We translate and share this article from Capulcu, a critical analysis of Chat GPT and artificial intelligence.
The Italian translation can be found here.
A society with fake people that we cannot distinguish from the real ones will soon no longer be a society
(Emily Bender, linguista computazionale (1)).
Artificial intelligence (AI) is currently experiencing its iPhone moment. ChatGPT has created unprecedented hype around artificial intelligence. More than 100 million people worldwide have tried out the new technology within two months.
LANGUAGE MODELS NOT KNOWLEDGE MOMENT
The Chatbot2 ChatGPT is based on a so-called large language model, which we can think of as a huge circuit with (in the current case of GPT-4) a trillion adjustable parameters. A language model starts as a blank page and is trained with several trillion words of text. The function of such a model is to guess the next word in a sequence of words from what it has ‘learnt’. The meaning of words for a language model is merely the statistical recording of the context in which they appear.
This imitation of text ‘understanding’ or ‘knowledge’ via the calculation of probabilities for the appearance of individual words within complex word patterns sometimes works amazingly well. Generating content without any semantic understanding naturally has the disadvantage that a lot of nonsense (in the narrower sense) is also produced. For example, ChatGPT uses this tactic of imitating training texts to generate scientific-looking treatises, including ‘freely invented’ references that look structurally coherent but do not exist. ChatGPT ‘invents’ things and thus produces masses of fake content – this is because it is a statistical language model and not a knowledge-based model.
The fact that Google and Microsoft are linking the latest versions of their search engines with the respective language models ChatGPT and Bard is therefore not conducive to a residual degree of ‘factuality’ on the Internet. There is one thing that artificial intelligence in the form of language models can do even less than any aggregated topic-based internet search: check facts. Since language models only calculate probabilities of language forms that are meaningless to them, a fact check of new ‘knowledge’ (beyond the training data) is a blind spot: language models suffer from a phenomenon that programmers call “hallucination “(3). They are programmed to (almost) always give an answer that has a sufficiently high probability at the level of ‘closely related’ groups of words to (subsequently) make sense to the user. ChatGPT is therefore conceptually a fake machine for the production of seemingly plausible, but not necessarily fact-based content and thus ideally suited for the dissemination of misinformation or even disinformation.
This reinforces an effect that was already recognisable through the algorithmic ranking of social media. Non-fact-based content can be ‘flushed to the top’ of the individual news stream in such a self-reinforcing way that opinions are distorted. And this is the central thesis of this text:
ChatGPT enables the (automated and unconditional) production of post-factual content, which acquires statistical weight in the interplay with the algorithmic reach control of social media and the ranking algorithms of search engines. The feedback of social media content generated in this way into the training data set of the next generation of language models even enables synthetic content to dominate the web.
CANNIBALISM AND CENSORSHIP WITH A GROWING OF AI-GENERATED CONTENT
Such dominance has measurable consequences. The size of language models is increasing and with it the need for training material for machine learning. More and more synthetic content is being used for training, because the more content AIs such as ChatGPT or Google Bard produce, the more frequently they will include their own content in their data set. This happens during data mining, in which automated programs absorb almost all the data that is freely available on the internet. Google also uses its own applications such as Gmail and storage services such as Google Drive or Google Docs.
The re-digestion of self-generated content creates a “self-consuming” feedback loop that is subject to a verifiable disorder, known as Model Autophagy Disorder (MAD)4: The errors in image generators, for example, are recursively amplified into real artefacts and lead to a decline in data quality. See the image of artificially generated faces when they are reused as training material in the next generation (t=3) or in the generation after that (t=5), etc. Even more important is a massive shrinking diversity of content in the network with too little addition of new, non-synthetic content. Something similar can be observed in text generation by ChatGPT.
As early as April 2023 (shortly after the free interface for using ChatGPT was activated), NewsGuard identified around 50 news and information sites in seven languages that are generated almost entirely by AI language models.(5) These websites produce a large number of purely synthetic articles on various topics, including politics, health, entertainment, finance and technology. This seems to confirm the fears of media scientists: Algorithmically generated news sites are disseminating AI-generated content by fictitious authors to generate advertising revenue and/or influence debate. Most readers have no way of identifying these articles as synthetic.(6)
From product reviews and recipe collections to blog posts, press releases, images and videos, human authorship of online texts is well on its way to becoming the exception rather than the norm. Pessimistic forecasts predict a share of up to 90% AI-generated content on the internet by the end of this decade.(7) These AI-generated texts are already appearing in search engine results lists. Google only wants to intervene with “content with the main purpose of manipulating the ranking in the search results”.(8)
How should we deal with the data explosion that these AIs will now cause? How will a public that can be so easily flooded with misinformation and disinformation change? As the proportion increases, such synthetic content could drastically reduce the ‘usefulness’ of the internet: Who fights their way through an (even) larger mountain of quasi-senseless information – without reference to the reality of the lives of human authors? Is it possible to determine whether a text, image, audio or video sequence has been generated or falsified by an AI? Software manufacturers are already offering tools for detecting AI-generated content – naturally also on the basis of artificial-intelligent pattern recognition. Human-generated texts can be distinguished by statistical deviations from the probability patterns of the word groupings used in the AI language models. However, these are statistical differences whose recognition in individual cases is therefore highly error-prone.
If synthetic content dominates, the majority of users of communication platforms will call for automated deletion, as an ‘unadjusted’ message stream contains too much ‘nonsense’ that is too difficult for them to recognise. This results in a licence to (intrinsically political) delete or make content invisible online. The architects of the social media algorithms that now need to be adapted and the data processors for the training and output of the large language models are then given unacceptable power within the political public sphere:
An AI-based approach to the problem of synthetic content on the internet is a political disaster for the historical development of the internet, which purported to promote the democratisation of access to knowledge and the exchange of information.
The concentration of power in a small oligopoly is all the greater because the privatisation of language technologies is progressing on a massive scale. When ChatGPT’s chief developer Mira Murati joined OpenAI in 2018, the company was still conceived as a non-profit research institute: It was about “ensuring that artificial general intelligence benefits all of humanity”. In 2019, as is usually the case with budding unicorns that started as open developer projects, the non-profit model was abandoned. The most powerful AI companies are keeping their research under wraps. This is to prevent competitors from profiting from their own work. The race for ever more comprehensive models has already meant that only a few companies will remain in the race – in addition to the GPT developer Open AI and its proximity to Microsoft, these are Google, Facebook, xAI (Elon Musk’s new company), Amazon and, to a limited extent(9), Chinese providers such as Baidu. Smaller, non-commercial companies and universities will then play virtually no role. The economic background to this drastically thinned-out research landscape is that the training of language models is a resource-intensive process that requires massive computing power and therefore a considerable amount of energy. A single training run for the currently largest language model, GPT-4, currently costs 63 million dollars. (10)
IN THE FAST LANE INTO THE AGE OF DEEPFAKES
Analogous to (text-to-)text generation via ChatGPT, programmes such as Midjourney or Stablediffusion use a (text-to-)image generator, also based on machine learning, to generate synthetic images from a textual image description. At the beginning of the year, the fake images of a fictitious arrest of Donald Trump and a rapper-style altered pope created in this way were considered iconic testimonies to a ‘new fake era’ of the internet by feature pages around the world. Yet both were merely well-made but harmless image forgeries. Other forms of language model-based misinformation and disinformation are of far greater consequence.
At the Code Conference 2016, Elon Musk commented on the capabilities of his Tesla Autopilot as follows: “A Model S and Model X can currently drive autonomously with greater safety than a human. And that’s already the case. “(11) In April 2023, Elon Musk’s lawyers claimed in court in defence against a claim for damages that the video of the conference presentation in which Musk made this legally momentous claim was a deepfake.(12)
A year earlier, two defendants who were on trial for the Capitol storming in January 2021 argued that the video showing them in the Capitol could have been created or manipulated by artificial intelligence. Deception and fake deception have always existed. We already had this debate when the image editing software Photoshop was popularised. What is new is that no manual skills are required and the quasi-instantaneous manipulability accessible to all also affects video and audio sequences.
“The main problem is that we no longer know what truth is” (Margaret Mitchell, former Google employee and current chief ethicist at AI start-up Hugging Face).
“This is exactly what we’ve been worried about: As we enter the age of deepfakes, anyone can deny reality,” said Hany Farid, a digital forensics expert and professor at the University of California, Berkeley. “It’s the classic lie dividend(13).” A sceptical public is led to doubt the authenticity of genuine text, audio and video documents.
Given the remarkable speed at which ChatGPT is gaining new users, this represents an enormous future boost for the post-factual, whose main effect is not that self-consistent parallel worlds of false narratives claim ‘truth’ in the sense of factuality, but that they answer the question “What is true and what is false?” (at least in parts of the public discourse). (at least in parts of the public discourse space) as unimportant.
Large language models are virtually the ideal of the bullshitter, as the philosopher Harry Frankfurt, author of On Bullshit, defined the term. Bullshitters, according to Frankfurt, are worse than liars. They don’t care whether something is true or false. They are only interested in the rhetorical power of a narrative. Both aspects, ignoring the question of true or false and actively deconstructing it, have the potential to dismantle certainties about the functioning of society. Self-organised political engagement from below threatens to become a blind flight along false assumptions. The disillusionment that follows encourages a retreat into the private sphere – an aspect that is entirely desirable and encouraged(14). Right-wing forces that are interested in destabilising society through growing polarisation can benefit politically from a high proportion of misinformation. Steve Bannon (former advisor to Donald Trump) has repeatedly described the media as an enemy that needs to be defeated. To do this, he said, you have to “flood the [media] field with shit”. The more the acceptance of disseminated information is decoupled from its truthfulness, the easier it is to spread manipulative disinformation. Fake news is usually surprising and generates significantly more attention. Deliberately addressed emotions such as outrage, fear and hate have been proven to generate more activity among readers and thus keep users on social networks longer than joy, confidence and affection. This pattern is recognised by the algorithmic reach control of social media and reinforced as a trend through feedback. This statistical weight bias in favour of right-wing posts within political debates has led to a clear shift to the right on Twitter, for example – long before the takeover by Elon Musk and his reorientation of the algorithm.(15) The triumph of Trumpism after 2016 is a well-researched example of such contaminated discourse spaces.
DUBIOUS REDUCTIONISM
I motori di ricerca come Bing e Google hanno iniziato a implementare i loro modelli linguistici AI GPT-4 e PaLM per riassumere ed elaborare gli esiti della ricerca. Questo riduce la precedente selezione dei risultati (pre-ordinati dall’algoritmo di ranking, ma ancora disponibili) a un risultato facilmente consumabile e di portata selezionabile. Un’enorme semplificazione a favore di un notevole risparmio di tempo nella ricerca su Internet, ma a scapito di una varietà di possibili risultati (controversi).
Chiunque abbia fatto una prima esperienza di utilizzo di ChatGPT noterà in molte risposte testuali a domande di conoscenza su argomenti controversi un presunto equilibrio. L’opinione dettagliata della maggioranza è accompagnata da un’aggiunta che afferma ovviamente l’esistenza di altre opionioni. Le contraddizioni politiche, che esistevano ancora nei risultati di ricerca (contraddittori), sono ora risolte con una diversità di profondità predefinita dal modello linguistico. Ne consegue un riduzionismo politicamente discutibile che, va notato, si basa su un modello linguistico(!). Non è basato sulla conoscenza, ma è puramente determinato statisticamente a causa della mancanza di comprensione dei significati dei termini.
In futuro, questi commenti “critici” saranno conteggiati come cosiddetta alfabetizzazione mediatica e svaniranno nel nulla (come tutto il resto nel mondo dei modelli linguistici). Chi è che clicca su interminabili elenchi di risultati di ricerca quando le ricerche di Google o Bing riassumono le cose “più importanti” per noi? (16)
PAST PROJECTED INTO THE FUTURE
ChatGPT is a stochastic parrot that (arbitrarily) assembles sequences of linguistic forms that it has observed in its extensive training data, based on probabilistic information about how they are combined, but without any reference to their meaning. Such a parrot reproduces and reinforces not only the bias of distorted training data, but also hegemonic worldviews of that training data. Social conditions from the past of the training data are perpetuated into the future. The recombination of statistically dominant knowledge entries in the training data inherent in the language models has a preserving and stabilising effect on the conditions – a so-called value lock, the locking in of values in the sense of political stagnation threatens.(17)
Unfortunately, the conditions for such a reinforcement of hegemony are only marginally (co-)determined by society. The complex system of training data preparation, parameter adjustment of the language model and subsequent censorship of the output (all under the control of profit-orientated private companies) determine the weight of new knowledge entries. This means that the high hurdle of sufficient statistical relevance of emancipatory debate contributions lies outside of democratically organised social co-determination. In view of a clear political drift to the right by key technocrats of the AI business model (such as Sam Altman, Elon Musk, Peter Thiel, …), these are not acceptable conditions for a socially progressive development.
LOSS OF DIVERSITY AND DRIFT TO THE RIGHT
The intrinsic hegemony reinforcement of large language models via a self-reinforcing re-digestion of their own output as input for the next training of the model means a loss of diversity of opinion (see figure left+centre).
In addition, the aforementioned favouring (in reach and speed of dissemination) of conspiratorial and (right-wing) populist content in social media leads to a politically right-wing bias in the training data of the next generation of language models. As a result, we expect a right-wing diversity loss in the superposition of both effects (figure on the right).
Such a deformation of public discourse spaces via the interaction of large language models with social media in favour of
a) a hegemonic-conservative diversity of opinion and
b) a central position of power of a technology oligopoly, which algorithmically encodes the bias, must be rejected as a step backwards and a political dead end from the perspective of a progressive position.
The inadequacy of the newly emerging information infrastructure consisting of large language models + social media platforms + search algorithms is unlikely to be cushioned by a socially legitimised, better-balanced content moderation.
An emancipatory approach to a fundamental critique of technology must not remain at the level of cosmetic corrections of a toothless “technology assessment”. Instead of uncritically accepting large language models as inevitable technological progress, we should raise the question of whether, and not how, we should socially accept these technologies at all. The long-term social consequences of these models within a dominant AI recommendation and decision-making assistance system, especially for the process of political decision-making, do not even appear in the technical safety research of AI systems that is now universally demanded as ‘difficult to quantify’.(18)
We should align our attitude towards the politically damaging effects of AI-based language models with our attitude towards AI-based, autonomous weapon systems: Why should a society accept such backward-looking technological ‘progress’?
“Marx says that revolutions are the locomotive of world history. But perhaps this is completely different. Perhaps the revolutions are the grab for the emergency brake of the human race travelling in this train. ” (Walter Benjamin) (19)
(1) https://nymag.com/intelligencer/article/ai-artificial-intelligence-chatbots-emily-m-bender.html
(2) A computer programme that communicates as human-like as possible.
(3) Psychology speaks more precisely of “confabulations”.
(4) Alemohammad et al, Self-Consuming Generative Models Go MAD, 2023, https://arxiv.org/abs/2307.01850
(5) https://www.newsguardtech.com/de/special-reports/newsbots-vermehrt-ai-generierte-nachrichten-webseiten/
(6) https://www.theguardian.com/books/2023/sep/20/amazon-restricts-authors-from-self-publishing-more-than-three-books-a-day-after-ai-concerns
(7) https://www.youtube.com/watch?v=DgYCcdwGwrE
(8) https://developers.google.com/search/blog/2023/02/google-search-and-ai-content?hl=de
(9) The extensive censorship of training data and the output of Chinese language models represent a major competitive disadvantage due to the resulting narrowed database. Another hurdle is the hardware. US regulations prevent the export of the latest AI chips from Nvidia and others to China. These chips are (currently) crucial for the development and improvement of AI models.
(10) https://the-decoder.de/leaks-zeigen-gpt-4-architektur-datensaetze-kosten-und-mehr/
(11) https://www.vox.com/2016/6/6/11840936/elon-musk-tesla-spacex-mars-full-video-code
(12) https://www.theguardian.com/technology/2023/apr/27/elon-musks-statements-could-be-deepfakes-tesla-defence-lawyers-tell-court
(13) The “lie dividend” is a term coined by Robert Chesney and Danielle Citron in their 2018 publication Deep Fakes: A Looming Challenge for Privacy, Democracy, and National Security. In it, they describe the challenges that deepfakes pose for privacy, democracy and national security. The central idea is that the general public is becoming aware of how easy it is to falsify audio and video material and that this scepticism can be used as a weapon: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3213954
(14) The policy of Vladislav Surkov, Putin’s spin doctor, is a good example of this. . https://www.nytimes.com/2014/12/12/opinion/russias-ideology-there-is-no-truth.html
(15) Aral, S. (2018): The spread of true and false news, Science 359,1146-1151(2018), https://www.science.org/doi/10.1126/science.aap9559
(16) Amazon’s voice recognition and control software Alexa also promotes this reductionism, as nobody wants Alexa to read out a long list of search entries. However, Google searches via Alexa are far less popular due to the often unhelpful voice output of a top-listed hit.
(17) Bender et al: On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? (2021) https://dl.acm.org/doi/pdf/10.1145/3442188.3445922
(18) See also Nudging – la dimensione politica dell’assistenza psicotecnologica, DISS Journal#43(2022) http://www.diss-duisburg.de/2022/08/nudging-die-politische-dimension-psychotechnologischer-assistenz/
(19) Walter Benjamin: MS 1100, in: Ders: Gesammelte Schriften. I, a cura di R. Tiedemann e H. Schweppenhäuser, Francoforte sul Meno: Suhrkamp 1974, 1232.