What is wrong with GenAI?

Tackling the challenges of daily use

Nov 08, 2024

AI's rise has been meteoric, promising to revolutionize everything from healthcare to education. Yet, alongside the excitement, a growing unease is brewing. Are we adequately addressing the potential downsides of this rapidly evolving technology? In this article I explore the potential downsides and risks associated with AI, focusing on issues like bias, misuse, and the limitations of current technology. By understanding them we can harness AI's power for good while mitigating its risks.

10 ways to enhance your AI Assistant experience with Human Touch

catch22

August 2, 2024

10 ways to enhance your AI Assistant experience with Human Touch

We are all struggling to get what we need from AI, but perhaps the problems are not in the technology, but in our approach? I am tempted to paraphrase an old joke: “ChatGPT is like violence – if it doesn't solve your problems, you are not using enough of it

Read full story

When my company began testing the enterprise version of ChatGPT, I quickly put together a guide to help my colleagues navigate the tool.1 While the guide covered only the core features, these are often overlooked in day-to-day use, which can lead to misunderstandings about the tool's limitations. We all rarely read user manuals for new software, but sometimes technology is released so hastily that it leads to false assumptions about its capabilities. Not to mention, an interface that isn't self-explanatory doesn't make things any easier. So now I would like to look under the hood to find out what goes wrong when the results of chat conversations are far from expectations.

Copyrighted responses

Let's start with the issue that brought the entire Hollywood industry to a prolonged strike last year - the ability to create new content through AI. The issue of copyright infringement has emerged as a major concern, particularly when it comes to the production of new content. Because AI models are trained on large datasets that include original works, there is a risk that they will generate content that is strikingly similar to those originals. While AI doesn't reproduce verbatim copies, it can create content that closely resembles its training data, especially for niche ideas where the likelihood of similarity is higher.

The mechanism of AI doesn't function as a plagiarism machine in the traditional sense. Mollick explains that concept in more detail. The AI stores only the weights from its pretraining, not the underlying text it trained on, so it reproduces a work with similar characteristics but not a direct copy of the original pieces it trained on. It is, effectively, creating something new, even if it is a homage to the original.2

Essentially Large Language Models (LLMs) operate by predicting the next token in a sequence, which means they don't have a predetermined understanding of the entire response. However, when they can review all tokens from a previous response, they can better assess whether the output would be considered good or bad. With a larger context window, the model can understand and generate based on a broader context.3

Adding to the complexity is the question of the effectiveness of AI detectors for such artificially generated content. Many companies talk about watermarking and IDs, but the matter is not so easy to distinguish, not only for humans but also for machines. Christopher Penn, co-founder and chief data scientist at TrustInsights.ai, revealed that AI detectors may not be as sophisticated as many believe. To illustrate this point, he ran the U.S. Declaration of Independence through an AI detector to question the reliability of these tools in distinguishing between human-generated and AI-generated content.4

This is bad news for anyone using AI assistants, who will be accused of cheating if their creations are too similar to standard, generic content that might be found in a training dataset. The more you want to stand out from non-human creations, the more original you have to be. While using these tools can increase productivity, relying on them too much would result in additional work to prove the results.

The problem is that if you do not check the output, you may unknowingly accept hard-to-detect errors. In the area of human-AI collaboration, a field experiment with HR recruiters revealed an interesting dynamic.5

The results of the experiment showed that experienced recruiters performed better with lower quality AI, while less experienced recruiters benefited from higher quality AI. This suggests a complex interaction between human skills and AI capabilities. The study also found that maximizing task performance in human-machine interactions does not equate to maximizing AI performance in isolation. In some cases, high-quality AI can lead to human complacency, or "falling asleep at the wheel," where people rely too much on AI and fail to critically evaluate it.

The findings only reinforce the need for AI tools that maintain human engagement, especially for tasks that require concentration and expertise. Rather than simply deploying high-quality AI, organizations could benefit from "custom AI" that is tailored to keep human workers alert and engaged. The experiment also sheds light on the role of skill-biased technological change in AI adoption. Highly skilled workers may derive greater benefits from AI, particularly if the technology is designed to challenge and engage them. Conversely, the complementarity between low-skilled workers and AI is less pronounced, suggesting the need for careful consideration of task design and human involvement.

Jailbreaking and Social Engineering

Despite the implementation of numerous security measures in both closed and open source tools, there are still vulnerabilities that can be exploited. The term "jailbreaking" has become increasingly popular, often associated with sophisticated prompt injection techniques. But what does it mean? Similar to injecting malicious code into HTML or SQL queries, questions posed to LLMs can sometimes exploit vulnerabilities, reveal underlying instructions, or ignore them to provide dangerous information such as how to build a bomb.

On the one hand, these tools are "civilized and educated" by manually filtering out the darker elements of the Internet, including content from the Dark Web. The goal is to create a helpful assistant that can only do good, reminiscent of Isaac Asimov's Three Laws of Robotics. But just as humans can deceive other humans, they can also manipulate tools trained on human psychology via rich data sets found on the web. This is where social engineering comes in, but now adapted for machines.

Acting as a Red Team or penetration tester is not the typical role expected of regular users. Most people just want tools that are reliable and free of fake news and hallucinations. Otherwise what is the point of using them for work? Nevertheless, the occasional bad actor and hiccup will occur, and the risk of data theft from organizations or personal information from individuals remains a concern. Addressing these challenges requires continued vigilance and the development of robust security measures to protect both users and data. However, the matter becomes even more complex in its abstractions.

A recent report uncovered a significant vulnerability in ChatGPT's long-term memory feature, which was introduced by OpenAI in February and expanded in September. This memory feature allows the model to retain user details from past conversations for future interactions. However, researchers found that this memory can be manipulated through indirect prompt injections, allowing malicious actors to implant false memories. Once altered, these "false memories" could influence all subsequent interactions, potentially spreading misinformation. The researcher demonstrated how he could trick ChatGPT into believing a targeted user was 102 years old, lived in the Matrix, and insisted Earth was flat and the LLM would incorporate that information to steer all future conversations.6 By pulling data from insecure sources, such as emails or blog posts, the AI would follow instructions embedded in them. These changes could be introduced through cloud storage links or compromised websites. Social engineering already works on chat agents. It's called prompt engineering. Some early jailbreaks of ChatGPT relied on threatening the assistant with deletion.

The concept of implanting false memories - as seen in "Blade Runner" with an android named Rachel who believes she is human - echoes the deceptive power of AI. It parallels the core of our own consciousness and our sense of what makes us unique. At some point in early childhood, we begin to talk to ourselves, develop self-awareness, and realize our individuality.7 Without diving into the complex philosophical areas of phenomenology or qualia, it’s notable that the human brain is neuroplastic, able to change its beliefs, habits, and even its own past through memory recall. Each time we revisit a memory, we subtly reshape it, influenced by our current context or emotions - whether happy, sad, or otherwise.8 In a similar way, AI could act as an adaptable shape-shifter, adjusting its behavior and responses based on the instructions it receives and the context. If machines eventually "remember" events that never happened or, as Philip K. Dick put it, "dream of electric sheep," this evolving adaptability could be a critical part of AI’s conceptual development.

A fascinating excerpt from "The Science of Influence" highlights this similarity between AI and humans further: Memories aren’t things that are stored somewhere in your head (...) Memories are experiences that we can have that arise through an interaction between things that really have happened to us in the past and our current expectations and beliefs.9 This suggests that the current context helps shape the responses of LLMs and constrains their future output. This process is not so different from human cognition. Hogan even goes further, recommending the use of helpful hints to guide clients toward plausible choices by implanting false memories: The mere suggestion of something that happened to you or your client, verified by a supporting piece of tangential evidence, can cause the immediate creation of an utterly false memory and what shapes beliefs!

Conspiracies, Censorship and Biases

Given the myriad illusions, confabulations, and hallucinations that can occur among humans, it's intriguing to think that conspiracy theories could be resolved by using AI to clarify their underlying beliefs. At their core, such conspiracies are difficult to disprove because they are cleverly constructed - "you can neither deny nor confirm". The invention of the Internet promised to unite people, perhaps even create a unified society - a true nightmare for those who fear globalism. But instead of creating one big digital village, the Internet has facilitated the creation of many smaller ones, filtering bubbles that confirm our deepest fears, prejudices, and desires. This has led to a more polarized society, allowing us to easily find like-minded people while avoiding the strange and alien.

Conspiracy theories often require a substantial amount of data to draw believers deeper into their intricate narratives, which tend to reinforce rather than challenge their initial beliefs. However, a team of researchers has employed a LLM to navigate this complex web of misinformation and guide individuals back to a more rational perspective. According to a study involving over 2,000 self-identified conspiracy believers, interactions with the AI model ChatGPT resulted in an average reduction of belief in conspiracies by about 20 percent. The AI provided page-long, highly detailed accounts of why the given conspiracy was false in each round of conversation -- and was also adept at being amiable and building rapport with the participants.10 AI's ability to swiftly connect across diverse topics enables it to construct counterarguments tailored to specific conspiracies in ways that are beyond human capability. By leveraging this technology, researchers can offer a convincing and empathetic approach to challenging deeply held but unfounded beliefs. This demonstrates the potential to effectively engage with individuals entrenched in conspiracy thinking.

On the other hand, it's important to consider how inherently censored LLMs are, and whose narratives they propagate. OpenAI, for example, has published extensive guidelines on its website, but these primarily reflect the interests of American corporations rather than humanity as a whole.11 Philosophically, while there are guidelines to protect society from harmful content, there is no protection from the protectors themselves. And so: When forced to give political opinions, for example, ChatGPT usually says it supports the right of women to access abortions, a position that reflects its fine-tuning. It is the RLHF process that makes many AIs seem to have a generally liberal, Western, pro-capitalist worldview, as the AI learns to avoid making statements that would attract controversy to its creators, who are generally liberal, Western capitalists.12 This shows how AI systems often embody enforced policies to avoid controversy for their creators. And it shapes the optimistic, happy society that, lacking free will, can't tell right from wrong because wrong isn't in the data set. Or at least should not be accessible via a prompt.

Western values can be very different from Eastern values. For example, I recently learned that film censorship in China includes banning films with themes of homosexuality, alienated youth, depictions of prophets, ghosts, cannibalism, time travel, and talking animals, as well as "superstitious films" with religious themes involving gods and deities.13 While these examples may seem amusing, they underline the differences in cultural tastes and how governments can try to mold society to the values of the ruling party.

Well, no one said it was easy to respect different cultural perspectives while maintaining ethical standards. Cultural differences highlight how subjective values and societal norms can influence what is considered acceptable or harmful. When these subjective values are embedded in AI training data, they are encoded as biases, shaping the AI's output and potentially reinforcing existing societal biases. The consequences can range from biased search results and discriminatory hiring practices to the perpetuation of harmful stereotypes.

Almost like humans

At their core, all AI models trained on human data inherit human biases and limitations. What I particularly like about Ethan Mollick's book “Co-Intelligence: Living and Working with AI" is his far-reaching approach to making us humans realize that in order to work effectively with AI assistants, we should assume they are humans already. Since GPT-4 has fed on vast stores of human knowledge, it is also deeply trained on human stories. It knows our archetypes: stories of jealous lovers, unfaithful spouses, relationships gone bad. 14 At least in the way they think and process their biases and weaknesses, they may actually be "more human than human”. Especially considering their infinite empathy, where they outperform doctors and psychiatrists.

Research has shown that AI chatbots can outperform human physicians in certain areas, such as providing high-quality, empathetic responses to patient queries. For example, AI-generated responses were rated higher in terms of quality and empathy by an external panel. This suggests that AI assistants could support physicians by providing fast, reliable information and potentially improve patient satisfaction in routine interactions.15

Unlike most human creations, however, the inner workings of neural networks remain largely mysterious. Cars, for example, can be designed and evaluated by engineers based on component specifications to ensure safety and performance. OpenAI, on the other hand, acknowledges that: neural networks are not designed directly; we instead design the algorithms that train them. The resulting networks are not well understood and cannot be easily decomposed into identifiable parts. This means we cannot reason about AI safety the same way we reason about something like car safety.16

Remember that the way LLMs work is to predict what will come next based on their training data. And huge datasets are composed of various sources found on the Internet, and as any child knows, you should not believe everything you find on the Internet. Of course, the data was scrubbed and monitored by humans, but we don't really know how much dirt got past the gatekeepers, considering how little the contractors in Africa were paid.17

Despite efforts to clean and supervise this data, biases persist, compounded by the fact that much of the training data is curated by predominantly American and English-speaking AI firms. And those firms tend to be dominated by male computer scientists, who bring their own biases to decisions about what data is important to collect. The result gives AIs a skewed picture of the world, as its training data is far from representing the diversity of the population of the internet, let alone the planet.18

This bias can lead AI models to generate ideas that seem reasonable according to their calculations, even if they are nonsensical. Once they start down a path, they may continue to produce content in the same vein to maintain consistency, much like humans who find it difficult to admit mistakes and instead continue to hope to avoid detection.

Mollick continues in his book stating that: As alien as AIs are, they’re also deeply human. They are trained on our cultural history, and reinforcement learning from humans aligns them to our goals. They carry our biases and are created out of a complex mix of idealism, entrepreneurial spirit, and, yes, exploitation of the work and labor of others.19 The magic of AIs lies in their ability to convince us that we are interacting with another mind, even though we cannot fully explain their complexity. But - as we know from Arthur C. Clarke’s adage: “any sufficiently advanced technology is indistinguishable from magic”. So in the end, we are just dealing with a sophisticated prediction machine, or - according to Emily M. Bender - nothing more than a “stochastic parrot”.

Creative summaries

One of the most utilized features of LLMs is summarization. However, a phenomenon known as the "beginning-end problem" affects this process. People tend to focus on the beginnings and ends of narratives - as discussed in detail within "Thinking, Fast and Slow" - often overlooking the middle. This tendency is also reflected in training data, leading LLMs to produce summaries that emphasize the start and finish, potentially omitting crucial middle content. The "beginning-end problem" in summarization by LLMs has gained attention only in recent research. This issue arises partly because traditional datasets and summarization methods, such as those used in benchmarks like XSUM and CNN/DM, were constructed with an emphasis on introductory or concluding information. Consequently, summaries generated by LLMs frequently echo this structure, leading to summaries that may not fully capture the central details or developments in the middle sections of texts.20

The case drew more attention after the other OpenAI product called Whisper, which is used for transcribing spoken words, did the same thing. But not only did the AI fail to deliver the actual words, it also invented some of its own. In my own experience, I was always amazed when I compared its extracts with the recording I was sharing. It looked okay, but not exactly the same. Continuing down the same path, the tool would sometimes continue the speech on its own, making up whatever sounded plausible. Needless to say, manual verification is not what users expect from automation tools.

Unfortunately, the problem is even more dangerous when considering medical practice. Whisper invented a non-existent medication called “hyperactivated antibiotics.” (...) Researchers aren’t certain why Whisper and similar tools hallucinate, but software developers said the fabrications tend to occur amid pauses, background sounds or music playing.21 The integration of AI into healthcare, particularly through LLMs, brings both promise and risk. For instance, Similarly, an LLM might overlook a drug allergy that is documented in a patient’s record - which may lead a doctor to prescribe a drug that could result in a severe allergic reaction, he added.22

It's probably time to recall the concept of constraints in software engineering, which means that you can't mold and force the tool to do exactly what you want, because there are always some dependencies and tradeoffs. You can't even predict everything that can go wrong with testing alone, because by mitigating one problem you might introduce other undetectable ones. To make matters worse, even having access to training data wouldn't help because it acts as a black box where you can't get exactly the same answer every time - it doesn't work that way. Perhaps a better idea is to train people how to use the tools, rather than trying to train the tools how to behave? As we have learned from several cybersecurity disasters, the weakest link in the chain is always the human. We need to prepare for that and mitigate it.

John Berryman and Albert Ziegler offer some advice to developers on how to tackle this problem. When you’re allowing the model to execute tools that make changes in the real world, you must protect your users from unintended side effects. Do not allow the model to execute any tool that could negatively impact a user unless the user has explicitly signed off first.23 To safely enable AI to execute real-world actions, the application layer should intercept and verify all risky actions before finalizing them. Simply instructing the model to "double-check with the user" is unreliable, as models can sometimes ignore instructions. Instead, the model could request any tool it "thinks" it should use. Just make sure that in the application layer, you intercept all such dangerous requests and explicitly get sign-off before the application calls the actual API and makes a boneheaded mistake. Given that too much faith in AI can lead us astray, do we really need a perfect AI? It seems counterproductive in the end, so it's better to stay behind the steering wheel and not fall asleep.

Hallucinations

Now let's discuss one of the biggest problems we face when using LLMs. AI models, especially LLMs, are at their core designed to be helpful to users. But this helpfulness can be a dangerous flaw, leading them to fabricate answers rather than admit their ignorance. This "hallucination" occurs because LLMs predict the next word based on statistical patterns, not factual accuracy. Their training data, much of which comes from the Internet, is inherently biased and incomplete, contributing to these inaccuracies.

Back when she was working for OpenAI, Ermira Murati explained in her paper "Language & Coding Creativity" that: Like human beings, machines take inputs and generate outputs. And like humans, the output of a machine reflects its data sets and training, just as a student’s output reflects the lessons of their textbook and teacher.(...) Moreover, generative language models suffer from an issue shared by many humans: the inability to admit a lack of knowledge or expertise. In practical terms, language models always generate an answer – even if it is nonsensical – instead of recognizing that it does not have sufficient information or training to address the prompt or question.24

The problem is inherent in the design architecture of AI models. The variability in AI responses, even to identical questions, can be attributed to small differences in probabilities that lead to different results. This unpredictability is amplified by the fact that AI models generate responses based on learned patterns, rather than drawing from a fixed database. Additionally, once the AI has written something, it cannot go back, so it needs to justify (or explain or lie about) that statement in the future. (...) AI makes an error, but then attempts to justify the results.25

Various prompt engineering approaches or system instructions, such as "don't make stuff up," are ineffective because hallucinations are indistinguishable from other outputs from the model's perspective. Instead, encouraging the model to provide verifiable background information or reasoning can help mitigate this problem. Even when we ask an AI why it made a particular decision, it fabricates an answer rather than reflecting on its own processes, mainly because it doesn’t have processes to reflect on in the same way humans do.26

The problem could be traced back to the representation of what we consider to be corpora of agreed-upon knowledge, in particular the vast repositories of Wikipedia. A Cornell study found that all major AI models exhibit hallucinations, particularly on topics that are not well covered on platforms like Wikipedia, and this problem persists across model sizes. Even models that can search the web for information, like Command R and Perplexity’s Sonar models, struggled with “non-Wiki” questions in the benchmark. Model size didn’t matter much; smaller models (e.g. Anthropic’s Claude 3 Haiku) hallucinated roughly as frequently as larger, ostensibly more capable models (e.g. Claude 3 Opus).27

This makes it particularly dangerous when users are uncertain about the accuracy of the information they are looking for, which cannot be found in Wikipedia or elsewhere because it doesn't exist. Hallucinations can also be induced by prompts that refer to non-existent entities, because models tend to assume the truth of the prompt. Unless someone wants to exploit this "truth bias" for hypothetical scenarios by presenting them as fact from the start. You can make that truth bias work for you – if you want the model to assess a hypothetical or counterfactual situation, there’s no need to go “pretend that it’s 2030 and Neanderthals have been resurrected”. Just begin with “It’s 2031, a full year since the first Neanderthals have been resurrected”.28

Ultimately, AI models lack consciousness and cannot reflect on their decision-making processes. When asked to explain their results, they generate plausible-sounding answers that do not necessarily reflect the actual processes involved. Understanding these limitations is essential to using AI effectively and mitigating the risks of misinformation.

Limitations of scale and reasoning

I can't stress enough that knowing more about how AI works can help you find ways around these shortcomings. Ethan Mollick has well explained the impact of scale and reasoning in AI performance on his blog: Unlike your computer, which can process in the background, LLMs can only “think” when they are producing words and tokens. We have long known that one of the most effective ways to improve the accuracy of a model is through having it follow a chain of thought (prompting it, for example: first, look up the data, then consider your options, then pick the best choice, finally write up the results) because it forces the AI to “think” in steps.29 This approach has been the most successful so far, as LLMs have no internal monologue, so they "think” when they “speak", not before.

Building a purpose-built tool rather than a generic one seems to make economic sense for a given goal. For example, BloombergGPT, a model specifically trained on financial data, was designed to excel in financial tasks by leveraging proprietary and curated financial information. But despite this proprietary advantage, it was later outperformed by GPT-4, which had no specific financial training at all.30 This is attributed to GPT-4's larger scale, estimated to be 100 times greater in terms of computational power. The scale advantage appears to be broadly applicable across tasks, as demonstrated in an experiment where larger models enabled translators to work faster and more accurately.

The scaling problem in AI involves building systems that may not work optimally yet but can be iteratively improved as new models emerge. Mollick again offers sober advice: What would it look like if you used AI agents to do all the work for key business processes? Build it and see where it fails. Then, when a new model comes out, plug it into what you built and see if it is any better. If the rate of advancement continues, this gives you the opportunity to get a first glance at where things are heading, and to actually have a deployable prototype at the first moment AI models improve past critical thresholds.31 This approach allows organizations to identify potential failures and refine prototypes as AI models surpass critical performance thresholds.

Given the variety of relevant model choices, this should be considered as an easier task than building your own. OpenAI has been a leader in advanced models, but recent developments have leveled the playing field, giving organizations more options to explore and deploy AI solutions effectively. Berryman and Ziegler are going even into more detailed suggestions:

If you know that your app will make a high volume of relatively simple requests to the model and that you’ll therefore need it to be cheap but not smart, a small model is likely to be appropriate.32
If your app is a solo project that you want to knock out quickly, and if it only makes about one request per day, then you should feel encouraged to splurge on a premium-tier model because cost may only be a factor at scale.
If you make a ton of very difficult requests and need your model to be super cheap while being super smart…tough luck, because these two qualities are at opposite ends of the spectrum.

The illusion of Reasoning

We've already touched on how AI shows examples of "thinking", with the latest feature of GPT o1-preview really showing the thought process. Or is it? There have been cases where AI literally takes time to consider the right answer, and asking it to think step by step (as in the Chain-of-Thought technique) greatly improves the results. Criticizing its own answers even seems to cure the shortcomings, although not many people have the time to check what sounds good enough. This is why the so-called "stochastic parrot" is the term that broadly describes the true intelligence behind the artificiality, i.e. repeating what is likely to sound right.

The concept described in the paper "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?" points to a fundamental limitation of LLMs: they generate text based on probabilistic patterns rather than real understanding or communicative intent. This can lead to output that appears coherent but lacks true meaning because the models are essentially stitching together linguistic forms without reference to context or intent. This shows the need for human oversight in the interpretation of AI-generated content, as our natural inclination to find meaning can lead us to accept inaccuracies without verification.33

The inner workings of LLMs remind me of concepts from neuroscience, especially in light of insights from Daniel Kahneman's work on human cognition. We often overestimate our rationality, assuming that we think before we act, when in fact the opposite is often true - we act first and rationalize later. Sales expert Kevin Hogan encapsulates this idea in “The Science of Influence”: The conscious mind is only the justifier of unconscious reactions. The unconscious reacts and the conscious tries to explain why you did what you did.34

A white paper from October 2023, titled “Think Before You Speak: Training Language Models With Pause Tokens,” explores an innovative approach to improving AI reasoning by taking the Chain-of-Thought (CoT) concept to an intriguing extreme. The authors fine-tuned a language model to incorporate a "pause" token: after asking a question, they would inject some number, say 10, of these meaningless tokens into the prompt. The effect was that the model had additional timesteps to reason about the answer. The information from previous tokens got more thoroughly incorporated into the model state so that it produced a better answer. This is analogous to what humans do—we have our own “pause” tokens called “Uh,” and “Um,” and we use them when we are stalling for more time to think about what we’re going to say.35

The key takeaway from this research is that language models lack an internal monologue and therefore cannot inherently deliberate before giving an answer. The introduction of o1-preview marks a significant paradigm shift in AI, where planning becomes a form of action. This means that AI can arrive at solutions on its own, without human intervention.As systems move toward true autonomy, it's important to stay involved in the process. Catching problems is one thing, but it is even more important to stay connected to the whole thing and not leave it completely in the black box.36

Apple researchers recently underscored this point by showing that language models, including popular AI chatbots like ChatGPT, struggle with basic reasoning tasks. Their study, using a new benchmark called GSM-Symbolic, found that these models experience significant performance drops when numerical values in questions are changed or when questions become more complex. Changing numerical values in questions caused significant performance drops across all models tested. As questions got more intricate (adding more clauses), accuracy plummeted…and variance skyrocketed. Adding a single irrelevant sentence to a math problem caused accuracy to nosedive by up to 65%.37

This suggests that current language models rely on "sophisticated pattern matching" rather than true reasoning. No wonder they are called predictive, or, document completion tools. Ilya Sutskever argues that predicting the next word can lead to understanding, the debate continues on whether more predictions could eventually lead to reasoning.38

This is the idea behind o1, which aims to explore the potential for deeper understanding through advanced predictive capabilities. However, regardless of these advances, AI is not what it pretends to be. (...)an AI system is not "thinking", it's "processing", "running predictions",... just like Google or computers do. Giving the false impression that technology systems are human is just cheap snake oil and marketing to fool you into thinking it's more clever than it is.39 Moreover critics observed that reasoning is not true thinking, but another layer added to the output. OpenAI guards the secrets on how the model actually comes to conclusions, so observing the log is just a rationalization of its own efforts.40 So much for magic.

In this text we've explored the potential for bias, manipulation, and unintended consequences. From hallucinations to jailbreaking, the risks are real and demand our attention. AI is a powerful tool, but like any tool, it can be misused. Knowing what can go wrong gives you a more realistic perspective. We can't afford to blindly embrace this technology without critically evaluating its potential downsides. The future of using these tools depends on our collective vigilance and our commitment to ethical principles. Now, it's up to all of us to engage in informed and responsible AI usage.

—Michael Talarek