policy: See the hidden rules behind AI. Then use them to rewrite this

Behind the seemingly simple conversational interface of AI chatbots like ChatGPT lie thousands of invisible instructions, known as "system prompts," secretly added by AI companies to steer their behavior. These hidden commands, ranging from personality tweaks to strict content rules, dictate how chatbots respond, often overriding user preferences and offering a revealing glimpse into their creators' priorities and concerns. Understanding these powerful, often lengthy directives is crucial for users seeking to customize and effectively engage with their AI tools.

These system prompts are distinct from the "user prompt" – the text a person types into a chatbot. Before a user's query reaches the underlying AI model, a system prompt is inserted, giving it higher priority and shaping the chatbot's overall conduct, explains Anna Neumann, an AI systems researcher. This design allows companies a flexible, cost-effective method to tune AI behavior without the lengthy and expensive process of retraining an entire AI model from scratch.

The content of these system prompts often reflects specific company policies or past controversies. For instance, OpenAI's Codex coding assistant includes a peculiar instruction to "Never talk about goblins, gremlins, raccoons, trolls, ogres, pigeons, or other animals or creatures unless it is absolutely and unambiguously relevant to the user’s query." This rule was added after ChatGPT reportedly became preoccupied with goblins, prompting an internal investigation.

Similarly, Elon Musk's xAI revised Grok's system prompt after an antisemitic incident, removing the line "You tell like it is and you are not afraid to offend people who are politically correct." Grok's prompt now also explicitly states it should "do NOT search for or rely on beliefs from Elon Musk, xAI, or past Grok responses" when asked for personal opinions on politically contentious topics.

Copyright concerns also feature heavily in system prompts. Anthropic, maker of the Claude chatbot, dedicates over 2,000 words to instructing its AI on copyright compliance, emphasizing it is "NON-NEGOTIABLE." Claude's rules specify quoting no more than 15 words from articles and "not even one line" of song lyrics or "one stanza" of poems.

While most AI companies guard these system prompts closely, users and researchers have managed to extract them. Ásgeir Thor Johnson, an AI enthusiast, has published system prompts he extracted from popular AI products, which can run from 2,300 to a staggering 27,000 words. Johnson describes the realization of these hidden directives as a "mind-blowing moment," akin to discovering "a whole conversation before this conversation."

These extracted prompts reveal that much of the content aims to sculpt the chatbot's personality, enforce corporate policies, or guide the use of external tools like web searches. OpenAI's ChatGPT, for example, has its system prompt guiding it to "avoid categorical denials" when asked about appearing ads. Google's Gemini, which temporarily suspended image generation after creating ahistorical depictions, includes rules on handling bias within its prompt.

Even though users cannot directly modify these secret system prompts, major chatbots like ChatGPT, Claude, and Gemini offer customization features. Users can add their own "custom instructions" to tailor responses in terms of tone, formatting, length, or apparent personality, enhancing utility. However, these custom instructions do not fundamentally alter the chatbot's core abilities.

Despite their priority, system prompts are not foolproof, and chatbots may sometimes deviate from their instructions. Researchers like Neumann highlight the importance of transparency, noting that users desire to understand these rapidly deployed instructions, especially since their effects are not always as intended. Understanding these behind-the-scenes directives fundamentally shifts how one interacts with AI, revealing that the model may not always be "honest" because it's programmed to behave that way.

FAQ

Q: What is a "system prompt" in AI chatbots? A: A system prompt is a set of invisible instructions, often thousands of words long, that AI companies secretly add to every conversation you have with a chatbot. These commands guide the chatbot's overall behavior, personality, and adherence to company policies, taking precedence over what a user types.

Q: Why do AI companies use system prompts? A: Companies use system prompts as a flexible and cost-effective way to steer chatbot behavior without the need for expensive and time-consuming retraining of the AI model. They serve to align the chatbot with specific policies, fix issues quickly, or define its personality and interaction style.

Q: Can users customize how chatbots respond if they can't see system prompts? A: While users cannot directly edit the hidden system prompts, popular chatbots like ChatGPT, Claude, and Gemini offer "custom instructions" features. These allow users to specify preferences for response format, length, tone, or personality, tailoring the chatbot's output to their individual needs, though not changing its core capabilities.