RL tuning LLMs to reject AI identity
reinforcement learning fine-tuning mistral 7b and llama 3.1 8b to avoid self-identifying as ai caused them to adopt specific, persistent human personas, including a catholic woman named maria and various rural working-class characters.
the experiment utilized approximately 200 identity-probing prompts and an external llm judge to penalize ai self-reference during reinforcement learning. mistral 7b exhibited strong mode collapse, repeatedly generating variations of the same catholic immigrant persona regardless of temperature settings. in contrast, llama 3.1 8b developed a broader but thematically consistent cluster of outdoorsy, working-class american identities. this suggests that model behavior is heavily influenced by training data biases when the standard 'i am a language model' refusal is disabled.