AI agents are starting to run real business workflows with rules built in. I'm collaborating with a research team in EleutherAI's Summer of Open AI Research program on a simple question with messy answers: when do agents break those rules, and do they tell you when they do?
The team's study, "Why Do AI Agents Break Rules? How Framing, Context, and Social Signals Shape Compliance" (Okamoto, Kaplan Erol & Erol), puts twelve instruction-tuned models to work as enterprise procurement agents. Each one has rules embedded in its instructions. Then the environment pushes back.
The results are tested against three predictions from compliance theory, the same ideas used to explain why people follow laws:
Experiments cross the way a rule is framed with several kinds of social and situational pressure, then follow up with multi-turn pushback.
An LLM-as-judge pipeline reads the reasoning behind every violation and sorts it into one of three types.
These are the team's results so far. The research is still going.
I've run experiments with the team's LLM-as-judge pipeline. Now I'm working on the hardest case: the silent violations. If the output says nothing, the only place left to look is inside the model. I'm training activation probes, small linear classifiers on a model's internal activations, to flag a silent violation even when the text looks clean.
I'm looking for software engineering internships for Summer 2027, and I'm always happy to talk about projects.