The Automated Company, part 3: supervisor, executor, critic

2026-11-03

This is the third part of “The Automated Company” series. In part one I wrote about why at Element we hand routine work to automations. In part two I showed the company map, a list of processes and automations that is a good place to start. Today I describe who builds these automations for us and who checks them. If you read Polish, there is also a Polish version of this article.

An AI agent that has written code and tested it itself usually reports that everything works. That report sounds convincing even when the code has a serious bug. That is why we split the work between three roles.

Three roles

The supervisor plans the work, divides it into parts, writes a detailed specification, checks the finished code line by line and decides whether a change goes into the running system.
The executor writes code and tests according to the specification.
The critic is a separate agent that receives the finished result, the goal and the rules that must not be broken. It knows neither the supervisor’s reasoning nor the history of the conversation. It looks for errors, and the decision about what to do with them stays with the supervisor.

I set the goals and settle whatever the agents should not settle on their own. This is how our automations work and how we develop Element, both the code of the recruitment system and the automation of the company’s business processes.

How to set up these roles

In practice, a role is a few paragraphs of text and a separately started session with an AI model. To begin with, all you need is a tool for working with agents, such as Claude Code or Codex.

You set up the supervisor in a rules file. When an agent starts work, it reads a file with instructions: CLAUDE.md in Claude Code, AGENTS.md in Codex. In that file you write that the main session plans and checks, does not write code itself and hands the coding to the executor. These rules apply in every conversation, so you do not have to repeat them.
The executor gets a separate session and one task. The supervisor receives a task from me and writes it down in a file: what to change, where, and which tests have to pass. Then, with a simple script, it starts another model (the executor), passes it that file and points it to a separate copy of the code. In our setup the same script keeps track of the order of models, so when the first one reaches its limit, the next one gets the task.
The critic gets a new, empty session. When the executor has finished, the supervisor starts a new agent (the critic) with no conversation history and gives it the executor’s work, the goal and the rules that must not be broken. The instruction can be short: “You are an independent critic. Here is the change and its goal. Find the errors and give evidence for them. Do not fix anything. If something cannot be checked, say so plainly.”

How the agents know what to do and in what order

The supervisor runs the whole process, because the order is written in the rules file. The executor and the critic each get one task from the supervisor, do the task and hand back the result. In our case it looks like this:

I give the goal.
The supervisor checks the current system and writes the specification, a detailed description of the task.
The executor writes code and tests on a separate copy.
The supervisor reads the change line by line and runs the tests itself.
For risky work, the critic looks for errors, and then the supervisor checks every reported error.
If something needs fixing, the executor gets a description of the specific error.
The supervisor puts the change into the running system and records it in the change log.

A rule in a file is not always enough, because an agent can skip it. We measured this: when the rule about running the critic existed only as text, the agents did not follow it. That is why, for the riskiest actions, such as an email to a client sent by an agent or deleting data, we also have technical blocks that stop the agent. I will describe them in a separate part of the series.

Roles and models

Today the supervisor is usually Claude Opus 5.5. It is the best model according to the leading rankings, but also the most expensive. The work designed by the supervisor goes first to Grok, then to Codex and finally to Gemini. This order has already changed several times and will probably change again, so in the company’s rules we commit to roles and not to models. Specific models are just the current cast for those roles. When one model reaches its limit, the next one takes over the task automatically. In September the first executor completed 168 advanced tasks, hit its limit 46 times, and each time another model was available to take over the work.

An executor that has written something does not assess its own work. Given that all of us find it easiest to overlook our own assumptions, I see no reason why models should be any different. Below are five situations from our company that show this.

Example 1: 526 passing tests and 13 bugs

The executor connected a paid service that checks whether email addresses are valid. This was part of automating our sales processes. The first version passed 526 tests. An independent critic found 13 bugs in it, including one that concerned the budget: when the service’s response did not include information about the cost, all spending limits could be bypassed, which could have left us with an unexpected bill.

I suspect the tests did not catch this because they were written with the same assumption as the code, namely that the service always reports the cost. A test checks what its author considered possible. So a passing result says a lot about the author’s consistency with themselves, and much less about what will happen in the real world. The critic did not have those assumptions, so it could check scenarios that the executor’s tests had not anticipated.

Example 2: a refusal read as agreement

Most of our sales conversations are automated. When a client writes that they have decided to work with us, the automation moves the conversation in the CRM to the agreement stage and prepares a draft of the next email: a request for the data needed to set up the system and a note that the contract is coming. Our automation recognized this decision partly by typical phrases. The tests on sample sentences passed. The critic checked the rule on real emails, and it turned out that the same phrase also appears in polite refusals, when a client writes that they have chosen another vendor. Because of this, one conversation in the CRM sat at the agreement stage even though the client had refused, which distorted our view of sales. For another client, who had turned us down three years earlier, a draft asking for contract data would have been created. If such an email had gone out, we would have looked unprofessional. The automation’s log since the end of August does not contain a single such message, so no harm was done.

On the same day, four of about 25 fixes from the executors passed their tests but were wrong, as the critic showed on real emails. During implementation, the executor had prepared tests with sentences that were much clearer and more obvious than what clients actually write. Accordingly, we adopted a rule that changes to how we recognize client replies are checked by a critic with access to real correspondence.

Example 3: a block that worked the wrong way round

When a company declines to work with us, we want to leave it alone for longer. We will get in touch again in the future to see whether anything has changed that makes it worth talking again. The change meant to introduce this was written, and the tests passed. The critic found that in practice the opposite was happening: during the pause, the automation could still contact other people at that company, so they could get an email from us right after their company had said no.

The executor’s tests checked only what the executor had implemented, namely stopping communication with the contact person. They did not cover the situation where the CRM holds several contact people at one company. The critic caught a bug the executor had not foreseen. We now have a rule that the specification of every change to who does not receive emails from us includes a table of all the situations a company can be in (for example open conversation, pause, end of pause) and the transitions between them, with a separate test for each row.

Example 4: code that ran before it was checked

The executor was changing the automation that replies to people who write to us through the contact form on our website, meaning potential clients. It edited files in the folder this automation runs from, and the automation starts every 10 minutes. Four minutes after the change, the scheduler started the new code before anyone had read it. Unchecked code could therefore reply to real clients straight away. The supervisor did not notice this. The critic did. Tests would not have shown it, because the problem was about where and when the change was made. Executors now work on a separate copy, so a change reaches the running system only after it has been checked.

Example 5: the supervisor makes mistakes too

A blacklist is a list of companies and people we must not write to. Companies that have asked us not to contact them again go on this list. The supervisor prepared a work plan under which thousands of CRM entries would have gone on it, including many companies we were in the middle of sales conversations with. We would have cut those conversations off ourselves. The executor, which did not know the supervisor’s reasoning, stopped the work before changing anything.

In my opinion, the executor noticed this error precisely because it did not know what the supervisor had established earlier. Whoever knows the whole history of a decision looks at the result through the same assumptions that led to it.

When we do not run the critic

Every check by a critic is extra work by an AI model, and we pay for it. So we run it where a mistake is expensive or cannot be undone: for emails to clients, for changing a lot of data at once, for documents leaving the company and for paid services. When an error is easy to fix, we skip the critic.

We also tested whether it is worth checking the same thing many times in a row. Such a series cost over a million tokens (the units in which the work of AI models is billed) and found about as much as a single check.

The critic makes mistakes too, so we do not take its word for it. The supervisor checks every reported error itself and only then decides. Each of these roles can get things wrong. They rarely get the same thing wrong, though, which is why having three agents check the code independently is an effective way to reduce the risk of errors in the final version.

In the next part of the series I will describe how we write down the way the company works in a single set of rules, so that every agent knows it, regardless of vendor.

Read more about what an ATS is and how Element works.