LLM18 – Day and Night

Last year while we were starting this journey, we talked about execution modes LLM3, but today it’s undeniable that the models can do a lot of work on their own, and in the previous article we talked about the need of a LLM17 – Paradigm shift.

In the previous article I started talking about letting the agents run, but the agents can run in multiple directions, they can start from a defined plan, they can go from an abstraction and make sense of it, or they can try to make a solution work on an existing system.

For the past few weeks I find myself defining the work in larger chunks, and sending the agent to «run errands» with incomplete information, along the lines of «I want to see how the UI would look if we added this messaging on that condition», before closing for the day. The next day I review it the same way I review email: sometimes it’s really useful, sometimes it’s spam. Today we will talk about these types of work modes and about what Matt Pocock names the Day Shift and the Night Shift.

The idea behind it is that we can separate the type of work that an agent can do autonomously, like research and detailed implementation, from the work that requires us to review, analyze and decide what the agent will be doing in the next autonomous session.

Autonomous work vs engaged work

We call autonomous work the set of tasks that, given a goal, an agent can execute without our constant steering, and produce the outcome we requested (or a very convincing explanation of why it could not be done). Enabling autonomous work requires three components:

• A remote / cloud-based execution environment (autonomous work happens when we are, quite literally Away From Keyboard as Matt mentions, otherwise our laptop can never go to sleep).
• A trusted sandbox and a permissions model like Claude’s auto mode.
• A mindset shift in how we define that outcome.

At the other end of the spectrum is the day shift, the times we are in flow, or what we will call engaged work for the sake of this conversation: This is where we think, we reflect, we debate, we go back and forth via the chat window, we bring colleagues into the mix (and sometimes their agents), we discuss an option, find gaps in our reasoning (or the agent’s) and produce a plan. An effective engaged working session requires context, the agent needs to know about the problem you are trying to solve, and also requires the mindset on how we define the result of this session, because we need to time box them.

Three types of autonomous work

Once we have the framing that the agent can do things on its own, the next question becomes what to do next? Based on my recent experience there are three types of tasks that trigger a «night shift» session, or an «away from keyboard» session.

The figure it out task: Comb through a set of wikis, codebases, documentation, Slack channels, public forums, and gather as much context and evidence as you need to make sense of a problem, a system, an environment, etc. This one is very useful when facing a new codebase or an unknown problem and can be very helpful to get a first draft. You can have a basic prompt, use a skill like research from Matt’s toolbox, or ideally have one tuned to your specific systems and sources of truth.

The will it blend? task: Try to apply a fix or a change to a codebase, with the goal of exploring how the system would behave. This is applying blunt object precision, and can be compared to how, when we didn’t have access to debuggers, we would modify our production code in order to print values, or fake behaviour. Faking the conditions of the system is suddenly less expensive, and we can get a tracer bullet through, bringing the old concept from Andy Hunt and Dave Thomas.

The relentless execution task: This is different, at this point we are no longer exploring, all guardrails apply, the code needs to fit our coding standards, needs to be tested, and needs to be small enough to be reviewed by someone, and by another agent. This is the endgame, this is the task we run once we know what we want and once we’ve stress-tested our system.

A mechanism

Jeff Bezos is famous for saying that «good intentions don’t work» and that the solution for a problem is not «try harder» but instead to have a mechanism to prevent mistakes or to improve a system. Here’s my five-step mechanism for dealing with this new paradigm of work.

1. {Autonomous} Figure it out: Let the agent roam free, give it a blurb, a codebase to explore, and a goal «find out why X works» or «I’m considering implementing Y, run a POC». The goal is short, abstract, but the outcome is defined. This is a conversation starter, not shippable code in any way.
2. {Engaged} Shape what you figured out: When we have a clear idea on the codebase and the problem, we can then start brainstorming the idea. Matt has made a few iterations of the grilling concept. Instead of telling the agent what to do and letting it assume the rest, have it interview you and have you voice the assumptions, leaving the thinking to you. The outcome is either a workable plan that we can stress-test (step three) or a go-ahead to jump straight to execution (step four).
3. {Autonomous} Will it blend?: At this point we can put our plan to the test by quite literally running it, mocking as many dependencies and making as many assumptions about the state of the system as we want, before our code runs in production. We can drop a few «what if?» questions to the agent, and in the spirit of step one, let it roam free.
4. {Autonomous} Relentless execution: At this point we’ve finished the cycle, we ran an exploratory session, we discussed the implementation, we explored the outcome, modified the implementation with the exploration, and now we are ready to go. Here’s where the rubber meets the road, this code must be tested, production-quality, and ideally, small enough to be tested by others and agents quickly.
5. {Engaged} Review: Regardless of the complexity, we should understand the feature we are building, so in this step we review the code, the evidence, the tests, the behaviour of the system… I’m not prescriptive here, I have a few personal tools where I don’t watch the code but I pay attention to the test cases we add, and a few systems where I make a point in reviewing every single line of code. At the end of the day, the name on the commit is ours, regardless of the tools we used, and the responsibility goes with the name.

As you can see there’s a clear distinction between the types of work that an agent can do on auto-pilot, and the work that needs our attention:

• Steps one, three and four are autonomous, but it’s a different type of autonomy each time. In steps one and three we are searching for information, boundaries are fluid, in step four we are implementing a disambiguated and stress-tested plan and the boundaries are set. We can then leave an agent working for some time.
• Steps two and five are the engaged review and analysis, and that’s where we need to be spending our time, not babysitting a terminal looking at the next token being produced.

And now what?

Chunking the work we send to the agent has a cost, which is the loss of instant gratification: we send a prompt, we see the result almost instantly and we have complicated feelings about it. With this process we define the work, we send it away, and we don’t look at it until we do a pass. It’s the same approach we have with email and messaging apps.

However, and like with messaging apps, reaching some version of this mechanism takes time and I’m still exploring. I still find myself talking to the agent, correcting the approaches, redoing some work and waiting for some tasks, so the mechanism of defining the autonomous work is not there yet.

What I’m trying to do is to keep related work close by. If I’m reviewing documents I do several reviews in one pass, if I’m reviewing code, same, and if I’m between meetings I’m most likely recapping, capturing action items and communicating. We can’t be productive all the time, and we shouldn’t aim for that.