Improving the Harness

Now that GenAI is all the rage, I will share more about it. Kent Beck says "nobody knows for sure, we can just explore and share what we found".

Here is a little trick I use to improve my AI workflow. Birgitta Böckeler defines harness as the support we give a coding agent to help it do its job, so harness engineering is all the work we do around that to make that possible. A part of the harness comes with your coding agent (i.e. inner harness - e.g. its system prompt), and there are parts which we add (i.e. outer harness - e.g. skills). The harness can contain feed-forward "guides" (all the markdown files) and feedback "sensors" (tests, linters and the like).

Learning From Past Conversations
While researching the different workflows people use with GenAI, I read an advice that from time to time, you should scan your agent conversations, check what the agent did wrong and then got right - either by itself or by your correction. Then update your guides to avoid this problem in the future. (Unfortunately, I cannot remember the source any more.) My first challenge was finding my past conversations. I started my GenAI journey with Codeium in September 2023, which became Windsurf which then became Devin. Where were my conversations? Windsurf was based on VSCode and I found the conversations in my home directory. They were in binary format which looked like protobuf and people were unable to decrypt them. Bummer. When I switched to Mistral (because of the CLOUD_Act and I like Europe's GDPR way more ;-) the situation improved. Mistral VIBE stored its conversations as JSON lines in a my home directory ~/​.vibe/​logs/​session/​session_*/​messages.jsonl. And even if it had hidden them, I could have modified its source code. That is a reasons I like Open Source.

Chunking Conversations
I used context discipline, aka focused agents, and work with short conversations, often starting with a single user prompt. The largest of these JSON files were around 600 KB, with more than 100 lines agent reasoning and tool calling. Reading these large conversations overflowed my context and I had to split them into chunks, preserving lines, using a little Python script. I used a threshold of 60 KB for each chunk which works. Using chunks kept my context limited but the agent did overlook mistakes and their fixes than came much later in the conversation.

An agent loop processed the chunks of each conversation and wrote out an result markdown analysis.md file for each conversation. Here is a part of my Windows Batch script:

for /d %%d in (session_*) do (
  if not exist %%~dpd%%d\analysis.md (
    call vibe -p "Run @%~dp0search_process_improvements_prompt.md for folder %%~dpd%%d"
  )
)

And the prompt (search_process_improvements_prompt.md) contains:

• Termination condition: If the folder contains an `analysis.md` its analysis is complete. Stop.
• Context: The given folder contains a conversation between a human and answers from an agent.
• Input: This conversation is split across smaller files named `chunk_*.jsonl`.
• Task: Read each chunk and search for things that went wrong and that later got corrected, either by retry or by the user.
• Output: Format the learnings as a short list suitable for documentation or rule files.
• Then process the next chunked file, keep appending to `analysis.md` until all chunks are done.

After the agent looped for some time, I got a bunch of individual analysis.md files. A final agent call combined, grouped by issue and sorted the findings by number of occurrences, i.e. more frequent issues came first in the result. As expected from GenAI documents, the output was too much. For my TDD with Agentic AI experiments the analysis of 50 sessions showed around 200 issues in 14 categories:

No.	Category	Occurrences	Priority	Impact
1	Line Endings	30+	Critical	Agent Tooling
2	Platform Commands	20+	Critical	Agent Tooling
3	File Editing Tools	25+	Critical	Agent Tooling
4	TDD Workflow	15+	High	Process Skills
5	File/Path Handling	15+	High	Agent Tooling
6	Code Quality	15+	High	Code/Design Documents
7	String Escaping	10+	High	?
8	Testing Practices	10+	High	Process Skills
9	Git Workflow	10+	High	Process Skills
10	Skill Adherence	10+	High	?
11	Refactoring	10+	High	Process Skills
12	Tool Usage	10+	Medium	Agent Tooling
13	Error Handling	5+	Medium	Code/Design Documents
14	Documentation	10+	Medium	Code/Design Documents

Fixing Agent Tooling Problems
The results covered different topics. Some issues alerted me of tooling problems. At the time I used VIBE v2.9.6, an early version which lacked Windows support. Categories 1, 2, 3, 5 and 12 resulted from that. For example VIBE's search_replace tool struggled with Windows line endings. My source code had a CRLF instead of the expected LF at the end of each line. I changed my workflow to call dos2unix on all source files repeatedly. (Later a VIBE upgrade made this unnecessary. VIBE is actively maintained and delivers a new version at least every week.)

Sidebar: Agents vs. Microsoft Windows
VIBE tried to call grep again and again. Some (many?) agents assume a Linux based environment when calling command line tools. Pi coding agent supported only Linux at the time I explored it. At first I did not notice, I use UnxUtils (GNU utilities for Win32) since many years, even long before MSYS2 or WSL were a thing. My Windows is "hybrid" anyway, e.g. I set a %HOME% variable, set HOME=%HOMEDRIVE%%HOMEPATH% which works great for Linux programs compiled for Windows. My batch scripts are a mess too, mixing CMD internals like for, MS-DOS/Windows32 utilities like xcopy, standard Unix-based shell commands like grep and tail and modern versions of my own commandline tool library.

VIBE failed at calling some Linux tool, it retried and got it right, slowing down my workflow. I downloaded VIBE's source code and pointed itself at it to analyse the issue with grep. (That is a reason I like Open Source.) In the end I installed Ripgrep rg to fix the problem, as VIBE calls it first. Now all code searches work, even better Ripgrep is much faster that plain grep. Win!

Fixing Development Process Problems
Categories 4, 8, 9 and 11 concerned the TDD process and how I wanted to use Git in the project. These added to the different agent SKILL.md files I used. While some of the findings were already in the skills - see category 10 - I added most as extra rules to my skills. Here is an example of the improved refactoring rules. I considered them obvious and skipped specifying them in the first place, but past conversation showed that the agent got it wrong and needed more guidance:

• Use grep to find all usages of a method before refactoring.
• When refactoring method signatures, update all call sites. Compile frequently.
• Search for all occurrences of the old method across the entire codebase before committing.
• When moving code, verify that the original has been removed, not just copied.
• Remove obsolete files during cleanup.

Fixing Documentation Problems
The rules from the remaining categories 6, 13, 14 were missing in different knowledge documents about the project setup, programming conventions and design guidelines. These were project and language specific rules. Not everything listed was relevant and much was already documented. Still I got some good addition to my harness both the inner and outer ones.

Prompt Injection?
I used the above process on different projects. When there were enough new conversations, I re-ran the whole thing. The loop let my agent churn away for hours in the background. In the end I scanned the resulting document and applied all new rules which looked relevant. I did not care if the analysis missed anything, this was not about correctness, rather a performance improvement. One final problem I noticed was that during some runs the agent forgot its task (to find process improvements) and instead changed the code in the project referenced in the conversation. Snippets of code appeared like a shadow of the past. What a suitable reminder that our new friends? are indeterministic and untrustworthy.

Finally Fizz Buzz Turing Machine

Now that our GenAI agents are doing all coding, what does remain for geek fun? I do not mean developing side projects or open source, I mean fiddling with code, tweaking it, exploring constructs like Only Exceptions for Flow Control. Such coding activities used to be practice or learning. What is their use now? At least they are entertaining (me). I use the tag Coding Fun to mark ideas of educational entertaining hobby coding.

Back to the Turing Machine
One of these ideas was to create Fizz Buzz on a Turing Machine (abbreviated TM) - you know, the thing with the tape of zeroes and ones. Last summer I created its building blocks, i.e. symbols, Tape, TuringMachine and TransitionTable classes. These parts built an Universal Turing Machine (abbreviated UTM), which ran any concrete TM. On top I added transition tables for the basic structures required by Fizz Bizz, e.g. numbers (encoded as binary) and operations to INC and compare them. Inspired (or biased) by assembly, I unknowingly recreated a Von Neumann architecture where some part of the tape contained the program code, specific symbols for each operation, and another part contained the working memory. Now I had a machine on top of a machine.

Arithmetic
The best part about "Entertainment Coding" is that you can stop whenever you want. I often stop when I have proven my idea. I pass on filling in all the details. And there are a lot of details to fill in for a Turing complete assembly - which I need for Fizz Bizz. The numbers are encoded as their bits. Moving the data cursor right or left one number needs only a few states and transitions. Increment's (INC) transition table, shown in the previous article, has around 8 lines. Decrement is the same. Comparing two numbers for being less-than or being equal-to needs 28 lines and duplicating a number to the next space on the right has already 34 lines. The number of state transitions grows with the number of cells involved (e.g. Which bit is this?), overflow (e.g. Do I have to add 1 to the bit because of the previous operation?) and number of operands (e.g. Do I have to move right to continue?). Adding or subtracting two numbers is the same flow, with more moving the read-write head and keeping the overflow. It is possible and much work, so skip it.

Flow Control
What about flow control? For Fizz Buzz I need two kinds of flow control:

1. A conditional to check if a number is equal another number. If they are equal, the following code is run, else the code is skipped. Skipping next instructions is a jump forward towards the end of the "if body", defined by the symbol ';'. (Why use a semicolon? I do not know, probably I did too much Java in my life ;-) Luckily I only need to check if a number is equal to 0, which is much simpler than comparing to an arbitrary number.
2. For looping - at the end of the loop - I need to check if a number is smaller than another number. This is like point 1 above. Then jump back to the beginning. This is a jump backwards to the start of the "loop body". I need a GOTO and a loop label as target to do this.

This is a minimalist design. It has no else branch, nested conditionals or nested loops. It models the required control flow for Fizz Buzz:

'L' ... start of the loop
   'i' ... current += 1

   'z' ... if (current % 15) == 0 then
      // write "FizzBuzz"
      'g' ... continue
   ';' ... end if

   // similar block for 3
   // similar block for 5

   // write current as string

   '<' ... if current < 100 then
      'g' ... continue
   ';' ... end if

'h' ... halt program

Because I miss else the body of each if must end the loop prematurely - in some programming languages called continue. Thus the last if checks the loop exit, i.e. the upper limit, e.g. 100. The last number must not be Fizz nor Buzz nor FizzBuzz.

Time for some low level details: Here is the transition table of GOTO:

Step	state	symbol	newState	newSymbol	direction
(1)	Ip_SwitchRight	'g'	Ip_Goto	same	L
(2)	Ip_Goto	'P'	same	same	L
(3)	Ip_Goto	'h'	Ip_GotoRightHalt	'P'	R
	Ip_GotoRightHalt	'P'	Ip_Goto	'h'	L
	Ip_Goto	'd'	Ip_GotoRightDup	'P'	R
	Ip_GotoRightDup	'P'	Ip_Goto	'd'	L
	Ip_Goto	'i'	Ip_GotoRightInc	'P'	R
	Ip_GotoRightInc	'P'	Ip_Goto	'i'	L
	Ip_Goto	'r'	Ip_GotoRightRightMove	'P'	R
	Ip_GotoRightRightMove	'P'	Ip_Goto	'r'	L
	Ip_Goto	'l'	Ip_GotoRightLeftMove	'P'	R
	Ip_GotoRightLeftMove	'P'	Ip_Goto	'l'	L
	Ip_Goto	'='	Ip_GotoRightEqual	'P'	R
	Ip_GotoRightEqual	'P'	Ip_Goto	'='	L
	Ip_Goto	'<'	Ip_GotoRightLess	'P'	R
	Ip_GotoRightLess	'P'	Ip_Goto	'<'	L
	Ip_Goto	';'	Ip_GotoRightEqualEnd	'P'	R
	Ip_GotoRightEqualEnd	'P'	Ip_Goto	';'	L
(4)	Ip_Goto	'L'	Ip_Restart	same	R

Transition (1) begins the GOTO states. Transition (2) starts moving left and skips the instruction pointer. Transition (3) moves the instruction pointer along each and every skipped instruction. Whenever I add a new instruction I have to update conditional and loop code, which is boring. (4) Goto's state transitions finish when it sees the symbol 'L'. Then the instruction process restarts with state Ip_Restart by looking at the next instruction right of the instruction pointer.

The Whole "Language"
Till now I created the following symbols:

symbol	memory	meaning
'0'/'1'	data	a bit of a number.
'$'	data	separator of a number or memory slot - a "byte" if you like.
'C'	data	data Cursor marking the current number on its right.
'P'	code	instruction Pointer left of the next instruction.
'h'	instruction	halt, stop the TM.
'l'/'r'	instruction	move the cursor one number to the left or right.
'i'	instruction	increment the current number by one and move cursor right.
'e'	instruction	decrement the current number.
'd'	instruction	duplicate current number to the right.
'3'	instruction	write the digits of 3 into the current number.
'<'	instruction	jump to end of block if current is not < next number.
'='	instruction	jump to end of block if current is not = next number.
'z'	instruction	jump to end of block if current number is not 0.
';'	label	mark the end of the (conditional) block.
'g'	instruction	jump backwards (moving 'P' left) until the label 'L'.
'L'	label	Loop label for goto (backwards) jump.

Instruction '3' is special, as would be '5' and 15. Data values are on the tape and copied around. It might save me some hassle if I can write arbitrary numbers programmatically. Then d3ll%lz...;lll would be the code to check for multiplies of 3 - if I had '%'. All my programs (till now) will match the regular expression P[hdierl3=<z;gL]+C([01]{7}\\$)*.

I need modulo ('%'), which extends subtraction (which extends decrement). Modulo is hard, too complicated for me to write it as state transitions. I could write a macro, i.e. separate code that gets executed whenever the UTM reads a '%'. This would be like Microcode of modern CPUs. Or I could create subroutines. If I avoid nesting them, a single instruction pointer symbol, e.g. 'R', as return address would be sufficient. For example, when the UTM reads 's', it writes the 'R' instead of the 'P'. Then it searches right for 'S' which is the label to start the subroutine and executes until it reads 'r' for return. On 'r' the instruction pointer goes back to 'R'. Four new symbol 'srSR' (2 instructions, 1 label, 1 pointer). Regular program flow can never see 'S' or 'R' and I omit adding them to if or goto. The 's' might be inside a loop or conditional. The 's' and 'r' are gotos looking in specific directions for specific labels, nothing conceptionally new.

The "Actual" Program
I have enough of this. This is a Turing Tarpit, a (esoteric) programming language that is universal but impractical and difficult because it offers no support for common tasks. [...] Using such a language is a form of mathematical recreation. Yes I can finish Fizz Buzz, but it is no fun. To conclude, here is my final piece of code to create the numbers from 1 to 16 on the tape:

P
Lrdllldlil<lg;   // loop creating numbers
i                // fix last element
llllllllllllllll // go back all = "l" * limit
hC
0000001$         // from 1
0001111$         // to limit - 1

I hope you had fun following me on my quest. The whole TM, UTM and transition logic is on my Git. I now appreciate how high level assembly is.

My Writing and GenAI

I follow my plan of focusing on fewer tasks. This puts a lot of pressure on my time. In addition I spent almost every evening of the last months exploring GenAI. My wife got sick of me talking about agents, tokens and the like. Besides my feelings about the change of our industry I did not write much. Having little time and less practise in writing made writing more unlikely - a vicious cycle. I had no idea where putting my thoughts would end up. It might become a rant. (Sidebar 1)

Working With Text
What to look for? I use writing to structure my thoughts. This is rubber ducking, a practise of talking through a problem. In doing so, your brain structures the problem as part of communication. This often lets us find the solution to the problem. I learnt about this technique from The Pragmatic Programmers, a book I recommend since 2011. The idea itself is way older and based on Joseph Weizenbaum's ELIZA from 1966. (You can try ELIZA here.)

More than once the outcome of my writing was different than I had expected or planned. I partitioned and analysed the problem during writing and found better results. This is the benefit. I lose if AI writes the text for me. It will format my notes nicely and present something that looks reasonable but it will not end with opposing results. And even if it would, I would reject them because I am unprepared for opposing results. I need to go through the work of structure and analysis to find and accept a different outcome than I had anticipated. My writing needs to be slow and meandering and cumbersome.

When writing about technical topics, e.g. the latest Coding Fun, I have to structure the material linearly to make it possible to read for people who are unfamiliar with what I want to present. This makes me reflect on the topic from a distance, after I had been in the fine details all the time. This summary adds to my understanding of the material.

On LinkedIn is a group of people who claim that they put as much thought and effort into their AI generated articles - which would also mean they get the same benefits. I do not know, maybe they do. But Mathias Verraes disagrees. He claims to recognise AI-generated text because he read so much. The generated text contains the same cliches and seems to be uniform - for sure statistically averaged. These articles are boring which makes them unreadable. (Sidebar 2)

I do not write enough and miss practice. Getting started is difficult, collecting ideas takes time and I edit everything more than once. I guess Mathias will resent my writing. ;-) I struggle with articulation as English is not my first language. I could use GenAI to improve my articulation - it would keep my ideas and thoughts, at least according to the proponents of GenAI. But I reject it. I want to improve my articulation myself. Articulation is a skill, writing is a skill, and both need practise. (Sidebar 3)

I like writing by hand, or should I call it writing manually: I am creating something and showing the world. Using GenAI for my writing removes all these benefits: I am not structuring, expressing, and polishing my thoughts. I am not summarising known material. And I am not practising and improving my writing. Dylan Beattie phrased it in his GeeCON 2026 keynote: "If something is fun and you're learning - using AI brakes both."

What Makes A Person?
Another goal of my writing is to share my knowledge. Other than expressing myself, sharing is a marketing tool, at least part of developer branding. As GenAI disturbs my beloved coding, I need to rethink my business. What makes me successful since 15 years? Besides my technical expertise I believe I am successful because I am authentic. I teach what I know and have experience from past projects and always state up front when I have no clue. (Sidebar 4) I avoid marketing buzz words and stay away from promising anything. I demand perfection from others and even more from myself. I live what I preach. I have no idea what my clients buy when they hire me for the first time (when they do not know me yet). I believe they buy the promise of my reputation. All my work comes through personal recommendations. (Thank you so much my friends.) For sure a large part of my business depends on my contacts and personal, authentic connections. Being honest, admitting mistakes and taking responsibility for my actions are important to me. No generated text, regardless how well structured and nicely sounding, supports those.

Generated Correspondence
In the early days of Chat-GPT I ran an inhouse Coderetreat for one of my clients. A local organiser sent an invitation to all developers. Instead of writing one or two sentences about the upcoming event, he generated a full page of text about being "delighted" to host such a "spectacular" event and more exaggerated terms. This is a common problem. Me friend Raimo writes about interacting with coworkers talking to us only through parrots which makes him frustrated or angry at times. I fail to understand why people would do that. (Sidebar 5) Particular in an corporate environment, nobody cares for style or typos in work emails. As time is money, long emails and chat messages are discouraged. The respect for our coworkers' time should stop us from creating (large amounts of) useless text. Please never send me AI generated text. I will never use AI for mails or personal messages.

I am a geek, spending my evenings alone, in a dark room filled with hardware and pizza boxes - at least kind of. And at the same time I like working with people. I value human connection. It makes much of my work meaningful and worthwhile. Human beings are social, everybody needs human connection somehow. To connect I need to express my feelings (like about accepting GenAI) and I want to be seen. AI is unable to express my feelings and it certainly can not make me seen. Connection is not happening if I do not expose myself.

My AI Writing Manifesto
After collecting notes and thoughts since May this year, structuring my thoughts and polishing then, I found clarity:

• I structure and formulate my thoughts and ideas myself. AI does not support me in digesting and distilling content.
• I summarise technical expertise myself. AI does not support me in finding deeper relations based on known material.
• I author and polish my writing myself. AI does not support me in practising and improving my writing skill.
• I keep a honest and authentic relation to my clients and peers. AI is neither.
• I respect your time and keep all messages short. AI generates bloat.
• I write all messages myself to express my feelings and to be seen. AI hides me.
• All images I use are real images, unless marked as AI generated.

---

Footer
I was massively distracted while writing. I kept finding notes from the last five months on different aspects of this. The topic had been on my mind for some time. After finishing the outline I streamlined the text and moved my secondary thoughts into this section. While off-topic these thoughts were reasonable - I might expand on them later.

1) Blog Rants: I have not written any rant since I went independent. This is a good thing. I do not miss the corporate politics and hidden agendas.

2) Hate of Generated Text: In my social "bubble", there is a crusade against AI generated text. I can relate. The massive creation of AI slop is speeding up the Enshittification of the Internet. For example: This January I suffered from pneumonia. Staying in bed I searched the web for articles comparing AC generators. I planned to buy one powering tools in the garden. The web search showed promising results. But each page turned out to contain generated text of products with deep links into Amazon, and no real descriptions nor any comparison. Maybe due my weakened state, I was devastated. I recognised that the Internet was broken. That was not the web I had learnt to trust for its usefulness.

3) Practice and Mastery: I am a big fan of deliberate practise. I practice deliberately since many years and have talked about it in the past. I use that approach for all areas which I care for and where I want to improve. I want to achieve mastery (as a path not a state). Maybe this is folly: I am in a position of privilege which allows me to reflect on my skills and put aside time to practise, which is like play, unproductive. I am able to laugh about failed attempts and try again. I like the experience of the flow you get after reaching a certain level of skill. Having invested huge amounts of time into practise makes me respect the effort and skills of other people. Coming back to AI writing - I am unable to respect neither the author nor the work using AI because there is little effort and skill required.

4) Working with technologies I do not know is an interesting situation as a coach. I always confess in the first meeting with the client when I have no experience with certain technologies they are using. 12 years ago, I worked with a company using PHP. Back then, they used PHP 5, which had a bad reputation. I had never seen any PHP before. I struggled a lot with its dereference operator -> and always used the more common . instead. When pair programming I always mistyped it and one of my pairing partners designed a T-shirt for me saying "this -> is not this .". In the end I knew more about PHP than most developers I paired with because I knew what to expect from a language and how to google it. Another time I worked with C# which I had never touched before. That was even easier, as at the time, C# was "Java in Pascal case". ;-) After learning some programming languages, at least the main stream ones, all languages look the same.

5) Why generate bloated text? I am wondering which universal human need is met when people generate work emails with GenAI, making them many times larger than necessary. It is not efficiency, as the prompt takes longer than the pure message. It is not reputation, as everybody recognises the generated text. (Ha ha sequences of "it is not" are a sign of AI generated text. Fear not dear reader, I am able to author bad writing without any AI any day. ;-) People generate content for Wikipedia - which is vandalism and go on when asked about it. Which need is met by these actions? I will have to investigate. The best I know is "because I can" which is self-efficacy, an important need indeed.