Improving the Harness

Now that GenAI is all the rage, I will share more about it. Kent Beck says "nobody knows for sure, we can just explore and share what we found".

Here is a little trick I use to improve my AI workflow. Birgitta Böckeler defines harness as the support we give a coding agent to help it do its job, so harness engineering is all the work we do around that to make that possible. A part of the harness comes with your coding agent (i.e. inner harness - e.g. its system prompt), and there are parts which we add (i.e. outer harness - e.g. skills). The harness can contain feed-forward "guides" (all the markdown files) and feedback "sensors" (tests, linters and the like).

Learning From Past Conversations
While researching the different workflows people use with GenAI, I read an advice that from time to time, you should scan your agent conversations, check what the agent did wrong and then got right - either by itself or by your correction. Then update your guides to avoid this problem in the future. (Unfortunately, I cannot remember the source any more.) My first challenge was finding my past conversations. I started my GenAI journey with Codeium in September 2023, which became Windsurf which then became Devin. Where were my conversations? Windsurf was based on VSCode and I found the conversations in my home directory. They were in binary format which looked like protobuf and people were unable to decrypt them. Bummer. When I switched to Mistral (because of the CLOUD_Act and I like Europe's GDPR way more ;-) the situation improved. Mistral VIBE stored its conversations as JSON lines in a my home directory ~/​.vibe/​logs/​session/​session_*/​messages.jsonl. And even if it had hidden them, I could have modified its source code. That is a reasons I like Open Source.

Chunking Conversations
I used context discipline, aka focused agents, and work with short conversations, often starting with a single user prompt. The largest of these JSON files were around 600 KB, with more than 100 lines agent reasoning and tool calling. Reading these large conversations overflowed my context and I had to split them into chunks, preserving lines, using a little Python script. I used a threshold of 60 KB for each chunk which works. Using chunks kept my context limited but the agent did overlook mistakes and their fixes than came much later in the conversation.

An agent loop processed the chunks of each conversation and wrote out an result markdown analysis.md file for each conversation. Here is a part of my Windows Batch script:

for /d %%d in (session_*) do (
  if not exist %%~dpd%%d\analysis.md (
    call vibe -p "Run @%~dp0search_process_improvements_prompt.md for folder %%~dpd%%d"
  )
)

And the prompt (search_process_improvements_prompt.md) contains:

• Termination condition: If the folder contains an `analysis.md` its analysis is complete. Stop.
• Context: The given folder contains a conversation between a human and answers from an agent.
• Input: This conversation is split across smaller files named `chunk_*.jsonl`.
• Task: Read each chunk and search for things that went wrong and that later got corrected, either by retry or by the user.
• Output: Format the learnings as a short list suitable for documentation or rule files.
• Then process the next chunked file, keep appending to `analysis.md` until all chunks are done.

After the agent looped for some time, I got a bunch of individual analysis.md files. A final agent call combined, grouped by issue and sorted the findings by number of occurrences, i.e. more frequent issues came first in the result. As expected from GenAI documents, the output was too much. For my TDD with Agentic AI experiments the analysis of 50 sessions showed around 200 issues in 14 categories:

No.	Category	Occurrences	Priority	Impact
1	Line Endings	30+	Critical	Agent Tooling
2	Platform Commands	20+	Critical	Agent Tooling
3	File Editing Tools	25+	Critical	Agent Tooling
4	TDD Workflow	15+	High	Process Skills
5	File/Path Handling	15+	High	Agent Tooling
6	Code Quality	15+	High	Code/Design Documents
7	String Escaping	10+	High	?
8	Testing Practices	10+	High	Process Skills
9	Git Workflow	10+	High	Process Skills
10	Skill Adherence	10+	High	?
11	Refactoring	10+	High	Process Skills
12	Tool Usage	10+	Medium	Agent Tooling
13	Error Handling	5+	Medium	Code/Design Documents
14	Documentation	10+	Medium	Code/Design Documents

Fixing Agent Tooling Problems
The results covered different topics. Some issues alerted me of tooling problems. At the time I used VIBE v2.9.6, an early version which lacked Windows support. Categories 1, 2, 3, 5 and 12 resulted from that. For example VIBE's search_replace tool struggled with Windows line endings. My source code had a CRLF instead of the expected LF at the end of each line. I changed my workflow to call dos2unix on all source files repeatedly. (Later a VIBE upgrade made this unnecessary. VIBE is actively maintained and delivers a new version at least every week.)

Sidebar: Agents vs. Microsoft Windows
VIBE tried to call grep again and again. Some (many?) agents assume a Linux based environment when calling command line tools. Pi coding agent supported only Linux at the time I explored it. At first I did not notice, I use UnxUtils (GNU utilities for Win32) since many years, even long before MSYS2 or WSL were a thing. My Windows is "hybrid" anyway, e.g. I set a %HOME% variable, set HOME=%HOMEDRIVE%%HOMEPATH% which works great for Linux programs compiled for Windows. My batch scripts are a mess too, mixing CMD internals like for, MS-DOS/Windows32 utilities like xcopy, standard Unix-based shell commands like grep and tail and modern versions of my own commandline tool library.

VIBE failed at calling some Linux tool, it retried and got it right, slowing down my workflow. I downloaded VIBE's source code and pointed itself at it to analyse the issue with grep. (That is a reason I like Open Source.) In the end I installed Ripgrep rg to fix the problem, as VIBE calls it first. Now all code searches work, even better Ripgrep is much faster that plain grep. Win!

Fixing Development Process Problems
Categories 4, 8, 9 and 11 concerned the TDD process and how I wanted to use Git in the project. These added to the different agent SKILL.md files I used. While some of the findings were already in the skills - see category 10 - I added most as extra rules to my skills. Here is an example of the improved refactoring rules. I considered them obvious and skipped specifying them in the first place, but past conversation showed that the agent got it wrong and needed more guidance:

• Use grep to find all usages of a method before refactoring.
• When refactoring method signatures, update all call sites. Compile frequently.
• Search for all occurrences of the old method across the entire codebase before committing.
• When moving code, verify that the original has been removed, not just copied.
• Remove obsolete files during cleanup.

Fixing Documentation Problems
The rules from the remaining categories 6, 13, 14 were missing in different knowledge documents about the project setup, programming conventions and design guidelines. These were project and language specific rules. Not everything listed was relevant and much was already documented. Still I got some good addition to my harness both the inner and outer ones.

Prompt Injection?
I used the above process on different projects. When there were enough new conversations, I re-ran the whole thing. The loop let my agent churn away for hours in the background. In the end I scanned the resulting document and applied all new rules which looked relevant. I did not care if the analysis missed anything, this was not about correctness, rather a performance improvement. One final problem I noticed was that during some runs the agent forgot its task (to find process improvements) and instead changed the code in the project referenced in the conversation. Snippets of code appeared like a shadow of the past. What a suitable reminder that our new friends? are indeterministic and untrustworthy.