Method
How we work.
Two weeks watching the work. A written report you can argue with. One thing built and tested against real jobs from your business. Then it is yours to keep.
01 · The method
The two weeks, and what lands on your desk.
The front page carries the short version. This is the same method with the parts that did not fit – who sits in the room, what a shadowing session actually involves, and what is in the report, section by section.
-
Weeks one and two – we watch.
It opens with a half-day workshop. In the room: whoever runs the work, the person who quotes, the person who schedules, and whoever chases the money. Four to six people is plenty. From our side it is the two people who would build the thing. There is no account manager and nobody taking notes for someone more senior you never meet.
Then three to five shadowing sessions, two to three hours each. We sit beside the person doing the job and watch them do it, at their desk or in the ute. We time things. How long a quote takes from the phone call to the send button. How many times the same address gets typed in. What happens when a supplier answers with a price that is out of date.
We take copies of twenty or thirty real jobs, with names removed if you want them removed. Those become the test set later, so it matters that they are the messy ones. We also ask for read-only access to the systems that hold the work – the job management software, the accounting file, the shared inbox. Nothing is changed during the fortnight.
We do not ask for a process diagram. A diagram shows the process someone wrote down once. We are there for the fourth job of the day running late.
-
End of week two – the report, then a 90-minute readout.
The report arrives before the meeting, so you can read it without us in the room. Then 90 minutes with the same people who were in the workshop. Seven sections, always in this order.
What is in the diagnostic report. Usually 15 to 25 pages, written in plain words. Section What it contains What we saw The work as it actually runs, step by step, with the timings we took while sitting there. Written so your own staff recognise it. The shortlist Five to eight things AI could take off you, ranked. Each one described as a job, not a technology. The scoring Hours saved each week, cost to build, and how likely it is to break. Every score shows the arithmetic and the assumption behind it, so you can argue with either. The recommendation Which one to do first and why, or that none of them is worth doing yet. What would have to be true The access, the data quality and the person on your side that each option needs. This is where most projects quietly die, so it is written down early. What we could not check The questions two weeks was not enough to answer, and what it would take to answer them. The quote Fixed price, fixed scope and a timeline for the top item, with the acceptance test written into it. Every number in the scoring traces to something we watched or something you told us. If the honest answer is to build none of it, that is what the recommendation says, and the diagnostic fee is the only money you spend.
-
Then we build one thing.
Fixed price, fixed scope, half up front and half when you accept it. One workflow, finished, rather than three started.
Before any code is written we agree what "working" means, in numbers, and you sign it. "Eight out of ten quotes go out without anyone editing them" is a definition. "It feels faster" is not, and it cannot be tested. That number becomes the acceptance test, and it is the same number you refuse to pay on.
Every week you see the software running with your own data in it. There is no status deck. We build against the real jobs collected during the two weeks, because invented test cases pass. Real ones find the customer with two site addresses, the invoice with a credit on it, and the job that was booked by text at nine at night.
-
Then we measure.
We take the baseline during the two weeks, before anything exists to flatter. Then the same measure, taken the same way, 30 days after it goes live and again at 90.
There is a good reason to insist on this. Researchers ran a controlled trial with sixteen experienced developers on 246 real tasks. They were 19% slower using AI tools, and believed afterwards that AI had sped them up by 20%. metr.org ↗ The researchers treat that result as historical and expect the same developers to be faster with today's tools, which we say because it is true. The part that survives is the gap between what people felt and what the clock said. People cannot tell. So we count.
If the number does not move, you get that in writing, with what we would do differently. A measurement you are not allowed to fail is not a measurement.
-
Then we hand it over, and stay if you want us.
Working software, written procedures, a walkthrough video for each workflow, the test set, and a plan for what to do when it goes wrong. Section 04 lists exactly what changes hands.
The monthly retainer is optional and you can leave on 30 days' notice. Plenty of clients should not take it. Take it when the workflow touches customers or money, when the models underneath it are moving, or when nobody in the building would notice for a fortnight if it quietly started getting things wrong.
02 · Testing
How we test it.
Before we hand anything over, we test it the way you would test a new apprentice.
We take a hundred real jobs from your business and run them through. Then we sit with you and go through what it got wrong. The output is a count, not a feeling. It got 84 of these right, here are the 16 it did not, and here is why each one failed.
Then we fix those and run the lot again. And we keep the hundred. When we change something in six months, we can show you that nothing which used to work has stopped working.
If it passes everything first go, the test was too easy, and we go and write a harder one.
The technical version
This part is for whoever reviews the work on your side. It describes the same procedure with its proper names, and the numbers we hold ourselves to.
Error analysis comes first.
We read at least 100 traces of the system doing the real task and write a short note on each failure in ordinary words. Those notes get grouped into a failure taxonomy, and each category gets counted. We stop adding categories when roughly twenty traces in a row turn up nothing new. Fixes then go in order of frequency, which is rarely the order in which they annoyed anyone. hamel.dev, checked 8 August 2026 ↗
Graders are binary.
Each check is a written rule with a pass or a fail. Scoring one to five hides the disagreement: two reviewers both give a 3 for opposite reasons, and the average looks stable while the system drifts. Pass or fail forces the argument to happen early, when it is cheap.
Judges get validated against human labels.
Where a model grades another model's output, we label a sample by hand first, then measure the judge against those labels. Both the true positive rate and the true negative rate are written down, and we use them to correct the judge's estimate of the real failure rate. The grading model is never the model being graded. platform.claude.com ↗ When two of us label the same set, agreement is reported with Cohen's kappa, which corrects for agreeing by chance. An unvalidated judge measures its own opinion. hamel.dev ↗
The regression suite runs in CI.
At least 100 examples: the core cases, one case for every bug ever fixed, and the edge cases we know about. We favour deterministic assertions – string matches, schema checks, the exact sequence of tool calls – over model-graded checks, because CI runs constantly and every model-graded check costs money on every run. hamel.dev, checked 8 August 2026 ↗
This is also what makes a model change answerable. When a provider ships a new version, the suite gives you the same 100 examples before and after, in an afternoon. The Digital Transformation Agency now publishes model AI clauses and AI procurement guidance for government buyers. dta.gov.au ↗ Questions of that kind are difficult to answer without a suite and straightforward with one.
Retrieval is measured on its own.
Where the system looks something up before it answers, the lookup is scored separately with recall at k, precision at k and mean reciprocal rank. Otherwise a wrong answer and a missing document look identical from the outside, and you spend a week rewriting a prompt that was never the problem. hamel.dev, checked 8 August 2026 ↗
A suite that always passes is not a suite.
If the pass rate sits at 100%, the examples are too easy and the suite has stopped telling you anything. A rate around 70% is usually the one doing real work. hamel.dev, checked 8 August 2026 ↗ We will show you a failing test on handover day and explain why we left it there.
03 · Data
Where your data goes.
Credentials are per client. The API accounts, the cloud account and any third-party subscription are opened in your name and billed to you. Nothing of yours runs through a shared Westringia account, so nothing of yours sits next to another client's work.
The provider terms matter more than anything we could promise on our own. Anthropic's commercial terms state that Anthropic may not train models on customer content from the services. anthropic.com ↗ OpenAI states that data sent to its API is not used to train or improve its models unless you opt in, and has said so since 1 March 2023. developers.openai.com ↗ Those are contract terms, they can change, and we read them again when they do. Last checked 8 August 2026.
Retention is a separate question from training. OpenAI keeps API logs for up to 30 days by default for abuse monitoring. developers.openai.com ↗ Anthropic offers a zero data retention arrangement, granted per organisation, and it does not cover every feature – batch jobs, stored files and parts of the agent tooling sit outside it. platform.claude.com ↗ We tell you which parts of your system are inside that arrangement and which are not, in writing, before anything goes live.
Where the work is processed depends on the model. Some can be pinned to Australian regions and some cannot, and it changes every few months. The dated per-model table is in field notes.
If personal information leaves the country, two things apply. Under Australian Privacy Principle 8 you must take reasonable steps first, and under section 16C you stay accountable for what the overseas recipient does with it. oaic.gov.au ↗ You cannot contract that away. You can design so that less of it has to leave, which is what we do.
The named list of services we use, and the countries they run in, is on our privacy page. It is dated, and it changes.
Here is what we do not have. We are not ISO 27001 certified. We hold no SOC 2 report and no IRAP assessment. If your procurement requires one of those, the honest answer today is no, and it is better to hear it in the first conversation than the fourth.
04 · Ownership
What you own when we are done.
All of it. That sounds like a slogan, so here is the list.
The code, in your repository.
From the first commit, not handed across at the end. We work inside your repository under your organisation. You watch it get built and you keep the history.
The accounts and the credentials.
Every key, account and subscription is in your name and on your card. Removing us is a permissions change you can make yourself in about ten minutes, and you do not need to ask us to do it.
Written procedures.
What the system does, what it deliberately does not do, what to check each week, and what to do when it stops. Written for the person who has to run it at seven in the morning.
A walkthrough video for each workflow.
Five to fifteen minutes, screen and voice, showing the real system doing the real job. New staff watch it instead of booking a training session with us.
The tests and the labelled examples.
The example set, the human labels, the graders and the CI job all go with it. That is the part most suppliers keep, and it is the part that lets the next person change something without breaking what already works.
An exit plan, in every statement of work.
Written before the work starts, not negotiated when the relationship is already sour. It names the accounts to transfer, what to switch off in what order, what the system costs a month to run without us, and what skills the next person needs.
If we get hit by a bus.
It is a fair question to ask a small firm, and you should ask it of anyone this size. Here is the answer without the sentiment.
Nothing we hold is needed to keep your system running. The code is in your repository, the credentials are in your accounts, the procedures are written and the tests run in your CI. We build on ordinary, widely used tooling, and we write down each choice and the reason for it. There is no Westringia runtime, no licence to renew and no part of it that only we can open.
The real cost you would carry is a handover. You would hire a competent contractor and give them a fortnight to read what we left behind. That is a genuine expense and we are not going to pretend otherwise. It is the reason the documentation is a deliverable with a price attached rather than a favour at the end.
05 · Technical
The technical section.
Skip this one if you do not build software. Nothing in it changes what is above. It is here because the people who review this work want the specifics, and they should not have to ask for them on a call.
Agent tools and MCP servers.
A model gets a small number of tools with narrow permissions: read a job, draft a reply, book a slot, send nothing to a customer without a human pressing the button. Where a system needs to be reachable by more than one assistant, we put it behind a Model Context Protocol server, so one set of tool definitions and permissions serves every caller. Where a single assistant would be carrying too many instructions, we split the work into sub-agents with their own tools and their own tests.
Retrieval.
Where the answer lives in your documents, we build a retrieval step and score it separately, as described in section 02. We start with the simplest approach that could work. A keyword search over a well-structured index often beats an embedding store on your data, and it is cheaper to run and easier to debug, so we test both rather than assume. Chunking, metadata filters and the refresh schedule are documented, because a stale index fails quietly and confidently.
Model routing and fallback.
Different steps get different models. A small fast one classifies and extracts, a larger one drafts anything a person will read. Each routed step has a second provider configured behind it, and we exercise the fallback path in testing rather than trusting it. Models are retired on a published schedule, so this is maintenance rather than paranoia. developers.openai.com ↗
Observability.
Every run gets a trace ID, and every message is logged with its source: user, tool, retrieval or model. Prompts and versions are logged alongside the output, so a run can be reproduced months later. This is what makes error analysis possible at all. Without it, the first serious complaint from a customer is simply unanswerable.
Cost per task.
We quote a cost per task before the build and track it after: tokens in, tokens out, retries, and how much of the context is being served from cache. Retries and long context are where the money actually goes. A workflow that costs a few cents a run is one proposition at ten runs a day and another at a thousand, and that arithmetic belongs in the business case rather than in the first invoice.
What we are not.
We are not a partner, reseller, or certified anything of OpenAI or Anthropic. We hold no badge or tier from either, and we have not applied for one. We buy the same API access you can buy and read the same documentation you can read. If a supplier tells you their tier gets you a better model, ask them which model. The routing, logging and cost tracking described here sit behind Lomandra, which we run ourselves, and we carry the pager for it.
Tell us what's slow.
Describe the part of your week that takes too long. You will get a reply from a person, usually the same day.
Or email hello@westringia.com directly.