From vibe coding to verified: Practising independent diagnosis

An assistant writes a retry loop. The tests pass. Before accepting it, can you explain what happens if the remote service completes the operation but the response never arrives?

That question is more useful to me than whether the developer used “vibe coding”. It asks whether someone understands a consequential behaviour of the code they are about to own. They might answer by reading the implementation, checking the API contract, or running a targeted experiment. Memorising the assistant's explanation would not settle it.

I use cognitive fitness here to mean the ability to form a diagnosis, test it, and revise it when evidence disagrees. It is a working definition for engineering practice, not a clinical measure. If we expect people to supervise AI-generated work, we should give them opportunities to exercise that ability and check whether our approach helps.

What the research does and does not establish

Two studies discussed earlier in this series help frame the question, but neither proves that coding assistants cause engineers to lose their skills.

Nataliya Kosmyna and colleagues' Your Brain on ChatGPT preprint studied an essay-writing task. It reports 54 participants in the first three sessions, with 18 completing a fourth session that changed tool conditions. The researchers observed differences in EEG connectivity, recall, and reported ownership across conditions.[1] Those results concern that task and sample. EEG connectivity is not a direct measure of engineering competence, and the study does not establish lasting impairment in software developers.

Hao-Ping Lee and colleagues surveyed 319 knowledge workers, collecting 936 examples of generative AI use. Greater confidence in AI was associated with less reported critical thinking; greater task-specific self-confidence was associated with more.[2] The study also describes critical thinking shifting towards verification and integration. This is self-reported, observational evidence, not a longitudinal test of skill loss.

Neither study tests the exercises below. I am proposing them as ways to make diagnostic capability visible, with evaluation built in. A team could try them and find that they add effort without improving judgement. That result would matter too.

Start with a prediction

For a practice session, use a synthetic service with a retrying client. Before revealing an assistant's proposed fix, ask the engineer to predict what will happen when a successful response is lost. Which operation might happen twice? What evidence would distinguish a retry from a second user request?

A short written prediction is enough. Then let the engineer use the assistant and normal documentation to investigate. Compare the initial prediction with the observed behaviour and the final diagnosis. The interesting part is where the explanation changed and why.

This avoids making “work without AI” the entire exercise. Tool use is part of engineering. The point is to preserve a moment in which the engineer commits to a testable view before the assistant supplies one.

Do this away from an active incident. Delaying mitigation so someone can complete a learning exercise would confuse training with response.

The same pattern works for code review. Before reading a generated rationale, ask what property the patch must preserve. For the retry example, that might be avoiding duplicate effects for the same request. The reviewer should then inspect the implementation and tests against that property, rather than rate how convincing the rationale sounds.

Make disagreement testable

A second exercise starts with an intentionally incomplete diagnosis in a synthetic case. The assistant's suggestion might fit a visible timeout while failing to account for a duplicate write. Ask the reviewer to identify the missing evidence and choose a check that separates the competing explanations.

The exercise should not reward disagreement for its own sake. Sometimes the recommendation is correct. Include those cases too, and ask what evidence warrants accepting it. Otherwise the team learns to perform scepticism rather than exercise judgement.

A useful review note might say, “The operation succeeded before the connection closed, so retrying without a stable request identifier can duplicate the effect.” That is a claim someone else can inspect. “I reviewed the answer carefully” tells us very little.

Explanations generated by the assistant can help the investigation, but they are not privileged access to its reasoning or proof of correctness. Give reviewers the relevant contract, trace, and diff. Let them reach a different conclusion without fighting the interface.

There is a coaching question here too: did the reviewer lack knowledge, overlook evidence, or lack access to it? Those failures need different responses. More mandatory explanation fields will not repair missing telemetry.

Check whether the capability transfers

Success on the same exercise a week later may show familiarity with the answer. I would use a different synthetic case that requires the same diagnostic move, such as distinguishing a rejected request from an accepted request whose acknowledgement was lost. The surface details should change without quietly making the task much harder.

Evaluate both assisted performance and a short diagnostic task without generated recommendations. Keep normal reference documentation available and define the tool conditions in advance. We are testing whether someone can investigate, not whether they can recite an API from memory.

The assessment needs a rubric. Did the person identify the relevant failure mode, choose a discriminating check, notice an unsafe scope, and recognise when the evidence was insufficient? Record time spent as a cost, not as a substitute for correctness. A fast wrong answer is still wrong; a correct answer that arrives too late may be operationally inadequate.

Compare with a baseline and, where practical, a comparable group or alternating task order. Experience, case difficulty, and familiarity with the code can all affect results. A small pilot will not isolate every cause, so report those limits rather than announce that cognitive decline has been prevented.

Avoid turning learning into surveillance

I would not rank engineers by override rate or the number of times they open an explanation. Low overrides can mean accurate recommendations. High overrides can mean a bad assistant. Neither is a clean measure of someone's judgement.

Keep the initial pilot focused on team learning, with clear limits on recording and retention. Discuss mistakes in terms of the evidence and the workflow. If people expect a wrong hypothesis to become a performance-management label, they have a reason to hide the very uncertainty the exercise is meant to surface.

And stop adding friction if it is merely generating paperwork. The useful outcome is an engineer who can explain which observation would change their mind, then obtain that observation. A page of mandatory prose is not an acceptable substitute.

For the next generated patch you review, choose one behaviour the tests might have missed. Predict it, test it, and keep the result. That gives you something concrete to learn from, whether the assistant was right or wrong.