Finding Flaws in Technical Documentation

 

An idea

I spent some time on a project and had finished updating the tutorials and reference documentation I'd created as Markdown .md files. Then I suddenly thought:

Why not use the AI to test the documentation?

So I asked ChatGPT if it could create an isolated new chat that read just the project's .md documentation files and then tried to use the tool.

Putting it to the test

My current chat session said it could, created a new chat, and expanded my sentence into an initial prompt that included a lot more detail on what to do and what it could not do.

It came back to me and said it would run and monitor the new chat and report back on any documentation deficiencies it found.

That led to several points, some valid and some not, that I then addressed.

One example was that the documentation didn't explicitly state that the tool was intended for Linux use. It included setup details for Linux and WSL, but that hadn't stopped the AI from trying to work out how to run it in a Windows environment.

An obvious assumption on my part, perhaps, but one worth making explicit.

A new, (to me), way to review documentation

I like the idea of getting AI to act as someone new to a project, trying to get something done and reporting on any problems encountered along the way.

It's a way of testing the documentation by actually trying to follow it, rather than just asking AI to review the wording.

Of course, the AI won't always get it right. Some of the points it raised weren't valid, so you still need to use your own judgement when deciding what to change.

Documentation for humans and AI

You still have to make documentation approachable for any user, without requiring them to use AI. But I'd expect more of my target audience to use AI to help them work with projects.

That means documentation has to serve two purposes: it needs to be usable by people reading it directly, but also clear enough for an AI assistant to use when helping someone get things done.

I'm not suggesting we should write documentation just for AI. But if AI is going to be another way people interact with our projects, it makes sense to consider how well our docs support that.

Perhaps we should be testing documentation in much the same way we test code: by seeing whether someone, or something, can actually follow it and accomplish the intended task.

Something worth trying on your next project?