In companies that have large engineering teams, a dedicated technical documentation organization is a necessary function. Our industry's understanding of the role of documentation, both as engineers and technical writers, has shifted its focus over the last decade.
Most notably, the way that TechDocs teams do their work has changed. Traditional technical writing, evaluated by “words on a page” (accuracy, length) has been replaced by a data-driven approach to content oversight and training. We now employ scalable automated tooling to work smarter, measure the effectiveness of our work, and to prove that we've had the impact we set out to achieve.
Some of the new measures of TechDocs effectiveness are:
- How many support requests are filed?
- How often are docs pages visited and updated?
- What is the code comment coverage?
- What is the reader sentiment of the docs?
- How long do readers spend on the page?
TechDocs: A Mission Statement
At Twitter, the TechDocs team has the following mission statement:
"The right knowledge at the right time. Great docs enable engineers to do a lot more in a lot less time. The TechDocs team exists to foster the culture of documentation."
Our goal through writing is deliver the right knowledge, at the right time, so that engineers can be more effective. We are also the champions of documentation in our company's engineering culture; ensuring that engineers care about docs, writing and maintaining their docs at a high level of quality.
Our role as owners and authors is also changing. For technical communicators, the need to author documents that require professional oversight still commands an important role in our work (especially outward-facing docs); however, the majority of our time is focused on empowering the subject matter experts (aka SMEs, software engineers in this case) to record their knowledge themselves. In this way, we can scale out the responsibilities of documentation authorship and maintenance to our engineers, and instead focus our effort on providing editorial assistance, expert level guidance and direction setting.
Four Factors for TechDocs Organizations
We can break down this approach into four functional areas of work:
- #Discovery - Deliver the right knowledge at the right time
- #Consistency - Documentation you can trust and consume easily
- #Culture - Engineers highly value creating and maintaining their own documentation
- #WriteTheDocs - Technical Communicators own and create high-impact content
#Discovery
The best documentation in the world is useless if it can't be found when it's needed. Many companies have documentation spread around in a number of separate silos. Even when it's all in one place, like a wiki, the content is usually poorly structured, making it hard to navigate and find content.
Tooling like Read The Docs can solve part of this problem by providing a single destination to consume, search and catalog content, but skilled technical communicators are required to build effective information architecture for end-user navigation.
#Consistency
As writers, we spend a lot of time thinking about things like voice, reading level and using unambiguous terminology to ensure that readers consume the information accurately and easily. This kind of consistency is important, but also larger-level concerns like document templates, titling guidelines, company glossaries, and tooling to support such standards, help both readers and authors to smooth the entire process.
For example, as a reader, when I go to a new doc titled "Tutorial" I should know what to expect from that. It should be similar in structure and intent to everything else called "Tutorial". If some docs are called "Runbooks" and some are "Tutorials", are they different? If so, the differences should be obvious and consistent within the documentation corpus. As a content author, when I'm making a new document, it should be clear whether I want to write a "Tutorial" or a "Runbook", and there should be separate templates for each that I can use, which provide the basic structure that defines that type of document.
Further, term usage should be consistent. When there are domain specific terms, or terms that are ambiguous in their general usage, but have an unambiguous domain-specific meaning, those terms should be well defined in a glossary. That glossary should be available to both readers and authors to ensure a common understanding of the intended meaning.
Every company should publish internal standards for their tech docs, and provide tools to help authors create that content using templates. Beyond that, TechDocs tooling should provide verification (linters, or other testing systems) to ensure that the published standards are adhered to. For that reason, it's best to choose standards which can be measured, or clearly indicate when something requires human oversight to verify.
#Culture
At Twitter, we have only four (4) full-time technical writers on our internal-facing TechDocs team, but we have a staff of nearly 2000 engineers working on roughly 4000 separate software projects. Since there is no way four writers could cover all that work, it's essential to leverage the massive resource of our engineering staff to write their own documentation.
To enable this, we have to ensure that authoring documentation is a delightful experience. We can create tools and other materials that are self-serve and fit into an engineer's normal daily workflow well. We need engineers to not only feel that writing docs is their responsibility, but also to feel motivated to keep the quality of their docs high, and regularly maintain them.
Our engineers should feel a sense of ownership of their documentation, and also the tools they use. For that reason, tying docs into your code review and issue tracker helps to keep docs on engineer's minds, and opening up your internal tooling for contributions from engineers motivates them to care about the tool.
#WriteTheDocs
Certain documentation is so critical to the success of the company that TechDocs must own the content, and ensure it is written at a high level of quality and regularly maintained.
Other docs require only guidance from the TechDocs team in terms of information architecture, outlining, and style. For those projects, the engineering team can author the content, working from an outline and list of tasks that TechDocs provides them on a regular basis.
TechDocs should operate like an internal agency, fielding requests and triaging them into three categories: "docs we own", "docs they own but we assist with", "docs they own and we don't assist with". Instead of a tech writer being embedded on a single project, they can work from a queue of open items from all projects, sharing the workload evenly.
Measuring TechDocs
Contemporary TechDocs teams need to be able to measure their impact to ensure they are doing the right things to improve documentation at their company. Without metrics, it's all guess work.
Now that we've established the mission of TechDocs and discussed the various factors of our work, some questions come up when we try to prove that we're succeeding in this mission:
- How do you know that your documentation is healthy?
- How do you know that your documentation is having a positive impact?
- What areas are you impacting? Who is benefiting?
- How to keep up with maintenance?
- How do you know if things are NOT going well?
Does your current tooling answer these questions? No, it doesn't. That means building the tools to measure, then building the dashboards, monitoring, alerting, and reporting that uses those measurements. We need better tooling, and we should probably share the effort by building it in the open.
Things We Can Measure
While the value of metrics is obvious, what is not so obvious is what kind of things we can accurately measure about documentation and documentation culture.
Here are some ideas:
- Doc Linter - Measure some basic features about a document to determine its quality, objectively (reading level/lexical density, reading time, doc contains contact email?, code snippets valid?, etc)
- Total Doc Commits - Overall, how many commits to doc files? High level of activity? What projects/topics?
- Drift - Measure the commit density distance between a piece of code and it's related docs. Commit density on code vs commit density on docs in past 30 days? (Note: density is # of adds/removes in total vs # of changesets) Did the code move forward, but the docs didn't? That's called '_drift_' and it's bad news!
- Traffic - How many people are reading the docs? Who are they? How long are they spending on the page?
- Reader Sentiment - Five star rating on page, NPS survey, etc. Are readers happy with the docs?
- Support Tickets - What questions are people asking about? What doc pages answer that? Top ten. Did a new doc on a topic reduce inbound support tickets on that topic?
- Shoulder Taps - Parse Hipchat/Campfire/Slack logs, look for 1:1 communication between engineers. Convos that start with "quick question", or just start with a question and don't mention lunch. How many? Use NLP to categorize topics/nouns. What topics are generating shoulder taps? How to track real-life shoulder taps?
- Docs Tickets - How many JIRA/GitHub Issues are filed related to documentation (either doc bugs, or doc improvement tasks for owners, etc). Increasing or decreasing over time? Shows improvement in docs culture?
Documentation Health
All those measurements are useful by themselves but that might be too many things for people to think about. After all, one of the big #culture problems around docs is that they are currently viewed as a "secondary concern" to the code. Any complexity around a "secondary concern" will cause folks to stop thinking about it.
To keep things simple, we can aggregate the metric signals into a single measure… your documentation "Health Score".
This is a number between 0 and 10, telling you how healthy your docs are. Each of the above signals can be implemented to provide a unit interval (a value between 0 and 1) which can be composited into a normed vector space, with a rank applied to each signal to influence the final score. You can tweak your algorithm to emphasize the signals your organization cares about.
This score could be displayed on the project homepage, so engineers who are project owners can see a clear indicator of how they are doing. You can provide those engineers with tools to iteratively improve this number. The score can be clickable, to see a detailed report of what is wrong. Your documentation build system can regularly email reports to project owners with stats about their project, to let them know how they are doing. When they are doing well, you can reward them with perks like shoutouts to their managers, or swag like stickers, shirts, or other material items.
Conclusion
Contemporary TechDocs, in businesses that produce software products, is no longer as simple as hiring tech writers. You can't just park more tech writers in chairs and tell them to crank away at writing docs, adding more warm bodies as the workload increases. Brute force doesn't work. It doesn't scale. As Martin Sustrik recently pointed out, we have to operate under new principles, which support a more scalable and modern approach.
A snow shovel works well to clear a driveway, but you'll need a more sophisticated approach to keep the roads safe and clear for an entire city.
NOTE: Significant portions of this blog post are taken nearly-verbatim, or lightly paraphrased from Twitter's TechDocs team documents which were a collaborative effort. Other pieces are directly ripped from conversations I've had with various Write The Docs community members. Specifically, co-authorship credits should go to my team at Twitter (Marko Gargenta, Nancy Groschwitz, Ricardo Cervera, Luc Perkins) as well as WTD community members Nik Blanchet and Garen Torikian.