Metrics Case Study: Total Time Reading (TTR)

In a previous article I discussed the need for a data driven approach to TechDocs. This article will discuss one possible documentation metric: Total Time Reading (TTR).

Definition and Implementation of TTR

So what exactly is Total Time Reading (TTR)? TTR is the total amount of time that a reader spends actively engaging with a piece of content. This is usually expressed in seconds, and recorded along with a URL for the content in a metrics database.

One way to implement this is by using the JavaScript library TimeMe by Jason Zissman. This library will keep track of the time a user is focused on the page. If they switch tabs, minimize the window, or otherwise unfocus the content, it will stop tracking their time until they return. I've used this library and find that it works pretty solidly.

Here's some example code:

<script type="text/javascript">
  $(document).ready(function () {
    // track user activity time
    TimeMe.setIdleDurationInSeconds(30);
    TimeMe.setCurrentPageName("my-home-page");
    TimeMe.initialize();

    // record page visit event when user leaves the page
    window.onbeforeunload = function (event) {
      xmlhttp=new XMLHttpRequest();
      xmlhttp.withCredentials = true;
      xmlhttp.open("POST", "/event/visit", false);
      xmlhttp.setRequestHeader("Content-type", "application/x-www-form-urlencoded");
      var event_data = {
        total_time_reading: TimeMe.getTimeOnCurrentPageInSeconds(),
        page: window.location.href
      };
      xmlhttp.send($.param(event_data));
    };
  });
</script>

That code will start the timer once the page fully loads, and then, using the window.onbeforeunload, will trigger an AJAX post to record the TTR metric and the URL, when the user closes the page.

Adding that to every page footer as part of your docs default HTML template will get you started collecting TTR measurements.

Interpreting the Data

Now that we have TTR recorded, what do we do with it? First thing to realize about TTR is that it also functions as a basic page hit tracker. It is a more precise measurement than just "did a user visit the page or not", but it can also be used in that way.

The most basic things you can use it for would be detecting hot-spots and dead-wood. Which pages get the most hits? Which pages get no hits, or very few?

Further, we can now look at the combination of how much time and how often. Frequent visits to a page, with low amount of time on it tells you what? Something about that page is drawing users in, but they don't need to spend much time there. Is that because they didn't find what they were looking for and left after scanning it? Or is it because the page is reference style documentation, for something very commonly used, but is perhaps a bit confusing with details that are easily forgotten.

How about a page that does not get very many visits, but always has a long TTR? Well… how long is "long" anyway? How many seconds should a user spend on that page?

You could calculate the Estimated Reading Time (ERT) for the page, then compare that to the amount of time that a users are actually spending on it. If a page has a ERT of ten (10) minutes, but users are only spending about five (5) minutes on it.. Why is that? Does the reader get distracted part-way through the page because the content is poorly written? Or maybe it is the kind of article that has more than one separate section, which can be used independently, and the user scans, finds the section they need, which only takes five (5) minutes to read.

It is dangerous to infer too much about what TTR means. However, it is useful to look at outliers. If a given page has a 20 minute ERT and only sees about two (2) minutes of engagement on a regular basis, it is worth looking at that to understand why that might be. In fact, you might ask your users to tell you (send out a survey asking for feedback, or pop-up a feedback form right on the "outlier" page).

Another approach: You could calculate the ERT, compare it to the TTR, and record a percentage like: TTR / ERT * 100. Maybe you could apply a rule, for your content, that a ratio of TTR to ERT less than 30% or more than 150% is considered "unhealthy" and the page should be looked at? YMMV, so tweak those numbers to suit your content and users. Perhaps you can configure that per page?

NOTE: Calculate Estimated Reading Time (ERT) using the reading-time library from Nicolas Gryman.

Even without a ERT to compare to, a pattern of very short engagements (less than one (1) minute) and very long engagements (>20 minutes) probably indicate a problem, in general. Most technical documentation should be broken up to be 20 minutes or less for the ERT, and it is rare than you can get anything useful out of a page in under one (1) minute as a reader. Flag these "unhealthy" TTR events and focus some time on improving those pages.

Also, you will want to look at changes in the norm. If a page suddenly gets a lot more or a lot less engagement, you might want to alert on that and try to understand why the user behaviour suddenly changed.

Conclusion

TTR is an interesting metric, but interpreting it can be difficult. It is very sensitive to the content, especially to the reading length. If you're going to track visits, you should also be tracking TTR, because it is pretty easy to do. Even if the data is hard to understand, it is better to collect more precise/rich data if possible.