Compressing ResourceTiming

November 7th, 2014

At SOASTA, we’re building tools and services to help our customers understand and improve the performance of their websites. Our mPulse product utilizes Real User Monitoring to capture data about page-load performance.

For browser-side data collection, mPulse uses Boomerang, which beacons every single page-load experience back to our real time analytics engine. Boomerang utilizes NavigationTiming when possible to relay accurate performance metrics about the page load, such as the timings of DNS, TCP, SSL and the HTTP response.

ResourceTiming is another important feature in modern browsers that gives JavaScript access to performance metrics about the page’s components fetched from the network, such as CSS, JavaScript and images. mPulse will soon be releasing a new feature that lets our customers view the complete waterfall of every visitor’s session, which can be a tremendous help in debugging performance issues.

The challenge with ResourceTiming is that it offers a lot of data if you want to beacon it all back to a server. For each resource, there’s data on:

  • URL
  • Initiating element (eg IMG)
  • Start time
  • Duration
  • Plus 11 other timestamps

Here’s an example of performance.getEntriesByType('resource') of a single resource:

{"responseEnd":2436.426999978721,"responseStart":2435.966999968514,
"requestStart":2435.7460000319406,"secureConnectionStart":0,
"connectEnd":2434.203000040725,"connectStart":2434.203000040725,
"domainLookupEnd":2434.203000040725,"domainLookupStart":2434.203000040725,
"fetchStart":2434.203000040725,"redirectEnd":0,"redirectStart":0,
"initiatorType":"internal","duration":2.2239999379962683,
"startTime":2434.203000040725,"entryType":"resource","name":"http://nicj.net/"}

JSON.stringify()‘d, that’s 469 bytes for this one resource.  Multiple that by each resource on your page, and you can quickly see that gathering and beaconing all of this data back to a server will take a lot of bandwidth and storage if you’re tracking this for every single visitor to your site. The HTTP Archive tells us that the average page is composed of 99 HTTP resources, with an average URL length of 85 bytes.

So for a rough estimate you could expect around 45 KB of ResourceTiming data per page load.

The Goal

We wanted to find a way to compress this data before we JSON serialize it and beacon it back to our server.

Philip Tellis, the author of Boomerang, and I have come up with several compression techniques that can reduce the above data to about 15% of it’s original size.

Techniques

Let’s start out with a single resouce, as you get back from window.performance.getEntriesByType("resource"):

{  
  "responseEnd":323.1100000002698,
  "responseStart":300.5000000000000,
  "requestStart":252.68599999981234,
  "secureConnectionStart":0,
  "connectEnd":0,
  "connectStart":0,
  "domainLookupEnd":0,
  "domainLookupStart":0,
  "fetchStart":252.68599999981234,
  "redirectEnd":0,
  "redirectStart":0,
  "duration":71.42400000045745,
  "startTime":252.68599999981234,
  "entryType":"resource",
  "initiatorType":"script",
  "name":"http://foo.com/js/foo.js"
}

Step 1: Drop some attributes

We don’t need:

  • entryType will always be resource
  • duration can always be calculated as responseEnd - startTime.
  • fetchStart will always be startTime (with no redirects) or redirectEnd (with redirects)
{  
  "responseEnd":323.1100000002698,
  "responseStart":300.5000000000000,
  "requestStart":252.68599999981234,
  "secureConnectionStart":0,
  "connectEnd":0,
  "connectStart":0,
  "domainLookupEnd":0,
  "domainLookupStart":0,
  "redirectEnd":0,
  "redirectStart":0,
  "startTime":252.68599999981234,
  "initiatorType":"script",
  "name":"http://foo.com/js/foo.js"
}

Step 2: Change into a fixed-size array

Since we know all of the attributes ahead of time, we can change the object into a fixed-sized array. We’ll create a new object where each key is the URL, and its value is a fixed-sized array. We’ll take care of duplicate URLs later:

{ "name": [initiatorType, startTime, redirectStart, redirectEnd,
   domainLookupStart, domainLookupEnd, connectStart, secureConnectionStart, 
   connectEnd, requestStart, responseStart, responseEnd] }

With our data:

{ "http://foo.com/foo.js": ["script", 252.68599999981234, 0, 0
   0, 0, 0, 0, 
   0, 252.68599999981234, 300.5000000000000, 323.1100000002698] }

Step 3: Drop microsecond timings

For our purposes, we don’t need sub-milliscond accuracy, so we can round all timings to the nearest millisecond:

{ "http://foo.com/foo.js": ["script", 252, 0, 0, 0, 0, 0, 0, 0, 252, 300, 323] }

Step 4: Trie

We can now use an optimized Trie to compress the URLs. A Trie is an optimized tree structure where associative array keys are compressed.

Mark Holland and Mike McCall discussed this technique at Velocity this year.

Here’s an example with multiple resources:

{
    "http://": {
        "foo.com/": {
            "js/foo.js": ["script", 252, 0, 0, 0, 0, 0, 0, 0, 252, 300, 323]
            "css/foo.css": ["css", 300, 0, 0, 0, 0, 0, 0, 0, 305, 340, 500]
        },
        "other.com/other.css": [...]
    }
}

Step 5: Offset from startTime

If we offset all of the timestamps from startTime (which they should always be larger than), they may use fewer characters:

{
    "http://": {
        "foo.com/": {
            "js/foo.js": ["script", 252, 0, 0, 0, 0, 0, 0, 0, 0, 48, 71],
            "css/foo.css": ["script", 300, 0, 0, 0, 0, 0, 5, 40, 200]
        },
        "other.com/other.css": [...]
    }
}

Step 6: Reverse the timestamps and drop any trailing 0s

The only two required timestamps in ResourceTiming are startTime and responseEnd. Other timestamps may be zero due to being a Cross-Origin resource, or a timestamp that was “zero” because it didn’t take any time offset from startTime, such as domainLookupStart if DNS was already resolved.

If we re-order the timestamps so that, after startTime, we put them in reverse order, we’re more likely to have the “zero” timestamps at the end of the array.

{ "name": [initiatorType, startTime, responseEnd, responseStart,
   requestStart, connectEnd, secureConnectionStart, connectStart,
   domainLookupEnd, domainLookupStart, redirectEnd, redirectStart] }
{
    "http://": {
        "foo.com/": {
            "js/foo.js": ["script", 252, 71, 48, 0, 0, 0, 0, 0, 0, 0, 0, 0]
            "css/foo.css": ["script", 300, 200, 40, 5, 0, 0, 0, 0, 0, 0, 0, 0]
        }
    }
}

Once we have all of the zero timestamps towards the end of the array, we can drop any repeating trailing zeros. When reading later, missing array values can be interpreted as zero.

{
    "http://": {
        "foo.com/": {
            "js/foo.js": ["script", 252, 71, 48]
            "css/foo.css": ["css", 300, 200, 40]
        }
    }
}

Step 7: Convert initiatorType into a lookup

Using a numeric lookup instead of a string will save some bytes for initiatorType:

var INITIATOR_TYPES = {
    "other": 0,
    "img": 1,
    "link": 2,
    "script": 3,
    "css": 4,
    "xmlhttprequest": 5
};
{
    "http://": {
        "foo.com/": {
            "js/foo.js": [3, 252, 71, 48]
            "css/foo.css": [4, 300, 200, 40]
        }
    }
}

Step 8: Use Base36 for numbers

Base 36 is convenient because it can result in smaller byte-size than Base-10 and has built-in browser support in JavaScript toString(36):

{
    "http://": {
        "foo.com/": {
            "js/foo.js": [3, "70", "1z", "1c"]
            "css/foo.css": [4, "8c", "5k", "14"]
        }
    }
}

Step 9: Compact the array into a string

A JSON string representation of an array (separated by commas) saves a few bytes during serialization. We’ll designate the first byte as the initiatorType:

{
    "http://": {
        "foo.com/": {
            "js/foo.js": "370,1z,1c",
            "css/foo.css": "48c,5k,14"
        }
    }
}

Step 10: Multiple hits

Finally, if there are multiple hits to the same resource, the keys (URLs) in the Trie will conflict with each other.

Let’s fix this by concatenating multiple hits to the same URL via a special character such as pipe | (see foo.js below):

{
    "http://": {
        "foo.com/": {
            "js/foo.js": "370,1z,1c|390,1,2",
            "css/foo.css": "48c,5k,14"
        }
    }
}

Step 11: Gzip or MsgPack

Applying gzip compression or MsgPack can give additional savings during transport and storage.

Results

Overall, the above techniques compress raw JSON.stringify(performance.getEntriesByType('resource')) to about 15% of its original size.

Taking a few sample pages:

  • Search engine home page
    • Raw: 1,000 bytes
    • Compressed: 172 bytes
  • Questions and answers page:
    • Raw: 5,453 bytes
    • Compressed: 789 bytes
  • News home page
    • Raw: 32,480 bytes
    • Compressed: 4,949 bytes

How-To

These compression techniques have been added to the latest version of Boomerang.

I’ve also released a small library that does the compression as well as de-compression of the optimized result: resourcetiming-compression.js.

This article also appears on soasta.com.

Share this:

  1. January 9th, 2015 at 15:02 | #1

    This is great advice. Resource Timing data provides a lot of insight, but can be large. The techniques you describe make it possible to send back the needed data with higher likelihood of success (over the alternatives of multiple beacons or POST).

    I found two additional techniques for compressing the data further.

    1. Deltas – Rather than send back every time property’s value, I convert them to deltas. For example, you have this structure with 12 array elements:

    “name”: [initiatorType,
    startTime,
    responseEnd,
    responseStart,
    requestStart,
    connectEnd,
    secureConnectionStart,
    connectStart,
    domainLookupEnd,
    domainLookupStart,
    redirectEnd,
    redirectStart]

    Here’s the “deltas” version of the same info:

    “name”: [initiatorType,
    startTime,
    content, // responseEnd – responseStart
    ttfb, // responseStart – requestStart
    connect, // connectEnd – connectStart
    ssl, // secureConnectionStart – connectStart
    dns, // domainLookupEnd – domainLookupStart,
    redir] // redirectEnd – redirectStart

    That reduces it from 12 array elements to 8.

    2. Key vs delimiter – As you mention, many of the values are zero. The problem with an array is that if the *latter* values are non-zero, all the intermediate zeroes *must* be specified to preserve position. Since we later convert the array to a string, rather than using a delimiter character (such as comma) I use single character keys – “C” for “content”, “T” for “ttfb”, etc. That way, it’s not necessary to preserve order, and all zero values can be left out.

    For example, in your approach if there’s a 12ms redirect but no time is spent on DNS or connect, the array is [“script”, 252, 71, 48, 0, 0, 0, 0, 0, 0, 0, 12, 0] which converts to the string “370,1z,1c,0,0,0,0,0,0,0,c”. The “deltas” approach array is [“script”, 252, 23, 48, 0, 0, 0, 12] which converts to the string “370CnT1cRc” – 10 chars vs 25.

    Note 1: I user upper case chars for my delimiter keys so they don’t get confused with the base 36 alpha characters.

    Note 2: In my approach I add a 9th element to the array: blocking. This is needed if you want to construct a waterfall.

    I bet there are a few places where these additional optimizations mess up. Given SOASTA’s huge amount of experience with RUM, it’d be great to hear your feedback on whether these are worthwhile.

  2. January 10th, 2015 at 06:32 | #2

    @ Steve Souders
    Steve,

    Thanks for the ideas, those are both great improvements.

    For the deltas version, the only concern I have is using it when re-constructing the complete Waterfall. For most cases of RT we’ve seen in the wild, the end/start timestamps line up (eg domainLookupEnd==connectStart, connectEnd==requestStart), but if they’re slightly off (say the browser didn’t immediately send the request after TCP connection, so connectEnd is 10 but requestStart is 50), then you’d be missing the fact that the overall duration is 40ms longer than the deltas suggest.

    I’ll do a bit of in-the-wild analysis to see if this is prevalent or not. I’d also be worried about future implementations (eg FF and maybe even Safari some day) having this issue and it not being obvious.

    The Key vs Delimiter approach is great. With the RT data we’ve seen so far, we’ve found it very rare for 0-in-the-middle timestamps, but you do suggest one (probably somewhat common) example if the resource was redirected to different content on the same domain. I like using the keys instead of a fixed array, and whether you’re doing “deltas” or all of the timestamps, you could use the same approach.

  3. Scott Povlot
    June 21st, 2015 at 15:17 | #3

    Nic,

    Has ResourceTiming Compression been added to the latest version of Boomerang yet? I don’t seem to find the code in the GitHub repository. What source file is this included in?

    Scott

  4. Wolfgang Egartner
    May 22nd, 2017 at 05:47 | #5

    Nic,

    many thx for sharing the details to the compression, which I found very helpful to unravel raw data from mpulse except for one small detail: sometimes there is a star in the base36 strings like “8*1mh” which I do not find in the article and I guess there is a special meaning to this.

    Would be great if you could provide some hint about how this is supposed to be handled.

    Best regards,
    Wolfgang

  5. Wolfgang Egartner
    June 1st, 2017 at 13:45 | #7

    Hi Nic,

    many thx!

    In the meantime I have found the decompression javascript on github, and found it very readable, It is all there, e.g. “*1” being just a separator between timings and sizes, just as you say.

    I have already successfully tested that in R and it works great.

    Thx again, this is really great stuff,
    Wolfgang

  6. EyalG
    September 11th, 2017 at 03:02 | #8

    Hi Nic,

    is there anyway to determine the response code by transfer size / decodedSize ?
    thanks!

    • September 11th, 2017 at 04:17 | #9

      @EyalG: You could probably infer a 304 vs 200 hit if the transferSize is minimal but the decodedSize is large (and responseStart/responseEnd would probably be equal or minimal).

      Not sure if there are any other codes you could determine or guess, unless you have ideas?

  1. No trackbacks yet.