{"id":200,"date":"2019-10-05T19:49:31","date_gmt":"2019-10-05T19:49:31","guid":{"rendered":"http:\/\/jwonsever.com\/wp\/?p=200"},"modified":"2019-10-05T19:49:31","modified_gmt":"2019-10-05T19:49:31","slug":"breaking-down-complicated-json","status":"publish","type":"post","link":"https:\/\/jwonsever.com\/wp\/?p=200","title":{"rendered":"Breaking Down Complicated JSON"},"content":{"rendered":"

Working title: \u201cUsing D3 flame charts to provide insight into the largest components of large, variable JSON objects\u201d.<\/span><\/p>\n

\"\"<\/a><\/p>\n

Summary<\/span><\/h2>\n

I\u2019ve written a tool to visualize the size of a JSON object. \u00a0 It takes any JSON object as input and creates a JSON format which D3 can consume to create a flame chart representing the size of the JSON and all of its subobjects.\u00a0 In the simplified example above, you can see that the \u201cfriends\u201d object in the list clearly should be the first optimization target.<\/span><\/p>\n

Code, example data, and README is available at <\/span>https:\/\/github.com\/Jwonsever\/json_flame<\/span><\/a>.<\/span><\/p>\n

Inspiration partly goes to <\/span>https:\/\/github.com\/spiermar\/burn<\/span><\/a> and <\/span>https:\/\/github.com\/spiermar\/d3-flame-graph<\/span><\/a>, which is used to convert profiler output to flame charts, partly to YourKit<\/a> and JProfiler<\/a>, and partly to the Chrome Dev Tools performance analysis.\u00a0 \u00a0 All of which provide great insight into performance problems using flame charts.<\/span><\/p>\n

The Story<\/span><\/h2>\n

This should be a quick little read.\u00a0 I was a Performance Engineer for the first couple years of my career, and while I no longer want to work on software performance every day, sometimes there is a problem that comes up that piques my interest.\u00a0 This post is about one.<\/span><\/p>\n

One of the most common performance problems, present in practically every web application, is an excess of data. \u00a0 Obviously, the less code you have to download, the faster your site will load. It\u2019s very easy to see how much time is spent downloading using Chrome Dev Tools, just go to the network tab, and <\/span>look at the blue bar!<\/span><\/i><\/p>\n

\"\"<\/a><\/p>\n

The green represents server time processing, and the blue is download time.\u00a0 In that time, neither the server nor the client is doing any work, and that waiting time is critical to optimize.\u00a0 While there are tons of types of data being sent to the web client, any of which can be too large, I\u2019ve found that excessively large JSON objects come up often and are one of the easiest problems to target.<\/span><\/p>\n

I recently ran into a problem where I had an extremely complicated JSON object which I didn\u2019t know the structure of.\u00a0 \u00a0 I simply wanted to diagnose which subobject were the largest contributors, so I know where to focus my optimization first.\u00a0 This is a common pattern, (Chrome memory and performance analysis give tools for this, its commonly used for <\/span>profiling<\/span><\/a>, etc.)\u00a0 Most engineers have seen flame charts in the past and used them for such purposes.\u00a0 Surprisingly, I wasn\u2019t able to find a way to visualize the components of JSON in this way, so I built one.\u00a0 Given it really only took a couple hours to get the first version of it working, I\u2019m still in disbelief that this didn\u2019t already exist.\u00a0 Maybe I just wasn\u2019t trying hard enough to find it.<\/span><\/p>\n

The Tool<\/span><\/h2>\n

The tool is relatively simple.\u00a0 It takes an input JSON, generates an out.json file with the performance data, then starts a local express node app to visualize the data using D3 using <\/span>d3-flame-graph<\/span><\/a>.<\/span><\/p>\n

Without any optional parameters, you will see every composite object.\u00a0 Unfortunately, that output often is impossible to read:<\/span><\/p>\n

\"\"<\/a><\/p>\n

It\u2019s a long array of the same object.\u00a0 Long lists of items are common and thus I added an extra mode: \u201c(-a)ggregate mode\u201d.<\/span><\/p>\n

Aggregate mode goes through an array and acts as if each of its child objects are the same object, summing up their contributions into a single object.<\/span><\/p>\n

This lets you take the first graph, and coalesce it into something much easier to parse visually:
\n\"\"<\/a>
\n<\/span><\/p>\n

I also added another optional mode: \u201c(-k)eys mode\u201d.\u00a0 The difference there is only that the objects keys are included in its representation.\u00a0 This makes a big difference if you are looking to determine excessively long keys and their contributions to the greater object.\u00a0 Generally, I find it provides a clearer representation to what is going on in the object, so I recommend using it as well.<\/span><\/p>\n

Next Steps<\/span><\/h2>\n

This seems like it should be a SAAS tool that I should put online somewhere for people to use.\u00a0 If anyone wants to see that, please contact me! Maybe we can work something out.<\/span><\/p>\n

Right now, the code abuses JSON parse and JSON stringify to determine the size of the JSON object.\u00a0 Ideally, the blob would be measured directly, so all of the supplied whitespace would be accounted for.\u00a0 There\u2019s a placeholder file \u201cmeasure_without_json_parse.js\u201d which marks this todo item.<\/span><\/p>\n

There will be a second blog soon called something like \u201cMinifying JSON\u201d with another cute trick to make JSON a little smaller.\u00a0 Watch out for it in the next couple weeks!<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

Working title: \u201cUsing D3 flame charts to provide insight into the largest components of large, variable JSON objects\u201d. Summary I\u2019ve written a tool to visualize the size of a JSON object. \u00a0 It takes any JSON object as input and creates a JSON format which D3 can consume to create a flame chart representing the<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-200","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/jwonsever.com\/wp\/index.php?rest_route=\/wp\/v2\/posts\/200","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/jwonsever.com\/wp\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/jwonsever.com\/wp\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/jwonsever.com\/wp\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/jwonsever.com\/wp\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=200"}],"version-history":[{"count":8,"href":"https:\/\/jwonsever.com\/wp\/index.php?rest_route=\/wp\/v2\/posts\/200\/revisions"}],"predecessor-version":[{"id":208,"href":"https:\/\/jwonsever.com\/wp\/index.php?rest_route=\/wp\/v2\/posts\/200\/revisions\/208"}],"wp:attachment":[{"href":"https:\/\/jwonsever.com\/wp\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=200"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/jwonsever.com\/wp\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=200"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/jwonsever.com\/wp\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=200"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}