Builder-web visualizations at Robur

let created = `Date (2022, 3, 7) in

We've added two new interactive visualizations to builder-web's build page. The first one called dependencies is a view of the build-binarys direct dependencies, and their respective direct dependencies. The second one is an ordered treemap of the space usage of a dissection of the binarys constituent libraries, which is a fork of Drup's modulectomy.

Both visualizations are implemented solely using CSS and SVG, to avoid JS. There are several reasons for not using JS when possible; this includes user privacy, trust and performance, which also results in less energy usage. On the serverside the visualizations are cached, for faster loadtimes and avoiding unneccesary computations.

This blogpost is a development story about the implementations and about the iterative design of the dependencies visualization.

Treemap: A dissection of the binary

Treemap is a fork of modulectomy, which already had an interactive CSS + SVG output. The goal of adding the visualization to builder-web is to enable the user to:

  • inspect what is in the binary
  • optimize the size of the binary based on what libs take up the most space, either by avoiding a certain library, or helping make the library leaner

We switched modulectomy away from visualizing symbols in the analyzed ELF binary (which uses the Owee library behind the scenes) - to visualizing OCaml modules/libraries instead. This both brought performance gains in the generation of the SVG, and for the user viewing it, because of fewer rendered elements. This change also made modulectomy work with ELF files generated on FreeBSD (which uses Clang).

To do this, the implementation now looks at code_begin..end and data_begin..end in the debug-binary, as well as C-symbols. To solve that the C-symbols are separate from OCaml module sections, we chose to manually include sets of symbols from popular OCaml libraries. We might look into automating this.

A bunch of fixes was also made - one was to correct the sizes of visualized boxes, and later we found that there were a lot of duplicate symbols generated by Owee, resulting in wrong sizes. We quickfixed the latter on the side of modulectomy, but will optimally be fixed on the side of Owee.

Finishing up the user experience - we chose to filter the smallest rectangles away from the treemap, which allows more overall area to the rectangles that matter. An extension, treemap-scale, was added to the SVG-output, which shows how big a part the included and excluded treemap-rectangles constitutes of the actual binary.

You interact with treemap-scale by hovering over parts of it, and the respective sizes are shown via a popup - the parts included are visually marked. The rectangles in the treemap represent the relative sizes of the modules/libraries in the binary. When hovering over these rectangles, you get a popup showing the size used.

Future work also includes automatically merging modules stemming from the same library.

Dependencies: A limited view of the dependency-graph

This visualization is an outshoot from Hannes opam-graph tool. The intention is to:

  • make an understandable view of complex dependency-graphs
  • be able to track how the dependencies change over time
    • this can be seen by comparing with old builds dependencies visualizations
  • give an overview of the amount of dependencies, e.g. for:
    • exploration of direct dependencies and their own dependency-sets
    • evaluating sharing of dependencies, e.g. for minimization of the codebase

The development was done in an iterative design process. First I looked at existing visualizations of dependency-graphs, which all were too complex to communicate anything clearly. After mocking up possible designs of limited views of the graph, I began prototyping the UI via Graphviz using ocamldot in opam-graph. This led to insights into the average size of the set of direct and transitive dependencies, and how much screen-space one would need to show different limited views of the full dependency-graph.

The final interactive visualization we also wanted to be done in CSS + SVG, as with treemap. The libraries used was Tyxml for typesafe SVG and Gg for vector-math. A problem that comes up when you use CSS + SVG is that both systems depend on the order of SVG elements. CSS can only select siblings in front of an element, and SVG has a paint-order defined by the order of elements. The limitation of CSS to only select forwards, becomes a problem when you want to make an interactive visualization where the interactive nodes have recursive interactive relationships.

I chose to avoid recursion by only making shared dependencies be marked by hovering over direct dependencies of the binary. Though a possible solution to imitate recursion could be to add all interactive nodes before all visibly reactive nodes - both could be placed near the same visual position.

Interaction happens by hovering over parts of the visualization. You both get a pop-up describing the node and visual cues. The binarys node is placed in the center. Each edge shooting out from it points at a direct dependency, and when hovering over these, all their shared dependencies with other direct dependencies are marked.

Future work includes filtering build/runtime dependencies, and making the visualizations mobile-compatible. We'd love feedback if you have other thoughts on how the visualizations could become more useful. Write to us at teamATroburDOTcoop.

The Robur team