gbuild: To boldly go where no build system has gone before ...

Posted by Bjoern Michaelsen on 26 July 2011

gbuild: To boldly go where no build system has gone before ...

This is the first post in a short series of blog posts about the new GNU make based build system that will soon be integrated into the DEV300 codeline. It is covering the memory usage of the new GNU make build system when using full dependencies.

Since the first announcement of plans to replace the old build system with a new solution, one of the focus points was the correct handling of dependencies. To handle this problem correctly, multiple approaches have been tried and one was to have all dependencies in one process because recursive make is considered harmful. As the gnumake2 cws is finally approaching the final status in the lifecycle of a feature branch (being "ready for QA" until "integrated"), it is time to address one of the main concerns reported by community members about it: memory usage.

The very idea of having all information about an build in one process -- including all dependencies -- might sound obnoxious and megalomaniac at first. But in general, memory got cheap in recent years and it was the time to at least consider this option.

By now, we can say it was well worth it: gnumake2 is now capable of building eight modules (framework, sfx2, svl, svtools, sw, toolkit, tools, xmloff) in one process, and we have a solid base to approximate the memory usage of a build that contains all dependencies in one process.

First we have to find a reasonable metric of "dependency-intensity" of a part of the build. Then we can try to relate that metric to the memory usage measured on migrated modules to extrapolate to a full build.

A simple metric is the number on #include-statements in hxx- and cxx-files. (That might be a very naïve assumption, but I have spent too much lifetime in the physics department of a university to be scared of a spherical cow.)

To measure the memory usage two methods have been used:

  1. Adding $(info finished parsing.) $(shell sleep 60) to the end of the main makefile allowed me to measure the heap size of a real make process with pmap -d $(ps -a|grep make|cut -f1 -d\ )|egrep -o writeable/private:.[0-9]+K|cut -f 2 -d\ on Linux 64-Bit.
  2. valgrinds massif provided a simple, but synthetic way to do the same (using useful-heap as a measure).
   include statements pmap (KiB)  massif (KiB)
 no module  0  632 407
 tools  964  1184 1115
 svl  1652  1660 1645
 toolkit  2276  3524 3210
 svtools  3768  5548 5179
 framework  6049  5188 4812
 sfx2  6065  6196 5961
 xmloff  6496  6860 6582
 framework, sfx2  12514  10276 9276
 sw  24087  29340 25943
 all migrated but sw  27670  23124 21145
 all migrated but xmloff, sfx2  38796  40088 35550
 all migrated  51757  50812 45129


Using a simple linear regression over the data, Calcs LINEST function tells me that one can assume a heap usage of 1010±40 bytes per include for the pmap data and 890±35 bytes for the massif data. This plot show no obvious systematic error in the assumption of the model:

gbuild scaling chart


The last two data points are the predictions of the model for a full build with and without binfilter:

  • without binfilter: 170-190 MiB (pmap), 150-170 MiB (massif)
  • with binfilter: 190-210 MiB (pmap), 170-180 MiB (massif)
So it can be assumed that a full build with all dependencies can be handled decently even by slower and smaller hardware as even common ARM machines have enough RAM to handle this.

Finishing note: gnumake2 will be integrated soon, so if you are using actively developing on the DEV300 codeline, it is advisable to check out the basics of it. A good starting point is the talk "Rebooting the Build System" [ ODP PDF video ] I gave at the OOoCon2010. For more detailed information see the Build Environment Effort section at the OOo wiki.

I hope to keep the posts about the build system coming in the next days. Next up: How to migrate a module to gbuild.


P.S.: In case an interested GNU make hacker comes across this post, here is some output from make -p for the 8 migrated modules:

2385 variable set hash-tables
# files hash-table stats:
# Load=21210/32768=65%, Rehash=5, Collisions=605939/1022290=59%

# # of strings in strcache: 32753 / lookups = 991181 / hits = 958428
# # of strcache buffers: 284 (* 8176 B/buffer = 2321984 B)
# strcache used: total = 2313570 (5196) / max = 8176 / min = 8170 / avg = 8175
# strcache free: total = 238 (2980) / max = 6 / min = 0 / avg = 0

# strcache hash-table stats:
# Load=32869/65536=50%, Rehash=3, Collisions=545530/1044947=52%

(This is a very raw mirror of the original blog post made to 16 Nov 2010. As per "... You grant Sun and all other users of the Website an irrevocable, worldwide, royalty-free, nonexclusive license to use, reproduce, modify, distribute, transmit, display, perform, adapt, resell and publish such Content (including in digital form) ..." )

This was originally published at 2011-07-26 10:03:00/2011-07-26 08:03:08 on livejournal.