autopkgtests for adults

"That's not a knife -- THAT's a knife."

I recently worked a bit to see this line showing up in my favorite editor:

ubtree0t-junit-subsequentcheck PASS

LibreOffice has multiple sets of testsuites and during the build of the package we run them all (although not yet on all platforms). However, LibreOffice depends on ~1/3 of main -- so there are a lot of things that might break LibreOffice. A lot of things just break at build time and not at run time and thus prevent the such a broken package to enter the archive in the first place as we run the tests during the build already. Thats as: unless the breakage is caused by an update of a dependency of LibreOffice, therefore making the LibreOffice package in the archive FTBFS (or at least broken) in a sneaky way. Thats whats happened for the e.g. the libjpeg, boost, kdelibs examples above.

But lets keep those aside for now and concentrate on the runtime issues. Running the tests at build-time is a good early-warning already and prevents some serious breakage to enter the archive. On the other hand, these tests do not run against LibreOffice as we install it in the system from packages -- it runs them against a installation set aside in the build tree. While I can not come up with an immediate example, where LibreOffice was broken when installed from the packages in a way that would have been detected by tests but missed when run against the in-tree installation, it would still be good to have the additional confidence that:

LibreOffice passes the tests as installed on the system
LibreOffice is not broken at runtime by some update of a dependency

In short: Its highly desirable to test that LibreOffice does still run and work as the ground below it keeps moving -- this is even more important when Ubuntu is considering to move towards a more rolling way of releasing. And we have a means to do that: Autopkgtests.

So, what was needed to get this working for LibreOffice?

First, some parts of the testsuites are quite large and -- as we run the tests during the build anyway -- are already build during the build. Therefore it made sense to package these, which was done very early in the cycle (actually: during UDS).

Second, we would need to get LibreOffice to run the tests without trying to build the product. That originally wasnt as easy as it may seem. For one, LibreOffice build system was reasonably expecting that you need a product to test it and therefore would have dependencies on the product to be build. In addition, when I started considering this, we still had a lot of the old build system around -- which was a pain to bend to your will. Luckily, these times are over. So, by now a patch changing some ~15 lines get us what we want.

Third, we need a config_host.mk (the output of ./configure), so that we can run the LibreOffice build. And for that, we unfortunately need the build dependencies (which are generated) of LibreOffice -- otherwise we would not really test what we did build. But for a missing feature of autopkgtests, we can not reuse the existing dependencies, but have to do manual double bookkeeping there. Im not thrilled by the prospect of hunting false positives there. Some possible ways out would be:

to package the config_host.mk file into the package containing the other testsuite helpers, but that would make that package architecture dependendant
or to not really specify the dependencies at all and pragmatically and greedily request the restrictions needs-root and breaks-testbed and then -- as we are root now -- run this before starting the tests:
```
apt-get build-dep -y libreoffice
```

Finally, we should be able to run:

apt-get build-dep libreoffice
apt-get install libreoffice-subsequentcheckbase
apt-get source libreoffice
cd libreoffice-*
./debian/tests/junit-subsequentcheck

and this should run the tests locally and headless -- and indeed it does and the tests happily finish and report success. Great, lets quickly check if it also runs in the 'official' VM with:

run-adt-test

Nope, and this is why I choose the Crocodile Dundee quote below the title for this post: The VM fails before it even starts the tests -- it does not even have enough discspace to copy in the LibreOffice source package. This needs to be fixed on the side of the image, there is nothing on the test side that could fix this. But to test if LibreOffice would finish if only the image could handle it, I began cannibalizing, removing one after another the directories of the icon-themes, translations and external sources from the package, each time getting a bit further: from failing to start to failing when installing the 501 additional packages and so on. With this hollowed out package, I could verify: yes, the autopkgtest would pass in the image, if only it had enough discspace.

Finally, once this is in the archive (or ppa) you will also be able to run:

apt-get build-dep libreoffice
apt-get install libreoffice-subsequentcheckbase
apt-get source libreoffice
cd libreoffice-*
libreoffice '--accept=pipe,name=blickenlights;urp'&
./debian/tests/junit-subsequentcheck 'connect:pipe,name=blinkenlights' 1

This will connect to the LibreOffice you started in the second-to-last step (which is not headless, but running in your session) and run the tests against it. The "1" tells it not to use parallelization, but just run one suite at a time, as otherwise you have a very good chance to lock/hang your own session by compiz (or the dash or other components) being mightly confused by all the windows flashing up and closing in fast progression. With "1" you might still get some test failures (mostly from the a11y integration) -- but at least your session will survive:

ZO RELAXEN UND WATSCHEN DER BLINKENLICHTEN.

Addendum:

Preparing an adt-image with:

./bin/prepare-testbed -r raring amd64 -S12GB

seems to solve the issue. The "df -h" at the start of the test reports some 3GB of free space (with 2.6GB being needed still to create a rw-copy of the source tree after that point). So 12GB is likely the size the images on Jenkins roughly currently need (plus maybe another 1GB of wiggle room).

Originally published on 2013-03-19 16:54:22 on wordpress.

Categories: #libreoffice #ubuntu