regression-search.pl


Summary

This script automates the process of searching old builds of Mozilla products to find the date when a particular bug appeared in the build. This allows you to greatly narrow down the number of checkins that may have caused the bug, which is frequently helpful in determining the proper way to fix the bug.

This is accomplished by performing a binary search over all of the available builds in order to narrow the range down as quickly as possible. After each build is downloaded, the script will unpack and launch it for you. At this time you should perform whatever steps are required to test for the presence of the bug you're looking for. For example, if the regression in question is that the Firefox Bookmarks menu doesn't get populated, you would open that menu to see if it listed the bookmarks you were expecting. Once you exit the application, the script will then ask you if the bug was present, and use that information to find the next build to test.

Alternatively, you can specify one of the automated test types. Currently there are three types of automated tests available, reftests, mochitests and perl tests.

Specifying --test-type=reftest to use a reftest enables the script to find a layout regression using the test you give as input. See Reftests for instructions on how to create a reftest manifest.

Specifying --test-type=mochitest to use a mochitest enables the script to run the test you give as input in the MochiTest test suite (http://developer.mozilla.org/en/docs/Mochitest). See MochiTests for more details.

If you specify --test-type=perl, the script will load the given perl file to use as a test. The script will call the function checkForBug, and use it's return value to determine how to proceed. You should return a true value if the bug was detected. This type of test is good for testing things that don't require interaction with the app, but can be found either in the output of running the app or in the app's profile folder. See Perl tests for more details.


Availibility

The script can be downloaded from http://db48x.net/hg/regression-search/archive/v0.6.1_release.tar.bz2, or from the Mercurial repository at http://db48x.net/hg/regression-search/.


Usage

  ./regression-search.pl [flags]


Options

  --branch    -b Which branch you want to test. This isn't exactly a CVS branch,
                 but is instead a string like 'trunk', 'mozilla1.8' or
                 'mozilla1.8.0'. Defaults to 'trunk'.
  --interval  -i Initial size of the time step. The interval specification needs
                 to be one that the Class::Date::Rel perl module can parse.
                 Basically that means a number followed by a suffix. Use M for
                 months, Y for years and d for days. The default is 3 months.
  --product   -p Which product you want to test. Defaults to 'firefox'.
  --profile   -P Selects a profile to use. Defaults to 'regression-search'.
  --selinux   -s Placate SELinux. Sometimes the SELinux attributes on the
                 libraries in the build aren't correct, so firefox won't run.
                 This option fixes them.
  --test-type -t Selects one of four methods of testing the product,
                 'reftest', 'mochitest', 'perl' or 'manual'.
                 The default is 'manual', which means the script will launch
                 the product for you to  determine manually if it exhibits
                 the bug.
  --test-file -f Specifies a file to load when running an automated test.
  --url       -u A url to load when doing a manual test.
  --verbose   -v Prints extra output during the test run. Useful when an
                 automated test needs debugging.


Reftests

A reftest is a test that compares the rendering of two webpages pixel by pixel. If they match, then the test is successful. There is a suite of reftests already in use by the developers to detect regressions when changes are made to the layout engine, but because these tests are not (and probbaly cannot be) exhaustive, regressions still happen. Using this test type you can determine when those regressions happened by designing a new reftest that highlights the problem. Generally you will design two html files that look identical when rendered correctly and render differently otherwise. You will then create a manifest file that tells the reftest framework how to perform the test:

    == a.html b.html

There are a number of other options you can specify in the manifest, see http://lxr.mozilla.org/mozilla/source/layout/tools/reftest/README.txt for the details.

Remember that if you use this test method, then your reftest will be just as useful for preventing regressions in the future if you add it to the suite of tests already available. Don't forget to attach it to the bug report, the developer who fixes the bug you've found will thank you!


MochiTests

Mochitest is an automated testing framework built on top of the MochiKit JavaScript libraries: http://developer.mozilla.org/en/docs/Mochitest. Tests are generally HTML, XML, or XUL files containing JavaScript code that checks for test conditions. See MDC for documentation on writing MochiTest unit tests: http://developer.mozilla.org/en/docs/Writing_MochiTest-based_unit_tests. The HTTP server for these tests runs on port 18888.


Perl tests

The perl test mode requires that you write a perl script to be run on each of the builds that checks to see if the bug is present. This type of test is useful when you want to check that a file exists in the build, or that the profile is created correctly. Even some tests that would still require some user input can be usefully automated in this fashion. Consider a test that checks a file is saved by the app correctly - the user may have to initiate the save, but the script can then check that it matches a reference.

Your script must contain a function called checkForBug which returns true if the bug is present in the current build. When this function is called it is passed the path to regression-search.pl, the name of the product the user is testing, the name of the profile the user is testing the product in, and the url (if any) that the user specified on the command line. Additionally, the current working directory will be that of the build to be tested. If the app needs to be run in order for the bug to detect the bug, your function is responsible for launching it.

If you need to run the app but you don't require that the user interact with it, consider passing --chrome chrome://close/content/close.xul on the app's command line. This simple xul file will immediately exit the app.

Because this script uses require to load your script, your script must return a true value. See the following template:

    sub checkForBug()
    {
        my ($dir, $product, $profile, $url, $verbosity) = @_;
        # …
        return 0;
    }
    1;


Changelog

Version 0.1 — 2007-07-09

Initial release.

Version 0.2 — 2007-07-24

Fixed a bug that caused it to look for builds in the wrong place when the current date moved across the three-month boundary.

Tweaked the code that calculates the next date to check so that if the first build you try exhibits the bug, it doesn't halve the interval. This means that you can search arbitrarily far back into the past if you need to. Before the furthest back you could search for the bug was twice the size of the initial interval, and then the script would exit giving you a meaningless regression range.

Added a Bonsai url to the output, though I haven't taken the time to output correct urls for anything other than the trunk builds.

Added this changelog.

Added the script to an Hg repository. See http://db48x.net/hg/regression-search/

Version 0.3 — 2007-07-25

You can now use a reftest to do an automated search.

Version 0.3.1 — 2007-07-25

Minor change to the docs.

Version 0.3.2 — 2007-07-29

Bug fix. Make sure the tarballs from different products won't get in the way of each other.

Version 0.4 — 2007-07-30

Cache html, so that it doesn't have to retrieve the list of builds each time through.

--url is now only used during a manual test, use --test-file during an automated test.

Added perl-based automated tests.

Added some more documentation for perl tests and reftests.

Version 0.4.1 — 2007-08-01

Fix two small bugs.

Version 0.5 — 2007-08-01

Ted added mochitest automated tests.

Fixed a horrible gaffe - since version 0.4 the script would download the same build each time through.

Tweaked the final output, so that it's more likely to form a complete sentence.

Version 0.5.1 — 2007-08-07

Some very minor changes that make it a little more robust when used from other apps.

A little bit better error handling during reftests.

Version 0.6 — 2007-08-11

The script will now search hourly builds once it has narrowed the regression range down to a single day. Hourly builds are only available for the previous 14 days at present.

The directories that cached builds are stored in are now named differently, as they include the time of the build as well as the date. Builds you've previously downloaded won't get in the way, but won't be used either.

The final regression range will now always use valid dates, rather than the human readable strings 'now' and 'the beginning of time'.

Version 0.6.1 — 2007-09-01

Added a verbose switch that prints extra output during the test run.

ftp.mozilla.org now holds fewer builds, so Ted changed it to compensate.


Bugs / To Do

Currently it assumes you want tarballs, but this is incorrect on windows and mac. The script should choose the correct file based on the platform. Of course, that also means the script needs to know what to do with zip files and dmg files, which isn't something I want to bother with right now. I'll need it by 1.0, however.

I need to write some fairly general perl tests, such as one to look for the appearance of a regression or a crash.

Because builds take up a lot of space, hourly builds are only available two weeks into the past. As such, regressions introduced earlier than this will have larger regression ranges. In order to combat this, I'd like to extend the script to build the app itself, doing a binary search through the checkins on a single day. I'm planning this for version 1.0

If it can't find a build for the right day, it should be clever and look for the day before or after, as long as that doesn't cause the regression range to become invalid.


Copyright

Copyright © 2007 Daniel Brooks

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

 regression-search.pl