"make quicktest" could be quicker and less noisy #591

New issue

Closed

opened 2009-01-23 02:55:02 +00:00 by warner · 17 comments

warner commented

2009-01-23 02:55:02 +00:00

In the recent setup.py refactoring (specifically the part that uses a setuptools plugin to invoke trial, instead of custom code in setup.py), the "quicktest" target is starting to look more like the not-so-quick "test" target.

On my workstation, the tahoe-1.2.0 release could do a "make quicktest TEST=allmydata.test.test_base62" in 1.0 seconds, and emits 17 lines (only one of which is not coming from trial, the line which tells you which 'trial' command line you would need to run the tests yourself). The underlying trial process reports a runtime of about 0.1 seconds. The corresponding "make test TEST=allmydata.test.test_base62" process takes 3.5s and emits 159 lines (so 142 lines of fluff). If I wanted more control over the trial command, I could cut/edit/paste the command line it presented.

In current trunk, the same quicktest command takes 2.5s and emits 99 lines (82 lines of fluff). The non-quick "test" command takes 3.6s and emits 177 lines (160 lines of fluff). In addition, the trial command line is no longer available, and I'm not even sure it is easy to figure out what the user might be able to run to invoke the same tests without the setup.py framework.

It's disappointing that the "quicktest" target is becoming just as slow and just as noisy as the "test" target. I don't mind the extra couple of seconds.. if it were 5 seconds, that would cut into my edit-test-repeat loop (by giving me time to get distracted while waiting for a single-test development cycle to complete), but 2.5s is not quite enough to hit that threshold. However, the extra 82 lines of fluff is annoying: when I tell emacs to re-run the unit test for whatever feature I'm working on right now, the test results no longer fit in the window that it creates.

So, it's low-priority, but I'd appreciate it if our ongoing setup.py work would consider brevity and speed of testing to be a priority, in particular the use-case of running a single test case (in an edit-test-repeat cycle).

I added the "quicktest" target to the Makefile long ago, just after Zooko added some dependencies to make sure that "make test" would cause a "make build" to occur first. This dependency reduces the surprise factor for a new developer who grabs a source tree and types "make test" without first typing "make". I don't mind that sort of handholding, but I wanted a way to tell the system that I know what I'm doing and I really do want to run just the tests and not do anything else. The "quicktest" target was added to run trial without the extra fluff. In addition, I don't want to be distracted by a lot of extra output. In the recent setup.py refactoring (specifically the part that uses a setuptools plugin to invoke trial, instead of custom code in setup.py), the "quicktest" target is starting to look more like the not-so-quick "test" target. On my workstation, the tahoe-1.2.0 release could do a "make quicktest TEST=allmydata.test.test_base62" in 1.0 seconds, and emits 17 lines (only one of which is not coming from trial, the line which tells you which 'trial' command line you would need to run the tests yourself). The underlying trial process reports a runtime of about 0.1 seconds. The corresponding "make test TEST=allmydata.test.test_base62" process takes 3.5s and emits 159 lines (so 142 lines of fluff). If I wanted more control over the trial command, I could cut/edit/paste the command line it presented. In current trunk, the same quicktest command takes 2.5s and emits 99 lines (82 lines of fluff). The non-quick "test" command takes 3.6s and emits 177 lines (160 lines of fluff). In addition, the trial command line is no longer available, and I'm not even sure it is easy to figure out what the user might be able to run to invoke the same tests without the setup.py framework. It's disappointing that the "quicktest" target is becoming just as slow and just as noisy as the "test" target. I don't mind the extra couple of seconds.. if it were 5 seconds, that would cut into my edit-test-repeat loop (by giving me time to get distracted while waiting for a single-test development cycle to complete), but 2.5s is not quite enough to hit that threshold. However, the extra 82 lines of fluff is annoying: when I tell emacs to re-run the unit test for whatever feature I'm working on right now, the test results no longer fit in the window that it creates. So, it's low-priority, but I'd appreciate it if our ongoing setup.py work would consider brevity and speed of testing to be a priority, in particular the use-case of running a single test case (in an edit-test-repeat cycle).

warner added the

labels 2009-01-23 02:55:02 +00:00

warner added this to the undecided milestone 2009-01-23 02:55:02 +00:00

zooko commented

2009-01-23 23:03:42 +00:00

fixed by changeset:2cf9505d5f986cfb

zooko added the

r/fixed

label 2009-01-23 23:03:42 +00:00

zooko closed this issue

2009-01-23 23:03:42 +00:00

warner commented

2009-01-23 23:10:19 +00:00

Author

In addition, it looks like the new code (which just runs bare 'trial') won't respect the PYTHON variable (which I think we used to respect, to run the tests under a different version of python than the default). I'd have to look at the old pre-setuptools_trial implementation, but I think we either parameterized $TRIAL or $PYTHON to allow this sort of control.

Unfortunately, no. 'make quicktest' now fails completely, because PYTHONPATH is not set up to locate the tahoe source code. The old code used setup.py to find out what PYTHONPATH ought to be (i.e. where the tahoe code and its dependencies went), and then spawned trial with that environment. The new Makefile could only work if Tahoe were already installed (like in /usr/lib), making it less-than-useful for development work. In addition, it looks like the new code (which just runs bare 'trial') won't respect the PYTHON variable (which I think we used to respect, to run the tests under a different version of python than the default). I'd have to look at the old pre-setuptools_trial implementation, but I think we either parameterized $TRIAL or $PYTHON to allow this sort of control.

warner added

p/major

and removed

p/minor

r/fixed

labels 2009-01-23 23:10:19 +00:00

warner reopened this issue

2009-01-23 23:10:19 +00:00

warner commented

2009-01-30 02:38:22 +00:00

Author

Once upon a time, the Makefile had code to figure out what that PYTHONPATH=
setting ought to be (the tahoe code gets placed somewhere under
CWD/support/lib, but it depends upon the version of Python in use, and it's
different on windows). That code was gross, because CWD might have spaces or
slashes, and the path names had to be escaped when they were passed all over
the place. So we moved that code into setup.py and added the 'setup.py trial'
target to set up PYTHONPATH and exec trial directly. Then that code moved
into the setuptools_trial plugin, but in the process it lost the
PYTHONPATH-setting code. So now we're back to considering adding code into
the Makefile again. Ick.

If I'm actually the only person who cares about running a single test
quickly, I could write a personal ~/bin/quicktest-tahoe which figures out the
PYTHONPATH= and spawns trial, and just use that. But I think anybody
developing new tahoe code will (or should) want this feature, so I'd really
like for it to be a part of the Tahoe source tree.

we had to change the Makefile (in changeset:a6eb434b57b0577d) to use "setup.py test" for both "test" and "quicktest", because otherwise the egg-info version gets mangled and test_runner fails. As a result, a tiny test (make quicktest TEST=allmydata.test.test_base62.T.test_ende_0x00) takes 4.0s to run and produces 91 lines of output, whereas the underlying trial process (which can be invoked with `PYTHONPATH=something trial allmydata.test.test_base62.T.test_ende_0x00`) runs in 750ms and emits 8 lines of output. Once upon a time, the Makefile had code to figure out what that PYTHONPATH= setting ought to be (the tahoe code gets placed somewhere under CWD/support/lib, but it depends upon the version of Python in use, and it's different on windows). That code was gross, because CWD might have spaces or slashes, and the path names had to be escaped when they were passed all over the place. So we moved that code into setup.py and added the 'setup.py trial' target to set up PYTHONPATH and exec trial directly. Then that code moved into the setuptools_trial plugin, but in the process it lost the PYTHONPATH-setting code. So now we're back to considering adding code into the Makefile again. Ick. If I'm actually the only person who cares about running a single test quickly, I could write a personal ~/bin/quicktest-tahoe which figures out the PYTHONPATH= and spawns trial, and just use that. But I think anybody developing new tahoe code will (or should) want this feature, so I'd really like for it to be a part of the Tahoe source tree.

zooko commented

2009-02-03 23:41:01 +00:00

Assigning to cgalvan. Chris: you're welcome, of course, to assign this to "nobody" to indicate that you don't plan to do it. Especially if you also include some notes about how it might be possible, so that "nobody" can use that knowledge when they get around to doing it. ;-)

warner commented

2009-02-06 05:37:48 +00:00

Author

Incidentally, running a tiny test (make test TEST=allmydata.test.test_base62.T.test_ende_0x00) on my modern Mac laptop takes 24s to run, which makes it really annoying for development purposes. I suspect that it takes longer on my Mac than on my linux box because the linux box has a bunch of dependencies available as .debs . From the output of that 'make test', it looks like it's half-rebuilding several of the dependencies each time.

The corresponding 'make quicktest' takes 12.6s on the Mac.

zooko commented

2009-02-06 06:08:02 +00:00

Please paste the output from the 24s run, including the "half-rebuilding" stuff.

warner commented

2009-02-06 07:49:19 +00:00

Author

Attachment macos-make-test.txt (17064 bytes) added

Mac OS-X 'make test (tinytest)' run, took 26 seconds

**Attachment** macos-make-test.txt (17064 bytes) added Mac OS-X 'make test (tinytest)' run, took 26 seconds

macos-make-test.txt

17 KiB

zooko commented

2009-02-12 04:13:46 +00:00

On my workstation yukyuk, python setup.py test -s allmydata.test.test_repairer.Repairer.test_repair_from_deletion_of_1 takes 8.8s, where PYTHONPATH=src/ trial allmydata.test.test_repairer.Repairer.test_repair_from_deletion_of_1 takes 4.6s. Also the former which uses (setuptools_trial) is very noisy. The fact that I don't need to add support/lib/python2.5/site-packages to my PYTHONPATH shows that I have all the dependencies installed into my system directories.

On my workstation yukyuk, `python setup.py test -s allmydata.test.test_repairer.Repairer.test_repair_from_deletion_of_1` takes 8.8s, where `PYTHONPATH=src/ trial allmydata.test.test_repairer.Repairer.test_repair_from_deletion_of_1` takes 4.6s. Also the former which uses (`setuptools_trial`) is very noisy. The fact that I don't need to add `support/lib/python2.5/site-packages` to my PYTHONPATH shows that I have all the dependencies installed into my system directories.

warner commented

2009-02-13 07:07:02 +00:00

Author

A complaint about the full "test" step, as opposed to the "quicktest" step:

On the edgy buildslave, the 'build' step (python setup.py build) takes 6 minutes, including the time to compile pycryptopp. Then the "test" step (python setup.py test) takes 5 minutes before it even starts running the first test, partially because it compiles pycryptopp a second time.
Here is the log from that "test" step, which also downloads+builds nevow, foolscap, zfec, simplejson, pyopenssl, pyutil, argparse, and zbase32, despite the fact that every single one was built by the previous "build" step.

Interestingly, many of these dependencies appear to be examined twice during the "test" step: the first examination notices that a suitable version is installed and available, but the second examination somehow comes to a different conclusion and decides to download+build the library.
Clearly this is somehow a bug in setuptools.

In general, the amount of time consumed by the test suite is starting to make the buildbot less useful: some of the slower builders take almost an hour before they provide useful unit-test results. If a typical developer machine took that long, the tests would be worthless for development purposes.

A complaint about the full "test" step, as opposed to the "quicktest" step: On the edgy buildslave, the 'build' step (python setup.py build) takes 6 minutes, including the time to compile pycryptopp. Then the "test" step (python setup.py test) takes 5 minutes before it even starts running the first test, partially because it compiles pycryptopp a **second** time. [Here](http://allmydata.org/buildbot/builders/edgy/builds/1997/steps/test-figleaf/logs/stdio) is the log from that "test" step, which also downloads+builds nevow, foolscap, zfec, simplejson, pyopenssl, pyutil, argparse, and zbase32, despite the fact that every single one was built by the previous "build" step. Interestingly, many of these dependencies appear to be examined twice during the "test" step: the first examination notices that a suitable version is installed and available, but the second examination somehow comes to a different conclusion and decides to download+build the library. Clearly this is somehow a bug in setuptools. In general, the amount of time consumed by the test suite is starting to make the buildbot less useful: some of the slower builders take almost an hour before they provide useful unit-test results. If a typical developer machine took that long, the tests would be worthless for development purposes.

zooko commented

2009-02-13 13:57:08 +00:00

I intend to look at this and related tickets after 1.3.0 release.

zooko commented

2009-04-07 17:11:53 +00:00

(http://mail.python.org/pipermail/distutils-sig/2009-April/011332.html)

PJE says:

"""
Install your eggs with --multi-version, and then only the eggs that are required for the running script will be added to sys.path or have their contents opened. (Installing them as zip files rather than directories may also speed this up.)
"""

#530 is about using --multi-version. I'm skeptical that zipped eggs will load faster, but it is worth measuring.

Unassigning cgalvan as he hasn't been active in a while.

(http://mail.python.org/pipermail/distutils-sig/2009-April/011332.html) PJE says: """ Install your eggs with --multi-version, and then only the eggs that are required for the running script will be added to sys.path or have their contents opened. (Installing them as zip files rather than directories may also speed this up.) """ #530 is about using --multi-version. I'm skeptical that zipped eggs will load faster, but it is worth measuring. Unassigning cgalvan as he hasn't been active in a while.

warner commented

2009-04-08 02:00:55 +00:00

Author

I have to say I'm about ready to give up on setuptools_trial .
Rebuilding dependencies takes an extra few seconds out of each
invocation. On my Mac laptop (which I've been using exclusively this
week), it builds all the dependencies in the wrong place (#657), which
takes forever, even with 'quicktest'. Running darcsver on a Mac seems to
take forever (maybe my 'darcs' was built by that version of ghc that
causes really slow fileio or something?). I can't pass through an
argument to run multiple tests at once. It doesn't print out a command
line that I could run to bypass all of the setuptools junk and run
trial by myself. And to manually hack on any of these, I have to unpack
the setuptools_trial source, edit something, repack it into a .tar.gz,
then let the Tahoe setup.py re-extract it. And the setuptools control
flow is not very obvious.. the one or two times I've tried to modify it
to fix these things, I wasn't able to figure out where argv gets passed.

So I'm tempted to throw out setuptools_trial and go back to the setup.py
code that we had before. That, or leave 'setup.py trial' to Zooko and
change the Makefile's 'quicktest' target (which is what I use all the
time) to run code that I've written.

I have to say I'm about ready to give up on setuptools_trial . Rebuilding dependencies takes an extra few seconds out of each invocation. On my Mac laptop (which I've been using exclusively this week), it builds all the dependencies in the wrong place (#657), which takes forever, even with 'quicktest'. Running darcsver on a Mac seems to take forever (maybe my 'darcs' was built by that version of ghc that causes really slow fileio or something?). I can't pass through an argument to run multiple tests at once. It doesn't print out a command line that I *could* run to bypass all of the setuptools junk and run trial by myself. And to manually hack on any of these, I have to unpack the setuptools_trial source, edit something, repack it into a .tar.gz, then let the Tahoe setup.py re-extract it. And the setuptools control flow is not very obvious.. the one or two times I've tried to modify it to fix these things, I wasn't able to figure out where argv gets passed. So I'm tempted to throw out setuptools_trial and go back to the setup.py code that we had before. That, or leave 'setup.py trial' to Zooko and change the Makefile's 'quicktest' target (which is what I use all the time) to run code that I've written.

zooko commented

2009-06-10 14:08:40 +00:00

Brian: are you satisfied using the current make quicktest? We can leave this ticket open because it remains true and it would be useful to improve setuptools_trial, but if it is not actively bothering then I will prioritize other things.

Brian: are you satisfied using the current `make quicktest`? We can leave this ticket open because it remains true and it would be useful to improve `setuptools_trial`, but if it is not actively bothering then I will prioritize other things.

daira commented

2011-07-21 20:28:04 +00:00

A side-effect of #1296 (which added tahoe debug trial, and made make quicktest use it) was that make quicktest is now much faster and quieter. I don't think it can be made significantly faster still without defeating the main goal of that ticket (to have all methods of running the test suite end up using the same sys.path). There is almost no fluff: the output starts

python bin/tahoe debug trial  allmydata
allmydata.test.test_backupdb
  BackupDB
    test_basic ...                                                         [OK]

Brian, are you now happy with the unfluffiness and speed of make quicktest?

A side-effect of #1296 (which added `tahoe debug trial`, and made `make quicktest` use it) was that `make quicktest` is now much faster and quieter. I don't think it can be made significantly faster still without defeating the main goal of that ticket (to have all methods of running the test suite end up using the same `sys.path`). There is almost no fluff: the output starts ``` python bin/tahoe debug trial allmydata allmydata.test.test_backupdb BackupDB test_basic ... [OK] ``` Brian, are you now happy with the unfluffiness and speed of `make quicktest`?

daira commented

2011-07-23 01:49:01 +00:00

There are some criticisms of the speed of the 'build' step in comment:369958 and comment:369961, but they don't really belong in this ticket about 'quicktest'. The one about rebuilding dependencies appears to be #657. Brian, if there is anything in those comments not covered by an existing ticket, please file a new one so that they don't get lost if this ticket is closed.

There are some criticisms of the speed of the 'build' step in [comment:369958](/tahoe-lafs/trac/issues/591#issuecomment-369958) and [comment:369961](/tahoe-lafs/trac/issues/591#issuecomment-369961), but they don't really belong in this ticket about 'quicktest'. The one about rebuilding dependencies appears to be #657. Brian, if there is anything in those comments not covered by an existing ticket, please file a new one so that they don't get lost if this ticket is closed.

daira commented

2013-04-23 05:10:21 +00:00

I think #1296 can be considered to have fixed this ticket.

daira added the

r/fixed

label 2013-04-23 05:10:21 +00:00

daira closed this issue

2013-04-23 05:10:21 +00:00

warner commented

2013-04-23 05:59:06 +00:00

Author

Yeah, I'll agree. There seems to be about 790ms of overhead (on my 2013 Mac Mini, running from SSD), but no extra output fluff, and no apparent attempt to rebuild anything. Good enough for me. Thanks!

Rows
Columns