make it work with bbfreeze #585

New issue

Closed

opened 2009-01-15 03:00:05 +00:00 by zooko · 29 comments

zooko commented

2009-01-15 03:00:05 +00:00

bbfreeze -- http://pypi.python.org/pypi/bbfreeze -- is an alternative to py2exe and (in the near future) to py2app. It is very easy to use:

easy_install bbfreeze
python ./setup.py bdist_bbfreeze

Except that it currently doesn't work for Tahoe on Windows -- something about our (very tricky and complicated) bin/tahoe hack is outwitting bbfreeze maybe?

bbfreeze -- <http://pypi.python.org/pypi/bbfreeze> -- is an alternative to py2exe and (in the near future) to py2app. It is very easy to use: ``` easy_install bbfreeze python ./setup.py bdist_bbfreeze ``` Except that it currently doesn't work for Tahoe on Windows -- something about our (very tricky and complicated) `bin/tahoe` hack is outwitting bbfreeze maybe?

zooko added the

labels 2009-01-15 03:00:05 +00:00

zooko added this to the undecided milestone 2009-01-15 03:00:05 +00:00

zooko commented

2009-04-07 03:02:38 +00:00

Author

Unassigning cgalvan since he hasn't been active in a couple of months.

daira added

t/enhancement

and removed

t/defect

labels 2009-11-08 01:08:12 +00:00

mids commented

2010-02-27 19:29:37 +00:00

Owner

This works for me on 1.6.0:

python setup.py build
python setup.py install
cd windows
bb-freeze tahoe.py

tahoe.exe is created in dist directory and runs okay.

This works for me on 1.6.0: ``` python setup.py build python setup.py install cd windows bb-freeze tahoe.py ``` tahoe.exe is created in dist directory and runs okay.

daira commented

2010-03-02 00:57:05 +00:00

For me mids' recipe does produce an executable, but it references an .egg file that does not contain any resources, such as .xhtml files. This results in exceptions when trying to view the WUI of a node run using this executable.

For me mids' recipe does produce an executable, but it references an `.egg` file that does not contain any resources, such as `.xhtml` files. This results in exceptions when trying to view the WUI of a node run using this executable.

daira commented

2010-03-02 01:01:29 +00:00

The exception when viewing the WUI was:

<type 'exceptions.WindowsError'>: [Error 2] The system cannot find the file specified:
'd:\\tahoe\\1.6.1\\tahoe-lafs\\windows\\dist\\allmydata_tahoe-1.6.0_r4256-py2.6.egg\\allmydata\\web\\directory.xhtml'

(That file indeed does not exist, although the directory does and contains .pyc files.)

zooko wrote on IRC:

[.egg]The is supposed to include [files]resource because (a) we said "include_package_data=True" as an argument to setup(), (b) that .xhtml file is registered under darcs revision control and (c) the setuptools_darcs plugin is installed.

The exception when viewing the WUI was: ``` <type 'exceptions.WindowsError'>: [Error 2] The system cannot find the file specified: 'd:\\tahoe\\1.6.1\\tahoe-lafs\\windows\\dist\\allmydata_tahoe-1.6.0_r4256-py2.6.egg\\allmydata\\web\\directory.xhtml' ``` (That file indeed does not exist, although the directory does and contains `.pyc` files.) zooko wrote on IRC: > [`.egg`]The is supposed to include [files]resource because (a) we said "include_package_data=True" as an argument to setup(), (b) that `.xhtml` file is registered under darcs revision control and (c) the setuptools_darcs plugin is installed.

zooko commented

2010-06-21 21:02:37 +00:00

Author

Replying to davidsarah:

For me mids' recipe does produce an executable, but it references an .egg file that does not contain any resources, such as .xhtml files. This results in exceptions when trying to view the WUI of a node run using this executable.

Hey! This was ticket #1033, which has been fixed. Maybe now mids's recipe will work! :-)

Replying to [davidsarah](/tahoe-lafs/trac/issues/585#issuecomment-369828): > For me mids' recipe does produce an executable, but it references an `.egg` file that does not contain any resources, such as `.xhtml` files. This results in exceptions when trying to view the WUI of a node run using this executable. Hey! This was ticket #1033, which has been fixed. Maybe now mids's recipe will work! :-)

zooko modified the milestone from undecided to soon

2010-06-21 21:02:37 +00:00

daira commented

2011-01-03 07:01:03 +00:00

Replying to [zooko]comment:7:

Replying to davidsarah:

For me mids' recipe does produce an executable, but it references an .egg file that does not contain any resources, such as .xhtml files. This results in exceptions when trying to view the WUI of a node run using this executable.

Hey! This was ticket #1033, which has been fixed. Maybe now mids's recipe will work! :-)

If it did before, it won't now, because the windows directory has been removed by #1282.

After much trial and error, I found the following recipe instead. First apply attachment:bb-freeze.darcs.patch to a checkout of trunk. Then from a cygwin prompt:

make clean
python setup.py build
sed 's/\\r//' <src/allmydata/_version.py >_version.new
mv -f _version.new src/allmydata/_version.py
export PYTHONPATH=support/Lib/site-packages
bb-freeze windows/tahoe.py

(cygwin is not essential, but we need sed to convert the CRLF line endings in _version.py, otherwise the compiler invoked by bb-freeze will fail to compile it. I don't know why it is so picky.)

The resulting dist\tahoe.exe has a bug that causes its argv array to be shifted, dropping sys.argv[0]. You can work around this by giving 'tahoe' as a dummy first argument.

I haven't tested it much, but it can run a functioning gateway node to the pubgrid, and list the public directory using the CLI. Unicode arguments and output even work!

Replying to [zooko]comment:7: > Replying to [davidsarah](/tahoe-lafs/trac/issues/585#issuecomment-369828): > > For me mids' recipe does produce an executable, but it references an `.egg` file that does not contain any resources, such as `.xhtml` files. This results in exceptions when trying to view the WUI of a node run using this executable. > > Hey! This was ticket #1033, which has been fixed. Maybe now mids's recipe will work! :-) If it did before, it won't now, because the windows directory has been removed by #1282. After much trial and error, I found the following recipe instead. First apply [attachment:bb-freeze.darcs.patch](/tahoe-lafs/trac/attachments/000078ac-a32c-5cfe-e764-98bee4072501) to a checkout of trunk. Then from a cygwin prompt: ``` make clean python setup.py build sed 's/\\r//' <src/allmydata/_version.py >_version.new mv -f _version.new src/allmydata/_version.py export PYTHONPATH=support/Lib/site-packages bb-freeze windows/tahoe.py ``` (cygwin is not essential, but we need sed to convert the CRLF line endings in _version.py, otherwise the compiler invoked by bb-freeze will fail to compile it. I don't know why it is so picky.) The resulting `dist\tahoe.exe` has a bug that causes its argv array to be shifted, dropping `sys.argv[0]`. You can work around this by giving 'tahoe' as a dummy first argument. I haven't tested it much, but it can run a functioning gateway node to the pubgrid, and list the public directory using the CLI. Unicode arguments and output even work!

daira commented

2011-01-03 07:01:50 +00:00

Attachment bb-freeze.darcs.patch (13398 bytes) added

Various hacks to get bb-freeze to mostly work. Do not apply to trunk. refs #585

**Attachment** bb-freeze.darcs.patch (13398 bytes) added Various hacks to get bb-freeze to mostly work. Do not apply to trunk. refs #585

bb-freeze.darcs.patch

13 KiB

daira commented

2011-01-03 07:14:24 +00:00

Attachment bb-freeze.darcs.2.patch (13928 bytes) added

Various hacks to get bb-freeze to mostly work. Definitely do not apply to trunk. This version of the patch includes the workaround of adding as the dummy 'tahoe' argument so you don't need to. refs #585

**Attachment** bb-freeze.darcs.2.patch (13928 bytes) added Various hacks to get bb-freeze to mostly work. Definitely do not apply to trunk. This version of the patch includes the workaround of adding as the dummy 'tahoe' argument so you don't need to. refs #585

bb-freeze.darcs.2.patch

14 KiB

daira commented

2011-01-03 07:20:45 +00:00

Replying to [davidsarah]comment:8:

I haven't tested it much, but it can run a functioning gateway node to the pubgrid, and list the public directory using the CLI.

And the WUI works without the resources problem in comment:369828, I meant to say.

Replying to [davidsarah]comment:8: > I haven't tested it much, but it can run a functioning gateway node to the pubgrid, and list the public directory using the CLI. And the WUI works without the resources problem in [comment:369828](/tahoe-lafs/trac/issues/585#issuecomment-369828), I meant to say.

daira commented

2011-01-04 12:03:24 +00:00

attachment:bb-freeze.darcs.3.patch (which depends on the patch for #1287) has now been updated to solve this problem in a way that doesn't interfere with the behaviour of the usual bin/tahoe.

This attachment also includes a patch to the bundled darcsver, to make it write _version.py with LF line endings, by changing "wt+" to "wb+" here. This matches all of the other Python source files, so should not cause any problems. (We should make a darcsver release, this patch to the bundled version is just for testing.)

windows/tahoe.py has been moved to static/tahoe.py because it is not specific to Windows. It should probably work for any [http://www.freehackers.org/Packaging_a_python_program static Python executable packager], although I haven't tested any others.

I've changed the patch to only omit the setuptools version and path from the output of get_package_versions_and_locations() when running the bb-frozen executable (which does not actually have setuptools as a runtime dependency). The problem that this is addressing is that if the executable tries to import setuptools, it can fail to find the original site module and exit at that point.

I also refactored a bunch of stuff in __init__.py and _auto_deps.py. get_package_versions_and_locations() is now in _auto_deps.py because it needs to be updated whenever install_requires is. It suppresses deprecation warnings while importing modules, so that the bb-frozen executable doesn't print these warnings. Finally, we now correctly check that the versions of imported modules actually satisfy the requirements (except for pyasn1, where we don't know the version number). Previously, require_auto_deps() only checked that the distributions that pkg_resources ''attempted'' to put on the sys.path satisfy the requirements that it was told to satisfy. That is not sufficient due to #1246, #1258 and possibly other bugs. I needed to simplify the pycrypto requirement to >= 2.3, since the new check doesn't support disjunctive requirements.

The recipe is now:

make clean
python setup.py build
export PYTHONPATH=support/Lib/site-packages
bb-freeze static/tahoe.py

(This should work on a Windows command prompt, modulo availability of 'make', and set instead of export.)

We need to be able to test the resulting executable. I will file another ticket for that.

I worked out why the arguments are off-by-one -- it is because the code to retrieve Unicode arguments in source:src/allmydata/windows/fixups.py depends on the script having been invoked via the Python interpreter, so that `unicode_argv[0]` (the python executable) needs to be skipped. The executable produced by bbfreeze, OTOH, is not invoked via the Python interpreter. This problem wouldn't have occurred before the fix to #1074 in Tahoe v1.8.0, which explains why it worked for mids. [attachment:bb-freeze.darcs.3.patch](/tahoe-lafs/trac/attachments/000078ac-a32c-5cfe-e764-1bbbfe2cb0a2) (which depends on the patch for #1287) has now been updated to solve this problem in a way that doesn't interfere with the behaviour of the usual `bin/tahoe`. This attachment also includes a patch to the bundled darcsver, to make it write `_version.py` with LF line endings, by changing "wt+" to "wb+" [here](http://tahoe-lafs.org/trac/darcsver/browser/trunk/darcsver/darcsvermodule.py#L50). This matches all of the other Python source files, so should not cause any problems. (We should make a darcsver release, this patch to the bundled version is just for testing.) `windows/tahoe.py` has been moved to `static/tahoe.py` because it is not specific to Windows. It should probably work for any [http://www.freehackers.org/Packaging_a_python_program static Python executable packager], although I haven't tested any others. I've changed the patch to only omit the setuptools version and path from the output of `get_package_versions_and_locations()` when running the bb-frozen executable (which does not actually have setuptools as a runtime dependency). The problem that this is addressing is that if the executable tries to import `setuptools`, it can fail to find the original `site` module and exit at that point. I also refactored a bunch of stuff in `__init__.py` and `_auto_deps.py`. `get_package_versions_and_locations()` is now in `_auto_deps.py` because it needs to be updated whenever `install_requires` is. It suppresses deprecation warnings while importing modules, so that the bb-frozen executable doesn't print these warnings. Finally, we now correctly check that the versions of imported modules actually satisfy the requirements (except for pyasn1, where we don't know the version number). Previously, `require_auto_deps()` only checked that the distributions that `pkg_resources` ''attempted'' to put on the `sys.path` satisfy the requirements that it was told to satisfy. That is not sufficient due to #1246, #1258 and possibly other bugs. I needed to simplify the pycrypto requirement to `>= 2.3`, since the new check doesn't support disjunctive requirements. The recipe is now: ``` make clean python setup.py build export PYTHONPATH=support/Lib/site-packages bb-freeze static/tahoe.py ``` (This should work on a Windows command prompt, modulo availability of 'make', and set instead of export.) We need to be able to test the resulting executable. I will file another ticket for that.

daira commented

2011-01-04 12:05:58 +00:00

Attachment bb-freeze.darcs.3.patch (41381 bytes) added

Make bbfreeze work. This also refactors allmydata/init.py and _auto_deps.py and implements more robust checking of dependency requirements. It includes a change to the bundled darcsvar, to write the _version.py file with LF line endings.

**Attachment** bb-freeze.darcs.3.patch (41381 bytes) added Make bbfreeze work. This also refactors allmydata/*init*.py and _auto_deps.py and implements more robust checking of dependency requirements. It includes a change to the bundled darcsvar, to write the _version.py file with LF line endings.

bb-freeze.darcs.3.patch

40 KiB

daira modified the milestone from soon to 1.9.0

2011-01-04 12:08:58 +00:00

daira commented

2011-01-07 10:41:19 +00:00

Attachment static-tahoe.darcs.patch (12074 bytes) added

static/tahoe.py: import mock (for 'tahoe debug trial') and xml.tree.minidom (to avoid noise from twisted plugins). refs #585

**Attachment** static-tahoe.darcs.patch (12074 bytes) added static/tahoe.py: import mock (for 'tahoe debug trial') and xml.tree.minidom (to avoid noise from twisted plugins). refs #585

static-tahoe.darcs.patch

12 KiB

daira commented

2011-01-07 10:44:34 +00:00

With attachment:bb-freeze.darcs.3.patch + attachment:static-tahoe.darcs.patch + attachment🎫1296:tahoe-debug-trial.darcs.patch , we pass everything except for skipping a few runner tests (which I will fix tomorrow).

With [attachment:bb-freeze.darcs.3.patch](/tahoe-lafs/trac/attachments/000078ac-a32c-5cfe-e764-1bbbfe2cb0a2) + [attachment:static-tahoe.darcs.patch](/tahoe-lafs/trac/attachments/000078ac-a32c-5cfe-e764-84db4bea70c7) + attachment:ticket:1296:tahoe-debug-trial.darcs.patch , we pass everything except for skipping a few runner tests (which I will fix tomorrow).

daira commented

2011-01-07 10:46:30 +00:00

Use dist\tahoe debug trial to test.

Use `dist\tahoe debug trial` to test.

daira commented

2011-01-15 05:14:15 +00:00

In [4938/ticket1306]:

Add static/tahoe.py, to support bbfreeze, py2exe etc. refs #585

In [4938/ticket1306]: ``` Add static/tahoe.py, to support bbfreeze, py2exe etc. refs #585 ```

daira commented

2011-01-15 05:14:16 +00:00

In [4939/ticket1306]:

static/tahoe.py: import mock (for 'tahoe debug trial') and xml.tree.minidom (to avoid noise from twisted plugins). refs #585

In [4939/ticket1306]: ``` static/tahoe.py: import mock (for 'tahoe debug trial') and xml.tree.minidom (to avoid noise from twisted plugins). refs #585 ```

daira commented

2011-01-15 05:14:17 +00:00

In [4942/ticket1306]:

Make bb-freeze work. This also implements more robust checking of dependency requirements, as discussed in ticket:585#comment:10 . refs #585

In [4942/ticket1306]: ``` Make bb-freeze work. This also implements more robust checking of dependency requirements, as discussed in ticket:585#comment:10 . refs #585 ```

daira commented

2011-01-15 05:14:17 +00:00

In [4943/ticket1306]:

Move get_package_versions_and_locations from __init__.py to _auto_deps.py (because it needs to be updated when dependencies change). refs #585

In [4943/ticket1306]: ``` Move get_package_versions_and_locations from __init__.py to _auto_deps.py (because it needs to be updated when dependencies change). refs #585 ```

daira modified the milestone from 1.9.0 to 1.8.2

2011-01-16 21:25:15 +00:00

daira commented

2011-01-19 05:40:51 +00:00

In changeset:93c1909847e9dc44:

bin/tahoe-script.template, src/windows/fixups.py: simplify the method of stripping initial arguments in sys.argv on Windows. This helps with bb-freeze and running tahoe via 'coverage'. Also includes some wording changes and minor refactoring of bin/tahoe-script.template. refs #585, #1303

In changeset:93c1909847e9dc44: ``` bin/tahoe-script.template, src/windows/fixups.py: simplify the method of stripping initial arguments in sys.argv on Windows. This helps with bb-freeze and running tahoe via 'coverage'. Also includes some wording changes and minor refactoring of bin/tahoe-script.template. refs #585, #1303 ```

zooko commented

2011-01-19 06:12:41 +00:00

Author

I reviewed changeset:93c1909847e9dc44 and it looks good to me.

zooko commented

2011-01-19 07:39:26 +00:00

Author

[20110113043040-93fa1-2a5ccfc48e2fe53b3fa5a91e913fbd300ccd98f4]/ticket1306 looks good. (I eyeballed it but didn't actually copy out the text and use diff to confirm that the text didn't get changed while it was getting moved.)

[20110113043040-93fa1-2a5ccfc48e2fe53b3fa5a91e913fbd300ccd98f4]/ticket1306 looks good. (I eyeballed it but didn't actually copy out the text and use `diff` to confirm that the text didn't get changed while it was getting moved.)

zooko commented

2011-01-19 07:42:35 +00:00

Author

[20110107085259-93fa1-386c985e5bebe82119025222c9c39a43aa5b40b4]/ticket1306 looks fine.

zooko commented

2011-01-19 07:48:40 +00:00

Author

Okay I read through [20110113042457-93fa1-b0c23a181f94c19e75407448bdd776c9a065f1a1]/ticket1306. I can't claim to understand it 100%, but I couldn't see any defect or think of any design problem with it. +0.9 :-)

zooko removed their assignment 2011-01-19 07:51:03 +00:00

warner was assigned by zooko

2011-01-19 07:51:03 +00:00

zooko commented

2011-01-20 09:37:36 +00:00

Author

David-Sarah is going to merge these patches into trunk.

warner was unassigned by zooko

2011-01-20 09:37:36 +00:00

daira was assigned by zooko

2011-01-20 09:37:36 +00:00

daira commented

2011-01-21 09:00:03 +00:00

In changeset:8587c63bdc291faf:

Make bb-freeze (and probably other static packaging tools) work. This updates various places where we assumed that the tahoe process was executed via the Python interpreter. It also allows tests to recursively invoke the same tahoe.exe, rather than bin/tahoe. refs #585

In changeset:8587c63bdc291faf: ``` Make bb-freeze (and probably other static packaging tools) work. This updates various places where we assumed that the tahoe process was executed via the Python interpreter. It also allows tests to recursively invoke the same tahoe.exe, rather than bin/tahoe. refs #585 ```

warner commented

2011-01-21 17:22:45 +00:00

I'm guessing that changeset:8587c63bdc291faf is meant to close this ticket. Please re-open if I'm wrong.

warner added the

r/fixed

label 2011-01-21 17:22:45 +00:00

warner closed this issue

2011-01-21 17:22:45 +00:00

warner commented

2011-01-21 23:54:09 +00:00

davidsarah mentioned on IRC about an hour ago that there's one more patch to go in (".tac files import pkg_resources, so that's still a dependency"). Reopening for that.

warner removed the

r/fixed

label 2011-01-21 23:54:09 +00:00

warner reopened this issue

2011-01-21 23:54:09 +00:00

daira commented

2011-01-22 03:42:30 +00:00

In changeset:c8ccb297fa7f499d:

src/allmydata/_auto_deps.py: setuptools is still an install requirement even in frozen builds, because .tac files import pkg_resources. refs #585

In changeset:c8ccb297fa7f499d: ``` src/allmydata/_auto_deps.py: setuptools is still an install requirement even in frozen builds, because .tac files import pkg_resources. refs #585 ```

daira commented

2011-01-22 04:57:17 +00:00

The PYTHONPATH is not right in the recipe in comment:369834. Using a Command Prompt, do:

python setup.py build
set PYTHONPATH=<tahoedir>\support\Lib\site-packages;<tahoedir>\src
bb-freeze static\tahoe.py

where <tahoedir> is the full path to the Tahoe source directory.

(make clean is only necessary if dependencies have changed.)

I'll add this to wiki/AdvancedInstall.

The PYTHONPATH is not right in the recipe in [comment:369834](/tahoe-lafs/trac/issues/585#issuecomment-369834). Using a Command Prompt, do: ``` python setup.py build set PYTHONPATH=<tahoedir>\support\Lib\site-packages;<tahoedir>\src bb-freeze static\tahoe.py ``` where `<tahoedir>` is the full path to the Tahoe source directory. (`make clean` is only necessary if dependencies have changed.) I'll add this to [wiki/AdvancedInstall](wiki/AdvancedInstall).

daira added the

r/fixed

label 2011-01-22 04:57:17 +00:00

daira closed this issue

2011-01-22 04:57:17 +00:00