Debian package: Missing dependency on Debian/etch #745

New issue

Closed

opened 2009-06-22 16:41:34 +00:00 by [4-tea-2] · 15 comments

[4-tea-2] commented

2009-06-22 16:41:34 +00:00

Owner

Starting tahoe after upgrading a Debian/etch system to use the Debian Tahoe package 1.4.1-r3916 fails with this error message:

tahoe@{hostname}:~$ tahoe start
Traceback (most recent call last):
  File "/usr/bin/tahoe", line 2, in <module>
    from allmydata.scripts import runner
  File "/usr/lib/python2.5/site-packages/allmydata/scripts/runner.py", line 9, in <module>
    pkg_resources.require('allmydata-tahoe')
  File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 626, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 524, in resolve
    raise DistributionNotFound(req)  # XXX put more info here
pkg_resources.DistributionNotFound: pysqlite>=2.0.5

Perhaps including python-pysqlite2 as a dependency in debian/control would fix this?

Starting tahoe after upgrading a Debian/etch system to use the Debian Tahoe package 1.4.1-r3916 fails with this error message: ``` tahoe@{hostname}:~$ tahoe start Traceback (most recent call last): File "/usr/bin/tahoe", line 2, in <module> from allmydata.scripts import runner File "/usr/lib/python2.5/site-packages/allmydata/scripts/runner.py", line 9, in <module> pkg_resources.require('allmydata-tahoe') File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 626, in require needed = self.resolve(parse_requirements(requirements)) File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 524, in resolve raise DistributionNotFound(req) # XXX put more info here pkg_resources.DistributionNotFound: pysqlite>=2.0.5 ``` Perhaps including python-pysqlite2 as a dependency in debian/control would fix this?

tahoe-lafs added the

labels 2009-06-22 16:41:34 +00:00

tahoe-lafs added this to the undecided milestone 2009-06-22 16:41:34 +00:00

[4-tea-2] commented

2009-06-22 16:42:23 +00:00

Author

Owner

Sorry, I forgot to mention that installing python-pysqlite2 manually fixed the problem.

warner commented

2009-06-24 04:09:48 +00:00

Hrm, on python2.5, it's not supposed to declare a dependency on pysqlite (since 2.5 comes with sqlite in the standard library).

Zooko?

Hrm, on python2.5, it's not supposed to declare a dependency on pysqlite (since 2.5 comes with sqlite in the standard library). Zooko?

warner added

c/packaging

and removed

c/unknown

labels 2009-06-24 04:09:48 +00:00

warner modified the milestone from undecided to 1.5.0

2009-06-24 04:09:48 +00:00

warner commented

2009-06-24 04:17:03 +00:00

#728 was for adding this change. I grepped the current source code and the pysqlite requirement is guarded by an if sys.version_info < (2,5). Could you make sure that sys.version_info looks correct on your system, and that this conditional is doing the right thing?

#728 was for adding this change. I grepped the current source code and the pysqlite requirement is guarded by an `if sys.version_info < (2,5)`. Could you make sure that sys.version_info looks correct on your system, and that this conditional is doing the right thing?

zooko commented

2009-06-25 15:59:20 +00:00

Assigning to [4-tea-2] to investigate.

warner commented

2009-06-30 18:58:19 +00:00

Incidentally, when Zooko asked me about adding this dependency declaration, I agreed to it only with the condition that it not cause exactly this sort of problem (setuptools has taught me to not trust its dependency calculations: I am constantly seeing it rebuild things which are already installed). If this turns out to be some sort of deep problem, I'll recommend that we remove the declaration.

zooko commented

2009-06-30 19:21:33 +00:00

setuptools installs requirements that are already installed (#657 ("python ./setup.py test" rebuilds packages), #717 (unnecessary rebuild of dependencies when tahoe-deps/ is present)), but I don't think I've seen it install something which wasn't required. My guess is that [4-tea-2] installed a .deb which didn't declare a dependency on sqlite2 (possibly because the .deb is in error? Or because it was a .deb for python 2.6?), and then ran it with Python 2.5.

[4-tea-2]: which .deb of Tahoe-LAFS did you install?

Thanks!

zooko commented

2009-07-02 20:34:31 +00:00

Formatted original report with wiki quoting.

zooko commented

2009-07-02 20:45:28 +00:00

Oh, when I wrote comment:372656, I was thinking that Python >= 2.6 comes with sqlite, but it is actually Python >= 2.5. [4-tea-2], would you please run the following and post the results?

PYTHONPATH=./support/lib/python2.5/site-packages/:$PYTHONPATH python -c 'import pkg_resources;print pkg_resources.require("allmydata-tahoe")'

? For example, here are the results on my machine:

$ PYTHONPATH=./support/lib/python2.4/site-packages/:$PYTHONPATH python -c 'import pkg_resources;print pkg_resources.require("allmydata-tahoe")'
[allmydata-tahoe 1.4.1-r3914 (/home/zooko/playground/allmydata/tahoe/trunk/src), pysqlite 2.5.5 (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site-packages/pysqlite-2.5.5-py2.4-solaris-2.11-i86pc.egg), Nevow 0.9.32 (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site-packages/Nevow-0.9.32-py2.4.egg), foolscap 0.4.1 (/usr/lib/python2.4/site-packages/foolscap-0.4.1-py2.4.egg), Twisted 8.2.0 (/usr/lib/python2.4/site-packages/Twisted-8.2.0-py2.4-solaris-2.11-i86pc.egg), zope.interface 3.5.1 (/usr/lib/python2.4/site-packages/zope.interface-3.5.1-py2.4-solaris-2.11-i86pc.egg), simplejson 2.0.7 (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site-packages/simplejson-2.0.7-py2.4-solaris-2.11-i86pc.egg), zfec 1.4.4 (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site-packages/zfec-1.4.4-py2.4-solaris-2.11-i86pc.egg), pycryptopp 0.5.4 (/usr/lib/python2.4/site-packages/pycryptopp-0.5.4-py2.4-solaris-2.11-i86pc.egg), setuptools 0.6c12dev (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site-packages/setuptools-0.6c12dev.egg), pyOpenSSL 0.8 (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site-packages/pyOpenSSL-0.8-py2.4-solaris-2.11-i86pc.egg), setuptools 0.6c12dev (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site-packages/setuptools-0.6c12dev.egg), pyutil 1.3.26 (/usr/lib/python2.4/site-packages/pyutil-1.3.26-py2.4.egg), argparse 0.8.0 (/usr/lib/python2.4/site-packages/argparse-0.8.0-py2.4.egg), setuptools 0.6c12dev (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site-packages/setuptools-0.6c12dev.egg), zbase32 1.1.0 (/usr/lib/python2.4/site-packages/zbase32-1.1.0-py2.4.egg)]

Then, please look at the _auto_deps.py file which is in the allmydata-tahoe package. For example, on my machine it is:

$ grep -A1 -B1 -i sqlite /home/zooko/playground/allmydata/tahoe/trunk/src/allmydata/_auto_deps.py

# Sqlite comes built into Python >= 2.5, and is provided by the "pysqlite"
# distribution for Python 2.4.
--
if sys.version_info < (2, 5):
    # pysqlite v2.0.5 was shipped in Ubuntu 6.06 LTS "dapper" and Nexenta NCP 1.
    install_requires.append("pysqlite >= 2.0.5")

The _auto_deps.py ought to be the same as the current trunk version: source:_auto_deps.py.

Finally, please look at the requires.txt file in the .egg-info, like this:

$ grep -i sqlite /home/zooko/playground/allmydata/tahoe/trunk/src/allmydata_tahoe.egg-info/requires.txt 
pysqlite >= 2.0.5

This shows that on my machine pysqlite appears in the requires.txt, because the .egg-info was built using Python 2.4. If you built with Python 2.5 then pysqlite should not appear in your .egg-info. You could try rm'rf the whole allmydata_tahoe.egg-info directory and rebuilding.

Thanks!

Oh, when I wrote [comment:372656](/tahoe-lafs/trac/issues/745#issuecomment-372656), I was thinking that Python >= 2.6 comes with sqlite, but it is actually Python >= 2.5. [4-tea-2], would you please run the following and post the results? ``` PYTHONPATH=./support/lib/python2.5/site-packages/:$PYTHONPATH python -c 'import pkg_resources;print pkg_resources.require("allmydata-tahoe")' ``` ? For example, here are the results on my machine: ``` $ PYTHONPATH=./support/lib/python2.4/site-packages/:$PYTHONPATH python -c 'import pkg_resources;print pkg_resources.require("allmydata-tahoe")' [allmydata-tahoe 1.4.1-r3914 (/home/zooko/playground/allmydata/tahoe/trunk/src), pysqlite 2.5.5 (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site-packages/pysqlite-2.5.5-py2.4-solaris-2.11-i86pc.egg), Nevow 0.9.32 (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site-packages/Nevow-0.9.32-py2.4.egg), foolscap 0.4.1 (/usr/lib/python2.4/site-packages/foolscap-0.4.1-py2.4.egg), Twisted 8.2.0 (/usr/lib/python2.4/site-packages/Twisted-8.2.0-py2.4-solaris-2.11-i86pc.egg), zope.interface 3.5.1 (/usr/lib/python2.4/site-packages/zope.interface-3.5.1-py2.4-solaris-2.11-i86pc.egg), simplejson 2.0.7 (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site-packages/simplejson-2.0.7-py2.4-solaris-2.11-i86pc.egg), zfec 1.4.4 (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site-packages/zfec-1.4.4-py2.4-solaris-2.11-i86pc.egg), pycryptopp 0.5.4 (/usr/lib/python2.4/site-packages/pycryptopp-0.5.4-py2.4-solaris-2.11-i86pc.egg), setuptools 0.6c12dev (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site-packages/setuptools-0.6c12dev.egg), pyOpenSSL 0.8 (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site-packages/pyOpenSSL-0.8-py2.4-solaris-2.11-i86pc.egg), setuptools 0.6c12dev (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site-packages/setuptools-0.6c12dev.egg), pyutil 1.3.26 (/usr/lib/python2.4/site-packages/pyutil-1.3.26-py2.4.egg), argparse 0.8.0 (/usr/lib/python2.4/site-packages/argparse-0.8.0-py2.4.egg), setuptools 0.6c12dev (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site-packages/setuptools-0.6c12dev.egg), zbase32 1.1.0 (/usr/lib/python2.4/site-packages/zbase32-1.1.0-py2.4.egg)] ``` Then, please look at the `_auto_deps.py` file which is in the `allmydata-tahoe` package. For example, on my machine it is: ``` $ grep -A1 -B1 -i sqlite /home/zooko/playground/allmydata/tahoe/trunk/src/allmydata/_auto_deps.py # Sqlite comes built into Python >= 2.5, and is provided by the "pysqlite" # distribution for Python 2.4. -- if sys.version_info < (2, 5): # pysqlite v2.0.5 was shipped in Ubuntu 6.06 LTS "dapper" and Nexenta NCP 1. install_requires.append("pysqlite >= 2.0.5") ``` The `_auto_deps.py` ought to be the same as the current trunk version: source:_auto_deps.py. Finally, please look at the `requires.txt` file in the `.egg-info`, like this: ``` $ grep -i sqlite /home/zooko/playground/allmydata/tahoe/trunk/src/allmydata_tahoe.egg-info/requires.txt pysqlite >= 2.0.5 ``` This shows that on my machine `pysqlite` appears in the `requires.txt`, because the .egg-info was built using Python 2.4. If you built with Python 2.5 then `pysqlite` should not appear in your `.egg-info`. You could try rm'rf the whole `allmydata_tahoe.egg-info` directory and rebuilding. Thanks!

zooko commented

2009-07-02 21:38:33 +00:00

Ooh, I get it. [4-tea-2] didn't build the .egg-info himself, it came with the .deb that we built. So, we have to decide whether we want our .deb to depend on python-pysqlite, or whether we want it to not completely support Python 2.4 (or we could even build different .deb's for each Python version). I'm okay with either of those two options. (Not so much with the Python-version-specific debs option.)

I guess that the build machine where the Debian packages are being created is using Python 2.4, so pysqlite is being added to the .egg-info's requires.txt.

Assigning to Brian to tell me what our policy is about .debs and pysqlite.

Ooh, I get it. [4-tea-2] didn't build the `.egg-info` himself, it came with the .deb that we built. So, we have to decide whether we want our .deb to depend on `python-pysqlite`, or whether we want it to not completely support Python 2.4 (or we could even build different .deb's for each Python version). I'm okay with either of those two options. (Not so much with the Python-version-specific debs option.) I guess that the build machine where the Debian packages are being created is using Python 2.4, so `pysqlite` is being added to the `.egg-info`'s `requires.txt`. Assigning to Brian to tell me what our policy is about .debs and pysqlite.

warner commented

2009-07-02 21:53:51 +00:00

On the other hand, the "standard" version of python is well-known per-platform (it's 2.4 in the older ones, 2.5 in the newer ones, and I guess Ubuntu Karmic will probably have 2.6). So we could say that when we build a .deb on a platform with /usr/bin/python=2.4, we declare the dependency upon pysqlite, and when we build it on a py2.5-using platform, we don't.

To implement this, we should update the misc/*/debian/control files to declare the pysqlite dependency on the older platforms, and leave the newer ones alone (not declaring it).

This also means that we should probably stop using the feisty rules for all platforms.. we at least need to identify the release for which python switched to 2.5 and use a newer set of rules for the 2.5-using set.

I'll try to look at this today, but I'm not sure I'll get the time.

Sigh. So, building separate .debs for py2.4/py2.5/py2.6 is painful (nobody does it anymore, now that debian-python-policy generates .pycs at install time). And I don't believe the .deb control-file "Depends:" format isn't rich enough to express (py>=2.5 or pysqlite). On the other hand, the "standard" version of python is well-known per-platform (it's 2.4 in the older ones, 2.5 in the newer ones, and I guess Ubuntu Karmic will probably have 2.6). So we could say that when we build a .deb on a platform with /usr/bin/python=2.4, we declare the dependency upon pysqlite, and when we build it on a py2.5-using platform, we don't. To implement this, we should update the misc/*/debian/control files to declare the pysqlite dependency on the older platforms, and leave the newer ones alone (not declaring it). This also means that we should probably stop using the feisty rules for all platforms.. we at least need to identify the release for which python switched to 2.5 and use a newer set of rules for the 2.5-using set. I'll try to look at this today, but I'm not sure I'll get the time.

warner commented

2009-07-03 02:23:29 +00:00

Ok, I think I understand this better now.

debian/etch ships with python2.4 as the standard /usr/bin/python
(http://packages.debian.org/etch/python).
our most recent tahoe .deb for etch
(http://allmydata.org/debian/dists/etch/tahoe/binary-i386/allmydata-tahoe_1.4.1-r3916_all.deb)
was built on an Etch system with python2.4
therefore its _auto_deps.py correctly declares a requirement on
pysqlite, enforced at runtime, via the call to require_auto_deps()
in src/allmydata/*init*.py
the .deb's .egg-info/requires.txt correctly includes a line which
requires pysqlite >= 2.0.5, which will probably be enforced by
anything that goes poking around .egg-info . In particular, the code that
was added to src/allmydata/scripts/runner.py (the main CLI entry point)
to call pkg_requires.require('allmydata-tahoe') will read the
.egg-info data and enforce its requirements.
however, the packaging (i.e. debian/control) does not declare this
requirement. This is a bug. Installing just this .deb (and no extra
non-Depends-required packages) and then running the supplied
/usr/bin/tahoe with python2.4 should fail with the same sort of exception
displayed, because _auto_deps.py insists on having pysqlite, but APT
did not know that python-pysqlite2 should have been installed.
But, 4-tea-2 appears to be using python2.5 to start the Tahoe node on this
etch system, because the traceback listed above shows
/usr/lib/python2.5/site-packages. Etch includes a python2.5 package
(http://packages.debian.org/etch/python2.5), but it does not touch
/usr/bin/python, so either 4-tea-2 has set up a symlink or installed
python2.5 outside the awareness of APT (perhaps directly to
/usr/local/bin). This is what was confusing me.
the .deb provides an allmydata_tahoe-1.4.1_r3916-py2.4.egg-info/
directory, and does not provide any .egg-info data for python2.5 . I
don't know how setuptools reacts to this: would it refuse to believe that
tahoe was actually installed when run under py2.5? If so, then I'm still
confused as to where 4-tea-2's tahoe was finding the .egg-info data.
when run under python2.5, the code in _auto_deps.py should not
require pysqlite
however the .egg-info line (generated on our Etch buildslave) still
requires it, and thus the scripts/runner.py call will demand it.

Why is that pkg_resources.require('allmydata-tahoe') line there? If it
weren't, then the correct runtime-computed dependencies in _auto_deps.py
would be enforced, and the incorrect package-build-time dependencies in
.egg-info/requires.txt would not.

Now, it happens that the .deb packaging for this is incorrect, and the Etch
.deb should declare a dependency on python-sqlite2 because that's needed by
the default version of python on an Etch system, and it happens that fixing
the packaging would also happen to make this problem go away. But it's still
wrong: this is a dependency that changes depending upon the version of Python
used to run bin/tahoe, and recording the value that was computed at
package-build time seems inappropriate. It should be possible to run tahoe
under py2.5 on this system without having pysqlite installed.

Ok, I think I understand this better now. * debian/etch ships with python2.4 as the standard /usr/bin/python (<http://packages.debian.org/etch/python>). * our most recent tahoe .deb for etch (<http://allmydata.org/debian/dists/etch/tahoe/binary-i386/allmydata-tahoe_1.4.1-r3916_all.deb>) was built on an Etch system with python2.4 * therefore its `_auto_deps.py` correctly declares a requirement on pysqlite, enforced at runtime, via the call to `require_auto_deps()` in `src/allmydata/*init*.py` * the .deb's .egg-info/requires.txt correctly includes a line which requires `pysqlite >= 2.0.5`, which will probably be enforced by anything that goes poking around .egg-info . In particular, the code that was added to src/allmydata/scripts/runner.py (the main CLI entry point) to call `pkg_requires.require('allmydata-tahoe')` will read the .egg-info data and enforce its requirements. * however, the packaging (i.e. debian/control) does not declare this requirement. This is a bug. Installing just this .deb (and no extra non-Depends-required packages) and then running the supplied /usr/bin/tahoe with python2.4 should fail with the same sort of exception displayed, because `_auto_deps.py` insists on having pysqlite, but APT did not know that python-pysqlite2 should have been installed. * But, 4-tea-2 appears to be using python2.5 to start the Tahoe node on this etch system, because the traceback listed above shows `/usr/lib/python2.5/site-packages`. Etch includes a python2.5 package (<http://packages.debian.org/etch/python2.5>), but it does not touch /usr/bin/python, so either 4-tea-2 has set up a symlink or installed python2.5 outside the awareness of APT (perhaps directly to /usr/local/bin). This is what was confusing me. * the .deb provides an allmydata_tahoe-1.4.1_r3916-py2.4.egg-info/ directory, and does not provide any .egg-info data for python2.5 . I don't know how setuptools reacts to this: would it refuse to believe that tahoe was actually installed when run under py2.5? If so, then I'm still confused as to where 4-tea-2's tahoe was finding the .egg-info data. * when run under python2.5, the code in `_auto_deps.py` should *not* require pysqlite * however the .egg-info line (generated on our Etch buildslave) still requires it, and thus the scripts/runner.py call will demand it. Why is that `pkg_resources.require('allmydata-tahoe')` line there? If it weren't, then the correct runtime-computed dependencies in _auto_deps.py would be enforced, and the incorrect package-build-time dependencies in .egg-info/requires.txt would not. Now, it happens that the .deb packaging for this is incorrect, and the Etch .deb should declare a dependency on python-sqlite2 because that's needed by the default version of python on an Etch system, and it happens that fixing the packaging would also happen to make this problem go away. But it's still wrong: this is a dependency that changes depending upon the version of Python used to run bin/tahoe, and recording the value that was computed at package-build time seems inappropriate. It should be possible to run tahoe under py2.5 on this system without having pysqlite installed.

zooko commented

2009-07-03 02:29:04 +00:00

I guess a quicker solution is just to have the allmydata-tahoe .deb always depend on python-sqlite. While we're at it, I can swap the order of attempted-import in source:src/allmydata/init.py and source:src/allmydata/scripts/backupdb.py, so that whatever python-sqlite package that is installed is the one that actually gets used (in preference to the sqlite module that comes in the Python standard library).

I guess a quicker solution is just to have the `allmydata-tahoe` .deb always depend on `python-sqlite`. While we're at it, I can swap the order of attempted-import in source:src/allmydata/*init*.py and source:src/allmydata/scripts/backupdb.py, so that whatever `python-sqlite` package that is installed is the one that actually gets used (in preference to the `sqlite` module that comes in the Python standard library).

zooko commented

2009-07-03 02:30:50 +00:00

My comment:372662 was in reply to your comment:372660, not your comment:372661.

My [comment:372662](/tahoe-lafs/trac/issues/745#issuecomment-372662) was in reply to your [comment:372660](/tahoe-lafs/trac/issues/745#issuecomment-372660), not your [comment:372661](/tahoe-lafs/trac/issues/745#issuecomment-372661).

warner commented

2009-07-03 16:24:57 +00:00

I updated the debian packaging rules last night, and now the py2.4-based distributions (etch and edgy, I believe) add a python-sqlite2 dependency, and the more recent ones do not. I'd prefer to not force py2.5-based platforms to install an extra package. And I'd prefer that we use the stdlib sqlite3 module in preference to an external pysqlite2 module. Think about it this way: when py2.4 is long dead, do you still want us to be requiring people to install a third-party module to duplicate stdlib functionality?

I updated the debian packaging rules last night, and now the py2.4-based distributions (etch and edgy, I believe) add a python-sqlite2 dependency, and the more recent ones do not. I'd prefer to not force py2.5-based platforms to install an extra package. And I'd prefer that we use the stdlib `sqlite3` module in preference to an external `pysqlite2` module. Think about it this way: when py2.4 is long dead, do you still want us to be requiring people to install a third-party module to duplicate stdlib functionality?