make debian packages of dependencies #251

New issue

Closed

opened 2007-12-31 23:54:39 +00:00 by warner · 11 comments

warner commented

2007-12-31 23:54:39 +00:00

To release tahoe-0.7.0, we need debian packages of all runtime dependencies
that aren't already in debian, so that the tahoe .debs will actually be
usable.

That means:

zfec-1.3.4
pycryptopp-0.2.10
foolscap-0.2.3

The zfec package should really depend upon argparse (/usr/bin/zfec requires
it), so that means a debian package for argparse too.

Post-0.7.0, if we implement #47 and depend upon pyutil, then we'll need to
make debian packages of pyutil-1.3.12 too.

Foolscap comes with its own debian packaging rules: just do 'make
debian-feisty' or whatever.

Zooko's packages use setuptools, and I've been using the 'stdeb' utility to
turn them into debian packages. It's not trivial. The first step is to
install stdeb (perhaps from http://stdeb.python-hosting.com/), then have it
generate a .orig.tar.gz, .dsc, and most importantly the .diff.gz debian
files. The next step is to unpack the tarball and apply the patch, then run
'debuild -uc -us' or './debian/rules binary' or dpkg-buildpackage (or some
other utility that I'm not familiar with) and see if it works. Note that it's
best if a copy of the .orig.tar.gz is present in the parent of the working
directory, because then debuild has something against which it can generate a
new .diff file. Also keep a copy of the stdeb-produced .diff.gz somewhere
else, since debuild will probably overwrite the one in the parent directory.

The resulting package must also be inspected to see if it has all the desired
files: usually documentation, examples, and README files are missing and/or
misplaced. Most importantly, the debian/control file needs to be updated to
list all the debian dependencies that the package requires: it can usually
figure out C/.so deps by itself, but it can't usually figure out python
dependencies. Build-time dependencies also go into debian/control . Edit
debian/rules and debian/control until things work the way they ought to, then
run 'debuild'/etc again to regenerate the .diff.gz . Then copy the
.orig.tar.gz and .diff.gz over to a new directory, unpack/patch/build to make
sure it all works. Then the source .orig.tar.gz and .diff.gz get copied to
buildslave@hanford:tahoe-debs/dists/DIST/main/source/ , and the .deb file
gets copied to main/binary-i386/ .

This needs to be done for each debian/ubuntu release we support: edgy,
feisty, gusty, etch, and sid. It would probably be nice to make amd64
versions as well, but we haven't ever done this before and it would require
changes to our repository config to actually make them reachable.

The same .diff.gz can probably be used for multiple releases, although
several minimum versions probably get recorded in the debian/control files,
so if you choose this approach, you might want to run stdeb on our oldest
supported platform (which would be edgy) rather than on the newest (sid).
Either 1: copy the .orig.tar.gz and .diff.gz to a new host, run debuild, then
upload everything to hanford, or 2: copy (on hanford) the .orig.tar.gz and
.diff.gz (and .dsc) into DIST/main/source/ for all dists, then go to the
debian build host and do 'apt-get update && apt-get source PACKAGENAME',
which will grab the source from hanford and unpack it. Doing 'apt-get
build-dep PACKAGENAME' will look at the build-time dependencies and install
them, which can save a lot of time. If you choose this path, you'll only have
to upload the .deb file to hanford.

Once you've added something to tahoe-debs/ , run 'make' in that directory and
it will update the index and push a copy out to org.

Once something has been packaged, it is frequently sufficient to apply the
old .diff.gz to the new release tarball, add an entry to debian/changelog,
modify the version number that stdeb unfortunately embeds in debian/rules,
then run debuild.

Debian build hosts:

edgy: slave-edgy.allmydata.com (inside the firewall)
feisty: tahoeslave-fesity (inside the firewall)
gutsy: tahoecs2 (in colo)
etch: slave-etch (inside the firewall)
sid: fluxx (inside the firewall)

For reference, we use slave3.allmydata.com (in colo) as a dapper buildslave.
We test tahoe on dapper but don't produce debian package for it (although I
do make foolscap packages for dapper).

To release tahoe-0.7.0, we need debian packages of all runtime dependencies that aren't already in debian, so that the tahoe .debs will actually be usable. That means: * zfec-1.3.4 * pycryptopp-0.2.10 * foolscap-0.2.3 The zfec package should really depend upon argparse (/usr/bin/zfec requires it), so that means a debian package for argparse too. Post-0.7.0, if we implement #47 and depend upon pyutil, then we'll need to make debian packages of pyutil-1.3.12 too. Foolscap comes with its own debian packaging rules: just do 'make debian-feisty' or whatever. Zooko's packages use setuptools, and I've been using the 'stdeb' utility to turn them into debian packages. It's not trivial. The first step is to install stdeb (perhaps from <http://stdeb.python-hosting.com/>), then have it generate a .orig.tar.gz, .dsc, and most importantly the .diff.gz debian files. The next step is to unpack the tarball and apply the patch, then run 'debuild -uc -us' or './debian/rules binary' or dpkg-buildpackage (or some other utility that I'm not familiar with) and see if it works. Note that it's best if a copy of the .orig.tar.gz is present in the parent of the working directory, because then debuild has something against which it can generate a new .diff file. Also keep a copy of the stdeb-produced .diff.gz somewhere else, since debuild will probably overwrite the one in the parent directory. The resulting package must also be inspected to see if it has all the desired files: usually documentation, examples, and README files are missing and/or misplaced. Most importantly, the debian/control file needs to be updated to list all the debian dependencies that the package requires: it can usually figure out C/.so deps by itself, but it can't usually figure out python dependencies. Build-time dependencies also go into debian/control . Edit debian/rules and debian/control until things work the way they ought to, then run 'debuild'/etc again to regenerate the .diff.gz . Then copy the .orig.tar.gz and .diff.gz over to a new directory, unpack/patch/build to make sure it all works. Then the source .orig.tar.gz and .diff.gz get copied to buildslave@hanford:tahoe-debs/dists/DIST/main/source/ , and the .deb file gets copied to main/binary-i386/ . This needs to be done for each debian/ubuntu release we support: edgy, feisty, gusty, etch, and sid. It would probably be nice to make amd64 versions as well, but we haven't ever done this before and it would require changes to our repository config to actually make them reachable. The same .diff.gz can probably be used for multiple releases, although several minimum versions probably get recorded in the debian/control files, so if you choose this approach, you might want to run stdeb on our oldest supported platform (which would be edgy) rather than on the newest (sid). Either 1: copy the .orig.tar.gz and .diff.gz to a new host, run debuild, then upload everything to hanford, or 2: copy (on hanford) the .orig.tar.gz and .diff.gz (and .dsc) into DIST/main/source/ for all dists, then go to the debian build host and do 'apt-get update && apt-get source PACKAGENAME', which will grab the source from hanford and unpack it. Doing 'apt-get build-dep PACKAGENAME' will look at the build-time dependencies and install them, which can save a lot of time. If you choose this path, you'll only have to upload the .deb file to hanford. Once you've added something to tahoe-debs/ , run 'make' in that directory and it will update the index and push a copy out to org. Once something has been packaged, it is frequently sufficient to apply the old .diff.gz to the new release tarball, add an entry to debian/changelog, modify the version number that stdeb unfortunately embeds in debian/rules, then run debuild. Debian build hosts: * edgy: slave-edgy.allmydata.com (inside the firewall) * feisty: tahoeslave-fesity (inside the firewall) * gutsy: tahoecs2 (in colo) * etch: slave-etch (inside the firewall) * sid: fluxx (inside the firewall) For reference, we use slave3.allmydata.com (in colo) as a dapper buildslave. We test tahoe on dapper but don't produce debian package for it (although I do make foolscap packages for dapper).

warner added the

labels 2007-12-31 23:54:39 +00:00

warner added this to the 0.7.0 milestone 2007-12-31 23:54:39 +00:00

warner commented

2008-01-01 00:16:01 +00:00

Author

ok, foolscap-0.2.3 debs have been created for i386 on all releases

warner commented

2008-01-01 00:40:49 +00:00

Author

I started to work on the pycrypto debs, but the new pycrypto requires darcsver and setuptools_darcs and stuff, and that got to be too much work, so I think I'll leave it for zooko. I suspect zfec has the same requirements.

zooko commented

2008-01-01 22:18:22 +00:00

Okay, I've just spent some time learning how to to this -- thank you, Brian, for the extensive notes -- and it seems like packaging things for Debian is a lot of work.

Could we de-prioritize this for v0.7.0?

My general strategy for Tahoe until now has been not to offer binaries, because that's more work for us, and I don't mind if Tahoe v0.7.0 is available only to that subset of people who are willing to build from source, and I would like to spend our development effort on improving the build-from-source process. (Which still needs a lot of work, as we speak.)

My intent had been to make Tahoe v0.7.0 more easily installable from source (and we've succeeded -- see the new install.html), and then to start offering binary packages in Tahoe v0.7.1.

When it is time for us to spend the development resources to support binary distributions, then my priorities for platforms to support would be: 1. Whatever Zandr wants to deploy Tahoe on (I guess that is Gutsy), 2. Windows, which would potentially help Mike, Dmitry, Greg Hazel, Fabrice (?), Peter, Richard Samantha, and lots of other people, 3. whatever platform Richard Samantha has in his lab. 4. More Debian-likes not already supported for Zandr as number 1 above, 5. Mac OS X (note that allmydata.com receives frequent requests for a Mac client).

So I accept responsibility for increasing the work-load of making Debian packages by my choice of how to split up Python packages. Part of my justification for doing that is that it decreases the work-load for development and for making Python packages, which is the kind of trade-off that led me to focus on source packaging this time around.

So I'm willing to spend my time and effort to support Debian, which is a platform that I love and which I consider very important, but I would like to prioritize other things (The Roadmap) over this issue for v0.7.0.

By the way, both darcsver and setuptools_darcs are build requirements, not install requirements, so presumably we don't need to produce .deb's for them. Hopefully that will make things easier.

"To release tahoe-0.7.0, we need debian packages of all runtime dependencies that aren't already in debian, so that the tahoe .debs will actually be usable. ... This needs to be done for each debian/ubuntu release we support: edgy, feisty, gusty, etch, and sid." Okay, I've just spent some time learning how to to this -- thank you, Brian, for the extensive notes -- and it seems like packaging things for Debian is a lot of work. Could we de-prioritize this for v0.7.0? My general strategy for Tahoe until now has been not to offer binaries, because that's more work for us, and I don't mind if Tahoe v0.7.0 is available only to that subset of people who are willing to build from source, and I would like to spend our development effort on improving the build-from-source process. (Which still needs a lot of work, as we speak.) My intent had been to make Tahoe v0.7.0 more easily installable from source (and we've succeeded -- see the new [install.html](http://allmydata.org/trac/tahoe/browser/docs/install.html)), and then to start offering binary packages in Tahoe v0.7.1. When it is time for us to spend the development resources to support binary distributions, then my priorities for platforms to support would be: 1. Whatever Zandr wants to deploy Tahoe on (I guess that is Gutsy), 2. Windows, which would potentially help Mike, Dmitry, Greg Hazel, Fabrice (?), Peter, Richard Samantha, and lots of other people, 3. whatever platform Richard Samantha has in his lab. 4. More Debian-likes not already supported for Zandr as number 1 above, 5. Mac OS X (note that allmydata.com receives frequent requests for a Mac client). So I accept responsibility for increasing the work-load of making Debian packages by my choice of how to split up Python packages. Part of my justification for doing that is that it decreases the work-load for development and for making Python packages, which is the kind of trade-off that led me to focus on source packaging this time around. So I'm willing to spend my time and effort to support Debian, which is a platform that I love and which I consider very important, but I would like to prioritize other things ([The Roadmap](http://allmydata.org/trac/tahoe/wiki/TracRoadmap)) over this issue for v0.7.0. By the way, both darcsver and setuptools_darcs are build requirements, not install requirements, so presumably we don't need to produce .deb's for them. Hopefully that will make things easier.

warner commented

2008-01-02 18:46:39 +00:00

Author

Could we de-prioritize this for v0.7.0?

I'm willing to accept that (although I can't say I'm happy about the
regression). The consequences are:

production nodes (testnet, perfnet, loadnet, i.e. tahoebs1-5 and tahoecs2)
will be unable to run from debian packages, and must therefore run from
source. I'm unwilling to have production machines using code installed via
easy_install, and I'm reluctant to have them use code from /usr/local :
these machines are a test case for how Zandr would deploy servers, so if
we're not going to run them from debian packages, the remaining option
which is the easiest to keep straight and to clean up after is running
from a local source tree.
we should stop updating our APT repository with new Tahoe .debs, since
they are unusable, and older ones (before these new dependencies went in)
are more usable than the new ones. In fact we should figure out what the
last known usable version is, and remove any .debs produced after that
date. Of course we should keep the buildbot .deb builder running.. we can
just turn off the automatic mirror-to-org script. We should also update
the .deb making code in our tree to express the new dependencies, so
that anyone who does try to use them gets a clear install error rather
than a cryptic runtime error.

Both of those are pretty easy to manage. So, ok, 0.7.0 will not have
functional .debs, but the first thing we do for 0.7.1 should be to make these
work again.

> Could we de-prioritize this for v0.7.0? I'm willing to accept that (although I can't say I'm happy about the regression). The consequences are: * production nodes (testnet, perfnet, loadnet, i.e. tahoebs1-5 and tahoecs2) will be unable to run from debian packages, and must therefore run from source. I'm unwilling to have production machines using code installed via easy_install, and I'm reluctant to have them use code from /usr/local : these machines are a test case for how Zandr would deploy servers, so if we're not going to run them from debian packages, the remaining option which is the easiest to keep straight and to clean up after is running from a local source tree. * we should stop updating our APT repository with new Tahoe .debs, since they are unusable, and older ones (before these new dependencies went in) are more usable than the new ones. In fact we should figure out what the last known usable version is, and remove any .debs produced after that date. Of course we should keep the buildbot .deb builder running.. we can just turn off the automatic mirror-to-org script. We should also update the .deb making code in our tree to express the new dependencies, so that anyone who *does* try to use them gets a clear install error rather than a cryptic runtime error. Both of those are pretty easy to manage. So, ok, 0.7.0 will not have functional .debs, but the first thing we do for 0.7.1 should be to make these work again.

warner commented

2008-01-02 18:53:18 +00:00

Author

oh, and I concur that darcsver and setuptools_darcs do not require debian packages, as long as they are buildable in the Desert Island mode: the debian/rules code that build the tahoe .deb will need to build these as well, to figure out the version number that should be put into the debian/changelog (and thus determine the .deb's version number).

warner commented

2008-01-02 20:53:10 +00:00

Author

Hm, you might have lucked out: according to our calc-deps.py, tahoe only currently requires zfec-1.0.3 and pycryptopp-0.2.8 (even though we ship newer versions of both as tarballs in misc/dependencies/). We already have zfec-1.1 .debs for all platforms, and we have a pycryptopp-0.2.8 for feisty (which means that I've already done the necessary debianization work for that version). The current versions of zfec and pycryptopp are harder to build because of the new dependencies, but if we can get away with using the older versions for tahoe-0.7.0, then we can yet manage to have 0.7.0 be debian installable.

I'll look at the pycryptopp-0.2.8 feisty .deb and see how much work it will take to build it on the other platforms.

warner commented

2008-01-02 21:12:46 +00:00

Author

ok, we've now got pycryptopp-0.2.8 debs on all platforms. I think this will be enough to allow tahoe .debs to be installed and run.

zooko added this to the 0.8.0 (Allmydata 3.0 Beta) milestone 2008-01-23 02:43:22 +00:00

zooko modified the milestone from 0.8.0 (Allmydata 3.0 Beta) to 0.9.0 (Allmydata 3.0 final)

2008-03-08 02:54:38 +00:00

zooko commented

2008-03-10 19:44:41 +00:00

As long as we can run production servers against zfec-1.1 and pycryptopp-0.2.8, then we are currently okay on this issue, so I'm moving this to Milestone 0.10.0.

warner commented

2008-05-15 19:53:31 +00:00

Author

This has become critical again, since tahoe trunk now depends upon a newer version of pycryptopp than we have .debs for. We'll need those .debs to update our production machines to code beyond 1.0.0

warner added

p/critical

and removed

p/major

labels 2008-05-15 19:53:31 +00:00

zooko commented

2008-05-18 23:02:20 +00:00

Okay! I can produce a python-pycryptopp-0.5.2-1_amd64.deb just by running ./setup.py sdist_dsc && cd deb_dist/pycryptopp-0.5.2 && dpkg-buildpackage -rfakeroot -uc -us.

I've figured out how to configure pycryptopp so that the README.txt, COPYING.GPL and COPYING.TGPPL.html go into the right place (/usr/share/docs/python-pycryptopp.

Next, I want to figure out how to make the resulting .deb automatically depend on python-setuptools based on the fact that the setup.py specifies that pycryptopp install_requires setuptools:

http://allmydata.org/trac/pycryptopp/browser/setup.py

This is non-trivial, because setuptools and debian don't use exactly the same namespace -- i.e. I need to write either a heuristic ("foo in setuptools world probably means python-foo in debian world") or a map. Hm... :-)

Well, okay, it might be best to just use the feature which is explicitly for this purpose in the stdeb config file... We'll see... :-)

Okay! I can produce a `python-pycryptopp-0.5.2-1_amd64.deb` just by running `./setup.py sdist_dsc && cd deb_dist/pycryptopp-0.5.2 && dpkg-buildpackage -rfakeroot -uc -us`. I've figured out how to configure pycryptopp so that the `README.txt`, `COPYING.GPL` and `COPYING.TGPPL.html` go into the right place (`/usr/share/docs/python-pycryptopp`. Next, I want to figure out how to make the resulting .deb automatically depend on `python-setuptools` based on the fact that the `setup.py` specifies that pycryptopp install_requires setuptools: <http://allmydata.org/trac/pycryptopp/browser/setup.py> This is non-trivial, because setuptools and debian don't use exactly the same namespace -- i.e. I need to write either a heuristic ("foo in setuptools world probably means python-foo in debian world") or a map. Hm... :-) Well, okay, it might be best to just use the feature which is explicitly for this purpose in the stdeb config file... We'll see... :-)

zooko commented

2008-05-29 23:05:15 +00:00

Okay, we have new .deb's for zfec, argparse, pyutil, and pycryptopp, and we have an easy way to make more of them when desired, so we're closing this ticket.

See also follow-on tickets #422 (stdeb: run from buildslaves), #423 (stdeb: use stdeb on tahoe itself), and #424 (stdeb: push to upstream).

Okay, we have new .deb's for `zfec`, `argparse`, `pyutil`, and `pycryptopp`, and we have an easy way to make more of them when desired, so we're closing this ticket. See also follow-on tickets #422 (stdeb: run from buildslaves), #423 (stdeb: use stdeb on tahoe itself), and #424 (stdeb: push to upstream).

Rows
Columns