have a separate build target to download any missing deps but not to compile or install them #1270

New issue

Closed

opened 2010-11-27 18:57:27 +00:00 by zooko · 8 comments

zooko commented

2010-11-27 18:57:27 +00:00

As mentioned by gdt in comment:381083, it would be useful for packagers if they could ask the build system to download any missing dependencies, but not to build them or install them.

This might be related to or satisfied by the "tahoe-deps" and/or SUMO tarball features, which we have sort-of working right now.

Let's use this ticket to work out what test would satisfy this feature request. gdt: how could we test this? Maybe just do a test like the one described in comment:21:/tahoe-lafs/trac/issues/27164 (the one which will go red if the code under test says "Downloading http") but we first run python setup.py justdeps. So then after running python setup.py justdeps, then it runs python setup.py justbuild and goes red if it says "Downloading http", then it runs python setup.py justinstall --prefix=$PREFIXDIR (and again goes red if it says "Downloading http"), then it runs $PREFIXDIR/bin/tahoe --version-and-path and goes red unless it prints out the right version and path.

Note that the test code doesn't have a fair chance of marking the code under test red if the system already has all deps already installed. There have to be some missing deps so that the code under test might be tempted to try to download those deps when it isn't supposed to.

As mentioned by gdt in [comment:381083](/tahoe-lafs/trac/issues/1200#issuecomment-381083), it would be useful for packagers if they could ask the build system to download any missing dependencies, but not to build them or install them. This might be related to or satisfied by the "tahoe-deps" and/or SUMO tarball features, which we have sort-of working right now. Let's use this ticket to work out what test would satisfy this feature request. gdt: how could we test this? Maybe just do a test like the one described in comment:21:[/tahoe-lafs/trac/issues/27164](/tahoe-lafs/trac/issues/27164) (the one which will go red if the code under test says "Downloading http") but we first run `python setup.py justdeps`. So then after running `python setup.py justdeps`, then it runs `python setup.py justbuild` and goes red if it says "Downloading http", then it runs `python setup.py justinstall --prefix=$PREFIXDIR` (and again goes red if it says "Downloading http"), then it runs `$PREFIXDIR/bin/tahoe --version-and-path` and goes red unless it prints out the right version and path. Note that the test code doesn't have a fair chance of marking the code under test red if the system already has all deps already installed. There have to be some missing deps so that the code under test might be tempted to try to download those deps when it isn't supposed to.

zooko added the

labels 2010-11-27 18:57:27 +00:00

zooko added this to the undecided milestone 2010-11-27 18:57:27 +00:00

warner commented

2010-11-27 20:25:21 +00:00

Incidentally, let's call this command "setup.py download-deps", or fetch-deps.. something more meaningful than "justXYZ".

In my "unsuck" branch, I'm taking the approach of embedding a list of (NAME, URL, SHA256) tuples, and it will refuse to expose a downloaded tarball which doesn't match the hash (download to tempfile, check hash, rename). I really really don't like the current setuptools behavior of downloading random things from HTTP websites and executing them.

Sounds like we need a buildslave environment which is specifically designated to have (or lack) certain dependencies. Part of the warmup for that build will be to assert that those dependencies are (or are not) present. If we had an easy way to spin up predefined VMs as buildslaves on demand, it'd be easier to accomplish that. Using personal machines as buildslaves (or really any machine that serves other purposes), such that they may be upgraded or improved for non-tahoe-buildbot reasons, makes it challenging. Incidentally, let's call this command "`setup.py download-deps`", or `fetch-deps`.. something more meaningful than "justXYZ". In my "unsuck" branch, I'm taking the approach of embedding a list of (NAME, URL, SHA256) tuples, and it will refuse to expose a downloaded tarball which doesn't match the hash (download to tempfile, check hash, rename). I really really don't like the current setuptools behavior of downloading random things from HTTP websites and executing them.

zooko commented

2010-11-27 21:14:28 +00:00

Author

Replying to warner:

Sounds like we need a buildslave environment which is specifically designated to have (or lack) certain dependencies.

Yes. See also #1256 (set up pbuilder for Debian/Ubuntu buildslaves), which is the Right Way to do this for Debian/Ubuntu.

Part of the warmup for that build will be to assert that those dependencies are (or are not) present. If we had an easy way to spin up predefined VMs as buildslaves on demand,

That would be sweet. :-) I would still want the personal/ad-hoc/varying buildslaves for other reasons though.

In my "unsuck" branch, I'm taking the approach of embedding a list of (NAME, URL, SHA256) tuples, and it will refuse to expose a downloaded tarball which doesn't match the hash (download to tempfile, check hash, rename). I really really don't like the current setuptools behavior of downloading random things from HTTP websites and executing them.

Good idea! setuptools/distribute has a similar functionality (using md5 instead of sha256). Let's write a build step test which goes red if you specified a specific hash and the package you got doesn't match that hash, and you go ahead and use it!

Replying to [warner](/tahoe-lafs/trac/issues/1270#issuecomment-382390): > Sounds like we need a buildslave environment which is specifically designated to have (or lack) certain dependencies. Yes. See also #1256 (set up pbuilder for Debian/Ubuntu buildslaves), which is the Right Way to do this for Debian/Ubuntu. > Part of the warmup for that build will be to assert that those dependencies are (or are not) present. If we had an easy way to spin up predefined VMs as buildslaves on demand, That would be sweet. :-) I would still want the personal/ad-hoc/varying buildslaves for other reasons though. > In my "unsuck" branch, I'm taking the approach of embedding a list of (NAME, URL, SHA256) tuples, and it will refuse to expose a downloaded tarball which doesn't match the hash (download to tempfile, check hash, rename). I really really don't like the current setuptools behavior of downloading random things from HTTP websites and executing them. Good idea! setuptools/distribute has a similar functionality (using md5 instead of sha256). Let's write a build step test which goes red if you specified a specific hash and the package you got doesn't match that hash, and you go ahead and use it!

gdt commented

2010-11-27 22:57:54 +00:00

Owner

From the packager point of view, what I want is much simpler.

Don't ever try to automatically get dependencies. Don't download, don't build. If it's not already there and satisfactory, just error out. The packager will then have to go package the dependency (which is fine), record in the current package's control files that there is a dependency, and then try to build again.
Don't depend on packages unless they are necessary. While I see the point of setuptools_darcsver in making released packages from darcs, the released packages seem to build just fine with that dependency removed.

I realize the above is totally in conflict with being user friendly, but hopefully this can be an option to enable for packaging systems. I don't yet understand if this is a general issue with python, or if the tahoe-world's package are somehow more automatic than normal.

I'm happy to work in ticket comments, but I think it would helpful to start a wiki page that documents the combined wisdom about goals and python behavior in various circumstances.

From the packager point of view, what I want is much simpler. * Don't ever try to automatically get dependencies. Don't download, don't build. If it's not already there and satisfactory, just error out. The packager will then have to go package the dependency (which is fine), record in the current package's control files that there is a dependency, and then try to build again. * Don't depend on packages unless they are necessary. While I see the point of setuptools_darcsver in making released packages from darcs, the released packages seem to build just fine with that dependency removed. I realize the above is totally in conflict with being user friendly, but hopefully this can be an option to enable for packaging systems. I don't yet understand if this is a general issue with python, or if the tahoe-world's package are somehow more automatic than normal. I'm happy to work in ticket comments, but I think it would helpful to start a wiki page that documents the combined wisdom about goals and python behavior in various circumstances.

zooko commented

2011-03-07 12:31:01 +00:00

Author

Replying to gdt:

From the packager point of view, what I want is much simpler.

Don't ever try to automatically get dependencies.

It sounds like you're saying you don't want to use the python setup.py justdeps command that would be created as a solution to this ticket. That's fine, but you're not saying that you actively want python setup.py justdeps to not exist, right? You can just not use it. So this is not the ticket you're looking for. Move along.

Replying to [gdt](/tahoe-lafs/trac/issues/1270#issuecomment-382392): > From the packager point of view, what I want is much simpler. > * Don't ever try to automatically get dependencies. It sounds like you're saying you don't want to use the `python setup.py justdeps` command that would be created as a solution to this ticket. That's fine, but you're not saying that you actively want `python setup.py justdeps` to not exist, right? You can just not use it. So this is not the ticket you're looking for. Move along.

gdt commented

2011-03-07 12:45:42 +00:00

Owner

I have no objection to "justdeps" existing. But I don't think it helps packagers in any significant way.

warner commented

2011-03-09 16:40:48 +00:00

I think we need a 'build' command that behaves like a traditional build command: exactly the way gdt described in comment:382392 . Personally, I'd like this command to get the short name ('make build' or 'python setup.py build', or 'python setup.py install', or something), and to give a longer name to the command that also calculates missing dependencies / downloads them / builds them / maybe-installs them. The "user-friendly" do-everything-for-me instructions would point people at the longer do-everything name, and the shorter just-build command would behave as packagers expect.

And yeah, a wiki page to describe our packaging goals, and what tools we already have (and would like to add) to accomplish them, would be a good thing.

I think we need a 'build' command that behaves like a traditional build command: exactly the way gdt described in [comment:382392](/tahoe-lafs/trac/issues/1270#issuecomment-382392) . Personally, I'd like this command to get the short name ('`make build`' or '`python setup.py build`', or '`python setup.py install`', or something), and to give a longer name to the command that also calculates missing dependencies / downloads them / builds them / maybe-installs them. The "user-friendly" do-everything-for-me instructions would point people at the longer do-everything name, and the shorter just-build command would behave as packagers expect. And yeah, a wiki page to describe our packaging goals, and what tools we already have (and would like to add) to accomplish them, would be a good thing.

zooko commented

2011-03-09 17:04:42 +00:00

Author

I really hope we don't spend a lot of time arguing about which command gets the shorter name. :-(

But...

While the "just build" behavior is the standard behavior for unix tools and packagers like gdt for a command with a short name like "install", the "automatically resolve dependencies" behavior is the standard behavior for Python tools for the command "python setup.py install".

Or at least it was two years ago when there was a large survey about such things performed by Tarek Ziadé. Perhaps it is changing.

I really hope we don't spend a lot of time arguing about which command gets the shorter name. :-( But... While the "just build" behavior is the standard behavior for unix tools and packagers like gdt for a command with a short name like "install", the "automatically resolve dependencies" behavior is the standard behavior for Python tools for the command "python setup.py install". Or at least it was two years ago when there was a large survey about such things performed by Tarek Ziadé. Perhaps it is changing.

warner commented

2016-03-26 21:35:14 +00:00

Now that we're on pip, the command to download the sources for all dependencies (but not compile or install them) is:

pip download tahoe-lafs

And pip wheel tahoe-lafs will download+compile, but not install, leaving you with a directory full of *.whl files.

The original request is more of an INVALID now, since we no longer want people to even attempt to use dependencies that are already present on their system (instead we recommend creating a new virtualenv, isolated from the system).

Now that we're on pip, the command to download the sources for all dependencies (but not compile or install them) is: ``` pip download tahoe-lafs ``` And `pip wheel tahoe-lafs` will download+compile, but not install, leaving you with a directory full of `*.whl` files. The original request is more of an INVALID now, since we no longer want people to even attempt to use dependencies that are already present on their system (instead we recommend creating a new virtualenv, isolated from the system).

Rows
Columns