tahoe-lafs/trac

users expect 'python setup.py install' to install dependencies #2063

New issue

Closed

opened 2013-08-24 08:24:14 +00:00 by sel · 17 comments

sel commented

2013-08-24 08:24:14 +00:00

Owner

Problem after installation in Fedora 19:

$ tahoe(most recent call last):
  File "/usr/bin/tahoe", line 6, in <module>
    from pkg_resources import load_entry_point
ImportError: No module named pkg_resources

Problem after installation in Fedora 19: ``` $ tahoe(most recent call last): File "/usr/bin/tahoe", line 6, in <module> from pkg_resources import load_entry_point ImportError: No module named pkg_resources ```

tahoe-lafs added the

labels 2013-08-24 08:24:14 +00:00

tahoe-lafs added this to the undecided milestone 2013-08-24 08:24:14 +00:00

daira commented

2013-08-24 08:35:18 +00:00

What command was used to install Tahoe?

sel commented

2013-08-24 08:55:28 +00:00

Author

Owner

su -c 'python setup.py install'

daira changed title from ~~problem after installation in Fedora 19~~ to users expect 'python setup.py install' to install dependencies

2013-08-24 10:07:07 +00:00

sel commented

2013-08-24 11:57:27 +00:00

Author

Owner

a list of all dependencies should do, we can then install them with a single command.

gdt commented

2013-08-24 12:11:29 +00:00

Author

Owner

I think part of the point of the bug is that user expectations are not necessarily right. There are basically three approaches

use a package from a packaging system. Users should be pointed to this as the standard approach. The overwhelming majority of uses of successful programs are like this.
run some command from sources and have it go find and install dependencies (presuambly only some, and not thing like python!!)
read the README, install the dependencies from a packaging system, and then build/install. This is the standard approach for people who are going to do development rather than just run.

My only concern is that the auto-install-dependencies approach is actively harmful for packaging systems, because package builds should not use the net and should be 100% byte-for-byte repeatable. So getting dependencies should be an explicit step the user has to ask for (perhaps this can be hinted at in the error if they aren't there).

I think part of the point of the bug is that user expectations are not necessarily right. There are basically three approaches * use a package from a packaging system. Users should be pointed to this as the standard approach. The overwhelming majority of uses of successful programs are like this. * run some command from sources and have it go find and install dependencies (presuambly only some, and not thing like python!!) * read the README, install the dependencies from a packaging system, and then build/install. This is the standard approach for people who are going to do development rather than just run. My only concern is that the auto-install-dependencies approach is actively harmful for packaging systems, because package builds should not use the net and should be 100% byte-for-byte repeatable. So getting dependencies should be an explicit step the user has to ask for (perhaps this can be hinted at in the error if they aren't there).

sel commented

2013-08-24 13:49:30 +00:00

Author

Owner

The separate step approach is unusual in fedora/centos, unless you are building from sources, in which case you should be prepared to build/install every required package.
The situation I found: 1) Some dependencies were discovered during the 'build' process, 2) Some other dependencies were discovered after the 'install' process, when actually running the tahoe binary.

The separate step approach is unusual in fedora/centos, unless you are building from sources, in which case you should be prepared to build/install every required package. The situation I found: 1) Some dependencies were discovered during the 'build' process, 2) Some other dependencies were discovered after the 'install' process, when actually running the tahoe binary.

gdt commented

2013-08-24 14:04:56 +00:00

Author

Owner

In pkgsrc, this is handled by declaring all the things tahoe needs as dependencies, so those packages are installed before the build starts. Doesn't RHEL/FC/CentOS do the same thing?

@sel: I am having trouble following you. Are you talking about somehow who is building a tahoe-lafs package using packaging control scripts? Or a user not using the packaging system who has downloaded source. Surely it's normal to have to have the prereqs installed first; you can't build emacs without first having gtk2+ (well, assuming you want it built that way, as an example). In pkgsrc, this is handled by declaring all the things tahoe needs as dependencies, so those packages are installed before the build starts. Doesn't RHEL/FC/CentOS do the same thing?

sel commented

2013-08-24 14:18:49 +00:00

Author

Owner

One thing: right now fedora has mock-1.1.32, but tahoe did not run until I installed mock-0.8.0. This is a downgrade with respect to the packaging system.

I downloaded the .zip unpacked it and followed the instructions. I didn't find a list of dependencies to install, I have discovered them from error messages by running the build multiple times, and after the install by running tahoe multiple times. One thing: right now fedora has mock-1.1.32, but tahoe did not run until I installed mock-0.8.0. This is a downgrade with respect to the packaging system.

gdt commented

2013-08-24 15:11:36 +00:00

Author

Owner

So perhaps you should open a new ticket "The build instructions do not clearly specify the list of dependencies", which should be easy to fix. I admit to not having read them in a really long time.

sel commented

2013-08-26 07:01:48 +00:00

Author

Owner

Replying to gdt:

@sel: I am having trouble following you. Are you talking about somehow who is building a tahoe-lafs package using packaging control scripts? Or a user not using the packaging system who has downloaded source. Surely it's normal to have to have the prereqs installed first; you can't build emacs without first having gtk2+ (well, assuming you want it built that way, as an example).

In pkgsrc, this is handled by declaring all the things tahoe needs as dependencies, so those packages are installed before the build starts. Doesn't RHEL/FC/CentOS do the same thing?

It should do the same thing, however the packaging is different from Debian. RHEL/FC/CentOS uses yum, which wants a .rpm

Replying to [gdt](/tahoe-lafs/trac/issues/2063#issuecomment-394949): > @sel: I am having trouble following you. Are you talking about somehow who is building a tahoe-lafs package using packaging control scripts? Or a user not using the packaging system who has downloaded source. Surely it's normal to have to have the prereqs installed first; you can't build emacs without first having gtk2+ (well, assuming you want it built that way, as an example). > > In pkgsrc, this is handled by declaring all the things tahoe needs as dependencies, so those packages are installed before the build starts. Doesn't RHEL/FC/CentOS do the same thing? It should do the same thing, however the packaging is different from Debian. RHEL/FC/CentOS uses yum, which wants a .rpm

sel commented

2013-08-26 07:09:28 +00:00

Author

Owner

Replying to gdt:

I think part of the point of the bug is that user expectations are not necessarily right. There are basically three approaches

use a package from a packaging system. Users should be pointed to this as the standard approach. The overwhelming majority of uses of successful programs are like this.

run some command from sources and have it go find and install dependencies (presuambly only some, and not thing like python!!)

read the README, install the dependencies from a packaging system, and then build/install. This is the standard approach for people who are going to do development rather than just run.

My only concern is that the auto-install-dependencies approach is actively harmful for packaging systems, because package builds should not use the net and should be 100% byte-for-byte repeatable. So getting dependencies should be an explicit step the user has to ask for (perhaps this can be hinted at in the error if they aren't there).

Generally I think the package system just downloads prebuilt packages. It is okay to have a command line option that must be specified to install dependencies, it is also okay to have a list of dependencies that can be installed in a separate step.

My concern is: once I have created the introducer, I need to create N storage nodes, that's why I would like to automate the installation as much as possible.

Replying to [gdt](/tahoe-lafs/trac/issues/2063#issuecomment-394947): > I think part of the point of the bug is that user expectations are not necessarily right. There are basically three approaches > * use a package from a packaging system. Users should be pointed to this as the standard approach. The overwhelming majority of uses of successful programs are like this. > * run some command from sources and have it go find and install dependencies (presuambly only some, and not thing like python!!) > * read the README, install the dependencies from a packaging system, and then build/install. This is the standard approach for people who are going to do development rather than just run. > > My only concern is that the auto-install-dependencies approach is actively harmful for packaging systems, because package builds should not use the net and should be 100% byte-for-byte repeatable. So getting dependencies should be an explicit step the user has to ask for (perhaps this can be hinted at in the error if they aren't there). Generally I think the package system just downloads *prebuilt* packages. It is okay to have a command line option that must be specified to install dependencies, it is also okay to have a list of dependencies that can be installed in a separate step.<br> My concern is: once I have created the introducer, I need to create N storage nodes, that's why I would like to automate the installation as much as possible.

zooko commented

2013-08-27 11:40:10 +00:00

I agree that it is a problem that users expect python setup.py install to install dependencies, but it doesn't do that. The fact that it doesn't do that is tracked by ticket #1494. The fact that users decide to try python setup.py install is the subject of this ticket. (I think some of gdt's comments from comment:394947 are the subject of other tickets, and it would probably help if gdt would find those tickets and cross-link them to this one.)

So: one possible patch to help with this ticket would be to add a line to [quickstart.rst]source:trunk/docs/quickstart.rst saying something like:

Note: running `python setup.py install` will copy only the Tahoe-LAFS package and not any of the dependent packages into your system. We don't currently support any way to install Tahoe-LAFS into your system. Instead, we recommend that you run it from the directory you built it in, as described in this document. See also the instructions for getting prebuilt packages from your operating system: <https://tahoe-lafs.org/trac/tahoe-lafs/wiki/Installation>

What do you think? Would text like that make it so people who read quickstart.rst stop trying python setup.py install?

(Then, if anyone ever fixes #1494 and makes python setup.py install work, we could remove this text.)

I agree that it is a problem that users expect `python setup.py install` to install dependencies, but it doesn't do that. The fact that it doesn't do that is tracked by ticket #1494. The fact that users decide to try `python setup.py install` is the subject of this ticket. (I think some of gdt's comments from [comment:394947](/tahoe-lafs/trac/issues/2063#issuecomment-394947) are the subject of *other* tickets, and it would probably help if gdt would find those tickets and cross-link them to this one.) So: one possible patch to help with *this* ticket would be to add a line to [quickstart.rst]source:trunk/docs/quickstart.rst saying something like: Note: running `python setup.py install` will copy only the Tahoe-LAFS package and not any of the dependent packages into your system. We don't currently support any way to install Tahoe-LAFS into your system. Instead, we recommend that you run it from the directory you built it in, as described in this document. See also the instructions for getting prebuilt packages from your operating system: <https://tahoe-lafs.org/trac/tahoe-lafs/wiki/Installation> What do you think? Would text like that make it so people who read quickstart.rst stop trying `python setup.py install`? (Then, if anyone ever fixes #1494 and makes `python setup.py install` *work*, we could remove this text.)

daira commented

2013-08-27 12:15:39 +00:00

See also #1494.

sel commented

2013-08-29 10:37:27 +00:00

Author

Owner

just one quick comment: The package names in Debian repositories are different from the package names in other distributions such as Fedora/CentOS. Generally Fedora package names have the 'python-' prefix.

gdt commented

2013-08-30 17:39:18 +00:00

Author

Owner

Replying to [sel]comment:11:

Generally I think the package system just downloads prebuilt packages. It is okay to have a command line option that must be specified to install dependencies, it is also okay to have a list of dependencies that can be installed in a separate step.

My concern is: once I have created the introducer, I need to create N storage nodes, that's why I would like to automate the installation as much as possible.

Sorry, I wasn't clear enough. There are two parts to packaging systems. One is the control files that take the upstream distribution file and build a binary package. The other is how users install binary packages. Both of these mechanisms are within the packaging system, and both have to manage dependencies by expressing that other packages are required first.

It seems that in most GNU/Linux systems, users almost never build packages from source using the packaging system, but just use binary packages. In pkgsrc, the norm is to build from source but using the packaging system, because of the huge number of OS/os-version/CPU tuples.

I think it's an artifact of tahoe being in python that there is this focus on setup scripts automatically fetching and installing dependencies. In general, step one of building something from source is to read the dependency list and make sure those are installed. So I think the focus should be on having tahoe available via packaging systems, rather than out-of-system dependency management. (I've done this for pkgsrc, and thus am mostly ignoring the setuptools kerfluffle.)

Replying to [sel]comment:11: > Generally I think the package system just downloads *prebuilt* packages. It is okay to have a command line option that must be specified to install dependencies, it is also okay to have a list of dependencies that can be installed in a separate step.<br> > My concern is: once I have created the introducer, I need to create N storage nodes, that's why I would like to automate the installation as much as possible. Sorry, I wasn't clear enough. There are two parts to packaging systems. One is the control files that take the upstream distribution file and build a binary package. The other is how users install binary packages. Both of these mechanisms are within the packaging system, and both have to manage dependencies by expressing that other packages are required first. It seems that in most GNU/Linux systems, users almost never build packages from source using the packaging system, but just use binary packages. In pkgsrc, the norm is to build from source but using the packaging system, because of the huge number of OS/os-version/CPU tuples. I think it's an artifact of tahoe being in python that there is this focus on setup scripts automatically fetching and installing dependencies. In general, step one of building something from source is to read the dependency list and make sure those are installed. So I think the focus should be on having tahoe available via packaging systems, rather than out-of-system dependency management. (I've done this for pkgsrc, and thus am mostly ignoring the setuptools kerfluffle.)

sel commented

2013-08-30 18:03:59 +00:00

Author

Owner

Replying to [gdt]comment:16:

Replying to [sel]comment:11:

Generally I think the package system just downloads prebuilt packages. It is okay to have a command line option that must be specified to install dependencies, it is also okay to have a list of dependencies that can be installed in a separate step.

My concern is: once I have created the introducer, I need to create N storage nodes, that's why I would like to automate the installation as much as possible.

Sorry, I wasn't clear enough. There are two parts to packaging systems. One is the control files that take the upstream distribution file and build a binary package. The other is how users install binary packages. Both of these mechanisms are within the packaging system, and both have to manage dependencies by expressing that other packages are required first.

It seems that in most GNU/Linux systems, users almost never build packages from source using the packaging system, but just use binary packages. In pkgsrc, the norm is to build from source but using the packaging system, because of the huge number of OS/os-version/CPU tuples.

I think that pkgsrc and yum are different, afaik yum uses separate packages for binaries (rpm) or sources (srpm), but I have never seen yum build packages from source.

I think it's an artifact of tahoe being in python that there is this focus on setup scripts automatically fetching and installing dependencies. In general, step one of building something from source is to read the dependency list and make sure those are installed. So I think the focus should be on having tahoe available via packaging systems, rather than out-of-system dependency management. (I've done this for pkgsrc, and thus am mostly ignoring the setuptools kerfluffle.)

There are a couple of situations where the packaging system doesn't help: for example the mock package (needed by tahoe) that comes with current fedora, is higher version (1.1.32) from the one needed by tahoe (0.8.0).

Replying to [gdt]comment:16: > Replying to [sel]comment:11: > > Generally I think the package system just downloads *prebuilt* packages. It is okay to have a command line option that must be specified to install dependencies, it is also okay to have a list of dependencies that can be installed in a separate step.<br> > > My concern is: once I have created the introducer, I need to create N storage nodes, that's why I would like to automate the installation as much as possible. > > Sorry, I wasn't clear enough. There are two parts to packaging systems. One is the control files that take the upstream distribution file and build a binary package. The other is how users install binary packages. Both of these mechanisms are within the packaging system, and both have to manage dependencies by expressing that other packages are required first. > > It seems that in most GNU/Linux systems, users almost never build packages from source using the packaging system, but just use binary packages. In pkgsrc, the norm is to build from source but using the packaging system, because of the huge number of OS/os-version/CPU tuples. I think that pkgsrc and yum are different, afaik yum uses separate packages for binaries (rpm) or sources (srpm), but I have never seen yum build packages from source. > I think it's an artifact of tahoe being in python that there is this focus on setup scripts automatically fetching and installing dependencies. In general, step one of building something from source is to read the dependency list and make sure those are installed. So I think the focus should be on having tahoe available via packaging systems, rather than out-of-system dependency management. (I've done this for pkgsrc, and thus am mostly ignoring the setuptools kerfluffle.) There are a couple of situations where the packaging system doesn't help: for example the mock package (needed by tahoe) that comes with current fedora, is higher version (1.1.32) from the one needed by tahoe (0.8.0).

zooko commented

2013-08-30 21:25:09 +00:00

Folks, instead of fixing it so that users don't expect automatic dependency-resolution from python setup.py install, we're going to fix it so that users get what they expect. That is the subject of ticket #1494. I'm closing this ticket as wontfix. If you want to push forward some other issues, for example things like "disallow automatic download of dependencies", please find the ticket about that and add a comment on that ticket.

Folks, instead of fixing it so that users don't expect automatic dependency-resolution from `python setup.py install`, we're going to fix it so that users get what they expect. That is the subject of ticket #1494. I'm closing this ticket as wontfix. If you want to push forward some other issues, for example things like "disallow automatic download of dependencies", please find the ticket about that and add a comment on that ticket.

zooko added the

r/wontfix

label 2013-08-30 21:25:09 +00:00

zooko closed this issue

2013-08-30 21:25:09 +00:00

daira commented

2013-08-31 12:03:06 +00:00

Replying to zooko:

Folks, instead of fixing it so that users don't expect automatic dependency-resolution from python setup.py install, we're going to fix it so that users get what they expect.

+1. The only reason this ticket was filed was that I was under the impression that not installing dependencies was standard behaviour for setup.py install, but apparently it's a regression in zetuptoolz.

That is the subject of ticket #1494.

Indeed.

If you want to push forward some other issues, for example things like "disallow automatic download of dependencies", please find the ticket about that and add a comment on that ticket.

That's ticket #1220.

Replying to [zooko](/tahoe-lafs/trac/issues/2063#issuecomment-394959): > Folks, instead of fixing it so that users don't expect automatic dependency-resolution from `python setup.py install`, we're going to fix it so that users get what they expect. +1. The only reason this ticket was filed was that I was under the impression that not installing dependencies was standard behaviour for ``setup.py install``, but apparently it's a regression in zetuptoolz. > That is the subject of ticket #1494. Indeed. > If you want to push forward some other issues, for example things like "disallow automatic download of dependencies", please find the ticket about that and add a comment on that ticket. That's ticket #1220.

Rows
Columns