consider using setuptools for tahoe #15

New issue

Closed

opened 2007-04-27 23:47:58 +00:00 by zooko · 7 comments

zooko commented

2007-04-27 23:47:58 +00:00

On the other hand, the GNUmakefile already works fine and it would seem like a lot of work to rewrite it entirely in terms of distutils/setuptools.

Maybe we can have both -- keep GNUmakefile for the build rules, but use setup.py for the user interface and distutils/setuptools for the distribution.

It might be nice to make "./setup.py" instead of "make" be the user-facing front end for building, installing, testing, and distributing. It might also be nice to take advantage of distutils/setuptools's many features such as automatically building tarballs and uploading them to the Cheeseshop (PyPI). On the other hand, the GNUmakefile already works fine and it would seem like a lot of work to rewrite it entirely in terms of distutils/setuptools. Maybe we can have both -- keep GNUmakefile for the build rules, but use setup.py for the user interface and distutils/setuptools for the distribution.

zooko added the

p/major

t/enhancement

labels 2007-04-27 23:47:58 +00:00

warner added the

c/packaging

label 2007-04-28 19:16:08 +00:00

zooko added

p/minor

and removed

p/major

labels 2007-04-30 05:43:43 +00:00

warner commented

2007-05-03 17:42:06 +00:00

Just some random thoughts:

I tend to throw in Makefiles in my python projects to remind me of what the other incantations are. The GNUMakefile in tahoe has grown much larger than any of those.

Part of the issue is that it does a number of things that are outside the scope of distutils/setuptools. figleaf conversion, building multiple independent sub-projects, creating/running nodes, pyflakes/development-utilities, and debian packaging are all things that setup.py does not traditionally know much about. Hence I'm not sure whether these things are best put into the rather non-pythonic Makefile or the not-quite-prepared-for-the-task setup.py .

I'm uncertain about setuptools in general: most of the other projects I've seen it in have been a hassle to deal with. Now that I know about --single-version-externally-managed that might be less of an issue than it was before. I'm still inclined to want a list of beneficial features before submitting our uses to the same sort of hassle that I remember dealing with.

Just some random thoughts: I tend to throw in Makefiles in my python projects to remind me of what the other incantations are. The GNUMakefile in tahoe has grown much larger than any of those. Part of the issue is that it does a number of things that are outside the scope of distutils/setuptools. figleaf conversion, building multiple independent sub-projects, creating/running nodes, pyflakes/development-utilities, and debian packaging are all things that setup.py does not traditionally know much about. Hence I'm not sure whether these things are best put into the rather non-pythonic Makefile or the not-quite-prepared-for-the-task setup.py . I'm uncertain about setuptools in general: most of the other projects I've seen it in have been a hassle to deal with. Now that I know about --single-version-externally-managed that might be less of an issue than it was before. I'm still inclined to want a list of beneficial features before submitting our uses to the same sort of hassle that I remember dealing with.

zooko commented

2007-05-22 15:57:58 +00:00

Author

I took a stab at setuptoolifying our setup.py, the results of which are in this branch. I haven't gotten "./setup.py test" to work, and I haven't yet moved functionality out of make and into python, so the benefits of this stab at it are only:

integration with cheeseshop metadata
easy/happy installation for people who like setuptools

However, this stab at it has at least this major drawback:

unhappiness for people who don't like setuptools (e.g. GNU stow users)

And therefore, it would be a big lose to switch to it at this time.

For the record, if I could somehow get past that fatal flaw -- possibly by offering two installation interfaces (make install and ./setup.py install), or by prominently documenting the --single-version-externally-managed switch -- and if I did the necessary work to get the other features working, then this would offer:

./setup.py test, which will hopefully in the future be the standard way to run tests. In the meantime we currently have the old standard way to run tests: make test
reduce the number of build/install mechanisms from two (makefile and setup.py) to one
automated production of tarballs and automated upload of same to cheeseshop
dependency management for people who don't have apt (ideally also producing debian packages with some tool like easy-deb)
more features of setuptools that I'm currently not thinking of
more features of setuptools that will be invented by the setuptools-loving community in the future

Having thought this through and talked about it quite a bit with Brian, I've realized that my over-riding motivation for being interested in setuptools is:

easy/happy installation for people who like setuptools

And an even bigger over-riding consideration is:

unhappiness for people who don't like setuptools (e.g. GNU stow users)

The latter category of people will never go away, and I will probably never become willing to do something that discourages that category in order to encourage the former category, so at this point I predict that tahoe will never use setuptools until someone figures out a way to make it more palatable to users of traditional unix-like package management tools.

I took a stab at setuptoolifying our setup.py, the results of which are in [this branch](http://allmydata.org/source/tahoe/branches/setuptools/). I haven't gotten "./setup.py test" to work, and I haven't yet moved functionality out of make and into python, so the benefits of this stab at it are only: * integration with cheeseshop metadata * easy/happy installation for people who like setuptools However, this stab at it has at least this major drawback: * unhappiness for people who don't like setuptools (e.g. `GNU stow` users) And therefore, it would be a big lose to switch to it at this time. For the record, if I could somehow get past that fatal flaw -- possibly by offering two installation interfaces (`make install` and `./setup.py install`), or by prominently documenting the `--single-version-externally-managed` switch -- and if I did the necessary work to get the other features working, then this would offer: * `./setup.py test`, which will hopefully in the future be the standard way to run tests. In the meantime we currently have the old standard way to run tests: `make test` * reduce the number of build/install mechanisms from two (makefile and setup.py) to one * automated production of tarballs and automated upload of same to cheeseshop * dependency management for people who don't have apt (ideally also producing debian packages with some tool like `easy-deb`) * more features of setuptools that I'm currently not thinking of * more features of setuptools that will be invented by the setuptools-loving community in the future Having thought this through and talked about it quite a bit with Brian, I've realized that my over-riding motivation for being interested in setuptools is: * easy/happy installation for people who like setuptools And an even bigger over-riding consideration is: * unhappiness for people who don't like setuptools (e.g. `GNU stow` users) The latter category of people will never go away, and I will probably never become willing to do something that discourages that category in order to encourage the former category, so at this point I predict that tahoe will never use setuptools until someone figures out a way to make it more palatable to users of traditional unix-like package management tools.

zooko commented

2007-06-04 18:16:38 +00:00

Author

(http://blog.ianbicking.org/pythons-makefile.html)

Re: the setup.py vs. makefile discussion

(http://blog.ianbicking.org/pythons-makefile.html) Re: the setup.py vs. makefile discussion

zooko commented

2007-07-10 18:47:18 +00:00

Author

So now we want to use another 3rd party package -- Bob Ippolito's json parser -- and here are our desiderata for using a package:

We can produce a Tahoe binary package for a given deployment target, ship that package to a user using that platform, they can install that package, and it will work. That is: user doesn't have to manually satisfy any dependencies.
Use the source code as it is written by upstream. That is: no patching required.
Use it as it is packaged by upstream. That is: we prefer to get a copy of the source code of the package as upstream prefers to distribute such source code, rather than by pulling from their revision control tool or so on.
Don't have name conflicts with existing packages on deployment target (either use extant one if any or else use our bundled one). That is: the user doesn't have to manually resolve any conflicts.
Make it convenient for someone to use other versions of the packages that we use e.g. system-wide packages or newer or alternate versions, etc.

So now we want to use another 3rd party package -- Bob Ippolito's json parser -- and here are our desiderata for using a package: 1. We can produce a Tahoe binary package for a given deployment target, ship that package to a user using that platform, they can install that package, and it will work. That is: user doesn't have to manually satisfy any dependencies. 2. Use the source code as it is written by upstream. That is: no patching required. 3. Use it as it is packaged by upstream. That is: we prefer to get a copy of the source code of the package as upstream prefers to distribute such source code, rather than by pulling from their revision control tool or so on. 4. Don't have name conflicts with existing packages on deployment target (either use extant one if any or else use our bundled one). That is: the user doesn't have to manually resolve any conflicts. 5. Make it convenient for someone to use other versions of the packages that we use e.g. system-wide packages or newer or alternate versions, etc.

zooko commented

2007-07-10 23:23:57 +00:00

Author

Things I Need To Do To Make Usage of setuptools Okay:

Bundle setuptools so that ez_setup.py never connects to the network.
Bundle dependent libs (in src tarball form) so that setup.py never connects to the network.
Hack ez_setup.py so that it accepts a "minimum version" of setuptools (RobK did this: changeset:03e9b0d4d77ee0ae).

Things I Need To Do To Make Usage of setuptools Okay: * Bundle setuptools so that ez_setup.py never connects to the network. * Bundle dependent libs (in src tarball form) so that setup.py never connects to the network. * Hack ez_setup.py so that it accepts a "minimum version" of setuptools (RobK did this: changeset:03e9b0d4d77ee0ae).