setuptools/easy_install/pip-installed package lacks documentation #1659

New issue

Open

opened 2012-01-18 03:23:59 +00:00 by nejucomo · 9 comments

nejucomo commented

2012-01-18 03:23:59 +00:00

I am a happy user of the pip installability feature, whereby I run "pip install allmydata-tahoe" to get a running tahoe installation.

This installation lacks documentation, at least where I sometimes find other python documentation in $VIRTUALENV_ROOT/share/doc/$PACKAGE_NAME.

Request: Fix the pypi distribution to bundle all documentation such that pip-installed tahoe provides documentation in this location.

If documentation is installed elsewhere, ensure that it is compatible with virtualenv. Specifically I should be able to install different versions of tahoe-lafs in different virtualenvs and have the correct copy of documentation for each install.

I am a happy user of the pip installability feature, whereby I run "pip install allmydata-tahoe" to get a running tahoe installation. This installation lacks documentation, at least where I sometimes find other python documentation in $VIRTUALENV_ROOT/share/doc/$PACKAGE_NAME. Request: Fix the pypi distribution to bundle all documentation such that pip-installed tahoe provides documentation in this location. If documentation is installed elsewhere, ensure that it is compatible with virtualenv. Specifically I should be able to install different versions of tahoe-lafs in different virtualenvs and have the correct copy of documentation for each install.

nejucomo added the

labels 2012-01-18 03:23:59 +00:00

nejucomo added this to the undecided milestone 2012-01-18 03:23:59 +00:00

nejucomo commented

2013-07-20 18:57:45 +00:00

Author

Note that the title of this ticket is misleading. This has nothing to do with pip. I added the pypi keyword, but this also is not specific to Tahoe-LAFS's PyPI page.

I just verified that if I download the pypi tarball or the self-hosted zip file, and then run:

$ python ./setup.py build
$ python ./setup.py install

-there are no doc files installed outside of the tarball/zipfile contents.

Note that the title of this ticket is misleading. This has nothing to do with `pip`. I added the `pypi` keyword, but this also is not specific to Tahoe-LAFS's [PyPI page](https://pypi.python.org/pypi/allmydata-tahoe/1.10.0). I just verified that if I download the [pypi tarball](https://pypi.python.org/pypi/allmydata-tahoe/1.10.0) or the [self-hosted zip file](https://tahoe-lafs.org/source/tahoe-lafs/releases/allmydata-tahoe-1.10.0.zip), and then run: ``` $ python ./setup.py build $ python ./setup.py install ``` -there are no doc files installed outside of the tarball/zipfile contents.

nejucomo commented

2013-07-20 19:01:55 +00:00

Author

I just added the proposed enhancement #2031 which is to serve ./docs from the web gateway. That would necessarily solve this issue by bundling all of ./docs as a package resource directory.

I just added the proposed enhancement #2031 which is to serve `./docs` from the web gateway. That would necessarily solve this issue by bundling all of `./docs` as a package resource directory.

daira changed title from ~~pip installed-package lacks documentation.~~ to setuptools/easy_install/pip-installed package lacks documentation

2013-07-22 23:02:40 +00:00

nejucomo commented

2013-07-26 05:48:49 +00:00

Author

I believe the summary should actually be changed to "packages installed from an extracted sdist tarball do not install documentation", but I haven't tested that thoroughly enough yet.

nejucomo commented

2013-07-26 05:54:42 +00:00

Author

I just created this change to setup.py which causes all of ./docs and some other useful files like README.txt to be installed in $PREFIX/share/doc/allmydata_tahoe when installed with pip.

Next I'd like to test that this is also the case when I run python ./setup.py install --single-version-externally-managed from an unpacked sdist.

If that also works, then I would propose to merge that branch.

There are a few sticking points in that branch:

It replicates some information from MANIFEST.in. (I found while doing this that MANIFEST.in refers to non-existent paths, and that python ./setup.py sdist does not complain about these missing files, so it has accumulated some bitrot.)
It may use the wrong path format for all platforms... I suspect it should use distutils.util.convert_path() somewhere.
There's no test code. Testing build/install scaffolding is difficult!

I just created [this change to setup.py](https://github.com/nejucomo/tahoe-lafs/compare/1659-bundle-docs-in-sdist) which causes all of `./docs` and some other useful files like `README.txt` to be installed in `$PREFIX/share/doc/allmydata_tahoe` when installed with pip. Next I'd like to test that this is also the case when I run `python ./setup.py install --single-version-externally-managed` from an unpacked sdist. If that also works, then I would propose to merge that branch. There are a few sticking points in that branch: * It replicates some information from `MANIFEST.in`. (I found while doing this that `MANIFEST.in` refers to non-existent paths, and that `python ./setup.py sdist` does not complain about these missing files, so it has accumulated some bitrot.) * It may use the wrong path format for all platforms... I suspect it should use `distutils.util.convert_path()` somewhere. * There's no test code. Testing build/install scaffolding is difficult!

nejucomo commented

2013-07-26 06:18:16 +00:00

Author

Here's a demo of the 1659-bundle-docs-in-sdist branch using pip to install from a git repository:

 $ git rev-parse --abbrev-ref HEAD ; git rev-parse HEAD
1659-bundle-docs-in-sdist
b0ca93b732ba47cdd14d9b70996525c7cc8c85a1


$ TMP="$(mktemp --tmpdir -d 1659-bundle-docs-in-sdist.demo-XXXXXX)"


$ virtualenv "$TMP"
Using real prefix '/usr/local/bin/../stow/Python-2.7.3-setuptools-0.6c11'
New python executable in /home/n/tmp/1659-bundle-docs-in-sdist.demo-I4S29A/bin/python
Installing setuptools...............done.
Installing pip.................done.


$ source "$TMP/bin/activate"
(1659-bundle-docs-in-sdist.demo-I4S29A)

$ pip install twisted > $TMP/pip-install-twisted.log # necessary work around; see https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2032
(1659-bundle-docs-in-sdist.demo-I4S29A)

$ pip install . > $TMP/pip-install-dot.log
(1659-bundle-docs-in-sdist.demo-I4S29A)

$ cd "$VIRTUAL_ENV"
(1659-bundle-docs-in-sdist.demo-I4S29A)

$ ls ./share/doc/allmydata-tahoe/
COPYING.GPL  COPYING.TGPPL.rst  CREDITS  docs  NEWS.rst  README.txt  relnotes.txt
(1659-bundle-docs-in-sdist.demo-I4S29A)

$ find share/doc/ -type d
share/doc/
share/doc/zfec
share/doc/pyutil
share/doc/pycryptopp
share/doc/allmydata-tahoe
share/doc/allmydata-tahoe/docs
share/doc/allmydata-tahoe/docs/historical
share/doc/allmydata-tahoe/docs/frontends
share/doc/allmydata-tahoe/docs/proposed
share/doc/allmydata-tahoe/docs/specifications
share/doc/allmydata-tahoe/docs/specifications/backends
share/doc/allmydata-tahoe/docs/man
(1659-bundle-docs-in-sdist.demo-I4S29A)

Here's a demo of the [1659-bundle-docs-in-sdist](https://github.com/nejucomo/tahoe-lafs/tree/1659-bundle-docs-in-sdist) branch using `pip` to install from a `git` repository: ``` $ git rev-parse --abbrev-ref HEAD ; git rev-parse HEAD 1659-bundle-docs-in-sdist b0ca93b732ba47cdd14d9b70996525c7cc8c85a1 $ TMP="$(mktemp --tmpdir -d 1659-bundle-docs-in-sdist.demo-XXXXXX)" $ virtualenv "$TMP" Using real prefix '/usr/local/bin/../stow/Python-2.7.3-setuptools-0.6c11' New python executable in /home/n/tmp/1659-bundle-docs-in-sdist.demo-I4S29A/bin/python Installing setuptools...............done. Installing pip.................done. $ source "$TMP/bin/activate" (1659-bundle-docs-in-sdist.demo-I4S29A) $ pip install twisted > $TMP/pip-install-twisted.log # necessary work around; see https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2032 (1659-bundle-docs-in-sdist.demo-I4S29A) $ pip install . > $TMP/pip-install-dot.log (1659-bundle-docs-in-sdist.demo-I4S29A) $ cd "$VIRTUAL_ENV" (1659-bundle-docs-in-sdist.demo-I4S29A) $ ls ./share/doc/allmydata-tahoe/ COPYING.GPL COPYING.TGPPL.rst CREDITS docs NEWS.rst README.txt relnotes.txt (1659-bundle-docs-in-sdist.demo-I4S29A) $ find share/doc/ -type d share/doc/ share/doc/zfec share/doc/pyutil share/doc/pycryptopp share/doc/allmydata-tahoe share/doc/allmydata-tahoe/docs share/doc/allmydata-tahoe/docs/historical share/doc/allmydata-tahoe/docs/frontends share/doc/allmydata-tahoe/docs/proposed share/doc/allmydata-tahoe/docs/specifications share/doc/allmydata-tahoe/docs/specifications/backends share/doc/allmydata-tahoe/docs/man (1659-bundle-docs-in-sdist.demo-I4S29A) ```

nejucomo commented

2013-07-26 06:36:15 +00:00

Author

Here is a demo which does not use pip nor virtualenv:

$ git rev-parse --abbrev-ref HEAD ; git rev-parse HEAD
1659-bundle-docs-in-sdist
b0ca93b732ba47cdd14d9b70996525c7cc8c85a1

$ TMP="$(mktemp --tmpdir -d lafs-install-no-virtual-env-or-pip.demo-XXXXXX)"

$ python ./setup.py sdist > "$TMP/sdist.log"
warning: no previously-included files matching '*~' found anywhere in distribution

$ tar -xf ./dist/allmydata-tahoe-1.10.0.post32.tar.gz -C /usr/local/src

$ cd /usr/local/src/allmydata-tahoe-1.10.0.post32/

$ python ./setup.py install --single-version-externally-managed --record "$TMP/install-record" --prefix='/usr/local' > "$TMP/install.log"
warning: no previously-included files matching '*~' found anywhere in distribution

$ find /usr/local/share/doc/allmydata-tahoe/ -type d
/usr/local/share/doc/allmydata-tahoe/
/usr/local/share/doc/allmydata-tahoe/docs
/usr/local/share/doc/allmydata-tahoe/docs/historical
/usr/local/share/doc/allmydata-tahoe/docs/frontends
/usr/local/share/doc/allmydata-tahoe/docs/proposed
/usr/local/share/doc/allmydata-tahoe/docs/specifications
/usr/local/share/doc/allmydata-tahoe/docs/specifications/backends
/usr/local/share/doc/allmydata-tahoe/docs/man

Here is a demo which does not use `pip` nor `virtualenv`: ``` $ git rev-parse --abbrev-ref HEAD ; git rev-parse HEAD 1659-bundle-docs-in-sdist b0ca93b732ba47cdd14d9b70996525c7cc8c85a1 $ TMP="$(mktemp --tmpdir -d lafs-install-no-virtual-env-or-pip.demo-XXXXXX)" $ python ./setup.py sdist > "$TMP/sdist.log" warning: no previously-included files matching '*~' found anywhere in distribution $ tar -xf ./dist/allmydata-tahoe-1.10.0.post32.tar.gz -C /usr/local/src $ cd /usr/local/src/allmydata-tahoe-1.10.0.post32/ $ python ./setup.py install --single-version-externally-managed --record "$TMP/install-record" --prefix='/usr/local' > "$TMP/install.log" warning: no previously-included files matching '*~' found anywhere in distribution $ find /usr/local/share/doc/allmydata-tahoe/ -type d /usr/local/share/doc/allmydata-tahoe/ /usr/local/share/doc/allmydata-tahoe/docs /usr/local/share/doc/allmydata-tahoe/docs/historical /usr/local/share/doc/allmydata-tahoe/docs/frontends /usr/local/share/doc/allmydata-tahoe/docs/proposed /usr/local/share/doc/allmydata-tahoe/docs/specifications /usr/local/share/doc/allmydata-tahoe/docs/specifications/backends /usr/local/share/doc/allmydata-tahoe/docs/man ```

zooko commented

2013-07-26 14:08:21 +00:00

Replying to nejucomo:

It may use the wrong path format for all platforms... I suspect it should use distutils.util.convert_path() somewhere.

There's no test code. Testing build/install scaffolding is difficult!

We have a (kludgey) way to test such things. Here is the output on buildbot:

A test of ['python', '-c', "import glob, os, subprocess, sys; os.mkdir('egginstalldir'); tahoe_egg = glob.glob(os.path.join('dist', '*.egg'))[0]; sys.exit(subprocess.call(['easy_install', '-d', 'egginstalldir', tahoe_egg]))"]:

https://tahoe-lafs.org/buildbot-tahoe-lafs/builders/Kyle%20OpenBSD%20amd64/builds/235/steps/install-to-egg/logs/stdio

A test of ['python', '-c', "import subprocess, sys;sys.exit(subprocess.call(['setup.py', 'install', '--single-version-externally-managed', '--record=record.txt', '--prefix', 'prefixinstalldir']sys.executable,))"]:

https://tahoe-lafs.org/buildbot-tahoe-lafs/builders/Kyle%20OpenBSD%20amd64/builds/235/steps/install-to-prefix/logs/stdio

In both cases, they are followed by executing the unit tests from within the installed location:

https://tahoe-lafs.org/buildbot-tahoe-lafs/builders/Kyle%20OpenBSD%20amd64/builds/235/steps/test-from-egg

and

https://tahoe-lafs.org/buildbot-tahoe-lafs/builders/Kyle%20OpenBSD%20amd64/builds/235/steps/test-from-prefixdir

So to test that the docs are installed correctly, I guess we could add another Buildbot BuildStep which runs after the install and before the unit-tests-run-from-installed, which just asserts that the right files are in place.

Nathan: interested in contributing such a test?

Replying to [nejucomo](/tahoe-lafs/trac/issues/1659#issuecomment-388440): > > * It may use the wrong path format for all platforms... I suspect it should use `distutils.util.convert_path()` somewhere. > * There's no test code. Testing build/install scaffolding is difficult! We have a (kludgey) way to test such things. Here is the output on buildbot: A test of `['python', '-c', "import glob, os, subprocess, sys; os.mkdir('egginstalldir'); tahoe_egg = glob.glob(os.path.join('dist', '*.egg'))[0]; sys.exit(subprocess.call(['easy_install', '-d', 'egginstalldir', tahoe_egg]))"]`: <https://tahoe-lafs.org/buildbot-tahoe-lafs/builders/Kyle%20OpenBSD%20amd64/builds/235/steps/install-to-egg/logs/stdio> A test of `['python', '-c', "import subprocess, sys;sys.exit(subprocess.call(['setup.py', 'install', '--single-version-externally-managed', '--record=record.txt', '--prefix', 'prefixinstalldir']sys.executable,))"]`: <https://tahoe-lafs.org/buildbot-tahoe-lafs/builders/Kyle%20OpenBSD%20amd64/builds/235/steps/install-to-prefix/logs/stdio> In both cases, they are followed by executing the unit tests from within the installed location: <https://tahoe-lafs.org/buildbot-tahoe-lafs/builders/Kyle%20OpenBSD%20amd64/builds/235/steps/test-from-egg> and <https://tahoe-lafs.org/buildbot-tahoe-lafs/builders/Kyle%20OpenBSD%20amd64/builds/235/steps/test-from-prefixdir> So to test that the docs are installed correctly, I guess we could add another Buildbot [BuildStep](wiki/BuildStep) which runs after the install and before the unit-tests-run-from-installed, which just asserts that the right files are in place. Nathan: interested in contributing such a test?