tahoe-lafs/trac

compilation errors with pyOpenSSL >= 0.13 on RHEL5 systems (and clones) #1637

New issue

Closed

opened 2011-12-15 19:21:28 +00:00 by daira · 10 comments

daira commented

2011-12-15 19:21:28 +00:00

Reported by Jimmy Tang:

[jtang@gateway allmydata-tahoe-1.9.0-SUMO]$ python setup.py build
running darcsver
setup.py darcsver: using extant version file ['src/allmydata/_version.py']
running develop
Not found: ../tahoe-deps
running egg_info
writing requirements to src/allmydata_tahoe.egg-info/requires.txt
writing src/allmydata_tahoe.egg-info/PKG-INFO
writing top-level names to src/allmydata_tahoe.egg-info/top_level.txt
writing dependency_links to src/allmydata_tahoe.egg-info/dependency_links.txt
writing entry points to src/allmydata_tahoe.egg-info/entry_points.txt
reading manifest file 'src/allmydata_tahoe.egg-info/SOURCES.txt'
writing manifest file 'src/allmydata_tahoe.egg-info/SOURCES.txt'
running build_ext
Processing setuptools_darcs-1.2.12.egg
removing '/home/jtang/allmydata-tahoe-1.9.0-SUMO/support/lib/python2.4/site-packages/setuptools_darcs-1.2.12.egg'
(and everything under it)
Copying setuptools_darcs-1.2.12.egg to
/home/jtang/allmydata-tahoe-1.9.0-SUMO/support/lib/python2.4/site-packages
setuptools-darcs 1.2.12 is already the active version in easy-install.pth

Installed /home/jtang/allmydata-tahoe-1.9.0-SUMO/support/lib/python2.4/site-packages/setuptools_darcs-1.2.12.egg
Creating /home/jtang/allmydata-tahoe-1.9.0-SUMO/support/lib/python2.4/site-packages/allmydata-tahoe.egg-link
(link to src)
allmydata-tahoe 1.9.0 is already the active version in easy-install.pth
Installing tahoe script to support/bin

Installed /home/jtang/allmydata-tahoe-1.9.0-SUMO/src
Processing dependencies for allmydata-tahoe==1.9.0
Searching for pyOpenSSL
Reading http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-dep-sdists/
Reading http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-eggs/
Best match: pyOpenSSL 0.13
Processing pyOpenSSL-0.13.tar.gz
Running pyOpenSSL-0.13/setup.py -q bdist_egg --dist-dir
/tmp/easy_install-gr--LP/pyOpenSSL-0.13/egg-dist-tmp-YkOeMn
warning: no previously-included files matching '*.pyc' found anywhere
in distribution
OpenSSL/ssl/connection.c: In function 'ssl_Connection_set_context':
OpenSSL/ssl/connection.c:289: warning: implicit declaration of
function 'SSL_set_SSL_CTX'
OpenSSL/ssl/connection.c: In function 'ssl_Connection_get_servername':
OpenSSL/ssl/connection.c:313: error: 'TLSEXT_NAMETYPE_host_name'
undeclared (first use in this function)
OpenSSL/ssl/connection.c:313: error: (Each undeclared identifier is
reported only once
OpenSSL/ssl/connection.c:313: error: for each function it appears in.)
OpenSSL/ssl/connection.c:320: warning: implicit declaration of
function 'SSL_get_servername'
OpenSSL/ssl/connection.c:320: warning: assignment makes pointer from
integer without a cast
OpenSSL/ssl/connection.c: In function 'ssl_Connection_set_tlsext_host_name':
OpenSSL/ssl/connection.c:346: warning: implicit declaration of
function 'SSL_set_tlsext_host_name'
error: Setup script exited with error: command 'gcc' failed with exit status 1

as far as I know it's due to the pyopenssl needing newer features of
openssl -- see

https://bugs.launchpad.net/pyopenssl/+bug/845445

Doing this somewhat makes the problem go away on RHEL5

diff --git a/src/allmydata/_auto_deps.py b/src/allmydata/_auto_deps.py
index bee1b5d..57d82b1 100644
--- a/src/allmydata/_auto_deps.py
+++ b/src/allmydata/_auto_deps.py
@@ -38,14 +38,13 @@ install_requires = [
     # or foolscap switching from pyOpenSSL to a different crypto library, we
     # need to update this declaration here.
     "foolscap >= 0.6.1",
-    "pyOpenSSL",
+    "pyOpenSSL <= 0.11",

the tests all pass after making the change.

Reported by Jimmy Tang: ``` [jtang@gateway allmydata-tahoe-1.9.0-SUMO]$ python setup.py build running darcsver setup.py darcsver: using extant version file ['src/allmydata/_version.py'] running develop Not found: ../tahoe-deps running egg_info writing requirements to src/allmydata_tahoe.egg-info/requires.txt writing src/allmydata_tahoe.egg-info/PKG-INFO writing top-level names to src/allmydata_tahoe.egg-info/top_level.txt writing dependency_links to src/allmydata_tahoe.egg-info/dependency_links.txt writing entry points to src/allmydata_tahoe.egg-info/entry_points.txt reading manifest file 'src/allmydata_tahoe.egg-info/SOURCES.txt' writing manifest file 'src/allmydata_tahoe.egg-info/SOURCES.txt' running build_ext Processing setuptools_darcs-1.2.12.egg removing '/home/jtang/allmydata-tahoe-1.9.0-SUMO/support/lib/python2.4/site-packages/setuptools_darcs-1.2.12.egg' (and everything under it) Copying setuptools_darcs-1.2.12.egg to /home/jtang/allmydata-tahoe-1.9.0-SUMO/support/lib/python2.4/site-packages setuptools-darcs 1.2.12 is already the active version in easy-install.pth Installed /home/jtang/allmydata-tahoe-1.9.0-SUMO/support/lib/python2.4/site-packages/setuptools_darcs-1.2.12.egg Creating /home/jtang/allmydata-tahoe-1.9.0-SUMO/support/lib/python2.4/site-packages/allmydata-tahoe.egg-link (link to src) allmydata-tahoe 1.9.0 is already the active version in easy-install.pth Installing tahoe script to support/bin Installed /home/jtang/allmydata-tahoe-1.9.0-SUMO/src Processing dependencies for allmydata-tahoe==1.9.0 Searching for pyOpenSSL Reading http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-dep-sdists/ Reading http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-eggs/ Best match: pyOpenSSL 0.13 Processing pyOpenSSL-0.13.tar.gz Running pyOpenSSL-0.13/setup.py -q bdist_egg --dist-dir /tmp/easy_install-gr--LP/pyOpenSSL-0.13/egg-dist-tmp-YkOeMn warning: no previously-included files matching '*.pyc' found anywhere in distribution OpenSSL/ssl/connection.c: In function 'ssl_Connection_set_context': OpenSSL/ssl/connection.c:289: warning: implicit declaration of function 'SSL_set_SSL_CTX' OpenSSL/ssl/connection.c: In function 'ssl_Connection_get_servername': OpenSSL/ssl/connection.c:313: error: 'TLSEXT_NAMETYPE_host_name' undeclared (first use in this function) OpenSSL/ssl/connection.c:313: error: (Each undeclared identifier is reported only once OpenSSL/ssl/connection.c:313: error: for each function it appears in.) OpenSSL/ssl/connection.c:320: warning: implicit declaration of function 'SSL_get_servername' OpenSSL/ssl/connection.c:320: warning: assignment makes pointer from integer without a cast OpenSSL/ssl/connection.c: In function 'ssl_Connection_set_tlsext_host_name': OpenSSL/ssl/connection.c:346: warning: implicit declaration of function 'SSL_set_tlsext_host_name' error: Setup script exited with error: command 'gcc' failed with exit status 1 ``` as far as I know it's due to the pyopenssl needing newer features of openssl -- see <https://bugs.launchpad.net/pyopenssl/+bug/845445> Doing this somewhat makes the problem go away on RHEL5 ``` diff --git a/src/allmydata/_auto_deps.py b/src/allmydata/_auto_deps.py index bee1b5d..57d82b1 100644 --- a/src/allmydata/_auto_deps.py +++ b/src/allmydata/_auto_deps.py @@ -38,14 +38,13 @@ install_requires = [ # or foolscap switching from pyOpenSSL to a different crypto library, we # need to update this declaration here. "foolscap >= 0.6.1", - "pyOpenSSL", + "pyOpenSSL <= 0.11", ``` the tests all pass after making the change.

daira added the

labels 2011-12-15 19:21:28 +00:00

daira added this to the undecided milestone 2011-12-15 19:21:28 +00:00

daira added

c/packaging

and removed

c/unknown

labels 2011-12-15 19:23:03 +00:00

daira commented

2011-12-15 19:27:48 +00:00

Author

On 15/12/11 13:51, Jimmy Tang wrote:

Is there a compelling reason for using newer versions of pyopenssl ?

No, but we dislike having to exclude newer versions of dependencies; it
means that security updates don't get picked up, for example.

On 15/12/11 13:51, Jimmy Tang wrote: > Is there a compelling reason for using newer versions of pyopenssl ? No, but we dislike having to exclude newer versions of dependencies; it means that security updates don't get picked up, for example.

daira changed title from ~~compilation errors with pyOpenSSL > 0.11 on RHEL5 systems (and clones)~~ to compilation errors with pyOpenSSL >= 0.13 on RHEL5 systems (and clones)

2011-12-15 19:32:28 +00:00

zooko commented

2011-12-15 19:59:39 +00:00

One solution to this for Tahoe-LAFS would be to build an egg to put in the pyOpenSSL column of the Python 2.4 rows of [//source/tahoe-lafs/deps/tahoe-lafs-dep-eggs/README.html https://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-eggs/README.html]. If someone with Python 2.4 linux could provide us with such an egg, that would probably fix this problem. (By having the build system detect and download that egg instead of attempting to compile pyOpenSSL from source.)

One solution to this for Tahoe-LAFS would be to build an egg to put in the `pyOpenSSL` column of the Python 2.4 rows of [//source/tahoe-lafs/deps/tahoe-lafs-dep-eggs/README.html <https://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-eggs/README.html>]. If someone with Python 2.4 linux could provide us with such an egg, that would probably fix this problem. (By having the build system detect and download that egg instead of attempting to compile pyOpenSSL from source.)

zooko commented

2011-12-29 00:23:02 +00:00

This is not a critical problem, but if there is an easy way to fix it for Tahoe-LAFS v1.9.1, I would be willing to do a little work on that. The next step is we need a volunteer who loves and/or operates RHEL to provide us with a precompiled egg and to promise us that he didn't insert any backdoors or allow anyone else to do so.

zooko modified the milestone from undecided to 1.9.1

2011-12-29 00:23:02 +00:00

zooko commented

2011-12-29 07:39:14 +00:00

Jimmy: would you be willing to build a binary package with Python 2.4 on your platform and give it to us?

jcftang commented

2011-12-29 09:44:31 +00:00

Owner

just note that my version of openssl is installed in a none-standard location, so the egg might not behave too well if any new functionality is used that requires the newer features of the updated openssl installation.

The above solution doesn't sound like a runner to me, I think trying to convince the pyopenssl people that rhel5 is still a valid and supported platform is a better way of getting fixed, but this isn't clear to me if rhel5 is supported or not on for pyopenssl 0.13

edit: rhel 5.8 recently got released, rhel5 will be around till at least 2014, I would expect people to migrate away from it en-masse in 2013.

Zooko: If you are happy with me submitting a pyopenssl 0.13 egg which links to a newer version of openssl than the stock version in rhel5 and the stock python 2.4 then I can cook one up later on in the day and upload it somewhere, just point me to some instructions and documentation on how you would like it done. just note that my version of openssl is installed in a none-standard location, so the egg might not behave too well if any new functionality is used that requires the newer features of the updated openssl installation. The above solution doesn't sound like a runner to me, I think trying to convince the pyopenssl people that rhel5 is still a valid and supported platform is a better way of getting fixed, but this isn't clear to me if rhel5 is supported or not on for pyopenssl 0.13 edit: rhel 5.8 recently got released, rhel5 will be around till at least 2014, I would expect people to migrate away from it en-masse in 2013.

zooko commented

2011-12-30 06:08:56 +00:00

jcftang: it appears to me that when you build a pyOpenSSL egg then it includes a copy of libcrypto.so in the resulting .egg, so that egg will work on a platform even if that version of libcrypto.so is not the standard one for that platform. Please try it! Let's see. :-)

I'm not sure what you mean about the pyOpenSSL people "supporting" RHEL 5. Jean-Paul "exarkun" Calderone is the the most active contributor to pyOpenSSL, and he said something on pyOpenSSL ticket 845445 that made it sound like he was willing to work on this, but lacked "a good environment set up to be able to easily test against specific versions of OpenSSL". Maybe we could offer to help him set up such an environment?

Also helping them build automation that produces .eggs automatically on different platforms would certainly help: pyOpenSSL ticket 668996.

jcftang: it appears to me that when you build a pyOpenSSL egg then it includes a copy of `libcrypto.so` in the resulting .egg, so that egg will work on a platform even if that version of `libcrypto.so` is not the standard one for that platform. Please try it! Let's see. :-) I'm not sure what you mean about the pyOpenSSL people "supporting" RHEL 5. Jean-Paul "exarkun" Calderone is the the most active contributor to pyOpenSSL, and he said something on [pyOpenSSL ticket 845445](https://bugs.launchpad.net/pyopenssl/+bug/845445) that made it sound like he was willing to work on this, but lacked "a good environment set up to be able to easily test against specific versions of OpenSSL". Maybe we could offer to help him set up such an environment? Also helping them build automation that produces .eggs automatically on different platforms would certainly help: [pyOpenSSL ticket 668996](https://bugs.launchpad.net/pyopenssl/+bug/668996).

jcftang commented

2011-12-30 11:32:48 +00:00

Owner

Attachment pyOpenSSL-0.13-py2.4-linux-x86_64.egg (307070 bytes) added

**Attachment** pyOpenSSL-0.13-py2.4-linux-x86_64.egg (307070 bytes) added

pyOpenSSL-0.13-py2.4-linux-x86_64.egg

300 KiB

jcftang commented

2011-12-30 11:45:05 +00:00

Owner

Also apologies for not reading ticket pyopenssl Bug #845445 more thoroughly, it looks like they see the problem which is good :).

Under linux I've been using a home brew build system (which allows multiple versions of software to be installed, for a cluster compute environment) but it isn't very mature, it may be of interest. This is how I install alternate versions of openssl, python, whatever for testing and deployment. This may be better discussed on the mailing list if there is interest.

I might be able to help setup something to automatically build some eggs for submission, but I would need to check for permission with the guys in work first before I can commit to anything.

zooko: I did not realise that building python eggs included a compiled binary blob, anyway as requested, attached to this ticket is an egg for rhel5.7 (SL5.7 to be exact), it was built using openssl 0.98o. Please test! Also apologies for not reading ticket pyopenssl Bug #845445 more thoroughly, it looks like they see the problem which is good :). Under linux I've been using a home brew build system (which allows multiple versions of software to be installed, for a cluster compute environment) but it isn't very mature, it may be of interest. This is how I install alternate versions of openssl, python, whatever for testing and deployment. This may be better discussed on the mailing list if there is interest. I might be able to help setup something to automatically build some eggs for submission, but I would need to check for permission with the guys in work first before I can commit to anything.

daira commented

2012-02-11 00:15:03 +00:00

Author

#1671 was a duplicate. The problem also happens on CentOS (which is a Redhat derivative).

daira commented

2012-03-31 23:23:22 +00:00

Author

This is pyOpenSSL's bug rather than our bug, therefore, it does not need to be in milestone 1.9.2. I'm leaving this open rather than marking it invalid just because we will need to make sure that an egg of the fixed pyOpenSSL is available.