tahoe-lafs/trac

I don't like pyopenssl #11

New issue

Closed

opened 2007-04-27 04:47:31 +00:00 by zooko · 34 comments

zooko commented

2007-04-27 04:47:31 +00:00

http://allmydata.org/source/pyOpenSSL-0.6.win32-py2.5

Which I copied from webcleaner.sf.net.

I wouldn't know how to rebuild this if we needed to, for example if someone discovered a critical security hole in it.

Also, pyopenssl's web site hasn't been updated since 2004.

Also, it apparently uses lots of memory for secure connections, although Brian might want to add more accurate and precise notes about that issue.

All in all, I would be really happy to find a well maintained, easy to compile alternative.

There is no known way (to me) way to get pyopenssl running on Windows without finding a binary that somebody has uploaded somewhere (i.e., not an actual supported package on an official project's web site or anything). For example, right now the best way for our Windows users to get pyOpenSSL is to get this binary: <http://allmydata.org/source/pyOpenSSL-0.6.win32-py2.5> Which I copied from webcleaner.sf.net. I wouldn't know how to rebuild this if we needed to, for example if someone discovered a critical security hole in it. Also, pyopenssl's web site hasn't been updated since 2004. Also, it apparently uses lots of memory for secure connections, although Brian might want to add more accurate and precise notes about that issue. All in all, I would be really happy to find a well maintained, easy to compile alternative.

zooko added the

p/major

t/defect

labels 2007-04-27 04:47:31 +00:00

warner added the

c/unknown

label 2007-04-28 19:18:23 +00:00

zooko commented

2007-04-29 16:30:14 +00:00

Author

It has a very nice straightforward explanation up front:

http://cspace.in/

I was thinking that it might be a nice optional underlay protocol for Foolscap.

Unfortunately it is GPL'ed, so it is a non-starter for Allmydata unless
tachyon.in wants to give us a more permissive licence.

But the OpenSSL Python wrappers that they wrote are permissively licensed:

http://tachyon.in/ncrypt/

And it works well on Windows:

http://tachyon.in/pipermail/ncrypt-users/2007-February/000016.html

Here's K. S. Sreeram's page:

http://sreeram.cc/

So all Python crypto libraries that I know of that do TLS and that have
compatible licences:

pyOpenSSL
tlslite
M2Crypto
ncrypt

I vaguely remember that Brian Warner investigated tlslite and had trouble with
it. I've heard bad things about M2Crypto. I would be interested in trying
ncrypt.

By the way, I was reminded while doing this browsing that we need to add "the
OpenSSL+GPL exception" to our licence.

http://en.wikipedia.org/wiki/OpenSSL#The_exception

The guy who made the tracdarcs plugin work is K. S. Sreeram. I was idly looking for alternate python crypto modules when I found ncrypt by K. S. Sreeram. Then I saw that ncrypt is sponsored by a p2p company, tachyon.in, which also makes a secure decentralized (?) instant messaging protocol: It has a very nice straightforward explanation up front: <http://cspace.in/> I was thinking that it might be a nice optional underlay protocol for Foolscap. Unfortunately it is GPL'ed, so it is a non-starter for Allmydata unless tachyon.in wants to give us a more permissive licence. But the OpenSSL Python wrappers that they wrote are permissively licensed: <http://tachyon.in/ncrypt/> And it works well on Windows: <http://tachyon.in/pipermail/ncrypt-users/2007-February/000016.html> Here's K. S. Sreeram's page: <http://sreeram.cc/> So all Python crypto libraries that I know of that do TLS and that have compatible licences: ``` pyOpenSSL tlslite M2Crypto ncrypt ``` I vaguely remember that Brian Warner investigated tlslite and had trouble with it. I've heard bad things about M2Crypto. I would be interested in trying ncrypt. By the way, I was reminded while doing this browsing that we need to add "the OpenSSL+GPL exception" to our licence. <http://en.wikipedia.org/wiki/OpenSSL#The_exception>

zooko commented

2007-04-30 04:16:43 +00:00

Author

I don't like pycrypto either. The official version's ctr mode is too slow, and AMK didn't accept my patch to speed it up. I never received a rejection letter either.

Actually, I do kind of like pycrypto, but it doesn't do TLS, and so we can probably replace both of our libraries -- pyOpenSSL and pycrypto -- with one which provides all the crypto functions we need and which is also portable and maintained and so on.

I don't like pycrypto either. The official version's ctr mode is too slow, and AMK didn't accept my patch to speed it up. I never received a rejection letter either. Actually, I *do* kind of like pycrypto, but it doesn't do TLS, and so we can probably replace both of our libraries -- pyOpenSSL and pycrypto -- with one which provides all the crypto functions we need and which is also portable and maintained and so on.

zooko changed title from ~~I don't like pyopenssl~~ to I don't like pyopenssl. or pycrypto

2007-04-30 04:16:43 +00:00

zooko added

c/code

and removed

c/unknown

labels 2007-04-30 05:33:14 +00:00

zooko commented

2007-05-02 04:07:23 +00:00

Author

We've added the OpenSSL exception to our licence.

zooko commented

2007-05-04 18:51:34 +00:00

Author

There appear to be people hacking on pycrypto other than AMK, as discovered by launchpad. I reported bugs on launchpad.

https://bugs.launchpad.net/pycrypto/+bug/112423

https://bugs.launchpad.net/pycrypto/+bug/112422

There appear to be people hacking on pycrypto other than AMK, as discovered by launchpad. I reported bugs on launchpad. <https://bugs.launchpad.net/pycrypto/+bug/112423> <https://bugs.launchpad.net/pycrypto/+bug/112422>

zooko added

p/minor

and removed

p/major

labels 2007-05-27 14:53:42 +00:00

evilrob commented

2007-06-05 00:24:46 +00:00

Owner

fwiw the allmydata.com 'ext' repository has a py24 native build of pyopenssl which was also found from trawling the web

evilrob commented

2007-06-05 00:25:31 +00:00

Owner

(which I suspect means that we don't have a py25 requirement on windows because of this dependency)

zooko commented

2007-06-17 15:53:10 +00:00

Author

See also ticket #199.

zooko commented

2007-11-28 18:56:52 +00:00

Author

Itamar pointed out that M2crypto has been integrated with Twisted, but on the other hand, Guido van Rossum had bad experiences with M2crypto:

http://www.artima.com/forums/flat.jsp?forum=106&thread=95863

Itamar pointed out that M2crypto has been integrated with Twisted, but on the other hand, Guido van Rossum had bad experiences with M2crypto: <http://www.artima.com/forums/flat.jsp?forum=106&thread=95863>

zooko commented

2007-11-30 16:29:25 +00:00

Author

Removing "pycrypto" from the subject line of this ticket, since we have removed our dependency on it by switching to pycryptopp. (This fix isn't committed to trunk yet, but I want to point people at this ticket who are interested in pyOpenSSL specifically.)

zooko changed title from ~~I don't like pyopenssl. or pycrypto~~ to I don't like pyopenssl

2007-11-30 16:29:25 +00:00

heikki commented

2007-12-03 04:19:33 +00:00

Owner

Replying to zooko:

Itamar pointed out that M2crypto has been integrated with Twisted, but on the other hand, Guido van Rossum had bad experiences with M2crypto:

http://www.artima.com/forums/flat.jsp?forum=106&thread=95863

I'd like to point out that Guido wrote that almost three years ago! As far as I know, all the issues he experienced were fixed long time ago.

Itamar is also correct: M2Crypto has a Twisted protocol wrapper, which can be used to do SSL instead of pyOpenSSL. It was modeled after similar thing in TLS Lite. We use Twisted in Chandler, and M2Crypto does the SSL part using this wrapper. More on Chandler at http://chandlerproject.org.

If you try M2Crypto and run into any issues, please let me know. I can't fix issues I don't know about. You can find the mailing list and bugzilla info on the M2Crypto homepage at http://chandlerproject.org/Projects/MeTooCrypto.

Heikki Toivonen - M2Crypto maintainer

Replying to [zooko](/tahoe-lafs/trac/issues/11#issuecomment-360235): > Itamar pointed out that M2crypto has been integrated with Twisted, but on the other hand, Guido van Rossum had bad experiences with M2crypto: > > <http://www.artima.com/forums/flat.jsp?forum=106&thread=95863> I'd like to point out that Guido wrote that almost three years ago! As far as I know, all the issues he experienced were fixed long time ago. Itamar is also correct: M2Crypto has a Twisted protocol wrapper, which can be used to do SSL instead of pyOpenSSL. It was modeled after similar thing in TLS Lite. We use Twisted in Chandler, and M2Crypto does the SSL part using this wrapper. More on Chandler at <http://chandlerproject.org>. If you try M2Crypto and run into any issues, please let me know. I can't fix issues I don't know about. You can find the mailing list and bugzilla info on the M2Crypto homepage at <http://chandlerproject.org/Projects/MeTooCrypto>. Heikki Toivonen - M2Crypto maintainer

zooko commented

2007-12-09 14:16:52 +00:00

Author

Heikki:

Thanks for the post. If M2Crypto is actively maintained, then this is a big advantage that it has over pyOpenSSL!

We could use M2Crypto for both our SSL needs and our filesystem crypto needs, thus removing the need for pyOpenSSL and removing the need for pycryptopp. (This latter part makes me a little sad because I like pycryptopp -- it is my newest baby.)

Brian: are you interested in using M2Crypto for foolscap's SSL layer?

Heikki: Thanks for the post. If M2Crypto is actively maintained, then this is a big advantage that it has over pyOpenSSL! We could use M2Crypto for both our SSL needs and our filesystem crypto needs, thus removing the need for pyOpenSSL and removing the need for pycryptopp. (This latter part makes me a little sad because I like pycryptopp -- it is my newest baby.) Brian: are you interested in using M2Crypto for foolscap's SSL layer?

zooko commented

2007-12-18 00:02:15 +00:00

Author

[http://yassl.com/]yassl is an implementation of SSL written on top of Crypto++. It is licensed under the traditional dual-licence model of "You can use it under the GPL, or you can ask us for a special licence.". I have already asked them for a special licence, and I'm waiting to hear back from them.

[<http://yassl.com/>]yassl is an implementation of SSL written on top of Crypto++. It is licensed under the traditional dual-licence model of "You can use it under the GPL, or you can ask us for a special licence.". I have already asked them for a special licence, and I'm waiting to hear back from them.

zooko commented

2008-01-13 22:36:55 +00:00

Author

Heikki Toivonen, the M2Crypto maintainer, posted the following comment in a discussion: "I suspect most
people use something more robust than plain M2Crypto for a server
application (like Apache or Twisted)."

This makes me think that M2Crypto might not be best for allmydata.org.

Heikki Toivonen, the M2Crypto maintainer, posted the following comment [in a discussion](http://article.gmane.org/gmane.comp.python.cryptography/494): "I suspect most people use something more robust than plain M2Crypto for a server application (like Apache or Twisted)." This makes me think that M2Crypto might not be best for allmydata.org.

zooko commented

2008-01-13 22:40:19 +00:00

Author

The aforementioned discussion leads to two further options:

http://www.brontes3d.com/opensource/straightssl.html

yet another openssl wrapper

http://pypi.python.org/pypi/ssl

This is a backport of the SSL implementation that is intended to be standard in Python >= v2.6.

The aforementioned discussion leads to two further options: 1. <http://www.brontes3d.com/opensource/straightssl.html> yet another openssl wrapper 2. <http://pypi.python.org/pypi/ssl> This is a backport of the SSL implementation that is intended to be standard in Python >= v2.6.

zooko commented

2008-01-13 22:41:00 +00:00

Author

Also, allmydata.com has decided that it would be okay to go ahead and use GPL'ed source code such as yassl, so that opens up some more options.

zooko commented

2008-01-13 22:49:15 +00:00

Author

HACK wonwin-mcbrootles-computer:~/playground/pyOpenSSL/pyOpenSSL-0.6$ python -c 'import OpenSSL'
Traceback (most recent call last):
  File "<string>", line 1, in ?
  File "/usr/local/stow/pyOpenSSL-0.6/lib/python2.4/site-packages/OpenSSL/__init__.py", line 11, in ?
    import rand, crypto, SSL, tsafe
ImportError: Failure linking new module: /usr/local/lib/python2.4/site-packages/OpenSSL/SSL.so: Symbol not found: _SSL_CTX_set_info_callback
  Referenced from: /usr/local/lib/python2.4/site-packages/OpenSSL/SSL.so
  Expected in: dynamic lookup

Oh, by the way, I forgot to mention that the reason I am looking at this ticket is that I can't compile pyOpenSSL 0.6 on Mac OS X against openssl-0.9.8g. There is a callback defined in openssl that takes (const SSL*, int, int), and pyOpenSSL passes a function that takes (SSL*, int, int). Patching pyOpenSSL to add the const keyword makes the gcc warning/error go away, but when I try to import it I still get: ``` HACK wonwin-mcbrootles-computer:~/playground/pyOpenSSL/pyOpenSSL-0.6$ python -c 'import OpenSSL' Traceback (most recent call last): File "<string>", line 1, in ? File "/usr/local/stow/pyOpenSSL-0.6/lib/python2.4/site-packages/OpenSSL/__init__.py", line 11, in ? import rand, crypto, SSL, tsafe ImportError: Failure linking new module: /usr/local/lib/python2.4/site-packages/OpenSSL/SSL.so: Symbol not found: _SSL_CTX_set_info_callback Referenced from: /usr/local/lib/python2.4/site-packages/OpenSSL/SSL.so Expected in: dynamic lookup ```

heikki commented

2008-01-14 00:23:59 +00:00

Owner

Replying to zooko:

Heikki Toivonen, the M2Crypto maintainer, posted the following comment in a discussion: "I suspect most
people use something more robust than plain M2Crypto for a server
application (like Apache or Twisted)."

This makes me think that M2Crypto might not be best for allmydata.org.

It really depends on what you need. If your website is not a high traffic site, M2Crypto is probably ok. But if you need traffic shaping, load balancing, guaranteed high availability etc. then I believe none of the simple libraries will be robust enough for you. That is why I mentioned Apache etc. which certainly can handle high traffic sites.

Replying to [zooko](/tahoe-lafs/trac/issues/11#issuecomment-360241): > Heikki Toivonen, the M2Crypto maintainer, posted the following comment [in a discussion](http://article.gmane.org/gmane.comp.python.cryptography/494): "I suspect most > people use something more robust than plain M2Crypto for a server > application (like Apache or Twisted)." > > This makes me think that M2Crypto might not be best for allmydata.org. It really depends on what you need. If your website is not a high traffic site, M2Crypto is probably ok. But if you need traffic shaping, load balancing, guaranteed high availability etc. then I believe none of the simple libraries will be robust enough for you. That is why I mentioned Apache etc. which certainly can handle high traffic sites.

warner commented

2008-01-15 04:33:08 +00:00

Most of what we need is just reactor.connectTCP, transport.startTLS(), and access to the certificate so we can validate it according to Foolscap's rules instead of the whole CA business (basically: the hash of the cert must match the server you were intending to connect to, no more, no less). My biggest concern about pyopenssl is an apparent memory consumption problem (something like 100kB to 1MB per open connection, although I haven't re-run those tests in a few years). If M2Crypto has comparable speed to pyopenssl, and it isn't too hard for users to obtain/build/install, then I'll seriously consider moving foolscap over to it.

FYI, I started playing with a port of Foolscap to M2Crypto. The Twisted interface seems pretty well implemented, but the way that you get access to the certificate (and the way you control validation) is pretty different, so I haven't gotten it working yet. When I get back next week I'll post my results and maybe beg Heikki for some help :). Most of what we need is just reactor.connectTCP, transport.startTLS(), and access to the certificate so we can validate it according to Foolscap's rules instead of the whole CA business (basically: the hash of the cert must match the server you were intending to connect to, no more, no less). My biggest concern about pyopenssl is an apparent memory consumption problem (something like 100kB to 1MB per open connection, although I haven't re-run those tests in a few years). If M2Crypto has comparable speed to pyopenssl, and it isn't too hard for users to obtain/build/install, then I'll seriously consider moving foolscap over to it.

heikki commented

2008-01-15 04:46:47 +00:00

Owner

Replying to warner:

FYI, I started playing with a port of Foolscap to M2Crypto. The Twisted interface seems pretty well implemented, but the way that you get access to the certificate (and the way you control validation) is pretty different, so I haven't gotten it working yet. When I get back next week I'll post my results and maybe beg Heikki for some help :).

M2Crypto is used in Chandler, and since the certificates are stored in a database in Chandler, the validation is a bit different. Also, Chandler will present the user with a dialog if there are errors in the SSL connection (the users can choose to ignore these), so this presents additional challenges for the SSL implementation. You can see how it is done in here:

http://svn.osafoundation.org/chandler/trunk/chandler/parcels/osaf/framework/certstore/ssl.py

Replying to [warner](/tahoe-lafs/trac/issues/11#issuecomment-360246): > FYI, I started playing with a port of Foolscap to M2Crypto. The Twisted interface seems pretty well implemented, but the way that you get access to the certificate (and the way you control validation) is pretty different, so I haven't gotten it working yet. When I get back next week I'll post my results and maybe beg Heikki for some help :). M2Crypto is used in Chandler, and since the certificates are stored in a database in Chandler, the validation is a bit different. Also, Chandler will present the user with a dialog if there are errors in the SSL connection (the users can choose to ignore these), so this presents additional challenges for the SSL implementation. You can see how it is done in here: <http://svn.osafoundation.org/chandler/trunk/chandler/parcels/osaf/framework/certstore/ssl.py>

zooko commented

2008-02-25 16:21:38 +00:00

Author

Rejoice -- exarkun and bigdog are working on pyOpenSSL. That makes the path of least resistance for us (continuing to use pyOpenSSL) also be the path of future promise, since exarkun is an excellent engineer.

zooko commented

2008-02-28 16:56:38 +00:00

Author

binary builds from exarkun and company:

http://buildbot.twistedmatrix.com/builds/

binary builds from exarkun and company: <http://buildbot.twistedmatrix.com/builds/>

zooko commented

2008-05-30 03:19:57 +00:00

Author

Okay, I like pyOpenSSL now since exarkun et alia are maintaining it. Closing as, um, "fixed".

Okay, I like `pyOpenSSL` now since exarkun et alia are maintaining it. Closing as, um, "fixed".

zooko added the

r/fixed

label 2008-05-30 03:19:57 +00:00

zooko closed this issue

2008-05-30 03:19:57 +00:00

zooko commented

2008-05-30 03:41:46 +00:00

Author

Re-opening this since I get unit test failures when I try to use the current pyOpenSSL-0.7 with Tahoe, and different unit test failures when I run pyOpenSSL-0.7's own unit tests. Here's the bug report for the pyOpenSSL project:

https://sourceforge.net/tracker/index.php?func=detail&aid=1978729&group_id=31249&atid=401758

zooko removed the

r/fixed

label 2008-05-30 03:41:46 +00:00

zooko reopened this issue

2008-05-30 03:41:46 +00:00

zooko commented

2008-05-30 04:12:32 +00:00

Author

Another reason why I still don't like pyOpenSSL, and therefore this ticket should remain open, is ticket #402 (bug in Twisted, triggered by pyOpenSSL-0.7).

warner modified the milestone from eventually to undecided

2008-06-01 20:47:58 +00:00

zooko commented

2008-08-12 17:10:26 +00:00

Author

Okay, now I like pyOpenSSL. See also #456 (it would be nice if the dependency on OpenSSL could be automatically resolved), but basically ticket #11 can finally be closed, thanks to JP Calderone and bigdog's stewardship of pyOpenSSL.

zooko added the

r/fixed

label 2008-08-12 17:10:26 +00:00

zooko closed this issue

2008-08-12 17:10:26 +00:00

launchpad commented

2008-10-31 15:33:21 +00:00

Owner

Updating Launchpad bug reference