HTTP proxy support for node to node communication #1007

New issue

Open

opened 2010-03-27 21:03:28 +00:00 by duck · 26 comments

duck commented

2010-03-27 21:03:28 +00:00

Owner

To use Tahoe-LAFS over the I2P anonymous network I have added HTTP proxy support to the Foolscap library. Most of the work is in Foolscap, but within Tahoe it must also be possible to enable this functionality and specify which HTTP proxy to use. All I2P nodes have this HTTP proxy enabled by default on 127.0.0.1:4444.

For the anonymous network use case, every connection to storage nodes and introducers will have to be through the anonymous network; it is not acceptable to mix anonymous and non-anonymous connections. The intention is to provide anonymity to both clients and node operators.

A reference implementation is available on http://duck.i2p.to/tahoe-lafs/ , as of today (2010-03-27) a test grid is operating inside I2P with 21 nodes, of which 6 storage nodes and 1 introducer.

Example of configuration in tahoe.cfg:

    [node]
    ...
    http_proxy = 127.0.0.1:4444

Snippit showing how this is used in node.py:

    http_proxy = self.get_config("node", "http_proxy", "")
    if http_proxy:
        self.tub.setOption("httpProxy", http_proxy)

To use Tahoe-LAFS over the [I2P anonymous network](http://www.i2p2.de/) I have added HTTP proxy support to the Foolscap library. Most of the work is in Foolscap, but within Tahoe it must also be possible to enable this functionality and specify which HTTP proxy to use. All I2P nodes have this HTTP proxy enabled by default on `127.0.0.1:4444`. For the anonymous network use case, every connection to storage nodes and introducers will have to be through the anonymous network; it is not acceptable to mix anonymous and non-anonymous connections. The intention is to provide anonymity to both clients and node operators. A reference implementation is available on <http://duck.i2p.to/tahoe-lafs/> , as of today (2010-03-27) a test grid is operating inside I2P with 21 nodes, of which 6 storage nodes and 1 introducer. Example of configuration in `tahoe.cfg`: ``` [node] ... http_proxy = 127.0.0.1:4444 ``` Snippit showing how this is used in `node.py`: ``` http_proxy = self.get_config("node", "http_proxy", "") if http_proxy: self.tub.setOption("httpProxy", http_proxy) ```

tahoe-lafs added the

labels 2010-03-27 21:03:28 +00:00

tahoe-lafs added this to the undecided milestone 2010-03-27 21:03:28 +00:00

duck commented

2010-03-27 21:06:05 +00:00

Author

Owner

Attachment trac_tahoe_http_proxy.txt (1218 bytes) added

HTTP Proxy support to Foolscap

**Attachment** trac_tahoe_http_proxy.txt (1218 bytes) added HTTP Proxy support to Foolscap

trac_tahoe_http_proxy.txt

1.2 KiB

duck commented

2010-03-27 21:45:51 +00:00

Author

Owner

Connected Foolscap tickets:

#150 HTTP proxy support
#151 Accept I2P destinations

Connected Foolscap tickets: * [#150](http://foolscap.lothar.com/trac/ticket/150) HTTP proxy support * [#151](http://foolscap.lothar.com/trac/ticket/151) Accept I2P destinations

daira modified the milestone from undecided to 1.7.0

2010-03-29 00:51:23 +00:00

daira commented

2010-04-12 20:56:43 +00:00

Requires a test that we enable the foolscap option when the http_proxy line is present.

(The foolscap changes will also need additions to the foolscap test suite.)

Requires a test that we enable the foolscap option when the `http_proxy` line is present. (The foolscap changes will also need additions to the foolscap test suite.)

daira commented

2010-04-12 21:02:04 +00:00

Replying to davidsarah:

Requires a test that we enable the foolscap option when the http_proxy line is present.

This should probably go in source:src/allmydata/test/test_client.py , I think.

Replying to [davidsarah](/tahoe-lafs/trac/issues/1007#issuecomment-377566): > Requires a test that we enable the foolscap option when the `http_proxy` line is present. This should probably go in source:src/allmydata/test/test_client.py , I think.

zooko commented

2010-06-17 04:27:16 +00:00

So the status of this ticket is that it is waiting for someone (ideally duck) to write tests, right? I guess that's what the "test" keyword means? Hey, let's make a new keyword: "test-needed". :-)

zooko modified the milestone from 1.7.0 to soon

2010-06-17 04:27:16 +00:00

zooko commented

2010-08-15 04:58:32 +00:00

See also #510 (use plain HTTP for storage server protocol).

zooko commented

2010-08-15 05:56:07 +00:00

So back in comment:377568 two months ago I set this ticket to "test-needed", and I haven't intended to do more on this ticket until duck (or someone) writes a test. But today I noticed that over on foolscap #150 and foolscap #151 duck has asked for the foolscap maintainer (Brian) to say whether he approves of the patches in principle and if so how to write a unit test for them. So the ball is back in our court. Brian: do you approve of these patches in principle?

So back in [comment:377568](/tahoe-lafs/trac/issues/1007#issuecomment-377568) two months ago I set this ticket to "test-needed", and I haven't intended to do more on this ticket until duck (or someone) writes a test. But today I noticed that over on [foolscap #150](http://foolscap.lothar.com/trac/ticket/150) and [foolscap #151](http://foolscap.lothar.com/trac/ticket/151) duck has asked for the foolscap maintainer (Brian) to say whether he approves of the patches in principle and if so how to write a unit test for them. So the ball is back in our court. Brian: do you approve of these patches in principle?

zooko commented

2010-10-06 17:52:52 +00:00

Brian: please tell duck whether he has a chance of getting his patches accepted into foolscap trunk (assuming of course that the patches pass quality requirements -- unit tests, code review, docs, etc.). I think duck has been blocked by not wanting to invest effort into his patches when he hasn't received any indication from the foolscap maintainer on whether those patches could ever have a chance of inclusion.

zooko commented

2010-10-22 14:49:41 +00:00

Oh look! Brian updated foolscap #150 and foolscap #151!

Oh look! Brian updated [foolscap #150](http://foolscap.lothar.com/trac/ticket/150) and [foolscap #151](http://foolscap.lothar.com/trac/ticket/151)!

duck commented

2011-01-12 21:40:30 +00:00

Author

Owner

Attachment 1007-http-proxy-support.patch (2052 bytes) added

HTTP Proxy support to Foolscap v2

**Attachment** 1007-http-proxy-support.patch (2052 bytes) added HTTP Proxy support to Foolscap v2

1007-http-proxy-support.patch

2 KiB

duck commented

2011-01-12 21:50:24 +00:00

Author

Owner

The suggestion of Brian to name the foolscap property http-proxy instead of httpProxy has been taken.

In addition to this an unit test has been implemented as suggested by davidsarah in comment:377566. test_node.py seemed to be a better place than test_client.py (comment:5) as this option applies to the node section and changes are made to node.py.

Please review version 2 of the patch.

The suggestion of Brian to name the foolscap property `http-proxy` instead of `httpProxy` has been taken. In addition to this an unit test has been implemented as suggested by davidsarah in [comment:377566](/tahoe-lafs/trac/issues/1007#issuecomment-377566). `test_node.py` seemed to be a better place than `test_client.py` (comment:5) as this option applies to the `node` section and changes are made to `node.py`. Please review version 2 of the patch.

daira commented

2011-01-12 22:00:58 +00:00

Given Brian's [comment:-1 on foolscap#150](@@http://foolscap.lothar.com/trac/ticket/150#comment:-1@@) that the proxy is actually relaying foolscap (which only initially looks a little like HTTP), I think the option name should not have "http" in it. Technically it is a "storage connection proxy", although that is a little long.

Another reason not to use "http" is that that could be confused with an HTTP proxy for the web-API (which we intentionally do not support).

Given Brian's [[comment:-1](/tahoe-lafs/trac/issues/1007#issuecomment--1) on foolscap#150](@@http://foolscap.lothar.com/trac/ticket/150#[comment:-1](/tahoe-lafs/trac/issues/1007#issuecomment--1)@@) that the proxy is actually relaying foolscap (which only initially looks a little like HTTP), I think the option name should not have "http" in it. Technically it is a "storage connection proxy", although that is a little long. Another reason not to use "http" is that that could be confused with an HTTP proxy for the web-API (which we intentionally do not support).

daira modified the milestone from soon to 1.9.0

2011-01-12 22:01:59 +00:00

duck commented

2011-01-16 13:29:56 +00:00

Author

Owner

Which name would you approve of for the "introducer and storage connection proxy that speaks a little HTTP proxy and is used by foolscap to make any outbound connections" configuration option?

foolscap-proxy
outbound-proxy
i2p-proxy
tub.proxy

Which name would you approve of for the "introducer and storage connection proxy that speaks a little HTTP proxy and is used by foolscap to make any outbound connections" configuration option? - foolscap-proxy - outbound-proxy - i2p-proxy - tub.proxy

zooko commented

2011-02-09 18:47:59 +00:00

Assigning to davidsarah to answer duck's question from comment:377577. (Brian may also have an opinion.)

Assigning to davidsarah to answer duck's question from [comment:377577](/tahoe-lafs/trac/issues/1007#issuecomment-377577). (Brian may also have an opinion.)

daira commented

2011-02-13 18:36:45 +00:00

outbound-proxy for the config option. It's not specific to i2p, so that rules out i2p-proxy. The existing tub. options control the tub for this storage server. And outbound-proxy is more specific than foolscap-proxy.

`outbound-proxy` for the config option. It's not specific to i2p, so that rules out `i2p-proxy`. The existing `tub.` options control the tub for *this* storage server. And `outbound-proxy` is more specific than `foolscap-proxy`.

daira commented

2011-02-13 22:42:04 +00:00

Replying to davidsarah:

outbound-proxy for the config option.

Make that outbound_proxy. Other config options use _ instead of -.

Replying to [davidsarah](/tahoe-lafs/trac/issues/1007#issuecomment-377579): > `outbound-proxy` for the config option. Make that `outbound_proxy`. Other config options use `_` instead of `-`.

zooko commented

2011-06-04 01:16:57 +00:00

Removing the review-needed tag until someone (perhaps duck) updates the name of the config option.

Removing the `review-needed` tag until someone (perhaps duck) updates the name of the config option.

zooko modified the milestone from 1.9.0 to soon

2011-07-27 18:21:54 +00:00

killyourtv commented

2013-08-07 15:37:33 +00:00

Author

Owner

Attachment 0001-outbound-proxy-support.patch (2753 bytes) added

outbound-proxy

**Attachment** 0001-outbound-proxy-support.patch (2753 bytes) added outbound-proxy

0001-outbound-proxy-support.patch

2.7 KiB

killyourtv commented

2013-08-07 15:41:31 +00:00

Author

Owner

Attached is an updated patch (0001-outbound-proxy-support.patch ) that changes the config option to outbound-proxy. The patch applies cleanly to current trunk.

(I'm still trying to learn Python so please forgive any n00b errors. I also need to learn how to write unit tests).

Attached is an updated patch (0001-outbound-proxy-support.patch ) that changes the config option to outbound-proxy. The patch applies cleanly to current trunk. (I'm still trying to learn Python so please forgive any n00b errors. I also need to learn how to write unit tests).

tahoe-lafs added

v/1.10.0

and removed

v/1.6.1

labels 2013-08-07 15:42:42 +00:00

daira commented

2013-08-31 01:04:04 +00:00

BTW, the Version field is intended to reflect the version in which an issue was first reported, so it's not necessary to update it unless it was originally set incorrectly.

daira added

v/1.6.1

and removed

v/1.10.0

labels 2013-08-31 01:04:04 +00:00

zooko commented

2013-09-29 20:22:32 +00:00

A potentially better way to accomplish it, which we are now poking at, is Twisted Endpoints -- see http://foolscap.lothar.com/trac/ticket/203 . One potential advantage of that — if it can be made to work — is that it might make it easier to support Tor, cjdns, SSL/TLS, IPv6, and maybe other cool networking protocols.

Jeff "psi" "ampernand" and I have been looking at this, and we don't think HTTP-proxying is the best way to do accomplish this. Because HTTP-proxying is meant to provide a request-response style, but Foolscap needs more of a "bidirectional byte-stream" style. A potentially better way to accomplish it, which we are now poking at, is Twisted Endpoints -- see <http://foolscap.lothar.com/trac/ticket/203> . One potential advantage of that — if it can be made to work — is that it might make it easier to support Tor, cjdns, SSL/TLS, IPv6, and maybe other cool networking protocols.