Handling Tor and i2p in NURLs #3888

New issue

Closed

opened 2022-04-06 15:40:28 +00:00 by itamarst · 11 comments

itamarst commented

2022-04-06 15:40:28 +00:00

NURLs, the new furl replacement that is HTTP based, does not currently support Tor or i2p syntactically. It just has a hostname field... The hyperlink library also has issues with parsing it if is forced in to the hostname, which may be legitimate "this is not a valid URI/URL" complaint.

One alternative is to switch to combo protocol URLs, e.g.:

pb://1WUX44xKjKdpGLohmFcBNuIRN-8rlv1Iij_7rQ@127.1:34399/jhjbc3bjbhk#v=1
pb+tor://1WUX44xKjKdpGLohmFcBNuIRN-8rlv1Iij_7rQ@127.1:34399/jhjbc3bjbhk#v=1
pb+i2p://1WUX44xKjKdpGLohmFcBNuIRN-8rlv1Iij_7rQ@sdsdfsdfds/jhjbc3bjbhk#v=1

Changing the NURL to this could be backwards compatible if pb:// is synonym for normal HTTPS, as above.

NURLs, the new furl replacement that is HTTP based, does not currently support Tor or i2p syntactically. It just has a hostname field... The `hyperlink` library also has issues with parsing it if is forced in to the hostname, which may be legitimate "this is not a valid URI/URL" complaint. One alternative is to switch to combo protocol URLs, e.g.: ``` pb://1WUX44xKjKdpGLohmFcBNuIRN-8rlv1Iij_7rQ@127.1:34399/jhjbc3bjbhk#v=1 pb+tor://1WUX44xKjKdpGLohmFcBNuIRN-8rlv1Iij_7rQ@127.1:34399/jhjbc3bjbhk#v=1 pb+i2p://1WUX44xKjKdpGLohmFcBNuIRN-8rlv1Iij_7rQ@sdsdfsdfds/jhjbc3bjbhk#v=1 ``` Changing the NURL to this could be backwards compatible if `pb://` is synonym for normal HTTPS, as above.

itamarst added the

labels 2022-04-06 15:40:28 +00:00

itamarst added this to the HTTP Storage Protocol milestone 2022-04-06 15:40:28 +00:00

itamarst commented

2022-04-06 16:49:29 +00:00

Author

Worth breaking out specific cases:

You can have a server listening on public Internet. Client uses Tor. In this case... normal NURL works, insofar as you just make sure to use the Tor proxy. But in practice something that understands Tahoe semantics would provide better anonymity, so the use case is "connect to this normal NURL, but use Tor".
You can have a server listening on a Tor private address, which ends with ".onion" or something. In this case maybe normal NURL works too, you could identify it's a special Tor address from teh TLD.
As in second Tor use case, I2P addresses might also be identifiable via ".i2p".

Worth breaking out specific cases: * You can have a server listening on public Internet. Client uses Tor. In this case... normal NURL works, insofar as you just make sure to use the Tor proxy. But in practice something that understands Tahoe semantics would provide better anonymity, so the use case is "connect to this normal NURL, but use Tor". * You can have a server listening on a Tor private address, which ends with ".onion" or something. In this case maybe normal NURL works too, you could identify it's a special Tor address from teh TLD. * As in second Tor use case, I2P addresses might also be identifiable via ".i2p".

meejah commented

2022-04-06 17:37:27 +00:00

I'm not sure that there's a problem with Tor here; onion services are on syntactically-valid .onion URLs with a well-known TLD of ".onion" (RFC 7686) which are "just" host-names.

Of course, you do need something that understands what to do with those (which in practice means "using SOCKS5 to the local Tor daemon"). For our case here, txtorcon understands all this and provides "IStreamClientEndpoint" or "IAgent" type interfaces.

I'm not sure that there's a problem with Tor here; onion services are on syntactically-valid `.onion` URLs with a well-known TLD of ".onion" (RFC 7686) which are "just" host-names. Of course, you do need something that understands what to do with those (which in practice means "using SOCKS5 to the local Tor daemon"). For our case here, txtorcon understands all this and provides "IStreamClientEndpoint" or "IAgent" type interfaces.

exarkun commented

2022-04-06 17:43:34 +00:00

The high-level question to answer is:

Given a string containing a NURL, how does the GBS client know how to set up the connection?

The answer implied by the ticket description is to look at the scheme and for "pb" use HTTP over TLS, for "pb+tor" use HTTP over Tor, and for "pb+i2p" use HTTP over I2P.

I'm not sure that there's a problem with Tor here; onion services are on syntactically-valid .onion URLs with a well-known TLD of ".onion" (RFC 7686) which are "just" host-names.

It sounds like this means there might be an alternate answer possible:

Look at the scheme. If it is "pb" and the hostname does not end with ".onion" then use HTTP over TLS. If it is "pb" and the hostname does end with ".onion" then use HTTP over Tor. If it is "pb+i2p" then use HTTP over I2P.

Did I understand the Tor interaction correctly?

If so, I still might prefer the "pb+tor" scheme because of the symmetry it maintains with the other two transports to be supported.

The high-level question to answer is: > Given a string containing a NURL, how does the GBS client know how to set up the connection? The answer implied by the ticket description is to look at the scheme and for "pb" use HTTP over TLS, for "pb+tor" use HTTP over Tor, and for "pb+i2p" use HTTP over I2P. > I'm not sure that there's a problem with Tor here; onion services are on syntactically-valid .onion URLs with a well-known TLD of ".onion" (RFC 7686) which are "just" host-names. It sounds like this means there might be an alternate answer possible: Look at the scheme. If it is "pb" and the hostname does not end with ".onion" then use HTTP over TLS. If it is "pb" and the hostname does end with ".onion" then use HTTP over Tor. If it is "pb+i2p" then use HTTP over I2P. Did I understand the Tor interaction correctly? If so, I still might prefer the "pb+tor" scheme because of the symmetry it maintains with the other two transports to be supported.

meejah commented

2022-04-06 17:49:51 +00:00

Yeah, "pb+tor" might be better because it also covers the case where you want to use Tor to a clearnet listener. And it's more explicit.

That said, though, "use tor for client-operations" is more of a client decision and doesn't necessarily need to be burned into the NURLS like that -- the only time you need to use Tor as a client is when the listener is on a .onion

There's a lot of other stuff to consider regarding "internet location privacy" so having e.g. a Tahoe option like "--location-privacy" would be more how I'd handle the "tor to clearnet services" situation, because there are other things you'd probably want to put over Tor and e.g. make sure you're not doing local DNS lookups or similar.

Yeah, "pb+tor" might be better because it also covers the case where you want to use Tor to a clearnet listener. And it's more explicit. That said, though, "use tor for client-operations" is more of a client decision and doesn't necessarily need to be burned into the NURLS like that -- the only time you _need_ to use Tor as a client is when the listener is on a .onion There's a _lot_ of other stuff to consider regarding "internet location privacy" so having e.g. a Tahoe option like "--location-privacy" would be more how I'd handle the "tor to clearnet services" situation, because there are other things you'd probably want to put over Tor and e.g. make sure you're not doing local DNS lookups or similar.

meejah commented

2022-04-06 18:01:02 +00:00

BTW, the tor example should look like pb+tor://1WUX44xKjKdpGLohmFcBNuIRN-8rlv1Iij_7rQ@fjblvrw2jrxnhtg67qpbzi45r7ofojaoo3orzykesly2j3c2m3htapid.onion:34399/jhjbc3bjbhk#v=1

BTW, the tor example should look like `pb+tor://1WUX44xKjKdpGLohmFcBNuIRN-8rlv1Iij_7rQ@fjblvrw2jrxnhtg67qpbzi45r7ofojaoo3orzykesly2j3c2m3htapid.onion:34399/jhjbc3bjbhk#v=1`

exarkun commented

2022-04-06 18:18:02 +00:00

Summarizing some further discussion from IRC ...

There is existing support for having a lot of control over how connections are made. This is in the connections section of tahoe.cfg and there's docs for it. Essentially, the GBS client should be prepared to use Tor even for a "pb://..." NURL because maybe the client is configured to prefer privacy-enabling connections even when they are not required to reach a server.

There is other configuration required to be a Tor client but it does not belong in NURLs. It covers, eg, the address of a Tor-ifying SOCKS server to which to connect. This configuration is already supported in the tor section of tahoe.cfg and this is how the GBS client can figure out how to create a Tor client endpoint (or IAgent) to use.

Foolscap's I2P support is somewhat more featureful than the pb+i2p:*... example given here. It accepts mostly-arbitrary key/value pairs in the connection hints. There's room in the pb+i2p scheme though - for example, in the query arguments. eg pb+i2p:*1WUX44xKjKdpGLohmFcBNuIRN-8rlv1Iij_7rQ@sdsdfsdfds/jhjbc3bjbhk#v=1&foo=bar

Summarizing some further discussion from IRC ... There is existing support for having a lot of control over how connections are made. This is in the `connections` section of tahoe.cfg and there's docs for it. Essentially, the GBS client should be prepared to use Tor even for a "pb://..." NURL because maybe the client is configured to prefer privacy-enabling connections even when they are not required to reach a server. There is other configuration required to be a Tor client but it does not belong in NURLs. It covers, eg, the address of a Tor-ifying SOCKS server to which to connect. This configuration is already supported in the `tor` section of tahoe.cfg and this is how the GBS client can figure out how to create a Tor client endpoint (or `IAgent`) to use. Foolscap's I2P support is somewhat more featureful than the `pb+i2p:*...` example given here. It accepts mostly-arbitrary key/value pairs in the connection hints. There's room in the `pb+i2p` scheme though - for example, in the query arguments. eg `pb+i2p:*1WUX44xKjKdpGLohmFcBNuIRN-8rlv1Iij_7rQ@sdsdfsdfds/jhjbc3bjbhk#v=1&foo=bar`

meejah commented

2022-06-10 19:33:31 +00:00

I think this all leads me to conclude that we shouldn't have pb+tor: scheme. There should just be a pb:// scheme and the other configuration tells a client whether to use Tor or not.

If the NURL ends in the well-defined TLD .onion then the client has to decide whether to use that NURL at all or not. If it has Tor configuration, then it can attempt to use it. If it does not, it should not attempt to use it.

I think this all leads me to conclude that we _shouldn't_ have `pb+tor:` scheme. There should just be a `pb://` scheme and the other configuration tells a client whether to use Tor or not. If the NURL ends in the well-defined TLD `.onion` then the client has to decide whether to use that NURL at all or not. If it has Tor configuration, then it can attempt to use it. If it does not, it should not attempt to use it.

itamarst commented

2022-07-25 14:19:17 +00:00

Author

For the moment, then, the actual logic for listening on these ports is implemented by Foolscap, there is no need for HTTP-specific code. However, we do need custom NURL generation for Tor (the .onion domain) and I2P. This can be done post-#3902, and sounds like we decided on pb:* for Tor and pb+i2p:* for I2P. Even though the Tor endpoint has same URL structure as normal endpoint, it still needs custom code to generate it.

In the long term we will need to reimplement the listening logic, but that might take a while due to need for backwards compatibility.

On the client side we need two things:

At minimum, HTTP client code that understands .onion URLs (#3910).
An HTTP-specific Tor policy for additional privacy, specifically to address timing analysis attacks, which would work both with .onion and public servers. For example, the Tor Browser has a bunch of policies about how it routes to different domains and to a specific domain to reduce this risk. This is probably a long-term thing (#3911).

More notes: the way we've decided to implement listening (#3902) is by sharing the Foolscap TCP port, and therefore the Foolscap listening and configuration-parsing logic, specifically `tub.port` and `tub.location`. For the moment, then, the actual logic for _listening_ on these ports is implemented by Foolscap, there is no need for HTTP-specific code. However, we do need custom NURL generation for Tor (the `.onion` domain) and I2P. This can be done post-#3902, and sounds like we decided on `pb:*` for Tor and `pb+i2p:*` for I2P. Even though the Tor endpoint has same URL structure as normal endpoint, it still needs custom code to generate it. In the long term we will need to reimplement the listening logic, but that might take a while due to need for backwards compatibility. On the _client_ side we need two things: 1. At minimum, HTTP client code that understands `.onion` URLs (#3910). 2. An HTTP-specific Tor policy for additional privacy, specifically to address timing analysis attacks, which would work both with `.onion` and public servers. For example, the Tor Browser has a bunch of policies about how it routes to different domains and to a specific domain to reduce this risk. This is probably a long-term thing (#3911).

itamarst commented

2022-09-15 14:08:47 +00:00

Author

So actually the spec for NURLs has a proposal; regardless of what gets chosen (spec or above), we should make sure everything is in sync!

https://github.com/tahoe-lafs/tahoe-lafs/blob/master/docs/specifications/url.rst

So actually the spec for NURLs has a proposal; regardless of what gets chosen (spec or above), we should make sure everything is in sync! <https://github.com/tahoe-lafs/tahoe-lafs/blob/master/docs/specifications/url.rst>

itamarst commented

2023-05-12 13:17:49 +00:00

Author

Re-reading:

Tor support in NURL scheme is not required. The client needs to detect .onion, yes, but that doesn't require a change in the URL scheme.
I2P does have extra options, but those are options the client will want to decide, it's not related to the NURL the server puts out. So as long as no one makes a real conflicting .i2p TLD then you can do same thing as Tor.

So real issue is maybe "are we doing right thing in HTTP client when given .onion and .i2p domains". And custom URLs are nice hint but maybe not actually required.

Re-reading: * Tor support in NURL scheme is not required. The client needs to detect .onion, yes, but that doesn't require a change in the URL scheme. * I2P does have extra options, but those are options the _client_ will want to decide, it's not related to the NURL the server puts out. So as long as no one makes a real conflicting .i2p TLD then you can do same thing as Tor. So real issue is maybe "are we doing right thing in HTTP client when given .onion and .i2p domains". And custom URLs are nice hint but maybe not actually required.

Rows
Columns