servers should attempt to open connections to clients #1086

New issue

Closed

opened 2010-06-16 03:57:43 +00:00 by zooko · 16 comments

zooko commented

2010-06-16 03:57:43 +00:00

To do this, servers should ask the introducer(s) to tell them about all clients and then servers should attempt to establish a foolscap connection to each client. If there is already such a connection (because, for example the client has already opened a connection to the server) then foolscap will automatically deduplicate and keep just one working connection.

Currently each client attempts to open a connection to each server. If it were the case that a server were behind NAT or firewall and connections couldn't be opened to that server, but that server were able to open connections out to clients, then if it did so the clients would be able to use that connection to issue requests to the server. To do this, servers should ask the introducer(s) to tell them about all clients and then servers should attempt to establish a foolscap connection to each client. If there is already such a connection (because, for example the client has already opened a connection to the server) then foolscap will automatically deduplicate and keep just one working connection.

zooko added the

labels 2010-06-16 03:57:43 +00:00

zooko added this to the eventually milestone 2010-06-16 03:57:43 +00:00

zooko commented

2010-06-16 04:06:30 +00:00

Author

[//pipermail/tahoe-dev/2010-June/004451.html]

warner commented

2010-06-16 18:06:27 +00:00

Now that I think about it, that does seem slightly backwards, since it's the NAT-bound server which is in the unusual position. So it would feel better to do something to the server (to indicate that it should connect to clients) rather than do something to the otherwise-normal clients.

I'd sort of like it if clients didn't have to publish contact information all the time (specifically, foolscap location hints).. that feels a bit weird too. I'm not sure how to accomplish both yet, though.

Yeah, something like that. As I mentioned on the email thread, one way to do this might be to have clients publish a special "please connect to me" announcement, which servers could then subscribe to and use to initiate outbound connections. Now that I think about it, that does seem slightly backwards, since it's the NAT-bound server which is in the unusual position. So it would feel better to do something to the server (to indicate that it should connect to clients) rather than do something to the otherwise-normal clients. I'd sort of like it if clients didn't have to publish contact information all the time (specifically, foolscap location hints).. that feels a bit weird too. I'm not sure how to accomplish both yet, though.

zooko commented

2010-07-26 04:38:39 +00:00

Author

What's the difference between publishing a "stub_client" record and publishing a "please connect to me" announcement?

I don't understand the motivation about doing something to the behind-NAT server instead of to the client and about not publishing contact information. I think it would be simple and effective to just have all clients publish contact information and all servers attempt to connect to all clients.

What's the difference between publishing a "stub_client" record and publishing a "please connect to me" announcement? I don't understand the motivation about doing something to the behind-NAT server instead of to the client and about not publishing contact information. I think it would be simple and effective to just have all clients publish contact information and all servers attempt to connect to all clients.

zooko commented

2011-07-31 20:31:12 +00:00

Author

See also #344 (more client-vs-server refactoring: servers-only shouldn't subscribe to storage announcements).

daira commented

2011-08-01 00:56:24 +00:00

Note that this would conflict with gdt's suggestion in /tahoe-lafs/trac/issues/26954#comment:18, that clients (actually non-servers) should only include 127.0.0.1 in their furls by default.

Note that this would conflict with gdt's suggestion in [/tahoe-lafs/trac/issues/26954](/tahoe-lafs/trac/issues/26954)#comment:18, that clients (actually non-servers) should only include 127.0.0.1 in their furls by default.

gdt commented

2011-08-01 12:54:37 +00:00

Owner

When I ran a server, one of the annoyances was it trying to open connections to RFC1918 addresses of servers behind NATs. If you assume mobile clients, then having a server behind a NAT means that sometimes one can connect and sometimes, not, and that's a mess with the current repair behavior. If you assume that all clients have global IP addresses and some servers don't, you inhabit a strange world. So I see this as addressing an odd case, and think that clients advertising any addresses at all and servers trying to connect to clients should be non-default behaviors.

zooko commented

2011-10-10 07:02:52 +00:00

Author

I think we should take one design strategy and "go all the way" with it, because either extreme is simpler for users to understand than what we currently have.

"all the way" in the minimal unix style is that servers never attempt to open connections to clients, and if a node has both a server and a client in it then the client part of it will not re-use a connection which a different node earlier established to the server part of it. In other words, the only thing that ever works is if the server is routeable, and if you want to run a server behind a firewall or NAT then you have to learn how to configure that firewall or NAT in order to give your server a routeable IP address. It is important to understand that if we do it this way then it is a bug if connections sometimes magically work when your server doesn't have a routable IP address and you didn't explicitly configure your firewall/NAT to forward this port. It is a bug because it confuses users when things "sometimes" work and it irritates minimal-unix-style people like gdt when programs do clever non-standard things.

"all the way" in the p2p style is that all nodes publish all of the "ways to contact me" that they can think of (including RFC1918 addresses and "my port number on localhost"), regardless of whether they are running a server or not, and all servers attempt to open connections to all clients and all clients attempt to open connections to all servers. This is non-standard and bothers expert sysadmins who want precise control over what TCP connection attempts are made, but it means that servers behind firewall/NAT can serve clients that are routeable without the server's operator having to learn how to configure their NAT. This can be understandable to users by explaining that the client and the server each try to connect to each other, and if it is possible for a TCP connection to be established in either of the two possible directions, then the client will be able to use the server. (Note that in addition to this approach supporting NATted-server + routeable client, it also supports client and server both on the same private network and neither one routeable from the public Internet.)

Both approaches have their merits and drawbacks and let's not get distracted by arguing which is "better". Currently we have something that is almost like the p2p style, except that servers don't attempt to open connections to clients and clients don't necessarily publish their contact info (if they aren't offering a server).

I think we should go ahead and make this style "complete" by the extensions I've requested in this ticket, which will I believe make it easier for users to understand and to use (even users who don't know how to configure their NAT device). I also think that in the future as we move to an HTTP-based transport instead of the current foolscap-based transport, we should probably switch to the extreme "minimal unix style".

tjgillies on IRC just had a confusing situation where he enabled storage service on a routeable node and this caused the gateway on that routeable node to start using a storage server on a behind-NAT node. This was understandably confusing to him. I think we should take one design strategy and "go all the way" with it, because either extreme is simpler for users to understand than what we currently have. "all the way" in the minimal unix style is that servers never attempt to open connections to clients, and if a node has both a server and a client in it then the client part of it will not re-use a connection which a different node earlier established to the server part of it. In other words, the only thing that ever works is if the server is routeable, and if you want to run a server behind a firewall or NAT then you have to learn how to configure that firewall or NAT in order to give your server a routeable IP address. It is important to understand that if we do it this way then **it is a bug** if connections sometimes magically work when your server doesn't have a routable IP address and you didn't explicitly configure your firewall/NAT to forward this port. It is a bug because it confuses users when things "sometimes" work and it irritates minimal-unix-style people like gdt when programs do clever non-standard things. "all the way" in the p2p style is that all nodes publish all of the "ways to contact me" that they can think of (including RFC1918 addresses and "my port number on localhost"), regardless of whether they are running a server or not, and all servers attempt to open connections to all clients and all clients attempt to open connections to all servers. This is non-standard and bothers expert sysadmins who want precise control over what TCP connection attempts are made, but it means that servers behind firewall/NAT can serve clients that are routeable without the server's operator having to learn how to configure their NAT. This can be understandable to users by explaining that the client and the server each try to connect to each other, and if it is possible for a TCP connection to be established in either of the two possible directions, then the client will be able to use the server. (Note that in addition to this approach supporting NATted-server + routeable client, it also supports client and server both on the same private network and neither one routeable from the public Internet.) Both approaches have their merits and drawbacks and let's not get distracted by arguing which is "better". Currently we have something that is *almost* like the p2p style, except that servers don't attempt to open connections to clients and clients don't necessarily publish their contact info (if they aren't offering a server). I think we should go ahead and make this style "complete" by the extensions I've requested in this ticket, which will I believe make it easier for users to understand and to use (even users who don't know how to configure their NAT device). I also think that in the future as we move to an HTTP-based transport instead of the current foolscap-based transport, we should probably switch to the extreme "minimal unix style".

zooko commented

2011-10-10 07:04:15 +00:00

Author

tjgillies added a FAQ entry on the wiki: wiki/FAQ?version=52#Q21_NAT

tjgillies added a FAQ entry on the wiki: [wiki/FAQ](wiki/FAQ)?version=52#Q21_NAT

gdt commented

2011-10-10 11:35:15 +00:00

Owner

This is also an argument for having v6 support working. It's boggling that Twisted is so far (10+ years) behind the curve.

On the notion of not arguing about better, two points:

we are talking about the protocol spec, not just the implementation behavior. tahoe-lafs suffers from not having a clear separation between these.
obviously different people have different preferences and different situations, so (less obviously) both styles should be supported
given support for both styles, a good question is whether it should be per-grid or per-system

As I've been using tahoe-lafs with a private grid, each node (process) is a client or a server. If the machine I want to use as a client happens to host a server node, then I run a client node (also). With the current implementation, client nodes have the aliases file and thus must be per-user, and server nodes are about the grid. (I am talking about the CLI, mounting, and WAPI, and reject the WUI as unsafe.) And to make servers that have private addresses reachable and usable to all clients, I would have to configure NAT translation. Given that, the client/server approach is sensible.

People using the pubgrid may want maximal connectivity, but if on a laptop/etc. they are going to find that servers come and go depending on whether their address is private or routable (and unfirewalled) at the moment. (But this is not currently helpful as tahoe does not work well with server churn (ticket:1209).)

So, it could be that the introducer should have a "connect-to-clients" flag, and if set then 1) clients publish contact information and 2) servers try to connect to clients (and use connections in either direction). I'd want this off, but it would be interesting to see what it should be on the pubgrid. Perhaps the client should have a connect-to-clients config with yes/introducer/no, where "introducer" means to respect the config value from the introducer. In a future with a distributed introducer function, this is just a piece of data that needs to be available from the function. And it could be that the flag is passed in the setup of each connection, so reverse channels are not used to those who say connect-to-clients=off.

It could also be that it's easier to make the flag be purely local, causing a client to publish contact information, causing a node that offers storage to connect to nodes that do not, trying to use the reverse channel, and allowing the reverse channel to be used.

I think it's an important property that if on a node of mine I set connect-to-clients=off, then my client node will not publish anything (I realize the introducer still knows about it) and my server node won't try to make connections to clients. (I do find repeated connection attempts to RFC1918 addresses annoying, and this helps but does not fix the issue.)

I concur with the notion that the implementation should be "all the way" in either the unix style (which I would call client/server) or the p2p style. (I am amused at being the canonical example of minimal.) This is also an argument for having v6 support working. It's boggling that Twisted is so far (10+ years) behind the curve. On the notion of not arguing about better, two points: * we are talking about the protocol spec, not just the implementation behavior. tahoe-lafs suffers from not having a clear separation between these. * obviously different people have different preferences and different situations, so (less obviously) both styles should be supported * given support for both styles, a good question is whether it should be per-grid or per-system As I've been using tahoe-lafs with a private grid, each node (process) is a client or a server. If the machine I want to use as a client happens to host a server node, then I run a client node (also). With the current implementation, client nodes have the aliases file and thus must be per-user, and server nodes are about the grid. (I am talking about the CLI, mounting, and WAPI, and reject the WUI as unsafe.) And to make servers that have private addresses reachable and usable to all clients, I would have to configure NAT translation. Given that, the client/server approach is sensible. People using the pubgrid may want maximal connectivity, but if on a laptop/etc. they are going to find that servers come and go depending on whether their address is private or routable (and unfirewalled) at the moment. (But this is not *currently* helpful as tahoe does not work well with server churn (ticket:1209).) So, it could be that the introducer should have a "connect-to-clients" flag, and if set then 1) clients publish contact information and 2) servers try to connect to clients (and use connections in either direction). I'd want this off, but it would be interesting to see what it should be on the pubgrid. Perhaps the client should have a connect-to-clients config with yes/introducer/no, where "introducer" means to respect the config value from the introducer. In a future with a distributed introducer function, this is just a piece of data that needs to be available from the function. And it could be that the flag is passed in the setup of each connection, so reverse channels are not used to those who say connect-to-clients=off. It could also be that it's easier to make the flag be purely local, causing a client to publish contact information, causing a node that offers storage to connect to nodes that do not, trying to use the reverse channel, and allowing the reverse channel to be used. I think it's an important property that if on a node of mine I set connect-to-clients=off, then my client node will not publish anything (I realize the introducer still knows about it) and my server node won't try to make connections to clients. (I do find repeated connection attempts to RFC1918 addresses annoying, and this helps but does not fix the issue.)

gdt commented

2011-10-10 11:43:48 +00:00

Owner

Related, if a server is only a server, and not a client (no associated uid/human), then it shouldn't try to open any connections (in the minimal style). So perhaps in addition to configuring storage on/off, the client behavior should be configurable. That makes it more clear that this ticket is (in part) talking about servers that don't have a need to be clients making connections to client-only nodes.

zooko commented

2011-10-10 17:42:57 +00:00

Author

In any case, if we supported multiple ways to do it (note: I'm almost always opposed to having more than one way to do it), then as long as the way that is friendly to non-technical users is the default then that advantage is preserved. If the user has to set a configuration setting in order to get that behavior then that advantage evaporates.

The main advantage of the p2p way is that a user might be able to successfully run a server even if they don't know what a "port number" is and they don't want to learn. Perhaps this is not a desirable result! Because such servers are likely to be unreliable anyway. In any case, if we supported multiple ways to do it (note: I'm almost always opposed to having more than one way to do it), then as long as the way that is friendly to non-technical users is the default then that advantage is preserved. If the user has to set a configuration setting in order to get that behavior then that advantage evaporates.

daira commented

2011-10-10 17:45:43 +00:00

Replying to zooko:

tjgillies added a FAQ entry on the wiki: wiki/FAQ?version=52#Q21_NAT

which says:

Ideally, all clients attempt to open connections to all servers, and all servers attempt to open connections to all clients. So, if the client is not behind NAT, then even if the server is behind NAT. However, this is not currently the case. Currently what it does is that all clients attempt to open connections to all servers, but if there is a connection between two Tahoe-LAFS processes (== Tahoe-LAFS nodes) it can re-use that connection for any client or server in either node. So, when you enabled a storage server on the public facing server, that caused the node behind NAT to initiate a TCP connection to the node on the public facing server. Once that connection was established, that enabled the node there to use the server behind NAT.

The current behaviour is a bug for the following reason: If a node is behind a NAT and is only intermittently connected, but has storage service enabled (by mistake, perhaps), then it will sometimes be used as a server and sometimes be inaccessible. Worse, it will only be accessible to some subset of the other nodes. This leads to poor preservation and availability. It is better for the NAT to cause the node to always be inaccessible as a server.

I also disagree that we should be changing the behaviour to one style now and then changing it to the opposite style later. It makes more sense to have a config switch (but I don't know where that switch should be).

Replying to [zooko](/tahoe-lafs/trac/issues/1086#issuecomment-378982): > tjgillies added a FAQ entry on the wiki: [wiki/FAQ](wiki/FAQ)?version=52#Q21_NAT which says: > Ideally, all clients attempt to open connections to all servers, and all servers attempt to open connections to all clients. So, if the client is not behind NAT, then even if the server is behind NAT. However, this is not currently the case. Currently what it does is that all clients attempt to open connections to all servers, but if there is a connection between two Tahoe-LAFS processes (== Tahoe-LAFS nodes) it can re-use that connection for any client or server in either node. So, when you enabled a storage server on the public facing server, that caused the node behind NAT to initiate a TCP connection to the node on the public facing server. Once that connection was established, that enabled the node there to *use* the server behind NAT. The current behaviour is a bug for the following reason: If a node is behind a NAT and is only intermittently connected, but has storage service enabled (by mistake, perhaps), then it will sometimes be used as a server and sometimes be inaccessible. Worse, it will only be accessible to some subset of the other nodes. This leads to poor preservation and availability. It is better for the NAT to cause the node to always be inaccessible as a server. I also disagree that we should be changing the behaviour to one style now and then changing it to the opposite style later. It makes more sense to have a config switch (but I don't know where that switch should be).

zooko commented

2011-10-10 18:02:29 +00:00

Author

Replying to davidsarah:

I also disagree that we should be changing the behaviour to one style now and then changing it to the opposite style later. It makes more sense to have a config switch (but I don't know where that switch should be).

Hm, well if we're going to converge on the minimal-unix style, this is mostly just a matter of eliminating the use of certain foolscap features. It isn't obvious how to eliminate them -- foolscap doesn't provide a way to turn them off. I mean features like: all objects in one foolscap tub can access all objects in the other foolscap tub as long as there is currently a TCP connection open to them. One simple way to make it so that this feature no longer allows clients to use an unrouteable server as long as that server's tub happens to have opened a connection to that client would be to stop running both clients and servers in the same tub! That appeals to me. Then if you want to run both a gateway and a server you have to launch two separate processes in your operating system.

Another feature which should probably be eliminated if we're going to the minimal unix style is the foolscap feature of listing multiple alternative IP addresses or hostnames, possibly including ones that are not routeable from the public Internet, and having a client attempt to connect to all of them in parallel and then use whichever connection succeeds. This feels like kind of a loss -- that feature is cool because it allows two nodes behind NAT to connect directly to one another over the LAN, for example. The minimal unix way to do that is that they cannot talk to each other until their human manually configures the NAT, in which case their packets get routed to the NAT box and then back to their LAN, or the human manually configures their IP address to be the LAN-scoped IP address, in which case they can connect to one another directly.

David-Sarah: the reason to go the rest of the way to one style now and then switch to the opposite extreme style later is that we plan to switch from foolscap to HTTP. Foolscap provides a lot of very cool features in the p2p style, and in fact it is hard to use Foolscap without the p2p style. HTTP would be hard to use in the p2p style. I think we should work with the tools rather than against them here.

Whatever we do, I hope we don't let this ticket just lie here because there are differing opinions about which direction to head. Heading in either direction would be an improvement, in my opinion, where the measurement of value I'm using is "How hard is it for users to understand and predict the behavior".

Replying to [davidsarah](/tahoe-lafs/trac/issues/1086#issuecomment-378986): > > I also disagree that we should be changing the behaviour to one style now and then changing it to the opposite style later. It makes more sense to have a config switch (but I don't know where that switch should be). Hm, well if we're going to converge on the minimal-unix style, this is mostly just a matter of eliminating the use of certain foolscap features. It isn't obvious how to eliminate them -- foolscap doesn't provide a way to turn them off. I mean features like: all objects in one foolscap tub can access all objects in the other foolscap tub as long as there is currently a TCP connection open to them. One simple way to make it so that this feature no longer allows clients to use an unrouteable server as long as that server's tub happens to have opened a connection to that client would be to stop running both clients and servers in the same tub! That appeals to me. Then if you want to run both a gateway and a server you have to launch two separate processes in your operating system. Another feature which should probably be eliminated if we're going to the minimal unix style is the foolscap feature of listing multiple alternative IP addresses or hostnames, possibly including ones that are not routeable from the public Internet, and having a client attempt to connect to all of them in parallel and then use whichever connection succeeds. This feels like kind of a loss -- that feature is cool because it allows two nodes behind NAT to connect directly to one another over the LAN, for example. The minimal unix way to do that is that they cannot talk to each other until their human manually configures the NAT, in which case their packets get routed to the NAT box and then back to their LAN, or the human manually configures their IP address to be the LAN-scoped IP address, in which case they can connect to one another directly. David-Sarah: the reason to go the rest of the way to one style now and then switch to the opposite extreme style later is that we plan to switch from foolscap to HTTP. Foolscap provides a lot of very cool features in the p2p style, and in fact it is hard to use Foolscap *without* the p2p style. HTTP would be hard to use in the p2p style. I think we should work with the tools rather than against them here. Whatever we do, I hope we don't let this ticket just lie here because there are differing opinions about which direction to head. Heading in either direction would be an improvement, in my opinion, where the measurement of value I'm using is "How hard is it for users to understand and predict the behavior".

daira commented

2011-10-10 18:20:20 +00:00

Replying to [zooko]comment:13:

Replying to davidsarah:

I also disagree that we should be changing the behaviour to one style now and then changing it to the opposite style later. It makes more sense to have a config switch (but I don't know where that switch should be).

Hm, well if we're going to converge on the minimal-unix style, this is mostly just a matter of eliminating the use of certain foolscap features. It isn't obvious how to eliminate them -- foolscap doesn't provide a way to turn them off.

The introducer is responsible for connecting clients to servers. If the introducer doesn't connect them, it doesn't matter which objects are in which vats, IIUC. There are no implicit connections due to objects being in the same vat; that would not be consistent with a capability model.

Replying to [zooko]comment:13: > Replying to [davidsarah](/tahoe-lafs/trac/issues/1086#issuecomment-378986): > > > > I also disagree that we should be changing the behaviour to one style now and then changing it to the opposite style later. It makes more sense to have a config switch (but I don't know where that switch should be). > > Hm, well if we're going to converge on the minimal-unix style, this is mostly just a matter of eliminating the use of certain foolscap features. It isn't obvious how to eliminate them -- foolscap doesn't provide a way to turn them off. The introducer is responsible for connecting clients to servers. If the introducer doesn't connect them, it doesn't matter which objects are in which vats, IIUC. There are no implicit connections due to objects being in the same vat; that would not be consistent with a capability model.

zooko commented

2013-06-25 15:54:33 +00:00

Author

adding keyword p2p to tag tickets that have to do with the open question of "Is Tahoe-LAFS client-server or is it peer-to-peer?": [//pipermail/tahoe-dev/2012-July/007533.html]

adding keyword `p2p` to tag tickets that have to do with the open question of "Is Tahoe-LAFS client-server or is it peer-to-peer?": [//pipermail/tahoe-dev/2012-July/007533.html]

warner commented

2016-08-30 06:28:43 +00:00

#760 added tahoe create-client (ok that landed a while ago)
#2759 moved us to ephemeral per-server Tubs, removing the notion of "one Tub per process"
#2816 (about to land) makes tub.port= (empty) mean "don't listen, don't set a location"

At this point, I think we'd all feel comfortable making tahoe create-client set tub.port= to disable listening. In this world, clients never listen, so servers cannot connect to client-only nodes.

So I'm going to close this as a WONTFIX.

Over the last few years, I think we've converged on believing that clients (nodes that do not have Storage or a Helper enabled) should not accept connections. We've made several changes recently to support this: * #760 added `tahoe create-client` (ok that landed a while ago) * #2759 moved us to ephemeral per-server Tubs, removing the notion of "one Tub per process" * #2816 (about to land) makes `tub.port=` (empty) mean "don't listen, don't set a location" At this point, I think we'd all feel comfortable making `tahoe create-client` set `tub.port=` to disable listening. In this world, clients never listen, so servers cannot connect to client-only nodes. So I'm going to close this as a WONTFIX.

Rows
Columns