tahoe-lafs/trac-2024-07-25

introducers.yaml causes foolscap assertion #2862

New issue

Closed

opened 2017-01-08 22:01:56 +00:00 by warner · 4 comments

warner commented

2017-01-08 22:01:56 +00:00

Owner

User "lazygravy" on IRC reported a node-startup exception that happens when private/introducers.yaml is used. The problem appears to be that the FURL given to foolscap is a unicode string, and there is code inside foolscap which asserts that the connection hints it sees are normal strings (not unicode). The exception (as recorded in twistd.log) looks like:

Unhandled error in Deferred:
  Traceback (most recent call last):
    File "/usr/local/lib/python2.7/site-packages/foolscap/pb.py", line 856, in connectTo
      rc.startConnecting(self)
    File "/usr/local/lib/python2.7/site-packages/foolscap/reconnector.py", line 78, in startConnecting
      self._connect()
    File "/usr/local/lib/python2.7/site-packages/foolscap/reconnector.py", line 110, in _connect
      d = self._tub.getReference(self._url)
    File "/usr/local/lib/python2.7/site-packages/foolscap/pb.py", line 794, in getReference
      return defer.maybeDeferred(self._getReference, sturdyOrURL)
  --- <exception caught here> ---
    File "/usr/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
      result = f(*args, **kw)
    File "/usr/local/lib/python2.7/site-packages/foolscap/pb.py", line 811, in _getReference
      d = self.getBrokerForTubRef(sturdy.getTubRef())
    File "/usr/local/lib/python2.7/site-packages/foolscap/referenceable.py", line 804, in getTubRef
      return TubRef(self.tubID, self.locationHints)
    File "/usr/local/lib/python2.7/site-packages/foolscap/referenceable.py", line 837, in __init__
      locationHints
  exceptions.AssertionError: [u'cz3uevcjrgwszq4ytvgisdw2nsdzgaxfan4nvh7uy5bxcbr7qv4q.b32.i2p']

I think the code that loads the YAML file needs to do an .encode("ascii") on any of the FURLs it extracts, before passing them to foolscap.

In addition, docs/configuration.rst had an invalid example for the private/introducers.yaml section (around line 920). It said:

introducers:
  petname2: furl = FURL2

but that will result in a data structure like introducers = {"petname2": "furl = FURL2"}, which causes an exception at node startup time like this:

  File "/usr/local/lib/python2.7/site-packages/allmydata/client.py", line 189, in __init__
    self.init_introducer_clients()
  File "/usr/local/lib/python2.7/site-packages/allmydata/client.py", line 268, in init_introducer_clients
    ic = IntroducerClient(self.tub, introducer['furl'],
exceptions.TypeError: string indices must be integers

Instead, the docs should recommend:

introducers:
  petname2:
    furl: FURL2

User "lazygravy" on IRC reported a node-startup exception that happens when `private/introducers.yaml` is used. The problem appears to be that the FURL given to foolscap is a unicode string, and there is code inside foolscap which asserts that the connection hints it sees are normal strings (not unicode). The exception (as recorded in twistd.log) looks like: ``` Unhandled error in Deferred: Traceback (most recent call last): File "/usr/local/lib/python2.7/site-packages/foolscap/pb.py", line 856, in connectTo rc.startConnecting(self) File "/usr/local/lib/python2.7/site-packages/foolscap/reconnector.py", line 78, in startConnecting self._connect() File "/usr/local/lib/python2.7/site-packages/foolscap/reconnector.py", line 110, in _connect d = self._tub.getReference(self._url) File "/usr/local/lib/python2.7/site-packages/foolscap/pb.py", line 794, in getReference return defer.maybeDeferred(self._getReference, sturdyOrURL) --- <exception caught here> --- File "/usr/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred result = f(*args, **kw) File "/usr/local/lib/python2.7/site-packages/foolscap/pb.py", line 811, in _getReference d = self.getBrokerForTubRef(sturdy.getTubRef()) File "/usr/local/lib/python2.7/site-packages/foolscap/referenceable.py", line 804, in getTubRef return TubRef(self.tubID, self.locationHints) File "/usr/local/lib/python2.7/site-packages/foolscap/referenceable.py", line 837, in __init__ locationHints exceptions.AssertionError: [u'cz3uevcjrgwszq4ytvgisdw2nsdzgaxfan4nvh7uy5bxcbr7qv4q.b32.i2p'] ``` I think the code that loads the YAML file needs to do an `.encode("ascii")` on any of the FURLs it extracts, before passing them to foolscap. In addition, docs/configuration.rst had an invalid example for the `private/introducers.yaml` section (around line 920). It said: ``` introducers: petname2: furl = FURL2 ``` but that will result in a data structure like `introducers = {"petname2": "furl = FURL2"}`, which causes an exception at node startup time like this: ``` File "/usr/local/lib/python2.7/site-packages/allmydata/client.py", line 189, in __init__ self.init_introducer_clients() File "/usr/local/lib/python2.7/site-packages/allmydata/client.py", line 268, in init_introducer_clients ic = IntroducerClient(self.tub, introducer['furl'], exceptions.TypeError: string indices must be integers ``` Instead, the docs should recommend: ``` introducers: petname2: furl: FURL2 ```

tahoe-lafs added the

labels 2017-01-08 22:01:56 +00:00

tahoe-lafs added this to the 1.12.1 milestone 2017-01-08 22:01:56 +00:00

warner commented

2017-01-08 23:20:34 +00:00

Author

Owner

I assembled a patch: PR in https://github.com/tahoe-lafs/tahoe-lafs/pull/391 is testing now

I assembled a patch: PR in <https://github.com/tahoe-lafs/tahoe-lafs/pull/391> is testing now

Brian Warner <warner@lothar.com> commented

2017-01-09 07:00:35 +00:00

Author

Owner

In 582931b/trunk:

docs: recommend proper introducers.yaml syntax

add a test to exercise that this syntax is parseable, and another to point
out that the previously-recommended syntax was not

refs ticket:2862

In [582931b/trunk](/tahoe-lafs/trac-2024-07-25/commit/582931b1c25cb873853b1213495469051ca6ddbb): ``` docs: recommend proper introducers.yaml syntax add a test to exercise that this syntax is parseable, and another to point out that the previously-recommended syntax was not refs ticket:2862 ```

Brian Warner <warner@lothar.com> commented

2017-01-09 07:00:36 +00:00

Author

Owner

In 64e344b/trunk:

introducer.yaml: ascii-fy furls before use

This fixes the foolscap-rejects-unicode-furls problem which prevented a
correctly-formatted introducers.yaml from working.

closes ticket:2862

In [64e344b/trunk](/tahoe-lafs/trac-2024-07-25/commit/64e344b03a0ecc449db8b6ee6897765498a387e5): ``` introducer.yaml: ascii-fy furls before use This fixes the foolscap-rejects-unicode-furls problem which prevented a correctly-formatted introducers.yaml from working. closes ticket:2862 ```

tahoe-lafs added the

fixed

label 2017-01-09 07:00:36 +00:00

Brian Warner <warner@lothar.com> closed this issue

2017-01-09 07:00:36 +00:00

meejah commented

2017-01-10 17:38:39 +00:00

Author

Owner

A comment on https://blog.torproject.org/blog/tor-heart-tahoe-lafs (January 6, 2017 by Anonymous) sounds possibly related to this (can't link directly to the comment):

After a series of tests and debugging, I managed to have the test Tahoe-LAFS tor hidden service Storage-Client nodes connect to one another without a Introducer and only using the servers.yaml file. It appears there is a slight bug in the servers.yaml coding which makes it presently necessary to add a #comment between the storage node entrys to be successful. To save the readers the effort and time I'll paste in a working server.yaml file with fictional node names, onion addresses and id strings. However the spaces and syntax must be exactly copied if you want this to work for you after changing to your Storage nicnames, Onion addresses and the Storage key Id strings located after ann: in your introducer_default_cache.yaml file.

example of locating the desired announce string to copy to your server.yaml file
~/.tahoe/private/introducer_default_cache.yaml

search for

key_s: v0-admjvlr3czr4w7fact5flbp2r4hawtqg6yz1l542ajrcp2lkyn3r

and

- ann:
anonymous-storage-FURL: pb://hrshycb12ngpiz4qs2jevzvmjsk34zne@tor:abcqea4xsfgpmbac.onion:20100/sm1owyxjoi23fohajeqgdevh7dxrc1mr

Create, add these strings to your new server.yaml file keeping the #storage: comment line between each added storage node using a text editor.

storage:
v0-admjvlr3czr4w7fact5flbp2r4hawtqg6yz1l542ajrcp2lkyn3r:
ann:
nickname: Someone
anonymous-storage-FURL: pb://hrshycb12ngpiz4qs2jevzvmjsk34zne@tor:abcqea4xsfgpmbac.onion:20100/sm1owyxjoi23fohajeqgdevh7dxrc1mr
#storage:
v0-3rpxlixushufwhh4fqnxsitmk1ys4nmusgadjgrtjfb2lk1s34ic:
ann:
nickname: Someonelse
anonymous-storage-FURL: pb://4l26hnjjcoxnvrcrhhycinplpen6zhur@tor:l12nvioyiufz4cwb.onion:20200/szdpxq5uv2cmagkr2lzzcduiuawvjhnp

Next, comment the introducer in your tahoe.cfg file #introducer.furl = pb:/ and then start/restart your tahoe node which in a few seconds should then successfully only connect to your Tahoe-LAFS tor hidden service storage nodes without a Introducer.

A comment on <https://blog.torproject.org/blog/tor-heart-tahoe-lafs> (January 6, 2017 by Anonymous) sounds possibly related to this (can't link directly to the comment): After a series of tests and debugging, I managed to have the test Tahoe-LAFS tor hidden service Storage-Client nodes connect to one another without a Introducer and only using the servers.yaml file. It appears there is a slight bug in the servers.yaml coding which makes it presently necessary to add a #comment between the storage node entrys to be successful. To save the readers the effort and time I'll paste in a working server.yaml file with fictional node names, onion addresses and id strings. However the spaces and syntax must be exactly copied if you want this to work for you after changing to your Storage nicnames, Onion addresses and the Storage key Id strings located after ann: in your introducer_default_cache.yaml file. example of locating the desired announce string to copy to your server.yaml file ~/.tahoe/private/introducer_default_cache.yaml search for ``` key_s: v0-admjvlr3czr4w7fact5flbp2r4hawtqg6yz1l542ajrcp2lkyn3r ``` and ``` - ann: anonymous-storage-FURL: pb://hrshycb12ngpiz4qs2jevzvmjsk34zne@tor:abcqea4xsfgpmbac.onion:20100/sm1owyxjoi23fohajeqgdevh7dxrc1mr ``` Create, add these strings to your new server.yaml file keeping the #storage: comment line between each added storage node using a text editor. ``` storage: v0-admjvlr3czr4w7fact5flbp2r4hawtqg6yz1l542ajrcp2lkyn3r: ann: nickname: Someone anonymous-storage-FURL: pb://hrshycb12ngpiz4qs2jevzvmjsk34zne@tor:abcqea4xsfgpmbac.onion:20100/sm1owyxjoi23fohajeqgdevh7dxrc1mr #storage: v0-3rpxlixushufwhh4fqnxsitmk1ys4nmusgadjgrtjfb2lk1s34ic: ann: nickname: Someonelse anonymous-storage-FURL: pb://4l26hnjjcoxnvrcrhhycinplpen6zhur@tor:l12nvioyiufz4cwb.onion:20200/szdpxq5uv2cmagkr2lzzcduiuawvjhnp ``` Next, comment the introducer in your tahoe.cfg file #introducer.furl = pb:/ and then start/restart your tahoe node which in a few seconds should then successfully only connect to your Tahoe-LAFS tor hidden service storage nodes without a Introducer.