2 patches for repository https://tahoe-lafs.org/source/tahoe/ticket999-S3-backend: Wed Jul 10 00:19:16 BST 2013 daira@jacaranda.org * Make introducer.furl unguessable [backport of git changeset 0a89b738bc05f17597555786b8f59dc05c46be0f/trunk to ticket999-S3-backend branch]. Closes #1802 for this branch. Previously, Introducers always used a swissnum of "introducer", so anyone who could learn the (public) tubid of the introducer would be able to connect to and use it. This changes new Introducers to use the same randomly-generated swissnum as clients and storage servers do, so that you absolutely must learn the introducer.furl from someone who knows it already before you can connect. This change also moves the location of the file that stores introducer.furl from BASEDIR/introducer.furl to BASEDIR/private/introducer.furl, since that's where we keep the private things. The first time an introducer is started with the new code, it will move any existing BASEDIR/introducer.furl into the new place. Note that this will not change the FURL of existing introducers: it will only affect newly created ones. When you change an introducer's FURL, you must also update all of the nodes (clients and storage servers) which connect to it, so upgrading it to an unguessable one isn't something we should do automatically. Wed Jul 10 01:55:33 BST 2013 daira@jacaranda.org * Censor the introducer and helper furls' swissnums from the web welcome page [backport of 9be1a94043ce1518d70ebe8b1d0c44b54ec6652f/trunk to ticket999-S3-backend branch]. refs #1802 New patches: [Make introducer.furl unguessable [backport of git changeset 0a89b738bc05f17597555786b8f59dc05c46be0f/trunk to ticket999-S3-backend branch]. Closes #1802 for this branch. daira@jacaranda.org**20130709231916 Ignore-this: 5ba82c60951c3a0c9cb51151c3758ddb Previously, Introducers always used a swissnum of "introducer", so anyone who could learn the (public) tubid of the introducer would be able to connect to and use it. This changes new Introducers to use the same randomly-generated swissnum as clients and storage servers do, so that you absolutely must learn the introducer.furl from someone who knows it already before you can connect. This change also moves the location of the file that stores introducer.furl from BASEDIR/introducer.furl to BASEDIR/private/introducer.furl, since that's where we keep the private things. The first time an introducer is started with the new code, it will move any existing BASEDIR/introducer.furl into the new place. Note that this will not change the FURL of existing introducers: it will only affect newly created ones. When you change an introducer's FURL, you must also update all of the nodes (clients and storage servers) which connect to it, so upgrading it to an unguessable one isn't something we should do automatically. ] { hunk ./docs/configuration.rst 297 This FURL tells the client how to connect to the introducer. Each Tahoe-LAFS grid is defined by an introducer. The introducer's FURL is - created by the introducer node and written into its base directory when - it starts, whereupon it should be published to everyone who wishes to - attach a client to that grid + created by the introducer node and written into its private base + directory when it starts, whereupon it should be published to everyone + who wishes to attach a client to that grid ``helper.furl = (FURL string, optional)`` hunk ./docs/configuration.rst 478 The Introducer node maintains some different state than regular client nodes. -``BASEDIR/introducer.furl`` +``BASEDIR/private/introducer.furl`` This is generated the first time the introducer node is started, and used again on subsequent runs, to give the introduction service a persistent hunk ./docs/frontends/CLI.rst 107 "``tahoe create-introducer [NODEDIR]``" is used to create the Introducer node. This node provides introduction services and nothing else. When started, this -node will produce an ``introducer.furl`` file, which should be published to all -clients. +node will produce a ``private/introducer.furl`` file, which should be +published to all clients. "``tahoe create-key-generator [NODEDIR]``" is used to create a special "key-generation" service, which allows a client to offload their RSA key hunk ./docs/running.rst 50 name of the directory is up to you), ``cd`` into it, and run "``tahoe create-introducer .``". Now run the introducer using "``tahoe start .``". After it starts, it will write a file named -``introducer.furl`` in that base directory. This file contains the URL -the other nodes must use in order to connect to this introducer. (Note -that "``tahoe run .``" doesn't work for introducers, this is a known -issue: `#937 `_.) +``introducer.furl`` into the ``private/`` subdirectory of that base +directory. This file contains the URL the other nodes must use in order +to connect to this introducer. (Note that "``tahoe run .``" doesn't +work for introducers, this is a known issue: `#937 +`_.) The "``tahoe run``" command above will run the node in the foreground. On Unix, you can run it in the background instead by using the hunk ./src/allmydata/introducer/server.py 2 -import time, os.path +import time, os.path, textwrap from base64 import b32decode from zope.interface import implements from twisted.application import service hunk ./src/allmydata/introducer/server.py 10 import allmydata from allmydata import node from allmydata.util import log, rrefutil +from allmydata.util.encodingutil import get_filesystem_encoding from allmydata.introducer.interfaces import \ RIIntroducerPublisherAndSubscriberService hunk ./src/allmydata/introducer/server.py 14 +class FurlFileConflictError(Exception): + pass + class IntroducerNode(node.Node): PORTNUMFILE = "introducer.port" NODETYPE = "introducer" hunk ./src/allmydata/introducer/server.py 34 introducerservice = IntroducerService(self.basedir) self.add_service(introducerservice) + old_public_fn = os.path.join(self.basedir, "introducer.furl").encode(get_filesystem_encoding()) + private_fn = os.path.join(self.basedir, "private", "introducer.furl").encode(get_filesystem_encoding()) + + if os.path.exists(old_public_fn): + if os.path.exists(private_fn): + msg = """This directory (%s) contains both an old public + 'introducer.furl' file, and a new-style + 'private/introducer.furl', so I cannot safely remove the old + one. Please make sure your desired FURL is in + private/introducer.furl, and remove the public file. If this + causes your Introducer's FURL to change, you need to inform + all grid members so they can update their tahoe.cfg. + """ + raise FurlFileConflictError(textwrap.dedent(msg)) + os.rename(old_public_fn, private_fn) + d = self.when_tub_ready() def _publish(res): hunk ./src/allmydata/introducer/server.py 52 - self.introducer_url = self.tub.registerReference(introducerservice, - "introducer") - self.log(" introducer is at %s" % self.introducer_url) - self.write_config("introducer.furl", self.introducer_url + "\n") + furl = self.tub.registerReference(introducerservice, + furlFile=private_fn) + self.log(" introducer is at %s" % furl, umid="qF2L9A") + self.introducer_url = furl # for tests d.addCallback(_publish) d.addErrback(log.err, facility="tahoe.init", level=log.BAD, umid="UaNs9A") hunk ./src/allmydata/test/test_introducer.py 13 from twisted.application import service from allmydata.interfaces import InsufficientVersionError from allmydata.introducer.client import IntroducerClient -from allmydata.introducer.server import IntroducerService +from allmydata.introducer.server import IntroducerService, FurlFileConflictError # test compatibility with old introducer .tac files from allmydata.introducer import IntroducerNode from allmydata.util import pollmixin hunk ./src/allmydata/test/test_introducer.py 24 log.msg(msg, **kw) class Node(testutil.SignalMixin, unittest.TestCase): - def test_loadable(self): - basedir = "introducer.IntroducerNode.test_loadable" + def test_furl(self): + basedir = "introducer.IntroducerNode.test_furl" os.mkdir(basedir) hunk ./src/allmydata/test/test_introducer.py 27 - q = IntroducerNode(basedir) + public_fn = os.path.join(basedir, "introducer.furl") + private_fn = os.path.join(basedir, "private", "introducer.furl") + q1 = IntroducerNode(basedir) d = fireEventually(None) hunk ./src/allmydata/test/test_introducer.py 31 - d.addCallback(lambda res: q.startService()) - d.addCallback(lambda res: q.when_tub_ready()) - d.addCallback(lambda res: q.stopService()) + d.addCallback(lambda res: q1.startService()) + d.addCallback(lambda res: q1.when_tub_ready()) + d.addCallback(lambda res: q1.stopService()) d.addCallback(flushEventualQueue) hunk ./src/allmydata/test/test_introducer.py 35 + def _check_furl(res): + # new nodes create unguessable furls in private/introducer.furl + ifurl = fileutil.read(private_fn) + self.failUnless(ifurl) + ifurl = ifurl.strip() + self.failIf(ifurl.endswith("/introducer"), ifurl) + + # old nodes created guessable furls in BASEDIR/introducer.furl + guessable = ifurl[:ifurl.rfind("/")] + "/introducer" + fileutil.write(public_fn, guessable+"\n", mode="w") # text + + # if we see both files, throw an error + self.failUnlessRaises(FurlFileConflictError, + IntroducerNode, basedir) + + # when we see only the public one, move it to private/ and use + # the existing furl instead of creating a new one + os.unlink(private_fn) + q2 = IntroducerNode(basedir) + d2 = fireEventually(None) + d2.addCallback(lambda res: q2.startService()) + d2.addCallback(lambda res: q2.when_tub_ready()) + d2.addCallback(lambda res: q2.stopService()) + d2.addCallback(flushEventualQueue) + def _check_furl2(res): + self.failIf(os.path.exists(public_fn)) + ifurl2 = fileutil.read(private_fn) + self.failUnless(ifurl2) + self.failUnlessEqual(ifurl2.strip(), guessable) + d2.addCallback(_check_furl2) + return d2 + d.addCallback(_check_furl) return d class ServiceMixin: hunk ./src/allmydata/test/test_runner.py 357 c1 = os.path.join(basedir, "c1") HOTLINE_FILE = os.path.join(c1, "suicide_prevention_hotline") TWISTD_PID_FILE = os.path.join(c1, "twistd.pid") - INTRODUCER_FURL_FILE = os.path.join(c1, "introducer.furl") + INTRODUCER_FURL_FILE = os.path.join(c1, "private", "introducer.furl") PORTNUM_FILE = os.path.join(c1, "introducer.port") NODE_URL_FILE = os.path.join(c1, "node.url") CONFIG_FILE = os.path.join(c1, "tahoe.cfg") hunk ./src/allmydata/test/test_runner.py 408 d.addCallback(lambda res: self.poll(_node_has_started)) def _started(res): - # read the introducer.furl and introducer.port files so we can check that their - # contents don't change on restart + # read the introducer.furl and introducer.port files so we can + # check that their contents don't change on restart self.furl = fileutil.read(INTRODUCER_FURL_FILE) self.failUnless(os.path.exists(PORTNUM_FILE)) self.portnum = fileutil.read(PORTNUM_FILE) } [Censor the introducer and helper furls' swissnums from the web welcome page [backport of 9be1a94043ce1518d70ebe8b1d0c44b54ec6652f/trunk to ticket999-S3-backend branch]. refs #1802 daira@jacaranda.org**20130710005533 Ignore-this: 7a1ca42a71c9ea63b1500439905a1d77 ] { hunk ./src/allmydata/test/test_introducer.py 16 from allmydata.introducer.server import IntroducerService, FurlFileConflictError # test compatibility with old introducer .tac files from allmydata.introducer import IntroducerNode -from allmydata.util import pollmixin +from allmydata.util import pollmixin, fileutil import allmydata.test.common_util as testutil class LoggingMultiService(service.MultiService): hunk ./src/allmydata/test/test_introducer.py 44 # old nodes created guessable furls in BASEDIR/introducer.furl guessable = ifurl[:ifurl.rfind("/")] + "/introducer" - fileutil.write(public_fn, guessable+"\n", mode="w") # text + fileutil.write(public_fn, guessable+"\n") # text # if we see both files, throw an error self.failUnlessRaises(FurlFileConflictError, hunk ./src/allmydata/test/test_web.py 74 class FakeUploader(service.Service): name = "uploader" + def __init__(self): + self.helper_furl = None + self.helper_connected = False def upload(self, uploadable): d = uploadable.get_size() d.addCallback(lambda size: uploadable.read(size)) hunk ./src/allmydata/test/test_web.py 89 d.addCallback(_got_data) return d def get_helper_info(self): - return (None, False) + return (self.helper_furl, self.helper_connected) class FakeIServer: def __init__(self, binaryserverid): hunk ./src/allmydata/test/test_web.py 517 d.addCallback(_check) return d + def test_introducer_status(self): + class MockIntroducerClient(object): + def __init__(self, connected): + self.connected = connected + def connected_to_introducer(self): + return self.connected + + d = defer.succeed(None) + + # introducer not connected, unguessable furl + def _set_introducer_not_connected_unguessable(ign): + self.s.introducer_furl = "pb://someIntroducer/secret" + self.s.introducer_client = MockIntroducerClient(False) + return self.GET("/") + d.addCallback(_set_introducer_not_connected_unguessable) + def _check_introducer_not_connected_unguessable(res): + html = res.replace('\n', ' ') + self.failUnlessIn('
Introducer: pb://someIntroducer/[censored]
', html) + self.failIfIn('pb://someIntroducer/secret', html) + self.failUnlessIn('
Connected to introducer?: no
', html) + d.addCallback(_check_introducer_not_connected_unguessable) + + # introducer connected, unguessable furl + def _set_introducer_connected_unguessable(ign): + self.s.introducer_furl = "pb://someIntroducer/secret" + self.s.introducer_client = MockIntroducerClient(True) + return self.GET("/") + d.addCallback(_set_introducer_connected_unguessable) + def _check_introducer_connected_unguessable(res): + html = res.replace('\n', ' ') + self.failUnlessIn('
Introducer: pb://someIntroducer/[censored]
', html) + self.failIfIn('pb://someIntroducer/secret', html) + self.failUnlessIn('
Connected to introducer?: yes
', html) + d.addCallback(_check_introducer_connected_unguessable) + + # introducer connected, guessable furl + def _set_introducer_connected_guessable(ign): + self.s.introducer_furl = "pb://someIntroducer/introducer" + self.s.introducer_client = MockIntroducerClient(True) + return self.GET("/") + d.addCallback(_set_introducer_connected_guessable) + def _check_introducer_connected_guessable(res): + html = res.replace('\n', ' ') + self.failUnlessIn('
Introducer: pb://someIntroducer/introducer
', html) + self.failUnlessIn('
Connected to introducer?: yes
', html) + d.addCallback(_check_introducer_connected_guessable) + return d + + def test_helper_status(self): + d = defer.succeed(None) + + # set helper furl to None + def _set_no_helper(ign): + self.s.uploader.helper_furl = None + self.s.uploader.helper_connnected = False + return self.GET("/") + d.addCallback(_set_no_helper) + def _check_no_helper(res): + html = res.replace('\n', ' ') + self.failUnlessIn('
  • Not running helper
    • ', html) + d.addCallback(_check_no_helper) + + # enable helper, not connected + def _set_helper_not_connected(ign): + self.s.uploader.helper_furl = "pb://someHelper/secret" + self.s.uploader.helper_connected = False + return self.GET("/") + d.addCallback(_set_helper_not_connected) + def _check_helper_not_connected(res): + html = res.replace('\n', ' ') + self.failUnlessIn('
    Helper: pb://someHelper/[censored]
    ', html) + self.failIfIn('pb://someHelper/secret', html) + self.failUnlessIn('
    Connected to helper?: no
    ', html) + d.addCallback(_check_helper_not_connected) + + # enable helper, connected + def _set_helper_connected(ign): + self.s.uploader.helper_furl = "pb://someHelper/secret" + self.s.uploader.helper_connected = True + return self.GET("/") + d.addCallback(_set_helper_connected) + def _check_helper_connected(res): + html = res.replace('\n', ' ') + self.failUnlessIn('
    Helper: pb://someHelper/[censored]
    ', html) + self.failIfIn('pb://someHelper/secret', html) + self.failUnlessIn('
    Connected to helper?: yes
    ', html) + d.addCallback(_check_helper_connected) + return d + def test_status(self): h = self.s.get_history() dl_num = h.list_all_download_statuses()[0].get_counter() hunk ./src/allmydata/web/root.py 201 return ctx.tag[ul] - def data_introducer_furl(self, ctx, data): - return self.client.introducer_furl + def data_introducer_furl_prefix(self, ctx, data): + ifurl = self.client.introducer_furl + # trim off the secret swissnum + (prefix, _, swissnum) = ifurl.rpartition("/") + if not ifurl: + return None + if swissnum == "introducer": + return ifurl + else: + return "%s/[censored]" % (prefix,) + def data_connected_to_introducer(self, ctx, data): if self.client.connected_to_introducer(): return "yes" hunk ./src/allmydata/web/root.py 217 return "no" - def data_helper_furl(self, ctx, data): + def data_helper_furl_prefix(self, ctx, data): try: uploader = self.client.getServiceNamed("uploader") except KeyError: hunk ./src/allmydata/web/root.py 223 return None furl, connected = uploader.get_helper_info() - return furl + if not furl: + return None + # trim off the secret swissnum + (prefix, _, swissnum) = furl.rpartition("/") + return "%s/[censored]" % (prefix,) + def data_connected_to_helper(self, ctx, data): try: uploader = self.client.getServiceNamed("uploader") hunk ./src/allmydata/web/welcome.xhtml 45
    connected- -
    Introducer:
    +
    Introducer:
    Connected to introducer?:
    hunk ./src/allmydata/web/welcome.xhtml 51
    connected- -
    Helper:
    +
    Helper:
    Connected to helper?:
    } Context: [TAG allmydata-tahoe-1.9.1.dev1 david-sarah@jacaranda.org**20120522033928 Ignore-this: 3df82d8713cfe8f6a8245250c0d49b60 ] Patch bundle hash: 2e5996948e3fc548c646510bbcb53329b599fc5d -----------------------------134698832021294017231450416348 Content-Disposition: form-data; name="description" Make introducer.furl unguessable and censor the introducer and helper furls' swissnums from the web welcome page [backport to ticket999-S3-backend branch]