Ticket #1802: 1802-port-to-ticket999.darcs.patch

2 patches for repository https://tahoe-lafs.org/source/tahoe/ticket999-S3-backend:

Wed Jul 10 00:19:16 BST 2013  daira@jacaranda.org
  * Make introducer.furl unguessable [backport of git changeset 0a89b738bc05f17597555786b8f59dc05c46be0f/trunk to ticket999-S3-backend branch]. Closes #1802 for this branch.
  
  Previously, Introducers always used a swissnum of "introducer", so
  anyone who could learn the (public) tubid of the introducer would be
  able to connect to and use it. This changes new Introducers to use the
  same randomly-generated swissnum as clients and storage servers do, so
  that you absolutely must learn the introducer.furl from someone who
  knows it already before you can connect.
  
  This change also moves the location of the file that stores
  introducer.furl from BASEDIR/introducer.furl to
  BASEDIR/private/introducer.furl, since that's where we keep the private
  things. The first time an introducer is started with the new code, it
  will move any existing BASEDIR/introducer.furl into the new place.
  
  Note that this will not change the FURL of existing introducers: it will
  only affect newly created ones. When you change an introducer's FURL,
  you must also update all of the nodes (clients and storage servers)
  which connect to it, so upgrading it to an unguessable one isn't
  something we should do automatically.

Wed Jul 10 01:55:33 BST 2013  daira@jacaranda.org
  * Censor the introducer and helper furls' swissnums from the web welcome page [backport of 9be1a94043ce1518d70ebe8b1d0c44b54ec6652f/trunk to ticket999-S3-backend branch]. refs #1802


New patches:

[Make introducer.furl unguessable [backport of git changeset 0a89b738bc05f17597555786b8f59dc05c46be0f/trunk to ticket999-S3-backend branch]. Closes #1802 for this branch.
daira@jacaranda.org**20130709231916
 Ignore-this: 5ba82c60951c3a0c9cb51151c3758ddb
 
 Previously, Introducers always used a swissnum of "introducer", so
 anyone who could learn the (public) tubid of the introducer would be
 able to connect to and use it. This changes new Introducers to use the
 same randomly-generated swissnum as clients and storage servers do, so
 that you absolutely must learn the introducer.furl from someone who
 knows it already before you can connect.
 
 This change also moves the location of the file that stores
 introducer.furl from BASEDIR/introducer.furl to
 BASEDIR/private/introducer.furl, since that's where we keep the private
 things. The first time an introducer is started with the new code, it
 will move any existing BASEDIR/introducer.furl into the new place.
 
 Note that this will not change the FURL of existing introducers: it will
 only affect newly created ones. When you change an introducer's FURL,
 you must also update all of the nodes (clients and storage servers)
 which connect to it, so upgrading it to an unguessable one isn't
 something we should do automatically.
] {
hunk ./docs/configuration.rst 297
 
     This FURL tells the client how to connect to the introducer. Each
     Tahoe-LAFS grid is defined by an introducer. The introducer's FURL is
-    created by the introducer node and written into its base directory when
-    it starts, whereupon it should be published to everyone who wishes to
-    attach a client to that grid
+    created by the introducer node and written into its private base
+    directory when it starts, whereupon it should be published to everyone
+    who wishes to attach a client to that grid
 
 ``helper.furl = (FURL string, optional)``
 
hunk ./docs/configuration.rst 478
 
 The Introducer node maintains some different state than regular client nodes.
 
-``BASEDIR/introducer.furl``
+``BASEDIR/private/introducer.furl``
 
   This is generated the first time the introducer node is started, and used
   again on subsequent runs, to give the introduction service a persistent
hunk ./docs/frontends/CLI.rst 107
 
 "``tahoe create-introducer [NODEDIR]``" is used to create the Introducer node.
 This node provides introduction services and nothing else. When started, this
-node will produce an ``introducer.furl`` file, which should be published to all
-clients.
+node will produce a ``private/introducer.furl`` file, which should be
+published to all clients.
 
 "``tahoe create-key-generator [NODEDIR]``" is used to create a special
 "key-generation" service, which allows a client to offload their RSA key
hunk ./docs/running.rst 50
 name of the directory is up to you), ``cd`` into it, and run
 "``tahoe create-introducer .``". Now run the introducer using
 "``tahoe start .``". After it starts, it will write a file named
-``introducer.furl`` in that base directory. This file contains the URL
-the other nodes must use in order to connect to this introducer. (Note
-that "``tahoe run .``" doesn't work for introducers, this is a known
-issue: `#937 <http://allmydata.org/trac/tahoe-lafs/ticket/937>`_.)
+``introducer.furl`` into the ``private/`` subdirectory of that base
+directory. This file contains the URL the other nodes must use in order
+to connect to this introducer. (Note that "``tahoe run .``" doesn't
+work for introducers, this is a known issue: `#937
+<http://allmydata.org/trac/tahoe-lafs/ticket/937>`_.)
 
 The "``tahoe run``" command above will run the node in the foreground.
 On Unix, you can run it in the background instead by using the
hunk ./src/allmydata/introducer/server.py 2
 
-import time, os.path
+import time, os.path, textwrap
 from base64 import b32decode
 from zope.interface import implements
 from twisted.application import service
hunk ./src/allmydata/introducer/server.py 10
 import allmydata
 from allmydata import node
 from allmydata.util import log, rrefutil
+from allmydata.util.encodingutil import get_filesystem_encoding
 from allmydata.introducer.interfaces import \
      RIIntroducerPublisherAndSubscriberService
 
hunk ./src/allmydata/introducer/server.py 14
+class FurlFileConflictError(Exception):
+    pass
+
 class IntroducerNode(node.Node):
     PORTNUMFILE = "introducer.port"
     NODETYPE = "introducer"
hunk ./src/allmydata/introducer/server.py 34
         introducerservice = IntroducerService(self.basedir)
         self.add_service(introducerservice)
 
+        old_public_fn = os.path.join(self.basedir, "introducer.furl").encode(get_filesystem_encoding())
+        private_fn = os.path.join(self.basedir, "private", "introducer.furl").encode(get_filesystem_encoding())
+
+        if os.path.exists(old_public_fn):
+            if os.path.exists(private_fn):
+                msg = """This directory (%s) contains both an old public
+                'introducer.furl' file, and a new-style
+                'private/introducer.furl', so I cannot safely remove the old
+                one. Please make sure your desired FURL is in
+                private/introducer.furl, and remove the public file. If this
+                causes your Introducer's FURL to change, you need to inform
+                all grid members so they can update their tahoe.cfg.
+                """
+                raise FurlFileConflictError(textwrap.dedent(msg))
+            os.rename(old_public_fn, private_fn)
+
         d = self.when_tub_ready()
         def _publish(res):
hunk ./src/allmydata/introducer/server.py 52
-            self.introducer_url = self.tub.registerReference(introducerservice,
-                                                             "introducer")
-            self.log(" introducer is at %s" % self.introducer_url)
-            self.write_config("introducer.furl", self.introducer_url + "\n")
+            furl = self.tub.registerReference(introducerservice,
+                                              furlFile=private_fn)
+            self.log(" introducer is at %s" % furl, umid="qF2L9A")
+            self.introducer_url = furl # for tests
         d.addCallback(_publish)
         d.addErrback(log.err, facility="tahoe.init",
                      level=log.BAD, umid="UaNs9A")
hunk ./src/allmydata/test/test_introducer.py 13
 from twisted.application import service
 from allmydata.interfaces import InsufficientVersionError
 from allmydata.introducer.client import IntroducerClient
-from allmydata.introducer.server import IntroducerService
+from allmydata.introducer.server import IntroducerService, FurlFileConflictError
 # test compatibility with old introducer .tac files
 from allmydata.introducer import IntroducerNode
 from allmydata.util import pollmixin
hunk ./src/allmydata/test/test_introducer.py 24
         log.msg(msg, **kw)
 
 class Node(testutil.SignalMixin, unittest.TestCase):
-    def test_loadable(self):
-        basedir = "introducer.IntroducerNode.test_loadable"
+    def test_furl(self):
+        basedir = "introducer.IntroducerNode.test_furl"
         os.mkdir(basedir)
hunk ./src/allmydata/test/test_introducer.py 27
-        q = IntroducerNode(basedir)
+        public_fn = os.path.join(basedir, "introducer.furl")
+        private_fn = os.path.join(basedir, "private", "introducer.furl")
+        q1 = IntroducerNode(basedir)
         d = fireEventually(None)
hunk ./src/allmydata/test/test_introducer.py 31
-        d.addCallback(lambda res: q.startService())
-        d.addCallback(lambda res: q.when_tub_ready())
-        d.addCallback(lambda res: q.stopService())
+        d.addCallback(lambda res: q1.startService())
+        d.addCallback(lambda res: q1.when_tub_ready())
+        d.addCallback(lambda res: q1.stopService())
         d.addCallback(flushEventualQueue)
hunk ./src/allmydata/test/test_introducer.py 35
+        def _check_furl(res):
+            # new nodes create unguessable furls in private/introducer.furl
+            ifurl = fileutil.read(private_fn)
+            self.failUnless(ifurl)
+            ifurl = ifurl.strip()
+            self.failIf(ifurl.endswith("/introducer"), ifurl)
+
+            # old nodes created guessable furls in BASEDIR/introducer.furl
+            guessable = ifurl[:ifurl.rfind("/")] + "/introducer"
+            fileutil.write(public_fn, guessable+"\n", mode="w") # text
+
+            # if we see both files, throw an error
+            self.failUnlessRaises(FurlFileConflictError,
+                                  IntroducerNode, basedir)
+
+            # when we see only the public one, move it to private/ and use
+            # the existing furl instead of creating a new one
+            os.unlink(private_fn)
+            q2 = IntroducerNode(basedir)
+            d2 = fireEventually(None)
+            d2.addCallback(lambda res: q2.startService())
+            d2.addCallback(lambda res: q2.when_tub_ready())
+            d2.addCallback(lambda res: q2.stopService())
+            d2.addCallback(flushEventualQueue)
+            def _check_furl2(res):
+                self.failIf(os.path.exists(public_fn))
+                ifurl2 = fileutil.read(private_fn)
+                self.failUnless(ifurl2)
+                self.failUnlessEqual(ifurl2.strip(), guessable)
+            d2.addCallback(_check_furl2)
+            return d2
+        d.addCallback(_check_furl)
         return d
 
 class ServiceMixin:
hunk ./src/allmydata/test/test_runner.py 357
         c1 = os.path.join(basedir, "c1")
         HOTLINE_FILE = os.path.join(c1, "suicide_prevention_hotline")
         TWISTD_PID_FILE = os.path.join(c1, "twistd.pid")
-        INTRODUCER_FURL_FILE = os.path.join(c1, "introducer.furl")
+        INTRODUCER_FURL_FILE = os.path.join(c1, "private", "introducer.furl")
         PORTNUM_FILE = os.path.join(c1, "introducer.port")
         NODE_URL_FILE = os.path.join(c1, "node.url")
         CONFIG_FILE = os.path.join(c1, "tahoe.cfg")
hunk ./src/allmydata/test/test_runner.py 408
         d.addCallback(lambda res: self.poll(_node_has_started))
 
         def _started(res):
-            # read the introducer.furl and introducer.port files so we can check that their
-            # contents don't change on restart
+            # read the introducer.furl and introducer.port files so we can
+            # check that their contents don't change on restart
             self.furl = fileutil.read(INTRODUCER_FURL_FILE)
             self.failUnless(os.path.exists(PORTNUM_FILE))
             self.portnum = fileutil.read(PORTNUM_FILE)
}
[Censor the introducer and helper furls' swissnums from the web welcome page [backport of 9be1a94043ce1518d70ebe8b1d0c44b54ec6652f/trunk to ticket999-S3-backend branch]. refs #1802
daira@jacaranda.org**20130710005533
 Ignore-this: 7a1ca42a71c9ea63b1500439905a1d77
] {
hunk ./src/allmydata/test/test_introducer.py 16
 from allmydata.introducer.server import IntroducerService, FurlFileConflictError
 # test compatibility with old introducer .tac files
 from allmydata.introducer import IntroducerNode
-from allmydata.util import pollmixin
+from allmydata.util import pollmixin, fileutil
 import allmydata.test.common_util as testutil
 
 class LoggingMultiService(service.MultiService):
hunk ./src/allmydata/test/test_introducer.py 44
 
             # old nodes created guessable furls in BASEDIR/introducer.furl
             guessable = ifurl[:ifurl.rfind("/")] + "/introducer"
-            fileutil.write(public_fn, guessable+"\n", mode="w") # text
+            fileutil.write(public_fn, guessable+"\n") # text
 
             # if we see both files, throw an error
             self.failUnlessRaises(FurlFileConflictError,
hunk ./src/allmydata/test/test_web.py 74
 
 class FakeUploader(service.Service):
     name = "uploader"
+    def __init__(self):
+        self.helper_furl = None
+        self.helper_connected = False
     def upload(self, uploadable):
         d = uploadable.get_size()
         d.addCallback(lambda size: uploadable.read(size))
hunk ./src/allmydata/test/test_web.py 89
         d.addCallback(_got_data)
         return d
     def get_helper_info(self):
-        return (None, False)
+        return (self.helper_furl, self.helper_connected)
 
 class FakeIServer:
     def __init__(self, binaryserverid):
hunk ./src/allmydata/test/test_web.py 517
         d.addCallback(_check)
         return d
 
+    def test_introducer_status(self):
+        class MockIntroducerClient(object):
+            def __init__(self, connected):
+                self.connected = connected
+            def connected_to_introducer(self):
+                return self.connected
+
+        d = defer.succeed(None)
+
+        # introducer not connected, unguessable furl
+        def _set_introducer_not_connected_unguessable(ign):
+            self.s.introducer_furl = "pb://someIntroducer/secret"
+            self.s.introducer_client = MockIntroducerClient(False)
+            return self.GET("/")
+        d.addCallback(_set_introducer_not_connected_unguessable)
+        def _check_introducer_not_connected_unguessable(res):
+            html = res.replace('\n', ' ')
+            self.failUnlessIn('<div>Introducer: <span class="data-chars">pb://someIntroducer/[censored]</span></div>', html)
+            self.failIfIn('pb://someIntroducer/secret', html)
+            self.failUnlessIn('<div>Connected to introducer?: <span>no</span></div>', html)
+        d.addCallback(_check_introducer_not_connected_unguessable)
+
+        # introducer connected, unguessable furl
+        def _set_introducer_connected_unguessable(ign):
+            self.s.introducer_furl = "pb://someIntroducer/secret"
+            self.s.introducer_client = MockIntroducerClient(True)
+            return self.GET("/")
+        d.addCallback(_set_introducer_connected_unguessable)
+        def _check_introducer_connected_unguessable(res):
+            html = res.replace('\n', ' ')
+            self.failUnlessIn('<div>Introducer: <span class="data-chars">pb://someIntroducer/[censored]</span></div>', html)
+            self.failIfIn('pb://someIntroducer/secret', html)
+            self.failUnlessIn('<div>Connected to introducer?: <span>yes</span></div>', html)
+        d.addCallback(_check_introducer_connected_unguessable)
+
+        # introducer connected, guessable furl
+        def _set_introducer_connected_guessable(ign):
+            self.s.introducer_furl = "pb://someIntroducer/introducer"
+            self.s.introducer_client = MockIntroducerClient(True)
+            return self.GET("/")
+        d.addCallback(_set_introducer_connected_guessable)
+        def _check_introducer_connected_guessable(res):
+            html = res.replace('\n', ' ')
+            self.failUnlessIn('<div>Introducer: <span class="data-chars">pb://someIntroducer/introducer</span></div>', html)
+            self.failUnlessIn('<div>Connected to introducer?: <span>yes</span></div>', html)
+        d.addCallback(_check_introducer_connected_guessable)
+        return d
+
+    def test_helper_status(self):
+        d = defer.succeed(None)
+
+        # set helper furl to None
+        def _set_no_helper(ign):
+            self.s.uploader.helper_furl = None
+            self.s.uploader.helper_connnected = False
+            return self.GET("/")
+        d.addCallback(_set_no_helper)
+        def _check_no_helper(res):
+            html = res.replace('\n', ' ')
+            self.failUnlessIn('<li>Not running helper</li>', html)
+        d.addCallback(_check_no_helper)
+
+        # enable helper, not connected
+        def _set_helper_not_connected(ign):
+            self.s.uploader.helper_furl = "pb://someHelper/secret"
+            self.s.uploader.helper_connected = False
+            return self.GET("/")
+        d.addCallback(_set_helper_not_connected)
+        def _check_helper_not_connected(res):
+            html = res.replace('\n', ' ')
+            self.failUnlessIn('<div>Helper: <span>pb://someHelper/[censored]</span></div>', html)
+            self.failIfIn('pb://someHelper/secret', html)
+            self.failUnlessIn('<div>Connected to helper?: <span>no</span></div>', html)
+        d.addCallback(_check_helper_not_connected)
+
+        # enable helper, connected
+        def _set_helper_connected(ign):
+            self.s.uploader.helper_furl = "pb://someHelper/secret"
+            self.s.uploader.helper_connected = True
+            return self.GET("/")
+        d.addCallback(_set_helper_connected)
+        def _check_helper_connected(res):
+            html = res.replace('\n', ' ')
+            self.failUnlessIn('<div>Helper: <span>pb://someHelper/[censored]</span></div>', html)
+            self.failIfIn('pb://someHelper/secret', html)
+            self.failUnlessIn('<div>Connected to helper?: <span>yes</span></div>', html)
+        d.addCallback(_check_helper_connected)
+        return d
+
     def test_status(self):
         h = self.s.get_history()
         dl_num = h.list_all_download_statuses()[0].get_counter()
hunk ./src/allmydata/web/root.py 201
 
         return ctx.tag[ul]
 
-    def data_introducer_furl(self, ctx, data):
-        return self.client.introducer_furl
+    def data_introducer_furl_prefix(self, ctx, data):
+        ifurl = self.client.introducer_furl
+        # trim off the secret swissnum
+        (prefix, _, swissnum) = ifurl.rpartition("/")
+        if not ifurl:
+            return None
+        if swissnum == "introducer":
+            return ifurl
+        else:
+            return "%s/[censored]" % (prefix,)
+
     def data_connected_to_introducer(self, ctx, data):
         if self.client.connected_to_introducer():
             return "yes"
hunk ./src/allmydata/web/root.py 217
         return "no"
 
-    def data_helper_furl(self, ctx, data):
+    def data_helper_furl_prefix(self, ctx, data):
         try:
             uploader = self.client.getServiceNamed("uploader")
         except KeyError:
hunk ./src/allmydata/web/root.py 223
             return None
         furl, connected = uploader.get_helper_info()
-        return furl
+        if not furl:
+            return None
+        # trim off the secret swissnum
+        (prefix, _, swissnum) = furl.rpartition("/")
+        return "%s/[censored]" % (prefix,)
+
     def data_connected_to_helper(self, ctx, data):
         try:
             uploader = self.client.getServiceNamed("uploader")
hunk ./src/allmydata/web/welcome.xhtml 45
 
   <div>
     <n:attr name="class">connected-<n:invisible n:render="string" n:data="connected_to_introducer" /></n:attr>
-    <div>Introducer: <span class="data-chars" n:render="string" n:data="introducer_furl" /></div>
+    <div>Introducer: <span class="data-chars" n:render="string" n:data="introducer_furl_prefix" /></div>
     <div>Connected to introducer?: <span n:render="string" n:data="connected_to_introducer" /></div>
   </div>
 
hunk ./src/allmydata/web/welcome.xhtml 51
   <div>
     <n:attr name="class">connected-<n:invisible n:render="string" n:data="connected_to_helper" /></n:attr>
-    <div>Helper: <span n:render="string" n:data="helper_furl" /></div>
+    <div>Helper: <span n:render="string" n:data="helper_furl_prefix" /></div>
     <div>Connected to helper?: <span n:render="string" n:data="connected_to_helper" /></div>
   </div>
 
}

Context:

[TAG allmydata-tahoe-1.9.1.dev1
david-sarah@jacaranda.org**20120522033928
 Ignore-this: 3df82d8713cfe8f6a8245250c0d49b60
] 
Patch bundle hash:
2e5996948e3fc548c646510bbcb53329b599fc5d

-----------------------------134698832021294017231450416348
Content-Disposition: form-data; name="description"

Make introducer.furl unguessable and censor the introducer and helper furls' swissnums from the web welcome page [backport to ticket999-S3-backend branch]