Wed Apr  1 10:15:17 CEST 2009  francois@ctrlaltdel.ch
  * CLI: Improved handling of non-ascii filenames
  
  The patch allows users to pass non-ascii filenames to the Tahoe CLI, it stores
  aliase names as UTF-8 and fix handling of non-ascii filenames on commands like
  tahoe cp or tahoe backup.
  
  As Tahoe internal filename storage is unicode, we have to decode the parameters
  we received according to the alleged filesystem encoding which varies depending
  on the platform.
  
  On Windows and MacOS X, the filesystem make uses of unicode strings so no
  decoding should ever happen.
  
  On Linux, the filesystem stores only bytestrings we have to be interepreted
  according to the current locale configuration. This operation can fail in which
  case an error is raised.
  

New patches:

[CLI: Improved handling of non-ascii filenames
francois@ctrlaltdel.ch**20090401081517
 
 The patch allows users to pass non-ascii filenames to the Tahoe CLI, it stores
 aliase names as UTF-8 and fix handling of non-ascii filenames on commands like
 tahoe cp or tahoe backup.
 
 As Tahoe internal filename storage is unicode, we have to decode the parameters
 we received according to the alleged filesystem encoding which varies depending
 on the platform.
 
 On Windows and MacOS X, the filesystem make uses of unicode strings so no
 decoding should ever happen.
 
 On Linux, the filesystem stores only bytestrings we have to be interepreted
 according to the current locale configuration. This operation can fail in which
 case an error is raised.
 
] {
hunk ./docs/frontends/CLI.txt 94
-In Tahoe v1.3.0, passing non-ascii characters to the cli is not guaranteed to
-work, although it might work on your platform, especially if your platform
-uses utf-8 encoding.
+As of Tahoe v1.3.1, filenames containing non-ascii characters are
+supported on the commande line if your terminal is correctly configured
+for UTF-8 support. This is usually the case on moderns GNU/Linux
+distributions.
+
+If your terminal doesn't support UTF-8, you will still be able to list
+directories but non-ascii characters will be replaced by a question mark
+(?) on display.
+
+Reading from and writing to files whose name contain non-ascii
+characters is also supported when your system correctly understand them.
+Under Unix, this is usually handled by locale settings. If Tahoe cannot
+correctly decode a filename, it will raise an error. In such case,
+you'll need to correct the name of your file, possibly with help from
+tools such as convmv.
hunk ./src/allmydata/scripts/cli.py 4
+from allmydata.util.stringutils import argv_to_unicode
hunk ./src/allmydata/scripts/cli.py 53
-        self.where = where
+        self.where = argv_to_unicode(where)
hunk ./src/allmydata/scripts/cli.py 58
-        self.alias = alias
+        self.alias = argv_to_unicode(alias)
hunk ./src/allmydata/scripts/cli.py 68
-        self.alias = alias
+        self.alias = argv_to_unicode(alias)
hunk ./src/allmydata/scripts/cli.py 87
-        self.where = where
+        self.where = argv_to_unicode(where)
hunk ./src/allmydata/scripts/cli.py 98
-        self.from_file = arg1
-        self.to_file = arg2
+        self.from_file = argv_to_unicode(arg1)
+
+        if arg2:
+            self.to_file = argv_to_unicode(arg2)
+        else:
+            self.to_file = None
+
hunk ./src/allmydata/scripts/cli.py 140
-            self.from_file = arg1
-            self.to_file = arg2
+            self.from_file = argv_to_unicode(arg1)
+            self.to_file =  argv_to_unicode(arg2)
hunk ./src/allmydata/scripts/cli.py 143
-            self.from_file = arg1 # might be "-"
+            self.from_file = argv_to_unicode(arg1) # might be "-"
hunk ./src/allmydata/scripts/cli.py 148
-        if self.from_file == "-":
+        if self.from_file == u"-":
hunk ./src/allmydata/scripts/cli.py 185
-        self.sources = args[:-1]
-        self.destination = args[-1]
+        self.sources = map(argv_to_unicode, args[:-1])
+        self.destination = argv_to_unicode(args[-1])
hunk ./src/allmydata/scripts/cli.py 190
-        self.where = where
+        self.where = argv_to_unicode(where)
hunk ./src/allmydata/scripts/cli.py 197
-        self.from_file = frompath
-        self.to_file = topath
+        self.from_file = argv_to_unicode(frompath)
+        self.to_file = argv_to_unicode(topath)
hunk ./src/allmydata/scripts/cli.py 205
-        self.from_file = frompath
-        self.to_file = topath
+        self.from_file = argv_to_unicode(frompath)
+        self.to_file = argv_to_unicode(topath)
hunk ./src/allmydata/scripts/cli.py 230
-        self.from_dir = localdir
-        self.to_dir = topath
+        self.from_dir = argv_to_unicode(localdir)
+        self.to_dir = argv_to_unicode(topath)
hunk ./src/allmydata/scripts/cli.py 279
-        self.where = where
+        self.where = argv_to_unicode(where)
hunk ./src/allmydata/scripts/cli.py 294
-        self.where = where
+        self.where = argv_to_unicode(where)
hunk ./src/allmydata/scripts/cli.py 306
-        self.where = where
+        self.where = argv_to_unicode(where)
hunk ./src/allmydata/scripts/cli.py 321
-        self.where = where
+        self.where = argv_to_unicode(where)
hunk ./src/allmydata/scripts/cli.py 337
-        self.where = where
+        self.where = argv_to_unicode(where)
hunk ./src/allmydata/scripts/common.py 3
+import codecs
hunk ./src/allmydata/scripts/common.py 5
-
+from allmydata.util.stringutils import unicode_to_url
hunk ./src/allmydata/scripts/common.py 104
-        f = open(aliasfile, "r")
+        f = codecs.open(aliasfile, "r", "utf-8")
hunk ./src/allmydata/scripts/common.py 111
-            cap = cap.strip()
+            cap = cap.strip().encode('ascii')
hunk ./src/allmydata/scripts/common.py 167
-    return "/".join([urllib.quote(s) for s in segments])
+    return "/".join([urllib.quote(unicode_to_url(s)) for s in segments])
hunk ./src/allmydata/scripts/tahoe_add_alias.py 3
+import codecs
+import sys
hunk ./src/allmydata/scripts/tahoe_add_alias.py 8
+from allmydata.util.stringutils import unicode_to_stdout
hunk ./src/allmydata/scripts/tahoe_add_alias.py 58
-    f = open(aliasfile, "a")
+    f = codecs.open(aliasfile, "a", "utf-8")
hunk ./src/allmydata/scripts/tahoe_add_alias.py 61
-    print >>stdout, "Alias '%s' created" % (alias,)
+    print >>stdout, "Alias '%s' created" % (unicode_to_stdout(alias),)
hunk ./src/allmydata/scripts/tahoe_backup.py 7
+import sys
hunk ./src/allmydata/scripts/tahoe_backup.py 13
+from allmydata.util.stringutils import fs_to_unicode, unicode_to_fs, unicode_to_stdout
+from allmydata.util.assertutil import precondition
+from twisted.python import usage
hunk ./src/allmydata/scripts/tahoe_backup.py 252
-            print >>self.options.stdout, msg
+            print >>self.options.stdout, unicode_to_stdout(msg)
hunk ./src/allmydata/scripts/tahoe_backup.py 255
+        precondition(isinstance(localpath, unicode), localpath)
hunk ./src/allmydata/scripts/tahoe_backup.py 264
-        for child in self.options.filter_listdir(os.listdir(localpath)):
+        for child in self.options.filter_listdir(os.listdir(unicode_to_fs(localpath))):
+            child = fs_to_unicode(child)
hunk ./src/allmydata/scripts/tahoe_backup.py 351
+        precondition(isinstance(childpath, unicode), childpath)
+
hunk ./src/allmydata/scripts/tahoe_backup.py 361
-            infileobj = open(os.path.expanduser(childpath), "rb")
+            infileobj = open(unicode_to_fs(os.path.expanduser(childpath)), "rb")
hunk ./src/allmydata/scripts/tahoe_cp.py 7
+import sys
hunk ./src/allmydata/scripts/tahoe_cp.py 11
+from twisted.python import usage
+from allmydata.util.stringutils import fs_to_unicode, unicode_to_fs, unicode_to_url
+from allmydata.util.assertutil import precondition
hunk ./src/allmydata/scripts/tahoe_cp.py 76
+        precondition(isinstance(pathname, unicode), pathname)
hunk ./src/allmydata/scripts/tahoe_cp.py 87
+        precondition(isinstance(pathname, unicode), pathname)
hunk ./src/allmydata/scripts/tahoe_cp.py 100
+        precondition(isinstance(pathname, unicode), pathname)
hunk ./src/allmydata/scripts/tahoe_cp.py 114
+        precondition(isinstance(pathname, unicode), pathname)
+
hunk ./src/allmydata/scripts/tahoe_cp.py 124
-        children = os.listdir(self.pathname)
+        children = os.listdir(unicode_to_fs(self.pathname))
hunk ./src/allmydata/scripts/tahoe_cp.py 126
+            n = fs_to_unicode(n)
hunk ./src/allmydata/scripts/tahoe_cp.py 142
+        precondition(isinstance(pathname, unicode), pathname)
+
hunk ./src/allmydata/scripts/tahoe_cp.py 152
-        children = os.listdir(self.pathname)
+        children = os.listdir(unicode_to_fs(self.pathname))
hunk ./src/allmydata/scripts/tahoe_cp.py 154
+            n = fs_to_unicode(n)
hunk ./src/allmydata/scripts/tahoe_cp.py 176
+        precondition(isinstance(name, unicode), name)
hunk ./src/allmydata/scripts/tahoe_cp.py 178
-        outf = open(pathname, "wb")
+        outf = open(unicode_to_fs(pathname), "wb")
hunk ./src/allmydata/scripts/tahoe_cp.py 367
-                                                   urllib.quote(name.encode('utf-8'))])
+                                                   urllib.quote(unicode_to_url(name))])
hunk ./src/allmydata/scripts/tahoe_ls.py 6
+from allmydata.util.stringutils import unicode_to_stdout
hunk ./src/allmydata/scripts/tahoe_ls.py 116
-        line.append(name + classify)
+        line.append(unicode_to_stdout(name) + classify)
hunk ./src/allmydata/scripts/tahoe_manifest.py 83
-                    print >>stdout, d["cap"], "/".join([p.encode("utf-8")
+                    print >>stdout, d["cap"], "/".join([unicode_to_stdout(p)
hunk ./src/allmydata/scripts/tahoe_mkdir.py 5
+from allmydata.util.stringutils import unicode_to_url
hunk ./src/allmydata/scripts/tahoe_mkdir.py 35
-                                           urllib.quote(path))
+                                           urllib.quote(unicode_to_url(path)))
hunk ./src/allmydata/test/test_cli.py 3
+import sys
hunk ./src/allmydata/test/test_cli.py 522
+        d.addCallback(lambda res: self.do_cli("create-alias", "études"))
+        def _check_create_unicode((rc,stdout,stderr)):
+            self.failUnlessEqual(rc, 0)
+            self.failIf(stderr)
+
+            # If stdout only supports ascii, accentuated characters are
+            # being replaced by '?'
+            if sys.stdout.encoding == "ANSI_X3.4-1968":
+                self.failUnless("Alias '?tudes' created" in stdout)
+            else:
+                self.failUnless("Alias 'études' created" in stdout)
+
+            aliases = get_aliases(self.get_clientdir())
+            self.failUnless(aliases[u"études"].startswith("URI:DIR2:"))
+        d.addCallback(_check_create_unicode)
+
+        d.addCallback(lambda res: self.do_cli("ls", "études:"))
+        def _check_ls1((rc, stdout, stderr)):
+            self.failUnlessEqual(rc, 0)
+            self.failIf(stderr)
+
+            self.failUnlessEqual(stdout, "")
+        d.addCallback(_check_ls1)
+
+        d.addCallback(lambda res: self.do_cli("put", "-", "études:uploaded.txt",
+          stdin="Blah blah blah"))
+
+        d.addCallback(lambda res: self.do_cli("ls", "études:"))
+        def _check_ls2((rc, stdout, stderr)):
+            self.failUnlessEqual(rc, 0)
+            self.failIf(stderr)
+
+            self.failUnlessEqual(stdout, "uploaded.txt\n")
+        d.addCallback(_check_ls2)
+
hunk ./src/allmydata/test/test_cli.py 778
+    def test_immutable_from_file_unicode(self):
+        # tahoe put file.txt "à trier.txt"
+        self.basedir = os.path.dirname(self.mktemp())
+        self.set_up_grid()
+
+        rel_fn = os.path.join(self.basedir, "DATAFILE")
+        abs_fn = os.path.abspath(rel_fn)
+        # we make the file small enough to fit in a LIT file, for speed
+        DATA = "short file"
+        f = open(rel_fn, "w")
+        f.write(DATA)
+        f.close()
+
+        d = self.do_cli("create-alias", "tahoe")
+
+        d.addCallback(lambda res:
+                      self.do_cli("put", rel_fn, "à trier.txt"))
+        def _uploaded((rc,stdout,stderr)):
+            readcap = stdout.strip()
+            self.failUnless(readcap.startswith("URI:LIT:"))
+            self.failUnless("201 Created" in stderr, stderr)
+            self.readcap = readcap
+        d.addCallback(_uploaded)
+
+        d.addCallback(lambda res:
+                      self.do_cli("get", "tahoe:à trier.txt"))
+        d.addCallback(lambda (rc,stdout,stderr):
+                      self.failUnlessEqual(stdout, DATA))
+
+        return d
+
hunk ./src/allmydata/test/test_cli.py 865
+        d = self.do_cli("create-alias", "tahoe")
+
+        # Use unicode strings when calling os functions
+        if sys.getfilesystemencoding() == "ANSI_X3.4-1968":
+            fn1 = os.path.join(self.basedir, u"Artonwall")
+        else:
+            fn1 = os.path.join(self.basedir, u"Ärtonwall")
hunk ./src/allmydata/test/test_cli.py 873
-        fn1 = os.path.join(self.basedir, "Ärtonwall")
hunk ./src/allmydata/test/test_cli.py 875
+        d.addCallback(lambda res: self.do_cli("cp", fn1.encode('utf-8'), "tahoe:Ärtonwall"))
+
+        d.addCallback(lambda res: self.do_cli("get", "tahoe:Ärtonwall"))
+        d.addCallback(lambda (rc,out,err): self.failUnlessEqual(out, DATA1))
hunk ./src/allmydata/test/test_cli.py 880
-        fn2 = os.path.join(self.basedir, "Metallica")
+
+        fn2 = os.path.join(self.basedir, u"Metallica")
hunk ./src/allmydata/test/test_cli.py 888
-        d = self.do_cli("create-alias", "tahoe")
-        d.addCallback(lambda res: self.do_cli("cp", fn1, "tahoe:"))
-        d.addCallback(lambda res: self.do_cli("cp", fn2, "tahoe:"))
-
-        d.addCallback(lambda res: self.do_cli("get", "tahoe:Ärtonwall"))
-        d.addCallback(lambda (rc,out,err): self.failUnlessEqual(out, DATA1))
+        d.addCallback(lambda res: self.do_cli("cp", fn2.encode('utf-8'), "tahoe:"))
hunk ./src/allmydata/test/test_cli.py 893
+        d.addCallback(lambda res: self.do_cli("ls", "tahoe:"))
+
hunk ./src/allmydata/test/test_cli.py 896
-    test_unicode_filename.todo = "This behavior is not yet supported, although it does happen to work (for reasons that are ill-understood) on many platforms.  See issue ticket #534."
hunk ./src/allmydata/test/test_cli.py 914
+class Mkdir(GridTestMixin, CLITestMixin, unittest.TestCase):
+    def test_unicode_mkdir(self):
+        self.basedir = os.path.dirname(self.mktemp())
+        self.set_up_grid()
+
+        d = self.do_cli("create-alias", "tahoe")
+        d.addCallback(lambda res: self.do_cli("mkdir", "tahoe:Motörhead"))
+
+        return d
+ 
+
hunk ./src/allmydata/test/test_cli.py 959
+        if sys.getfilesystemencoding() == "ANSI_X3.4-1968":
+            self.writeto(u"parent/artonwall.txt", "Marmelade Jacuzzi")
+        else:
+            self.writeto(u"parent/ärtonwall.txt", "Marmelade Jacuzzi")
+
hunk ./src/allmydata/test/test_cli.py 988
-            # foo.txt, bar.txt, blah.txt
-            self.failUnlessEqual(fu, 3)
+            # foo.txt, bar.txt, blah.txt, ärtonwall.txt
+            self.failUnlessEqual(fu, 4)
hunk ./src/allmydata/test/test_cli.py 1038
-                # foo.txt, bar.txt, blah.txt
+                # foo.txt, bar.txt, blah.txt, ärtonwall.txt
hunk ./src/allmydata/test/test_cli.py 1040
-                self.failUnlessEqual(fr, 3)
+                self.failUnlessEqual(fr, 4)
hunk ./src/allmydata/test/test_cli.py 1068
-                self.failUnlessEqual(fchecked, 3)
+                self.failUnlessEqual(fchecked, 4)
hunk ./src/allmydata/test/test_cli.py 1070
-                self.failUnlessEqual(fr, 3)
+                self.failUnlessEqual(fr, 4)
hunk ./src/allmydata/test/test_cli.py 1116
-                # old bar.txt
-                self.failUnlessEqual(fr, 1)
+                # old bar.txt, ärtonwall.txt
+                self.failUnlessEqual(fr, 2)
hunk ./src/allmydata/test/test_cli.py 1156
-            self.failUnlessEqual(fu, 5)
+            self.failUnlessEqual(fu, 6)
addfile ./src/allmydata/util/stringutils.py
hunk ./src/allmydata/util/stringutils.py 1
+"""
+Functions used to convert inputs from whatever encoding used in the system to
+unicode and back.
+
+TODO:
+  * Accept two cli arguments --argv-encoding and --filesystem-encoding
+"""
+
+import sys
+from allmydata.util.assertutil import precondition
+from twisted.python import usage
+
+def argv_to_unicode(s):
+    """
+    Decode given argv element to unicode.
+    """
+    # sys.argv encoding detection in Python is not trivial so utf-8 is
+    # currently used by default and an informative error message is given if
+    # the argument cannot be correctly decoded.
+
+    precondition(isinstance(s, str), s)
+    try:
+        return unicode(s, 'utf-8')
+    except UnicodeEncodeError:
+        raise usageError("Argument '%s' cannot be decoded as UTF-8." % s)
+
+def fs_to_unicode(s):
+    """
+    Decode a filename (or a directory name) to unicode using the same encoding
+    as the filesystem.
+    """
+    # Filename encoding detection is a little bit better thanks to
+    # getfilesystemencoding() in the sys module. However, filenames can be
+    # encoded using another encoding than the one used on the filesystem.
+    
+    precondition(isinstance(s, str), s)
+    encoding = sys.getfilesystemencoding()
+    try:
+        return unicode(s, encoding)
+    except UnicodeDecodeError:
+        raise usage.UsageError("Filename '%s' cannot be decoded using the current encoding of your filesystem (%s). Please rename this file." % (s, encoding))
+
+def unicode_to_fs(s):
+    """
+    Encode an unicode object used in file or directoy name.
+    """
+
+    precondition(isinstance(s, unicode), s)
+    encoding = sys.getfilesystemencoding()
+    try:
+        return s.encode(encoding)
+    except UnicodeEncodeError:
+        raise usage.UsageError("Filename '%s' cannot be encoded using the current encoding of your filesystem (%s). Please configure your locale correctly or rename this file." % (s, encoding))
+
+def unicode_to_url(s):
+    """
+    Encode an unicode object used in an URL.
+    """
+    # According to RFC 2718, non-ascii characters in url's must be UTF-8 encoded.
+
+    precondition(isinstance(s, unicode), s)
+    return s.encode('utf-8')
+
+def unicode_to_stdout(s):
+    """
+    Encode an unicode object for representation on stdout.
+    """
+
+    precondition(isinstance(s, unicode), s)
+    return s.encode(sys.stdout.encoding, 'replace')
}

Context:

[docs: setup: Norm Hardy suggested that it would be easier if users realized that they already had Python (especially true for Mac users)
zooko@zooko.com**20090325035459
 Ignore-this: e1bb76a1be4d6d541090d8d9e7e73db9
] 
[expirer: include crawler progress in the JSON status output
warner@allmydata.com**20090324205137
 Ignore-this: f65ccc0a57ed7c5d9565131350d87d2d
] 
[docs/proposed: new Accounting overview, discuss in #666
warner@allmydata.com**20090324015752
 Ignore-this: 7379396536e85194b1c1bec21288adc7
] 
[docs/proposed: move old accounting docs out of the way
warner@allmydata.com**20090324015457
 Ignore-this: 196fe85ef3592f7ef8e58a101eed0494
] 
[move GC docs out of proposed/, since it's all implemented now. Add reference to configuration.txt . Add expire.*= suggestions to tahoe.cfg .
warner@allmydata.com**20090323230820
 Ignore-this: ce0ec977f49c4a3642cf3a7ea7789dc4
] 
[storage: use constant-time comparison for write-enablers and lease-secrets
warner@lothar.com**20090323032128
 Ignore-this: efbbb15242b3f306c32498aae7fd75d
] 
[hashutil: add constant-time comparison function, to avoid timing attacks when python's short-circuiting data-dependent == operator is used to, say, check a write-enabler
warner@lothar.com**20090323032055
 Ignore-this: c5c1f5e529ab1b352c949f3e0d9abf20
] 
[WUI: fix display of empty directories, it threw an exception before
warner@allmydata.com**20090320235809
 Ignore-this: e598bb806d75411d202ba90fc251ad2b
] 
[storage webstatus: insert spaces when we're configured to expire multiple sharetypes
warner@allmydata.com**20090320224450
 Ignore-this: 18f44396384c8598d984eaa11dc92cf7
] 
[expirer: tolerate empty buckets, refactor bucketsize-counting code a bit, don't increment -mutable/-immutable counters unless we actually know the sharetype
warner@allmydata.com**20090320191816
 Ignore-this: 16827fe12b874a0205b0a7fdb6b7f960
] 
[storage: improve wording of status message
warner@allmydata.com**20090319184837
 Ignore-this: 66684623d50c89fa02915665dbbfec98
] 
[parse_date: insist that it returns an int
warner@allmydata.com**20090319184443
 Ignore-this: a1f5ae0c16a151d218c06fc4411f03c1
] 
[storage status: report expiration-cutoff-date like 19-Mar-2009 (as opposed to the tahoe.cfg input format of 2009-03-19), for redundancy: someone who gets the month and day switched will have a better chance to spot the problem in the storage-status output if it's in a different format
warner@allmydata.com**20090319180756
 Ignore-this: df2c3778260c6389d32f4b47b004ab35
] 
[expirer: clean up constructor args, add tahoe.cfg controls, use cutoff_date instead of date_cutoff
warner@allmydata.com**20090319010009
 Ignore-this: 2b6aaa6d5e6ff9fd417f32978b443fd2
] 
[util/time_format: new routine to parse dates like 2009-03-18, switch expirer to use it. I'd prefer to use 18-Mar-2009, but it is surprisingly non-trivial to build a parser that will take UTC dates instead of local dates
warner@allmydata.com**20090319005814
 Ignore-this: 4f08b301c780a78084fa02a09b5dda3
] 
[expirer: change setup, config options, in preparation for adding tahoe.cfg controls
warner@allmydata.com**20090319002138
 Ignore-this: b23a53e97f2a9fb7a005e9fe40e83fac
] 
[expirer: track mutable-vs-immutable sharecounts and sizes, report them on the web status page for comparison
warner@allmydata.com**20090318202504
 Ignore-this: 87e809bf5dedef3f0bc8f4a7b90e42d2
] 
[add utility function to parse durations, for lease-expiration config
warner@lothar.com**20090317070117
 Ignore-this: 2f4fa4352c887b4767969f0ea7931644
] 
[expirer: add mode to expire only-mutable or only-immutable shares
warner@lothar.com**20090317065118
 Ignore-this: b0b25427e3b1516bdfe293528b8e4a4e
] 
[tahoe_ls.py: add comment about error cases to improve
warner@lothar.com**20090317051206
 Ignore-this: 7c678b92a55b2f7c6f0d96fb6ece74ed
] 
[GC: add date-cutoff -based expiration, add proposed docs
warner@lothar.com**20090317051041
 Ignore-this: a5c0ecbcc2666eb04f2daa67331d1948
] 
[test_deepcheck: remove the 10s timeout: our dapper buildslave requires 30s, and the reduced timeout was only there because this tests fails by timeout rather than explicitly
warner@lothar.com**20090316005134
 Ignore-this: 4ce081283c26719be513b5ce8381a81a
] 
[tahoe cp -r: add --caps-only flag, to write filecaps into local files instead of actual file contents. Used only for debugging and as a quick tree-comparison tool.
warner@lothar.com**20090315231958
 Ignore-this: 8ecdf2b08601ae9e9fec5885bf640262
] 
[dirnode deep_traverse: insert a turn break (fireEventually) at least once every 100 files, otherwise a CHK followed by more than 158 LITs can overflow the stack, sort of like #237.
warner@allmydata.com**20090313233135
 Ignore-this: 39b78faa947ed9461f2d120f6843e59f
] 
[consolidate: remove pointless 'else' after for loop
warner@allmydata.com**20090313082751
 Ignore-this: 517f41316f7ae40b92b9eef8269a4b69
] 
[consolidate: add eta, flush stdout
warner@allmydata.com**20090313082451
 Ignore-this: 845f63adccc32557c6864ae6120ba836
] 
[consolidate: tolerate unicode dirnames
warner@allmydata.com**20090313065402
 Ignore-this: 7e65703ed3d12d4bd5ec14b693e5f61f
] 
[dirnode.py: when doing deep-traverse, walk each directory in alphabetical order, to make things like 'manifest' more predictable
warner@allmydata.com**20090313065046
 Ignore-this: 9a80055a93a6b11853d4e8202bacec14
] 
[consolidator: add progress to scan-old-directory passes
warner@allmydata.com**20090313054728
 Ignore-this: adc67a34f4f19fd58c5bc76301b3df36
] 
[consolidator: fix cycle detection to not trigger on merely shared directories, add snapshot counter to progress
warner@allmydata.com**20090313042229
 Ignore-this: eba2cf9f1b1364b8e4c5ae4fa030a99f
] 
[consolidator: re-use more directories, add total directories seen-vs-used counts
warner@allmydata.com**20090313034801
 Ignore-this: 6e743d2940a9fe129cee31008c894d70
] 
[tahoe_backup.py: tolerate more time formats
warner@allmydata.com**20090313011600
 Ignore-this: ca74f56f0dce7d19810c5a7a75bc623c
] 
[consolidator: add more verbose traversal of directories
warner@allmydata.com**20090312232900
 Ignore-this: 8ff0e17c6174566832a566a111032db4
] 
[consolidate: create multiple numbered backups of the original Archives directory, not just the first time
warner@allmydata.com**20090312230427
 Ignore-this: e4985f76969b584d099b050781aa561c
] 
[add 'tahoe debug consolidate' command, to merge directories created by repeated 'tahoe cp -r' or the allmydata win32 backup tool, into the form that would have been created by 'tahoe backup'.
warner@allmydata.com**20090312205606
 Ignore-this: 66569ca2190aa7b0f9199bcf09dcb27e
] 
[tahoe_cp.py: return 0 for success, instead of None
warner@allmydata.com**20090312205345
 Ignore-this: 808f6d8617f8c4e7fde455e6c7639fab
] 
[setup: specify in the debian/control files that tahoe is compatible with Python 2.6
zooko@zooko.com**20090311225902
 Ignore-this: d0793013e4c868d92793d932ef92a62d
] 
[setup: update the debian/copyright text to reflect the current licences
zooko@zooko.com**20090311152952
 Ignore-this: 806a95b1b79d6bb20507db5c7201af45
] 
[expirer: fix prediction math, thanks to Zandr for the catch
warner@allmydata.com**20090309204217
 Ignore-this: e20f784483ea1a15e35627fe95506272
] 
[storage.expirer: oops, fix upgrade-handler code
warner@lothar.com**20090309035516
 Ignore-this: af8bc0ea1c7f305b33c06f4a9bf58de3
] 
[storage.expirer: handle upgrades better
warner@lothar.com**20090309034220
 Ignore-this: dd654305b94bc3209426a3e008d57363
] 
[storage.expirer: exercise the last missing line of webstatus code
warner@lothar.com**20090309033828
 Ignore-this: fc4aa34734cae32eec1db623ca0b145b
] 
[expirer: tolerate corrupt shares, add them to the state and history for future examination
warner@lothar.com**20090309030840
 Ignore-this: 5ae7e68471ed700cc68beb408e0f303
] 
[storage/immutable: raise a specific error upon seeing a bad version number, instead of using assert. Also wrap to 80cols.
warner@lothar.com**20090309030732
 Ignore-this: 5331d9680ffceff029fbbbcdece7f282
] 
[storage: add test coverage for BucketReader.__repr__
warner@lothar.com**20090309030527
 Ignore-this: 60ce4e8c5200cc9885a3057f914e78b4
] 
[storage/mutable: raise a specific error upon seeing bad magic, instead of using assert
warner@lothar.com**20090309020201
 Ignore-this: 8daa77362902f5d6ef793e9602a1383b
] 
[nodeadmin: node stops itself if a hotline file hasn't been touched in 120 seconds now, instead of in 60 seconds
zooko@zooko.com**20090308025039
 Ignore-this: a9cfc605d3fcad912bd2d0b35d5143ec
 A test failed on draco (MacPPC) because it took 67.1 seconds to get around to running the test, and the node had already stopped itself when the hotline file was 60 seconds old.
] 
[expirer: make web display a bit more consistent
warner@lothar.com**20090307221442
 Ignore-this: 7fec9d9bffc0bddeb51c1baa8e7ea020
] 
[web/storage.py: tolerate unknown-future displays, I'm not sure why LeaseCrawler.test_unpredictable_future didn't catch this
warner@lothar.com**20090307220243
 Ignore-this: 3d4e5baa8cc6d5d26edcea29fda8593d
] 
[tahoe_cp.py: improve error reporting slightly: don't json-interpret HTTP errors, pass through tahoe webapi error messages
warner@lothar.com**20090307114051
 Ignore-this: a8beccb67adb082a92509d439b27d68e
] 
[test_cli.py: assert that 'ls' on an unrecoverable file now gives a better error message
warner@lothar.com**20090307110815
 Ignore-this: 18e9758e4d0ca7faeaf5bd481c2d72ff
] 
[test_web: test behavior of broken-dirnode GET, both html and json
warner@lothar.com**20090307105707
 Ignore-this: c0e5b45eee28959f899efa1bd189d6bd
] 
[web: when a dirnode can't be read, emit a regular HTML page but with the child-table and upload-forms replaced with an apologetic message. Make sure to include the 'get info' links so the user can do a filecheck
warner@lothar.com**20090307105601
 Ignore-this: f949d6bd58c0c2fd60fd5fa730115f3f
] 
[web/common: split out exception-to-explanation+code mapping to a separate humanize_failure() function, so it can be used by other code. Add explanation for mutable UnrecoverableFileError.
warner@lothar.com**20090307105408
 Ignore-this: 92f326804ba73bb446c5df5992b7d72e
] 
[test_cli: use explicit (and stable) testdirs, instead of using self.mktemp
warner@lothar.com**20090307090428
 Ignore-this: 7c58d159e4f33d01635c3445d9e591f9
] 
[test_storage: solaris doesn't appear to give the same block count as other platforms, so don't assert as much about 'diskbytes' recovered
warner@lothar.com**20090307084518
 Ignore-this: 55b35c094ce78c50c8ede42062c5ea13
] 
[tahoe_cp.py: don't be verbose unless --verbose is passed: default is one line of success
warner@lothar.com**20090307084421
 Ignore-this: f75bb818081e0c2993378154775a69d1
] 
[setup.cfg: use allmydata.org as a mirror for some of our necessary dependencies
warner@allmydata.com**20090307045753
 Ignore-this: ce174de4ad2dd96f184eadf4d62ea779
] 
[storage: add a lease-checker-and-expirer crawler, plus web status page.
warner@allmydata.com**20090307044517
 Ignore-this: 4355224f89b959c6f1a256a7e6c88c3b
 
 This walks slowly through all shares, examining their leases, deciding which
 are still valid and which have expired. Once enabled, it will then remove the
 expired leases, and delete shares which no longer have any valid leases. Note
 that there is not yet a tahoe.cfg option to enable lease-deletion: the
 current code is read-only. A subsequent patch will add a tahoe.cfg knob to
 control this, as well as docs. Some other minor items included in this patch:
 
  tahoe debug dump-share has a new --leases-only flag
  storage sharefile/leaseinfo code is cleaned up
  storage web status page (/storage) has more info, more tests coverage
  space-left measurement on OS-X should be more accurate (it was off by 2048x)
   (use stat .f_frsize instead of f_bsize)
] 
[setup: comment-out the use of the directory full of dependency packages on the test grid -- the test grid is too unreliable!  :-(
zooko@zooko.com**20090307044332
 Ignore-this: c09f2e5f0b383b4227df4b07c267cb05
] 
[setup: create a "make deb-lenny-head" target
zooko@zooko.com**20090306191057
 Ignore-this: 4b2ff187a3d08dcfe9318980ca92f097
 I made this patch by copying [20090305220021-92b7f-89d987c7d05306b5cb03a64f2956a652c10a7296] and changing the name from "intrepid" to "lenny".  I haven't tested it.
] 
[trivial: remove unused import detected by pyflakes, and remove trailing whitespace
zooko@zooko.com**20090305223204
 Ignore-this: 991001a50da1a357a519c3cb880d7ee1
] 
[setup: enable build of .debs for Ubuntu Intrepid, thanks to DarKNesS_WolF
zooko@zooko.com**20090305220021
 Ignore-this: 88dbb3f72c2446b7734ac437189b67df
] 
[trivial: use more specific function for ascii-encoding storage index
zooko@zooko.com**20090222175751
 Ignore-this: c1a78a71cf04d7dbd80a71106f9b21a1
] 
[doc: update how_to_make_a_tahoe_release.txt
zooko@zooko.com**20090222175739
 Ignore-this: 6a2e1592741b362bc170167a9cadc0b
] 
[test_cli: validate non-HTML error response of 'tahoe get' on an unrecoverable file
warner@lothar.com**20090304041146] 
[web: full patch for HTML-vs-plaintext traceback renderings, improve test coverage of exception rendering
warner@lothar.com**20090304035630] 
[web/common.py: use 'Accept:' header to control HTML-vs-text/plain traceback renderings
warner@lothar.com**20090304035457] 
[test/common: rearrange shouldHTTPError
warner@lothar.com**20090304014059] 
[web: move plural() to common.py
warner@lothar.com**20090304014019] 
[add more information to NotEnoughSharesError, split out new exceptions for no-servers and no-source-of-ueb-hash
warner@lothar.com**20090304013715] 
[test_web: move shouldHTTPError into common.WebErrorMixin
warner@lothar.com**20090303225620] 
[test_web: clean up shouldHTTPError methods
warner@lothar.com**20090303224604] 
[immutable checker add-lease: catch remote IndexError here too
warner@allmydata.com**20090227071724
 Ignore-this: 94ee6064ce26409381c6451cd50fdade
] 
[servermap add-lease: fix the code that's supposed to catch remote IndexErrors, I forgot that they present as ServerFailures instead. This should stop the deluge of Incidents that occur when you do add-lease against 1.3.0 servers
warner@allmydata.com**20090227070426
 Ignore-this: 3d7bc87d9587b51d44b27317d9eded23
] 
[rrefutil: add check_remote utility function
warner@allmydata.com**20090227065957
 Ignore-this: d859d0fae87b7e84ad3c2893350ed519
] 
[rrefutil: add trap_remote utility and friends
warner@allmydata.com**20090227065524
 Ignore-this: a594050cdd9bcca073d8029819dbc35
] 
[web: improve layout of storage status with a table
warner@allmydata.com**20090227015838
 Ignore-this: 5c12f5e296ad1cf224f85e307813eb09
] 
[crawler: add ETA to get_progress()
warner@allmydata.com**20090227014248
 Ignore-this: 27ae76c0530b83323209be70df3d8a4b
] 
[Fix for bug #645, correct path handling logic so that it works from sshfs
Alberto Berti <alberto@metapensiero.it>**20090226150237
 Ignore-this: e9c1b2d48ebf4ba68100d76e54154a78
] 
[wui: s/Provisioning/Reliability/ ; suggested by Terrell
zooko@zooko.com**20090226030904
 Ignore-this: f9559702e760dc3c57e2fc36f087d91e
] 
[startstop_node.py: when restarting, tolerate a stale pid too
warner@allmydata.com**20090226015245
 Ignore-this: 59197d3f400417be64fa0f91a64fae12
] 
[web: fix the ERROR: line to work the same in python2.4 and 2.5
warner@lothar.com**20090225074621] 
[test_cli/test_web: fix spurious test failure on solaris (maybe python2.4?) due to variations in the way that exceptions are stringified
warner@allmydata.com**20090225060128
 Ignore-this: 829fb7d67fd199babdde1443b64a5381
] 
[CLI: modify 'tahoe manifest' and 'tahoe deep-check' to report ERROR: properly. For #590.
warner@allmydata.com**20090225054415
 Ignore-this: 99162f894fdd24112a869e14848c3dea
] 
[webapi: modify streaming deep-manifest/deep-checker to emit an ERROR: line if they encounter an unrecoverable+untraversable directory. For #590.
warner@allmydata.com**20090225051335
 Ignore-this: e6bc49368fb0e7ada7cff477fd63ffe6
] 
[scripts/common: fix alias handling on windows again, emit slightly nicer error message in response to an unknown alias
warner@allmydata.com**20090225042136
 Ignore-this: 76df800d131aed6701b5c7408105b134
] 
[#165: make 'tahoe restart --force' the default behavior: warn but do not stop if restart is used on something that wasn't a running node, and always try to start it afterwards. This is particularly important for #315 (restart -m), because otherwise a single not-already-running node will prevent all nodes from being restarted, resulting in longer downtime than necessary
warner@allmydata.com**20090225034213
 Ignore-this: 26d8b0eea0279ca793faf23c5e948d90
] 
[test_deepcheck: switch deep-check tests to use no-network too. This cuts the runtime down by about 50%
warner@allmydata.com**20090225030457
 Ignore-this: b3a98ed18c5752c9016c047e95d42b
] 
[test_deepcheck: convert MutableChecker to no-network GridTest
warner@allmydata.com**20090225020010
 Ignore-this: eccba7fda129330b642886271a61a573
] 
[tests/no_network: move GET into the GridTestMixin class
warner@allmydata.com**20090225003300
 Ignore-this: 7779ad38c2d687ae328ba3cb6164a7a4
] 
[common_web.py: oops, add .fields to the other FakeRequest
warner@allmydata.com**20090225000459
 Ignore-this: 7144269b5e083553ee2c3e7afea00604
] 
[test_cli: exercise the recent tolerate-'c:\dir\file.txt' fix in scripts/common, recorded in a separate match to make it easier to merge the fix to prod
warner@allmydata.com**20090224235620
 Ignore-this: 2cae196dd4ccb578b2abae085376e0d7
] 
[scripts/common: on windows, tolerate paths like 'c:\dir\file.txt', by treating single-letter aliases on windows/cygwin as non-aliases
warner@allmydata.com**20090224235522
 Ignore-this: 96d37644b7f81ac768ff4a1d1915eb46
] 
[test/common_web.py: add a .fields attribute to our FakeRequest, since we support versions of Nevow that are old enough to not do it themselves
warner@allmydata.com**20090224232050
 Ignore-this: 239aaed746128e54e886762782541a1f
] 
[Two small fixes on documentation for cli backup command.
Alberto Berti <alberto@metapensiero.it>**20090224223634
 Ignore-this: 5634a6dadad6e4e43a112de7fe5c74c
] 
[test_web: add (disabled) test to see what happens when deep-check encounters an unrecoverable directory. We still need code changes to improve this behavior.
warner@lothar.com**20090224214017
 Ignore-this: e839f1b0ec40f53fedcd809c2a30d5f9
] 
[Add elapsed timestamp to cli backup command final summary.
Alberto Berti <alberto@metapensiero.it>**20090224171425
 Ignore-this: 9a042d11f95ee9f6858a5096d513c0bc
] 
[Added documentation for '--exclude' and friends cli backup command.
Alberto Berti <alberto@metapensiero.it>**20090224153049
 Ignore-this: bbc791fa56e38535bb82cc3077ffde90
] 
[test_repairer: change to use faster no_network.GridTestMixin, split Verifier tests into separate cases, refactor judgement funcs into shared methods
warner@lothar.com**20090224041506
 Ignore-this: 584ce72d6276da5edc00562793d4ee53
] 
[immutable/checker.py: trap ShareVersionIncompatible too. Also, use f.check
warner@lothar.com**20090224041405
 Ignore-this: b667e8d3192116293babcacdeed42898
 instead of examining the value returned by f.trap, because the latter appears
 to squash exception types down into their base classes (i.e. since
 ShareVersionIncompatible is a subclass of LayoutInvalid,
 f.trap(Failure(ShareVersionIncompatible)) == LayoutInvalid).
 
 All this resulted in 'incompatible' shares being misclassified as 'corrupt'.
] 
[immutable/layout.py: wrap to 80 cols, no functional changes
warner@lothar.com**20090224005837
 Ignore-this: 40019480180ec34141506a28d7711608
] 
[test_repairer: change Repairer to use much-faster no_network.GridTestMixin. As a side-effect, fix what I think was a bug: some of the assert-minimal-effort-expended checks were mixing write counts and allocate counts
warner@lothar.com**20090223234227
 Ignore-this: d58bd0a909f9939775730cda4a858cae
] 
[test/no_network.py: add a basic stats provider
warner@lothar.com**20090223233937
 Ignore-this: c9f3cc4eed99cfc36f68938ceff4162c
] 
[tests: stop using setUpClass/tearDownClass, since they've been deprecated in Twisted-8.2.0
warner@lothar.com**20090223204312
 Ignore-this: 24c6592141cf64103530c024f93a5b88
] 
[test_checker: improve test coverage for checker results
warner@lothar.com**20090223201943
 Ignore-this: 83e173602f0f4c811a7a9893d85385df
] 
[Fixed tests again so they will pass on windows.
Alberto Berti <alberto@metapensiero.it>**20090223003502
 Ignore-this: 80d5074e7153642a2fa2a77958bfb50d
] 
[misc/*: remove RuntimeError too
warner@lothar.com**20090222233401
 Ignore-this: b76f8a184f75bb28eb9d8002f957936a
] 
[scripts: stop using RuntimeError, for #639
warner@lothar.com**20090222233106
 Ignore-this: 686a424442670fffbd4d1816c284a601
] 
[mutable/publish: stop using RuntimeError, for #639
warner@lothar.com**20090222233056
 Ignore-this: 2a80a661c7850d97357caddad48c6e9d
] 
[remove more RuntimeError from unit tests, for #639
warner@lothar.com**20090222232855
 Ignore-this: 1a1c3e1457f3f29ba7101fe406ee5f43
] 
[stop using RuntimeError in unit tests, for #639
warner@lothar.com**20090222232722
 Ignore-this: 475ce0c0dcd7a1f5ed83ef460312efea
] 
[ftpd/sftpd: stop using RuntimeError, for #639
warner@lothar.com**20090222232426
 Ignore-this: 97001362c4ba9e94b2e254e229b79987
] 
[Added tests for the cse when listdir is an iterator
Alberto Berti <alberto@metapensiero.it>**20090222224356
 Ignore-this: 218fb2aba02c28b4b1e5324bdb5adeaa
] 
[Fixed tests so that they pass also on buildbots.
Alberto Berti <alberto@metapensiero.it>**20090222224311
 Ignore-this: fcb91cd6acf028382411d23d380a4576
] 
[Use failUnlessEqual instead of failUnless(a == b)
Alberto Berti <alberto@metapensiero.it>**20090222224214
 Ignore-this: 8f9144632e3ac9acb4726fb48a083bf4
] 
[Better implementation of filtering algorithm.
Alberto Berti <alberto@metapensiero.it>**20090222224049
 Ignore-this: 67a8bd2f99bcc87ca2443bef13370a87
] 
[Removed '.hgrags' from vcs excludes
Alberto Berti <alberto@metapensiero.it>**20090222223946
 Ignore-this: 3e94c22fc9d85f380ee11fb8bdb4d1e9
] 
[docs: CREDITS to Alberto Berti
zooko@zooko.com**20090222193314
 Ignore-this: 74d370ada3234cce9e58aec15d739f71
] 
[Added tests for the --exclude* options of backup command.
Alberto Berti <alberto@metapensiero.it>**20090222165106
 Ignore-this: f1b931cf2e7929ce47b737c022bca707
] 
[Added --exclude, --exclude-from and --exclude-vcs options to backup command.
Alberto Berti <alberto@metapensiero.it>**20090222170829
 Ignore-this: 4912890229cd54a2f61f14f06bc4afcc
 
 It is still impossible to specify absolute exclusion path, only
 relative. I must check with tar or rsync how they allow them to be
 specified.
] 
[Raise a more explanatory exception for errors encountered during backup processing.
Alberto Berti <alberto@metapensiero.it>**20090222170252
 Ignore-this: f6b8ffe2a903ba07a2c1c59130dac1e4
] 
[Added tests for the fixed alias related command's synopsis
Alberto Berti <alberto@metapensiero.it>**20090222163732
 Ignore-this: 4432b4e88e990ba53a5b3fe0f12db2ac
] 
[Add missing synopsis and descriptions for alias commands.
Alberto Berti <alberto@metapensiero.it>**20090221003106
 Ignore-this: 8aedd03d36d92d912102c7f29e4ca697
] 
[docs: move many specification-like documents into specifications/
warner@lothar.com**20090222054054
 Ignore-this: a4110cc478198c0611205aba1ccf54f4
] 
[test_web.py: increase test coverage of web.status.plural()
warner@lothar.com**20090222000116
 Ignore-this: 3138c9d5d2410d8e1121e9b2ed694169
] 
[crawler: fix performance problems: only save state once per timeslice (not after every bucket), don't start the crawler until 5 minutes after node startup
warner@lothar.com**20090221205649
 Ignore-this: e6551569982bd31d19779ff15c2d6f58
] 
[test_system: oops, don't assume that all files in storage/ are in a deep storage/shares/prefix/si/shnum path, since now the crawler pickle has a short path
warner@lothar.com**20090221061710
 Ignore-this: fde76d0e5cae853014d1bb18b5f17dae
] 
[crawler: tolerate low-resolution system clocks (i.e. windows)
warner@lothar.com**20090221061533
 Ignore-this: 57286a3abcaf44f6d1a78c3c1ad547a5
] 
[BucketCountingCrawler: store just the count, not cycle+count, since it's too easy to make usage mistakes otherwise
warner@lothar.com**20090221035831
 Ignore-this: 573b6f651af74380cdd64059fbbdda4b
] 
[test_storage: startService the server, as is now the standard practice
warner@lothar.com**20090221035755
 Ignore-this: 3999889bd628fe4039bbcf1b29160453
] 
[crawler: load state from the pickle in init, rather than waiting until startService, so get_state() can be called early
warner@lothar.com**20090221035720
 Ignore-this: ecd128a5f4364c0daf4b72d791340b66
] 
[BucketCountingCrawler: rename status and state keys to use 'bucket' instead of 'share', because the former is more accurate
warner@lothar.com**20090221034606
 Ignore-this: cf819f63fac9506c878d6c9715ce35b7
] 
[storage: also report space-free-for-root and space-free-for-nonroot, since that helps users understand the space-left-for-tahoe number better
warner@lothar.com**20090221032856
 Ignore-this: 9fdf0475f758acd98b73026677170b45
] 
[storage: add bucket-counting share crawler, add its output (number of files+directories maintained by a storage server) and status to the webapi /storage page
warner@lothar.com**20090221030408
 Ignore-this: 28761c5e076648026bc5f518506db65c
] 
[storage: move si_b2a/si_a2b/storage_index_to_dir out of server.py and into common.py
warner@lothar.com**20090221030309
 Ignore-this: 645056428ab797f0b542831c82bf192a
] 
[crawler: add get_progress, clean up get_state
warner@lothar.com**20090221002743
 Ignore-this: 9bea69f154c75b31a53425a8ea67789b
] 
[web/storage: make sure we can handle platforms without os.statvfs too
warner@lothar.com**20090220220353
 Ignore-this: 79d4cb8482a8543b9759dc949c86c587
] 
[crawler: provide for one-shot crawlers, which stop after their first full cycle, for share-upgraders and database-populaters
warner@lothar.com**20090220211911
 Ignore-this: fcdf72c5ffcafa374d376388be6fa5c5
] 
[web: add Storage status page, improve tests
warner@lothar.com**20090220202926
 Ignore-this: e34d5270dcf0237fe72f573f717c7a4
] 
[storage: include reserved_space in stats
warner@lothar.com**20090220202920
 Ignore-this: b5b480fe0abad0148ecad0c1fb47ecae
] 
[web/check_results: sort share identifiers in the sharemap display
warner@lothar.com**20090220182922
 Ignore-this: 5c7bfcee3e15c7082c3653eb8a460960
] 
[webapi: pass client through constructor arguments, remove IClient, should make it easier to test web renderers in isolation
warner@lothar.com**20090220181554
 Ignore-this: e7848cd1bee8faf2ce7aaf040b9bf8e3
] 
[test/no_network: do startService on the storage servers, make it easier to customize the storage servers
warner@lothar.com**20090220022254
 Ignore-this: e62f328721c007e4c5ee023a6efdf66d
] 
[crawler: modify API to support upcoming bucket-counting crawler
warner@lothar.com**20090220013142
 Ignore-this: 808f8382837b13082f8b245db2ebee06
] 
[test_backupdb: make the not-a-database file larger, since the older sqlite-2.3.2 on OS-X is easily fooled
warner@lothar.com**20090220000409
 Ignore-this: 694d2ca5053bb96e91670765d0cedf2e
] 
[web/reliability: add parameter descriptions, adapted from a patch from Terrell Russell.
warner@lothar.com**20090219222918
 Ignore-this: 835f5ab01e1aff31b2ff9febb9a51f3
] 
[test_crawler: hush pyflakes
warner@lothar.com**20090219202340
 Ignore-this: 765d22c9c9682cc86c5205dc130500af
] 
[test_crawler: disable the percentage-of-cpu-used test, since it is too unreliable on our slow buildslaves. But leave the code in place for developers to run by hand.
warner@lothar.com**20090219201654
 Ignore-this: ff7cf5cfa79c6f2ef0cf959495dd989a
] 
[reliability.py: fix the numpy conversion, it was completely broken. Thanks to Terrell Russell for the help.
warner@lothar.com**20090219195515
 Ignore-this: f2b1eb65855111b338e1487feee1bbcf
] 
[reliability: switch to NumPy, since Numeric is deprecated
warner@lothar.com**20090219074435
 Ignore-this: f588a68e9bcd3b0bc3653570882b6fd5
] 
[setup.py: fix pyflakes complaints
warner@lothar.com**20090219073643
 Ignore-this: a314e5456b0a796bc9f70232a119ec68
] 
[move show-tool-versions out of setup.py and into a separate script in misc/ , since setuptools is trying to build and install a bunch of stuff first
warner@lothar.com**20090219073558
 Ignore-this: 9e56bc43026379212e6b6671ed6a1fd4
] 
[test_crawler: don't require >=1 cycle on cygwin
warner@lothar.com**20090219065818
 Ignore-this: b8d2d40f26aeb30a7622479840a04635
] 
[setup.py: add show_tool_versions command, for the benefit of a new buildbot step
warner@lothar.com**20090219062436
 Ignore-this: 21d761c76a033e481831584bedc60c86
] 
[setup.py: wrap to 80 cols, no functional changes
warner@lothar.com**20090219055751
 Ignore-this: d29e57c6ee555f2ee435667b7e13e60b
] 
[crawler: use fileutil.move_info_place in preference to our own version
warner@lothar.com**20090219051342
 Ignore-this: ee4e46f3de965610503ba36b28184db9
] 
[fileutil: add move_into_place(), to perform the standard unix trick of atomically replacing a file, with a fallback for windows
warner@lothar.com**20090219051310
 Ignore-this: c1d35e8ca88fcb223ea194513611c511
] 
[crawler: fix problems on windows and our slow cygwin slave
warner@lothar.com**20090219042431
 Ignore-this: 8019cb0da79ba00c536183a6f57b4cab
] 
[#633: first version of a rate-limited interruptable share-crawler
warner@lothar.com**20090219034633
 Ignore-this: 5d2d30c743e3b096a8e775d5a9b33601
] 
[change StorageServer to take nodeid in the constructor, instead of assigning it later, since it's cleaner and because the original problem (Tubs not being ready until later) went away
warner@lothar.com**20090218222301
 Ignore-this: 740d582f20c93bebf60e21d9a446d3d2
] 
[test_system: split off checker tests to test_deepcheck.py, this file is too big
warner@lothar.com**20090218214234
 Ignore-this: 82bf8db81dfbc98224bbf694054a8761
] 
[break storage.py into smaller pieces in storage/*.py . No behavioral changes.
warner@lothar.com**20090218204655
 Ignore-this: 312d408d1cacc5a764d791b53ebf8f91
] 
[immutable/layout: minor change to repr name
warner@lothar.com**20090218204648
 Ignore-this: c8781ef15b7dea63b39236a1899b86ce
] 
[docs: add lease-tradeoffs diagram
warner@lothar.com**20090218204137
 Ignore-this: c22a589ad465dac846da834c30dc4083
] 
[interfaces.py: allow add/renew/cancel-lease to return Any, so that 1.3.1 clients (the first to use these calls) can tolerate future storage servers which might return something other than None
warner@lothar.com**20090218192903
 Ignore-this: dcbb704a05416ecc66d90fb486c3d75b
] 
[docs/debian.txt: minor edit
warner@lothar.com**20090218032212
 Ignore-this: 64ff1fb163ffca4bcfd920254f1cf866
] 
[add --add-lease to 'tahoe check', 'tahoe deep-check', and webapi.
warner@lothar.com**20090218013243
 Ignore-this: 176b2006cef5041adcb592ee83e084dd
] 
[change RIStorageServer.remote_add_lease to exit silently in case of no-such-bucket, instead of raising IndexError, because that makes the upcoming --add-lease feature faster and less noisy
warner@lothar.com**20090218013053
 Ignore-this: 6fdfcea2c832178f1ce72ab0ff510f3a
] 
[CLI #590: convert 'tahoe deep-check' to streaming form, improve display, add tests
warner@lothar.com**20090217231511
 Ignore-this: 6d88eb94b1c877eacc8c5ca7d0aac776
] 
[interfaces.py: document behavior of add_lease/renew_lease/cancel_lease, before I change it
warner@lothar.com**20090217194809
 Ignore-this: 703c6712926b8edb19d55d790b65a400
] 
[test_backupdb: improve error messages if the test fails
warner@lothar.com**20090217170838
 Ignore-this: ef657e87c66e4304d3e0aca9831b84c
] 
[webapi #590: add streaming deep-check. Still need a CLI tool to use it.
warner@lothar.com**20090217053553
 Ignore-this: a0edd3d2a531c48a64d8397f7e4b208c
] 
[test_web.Grid: change the CHECK() function to make it easier to test t= values with hyphens in them
warner@lothar.com**20090217050034
 Ignore-this: 410c08735347c2057df52f6716520228
] 
[test_web: improve checker-results coverage with a no-network -based test, enhance no-network harness to assist, fix some bugs in web/check_results.py that were exposed
warner@lothar.com**20090217041242
 Ignore-this: fe54bb66a9ae073c002a7af51cd1e18
] 
[web: fix handling of reliability page when Numeric is not available
warner@lothar.com**20090217015658
 Ignore-this: 9d329182f1b2e5f812e5e7eb5f4cf2ed
] 
[test/no_network: update comments with setup timing: no_network takes 50ms, SystemTestMixin takes 2s (on my laptop)
warner@lothar.com**20090217000643
 Ignore-this: cc778fa3219775b25057bfc9491f8f34
] 
[test_upload: rewrite in terms of no-network GridTestMixin, improve no_network.py as necessary
warner@lothar.com**20090216234457
 Ignore-this: 80a341d5aa3036d24de98e267499d70d
] 
[test_download: rewrite in terms of no-network GridTestMixin, improve no_network.py as necessary
warner@lothar.com**20090216233658
 Ignore-this: ec2febafd2403830519120fb3f3ca04e
] 
[test_dirnode.py: convert Deleter to new no-network gridtest
warner@lothar.com**20090216232348
 Ignore-this: 8041739442ec4db726675e48f9775ae9
] 
[test_cli.py: modify to use the new 'no-network' gridtest instead of SystemTestMixin, which speeds it up from 73s to 43s on my system
warner@lothar.com**20090216232005
 Ignore-this: ec6d010c9182aa72049d1fb894cf890e
] 
[tests: fix no_network framework to work with upload/download and checker
warner@lothar.com**20090216231947
 Ignore-this: 74b4dbd66b8384ae7c7544969fe4f744
] 
[client.py: improve docstring
warner@lothar.com**20090216231532
 Ignore-this: bbaa9e3f63fdb0048e3125c4681b2d1f
] 
[test_cli: add test coverage for help strings
warner@lothar.com**20090216210833
 Ignore-this: d2020849107f687448e159a19d0e5dab
] 
[test/no_network: new test harness, like system-test but doesn't use the network so it's faster
warner@lothar.com**20090216205844
 Ignore-this: 31678f7bdef30b0216fd657fc6145534
] 
[interfaces.py: minor docstring edit
warner@lothar.com**20090216205816
 Ignore-this: cec3855070197f7920b370f95e8b07bd
] 
[setup: if you sdist_dsc (to produce the input files for dpkg-buildpackage) then run darcsver first
zooko@zooko.com**20090216201558
 Ignore-this: b85be51b3d4a9a19a3366e690f1063e2
] 
[doc: a few edits to docs made after the 1.3.0 release
zooko@zooko.com**20090216201539
 Ignore-this: dbff3b929d88134d862f1dffd1ef068a
] 
[test_cli: improve test coverage slightly
warner@lothar.com**20090216030451
 Ignore-this: e01ccc6a6fb44aaa4fb14fe8669e2065
] 
[test_util: get almost full test coverage of dictutil, starting with the original pyutil tests as a base. The remaining three uncovered lines involve funny cases of ValueOrderedDict that I can't figure out how to get at
warner@lothar.com**20090216023210
 Ignore-this: dc1f0c6d8c003c0ade38bc8f8516b04d
] 
[provisioning/reliability: add tests, hush pyflakes, remove dead code, fix web links
warner@lothar.com**20090215222451
 Ignore-this: 7854df3e0130d9388f06efd4c797262f
] 
[util/statistics: add tests, fix mean_repair_cost
warner@lothar.com**20090215222326
 Ignore-this: c576eabc74c23b170702018fc3c122d9
] 
[test_repairer: hush pyflakes
warner@lothar.com**20090215222310
 Ignore-this: 875eb52e86077cda77efd02da77f8cfa
] 
[lossmodel.lyx: move draft paper into docs/proposed/, since it's unfinished
warner@lothar.com**20090215221905
 Ignore-this: 7f7ee204e47fd66932759c94deefe68
] 
[build a 'reliability' web page, with a simulation of file decay and repair over time
warner@lothar.com**20090213234234
 Ignore-this: 9e9623eaac7b0637bbd0071f082bd345
] 
[More lossmodel work, on repair.
Shawn Willden <shawn-tahoe@willden.org>**20090116025648] 
[Loss model work (temp1)
Shawn Willden <shawn@willden.org>**20090115030058] 
[Statistics module
Shawn Willden <shawn-tahoe@willden.org>**20090114021235
 
 Added a statistics module for calculating various facets of
 share survival statistics.
] 
[docs: relnotes-short.txt
zooko@zooko.com**20090215163510
 Ignore-this: 683649bb13499bbe0e5cea2e1716ff59
 linkedin.com imposed a strict limit on the number of characters I could post.  This forced me to prune and prune and edit and edit until relnotes.txt was a quarter of its former size.  Here's the short version.
] 
[TAG allmydata-tahoe-1.3.0
zooko@zooko.com**20090214000556
 Ignore-this: aa6c9a31a14a58ad2298cb7b08d3ea70
] 
Patch bundle hash:
0bedf208c0799063a51adaab9d20cd288d2b614c