Mon Apr 27 11:58:49 CEST 2009 francois@ctrlaltdel.ch * cli: plumbing for unicode support Mon Apr 27 10:59:31 CEST 2009 francois@ctrlaltdel.ch * cli: tahoe backup unicode support Mon Apr 27 11:01:44 CEST 2009 francois@ctrlaltdel.ch * cli: tahoe manifest unicode support Mon Apr 27 11:18:50 CEST 2009 francois@ctrlaltdel.ch * cli: tahoe ls unicode support Mon Apr 27 11:18:20 CEST 2009 francois@ctrlaltdel.ch * cli: tahoe get unicode support Mon Apr 27 11:00:59 CEST 2009 francois@ctrlaltdel.ch * cli: tahoe webopen unicode support Mon Apr 27 10:58:47 CEST 2009 francois@ctrlaltdel.ch * cli: tahoe rm unicode support Mon Apr 27 10:59:13 CEST 2009 francois@ctrlaltdel.ch * cli: tahoe ln unicode support Mon Apr 27 11:02:13 CEST 2009 francois@ctrlaltdel.ch * cli: tahoe status unicode support Mon Apr 27 11:02:26 CEST 2009 francois@ctrlaltdel.ch * cli: tahoe check unicode support Mon Apr 27 11:02:43 CEST 2009 francois@ctrlaltdel.ch * cli: tahoe deep-check unicode support Mon Apr 27 11:10:47 CEST 2009 francois@ctrlaltdel.ch * cli: tahoe mkdir unicode support Mon Apr 27 11:12:52 CEST 2009 francois@ctrlaltdel.ch * docs: explain the current unicode support in CLI Mon Apr 27 15:52:03 CEST 2009 francois@ctrlaltdel.ch * cli: tahoe cp unicode support Wed Apr 29 18:20:03 CEST 2009 francois@ctrlaltdel.ch * cli: tahoe put unicode support + encode path elements using unicode_to_url Both changes (tahoe put and url encoding) had to be included in the same patch to prevent failing tests. Mon Apr 27 11:58:07 CEST 2009 francois@ctrlaltdel.ch * cli: aliases unicode support Mon Apr 27 15:52:39 CEST 2009 francois@ctrlaltdel.ch * cli: previous cli related patches should fix #534 New patches: [cli: plumbing for unicode support francois@ctrlaltdel.ch**20090427095849 Ignore-this: 9cc8871d8fe54cde0dba52912754e4f4 ] { hunk ./src/allmydata/scripts/cli.py 4 import os.path, re, sys, fnmatch from twisted.python import usage from allmydata.scripts.common import BaseOptions, get_aliases +from allmydata.util.stringutils import argv_to_unicode NODEURL_RE=re.compile("http://([^:]*)(:([1-9][0-9]*))?") hunk ./src/allmydata/scripts/common.py 4 import os, sys, urllib from twisted.python import usage - +from allmydata.util.stringutils import unicode_to_url class BaseOptions: # unit tests can override these to point at StringIO instances hunk ./src/allmydata/test/test_cli.py 9 import urllib import re import simplejson +import sys from allmydata.util import fileutil, hashutil, base32 from allmydata import uri addfile ./src/allmydata/util/stringutils.py hunk ./src/allmydata/util/stringutils.py 1 +""" +Functions used to convert inputs from whatever encoding used in the system to +unicode and back. + +TODO: + * Accept two cli arguments --argv-encoding and --filesystem-encoding +""" + +import sys +from allmydata.util.assertutil import precondition +from twisted.python import usage + +def argv_to_unicode(s): + """ + Decode given argv element to unicode. + """ + # sys.argv encoding detection in Python is not trivial so utf-8 is + # currently used by default and an informative error message is given if + # the argument cannot be correctly decoded. + + precondition(isinstance(s, str), s) + try: + return unicode(s, 'utf-8') + except UnicodeEncodeError: + raise usageError("Argument '%s' cannot be decoded as UTF-8." % s) + +def fs_to_unicode(s): + """ + Decode a filename (or a directory name) to unicode using the same encoding + as the filesystem. + """ + # Filename encoding detection is a little bit better thanks to + # getfilesystemencoding() in the sys module. However, filenames can be + # encoded using another encoding than the one used on the filesystem. + + precondition(isinstance(s, str), s) + encoding = sys.getfilesystemencoding() + try: + return unicode(s, encoding) + except UnicodeDecodeError: + raise usage.UsageError("Filename '%s' cannot be decoded using the current encoding of your filesystem (%s). Please rename this file." % (s, encoding)) + +def unicode_to_fs(s): + """ + Encode an unicode object used in file or directoy name. + """ + + precondition(isinstance(s, unicode), s) + encoding = sys.getfilesystemencoding() + try: + return s.encode(encoding) + except UnicodeEncodeError: + raise usage.UsageError("Filename '%s' cannot be encoded using the current encoding of your filesystem (%s). Please configure your locale correctly or rename this file." % (s, encoding)) + +def unicode_to_url(s): + """ + Encode an unicode object used in an URL. + """ + # According to RFC 2718, non-ascii characters in url's must be UTF-8 encoded. + + precondition(isinstance(s, unicode), s) + return s.encode('utf-8') + +def unicode_to_stdout(s): + """ + Encode an unicode object for representation on stdout. + """ + + precondition(isinstance(s, unicode), s) + return s.encode(sys.stdout.encoding, 'replace') } [cli: tahoe backup unicode support francois@ctrlaltdel.ch**20090427085931 Ignore-this: 1270f10115dfbc6a1661678cce279072 ] { hunk ./src/allmydata/scripts/cli.py 225 self['exclude'] = set() def parseArgs(self, localdir, topath): - self.from_dir = localdir - self.to_dir = topath + self.from_dir = argv_to_unicode(localdir) + self.to_dir = argv_to_unicode(topath) def getSynopsis(Self): return "%s backup FROM ALIAS:TO" % os.path.basename(sys.argv[0]) hunk ./src/allmydata/scripts/tahoe_backup.py 7 import urllib import simplejson import datetime +import sys from allmydata.scripts.common import get_alias, escape_path, DEFAULT_ALIAS from allmydata.scripts.common_http import do_http from allmydata import uri hunk ./src/allmydata/scripts/tahoe_backup.py 13 from allmydata.util import time_format from allmydata.scripts import backupdb +from allmydata.util.stringutils import fs_to_unicode, unicode_to_fs, unicode_to_stdout +from allmydata.util.assertutil import precondition +from twisted.python import usage class HTTPError(Exception): pass hunk ./src/allmydata/scripts/tahoe_backup.py 252 def verboseprint(self, msg): if self.verbosity >= 2: - print >>self.options.stdout, msg + print >>self.options.stdout, unicode_to_stdout(msg) def process(self, localpath, olddircap): hunk ./src/allmydata/scripts/tahoe_backup.py 255 + precondition(isinstance(localpath, unicode), localpath) # returns newdircap self.verboseprint("processing %s, olddircap %s" % (localpath, olddircap)) hunk ./src/allmydata/scripts/tahoe_backup.py 264 olddircontents = self.readdir(olddircap) newdircontents = {} # childname -> (type, rocap, metadata) - for child in self.options.filter_listdir(os.listdir(localpath)): + for child in self.options.filter_listdir(os.listdir(unicode_to_fs(localpath))): + child = fs_to_unicode(child) childpath = os.path.join(localpath, child) if os.path.isdir(childpath): metadata = get_local_metadata(childpath) hunk ./src/allmydata/scripts/tahoe_backup.py 351 return contents def upload(self, childpath): + precondition(isinstance(childpath, unicode), childpath) + #self.verboseprint("uploading %s.." % childpath) metadata = get_local_metadata(childpath) hunk ./src/allmydata/scripts/tahoe_backup.py 361 if must_upload: self.verboseprint("uploading %s.." % childpath) - infileobj = open(os.path.expanduser(childpath), "rb") + infileobj = open(unicode_to_fs(os.path.expanduser(childpath)), "rb") url = self.options['node-url'] + "uri" resp = do_http("PUT", url, infileobj) if resp.status not in (200, 201): hunk ./src/allmydata/test/test_cli.py 875 self.writeto("parent/subdir/bar.txt", "bar\n" * 1000) self.writeto("parent/blah.txt", "blah") + if sys.getfilesystemencoding() == "ANSI_X3.4-1968": + self.writeto(u"parent/artonwall.txt", "Marmelade Jacuzzi") + else: + self.writeto(u"parent/ärtonwall.txt", "Marmelade Jacuzzi") + def do_backup(use_backupdb=True, verbose=False): cmd = ["backup"] if not have_bdb or not use_backupdb: hunk ./src/allmydata/test/test_cli.py 904 self.failUnlessEqual(err, "") self.failUnlessEqual(rc, 0) fu, fr, dc, dr = self.count_output(out) - # foo.txt, bar.txt, blah.txt - self.failUnlessEqual(fu, 3) + # foo.txt, bar.txt, blah.txt, ärtonwall.txt + self.failUnlessEqual(fu, 4) self.failUnlessEqual(fr, 0) # empty, home, home/parent, home/parent/subdir self.failUnlessEqual(dc, 4) hunk ./src/allmydata/test/test_cli.py 954 self.failUnlessEqual(rc, 0) if have_bdb: fu, fr, dc, dr = self.count_output(out) - # foo.txt, bar.txt, blah.txt + # foo.txt, bar.txt, blah.txt, ärtonwall.txt self.failUnlessEqual(fu, 0) hunk ./src/allmydata/test/test_cli.py 956 - self.failUnlessEqual(fr, 3) + self.failUnlessEqual(fr, 4) # empty, home, home/parent, home/parent/subdir self.failUnlessEqual(dc, 0) self.failUnlessEqual(dr, 4) hunk ./src/allmydata/test/test_cli.py 984 self.failUnlessEqual(rc, 0) fu, fr, dc, dr = self.count_output(out) fchecked, dchecked, dread = self.count_output2(out) - self.failUnlessEqual(fchecked, 3) + self.failUnlessEqual(fchecked, 4) self.failUnlessEqual(fu, 0) hunk ./src/allmydata/test/test_cli.py 986 - self.failUnlessEqual(fr, 3) + self.failUnlessEqual(fr, 4) # TODO: backupdb doesn't do dirs yet; when it does, this will # change to dchecked=4, and maybe dread=0 self.failUnlessEqual(dchecked, 0) hunk ./src/allmydata/test/test_cli.py 1032 fu, fr, dc, dr = self.count_output(out) # new foo.txt, surprise file, subfile, empty self.failUnlessEqual(fu, 4) - # old bar.txt - self.failUnlessEqual(fr, 1) + # old bar.txt, ärtonwall.txt + self.failUnlessEqual(fr, 2) # home, parent, subdir, blah.txt, surprisedir self.failUnlessEqual(dc, 5) self.failUnlessEqual(dr, 0) hunk ./src/allmydata/test/test_cli.py 1072 self.failUnlessEqual(err, "") self.failUnlessEqual(rc, 0) fu, fr, dc, dr = self.count_output(out) - self.failUnlessEqual(fu, 5) + self.failUnlessEqual(fu, 6) self.failUnlessEqual(fr, 0) self.failUnlessEqual(dc, 0) self.failUnlessEqual(dr, 5) } [cli: tahoe manifest unicode support francois@ctrlaltdel.ch**20090427090144 Ignore-this: d2d3ca0614d6cbf725a4b0a0f06f7694 ] { hunk ./src/allmydata/scripts/cli.py 289 ("raw", "r", "Display raw JSON data instead of parsed"), ] def parseArgs(self, where=''): - self.where = where + self.where = argv_to_unicode(where) def getSynopsis(self): return "%s manifest [ALIAS:PATH]" % (os.path.basename(sys.argv[0]),) hunk ./src/allmydata/scripts/tahoe_manifest.py 83 try: print >>stdout, d["cap"], "/".join(d["path"]) except UnicodeEncodeError: - print >>stdout, d["cap"], "/".join([p.encode("utf-8") + print >>stdout, d["cap"], "/".join([unicode_to_stdout(p) for p in d["path"]]) def manifest(options): } [cli: tahoe ls unicode support francois@ctrlaltdel.ch**20090427091850 Ignore-this: 38257c3ddea0cbab857c2eb194c349ec ] { hunk ./src/allmydata/scripts/cli.py 87 ("json", None, "Show the raw JSON output"), ] def parseArgs(self, where=""): - self.where = where + self.where = argv_to_unicode(where) longdesc = """List the contents of some portion of the virtual drive.""" hunk ./src/allmydata/scripts/tahoe_ls.py 6 import simplejson from allmydata.scripts.common import get_alias, DEFAULT_ALIAS, escape_path from allmydata.scripts.common_http import do_http +from allmydata.util.stringutils import unicode_to_stdout def list(options): nodeurl = options['node-url'] hunk ./src/allmydata/scripts/tahoe_ls.py 116 line.append(ctime_s) if not options["classify"]: classify = "" - line.append(name + classify) + line.append(unicode_to_stdout(name) + classify) if options["uri"]: line.append(uri) if options["readonly-uri"]: } [cli: tahoe get unicode support francois@ctrlaltdel.ch**20090427091820 Ignore-this: 9cae48ab0055113379beab9d4ee504e5 ] hunk ./src/allmydata/scripts/cli.py 98 # tahoe get FOO bar # write to local file # tahoe get tahoe:FOO bar # same - self.from_file = arg1 - self.to_file = arg2 + self.from_file = argv_to_unicode(arg1) + + if arg2: + self.to_file = argv_to_unicode(arg2) + else: + self.to_file = None + if self.to_file == "-": self.to_file = None [cli: tahoe webopen unicode support francois@ctrlaltdel.ch**20090427090059 Ignore-this: 2b972b5f3f240dfbb7545b5acd388f81 ] hunk ./src/allmydata/scripts/cli.py 279 class WebopenOptions(VDriveOptions): def parseArgs(self, where=''): - self.where = where + self.where = argv_to_unicode(where) def getSynopsis(self): return "%s webopen [ALIAS:PATH]" % (os.path.basename(sys.argv[0]),) [cli: tahoe rm unicode support francois@ctrlaltdel.ch**20090427085847 Ignore-this: dadad2700725c85440b9c34d8ce3c68c ] hunk ./src/allmydata/scripts/cli.py 197 class MvOptions(VDriveOptions): def parseArgs(self, frompath, topath): - self.from_file = frompath - self.to_file = topath + self.from_file = argv_to_unicode(frompath) + self.to_file = argv_to_unicode(topath) def getSynopsis(self): return "%s mv FROM TO" % (os.path.basename(sys.argv[0]),) [cli: tahoe ln unicode support francois@ctrlaltdel.ch**20090427085913 Ignore-this: 82ced82ded50a925a5650ae6000b46a2 ] hunk ./src/allmydata/scripts/cli.py 205 class LnOptions(VDriveOptions): def parseArgs(self, frompath, topath): - self.from_file = frompath - self.to_file = topath + self.from_file = argv_to_unicode(frompath) + self.to_file = argv_to_unicode(topath) def getSynopsis(self): return "%s ln FROM TO" % (os.path.basename(sys.argv[0]),) [cli: tahoe status unicode support francois@ctrlaltdel.ch**20090427090213 Ignore-this: 370469b0cea44162bfa8d11e58ee72d4 ] hunk ./src/allmydata/scripts/cli.py 306 ("raw", "r", "Display raw JSON data instead of parsed"), ] def parseArgs(self, where=''): - self.where = where + self.where = argv_to_unicode(where) def getSynopsis(self): return "%s stats [ALIAS:PATH]" % (os.path.basename(sys.argv[0]),) [cli: tahoe check unicode support francois@ctrlaltdel.ch**20090427090226 Ignore-this: 59700729d6dc91c77ae46f4d053e79ce ] hunk ./src/allmydata/scripts/cli.py 321 ("add-lease", None, "Add/renew lease on all shares"), ] def parseArgs(self, where=''): - self.where = where + self.where = argv_to_unicode(where) def getSynopsis(self): return "%s check [ALIAS:PATH]" % (os.path.basename(sys.argv[0]),) [cli: tahoe deep-check unicode support francois@ctrlaltdel.ch**20090427090243 Ignore-this: 2d0bb5592ae8735a184e325bd1515dec ] hunk ./src/allmydata/scripts/cli.py 337 ("verbose", "v", "Be noisy about what is happening."), ] def parseArgs(self, where=''): - self.where = where + self.where = argv_to_unicode(where) def getSynopsis(self): return "%s deep-check [ALIAS:PATH]" % (os.path.basename(sys.argv[0]),) [cli: tahoe mkdir unicode support francois@ctrlaltdel.ch**20090427091047 Ignore-this: 1375e8ea621d862ed4e261a8fc6cd772 ] { hunk ./src/allmydata/scripts/cli.py 53 class MakeDirectoryOptions(VDriveOptions): def parseArgs(self, where=""): - self.where = where + self.where = argv_to_unicode(where) longdesc = """Create a new directory, either unlinked or as a subdirectory.""" class AddAliasOptions(VDriveOptions): hunk ./src/allmydata/scripts/tahoe_mkdir.py 5 import urllib from allmydata.scripts.common_http import do_http, check_http_error from allmydata.scripts.common import get_alias, DEFAULT_ALIAS +from allmydata.util.stringutils import unicode_to_url def mkdir(options): nodeurl = options['node-url'] hunk ./src/allmydata/scripts/tahoe_mkdir.py 35 path = path[:-1] # path (in argv) must be "/".join([s.encode("utf-8") for s in segments]) url = nodeurl + "uri/%s/%s?t=mkdir" % (urllib.quote(rootcap), - urllib.quote(path)) + urllib.quote(unicode_to_url(path))) resp = do_http("POST", url) check_http_error(resp, stderr) new_uri = resp.read().strip() hunk ./src/allmydata/test/test_cli.py 841 dn, "tahoe:")) return d +class Mkdir(GridTestMixin, CLITestMixin, unittest.TestCase): + def test_unicode_mkdir(self): + self.basedir = os.path.dirname(self.mktemp()) + self.set_up_grid() + + d = self.do_cli("create-alias", "tahoe") + d.addCallback(lambda res: self.do_cli("mkdir", "tahoe:Motörhead")) + + return d + + class Backup(GridTestMixin, CLITestMixin, StallMixin, unittest.TestCase): def writeto(self, path, data): } [docs: explain the current unicode support in CLI francois@ctrlaltdel.ch**20090427091252 Ignore-this: 42f2c3107c0c94172f878559944779ca ] hunk ./docs/frontends/CLI.txt 94 These commands also use a table of "aliases" to figure out which directory they ought to use a starting point. This is explained in more detail below. -In Tahoe v1.3.0, passing non-ascii characters to the cli is not guaranteed to -work, although it might work on your platform, especially if your platform -uses utf-8 encoding. +As of Tahoe v1.3.1, filenames containing non-ascii characters are +supported on the commande line if your terminal is correctly configured +for UTF-8 support. This is usually the case on moderns GNU/Linux +distributions. + +If your terminal doesn't support UTF-8, you will still be able to list +directories but non-ascii characters will be replaced by a question mark +(?) on display. + +Reading from and writing to files whose name contain non-ascii +characters is also supported when your system correctly understand them. +Under Unix, this is usually handled by locale settings. If Tahoe cannot +correctly decode a filename, it will raise an error. In such case, +you'll need to correct the name of your file, possibly with help from +tools such as convmv. === Starting Directories === [cli: tahoe cp unicode support francois@ctrlaltdel.ch**20090427135203 Ignore-this: cd26cdd0753625d1c1252cffd976abd6 ] { hunk ./src/allmydata/scripts/cli.py 185 def parseArgs(self, *args): if len(args) < 2: raise usage.UsageError("cp requires at least two arguments") - self.sources = args[:-1] - self.destination = args[-1] + self.sources = map(argv_to_unicode, args[:-1]) + self.destination = argv_to_unicode(args[-1]) class RmOptions(VDriveOptions): def parseArgs(self, where): hunk ./src/allmydata/scripts/cli.py 190 - self.where = where + self.where = argv_to_unicode(where) def getSynopsis(self): return "%s rm VDRIVE_FILE" % (os.path.basename(sys.argv[0]),) hunk ./src/allmydata/scripts/tahoe_cp.py 7 import simplejson from cStringIO import StringIO from twisted.python.failure import Failure +import sys from allmydata.scripts.common import get_alias, escape_path, DefaultAliasMarker from allmydata.scripts.common_http import do_http from allmydata import uri hunk ./src/allmydata/scripts/tahoe_cp.py 11 +from twisted.python import usage +from allmydata.util.stringutils import fs_to_unicode, unicode_to_fs, unicode_to_url +from allmydata.util.assertutil import precondition def ascii_or_none(s): if s is None: hunk ./src/allmydata/scripts/tahoe_cp.py 76 class LocalFileSource: def __init__(self, pathname): + precondition(isinstance(pathname, unicode), pathname) self.pathname = pathname def need_to_copy_bytes(self): hunk ./src/allmydata/scripts/tahoe_cp.py 87 class LocalFileTarget: def __init__(self, pathname): + precondition(isinstance(pathname, unicode), pathname) self.pathname = pathname def put_file(self, inf): outf = open(self.pathname, "wb") hunk ./src/allmydata/scripts/tahoe_cp.py 100 class LocalMissingTarget: def __init__(self, pathname): + precondition(isinstance(pathname, unicode), pathname) self.pathname = pathname def put_file(self, inf): hunk ./src/allmydata/scripts/tahoe_cp.py 114 class LocalDirectorySource: def __init__(self, progressfunc, pathname): + precondition(isinstance(pathname, unicode), pathname) + self.progressfunc = progressfunc self.pathname = pathname self.children = None hunk ./src/allmydata/scripts/tahoe_cp.py 124 if self.children is not None: return self.children = {} - children = os.listdir(self.pathname) + children = os.listdir(unicode_to_fs(self.pathname)) for i,n in enumerate(children): hunk ./src/allmydata/scripts/tahoe_cp.py 126 + n = fs_to_unicode(n) self.progressfunc("examining %d of %d" % (i, len(children))) pn = os.path.join(self.pathname, n) if os.path.isdir(pn): hunk ./src/allmydata/scripts/tahoe_cp.py 142 class LocalDirectoryTarget: def __init__(self, progressfunc, pathname): + precondition(isinstance(pathname, unicode), pathname) + self.progressfunc = progressfunc self.pathname = pathname self.children = None hunk ./src/allmydata/scripts/tahoe_cp.py 152 if self.children is not None: return self.children = {} - children = os.listdir(self.pathname) + children = os.listdir(unicode_to_fs(self.pathname)) for i,n in enumerate(children): hunk ./src/allmydata/scripts/tahoe_cp.py 154 + n = fs_to_unicode(n) self.progressfunc("examining %d of %d" % (i, len(children))) pn = os.path.join(self.pathname, n) if os.path.isdir(pn): hunk ./src/allmydata/scripts/tahoe_cp.py 176 return LocalDirectoryTarget(self.progressfunc, pathname) def put_file(self, name, inf): + precondition(isinstance(name, unicode), name) pathname = os.path.join(self.pathname, name) hunk ./src/allmydata/scripts/tahoe_cp.py 178 - outf = open(pathname, "wb") + outf = open(unicode_to_fs(pathname), "wb") while True: data = inf.read(32768) if not data: hunk ./src/allmydata/scripts/tahoe_cp.py 367 if self.writecap: url = self.nodeurl + "/".join(["uri", urllib.quote(self.writecap), - urllib.quote(name.encode('utf-8'))]) + urllib.quote(unicode_to_url(name))]) self.children[name] = TahoeFileTarget(self.nodeurl, mutable, writecap, readcap, url) else: hunk ./src/allmydata/test/test_cli.py 799 def test_unicode_filename(self): self.basedir = "cli/Cp/unicode_filename" self.set_up_grid() + d = self.do_cli("create-alias", "tahoe") + + # Use unicode strings when calling os functions + if sys.getfilesystemencoding() == "ANSI_X3.4-1968": + fn1 = os.path.join(self.basedir, u"Artonwall") + else: + fn1 = os.path.join(self.basedir, u"Ärtonwall") hunk ./src/allmydata/test/test_cli.py 807 - fn1 = os.path.join(self.basedir, "Ärtonwall") DATA1 = "unicode file content" open(fn1, "wb").write(DATA1) hunk ./src/allmydata/test/test_cli.py 809 + d.addCallback(lambda res: self.do_cli("cp", fn1.encode('utf-8'), "tahoe:Ärtonwall")) + + d.addCallback(lambda res: self.do_cli("get", "tahoe:Ärtonwall")) + d.addCallback(lambda (rc,out,err): self.failUnlessEqual(out, DATA1)) + hunk ./src/allmydata/test/test_cli.py 815 - fn2 = os.path.join(self.basedir, "Metallica") + fn2 = os.path.join(self.basedir, u"Metallica") DATA2 = "non-unicode file content" open(fn2, "wb").write(DATA2) hunk ./src/allmydata/test/test_cli.py 822 # Bug #534 # Assure that uploading a file whose name contains unicode character doesn't # prevent further uploads in the same directory - d = self.do_cli("create-alias", "tahoe") - d.addCallback(lambda res: self.do_cli("cp", fn1, "tahoe:")) - d.addCallback(lambda res: self.do_cli("cp", fn2, "tahoe:")) - - d.addCallback(lambda res: self.do_cli("get", "tahoe:Ärtonwall")) - d.addCallback(lambda (rc,out,err): self.failUnlessEqual(out, DATA1)) + d.addCallback(lambda res: self.do_cli("cp", fn2.encode('utf-8'), "tahoe:")) d.addCallback(lambda res: self.do_cli("get", "tahoe:Metallica")) d.addCallback(lambda (rc,out,err): self.failUnlessEqual(out, DATA2)) hunk ./src/allmydata/test/test_cli.py 827 + d.addCallback(lambda res: self.do_cli("ls", "tahoe:")) + return d test_unicode_filename.todo = "This behavior is not yet supported, although it does happen to work (for reasons that are ill-understood) on many platforms. See issue ticket #534." } [cli: tahoe put unicode support + encode path elements using unicode_to_url francois@ctrlaltdel.ch**20090429162003 Ignore-this: 6285a8c89837290173e9a4d3744e1173 Both changes (tahoe put and url encoding) had to be included in the same patch to prevent failing tests. ] { hunk ./src/allmydata/scripts/cli.py 140 # tahoe put bar tahoe:FOO # same if arg1 is not None and arg2 is not None: - self.from_file = arg1 - self.to_file = arg2 + self.from_file = argv_to_unicode(arg1) + self.to_file = argv_to_unicode(arg2) elif arg1 is not None and arg2 is None: hunk ./src/allmydata/scripts/cli.py 143 - self.from_file = arg1 # might be "-" + self.from_file = argv_to_unicode(arg1) # might be "-" self.to_file = None else: self.from_file = None hunk ./src/allmydata/scripts/cli.py 148 self.to_file = None - if self.from_file == "-": + if self.from_file == u"-": self.from_file = None def getSynopsis(self): hunk ./src/allmydata/scripts/common.py 166 def escape_path(path): segments = path.split("/") - return "/".join([urllib.quote(s) for s in segments]) + return "/".join([urllib.quote(unicode_to_url(s)) for s in segments]) hunk ./src/allmydata/test/test_cli.py 743 d.addCallback(lambda (rc,out,err): self.failUnlessEqual(out, DATA2)) return d + def test_immutable_from_file_unicode(self): + # tahoe put file.txt "à trier.txt" + self.basedir = os.path.dirname(self.mktemp()) + self.set_up_grid() + + rel_fn = os.path.join(self.basedir, "DATAFILE") + abs_fn = os.path.abspath(rel_fn) + # we make the file small enough to fit in a LIT file, for speed + DATA = "short file" + f = open(rel_fn, "w") + f.write(DATA) + f.close() + + d = self.do_cli("create-alias", "tahoe") + + d.addCallback(lambda res: + self.do_cli("put", rel_fn, "à trier.txt")) + def _uploaded((rc,stdout,stderr)): + readcap = stdout.strip() + self.failUnless(readcap.startswith("URI:LIT:")) + self.failUnless("201 Created" in stderr, stderr) + self.readcap = readcap + d.addCallback(_uploaded) + + d.addCallback(lambda res: + self.do_cli("get", "tahoe:à trier.txt")) + d.addCallback(lambda (rc,stdout,stderr): + self.failUnlessEqual(stdout, DATA)) + + return d + class List(GridTestMixin, CLITestMixin, unittest.TestCase): def test_list(self): self.basedir = "cli/List/list" } [cli: aliases unicode support francois@ctrlaltdel.ch**20090427095807 Ignore-this: e2c49f5851ab89819acc633a196cc3df ] { hunk ./src/allmydata/scripts/cli.py 58 class AddAliasOptions(VDriveOptions): def parseArgs(self, alias, cap): - self.alias = alias + self.alias = argv_to_unicode(alias) self.cap = cap def getSynopsis(self): hunk ./src/allmydata/scripts/cli.py 68 class CreateAliasOptions(VDriveOptions): def parseArgs(self, alias): - self.alias = alias + self.alias = argv_to_unicode(alias) def getSynopsis(self): return "%s create-alias ALIAS" % (os.path.basename(sys.argv[0]),) hunk ./src/allmydata/scripts/common.py 3 import os, sys, urllib +import codecs from twisted.python import usage from allmydata.util.stringutils import unicode_to_url hunk ./src/allmydata/scripts/common.py 104 except EnvironmentError: pass try: - f = open(aliasfile, "r") + f = codecs.open(aliasfile, "r", "utf-8") for line in f.readlines(): line = line.strip() if line.startswith("#") or not line: hunk ./src/allmydata/scripts/common.py 111 continue name, cap = line.split(":", 1) # normalize it: remove http: prefix, urldecode - cap = cap.strip() + cap = cap.strip().encode('ascii') aliases[name] = uri.from_string_dirnode(cap).to_string() except EnvironmentError: pass hunk ./src/allmydata/scripts/tahoe_add_alias.py 3 import os.path +import codecs +import sys from allmydata import uri from allmydata.scripts.common_http import do_http, check_http_error from allmydata.scripts.common import get_aliases hunk ./src/allmydata/scripts/tahoe_add_alias.py 8 +from allmydata.util.stringutils import unicode_to_stdout def add_alias(options): nodedir = options['node-directory'] hunk ./src/allmydata/scripts/tahoe_add_alias.py 58 new_uri = resp.read().strip() # probably check for others.. - f = open(aliasfile, "a") + f = codecs.open(aliasfile, "a", "utf-8") f.write("%s: %s\n" % (alias, new_uri)) f.close() hunk ./src/allmydata/scripts/tahoe_add_alias.py 61 - print >>stdout, "Alias '%s' created" % (alias,) + print >>stdout, "Alias '%s' created" % (unicode_to_stdout(alias),) return 0 def list_aliases(options): hunk ./src/allmydata/test/test_cli.py 522 self._test_webopen(["two:"], self.two_url) d.addCallback(_test_urls) + d.addCallback(lambda res: self.do_cli("create-alias", "études")) + def _check_create_unicode((rc,stdout,stderr)): + self.failUnlessEqual(rc, 0) + self.failIf(stderr) + + # If stdout only supports ascii, accentuated characters are + # being replaced by '?' + if sys.stdout.encoding == "ANSI_X3.4-1968": + self.failUnless("Alias '?tudes' created" in stdout) + else: + self.failUnless("Alias 'études' created" in stdout) + + aliases = get_aliases(self.get_clientdir()) + self.failUnless(aliases[u"études"].startswith("URI:DIR2:")) + d.addCallback(_check_create_unicode) + + d.addCallback(lambda res: self.do_cli("ls", "études:")) + def _check_ls1((rc, stdout, stderr)): + self.failUnlessEqual(rc, 0) + self.failIf(stderr) + + self.failUnlessEqual(stdout, "") + d.addCallback(_check_ls1) + + d.addCallback(lambda res: self.do_cli("put", "-", "études:uploaded.txt", + stdin="Blah blah blah")) + + d.addCallback(lambda res: self.do_cli("ls", "études:")) + def _check_ls2((rc, stdout, stderr)): + self.failUnlessEqual(rc, 0) + self.failIf(stderr) + + self.failUnlessEqual(stdout, "uploaded.txt\n") + d.addCallback(_check_ls2) + + d.addCallback(lambda res: self.do_cli("get", "études:uploaded.txt")) + def _check_get((rc, stdout, stderr)): + self.failUnlessEqual(rc, 0) + self.failIf(stderr) + self.failUnlessEqual(stdout, "Blah blah blah") + d.addCallback(_check_get) + return d class Put(GridTestMixin, CLITestMixin, unittest.TestCase): } [cli: previous cli related patches should fix #534 francois@ctrlaltdel.ch**20090427135239 Ignore-this: 82709853d0d871651b6673929616ec08 ] hunk ./src/allmydata/test/test_cli.py 903 d.addCallback(lambda res: self.do_cli("ls", "tahoe:")) return d - test_unicode_filename.todo = "This behavior is not yet supported, although it does happen to work (for reasons that are ill-understood) on many platforms. See issue ticket #534." def test_dangling_symlink_vs_recursion(self): if not hasattr(os, 'symlink'): Context: [docs: setup: Norm Hardy suggested that it would be easier if users realized that they already had Python (especially true for Mac users) zooko@zooko.com**20090325035459 Ignore-this: e1bb76a1be4d6d541090d8d9e7e73db9 ] [expirer: include crawler progress in the JSON status output warner@allmydata.com**20090324205137 Ignore-this: f65ccc0a57ed7c5d9565131350d87d2d ] [docs/proposed: new Accounting overview, discuss in #666 warner@allmydata.com**20090324015752 Ignore-this: 7379396536e85194b1c1bec21288adc7 ] [docs/proposed: move old accounting docs out of the way warner@allmydata.com**20090324015457 Ignore-this: 196fe85ef3592f7ef8e58a101eed0494 ] [move GC docs out of proposed/, since it's all implemented now. Add reference to configuration.txt . Add expire.*= suggestions to tahoe.cfg . warner@allmydata.com**20090323230820 Ignore-this: ce0ec977f49c4a3642cf3a7ea7789dc4 ] [storage: use constant-time comparison for write-enablers and lease-secrets warner@lothar.com**20090323032128 Ignore-this: efbbb15242b3f306c32498aae7fd75d ] [hashutil: add constant-time comparison function, to avoid timing attacks when python's short-circuiting data-dependent == operator is used to, say, check a write-enabler warner@lothar.com**20090323032055 Ignore-this: c5c1f5e529ab1b352c949f3e0d9abf20 ] [WUI: fix display of empty directories, it threw an exception before warner@allmydata.com**20090320235809 Ignore-this: e598bb806d75411d202ba90fc251ad2b ] [storage webstatus: insert spaces when we're configured to expire multiple sharetypes warner@allmydata.com**20090320224450 Ignore-this: 18f44396384c8598d984eaa11dc92cf7 ] [expirer: tolerate empty buckets, refactor bucketsize-counting code a bit, don't increment -mutable/-immutable counters unless we actually know the sharetype warner@allmydata.com**20090320191816 Ignore-this: 16827fe12b874a0205b0a7fdb6b7f960 ] [storage: improve wording of status message warner@allmydata.com**20090319184837 Ignore-this: 66684623d50c89fa02915665dbbfec98 ] [parse_date: insist that it returns an int warner@allmydata.com**20090319184443 Ignore-this: a1f5ae0c16a151d218c06fc4411f03c1 ] [storage status: report expiration-cutoff-date like 19-Mar-2009 (as opposed to the tahoe.cfg input format of 2009-03-19), for redundancy: someone who gets the month and day switched will have a better chance to spot the problem in the storage-status output if it's in a different format warner@allmydata.com**20090319180756 Ignore-this: df2c3778260c6389d32f4b47b004ab35 ] [expirer: clean up constructor args, add tahoe.cfg controls, use cutoff_date instead of date_cutoff warner@allmydata.com**20090319010009 Ignore-this: 2b6aaa6d5e6ff9fd417f32978b443fd2 ] [util/time_format: new routine to parse dates like 2009-03-18, switch expirer to use it. I'd prefer to use 18-Mar-2009, but it is surprisingly non-trivial to build a parser that will take UTC dates instead of local dates warner@allmydata.com**20090319005814 Ignore-this: 4f08b301c780a78084fa02a09b5dda3 ] [expirer: change setup, config options, in preparation for adding tahoe.cfg controls warner@allmydata.com**20090319002138 Ignore-this: b23a53e97f2a9fb7a005e9fe40e83fac ] [expirer: track mutable-vs-immutable sharecounts and sizes, report them on the web status page for comparison warner@allmydata.com**20090318202504 Ignore-this: 87e809bf5dedef3f0bc8f4a7b90e42d2 ] [add utility function to parse durations, for lease-expiration config warner@lothar.com**20090317070117 Ignore-this: 2f4fa4352c887b4767969f0ea7931644 ] [expirer: add mode to expire only-mutable or only-immutable shares warner@lothar.com**20090317065118 Ignore-this: b0b25427e3b1516bdfe293528b8e4a4e ] [tahoe_ls.py: add comment about error cases to improve warner@lothar.com**20090317051206 Ignore-this: 7c678b92a55b2f7c6f0d96fb6ece74ed ] [GC: add date-cutoff -based expiration, add proposed docs warner@lothar.com**20090317051041 Ignore-this: a5c0ecbcc2666eb04f2daa67331d1948 ] [test_deepcheck: remove the 10s timeout: our dapper buildslave requires 30s, and the reduced timeout was only there because this tests fails by timeout rather than explicitly warner@lothar.com**20090316005134 Ignore-this: 4ce081283c26719be513b5ce8381a81a ] [tahoe cp -r: add --caps-only flag, to write filecaps into local files instead of actual file contents. Used only for debugging and as a quick tree-comparison tool. warner@lothar.com**20090315231958 Ignore-this: 8ecdf2b08601ae9e9fec5885bf640262 ] [dirnode deep_traverse: insert a turn break (fireEventually) at least once every 100 files, otherwise a CHK followed by more than 158 LITs can overflow the stack, sort of like #237. warner@allmydata.com**20090313233135 Ignore-this: 39b78faa947ed9461f2d120f6843e59f ] [consolidate: remove pointless 'else' after for loop warner@allmydata.com**20090313082751 Ignore-this: 517f41316f7ae40b92b9eef8269a4b69 ] [consolidate: add eta, flush stdout warner@allmydata.com**20090313082451 Ignore-this: 845f63adccc32557c6864ae6120ba836 ] [consolidate: tolerate unicode dirnames warner@allmydata.com**20090313065402 Ignore-this: 7e65703ed3d12d4bd5ec14b693e5f61f ] [dirnode.py: when doing deep-traverse, walk each directory in alphabetical order, to make things like 'manifest' more predictable warner@allmydata.com**20090313065046 Ignore-this: 9a80055a93a6b11853d4e8202bacec14 ] [consolidator: add progress to scan-old-directory passes warner@allmydata.com**20090313054728 Ignore-this: adc67a34f4f19fd58c5bc76301b3df36 ] [consolidator: fix cycle detection to not trigger on merely shared directories, add snapshot counter to progress warner@allmydata.com**20090313042229 Ignore-this: eba2cf9f1b1364b8e4c5ae4fa030a99f ] [consolidator: re-use more directories, add total directories seen-vs-used counts warner@allmydata.com**20090313034801 Ignore-this: 6e743d2940a9fe129cee31008c894d70 ] [tahoe_backup.py: tolerate more time formats warner@allmydata.com**20090313011600 Ignore-this: ca74f56f0dce7d19810c5a7a75bc623c ] [consolidator: add more verbose traversal of directories warner@allmydata.com**20090312232900 Ignore-this: 8ff0e17c6174566832a566a111032db4 ] [consolidate: create multiple numbered backups of the original Archives directory, not just the first time warner@allmydata.com**20090312230427 Ignore-this: e4985f76969b584d099b050781aa561c ] [add 'tahoe debug consolidate' command, to merge directories created by repeated 'tahoe cp -r' or the allmydata win32 backup tool, into the form that would have been created by 'tahoe backup'. warner@allmydata.com**20090312205606 Ignore-this: 66569ca2190aa7b0f9199bcf09dcb27e ] [tahoe_cp.py: return 0 for success, instead of None warner@allmydata.com**20090312205345 Ignore-this: 808f6d8617f8c4e7fde455e6c7639fab ] [setup: specify in the debian/control files that tahoe is compatible with Python 2.6 zooko@zooko.com**20090311225902 Ignore-this: d0793013e4c868d92793d932ef92a62d ] [setup: update the debian/copyright text to reflect the current licences zooko@zooko.com**20090311152952 Ignore-this: 806a95b1b79d6bb20507db5c7201af45 ] [expirer: fix prediction math, thanks to Zandr for the catch warner@allmydata.com**20090309204217 Ignore-this: e20f784483ea1a15e35627fe95506272 ] [storage.expirer: oops, fix upgrade-handler code warner@lothar.com**20090309035516 Ignore-this: af8bc0ea1c7f305b33c06f4a9bf58de3 ] [storage.expirer: handle upgrades better warner@lothar.com**20090309034220 Ignore-this: dd654305b94bc3209426a3e008d57363 ] [storage.expirer: exercise the last missing line of webstatus code warner@lothar.com**20090309033828 Ignore-this: fc4aa34734cae32eec1db623ca0b145b ] [expirer: tolerate corrupt shares, add them to the state and history for future examination warner@lothar.com**20090309030840 Ignore-this: 5ae7e68471ed700cc68beb408e0f303 ] [storage/immutable: raise a specific error upon seeing a bad version number, instead of using assert. Also wrap to 80cols. warner@lothar.com**20090309030732 Ignore-this: 5331d9680ffceff029fbbbcdece7f282 ] [storage: add test coverage for BucketReader.__repr__ warner@lothar.com**20090309030527 Ignore-this: 60ce4e8c5200cc9885a3057f914e78b4 ] [storage/mutable: raise a specific error upon seeing bad magic, instead of using assert warner@lothar.com**20090309020201 Ignore-this: 8daa77362902f5d6ef793e9602a1383b ] [nodeadmin: node stops itself if a hotline file hasn't been touched in 120 seconds now, instead of in 60 seconds zooko@zooko.com**20090308025039 Ignore-this: a9cfc605d3fcad912bd2d0b35d5143ec A test failed on draco (MacPPC) because it took 67.1 seconds to get around to running the test, and the node had already stopped itself when the hotline file was 60 seconds old. ] [expirer: make web display a bit more consistent warner@lothar.com**20090307221442 Ignore-this: 7fec9d9bffc0bddeb51c1baa8e7ea020 ] [web/storage.py: tolerate unknown-future displays, I'm not sure why LeaseCrawler.test_unpredictable_future didn't catch this warner@lothar.com**20090307220243 Ignore-this: 3d4e5baa8cc6d5d26edcea29fda8593d ] [tahoe_cp.py: improve error reporting slightly: don't json-interpret HTTP errors, pass through tahoe webapi error messages warner@lothar.com**20090307114051 Ignore-this: a8beccb67adb082a92509d439b27d68e ] [test_cli.py: assert that 'ls' on an unrecoverable file now gives a better error message warner@lothar.com**20090307110815 Ignore-this: 18e9758e4d0ca7faeaf5bd481c2d72ff ] [test_web: test behavior of broken-dirnode GET, both html and json warner@lothar.com**20090307105707 Ignore-this: c0e5b45eee28959f899efa1bd189d6bd ] [web: when a dirnode can't be read, emit a regular HTML page but with the child-table and upload-forms replaced with an apologetic message. Make sure to include the 'get info' links so the user can do a filecheck warner@lothar.com**20090307105601 Ignore-this: f949d6bd58c0c2fd60fd5fa730115f3f ] [web/common: split out exception-to-explanation+code mapping to a separate humanize_failure() function, so it can be used by other code. Add explanation for mutable UnrecoverableFileError. warner@lothar.com**20090307105408 Ignore-this: 92f326804ba73bb446c5df5992b7d72e ] [test_cli: use explicit (and stable) testdirs, instead of using self.mktemp warner@lothar.com**20090307090428 Ignore-this: 7c58d159e4f33d01635c3445d9e591f9 ] [test_storage: solaris doesn't appear to give the same block count as other platforms, so don't assert as much about 'diskbytes' recovered warner@lothar.com**20090307084518 Ignore-this: 55b35c094ce78c50c8ede42062c5ea13 ] [tahoe_cp.py: don't be verbose unless --verbose is passed: default is one line of success warner@lothar.com**20090307084421 Ignore-this: f75bb818081e0c2993378154775a69d1 ] [setup.cfg: use allmydata.org as a mirror for some of our necessary dependencies warner@allmydata.com**20090307045753 Ignore-this: ce174de4ad2dd96f184eadf4d62ea779 ] [storage: add a lease-checker-and-expirer crawler, plus web status page. warner@allmydata.com**20090307044517 Ignore-this: 4355224f89b959c6f1a256a7e6c88c3b This walks slowly through all shares, examining their leases, deciding which are still valid and which have expired. Once enabled, it will then remove the expired leases, and delete shares which no longer have any valid leases. Note that there is not yet a tahoe.cfg option to enable lease-deletion: the current code is read-only. A subsequent patch will add a tahoe.cfg knob to control this, as well as docs. Some other minor items included in this patch: tahoe debug dump-share has a new --leases-only flag storage sharefile/leaseinfo code is cleaned up storage web status page (/storage) has more info, more tests coverage space-left measurement on OS-X should be more accurate (it was off by 2048x) (use stat .f_frsize instead of f_bsize) ] [setup: comment-out the use of the directory full of dependency packages on the test grid -- the test grid is too unreliable! :-( zooko@zooko.com**20090307044332 Ignore-this: c09f2e5f0b383b4227df4b07c267cb05 ] [setup: create a "make deb-lenny-head" target zooko@zooko.com**20090306191057 Ignore-this: 4b2ff187a3d08dcfe9318980ca92f097 I made this patch by copying [20090305220021-92b7f-89d987c7d05306b5cb03a64f2956a652c10a7296] and changing the name from "intrepid" to "lenny". I haven't tested it. ] [trivial: remove unused import detected by pyflakes, and remove trailing whitespace zooko@zooko.com**20090305223204 Ignore-this: 991001a50da1a357a519c3cb880d7ee1 ] [setup: enable build of .debs for Ubuntu Intrepid, thanks to DarKNesS_WolF zooko@zooko.com**20090305220021 Ignore-this: 88dbb3f72c2446b7734ac437189b67df ] [trivial: use more specific function for ascii-encoding storage index zooko@zooko.com**20090222175751 Ignore-this: c1a78a71cf04d7dbd80a71106f9b21a1 ] [doc: update how_to_make_a_tahoe_release.txt zooko@zooko.com**20090222175739 Ignore-this: 6a2e1592741b362bc170167a9cadc0b ] [test_cli: validate non-HTML error response of 'tahoe get' on an unrecoverable file warner@lothar.com**20090304041146] [web: full patch for HTML-vs-plaintext traceback renderings, improve test coverage of exception rendering warner@lothar.com**20090304035630] [web/common.py: use 'Accept:' header to control HTML-vs-text/plain traceback renderings warner@lothar.com**20090304035457] [test/common: rearrange shouldHTTPError warner@lothar.com**20090304014059] [web: move plural() to common.py warner@lothar.com**20090304014019] [add more information to NotEnoughSharesError, split out new exceptions for no-servers and no-source-of-ueb-hash warner@lothar.com**20090304013715] [test_web: move shouldHTTPError into common.WebErrorMixin warner@lothar.com**20090303225620] [test_web: clean up shouldHTTPError methods warner@lothar.com**20090303224604] [immutable checker add-lease: catch remote IndexError here too warner@allmydata.com**20090227071724 Ignore-this: 94ee6064ce26409381c6451cd50fdade ] [servermap add-lease: fix the code that's supposed to catch remote IndexErrors, I forgot that they present as ServerFailures instead. This should stop the deluge of Incidents that occur when you do add-lease against 1.3.0 servers warner@allmydata.com**20090227070426 Ignore-this: 3d7bc87d9587b51d44b27317d9eded23 ] [rrefutil: add check_remote utility function warner@allmydata.com**20090227065957 Ignore-this: d859d0fae87b7e84ad3c2893350ed519 ] [rrefutil: add trap_remote utility and friends warner@allmydata.com**20090227065524 Ignore-this: a594050cdd9bcca073d8029819dbc35 ] [web: improve layout of storage status with a table warner@allmydata.com**20090227015838 Ignore-this: 5c12f5e296ad1cf224f85e307813eb09 ] [crawler: add ETA to get_progress() warner@allmydata.com**20090227014248 Ignore-this: 27ae76c0530b83323209be70df3d8a4b ] [Fix for bug #645, correct path handling logic so that it works from sshfs Alberto Berti **20090226150237 Ignore-this: e9c1b2d48ebf4ba68100d76e54154a78 ] [wui: s/Provisioning/Reliability/ ; suggested by Terrell zooko@zooko.com**20090226030904 Ignore-this: f9559702e760dc3c57e2fc36f087d91e ] [startstop_node.py: when restarting, tolerate a stale pid too warner@allmydata.com**20090226015245 Ignore-this: 59197d3f400417be64fa0f91a64fae12 ] [web: fix the ERROR: line to work the same in python2.4 and 2.5 warner@lothar.com**20090225074621] [test_cli/test_web: fix spurious test failure on solaris (maybe python2.4?) due to variations in the way that exceptions are stringified warner@allmydata.com**20090225060128 Ignore-this: 829fb7d67fd199babdde1443b64a5381 ] [CLI: modify 'tahoe manifest' and 'tahoe deep-check' to report ERROR: properly. For #590. warner@allmydata.com**20090225054415 Ignore-this: 99162f894fdd24112a869e14848c3dea ] [webapi: modify streaming deep-manifest/deep-checker to emit an ERROR: line if they encounter an unrecoverable+untraversable directory. For #590. warner@allmydata.com**20090225051335 Ignore-this: e6bc49368fb0e7ada7cff477fd63ffe6 ] [scripts/common: fix alias handling on windows again, emit slightly nicer error message in response to an unknown alias warner@allmydata.com**20090225042136 Ignore-this: 76df800d131aed6701b5c7408105b134 ] [#165: make 'tahoe restart --force' the default behavior: warn but do not stop if restart is used on something that wasn't a running node, and always try to start it afterwards. This is particularly important for #315 (restart -m), because otherwise a single not-already-running node will prevent all nodes from being restarted, resulting in longer downtime than necessary warner@allmydata.com**20090225034213 Ignore-this: 26d8b0eea0279ca793faf23c5e948d90 ] [test_deepcheck: switch deep-check tests to use no-network too. This cuts the runtime down by about 50% warner@allmydata.com**20090225030457 Ignore-this: b3a98ed18c5752c9016c047e95d42b ] [test_deepcheck: convert MutableChecker to no-network GridTest warner@allmydata.com**20090225020010 Ignore-this: eccba7fda129330b642886271a61a573 ] [tests/no_network: move GET into the GridTestMixin class warner@allmydata.com**20090225003300 Ignore-this: 7779ad38c2d687ae328ba3cb6164a7a4 ] [common_web.py: oops, add .fields to the other FakeRequest warner@allmydata.com**20090225000459 Ignore-this: 7144269b5e083553ee2c3e7afea00604 ] [test_cli: exercise the recent tolerate-'c:\dir\file.txt' fix in scripts/common, recorded in a separate match to make it easier to merge the fix to prod warner@allmydata.com**20090224235620 Ignore-this: 2cae196dd4ccb578b2abae085376e0d7 ] [scripts/common: on windows, tolerate paths like 'c:\dir\file.txt', by treating single-letter aliases on windows/cygwin as non-aliases warner@allmydata.com**20090224235522 Ignore-this: 96d37644b7f81ac768ff4a1d1915eb46 ] [test/common_web.py: add a .fields attribute to our FakeRequest, since we support versions of Nevow that are old enough to not do it themselves warner@allmydata.com**20090224232050 Ignore-this: 239aaed746128e54e886762782541a1f ] [Two small fixes on documentation for cli backup command. Alberto Berti **20090224223634 Ignore-this: 5634a6dadad6e4e43a112de7fe5c74c ] [test_web: add (disabled) test to see what happens when deep-check encounters an unrecoverable directory. We still need code changes to improve this behavior. warner@lothar.com**20090224214017 Ignore-this: e839f1b0ec40f53fedcd809c2a30d5f9 ] [Add elapsed timestamp to cli backup command final summary. Alberto Berti **20090224171425 Ignore-this: 9a042d11f95ee9f6858a5096d513c0bc ] [Added documentation for '--exclude' and friends cli backup command. Alberto Berti **20090224153049 Ignore-this: bbc791fa56e38535bb82cc3077ffde90 ] [test_repairer: change to use faster no_network.GridTestMixin, split Verifier tests into separate cases, refactor judgement funcs into shared methods warner@lothar.com**20090224041506 Ignore-this: 584ce72d6276da5edc00562793d4ee53 ] [immutable/checker.py: trap ShareVersionIncompatible too. Also, use f.check warner@lothar.com**20090224041405 Ignore-this: b667e8d3192116293babcacdeed42898 instead of examining the value returned by f.trap, because the latter appears to squash exception types down into their base classes (i.e. since ShareVersionIncompatible is a subclass of LayoutInvalid, f.trap(Failure(ShareVersionIncompatible)) == LayoutInvalid). All this resulted in 'incompatible' shares being misclassified as 'corrupt'. ] [immutable/layout.py: wrap to 80 cols, no functional changes warner@lothar.com**20090224005837 Ignore-this: 40019480180ec34141506a28d7711608 ] [test_repairer: change Repairer to use much-faster no_network.GridTestMixin. As a side-effect, fix what I think was a bug: some of the assert-minimal-effort-expended checks were mixing write counts and allocate counts warner@lothar.com**20090223234227 Ignore-this: d58bd0a909f9939775730cda4a858cae ] [test/no_network.py: add a basic stats provider warner@lothar.com**20090223233937 Ignore-this: c9f3cc4eed99cfc36f68938ceff4162c ] [tests: stop using setUpClass/tearDownClass, since they've been deprecated in Twisted-8.2.0 warner@lothar.com**20090223204312 Ignore-this: 24c6592141cf64103530c024f93a5b88 ] [test_checker: improve test coverage for checker results warner@lothar.com**20090223201943 Ignore-this: 83e173602f0f4c811a7a9893d85385df ] [Fixed tests again so they will pass on windows. Alberto Berti **20090223003502 Ignore-this: 80d5074e7153642a2fa2a77958bfb50d ] [misc/*: remove RuntimeError too warner@lothar.com**20090222233401 Ignore-this: b76f8a184f75bb28eb9d8002f957936a ] [scripts: stop using RuntimeError, for #639 warner@lothar.com**20090222233106 Ignore-this: 686a424442670fffbd4d1816c284a601 ] [mutable/publish: stop using RuntimeError, for #639 warner@lothar.com**20090222233056 Ignore-this: 2a80a661c7850d97357caddad48c6e9d ] [remove more RuntimeError from unit tests, for #639 warner@lothar.com**20090222232855 Ignore-this: 1a1c3e1457f3f29ba7101fe406ee5f43 ] [stop using RuntimeError in unit tests, for #639 warner@lothar.com**20090222232722 Ignore-this: 475ce0c0dcd7a1f5ed83ef460312efea ] [ftpd/sftpd: stop using RuntimeError, for #639 warner@lothar.com**20090222232426 Ignore-this: 97001362c4ba9e94b2e254e229b79987 ] [Added tests for the cse when listdir is an iterator Alberto Berti **20090222224356 Ignore-this: 218fb2aba02c28b4b1e5324bdb5adeaa ] [Fixed tests so that they pass also on buildbots. Alberto Berti **20090222224311 Ignore-this: fcb91cd6acf028382411d23d380a4576 ] [Use failUnlessEqual instead of failUnless(a == b) Alberto Berti **20090222224214 Ignore-this: 8f9144632e3ac9acb4726fb48a083bf4 ] [Better implementation of filtering algorithm. Alberto Berti **20090222224049 Ignore-this: 67a8bd2f99bcc87ca2443bef13370a87 ] [Removed '.hgrags' from vcs excludes Alberto Berti **20090222223946 Ignore-this: 3e94c22fc9d85f380ee11fb8bdb4d1e9 ] [docs: CREDITS to Alberto Berti zooko@zooko.com**20090222193314 Ignore-this: 74d370ada3234cce9e58aec15d739f71 ] [Added tests for the --exclude* options of backup command. Alberto Berti **20090222165106 Ignore-this: f1b931cf2e7929ce47b737c022bca707 ] [Added --exclude, --exclude-from and --exclude-vcs options to backup command. Alberto Berti **20090222170829 Ignore-this: 4912890229cd54a2f61f14f06bc4afcc It is still impossible to specify absolute exclusion path, only relative. I must check with tar or rsync how they allow them to be specified. ] [Raise a more explanatory exception for errors encountered during backup processing. Alberto Berti **20090222170252 Ignore-this: f6b8ffe2a903ba07a2c1c59130dac1e4 ] [Added tests for the fixed alias related command's synopsis Alberto Berti **20090222163732 Ignore-this: 4432b4e88e990ba53a5b3fe0f12db2ac ] [Add missing synopsis and descriptions for alias commands. Alberto Berti **20090221003106 Ignore-this: 8aedd03d36d92d912102c7f29e4ca697 ] [docs: move many specification-like documents into specifications/ warner@lothar.com**20090222054054 Ignore-this: a4110cc478198c0611205aba1ccf54f4 ] [test_web.py: increase test coverage of web.status.plural() warner@lothar.com**20090222000116 Ignore-this: 3138c9d5d2410d8e1121e9b2ed694169 ] [crawler: fix performance problems: only save state once per timeslice (not after every bucket), don't start the crawler until 5 minutes after node startup warner@lothar.com**20090221205649 Ignore-this: e6551569982bd31d19779ff15c2d6f58 ] [test_system: oops, don't assume that all files in storage/ are in a deep storage/shares/prefix/si/shnum path, since now the crawler pickle has a short path warner@lothar.com**20090221061710 Ignore-this: fde76d0e5cae853014d1bb18b5f17dae ] [crawler: tolerate low-resolution system clocks (i.e. windows) warner@lothar.com**20090221061533 Ignore-this: 57286a3abcaf44f6d1a78c3c1ad547a5 ] [BucketCountingCrawler: store just the count, not cycle+count, since it's too easy to make usage mistakes otherwise warner@lothar.com**20090221035831 Ignore-this: 573b6f651af74380cdd64059fbbdda4b ] [test_storage: startService the server, as is now the standard practice warner@lothar.com**20090221035755 Ignore-this: 3999889bd628fe4039bbcf1b29160453 ] [crawler: load state from the pickle in init, rather than waiting until startService, so get_state() can be called early warner@lothar.com**20090221035720 Ignore-this: ecd128a5f4364c0daf4b72d791340b66 ] [BucketCountingCrawler: rename status and state keys to use 'bucket' instead of 'share', because the former is more accurate warner@lothar.com**20090221034606 Ignore-this: cf819f63fac9506c878d6c9715ce35b7 ] [storage: also report space-free-for-root and space-free-for-nonroot, since that helps users understand the space-left-for-tahoe number better warner@lothar.com**20090221032856 Ignore-this: 9fdf0475f758acd98b73026677170b45 ] [storage: add bucket-counting share crawler, add its output (number of files+directories maintained by a storage server) and status to the webapi /storage page warner@lothar.com**20090221030408 Ignore-this: 28761c5e076648026bc5f518506db65c ] [storage: move si_b2a/si_a2b/storage_index_to_dir out of server.py and into common.py warner@lothar.com**20090221030309 Ignore-this: 645056428ab797f0b542831c82bf192a ] [crawler: add get_progress, clean up get_state warner@lothar.com**20090221002743 Ignore-this: 9bea69f154c75b31a53425a8ea67789b ] [web/storage: make sure we can handle platforms without os.statvfs too warner@lothar.com**20090220220353 Ignore-this: 79d4cb8482a8543b9759dc949c86c587 ] [crawler: provide for one-shot crawlers, which stop after their first full cycle, for share-upgraders and database-populaters warner@lothar.com**20090220211911 Ignore-this: fcdf72c5ffcafa374d376388be6fa5c5 ] [web: add Storage status page, improve tests warner@lothar.com**20090220202926 Ignore-this: e34d5270dcf0237fe72f573f717c7a4 ] [storage: include reserved_space in stats warner@lothar.com**20090220202920 Ignore-this: b5b480fe0abad0148ecad0c1fb47ecae ] [web/check_results: sort share identifiers in the sharemap display warner@lothar.com**20090220182922 Ignore-this: 5c7bfcee3e15c7082c3653eb8a460960 ] [webapi: pass client through constructor arguments, remove IClient, should make it easier to test web renderers in isolation warner@lothar.com**20090220181554 Ignore-this: e7848cd1bee8faf2ce7aaf040b9bf8e3 ] [test/no_network: do startService on the storage servers, make it easier to customize the storage servers warner@lothar.com**20090220022254 Ignore-this: e62f328721c007e4c5ee023a6efdf66d ] [crawler: modify API to support upcoming bucket-counting crawler warner@lothar.com**20090220013142 Ignore-this: 808f8382837b13082f8b245db2ebee06 ] [test_backupdb: make the not-a-database file larger, since the older sqlite-2.3.2 on OS-X is easily fooled warner@lothar.com**20090220000409 Ignore-this: 694d2ca5053bb96e91670765d0cedf2e ] [web/reliability: add parameter descriptions, adapted from a patch from Terrell Russell. warner@lothar.com**20090219222918 Ignore-this: 835f5ab01e1aff31b2ff9febb9a51f3 ] [test_crawler: hush pyflakes warner@lothar.com**20090219202340 Ignore-this: 765d22c9c9682cc86c5205dc130500af ] [test_crawler: disable the percentage-of-cpu-used test, since it is too unreliable on our slow buildslaves. But leave the code in place for developers to run by hand. warner@lothar.com**20090219201654 Ignore-this: ff7cf5cfa79c6f2ef0cf959495dd989a ] [reliability.py: fix the numpy conversion, it was completely broken. Thanks to Terrell Russell for the help. warner@lothar.com**20090219195515 Ignore-this: f2b1eb65855111b338e1487feee1bbcf ] [reliability: switch to NumPy, since Numeric is deprecated warner@lothar.com**20090219074435 Ignore-this: f588a68e9bcd3b0bc3653570882b6fd5 ] [setup.py: fix pyflakes complaints warner@lothar.com**20090219073643 Ignore-this: a314e5456b0a796bc9f70232a119ec68 ] [move show-tool-versions out of setup.py and into a separate script in misc/ , since setuptools is trying to build and install a bunch of stuff first warner@lothar.com**20090219073558 Ignore-this: 9e56bc43026379212e6b6671ed6a1fd4 ] [test_crawler: don't require >=1 cycle on cygwin warner@lothar.com**20090219065818 Ignore-this: b8d2d40f26aeb30a7622479840a04635 ] [setup.py: add show_tool_versions command, for the benefit of a new buildbot step warner@lothar.com**20090219062436 Ignore-this: 21d761c76a033e481831584bedc60c86 ] [setup.py: wrap to 80 cols, no functional changes warner@lothar.com**20090219055751 Ignore-this: d29e57c6ee555f2ee435667b7e13e60b ] [crawler: use fileutil.move_info_place in preference to our own version warner@lothar.com**20090219051342 Ignore-this: ee4e46f3de965610503ba36b28184db9 ] [fileutil: add move_into_place(), to perform the standard unix trick of atomically replacing a file, with a fallback for windows warner@lothar.com**20090219051310 Ignore-this: c1d35e8ca88fcb223ea194513611c511 ] [crawler: fix problems on windows and our slow cygwin slave warner@lothar.com**20090219042431 Ignore-this: 8019cb0da79ba00c536183a6f57b4cab ] [#633: first version of a rate-limited interruptable share-crawler warner@lothar.com**20090219034633 Ignore-this: 5d2d30c743e3b096a8e775d5a9b33601 ] [change StorageServer to take nodeid in the constructor, instead of assigning it later, since it's cleaner and because the original problem (Tubs not being ready until later) went away warner@lothar.com**20090218222301 Ignore-this: 740d582f20c93bebf60e21d9a446d3d2 ] [test_system: split off checker tests to test_deepcheck.py, this file is too big warner@lothar.com**20090218214234 Ignore-this: 82bf8db81dfbc98224bbf694054a8761 ] [break storage.py into smaller pieces in storage/*.py . No behavioral changes. warner@lothar.com**20090218204655 Ignore-this: 312d408d1cacc5a764d791b53ebf8f91 ] [immutable/layout: minor change to repr name warner@lothar.com**20090218204648 Ignore-this: c8781ef15b7dea63b39236a1899b86ce ] [docs: add lease-tradeoffs diagram warner@lothar.com**20090218204137 Ignore-this: c22a589ad465dac846da834c30dc4083 ] [interfaces.py: allow add/renew/cancel-lease to return Any, so that 1.3.1 clients (the first to use these calls) can tolerate future storage servers which might return something other than None warner@lothar.com**20090218192903 Ignore-this: dcbb704a05416ecc66d90fb486c3d75b ] [docs/debian.txt: minor edit warner@lothar.com**20090218032212 Ignore-this: 64ff1fb163ffca4bcfd920254f1cf866 ] [add --add-lease to 'tahoe check', 'tahoe deep-check', and webapi. warner@lothar.com**20090218013243 Ignore-this: 176b2006cef5041adcb592ee83e084dd ] [change RIStorageServer.remote_add_lease to exit silently in case of no-such-bucket, instead of raising IndexError, because that makes the upcoming --add-lease feature faster and less noisy warner@lothar.com**20090218013053 Ignore-this: 6fdfcea2c832178f1ce72ab0ff510f3a ] [CLI #590: convert 'tahoe deep-check' to streaming form, improve display, add tests warner@lothar.com**20090217231511 Ignore-this: 6d88eb94b1c877eacc8c5ca7d0aac776 ] [interfaces.py: document behavior of add_lease/renew_lease/cancel_lease, before I change it warner@lothar.com**20090217194809 Ignore-this: 703c6712926b8edb19d55d790b65a400 ] [test_backupdb: improve error messages if the test fails warner@lothar.com**20090217170838 Ignore-this: ef657e87c66e4304d3e0aca9831b84c ] [webapi #590: add streaming deep-check. Still need a CLI tool to use it. warner@lothar.com**20090217053553 Ignore-this: a0edd3d2a531c48a64d8397f7e4b208c ] [test_web.Grid: change the CHECK() function to make it easier to test t= values with hyphens in them warner@lothar.com**20090217050034 Ignore-this: 410c08735347c2057df52f6716520228 ] [test_web: improve checker-results coverage with a no-network -based test, enhance no-network harness to assist, fix some bugs in web/check_results.py that were exposed warner@lothar.com**20090217041242 Ignore-this: fe54bb66a9ae073c002a7af51cd1e18 ] [web: fix handling of reliability page when Numeric is not available warner@lothar.com**20090217015658 Ignore-this: 9d329182f1b2e5f812e5e7eb5f4cf2ed ] [test/no_network: update comments with setup timing: no_network takes 50ms, SystemTestMixin takes 2s (on my laptop) warner@lothar.com**20090217000643 Ignore-this: cc778fa3219775b25057bfc9491f8f34 ] [test_upload: rewrite in terms of no-network GridTestMixin, improve no_network.py as necessary warner@lothar.com**20090216234457 Ignore-this: 80a341d5aa3036d24de98e267499d70d ] [test_download: rewrite in terms of no-network GridTestMixin, improve no_network.py as necessary warner@lothar.com**20090216233658 Ignore-this: ec2febafd2403830519120fb3f3ca04e ] [test_dirnode.py: convert Deleter to new no-network gridtest warner@lothar.com**20090216232348 Ignore-this: 8041739442ec4db726675e48f9775ae9 ] [test_cli.py: modify to use the new 'no-network' gridtest instead of SystemTestMixin, which speeds it up from 73s to 43s on my system warner@lothar.com**20090216232005 Ignore-this: ec6d010c9182aa72049d1fb894cf890e ] [tests: fix no_network framework to work with upload/download and checker warner@lothar.com**20090216231947 Ignore-this: 74b4dbd66b8384ae7c7544969fe4f744 ] [client.py: improve docstring warner@lothar.com**20090216231532 Ignore-this: bbaa9e3f63fdb0048e3125c4681b2d1f ] [test_cli: add test coverage for help strings warner@lothar.com**20090216210833 Ignore-this: d2020849107f687448e159a19d0e5dab ] [test/no_network: new test harness, like system-test but doesn't use the network so it's faster warner@lothar.com**20090216205844 Ignore-this: 31678f7bdef30b0216fd657fc6145534 ] [interfaces.py: minor docstring edit warner@lothar.com**20090216205816 Ignore-this: cec3855070197f7920b370f95e8b07bd ] [setup: if you sdist_dsc (to produce the input files for dpkg-buildpackage) then run darcsver first zooko@zooko.com**20090216201558 Ignore-this: b85be51b3d4a9a19a3366e690f1063e2 ] [doc: a few edits to docs made after the 1.3.0 release zooko@zooko.com**20090216201539 Ignore-this: dbff3b929d88134d862f1dffd1ef068a ] [test_cli: improve test coverage slightly warner@lothar.com**20090216030451 Ignore-this: e01ccc6a6fb44aaa4fb14fe8669e2065 ] [test_util: get almost full test coverage of dictutil, starting with the original pyutil tests as a base. The remaining three uncovered lines involve funny cases of ValueOrderedDict that I can't figure out how to get at warner@lothar.com**20090216023210 Ignore-this: dc1f0c6d8c003c0ade38bc8f8516b04d ] [provisioning/reliability: add tests, hush pyflakes, remove dead code, fix web links warner@lothar.com**20090215222451 Ignore-this: 7854df3e0130d9388f06efd4c797262f ] [util/statistics: add tests, fix mean_repair_cost warner@lothar.com**20090215222326 Ignore-this: c576eabc74c23b170702018fc3c122d9 ] [test_repairer: hush pyflakes warner@lothar.com**20090215222310 Ignore-this: 875eb52e86077cda77efd02da77f8cfa ] [lossmodel.lyx: move draft paper into docs/proposed/, since it's unfinished warner@lothar.com**20090215221905 Ignore-this: 7f7ee204e47fd66932759c94deefe68 ] [build a 'reliability' web page, with a simulation of file decay and repair over time warner@lothar.com**20090213234234 Ignore-this: 9e9623eaac7b0637bbd0071f082bd345 ] [More lossmodel work, on repair. Shawn Willden **20090116025648] [Loss model work (temp1) Shawn Willden **20090115030058] [Statistics module Shawn Willden **20090114021235 Added a statistics module for calculating various facets of share survival statistics. ] [docs: relnotes-short.txt zooko@zooko.com**20090215163510 Ignore-this: 683649bb13499bbe0e5cea2e1716ff59 linkedin.com imposed a strict limit on the number of characters I could post. This forced me to prune and prune and edit and edit until relnotes.txt was a quarter of its former size. Here's the short version. ] [TAG allmydata-tahoe-1.3.0 zooko@zooko.com**20090214000556 Ignore-this: aa6c9a31a14a58ad2298cb7b08d3ea70 ] Patch bundle hash: 6422bf98b0a47e7b946934be2164e89ae0c9f6b2