1 patch for repository http://tahoe-lafs.org/source/tahoe-lafs/trunk: Sat Oct 30 22:30:47 CEST 2010 francois@ctrlaltdel.ch * tahoe backup: perform tilde expansion in exclude-from filename New patches: [tahoe backup: perform tilde expansion in exclude-from filename francois@ctrlaltdel.ch**20101030203047 Ignore-this: 1020962754010210575ad54a212b544a ] { hunk ./src/allmydata/scripts/cli.py 4 import os.path, re, sys, fnmatch from twisted.python import usage from allmydata.scripts.common import BaseOptions, get_aliases, get_default_nodedir, DEFAULT_ALIAS -from allmydata.util.encodingutil import argv_to_unicode, argv_to_abspath, quote_output +from allmydata.util.encodingutil import argv_to_unicode, argv_to_abspath, \ + quote_output, abspath_expanduser_unicode NODEURL_RE=re.compile("http(s?)://([^:]*)(:([1-9][0-9]*))?") hunk ./src/allmydata/scripts/cli.py 311 def opt_exclude_from(self, filepath): """Ignore file matching glob patterns listed in file, one per line. The file is assumed to be in the argv encoding.""" + self.exclude_from = argv_to_unicode(filepath) try: hunk ./src/allmydata/scripts/cli.py 313 - exclude_file = file(filepath) + exclude_file = file(abspath_expanduser_unicode(self.exclude_from)) except: raise BackupConfigurationError('Error opening exclude file %r.' % filepath) try: hunk ./src/allmydata/test/test_cli.py 35 from allmydata.util.assertutil import precondition from allmydata.util.encodingutil import listdir_unicode, unicode_platform, \ quote_output, get_output_encoding, get_argv_encoding, get_filesystem_encoding, \ - unicode_to_output, unicode_to_argv, to_str + unicode_to_output, unicode_to_argv, to_str, argv_to_unicode from allmydata.util.fileutil import abspath_expanduser_unicode timeout = 480 # deep_check takes 360s on Zandr's linksys box, others take > 240s hunk ./src/allmydata/test/test_cli.py 1870 self._check_filtering(filtered, root_listdir, (u'lib.a', u'_darcs', u'subdir'), (nice_doc,)) + def test_exclude_from_tilde_expansion(self): + # ensure that tilde expansion is performed on exclude-from argument + exclude_file = '~/.tahoe/excludes.dummy' + backup_options = cli.BackupOptions() + + # Exception BackupConfigurationError is catched because the file + # exclude_file probably doesn't exist on the system on which this test + # is run, but it doesn't matter + try: + backup_options.parseOptions(['--exclude-from', exclude_file, '~', 'tahoe:']) + except cli.BackupConfigurationError: + pass + + self.failUnlessEqual(abspath_expanduser_unicode(backup_options.exclude_from), + abspath_expanduser_unicode(argv_to_unicode(exclude_file))) + def test_ignore_symlinks(self): if not hasattr(os, 'symlink'): raise unittest.SkipTest("Symlinks are not supported by Python on this platform.") } Context: [setup: run require_auto_deps() before attempting to import any deps in __init__.py zooko@zooko.com**20101030081035 Ignore-this: ffcaf2450628543e020e9919e455f691 For one thing, this makes missing-dependency failures into DistributionNotFound errors instead of ImportErrors, which might be more useful to the user. For another thing, if someone is using distributions that were installed with --multi-version, then they might be not importable until after require_auto_deps() has been run. (The docs claim that this would be the case, but we don't have an example of this happening at this time.) ] [setup: show-tool-versions: emit module and __version__ information even when module name != distribution (package) name, and add TwistedCore, TwistedWeb, and TwistedConch zooko@zooko.com**20101030070233 Ignore-this: 3df19910090d44502ddeeef5d9c29a7 ] [misc/build_helpers/test-with-fake-pkg.py: look for eggs in the parent of the src directory. refs #1190 david-sarah@jacaranda.org**20101030034303 Ignore-this: 4a3cf286272cdb5d06aac15fb5998b33 ] [scripts/runner.py: fix unused import of allmydata. refs #1190 david-sarah@jacaranda.org**20101030003149 Ignore-this: b2fc67f6192ea7ccf8a5ad010ce74a64 ] [scripts/runner.py: remove pkg_resources.require() calls. These are at best redundant because we have already called _auto_deps.require_auto_deps() (from allmydata.__init__) at that point, and they are causing failure of the test-from-prefixdir step on some buildslaves. refs #1190 david-sarah@jacaranda.org**20101029235328 Ignore-this: e00dee63acc7b76a5755025d75abf524 ] [misc/build_helpers/run_trial.py: look for zetuptoolz egg in the parent directory, not the cwd of run_trial. refs #1190 david-sarah@jacaranda.org**20101029230329 Ignore-this: 1596fb8c290d1c706f079701b1857db8 ] [bundled zetuptoolz: if __main__.__requires__ exists then do not add packages to the working set if they provide an incompatible version of a package. Also put a complete __requires__ listing the transitive closure of dependencies at the beginning of generated scripts, rather than a shallow __requires__ specifying only the application version. refs #1190 david-sarah@jacaranda.org**20101029223111 Ignore-this: a95f1967884340e53bf3adf90db40cfc ] [setup.py, misc/build_helpers/run_trial.py: use undocumented __requires__ variable to cause setuptools/zetuptoolz to put the correct versions of dependencies on sys.path. Also ensure that run_trial adds the bundled zetuptoolz egg at the start of sys.path if present. Make the source directory comparison work correctly for the test-with-fake-pkg build step. refs #1190 david-sarah@jacaranda.org**20101029222825 Ignore-this: 8b09366eb6ce3d55c7db5239077a0fac ] [test_runner.py: fix error in BinTahoe.test_version_no_noise introduced by last patch. refs #1235 david-sarah@jacaranda.org**20101029221123 Ignore-this: 4bf21ea34768e8e6adf104e56f939fd0 ] [test_runner.py: also allow 'from pkg_resources import load_entry_point' as noise. refs #1235. david-sarah@jacaranda.org**20101029204246 Ignore-this: a47440aa2cdd29ce55ac7c6c7f4bcaf2 ] [test_runner.py: if the only noise is 'UserWarning: Unbuilt egg for setuptools', skip instead of failing the no_noise tests. This version avoids 'any' to be compatible with Python < 2.5. refs #1235. david-sarah@jacaranda.org**20101029191804 Ignore-this: 83ca1543fc9673e664a8eeefe1eba429 ] [NEWS: clarify (strengthen) description of what backdoors.rst declares, and add bugfix entries for 'tahoe cp' and Windows console bugs. refs #1216, #1224, #1232 david-sarah@jacaranda.org**20101028180046 Ignore-this: 1c3eef3cd353b06b6ee00ce87c5ef59a ] [make ResponseCache smarter to avoid memory leaks: don't record timestamps, use DataSpans to merge entries, and clear the cache when we see a new seqnum. refs #1045, #1229 david-sarah@jacaranda.org**20101027043302 Ignore-this: 88fd6fba7f35a2f8af1693b92718f5f3 ] [windows/fixups.py: limit length of string passed in a single call to WriteConsoleW. fixes #1232. david-sarah@jacaranda.org**20101027021636 Ignore-this: fbd99e0d22493974696d37925d97c7d6 ] [scripts/tahoe_backup.py: oops, fix missing import, thanks pyflakes Brian Warner **20101029094223 Ignore-this: 285c35af824935641a5be35c008b080c test_cli.py: hush minor pyflakes complaint ] [mutable/servermap.py: update comment. Closes #1231. Brian Warner **20101029091424 Ignore-this: 80bf854123fc254e097a81b82bdf4990 ] [tahoe_cp.py: Don't call urllib.quote with an Unicode argument, fix #1224 Brian Warner **20101029084520 Ignore-this: 5524722d5e5babbb73ca0969d54967f6 tahoe_backup.py: Fix another (potential) occurrence of calling urllib.quote() with an Unicode parameter ] [fix #1223, crash+inefficiency during repair due to read overrun Brian Warner **20101029082036 Ignore-this: e6aa0295ad254544da3b5cc41b33d862 * repairer (really the uploader) reads beyond end of input file (Uploadable) * new-downloader does not tolerate overreads * uploader does lots of tiny reads (inefficient) This fixes the last two. The uploader still does a single overread at the end of the input file, but now that's ok so we can leave it in place. The uploader now expects the Uploadable to behave like a normal disk file (reading beyond EOF will return less data than was asked for), and now the new-downloadable behaves that way. ] [add misc/build_helpers/test-with-fake-pkg.py. refs #1190 david-sarah@jacaranda.org**20101029025150 Ignore-this: 995f220962708f1bad83092161130f67 ] [startstop_node.py: pyflakes import fix. refs #1190 david-sarah@jacaranda.org**20101028014805 Ignore-this: 369ef5022c8ee5a0d8341af01553bcef ] ['tahoe start': use import+call rather than find+spawn "Brian Warner "**20101027061342 This removes the need to use a locally-built (dependency) bin/twistd, and removes a big chunk of behavior differences between unix and windows. It also happens to resolve the "client node probably started" uncertainty. Might help with #1190, #602, and #71. ] [docs/known_issues.rst: Add section on traffic analysis. Fix URL for current version of file. david-sarah@jacaranda.org**20101024234259 Ignore-this: f3416e79d3bb833f5118da23e85723ad ] [test_mutable.py: add test for ResponseCache memory leak. refs #1045, #1129 david-sarah@jacaranda.org**20101024193409 Ignore-this: 3aee7f0677956cc6deaccb4d5b8e415f ] [test_encodingutil.py: test_argv_to_unicode modified the wrong encoding variable. fixes #1214 david-sarah@jacaranda.org**20101023035810 Ignore-this: e5f1f849931b96939facc53d93ff61c5 ] [docs/running.html: fix missing end-quote, and change frontends/ doc references to .rst. david-sarah@jacaranda.org**20101024171500 Ignore-this: 47c645a6595e1790b1d1adfa71af0e1d ] [docs/running.html: 'tahoe create-client' now creates a node with storage disabled. Also change configuration.txt references to configuration.rst. david-sarah@jacaranda.org**20101024170431 Ignore-this: e5b048055494ba3505bb8a506610681c ] [doc: add explanation of the motivation for the surprising and awkward API to erasure coding zooko@zooko.com**20101015060202 Ignore-this: 428913ff6e1bf5b393deffb1f20b949b ] [setup: catch and log ValueError from locale.getdefaultlocale() in show-tool-versions.py zooko@zooko.com**20101015054440 Ignore-this: 827d91490562c32ed7cf6526dfded773 I got a bug report from Mathias Baert showing that locale.getdefaultlocale() raises an exception on his Mac OS X system. Heh. ] [docs: update how-to-make-a-release doc with a few tweaks from the 1.8.0 process zooko@zooko.com**20101015054413 Ignore-this: ca5e9478531a3393792ae283239549dd ] [docs: update NEWS ref: #1216 zooko@zooko.com**20101015053719 Ignore-this: 2e0b92e4145d667cdf075e64b7965530 ] [docs: fix tab-vs-spaces, make some CLI examples /"literal", wrap some to Brian Warner **20101015060606 Ignore-this: eae08bdf0afb19a2fbf41c31e70a8122 80-cols, remove spurious whitespace. Add rst2html.py rule to Makefile. ] [docs: add Peter Secor, Shawn Willden, and Terrell Russell as signatories to docs/backdoors.rst zooko@zooko.com**20101015053242 Ignore-this: c77adf819d664f673e17c4aaeb353f33 ] [docs: convert all .txt docs to .rst thanks to Ravi Pinjala zooko@zooko.com**20101015052913 Ignore-this: 178a5122423189ecfc45b142314a78ec fixes #1225 ] [docs: add statement on our refusal to insert backdoors zooko@zooko.com**20101006051147 Ignore-this: 644d308319a7b80c4434bdff9760404a ] [setup: add --multi-version to the "setup.py develop" command-line zooko@zooko.com**20101005182350 Ignore-this: 709155cc21caff29826b8d41a8c8d63d fixes #530. I earlier tried this twice (see #530 for history) and then twice rolled it back due to some problems that arose. However, I didn't write down what the problems were in enough detail on the ticket that I can tell today whether those problems are still issues, so here goes the third attempt. (I did write down on the ticket that it would not create site.py or .pth files in the target directory with --multi-version mode, but I didn't explain why *that* was a problem.) ] [setup: use execfile to access _auto_deps.py in its proper location of src/allmydata/ instead of copying it into place when setup.py is executed zooko@zooko.com**20100906055714 Ignore-this: c179b42672d775580afad40121f86812 ] [trivial: M-x whitespace-cleanup zooko@zooko.com**20100903144712 Ignore-this: 1bb764d11ac69b4a35ea091cfb13158a ] [minor: remove unused interface declaration, change allmydata.org to tahoe-lafs.org in email address, fix wording in relnotes.txt zooko@zooko.com**20100930153708 Ignore-this: a452969228afed2774de375e29fa3048 ] [immutable/repairer.py: don't use the default happiness setting when repairing Kevan Carstensen **20100927200102 Ignore-this: bd704d9744b970849da8d46a16b8089a ] [NEWS: note dependency updates to pycryptopp and pycrypto. david-sarah@jacaranda.org**20100924191207 Ignore-this: eeaf5c9c9104f24c450c2ec4482ac1ee ] [TAG allmydata-tahoe-1.8.0 zooko@zooko.com**20100924021631 Ignore-this: 494ca0a885c5e20c883845fc53e7ab5d ] [docs: timestamp the 1.8.0 release zooko@zooko.com**20100924021552 Ignore-this: dbacb97c0f9994532f38a5612cecef65 ] [docs: update quickstart.html to include "In Case Of Trouble" and to mention Python 2.7 zooko@zooko.com**20100924020851 Ignore-this: 58131cf98bd1c33d8c8d0dc225298ed9 ] [docs: mention default values of K, H, and M zooko@zooko.com**20100924020245 Ignore-this: ab825b7415142b4394599f909ea31934 ] [setup: add flappclient to the set of tools reported on by show-tool-versions zooko@zooko.com**20100923072325 Ignore-this: 9a35eb80502712c510b9feea483f6f01 ] [setup: raise minimum required version of pycryptopp, even on non-x86/amd64 platforms, to v0.5.14 of pycryptopp zooko@zooko.com**20100922070808 Ignore-this: 651ec05827a5569508d765968900bad0 ] [setup: allow an older version of pycryptopp if we are not on x86 or amd64 architecture (which are the architectures on which older versions of Crypto++ had bugs in asm implementations of AES and SHA-256) zooko@zooko.com**20100921140344 Ignore-this: cac01ed5d771bdf806a0d34016f6a54e ] [setup: force the bundled version of darcsver to take precedence over any other version of darcsver zooko@zooko.com**20100921073258 Ignore-this: e1bb7bb470e0edbf63a8014f7ec11ec9 ] [setup: upgrade bundled copy of darcsver to v1.6.3 zooko@zooko.com**20100921073046 Ignore-this: 8914b566cf028f9d118905783d6e3364 ] [setup: format the output of show-tool-versions for better human-readability zooko@zooko.com**20100920225415 Ignore-this: 9894d42b37c5c12eb0c1af1c39709ac2 ] [relnotes.txt and docs/quickstart.html for 1.8.0 release david-sarah@jacaranda.org**20100919050335 Ignore-this: 9ef6499236d07309fb4df983f9a0a5cd ] [_auto_deps.py: require pycrypto >= 0.5.20 (fixes bugs in SHA-256 and AES) david-sarah@jacaranda.org**20100919045743 Ignore-this: b6864315e89c1fb967a3bd1f436251d5 ] [NEWS, docs/known_issues.txt for 1.8.0 release david-sarah@jacaranda.org**20100919044412 Ignore-this: d8901578b0c0c20e42daaae23879b091 ] [docs: a few simple updates to links and naming, and also recommend torsocks instead of the old, unmaintained tsocks, for use with Tor zooko@zooko.com**20100919011636 Ignore-this: ce44fafc565506fcac647379f8ecfa5b ] [setup: make show-tool-versions report entire verbose exception stack traces when it fails to find a Python package zooko@zooko.com**20100919011554 Ignore-this: 2bfcc6abd0e71c5b98b16d52b67dd23 ] [setup: add information about setuptools_darcs and darcsver to the show-tool-versions output zooko@zooko.com**20100918220410 Ignore-this: 47f1924651372e3375f65dd4773db673 ] [docs/quickstart.html: note dependency on python development files, e.g. on Ubuntu david-sarah@jacaranda.org**20100914040456 Ignore-this: 8f2ab59d3048de1ea0fbea5850843045 ] [TAG allmydata-tahoe-1.8.0c4 zooko@zooko.com**20100912062225 Ignore-this: 4c99e0eed253e2a38ed9ea4f4db5ad77 ] [docs: update relnotes.txt, NEWS, and quickstart.html for the 1.8.0c4 release zooko@zooko.com**20100912061423 Ignore-this: bb17f4c54ba390fdcc74eb5d5017373 ] [misc: add docstring to bench_spans.py telling how to run it zooko@zooko.com**20100911041442 Ignore-this: 4258211a81b8eac63e073889026176d6 ref #1182 ] [test_download.py: simplify and possibly fix test_lost_servers; refs #1197 david-sarah@jacaranda.org**20100912003251 Ignore-this: 90e88e764e8dcd6c1e5254d493c8f41a ] [test_runner.py: fix spurious failures of test_path on case-insensitive filesystems, and restore the check that the version is not 'unknown'. david-sarah@jacaranda.org**20100911221913 Ignore-this: 615ea1b18798af7eae78f6bd71667df6 ] [docs/frontends/webapi.txt: document that the meaning of the 'healthy' field may change in future to reflect servers-of-happiness; refs #614 david-sarah@jacaranda.org**20100911003147 Ignore-this: 4661d576c145cc2b641481b70e34e357 ] [docs/frontends/webapi.txt: note that 'count-good-share-hosts' is computed incorrectly; refs #1115 david-sarah@jacaranda.org**20100911002548 Ignore-this: 606989661c95a6db109f8fb7bd9a8bf9 ] [docs/frontends/FTP-and-SFTP.txt: docs/performance.txt, architecture.txt: updates taking into account new downloader (revised). refs #798 david-sarah@jacaranda.org**20100910195422 Ignore-this: 5774da17f734231fefe6454a80e81455 ] [docs/frontends/FTP-and-SFTP.txt: warn users about connecting to the FTP and SFTP servers remotely. Fixes #1192 david-sarah@jacaranda.org**20100910193234 Ignore-this: 7e403e9f349dc38f49197eb0835d47c5 ] [test: make tests stop relying on pyutil version class accepting the string 'unknown' for its version, and make them forward-compatible with the future Python Rational Version Numbering standard zooko@zooko.com**20100910154135 Ignore-this: d051b071f33595493be5df218f5015a6 ] [setup: copy in this fix from zetuptoolz and the accompanying new version number of zetuptoolz: http://tahoe-lafs.org/trac/zetuptoolz/ticket/1 zooko@zooko.com**20100910061411 Ignore-this: cb0ddce66b2a71666df3e22375fa581a ] [immutable download: have the finder inform its share consumer "no more shares" in a subsequent tick, thus avoiding accidentally telling it "no more shares" now and then telling it "here's another share" in a subsequent tick Brian Warner **20100910043038 Ignore-this: 47595fb2b87867d3d75695d51344c484 fixes #1191 Patch by Brian. This patch description was actually written by Zooko, but I forged Brian's name on the "author" field so that he would get credit for this patch in revision control history. ] [immutable downloader: add a test specifically of whether the finder sometimes announces "no more shares ever" and then announces a newly found share zooko@zooko.com**20100909041654 Ignore-this: ec0d5febc499f974b167465290770abd (The current code fails this test, ref #1191.) ] [docs/frontends/FTP-and-SFTP.txt : ftpd and sftpd doesn't listen on loopback interface only marc.doudiet@nimag.net**20100813140853 Ignore-this: 5b5dfd0e5991a2669fe41ba13ea21bd4 ] [tests: assign the storage servers to a fixed order which triggers a bug in new downloader every time this test is run (formerly this test would detect the bug in new-downloader only sporadically) zooko@zooko.com**20100904041515 Ignore-this: 33155dcc03e84217ec5541addd3a16fc If you are investigating the bug in new-downloader, one way to investigate might be to change this ordering to a different fixed order (e.g. rotate by 4 instead of rotate by 5) and observe how the behavior of new-downloader differs in that case. ] [TAG allmydata-tahoe-1.8.0c3 zooko@zooko.com**20100902212140 Ignore-this: e4550de37f57e5c1a591e549a104565d ] [docs: update relnotes.txt for v1.8.0c3 zooko@zooko.com**20100902212111 Ignore-this: 7211f79f4c388c9e8ff0d05f22eb3ba2 ] [download status: fix bug from me committing the wrong one of François's #1172 patches, fixes #1172 zooko@zooko.com**20100902161541 Ignore-this: aeaa3befa632dbc7216686bb67a9695f ] [DownloadStatus: show active immutable downloads in Active Operations, Fix #1172 francois@ctrlaltdel.ch**20100902101728 Ignore-this: 47d2b214bbf6e4713890f0ba4d4beecf ] [tests: bump up the allowed number of reads zooko@zooko.com**20100902053801 Ignore-this: 9450a720c9c5f51d63454029673cca16 Kyle's OpenBSD buildslave used 41 reads when doing this test. The fact that I'm blindly bumping this number up to match the observed behavior probably means this isn't a good criterion to be testing for anyway. But perhaps someone else (Brian) could investigate why that run on Kyle's OpenBSD box took four more reads than we expected, and whether the fact that it took 41 reads to do this operation is indicative of an actual problem. ] [SegmentFetcher: use new diversity-seeking share-selection algorithm, and "Brian Warner "**20100901013702 deliver all shares at once instead of feeding them out one-at-a-time. Also fix distribution of real-number-of-segments information: now all CommonShares (not just the ones used for the first segment) get a correctly-sized hashtree. Previously, the late ones might not, which would make them crash and get dropped (causing the download to fail if the initial set were insufficient, perhaps because one of their servers went away). Update tests, add some TODO notes, improve variable names and comments. Improve logging: add logparents, set more appropriate levels. ] [Share: drop received data after each block finishes. Quick fix for the #1170 spans.py complexity bug. "Brian Warner "**20100901013558] [docs: a couple of small edits to CREDITS and how_to_make_a_tahoe-lafs_release.txt zooko@zooko.com**20100829222758 Ignore-this: cfdb414f86dfd581b5f1d4d94231b85c ] [add simulator to explore the trade-offs for hashed-based digital signatures zooko@zooko.com**20100819030630 Ignore-this: 284e6d8b4140d2ecd9a4b14247d0816f ] [_auto_deps.py: change pycrypto version requirement to avoid https://bugs.launchpad.net/pycrypto/+bug/620253 david-sarah@jacaranda.org**20100829230038 Ignore-this: e58f98ef262444067fc4b31fad23e40b ] [docs: update NEWS a bit about New-Downloader zooko@zooko.com**20100819021446 Ignore-this: 31a6e2fb0a6e3d19f73481e99070da7a ] [misc: add benchmarking tool for spans zooko@zooko.com**20100819021420 Ignore-this: 569327a1908a07e5fb634526bed515b2 ] [web: refactor rate computation, fixes #1166 francois@ctrlaltdel.ch**20100815141933 Ignore-this: d25491858e137894142eaa67c75b0439 ] [docs: doc of the download status page zooko@zooko.com**20100814054117 Ignore-this: a82ec33da3c39a7c0d47a7a6b5f81bbb ref: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1169#comment:1 ] [docs: NEWS: edit English usage, remove ticket numbers for regressions vs. 1.7.1 that were fixed again before 1.8.0c2 zooko@zooko.com**20100811071758 Ignore-this: 993f5a1e6a9535f5b7a0bd77b93b66d0 ] [docs: NEWS: more detail about new-downloader zooko@zooko.com**20100811071303 Ignore-this: 9f07da4dce9d794ce165aae287f29a1e ] [TAG allmydata-tahoe-1.8.0c2 david-sarah@jacaranda.org**20100810073847 Ignore-this: c37f732b0e45f9ebfdc2f29c0899aeec ] Patch bundle hash: a131984fa1c6366d292f5b7db7902f7979ae9c9b