1 patch for repository davidsarah@tahoe-lafs.org:/home/source/darcs/tahoe/ticket999-S3-backend: Wed Oct 19 02:57:28 BST 2011 david-sarah@jacaranda.org * S3 backend: restore max_space option (partially implemented). New patches: [S3 backend: restore max_space option (partially implemented). david-sarah@jacaranda.org**20111019015728 Ignore-this: 3c224bd816bb0fde0b4538fc3c77cdca ] { hunk ./docs/backends/S3.rst 46 This controls which Amazon S3 region will be used. The options are ``US`` and ``EU``. The default is ``US``. +``s3.max_space = (quantity of space, optional)`` + + This tells the server to limit how much space can be used in the S3 + bucket. Before each share is uploaded, the server will ask S3 for the + current bucket usage, and will only accept the share if it does not cause + the usage to grow above this limit. + + The string contains a number, with an optional case-insensitive scale + suffix like "K" or "M" or "G", and an optional "B" or "iB" suffix. So + "100MB", "100M", "100000000B", "100000000", and "100000kb" all mean the + same thing. Likewise, "1MiB", "1024KiB", and "1048576B" all mean the same + thing. + + If ``s3.max_space`` is omitted, the default behavior is to allow + unlimited usage. + + +Once configured, the WUI "storage server" page will provide information about +how much space is being used and how many shares are being stored. + hunk ./src/allmydata/storage/backends/s3/s3_backend.py 32 from allmydata.storage.backends.s3.s3_bucket import S3Bucket readonly = config.get_config("storage", "readonly", False, boolean=True) + max_space = config.get_config_size("storage", "max_space", None) corruption_advisory_dir = storedir.child("corruption-advisories") accesskeyid = config.get_config("storage", "s3.access_key_id") hunk ./src/allmydata/storage/backends/s3/s3_backend.py 42 bucketname = config.get_config("storage", "s3.bucket") s3bucket = S3Bucket(accesskeyid, secretkey, region, url, bucketname) - return S3Backend(s3bucket, readonly, corruption_advisory_dir) + return S3Backend(s3bucket, readonly, max_space, corruption_advisory_dir) class S3Backend(Backend): hunk ./src/allmydata/storage/backends/s3/s3_backend.py 48 implements(IStorageBackend) - def __init__(self, s3bucket, readonly=False, corruption_advisory_dir=None): + def __init__(self, s3bucket, readonly=False, max_space=None, corruption_advisory_dir=None): Backend.__init__(self) self._s3bucket = s3bucket self._readonly = readonly hunk ./src/allmydata/storage/backends/s3/s3_backend.py 52 + if max_space is None: + self._max_space = 2**64 + else: + self._max_space = int(max_space) # we don't actually create the corruption-advisory dir until necessary self._corruption_advisory_dir = corruption_advisory_dir hunk ./src/allmydata/storage/backends/s3/s3_backend.py 86 return S3ShareSet(storageindex, self._s3bucket, self._incomingset) def fill_in_space_stats(self, stats): + stats['storage_server.max_space'] = self._max_space + # TODO: query space usage of S3 bucket stats['storage_server.accepting_immutable_shares'] = int(not self._readonly) hunk ./src/allmydata/storage/backends/s3/s3_backend.py 95 if self._readonly: return 0 # TODO: query space usage of S3 bucket - return 2**64 + return self._max_space class S3ShareSet(ShareSet): hunk ./src/allmydata/test/test_client.py 215 "s3.bucket = test\n") self.failUnlessRaises(MissingConfigEntry, client.Client, basedir) + def test_s3_config_max_space_good(self): + basedir = "client.Basic.test_s3_config_max_space_good" + os.mkdir(basedir) + self._write_s3secret(basedir) + fileutil.write(os.path.join(basedir, "tahoe.cfg"), + BASECONFIG + + "[storage]\n" + + "enabled = true\n" + + "backend = s3\n" + + "s3.access_key_id = keyid\n" + + "s3.bucket = test\n" + + "max_space = 1GB\n") + c = client.Client(basedir) + server = c.getServiceNamed("storage") + self.failUnless(isinstance(server.backend, S3Backend), server.backend) + self.failUnlessReallyEqual(server.backend._max_space, 1*1000*1000*1000) + + def test_s3_config_max_space_bad(self): + basedir = "client.Basic.test_s3_config_max_space_bad" + os.mkdir(basedir) + self._write_s3secret(basedir) + fileutil.write(os.path.join(basedir, "tahoe.cfg"), + BASECONFIG + + "[storage]\n" + + "enabled = true\n" + + "backend = s3\n" + + "s3.access_key_id = keyid\n" + + "s3.bucket = test\n" + + "max_space = bogus\n") + self.failUnlessRaises(InvalidValueError, client.Client, basedir) + def _permute(self, sb, key): return [ s.get_serverid() for s in sb.get_servers_for_psi(key) ] } Context: [S3 backend: remove max_space option. refs #999 david-sarah@jacaranda.org**20111018224057 Ignore-this: 6fc56d5ea43dae4ac9604a7cf15e7ce6 ] [test_storage.py, test_crawler.py: change 'bucket' terminology to 'shareset' where appropriate. refs #999 david-sarah@jacaranda.org**20111018183242 Ignore-this: b8cf782836d1558fc99ff13fa08b2b9f ] [Add some __repr__ methods. refs #999 david-sarah@jacaranda.org**20111018064409 Ignore-this: 9b901ee8aaaa9f9895f6a3b4e5f41a21 ] [Fix race conditions in crawler tests. (storage.LeaseCrawler.test_unpredictable_future may still be racy.) refs #999 david-sarah@jacaranda.org**20111018064303 Ignore-this: 58b2791250d14f7e8a6284190d7872e8 ] [Allow crawlers and storage servers to use a deterministic clock, for testing. We do not yet take advantage of this in tests. refs #999 david-sarah@jacaranda.org**20111018063926 Ignore-this: 171bf7275a978a93108b0835d06834ea ] [Change IShareSet.get_shares[_synchronous] to return a pair (list of share objects, set of corrupt shnums). This is necessary to allow crawlers to record but skip over corrupt shares. This patch also changes the behaviour of storage servers to ignore corrupt shares on read, which may or may not be what we want. Note that the S3 backend does not yet report corrupt shares. refs #999 david-sarah@jacaranda.org**20111018063423 Ignore-this: 35abee65334aa3a92471266f5789f452 ] [test_storage.py: cleanup to style of test_limited_history to match other tests. refs #999 david-sarah@jacaranda.org**20111016044311 Ignore-this: 3024764ffb419917eeeb3ecd554fb421 ] [Change accesses of ._sharehomedir on a disk shareset to _get_sharedir(). refs #999 david-sarah@jacaranda.org**20111016044229 Ignore-this: 5b2b9e11f56f0af0588c69ca930f60fd ] [Disk backend: make sure that disk shares with a storageindex of None (as sometimes created by test code) can be printed using __repr__. refs #999 david-sarah@jacaranda.org**20111016035131 Ignore-this: a811b53c20fdde5ad60471c5e4961a24 ] [scripts/debug.py: fix stale code in describe_share that had not been updated for changes in share interfaces. refs #999 david-sarah@jacaranda.org**20111016034913 Ignore-this: 7f2469392b9e6fc64c354ce5a5568a68 ] [test_storage.py: fix a bug in _backdate_leases (it was returning too early). refs #999 david-sarah@jacaranda.org**20111016014038 Ignore-this: 658cf2e2c0dc87032988f1d4db62f267 ] [Undo partial asyncification of crawlers, and enable crawlers only for the disk backend. refs #999 david-sarah@jacaranda.org**20111014061902 Ignore-this: ce7303610878b1b051cf54604796bdde ] [test_storage.py: fix two bugs in test_no_st_blocks -- the _cleanup function was being called too early, and we needed to treat directories as using no space in order for the configured-sharebytes == configured-diskbytes check to be correct. refs #999 david-sarah@jacaranda.org**20111014025840 Ignore-this: a20434e28eda165bed2021f0dafa676c ] [test_storage.py: print more info when checks fail. refs #999 david-sarah@jacaranda.org**20111013234159 Ignore-this: 2989f30c24362ee6a80a7f8f3d5aad9 ] [test_storage.py: remove some redundant coercions to bool. refs #999 david-sarah@jacaranda.org**20111013233520 Ignore-this: 3fa9baaf7e41831a24b8cfa0ef5ec5e4 ] [test_storage: in test_no_st_blocks, print the rec 'dict' if checking one of its fields fails. refs #999 david-sarah@jacaranda.org**20111013232802 Ignore-this: cf18a119d80f11b1ba8681c4285c0198 ] [test_storage: fix some typos introduced when asyncifying test_immutable_leases. refs #999 david-sarah@jacaranda.org**20111013232618 Ignore-this: 28a5e1377d7198191d5771e09826af5b ] [test_storage: rename the two test_leases methods to ServerTest.test_immutable_leases and MutableServer.test_mutable_leases. refs #999 david-sarah@jacaranda.org**20111013232538 Ignore-this: 7a3ccfd237db7a3c5053fe90c3bed1f3 ] [test_storage.py: fix a typo (d vs d2) in test_remove_incoming. refs #999 david-sarah@jacaranda.org**20111013222822 Ignore-this: 71ad69489698865748cd32bc2c8b2fc1 ] [docs/backends/S3.rst: note that storage servers should use different buckets. refs #999 david-sarah@jacaranda.org**20111013050647 Ignore-this: fec8d3bf114bbcf20165a5850aa25aac ] [S3 backend: keep track of incoming shares, so that the storage server can avoid creating BucketWriters for shnums that have an incoming share. refs #999 david-sarah@jacaranda.org**20111013035040 Ignore-this: f1c33357553d68748f970c0c9e19d538 ] [test_storage.py: test_read_old_share and test_write_and_read_share should only expect to be able to read input share data. refs #999 david-sarah@jacaranda.org**20111013032825 Ignore-this: bec4a26f13f105cc84261f2e1b028302 ] [misc/check-interfaces.py: print a warning if a .pyc or .pyo file exists without a corresponding .py file. david-sarah@jacaranda.org**20111012233609 Ignore-this: 35f04939360c6d3b1e8e0c2e9e712d80 ] [storage/backends/base.py: allow readv to work for both mutable and immutable shares. refs #999 david-sarah@jacaranda.org**20111012232802 Ignore-this: a266704981739e7c1217f352aee153fe ] [S3 backend: correct list_objects to list_all_objects in IS3Bucket. refs #999 david-sarah@jacaranda.org**20111012232713 Ignore-this: 7f9dc7946e9866f71e16f3a595f0218e ] [Null backend: make NullShareSet inherit from ShareSet, which should implement readv correctly. Remove its implementation of testv_and_readv_and_writev since the one from ShareSet should work (if it doesn't that would be a separate bug). refs #999 david-sarah@jacaranda.org**20111012232600 Ignore-this: 404757cc6f7e29c2b927258af31d55ce ] [Remove test_backends.py, since all its tests are now redundant with tests in test_storage.py or test_client.py. refs #999 david-sarah@jacaranda.org**20111012232316 Ignore-this: f601a8165058773075ce80d96586b0d9 ] [test_storage.py: add test_write_and_read_share and test_read_old_share originally from test_backends.py. refs #999 david-sarah@jacaranda.org**20111012232124 Ignore-this: 805cd42094d3948ffdf957f44e0d146d ] [test_download.py: fix and reenable Corruption.test_each_byte. Add a comment noting that catalog_detection = True has bitrotted. refs #999 david-sarah@jacaranda.org**20111012041219 Ignore-this: b9fa9ce7406811cd5a9d4a49666b1ab0 ] [no_network.py: fix delete_all_shares. refs #999 david-sarah@jacaranda.org**20111012033458 Ignore-this: bfe9225562454f153a921277b43ac848 ] [S3 backend: fix corruption advisories and listing of shares for mock S3 bucket. refs #999 david-sarah@jacaranda.org**20111012033443 Ignore-this: 9d655501062888be6ee391e426c90a13 ] [test_storage.py: asyncify some more tests, and fix create methods. refs #999 david-sarah@jacaranda.org**20111012025739 Ignore-this: 1574d8175917665f44d278d13f815bb9 ] [test_storage.py: add a test that we can create a share, exercising the backend's get_share and get_shares methods. This may explicate particular kinds of backend failure better than the existing tests. refs #999 david-sarah@jacaranda.org**20111012025514 Ignore-this: f52983e4f3d96ea26ef25856d4cc92ce ] [test_storage.py: Move test_seek to its own class, since it is independent of the backend. Also move test_reserved_space to ServerWithDiskBackend, since reserved_space is specific to that backend. refs #999 david-sarah@jacaranda.org**20111012025149 Ignore-this: 281de8befe51e24cd638886fb5063cd2 ] [util/deferredutil.py: remove unneeded utility functions. refs #999 david-sarah@jacaranda.org**20111012024440 Ignore-this: 17380afd9079442785d0cb78876c7fd5 ] [Move configuration of each backend into the backend itself. refs #999 david-sarah@jacaranda.org**20111012014004 Ignore-this: c337c43e4c4a05617de62f4acf7119d0 ] [test_storage.py: fix test failures in MDMFProxies. refs #999 david-sarah@jacaranda.org**20111012000848 Ignore-this: 798f2a4e960ee444e401a10748afeb08 ] [test_storage.py: cosmetics. refs #999 david-sarah@jacaranda.org**20111012000442 Ignore-this: d3514fa8d69f38d3b45204e2224152d5 ] [storage/backends/disk/disk_backend.py: trivial fix to a comment. #refs 999 david-sarah@jacaranda.org**20111011165704 Ignore-this: b9031b01ef643cb973a41af277d941c0 ] [check-miscaptures.py: report the number of files that were not analysed due to syntax errors (and don't count them in the number of suspicious captures). refs #1555 david-sarah@jacaranda.org**20111009050301 Ignore-this: 62ee03f4b8a96c292e75c097ad87d52e ] [check-miscaptures.py: handle corner cases around default arguments correctly. Also make a minor optimization when there are no assigned variables to consider. refs #1555 david-sarah@jacaranda.org**20111009045023 Ignore-this: f49ece515620081da1d745ae6da19d21 ] [check-miscaptures.py: Python doesn't really have declarations; report the topmost assignment. refs #1555 david-sarah@jacaranda.org**20111009044800 Ignore-this: 4905c9dfe7726f433333e216a6760a4b ] [check-miscaptures.py: handle destructuring function arguments correctly. refs #1555 david-sarah@jacaranda.org**20111009044710 Ignore-this: f9de7d95e94446507a206c88d3f98a23 ] [check-miscaptures.py: check while loops and list comprehensions as well as for loops. Also fix a pyflakes warning. refs #1555 david-sarah@jacaranda.org**20111009044022 Ignore-this: 6526e4e315ca6461b1fbc2da5568e444 ] [Fix pyflakes warnings in misc/ directories other than misc/build_helpers. refs #1557 david-sarah@jacaranda.org**20111007033031 Ignore-this: 7daf5862469732d8cabc355266622b74 ] [Makefile: include misc/ directories other than misc/build_helpers in SOURCES. refs #1557 david-sarah@jacaranda.org**20111007032958 Ignore-this: 31376ec01401df7972e83341dc65aa05 ] [util/happinessutil.py: suppress a warning from check-miscaptures. (It is not a bug because the capturing function is only used by a 'map' in the same iteration.) refs #1556 david-sarah@jacaranda.org**20111009052106 Ignore-this: 16a62844bae083800d6b6a7334abc9bc ] [misc/coding_tools/make-canary-files.py: fix a suspicious capture reported by check-miscaptures (although it happens not to be a bug because the callback will be processed synchronously). refs #1556 david-sarah@jacaranda.org**20111009050531 Ignore-this: 2d1a696955a4c1f7d9c649d4ecefd7de ] [Fix two pyflakes warnings about unused imports. refs #999 david-sarah@jacaranda.org**20111011051745 Ignore-this: 23c17f8eb36a30f4e3b662a778bc4bb7 ] [test_storage.py: fix asyncification of three tests in MDMFProxies. refs #999 david-sarah@jacaranda.org**20111011051319 Ignore-this: a746cc2ed1f4fbcf95bad7624a0544e9 ] [test_storage.py: fix a trivial bug in MDMFProxies.test_write. refs #999 david-sarah@jacaranda.org**20111011045645 Ignore-this: 943c6da82eca7b2d247cfb7d75afc9b7 ] [test_storage.py: fix a typo in test_null_backend. refs #999 david-sarah@jacaranda.org**20111011045133 Ignore-this: ddf00d1d65182d520904168827c792c4 ] [test_storage.py: fix a bug introduced by asyncification of test_allocate. refs #999 david-sarah@jacaranda.org**20111011044131 Ignore-this: 1460b8a713081c8bbe4d298ab39f264f ] [test_storage.py: make MutableServer.test_leases pass. refs #999 david-sarah@jacaranda.org**20111011002917 Ignore-this: dce275f6508a7cfe31c0af82483eea97 ] [test/common.py: in shouldFail and shouldHTTPError, when the raised exception does not include the expected substring (or, for shouldHTTPError, when the status code is wrong), mention which test that happened in. david-sarah@jacaranda.org**20111011002227 Ignore-this: 836cabe9ef774617122905b214a0b8e8 ] [test/mock_s3.py: fix a bug that was causing us to use the wrong directory for share files. refs #999 david-sarah@jacaranda.org**20111010231344 Ignore-this: bc63757f5dd8d31643bd9919f2ecd98c ] [Add fileutil.fp_list(fp) which is like fp.children(), but returns [] in case of a directory that does not exist. Use it to simplify the disk backend and mock S3 bucket implementations. refs #999 david-sarah@jacaranda.org**20111010231146 Ignore-this: fd4fa8b1446fc7e5c03631b4092c20cc ] [S3 backend: move the implementation of list_objects from s3_bucket.py to s3_common.py, making s3_bucket.py simpler and list_objects easier to test independently. refs #999 david-sarah@jacaranda.org**20111010230751 Ignore-this: 2f9a8f75671e87d2caba2ac6c6d4bdfd ] [Make unlink() on share objects consistently idempotent. refs #999 david-sarah@jacaranda.org**20111010204417 Ignore-this: 1dc559fdd89b7135cec64ffca62fc96a ] [Null backend: implement unlink and readv more correctly. refs #999 david-sarah@jacaranda.org**20111010204404 Ignore-this: 3386bc2a1cd0ff6268def31c5c5ce3a1 ] [test_download.py: fix test_download_failover (it should tolerate non-existing shares in _clobber_most_shares). refs #999 david-sarah@jacaranda.org**20111010204142 Ignore-this: db648ffcbd5e37cf236f49ecc1e720fc ] [interfaces.py: resolve another conflict with trunk. refs #999 david-sarah@jacaranda.org**20111010200903 Ignore-this: 163067eab9a5c71e12c6cac058a03832 ] [interfaces.py: fix a typo in the name of IMutableSlotWriter.put_encprivkey. refs #393 david-sarah@jacaranda.org**20111010194642 Ignore-this: eb65439e8dd891c169b43b1679c29238 ] [interfaces.py: resolve conflicts with trunk. refs #999 david-sarah@jacaranda.org**20111010195634 Ignore-this: 8e02c7933392491ba3deb678c5bc5876 ] [interfaces.py: remove get_extension_params and set_extension_params methods from IMutableFileURI. refs #393, #1526 david-sarah@jacaranda.org**20111010194842 Ignore-this: 6012be6fcc12f560aeeeac0be2d337d1 ] [Instrument some assertions to report the failed values. refs #999 david-sarah@jacaranda.org**20111010191733 Ignore-this: 4e886faa5909bf703af8228194ae759c ] [test_storage.py: move some tests that were not applicable to all backends out of ServerTest. refs #999 david-sarah@jacaranda.org**20111010181214 Ignore-this: d2310591c71c4d2d2c5ff4e316f15542 ] [storage/backends/disk/mutable.py: put back a correct assertion that had been disabled. storage/base.py: fix the bug that was causing that assertion to fail. refs #999 david-sarah@jacaranda.org**20111009232142 Ignore-this: dbb644f596bf3c42575b9d9fadc2c9d9 ] [test_storage.py: fix a trivial bug in LeaseCrawler.test_unpredictable_future. refs #999 david-sarah@jacaranda.org**20111007195753 Ignore-this: 357539b4cf8b7455c1787ac591b0ee23 ] [Asyncification, and resolution of conflicts. #999 david-sarah@jacaranda.org**20111007193418 Ignore-this: 29b15345aecd3adeef2c2392ca90d4ff ] [disk backend: size methods should no longer return Deferreds. refs #999 david-sarah@jacaranda.org**20111007193327 Ignore-this: 8d32ffdbb81d88a30352f344e385feff ] [Ensure that helper classes are not treated as test cases. Also fix a missing mixin. refs #999 david-sarah@jacaranda.org**20111007081439 Ignore-this: a222110248f378d91c232799bcd5d3a6 ] [More miscapture fixes. refs #999 david-sarah@jacaranda.org**20111007080916 Ignore-this: 85004d4e3a609a2ef70a38164897ff02 ] [Partially asyncify crawlers. refs #999 david-sarah@jacaranda.org**20111007080657 Ignore-this: f6a15b81592bfff33ccf09301dbdfca1 ] [unlink() on share objects should be idempotent. refs #999 david-sarah@jacaranda.org**20111007080615 Ignore-this: ff87d0b30fc81dd8e90bc5c6852955eb ] [Make sure that get_size etc. work correctly on an ImmutableS3ShareForWriting after it has been closed. Also simplify by removing the _end_offset attribute. refs #999 david-sarah@jacaranda.org**20111007080342 Ignore-this: 9d8ce463daee2ef1b7dc33aca70a0379 ] [Remove an inapplicable comment. refs #999 david-sarah@jacaranda.org**20111007080128 Ignore-this: 1c7fae3ffc9ac412ad9ab6e411ef9be7 ] [Remove unused load method and _loaded attribute from s3/mutable.py. refs #999 david-sarah@jacaranda.org**20111007080051 Ignore-this: b92ed543f7dcf56df0510de8515230e1 ] [Fix a duplicate umid. refs #999 david-sarah@jacaranda.org**20111007080001 Ignore-this: bb9519d15220ab7350731abf19038c2e ] [Fix some miscapture bugs. refs #999 david-sarah@jacaranda.org**20111007075915 Ignore-this: 88dc8ec49e93e5d62c2abb8f496706cb ] [Add a _get_sharedir() method on IShareSet, implemented by the disk and mock S3 backends, for use by tests. refs #999 david-sarah@jacaranda.org**20111007075645 Ignore-this: 9e9ce0d244785da8ac4a3c0aa948ddce ] [Add misc/coding_tools/check-miscaptures.py to detect incorrect captures of variables declared in a for loop, and a 'make check-miscaptures' Makefile target to run it. (It is also run by 'make code-checks'.) This is a rewritten version that reports much fewer false positives, by determining captured variables more accurately. fixes #1555 david-sarah@jacaranda.org**20111007074121 Ignore-this: 51318e9678d132c374ea557ab955e79e ] [Add a get_share method to IShareSet, to get a specific share. refs #999 david-sarah@jacaranda.org**20111007075426 Ignore-this: 493ddfe83414208f08a22b9f327d6b69 ] [Fix some more potential bugs in test code exposed by check-miscaptures.py. refs #1556 david-sarah@jacaranda.org**20111007033847 Ignore-this: aec8a543e9b5c3563b60692c647439a8 ] [Fix some potential bugs (in non-test code) exposed by check-miscaptures.py. refs #1556 david-sarah@jacaranda.org**20111007032444 Ignore-this: bac9ed65b21c2136c4db2482b3c093f7 ] [Fix some potential bugs in test code exposed by check-miscaptures.py. refs #1556 david-sarah@jacaranda.org**20111007023443 Ignore-this: e48b2c2d200521d6f28c737994ce3a2a ] [misc/simulators/hashbasedsig.py: simplify by removing unnecessary local function that captured a variable declared in a for loop (this was not a bug, but the code was unclear). Also fix a pyflakes warning about an import. refs #1556 david-sarah@jacaranda.org**20111007023001 Ignore-this: 446c94efae02ded5e85eb3335ca5e69 ] [immutable/literal.py: add pauseProducing method to LiteralProducer. refs #1537 david-sarah@jacaranda.org**20111003195239 Ignore-this: 385ee3379a2819381937357f1eac457 ] [docs: fix the rst formatting of COPYING.TGPPL.rst zooko@zooko.com**20111003043333 Ignore-this: c5fbc83f4a3db81a0c95b27053c463c5 Now it renders correctly both on trac and with rst2html --verbose from docutils v0.8.1. ] [MDMF: remove extension fields from caps, tolerate arbitrary ones. Fixes #1526 Brian Warner **20111001233553 Ignore-this: 335e1690aef1146a2c0b8d8c18c1cb21 The filecaps used to be produced with hints for 'k' and segsize, but they weren't actually used, and doing so had the potential to limit how we change those filecaps in the future. Also the parsing code had some problems dealing with other numbers of extensions. Removing the existing fields and making the parser tolerate (and ignore) extra ones makes MDMF more future-proof. ] [More asyncification of tests. Also fix some bugs due to capture of slots in for loops. refs #999 david-sarah@jacaranda.org**20111004010813 Ignore-this: 15bf68748ab737d1edc24552ce192f8b ] [s3/s3_common.py: remove incorrect 'self' arguments from interface methods in IS3Bucket. refs #999 david-sarah@jacaranda.org**20111004010745 Ignore-this: d5f66be90062292164d3f017aef3d6f4 ] [no_network.py: Clean up whitespace around code changed by previous patch. david-sarah@jacaranda.org**20111004010407 Ignore-this: 647ec8a9346dca1a41212ab250619b72 ] [no_network.py: Fix potential bugs in some tests due to capture of slots in for loops. david-sarah@jacaranda.org**20111004010231 Ignore-this: 9c496877613a3befd54979e5de6e63d2 ] [Add a share._get_filepath() method used by tests to get the FilePath for a share, rather than accessing the _home attribute. refs #999 david-sarah@jacaranda.org**20111004004604 Ignore-this: ec873e356b7ebd74f52336dd92dea8aa ] [s3/immutable.py: minor simplification in ImmutableS3ShareForReading. refs #999 david-sarah@jacaranda.org**20110930212714 Ignore-this: d71e2466231f695891a6b8d1df945687 ] [free up the buffer used to hold data while it is being written to ImmutableS3ShareForWriting zooko@zooko.com**20110930060238 Ignore-this: 603b2c8bb1f4656bdde5876ac95aa5c9 ] [FIX THE BUG! zooko@zooko.com**20110930032140 Ignore-this: fd32c4ac3054ae6fc2b9433f113b2fd6 ] [fix another bug in ImmutableShareS3ForWriting zooko@zooko.com**20110930025701 Ignore-this: 6ad7bd17111b12d96991172fbe04d76 ] [really fix the bug in ImmutableS3ShareForWriting zooko@zooko.com**20110930023501 Ignore-this: 36a7804433cab667566d119af7223425 ] [Add dummy lease methods to immutable S3 share objects. refs #999 david-sarah@jacaranda.org**20110930021703 Ignore-this: 7c21f140020edd64027c71be0f32c2b2 ] [fix bug in ImmutableS3ShareForWriting zooko@zooko.com**20110930020535 Ignore-this: f7f63d2fc2086903a195cc000f306b88 ] [test_storage.py: Server class uses ShouldFailMixin. refs #999 david-sarah@jacaranda.org**20110930001349 Ignore-this: 4cf1ef21bbf85d7fe52ab660f59ff237 ] [mock_s3.py: fix bug in MockS3Error constructor. refs #999 david-sarah@jacaranda.org**20110930001326 Ignore-this: 4d0ebd9120fc8e99b15924c671cd0927 ] [return res zooko@zooko.com**20110930000446 Ignore-this: 6f73b3e389612c73c6590007229ad8e ] [fix doc to say that secret access key goes into private/s3secret zooko@zooko.com**20110930000256 Ignore-this: c054ff78041a05b3177b3c1b3e9d4ae7 ] [s3_bucket.py: fix an incorrect argument signature for list_objects. refs #999 david-sarah@jacaranda.org**20110929235646 Ignore-this: f02e3a23f28fadef71c70fd0b1592ba6 ] [Make sure that the statedir is created before trying to use it. refs #999 david-sarah@jacaranda.org**20110929234845 Ignore-this: b5f0529b1f2a5b5250c2ee2091cbe24b ] [test/mock_s3.py: fix a typo. refs #999 david-sarah@jacaranda.org**20110929234808 Ignore-this: ccdff591f9b301f7f486454a4366c2b3 ] [test_storage.py: only run test_large_share on the disk backend. (It will wedge your machine if run on the S3 backend with MockS3Bucket.) refs #999 david-sarah@jacaranda.org**20110929234725 Ignore-this: ffa7c08458ee0159455b6f1cd1c3ff48 ] [Fixes to S3 config parsing, with tests. refs #999 david-sarah@jacaranda.org**20110929225014 Ignore-this: 19aa5a3e9575b0c2f77b19fe1bcbafcb ] [Add missing src/allmydata/test/mock_s3.py (mock implementation of an S3 bucket). refs #999 david-sarah@jacaranda.org**20110929212229 Ignore-this: a1433555d4bb0b8b36fb80feb122187b ] [Make the s3.region option case-insensitive (txaws expects uppercase). refs #999 david-sarah@jacaranda.org**20110929211606 Ignore-this: def83d3fa368c315573e5f1bad5ee7f9 ] [Fix missing add_lease method on ImmutableS3ShareForWriting. refs #999 david-sarah@jacaranda.org**20110929211524 Ignore-this: 832f0d94f912b17006b0dbaab94846b6 ] [Add missing src/allmydata/storage/backends/s3/s3_bucket.py. refs #999 david-sarah@jacaranda.org**20110929211416 Ignore-this: aa783c5d7c32af172b5c5a3d62c3faf2 ] [scripts/debug.py: repair stale code, and use the get_disk_share function defined by disk_backend instead of duplicating it. refs #999 david-sarah@jacaranda.org**20110929211252 Ignore-this: 5dda548e8703e35f0c103467346627ef ] [Fix a bug in the new config parsing code when reserved_space is not present for a disk backend. refs #999 david-sarah@jacaranda.org**20110929211106 Ignore-this: b05bd3c4ff7d90b5ecb1e6a54717b735 ] [test_storage.py: Avoid using the same working directory for different test classes. refs #999 david-sarah@jacaranda.org**20110929210954 Ignore-this: 3a01048e941c61c603eec603d064bebb ] [More asycification of tests. refs #999 david-sarah@jacaranda.org**20110929210727 Ignore-this: 87690a62f89a07e63b859c24948d262d ] [Fix a bug in disk_backend.py. refs #999 david-sarah@jacaranda.org**20110929182511 Ignore-this: 4f9a62adf03fc3221e46b54f7a4a960b ] [docs/backends/S3.rst: add s3.region option. Also minor changes to configuration.rst. refs #999 david-sarah@jacaranda.org**20110929182442 Ignore-this: 2992ead5f8d9357a0d9b912b1e0bd932 ] [Updates to test_backends.py. refs #999 david-sarah@jacaranda.org**20110929182016 Ignore-this: 3bac19179308e6f27e54c45c7cad4dc6 ] [Implement selection of backends from tahoe.cfg options. Also remove the discard_storage parameter from the disk backend. refs #999 david-sarah@jacaranda.org**20110929181754 Ignore-this: c7f78e7db98326723033f44e56858683 ] [test_storage.py: fix an incorrect argument in construction of S3Backend. refs #999 david-sarah@jacaranda.org**20110929081331 Ignore-this: 33ad68e0d3a15e3fa1dda90df1b8365c ] [Move the implementation of lease methods to disk_backend.py, and add stub implementations in s3_backend.py that raise NotImplementedError. Fix the lease methods in the disk backend to be synchronous. Also make sure that get_shares() returns a Deferred list sorted by shnum. refs #999 david-sarah@jacaranda.org**20110929081132 Ignore-this: 32cbad21c7236360e2e8e84a07f88597 ] [Make the make_bucket_writer method synchronous. refs #999 david-sarah@jacaranda.org**20110929080712 Ignore-this: 1de299e791baf1cf1e2a8d4b593e8ba1 ] [Add get_s3_share function in place of S3ShareSet._load_shares. refs #999 david-sarah@jacaranda.org**20110929080530 Ignore-this: f99665979612e42ecefa293bda0db5de ] [Complete the splitting of the immutable IStoredShare interface into IShareForReading and IShareForWriting. Also remove the 'load' method from shares, and other minor interface changes. refs #999 david-sarah@jacaranda.org**20110929075544 Ignore-this: 8c923051869cf162d9840770b4a08573 ] [split Immutable S3 Share into for-reading and for-writing classes, remove unused (as far as I can tell) methods, use cStringIO for buffering the writes zooko@zooko.com**20110929055038 Ignore-this: 82d8c4488a8548936285a975ef5a1559 TODO: define the interfaces that the new classes claim to implement ] [Comment out an assertion that was causing all mutable tests to fail. THIS IS PROBABLY WRONG. refs #999 david-sarah@jacaranda.org**20110929041110 Ignore-this: 1e402d51ec021405b191757a37b35a94 ] [Fix some incorrect or incomplete asyncifications. refs #999 david-sarah@jacaranda.org**20110929040800 Ignore-this: ed70e9af2190217c84fd2e8c41de4c7e ] [Add some debugging assertions that share objects are not Deferred. refs #999 david-sarah@jacaranda.org**20110929040657 Ignore-this: 5c7f56a146f5a3c353c6fe5b090a7dc5 ] [scripts/debug.py: take account of some API changes. refs #999 david-sarah@jacaranda.org**20110929040539 Ignore-this: 933c3d44b993c041105038c7d4514386 ] [Make get_sharesets_for_prefix synchronous for the time being (returning a Deferred breaks crawlers). refs #999 david-sarah@jacaranda.org**20110929040136 Ignore-this: e94b93d4f3f6173d9de80c4121b68748 ] [More asyncification of tests. refs #999 david-sarah@jacaranda.org**20110929035644 Ignore-this: 28b650a9ef593b3fd7524f6cb562ad71 ] [no_network.py: add some assertions that the things we wrap using LocalWrapper are not Deferred (which is not supported and causes hard-to-debug failures). refs #999 david-sarah@jacaranda.org**20110929035537 Ignore-this: fd103fbbb54fbbc17b9517c78313120e ] [Add some debugging code (switched off) to no_network.py. When switched on (PRINT_TRACEBACKS = True), this prints the stack trace associated with the caller of a remote method, mitigating the problem that the traceback normally gets lost at that point. TODO: think of a better way to preserve the traceback that can be enabled by default. refs #999 david-sarah@jacaranda.org**20110929035341 Ignore-this: 2a593ec3ee450719b241ea8d60a0f320 ] [Use factory functions to create share objects rather than their constructors, to allow the factory to return a Deferred. Also change some methods on IShareSet and IStoredShare to return Deferreds. Refactor some constants associated with mutable shares. refs #999 david-sarah@jacaranda.org**20110928052324 Ignore-this: bce0ac02f475bcf31b0e3b340cd91198 ] [Work in progress for asyncifying the backend interface (necessary to call txaws methods that return Deferreds). This is incomplete so lots of tests fail. refs #999 david-sarah@jacaranda.org**20110927073903 Ignore-this: ebdc6c06c3baa9460af128ec8f5b418b ] [mutable/publish.py: don't crash if there are no writers in _report_verinfo. refs #999 david-sarah@jacaranda.org**20110928014126 Ignore-this: 9999c82bb3057f755a6e86baeafb8a39 ] [scripts/debug.py: fix incorrect arguments to dump_immutable_share. refs #999 david-sarah@jacaranda.org**20110928014049 Ignore-this: 1078ee3f06a2f36b29e0cf694d2851cd ] [test_system.py: more debug output for a failing check in test_filesystem. refs #999 david-sarah@jacaranda.org**20110928014019 Ignore-this: e8bb77b8f7db12db7cd69efb6e0ed130 ] [test_system.py: incorrect arguments were being passed to the constructor for MutableDiskShare. refs #999 david-sarah@jacaranda.org**20110928013857 Ignore-this: e9719f74e7e073e37537f9a71614b8a0 ] [Undo an incompatible change to RIStorageServer. refs #999 david-sarah@jacaranda.org**20110928013729 Ignore-this: bea4c0f6cb71202fab942cd846eab693 ] [mutable/publish.py: resolve conflicting patches. refs #999 david-sarah@jacaranda.org**20110927073530 Ignore-this: 6154a113723dc93148151288bd032439 ] [test_storage.py: fix test_no_st_blocks. refs #999 david-sarah@jacaranda.org**20110927072848 Ignore-this: 5f12b784920f87d09c97c676d0afa6f8 ] [Cleanups to S3 backend (not including Deferred changes). refs #999 david-sarah@jacaranda.org**20110927071855 Ignore-this: f0dca788190d92b1edb1ee1498fb34dc ] [Cleanups to disk backend. refs #999 david-sarah@jacaranda.org**20110927071544 Ignore-this: e9d3fd0e85aaf301c04342fffdc8f26 ] [test_storage.py: fix test_status_bad_disk_stats. refs #999 david-sarah@jacaranda.org**20110927071403 Ignore-this: 6108fee69a60962be2df2ad11b483a11 ] [util/deferredutil.py: add some utilities for asynchronous iteration. refs #999 david-sarah@jacaranda.org**20110927070947 Ignore-this: ac4946c1e5779ea64b85a1a420d34c9e ] [Add 'has-immutable-readv' to server version information. refs #999 david-sarah@jacaranda.org**20110923220935 Ignore-this: c3c4358f2ab8ac503f99c968ace8efcf ] [Minor cleanup to disk backend. refs #999 david-sarah@jacaranda.org**20110923205510 Ignore-this: 79f92d7c2edb14cfedb167247c3f0d08 ] [Update the S3 backend. refs #999 david-sarah@jacaranda.org**20110923205345 Ignore-this: 5ca623a17e09ddad4cab2f51b49aec0a ] [Update the null backend to take into account interface changes. Also, it now records which shares are present, but not their contents. refs #999 david-sarah@jacaranda.org**20110923205219 Ignore-this: 42a23d7e253255003dc63facea783251 ] [Make EmptyShare.check_testv a simple function. refs #999 david-sarah@jacaranda.org**20110923204945 Ignore-this: d0132c085f40c39815fa920b77fc39ab ] [The cancel secret needs to be unique, even if it isn't explicitly provided. refs #999 david-sarah@jacaranda.org**20110923204914 Ignore-this: 6c44bb908dd4c0cdc59506b2d87a47b0 ] [Implement readv for immutable shares. refs #999 david-sarah@jacaranda.org**20110923204611 Ignore-this: 24f14b663051169d66293020e40c5a05 ] [Remove redundant si_s argument from check_write_enabler. refs #999 david-sarah@jacaranda.org**20110923204425 Ignore-this: 25be760118dbce2eb661137f7d46dd20 ] [interfaces.py: add fill_in_space_stats method to IStorageBackend. refs #999 david-sarah@jacaranda.org**20110923203723 Ignore-this: 59371c150532055939794fed6c77dcb6 ] [Add incomplete S3 backend. refs #999 david-sarah@jacaranda.org**20110923041314 Ignore-this: b48df65699e3926dcbb87b5f755cdbf1 ] [Move advise_corrupt_share to allmydata/storage/backends/base.py, since it will be common to the disk and S3 backends. refs #999 david-sarah@jacaranda.org**20110923041115 Ignore-this: 782b49f243bd98fcb6c249f8e40fd9f ] [A few comment cleanups. refs #999 david-sarah@jacaranda.org**20110923041003 Ignore-this: f574b4a3954b6946016646011ad15edf ] [mutable/publish.py: elements should not be removed from a dictionary while it is being iterated over. refs #393 david-sarah@jacaranda.org**20110923040825 Ignore-this: 135da94bd344db6ccd59a576b54901c1 ] [Blank line cleanups. david-sarah@jacaranda.org**20110923012044 Ignore-this: 8e1c4ecb5b0c65673af35872876a8591 ] [Reinstate the cancel_lease methods of ImmutableDiskShare and MutableDiskShare, since they are needed for lease expiry. refs #999 david-sarah@jacaranda.org**20110922183323 Ignore-this: a11fb0dd0078ff627cb727fc769ec848 ] [Fix most of the crawler tests. refs #999 david-sarah@jacaranda.org**20110922183008 Ignore-this: 116c0848008f3989ba78d87c07ec783c ] [Fix some more test failures. refs #999 david-sarah@jacaranda.org**20110922045451 Ignore-this: b726193cbd03a7c3d343f6e4a0f33ee7 ] [uri.py: resolve a conflict between trunk and the pluggable-backends patches. refs #999 david-sarah@jacaranda.org**20110921222038 Ignore-this: ffeeab60d8e71a6a29a002d024d76fcf ] [Fix more shallow bugs, mainly FilePathification. Also, remove the max_space_per_bucket parameter from BucketWriter since it can be obtained from the _max_size attribute of the share (via a new get_allocated_size() accessor). refs #999 david-sarah@jacaranda.org**20110921221421 Ignore-this: 600e3ccef8533aa43442fa576c7d88cf ] [More fixes to tests needed for pluggable backends. refs #999 david-sarah@jacaranda.org**20110921184649 Ignore-this: 9be0d3a98e350fd4e17a07d2c00bb4ca ] [docs/backends/S3.rst, disk.rst: describe type of space settings as 'quantity of space', not 'str'. refs #999 david-sarah@jacaranda.org**20110921031705 Ignore-this: a74ed8e01b0a1ab5f07a1487d7bf138 ] [docs/backends/S3.rst: remove Issues section. refs #999 david-sarah@jacaranda.org**20110921031625 Ignore-this: c83d8f52b790bc32488869e6ee1df8c2 ] [Fix some incorrect attribute accesses. refs #999 david-sarah@jacaranda.org**20110921031207 Ignore-this: f1ea4c3ea191f6d4b719afaebd2b2bcd ] [docs/backends: document the configuration options for the pluggable backends scheme. refs #999 david-sarah@jacaranda.org**20110920171737 Ignore-this: 5947e864682a43cb04e557334cda7c19 ] [Work-in-progress, includes fix to bug involving BucketWriter. refs #999 david-sarah@jacaranda.org**20110920033803 Ignore-this: 64e9e019421454e4d08141d10b6e4eed ] [Pluggable backends -- all other changes. refs #999 david-sarah@jacaranda.org**20110919233256 Ignore-this: 1a77b6b5d178b32a9b914b699ba7e957 ] [Pluggable backends -- new and moved files, changes to moved files. refs #999 david-sarah@jacaranda.org**20110919232926 Ignore-this: ec5d2d1362a092d919e84327d3092424 ] [interfaces.py: 'which -> that' grammar cleanup. david-sarah@jacaranda.org**20110825003217 Ignore-this: a3e15f3676de1b346ad78aabdfb8cac6 ] [test/test_runner.py: BinTahoe.test_path has rare nondeterministic failures; this patch probably fixes a problem where the actual cause of failure is masked by a string conversion error. david-sarah@jacaranda.org**20110927225336 Ignore-this: 6f1ad68004194cc9cea55ace3745e4af ] [docs/configuration.rst: add section about the types of node, and clarify when setting web.port enables web-API service. fixes #1444 zooko@zooko.com**20110926203801 Ignore-this: ab94d470c68e720101a7ff3c207a719e ] [TAG allmydata-tahoe-1.9.0a2 warner@lothar.com**20110925234811 Ignore-this: e9649c58f9c9017a7d55008938dba64f ] Patch bundle hash: afa65f364a9eb7e0fbe701e69d8edb148b9e16b4