Thu Sep 1 02:41:44 MDT 2011 wilcoxjg@gmail.com
* storage/mutable.py: special characters in struct.foo arguments indicate standard as opposed to native sizes, we should be using these characters in these asserts
New patches:
[storage/mutable.py: special characters in struct.foo arguments indicate standard as opposed to native sizes, we should be using these characters in these asserts
wilcoxjg@gmail.com**20110901084144
Ignore-this: 28ace2b2678642e4d7269ddab8c67f30
] hunk ./src/allmydata/storage/mutable.py 31
# 9 ?? n*92 extra leases
-assert struct.calcsize("L"), 4 # The struct module doc says that L's are 4 bytes in size.
-assert struct.calcsize("Q"), 8 # The struct module doc says that Q's are 8 bytes in size (at least with big-endian ordering).
+assert struct.calcsize(">L") == 4 # The struct module doc says that L's are 4 bytes in size.
+assert struct.calcsize(">Q") == 8 # The struct module doc says that Q's are 8 bytes in size (at least with big-endian ordering).
class MutableShareFile:
Context:
[docs/write_coordination.rst: fix formatting and add more specific warning about access via sshfs.
david-sarah@jacaranda.org**20110831232148
Ignore-this: cd9c851d3eb4e0a1e088f337c291586c
]
[test_mutable.Version: consolidate some tests, reduce runtime from 19s to 15s
warner@lothar.com**20110831050451
Ignore-this: 64815284d9e536f8f3798b5f44cf580c
]
[mutable/retrieve: handle the case where self._read_length is 0.
Kevan Carstensen <kevan@isnotajoke.com>**20110830210141
Ignore-this: fceafbe485851ca53f2774e5a4fd8d30
Note that the downloader will still fetch a segment for a zero-length
read, which is wasteful. Fixing that isn't specifically required to fix
#1512, but it should probably be fixed before 1.9.
]
[NEWS: added summary of all changes since 1.8.2. Needs editing.
Brian Warner <warner@lothar.com>**20110830163205
Ignore-this: 273899b37a899fc6919b74572454b8b2
]
[test_mutable.Update: only upload the files needed for each test. refs #1500
Brian Warner <warner@lothar.com>**20110829072717
Ignore-this: 4d2ab4c7523af9054af7ecca9c3d9dc7
This first step shaves 15% off the runtime: from 139s to 119s on my laptop.
It also fixes a couple of places where a Deferred was being dropped, which
would cause two tests to run in parallel and also confuse error reporting.
]
[Let Uploader retain History instead of passing it into upload(). Fixes #1079.
Brian Warner <warner@lothar.com>**20110829063246
Ignore-this: 3902c58ec12bd4b2d876806248e19f17
This consistently records all immutable uploads in the Recent Uploads And
Downloads page, regardless of code path. Previously, certain webapi upload
operations (like PUT /uri/$DIRCAP/newchildname) failed to pass the History
object and were left out.
]
[Fix mutable publish/retrieve timing status displays. Fixes #1505.
Brian Warner <warner@lothar.com>**20110828232221
Ignore-this: 4080ce065cf481b2180fd711c9772dd6
publish:
* encrypt and encode times are cumulative, not just current-segment
retrieve:
* same for decrypt and decode times
* update "current status" to include segment number
* set status to Finished/Failed when download is complete
* set progress to 1.0 when complete
More improvements to consider:
* progress is currently 0% or 100%: should calculate how many segments are
involved (remembering retrieve can be less than the whole file) and set it
to a fraction
* "fetch" time is fuzzy: what we want is to know how much of the delay is not
our own fault, but since we do decode/decrypt work while waiting for more
shares, it's not straightforward
]
[Teach 'tahoe debug catalog-shares about MDMF. Closes #1507.
Brian Warner <warner@lothar.com>**20110828080931
Ignore-this: 56ef2951db1a648353d7daac6a04c7d1
]
[debug.py: remove some dead comments
Brian Warner <warner@lothar.com>**20110828074556
Ignore-this: 40e74040dd4d14fd2f4e4baaae506b31
]
[hush pyflakes
Brian Warner <warner@lothar.com>**20110828074254
Ignore-this: bef9d537a969fa82fe4decc4ba2acb09
]
[MutableFileNode.set_downloader_hints: never depend upon order of dict.values()
Brian Warner <warner@lothar.com>**20110828074103
Ignore-this: caaf1aa518dbdde4d797b7f335230faa
The old code was calculating the "extension parameters" (a list) from the
downloader hints (a dictionary) with hints.values(), which is not stable, and
would result in corrupted filecaps (with the 'k' and 'segsize' hints
occasionally swapped). The new code always uses [k,segsize].
]
[layout.py: fix MDMF share layout documentation
Brian Warner <warner@lothar.com>**20110828073921
Ignore-this: 3f13366fed75b5e31b51ae895450a225
]
[teach 'tahoe debug dump-share' about MDMF and offsets. refs #1507
Brian Warner <warner@lothar.com>**20110828073834
Ignore-this: 3a9d2ef9c47a72bf1506ba41199a1dea
]
[test_mutable.Version.test_debug: use splitlines() to fix buildslaves
Brian Warner <warner@lothar.com>**20110828064728
Ignore-this: c7f6245426fc80b9d1ae901d5218246a
Any slave running in a directory with spaces in the name was miscounting
shares, causing the test to fail.
]
[test_mutable.Version: exercise 'tahoe debug find-shares' on MDMF. refs #1507
Brian Warner <warner@lothar.com>**20110828005542
Ignore-this: cb20bea1c28bfa50a72317d70e109672
Also changes NoNetworkGrid to put shares in storage/shares/ .
]
[test_mutable.py: oops, missed a .todo
Brian Warner <warner@lothar.com>**20110828002118
Ignore-this: fda09ae86481352b7a627c278d2a3940
]
[test_mutable: merge davidsarah's patch with my Version refactorings
warner@lothar.com**20110827235707
Ignore-this: b5aaf481c90d99e33827273b5d118fd0
]
[Make the immutable/read-only constraint checking for MDMF URIs identical to that for SSK URIs. refs #393
david-sarah@jacaranda.org**20110823012720
Ignore-this: e1f59d7ff2007c81dbef2aeb14abd721
]
[Additional tests for MDMF URIs and for zero-length files. refs #393
david-sarah@jacaranda.org**20110823011532
Ignore-this: a7cc0c09d1d2d72413f9cd227c47a9d5
]
[Additional tests for zero-length partial reads and updates to mutable versions. refs #393
david-sarah@jacaranda.org**20110822014111
Ignore-this: 5fc6f4d06e11910124e4a277ec8a43ea
]
[test_mutable.Version: factor out some expensive uploads, save 25% runtime
Brian Warner <warner@lothar.com>**20110827232737
Ignore-this: ea37383eb85ea0894b254fe4dfb45544
]
[SDMF: update filenode with correct k/N after Retrieve. Fixes #1510.
Brian Warner <warner@lothar.com>**20110827225031
Ignore-this: b50ae6e1045818c400079f118b4ef48
Without this, we get a regression when modifying a mutable file that was
created with more shares (larger N) than our current tahoe.cfg . The
modification attempt creates new versions of the (0,1,..,newN-1) shares, but
leaves the old versions of the (newN,..,oldN-1) shares alone (and throws a
assertion error in SDMFSlotWriteProxy.finish_publishing in the process).
The mixed versions that result (some shares with e.g. N=10, some with N=20,
such that both versions are recoverable) cause problems for the Publish code,
even before MDMF landed. Might be related to refs #1390 and refs #1042.
]
[layout.py: annotate assertion to figure out 'tahoe backup' failure
Brian Warner <warner@lothar.com>**20110827195253
Ignore-this: 9b92b954e3ed0d0f80154fff1ff674e5
]
[Add 'tahoe debug dump-cap' support for MDMF, DIR2-CHK, DIR2-MDMF. refs #1507.
Brian Warner <warner@lothar.com>**20110827195048
Ignore-this: 61c6af5e33fc88e0251e697a50addb2c
This also adds tests for all those cases, and fixes an omission in uri.py
that broke parsing of DIR2-MDMF-Verifier and DIR2-CHK-Verifier.
]
[MDMF: more writable/writeable consistentifications
warner@lothar.com**20110827190602
Ignore-this: 22492a9e20c1819ddb12091062888b55
]
[MDMF: s/Writable/Writeable/g, for consistency with existing SDMF code
warner@lothar.com**20110827183357
Ignore-this: 9dd312acedbdb2fc2f7bef0d0fb17c0b
]
[setup.cfg: remove no-longer-supported test_mac_diskimage alias. refs #1479
david-sarah@jacaranda.org**20110826230345
Ignore-this: 40e908b8937322a290fb8012bfcad02a
]
[test_mutable.Update: increase timeout from 120s to 400s, slaves are failing
Brian Warner <warner@lothar.com>**20110825230140
Ignore-this: 101b1924a30cdbda9b2e419e95ca15ec
]
[tests: fix check_memory test
zooko@zooko.com**20110825201116
Ignore-this: 4d66299fa8cb61d2ca04b3f45344d835
fixes #1503
]
[TAG allmydata-tahoe-1.9.0a1
warner@lothar.com**20110825161122
Ignore-this: 3cbf49f00dbda58189f893c427f65605
]
Patch bundle hash:
7b28aaf9e2830e6259ded7e99dfcde7376d25c19