tahoe-lafs/trac-2024-07-25

KeyError exception seen in log when doing a mutable publish on the pubgrid #1042

New issue

Open

opened 2010-05-15 03:30:09 +00:00 by davidsarah · 6 comments

davidsarah commented

2010-05-15 03:30:09 +00:00

Owner

The file in question was mutable.txt in this directory.

See attached log. This happened when publishing a mutable file version using SFTP (with WinSCP client on Windows), but I don't think that's directly relevant. It may be relevant that there were only two servers connected to the grid at the time. The same error happened again on a retry. The file in question was mutable.txt in [this directory](http://pubgrid.tahoe-lafs.org/uri/URI%3ADIR2%3Atunj3c3jt4znroz6fjwxvpveka%3Azzjsx5hwgjymia4nixlj37t7476jm3ypfqgqplj6qhgpykqylsla/).

tahoe-lafs added the

labels 2010-05-15 03:30:09 +00:00

tahoe-lafs added this to the undecided milestone 2010-05-15 03:30:09 +00:00

davidsarah commented

2010-05-15 03:33:39 +00:00

Author

Owner

Attachment log.txt (9318 bytes) added

Log showing KeyError exception

**Attachment** log.txt (9318 bytes) added Log showing [KeyError](wiki/KeyError) exception

log.txt

9.1 KiB

davidsarah commented

2010-05-15 03:37:45 +00:00

Author

Owner

The two servers connected were "shiny" (etnv37dedl2kef5c5ny2obgftav2eelw) and the public gateway "soultcer-skulls" (sp26qyqclbwjv6lgqdxik5lfdlxrkpnz). "shiny" is my server and is only on-line when I am. The log is from "shiny".

davidsarah commented

2010-05-19 21:01:23 +00:00

Author

Owner

I have also seen this (on the same file) with tahoe put, so it's definitely not SFTP-specific. The encoding parameters had been set to N = 2, k = 2, happy = 2 (but the #778 patches had not been applied).

I have also seen this (on the same file) with `tahoe put`, so it's definitely not SFTP-specific. The encoding parameters had been set to N = 2, k = 2, happy = 2 (but the #778 patches had not been applied).

zooko commented

2010-05-27 22:12:59 +00:00

Author

Owner

It's really bothering me that mutable file upload and download behavior is so finicky, buggy, inefficient, hard to understand, different from immutable file upload and download behavior, etc. So I'm putting a bunch of tickets into the "1.8" Milestone. I am not, however, at this time, volunteering to work on these tickets, so it might be a mistake to put them into the 1.8 Milestone, but I really hope that someone else will volunteer or that I will decide to do it myself. :-)

tahoe-lafs modified the milestone from undecided to 1.8.0

2010-05-27 22:12:59 +00:00

tahoe-lafs modified the milestone from 1.8.0 to eventually

2010-08-14 06:45:17 +00:00

Brian Warner <warner@lothar.com> commented

2011-08-27 22:51:05 +00:00

Author

Owner

In changeset:370e6f271e40945b:

SDMF: update filenode with correct k/N after Retrieve. Fixes #1510.

Without this, we get a regression when modifying a mutable file that was
created with more shares (larger N) than our current tahoe.cfg . The
modification attempt creates new versions of the (0,1,..,newN-1) shares, but
leaves the old versions of the (newN,..,oldN-1) shares alone (and throws a
assertion error in SDMFSlotWriteProxy.finish_publishing in the process).

The mixed versions that result (some shares with e.g. N=10, some with N=20,
such that both versions are recoverable) cause problems for the Publish code,
even before MDMF landed. Might be related to refs #1390 and refs #1042.

In changeset:370e6f271e40945b: ``` SDMF: update filenode with correct k/N after Retrieve. Fixes #1510. Without this, we get a regression when modifying a mutable file that was created with more shares (larger N) than our current tahoe.cfg . The modification attempt creates new versions of the (0,1,..,newN-1) shares, but leaves the old versions of the (newN,..,oldN-1) shares alone (and throws a assertion error in SDMFSlotWriteProxy.finish_publishing in the process). The mixed versions that result (some shares with e.g. N=10, some with N=20, such that both versions are recoverable) cause problems for the Publish code, even before MDMF landed. Might be related to refs #1390 and refs #1042. ```

davidsarah commented

2011-08-27 23:24:51 +00:00

Author

Owner

The bug in #1510 caused a situation where there were stored shares for an SDMF file with higher shnums than expected. When the resulting file is updated using version 1.8.2 (or earlier, presumably), this would cause a KeyError almost identical to the one in log.txt. This situation is still inadequately tested, but note that it may cause different errors after 1.9 (specifically, after the MDMF changes were landed) than before 1.9.

The bug in #1510 caused a situation where there were stored shares for an SDMF file with higher shnums than expected. When the resulting file is updated using version 1.8.2 (or earlier, presumably), this would cause a KeyError almost identical to the one in [log.txt](/tahoe-lafs/trac-2024-07-25/attachments/000078ac-8b2d-d04e-8757-89dbeb08d7aa). This situation is still inadequately tested, but note that it may cause different errors after 1.9 (specifically, after the MDMF changes were landed) than before 1.9.