tahoe-lafs/trac

webapi PUT via multiple nodes can cause directory corruption but does not report UncoordinatedWriteError #970

New issue

Open

opened 2010-02-23 05:28:21 +00:00 by stott · 6 comments

stott commented

2010-02-23 05:28:21 +00:00

Owner

Multiple simultaneous Tahoe put(s) via web API cause directory level corruption resulting in no recoverable data.

To recreate

Step 1.) Create Directory ; Get directory writecap.

Step 2.) Using 61 .JPG files avg 1.7MB use test.sh script to put files to Tahoe-Lafs.

bash-3.2$ du -sh .
102m

bash-3.2$ ls *.JPG |wc -l
61

bash-3.2$ cat test.sh
#!/bin/sh

# From Directory listing itself == Directory Write CAP
FW="URI:DIR2:tuz27wvy27ua4mt5lyotllbyke:phzv6ilb5gssi3zy33nki62zcudqjzyv7v7w4qaavwn5kuh2hawa"



X=3456
for I in `ls *.JPG`
	do
curl -T $I  http://10.20.0.151:$X/uri/$FW/$I & 
#echo "curl -T $I  http://10.20.0.151:$X/uri/$FW/$I & "
         X=`expr $X + 1`
        if [ $X -le 3500 ] ; then 
           echo "Submitting $I"
           else X=3456;
        fi  
        done

Error returned from curl

UnrecoverableFileError: the directory (or mutable file) could not be retrieved, because there were insufficient good shares. This might indicate that no servers were connected, insufficient servers were connected, the URI was corrupt, or that shares have been lost due to server departure, hard drive failure, or disk corruption. You should perform a filecheck on this object to learn more.

Error generated when trying to retrieve known good URI from child:

attachment:970-paste.txt

Multiple simultaneous Tahoe put(s) via web API cause directory level corruption resulting in no recoverable data. To recreate Step 1.) Create Directory ; Get directory writecap. Step 2.) Using 61 .JPG files avg 1.7MB use test.sh script to put files to Tahoe-Lafs. ------------------- bash-3.2$ du -sh . 102m ------------------- bash-3.2$ ls *.JPG |wc -l 61 ------------------- ``` bash-3.2$ cat test.sh #!/bin/sh # From Directory listing itself == Directory Write CAP FW="URI:DIR2:tuz27wvy27ua4mt5lyotllbyke:phzv6ilb5gssi3zy33nki62zcudqjzyv7v7w4qaavwn5kuh2hawa" X=3456 for I in `ls *.JPG` do curl -T $I http://10.20.0.151:$X/uri/$FW/$I & #echo "curl -T $I http://10.20.0.151:$X/uri/$FW/$I & " X=`expr $X + 1` if [ $X -le 3500 ] ; then echo "Submitting $I" else X=3456; fi done ``` ------------------------------------------- Error returned from curl ``` UnrecoverableFileError: the directory (or mutable file) could not be retrieved, because there were insufficient good shares. This might indicate that no servers were connected, insufficient servers were connected, the URI was corrupt, or that shares have been lost due to server departure, hard drive failure, or disk corruption. You should perform a filecheck on this object to learn more. ``` ------------------------ Error generated when trying to retrieve known good URI from child: [attachment:970-paste.txt](/tahoe-lafs/trac/attachments/000078ac-636f-d04e-be81-b63796148caa) -------------------------

tahoe-lafs added the

labels 2010-02-23 05:28:21 +00:00

tahoe-lafs added this to the undecided milestone 2010-02-23 05:28:21 +00:00

daira commented

2010-02-23 07:08:53 +00:00

Note that the script is trying to upload using a different webapi server instance for each file, contrary to the [prime coordination directive]source:docs/frontends/webapi.txt?rev=4236#L1786.

However, this should probably have caused an UncoordinatedWriteError, rather than UnrecoverableFileError.

Note that the script is trying to upload using a different webapi server instance for each file, contrary to the [prime coordination directive]source:docs/frontends/webapi.txt?rev=4236#L1786. However, this should probably have caused an UncoordinatedWriteError, rather than UnrecoverableFileError.

daira added

c/code-frontend-web

and removed

labels 2010-02-23 07:08:53 +00:00

daira changed title from ~~Tahoe Put can cause directory corruption~~ to webapi PUT via multiple nodes can cause directory corruption but does not report UncoordinatedWriteError

2010-02-23 07:08:53 +00:00

warner commented

2010-02-23 10:00:16 +00:00

wow, it sounds like you were actually able to provoke a real UCWE! Well, a real collision, at least.

So, did really none of the PUT commands result in an error? I would have expected at least one of them to emit a UCWE. Re-running the test and sending the output of each curl instance to a separate logfile would help answer this question. Also double-checking that curl emits errors to stdout when it gets a 500 or whatever HTTP error code UCWE maps to.

If you could, please do a file-check (with --verify) on the directory in question. With the dircap you show, the command would be "tahoe check --verify --raw $FW". I'm expecting to see a small number of shares of each version, for several different versions.

The file-check output will tell us, but what were the encoding parameters in use when you ran this test? I know from another ticket you were experimenting with parameters on the order of 40-of-50.. if the dirnodes (and other mutable files) were created with these same parameters, they'd be much more vulnerable to UCWE than with the normal 3-of-10 encoding. If that was a factor here, we might want to consider separate encoding-parameter configs for dirnodes (or perhaps for all mutable files), so that you can use safer 3-of-10 for them and more efficient 40-of-50 for immutable bulk data. (Note that protection from UCWE comes from small "k", whereas the usual reliability against server problems comes from having a large N-k).

(also, incidentally, when pasting large shell transcripts into a Trac page like this one, you should wrap the block with triple-curlies, so that Trac will not try to interpret the comments as WikiFormatting. And please attach other things as trac attachments instead of e.g. codepad links, because a few months from now when somebody comes back to look at this ticket, the pastebin will have expired and the contents lost)

wow, it sounds like you were actually able to provoke a real UCWE! Well, a real collision, at least. So, did really none of the PUT commands result in an error? I would have expected at least one of them to emit a UCWE. Re-running the test and sending the output of each curl instance to a separate logfile would help answer this question. Also double-checking that curl emits errors to stdout when it gets a 500 or whatever HTTP error code UCWE maps to. If you could, please do a file-check (with --verify) on the directory in question. With the dircap you show, the command would be "`tahoe check --verify --raw $FW`". I'm expecting to see a small number of shares of each version, for several different versions. The file-check output will tell us, but what were the encoding parameters in use when you ran this test? I know from another ticket you were experimenting with parameters on the order of 40-of-50.. if the dirnodes (and other mutable files) were created with these same parameters, they'd be much more vulnerable to UCWE than with the normal 3-of-10 encoding. If that was a factor here, we might want to consider separate encoding-parameter configs for dirnodes (or perhaps for all mutable files), so that you can use safer 3-of-10 for them and more efficient 40-of-50 for immutable bulk data. (Note that protection from UCWE comes from small "k", whereas the usual reliability against server problems comes from having a large N-k). (also, incidentally, when pasting large shell transcripts into a Trac page like this one, you should wrap the block with triple-curlies, so that Trac will not try to interpret the comments as [WikiFormatting](wiki/WikiFormatting). And please attach other things as trac attachments instead of e.g. codepad links, because a few months from now when somebody comes back to look at this ticket, the pastebin will have expired and the contents lost)

zooko commented

2010-02-23 13:27:15 +00:00

Careful here -- the only safe solution is to coordinate writes. Having different encoding parameters such as a smaller k may reduce the amount of damage to your data when an uncoordinated write is encountered (or it may not -- I vaguely recall that the reasoning about that was surprisingly subtle), but one shouldn't rely on that to protect one's data.

Careful here -- the *only* safe solution is to coordinate writes. Having different encoding parameters such as a smaller `k` may reduce the amount of damage to your data when an uncoordinated write is encountered (or it may not -- I vaguely recall that the reasoning about that was surprisingly subtle), but one shouldn't rely on that to protect one's data.

zooko commented

2010-02-23 13:28:23 +00:00

source:docs/specifications/mutable.txt@3696#L48 The Prime Coordination Directive]

source:docs/specifications/mutable.txt@3696#L48 The Prime Coordination Directive]

zooko commented

2010-02-23 13:32:57 +00:00

Hm, the docs in source:docs/specifications/mutable.txt@3696#L48 suggest the following four strategies:

don't give the read-write URI to anyone else. Dirnodes in a private
directory generally satisfy this case, as long as you don't use two
clients on the same account at the same time
if you give a read-write URI to someone else, stop using it yourself. An
inbox would be a good example of this.
if you give a read-write URI to someone else, call them on the phone
before you write into it
build an automated mechanism to have your agents coordinate writes.
For example, we expect a future release to include a FURL for a
"coordination server" in the dirnodes. The rule can be that you must
contact the coordination server and obtain a lock/lease on the file
before you're allowed to modify it.

Then later they mention that a Tahoe-LAFS gateway automatically serializes requests that go through it to the same mutable file or directory. Perhaps that should be promoted to a fifth strategy: make all your writes to a certain mutable file or directory go through a certain Tahoe-LAFS gateway.

Hm, the docs in source:docs/specifications/mutable.txt@3696#L48 suggest the following four strategies: * don't give the read-write URI to anyone else. Dirnodes in a private directory generally satisfy this case, as long as you don't use two clients on the same account at the same time * if you give a read-write URI to someone else, stop using it yourself. An inbox would be a good example of this. * if you give a read-write URI to someone else, call them on the phone before you write into it * build an automated mechanism to have your agents coordinate writes. For example, we expect a future release to include a FURL for a "coordination server" in the dirnodes. The rule can be that you must contact the coordination server and obtain a lock/lease on the file before you're allowed to modify it. Then later they mention that a Tahoe-LAFS gateway automatically serializes requests that go through it to the same mutable file or directory. Perhaps that should be promoted to a fifth strategy: make all your writes to a certain mutable file or directory go through a certain Tahoe-LAFS gateway.

daira commented

2011-08-17 00:13:04 +00:00

Attachment 970-paste.txt (4450 bytes) added

Error generated when trying to retrieve known good URI from child

**Attachment** 970-paste.txt (4450 bytes) added Error generated when trying to retrieve known good URI from child

970-paste.txt

4.3 KiB

Sign in to join this conversation.

No labels

c/code-dirnodes

c/code-encoding

c/code-frontend

c/code-frontend-cli

c/code-frontend-ftp-sftp

c/code-frontend-magic-folder

c/code-frontend-web

c/code-nodeadmin

c/code-peerselection

c/dev-infrastructure

kw:AttributeError

kw:DataUnavailable

kw:DeadReferenceError

kw:GetLastError

kw:IFinishableConsumer

kw:LeastAuthority

kw:RIStorageServer

kw:UncoordinatedWriteError

kw:access-control

kw:accessibility

kw:accounting-crawler

kw:anti-censorship

kw:api_auth_token

kw:authentication

kw:availability

kw:backward-compatibility

kw:blocks-cloud-deployment

kw:blocks-cloud-merge

kw:blocks-magic-folder-merge

kw:blocks-merge

kw:blocks-release

kw:brians-opinion-needed

kw:build-helpers

kw:cloud-backend

kw:coding-standards

kw:coding-tools

kw:coding_tools

kw:compatibility

kw:confidentiality

kw:configuration

kw:configuration.txt

kw:connectivity

kw:control.furl

kw:coordination

kw:coveralls.io

kw:create-container

kw:cryptography

kw:cryptography-lib

kw:denial-of-service

kw:desert-island

kw:desert-island-build

kw:design-review-needed

kw:dev-infrastructure

kw:directory-page

kw:disk-backend

kw:earth-dragon

kw:erasure-coding

kw:excess-authority

kw:extensibility

kw:file-descriptor

kw:floatingpoint

kw:forward-compatibility

kw:forward-secrecy

kw:garbage-collection

kw:google-cloud-storage

kw:google-drive-backend

kw:grid-manager

kw:illustration

kw:inlineCallbacks

kw:integration-test

kw:interoperability

kw:interstellar-exploration

kw:introduction

kw:key-value-store

kw:magic-folder

kw:manual-test-needed

kw:more-info-needed

kw:mountain-lion

kw:multiuser-gateway

kw:needs-review

kw:notification

kw:notifyOnDisconnect

kw:openitp-packaging

kw:operation-helpers

kw:optimization

kw:organization

kw:otf-magic-folder-objective1

kw:otf-magic-folder-objective2

kw:otf-magic-folder-objective3

kw:otf-magic-folder-objective4

kw:otf-magic-folder-objective5

kw:otf-magic-folder-objective6

kw:peer-selection

kw:pkg_resources

kw:preservation

kw:pycrypto-lib

kw:pyfilesystem

kw:random-access

kw:raspberry-pi

kw:release-blocker

kw:removable-disk

kw:reserved_space

kw:response-needed

kw:response-time

kw:review-needed

kw:self-contained

kw:servers-of-happiness

kw:setup_requires

kw:setuptools_darcs

kw:simultaneous

kw:space-efficiency

kw:static-analysis

kw:stats_gatherer

kw:tahoe-add-alias

kw:tahoe-archive

kw:tahoe-backup

kw:tahoe-create-alias

kw:tahoe-create-introducer

kw:tahoe-deep-check

kw:tahoe-deepcheck

kw:tahoe-lafs-trac-stream

kw:tahoe-list-aliases

kw:tahoe-magic-folder

kw:tahoe-manifest

kw:tahoe-restart

kw:tahoe-unlink

kw:tahoe-webopen

kw:test-and-set

kw:test-from-egg

kw:ticket999-s3-backend

kw:to-be-closed-on-2011-08-01

kw:tor-protocol

kw:transparency

kw:tub.location

kw:twisted-trial

kw:uncoordinated-writes

kw:unfinished-business

kw:unhandled-error

kw:upload-helper

kw:visualization

kw:volunteergrid2

kw:warners-opinion-needed

kw:welcome-page

kw:windows-related

kw:world-domination

kw:write-enabler

kw:zookos-opinion-needed

kw:zope.interface

p/supercritical

r/cannot reproduce

r/somebody else's problem

r/was already fixed

v/1.9.0-s3branch

No milestone

No project

No assignees

4 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: tahoe-lafs/trac#970

No description provided.