1 patch for repository zooko@dev.allmydata.org:/home/darcs/tahoe-lafs/trunk: Mon Jan 3 23:54:55 MST 2011 zooko@zooko.com * docs: update performance.rst to describe the difference between already-uploaded and not-already-uploaded, to parameterize segment size, and to use "~A" to mean "approximately A" New patches: [docs: update performance.rst to describe the difference between already-uploaded and not-already-uploaded, to parameterize segment size, and to use "~A" to mean "approximately A" zooko@zooko.com**20110104065455 Ignore-this: 8df0d79a062ee19854c0211bd202f606 ] { hunk ./docs/performance.rst 17 10. `Performing a file-verify on an A-byte file`_ 11. `Repairing an A-byte file (mutable or immutable)`_ +``K`` indicates the number of shares required to reconstruct the file +(default: 3) + +``N`` indicates the total number of shares produced (default: 10) + +``S`` indicates the segment size (default: 128 KiB) + +``A`` indicates the number of bytes in a file + +``B`` indicates the number of bytes of a file which are being read or +written + +``G`` indicates the number of storage servers on your grid + Publishing an ``A``-byte immutable file ======================================= hunk ./docs/performance.rst 34 -network: A +when the file is already uploaded +--------------------------------- + +If the file is already uploaded with the exact same contents, same +erasure coding parameters (K, N), and same added convergence secret, +then it reads the whole file from disk one time while hashing it to +compute the storage index, then contacts about N servers to ask each +one to store a share. All of the servers reply that they already have +a copy of that share, and the upload is done. + +disk: A + +cpu: ~A + +network: ~N + +memory footprint: N/K*S + +when the file is not already uploaded +------------------------------------- + +If the file is not already uploaded with the exact same contents, same +erasure coding parameters (K, N), and same added convergence secret, +then it reads the whole file from disk one time while hashing it to +compute the storage index, then contacts about N servers to ask each +one to store a share. Then it uploads each share to a storage server. + +disk: 2*A + +cpu: 2*~A hunk ./docs/performance.rst 65 -memory footprint: N/k*128KiB +network: ~N + ~A hunk ./docs/performance.rst 67 -notes: An immutable file upload requires an additional I/O pass over the entire -source file before the upload process can start, since convergent -encryption derives the encryption key in part from the contents of the -source file. +memory footprint: N/K*S Publishing an ``A``-byte mutable file ===================================== hunk ./docs/performance.rst 74 network: A -memory footprint: N/k*A +memory footprint: N/K*A hunk ./docs/performance.rst 76 -cpu: O(A) + a large constant for RSA keypair generation +cpu: ~A + a large constant for RSA keypair generation notes: Tahoe-LAFS generates a new RSA keypair for each mutable file that it publishes to a grid. This takes up to 1 or 2 seconds on a typical desktop PC. hunk ./docs/performance.rst 91 network: B -memory footprint: 128KiB +cpu: ~A hunk ./docs/performance.rst 93 -notes: When Tahoe-LAFS 1.8.0 or later is asked to read an arbitrary range -of an immutable file, only the 128-KiB segments that overlap the +notes: When Tahoe-LAFS 1.8.0 or later is asked to read an arbitrary +range of an immutable file, only the S-byte segments that overlap the requested range will be downloaded. (Earlier versions would download from the beginning of the file up hunk ./docs/performance.rst 117 network: A -memory footprint: N/k*A +memory footprint: N/K*A notes: If you upload a changed version of a mutable file that you earlier put onto your grid with, say, 'tahoe put --mutable', hunk ./docs/performance.rst 132 network: A -memory footprint: N/k*A +memory footprint: N/K*A notes: Modifying any part of a mutable file in Tahoe-LAFS requires that the entire file be downloaded, modified, held in memory while it is hunk ./docs/performance.rst 147 Adding an entry to an ``A``-entry directory =========================================== -network: O(A) +network: ~A hunk ./docs/performance.rst 149 -memory footprint: N/k*A +memory footprint: N/K*A notes: In Tahoe-LAFS, directories are implemented as specialized mutable files. So adding an entry to a directory is essentially adding B hunk ./docs/performance.rst 158 Listing an ``A`` entry directory ================================ -network: O(A) +network: ~A hunk ./docs/performance.rst 160 -memory footprint: N/k*A +memory footprint: N/K*A notes: Listing a directory requires that the mutable file storing the directory be downloaded from the grid. So listing an A entry hunk ./docs/performance.rst 170 Performing a file-check on an ``A``-byte file ============================================= -network: O(S), where S is the number of servers on your grid +network: ~G, where G is the number of servers on your grid memory footprint: negligible hunk ./docs/performance.rst 182 Performing a file-verify on an ``A``-byte file ============================================== -network: N/k*A +network: N/K*A hunk ./docs/performance.rst 184 -memory footprint: N/k*128KiB +memory footprint: N/K*S notes: To verify a file, Tahoe-LAFS downloads all of the ciphertext shares that were originally uploaded to the grid and integrity hunk ./docs/performance.rst 195 Repairing an ``A``-byte file (mutable or immutable) =================================================== -network: variable; up to around O(A) +network: variable; up to around ~A hunk ./docs/performance.rst 197 -memory footprint: from 128KiB to (1+N/k)*128KiB +memory footprint: from S to (1+N/K)*S notes: To repair a file, Tahoe-LAFS downloads the file, and generates/uploads missing shares in the same way as when it initially uploads the file. } Context: [NEWS: update entry for removal of Mac and Windows apps. refs #1282 david-sarah@jacaranda.org**20101226042245 Ignore-this: c8099bc6e8235718d042c9a13c1e2425 ] [Move dependency imports from windows/depends.py (which has gone away) into src/allmydata/windows/tahoesvc.py. Also fix a pyflakes warning, and change the service display name from 'Allmydata Tahoe Node' to 'Tahoe-LAFS node'. refs #1282 david-sarah@jacaranda.org**20101226042100 Ignore-this: ee45f324934e1251380206dbee6346d0 ] [Remove unmaintained Windows GUI app, except for windows/tahoesvc.py which is moved to src/allmydata/windows. refs #1282 david-sarah@jacaranda.org**20101226040237 Ignore-this: cae37b6622a7dd5940acc7d3e6a98b90 ] [Remove the Makefile targets relating to the Mac GUI app. refs #1282 david-sarah@jacaranda.org**20101226025859 Ignore-this: 75303be783974b41138744ec62b07965 ] [NEWS: remove unmaintained Mac GUI app. refs #1282 david-sarah@jacaranda.org**20101226020858 Ignore-this: 40474a07f4a550b48563d35350be7ab5 ] [Remove unmaintained Mac GUI app. fixes #1282 david-sarah@jacaranda.org**20101226020508 Ignore-this: b3613bf1abfd284d542bf7c753ec557a ] [Remove src/allmydata/util/find_exe.py which is no longer used. fixes #1150 david-sarah@jacaranda.org**20101226023206 Ignore-this: 7436c9b53bf210aed34a1a973cd9cace ] [status_web_pages_review.darcs.patch freestorm77@gmail.com**20110102034214 Ignore-this: 29f1ecb36177f10f3f846b3d56b313b2 I make some changes on status web pages status.xhtml: - Delete unused webform_css link - Align tables on the left tahoe-css: - Do some minor changes on code synthax - changes table.status-download-events style to look like other tables status.py: - Align table on the left - Changes table header - Add heading tags - Modify google api graph: add image border, calculate height to feet data signed-off-by: zooko@zooko.com fixes #1219 ] [test_storage.py: fix a pyflakes unused import warning. david-sarah@jacaranda.org**20101231220756 Ignore-this: df08231540cb7dff9d2b038e47ab30ee ] [test_storage.py: leave at least 512 MiB free when running test_large_share. refs #1195 david-sarah@jacaranda.org**20101231203215 Ignore-this: b2144c0341c3452b5d4ba219e284ea0e ] [storage: use fileutil's version of get_disk_stats() and get_available_space(), use mockery/fakery in tests, enable large share test on platforms with sparse files and if > 4 GiB of disk space is currently available zooko@zooko.com**20100910173629 Ignore-this: 1304f1164c661de6d5304f993eb9b27b ] [fileutil: copy in the get_disk_stats() and get_available_space() functions from storage/server.py zooko@zooko.com**20100910173520 Ignore-this: 8b15569715f710f4fc5092f7ca109253 ] [Update foolscap version requirement to 0.6.0, to address http://foolscap.lothar.com/trac/ticket/167 david-sarah@jacaranda.org**20101231060039 Ignore-this: 98d2b8086a1a500b9f4565bca5a3810 ] [docs/webapi.rst: typos. david-sarah@jacaranda.org**20101230034422 Ignore-this: d1f5166d72cc711f7e0d9981eac9105e ] [docs/webapi.rst: capitalization, formatting of section on URL character encoding, and a correction about Internet Explorer. david-sarah@jacaranda.org**20101230034049 Ignore-this: b3b9819d2fb264b4cdc5c8afd4e8c48d ] [docs: corrections and clarifications. david-sarah@jacaranda.org**20101227051056 Ignore-this: e33202858c7644c58f3f924b164294b6 ] [docs: more formatting cleanups and corrections. Spell webapi and wapi as web-API. david-sarah@jacaranda.org**20101227050533 Ignore-this: 18b23cbfb780df585d8a722a1ec63e94 ] [docs/debian.rst: bring description of building dependencies from source up-to-date, and change hostname from allmydata.com to tahoe-lafs.org. david-sarah@jacaranda.org**20101212222912 Ignore-this: f38462afc88b4475195610385a28391c ] [docs/architecture.rst: correct rst syntax. david-sarah@jacaranda.org**20101212202003 Ignore-this: 3fbe12feb28bec6f1c63aedbc79aad21 ] [docs/architecture.rst: formatting. david-sarah@jacaranda.org**20101212201719 Ignore-this: 305fa5dfc2939355eaf6d0d2161eb1ff ] [docs: linkification, wording improvements. david-sarah@jacaranda.org**20101212201234 Ignore-this: 4e67287f527a8bc728cfbd93255d2aae ] [docs: formatting. david-sarah@jacaranda.org**20101212201115 Ignore-this: 2e0ed394ac7726651d3a4f2c4b0d3798 ] [docs/configuration.rst: more formatting tweaks; which -> that. david-sarah@jacaranda.org**20101212195522 Ignore-this: a7becb7021854ca5a90edd892b36fdd7 ] [docs/configuration.rst: more changes to formatting. david-sarah@jacaranda.org**20101212194511 Ignore-this: 491aac33e5f5268d224359f1447d10be ] [docs/configuration.rst: changes to formatting (mainly putting commands and filenames in monospace). david-sarah@jacaranda.org**20101212181828 Ignore-this: 8a1480e2d5f43bee678476424615b50f ] [scripts/backupdb.py: more accurate comment about path field. david-sarah@jacaranda.org**20101212170320 Ignore-this: 50e47a2228a85207bbcd188a78a0d4e6 ] [scripts/cli.py: fix missing 'put' in usage example for 'tahoe put'. david-sarah@jacaranda.org**20101212170207 Ignore-this: 2cbadf066fff611fc03d3c0ff97ce6ec ] [docs/frontends/CLI.rst: changes to formatting (mainly putting commands and filenames in monospace), and to command syntax to reflect that DIRCAP/... is accepted. Clarify the syntax of 'tahoe put' and other minor corrections. Tahoe -> Tahoe-LAFS. david-sarah@jacaranda.org**20101212165800 Ignore-this: a123ef6b564aa8624d1e79c97068ea12 ] [docs/frontends/CLI.rst: Unicode arguments to 'tahoe' work on Windows as of v1.7.1. david-sarah@jacaranda.org**20101212063740 Ignore-this: 3977a99dfa86ac33a44171deaf43aaab ] [docs/known_issues.rst: fix title and linkify another URL. refs #1225 david-sarah@jacaranda.org**20101212062817 Ignore-this: cc91287f7fb51c23440b3d2fe79c449c ] [docs/known_issues.rst: fix an external link. refs #1225 david-sarah@jacaranda.org**20101212062435 Ignore-this: b8cbf12f353131756c358965c48060ec ] [Fix a link from uri.rst to dirnodes.rst. refs #1225 david-sarah@jacaranda.org**20101212054502 Ignore-this: af6205299f5c9a33229cab259c00f9d5 ] [Fix a link from webapi.rst to FTP-and-SFTP.rst. refs #1225 david-sarah@jacaranda.org**20101212053435 Ignore-this: 2b9f88678c3447ea860d6b61e8799858 ] [More specific hyperlink to architecture.rst from helper.rst. refs #1225 david-sarah@jacaranda.org**20101212052607 Ignore-this: 50424c768fca481252fabf58424852dc ] [Update hyperlinks between docs, and linkify some external references. refs #1225 david-sarah@jacaranda.org**20101212051459 Ignore-this: cd43a4c3d3de1f832abfa88d5fc4ace1 ] [docs/specifications/dirnodes.rst: fix references to mutable.rst. refs #1225 david-sarah@jacaranda.org**20101212012720 Ignore-this: 6819b4b4e06e947ee48b365e840db37d ] [docs/specifications/mutable.rst: correct the magic string for v1 mutable containers. refs #1225 david-sarah@jacaranda.org**20101212011400 Ignore-this: 99a5fcdd40cef83dbb08f323f6cdaaca ] [Move .txt files in docs/frontends and docs/specifications to .rst. refs #1225 david-sarah@jacaranda.org**20101212010251 Ignore-this: 8796d35d928370f7dc6ad2dafdc1c0fe ] [Convert docs/frontends and docs/specifications to reStructuredText format (not including file moves). david-sarah@jacaranda.org**20101212004632 Ignore-this: e3ceb2d832d73875abe48624ddbb5622 ] [scripts/cli.py: remove the disclaimer in the help for 'tahoe cp' that it does not handle non-ASCII filenames well. (At least, we intend to handle them.) david-sarah@jacaranda.org**20101130002145 Ignore-this: 94c003efaa20b9eb4a83503d79844ca ] [relnotes.txt: fifth -> sixth labor-of-love release zooko@zooko.com**20101129045647 Ignore-this: 21c245015268b38916e3a138d256c09d ] [Makefile: BB_BRANCH is set to the empty string for trunk, not the string 'trunk'. david-sarah@jacaranda.org**20101128233512 Ignore-this: 5a7ef8eb10475636d21b91e25b56c369 ] [relnotes.txt: eleventh -> twelfth release. david-sarah@jacaranda.org**20101128223321 Ignore-this: 1e26410156a665271c1170803dea2c0d ] [relnotes.tst: point to known_issues.rst, not known_issues.txt. david-sarah@jacaranda.org**20101128222918 Ignore-this: 60194eb4544cac446fe4f60b3e34b887 ] [quickstart.html: fix link to point to allmydata-tahoe-1.8.1.zip. david-sarah@jacaranda.org**20101128221728 Ignore-this: 7b3ee86f8256aa12f5d862f689f3ee29 ] [TAG allmydata-tahoe-1.8.1 david-sarah@jacaranda.org**20101128212336 Ignore-this: 9c18bdeaef4822f590d2a0d879e00621 ] Patch bundle hash: 4d8ada6ea9d68920ae22f7d34a651554b45b3d07