diff --git a/Performance.md b/Performance.md new file mode 100644 index 0000000..5a6103f --- /dev/null +++ b/Performance.md @@ -0,0 +1,72 @@ +Some basic notes on performance: + +## Memory Footprint + +The [MemoryFootprint](MemoryFootprint) page has more specific information. + +The +[graph](http://allmydata.org/tahoe-figleaf-graph/hanford.allmydata.com-tahoe_memstats.html) +munin shows our static memory footprint (starting a node but not doing +anything with it) to be about 24MB. Uploading one file at a time gets the +node to about 29MB. (we only process one segment at a time, so peak memory +consumption occurs when the file is a few MB in size and does not grow beyond +that). Uploading multiple files at once would increase this. + +## Network Speed + +### test results + +Using a 3-server testnet in colo and an uploading node at home (on a DSL line +that gets about 150kBps upstream and has a 14ms ping time to colo) using +0.5.1-34 takes 820ms-900ms per 1kB file uploaded (80-90s for 100 files, 819s +for 1000 files). + +'scp' of 3.3kB files (simulating expansion) takes 8.3s for 100 files and 79s +for 1000 files, 80ms each. + +Doing the same uploads locally on my laptop (both the uploading node and the +storage nodes are local) takes 46s for 100 1kB files and 369s for 1000 files. + + + +### Roundtrips + +The 0.5.1 release requires about 9 roundtrips for each share it uploads. The +upload algorithm sends data to all shareholders in parallel, but these 9 +phases are done sequentially. The phases are: + + 1. allocate_buckets + 1. send_subshare (once per segment) + 1. send_plaintext_hash_tree + 1. send_crypttext_hash_tree + 1. send_subshare_hash_trees + 1. send_share_hash_trees + 1. send_UEB + 1. close + 1. dirnode update + +We need to keep the send_subshare calls sequential (to keep our memory +footprint down), and we need a barrier between the close and the dirnode +update (for robustness and clarity), but the others could be pipelined. +9*14ms=126ms, which accounts for about 15% of the measured upload time. + +## Storage Servers + +ext3 (on tahoebs1) refuses to create more than 32000 subdirectories in a +single parent directory. In 0.5.1, this appears as a limit on the number of +buckets (one per storage index) that any [StorageServer](StorageServer) can hold. A simple +nested directory structure will work around this.. the following code would +let us manage 33.5G shares: + +``` + from idlib import b2a + os.path.join(b2a(si[:2]), b2a(si[2:4]), b2a(si)) +``` + +This limitation is independent of problems of memory use and lookup speed. +Once the number of buckets is large, the filesystem may take a long time (and +multiple disk seeks) to determine if a bucket is present or not. The +provisioning page suggests how frequently these lookups will take place, and +we can compare this against the time each one will take to see if we can keep +up or not. If and when necessary, we'll move to a more sophisticated storage +server design (perhaps with a database to locate shares).