diff --git a/Performance/Sep2011.md b/Performance/Sep2011.md index 8afa3d9..bac69f5 100644 --- a/Performance/Sep2011.md +++ b/Performance/Sep2011.md @@ -23,15 +23,10 @@ significantly with larger k. ## MDMF (trunk) MDMF is fast! Trunk downloads 1MB/10MB/100MB MDMF files at around 4MBps. -~~The download speed seems unaffected by k (from 1 to 60)~~. Partial -reads take the expected amount of time: O(data_read), slightly quantized -near the 128KiB segment size. +Download speed drops linearly with k, k=60 takes roughly 2x more time +than k=30. Partial reads take the expected amount of time: O(data_read), +slightly quantized near the 128KiB segment size. -(update 25-Sep: the test harness appears to have been flawed, and all -MDMF files were actually k=3. A new test is in the works to get proper -data on how MDMF read speed varies with k). - -* MDMF read versus k, 100MB [MDMF-100MB-vs-k.png](../raw/attachments/Performance/Sep2011/MDMF-100MB-vs-k.png) (timing5.out) * MDMF partial reads, 100MB [MDMF-100MB-partial.png](../raw/attachments/Performance/Sep2011/MDMF-100MB-partial.png) (timing6.out) * MDMF partial reads, 1MB [MDMF-1MB-partial.png](../raw/attachments/Performance/Sep2011/MDMF-1MB-partial.png) (timing6.out) @@ -43,14 +38,14 @@ connections. Compared to the old-downloader (in 1.7.1), downloads tend to be about 3x slower. (note that this delay is probably completely hidden on slow WAN links, and it's only the fast LAN connections of the atlasperf1 grid that exposes the delay). In addition, both old and new downloaders suffer from a -linear slowdown as k increases. On the new-downloader, k=60 takes roughly 3x -more time than k=15. Trunk contains a fix for #1268 that might improve speeds +linear slowdown as k increases. On the new-downloader, k=60 takes roughly 2x +more time than k=30. Trunk contains a fix for #1268 that might improve speeds by 5% compared to 1.8.2. Partial reads take the expected amount of time, although the segsize-quantization was nowhere nearly as clear as with MDMF. * CHK (1.7.1/1.8.2/trunk) read versus k, 1MB [CHK-1MB-vs-k.png](../raw/attachments/Performance/Sep2011/CHK-1MB-vs-k.png) (t4/t/t3) -* CHK (1.7.1/1.8.2/trunk) read versus k, 100MB [CHK-100MB-vs-k.png](../raw/attachments/Performance/Sep2011/CHK-100MB-vs-k.png) (t4/t/t3) +* CHK (1.7.1/1.8.2/trunk) and MDMF (trunk) read versus k, 100MB [CHKMDMF-100MB-vs-k.png](../raw/attachments/Performance/Sep2011/CHKMDMF-100MB-vs-k.png) (t4/t/t3/t8) * CHK (1.8.2) read versus segsize, 1MB [CHK-1MB-vs-segsize.png](../raw/attachments/Performance/Sep2011/CHK-1MB-vs-segsize.png) (t2) * CHK (1.8.2) read versus segsize, 100MB [CHK-100MB-vs-segsize.png](../raw/attachments/Performance/Sep2011/CHK-100MB-vs-segsize.png) (t2) @@ -73,17 +68,19 @@ Likely fixes include: > old-downloader cheated by reading the whole hash tree at once, violating > the memory footprint goals (requires O(numsegments) memory), but probably > tolerable unless the filesize is really large. + * note that 1.7.1 "cheats" with both the crypttext_hash_tree and the + block_hash_tree. MDMF in trunk cheats with the block_hash_tree and + does not have a crypttext_hash_tree * encourage use of larger segsize for large files (at the expense of > alacrity) * use unencrypted HTTP for share reads -readv() is the least-work/most-promising, ~~since MDMF has readv() and -achieves high speeds. First step is to do whatever MDMF is doing~~ +readv() is the least-work/most-promising, since MDMF has readv() and +is faster than trunk, although not as fast as 1.7.1. ## Future Tests * measure alacrity: ask for random single byte, measure elapsed time -* measure partial-read speeds for CHK * measure SDMF/MDMF modification times * measure upload times * using existing data as a baseline, detect outliers in real-time during the