tahoe-lafs/trac-2024-07-25
2
0
Issues 1.1k Wiki Activity

fix MDMF tests to include proper 'k' graph 

[Imported from Trac: page Performance/Sep2011, version 4]
warner 2011-09-27 03:20:30 +00:00
parent ab1a5dd059
commit fd48ebb245
1 changed files with 11 additions and 14 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

25
Performance/Sep2011.md

@ -23,15 +23,10 @@ significantly with larger k.
## MDMF (trunk) ## MDMF (trunk)
MDMF is fast! Trunk downloads 1MB/10MB/100MB MDMF files at around 4MBps. MDMF is fast! Trunk downloads 1MB/10MB/100MB MDMF files at around 4MBps.
~~The download speed seems unaffected by k (from 1 to 60)~~. Partial Download speed drops linearly with k, k=60 takes roughly 2x more time
reads take the expected amount of time: O(data_read), slightly quantized than k=30. Partial reads take the expected amount of time: O(data_read),
near the 128KiB segment size. slightly quantized near the 128KiB segment size.
(update 25-Sep: the test harness appears to have been flawed, and all
MDMF files were actually k=3. A new test is in the works to get proper
data on how MDMF read speed varies with k).
* MDMF read versus k, 100MB [MDMF-100MB-vs-k.png](../raw/attachments/Performance/Sep2011/MDMF-100MB-vs-k.png) (timing5.out)
* MDMF partial reads, 100MB [MDMF-100MB-partial.png](../raw/attachments/Performance/Sep2011/MDMF-100MB-partial.png) (timing6.out) * MDMF partial reads, 100MB [MDMF-100MB-partial.png](../raw/attachments/Performance/Sep2011/MDMF-100MB-partial.png) (timing6.out)
* MDMF partial reads, 1MB [MDMF-1MB-partial.png](../raw/attachments/Performance/Sep2011/MDMF-1MB-partial.png) (timing6.out) * MDMF partial reads, 1MB [MDMF-1MB-partial.png](../raw/attachments/Performance/Sep2011/MDMF-1MB-partial.png) (timing6.out)
@ -43,14 +38,14 @@ connections. Compared to the old-downloader (in 1.7.1), downloads tend to be
about 3x slower. (note that this delay is probably completely hidden on slow about 3x slower. (note that this delay is probably completely hidden on slow
WAN links, and it's only the fast LAN connections of the atlasperf1 grid that WAN links, and it's only the fast LAN connections of the atlasperf1 grid that
exposes the delay). In addition, both old and new downloaders suffer from a exposes the delay). In addition, both old and new downloaders suffer from a
linear slowdown as k increases. On the new-downloader, k=60 takes roughly 3x linear slowdown as k increases. On the new-downloader, k=60 takes roughly 2x
more time than k=15. Trunk contains a fix for #1268 that might improve speeds more time than k=30. Trunk contains a fix for #1268 that might improve speeds
by 5% compared to 1.8.2. Partial reads take the expected amount of time, by 5% compared to 1.8.2. Partial reads take the expected amount of time,
although the segsize-quantization was nowhere nearly as clear as with MDMF. although the segsize-quantization was nowhere nearly as clear as with MDMF.
* CHK (1.7.1/1.8.2/trunk) read versus k, 1MB [CHK-1MB-vs-k.png](../raw/attachments/Performance/Sep2011/CHK-1MB-vs-k.png) (t4/t/t3) * CHK (1.7.1/1.8.2/trunk) read versus k, 1MB [CHK-1MB-vs-k.png](../raw/attachments/Performance/Sep2011/CHK-1MB-vs-k.png) (t4/t/t3)
* CHK (1.7.1/1.8.2/trunk) read versus k, 100MB [CHK-100MB-vs-k.png](../raw/attachments/Performance/Sep2011/CHK-100MB-vs-k.png) (t4/t/t3) * CHK (1.7.1/1.8.2/trunk) and MDMF (trunk) read versus k, 100MB [CHKMDMF-100MB-vs-k.png](../raw/attachments/Performance/Sep2011/CHKMDMF-100MB-vs-k.png) (t4/t/t3/t8)
* CHK (1.8.2) read versus segsize, 1MB [CHK-1MB-vs-segsize.png](../raw/attachments/Performance/Sep2011/CHK-1MB-vs-segsize.png) (t2) * CHK (1.8.2) read versus segsize, 1MB [CHK-1MB-vs-segsize.png](../raw/attachments/Performance/Sep2011/CHK-1MB-vs-segsize.png) (t2)
* CHK (1.8.2) read versus segsize, 100MB [CHK-100MB-vs-segsize.png](../raw/attachments/Performance/Sep2011/CHK-100MB-vs-segsize.png) (t2) * CHK (1.8.2) read versus segsize, 100MB [CHK-100MB-vs-segsize.png](../raw/attachments/Performance/Sep2011/CHK-100MB-vs-segsize.png) (t2)
@ -73,17 +68,19 @@ Likely fixes include:
> old-downloader cheated by reading the whole hash tree at once, violating > old-downloader cheated by reading the whole hash tree at once, violating
> the memory footprint goals (requires O(numsegments) memory), but probably > the memory footprint goals (requires O(numsegments) memory), but probably
> tolerable unless the filesize is really large. > tolerable unless the filesize is really large.
* note that 1.7.1 "cheats" with both the crypttext_hash_tree and the
block_hash_tree. MDMF in trunk cheats with the block_hash_tree and
does not have a crypttext_hash_tree
* encourage use of larger segsize for large files (at the expense of * encourage use of larger segsize for large files (at the expense of
> alacrity) > alacrity)
* use unencrypted HTTP for share reads * use unencrypted HTTP for share reads
readv() is the least-work/most-promising, ~~since MDMF has readv() and readv() is the least-work/most-promising, since MDMF has readv() and
achieves high speeds. First step is to do whatever MDMF is doing~~ is faster than trunk, although not as fast as 1.7.1.
## Future Tests ## Future Tests
* measure alacrity: ask for random single byte, measure elapsed time * measure alacrity: ask for random single byte, measure elapsed time
* measure partial-read speeds for CHK
* measure SDMF/MDMF modification times * measure SDMF/MDMF modification times
* measure upload times * measure upload times
* using existing data as a baseline, detect outliers in real-time during the * using existing data as a baseline, detect outliers in real-time during the