tahoe-lafs/trac-2024-07-25
2
0
Issues 1.1k Wiki Activity

fix MDMF tests to include proper 'k' graph 

[Imported from Trac: page Performance/Sep2011, version 4]
warner 2011-09-27 03:20:30 +00:00
parent ab1a5dd059
commit fd48ebb245
1 changed files with 11 additions and 14 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

25
Performance/Sep2011.md

@ -23,15 +23,10 @@ significantly with larger k.
## MDMF (trunk)
MDMF is fast! Trunk downloads 1MB/10MB/100MB MDMF files at around 4MBps.
~~The download speed seems unaffected by k (from 1 to 60)~~. Partial
reads take the expected amount of time: O(data_read), slightly quantized
near the 128KiB segment size.
Download speed drops linearly with k, k=60 takes roughly 2x more time
than k=30. Partial reads take the expected amount of time: O(data_read),
slightly quantized near the 128KiB segment size.
(update 25-Sep: the test harness appears to have been flawed, and all
MDMF files were actually k=3. A new test is in the works to get proper
data on how MDMF read speed varies with k).
* MDMF read versus k, 100MB [MDMF-100MB-vs-k.png](../raw/attachments/Performance/Sep2011/MDMF-100MB-vs-k.png) (timing5.out)
* MDMF partial reads, 100MB [MDMF-100MB-partial.png](../raw/attachments/Performance/Sep2011/MDMF-100MB-partial.png) (timing6.out)
* MDMF partial reads, 1MB [MDMF-1MB-partial.png](../raw/attachments/Performance/Sep2011/MDMF-1MB-partial.png) (timing6.out)
@ -43,14 +38,14 @@ connections. Compared to the old-downloader (in 1.7.1), downloads tend to be
about 3x slower. (note that this delay is probably completely hidden on slow
WAN links, and it's only the fast LAN connections of the atlasperf1 grid that
exposes the delay). In addition, both old and new downloaders suffer from a
linear slowdown as k increases. On the new-downloader, k=60 takes roughly 3x
more time than k=15. Trunk contains a fix for #1268 that might improve speeds
linear slowdown as k increases. On the new-downloader, k=60 takes roughly 2x
more time than k=30. Trunk contains a fix for #1268 that might improve speeds
by 5% compared to 1.8.2. Partial reads take the expected amount of time,
although the segsize-quantization was nowhere nearly as clear as with MDMF.
* CHK (1.7.1/1.8.2/trunk) read versus k, 1MB [CHK-1MB-vs-k.png](../raw/attachments/Performance/Sep2011/CHK-1MB-vs-k.png) (t4/t/t3)
* CHK (1.7.1/1.8.2/trunk) read versus k, 100MB [CHK-100MB-vs-k.png](../raw/attachments/Performance/Sep2011/CHK-100MB-vs-k.png) (t4/t/t3)
* CHK (1.7.1/1.8.2/trunk) and MDMF (trunk) read versus k, 100MB [CHKMDMF-100MB-vs-k.png](../raw/attachments/Performance/Sep2011/CHKMDMF-100MB-vs-k.png) (t4/t/t3/t8)
* CHK (1.8.2) read versus segsize, 1MB [CHK-1MB-vs-segsize.png](../raw/attachments/Performance/Sep2011/CHK-1MB-vs-segsize.png) (t2)
* CHK (1.8.2) read versus segsize, 100MB [CHK-100MB-vs-segsize.png](../raw/attachments/Performance/Sep2011/CHK-100MB-vs-segsize.png) (t2)
@ -73,17 +68,19 @@ Likely fixes include:
> old-downloader cheated by reading the whole hash tree at once, violating
> the memory footprint goals (requires O(numsegments) memory), but probably
> tolerable unless the filesize is really large.
* note that 1.7.1 "cheats" with both the crypttext_hash_tree and the
block_hash_tree. MDMF in trunk cheats with the block_hash_tree and
does not have a crypttext_hash_tree
* encourage use of larger segsize for large files (at the expense of
> alacrity)
* use unencrypted HTTP for share reads
readv() is the least-work/most-promising, ~~since MDMF has readv() and
achieves high speeds. First step is to do whatever MDMF is doing~~
readv() is the least-work/most-promising, since MDMF has readv() and
is faster than trunk, although not as fast as 1.7.1.
## Future Tests
* measure alacrity: ask for random single byte, measure elapsed time
* measure partial-read speeds for CHK
* measure SDMF/MDMF modification times
* measure upload times
* using existing data as a baseline, detect outliers in real-time during the