tahoe-lafs/trac-2024-07-25

speed up directories more #752

New issue

Open

opened 2009-07-09 04:28:43 +00:00 by zooko · 2 comments

zooko commented

2009-07-09 04:28:43 +00:00

Owner

Even after changeset:efafcfb91a09b4de, changeset:903005a52830ba96, changeset:c0d1e7deaec145d6, and the not-yet-committed patch to generate IVs with hashes of writecaps (#750) and a not-yet-committed patch to run with -OO optimization mode, directories are still too slow. This set of five patches speeds directories up significantly:

on my amd64 workstation yukyuk:

before these five optimization patches: unpack and repack a 64-entry directory: 0.10 seconds, a 1024-entry directory: 1.75s, a 4096-entry directory: 7.75s, a 65536-entry directory: 441.65s.

after these five optimization patches: unpack and repack a 64-entry directory: 0.08s, a 1024-entry directory: 1.3s, a 4096-entry directory: 4.95s, a 65536-entry directory: 115.4s

on Zandr's 266 MHz armel box:

before these five optimization patches: unpack and repack a 64-entry directory: 2.19 seconds, a 1024-entry directory: 35.48s, a 4096-entry directory: 183.05s (65536-entry directory not measured).

after these five optimization patches: unpack and repack a 64-entry directory: 2.16s, a 1024-entry directory: 32.55s, a 4096-entry directory: 124.28s (a 65536-entry directory not measured).

So this is a significant speed-up. Except wait -- 5s to open a directory with 4096 entries on an amd64 workstation, or 2 minutes to open it on a 266 MHz armel NAS? We can do better than that!

Even after changeset:efafcfb91a09b4de, changeset:903005a52830ba96, changeset:c0d1e7deaec145d6, *and* the not-yet-committed patch to generate IVs with hashes of writecaps (#750) *and* a not-yet-committed patch to run with `-OO` optimization mode, directories are still too slow. This set of five patches speeds directories up significantly: on my amd64 workstation yukyuk: before these five optimization patches: unpack and repack a 64-entry directory: 0.10 seconds, a 1024-entry directory: 1.75s, a 4096-entry directory: 7.75s, a 65536-entry directory: 441.65s. after these five optimization patches: unpack and repack a 64-entry directory: 0.08s, a 1024-entry directory: 1.3s, a 4096-entry directory: 4.95s, a 65536-entry directory: 115.4s on Zandr's 266 MHz armel box: before these five optimization patches: unpack and repack a 64-entry directory: 2.19 seconds, a 1024-entry directory: 35.48s, a 4096-entry directory: 183.05s (65536-entry directory not measured). after these five optimization patches: unpack and repack a 64-entry directory: 2.16s, a 1024-entry directory: 32.55s, a 4096-entry directory: 124.28s (a 65536-entry directory not measured). So this is a significant speed-up. Except wait -- 5s to open a directory with 4096 entries on an amd64 workstation, or 2 minutes to open it on a 266 MHz armel NAS? We can do better than that!

tahoe-lafs added the

labels 2009-07-09 04:28:43 +00:00

tahoe-lafs added this to the undecided milestone 2009-07-09 04:28:43 +00:00

davidsarah commented

2009-12-13 05:08:09 +00:00

Author

Owner

Er, where are the patches?

zooko commented

2009-12-13 05:51:56 +00:00

Author

Owner

Sorry for the confusion. This sounds like I have five more patches to offer, but actually the five patches I meant were changeset:efafcfb91a09b4de, changeset:d71adaf1ca1f8b7d (not as I originally posted changeset:903005a52830ba96), changeset:c0d1e7deaec145d6, changeset:c1d5717cf0ecd68f (from #750), and the patch to use python optimization mode was never committed. I don't know if I still have that patch around somewhere and I don't remember how I did it.