tahoe-lafs/trac-2024-07-25

measure the effect of a more memory-local zfec variant on a webapi server #236

New issue

Open

opened 2007-12-18 22:36:57 +00:00 by zooko · 1 comment

zooko commented

2007-12-18 22:36:57 +00:00

Owner

The preliminary results of the load tests ("the webapi servers are the busy points. Each webapi server is on its own machine. Each one is consuming anywhere from 30% to 70% CPU.") suggest that zfec might be a bottleneck in a complete system (grid plus webapi servers). I have some patches to zfec which constrain its use of memory to be more localized in order to fit into smaller caches.

I will measure whethermany zfec processes running in parallel on one server-sized machine (an Athlon64 2.4 GHz, 512 KiB L2 cache, 4 GiB DDR2 RAM) benefit or suffer from this locality optimization.

The preliminary results of the load tests ("the webapi servers are the busy points. Each webapi server is on its own machine. Each one is consuming anywhere from 30% to 70% CPU.") suggest that zfec might be a bottleneck in a complete system (grid plus webapi servers). I have some patches to zfec which constrain its use of memory to be more localized in order to fit into smaller caches. I will measure whethermany zfec processes running in parallel on one server-sized machine (an Athlon64 2.4 GHz, 512 KiB L2 cache, 4 GiB DDR2 RAM) benefit or suffer from this locality optimization.

tahoe-lafs added the

labels 2007-12-18 22:36:57 +00:00

tahoe-lafs added this to the eventually milestone 2007-12-18 22:36:57 +00:00

warner commented

2008-03-08 00:53:51 +00:00

Author

Owner

FYI, the new upload/download status timing pages tell me that for a 5MB
3-of-10 download, zfec is running at 89MBps, whereas AES is running at
5.6MBps. So at least for one-at-a-time downloads, AES is far more of a
bottleneck than zfec.

For a machine that is in colo, a 2MB download runs zfec at 587MBps and AES at
12MBps. A 16MB download runs zfec at 320MBps and AES at 12MBps.

FYI, the new upload/download status timing pages tell me that for a 5MB 3-of-10 download, zfec is running at 89MBps, whereas AES is running at 5.6MBps. So at least for one-at-a-time downloads, AES is far more of a bottleneck than zfec. For a machine that is in colo, a 2MB download runs zfec at 587MBps and AES at 12MBps. A 16MB download runs zfec at 320MBps and AES at 12MBps.

tahoe-lafs modified the milestone from eventually to undecided

2008-06-01 21:05:08 +00:00

tahoe-lafs added

and removed

labels 2009-12-13 05:04:01 +00:00

Sign in to join this conversation.

No labels

LeastAuthority.com automation

cannot reproduce

code-frontend-cli

code-frontend-ftp-sftp

code-frontend-magic-folder

code-frontend-web

code-peerselection

dev-infrastructure

somebody else's problem

was already fixed

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: tahoe-lafs/trac-2024-07-25#236

No description provided.