tahoe-lafs/trac-2024-07-25

Performance hit by a million cuts #4079

New issue

Open

opened 2023-12-08 15:34:38 +00:00 by itamarst · 0 comments

itamarst commented

2023-12-08 15:34:38 +00:00

Owner

Tahoe uses a decent amount of CPU.

Profiling shows just lots and lots and lots of tiny little actions. Question is, is there any obvious culprint beyond "it's Python", and if so, can we do anything about it.

Things already investigated:

I measured overhead of using plain Deferreds. I'd estimate it's maybe 1% of CPU usage? So probably not that. In future Python versions this might become more significant depending how good the theoretical proposed JIT ends up being, as callbacks may prevent optimizations at that level. But that's not quite the same thing as overhead.

Things to investigate:

inlineCallbacks probably has overhead too. I didn't measure it yet.
HTTP overhead. Twisted's HTTP parser probably isn't the fastest.
Chunk size. If chunking is done in too small of a unit, this increases all the overhead. Larger chunk sizes will help. See #4080

Tahoe uses a decent amount of CPU. Profiling shows just lots and lots and lots of tiny little actions. Question is, is there any obvious culprint beyond "it's Python", and if so, can we do anything about it. Things already investigated: * I measured overhead of using plain Deferreds. I'd estimate it's maybe 1% of CPU usage? So probably not that. In future Python versions this might become more significant depending how good the theoretical proposed JIT ends up being, as callbacks may prevent optimizations at that level. But that's not quite the same thing as overhead. Things to investigate: * inlineCallbacks probably has overhead too. I didn't measure it yet. * HTTP overhead. Twisted's HTTP parser probably isn't the fastest. * Chunk size. If chunking is done in too small of a unit, this increases all the overhead. Larger chunk sizes will help. See #4080

tahoe-lafs added the

labels 2023-12-08 15:34:38 +00:00

tahoe-lafs added this to the Performance and Benchmarking milestone 2023-12-08 15:34:38 +00:00

Sign in to join this conversation.

No labels

LeastAuthority.com automation

cannot reproduce

code-frontend-cli

code-frontend-ftp-sftp

code-frontend-magic-folder

code-frontend-web

code-peerselection

dev-infrastructure

somebody else's problem

was already fixed

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: tahoe-lafs/trac-2024-07-25#4079

No description provided.