tahoe-lafs/trac

very large memory usage when adding to large directories #379

New issue

Closed

opened 2008-04-12 05:44:48 +00:00 by warner · 3 comments

warner commented

2008-04-12 05:44:48 +00:00

We've seen the webapi servers that are participating in the allmydata
migration effort have memory usage spikes that jump to an incredible 3.1GB
over the past two days. The CPU usage goes to 100% for about 5 minutes while
this happens. This is occurring about once every 15 minutes, causing the
migration process to run significantly slower.

We've isolated the problem to the following items:

our RemoteInterface for mutable share writes (i.e.
slot_testv_and_readv_and_writev) declares a maximum share data size of 1
MiB, i.e. 1048576 bytes, but the maximum size of a mutable file (3.5 MB)
leads to shares that can exceed this. This occurs for directories of
about 9600 entries
Adding new children to such a directory causes an outbound foolscap
Violation, because the share-size constraint is being violated
the Violation is a Failure object, and Twisted (specifically
maybeDeferred) causes the cleanFailure method to be run, which turns
the entire stack trace (including local variables for each frame) into
a bunch of strings
the 1MB-ish shares are present in the argument list for about a dozen
stack frames (in the locals). Every one of these strings gets repr'ed,
and since they're binary data, each repr'ing gets a 4x expansion.
the memory usage is coming from the dozens of copies of this expanded
share data inside the Failure's .stack and .frames attributes.
in addition, the way we're using DeferredList fails to catch
errors in callRemote: servers fail to accept shares, but we don't
notice.

What we need to do to address this:

remove the size constraint on ShareData. This will stop these
particular Violations from happening
Fix twisted ticket Twisted#2466,
somehow, then require the resulting release of twisted.
add fireOnOneErrback=True to the uses of DeferredList in mutable.py,
to properly catch exceptions

Any exception in share transmission is likely to consume this sort of memory.
Workarounds we could conceivably implement before the Twisted problem gets
fixed:

avoid holding large strings in local variables: pass them as instance
attributes instead, be careful to delete them from locals as soon as
possible
hold large strings in some sort of object (with a repr() that doesn't
show the whole thing) instead of a real string, and teach Foolscap
(via ISliceable) to serialize them as strings.
have Foolscap trim Failures in callRemote somehow
try to break up the stacks by using fireEventually in more places
rewrite Foolscap's outbound constraint checking to not do everything
on the same stack

Our current plan is to fix the constraint and then hope that we don't trigger
other failures while we contribute to the Twisted ticket and wait for a new
release. Later refactorings of share management will probably put more data
in instance attributes rather than being passed through method arguments.

If necessary, we can ship tahoe with a patched version of Twisted.

We've seen the webapi servers that are participating in the allmydata migration effort have memory usage spikes that jump to an incredible 3.1GB over the past two days. The CPU usage goes to 100% for about 5 minutes while this happens. This is occurring about once every 15 minutes, causing the migration process to run significantly slower. We've isolated the problem to the following items: * our RemoteInterface for mutable share writes (i.e. slot_testv_and_readv_and_writev) declares a maximum share data size of 1 MiB, i.e. 1048576 bytes, but the maximum size of a mutable file (3.5 MB) leads to shares that can exceed this. This occurs for directories of about 9600 entries * Adding new children to such a directory causes an outbound foolscap Violation, because the share-size constraint is being violated * the Violation is a Failure object, and Twisted (specifically maybeDeferred) causes the cleanFailure method to be run, which turns the entire stack trace (including local variables for each frame) into a bunch of strings * the 1MB-ish shares are present in the argument list for about a dozen stack frames (in the locals). Every one of these strings gets repr'ed, and since they're binary data, each repr'ing gets a 4x expansion. * the memory usage is coming from the dozens of copies of this expanded share data inside the Failure's .stack and .frames attributes. * in addition, the way we're using `DeferredList` fails to catch errors in callRemote: servers fail to accept shares, but we don't notice. What we need to do to address this: * remove the size constraint on `ShareData`. This will stop these particular Violations from happening * Fix twisted ticket [Twisted#2466](http://twistedmatrix.com/trac/ticket/2466), somehow, then require the resulting release of twisted. * add fireOnOneErrback=True to the uses of `DeferredList` in mutable.py, to properly catch exceptions Any exception in share transmission is likely to consume this sort of memory. Workarounds we could conceivably implement before the Twisted problem gets fixed: * avoid holding large strings in local variables: pass them as instance attributes instead, be careful to delete them from locals as soon as possible * hold large strings in some sort of object (with a repr() that doesn't show the whole thing) instead of a real string, and teach Foolscap (via `ISliceable`) to serialize them as strings. * have Foolscap trim Failures in callRemote somehow * try to break up the stacks by using fireEventually in more places * rewrite Foolscap's outbound constraint checking to not do everything on the same stack Our current plan is to fix the constraint and then hope that we don't trigger other failures while we contribute to the Twisted ticket and wait for a new release. Later refactorings of share management will probably put more data in instance attributes rather than being passed through method arguments. If necessary, we can ship tahoe with a patched version of Twisted.

warner added the

labels 2008-04-12 05:44:48 +00:00

warner commented

2008-04-12 18:13:32 +00:00

Author

I changed the remote interface to remove the size constraint, and I fixed the use of DeferredList to propagate errors back to the caller.

I updated our production webapi servers with these change, and they stopped using 3GB of memory every time someone tried to exceed the directory size limit.

I didn't update our storage servers to match, however, so what's probably happening right now is that the remote end is raising the same Violation. However this should involve less memory: the inbound constraint is being tested on each token and rejecting the message immediately (precisely what the foolscap constraint mechanism is designed for). So the caller is still likely to get a very weird error (a foolscap constraint Violation) rather than hitting the earlier "mutable files must be < 3.5MB" check.

I changed the remote interface to remove the size constraint, and I fixed the use of `DeferredList` to propagate errors back to the caller. I updated our production webapi servers with these change, and they stopped using 3GB of memory every time someone tried to exceed the directory size limit. I didn't update our storage servers to match, however, so what's probably happening right now is that the remote end is raising the same Violation. However this should involve less memory: the inbound constraint is being tested on each token and rejecting the message immediately (precisely what the foolscap constraint mechanism is designed for). So the caller is still likely to get a very weird error (a foolscap constraint Violation) rather than hitting the earlier "mutable files must be < 3.5MB" check.

warner commented

2008-04-24 23:54:36 +00:00

Author

the newly-refactoried mutable files no longer use DeferredList.

I think this ticket has served its purpose, and hopefully future searchers will find it in the archives when they run into this sort of problem again.

the newly-refactoried mutable files no longer use `DeferredList`. I think this ticket has served its purpose, and hopefully future searchers will find it in the archives when they run into this sort of problem again.

warner added the

label 2008-04-24 23:54:36 +00:00

warner closed this issue

2008-04-24 23:54:36 +00:00

warner added

c/code-dirnodes

and removed

labels 2008-04-24 23:54:43 +00:00

zooko commented

2008-05-05 21:08:36 +00:00

Milestone 1.0.1 deleted

Milestone 1.0.1 deleted

zooko added this to the 1.1.0 milestone 2008-05-05 21:08:36 +00:00

Sign in to join this conversation.

No labels

c/code-dirnodes

c/code-encoding

c/code-frontend

c/code-frontend-cli

c/code-frontend-ftp-sftp

c/code-frontend-magic-folder

c/code-frontend-web

c/code-nodeadmin

c/code-peerselection

c/dev-infrastructure

kw:AttributeError

kw:DataUnavailable

kw:DeadReferenceError

kw:GetLastError

kw:IFinishableConsumer

kw:LeastAuthority

kw:RIStorageServer

kw:UncoordinatedWriteError

kw:access-control

kw:accessibility

kw:accounting-crawler

kw:anti-censorship

kw:api_auth_token

kw:authentication

kw:availability

kw:backward-compatibility

kw:blocks-cloud-deployment

kw:blocks-cloud-merge

kw:blocks-magic-folder-merge

kw:blocks-merge

kw:blocks-release

kw:brians-opinion-needed

kw:build-helpers

kw:cloud-backend

kw:coding-standards

kw:coding-tools

kw:coding_tools

kw:compatibility

kw:confidentiality

kw:configuration

kw:configuration.txt

kw:connectivity

kw:control.furl

kw:coordination

kw:coveralls.io

kw:create-container

kw:cryptography

kw:cryptography-lib

kw:denial-of-service

kw:desert-island

kw:desert-island-build

kw:design-review-needed

kw:dev-infrastructure

kw:directory-page

kw:disk-backend

kw:earth-dragon

kw:erasure-coding

kw:excess-authority

kw:extensibility

kw:file-descriptor

kw:floatingpoint

kw:forward-compatibility

kw:forward-secrecy

kw:garbage-collection

kw:google-cloud-storage

kw:google-drive-backend

kw:grid-manager

kw:illustration

kw:inlineCallbacks

kw:integration-test

kw:interoperability

kw:interstellar-exploration

kw:introduction

kw:key-value-store

kw:magic-folder

kw:manual-test-needed

kw:more-info-needed

kw:mountain-lion

kw:multiuser-gateway

kw:needs-review

kw:notification

kw:notifyOnDisconnect

kw:openitp-packaging

kw:operation-helpers

kw:optimization

kw:organization

kw:otf-magic-folder-objective1

kw:otf-magic-folder-objective2

kw:otf-magic-folder-objective3

kw:otf-magic-folder-objective4

kw:otf-magic-folder-objective5

kw:otf-magic-folder-objective6

kw:peer-selection

kw:pkg_resources

kw:preservation

kw:pycrypto-lib

kw:pyfilesystem

kw:random-access

kw:raspberry-pi

kw:release-blocker

kw:removable-disk

kw:reserved_space

kw:response-needed

kw:response-time

kw:review-needed

kw:self-contained

kw:servers-of-happiness

kw:setup_requires

kw:setuptools_darcs

kw:simultaneous

kw:space-efficiency

kw:static-analysis

kw:stats_gatherer

kw:tahoe-add-alias

kw:tahoe-archive

kw:tahoe-backup

kw:tahoe-create-alias

kw:tahoe-create-introducer

kw:tahoe-deep-check

kw:tahoe-deepcheck

kw:tahoe-lafs-trac-stream

kw:tahoe-list-aliases

kw:tahoe-magic-folder

kw:tahoe-manifest

kw:tahoe-restart

kw:tahoe-unlink

kw:tahoe-webopen

kw:test-and-set

kw:test-from-egg

kw:ticket999-s3-backend

kw:to-be-closed-on-2011-08-01

kw:tor-protocol

kw:transparency

kw:tub.location

kw:twisted-trial

kw:uncoordinated-writes

kw:unfinished-business

kw:unhandled-error

kw:upload-helper

kw:visualization

kw:volunteergrid2

kw:warners-opinion-needed

kw:welcome-page

kw:windows-related

kw:world-domination

kw:write-enabler

kw:zookos-opinion-needed

kw:zope.interface

p/supercritical

r/cannot reproduce

r/somebody else's problem

r/was already fixed

v/1.9.0-s3branch

No milestone

No project

No assignees

2 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: tahoe-lafs/trac#379

No description provided.