tahoe-lafs/trac-2024-07-25
2
0
Issues 1.1k Wiki Activity

memory leak in allmydata.test.test_web #1729

New issue
Closed
opened 2012-05-05 06:05:47 +00:00 by zooko · 7 comments
zooko commented 2012-05-05 06:05:47 +00:00
Owner
Copy link

Running allmydata.test.test_web with --until-failure results in memory usage increasing until it fails when a subprocess can't be created:

time  ./bin/tahoe @trial --rterror --until-failure allmydata.test.test_web

…

    test_welcome_page_mkdir_button ...                                     [OK]

-------------------------------------------------------------------------------
Ran 243 tests in 22.676s

PASSED (successes=243)
Test Pass 13
allmydata.test.test_web
  Grid
    test_add_lease ...                                                     [OK]
    test_blacklist ...                                                     [OK]
    test_deep_add_lease ...                                                [OK]
    test_deep_check ...                                                    [OK]
    test_deep_check_and_repair ...                                         [OK]
    test_exceptions ...                                                    [OK]
    test_filecheck ...                                                     [OK]
    test_immutable_unknown ...                                             [OK]
    test_mutant_dirnodes_are_omitted ...                                   [OK]
    test_repair_html ...                                                   [OK]
    test_repair_json ...                                                   [OK]
    test_unknown ...                                                       [OK]
  IntroducerWeb
    test_welcome ... Node._startService failed, aborting
[Failure instance: Traceback: <type 'exceptions.OSError'>: [Errno 12] Cannot allocate memory
/usr/lib/python2.7/threading.py:524:__bootstrap
/usr/lib/python2.7/threading.py:551:__bootstrap_inner
/usr/lib/python2.7/threading.py:504:run
--- <exception caught here> ---
/home/zooko/playground/twisted/twisted/twisted/python/threadpool.py:167:_worker
/home/zooko/playground/twisted/twisted/twisted/python/context.py:118:callWithContext
/home/zooko/playground/twisted/twisted/twisted/python/context.py:81:callWithContext
/home/zooko/playground/tahoe-lafs/dw/src/allmydata/util/iputil.py:224:_synchronously_find_addresses_via_config
/home/zooko/playground/tahoe-lafs/dw/src/allmydata/util/iputil.py:238:_query
/usr/lib/python2.7/subprocess.py:679:__init__
/usr/lib/python2.7/subprocess.py:1143:_execute_child
]
calling os.abort()

real    4m41.030s
user    4m20.853s
sys     0m13.239s

This is on my Macbook Pro running Ubuntu 12.04. Tahoe is current (darcs) trunk:

$ darcs pull
Pulling from "zooko@tahoe-lafs.org:/home/source/darcs/tahoe-lafs/trunk"...
No remote changes to pull in!
$ python setup.py update_version
running update_version
darcsver: wrote '1.9.0-r5493' into src/allmydata/_version.py
$ ./bin/tahoe --version
allmydata-tahoe: 1.9.0-r5493,
foolscap: 0.6.3.post0,
pycryptopp: 0.6.0.1206569328141510525648634803928199668821045408958,
zfec: 1.4.24,
Twisted: 10.1.0,
Nevow: 0.10.0,
zope.interface: unknown,
python: 2.7.3,
platform: Linux-Ubuntu_12.04-x86_64-64bit_ELF,
pyOpenSSL: 0.12,
simplejson: 2.3.2,
pycrypto: 2.4.1,
pyasn1: unknown,
mock: 0.8.0beta3,
sqlite3: 2.6.0 [sqlite 3.7.9],
setuptools: 0.6c16dev3
Running `allmydata.test.test_web` with `--until-failure` results in memory usage increasing until it fails when a subprocess can't be created: ``` time ./bin/tahoe @trial --rterror --until-failure allmydata.test.test_web ``` … ``` test_welcome_page_mkdir_button ... [OK] ------------------------------------------------------------------------------- Ran 243 tests in 22.676s PASSED (successes=243) Test Pass 13 allmydata.test.test_web Grid test_add_lease ... [OK] test_blacklist ... [OK] test_deep_add_lease ... [OK] test_deep_check ... [OK] test_deep_check_and_repair ... [OK] test_exceptions ... [OK] test_filecheck ... [OK] test_immutable_unknown ... [OK] test_mutant_dirnodes_are_omitted ... [OK] test_repair_html ... [OK] test_repair_json ... [OK] test_unknown ... [OK] IntroducerWeb test_welcome ... Node._startService failed, aborting [Failure instance: Traceback: <type 'exceptions.OSError'>: [Errno 12] Cannot allocate memory /usr/lib/python2.7/threading.py:524:__bootstrap /usr/lib/python2.7/threading.py:551:__bootstrap_inner /usr/lib/python2.7/threading.py:504:run --- <exception caught here> --- /home/zooko/playground/twisted/twisted/twisted/python/threadpool.py:167:_worker /home/zooko/playground/twisted/twisted/twisted/python/context.py:118:callWithContext /home/zooko/playground/twisted/twisted/twisted/python/context.py:81:callWithContext /home/zooko/playground/tahoe-lafs/dw/src/allmydata/util/iputil.py:224:_synchronously_find_addresses_via_config /home/zooko/playground/tahoe-lafs/dw/src/allmydata/util/iputil.py:238:_query /usr/lib/python2.7/subprocess.py:679:__init__ /usr/lib/python2.7/subprocess.py:1143:_execute_child ] calling os.abort() real 4m41.030s user 4m20.853s sys 0m13.239s ``` This is on my Macbook Pro running Ubuntu 12.04. Tahoe is current (darcs) trunk: ``` $ darcs pull Pulling from "zooko@tahoe-lafs.org:/home/source/darcs/tahoe-lafs/trunk"... No remote changes to pull in! $ python setup.py update_version running update_version darcsver: wrote '1.9.0-r5493' into src/allmydata/_version.py $ ./bin/tahoe --version allmydata-tahoe: 1.9.0-r5493, foolscap: 0.6.3.post0, pycryptopp: 0.6.0.1206569328141510525648634803928199668821045408958, zfec: 1.4.24, Twisted: 10.1.0, Nevow: 0.10.0, zope.interface: unknown, python: 2.7.3, platform: Linux-Ubuntu_12.04-x86_64-64bit_ELF, pyOpenSSL: 0.12, simplejson: 2.3.2, pycrypto: 2.4.1, pyasn1: unknown, mock: 0.8.0beta3, sqlite3: 2.6.0 [sqlite 3.7.9], setuptools: 0.6c16dev3 ```
tahoe-lafs added the
code
normal
defect
1.9.1
 labels 2012-05-05 06:05:47 +00:00
tahoe-lafs added this to the undecided milestone 2012-05-05 06:05:47 +00:00
warner commented 2012-05-22 21:23:13 +00:00
Author
Owner
Copy link

There are four test classes in test_web: Web, IntroducerWeb, Util,
and Grid. I ruled out Util and IntroducerWeb (running them for
several minutes in a row has a flat memory footprint). Running Grid
by itself uses a bunch but seems to peak at 78M VMSize after about
2 or 3 passes. Running Web by itself consumes more and more memory.

I'm pretty sure this is due to the "all_contents" class-level dictionary in allmydata.test.common.FakeCHKFileNode and FakeMutableFileNode. I'll look into parametizing that with a container which can be discarded between test runs.

There are four test classes in test_web: Web, IntroducerWeb, Util, and Grid. I ruled out Util and IntroducerWeb (running them for several minutes in a row has a flat memory footprint). Running Grid by itself uses a bunch but seems to peak at 78M VMSize after about 2 or 3 passes. Running Web by itself consumes more and more memory. I'm pretty sure this is due to the "all_contents" class-level dictionary in `allmydata.test.common.FakeCHKFileNode` and `FakeMutableFileNode`. I'll look into parametizing that with a container which can be discarded between test runs.
warner commented 2012-05-22 22:31:33 +00:00
Author
Owner
Copy link

Yeah, that seemed to be the problem. The patch I'm about to fix moves that container out into the FakeClient created fresh for each test case, which removes the lingering buildup. With that in place, I see the memory usage for --until-failure looping of test_web.Web grow to maybe 250MB or 300MB and then drop back to 150MB (as GC kicks in). I think that closes the leak, although it might be nice to identify why it's still using so much RAM (maybe the test files it's uploading are excessively large).

Yeah, that seemed to be the problem. The patch I'm about to fix moves that container out into the FakeClient created fresh for each test case, which removes the lingering buildup. With that in place, I see the memory usage for `--until-failure` looping of `test_web.Web` grow to maybe 250MB or 300MB and then drop back to 150MB (as GC kicks in). I think that closes the leak, although it might be nice to identify why it's still using so much RAM (maybe the test files it's uploading are excessively large).
Brian Warner <warner@lothar.com> commented 2012-05-22 23:07:30 +00:00
Author
Owner
Copy link

In changeset:bfee999e20aa9fdc:

test_web.py: fix memory leak when run with --until-failure

The Fake*Node classes in test/common.py were accumulating share data in
a class-level dictionary, which persisted from one test run to the next.
As a result, running test_web.py over and over (with trial's
--until-failure feature) made this dictionary grow without bound,
eventually running out of memory.

This fix moves that dictionary into the FakeClient built fresh for each
test, so it doesn't build up. It does the same thing for "file_types",
which was much smaller but still lived at the class level.

Closes #1729
In changeset:bfee999e20aa9fdc: ``` test_web.py: fix memory leak when run with --until-failure The Fake*Node classes in test/common.py were accumulating share data in a class-level dictionary, which persisted from one test run to the next. As a result, running test_web.py over and over (with trial's --until-failure feature) made this dictionary grow without bound, eventually running out of memory. This fix moves that dictionary into the FakeClient built fresh for each test, so it doesn't build up. It does the same thing for "file_types", which was much smaller but still lived at the class level. Closes #1729 ```
tahoe-lafs added the
fixed
 label 2012-05-22 23:07:30 +00:00
Brian Warner <warner@lothar.com> closed this issue 2012-05-22 23:07:30 +00:00
Brian Warner <warner@lothar.com> commented 2012-05-22 23:08:08 +00:00
Author
Owner
Copy link

In changeset:bfee999e20aa9fdc:

test_web.py: fix memory leak when run with --until-failure

The Fake*Node classes in test/common.py were accumulating share data in
a class-level dictionary, which persisted from one test run to the next.
As a result, running test_web.py over and over (with trial's
--until-failure feature) made this dictionary grow without bound,
eventually running out of memory.

This fix moves that dictionary into the FakeClient built fresh for each
test, so it doesn't build up. It does the same thing for "file_types",
which was much smaller but still lived at the class level.

Closes #1729
In changeset:bfee999e20aa9fdc: ``` test_web.py: fix memory leak when run with --until-failure The Fake*Node classes in test/common.py were accumulating share data in a class-level dictionary, which persisted from one test run to the next. As a result, running test_web.py over and over (with trial's --until-failure feature) made this dictionary grow without bound, eventually running out of memory. This fix moves that dictionary into the FakeClient built fresh for each test, so it doesn't build up. It does the same thing for "file_types", which was much smaller but still lived at the class level. Closes #1729 ```
tahoe-lafs modified the milestone from undecided to 1.10.0 2012-05-22 23:09:01 +00:00
Brian Warner <warner@lothar.com> commented 2012-05-23 05:28:01 +00:00
Author
Owner
Copy link

In changeset:5503/1.9.2:

test_web.py: fix memory leak when run with --until-failure

The Fake*Node classes in test/common.py were accumulating share data in
a class-level dictionary, which persisted from one test run to the next.
As a result, running test_web.py over and over (with trial's
--until-failure feature) made this dictionary grow without bound,
eventually running out of memory.

This fix moves that dictionary into the FakeClient built fresh for each
test, so it doesn't build up. It does the same thing for "file_types",
which was much smaller but still lived at the class level.

Closes #1729
In changeset:5503/1.9.2: ``` test_web.py: fix memory leak when run with --until-failure The Fake*Node classes in test/common.py were accumulating share data in a class-level dictionary, which persisted from one test run to the next. As a result, running test_web.py over and over (with trial's --until-failure feature) made this dictionary grow without bound, eventually running out of memory. This fix moves that dictionary into the FakeClient built fresh for each test, so it doesn't build up. It does the same thing for "file_types", which was much smaller but still lived at the class level. Closes #1729 ```
davidsarah commented 2012-05-23 05:28:30 +00:00
Author
Owner
Copy link

Safe to apply to 1.9.2 since it only affects tests.

Safe to apply to 1.9.2 since it only affects tests.
tahoe-lafs modified the milestone from 1.10.0 to 1.9.2 2012-05-23 05:28:30 +00:00
Brian Warner <warner@lothar.com> commented 2012-07-10 20:04:56 +00:00
Author
Owner
Copy link

In changeset:5852/cloud-backend:

test_web.py: fix memory leak when run with --until-failure

The Fake*Node classes in test/common.py were accumulating share data in
a class-level dictionary, which persisted from one test run to the next.
As a result, running test_web.py over and over (with trial's
--until-failure feature) made this dictionary grow without bound,
eventually running out of memory.

This fix moves that dictionary into the FakeClient built fresh for each
test, so it doesn't build up. It does the same thing for "file_types",
which was much smaller but still lived at the class level.

Closes #1729
In changeset:5852/cloud-backend: ``` test_web.py: fix memory leak when run with --until-failure The Fake*Node classes in test/common.py were accumulating share data in a class-level dictionary, which persisted from one test run to the next. As a result, running test_web.py over and over (with trial's --until-failure feature) made this dictionary grow without bound, eventually running out of memory. This fix moves that dictionary into the FakeClient built fresh for each test, so it doesn't build up. It does the same thing for "file_types", which was much smaller but still lived at the class level. Closes #1729 ```
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
No results found.
No results found.
Labels
Clear labels   
0.2.0

   
0.3.0

   
0.4.0

   
0.5.0

   
0.5.1

   
0.6.0

   
0.6.1

   
0.7.0

   
0.8.0

   
0.9.0

   
1.0.0

   
1.1.0

   
1.10.0

   
1.10.1

   
1.10.2

   
1.10a2

   
1.11.0

   
1.12.0

   
1.12.1

   
1.13.0

   
1.14.0

   
1.15.0

   
1.15.1

   
1.2.0

   
1.3.0

   
1.4.1

   
1.5.0

   
1.6.0

   
1.6.1

   
1.7.0

   
1.7.1

   
1.7β

   
1.8.0

   
1.8.1

   
1.8.2

   
1.8.3

   
1.8β

   
1.9.0

   
1.9.0-s3branch

   
1.9.0a1

   
1.9.0a2

   
1.9.0b1

   
1.9.1

   
1.9.2

   
1.9.2a1

   
LeastAuthority.com automation

   
blocker

   
cannot reproduce

   
cloud-branch

   
code

catch-all for anything that can be fixed by writing code but isn't classified better

  
code-dirnodes

anything involving tahoe directories

  
code-encoding

immutable-file encoding format/layout, cap format, encoding/decoding algorithms

  
code-frontend

general frontend things: new user-facing interfaces, FTP/SFTP servers, FUSE-like interfaces

  
code-frontend-cli

CLI tools

  
code-frontend-ftp-sftp

   
code-frontend-magic-folder

   
code-frontend-web

webapi interfaces, user-facing "WUI" interfaces

  
code-mutable

anything involving mutable files: encoding/decoding, share format

  
code-network

Introducer, locating storage servers, establishing/maintaining connections to them

  
code-nodeadmin

node startup/shutdown, tools to create nodes, logging tools, management/monitoring tools, manhole, disk-space-limiting tools

  
code-peerselection

server selection/sorting algorithm, rebalancing code, allocation policy, swarming-download

  
code-storage

storage-server implementation, share-storage directory layout, client-to-server share transfer protocol

  
contrib

problems with non-core things that live in the source:contrib/ directory

  
critical

   
defect

   
dev-infrastructure

things that help us build Tahoe: buildbot, automated speed/memory/code-coverage tests and results-displayers, IRC channels, mailing lists, this Trac instance

  
documentation

docs items. Also use 'docs' keyword on tickets in more specific components.

  
duplicate

   
enhancement

   
fixed

   
invalid

   
major

   
minor

   
n/a

   
normal

   
operational

things involved in running a tahoe grid, external tools to monitor/maintain a grid (like "nodeadmin" but larger-scale). Was also used for issues with the allmydata.com grid.

  
packaging

setup.py script, library dependencies, binary packages, licensing, installation instructions

  
somebody else's problem

   
supercritical

   
task

   
trivial

   
unknown

Uncategorized. All tickets default to this component until someone updates it.

  
was already fixed

   
website

things about the tahoe-lafs.org website (and sometimes this Trac instance, shared with 'dev-infrastructure')

  
wontfix

   
worksforme
No labels
0.2.0
0.3.0
0.4.0
0.5.0
0.5.1
0.6.0
0.6.1
0.7.0
0.8.0
0.9.0
1.0.0
1.1.0
1.10.0
1.10.1
1.10.2
1.10a2
1.11.0
1.12.0
1.12.1
1.13.0
1.14.0
1.15.0
1.15.1
1.2.0
1.3.0
1.4.1
1.5.0
1.6.0
1.6.1
1.7.0
1.7.1
1.7β
1.8.0
1.8.1
1.8.2
1.8.3
1.8β
1.9.0
1.9.0-s3branch
1.9.0a1
1.9.0a2
1.9.0b1
1.9.1
1.9.2
1.9.2a1
LeastAuthority.com automation
blocker
cannot reproduce
cloud-branch
code
code-dirnodes
code-encoding
code-frontend
code-frontend-cli
code-frontend-ftp-sftp
code-frontend-magic-folder
code-frontend-web
code-mutable
code-network
code-nodeadmin
code-peerselection
code-storage
contrib
critical
defect
dev-infrastructure
documentation
duplicate
enhancement
fixed
invalid
major
minor
n/a
normal
operational
packaging
somebody else's problem
supercritical
task
trivial
unknown
was already fixed
website
wontfix
worksforme
Milestone
Clear milestone
No items
No milestone
1.9.2
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: tahoe-lafs/trac-2024-07-25#1729
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?