tahoe-lafs/trac-2024-07-25

disk space resource limits on storage servers #34

New issue

Closed

opened 2007-05-04 05:54:55 +00:00 by zooko · 4 comments

zooko commented

2007-05-04 05:54:55 +00:00

Owner

Currently if you run a storage server, then clients can ask you to store data, and you always say yes. Make it possible to configure a storage server to have a limit on how much disk space it will use, and when it is full it says no.

tahoe-lafs added the

code

major

defect

labels 2007-05-04 05:54:55 +00:00

zooko commented

2007-05-04 05:59:02 +00:00

Author

Owner

also mentioned in source:roadmap.txt

warner commented

2007-06-06 21:56:24 +00:00

Author

Owner

Implementation notes: I'm thinking of having the node do a 'du -s' of the storage directory on startup and keep that value around as its estimate of total storage used. Each time a lease request arrives, subtract the estimate from the max-space target to decide whether we'll accept the lease or not. Keep a list of pending incoming shares and count them against the storage space (so that four billion lease requests that arrive at the same time won't all be honored). When the bucket is closed, do a 'du -s' of the bucket and add it to the estimate. Likewise, when a bucket is deleted, do a 'du -s' first and remove that from the estimate.

Sound reasonable? I'll look to amdlib.utils.fileutil for 'du -s' code, so we have something that'll work on windows too.

zooko commented

2007-06-07 03:00:46 +00:00

Author

Owner

Sounds reasonable!

warner commented

2007-07-04 00:43:55 +00:00

Author

Owner

It's not 100% accurate right now: buckets which are being filled count against the limit, but the metadata that goes into them is only counted once the bucket is closed. The whole storage pool is sized for real at startup, and the amount consumed is modified after that as buckets come and go. The sizelimit does not take into account space consumed by directories, but rather only pays attention to files.

I just implemented this, in changeset:9ddb92965161a067, changeset:c80ea7d69399b153, and changeset:94e6e6160b24959c. Write a number of bytes (perhaps with a suffix like kB or Mb or G) to a file named 'sizelimit' in the client's base directory. It will be read at node startup and used to constrain the [StorageServer](wiki/StorageServer) to that size. It's not 100% accurate right now: buckets which are being filled count against the limit, but the metadata that goes into them is only counted once the bucket is closed. The whole storage pool is sized for real at startup, and the amount consumed is modified after that as buckets come and go. The sizelimit does not take into account space consumed by directories, but rather only pays attention to files.

tahoe-lafs added the

fixed

label 2007-07-04 00:43:55 +00:00

warner closed this issue

2007-07-04 00:43:55 +00:00

tahoe-lafs added this to the 0.5.0 milestone 2007-07-04 00:44:08 +00:00