tahoe-lafs/trac-2024-07-25

How does tahoe filesystem layer (lafs) handle lots of file-upload tasks? #173

New issue

Open

opened 2007-10-11 03:36:54 +00:00 by zooko · 3 comments

zooko commented

2007-10-11 03:36:54 +00:00

Owner

Rob, Peter, Justin and I just fixed up the commercial, proprietary cousin of Tahoe -- the http://allmydata.com product -- to behave better when the user asks it to upload 100,000 files ASAP. How would Tahoe handle a similar situation? To close this ticket, implement an automated measurement (akin to the memory and upload/download rate measurements) which measures how much RAM, wall-clock time, CPU load, etc. a Tahoe node requires to simultaneously upload 1000, 10,000, or 100,000 files.

Rob, Peter, Justin and I just fixed up the commercial, proprietary cousin of Tahoe -- the <http://allmydata.com> product -- to behave better when the user asks it to upload 100,000 files ASAP. How would Tahoe handle a similar situation? To close this ticket, implement an automated measurement (akin to the [memory and upload/download rate](wiki/Performance) measurements) which measures how much RAM, wall-clock time, CPU load, etc. a Tahoe node requires to simultaneously upload 1000, 10,000, or 100,000 files.

tahoe-lafs added the

labels 2007-10-11 03:36:54 +00:00

tahoe-lafs added this to the undecided milestone 2007-10-11 03:36:54 +00:00

tahoe-lafs added

0.6.1

and removed

0.6.0

labels 2007-10-19 23:11:38 +00:00

zooko commented

2008-02-28 01:58:33 +00:00

Author

Owner

Fabrice, the entrepeneur behind Allmydata, says that our reference test should be 25,000 files and 100 GB -- "VERY WELL and RELIABLY!". He says this is necessary to be competitive with Carbonite, Mozy, etc.. He promises to raise the bar to 100,000 files and 500 GB as soon as we reach 25,000 files and 100 GB. ;-)

Putting this into the 0.9.0 (Allmydata 3.0 final) Milestone.

tahoe-lafs modified the milestone from undecided to 0.9.0 (Allmydata 3.0 final)

2008-02-28 01:58:33 +00:00

zooko commented

2008-03-10 19:57:45 +00:00

Author

Owner

This ticket is just for the tahoe filesystem layer ("laughfs"). See also #337 (how does the whole system including the Windows native client handle lots of file-upload tasks?).

This ticket is hereby vagueified to be unclear about whether we're talking about lots of parallel upload tasks or a lot of sequential upload tasks or some combination thereof.

This ticket is just for the tahoe filesystem layer ("laughfs"). See also #337 (how does the whole system including the Windows native client handle lots of file-upload tasks?). This ticket is hereby vagueified to be unclear about whether we're talking about lots of parallel upload tasks or a lot of sequential upload tasks or some combination thereof.

tahoe-lafs changed title from ~~How does tahoe handle lots of simultaneous file-upload tasks?~~ to How does tahoe filesystem layer (laughfs) handle lots of file-upload tasks?

2008-03-10 19:57:45 +00:00

warner commented

2008-03-10 20:04:18 +00:00

Author

Owner

incidentally, I'm -1 on the "laughfs" term. I have no idea what it refers to.

tahoe-lafs changed title from ~~How does tahoe filesystem layer (laughfs) handle lots of file-upload tasks?~~ to How does tahoe filesystem layer (lafs) handle lots of file-upload tasks?

2008-03-21 22:31:09 +00:00

tahoe-lafs added this to the undecided milestone 2008-03-21 22:31:29 +00:00