tahoe-lafs/trac

stop using share crawler for anything except constructing a leasedb #1834

New issue

Open

opened 2012-10-30 22:44:21 +00:00 by zooko · 5 comments

zooko commented

2012-10-30 22:44:21 +00:00

I think we should try to use a "share crawler" — a long-running, persistent, duty-cycle-limited process that visits every share held by a storage server — for as little as possible.

And, I think that the only thing that we can't do in a different way is: construct a leasedb when we are first upgrading the server to a leasedb-capable version, or the leasedb has been lost or corrupted.

Here are the other things that are currently done by crawlers and how I think they should be done differently:

Updating and/or checking the leases on shares to see if they have expired;

On David-Sarah's 666-accounting branch, this is now done for all shares by a single, synchronous command/query to leasedb. (#666)

Delete shares that have lost all their leases (by cancellation or expiry);

I propose that this be done instead by the storage server maintaining a persistent set of shares to be deleted. When lease-updating step (which, in #666, is synchronous and fast) has identified a share that has no more leases, the share's id gets added to the persistent set of shares to delete. A long-running, persistent, duty-cycle-limited processes deletes those shares from the backend and removes their ids from the set of shares-to-delete. This is cleaner and more efficient than using a crawler, which has to visit all shares and which never stops twitching, since this has to visit only shares that have been marked as to-delete, and it quiesces when there is nothing to delete. (#1833 — storage server deletes garbage shares itself instead of waiting for crawler to notice them)

Discover newly added shares that the operator copied into the backend without notifying the storage server;

I propose that we stop supporting this use case. It can be replaced by some combination of: 1. requiring you to run a tahoe-lafs storage client tool (a share migration tool) to upload the shares through the server instead of copying the shares directly into the backend, 2. various kludgy workarounds, 3. a new tool for registering specific storage indexes in the leasedb after you've added the shares directly into the backend, or 4. simply requiring that the operator manually trigger the crawler to start instead of expecting the crawler to run continuously. (#1835 — stop grovelling the whole storage backend looking for externally-added shares to add a lease to)

Count how many shares you have;

This can be nicely replaced by leasedb (a simple SQL "COUNT" query), and also the functionality can be extended to compute the aggregate sizes of data in addition to the mere number of objects, which would be very useful for customers of LeastAuthority.com (who pay per byte), among others. (#1836 — stop crawling share files in order to figure out how many shares you have)

What would be better about removing these uses of crawler?

The storage server would be more efficient in terms of accesses to its storage backend. This might turn out to matter when the storage backend is a cloud storage service and you pay per API call. (Or it might not, if the cost is cheap enough and the crawly way to do it is efficient enough.)
The crawling would be a quiescent process — something that finishes its job and then stops, and doesn't start again unless a user tells it to. I like this way of doing things. See wiki/FAQ#Q18_unobtrusive_software .
Some of these operations would be faster and better if done in the newly proposed way instead of by relying on a crawler.

I think we should try to use a "share crawler" — a long-running, persistent, duty-cycle-limited process that visits every share held by a storage server — for as little as possible. And, I think that the only thing that we can't do in a different way is: construct a leasedb when we are first upgrading the server to a leasedb-capable version, or the leasedb has been lost or corrupted. Here are the other things that are currently done by crawlers and how I think they should be done differently: * Updating and/or checking the leases on shares to see if they have expired; On David-Sarah's 666-accounting branch, this is now done for all shares by a single, synchronous command/query to leasedb. (#666) * Delete shares that have lost all their leases (by cancellation or expiry); I propose that this be done instead by the storage server maintaining a persistent set of shares to be deleted. When lease-updating step (which, in #666, is synchronous and fast) has identified a share that has no more leases, the share's id gets added to the persistent set of shares to delete. A long-running, persistent, duty-cycle-limited processes deletes those shares from the backend and removes their ids from the set of shares-to-delete. This is cleaner and more efficient than using a crawler, which has to visit *all* shares and which never stops twitching, since this has to visit only shares that have been marked as to-delete, and it quiesces when there is nothing to delete. (#1833 — storage server deletes garbage shares itself instead of waiting for crawler to notice them) * Discover newly added shares that the operator copied into the backend without notifying the storage server; I propose that we stop supporting this use case. It can be replaced by some combination of: 1. requiring you to run a tahoe-lafs storage client tool (a share migration tool) to upload the shares through the server instead of copying the shares directly into the backend, 2. various kludgy workarounds, 3. a new tool for registering specific storage indexes in the leasedb after you've added the shares directly into the backend, or 4. simply requiring that the operator manually trigger the crawler to start instead of expecting the crawler to run continuously. (#1835 — stop grovelling the whole storage backend looking for externally-added shares to add a lease to) * Count how many shares you have; This can be nicely replaced by leasedb (a simple SQL "COUNT" query), and also the functionality can be extended to compute the aggregate sizes of data in addition to the mere number of objects, which would be very useful for customers of LeastAuthority.com (who pay per byte), among others. (#1836 — stop crawling share files in order to figure out how many shares you have) What would be better about removing these uses of crawler? 1. The storage server would be more efficient in terms of accesses to its storage backend. This might turn out to matter when the storage backend is a cloud storage service and you pay per API call. (Or it might not, if the cost is cheap enough and the crawly way to do it is efficient enough.) 2. The crawling would be a quiescent process — something that finishes its job and then stops, and doesn't start again unless a user tells it to. I like this way of doing things. See [wiki/FAQ#Q18_unobtrusive_software](wiki/FAQ#Q18_unobtrusive_software) . 3. Some of these operations would be faster and better if done in the newly proposed way instead of by relying on a crawler.

zooko added the

labels 2012-10-30 22:44:21 +00:00

zooko added this to the undecided milestone 2012-10-30 22:44:21 +00:00

zooko commented

2012-11-07 16:54:05 +00:00

Author

I'm pretty interested in taking this design to the extreme to get the best efficiency. In that extreme, we never go to persistent storage for either read or write (or existence check) — which requires at least a disk seek for a direct-attached-storage backend or at least a cloud service API request for a cloud backend — unless the leasedb told us to go to persistent storage. (Except, in the case that we're currently building or rebuilding leasedb by crawling persistent storage.)

I'm pretty interested in taking this design to the extreme to get the best efficiency. In that extreme, we *never* go to persistent storage for either read or write (or existence check) — which requires at least a disk seek for a direct-attached-storage backend or at least a cloud service API request for a cloud backend — unless the leasedb *told* us to go to persistent storage. (Except, in the case that we're currently building or rebuilding leasedb by crawling persistent storage.)

daira commented

2013-05-28 02:01:55 +00:00

Replying to zooko:

I'm pretty interested in taking this design to the extreme to get the best efficiency. In that extreme, we never go to persistent storage for either read or write (or existence check) — which requires at least a disk seek for a direct-attached-storage backend or at least a cloud service API request for a cloud backend — unless the leasedb told us to go to persistent storage. (Except, in the case that we're currently building or rebuilding leasedb by crawling persistent storage.)

I agree with returning a positive response to an existence check (DYHB) when the leasedb says we have a share. The case where we turn out not to actually have the share is an error that the downloader, uploader, or repairer should tolerate anyway.

I think that returning a negative response to an existence check when the leasedb says we don't have a share is more debatable. In principle, this shouldn't affect latency of downloads because the downloader should use the first shares.needed servers that respond positively, so the latency of negative responses shouldn't matter.

Replying to [zooko](/tahoe-lafs/trac/issues/1834#issuecomment-391375): > I'm pretty interested in taking this design to the extreme to get the best efficiency. In that extreme, we *never* go to persistent storage for either read or write (or existence check) — which requires at least a disk seek for a direct-attached-storage backend or at least a cloud service API request for a cloud backend — unless the leasedb *told* us to go to persistent storage. (Except, in the case that we're currently building or rebuilding leasedb by crawling persistent storage.) I agree with returning a positive response to an existence check (DYHB) when the leasedb says we have a share. The case where we turn out not to actually have the share is an error that the downloader, uploader, or repairer should tolerate anyway. I think that returning a negative response to an existence check when the leasedb says we don't have a share is more debatable. In principle, this shouldn't affect latency of downloads because the downloader should use the first `shares.needed` servers that respond positively, so the latency of negative responses shouldn't matter.

zooko commented

2013-11-14 19:05:25 +00:00

Author

This would fix #1886.

This would fix #1886.

zooko commented

2013-11-14 19:08:01 +00:00

Author

This ticket would obviate the need for lease-checking-crawler and bucket-counting-crawler to deal with corrupted shares (#812), since those two things would no longer exist. However, we might then want to invent a share-verifying crawler specifically to check for corrupted shares.

This ticket would obviate the need for lease-checking-crawler and bucket-counting-crawler to deal with corrupted shares (#812), since those two things would no longer exist. However, we might then want to invent a share-verifying crawler specifically to check for corrupted shares.

zooko commented

2013-11-14 19:08:32 +00:00

Author

This would fix #1280.

This would fix #1280.

Sign in to join this conversation.

No labels

c/code-dirnodes

c/code-encoding

c/code-frontend

c/code-frontend-cli

c/code-frontend-ftp-sftp

c/code-frontend-magic-folder

c/code-frontend-web

c/code-nodeadmin

c/code-peerselection

c/dev-infrastructure

kw:AttributeError

kw:DataUnavailable

kw:DeadReferenceError

kw:GetLastError

kw:IFinishableConsumer

kw:LeastAuthority

kw:RIStorageServer

kw:UncoordinatedWriteError

kw:access-control

kw:accessibility

kw:accounting-crawler

kw:anti-censorship

kw:api_auth_token

kw:authentication

kw:availability

kw:backward-compatibility

kw:blocks-cloud-deployment

kw:blocks-cloud-merge

kw:blocks-magic-folder-merge

kw:blocks-merge

kw:blocks-release

kw:brians-opinion-needed

kw:build-helpers

kw:cloud-backend

kw:coding-standards

kw:coding-tools

kw:coding_tools

kw:compatibility

kw:confidentiality

kw:configuration

kw:configuration.txt

kw:connectivity

kw:control.furl

kw:coordination

kw:coveralls.io

kw:create-container

kw:cryptography

kw:cryptography-lib

kw:denial-of-service

kw:desert-island

kw:desert-island-build

kw:design-review-needed

kw:dev-infrastructure

kw:directory-page

kw:disk-backend

kw:earth-dragon

kw:erasure-coding

kw:excess-authority

kw:extensibility

kw:file-descriptor

kw:floatingpoint

kw:forward-compatibility

kw:forward-secrecy

kw:garbage-collection

kw:google-cloud-storage

kw:google-drive-backend

kw:grid-manager

kw:illustration

kw:inlineCallbacks

kw:integration-test

kw:interoperability

kw:interstellar-exploration

kw:introduction

kw:key-value-store

kw:magic-folder

kw:manual-test-needed

kw:more-info-needed

kw:mountain-lion

kw:multiuser-gateway

kw:needs-review

kw:notification

kw:notifyOnDisconnect

kw:openitp-packaging

kw:operation-helpers

kw:optimization

kw:organization

kw:otf-magic-folder-objective1

kw:otf-magic-folder-objective2

kw:otf-magic-folder-objective3

kw:otf-magic-folder-objective4

kw:otf-magic-folder-objective5

kw:otf-magic-folder-objective6

kw:peer-selection

kw:pkg_resources

kw:preservation

kw:pycrypto-lib

kw:pyfilesystem

kw:random-access

kw:raspberry-pi

kw:release-blocker

kw:removable-disk

kw:reserved_space

kw:response-needed

kw:response-time

kw:review-needed

kw:self-contained

kw:servers-of-happiness

kw:setup_requires

kw:setuptools_darcs

kw:simultaneous

kw:space-efficiency

kw:static-analysis

kw:stats_gatherer

kw:tahoe-add-alias

kw:tahoe-archive

kw:tahoe-backup

kw:tahoe-create-alias

kw:tahoe-create-introducer

kw:tahoe-deep-check

kw:tahoe-deepcheck

kw:tahoe-lafs-trac-stream

kw:tahoe-list-aliases

kw:tahoe-magic-folder

kw:tahoe-manifest

kw:tahoe-restart

kw:tahoe-unlink

kw:tahoe-webopen

kw:test-and-set

kw:test-from-egg

kw:ticket999-s3-backend

kw:to-be-closed-on-2011-08-01

kw:tor-protocol

kw:transparency

kw:tub.location

kw:twisted-trial

kw:uncoordinated-writes

kw:unfinished-business

kw:unhandled-error

kw:upload-helper

kw:visualization

kw:volunteergrid2

kw:warners-opinion-needed

kw:welcome-page

kw:windows-related

kw:world-domination

kw:write-enabler

kw:zookos-opinion-needed

kw:zope.interface

p/supercritical

r/cannot reproduce

r/somebody else's problem

r/was already fixed

v/1.9.0-s3branch

No milestone

No project

No assignees

2 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: tahoe-lafs/trac#1834

No description provided.