tahoe-lafs/trac

write-only caps #796

New issue

Open

opened 2009-08-22 23:55:34 +00:00 by warner · 2 comments

warner commented

2009-08-22 23:55:34 +00:00

Daira Hopwood points out an even more interesting
direction to take in a recent tahoe-dev posting:

http://allmydata.org/pipermail/tahoe-dev/2009-August/002653.html

The goal is to have one cap (used frequently and stored online)
to do write-only backups, and a different cap (used only for
recovery and stored offline) to perform the reads. The effect
would be close to that of the Mac OS-X shared public "Drop Box"
folder, or of GPG-encrypting a piece of data to a private key
that is held offline: normally a one-way operation, but when you
need to, you open up the vault and pull out the decryption key.

This would be pretty cool. This ticket is to sketch out what the
crypto layout would look like. #795 (append-only files) will be
a starting point, and there will certainly be an asymmetric
encryption/decryption keypair involved.

From the UI point of view, you'd have some sort of magic
append-only no-reading directory cap, which you keep in your
private/alises table. There would be a corresponding
read-everything cap (or maybe just the full-fledged writecap;
these could be stored separately), which you keep in a vault and
only type in to test the system and to recover data. Then you
type "tahoe backup ~ backup-appendonlycap:", and you expect that
this unreadable "backup-appendonlycap:" object will acquire
another child, with a timestamp name that is hopefully (but not
guaranteedly) unique.

You might also like the unchanged-directory-sharing properties
of "tahoe backup" to keep working, so that you don't spend a lot
of time or disk on things that haven't changed. I don't know if
it's possible to accomplish this without recording some
information which would violate the no-reading properties of the
parent. This would probably be easier to pull off if we have
immutable directories (#607). I suspect that you'll still have
to read and hash your whole disk, and generate the CHK
identifiers, and then discover that they're already uploaded. So
you might save the storage space and the upload bandwidth, but
not the local disk IO.

(hm, so the current backupdb would record the uploaded filecaps,
which starts to violate the goals once the original file gets
deleted and the backupdb doesn't also delete the stored filecap.
But if your local filesystem allows you to attach metadata to
the files you're backing up, then just attach the tahoe filecap
and a ctime/mtime/filesize snapshot to the original file, so the
filecap dies with the file. The backup process would look for
this metadata, compare the ctime/mtime/size snapshot to decide
if the cached filecap is stale, then upload or not. This would
be pretty slick, actually, and I think several modern
filesystems let you attach this sort of metadata (HFS+ for one).
If you can attach metadata to directories, then you write the
verifycap of the immutable dirnode last used for that directory:
on each new backup, you figure out the new dirnode contents,
hash them into the CHK key, hash that and compare it against
the verifycap, if they match then boom now you have the dirnode
readcap for going up to the parent, if they don't match then you
must upload the new version of that dirnode. This avoids keeping
the old dircap cleartext around. The only remaining security
issue is that you'd be keeping the individual filecaps around
for old versions, until the next "tahoe backup" process came
along and replaced them, but this is a much smaller exposure
than the dirnodes. It would leak the following information: if
an attacker gets a copy of your disk at time T=2, they might be
able to learn the contents of modified-but-not-deleted files
that we previously backed up at time T=1.)

It's probably ok for the "tahoe backup" process to upload files
and create directories, generating temporary caps which it is
obligated to forget after the top-level append operation. If the
whole backup is created out of immutable objects, the only
mutable slot is the top-most timestamped-version holding
directory, and that's where the append-only operation would be
used.

I'm trying to imagine if it would make sense to add an
"append-only" or "write-only-no-reading" column to the dirnode
table (to provide something like "transitive append-only-ness").
I'm not even sure if that's sane, so I'll put off thinking about
it until later. (if you can't read, is "transitive" even
defined?).

Daira Hopwood points out an even more interesting direction to take in a recent tahoe-dev posting: <http://allmydata.org/pipermail/tahoe-dev/2009-August/002653.html> The goal is to have one cap (used frequently and stored online) to do write-only backups, and a different cap (used only for recovery and stored offline) to perform the reads. The effect would be close to that of the Mac OS-X shared public "Drop Box" folder, or of GPG-encrypting a piece of data to a private key that is held offline: normally a one-way operation, but when you need to, you open up the vault and pull out the decryption key. This would be pretty cool. This ticket is to sketch out what the crypto layout would look like. #795 (append-only files) will be a starting point, and there will certainly be an asymmetric encryption/decryption keypair involved. From the UI point of view, you'd have some sort of magic append-only no-reading directory cap, which you keep in your private/alises table. There would be a corresponding read-everything cap (or maybe just the full-fledged writecap; these could be stored separately), which you keep in a vault and only type in to test the system and to recover data. Then you type "tahoe backup ~ backup-appendonlycap:", and you expect that this unreadable "backup-appendonlycap:" object will acquire another child, with a timestamp name that is hopefully (but not guaranteedly) unique. You might also like the unchanged-directory-sharing properties of "tahoe backup" to keep working, so that you don't spend a lot of time or disk on things that haven't changed. I don't know if it's possible to accomplish this without recording some information which would violate the no-reading properties of the parent. This would probably be easier to pull off if we have immutable directories (#607). I suspect that you'll still have to read and hash your whole disk, and generate the CHK identifiers, and then discover that they're already uploaded. So you might save the storage space and the upload bandwidth, but not the local disk IO. (hm, so the current backupdb would record the uploaded filecaps, which starts to violate the goals once the original file gets deleted and the backupdb doesn't also delete the stored filecap. But if your local filesystem allows you to attach metadata to the files you're backing up, then just attach the tahoe filecap and a ctime/mtime/filesize snapshot to the original file, so the filecap dies with the file. The backup process would look for this metadata, compare the ctime/mtime/size snapshot to decide if the cached filecap is stale, then upload or not. This would be pretty slick, actually, and I think several modern filesystems let you attach this sort of metadata (HFS+ for one). If you can attach metadata to directories, then you write the verifycap of the immutable dirnode last used for that directory: on each new backup, you figure out the new dirnode contents, hash them into the CHK key, hash *that* and compare it against the verifycap, if they match then boom now you have the dirnode readcap for going up to the parent, if they don't match then you must upload the new version of that dirnode. This avoids keeping the old dircap cleartext around. The only remaining security issue is that you'd be keeping the individual filecaps around for old versions, until the next "tahoe backup" process came along and replaced them, but this is a much smaller exposure than the dirnodes. It would leak the following information: if an attacker gets a copy of your disk at time T=2, they might be able to learn the contents of modified-but-not-deleted files that we previously backed up at time T=1.) It's probably ok for the "tahoe backup" process to upload files and create directories, generating temporary caps which it is obligated to forget after the top-level append operation. If the whole backup is created out of immutable objects, the only mutable slot is the top-most timestamped-version holding directory, and that's where the append-only operation would be used. I'm trying to imagine if it would make sense to add an "append-only" or "write-only-no-reading" column to the dirnode table (to provide something like "transitive append-only-ness"). I'm not even sure if that's sane, so I'll put off thinking about it until later. (if you can't read, is "transitive" even defined?).

warner added the

labels 2009-08-22 23:55:34 +00:00

warner added this to the undecided milestone 2009-08-22 23:55:34 +00:00

daira commented

2009-10-28 03:31:01 +00:00

Tagging issues relevant to new cap protocol design.

Tagging issues relevant to new cap protocol design.

zooko commented

2010-07-12 17:48:31 +00:00

See also the notes about append-only caps in wiki/NewCapDesign.

See also the notes about append-only caps in [wiki/NewCapDesign](wiki/NewCapDesign).

zooko changed title from ~~write-only backup caps~~ to write-only caps

2013-08-13 18:53:44 +00:00

Sign in to join this conversation.

No labels

c/code-dirnodes

c/code-encoding

c/code-frontend

c/code-frontend-cli

c/code-frontend-ftp-sftp

c/code-frontend-magic-folder

c/code-frontend-web

c/code-nodeadmin

c/code-peerselection

c/dev-infrastructure

kw:AttributeError

kw:DataUnavailable

kw:DeadReferenceError

kw:GetLastError

kw:IFinishableConsumer

kw:LeastAuthority

kw:RIStorageServer

kw:UncoordinatedWriteError

kw:access-control

kw:accessibility

kw:accounting-crawler

kw:anti-censorship

kw:api_auth_token

kw:authentication

kw:availability

kw:backward-compatibility

kw:blocks-cloud-deployment

kw:blocks-cloud-merge

kw:blocks-magic-folder-merge

kw:blocks-merge

kw:blocks-release

kw:brians-opinion-needed

kw:build-helpers

kw:cloud-backend

kw:coding-standards

kw:coding-tools

kw:coding_tools

kw:compatibility

kw:confidentiality

kw:configuration

kw:configuration.txt

kw:connectivity

kw:control.furl

kw:coordination

kw:coveralls.io

kw:create-container

kw:cryptography

kw:cryptography-lib

kw:denial-of-service

kw:desert-island

kw:desert-island-build

kw:design-review-needed

kw:dev-infrastructure

kw:directory-page

kw:disk-backend

kw:earth-dragon

kw:erasure-coding

kw:excess-authority

kw:extensibility

kw:file-descriptor

kw:floatingpoint

kw:forward-compatibility

kw:forward-secrecy

kw:garbage-collection

kw:google-cloud-storage

kw:google-drive-backend

kw:grid-manager

kw:illustration

kw:inlineCallbacks

kw:integration-test

kw:interoperability

kw:interstellar-exploration

kw:introduction

kw:key-value-store

kw:magic-folder

kw:manual-test-needed

kw:more-info-needed

kw:mountain-lion

kw:multiuser-gateway

kw:needs-review

kw:notification

kw:notifyOnDisconnect

kw:openitp-packaging

kw:operation-helpers

kw:optimization

kw:organization

kw:otf-magic-folder-objective1

kw:otf-magic-folder-objective2

kw:otf-magic-folder-objective3

kw:otf-magic-folder-objective4

kw:otf-magic-folder-objective5

kw:otf-magic-folder-objective6

kw:peer-selection

kw:pkg_resources

kw:preservation

kw:pycrypto-lib

kw:pyfilesystem

kw:random-access

kw:raspberry-pi

kw:release-blocker

kw:removable-disk

kw:reserved_space

kw:response-needed

kw:response-time

kw:review-needed

kw:self-contained

kw:servers-of-happiness

kw:setup_requires

kw:setuptools_darcs

kw:simultaneous

kw:space-efficiency

kw:static-analysis

kw:stats_gatherer

kw:tahoe-add-alias

kw:tahoe-archive

kw:tahoe-backup

kw:tahoe-create-alias

kw:tahoe-create-introducer

kw:tahoe-deep-check

kw:tahoe-deepcheck

kw:tahoe-lafs-trac-stream

kw:tahoe-list-aliases

kw:tahoe-magic-folder

kw:tahoe-manifest

kw:tahoe-restart

kw:tahoe-unlink

kw:tahoe-webopen

kw:test-and-set

kw:test-from-egg

kw:ticket999-s3-backend

kw:to-be-closed-on-2011-08-01

kw:tor-protocol

kw:transparency

kw:tub.location

kw:twisted-trial

kw:uncoordinated-writes

kw:unfinished-business

kw:unhandled-error

kw:upload-helper

kw:visualization

kw:volunteergrid2

kw:warners-opinion-needed

kw:welcome-page

kw:windows-related

kw:world-domination

kw:write-enabler

kw:zookos-opinion-needed

kw:zope.interface

p/supercritical

r/cannot reproduce

r/somebody else's problem

r/was already fixed

v/1.9.0-s3branch

No milestone

No project

No assignees

3 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: tahoe-lafs/trac#796

No description provided.