tahoe-lafs/trac-2024-07-25

self-update command #2761

New issue

Open

opened 2016-03-30 19:06:43 +00:00 by warner · 3 comments

warner commented

2016-03-30 19:06:43 +00:00

Owner

I had a wild idea this morning, related to #2055.

Start by using a hash-enabled requirements.txt file for the primary install: the install process would look like:

% wget https://tahoe-lafs.org/downloads/tahoe-lafs-requirements.txt
% virtualenv ~/.tahoe-venv
% ~/.tahoe-venv/bin/pip install --require-hashes -r tahoe-lafs-requirements.txt
% ~/.tahoe-venv/bin/tahoe --version

The first wget is TOFU (and depends upon TLS and the CAs), but after that you're installing exactly whatever the tahoe project said you should be installing.

Maybe we even encapsulate all that into an installer stub, like the "lightweight" installers that a lot of professional apps (Firefox, Dropbox, Chrome) are using. You download the (small) installer application in your browser, you run it, then that does the longer full download itself (with retries and hash-checking and localization and nicer error messages and install instructions).

(hm, maybe pipsi should acquire a --requirements= mode, that'd be even easier)

Now here's the wild part: if Tahoe knows that it's installed in a virtualenv, then it could run "pip" to upgrade things. We could add a "tahoe admin upgrade" command which:

downloads the current tahoe-lafs-requirements.txt file
checks an !Ed25519 signature against a baked-in release key
checks that the version string is greater than the current one
build wheels of all necessary dependencies, with --require-hashes
install those wheels into the current virtualenv

The process would we safer than the original install: the reliance set is only that !Ed25519 key (and the people who use it to sign the requirements file).

I suspect Google's cloud-services CLI tool (gcloud) does something similar. And it's safer than a pip install -U of everything that pip list --outdated reports: we can stick with specific versions (and hashes) that we've tested.

I had a wild idea this morning, related to #2055. Start by using a [hash-enabled requirements.txt](https://pip.pypa.io/en/stable/reference/pip_install/#hash-checking-mode) file for the primary install: the install process would look like: ``` % wget https://tahoe-lafs.org/downloads/tahoe-lafs-requirements.txt % virtualenv ~/.tahoe-venv % ~/.tahoe-venv/bin/pip install --require-hashes -r tahoe-lafs-requirements.txt % ~/.tahoe-venv/bin/tahoe --version ``` The first wget is TOFU (and depends upon TLS and the CAs), but after that you're installing exactly whatever the tahoe project said you should be installing. Maybe we even encapsulate all that into an installer stub, like the "lightweight" installers that a lot of professional apps (Firefox, Dropbox, Chrome) are using. You download the (small) installer application in your browser, you run it, then that does the longer full download itself (with retries and hash-checking and localization and nicer error messages and install instructions). (hm, maybe [pipsi](https://github.com/mitsuhiko/pipsi/) should acquire a `--requirements=` mode, that'd be even easier) Now here's the wild part: if Tahoe knows that it's installed in a virtualenv, then it could run "pip" to upgrade things. We could add a "tahoe admin upgrade" command which: * downloads the current `tahoe-lafs-requirements.txt` file * checks an !Ed25519 signature against a baked-in release key * checks that the version string is greater than the current one * build wheels of all necessary dependencies, with `--require-hashes` * install those wheels into the current virtualenv The process would we safer than the original install: the reliance set is only that !Ed25519 key (and the people who use it to sign the requirements file). I suspect Google's cloud-services CLI tool (`gcloud`) does something similar. And it's safer than a `pip install -U` of everything that `pip list --outdated` reports: we can stick with specific versions (and hashes) that we've tested.

tahoe-lafs added the

labels 2016-03-30 19:06:43 +00:00

tahoe-lafs added this to the undecided milestone 2016-03-30 19:06:43 +00:00

leif commented

2016-03-31 11:18:27 +00:00

Author

Owner

As I wrote recently on Ars Technica, I think a basic requirement for new update mechanisms being deployed at this point in history is that they not introduce new single points of failure like the one Ed25519 release key described above!

Anyway, it seems like this is something that should be implemented in pip or somewhere else outside of Tahoe. Perhaps pip could provide an API for applications like Tahoe to check if they are the latest version, and even initiate upgrades? And/or perhaps pip could have a cronjob to auto-update certain packages and their dependencies? I think removing the single points of failure from pip's authenticity-checking regime should be a much higher priority, though!

As I [wrote](http://arstechnica.com/security/2016/02/most-software-already-has-a-golden-key-backdoor-its-called-auto-update/) recently on Ars Technica, I think a basic requirement for new update mechanisms being deployed at this point in history is that they not introduce new single points of failure like the one Ed25519 release key described above! Anyway, it seems like this is something that should be implemented in pip or somewhere else outside of Tahoe. Perhaps pip could provide an API for applications like Tahoe to check if they are the latest version, and even initiate upgrades? And/or perhaps pip could have a cronjob to auto-update certain packages and their dependencies? I think removing the single points of failure from pip's authenticity-checking regime should be a much higher priority, though!

daira commented

2016-03-31 14:45:42 +00:00

Author

Owner

IMHO this is not something that should be done per-application. If it is, >90% of applications will get it wrong (and that's optimistic). Even if every application did implement something secure by itself, then the resulting agglomeration of separate mechanisms would be completely unmanageable.

warner commented

2016-03-31 18:29:30 +00:00

Author

Owner

(hey, that gives us a t-shirt version: "MPOC bad! SPOC better! QUOC best!")

I think the pip folks (especially dstufft) are keen on moving in this direction, at least partially. They've been working to add signatures to wheel files, and per-project pubkeys to the PyPI entries.

But everything I've seen so far is one-wheel-at-a-time. I don't think there's any support yet for whole-project hash-locked requirements.txt -based installs. Using requirements files, and therefore hashes, must be done manually, by downloading a requirements.txt (possibly by doing a git checkout or unpacking a tarball), then running pip install -r requirements.txt. There's no pip install tahoe-lafs -like thing that will automatically fetch one for you.

So anyways, I suspect that we'd have to be a trail-breaker here. We could prototype this in Tahoe and then help push it upstream into pip. I agree that this wants to live somewhere larger than just Tahoe, but it's not there yet, and ya gotta start somewhere.

Leif: oh, sure, that !Ed25519 signature could be implemented as a quorum/TUF-style signature bundle, to turn that single-point-of-compromise into a quorum-of-compromise. I think that's extra credit, though, because almost all of the current schemes have multiple points of compromise. (hey, that gives us a t-shirt version: "MPOC bad! SPOC better! QUOC best!") I think the pip folks (especially dstufft) are keen on moving in this direction, at least partially. They've been working to add signatures to wheel files, and per-project pubkeys to the PyPI entries. But everything I've seen so far is one-wheel-at-a-time. I don't think there's any support yet for whole-project hash-locked requirements.txt -based installs. Using requirements files, and therefore hashes, must be done manually, by downloading a requirements.txt (possibly by doing a git checkout or unpacking a tarball), then running `pip install -r requirements.txt`. There's no `pip install tahoe-lafs` -like thing that will automatically fetch one for you. So anyways, I suspect that we'd have to be a trail-breaker here. We could prototype this in Tahoe and then help push it upstream into pip. I agree that this wants to live somewhere larger than just Tahoe, but it's not there yet, and ya gotta start somewhere.