integrity failure in mutable retrieve (SDMF+MDMF) in 1.9.0 #1654

New issue

Closed

opened 2012-01-09 04:44:37 +00:00 by warner · 7 comments

warner commented

2012-01-09 04:44:37 +00:00

This is the ticket where we will track the details of a failure of assurance of integrity. For now, here is the announcement:

Dear Tahoe-LAFS Users:

Kevan Carstensen of the Tahoe-LAFS core team has discovered a security
vulnerability in Tahoe-LAFS v1.9.0 which would allow a sufficiently clever
attacker to corrupt the retrieval of mutable files or directories which are
retrieved with v1.9.0 or, in some cases, to corrupt the stored copy of
mutable files or directories which are updated with v1.9.0.

The recommended defensive action for all users is to downgrade to v1.8.3, or
to refrain from using mutable files (either SDMF or MDMF) with 1.9.0.

A FAQ about downgrading from 1.9.0 to 1.8.3, which was written before we
discovered this critical security vulnerability, is here:

https://tahoe-lafs.org/pipermail/tahoe-dev/2011-December/006905.html

The FAQ is no longer accurate about 1.9.0 being free of dangerous flaws, but
it is still accurate about 1.8.3 being free of compatibility problems.

We'll be providing a patch soon. We are still writing tests for it and
searching for other similar bugs and so on. Of course, as soon as we release
the patch, this will inform any attackers of exactly what they could do to
users of 1.9.0. Therefore, if there are any users who are especially
security-sensitive, then they should downgrade to 1.8.3 before we release the
patch, or else they should suspend their use of mutable files and directories
until we released the patch and they've applied it.

Once we are ready to publish the details of the issue we will post them to
this issue tracker ticket:

https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1654

Please feel free to contact me with any questions or concerns, using GPG
encryption. Please Cc: Brian Warner, David-Sarah Hopwood, and Kevan
Carstensen on all such email.

Regards,

Zooko Wilcox-O'Hearn, on behalf of the Tahoe-LAFS core team


GPG fingerprints:

Brian Warner <warner-tahoe@lothar.com> 967E FE06 6998 7241 1A77  DF36 D43B 4C9C 7322 5AAF
David-Sarah Hopwood <david-sarah@jacaranda.org> 3D6A 08E9 1262 3E9A 00B2  1BDC 067F 4920 98CF 2762
Kevan Carstensen <kevan@isnotajoke.com> 7E1E 99DB 97B1 DD5F 8154  5973 8E6B 2106 2425 D7AE
Zooko Wilcox-O'Hearn <zooko@zooko.com> A60B 7EE1 7164 D0C5 F137  3868 5F22 F428 242B E85F

This is the ticket where we will track the details of a failure of assurance of integrity. For now, here is the announcement: ``` Dear Tahoe-LAFS Users: Kevan Carstensen of the Tahoe-LAFS core team has discovered a security vulnerability in Tahoe-LAFS v1.9.0 which would allow a sufficiently clever attacker to corrupt the retrieval of mutable files or directories which are retrieved with v1.9.0 or, in some cases, to corrupt the stored copy of mutable files or directories which are updated with v1.9.0. The recommended defensive action for all users is to downgrade to v1.8.3, or to refrain from using mutable files (either SDMF or MDMF) with 1.9.0. A FAQ about downgrading from 1.9.0 to 1.8.3, which was written before we discovered this critical security vulnerability, is here: https://tahoe-lafs.org/pipermail/tahoe-dev/2011-December/006905.html The FAQ is no longer accurate about 1.9.0 being free of dangerous flaws, but it is still accurate about 1.8.3 being free of compatibility problems. We'll be providing a patch soon. We are still writing tests for it and searching for other similar bugs and so on. Of course, as soon as we release the patch, this will inform any attackers of exactly what they could do to users of 1.9.0. Therefore, if there are any users who are especially security-sensitive, then they should downgrade to 1.8.3 before we release the patch, or else they should suspend their use of mutable files and directories until we released the patch and they've applied it. Once we are ready to publish the details of the issue we will post them to this issue tracker ticket: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1654 Please feel free to contact me with any questions or concerns, using GPG encryption. Please Cc: Brian Warner, David-Sarah Hopwood, and Kevan Carstensen on all such email. Regards, Zooko Wilcox-O'Hearn, on behalf of the Tahoe-LAFS core team GPG fingerprints: Brian Warner <warner-tahoe@lothar.com> 967E FE06 6998 7241 1A77 DF36 D43B 4C9C 7322 5AAF David-Sarah Hopwood <david-sarah@jacaranda.org> 3D6A 08E9 1262 3E9A 00B2 1BDC 067F 4920 98CF 2762 Kevan Carstensen <kevan@isnotajoke.com> 7E1E 99DB 97B1 DD5F 8154 5973 8E6B 2106 2425 D7AE Zooko Wilcox-O'Hearn <zooko@zooko.com> A60B 7EE1 7164 D0C5 F137 3868 5F22 F428 242B E85F ```

warner added the

labels 2012-01-09 04:44:37 +00:00

warner added this to the undecided milestone 2012-01-09 04:44:37 +00:00

zooko added

p/supercritical

and removed

p/major

labels 2012-01-09 06:54:32 +00:00

zooko modified the milestone from undecided to 1.9.1

2012-01-09 06:54:32 +00:00

zooko commented

2012-01-09 06:55:02 +00:00

Attachment 1.9.0-integrity-fail-warning.txt.asc (10253 bytes) added

**Attachment** 1.9.0-integrity-fail-warning.txt.asc (10253 bytes) added

1.9.0-integrity-fail-warning.txt.asc

10 KiB

warner commented

2012-01-09 08:10:44 +00:00

Author

fixed formatting of announcement, removed pubkeys, removed signature wrapper (not verifiable here)

daira added

c/code-mutable

and removed

c/unknown

labels 2012-01-09 18:05:33 +00:00

zooko modified the milestone from 1.9.2 to 1.9.1

2012-01-11 13:27:38 +00:00

Brian Warner <warner@lothar.com> commented

2012-01-12 23:18:08 +00:00

Owner

In changeset:d10f260f36a9da6f:

retrieve.py: unconditionally check share-hash-tree. Fixes #1654.

Add Kevan's unit test, update known_issues.rst

In changeset:d10f260f36a9da6f: ``` retrieve.py: unconditionally check share-hash-tree. Fixes #1654. Add Kevan's unit test, update known_issues.rst ```

tahoe-lafs added the

r/fixed

label 2012-01-12 23:18:08 +00:00

Brian Warner <warner@lothar.com> closed this issue

2012-01-12 23:18:08 +00:00

Brian Warner <warner@lothar.com> commented

2012-01-12 23:19:02 +00:00

Owner

In changeset:d10f260f36a9da6f:

retrieve.py: unconditionally check share-hash-tree. Fixes #1654.

Add Kevan's unit test, update known_issues.rst

In changeset:d10f260f36a9da6f: ``` retrieve.py: unconditionally check share-hash-tree. Fixes #1654. Add Kevan's unit test, update known_issues.rst ```

warner commented

2012-01-12 23:32:33 +00:00

Author

Dear Tahoe-LAFS Users:

On 08-Jan-2012, Tahoe-LAFS core member Kevan Carstensen (author of the
MDMF code) discovered a serious bug in v1.9.0 (the current stable
release) that allows attackers to corrupt downloads of mutable files in
certain cases. We've released Tahoe-LAFS v1.9.1 which removes this
vulnerability. All users are encouraged to upgrade immediately to
v1.9.1, or to downgrade to v1.8.3.

v1.9.0 was released about two months ago. As far as we know, ArchLinux
is the only distribution to have packaged v1.9.0 (the others are still
on v1.8.3, which is safe). So if you get your Tahoe-LAFS through a
non-ArchLinux package, you're probably fine. If you build it yourself,
you should upgrade.

In Tahoe, files are encrypted, and then encoded into multiple redundant
shares. Integrity-checking information (Merkle hash trees) are included
in the shares to detect corruption. When downloading, these hashes are
checked before combining the shares in the decoder, which generates
ciphertext that can be decrypted into the original file. Mutable files
have two sets of hash trees, the "share hash tree" (which covers all
shares), and the "block hash trees" (which sit under the share-hash-tree
and cover the individual blocks that make up each share, one block per
128KiB segment of the original file).

The new mutable downloader released in v1.9.0, which supports both the
old-style SDMF format and the new MDMF format, has a bug in which the
share-hash-tree check is accidentaly bypassed when the Merkle hash tree
is already fully populated. This doesn't normally occur, but shares can
contain additional hash-tree nodes beyond the ones they strictly need.
An attacker could modify one share to include the entire tree, then
change the block data in the remaining shares. They would need to update
the block-hash-trees in those doctored shares, but because of the bug,
these tree roots will not be compared against the share hash tree.

The attacker is thus able to control the input to the ZFEC decoder for
all but the first share received (which must have valid block data).
This gives them the ability to flip bits of the plaintext without
triggering the CorruptShareError exceptions that share corruption would
normally produce, causing corrupted plaintext to be delivered to an
unwitting client.

To exploit this bug, the attacker must be able to deliver multiple
modified shares to your client, in a particular order: this means they
must control one or more of your storage servers.

Note that this does not directly reveal the plaintext to the attacker
(this is an integrity failure, not a confidentiality failure). However,
"encryption without authentication" is never a safe state of affairs,
and can frequently be exploited to reveal information about the
plaintext (perhaps by inducing observable failures by flipping bits in
messages of a known format). In addition, clients which read corrupted
data as part of a read-modify-write operation (such as directory
modifications) may then write the corrupted data back out to the file,
making the corruption persist even after the client has been fixed.

v1.9.1 fixes this by removing the accidental "if" clause, making the
share-hash-tree check unconditional.

The specific bug is in source:src/allmydata/mutable/retrieve.py,
Retrieve._validate_block, around the call to
share_hash_tree.set_hashes(), and was introduced in git revisionid
ac3b2647dd2c45cd1ddbf5b130ee5a780c66c73b with the MDMF-capable
downloader rewrite around 01-Aug-2011. The bug was first present in
shipping code in Tahoe-LAFS-1.9.0, on 30-Oct-2011. It was fixed in
commit [9b4b03a474a2c9050c8347459ab6698839be7288], shipped in
Tahoe-LAFS-1.9.1 on 12-Jan-2012. We are continuing to audit the 1.9.x
mutable downloader code to search for other potential bugs.

Dear Tahoe-LAFS Users: On 08-Jan-2012, Tahoe-LAFS core member Kevan Carstensen (author of the MDMF code) discovered a serious bug in v1.9.0 (the current stable release) that allows attackers to corrupt downloads of mutable files in certain cases. We've released Tahoe-LAFS v1.9.1 which removes this vulnerability. All users are encouraged to upgrade immediately to v1.9.1, or to downgrade to v1.8.3. v1.9.0 was released about two months ago. As far as we know, [ArchLinux](wiki/ArchLinux) is the only distribution to have packaged v1.9.0 (the others are still on v1.8.3, which is safe). So if you get your Tahoe-LAFS through a non-ArchLinux package, you're probably fine. If you build it yourself, you should upgrade. In Tahoe, files are encrypted, and then encoded into multiple redundant shares. Integrity-checking information (Merkle hash trees) are included in the shares to detect corruption. When downloading, these hashes are checked before combining the shares in the decoder, which generates ciphertext that can be decrypted into the original file. Mutable files have two sets of hash trees, the "share hash tree" (which covers all shares), and the "block hash trees" (which sit under the share-hash-tree and cover the individual blocks that make up each share, one block per 128KiB segment of the original file). The new mutable downloader released in v1.9.0, which supports both the old-style SDMF format and the new MDMF format, has a bug in which the share-hash-tree check is accidentaly bypassed when the Merkle hash tree is already fully populated. This doesn't normally occur, but shares can contain additional hash-tree nodes beyond the ones they strictly need. An attacker could modify one share to include the entire tree, then change the block data in the remaining shares. They would need to update the block-hash-trees in those doctored shares, but because of the bug, these tree roots will not be compared against the share hash tree. The attacker is thus able to control the input to the ZFEC decoder for all but the first share received (which must have valid block data). This gives them the ability to flip bits of the plaintext without triggering the [CorruptShareError](wiki/CorruptShareError) exceptions that share corruption would normally produce, causing corrupted plaintext to be delivered to an unwitting client. To exploit this bug, the attacker must be able to deliver multiple modified shares to your client, in a particular order: this means they must control one or more of your storage servers. Note that this does not directly reveal the plaintext to the attacker (this is an integrity failure, not a confidentiality failure). However, "encryption without authentication" is never a safe state of affairs, and can frequently be exploited to reveal information about the plaintext (perhaps by inducing observable failures by flipping bits in messages of a known format). In addition, clients which read corrupted data as part of a read-modify-write operation (such as directory modifications) may then write the corrupted data back out to the file, making the corruption persist even after the client has been fixed. v1.9.1 fixes this by removing the accidental "if" clause, making the share-hash-tree check unconditional. The specific bug is in source:src/allmydata/mutable/retrieve.py, Retrieve._validate_block, around the call to share_hash_tree.set_hashes(), and was introduced in git revisionid ac3b2647dd2c45cd1ddbf5b130ee5a780c66c73b with the MDMF-capable downloader rewrite around 01-Aug-2011. The bug was first present in shipping code in Tahoe-LAFS-1.9.0, on 30-Oct-2011. It was fixed in commit [9b4b03a474a2c9050c8347459ab6698839be7288], shipped in Tahoe-LAFS-1.9.1 on 12-Jan-2012. We are continuing to audit the 1.9.x mutable downloader code to search for other potential bugs.

gdt commented

2012-01-13 01:16:08 +00:00

Owner

As far as we know, ArchLinux? is the only distribution to have packaged v1.9.0 (the others are still on v1.8.3, which is safe). So if you get your Tahoe-LAFS through a non-ArchLinux? package, you're probably fine.

pkgsrc also packaged 1.9.0. (However, pkgsrc isn't a "distribution", in that it is not a Linux kernel and a group of user-space programs. Instead, it is a packaging system that supports multiple operating systems.)
The tahoe-lafs entry in pkgsrc was updated about 2011-01-13T0115Z to 1.9.1.

> As far as we know, [ArchLinux](wiki/ArchLinux)? is the only distribution to have packaged v1.9.0 (the others are still on v1.8.3, which is safe). So if you get your Tahoe-LAFS through a non-ArchLinux? package, you're probably fine. pkgsrc also packaged 1.9.0. (However, pkgsrc isn't a "distribution", in that it is not a Linux kernel and a group of user-space programs. Instead, it is a packaging system that supports multiple operating systems.) The tahoe-lafs entry in pkgsrc was updated about 2011-01-13T0115Z to 1.9.1.