<!--{{{-->
<link rel='alternate' type='application/rss+xml' title='RSS' href='index.xml' />
<!--}}}-->

Background: #fff
Foreground: #000
PrimaryPale: #8cf
PrimaryLight: #18f
PrimaryMid: #04b
PrimaryDark: #014
SecondaryPale: #ffc
SecondaryLight: #fe8
SecondaryMid: #db4
SecondaryDark: #841
TertiaryPale: #eee
TertiaryLight: #ccc
TertiaryMid: #999
TertiaryDark: #666
Error: #f88

/*{{{*/
body {background:[[ColorPalette::Background]]; color:[[ColorPalette::Foreground]];}

a {color:[[ColorPalette::PrimaryMid]];}
a:hover {background-color:[[ColorPalette::PrimaryMid]]; color:[[ColorPalette::Background]];}
a img {border:0;}

h1,h2,h3,h4,h5,h6 {color:[[ColorPalette::SecondaryDark]]; background:transparent;}
h1 {border-bottom:2px solid [[ColorPalette::TertiaryLight]];}
h2,h3 {border-bottom:1px solid [[ColorPalette::TertiaryLight]];}

.button {color:[[ColorPalette::PrimaryDark]]; border:1px solid [[ColorPalette::Background]];}
.button:hover {color:[[ColorPalette::PrimaryDark]]; background:[[ColorPalette::SecondaryLight]]; border-color:[[ColorPalette::SecondaryMid]];}
.button:active {color:[[ColorPalette::Background]]; background:[[ColorPalette::SecondaryMid]]; border:1px solid [[ColorPalette::SecondaryDark]];}

.header {background:[[ColorPalette::PrimaryMid]];}
.headerShadow {color:[[ColorPalette::Foreground]];}
.headerShadow a {font-weight:normal; color:[[ColorPalette::Foreground]];}
.headerForeground {color:[[ColorPalette::Background]];}
.headerForeground a {font-weight:normal; color:[[ColorPalette::PrimaryPale]];}

.tabSelected{color:[[ColorPalette::PrimaryDark]];
	background:[[ColorPalette::TertiaryPale]];
	border-left:1px solid [[ColorPalette::TertiaryLight]];
	border-top:1px solid [[ColorPalette::TertiaryLight]];
	border-right:1px solid [[ColorPalette::TertiaryLight]];
}
.tabUnselected {color:[[ColorPalette::Background]]; background:[[ColorPalette::TertiaryMid]];}
.tabContents {color:[[ColorPalette::PrimaryDark]]; background:[[ColorPalette::TertiaryPale]]; border:1px solid [[ColorPalette::TertiaryLight]];}
.tabContents .button {border:0;}

#sidebar {}
#sidebarOptions input {border:1px solid [[ColorPalette::PrimaryMid]];}
#sidebarOptions .sliderPanel {background:[[ColorPalette::PrimaryPale]];}
#sidebarOptions .sliderPanel a {border:none;color:[[ColorPalette::PrimaryMid]];}
#sidebarOptions .sliderPanel a:hover {color:[[ColorPalette::Background]]; background:[[ColorPalette::PrimaryMid]];}
#sidebarOptions .sliderPanel a:active {color:[[ColorPalette::PrimaryMid]]; background:[[ColorPalette::Background]];}

.wizard {background:[[ColorPalette::PrimaryPale]]; border:1px solid [[ColorPalette::PrimaryMid]];}
.wizard h1 {color:[[ColorPalette::PrimaryDark]]; border:none;}
.wizard h2 {color:[[ColorPalette::Foreground]]; border:none;}
.wizardStep {background:[[ColorPalette::Background]]; color:[[ColorPalette::Foreground]];
	border:1px solid [[ColorPalette::PrimaryMid]];}
.wizardStep.wizardStepDone {background:[[ColorPalette::TertiaryLight]];}
.wizardFooter {background:[[ColorPalette::PrimaryPale]];}
.wizardFooter .status {background:[[ColorPalette::PrimaryDark]]; color:[[ColorPalette::Background]];}
.wizard .button {color:[[ColorPalette::Foreground]]; background:[[ColorPalette::SecondaryLight]]; border: 1px solid;
	border-color:[[ColorPalette::SecondaryPale]] [[ColorPalette::SecondaryDark]] [[ColorPalette::SecondaryDark]] [[ColorPalette::SecondaryPale]];}
.wizard .button:hover {color:[[ColorPalette::Foreground]]; background:[[ColorPalette::Background]];}
.wizard .button:active {color:[[ColorPalette::Background]]; background:[[ColorPalette::Foreground]]; border: 1px solid;
	border-color:[[ColorPalette::PrimaryDark]] [[ColorPalette::PrimaryPale]] [[ColorPalette::PrimaryPale]] [[ColorPalette::PrimaryDark]];}

.wizard .notChanged {background:transparent;}
.wizard .changedLocally {background:#80ff80;}
.wizard .changedServer {background:#8080ff;}
.wizard .changedBoth {background:#ff8080;}
.wizard .notFound {background:#ffff80;}
.wizard .putToServer {background:#ff80ff;}
.wizard .gotFromServer {background:#80ffff;}

#messageArea {border:1px solid [[ColorPalette::SecondaryMid]]; background:[[ColorPalette::SecondaryLight]]; color:[[ColorPalette::Foreground]];}
#messageArea .button {color:[[ColorPalette::PrimaryMid]]; background:[[ColorPalette::SecondaryPale]]; border:none;}

.popupTiddler {background:[[ColorPalette::TertiaryPale]]; border:2px solid [[ColorPalette::TertiaryMid]];}

.popup {background:[[ColorPalette::TertiaryPale]]; color:[[ColorPalette::TertiaryDark]]; border-left:1px solid [[ColorPalette::TertiaryMid]]; border-top:1px solid [[ColorPalette::TertiaryMid]]; border-right:2px solid [[ColorPalette::TertiaryDark]]; border-bottom:2px solid [[ColorPalette::TertiaryDark]];}
.popup hr {color:[[ColorPalette::PrimaryDark]]; background:[[ColorPalette::PrimaryDark]]; border-bottom:1px;}
.popup li.disabled {color:[[ColorPalette::TertiaryMid]];}
.popup li a, .popup li a:visited {color:[[ColorPalette::Foreground]]; border: none;}
.popup li a:hover {background:[[ColorPalette::SecondaryLight]]; color:[[ColorPalette::Foreground]]; border: none;}
.popup li a:active {background:[[ColorPalette::SecondaryPale]]; color:[[ColorPalette::Foreground]]; border: none;}
.popupHighlight {background:[[ColorPalette::Background]]; color:[[ColorPalette::Foreground]];}
.listBreak div {border-bottom:1px solid [[ColorPalette::TertiaryDark]];}

.tiddler .defaultCommand {font-weight:bold;}

.shadow .title {color:[[ColorPalette::TertiaryDark]];}

.title {color:[[ColorPalette::SecondaryDark]];}
.subtitle {color:[[ColorPalette::TertiaryDark]];}

.toolbar {color:[[ColorPalette::PrimaryMid]];}
.toolbar a {color:[[ColorPalette::TertiaryLight]];}
.selected .toolbar a {color:[[ColorPalette::TertiaryMid]];}
.selected .toolbar a:hover {color:[[ColorPalette::Foreground]];}

.tagging, .tagged {border:1px solid [[ColorPalette::TertiaryPale]]; background-color:[[ColorPalette::TertiaryPale]];}
.selected .tagging, .selected .tagged {background-color:[[ColorPalette::TertiaryLight]]; border:1px solid [[ColorPalette::TertiaryMid]];}
.tagging .listTitle, .tagged .listTitle {color:[[ColorPalette::PrimaryDark]];}
.tagging .button, .tagged .button {border:none;}

.footer {color:[[ColorPalette::TertiaryLight]];}
.selected .footer {color:[[ColorPalette::TertiaryMid]];}

.sparkline {background:[[ColorPalette::PrimaryPale]]; border:0;}
.sparktick {background:[[ColorPalette::PrimaryDark]];}

.error, .errorButton {color:[[ColorPalette::Foreground]]; background:[[ColorPalette::Error]];}
.warning {color:[[ColorPalette::Foreground]]; background:[[ColorPalette::SecondaryPale]];}
.lowlight {background:[[ColorPalette::TertiaryLight]];}

.zoomer {background:none; color:[[ColorPalette::TertiaryMid]]; border:3px solid [[ColorPalette::TertiaryMid]];}

.imageLink, #displayArea .imageLink {background:transparent;}

.annotation {background:[[ColorPalette::SecondaryLight]]; color:[[ColorPalette::Foreground]]; border:2px solid [[ColorPalette::SecondaryMid]];}

.viewer .listTitle {list-style-type:none; margin-left:-2em;}
.viewer .button {border:1px solid [[ColorPalette::SecondaryMid]];}
.viewer blockquote {border-left:3px solid [[ColorPalette::TertiaryDark]];}

.viewer table, table.twtable {border:2px solid [[ColorPalette::TertiaryDark]];}
.viewer th, .viewer thead td, .twtable th, .twtable thead td {background:[[ColorPalette::SecondaryMid]]; border:1px solid [[ColorPalette::TertiaryDark]]; color:[[ColorPalette::Background]];}
.viewer td, .viewer tr, .twtable td, .twtable tr {border:1px solid [[ColorPalette::TertiaryDark]];}

.viewer pre {border:1px solid [[ColorPalette::SecondaryLight]]; background:[[ColorPalette::SecondaryPale]];}
.viewer code {color:[[ColorPalette::SecondaryDark]];}
.viewer hr {border:0; border-top:dashed 1px [[ColorPalette::TertiaryDark]]; color:[[ColorPalette::TertiaryDark]];}

.highlight, .marked {background:[[ColorPalette::SecondaryLight]];}

.editor input {border:1px solid [[ColorPalette::PrimaryMid]];}
.editor textarea {border:1px solid [[ColorPalette::PrimaryMid]]; width:100%;}
.editorFooter {color:[[ColorPalette::TertiaryMid]];}

#backstageArea {background:[[ColorPalette::Foreground]]; color:[[ColorPalette::TertiaryMid]];}
#backstageArea a {background:[[ColorPalette::Foreground]]; color:[[ColorPalette::Background]]; border:none;}
#backstageArea a:hover {background:[[ColorPalette::SecondaryLight]]; color:[[ColorPalette::Foreground]]; }
#backstageArea a.backstageSelTab {background:[[ColorPalette::Background]]; color:[[ColorPalette::Foreground]];}
#backstageButton a {background:none; color:[[ColorPalette::Background]]; border:none;}
#backstageButton a:hover {background:[[ColorPalette::Foreground]]; color:[[ColorPalette::Background]]; border:none;}
#backstagePanel {background:[[ColorPalette::Background]]; border-color: [[ColorPalette::Background]] [[ColorPalette::TertiaryDark]] [[ColorPalette::TertiaryDark]] [[ColorPalette::TertiaryDark]];}
.backstagePanelFooter .button {border:none; color:[[ColorPalette::Background]];}
.backstagePanelFooter .button:hover {color:[[ColorPalette::Foreground]];}
#backstageCloak {background:[[ColorPalette::Foreground]]; opacity:0.6; filter:'alpha(opacity:60)';}
/*}}}*/

/*{{{*/
* html .tiddler {height:1%;}

body {font-size:.75em; font-family:arial,helvetica; margin:0; padding:0;}

h1,h2,h3,h4,h5,h6 {font-weight:bold; text-decoration:none;}
h1,h2,h3 {padding-bottom:1px; margin-top:1.2em;margin-bottom:0.3em;}
h4,h5,h6 {margin-top:1em;}
h1 {font-size:1.35em;}
h2 {font-size:1.25em;}
h3 {font-size:1.1em;}
h4 {font-size:1em;}
h5 {font-size:.9em;}

hr {height:1px;}

a {text-decoration:none;}

dt {font-weight:bold;}

ol {list-style-type:decimal;}
ol ol {list-style-type:lower-alpha;}
ol ol ol {list-style-type:lower-roman;}
ol ol ol ol {list-style-type:decimal;}
ol ol ol ol ol {list-style-type:lower-alpha;}
ol ol ol ol ol ol {list-style-type:lower-roman;}
ol ol ol ol ol ol ol {list-style-type:decimal;}

.txtOptionInput {width:11em;}

#contentWrapper .chkOptionInput {border:0;}

.externalLink {text-decoration:underline;}

.indent {margin-left:3em;}
.outdent {margin-left:3em; text-indent:-3em;}
code.escaped {white-space:nowrap;}

.tiddlyLinkExisting {font-weight:bold;}
.tiddlyLinkNonExisting {font-style:italic;}

/* the 'a' is required for IE, otherwise it renders the whole tiddler in bold */
a.tiddlyLinkNonExisting.shadow {font-weight:bold;}

#mainMenu .tiddlyLinkExisting,
	#mainMenu .tiddlyLinkNonExisting,
	#sidebarTabs .tiddlyLinkNonExisting {font-weight:normal; font-style:normal;}
#sidebarTabs .tiddlyLinkExisting {font-weight:bold; font-style:normal;}

.header {position:relative;}
.header a:hover {background:transparent;}
.headerShadow {position:relative; padding:4.5em 0em 1em 1em; left:-1px; top:-1px;}
.headerForeground {position:absolute; padding:4.5em 0em 1em 1em; left:0px; top:0px;}

.siteTitle {font-size:3em;}
.siteSubtitle {font-size:1.2em;}

#mainMenu {position:absolute; left:0; width:10em; text-align:right; line-height:1.6em; padding:1.5em 0.5em 0.5em 0.5em; font-size:1.1em;}

#sidebar {position:absolute; right:3px; width:16em; font-size:.9em;}
#sidebarOptions {padding-top:0.3em;}
#sidebarOptions a {margin:0em 0.2em; padding:0.2em 0.3em; display:block;}
#sidebarOptions input {margin:0.4em 0.5em;}
#sidebarOptions .sliderPanel {margin-left:1em; padding:0.5em; font-size:.85em;}
#sidebarOptions .sliderPanel a {font-weight:bold; display:inline; padding:0;}
#sidebarOptions .sliderPanel input {margin:0 0 .3em 0;}
#sidebarTabs .tabContents {width:15em; overflow:hidden;}

.wizard {padding:0.1em 1em 0em 2em;}
.wizard h1 {font-size:2em; font-weight:bold; background:none; padding:0em 0em 0em 0em; margin:0.4em 0em 0.2em 0em;}
.wizard h2 {font-size:1.2em; font-weight:bold; background:none; padding:0em 0em 0em 0em; margin:0.4em 0em 0.2em 0em;}
.wizardStep {padding:1em 1em 1em 1em;}
.wizard .button {margin:0.5em 0em 0em 0em; font-size:1.2em;}
.wizardFooter {padding:0.8em 0.4em 0.8em 0em;}
.wizardFooter .status {padding:0em 0.4em 0em 0.4em; margin-left:1em;}
.wizard .button {padding:0.1em 0.2em 0.1em 0.2em;}

#messageArea {position:fixed; top:2em; right:0em; margin:0.5em; padding:0.5em; z-index:2000; _position:absolute;}
.messageToolbar {display:block; text-align:right; padding:0.2em 0.2em 0.2em 0.2em;}
#messageArea a {text-decoration:underline;}

.tiddlerPopupButton {padding:0.2em 0.2em 0.2em 0.2em;}
.popupTiddler {position: absolute; z-index:300; padding:1em 1em 1em 1em; margin:0;}

.popup {position:absolute; z-index:300; font-size:.9em; padding:0; list-style:none; margin:0;}
.popup .popupMessage {padding:0.4em;}
.popup hr {display:block; height:1px; width:auto; padding:0; margin:0.2em 0em;}
.popup li.disabled {padding:0.4em;}
.popup li a {display:block; padding:0.4em; font-weight:normal; cursor:pointer;}
.listBreak {font-size:1px; line-height:1px;}
.listBreak div {margin:2px 0;}

.tabset {padding:1em 0em 0em 0.5em;}
.tab {margin:0em 0em 0em 0.25em; padding:2px;}
.tabContents {padding:0.5em;}
.tabContents ul, .tabContents ol {margin:0; padding:0;}
.txtMainTab .tabContents li {list-style:none;}
.tabContents li.listLink { margin-left:.75em;}

#contentWrapper {display:block;}
#splashScreen {display:none;}

#displayArea {margin:1em 17em 0em 14em;}

.toolbar {text-align:right; font-size:.9em;}

.tiddler {padding:1em 1em 0em 1em;}

.missing .viewer,.missing .title {font-style:italic;}

.title {font-size:1.6em; font-weight:bold;}

.missing .subtitle {display:none;}
.subtitle {font-size:1.1em;}

.tiddler .button {padding:0.2em 0.4em;}

.tagging {margin:0.5em 0.5em 0.5em 0; float:left; display:none;}
.isTag .tagging {display:block;}
.tagged {margin:0.5em; float:right;}
.tagging, .tagged {font-size:0.9em; padding:0.25em;}
.tagging ul, .tagged ul {list-style:none; margin:0.25em; padding:0;}
.tagClear {clear:both;}

.footer {font-size:.9em;}
.footer li {display:inline;}

.annotation {padding:0.5em; margin:0.5em;}

* html .viewer pre {width:99%; padding:0 0 1em 0;}
.viewer {line-height:1.4em; padding-top:0.5em;}
.viewer .button {margin:0em 0.25em; padding:0em 0.25em;}
.viewer blockquote {line-height:1.5em; padding-left:0.8em;margin-left:2.5em;}
.viewer ul, .viewer ol {margin-left:0.5em; padding-left:1.5em;}

.viewer table, table.twtable {border-collapse:collapse; margin:0.8em 1.0em;}
.viewer th, .viewer td, .viewer tr,.viewer caption,.twtable th, .twtable td, .twtable tr,.twtable caption {padding:3px;}
table.listView {font-size:0.85em; margin:0.8em 1.0em;}
table.listView th, table.listView td, table.listView tr {padding:0px 3px 0px 3px;}

.viewer pre {padding:0.5em; margin-left:0.5em; font-size:1.2em; line-height:1.4em; overflow:auto;}
.viewer code {font-size:1.2em; line-height:1.4em;}

.editor {font-size:1.1em;}
.editor input, .editor textarea {display:block; width:100%; font:inherit;}
.editorFooter {padding:0.25em 0em; font-size:.9em;}
.editorFooter .button {padding-top:0px; padding-bottom:0px;}

.fieldsetFix {border:0; padding:0; margin:1px 0px 1px 0px;}

.sparkline {line-height:1em;}
.sparktick {outline:0;}

.zoomer {font-size:1.1em; position:absolute; overflow:hidden;}
.zoomer div {padding:1em;}

* html #backstage {width:99%;}
* html #backstageArea {width:99%;}
#backstageArea {display:none; position:relative; overflow: hidden; z-index:150; padding:0.3em 0.5em 0.3em 0.5em;}
#backstageToolbar {position:relative;}
#backstageArea a {font-weight:bold; margin-left:0.5em; padding:0.3em 0.5em 0.3em 0.5em;}
#backstageButton {display:none; position:absolute; z-index:175; top:0em; right:0em;}
#backstageButton a {padding:0.1em 0.4em 0.1em 0.4em; margin:0.1em 0.1em 0.1em 0.1em;}
#backstage {position:relative; width:100%; z-index:50;}
#backstagePanel {display:none; z-index:100; position:absolute; width:90%; margin:0em 3em 0em 3em; padding:1em 1em 1em 1em;}
.backstagePanelFooter {padding-top:0.2em; float:right;}
.backstagePanelFooter a {padding:0.2em 0.4em 0.2em 0.4em;}
#backstageCloak {display:none; z-index:20; position:absolute; width:100%; height:100px;}

.whenBackstage {display:none;}
.backstageVisible .whenBackstage {display:block;}
/*}}}*/

/***
StyleSheet for use when a translation requires any css style changes.
This StyleSheet can be used directly by languages such as Chinese, Japanese and Korean which need larger font sizes.
***/
/*{{{*/
body {font-size:0.8em;}
#sidebarOptions {font-size:1.05em;}
#sidebarOptions a {font-style:normal;}
#sidebarOptions .sliderPanel {font-size:0.95em;}
.subtitle {font-size:0.8em;}
.viewer table.listView {font-size:0.95em;}
/*}}}*/

/*{{{*/
@media print {
#mainMenu, #sidebar, #messageArea, .toolbar, #backstageButton, #backstageArea {display: none ! important;}
#displayArea {margin: 1em 1em 0em 1em;}
/* Fixes a feature in Firefox 1.5.0.2 where print preview displays the noscript content */
noscript {display:none;}
}
/*}}}*/

<!--{{{-->
<div class='header' macro='gradient vert [[ColorPalette::PrimaryLight]] [[ColorPalette::PrimaryMid]]'>
<div class='headerShadow'>
<span class='siteTitle' refresh='content' tiddler='SiteTitle'></span>&nbsp;
<span class='siteSubtitle' refresh='content' tiddler='SiteSubtitle'></span>
</div>
<div class='headerForeground'>
<span class='siteTitle' refresh='content' tiddler='SiteTitle'></span>&nbsp;
<span class='siteSubtitle' refresh='content' tiddler='SiteSubtitle'></span>
</div>
</div>
<div id='mainMenu' refresh='content' tiddler='MainMenu'></div>
<div id='sidebar'>
<div id='sidebarOptions' refresh='content' tiddler='SideBarOptions'></div>
<div id='sidebarTabs' refresh='content' force='true' tiddler='SideBarTabs'></div>
</div>
<div id='displayArea'>
<div id='messageArea'></div>
<div id='tiddlerDisplay'></div>
</div>
<!--}}}-->

<!--{{{-->
<div class='toolbar' macro='toolbar [[ToolbarCommands::ViewToolbar]]'></div>
<div class='title' macro='view title'></div>
<div class='subtitle'><span macro='view modifier link'></span>, <span macro='view modified date'></span> (<span macro='message views.wikified.createdPrompt'></span> <span macro='view created date'></span>)</div>
<div class='tagging' macro='tagging'></div>
<div class='tagged' macro='tags'></div>
<div class='viewer' macro='view text wikified'></div>
<div class='tagClear'></div>
<!--}}}-->

<!--{{{-->
<div class='toolbar' macro='toolbar [[ToolbarCommands::EditToolbar]]'></div>
<div class='title' macro='view title'></div>
<div class='editor' macro='edit title'></div>
<div macro='annotations'></div>
<div class='editor' macro='edit text'></div>
<div class='editor' macro='edit tags'></div><div class='editorFooter'><span macro='message views.editor.tagPrompt'></span><span macro='tagChooser excludeLists'></span></div>
<!--}}}-->

To get started with this blank TiddlyWiki, you'll need to modify the following tiddlers:
* SiteTitle & SiteSubtitle: The title and subtitle of the site, as shown above (after saving, they will also appear in the browser title bar)
* MainMenu: The menu (usually on the left)
* DefaultTiddlers: Contains the names of the tiddlers that you want to appear when the TiddlyWiki is opened
You'll also need to enter your username for signing your edits: <<option txtUserName>>

These InterfaceOptions for customising TiddlyWiki are saved in your browser

Your username for signing your edits. Write it as a WikiWord (eg JoeBloggs)

<<option txtUserName>>
<<option chkSaveBackups>> SaveBackups
<<option chkAutoSave>> AutoSave
<<option chkRegExpSearch>> RegExpSearch
<<option chkCaseSensitiveSearch>> CaseSensitiveSearch
<<option chkAnimate>> EnableAnimations

----
Also see [[AdvancedOptions]]

<<importTiddlers>>

Yesterday I posted [[a note|http://allmydata.org/pipermail/tahoe-dev/2008-July/000729.html]] to tahoe-dev about garbage collection and did some work on [[my paper for StorageSS08|http://allmydata.org/pipermail/tahoe-dev/2008-July/000716.html]].  It turns out that they've changed the due date from August 3 to August 18.  Great!  Now I'm starting to work on it and it isn't the last minute!

Today I'm going to pair up with Brian, either on the paper or on immutable file checking and repair (and possibly improved download), or perhaps on both.

I'm investigating [[this Debian/Ubuntu bug|https://bugs.launchpad.net/debian/+source/python-setuptools/+bug/254035]], the better to make Brian accept Tahoe's reliance on setuptools.  We could work around this Debian/Ubuntu bug by using only our own bundled copy of setuptools that comes with Tahoe, but I'm sure Brian would be more accepting of it if the version of setuptools that came with Debian/Ubuntu worked...

I'm feeling overwhelmed with things I Ought To Do.  The top priority is new checker/verifier/repairer for immutable files for Tahoe -- it is, unless I'm forgetting something -- the last feature that needs to be implemented for the Tahoe 1.3 release.

But I also have a hard deadline of August 18 for the final version of [[my paper for StorageSS08|http://allmydata.org/pipermail/tahoe-dev/2008-July/000716.html]].

Per some of [[the reviewers' suggestions|http://allmydata.org/pipermail/tahoe-dev/2008-July/000656.html]] I'm (re-)reading some of the storage papers by the estimable [[David Mazières|http://www.scs.stanford.edu/~dm]], starting with [[the SUNDR paper|http://www.scs.stanford.edu/~dm/home/papers/li:sundr.pdf]].

Also some hackers have started [[revitalizing the darcs project|http://lists.osuosl.org/pipermail/darcs-users/2008-August/012939.html]], and I have some responsibilities, to do with building Windows binaries of darcs, but more-over to do with automation: automated testing, automated building of binaries on all supported platforms, and automated performance measurement.  Unfortunately I'm not going to be able to contribute anything to this project until at least after August 18.

Of course there are also one zillion other things that I need to, want to, or ought to do ASAP...

I [[offered|http://mail.python.org/pipermail/python-dev/2008-August/081758.html]] some ascii-encoding code of mine to the Python project.

I updated a [[darcs code browser web site|http://allmydata.org/trac/darcs-2/browser]] so that the darcs folks can link to it from [[the darcs home page|http://darcs.net]].

I contributed [[a patch|http://mail.python.org/pipermail/distutils-sig/2008-August/009815.html]] for the [[setuptools|http://peak.telecommunity.com/DevCenter/setuptools]] documents.

I updated a couple of tickets on [[http://allmydata.org|http://allmydata.org]]: [[#456 (it would be nice if the dependency on OpenSSL could be automatically resolved)|http://allmydata.org/trac/tahoe/ticket/456#comment:6]] and [[#402 (bug in Twisted, triggered by pyOpenSSL-0.7)|http://allmydata.org/trac/tahoe/ticket/402#comment:18]].

I worked on reproducing [[this problem|http://allmydata.org/pipermail/tahoe-dev/2008-August/000736.html]] with installing Tahoe on Mac OS X.  I'm pretty sure that the problem in Tahoe-1.2.0 was already fixed by [[this patch|http://allmydata.org/trac/tahoe/changeset/2803]], but in attempting to reproduce David Evans's experience to be sure before I wrote back to him I ran into a related bug in the Mac OS X packaging of pyOpenSSL.

So I helped the [[pyOpenSSL|https://launchpad.net/pyopenssl]] developers run unit tests of pyOpenSSL on my Mac: [[https://bugs.launchpad.net/pyopenssl/+bug/236170|https://bugs.launchpad.net/pyopenssl/+bug/236170 #  test failures on Mac OS X with openssl-0.9.8[gh] ]].  Here is the [[pyOpenSSL buildbot|http://buildbot.twistedmatrix.com/waterfall-pyopenssl]].

I chatted quite a bit with ~RobK, the allmydata, UK arm.  I encouraged him to spend some time creating beautiful, eye-popping, richly informative visualizations of Tahoe.

I queried Brian Warner about exactly what he wants from darcs and took notes.  Tomorrow I'll post those notes to the darcs folks.  Maybe Brian will get what he wants out of darcs.  Or maybe someone else will eventually benefit from those kinds of improvements even if it is too late for Tahoe to do.  Or if that doesn't work then at the very least there will be a good document showing what it was about darcs that was unsatisfactory.

Okay, it is now time for me to head home and prepare for aikido class, and I've done exactly nothing on the two major urgent important tasks that I mentioned at the beginning of the day: ~StorageSS08 paper and new file checking/verifier/repair.  I think I should plan some complete blanket block on distractions tomorrow -- no e-mail, no IRC, no IM.  If Brian (or anyone) wants to talk to me about ~StorageSS08 paper or about new file checking/verifier/repair then they'll have to call me...  Oh wait, I still [[don't have phone service|http://www.dailycamera.com/news/2008/aug/08/south-boulder-phone-repair-70-percent-finished]].  I guess they'll have to call my wife's cell phone and ask her to come find me and tell me to call them back.

Good morning, world!  It is 7:10 AM in Boulder, Colorado and the sun is shining!

My wife has some meetings and classes today, so I am going to be responsible for children in the morning and in the evening.  I'll probably just tell the boys to run and play and then get work done.  They are a lot better at entertaining themselves for hours at a time than they were last summer.

School starts next week -- [[Boulder Valley School District calendar|http://www.bvsd.org/calendar/Pages/default.aspx]].

I updated http://allmydata.org/buildbot-darcs to always redirect to http://buildbot.darcs.net, which is a nicer URL and which also correctly serves up [[its master.cfg file|http://<b style="color: black; background-color: rgb(255, 255, 102);">buildbot</b>.darcs.net/master.cfg]].  This was because [[Petr|http://is.muni.cz/lide/?fakulta=1433;obdobi=2825;studium=150956;jazyk=en;uco=139761]] [[Rockai|http://behindkde.org/people/mornfall/]] tried to access the master.cfg from the old URL and it gave him an error.  Hopefully the reason he is looking at that master.cfg is in order to automate more [[darcs|http://darcs.net]] processes such as building binaries and running benchmarks.

I see that my Mac OS 10.4 builder is now green on [[the pyOpenSSL buildbot|http://buildbot.twistedmatrix.com/waterfall-pyopenssl]].

I updated [[Twisted ticket #2234 (Select default reactor based on platform and available libraries)|http://twistedmatrix.com/trac/ticket/2234#comment:6]], about what reactor Twisted should use if the user didn't specify any preference, and incidentally about integrating trial with setuptools.

I read some of the recent [[darcs-users mailing list traffic|http://lists.osuosl.org/pipermail/darcs-users/2008-August/date.html]].  There are a lot of people eager to improve darcs in various ways.  I was reminded of an earlier [[post from Brian Warner|http://allmydata.org/pipermail/tahoe-dev/2007-November/000228.html]] explaining what he wants from his revision control tool.  (I was reminded of it because it was hyperlinked from [[the darcs wiki|http://wiki.darcs.net]].)  In order to communicate his needs to the darcs developers, I probably just need to link to that letter and then reiterate the essential points, which he expressed succinctly in IRC conversation yesterday.

Finally closed [[ticket #11 (I don't like pyOpenSSL)|http://allmydata.org/trac/tahoe/ticket/11]].

I noticed that the darcs network tests are hanging on two solaris boxes and an ~OpenBSD box -- see [[the darcs buildbot|http://buildbot.darcs.net/waterfall]] for tall yellow stripes.  One of those solaris boxes is mine, so I'm going to try upgrading the [[libcurl|http://curl.haxx.se/]] on that box and see if that changes the behavior.

I opened [[a new ticket|http://bugs.darcs.net/issue991]] about a bug in darcs -- stack space overflow when you try to "darcs put" 10,000 patches over ssh.

In the attempt to get an instant messaging client with [[OTR|http://www.cypherpunks.ca/otr]] I installed [[Adium|http://www.adiumx.com]] on my Macbook Pro and [[kopete|http://kopete.kde.org]] on my Linux laptop.  Kopete didn't work for me, so I opened [[a bug report|https://bugs.launchpad.net/ubuntu/+source/kopete-plugin-otr-kde4/+bug/257390]] for it on launchpad and installed [[pidgin|http://www.pidgin.im]] which did work.

I installed [[iTerm|http://iterm.sourceforge.net]] on my Macbook Pro.  Hopefully this will end my persistent problems with Terminal.app, such as cut and paste eating the last character of every line and/or eating the line feed chars.

Okay, now I need to figure out how to integrate this blog with the internal allmydata.com employees' blog so that my co-workers can easily see what I am doing.

Oh, yesterday I finished re-reading [[SUNDR|http://www.scs.cs.nyu.edu/~dm/papers/li:sundr.pdf]].  I didn't actually understand the key mechanism to enable concurrency while retaining fork-consistency, but I don't think I really need to in order to correctly cite it in [[my StorageSS08 paper|http://allmydata.org/pipermail/tahoe-dev/2008-July/000656.html]].

I opened a ticket on the darcs issue tracker to record Brian's desire for short, secure, efficient version identifiers.  I think it is a valuable feature, and that it is more feasible for darcs than people think.  [[http://bugs.darcs.net/issue992 (short, secure, fast version identifiers)|http://bugs.darcs.net/issue992]]

I made a small note about [[issue #348 (BuildBot step to run tests from package)|http://allmydata.org/trac/tahoe/ticket/348#comment:5]].

The openssl developers [[were arguing|http://www.mail-archive.com/openssl-dev@openssl.org/msg24270.html]] about how to be compatible with valgrind debugging while minimizing the chances of another [["Debian OpenSSL debacle"|http://google.com/search?q=debian+openssl+debacle]] or other problems.  I subscribed to the openssl-dev list just to post [[a suggestion|http://marc.info/?l=openssl-dev&m=121856771027223&w=2]] for what seems like the best way to do it.

Ooh, Danny O'Brien wrote back and asked if I would mind if he wrote some documentation for Tahoe that was less technical than what we've got now.  Would I MIND?  Whoo!  That would be great!

[[Professor James Plank|http://www.cs.utk.edu/~plank]] and his student Katie Schuman have done some benchmarks of open source erasure coding software.  Here are [[their results so far|http://www.cs.utk.edu/~plank/plank/papers/CS-08-625.html]].  Zfec comes out looking pretty good.  I need to write a few suggestions for them on their paper.

Yesterday I wrote [[a letter to Jeremy Fitzhardinge on tahoe-dev|http://allmydata.org/pipermail/tahoe-dev/2008-August/000743.html]] about convergent encryption in Tahoe, the "Chosen Protocol Attack" (Kelsey, Schneier, Wagner), and the Tiger hash.  And I wrote to [img[Danny O'Brien|/file/URI%3ACHK%3Av3rks3qwih4ietrtzwrfuur4ru%3Akz3gcobjest4bdr2yugwphk26bzx3beb3t6baqvusn2eo3smw5ja%3A3%3A10%3A108574/@@named=/danny_obrien_on_cover_of_wired.png]] <-- Danny O'Brien urging him to write docs for Tahoe and suggesting that he contact my business leader at [[allmydata.com|http://allmydata.com]], [img[Peter Secor|http://www.npost.com/images/interviews/peter_secor.jpg]] <-- Peter Secor.

One advantage of having Danny O'Brien write the user docs for Tahoe is that they'll probably turn out [[a lot funnier|http://www.ntk.net]] than the ones [img[Zooko being silly|/file/URI:CHK:ywcrkt5vm3jiwdtyb43n4kgfpu:qydseoirjn5kbpoydgptlaim2spas7a6fsbhdcg6umj2f3sqqdpq:3:10:49576/@@named=/Zooko_being_silly.jpg]] <-- I write.

Today I learned how to inline images into tiddlywiki.

I spent some time working darcs today -- chatting with darcs hackers who are very active in the #darcs channel on irc.freenode.net nowadays, debugged an issue with the latest libcurl and the latest darcs on solaris, and urged someone (Eric Kow) to plan a darcs-2.0.3 release in order to get the latest improvements out there and to exercise the machinery of non-~David-Roundy-led releases.  I really ought to test out Gwern's cabalization patches on Windows soon.

Okay now I would really like to explore using FND's ~HTTPSavingPlugin so that I can use modern non-hacked ~TiddlyWiki code on my tahoe-hosted blog, but instead I'm going to work on [[my StorageSS08 paper|http://allmydata.org/pipermail/tahoe-dev/2008-July/000716.html]].

[[agl|http://www.imperialviolet.org]] wrote to tell me about his Curve22519 implementation.  On request, he explained his reasons to choose Curve25519 instead of FIPS curves.  He says that other curves than Curve25519 ''may'' require validation of the public key before computation of ~Diffie-Hellman and ''may'' be insecure if you use the same public key for many ~Diffie-Hellman agreements.  I think and hope that he is wrong about both, but I need to investigate more...

Hooray!  [[Allmydata.com|http://www.allmydata.com/index.php?tracking=zookos_hlog]] just released the new native Windows client for Tahoe.  It is called Allmydata 3.1.  If you or someone you know uses Windows, check it out!

Good morning, World!

It is 8:54 here, and it looks like another beautiful day.  So far this morning, I've done a bunch of home administration work (with ~OpenOffice, Firefox, ~XEmacs, and Darcs -- maybe adding Gnucash in the future), some chatting on the #darcs IRC channel and opened [[a ticket on the darcs bug tracker|http://bugs.darcs.net/issue997]] showing them example code from Tahoe of how to program your buildbot to compile and upload packages.

Yay!  The new [[LWN.net|http://lwn.net]] is here!

Followed [[Danny O'Brien's blog|http://www.oblomovka.com]] to [[Danny O'Brien's delicious links|http://delicious.com/malaclyps]] to [[Wes Felter's blog|http://wmf.editthispage.com]].  Oh dear -- I need to stop reading the Internet and starting [[writing|http://zooko.com/lafs.pdf]].

I got up early this morning.

I'm making some progress on [[the paper|http://zooko.com/lafs.pdf]].  There are a lot more useful [[reviewer comments|http://allmydata.org/pipermail/tahoe-dev/2008-July/000656.html]] that I need to use.

I printed out [[Efficient byzantine-tolerant erasure-coded storage (2004)|http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.10.2568]], which I probably need to cite in lafs.pdf, and I wrote [[a note to tahoe-dev|http://allmydata.org/pipermail/tahoe-dev/2008-August/000756.html]] about [[A Performance Comparison of Open-Source Erasure Coding Libraries for Storage Applications|http://www.cs.utk.edu/~plank/plank/papers/CS-08-625.html]].

I chatted quite a bit with ~R-K, the mysterious off-shore hacker.  I urged him to dive right in and write a Mac client for Tahoe, doing it just like he wants in terms of technology (Cocoa) and features (whatever features he likes), instead of spending time formulating a business requirements document.  And I urged him to Release Early, Release Often and make it open source.

Because, you know, Time is of the Essence, and cranking out some cool hack in the way that you like is a good strategy.

Hooray!  The paper for [[StorageSS'08|http://storagess.org]] is finished!

[[http://zooko.com/lafs.pdf|http://zooko.com/lafs.pdf]]

I submitted [[a patch to setuptools|http://bugs.python.org/setuptools/issue14]].

I updated [[dupfilefind|http://allmydata.org/trac/dupfilefind]] and [[posted about it|http://allmydata.org/pipermail/tahoe-dev/2008-August/000760.html]].

* Worked on getting the pyOpenSSL project to [[build and host binaries|https://bugs.launchpad.net/pyopenssl/+bug/238658]] so that people can install Tahoe without a C compiler and ~OpenSSL development headers.
* Helped the Debian developer fellow, Micah, post to the tahoe-dev list.
* Helped the darcs folks set up [[a buildslave on NetBSD|http://buildbot.darcs.net/waterfall]] and tried to figure out why the darcs buildslave on our Windows virtual machine gets hung and can't kill the darcs process.
* Read the latest news from [[the SHA-3 competition|http://ehash.iaik.tugraz.at/wiki/The_SHA-3_Zoo]].  Boole is broken!  ~CubeHash is analyzed by two external teams!  ~EnRUPT is updated!
* Tweaked [[the spreadsheet|http://docs.google.com]] -- I had incorrectly told Peter that the HP DL 185 G5 cost a mere $1650, which would reduce our "Storage cost per GB per year for user space" from $0.94 to $0.34.  The actual [[cost of the HP when properly configured|http://allmydata.org/~zooko/hp-DL185-G5-configured.html]] is $2028, which apparently changes our storage cost per GB per year for user space from $0.34 to $0.35.  On the other hand, we can get Seagate Barracuda 7200.11 1.5 TB for [[around $150 each|http://storagereview.pricegrabber.com/search_getprod.php/masterid=88928483/st=product_tab]], which puts it back down to $0.33.  Also the Redundancy percentage was set to 101% instead of its current value of 333%, but that makes no difference as the other numbers were calculated to allow for 300%.  I wrote to my favorite system builder [[Kevin Chalker|http://kc-computers.com]] to ask if he could build rackmount systems with a ~SATA-drives-per-dollar as high as the HP.
* Brian pointed out that [[zbase32|http://pypi.python.org/pypi/zbase32]] didn't work on Python 2.6 because of the new reserved status of the token "as".  I uploaded a fix.
* Hooray!!  [[My hlog|http://tahoebs1.allmydata.com:8123/uri/URI:DIR2-RO:hgvn7nhforxhfxbx3nbej53qoi:yhbnnuxl4o2hr4sxuocoi735t6lcosdin72axkrcboulfslwbfwq/wiki.html]] is back!!
* See also [[my old-fashioned, centralized blog|https://zooko.com/log.html]].

This morning I read Scott Aaronson's research statement about [[computational complexity theory and quantum computing|http://www.scottaaronson.com/research.pdf]].  I understood little of it, except that our secure hashes need to have 3n bits for n-bit resistance to collisions, in case quantum computers ever work out.  I also read the ~SHA-3 submission for the [[SWIFFTX|http://www.eecs.harvard.edu/~alon/PAPERS/lattices/swifftx.pdf]] -- it is interesting because being able to break a random variant of SWIFFTX would entail that you can solve a hard problem in algebraic number theory.  But it is too big and slow for my purposes -- it looks like you would have to use thousands of bits of state to compute it at all, and even if you can use a desktop CPU with SIMD, it still takes 57 cycles per byte.

My plan to organize a Boulder Hack Fest while Jake Appelbaum is in town hasn't worked out because it is Thanksgiving.  I'm looking into having a Hack Fest the following week.

This morning I made sure that the Windows buildslave for darcs was running and posted [[about the plan|http://lists.osuosl.org/pipermail/darcs-users/2008-November/016025.html]] for releasing a new version of darcs for Windows.

I checked on the status  of my tickets on the divmod trac: [[nevow #2798 (setup.py install --home is broken :-()|http://divmod.org/trac/ticket/2798]], [[nevow #2699 (build nevow without importing nevow)|http://divmod.org/trac/ticket/2699]], [[nevow #2698 (please mail me when my tickets change)|http://divmod.org/trac/ticket/2698]], [[nevow #2629 (Nevow doesn't declare its dependency on Twisted in a machine-parseable way)|http://divmod.org/trac/ticket/2629]], [[nevow #2527 (easy_install compatibility)|http://divmod.org/trac/ticket/2527]]. They are all blocked waiting for someone else to do something.  Same with the pyOpenSSL binaries ticket [[pyopenssl #238658 (please provide binaries)|https://bugs.launchpad.net/pyopenssl/+bug/238658]].

I spent a minute forlornly wishing that someone other than me would spend the time to fix [[buildbot #212 (buildbot doesn't respond to darcs tags)|http://buildbot.net/trac/ticket/212]].

I made a new release of [[pycryptopp|http://allmydata.org/trac/pycryptopp]] -- v0.5.11.  The improvement over v0.5.10 is that the buggy ecdsa wrapper has been commented out.

I helped Dan ~McNair set up [[an Arch Linux buildbot for Tahoe|http://allmydata.org/buildbot/builders/Dan%20ArchLinux]] and helped Micah Anderson subscribe to tahoe-dev.

Oh, good, it turns out that the quantum algorithm to find hash collisions, which takes only about 2^(n/3) time, also takes about 2^(n/3) size of quantum computer.  Therefore the overall cost to find collisions in an ideal hash function, whether using classical or quantum computation, is probably about 2^(n/2).  On the other hand there ''is'' something that quantum computers -- if they ever come into existence -- will be able to do cheaper than classical computers, and that is find pre-images of an ideal hash function in a mere 2^(n/2) cost instead of the 2^n cost that a classical computer requires (Grover's Algorithm).

So as long as you use a hash function which prevents people from cheaply finding collisions (in today's, classical, terms), then you'll probably be okay.  This means it is still probably a reasonable strategy for Tahoe to use 192-bit-output hash functions in the next revision of its crypto capability scheme.

Thanks to ~DJBernstein for posting about that topic to the NIST mailing list.

I might go to [[the Boulder Hacking In Public Society|http://hackingsociety.org/chapters/blug]] meeting tonight.

I worked on packaging issues -- older installs of Tahoe could conflict with newer ones.  The fix to that (using the """"--multi-version"""" flag) led to other problems.  At the moment, most of the buildbots are red because of this.

Worked with Chris Galvan on setuptools_trial and packaging.  Worked with Greg Hazel on making darcs work on Windows for him and on making Python 2.6 work for me on Windows.  Worked with Brian Warner on versioning for Tahoe.  Worked on upgrading my hlog (Amber points out that it should be called a "clog") to the new version of Tiddly Wiki and the separate ~HTTPSavingPlugin.  It doesn't work yet.

Things I did yesterday -- 2008-11-24:
* upgrade this klog to use the current version of ~TiddlyWiki and the ~HTTPSavingPlugin thanks to help from FND
* open some tickets about improvements I'd like to see:
** [[tiddly_on_tahoe #1 (it says "saving please wait...done" *after* it is finished saving)|http://allmydata.org/trac/tiddly_on_tahoe/ticket/1]]
** [[tiddly_on_tahoe #2 (don't offer the option to save changes when you are viewing read-only)|http://allmydata.org/trac/tiddly_on_tahoe/ticket/2]]
* spent a couple of minutes helping the darcs hackers wrangle [[the darcs buildbot|http://buildbot.darcs.net]] -- they now have automated uploads of binaries for Solaris and ~OpenBSD
* give Francois Deppierraz an account on the allmydata.com dapper machine so he can work on [[tahoe #534 ("tahoe cp" command encoding issue)|http://allmydata.org/trac/tahoe/ticket/534]]
* wrote a [[letter|http://www.nabble.com/poly1305-p20684551.html]] to Wei Dai about MAC algorithms in Crypto++ (finished that letter and sent it today - -2008-11-25)
* investigated a problem on the Tahoe trac as reported by Nathan Wilcox
* wrote a letter to Ian Goldberg about his advice on how to prove security (and improve security) for [[semi-private keys|http://allmydata.org/~zooko/lafs.pdf]]
* looked at the list of open tickets from my [[21 November 2008]] journal entry -- alas, none of them have been improved by anyone else since the last time I looked
* planned ~ETAs for immutable file repairer:
** ~ValidatedUEB ETA 2008-11-25
** Verifier ETA 2008-11-26
** Repairer ETA 2008-12-02

Things I did so far today -- 2008-11-25:
* chat with Debian Python packagers about using [[my stdeb hacks|https://code.launchpad.net/~astraw/stdeb/autofind-depends]] in [[their find_python_dependencies.py|http://svn.debian.org/viewsvn/*checkout*/python-modules/tools/find_python_dependencies.py]]
* learn how to use proper (~ISO-8601'ish) daystamps on ~TiddlyWiki and to automatically open today's journal entry, thanks to the ever-helpful FND a.k.a. ~Ace_NoOne
* fix the tahoe/pyutil "platform description string" feature to detect Arch Linux correctly, with the help of Daenyth from IRC
* add Dan ~McNair's builder to the list of builders to be run on patch commits and restart [[the tahoe buildbot|http://allmydata.org/buildbot/waterfall?show_events=true]]
* checked whether the version of simplejson could explain [[tahoe #534 ("tahoe cp" command encoding issue)|http://allmydata.org/trac/tahoe/ticket/534]] -- unlikely
* posted [[about TiddlyWiki on Tahoe|http://allmydata.org/pipermail/tahoe-dev/2008-November/000893.html]] to the tahoe-dev list
* chatted with Brian for a few minute about [[tahoe #540 (inappropriate "uncoordinated write error" after handling a server failure)|http://allmydata.org/trac/tahoe/ticket/540]]
* helped Francois fix [[tahoe #534 ("tahoe cp" command encoding issue)|http://allmydata.org/trac/tahoe/ticket/534]]
* try to persuade maintainers of setuptools and setuptools forks to accept my patch to respect PYTHONPATH: [[message to distutils-sig|http://mail.python.org/pipermail/distutils-sig/2008-November/010539.html]] (actually the "Enstaller" setuptools fork already accepted my patches, but that one is not publicly supported at this time -- it used only internally by the Enthought corporation).  Hm, this seems to have triggered at least one setuptools fork to become more active.  See the distutils-sig mailing list archives for details.
* respond to Denis Bonnenfant's [[message to tahoe-dev|http://allmydata.org/pipermail/tahoe-dev/2008-November/000892.html]]
* change the default port number from 8123 to 3456 [[tahoe ticket #536 (port number conflict: 8123 is (or was) used by polipo and is blocked by TorButton)|http://allmydata.org/trac/tahoe/ticket/536]]

Today is our eighth wedding anniversary.

Things I did so far today:

* chatted with Roger Dingledine of Tor about the port conflict
* read about [[the LANE hash function|http://www.cosic.esat.kuleuven.be/lane/moreinfo.shtml]] which was submitted for the ~SHA-3 contest (see [[SHA-3 benchmarks and the importance of 32-bit CPUs]])
* learned about ~ECIES-ISO -- [[Shoup's ISO doc|http://www.shoup.net/iso/std6.pdf]], [[David Hopwood's page|http://www.users.zetnet.co.uk/hopwood/crypto/scan/ca.html#ECIES-ISO]], [[wikipedia page|http://en.wikipedia.org/wiki/Integrated_Encryption_Scheme]]; ~ECIES-ISO is a nice simple scheme compared to [[the ElGamal encryption scheme|http://en.wikipedia.org/wiki/ElGamal_encryption]].
* learned about ~PSEC-KEM and ~ACE-KEM and decided that ~ECIES-ISO would serve Tahoe better (with ~CofactorMode=0, ~OldCofactorMode=0, ~CheckMode=1 and ~SingleHashMode=0)
* With the help of cdent and Jeremy Ruston, I fixed a bug in [[Tiddly-On-Tahoe|http://allmydata.org/trac/tiddly_on_tahoe]] in which it was extra-encoding the body when saving.  Each time I clicked "Save Changes", it escaped all of the previous escape characters, thus doubling the size of the strings encoding encodings of encodings of "David Mazières"; I noticed once the file size was up to 8 MB and it was taking an excessively long time to load and my web browsers were offering to kill the ~JavaScript because it was taking too long.
* noticed that there exists this project named [[CorePy|http://corepy.org]], which is an assembly language programming tool integrated with Python.  They announced their 1.0 release on [[lwn.net|http://lwn.net]] this week.  Mmmm...
* I tried in vain to configure my server -- nooxie.zooko.com -- to send outgoing mail from [[nmh|http://www.nongnu.org/nmh]]; So my quest to free myself from the Apple Mail.app MUA didn't make much progress.
* I did a little bit of work on the ~ValidatedUEB part of repairer -- the step of repairer that was supposed to be done yesterday.  :-(  But at least I got started.  :-)

For my morning's economic education, I listened to [[this week's econtalk|http://www.econtalk.org/archives/2008/11/hazlett_on_tele.html]], Thomas Hazlett on net neutrality, the Microsoft anti-trust case, the original browser wars, etc..  This isn't a good econtalk.  Econtalk doesn't work very well when the guest and the host (Russ Roberts) share biases and don't succeed at critically challenging their own biases.  (This one also seems to be marred by factual errors on the part of Thomas Hazlett -- see the comments on the econtalk blog for extensive discussion.)

If you are new to econtalk try one of the good ones: [[Shirky on Coase, Collaboration and Here Comes Everybody|http://www.econtalk.org/archives/2008/10/shirky_on_coase.html]], [[Patri Friedman on Seasteading|http://www.econtalk.org/archives/2008/10/patri_friedman.html]], [[Kling on Freddie and Fannie and the Recent History of the U.S. Housing Market|http://www.econtalk.org/archives/2008/09/kling_on_freddi.html]], [[Brook on Vermeer's Hat and the Dawn of Global Trade|http://www.econtalk.org/archives/2008/02/brook_on_vermee.html]], [[Hanson on Health|http://www.econtalk.org/archives/2007/05/hanson_on_healt.html]], [[Taleb on Black Swans|http://www.econtalk.org/archives/2007/04/taleb_on_black.html]].

11:00 ~UTC-7

I created a new tiddler: [[things to read]].

I'm learning how to customize ~TiddlyWiki.  Today I learned how to put things on the sidebar over on the right -- the first thing I put there is a blogroll, and the next thing is a link to [[things to read]].

Everything that I've learned about how to use ~TiddlyWiki is due to the friendly ~TiddlyWiki hacker [[FND|http://fnd.lewcid.org/blog]], who answers my questions on the #tiddlywiki IRC channel.

reading web pages:
* [[Matthew Garrett on power management|http://www.codon.org.uk/~mjg59/power/good_practices.html]] (found via [[LWN.net|http://lwn.net]])

There is a big-picture issue that people are missing; should your operating system and CPU be designed to execute tasks as quickly as possible in order to get them over with, or in a slow, foot-dragging way in order to save energy?  Obviously in an ideal world the applications are doing things that users need done, and so obviously the first strategy is the only answer.  However, in the real world many or even most of the tasks are useless, such as busy loops checking "Has anything changed yet?  Has anything changed yet?  Has anything changed yet?" every 100 milliseconds.  Those busy loops are not going to run to completion and then stop, they are going to keep running endlessly, and they are not going to accomplish anything for the user (until the time comes that the answer changes from "No" to "Yes").  If that is the workload, then the second strategy is better, because it can endure the wastage so that there is still some power left when the user wants to actually do something.

The wintel world ([[led by Transmeta|http://en.wikipedia.org/wiki/LongRun]]) seems to have pursued the latter strategy. Linux hackers like Matthew Garrett naturally tend toward the former strategy, along with the injunction the app writers ought to "Fix your apps!".  That's fine too -- //viva la difference!// -- but I wonder what happens when [[the app writers don't|http://lwn.net/Articles/308446]] (Beagle).

This reminds me of the trouble that we had with memory protection and scheduling in the previous generation of operating systems -- Windows < NT and Mac OS < X.  Back then (in the 90's), users would say "Hey my whole system just froze up!", and experts would explain to them "Ah, you need to figure out which of the apps you were running was naughty and tell the author to fix his app!".  These experts would argue that it wasn't the operating system's fault if apps misbehaved like that.

Eventually the makers of the operating systems took responsibility for preserving the user experience //even in the presence of naughty apps//, by enforcing memory protection and pre-emption at the OS level.  Today, the Linux community thinks of battery life as a communal resource that any app can drink from as much as it likes, and therefore they complain about apps which drink too much (see link "the app writers don't", above, about Beagle).  It would be interesting if the users could tell the operating system to stop giving power to Beagle until further notice.  The hard part is how the user can know what they want and communicate it to the OS.  It's a user interface issue.  Imagine a little animation showing different apps sucking juice out of your battery.  The user might notice that the Beagle app is //always// there, draining the battery, and they might choose to gesture to the OS to cut Beagle off and see what happens.

found [[a bug in TiddlyWiki|http://trac.tiddlywiki.org/ticket/373]]

* wrote to Denis Bonnenfant on [[tahoe-dev|http://allmydata.org/pipermail/tahoe-dev/2008-December/date.html]]
* tried and failed to get ~OpenSSL to build on Windows using gcc's -mno-cygwin option to produce native win32 executables; ~OpenSSL's build system is an awful mess of shell and perl scripts. It appears that the developers assumed that if you want to build it for Windows then (a) you have a Microsoft compiler and (b) you don't mind doing a whole bunch of custom steps manually; On the other hand, I have a Free Software compiler, and I would like for it to JUST WORK.  Wouldn't it be nice if the same command, such as "./config && make", would work on both unix //and// Windows?  Argh.  pyOpenSSL's build system is an awful mess of Python scripts, and the rest of my criticism of ~OpenSSL's build system applies equally well to pyOpenSSL's build system.  I was amused to notice that the pyOpenSSL build script has a hardcoded assumption that if you are building pyOpenSSL on Macintosh that you've already installed the //fink// version of ~OpenSSL!  Great.
* complained bitterly about ~OpenSSL and pyOpenSSL and beg someone to build binary eggs for Windows [[launchpad #238658|https://bugs.launchpad.net/pyopenssl/+bug/238658]]
* contribute [[small patch to pyOpenSSL|http://bazaar.launchpad.net/~zooko/pyopenssl/buildbinaries/revision/81]]
* update [[setuptools_darcs|http://allmydata.org/trac/setuptools_darcs/timeline]]
* Ugh.  I'm sick.  'Tis the season; everyone else I know has been sick recently.  Oh well.
* wrote [[a letter to tahoe-dev|http://allmydata.org/pipermail/tahoe-dev/2008-December/000924.html]] about caps-in-~URLs

* spent most of yesterday trying to build ~OpenSSL and pyOpenSSL for Windows using mingw
* read about [[the TIB3 hash function|http://ehash.iaik.tugraz.at/wiki/TIB3]]; I like it because it is a traditional, conservative design, because they explain their motivations in detail in the document, and because it seems to have good performance -- around 8 to 16 cycles per byte on either 64-bit //or// 32-bit ~CPUs (see my earlier note [[SHA-3 benchmarks and the importance of 32-bit CPUs]]).
* listened to [[this week's econtalk|http://www.econtalk.org/archives/2008/12/rauchway_on_the.html]] -- "Rauchway on the Great Depresson and the New Deal"
* posted to openssl-dev about my attempt to compile ~OpenSSL for Windows using mingw
* did a lot more work on ~ValidatedUEB

* 5:00 AM: updated the tickets about packaging pywin32 [[sourceforge #1799934|https://sourceforge.net/tracker2/?func=detail&aid=1799934&group_id=78018&atid=551954]], [[setuptools #18|http://bugs.python.org/setuptools/issue18]], [[twisted #3238|http://twistedmatrix.com/trac/ticket/3238]]
* my [[letter to openssl-dev|http://www.mail-archive.com/openssl-dev@openssl.org/msg24959.html]] finally went through
* posted to tahoe-dev about [[managing port numbers|http://allmydata.org/pipermail/tahoe-dev/2008-December/000928.html]]
* got almost all the tests passing with the new ~ValidatedExtendedURIProxy and ~ValidatedCrypttextHashTreeProxy ; worried about how much longer before Repairer is actually done
* saw this [[cool 5-minute video|http://project.haskell.org/camp/unique]] showing camp (a darcs variant) and git side-by-side with graphical patch trees; Nice!
* If you haven't seen this note of mine before, and you are interested in revision control tools, read this: [[bad merge|https://zooko.com/badmerge/concrete-good-semantics.html]]

* tests finally pass with the new ~ValidatedExtendedURIProxy and ~ValidatedCrypttextHashTreeProxy!
* here's something I want to be able to find/remember later, so I'm mentioning it on my klog: [[JACK Timemachine|http://plugin.org.uk/timemachine]]
* tried to get pyOpenSSL packaged for setuptools for Windows ([[tahoe #456|http://allmydata.org/trac/tahoe/ticket/456]] == [[pyopenssl #238658|https://bugs.launchpad.net/pyopenssl/+bug/238658]])

Beautiful, soft, dry white snow has been drifting down all morning.  The boys and Amber happily shoveled the driveway and sidewalk.  Irby asked if he could please take the snow shovel with him and shovel the sidewalk all the way to school, but I explained that we would be late in that case, and maybe we could try it another day when we left early.

I updated the list of [[issue tickets]] issues in open source projects that I'm contributing to in one way or another.  [[issue tickets]] is always visible in my side bar over there """---->""".

(at the top of the right-hand-side-bar)

Here is my paper on Tahoe: [[lafs.pdf|http://testgrid.allmydata.org:3567/file/URI%3ACHK%3Alqigipzuthqasj6x2f2tsyrwsu%3Akesgqz3z3prnyuzli7hl5vpigbpipjmzzyhtyngvak2wet55rspq%3A3%3A10%3A275101/@@named=/lafs.pdf]].

There is a thick blanket of soft snow lying around the house.  Amber just brought me fresh-baked gingerbread cookies, milk, and coffee.  Elliot has a friend over for a play-date, and the play-date involved baking cookies.

* I wrote [[a letter to distutils-sig|http://mail.python.org/pipermail/distutils-sig/2008-December/010593.html]].  I'm concerned that the fact that Tahoe behaves differently when you pass {{{?t=info}}} (and [[a few other query args|http://allmydata.org/trac/tahoe/browser/docs/frontends/webapi.txt]], when apache just ignored the query arg might inhibit Tahoe being used as a generic web host.
* ~ValidatedExtendedURIProxy and ~ValidatedCrypttextHashTreeProxy pass all tests!  And code coverage is excellent and Brian reviewed the code and I changed most of the things he mentioned.
* I finally got ~OpenSSL and pyOpenSSL working on Windows, and tested out the new setup/build/install process on Windows.  It is almost there -- I just have to decide how to make the tahoe executable plus PYTHONPATH available.

* [[happiness is contagious|http://www.latimes.com/features/health/la-sci-happy5-2008dec05,0,5449915.story]]; This news article contains many interesting assertions.
* We've finally got pyOpenSSL binary eggs for Windows.  Thanks, Chris Galvan!
* worked on the new packaging system for Tahoe, with Chris Galvan's help
* realized that my darcs executable was very slow due to a bug in ~GHC-6.8.2, and tried (with mixed success) to compile ~GHC-6.10.1 on various machines
* learned more about how {{{easy_install}}} parses web pages looking for package files
* fixed [[tahoe #553|http://allmydata.org/trac/tahoe/ticket/553]]: "More Info" link should point to a file/dir, not a dir+childname
* learned about [[transclusion in TiddlyWiki|http://www.tiddlywiki.org/wiki/Including_External_Content]] -- hopefully ~TiddlyWiki on Tahoe with transclusion will make the ultimate decentralized, shared wiki
* finally committed [[refactor handling of URI Extension Block and crypttext hash tree|http://allmydata.org/trac/tahoe/changeset/20081205141754-92b7f-f3ee4370feeab166962456792d95fdca9bb2cfab]]

The next step for repairer: integrate refactored URI Extension Block with my new checker code.

* I updated my patch to Dungeon Crawl to make a better god Xom.  To play it, telnet to {{{crawl.develz.org 345}}} and play the current trunk version.

I updated a few [[issue tickets]]:

[[buildbot #236|http://buildbot.net/trac/ticket/236]]: show elapsed time for steps -- @@fixed@@
[[buildbot #395|http://buildbot.net/trac/ticket/395]]: when i change the vcs executable, buildslave stops being able to invoke it until I restart buildslave
[[buildbot #396|http://buildbot.net/trac/ticket/396]]: Older builds
[[buildbot #252|http://buildbot.net/trac/ticket/252]]: side-effecty operations (Force Builder) should be ~POSTs

Today is Game Day at Chez O'Whielacronx.

I [[complained about Python 3.0|http://lwn.net/Articles/310071]] on lwn.net.

[[tahoe #555|http://allmydata.org/trac/tahoe/ticket/555]]: tahoe .deb cannot be installed on hardy: simplejson dependency is too new
[[tahoe #530|http://allmydata.org/trac/tahoe/ticket/530]]: use setuptools's """--multi-version""" mode
[[tahoe #534|http://allmydata.org/trac/tahoe/ticket/534]]: "tahoe cp" command encoding issue

FND helped me with CSS for this klog.

* using my newly gained knowledge of Tiddly CSS thanks to FND, I made the "side stuff" tiddler at the top of the sidebar prettier and more noticeable
* [[tahoe hacking|http://allmydata.org/trac/tahoe/timeline?from=2008-12-08&daysback=1&ticket=on&ticket_details=on&changeset=on&milestone=on&wiki=on&update=Update]]
** loosened requirements on {{{simplejson}}} and {{{setuptools}}} to ease packaging for Ubuntu gutsy and hardy (this results in a test failure on dapper, but nobody understands why)
** more tidying up of download code in preparation for merging new repairer
** merged new repairer and tested it, but it doesn't pass all tests so it isn't committed to trunk yet
** worked with Brian on refactoring and optimizing download code -- fetch only the needed parts of the Merkle Trees

Allmydata.com has an iPhone product now!  [[the allmydata.com blog|http://www.allmydata.com/blog]]

* I submitted a patch to Twisted: [[twisted #3568|http://twistedmatrix.com/trac/ticket/3568]]: ERROR from conch test when pycrypto is not installed
* I've found a major performance issue in darcs -- sometimes it tries repeatedly to establish a connection to a server, and it waits to see if the server answers, so if that server isn't answering then it takes hours to do anything.  [[darcs #1153|http://bugs.darcs.net/issue1153]]: darcs waits to hear back from servers unnecessarily
* I'm working on a patch for Twisted to avoid {{{repr}}}'ing strings in {{{Failure}}}s: [[twisted #2466|http://twistedmatrix.com/trac/ticket/2466]]: Failures use a lot of memory
* --I'm working on a bug report for ghc-6.10.1, which doesn't build out of the box on my GNU/~OpenSolaris server.-- (Forget it: couldn't reproduce it.)
* Uploaded videos encoded in [[the dirac codec|http://diracvideo.org]] to [[testgrid-shared-directory/video/dirac codec -- use VLC to play|http://testgrid.allmydata.org:3567/uri/URI%3ADIR2%3Adjrdkfawoqihigoett4g6auz6a%3Ajx5mplfpwexnoqff7y5e4zjus4lidm76dcuarpct7cckorh2dpgq/video/dirac codec -- use VLC to play]]
* wrote to [[Kevin Chalker|http://kc-computers.com]] asking how many SATA connectors per dollar he could build
* [[setuptools #17|http://bugs.python.org/setuptools/issue17]]: easy_install will install a package that is already there; This issue should probably be renamed in light of the fact that it seems to cause a worse failure nowadays with the proposed Debian packages for {{{foolscap}}} and {{{tahoe-lafs}}}.

* posted to the NIST ~SHA-3 forum quoting [[SHA-3 benchmarks and the importance of 32-bit CPUs]]
* started tracking this issue: [[tiddlywiki #658|http://trac.tiddlywiki.org/ticket/658]]: ~SiteUrl as current document location
* chatted with FND on the ~TiddlyWiki IRC channel:
** about [[tiddlywiki #658|http://trac.tiddlywiki.org/ticket/658]] ({{{SiteUrl}}} as current document location), which breaks the tiddly_on_tahoe RSS feed in its default setting; fixed the RSS feed of this klog per Peter Secor's bug report.
** about the cryptographic-capabilities-in-~URLs security model (I referenced [[Nathan Wilcox's page on the "Hack Tahoe!" hall of fame|http://hacktahoe.org/nathan_wilcox.html]] and [[this post on cap-talk|http://www.eros-os.org/pipermail/cap-talk/2008-December/011844.html]])
** about ~JavaScript -- he taught me how to use Firebug and I learned some ~JavaScript and fixed [[tiddly_on_tahoe #2|http://allmydata.org/trac/tiddly_on_tahoe/ticket/2]]: don't offer the option to save changes when you are viewing read-only -- @@fixed!@@
* I chatted quite a bit with Kragen Sitaker.  One topic was how [[git|http://git.or.cz]] has wide acceptance and is widely used as a substrate to build new things (such as [[github|http://github.com]], which is a social networking site built on a decentralized revision control tool), and how that came about.  I am hoping to apply some of those lessons to Tahoe.

Amber updated [[her home page|http://www.cs.toronto.edu/~amber]] for the first time since, I think, 2003.

Today I added a bunch of items to my tiddler named [[things to read]]:
* IEEE Spectrum: [[A Fairer, Faster Internet Protocol|http://spectrum.ieee.org/dec08/7027]] (as [[recommended by Wes Felter|http://wmf.editthispage.com/2008/12/04]])
* Ben Laurie, Abe Singer: [[Choose the Red Pill and the Blue Pill|http://www.links.org/files/nspw36.pdf]] (as [[recommended by Wes Felter|http://wmf.editthispage.com/2008/12/08]])
* Ben Laurie, Eric Sachs: [[Usability of Stronger Authentication Options|http://sites.google.com/site/oauthgoog/UXFedLogin/strongauth]] (as [[recommended by Wes Felter|http://wmf.editthispage.com/2008/12/03]])
* Jeff Bonwick (inventor of ZFS) on timeouts (as [[recommended by Wes Felter|http://wmf.editthispage.com/2008/11/29]])
* [[this week's LWN|http://lwn.net/Articles/309663]]

* I'm going to [[Boulder Linux Users Group|http://lug.boulder.co.us]] tonight.
* added ticket: [[tiddly_on_tahoe #3|http://allmydata.org/trac/tiddly_on_tahoe/ticket/3]] (offer a read-only cap to the user)
* finished reading and removed from my [[things to read]]: [[this week's LWN|http://lwn.net/Articles/309663]] (featuring an interview with Vernor Vinge); The "interview" with Vernor Vinge was only mildly interesting -- it sounded like it was excerpts from a much more interesting conversation or speech.
* added [[pycryptopp #12|http://allmydata.org/trac/pycryptopp/ticket/12]] (automatic wrappers for all of Crypto++) to [[issue tickets]]
* read these slides by Josh Berkus on [[Ten Ways To Destroy Your Open-Source Community|http://www.powerpostgresql.com/download/TFCKUpload/25.pdf]] and a few follow-up comments about it: [[Zack Urlocker|http://weblog.infoworld.com/openresource/archives/2008/05/josh_berkus_on.html]], [[Savio Rodrigues|http://weblog.infoworld.com/openresource/archives/2008/05/questioning_jos.html]], [[Josh Berkus again|http://it.toolbox.com/blogs/database-soup/community-destroyers-24309]]
* Here are some more [[things to read]] from Wes Felter's blog.  Does he really read all of these?  He must be a fast reader.
** Greenan, Long, Miller, Schwarz, Wylie: [[A Spin-Up Saved is Energy Earned: Achieving Power-Efficient, Erasure-Coded Storage|http://www.usenix.org/events/hotdep08/tech/full_papers/greenan/greenan_html]]  (as [[recommended by Wes Felter|http://wmf.editthispage.com/2008/12/10]])
** Andreas Merkel, Frank Bellosa: [[Memory-aware Scheduling for Energy Efficiency on Multicore Processors|http://www.usenix.org/events/hotpower08/tech/full_papers/merkel/merkel_html]] (as [[recommended by Wes Felter|http://wmf.editthispage.com/2008/12/10]])
----
Yesterday I suggested to a group of eminent computer security researchers from major research institutions that instead of asking the U.S. Government for money to build a maze inside our farmhouse to protect us from the zombie hordes, they should ask for money to build a forcefield.
----
Here's a funny picture I found via Sameer's blog:
[img[Big Three Advertisement|/file/URI%3ACHK%3Av4kdjf3r7ekuwlcixqcrxxxnde%3Awz6chffifbpggdgsxgytgusgvfipedazpe7v6uopbwk4b5zs4kua%3A3%3A10%3A77968/@@named=/bigthree.jpg/@@named=/BigThree.jpg]] 

* read about [[CubeHash|http://cubehash.cr.yp.to]]
I like it!  It's very simple to understand, has obvious tuning knobs, and offers very good performance (depending on the next few years of cryptanalysis).  The only thing that I don't like about it is that there is no provision for having a smaller state.  Contrast with [[EnRUPT|http://enrupt.com]], which is specified to use a smaller state for smaller outputs, for example the 192-bit output size that I would like to use for tahoe-lafs would use a mere 384-bit state, where for 512-bit output, it would specify to use 1024 bits of state (just like ~CubeHash does).  Requiring fewer bytes of RAM for state can make the algorithm fit into more uses.  My friend Sebastian was skeptical of my claim that 128 bytes of RAM might turn out to be too much to ask for some applications, until I said "The fewer bytes of RAM we need, then //the smaller the motes of dust// that we can run ~SHA-3 on.".  That convinced him.
* finished "Scott Contini, Ron Steinfeld, Josef Pieprzyk, Krystian Matusiewicz: [[A critical look at cryptographic hash function literature (2007)|http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.83.7429]]" with Amber and moved it from [[things to read]] to [[things read]]
It was a good survey and correctly pointed out some theoretical inconsistencies. The most novel suggestion for improvement was to evaluate the security of a hash function in terms of some application of that hash function.  What would such-and-such an application require from its hash function for such-and-such an application to be secure?  It's an interesting idea.  In the context of public-key cryptography secure hash functions are often called upon to do jobs that are easier than the job of being a collision-resistant, pre-image resistant, and second-pre-image resistant.  For example, ~ECIES-ISO needs a ~Key-Derivation Function, and what it needs from that function is merely that if compute the function without letting your enemy know what inputs you are feeding to it, then it doesn't produce outputs that your enemy can guess.  (Oh, and also that if you call it twice with two diversifiers to get two different keys, that it doesn't give you a pair of keys such that if you use one of them for a Message Authentication Code and the other one for a Cipher, that this doesn't somehow ruin the security of your Message Authentication Code and/or your Cipher.  If it were even //possible// for the ~Key-Derivation Function to do this then this would mean that your MAC and/or Cipher were badly flawed, of course.)

What does tahoe-lafs need from its secure hash functions?  Well, we use them in lots of ways, including ~Key-Derivation, just like ~ECIES-ISO does.  One way that tahoe-lafs uses a secure hash function is as an //identifier// of a file.  In that use the traditional requirements of collision-resistance, pre-image resistance and second-pre-image resistance seem just right.  Except for the caveat, as I [[mentioned earlier|today's crypto education]] that what we //really// care about is //confidentiality// which seems to diverge slightly from what cryptographers currently understand as pre-image resistance.  My notion of //confidentiality// of a secure hash function is that I don't want you to be able to find //my pre-image// -- the one I started with -- after I tell you the image.  I don't care if you can pick a random image and then find its pre-image, and I don't care if you can find a //different// pre-image for my image!  (If you do so, then you and I have collectively violated collision resistance, but you haven't violated my confidentiality.)

* the December edition of Bruce Schneier's newsletter [[CryptoGram|http://www.schneier.com/crypto-gram-0812.html]] is out; It seems to be chock full of provocative and interesting observations.  {{{CryptoGram}}} doesn't contain any text that hasn't previously been published on Bruce Schneier's daily blog or in magazine articles, and yet I feel like I get much better value for my time by reading the monthly edition than by reading the daily edition.  I suspect that same improvement could be applied to many sources of news and views.  Nicholas Nassim Taleb mentioned something similar in [[The Black Swan|http://www.amazon.com/exec/obidos/ASIN/1400063515]].
* the ~TahoePlugin for ~TiddlyWiki is almost finished
* I submitted [[The Transitive Grace Period Public Licence, v1.0|https://zooko.com/repos/pyutil/pyutil/COPYING.TGPPL.html]] to the [[Open Source Initiative for approval|http://www.crynwr.com/cgi-bin/ezmlm-cgi?17:mss:462:200812:ofpndmgcgmbhbmimpkpe]].
* Okay! {{{tiddly_on_tahoe}}} is ready for wider use!  The way to get started is to follow [[the instructions to create a tiddly_on_tahoe instance|http://testgrid.allmydata.org:3567/uri/URI:DIR2-RO:7h7syiurogz5erc2au74tjwguu:h7bdxvjtvidlkcdbld3j2d5sbgyzsbqs7wdnu6yznqrejzssc5za/wiki.html]].

* buying/making Christmas presents
* chatting with Chris Galvan about tahoe-lafs packaging

TODO:
* enter Eric Shulman's bug reports about {{{tiddly_on_tahoe}}} into [[http://allmydata.org/trac/tiddly_on_tahoe|http://allmydata.org/trac/tiddly_on_tahoe]] or get him to do so

NEXT:
* announce {{{tiddly_on_tahoe}}} @@finished@@
* work on Repairer
** next step: commit the logging refactoring patch that I was using to debug the new Checker

* created [[tiddly_on_tahoe #7|http://allmydata.org/trac/tiddly_on_tahoe/ticket/7]] (wrong error message when server is unreachable) and added it to [[issue tickets]]
* This morning I thought about making a challenge problem for the ~SHA-3 contest to illustrate [[my ideas about target-pre-image-resistance|today's crypto education]].  I didn't quite figure out how to illustrate it.  Maybe at some point (Christmas vacation?) I'll study [[the LANE hash function|http://www.cosic.esat.kuleuven.be/lane]] more.
* 10:30 Finally finished with morning take-the-children-to-school chores and starting the day's Repairer work.
* Yay!  There's somebody else besides me who thinks that {{{tiddly_on_tahoe}}} is [["completely frickin' brilliant"|http://www.eros-os.org/pipermail/cap-talk/2008-December/011878.html]].
* Today I've been spending a lot of time on the phone with credit card companies persuading them to lower my interest rates.
* added [[tahoe #558|http://allmydata.org/trac/tahoe/ticket/558]] (kpreid says that the -SUMO tarballs don't exist) to [[issue tickets]]

* "How the Barack Obama tapped into a powerful and only [[recently studied human emotion called 'elevation'|http://www.slate.com/id/2205150/pagenum/all/#p2]]." -- from [[Paul Hsieh|http://geekpress.com]]
* been hacking on Verifier/Repairer so far today

* Checker Verifier Repairer Checker Verifier Repairer Checker Verifier Repairer

* added [[twisted 3586|http://twistedmatrix.com/trac/ticket/3586]] (I want to install twisted without a c compiler) to [[issue tickets]]
* working on Download/Checker/Verifier/Repairer
* my brother Josh is here!  Hooray!  :-)

* Christmas party was a big success
* hacking on Downloader/Checker/Verifier/Repairer ; I love how thorough our tests are.

We drove to my mom's house to get there ahead of [[the storm|http://www.wunderground.com/wundermap/?lat=35.19670&lon=-101.84660&zoom=4&type=hyb&units=english&rad=1&rad.num=1&rad.spd=25&rad.opa=70&rad.stm=0&wxsn=1&wxsn.mode=tw&svr=1&svr.opa=70&cams=0&sat=0&riv=0&mm=0&hur=0&fire=0&tor=0&ndfd=0&pix=0]].

I hacked on Downloader/Checker/Verifier/Repairer in the car until I accidentally fell asleep at about 3 AM.  My brother Josh helped me debug.  I spent many hours trying to figure out where in the code was something handling all kinds of Failure and silently dropping it.  (This is the Twisted Python equivalent of catching all kinds of exception and then silently dropping it.)  It turned out to be in the stub Verifier class which is just a place-holder.  This is a class which I have already rewritten but I wasn't using my new version -- I was testing on trunk because I'm trying to test and commit each of my changes in a modular way.  Anyway, it was a frustrating expenditure of hours trying to figure out why exceptions/Failures were vanishing, but at least I got more familiar with the code and the tools.  Also I got to train Josh up on the details of Twisted failure handling.

Note to Python and Twisted programmers: please be very careful about catching all kinds of exception with an except: block and likewise about handling all kinds of Failure with an errback.

* This afternoon I wrote [[a short letter|will SHA-3 replace the current standard secure hash algorithm -- MD5?]] to the cryptographers designing ~SHA-3.
* Added [[buildbot #407|http://buildbot.net/trac/ticket/407]] ({{{darcs_buildbot}}} uses {{{.encode('ascii')}}}, but {{{.encode('utf-8')}}} works better @@patch submitted@@) to [[issue tickets]]
* fixed the tahoe-lafs trac so that drewp and nejucomo can open tickets

Now Terrell can't log into the trac.  Grr.

I'm stumped about unicode handling in the tahoe cli.  I posted [[a plea for help|http://allmydata.org/pipermail/tahoe-dev/2008-December/000948.html]].

It is the day after Christmas.

 * [[The science of shopping|http://www.economist.com/science/displayStory.cfm?story_id=12792420&source=hptextfeature]] by The Economist; Okay people, now is the time to talk about the rules for machines that manipulate the brains of consumers.  Now, while it is still mostly theoretical.
 * http://events.ccc.de/congress/2008/wiki/Distributed_Storage_Grid
 * added [[pyopenssl #311600|https://bugs.launchpad.net/pyopenssl/+bug/311600]] (please update http://pypi.python.org/simple/pyOpenSSL) to [[issue tickets]]
 * added James Hamilton -- [[The Cost of Bulk Cold Storage|http://perspectives.mvdirona.com/2008/12/22/TheCostOfBulkColdStorage.aspx]] as [[recommended by Wes Felter|http://wmf.editthispage.com/2008/12/27]] to [[things to read]]
 * Downloader/Checker/Verifier/Repairer; Wow, we really have some good test coverage here.

* http://events.ccc.de/congress/2008/Fahrplan/events/2875.en.html / [[Tor Bjørstad's slides|http://rosetta.nwo.no/~tor/slides.pdf]]
* http://events.ccc.de/congress/2008/Fahrplan/events/3023.en.html
* added [[foolscap #105|http://foolscap.lothar.com/trac/ticket/105]] //(make it easy to distinguish server-side failures/exceptions from client-side)// to [[issue tickets]]

Today we drive home to Boulder from my mom's farm house in New Mexico.  I stayed up late hacking on Downloader/Checker/Verifier/Repairer and updating release docs.  I'm going to be sleepy!  Maybe I can borrow a pillow from my mom...

* Yep, that CCC presentation was a doozy all right.
* Downloader/Storage/Checker/Verifier/Repairer
* Happy New Year, folks!

Happy New Year!

[[Help Zooko Choose Books]]

* added [[darcs #1298|http://bugs.darcs.net/issue1298]] //darcs failed: Malformed patch bundle: '{' is not 'Context:'// to [[issue tickets]]

It's the 3rd day of the New Year here at Zooko World Headquarters, and it is one day since we posted our literary invitation, and so far nobody has written in to [[Help Zooko Choose Books]].  Act now, during your moment of opportunity!

It occurs to us that many readers might find themselves devoid of opinions about that particular slate of books, and so we hereby open up the voting to any book that you recommend that we can get from amazon.com using our Christmas gift certificate.

* I created a new tiddler: [[collection of failures]].

* {{{<kpreid> guards with authority is a hazardous thing...}}}

* added [[debian #510901|http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=510901]] //python-foolscap: should advertise [secure_connections] feature to setuptools// to [[issue tickets]]

* added [[foolscap #108|http://foolscap.lothar.com/trac/ticket/108]] //set base to "." if not running from source (so {{{flogtool}}} works on Windows)// to [[issue tickets]]
* posted [[a note about the "Hack Tahoe!" contest|http://www.eros-os.org/pipermail/cap-talk/2009-January/011931.html]] in the attempt to contribute to [[the W3C Technical Architecture Group discussion|http://www.w3.org/2001/tag/2008/12/10-minutes]]
* added [[darcs #1303|http://bugs.darcs.net/issue1303]] //proposal: make "darcs changes" interactive by default// to [[issue tickets]]
* added [[ubuntu #314468|https://bugs.launchpad.net/hardy-backports/+bug/314468]] //Please backport setuptools-0.6c9 from Intrepid// to [[issue tickets]]
* responded to Brian's patch review [[on tahoe-dev|http://allmydata.org/pipermail/tahoe-dev/2009-January/thread.html#962]]

* added [[darcs #26|http://bugs.darcs.net/issue26]] //Darcs needs real MIME parsing, fails with Mail.app, Courier// to [[issue tickets]]
* finally got tests passing for download-with-no-decryption; Now hooking together download-with-no-decryption and upload-with-no-encryption should result in Repairer.

Oh boy!  [[CodeCon|http://codecon.org]] is back!  I hope I can arrange to be in San Francisco that week.

Here's an experiment in using {{{tiddly_on_tahoe}}} to collaborate with my brother, Josh: [[biowiki|http://testgrid.allmydata.org:3567/uri/URI:DIR2-RO:5ktwwvxbracbwbdrctmalqhu6e:i6d7l5ylzz4dx3zzxmrtqq7lxzhauz63ufhxnn7ckm2e7squhk4a/biowiki.html]].  Other are invited.

The topic that I want to share notes with him on is the prospect that low-carb dieting makes you age slower, and of course any related topics that attract our attention along the way.

* added //[[Martin C. Atkins|http://www.mca-ltd.com/martin/]] [[An Introduction to Ten15 - A personal retrospective.|http://www.mca-ltd.com/martin/Ten15/introduction.html]]// to [[things to read]]
* Repairer works!  Hooray!  There are a few more tests to fix or to mark TODO, and I'll write a commit message and push it into trunk tomorrow.

* I read [[caja-spec-2008-06-07|http://code.google.com/p/google-caja/downloads/detail?name=caja-spec-2008-06-07.pdf]].  Oh boy -- I'm excited about Caja!  I especially like the idea of Cajita: a nice clean, securable dynamic language which is a subset of ~JavaScript, so it is fully compatible with all of our existing ~JavaScript source code, tools, deployments, and programmer-brains.
* On the subject of [[Help Zooko Choose Books]], [[Myers Carpenter|http://icepick.info]] wrote in to say that he's reading [[Programming Erlang: Software For a Concurrent World|http://amazon.com/exec/obidos/ASIN/193435600X]], which he finds more interesting than Haskell because of the distributed, fault-tolerant, clustering, database features that ship with it.

* started reading [[The Transparent Society|http://books.google.com/books?id=brjIK1dnoYgC&dq=the+transparent+society&printsec=frontcover&source=bn&hl=en&sa=X&oi=book_result&resnum=4&ct=result]] by David Brin
* added [[twisted #2234|http://twistedmatrix.com/trac/ticket/2234]] //Select default reactor based on platform and available libraries// to [[issue tickets]]
* added [[twisted #3529|http://twistedmatrix.com/trac/ticket/3529]] //closing stdout in a child process on cygwin means that process doesn't receive bytes from stdin anymore. I think.// to [[issue tickets]]

* resumed discussion of [[the TGPPL|http://allmydata.org/source/tahoe/trunk/COPYING.TGPPL.html]] on [[the OSI licence-review list|http://www.crynwr.com/cgi-bin/ezmlm-cgi?17:mss:478:200901:ofpndmgcgmbhbmimpkpe]]; The contributors to that list have expressed great reluctance to put the OSI stamp on the TGPPL, not because there is any question about whether it is an open source licence, but because they don't want more open source licences to come into existence.
* added [[darcs #1217|http://bugs.darcs.net/issue1217]] //darcs put fails with 'darcs failed:  Malformed patch bundle: '{' is not 'Context:' '// to [[issue tickets]]

* Last week I said [[darcs is not trying to understand your source code]] and that eventually someone would steal or reinvent the simple idea of darcs merge and implement it in another revision control tool such as git.  Self-fulfilling prophecy -- Miklos Vajna, who is one of the git hackers, saw my post and is [[investigating how to make git do the right thing|http://article.gmane.org/gmane.comp.version-control.git/105748]] on my "badmerge" example.
* Whoo!  Josh gave me an [[OpenMoko Neo FreeRunner|http://wiki.openmoko.org/wiki/Neo_FreeRunner]] for Christmas!  I have no idea what I'm going to do with it.
* [[The $300 Million Button|http://www.uie.com/articles/three_hund_million_button]] as linked from [[Wes Felter's blog|http://wmf.editthispage.com]]
* added [[foolscap #109|http://foolscap.lothar.com/trac/ticket/109]] //make a "flogtool" executable that works on Windows// to [[issue tickets]]
* I updated the tags on this klog.  If you click on the word "Tags" over on the right-hand side you can browse entries by topic.  Also, of course, you can use the search box to find entries by text.  Let me know if it useful!
* This morning I posted to the darcs-users mailing list about [[how you can store a darcs repository on Tahoe|http://lists.osuosl.org/pipermail/darcs-users/2009-January/017199.html]].  :-)

* added //Michal Rjaško: [[Properties of Cryptographic Hash Functions|http://eprint.iacr.org/2008/527]]// to [[things to read]]
* added //Abhishek Parakh, Subhash Kak: [[A Recursive Threshold Visual Cryptography Scheme|http://eprint.iacr.org/2008/535]]// to [[things to read]]; (A good feature of visual cryptography papers is that they always come with pictures!)
* Yesterday I read //Peter Gutmann, [[The Crypto Gardening Guide and Planting Tips|http://www.cs.auckland.ac.nz/~pgut001/pubs/crypto_guide.txt]]//.
* Today I read //James Hamilton, [[The Case For Low-Cost, Low-Power Servers|http://perspectives.mvdirona.com/2009/01/15/TheCaseForLowCostLowPowerServers.aspx]]// (via [[Wes Felter's blog|http://wmf.editthispage.com/2009/01/16]]), and posted a question about it to James Hamilton's blog.
* added //Yehuda Lindell [[Adaptively Secure Two-Party Computation with Erasures|http://eprint.iacr.org/2009/031]]// to [[things to read]]
* added //[[foolscap #111|http://foolscap.lothar.com/trac/ticket/111]] -- timestamps of incident files -- TZ indicator please// to [[issue tickets]]
* added //[[foolscap #112|http://foolscap.lothar.com/trac/ticket/112]] -- timestamps of incident files -- ~ISO-8601'ish// to [[issue tickets]]
* added //[[foolscap #113|http://foolscap.lothar.com/trac/ticket/113]] -- timestamps of incident files -- UTC// to [[issue tickets]]
* added //Kevin D. Bowers, Ari Juels, and Alina Oprea [[HAIL: A High-Availability and Integrity Layer for Cloud Storage|http://eprint.iacr.org/2008/489]]// to [[things to read]]

* packaged darcs-2.2.0 for Windows and posted [[a note|http://lists.osuosl.org/pipermail/darcs-users/2009-January/017272.html]]; The darcs-2.2.0 packages for Windows are hosted on the Tahoe test grid.
* We're going to have a picnic in Martin Park around 15:00 today!
* still reading //[[The Transparent Society|http://books.google.com/books?id=brjIK1dnoYgC&dq=the+transparent+society&printsec=frontcover&source=bn&hl=en&sa=X&oi=book_result&resnum=4&ct=result]] by David Brin//

* posted [[another request|http://www.crynwr.com/cgi-bin/ezmlm-cgi?17:mss:479:200901:ofpndmgcgmbhbmimpkpe]] to the Open Source Initiative about the [[Transitive Grace Period Public Licence|https://zooko.com/tgppl.html]]; I hope to get the stamp of Open Sourciness for the TGPPL as a Valentine's Day present!
* Tim O'Reilly says [[Work On Stuff That Matters|http://radar.oreilly.com/2009/01/work-on-stuff-that-matters-fir.html]]: "Money is like gas in the car -- you need to pay attention or you'll end up on the side of the road -- but a well-lived life is not a tour of gas stations!"
* posted to distutils-sig about [[how Brian really wants his PYTHONPATH back|http://mail.python.org/pipermail/distutils-sig/2009-January/010755.html]] (let's fork setuptools)
* added the following to [[issue tickets]]: 
** //[[tahoe #424|http://allmydata.org/trac/tahoe/ticket/424]]: stdeb: push to upstream//
** //[[tahoe #423|http://allmydata.org/trac/tahoe/ticket/423]]: stdeb: use stdeb on tahoe itself//
** //[[tahoe #422|http://allmydata.org/trac/tahoe/ticket/422]]: stdeb: run from buildslaves//
** //[[setuptools #57|http://bugs.python.org/setuptools/issue57]]: {{{develop}}} doesn't create {{{.pth}}} files and {{{site.py}}} if {{{--multi-version}}}//
** //[[darcs #1255|http://bugs.darcs.net/issue1255]]: darcs put tries to convert to darcs-2-format?// and retired //darcs #1217// from [[issue tickets]] to [[issue tickets closed]]
* organized [[issue tickets]] a bit by adding some hierarchical layout, since it is growing so big
* retired //[[debian #510901|http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=510901]]: python-foolscap: should advertise [secure_connections] feature to setuptools// to [[issue tickets closed]] -- Debian hacker Stephan Peijnik quickly fixed it in response to Brian Warner's request. 
* finally got [[the tahoe-lafs buildbot|http://allmydata.org/buildbot/waterfall?show_events=true]] back to working order; Now it is testing that the instructions in [["Install Tahoe"|http://allmydata.org/source/tahoe/trunk/docs/install.html]] work on all supported platforms, as well as a few other useful things.
* worked with Zandr on getting a Windows machine that JP Calderone can use for //[[pyopenssl #238658|https://bugs.launchpad.net/pyopenssl/+bug/238658]]: package pyOpenSSL for easy_install on Windows//
* added //Gaetan Leurent and Phong Q. Nguyen: [[How Risky is the Random-Oracle Model?|http://eprint.iacr.org/2008/441]]// to [[things I have read]]
* added //Ewan Fleischmann, Christian Forler, and Michael Gorski: [[Classification of the SHA-3 Candidates (2009-09-19 edition)|http://eprint.iacr.org/2008/511]]// to [[things to read]]

yesterday:
* offered to help with //[[pyflakes #2720|http://divmod.org/trac/ticket/2720]]: Release Pyflakes//
* help with //[[buildbot #407|http://buildbot.net/trac/ticket/407]]: {{{darcs_buildbot}}} uses {{{.encode('ascii')}}}, but {{{.encode('utf-8')}}} works better// (by pointing out that {{{utf-8}}} already seems to work)

* Oh, [[this week's EconTalk|http://econtalk.org]] is with Eric S. Raymond.  That's interesting.
* This week's [[LWN|http://lwn.net]] is good, as always.

* [[A Performance Evaluation and Examination of Open-Source Erasure Coding Libraries For Storage|http://www.cs.utk.edu/~plank/plank/papers/FAST-2009.html]], by Jim Plank, Jianqiang Luo, Catherine Schuman, Lihao Xu, and Zooko ~Wilcox-O'Hearn will be presented at [[FAST-2009: 7th USENIX Conference on File and Storage Technologies|http://www.usenix.org/events/fast09]].
* added to [[issue tickets]]:
** //[[nevow #2713|http://divmod.org/trac/ticket/2713]]: setup.py installs tests, but not documentation//
** //[[nevow #2830|http://divmod.org/trac/ticket/2830]]: setup.py incorrectly declares twisted.plugins to be a package//

* Update on my request for the stamp of ~OSD-conformance on [[the Transitive Grace Period Public Licence|http://allmydata.org/source/tahoe/trunk/COPYING.TGPPL.html]] from the Open Source Initiative: I don't know who exactly is on the OSI's Board of Directors, but so far Russell Nelson and Bruce Perens have posted to [[the license-review list|http://www.crynwr.com/cgi-bin/ezmlm-cgi?17:iis:498]] saying that they will not certify the TGPPL as conformant, and questioning my motives.  Neither of them have actually suggested that the TGPPL is not ~OSD-conformant.  Obviously the TGPPL is an open source licence, and neither Nelson nor Perens have suggested otherwise.  Presumably if the OSI officially rejects the licence then it will have to state the reason for rejection.
* added to [[things to read]]:
** //George Bray: [[Obesity: a failure of homeostasis because of hedonic rewards: response to the letter from Gary Taubes|http://testgrid.allmydata.com:3567/file/URI:CHK:q3cwozbit4be2jxmxfre65wzdq:h6rm26yuvcjttexlwhdi6lfcgqoxkordv52xxkwyhxphkytcfuva:3:10:116345/@@named=/Bray_Taubes_rebuttal_rebuttal_rebuttal.pdf]]// (thanks to Gary Taubes for bringing it to my attention as I requested in [[rebuttals to "Good Calories, Bad Calories"]])
** //Bryan Cantrill, and Jeff Bonwick: [[Real-World CONCURRENCY|http://mags.acm.org/queue/200809/?folio=16&CFID=19616687&CFTOKEN=90220359]]// (thanks to Kragen Sitaker for bringing it to my attention)
* I've been trying to use Konqueror from KDE 4.  So far it is performing badly and behaving badly, compared to Firefox-3.
* added to [[issue tickets]]:
** //[[konqueror #321636|https://bugs.launchpad.net/ubuntu/+source/kdebase-kde4/+bug/321636]]: kioslave crashes when logging into my issue tracker//
** //[[konqueror #321656|https://bugs.launchpad.net/ubuntu/+source/kdebase-kde4/+bug/321656]]: iso-8859-1 and/or utf-8 character not decoded properly//
* [[Decentralized Money]]

* fixed a few bugs in {{{allmydata.test.test_runner}}}, most of which only affected Windows: [[[3482]|http://allmydata.org/trac/tahoe/changeset/20090127044046-92b7f-7f2a4a94dd1e18f7397f5ff2900136e1d19ddcb4]], [[[3483]|http://allmydata.org/trac/tahoe/changeset/20090127203245-92b7f-d573b92533ffde9c4b22eefb0e9d4b8eae2cc213]], [[[3484]|http://allmydata.org/trac/tahoe/changeset/20090127203505-92b7f-ee4a6bb8114e1717b62c818cbf08f27a52021415]], [[[3485]|http://allmydata.org/trac/tahoe/changeset/20090127203717-92b7f-4e916910ac3838630fc4c15697371c0d8534af82]]
* created //[[tahoe #596|http://allmydata.org/trac/tahoe/ticket/596]]: storage servers should announce that they support over-read//
* uploaded Twisted binary eggs for Linux to [[the tahoe dependencies repository|http://testgrid.allmydata.org:3567/uri/URI%3ADIR2-RO%3Asnrfwfxatrci35zdgjnzxxx2ke%3Aunarxv347edtku3xzmefy4mcdmfngxzeb72iyqcadbjzjpczjx5a]]
* uploaded ~CGalvan's py 2.4 pyOpenSSL egg to the tahoe dependencies repository
* uploaded zope.interface bdist_eggs for Linux to the tahoe dependencies repository
* got JP Calderone access to a Windows machine (mostly for //[[pyopenssl #238658|https://bugs.launchpad.net/pyopenssl/+bug/238658]]: package pyOpenSSL for easy_install on Windows / please provide binaries//) (thanks to Zandr)
* applied Eric Kow's patches to [[the darcs buildbot|http://buildbot.darcs.net/waterfall]] to build darcs with [[the Haskell cabal tool|http://www.haskell.org/cabal] instead of with autoconf and GNU make
* pushed on the distutils ticket about proceeding after failing to build extension modules: //[[distutils #4706|http://bugs.python.org/issue4706]]: try to build a C module, but don't worry if it doesn't work// (added to [[issue tickets]])
* added to [[things to read]]:
** Daniel J. Bernstein, and Adam Langley [[Crit-bit Trees|http://www.imperialviolet.org/binary/critbit.pdf]]
** Stanford Encyclopedia of Philosophy: [[Bayes' Theorem|http://plato.stanford.edu/entries/bayes-theorem]]

Things That I Could Do (unordered)
* post about hosting eggs on tahoe
* write to Shawn Willden on tahoe-dev about backup designs
* write to Shawn Willden on tahoe-dev about random access read of immutable file contents
* move reactor selection based on sys.platform from tahoe's setup.py to setuptools_trial and push on the Twisted ticket
* fix test failure of share corruption on Windows
* fix that problem with shutdown of tahoe node (?) on Windows (unit test failure)
* upload Twisted binary eggs for more platforms
* build and upload zope.interface bdist_eggs for more platforms
* see about http://www.mozilla.org/projects/netlib/dirindexformat.html and if tahoe wui/wapi should serve it up
* request more binary eggs from zope.interface upstream
* report nejucomo's and dreid's egg boos to distutils-sig
* fix the build failure on hardy-py2.6 due to the {{{#!/usr/bin/env python}}} differing from the Python used to build (seek Chris Galvan's advice about how to fix this)
* fix all those {{{SUCCESS!?!}}} and {{{TODO}}} regarding handling of randomly corrupted shares
* fix //[[tahoe #596|http://allmydata.org/trac/tahoe/ticket/596]]: storage servers should announce that they support over-read//
* enable large_share test on all platforms other than OSX
* enable runner tests on Windows
* fix the deb builders

To Do Next:
* fix the deb builders
Things Done So Far Today:
* posted about [[the limits of reliability estimates|http://allmydata.org/pipermail/tahoe-dev/2009-January/001058.html]] to tahoe-dev
* persuaded Jack Lloyd to let me use [[his code|http://www.randombit.net/bitbashing/programming/forward_error_correction_using_simd.html]] in zfec
* [[fixed a bug|http://allmydata.org/trac/darcsver/changeset/42]] in [[darcsver|http://pypi.python.org/pypi/darcsver]] that Nathan discovered when trying to install tahoe with [[the install.html|http://allmydata.org/source/tahoe/trunk/docs/install.html]], [[fixed another bug|http://allmydata.org/trac/darcsver/changeset/44]] in darcsver, released darcsver-1.1.8, bundled darcsver-1.1.8 with tahoe
* observed that {{{python ./setup.py darcsver}}} takes 55 seconds on this one darcs repository I have; observed that it takes 15 seconds even after running {{{darcs optimize}}} on that repository; submitted a copy of that repository to the darcs folks for their new project of optimizing darcs with automated benchmarking of real-live repositories
* helped the darcs hackers get [[the darcs buildbot|http://buildbot.darcs.net/waterfall]] working better
* [[wrote to Shawn Willden on tahoe-dev|http://allmydata.org/pipermail/tahoe-dev/2009-January/001054.html]] about random access read of immutable file contents
* [[reported nejucomo's and dreid's egg boos to distutils-sig|http://mail.python.org/pipermail/distutils-sig/2009-January/010816.html]]
* opened //[[ubuntu p7zip #322481|https://bugs.launchpad.net/ubuntu/+source/p7zip/+bug/322481]]: took an order of magnitude longer than expected to compress//
* updated the docs and metadata of setuptools_darcs (see [[its timeline|http://allmydata.org/trac/setuptools_darcs/timeline?from=2009-01-28&daysback=1&changeset=on&update=Update]]) and uploaded [[setuptools_darcs-1.2.3|http://pypi.python.org/pypi/setuptools_darcs]]
* updated the docs and metadata of darcsver (see [[its timeline|http://allmydata.org/trac/darcsver/timeline?from=2009-01-28&daysback=1&changeset=on&update=Update]]) and uploaded [[darcsver-1.1.9|http://pypi.python.org/pypi/darcsver]]
* made setuptools_trial specify the poll reactor on linux or cygwin, updated its metadata (see its [[timeline|http://allmydata.org/trac/setuptools_trial/timeline?from=2009-01-28&daysback=1&changeset=on&update=Update]]), and uploaded [[setuptools_trial-0.5|http://pypi.python.org/pypi/setuptools_trial]]
* poked //[[twisted #2234|http://twistedmatrix.com/trac/ticket/2234]]: Select default reactor based on platform and available libraries//
* enabled {{{allmydata.test.test_storage.test_large_share}}} on all platforms

Things That Ought To Be Done for the Tahoe-1.3 release:
* fix test failure of share corruption on Windows
* fix that problem with shutdown of tahoe node (?) on Windows (unit test failure)
* fix the build failure on hardy-py2.6 due to the {{{#!/usr/bin/env python}}} differing from the Python used to build (seek Chris Galvan's advice about how to fix this)
* fix //[[tahoe #596|http://allmydata.org/trac/tahoe/ticket/596]]: storage servers should announce that they support over-read//
* put this list and a few other open tickets onto [[The Roadmap for tahoe-1.3|http://allmydata.org/trac/tahoe/roadmap]]
Things That I Could Do (unordered):
* post about hosting eggs on tahoe
* write to Shawn Willden on tahoe-dev about backup designs
* move reactor selection based on sys.platform from tahoe's setup.py to setuptools_trial and push on the Twisted ticket
* upload Twisted binary eggs for more platforms
* build and upload zope.interface bdist_eggs for more platforms
* see about http://www.mozilla.org/projects/netlib/dirindexformat.html and if tahoe wui/wapi should serve it up
* request more binary eggs from zope.interface upstream
* enable runner tests on Windows
* fix all those {{{SUCCESS!?!}}} and {{{TODO}}} regarding handling of randomly corrupted shares

Good morning.

* [[This week's econtalk|http://www.econtalk.org/archives/2009/01/roberts_and_han.html]] is excellent so far.  Russ Roberts is having a crisis of faith. Enlisting Robin Hanson as the patient confessor, he turns his practice of economic analysis to focus on... his practice of economic analysis!  I love this kind of thing.  I've listened to only half of it so far.  Will Russ reaffirm his faith in the free market?  Will Robin say something which devastates the belief systems of all listeners in a single vast wave of disillusionment?  Tune in to find out.
* [[This week's LWN|http://lwn.net/Articles/316190]] reports on Simon Phipp's speech about "the third wave of free software".
* added to [[issue tickets]]:
** //[[http://buildbot.net/trac/ticket/266]]: I wish to tell my buildmaster: "restart yourself the next time you quiesce"//

What I'm Doing Right Now:
* Debugging [[use setuptools's --multi-version mode|http://allmydata.org/trac/tahoe/ticket/530]] / [[develop doesn't create .pth files and site.py if --multi-version|http://bugs.python.org/setuptools/issue57]]
Argh, I give up on debugging that.  We'll just require users to uninstall any old conflicting packages before building Tahoe.

I'm working with Nathan and Brian and ~FriAM on Tahoe security architecture vs. ~JavaScript.  I'm still waiting for Collin Jackson, or Nathan Wilcox, or ''someone'' to demonstrate how the current tahoe security architecture's mismatch with the current web browser security architecture leads to an exploitable problem.
I think the best long-term strategy is:
* tell the web browsers that two web pages loaded from tahoe are from ''different'' origins
* have the tahoe web server do Caja verification/rewriting on all pages that it serves
Note that this "two pages from same server are actually of different origin" situation is exactly the situation with mash-ups.  You want to be able to serve up two different ~JavaScript applets from two different authors, from the same server and running on the same page, while telling the browser that these two applets are from different origins.  At least, if you're a capability-security person, that's what you want.
* fixed [[the deb-builders|http://allmydata.org/buildbot/waterfall?show_events=false&branch=&builder=edgy&builder=feisty2.5&builder=etch&builder=gutsy&builder=hardy&builder=deb-edgy&builder=deb-feisty&builder=deb-etch&builder=deb-gutsy&reload=none]] yesterday
* added to [[things to read]]:
** Tyler Close: [[ACLs don't|http://waterken.sourceforge.net/aclsdont]]

Wow!  [[Mark Miller|http://research.google.com/pubs/author35958.html]] said he liked my blog!

* wrote a note to cap-talk about [[why it is important to publish "ACLs don't"|http://www.eros-os.org/pipermail/cap-talk/2009-January/012073.html]] 
* wrote a note to distutils-sig about how [[stdeb can be used to produce real Debian packages|http://mail.python.org/pipermail/distutils-sig/2009-January/010912.html]]
* I've been building binary Python eggs of many Tahoe dependencies on many platforms and uploading them to [[this directory|http://testgrid.allmydata.org:3567/uri/URI:DIR2-RO:snrfwfxatrci35zdgjnzxxx2ke:unarxv347edtku3xzmefy4mcdmfngxzeb72iyqcadbjzjpczjx5a/index.html]].
* wrote a note to distutils-sig about [[hosting Tahoe's dependency packages on a Tahoe grid|http://mail.python.org/pipermail/distutils-sig/2009-January/010915.html]]
* Heh heh heh.  This morning google was warning me that "This site might harm your computer.".  It was warning me about that for ''every'' site that it suggested, for ''every'' query that I asked about.  Truer words were never spoken, Mr. Google!  Here's a screen shot of one of the queries.  I got the same results when I searched for "sex" and for "frisbees".  (Click the image below for the full screen shot.)
[img[This site may harm your computer. -- cropped|/file/URI%3ACHK%3Asx2qslmo2lrjnaslhlr4t6rk7e%3Ajl6frjdvmfs2kuq7pfwov3t4iqcbraz6ey5ul7lrxakzlh6jqnba%3A3%3A10%3A66528
/@@named=/This_site_may_harm_your_computer-cropped.png][/file/URI:CHK:kry2elgkh42hzpfwkh4b3mhksi:xdcewsphgz5f4js3cf4hszbie4kcz4uuds4djbnztel2wkzbscbq:3:10:171851/@@named=/This_site_may_harm_your_computer.png]]
(P.S.  Here is [[a screenshot|http://folk.ntnu.no/brasetvi/google_considered_harmful.png]] sent in by an alert reader who googled for google.  I should have thought of that.)

What I'm Working On Right Now:
* figure out why my Windows Tahoe node doesn't connect to all the testgrid blockservers
* make green the rest of [[the tahoe buildbots|http://allmydata.org/buildbot/waterfall]] (cygwin and Windows)
Then:
* ask everyone on tahoe-dev to test out the current trunk in preparation for the 1.3.0 release
* close the rest of these tickets for the Tahoe-1.3.0 release: [[the Tahoe Roadmap|http://allmydata.org/trac/tahoe/query?status=assigned&status=new&status=reopened&group=status&milestone=1.3.0]]
Then:
* tag tahoe-1.3.0, upload tarballs and .debs, send out a release announcement far and wide letting people know about the existence of tahoe-1.3.0
Done:
* greenified the edgy and dapper buildslaves
* added to [[issue tickets]]
** //[[foolscap #107|http://foolscap.lothar.com/trac/ticket/107]]: exceptions.~KeyError: "unable to find reference for name//

* Here is [[the RSS feed of this blog|http://testgrid.allmydata.org:3567/uri/URI:DIR2-RO:j74uhg25nwdpjpacl6rkat2yhm:kav7ijeft5h7r7rxdp5bgtlt3viv32yabqajkrdykozia5544jqa/wiki.xml]].
* I went to [[Boulder Open Coffee|http://boulder.me]] this morning.  The host was a funny venture capitalist named [[Jason Mendelson|http://jasonmendelson.com]].  The back room of the coffee shop was crowded with tech industry people.  I think I flubbed my 10-second introduction, describing [[tahoe|http://allmydata.org]] as a "secure decentralized filesystem", which did not elicit any detectable interest.  Perhaps I should have said that there exists a "decentralized web app" (this blog).  Oh well -- maybe I'll have another 10-second chance someday.<br>I was later pleasantly surprised when the fellow on my right, Manisch, asked me a few good questions (about naming and availability), quickly understand the architecture, and described some interesting related work.
* I finally finished listening to [[EconTalk 2009-01-26|http://www.econtalk.org/archives/2009/01/roberts_and_han.html]].  It was fascinating.  I wonder if thinkers like Hanson and Roberts will turn their current ideas about how science is practiced into an empirical scientific endeavour instead of merely a fascinating philosophical rumination.
* I've renewed [[my plea|http://www.crynwr.com/cgi-bin/ezmlm-cgi?17:mss:512:200902:ofpndmgcgmbhbmimpkpe]] for the Open Source Initiative to certify that [[the Transitive Grace Period Public Licence|http://allmydata.org/source/tahoe/trunk/COPYING.TGPPL.html]] is open source.  The discussion is valuable, although personally painful for me -- I get physically shaky when reading or writing on [[the thread|http://www.crynwr.com/cgi-bin/ezmlm-cgi?17:sss:512:200902:ofpndmgcgmbhbmimpkpe#b]].  One way in which the discussion is valuable is that it vividly reminds me of the fact that the TGPPL needs a careful FAQ and other such supporting material.  (Thanks also to Wes Felter for reminding me of that.)
* added to [[issue tickets]]:
** //[[tahoe #331|http://allmydata.org/trac/tahoe/ticket/331]]: add DSA to pycryptopp - serialize pubkeys with less fluff//
** //[[tahoe #605|http://allmydata.org/trac/tahoe/ticket/605]]: delayed connection on Windows//
** //[[tahoe #556|http://allmydata.org/trac/tahoe/ticket/556]]: prepend 'application-version' with the name of this particular application//
** //[[tahoe #217|http://allmydata.org/trac/tahoe/ticket/217]]: ~DSA-based mutable files -- small ~URLs, fast file creation//
* Eric Shulman pointed out that all I needed to say to get the attention of the business people was "Tahoe is a cloud storage technology.".  Now I'm armed for next time I find myself in that position.

* added a comment to //[[setuptools #20|http://bugs.python.org/setuptools/issue20]]: package required at build time seems to be not fully present at install time?// explaining how the current attempt to fix it leads to a different problem for tahoe
Here's a joke, from [[Ian Grigg|http://financialcryptography.com]]:
<<<
Teacher: If there are ten sheep in the pen, and one sheep jumps out, then how many sheep are left in the pen?
Girl: None.
Teacher (sadly): You don't understand arithmetic.
Girl: No, you don't understand sheep!
<<<

Good morning!  The setup/packaging/installing problem on tahoe trunk appear to be fixed, judging by [[the mostly-green buildbot|http://allmydata.org/buildbot/waterfall?show_events=true]].  In order to do this I had to create a toothpick of [[setuptools|http://peak.telecommunity.com/DevCenter/setuptools]].  A toothpick is like a fork, but smaller and best used for only one user.  The toothpick is being maintained under the whimsical name {{{zetuptoolz}}} on [[the allmydata trac|http://allmydata.org/trac/zetuptoolz]].  As soon as tahoe-1.3.0 is out, I'll try to merge my toothpick with upstream setuptools or one of its variants or alternatives.

I'm going to [[this talk by Peter Braam|http://www.cs.colorado.edu/events/colloquia/2008-2009/braam.html]] this afternoon.  So I'm reading [[the wikipedia about about Lustre|http://en.wikipedia.org/wiki/Lustre_(file_system)]].

Now that the OSI understand the intent of [[the Transitive Grace Period Public Licence|http://allmydata.org/source/tahoe/trunk/COPYING.TGPPL.html]] as being a //transitive public licence// (i.e. like the GPL in this respect), [[they want to think about it more|http://www.crynwr.com/cgi-bin/ezmlm-cgi?17:mss:518:200902:ofpndmgcgmbhbmimpkpe]].

add the following to the list of Things To Do After 1.3.0 release: write up the notes from the discussion with Brian about how to document standardizable explanation of tahoe:
* document 1: file formats: integrity, confidentiality, erasure-coding; immutable and mutable files (this would basically be my [[lafs.pdf|http://allmydata.org/~zooko/lafs.pdf]] with enough detail to facilitate compatible implementation)
* document 2: storage server protocol: how to connect to storage servers and upload/download files or parts thereof; foolscap protocol (extant), HTTP protocol (future work)
* document 3: peer selection ; This is the hard one.  We can document the "~TahoeTwo" system, which works okay for [[the allmydata.com use case|http://allmydata.org/trac/tahoe/wiki/UseCases]], but there are all sorts of use cases that are ill-served by the "~TahoeTwo" design -- more than a few hundred servers per grid, non-clique grids, secure decentralized introduction, different needs for reliability, availability, efficiency, etc. (some user wants Tahoe to store K+1 shares of each file in each of their several separate data centers, Shawn wants it to store exactly K shares of each file on his mom's computer, if that file is tagged as being a family photo), how to tell which grid or grids or set-of-servers a given cap can be found in.
* document 4: directories ; How to encode a set of caps, each annotated with metadata such as a name, timestamp, etc., in a file.  Traversal caps?
This project is very exciting because it could facilitate other people re-using tahoe, re-implementing it, or at least stealing its best ideas.  The walls of separation between these four documents are there to limit the damage from design mistakes, and by the same token to explicitly identify what each part requires of the other parts.
In particular I am eager to wall off the designs in document 3 from the others, as it is a field littered with the corpses of attractive designs.

Things to do for the tahoe-1.3.0 release; see also [[the Tahoe Roadmap|http://allmydata.org/trac/tahoe/roadmap]]:
* remove the cygwin buildbot from the list of supported platforms, move it to a new list of unsupported platforms, report this issue to twisted and cygwin via bug reports with minimal test cases ; //or// fix the cygwin buildslave.  (I contributed a small test script which shows the problem over on //** [[twisted #3529|http://twistedmatrix.com/trac/ticket/3529]]: closing stdout in a child process on cygwin means that process doesn't receive bytes from stdin anymore. I think.//.)
* remove Dan's ~ArchLinux buildbot from the list of supported platforms, //or// fix it.  (I sent Dan, Chris Galvan, and Brian mail about that behavior.)
* clean up the repairer tests -- probably comment many of them out since they are written to test a repairer that discriminates among servers on upload, probably add a verify run after the remaining ones so that it is judging more carefully the state of the shares after repair instead of just judging the repairer's report
* figure out why dreid and Andrzej Falout are both reporting bad performance from the prod grid
Things to do after the tahoe-1.3.0 release:
* //[[tahoe #608|http://allmydata.org/trac/tahoe/ticket/608]]: premature abort of upload if some shares were already present and some servers fail// (pushed out of the 1.3.0 milestone)
* write back on [[this thread on tahoe-dev|http://allmydata.org/pipermail/tahoe-dev/2009-January/001056.html]] and say "No, no, it really is almost as easy as Shawn originally thought.  Go, Shawn, go."
* figure out why setuptools/zetuptoolz is rebuilding things when {{{python ./setup.py test}}} after it just built them when {{{python ./setup.py build}}}
* pycryptopp improvements -- link against system libcryptopp.so for Debian and Fedora packagers, add new improved ECDSA, build out more buildbots, etc.
* accounting/garbage-collection/quotas/etc.
* write up all those documents described in [[2009-02-06]]
* write up my new idea for immutable crypto caps
* start backing up all my personal files with tahoe
* one zillion other things
Added to [[things to read]]:
* new IETF group [[on Massively Multiplayer Online interop|http://trac.tools.ietf.org/bof/trac/wiki/MmoxCharter]]
* Tyler Close: [[ACLs don't|http://waterken.sourceforge.net/aclsdont]] and [[the ensuing conversation on cap-talk|http://www.eros-os.org/pipermail/cap-talk/2009-January/012030.html]]
* Nate Foster, Benjamin C. Pierce, and Michael Greenberg, et al.: [[Harmony/Boomerang|http://www.seas.upenn.edu/~harmony]], A bidirectional programming language for ad-hoc, textual data

Thanks especially to Brian Warner, we got a lot closer to the tahoe-1.3.0 release today!  See [[the Roadmap|http://allmydata.org/trac/tahoe/roadmap]] and [[the Timeline|http://allmydata.org/trac/tahoe/timeline]] for details, and [[the tahoe-dev list|http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev]] for social context.

added //[[twisted #3649|http://twistedmatrix.com/trac/ticket/3649]]: more specific warning about plugin cache// to [[issue tickets]]

Today is the day that we'll close all [[open tickets for tahoe-1.3.0|http://allmydata.org/trac/tahoe/query?status=assigned&status=new&status=reopened&group=status&milestone=1.3.0]].  I'll also send out a plea to [[the tahoe-dev mailing list|http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev]] to test it and report bugs.  Hopefully tomorrow is the day that we tag it and send out the release announcement far and wide.

added to [[things to read]] a couple of things that Nathan just put on //his// "things to read":
* Ken Thompson: [[Reflections on Trusting Trust|http://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf]]
* Helen J. Wang, Xiaofeng Fan, Jon Howell, and Collin Jackson: [[Protection and Communication Abstractions for Web Browsers in MashupOS|http://research.microsoft.com/en-us/um/people/helenw/papers/sosp07mashupos.pdf]]

* helped Nathan with his newly announced [[tahoewapi.js|http://allmydata.org/pipermail/tahoe-dev/2009-February/001158.html]]
* got some help from other people on packaging Tahoe for Fedora and Debian -- [[working around subtle C++ linking issues|http://allmydata.org/pipermail/tahoe-dev/2009-February/001159.html]]
* posted [[a beta-test of the tahoe-1.3.0 Release Announcement|http://allmydata.org/pipermail/tahoe-dev/2009-February/001151.html]] :-)
* posted about [[tahoe RelatedProjects|http://allmydata.org/pipermail/tahoe-dev/2009-February/001157.html]]
* added to [[issue tickets]]: //[[tahoe #615|http://allmydata.org/trac/tahoe/ticket/615]]: Can ~JavaScript loaded from Tahoe access all your content which is loaded from Tahoe?//
* made a new tiddler on this blog: [[things to do after the tahoe-1.3.0 release]]

Hello, world!  Tahoe-1.3.0 looks to be just about done.  I'm editing docs and release notes and changing the format of the "application version string" and running extra tests on repairer, but basically what is in trunk now is going to be released as Tahoe-1.3.0 today.  So hurry up and test trunk for me and report bugs!

added to [[issue tickets]]:
* //[[konqueror #184157|http://bugs.kde.org/show_bug.cgi?id=184157]]: crashed (I might have just hit the "back" button)//
* //[[tiddly_on_tahoe #9|http://allmydata.org/trac/tiddly_on_tahoe/ticket/9]]: can't save from Konqueror-4.2.0//
* //[[darcsver #2|http://allmydata.org/trac/darcsver/ticket/2]]: use "darcs query" to get count of patches faster//

* I updated [[Help Zooko Choose Books]] to reflect Myers's recommendation.  I'm increasingly interested in Erlang for similar reasons as Myers is -- I'm very interested in the "crash-only" design for high availability and correctness.  Also I'm told that Erlang is an iota away from being a capability-secure programming language.
* Okay [[Tahoe-1.3.0|http://allmydata.org/trac/tahoe/browser/relnotes.txt?rev=20090214000500-92b7f-26d07f519c69a3e086ee4599620f6d2c0c9b2fcd]] is out!  Now I'm looking at a huge backlog of tasks.  Last night I submitted a proposal to present Tahoe at [[CodeCon 2009|http://codecon.org]].  (I offered to suddenly and violently destroy a hard drive or two on stage.)  Many of the tasks on this huge backlog involve writing mail to [[tahoe-dev|http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev]].
* submitted a couple of patches to fix a couple of bugs in [[stdeb|http://github.com/astraw/stdeb/tree/master]]
* helped Nils Durner host his new {{{libtahoeclient_webapi}}} project on http://allmydata.org/trac
* had a nice chat at a local coffeeshop with Peter Braam

I got audio working on my linux workstation.  Hey, that was painless!
* working with François Deppierraz, ~Jan-Benedict Glaw, Shawn Willden and perhaps others on making tahoe handle the tangled mess that is character-set encoding (//[[tahoe #534|http://allmydata.org/trac/tahoe/ticket/534]]: "tahoe cp" command encoding issue//, added to [[issue tickets]])
* posted about how to automatically produce .deb packages from Python source code [[on distutils-sig|http://mail.python.org/pipermail/distutils-sig/2009-February/010998.html]]
* re-iterated my [[Request For Approval As Open-Source-Definition-Conformant|http://www.crynwr.com/cgi-bin/ezmlm-cgi?17:mss:529:200902:ofpndmgcgmbhbmimpkpe]] for [[The Transitive Grace Period Public Licence|http://allmydata.org/source/tahoe/trunk/COPYING.TGPPL.html]]
* //update!// and then I [[challenged the Open Source Initiative to sue me|http://www.crynwr.com/cgi-bin/ezmlm-cgi?17:mss:531:200902:ofpndmgcgmbhbmimpkpe]]
* I submitted the Transitive Grace Period Public Licence to [[the debian-legal mailing list for review|http://lists.debian.org/debian-legal/2009/02/msg00047.html]].
added to [[issue tickets]]:
* //[[twisted #1956|http://twistedmatrix.com/trac/ticket/1956]]: Make a less sucky producer/consumer API//

[[The Open Source Initiative Stumbles]]

Tonight I'm going to see [[Children of Men at the International Film Series|http://internationalfilmseries.com/event_detail.php?event_id=9509]].  Meet me in the square with the giant buffalo at 21:00.

I want to make it possible to use the [[Brainpool ECC curves in Crypto++|http://groups.google.com/group/cryptopp-users/browse_thread/thread/b04fd594c5fc833]], but where are the test vectors?  I'm writing letters to Hal Finney and Niels Ferguson to ask if they have some ideas.

added to [[issue tickets]]
* //[[amarok #184834|https://bugs.kde.org/show_bug.cgi?id=184834]]: amarok crashes on "Quit"//

//(My mom, and my friend Nat and others, have pointed out that my blog is chock full of acronymy goodness and they can get only the vaguest idea of what I'm on about, mostly by reading the verbs.  I don't like those sorts of communication gaps (even though they are inevitable), so I've decided to try the exercise of translating each item that I post from programmer-speak to English.  If it is hard to translate into English, maybe this tells us something about what it means in its original programmer-speak.)//

I've decided to attend [[PyCon|http://us.pycon.org/2009/about/]] this year!  Whoo!  //''Mom'': that means a giant party in Chicago where we learn, write programs, and engage in "professional networking", such as by consuming intoxicants.//

[[Bespin+Tahoe]]

Irby and Elliot are really enjoying their [[OLPC XOs|http://en.wikipedia.org/wiki/OLPC_XO-1]], which a generous relative gave to them a year ago for Christmas.  They play [[Battle for Wesnoth|http://wesnoth.org]] quite a lot, but they also explore the more explicitly educational activities.  A couple of days ago I browsed the [[list of Activities|http://wiki.laptop.org/go/Activities/All]] and chose this one named [[Physics (Activity)|http://wiki.laptop.org/go/Physics_(activity)]].  It is great!  Irby has spent many hours now configuring different 2-D objects to fall and bump and bend and scrape each other.

Good morning, world!

A bunch of Free Software/Open Source hackers from around the world have volunteered their efforts to the Tahoe project.  Come join the fun!  http://allmydata.org .  The [[tahoe-dev mailing list|http://allmydata.org/pipermail/tahoe-dev/2009-February/date.html]] is the center.

(This offer does not apply if contributing to a Free/Open Source project to create a secure, decentralized storage grid doesn't sound like "fun" to you.)


things to do (I'm taking notes here so I don't forget any of them):
* //[[tahoe #331|http://allmydata.org/trac/tahoe/ticket/331]]: add DSA to pycryptopp - serialize pubkeys with less fluff// == //[[pycryptopp #3|http://allmydata.org/trac/pycryptopp/ticket/3]]: serialize ecdsa keys without the fluff// (next step: separate out my patches to embed C++ objects directly into Python objects and save it aside for later)
* review [[François Deppierraz's latest patch|http://allmydata.org/trac/tahoe/ticket/534]] to fix handling of non-ascii characters on all platforms
* respond to Nils Durner's wiki entry on [[building tahoe for Wndows|http://allmydata.org/trac/tahoe/wiki/WindowsBuild]]
* compare Nils Durner's summary of how [[GNUnet|http://gnunet.org]] finds its local state files on Windows with how tahoe currently does it (which was implemented by RobK)
* post my note about Jack Lloyd's attack on rsync
* fix Eugen Leitl's contributed buildslave so it builds Debian packages for Lenny/amd64
* figure out why yukyuk is getting signal 11 errors sometimes in test_backupdb
* fix buildbot breakage due to recent patch to improve the checker/repairer unit tests
* post about buildbots to tahoe-dev
* set up an introducer on nooxie for the nascent Community Grid
* review [[DarKnesS_WolF's patch|http://allmydata.org/trac/tahoe/ticket/638]] to build .deb's for Intrepid


//''Mom:'' Just read the part about people volunteering.//

Here's my reply to Jack Lloyd's attack on rsync: [[collision-resistance vs. second-preimage-resistance|http://allmydata.org/pipermail/tahoe-dev/2009-February/001309.html]].

//''Mom:'' I'm trying to figure out how to make ~URLs which are nice and short but which are secure in the sense that nobody can make you get a different page than the page intended by the person who gave you the URL.  Other people might be able to use the underlying cryptographic principles to make other things.  Is that a good enough translation?//

Hello folks!  Sorry for the blog outage.  I got [[laid off|http://allmydata.org/pipermail/tahoe-dev/2009-March/001461.html]], Amber is pregnant, and the Tahoe test grid that this blog is hosted on [[suffered a bug|http://allmydata.org/trac/tahoe/ticket/651]].  But I'm back!  Hope you missed me!  Don't forget to subcribe to [[my RSS feed|http://testgrid.allmydata.org:3567/uri/URI:DIR2-RO:j74uhg25nwdpjpacl6rkat2yhm:kav7ijeft5h7r7rxdp5bgtlt3viv32yabqajkrdykozia5544jqa/wiki.xml]].

All Hackers who are going to be near Boulder, Colorado at the beginning of April, please write to [[zooko@zooko.com|mailto:zooko@zooko.com]]!

Also, I'm going to [[PyCon 2009|http://us.pycon.org/2009/about/]]!  Whoo!

[[PyCon|http://us.pycon.org/2009/about]] was great fun.  It was a pleasure to find out that [[Tahoe|http://allmydata.org]] has become a widely known project within the Python community.  Most people have heard of it, and many people find it interesting.

NEWSFLASH!  Tahoe is eligible for Google Summer of Code sponsorship, thanks to the generous Python community in the form of the [[Python Software Foundation|http://www.python.org/psf]] umbrella organization!  If you are a student, or you know a student, who wants to hack on Tahoe under google's sponsorship this summer, then [[read this now|http://allmydata.org/pipermail/tahoe-dev/2009-April/001516.html]]!  :-)

(P.S.  This is not Fake News as in the Internet tradition of April first.  It is true.  And you have less than 48 hours to sign up, so get on with it.)

//~CodeCon!  Whoo!//  I'm going to [[CodeCon|http://codecon.org]]!  I'll be presenting [[Tahoe|http://allmydata.org]] there.

(//''Mom:'' Major hacker party in San Francisco.//)

//[[Nicholas Nassim Taleb returns to econtalk|http://www.econtalk.org/archives/2009/03/taleb_on_the_fi.html]]// It has been two years since [[Nicholas Nassim Taleb's first appearance on econtalk.org|http://www.econtalk.org/archives/2007/04/taleb_on_black.html]]. Now, he gets introduced by the host, Russ Roberts, as "one of the few thoughtful people alive today who can say 'I told you so.'".

I got interested in [[Nicholas Nassim Taleb|http://www.fooledbyrandomness.com]] because of his first appearance on econtalk. (I got interested in econtalk because of [[Robin Hanson's first appearance|http://www.econtalk.org/archives/2007/05/hanson_on_healt.html]].) I've subsequently read Taleb's book //[[The Black Swan|http://www.randomhouse.com/catalog/display.pperl/9781400063512.html]]// and found it to be rife with errors and ridiculousities, and yet delightful, profound and overwhelmingly important.

This reprise does not disappoint. Far from simply saying 'I told you so.', Taleb and Roberts provide important insights into why things like this happen and what to do about it.

(//''Mom:'' That all makes sense, right?  Listen to the podcasts.//)

Hello I am at ~CodeCon.

I am showing Zack Weinberg how I update my blog.

My personal server which runs http://zooko.com is unreachable at the moment.  It'll probably be back on-line later today, but in the meantime if you were looking for my résumé.html, here it is on the Tahoe decentralized storage grid: [[résumé.html|../../file/URI%3ACHK%3Ano2l46woyeri6xmhcrhhomgr5a%3A5p7cxw7ofacblmctmjtgmhi6jq7g5wf77tx6befn2rjsfpedzkia%3A3%3A10%3A8328/@@named@@/r%C3%A9sum%C3%A9.html]].  :-)

* ~SHA-1 broken?  [[These slides from Eurocrypt 2009 rump session|http://eurocrypt2009rump.cr.yp.to/837a0a8086fa6ca714249409ddfae43d.pdf]] by Cameron ~McDonald, Philip Hawkes, and Josef Pieprzyk, presented by Greg Rose, seem to say that they figured out how to compute ~SHA-1 collisions with 2^^52^^ cost.  The previous best-known technique required 2^^63^^ computations, so if this is right then it puts ~SHA-1 squarely in the category of things that [[some people|http://ioerror.livejournal.com]] will abuse and exploit just to make a point.  [[Here's my take|http://allmydata.org/pipermail/tahoe-dev/2009-April/001663.html]] in a message to tahoe-dev.
* Matt Mackall has written a new tool to give a [[useful measurement of how much memory|http://lwn.net/SubscriberLink/329458/d28c2d45a663045a]] your Linux program is using.  Hooray!
* Added to [[things I have read]]:
** Stefan Tillich: [[Hardware Implementation of the SHA-3 Candidate Skein|http://eprint.iacr.org/2009/159]];  Comparing this to [[the EnRUPT SHA-3 proposal|http://www.enrupt.com/SHA3/Supporting_Documentation/EnRUPT_Specification.pdf]] (page 24) suggests that ~EnRUPT is probably better than Skein both ASIC and FPGA, in both area and speed.  This is true even after remembering that ~EnRUPT as it was proposed was broken, and that the current proposal is to double (or so) the number of rounds, so I probably need to mentally double all of the "How many seconds to process 1 Gbit" in the ~EnRUPT results.
* Okay, http://zooko.com is back, thanks to the heroic efforts of Brian Warner and rootard.  I am very pleased that my blog remained fully functional (for both read and write) while my server was off-line.  (Because this blog is on [[Tahoe-LAFS|http://allmydata.org]].)

I picked up the third book in //[[The Golden Age|http://www.amazon.com/dp/0765349086]]// from the library today, and I've read more than half of it already. I crowed with pleasure when the opening pages started with an intense space battle including previously unknown laws of physics, mind virusses, nanotech, and tactical personality implantation, and then the survivors sat down to argue strategy using, among other ideas, detailed renditions of Ayn Rand's "fallacy of the stolen concept" and David Ricardo's "comparative advantage". (Those names were not used, but I recognized the ideas.)

finished //[[The Golden Age|http://www.amazon.com/dp/0812579844]]// and re-organized my [[things to read]] tiddler;   //[[The Golden Age|http://www.amazon.com/dp/0812579844]]// was satisfying both as an adventure novel with an admirable hero (I always like those) and as a thought-provoking exploration of many ideas.

Hooray!  I finally have an intuition for Bayes' Theorem thanks to [[this visualization|http://blog.oscarbonilla.com/2009/05/visualizing-bayes-theorem]].  Thank you, Oscar Bonilla!

This journal has experienced an outage because the http://tahoebs1.allmydata.com:8123 public web gateway to the Tahoe test grid is down.  Of course, the journal itself is distributed and accessible even when that gateway is broken, but only if people use a different web gateway (ideally, a local web gateway running on their own computer and access with http://127.0.0.1:8123).

But, it is inconvenient to explain that, so I've just been giving people [[a hyperlink going through the tahoebs1 gateway|http://tahoebs1.allmydata.com:8123/uri/URI:DIR2-RO:hgvn7nhforxhfxbx3nbej53qoi:yhbnnuxl4o2hr4sxuocoi735t6lcosdin72axkrcboulfslwbfwq/wiki.html]].  Therefore, when that single point fails, my blog becomes unavailable to them.

Hm...

Got up this morning and checked the status of all the issues which interfere with the automated installation of libraries that Tahoe depends on:

* [[pyflakes #2709 (Pyflakes svn doesn't install properly due to missing packages)|http://divmod.org/trac/ticket/2709]]
* [[nevow #2798 (setup.py install --home is broken :-()|http://divmod.org/trac/ticket/2798]]
* [[nevow #2699 (build nevow without importing nevow)|http://divmod.org/trac/ticket/2699]]
* [[nevow #2629 (Nevow doesn't declare its dependency on Twisted in a machine-parseable way)|http://divmod.org/trac/ticket/2629]]
* [[nevow #2527 (easy_install compatibility)|http://divmod.org/trac/ticket/2527]]
* [[pyopenssl #238658 (please provide binaries)|https://bugs.launchpad.net/pyopenssl/+bug/238658]]
* [[setuptools #53 (respect the PYTHONPATH)|http://bugs.python.org/setuptools/issue53]]
* [[setuptools #54 (be more like distutils with regard to --prefix=)|http://bugs.python.org/setuptools/issue54]]
* [[setuptools_trial #1 (add all trial args that tahoe and/or Brian want to use)|http://allmydata.org/trac/setuptools_trial/ticket/1]]

and

* [[buildbot #212 (buildbot doesn't respond to darcs tags)|http://buildbot.net/trac/ticket/212]]

Grr...  And today the Hack Tahoe! contest is down.

Hm...

I've been reading a lot about hash functions in order to contribute to the ~SHA-3 project.  Here's a good one: [[Merkle–Damgård revisited: How to construct a hash function|http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.60.2072]] by ~Jean-Sébastien Coron, Yevgeniy Dodis, Cécile Malinaud, Prashant Puniya.

small bits of hackery:

darcs: I helped Greg H use darcs.  I deleted spam tickets from http://allmydata.org/trac/darcs-2 and posted [[a message|http://lists.osuosl.org/pipermail/darcs-users/2008-July/012587.html]] to the darcs-users mailing list asking if they could link to it from http://darcs.net.  I updated [[darcs ticket#693|http://bugs.darcs.net/issue693]].

my development tools: I've figured out, thanks to Brian Warner's timely help, that the problem with my Linux workstation locking up is [[this driver issue|https://bugs.launchpad.net/ubuntu/+source/linux/+bug/136836]].  I'm compiling the shiny new Linux 2.6.26 kernel to see if that changes the behavior.  (Update: so far so good, kernel 2.6.26 has yet to lock up.  I'm loading the machine up with buildslaves for [[pycryptopp|http://allmydata.org/buildbot-pycryptopp/waterfall]], [[tahoe|http://allmydata.org/buildbot/waterfall?show_events=false]], and eventually also zfec and other projects in order to stress it.)

open source organization: added https://launchpad.net/foolscap and https://launchpad.net/allmydata.org as being related to the https://launchpad.net/tx superproject.

Today I'm going to be pair-programming with Brian Warner from noon to 15:00 and then from 17:00 to 19:00 (Mountain Time).  I'm pleased that Amber's powerful Linux workstation (yukyuk) seems to be stable now that I've upgraded it to Linux 2.6.26.  You can see that it comes in second in [[the builder races|http://allmydata.org/buildbot/waterfall?show_events=false]].  I'll probably use that for the pair-programming instead of my trusty old Toshiba laptop (~Pentium-M 1.8 ~GHz).

This morning I wrote [[a note|http://lists.osuosl.org/pipermail/darcs-users/2008-July/012607.html]] to the darcs-users mailing list about optimizing darcs-2 performance.

Also of course I'm fixing [[the Tahoe builders|http://allmydata.org/buildbot/waterfall?show_events=false]], some of which are failing due to bugs in 3rd party libraries we use and due to issues in how the Tahoe build system manages which versions of the libraries that it builds.

Whenever I'm not fixing the buildslaves or pair-programming with Brian, I intend to be working on fixing up [[my submission to StorageSS08|http://allmydata.org/pipermail/tahoe-dev/2008-July/000656.html]], which is due Sunday, August 3.

Okay, yesterday went pretty well.  I didn't pair program with Brian as much as planned because I instead spent a long time setting up my development environment and remote access to it for Brian on Amber's newly reinvigorated Linux workstation.  We did get an hour and a half of pair-programming, during which time we made some good unit tests for immutable file checking and repair: [[[2813]|http://allmydata.org/trac/tahoe/changeset/2813]], [[[2814]|http://allmydata.org/trac/tahoe/changeset/2814]].

I also improved our packaging, unit tests, and added a builder.  The [[trac Timeline for yesterday|http://allmydata.org/trac/tahoe/timeline?from=2008-07-30&daysback=1&ticket=on&ticket_details=on&changeset=on&milestone=on&wiki=on&update=Update]] shows all changes that anyone committed to the wiki, the ticket database, or the source code yesterday.

I've also been doing some open source "community building" work.  That is: I've been contributing to the community development process that produces [[darcs|http://darcs.net]] and [[setuptools|http://peak.telecommunity.com/DevCenter/setuptools]].  We'll see how that works out.

Today I hope to work on [[my submission to StorageSS08|http://allmydata.org/pipermail/tahoe-dev/2008-July/000656.html]], which is due Sunday, August 3.

Whoops, I allowed days to go by without hlog updates.  It is hard to be motivated to write here since I don't know if anyone other than me is reading it.

Perhaps I should just think of it as a personal hack diary -- that would be a useful thing to have even if nobody else read it.

Okay, self, you really need to work on immutable file checking and repair and download ASAP.  Starting with checking.

//(My mom, and others, have pointed out that my blog is chock full of jargony goodness and they can get only the vaguest idea of what I'm on about, mostly by reading the verbs.  I don't like those sorts of communication gaps (even though they are inevitable), so I've decided to try the exercise of translating each item that I post from programmer-speak to English.  If it is hard to translate into English, maybe this tells us something about what it means in its original programmer-speak.)//

Last week I saw from [[Simon Phipp's blog|http://blogs.sun.com/webmink]] this interesting project named [[Bespin|https://bespin.mozilla.com]].  It is a programmer's text editor implemented in ~JavaScript and [[the Canvas element|http://en.wikipedia.org/wiki/Canvas_(HTML_element)]].  That same day Tom Lord mentioned Bespin to me in email.  Bespin is exciting stuff!  I really want to hack it to save the files to a [[tahoe-lafs|http://allmydata.org]] grid.  If you saved your files to a tahoe-lafs grid, then you wouldn't have to rely on the server to make sure your file isn't deleted or corrupted, that its contents aren't [[stolen by someone else|http://www.emergentchaos.com/archives/2009/01/breach_misdirection.html]], nor that permission to write to your file was accidentally given to someone else, nor that permission to write to your file was accidentally //denied// to someone with whom you had intended to share it.  (By the way, the list of claims in that sentence should arouse great skepticism in security and distributed systems experts.  On the other hand, webapp authors and users probably paid little attention, thinking to themselves "Well, why ''shouldn't'' it Just Work?".  I love this chaos.)

I had a lot of fun hacking [[TiddlyWiki|http://tiddlywiki.com]] to store itself to a tahoe-lafs grid, with the help of ~TiddlyWiki hackers [[FND|http://fnd.lewcid.org]], [[Eric Shulman|http://about.unamesa.org/Eric+Shulman]], and [[Jeremy Ruston|http://jermolene.com]].  The result was two small ~JavaScript plugins that anyone can add to their ~TiddlyWiki to make their ~TiddlyWiki into a ''gridapp'': [[HTTPSavingPlugin.js|http://allmydata.org/trac/tiddly_on_tahoe/browser/tahoe_tiddly/HTTPSavingPlugin.js]] and [[TahoePlugin.js|http://allmydata.org/trac/tiddly_on_tahoe/browser/tahoe_tiddly/TahoePlugin.js]].

I would love to do something similar for Bespin, but I can tell that I need to focus on the tahoe core right now.  The [[announcement of tahoe-lafs 1.3.0|http://allmydata.org/trac/tahoe/browser/relnotes.txt?rev=20090214000500-92b7f-26d07f519c69a3e086ee4599620f6d2c0c9b2fcd]] has encouraged more people to [[start contributing to Tahoe|http://allmydata.org/pipermail/tahoe-dev/2009-February/date.html]], and I'm busy teaching them how to get their patches accepted into the tahoe core.

That said, if any ~JavaScript/Caja/Tahoe/Bespin/~TiddlyWiki hackers out there want to have a go at {{{Bespin+Tahoe}}}, do let me know and I'll give you a little help.  :-)

//''Mom:'' Hm...  This one is pretty hard to translate into English.  I think that in the future new applications such as word processors, spreadsheets, tax software, photo album browsers, email, etc. will live on a web page and you will use them by pointing your web browser at that page.  I think the most fact about this is that the market for software is likely to get more competitive -- a greater marketplace of useful software, easier to use and with better features, that you are able to try out and then adopt. A large part of the motivation for the Tahoe project is to explore whether we can get all those values and //still// let you retain ownership of your own data.//

//{{{
config.options.txtTheme = "WritableTheme";
//}}}

I've been spending a little bit of time thinking about how gaming services like //World of Warcraft// and (don't-call-it-a-gaming-service) //Second Life// have succeeded where we at ~DigiCash failed -- convenient, widely-used, programmable digital cash.  A problem is that each of these new currencies are centrally controlled by one entity.  This limits the scope of who will rely on that currency and how much value they will risk on that currency.  There are ideas floating around about how to facilitate transactions between currencies, but this would not solve the problem.  //A plethora of competing centralized services is not the same as a decentralized service.//  Even if it were cheap and convenient to trade some ~LindenBucks for some ~WoW Gold, this would only lead us back to the equivalent of the modern nation-state currencies: mostly centralized (because of //the Network Externality//), heavily taxed/regulated/manipulated, and prone to disastrous failure.  What I want is a currency which everyone can cheaply and conveniently use but which ''no-one'' has the power to manipulate.  No-one has the power to inflate or deflate the currency supply, no-one has the power to monitor, tax, or prevent transactions.  Truly the digital equivalent of gold, during the times and places when gold was the universal currency.  See the [[BitGold|http://unenumerated.blogspot.com/2005/12/bit-gold.html]] idea by Nick Szabo and [[b-money|http://www.weidai.com/bmoney.txt]] idea by Wei Dai, and recent effort to actually implement something along these lines: [[BitCoin|http://www.bitcoin.org]] by Satoshi Nakamoto.

[[2009-05-05]]
[[2009-04-30]]
[[2009-04-29]]
[[2009-04-28]]
[[Laptop Versus Axe]]

I'm re-reading [[Good Calories, Bad Calories|http://amazon.com/exec/obidos/ASIN/1400040787]] by Gary Taubes.  The first time I read it aloud with my wife, Amber, and this time I'm reading it aloud with my brother Josh, who has recently decided to become a statistician.  (Reading it aloud to him seemed to be necessary in order to persuade him to read it, and so far it has worked well -- he is now very interested in the book.)

I've just finished reading Part One (of three) for the second time, and the words that keep coming to mind are "tour de force".  Part One is a beautiful example of investigative journalism in action.  By the traditional method of digging into the facts, reading the original historical sources, and interviewing the actors, and by the less traditional method of reading the relevant scientific literature, Taubes has uncovered a scientific and public policy failure of historic proportions.  Basically, some influential diet researchers managed to turn their hypothesis -- the low-fat diet hypothesis -- into the national policy of the United States of America starting in the late 70's.  The evidence in support of the hypothesis was tenuous, but confirmation bias -- that all too human weakness -- is richly stimulated by narrative flow combined with noisy data such as epidemiological observations, and so politicians and nutrition researchers alike (but not, as Josh points out, statisticians) came to believe that the hypothesis //must// be true, and that future experiments or observations would confirm it.

This is why everyone I know was taught, starting in the 1980's, that we should reduce the amount of fat -- especially saturated fat -- in our diet.

But it turned out that the subsequent experiments and observations did //not// confirm the hypothesis.  The succession of results over the last 30 years casts increasing doubt on the hypothesis.  The evidence now suggests that reducing your saturated fat intake is bad for your health, or at least not good for your health, or if it is helpful then its benefits are so small that they can't be detected.  Indeed there is good reason to believe that the low-fat diet hypothesis and the official policy promulgating it are actually one of the //causes// of the current epidemics of obesity, diabetes, and other problems.  If that is true, then the costs -- in dollars and in lives -- of this mistake are enormous.

In a thread on the "overcoming bias" blog, Hal Finney [[wrote|http://overcomingbias.com/2008/07/gary-taubes-goo.html#comment-124240968]]:

> My problem is, why can't I trust the people I pay to be experts on this topic?
> ...
> Why is it that I, or Gary Taubes, can read the literature and learn the truth, while a professional with far more experience and knowledge than either of us is unable to do so? This is the fundamental paradox I run into again and again on controversial topics.
> ...
> Two possible answers are, A, you can't, so trust the experts; or B, the experts are committed to the conventional wisdom and institutional forces prevent them from acknowledging the truth, while you are free from such prejudices. All I can really say is, I'll be really angry if B turns out to be true. Why should I have to do every damn thing for myself? Why can't I live in a world where people have a reasonable level of competence, where experts actually have expertise? For now, I just hope that A is correct

I think this is one of the most interesting aspects of this book, to me -- a compelling, factual investigation of Hal's question: Why can't we trust the experts on this topic?  I'm afraid Hal is going to be angry.  And rightly so.

It is not a diet book.  (If you need to lose weight, don't read this book.  It makes no specific dietary recommendations and you'll probably find it boring unless you are fascinated by the topic of "Science As She Is Practiced", in which case it is exhilarating.)

One lesson that I draw from this story is the importance of //experiment// in science.  Before //Good Calories, Bad Calories// was written, I had already become interested in this topic.  It all started in the year 1999, when I learned that my then girlfriend, Amber, was following some weird New Age fad diet -- a "low-carbohydrate" diet.  Concerned, I started paying attention whenever I learned about a new clinical trial in which this much-hyped fad diet was tested by experiment.  To my surprise, every experiment yielded results which were inconsistent with the established theory.  I read the abstracts and press releases and news coverage on probably half a dozen clinical trials over the next few years, and usually the scientists who had conducted the experiment were surprised too.  They, or other scientists in the field who were interviewed by journalists, would say something like "This is really weird -- we can't explain why the results are like that.".  (By the way, I still have notes naming those studies if you want them.)  I began to suspect that there was something amiss in the reigning theory.

Further reading, discussion with Amber, and personal experimentation persuaded me that carbohydrates are bad for you, and so when I read //Good Calories, Bad Calories//, the content about nutrition experiments and epidemiology wasn't too surprising.  The lesson that I drew from this is that it is easy to have too much confidence in a hypothesis which is based on ex post facto observation and theory.  For many years now I have found myself in the strange position of knowing almost nothing about biochemistry, human metabolism, or epidemiology, and yet being pretty sure that the experts who did know about these things were incorrect in their conclusions, for the simple reason that I had seen the results of several experiments designed to test their hypothesis, and the results had come out inconsistent with their hypothesis.  (Okay, nowadays I know a tad more about biochemistry and metabolism and epidemiology, almost all thanks to Amber, Josh, and Gary Taubes.)

This reminds me of Robin Hanson's [[contribution|http://www.overcomingbias.com/2008/07/gary-taubes-goo.html#comment-124037158]] to the overcoming bias blog thread: "Deviant prediction market estimates here would of course influence me much more [than scientific consensus]".

I think prediction markets might be a way to head off this kind of scientific fiasco.  I can't really justify this idea -- why should prediction markets work any better than the "market" for scientific reputation already does?  Nonetheless, I have a strong feeling that prediction markets would somehow focus more attention on experiment, or on other objective results which are robust against bias or interference, and less on the arts of persuasion and politics which can be used to build scientific consensus.

(Or at least, if the prediction market doesn't have this effect, then maybe I'll get lucky again -- correctly detect a major error long before the established scientists and officials detect it or admit to it -- and make myself rich by speculating.  ;-))


Okay, so I strongly recommend to all intelligent people who are interested in science, public policy, and/or human nutrition that they read this book.  By the way, I haven't even mentioned Parts Two and Three yet, which contain a wealth of intriguing ideas about such topics as aging (yes, I know it sounds crazy, but there is good reason to believe that low-carb dieting makes certain kinds of aging go slower).  The literary quality of the book drops in Part Three (perhaps there was a time crunch in the process of writing and editing it?) and Taubes repeats himself far too many times in Part Three, saying that the reigning hypothesis of human weight regulation is ''wrong, wrong wrong'' and ''utterly inconsistent with plenty of observed facts''.  This is a shame, because the writing in the first two Parts is skillful and engaging.  I will forgive you for skimming a few pages here and there of Part Three, but I will not forgive you for skipping this book and continuing to have a confident opinion about human nutrition.


P.S.  There is [[a thread|http://www.overcomingbias.com/2009/01/open-thread.html#comment-143978054]] over at the overcomingbias blog where I posted a link to this book review, so maybe the deep thinkers there will have something to say about it.

/***
|''Name''|HTTPSavingPlugin|
|''Description''|<...>|
|''Author''|Zooko|
|''Contributors''|FND|
|''Version''|0.2.1|
|''Status''|@@experimental@@|
|''Source''|http://allmydata.org/trac/tiddly_on_tahoe|
|''CodeRepository''|http://allmydata.org/source/tiddly_on_tahoe/trunk/|
|''License''|GPLv2+ or TGPPLv1.0+|
|''Keywords''|<...>|
!Description
<...>
!Notes
This plugin is being developed for [[Tiddly on Tahoe|http://allmydata.org/trac/tiddly_on_tahoe]].
***/
/* The following comment is to let jslint know which variables are supposed to be global. */
/*global clearMessage, config, getPath, readOnly, saveChanges, saveTest, showBackstage, store, story, version, convertUriToUTF8, convertUnicodeToFileFormat, getLocalPath, loadRemoteFile, locateStoreArea, saveBackup, saveEmpty, saveFile, saveMain, saveRss, unescape, displayMessage, httpReq */
//{{{
if (!version.extensions.HTTPSavingPlugin) { //# ensure that the plugin is only installed once
	version.extensions.HTTPSavingPlugin = { installed: true };

	(function () { //# wrapper
		readOnly = false;
		config.options.chkHttpReadOnly = false;
		showBackstage = true;

		saveTest = function () {
			var s = document.getElementById("saveTest");
			/*if (s.hasChildNodes()) {
			  alert(config.messages.savedSnapshotError);
			  }*/
			s.appendChild(document.createTextNode("savetest"));
		};

		// Save this TiddlyWiki with the pending changes
		saveChanges = function (onlyIfDirty, tiddlers) {
			var originalPath, localCallback, result;
			if (onlyIfDirty && !store.isDirty()) {
				return;
			}
			clearMessage();
			// Get the URL of the document
			originalPath = getPath(document.location.toString());
			// Load the original file
			localCallback = function (status, context, original, url, xhr) {
				//log("loaded remote file from ", originalPath);
				/*log("got callback status ", status, "\n", context: ", context, "\n",
				  URL: ", url, "\n", XHR: ", xhr);*/
				if (original === null) {
					alert(config.messages.cantSaveError);
					if (store.tiddlerExists(config.messages.saveInstructions)) {
						story.displayTiddler(null, config.messages.saveInstructions);
					}
					return;
				}
				// Locate the storeArea div's
				var posDiv = locateStoreArea(original);
				if (!posDiv) {
					alert(config.messages.invalidFileError.format([originalPath]));
					return;
				}
				saveRss(originalPath);
				saveEmpty(originalPath, original, posDiv);
				saveMain(originalPath, original, posDiv);
			};
			result = loadRemoteFile(originalPath, localCallback);
			//log("result from loadRemoteFile: ", result);
			return true;
		};

		// override and disable saveBackup()
		saveBackup = function (localPath, original) {};

		// override and disable getLocalPath()
		getLocalPath = function (origPath) {};

		// override getPath()
		getPath = function (origPath) {
			var originalPath, argPos, hashPos, resultPath;
			originalPath = convertUriToUTF8(origPath, config.options.txtFileSystemCharSet);
			// Remove any location or query part of the URL
			argPos = originalPath.indexOf("?");
			if (argPos !== -1) {
				originalPath = originalPath.substr(0, argPos);
			}
			hashPos = originalPath.indexOf("#");
			if (hashPos !== -1) {
				originalPath = originalPath.substr(0, hashPos);
			}
			// Convert file://localhost/ to file:///
			if (originalPath.indexOf("file://localhost/") === 0) {
				originalPath = "file://" + originalPath.substr(16);
			}
			// Convert to a native file format
			if (originalPath.indexOf("http://") === 0) { // HTTP file
				resultPath = originalPath;
			} else if (originalPath.charAt(9) === ":") { // PC local file
				resultPath = unescape(originalPath.substr(8)).replace(new RegExp("/", "g"), "\\");
			} else if (originalPath.indexOf("file://///") === 0) { // Firefox PC network file
				resultPath = "\\\\" + unescape(originalPath.substr(10)).replace(new RegExp("/", "g"), "\\");
			} else if (originalPath.indexOf("file:///") === 0) { // *nix local file
				resultPath = unescape(originalPath.substr(7));
			} else if (originalPath.indexOf("file:/") === 0) { // *nix local file
				resultPath = unescape(originalPath.substr(5));
			} else { // PC local file
				resultPath = "\\\\" + unescape(originalPath.substr(7)).replace(new RegExp("/", "g"), "\\");
			}
			return resultPath;
		};

		// override saveFile()
		saveFile = function (fileUrl, content, callb) {
			displayMessage("saving... please wait"); // XXX: belongs into command handler -- TODO: i18n
			//alert("whee! about to save to " + fileUrl);
			var localCallback = function (status, params, responseText, url, xhr) {
				if (!status) {
					displayMessage("saving failed: " + responseText);
				}
			};
			return httpReq("PUT", fileUrl, localCallback, null, null, content, "text/html;charset=utf-8");
		};

		// override convertUnicodeToFileFormat()
		convertUnicodeToFileFormat = function (s)
		{
			return s;
		};

	})(); //# end of wrapper
} //# end of "install only once"
//}}}

My mom gave me a gift certificate for amazon.com for Christmas.  Yay!

What shall I get with it?  I'm considering:
* [[Programming Erlang: Software For a Concurrent World|http://amazon.com/exec/obidos/ASIN/193435600X]] by Joe Armstrong (suggested by [[Myers|http://icepick.info]] -- he's reading it right now)
* [[Security Engineering: A Guide to Building Dependable Distributed Systems|http://amazon.com/exec/obidos/ASIN/0470068523]] by Ross Anderson
* [[Concepts, Techniques, and Models of Computer Programming|http://amazon.com/exec/obidos/ASIN/0262220695]] by Peter Van Roy and Seif Haridi
* [[Hacker's Delight|http://amazon.com/exec/obidos/ASIN/0201914654]] by Henry S. Warren
* [[Real World Haskell|http://amazon.com/exec/obidos/ASIN/0596514980]] by Bryan O'Sullivan, John Goerzen, and Don Stewart
* [[The New School of Information Security|http://amazon.com/exec/obidos/ASIN/0321502787]] by Adam Shostack and Andrew Stewart

I presented [[Tahoe-LAFS|http://allmydata.org]] at ~CodeCon last weekend.  [[CodeCon|http://www.codecon.org/2009/schedule.html]]'s prime directive is that every presentation has to have a live demo of working code, and that the presenter has to be an author of that code.

For my demo, I leaned an axe against the speaker's podium, strapped safety goggles around my neck, and then I showed three laptops on stage, each running a Tahoe node, and then uploaded a movie file to the Tahoe grid made up of those three nodes.  (This means the file gets automatically encrypted, digitally signed, and erasure-coded.)

Then I explained that after uploading your movie to the Tahoe grid, you might turn off your Tahoe node and go away.  And while you are gone, something BAD might happen...

<html><object width="425" height="344"><param name="movie" value="http://www.youtube.com/v/ztbIwH7gz7o&hl=de&fs=1"></param><param name="allowFullScreen" value="true"></param><param name="allowscriptaccess" value="always"></param><embed src="http://www.youtube.com/v/ztbIwH7gz7o&hl=de&fs=1" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="425" height="344"></embed></object></html>

KRUSH! KRUSH! I KRUSH YOUR FILE!

(Note that the axe breaks into pieces on the first blow.)

The laptop was donated by Adam Langley. The axe was provided by Brian Warner.  The video recording was provided by Jake Appelbaum.

[[the same video in the freedom-compatible Theora format|http://testgrid.allmydata.org:3567/file/URI%3ACHK%3Auduug6b7m2p4wlvtosozisg2su%3Azy7xex4ocpico53uxex4pb6cpohbuqzpxoym4ioucvlrccgii6ia%3A3%3A10%3A35133367/@@named=/Axe_Versus_Laptop-ffmpeg2theora-audioquality7.ogv]]

<!--{{{-->
<link rel='alternate' type='application/rss+xml' title='RSS' href='wiki.xml' />
<!--}}}-->

<!--{{{-->
<div class='header' macro='gradient vert [[ColorPalette::PrimaryLight]] [[ColorPalette::PrimaryMid]]'>
<div class='headerShadow'>
<div id='accessControlExplanationDivId' macro='accessControlExplanation'></div>
<span class='siteTitle' refresh='content' tiddler='SiteTitle'></span>&nbsp;
<span class='siteSubtitle' refresh='content' tiddler='SiteSubtitle'></span>
</div>
<div class='headerForeground'>
<div id='accessControlExplanationDivId' macro='accessControlExplanation'></div>
<span class='siteTitle' refresh='content' tiddler='SiteTitle'></span>&nbsp;
<span class='siteSubtitle' refresh='content' tiddler='SiteSubtitle'></span>
</div>
</div>
<div id='sidebar'>
<div id='sideStuff' refresh='content' tiddler='side stuff'></div>
<div id='sidebarOptions' refresh='content' tiddler='SideBarOptions'></div>
<div id='sidebarTabs' refresh='content' force='true' tiddler='SideBarTabs'></div>
</div>
<div id='displayArea'>
<div id='messageArea'></div>
<div id='tiddlerDisplay'></div>
</div>
<!--}}}-->

There seem to be a lot of Python libraries which are for holding miscellaneous tools.  I'm the maintainer of one myself -- [[pyutil|http://pypi.python.org/pypi/pyutil]].  How many others are there?  I've decided to start accumulating pointers to the ones that I notice:

|!Python utility library|!licence|
|[[pyutil|hthttp://pypi.python.org/pypi/pyutil]]|GPL or TGPPL|
|[[itools|http://pypi.python.org/pypi/itools]]|GPL|
|[[RO|http://pypi.python.org/pypi/RO]]|GPL, except for one module!|
|[[boduch|http://pypi.python.org/pypi/boduch]]|GNU Affero General Public License|

{{{
From: zooko 
Date: February 5, 2008 12:15:53 PM MST
To: Multiple recipients of list 
Subject: bulk data use cases -- SHA-256 is too slow
}}}

Folks:

Cryptographic hash functions were invented for hashing small variable-length strings, such as human-readable text documents, public keys, or certificates, into tiny fixed-length strings in order to sign them. When considering such usage, the inputs to the hash function are short -- often only hundreds or thousands of bytes, rarely as much as a million bytes. Also, the computational cost of the hash function is likely to be swamped by the computational cost of the public key operation.

Later, hash functions were pressed into service in ~MACs as exemplified by HMAC. In that usage, the inputs to the hash function tend to be small -- typically hundreds of bytes in a network packet. Also, the network is often the limiting factor on performance, in which case the time to compute the MAC is not the performance bottleneck.

I would like to draw your attention to another way that cryptographic hash functions have been pressed into service -- as core security mechanisms in a myriad of bulk data systems. Examples include local filesystems (e.g. {{{ZFS}}} [1]), decentralized filesystems (e.g. a project that I hack on: {{{tahoe-lafs}}} [2]), p2p file-sharing tools (e.g. {{{BitTorrent}}} [3], {{{Bitzi}}} [4]), decentralized revision control tools (e.g. {{{monotone}}} [5], {{{git}}} [6], {{{mercurial}}} [7], {{{darcs}}} [8]), intrusion detection systems (e.g. {{{Samhain}}} [9]), and software package tools (e.g. {{{Microsoft CLR strong names}}} [10], {{{Python setuptools}}} [11], {{{Debian control files}}} [12], {{{Ubuntu system-integrity-check}}} [13]).

Commonly in this third category of uses the size of the data being hashed can be large -- millions, billions or even trillions of bytes at once -- and there is no public key operation or network delay to hide the cost of the hash function. The hash function typically sits squarely on the critical path of certain operations, and the speed of the hash function is the limiting factor for the speed of those operations.

Something else common about these applications are that the designers are cryptographically unsophisticated, compared to designers in the earlier two use cases. It is not uncommon within those communities for the designers to believe that hash collisions are not a problem as long as second pre-image attacks are impossible, or to believe that the natural redundancy and structure of their formats protect them ("only meaningless files can have hash collisions", they say).

A consequence of these conditions is that raw speed of a hash function is very important for adoption in these systems. If you browse the references I've given above, you'll find that ~SHA-1, Tiger, and ~MD5 (!!) are commonly used, and ~SHA-256 is rare. In fact, of all the examples listed above, ~SHA-256 is used only in my own project -- tahoe-lafs. It is available in ZFS, but it is never turned on because it is too slow compared to the alternative non-cryptographic checksum.

I should emphasize that this is not just a matter of legacy -- it is not just that these older hash functions have been "grandfathered in". Legacy is certainly a very important part of it, but newly designed and deployed systems often use ~SHA-1. Linus Torvalds chose to use ~SHA-1 in his newly designed {{{git}}} decentralized revision control tool, //after// the original 2005-02-15 Wang et al. attack was announced, and roundly mocked people who suggested that he choose a more secure alternative [6]. I recently plead with the developers of the {{{darcs}}} revision control tool that they should not use ~SHA-1 for their new, backwards-incompatible design. (The issue currently hangs on whether I can find a sufficiently fast implementation of ~SHA-256 or Tiger with Haskell bindings.)

Because of my exposure to these systems, I was surprised to see a few comments recently on this mailing list that ~SHA-256 is fast enough. My surprise abated when I decided that the commentors are coming from a background where the first two use cases -- public key signatures and ~MACs -- are common, and they may not be aware that ~SHA-256 is potentially too slow for some other use cases.

Regards,

Zooko O'Whielacronx

[1] http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Tuning_ZFS_Checksums
[2] http://allmydata.org
[3] http://en.wikipedia.org/wiki/BitTorrent_%28protocol%29
[4] http://bitzi.com/developer/bitprint
[5] http://www.venge.net/mtn-wiki/FutureCryptography
[6] http://www.gelato.unsw.edu.au/archives/git/0506/5299.html
[7] http://www.selenic.com/pipermail/mercurial/2005-August/003832.html
[8] http://www.nabble.com/announcing-darcs-2.0.0pre3-tt15027931.html#a15048993
[9] http://la-samhna.de/samhain/manual/hash-function.html
[10] http://blogs.msdn.com/shawnfa/archive/2005/02/28/382027.aspx
[11] http://peak.telecommunity.com/DevCenter/setuptools
[12] http://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Files
[13] https://wiki.ubuntu.com/IntegrityCheck

This morning I perused [[the current benchmark results|http://bench.cr.yp.to/results-hash.html]] for the ~SHA-3 contest; (Note that only a few of the ~SHA-3 candidates have yet been benchmarked -- the vast majority of them have yet to appear on these graphs.) Here are [[the graphs on a powerful modern 64-bit Intel chip|http://bench.cr.yp.to/graph-hash/amd64-nmiv003.png]]: //edonr512// and //bmw512// are actually about as efficient as ~MD5 -- hooray!  But here are [[the graphs on a 32-bit PowerPC|http://bench.cr.yp.to/graph-hash/ppc32-nmi0042.png]]: not so encouraging.  Here are [[the graphs on a 64-bit PowerPC|http://bench.cr.yp.to/graph-hash/ppc64-nmi0055.png]]: also not so encouraging.  64-bit PPC is the architecture used in the Sony Playstation 3 and the Microsoft Xbox 360.  32-bit PPC is the architecture used in the Nintendo Wii, which is currently out-selling the other two consoles //combined//.

There is a tendency among programmers, who mainly work on ~PCs, to think of 32-bit architectures as the past and 64-bit as the future.  However, if you look at the numbers, it appears that 32-bit chips are becoming more and more common, not less and less.  In 2008, about 300 million ~PCs were shipped [[[1]|http://news.cnet.com/8301-10784_3-9902716-7.html]].  I guesstimate that about half of those, including the new, fast-growing segment of "netbooks" such as OLPC XO and Asus EEE PC, were 32-bit.  In the same year, //billions// of 32-bit embedded devices were shipped.  The ARM architecture alone shipped 3 billion units in 2008, and other architectures such as MIPS and 32-bit x86 shipped uncounted billions of embedded units.  Many of the newest and fastest-growing products such as the iPhone are 32-bit.

Rather than dying out, 32-bit ~CPUs are getting smaller, cheaper, lower-power, and becoming more and more common and important.

Back in 1997, during the AES contest, a smart fellow accurately predicted "The lesson of the past 20 years of computing seems to be that while the high end always gets better, the low end never goes away. We're still programming tiny 8-bit microprocessors; instead of being used in desktop computers, they're in cellular phones, automobiles, electrical meters, and smart cards. These processors will be around for a long time to come, in ~Dick-Tracy-like wristwatch communicators, small affixable processors, [...] and who knows what else (nanotechnology?)."

The same appears to be true of the 32-bit architectures.

(Other references: that were used in the writing of this article: [[[2]|http://news.cnet.com/8301-13924_3-10076795-64.html]], [[[3]|http://news.cnet.com/8301-10784_3-9902716-7.html]], [[[4]|http://landley.net/ols/ols2007/platforms.txt]], [[[4]|http://arstechnica.com/news.ars/post/20081017-september-npd-numbers-star-wars-trumps-flagging-us-economy.html]], [[[5]|http://www.shacknews.com/onearticle.x/55644]].)

<<search>><<closeAll>><<permaview>><<newTiddler>><<newJournal "YYYY-0MM-0DD">><<saveChanges>><<slider chkSliderOptionsPanel OptionsPanel "options »" "Change TiddlyWiki advanced options">>

a chronological arrangement of Zooko's work/play ; also known as a "klog"

Zooko's Hack Log

http://testgrid.allmydata.org:3567/uri/URI:DIR2-RO:j74uhg25nwdpjpacl6rkat2yhm:kav7ijeft5h7r7rxdp5bgtlt3viv32yabqajkrdykozia5544jqa/wiki.html

#sideStuff { background-color: #ccc; margin-bottom: 1em; padding: 5px;}
#accessControlExplanationDivId { margin-bottom: 3em;}

/***
|''Name''|TahoePlugin|
|''Description''|<...>|
|''Author''|Zooko|
|''Contributors''|FND, EricShulman|
|''Version''|0.2.3|
|''Requires''|HTTPSavingPlugin| 
|''Status''|@@experimental@@|
|''Source''|http://allmydata.org/trac/tiddly_on_tahoe|
|''CodeRepository''|http://allmydata.org/source/tiddly_on_tahoe/trunk/|
|''License''|GPLv2+ or TGPPLv1.0+|
|''Keywords''|<...>|
!Description
<...>
!Notes
This plugin is being developed for [[Tiddly on Tahoe|http://allmydata.org/trac/tiddly_on_tahoe]].
***/
//{{{
/* The following comment is to let jslint know which variables are supposed to be global. */
/*global version, readOnly, showBackstage, config, loadRemoteFile, wikify */
if (!version.extensions.TahoePlugin) { //# ensure that the plugin is only installed once
	version.extensions.TahoePlugin = { installed: true };

	(function () { //# wrapper
		var BASE32CHAR, BASE32CHAR_3bits, BASE32CHAR_1bits, SEP, NUMBER, HTTPLEAD, BASE32STR_128bits, BASE32STR_256bits, ALPHANUMERIC_STRING, TAHOE_FUTURE_IMMUTABLE_CAP_RE_STR, TAHOE_FUTURE_READONLY_CAP_RE_STR, TAHOE_FUTURE_WRITABLE_CAP_RE_STR, TAHOE_IMMUTABLE_CAP_RE_STR, TAHOE_READONLY_FILE_CAP_RE_STR, TAHOE_READONLY_DIR_CAP_RE_STR, TAHOE_WRITABLE_FILE_CAP_RE_STR, TAHOE_WRITABLE_DIR_CAP_RE_STR, TAHOE_NONWRITABLE_THING_CAP_RE_STR, TAHOE_WRITABLE_THING_CAP_RE_STR, TAHOE_ANY_CAP_RE_STR, splitTahoeURL, scrapeOutReadonlyCap, diminishToReadonlyCap, getReadonlyURLToThisPage;

		BASE32CHAR = '[abcdefghijklmnopqrstuvwxyz234567]';
		BASE32CHAR_3bits = '[aqiyemu4]';
		BASE32CHAR_1bits = '[aq]';
		SEP = '(?::|%3A)';
		NUMBER = '[0-9]+';
		HTTPLEAD = 'https?://(?:[^:/]+)(?::' + NUMBER + ')?/(uri|file|cap)/?';

		BASE32STR_128bits = '(' + BASE32CHAR + '{25}' + BASE32CHAR_3bits + ')';
		BASE32STR_256bits = '(' + BASE32CHAR + '{51}' + BASE32CHAR_1bits + ')';

		ALPHANUMERIC_STRING = '[A-Za-z0-9]+';

		// This is speculative: maybe in the future there will be a version of Tahoe where caps 
		// start with these symbols, and if so then this JavaScript code will magically work with 
		// that version of Tahoe.
		TAHOE_FUTURE_IMMUTABLE_CAP_RE_STR = "i_" + ALPHANUMERIC_STRING;
		TAHOE_FUTURE_READONLY_CAP_RE_STR = "r_" + ALPHANUMERIC_STRING;
		TAHOE_FUTURE_WRITABLE_CAP_RE_STR = "W_" + ALPHANUMERIC_STRING;

		TAHOE_IMMUTABLE_CAP_RE_STR = "(?:URI" + SEP + "CHK" + SEP + BASE32STR_128bits + SEP + BASE32STR_256bits + SEP + NUMBER + SEP + NUMBER + SEP + NUMBER + '|' + TAHOE_FUTURE_IMMUTABLE_CAP_RE_STR + ')';
		TAHOE_READONLY_FILE_CAP_RE_STR = "URI" + SEP + "SSK-RO" + SEP + BASE32STR_128bits + SEP + BASE32STR_256bits;
		TAHOE_READONLY_DIR_CAP_RE_STR = "URI" + SEP + "DIR2-RO" + SEP + BASE32STR_128bits + SEP + BASE32STR_256bits;
		TAHOE_WRITABLE_FILE_CAP_RE_STR = "URI" + SEP + "SSK" + SEP + BASE32STR_128bits + SEP + BASE32STR_256bits;
		TAHOE_WRITABLE_DIR_CAP_RE_STR = "URI" + SEP + "DIR2" + SEP + BASE32STR_128bits + SEP + BASE32STR_256bits;

		TAHOE_NONWRITABLE_THING_CAP_RE_STR = '(' + TAHOE_READONLY_FILE_CAP_RE_STR + '|' + TAHOE_READONLY_DIR_CAP_RE_STR + '|' + TAHOE_IMMUTABLE_CAP_RE_STR + '|' + TAHOE_FUTURE_IMMUTABLE_CAP_RE_STR + '|' + TAHOE_FUTURE_READONLY_CAP_RE_STR + ')';
		TAHOE_WRITABLE_THING_CAP_RE_STR = '(' + TAHOE_WRITABLE_DIR_CAP_RE_STR + '|' + TAHOE_WRITABLE_FILE_CAP_RE_STR + '|' + TAHOE_FUTURE_WRITABLE_CAP_RE_STR + ')';

		TAHOE_ANY_CAP_RE_STR = '(' + TAHOE_NONWRITABLE_THING_CAP_RE_STR + '|' + TAHOE_WRITABLE_THING_CAP_RE_STR + ')';

		readOnly = document.location.toString().match(new RegExp(HTTPLEAD + TAHOE_NONWRITABLE_THING_CAP_RE_STR));
		showBackstage = !readOnly;
		config.options.chkHttpReadOnly = false;
		
		/* Returns server (which is "http://$HOST:$PORT/uri"), cap, and suffix, which can be a 
		   path from the cap through the tahoe filesystem and/or trailing extra arguments. */
		splitTahoeURL = function (someURL) {
			var u, urlSuffix, candidate_cap, urlPrefix;

			u = someURL.split('/');
			urlSuffix = [];
			candidate_cap = u.pop();
			urlPrefix = u.join('/');
			while ((u.length > 0) && (!urlPrefix.match(new RegExp("^" + HTTPLEAD + "$")))) {
				urlSuffix.unshift(candidate_cap);
				candidate_cap = u.pop();
				urlPrefix = u.join('/');
			}
			// Okay we've found the HTTPLEAD.  Is the following thing shaped like a Tahoe capability?
			if (candidate_cap.match(new RegExp(TAHOE_ANY_CAP_RE_STR))) {
				// Yes!
				return {'urlPrefix': urlPrefix, 'cap': candidate_cap, 'urlSuffix': urlSuffix};
			} else {
				// No!
				return;
			}
		};

		scrapeOutReadonlyCap = function (metadata) {
			// example of tahoe-lafs json-encoded metadata:
			// [
			// "dirnode", 
			// {
			//  "rw_uri": "URI:DIR2:ouojn4oj2fa7fphdf54hz5bfaq:rf56nzb6klj3ctvssqghy2ugalp6wundystbysxujodttrhxbqwa", 
			//  "ro_uri": "URI:DIR2-RO:sznrgoyz7lbjorhe4ipzcnmluy:rf56nzb6klj3ctvssqghy2ugalp6wundystbysxujodttrhxbqwa", 
			//  "children": {
			//   "tw_empty.html": [
			//    "filenode", 
			//    {
			//     "mutable": false, 
			//     "metadata": {
			//      "ctime": 1229263396.69, 
			//      "mtime": 1229263396.69
			//     }, 
			//     "ro_uri": "URI:CHK:cofm2lm3ywu4r4efeqwjzuzyeq:dfw7oi65smf7dhtcx6wvr4ouazswprhwkvc3uopqtmvn3e7cactq:3:10:295520", 
			//     "size": 295520
			//    }
			//   ]
			//  }, 
			//  "mutable": true
			// }
			//]

			// another example:
			// [
			//  "filenode", 
			//  {
			//   "rw_uri": "URI:SSK:ouojn4oj2fa7fphdf54hz5bfaq:rf56nzb6klj3ctvssqghy2ugalp6wundystbysxujodttrhxbqwa", 
			//   "mutable": true, 
			//   "ro_uri": "URI:SSK-RO:sznrgoyz7lbjorhe4ipzcnmluy:rf56nzb6klj3ctvssqghy2ugalp6wundystbysxujodttrhxbqwa", 
			//   "size": "?"
			//  }
			// ]
			var matchobj = metadata.match(new RegExp("^\\s*\\[[^\\[]*\"ro_uri\"\\s*:\\s*\"([^\"]*)\""));
			if (matchobj) {
				return matchobj[1];
			}
		};

		diminishToReadonlyCap = function (urlPrefix, writableCap, callback) {
			var queryURL = [urlPrefix, writableCap, "?t=json"].join("/");

			loadRemoteFile(queryURL, function (success, param, txt, src, xhr) {
				if (success) {
					callback(scrapeOutReadonlyCap(txt));
				}
			}); 
		};
    
		getReadonlyURLToThisPage = function (callback) {
			if (document.location.tahoeDiminishedCapabilityURL) {
				return callback(document.location.tahoeDiminishedCapabilityURL);
			} else {
				var pieces = splitTahoeURL(document.location.toString());
				diminishToReadonlyCap(pieces.urlPrefix, pieces.cap, function (diminishedCap) {
					var diminishedURL = pieces.urlPrefix + "/" + diminishedCap + "/" + pieces.urlSuffix;
					document.location.tahoeDiminishedCapabilityURL = diminishedURL;
					callback(diminishedURL);
				});
			}
		};

		config.macros.accessControlExplanation = {
	
			handler: function (place, macroName, params, wikifier, paramString, tiddler) {
				if (document.location.toString().match(new RegExp(HTTPLEAD + TAHOE_IMMUTABLE_CAP_RE_STR))) {
					wikify("This is an immutable view of this page.  Using this URL will always show this version of this page, even if a newer version has been uploaded.", place);
				} else if (document.location.toString().match(new RegExp(HTTPLEAD + TAHOE_NONWRITABLE_THING_CAP_RE_STR))) {
					wikify("This is a read-only view of this page.  If you share this URL with someone, they will be able to see the most recent version of this page, but not to change the page.", place);
				} else if (document.location.toString().match(new RegExp(HTTPLEAD + TAHOE_WRITABLE_THING_CAP_RE_STR))) {
					getReadonlyURLToThisPage(function (readonlyCap) {
						wikify("This is a writable view of this page.  If you share this URL with someone, they will be able to change this page.  Click here for a [[read-only view of this page|" + readonlyCap + "]].", place);
					});
				} else {
					wikify("You are accessing this page not through the Tahoe-LAFS secure, distributed filesystem.", place);
				}
			}
		};
	})(); //# end of wrapper
} //# end of "install only once"
//}}}

Several people have publicly and privately voiced their discontent with the way that the Open Source Initiative is handling [[my Request For Approval|http://www.crynwr.com/cgi-bin/ezmlm-cgi?17:sss:478:200901:ofpndmgcgmbhbmimpkpe#b]] for [[The Transitive Grace Period Public Licence|http://zooko.com/tgppl.pdf]] and [[my accusation of wrongdoing|http://www.crynwr.com/cgi-bin/ezmlm-cgi?17:mss:491:200901:hnghfggnikagbelnanba]].  I'm glad that other people are paying attention because there is an issue at stake which is wider than the TGPPL itself -- it has to do with the OSI's role in the Open Source/Free Software community.

The OSI is trying to exercise a general authority to encourage or suppress the use of open source licences.  That's fine -- perhaps they //should// have that authority.  But if so, it needs to be explicitly stated and transparently executed so that the wider community understands what they are doing and what is meant by the phrase "~OSI-approved".  Currently they are trying to accomplish their goal of discriminating //among// open source licences by overusing their community-sanctioned authority to judge //whether// a licence is open source.  That is, they are refusing to formally recognize a licence (the TGPPL) as ~Open-Source-Definition-conformant even though they admit that it //is// ~OSD-conformant, because they wish to reduce the number of open source licences in use (the "licence proliferation problem").  Since this is inconsistent with the community's expectations and with the OSI's own documentation, it is wrong.

One particularly frustrating thing about this is that OSI [[has already initiated|https://ideas.opensource.org/attachment/ticket/157/email]] and is [[currently working on|http://www.crynwr.com/cgi-bin/ezmlm-cgi?17:msn:491:hnghfggnikagbelnanba]] a plan to separate out the question of whether a licence is open source from the question of whether they recommend it.  They could go ahead and formally acknowledge what they have already publicly admitted -- that the TGPPL is an open source licence -- and proceed with their plan to set up an explicit, transparent, and community-sanctioned process for formalizing their broader judgments.  This would protect such trust and confidence as the people vest in OSI.  I have already had several well-known members of the community tell me privately that they recommend simply ignoring the Open Source Initiative.  That's too bad, and it is unnecessary.

This klog is insecure.

Last night while saying "Good-bye" to Trevor Stone as he left Game Night At The O'Whielacronxes House, I mentioned to him that I was now I was now linking to his blog from mine.  (He replied "Oh.  I didn't know you had a blog.")

A moment later I realized that my klog is insecure.  I realized that there are a few people that have the ability to change its contents, to whom I hadn't intended to grant that ability.  Sigh.  Well, learning about things like this is why I use {{{tiddly_on_tahoe}}} for this klog (see [[about this klog]]).  Computer security is often not about math, but about [[human-computer interaction|http://en.wikipedia.org/wiki/Human-computer_interaction]], and therefore many important questions about not theoretical questions best answered with proofs but empirical questions best answered by observation and experiment.  This klog is an experiment (a barely controlled one), and it has now delivered its second useful result to me.  Hopefully I'll write up my conclusions for [[the tahoe-dev list|http://allmydata.org/pipermail/tahoe-dev]], but for now it is much more urgent that I finish release Tahoe-1.3.0.
added to [[things to read]]:
* [[Usable Security Blog|http://usablesecurity.com]]
P.S.  If any of those people to whom I've accidentally given write-access to my klog read this, figure out what I meant, //and// actually exercise that access to edit this page without my help, then I'll praise them and buy them a congratulatory drink.  If you're not one of those people, you'll require the cooperation of one of those people in order to get write access to my klog.  I'll say no more, for now.

|StyleSheet|##AuthorStyles|
|StyleSheetReadOnly|##ReaderStyles|

!AuthorStyles
/*{{{*/
[[StyleSheet]]
body {
	background: #eee;
}
/*}}}*/

!ReaderStyles
/*{{{*/
[[StyleSheet]]
body {
}
/*}}}*/

http://xkcd.com/428

My co-workers at [[allmydata.com|http://www.allmydata.com/index.php?tracking=zookos_hlog]] and hacking partners at [[allmydata.org|http://allmydata.org]] would like to know what I'm doing or have recently done or intend to do soon.  So would I!  And so would you, or else you wouldn't be reading this.

By the way, this klog is a [[TiddlyWiki|http://tiddlywiki.com]] which is stored on a Tahoe decentralized filesystem.  That means it is a //decentralized web app//.  Is that not cool?  I think that's awesome.  This also means, as Zandr informed me, that this decentralized version of ~TiddlyWiki Just Works on the iPhone, unlike the normal ~TiddlyWiki.

If this klog doesn't satisfy your thirst to know what Zooko is up to, you can also peruse the allmydata.org [[Timeline|http://allmydata.org/trac/tahoe/timeline]] and the [[tahoe-dev|http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev]] mailing list.

You can have a ~TiddlyWiki-on-Tahoe of your very own -- just follow [[these instructions|http://allmydata.org/pipermail/tahoe-dev/2008-December/000938.html]].

Here are some assorted documents about technical failures.  I'm mainly using this as a permanent repository of this information for my own future reference, since the original sources themselves, as well as their context, will probably be lost in a couple of years.  Obviously there is a lot more that I could do for this collection: better and more thorough acquisition of failure "war stories" from the wild (which reminds me that I ought to subscribe to RISKS digest again), better curating of this collection (i.e. explaining the context of each one, and analyzing them), etc..
So far the first two have to do with the way that tiny little details about clocks can lead to bigger failures.  I think one of the reasons that I am paying attention to these stories is that Brian doesn't entirely share my intuitions about safety engineering and clocks.  :-)
* The cron job for the Debian/Ubuntu update tool "apt" can get stuck, depending on when Daylight Savings Time happens in your timezone and time of day that the cron job starts and completes: [[apt lost progress due to Daylight Savings Time|../../file/URI%3ACHK%3Axuwyk7jji7nirgwmef3mhdpr74%3A6ba7od2ydryediodgzcstfikzb7caty6drapuozmcj7yxwliztxa%3A3%3A10%3A15331/@@named=/bugreport.cgi%3Fbug%3D523213.html]]
* Every Zune in the world locked up on December 31, 2008: [[Zune lock up on December 31, 2008|../../file/URI%3ACHK%3A27dycwcdsncqcdeiewmarhyqf4%3Aswt2xah4mfzd35djubfdoq7kgxa5g2izehnxnmsuw2zhnbflapca%3A3%3A10%3A278416/@@named=/38143-cause-zune-30-leapyear-problem-isolated.html]]
Here is [[the tahoe dir|http://testgrid.allmydata.com:3567/uri/URI:DIR2-RO:h7pjn6prggf37p4of47b5ustb4:h7vf55ularzdwnoyp6hi5bqaka4wixstnhlz43xbaowohz3upnwa]] where I am keeping these.

//My mom and others have pointed out that my blog is chock full of acronymy goodness and they can get only the vaguest idea of what I'm talking about, mostly by reading the verbs.  I don't like those sorts of communication gaps (even though they are inevitable), so I've decided to try the exercise of translating each item that I post from programmer-speak to English.  If it is hard to translate into English, maybe this tells us something about what it means in its original programmer-speak.//

(I wrote this as a response to Joseph Miklojcik's blog entry [[Darcs' Theory of Patches Isn't Useful|http://jfm3-repl.blogspot.com/2009/01/darcs-theory-of-patches-isnt-useful.html]].)

Hi!  I'm a darcs-lover.  Thanks for writing this blog entry.

Your criticism of darcs is a common one.  It is nice that you write a concrete example -- that helps focus the mind.

The only thing that I would disagree with you about is your suggestion that darcs is //intended// to somehow detect or solve the semantic issues that you illustrate, or that darcs's failure to do so means that darcs isn't useful.

I've been struggling for a while now to articulate how it is that darcs can use more information than git does in computing its merges, and compute correct answers to more interesting questions about the source code, without actually attempting to reach all the way up and answer questions about the semantics of the program (or other text or data) which is being merged.

To someone who is used to cvs/svn/git style merge, darcs's merge seems to be doing something magical, and suspiciously close to "guessing what the user meant". Of course it isn't actually doing either of these things. (To place blame where due, darcs's merge algorithm is woefully under-documented and is mixed in with a whole bunch of other ideas, so if the git users fail to understand what darcs is doing, this is mostly darcs's fault for not explaining it properly.  I expect that eventually someone will either steal or rediscover the simple idea of darcs merge and integrate it into a different revision control tool -- possibly git.)

So what is this "simple idea"?  It's like this: suppose you've written a patch on your branch which changes a line of code in a file.  Now suppose you give your patch to someone else, but over in their branch that file has moved.  The simple idea is that //your patch should be applied to the right file//.  It should not be applied to "whatever file is currently located at that pathname in the other branch", and it should not be applied to "whatever file has contents like the contents of the original file".  It should be applied to ''that file'', even if the file has moved locations or changed its contents.  This is what users naively expect the revision control tool to do, and the revision control tool is able to do it without any magic or "guessing what the user meant", and it is more useful than the alternatives, so that's what should be done.

Tthis same simple idea can be used to determine "which lines of code within a file should this hunk apply to", just as "which file within the tree should this patch apply to".  Here is my concrete example: [[badmerge/simple.html|https://zooko.com/badmerge/simple.html]].

I hope that you can see how an algorithm which computes an answer to this question: "where did the file move to in this tree" or "where did the line move to in this file" can be reliable and useful without attempting to interpret the meaning of C source code (nor of HTML or English text if that's what you use your revision control tool for).

Note for nitpickers: yes, in order to correctly compute "where did the file move to in this tree", darcs needs to be explicitly told which actions are file mv's and which are deletion of the old file followed by creation of a new file (with suspiciously similar contents) in a different location.  Some nitpickers have argued that explicitly writing {{{darcs mv file new/path/to/file}}} is too much to ask of users and that instead the source control tool is going to have to figure out what happened after the fact once the user has run {{{mv file new/path/to/file}}}.  When they were arguing this in the year 2000, I agreed with them that this might turn out to be a serious problem, but in the ensuing years it has turned out that the vast majority of darcs users are happy to write {{{darcs mv file new/path/to/file}}}, and that on the occasions that they forget to do so the consequences are not too bad, so in 2009, this argument is wrong and should not be repeated.  There is a similar but subtler nitpick concerning which-lines-in-a-file but I'm going to stop here.

("OSD" stands for the [["Open Source Definition"|http://www.opensource.org/docs/osd]].)

From: Zooko
To: the Open Source Initiative
Date: Wed, 21 Jan 2009 17:19:29 UTC
Subject: [[TGPPL, deterring licence proliferation by withholding the stamp of OSD-conformance|http://www.crynwr.com/cgi-bin/ezmlm-cgi?17:mss:491:200901:hnghfggnikagbelnanba]]

"It appears that in your concern about licence proliferation you have started to use your power of withholding certification of ~OSD-conformance as a general purpose tool to deter the use of [open source] licences that you don't like.  This is wrong." 

Here are some tickets on open source projects that I'm interested in tracking.  See also [[issue tickets closed]].
* [[nautilus (launchpad #372287)|https://bugs.launchpad.net/ubuntu/+source/nautilus/+bug/372287]]: If you drag and drop a file whose name is invalidly encoded and whose mangled name collides with an extant file then the window doesn't refresh. (sometimes!?)
* [[python #5720|http://bugs.python.org/issue5720]]: ctime: I don't think that word means what you think it means.
* tahoe
** [[tiddly_on_tahoe #9|http://allmydata.org/trac/tiddly_on_tahoe/ticket/9]]: can't save from Konqueror-4.2.0
** [[tahoe #615|http://allmydata.org/trac/tahoe/ticket/615]]: Can ~JavaScript loaded from Tahoe access all your content which is loaded from Tahoe?
** [[tahoe #608|http://allmydata.org/trac/tahoe/ticket/608]]: premature abort of upload if some shares were already present and some servers fail
** [[tahoe #331|http://allmydata.org/trac/tahoe/ticket/331]]: add DSA to pycryptopp - serialize pubkeys with less fluff
** [[tahoe #605|http://allmydata.org/trac/tahoe/ticket/605]]: delayed connection on Windows
** [[tahoe #591|http://allmydata.org/trac/tahoe/ticket/591]]: "make quicktest" could be quicker and less noisy
** [[tahoe #556|http://allmydata.org/trac/tahoe/ticket/556]]: prepend 'application-version' with the name of this particular application
** [[tahoe #534|http://allmydata.org/trac/tahoe/ticket/534]]: "tahoe cp" command encoding issue
** [[tahoe #217|http://allmydata.org/trac/tahoe/ticket/217]]: ~DSA-based mutable files -- small ~URLs, fast file creation
** [[tahoe #596|http://allmydata.org/trac/tahoe/ticket/596]]: storage servers should announce that they support over-read
** [[tahoe #424|http://allmydata.org/trac/tahoe/ticket/424]]: stdeb: push to upstream
** [[tahoe #423|http://allmydata.org/trac/tahoe/ticket/423]]: stdeb: use stdeb on tahoe itself
** [[tahoe #422|http://allmydata.org/trac/tahoe/ticket/422]]: stdeb: run from buildslaves
** [[tahoe #530|http://allmydata.org/trac/tahoe/ticket/530]]: use setuptools's """--multi-version""" mode
** [[tahoe #534|http://allmydata.org/trac/tahoe/ticket/534]]: "tahoe cp" command encoding issue
** [[tahoe #558|http://allmydata.org/trac/tahoe/ticket/558]]: kpreid says that the -SUMO tarballs don't exist
* pycryptopp
** [[pycryptopp #2|http://allmydata.org/trac/pycryptopp/ticket/2]]: deterministic generation of private key from small seed
** [[pycryptopp #3|http://allmydata.org/trac/pycryptopp/ticket/3]]: serialize ecdsa keys without the fluff
** [[pycryptopp #9|http://allmydata.org/trac/pycryptopp/ticket/9]]: link against existing (system) libcrypto++.so
** [[pycryptopp #11|http://allmydata.org/trac/pycryptopp/ticket/11]]: submit patch for Brainpool ECC to Crypto++
** [[pycryptopp #12|http://allmydata.org/trac/pycryptopp/ticket/12]]: automatic wrappers for all of Crypto++
** [[pycryptopp #13|http://allmydata.org/trac/pycryptopp/ticket/13]]: DSA "semi-private"/intermediate keys
* [[ubuntu p7zip #322481|https://bugs.launchpad.net/ubuntu/+source/p7zip/+bug/322481]]: took an order of magnitude longer than expected to compress
* [[amarok #184834|https://bugs.kde.org/show_bug.cgi?id=184834]]: amarok crashes on "Quit" 
* konqueror
** [[konqueror #184157|http://bugs.kde.org/show_bug.cgi?id=184157]]: crashed (I might have just hit the "back" button)
** [[konqueror #321636|https://bugs.launchpad.net/ubuntu/+source/kdebase-kde4/+bug/321636]]: kioslave crashes when logging into my issue tracker
** [[konqueror #321656|https://bugs.launchpad.net/ubuntu/+source/kdebase-kde4/+bug/321656]]: iso-8859-1 and/or utf-8 character not decoded properly
* [[pyopenssl #238658|https://bugs.launchpad.net/pyopenssl/+bug/238658]]: package pyOpenSSL for easy_install on Windows -- @@almost fixed@@
* [[tiddly_on_tahoe #7|http://allmydata.org/trac/tiddly_on_tahoe/ticket/7]]: wrong error message when server is unreachable
* [[pywin32 #1799934|https://sourceforge.net/tracker2/?func=detail&aid=1799934&group_id=78018&atid=551954]]: easy_install pywin32 -- @@almost fixed@@
* setuptools
** [[setuptools #20|http://bugs.python.org/setuptools/issue20]]: package required at build time seems to be not fully present at install time?
** [[setuptools #57|http://bugs.python.org/setuptools/issue57]]: {{{develop}}} doesn't create {{{.pth}}} files and {{{site.py}}} if {{{--multi-version}}}
** [[setuptools #53|http://bugs.python.org/setuptools/issue53]]: respect the PYTHONPATH
** [[setuptools #54|http://bugs.python.org/setuptools/issue54]]: be more like distutils with regard to """--prefix="""
** [[setuptools #17|http://bugs.python.org/setuptools/issue17]]: easy_install will install a package that is already there; This issue should probably be renamed in light of the fact that it seems to cause a worse failure nowadays with the proposed Debian packages for {{{foolscap}}} and {{{tahoe-lafs}}}.
* nevow
** [[nevow #2830|http://divmod.org/trac/ticket/2830]]: setup.py incorrectly declares twisted.plugins to be a package
** [[nevow #2713|http://divmod.org/trac/ticket/2713]]: setup.py installs tests, but not documentation
** [[nevow #2629|http://divmod.org/trac/ticket/2629]]: Nevow doesn't declare its dependency on Twisted in a machine-parseable way
** [[nevow #2699|http://divmod.org/trac/ticket/2699]]: build nevow without importing nevow
** [[nevow #2798|http://divmod.org/trac/ticket/2798]]: setup.py install """--home""" is broken :-(
* [[pyflakes #2720|http://divmod.org/trac/ticket/2720]]: Release Pyflakes
* buildbot
** [[buildbot #266|http://buildbot.net/trac/ticket/266]]: I wish to tell my buildmaster: "restart yourself the next time you quiesce"
** [[buildbot #212|http://buildbot.net/trac/ticket/212]]: buildbot doesn't respond to darcs tags
** [[buildbot #395|http://buildbot.net/trac/ticket/395]]: when i change the vcs executable, buildslave stops being able to invoke it until I restart buildslave
** [[buildbot #396|http://buildbot.net/trac/ticket/396]]: Older builds
** [[buildbot #252|http://buildbot.net/trac/ticket/252]]: side-effecty operations (Force Builder) should be ~POSTs
** [[buildbot #407|http://buildbot.net/trac/ticket/407]]: {{{darcs_buildbot}}} uses {{{.encode('ascii')}}}, but {{{.encode('utf-8')}}} works better @@patch submitted@@
* twisted
** [[twisted #3649|http://twistedmatrix.com/trac/ticket/3649]]: more specific warning about plugin cache
** [[twisted #3568|http://twistedmatrix.com/trac/ticket/3568]]: ERROR from conch test when pycrypto is not installed @@patch submitted@@
** [[twisted #2466|http://twistedmatrix.com/trac/ticket/2466]]: Failures use a lot of memory @@patch submitted@@
** [[twisted #3586|http://twistedmatrix.com/trac/ticket/3586]]: I want to install twisted without a c compiler
** [[twisted #2234|http://twistedmatrix.com/trac/ticket/2234]]: Select default reactor based on platform and available libraries
** [[twisted #1956|http://twistedmatrix.com/trac/ticket/1956]]: Make a less sucky producer/consumer API
** [[twisted #3529|http://twistedmatrix.com/trac/ticket/3529]]: closing stdout in a child process on cygwin means that process doesn't receive bytes from stdin anymore. I think.
* darcs
** [[darcsver #2|http://allmydata.org/trac/darcsver/ticket/2]]: use "darcs query" to get count of patches faster
** [[darcs #1153|http://bugs.darcs.net/issue1153]]: darcs waits to hear back from servers unnecessarily
** [[darcs #1303|http://bugs.darcs.net/issue1303]]: proposal: make "darcs changes" interactive by default
** [[darcs #26|http://bugs.darcs.net/issue26]]: Darcs needs real MIME parsing, fails with Mail.app, Courier
** [[darcs #1255|http://bugs.darcs.net/issue1255]]: darcs put tries to convert to darcs-2-format?
* foolscap
** [[foolscap #107|http://foolscap.lothar.com/trac/ticket/107]]: exceptions.~KeyError: "unable to find reference for name
** [[foolscap #105|http://foolscap.lothar.com/trac/ticket/105]]: make it easy to distinguish server-side failures/exceptions from client-side
** [[foolscap #108|http://foolscap.lothar.com/trac/ticket/108]]: set base to "." if not running from source (so {{{flogtool}}} works on Windows)
** [[foolscap #109|http://foolscap.lothar.com/trac/ticket/109]]: make a "flogtool" executable that works on Windows
** [[foolscap #111|http://foolscap.lothar.com/trac/ticket/111]]: timestamps of incident files -- TZ indicator please
** [[foolscap #112|http://foolscap.lothar.com/trac/ticket/112]]: timestamps of incident files -- ~ISO-8601'ish
** [[foolscap #113|http://foolscap.lothar.com/trac/ticket/113]]: timestamps of incident files -- UTC
* [[ubuntu #314468|https://bugs.launchpad.net/hardy-backports/+bug/314468]]: Please backport setuptools-0.6c9 from Intrepid.

* [[debian #510901|http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=510901]]: python-foolscap: should advertise [secure_connections] feature to setuptools @@fixed!@@
* [[tiddlywiki #658|http://trac.tiddlywiki.org/ticket/658]]: ~SiteUrl as current document location @@fixed@@
* [[darcs #1217|http://bugs.darcs.net/issue1217]]: darcs put fails with 'darcs failed:  Malformed patch bundle: '{' is not 'Context:' ' -- @@superceded by darcs #1255@@
* [[pyopenssl #311600|https://bugs.launchpad.net/pyopenssl/+bug/311600]]: please update http://pypi.python.org/simple/pyOpenSSL -- @@fixed@@
* [[tahoe #555|http://allmydata.org/trac/tahoe/ticket/555]]: tahoe .deb cannot be installed on hardy: simplejson dependency is too new -- @@fixed@@
* [[tiddly_on_tahoe #2|http://allmydata.org/trac/tiddly_on_tahoe/ticket/2]]: don't offer the option to save changes when you are viewing read-only -- @@fixed!@@
* [[tiddly_on_tahoe #3|http://allmydata.org/trac/tiddly_on_tahoe/ticket/3]]: offer a read-only cap to the user -- @@fixed!@@
* [[nevow #2527|http://divmod.org/trac/ticket/2527]]: easy_install compatibility -- @@hooray!  This issue is finally fixed!@@
* [[pyflakes #2709|http://divmod.org/trac/ticket/2709]]: Pyflakes svn doesn't install properly due to missing packages -- @@hooray!  Fixed!@@
* [[buildbot #236|http://buildbot.net/trac/ticket/236]]: show elapsed time for steps -- @@fixed@@

I submitted the following proposal in response to [[the CodeCon RFP|http://www.codecon.org/2009/cfp.txt]]:

Project name: Tahoe, the ~Least-Authority Filesystem
Track: code
url of home page: http://allmydata.org
tagline: a secure, decentralized, fault-tolerant storage network //(whoops I forgot to say ''"cloud"'' again)//
presenter: Zooko ~Wilcox-O'Hearn, http://zooko.com
alternate/backup/co- presenter: Brian Warner, http://lothar.com

project history: In 2006 I got to start fresh on inventing a secure, decentralized storage network, after the failure of Mojo Nation (for which I was partially responsible), the failure of Mnet (for which I was primarily responsible), the observed failures of Freenet, and the ongoing failure of a proprietary commercial backup system written by allmydata.com (for which I was partially responsible), not to mention a few other failures that I also tried to learn from.  I tried to learn from the success of ~BitTorrent by starting fresh and limiting the scope.  Also, I was blessed with a supportive company and the kick-ass engineering skills of Brian Warner, and I finally got a secure, decentralized storage network that didn't fail!  allmydata.com deployed Tahoe a year ago and copied all of their customer data over to Tahoe from the old proprietary system.  Open source hackers are building on it.  It works!

novelty: Tahoe is comparable to Freenet, ~OceanStore, and Mojo Nation.  It avoids some of the trickier problems in this space by limiting the scope: Tahoe assumes that the set of storage servers is not too large or dynamic, and that there are enough servers that are at least moderately reliable.  This means it doesn't even *try* to solve the Very Hard Problem of sharing storage with millions of anonymous strangers, but on the other hand it does a fine job of sharing storage among a couple hundred moderately reliable servers, such as a "friendnet" (home computers operated by your friends and family) or the allmydata.com commercial grid.  On top of this pool of moderately-reliable servers, Tahoe adds encryption for confidentiality and integrity, erasure-coding for high reliability, and capabilities for file-sharing.  The "Principle of Least Authority" design means that the system relies on each component as little as possible -- security properties such as confidentiality, integrity, and access control are all guaranteed by the client on its own behalf using cryptography instead of relying on the servers to cooperate in providing those properties.  To get the reliability properties that it wants the client *does* require the help of the servers, but by the power of erasure coding, only a subset of the servers need to perform only moderately well for the reliability properties to hold.  Tahoe is the only open source project that I know of which offers these sorts of properties in a practical system that many people use every day.

demo: I haven't thought this through all the way, but at several hacker parties in the past we've had partiers install Tahoe on their laptops and form a "temporary autonomous zone" storage system on which to share music and movies.  When the laptops close up and go home, the temporary autonomous zone is destroyed and all of the files become unrecoverable (unless a quorum of the partiers were to later reconvene and reconnect their laptops).  Maybe we could figure out a way to have some such live audience participation in the demo.  It has worked at parties with dozens of attendees, but I'm not sure if it would fit into a ~CodeCon demo.  Failing that, I can always demo the user-facing applications that run from Tahoe, such as streaming movies and "gridapps", which are ~JavaScript applets that are stored in Tahoe and executed in your web browser.  Maybe I could cook up some sort of demo involving suddenly and violently destroying one of the storage servers and then demonstrating that all the content is still available because of the survival of the other ones.  Hey, that sounds like fun!  As you can see, I don't have a precise plan yet.  Nor money to spend on a sacrificial removable hard drive or two.  :-)

slides: I have none prepared specifically for ~CodeCon yet.  Here is the peer-reviewed short paper that I presented at the Storage Security and Survivability Workshop -- http://allmydata.org/~zooko/lafs.pdf .  Here are the slides that I used at that presentation: http://zooko.com/lafs/presentation/index.html .  At that presentation I did actually load each of the slides on demand from a live Tahoe grid so it was a demo as well as a presentation.

future plans: 1.  Support more and more people building on top of Tahoe, such as allmydata.com's backup business, and several open source projects that are currently building on top of Tahoe.  I'm especially interested in "gridapps", which might evolve into a distributed computation platform that can be built with the world's vast supply of web app development expertise.  "Gridapps" look exactly like web apps, but all of their storage is in the decentralized, secure tahoe grid, and they have access to the convenient capability-based file-sharing API (over HTTP), so they could do some interesting things.

Future plan #2: fix the glaring deficiencies that we already know about, plus all the new ones that will be revealed in the process of Future plan #1.

Future plan #3: document the file formats and protocols in sufficient precision that others could write a compatible implementation from the spec.

Future plan #4: design better-performing and safer cryptographic mechanisms and better-performing and more versatile filesystem semantics.

[a letter I wrote to a discussion group]

During the meeting yesterday morning someone's voice -- I believe it was the voice of Bill Frantz -- said that he preferred to avoid public key cryptography when possible because it could be broken by quantum computers, if they existed.  This is true of all the big ones -- RSA, DSA, ECDSA, etc.  Fortunately, there are a few public key crypto systems which (as far as we know) aren't vulnerable to quantum computers: Merkle Hash-based signatures, ~McEliece code-based crypto, lattice-based crypto, and something that I completely don't understand called multivariate-quadratic-equations.  Here is Daniel J. Bernstein's overview at a conference of the topic of "post-quantum cryptography" in 2008:

http://math.uc.edu/~aac/pqcrypto2008/presentations/bernstein.pdf

The lattice-based ones are especially interesting to me because they are already implemented and used today, and because they are faster than RSA or ECDSA and have smaller key sizes than RSA.  See the part about "NTRU" in here:

http://postquantum.cr.yp.to/pqcrypto2006record.pdf

Unfortunately they have much larger key sizes and costs of key generation than ECDSA, and there are also many patents currently in force on lattice-based techniques, so I don't expect to use them any time soon.

However, when thinking about the longer-term, it seems like there is a good chance that public key crypto will continue to work even if real quantum computers are built.  On the other hand, don't forget the time Blueshell had a humor fit at Pham's faith in public-key cryptography.

Regards,

Zooko

(Here's the link to [[the conversational context|http://www.overcomingbias.com/2009/01/open-thread.html#comment-144823758]].  See also my earlier post -- a book review: [[Good Calories, Bad Calories]].)

Hal:

You raise a good objection that you can't tell if Taubes is omitting important contrary data or arguments. How can one find out if such important contraries exist but are omitted? An efficient way to do so is debate, which for large written works like this takes the form of a rebuttal. If Taubes has omitted significant evidence or important argument, then people who know a lot about that evidence and argument and who believe Taubes is wrong can be relied upon to inform us about them.

I'm aware of one such rebuttal of "Good Calories, Bad Calories". (Only one!! If anyone has more, please let me know.) It is by Dr. George Bray, who is, according to low-carb guru Dr. Mike Eades "probably the most renowned figure in the field of obesity research today", and whose contributions to the field are mentioned in the book itself. Here's the link to the full rebuttal:

[[bray-review-of-gcbc.pdf|http://www.proteinpower.com/drmike/wp-content/uploads/2008/07/bray-review-of-gcbc.pdf]]

Unfortunately, Dr. Bray seems to have misunderstood or even failed to read important parts of the work he is rebutting, since he claims that the book omits the distinction between low-density lipoproteins and high-density lipoproteins, which it does not, and that it evinces a misunderstanding of the First Law of Thermodynamics, which it does not.

That last part is really the key: Dr. Bray and his colleagues are committing the classic error of looking at a relation and assuming the direction of causation. The First Law of Thermodynamics dictates that delta energy storage (roughly, weight gain), equals energy in minus energy out (given a few plausible assumptions about what counts as a "closed system" in this case). Everyone in the debate agrees on that point. What the First Law of Thermodynamics does not tell us is the direction of causation. Does energy imbalance cause obesity, or does obesity cause energy imbalance? (Or more complex combinations of causation?) Dr. Bray and company intuitively believe the former direction: they think the causation must flow from human decisions to eat more or less food, and human decisions to exercise more or less, to deposition of fat in human fat cells. This is not the only causal explanation which is consistent with the First Law of Thermodynamics, but Dr. Bray appears to think that it is. We can tell, because he seems to think that if Taubes disagrees with this causal direction, then Taubes must misunderstand the First Law of Thermodynamics. We can also tell by the way Bray asserts that direction of causation without justification, perhaps because he thinks it is too obvious to require justification or that it is the only logical explanation -- search in the text of his rebuttal for the phrase "result of".

I have a hypothesis about why so many well-versed researchers make this unjustified assumption: it is because of their belief in Free Will. If the arrow of causation has the pointy end aiming at human decisions, then this violates the notion that humans are free to choose their own fate, and this is either inconceivable or abhorrent. Therefore, the arrow of causation must have the blunt end towards human decisions and the pointy end towards weight gain. Taubes doesn't really explore the notion of Free Will in his book -- too bad. Room for follow-up work.

By the way, here is Taubes's rebuttal to Bray's rebuttal. Now that you've read mine, you don't need to read Taubes's so much. ;-)

[[taubes-response-to-bray-ob-reviews.pdf|http://www.proteinpower.com/drmike/wp-content/uploads/2008/07/taubes-response-to-bray-ob-reviews.pdf]]

Anyway, back to Hal's original question: how can you tell if Taubes is omitting some important pieces? I think rebuttal is the best way to tell. This rebuttal by Bray does point out some omissions in "Good Calories, Bad Calories", although unfortunately it also (I think) incorrectly alleges some other omissions. This points up the problem with this approach -- how do we know that Dr. Bray hasn't failed to notice more important omissions in "Good Calories, Bad Calories"? Especially since he made those two huge blunders I described above. That's why I'm hoping for more better rebuttals. But ultimately, we can't know. Taubes could be omitting tremendously important aspects in his book. Bray could be omitting to point out omissions. I'm still totally willing to put down cash on Taubes being the righter of the two (although I might want to first dig into those meta-studies the Bray mentions which covered five studies of law-carb diets). Too bad there's no legal, high-volume open market for such a bet.

blogroll:
[[Mark Seaborn|http://lackingrhoticity.blogspot.com]], [[Sameer Parekh|http://blog.creativedestruction.com]], [[Wes Felter|http://wmf.editthispage.com]], [[Paul Hsieh|http://geekpress.com]], [[Trevor Stone|http://flwyd.livejournal.com]], [[Brian Warner|http://www.lothar.com/blog]], [[Chris Hibbert|http://pancrit.blogspot.com]]

reading:
[[things to read]]; [[things read|things I have read]]; currently reading: //[[Intellectual Property and Open Source|http://www.amazon.com/dp/0596517963]]// by Van Lindberg

open source hacking:
[[issue tickets]]; [[tickets closed|issue tickets closed]]

trying to learn from failures: [[collection of failures]]

[img[RSS feed|/file/URI:CHK:om3vmqaqojftyfbcupux63oocu:xvhw2ke7ikb3d676ngbfvup42bzyt7gpfmwnf7jt3jepbruufcxq:3:10:689/@@named=/feed-icon-14x14.png][http://testgrid.allmydata.org:3567/uri/URI:DIR2-RO:j74uhg25nwdpjpacl6rkat2yhm:kav7ijeft5h7r7rxdp5bgtlt3viv32yabqajkrdykozia5544jqa/wiki.xml]] [[RSS feed for this blog|http://testgrid.allmydata.org:3567/uri/URI:DIR2-RO:j74uhg25nwdpjpacl6rkat2yhm:kav7ijeft5h7r7rxdp5bgtlt3viv32yabqajkrdykozia5544jqa/wiki.xml]]

Here are some things I have read.
* [[2009-05-05]]
** John C. Wright: //[[The Golden Age|http://www.amazon.com/dp/0812579844]]// trilogy
** Jeff Bonwick (inventor of ZFS): [[on timeouts|http://storagemojo.com/2008/11/25/stupid-storage-failures/#comments]] (as [[recommended by Wes Felter|http://wmf.editthispage.com/2008/11/29]])
** Oscar Bonilla: [[Visualizing Bayes' Theorem|http://oscarbonilla.com/2009/05/visualizing-bayes-theorem]]
* [[2009-04-29]]
** Stefan Tillich: [[Hardware Implementation of the SHA-3 Candidate Skein|http://eprint.iacr.org/2009/159]]; See journal entry on [[2009-04-29]] for comments.
* 2009-01-20 (By the way, I read a lot of things between 2009-01-20 and 2009-04-29, but I forgot to note them here.  Sorry.)
** Ewan Fleischmann, Christian Forler, and Michael Gorski: [[Classification of the SHA-3 Candidates (2009-09-19 edition)|http://eprint.iacr.org/2008/511]]
* 2009-01-18
** Huseyin Hisil, Kenneth ~Koon-Ho Wong, Gary Carter, and Ed Dawson: [[Twisted Edwards Curves Revisited|http://eprint.iacr.org/2008/522]]; I don't really understand much of the math, but it looks like elliptic curve cryptography is going to get more efficient and safer as results like these trickle down to practice.
** James Hamilton: [[The Case For Low-Cost, Low-Power Servers|http://perspectives.mvdirona.com/2009/01/15/TheCaseForLowCostLowPowerServers.aspx]] (via [[Wes Felter's blog|http://wmf.editthispage.com/2009/01/16]]), and posted a question about it to James Hamilton's blog
* 2009-01-17
** Peter Gutmann: [[The Crypto Gardening Guide and Planting Tips|http://www.cs.auckland.ac.nz/~pgut001/pubs/crypto_guide.txt]]
* earlier
** James Hamilton: [[The Cost of Bulk Cold Storage|http://perspectives.mvdirona.com/2008/12/22/TheCostOfBulkColdStorage.aspx]] as [[recommended by Wes Felter|http://wmf.editthispage.com/2008/12/27]]
** [[Some Results from EDOS on Dependency Checking|http://mancoosi.org/edos]]

*  Scott Contini, Ron Steinfeld, Josef Pieprzyk, Krystian Matusiewicz: [[A critical look at cryptographic hash function literature (2007)|http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.83.7429]]

* finish letter to Shawn, Josh, tahoe-dev //Re: [tahoe-dev] Fwd: On the value of "proofs"...//
* read ~David-Sarah Hopwood //[[Jacaranda Language Specification, draft 0.3|http://www.eros-os.org/pipermail/e-lang/2008-August/012974.html]]// and twenty-six other [[things to read]]
* write letter to friam about provenance of the git ~Cryptographic-Hash-Function-Directed-Acyclic-Graph (~CHF-DAG) idea, starting with Graydon Hoare's recollections posted to uvc-reviewers
* reduce the number of incomplete drafts of e-mails in my Drafts folder -- currently twenty-six (hm, twenty-six Things To Read, twenty-six unfinished drafts of Things For Other People To Read...)
* catch up on reading other people's blogs
* talk about [[darcsver|http://allmydata.org/trac/darcsver]] with Brian

(Copied from [[2009-02-07]]:)
* //[[tahoe #608|http://allmydata.org/trac/tahoe/ticket/608]]: premature abort of upload if some shares were already present and some servers fail// (pushed out of the 1.3.0 milestone)
* write back on [[this thread on tahoe-dev|http://allmydata.org/pipermail/tahoe-dev/2009-January/001056.html]] and say "No, no, it really is almost as easy as Shawn originally thought.  Go, Shawn, go."
* figure out why setuptools/zetuptoolz is rebuilding things when {{{python ./setup.py test}}} after it just built them when {{{python ./setup.py build}}} //[[tahoe #591|http://allmydata.org/trac/tahoe/ticket/591]]: "make quicktest" could be quicker and less noisy// (added to [[issue tickets]])
* pycryptopp improvements -- link against system libcryptopp.so for Debian and Fedora packagers (//[[pycryptopp #9|http://allmydata.org/trac/pycryptopp/ticket/9]]: link against existing (system) libcrypto++.so//), add new improved ECDSA (//[[pycryptopp #13|http://allmydata.org/trac/pycryptopp/ticket/13]]: DSA "semi-private"/intermediate keys//, //[[pycryptopp #2|http://allmydata.org/trac/pycryptopp/ticket/2]]: deterministic generation of private key from small seed//, //[[pycryptopp #3|http://allmydata.org/trac/pycryptopp/ticket/3]]: serialize ecdsa keys without the fluff//, //[[pycryptopp #11|http://allmydata.org/trac/pycryptopp/ticket/11]]: submit patch for Brainpool ECC to Crypto++//), build out more pycryptopp buildbots, etc., (all added to [[issue tickets]])
* accounting/garbage-collection/quotas/etc.
* write up all those documents described in [[2009-02-06]]
* write up my new idea for immutable crypto caps
* start backing up all my personal files with tahoe
* one zillion other things

Here are some things that I hope I'll have time to read someday.  I'm posting them here so that (a) I can easily find them again or be reminded of their existence, and (b) people who've already read them or who otherwise have useful things to share with me will know that I'm interested in them.

[[computer security and reliability]]:
* ~David-Sarah Hopwood: [[Jacaranda Language Specification, draft 0.3|http://www.eros-os.org/pipermail/e-lang/2008-August/012974.html]] (also on the topic of [[programming languages]])
* Ken Thompson: [[Reflections on Trusting Trust|http://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf]]
* Helen J. Wang, Xiaofeng Fan, Jon Howell, Collin Jackson: [[Protection and Communication Abstractions for Web Browsers in MashupOS|http://research.microsoft.com/en-us/um/people/helenw/papers/sosp07mashupos.pdf]] (also on the topic of [[operating systems]])
* Tyler Close: [[ACLs don't|http://waterken.sourceforge.net/aclsdont]] and [[the ensuing conversation on cap-talk|http://www.eros-os.org/pipermail/cap-talk/2009-January/012030.html]]
* [[Martin C. Atkins|http://www.mca-ltd.com/martin/]]: [[An Introduction to Ten15 - A personal retrospective.|http://www.mca-ltd.com/martin/Ten15/introduction.html]]
* [[Carl Hewitt|http://carlhewitt.info]] ([[on wikipedia|http://en.wikipedia.org/wiki/Carl_Hewitt]]): [[A historical perspective on developing foundations for privacy-friendly client cloud computing: The Paradigm Shift from “Inconsistency Denial” to “Semantic Integration”|http://perspective.carlhewitt.info]]
* Ben Laurie, Abe Singer: [[Choose the Red Pill and the Blue Pill|http://www.links.org/files/nspw36.pdf]] (as [[recommended by Wes Felter|http://wmf.editthispage.com/2008/12/08]])
* Ben Laurie, Eric Sachs: [[Usability of Stronger Authentication Options|http://sites.google.com/site/oauthgoog/UXFedLogin/strongauth]] (as [[recommended by Wes Felter|http://wmf.editthispage.com/2008/12/03]])
* google: [[Browser Security Handbook|http://code.google.com/p/browsersec/wiki/Main]]
[[cryptography]] and [[secure hash functions]] (which means also [[computer security]]):
* Phil Rogaway: [[Formalizing Human Ignorance|http://www.cs.ucdavis.edu/~rogaway/papers/ignorance.html]] ([[on citeseerx|http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.65.291]], [[on eprint.iacr.org|http://eprint.iacr.org/2006/281]])
* Whitfield Diffie, Martin Hellman: [[New Directions in Cryptography|http://groups.csail.mit.edu/cis/crypto/classes/6.857/papers/diffie-hellman.pdf]], 1976
* Kevin D. Bowers, Ari Juels, Alina Oprea: [[HAIL: A High-Availability and Integrity Layer for Cloud Storage|http://eprint.iacr.org/2008/489]]
* Gaetan Leurent, Phong Q. Nguyen: [[How Risky is the Random-Oracle Model?|http://eprint.iacr.org/2008/441]]
* Michal Rjaško: [[Properties of Cryptographic Hash Functions|http://eprint.iacr.org/2008/527]]
* Abhishek Parakh, Subhash Kak: [[A Recursive Threshold Visual Cryptography Scheme|http://eprint.iacr.org/2008/535]]; (A good feature of papers about visual cryptography is that they always come with pictures!)
* Yehuda Lindell: [[Adaptively Secure Two-Party Computation with Erasures|http://eprint.iacr.org/2009/031]]
[[programming languages]]:
* Nate Foster, Benjamin C. Pierce, Michael Greenberg, et al.: [[Harmony/Boomerang|http://www.seas.upenn.edu/~harmony]], A bidirectional programming language for ad-hoc, textual data
[[computer performance]]:
* Daniel J. Bernstein, Adam Langley: [[Crit-bit Trees|http://www.imperialviolet.org/binary/critbit.pdf]]
* Bryan Cantrill, Jeff Bonwick: [[Real-World CONCURRENCY|http://mags.acm.org/queue/200809/?folio=16&CFID=19616687&CFTOKEN=90220359]] //(thanks to Kragen Sitaker for bringing it to my attention)//
* IEEE Spectrum: [[A Fairer, Faster Internet Protocol|http://spectrum.ieee.org/dec08/7027]] (as [[recommended by Wes Felter|http://wmf.editthispage.com/2008/12/04]])
* Greenan, Long, Miller, Schwarz, Wylie: [[A Spin-Up Saved is Energy Earned: Achieving Power-Efficient, Erasure-Coded Storage|http://www.usenix.org/events/hotdep08/tech/full_papers/greenan/greenan_html]]  (as [[recommended by Wes Felter|http://wmf.editthispage.com/2008/12/10]])
* Andreas Merkel, Frank Bellosa: [[Memory-aware Scheduling for Energy Efficiency on Multicore Processors|http://www.usenix.org/events/hotpower08/tech/full_papers/merkel/merkel_html]] (as [[recommended by Wes Felter|http://wmf.editthispage.com/2008/12/10]])
[[science]], [[health]], and [[politics and economics]]:
* Stanford Encyclopedia of Philosophy: [[Bayes' Theorem|http://plato.stanford.edu/entries/bayes-theorem]]
* George Bray: [[Obesity: a failure of homeostasis because of hedonic rewards: response to the letter from Gary Taubes|http://testgrid.allmydata.com:3567/file/URI:CHK:q3cwozbit4be2jxmxfre65wzdq:h6rm26yuvcjttexlwhdi6lfcgqoxkordv52xxkwyhxphkytcfuva:3:10:116345/@@named=/Bray_Taubes_rebuttal_rebuttal_rebuttal.pdf]] //(thanks to Gary Taubes for bringing it to my attention as I requested in [[rebuttals to "Good Calories, Bad Calories"]])//
* Leamer: [[on housing and the business cycle, posted in 1997|http://www.anderson.ucla.edu/Documents/areas/adm/media/leamer_housing_business_cycle.pdf]] (as [[recommended by Russ Roberts|http://cafehayek.typepad.com/hayek/2008/11/housing-and-the.html]])

See also [[things I have read]].

For my morning's crypto education I learned a little bit more about [[the LANE hash function|http://www.cosic.esat.kuleuven.be/publications/article-1182.pdf]].  The message expansion part of LANE can be seen as an error-detection code.  This is interesting to me, and it reminds me that the notions of collision-resistance and (especially) pre-image resistance ought, perhaps, to be clarified to include targets.

Informally, you care about the notion of pre-image resistance because you want to be able to give someone //H(x)// without revealing to them information about //x//.  But cryptographers seem to say that an attacker wins if he is able to learn the pre-image //y// about ''any'' image //H(y)// (excepting those that he already knew before he started -- i.e. excluding the trivial move of calculating //H(y)// from //y// and then saying you know the pre-image of //H(y)//).  This "general pre-image-resistance" notion isn't necessarily a stronger notion than the more natural "targetted-image pre-image-resistance" notion, although it might seem like it is at first.

That is: there could be some hash function for which an attacker can't come up with a pre-image for some arbitrarily chosen image, but which he //can// come up with a pre-image for your image.  :-)  (Because, the images you care about may be a special, non-random, non-uniformly distributed subset of all images, and because by telling the attacker an image, you are giving him information.)

It seems unlikely that any real hash function would be like this -- but the message-expansion step in LANE makes me wonder.

It seems to me, intuitively, that using an error-detecting code for the message-expansion step of LANE strengthens LANE against the attacker who wants to find any pre-image that he can, but weakens it against the attacker who wants to find //your pre-image// of some image that you told him.  This is because the message expansion code is adding redundancy but not information, and it is therefore //reducing the fraction of internal states which have pre-images at all//.  So if you are a cryptographer trying to make a name for yourself by coming up with a pre-image attack on LANE, and you've laboriously (with the help of complex algorithms and high-powered computers) worked out some prospective images which are reached from some internal states, then you might be disappointed to find that none of those internal states are valid according to the error-detection code, so there are no pre-images that lead to those internal states.  On the other hand, if you are an attacker who has been given an actual image by someone who hopes that you won't learn their pre-image, then the error-detection code ought to help you, by letting you cheaply determine whether a given internal state is right.  The error-detection property of the message expansion can't actually //hinder// your attempt to find the user's original pre-image, because by definition that pre-image is a valid input.  It could hinder your attempt to find a //different// pre-image that maps to the user's image.

A related puzzler is this: suppose you have an important secret //y//, and you compute //h = H(y)// and tell //h// to someone else.  Now suppose that other person is able to find a different pre-image //y′//, unrelated to your //y//, such that //h = H(y′)//.  Good for him!  He has violated the pre-image resistance property of //H//.  But he hasn't violated your confidentiality.  Hm.  Isn't your confidentiality the actual motivation for caring about pre-image resistance in the first place?

I know that cryptographers have formalized a notion of "target collision resistance", a.k.a. "universal one-way hash functions".  Have they done the same for "target pre-image resistance"?  Oh, maybe [[Herding Hash Functions and the Nostradamus Attack|http://www.cs.washington.edu/homes/yoshi/papers/EC06/herding.pdf]] by Kelsey and Kohno is relevant.  (See also [[the amusing demonstration|http://www.win.tue.nl/hashclash/Nostradamus]] in which some Dutch mathematicians used a ~PlayStation 3 to correctly predict the outcome of the 2008 USA Presidential Election.)

Thanks to Ghoti from IRC for challenging me to think more carefully about LANE.  Thanks to Ruptor from IRC for teaching me cryptography in the mornings.

Post script: ooh, this paper looks perfect for me: [[A Critical Look at Cryptographic Hash Function Literature|http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.83.7429]] by Scott Contini, Ron Steinfeld, Josef Pieprzyk, and Krystian Matusiewicz.  However, reading it will have to wait until I've finished the Repairer for ~Tahoe-LAFS!

(Also in this mornings crypto exploration: [[performance of the crypto hardware in the Ultrasparc T2|http://blogs.sun.com/sprack/entry/ultrasparc_t2_crypto_performance]].)

{{{
From: zooko
Date: December 22, 2008 15:02:40 PM MST
To: Multiple recipients of list
Subject: will SHA-3 replace the current standard secure hash algorithm -- MD5?
}}}

Folks:

Below, I re-post [[a letter|SHA-256 is too slow]] that I wrote to this list last February.  I think that this letter, which some of you may not have seen, casts light on why we have different assumptions about valid security/performance trade-offs.  It is because secure hashes are used today for more than just their original purpose.

Since I wrote this letter, I did some more snooping around, and I learned that the situation is even more extreme than I thought -- for some areas of endeavour, ~MD5 is actually the standard secure hash algorithm in 2008.

I chatted with a couple of friends who are information security consultants -- they get paid big bucks by household-name corporations to audit source code and systems for security flaws.  I asked them what kinds of secure hash functions they see used in the wild.  They answered that ~MD5 was the most common, occasionally ~SHA-1, in large part because it is a default value on the Java Cryptography Extensions, and they have never seen any other secure hash functions in client systems.

I chatted with a friend who works at the Internet Archive -- all files stored at the Internet Archive are identified by their ~MD5 hashes.

I noticed that there was a new release of the Haskell compiler GHC.  One of the new features is that it uses ~MD5 to identify code modules.

I learned more about the "computer forensics" field.  ~MD5 appears to be the standard mechanism to identify files in that field.  I read discussion forums in which computer forensics practitioners asked each other whether the cryptographic attacks on ~MD5 that they had heard about meant that they needed to change their practice.  The consensus seemed to be that they could continue using ~MD5 for now.

Finally, I was intrigued to see that NIST, of all organizations, uses and recommends the use of ~MD5 (in addition to ~SHA-1), as part of its "National Software Reference Library", which supports digital forensics.  This document explaining why NIST believes that this is safe is fascinating:

http://www.nsrl.nist.gov/Documents/analysis/draft-060530.pdf


The wide gap between the performance needs of using a secure hash function for public key cryptography versus using it for bulk data identification and integrity checking (which is what I use it for at my day job), make me wonder if ~SHA-3 should include variants or officially recommended tuning parameters so that people identifying large files can use a ~SHA-3 which is at least as fast as ~MD5 or Tiger, while people who are signing thousand-year documents can use a ~SHA-3 which is more expensive but safer.  (By the way, I tend to think that HMAC shouldn't be weighted heavily as a use case for ~SHA-3 simply because people should stop using HMAC and start using ~Carter-Wegman ~MACs instead such as Poly1305 or VMAC.)


Regards,

Zooko O'Whielacronx