tahoe-lafs/trac
2
0
Issues 1.1k Wiki Activity

Too many open files #2342

New issue
Open
opened 2014-11-27 16:11:00 +00:00 by zooko · 5 comments
zooko commented 2014-11-27 16:11:00 +00:00
Copy link

I'm working with a client company that says they have hundreds of thousands of people inconvenienced by their Tahoe-LAFS installation failing. Initial investigation reveals this in the twistd.log of their gateway node:

2014-11-20 13:33:36+0530 [HTTPChannel,143180,192.168.51.230] Unhandled Error
        Traceback (most recent call last):
          File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/python/log.py", line 88, in cal
lWithLogger
            
          File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/python/log.py", line 73, in cal
lWithContext
            
          File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/python/context.py", line 118, i
n callWithContext
            
          File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/python/context.py", line 81, in
 callWithContext
            
        --- <exception caught here> ---
          File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/internet/posixbase.py", line 61
4, in _doReadOrWrite
            
          File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/internet/tcp.py", line 215, in 
doRead
            
          File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/internet/tcp.py", line 221, in 
_dataReceived
            
          File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/protocols/basic.py", line 581, 
in dataReceived
            
          File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/web/http.py", line 1609, in lin
eReceived
            
HeadersReceived

          File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/web/http.py", line 694, in gotLength

          File "/usr/lib64/python2.6/tempfile.py", line 475, in TemporaryFile

          File "/usr/lib64/python2.6/tempfile.py", line 228, in _mkstemp_inner

        exceptions.OSError: [Errno 24] Too many open files: '/GFS2_tahoe/.tahoe-filestore1/tmp/tmpmKDgf6'
I'm working with a client company that says they have hundreds of thousands of people inconvenienced by their Tahoe-LAFS installation failing. Initial investigation reveals this in the twistd.log of their gateway node: ``` 2014-11-20 13:33:36+0530 [HTTPChannel,143180,192.168.51.230] Unhandled Error Traceback (most recent call last): File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/python/log.py", line 88, in cal lWithLogger File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/python/log.py", line 73, in cal lWithContext File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/python/context.py", line 118, i n callWithContext File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/python/context.py", line 81, in callWithContext --- <exception caught here> --- File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/internet/posixbase.py", line 61 4, in _doReadOrWrite File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/internet/tcp.py", line 215, in doRead File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/internet/tcp.py", line 221, in _dataReceived File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/protocols/basic.py", line 581, in dataReceived File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/web/http.py", line 1609, in lin eReceived HeadersReceived File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/web/http.py", line 694, in gotLength File "/usr/lib64/python2.6/tempfile.py", line 475, in TemporaryFile File "/usr/lib64/python2.6/tempfile.py", line 228, in _mkstemp_inner exceptions.OSError: [Errno 24] Too many open files: '/GFS2_tahoe/.tahoe-filestore1/tmp/tmpmKDgf6' ```
zooko added the
c/code
p/normal
t/defect
v/1.10.0
 labels 2014-11-27 16:11:00 +00:00
zooko added this to the undecided milestone 2014-11-27 16:11:00 +00:00
zooko commented 2014-11-27 16:53:59 +00:00
Author
Copy link

Here's a branch to tweak the increase_rlimits() code and to printout what it does for diagnostics: https://github.com/zooko/tahoe-lafs/blob/2342-Too-many-open-files/src/allmydata/util/iputil.py

Here's a branch to tweak the increase_rlimits() code and to printout what it does for diagnostics: <https://github.com/zooko/tahoe-lafs/blob/2342-Too-many-open-files/src/allmydata/util/iputil.py>
zooko commented 2014-11-28 17:39:53 +00:00
Author
Copy link

My consulting client (codenamed "WAG") reported this result on their RHEL server:

# python iputil.py v110
HELLO WORLD 000
HELLO WORLD 011
012 getrlimit(RLIMIT_NOFILE) before setrlimit: (1024, 4096)

Also this tweep ran it on CentOS and reported that with the patch (https://github.com/zooko/tahoe-lafs/blob/2342-Too-many-open-files/src/allmydata/util/iputil.py) the limit was raised to 4096:

https://twitter.com/brouhaha/status/538085487622107136

CentOS 6.5: before setrlimit: (1024, 4096) ... after setrlimit(4096, 4096): (4096, 4096)
My consulting client (codenamed "WAG") reported this result on their RHEL server: ``` # python iputil.py v110 HELLO WORLD 000 HELLO WORLD 011 012 getrlimit(RLIMIT_NOFILE) before setrlimit: (1024, 4096) ``` Also this tweep ran it on CentOS and reported that with the patch (<https://github.com/zooko/tahoe-lafs/blob/2342-Too-many-open-files/src/allmydata/util/iputil.py>) the limit was raised to 4096: <https://twitter.com/brouhaha/status/538085487622107136> ``` CentOS 6.5: before setrlimit: (1024, 4096) ... after setrlimit(4096, 4096): (4096, 4096) ```
zooko commented 2014-11-28 17:48:30 +00:00
Author
Copy link

See also #1794, #812, and #1278. I currently believe the underlying problem has to do with bad handling of corrupted shares (per comment:374048).

See also #1794, #812, and #1278. I currently believe the underlying problem has to do with bad handling of corrupted shares (per [comment:374048](/tahoe-lafs/trac/issues/812#issuecomment-374048)).
warner commented 2014-11-29 01:55:06 +00:00
Copy link

Seems like a decent theory.. you might add a timed loop that counts/logs the number of allmydata.immutable.upload.Uploader instances, and/or upload.FileHandle+subclasses (specifically FileName). If an upload gets wedged and stops making any progress, it will hold a filehandle open forever, and eventually an open() will fail like that.

You might also lsof the client in question and see what filehandles it has open: if it's this problem, there'll be a lot of /tmp files in the list, recently opened by HTTP uploads but not yet pushed out to the grid.

Seems like a decent theory.. you might add a timed loop that counts/logs the number of `allmydata.immutable.upload.Uploader` instances, and/or `upload.FileHandle`+subclasses (specifically `FileName`). If an upload gets wedged and stops making any progress, it will hold a filehandle open forever, and eventually an `open()` will fail like that. You might also `lsof` the client in question and see what filehandles it has open: if it's this problem, there'll be a lot of `/tmp` files in the list, recently opened by HTTP uploads but not yet pushed out to the grid.
warner commented 2014-11-29 17:20:56 +00:00
Copy link

Also, it might be appropriate to add a failsafe timer to Uploaders, something that fires once every minute or five minutes, and checks to see if any progress has been made, and self-destructs if not. We don't like heuristics, but sometimes they're a good hedge against weird unpredictable things happening.

Also, it might be appropriate to add a failsafe timer to `Uploader`s, something that fires once every minute or five minutes, and checks to see if any progress has been made, and self-destructs if not. We don't like heuristics, but sometimes they're a good hedge against weird unpredictable things happening.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
No results found.
No results found.
Labels
Clear labels   
c/code

catch-all for anything that can be fixed by writing code but isn't classified better

  
c/code-dirnodes

anything involving tahoe directories

  
c/code-encoding

immutable-file encoding format/layout, cap format, encoding/decoding algorithms

  
c/code-frontend

general frontend things: new user-facing interfaces, FTP/SFTP servers, FUSE-like interfaces

  
c/code-frontend-cli

CLI tools

  
c/code-frontend-ftp-sftp

   
c/code-frontend-magic-folder

   
c/code-frontend-web

webapi interfaces, user-facing "WUI" interfaces

  
c/code-mutable

anything involving mutable files: encoding/decoding, share format

  
c/code-network

Introducer, locating storage servers, establishing/maintaining connections to them

  
c/code-nodeadmin

node startup/shutdown, tools to create nodes, logging tools, management/monitoring tools, manhole, disk-space-limiting tools

  
c/code-peerselection

server selection/sorting algorithm, rebalancing code, allocation policy, swarming-download

  
c/code-storage

storage-server implementation, share-storage directory layout, client-to-server share transfer protocol

  
c/contrib

problems with non-core things that live in the source:contrib/ directory

  
c/dev-infrastructure

things that help us build Tahoe: buildbot, automated speed/memory/code-coverage tests and results-displayers, IRC channels, mailing lists, this Trac instance

  
c/docs

docs items. Also use 'docs' keyword on tickets in more specific components.

  
c/operational

things involved in running a tahoe grid, external tools to monitor/maintain a grid (like "nodeadmin" but larger-scale). Was also used for issues with the allmydata.com grid.

  
c/packaging

setup.py script, library dependencies, binary packages, licensing, installation instructions

  
c/unknown

Uncategorized. All tickets default to this component until someone updates it.

  
c/website

things about the tahoe-lafs.org website (and sometimes this Trac instance, shared with 'dev-infrastructure')

  
kw:2pc

   
kw:410

   
kw:9p

   
kw:ActivePerl

   
kw:AttributeError

   
kw:DataUnavailable

   
kw:DeadReferenceError

   
kw:DoS

   
kw:FileZilla

   
kw:GetLastError

   
kw:IFinishableConsumer

   
kw:K

   
kw:LeastAuthority

   
kw:Makefile

   
kw:RIStorageServer

   
kw:StringIO

   
kw:UncoordinatedWriteError

   
kw:about

   
kw:access

   
kw:access-control

   
kw:accessibility

   
kw:accounting

   
kw:accounting-crawler

   
kw:add-only

   
kw:aes

   
kw:aesthetics

   
kw:alias

   
kw:aliases

   
kw:aliens

   
kw:allmydata

   
kw:amazon

   
kw:ambient

   
kw:annotations

   
kw:anonymity

   
kw:anonymous

   
kw:anti-censorship

   
kw:api_auth_token

   
kw:appearance

   
kw:appname

   
kw:apport

   
kw:archive

   
kw:archlinux

   
kw:argparse

   
kw:arm

   
kw:assertion

   
kw:attachment

   
kw:auth

   
kw:authentication

   
kw:automation

   
kw:avahi

   
kw:availability

   
kw:aws

   
kw:azure

   
kw:backend

   
kw:backoff

   
kw:backup

   
kw:backupdb

   
kw:backward-compatibility

   
kw:bandwidth

   
kw:basedir

   
kw:bayes

   
kw:bbfreeze

   
kw:beta

   
kw:binaries

   
kw:binutils

   
kw:bitcoin

   
kw:bitrot

   
kw:blacklist

   
kw:blocker

   
kw:blocks-cloud-deployment

   
kw:blocks-cloud-merge

   
kw:blocks-magic-folder-merge

   
kw:blocks-merge

   
kw:blocks-raic

   
kw:blocks-release

   
kw:blog

   
kw:bom

   
kw:bonjour

   
kw:branch

   
kw:branding

   
kw:breadcrumbs

   
kw:brians-opinion-needed

   
kw:browser

   
kw:bsd

   
kw:build

   
kw:build-helpers

   
kw:buildbot

   
kw:builders

   
kw:buildslave

   
kw:buildslaves

   
kw:cache

   
kw:cap

   
kw:capleak

   
kw:captcha

   
kw:cast

   
kw:centos

   
kw:cffi

   
kw:chacha

   
kw:charset

   
kw:check

   
kw:checker

   
kw:chroot

   
kw:ci

   
kw:clean

   
kw:cleanup

   
kw:cli

   
kw:cloud

   
kw:cloud-backend

   
kw:cmdline

   
kw:code

   
kw:code-checks

   
kw:coding-standards

   
kw:coding-tools

   
kw:coding_tools

   
kw:collection

   
kw:compatibility

   
kw:completion

   
kw:compression

   
kw:confidentiality

   
kw:config

   
kw:configuration

   
kw:configuration.txt

   
kw:conflict

   
kw:connection

   
kw:connectivity

   
kw:consistency

   
kw:content

   
kw:control

   
kw:control.furl

   
kw:convergence

   
kw:coordination

   
kw:copyright

   
kw:corruption

   
kw:cors

   
kw:cost

   
kw:coverage

   
kw:coveralls

   
kw:coveralls.io

   
kw:cpu-watcher

   
kw:cpyext

   
kw:crash

   
kw:crawler

   
kw:crawlers

   
kw:create-container

   
kw:cruft

   
kw:crypto

   
kw:cryptography

   
kw:cryptography-lib

   
kw:cryptopp

   
kw:csp

   
kw:curl

   
kw:cutoff-date

   
kw:cycle

   
kw:cygwin

   
kw:d3

   
kw:daemon

   
kw:darcs

   
kw:darcsver

   
kw:database

   
kw:dataloss

   
kw:db

   
kw:dead-code

   
kw:deb

   
kw:debian

   
kw:debug

   
kw:deep-check

   
kw:defaults

   
kw:deferred

   
kw:delete

   
kw:deletion

   
kw:denial-of-service

   
kw:dependency

   
kw:deployment

   
kw:deprecation

   
kw:desert-island

   
kw:desert-island-build

   
kw:design

   
kw:design-review-needed

   
kw:detection

   
kw:dev-infrastructure

   
kw:devpay

   
kw:directory

   
kw:directory-page

   
kw:dirnode

   
kw:dirnodes

   
kw:disconnect

   
kw:discovery

   
kw:disk

   
kw:disk-backend

   
kw:distribute

   
kw:distutils

   
kw:dns

   
kw:do_http

   
kw:doc-needed

   
kw:docker

   
kw:docs

   
kw:docs-needed

   
kw:dokan

   
kw:dos

   
kw:download

   
kw:downloader

   
kw:dragonfly

   
kw:drop-upload

   
kw:duplicity

   
kw:dusty

   
kw:earth-dragon

   
kw:easy

   
kw:ec2

   
kw:ecdsa

   
kw:ed25519

   
kw:egg-needed

   
kw:eggs

   
kw:eliot

   
kw:email

   
kw:empty

   
kw:encoding

   
kw:endpoint

   
kw:enterprise

   
kw:enum34

   
kw:environment

   
kw:erasure

   
kw:erasure-coding

   
kw:error

   
kw:escaping

   
kw:etag

   
kw:etch

   
kw:evangelism

   
kw:eventual

   
kw:example

   
kw:excess-authority

   
kw:exec

   
kw:exocet

   
kw:expiration

   
kw:extensibility

   
kw:extension

   
kw:failure

   
kw:fedora

   
kw:ffp

   
kw:fhs

   
kw:figleaf

   
kw:file

   
kw:file-descriptor

   
kw:filename

   
kw:filesystem

   
kw:fileutil

   
kw:fips

   
kw:firewall

   
kw:first

   
kw:floatingpoint

   
kw:flog

   
kw:foolscap

   
kw:forward-compatibility

   
kw:forward-secrecy

   
kw:forwarding

   
kw:free

   
kw:freebsd

   
kw:frontend

   
kw:fsevents

   
kw:ftp

   
kw:ftpd

   
kw:full

   
kw:furl

   
kw:fuse

   
kw:garbage

   
kw:garbage-collection

   
kw:gateway

   
kw:gatherer

   
kw:gc

   
kw:gcc

   
kw:gentoo

   
kw:get

   
kw:git

   
kw:git-annex

   
kw:github

   
kw:glacier

   
kw:globalcaps

   
kw:glossary

   
kw:google-cloud-storage

   
kw:google-drive-backend

   
kw:gossip

   
kw:governance

   
kw:grid

   
kw:grid-manager

   
kw:gridid

   
kw:gridsync

   
kw:grsec

   
kw:gsoc

   
kw:gvfs

   
kw:hackfest

   
kw:hacktahoe

   
kw:hang

   
kw:hardlink

   
kw:heartbleed

   
kw:heisenbug

   
kw:help

   
kw:helper

   
kw:hint

   
kw:hooks

   
kw:how

   
kw:how-to

   
kw:howto

   
kw:hp

   
kw:hp-cloud

   
kw:html

   
kw:http

   
kw:https

   
kw:i18n

   
kw:i2p

   
kw:i2p-collab

   
kw:illustration

   
kw:image

   
kw:immutable

   
kw:impressions

   
kw:incentives

   
kw:incident

   
kw:init

   
kw:inlineCallbacks

   
kw:inotify

   
kw:install

   
kw:installer

   
kw:integration

   
kw:integration-test

   
kw:integrity

   
kw:interactive

   
kw:interface

   
kw:interfaces

   
kw:interoperability

   
kw:interstellar-exploration

   
kw:introducer

   
kw:introduction

   
kw:iphone

   
kw:ipkg

   
kw:iputil

   
kw:ipv6

   
kw:irc

   
kw:jail

   
kw:javascript

   
kw:joke

   
kw:jquery

   
kw:json

   
kw:jsui

   
kw:junk

   
kw:key-value-store

   
kw:kfreebsd

   
kw:known-issue

   
kw:konqueror

   
kw:kpreid

   
kw:kvm

   
kw:l10n

   
kw:lae

   
kw:large

   
kw:latency

   
kw:leak

   
kw:leasedb

   
kw:leases

   
kw:libgmp

   
kw:license

   
kw:licenss

   
kw:linecount

   
kw:link

   
kw:linux

   
kw:lit

   
kw:localhost

   
kw:location

   
kw:locking

   
kw:logging

   
kw:logo

   
kw:loopback

   
kw:lucid

   
kw:mac

   
kw:macintosh

   
kw:magic-folder

   
kw:manhole

   
kw:manifest

   
kw:manual-test-needed

   
kw:map

   
kw:mapupdate

   
kw:max_space

   
kw:mdmf

   
kw:memcheck

   
kw:memory

   
kw:memory-leak

   
kw:mesh

   
kw:metadata

   
kw:meter

   
kw:migration

   
kw:mime

   
kw:mingw

   
kw:minimal

   
kw:misc

   
kw:miscapture

   
kw:mlp

   
kw:mock

   
kw:more-info-needed

   
kw:mountain-lion

   
kw:move

   
kw:multi-users

   
kw:multiple

   
kw:multiuser-gateway

   
kw:munin

   
kw:music

   
kw:mutability

   
kw:mutable

   
kw:mystery

   
kw:names

   
kw:naming

   
kw:nas

   
kw:navigation

   
kw:needs-review

   
kw:needs-spawn

   
kw:netbsd

   
kw:network

   
kw:nevow

   
kw:new-user

   
kw:newcaps

   
kw:news

   
kw:news-done

   
kw:news-needed

   
kw:newsletter

   
kw:newurls

   
kw:nfc

   
kw:nginx

   
kw:nixos

   
kw:no-clobber

   
kw:node

   
kw:node-url

   
kw:notification

   
kw:notifyOnDisconnect

   
kw:nsa310

   
kw:nsa320

   
kw:nsa325

   
kw:numpy

   
kw:objects

   
kw:old

   
kw:openbsd

   
kw:openitp-packaging

   
kw:openssl

   
kw:openstack

   
kw:opensuse

   
kw:operation-helpers

   
kw:operational

   
kw:operations

   
kw:ophandle

   
kw:ophandles

   
kw:ops

   
kw:optimization

   
kw:optional

   
kw:options

   
kw:organization

   
kw:os

   
kw:os.abort

   
kw:ostrom

   
kw:osx

   
kw:osxfuse

   
kw:otf-magic-folder-objective1

   
kw:otf-magic-folder-objective2

   
kw:otf-magic-folder-objective3

   
kw:otf-magic-folder-objective4

   
kw:otf-magic-folder-objective5

   
kw:otf-magic-folder-objective6

   
kw:p2p

   
kw:packaging

   
kw:partial

   
kw:password

   
kw:path

   
kw:paths

   
kw:pause

   
kw:peer-selection

   
kw:performance

   
kw:permalink

   
kw:permissions

   
kw:persistence

   
kw:phone

   
kw:pickle

   
kw:pip

   
kw:pipermail

   
kw:pkg_resources

   
kw:placement

   
kw:planning

   
kw:policy

   
kw:port

   
kw:portability

   
kw:portal

   
kw:posthook

   
kw:pratchett

   
kw:preformance

   
kw:preservation

   
kw:privacy

   
kw:process

   
kw:profile

   
kw:profiling

   
kw:progress

   
kw:proxy

   
kw:publish

   
kw:pyOpenSSL

   
kw:pyasn1

   
kw:pycparser

   
kw:pycrypto

   
kw:pycrypto-lib

   
kw:pycryptopp

   
kw:pyfilesystem

   
kw:pyflakes

   
kw:pylint

   
kw:pypi

   
kw:pypy

   
kw:pysqlite

   
kw:python

   
kw:python3

   
kw:pythonpath

   
kw:pyutil

   
kw:pywin32

   
kw:quickstart

   
kw:quiet

   
kw:quotas

   
kw:quoting

   
kw:raic

   
kw:rainhill

   
kw:random

   
kw:random-access

   
kw:range

   
kw:raspberry-pi

   
kw:reactor

   
kw:readonly

   
kw:rebalancing

   
kw:recovery

   
kw:recursive

   
kw:redhat

   
kw:redirect

   
kw:redressing

   
kw:refactor

   
kw:referer

   
kw:referrer

   
kw:regression

   
kw:rekey

   
kw:relay

   
kw:release

   
kw:release-blocker

   
kw:reliability

   
kw:relnotes

   
kw:remote

   
kw:removable

   
kw:removable-disk

   
kw:rename

   
kw:renew

   
kw:repair

   
kw:replace

   
kw:report

   
kw:repository

   
kw:research

   
kw:reserved_space

   
kw:response-needed

   
kw:response-time

   
kw:restore

   
kw:retrieve

   
kw:retry

   
kw:review

   
kw:review-needed

   
kw:reviewed

   
kw:revocation

   
kw:roadmap

   
kw:rollback

   
kw:rpm

   
kw:rsa

   
kw:rss

   
kw:rst

   
kw:rsync

   
kw:rusty

   
kw:s3

   
kw:s3-backend

   
kw:s3-frontend

   
kw:s4

   
kw:same-origin

   
kw:sandbox

   
kw:scalability

   
kw:scaling

   
kw:scheduling

   
kw:schema

   
kw:scheme

   
kw:scp

   
kw:scripts

   
kw:sdist

   
kw:sdmf

   
kw:security

   
kw:self-contained

   
kw:server

   
kw:servermap

   
kw:servers-of-happiness

   
kw:service

   
kw:setup

   
kw:setup.py

   
kw:setup_requires

   
kw:setuptools

   
kw:setuptools_darcs

   
kw:sftp

   
kw:shared

   
kw:shareset

   
kw:shell

   
kw:signals

   
kw:simultaneous

   
kw:six

   
kw:size

   
kw:slackware

   
kw:slashes

   
kw:smb

   
kw:sneakernet

   
kw:snowleopard

   
kw:socket

   
kw:solaris

   
kw:space

   
kw:space-efficiency

   
kw:spam

   
kw:spec

   
kw:speed

   
kw:sqlite

   
kw:ssh

   
kw:ssh-keygen

   
kw:sshfs

   
kw:ssl

   
kw:stability

   
kw:standards

   
kw:start

   
kw:startup

   
kw:static

   
kw:static-analysis

   
kw:statistics

   
kw:stats

   
kw:stats_gatherer

   
kw:status

   
kw:stdeb

   
kw:storage

   
kw:streaming

   
kw:strports

   
kw:style

   
kw:stylesheet

   
kw:subprocess

   
kw:sumo

   
kw:survey

   
kw:svg

   
kw:symlink

   
kw:synchronous

   
kw:tac

   
kw:tahoe-*

   
kw:tahoe-add-alias

   
kw:tahoe-admin

   
kw:tahoe-archive

   
kw:tahoe-backup

   
kw:tahoe-check

   
kw:tahoe-cp

   
kw:tahoe-create-alias

   
kw:tahoe-create-introducer

   
kw:tahoe-debug

   
kw:tahoe-deep-check

   
kw:tahoe-deepcheck

   
kw:tahoe-lafs-trac-stream

   
kw:tahoe-list-aliases

   
kw:tahoe-ls

   
kw:tahoe-magic-folder

   
kw:tahoe-manifest

   
kw:tahoe-mkdir

   
kw:tahoe-mount

   
kw:tahoe-mv

   
kw:tahoe-put

   
kw:tahoe-restart

   
kw:tahoe-rm

   
kw:tahoe-run

   
kw:tahoe-start

   
kw:tahoe-stats

   
kw:tahoe-unlink

   
kw:tahoe-webopen

   
kw:tahoe.css

   
kw:tahoe_files

   
kw:tahoewapi

   
kw:tarball

   
kw:tarballs

   
kw:tempfile

   
kw:templates

   
kw:terminology

   
kw:test

   
kw:test-and-set

   
kw:test-from-egg

   
kw:test-needed

   
kw:testgrid

   
kw:testing

   
kw:tests

   
kw:throttling

   
kw:ticket999-s3-backend

   
kw:tiddly

   
kw:time

   
kw:timeout

   
kw:timing

   
kw:to

   
kw:to-be-closed-on-2011-08-01

   
kw:tor

   
kw:tor-protocol

   
kw:torsocks

   
kw:tox

   
kw:trac

   
kw:transparency

   
kw:travis

   
kw:travis-ci

   
kw:trial

   
kw:trickle

   
kw:trivial

   
kw:truckee

   
kw:tub

   
kw:tub.location

   
kw:twine

   
kw:twistd

   
kw:twistd.log

   
kw:twisted

   
kw:twisted-14

   
kw:twisted-trial

   
kw:twitter

   
kw:twn

   
kw:txaws

   
kw:type

   
kw:typeerror

   
kw:ubuntu

   
kw:ucwe

   
kw:ueb

   
kw:ui

   
kw:unclean

   
kw:uncoordinated-writes

   
kw:undeletable

   
kw:unfinished-business

   
kw:unhandled-error

   
kw:unhappy

   
kw:unicode

   
kw:unit

   
kw:unix

   
kw:unlink

   
kw:update

   
kw:upgrade

   
kw:upload

   
kw:upload-helper

   
kw:uri

   
kw:url

   
kw:usability

   
kw:use-case

   
kw:utf-8

   
kw:util

   
kw:uwsgi

   
kw:ux

   
kw:validation

   
kw:variables

   
kw:vdrive

   
kw:verify

   
kw:verlib

   
kw:version

   
kw:versioning

   
kw:versions

   
kw:video

   
kw:virtualbox

   
kw:virtualenv

   
kw:vista

   
kw:visualization

   
kw:visualizer

   
kw:vm

   
kw:volunteergrid2

   
kw:volunteers

   
kw:vpn

   
kw:wapi

   
kw:warners-opinion-needed

   
kw:warning

   
kw:weapi

   
kw:web

   
kw:web.port

   
kw:webapi

   
kw:webdav

   
kw:webdrive

   
kw:webport

   
kw:websec

   
kw:website

   
kw:websocket

   
kw:welcome

   
kw:welcome-page

   
kw:welcomepage

   
kw:wiki

   
kw:win32

   
kw:win64

   
kw:windows

   
kw:windows-related

   
kw:winscp

   
kw:workaround

   
kw:world-domination

   
kw:wrapper

   
kw:write-enabler

   
kw:wui

   
kw:x86

   
kw:x86-64

   
kw:xhtml

   
kw:xml

   
kw:xss

   
kw:zbase32

   
kw:zetuptoolz

   
kw:zfec

   
kw:zookos-opinion-needed

   
kw:zope

   
kw:zope.interface

   
p/blocker

   
p/critical

   
p/major

   
p/minor

   
p/normal

   
p/supercritical

   
p/trivial

   
r/cannot reproduce

   
r/duplicate

   
r/fixed

   
r/invalid

   
r/somebody else's problem

   
r/was already fixed

   
r/wontfix

   
r/worksforme

   
t/defect

   
t/enhancement

   
t/task

   
v/0.2.0

   
v/0.3.0

   
v/0.4.0

   
v/0.5.0

   
v/0.5.1

   
v/0.6.0

   
v/0.6.1

   
v/0.7.0

   
v/0.8.0

   
v/0.9.0

   
v/1.0.0

   
v/1.1.0

   
v/1.10.0

   
v/1.10.1

   
v/1.10.2

   
v/1.10a2

   
v/1.11.0

   
v/1.12.0

   
v/1.12.1

   
v/1.13.0

   
v/1.14.0

   
v/1.15.0

   
v/1.15.1

   
v/1.2.0

   
v/1.3.0

   
v/1.4.1

   
v/1.5.0

   
v/1.6.0

   
v/1.6.1

   
v/1.7.0

   
v/1.7.1

   
v/1.7β

   
v/1.8.0

   
v/1.8.1

   
v/1.8.2

   
v/1.8.3

   
v/1.8β

   
v/1.9.0

   
v/1.9.0-s3branch

   
v/1.9.0a1

   
v/1.9.0a2

   
v/1.9.0b1

   
v/1.9.1

   
v/1.9.2

   
v/1.9.2a1

   
v/cloud-branch

   
v/unknown
No labels
c/code
c/code-dirnodes
c/code-encoding
c/code-frontend
c/code-frontend-cli
c/code-frontend-ftp-sftp
c/code-frontend-magic-folder
c/code-frontend-web
c/code-mutable
c/code-network
c/code-nodeadmin
c/code-peerselection
c/code-storage
c/contrib
c/dev-infrastructure
c/docs
c/operational
c/packaging
c/unknown
c/website
kw:2pc
kw:410
kw:9p
kw:ActivePerl
kw:AttributeError
kw:DataUnavailable
kw:DeadReferenceError
kw:DoS
kw:FileZilla
kw:GetLastError
kw:IFinishableConsumer
kw:K
kw:LeastAuthority
kw:Makefile
kw:RIStorageServer
kw:StringIO
kw:UncoordinatedWriteError
kw:about
kw:access
kw:access-control
kw:accessibility
kw:accounting
kw:accounting-crawler
kw:add-only
kw:aes
kw:aesthetics
kw:alias
kw:aliases
kw:aliens
kw:allmydata
kw:amazon
kw:ambient
kw:annotations
kw:anonymity
kw:anonymous
kw:anti-censorship
kw:api_auth_token
kw:appearance
kw:appname
kw:apport
kw:archive
kw:archlinux
kw:argparse
kw:arm
kw:assertion
kw:attachment
kw:auth
kw:authentication
kw:automation
kw:avahi
kw:availability
kw:aws
kw:azure
kw:backend
kw:backoff
kw:backup
kw:backupdb
kw:backward-compatibility
kw:bandwidth
kw:basedir
kw:bayes
kw:bbfreeze
kw:beta
kw:binaries
kw:binutils
kw:bitcoin
kw:bitrot
kw:blacklist
kw:blocker
kw:blocks-cloud-deployment
kw:blocks-cloud-merge
kw:blocks-magic-folder-merge
kw:blocks-merge
kw:blocks-raic
kw:blocks-release
kw:blog
kw:bom
kw:bonjour
kw:branch
kw:branding
kw:breadcrumbs
kw:brians-opinion-needed
kw:browser
kw:bsd
kw:build
kw:build-helpers
kw:buildbot
kw:builders
kw:buildslave
kw:buildslaves
kw:cache
kw:cap
kw:capleak
kw:captcha
kw:cast
kw:centos
kw:cffi
kw:chacha
kw:charset
kw:check
kw:checker
kw:chroot
kw:ci
kw:clean
kw:cleanup
kw:cli
kw:cloud
kw:cloud-backend
kw:cmdline
kw:code
kw:code-checks
kw:coding-standards
kw:coding-tools
kw:coding_tools
kw:collection
kw:compatibility
kw:completion
kw:compression
kw:confidentiality
kw:config
kw:configuration
kw:configuration.txt
kw:conflict
kw:connection
kw:connectivity
kw:consistency
kw:content
kw:control
kw:control.furl
kw:convergence
kw:coordination
kw:copyright
kw:corruption
kw:cors
kw:cost
kw:coverage
kw:coveralls
kw:coveralls.io
kw:cpu-watcher
kw:cpyext
kw:crash
kw:crawler
kw:crawlers
kw:create-container
kw:cruft
kw:crypto
kw:cryptography
kw:cryptography-lib
kw:cryptopp
kw:csp
kw:curl
kw:cutoff-date
kw:cycle
kw:cygwin
kw:d3
kw:daemon
kw:darcs
kw:darcsver
kw:database
kw:dataloss
kw:db
kw:dead-code
kw:deb
kw:debian
kw:debug
kw:deep-check
kw:defaults
kw:deferred
kw:delete
kw:deletion
kw:denial-of-service
kw:dependency
kw:deployment
kw:deprecation
kw:desert-island
kw:desert-island-build
kw:design
kw:design-review-needed
kw:detection
kw:dev-infrastructure
kw:devpay
kw:directory
kw:directory-page
kw:dirnode
kw:dirnodes
kw:disconnect
kw:discovery
kw:disk
kw:disk-backend
kw:distribute
kw:distutils
kw:dns
kw:do_http
kw:doc-needed
kw:docker
kw:docs
kw:docs-needed
kw:dokan
kw:dos
kw:download
kw:downloader
kw:dragonfly
kw:drop-upload
kw:duplicity
kw:dusty
kw:earth-dragon
kw:easy
kw:ec2
kw:ecdsa
kw:ed25519
kw:egg-needed
kw:eggs
kw:eliot
kw:email
kw:empty
kw:encoding
kw:endpoint
kw:enterprise
kw:enum34
kw:environment
kw:erasure
kw:erasure-coding
kw:error
kw:escaping
kw:etag
kw:etch
kw:evangelism
kw:eventual
kw:example
kw:excess-authority
kw:exec
kw:exocet
kw:expiration
kw:extensibility
kw:extension
kw:failure
kw:fedora
kw:ffp
kw:fhs
kw:figleaf
kw:file
kw:file-descriptor
kw:filename
kw:filesystem
kw:fileutil
kw:fips
kw:firewall
kw:first
kw:floatingpoint
kw:flog
kw:foolscap
kw:forward-compatibility
kw:forward-secrecy
kw:forwarding
kw:free
kw:freebsd
kw:frontend
kw:fsevents
kw:ftp
kw:ftpd
kw:full
kw:furl
kw:fuse
kw:garbage
kw:garbage-collection
kw:gateway
kw:gatherer
kw:gc
kw:gcc
kw:gentoo
kw:get
kw:git
kw:git-annex
kw:github
kw:glacier
kw:globalcaps
kw:glossary
kw:google-cloud-storage
kw:google-drive-backend
kw:gossip
kw:governance
kw:grid
kw:grid-manager
kw:gridid
kw:gridsync
kw:grsec
kw:gsoc
kw:gvfs
kw:hackfest
kw:hacktahoe
kw:hang
kw:hardlink
kw:heartbleed
kw:heisenbug
kw:help
kw:helper
kw:hint
kw:hooks
kw:how
kw:how-to
kw:howto
kw:hp
kw:hp-cloud
kw:html
kw:http
kw:https
kw:i18n
kw:i2p
kw:i2p-collab
kw:illustration
kw:image
kw:immutable
kw:impressions
kw:incentives
kw:incident
kw:init
kw:inlineCallbacks
kw:inotify
kw:install
kw:installer
kw:integration
kw:integration-test
kw:integrity
kw:interactive
kw:interface
kw:interfaces
kw:interoperability
kw:interstellar-exploration
kw:introducer
kw:introduction
kw:iphone
kw:ipkg
kw:iputil
kw:ipv6
kw:irc
kw:jail
kw:javascript
kw:joke
kw:jquery
kw:json
kw:jsui
kw:junk
kw:key-value-store
kw:kfreebsd
kw:known-issue
kw:konqueror
kw:kpreid
kw:kvm
kw:l10n
kw:lae
kw:large
kw:latency
kw:leak
kw:leasedb
kw:leases
kw:libgmp
kw:license
kw:licenss
kw:linecount
kw:link
kw:linux
kw:lit
kw:localhost
kw:location
kw:locking
kw:logging
kw:logo
kw:loopback
kw:lucid
kw:mac
kw:macintosh
kw:magic-folder
kw:manhole
kw:manifest
kw:manual-test-needed
kw:map
kw:mapupdate
kw:max_space
kw:mdmf
kw:memcheck
kw:memory
kw:memory-leak
kw:mesh
kw:metadata
kw:meter
kw:migration
kw:mime
kw:mingw
kw:minimal
kw:misc
kw:miscapture
kw:mlp
kw:mock
kw:more-info-needed
kw:mountain-lion
kw:move
kw:multi-users
kw:multiple
kw:multiuser-gateway
kw:munin
kw:music
kw:mutability
kw:mutable
kw:mystery
kw:names
kw:naming
kw:nas
kw:navigation
kw:needs-review
kw:needs-spawn
kw:netbsd
kw:network
kw:nevow
kw:new-user
kw:newcaps
kw:news
kw:news-done
kw:news-needed
kw:newsletter
kw:newurls
kw:nfc
kw:nginx
kw:nixos
kw:no-clobber
kw:node
kw:node-url
kw:notification
kw:notifyOnDisconnect
kw:nsa310
kw:nsa320
kw:nsa325
kw:numpy
kw:objects
kw:old
kw:openbsd
kw:openitp-packaging
kw:openssl
kw:openstack
kw:opensuse
kw:operation-helpers
kw:operational
kw:operations
kw:ophandle
kw:ophandles
kw:ops
kw:optimization
kw:optional
kw:options
kw:organization
kw:os
kw:os.abort
kw:ostrom
kw:osx
kw:osxfuse
kw:otf-magic-folder-objective1
kw:otf-magic-folder-objective2
kw:otf-magic-folder-objective3
kw:otf-magic-folder-objective4
kw:otf-magic-folder-objective5
kw:otf-magic-folder-objective6
kw:p2p
kw:packaging
kw:partial
kw:password
kw:path
kw:paths
kw:pause
kw:peer-selection
kw:performance
kw:permalink
kw:permissions
kw:persistence
kw:phone
kw:pickle
kw:pip
kw:pipermail
kw:pkg_resources
kw:placement
kw:planning
kw:policy
kw:port
kw:portability
kw:portal
kw:posthook
kw:pratchett
kw:preformance
kw:preservation
kw:privacy
kw:process
kw:profile
kw:profiling
kw:progress
kw:proxy
kw:publish
kw:pyOpenSSL
kw:pyasn1
kw:pycparser
kw:pycrypto
kw:pycrypto-lib
kw:pycryptopp
kw:pyfilesystem
kw:pyflakes
kw:pylint
kw:pypi
kw:pypy
kw:pysqlite
kw:python
kw:python3
kw:pythonpath
kw:pyutil
kw:pywin32
kw:quickstart
kw:quiet
kw:quotas
kw:quoting
kw:raic
kw:rainhill
kw:random
kw:random-access
kw:range
kw:raspberry-pi
kw:reactor
kw:readonly
kw:rebalancing
kw:recovery
kw:recursive
kw:redhat
kw:redirect
kw:redressing
kw:refactor
kw:referer
kw:referrer
kw:regression
kw:rekey
kw:relay
kw:release
kw:release-blocker
kw:reliability
kw:relnotes
kw:remote
kw:removable
kw:removable-disk
kw:rename
kw:renew
kw:repair
kw:replace
kw:report
kw:repository
kw:research
kw:reserved_space
kw:response-needed
kw:response-time
kw:restore
kw:retrieve
kw:retry
kw:review
kw:review-needed
kw:reviewed
kw:revocation
kw:roadmap
kw:rollback
kw:rpm
kw:rsa
kw:rss
kw:rst
kw:rsync
kw:rusty
kw:s3
kw:s3-backend
kw:s3-frontend
kw:s4
kw:same-origin
kw:sandbox
kw:scalability
kw:scaling
kw:scheduling
kw:schema
kw:scheme
kw:scp
kw:scripts
kw:sdist
kw:sdmf
kw:security
kw:self-contained
kw:server
kw:servermap
kw:servers-of-happiness
kw:service
kw:setup
kw:setup.py
kw:setup_requires
kw:setuptools
kw:setuptools_darcs
kw:sftp
kw:shared
kw:shareset
kw:shell
kw:signals
kw:simultaneous
kw:six
kw:size
kw:slackware
kw:slashes
kw:smb
kw:sneakernet
kw:snowleopard
kw:socket
kw:solaris
kw:space
kw:space-efficiency
kw:spam
kw:spec
kw:speed
kw:sqlite
kw:ssh
kw:ssh-keygen
kw:sshfs
kw:ssl
kw:stability
kw:standards
kw:start
kw:startup
kw:static
kw:static-analysis
kw:statistics
kw:stats
kw:stats_gatherer
kw:status
kw:stdeb
kw:storage
kw:streaming
kw:strports
kw:style
kw:stylesheet
kw:subprocess
kw:sumo
kw:survey
kw:svg
kw:symlink
kw:synchronous
kw:tac
kw:tahoe-*
kw:tahoe-add-alias
kw:tahoe-admin
kw:tahoe-archive
kw:tahoe-backup
kw:tahoe-check
kw:tahoe-cp
kw:tahoe-create-alias
kw:tahoe-create-introducer
kw:tahoe-debug
kw:tahoe-deep-check
kw:tahoe-deepcheck
kw:tahoe-lafs-trac-stream
kw:tahoe-list-aliases
kw:tahoe-ls
kw:tahoe-magic-folder
kw:tahoe-manifest
kw:tahoe-mkdir
kw:tahoe-mount
kw:tahoe-mv
kw:tahoe-put
kw:tahoe-restart
kw:tahoe-rm
kw:tahoe-run
kw:tahoe-start
kw:tahoe-stats
kw:tahoe-unlink
kw:tahoe-webopen
kw:tahoe.css
kw:tahoe_files
kw:tahoewapi
kw:tarball
kw:tarballs
kw:tempfile
kw:templates
kw:terminology
kw:test
kw:test-and-set
kw:test-from-egg
kw:test-needed
kw:testgrid
kw:testing
kw:tests
kw:throttling
kw:ticket999-s3-backend
kw:tiddly
kw:time
kw:timeout
kw:timing
kw:to
kw:to-be-closed-on-2011-08-01
kw:tor
kw:tor-protocol
kw:torsocks
kw:tox
kw:trac
kw:transparency
kw:travis
kw:travis-ci
kw:trial
kw:trickle
kw:trivial
kw:truckee
kw:tub
kw:tub.location
kw:twine
kw:twistd
kw:twistd.log
kw:twisted
kw:twisted-14
kw:twisted-trial
kw:twitter
kw:twn
kw:txaws
kw:type
kw:typeerror
kw:ubuntu
kw:ucwe
kw:ueb
kw:ui
kw:unclean
kw:uncoordinated-writes
kw:undeletable
kw:unfinished-business
kw:unhandled-error
kw:unhappy
kw:unicode
kw:unit
kw:unix
kw:unlink
kw:update
kw:upgrade
kw:upload
kw:upload-helper
kw:uri
kw:url
kw:usability
kw:use-case
kw:utf-8
kw:util
kw:uwsgi
kw:ux
kw:validation
kw:variables
kw:vdrive
kw:verify
kw:verlib
kw:version
kw:versioning
kw:versions
kw:video
kw:virtualbox
kw:virtualenv
kw:vista
kw:visualization
kw:visualizer
kw:vm
kw:volunteergrid2
kw:volunteers
kw:vpn
kw:wapi
kw:warners-opinion-needed
kw:warning
kw:weapi
kw:web
kw:web.port
kw:webapi
kw:webdav
kw:webdrive
kw:webport
kw:websec
kw:website
kw:websocket
kw:welcome
kw:welcome-page
kw:welcomepage
kw:wiki
kw:win32
kw:win64
kw:windows
kw:windows-related
kw:winscp
kw:workaround
kw:world-domination
kw:wrapper
kw:write-enabler
kw:wui
kw:x86
kw:x86-64
kw:xhtml
kw:xml
kw:xss
kw:zbase32
kw:zetuptoolz
kw:zfec
kw:zookos-opinion-needed
kw:zope
kw:zope.interface
p/blocker
p/critical
p/major
p/minor
p/normal
p/supercritical
p/trivial
r/cannot reproduce
r/duplicate
r/fixed
r/invalid
r/somebody else's problem
r/was already fixed
r/wontfix
r/worksforme
t/defect
t/enhancement
t/task
v/0.2.0
v/0.3.0
v/0.4.0
v/0.5.0
v/0.5.1
v/0.6.0
v/0.6.1
v/0.7.0
v/0.8.0
v/0.9.0
v/1.0.0
v/1.1.0
v/1.10.0
v/1.10.1
v/1.10.2
v/1.10a2
v/1.11.0
v/1.12.0
v/1.12.1
v/1.13.0
v/1.14.0
v/1.15.0
v/1.15.1
v/1.2.0
v/1.3.0
v/1.4.1
v/1.5.0
v/1.6.0
v/1.6.1
v/1.7.0
v/1.7.1
v/1.7β
v/1.8.0
v/1.8.1
v/1.8.2
v/1.8.3
v/1.8β
v/1.9.0
v/1.9.0-s3branch
v/1.9.0a1
v/1.9.0a2
v/1.9.0b1
v/1.9.1
v/1.9.2
v/1.9.2a1
v/cloud-branch
v/unknown
Milestone
Clear milestone
No items
No milestone
undecided
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: tahoe-lafs/trac#2342
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?