tahoe-lafs/trac

Unicode stdout/stderr replacement on Windows fails to print large strings #1232

New issue

Closed

opened 2010-10-27 02:29:48 +00:00 by daira · 7 comments

daira commented

2010-10-27 02:29:48 +00:00

While trying to debug #1045, I tried to print a ResponseCache object and got the following exception:

  File "d:\tahoe\tahoe-1.8.0c2\src\allmydata\windows\fixups.py", line 134, in write
    raise IOError("WriteConsoleW returned %r, n.value = %r" % (retval, n.value))
exceptions.IOError: WriteConsoleW returned 0, n.value = 0L

This is a bug in the Unicode stdout/stderr replacement in source:src/allmydata/windows/fixups.py that was added in Tahoe-LAFS v1.8 beta. On my machine it fails when trying to print more than 26608 characters at once to stdout or stderr. This appears to be an undocumented limitation of the Windows WriteConsoleW API call.

Technically this is a regression since v1.7.1, although we very rarely print a string that large. The fix is trivial; just limit the length of data passed in a single call to WriteConsoleW.

While trying to debug #1045, I tried to print a `ResponseCache` object and got the following exception: ``` File "d:\tahoe\tahoe-1.8.0c2\src\allmydata\windows\fixups.py", line 134, in write raise IOError("WriteConsoleW returned %r, n.value = %r" % (retval, n.value)) exceptions.IOError: WriteConsoleW returned 0, n.value = 0L ``` This is a bug in the Unicode stdout/stderr replacement in source:src/allmydata/windows/fixups.py that was added in Tahoe-LAFS v1.8 beta. On my machine it fails when trying to print more than 26608 characters at once to stdout or stderr. This appears to be an undocumented limitation of the Windows `WriteConsoleW` API call. Technically this is a regression since v1.7.1, although we very rarely print a string that large. The fix is trivial; just limit the length of data passed in a single call to `WriteConsoleW`.

daira added the

labels 2010-10-27 02:29:48 +00:00

daira added this to the 1.8.1 milestone 2010-10-27 02:29:48 +00:00

daira commented

2010-10-27 02:36:12 +00:00

Author

Correction: the [MSDN documentation for WriteConsoleW](http://msdn.microsoft.com/en-us/library/ms687401%28VS.85%29.aspx) does document a length limitation, but it says 64 KiB (which would be 32767 UTF-16 characters and a terminating NUL), not 26608 characters. The fix is the same regardless; I'll limit it to 10000 UTF-16 characters.

Correction: the [MSDN documentation for [WriteConsole](wiki/WriteConsole)W](http://msdn.microsoft.com/en-us/library/ms687401%28VS.85%29.aspx) does document a length limitation, but it says 64 KiB (which would be 32767 UTF-16 characters and a terminating NUL), not 26608 characters. The fix is the same regardless; I'll limit it to 10000 UTF-16 characters.

daira commented

2010-10-27 02:42:39 +00:00

Author

Attachment workaround-windows-writeconsole-suckage.darcs.patch (5408 bytes) added

windows/fixups.py: limit length of string passed in a single call to WriteConsoleW. fixes #1232.

**Attachment** workaround-windows-writeconsole-suckage.darcs.patch (5408 bytes) added windows/fixups.py: limit length of string passed in a single call to [WriteConsole](wiki/WriteConsole)W. fixes #1232.

workaround-windows-writeconsole-suckage.darcs.patch

5.3 KiB

warner commented

2010-10-29 09:13:19 +00:00

looks good to me.

looks good to me.

daira commented

2010-10-29 19:43:09 +00:00

Author

In changeset:25d8103dde95d784:

windows/fixups.py: limit length of string passed in a single call to WriteConsoleW. fixes #1232.

In changeset:25d8103dde95d784: ``` windows/fixups.py: limit length of string passed in a single call to WriteConsoleW. fixes #1232. ```

daira added the

label 2010-10-29 19:43:09 +00:00

daira closed this issue

2010-10-29 19:43:09 +00:00

daira commented

2010-10-29 19:43:14 +00:00

Author

In changeset:2610f8e0aa6e2221:

NEWS: clarify (strengthen) description of what backdoors.rst declares, and add bugfix entries for 'tahoe cp' and Windows console bugs. refs #1216, #1224, #1232

In changeset:2610f8e0aa6e2221: ``` NEWS: clarify (strengthen) description of what backdoors.rst declares, and add bugfix entries for 'tahoe cp' and Windows console bugs. refs #1216, #1224, #1232 ```

daira commented

2011-01-09 08:47:29 +00:00

Author

Thread about this issue that seems to suggest the limit varies between machines:

(@@http://www.mail-archive.com/log4net-dev@logging.apache.org/msg00661.html@@)

I have not seen any reports of it failing with < 10000 characters, so our current code should work. However, I'm attempting to file the bug with Microsoft's bug reporting system, such as it is.

Thread about this issue that seems to suggest the limit varies between machines: (@@http://www.mail-archive.com/log4net-dev@logging.apache.org/msg00661.html@@) I have not seen any reports of it failing with < 10000 characters, so our current code should work. However, I'm attempting to file the bug with Microsoft's bug reporting system, such as it is.

daira commented

2011-03-27 14:13:00 +00:00

Author

It is possible for calls to the console functions by multiple threads to cause this error even when the amount written in each call is limited to 10000 characters (20000 bytes). However Tahoe-LAFS is single-threaded (almost; there are a few uses of deferToThread, but I don't think we print from those threads), so this shouldn't affect us. I'm pointing it out here because I know this ticket is referenced from tickets in other projects.

# Warning: this test may DoS your system.

from threading import Thread
import sys
from ctypes import WINFUNCTYPE, windll, POINTER, byref
from ctypes.wintypes import BOOL, HANDLE, DWORD, LPVOID, LPCVOID

GetStdHandle = WINFUNCTYPE(HANDLE, DWORD)(("GetStdHandle", windll.kernel32))
WriteFile = WINFUNCTYPE(BOOL, HANDLE, LPCVOID, DWORD, POINTER(DWORD), LPVOID) \
                        (("WriteFile", windll.kernel32))
GetLastError = WINFUNCTYPE(DWORD)(("GetLastError", windll.kernel32))
STD_OUTPUT_HANDLE = DWORD(-11)
INVALID_HANDLE_VALUE = DWORD(-1).value

hStdout = GetStdHandle(STD_OUTPUT_HANDLE)
assert hStdout is not None and hStdout != INVALID_HANDLE_VALUE

LENGTH = 20000 #bytes
THREADS = 10

data = b'\x08'*LENGTH

def run():
    n = DWORD(0)
    while True:
        ret = WriteFile(hStdout, data, LENGTH, byref(n), None)
        if ret == 0 or n.value != LENGTH:
            print("WriteFile returned %d, bytes written = %d, last error = %d"
                  % (ret, n.value, GetLastError()))
            sys.exit(1)

for i in range(THREADS):
    Thread(target=run).start()

It is possible for calls to the console functions by multiple threads to cause this error even when the amount written in each call is limited to 10000 characters (20000 bytes). However Tahoe-LAFS is single-threaded (almost; there are a few uses of `deferToThread`, but I don't think we print from those threads), so this shouldn't affect us. I'm pointing it out here because I know this ticket is referenced from tickets in other projects. ``` # Warning: this test may DoS your system. from threading import Thread import sys from ctypes import WINFUNCTYPE, windll, POINTER, byref from ctypes.wintypes import BOOL, HANDLE, DWORD, LPVOID, LPCVOID GetStdHandle = WINFUNCTYPE(HANDLE, DWORD)(("GetStdHandle", windll.kernel32)) WriteFile = WINFUNCTYPE(BOOL, HANDLE, LPCVOID, DWORD, POINTER(DWORD), LPVOID) \ (("WriteFile", windll.kernel32)) GetLastError = WINFUNCTYPE(DWORD)(("GetLastError", windll.kernel32)) STD_OUTPUT_HANDLE = DWORD(-11) INVALID_HANDLE_VALUE = DWORD(-1).value hStdout = GetStdHandle(STD_OUTPUT_HANDLE) assert hStdout is not None and hStdout != INVALID_HANDLE_VALUE LENGTH = 20000 #bytes THREADS = 10 data = b'\x08'*LENGTH def run(): n = DWORD(0) while True: ret = WriteFile(hStdout, data, LENGTH, byref(n), None) if ret == 0 or n.value != LENGTH: print("WriteFile returned %d, bytes written = %d, last error = %d" % (ret, n.value, GetLastError())) sys.exit(1) for i in range(THREADS): Thread(target=run).start() ```

Sign in to join this conversation.

No labels

c/code-dirnodes

c/code-encoding

c/code-frontend

c/code-frontend-cli

c/code-frontend-ftp-sftp

c/code-frontend-magic-folder

c/code-frontend-web

c/code-nodeadmin

c/code-peerselection

c/dev-infrastructure

kw:AttributeError

kw:DataUnavailable

kw:DeadReferenceError

kw:GetLastError

kw:IFinishableConsumer

kw:LeastAuthority

kw:RIStorageServer

kw:UncoordinatedWriteError

kw:access-control

kw:accessibility

kw:accounting-crawler

kw:anti-censorship

kw:api_auth_token

kw:authentication

kw:availability

kw:backward-compatibility

kw:blocks-cloud-deployment

kw:blocks-cloud-merge

kw:blocks-magic-folder-merge

kw:blocks-merge

kw:blocks-release

kw:brians-opinion-needed

kw:build-helpers

kw:cloud-backend

kw:coding-standards

kw:coding-tools

kw:coding_tools

kw:compatibility

kw:confidentiality

kw:configuration

kw:configuration.txt

kw:connectivity

kw:control.furl

kw:coordination

kw:coveralls.io

kw:create-container

kw:cryptography

kw:cryptography-lib

kw:denial-of-service

kw:desert-island

kw:desert-island-build

kw:design-review-needed

kw:dev-infrastructure

kw:directory-page

kw:disk-backend

kw:earth-dragon

kw:erasure-coding

kw:excess-authority

kw:extensibility

kw:file-descriptor

kw:floatingpoint

kw:forward-compatibility

kw:forward-secrecy

kw:garbage-collection

kw:google-cloud-storage

kw:google-drive-backend

kw:grid-manager

kw:illustration

kw:inlineCallbacks

kw:integration-test

kw:interoperability

kw:interstellar-exploration

kw:introduction

kw:key-value-store

kw:magic-folder

kw:manual-test-needed

kw:more-info-needed

kw:mountain-lion

kw:multiuser-gateway

kw:needs-review

kw:notification

kw:notifyOnDisconnect

kw:openitp-packaging

kw:operation-helpers

kw:optimization

kw:organization

kw:otf-magic-folder-objective1

kw:otf-magic-folder-objective2

kw:otf-magic-folder-objective3

kw:otf-magic-folder-objective4

kw:otf-magic-folder-objective5

kw:otf-magic-folder-objective6

kw:peer-selection

kw:pkg_resources

kw:preservation

kw:pycrypto-lib

kw:pyfilesystem

kw:random-access

kw:raspberry-pi

kw:release-blocker

kw:removable-disk

kw:reserved_space

kw:response-needed

kw:response-time

kw:review-needed

kw:self-contained

kw:servers-of-happiness

kw:setup_requires

kw:setuptools_darcs

kw:simultaneous

kw:space-efficiency

kw:static-analysis

kw:stats_gatherer

kw:tahoe-add-alias

kw:tahoe-archive

kw:tahoe-backup

kw:tahoe-create-alias

kw:tahoe-create-introducer

kw:tahoe-deep-check

kw:tahoe-deepcheck

kw:tahoe-lafs-trac-stream

kw:tahoe-list-aliases

kw:tahoe-magic-folder

kw:tahoe-manifest

kw:tahoe-restart

kw:tahoe-unlink

kw:tahoe-webopen

kw:test-and-set

kw:test-from-egg

kw:ticket999-s3-backend

kw:to-be-closed-on-2011-08-01

kw:tor-protocol

kw:transparency

kw:tub.location

kw:twisted-trial

kw:uncoordinated-writes

kw:unfinished-business

kw:unhandled-error

kw:upload-helper

kw:visualization

kw:volunteergrid2

kw:warners-opinion-needed

kw:welcome-page

kw:windows-related

kw:world-domination

kw:write-enabler

kw:zookos-opinion-needed

kw:zope.interface

p/supercritical

r/cannot reproduce

r/somebody else's problem

r/was already fixed

v/1.9.0-s3branch

No milestone

No project

No assignees

2 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: tahoe-lafs/trac#1232

No description provided.