Ticket #1074: unicode-args-and-rm-tahoe-exe.4.dpatch

Thu Jul 15 06:31:15 GMT Daylight Time 2010  david-sarah@jacaranda.org
  * Changes to Tahoe needed to work with new zetuptoolz (that does not use .exe wrappers on Windows), and to support Unicode arguments and stdout/stderr -- v4.

New patches:

[Changes to Tahoe needed to work with new zetuptoolz (that does not use .exe wrappers on Windows), and to support Unicode arguments and stdout/stderr -- v4.
david-sarah@jacaranda.org**20100715053115
 Ignore-this: 8bcab59c644bda66493c9e51e91ccc29
] {
hunk ./bin/tahoe-script.template 8
 where = os.path.realpath(sys.argv[0])
 base = os.path.dirname(os.path.dirname(where))
 
+if sys.platform == "win32":
+    installed_tahoe = os.path.join(os.path.dirname(sys.executable), 'Scripts', 'tahoe.pyscript')
+else:
+    installed_tahoe = "/usr/bin/tahoe"
+
 whoami = '''\
 I am a "bin/tahoe" executable who is only for the convenience of running
 Tahoe from its source distribution -- I work only when invoked as the "tahoe"
hunk ./bin/tahoe-script.template 27
     print '''\
 I just tried to run and found that I am not living in such a directory, so I
 am stopping now. To run Tahoe after it has been is installed, please execute
-my brother, also named "tahoe", who gets installed into the appropriate place
-for executables when you run "make install" (perhaps as /usr/bin/tahoe).
-'''
+my brother, who gets installed into the appropriate place for executables
+when you run "make install" (perhaps as %s).
+''' % (installed_tahoe,)
     sys.exit(1)
 
 # we've found our home. Put the tahoe support/lib etc. in our PYTHONPATH.
hunk ./bin/tahoe-script.template 49
     pp = supportdir
 os.environ["PYTHONPATH"] = pp
 
-# find the location of the tahoe executable.
-bin_dir = "bin"
+# find commandline args and the location of the tahoe executable.
 if sys.platform == "win32":
hunk ./bin/tahoe-script.template 51
-    bin_dir = "Scripts"
-executable = os.path.join(base, "support", bin_dir, "tahoe")
+    import re
+    from ctypes import WINFUNCTYPE, POINTER, byref, c_wchar_p, c_int, windll
+
+    GetCommandLineW = WINFUNCTYPE(c_wchar_p)(("GetCommandLineW", windll.kernel32))
+    CommandLineToArgvW = WINFUNCTYPE(POINTER(c_wchar_p), c_wchar_p, POINTER(c_int)) \
+                            (("CommandLineToArgvW", windll.shell32))
+
+    argc = c_int(0)
+    argv_unicode = CommandLineToArgvW(GetCommandLineW(), byref(argc))
+
+    # See src/allmydata/scripts/runner.py for the corresponding unmangler.
+    # Note that this doesn't escape \x7F. If it did, test_unicode_arguments_and_output
+    # in test_runner.py wouldn't work.
+    def mangle(s):
+        return str(re.sub(ur'[^\x20-\x7F]', lambda m: u'\x7F%x;' % (ord(m.group(0)),), s))
+
+    argv = [mangle(argv_unicode[i]) for i in xrange(1, argc.value)]
+    local_tahoe = "Scripts\\tahoe.pyscript"
+else:
+    argv = sys.argv
+    local_tahoe = "bin/tahoe"
+
+script = os.path.join(base, "support", local_tahoe)
 
 try:
hunk ./bin/tahoe-script.template 76
-    res = subprocess.call([executable] + sys.argv[1:], env=os.environ)
+    res = subprocess.call([sys.executable, script] + argv[1:], env=os.environ)
 except (OSError, IOError), le:
     if le.args[0] == errno.ENOENT:
         print whoami
hunk ./bin/tahoe-script.template 80
+
         print '''\
 I just tried to run and could not find my brother, named
hunk ./bin/tahoe-script.template 83
-"../support/bin/tahoe". To run Tahoe when it is installed, please execute my
-brother, also named "tahoe", who gets installed into the appropriate place
-for executables when you run "make install" (perhaps as /usr/bin/tahoe).
-'''
+"../support/%s". To run Tahoe when it is installed, please execute my
+brother, who gets installed into the appropriate place for executables
+when you run "make install" (perhaps as %s).
+''' % (local_tahoe, installed_tahoe)
         raise
 except Exception, le:
     print whoami
hunk ./bin/tahoe-script.template 91
     print '''\
-I just tried to invoke my brother, named "../support/bin/tahoe" and got an
+I just tried to invoke my brother, named "../support/%s" and got an
 exception.
hunk ./bin/tahoe-script.template 93
-'''
+''' % (local_tahoe,)
     raise
 else:
     sys.exit(res)
hunk ./docs/quickstart.html 34
 
     <p>Unpack the zip file and cd into the top-level directory.</p>
 
+    <p>If you are on Windows and have not already done so, run <cite>python setup.py winsetup</cite>. This makes the registry and environment changes needed to run the <cite>tahoe</cite> command-line script. You may need to accept an elevation or other confirmation prompt.</p>
+
     <p>Run <cite>python setup.py build</cite> to generate the <cite>tahoe</cite> executable in a subdirectory of the current directory named <cite>bin</cite>.</p>
 
     <p>Optionally run <cite>python setup.py test</cite> to verify that it passes all of its self-tests.</p>
hunk ./setup.py 238
     def run(self):
         bin_tahoe_template = os.path.join("bin", "tahoe-script.template")
 
-        # Create the 'tahoe-script.py' file under the 'bin' directory. The
-        # 'tahoe-script.py' file is exactly the same as the
-        # 'tahoe-script.template' script except that the shebang line is
-        # rewritten to use our sys.executable for the interpreter. On
-        # Windows, create a tahoe.exe will execute it. On non-Windows, make a
-        # symlink to it from 'tahoe'. The tahoe.exe will be copied from the
-        # setuptools egg's cli.exe and this will work from a zip-safe and
-        # non-zip-safe setuptools egg.
+        if sys.platform == 'win32':
+            # 'tahoe' script is needed for cygwin
+            script_names = ["tahoe.pyscript", "tahoe"]
+        else:
+            script_names = ["tahoe"]
+
+        # Create the tahoe script file under the 'bin' directory. This
+        # file is exactly the same as the 'tahoe-script.template' script
+        # except that the shebang line is rewritten to use our sys.executable
+        # for the interpreter.
         f = open(bin_tahoe_template, "rU")
         script_lines = f.readlines()
         f.close()
hunk ./setup.py 251
-        script_lines[0] = "#!%s\n" % sys.executable
-        tahoe_script = os.path.join("bin", "tahoe-script.py")
-        f = open(tahoe_script, "w")
-        for line in script_lines:
-            f.write(line)
-        f.close()
-        if sys.platform == "win32":
-            from pkg_resources import require
-            setuptools_egg = require("setuptools")[0].location
-            if os.path.isfile(setuptools_egg):
-                z = zipfile.ZipFile(setuptools_egg, 'r')
-                for filename in z.namelist():
-                    if 'cli.exe' in filename:
-                        cli_exe = z.read(filename)
-            else:
-                cli_exe = os.path.join(setuptools_egg, 'setuptools', 'cli.exe')
-            tahoe_exe = os.path.join("bin", "tahoe.exe")
-            if os.path.isfile(setuptools_egg):
-                f = open(tahoe_exe, 'wb')
-                f.write(cli_exe)
-                f.close()
-            else:
-                shutil.copy(cli_exe, tahoe_exe)
-        else:
+        script_lines[0] = '#!%s\n' % sys.executable
+        for script_name in script_names:
+            tahoe_script = os.path.join("bin", script_name)
             try:
hunk ./setup.py 255
-                os.remove(os.path.join('bin', 'tahoe'))
-            except:
-                # okay, probably it was already gone
-                pass
-            os.symlink('tahoe-script.py', os.path.join('bin', 'tahoe'))
+                os.remove(tahoe_script)
+            except Exception:
+                if os.path.exists(tahoe_script):
+                   raise
+            f = open(tahoe_script, "wb")
+            for line in script_lines:
+                f.write(line)
+            f.close()
+
+            # chmod +x
+            old_mode = stat.S_IMODE(os.stat(tahoe_script)[stat.ST_MODE])
+            new_mode = old_mode | (stat.S_IXUSR | stat.S_IRUSR |
+                                   stat.S_IXGRP | stat.S_IRGRP |
+                                   stat.S_IXOTH | stat.S_IROTH )
+            os.chmod(tahoe_script, new_mode)
 
hunk ./setup.py 271
-        # chmod +x bin/tahoe-script.py
-        old_mode = stat.S_IMODE(os.stat(tahoe_script)[stat.ST_MODE])
-        new_mode = old_mode | (stat.S_IXUSR | stat.S_IRUSR |
-                               stat.S_IXGRP | stat.S_IRGRP |
-                               stat.S_IXOTH | stat.S_IROTH )
-        os.chmod(tahoe_script, new_mode)
+        old_tahoe_exe = os.path.join("bin", "tahoe.exe")
+        try:
+            os.remove(old_tahoe_exe)
+        except Exception:
+            if os.path.exists(old_tahoe_exe):
+                raise
 
 class MySdist(sdist.sdist):
     """ A hook in the sdist command so that we can determine whether this the
hunk ./src/allmydata/scripts/runner.py 12
 pkg_resources.require('allmydata-tahoe')
 from allmydata.scripts.common import BaseOptions
 from allmydata.scripts import debug, create_node, startstop_node, cli, keygen, stats_gatherer
+from allmydata.util.encodingutil import quote_output, get_argv_encoding
 
 def GROUP(s):
     # Usage.parseOptions compares argv[1] against command[0], so it will
hunk ./src/allmydata/scripts/runner.py 22
 
 
 class Options(BaseOptions, usage.Options):
-    synopsis = "Usage:  tahoe <command> [command options]"
+    synopsis = "\nUsage:  tahoe <command> [command options]"
     subCommands = ( GROUP("Administration")
                     +   create_node.subCommands
                     +   keygen.subCommands
hunk ./src/allmydata/scripts/runner.py 45
 
 def runner(argv,
            run_by_human=True,
-           stdin=sys.stdin, stdout=sys.stdout, stderr=sys.stderr,
+           stdin=None, stdout=None, stderr=None,
            install_node_control=True, additional_commands=None):
 
hunk ./src/allmydata/scripts/runner.py 48
+    stdin  = stdin  or sys.stdin
+    stdout = stdout or sys.stdout
+    stderr = stderr or sys.stderr
+
     config = Options()
     if install_node_control:
         config.subCommands.extend(startstop_node.subCommands)
hunk ./src/allmydata/scripts/runner.py 70
         c = config
         while hasattr(c, 'subOptions'):
             c = c.subOptions
-        print str(c)
-        print "%s:  %s" % (sys.argv[0], e)
+        print >>stdout, str(c)
+        try:
+            msg = e.args[0].decode(get_argv_encoding())
+        except Exception:
+            msg = repr(e)
+        print >>stdout, "%s:  %s\n" % (sys.argv[0], quote_output(msg, quotemarks=False))
         return 1
 
     command = config.subCommand
hunk ./src/allmydata/scripts/runner.py 110
 
     return rc
 
+
 def run(install_node_control=True):
hunk ./src/allmydata/scripts/runner.py 112
-    rc = runner(sys.argv[1:])
+    if sys.platform == "win32":
+        from allmydata.windows.fixups import initialize
+        initialize()
+
+    rc = runner(sys.argv[1:], install_node_control=install_node_control)
     sys.exit(rc)
hunk ./src/allmydata/test/__init__.py 28
 # we disable incident reporting for all unit tests.
 disable_foolscap_incidents()
 
+import sys
+if sys.platform == "win32":
+    from allmydata.windows.fixups import initialize
+    initialize()
+
hunk ./src/allmydata/test/test_encodingutil.py 28
     print
     print "class MyWeirdOS(EncodingUtil, unittest.TestCase):"
     print "    uname = '%s'" % ' '.join(platform.uname())
-    if sys.platform != "win32":
-        print "    argv = %s" % repr(sys.argv[1])
+    print "    argv = %s" % repr(sys.argv[1])
     print "    platform = '%s'" % sys.platform
     print "    filesystem_encoding = '%s'" % sys.getfilesystemencoding()
hunk ./src/allmydata/test/test_encodingutil.py 31
-    print "    output_encoding = '%s'" % sys.stdout.encoding
-    print "    argv_encoding = '%s'" % (sys.platform == "win32" and 'ascii' or sys.stdout.encoding)
+    print "    output_encoding = '%s'" % (sys.platform == "win32" and 'utf-8' or sys.stdout.encoding)
+    print "    argv_encoding = '%s'" % (sys.platform == "win32" and 'utf-8' or sys.stdout.encoding)
 
     try:
         tmpdir = tempfile.mkdtemp()
hunk ./src/allmydata/test/test_encodingutil.py 80
         self.failUnlessReallyEqual(get_output_encoding(), 'utf-8')
 
         mock_stdout.encoding = 'koi8-r'
+        expected = sys.platform == "win32" and 'utf-8' or 'koi8-r'
         _reload()
hunk ./src/allmydata/test/test_encodingutil.py 82
-        self.failUnlessReallyEqual(get_output_encoding(), 'koi8-r')
+        self.failUnlessReallyEqual(get_output_encoding(), expected)
 
         mock_stdout.encoding = 'nonexistent_encoding'
hunk ./src/allmydata/test/test_encodingutil.py 85
-        self.failUnlessRaises(AssertionError, _reload)
+        if sys.platform == "win32":
+            _reload()
+            self.failUnlessReallyEqual(get_output_encoding(), 'utf-8')
+        else:
+            self.failUnlessRaises(AssertionError, _reload)
 
     @patch('locale.getpreferredencoding')
     def test_get_output_encoding_not_from_stdout(self, mock_locale_getpreferredencoding):
hunk ./src/allmydata/test/test_encodingutil.py 101
         old_stdout = sys.stdout
         sys.stdout = DummyStdout()
         try:
+            expected = sys.platform == "win32" and 'utf-8' or 'koi8-r'
             _reload()
hunk ./src/allmydata/test/test_encodingutil.py 103
-            self.failUnlessReallyEqual(get_output_encoding(), 'koi8-r')
+            self.failUnlessReallyEqual(get_output_encoding(), expected)
 
             sys.stdout.encoding = None
             _reload()
hunk ./src/allmydata/test/test_encodingutil.py 107
-            self.failUnlessReallyEqual(get_output_encoding(), 'koi8-r')
+            self.failUnlessReallyEqual(get_output_encoding(), expected)
 
             mock_locale_getpreferredencoding.return_value = None
             _reload()
hunk ./src/allmydata/test/test_encodingutil.py 129
         # Encoding koi8-r cannot represent e-grave
         mock.encoding = 'koi8-r'
         _reload()
-        self.failUnlessRaises(UnicodeEncodeError, unicode_to_output, lumiere_nfc)
+        if sys.platform == "win32":
+            self.failUnlessReallyEqual(unicode_to_output(lumiere_nfc), 'lumi\xc3\xa8re')
+        else:
+            self.failUnlessRaises(UnicodeEncodeError, unicode_to_output, lumiere_nfc)
 
     @patch('os.listdir')
     def test_no_unicode_normalization(self, mock):
hunk ./src/allmydata/test/test_encodingutil.py 182
                               listdir_unicode,
                               u'/' + lumiere_nfc)
 
+
 class EncodingUtil(ReallyEqualMixin):
     def setUp(self):
         # Mock sys.platform because unicode_platform() uses it
hunk ./src/allmydata/test/test_encodingutil.py 214
 
         mock.encoding = self.output_encoding
         _reload()
-        self.failUnlessReallyEqual(unicode_to_output(lumiere_nfc), self.output)
+        self.failUnlessReallyEqual(unicode_to_output(lumiere_nfc), self.argv)
 
     def test_unicode_platform(self):
         matrix = {
hunk ./src/allmydata/test/test_encodingutil.py 296
 
 class UbuntuKarmicUTF8(EncodingUtil, unittest.TestCase):
     uname = 'Linux korn 2.6.31-14-generic #48-Ubuntu SMP Fri Oct 16 14:05:01 UTC 2009 x86_64'
-    output = 'lumi\xc3\xa8re'
     argv = 'lumi\xc3\xa8re'
     platform = 'linux2'
     filesystem_encoding = 'UTF-8'
hunk ./src/allmydata/test/test_encodingutil.py 305
 
 class UbuntuKarmicLatin1(EncodingUtil, unittest.TestCase):
     uname = 'Linux korn 2.6.31-14-generic #48-Ubuntu SMP Fri Oct 16 14:05:01 UTC 2009 x86_64'
-    output = 'lumi\xe8re'
     argv = 'lumi\xe8re'
     platform = 'linux2'
     filesystem_encoding = 'ISO-8859-1'
hunk ./src/allmydata/test/test_encodingutil.py 312
     argv_encoding = 'ISO-8859-1'
     dirlist = ['test_file', 'Blah blah.txt', '\xc4rtonwall.mp3']
 
-class WindowsXP(EncodingUtil, unittest.TestCase):
+class Windows(EncodingUtil, unittest.TestCase):
     uname = 'Windows XP 5.1.2600 x86 x86 Family 15 Model 75 Step ping 2, AuthenticAMD'
hunk ./src/allmydata/test/test_encodingutil.py 314
-    output = 'lumi\x8are'
-    platform = 'win32'
-    filesystem_encoding = 'mbcs'
-    output_encoding = 'cp850'
-    argv_encoding = 'ascii'
-    dirlist = [u'Blah blah.txt', u'test_file', u'\xc4rtonwall.mp3']
-
-class WindowsXP_UTF8(EncodingUtil, unittest.TestCase):
-    uname = 'Windows XP 5.1.2600 x86 x86 Family 15 Model 75 Step ping 2, AuthenticAMD'
-    output = 'lumi\xc3\xa8re'
-    platform = 'win32'
-    filesystem_encoding = 'mbcs'
-    output_encoding = 'cp65001'
-    argv_encoding = 'ascii'
-    dirlist = [u'Blah blah.txt', u'test_file', u'\xc4rtonwall.mp3']
-
-class WindowsVista(EncodingUtil, unittest.TestCase):
-    uname = 'Windows Vista 6.0.6000 x86 x86 Family 6 Model 15 Stepping 11, GenuineIntel'
-    output = 'lumi\x8are'
+    argv = 'lumi\xc3\xa8re'
     platform = 'win32'
     filesystem_encoding = 'mbcs'
hunk ./src/allmydata/test/test_encodingutil.py 317
-    output_encoding = 'cp850'
-    argv_encoding = 'ascii'
+    output_encoding = 'utf-8'
+    argv_encoding = 'utf-8'
     dirlist = [u'Blah blah.txt', u'test_file', u'\xc4rtonwall.mp3']
 
 class MacOSXLeopard(EncodingUtil, unittest.TestCase):
hunk ./src/allmydata/test/test_encodingutil.py 323
     uname = 'Darwin g5.local 9.8.0 Darwin Kernel Version 9.8.0: Wed Jul 15 16:57:01 PDT 2009; root:xnu-1228.15.4~1/RELEASE_PPC Power Macintosh powerpc'
-    output = 'lumi\xc3\xa8re'
     argv = 'lumi\xc3\xa8re'
     platform = 'darwin'
     filesystem_encoding = 'utf-8'
hunk ./src/allmydata/test/test_runner.py 11
 from cStringIO import StringIO
 from allmydata.util import fileutil, pollmixin
 from allmydata.scripts import runner
+from allmydata.util.encodingutil import unicode_to_argv, unicode_to_output
 
 from allmydata.test import common_util
 import allmydata
hunk ./src/allmydata/test/test_runner.py 17
 
 bintahoe = os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(allmydata.__file__))), 'bin', 'tahoe')
-if sys.platform == "win32":  # TODO: should this include cygwin?
-    bintahoe += ".exe"
+if sys.platform == "win32":
+    bintahoe += ".pyscript"
 
 
 class SkipMixin:
hunk ./src/allmydata/test/test_runner.py 52
         d.addCallback(_cb)
         return d
 
+    def test_unicode_arguments_and_output(self):
+        self.skip_if_cannot_run_bintahoe()
+
+        tricky = u"\u2621"
+        try:
+            tricky_arg = unicode_to_argv(tricky)
+            tricky_out = unicode_to_output(tricky)
+        except UnicodeEncodeError:
+            raise unittest.SkipTest("A non-ASCII argument/output could not be encoded on this platform.")
+
+        d = utils.getProcessOutputAndValue(bintahoe, args=[tricky_arg], env=os.environ)
+        def _cb(res):
+            out, err, rc_or_sig = res
+            self.failUnlessEqual(rc_or_sig, 1, str((out, err, rc_or_sig)))
+            self.failUnlessIn("Unknown command: "+tricky_out, out)
+        d.addCallback(_cb)
+        return d
+
 
 class CreateNode(unittest.TestCase):
     # exercise "tahoe create-node", create-introducer,
hunk ./src/allmydata/util/encodingutil.py 15
 from allmydata.util import log
 
 
-def _canonical_encoding(encoding):
+def canonical_encoding(encoding):
     if encoding is None:
         log.msg("Warning: falling back to UTF-8 encoding.", level=log.WEIRD)
         encoding = 'utf-8'
hunk ./src/allmydata/util/encodingutil.py 25
     elif encoding == "us-ascii" or encoding == "646" or encoding == "ansi_x3.4-1968":
         encoding = 'ascii'
 
+    return encoding
+
+def check_encoding(encoding):
     # sometimes Python returns an encoding name that it doesn't support for conversion
     # fail early if this happens
     try:
hunk ./src/allmydata/util/encodingutil.py 35
     except (LookupError, AttributeError):
         raise AssertionError("The character encoding '%s' is not supported for conversion." % (encoding,))
 
-    return encoding
-
 filesystem_encoding = None
 output_encoding = None
 argv_encoding = None
hunk ./src/allmydata/util/encodingutil.py 43
 def _reload():
     global filesystem_encoding, output_encoding, argv_encoding, is_unicode_platform
 
-    filesystem_encoding = _canonical_encoding(sys.getfilesystemencoding())
-
-    outenc = None
-    if hasattr(sys.stdout, 'encoding'):
-        outenc = sys.stdout.encoding
-    if outenc is None:
-        try:
-            outenc = locale.getpreferredencoding()
-        except Exception:
-            pass  # work around <http://bugs.python.org/issue1443504>
-    output_encoding = _canonical_encoding(outenc)
+    filesystem_encoding = canonical_encoding(sys.getfilesystemencoding())
+    check_encoding(filesystem_encoding)
 
     if sys.platform == 'win32':
hunk ./src/allmydata/util/encodingutil.py 47
-        # Unicode arguments are not supported on Windows yet; see #565 and #1074.
-        argv_encoding = 'ascii'
+        # On Windows we install UTF-8 stream wrappers for sys.stdout and
+        # sys.stderr, and reencode the arguments as UTF-8 (see scripts/runner.py).
+        output_encoding = 'utf-8'
     else:
hunk ./src/allmydata/util/encodingutil.py 51
-        argv_encoding = output_encoding
+        outenc = None
+        if hasattr(sys.stdout, 'encoding'):
+            outenc = sys.stdout.encoding
+        if outenc is None:
+            try:
+                outenc = locale.getpreferredencoding()
+            except Exception:
+                pass  # work around <http://bugs.python.org/issue1443504>
+        output_encoding = canonical_encoding(outenc)
+
+    check_encoding(output_encoding)
+    argv_encoding = output_encoding
+
     is_unicode_platform = sys.platform in ["win32", "darwin"]
 
 _reload()
hunk ./src/allmydata/util/encodingutil.py 99
         raise usage.UsageError("Argument %s cannot be decoded as %s." %
                                (quote_output(s), argv_encoding))
 
+def unicode_to_argv(s):
+    """
+    Encode the given Unicode argument as a bytestring. On Windows, this uses a
+    mangled encoding that will be reversed by code in runner.py.
+    """
+    precondition(isinstance(s, unicode), s)
+
+    if sys.platform == "win32":
+        # This must be the same as 'mangle' in bin/tahoe-script.template.
+        return str(re.sub(ur'[^\x20-\x7F]', lambda m: u'\x7F%x;' % (ord(m.group(0)),), s))
+    else:
+        return s.decode(argv_encoding)
+
 def unicode_to_url(s):
     """
     Encode an unicode object used in an URL.
addfile ./src/allmydata/windows/fixups.py
hunk ./src/allmydata/windows/fixups.py 1
+
+done = False
+
+def initialize():
+    global done
+    import sys
+    if sys.platform != "win32" or done:
+        return True
+    done = True
+
+    import codecs, re
+    from ctypes import WINFUNCTYPE, windll, CFUNCTYPE, cdll, POINTER, byref, \
+        c_wchar_p, c_char_p, c_void_p, c_int, c_size_t
+    from allmydata.util import log
+    from allmydata.util.encodingutil import canonical_encoding
+
+    # Work around <http://bugs.python.org/issue6058>.
+    codecs.register(lambda name: name == 'cp65001' and codecs.lookup('utf-8') or None)
+
+    # Make Unicode console output work independently of the current code page.
+    # This also fixes <http://bugs.python.org/issue1602>.
+    try:
+        STDOUT_FILENO = 1
+        STDERR_FILENO = 2
+        real_stdout = hasattr(sys.stdout, 'fileno') and sys.stdout.fileno() == STDOUT_FILENO
+        real_stderr = hasattr(sys.stderr, 'fileno') and sys.stderr.fileno() == STDERR_FILENO
+
+        def force_utf8(stream, name):
+            if hasattr(stream, 'encoding') and canonical_encoding(stream.encoding) != 'utf-8':
+                log.msg("%s (%r) had encoding %r, but we're going to write UTF-8 to it" %
+                        (name, stream, stream.encoding), level=log.CURIOUS)
+            stream.encoding = 'utf-8'
+
+        if not real_stdout:
+            force_utf8(sys.stdout, "sys.stdout")
+
+        if not real_stderr:
+            force_utf8(sys.stderr, "sys.stderr")
+
+        if real_stdout or real_stderr:
+            # FILE * _fdopen(int fd, const char *mode);
+            # #define _IOLBF 0x0040
+            # int setvbuf(FILE *stream, char *buffer, int mode, size_t size);
+            # #define _O_U8TEXT 0x40000
+            # int _setmode(int fd, int mode);
+            # int fputws(const wchar_t *ws, FILE *stream);
+            # int fflush(FILE *stream);
+
+            c_runtime = cdll.msvcrt
+            NULL = None
+            _fdopen = CFUNCTYPE(c_void_p, c_int, c_char_p)(("_fdopen", c_runtime))
+            _IOLBF = 0x0040
+            setvbuf = CFUNCTYPE(c_int, c_void_p, c_char_p, c_int, c_size_t)(("setvbuf", c_runtime))
+            _O_U8TEXT = 0x40000
+            _setmode = CFUNCTYPE(c_int, c_int, c_int)(("_setmode", c_runtime))
+            fputws = CFUNCTYPE(c_int, c_wchar_p, c_void_p)(("fputws", c_runtime));
+            fflush = CFUNCTYPE(c_int, c_void_p)(("fflush", c_runtime));
+
+            buffer_chars = 1024
+
+            class UnicodeOutput:
+                def __init__(self, fileno, name):
+                    self._stream = _fdopen(fileno, "w")
+                    assert self._stream is not NULL
+
+                    # Deep magic. MSVCRT supports writing wide-oriented output to stdout/stderr
+                    # to the console using the Unicode APIs, but it does the conversion in the
+                    # stdio buffer, so you need that buffer to be as large as the maximum amount
+                    # you're going to write in a single call (in bytes, not characters).
+                    setvbuf(self._stream, NULL, _IOLBF, buffer_chars*4 + 100)
+                    _setmode(fileno, _O_U8TEXT)
+
+                    self._fileno = fileno
+                    self.closed = False
+                    self.softspace = False
+                    self.mode = 'w'
+                    self.encoding = 'utf-8'
+                    self.name = name
+
+                def isatty(self):
+                    return False
+                def close(self):
+                    self.closed = True
+                    self.flush()
+                def fileno(self):
+                    return self._fileno
+                def flush(self):
+                    fflush(self._stream)
+
+                def write(self, text):
+                    if not isinstance(text, unicode):
+                        text = str(text).decode('utf-8')
+                    for i in xrange(0, len(text), buffer_chars):
+                        fputws(text[i:(i+buffer_chars)], self._stream)
+                        fflush(self._stream)
+
+                def writelines(self, lines):
+                    for line in lines:
+                        self.write(line)
+
+            if real_stdout:
+                sys.stdout = UnicodeOutput(STDOUT_FILENO, '<Unicode stdout>')
+
+            if real_stderr:
+                sys.stderr = UnicodeOutput(STDERR_FILENO, '<Unicode stderr>')
+    except Exception, e:
+        log.msg("exception %r while fixing up sys.stdout and sys.stderr" % (e,), level=log.WEIRD)
+
+    # Unmangle command-line arguments.
+    GetCommandLineW = WINFUNCTYPE(c_wchar_p)(("GetCommandLineW", windll.kernel32))
+    CommandLineToArgvW = WINFUNCTYPE(POINTER(c_wchar_p), c_wchar_p, POINTER(c_int)) \
+                            (("CommandLineToArgvW", windll.shell32))
+
+    argc = c_int(0)
+    argv_unicode = CommandLineToArgvW(GetCommandLineW(), byref(argc))
+
+    def unmangle(s):
+        return re.sub(ur'\x7f[0-9a-fA-F]*\;', lambda m: unichr(int(m.group(0)[1:-1], 16)), s)
+
+    try:
+        sys.argv = [unmangle(argv_unicode[i]).encode('utf-8') for i in xrange(1, argc.value)]
+    except Exception, e:
+        print >>sys.stderr, "%s:  could not unmangle Unicode arguments" % (sys.argv[0],)
+        print >>sys.stderr, [argv_unicode[i] for i in xrange(1, argc.value)]
+        raise
}

Context:

[Rename stringutils to encodingutil, and drop open_unicode (since the Python 'open' function works fine with Unicode paths).
david-sarah@jacaranda.org**20100713042715
 Ignore-this: fa2bb6b5d48ce9ba7ea2b1afc9f3b7b4
] 
[Resolve conflicts between NFC normalization changes, and post-1.7 branch.
david-sarah@jacaranda.org**20100618021642
 Ignore-this: ed3d0d71b761b1317cff9a4c92c2f5cb
] 
[trivial: a copy of tiny code-cleanups suggested by Kevan in reviewing #967
zooko@zooko.com**20100617045339
 Ignore-this: 274b3fdbf1531aa053f484a5d47d7447
] 
[minor code clean-up in dirnode.py
zooko@zooko.com**20100221052527
 Ignore-this: b01bfb47638f5a64256bb306e0022066
 Impose micro-POLA by passing only the writekey instead of the whole node object to {{{_encrypt_rw_uri()}}}. Remove DummyImmutableFileNode in nodemaker.py, which is obviated by this. Add micro-optimization by precomputing the netstring of the empty string and branching on whether the writekey is present or not outside of {{{_encrypt_rw_uri()}}}. Add doc about writekey to docstring.
] 
[Move EncryptedTemporaryFile from SFTP frontend to allmydata.util.fileutil, and make the FTP frontend also use it (fixing #1083).
david-sarah@jacaranda.org**20100711213721
 Ignore-this: e452e8ca66391aa2a1a49afe0114f317
] 
[Add tests of caps from the future that have non-ASCII characters in them (encoded as UTF-8). The changes to test_uri.py, test_client.py, and test_dirnode.py add tests of non-ASCII future caps in addition to the current tests. The changes to test_web.py just replace the tests of all-ASCII future caps with tests of non-ASCII future caps. We also change uses of failUnlessEqual to failUnlessReallyEqual, in order to catch cases where the type of a string is not as expected.
david-sarah@jacaranda.org**20100711200252
 Ignore-this: c2f193352369d32e06865f8f3e951894
] 
[Allow URIs passed in the initial JSON for t=mkdir-with-children, t=mkdir-immutable to be Unicode. Also pass the name of each child into nodemaker.create_from_cap for error reporting.
david-sarah@jacaranda.org**20100711195525
 Ignore-this: deac32d8b91ba26ede18905d3f7d2b93
] 
[docs/logging.txt: note that setting flogging vars might affect tests with race conditions.
david-sarah@jacaranda.org**20100712050721
 Ignore-this: fc1609d215fcd5561a57fd1226206f27
] 
[test_storage.py: potential fix for failures when logging is enabled.
david-sarah@jacaranda.org**19700713040546
 Ignore-this: 5815693a0df3e64c52c3c6b7be2846c7
] 
[SFTP: address some of the comments in zooko's review (#1106).
david-sarah@jacaranda.org**20100712025537
 Ignore-this: c3921638a2d4f1de2a776ae78e4dc37e
] 
[upcase_since_on_welcome
terrellrussell@gmail.com**20100708193903] 
[server_version_on_welcome_page.dpatch.txt
freestorm77@gmail.com**20100605191721
 Ignore-this: b450c76dc875f5ac8cca229a666cbd0a
 
 
 - The storage server version is 0 for all storage nodes in the Welcome Page
 
 
] 
[NEWS: add NEWS snippets about two recent patches
zooko@zooko.com**20100708162058
 Ignore-this: 6c9da6a0ad7351a960bdd60f81532899
] 
[directory_html_top_banner.dpatch
freestorm77@gmail.com**20100622205301
 Ignore-this: 1d770d975e0c414c996564774f049bca
 
 The div tag with the link "Return to Welcome page" on the directory.xhtml page is not correct
 
] 
[tahoe_css_toolbar.dpatch
freestorm77@gmail.com**20100622210046
 Ignore-this: 5b3ebb2e0f52bbba718a932f80c246c0
 
 CSS modification to be correctly diplayed with Internet Explorer 8
 
 The links on the top of page directory.xhtml are not diplayed in the same line as display with Firefox.
 
] 
[runnin_test_tahoe_css.dpatch
freestorm77@gmail.com**20100622214714
 Ignore-this: e0db73d68740aad09a7b9ae60a08c05c
 
 Runnin test for changes in tahoe.css file
 
] 
[runnin_test_directory_xhtml.dpatch
freestorm77@gmail.com**20100622201403
 Ignore-this: f8962463fce50b9466405cb59fe11d43
 
 Runnin test for diretory.xhtml top banner
 
] 
[stringutils.py: tolerate sys.stdout having no 'encoding' attribute.
david-sarah@jacaranda.org**20100626040817
 Ignore-this: f42cad81cef645ee38ac1df4660cc850
] 
[quickstart.html: python 2.5 -> 2.6 as recommended version
david-sarah@jacaranda.org**20100705175858
 Ignore-this: bc3a14645ea1d5435002966ae903199f
] 
[SFTP: don't call .stopProducing on the producer registered with OverwriteableFileConsumer (which breaks with warner's new downloader).
david-sarah@jacaranda.org**20100628231926
 Ignore-this: 131b7a5787bc85a9a356b5740d9d996f
] 
[docs/how_to_make_a_tahoe-lafs_release.txt: trivial correction, install.html should now be quickstart.html.
david-sarah@jacaranda.org**20100625223929
 Ignore-this: 99a5459cac51bd867cc11ad06927ff30
] 
[setup: in the Makefile, refuse to upload tarballs unless someone has passed the environment variable "BB_BRANCH" with value "trunk"
zooko@zooko.com**20100619034928
 Ignore-this: 276ddf9b6ad7ec79e27474862e0f7d6
] 
[trivial: tiny update to in-line comment
zooko@zooko.com**20100614045715
 Ignore-this: 10851b0ed2abfed542c97749e5d280bc
 (I'm actually committing this patch as a test of the new eager-annotation-computation of trac-darcs.)
] 
[docs: about.html link to home page early on, and be decentralized storage instead of cloud storage this time around
zooko@zooko.com**20100619065318
 Ignore-this: dc6db03f696e5b6d2848699e754d8053
] 
[docs: update about.html, especially to have a non-broken link to quickstart.html, and also to comment out the broken links to "for Paranoids" and "for Corporates"
zooko@zooko.com**20100619065124
 Ignore-this: e292c7f51c337a84ebfeb366fbd24d6c
] 
[TAG allmydata-tahoe-1.7.0
zooko@zooko.com**20100619052631
 Ignore-this: d21e27afe6d85e2e3ba6a3292ba2be1
] 
Patch bundle hash:
bc0e02a3adc819e598af6dda619dcf5a10d23985