os.path.expanduser (and therefore abspath_expanduser_unicode) fails on Windows if username or home directory contains non-ASCII characters

[vikarti]

attempt to perform setup.py test results in

argparse 0.9.1 is already the active version in easy-install.pth                                                                                                                                                16:34

Using c:\tahoe\allmydata-tahoe-1.9.1\support\lib\site-packages\argparse-0.9.1-py2.7-win32.egg
Searching for zbase32==1.1.3
Best match: zbase32 1.1.3
Processing zbase32-1.1.3-py2.7.egg
zbase32 1.1.3 is already the active version in easy-install.pth

Using c:\tahoe\allmydata-tahoe-1.9.1\support\lib\site-packages\zbase32-1.1.3-py2.7.egg
Finished processing dependencies for allmydata-tahoe==1.9.1

Setting up environment to run scripts for the current user...
Already done: associate the filetype .pyscript with Python.File for the current user.
Already done: add .pyscript to the user environment variable PATHEXT.
Already done: add .pyw to the user environment variable PATHEXT.
running make_executable
running build
running build_py

Setting up environment to run scripts for the current user...
Already done: associate the filetype .pyscript with Python.File for the current user.
Already done: add .pyscript to the user environment variable PATHEXT.
Already done: add .pyw to the user environment variable PATHEXT.
running trial
Traceback (most recent call last):
  File "C:\Tahoe\allmydata-tahoe-1.9.1\support\Scripts\tahoe.pyscript", line 16, in <module>
    load_entry_point('allmydata-tahoe==1.9.1', 'console_scripts', 'tahoe')()
  File "C:\Tahoe\allmydata-tahoe-1.9.1\support\Lib\site-packages\setuptools-0.6c16dev3.egg\pkg_resources.py", line 318, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "C:\Tahoe\allmydata-tahoe-1.9.1\support\Lib\site-packages\setuptools-0.6c16dev3.egg\pkg_resources.py", line 2245, in load_entry_point
    return ep.load()
  File "C:\Tahoe\allmydata-tahoe-1.9.1\support\Lib\site-packages\setuptools-0.6c16dev3.egg\pkg_resources.py", line 1978, in load
    entry = __import__(self.module_name, globals(),globals(), ['__name__'])
  File "c:\tahoe\allmydata-tahoe-1.9.1\src\allmydata\scripts\runner.py", line 7, in <module>
    from allmydata.scripts.common import BaseOptions
  File "c:\tahoe\allmydata-tahoe-1.9.1\src\allmydata\scripts\common.py", line 19, in <module>
    path = abspath_expanduser_unicode(u"~/.tahoe")
  File "c:\tahoe\allmydata-tahoe-1.9.1\src\allmydata\util\fileutil.py", line 291, in abspath_expanduser_unicode
    path = os.path.expanduser(path)
  File "C:\Python27\lib\ntpath.py", line 301, in expanduser
    return userhome + path[i:]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc0 in position 9: ordinal not in range(128)

C:\Tahoe\allmydata-tahoe-1.9.1>                                                                                                                                                                                     ^

Host:Windows 7,Python 2.7(same issue - with 2.6.6) This problem happens if current username is not ASCII (it is by default on some international localizations of Windows is this bug with python on windows?

[davidsarah]

The bug might be in os.path.expanduser or we might be calling it incorrectly; I'll have to check its specification.

[davidsarah]

os.path.expanduser appears to be hopelessly buggy when the username is non-ASCII (it returns the wrong answer even when it doesn't fail). We were already working around two bugs in the path functions (​http://bugs.python.org/issue1669539 and ​http://bugs.python.org/issue5827), so it might be simpler just to write our own implementation of expanduser from scratch.

[davidsarah]

Replying to davidsarah:

We were already working around two bugs in the path functions ...

Actually three, the other is ​http://bugs.python.org/issue3426 .

[daira]

See also #2235: by making all user-expansions go through fileutil.abspath_expanduser_unicode, we'll only need to fix this problem in that function.

[daira]

I'm working on this in the branch for #2235.

[daira]

See ​https://github.com/tahoe-lafs/tahoe-lafs/pull/138.

[daira]

Fixed on the pr138 branch which was merged in [3afe827ad0cbdb41b2928d17c8bfbbaf5102acc8/trunk].

(Woohoo! This set of bugs has been irking me for some time.)