Start porting allmydata.scripts to Python 3

[jaraco]

​Pull Request 975

Specifically covers allmydata.test.cli.test_create, allmydata.test.cli.test_runner, allmydata.scripts.create_node.

[jaraco]

I learned that for the scripts, even after adding them to 'PORTED_MODULES', none of the tests fail, so even syntax errors aren't caught by the test suite. I'll need to find another way to ascertain the routines necessary to validate an effective port.

[jaraco]

I found at least one failure mode that can be fixed:

$ .tox/py36/bin/python -m allmydata.scripts.runner
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/jaraco/code/public/tahoe-lafs/.tox/py36/lib/python3.6/site-packages/allmydata/scripts/runner.py", line 251, in <module>
    run()
  File "/Users/jaraco/code/public/tahoe-lafs/.tox/py36/lib/python3.6/site-packages/allmydata/scripts/runner.py", line 182, in run
    assert sys.version_info < (3,), u"Tahoe-LAFS does not run under Python 3. Please use Python 2.7.x."
AssertionError: Tahoe-LAFS does not run under Python 3. Please use Python 2.7.x.

[jaraco]

After ​5bf2b09b8, I've run some basic tests on runner, and it appears to be working with -q and -V and properly dispatching commands, so I'm prepared to declare that module as ported.

[jaraco]

Attempting to run create-client results in an error:

tahoe-lafs 3603.scripts $ .tox/py36/bin/python -m allmydata.scripts.runner create-client
/Users/jaraco/code/public/tahoe-lafs/src/allmydata/scripts/runner.py:182: UserWarning: Support for Python 3 is experimental. Use at your own risk.
  warnings.warn("Support for Python 3 is experimental. Use at your own risk.")
Traceback (most recent call last):
  File "/Users/jaraco/code/public/tahoe-lafs/.tox/py36/lib/python3.6/site-packages/twisted/internet/defer.py", line 151, in maybeDeferred
    result = f(*args, **kw)
  File "/Users/jaraco/code/public/tahoe-lafs/src/allmydata/scripts/create_node.py", line 469, in create_client
    return create_node(config)
  File "/Users/jaraco/code/public/tahoe-lafs/.tox/py36/lib/python3.6/site-packages/twisted/internet/defer.py", line 1613, in unwindGenerator
    return _cancellableInlineCallbacks(gen)
  File "/Users/jaraco/code/public/tahoe-lafs/.tox/py36/lib/python3.6/site-packages/twisted/internet/defer.py", line 1529, in _cancellableInlineCallbacks
    _inlineCallbacks(None, g, status)
--- <exception caught here> ---
  File "/Users/jaraco/code/public/tahoe-lafs/.tox/py36/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/Users/jaraco/code/public/tahoe-lafs/src/allmydata/scripts/create_node.py", line 405, in create_node
    precondition(isinstance(basedir, unicode), basedir)
builtins.NameError: name 'unicode' is not defined

[jaraco]

With ​36b7fdaecf, create-client now succeeds.

tahoe-lafs 3603.scripts $ .tox/py36/bin/python -m allmydata.scripts.runner create-client
Node created in '/Users/jaraco/.tahoe'
 Please add introducers to '/Users/jaraco/.tahoe/private/introducers.yaml'!
 The node cannot connect to a grid without it.
 Please set [node]nickname= in '/Users/jaraco/.tahoe/tahoe.cfg'

[jaraco]

Run command fails with:

tahoe-lafs 3603.scripts $ .tox/py36/bin/python -m allmydata.scripts.runner run
'tahoe run' in '/Users/jaraco/.tahoe'
running node in '/Users/jaraco/.tahoe'
2021-01-30T12:26:27-0500 [twisted.scripts._twistd_unix.UnixAppLogger#info] twistd 20.3.0 (/Users/jaraco/code/public/tahoe-lafs/.tox/py36/bin/python 3.6.6) starting up.
2021-01-30T12:26:27-0500 [twisted.scripts._twistd_unix.UnixAppLogger#info] reactor class: twisted.internet.selectreactor.SelectReactor.
2021-01-30T12:26:29-0500 [-] Foolscap logging initialized
2021-01-30T12:26:29-0500 [-] Note to developers: twistd.log does not receive very much.
2021-01-30T12:26:29-0500 [-] Use 'flogtool tail -c NODEDIR/private/logport.furl' instead
2021-01-30T12:26:29-0500 [-] and read docs/logging.rst
2021-01-30T12:26:32-0500 [-] Main loop terminated.
2021-01-30T12:26:32-0500 [twisted.scripts._twistd_unix.UnixAppLogger#info] Server Shut Down.

Unknown error
Traceback (most recent call last):
  File "/Users/jaraco/code/public/tahoe-lafs/.tox/py36/lib/python3.6/site-packages/twisted/internet/defer.py", line 151, in maybeDeferred
    result = f(*args, **kw)
  File "/Users/jaraco/code/public/tahoe-lafs/src/allmydata/client.py", line 239, in create_client
    _client_factory=_client_factory,
  File "/Users/jaraco/code/public/tahoe-lafs/.tox/py36/lib/python3.6/site-packages/twisted/internet/defer.py", line 1613, in unwindGenerator
    return _cancellableInlineCallbacks(gen)
  File "/Users/jaraco/code/public/tahoe-lafs/.tox/py36/lib/python3.6/site-packages/twisted/internet/defer.py", line 1529, in _cancellableInlineCallbacks
    _inlineCallbacks(None, g, status)
--- <exception caught here> ---
  File "/Users/jaraco/code/public/tahoe-lafs/.tox/py36/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/Users/jaraco/code/public/tahoe-lafs/src/allmydata/client.py", line 290, in create_client_from_config
    storage_broker,
  File "/Users/jaraco/code/public/tahoe-lafs/src/allmydata/client.py", line 669, in __init__
    self.init_web(webport) # strports string
  File "/Users/jaraco/code/public/tahoe-lafs/src/allmydata/client.py", line 1024, in init_web
    staticdir,
  File "/Users/jaraco/code/public/tahoe-lafs/src/allmydata/webish.py", line 208, in __init__
    self.buildServer(webport, tempdir, nodeurl_path, staticdir)
  File "/Users/jaraco/code/public/tahoe-lafs/src/allmydata/webish.py", line 228, in buildServer
    s = strports.service(webport, self.site)
  File "/Users/jaraco/code/public/tahoe-lafs/.tox/py36/lib/python3.6/site-packages/twisted/application/strports.py", line 40, in service
    endpoints.serverFromString(reactor, description), factory)
  File "/Users/jaraco/code/public/tahoe-lafs/.tox/py36/lib/python3.6/site-packages/twisted/internet/endpoints.py", line 1731, in serverFromString
    nameOrPlugin, args, kw = _parseServer(description, None)
  File "/Users/jaraco/code/public/tahoe-lafs/.tox/py36/lib/python3.6/site-packages/twisted/internet/endpoints.py", line 1649, in _parseServer
    getPlugins(IStreamServerEndpointStringParser), endpointType
  File "/Users/jaraco/code/public/tahoe-lafs/.tox/py36/lib/python3.6/site-packages/twisted/internet/endpoints.py", line 1665, in _matchPluginToPrefix
    raise ValueError("Unknown endpoint type: '%s'" % (endpointType,))
builtins.ValueError: Unknown endpoint type: 'b'tcp'

[jaraco]

Looking at the config file, it looks like the config generated by create-client has some unwanted bytes literals:

[node]
nickname = b''
reveal-IP-address = true
web.port = b'tcp:3456:interface=127.0.0.1'

[jaraco]

In ​57cb88638, I've addressed that issue by opening the config file using io.open with encoding='utf-8' and then writing text.

[jaraco]

Attempting to run an invalid command, the rendering doesn't look right:

$ .tox/py36/bin/python -m allmydata.scripts.runner start out
...
/Users/jaraco/code/public/tahoe-lafs/src/allmydata/scripts/runner.py:  "UsageError('Unknown command: start',)"

Compare that to what's expected to be rendered on Python 2:

/Users/jaraco/code/public/tahoe-lafs/src/allmydata/scripts/runner.py:  Unknown command: start

[jaraco]

The issue ​traces to this line, where the message fails to be "decoded" when it's already text.

[jaraco]

That behavior ​traces to 37b07a545f.

[jaraco]

Fixed in 621ae58ab.

[jaraco]

Running test_runner on Python 3 (tox -e py36 -- allmydata.test.test_runner) reveals many failures. That might be a good module to port next.

[jaraco]

It occurs to me that the PR at its current state is probably in a good place for review. It doesn't close this issue, but at least addresses some concerns.

[GitHub <noreply@…>]

In dddcff09/trunk:

Merge pull request #975 from tahoe-lafs/3603.scripts

Port scripts.create_node to Python 3

Fixes ticket:3603