#2998 closed task (somebody else's problem)

Port foolscap to Python3

Reported by: pythonhacker Owned by:
Priority: normal Milestone: undecided
Component: code Version: 1.12.1
Keywords: Python3 Cc:
Launchpad Bug:

Description

Part of dependencies of tahoe-lfs, foolscap is only supported for Python2. This ticket addresses its porting steps to Python3.

Change History (10)

comment:1 Changed at 2019-06-04T18:45:29Z by pythonhacker

Considering the discussions on https://github.com/warner/foolscap/issues/48 - not starting anything on this for now.

comment:2 Changed at 2019-07-09T15:06:16Z by pythonhacker

Restarted work last week.

comment:3 Changed at 2019-07-09T15:11:55Z by pythonhacker

Figured out top files (modules) in foolscap source code by tracing system calls by running tahoe-lafs unit tests. These are,

Top 30 modules in decreasing order of sorted calls...

    foolscap/base32.py => 4194408
    foolscap/logging/log.py => 2540325
    foolscap/stringchain.py => 2403614
    foolscap/banana.py => 1954946
    foolscap/slicer.py => 664050
    foolscap/logging/incident.py => 453776
    foolscap/eventual.py => 448841
    foolscap/call.py => 287651
    foolscap/broker.py => 179140
    foolscap/constraint.py => 148022
    foolscap/slicers/root.py => 127975
    foolscap/slicers/dict.py => 106782
    foolscap/slicers/list.py => 103810
    foolscap/referenceable.py => 82494
    foolscap/slicers/tuple.py => 75613
    foolscap/schema.py => 70142
    foolscap/remoteinterface.py => 62729
    foolscap/negotiate.py => 48454
    foolscap/pb.py => 14599
    foolscap/slicers/bool.py => 8966
    foolscap/slicers/none.py => 8714
    foolscap/crypto.py => 5223
    foolscap/info.py => 3799
    foolscap/slicers/set.py => 2617
    foolscap/connection.py => 2518
    foolscap/reconnector.py => 2136
    foolscap/furl.py => 1840
    foolscap/util.py => 850
    foolscap/connections/tcp.py => 627
    foolscap/vocab.py => 538

Idea is to get the most "popular" source files so as to form some early foundation of the porting.

Last edited at 2019-07-09T15:12:37Z by pythonhacker (previous) (diff)

comment:4 Changed at 2019-07-16T05:47:24Z by pythonhacker

Fixed all the Python3 basic port issues, now pyetst is collecting everything, tests are all executing but of course failing - which is progress at this stage :-)

	
$ pytest .
================================================= test session starts ==================================================
platform linux -- Python 3.6.7, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: /home/anand/projects/anvetsu/tahoe/foolscap3
collected 534 items                                                                                                    

test__versions.py ...                                                                                            [  0%]
test_appserver.py FFFFFFFFFFFFFFFF...F                                                                           [  4%]
test_banana.py .............FE.................FEF.........FEFE..FE.....................FE....F...........FE.FEs [ 20%]
.Fss....FE.FEs..........F...                                                                                     [ 25%]
test_call.py .......................FE......FE...................FFFF                                            [ 35%]
test_connection.py .......FFFFFFF...FFFFFFFFFFFFFFFFFFFFFF                                                       [ 43%]
test_copyable.py .....FF..                                                                                       [ 44%]
test_crypto.py FFFF                                                                                              [ 45%]
test_eventual.py ...                                                                                             [ 46%]
test_gifts.py FFFFFFFFFFFFFFFF                                                                                   [ 49%]
test_info.py .FFFF.                                                                                              [ 50%]
test_interfaces.py .................                                                                             [ 53%]
test_keepalive.py FFFFF                                                                                          [ 54%]
test_listener.py FFFFF                                                                                           [ 55%]
test_logging.py .....F..F......F.FFFFFF..FFFFFFFFFFFFFFFFFFFFF.F.....FFFFFFFFEFE..                               [ 67%]
test_loopback.py F                                                                                               [ 67%]
test_negotiate.py FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF                                                        [ 74%]
test_observer.py .                                                                                               [ 75%]
test_pb.py .................FFFFFF.FFFFFF                                                                        [ 80%]
test_promise.py .....................                                                                            [ 84%]
test_reconnector.py FFFFFFF                                                                                      [ 85%]
test_reference.py .....                                                                                          [ 86%]
test_registration.py FFF                                                                                         [ 87%]
test_schema.py ..F.....F..F.FF....                                                                               [ 91%]
test_serialize.py ....FFFE                                                                                       [ 92%]
test_stringchain.py ........                                                                                     [ 93%]
test_sturdyref.py .F...                                                                                          [ 94%]
test_tub.py FFFFFFFFFFFFFFFF                                                                                     [ 97%]
test_unreachable.py FFF                                                                                          [ 98%]
test_util.py .........                                                                                           [100%]

comment:5 Changed at 2019-07-17T18:02:45Z by pythonhacker

I am working on test_banana as it seems to test the biggest and most important module (banana.py).

Reduced erros from a starting 109 to 41.

$ pytest test_banana.py
================================================= test session starts ==================================================
platform linux -- Python 3.6.7, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: /home/anand/projects/anvetsu/tahoe/foolscap3
collected 115 items                                                                                                    

...

-- Docs: https://docs.pytest.org/en/latest/warnings.html
========================= 41 failed, 69 passed, 5 skipped, 2 warnings, 1 error in 3.81 seconds =========================

I tried the py3 and py3.2 branches of warner's code and they have more errors in Python3 so clearly at this point my fork is at a better position.

Version 0, edited at 2019-07-17T18:02:45Z by pythonhacker (next)

comment:6 Changed at 2019-07-18T12:42:16Z by pythonhacker

Failures reduced to 22. banana.py will soon pass with another session of porting.

$ pytest test_banana.py
================================================= test session starts ==================================================
platform linux -- Python 3.6.7, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: /home/anand/projects/anvetsu/tahoe/foolscap3
collected 115 items                                                                                                    

test_banana.py ..........................................s....FFFFFFFFF..FFFFFFFF........F.FF.F.F......s..ss.... [ 84%]
...s..............                                                                                               [100%]

======================================================= FAILURES =======================================================


============================= 22 failed, 88 passed, 5 skipped, 2 warnings in 1.81 seconds ==============================

comment:7 Changed at 2019-07-22T15:30:55Z by pythonhacker

Yo! One milestone reached. All tests passing on test_banana.py!!!

(tahoe3) anand@anand-HP-Pavilion-Laptop-15-cc1xx:~/projects/anvetsu/tahoe/foolscap3/src/foolscap/test {2019-07-22 20:58:39}

$ pytest test_banana.py
========================================== test session starts ===========================================
platform linux -- Python 3.6.7, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: /home/anand/projects/anvetsu/tahoe/foolscap3
collected 115 items                                                                                      

test_banana.py ................................................................................... [ 72%]
.....s..ss.......s..............                                                                   [100%]

============================================ warnings summary ============================================
test_banana.py:300
  /home/anand/projects/anvetsu/tahoe/foolscap3/src/foolscap/test/test_banana.py:300: PytestCollectionWarning: cannot collect test class 'TestTransport' because it has a __init__ constructor (from: src/foolscap/test/test_banana.py)
    class TestTransport(io.StringIO):

test_banana.py:307
  /home/anand/projects/anvetsu/tahoe/foolscap3/src/foolscap/test/test_banana.py:307: PytestCollectionWarning: cannot collect test class 'TestBytesTransport' because it has a __init__ constructor (from: src/foolscap/test/test_banana.py)
    class TestBytesTransport(io.BytesIO):

-- Docs: https://docs.pytest.org/en/latest/warnings.html
=========================== 111 passed, 4 skipped, 2 warnings in 0.43 seconds ============================

comment:8 Changed at 2019-07-22T18:48:52Z by pythonhacker

Made the tests retroactively pass on Python2 also ... after the Python3 changes. Ain't it cool ?

(tahoe) anand@anand-HP-Pavilion-Laptop-15-cc1xx:~/projects/anvetsu/tahoe/foolscap3/src/foolscap/test {2019-07-2
3 00:17:43}
$ pytest test_banana.py
========================================== test session starts ===========================================
platform linux2 -- Python 2.7.15rc1, pytest-4.3.1, py-1.8.0, pluggy-0.9.0
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/anand/projects/anvetsu/tahoe/foolscap3/src/foolscap/test/.hypothesis/examples')
rootdir: /home/anand/projects/anvetsu/tahoe/foolscap3, inifile:
plugins: twisted-1.10, hypothesis-4.27.0
collected 115 items                                                                                      

test_banana.py ................................................................................... [ 72%]
.....s..ss.......s..............                                                                   [100%]

============================================ warnings summary ============================================
test_banana.py:301
  /home/anand/projects/anvetsu/tahoe/foolscap3/src/foolscap/test/test_banana.py:301: PytestWarning: cannot collect test class 'TestTransport' because it has a __init__ constructor
    class TestTransport(io.StringIO):

test_banana.py:308
  /home/anand/projects/anvetsu/tahoe/foolscap3/src/foolscap/test/test_banana.py:308: PytestWarning: cannot collect test class 'TestBytesTransport' because it has a __init__ constructor
    class TestBytesTransport(io.BytesIO):

-- Docs: https://docs.pytest.org/en/latest/warnings.html
=========================== 111 passed, 4 skipped, 2 warnings in 0.32 seconds ============================

comment:9 Changed at 2019-09-03T10:06:59Z by pythonhacker

https://docs.google.com/spreadsheets/d/1M_l5FjBWtzUEfY7NXyt95TydpVqC9KrwD2DtBDLU6Dw/edit#gid=0

Approx plan for tests porting. Higher priority indicates higher priority temporally.

comment:10 Changed at 2020-01-09T18:22:03Z by exarkun

  • Resolution set to somebody else's problem
  • Status changed from new to closed

This is being tracked upstream now - https://github.com/warner/foolscap/issues/48

Note: See TracTickets for help on using tickets.