Opened at 2019-03-13T18:32:04Z
Closed at 2020-01-09T18:22:03Z
#2998 closed task (somebody else's problem)
Port foolscap to Python3
| Reported by: | pythonhacker | Owned by: | |
|---|---|---|---|
| Priority: | normal | Milestone: | undecided |
| Component: | code | Version: | 1.12.1 |
| Keywords: | Python3 | Cc: | |
| Launchpad Bug: |
Description
Part of dependencies of tahoe-lfs, foolscap is only supported for Python2. This ticket addresses its porting steps to Python3.
Change History (10)
comment:1 Changed at 2019-06-04T18:45:29Z by pythonhacker
comment:2 Changed at 2019-07-09T15:06:16Z by pythonhacker
Restarted work last week.
comment:3 Changed at 2019-07-09T15:11:55Z by pythonhacker
Figured out top files (modules) in foolscap source code by tracing system calls by running tahoe-lafs unit tests. These are,
Top 30 modules in decreasing order of sorted calls...
foolscap/base32.py => 4194408
foolscap/logging/log.py => 2540325
foolscap/stringchain.py => 2403614
foolscap/banana.py => 1954946
foolscap/slicer.py => 664050
foolscap/logging/incident.py => 453776
foolscap/eventual.py => 448841
foolscap/call.py => 287651
foolscap/broker.py => 179140
foolscap/constraint.py => 148022
foolscap/slicers/root.py => 127975
foolscap/slicers/dict.py => 106782
foolscap/slicers/list.py => 103810
foolscap/referenceable.py => 82494
foolscap/slicers/tuple.py => 75613
foolscap/schema.py => 70142
foolscap/remoteinterface.py => 62729
foolscap/negotiate.py => 48454
foolscap/pb.py => 14599
foolscap/slicers/bool.py => 8966
foolscap/slicers/none.py => 8714
foolscap/crypto.py => 5223
foolscap/info.py => 3799
foolscap/slicers/set.py => 2617
foolscap/connection.py => 2518
foolscap/reconnector.py => 2136
foolscap/furl.py => 1840
foolscap/util.py => 850
foolscap/connections/tcp.py => 627
foolscap/vocab.py => 538
Idea is to get the most "popular" source files so as to form some early foundation of the porting.
comment:4 Changed at 2019-07-16T05:47:24Z by pythonhacker
Fixed all the Python3 basic port issues, now pyetst is collecting everything, tests are all executing but of course failing - which is progress at this stage :-)
$ pytest . ================================================= test session starts ================================================== platform linux -- Python 3.6.7, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 rootdir: /home/anand/projects/anvetsu/tahoe/foolscap3 collected 534 items test__versions.py ... [ 0%] test_appserver.py FFFFFFFFFFFFFFFF...F [ 4%] test_banana.py .............FE.................FEF.........FEFE..FE.....................FE....F...........FE.FEs [ 20%] .Fss....FE.FEs..........F... [ 25%] test_call.py .......................FE......FE...................FFFF [ 35%] test_connection.py .......FFFFFFF...FFFFFFFFFFFFFFFFFFFFFF [ 43%] test_copyable.py .....FF.. [ 44%] test_crypto.py FFFF [ 45%] test_eventual.py ... [ 46%] test_gifts.py FFFFFFFFFFFFFFFF [ 49%] test_info.py .FFFF. [ 50%] test_interfaces.py ................. [ 53%] test_keepalive.py FFFFF [ 54%] test_listener.py FFFFF [ 55%] test_logging.py .....F..F......F.FFFFFF..FFFFFFFFFFFFFFFFFFFFF.F.....FFFFFFFFEFE.. [ 67%] test_loopback.py F [ 67%] test_negotiate.py FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF [ 74%] test_observer.py . [ 75%] test_pb.py .................FFFFFF.FFFFFF [ 80%] test_promise.py ..................... [ 84%] test_reconnector.py FFFFFFF [ 85%] test_reference.py ..... [ 86%] test_registration.py FFF [ 87%] test_schema.py ..F.....F..F.FF.... [ 91%] test_serialize.py ....FFFE [ 92%] test_stringchain.py ........ [ 93%] test_sturdyref.py .F... [ 94%] test_tub.py FFFFFFFFFFFFFFFF [ 97%] test_unreachable.py FFF [ 98%] test_util.py ......... [100%]
comment:5 Changed at 2019-07-17T18:02:45Z by pythonhacker
I am working on test_banana as it seems to test the biggest and most important module (banana.py).
Reduced errors from a starting 109 to 41.
$ pytest test_banana.py ================================================= test session starts ================================================== platform linux -- Python 3.6.7, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 rootdir: /home/anand/projects/anvetsu/tahoe/foolscap3 collected 115 items ... -- Docs: https://docs.pytest.org/en/latest/warnings.html ========================= 41 failed, 69 passed, 5 skipped, 2 warnings, 1 error in 3.81 seconds =========================
I tried the py3 and py3.2 branches of warner's code and they have more errors in Python3 so clearly at this point my fork is at a better position.
comment:6 Changed at 2019-07-18T12:42:16Z by pythonhacker
Failures reduced to 22. banana.py will soon pass with another session of porting.
$ pytest test_banana.py ================================================= test session starts ================================================== platform linux -- Python 3.6.7, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 rootdir: /home/anand/projects/anvetsu/tahoe/foolscap3 collected 115 items test_banana.py ..........................................s....FFFFFFFFF..FFFFFFFF........F.FF.F.F......s..ss.... [ 84%] ...s.............. [100%] ======================================================= FAILURES ======================================================= ============================= 22 failed, 88 passed, 5 skipped, 2 warnings in 1.81 seconds ==============================
comment:7 Changed at 2019-07-22T15:30:55Z by pythonhacker
Yo! One milestone reached. All tests passing on test_banana.py!!!
(tahoe3) anand@anand-HP-Pavilion-Laptop-15-cc1xx:~/projects/anvetsu/tahoe/foolscap3/src/foolscap/test {2019-07-22 20:58:39}
$ pytest test_banana.py
========================================== test session starts ===========================================
platform linux -- Python 3.6.7, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: /home/anand/projects/anvetsu/tahoe/foolscap3
collected 115 items
test_banana.py ................................................................................... [ 72%]
.....s..ss.......s.............. [100%]
============================================ warnings summary ============================================
test_banana.py:300
/home/anand/projects/anvetsu/tahoe/foolscap3/src/foolscap/test/test_banana.py:300: PytestCollectionWarning: cannot collect test class 'TestTransport' because it has a __init__ constructor (from: src/foolscap/test/test_banana.py)
class TestTransport(io.StringIO):
test_banana.py:307
/home/anand/projects/anvetsu/tahoe/foolscap3/src/foolscap/test/test_banana.py:307: PytestCollectionWarning: cannot collect test class 'TestBytesTransport' because it has a __init__ constructor (from: src/foolscap/test/test_banana.py)
class TestBytesTransport(io.BytesIO):
-- Docs: https://docs.pytest.org/en/latest/warnings.html
=========================== 111 passed, 4 skipped, 2 warnings in 0.43 seconds ============================
comment:8 Changed at 2019-07-22T18:48:52Z by pythonhacker
Made the tests retroactively pass on Python2 also ... after the Python3 changes. Ain't it cool ?
(tahoe) anand@anand-HP-Pavilion-Laptop-15-cc1xx:~/projects/anvetsu/tahoe/foolscap3/src/foolscap/test {2019-07-2
3 00:17:43}
$ pytest test_banana.py
========================================== test session starts ===========================================
platform linux2 -- Python 2.7.15rc1, pytest-4.3.1, py-1.8.0, pluggy-0.9.0
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/anand/projects/anvetsu/tahoe/foolscap3/src/foolscap/test/.hypothesis/examples')
rootdir: /home/anand/projects/anvetsu/tahoe/foolscap3, inifile:
plugins: twisted-1.10, hypothesis-4.27.0
collected 115 items
test_banana.py ................................................................................... [ 72%]
.....s..ss.......s.............. [100%]
============================================ warnings summary ============================================
test_banana.py:301
/home/anand/projects/anvetsu/tahoe/foolscap3/src/foolscap/test/test_banana.py:301: PytestWarning: cannot collect test class 'TestTransport' because it has a __init__ constructor
class TestTransport(io.StringIO):
test_banana.py:308
/home/anand/projects/anvetsu/tahoe/foolscap3/src/foolscap/test/test_banana.py:308: PytestWarning: cannot collect test class 'TestBytesTransport' because it has a __init__ constructor
class TestBytesTransport(io.BytesIO):
-- Docs: https://docs.pytest.org/en/latest/warnings.html
=========================== 111 passed, 4 skipped, 2 warnings in 0.32 seconds ============================
comment:9 Changed at 2019-09-03T10:06:59Z by pythonhacker
https://docs.google.com/spreadsheets/d/1M_l5FjBWtzUEfY7NXyt95TydpVqC9KrwD2DtBDLU6Dw/edit#gid=0
Approx plan for tests porting. Higher priority indicates higher priority temporally.
comment:10 Changed at 2020-01-09T18:22:03Z by exarkun
- Resolution set to somebody else's problem
- Status changed from new to closed
This is being tracked upstream now - https://github.com/warner/foolscap/issues/48
Note: See
TracTickets for help on using
tickets.
Considering the discussions on https://github.com/warner/foolscap/issues/48 - not starting anything on this for now.