Port foolscap to Python3 #2998


    foolscap/base32.py => 4194408
    foolscap/logging/log.py => 2540325
    foolscap/stringchain.py => 2403614
    foolscap/banana.py => 1954946
    foolscap/slicer.py => 664050
    foolscap/logging/incident.py => 453776
    foolscap/eventual.py => 448841
    foolscap/call.py => 287651
    foolscap/broker.py => 179140
    foolscap/constraint.py => 148022
    foolscap/slicers/root.py => 127975
    foolscap/slicers/dict.py => 106782
    foolscap/slicers/list.py => 103810
    foolscap/referenceable.py => 82494
    foolscap/slicers/tuple.py => 75613
    foolscap/schema.py => 70142
    foolscap/remoteinterface.py => 62729
    foolscap/negotiate.py => 48454
    foolscap/pb.py => 14599
    foolscap/slicers/bool.py => 8966
    foolscap/slicers/none.py => 8714
    foolscap/crypto.py => 5223
    foolscap/info.py => 3799
    foolscap/slicers/set.py => 2617
    foolscap/connection.py => 2518
    foolscap/reconnector.py => 2136
    foolscap/furl.py => 1840
    foolscap/util.py => 850
    foolscap/connections/tcp.py => 627
    foolscap/vocab.py => 538

Idea is to get the most "popular" source files so as to form some early foundation of the porting.

Figured out top files (modules) in foolscap source code by tracing system calls by running tahoe-lafs unit tests. These are, Top 30 modules in decreasing order of sorted calls... ``` foolscap/base32.py => 4194408 foolscap/logging/log.py => 2540325 foolscap/stringchain.py => 2403614 foolscap/banana.py => 1954946 foolscap/slicer.py => 664050 foolscap/logging/incident.py => 453776 foolscap/eventual.py => 448841 foolscap/call.py => 287651 foolscap/broker.py => 179140 foolscap/constraint.py => 148022 foolscap/slicers/root.py => 127975 foolscap/slicers/dict.py => 106782 foolscap/slicers/list.py => 103810 foolscap/referenceable.py => 82494 foolscap/slicers/tuple.py => 75613 foolscap/schema.py => 70142 foolscap/remoteinterface.py => 62729 foolscap/negotiate.py => 48454 foolscap/pb.py => 14599 foolscap/slicers/bool.py => 8966 foolscap/slicers/none.py => 8714 foolscap/crypto.py => 5223 foolscap/info.py => 3799 foolscap/slicers/set.py => 2617 foolscap/connection.py => 2518 foolscap/reconnector.py => 2136 foolscap/furl.py => 1840 foolscap/util.py => 850 foolscap/connections/tcp.py => 627 foolscap/vocab.py => 538 ``` Idea is to get the most "popular" source files so as to form some early foundation of the porting.

pythonhacker commented

2019-07-16 05:47:24 +00:00

Author

Owner

Fixed all the Python3 basic port issues, now pyetst is collecting everything, tests are all executing but of course failing - which is progress at this stage :-)

	
$ pytest .
================================================= test session starts ==================================================
platform linux -- Python 3.6.7, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: /home/anand/projects/anvetsu/tahoe/foolscap3
collected 534 items                                                                                                    

test__versions.py ...                                                                                            [  0%]
test_appserver.py FFFFFFFFFFFFFFFF...F                                                                           [  4%]
test_banana.py .............FE.................FEF.........FEFE..FE.....................FE....F...........FE.FEs [ 20%]
.Fss....FE.FEs..........F...                                                                                     [ 25%]
test_call.py .......................FE......FE...................FFFF                                            [ 35%]
test_connection.py .......FFFFFFF...FFFFFFFFFFFFFFFFFFFFFF                                                       [ 43%]
test_copyable.py .....FF..                                                                                       [ 44%]
test_crypto.py FFFF                                                                                              [ 45%]
test_eventual.py ...                                                                                             [ 46%]
test_gifts.py FFFFFFFFFFFFFFFF                                                                                   [ 49%]
test_info.py .FFFF.                                                                                              [ 50%]
test_interfaces.py .................                                                                             [ 53%]
test_keepalive.py FFFFF                                                                                          [ 54%]
test_listener.py FFFFF                                                                                           [ 55%]
test_logging.py .....F..F......F.FFFFFF..FFFFFFFFFFFFFFFFFFFFF.F.....FFFFFFFFEFE..                               [ 67%]
test_loopback.py F                                                                                               [ 67%]
test_negotiate.py FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF                                                        [ 74%]
test_observer.py .                                                                                               [ 75%]
test_pb.py .................FFFFFF.FFFFFF                                                                        [ 80%]
test_promise.py .....................                                                                            [ 84%]
test_reconnector.py FFFFFFF                                                                                      [ 85%]
test_reference.py .....                                                                                          [ 86%]
test_registration.py FFF                                                                                         [ 87%]
test_schema.py ..F.....F..F.FF....                                                                               [ 91%]
test_serialize.py ....FFFE                                                                                       [ 92%]
test_stringchain.py ........                                                                                     [ 93%]
test_sturdyref.py .F...                                                                                          [ 94%]
test_tub.py FFFFFFFFFFFFFFFF                                                                                     [ 97%]
test_unreachable.py FFF                                                                                          [ 98%]
test_util.py .........                                                                                           [100%]

Fixed all the Python3 basic port issues, now pyetst is collecting everything, tests are all executing but of course failing - which is progress at this stage :-) ``` $ pytest . ================================================= test session starts ================================================== platform linux -- Python 3.6.7, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 rootdir: /home/anand/projects/anvetsu/tahoe/foolscap3 collected 534 items test__versions.py ... [ 0%] test_appserver.py FFFFFFFFFFFFFFFF...F [ 4%] test_banana.py .............FE.................FEF.........FEFE..FE.....................FE....F...........FE.FEs [ 20%] .Fss....FE.FEs..........F... [ 25%] test_call.py .......................FE......FE...................FFFF [ 35%] test_connection.py .......FFFFFFF...FFFFFFFFFFFFFFFFFFFFFF [ 43%] test_copyable.py .....FF.. [ 44%] test_crypto.py FFFF [ 45%] test_eventual.py ... [ 46%] test_gifts.py FFFFFFFFFFFFFFFF [ 49%] test_info.py .FFFF. [ 50%] test_interfaces.py ................. [ 53%] test_keepalive.py FFFFF [ 54%] test_listener.py FFFFF [ 55%] test_logging.py .....F..F......F.FFFFFF..FFFFFFFFFFFFFFFFFFFFF.F.....FFFFFFFFEFE.. [ 67%] test_loopback.py F [ 67%] test_negotiate.py FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF [ 74%] test_observer.py . [ 75%] test_pb.py .................FFFFFF.FFFFFF [ 80%] test_promise.py ..................... [ 84%] test_reconnector.py FFFFFFF [ 85%] test_reference.py ..... [ 86%] test_registration.py FFF [ 87%] test_schema.py ..F.....F..F.FF.... [ 91%] test_serialize.py ....FFFE [ 92%] test_stringchain.py ........ [ 93%] test_sturdyref.py .F... [ 94%] test_tub.py FFFFFFFFFFFFFFFF [ 97%] test_unreachable.py FFF [ 98%] test_util.py ......... [100%] ```

pythonhacker commented

2019-07-17 18:02:45 +00:00

Author

Owner

I am working on test_banana as it seems to test the biggest and most important module (banana.py).

Reduced failures from a starting 109 to 41.


$ pytest test_banana.py
================================================= test session starts ==================================================
platform linux -- Python 3.6.7, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: /home/anand/projects/anvetsu/tahoe/foolscap3
collected 115 items                                                                                                    

...

-- Docs: https://docs.pytest.org/en/latest/warnings.html
========================= 41 failed, 69 passed, 5 skipped, 2 warnings, 1 error in 3.81 seconds =========================

I tried the py3 and py3.2 branches of warner's code and they have more errors in Python3 so clearly at this point my fork is at a better position.

I am working on `test_banana` as it seems to test the biggest and most important module (banana.py). Reduced failures from a starting 109 to 41. ``` $ pytest test_banana.py ================================================= test session starts ================================================== platform linux -- Python 3.6.7, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 rootdir: /home/anand/projects/anvetsu/tahoe/foolscap3 collected 115 items ... -- Docs: https://docs.pytest.org/en/latest/warnings.html ========================= 41 failed, 69 passed, 5 skipped, 2 warnings, 1 error in 3.81 seconds ========================= ``` I tried the py3 and py3.2 branches of warner's code and they have more errors in Python3 so clearly at this point my fork is at a better position.

pythonhacker commented

2019-07-18 12:42:16 +00:00

Author

Owner

Failures reduced to 22. banana.py will soon pass with another session of porting.

$ pytest test_banana.py
================================================= test session starts ==================================================
platform linux -- Python 3.6.7, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: /home/anand/projects/anvetsu/tahoe/foolscap3
collected 115 items                                                                                                    

test_banana.py ..........................................s....FFFFFFFFF..FFFFFFFF........F.FF.F.F......s..ss.... [ 84%]
...s..............                                                                                               [100%]

======================================================= FAILURES =======================================================


============================= 22 failed, 88 passed, 5 skipped, 2 warnings in 1.81 seconds ==============================

Failures reduced to 22. banana.py will soon pass with another session of porting. ``` $ pytest test_banana.py ================================================= test session starts ================================================== platform linux -- Python 3.6.7, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 rootdir: /home/anand/projects/anvetsu/tahoe/foolscap3 collected 115 items test_banana.py ..........................................s....FFFFFFFFF..FFFFFFFF........F.FF.F.F......s..ss.... [ 84%] ...s.............. [100%] ======================================================= FAILURES ======================================================= ============================= 22 failed, 88 passed, 5 skipped, 2 warnings in 1.81 seconds ============================== ```

pythonhacker commented

2019-07-22 15:30:55 +00:00

Author

Owner

Yo! One milestone reached. All tests passing on test_banana.py!!!

(tahoe3) anand@anand-HP-Pavilion-Laptop-15-cc1xx:~/projects/anvetsu/tahoe/foolscap3/src/foolscap/test {2019-07-22 20:58:39}

$ pytest test_banana.py
========================================== test session starts ===========================================
platform linux -- Python 3.6.7, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: /home/anand/projects/anvetsu/tahoe/foolscap3
collected 115 items                                                                                      

test_banana.py ................................................................................... [ 72%]
.....s..ss.......s..............                                                                   [100%]

============================================ warnings summary ============================================
test_banana.py:300
  /home/anand/projects/anvetsu/tahoe/foolscap3/src/foolscap/test/test_banana.py:300: PytestCollectionWarning: cannot collect test class 'TestTransport' because it has a __init__ constructor (from: src/foolscap/test/test_banana.py)
    class TestTransport(io.StringIO):

test_banana.py:307
  /home/anand/projects/anvetsu/tahoe/foolscap3/src/foolscap/test/test_banana.py:307: PytestCollectionWarning: cannot collect test class 'TestBytesTransport' because it has a __init__ constructor (from: src/foolscap/test/test_banana.py)
    class TestBytesTransport(io.BytesIO):

-- Docs: https://docs.pytest.org/en/latest/warnings.html
=========================== 111 passed, 4 skipped, 2 warnings in 0.43 seconds ============================

Yo! One milestone reached. All tests passing on test_banana.py!!! ``` (tahoe3) anand@anand-HP-Pavilion-Laptop-15-cc1xx:~/projects/anvetsu/tahoe/foolscap3/src/foolscap/test {2019-07-22 20:58:39} $ pytest test_banana.py ========================================== test session starts =========================================== platform linux -- Python 3.6.7, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 rootdir: /home/anand/projects/anvetsu/tahoe/foolscap3 collected 115 items test_banana.py ................................................................................... [ 72%] .....s..ss.......s.............. [100%] ============================================ warnings summary ============================================ test_banana.py:300 /home/anand/projects/anvetsu/tahoe/foolscap3/src/foolscap/test/test_banana.py:300: PytestCollectionWarning: cannot collect test class 'TestTransport' because it has a __init__ constructor (from: src/foolscap/test/test_banana.py) class TestTransport(io.StringIO): test_banana.py:307 /home/anand/projects/anvetsu/tahoe/foolscap3/src/foolscap/test/test_banana.py:307: PytestCollectionWarning: cannot collect test class 'TestBytesTransport' because it has a __init__ constructor (from: src/foolscap/test/test_banana.py) class TestBytesTransport(io.BytesIO): -- Docs: https://docs.pytest.org/en/latest/warnings.html =========================== 111 passed, 4 skipped, 2 warnings in 0.43 seconds ============================ ```

pythonhacker commented

2019-07-22 18:48:52 +00:00

Author

Owner

Made the tests retroactively pass on Python2 also ... after the Python3 changes. Ain't it cool ?

(tahoe) anand@anand-HP-Pavilion-Laptop-15-cc1xx:~/projects/anvetsu/tahoe/foolscap3/src/foolscap/test {2019-07-2
3 00:17:43}
$ pytest test_banana.py
========================================== test session starts ===========================================
platform linux2 -- Python 2.7.15rc1, pytest-4.3.1, py-1.8.0, pluggy-0.9.0
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/anand/projects/anvetsu/tahoe/foolscap3/src/foolscap/test/.hypothesis/examples')
rootdir: /home/anand/projects/anvetsu/tahoe/foolscap3, inifile:
plugins: twisted-1.10, hypothesis-4.27.0
collected 115 items                                                                                      

test_banana.py ................................................................................... [ 72%]
.....s..ss.......s..............                                                                   [100%]

============================================ warnings summary ============================================
test_banana.py:301
  /home/anand/projects/anvetsu/tahoe/foolscap3/src/foolscap/test/test_banana.py:301: PytestWarning: cannot collect test class 'TestTransport' because it has a __init__ constructor
    class TestTransport(io.StringIO):

test_banana.py:308
  /home/anand/projects/anvetsu/tahoe/foolscap3/src/foolscap/test/test_banana.py:308: PytestWarning: cannot collect test class 'TestBytesTransport' because it has a __init__ constructor
    class TestBytesTransport(io.BytesIO):

-- Docs: https://docs.pytest.org/en/latest/warnings.html
=========================== 111 passed, 4 skipped, 2 warnings in 0.32 seconds ============================

Made the tests retroactively pass on Python2 also ... after the Python3 changes. Ain't it cool ? ``` (tahoe) anand@anand-HP-Pavilion-Laptop-15-cc1xx:~/projects/anvetsu/tahoe/foolscap3/src/foolscap/test {2019-07-2 3 00:17:43} $ pytest test_banana.py ========================================== test session starts =========================================== platform linux2 -- Python 2.7.15rc1, pytest-4.3.1, py-1.8.0, pluggy-0.9.0 hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/anand/projects/anvetsu/tahoe/foolscap3/src/foolscap/test/.hypothesis/examples') rootdir: /home/anand/projects/anvetsu/tahoe/foolscap3, inifile: plugins: twisted-1.10, hypothesis-4.27.0 collected 115 items test_banana.py ................................................................................... [ 72%] .....s..ss.......s.............. [100%] ============================================ warnings summary ============================================ test_banana.py:301 /home/anand/projects/anvetsu/tahoe/foolscap3/src/foolscap/test/test_banana.py:301: PytestWarning: cannot collect test class 'TestTransport' because it has a __init__ constructor class TestTransport(io.StringIO): test_banana.py:308 /home/anand/projects/anvetsu/tahoe/foolscap3/src/foolscap/test/test_banana.py:308: PytestWarning: cannot collect test class 'TestBytesTransport' because it has a __init__ constructor class TestBytesTransport(io.BytesIO): -- Docs: https://docs.pytest.org/en/latest/warnings.html =========================== 111 passed, 4 skipped, 2 warnings in 0.32 seconds ============================ ```

pythonhacker commented

2019-09-03 10:06:59 +00:00

Author

Owner

(https://docs.google.com/spreadsheets/d/1M_l5FjBWtzUEfY7NXyt95TydpVqC9KrwD2DtBDLU6Dw/edit#gid=0)

Approx plan for tests porting. Higher priority indicates higher priority temporally.

(https://docs.google.com/spreadsheets/d/1M_l5FjBWtzUEfY7NXyt95TydpVqC9KrwD2DtBDLU6Dw/edit#gid=0) Approx plan for tests porting. Higher priority indicates higher priority temporally.