View Issue Details

IDProjectCategoryView StatusLast Update
0005560GNUnetintegration testspublic2019-02-28 11:17
Reporterng0Assigned ToChristian Grothoff 
PrioritynormalSeverityminorReproducibilityhave not tried
Status closedResolutionfixed 
Product VersionSVN HEAD 
Target Version0.11.0Fixed in Version0.11.0 
Summary0005560: python3.7 support for test suite
DescriptionThe integration tests are currently failing on python3.7.
I will fix this (probably) by introducing a dependency on 'future' to make gnunet tests succeed with python2.7 and at least python3.7.
Tested will be only python3.7, since that's what gnunet picks up for a while now on my system (binary: python3.7).
TagsNo tags attached.

Relationships

related to 0005563 closedng0 check for python3.7 for python related tests 

Activities

ng0

2019-02-12 14:43

developer   ~0013684

In my opinion we should put an emphasis on python3.7 here and consider everything else an option.
Switch to officially support python3.7.
The bytes/str fallacy in the integration-tests is taking some time since I'm solving this in a compatible way.

ng0

2019-02-12 22:02

developer   ~0013692

We have roughly 11 months left with official support for python2, I think it makes more sense to simply ignore python2 and have at the very least python3.4 support (I'm building with python3.7 already).
python2 won't go away, but I think it will be easier to support only one version of a language. No idea if I can make this in time for 0.11.0, I'll try

ng0

2019-02-12 22:23

developer   ~0013693

Last edited: 2019-02-13 17:27

View 12 revisions

I'm running tests with 3.7 now in all directories containing python files, updating this post later on:

* src/statistics passes with python3.7 (pkgsrc, netbsd current)
* src/fs passes with python3.7 (pkgsrc, netbsd current)
* src/arm passes the python test (3 tests pass, 1 fails) with python3.7 (pkgsrc, netbsd current)
* src/dht passes with python3.7 (pkgsrc, netbsd current), failure fixed in HEAD
* src/peertool-info passes with python3.7 (pkgsrc, netbsd current)
* src/revocation passes the python test (1 tests pass, 1 fails) with python3.7 (pkgsrc, netbsd current)
* src/consensus passes with python3.7 (pkgsrc, netbsd current)
* src/testbed: should be python3.7 ready, the only python file in there is rather version neutral (no lines which could throw off py2 or py3..py37)

TODO:
* integration-tests

ng0

2019-02-13 01:27

developer   ~0013696

runtime problems in moving to python3.7:
* gnunet-qr creates a permanent optional runtime dependency on python2.7 (see 0005562).

ng0

2019-02-13 15:23

developer   ~0013699

dht python test failure fixed in https://git.gnunet.org/gnunet.git/commit/?id=aaca8161989c5af3f46251cd3e8d5fb531a5cb01

ng0

2019-02-13 22:54

developer  

test-suite.log (29,042 bytes)

ng0

2019-02-13 22:56

developer   ~0013715

Last edited: 2019-02-13 22:56

View 2 revisions

Christian, Please see if/how I messed up the tests for integration-tests, with python3.7

Christian Grothoff

2019-02-13 22:56

manager   ~0013716

Why was this assigned to me?

Christian Grothoff

2019-02-13 22:57

manager   ~0013717

Ah, concurrency issue ;-). Didn't see your comment.

ng0

2019-02-13 22:57

developer   ~0013718

see my latest comment, with the addition of the log file. I'm simply not sure if I understood the tests right.

ng0

2019-02-13 23:00

developer   ~0013719

Last edited: 2019-02-13 23:05

View 4 revisions

The core of the python2 -> python3.7 migration issue here was the "res" variable in print of class StatisticsCondition producing a TypeError, where str can not be concatenated with byte


Tested it like this:
cd src/integration-tests
PATH=$PATH:/home/ng0/src/gnunet/gnunet:/home/ng0/opt/bin TMP=/tmp gmake check

which might be wrong but worked for most other tests.


Not assuming you kept up with python2/3 problems:
https://eli.thegreenplace.net/2012/01/30/the-bytesstr-dichotomy-in-python-3
https://python-future.org/compatible_idioms.html
showcases some of the differences.

Christian Grothoff

2019-02-13 23:31

manager   ~0013724

FAIL: test_integration_bootstrap_and_connect.py:

This log suggests you're giving the wrong config file to one of the peers, resulting in both trying the same ports, which is why it fails.

Christian Grothoff

2019-02-13 23:43

manager   ~0013725

FAIL: test_integration_disconnect.py:
Configuration was not good, used some plugins that should not be tested here, plus TCP plugin somehow used link local v6 addresses which are (in 0.11) not allowed by the validation routine causing the assertion. Maybe in combination this caused the peers to not connect; I've modified the configuration and TCP plugin to alleviate those specific issues.

Christian Grothoff

2019-02-13 23:44

manager   ~0013726

FAIL: test_integration_disconnect_nat.py:
Again, seems we don't use the right configuration files anymore, launching both peers on the same port(s).

Christian Grothoff

2019-02-13 23:44

manager   ~0013727

FAIL: test_integration_reconnect.py:
Again, seems we don't use the right configuration files anymore, launching both peers on the same port(s).

Christian Grothoff

2019-02-13 23:45

manager   ~0013728

FAIL: test_integration_clique.py:
Strangely, only the NAT service seems to be having a problem with shared ports. Maybe an issue in the configuration, the NAT service is relatively new.

Christian Grothoff

2019-02-13 23:51

manager   ~0013729

Indeed the NAT config was wrong, fixed in a2de1176e..9188dd31a

ng0

2019-02-13 23:52

developer   ~0013731

is the output in my original log wrong or causing problems? I did not expect the bytes ('b') to appear as a prefix to all lines.

Christian Grothoff

2019-02-14 00:01

manager   ~0013735

That 'b' is indeed not expected (and likely a minor bug), but I doubt it is the cause of the test failure.

ng0

2019-02-14 00:42

developer   ~0013736

for the record of the ticket - as written in irc - you need py37-future or py27-future (or your equivalent of it) to get the module "builtins" (or what it was) with python2.7 (not sure about 3.7, but I used future).

ng0

2019-02-14 00:45

developer   ~0013737

According to the internet:

AM_PATH_PYTHON holds PYTHON as a 'precious variable'.

it can then be overridden with:
$ PYTHON=python2 ./configure

ng0

2019-02-14 00:47

developer   ~0013738

Tested in a make check, and it's not that obvious, needs more than 5 seconds of additional work.

Christian Grothoff

2019-02-16 15:57

manager   ~0013826

Ok, for me they still fail, right now like this:

Feb 16 15:51:54-272895 gnunet-daemon-topology-102811 WARNING `write' failed on file `/tmp/gnunet//c_bootstrap_server//.config/gnunet//topology/friends.txt' at friends.c:82 with error: No such file or directory
Feb 16 15:51:59-305976 gnunet-daemon-topology-102834 WARNING `write' failed on file `/tmp/gnunet//c_no_nat_client//.config/gnunet//topology/friends.txt' at friends.c:82 with error: No such file or directory
Check had timeout after 180 seconds
Peers did not connect
[plus stats output where clearly nothing worked]

Christian Grothoff

2019-02-16 15:59

manager   ~0013827

Looking at the peers _while_ the test is running shows that they _do_ connect, so probably the way the test case either grabs the statistics or checks for connectivity was broken.

Christian Grothoff

2019-02-19 02:02

manager   ~0013865

Integration tests pass again (at least for me).

Issue History

Date Modified Username Field Change
2019-02-12 12:29 ng0 New Issue
2019-02-12 12:30 ng0 Assigned To => ng0
2019-02-12 12:30 ng0 Status new => assigned
2019-02-12 14:43 ng0 Note Added: 0013684
2019-02-12 22:02 ng0 Note Added: 0013692
2019-02-12 22:23 ng0 Note Added: 0013693
2019-02-12 22:23 ng0 Note Edited: 0013693 View Revisions
2019-02-13 01:13 ng0 Target Version => 0.11.1
2019-02-13 01:16 ng0 Summary python 2 + 3 support for integration tests (and rest of python scripts) => python 2 + 3 support for integration tests (and rest of python scripts) (ie: python3.7 support)
2019-02-13 01:27 ng0 Note Added: 0013696
2019-02-13 02:01 ng0 Note Edited: 0013693 View Revisions
2019-02-13 02:07 ng0 Note Edited: 0013693 View Revisions
2019-02-13 02:21 ng0 Note Edited: 0013693 View Revisions
2019-02-13 02:31 ng0 Note Edited: 0013693 View Revisions
2019-02-13 15:23 ng0 Note Added: 0013699
2019-02-13 15:27 ng0 Target Version 0.11.1 => 0.11.0
2019-02-13 15:27 ng0 Summary python 2 + 3 support for integration tests (and rest of python scripts) (ie: python3.7 support) => python3.7 support for test suite
2019-02-13 15:44 ng0 Note Edited: 0013693 View Revisions
2019-02-13 15:50 ng0 Note Edited: 0013693 View Revisions
2019-02-13 15:52 ng0 Note Edited: 0013693 View Revisions
2019-02-13 15:54 ng0 Note Edited: 0013693 View Revisions
2019-02-13 15:56 ng0 Note Edited: 0013693 View Revisions
2019-02-13 17:27 ng0 Note Edited: 0013693 View Revisions
2019-02-13 17:41 ng0 Relationship added parent of 0005563
2019-02-13 17:42 ng0 Relationship replaced related to 0005563
2019-02-13 22:54 ng0 File Added: test-suite.log
2019-02-13 22:55 ng0 Assigned To ng0 => Christian Grothoff
2019-02-13 22:56 ng0 Note Added: 0013715
2019-02-13 22:56 Christian Grothoff Note Added: 0013716
2019-02-13 22:56 Christian Grothoff Assigned To Christian Grothoff =>
2019-02-13 22:56 Christian Grothoff Status assigned => confirmed
2019-02-13 22:56 ng0 Note Edited: 0013715 View Revisions
2019-02-13 22:57 Christian Grothoff Note Added: 0013717
2019-02-13 22:57 ng0 Note Added: 0013718
2019-02-13 23:00 ng0 Note Added: 0013719
2019-02-13 23:00 ng0 Note Edited: 0013719 View Revisions
2019-02-13 23:02 ng0 Note Edited: 0013719 View Revisions
2019-02-13 23:05 ng0 Note Edited: 0013719 View Revisions
2019-02-13 23:31 Christian Grothoff Note Added: 0013724
2019-02-13 23:43 Christian Grothoff Note Added: 0013725
2019-02-13 23:44 Christian Grothoff Note Added: 0013726
2019-02-13 23:44 Christian Grothoff Note Added: 0013727
2019-02-13 23:45 Christian Grothoff Note Added: 0013728
2019-02-13 23:51 Christian Grothoff Note Added: 0013729
2019-02-13 23:52 ng0 Note Added: 0013731
2019-02-14 00:01 Christian Grothoff Note Added: 0013735
2019-02-14 00:42 ng0 Note Added: 0013736
2019-02-14 00:45 ng0 Note Added: 0013737
2019-02-14 00:47 ng0 Note Added: 0013738
2019-02-16 15:57 Christian Grothoff Note Added: 0013826
2019-02-16 15:59 Christian Grothoff Note Added: 0013827
2019-02-17 22:37 Christian Grothoff Assigned To => Christian Grothoff
2019-02-17 22:37 Christian Grothoff Status confirmed => assigned
2019-02-17 22:38 Christian Grothoff Product Version => SVN HEAD
2019-02-19 02:02 Christian Grothoff Status assigned => resolved
2019-02-19 02:02 Christian Grothoff Resolution open => fixed
2019-02-19 02:02 Christian Grothoff Fixed in Version => 0.11.0
2019-02-19 02:02 Christian Grothoff Note Added: 0013865
2019-02-28 11:17 Christian Grothoff Status resolved => closed