View Issue Details

IDProjectCategoryView StatusLast Update
0005560GNUnetintegration testspublic2019-02-28 11:17
Reporternikita Assigned ToChristian Grothoff  
PrioritynormalSeverityminorReproducibilityhave not tried
Status closedResolutionfixed 
Product VersionGit master 
Target Version0.11.0Fixed in Version0.11.0 
Summary0005560: python3.7 support for test suite
DescriptionThe integration tests are currently failing on python3.7.
I will fix this (probably) by introducing a dependency on 'future' to make gnunet tests succeed with python2.7 and at least python3.7.
Tested will be only python3.7, since that's what gnunet picks up for a while now on my system (binary: python3.7).
TagsNo tags attached.

Relationships

related to 0005563 closednikita check for python3.7 for python related tests 

Activities

nikita

2019-02-12 14:43

developer   ~0013684

In my opinion we should put an emphasis on python3.7 here and consider everything else an option.
Switch to officially support python3.7.
The bytes/str fallacy in the integration-tests is taking some time since I'm solving this in a compatible way.

nikita

2019-02-12 22:02

developer   ~0013692

We have roughly 11 months left with official support for python2, I think it makes more sense to simply ignore python2 and have at the very least python3.4 support (I'm building with python3.7 already).
python2 won't go away, but I think it will be easier to support only one version of a language. No idea if I can make this in time for 0.11.0, I'll try

nikita

2019-02-12 22:23

developer   ~0013693

Last edited: 2019-02-13 17:27

View 12 revisions

I'm running tests with 3.7 now in all directories containing python files, updating this post later on:

* src/statistics passes with python3.7 (pkgsrc, netbsd current)
* src/fs passes with python3.7 (pkgsrc, netbsd current)
* src/arm passes the python test (3 tests pass, 1 fails) with python3.7 (pkgsrc, netbsd current)
* src/dht passes with python3.7 (pkgsrc, netbsd current), failure fixed in HEAD
* src/peertool-info passes with python3.7 (pkgsrc, netbsd current)
* src/revocation passes the python test (1 tests pass, 1 fails) with python3.7 (pkgsrc, netbsd current)
* src/consensus passes with python3.7 (pkgsrc, netbsd current)
* src/testbed: should be python3.7 ready, the only python file in there is rather version neutral (no lines which could throw off py2 or py3..py37)

TODO:
* integration-tests

nikita

2019-02-13 01:27

developer   ~0013696

runtime problems in moving to python3.7:
* gnunet-qr creates a permanent optional runtime dependency on python2.7 (see 0005562).

nikita

2019-02-13 15:23

developer   ~0013699

dht python test failure fixed in https://git.gnunet.org/gnunet.git/commit/?id=aaca8161989c5af3f46251cd3e8d5fb531a5cb01

nikita

2019-02-13 22:54

developer  

test-suite.log (29,042 bytes)   
==============================================================
   gnunet 0.11.0pre66: src/integration-tests/test-suite.log
==============================================================

# TOTAL: 6
# PASS:  0
# SKIP:  0
# XFAIL: 0
# FAIL:  6
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: test_integration_bootstrap_and_connect.py
===============================================

Feb 13 21:26:11-108421 util-service-12508 WARNING `bind' failed for `/tmp/gnunet-system-runtime//test-service-arm-4': address already in use
Feb 13 21:26:11-108679 arm-12508 ERROR `bind' failed at service.c:1845 with error: Address already in use
Feb 13 21:26:11-108710 util-service-12508 WARNING `bind' failed for port 20004 (IPv6): address already in use
Feb 13 21:26:11-108766 arm-12508 ERROR `bind' failed at service.c:1845 with error: Address already in use
Feb 13 21:26:11-108829 util-service-12508 WARNING `bind' failed for port 20004 (IPv4): address already in use
Feb 13 21:26:11-108998 arm-12508 ERROR `bind' failed at service.c:1845 with error: Address already in use
Feb 13 21:26:11-109013 arm-12508 ERROR Could not bind to any of the ports I was supposed to, refusing to run!
Feb 13 21:26:16-138378 util-service-6562 WARNING `bind' failed for `/tmp/gnunet-system-runtime//test-service-arm-18': address already in use
Feb 13 21:26:16-138601 arm-6562 ERROR `bind' failed at service.c:1845 with error: Address already in use
Feb 13 21:26:16-138632 util-service-6562 WARNING `bind' failed for port 20022 (IPv6): address already in use
Feb 13 21:26:16-138697 arm-6562 ERROR `bind' failed at service.c:1845 with error: Address already in use
Feb 13 21:26:16-138722 util-service-6562 WARNING `bind' failed for port 20022 (IPv4): address already in use
Feb 13 21:26:16-138784 arm-6562 ERROR `bind' failed at service.c:1845 with error: Address already in use
Feb 13 21:26:16-138797 arm-6562 ERROR Could not bind to any of the ports I was supposed to, refusing to run!
Check had timeout after 180 seconds
Peers did not connect
b'6DQC ./confs/c_no_nat_client.conf statistics condition in subsystem "transport   " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'6DQC ./confs/c_no_nat_client.conf statistics condition in subsystem "core        " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'6DQC ./confs/c_no_nat_client.conf statistics condition in subsystem "topology    " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'6DQC ./confs/c_no_nat_client.conf statistics condition in subsystem "dht         " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'6DQC ./confs/c_no_nat_client.conf statistics condition in subsystem "fs          " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'0AZH ./confs/c_bootstrap_server.conf statistics condition in subsystem "transport   " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'0AZH ./confs/c_bootstrap_server.conf statistics condition in subsystem "core        " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'0AZH ./confs/c_bootstrap_server.conf statistics condition in subsystem "topology    " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'0AZH ./confs/c_bootstrap_server.conf statistics condition in subsystem "dht         " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'0AZH ./confs/c_bootstrap_server.conf statistics condition in subsystem "fs          " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
0 out of 10 conditions fulfilled
Test failed
FAIL test_integration_bootstrap_and_connect.py (exit status: 1)

FAIL: test_integration_disconnect.py
====================================

Feb 13 21:29:34-405324 util-os-installation-60 WARNING `access' failed on file `/home/ng0/opt/lib//gnunet/libexec/gnunet-helper-nat-server' at os_installation.c:890 with error: No such file or directory
Feb 13 21:29:34-437532 transport-https_server-6409 ERROR No usable TLS certificate found and creating one at `/tmp/gnunet//c_bootstrap_server//.local/share/gnunet//transport/https.key//tmp/gnunet//c_bootstrap_server//.local/share/gnunet//transport/https.cert' failed!
Feb 13 21:29:34-437659 transport-https_server-6409 ERROR Could not load or create server certificate! Loading plugin failed!
Feb 13 21:29:34-438164 transport-6409 ERROR Failed to load transport plugin for `libgnunet_plugin_transport_https_server'
Feb 13 21:29:39-472342 arm-21867 WARNING Unable to bind listening socket for service `nat-auto' to address `/tmp/gnunet-system-runtime//gnunet-service-nat-auto.sock': Address already in use
Feb 13 21:29:39-472803 arm-21867 WARNING Unable to bind listening socket for service `nat' to address `/tmp/gnunet-system-runtime//gnunet-service-nat.sock': Address already in use
Feb 13 21:29:39-597233 transport-759 WARNING External protocol violation detected at plugin_transport_tcp.c:2977.
Feb 13 21:29:42-452193 nat-60 WARNING upnpc failed to create port mapping
Feb 13 21:29:42-479512 nat-60 WARNING upnpc failed to create port mapping
Feb 13 21:29:49-417432 nat-60 WARNING upnpc failed to create port mapping
Feb 13 21:29:49-457408 nat-60 WARNING upnpc failed to create port mapping
Feb 13 21:29:49-457558 nat-60 WARNING upnpc failed to create port mapping
Feb 13 21:29:49-457661 nat-60 WARNING upnpc failed to create port mapping
Feb 13 21:29:49-457795 nat-60 WARNING upnpc failed to create port mapping
Feb 13 21:29:49-457913 nat-60 WARNING upnpc failed to create port mapping
Feb 13 21:29:54-509634 nat-60 WARNING upnpc failed to create port mapping
Feb 13 21:29:54-509791 nat-60 WARNING upnpc failed to create port mapping
Feb 13 21:29:54-509927 nat-60 WARNING upnpc failed to create port mapping
Feb 13 21:29:54-510082 nat-60 WARNING upnpc failed to create port mapping
Feb 13 21:30:47-960421 core-3534 WARNING Ignoring duplicate EPHEMERAL_KEY from VSFK
Feb 13 21:32:57-715706 util-client-6409 WARNING Error during sending message of type 17
Starting peer using cfg ./confs/c_bootstrap_server.conf
Starting peer using cfg ./confs/c_no_nat_client.conf
Peers started, running check
Check had timeout after 180 seconds
Peers failed to connected!
b'6DQC ./confs/c_no_nat_client.conf statistics condition in subsystem "transport   " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'6DQC ./confs/c_no_nat_client.conf statistics condition in subsystem "core        " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'6DQC ./confs/c_no_nat_client.conf statistics condition in subsystem "topology    " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'6DQC ./confs/c_no_nat_client.conf statistics condition in subsystem "dht         " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'6DQC ./confs/c_no_nat_client.conf statistics condition in subsystem "fs          " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'0AZH ./confs/c_bootstrap_server.conf statistics condition in subsystem "transport   " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'0AZH ./confs/c_bootstrap_server.conf statistics condition in subsystem "core        " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'0AZH ./confs/c_bootstrap_server.conf statistics condition in subsystem "topology    " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'0AZH ./confs/c_bootstrap_server.conf statistics condition in subsystem "dht         " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'0AZH ./confs/c_bootstrap_server.conf statistics condition in subsystem "fs          " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
0 out of 10 conditions fulfilled
Stopping peer using cfg ./confs/c_bootstrap_server.conf
Stopping peer using cfg ./confs/c_no_nat_client.conf
Test failed
FAIL test_integration_disconnect.py (exit status: 1)

FAIL: test_integration_disconnect_nat.py
========================================

Feb 13 21:32:57-919666 util-os-installation-18184 WARNING `access' failed on file `/home/ng0/opt/lib//gnunet/libexec/gnunet-helper-nat-server' at os_installation.c:890 with error: No such file or directory
Feb 13 21:32:57-938845 transport-https_server-7824 ERROR No usable TLS certificate found and creating one at `/tmp/gnunet//c_bootstrap_server//.local/share/gnunet//transport/https.key//tmp/gnunet//c_bootstrap_server//.local/share/gnunet//transport/https.cert' failed!
Feb 13 21:32:57-938977 transport-https_server-7824 ERROR Could not load or create server certificate! Loading plugin failed!
Feb 13 21:32:57-939466 transport-7824 ERROR Failed to load transport plugin for `libgnunet_plugin_transport_https_server'
Feb 13 21:33:02-971446 util-service-25544 WARNING `bind' failed for `/tmp/gnunet-system-runtime//test-service-arm-46': address already in use
Feb 13 21:33:02-971612 arm-25544 ERROR `bind' failed at service.c:1845 with error: Address already in use
Feb 13 21:33:02-971634 util-service-25544 WARNING `bind' failed for port 20058 (IPv6): address already in use
Feb 13 21:33:02-971678 arm-25544 ERROR `bind' failed at service.c:1845 with error: Address already in use
Feb 13 21:33:02-971696 util-service-25544 WARNING `bind' failed for port 20058 (IPv4): address already in use
Feb 13 21:33:02-971737 arm-25544 ERROR `bind' failed at service.c:1845 with error: Address already in use
Feb 13 21:33:02-971745 arm-25544 ERROR Could not bind to any of the ports I was supposed to, refusing to run!
Feb 13 21:33:05-989818 nat-18184 WARNING upnpc failed to create port mapping
Feb 13 21:33:12-932807 nat-18184 WARNING upnpc failed to create port mapping
Feb 13 21:33:12-932948 nat-18184 WARNING upnpc failed to create port mapping
Feb 13 21:33:12-970971 nat-18184 WARNING upnpc failed to create port mapping
Feb 13 21:33:12-971250 nat-18184 WARNING upnpc failed to create port mapping
Feb 13 21:33:12-971740 nat-18184 WARNING upnpc failed to create port mapping
Feb 13 21:33:12-972381 nat-18184 WARNING upnpc failed to create port mapping
Feb 13 21:33:12-972584 nat-18184 WARNING upnpc failed to create port mapping
Feb 13 21:34:09-333907 core-4712 WARNING Ignoring duplicate EPHEMERAL_KEY from VSFK
Starting peer using cfg ./confs/c_bootstrap_server.conf
Starting peer using cfg ./confs/c_nat_client.conf
Peers started, running check
Check had timeout after 180 seconds
Peers failed to connected!
b'VSFK ./confs/c_nat_client.conf statistics condition in subsystem "transport   " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'VSFK ./confs/c_nat_client.conf statistics condition in subsystem "core        " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'VSFK ./confs/c_nat_client.conf statistics condition in subsystem "topology    " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'VSFK ./confs/c_nat_client.conf statistics condition in subsystem "dht         " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'VSFK ./confs/c_nat_client.conf statistics condition in subsystem "fs          " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'0AZH ./confs/c_bootstrap_server.conf statistics condition in subsystem "transport   " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'0AZH ./confs/c_bootstrap_server.conf statistics condition in subsystem "core        " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'0AZH ./confs/c_bootstrap_server.conf statistics condition in subsystem "topology    " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'0AZH ./confs/c_bootstrap_server.conf statistics condition in subsystem "dht         " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'0AZH ./confs/c_bootstrap_server.conf statistics condition in subsystem "fs          " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
0 out of 10 conditions fulfilled
Stopping peer using cfg ./confs/c_bootstrap_server.conf
Stopping peer using cfg ./confs/c_nat_client.conf
Test failed
FAIL test_integration_disconnect_nat.py (exit status: 1)

FAIL: test_integration_reconnect.py
===================================

Feb 13 21:36:20-974427 transport-https_server-27020 ERROR No usable TLS certificate found and creating one at `/tmp/gnunet//c_bootstrap_server//.local/share/gnunet//transport/https.key//tmp/gnunet//c_bootstrap_server//.local/share/gnunet//transport/https.cert' failed!
Feb 13 21:36:20-974573 transport-https_server-27020 ERROR Could not load or create server certificate! Loading plugin failed!
Feb 13 21:36:20-975055 transport-27020 ERROR Failed to load transport plugin for `libgnunet_plugin_transport_https_server'
Feb 13 21:36:20-985662 util-os-installation-17967 WARNING `access' failed on file `/home/ng0/opt/lib//gnunet/libexec/gnunet-helper-nat-server' at os_installation.c:890 with error: No such file or directory
Feb 13 21:36:21-007069 arm-27215 WARNING Unable to bind listening socket for service `nat-auto' to address `/tmp/gnunet-system-runtime//gnunet-service-nat-auto.sock': Address already in use
Feb 13 21:36:21-007363 arm-27215 WARNING Unable to bind listening socket for service `nat' to address `/tmp/gnunet-system-runtime//gnunet-service-nat.sock': Address already in use
Feb 13 21:36:21-071145 util-service-26766 WARNING `bind' failed for `/tmp/gnunet-system-runtime//test-service-arm-4': address already in use
Feb 13 21:36:21-071376 arm-26766 ERROR `bind' failed at service.c:1845 with error: Address already in use
Feb 13 21:36:21-071406 util-service-26766 WARNING `bind' failed for port 20004 (IPv6): address already in use
Feb 13 21:36:21-071471 arm-26766 ERROR `bind' failed at service.c:1845 with error: Address already in use
Feb 13 21:36:21-071496 util-service-26766 WARNING `bind' failed for port 20004 (IPv4): address already in use
Feb 13 21:36:21-071559 arm-26766 ERROR `bind' failed at service.c:1845 with error: Address already in use
Feb 13 21:36:21-071572 arm-26766 ERROR Could not bind to any of the ports I was supposed to, refusing to run!
Feb 13 21:36:26-136249 util-service-26153 WARNING `bind' failed for `/tmp/gnunet-system-runtime//test-service-arm-18': address already in use
Feb 13 21:36:26-136517 arm-26153 ERROR `bind' failed at service.c:1845 with error: Address already in use
Feb 13 21:36:26-136551 util-service-26153 WARNING `bind' failed for port 20022 (IPv6): address already in use
Feb 13 21:36:26-136627 arm-26153 ERROR `bind' failed at service.c:1845 with error: Address already in use
Feb 13 21:36:26-136656 util-service-26153 WARNING `bind' failed for port 20022 (IPv4): address already in use
Feb 13 21:36:26-136727 arm-26153 ERROR `bind' failed at service.c:1845 with error: Address already in use
Feb 13 21:36:26-136742 arm-26153 ERROR Could not bind to any of the ports I was supposed to, refusing to run!
Feb 13 21:36:35-992573 nat-17967 WARNING upnpc failed to create port mapping
Feb 13 21:36:35-992735 nat-17967 WARNING upnpc failed to create port mapping
Feb 13 21:36:35-992862 nat-17967 WARNING upnpc failed to create port mapping
Feb 13 21:36:35-992975 nat-17967 WARNING upnpc failed to create port mapping
Feb 13 21:36:35-993089 nat-17967 WARNING upnpc failed to create port mapping
Feb 13 21:36:35-993194 nat-17967 WARNING upnpc failed to create port mapping
Feb 13 21:36:36-062610 nat-17967 WARNING upnpc failed to create port mapping
Feb 13 21:36:36-062755 nat-17967 WARNING upnpc failed to create port mapping
Feb 13 21:36:36-062869 nat-17967 WARNING upnpc failed to create port mapping
Feb 13 21:36:36-062981 nat-17967 WARNING upnpc failed to create port mapping
Feb 13 21:39:36-544557 util-client-27020 WARNING Error during sending message of type 17
Removing /tmp/c_bootstrap_server
Removing /tmp/c_no_nat_client
Starting peer using cfg ./confs/c_bootstrap_server.conf
Starting peer using cfg ./confs/c_no_nat_client.conf
Starting peer using cfg ./confs/c_bootstrap_server.conf
Starting peer using cfg ./confs/c_no_nat_client.conf
Check had timeout after 180 seconds
Peers failed to connect
b'6DQC ./confs/c_no_nat_client.conf statistics condition in subsystem "transport   " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'6DQC ./confs/c_no_nat_client.conf statistics condition in subsystem "core        " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'6DQC ./confs/c_no_nat_client.conf statistics condition in subsystem "topology    " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'6DQC ./confs/c_no_nat_client.conf statistics condition in subsystem "fs          " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'0AZH ./confs/c_bootstrap_server.conf statistics condition in subsystem "transport   " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'0AZH ./confs/c_bootstrap_server.conf statistics condition in subsystem "core        " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'0AZH ./confs/c_bootstrap_server.conf statistics condition in subsystem "topology    " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'0AZH ./confs/c_bootstrap_server.conf statistics condition in subsystem "fs          " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
0 out of 8 conditions fulfilled
Stopping peer using cfg ./confs/c_bootstrap_server.conf
Stopping peer using cfg ./confs/c_no_nat_client.conf
Removing /tmp/c_bootstrap_server
Removing /tmp/c_no_nat_client
Test failed
FAIL test_integration_reconnect.py (exit status: 1)

FAIL: test_integration_reconnect_nat.py
=======================================

Feb 13 21:39:36-738073 transport-https_server-19976 ERROR No usable TLS certificate found and creating one at `/tmp/gnunet//c_bootstrap_server//.local/share/gnunet//transport/https.key//tmp/gnunet//c_bootstrap_server//.local/share/gnunet//transport/https.cert' failed!
Feb 13 21:39:36-738217 transport-https_server-19976 ERROR Could not load or create server certificate! Loading plugin failed!
Feb 13 21:39:36-738704 transport-19976 ERROR Failed to load transport plugin for `libgnunet_plugin_transport_https_server'
Feb 13 21:39:36-748834 util-os-installation-24797 WARNING `access' failed on file `/home/ng0/opt/lib//gnunet/libexec/gnunet-helper-nat-server' at os_installation.c:890 with error: No such file or directory
Feb 13 21:39:36-764981 arm-7481 WARNING Unable to bind listening socket for service `nat-auto' to address `/tmp/gnunet-system-runtime//gnunet-service-nat-auto.sock': Address already in use
Feb 13 21:39:36-765212 arm-7481 WARNING Unable to bind listening socket for service `nat' to address `/tmp/gnunet-system-runtime//gnunet-service-nat.sock': Address already in use
Feb 13 21:39:36-836547 transport-19976 WARNING External protocol violation detected at plugin_transport_tcp.c:2977.
Feb 13 21:39:36-841986 util-service-20365 WARNING `bind' failed for `/tmp/gnunet-system-runtime//test-service-arm-4': address already in use
Feb 13 21:39:36-842230 arm-20365 ERROR `bind' failed at service.c:1845 with error: Address already in use
Feb 13 21:39:36-842259 util-service-20365 WARNING `bind' failed for port 20004 (IPv6): address already in use
Feb 13 21:39:36-842319 arm-20365 ERROR `bind' failed at service.c:1845 with error: Address already in use
Feb 13 21:39:36-842343 util-service-20365 WARNING `bind' failed for port 20004 (IPv4): address already in use
Feb 13 21:39:36-842401 arm-20365 ERROR `bind' failed at service.c:1845 with error: Address already in use
Feb 13 21:39:36-842413 arm-20365 ERROR Could not bind to any of the ports I was supposed to, refusing to run!
Feb 13 21:39:41-900746 util-service-19742 WARNING `bind' failed for `/tmp/gnunet-system-runtime//test-service-arm-46': address already in use
Feb 13 21:39:41-900913 arm-19742 ERROR `bind' failed at service.c:1845 with error: Address already in use
Feb 13 21:39:41-900935 util-service-19742 WARNING `bind' failed for port 20058 (IPv6): address already in use
Feb 13 21:39:41-900979 arm-19742 ERROR `bind' failed at service.c:1845 with error: Address already in use
Feb 13 21:39:41-900998 util-service-19742 WARNING `bind' failed for port 20058 (IPv4): address already in use
Feb 13 21:39:41-901038 arm-19742 ERROR `bind' failed at service.c:1845 with error: Address already in use
Feb 13 21:39:41-901047 arm-19742 ERROR Could not bind to any of the ports I was supposed to, refusing to run!
Feb 13 21:39:44-793885 nat-24797 WARNING upnpc failed to create port mapping
Feb 13 21:39:51-766880 nat-24797 WARNING upnpc failed to create port mapping
Feb 13 21:39:51-767072 nat-24797 WARNING upnpc failed to create port mapping
Feb 13 21:39:51-767212 nat-24797 WARNING upnpc failed to create port mapping
Feb 13 21:39:51-767348 nat-24797 WARNING upnpc failed to create port mapping
Feb 13 21:39:51-767466 nat-24797 WARNING upnpc failed to create port mapping
Feb 13 21:39:51-826835 nat-24797 WARNING upnpc failed to create port mapping
Feb 13 21:39:51-826978 nat-24797 WARNING upnpc failed to create port mapping
Removing /tmp/c_bootstrap_server
Removing /tmp/c_nat_client
Starting peer using cfg ./confs/c_bootstrap_server.conf
Starting peer using cfg ./confs/c_nat_client.conf
Starting peer using cfg ./confs/c_bootstrap_server.conf
Starting peer using cfg ./confs/c_nat_client.conf
Check had timeout after 180 seconds
Peers failed to connect
b'VSFK ./confs/c_nat_client.conf statistics condition in subsystem "transport   " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'VSFK ./confs/c_nat_client.conf statistics condition in subsystem "core        " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'VSFK ./confs/c_nat_client.conf statistics condition in subsystem "topology    " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'VSFK ./confs/c_nat_client.conf statistics condition in subsystem "fs          " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'0AZH ./confs/c_bootstrap_server.conf statistics condition in subsystem "transport   " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'0AZH ./confs/c_bootstrap_server.conf statistics condition in subsystem "core        " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'0AZH ./confs/c_bootstrap_server.conf statistics condition in subsystem "topology    " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
b'0AZH ./confs/c_bootstrap_server.conf statistics condition in subsystem "fs          " : "# peers connected             " : (expected/real value) 1 != NaN FAIL!'
0 out of 8 conditions fulfilled
Stopping peer using cfg ./confs/c_bootstrap_server.conf
Stopping peer using cfg ./confs/c_nat_client.conf
Removing /tmp/c_bootstrap_server
Removing /tmp/c_nat_client
Test failed
FAIL test_integration_reconnect_nat.py (exit status: 1)

FAIL: test_integration_clique.py
================================

Feb 13 21:42:51-870720 transport-https_server-11873 ERROR No usable TLS certificate found and creating one at `/tmp/gnunet//c_bootstrap_server//.local/share/gnunet//transport/https.key//tmp/gnunet//c_bootstrap_server//.local/share/gnunet//transport/https.cert' failed!
Feb 13 21:42:51-870864 transport-https_server-11873 ERROR Could not load or create server certificate! Loading plugin failed!
Feb 13 21:42:51-871346 transport-11873 ERROR Failed to load transport plugin for `libgnunet_plugin_transport_https_server'
Feb 13 21:42:51-881478 util-os-installation-16368 WARNING `access' failed on file `/home/ng0/opt/lib//gnunet/libexec/gnunet-helper-nat-server' at os_installation.c:890 with error: No such file or directory
Feb 13 21:42:56-896337 arm-6217 WARNING Unable to bind listening socket for service `nat-auto' to address `/tmp/gnunet-system-runtime//gnunet-service-nat-auto.sock': Address already in use
Feb 13 21:42:56-896546 arm-6217 WARNING Unable to bind listening socket for service `nat' to address `/tmp/gnunet-system-runtime//gnunet-service-nat.sock': Address already in use
Feb 13 21:42:56-997111 transport-22509 WARNING External protocol violation detected at plugin_transport_tcp.c:2977.
Feb 13 21:43:01-957284 arm-12749 WARNING Unable to bind listening socket for service `nat-auto' to address `/tmp/gnunet-system-runtime//gnunet-service-nat-auto.sock': Address already in use
Feb 13 21:43:01-957578 arm-12749 WARNING Unable to bind listening socket for service `nat' to address `/tmp/gnunet-system-runtime//gnunet-service-nat.sock': Address already in use
Feb 13 21:43:02-013912 transport-11873 WARNING External protocol violation detected at plugin_transport_tcp.c:2977.
Feb 13 21:43:06-895977 nat-16368 WARNING upnpc failed to create port mapping
Feb 13 21:43:06-896166 nat-16368 WARNING upnpc failed to create port mapping
Feb 13 21:43:06-896326 nat-16368 WARNING upnpc failed to create port mapping
Feb 13 21:43:06-896476 nat-16368 WARNING upnpc failed to create port mapping
Feb 13 21:43:06-896613 nat-16368 WARNING upnpc failed to create port mapping
Feb 13 21:43:06-896745 nat-16368 WARNING upnpc failed to create port mapping
Feb 13 21:43:11-948135 nat-16368 WARNING upnpc failed to create port mapping
Feb 13 21:43:11-948362 nat-16368 WARNING upnpc failed to create port mapping
Feb 13 21:43:11-948543 nat-16368 WARNING upnpc failed to create port mapping
Feb 13 21:43:11-948684 nat-16368 WARNING upnpc failed to create port mapping
Feb 13 21:43:17-000273 nat-16368 WARNING upnpc failed to create port mapping
Feb 13 21:43:17-000406 nat-16368 WARNING upnpc failed to create port mapping
Removing /tmp/c_bootstrap_server
Removing /tmp/c_no_nat_client
Removing /tmp/c_nat_client
Starting peer using cfg ./confs/c_bootstrap_server.conf
Starting peer using cfg ./confs/c_no_nat_client.conf
Starting peer using cfg ./confs/c_nat_client.conf
Peers started, running check
Check had timeout after 180 seconds
b'6DQC ./confs/c_no_nat_client.conf statistics condition in subsystem "transport   " : "# peers connected             " : (expected/real value) 2 != NaN FAIL!'
b'6DQC ./confs/c_no_nat_client.conf statistics condition in subsystem "core        " : "# peers connected             " : (expected/real value) 2 != NaN FAIL!'
b'6DQC ./confs/c_no_nat_client.conf statistics condition in subsystem "topology    " : "# peers connected             " : (expected/real value) 2 != NaN FAIL!'
b'6DQC ./confs/c_no_nat_client.conf statistics condition in subsystem "dht         " : "# peers connected             " : (expected/real value) 2 != NaN FAIL!'
b'6DQC ./confs/c_no_nat_client.conf statistics condition in subsystem "fs          " : "# peers connected             " : (expected/real value) 2 != NaN FAIL!'
b'VSFK ./confs/c_nat_client.conf statistics condition in subsystem "transport   " : "# peers connected             " : (expected/real value) 2 != NaN FAIL!'
b'VSFK ./confs/c_nat_client.conf statistics condition in subsystem "core        " : "# peers connected             " : (expected/real value) 2 != NaN FAIL!'
b'VSFK ./confs/c_nat_client.conf statistics condition in subsystem "topology    " : "# peers connected             " : (expected/real value) 2 != NaN FAIL!'
b'VSFK ./confs/c_nat_client.conf statistics condition in subsystem "dht         " : "# peers connected             " : (expected/real value) 2 != NaN FAIL!'
b'VSFK ./confs/c_nat_client.conf statistics condition in subsystem "fs          " : "# peers connected             " : (expected/real value) 2 != NaN FAIL!'
b'0AZH ./confs/c_bootstrap_server.conf statistics condition in subsystem "transport   " : "# peers connected             " : (expected/real value) 2 != NaN FAIL!'
b'0AZH ./confs/c_bootstrap_server.conf statistics condition in subsystem "core        " : "# peers connected             " : (expected/real value) 2 != NaN FAIL!'
b'0AZH ./confs/c_bootstrap_server.conf statistics condition in subsystem "topology    " : "# peers connected             " : (expected/real value) 2 != NaN FAIL!'
b'0AZH ./confs/c_bootstrap_server.conf statistics condition in subsystem "dht         " : "# peers connected             " : (expected/real value) 2 != NaN FAIL!'
b'0AZH ./confs/c_bootstrap_server.conf statistics condition in subsystem "fs          " : "# peers connected             " : (expected/real value) 2 != NaN FAIL!'
0 out of 15 conditions fulfilled
Failed to connect clique
Stopping peer using cfg ./confs/c_bootstrap_server.conf
Stopping peer using cfg ./confs/c_no_nat_client.conf
Stopping peer using cfg ./confs/c_nat_client.conf
Removing /tmp/c_bootstrap_server
Removing /tmp/c_no_nat_client
Removing /tmp/c_nat_client
Test failed
FAIL test_integration_clique.py (exit status: 1)

test-suite.log (29,042 bytes)   

nikita

2019-02-13 22:56

developer   ~0013715

Last edited: 2019-02-13 22:56

View 2 revisions

Christian, Please see if/how I messed up the tests for integration-tests, with python3.7

Christian Grothoff

2019-02-13 22:56

manager   ~0013716

Why was this assigned to me?

Christian Grothoff

2019-02-13 22:57

manager   ~0013717

Ah, concurrency issue ;-). Didn't see your comment.

nikita

2019-02-13 22:57

developer   ~0013718

see my latest comment, with the addition of the log file. I'm simply not sure if I understood the tests right.

nikita

2019-02-13 23:00

developer   ~0013719

Last edited: 2019-02-13 23:05

View 4 revisions

The core of the python2 -> python3.7 migration issue here was the "res" variable in print of class StatisticsCondition producing a TypeError, where str can not be concatenated with byte


Tested it like this:
cd src/integration-tests
PATH=$PATH:/home/ng0/src/gnunet/gnunet:/home/ng0/opt/bin TMP=/tmp gmake check

which might be wrong but worked for most other tests.


Not assuming you kept up with python2/3 problems:
https://eli.thegreenplace.net/2012/01/30/the-bytesstr-dichotomy-in-python-3
https://python-future.org/compatible_idioms.html
showcases some of the differences.

Christian Grothoff

2019-02-13 23:31

manager   ~0013724

FAIL: test_integration_bootstrap_and_connect.py:

This log suggests you're giving the wrong config file to one of the peers, resulting in both trying the same ports, which is why it fails.

Christian Grothoff

2019-02-13 23:43

manager   ~0013725

FAIL: test_integration_disconnect.py:
Configuration was not good, used some plugins that should not be tested here, plus TCP plugin somehow used link local v6 addresses which are (in 0.11) not allowed by the validation routine causing the assertion. Maybe in combination this caused the peers to not connect; I've modified the configuration and TCP plugin to alleviate those specific issues.

Christian Grothoff

2019-02-13 23:44

manager   ~0013726

FAIL: test_integration_disconnect_nat.py:
Again, seems we don't use the right configuration files anymore, launching both peers on the same port(s).

Christian Grothoff

2019-02-13 23:44

manager   ~0013727

FAIL: test_integration_reconnect.py:
Again, seems we don't use the right configuration files anymore, launching both peers on the same port(s).

Christian Grothoff

2019-02-13 23:45

manager   ~0013728

FAIL: test_integration_clique.py:
Strangely, only the NAT service seems to be having a problem with shared ports. Maybe an issue in the configuration, the NAT service is relatively new.

Christian Grothoff

2019-02-13 23:51

manager   ~0013729

Indeed the NAT config was wrong, fixed in a2de1176e..9188dd31a

nikita

2019-02-13 23:52

developer   ~0013731

is the output in my original log wrong or causing problems? I did not expect the bytes ('b') to appear as a prefix to all lines.

Christian Grothoff

2019-02-14 00:01

manager   ~0013735

That 'b' is indeed not expected (and likely a minor bug), but I doubt it is the cause of the test failure.

nikita

2019-02-14 00:42

developer   ~0013736

for the record of the ticket - as written in irc - you need py37-future or py27-future (or your equivalent of it) to get the module "builtins" (or what it was) with python2.7 (not sure about 3.7, but I used future).

nikita

2019-02-14 00:45

developer   ~0013737

According to the internet:

AM_PATH_PYTHON holds PYTHON as a 'precious variable'.

it can then be overridden with:
$ PYTHON=python2 ./configure

nikita

2019-02-14 00:47

developer   ~0013738

Tested in a make check, and it's not that obvious, needs more than 5 seconds of additional work.

Christian Grothoff

2019-02-16 15:57

manager   ~0013826

Ok, for me they still fail, right now like this:

Feb 16 15:51:54-272895 gnunet-daemon-topology-102811 WARNING `write' failed on file `/tmp/gnunet//c_bootstrap_server//.config/gnunet//topology/friends.txt' at friends.c:82 with error: No such file or directory
Feb 16 15:51:59-305976 gnunet-daemon-topology-102834 WARNING `write' failed on file `/tmp/gnunet//c_no_nat_client//.config/gnunet//topology/friends.txt' at friends.c:82 with error: No such file or directory
Check had timeout after 180 seconds
Peers did not connect
[plus stats output where clearly nothing worked]

Christian Grothoff

2019-02-16 15:59

manager   ~0013827

Looking at the peers _while_ the test is running shows that they _do_ connect, so probably the way the test case either grabs the statistics or checks for connectivity was broken.

Christian Grothoff

2019-02-19 02:02

manager   ~0013865

Integration tests pass again (at least for me).

Issue History

Date Modified Username Field Change
2019-02-12 12:29 nikita New Issue
2019-02-12 12:30 nikita Assigned To => nikita
2019-02-12 12:30 nikita Status new => assigned
2019-02-12 14:43 nikita Note Added: 0013684
2019-02-12 22:02 nikita Note Added: 0013692
2019-02-12 22:23 nikita Note Added: 0013693
2019-02-12 22:23 nikita Note Edited: 0013693 View Revisions
2019-02-13 01:13 nikita Target Version => 0.11.1
2019-02-13 01:16 nikita Summary python 2 + 3 support for integration tests (and rest of python scripts) => python 2 + 3 support for integration tests (and rest of python scripts) (ie: python3.7 support)
2019-02-13 01:27 nikita Note Added: 0013696
2019-02-13 02:01 nikita Note Edited: 0013693 View Revisions
2019-02-13 02:07 nikita Note Edited: 0013693 View Revisions
2019-02-13 02:21 nikita Note Edited: 0013693 View Revisions
2019-02-13 02:31 nikita Note Edited: 0013693 View Revisions
2019-02-13 15:23 nikita Note Added: 0013699
2019-02-13 15:27 nikita Target Version 0.11.1 => 0.11.0
2019-02-13 15:27 nikita Summary python 2 + 3 support for integration tests (and rest of python scripts) (ie: python3.7 support) => python3.7 support for test suite
2019-02-13 15:44 nikita Note Edited: 0013693 View Revisions
2019-02-13 15:50 nikita Note Edited: 0013693 View Revisions
2019-02-13 15:52 nikita Note Edited: 0013693 View Revisions
2019-02-13 15:54 nikita Note Edited: 0013693 View Revisions
2019-02-13 15:56 nikita Note Edited: 0013693 View Revisions
2019-02-13 17:27 nikita Note Edited: 0013693 View Revisions
2019-02-13 17:41 nikita Relationship added parent of 0005563
2019-02-13 17:42 nikita Relationship replaced related to 0005563
2019-02-13 22:54 nikita File Added: test-suite.log
2019-02-13 22:55 nikita Assigned To nikita => Christian Grothoff
2019-02-13 22:56 nikita Note Added: 0013715
2019-02-13 22:56 Christian Grothoff Note Added: 0013716
2019-02-13 22:56 Christian Grothoff Assigned To Christian Grothoff =>
2019-02-13 22:56 Christian Grothoff Status assigned => confirmed
2019-02-13 22:56 nikita Note Edited: 0013715 View Revisions
2019-02-13 22:57 Christian Grothoff Note Added: 0013717
2019-02-13 22:57 nikita Note Added: 0013718
2019-02-13 23:00 nikita Note Added: 0013719
2019-02-13 23:00 nikita Note Edited: 0013719 View Revisions
2019-02-13 23:02 nikita Note Edited: 0013719 View Revisions
2019-02-13 23:05 nikita Note Edited: 0013719 View Revisions
2019-02-13 23:31 Christian Grothoff Note Added: 0013724
2019-02-13 23:43 Christian Grothoff Note Added: 0013725
2019-02-13 23:44 Christian Grothoff Note Added: 0013726
2019-02-13 23:44 Christian Grothoff Note Added: 0013727
2019-02-13 23:45 Christian Grothoff Note Added: 0013728
2019-02-13 23:51 Christian Grothoff Note Added: 0013729
2019-02-13 23:52 nikita Note Added: 0013731
2019-02-14 00:01 Christian Grothoff Note Added: 0013735
2019-02-14 00:42 nikita Note Added: 0013736
2019-02-14 00:45 nikita Note Added: 0013737
2019-02-14 00:47 nikita Note Added: 0013738
2019-02-16 15:57 Christian Grothoff Note Added: 0013826
2019-02-16 15:59 Christian Grothoff Note Added: 0013827
2019-02-17 22:37 Christian Grothoff Assigned To => Christian Grothoff
2019-02-17 22:37 Christian Grothoff Status confirmed => assigned
2019-02-17 22:38 Christian Grothoff Product Version => Git master
2019-02-19 02:02 Christian Grothoff Status assigned => resolved
2019-02-19 02:02 Christian Grothoff Resolution open => fixed
2019-02-19 02:02 Christian Grothoff Fixed in Version => 0.11.0
2019-02-19 02:02 Christian Grothoff Note Added: 0013865
2019-02-28 11:17 Christian Grothoff Status resolved => closed