View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0003684 | GNUnet | UDP transport | public | 2015-02-14 08:20 | 2018-06-07 00:25 |
| Reporter | ell | Assigned To | Christian Grothoff | ||
| Priority | high | Severity | minor | Reproducibility | have not tried |
| Status | closed | Resolution | fixed | ||
| OS | Xubuntu-14.04-64amd | ||||
| Product Version | Git master | ||||
| Target Version | 0.11.0pre66 | Fixed in Version | 0.11.0pre66 | ||
| Summary | 0003684: Revision 35254: Overnight run fills memory and swap completely | ||||
| Description | Running gnunet over night makes RAM and SWAP completely full. Killing with gnunet-arm -e results in a never ending process. The tail of gnunet.log is added. | ||||
| Tags | No tags attached. | ||||
| Attached Files | gnunet.log (93,121 bytes)
Feb 14 07:44:39-218053 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:39-218173 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:39-218459 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:39-218713 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:39-218948 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:39-278952 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:39-279530 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:39-340200 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:39-340340 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:39-340450 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:39-340562 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:39-340969 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:39-341110 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:39-341219 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:39-402881 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:39-404494 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:39-405267 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:39-478858 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:39-479145 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:39-479938 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:39-482020 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:39-482154 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:39-482273 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:39-482875 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:39-487910 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:39-488110 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:39-494002 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:39-494624 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:39-495137 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:39-495647 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:39-497611 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:39-497859 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:39-519064 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:39-519419 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:39-520040 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:39-520551 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:40-900427 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:44:41-067190 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:41-067601 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:41-077071 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:41-079989 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:41-080166 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:41-080239 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:41-080550 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:41-081540 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:41-259932 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:41-260258 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:41-260444 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:41-388287 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:41-388920 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:41-389048 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:41-389248 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:41-433054 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:41-433403 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:41-433561 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:41-520022 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:41-552265 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:41-701648 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:41-701799 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:42-220946 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:42-386692 transport-2734 WARNING Failed to send PING to `4YCN' at `udp.0.[1234::1]:2086' Feb 14 07:44:42-388799 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:42-456998 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:42-534799 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:42-592965 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:42-593438 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:42-593925 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:42-594377 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:42-594689 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:42-595115 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:43-237871 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:43-303590 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:43-304028 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:43-304641 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:43-305082 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:43-337284 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:44:43-357719 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:44-452899 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:44:44-817418 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:44:44-827649 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:44:44-828781 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:44:45-079455 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:45-080078 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:45-080223 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:45-080367 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:45-186363 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:45-246191 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:45-246837 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:45-251558 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:45-252207 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:45-252425 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:45-323762 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:45-323955 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:45-434182 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:45-435772 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:45-663945 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:45-972234 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:46-310091 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:46-397925 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:46-501099 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:44:46-501249 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:44:46-526753 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:44:46-695842 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:46-695966 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:46-726176 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:44:46-893773 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:46-894362 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:46-923603 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:46-999027 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:47-000144 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:47-000447 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:47-571749 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:44:47-572578 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:47-572703 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:47-572776 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:47-807514 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:47-818194 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:44:47-818869 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:44:48-478752 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:48-478901 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:48-602666 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:48-603003 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:48-603946 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:48-605448 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:48-605903 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:48-612776 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:48-619127 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:48-622641 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:48-903767 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:49-111120 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:44:49-124984 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:49-127108 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:44:49-133864 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:44:49-197172 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:44:49-339215 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:44:49-670283 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:49-697018 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:49-697248 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:50-092978 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:50-163137 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:50-194085 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:50-300508 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:50-301368 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:50-301811 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:50-499463 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:44:50-501219 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:44:50-518178 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:44:50-539749 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:44:50-613626 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:50-704355 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:50-704675 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:50-931374 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:50-933530 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:44:50-934855 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:44:50-976781 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:51-027966 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:44:51-103673 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:51-103989 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:51-265640 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:51-265761 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:51-399531 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:51-399891 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:51-808929 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:51-884581 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:52-223969 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:52-403430 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:52-444990 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:52-570315 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:52-626282 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:52-626425 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:52-626513 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:53-068926 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:44:53-086559 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:44:53-107224 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:53-271007 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:53-338152 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:53-710956 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:53-711115 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:53-711803 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:53-719591 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:54-306930 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:54-316501 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:54-489629 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:54-489932 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:54-831136 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:54-922520 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:54-931254 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:55-006557 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:55-616298 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:55-688539 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:55-931391 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:55-931572 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:55-931715 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:56-406801 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:56-437558 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:56-437676 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:56-745413 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:56-945285 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:57-112092 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:44:58-240351 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:58-262778 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:58-668125 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:58-753236 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:58-849214 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:44:59-098789 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:59-365116 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:59-523036 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:59-833211 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:59-833366 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:59-833578 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:59-833656 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:59-900442 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:44:59-970029 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:00-385700 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:45:00-385810 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:45:00-397275 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:00-552683 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:01-132878 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:45:01-132972 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:45:01-282076 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:01-356289 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:02-355874 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:02-356092 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:02-356310 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:02-572011 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:02-932084 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:45:05-507412 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:45:06-112099 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:06-800501 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:45:08-116491 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:08-308656 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:08-458114 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:45:08-813251 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:08-813429 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:08-813522 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:09-019905 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:09-034061 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:09-034253 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:10-692099 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:10-853468 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:10-853593 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:11-225724 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:11-995707 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:45:12-630258 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:13-002803 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:14-653795 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:14-743620 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:15-520170 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:15-520289 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:15-520850 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:15-520934 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:16-039179 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:16-039464 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:16-039521 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:16-205721 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:16-326577 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:16-610848 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:16-612941 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:16-613391 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:16-617333 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:17-512554 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:17-515027 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:17-555971 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:17-693022 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:17-886027 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:45:18-013478 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:18-365058 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:19-194372 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:45:19-631005 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:20-115354 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:45:21-082988 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:45:21-083053 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:45:21-604412 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:45:22-791410 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:22-860227 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:22-883429 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:23-004748 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:23-014926 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:23-026669 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:23-233695 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:23-233774 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:23-247666 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:23-261882 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:23-855742 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:45:24-393489 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:24-515392 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:24-515520 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:25-218857 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:45:26-281115 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:45:26-463881 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:27-405442 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:27-428246 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:27-468878 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:45:28-136702 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:45:28-183789 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:28-183892 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:28-338783 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:28-458770 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:28-458865 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:28-608161 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:45:28-608266 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:45:28-773282 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:28-773626 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:28-774091 cadet-2726 WARNING Message `External protocol violation detected at gnunet-service-cadet_tunnel.c:1947.' repeated 1001 times in the last 16 m Feb 14 07:45:29-031478 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:29-031573 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:29-031626 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:29-144428 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:45:29-240608 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:45:29-282565 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:29-304077 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:29-453657 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:45:29-559198 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:29-587392 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:29-587922 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:29-588424 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:29-589016 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:30-445002 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:30-445123 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:32-790732 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:45:32-986285 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:32-986445 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:32-986583 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:32-986662 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:33-009306 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:33-588006 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:33-588146 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:35-248176 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:35-329259 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:36-114542 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:45:36-467517 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:36-534171 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:36-534271 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:36-882563 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:37-483783 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:37-559208 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:37-559548 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:37-798170 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:37-798280 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:37-884188 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:38-237923 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:38-313385 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:38-706391 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:38-787067 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:38-842281 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:45:38-992838 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:38-993105 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:39-091199 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:45:39-093691 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:39-231508 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:39-358066 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:39-371816 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:39-506224 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:39-542163 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:39-542441 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:39-542646 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:39-610186 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:39-676926 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:39-912573 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:39-912706 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:39-912780 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:39-912871 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:39-920250 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:39-920399 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:39-920495 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:39-920567 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:40-118495 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:40-118642 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:40-118715 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:40-170836 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:40-171347 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:40-227693 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:40-228099 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:40-228400 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:40-228687 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:40-229001 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:40-830558 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:40-830728 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:40-830803 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:40-830878 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:40-830956 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:40-901094 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:45:40-904082 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:40-906597 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:40-926550 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:41-047183 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:41-047444 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:41-047787 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:41-047916 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:41-097169 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:41-143819 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:41-173220 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:41-178523 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:41-178704 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:41-178778 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:41-192095 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:41-193011 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:41-298997 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:41-307721 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:41-307816 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:41-307867 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:41-525571 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:41-525971 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:41-526169 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:41-567626 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:41-617350 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:41-842395 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:41-842491 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:41-842552 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:41-860083 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:41-948228 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:41-948568 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:41-948713 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:42-223447 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:42-292037 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:42-292134 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:42-292218 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:42-292306 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:42-642053 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:43-200141 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:43-200312 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:43-200419 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:43-272164 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:43-337812 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:45:43-597309 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:44-453864 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:45:44-548332 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:44-548465 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:44-548539 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:44-817511 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:44-827985 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:45:44-828075 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:45:44-829143 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:45:45-016379 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:45-016495 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:45-016642 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:45-155953 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:45-157314 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:45-220883 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:45-590434 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:45-635266 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:45-641643 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:46-283534 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:46-386457 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:46-386637 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:46-386782 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:46-386920 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:46-419882 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:46-420022 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:46-527043 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:45:46-726674 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:45:47-137235 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:47-572124 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:45:47-818675 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:45:47-819205 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:45:48-112553 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:48-219885 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:48-220057 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:48-220164 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:48-220305 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:49-111947 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:45:49-127612 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:45:49-197578 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:45:49-339739 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:45:49-969899 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:50-261755 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:50-499868 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:45:50-507166 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:50-507271 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:50-518465 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:45:50-540511 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:45:50-893366 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:50-934766 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:45:50-935252 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:45:51-028270 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:45:52-066425 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:52-066556 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:52-107639 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:52-277774 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:52-560200 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:52-560469 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:52-628951 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:53-069277 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:45:53-086817 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:45:53-240099 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:53-241041 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:53-241136 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:53-241209 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:53-241280 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:53-280867 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:53-280998 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:53-735637 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:54-276006 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:54-929550 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:55-366495 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:55-366679 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:55-366751 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:55-366879 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:55-449667 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:55-770120 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:55-770258 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:56-181551 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:56-181811 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:56-280289 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:56-353044 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:56-384285 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:56-645841 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:56-885117 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:57-114698 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:45:58-849641 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:45:59-002757 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:59-002976 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:59-003051 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:59-003126 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:59-100994 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:59-437919 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:45:59-692164 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:45:59-803228 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:00-386739 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:46:00-386834 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:46:00-572797 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:00-644373 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:00-924065 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:01-133536 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:46:01-455792 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:01-884212 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:02-071120 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:02-071537 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:02-271666 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:02-397183 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:02-397361 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:02-397923 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:02-719491 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:02-733648 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:02-912952 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:02-932548 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:46:03-065836 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:03-066011 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:03-504098 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:03-609171 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:04-260568 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:04-340796 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:04-341049 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:04-344095 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:04-422048 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:04-480881 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:04-595060 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:04-635219 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:04-635706 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:04-642796 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:04-800174 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:05-237892 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:05-362529 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:05-427596 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:05-507639 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:46:05-563642 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:05-632776 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:05-633296 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:05-633416 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:05-633545 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:05-707750 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:05-849267 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:05-849412 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:05-849518 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:05-914445 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:06-156262 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:06-161575 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:46:06-443064 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:06-648576 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:06-648777 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:06-800851 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:46:07-138685 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:07-195679 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:07-403702 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:08-156948 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:08-252351 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:08-252473 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:08-295192 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:08-358350 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:08-358546 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:08-460101 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:46:08-638081 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:08-638448 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:08-780374 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:08-780490 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:08-819372 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:46:09-411983 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:46:09-436315 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:46:09-851210 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:09-906175 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:09-906346 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:10-055787 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:10-055965 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:10-391808 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:46:10-400875 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:46:10-691797 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:10-692058 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:10-692249 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:11-617237 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:11-914622 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:11-996167 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:46:12-380358 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:12-430709 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:12-564088 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:12-872429 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:13-078030 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:13-618374 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:14-391737 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:14-391870 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:14-460274 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:14-460423 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:14-633327 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:14-633479 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:14-826192 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:46:16-059002 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:46:16-060693 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:46:17-522905 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:46:18-014070 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:18-321686 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:18-321885 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:18-321942 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:18-365374 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:18-967680 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:18-967816 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:18-967893 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:19-062443 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:19-194715 fs-2731 WARNING Datastore lookup already took 4 m! Feb 14 07:46:19-266891 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:19-631709 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:19-832936 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:19-833026 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:19-833075 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:19-833121 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:19-834531 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:19-834626 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:19-834800 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:20-031669 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:20-115453 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:20-115685 fs-2731 WARNING Datastore lookup already took 4 m! Feb 14 07:46:20-115733 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:20-354249 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:20-446015 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:20-446468 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:20-446533 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:21-083205 fs-2731 WARNING Datastore lookup already took 4 m! Feb 14 07:46:21-083358 fs-2731 WARNING Datastore lookup already took 4 m! Feb 14 07:46:21-136770 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:21-136864 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:21-136917 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:21-260116 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:21-604636 fs-2731 WARNING Datastore lookup already took 4 m! Feb 14 07:46:22-066420 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:22-142371 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:22-786840 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:22-795368 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:22-860453 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:22-883742 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:23-005168 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:23-015189 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:23-026964 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:23-077658 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:23-233979 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:23-234185 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:23-262234 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:23-436766 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:23-499229 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:23-714743 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:23-859794 fs-2731 WARNING Datastore lookup already took 4 m! Feb 14 07:46:23-992366 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:23-992505 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:23-997192 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:23-997357 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:24-975807 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:25-192127 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:25-220084 fs-2731 WARNING Datastore lookup already took 4 m! Feb 14 07:46:25-256096 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:25-960934 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:25-961083 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:25-961157 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:26-282802 fs-2731 WARNING Datastore lookup already took 4 m! Feb 14 07:46:26-359925 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:26-467786 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:26-501495 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:26-873995 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:26-874179 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:27-147047 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:27-281955 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:27-471957 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:46:28-036327 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:28-137576 fs-2731 WARNING Datastore lookup already took 4 m! Feb 14 07:46:28-336882 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:28-479254 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:46:28-609299 fs-2731 WARNING Datastore lookup already took 4 m! Feb 14 07:46:28-609420 fs-2731 WARNING Datastore lookup already took 4 m! Feb 14 07:46:29-146022 fs-2731 WARNING Datastore lookup already took 4 m! Feb 14 07:46:29-243403 fs-2731 WARNING Datastore lookup already took 4 m! Feb 14 07:46:29-287418 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:46:29-319873 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:46:29-327426 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:46:29-405390 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:29-405533 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:29-405606 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:29-455400 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:46:29-490429 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:29-593657 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:46:29-595402 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:46:29-595490 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:46:29-690882 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:29-978498 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:46:29-978609 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:46:30-230698 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:30-230796 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:30-230849 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:30-446359 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:30-446488 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:30-849325 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:30-849695 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:30-849903 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:31-180423 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:31-180562 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:31-180656 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:31-948916 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:31-949060 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:32-268943 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:32-270314 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:32-799850 fs-2731 WARNING Datastore lookup already took 4 m! Feb 14 07:46:33-036773 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:33-036895 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:33-103341 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:33-235367 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:33-235719 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:34-179198 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:34-179546 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:34-179772 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:34-179958 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:34-180141 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:34-247696 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:34-393578 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:35-149790 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:35-151878 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:35-651247 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:35-651571 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:35-651829 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:35-932283 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:35-932402 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:35-932929 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:35-933239 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:36-004251 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:36-063560 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:36-063696 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:36-118012 fs-2731 WARNING Datastore lookup already took 4 m! Feb 14 07:46:36-428818 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:36-469276 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:36-548132 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:36-661873 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:36-663126 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:36-890462 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:37-636414 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:37-636549 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:37-799525 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:37-799704 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:37-873340 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:37-873698 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:37-911890 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:37-912899 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:37-928390 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:38-302239 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:38-302399 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:38-302633 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:38-302896 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:38-789145 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:38-795588 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:38-795729 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:38-847907 fs-2731 WARNING Datastore lookup already took 4 m! Feb 14 07:46:39-098112 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:39-098261 fs-2731 WARNING Datastore lookup already took 4 m! Feb 14 07:46:39-292230 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:39-359370 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:39-372162 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:39-680000 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:39-766558 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:39-838523 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:39-838767 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:39-899661 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:39-899825 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:40-095897 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:46:40-096018 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:46:40-101833 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:40-235024 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:46:40-545259 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:40-545417 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:40-866364 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:40-911437 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:40-911525 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:41-147507 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:41-175668 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:41-192442 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:41-299361 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:41-570681 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:41-619547 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:41-949767 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:42-033893 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:42-401368 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:42-492156 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:42-492305 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:42-648433 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:46:42-740713 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:43-140290 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:43-170317 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:46:43-275643 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:43-333347 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:43-499535 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:46:43-603138 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:46:43-610450 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:46:44-149210 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:44-262106 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:44-354956 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:44-459681 fs-2731 WARNING Datastore lookup already took 4 m! Feb 14 07:46:44-490332 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:44-806244 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:44-819619 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:44-831590 fs-2731 WARNING Datastore lookup already took 4 m! Feb 14 07:46:44-831698 fs-2731 WARNING Datastore lookup already took 4 m! Feb 14 07:46:44-831755 fs-2731 WARNING Datastore lookup already took 4 m! Feb 14 07:46:44-939595 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:46:45-006697 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:45-018191 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:45-018259 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:45-156311 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:45-233650 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:46:45-591395 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:45-607808 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:45-642732 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:45-674143 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:45-922407 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:46-052817 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:46-052996 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:46-285834 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:46-433572 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:46:46-727116 fs-2731 WARNING Datastore lookup already took 4 m! Feb 14 07:46:47-118228 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:47-549984 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:47-561243 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:47-600043 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:46:47-795284 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:47-795468 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:47-835692 fs-2731 WARNING Datastore lookup already took 4 m! Feb 14 07:46:47-864528 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:48-612728 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:48-670126 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:48-694962 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:48-737698 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:49-131896 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:46:49-199915 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:46:49-298013 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:49-298150 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:49-372820 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:49-611597 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:49-748012 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:49-899572 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:49-968714 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:49-971387 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:50-035578 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:50-119344 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:50-179692 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:50-198597 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:50-500558 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:46:50-519878 fs-2731 WARNING Datastore lookup already took 4 m! Feb 14 07:46:50-829349 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:50-895584 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:50-915556 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:50-936404 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:46:51-031559 fs-2731 WARNING Datastore lookup already took 3 m! Feb 14 07:46:51-299985 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:52-630470 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:53-075086 fs-2731 WARNING Datastore lookup already took 4 m! Feb 14 07:46:53-281120 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:53-281265 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:53-481906 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:53-560352 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:53-560497 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:53-737999 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:53-832944 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:53-898780 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:53-902871 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:54-279550 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:54-929823 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:55-488974 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:55-551486 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:55-833347 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:55-967896 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:56-073297 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:56-198391 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:46:57-101336 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:58-150469 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:46:58-350727 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:58-414446 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:58-495436 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:58-561021 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:58-851913 fs-2731 WARNING Datastore lookup already took 4 m! Feb 14 07:46:59-103538 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:46:59-211777 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:59-692256 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:59-731029 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:46:59-798502 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:00-387103 fs-2731 WARNING Datastore lookup already took 4 m! Feb 14 07:47:00-387209 fs-2731 WARNING Datastore lookup already took 4 m! Feb 14 07:47:00-498875 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:00-615996 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:00-757720 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:00-782924 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:00-791349 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:01-136818 fs-2731 WARNING Datastore lookup already took 4 m! Feb 14 07:47:01-306846 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:01-306976 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:01-348943 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:01-349078 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:01-349448 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:01-407905 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:01-484619 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:01-513099 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:01-535256 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:01-550108 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:01-598410 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:01-745548 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:01-826602 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:01-885231 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:01-983042 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:02-041203 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:02-110404 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:02-727908 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:47:02-932980 fs-2731 WARNING Datastore lookup already took 4 m! Feb 14 07:47:03-019321 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:03-095606 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:03-441054 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:03-504815 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:47:03-540047 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:03-603858 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:03-647381 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:03-885203 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:04-272825 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:04-272989 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:04-273095 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:04-273201 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:04-855678 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:04-855851 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:06-168251 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:47:06-950836 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:06-951000 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:07-183671 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:07-252507 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:07-253106 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:07-253194 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:07-331980 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:07-332119 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:07-394097 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:07-694445 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:07-766312 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:07-766439 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:07-838359 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:07-838793 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:07-839083 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:08-230489 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:08-230784 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:08-230936 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:08-359053 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:08-359176 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:08-436467 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:08-436596 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:08-461112 fs-2731 WARNING Datastore lookup already took 4 m! Feb 14 07:47:08-568223 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:08-653728 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:08-819731 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:47:08-883827 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:08-985397 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:09-309325 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:09-914116 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:47:09-918083 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:10-401540 fs-2731 WARNING Datastore lookup already took 2 m! Feb 14 07:47:10-791053 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:11-992882 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:11-996410 fs-2731 WARNING Datastore lookup already took 4 m! Feb 14 07:47:12-032871 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:12-093584 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:12-167393 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:12-425674 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:13-081186 fs-2731 WARNING Datastore lookup already took 1 m! Feb 14 07:47:15-274187 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:15-547823 ats-connectivity-api-2734 WARNING ATS connection died (code 1), reconnecting Feb 14 07:47:15-602935 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:15-635865 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:15-636001 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:15-771087 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:15-793194 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:47:15-793263 transport-2734 ERROR Assertion failed at plugin_transport_http_server.c:1666. Feb 14 07:48:28-497139 datastore-2738 WARNING Could not save some persistent statistics gnunet@Xubu-14:~$ gnunet-arm -e leak.log (1,119 bytes)
==16882== Memcheck, a memory error detector ==16882== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al. ==16882== Using Valgrind-3.10.0.SVN and LibVEX; rerun with -h for copyright info ==16882== Command: /usr/local/lib//gnunet/libexec/gnunet-service-transport -c /etc/gnunet.conf ==16882== Parent PID: 16868 ==16882== ==16882== ==16882== HEAP SUMMARY: ==16882== in use at exit: 5,019 bytes in 31 blocks ==16882== total heap usage: 13,818,794 allocs, 13,818,763 frees, 2,810,614,721 bytes allocated ==16882== ==16882== LEAK SUMMARY: ==16882== definitely lost: 0 bytes in 0 blocks ==16882== indirectly lost: 0 bytes in 0 blocks ==16882== possibly lost: 0 bytes in 0 blocks ==16882== still reachable: 5,019 bytes in 31 blocks ==16882== suppressed: 0 bytes in 0 blocks ==16882== Reachable blocks (those to which a pointer was found) are not shown. ==16882== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==16882== ==16882== For counts of detected and suppressed errors, rerun with: -v ==16882== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0) | ||||
| related to | 0003687 | closed | Christian Grothoff | WARNING `accept' failed at connection.c:345 with error: Resource temporarily unavailable |
| related to | 0003695 | closed | Christian Grothoff | UDP reports multiple session instances to same peer |
|
|
Looks like to be an issue of gnunet-gtk. --No, it is not an issue of gnunet-gtk, it seems to be an issue of gnunet-service-transport as admin-tools indicate. It is "gnunet-service-transport" that fills in a strictly monoton increasing way memory and swap and never releases any data. So I can not leave the machine really unguided. To get rid of this data, I have to restart gnunet, and refilling restarts also again. |
|
|
Ok, sounds like we have a major memory leak. Setting [transport] PREFIX = valgrind --leak-check=yes --log-file=/tmp/leak.log in the configuration file should cause memory leaks to be logged to '/tmp/leak.log' -- but only on 'regular' exit. So running a peer for a few hours with this setting and then grabbing /tmp/leak.log should help speed up our diagnostics. (I'd do so right now, but I'm not at work this week.) |
|
|
leak.log after 1/2 hour run is uploaded. The strictly monoton memory consumption of memcheck-amd64- (=disguised gnunet-service-transport) started at approximately 173 972 K and came to about 729 300 K when gnunet was stopped via gnunet-arm -e. If wanted, I can upload gnunet.log of this gnunet-run. |
|
|
Sorry, no leaks after 1/2h. Now, the increase to just 730k may even be expected, as it is conceivable that the peer creates more connections and thus needs more buffer space for those connections. 600k for buffers is not _that_ much. OTOH, it is odd that after 1/2h a major leak does not even show. One possible explanation is typically that it is not a classical leak but memory that is freed on exit, just allocated, used, and then not used for a long time before being finally freed. Such a leak is much harder to find, but might be identified using PREFIX = valgrind --tool=massif --massif-out-file=/tmp/massif.out --alloc-fn=GNUNET_malloc However, the output of this tool is likely a rather large file and we might have to play some more with the options to get those right. So maybe Matthias can try this? |
|
|
Additional observations: 1.) The incremental chunks seem to be of size 264 K (or integer multiples of 264 K) 2.) I have seen datastore-warnings from 1 m to 10 m the like: fs-16875 WARNING Datastore lookup already took 1 m! Can this warning be connected to the problem? I will test: PREFIX = valgrind --tool=massif --massif-out-file=/tmp/massif.out --alloc-fn=GNUNET_malloc ++++No massif.out-file is created. Sorry, I was too impatient, I retry. |
|
|
Oh, wait, did you say datastore??? In that case the valgrind commands must be in [datastore] instead of [transport]. (And I need glasses to read more carefully.) |
|
|
No glasses needed. massif.out is uploaded. Point 2.) was an idea, maybe it could be connected to the problem. Should I test [datastore] with valgrind or is this not needed? |
|
|
Well, if datastore is the process that uses too much memory, then yes, we should analyze that one ;-) (first with the leak checker...). |
|
|
The consuming process is definitely gnunet-service-transport. |
|
|
Ok, I fixed one particular leak in transport. If this still happens, one question I have is how much memory your system has, and how much bandwidth you have configured for GNUnet (as more bandwidth = higher expected memory consumption). |
|
|
Really sorry, but the problem still exists. Gnunet-service-transport consumes very slowly but continously and does never free the consumed memory, so after some hours memory and swap are completely full. The only way for me to free memory and swap is stop/start gnunet. Configured memory is 4 GiB and configured swap is also 4 GiB. Filling of memory by gnunet starts approximately at 0.8 GiB memory. Bandwidth is not expressly configured for gnunet, I let it work with the built-in standards of gnunet. What I see in watching network-history is that receiving signals seldom exceed 140 KiB/s and that my sending signals in average are about 200 KiB/s, peaks may come up to 500 KiB/s and very seldom peaks go up to 1000 KiB/s, so it would not be my favorite idea that it is a network problem. |
|
|
Ok, let's look further than. Which transport plugins do you have active (in section [transport], the option PLUGINS)? |
|
|
In /etc/gnunet.conf: ... [transport] PLUGINS = tcp udp http_client https_client http_server https_server ... |
|
|
Try: [transport] NEIGHBOUR_LIMIT = 5 And see if that caps memory usage. |
|
|
Amatus: I'd rather see him try with/without udp and/or http server/client. I suspect it's one of the plugins. |
|
|
Test result of: [transport] NEIGHBOUR_LIMIT = 5 Test was running from about 19:00 to 23:30 and filled nearly the complete memory. On monday I will test switching off udp, http_server, http_client. Now I go for sleep and dreams... |
|
|
svn up ->35298 Yes, the bug is alive in 35298. Test no udp configured: "PLUGINS = tcp http_client https_client http_server https_server" Result: Sequence of memory consumption begins at 17976 K comes up to 19400 K and then comes down to 18672 K and again up to 19400 K and down to 19356 K. Well, looks like the bug lives somewhere in udp. Also seen: lots of "transport-11133 ERROR Assertion failed at plugin_transport_http_server.c:1666" but unclear if connected to this bug. |
|
|
I am running a peer for server quite some days with: [transport] PLUGINS = tcp https_client http_client udp [nat] BEHIND_NAT=YES Peer is connected to several other (3-6) peers (core) I have attached some stats showing the difference between 4 days of running Transport memory consumption is slightly increasing from 110 to 117 MB ... but not thrashing the system... Interesting is that ATS is running with 100% CPU load ... see related bug 0003689 |
|
|
I reactivated udp: [transport] PLUGINS = tcp http_client https_client http_server https_server udp and again I see the strictly monotone memory consuming. I can observe connections using tcp, http_server, https_server to me. I do not observe udp- and *-client-connections. CPU-load of gnunet-service-ats is about 1%. Most cpu-load is for datastore, fs, gtk. Ok, you can not verify the problem. I can do tests for you, assuming I am equipped with the necessary test utensils and test directives. |
|
|
I found and fixed one (minor) leak in SVN 35330. But didn't seem big enough to justify running out of memory overnight. |
|
|
Maybe you can get some helpful info via the uploaded video. Upload seems not to work, so use http://srv1.nerdcamp.net/html/Bildschirmaufnahme_2015-03-02_10-06-19.mp4 |
|
|
I've been running a peer now for quite some time with TCP + UDP, and I don't see any significant memory utilization by transport. FS takes like 250 MB, but the rest is virtually invisible. Please let me know if you still get this with SVN HEAD. |
|
|
Two news, the good one first: Trashing memory and swap is gone with svn 35363 ( disappeared already in 35339) The bad news: udp-connections shown by peerinfo-gtk in gnunet-gtk are not displayed in the same way as tcp, http, https. udp-connections are displayed under 64TF-item, contrasting to tcp-connections listed not under 64TF-peer. |
|
|
Ok, thanks for the confirmation. As for the gnunet-peerinfo-gtk issue, I don't understand from what you write, but please simply file a separate bug and maybe attach a screenshot for illustration ;-). |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2015-02-14 08:20 | ell | New Issue | |
| 2015-02-14 08:20 | ell | File Added: gnunet.log | |
| 2015-02-14 09:47 | ell | Note Added: 0008879 | |
| 2015-02-15 15:00 | ell | Note Edited: 0008879 | |
| 2015-02-17 19:35 | Christian Grothoff | Note Added: 0008884 | |
| 2015-02-18 09:35 | ell | Note Added: 0008887 | |
| 2015-02-18 09:41 | ell | File Added: leak.log | |
| 2015-02-18 11:41 | Christian Grothoff | Note Added: 0008888 | |
| 2015-02-18 11:43 | Christian Grothoff | Assigned To | => Matthias Wachs |
| 2015-02-18 11:43 | Christian Grothoff | Status | new => assigned |
| 2015-02-18 13:21 | ell | Note Added: 0008889 | |
| 2015-02-18 13:27 | ell | Note Edited: 0008889 | |
| 2015-02-18 14:39 | ell | Note Edited: 0008889 | |
| 2015-02-18 15:10 | Christian Grothoff | Note Added: 0008891 | |
| 2015-02-18 16:22 | ell | File Added: massif.out | |
| 2015-02-18 16:29 | ell | Note Added: 0008893 | |
| 2015-02-18 16:37 | Christian Grothoff | Note Added: 0008894 | |
| 2015-02-18 17:08 | Christian Grothoff | Assigned To | Matthias Wachs => Christian Grothoff |
| 2015-02-18 17:27 | ell | Note Added: 0008895 | |
| 2015-02-18 22:53 | Christian Grothoff | Assigned To | Christian Grothoff => Matthias Wachs |
| 2015-02-18 22:53 | Christian Grothoff | Category | datastore service => transport service |
| 2015-02-19 14:44 | Christian Grothoff | Relationship added | related to 0003687 |
| 2015-02-21 21:46 | Christian Grothoff | Note Added: 0008907 | |
| 2015-02-21 21:47 | Christian Grothoff | Status | assigned => feedback |
| 2015-02-22 10:31 | ell | Note Added: 0008911 | |
| 2015-02-22 10:31 | ell | Status | feedback => assigned |
| 2015-02-22 12:38 | Christian Grothoff | Note Added: 0008912 | |
| 2015-02-22 13:31 | ell | Note Added: 0008913 | |
| 2015-02-22 18:22 | amatus | Note Added: 0008914 | |
| 2015-02-22 23:00 | Christian Grothoff | Note Added: 0008915 | |
| 2015-02-22 23:50 | ell | Note Added: 0008918 | |
| 2015-02-23 09:46 | ell | Note Added: 0008921 | |
| 2015-02-24 10:12 | Matthias Wachs | Note Added: 0008925 | |
| 2015-02-24 10:13 | Matthias Wachs | File Added: stat-feb-20.png | |
| 2015-02-24 10:13 | Matthias Wachs | File Added: stat-feb-24.png | |
| 2015-02-24 10:17 | Matthias Wachs | Note Edited: 0008925 | |
| 2015-02-24 11:21 | ell | Note Added: 0008927 | |
| 2015-02-24 12:14 | ell | Note Edited: 0008927 | |
| 2015-02-25 09:34 | ell | Note Edited: 0008927 | |
| 2015-02-28 14:44 | Christian Grothoff | Category | transport service => UDP transport |
| 2015-02-28 14:44 | Christian Grothoff | Priority | normal => high |
| 2015-02-28 17:51 | Christian Grothoff | Relationship added | related to 0003695 |
| 2015-03-01 23:55 | Christian Grothoff | Note Added: 0008954 | |
| 2015-03-02 10:23 | ell | Note Added: 0008958 | |
| 2015-03-02 10:45 | ell | Note Edited: 0008958 | |
| 2015-03-07 20:39 | Christian Grothoff | Assigned To | Matthias Wachs => Christian Grothoff |
| 2015-03-08 21:38 | Christian Grothoff | Note Added: 0009008 | |
| 2015-03-08 21:38 | Christian Grothoff | Status | assigned => feedback |
| 2015-03-09 08:07 | ell | Note Added: 0009009 | |
| 2015-03-09 08:07 | ell | Status | feedback => assigned |
| 2015-03-09 09:11 | Christian Grothoff | Note Added: 0009010 | |
| 2015-03-09 09:11 | Christian Grothoff | Status | assigned => resolved |
| 2015-03-09 09:11 | Christian Grothoff | Fixed in Version | => 0.11.0pre66 |
| 2015-03-09 09:11 | Christian Grothoff | Resolution | open => fixed |
| 2015-03-09 09:11 | Christian Grothoff | Target Version | => 0.11.0pre66 |
| 2018-06-07 00:25 | Christian Grothoff | Status | resolved => closed |
