View Issue Details

IDProjectCategoryView StatusLast Update
0005520GNUnetwebpagepublic2019-02-28 11:17
Reporterng0Assigned Tong0 
PrioritylowSeverityminorReproducibilityhave not tried
Status closedResolutionfixed 
Product VersionSVN HEAD 
Target Version0.11.0Fixed in Version0.11.0 
Summary0005520: pybabel does not include current directory in sys.path on first run, build crashes.
Descriptiontemplate.py:
  * ...
    The mechanism is currently like an USB cable, 1st attempt fails,
    second succeeds. reproducible, annoying, maybe don't fix it.

You should keep in mind that we need to support python2.7 and *all*
python3.x version up to the most recent version, which implies all
the changes in python relative imports in those minor versions.

To replicate, run gmake with all dependencies in your path and assuming
a python3.7 version.
Tagswebsite

Activities

catonano

2019-01-30 02:39

reporter   ~0013515

I run make (on Ubuntu, so I guess gmake was run under the hood) a few times and I run into no failure

I can't reproduce this USB cable like behavior ¯\_(ツ)_/¯

ng0

2019-01-30 11:55

developer   ~0013519

Last edited: 2019-01-30 11:56

View 2 revisions

Hm, weird.
Are you able to produce very verbose output, like running dtrace or temporarily changing the website files to verbosely output what's being opened and searched at?

We were able to produce what I reported on GuixSD from a couple of months ago, NetBSD 8.0-Release, and whatever Grothoff ran when Grothoff reported it to me.

I'm just curious, because that means that Ubuntu is doing something different and that not all systems are affected.

catonano

2019-01-31 06:35

reporter   ~0013529

I was confused

I did observer the usb cable like behavior indeed

I'll try to figure this out

ng0

2019-01-31 17:51

developer   ~0013537

Shall I simply assign this issue to you?

amz3

2019-02-02 17:24

reporter   ~0013560

It is a good pratice to paste the stacktrace of the error.

I tried to run 'make' and fixed the issues I have encountered.

0001-PYTHONPATH-shenanigan.patch (2,559 bytes)
From ce7f7090f02d5d85a3d70bc8ef5694c92aee0c16 Mon Sep 17 00:00:00 2001
From: Amirouche <amirouche.boubekki@gmail.com>
Date: Sat, 2 Feb 2019 17:00:08 +0100
Subject: [PATCH 1/3] PYTHONPATH shenanigan

Running:

  git clean -fxd && make

Will produce the following error:

  extracting messages from about.html.j2 (encoding="utf-8", lstrip_blocks="True", trim_blocks="True")
  Traceback (most recent call last):
    File "/gnu/store/2dv8956ggc15s3yidadkw38v6ksi36k5-python-babel-2.6.0/bin/.pybabel-real", line 11, in <module>
     load_entry_point('Babel==2.6.0', 'console_scripts', 'pybabel')()
    File "/gnu/store/2dv8956ggc15s3yidadkw38v6ksi36k5-python-babel-2.6.0/lib/python3.7/site-packages/babel/messages/frontend.py", line 911, in main
     return CommandLineInterface().run(sys.argv)
    File "/gnu/store/2dv8956ggc15s3yidadkw38v6ksi36k5-python-babel-2.6.0/lib/python3.7/site-packages/babel/messages/frontend.py", line 835, in run
     return cmdinst.run()
    File "/gnu/store/2dv8956ggc15s3yidadkw38v6ksi36k5-python-babel-2.6.0/lib/python3.7/site-packages/babel/messages/frontend.py", line 470, in run
     for filename, lineno, message, comments, context in extracted:
    File "/gnu/store/2dv8956ggc15s3yidadkw38v6ksi36k5-python-babel-2.6.0/lib/python3.7/site-packages/babel/messages/extract.py", line 157, in extract_from_dir
     dirpath=absname,
    File "/gnu/store/2dv8956ggc15s3yidadkw38v6ksi36k5-python-babel-2.6.0/lib/python3.7/site-packages/babel/messages/extract.py", line 212, in check_and_call_extract_file
     strip_comment_tags=strip_comment_tags
    File "/gnu/store/2dv8956ggc15s3yidadkw38v6ksi36k5-python-babel-2.6.0/lib/python3.7/site-packages/babel/messages/extract.py", line 241, in extract_from_file
     strip_comment_tags))
    File "/gnu/store/2dv8956ggc15s3yidadkw38v6ksi36k5-python-babel-2.6.0/lib/python3.7/site-packages/babel/messages/extract.py", line 294, in extract
     func = getattr(__import__(module, {}, {}, [attrname]), attrname)
  ModuleNotFoundError: No module named 'i18nfix'

So we add current working directory aka. CWD in the pythonpath. But
since guix also has changed PYTHONPATH env variable we need to suffix
it. In bash that would have been simply:

export PYTHONPATH=$(pwd):$PYTHONPATH
---
 config.mk | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/config.mk b/config.mk
index 6bce49e..026cacb 100644
--- a/config.mk
+++ b/config.mk
@@ -1,4 +1,5 @@
-#PYTHONPATH="$PYTHONPATH:$(pwd)"
+GUIX_PYTHONPATH:=$(PYTHONPATH)
+PYTHONPATH=$(PWD):$(GUIX_PYTHONPATH)
 
 DEBUG=0
 
-- 
2.19.1

amz3

2019-02-02 17:24

reporter  

0002-cosmit.patch (1,192 bytes)
From 4bdccc555eacc6a8e66507cf1ef99a946f97284a Mon Sep 17 00:00:00 2001
From: Amirouche <amirouche.boubekki@gmail.com>
Date: Sat, 2 Feb 2019 17:03:15 +0100
Subject: [PATCH 2/3] cosmit

See previous commit
---
 template.py | 14 +-------------
 1 file changed, 1 insertion(+), 13 deletions(-)

diff --git a/template.py b/template.py
index fa978b7..0af50f6 100755
--- a/template.py
+++ b/template.py
@@ -20,19 +20,7 @@ import gettext
 import glob
 import codecs
 import jinja2
-
-# FIXME: lint will complain about this. Do NOT! fix this by writing
-# import i18nfix again, send an email to our developer list if you
-# have a problem with your system specific python integration!
-# PACKAGE_PARENT = '..'
-# SCRIPT_DIR = os.path.dirname(os.path.realpath(os.path.join(os.getcwd(), os.path.expanduser(__file__))))
-# sys.path.append(os.path.normpath(os.path.join(SCRIPT_DIR, PACKAGE_PARENT)))
-# sys.path.append(os.getcwd())
-#from www import i18nfix
-try:
-    from . import i18nfix
-except ImportError:
-    import i18nfix
+import i18nfix
 
 env = jinja2.Environment(loader=jinja2.FileSystemLoader(os.path.dirname(__file__)),
                          extensions=["jinja2.ext.i18n"],
-- 
2.19.1

0002-cosmit.patch (1,192 bytes)

amz3

2019-02-02 17:24

reporter   ~0013561

see previous patch

0002-cosmit-2.patch (1,192 bytes)
From 4bdccc555eacc6a8e66507cf1ef99a946f97284a Mon Sep 17 00:00:00 2001
From: Amirouche <amirouche.boubekki@gmail.com>
Date: Sat, 2 Feb 2019 17:03:15 +0100
Subject: [PATCH 2/3] cosmit

See previous commit
---
 template.py | 14 +-------------
 1 file changed, 1 insertion(+), 13 deletions(-)

diff --git a/template.py b/template.py
index fa978b7..0af50f6 100755
--- a/template.py
+++ b/template.py
@@ -20,19 +20,7 @@ import gettext
 import glob
 import codecs
 import jinja2
-
-# FIXME: lint will complain about this. Do NOT! fix this by writing
-# import i18nfix again, send an email to our developer list if you
-# have a problem with your system specific python integration!
-# PACKAGE_PARENT = '..'
-# SCRIPT_DIR = os.path.dirname(os.path.realpath(os.path.join(os.getcwd(), os.path.expanduser(__file__))))
-# sys.path.append(os.path.normpath(os.path.join(SCRIPT_DIR, PACKAGE_PARENT)))
-# sys.path.append(os.getcwd())
-#from www import i18nfix
-try:
-    from . import i18nfix
-except ImportError:
-    import i18nfix
+import i18nfix
 
 env = jinja2.Environment(loader=jinja2.FileSystemLoader(os.path.dirname(__file__)),
                          extensions=["jinja2.ext.i18n"],
-- 
2.19.1

0002-cosmit-2.patch (1,192 bytes)

amz3

2019-02-02 17:25

reporter   ~0013562

fix %o error

0003-apparently-percent-character-must-be-escaped.patch (1,038 bytes)
From 2d426bef7ca07775d671071c676100c36dd50aad Mon Sep 17 00:00:00 2001
From: Amirouche <amirouche.boubekki@gmail.com>
Date: Sat, 2 Feb 2019 17:23:58 +0100
Subject: [PATCH 3/3] apparently percent character must be escaped

---
 gnurl.html.j2 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gnurl.html.j2 b/gnurl.html.j2
index 3294723..5dd17dd 100644
--- a/gnurl.html.j2
+++ b/gnurl.html.j2
@@ -82,7 +82,7 @@
     move to something simpler. As the library gets a new name, we do not
     have to worry about tons of packages breaking as soon as one rebuilds
     it. So renaming itself and saying that "libgnurl = libcurl with only
-    HTTP/HTTPS support and GnuTLS" fixes 99% of the problems that darkened
+    HTTP/HTTPS support and GnuTLS" fixes 99%% of the problems that darkened
     my mood. Note that this pretty much CANNOT be done without a fork, as
     renaming is an essential part of the fix. Now, there might be creative
     solutions to achieve the same thing within the standard cURL build
-- 
2.19.1

ng0

2019-02-02 17:52

developer   ~0013563

I've applied the 3 patches locally, but they didn't fix the original problem at
first. Runs after that are okay.

ng0

2019-02-02 17:54

developer   ~0013564

iirc you have commit access to the website, if not ask for it.
applied and resolved.

Thanks!

ng0

2019-02-03 09:53

developer   ~0013565

Last edited: 2019-02-03 10:03

View 2 revisions

according to catonano it persists on their system.
I assume amz3 fixed it on guix, and for myself it works on NetBSD 8.0 with pkgsrc current.

catonano

2019-02-03 11:08

reporter   ~0013567

I have an update:

I did this:

export PYTHONPATH=$PWD:$PYTHONPATH

and then

pybabel extract -F locale/babel.map -o locale/messages.pot .

which is the same line that Make runs

And doing it like this, it works, even if I edit a file and do that again

So it seems that when Make runs it, PYTHONPATH doesn't include the local folder

I don't know Make enough to troubleshoot this

catonano

2019-02-03 11:41

reporter   ~0013568

I have a new update

I simply re checked it out and it works

It's as if the old checkout is keeping some state. hidden somewhere

The Autotools based system is known to be a bit too "stateful" :-/

I wouldn't want to close this yet, I'd like someone else to test this some more

dvn

2019-02-03 20:47

developer   ~0013572

This is still broken the same way for me. I just did a fresh clone, to test, and I get the following stacktrace the first time I run `make`, and it works without error the second time.

```
pybabel extract -F locale/babel.map -o locale/messages.pot .
extracting messages from about.html.j2 (encoding="utf-8", lstrip_blocks="True", trim_blocks="True")
Traceback (most recent call last):
  File "/usr/bin/pybabel", line 11, in <module>
    load_entry_point('Babel==2.4.0', 'console_scripts', 'pybabel')()
  File "/usr/lib/python3/dist-packages/babel/messages/frontend.py", line 908, in main
    return CommandLineInterface().run(sys.argv)
  File "/usr/lib/python3/dist-packages/babel/messages/frontend.py", line 832, in run
    return cmdinst.run()
  File "/usr/lib/python3/dist-packages/babel/messages/frontend.py", line 467, in run
    for filename, lineno, message, comments, context in extracted:
  File "/usr/lib/python3/dist-packages/babel/messages/extract.py", line 156, in extract_from_dir
    dirpath=absname,
  File "/usr/lib/python3/dist-packages/babel/messages/extract.py", line 211, in check_and_call_extract_file
    strip_comment_tags=strip_comment_tags
  File "/usr/lib/python3/dist-packages/babel/messages/extract.py", line 240, in extract_from_file
    strip_comment_tags))
  File "/usr/lib/python3/dist-packages/babel/messages/extract.py", line 293, in extract
    func = getattr(__import__(module, {}, {}, [attrname]), attrname)
ModuleNotFoundError: No module named 'i18nfix'
Makefile:12: recipe for target 'locale/messages.pot' failed
make: *** [locale/messages.pot] Error 1
```

dvn

2019-02-03 22:27

developer   ~0013573

I should add: This was run on Ubuntu 18.04.1

catonano

2019-02-04 00:17

reporter   ~0013574

Thanks for helping me test !!

I appreciate that !

Yes, I'm on Ubuntu 18.04.1 too

I installed 2 more packages, with pip, in the environment created with venv

You could try that.

In fact, it might be that I was missing those packages (or just one of them) and it was not all Make's fault

one is: "i18n"

Like: pip install i18n

and another is

pip install python-i18n

one of those packages probably provides the i18inf namespace

which one ?

I don't know

I just installed them frantically out of desperation but then I didn't try with each one of them alone

I think our dependencies should be listed explicitly somewhere but it seems they aren't

Please let me know if this helps you !

ng0

2019-02-04 00:27

developer   ~0013575

Please read the commit history. i18n is the local module named i18n, not a system wide one.

dvn

2019-02-04 01:04

developer   ~0013576

Last edited: 2019-02-04 01:04

View 2 revisions

> I think our dependencies should be listed explicitly somewhere but it seems they aren't

Line 7 of README is:

> 7: Requires python3-jinja2, python3-babel, and gettext.

Are there additional dependencies? If so, let's add them

catonano

2019-02-04 07:46

reporter   ~0013577

No, I was wrong

the 2 packages I mentioned do not affect the make process

The only element affecting that is the PYTHONPATH environment variable

If it contains the local folder, the process will run

Otherwise it won't

So the line no. 7 of the README seems correct. Those are the deps: jinaj2, babel and gettext

ng0

2019-02-04 09:45

developer   ~0013578

I'm not sure what your angle on "still fails" is here.
Initially I chose a compromise on the bitter treatment of local and relative imports in all available python versions. That still failed.
Now after amz3's commits we have a version which fails one time and succeeds the times afterwards (no failures after the initial first failure).
That's an improvement.

I seriously *hate* python cross-version, cross-system work. It would make more sense to work on the website first before we fix this quirk. I've been there too many times with other people, you can spend days and weeks discussing this even with stuff like pyenv, pip, etc pp.

catonano

2019-02-04 09:45

reporter   ~0013579

in the Makefile there are these 2 lines:

# Extract translateable strings from jinja2 templates.
locale/messages.pot: *.j2 common/*.j2.inc
        $(BABEL) extract -F locale/babel.map -o locale/messages.pot .


this call to pybabel shuld happen in an environment in which PYTHONPATH includes the local directory

amz3 suggested me something similar but in pastebin and now I don't find that again

I'm not sure how to do that in the make language

ng0

2019-02-04 09:47

developer   ~0013580

> amz3 suggested me something similar but in pastebin and now I don't find that again

was this on the irc channel? I can look into our logs.

catonano

2019-02-04 09:48

reporter   ~0013581

ok, let's move to something else for now

I just thought this could have affected the CI process

ng0

2019-02-04 09:49

developer   ~0013582

Yes, you are right. if we build from CI only, we must fix this.

catonano

2019-02-04 09:50

reporter   ~0013583

> was this on the irc channel? I can look into our logs

yes, that was on the irc channel, on saturday evening

ng0

2019-02-04 09:56

developer   ~0013584

I did not push "cosmit-2" because I don't understand why we have 2 identical patches up. (cosmit, cosmit-2)

The paste is gone. Just ask amz3 to send a patch.

catonano

2019-02-04 12:54

reporter   ~0013585

Ok

I already wrote this on the mailing list but I guess it's appropriate to write this here too for the sake of completeness

In the Makefile, I changed these lines

# Extract translateable strings from jinja2 templates.
locale/messages.pot: *.j2 common/*.j2.inc
        $(BABEL) extract -F locale/babel.map -o locale/messages.pot .

to

# Extract translateable strings from jinja2 templates.
locale/messages.pot: *.j2 common/*.j2.inc
        PYTHONPATH=$(PYTHONPATH) $(BABEL) extract -F locale/babel.map -o locale/messages.pot .

hat is, I am setting the PYTHONPATH before calling Babel

this seems to fix the problem

You can test this yourself.
Apply this edit, then change a file and try to build.
Then change it again and build again.
It should build in both cases

I would appreciate if anyone would confirm this

ng0

2019-02-04 13:06

developer   ~0013586

Last edited: 2019-02-04 13:08

View 2 revisions

on the one hand this works, on the other hand it suddenly seems to require bash.
My /bin/sh in netbsd (not bash, not dash,...) chokes on it.

edit: it won't work with bash at all. log paste coming soon.

ng0

2019-02-04 13:11

developer  

log.log (6,336 bytes)

ng0

2019-02-04 13:40

developer   ~0013587

We already include the PYTHONPATH from config.mk on a global level. This wouldn't change anything I think?

ng0

2019-02-04 13:45

developer   ~0013588

Last edited: 2019-02-04 13:48

View 2 revisions

What will be our CI? We should have at least that building without issues.

catonano

2019-02-04 14:48

reporter   ~0013590

> We already include the PYTHONPATH from config.mk on a global level. This wouldn't change anything I think?

as far as I understamd, config.mk declares PYTHONPATH but then it's not used in the Makefile

ng0

2019-02-04 14:58

developer   ~0013591

Last edited: 2019-02-04 14:59

View 2 revisions

This approach you suggested crashes with what you suggested and with a simple export of the same variable.

ng0

2019-02-04 15:19

developer   ~0013592

Last edited: 2019-02-04 15:24

View 2 revisions

Can you uncomment in template.py:

# DEBUG OUTPUT: print(sys.path)


relevant here is just:

print(sys.path)


For me this returns the absolute path to the current directory as it should be as the first entry, followed by other python specific ones.
You are also welcome to switch over the build system to something more appropriate if you find that this fixes the bug.
I'm still reading into the combination of pybabel, jinja2 and Makefiles.

dvn

2019-02-04 17:01

developer   ~0013593

In regards to CI, we can just run make twice, so no _serious_ problem.

dvn

2019-02-04 20:01

developer   ~0013594

I don't think this should be part of the 0.11.0 release roadmap.

catonano

2019-02-04 20:48

reporter   ~0013596

if make can be run twice, I would do that and move on

I thought this wasn't possible

ng0

2019-02-06 14:38

developer   ~0013610

ack.

Thanks!

ng0

2019-02-17 18:02

developer   ~0013846

Further notes:

not limited to gmake, also happens with bmake / NetBSD's make.

ng0

2019-02-17 18:16

developer   ~0013847

I think this is a problem with pybabel-3.7 not picking up the local modulepath in time.
looking into this.

ng0

2019-02-17 19:28

developer   ~0013848

This is fixed on NetBSD with make now.
This should work on every system.

If it doesn't work on your system --- I haven't tested this on Guix, Nix, or other odd ones --- you are welcome to send patches which don't break it for everyone or apply breaking patches locally.

Issue History

Date Modified Username Field Change
2019-01-27 16:06 ng0 New Issue
2019-01-29 15:15 catonano Tag Attached: website
2019-01-30 02:39 catonano Note Added: 0013515
2019-01-30 11:55 ng0 Note Added: 0013519
2019-01-30 11:56 ng0 Note Edited: 0013519 View Revisions
2019-01-30 19:35 ng0 Category other => webpage
2019-01-30 19:35 ng0 Target Version => 0.11.0
2019-01-31 06:35 catonano Note Added: 0013529
2019-01-31 17:51 ng0 Note Added: 0013537
2019-01-31 17:51 ng0 Status new => acknowledged
2019-02-02 17:24 amz3 File Added: 0001-PYTHONPATH-shenanigan.patch
2019-02-02 17:24 amz3 Note Added: 0013560
2019-02-02 17:24 amz3 File Added: 0002-cosmit.patch
2019-02-02 17:24 amz3 File Added: 0002-cosmit-2.patch
2019-02-02 17:24 amz3 Note Added: 0013561
2019-02-02 17:25 amz3 File Added: 0003-apparently-percent-character-must-be-escaped.patch
2019-02-02 17:25 amz3 Note Added: 0013562
2019-02-02 17:52 ng0 Note Added: 0013563
2019-02-02 17:54 ng0 Assigned To => ng0
2019-02-02 17:54 ng0 Status acknowledged => resolved
2019-02-02 17:54 ng0 Resolution open => fixed
2019-02-02 17:54 ng0 Note Added: 0013564
2019-02-03 09:46 Christian Grothoff Product Version => SVN HEAD
2019-02-03 09:46 Christian Grothoff Fixed in Version => 0.11.0
2019-02-03 09:53 ng0 Assigned To ng0 =>
2019-02-03 09:53 ng0 Status resolved => feedback
2019-02-03 09:53 ng0 Resolution fixed => reopened
2019-02-03 09:53 ng0 Note Added: 0013565
2019-02-03 10:03 ng0 Note Edited: 0013565 View Revisions
2019-02-03 11:08 catonano Note Added: 0013567
2019-02-03 11:41 catonano Note Added: 0013568
2019-02-03 20:47 dvn Note Added: 0013572
2019-02-03 22:27 dvn Note Added: 0013573
2019-02-04 00:17 catonano Note Added: 0013574
2019-02-04 00:27 ng0 Note Added: 0013575
2019-02-04 00:27 ng0 Status feedback => new
2019-02-04 01:04 dvn Note Added: 0013576
2019-02-04 01:04 dvn Note Edited: 0013576 View Revisions
2019-02-04 07:46 catonano Note Added: 0013577
2019-02-04 09:45 ng0 Note Added: 0013578
2019-02-04 09:45 catonano Note Added: 0013579
2019-02-04 09:47 ng0 Note Added: 0013580
2019-02-04 09:48 catonano Note Added: 0013581
2019-02-04 09:49 ng0 Note Added: 0013582
2019-02-04 09:50 catonano Note Added: 0013583
2019-02-04 09:56 ng0 Note Added: 0013584
2019-02-04 10:02 ng0 Priority normal => high
2019-02-04 10:02 ng0 Status new => confirmed
2019-02-04 10:02 ng0 Fixed in Version 0.11.0 =>
2019-02-04 12:54 catonano Note Added: 0013585
2019-02-04 13:06 ng0 Note Added: 0013586
2019-02-04 13:08 ng0 Note Edited: 0013586 View Revisions
2019-02-04 13:11 ng0 File Added: log.log
2019-02-04 13:40 ng0 Note Added: 0013587
2019-02-04 13:45 ng0 Note Added: 0013588
2019-02-04 13:48 ng0 Note Edited: 0013588 View Revisions
2019-02-04 14:48 catonano Note Added: 0013590
2019-02-04 14:58 ng0 Note Added: 0013591
2019-02-04 14:59 ng0 Note Edited: 0013591 View Revisions
2019-02-04 15:19 ng0 Note Added: 0013592
2019-02-04 15:24 ng0 Note Edited: 0013592 View Revisions
2019-02-04 15:40 ng0 Summary fix USB-like behavior on new website generation => pybabel does not include current directory in sys.path on first run, build crashes.
2019-02-04 17:01 dvn Note Added: 0013593
2019-02-04 20:01 dvn Note Added: 0013594
2019-02-04 20:48 catonano Note Added: 0013596
2019-02-06 14:38 ng0 Note Added: 0013610
2019-02-06 14:38 ng0 Assigned To => ng0
2019-02-06 14:38 ng0 Status confirmed => acknowledged
2019-02-06 14:39 ng0 Priority high => low
2019-02-06 14:39 ng0 Status acknowledged => confirmed
2019-02-06 14:39 ng0 Target Version 0.11.0 => 0.11.1
2019-02-06 14:39 ng0 Assigned To ng0 =>
2019-02-17 18:02 ng0 Note Added: 0013846
2019-02-17 18:16 ng0 Note Added: 0013847
2019-02-17 18:16 ng0 Assigned To => ng0
2019-02-17 18:16 ng0 Status confirmed => assigned
2019-02-17 19:28 ng0 Status assigned => resolved
2019-02-17 19:28 ng0 Resolution reopened => fixed
2019-02-17 19:28 ng0 Fixed in Version => 0.11.0
2019-02-17 19:28 ng0 Note Added: 0013848
2019-02-20 12:20 Christian Grothoff Target Version 0.11.1 => 0.11.0
2019-02-28 11:17 Christian Grothoff Status resolved => closed