View Issue Details

IDProjectCategoryView StatusLast Update
0006264TalerWeb site(s)public2020-07-24 11:56
ReporterChristian Grothoff Assigned ToChristian Grothoff  
PrioritylowSeverityfeatureReproducibilityN/A
Status resolvedResolutionfixed 
Platformi7OSDebian GNU/LinuxOS Versionsqueeze
Product Versiongit (master) 
Target Version0.8Fixed in Version0.8 
Summary0006264: setup buildbot job to check website for dead links
DescriptionI don't know which Free Software would work best here, ideally something available in Debian. Please identify a suitable package, let someone with root know to install it on taler.net, and deploy it via buildbot. If there are broken links, it should send e-mail notifications - we can define a functional alias pointing to Stefan.
TagsNo tags attached.

Activities

buckE

2020-05-27 09:13

reporter   ~0015973

This is very clear, thank you for the definition.

buckE

2020-06-10 08:01

reporter   ~0016249

Christian: what MTA should I use in the script? "mail" etc. is not found on the system, and I know there are some planned changes. Please suggest a method of sending mail from the correct account (which I assume is whichever user account manages the 'sites-worker' buildbot worker and runs ./build-sites.sh - I'm still not sure which account that is yet but if you know, please mention that here).

Christian Grothoff

2020-06-10 11:36

manager   ~0016252

Ah, I think _buildbot_ has a way to send messages to the last committer. At least we sometimes from some job get e-mails. So buildbot has that capability. Likely the shell script wrapper only has to return a failure code *and* the buildbot job must be configured accordingly.

Buildbot can also figure out for you who are the committers since the built broke, so you can notify the right people. So the e-mail should be done by buildbot, the body should be the output of the sphinx command, and the main modifications are likely to (a) make the shell script return an error code to indicate failure, and (b) tell buildbot to inform the developers. --- and it is possible that (b) is already the case, and only (a) needs to be done.

buckE

2020-06-11 06:16

reporter   ~0016254

Your reply makes sense for https://bugs.gnunet.org/view.php?id=6223 but this issue is different because:

1 - sphinx is not involved (so the body is actually going to be a wget log - clumsy but I like the simplicity of using wget instead of a less common program)

2 - we aren't e-mailing the most recent committer, but Stefan

So I *think* it will still be necessary to have an MTA on the server to send this e-mail. At least that seems the most elegant. And since this is already something you are considering, would it make sense to wait for that or investigate some other method like a hack to buildbot?

Christian Grothoff

2020-06-11 09:53

manager   ~0016257

We could e-mail the most recent committer, if buildbot truly cannot e-mail Stefan. We could also just post to IRC, that would also suffice.
Oh, and yes, replace sphinx by wget here, I confused the two bugs.

buckE

2020-06-14 07:59

reporter   ~0016265

OK I'll do IRC first, then look into how specific the e-mail feature can get. Hopefully it is as simple as writing a new buildbot service specific to Stefan, and scheduling that service.

Would you like this to be under a "linkchecker" user? If so please create that account and give me access.

I assume I should create a "bootstrap-linkchecker" file? like "deployment/bootstrap-docbuilder"?

Christian Grothoff

2020-06-14 13:19

manager   ~0016266

linkchecker@taler.net is now setup. And yes, deployment scripts that will make it trivial in the future to migrate the setup to a new system are always appreciated.

buckE

2020-06-18 07:28

reporter   ~0016295

linkchecker BUILDBOT job running successfully.

e-mail notifications through Buildbot successful (tested sending to self)

Currently configured to send to self (for temporary testing only) and to brokenlinks@taler.net.

CHRISTIAN PLEASE DO: Please alias brokenlinks@taler.net to Stefan, etc.

CHRISTIAN QUESTION:

Currently, output of wget is written to /home/linkchecker/linkchecker.log. This contains a list of broken links. I like using wget for this, but buildbot e-mail does not seem to have the ability to send arbitrary attachments (only buildbot logs). So I think we have these options:

1 - Provide Stefan wtih shell access to linkchecker@taler.net (I know, we just discussed this right? Bad option.)

2 - We make linkchecker.log available via nginx, perhaps behind an http password?

3 - configure MTA on taler.net to use localhost `mail` command instead of buildbot reporter

4 - Something else I am not thinking of. Maybe do not use wget (but it's really so appropriate - and then what exactly would be better for solving this issue anyway)?

Christian Grothoff

2020-06-18 14:59

manager   ~0016301

I would just "cat linkchecker.log" as one of the build steps. Then this _is_ what the buildbot logs, as the output of all the commands you run in buildbot becomes part of the log. So I'd simply "if failed links, cat linkchecker, exit with failure (=> buildbot creates e-mail with the output from 'cat')". And if there are no failed links, simply exit with success (=> no e-mail).

buckE

2020-06-19 08:04

reporter   ~0016313

> I would just "cat linkchecker.log" as one of the build steps.

Oh, clever. Done. Confirmed working (log comes as attachment in e-mail). Job runs at 0745 daily.

Please set up the email alias from `brokenlinks@taler.net` to stefan or anyone you want. I can also specify specific users if you want.

Issue History

Date Modified Username Field Change
2020-05-27 09:03 Christian Grothoff New Issue
2020-05-27 09:03 Christian Grothoff Status new => assigned
2020-05-27 09:03 Christian Grothoff Assigned To => buckE
2020-05-27 09:13 buckE Note Added: 0015973
2020-06-10 08:01 buckE Note Added: 0016249
2020-06-10 11:36 Christian Grothoff Note Added: 0016252
2020-06-11 06:16 buckE Note Added: 0016254
2020-06-11 09:53 Christian Grothoff Note Added: 0016257
2020-06-14 07:59 buckE Note Added: 0016265
2020-06-14 13:19 Christian Grothoff Note Added: 0016266
2020-06-18 07:28 buckE Note Added: 0016295
2020-06-18 14:59 Christian Grothoff Note Added: 0016301
2020-06-19 08:04 buckE Assigned To buckE => Christian Grothoff
2020-06-19 08:04 buckE Status assigned => resolved
2020-06-19 08:04 buckE Resolution open => fixed
2020-06-19 08:04 buckE Note Added: 0016313
2020-07-24 11:56 Christian Grothoff Target Version => 0.8
2020-07-24 11:56 Christian Grothoff Fixed in Version => 0.8