View Issue Details

IDProjectCategoryView StatusLast Update
0006144TalerWeb site(s)public2020-04-20 23:14
ReporterChristian GrothoffAssigned TobuckE 
PrioritynormalSeverityfeatureReproducibilityN/A
Status resolvedResolutionfixed 
Platformi7OSDebian GNU/LinuxOS Versionsqueeze
Product Versiongit (master) 
Target VersionFixed in Version 
Summary0006144: WooCommerce installation / demonstrator
DescriptionBFH students developed last year a Taler payment plugin for WooCommerce. We should add the resulting demonstrator as woocommerce.demo.taler.net.

For this, we need a VM running WooCommerce with the Taler payment plugin, including _good_ instructions and/or shell scripts for how to set it up from scratch. The resulting VM should be integrated with demo.taler.net via a reverse-proxy setup configuration file in the existing nginx HTTPS server (for the respective virtual host name).

As the student plugin is a bit dated (we changed some (minor) aspects of the protocol), getting it to run will likely require some (minor) modifications to the existing payment plugin.
TagsNo tags attached.

Activities

buckE

2020-04-02 07:09

developer   ~0015486

Excellent, thank you. Where should the " _good_ instructions and/or shell scripts" (which will be later used to build "woocommerce.demo.taler.nen" be committed?

Christian Grothoff

2020-04-02 10:03

manager   ~0015487

I'd put it into our Git repository with the WooCommere-Taler extension.

Christian Grothoff

2020-04-02 10:06

manager   ~0015488

Oh, I should say the 'generic' instructions should go into that Woocommerce-Taler extension Git repo.

The very specific details for our _demo_ deployment should likely go into the onboarding manual in docs.git, and scripts specific to our installation (like the nginx configuration) go into the gv-maintenance.git (to which you probably don't *yet* have access).

buckE

2020-04-03 00:41

developer   ~0015493

Okay. First commit made, just beginning the process. I moved the plugin to a /plugin directory for clarity. Yes the demo will be a different project essentially. A special case of this general one.

buckE

2020-04-03 05:17

developer   ~0015494

Making good progress. Plugin is installed, some phrasing updated, and docs, scripts, assets are progressing well. Please consider making a backup of your current copy of the repo before doing a pull. I made major changes to the directory structure and minor changes to the plugin.

Christian, can you create a woocommerce-taler-demo-test@taler.net e-mail forwarder that forwards to me? I will use it to set up a demo *merchant* account and my personal customer account can be used to test as a customer. Or do you have a better suggestion?

Christian Grothoff

2020-04-03 10:35

manager   ~0015496

E-mail alias should be setup now.

buckE

2020-04-04 03:07

developer   ~0015517

E-mail alias seems to be working (buck@taler.net), but I am confused how to acquire the API Key that I need to test the plugin. I thought I would find it by creating a new Taler account and clicking on https://backoffice.demo.taler.net/ when logged on. But that was something different.

https://bank.demo.taler.net/profile does not offer this.

The page https://docs.taler.net/core/api-merchant.html discusses the API and it looks like `merchant_pub` is the variable holding the public key for the merchant. But I still don't see where to acquire that key.

Also the "GNU Taler Payment URL" is not clear to me.

It's probably right in front of me somewhere. Can you point me to where?

API-Key-Field.png (63,290 bytes)
API-Key-Field.png (63,290 bytes)

Christian Grothoff

2020-04-04 10:16

manager   ~0015523

See section 6.1.4:
  headers={"Authorization": "ApiKey sandbox"}

"ApiKey sandbox" is the actual API-key of the payment URL
http(s)://backend.demo.taler.net/ (because it is for testing, also HTTP works!).

buckE

2020-04-05 01:54

developer   ~0015530

Ah, okay. This info is for the demo and it will allow me to test this WooCommerce plugin. But what I'm looking for is how I can *acquire* an API key. Because I am writing a doc for end-users of the Taler WordPress/WooCommerce plugin, and most of those users will not be using the demo site, or have a Taler server of their own. They will be webstore owners who make an arrangement with a Taler provider, and part of that arrangement will be to sign up for an API key. If this is a process that can be done online (instead of manually, in a conversation between parties), then I would like to know how so I can add this to the docs.

buckE

2020-04-05 01:55

developer   ~0015531

Also is there a good way to verify that the API key is being accepted? I am sending requests to the backend using curl/json. I do not get authorized:

```
$ curl -X GET -H "Content-type: application/json" -H "Accept: application/json" 'https://backend.demo.taler.net/check-payment{"Authorization":"ApiKey sandbox"}'
<html>
<head><title>401 Authorization Required</title></head>
<body>
<center><h1>401 Authorization Required</h1></center>
<hr><center>nginx</center>
</body>
</html>
```

It does not look like the Taler plugin generates logs (or maybe I don't know where to find them) so I am looking for a place to start. The debugging is necessary because after filling out the fields with the information you gave me above, adding an item to the card, and attempting to pay with GNU Taler, I see this error:

`There seems to be a problem with the payment process, please try again or send the following message to a system administrator: 400 - request`

This could be anything. Bad address, wallet issue, etc. So I want to debug one step at a time, beginning with manual (curl/json?) authorizing with the API key.

Christian Grothoff

2020-04-05 02:08

manager   ~0015532

The API key and authorization is completely handled by nginx. So users that do not run their own Taler merchant backend will have to obtain it from their backend provider -- and that backend provider would have to implement the access control in some kind of reverse proxy (like we do with nginx).

As for your failing authorizations, I am confused. Here is the respective excerpt from the nginx configuration:

    # match the ApiKey part ignoring case, and the actual key
    # with case-sensitivity on.
    if ($http_authorization !~ "(?i)ApiKey (?-i)sandbox") {
      return 401;
    }

I've also attached the full configuration of nginx for your reference. That's where the 401 comes from...

demo.site (13,336 bytes)

buckE

2020-04-05 04:56

developer   ~0015533

I don't see where $http_authorization is defined. What is this referring to? My experiences with nginx authentication use something like this: https://docs.nginx.com/nginx/admin-guide/security-controls/configuring-http-basic-authentication/#config.

Are there any examples at this time where this authentication method and API Key are working? I would like to see the exact string that is being sent to the webserver in such a working example.

Christian Grothoff

2020-04-05 12:21

manager   ~0015534

$ curl -X GET -H "Content-type: application/json" -H "Accept: application/json" -H "Authorization: ApiKey sandbox" 'https://backend.demo.taler.net/check-payment'
{
  "code": 1008,
  "hint": "order_id required"
}

buckE

2020-04-07 05:49

developer   ~0015539

Making progress. I don't know how much information you like so I am giving you a medium amount of detail.

The plugin definitely has problems. Some fields are mislabeled, some are missing, but that is not the cause of the earlier problem. This error above was a generic WooCommerce error that would have been shown regardless of the backend error. The backend error was caused because I was using GBP, not KUDOS. I added KUDOS to the WordPress site and now I have a new problem (so, progress!, and added to the Howto doc for people who will use KUDOS or other custom currency.)

Go to the checkout page, enter data, start the transaction, I am told "function xdg-open wants to open this application" which was very strange to me, but I chose to accept. And then the checkout screen hangs (can not click anything.) Checking on the transaction shows me this:

$ curl -X GET -H "Content-type: application/json" -H "Accept: application/json" -H "Authorization: ApiKey sandbox" 'https://backend.demo.taler.net/check-payment?order_id=wc_order_P1jNw5WBqZEBX-15'
{
  "taler_pay_uri": "taler://pay/backend.demo.taler.net/-/-/wc_order_P1jNw5WBqZEBX-15",
  "contract_url": "https://backend.demo.taler.net/public/proposal?instance=default&order_id=wc_order_P1jNw5WBqZEBX-15",
  "paid": false,
  "already_paid_order_id": null


So I think that the problem is that the wallet is not being called. That was in Brave, and also the fee (40KUDOS) was more than I had in the wallet. So I am ignoring that problem for now.

Lowering the price to 3KUDOS, and using a Firefox browser with a wallet that has 3.x KUDOS in it, I make the transaction and I get this error:

```
The address wasn’t understood

Firefox doesn’t know how to open this address, because one of the following protocols (taler) isn’t associated with any program or is not allowed in this context.

    You might need to install other software to open this address.
```

The URL in the URL bar is "taler://pay/backend.demo.taler.net/-/-/wc_order_XIQ3OvPphCHzt-16"

So I think now the problem is with the taler *browser* extension, not the WordPress plugin. Does this sound correct to you?

How do you expect a browser to learn how to handle a `taler://` URI?

Note that this is different from the Brave error, because with the Brave error, the transaction was registered on the backend. But in this case, the same `curl` command shows that the backend was not informed:

```
$ curl -X GET -H "Content-type: application/json" -H "Accept: application/json" -H "Authorization: ApiKey sandbox" 'https://backend.demo.taler.net/check-payment?order_id=wc_order_P1jNw5WBqZEBX-16'
{
  "code": 2913,
  "hint": "unknown order_id"
}
```

So we have several problems, but I am sure you will agree that that's good. And maybe your understanding of the system will suggest a good place to start debugging.

Note: Chromium Browser behavior is exactly like Brave. xdg-open window asks: "https://10.137.0.28 wants to open this application." and I choose to allow, not cancel.

buckE

2020-04-07 07:11

developer   ~0015543

On another note, is there a "standard" for the currency symbol for KUDOS? I chose "KS" not for the abbreviation (like GPB/EUR) but the symbol like "$". I think it is important to be consistent whenever possible.

Christian Grothoff

2020-04-07 10:11

manager   ~0015545

The debugging of where/how things changed in the protocol (wallet-interaction, backend) is not something I expected you to (primarily) do. Let's get the site running on taler.net, and *then* Florian, Marcello and I will take a look and (likely) fix the things that are due to (usually trivial, once understood) protocol changes.

I don't see any documentation/instructions on how to setup Wordpress/Woocommerce/etc. on the server in the Git yet.

As for your questions: we change the version when it runs ;-). exchange != backend: exchange = exchange.git, backend = merchant.git.

buckE

2020-04-08 06:51

developer   ~0015567

nginx config instructions and sample wordpress.nginx.site file added to repo (under server-build/Server-Build-Instructions.md and server-build/assets/wordpress.nginx.site)

Tested working in my dev VM. (YMMV if installed on a site with other nginx configs.)

buckE

2020-04-08 06:56

developer   ~0015568

At this point, server-build/Server-Build-Instructions.md should have clear directions for building a Debian 10 server from scratch to end with a WordPress/WooCommerce site, running on apache2 or nginx, with a LetsEncrypt or self-signed SSL, with GNU Taler configured, KUDOS supported, a sample product added, and it should fail in the above way.

If you have problems, or you need something else from me please tell me. (ex. Maybe the error above is not with the plugin after all?) But at this time I will assume I am completed with my side, and I am very interested what solves the problem I describe in https://bugs.gnunet.org/view.php?id=6144#c15539

Christian Grothoff

2020-04-08 13:59

manager   ~0015582

Ok, I'll take a look at the build instructions and will report back.

Christian Grothoff

2020-04-08 19:19

manager   ~0015591

Ok, let's see.

Given that PHP is a mess with nginx, the choice of Apache makes sense.

Not sure about MariaDB, but good enough.

Firewall: eh, no. Sure, you can leave a remark to ensure that those ports ought to be open, but please don't do this on our systems.

Let's encrypt: doesn't _necessarily_ belong here IMO, good for generic instructions, but just to be clear on our server you'll need to set it up with just _HTTP_. TLS termination will be done by nginx.

wget: not sure about using -latest explicitly here, that _may_ eventually result in an incompatible/broken version. I'd at least use $VERSION and then one can do VERSION=latest in the shell script (see below).

Overall, the current instructions result in a typing-marathon (see my initial report "and/or shell scripts for how to set it up from scratch"). We should minimize the instructions more and at least for our setup replace almost everything with "run # woocommerce-installer-script.sh". The shell script can then have the details from your write-up as comments. Steps that cannot be fully automated (like setting password for MariaDB) could be output (echo) at the end with some text "reminder: you should probably do this" including of course why.

Nevertheless, good starting point.

So next steps:
(1) Boil instructions down to what we actually need, i.e. create a (robust!) shell-script that does all of the main setup logic we need (HTTP-only, no firewall). So assume 'minimal' Debian installation was finished, and then the instructions should be "run this shell script", and _then_ optionally "run this to set passwords" and otherwise "configure now like this the Browser".

(2) You can now log into test-woocommerce@taler.net. The user is in the 'kvm' group. Please setup a minimal Debian 10 VM that listens on TCP:localhost:9999, and forwards that traffic to the internal Apache2:80 (http). Run your installation script. https://woocommerce.test.taler.net/ is already setup (DNS, TLS, X509) as a reverse proxy to http://localhost:9999/.

(3) Try to automate our _specific_ setup. I.e. find a way to avoid having to use the browser at all. Maybe there is some Wordpress CLI tool (similar to drush for Drupal), or worst case hack the settings with SQL into the database from a shell script. The script should take key parameters as arguments, like the currency (KUDOS) and the URL of the merchant backend.

Please make sure that (2) and (3) are _very_ easy and fast to do. Ideally have a single (shell) command to run to generate the Debian kvm/qemu VM, setup an easy way to login to the guest VM for anyone who is "test-woocommerce" on the host. We'll want to be able to easily and *reliably* re-do the setup, including on other systems as well as for the woocommerce.demo.taler.net-site.

buckE

2020-04-08 21:34

developer   ~0015593

I'm confused. Why are you talking about "our server?" The task as I understood it was to document how a generic person who wants to sell his products online can build a Debian/WordPress/WooCommerce/GNU-Taler shop from scratch. I think the instructions do this. We did not lose any time over the confusion, because everything up until installing GNU Taler was very quick. But I mention this because it is not the only task I have where I am learning the *actual* task later.

Also, if the goal is to install something on one particular server, why would we have generic instructions and/or BASH scripts for a one-time installation? You understand why I'm asking, right? Why automate something that we will do one time? I think this is an indication that I still do not know the *actual* goal.

But now that you define (2) and (3) I can re-do this to that end. I am not familiar with WP-CLI or WC-CLI but they should be not very difficult to work with. Notes:

> wget: not sure about using -latest explicitly here,

Agreed. This is why I preferred instructions to a shell script for the generic instructions. If it breaks, it will be clear where and why. For example, installing PHP7 instead of PHP13 or whatever. But this will be different for a specific installation, and the script will specify "this script installs WordPress version X, PHP7.3, etc."

> Let's encrypt: doesn't _necessarily_ belong here IMO, good for generic instructions,

Right, this was intended to be generic instructions for a GNU Taler user who wants to create a WooCommerce store.

> irewall: eh, no. Sure, you can leave a remark to ensure that those ports ought to be open, but please don't do this on our systems.

Right. Again, I had no indiciation this had anything to do with "our systems," I thought I was writing docs for end users.

Basically I think this is a completely new task that borrows nothing from the original one except how to add KUDOS. Okay, I'll get started.

Christian Grothoff

2020-04-08 21:53

manager   ~0015594

Ok, let me clarify:
1) we need instructions for _others_ to reproduce their own setup. Good, we have some of those. As I tried
    to indicate, I'm pretty happy with those even though I think some more scripting can reduce the hassle
    to the user. But, that was only half the goal:
2) we ALSO need to do our own deployment (for testing and for demonstration). We ALSO want to be
    prepared to re-do our own deployment reasonably quickly (migrating servers, updating to a new version,
   we also usually run 2-4 setups on the same server (!) of almost everything (!) for testing and development).
   Finally, we want things to be maximally reproduceable. Hence this should be semi-automated.

buckE

2020-04-08 23:09

developer   ~0015595

Thank you. It's making much more sense.

buckE

2020-04-09 08:16

developer   ~0015598

Do you want me to commit the (4GB) base VM image?

buckE

2020-04-09 09:13

developer   ~0015599

(More like 2G now)

Christian Grothoff

2020-04-09 09:47

manager   ~0015600

Obviously not. We very, very rarely put generated code into Git. The image should simply live in /home/woocommerce on gv.taler.net. The shell-script to create such an image should live in Git.

buckE

2020-04-10 01:48

developer   ~0015606

Is it feasible to add KUDOS support to the plugin? It would avoid one more hack:

https://docs.woocommerce.com/document/add-a-custom-currency-symbol/#

Status: At this time I have a script that goes from a basic Debian VM to a functional basic WP site available to host at port 9999, and I have that basic Debian VM. This is working and reproducible.

But going from functional WP site to the end goal by cli is much more complicated due to the nature of WP. The wp-cli tool doesn't allow things like including custom code in a child theme's function.php file. Sure I can do that on the BASH terminal - until there is a new theme with a new version of WP and we have to re-create the script for that customization.

(I'm also not sure how to build a VM with a functioning OS without logging into it to install the OS, or without importing a filesystem, but that's for later.)

One suggestion I have is that I create a base WooTaler VM that has a functioning site that just turns on and runs. Then on this guest system, if a user wanted to upgrade it (ex: when GNU Taler Plugin is upgraded), that would require logging on, git pull an *update* script for Debian/PHP/WP/WC/GNU-Taler-Plugin (that was requested/tested for package compatibility at that time), run it, test the website, log off the VM. Of course that's just as easy as rebuilding the VM.

Note: gv.taler.net does not have rsync installed. I don't know if that is intentional but I would eventually want to use it to upload the VM.

buckE

2020-04-10 09:22

developer   ~0015607

> The script should take key parameters as arguments, like the currency (KUDOS) and the URL of the merchant backend.

Okay if we want to make that part of the script, the "recommended" method would be to add wp-cli support to the taler plugin (maybe a good idea anyway?) : https://wpreset.com/add-wp-cli-support-wordpress-plugin/

If you don't want to do that, I assume I can set the parameters manually in the PHP of the plugin, but depending on how the plugin is written, it may or may not be overwritten on upgrade.

Christian Grothoff

2020-04-10 11:50

manager   ~0015610

Yes, we should probably avoid the KUDOS hack by putting something into the plugin, that sounds certainly better than hacking custom code in a child theme's function.php file. Or re-use some other currency symbol, I just liked the idea of using "KU".

rsync: eh, I had expected you'd create the 'final' VM on the server locally (should save you a ton of bandwidth, and if things are sufficiently automated, shouldn't be extra work). But sure, we can install it.

wp-cli support in the plugin: yes, that would be good to add. If this is something _you_ feel you can do, that would be perfect (as a follow-up task).

About "(I'm also not sure how to build a VM with a functioning OS without logging into it to install the OS, or without importing a filesystem, but that's for later.)": I was thinking of using the FAI/Simple-CDD facilities of Debian:
https://debian-handbook.info/browse/stable/sect.automated-installation.html
This is _also_ what I was thinking for the 'bigger' task of creating a simple USB stick for a fully-automated reproduceable exchange deployment (except there we then need to also create some Git-based setup to interact/upgrade the running system).

About upgrade-script: sure, having an upgrade script would be nice, too. Especially for production deployments where one would want to minimize downtime and also not throw away the DB :-).

Christian Grothoff

2020-04-10 11:51

manager   ~0015611

Oh, and as long as *our* plugin doesn't support wp-cli, it's OK to leave those steps interactive in the Browser. Please open a bug that our plugin should support wp-cli, pointing to the wp-cli documentation and saying which options are relevant here.

buckE

2020-04-11 06:45

developer   ~0015619

TL;DR: There exists a README and scripts that successfully build a server and site when run on barebones Debian. Automating the image creation is next. Should be easy compared to the wp-cli nonsense.

> Yes, we should probably avoid the KUDOS hack by putting something into the plugin, that sounds certainly better than hacking custom code in a child theme's function.php file. Or re-use some other currency symbol, I just liked the idea of using "KU".

For now the script provides and exit instruction to handle this in the GUI along with a couple of other unavoidable things.

> I had expected you'd create the 'final' VM on the server locally

Okay but that seems reckless. And it would provide no opportunity for debugging if there are errors. But now that I'm understanding the actual specs better, I do see how this *could* work and if you like actually doing it, that's what I will do. No need for rsync then at this time (but it is useful, no?)

> https://debian-handbook.info/browse/stable/sect.automated-installation.html

Okay now I'm starting to understand the project goals. I'll look into this.

> This is _also_ what I was thinking for the 'bigger' task of creating a simple USB stick

Interesting. There must be parts of that project I don't yet understand.

> having an upgrade script would be nice, too

That was a suggestion if we would have a base disk image. If we're creating the image on the fly, this seems redundant?

> as long as *our* plugin doesn't support wp-cli, it's OK to leave those steps interactive in the Browser

Done.

> Please open a bug that our plugin should support wp-cli, pointing to the wp-cli documentation and saying which options are relevant here.

Okay. Although you also said "wp-cli support in the plugin: yes, that would be good to add. If this is something _you_ feel you can do, that would be perfect (as a follow-up task)" so I am confused but I will open the bug and assign it to you. You can reassign how you want.

Christian Grothoff

2020-04-11 11:11

manager   ~0015621

rsync is already installed, even if you won't need it this time.

I wouldn't say that we "are creating the image on the fly". We *automate* doing so, sure. But as I said, the upgrade script has a different purpose (not so much for our deployment, but for end-users that need good uptime). Anyway, the upgrade script is not really important IMO (and was not part of the original plan).

cli-bug: it's OK, let's discuss it once we're done with the existing bugs (this one, and the exchange.deb).

buckE

2020-04-13 03:18

developer   ~0015632

I'm looking at FAI and it is very interesting. I will use it for this if you want, but it is not more elegant than my suggestion of using a base VM image. Basically my concern from above: "I'm also not sure how to build a VM with a functioning OS without logging into it to install the OS, or without importing a filesystem" is not answered by FAI. Based on my reading, FAI (not surprisingly) requires importing a filesystem, and also having the FAI server available to every client that will manifest a VM. So it is less elegant than having a base .img file (although the required FAI ISO is *smaller* than the base IMG file would be, but I don't see how that helps in this specific case.)

However if the real purpose of this task is as a trial for the larger task, (a) I'm not sure there aren't better ways to accomplish that either (really, FAI seems not different in principle from using Puppet or Ansible) but (b) okay at least I understand - though I wish I'd known the real task when I'd started. Anyway I'm proceeding and at worst I will be much more familiar with FAI shortly.

But the image is ready at this time if we changed course to meet the specific goals of an automated WooTaler (my nickname for it) installation. (And the docs are written for a non-automated general purpose install.)

Christian Grothoff

2020-04-13 11:01

manager   ~0015638

The FAI has the advantage that it works for bare-metal installations, which is what we will want for the Taler exchange deployment eventually. So you getting up to speed on FAI here has other advantages.

Eh, FAI doesn't require importing a filesystem *or* running an FAI server. Maybe you are reading up on something else!? See Debian handbook I quoted above, section 12.3.2.1, choices 1 or 2: local preseed.cfg, stored in initrd root or on the USB key. Very easy. (I'd use the 2nd option.). That said, FAI is just the base tool, likely we'll want the full Simple-CDD (12.3.3) in the end.

So all you do is (1) create the simple-sdd.conf and then given that configuration _anyone_ can (2) invoke build-simple-cdd to create an ISO for the installer image, and (3) run, qemu to boot from that installer image, plus a target drive to create the image. Finally, (4) run qemu against that image, and voila Wordpress/WooCommerce. Steps (2-4) can be in a shell script:

#!/bin/sh
# create installer
build-simple-cdd ARGS installer.img
# create VM image
qemu ARGS installer.img target.img
echo "You can now launch with $ qemu ARGS target.img"

(Admittedly, the shell script won't be that simple for long, given that you'd want to test for the tools like build-simple-cdd to be locally available, actually create the target.img, maybe take arguments for the size of the on-disk image, etc.).

This way, our Git will contain a tiny configuration file and a shell script and some instructions (README), and everything else, all binaries, can be generated by the user from source!

buckE

2020-04-15 09:32

developer   ~0015652

Ah. Well, okay. I think simple-cdd is a complicated solution to a simple problem, and if you always had this in mind it would have been nice to read in the original spec, *but* it's no problem.

> Eh, FAI doesn't require importing a filesystem *or* running an FAI server.

Actually I think it does, according to the docs (yes, the link you posted.) Using any method other than simple-cdd, FAI needs it for the basic setup (ex: initrd lives on the client) but even when using simple.cdd it appears it is necessary when doing upgrades (because debconf may require different options for new versions of Debian.) But that might not matter for this task, since upgrades might not be needed (or can be done with an upgrade script?) and for the larger task having a server will be a possibility I guess? FAI/simple-cdd won't be difficult, but it's definitely not a tool I would have used for this task without specific request or else this would have been done some time ago.

Anyway, let's call this the final spec (please) and I'll take care of it shortly.

Christian Grothoff

2020-04-15 10:44

manager   ~0015653

When I used FAI in the past, I didn't need a server. I still don't find anything in the handbook link I sent to suggest otherwise. And I'm not sure I see how it helps with upgrades, there clearly one needs to make a new configuration image. Anyway, if simple-cdd allows you to do it without, then just do that and we'll both be happy. ;-).

buckE

2020-04-16 09:08

developer   ~0015665

> When I used FAI in the past, I didn't need a server.

If you can provide the details of this, it will be useful information. Other than simple-cdd, each method I looked at did require this. s-cdd does not require a server (so far?) but it is a solution I would never have considered, still do not like for this task, and each attempted build is taking about 3 hours before failing and trying again, so it will take a while. But I take it that it is the solution you like for this, so I will simply proceed with it.

Christian Grothoff

2020-04-16 09:18

manager   ~0015666

Sorry, it was in the *distant* past (~ 20 years ago). Is s-cdd CPU, IO or bandwidth-bound? What takes so long?

buckE

2020-04-16 10:51

developer   ~0015667

The current error is:

build/debian-cd: Use of uninitialized value $default_desktop in regexp compilation at /home/buck.../update_tasks line 255

Errors like this are not out of the ordinary of course. This is just where I'm at at the moment. Creating the s-cdd structure took about 8 minutes, customizing the file was another 10. Now I'm just debugging this problem, and each edit to the preseed file (tasksel/desktop "none" or "false" or "tasksel-kde" etc) requires a rebuild (actually I learned it doesn't, but only after my second one.) It takes so long because (as I understand the doc), s-cdd builds a custom debian install CD based on the preseed (which will have defaults that make the installation non-interactive - isn't this what you said you wanted? I'm doing it because you've specifically asked, not because I think I'm on anything like a good track.)

I don't see any other examples of this error online, although there may be a tasksel bug regarding desktop environments for a close version, not sure yet.

To the other point:

> When I used FAI in the past, I didn't need a server.

The issue is server *or* existing filesystem. And of course that relevant to the tools I was looking at before you specified using simple-cdd. From the doc:

There are several places where the installer can get a preseeding file:

* in the initrd used to start the machine; in this case (meaning it requires an existing filesystem - that has initrd)

* on the boot media (which is explicitly out of scope, except that we're creating the boot media on the fly with simple-cdd, though I still don't see why)

* from the network (again, explicitly out of scope)

So anyway, that's where I am. Particularly annoying since this VM will not need a desktop. But that's life. Just working through an error message. But at least I think I don't have to rebuild each time anymore.

buckE

2020-04-17 10:44

developer   ~0015675

At this time I think there was a bug in the version of tasksel in buster. And another bug in the upgraded version.

Specifically: I upgraded my system as per Debian project's recommendations (really only recommended upgrading tasksel but I upgraded my build system to bullseye), got a new error (related to bullseye InRelease file, and error is the same on my custom build and the default my-simple-cdd files), and reported that to reportbug. Putting this on hold until I receive a response from Debian.

If using the Demo WooTaler site is a priority in the meantime, the VM can be made available and instructions are in woocommerce-taler.git/server-build/QEMU-autobuild/README.md.

Christian Grothoff

2020-04-17 10:57

manager   ~0015676

Ok. Please add a link here to the Debian bug. Putting up WooTaler is not a 'priority', it is/was just a good introductory task for you.

buckE

2020-04-18 06:12

developer   ~0015687

Bug is here:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=956958

Note that there was a different error on buster ($default_desktop not defined) and other bug reports suggested to me this could have been fixed in recent version of tasksel, which is why I upgraded to bullseye. It is not clear if that first error will reappear after resolving this issue, and I see that taler.net runs stretch so it may be necessary to upgrade tasksel if you really want to run simple-cdd when these are resolved.

> Putting up WooTaler is not a 'priority'

Okay. My nerd-brain has trouble with that because that is explicitly the task. Because "introductory task" aside, there is a question of which is a more elegant solution for this actual task:

1. upload a VM; run it with a reverse proxy

2. install pre-requisites for simple-cdd; git pull the preseed, postinst, etc. files; possbly install a custom version of tasksel; run simple-cdd; which downloads a custom Debian install CD; and creates a custom VM; and then runs a postinst script that; does another git pull for the buildWootaler.sh file; and builds the VM from step (1); which requires user intervention that is already completed in the Demo Site VM.

I have no complaint against simple-cdd per se but the original task reads:

"For this, we need a VM running WooCommerce with the Taler payment plugin, including _good_ instructions and/or shell scripts for how to set it up from scratch. The resulting VM should be integrated with demo.taler.net via a reverse-proxy setup configuration file in the existing nginx HTTPS server (for the respective virtual host name)."

(1), above, answers this (plus my instructions in the git repo)

So as someone who tends to be obsessively focused on staying in scope and making deliverables specific to specific tasks, I would like to suggest using the VM for this task, and then having a broader conversation about the larger automated build server idea, and simple-cdd will certainly be one part of that. I am skeptical it will be elegant for that eiher, but it is *cool* and it looks easy enough to use once these errors are out of the way.

Christian Grothoff

2020-04-19 00:38

manager   ~0015689

Sure, go ahead and use the image so we can close this one. Note that we will need to get a working simple-cdd image generator for the exchange deployment.

buckE

2020-04-20 05:11

developer   ~0015696

> Sure, go ahead and use the image so we can close this one.

Great! Currently rsync-ing two images to `/home/demo-blue`:

1 - WooTaler.img-Base - the base image, inside which you can run the build script (buildWooTaler.sh) if you want to (though I still do not understand the reason for this script, anyway it exists and it works.)

2 - WooTaler.img-LiveSite - a fully-functional WP/WC/GNUTaler site VM with sample product and appropriate copy (recommended)

Run at Host's :9999 with: "$ qemu-system-x86_64 -m 1024 -enable-kvm -k en_us -hda /home/demo-blue/WooTaler.img-LiveSite -net nic -net user,hostfwd=tcp::9999-:80"

Further instructions (not really needed) in woocommerce-taler.git/server-build/QEMU-autobuild/README.md

> Note that we will need to get a working simple-cdd image generator for the exchange deployment.

You mean the "big"/USB system you and I have talked about? Sure, if you have decided you want to use simple-cdd for the big project that's no problem.

Final note: woocommerce-taler/server-build/my-simple-cdd has the working-copy profiles/ files that could eventually build this WooTaler VM using simple-cdd once the bug is cleared up. (But I'd need to test them in a dev environment before recommending running them live.)

Christian Grothoff

2020-04-20 10:51

manager   ~0015699

Rationale for (1): we'll eventually want to do the full re-building from source in a continuous integration. Ideally _including_ building the base image (hence the simple-cdd for that). But that can wait.
As for the 'big' USB/System: yes, that's the one. As for 'decided', I just don't know anything else that would satisfy our requirements. If you have something, let me know. Now, I hope the result will be "tiny" (at least smaller than WooCommerce!).

buckE

2020-04-20 11:06

developer   ~0015703

> Now, I hope the result will be "tiny" (at least smaller than WooCommerce!).

Yeah I was surprised how big this VM needs to be.

> we'll eventually want to do the full re-building from source in a continuous integration

This is what I don't understand since there are so many things that will have to be customized in WP/WC manually, and each VM build will be singular and last a long time. So the time to create a new build script + inevitable manual GUI input is probably significantly more than the time to just update the single VM manually.

Also if it is not a single-use VM, and there are multiple projects using the VM-build script, they would all have to install and be comfortable with simple-cdd. Does that mean that Fedora users are out?

> As for 'decided', I just don't know anything else that would satisfy our requirements.

I haven't read the actual requirements/project details so I can't offer suggestions at this time (other than very generic thoughts like we discussed (ex, ansible, which I don't really want to use if it is avoidable.)

Current status: 1.7/1.8GB transferred on WooTaler.img-Base. I'll set the other going as soon as the first is done.

buckE

2020-04-20 11:33

developer   ~0015705

Current status: WooTaler.img-Base uploaded. WooTaler.img-LiveSite (recommended) uploading now

buckE

2020-04-20 23:12

developer   ~0015710

WooTaler.img-LiveSite (recommended) is uploaded.

buckE

2020-04-20 23:14

developer   ~0015711

> BFH students developed last year a Taler payment plugin for WooCommerce.

Changes made to plugin (mostly language/translation issues)

> We should add the resulting demonstrator as woocommerce.demo.taler.net.

Out of scope (re: requires root access)

> For this, we need a VM running WooCommerce with the Taler payment plugin,

VM available in /home/demo-blue

> including _good_ instructions and/or shell scripts for how to set it up from scratch.

Instructions and shell scripts available in woocommerce-taler.git

> The resulting VM should be integrated with demo.taler.net via a reverse-proxy setup configuration file in the existing nginx HTTPS server (for the respective virtual host name).

Out of scope (re: requires root access)

Issue History

Date Modified Username Field Change
2020-04-01 16:20 Christian Grothoff New Issue
2020-04-01 16:20 Christian Grothoff Status new => assigned
2020-04-01 16:20 Christian Grothoff Assigned To => buckE
2020-04-02 07:09 buckE Note Added: 0015486
2020-04-02 10:03 Christian Grothoff Note Added: 0015487
2020-04-02 10:06 Christian Grothoff Note Added: 0015488
2020-04-03 00:41 buckE Note Added: 0015493
2020-04-03 05:17 buckE Note Added: 0015494
2020-04-03 10:35 Christian Grothoff Note Added: 0015496
2020-04-04 03:07 buckE File Added: API-Key-Field.png
2020-04-04 03:07 buckE Note Added: 0015517
2020-04-04 10:16 Christian Grothoff Note Added: 0015523
2020-04-05 01:54 buckE Note Added: 0015530
2020-04-05 01:55 buckE Note Added: 0015531
2020-04-05 02:08 Christian Grothoff File Added: demo.site
2020-04-05 02:08 Christian Grothoff Note Added: 0015532
2020-04-05 04:56 buckE Note Added: 0015533
2020-04-05 12:21 Christian Grothoff Note Added: 0015534
2020-04-07 05:49 buckE Note Added: 0015539
2020-04-07 07:11 buckE Note Added: 0015543
2020-04-07 10:11 Christian Grothoff Note Added: 0015545
2020-04-08 06:51 buckE Note Added: 0015567
2020-04-08 06:56 buckE Note Added: 0015568
2020-04-08 13:59 Christian Grothoff Note Added: 0015582
2020-04-08 13:59 Christian Grothoff Assigned To buckE => Christian Grothoff
2020-04-08 19:19 Christian Grothoff Note Added: 0015591
2020-04-08 19:19 Christian Grothoff Assigned To Christian Grothoff => buckE
2020-04-08 21:34 buckE Note Added: 0015593
2020-04-08 21:53 Christian Grothoff Note Added: 0015594
2020-04-08 23:09 buckE Note Added: 0015595
2020-04-09 08:16 buckE Note Added: 0015598
2020-04-09 09:13 buckE Note Added: 0015599
2020-04-09 09:47 Christian Grothoff Note Added: 0015600
2020-04-10 01:48 buckE Note Added: 0015606
2020-04-10 09:22 buckE Note Added: 0015607
2020-04-10 11:50 Christian Grothoff Note Added: 0015610
2020-04-10 11:51 Christian Grothoff Note Added: 0015611
2020-04-11 06:45 buckE Note Added: 0015619
2020-04-11 11:11 Christian Grothoff Note Added: 0015621
2020-04-13 03:18 buckE Note Added: 0015632
2020-04-13 11:01 Christian Grothoff Note Added: 0015638
2020-04-15 09:32 buckE Note Added: 0015652
2020-04-15 10:44 Christian Grothoff Note Added: 0015653
2020-04-16 09:08 buckE Note Added: 0015665
2020-04-16 09:18 Christian Grothoff Note Added: 0015666
2020-04-16 10:51 buckE Note Added: 0015667
2020-04-17 10:44 buckE Note Added: 0015675
2020-04-17 10:57 Christian Grothoff Note Added: 0015676
2020-04-18 06:12 buckE Note Added: 0015687
2020-04-19 00:38 Christian Grothoff Note Added: 0015689
2020-04-20 05:11 buckE Note Added: 0015696
2020-04-20 10:51 Christian Grothoff Note Added: 0015699
2020-04-20 11:06 buckE Note Added: 0015703
2020-04-20 11:33 buckE Note Added: 0015705
2020-04-20 23:12 buckE Note Added: 0015710
2020-04-20 23:14 buckE Status assigned => resolved
2020-04-20 23:14 buckE Resolution open => fixed
2020-04-20 23:14 buckE Note Added: 0015711