View Issue Details

IDProjectCategoryView StatusLast Update
0006906Talerwallet-corepublic2021-08-24 16:22
Reportersebasjm Assigned ToFlorian Dold  
PriorityhighSeveritymajorReproducibilityalways
Status closedResolutionfixed 
Product Versiongit (master) 
Target Version0.8Fixed in Version0.8 
Summary0006906: problems using wallet-core in a project with webpack
DescriptionTrying to use storybook with wallet-web-extension and the latest code I get 3 error in the build process [1] that seems to be related to the module format.
After exploring for some time I found that webpack has some issues with the new format [2]

Using webpack 5 seems to have to module format parsed correctly but I get some other problems with the import that are easily addressable. [3]
Storybook use webpack 4 but there is a migration to webpack 5 still in beta.

I have created a repo that reproduce the problem, but should we add this as a integration test into the wallet-core repo?
This example does not reproduce 2 of 3 errors (related to using the break) which I think it may be related to a webpack configuration.

https://github.com/sebasjm/wallet-core-webpack-packaging

To reproduce:
 * pnpm install
 * pnpm install $WALLET_CORE_FOLDER/packages/taler-wallet-core/
 * pnpx webpack

$ pnpx --version
6.2.1

[1] 3 errors found in storybook build process

ERROR in ../taler-wallet-core/lib/crypto/primitives/nacl-fast.js 14:44
Module parse failed: Unexpected token (14:44)
File was processed with these loaders:
 * ../../node_modules/.pnpm/babel-loader@8.2.2_105470dfa535e5f605a4a54ef7d81e87/node_modules/babel-loader/lib/index.js
You may need an additional loader to handle the result of these loaders.
| // We need this require function to synchronously
| // import the "crypto" module in the CSPRNG initialization.
> require = mod.default.createRequire(import.meta.url);
| }
|
 @ ../taler-wallet-core/lib/crypto/talerCrypto.js 36:0-50 54:9-25 170:13-39 174:9-29 177:11-20 184:10-25 185:9-18 188:22-56 189:10-25 190:9-18 318:18-34 326:18-34 334:9-18 337:13-39 338:9-27 341:9-34 344:13-27
 @ ../taler-wallet-core/lib/index.js
 @ ./src/wxApi.ts
 @ ./src/wallet/Withdraw.tsx
 @ ./src/wallet/Withdraw.stories.tsx
 @ ./src sync ^\.(?:(?:^|\/|(?:(?:(?!(?:^|\/)\.).)*?)\/)(?!\.)(?=.)[^/]*?\.stories\.tsx)$
 @ ./.storybook/generated-stories-entry.js

ERROR in ../taler-wallet-core/lib/operations/reserves.js 1136:20
Module parse failed: Unsyntactic break (1136:20)
File was processed with these loaders:
 * ../../node_modules/.pnpm/babel-loader@8.2.2_105470dfa535e5f605a4a54ef7d81e87/node_modules/babel-loader/lib/index.js
You may need an additional loader to handle the result of these loaders.
| return Promise.resolve(processReserveImpl(ws, reservePub, true)).then($return, $error);
| } else {
> break;
| }
| } catch ($boundEx) {
 @ ../taler-wallet-core/lib/wallet.js 43:0-119 128:31-45 697:29-42 710:19-38 849:21-47 884:14-28
 @ ../taler-wallet-core/lib/index.js
 @ ./src/wxApi.ts
 @ ./src/wallet/Withdraw.tsx
 @ ./src/wallet/Withdraw.stories.tsx
 @ ./src sync ^\.(?:(?:^|\/|(?:(?:(?!(?:^|\/)\.).)*?)\/)(?!\.)(?=.)[^/]*?\.stories\.tsx)$
 @ ./.storybook/generated-stories-entry.js

ERROR in ../taler-wallet-core/lib/operations/transactions.js 491:12
Module parse failed: Unsyntactic break (491:12)
File was processed with these loaders:
 * ../../node_modules/.pnpm/babel-loader@8.2.2_105470dfa535e5f605a4a54ef7d81e87/node_modules/babel-loader/lib/index.js
You may need an additional loader to handle the result of these loaders.
| return Promise.resolve(processWithdrawGroup(ws, withdrawalGroupId, true)).then(function ($await_27) {
| try {
> break;
| } catch ($boundEx) {
| return $error($boundEx);
 @ ../taler-wallet-core/lib/wallet.js 46:0-97 1154:31-46 1363:31-48 1373:31-47
 @ ../taler-wallet-core/lib/index.js
 @ ./src/wxApi.ts
 @ ./src/wallet/Withdraw.tsx
 @ ./src/wallet/Withdraw.stories.tsx
 @ ./src sync ^\.(?:(?:^|\/|(?:(?:(?!(?:^|\/)\.).)*?)\/)(?!\.)(?=.)[^/]*?\.stories\.tsx)$
 @ ./.storybook/generated-stories-entry.js



[2]
https://www.npmjs.com/package/@open-wc/webpack-import-meta-loader
https://github.com/webpack/webpack/issues/6719


[3] errors found in webpack 5, hiding repeated errors


ERROR in ../../taler/wallet-core/packages/taler-wallet-core/lib/crypto/primitives/nacl-fast.js 7:0-30
Module not found: Error: Can't resolve 'module' in '/home/sebasjm/Work/taler/wallet-core/packages/taler-wallet-core/lib/crypto/primitives'
 @ ../../taler/wallet-core/packages/taler-wallet-core/lib/crypto/talerCrypto.js 19:0-50 33:11-27 129:17-43 133:11-31 136:15-24 141:14-29 142:11-20 145:26-60 146:14-29 147:11-20 259:22-38 264:22-38 269:11-20 272:17-43 273:11-29 276:11-36 279:15-29
 @ ../../taler/wallet-core/packages/taler-wallet-core/lib/index.js 35:0-40 35:0-40
 @ ./src/index.js 3:0-70



ERROR in ../../taler/wallet-core/packages/taler-wallet-core/lib/crypto/workers/cryptoImplementation.js 29:0-42
Module not found: Error: Can't resolve '../../util/timer' in '/home/sebasjm/Work/taler/wallet-core/packages/taler-wallet-core/lib/crypto/workers'
Did you mean 'timer.js'?
BREAKING CHANGE: The request '../../util/timer' failed to resolve only because it was resolved as fully specified
(probably because the origin is a '*.mjs' file or a '*.js' file where the package.json contains '"type": "module"').
The extension in the request is mandatory for it to be fully specified.
Add the extension to the request.
 @ ../../taler/wallet-core/packages/taler-wallet-core/lib/index.js 33:0-80 33:0-80
 @ ./src/index.js 3:0-70



ERROR in ../../taler/wallet-core/packages/taler-wallet-core/lib/crypto/workers/nodeThreadWorker.js 16:0-20
Module not found: Error: Can't resolve 'os' in '/home/sebasjm/Work/taler/wallet-core/packages/taler-wallet-core/lib/crypto/workers'

BREAKING CHANGE: webpack < 5 used to include polyfills for node.js core modules by default.
This is no longer the case. Verify if you need this module and configure a polyfill for it.

If you want to include a polyfill, you need to:
    - add a fallback 'resolve.fallback: { "os": require.resolve("os-browserify/browser") }'
    - install 'os-browserify'
If you don't want to include a polyfill, you can use an empty module like this:
    resolve.fallback: { "os": false }
 @ ../../taler/wallet-core/packages/taler-wallet-core/lib/index.js 32:0-53 32:0-53
 @ ./src/index.js 3:0-70


TagsNo tags attached.

Activities

Florian Dold

2021-06-17 14:33

manager   ~0017963

I think the errors we get with wp5 are due to import statements sometimes not using the ".js" extension.

That is, we should always write

  import { runBackupCycle } from "./operations/backup.js";

and *not*

  import { runBackupCycle } from "./operations/backup";

Sebastian: Any idea what the issues in wp4 are, besides the missing .js extensions?

sebasjm

2021-06-17 14:40

developer   ~0017964

webpack 4 doesnt like this line

require = mod.default.createRequire(import.meta.url);

here https://git.taler.net/wallet-core.git/tree/packages/taler-wallet-core/src/crypto/primitives/nacl-fast.ts#n15

sebasjm

2021-06-17 14:48

developer   ~0017965

> I think the errors we get with wp5 are due to import statements sometimes not using the ".js" extension.

Yes, a rename of the imports will work.

There are still two others problems:

* Can't resolve 'module' here https://git.taler.net/wallet-core.git/tree/packages/taler-wallet-core/src/crypto/primitives/nacl-fast.ts#n8

Which I think we may replace with a dynamic import (maybe, not sure)

* Can't resolve 'os' here https://git.taler.net/wallet-core.git/tree/packages/taler-wallet-core/src/crypto/workers/nodeThreadWorker.ts

Same as above i think but I don't know how and where it is used to decide if we should use a polyfill or just dynamic import to handle the absence.

Other strategy could be to have 1 entry point for nodejs environment and other for browser environment which IMHO will end up in better code.

Florian Dold

2021-06-17 15:02

manager   ~0017966

Can you elaborate how you would do the multiple entry points?

The problem with using dynamic ES6-style imports is that:
* import(...) is asynchronous
* rollup (and AFAICT also webpack) doesn't support top-level await, so we'd have to restructure the code that accesses crypto functions, as the random number generator would have to be initialized asynchronously.

Let me know if you have any other ideas, too!

sebasjm

2021-06-17 15:31

developer   ~0017967

> Can you elaborate how you would do the multiple entry points?

Using the browsing field in the package json seems promising
https://docs.npmjs.com/cli/v7/configuring-npm/package-json#browser

Although I don't see the "type" and "module" fields there, so this behavior seems to be decided with the bundler.


If this browser field thing in the package_json does not work, an ugly but easy alternative could be

browser should import:

import { someApi } from '@gnu-taler/taler-wallet-core/browser'

nodejs apps should import:

import { someApi } from '@gnu-taler/taler-wallet-core/node'


this folders will have the index.js file with different setup but exporting the same API.

sebasjm

2021-06-17 15:36

developer   ~0017968

So an action plan could be:
 * rename imports to have the .js
 * add browser field in package json with different entry point
 * splint index file in wallet-core: browser-index.ts, node-index.ts and common-index.ts
 * configure rollup
 * refactor code to move require of "os" and "crypto" to node-index and equivalent into browser-index.ts
 * profit

sebasjm

2021-06-17 16:02

developer   ~0017969

FYI
https://github.com/webpack/webpack/issues/4674#issuecomment-355853969

sebasjm

2021-06-17 17:06

developer   ~0017970

profit achieved

I will fix imports and push into master if no comments, I still having this problem with webpack 4 that I will try to fix it after push


Module parse failed: Unsyntactic break (491:12)
File was processed with these loaders:
 * ../../node_modules/.pnpm/babel-loader@8.2.2_105470dfa535e5f605a4a54ef7d81e87/node_modules/babel-loader/lib/index.js
You may need an additional loader to handle the result of these loaders.
| return Promise.resolve(processWithdrawGroup(ws, withdrawalGroupId, true)).then(function ($await_27) {
| try {
> break;
| } catch ($boundEx) {
| return $error($boundEx);

sebasjm

2021-06-17 17:53

developer   ~0017972

fixed in 46f3fcbb..1c7423db

sebasjm

2021-06-17 18:22

developer   ~0017973

The missing issue with "Unsyntactic break" was related to babel.
Any project that use babel will not able to load wallet-core if wallet-core contains code that use break statement inside a block scoped case of a switch....case statement. Very rare.

https://github.com/babel/babel/issues/8709

Fixed in a4902de8..954ed239

Issue History

Date Modified Username Field Change
2021-06-17 14:22 sebasjm New Issue
2021-06-17 14:22 sebasjm Status new => assigned
2021-06-17 14:22 sebasjm Assigned To => Florian Dold
2021-06-17 14:33 Florian Dold Note Added: 0017963
2021-06-17 14:40 sebasjm Note Added: 0017964
2021-06-17 14:48 sebasjm Note Added: 0017965
2021-06-17 15:02 Florian Dold Note Added: 0017966
2021-06-17 15:31 sebasjm Note Added: 0017967
2021-06-17 15:36 sebasjm Note Added: 0017968
2021-06-17 16:02 sebasjm Note Added: 0017969
2021-06-17 17:06 sebasjm Note Added: 0017970
2021-06-17 17:53 sebasjm Note Added: 0017972
2021-06-17 18:22 sebasjm Note Added: 0017973
2021-06-18 13:30 sebasjm Status assigned => resolved
2021-06-18 13:30 sebasjm Resolution open => fixed
2021-07-30 13:56 Christian Grothoff Fixed in Version => 0.8
2021-07-30 13:59 Christian Grothoff Target Version git (master) => 0.8.1
2021-07-30 14:01 Christian Grothoff Target Version 0.8.1 => 0.8
2021-08-24 16:22 Christian Grothoff Status resolved => closed
2023-04-13 20:36 Florian Dold Category wallet (TS core) => wallet-core