View Issue Details

IDProjectCategoryView StatusLast Update
0005010Talerexchangepublic2021-09-02 18:22
ReporterChristian Grothoff Assigned ToChristian Grothoff  
PrioritynormalSeveritymajorReproducibilityrandom
Status closedResolutionfixed 
Platformi7OSDebian GNU/LinuxOS Versionsqueeze
Product Versiongit (master) 
Target Version0.4Fixed in Version0.4 
Summary0005010: DB serializability violations not always handled gracefully
DescriptionSometimes the DB plugin returns a "hard" error code for a serializabiltiy violation, causing the exchange to not retry the request, but go all 500 on the user.

We need to go over the code and properly identify all places where serializability violations can occur and double-check their handling.
Steps To ReproduceRun multiple instances of

taler-merchant-generate-payments -n 1000 -e -m

against one merchant/exchange backend produces errors like:

May 14 15:31:46-114418 taler-exchange-httpd-12796 ERROR Query failed at plugin_exchangedb_postgres.c:2065: could not serialize access due to read/write dependencies among transactions/Reason code: Canceled on identification as a pivot, during write./ERROR: could not serialize access due to read/write dependencies among transactions
DETAIL: Reason code: Canceled on identification as a pivot, during write.
HINT: The transaction might succeed if retried.
/PGRES_FATAL_ERROR/ERROR: could not serialize access due to read/write dependencies among transactions
DETAIL: Reason code: Canceled on identification as a pivot, during write.
HINT: The transaction might succeed if retried.

May 14 15:31:46-114486 taler-exchange-httpd-12796 ERROR Assertion failed at plugin_exchangedb_postgres.c:2483.
May 14 15:31:46-114502 taler-exchange-httpd-12796 ERROR Assertion failed at taler-exchange-httpd_db.c:872.

and

May 14 14:10:53-171686 taler-merchant-generate-payments-4409 ERROR Unexpected response code 500 to command withdraw-coin-3
{"error": "internal error", "code": 1108, "hint": "Failure in database interaction"}May 14 14:10:53-171790 taler-merchant-generate-payments-4409 ERROR Assertion failed at taler-merchant-generate-payments.c:680.


Configuration file in use for this against ~0.3 is attached.
TagsNo tags attached.

Activities

Christian Grothoff

2017-05-14 15:40

manager  

taler.conf (4,053 bytes)

Christian Grothoff

2017-05-25 02:07

manager   ~0012159

6bb2631..6029ed7 contains some initial work in this direction, adding a "state" to the sessions to track serializability violations, logic to test the PQ return code against this situation, and modifies some (!) of the functions to return #GNUNET_NO if such a violation is detected.

TODO:
* broaden the logic to all functions in the API
* actually handle the return code in all clients using the API
* make the same changes also in the auditor and merchant backend DB logic

Christian Grothoff

2017-06-25 13:05

manager   ~0012268

Exchange and auditor have been updated, merchant backend is still TBD.

Christian Grothoff

2017-06-25 13:11

manager   ~0012269

For the merchant, the payment generators (sometimes, after a while) fail IF one runs 2 merchant instances and 4 payment generators. While the merchant backend fails to log a proper error message, I suspect this is due to this issue not being fixed yet in the merchant backend.

Christian Grothoff

2017-06-29 19:25

manager   ~0012282

Fixed merchantdb handling of soft errors (by visual inspection); found that generate-payments test failed in part because of timeout given large values of '-n'. Adjusted timeout based on value given to '-n'.

Christian Grothoff

2021-09-02 18:14

manager   ~0018295

Fix committed to master branch.

Related Changesets

exchange: master a0c66e79

2017-06-19 20:12:00

Christian Grothoff

Details Diff
fix 0005010 for /track/transfer Affected Issues
0005010
mod - src/exchange/taler-exchange-httpd_track_transfer.c Diff File
mod - src/exchangedb/plugin_exchangedb_postgres.c Diff File
mod - src/exchangedb/test_exchangedb.c Diff File
mod - src/include/taler_exchangedb_plugin.h Diff File

exchange: master 703c54a2

2017-06-19 20:46:24

Christian Grothoff

Details Diff
fix 0005010 for /track/transaction Affected Issues
0005010
mod - src/exchange/taler-exchange-httpd_admin.c Diff File
mod - src/exchange/taler-exchange-httpd_db.c Diff File
mod - src/exchange/taler-exchange-httpd_db.h Diff File
mod - src/exchange/taler-exchange-httpd_track_transaction.c Diff File
mod - src/exchangedb/plugin_exchangedb_postgres.c Diff File
mod - src/exchangedb/test_exchangedb.c Diff File
mod - src/include/taler_exchangedb_plugin.h Diff File

exchange: master d4884c0c

2017-06-23 13:16:12

Christian Grothoff

Details Diff
Fix 0005010 for keystate Affected Issues
0005010
mod - src/exchange/taler-exchange-httpd_db.c Diff File
mod - src/exchange/taler-exchange-httpd_keystate.c Diff File
mod - src/exchangedb/perf_taler_exchangedb_interpreter.c Diff File
mod - src/exchangedb/plugin_exchangedb_postgres.c Diff File
mod - src/exchangedb/test_exchangedb.c Diff File
mod - src/include/taler_exchangedb_plugin.h Diff File

exchange: master 2d662e3f

2017-06-24 12:15:11

Christian Grothoff

Details Diff
fix 0005010 for taler-exchange-aggregator Affected Issues
0005010
mod - src/exchange/taler-exchange-aggregator.c Diff File
mod - src/exchangedb/plugin_exchangedb_postgres.c Diff File
mod - src/exchangedb/test_exchangedb.c Diff File
mod - src/include/taler_exchangedb_plugin.h Diff File

exchange: master c42d5447

2017-06-24 16:15:42

Christian Grothoff

Details Diff
convert rest of exchangedb plugin API to fix 0005010-issues Affected Issues
0005010
mod - src/auditor/taler-auditor.c Diff File
mod - src/exchangedb/plugin_exchangedb_postgres.c Diff File
mod - src/exchangedb/test_exchangedb.c Diff File
mod - src/include/taler_exchangedb_plugin.h Diff File

exchange: master 4c6ef743

2017-06-25 01:46:19

Christian Grothoff

Details Diff
update taler-auditor to fix 0005010 in auditor as well, technically done, but untested Affected Issues
0005010
mod - src/auditor/taler-auditor.c Diff File

merchant: master 515e970f

2017-06-29 18:54:04

Christian Grothoff

Details Diff
fix 0005010 for /track/transfer Affected Issues
0005010
mod - src/backend/taler-merchant-httpd_track-transfer.c Diff File

Issue History

Date Modified Username Field Change
2017-05-14 15:40 Christian Grothoff New Issue
2017-05-14 15:40 Christian Grothoff Status new => assigned
2017-05-14 15:40 Christian Grothoff Assigned To => Christian Grothoff
2017-05-14 15:40 Christian Grothoff File Added: taler.conf
2017-05-25 02:07 Christian Grothoff Note Added: 0012159
2017-06-25 13:05 Christian Grothoff Note Added: 0012268
2017-06-25 13:11 Christian Grothoff Note Added: 0012269
2017-06-29 19:25 Christian Grothoff Note Added: 0012282
2017-06-29 20:01 Christian Grothoff Status assigned => resolved
2017-06-29 20:01 Christian Grothoff Resolution open => fixed
2017-06-29 20:01 Christian Grothoff Fixed in Version => 0.4
2017-06-29 20:01 Christian Grothoff Target Version 0.5 => 0.4
2017-10-18 15:42 Christian Grothoff Status resolved => closed
2021-09-02 18:13 Christian Grothoff Changeset attached => Taler-exchange master 4c6ef743
2021-09-02 18:13 Christian Grothoff Changeset attached => Taler-exchange master c42d5447
2021-09-02 18:13 Christian Grothoff Changeset attached => Taler-exchange master 2d662e3f
2021-09-02 18:13 Christian Grothoff Changeset attached => Taler-exchange master d4884c0c
2021-09-02 18:13 Christian Grothoff Changeset attached => Taler-exchange master 703c54a2
2021-09-02 18:13 Christian Grothoff Changeset attached => Taler-exchange master a0c66e79
2021-09-02 18:14 Christian Grothoff Note Added: 0018295
2021-09-02 18:22 Christian Grothoff Changeset attached => Taler-merchant master 515e970f