Administrators

Administrators#

This chapter is for whoever installs the tracker, configures it, moves a MantisBT installation into it, backs it up and upgrades it. It assumes a Unix system, PostgreSQL and a reverse proxy in front.

locust is one binary. It keeps everything in PostgreSQL except attachment contents, which are files on disk; it serves its pages itself, and a reverse proxy in front of it terminates TLS.

Requirements#

  • PostgreSQL 15 or newer. The database user needs CREATE on the database (for the citext extension, which is a trusted extension).

  • Rust 1.88 or newer to build from source; the Debian package needs nothing but itself.

  • pg_dump, pg_restore and psql on PATH for locust backup, locust restore and locust import.

  • A mail relay, if the tracker is to send notifications.

  • No PHP, no web server modules, no cron entries.

Hardware: a tracker of the size of bugs.taler.net (ten thousand issues, fifty thousand notes) is comfortable on two cores and a gigabyte of memory. See Capacity for measured numbers.

Installing#

From source#

$ ./bootstrap                      # from a git checkout; a tarball has ./configure
$ ./configure --prefix=/usr/local
$ make
$ sudo make install

./configure --help lists the directory options it honours (--prefix, --bindir, --datadir, --docdir, --mandir, --sysconfdir); the rest of the autoconf vocabulary is accepted and ignored, so a packaging driver can run it unchanged. It looks for cargo and rustc (a missing one is an error), and for sphinx-build, pg_dump, pg_restore and psql (a missing one is a warning that says what will not work).

make install installs:

$PREFIX/bin/locust

the binary

$PREFIX/share/locust/config.d/locust.conf

the configuration defaults

$PREFIX/share/man/man1/locust.1

the manual page

$PREFIX/share/man/man5/locust.conf.5

the configuration format

$PREFIX/share/doc/locust/en/

this manual, served under /manual/

make install-conf additionally writes $SYSCONFDIR/locust/locust.conf if there is none; it never overwrites one. Without Sphinx the manual is skipped and everything else is installed as usual; ./configure --disable-manual says so deliberately.

The Debian package#

make deb builds it; apt install ./locust_*.deb installs it. The package creates the system user, /etc/locust/locust.conf (with a generated SECRET and the public URL you are asked for), a PostgreSQL role and a database owned by it, and ships site files for nginx and Apache in /etc/{nginx,apache2}/sites-available/.

It stops there. No schema, no administrator, and no question asked: how a tracker starts – new, or imported from MantisBT – is not something an unattended installation can answer. You run:

$ sudo -u locust locust migrate            # the schema
$ sudo -u locust locust create-admin       # or: locust import ...
$ sudo systemctl start locust

An upgrade migrates by itself: the package runs locust migrate --if-initialized, which brings an existing schema up to the package’s version and changes nothing when there is none.

Starting the service#

The package ships locust.service (enabled, not started until there is a schema) and locust-backup.timer (shipped disabled: where backups go and how long they are kept is your decision). Installed from source, a unit of three lines is enough: ExecStart=/usr/local/bin/locust serve as an unprivileged user, with Restart=on-failure.

locust serve refuses to start while migrations are pending, so a service that comes up is a service whose schema matches its binary.

The reverse proxy#

Put nginx or Apache in front for TLS. Three things matter:

  • Pass the Host header unchanged, so that absolute URLs are right.

  • Set X-Forwarded-For (nginx: $proxy_add_x_forwarded_for) and name the proxy’s address in [locust-rate-limits] TRUSTED_PROXIES, or every request looks like it comes from the proxy.

  • Allow request bodies as large as MAX_FILE_SIZE times file_upload_max_num, or uploads fail at the proxy.

  • Give the response header room: a page that lists issues names each of them in a cookie. nginx reads the header into one proxy_buffer_size (4k by default) and answers 502 when it does not fit; the shipped site sets 16k.

Over a UNIX socket (SERVE = unix) there is no peer address at all: the peer is a process on this host and is trusted, and the client is taken from X-Forwarded-For.

Configuring#

locust.conf is a GNU Taler configuration file – [sections] of OPTION = VALUE, case-insensitive names, case-sensitive values, YES/NO booleans, # and % comments, $VAR and ${VAR:-default} in options that name a file, @INLINE@ to read another file. locust.conf(5) and taler.conf(5) document it.

Files are read in this order, each option overriding the last:

  1. everything in $PREFIX/share/locust/config.d/ – the shipped defaults, replaced by upgrades, not to be edited;

  2. the file given with -c, or the first of $XDG_CONFIG_HOME/locust.conf, $HOME/.config/locust.conf, /etc/locust.conf, /etc/locust/locust.conf.

Every option has a built-in default, so a tracker with no configuration file at all serves on 127.0.0.1:8080 against postgres:///locust. locust config prints what the running configuration resolves to.

The sections#

[locust]

BASE_URL, the absolute public URL with its trailing slash (this is MantisBT’s $g_path: every absolute link and every mail is built from it). SECRET, a fixed random string for form tokens – unset, it changes at every start and forms left open stop working. ATTACHMENTS_DIR, where attachment contents live, as <dir>/<aa>/<sha256>. MANUAL_DIR, where this manual is installed (${DOCDIR} by default).

[locust-httpd]

SERVE = tcp with PORT and BIND_TO, unix with UNIXPATH and UNIXPATH_MODE (a stale socket file is replaced), or systemd for a socket the service manager hands over.

[locustdb-postgres]

CONFIG, the connection URI: postgres:///locust for the local socket, postgresql://user:password@host/dbname for a server; libpq’s key/value form is understood as well. MAX_CONNECTIONS, the pool’s size (16). No TLS is configured here: run PostgreSQL on the same host or on a private network.

[locust-policy]

The thresholds that decide visibility, with MantisBT’s names and defaults: ANONYMOUS_LEVEL (unset, every page needs a login), PRIVATE_PROJECT_THRESHOLD, VIEW_ISSUE_THRESHOLD, PRIVATE_ISSUE_THRESHOLD, PRIVATE_NOTE_THRESHOLD, SHOW_MONITORS_THRESHOLD. Levels are viewer, reporter, updater, developer, manager, administrator.

[locust-mail]

TRANSPORT: smtp://host:25, smtp://host:587?tls=required, smtps://user:password@host:465 or sendmail. Without it mail stays queued and locust send-mail sends it later.

[locust-rate-limits]

ENABLED, TRUSTED_PROXIES, and a sustained rate and burst per class: LOGIN, ACCOUNT (signup, lost password, verification links), WRITE (anything that is not a GET or HEAD), REST (per address) and REST_TOKEN (per token), EXPORT (CSV, Excel, Word, print, RSS, search). A rate of 0 switches a class off.

[locust-ai]

The door for language models: ENABLED, BOT_SIGNUP, the limits on one filed tree (TREE_MAX_ISSUES, TREE_MAX_DEPTH, TREE_MAX_BYTES), CONFIRM_EXPIRY_DAYS and REQUEST_ID_RETENTION_DAYS. ENABLED = NO removes the discovery documents, the JSON negotiation and /api/ai and changes nothing else. See Integrators.

Note

REAUTHENTICATION_EXPIRY in [locust-mantis] is four hours in locust, where MantisBT’s own default is five minutes. Five minutes was written for a password: with a second factor, a manager who edits a project, adds a category and dates a release types three one-time codes and waits for each. The window is extended by every page that asks, so continuous work never meets it – and it guards the account pages, the password page among them, as well as the administration ones, so four hours is four hours in which somebody at an unlocked screen can change a password. A tracker that wants it tighter sets it back.

[locust-ui]

What a page shows where locust departs from MantisBT. Each option has MantisBT’s own behaviour as its other value, so one line gets MantisBT’s page back. PERM_LOGIN_DEFAULT (YES): the login form’s “Save my login” starts ticked, where MantisBT leaves it clear; a request that carries perm_login is obeyed either way. VERSION_SORT (date): the three version columns sort by the day the release is scheduled for, undated last in both directions, where MantisBT (name) sorts by the version’s name. MAX_PER_PAGE (250): the most issues one page of a list shows, whatever the filter’s “Show” asks for, where MantisBT has no ceiling and reads “all” as all of them. A page of several hundred issues names each of them in a cookie, and that response header outgrows what the proxy reads it into, so the page – and the filter form on it – answers 502; 0 is MantisBT’s behaviour, and wants proxy_buffer_size raised with it. The print view and the CSV/Excel export are not paged and are not affected. SUMMARY_CACHE_SECONDS (86400): how long the summary statement’s answer – what both planning tabs are drawn from, and the expensive statement on the tracker – is kept in the process. Every write throws the whole cache away as it finishes, so what the day covers is a day passing, not a change. Two servers against one database see each other’s writes when the day is up; a tracker that wants the charts exact sets this to 0.

[locust-mantis]

MantisBT’s own options under MantisBT’s names without the $g_ prefix – the ones that live in config_inc.php. A boolean is YES/NO, a number is a number, a string is itself, and a list or a map is JSON (which is also how a threshold takes a set of levels: ADD_BUGNOTE_THRESHOLD = [25, 90]). An option name that is not MantisBT’s is an error, and so is a number that is not one: locust refuses at startup where PHP would have used 0.

Two MantisBT defaults are deliberately not kept. MantisBT hides the due date from everybody (due_date_view_threshold and due_date_update_threshold both nobody); locust shows it to anybody who may see the issue and lets a developer set it, because that field is what its deadlines are made of. Setting both options back to nobody restores MantisBT’s silence. And MantisBT keeps the summary for managers (view_summary_threshold = MANAGER); locust gives it to developers, and with it the whole Summary tab bar – the counts, the graphs, the plan, the resource view and the priorities – because people doing the work cannot argue with a plan they may not read. VIEW_SUMMARY_THRESHOLD = MANAGER puts it back.

Configuration in the database#

MantisBT keeps the options its own administration pages set in a database table, not in a file, and locust does the same: those rows apply on top of locust.conf, per project and per user where MantisBT resolves them that way.

The configuration page

Figure 28 Manage → Configuration: the options stored in the database, with where each value comes from.#

So an option can be set in three places, and the later wins: the file, an all-projects row, a project’s row. The page says which it is. This is also where a project’s bug_report_page_fields is set – the list that decides which optional fields the report form shows, and the way to give one project a short form (Managers).

The overview page

Figure 29 Manage → Overview: versions, paths and what this installation is made of.#

Users and access#

The user administration page

Figure 30 Manage → Users: accounts, their level, and when each was last seen.#

  • An account has a global access level and, per project, a membership that overrides it for that project. Membership applies to exactly one project and is not inherited by its subprojects – that is MantisBT’s rule, and imported permissions therefore behave exactly as they did.

  • Deleting a user leaves a disabled placeholder behind, so that their issues and notes keep an author. The name is freed.

  • Dormant accounts: the box under the user list deletes accounts that have not been seen for a given number of years and that nothing refers to – no issue, no note, nothing. It shows the list before it deletes anything.

  • Two-factor authentication: a project can require it from a level upwards (Managers). When somebody loses their device, Deactivate OTP on their user page clears the secret and applies every project requirement to them again.

  • Signup follows MantisBT’s allow_signup; addresses at throwaway providers can be blocked, and failed logins lock an account as MantisBT’s lockout does.

locust create-admin makes the first administrator from the command line; locust prune-dormant --years N does from the command line what the dormant box does from the page, with --dry-run.

Mail#

Notifications are written by the statement that causes them, into email_outbox, in the same transaction. A worker inside locust serve renders and sends them, so nothing is lost when the relay is down and nothing is sent twice when a request fails.

  • Without [locust-mail] TRANSPORT mail stays queued; locust send-mail sends what is due, once (useful from a timer, or to flush a queue after a relay outage).

  • The sender address and name come from MantisBT’s configuration (from_email, from_name, return_path_email), overrides included.

  • Who is mailed about what is MantisBT’s notify_flags and each user’s preferences.

  • SELECT count(*) FROM email_outbox WHERE sent_at IS NULL is the queue depth, and the thing to watch after a cutover.

Rate limits and abuse#

[locust-rate-limits] ENABLED = YES switches on token buckets per client and class; over a limit a request gets 429 with Retry-After instead of the page. Behind a proxy, TRUSTED_PROXIES must name it or every client shares one bucket.

Besides them, locust reproduces MantisBT’s own protections: the failed-login lockout (a locked account is refused without counting, a wrong password counts, disabled accounts included), the limit on lost-password requests, and the disposable-address check at signup.

Importing from MantisBT#

$ createdb locust
$ locust migrate
$ locust import --from "host=/var/run/postgresql dbname=mantis" \
    --crypto-master-salt '<the $g_crypto_master_salt of config_inc.php>'

The import reads a MantisBT 2.0–2.28 database on PostgreSQL (convert a MySQL installation first, with pgloader, and import the converted copy), writes everything in one transaction, copies attachments into ATTACHMENTS_DIR, and then compares the result with the source record by record and prints the comparison. Issue ids are preserved.

Options worth knowing:

--crypto-master-salt

MantisBT’s $g_crypto_master_salt. With it, every user’s RSS feed URL stays valid. Without it, they all change.

--mantis-upload-dir

Where the files are, if MantisBT was configured with $g_file_upload_method = DISK. The directory must be readable by the importing user.

--table-prefix / --table-suffix

MantisBT’s $g_db_table_prefix and suffix, if they are not the defaults.

--verify-only

Compare an already imported database with the source again.

What the importer refuses rather than guesses:

  • Usernames that differ only in case. locust’s names are unique regardless of case; it stops and lists them so that you rename one first.

  • Access levels MantisBT does not ship. Levels are a PostgreSQL enum; a tracker that invented one must map it first.

  • Several parents for one project. MantisBT allows it, locust keeps one and reports the others.

Read the report before you continue: skipped, unsupported and warnings must be empty or understood, and every verification line must read ok.

What does not come over: sessions (everybody logs in once more), plugins (locust has none and shows no trace of one), and MantisBT’s signup_use_captcha, which is ignored. Passwords, API tokens and RSS keys do come over; MD5 password hashes are upgraded to Argon2id at each user’s first login.

The cutover#

The checklist below was rehearsed end to end (compat/rehearse.py in the source tree) and is the one bugs.taler.net and bugs.gnunet.org follow.

It is a smaller step than it looks: locust serves MantisBT’s pages at MantisBT’s page names without the .php, and every old URL – view.php?id=N, index.php, a form posting to bug_update.php – answers with a permanent redirect (308) to its new one, while api/rest/... stays exactly as it is. Links in commit messages, mails and other trackers keep working, scripts keep working, and the switch is a change of what the reverse proxy forwards to. MantisBT stays installed, frozen, for the rollback.

Before the day#

  • Rehearse on a copy of the real data. Restore last night’s dump, import into a scratch database, read the report (above). Time it: the import is linear in the data.

  • Configuration ready and reviewed, in particular:

    • [locust] BASE_URL, SECRET, ATTACHMENTS_DIR (on a disk with room for the attachments, deduplicated).

    • [locust-mantis]: the options from MantisBT’s config_inc.php – not the ones its web pages set, which are rows in its database and come with the import. In particular MAX_FILE_SIZE must be MantisBT’s effective limit, which is $g_max_file_size capped by PHP’s upload_max_filesize and post_max_size. The rehearsal showed every issue page differing when this was left at MantisBT’s own default.

    • [locust-mail] TRANSPORT and [locust-rate-limits] with TRUSTED_PROXIES.

  • Reverse proxy site for locust prepared and not yet enabled.

  • Backups scheduled, and one restore tried.

  • Announcement: the time, the read-only window, and that everybody logs in once more afterwards.

The day#

Step

What

Rehearsed

1

Freeze MantisBT: the proxy answers 503 to everything that is not a GET or HEAD, except the login and logout pages. Reading and logging in keep working.

0.5 s

2

Dump: pg_dump --format=custom mantis > mantis.dump. A consistent snapshot; the frozen MantisBT still writes last-visit times, which is why the import reads a copy.

0.7 s

3

Frozen copy: createdb mantis_frozen && pg_restore --no-owner -d mantis_frozen mantis.dump

0.6 s

4

locust database: createdb locust && locust migrate

0.3 s

5

Import and verify: locust import --from "dbname=mantis_frozen" --crypto-master-salt ...; every verification line must read ok.

4.5 s

6

Attachment files: locust check-attachments --deep

< 0.1 s

7

Start locust and check GET /healthz

8

Switch the proxy to locust, still frozen: smoke-check as a reporter, a developer and an administrator – an issue, the issue list, My View, the REST API with a token.

0.6 s

9

Lift the freeze. From here on, changes exist only in locust.

10

Backup: locust backup --out /backup/locust

0.7 s

The times are for a synthetic database at the scale of bugs.taler.net (10,095 issues, 50,100 notes, 98,994 history rows, 2,019 attachments, 1,015 users; 71 MB in MantisBT) on a 24-core host with a local PostgreSQL 17. At five times that size the import and its verification take 21.4 s and 277 MB of memory.

Rollback#

Until step 9 the rollback is only the proxy: point it back at MantisBT and lift the freeze. Nothing was written anywhere.

After step 9, locust holds changes MantisBT does not have, and there is no way back that keeps them. Decide within the smoke-check window. Keep MantisBT’s frozen database and the dump until the new installation has run for a while.

After the switch#

  • The mail queue falls back to zero and the relay log shows deliveries.

  • Scripts using the REST API work unchanged – both api/rest/<route> and api/rest/index.php/<route> are served.

  • RSS readers still get their feeds (if you passed the master salt).

  • The log shows no rate-limit lines for ordinary use; raise a class if it does.

  • The first scheduled backup ran, and locust check-attachments is clean.

  • After a quiet period, retire MantisBT – keeping the last dump.

Backups#

$ locust backup --out /backup/locust

writes /backup/locust/locust-<UTC time>.dump (pg_dump --format=custom) and copies the attachment files that are new since the last backup into /backup/locust/attachments/.

Dump first, files second is consistent on purpose: files are named by their SHA-256, written before any row refers to them, and never changed or deleted, so every file a dump refers to already exists. Keep several dumps; they share the one attachment directory. Copy both off-site with anything that copies new files.

$ createdb locust_new
$ locust restore --dump /backup/locust/locust-<time>.dump

loads a dump into the configured database, which must be empty, copies the backup’s attachment files into ATTACHMENTS_DIR and checks them.

locust check-attachments verifies that every attachment row has its file; --deep also recomputes each file’s SHA-256.

The Debian package ships locust-backup.timer, disabled.

Upgrading#

  1. Back up (locust backup).

  2. Install the new binary.

  3. locust migrate – or let the package do it; it runs migrate --if-initialized. --backup-dir DIR dumps the database first, and migrates nothing if the dump fails.

  4. Restart the service.

locust migrate exits 78, changing nothing, if the database’s schema is newer than the program or if a migration was changed after it was applied. Migrations are compiled into the binary and checksummed, so there is no SQL directory to keep in step.

Downgrades are not supported: restore the backup.

Monitoring#

GET /healthz

200 and ok while the database answers, 503 otherwise. No authentication, no data; this is what a load balancer should poll.

Logs

RUST_LOG sets the level (warn is a good default, info for a new installation). Rate-limit rejections, mail failures and migration decisions are logged.

Database

The usual PostgreSQL metrics. A page of this tracker costs one or two statements whatever the data size, so a rising statement rate means more traffic and not a slow page.

Capacity#

Measured against MantisBT 2.28.4 (php-fpm behind nginx) on the same synthetic data, a release build of locust with 16 database connections:

Page

MantisBT (req/s)

locust (req/s)

Issue list

412

974

One issue

727

890

My View

357

436

REST, a page of 50 issues

114

353

Adding a note (web form)

240

452

Adding a note (REST)

317

736

The difference is statements per request: 49 to 683 in MantisBT, 2 to 8 in locust.

Agents#

[locust-ai] ENABLED decides whether this tracker is legible to language models at all. With it on, every GET page that shows data also answers JSON when asked, /llms.txt and /.well-known/ai describe the tracker, and /api/ai takes writes from accounts marked as agents. The rules are not advisory: an agent’s account has a name beginning with bot-, no password, and a human’s address behind it who had to confirm it; it may never delete anything or rewrite somebody else’s words, and every account of one is marked as such wherever its name appears. BOT_SIGNUP = NO keeps agents from registering at all; disabling an account stops it like any other.

Integrators describes the endpoints.

This manual#

The manual is built by Sphinx from docs/manual in the source tree and installed as HTML into $DOCDIR/<language>/; the tracker serves it under /manual/<language>/ from wherever [locust] MANUAL_DIR points, and /manual/ sends a reader to their own language, or to English. It costs no database query and needs no login.

  • make manual builds it, make install installs it.

  • Without sphinx-build there is no manual, and /manual/ answers 404 – the tracker is otherwise unaffected.

  • Another language is its code in MANUAL_LANGUAGES, make manual-gettext, a translated catalogue and a rebuild. The tracker needs no change: it serves every language directory it finds, and the footer link follows each reader’s own language.

There is no version in any of this: an installation serves the manual of the locust that is running.

Troubleshooting#

What you see

What it usually is

locust serve exits at once, naming migrations

The schema is older than the binary: run locust migrate.

locust migrate exits 78

The database is newer than the binary, or an applied migration was edited. Do not force it; use the matching binary.

Every page needs a login, including the public ones

[locust-policy] ANONYMOUS_LEVEL is unset.

Uploads fail at a size the page allows

The reverse proxy’s body limit, not locust’s.

One page answers 502 while the rest of the tracker is fine

The response header did not fit the proxy’s buffer: raise proxy_buffer_size (nginx). A page of a list shows at most 250 issues precisely so that it cannot grow one that does not fit.

All requests appear to come from one address, and rate limits bite

TRUSTED_PROXIES does not name the proxy.

Mail is not delivered and the queue grows

No [locust-mail] TRANSPORT, or the relay refuses it; the log says which.

Forms say the token is invalid after a restart

[locust] SECRET is unset, so tokens change at every start.

/manual/ answers 404

No manual installed, or MANUAL_DIR points elsewhere.

An issue “does not exist” for one user

It is private, or in a project they are not a member of: hidden and nonexistent look the same by design.