0008773: stale fuse-overlayfs processes leftover from podman

ID	Project	Category	View Status	Date Submitted	Last Update
0008773	Taler	deployment and operations	public	2024-04-25 14:58	2024-05-15 16:03

Reporter	dvn	Assigned To	dvn	Resolution deadline
Implementer		Reviewer
Release signed by		Deployed by
Verifier		Communicator
Data protection authority notification required	No
CSIRT notification required	No
Estimated effort (hours)		Actual effort (hours)	0
Fast-track	No
Work package	none
Priority	normal	Severity	minor	Reproducibility	sometimes
Status	closed	Resolution	fixed
Target Version	0.11

Summary	0008773: stale fuse-overlayfs processes leftover from podman
Description	On gv.taler.net there are many idle fuse-overlayfs processes apparently leftover from podman. Running `ps aux \| grep fuse-overlayfs \| wc -l` resulted in 458 items. Podman should be removing these mounts when a container shuts down, but there is a known bug that prevents this under some (unknown) circumstances. See: https://github.com/containers/podman/issues/7061 We may need to implement our own cleanup of these on a cron.
Tags	No tags attached.

Date Modified	Username	Field	Change
2024-04-25 14:58	dvn	New Issue
2024-04-25 14:58	dvn	Status	new => assigned
2024-04-25 14:58	dvn	Assigned To	=> dvn
2024-04-25 15:01	dvn	Note Added: 0022307
2024-05-02 16:25	dvn	Status	assigned => informing
2024-05-02 16:25	dvn	Resolution	open => fixed
2024-05-02 16:25	dvn	Note Added: 0022329
2024-05-15 16:02	Christian Grothoff	Target Version	=> 0.11
2024-05-15 16:03	Christian Grothoff	Status	informing => closed

dvn 2024-04-25 15:01 reporter ~0022307	I've figured out a command to clean these up: `podman unshare umount $(podman unshare mount \| grep fuse \| grep -v connections \| cut -d' ' -f3)`

dvn 2024-05-02 16:25 reporter ~0022329	Added a cronjob on user `container-worker` that runs the aforementioned command to remove stale fuse mounts at 05:00 every day. It has been running for a few days and appears to be working as intended.