We occasionally encounter the error:
Command error:
docker: failed to prepare extraction snapshot "extract-[...]": NotFound: lease does not exist: not found
when trying to run a Nextflow job that requires a docker container at the same time when our timed systemd service executes: docker system prune -fa
I tried to research the error but only found an obscure gitlab bug, but our error seems image and repository independent as it has occurred multiple times already.
The logs reveal that the error occurs shortly after the timer running the docker system prune -fa but the logs from the prune are not listing the requested image at all (not among the deleted ones).
I am wondering whether there is some race condition where docker system prune -fa also resets leases which then might fail the preparation steps as described in the error message.
I was unable to reproduce this issue so far, but it has occurred multiple times.
