Containers end up in non repairable failed state: Task Error removeVolume: Error removing volume - volume has active mounts

[makemrproper]

Hi

I have suddenly been seeing more and more apps ending up in an unrecoverable state. Last week I had 1 app in this state. Today I'm seeing 4 apps in the same error condition. Luckily none of my business critical containers are affected yet. It seems to be "spreading" Please help!!!

ERRORS as seen from the web interface
I am seeing this from the Container Info view in the web interface, per container. The error message and condition is the same across all four containers.

Task Error

If a configuration, update, restore or backup action resulted in an error, you can retry the task.

An error occurred during the configure operation: Docker Error: removeVolume: Error removing volume: (HTTP code 500) server error - remove e33adee2-f48d-48e8-8777-e281413d2576-localstorage: volume has active mounts

ERRORS as seen in the LOG

The log for the container contains these recent entries:

Dec 13 10:05:30 box:apptask hostname.com error reconfiguring : BoxError: removeVolume: Error removing volume: (HTTP code 500) server error - remove e33adee2-f48d-48e8-8777-e281413d2576-localstorage: volume has active mounts

Dec 13 10:05:30 box:apptask hostname.com updating app with values: {"installationState":"error","error":{"message":"removeVolume: Error removing volume: (HTTP code 500) server error - remove e33adee2-f48d-48e8-8777-e281413d2576-localstorage: volume has active mounts ","reason":"Docker Error","taskId":"6632","installationState":"pending_configure"}}

Dec 13 10:05:30 box:taskworker Task took 3.779 seconds

Dec 13 10:05:30 box:tasks setCompleted - 6632: {"result":null,"error":{"stack":"BoxError: removeVolume: Error removing volume: (HTTP code 500) server error - remove e33adee2-f48d-48e8-8777-e281413d2576-localstorage: volume has active mounts \n at Object.callback (/home/yellowtent/box/src/docker.js:686:64)\n at /home/yellowtent/box/node_modules/dockerode/lib/volume.js:81:12\n at /home/yellowtent/box/node_modules/docker-modem/lib/modem.js:265:7\n at getCause (/home/yellowtent/box/node_modules/docker-modem/lib/modem.js:287:7)\n at Modem.buildPayload (/home/yellowtent/box/node_modules/docker-modem/lib/modem.js:256:5)\n at IncomingMessage.<anonymous> (/home/yellowtent/box/node_modules/docker-modem/lib/modem.js:232:14)\n at IncomingMessage.emit (events.js:203:15)\n at endReadableNT (_stream_readable.js:1145:12)\n at process._tickCallback (internal/process/next_tick.js:63:19)","name":"BoxError","reason":"Docker Error","details":{},"message":"removeVolume: Error removing volume: (HTTP code 500) server error - remove e33adee2-f48d-48e8-8777-e281413d2576-localstorage: volume has active mounts "}}

Dec 13 10:05:30 box:tasks 6632: {"percent":100,"result":null,"error":{"stack":"BoxError: removeVolume: Error removing volume: (HTTP code 500) server error - remove e33adee2-f48d-48e8-8777-e281413d2576-localstorage: volume has active mounts \n at Object.callback (/home/yellowtent/box/src/docker.js:686:64)\n at /home/yellowtent/box/node_modules/dockerode/lib/volume.js:81:12\n at /home/yellowtent/box/node_modules/docker-modem/lib/modem.js:265:7\n at getCause (/home/yellowtent/box/node_modules/docker-modem/lib/modem.js:287:7)\n at Modem.buildPayload (/home/yellowtent/box/node_modules/docker-modem/lib/modem.js:256:5)\n at IncomingMessage.<anonymous> (/home/yellowtent/box/node_modules/docker-modem/lib/modem.js:232:14)\n at IncomingMessage.emit (events.js:203:15)\n at endReadableNT (_stream_readable.js:1145:12)\n at process._tickCallback (internal/process/next_tick.js:63:19)","name":"BoxError","reason":"Docker Error","details":{},"message":"removeVolume: Error removing volume: (HTTP code 500) server error - remove e33adee2-f48d-48e8-8777-e281413d2576-localstorage: volume has active mounts "}}

Environment info
HOST: DigitalOcean
The server has
RAM: 4gb out of 8 gb ram free
DISK: 20gb out of 80gb disk storage free
CPU: 10:38:57 up 31 min, 1 user, load average: 0.26, 0.64, 1.84
Uptime: Just performed a few reboots, to be sure, up 31 minutes
OS: Ubuntu 1604
Docker: Docker version 19.03.12, build 48a66213fe

[makemrproper]

Update:
I was able to restore the containers to a healthy state.
I did not solve the root cause, and I do not know if this will occur again.
"Repair" "Restart" and so on did not work.

Solution:
Restore container from backup and all was well. The last backup point was performed 4 days ago, when the problem started occurring.

[girish]

@makemrproper I have seen this as well . I am not sure what's causing this. But I will keep this thread open, just in case someone else hits this. I has to restart docker to "fix this" (systemctl restart docker)