Container will not restart due to chmod operation not permitted. Only works after a fresh install.

[AlvaroMartinezB]

Checklist

• Have you pulled and found the error with jc21/nginx-proxy-manager:latest docker image?
  • Yes
• Are you sure you're not using someone else's docker image?
  • Yes
• Have you searched for similar issues (both open and closed)?
  • Yes

Describe the bug

The NPM container will not start after restarting the docker engine for any reason. It works fine on a fresh install, but fails to start after any restart. The error happens during "Setting ownership ..."

2.12.1 (lastest)

To Reproduce
Steps to reproduce the behavior:

1. Configure an NPM instance. Log in, setup credentials, configure proxy host with let's encrypt SSL.
2. Restart docker engine (for any reason, like updates or applying a memory size change)
3. Try to start NPM
4. Observe error in terminal logs

Expected behavior

NPM should start with no issues.

Screenshots

Operating System

Ubuntu 24.04.1

Additional context

Here are the terminal logs:

npm-app-1 | ❯ Configuring npm user ...
npm-app-1 | 0
npm-app-1 | usermod: no changes
npm-app-1 | ❯ Configuring npm group ...
npm-app-1 | ❯ Checking paths ...
npm-app-1 | ❯ Setting ownership ...
npm-app-1 | chown: changing ownership of '/etc/letsencrypt/live/npm-1/cert.pem': Operation not permitted
npm-app-1 | chown: changing ownership of '/etc/letsencrypt/live/npm-1/chain.pem': Operation not permitted
npm-app-1 | chown: changing ownership of '/etc/letsencrypt/live/npm-1/fullchain.pem': Operation not permitted
npm-app-1 | chown: changing ownership of '/etc/letsencrypt/live/npm-1/privkey.pem': Operation not permitted
npm-app-1 | chown: changing ownership of '/etc/letsencrypt/live/npm-2/cert.pem': Operation not permitted
npm-app-1 | chown: changing ownership of '/etc/letsencrypt/live/npm-2/chain.pem': Operation not permitted
npm-app-1 | chown: changing ownership of '/etc/letsencrypt/live/npm-2/fullchain.pem': Operation not permitted
npm-app-1 | chown: changing ownership of '/etc/letsencrypt/live/npm-2/privkey.pem': Operation not permitted
npm-app-1 | s6-rc: warning: unable to start service prepare: command exited 1
npm-app-1 | /run/s6/basedir/scripts/rc.init: warning: s6-rc failed to properly bring all the services up! Check your logs (in /run/uncaught-logs/current if you have in-container logging) for more information.

[TimelessFun]

Same issue here except I can't even install it. It just won't run at all

[suprexus]

Same here.
Work around for me was to change permissions of the letsencrypt/live directory on the host itself.

sudo chmod -R 775 letsencrypt_dir/live

Survives container restarts after that.

[sonpeter88]

I have the same issue.
sudo chmopd -R 775 /live
doesn't seem to work for me

[suprexus]

Check your path - /live doesn't seem correct. Check your docker config to see where you're saving the letsencrypt files.

sudo chmod -R 775 letsencrypt_dir/live

[sonpeter88]

The path seems correct. I just ommitted putting in letsencrypt_dir since it's different for everyone. the live folder has the application folder with pem files within it.

the files have rwx permissions

[suprexus]

Gotcha - What user owns the directory? If you're specifying a PUID in your docker compose file, it should be the same user.

[sonpeter88]

I actually tried specifying user/group (1000:1000) in my docker compose but it got angry that it has to run as root so i assume my docker is already running as root.

ls -al in letsencrypt_dir/live/npm-3 folder shows

lrwxrwxrwx1 peter peter
which is uid=1000 and gid=1000

[jaysee260]

In the same boat as @sonpeter88 ... sudo chmod -R 775 /path/to/letsencrypt/live in my host machine doesn't seem to work either 😞 .

[sonpeter88]

okay I figured out a way to get it to work. The files in live folder are symlinks to the ../../archive/ folders and aren't happy when you try to chmod them on my ubuntu machine. I deleted all the symlinks in the live folder, and directly copied the pem files from the archive folder. Once I've done that and gave the files the correct permission, the container was able to restart fine.
Thanks for the tip @suprexus

[jaysee260]

okay I figured out a way to get it to work. The files in live folder are symlinks to the ../../archive/ folders and aren't happy when you try to chmod them on my ubuntu machine. I deleted all the symlinks in the live folder, and directly copied the pem files from the archive folder. Once I've done that and gave the files the correct permission, the container was able to restart fine. Thanks for the tip @suprexus

heh, really nice catch @sonpeter88 ... thanks, I tried it and this works!

[jaysee260]

Hate to report this, but this doesn't really seem to be a stable/lasting solution...

1. When you try to renew the LetsEncrypt cert, you get an error saying symlinks were expected on the *.pem files.
2. I suddenly started getting an issue where I could just not login into the Nginx Manager portal as my admin user.

I'm sure I was using the right credentials, I have them written down. Would straight up just fail authentication. I got so frustrated I just restarted everything from scratch. If it happens again, I'll capture logs and share.

Anyway, this seems like a temporary workaround at best.

[suprexus]

I didn't change the symlinks - What I probably did do was change the permissions on the entire npm directory to include the ../../archive/. Try recreating the symlinks and chmod the entire directory.

[jaysee260]

@suprexus -sigh- that doesn't seem to work, still getting

2024-12-21 10:09:57 ❯ Configuring npm user ...
2024-12-21 10:09:57 ❯ Configuring npm group ...
2024-12-21 10:09:58 ❯ Checking paths ...
2024-12-21 10:09:58 ❯ Setting ownership ...
2024-12-21 10:09:57 useradd warning: npm's uid 0 outside of the UID_MIN 1000 and UID_MAX 60000 range.
2024-12-21 10:09:58 chown: changing ownership of '/etc/letsencrypt/live/npm-1/chain.pem': Operation not permitted
2024-12-21 10:09:58 chown: changing ownership of '/etc/letsencrypt/live/npm-1/cert.pem': Operation not permitted
2024-12-21 10:09:58 chown: changing ownership of '/etc/letsencrypt/live/npm-1/fullchain.pem': Operation not permitted
2024-12-21 10:09:58 chown: changing ownership of '/etc/letsencrypt/live/npm-1/privkey.pem': Operation not permitted
2024-12-21 10:09:58 s6-rc: warning: unable to start service prepare: command exited 1
2024-12-21 10:09:58 /run/s6/basedir/scripts/rc.init: warning: s6-rc failed to properly bring all the services up! Check your logs (in /run/uncaught-logs/current if you have in-container logging) for more information.

Even went as far as making sure I set the proper permissions to the entire letsencrypt directory with sudo chmod -R 775 /path/to/letsencrypt.

The only thing that's "worked" so far is @sonpeter88's suggestion to remove the symlinks and copy the *.pem files from ../../archive, but as pointed out earlier, that solution is unstable, and causes other issues.

Will keep digging...