All certificate renewals suddenly fail. Has worked for months if not years.

[Waldorf3]

After using this container for months, if not years, with minimal interaction necessary, suddenly certificates are no longer automatically updated. Nothing changed in my environment so I'm a bit at a loss what happened.

From the log:

nginx-proxy-manager-app-1  | [9/15/2022] [4:50:19 PM] [Global   ] › ℹ  info      Generating MySQL knex configuration from environment variables
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:19 PM] [Global   ] › ⬤  debug     Wrote db configuration to config file: ./config/production.json
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:20 PM] [Migrate  ] › ℹ  info      Current database version: 20211108145214
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:20 PM] [Setup    ] › ℹ  info      Creating a new JWT key pair...
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:24 PM] [Setup    ] › ℹ  info      Wrote JWT key pair to config file: /app/config/production.json
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:24 PM] [Setup    ] › ℹ  info      Logrotate Timer initialized
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:24 PM] [Setup    ] › ℹ  info      Logrotate completed.
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:24 PM] [IP Ranges] › ℹ  info      Fetching IP Ranges from online services...
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:24 PM] [IP Ranges] › ℹ  info      Fetching https://ip-ranges.amazonaws.com/ip-ranges.json
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:24 PM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v4
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:24 PM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v6
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:24 PM] [SSL      ] › ℹ  info      Let's Encrypt Renewal Timer initialized
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:24 PM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:24 PM] [IP Ranges] › ℹ  info      IP Ranges Renewal Timer initialized
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:24 PM] [Global   ] › ℹ  info      Backend PID 246 listening on port 3000 ...
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:49 PM] [Express  ] › ⚠  warning   invalid signature
nginx-proxy-manager-app-1  | [9/15/2022] [4:52:43 PM] [SSL      ] › ✖  error     Error: Command failed: certbot renew --non-interactive --quiet --config "/etc/letsencrypt.ini" --preferred-challenges "dns,http" --disable-hook-validation
nginx-proxy-manager-app-1  | Failed to renew certificate npm-1 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-10 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-11 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-12 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-13 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-2 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-25 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-3 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-31 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-32 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-34 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-35 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-36 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-38 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-39 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-4 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-40 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-41 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-42 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-5 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-6 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-7 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-8 with error: urn:ietf:params:acme:error:rateLimited :: There were too many requests of a given type :: Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/failed-validation-limit/
nginx-proxy-manager-app-1  | Failed to renew certificate npm-9 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | All renewals failed. The following certificates could not be renewed:
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-1/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-10/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-11/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-12/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-13/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-2/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-25/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-3/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-31/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-32/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-34/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-35/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-36/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-38/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-39/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-4/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-40/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-41/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-42/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-5/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-6/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-7/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-8/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-9/fullchain.pem (failure)
nginx-proxy-manager-app-1  | 24 renew failure(s), 0 parse failure(s)
nginx-proxy-manager-app-1  |
nginx-proxy-manager-app-1  |     at ChildProcess.exithandler (node:child_process:399:12)
nginx-proxy-manager-app-1  |     at ChildProcess.emit (node:events:526:28)
nginx-proxy-manager-app-1  |     at maybeClose (node:internal/child_process:1092:16)
nginx-proxy-manager-app-1  |     at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5)
nginx-proxy-manager-app-1  | `QueryBuilder#allowEager` method is deprecated. You should use `allowGraph` instead. `allowEager` method will be removed in 3.0
nginx-proxy-manager-app-1  | `QueryBuilder#eager` method is deprecated. You should use the `withGraphFetched` method instead. `eager` method will be removed in 3.0
nginx-proxy-manager-app-1  | QueryBuilder#omit is deprecated. This method will be removed in version 3.0
nginx-proxy-manager-app-1  | Model#$omit is deprected and will be removed in 3.0.
nginx-proxy-manager-app-1  | Duplicate relation "access_list" in a relation expression. You should use "a.[b, c]" instead of "[a.b, a.c]". This will cause an error in objection 2.0
nginx-proxy-manager-app-1  | [9/15/2022] [4:55:08 PM] [Nginx    ] › ℹ  info      Reloading Nginx
nginx-proxy-manager-app-1  | [9/15/2022] [4:55:30 PM] [Nginx    ] › ℹ  info      Reloading Nginx
nginx-proxy-manager-app-1  | [9/15/2022] [4:55:53 PM] [Nginx    ] › ℹ  info      Reloading Nginx
nginx-proxy-manager-app-1  | [9/15/2022] [4:55:58 PM] [SSL      ] › ℹ  info      Requesting Let'sEncrypt certificates for Cert #46: myapp.mydomain.com
nginx-proxy-manager-app-1  | [9/15/2022] [4:55:58 PM] [SSL      ] › ℹ  info      Command: certbot certonly --config "/etc/letsencrypt.ini" --cert-name "npm-46" --agree-tos --authenticator webroot --email "me@mydomain.com" --preferred-challenges "dns,http" --domains "myapp.mydomain.com"
nginx-proxy-manager-app-1  | [9/15/2022] [4:56:05 PM] [Nginx    ] › ℹ  info      Reloading Nginx
nginx-proxy-manager-app-1  | [9/15/2022] [4:56:05 PM] [Express  ] › ⚠  warning   Command failed: certbot certonly --config "/etc/letsencrypt.ini" --cert-name "npm-46" --agree-tos --authenticator webroot --email "me@mydomain.com" --preferred-challenges "dns,http" --domains "myapp.mydomain.com"
nginx-proxy-manager-app-1  | Saving debug log to /var/log/letsencrypt/letsencrypt.log
nginx-proxy-manager-app-1  | Some challenges have failed.
nginx-proxy-manager-app-1  | Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

[the1ts]

@Waldorf3 There is a problem with renewing if force SSL is turned on, if that is true for you, turn off force SSL, renew then turn back on.
Its awaiting a fix to be merged #2038

[huntitus]

We have the same symptoms. The certificates not renewed by automatically. When I try to renew manually on the GUI, it provide an "internal error" message (without any error code, or details). Sadly this solution is not reliable :/

Edit:

• Turning off the "force SSL" option not worked for me.
• This solution, (from 2022 March 24) worked for me: Renew now on SSL Certificates page gives internal error #1816 (comment)

Looks like this problem is very old :/ and still exist.

[Waldorf3]

We have the same symptoms. The certificates not renewed by automatically. When I try to renew manually on the GUI, it provide an "internal error" message (without any error code, or details). Sadly this solution is not reliable :/

Edit:

• Turning off the "force SSL" option not worked for me.
• This solution, (from 2022 March 24) worked for me: Renew now on SSL Certificates page gives internal error #1816 (comment)

Looks like this problem is very old :/ and still exist.

I already tried the "uncheck force ssl and reissue cert", it just throws an "internal error".

I'm running NPM in docker, not sure how to start fiddling with a script to fix this. Would be better if the author would acknowledge the bug and offer a proper solution.

[Waldorf3]

I'm just realizing I might be barking up the wrong tree. Is jc21 the actual developer/maintainer of this code, or just the docker-packager? If not, who is responsible for the code, who can fix this bug?

I actually find the NPM SSL subsystem to be quite fragile. If for example you try to enable SSL for a site that does not have a root directory, such as for example ubooquity (requires http://ubooquity/ubooquity) it will also fail with a nondescript "internal error", and only way to fix is manually cleaning up the database. That's a viable solution for a single failed certificate, not for a system with 30 or more proxy hosts with failed certs.

[the1ts]

JC21 is the developer and the only fragility at the moment that I see is the renew issue outstanding a merge.
I simply don't see how NPM fronting for ubooquity results in an internal error that requires DB cleanup. The application isn't even hit for letsencrypt SSL certs to be created via the HTTP auth method.
Also picking a pretty broken by modern standards application like ubooquity to measure success is odd. It has, as you say, no root directory so requires special measures and also a different port for admin so requires special measures again, no tool calling itself easy to use can be expected to handle both those broken by modern standards decisions.
Often internal errors are after people remove active SSL certs breaking Nginx config i.e. ignoring the warning. What specific errors are you still getting?

[EDIflyer]

Just to confirm that PR #2038 by @the1ts seems to be doing the trick for me.

[github-actions]

Issue is now considered stale. If you want to keep it open, please comment 👍

[Saibamen]

👍

[jdhorner]

👍

[e1ke]

👍
have the same issue with jc21/nginx-proxy-manager:2.12.2
turning off 'force SSL', renewing the certificates by hand and turning 'force SSL' on again worked for me but this will be pretty annoying if that is the only workaround for this issue and not a permanent solution.
Strange that it worked for years and now suddenly stopped working. I migrated to another server by copying the data volume, maybe there is an issue with file permission or sth? I can't figure it out

edit
mh, after ignoring the HSTS settings, it worked even with 'force SSL' activated:

I think that works as a permament solution for me

[EDIflyer]

@e1ke if you want a more permanent solution that allows HSTS to be turned on then feel free to use nginxproxymanager/nginx-proxy-manager-dev:pr-3121 from my PR at #3121 - I live in hope that it'll be merged one of these days!