Logging into Cloudron with OpenID Fails After Update to 7.7.0

[pathab]

Hello, the login via Cloudron OpenID no longer works for all apps since I updated to V7.7.0. I have already tried several things, reinstalled apps and now also reinstalled the entire server and restored it from the backup. What else can I do?

The following apps are affected:

• Matomo (Unexpected response from the OAuth service.)
• Typebot
• Gitea (/user/oauth2/cloudron for xxx.xxx.xxx.xxx:0, 500 Internal Server Error)
• Rallly

All apps are up to date.

[pathab]

It looks like https://my.domain.xyz/.well-known/openid-configuration is not accessible, but how can I fix it?

[girish]

@pathab The dashboard itself works but not the .well-known ? Are you behind Cloudflare?

[pathab]

@girish No, but I was just able to find the problem. I tried to activate IPv6 after the update, but forgot to add the AAAA DNS records for the domain. I have now made the entries and it now works.

[pathab]

OpenID authentication now only does not work with Typebot and Rally.
I have now also tested it with a fresh installation of Typebot:

box:taskworker Task took 66.659 seconds
Mar 14 00:15:47 ' at /app/code/builder/node_modules/.pnpm/openid-client@5.6.4/node_modules/openid-client/lib/helpers/request.js:140:13\n' +
Mar 14 00:15:47 ' at async AuthHandler (/app/code/builder/node_modules/.pnpm/next-auth@4.22.1_next@14.1.0_nodemailer@6.9.3_react-dom@18.2.0_react@18.2.0/node_modules/next-auth/core/index.js:260:26)\n' +
Mar 14 00:15:47 ' at async D (/app/code/builder/apps/builder/.next/server/chunks/524.js:1:7871)\n' +
Mar 14 00:15:47 ' at async Issuer.discover (/app/code/builder/node_modules/.pnpm/openid-client@5.6.4/node_modules/openid-client/lib/issuer.js:143:22)\n' +
Mar 14 00:15:47 ' at async K (/app/code/builder/node_modules/.pnpm/next@14.1.0_@babel+core@7.22.9_react-dom@18.2.0_react@18.2.0/node_modules/next/dist/compiled/next-server/pages-api.runtime.prod.js:20:16545)\n' +
Mar 14 00:15:47 ' at async NextAuthApiHandler (/app/code/builder/node_modules/.pnpm/next-auth@4.22.1_next@14.1.0_nodemailer@6.9.3_react-dom@18.2.0_react@18.2.0/node_modules/next-auth/next/index.js:22:19)\n' +
Mar 14 00:15:47 ' at async Object.signin (/app/code/builder/node_modules/.pnpm/next-auth@4.22.1_next@14.1.0_nodemailer@6.9.3_react-dom@18.2.0_react@18.2.0/node_modules/next-auth/core/routes/signin.js:38:24)\n' +
Mar 14 00:15:47 ' at async U.render (/app/code/builder/node_modules/.pnpm/next@14.1.0_@babel+core@7.22.9_react-dom@18.2.0_react@18.2.0/node_modules/next/dist/compiled/next-server/pages-api.runtime.prod.js:20:16981)',
Mar 14 00:15:47 ' at async getAuthorizationUrl (/app/code/builder/node_modules/.pnpm/next-auth@4.22.1_next@14.1.0_nodemailer@6.9.3_react-dom@18.2.0_react@18.2.0/node_modules/next-auth/core/lib/oauth/authorization-url.js:70:18)\n' +
Mar 14 00:15:47 ' at async openidClient (/app/code/builder/node_modules/.pnpm/next-auth@4.22.1_next@14.1.0_nodemailer@6.9.3_react-dom@18.2.0_react@18.2.0/node_modules/next-auth/core/lib/oauth/client.js:16:14)\n' +
Mar 14 00:15:47 [next-auth][error][SIGNIN_OAUTH_ERROR]
Mar 14 00:15:47 error: {
Mar 14 00:15:47 https://next-auth.js.org/errors#signin_oauth_error outgoing request timed out after 3500ms {
Mar 14 00:15:47 message: 'outgoing request timed out after 3500ms'
Mar 14 00:15:47 message: 'outgoing request timed out after 3500ms',
Mar 14 00:15:47 name: 'RPError'
Mar 14 00:15:47 providerId: 'custom-oauth',
Mar 14 00:15:47 stack: 'RPError: outgoing request timed out after 3500ms\n' +
Mar 14 00:15:47 }
Mar 14 00:15:47 },

[nebulon]

So to confirm, dashboard and other apps work, only the typebot and rallly instances wont?

[pathab]

@nebulon Correct!

[nebulon]

And if you install a new surfer instance for example, that also works fine? I am asking since we had often router hairpin issues in the past, where apps were not able to reach the OpenID provider by its public origin. So fresh surfer instance would test this. If that works, then this is really local to those apps.

[pathab]

The installation actually fails

[nebulon]

Seems like the server has some issue with ipv6 connectivity then. Try to disable it on the server side and (if setup) remove the AAAA DNS records. Just to rule out that potential issue angle for now.

Depending on your server provider you may or may not able to disable it there, otherwise sysctl -w net.ipv6.conf.all.disable_ipv6=1 might work.

[pathab]

Ok, I have now deleted the DNS setting, deactivated the IPv6 setting and restarted the server. Now OpenID no longer works for all apps and https://my.domain.com/.well-known/openid-configuration is no longer accessible.
But now I was able to perform a completely fresh installation of Surfer. However, the login via OpenID does not work there either.

[nebulon]

If you open a webterminal into any app, can you curl -v https://my.yourdomain.com/.well-known/openid-configuration if not, does the DNS resolve correctly? If yes, I guess it is also a hairpinning issue then.

[pathab]

Yes, the IP is resolved correctly. But without response. Could it be that this nginx route is no longer working properly?

[girish]

@pathab does curl -4 -v https://my.yourdomain.com/.well-known/openid-configuration work? This forces IPv4. Maybe some caching issue somewhere is causing a IPv6 query.

[pathab]

Unfortunately, it's the same story.

[girish]

@pathab what is the output? also, does the curl not work only on the server or from anywhere?

[pathab]

the output is the same everywhere

*   Trying xxx.xxx.xxx.xxx:443...
* connect to xxx.xxx.xxx.xxx port 443 failed: Connection timed out
* Failed to connect to my.domain.com port 443 after 131026 ms: Connection timed out
* Closing connection 0
curl: (28) Failed to connect to my.domain.com port 443 after 131026 ms: Connection timed out

[girish]

@pathab are you hosting at home? If so, https://docs.cloudron.io/troubleshooting/#hairpin-nat is the most likely issue. Your network has no mechanism to reach itself via public IP.

But, before we go into this, another test. Can you try to curl that URL from another network altogether ? Say via your mobile phone network or equivalent? Does that work ?

[pathab]

It seems like you're right and the new modem is the problem, sorry I forgot to mention that - didn't expect that to be the cause. (Because cloudron has been running very smoothly for over a year now).
Well, I have now moved the server to a VPS. Everything seems to be working there now. I will try to get my cloudron home server up and running again at a later date. Thank you very much for your time and support!

[pathab]

btw. I just love how easy it is to transfer cloudron to a new server!