can't enable 2FA, now can't log in! --- Redis issues

[humptydumpty]

I decided to mess around some more with the settings hoping to break something again and I did! I changed the memory limit from 2GB down to 1.5GB and while the app was applying the changes, it spat an error:

Addons Error: Error waiting for redis-b261fedc-d6d5-4c52-994e-d4a2a04afa6a. Status code: 200 message: process not running

Redis was set at 150MB. Resized it to 1GB. Then, I resized the RAM again and it didn't run into any errors. I stopped and started the app with no issues. Looks like it's running flawlessly with a higher Redis size.

@nebulon maybe the default Redis size of 150MB should be higher for new installations?

[humptydumpty]

Yesterday, I lowered Redis to 512MB and left it for the night. I woke up today with multiple notifications that Redis ran out of memory again. I set it to 1GB again, let's see if it keeps crashing.

Content of error notification in CR dash:

{
  "event": {
    "status": "oom",
    "id": "7633e58f336ce8e6a8219c5024ebec767b33705b47a86a888e2e60b4a83e4cfd",
    "from": "cloudron/redis:3.5.0@sha256:ee6da2599a72afaec1d80c41db9b5fe79c882fb920195659e871501ea2e94d18",
    "Type": "container",
    "Action": "oom",
    "Actor": {
      "ID": "7633e58f336ce8e6a8219c5024ebec767b33705b47a86a888e2e60b4a83e4cfd",
      "Attributes": {
        "image": "cloudron/redis:3.5.0@sha256:ee6da2599a72afaec1d80c41db9b5fe79c882fb920195659e871501ea2e94d18",
        "isCloudronManaged": "true",
        "location": "work.mydomain.com",
        "name": "redis-b261fedc-d6d5-4c52-994e-d4a2a04afa6a"
      }
    },
    "scope": "local",
    "time": 1690544668,
    "timeNano": 1690544668119625700
  },
  "containerId": "7633e58f336ce8e6a8219c5024ebec767b33705b47a86a888e2e60b4a83e4cfd",
  "addonName": "redis",
  "app": {
    "id": "b261fedc-d6d5-4c52-994e-d4a2a04afa6a",
    "appStoreId": "io.leantime.cloudronapp",
    "installationState": "installed",
    "runState": "running",
    "health": "healthy",
    "containerId": "6df6ee7d6e0b79ac6b0e0d93c95be82525ecbb2327e35e1c188b0659c50324a0",
    "memoryLimit": 2147483648,
    "cpuShares": 512,
    "label": "Work",
    "taskId": null,
    "sso": false,
    "enableBackup": true,
    "proxyAuth": false,
    "containerIp": "172.18.17.2",
    "crontab": null,
    "creationTime": "2023-07-27T19:25:56.000Z",
    "updateTime": "2023-07-27T19:25:56.000Z",
    "enableAutomaticUpdate": true,
    "upstreamUri": "",
    "enableMailbox": true,
    "mailboxDisplayName": "",
    "mailboxName": "work.app",
    "mailboxDomain": "mydomain.com",
    "enableInbox": false,
    "inboxName": null,
    "inboxDomain": null,
    "storageVolumeId": null,
    "storageVolumePrefix": null,
    "ts": "2023-07-28T00:15:44.000Z",
    "healthTime": "2023-07-28T11:44:20.000Z",
    "hasIcon": false,
    "hasAppStoreIcon": true,
    "manifest": {
      "id": "io.leantime.cloudronapp",
      "title": "Leantime",
      "author": "Marcel Folaron marcel@leantime.io",
      "description": "### Overview\n\nLeantime is a strategic open source project management system for innovative companies and teams looking to go from start to finish.\nBuilt for the non-project manager, we combine the plans and the work while making it easy for everyone on the team to use.\nIt's an alternative to ClickUp, Notion, and Asana. As simple as Trello but as feature rich as Jira.\n",
      "changelog": "* Add cron job runner every 5min\n* Fix uploading of company logo\n",
      "tagline": "Project management system",
      "postInstallMessage": "This app is pre-setup with an admin account. The initial credentials are:\n\n**Username**: admin@cloudron.local<br/>\n**Password**: changeme<br/>\n\nPlease change the password immediately.\n",
      "version": "0.2.0",
      "upstreamVersion": "2.3.27",
      "minBoxVersion": "7.3.0",
      "healthCheckPath": "/",
      "icon": "file://logo.png",
      "memoryLimit": 536870912,
      "httpPort": 8080,
      "addons": {
        "localstorage": {},
        "mysql": {},
        "sendmail": {},
        "scheduler": {
          "cron": {
            "schedule": "*/15 * * * *",
            "command": "/app/pkg/cron.sh"
          }
        },
        "redis": {
          "noPassword": true
        }
      },
      "manifestVersion": 2,
      "mediaLinks": [
        "https://screenshots.cloudron.io/io.leantime.cloudronapp/leantime_01.png"
      ],
      "website": "https://leantime.io",
      "forumUrl": "https://forum.cloudron.io/category/171/leantime",
      "documentationUrl": "https://docs.cloudron.io/apps/leantime/",
      "contactEmail": "support@cloudron.io",
      "tags": [
        "project",
        "management",
        "asana",
        "notion"
      ],
      "dockerImage": "cloudron/io.leantime.cloudronapp:20230725-152314-391ce997f"
    },
    "tags": [],
    "reverseProxyConfig": {},
    "portBindings": {},
    "accessRestriction": {
      "users": [
        "uid-98bc0e1b-b180-40b3-a2f2-841972a36235",
        "uid-8b545e3c-f964-4a5b-9ea2-626c172bef50",
        "uid-6e2d5af4-3e0d-4c5e-885c-387284fd28e5",
        "uid-7484a80c-9bd0-4bf1-a06e-c930105a0c0c"
      ],
      "groups": []
    },
    "operators": null,
    "debugMode": null,
    "servicesConfig": {
      "redis": {
        "memoryLimit": 536870912,
        "recoveryMode": false
      }
    },
    "secondaryDomains": [],
    "redirectDomains": [],
    "aliasDomains": [],
    "subdomain": "work",
    "domain": "mydomain.com",
    "certificate": null,
    "env": {},
    "mounts": [],
    "error": null,
    "iconUrl": "/api/v1/apps/b261fedc-d6d5-4c52-994e-d4a2a04afa6a/icon",
    "fqdn": "work.mydomain.com"
  }
}

[humptydumpty]

I created a new user. I clicked on the invite email that Leantime sent and set a password for the new account. Then, I activated 2FA. I logged out and tried to log back in and ran into the same login error. However, resetting the password seems to fix the issue permanently. Something is corrupting the passwords on newly create accounts including the admin account that the package creates on first install.

To make this short, there are two issues that need looking into:

• Redis OOM - I have it set to 1GB, and it's 80% full on idle.
• Password Corruption - for all accounts and can be fixed by resetting the password and issue is gone permanently.

[humptydumpty]

Redis didn't crash (yet) but it's getting there. Now it's 95% full and I haven't been using the app!

[RoundHouse1924]

@humptydumpty What happens when you restart Redis?

[humptydumpty]

If I restart Redis, it goes down a bit, but rises after a while.

[robi]

It looks like the Leantime app using redis is not using it well, not freeing memory. Needs to be more lean with Redis

[humptydumpty]

@robi can this be fixed in the packaging or should I report this upstream?

[robi]

@humptydumpty upstream.

[humptydumpty]

@robi Thanks. I reported this upstream here: https://github.com/Leantime/leantime/issues/1780

[humptydumpty]

It looks like this will take a while to get fixed upstream. Is it possible to disable Redis for now?

[girish]

@humptydumpty we have a build now without redis, will publish shortly.

[girish]

Published new package.

[humptydumpty]

@girish Thanks for the update!

[LoudLemur]

@humptydumpty Did you manage to get Two Factor Authentication (2FA) working?

After initial creation of an account, I tried to set up 2FA, but it wouldn't let me.

[humptydumpty]

Yes, it’s working fine. You’re probably running into the issue where the password gets corrupted on first login. Log out, reset the password using the forgot pass button and try 2fa again. Iirc, this is fixed in the next upstream update.