Graphite stuck starting

[eyecreate]

On my cloudron instance, I can't see any disk stats because graphite is stuck starting. There seems to be a migration issue or something.

Mar 14 17:34:59 Apply all migrations: account, admin, auth, contenttypes, dashboard, events, sessions, tagging, tags, url_shortener
Mar 14 17:34:59 Running migrations:
Mar 14 17:34:59 Traceback (most recent call last):
Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 82, in _execute
Mar 14 17:34:59 return self.cursor.execute(sql)
Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/backends/sqlite3/base.py", line 381, in execute
Mar 14 17:34:59 return Database.Cursor.execute(self, query)
Mar 14 17:34:59 sqlite3.OperationalError: table "account_mygraph" already exists
Mar 14 17:34:59 The above exception was the direct cause of the following exception:
Mar 14 17:34:59 Traceback (most recent call last):
Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/bin/django-admin.py", line 5, in <module>
Mar 14 17:34:59 management.execute_from_command_line()
Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
Mar 14 17:34:59 utility.execute()
Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 375, in execute
Mar 14 17:34:59 self.fetch_command(subcommand).run_from_argv(self.argv)
Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 323, in run_from_argv
Mar 14 17:34:59 self.execute(*args, **cmd_options)
Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 364, in execute
Mar 14 17:34:59 output = self.handle(*args, **options)
Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 83, in wrapped
Mar 14 17:34:59 res = handle_func(*args, **kwargs)
Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/core/management/commands/migrate.py", line 232, in handle
Mar 14 17:34:59 post_migrate_state = executor.migrate(
Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 117, in migrate
Mar 14 17:34:59 state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
Mar 14 17:34:59 state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 245, in apply_migration
Mar 14 17:34:59 state = migration.apply(state, schema_editor)
Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/migrations/migration.py", line 124, in apply
Mar 14 17:34:59 operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/migrations/operations/models.py", line 92, in database_forwards
Mar 14 17:34:59 schema_editor.create_model(model)
Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/backends/base/schema.py", line 307, in create_model
Mar 14 17:34:59 self.execute(sql, params or None)
Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/backends/base/schema.py", line 137, in execute
Mar 14 17:34:59 cursor.execute(sql, params)
Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 67, in execute
Mar 14 17:34:59 return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
Mar 14 17:34:59 return executor(sql, params, many, context)
Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 84, in _execute
Mar 14 17:34:59 return self.cursor.execute(sql, params)
Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/utils.py", line 89, in __exit__
Mar 14 17:34:59 raise dj_exc_value.with_traceback(traceback) from exc_value
Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 82, in _execute
Mar 14 17:34:59 return self.cursor.execute(sql)
Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/backends/sqlite3/base.py", line 381, in execute
Mar 14 17:34:59 return Database.Cursor.execute(self, query)
Mar 14 17:34:59 django.db.utils.OperationalError: table "account_mygraph" already exists
Mar 14 17:34:59 Applying account.0001_initial...
Mar 14 17:35:52 Graphite already initialized
Mar 14 17:35:52 Operations to perform:
Mar 14 17:35:52 Apply all migrations: account, admin, auth, contenttypes, dashboard, events, sessions, tagging, tags, url_shortener
Mar 14 17:35:52 Running migrations:
Mar 14 17:35:52 Traceback (most recent call last):
Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 82, in _execute
Mar 14 17:35:52 return self.cursor.execute(sql)
Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/backends/sqlite3/base.py", line 381, in execute
Mar 14 17:35:52 return Database.Cursor.execute(self, query)
Mar 14 17:35:52 sqlite3.OperationalError: table "account_mygraph" already exists
Mar 14 17:35:52 The above exception was the direct cause of the following exception:
Mar 14 17:35:52 Traceback (most recent call last):
Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/bin/django-admin.py", line 5, in <module>
Mar 14 17:35:52 management.execute_from_command_line()
Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
Mar 14 17:35:52 utility.execute()
Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 375, in execute
Mar 14 17:35:52 self.fetch_command(subcommand).run_from_argv(self.argv)
Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 323, in run_from_argv
Mar 14 17:35:52 self.execute(*args, **cmd_options)
Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 364, in execute
Mar 14 17:35:52 output = self.handle(*args, **options)
Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 83, in wrapped
Mar 14 17:35:52 res = handle_func(*args, **kwargs)
Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/core/management/commands/migrate.py", line 232, in handle
Mar 14 17:35:52 post_migrate_state = executor.migrate(
Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 117, in migrate
Mar 14 17:35:52 state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
Mar 14 17:35:52 state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 245, in apply_migration
Mar 14 17:35:52 state = migration.apply(state, schema_editor)
Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/migrations/migration.py", line 124, in apply
Mar 14 17:35:52 operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/migrations/operations/models.py", line 92, in database_forwards
Mar 14 17:35:52 schema_editor.create_model(model)
Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/backends/base/schema.py", line 307, in create_model
Mar 14 17:35:52 self.execute(sql, params or None)
Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/backends/base/schema.py", line 137, in execute
Mar 14 17:35:52 cursor.execute(sql, params)
Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 67, in execute
Mar 14 17:35:52 return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
Mar 14 17:35:52 return executor(sql, params, many, context)
Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 84, in _execute
Mar 14 17:35:52 return self.cursor.execute(sql, params)
Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/utils.py", line 89, in __exit__
Mar 14 17:35:52 raise dj_exc_value.with_traceback(traceback) from exc_value
Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 82, in _execute
Mar 14 17:35:52 return self.cursor.execute(sql)
Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/backends/sqlite3/base.py", line 381, in execute
Mar 14 17:35:52 return Database.Cursor.execute(self, query)
Mar 14 17:35:52 django.db.utils.OperationalError: table "account_mygraph" already exists
Mar 14 17:35:52 Applying account.0001_initial...

Is there a way I can correct this in the filesystem or is a patch needed?

[murgero]

@eyecreate Give the graphite service some more RAM. I had to do this a long time ago, thought it was patch but as a quick work around give it a try.

[nebulon]

From another use reporting the same issue, if you don't care about stat history, you can purge the graphite database when running the following commands via SSH into your server:

docker exec -ti graphite /bin/bash
> rm -rf /var/lib/graphite/* && rm /var/lib/graphite/.initialized

Then restart the service via the dashboard.

edited the .initialized path

[eyecreate]

@nebulon Adding memory didn't do anything, but taking your direction to erase database ended up working out.

Note that /var/lib/.initialized didn't exist and was in the graphite directory. Also, I had to manually stop and start the container and execute the command right away or docker wouldn't let me do anything because the container was in the process of restarting/starting.

[nebulon]

@eyecreate ah good catch about the .initialized file, I've adjusted my snippet.