Graphite stuck starting
-
On my cloudron instance, I can't see any disk stats because graphite is stuck starting. There seems to be a migration issue or something.
Mar 14 17:34:59 Apply all migrations: account, admin, auth, contenttypes, dashboard, events, sessions, tagging, tags, url_shortener Mar 14 17:34:59 Running migrations: Mar 14 17:34:59 Traceback (most recent call last): Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 82, in _execute Mar 14 17:34:59 return self.cursor.execute(sql) Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/backends/sqlite3/base.py", line 381, in execute Mar 14 17:34:59 return Database.Cursor.execute(self, query) Mar 14 17:34:59 sqlite3.OperationalError: table "account_mygraph" already exists Mar 14 17:34:59 The above exception was the direct cause of the following exception: Mar 14 17:34:59 Traceback (most recent call last): Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/bin/django-admin.py", line 5, in <module> Mar 14 17:34:59 management.execute_from_command_line() Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 381, in execute_from_command_line Mar 14 17:34:59 utility.execute() Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 375, in execute Mar 14 17:34:59 self.fetch_command(subcommand).run_from_argv(self.argv) Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 323, in run_from_argv Mar 14 17:34:59 self.execute(*args, **cmd_options) Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 364, in execute Mar 14 17:34:59 output = self.handle(*args, **options) Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 83, in wrapped Mar 14 17:34:59 res = handle_func(*args, **kwargs) Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/core/management/commands/migrate.py", line 232, in handle Mar 14 17:34:59 post_migrate_state = executor.migrate( Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 117, in migrate Mar 14 17:34:59 state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial) Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards Mar 14 17:34:59 state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial) Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 245, in apply_migration Mar 14 17:34:59 state = migration.apply(state, schema_editor) Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/migrations/migration.py", line 124, in apply Mar 14 17:34:59 operation.database_forwards(self.app_label, schema_editor, old_state, project_state) Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/migrations/operations/models.py", line 92, in database_forwards Mar 14 17:34:59 schema_editor.create_model(model) Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/backends/base/schema.py", line 307, in create_model Mar 14 17:34:59 self.execute(sql, params or None) Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/backends/base/schema.py", line 137, in execute Mar 14 17:34:59 cursor.execute(sql, params) Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 67, in execute Mar 14 17:34:59 return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers Mar 14 17:34:59 return executor(sql, params, many, context) Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 84, in _execute Mar 14 17:34:59 return self.cursor.execute(sql, params) Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/utils.py", line 89, in __exit__ Mar 14 17:34:59 raise dj_exc_value.with_traceback(traceback) from exc_value Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 82, in _execute Mar 14 17:34:59 return self.cursor.execute(sql) Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/backends/sqlite3/base.py", line 381, in execute Mar 14 17:34:59 return Database.Cursor.execute(self, query) Mar 14 17:34:59 django.db.utils.OperationalError: table "account_mygraph" already exists Mar 14 17:34:59 Applying account.0001_initial... Mar 14 17:35:52 Graphite already initialized Mar 14 17:35:52 Operations to perform: Mar 14 17:35:52 Apply all migrations: account, admin, auth, contenttypes, dashboard, events, sessions, tagging, tags, url_shortener Mar 14 17:35:52 Running migrations: Mar 14 17:35:52 Traceback (most recent call last): Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 82, in _execute Mar 14 17:35:52 return self.cursor.execute(sql) Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/backends/sqlite3/base.py", line 381, in execute Mar 14 17:35:52 return Database.Cursor.execute(self, query) Mar 14 17:35:52 sqlite3.OperationalError: table "account_mygraph" already exists Mar 14 17:35:52 The above exception was the direct cause of the following exception: Mar 14 17:35:52 Traceback (most recent call last): Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/bin/django-admin.py", line 5, in <module> Mar 14 17:35:52 management.execute_from_command_line() Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 381, in execute_from_command_line Mar 14 17:35:52 utility.execute() Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 375, in execute Mar 14 17:35:52 self.fetch_command(subcommand).run_from_argv(self.argv) Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 323, in run_from_argv Mar 14 17:35:52 self.execute(*args, **cmd_options) Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 364, in execute Mar 14 17:35:52 output = self.handle(*args, **options) Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 83, in wrapped Mar 14 17:35:52 res = handle_func(*args, **kwargs) Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/core/management/commands/migrate.py", line 232, in handle Mar 14 17:35:52 post_migrate_state = executor.migrate( Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 117, in migrate Mar 14 17:35:52 state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial) Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards Mar 14 17:35:52 state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial) Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 245, in apply_migration Mar 14 17:35:52 state = migration.apply(state, schema_editor) Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/migrations/migration.py", line 124, in apply Mar 14 17:35:52 operation.database_forwards(self.app_label, schema_editor, old_state, project_state) Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/migrations/operations/models.py", line 92, in database_forwards Mar 14 17:35:52 schema_editor.create_model(model) Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/backends/base/schema.py", line 307, in create_model Mar 14 17:35:52 self.execute(sql, params or None) Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/backends/base/schema.py", line 137, in execute Mar 14 17:35:52 cursor.execute(sql, params) Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 67, in execute Mar 14 17:35:52 return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers Mar 14 17:35:52 return executor(sql, params, many, context) Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 84, in _execute Mar 14 17:35:52 return self.cursor.execute(sql, params) Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/utils.py", line 89, in __exit__ Mar 14 17:35:52 raise dj_exc_value.with_traceback(traceback) from exc_value Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 82, in _execute Mar 14 17:35:52 return self.cursor.execute(sql) Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/backends/sqlite3/base.py", line 381, in execute Mar 14 17:35:52 return Database.Cursor.execute(self, query) Mar 14 17:35:52 django.db.utils.OperationalError: table "account_mygraph" already exists Mar 14 17:35:52 Applying account.0001_initial...Is there a way I can correct this in the filesystem or is a patch needed?
-
On my cloudron instance, I can't see any disk stats because graphite is stuck starting. There seems to be a migration issue or something.
Mar 14 17:34:59 Apply all migrations: account, admin, auth, contenttypes, dashboard, events, sessions, tagging, tags, url_shortener Mar 14 17:34:59 Running migrations: Mar 14 17:34:59 Traceback (most recent call last): Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 82, in _execute Mar 14 17:34:59 return self.cursor.execute(sql) Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/backends/sqlite3/base.py", line 381, in execute Mar 14 17:34:59 return Database.Cursor.execute(self, query) Mar 14 17:34:59 sqlite3.OperationalError: table "account_mygraph" already exists Mar 14 17:34:59 The above exception was the direct cause of the following exception: Mar 14 17:34:59 Traceback (most recent call last): Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/bin/django-admin.py", line 5, in <module> Mar 14 17:34:59 management.execute_from_command_line() Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 381, in execute_from_command_line Mar 14 17:34:59 utility.execute() Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 375, in execute Mar 14 17:34:59 self.fetch_command(subcommand).run_from_argv(self.argv) Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 323, in run_from_argv Mar 14 17:34:59 self.execute(*args, **cmd_options) Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 364, in execute Mar 14 17:34:59 output = self.handle(*args, **options) Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 83, in wrapped Mar 14 17:34:59 res = handle_func(*args, **kwargs) Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/core/management/commands/migrate.py", line 232, in handle Mar 14 17:34:59 post_migrate_state = executor.migrate( Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 117, in migrate Mar 14 17:34:59 state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial) Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards Mar 14 17:34:59 state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial) Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 245, in apply_migration Mar 14 17:34:59 state = migration.apply(state, schema_editor) Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/migrations/migration.py", line 124, in apply Mar 14 17:34:59 operation.database_forwards(self.app_label, schema_editor, old_state, project_state) Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/migrations/operations/models.py", line 92, in database_forwards Mar 14 17:34:59 schema_editor.create_model(model) Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/backends/base/schema.py", line 307, in create_model Mar 14 17:34:59 self.execute(sql, params or None) Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/backends/base/schema.py", line 137, in execute Mar 14 17:34:59 cursor.execute(sql, params) Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 67, in execute Mar 14 17:34:59 return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers Mar 14 17:34:59 return executor(sql, params, many, context) Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 84, in _execute Mar 14 17:34:59 return self.cursor.execute(sql, params) Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/utils.py", line 89, in __exit__ Mar 14 17:34:59 raise dj_exc_value.with_traceback(traceback) from exc_value Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 82, in _execute Mar 14 17:34:59 return self.cursor.execute(sql) Mar 14 17:34:59 File "/usr/lib/python3/dist-packages/django/db/backends/sqlite3/base.py", line 381, in execute Mar 14 17:34:59 return Database.Cursor.execute(self, query) Mar 14 17:34:59 django.db.utils.OperationalError: table "account_mygraph" already exists Mar 14 17:34:59 Applying account.0001_initial... Mar 14 17:35:52 Graphite already initialized Mar 14 17:35:52 Operations to perform: Mar 14 17:35:52 Apply all migrations: account, admin, auth, contenttypes, dashboard, events, sessions, tagging, tags, url_shortener Mar 14 17:35:52 Running migrations: Mar 14 17:35:52 Traceback (most recent call last): Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 82, in _execute Mar 14 17:35:52 return self.cursor.execute(sql) Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/backends/sqlite3/base.py", line 381, in execute Mar 14 17:35:52 return Database.Cursor.execute(self, query) Mar 14 17:35:52 sqlite3.OperationalError: table "account_mygraph" already exists Mar 14 17:35:52 The above exception was the direct cause of the following exception: Mar 14 17:35:52 Traceback (most recent call last): Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/bin/django-admin.py", line 5, in <module> Mar 14 17:35:52 management.execute_from_command_line() Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 381, in execute_from_command_line Mar 14 17:35:52 utility.execute() Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 375, in execute Mar 14 17:35:52 self.fetch_command(subcommand).run_from_argv(self.argv) Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 323, in run_from_argv Mar 14 17:35:52 self.execute(*args, **cmd_options) Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 364, in execute Mar 14 17:35:52 output = self.handle(*args, **options) Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 83, in wrapped Mar 14 17:35:52 res = handle_func(*args, **kwargs) Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/core/management/commands/migrate.py", line 232, in handle Mar 14 17:35:52 post_migrate_state = executor.migrate( Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 117, in migrate Mar 14 17:35:52 state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial) Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards Mar 14 17:35:52 state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial) Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 245, in apply_migration Mar 14 17:35:52 state = migration.apply(state, schema_editor) Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/migrations/migration.py", line 124, in apply Mar 14 17:35:52 operation.database_forwards(self.app_label, schema_editor, old_state, project_state) Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/migrations/operations/models.py", line 92, in database_forwards Mar 14 17:35:52 schema_editor.create_model(model) Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/backends/base/schema.py", line 307, in create_model Mar 14 17:35:52 self.execute(sql, params or None) Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/backends/base/schema.py", line 137, in execute Mar 14 17:35:52 cursor.execute(sql, params) Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 67, in execute Mar 14 17:35:52 return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers Mar 14 17:35:52 return executor(sql, params, many, context) Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 84, in _execute Mar 14 17:35:52 return self.cursor.execute(sql, params) Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/utils.py", line 89, in __exit__ Mar 14 17:35:52 raise dj_exc_value.with_traceback(traceback) from exc_value Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 82, in _execute Mar 14 17:35:52 return self.cursor.execute(sql) Mar 14 17:35:52 File "/usr/lib/python3/dist-packages/django/db/backends/sqlite3/base.py", line 381, in execute Mar 14 17:35:52 return Database.Cursor.execute(self, query) Mar 14 17:35:52 django.db.utils.OperationalError: table "account_mygraph" already exists Mar 14 17:35:52 Applying account.0001_initial...Is there a way I can correct this in the filesystem or is a patch needed?
-
From another use reporting the same issue, if you don't care about stat history, you can purge the graphite database when running the following commands via SSH into your server:
docker exec -ti graphite /bin/bash > rm -rf /var/lib/graphite/* && rm /var/lib/graphite/.initializedThen restart the service via the dashboard.
edited the
.initializedpath -
From another use reporting the same issue, if you don't care about stat history, you can purge the graphite database when running the following commands via SSH into your server:
docker exec -ti graphite /bin/bash > rm -rf /var/lib/graphite/* && rm /var/lib/graphite/.initializedThen restart the service via the dashboard.
edited the
.initializedpath@nebulon Adding memory didn't do anything, but taking your direction to erase database ended up working out.
Note that /var/lib/.initialized didn't exist and was in the graphite directory. Also, I had to manually stop and start the container and execute the command right away or docker wouldn't let me do anything because the container was in the process of restarting/starting.
-
@nebulon Adding memory didn't do anything, but taking your direction to erase database ended up working out.
Note that /var/lib/.initialized didn't exist and was in the graphite directory. Also, I had to manually stop and start the container and execute the command right away or docker wouldn't let me do anything because the container was in the process of restarting/starting.
