Solved 150% CPU from Minio after recent upgrade
Apr 20 19:07:53 API: SYSTEM() Apr 20 19:07:53 Time: 02:07:53 UTC 04/21/2021 Apr 20 19:07:53 DeploymentID: a5a5e188-c10b-418b-b42a-3ea60efe5964 Apr 20 19:07:53 Error: Operation timed out (cmd.OperationTimedOut) Apr 20 19:07:53 1: cmd/data-usage.go:56:cmd.storeDataUsageInBackend() Apr 20 19:07:53 Apr 20 19:08:20 ==> Changing ownership Apr 20 19:08:20 ==> Starting minio Apr 20 19:08:20 Apr 20 19:08:20 API: SYSTEM() Apr 20 19:08:20 Time: 02:08:20 UTC 04/21/2021 Apr 20 19:08:20 DeploymentID: a5a5e188-c10b-418b-b42a-3ea60efe5964 Apr 20 19:08:20 Error: Unable to apply scanner config: found invalid keys (bitrotscan=off ) for 'scanner' sub-system, use 'mc admin config reset myminio scanner' to fix invalid keys (*fmt.wrapError) Apr 20 19:08:20 7: cmd/config-current.go:575:cmd.lookupConfigs() Apr 20 19:08:20 6: cmd/config-current.go:750:cmd.loadConfig() Apr 20 19:08:20 5: cmd/config.go:233:cmd.initConfig() Apr 20 19:08:20 4: cmd/config.go:193:cmd.(*ConfigSys).Init() Apr 20 19:08:20 3: cmd/server-main.go:380:cmd.initAllSubsystems() Apr 20 19:08:20 2: cmd/server-main.go:317:cmd.initServer() Apr 20 19:08:20 1: cmd/server-main.go:524:cmd.serverMain() Apr 20 19:08:20 IAM initialization complete
Can you report this upstream? Seems like maybe some minio issue. Is this all the time or only during some backup or some operation?
I don't see the high CPU use in our minio instances atleast. Is anyone else see high CPU usage?
@girish IDK that it is an upstream issue.
I restarted, stopped, and started it several times, each time it goes into the above state.
This instance has been running idle for quite some time, so it could be a configuration issue with a recent update or something similar.
New install doesn't appear to do that.
@robi It seems like some upstream issue. The config.json is somehow missing the
scannerkey, not sure why. But running
mc admin config reset myminio scannerusing the CLI tool seems to fixup your instance.
@girish wow, thank you for figuring that out.
Strangeness of the most High.