SOLVED 150% CPU from Minio after recent upgrade

[robi]

Apr 20 19:07:53 API: SYSTEM()
Apr 20 19:07:53 Time: 02:07:53 UTC 04/21/2021
Apr 20 19:07:53 DeploymentID: a5a5e188-c10b-418b-b42a-3ea60efe5964
Apr 20 19:07:53 Error: Operation timed out (cmd.OperationTimedOut)
Apr 20 19:07:53 1: cmd/data-usage.go:56:cmd.storeDataUsageInBackend()
Apr 20 19:07:53
Apr 20 19:08:20 ==> Changing ownership
Apr 20 19:08:20 ==> Starting minio
Apr 20 19:08:20
Apr 20 19:08:20 API: SYSTEM()
Apr 20 19:08:20 Time: 02:08:20 UTC 04/21/2021
Apr 20 19:08:20 DeploymentID: a5a5e188-c10b-418b-b42a-3ea60efe5964
Apr 20 19:08:20 Error: Unable to apply scanner config: found invalid keys (bitrotscan=off ) for 'scanner' sub-system, use 'mc admin config reset myminio scanner' to fix invalid keys (*fmt.wrapError)
Apr 20 19:08:20 7: cmd/config-current.go:575:cmd.lookupConfigs()
Apr 20 19:08:20 6: cmd/config-current.go:750:cmd.loadConfig()
Apr 20 19:08:20 5: cmd/config.go:233:cmd.initConfig()
Apr 20 19:08:20 4: cmd/config.go:193:cmd.(*ConfigSys).Init()
Apr 20 19:08:20 3: cmd/server-main.go:380:cmd.initAllSubsystems()
Apr 20 19:08:20 2: cmd/server-main.go:317:cmd.initServer()
Apr 20 19:08:20 1: cmd/server-main.go:524:cmd.serverMain()
Apr 20 19:08:20 IAM initialization complete

[girish]

Can you report this upstream? Seems like maybe some minio issue. Is this all the time or only during some backup or some operation?

I don't see the high CPU use in our minio instances atleast. Is anyone else see high CPU usage?

[robi]

@girish IDK that it is an upstream issue.

I restarted, stopped, and started it several times, each time it goes into the above state.

This instance has been running idle for quite some time, so it could be a configuration issue with a recent update or something similar.

New install doesn't appear to do that.

[girish]

@robi It seems like some upstream issue. The config.json is somehow missing the scanner key, not sure why. But running mc admin config reset myminio scanner using the CLI tool seems to fixup your instance.

[robi]

@girish wow, thank you for figuring that out.

Strangeness of the most High.