API hangs sometimes because too much usage?

[nebulon]

@johannesjom at least for me it works so far without issues, but I only one user on that instance. The next time it hangs, can you just open a webterminal into the app and restart only the languagetool process via supervisorctl restart languagetool just to narrow down the root cause.

[johannesjom]

@nebulon
Yes, that did help:

root@d0fe2428-fca3-4904-920e-59f2f78fbce4:/app/code# supervisorctl restart languagetool
languagetool: stopped
languagetool: started
root@d0fe2428-fca3-4904-920e-59f2f78fbce4:/app/code# 

[nebulon]

@johannesjom good, so then at least we know which component got stuck. Was there anything suspicious in the logs for that?

[johannesjom]

@nebulon Except for a "Feb 24 13:47:26 java.lang.OutOfMemoryError: Java heap space", but which seems to have fixed itself, I couldn't see anything. i'll report back as soon as I have a suspicion.

[johannesjom]

@nebulon
Thanks to Uptime-Kuma I can monitor the error quite well:
Can you/me somehow configure the Cloudron health check to restart the containers directly when the API becomes unreachable? Thanks!

2023-03-02T09:07:18.000Z 2023-03-02 09:07:18.156 +0000 INFO  org.languagetool.server.LanguageToolHttpHandler Handled request in 12974ms; sending code 500
2023-03-02T09:07:20.000Z 172.18.0.1 - - [02/Mar/2023:09:07:20 +0000] "GET / HTTP/1.1" 200 830 "-" "Mozilla (CloudronHealth)"
2023-03-02T09:07:27.000Z Exception in thread "LtThreadPoolExecutorMonitor" java.lang.OutOfMemoryError: Java heap space
2023-03-02T09:07:27.000Z 2023-03-02 09:07:27.388 +0000 ERROR org.languagetool.server.LanguageToolHttpHandler An error has occurred: 'java.lang.RuntimeException: Could not activate rules, detected: nl', sending HTTP code 500. Access from 0:0:0:0:0:0:0:1, HTTP user agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/110.0, User agent param: webextension-firefox-ng, v: 6.0.1, Referrer: null, language: auto, h: 5, r: 180, time: 22206m: ALL_BUT_TEXTLEVEL_ONLY, l: PICKY, iID: 55454:1677747715859, Stacktrace follows:java.lang.RuntimeException: java.lang.RuntimeException: Could not activate rules, detected: nl
2023-03-02T09:07:27.000Z at org.languagetool.server.TextChecker.checkText(TextChecker.java:507)
2023-03-02T09:07:27.000Z at org.languagetool.server.ApiV2.handleCheckRequest(ApiV2.java:172)
2023-03-02T09:07:27.000Z at org.languagetool.server.ApiV2.handleRequest(ApiV2.java:83)
2023-03-02T09:07:27.000Z at org.languagetool.server.LanguageToolHttpHandler.handle(LanguageToolHttpHandler.java:185)
2023-03-02T09:07:27.000Z at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
2023-03-02T09:07:27.000Z at jdk.httpserver/sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:82)
2023-03-02T09:07:27.000Z at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:80)
2023-03-02T09:07:27.000Z at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:848)
2023-03-02T09:07:27.000Z at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
2023-03-02T09:07:27.000Z at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:817)
2023-03-02T09:07:27.000Z at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2023-03-02T09:07:27.000Z at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2023-03-02T09:07:27.000Z at java.base/java.lang.Thread.run(Thread.java:829)
2023-03-02T09:07:27.000Z Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Could not activate rules
2023-03-02T09:07:27.000Z at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
2023-03-02T09:07:27.000Z at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
2023-03-02T09:07:27.000Z at org.languagetool.server.TextChecker.checkText(TextChecker.java:489)
2023-03-02T09:07:27.000Z ... 12 more
2023-03-02T09:07:27.000Z Caused by: java.lang.RuntimeException: Could not activate rules
2023-03-02T09:07:27.000Z at org.languagetool.JLanguageTool.<init>(JLanguageTool.java:343)
2023-03-02T09:07:27.000Z at org.languagetool.server.Pipeline.<init>(Pipeline.java:64)
2023-03-02T09:07:27.000Z at org.languagetool.server.PipelinePool.createPipeline(PipelinePool.java:117)
2023-03-02T09:07:27.000Z at org.languagetool.server.PipelinePool.getPipeline(PipelinePool.java:78)
2023-03-02T09:07:27.000Z at org.languagetool.server.TextChecker.getPipelineResults(TextChecker.java:797)
2023-03-02T09:07:27.000Z at org.languagetool.server.TextChecker.getRuleMatches(TextChecker.java:751)
2023-03-02T09:07:27.000Z at org.languagetool.server.TextChecker.lambda$checkText$5(TextChecker.java:472)
2023-03-02T09:07:27.000Z at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2023-03-02T09:07:27.000Z ... 3 more
2023-03-02T09:07:27.000Z Caused by: java.io.IOException: Cannot load or parse input stream of '/org/languagetool/rules/nl/grammar.xml'
2023-03-02T09:07:27.000Z at org.languagetool.rules.patterns.PatternRuleLoader.getRules(PatternRuleLoader.java:78)
2023-03-02T09:07:27.000Z at org.languagetool.Language.getPatternRules(Language.java:637)
2023-03-02T09:07:27.000Z at org.languagetool.JLanguageTool.activateDefaultPatternRules(JLanguageTool.java:643)
2023-03-02T09:07:27.000Z at org.languagetool.JLanguageTool.<init>(JLanguageTool.java:336)
2023-03-02T09:07:27.000Z ... 10 more
2023-03-02T09:07:27.000Z Caused by: java.lang.RuntimeException: Could not create filter class using constructor public org.languagetool.rules.nl.DutchSuppressMisspelledSuggestionsFilter() throws java.io.IOException
2023-03-02T09:07:27.000Z at org.languagetool.rules.patterns.RuleFilterCreator.getFilter(RuleFilterCreator.java:54)
2023-03-02T09:07:27.000Z at org.languagetool.rules.patterns.XMLRuleHandler.setRuleFilter(XMLRuleHandler.java:670)
2023-03-02T09:07:27.000Z at org.languagetool.rules.patterns.PatternRuleHandler.createRules(PatternRuleHandler.java:699)
2023-03-02T09:07:27.000Z at org.languagetool.rules.patterns.PatternRuleHandler.createRules(PatternRuleHandler.java:726)
2023-03-02T09:07:27.000Z at org.languagetool.rules.patterns.PatternRuleHandler.createRules(PatternRuleHandler.java:726)
2023-03-02T09:07:27.000Z at org.languagetool.rules.patterns.PatternRuleHandler.createRules(PatternRuleHandler.java:726)
2023-03-02T09:07:27.000Z at org.languagetool.rules.patterns.PatternRuleHandler.endElement(PatternRuleHandler.java:433)
2023-03-02T09:07:27.000Z at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:610)
2023-03-02T09:07:27.000Z at java.xml/com.sun.org.apache.xerces.internal.impl.dtd.XMLDTDValidator.endNamespaceScope(XMLDTDValidator.java:1985)
2023-03-02T09:07:27.000Z at java.xml/com.sun.org.apache.xerces.internal.impl.dtd.XMLDTDValidator.handleEndElement(XMLDTDValidator.java:1934)
2023-03-02T09:07:27.000Z at java.xml/com.sun.org.apache.xerces.internal.impl.dtd.XMLDTDValidator.endElement(XMLDTDValidator.java:865)
2023-03-02T09:07:27.000Z at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1718)
2023-03-02T09:07:27.000Z at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2883)
2023-03-02T09:07:27.000Z at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:605)
2023-03-02T09:07:27.000Z at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:534)
2023-03-02T09:07:27.000Z at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:888)
2023-03-02T09:07:27.000Z at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:824)
2023-03-02T09:07:27.000Z at java.xml/com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
2023-03-02T09:07:27.000Z at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1216)
2023-03-02T09:07:27.000Z at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:635)
2023-03-02T09:07:27.000Z at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:324)
2023-03-02T09:07:27.000Z at java.xml/javax.xml.parsers.SAXParser.parse(SAXParser.java:197)
2023-03-02T09:07:27.000Z at org.languagetool.rules.patterns.PatternRuleLoader.getRules(PatternRuleLoader.java:75)
2023-03-02T09:07:27.000Z ... 13 more
2023-03-02T09:07:27.000Z Caused by: java.lang.reflect.InvocationTargetException
2023-03-02T09:07:27.000Z at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
2023-03-02T09:07:27.000Z at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
2023-03-02T09:07:27.000Z at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
2023-03-02T09:07:27.000Z at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
2023-03-02T09:07:27.000Z at org.languagetool.rules.patterns.RuleFilterCreator.getFilter(RuleFilterCreator.java:47)
2023-03-02T09:07:27.000Z ... 35 more
2023-03-02T09:07:27.000Z Caused by: java.lang.OutOfMemoryError: Java heap space
2023-03-02T09:07:27.000Z

[girish]

@johannesjom think the issue is that we don't pass the memory argument to JVM. Let me think... we have seen this before.

[girish]

Right, in metabase and traccar, we pass export JAVA_OPTS="${JAVA_OPTS} -Xmx${ram_mb}". I will fix the package.

[girish]

@johannesjom I have pushed an update. Please update and let me know if it helps.

[johannesjom]

@girish Thank you & I will test it!

[johannesjom]

@johannesjom Seems solved.