API hangs sometimes because too much usage?
-
Hello all,
when I try to have larger texts corrected, it sometimes happens that the API hangs and stops responding.According to Health-check everything is ok, but the API itself does not respond anymore. A reboot in the cloudron backend solves the problem.
Does anyone else have this problem?
Thanks and greetings
johannesjom -
@johannesjom at least for me it works so far without issues, but I only one user on that instance. The next time it hangs, can you just open a webterminal into the app and restart only the languagetool process via
supervisorctl restart languagetool
just to narrow down the root cause. -
-
@nebulon
Yes, that did help:root@d0fe2428-fca3-4904-920e-59f2f78fbce4:/app/code# supervisorctl restart languagetool languagetool: stopped languagetool: started root@d0fe2428-fca3-4904-920e-59f2f78fbce4:/app/code#
-
@johannesjom good, so then at least we know which component got stuck. Was there anything suspicious in the logs for that?
-
@nebulon Except for a "Feb 24 13:47:26 java.lang.OutOfMemoryError: Java heap space", but which seems to have fixed itself, I couldn't see anything. i'll report back as soon as I have a suspicion.
-
@nebulon
Thanks to Uptime-Kuma I can monitor the error quite well:
Can you/me somehow configure the Cloudron health check to restart the containers directly when the API becomes unreachable? Thanks!2023-03-02T09:07:18.000Z 2023-03-02 09:07:18.156 +0000 INFO org.languagetool.server.LanguageToolHttpHandler Handled request in 12974ms; sending code 500 2023-03-02T09:07:20.000Z 172.18.0.1 - - [02/Mar/2023:09:07:20 +0000] "GET / HTTP/1.1" 200 830 "-" "Mozilla (CloudronHealth)" 2023-03-02T09:07:27.000Z Exception in thread "LtThreadPoolExecutorMonitor" java.lang.OutOfMemoryError: Java heap space 2023-03-02T09:07:27.000Z 2023-03-02 09:07:27.388 +0000 ERROR org.languagetool.server.LanguageToolHttpHandler An error has occurred: 'java.lang.RuntimeException: Could not activate rules, detected: nl', sending HTTP code 500. Access from 0:0:0:0:0:0:0:1, HTTP user agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/110.0, User agent param: webextension-firefox-ng, v: 6.0.1, Referrer: null, language: auto, h: 5, r: 180, time: 22206m: ALL_BUT_TEXTLEVEL_ONLY, l: PICKY, iID: 55454:1677747715859, Stacktrace follows:java.lang.RuntimeException: java.lang.RuntimeException: Could not activate rules, detected: nl 2023-03-02T09:07:27.000Z at org.languagetool.server.TextChecker.checkText(TextChecker.java:507) 2023-03-02T09:07:27.000Z at org.languagetool.server.ApiV2.handleCheckRequest(ApiV2.java:172) 2023-03-02T09:07:27.000Z at org.languagetool.server.ApiV2.handleRequest(ApiV2.java:83) 2023-03-02T09:07:27.000Z at org.languagetool.server.LanguageToolHttpHandler.handle(LanguageToolHttpHandler.java:185) 2023-03-02T09:07:27.000Z at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77) 2023-03-02T09:07:27.000Z at jdk.httpserver/sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:82) 2023-03-02T09:07:27.000Z at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:80) 2023-03-02T09:07:27.000Z at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:848) 2023-03-02T09:07:27.000Z at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77) 2023-03-02T09:07:27.000Z at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:817) 2023-03-02T09:07:27.000Z at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 2023-03-02T09:07:27.000Z at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 2023-03-02T09:07:27.000Z at java.base/java.lang.Thread.run(Thread.java:829) 2023-03-02T09:07:27.000Z Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Could not activate rules 2023-03-02T09:07:27.000Z at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) 2023-03-02T09:07:27.000Z at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) 2023-03-02T09:07:27.000Z at org.languagetool.server.TextChecker.checkText(TextChecker.java:489) 2023-03-02T09:07:27.000Z ... 12 more 2023-03-02T09:07:27.000Z Caused by: java.lang.RuntimeException: Could not activate rules 2023-03-02T09:07:27.000Z at org.languagetool.JLanguageTool.<init>(JLanguageTool.java:343) 2023-03-02T09:07:27.000Z at org.languagetool.server.Pipeline.<init>(Pipeline.java:64) 2023-03-02T09:07:27.000Z at org.languagetool.server.PipelinePool.createPipeline(PipelinePool.java:117) 2023-03-02T09:07:27.000Z at org.languagetool.server.PipelinePool.getPipeline(PipelinePool.java:78) 2023-03-02T09:07:27.000Z at org.languagetool.server.TextChecker.getPipelineResults(TextChecker.java:797) 2023-03-02T09:07:27.000Z at org.languagetool.server.TextChecker.getRuleMatches(TextChecker.java:751) 2023-03-02T09:07:27.000Z at org.languagetool.server.TextChecker.lambda$checkText$5(TextChecker.java:472) 2023-03-02T09:07:27.000Z at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) 2023-03-02T09:07:27.000Z ... 3 more 2023-03-02T09:07:27.000Z Caused by: java.io.IOException: Cannot load or parse input stream of '/org/languagetool/rules/nl/grammar.xml' 2023-03-02T09:07:27.000Z at org.languagetool.rules.patterns.PatternRuleLoader.getRules(PatternRuleLoader.java:78) 2023-03-02T09:07:27.000Z at org.languagetool.Language.getPatternRules(Language.java:637) 2023-03-02T09:07:27.000Z at org.languagetool.JLanguageTool.activateDefaultPatternRules(JLanguageTool.java:643) 2023-03-02T09:07:27.000Z at org.languagetool.JLanguageTool.<init>(JLanguageTool.java:336) 2023-03-02T09:07:27.000Z ... 10 more 2023-03-02T09:07:27.000Z Caused by: java.lang.RuntimeException: Could not create filter class using constructor public org.languagetool.rules.nl.DutchSuppressMisspelledSuggestionsFilter() throws java.io.IOException 2023-03-02T09:07:27.000Z at org.languagetool.rules.patterns.RuleFilterCreator.getFilter(RuleFilterCreator.java:54) 2023-03-02T09:07:27.000Z at org.languagetool.rules.patterns.XMLRuleHandler.setRuleFilter(XMLRuleHandler.java:670) 2023-03-02T09:07:27.000Z at org.languagetool.rules.patterns.PatternRuleHandler.createRules(PatternRuleHandler.java:699) 2023-03-02T09:07:27.000Z at org.languagetool.rules.patterns.PatternRuleHandler.createRules(PatternRuleHandler.java:726) 2023-03-02T09:07:27.000Z at org.languagetool.rules.patterns.PatternRuleHandler.createRules(PatternRuleHandler.java:726) 2023-03-02T09:07:27.000Z at org.languagetool.rules.patterns.PatternRuleHandler.createRules(PatternRuleHandler.java:726) 2023-03-02T09:07:27.000Z at org.languagetool.rules.patterns.PatternRuleHandler.endElement(PatternRuleHandler.java:433) 2023-03-02T09:07:27.000Z at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:610) 2023-03-02T09:07:27.000Z at java.xml/com.sun.org.apache.xerces.internal.impl.dtd.XMLDTDValidator.endNamespaceScope(XMLDTDValidator.java:1985) 2023-03-02T09:07:27.000Z at java.xml/com.sun.org.apache.xerces.internal.impl.dtd.XMLDTDValidator.handleEndElement(XMLDTDValidator.java:1934) 2023-03-02T09:07:27.000Z at java.xml/com.sun.org.apache.xerces.internal.impl.dtd.XMLDTDValidator.endElement(XMLDTDValidator.java:865) 2023-03-02T09:07:27.000Z at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1718) 2023-03-02T09:07:27.000Z at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2883) 2023-03-02T09:07:27.000Z at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:605) 2023-03-02T09:07:27.000Z at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:534) 2023-03-02T09:07:27.000Z at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:888) 2023-03-02T09:07:27.000Z at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:824) 2023-03-02T09:07:27.000Z at java.xml/com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141) 2023-03-02T09:07:27.000Z at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1216) 2023-03-02T09:07:27.000Z at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:635) 2023-03-02T09:07:27.000Z at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:324) 2023-03-02T09:07:27.000Z at java.xml/javax.xml.parsers.SAXParser.parse(SAXParser.java:197) 2023-03-02T09:07:27.000Z at org.languagetool.rules.patterns.PatternRuleLoader.getRules(PatternRuleLoader.java:75) 2023-03-02T09:07:27.000Z ... 13 more 2023-03-02T09:07:27.000Z Caused by: java.lang.reflect.InvocationTargetException 2023-03-02T09:07:27.000Z at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) 2023-03-02T09:07:27.000Z at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) 2023-03-02T09:07:27.000Z at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) 2023-03-02T09:07:27.000Z at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) 2023-03-02T09:07:27.000Z at org.languagetool.rules.patterns.RuleFilterCreator.getFilter(RuleFilterCreator.java:47) 2023-03-02T09:07:27.000Z ... 35 more 2023-03-02T09:07:27.000Z Caused by: java.lang.OutOfMemoryError: Java heap space 2023-03-02T09:07:27.000Z
-
@johannesjom think the issue is that we don't pass the memory argument to JVM. Let me think... we have seen this before.
-
@johannesjom I have pushed an update. Please update and let me know if it helps.
-
@girish Thank you & I will test it!
-
@johannesjom Seems solved.
-