java.lang.RuntimeException: Expected at least '1grams', '2grams', and '3grams' sub directories but only got [3grams] in /app/data/ngrams/en

[luckow]

The moment you try to perform a language check after activating the en n-gram data set:

Jan 27 11:49:49 2023-01-27 10:49:49 +0000 Starting LanguageTool 6.0 (build date: 2022-12-29 12:13:11 +0000, e44dbb0) server on http://localhost:8010...
Jan 27 11:49:49 2023-01-27 10:49:49 +0000 Server started
Jan 27 11:49:50 172.18.0.1 - - [27/Jan/2023:10:49:50 +0000] "GET / HTTP/1.1" 200 810 "-" "Mozilla (CloudronHealth)"
Jan 27 11:50:00 172.18.0.1 - - [27/Jan/2023:10:50:00 +0000] "GET / HTTP/1.1" 200 810 "-" "Mozilla (CloudronHealth)"
Jan 27 11:50:10 172.18.0.1 - - [27/Jan/2023:10:50:10 +0000] "GET / HTTP/1.1" 200 810 "-" "Mozilla (CloudronHealth)"
Jan 27 11:50:12 2023-01-27 10:50:12.527 +0000 INFO org.languagetool.server.LanguageToolHttpHandler Handling POST /v2/check
Jan 27 11:50:12 2023-01-27 10:50:12.527 +0000 INFO org.languagetool.server.LanguageToolHttpHandler Handling POST /v2/check
Jan 27 11:50:15 2023-01-27 10:50:15.099 +0000 ERROR org.languagetool.server.LanguageToolHttpHandler An error has occurred: 'java.lang.RuntimeException: Expected at least '1grams', '2grams', and '3grams' sub directories but only got [3grams] in /app/data/ngrams/en, detected: en-US', sending HTTP code 500. Access from 0:0:0:0:0:0:0:1, HTTP user agent: Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0, User agent param: webextension-firefox-ng, v: 6.0.1, Referrer: null, language: en-gb, h: 2, r: 2, time: 2589m: TEXTLEVEL_ONLY, l: PICKY, iID: 94148:1674816428889, Stacktrace follows:java.lang.RuntimeException: java.lang.RuntimeException: Expected at least '1grams', '2grams', and '3grams' sub directories but only got [3grams] in /app/data/ngrams/en, detected: en-US
Jan 27 11:50:15 at org.languagetool.server.TextChecker.checkText(TextChecker.java:507)
Jan 27 11:50:15 at org.languagetool.server.ApiV2.handleCheckRequest(ApiV2.java:172)
Jan 27 11:50:15 at org.languagetool.server.ApiV2.handleRequest(ApiV2.java:83)
Jan 27 11:50:15 at org.languagetool.server.LanguageToolHttpHandler.handle(LanguageToolHttpHandler.java:185)
Jan 27 11:50:15 at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
Jan 27 11:50:15 at jdk.httpserver/sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:82)
Jan 27 11:50:15 at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:80)
Jan 27 11:50:15 at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:848)
Jan 27 11:50:15 at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
Jan 27 11:50:15 at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:817)
Jan 27 11:50:15 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Jan 27 11:50:15 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Jan 27 11:50:15 at java.base/java.lang.Thread.run(Thread.java:829)
Jan 27 11:50:15 Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Expected at least '1grams', '2grams', and '3grams' sub directories but only got [3grams] in /app/data/ngrams/en
Jan 27 11:50:15 at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
Jan 27 11:50:15 at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
Jan 27 11:50:15 at org.languagetool.server.TextChecker.checkText(TextChecker.java:489)
Jan 27 11:50:15 ... 12 more
Jan 27 11:50:15 Caused by: java.lang.RuntimeException: Expected at least '1grams', '2grams', and '3grams' sub directories but only got [3grams] in /app/data/ngrams/en
Jan 27 11:50:15 at org.languagetool.languagemodel.LuceneSingleIndexLanguageModel.validateDirectory(LuceneSingleIndexLanguageModel.java:72)
Jan 27 11:50:15 at org.languagetool.languagemodel.LuceneSingleIndexLanguageModel.doValidateDirectory(LuceneSingleIndexLanguageModel.java:109)

[luckow]

My mistake. I restarted the app too early.
After the full download of almost 8 GB EN n-gram dataset, you need to give your Cloudron instance enough time to unpack and move the files to (by default) /app/data/ngram/en.

In my case, it took more than 15 minutes to meet the requirements.
The moment you restarted the app too early, the zip file was not unzipped completely. In this case the error occurs and you have to start again from the beginning.

Search your logs for ===> en ngram dataset has been installed.

[nebulon]

did the n-gram download finish and is this also happening after an app restart?

[luckow]

@nebulon yesterday I followed the documentation from here https://docs.cloudron.io/apps/languagetool/.
By the way: Is there a hint if it is important to comment out NGRAM_DATASET_PATH? In my case I only commented out NGRAM_DATASET=("en").

And yes: it is a permanent error. Every time I restarted the box, the same error occurred. Fun fact: the moment I comment NGRAM_DATASET=("en") again and restart the app, the error still occurs.

Q: is there an option to "redownload"?

[nebulon]

@luckow NGRAM_DATASET_PATH has a default set to /app/data/ngram so that does not need to be set, it is only there if someone want to move that to a volume due to the required disk size.

Currently you have to delete the folder for that specific language to trigger a redownload after app restart.

[luckow]

My mistake. I restarted the app too early.
After the full download of almost 8 GB EN n-gram dataset, you need to give your Cloudron instance enough time to unpack and move the files to (by default) /app/data/ngram/en.

In my case, it took more than 15 minutes to meet the requirements.
The moment you restarted the app too early, the zip file was not unzipped completely. In this case the error occurs and you have to start again from the beginning.

Search your logs for ===> en ngram dataset has been installed.

[nebulon]

The latest package version does not require the manual restart anymore, but will restart the internal component automatically after the download has finished.