It crawls sites from your server and shows status codes, titles, meta descriptions, headings, structured data, issues, internal and external links and a site-structure graph. Crawls are saved, reloadable and exportable (CSV, Excel, JSON, XML), and a REST API lets other tools start crawls and fetch results.

Install (Cloudron 9.1 or later): from Community Apps, or add this versions URL under Community apps in your dashboard settings, or run:

cloudron install --versions-url https://raw.githubusercontent.com/marcusquinn/cloudron-librecrawl-app/main/CloudronVersions.json --location librecrawl

Sign in as admin with the generated password from /app/data/admin-password (the post-install note shows how), then change it.

What the package does

• Signed-in use only. Registration and guest access are off, and modes that skip sign-in stop the app at startup. Create accounts with librecrawl-user add in the app's Terminal, one per person or per site.
• Users, settings, and crawls live in SQLite under /app/data and are backed up. The database is snapshotted before a new LibreCrawl version migrates it.
• A daily task removes finished crawls older than 90 days (CRAWL_RETENTION_DAYS in /app/data/env).
• Memory defaults to 2 GiB. On a test, a 5,000-page HTML crawl peaked at 1.16 GiB. Allow 4 GiB for several large crawls in the same hour, or for JavaScript rendering beyond small sites.
• Chromium rendering is available but off by default. tini runs as PID 1 so Chromium's helper processes are reaped.

Please read before giving anyone an account:

LibreCrawl has no filter for private addresses.

Anyone who can sign in can make the server request any address it can reach, including other apps on your Cloudron and your private network.

Only create accounts for people and sites you trust. An upstream pull request proposes a guard; it isn't merged yet.

Upstream behaviour worth knowing

• The crawl delay only slows JavaScript crawls. HTML crawls are limited only by concurrency, plus up to five HEAD checks at a time.
• JavaScript crawls send their own user agent (jsUserAgent). Set both if a firewall allowlists your crawler.
• The page limit counts fetched pages, but results also list every image and external link that was only HEAD-checked.
• A crawl session idle for an hour is dropped, even mid-crawl. API clients should keep polling or use the saved-crawl endpoints.

Source, docs and issues: https://github.com/marcusquinn/cloudron-librecrawl-app.

It's packaged independently of the LibreCrawl project.

Feedback welcome, especially memory figures from larger or JavaScript-heavy sites.

Website & screenshots

• https://librecrawl.com/