LibreCrawl: self-hosted website crawler for SEO audits (community package) - open-source alternative to Screaming Frog, Sitebulb, DeepCrawl, etc
-
I've packaged LibreCrawl (MIT), a self-hosted website crawler for SEO audits, as a Cloudron community app.
It crawls sites from your server and shows status codes, titles, meta descriptions, headings, structured data, issues, internal and external links and a site-structure graph. Crawls are saved, reloadable and exportable (CSV, Excel, JSON, XML), and a REST API lets other tools start crawls and fetch results.
Install (Cloudron 9.1 or later): from Community Apps, or add this versions URL under Community apps in your dashboard settings, or run:
cloudron install --versions-url https://raw.githubusercontent.com/marcusquinn/cloudron-librecrawl-app/main/CloudronVersions.json --location librecrawlSign in as
adminwith the generated password from/app/data/admin-password(the post-install note shows how), then change it.What the package does
- Signed-in use only. Registration and guest access are off, and modes that skip sign-in stop the app at startup. Create accounts with
librecrawl-user addin the app's Terminal, one per person or per site. - Users, settings, and crawls live in SQLite under
/app/dataand are backed up. The database is snapshotted before a new LibreCrawl version migrates it. - A daily task removes finished crawls older than 90 days (
CRAWL_RETENTION_DAYSin/app/data/env). - Memory defaults to 2 GiB. On a test, a 5,000-page HTML crawl peaked at 1.16 GiB. Allow 4 GiB for several large crawls in the same hour, or for JavaScript rendering beyond small sites.
- Chromium rendering is available but off by default. tini runs as PID 1 so Chromium's helper processes are reaped.
Please read before giving anyone an account:
LibreCrawl has no filter for private addresses.
Anyone who can sign in can make the server request any address it can reach, including other apps on your Cloudron and your private network.
Only create accounts for people and sites you trust. An upstream pull request proposes a guard; it isn't merged yet.
Upstream behaviour worth knowing
- The crawl delay only slows JavaScript crawls. HTML crawls are limited only by concurrency, plus up to five HEAD checks at a time.
- JavaScript crawls send their own user agent (
jsUserAgent). Set both if a firewall allowlists your crawler. - The page limit counts fetched pages, but results also list every image and external link that was only HEAD-checked.
- A crawl session idle for an hour is dropped, even mid-crawl. API clients should keep polling or use the saved-crawl endpoints.
Source, docs and issues: https://github.com/marcusquinn/cloudron-librecrawl-app.
It's packaged independently of the LibreCrawl project.
Feedback welcome, especially memory figures from larger or JavaScript-heavy sites.
Website & screenshots
Web Design & Development: https://www.evergreen.je
Technology & Apps: https://www.marcusquinn.com - Signed-in use only. Registration and guest access are off, and modes that skip sign-in stop the app at startup. Create accounts with
-
I've packaged LibreCrawl (MIT), a self-hosted website crawler for SEO audits, as a Cloudron community app.
It crawls sites from your server and shows status codes, titles, meta descriptions, headings, structured data, issues, internal and external links and a site-structure graph. Crawls are saved, reloadable and exportable (CSV, Excel, JSON, XML), and a REST API lets other tools start crawls and fetch results.
Install (Cloudron 9.1 or later): from Community Apps, or add this versions URL under Community apps in your dashboard settings, or run:
cloudron install --versions-url https://raw.githubusercontent.com/marcusquinn/cloudron-librecrawl-app/main/CloudronVersions.json --location librecrawlSign in as
adminwith the generated password from/app/data/admin-password(the post-install note shows how), then change it.What the package does
- Signed-in use only. Registration and guest access are off, and modes that skip sign-in stop the app at startup. Create accounts with
librecrawl-user addin the app's Terminal, one per person or per site. - Users, settings, and crawls live in SQLite under
/app/dataand are backed up. The database is snapshotted before a new LibreCrawl version migrates it. - A daily task removes finished crawls older than 90 days (
CRAWL_RETENTION_DAYSin/app/data/env). - Memory defaults to 2 GiB. On a test, a 5,000-page HTML crawl peaked at 1.16 GiB. Allow 4 GiB for several large crawls in the same hour, or for JavaScript rendering beyond small sites.
- Chromium rendering is available but off by default. tini runs as PID 1 so Chromium's helper processes are reaped.
Please read before giving anyone an account:
LibreCrawl has no filter for private addresses.
Anyone who can sign in can make the server request any address it can reach, including other apps on your Cloudron and your private network.
Only create accounts for people and sites you trust. An upstream pull request proposes a guard; it isn't merged yet.
Upstream behaviour worth knowing
- The crawl delay only slows JavaScript crawls. HTML crawls are limited only by concurrency, plus up to five HEAD checks at a time.
- JavaScript crawls send their own user agent (
jsUserAgent). Set both if a firewall allowlists your crawler. - The page limit counts fetched pages, but results also list every image and external link that was only HEAD-checked.
- A crawl session idle for an hour is dropped, even mid-crawl. API clients should keep polling or use the saved-crawl endpoints.
Source, docs and issues: https://github.com/marcusquinn/cloudron-librecrawl-app.
It's packaged independently of the LibreCrawl project.
Feedback welcome, especially memory figures from larger or JavaScript-heavy sites.
Website & screenshots
@marcusquinn nice work
- Signed-in use only. Registration and guest access are off, and modes that skip sign-in stop the app at startup. Create accounts with
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login