<![CDATA[grab-site]]>https://github.com/ArchiveTeam/grab-site

grab-site

Build status

grab-site is an easy preconfigured web crawler designed for backing up websites. Give grab-site a URL and it will recursively crawl the site and write WARC files. Internally, grab-site uses a fork of wpull for crawling.

grab-site gives you

  • a dashboard with all of your crawls, showing which URLs are being grabbed, how many URLs are left in the queue, and more.

  • the ability to add ignore patterns when the crawl is already running. This allows you to skip the crawling of junk URLs that would otherwise prevent your crawl from ever finishing. See below.

  • an extensively tested default ignore set (global) as well as additional (optional) ignore sets for forums, reddit, etc.

  • duplicate page detection: links are not followed on pages whose content duplicates an already-seen page.

The URL queue is kept on disk instead of in memory. If you're really lucky, grab-site will manage to crawl a site with ~10M pages.

dashboard screenshot

]]>https://forum.cloudron.io/topic/8420/grab-siteRSS for NodeTue, 21 Apr 2026 05:00:30 GMTTue, 10 Jan 2023 23:24:35 GMT60<![CDATA[Reply to grab-site on Wed, 11 Jan 2023 09:59:25 GMT]]>@robi grab-site is a great suggestion and I hope Cloudron supports it. @jdaviescoates makes a good recommendation too.

After the website is grabbed, the next phase is reading and searching it offline. I don't know if you have had much joy trying that with grab-site.

If grab-site can be supported, it is not very far from being able to support YaCy too, which also visits websites and crawls the pages. There is a request for YaCy support on Cloudron here:

https://forum.cloudron.io/topic/2715/yacy-decentralized-web-search?_=1673430654350

]]>https://forum.cloudron.io/post/59850https://forum.cloudron.io/post/59850Wed, 11 Jan 2023 09:59:25 GMT<![CDATA[Reply to grab-site on Wed, 11 Jan 2023 05:11:38 GMT]]>Useful utility.

This free tool https://www.httrack.com/ does this very well too.

]]>https://forum.cloudron.io/post/59840https://forum.cloudron.io/post/59840Wed, 11 Jan 2023 05:11:38 GMT<![CDATA[Reply to grab-site on Tue, 10 Jan 2023 23:52:04 GMT]]>@robi This looks very interesting!

]]>https://forum.cloudron.io/post/59833https://forum.cloudron.io/post/59833Tue, 10 Jan 2023 23:52:04 GMT