SOLVED Is a non JS version possible? How to wget files?

[jdaviescoates]

@scooke notified me that people on reddit have found my Tools for Change library of resources:

https://www.reddit.com/r/opendirectories/comments/l0hbvw/just_a_collection_of_a_tons_of_pdfs/

Quite a few people have noted that they are unable to download it all using wget etc.

Is there are way to let people browse the files without JS and to be able to download using wget?

[marcusquinn]

@jdaviescoates I use "SiteSucker" for that kind of thing. I'm on Mac but there might be a Windoze version.

[nebulon]

Do you mean like downloading a whole folder tree or so? The individual pdfs work fine to wget.

Otherwise how is this done with wget for other pages? I guess some recoursive link grabbing?

[jdaviescoates]

@nebulon said in Is a non JS version possible? How to wget files?:

Do you mean like downloading a whole folder tree or so?

Yes, for people who want to download the whole thing. All the folders at once.

[nebulon]

@jdaviescoates hm indeed this is currently not supported. There are no server-side rendering capabilities to provide such a page. Do you think giving the option to download a folder tree as a zip or tarball would make sense?

[mehdi]

@nebulon I think a simple server-side rendering of a list of links when listing a folder would be better than the ability to download a hole folder as tar. That's the way nginx / apache do it, and it shouldn't be too hard to implement IMO

[robi]

you can provide a convenience zip/tgz via the terminal.. and with a cron feature do it regularly.

[nebulon]

I've added basic support for file listing together with the <noscript> tag, which allows the usage of wget -r or also text based browsers like lynx.

Conveniently it also removes one server roundtrip for the dynamic version as the template also injects the folder content with a <script> tag to pickup by the vuejs app.

This is only for the public page so far and pending a surfer package release (hopefully tomorrow)

If anyone is interested, the change is at https://github.com/cloudron-io/surfer/commit/4d70d060961422a50528c9186061d458d261aa3d

I actually had a hard time to test this in firefox, since the noscript addon would simply not execute javascript, but ignores the <noscript> tag. Does anyone know how to test such a scenario?

[mehdi]

@nebulon What I do in this case is not put the elements destined for non-JS environment in a noscript tag, but rather just put them at the root of the element that's gonna be the Vue.JS app. So, as soon as Vue.JS takes over, it disappears.

[nebulon]

@mehdi thats kinda smart, I have to see how much flickering that adds to the UI as such, but otherwise I think I'd prefer your solution.

[jdaviescoates]

@nebulon sounds like you're making great progress, thanks!

[nebulon]

@jdaviescoates I published a new version 5.16.0 which has basic noscript support.
I decided against @mehdi 's suggestion, since the UI flicker during folder navigation was a bit too much. Generally I would have preferred that though.

[jdaviescoates]

@nebulon great, thanks! I'll let that reddit thread know and see if the people who were having issues before are OK now

[jdaviescoates]

@nebulon I just tried doing wget -r myself and it worked perfectly!

However, if you e.g. install noscript extension and visit it you can't actually scroll down pages.

So even though it's now possible to wget the files (which is great, thanks!), it's still not possible to simply browse the files in a browser without javascript enabled.

It'd be awesome if that could be made possible too!?

[mehdi]

@jdaviescoates NoScript is different from "a browser without javascript enabled". NoScript removes the script tags, but does not disable javascript. If you go to another browser, actually disable javascript in the preferences, it should work

[jdaviescoates]

@mehdi said in Is a non JS version possible? How to wget files?:

If you go to another browser, actually disable javascript in the preferences, it should work

I just tried that too, but it doesn't seem to work. Still can't scroll down pages e.g. https://library.uniteddiversity.coop/Cooperatives/

[nebulon]

As @mehdi mentiones the NoScript extension is not the right thing here, but if you go to about:config in Firefox and set javascript.enabled to false, then even without a browser restart you would see the static page instead. Haven't tested chrome based browser though.

[jdaviescoates]

@nebulon thanks, but even just doing that (and the equivalent in Chromium) it's not that I don't see the static page, I do. It's that you can't scroll down the page and browse files further down the page.

e.g. turn off javascript and go here: https://library.uniteddiversity.coop/Cooperatives/ and you can't get past here:

[nebulon]

@jdaviescoates well that is the basic noscript page for now. I guess it is not sorted nor gives indication of the filetype, yet.

[jdaviescoates]

@nebulon said in Is a non JS version possible? How to wget files?:

I guess it is not sorted nor gives indication of the filetype, yet.

Well, it looks like it is sorted alphabetically anyway and you can see the filetype anyway from the filename.

Just can't scroll down the page.

But you've solved the primary issue of not being able to wget the files so thanks for that.