Paperless - indexing and archiving scanned documents

doodlemania2

@doodlemania2 just an fyi - this is working well, but the "consumption" directory isn't polling so still tinkering with that. I am able to upload manually and auto tagging and pdf extraction is working. so, we're about 95% there I think.

scooke

@doodlemania2 Keep up the good work!

absnt_mind

@doodlemania2 Very excited about your efforts on this! I appreciate you, sir! I'm a vanilla PHP and JS developer for the most part, so node, docker, manifests, build scripts, all of it gets a little sketchy if I'm the one on it. I would love to learn how to package Cloudron apps, though. Maybe I'll have a look at some of the material that's available on that here later.
I have been aching for a good document management system with user permissions for probably the last 12 months or so.

doodlemania2

@girish - I'm officially stuck. The app is ready and I think the tests might even work except for the consumption directory.

Described my issue here: https://github.com/jonaswinkler/paperless-ng/issues/438.

Tried moving the supervisors around, nothing.

If I run the consume script from shell, works like a champ. Open to ideas.

jimcavoli

@doodlemania2 Is that a periodic job? Even if not, could we just hack around it with a small caveat by using the shell command and scheduler add-on? It'd be a minor thing, I think, for the Cloudron package to run an "every minute" or something consumption

doodlemania2

@jimcavoli That's a fabulous idea - I will give that a try - it uses inotify on the FS to monitor for changes (intended to always run), but this could be a fabulous workaround! ON IT!

jimcavoli

@doodlemania2 Sure thing! Also, didn't know it was inotify under the hood - that has a fraught history with dockerized runtimes, so this is likely your best option anyway

doodlemania2

@jimcavoli alrighty - that seems to have worked (running it with cloudron scheduler). now just need someone to help me with tests and it can ship!

girish

@doodlemania2 I marked this as WIP now! Great progress.

ultraviolet

hi all,

I was actually working on this a few weeks ago. I forked @doodlemania2's repo and updated it to the latest version. I also enabled SFTP access I am also working to implement the proxyauth addon to pass though to the django app (still wip) as it does not support LDAP natively.

I don't have access to fork in the cloudron gitlab so I had to copy to my github account. You can find the repo here:
https://github.com/campbellmcgregor/paperless-ng-cloudron

thanks

doodlemania2

@ultraviolet would you send me a PR? I'd be delighted to merge those in!

ultraviolet

@doodlemania2 sure next time I am at a desk I will send you a PR

girish

@ultraviolet said in Paperless - indexing and archiving scanned documents:

I don't have access to fork in the cloudron gitlab so I had to copy to my github account

Whoops, sorry about that. By default, accounts cannot fork since we get too much spam. I assume you are https://git.cloudron.io/admin/users/ultraviolet ? I have increased the project limit already.

ultraviolet

@girish yup that is me thanks!

ultraviolet

@doodlemania2 sent you a PR

doodlemania2

@ultraviolet Merged! That's fantastic - great work!!!

ultraviolet

@doodlemania2 thanks, I may send you another PR in a day or so hit a few walls with the proxyauth.

girish

@ultraviolet @doodlemania2 Per https://github.com/jonaswinkler/paperless-ng/discussions/654, it seems if someone knows some python we can add proper ldap support.

doodlemania2

@girish oooo shiny! @ultraviolet -- that you?

timconsidine

@doodlemania2 great work, thank you for making this available.
I built (very smooth) and installed on my Cloudron.
As a noob to paperless-ng, I am not sure what is needed for "processing".
I uploaded few documents to test and they seem stuck in 'processing' in dashboard view.
A tag and correspondent don't seem to get auto-applied.
Is this functionality working ?
Do I need to do something ?

timconsidine

@timconsidine Seems last was my noob issue.
The dashboard clears eventually and tags can be retro-applied with

root@<app-instance>:/app/code/src# python3 manage.py document_retagger -T

New Q : I'm looking to add schedule tasks but cannot see in doc a list of task.function. Is there one available?

doodlemania2

@timconsidine Glad you got it working - I'm note sure how the scheduled tasks stuff works - since it does what i need out of the box, I just ignored those

timconsidine

@doodlemania2 no problem, I understand, and likewise I have started using it as it does everything I currently want 'out of the box'.
I just don't like to miss a trick by getting even more value out of it by using 'advanced options'.

I have to say : thank you again. It's an awesome app, and solves a problem of coordinating docs between users (in my case purchase invoices needing review and input). Previously I have been using a shared folder in Nextcloud, and although that works, I have had issues when some user's sync to Nextcloud stops, and everyong starts shouting at each other because they are seeing different views of the world.

The only things not there which I can identify which would be nice :

"tag tree" view, which I notice Teedy has. Just a UI advantage
a way to remove tags w/o manually doing so, based on some conditions

But hey ho, just "nice to have" not essentials.

doodlemania2

@timconsidine outstanding! really glad it works for you

scooke

@timconsidine Hello! To get this installed, did you download the source, cd into the directory and run cloudron build then cloudron install? I only once ever managed to install a non-official app but I totally forget how I did it!

One place I get stuck on conceptually is at https://docs.cloudron.io/custom-apps/tutorial/, where it tells us to build:

# build the app
$ docker build -t username/nodejs-app:1.0.0 .

What in the world is "username" supposed to be? Just a bit earlier it tells us to git clone https://git.cloudron.io/cloudron/tutorial-nodejs-app, so why wouldn't the build section just use cloudron as the username? And if it isn't cloudron but myself, does this mean it is going to be put in a directory in the directory I'm building in? Anyway, this stuff confuses the heck out of me.

timconsidine

@scooke No I didn't use cloudron build
My process is :

install cloudron cli (but I guess you did this)
have your own docker repository : this is what is meant by username. I think it is best described as reponame. I am not so sophisticated as to have my own private repository system. I just use the free account on https://hub.docker.com/
install Docker Desktop for your platform
on local machine create a directory e.g. /cloudron-custom/appname and change into the appname directory
download a zip file of the app source and unzip into the appname directory
make sure you have Docker Desktop running (which logs you into the free hub.docker account)
then build using docker build -t <reponame>/<appname>:<tag> . NB my change of terminology, and remember the trailing dot
if I understand correctly, tag is only to distinguish different builds of the app. Initially i just ignored these, but probably good practice. But they can be anything (date, codeword)
the built app is now just on your local machine so get it into the reppository system you chose, e.g. hub.docker.com using docker push <reponame>/<appname>:<tag>
then it is available to cloudron cli so cloudron install --image <reponame>/<appname>:<tag> will ask you what location you want (e.g. appname.yourdomain.com)
generally it is seamless from there, watch the pretty dots and when finished it should be in your Cloudron dashboard, ready to set access permissions or resources if you don't want the default.

I guess this is not a sophisticated process, it has multiple steps, but I find that easier when something goes wrong to debug.

I think my build of Paperless-NG is available publicly at tcmbp132021/paperless-ng:derek210709
I've never tried it but maybe you could save some effort and try just doing cloudron install --image tcmbp132021/paperless-ng:derek210709
Maybe there will be some permission issues, but it says it is public. If you try that, let me know, it will be useful to know.

I hope that's not too granular for anybody, personally I just like having step-by-step instructions. Good luck and shout if you run into problems. You will succeed !

scooke

@timconsidine This is super helpful. I guess another thing that confuses me is, if I understand this process, the hub.docker.com site is going to end up (or already is) jam-packed with 1001 Docker images from everyone and their grandmother. Seems ineffecient and confusing to me.

I'll try skipping to the install step with the filled in <reponame>:<appname> details you gave. Thanks!

timconsidine

@scooke I think that's Docker's problem !
No different to any other SaaS-style hosted service. Providing performance on your account is OK, it's probably nothing to worry about.

Or alternatively set up a private repository. Isn't there a Cloudron app already for that ?

I've not been brave enough / had enough time to do that yet.

scooke

@doodlemania2 With the help of @timconsidine I have this installed also. Looks and runs smooth! Thank you!

nebulon

So this app is published as unstable in the paperless-ng flavor, thanks to @doodlemania2 for the majority of the initial packaging!

The forum section for further feedback and issues is now https://forum.cloudron.io/category/137/paperless-ng so I will lock this topic.

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Solved Paperless - indexing and archiving scanned documents