indexing of office documents?

[fbartels]

@doodlemania2 I was looking for some further information on this.

Paperless-ng uses Tika (to extract data from other file types) and Gotenberg (for pdf conversion) for this.

More at https://paperless-ng.readthedocs.io/en/latest/configuration.html#tika-settings

[ChristopherMag]

This is something I need as well, working on rolling out PaperlessNGX to replace an existing storage system and need to be able to store office documents as well so that all documents related to an entity are together.

It looks like Tika and Gotenberg both have docker containers available.

Do we need to have separate Tika and Gotenberg apps like we have OnlyOffice and Collabra Online as a separate apps though they are not usable on their own and are used by NextCloud for handling office documents?

[ChristopherMag]

For anyone wanting to get this up and running quickly if you have docker running on another system you can run the following:

docker run -d --restart unless-stopped -p 3000:3000 gotenberg/gotenberg
docker run -d --restart unless-stopped -p 9998:9998 apache/tika

And then add the following to your paperless.conf:

# Tika
PAPERLESS_TIKA_ENABLED=true
PAPERLESS_TIKA_ENDPOINT=http://<DockerHostnameOrIPGoesHere>:9998
PAPERLESS_TIKA_GOTENBERG_ENDPOINT=http://<DockerHostnameOrIPGoesHere>:3000

After this you can upload xlsx, docx, etc. to paperless-ngx.

In my testing if the Docker host running the Tika and Gotenberg containers goes down paperless-ngx keeps working fine but you won't be able to upload additional xlsx/docx/etc. documents until you restart the containers which works out fine as the reliability of paperless-ngx being accessible is way more important than this one feature working for us.

[jdaviescoates]

@ChristopherMag thanks for sharing!

@staff can we get this into the package somehow?

[girish]

@jdaviescoates yes, I will put this as a point to investigate for next release. I think we have to investigate what nextcloud needs for FTS as well as apps like these and design accordingly. (Maybe they have to become addons or alternately we can just put them in the app itself).

[jdaviescoates]

@girish said in indexing of office documents?:

I think we have to investigate what nextcloud needs for FTS as well

On that, I spotted recently that they do now at least mention Solr under Platform Apps over on https://github.com/nextcloud/fulltextsearch but as far as I can tell from the linked wiki https://github.com/nextcloud/fulltextsearch/wiki to date there is still only an Elastic Search Platform App.

[timconsidine]

@ChristopherMag do you or anyone else have experience of using this kind of setup for Mac OS document formats like Pages and Numbers ?

[ChristopherMag]

@timconsidine It looks like Apache Tika supports the document formats from the iWork suite like pages.

I tried to upload a .pages file to paperless-ngx with Tika and gotenberg configured and paperless popped up a failure message with the the error File type application/zip not supported.

I believe this signals.py file in the paperless-ngx project would need to add support for the various iWork software suite formats to resolve this error and get this working assuming you already have Tika and gotenberg setup and working with paperless-ngx.

You could probably open a github issue in the paperless-ngx repository on Github and see if they can assist with adding support fort his.

[necrevistonnezr]

Any updates on this?

[girish]

@necrevistonnezr nothing yet...

[neurokrish]

Hi, is there an update on this? I tried @ChristopherMag 's suggestion. However, I get connection refused for Tika. Is this something to do with iptables? How can I allow connection to the container for paperless app to access?

EDIT: I must say that I have installed docker - tika and gotenberg in the same system as Cloudron.

[nebulon]

I don't think we have an update on this yet. Possibly your containers are not within the same docker network on the system? Either way adding docker container on the side of Cloudron will break on Cloudron updates, so this is not very useful to investigate as such. Have you instead tried to run the required services on a separate isolated server instead?

[neurokrish]

@nebulon , thanks for your reply. Tried both ways, containers outside and inside Cloudron network. Good to know doing the later will break updates. Removed those containers now. Is it difficult to pre-install these containers via the app itself? Alternatively, may be provide them as separate installations as separate Cloudron apps which can be linked to paperless?

[nebulon]

Unless Tika and Gotenburg are useful for other apps, it may make more sense to actually package them as part of paperless and pre-configure everything.

Does anyone have experience on the memory requirement for those?

[ChristopherMag]

Fyi gotenberg publishes cloudron specific images now. Not sure the history of how or why that was started but I would assume those are meant to be used as a cloudron app though I don't see any app in the app store for it.

PS, don't use these images for your own gotenberg instance that your integrating with paperless, they exist hopefully to make it easier one day to run gotenberg on cloudron directly.

[necrevistonnezr]

@ChristopherMag It says ‘cloudrun’ - sure it’s just a typo or does it mean something like ‘cloud-run’?

[ChristopherMag]

@necrevistonnezr Wow, your right, those images probably have nothing to do with cloudron!

Thanks for pointing that out, for all others, please disregard my previous comment.