Mayan EDMS

nebulon

Mayan EDMS is a Free Open Source Electronic Document Management System, coded in the Python language using the Django web application framework and released under the Apache 2.0 License. It provides an electronic vault or repository for electronic documents.

https://mayan.readthedocs.io/en/latest/
https://hub.docker.com/r/mayanedms/mayanedms/

nebulon

The upstream link is https://www.mayan-edms.com/

ultraviolet

Hi all,

I have started to package this and it it working but the initial start script just not executing during install. If you run the /app/pkg/start.sh it will work fine.

I have been working on and off with this for a while but could use some help in getting the last bit over the line because it is driving me nuts.

Repo is here

Any help appreciated.

ultraviolet

I am seeing errors as supervisord tries to write logs to the read only folder /var/logs what is the best thing todo? Any suggestions?

nebulon

There are two settings to be done for supervisor to work well with regards to logging. The actual service file for an application should have those options set:

stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0

As well as in the Dockerfile adding the following line will ensure supervisor itself writes to a writeable location:

RUN sed -e 's,^logfile=.*$,logfile=/run/supervisord.log,' -i /etc/supervisor/supervisord.conf

ultraviolet

Thanks @nebulon I think that was the last piece of the puzzle!

ultraviolet

This is working and now needs some people to test installs etc. it is still rough around the edges, needs ldap and mail etc. but I will get this sorted over the coming days.

I should mention the initial install takes around 10 mins as there is a lot of database migrations and django stuff to get though.

nebulon

I've just tried it and it seems to work very well so far!

scooke

I'll try it out. What is the link please?

ultraviolet

https://github.com/euanmcgregor/mayan-cloudron

ergabergblerg

@ultraviolet Thanks so much for setting this up. I tried building and running on my server but the heartbeat never completes. I'm assuming the problem has to do with this bit in my logs:

Oct 10 21:50:56 Traceback (most recent call last):
Oct 10 21:50:56 File "/app/data/venv/mayan-edms/bin/mayan-edms.py", line 10, in <module>
Oct 10 21:50:56 execute_from_command_line(sys.argv)
Oct 10 21:50:56 File "/app/data/venv/mayan-edms/lib/python3.7/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
Oct 10 21:50:56 utility.execute()
Oct 10 21:50:56 File "/app/data/venv/mayan-edms/lib/python3.7/site-packages/django/core/management/__init__.py", line 357, in execute
Oct 10 21:50:56 django.setup()
Oct 10 21:50:56 File "/app/data/venv/mayan-edms/lib/python3.7/site-packages/django/__init__.py", line 24, in setup
Oct 10 21:50:56 apps.populate(settings.INSTALLED_APPS)
Oct 10 21:50:56 File "/app/data/venv/mayan-edms/lib/python3.7/site-packages/django/apps/registry.py", line 91, in populate
Oct 10 21:50:56 app_config = AppConfig.create(entry)
Oct 10 21:50:56 File "/app/data/venv/mayan-edms/lib/python3.7/site-packages/django/apps/config.py", line 116, in create
Oct 10 21:50:56 mod = import_module(mod_path)
Oct 10 21:50:56 File "/app/data/venv/mayan-edms/lib/python3.7/importlib/__init__.py", line 127, in import_module
Oct 10 21:50:56 return _bootstrap._gcd_import(name[level:], package, level)
Oct 10 21:50:56 File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
Oct 10 21:50:56 File "<frozen importlib._bootstrap>", line 983, in _find_and_load
Oct 10 21:50:56 File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
Oct 10 21:50:56 File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
Oct 10 21:50:56 File "<frozen importlib._bootstrap_external>", line 728, in exec_module
Oct 10 21:50:56 File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
Oct 10 21:50:56 File "/app/data/venv/mayan-edms/lib/python3.7/site-packages/mayan/apps/appearance/apps.py", line 3, in <module>
Oct 10 21:50:56 from mayan.apps.common.apps import MayanAppConfig
Oct 10 21:50:56 File "/app/data/venv/mayan-edms/lib/python3.7/site-packages/mayan/apps/common/apps.py", line 35, in <module>
Oct 10 21:50:56 from .tasks import task_delete_stale_uploads # NOQA - Force task registration
Oct 10 21:50:56 File "/app/data/venv/mayan-edms/lib/python3.7/site-packages/mayan/apps/common/tasks.py", line 7, in <module>
Oct 10 21:50:56 from mayan.celery import app
Oct 10 21:50:56 File "/app/data/venv/mayan-edms/lib/python3.7/site-packages/mayan/celery.py", line 5, in <module>
Oct 10 21:50:56 from .runtime import celery_class
Oct 10 21:50:56 File "/app/data/venv/mayan-edms/lib/python3.7/site-packages/mayan/runtime.py", line 5, in <module>
Oct 10 21:50:56 celery_class = import_string(setting_celery_class.value)
Oct 10 21:50:56 File "/app/data/venv/mayan-edms/lib/python3.7/site-packages/django/utils/module_loading.py", line 17, in import_string
Oct 10 21:50:56 module = import_module(module_path)
Oct 10 21:50:56 File "/app/data/venv/mayan-edms/lib/python3.7/importlib/__init__.py", line 127, in import_module
Oct 10 21:50:56 return _bootstrap._gcd_import(name[level:], package, level)
Oct 10 21:50:56 File "/app/data/venv/mayan-edms/lib/python3.7/site-packages/celery/__init__.py", line 153, in <module>
Oct 10 21:50:56 from . import local # noqa
Oct 10 21:50:56 File "/app/data/venv/mayan-edms/lib/python3.7/site-packages/celery/local.py", line 17, in <module>
Oct 10 21:50:56 from .five import PY3, bytes_if_py2, items, string, string_t
Oct 10 21:50:56 File "/app/data/venv/mayan-edms/lib/python3.7/site-packages/celery/five.py", line 7, in <module>
Oct 10 21:50:56 import vine.five
Oct 10 21:50:56 ModuleNotFoundError: No module named 'vine.five'

girish

@ergabergblerg Try this repo https://git.cloudron.io/cloudron/mayan-edms-app/ . There is still some celery error but I can login. Have to look into it later this week.

ultraviolet

ah yes there was an error. I have fixed it in my dev box. It was an unpinned dependency, there are also issues with upgrading so it is still very much beta.

I will publish the change shortly.

@girish could you give me access to the cloudron hosted repo so I can make my commits direct?

The latest push in my own public repo here has the fix. For info this is the change that pins the dependency to allow compiling. I have mentioned it in the upstream project and it should be fixed soon if it has not been already.

ultraviolet

Ok I have been testing the upgrading which is tricky for me to figure out. Could use a fresh pair of eyes.

This is how this image is working:

Build image with deps
pull the image version from the upstream repo based on the version number in the dockerfile
run down and install everything
Once complete the start.sh script in /app/pkg kicks off.
After defining all the environment variables it will run this line of code:

if [ ! -e "/app/data/init-completed" ]; then
  /app/data/venv/mayan-edms/bin/mayan-edms.py initialsetup
  echo "Setting up for initial install, this may take some time"
  touch /app/data/init-completed
fi

If there is a file called init-completed it will skip it.

Now this is where the issue with upgrades come in I believe there is a command /app/data/venv/mayan-edms/bin/mayan-edms.py performupgrade.

I have tried testing upgrades for a few hours but when it starts up all the services I simply get access denied issues.

To replicate:

Clone my repo https://github.com/euanmcgregor/mayan-cloudron
Do an install to your cloudron - check that it works and login to check the version it should
Alter the dockerfile and change the variable from ENV MAYAN_VERSION=3.4.17 to ENV MAYAN_VERSION=3.5.1 which is the latest version
Perform a cloudron update

Hope this make sense any help is appreciated.

girish

@ultraviolet You should have permissions now, thanks

ultraviolet

@girish thanks

I am going back to the drawing board for this one. So watch this space all....

ultraviolet

I have come up with another way of installing to allow me to update the package. In the start.sh it will do it's initial install but that function of the app installs further items into /app/code/ which during an non debug session is RO. Is there any other way to allow this to install then seal the system to RO?

An example during initialsetup:

  Applying tags.0006_documenttag... OK
  Applying tags.0007_auto_20170118_1758... OK
  Applying tags.0008_auto_20180917_0646... OK
  Applying user_management.0001_initial... OK
  Applying web_links.0001_initial... OK
  Applying web_links.0002_auto_20191210_0436... OK
  Applying web_links.0003_auto_20191211_0233... OK
Installing package: Bootstrap (=3.4.1)... Downloading... Verifying... Extracting... Patching files... Traceback (most recent call last):
  File "/usr/lib/python3.7/pathlib.py", line 1258, in mkdir
    self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/app/code/venv/lib/python3.7/site-packages/mayan/apps/appearance/static/appearance/node_modules/bootstrap'

<snip>


    self._accessor.mkdir(self, mode)
OSError: [Errno 30] Read-only file system: '/app/code/venv/lib/python3.7/site-packages/mayan/apps/appearance/static/appearance/node_modules'

Or if there is any other suggestions that I could try to get this installed?

EDIT: I should add I am unable to do this during build as the app needs the database etc. to migrate and change the database

robi

@ultraviolet you can make a note of all the errors and softlink them to someplace else for the build.

ultraviolet

@robi good idea, never thought of that. Thanks

ultraviolet

ok think we are good to go. I have pushed the code to my github the repo I will merge with the Cloudron hosted git soon.

Just need some people to test install etc. and then will look at getting some tests into the package. It is still on version 3.4.17 as the latest version 3.5.1 has some issues in the celery queues which I will look at once that branch is a bit more stable.

daniel.goletz

When you need a DAU for testing, i'm in!

jdaviescoates

@daniel-goletz said in Mayan EDMS:

DAU

Daily Active User?

envisible

The latest 3.5.2 release seems to working for me know (just change the version number in the Dockerfile), it fixes the celery errors and the Mayan EDMS API is working now. The only issue I have is a folder permissions issue, but that's an easy fix (with chmod).

girish

@envisible What is the folder permissions issue?

envisible

@girish
when I add the a watch folder in (via the file manager) in /app/data/media/ and try to process the docs, I get:

Oct 29 16:41:07 mayan.apps.sources.tasks <44> [ERROR] "task_check_interval_source() line 45 Error processing source id: 2; [Errno 13] Permission denied: '/app/data/media/t/104-10005-10016.pdf'"
Oct 29 16:41:07 [2020-10-29 16:41:07,424: ERROR/ForkPoolWorker-1] Error processing source id: 2; [Errno 13] Permission denied: '/app/data/media/t/104-10005-10016.pdf'

When I change the permission to cloudron (instead of root all via the file manager) I get:

Oct 29 16:37:29 mayan.apps.converter.backends.python <44> [ERROR] "get_page_count() line 185 Exception determining page count using Pillow; cannot identify image file <File: /app/data/media/document_storage/9692aadc-df6a-40cd-aab0-eb17f8063486>"
Oct 29 16:37:29 [2020-10-29 16:37:29,263: ERROR/ForkPoolWorker-1] Exception determining page count using Pillow; cannot identify image file <File: /app/data/media/document_storage/9692aadc-df6a-40cd-aab0-eb17f8063486>

But if I chmod -R 777 the folder it all works perfectly; I haven't played around with lesser permissions than that though.

girish

@envisible Thanks! I will look into getting this app published atleast as unstable then to get more testers.

ctrl

@girish testee here

marcusquinn

Nice work! Vids in the docs look cool: https://docs.mayan-edms.com/

I'd like to give this a run too.

Apparently this is popular with Lawyers & Finance companies: https://www.docmoto.com/videos/ so a FOSS alternative would be interesting for document-sensitive industries.

subven

Hmm looks nice but I ask myself if it's compatible to german GoBD and EU GDPR laws. I use ecoDMS at the moment.

girish

@subven As a general question, what could make self-hosted software incompatible with EU GDPR?

subven

@girish there is nothing to complain about selfhosted services in general, at least in terms of GDPR. According to TMG (Telemediengesetz) It's binding to have an imprint (at least in Germany) and an data protection agreement if you serve websites and stuff to the public. At some point (size or field of operation) you need to have an data protection officer for your company.

Regarding GoBD however, there is the problem that (if you host the software yourself) you usually have the possibility to manipulate data because you own the server or storage. GoBD is all about storing your business-related communication and financial data in a tamper-proof manner. It's the "principles for the proper management and storage of books, records and documents in electronic form as well as for data access." This is why the "best" solution seems to be SaaS. The law is even relevant if you're self employed but auditors proceed according to proportionality and traceability if you are "small". For example: Software such as Invoice Ninja is not GoBD compliant in Germany because you can edit/delete things afterwards. How do you do it right you may ask? It's complicated...

Modern DMS can counteract this issue because they store your data securely and document all changes you made. Sadly this sometimes conflicts with GDPR article 15, 16 and 17 (30 maybe too). A DMS must therefore also be GDPR compliant.

While being good in theory, these laws/rules are extremely difficult to implement and follow in practice.

Hope you don't regret that you've asked.

marcusquinn

@subven Sounds like something that could be solved by having DB backups to a Git repo, since the history is an immutable ledger.

It would be a nasty situation to ever get into but it would be an even nastier person or auditor that tried to manipulate past data.

With the principle of innocent until proven guilty, I'm certain it would be sufficient to provide "access" (not a copy) of the DB backups Git repo for forensic analysis if they really thought there was something amiss.

In my opinion, it's more effort to be 99% honest than 100% honest though, and generally the dishonest are also lazy, so having put in place Git backups to show immutable records, it would be as likely as meeting aliens to find someone that then tried to tamper with them as well.

Personally, I'm surprised the whole legal industry hasn't moved to Git for documenting anyway, since the law is a freeform codification of social contracts.

@necrevistonnezr I expect is our resident expert here?

necrevistonnezr

@subven

Indeed, GoBD and GDPR seem to contradict each other:

The GDPR requires a purpose for the storage of personal data and its deletetion if such purpose does not exist (anymore).
The GoBD deals with the retention of documents in order to comply with tax obligations.
Important: The GoBD does not stipulate if certain documents should be retained, only how. The "if" - the obligation to keep accounts and records - results from a variety of commercial and tax law regulations.

However:

The GoBD does not set any time limits for the retention of data but merely states: If there are obligations to retain data, such data must be retained in a certain way.
The GDPR does not contain any concrete time limits for the retention or subseqent deletion, either. It rather stipulates general principles of storage limitation and data minimization: According to Art. 5 GDPR, data may only be stored for as long as it is necessary and appropriate for a previously defined, clear and legitimate purpose; such purpose can also consist of precisely those storage obligations that GoBD deals with.

In other words: If there's a legal obligation (e.g. pursuant to tax law) to keep records, the GoBD stipulates how to keep those records and, pursuant to Art. 5 GDPR, such legal obligation legitimizes the retention of data (i.e. the invididual may not request the deletion of data for the legal retention period).

Therefore, GoBD and GDPR are not really competing sets of rules. The applicable test is:

Is there a tax law obligation to retain documents?
Does the GDPR principles of of storage limitation and data minimization require that documents be deleted?
It's possible that not all data of a document is relevant for the purpose of storage. In such cases, one solution is to redact certain information in the document to comply with both GoBD and GDPR. In order to avoid a (unreasonable) individual case examination for each document, some DMS apply a deletion concept based on e.g. legal retention periods.

nebulon

Just wanted to give an update. I have started to prepare this for release. I added the tests and required meta information now and the app works well so far.

Mayan supports email sending setup as well as receiving documents via an inbox. I am currently trying to get both auto-setup. The current package still requires manual setup from the corresponding env variables made available from the addons.

marcusquinn

@nebulon

nebulon

I have hit some road blocks with the current packaging. While it basically works, I only now found out that the original image would put the actual code/virtualenv into /app/data which is wrong. To proceed here I tried to reach out to upstream devs at https://gitlab.com/mayan-edms/mayan-edms/-/issues/947 lets see how we can proceed with hopefully some input there.

tobs

Nothing new for a long time ...
Would be nice to have an EDMS like Mayan on cloudron!

girish

@tobs have you seen https://forum.cloudron.io/category/1/announcements for new apps ?

tobs

@girish ... yes, but I've seen nothing concerning Mayan

girish

@tobs ah, i misunderstood what you said. I have pinged @nebulon about this.

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Mayan EDMS