ERPNext - cost-effective ERP solution

[jadudm]

@andreasdueren I've packaged one or two things for myself on Cloudron, and I took a look at Frappe/ERPnext.

First, for the thread: Cloudron does not run Docker images "as-is." Or, if you prefer, simply because a project runs in Docker does not mean that it will immediately be runnable under Cloudron. A Cloudron app needs to be packaged up so that it will "play nicely" with the control architecture that Cloudron provides. Put simply, to get that friendly Cloudron experience, some work is needed when packaging an app.

In the case of ERPnext, it has a compose file that specifies many software services. Traefik is used for routing and load balancing (I assume); Nginx fronts the service; it seems like Frappe (the API backend) is written in Python (another service). There's worker processes of several flavors, a scheduling service, a Redis cache, MariaDB (which, for porting to Cloudron, we'd want to integrate with it's built-in DB add-on), the site creator service... and a large number of storage volumes.

Cloudron does not, to the best of my understanding, support running docker-compose files. As a result, to package this, we'd have to pull all of these services into a single container image. That would take some thinking, especially since Docker "likes" to have one process per container. Or, if there is another way/it is possible, I don't personally know how to package up a multi-container Cloudron application.

The Cloudron team may have something else to say, but I thought I'd drop a note in the thread that helps explain why this app is a more complex proposition than others (perhaps) when it comes to packaging. Yes, it is open, and yes, it installs easily on a VM when you do a docker-compose up. Unfortunately, that is not the same as packaging things up to run under the Cloudron framework.

[andreasdueren]

@jadudm Fair and not a problem. I'm just a little bit disappointed by the lack of communication. If it won't get packaged for a while then a short note would have been sufficient instead of letting people like me guessing whats planned next.

[girish]

@andreasdueren We are not working on packaging this. We try to leave a note when we start packaging an app. Currently, we are working on 7.1 - https://forum.cloudron.io/topic/5982/what-s-coming-in-cloudron-7-1 and there's also many existing apps that need to be updated.

[girish]

For future reference, I found https://github.com/frappe/frappe/wiki/The-Hitchhiker's-Guide-to-Installing-Frappe-on-Linux . Fairly complicated!

[andreasdueren]

@girish Thank you, that makes it easier forme to decide on my next steps with ERPNext. I'll check out that guide.

[Aizat]

@andreasdueren Hey, sorry for the late reply. I have followed this protocol closely and it works like a charm: Youtube Video

It's fairly straightforward if you apply the same method as the video above, it uses the GitHub method that was posted here.
ERPnext is a solid system, worth the hassle. Good luck
Kind regards.

[infogulch]

From what I've seen looking at the frappe/erpnext system design, it looks like a lot of the service complexity comes from their "bench" system tool that provides multitenancy / "environment duplication" capabilities via their backup and restore system. From my experience being able to branch environments is vital for these kinds of tools, but... that's basically cloudron's core competency. With a bit of work cloudron could probably match any missing features that "bench" has over it.

With bench factored out and its features provided by cloudron directly, I think you'd be left with a single python service that needs redis, mariadb/mysql, and postgresql, which cloudron provides as services already.

Notes:

• "frappe" is the company and also the name of the development/deployment framework for erpnext and other applications developed by the same team
• The frappe/frappe-worker docker image referred to in the frappe_docker/compose.yaml file is built from the source in frappe dir in the frappe/frappe repo
• The bench worker --queue short command (and similar) runs scheduler.py:start_worker()
• The queue workers are a Python RQ-based job scheduling/background task system. RQ uses redis queues

[nj]

Teaser

[nj]

Two major issues that I ran into while testing on cloudron.

• frappe framework requires the database name and database username to be the same (its hardcoded and all over the place). When testing locally it worked but in cloudron, db name is appended with “db” and username with “user”

• frappe framework needs root access to the database, eg. password of the user named “postgres” (its hardcoded, even if —dbroot-username exists, it’s for mariadb only)

That’s why I’m running postgres server on the Docker container itself, and not use any db addons. I’ll map the db storage to /app/data so it gets backed up regularly.

See you guys with good news next time. Most apps made in Python seem to write all over the filesystem so I’m testing it on readonly environment; once done ErpNext should be available soon. Thank you for your patience.

[girish]

@nj said in ERPNext - cost-effective ERP solution:

frappe framework requires the database name and database username to be the same (its hardcoded and all over the place). When testing locally it worked but in cloudron, db name is appended with “db” and username with “user”

That's a pretty rough requirement but we can fix the platform code I guess to have an option to generate such names.

frappe framework needs root access to the database, eg. password of the user named “postgres” (its hardcoded, even if —dbroot-username exists, it’s for mariadb only)

Keeps getting rougher There's no chance we will allow an app to have root db access.

Is the situation any better with mariadb instead of postgresql?

[infogulch]

frappe framework needs root access to the database

Surely this is not actually necessary to run the app, but is just part of their custom dynamic deployment system (ick). I hope it's possible to extricate the actual app from the framework...

[nj]

@girish for the initial release, I've given up on the idea of using Cloudron addons for now. Postgres is also getting over-complicated so I think I'd stick with MariaDB. Let's see how it goes. The build-run-test cycle is quite tedious when there are all kinds of new errors to resolve. ErpNext turned out to be a lot harder than I expected.

Current status: frappe installed, up and running. ErpNext won't install, and require all its dependencies (apps) to be installed. My target is to publish it by this week.

[nj]

@infogulch said in ERPNext - cost-effective ERP solution:

frappe framework needs root access to the database

Surely this is not actually necessary to run the app, but is just part of their custom dynamic deployment system (ick). I hope it's possible to extricate the actual app from the framework...

I thought so too. Unfortunately, it keeps asking for "postgres super user password".

[LoudLemur]

@nj Thanks for all your hard work. Few have the necessary skills.

[nj]

Help Needed. Please check issue on github.

I'm inches away from either successfully running ErpNext or quitting the idea of packaging it. Never had I ever stuck with this kind of stupid errors.

When everything goes smooth, one of the modules (Payment Module in particular) make the entire table crash in the middle of loading the modules. Fix one error, then another pops up, then another.

I no longer have time nor patience to package this after this week. Here's the progress.. github.com/njsubedi/cloudron-erpnext if anyone has time, skill and patience, please go ahead and continue packaging this piece of sofware.

If anyone knows people from Frappe, please tell them to stop putting spaces and uppercase letters in table names, and at least retry any database operation instead of leaving the entire database in broken state when something fails, then have the user restart the minutes long process from the beginning.

Hours spent: 100+

Please check the issue on Github

[marcusquinn]

@nj You might not want to hear this, but in 100 hours, you could probably reproduce anything you need from ERPNext in EspoCRM using the Entity Manager to build copies of any specific data structures and Reports for anything specific there.

That's basically a lot of what I've been doing lately, taking Espo from being a CRM to an ERP system.

[nj]

@marcusquinn I could do the entire EspoCRM in Directus if I had to.

I wanted to package it because it's one of the most wanted apps, but I think it takes someone else who knows python and databases more than me.

[marcusquinn]

@nj After 18 months working with Espo, I doubt it, and I'm a fan of Directus too.

I see value in having ERPNext, and Odoo, packaged and available too. Even if only so that people can compare all of these similar apps, and take inspiration from all of them to make the best solution with the one they choose to develop among them.

I think you're right though, this one needs cooperation from the original author if it's not versatile enough to handle errors in building in Cloudron's tighter security Docker environment.

[nj]

Trailer
You know what's disappointing? I started packaging ERPNext for the HR module. Unfortunately, the HR Module is not available after installation. Now I'm here thinking where the hell it go, and why I spent weeks packaging something that doesn't have the particular module I was looking for.

Update: Looks like version-14 of ERPNext does not have HR module. Only version-13 has it. Now packaging version 13.

[subven]

@nj said in ERPNext - cost-effective ERP solution:

Looks like version-14 of ERPNext does not have HR module.

They refactored the HR and Payroll module into a "new" app --> #31467. Maybe this could still be installed within the current V14 docker app --> ERPNext HRMS installation.

PS: Thank you for your hard work

[nj]

@subven oh, wow! How did I miss that!

[Aizat]

@nj Not sure if this is what you are looking for, but if you wish to put on a donation pot, I would be happy to contribute where I can, as a token of thanks for your hard work and time!
Thank you again for all the hard work.

[nj]

@Aizat I'm looking for someone who know MariaDB and Python3 for the project. I've successfully packaged ErpNext, but I had to patch the source code of "Payments" and "HRMS" modules that crash the database tables during installation. Instead of cash, I'm looking for few other people who would pull the repo, and see if they see the same kind of crashes.

[nj]

@girish @nebulon would you look into it?

[infogulch]

@nj I tried installing it to my cloudron and it built successfully, but I ran into an issue with the cloudron cli during installation saying that I need to specify a subdomain (when I clearly did specify a subdomain), then I ran out of time to mess with it until later this week.

[nj]

Okay, the errors have gone. I'd say I'm 99% done packaging. Will post the good news soon.

[nj]

Okay, here is the good news. Please try it out and let me know if it works.

If you're not ready to build the image yourself, you can simply install the image that I created. cloudron install --image njsubedi/cloudron-erpnext -l erp.<yourdomain.tld>.

ErpNext v14 is now available on Cloudron. See the README file for usage instructions.

Features

• ErpNext v14 - running on Frappe Framework v14
• HRMS Module - Includes HR and Payroll modules
• Plug n Play - Automatically configured to use the subdomain you install it on, no setup needed
• Independent - Does not depend on Cloudron Addons for redis, mysql or such

Gotchas

• Installs its own database servers, but data resides inside /app/data, so it gets backed-up, no not a big issue
• Hasn't been tested a lot, but it's working pretty fine
• Multi-tenancy isn't set up, even though that's pretty trivial to do

Good to know

• After installation, look into /app/data folder using cloudron's file manager for a file that ends with -credential.txt.
• Raise issues on Github if you run into problems, not here.

[LoudLemur]

@nj said in ERPNext - cost-effective ERP solution:

Okay, here is the good news. Please try it out and let me know if it works.

If you're not ready to build the image yourself, you can simply install the image that I created. cloudron install --image njsubedi/cloudron-erpnext -l erp.<yourdomain.tld>.

ErpNext v14 is now available on Cloudron. See the README file for usage instructions.

Features

• ErpNext v14 - running on Frappe Framework v14
• HRMS Module - Includes HR and Payroll modules
• Plug n Play - Automatically configured to use the subdomain you install it on, no setup needed
• Independent - Does not depend on Cloudron Addons for redis, mysql or such

Gotchas

• Installs its own database servers, but data resides inside /app/data, so it gets backed-up, no not a big issue
• Hasn't been tested a lot, but it's working pretty fine
• Multi-tenancy isn't set up, even though that's pretty trivial to do

Good to know

• After installation, look into /app/data folder using cloudron's file manager for a file that ends with -credential.txt.
• Raise issues on Github if you run into problems, not here.

You are a Free Software hero right there, @nj !

[marcusquinn]

@nj bonus points to you sir!

[subven]

Can somebody review the code?

@girish what else needs to be done so that it can be released as an unstable app in the store?

Some concerns:

• only dev branch for hrms and payment module and no release (yes I'm aware that only the dev branch exists - this has to be adressed in the future)
• does not use Cloudrons Redis
• Will some of the database mitigations break the insallation in the future?

I will wait until the unstable release at the Cloudron App Store and start testing then

[nj]

I’ve added Cloudron LDAP support. I’ll push the changes this evening.

I think I can do Cloudron redis too. Looks doable.

I think after the default site is created, I can dump the db and import that to cloudron MySQL, but still ErpNext devs recommend Mariadb support only, and it needs the database name to be the same as the db user name. Unless there’s support from Cloudron devs, it’s a blocker. But I have set db path to be inside /app/data, so backup is not an issue. Also tried migration and backup-restore, which worked fine.

Since I’m actively using it at my company I’ll keep supporting this app. Also Keycloak, Outline and Odoo.

[hakunamatata]

@nj Thanks for your hard work with this! I am looking forward to the day this becomes available in the app store us n00bs.

[LoudLemur]

@nj What is the latest on Odoo? Is it still inching its way to beta on Cloudron?

[mdreira]

@nj ERPNext 14 also have an LMS, right?

https://erpnext.com/open-source-education

[micmc]

@mdreira said in ERPNext - cost-effective ERP solution:

@nj ERPNext 14 also have an LMS, right?

https://erpnext.com/open-source-education

Seem like yes, indeed, as it is written on the quoted page. And it seems to be very powerful as well.

[mdreira]

@micmc @nj Yes.

After reviewing Odoo and Flectra, in their community and enterprise versions, I think that ERPNext is a very good option because it is 100% free software.

And it has a lot of features.

As @nebulon says, it is a cost-effective ERP solution.

[marcusquinn]

@mdreira The big gotchya with Odoo is that the most useful part of it, the Studio addon, is not in the community edition.

With ERPNext, their equivalent no-code custom data model builder is included.

[benneic]

This post is deleted!

[mdreira]

@girish @nebulon so, could erpnext be a reality in Cloudron in the short term?

(I know you are working in the new update)

[privsec]

Such a great solution and offering.

This would solve quite a few problems a client of mine is experiencing trying to solve.

Looking forward to this!

[mdreira]

Wow, this app wishlist post started at 2018…

[plusone-nick]

@mdreira Yeahhh...all good things take time, plus ERP next at the time was not what it is now lol but that's even more reason to get it onboarded now =]

[Aizat]

@plusone-nick I'm hoping the Cloudron team will support to include this formally in Cloudron soon. There might be some hindrances, but I think it should work relatively fine now, right?

[spacetime]

I add my request for ERPNext!

[infogulch]

I was able to successfully build and run the package nj published on github for the last couple of months.

It seems to work fine, though I have not had a chance to evaluate how the upgrade and backup/restore process goes (I still have concerns about how it would interact with cloudron long-term), and I've only just started setup for a few modules.

From my (limited) experience, if you want to use ERPNext, be prepared for an extended commissioning phase. The product is both wide and deep, you will be taking considerable time to learn the ERPNext system and configuring it to work with your business process. ERPNext is moderately opinionated on how things should work, an approach I generally appreciate, but I found integration to be a big task as it pushed me to shift my expectation on how things should be organized to the "ERPNext-way". To be fair, this could be as much my ignorance of ERP systems in general as ERPNext's particular obtuseness, but I suspect the truth is that any large integrated bookkeeping system will exhibit similar commissioning/integration difficulties by the very nature of its size and scope.

I don't mean to push anyone away from the product, just to lay out realistic expectations of how much effort it will take on your part to integrate a product of this scope into your business.

[nj]

I have published an update for ERPNext on Github - cloudron-erpnext-v0.20.0 This version is stable, but has breaking changes from the first release, so instead of updating you'll need to export and re-import the documents manually. This won't be necessary from future updates.

1. Built on the base image version 4.0 (Ubuntu 22.04), so the source code is much lesser and simpler.
2. Uses Cloudron's redis service, as requested by one of the members.
3. The webserver and supervisor run as a normal (non-root) user so it's a bit more secure
4. Existing apps (frappe, erpnext) etc can be updated easily from the terminal. There will be no need to update this app anymore from this point onward.
5. New apps (like hrms, frappedesk) can also be added or removed easily.
6. LDAP Settings can be configured if needed using the included script.
7. There's preliminary support for multi-site setup.

I request someone to test this package and give some feedback. Thank you.

If building the app sounds complex, you can simply use the existing package by running cloudron install --image njsubedi/cloudron-erpnext:20230104-200417-70210de70 -l erp.your-domain.tld

Note: the latest image can be found at https://hub.docker.com/r/njsubedi/cloudron-erpnext/tags . Replace 20230104-200417-70210de70 with the latest tag.

[jdaviescoates]

@nj great, thanks!

@staff are you going to take a look at this and get it into the App Store?

[nj]

@jdaviescoates I doubt that because of a few reasons.

• ERPNext cannot use the MySQL addon because it needs MariaDB and needs to have exact same name for database username and the database name (eg. db_name and db_username must be same),

• ERPNext cannot use the Postgrest addon because it needs access to the database root user's password, which seems impossible to grant. There might be a possibility if that particular user had permissions to create new databases and user, which isn't possible right now.

• A separate MariaDB service is running inside the container. the database files are stored at /app/data so they can be backed up. This might not be a good idea but many apps that use file-based database are already using this so this might not be an issue.

• The frappe-bench folder is huge (~1GB) and needs to be writable at runtime. Hence the backup size grows fast.

If there's no problem running an extra background process for mariadb, and storing the data in /app/data folder, I think the team can look into packaging it. Good thing is this app doesn't need regular update as you can run bench update command to automatically update the Frappe framework and apps like Erpnext.

[LoudLemur]

@nj You do a lot of hard work for Cloudron. Thank you very much.

[MooCloud_Matt]

@nj
What you suggested is actually really hard to backup.
And it's not the best to have too many process in the same container.
Docker container differentiate from other container tecnologie to be single service container, even if they support multi services is not the best, especially for db.
Due to resources management especially.

[rookiee]

@nj hey thanks for doing god works, i tested it and works really well.
but i running to some problem when adding another erpnext site using different domain, seeing this message when visiting the url

You are seeing this page because the DNS record of erp.redacted.com is set to this server's IP but Cloudron has no app configured for this domain.

i'm running this command succesfully in the terminal

gosu cloudron bench config dns_multitenant on 

gosu cloudron bench new-site erp.redacted.com

i'm sorry if this sound stupid, im new to cloudron

[timconsidine]

@nj thanks for comments.

Strikes me that someone needs to be brave and just say 'ERPNext will not be packaged for Cloudron', as much in fairness to those who are really keen on it so they know it won't happen and they can look to self-host on a different VPS.

I don't have a dog in the fight, just trying to improve clarity and reduce "app anxiety".

There is no mechanism to "close" an entry in AppWIshlist. Maybe there needs to be one?

Of course, if I am wrong, that's fine, the exploration can continue.

[subven]

@timconsidine you can of course install ERPNext at Cloudron with the package provided by @nj It just needs some extra care and is not listed at the app store. Cloudron was always very open to custom package development and will push its efforts even further in the future.

There is no need to close the listing at the App Wishlist since there is still high demand for ERPNext and maybe at some point the app store will open up to list community maintained packages.

[nj]

To be honest I started packaging the most voted apps and ERPNext happened to be one of them. Also it saves me from installing 99 other apps now.

The good things about this package are:

• I don’t need to update it anymore; I can simply update the entire system without reinstalling this package

• I’m still working to bring multi-domain to it. For now dns based multi-tenancy is showing some issues but I will fix it soon

• If everything goes south, I can create a backup and move it to another ErpNext installation without an issue

• Cloudron totally takes care of backup and restore and I have already tested it

The bad things are:

• There are two processes running (mariadb and supervisor) and cloudron will only restart the container if supervisor goes out; but maybe I’ll add a mechanism to restart mariadb myself

• the backup size will be >1GB but if I’m using incremental backup this isn’t a big issue

• this app is not in the app store; but I’m going to keep the package maintained, because managing an instance of Erpnext on another VPS is going to be harder than maintaining it on Cloudron because I don’t have to care about ssl, redis, backups, ldap, etc.

So, guys, I’ll leave it here. Will let you know when an update is out.

[jdaviescoates]

@subven said in ERPNext - cost-effective ERP solution:

maybe at some point the app store will open up to list community maintained packages.

Sounds like that is in the pipeline:

@girish said in What's coming in 7.4:

Community repo (appstore)

[MooCloud_Matt]

@nj said in ERPNext - cost-effective ERP solution:

Cloudron totally takes care of backup and restore and I have already tested it

Not really, cloudron takes a snapshot of /app/data, and your database I guess don't create a dump before that happened.
That's not really stable as a solution.

If you could like get a dump every x time, that could be a solution, even if not perfect

[nj]

@MooCloud_Matt you don’t need to take a database dump because the entire mariadb database lies inside the /app/data folder.

[MooCloud_Matt]

@nj
that is correct if you have ZFS or BTRFS, that can do a snapshot instantaneity.
But cloudron uses a simple compression or rsync, which means that from when it starts and when it stops it takes time and in that time, data could be changed and the relationship could come out corrupted.

postgress in their documentation, for example, suggests that if you want to use an fs snapshot you should stop the DB.

[LoudLemur]

@nj I do not know if it is permissable or even possible but how about trying to use the Cloudron Demo to setup uour ERPNext?
We could take a look at it there.

@girish ?

[nj]

@LoudLemur you are right in that case. I didn’t worry because my backup schedule is at 3:00AM when nobody really used the app. If you expect people to write to the database during backup that’s an issue for sure.

Fortunately, ErpNext has a solution for that. We can put the app in maintenance mode for a while, and as I have already setup this ErpNext package to allow database read operations during maintenance, only writes will fail, which means websites etc will still be working fine.

Unfortunately, Cloudron doesn’t have a scheduler or cron system that can trigger a script before starting to backup and another script after backup success / failure. If @girish added a cron format like @prebackup @postbackup and @onbackupfail or simply added those options to the scheduler manifest, it would be perfect.

Either we could put the app in maintenance mode right before backup and put the site back on afterwards . I could even run mariadb inside /run/mariadb instead of /app/data ; I would only create database backup at /app/data/mariadb-dump-datetime.sql right before backup and delete it when backup task completes.

Also I’d like to see/know how the database addon handles backups and restoration.