Invoice ninja server 500 internal error

[AartJansen]

@girish these are the logs I see when attempting to view or generate a pdf

Jun 20 08:09:54 172.18.0.1 - - [19/Jun/2024:20:09:54 +0000] "GET /storage/wy7367YjZKJ1MeibNz7hWcqVUuzfvlYV/RE1ZVEX72zOe7y3YneTYcG4Y9oae9HMlDUOZg95d.png HTTP/1.1" 200 9907 "-" "-"
Jun 20 08:09:55 172.18.0.1 - - [19/Jun/2024:20:09:54 +0000] "GET /client/invoice/ZZ0cyw1hzyF4YLY5afg89hOIWKLCrkM5/download?t=1718827794306 HTTP/1.1" 500 1781 "-" "Dart/3.3 (dart:io)"
Jun 20 08:10:00 172.18.0.1 - - [19/Jun/2024:20:10:00 +0000] "GET / HTTP/1.1" 302 2041 "-" "Mozilla (CloudronHealth)"
Jun 20 08:10:00 2024-06-19 20:10:00 Running [queue-size-job] ..................... 12ms DONE <30>1 2024-06-19T20:10:00Z cloudron 484b2dd6-7856-4a2f-a637-cac90510d702 1285 484b2dd6-7856-4a2f-a637-cac90510d702 - 2024-06-19 20:10:00 Running [task-scheduler-job] .................. 2ms DONE <30>1 2024-06-19T20:10:00Z cloudron 484b2dd6-7856-4a2f-a637-cac90510d702 1285 484b2dd6-7856-4a2f-a637-cac90510d702 - 2024-06-19 20:10:00 Running [Callback] ............................ 7ms DONE
Jun 20 08:10:00 2024-06-19T20:10:00.000Z
Jun 20 08:10:01 2024-06-19 20:10:01 App\Jobs\Ninja\QueueSize ....................... RUNNING
Jun 20 08:10:01 2024-06-19 20:10:01 App\Jobs\Ninja\QueueSize ................... 3.26ms DONE
Jun 20 08:10:01 2024-06-19 20:10:01 App\Jobs\Ninja\TaskScheduler ................... RUNNING
Jun 20 08:10:01 2024-06-19 20:10:01 App\Jobs\Ninja\TaskScheduler ............... 3.24ms DONE
Jun 20 08:10:10 172.18.0.1 - - [19/Jun/2024:20:10:10 +0000] "GET / HTTP/1.1" 302 2041 "-" "Mozilla (CloudronHealth)"
Jun 20 08:10:20 172.18.0.1 - - [19/Jun/2024:20:10:20 +0000] "GET / HTTP/1.1" 302 2041 "-" "Mozilla (CloudronHealth)"
Jun 20 08:10:30 172.18.0.1 - - [19/Jun/2024:20:10:30 +0000] "GET / HTTP/1.1" 302 2041 "-" "Mozilla (CloudronHealth)"
Jun 20 08:10:40 172.18.0.1 - - [19/Jun/2024:20:10:40 +0000] "GET / HTTP/1.1" 302 2041 "-" "Mozilla (CloudronHealth)"
Jun 20 08:10:50 172.18.0.1 - - [19/Jun/2024:20:10:50 +0000] "GET / HTTP/1.1" 302 2041 "-" "Mozilla (CloudronHealth)"
Jun 20 08:11:00 172.18.0.1 - - [19/Jun/2024:20:11:00 +0000] "GET / HTTP/1.1" 302 2041 "-" "Mozilla (CloudronHealth)"
Jun 20 08:11:00 INFO No scheduled commands are ready to run. <30>1 2024-06-19T20:11:00Z cloudron 484b2dd6-7856-4a2f-a637-cac90510d702 1285 484b2dd6-7856-4a2f-a637-cac90510d702 -
Jun 20 08:11:00 172.18.0.1 - - [19/Jun/2024:20:11:00 +0000] "GET /storage/wy7367YjZKJ1MeibNz7hWcqVUuzfvlYV/RE1ZVEX72zOe7y3YneTYcG4Y9oae9HMlDUOZg95d.png HTTP/1.1" 200 9907 "-" "-"
Jun 20 08:11:01 172.18.0.1 - - [19/Jun/2024:20:10:59 +0000] "GET /client/quote/ZiFMzl5VdAaixCswv6S62SyUYfzRBenm/download?t=1718827860001 HTTP/1.1" 500 1781 "-" "Dart/3.3 (dart:io)"
Jun 20 08:11:10 172.18.0.1 - - [19/Jun/2024:20:11:10 +0000] "GET / HTTP/1.1" 302 2041 "-" "Mozilla (CloudronHealth)"
Jun 20 08:11:20 172.18.0.1 - - [19/Jun/2024:20:11:20 +0000] "GET / HTTP/1.1" 302 2041 "-" "Mozilla (CloudronHealth)"

[AartJansen]

@humptydumpty How do I find the health check ?

[AartJansen]

OK, I added the
PDF_GENERATOR=hosted_ninja
line to the env file, restarted the app and its working again.
Thanks humptydumpty.

[humptydumpty]

@AartJansen No problem. The health check can be found in the bottom left corner by clicking on the i icon.

[girish]

I could reproduce the problem in my test instance. Debugging as to what has changed.

[luckow]

Hmpf. PDF_GENERATOR=hosted_ninja works, but it feels wrong to add a third-party provider to my invoicing app. An app where the creation of PDFs at the end of the month does not work is also a problem.

[girish]

This should be fixed now in the latest package.

[humptydumpty]

@girish Do I need to undo my temp fix that I did above after updating?

[luckow]

@humptydumpty I have removed PDF_GENERATOR=hosted_ninja and can confirm that the PDF creation works.
Thank you Cloudron team.

[humptydumpty]

@luckow Invoices/quotes emails are not being sent to the client when I remove PDF_GENERATOR=hosted_ninja

Create a fake client & invoice and send it to yourself. Is it working?

A tip I received from the upstream dev is to BCC yourself by going to:

IN dashboard > settings > Advanced Settings > Email Settings >BCC Email

[humptydumpty]

@girish email delivery doesn't work for me unless I keep PDF_GENERATOR=hosted_ninja in the env file. PDF generation works fine with or without that line now. Should I report this upstream as a new/separate issue?

[AartJansen]

oh crap neither does mine!
I've sent quite a few reminders today, and new invoices, I take it they are just failing silently and not queued anywhere?

[luckow]

You are right. There is something wrong with sending emails. I tried to send offers to my own e-mail address and checked https://my.example.org/#/emails-eventlog.
The emails are going out, but to a completely different email address.
Investigating (Invoice Ninja 5.10.2)

[luckow]

ok. a few hours later.

• Fresh install of invoice ninja seems to work.
• my updated instance sends email exclusively to the mail address from user 1.
• dashboard Recent Activity shows the problem

sending mails works
Stephan Luckow emailed invoice RG-240481 for ACME inc. to Jane Doe

sending mails does not work anymore
Stephan Luckow emailed quote 0023-0052 for ACME inc. to

to is empty. Maybe there is a default setting in the code, if no "to" is set, it will be sent to the administrator account (or user 1).

I have restored old versions of IN from backups.
Up to v1.13.5, sending mails works. In v1.13.6 I have a problem with logging into my account. Next I will test v1.13.7.

Side quest: In the latest version (in my updated instance) it is impossible to change the mail address of user 1 via the user interface.

[luckow]

In my case, the problem starts with the emails in v1.13.7. Maybe in v1.13.6, but I can't log in in that version.
Next: updating from v1.13.5

[luckow]

@luckow ok. reproducible builds / updates. v1.13.5 -> v1.13.6 (no log-in possible) -> v1.13.7 (mails do not work)
next: migrate broken app into a new installation.

[luckow]

The moment I restore the settings (broken updated instance -> fresh instance), the new instance shows the same behavior. Data is not a problem.

[luckow]

restoring from a working v1.13.5 export -> fresh v1.14.2 produces same behaviour. the moment i restore settings, bam.
comparing json files is not one of my daily tasks, so I decide to do that:

• install a fresh IN
• import data only
• manually click configuration options.

Fun fact: It is possible to set a password that starts with a space for the login. ONLY for login. After you have logged in and your action requires a password confirmation, the leading space will be removed, and your password is incorrect.

[nebulon]

@luckow maybe good to report this password trimming upstream. Great catch and unlikely it gets detected by the devs on their own.

[humptydumpty]

That leaves the email deliverability issue. How is PDF generation related to emails not getting sent?