Data Importer Tool

[JLX89]

Hello All!
Is there a way we can get the Data Importer Tool integrated into Firefly? It looks like it is a separate tool, so if this is the case -- I would assume it would be a separate app?

Thank you!

[girish]

@JLX89 We haven't looked into it but it seems like you can install this on the LAMP app - https://docs.firefly-iii.org/data-importer/installation/self-hosted/

[JLX89]

Yeah, I'm seeing that now -- will give it a shot and report back with directions.

[JLX89]

So I've got this running in a LAMP App, but there are some scenarios to keep in mind:

• Both services listed for the "automation" portion of imports (Spectre and Nordigen/ seem to be most other countries, except the United States

• There doesn't seem to be an authentication gateway to prevent "public" access and your Personal Access Token is embedded in the .env file.

@girish, @nebulon or others might be able to provide more information if we can place the Cloudron Auth gateway similar to what is used for the Firefly App.

NOTE: This was a quick run at getting this up and running, so ensure you review your configuration properly.

---

Instructions for Installing the Data Importer Application

1. Install a fresh LAMP App from the Cloudron Store.

   • I typically disable indexing via App > Security.

2. Open the file manager and open the PHP_VERSION file and change the value to PHP 8.2:

; Set the desired PHP version in this file
; Restart app for changes to take effect
PHP_VERSION=8.2

3. Save your file and restart the LAMP App.
4. Launch the terminal and complete the following commands:

cd /app/data/
composer create-project firefly-iii/data-importer

5. Stay in the Terminal and next you'll want to make sure the permissions are correct by running the following commands:

sudo chown -R www-data:www-data /app/data/data-importer
sudo chmod -R 775 /app/data/data-importer/storage

6. Next, you're going to need to update your Apache configuration to point from the default /app/data/public/ folder to /app/data/data-importer/public/. Open /app/data/apache/app.conf and update the following lines:

FROM
Find the lines that have the previous public route of /app/data/public and change them to /app/data/data-importer/public.

• For good measure, I would disable phpMyAdmin also in the apache.conf file:
  # Include "/app/code/apache/phpmyadmin.conf"

7. Restart the LAMP App

8. Launch the file manager and go to /app/data/data-importer and open the .env file and update the file as required.

   • Sample .env: Link to copy of sample env. You can also find the values required to be changed below.
   • Personal Access Token: You're going to need to generate a Personal Access Token from Firefly at: https://firefly.domain.tld/profile > OAUTH > Personal Token

FIREFLY_III_URL={https://firefly.domain.tld}
VANITY_URL={https://firefly.domain.tld}
FIREFLY_III_ACCESS_TOKEN={ACCESS-TOKEN}
AUTO_IMPORT_SECRET={GENERATED_SECRET}
TZ={America/New_York}
ENABLE_MAIL_REPORT=false
MAIL_DESTINATION={EMAIL-ADDRESS}
MAIL_FROM_ADDRESS=getenv('CLOUDRON_MAIL_FROM')
MAIL_HOST=getenv('CLOUDRON_MAIL_SMTP_SERVER')
MAIL_PORT=getenv('CLOUDRON_MAIL_SMTP_PORT') 
MAIL_USERNAME=getenv('CLOUDRON_MAIL_SMTP_USERNAME')
MAIL_PASSWORD=getenv('CLOUDRON_MAIL_SMTP_PASSWORD')
MAIL_ENCRYPTION=null
REDIS_HOST=getenv('CLOUDRON_REDIS_HOST')
REDIS_PASSWORD=etenv('CLOUDRON_REDIS_PASSWORD')
REDIS_PORT=getenv('CLOUDRON_REDIS_PORT')

9. Restart the LAMP App and you can access the interface at: https://firefly-importer.domain.tld

If anyone has any feedback or sees that I missed anything please let me know.

Thanks!

[girish]

@JLX89 said in Data Importer Tool:

There doesn't seem to be an authentication gateway to prevent "public" access and your Personal Access Token is embedded in the .env file.

How is this usually done upstream? Note that you have access to the htaccess file, so you can put an basic auth guard in place.

[JLX89]

@girish I didn't actually see much mentioned about how it's completed upstream. There is a public instance setup run by Firefly. I ended up going the .htaccess route and need to finish a few other configuration items to ensure the .htaccess hasn't interfered with anything.

[robi]

@JLX89 NIce, when you're done, please post the contents of the .htaccess config for future readers.

[JLX89]

@robi Sorry for the delay on this! I ended up going the .htaccess / .htpassd route to secure it for the time being.

It would be great if we could use the proxyAuth option and use that as a "drop in" for a LAMP stack.

[robi]

@JLX89 and the contents of .htaccess?

[JLX89]

I'm using pretty much the standard, the default .htaccess that comes with the data-importer tool, and the generated .htpasswd file I created. I've just added the below. Just update the # Whitelist API Requests line with API requests from the services you're using with the data-import tool.

# Password Protect Entire Directory
AuthName "Restricted Access" 
AuthType Basic 
AuthUserFile /path/to/.htpasswd 
require valid-user
# Whitelist API Requests
Allow from api.domain.com

[cylon]

@JLX89 said in Data Importer Tool:

9. Restart the LAMP App and you can access the interface at: https://firefly-importer.domain.tld

Many thanks for this, very helpful !
However after doing all this configuration, I cannot access to the data-importer at https://firefly-importer.{my-domain.tld} I have a DNS error.

I don't see in the .env where the URL is set ? Also, I guess I need to create manually add an entry to my DNS record for the firefly-importer sub domain ?

[JLX89]

@cylon Did you update the app "public route" in your Apache configuration from /app/data/public to /app/data/data-importer/public and restart the app? You'll also want to double check the permissions:

sudo chown -R www-data:www-data /app/data/data-importer
sudo chmod -R 775 /app/data/data-importer/storage

The DNS is handled by the LAMP app from the settings you chose when you deployed the app.

[cylon]

oh ok I thought I had to use https://firefly-importer.{my-domain.tld} but it actually is https://{my-lamp-subdomain}.{my-domain.tld} my bad

so it's actually working just fine thx

[JLX89]

@cylon Great to hear!

[dadams-AU]

hey - i tried to build this a couple of times. i keep getting a 403 apache error. any thoughts?

[robi]

@dadams-AU fix your apache config?

[JLX89]

@dadams-AU More than likely along the lines of what @robi said.

You'll need to retrace your steps and ensure you completed the configuration change in the public routes in /app/data/apache/app.conf and ensure that you set the permissions correctly using:

sudo chown -R www-data:www-data /app/data/data-importer
sudo chmod -R 775 /app/data/data-importer/storage

If you haven't restarted the LAMP app, you'll need to do that before all of the previous settings work correctly.

[jadsmrizk]

Hello all!
I am following this tutorial but apparently, recent versions of the data importer tool require PHP 8.3 which isn't supported by the LAMP app. Any workarounds?

Thank you

[girish]

@jadsmrizk The latest lamp app has 8.3 now

[jadsmrizk]

@girish thank you. Will give it a try