Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse

Cloudron Forum

Apps | Demo | Docs | Install

Lots of dataviz apps, but what to use for data sources?

Scheduled Pinned Locked Moved Apps
10 Posts 4 Posters 121 Views
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • ei8fdbE Offline
    ei8fdbE Offline
    ei8fdb
    wrote on last edited by ei8fdb
    #1

    Hi all,

    I've been playing around with data visualisation options for some work. Right now it's all "somewhere on the internet" using different dataviz services.

    I know cloudron packages a lot of different dataviz tools, but I (possibly mistakenly) think they're DB driven - get your data into a DB then it can visualise it.

    Most of the datasources I'm looking at are file based, CSV, etc. I know this isn't so efficient but the data isn't real time - e.g. world population to date, average global income etc.

    (Also I've kept away from DBs this far because they scare the crap out of me. I had an exchange DB failure and loss when I was a sysadmin about 15 years ago and it was the worst experience of my life!)

    I'm looking for some advice on the "engineering" side of dataviz.

    What I'm looking for

    Recommendations on what packages available currently on Cloudron I use to achieve the following :

    I can put data sources (CSV,tsv, xls) somewhere hosted on Cloudron, then use one of the visualisation package to visualise that data.

    I may not be explaining my use case clearly enough, so will clarify as best I can.

    Thanks!

    L luckowL 2 Replies Last reply
    1
  • L Offline
    L Offline
    LoudLemur
    replied to ei8fdb on last edited by LoudLemur
    #2

    @ei8fdb Kaggle is good as the data there is used for Machine Learning:

    https://www.kaggle.com/datasets

    DataUSA:
    https://datausa.io/

    Politics:
    https://data.fivethirtyeight.com/

    If you can abide proprietary, Big Data, Google:
    https://www.google.com/publicdata/directory

    The census office provides updates periodically. The UK's came out a few months ago, so is quite current.

    ei8fdbE 1 Reply Last reply
    0
  • ei8fdbE Offline
    ei8fdbE Offline
    ei8fdb
    replied to LoudLemur on last edited by
    #3

    Hi @LoudLemur thanks for the data source links.

    What I'm looking for is Cloudron packages I can use.

    1 Reply Last reply
    0
  • luckowL Offline
    luckowL Offline
    luckow translator
    replied to ei8fdb on last edited by
    #4

    @ei8fdb To be fair, this is what Metabase supports by default:

    920aa736-c5dc-41f8-810e-f11a7bc024a8-image.png

    And this is superset

    0c5504a1-4476-4a51-889d-76def67750fe-image.png

    Without anything from the communities (like csv addons/plugins) you need to find a way to transform your data to a supported database. MySQL and Postgres are the preferred choices if you want to store your data on your local Cloudron instance.

    For a start (to try out the Cloudron apps) connect to a Cloudron app via the dashboard and look in the env file (if there is no env, use the terminal and 'env' on the console). Look for the credentials for the database addon and use that information in superset or metabase.

    Later, install a LAMP (which has the mysql addon enabled) and try to figure out how to import your data directly into the database. https://docs.cloudron.io/apps/lamp/

    I have never used it, but one of my favorite tools for cleaning messy data https://openrefine.org/ has a sql exporter https://openrefine.org/docs/manual/exporting#sql-statement-exporter
    Maybe that will help.

    Pronouns: he/him | Primary language: German

    ei8fdbE 1 Reply Last reply
    1
  • girishG Offline
    girishG Offline
    girish Staff
    wrote on last edited by
    #5

    SQLite works very well for all this. Our database is MySQL but we don't want to give the dataviz apps access to the database, if we can avoid it. So, we have a script that creates a sqlite database with the values we need from MySQL. And then use sqlite database in the apps.

    luckowL ei8fdbE 2 Replies Last reply
    1
  • luckowL Offline
    luckowL Offline
    luckow translator
    replied to girish on last edited by
    #6

    @girish From what I remember of my test cases, some of the fancy functions in these applications do not work with sqlite data sources.

    Pronouns: he/him | Primary language: German

    1 Reply Last reply
    2
  • ei8fdbE Offline
    ei8fdbE Offline
    ei8fdb
    replied to luckow on last edited by
    #7

    Thanks for the helpful answer @luckow.

    @luckow said in Lots of dataviz apps, but what to use for data sources?:

    you need to find a way to transform your data to a supported database

    Am I right in saying my assumption is correct - all the datavis packages available on Cloudron require databases? This isn't a criticism, I only want to make sure I've not missed something.

    If my assumption is correct, I guess the next question is:

    from the package/options available now on Cloudron - what is the easiest option to use to create a DB, of whatever type, MySQL, Postgres, SQLite, whatever?

    Any suggestions are welcome!

    And yes mentioning Openrefine - this is one of the tools I am using. I would like to have the data I'm visualising to be available online, and possibly interactive in some way. Hence my search for something available via Cloudron.

    1 Reply Last reply
    0
  • ei8fdbE Offline
    ei8fdbE Offline
    ei8fdb
    replied to girish on last edited by
    #8

    @girish said in Lots of dataviz apps, but what to use for data sources?:

    SQLite works very well for all this.

    Thanks @girish, and it is this - creating a database - step that I'm unfamiliar with.

    Can I use a LAMP stack app to create a DB server, as (I think) @luckow was suggesting? Sorry for the very basic question!

    girishG 1 Reply Last reply
    0
  • girishG Offline
    girishG Offline
    girish Staff
    replied to ei8fdb on last edited by
    #9

    @ei8fdb SQLite is just a file , no server needs to be run. Similar to CSV being a file and needs no server. Main difference is that you can run SQL queries with SQLite databases.

    1 Reply Last reply
    0
  • luckowL Offline
    luckowL Offline
    luckow translator
    wrote on last edited by
    #10

    ok. a quick try.

    • Download https://www.kaggle.com/datasets/justin2028/total-emissions-per-country-2000-2020
    • Install LAMP (important: give more RAM)
    • import CSV into openrefine
    • export sql from openrefine
      0e0699da-938b-4096-9fb4-3f4d9bbd02c7-image.png
    • import sql export into lamp via phpmyadmin
      fd1ff46c-5bb3-4cfb-a82e-b4124d356097-image.png
    • fire up the filemanager in lamp and look for credentials.txt
      b2b2ff22-f824-4c6f-866e-a76c6e2ce58a-image.png
    • install superset or metabase.
    • in my case adding the mysql credentials into superset
      5b95644f-5c44-4977-9dd7-ac7366b387e1-image.png
    • a click on connect brings
      1917edfd-dc57-40ee-b762-6ef1d7685cc1-image.png
    • et voilĂ 
      ec702395-b5b2-4166-9e9e-9d4fae5e7457-image.png

    Without being a pro in mysql, this worked for me

    7b177fcb-2227-4219-89ab-ae503e43d6f3-image.png

    Pronouns: he/him | Primary language: German

    1 Reply Last reply
    6

  • Login

  • Don't have an account? Register

  • Login or register to search.
  • First post
    Last post
0
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Login

  • Don't have an account? Register

  • Login or register to search.