Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Search
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse
Brand Logo

Cloudron Forum

Apps | Demo | Docs | Install
  1. Cloudron Forum
  2. Discuss
  3. Lots of dataviz apps, but what to use for data sources?

Lots of dataviz apps, but what to use for data sources?

Scheduled Pinned Locked Moved Discuss
10 Posts 4 Posters 1.3k Views 3 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • ei8fdbE Offline
      ei8fdbE Offline
      ei8fdb
      wrote on last edited by ei8fdb
      #1

      Hi all,

      I've been playing around with data visualisation options for some work. Right now it's all "somewhere on the internet" using different dataviz services.

      I know cloudron packages a lot of different dataviz tools, but I (possibly mistakenly) think they're DB driven - get your data into a DB then it can visualise it.

      Most of the datasources I'm looking at are file based, CSV, etc. I know this isn't so efficient but the data isn't real time - e.g. world population to date, average global income etc.

      (Also I've kept away from DBs this far because they scare the crap out of me. I had an exchange DB failure and loss when I was a sysadmin about 15 years ago and it was the worst experience of my life!)

      I'm looking for some advice on the "engineering" side of dataviz.

      What I'm looking for

      Recommendations on what packages available currently on Cloudron I use to achieve the following :

      I can put data sources (CSV,tsv, xls) somewhere hosted on Cloudron, then use one of the visualisation package to visualise that data.

      I may not be explaining my use case clearly enough, so will clarify as best I can.

      Thanks!

      L luckowL 2 Replies Last reply
      1
      • ei8fdbE ei8fdb

        Hi all,

        I've been playing around with data visualisation options for some work. Right now it's all "somewhere on the internet" using different dataviz services.

        I know cloudron packages a lot of different dataviz tools, but I (possibly mistakenly) think they're DB driven - get your data into a DB then it can visualise it.

        Most of the datasources I'm looking at are file based, CSV, etc. I know this isn't so efficient but the data isn't real time - e.g. world population to date, average global income etc.

        (Also I've kept away from DBs this far because they scare the crap out of me. I had an exchange DB failure and loss when I was a sysadmin about 15 years ago and it was the worst experience of my life!)

        I'm looking for some advice on the "engineering" side of dataviz.

        What I'm looking for

        Recommendations on what packages available currently on Cloudron I use to achieve the following :

        I can put data sources (CSV,tsv, xls) somewhere hosted on Cloudron, then use one of the visualisation package to visualise that data.

        I may not be explaining my use case clearly enough, so will clarify as best I can.

        Thanks!

        L Offline
        L Offline
        LoudLemur
        wrote on last edited by LoudLemur
        #2

        @ei8fdb Kaggle is good as the data there is used for Machine Learning:

        https://www.kaggle.com/datasets

        DataUSA:
        https://datausa.io/

        Politics:
        https://data.fivethirtyeight.com/

        If you can abide proprietary, Big Data, Google:
        https://www.google.com/publicdata/directory

        The census office provides updates periodically. The UK's came out a few months ago, so is quite current.

        ei8fdbE 1 Reply Last reply
        0
        • L LoudLemur

          @ei8fdb Kaggle is good as the data there is used for Machine Learning:

          https://www.kaggle.com/datasets

          DataUSA:
          https://datausa.io/

          Politics:
          https://data.fivethirtyeight.com/

          If you can abide proprietary, Big Data, Google:
          https://www.google.com/publicdata/directory

          The census office provides updates periodically. The UK's came out a few months ago, so is quite current.

          ei8fdbE Offline
          ei8fdbE Offline
          ei8fdb
          wrote on last edited by
          #3

          Hi @LoudLemur thanks for the data source links.

          What I'm looking for is Cloudron packages I can use.

          1 Reply Last reply
          0
          • ei8fdbE ei8fdb

            Hi all,

            I've been playing around with data visualisation options for some work. Right now it's all "somewhere on the internet" using different dataviz services.

            I know cloudron packages a lot of different dataviz tools, but I (possibly mistakenly) think they're DB driven - get your data into a DB then it can visualise it.

            Most of the datasources I'm looking at are file based, CSV, etc. I know this isn't so efficient but the data isn't real time - e.g. world population to date, average global income etc.

            (Also I've kept away from DBs this far because they scare the crap out of me. I had an exchange DB failure and loss when I was a sysadmin about 15 years ago and it was the worst experience of my life!)

            I'm looking for some advice on the "engineering" side of dataviz.

            What I'm looking for

            Recommendations on what packages available currently on Cloudron I use to achieve the following :

            I can put data sources (CSV,tsv, xls) somewhere hosted on Cloudron, then use one of the visualisation package to visualise that data.

            I may not be explaining my use case clearly enough, so will clarify as best I can.

            Thanks!

            luckowL Online
            luckowL Online
            luckow
            translator
            wrote on last edited by
            #4

            @ei8fdb To be fair, this is what Metabase supports by default:

            920aa736-c5dc-41f8-810e-f11a7bc024a8-image.png

            And this is superset

            0c5504a1-4476-4a51-889d-76def67750fe-image.png

            Without anything from the communities (like csv addons/plugins) you need to find a way to transform your data to a supported database. MySQL and Postgres are the preferred choices if you want to store your data on your local Cloudron instance.

            For a start (to try out the Cloudron apps) connect to a Cloudron app via the dashboard and look in the env file (if there is no env, use the terminal and 'env' on the console). Look for the credentials for the database addon and use that information in superset or metabase.

            Later, install a LAMP (which has the mysql addon enabled) and try to figure out how to import your data directly into the database. https://docs.cloudron.io/apps/lamp/

            I have never used it, but one of my favorite tools for cleaning messy data https://openrefine.org/ has a sql exporter https://openrefine.org/docs/manual/exporting#sql-statement-exporter
            Maybe that will help.

            Pronouns: he/him | Primary language: German

            ei8fdbE 1 Reply Last reply
            1
            • girishG Offline
              girishG Offline
              girish
              Staff
              wrote on last edited by
              #5

              SQLite works very well for all this. Our database is MySQL but we don't want to give the dataviz apps access to the database, if we can avoid it. So, we have a script that creates a sqlite database with the values we need from MySQL. And then use sqlite database in the apps.

              luckowL ei8fdbE 2 Replies Last reply
              1
              • girishG girish

                SQLite works very well for all this. Our database is MySQL but we don't want to give the dataviz apps access to the database, if we can avoid it. So, we have a script that creates a sqlite database with the values we need from MySQL. And then use sqlite database in the apps.

                luckowL Online
                luckowL Online
                luckow
                translator
                wrote on last edited by
                #6

                @girish From what I remember of my test cases, some of the fancy functions in these applications do not work with sqlite data sources.

                Pronouns: he/him | Primary language: German

                1 Reply Last reply
                2
                • luckowL luckow

                  @ei8fdb To be fair, this is what Metabase supports by default:

                  920aa736-c5dc-41f8-810e-f11a7bc024a8-image.png

                  And this is superset

                  0c5504a1-4476-4a51-889d-76def67750fe-image.png

                  Without anything from the communities (like csv addons/plugins) you need to find a way to transform your data to a supported database. MySQL and Postgres are the preferred choices if you want to store your data on your local Cloudron instance.

                  For a start (to try out the Cloudron apps) connect to a Cloudron app via the dashboard and look in the env file (if there is no env, use the terminal and 'env' on the console). Look for the credentials for the database addon and use that information in superset or metabase.

                  Later, install a LAMP (which has the mysql addon enabled) and try to figure out how to import your data directly into the database. https://docs.cloudron.io/apps/lamp/

                  I have never used it, but one of my favorite tools for cleaning messy data https://openrefine.org/ has a sql exporter https://openrefine.org/docs/manual/exporting#sql-statement-exporter
                  Maybe that will help.

                  ei8fdbE Offline
                  ei8fdbE Offline
                  ei8fdb
                  wrote on last edited by
                  #7

                  Thanks for the helpful answer @luckow.

                  @luckow said in Lots of dataviz apps, but what to use for data sources?:

                  you need to find a way to transform your data to a supported database

                  Am I right in saying my assumption is correct - all the datavis packages available on Cloudron require databases? This isn't a criticism, I only want to make sure I've not missed something.

                  If my assumption is correct, I guess the next question is:

                  from the package/options available now on Cloudron - what is the easiest option to use to create a DB, of whatever type, MySQL, Postgres, SQLite, whatever?

                  Any suggestions are welcome!

                  And yes mentioning Openrefine - this is one of the tools I am using. I would like to have the data I'm visualising to be available online, and possibly interactive in some way. Hence my search for something available via Cloudron.

                  1 Reply Last reply
                  0
                  • girishG girish

                    SQLite works very well for all this. Our database is MySQL but we don't want to give the dataviz apps access to the database, if we can avoid it. So, we have a script that creates a sqlite database with the values we need from MySQL. And then use sqlite database in the apps.

                    ei8fdbE Offline
                    ei8fdbE Offline
                    ei8fdb
                    wrote on last edited by
                    #8

                    @girish said in Lots of dataviz apps, but what to use for data sources?:

                    SQLite works very well for all this.

                    Thanks @girish, and it is this - creating a database - step that I'm unfamiliar with.

                    Can I use a LAMP stack app to create a DB server, as (I think) @luckow was suggesting? Sorry for the very basic question!

                    girishG 1 Reply Last reply
                    0
                    • ei8fdbE ei8fdb

                      @girish said in Lots of dataviz apps, but what to use for data sources?:

                      SQLite works very well for all this.

                      Thanks @girish, and it is this - creating a database - step that I'm unfamiliar with.

                      Can I use a LAMP stack app to create a DB server, as (I think) @luckow was suggesting? Sorry for the very basic question!

                      girishG Offline
                      girishG Offline
                      girish
                      Staff
                      wrote on last edited by
                      #9

                      @ei8fdb SQLite is just a file , no server needs to be run. Similar to CSV being a file and needs no server. Main difference is that you can run SQL queries with SQLite databases.

                      1 Reply Last reply
                      0
                      • luckowL Online
                        luckowL Online
                        luckow
                        translator
                        wrote on last edited by
                        #10

                        ok. a quick try.

                        • Download https://www.kaggle.com/datasets/justin2028/total-emissions-per-country-2000-2020
                        • Install LAMP (important: give more RAM)
                        • import CSV into openrefine
                        • export sql from openrefine
                          0e0699da-938b-4096-9fb4-3f4d9bbd02c7-image.png
                        • import sql export into lamp via phpmyadmin
                          fd1ff46c-5bb3-4cfb-a82e-b4124d356097-image.png
                        • fire up the filemanager in lamp and look for credentials.txt
                          b2b2ff22-f824-4c6f-866e-a76c6e2ce58a-image.png
                        • install superset or metabase.
                        • in my case adding the mysql credentials into superset
                          5b95644f-5c44-4977-9dd7-ac7366b387e1-image.png
                        • a click on connect brings
                          1917edfd-dc57-40ee-b762-6ef1d7685cc1-image.png
                        • et voilĂ 
                          ec702395-b5b2-4166-9e9e-9d4fae5e7457-image.png

                        Without being a pro in mysql, this worked for me

                        7b177fcb-2227-4219-89ab-ae503e43d6f3-image.png

                        Pronouns: he/him | Primary language: German

                        1 Reply Last reply
                        6
                        • nebulonN nebulon moved this topic from Apps on
                        Reply
                        • Reply as topic
                        Log in to reply
                        • Oldest to Newest
                        • Newest to Oldest
                        • Most Votes


                          • Login

                          • Don't have an account? Register

                          • Login or register to search.
                          • First post
                            Last post
                          0
                          • Categories
                          • Recent
                          • Tags
                          • Popular
                          • Bookmarks
                          • Search