Prefect - the easiest way to automate your data

[rmdes]

#datascience

website : https://www.prefect.io/
github : https://github.com/PrefectHQ/prefect
docker : https://hub.docker.com/r/prefecthq/prefect
docs : https://docs.prefect.io/core/getting_started/installation.html

[Lonkle]

I'm interested. Have you used this app before?

[jdaviescoates]

@marcusquinn will like this

[Lonkle]

@jdaviescoates Him and I tend to have similar interests, but I am always more developer centric. So, I'm interested in this, but it needs an API for me to want to convert it.

[marcusquinn]

@jdaviescoates Yeah, kinda depends a lot on the apps it has prebuilt connectors too, that's where the real value is.

Integromat is still the leader in that space for me, n8n looks good for more focus on FOSS app connectors and then there's Datafire and StackStorm to look at.

Love automating data-flows though, well, hate repetitive work

[robi]

@marcusquinn said in Prefect - the easiest way to automate your data:

Love automating data-flows though, well, hate repetitive work

On that note, I think we need a repository of data flows as they're often non obvious. This would help with more integrations and at least inspiration.

[Lonkle]

@robi said in Prefect - the easiest way to automate your data:

On that note, I think we need a repository of data flows as they're often non obvious. This would help with more integrations and at least inspiration.

What exactly do you mean by this? I think I agree with you, but I need to understand more of what you mean?

[robi]

@Lonk recepies.

[Lonkle]

@robi said in Prefect - the easiest way to automate your data:

recepies

Of what? That's what I'm confused about? "data flows" of what kind of data?

[robi]

@Lonk yes, data flows, that's the topic of my comment.

ever make a flow chart? a list of rules?

it's a collection of those.

as for data, it doesn't matter, it's any kind.

[Lonkle]

@robi said in Prefect - the easiest way to automate your data:

I think we need a repository of data flows as they're often non obvious

That seemed like a specific kind of data sentence.

[malvim]

@Lonk Yeah, I'm also unclear as to what would a "repository of data flows" look like in our context. I love these types of apps, but I couldn't understand what @robi was talking about, as well. Sounds interesting, I just couldn't understand it.

@robi do you have any examples of what this would mean, or similar software/sites/whatever to what you mean?

[robi]

@malvim think IFTTT, Zapier, etc.. understand now?

When @marcusquinn builds a data flow automation, it's a template for others to use and modify.

[Lonkle]

@robi said in Prefect - the easiest way to automate your data:

@malvim think IFTTT, Zapier, etc.. understand now?

When @marcusquinn builds a data flow automation, it's a template for others to use and modify.

Ah, so you're talking about in this app then, I understand. So, you've used it? I didn't know what it meant by data flows until I more closely inspected the screenshots and you compared it to IFTTT, Zapier, etc.

[robi]

@Lonk No, I merely referenced Marcus' comment.
Doesn't mean I used the app. (I haven't.)

[marcusquinn]

Integromat is good for sharing workflows, so I guess something like that. Not looked at the app yet but always happy to share if we create something useful if it has a method for storage that can be easily shared.

Did some nifty accounting automations with Google Vision for extracting text from scanned documents and storing all in Xero for search-ability.

All these ETL apps tend to be as useful as their pre-built connectors, when you have to write them as well, it can be quicker to just use whatever app already has them, and with development we rarely get time to come back and refactor for the preferred app, despite the continual dream of having less apps that do more.