How to install Elasticsearch on Mastodon?

[shanelord01]

@girish said in How to install Elasticsearch on Mastodon?:

@username said in How to install Elasticsearch on Mastodon?:

Do I need an external server, or should I install an elasticsearch app on Cloudron, or can I install it directly from the Ubuntu via code.

You will need an external server because we don't support installing additional packages on the server since it might break updates. At some point we do plan to integrate ElasticSearch into Cloudron. I think some other apps like nextcloud require this as well. For the moment, just install it on a ubuntu or equivalent outside Cloudron.

I've just tried getting an external Linode running Elasticsearch running with my Cloudron Mastodon server. No luck. Got the server running, but no idea how to get the Cloudron Mastodon server to connect and create the "ElasticSearch indices and fill them with data".

Anyone got any guidance/steps on how to get this working?

Tried this but no luck: https://docs.joinmastodon.org/admin/optional/elasticsearch/

[alwynispat]

@shanelord01 I've got mine working. It's only advisable to have it running from with in the same server otherwise you'll have to expose it to the internet.
I'm hosting my ES in docker but its similar.

Here's what I've done:

1. Set your network.host and discovery.seed_hosts to 0.0.0.0, this is so that it'll be open to the internet
2. Set your Port to something other than 9200, 9920 for example.
3. Ensure that your firewall/port forwarding is setup.
4. Configure your env.production

ES_ENABLED=true
ES_HOST=[ES IP Address]
ES_PORT=9920

5. I recommend a reboot of the app
6. Run bin/tootctl search deploy in the app terminal

[ei8fdb]

@girish said in How to install Elasticsearch on Mastodon?:

planned to add an elasticsearch addon

As an add-on? Are you hinting at something new? Or did I miss something about Cloudron add-ons?

[nichu42]

The demand for ElasticSearch is about to increase rapidly very soon:
Full text search has just been merged in Mastodon main branch, and will be in the next version 4.2.0 (planned release date: 1st half of September).

https://github.com/mastodon/mastodon/pull/26344

[shanelord01]

@nichu42 Completely agree. Perhaps it’s time for a package including both (or at least instructions on hire to set it up).

Personally I’d prefer a one stop all in one install.

[nichu42]

Now that 4.2.0 has been released, what's the plan / progress with ElasticSearch, @girish?

[redegelde]

@nichu42 and when does 4,2 become available .
Great new updates

[nichu42]

@redegelde It already is available on Github. I suppose the Cloudron package will follow shortly. The Cloudron team is usually very quick with providing updates.

[scooke]

I don't blame the Staff for not implementing this (yet). The RAM requirements are simply huge. Even as an add-on, how many users are going to "add this on" and then watch their Cloudron go down in flames. Imagine the Forum then! And then we'll all be asking/demanding/pleading with the Staff to fix it, tweak it, optimize it, "it should be clearer that I need a 16GB ram machine for Cloudron" complaints that will probably even turn into "don't use CLourdon, it's a memory beast" posts here and there. If anything, it would be nice if Cloudron just made it simpler to connect to an ES install.

[nichu42]

@scooke Thanks for the insights. I use Cloudron because it's very easy to use, not to save money. So I'm open to whichever solution they work out. If there won't be a solution from Cloudron, I'd appreciate if the team would let us know. I'd start finding one myself in that case, though that's an adventure I try to avoid.

[girish]

@nichu42 you should already be able to connect an external ES instance. I think this is the best way if you badly need this now.

For integrating with cloudron, ES is a beast with its memory requirements as @scooke pointed out. It will take us a while to get to it and needs much thought. It's not in immediate roadmap.

[nichu42]

@girish Thanks for the clarification!

[girish]

@nichu42 I was just looking at it briefly. In case you go down this route... For some reason, they only support ES7 per https://github.com/mastodon/mastodon/issues/26544#issuecomment-1684005007 which is almost https://www.elastic.co/support/eol .

Also, Nextcloud ES integration now requires ES8 which could conflict with the previous ES7 requirement.

Finally, https://github.com/mastodon/mastodon/issues/18535 suggests OpenSearch 2.4.x might be supported.

Let me if you come up with something!

[redegelde]

great work.
For the time frame, when will Elasticsearch become avaible or not
i understand this is needed with the 4.2 version

[jdaviescoates]

@redegelde said in How to install Elasticsearch on Mastodon?:

For the time frame, when will Elasticsearch become avaible or not

As @girish said it's not yet on the roadmap, so I don't think it'll really be possible to guesitmate the time frame.

@redegelde said in How to install Elasticsearch on Mastodon?:

i understand this is needed with the 4.2 version

To be clear, it is not required by 4.2 (as in you can use 4.2 without it). But yes, it can be used with 4.2 and the Cloudron package for 4.2 is already available.

[jdaviescoates]

@girish said in How to install Elasticsearch on Mastodon?:

@nichu42 I was just looking at it briefly. In case you go down this route... For some reason, they only support ES7 per https://github.com/mastodon/mastodon/issues/26544#issuecomment-1684005007 which is almost https://www.elastic.co/support/eol .

Also, Nextcloud ES integration now requires ES8 which could conflict with the previous ES7 requirement.

Finally, https://github.com/mastodon/mastodon/issues/18535 suggests OpenSearch 2.4.x might be supported.

I note that the Mastodon docs do now say:

Mastodon is tested with ElasticSearch version 7. It should support OpenSearch, as well as ElectisSearch versions 6 and 8, but those setups are not officially supported.

(also, I've just submitted a pull request to fix their tiny Elastis typo )

[msbt]

@jdaviescoates in that PR is room for improvement

[nichu42]

Got my ES up and running on a second VPS. In case you want to do the same, this guide was extremely helpful: https://blog.thms.uk/2023/09/mastodon-elasticsearch

[jdaviescoates]

@nichu42 nice one, thanks for sharing!

[shanelord01]

@nichu42 Brilliant stuff.

For me on Linode, all I did extra was to create a VLAN and connect eth1 on my Cloudron to it, as well as eth1 on my Elasticsearch server. Then used this private link to connect the two.

Edit : Actually damn while troublshooting why I was having an issue getting it to work, it seems Cloudron does not recognise multiple NIC/eth interfaces. So I can't have eth1 on a VLAN with a 10.0.0.x/24 address.

See Linode setup here: https://www.linode.com/docs/products/networking/vlans/get-started/

Cloudron via "ip link show" isn't showing the eth1 interface at all. @girish any way to fix this toi make it work?

Edit 2: Make sure to troubleshoot the network.host on the Elasticsearch server and port 9200 is open in iptables - mine was refusing connections for some reason. Working on VLAN now.