How to install Elasticsearch on Mastodon?
-
I don't blame the Staff for not implementing this (yet). The RAM requirements are simply huge. Even as an add-on, how many users are going to "add this on" and then watch their Cloudron go down in flames. Imagine the Forum then! And then we'll all be asking/demanding/pleading with the Staff to fix it, tweak it, optimize it, "it should be clearer that I need a 16GB ram machine for Cloudron" complaints that will probably even turn into "don't use CLourdon, it's a memory beast" posts here and there. If anything, it would be nice if Cloudron just made it simpler to connect to an ES install.
-
@scooke Thanks for the insights. I use Cloudron because it's very easy to use, not to save money. So I'm open to whichever solution they work out. If there won't be a solution from Cloudron, I'd appreciate if the team would let us know. I'd start finding one myself in that case, though that's an adventure I try to avoid.
-
@nichu42 you should already be able to connect an external ES instance. I think this is the best way if you badly need this now.
For integrating with cloudron, ES is a beast with its memory requirements as @scooke pointed out. It will take us a while to get to it and needs much thought. It's not in immediate roadmap.
-
@nichu42 I was just looking at it briefly. In case you go down this route... For some reason, they only support ES7 per https://github.com/mastodon/mastodon/issues/26544#issuecomment-1684005007 which is almost https://www.elastic.co/support/eol .
Also, Nextcloud ES integration now requires ES8 which could conflict with the previous ES7 requirement.
Finally, https://github.com/mastodon/mastodon/issues/18535 suggests OpenSearch 2.4.x might be supported.
Let me if you come up with something!
-
@redegelde said in How to install Elasticsearch on Mastodon?:
For the time frame, when will Elasticsearch become avaible or not
As @girish said it's not yet on the roadmap, so I don't think it'll really be possible to guesitmate the time frame.
@redegelde said in How to install Elasticsearch on Mastodon?:
i understand this is needed with the 4.2 version
To be clear, it is not required by 4.2 (as in you can use 4.2 without it). But yes, it can be used with 4.2 and the Cloudron package for 4.2 is already available.
-
@girish said in How to install Elasticsearch on Mastodon?:
@nichu42 I was just looking at it briefly. In case you go down this route... For some reason, they only support ES7 per https://github.com/mastodon/mastodon/issues/26544#issuecomment-1684005007 which is almost https://www.elastic.co/support/eol .
Also, Nextcloud ES integration now requires ES8 which could conflict with the previous ES7 requirement.
Finally, https://github.com/mastodon/mastodon/issues/18535 suggests OpenSearch 2.4.x might be supported.
I note that the Mastodon docs do now say:
Mastodon is tested with ElasticSearch version 7. It should support OpenSearch, as well as ElectisSearch versions 6 and 8, but those setups are not officially supported.
(also, I've just submitted a pull request to fix their tiny Elastis typo )
-
@jdaviescoates in that PR is room for improvement
-
Got my ES up and running on a second VPS. In case you want to do the same, this guide was extremely helpful: https://blog.thms.uk/2023/09/mastodon-elasticsearch
-
@nichu42 Brilliant stuff.
For me on Linode, all I did extra was to create a VLAN and connect eth1 on my Cloudron to it, as well as eth1 on my Elasticsearch server. Then used this private link to connect the two.
Edit : Actually damn while troublshooting why I was having an issue getting it to work, it seems Cloudron does not recognise multiple NIC/eth interfaces. So I can't have eth1 on a VLAN with a 10.0.0.x/24 address.See Linode setup here: https://www.linode.com/docs/products/networking/vlans/get-started/
Cloudron via "ip link show" isn't showing the eth1 interface at all. @girish any way to fix this toi make it work?Edit 2: Make sure to troubleshoot the network.host on the Elasticsearch server and port 9200 is open in iptables - mine was refusing connections for some reason. Working on VLAN now.
-
@shanelord01 Cloudron is just built on top of Ubuntu. We don't do anything to change visibility of network interfaces . All this to say "Cloudron does not recognize multiple NIC" is actually "Linux does recognize multiple NIC" . The issue is either in your networking setup or VM setup or VPS setup . Just giving you some ideas to debug
-
@girish My VM host (when I SSH into it) is showing the eth1 interface correctly via "ip link show"
When I use the Cloudron Terminal under my Mastodon app, it does not show the eth1 interface, only loopback and eth0.
So it is not my host.
Outcome is, Cloudron can't see the VLAN network on Linode - at least nothing is showing in the UI, and I am unable to use ping in the Cloudron terminal for some reason.
I'll keep testing.
-
@shanelord01 ah, I understood what you are saying now! The Cloudron Web Terminal is a terminal into the container. The container's network interfaces are completely different from those on the host. So, what you are seeing ins expected.
If I understand correctly now: you put your ES and Cloudron on the same VLAN. You are able to ping the ES from Cloudron host machine but unable to do so from the web terminal. Correct?
-
I tried this out of curiosity.
I installed Cloudron and attached a VLAN. Then, from a web terminal of an app
root@3270ef46-f6cc-463a-bff2-33062223f536:/app/code# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 23: eth0@if24: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 02:42:ac:12:13:23 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 172.18.19.35/16 brd 172.18.255.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fd00:c107:d509::8/64 scope global nodad valid_lft forever preferred_lft forever inet6 fe80::42:acff:fe12:1323/64 scope link valid_lft forever preferred_lft forever
I ran
nc -l 3000
on another server which is also in same LAN. Then, I can connect from app terminal:root@3270ef46-f6cc-463a-bff2-33062223f536:/app/code# telnet 192.0.2.3 3000 Trying 192.0.2.3... Connected to 192.0.2.3. Escape character is '^]'.
-
Thanks - I've done some further testing. You are right - it does work.
Seems the Debian host I was using on Linode (using "Secure Your Server" image) was blocking port 9200 by default so had to adjust iptables to allow external access. Had a hard time tracking that down.
Added:
iptables -I INPUT -p tcp --dport 9200 --syn -j ACCEPT
then saved it. Worked immediately.