Mattermost import - boards broken

[djxx]

Update - in the browser console I can see an API call to join me as an admin to the board failing with a 404. Could it be that the Cloudron app has not allowed the /boards/ endpoints?

Request URL:
https://mattermost.domain.com/plugins/focalboard/api/v2/boards/<board_id>/join?allow_admin
Request Method:
POST
Status Code:
404 Not Found
{"error":"{boards} not found","errorCode":404}

[nebulon]

I don't really know what mattermost focalboards are, but given that is a 404 and the path container "plugins", is it possible that this plugin needs to be enabled first on this instance?

[djxx]

The plugin is indeed enabled. The page itself opens up at https://mattermost.domain.com/boards/, but it seems some of the API calls a bit deeper under that domain fail. How can I confirm this is not a Cloudron containerization issue before raising it to the mattermost team?

[girish]

@djxx the initial error seems similar to https://github.com/mattermost/focalboard/issues/4828 . It's trying to open some log file but can't because of the readonly filesystem. You can try this:

• Put the app in recovery mode . Repair -> Recovery . This also makes the filesystem temporarily readwrite
• Open web terminal and start app manually using /app/pkg/start.sh
• Now try importing

[djxx]

@girish Thanks for the tip. Unfortunately the behavior is the same. I still get that message loop and I still see the 404 error coming back from the API call.

[girish]

@djxx as in you still see the readonly file system error?

[djxx]

@girish - Yes. I still see the read only error, and the boards still do not function and return 404 errors from the API call.

[girish]

@djxx just to double check, can you try touch /app/code/focalboard_errors.log with the web terminal? Does it work or give the same EROFS error?

[djxx]

@girish Yes - same error:

root@systemid:/app/code# touch /app/code/focalboard_errors.log
touch: cannot touch '/app/code/focalboard_errors.log': Read-only file system

[girish]

@djxx this should not be the case. If you put the app in Recovery mode, the filesystem is full writable.

Which version of Cloudron are you on?

[djxx]

Just updated to 8.0.4 last night.

I touched the file and confirmed it exists. I got the same error so I also tried to chmod 777 it, and it still gave the error. Maybe it's not writing where we think it is? Perhaps it's hard-coded to some path outside of the packaged directory?

In any case, it looks like it first has an error not finding {boards}, and then complains about not being able to write that error out.

{"timestamp":"2024-08-23 18:35:40.374 Z","level":"error","msg":"{boards} not found","caller":"app/plugin_api.go:1011","plugin_id":"focalboard"}
{"timestamp":"2024-08-23 18:35:40.375 Z","level":"info","msg":"can't open new logfile: open focalboard_errors.log: permission denied\n","caller":"io/io.go:432","plugin_id":"focalboard","source":"plugin_stderr"}
{"timestamp":"2024-08-23 18:35:40.421 Z","level":"info","msg":"can't open new logfile: open focalboard_errors.log: permission denied\n","caller":"io/io.go:432","plugin_id":"focalboard","source":"plugin_stderr"}
{"timestamp":"2024-08-23 18:35:40.422 Z","level":"error","msg":"{boards} not found","caller":"app/plugin_api.go:1011","plugin_id":"focalboard"}

[djxx]

Not sure why this ticket didn't show up in searches before, but it looks like it's a known issue upstream: https://github.com/mattermost/focalboard/issues/5019

My source serve's database was MySQL. I imported using an mmctl export so I didn't expect to have to deal with database issues, but it seems there are still problems. The issue above is showing the exact same error messages as me (except can't open log file), so I think the log error is just a red herring.

[girish]

@djxx good find . I guess we have to wait for upstream to be fixed then.

[djxx]

Who's ready for a book?

TL;DR - If you're trying to import boards from another server, especially with a different version, you're gonna have a bad time.

I'm nearly done moving one of my servers from NethServer to Cloudron. The Mattermost version on NethServer is very old, and it was before the forced migration to Postgres. This is already a bad thing, because there are known issues with doing a database restore from a MySQL instance into a Postgres instance.

I backed up my old Mattermost using the mmctl command, and after stepping on a few rakes I was finally able to get a file exported and imported. Tip: if you have any custom emoticons over their 512kb limit, your import will fail with obscure size errors. It's easiest to fix this in your original system and re-export, but you can also replace the emoticon file with a smaller version with the same name. Oh, and you'll most likely need to increase the RAM given to Mattermost in Cloudron if you're doing an import.

After looking around for a bit at my imported instance, I noticed that my boards were gone. I'm not sure if they just weren't in the export file, if they were excluded because the plugin wasn't installed on the fresh instance before doing the import, or because the database change made it impossible to import them. In any case, I ended up with a Mattermost instance with no boards - even after re-installing the plugin.

A quick read through their documentation showed that I should be able to export a .boardarchive file from each board and import it into my new server. I resurrected the old server long enough to export these files and tried to import them, that's when I started having problems that prompted me to write this post.

There were two primary problems:

#1 - {boards} not found

This error message is very unhelpful and doesn't give a good indication of what's happening. It turns out, this was caused by app / cookie data stored in my browser. There are preferences stored called lastViewId and lastBoardId which focalboard always uses to query when loading. Many requests to focalboard would succeed, but then this one would fail: /boards/<board_id>/blocks?all=true

Since the domain name of my restored instance was the same, my browser was using these preferences to fetch the board ID from my old instance, which didn't exist in the new instance.

The solution here is to clear your site data and then refresh. It will detect that you don't have a preference and then properly fetch and load the boards.

You may also need to do this if you're quickly creating and deleting boards while troubleshooting the import. You create a board, it stores it as your preference, you delete the board, BOOM, your UI is broken because it starts trying to fetch data for the board you just deleted.

#2 - 429 "Something went wrong"

As I was troubleshooting the above, and quickly flipping between boards, I noticed that the entire app would become broken, with lots of these 429 errors in the console. It looks like the default rate limit (either set by Mattermost or Cloudron) is WAY too low at 10 per second, and a burst of 100. Each page load on a board is at least 10-15 queries to get it fully loaded. So if you have a couple tabs open, or the app and a tab, you'll immediately hit the rate limit. Thankfully, Cloudron makes this very easy to fix: just use the File Manager to edit config.json, update the RateLimitSettings section, then restart the app.

Now can I load my boards?

You wish.
When I went to import the .boardarchive files I noticed that it was setting all users to me, and all timestamps to the time of import. After inspecting the file I can see that it does have all this data, but it's not making it through to the import. One obvious thing is that the user IDs are not the same. Obviously user IDs aren't the same because I'm migrating to another server.

I came up with a script (shown below) to update the .boardarchive files to do a find/replace of user IDs from old to new. This works, but only for assignee. I tried many things to get the created by and comment by to map properly, but nothing worked. Eventually I looked in the source code and it's obvious that it's not even trying. Their docs say

After importing a Focalboard backup from one Mattermost instance into another (such as during a migration from Mattermost Cloud to self-hosted), card timestamps will be updated based on the import date, and cards won't correctly identify users whose user IDs differ across Mattermost instances.

This is putting it gently. It doesn't, even, try:

...
block.ModifiedBy = userID
block.UpdateAt = now
...

It makes zero attempt to find the user, or restore the timestamps.

So the simple truth is that if you're importing a .boardarchive, you're going to lose almost all user and timestamp data.

I can see multiple issues open in their bug tracker that are variations of items #1 and #2 shown above, with no response for a long time. Overall, I'm very very disappointed with focalboards and am not sure how much longer it can hang on as a product with so many open bugs and poorly implemented features, such as imports.

If I could do it all over again, I would try harder to make the SQL import work to retain the data. Unfortunately, I've already wasted a lot of time on this, and my boards aren't that important (thankfully).

Next time I need to move these focalboards, I think I'll just switch to a different product. And next time I need to move chat system, I may look for something else that's self-hosted and has a good board integration.

Appendix - Fix your .boardarchive files to at least retain assignee

I wrote a script that will take two inputs:

• A file called ids.csv, with three columns: old_id, new_id, description
• .boardarchive files in the same directory.

So your directory structure would look like this:

• ids.csv
• some_board.boardarchive
• fix.sh

Running the fix.sh will do the following for each .boardarchive file in the current directory:

• Unzip the file to a temporary directory
• Find the board.jsonl file that contains the bord data
• Do a find/replace of every old ID to every new ID in the file
• Save the file
• Re-zip the archive
• Clean up the temporary directory

So as long as you export out the IDs from your old and new instance, put them in the CSV, and then run this script you can import a .boardarchive and retain the assigned users.

fix.sh

#!/bin/bash

# Loop through all files in the current directory with the extension "boardarchive"
for file in "$PWD"/*boardarchive; do
  echo "Processing $file"
  # Unzip the file and extract it to a temporary directory
  temp_dir=$(mktemp -d)
  unzip "$file" -d "$temp_dir"

  # Find the board.jsonl file in the unzipped archive
  jsonl_file=$(find "$temp_dir" -type f -name board.jsonl)
  if [ ! -f "$jsonl_file" ]; then
    echo "Error: Could not find board.jsonl file in $file"
    continue
  fi

  # Read ids from ids.csv and perform replacements on board.jsonl
  while IFS=, read -r old_id new_id description; do
    echo "Replacing $old_id with $new_id for $description"
    sed -i "s/$old_id/$new_id/g" "$jsonl_file"
  done < ids.csv

  # Update the archive with the updated board.jsonl file
  pushd $temp_dir
  zip -qr "$file" ./
  popd
  echo "Updated $file"

  # Clean up temporary directory
  rm -rf "$temp_dir"
done
``

[girish]

@djxx oh wow, thanks for the detailed write up! Hopefully it's useful for other people migrating as well. I will add a link to your report from our docs.