Bluesky Personal Data Server

[jdaviescoates]

@jdaviescoates said in Bluesky Personal Data Server:

For starters self hosting a Relay needs at least 4.5TB space.

As per https://whtwnd.com/bnewbold.net/3lbvbtqrg5t2t this has already increased to 16TB!

So, yeah, hosting a Bluesky Relay not really a viable thing to individually self host.

But I guess the PDS more viable.

BTW this thread and the linked blog exchanges well worth a read

https://social.coop/@cwebber/113647109852249805

In short: Bluesky / ATproto is not (and will not be) really very decentralized. Still better than X though.

[scooke]

@jdaviescoates said in Bluesky Personal Data Server:

Still better than X though

Anything is better than X, on principle!

[rmdes]

Most people would be more interested to run their own PDS than a Relay I think, it's probably going to be bit by bit, i'm not expecting Bluesky to be born in a desired state of perfection because nothing on the internet is ever born that way but I do hope that over time more and more parts of Bluesky will able to be run by users if they choose so

[nichu42]

What advantages does it have to self-host a PDS?

[jdaviescoates]

@rmdes said in Bluesky Personal Data Server:

I do hope that over time more and more parts of Bluesky will able to be run by users if they choose so

If you read that thread and linked posts it's pretty clear that it'll never be very decentralised.

[bmann]

@shanelord01 thanks for bringing this up Shane. I would like to see it packaged, it will probably need to start as a custom app.

I installed one myself taking over a full KVM-1 VPS at Hostinger ($4/month).

I'm an AT Protocol expert and maintain a tech talks Ghost blog at https://atprotocol.dev (hosted on my Cloudron server!).

There are some misunderstandings in this thread about architecture and frankly I'm not going to argue about decentralization. Running a PDS means user ownership of your data, which seems very on mission for Cloudron.

It also handles auth, acting as your OAuth server wherever you log into with your account. There are multiple non-Bluesky apps with different data types (called Lexicons). A few of my favourites:

• Smoke Signal - events & RSVP https://smokesignal.events
• White Wind - markdown blogs https://whtwnd.com
• Frontpage - link ranking & discussion https://frontpage.fyi
• Recipe Exchange - recipe storage https://recipe.exchange

When you log into other AT Prototocol powered apps, the data is written to your own PDS. Here's my (Bluesky hosted) account with all the data types: https://atproto-browser.vercel.app/at/bmann.ca

It is also extremely low resource usage. Runs on NodeJS and SQL lite and some web socket connections.

I haven't gotten far in learning Cloudron packaging myself yet, but I'll update this thread if I get something guying.

[sfeldkamp]

I'm taking a stab at packaging the Bluesky PDS. If anyone else is currently in progress on this or knows of a reason why it won't be possible, now is a good time to speak up!

[scooke]

@sfeldkamp Good luck, best wishes, it'd be a nice holiday gift for us all!

[ekevu123]

@sfeldkamp It works, I could set the app up. But I didn't migrate my data, because apparently, you can't go back to cloud later, and I wasn't sure if I wanted to maintain this at the moment.

I can share a tutorial, but you came first, and I don't want to spoil the fun for you, so feel free to try first

[sfeldkamp]

@ekevu123

Please share! I'll take whatever help you can offer. Or post a submission if you have it done already and I can test it out.

According to the docs Cloudron will take over maintenance of the app image after it's published.

And Bluesky recently implemented incoming account migration so now a user can go back to Bluesky hosted PDS.
https://docs.bsky.app/blog/incoming-migration

[timconsidine]

@sfeldkamp said in Bluesky Personal Data Server:

According to the docs Cloudron will take over maintenance of the app image after it's published.

They will …. But better to be prepared to maintain until they adopt it. Sometimes it’s quick, sometimes not.

[ekevu123]

Okay, find below information about how the app runs. What I haven't tested:

• Setting up notification e-mails (should be simple using SMTP)
• Actually transferring the data (a test account could be set up)
• Handling the integrated update mechanism: so far, all updates would need to be done manually.

The Core Concept

Standard deployment (Bluesky PDS original):

• Docker Compose file with 3 separate containers:
  • pds - The main application
  • caddy - Reverse proxy and TLS termination
  • watchtower - Automatic container updates

Cloudron deployment:

• Only 1 container - The main application (pds)
• Cloudron provides everything else:
  • Reverse proxy (TLS, HTTPS)
  • Health monitoring
  • Storage management
  • Update mechanism

Remove from the application:

• Caddy service - Cloudron's reverse proxy handles HTTPS/TLS
• Watchtower service - Cloudron's update system handles updates
• Docker Compose file - Cloudron doesn't use compose; it builds from Dockerfile

Keep from the application:

• The main app (Node.js + PDS in this case)
• The Dockerfile (but modify it to use a startup script)
• All application code and logic

Files You Need to Create

Three new files are required:

1. startup.sh - Environment validation and initialization

2. CloudronManifest.json - Cloudron deployment configuration

3. Modifications to Dockerfile - Add startup script and health check

#!/bin/bash
set -e

# Startup script for Bluesky PDS on Cloudron
# This script validates required environment variables and initializes the application

# Required environment variables
REQUIRED_VARS=(
  "PDS_HOSTNAME"
  "PDS_JWT_SECRET"
  "PDS_ADMIN_PASSWORD"
  "PDS_PLC_ROTATION_KEY_K256_PRIVATE_KEY_HEX"
)

# Check required variables
echo "Validating environment variables..."
for var in "${REQUIRED_VARS[@]}"; do
  if [[ -z "${!var:-}" ]]; then
    echo "ERROR: Required environment variable $var is not set"
    exit 1
  fi
done

# Set default data directory if not specified
PDS_DATA_DIRECTORY="${PDS_DATA_DIRECTORY:-/app/data}"
PDS_BLOBSTORE_DISK_LOCATION="${PDS_BLOBSTORE_DISK_LOCATION:-$PDS_DATA_DIRECTORY/blocks}"
PDS_BLOB_UPLOAD_LIMIT="${PDS_BLOB_UPLOAD_LIMIT:-104857600}"

# Set default service URLs (point to public AT Protocol network)
PDS_DID_PLC_URL="${PDS_DID_PLC_URL:-https://plc.directory}"
PDS_BSKY_APP_VIEW_URL="${PDS_BSKY_APP_VIEW_URL:-https://api.bsky.app}"
PDS_BSKY_APP_VIEW_DID="${PDS_BSKY_APP_VIEW_DID:-did:web:api.bsky.app}"
PDS_REPORT_SERVICE_URL="${PDS_REPORT_SERVICE_URL:-https://mod.bsky.app}"
PDS_REPORT_SERVICE_DID="${PDS_REPORT_SERVICE_DID:-did:plc:ar7c4by46qjdydhdevvrndac}"
PDS_CRAWLERS="${PDS_CRAWLERS:-https://bsky.network}"

# Set defaults for optional variables
LOG_ENABLED="${LOG_ENABLED:-true}"
PDS_PORT="${PDS_PORT:-3000}"
NODE_ENV="${NODE_ENV:-production}"

# Create required directories
echo "Initializing data directories..."
mkdir -p "$PDS_DATA_DIRECTORY"
mkdir -p "$PDS_BLOBSTORE_DISK_LOCATION"

# Export all PDS variables for the application
export PDS_HOSTNAME
export PDS_JWT_SECRET
export PDS_ADMIN_PASSWORD
export PDS_PLC_ROTATION_KEY_K256_PRIVATE_KEY_HEX
export PDS_DATA_DIRECTORY
export PDS_BLOBSTORE_DISK_LOCATION
export PDS_BLOB_UPLOAD_LIMIT
export PDS_DID_PLC_URL
export PDS_BSKY_APP_VIEW_URL
export PDS_BSKY_APP_VIEW_DID
export PDS_REPORT_SERVICE_URL
export PDS_REPORT_SERVICE_DID
export PDS_CRAWLERS
export LOG_ENABLED
export PDS_PORT
export NODE_ENV

# Optional environment variables (only export if set)
if [[ -n "${PDS_EMAIL_SMTP_URL:-}" ]]; then
  export PDS_EMAIL_SMTP_URL
fi
if [[ -n "${PDS_EMAIL_FROM_ADDRESS:-}" ]]; then
  export PDS_EMAIL_FROM_ADDRESS
fi
if [[ -n "${PDS_PRIVACY_POLICY_URL:-}" ]]; then
  export PDS_PRIVACY_POLICY_URL
fi
if [[ -n "${LOG_DESTINATION:-}" ]]; then
  export LOG_DESTINATION
fi
if [[ -n "${LOG_LEVEL:-}" ]]; then
  export LOG_LEVEL
fi

echo "Starting Bluesky PDS on Cloudron"
echo "  Hostname: $PDS_HOSTNAME"
echo "  Data directory: $PDS_DATA_DIRECTORY"
echo "  Blob storage: $PDS_BLOBSTORE_DISK_LOCATION"
echo "  Port: $PDS_PORT"

# Start the application
exec node --enable-source-maps index.js

FROM node:20.19-alpine3.22 as build

RUN corepack enable

# Build goat binary
ENV CGO_ENABLED=0
ENV GODEBUG="netdns=go"
WORKDIR /tmp
RUN apk add --no-cache git go
RUN git clone https://github.com/bluesky-social/goat.git && cd goat && git checkout v0.1.2 && go build -o /tmp/goat-build .

# Move files into the image and install
WORKDIR /app
COPY ./service ./
RUN corepack prepare --activate
RUN pnpm install --production --frozen-lockfile > /dev/null

# Uses assets from build stage to reduce build size
FROM node:20.19-alpine3.22

RUN apk add --update dumb-init bash

# Avoid zombie processes, handle signal forwarding
ENTRYPOINT ["dumb-init", "--"]

WORKDIR /app
COPY --from=build /app /app
COPY --from=build /tmp/goat-build /usr/local/bin/goat
COPY startup.sh /app/startup.sh

RUN chmod +x /app/startup.sh

EXPOSE 3000
ENV PDS_PORT=3000
ENV NODE_ENV=production
# potential perf issues w/ io_uring on this version of node
ENV UV_USE_IO_URING=0

# Health check to verify PDS is running and responsive
HEALTHCHECK --interval=30s --timeout=10s --start-period=40s --retries=3 \
  CMD node -e "require('http').get('http://localhost:3000/xrpc/_health', (r) => {if (r.statusCode !== 200) throw new Error(r.statusCode)})" || exit 1

CMD ["/app/startup.sh"]

LABEL org.opencontainers.image.source=https://github.com/bluesky-social/pds
LABEL org.opencontainers.image.description="AT Protocol PDS"
LABEL org.opencontainers.image.licenses=MIT

{
  "id": "io.cloudron.bluesky-pds",
  "version": "1.0.0",
  "manifestVersion": 2,
  "title": "Bluesky PDS",
  "description": "A Personal Data Server for AT Protocol and Bluesky",
  "tagline": "Self-hosted Bluesky server",
  "author": "Bluesky Social",
  "website": "https://github.com/bluesky-social/pds",
  "documentationUrl": "https://github.com/bluesky-social/pds",
  "tags": ["chat", "sync"],

  "httpPort": 3000,
  "healthCheckPath": "/xrpc/_health",

  "addons": {
    "localstorage": {
      "volumePath": "/app/data"
    },
    "sendmail": {}
  }
}

Make sure to change all hardcoded references for the data directory /pds to /app/data!

Then make sure to set all the required env variables. The admin password is for the server app, not your bluesky account.

As I said, this gets the app to run and report positively to the healthcheck, now does someone want to test this?

P.S.: I have now created and deleted a user via curl, and verified that it persists a server restart.

[sfeldkamp]

This was helpful and validated the direction I was headed with it! I am working from a fork of bluesky-social/pds at https://github.com/sfeldkamp/cloudron-bluesky-pds.

Current status is that it's running! I can add invite codes through the app's web terminal with goat CLI. I can register a user account on the PDS and post, reply, like, and follow from it. These all are pushed to the Bluesky firehose and can be seen in the Bluesky App by other accounts.

Still to be debugged / tested:

• Newly created account on the PDS shows "Invalid Handle" warning in Bluesky App.
• Account migration with goat CLI to the PDS from a bsky.social PDS.
• Account migration with goat CLI from the PDS back to bsky.social PDS.
• General soak test to feel comfortable that all parts of the PDS are working correctly.

I have some time off later this week, so I think I can make progress with these things.

[sfeldkamp]

I have resolved the issue with Invalid Handle. The problem was not having a *.[app.domain.com] wildcard domain mapped for the app. There's unfortunately no way to do this with the CloudronManifest.json file during app installation so this will have to be a manual step for installers.

I've tested migrating an account from a Bluesky PDS and that seems good. Tested making and restoring backups in Cloudron. That seems good.

The only remaining work to be done, that I am aware of, is to implement the update mechanism. I'm uncertain whether I should (or even can) do that or if that is something the Cloudron staff will do as they prepare it for the App Store.

https://github.com/sfeldkamp/cloudron-bluesky-pds

If anyone has any guidance on what more I should do, or a better way to "submit" this to the staff let me know.

I will likely move my live account sometime soon to be self hosted on my custom app install, as I expect that process to take awhile. If anyone would like to try it out themselves or review the code, please feel free!

[robi]

@sfeldkamp see https://forum.cloudron.io/post/117766 for aliasDomains in CloudronManifest.json

[sfeldkamp]

@robi

That's the Cloudron CLI, not the CloudronManifest.json. I asked a follow up question though to clarify though, so thank you.

[sfeldkamp]

I added a checklist item to the CloudronManifest.json that makes it as clear as possible for the user to add the wildcard alias. I just couldn't find any way to do this automatically.

I've also moved my primary PDS on the ATProto network to be hosted on my Cloudron custom app. So far so good, no problems I've noticed!

[girish]

@sfeldkamp does the app have any UI?

[sfeldkamp]

@girish

/ - An informational landing page. You can see it here: https://at.sethfeldkamp.com/
/oauth - OAuth routes

Then data is available from
/xrpc - Http API
/.well-known/atproto-did - DID verification

That's all I can find for routes in the source code.

https://github.com/bluesky-social/atproto/tree/main/packages/pds

[girish]

@sfeldkamp yeah, that works for the Cloudron package.