Bluesky Personal Data Server

[timconsidine]

@sfeldkamp said in Bluesky Personal Data Server:

According to the docs Cloudron will take over maintenance of the app image after it's published.

They will …. But better to be prepared to maintain until they adopt it. Sometimes it’s quick, sometimes not.

[ekevu123]

Okay, find below information about how the app runs. What I haven't tested:

• Setting up notification e-mails (should be simple using SMTP)
• Actually transferring the data (a test account could be set up)
• Handling the integrated update mechanism: so far, all updates would need to be done manually.

The Core Concept

Standard deployment (Bluesky PDS original):

• Docker Compose file with 3 separate containers:
  • pds - The main application
  • caddy - Reverse proxy and TLS termination
  • watchtower - Automatic container updates

Cloudron deployment:

• Only 1 container - The main application (pds)
• Cloudron provides everything else:
  • Reverse proxy (TLS, HTTPS)
  • Health monitoring
  • Storage management
  • Update mechanism

Remove from the application:

• Caddy service - Cloudron's reverse proxy handles HTTPS/TLS
• Watchtower service - Cloudron's update system handles updates
• Docker Compose file - Cloudron doesn't use compose; it builds from Dockerfile

Keep from the application:

• The main app (Node.js + PDS in this case)
• The Dockerfile (but modify it to use a startup script)
• All application code and logic

Files You Need to Create

Three new files are required:

1. startup.sh - Environment validation and initialization

2. CloudronManifest.json - Cloudron deployment configuration

3. Modifications to Dockerfile - Add startup script and health check

#!/bin/bash
set -e

# Startup script for Bluesky PDS on Cloudron
# This script validates required environment variables and initializes the application

# Required environment variables
REQUIRED_VARS=(
  "PDS_HOSTNAME"
  "PDS_JWT_SECRET"
  "PDS_ADMIN_PASSWORD"
  "PDS_PLC_ROTATION_KEY_K256_PRIVATE_KEY_HEX"
)

# Check required variables
echo "Validating environment variables..."
for var in "${REQUIRED_VARS[@]}"; do
  if [[ -z "${!var:-}" ]]; then
    echo "ERROR: Required environment variable $var is not set"
    exit 1
  fi
done

# Set default data directory if not specified
PDS_DATA_DIRECTORY="${PDS_DATA_DIRECTORY:-/app/data}"
PDS_BLOBSTORE_DISK_LOCATION="${PDS_BLOBSTORE_DISK_LOCATION:-$PDS_DATA_DIRECTORY/blocks}"
PDS_BLOB_UPLOAD_LIMIT="${PDS_BLOB_UPLOAD_LIMIT:-104857600}"

# Set default service URLs (point to public AT Protocol network)
PDS_DID_PLC_URL="${PDS_DID_PLC_URL:-https://plc.directory}"
PDS_BSKY_APP_VIEW_URL="${PDS_BSKY_APP_VIEW_URL:-https://api.bsky.app}"
PDS_BSKY_APP_VIEW_DID="${PDS_BSKY_APP_VIEW_DID:-did:web:api.bsky.app}"
PDS_REPORT_SERVICE_URL="${PDS_REPORT_SERVICE_URL:-https://mod.bsky.app}"
PDS_REPORT_SERVICE_DID="${PDS_REPORT_SERVICE_DID:-did:plc:ar7c4by46qjdydhdevvrndac}"
PDS_CRAWLERS="${PDS_CRAWLERS:-https://bsky.network}"

# Set defaults for optional variables
LOG_ENABLED="${LOG_ENABLED:-true}"
PDS_PORT="${PDS_PORT:-3000}"
NODE_ENV="${NODE_ENV:-production}"

# Create required directories
echo "Initializing data directories..."
mkdir -p "$PDS_DATA_DIRECTORY"
mkdir -p "$PDS_BLOBSTORE_DISK_LOCATION"

# Export all PDS variables for the application
export PDS_HOSTNAME
export PDS_JWT_SECRET
export PDS_ADMIN_PASSWORD
export PDS_PLC_ROTATION_KEY_K256_PRIVATE_KEY_HEX
export PDS_DATA_DIRECTORY
export PDS_BLOBSTORE_DISK_LOCATION
export PDS_BLOB_UPLOAD_LIMIT
export PDS_DID_PLC_URL
export PDS_BSKY_APP_VIEW_URL
export PDS_BSKY_APP_VIEW_DID
export PDS_REPORT_SERVICE_URL
export PDS_REPORT_SERVICE_DID
export PDS_CRAWLERS
export LOG_ENABLED
export PDS_PORT
export NODE_ENV

# Optional environment variables (only export if set)
if [[ -n "${PDS_EMAIL_SMTP_URL:-}" ]]; then
  export PDS_EMAIL_SMTP_URL
fi
if [[ -n "${PDS_EMAIL_FROM_ADDRESS:-}" ]]; then
  export PDS_EMAIL_FROM_ADDRESS
fi
if [[ -n "${PDS_PRIVACY_POLICY_URL:-}" ]]; then
  export PDS_PRIVACY_POLICY_URL
fi
if [[ -n "${LOG_DESTINATION:-}" ]]; then
  export LOG_DESTINATION
fi
if [[ -n "${LOG_LEVEL:-}" ]]; then
  export LOG_LEVEL
fi

echo "Starting Bluesky PDS on Cloudron"
echo "  Hostname: $PDS_HOSTNAME"
echo "  Data directory: $PDS_DATA_DIRECTORY"
echo "  Blob storage: $PDS_BLOBSTORE_DISK_LOCATION"
echo "  Port: $PDS_PORT"

# Start the application
exec node --enable-source-maps index.js

FROM node:20.19-alpine3.22 as build

RUN corepack enable

# Build goat binary
ENV CGO_ENABLED=0
ENV GODEBUG="netdns=go"
WORKDIR /tmp
RUN apk add --no-cache git go
RUN git clone https://github.com/bluesky-social/goat.git && cd goat && git checkout v0.1.2 && go build -o /tmp/goat-build .

# Move files into the image and install
WORKDIR /app
COPY ./service ./
RUN corepack prepare --activate
RUN pnpm install --production --frozen-lockfile > /dev/null

# Uses assets from build stage to reduce build size
FROM node:20.19-alpine3.22

RUN apk add --update dumb-init bash

# Avoid zombie processes, handle signal forwarding
ENTRYPOINT ["dumb-init", "--"]

WORKDIR /app
COPY --from=build /app /app
COPY --from=build /tmp/goat-build /usr/local/bin/goat
COPY startup.sh /app/startup.sh

RUN chmod +x /app/startup.sh

EXPOSE 3000
ENV PDS_PORT=3000
ENV NODE_ENV=production
# potential perf issues w/ io_uring on this version of node
ENV UV_USE_IO_URING=0

# Health check to verify PDS is running and responsive
HEALTHCHECK --interval=30s --timeout=10s --start-period=40s --retries=3 \
  CMD node -e "require('http').get('http://localhost:3000/xrpc/_health', (r) => {if (r.statusCode !== 200) throw new Error(r.statusCode)})" || exit 1

CMD ["/app/startup.sh"]

LABEL org.opencontainers.image.source=https://github.com/bluesky-social/pds
LABEL org.opencontainers.image.description="AT Protocol PDS"
LABEL org.opencontainers.image.licenses=MIT

{
  "id": "io.cloudron.bluesky-pds",
  "version": "1.0.0",
  "manifestVersion": 2,
  "title": "Bluesky PDS",
  "description": "A Personal Data Server for AT Protocol and Bluesky",
  "tagline": "Self-hosted Bluesky server",
  "author": "Bluesky Social",
  "website": "https://github.com/bluesky-social/pds",
  "documentationUrl": "https://github.com/bluesky-social/pds",
  "tags": ["chat", "sync"],

  "httpPort": 3000,
  "healthCheckPath": "/xrpc/_health",

  "addons": {
    "localstorage": {
      "volumePath": "/app/data"
    },
    "sendmail": {}
  }
}

Make sure to change all hardcoded references for the data directory /pds to /app/data!

Then make sure to set all the required env variables. The admin password is for the server app, not your bluesky account.

As I said, this gets the app to run and report positively to the healthcheck, now does someone want to test this?

P.S.: I have now created and deleted a user via curl, and verified that it persists a server restart.

[sfeldkamp]

This was helpful and validated the direction I was headed with it! I am working from a fork of bluesky-social/pds at https://github.com/sfeldkamp/cloudron-bluesky-pds.

Current status is that it's running! I can add invite codes through the app's web terminal with goat CLI. I can register a user account on the PDS and post, reply, like, and follow from it. These all are pushed to the Bluesky firehose and can be seen in the Bluesky App by other accounts.

Still to be debugged / tested:

• Newly created account on the PDS shows "Invalid Handle" warning in Bluesky App.
• Account migration with goat CLI to the PDS from a bsky.social PDS.
• Account migration with goat CLI from the PDS back to bsky.social PDS.
• General soak test to feel comfortable that all parts of the PDS are working correctly.

I have some time off later this week, so I think I can make progress with these things.

[sfeldkamp]

I have resolved the issue with Invalid Handle. The problem was not having a *.[app.domain.com] wildcard domain mapped for the app. There's unfortunately no way to do this with the CloudronManifest.json file during app installation so this will have to be a manual step for installers.

I've tested migrating an account from a Bluesky PDS and that seems good. Tested making and restoring backups in Cloudron. That seems good.

The only remaining work to be done, that I am aware of, is to implement the update mechanism. I'm uncertain whether I should (or even can) do that or if that is something the Cloudron staff will do as they prepare it for the App Store.

https://github.com/sfeldkamp/cloudron-bluesky-pds

If anyone has any guidance on what more I should do, or a better way to "submit" this to the staff let me know.

I will likely move my live account sometime soon to be self hosted on my custom app install, as I expect that process to take awhile. If anyone would like to try it out themselves or review the code, please feel free!

[robi]

@sfeldkamp see https://forum.cloudron.io/post/117766 for aliasDomains in CloudronManifest.json

[sfeldkamp]

@robi

That's the Cloudron CLI, not the CloudronManifest.json. I asked a follow up question though to clarify though, so thank you.

[sfeldkamp]

I added a checklist item to the CloudronManifest.json that makes it as clear as possible for the user to add the wildcard alias. I just couldn't find any way to do this automatically.

I've also moved my primary PDS on the ATProto network to be hosted on my Cloudron custom app. So far so good, no problems I've noticed!

[girish]

@sfeldkamp does the app have any UI?

[sfeldkamp]

@girish

/ - An informational landing page. You can see it here: https://at.sethfeldkamp.com/
/oauth - OAuth routes

Then data is available from
/xrpc - Http API
/.well-known/atproto-did - DID verification

That's all I can find for routes in the source code.

https://github.com/bluesky-social/atproto/tree/main/packages/pds

[girish]

@sfeldkamp yeah, that works for the Cloudron package.