Scaling Puma Web Workers and Sidekiq Queues for Mastodon Hubs

By 4FreeForums Editors · Published October 28, 2026 · Sources linked below

Running a decentralized social community requires balancing incoming web requests, high-throughput background processing, and real-time feeds. When traffic spikes across the Fediverse, administrators must properly allocate system resources across three core process types: web handling, streaming connections, and background tasks. You can evaluate how these requirements match other platforms in our Forum Software Server Requirements Compared guide, or explore community moderation essentials in our Mastodon Moderation Tools: An At-a-Glance Profile. The following checklist details the exact configuration variables, background queue structures, and database connection limits documented in the Scaling up your server - Mastodon documentation.

1. Puma Web Worker and Thread Allocation Checklist

The web layer uses Puma to process short-lived HTTP requests for the application. Tuning this layer requires balancing memory allocation against processor capacity:

• Control worker processes: Set WEB_CONCURRENCY in your environment to define how many worker processes Puma runs.
• Set threads per process: Configure MAX_THREADS to specify the number of threads assigned to each Puma worker process.
• Evaluate resource trade-offs: Threads share the memory space of their parent process. Separate processes allocate their own memory, though they share some memory via copy-on-write.
• Monitor hardware limits: A larger number of threads maxes out your CPU first, while a larger number of processes maxes out your RAM first.
• Prioritize throughput: In terms of application throughput, more processes are better than more threads.

2. Sidekiq Concurrency and Database Pool Checklist

Background processing offloads asynchronous tasks to ensure user-facing HTTP responses remain quick and to keep request aborts from interrupting essential jobs. As documented in the Scaling up your server - Mastodon documentation, Sidekiq concurrency requires specific command-line arguments and matching database pools:

• Specify thread counts on launch: The thread count for Sidekiq is regulated by a command-line flag rather than an environment variable. For example, running bundle exec sidekiq -c 15 initiates a Sidekiq process with 15 threads.
• Scale background delivery: While web threads alter responsiveness for end users, Sidekiq threads dictate how fast posts deliver from the author to remote recipients and how quickly outbound emails dispatch.
• Match the database pool: Every running background thread requires its own database connection. Set the DB_POOL environment variable to a number at least equal to the total number of threads configured for that process.
• Launch dedicated worker processes: Administrators can start multiple Sidekiq processes with varied arguments using distinct systemd service files to maintain parallel job handling.

3. Sidekiq Queue Priority and Routing Checklist

Sidekiq manages tasks through prioritized queues defined by how severely an unworked queue degrades local member experience. Sidekiq processes jobs sequentially: it evaluates tasks in the first queue, and only if it finds none does it check the subsequent queue. You can isolate queues by passing arguments like bundle exec sidekiq -q default. The queues rank in descending importance:

1. default: Processes all tasks that directly affect local users.
2. push: Handles delivery of activity payloads out to external servers.
3. ingress: Manages incoming remote activities. It has lower priority than default so local users still see their posts when the instance is under load.
4. mailers: Manages the delivery of notification emails.
5. pull: Carries lower-priority administrative tasks, such as imports, backups, resolving remote threads, user deletions, and reply forwarding.
6. scheduler: Executes periodic cron tasks, including refreshing trending hashtags and clearing log files. This queue must never run in more than one Sidekiq process at a time.
7. fasp: Manages interactions with external servers providing FASP services.

4. Streaming API Multi-Process Checklist

The streaming API server handles persistent HTTP and WebSocket connections so clients receive real-time notifications and timeline updates:

• Configure base URLs: Set STREAMING_API_BASE_URL to place streaming traffic on a separate subdomain or distinct load balancer.
• Scale across local ports: Run multiple streaming server instances by setting different values for PORT (such as ports 4000, 4001, and 4002) and routing traffic across them using nginx with a least_conn upstream directive.
• Manage service units: Start individual systemd services such as mastodon-streaming@4000.service, mastodon-streaming@4001.service, and mastodon-streaming@4002.service instead of relying on a single default process.
• Respect pool caps: Unlike older clustering configurations that created connection pools per internal cluster worker, a single streaming process uses at maximum DB_POOL PostgreSQL connections.
• Track Prometheus metrics: Monitor individual streaming processes directly via the /metrics endpoint using Prometheus rather than hitting the reverse proxy. Watch metrics such as mastodon_streaming_connected_clients, mastodon_streaming_connected_channels, mastodon_streaming_messages_sent_total, and mastodon_streaming_redis_messages_received_total.

5. PgBouncer Connection Pooling Setup Checklist

Because the total count of open PostgreSQL connections generally equals the combined thread count across Puma, Sidekiq, and the streaming API, larger servers can easily exceed the default PostgreSQL limit of 100 connections:

• Deploy connection pooling: Install the pooling service on Debian and Ubuntu systems using sudo apt install pgbouncer.
• Configure transaction mode: In /etc/pgbouncer/pgbouncer.ini, change pool_mode from session to transaction. This ensures PostgreSQL connections open at the beginning of a transaction and terminate immediately when it completes.
• Define connection limits: Keep the default settings initially with listen_port = 6432, max_client_conn = 100, and default_pool_size = 20, increasing pool limits later as load grows.
• Set authentication credentials: Add hashed md5 passwords to /etc/pgbouncer/userlist.txt for the mastodon and pgbouncer users, specifying auth_type = md5 in your configuration.
• Disable prepared statements: In Mastodon's .env.production, configure PREPARED_STATEMENTS=false because transaction-based pooling cannot support prepared statements.
• Update database port: Change DB_PORT=6432 in .env.production to route application queries through PgBouncer.
• Bypass pooler for migrations: Because database migrations cannot run through PgBouncer, execute schema updates by specifying the direct PostgreSQL port: RAILS_ENV=production DB_PORT=5432 bundle exec rails db:migrate.

6. Redis Partitioning and Caching Checklist

Redis maintains home feeds, list feeds, background Sidekiq queues, and streaming subscriptions. While Postgres data must never be lost, high-traffic hubs can separate volatile caches to protect primary memory queues:

• Allocate a dedicated cache store: Set CACHE_REDIS_URL (or separate entries like CACHE_REDIS_HOST and CACHE_REDIS_PORT) to isolate temporary caching from persistent application data.
• Reduce disk operations: For the dedicated cache Redis instance, disable background disk saves to free I/O capacity, since cache data loss during a restart will not break state.
• Monitor connection usage: Staff with DevOps permissions can use the PgHero panel within the Administration view to observe active PostgreSQL connection usage.

Sources

• Scaling up your server - Mastodon documentation

Published October 28, 2026. No affiliate links on this page. Facts come from the sources linked above; check them for updates.