Digging deeper
PubSub: messages between processes
smeltery::pubsub::PubSub carries messages between the processes of one app (Sparks' Broadcast sends through
it). Every app has one: PubSub::of(&app), or app.service::<PubSub>(). A message has a topic and a JSON payload
of at most 64 KiB:
use serde_json::json;
use smeltery::pubsub::PubSub;
use smeltery::{App, Result};
async fn watch_prices(app: App) -> Result<()> {
let pubsub = PubSub::of(&app).expect("every app has one");
let mut prices = pubsub.subscribe("prices");
pubsub.publish("prices", &json!({ "symbol": "ORE", "price": 7 })).await?;
while let Ok(message) = prices.recv().await {
println!("{} (from another process: {})", message.payload, message.remote);
}
Ok(())
}
publish delivers to the subscribers in this process at once and waits for the driver to take the message for the
other processes (it returns the driver's error). forward(topic, &payload) sends to the other processes only and
never waits: the message goes into a queue of 1024 that one task sends on; a full queue or a driver error drops it,
counts it (dropped()) and logs it at most once a minute. At shutdown the queue keeps taking messages while they
come (for work that stops on the same signal), then sends what is left within 2 s; a message forwarded after that,
or left unsent, is counted as dropped too. subscribe(topic) receives the topic's messages from
every process. Each topic has its own buffer of 1024 messages in a process, so a burst on one topic never makes the
subscribers of another fall behind; a subscriber more than 1024 messages behind its topic gets RecvError::Lagged
and then receives again (past 256 subscribed topics in one process, further topics share one buffer; the
framework's topics auth, anvil and sparks always have their own).
Delivery is at most once and in order per publishing process: a message sent while the database or Redis is down is
lost, so a page or client reloads its state after a reconnect.
PUBSUB_DRIVER decides how messages reach the other processes. The process logs the driver it chose at start.
PUBSUB_DRIVER |
Messages reach |
|---|---|
auto (default) |
decided by what the process runs: serve with its background work uses local; serve --no-agents (or WATCHFIRE_IN_SERVE=false, or ANVIL_IN_SERVE=false), work and Anvil's anvil process use redis when CACHE_STORE=redis (feature redis), else database when the app has a database, else local with a warning. Console commands and TestApp use local |
local |
this process only |
database |
every process on the app's database: the pubsub_messages table, which each process reads every PUBSUB_POLL_MS (default 250) while something in it subscribes (a web process with Sparks always does) |
redis |
every process on the Redis server of REDIS_URL (feature redis): PUBLISH / SUBSCRIBE on the channel <CACHE_PREFIX>pubsub |
Several serve processes behind a load balancer each count as one process under auto, and so does a serve that
runs its background work next to an extra work process: give them PUBSUB_DRIVER=database or redis. An explicit database or redis driver that lacks its database, APP_KEY or
feature stops serve and work from starting. TestApp uses local unless TEST_PUBSUB_DRIVER names another
driver. Messages that leave the process are encrypted and authenticated with AES-256-GCM under a key derived from
APP_KEY, so every process needs the same APP_KEY; a message sealed under another key, or sent more than 5
minutes earlier (by the sender's clock; a message written into the table or Redis again later), is skipped, counted
and logged, so the clocks of the app's machines must agree within 5 minutes (NTP). Every process of the database
driver deletes rows older than a minute every 10 s (in statements of 500
rows, for at most 2 s each time). The redis driver checks its subscriber connection with PING every 30 s and
reconnects when a PING goes unanswered for 2 s, so a connection dropped silently by a NAT or load balancer is
replaced.
New apps with Watchfire (headless apps, and web apps with the watchfire building block) and web apps with the
anvil building block have the migration of the pubsub_messages table (smeltery::pubsub::migrations::{up, down}; MEDIUMTEXT payloads on MySQL).
smeltery pubsub:install adds the same migration to an app without it; run smeltery migrate after it.