Created
September 5, 2025 14:10
-
-
Save rmoriz/1d3691961e4c8342cdd42a245e700acf to your computer and use it in GitHub Desktop.
Revisions
-
rmoriz created this gist
Sep 5, 2025 .There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,134 @@ # Architecture (Rust Rewrite) ## 1. Architectural Goals - Strong isolation of protocol handling, domain logic, persistence, and delivery. - Trait-based adapters for storage, queue, media processing, auth providers. - Predictable performance through bounded queues + backpressure. - Minimal unsafe Rust (only in well-reviewed media bindings layer if required). - Observability built-in (tracing spans at boundaries). ## 2. High-Level Component Map [Client Apps] → HTTP API Layer → (AuthN/Z, Rate Limit, Validation) → Domain Services → Repos (DB + Cache) → Persistence (Postgres/SQLite) Federation Inbound (HTTP Inbox) → Signature Verification → Activity Router → Domain Services Federation Outbound Queue → Delivery Workers → Remote Inboxes Media Upload → Media Pipeline (Validation, Transform, Thumbnail, Blurhash, Store) → Storage Backend Schedulers (cron-like) → Tasks (Domain Subscription Fetch, Media Cleanup, Key Refresh) Telemetry Exporter → OTLP / Prometheus / Logs ## 3. Layered Design - Interface Layer: HTTP (Axum or Actix-web) + JSON (Serde) + OpenAPI generation. - Application Layer: Coordinators / Services orchestrating domain operations (PostingService, FederationService, ModerationService, AccountService, MediaService, PermissionSubscriptionService). - Domain Layer: Pure logic objects + value types (Status, Visibility, InteractionPolicy, DomainPermissionSet, ActivityEnvelope) with invariants enforced in constructors. - Infrastructure Layer: Repositories (trait) + Adapters (SQLx/Postgres, SQLx/Sqlite), StorageAdapter (LocalFs, S3), QueueAdapter (InMemory, Redis future), MediaProcessor, SignatureVerifier, OIDCClient, TOTPGenerator, MarkdownRenderer, HTMLSanitizer. ## 4. Bounded Contexts 1. Accounts & Auth 2. Posting & Timelines 3. Federation 4. Moderation & Domain Permissions 5. Media & Storage 6. Configuration & Scheduling 7. Observability Each context exposes service traits; cross-context communication via domain events (in-process bus initial, pluggable later). ## 5. Key Modules (Rust Crates) - core-domain: Entity/value types, errors, result enums, time utilities. - core-infra: Shared adapter traits (Repository<T>, Queue, Storage, Clock, Hasher, Signer). - persistence-sqlx: Concrete impl for repositories (feature flags: postgres, sqlite). - http-api: Route definitions, request/response DTOs, auth middleware, OpenAPI generator. - federation: ActivityPub (serialize/deserialize, signature verify, activity router, delivery batching, dedupe store). - media-pipeline: Upload validators, transcoding workers, blurhash, exif stripping. - scheduling: Cron-like scheduler (time grid + durable last-run persistence). - moderation: Reports, domain permissions engine, spam filter heuristics interface. - auth: Local credential store, OIDC client, TOTP, OAuth2 provider (tokens, scopes), rate limiting + throttling. - timeline: Feed assembly (home, public, hashtag) with pagination strategy + caching. - config: Strong typed config loader (file + env + overrides) with validation. - telemetry: Tracing, metrics, logging façades. - cli: Admin commands (promote user, migrate, export/import, maintenance tasks). - service-bin: Composition root main binary wiring crates + DI. ## 6. Data Flow Examples ### Posting Client → /api/v1/statuses → Auth → PostingService.create() → (validate content policy, parse Markdown, sanitize) → MediaService link attachments → Repository(status).save() → FanoutPlan build → OutboundQueue.enqueue(deliveries) → TimelineAssembler.insert(home + possibly public) → Return hydrated Status DTO. ### Federation Inbound Create Remote POST inbox → SignatureVerifier.verify(request) → ActivityParser.deserialize(Create) → IdempotencyStore.check() → FederationService.handle_create(note) → Map to Status entity (visibility + local-only rules) → Save → Insert into timelines of permitted followers → Ack 2xx. ### Domain Permission Subscription Scheduler tick → SubscriptionService.fetchAllDue() → For each: Downloader → Parser (CSV/JSON/plain) → Diff existing → Drafts->Update / Blocks/Allows apply (policy resolution algorithm: priority descending) → Persist changes + emit events (for logging/metrics) → Next run timestamp. ## 7. Persistence Strategy - SQL schema normalized: accounts, statuses, media_attachments, polls, poll_options, follows, follow_requests, favourites, announces, interaction_policies, reports, domain_permissions, domain_permission_subscriptions, drafts, emojis, oauth_apps, oauth_tokens, deliveries (outbound queue durable), delivery_attempts (history), config_snapshots. - SQLite + Postgres via SQLx with feature gating; migrations stored as pure SQL; migration runner crate. - Secondary indexes: statuses(account_id, created_at DESC), deliveries(state, next_attempt_at), domain_permissions(domain, type, priority), hashtag_status(status_id, hashtag_id), timeline(home_account_id, status_id). ## 8. Caching Strategy - In-memory LRU (config target memory) for hot accounts, public keys, instance metadata, hashtags, recent statuses by id. - Optionally integrate Redis for distributed cache (future). Trait: Cache<K,V> with get/set/ invalidate. ## 9. Queue / Concurrency Model Initial: In-memory MPMC bounded channel (size configurable). Durable: deliveries table where workers poll (SELECT ... FOR UPDATE SKIP LOCKED) to allow horizontal scaling; memory channel acts as write-through cache of ready deliveries. Backpressure: When channel full, posting waits (bounded timeout) -> apply overflow policy (drop lowest priority or block). Metrics: queue_depth, delivery_latency, retry_count. ## 10. Federation Delivery Algorithm 1. Group recipients by shared inbox vs inbox. 2. Serialize Activity JSON once per unique body variant. 3. Batch sign (HTTP Signature) per target domain (reuse digest + date header). 4. Enqueue Delivery records (pending, attempt=0, next_attempt_at=now). 5. Worker picks pending sorted by next_attempt_at; concurrency = sender_multiplier * CPU. 6. Retry policy: exponential backoff with jitter (2^n * base) capped; mark failed after N attempts (config) -> dead letter table. 7. Idempotency: store delivered activity id + target domain hash to skip duplicate. ## 11. Media Pipeline Stages: Accept (size/type sniff) → Store original temp → Transcode (if video/audio) → Generate thumbnails + blurhash → Strip EXIF → Persist variants → Commit metadata row (transaction) → Publish status referencing attachment IDs. Pool sizing deterministic; tasks instrumented with tracing spans. ## 12. Security Considerations - Strict Content Security Policy builder including dynamic storage origins. - Signature verification enforces digest, date skew, key fetch caching with expiry + pinning. - HTML sanitization allowlist minimal (links, emphasis, code blocks, lists, blockquote) no inline event handlers. - Rate limiting pre-auth (IP) + post-auth (account) buckets. - TOTP secrets stored encrypted at rest (libsodium or ring) with key rotation plan. - All secrets loaded via env; config file redaction in logs. ## 13. Configuration System Single layered loader: (Defaults → File (TOML/YAML) → Env var prefix GTS_ → CLI flags). Validation pass building typed Config struct; produce diff summary on startup. Hot-reload limited to non-critical fields (log level, exposure toggles) via signals. ## 14. Observability Spans: http.request, db.query (sampling threshold), federation.outbound, federation.inbound, media.process, scheduler.run, subscription.fetch. Metrics: counter(deliveries_total{result}), gauge(queue_depth), histogram(request_duration_seconds{route}), gauge(db_pool_in_use), histogram(media_process_seconds), counter(spam_messages_filtered), gauge(active_sessions). Log correlation: request-id header propagate into spans. ## 15. Extensibility Points Traits: Storage, Queue, MediaTranscoder, SpamFilter, AuthProvider, KeyFetcher, MarkdownRenderer, Sanitizer. Plugin architecture (Phase 2): dynamic library or Wasm sandbox for SpamFilter + MarkdownRenderer replacements. ## 16. Migration Strategy from Go Version - Export data (SQL dump + object storage copy) → Import translator tool (maps Go schema to Rust schema via staged adapters) → Validate counts (accounts, statuses, follows) ±0%. Tool crate: migrator-go2rust. - Provide read-only compatibility mode: Rust server boot with Go DB schema + shadow write into new tables until cutover. ## 17. Testing Strategy - Unit: domain invariants (visibility, interaction policy resolution, domain permission merging). - Property: delivery dedupe, permission subscription diff algorithm. - Integration: federation scenarios with test harness (embedded remote actor fixtures). - Load: timeline read, posting fanout, media bursts. - Security: fuzz signature parser, markdown sanitizer. ## 18. Deployment Topologies 1. Single Binary (SQLite + local storage) — small personal. 2. Single Binary (Postgres + S3) — small community. 3. Split: API + Workers (shared Postgres + S3) — medium scale. 4. Scaled: API (N replicas), Workers (M replicas), Redis/Queue (future), CDN for media — larger deployment. ## 19. Future Evolution - Replace in-process queue with pluggable distributed (NATS or Redis streams) preserving Delivery trait contract. - Add relay ingestion publisher crate. - Introduce reputation scoring module feeding SpamFilter trait. ## 20. Open Technical Questions - Adopt Axum vs Actix: leaning Axum + tower for middleware simplicity, hyper alignment. - Choose SQL migration tool (refinery vs sqlx migrate). Prefer sqlx migrate for fewer deps. - Offload video transcoding to separate microservice? Start in-process, abstract early. --- End Architecture. This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,125 @@ # Dependencies (Current Go vs Proposed Rust) ## 1. Categories - Web / HTTP - Serialization & Parsing - Federation / Protocol - Storage & Media - Auth & Security - Database & ORM - Observability - Utilities (validation, rate limiting, caching, scheduling) ## 2. Current Go Dependencies (Representative) | Purpose | Go Library | Notes | |---------|-----------|-------| | HTTP routing | gin-gonic/gin | High performance, middleware stack | | Middleware (cors, gzip, sessions) | gin-contrib/* | Assorted features | | OAuth2 server | superseriousbusiness/oauth2 (fork) | Token issuance, scopes | | OIDC client | coreos/go-oidc | External IdP integration | | ActivityPub / ActivityStreams | superseriousbusiness/activity (fork go-fed) | Extended vocab/custom terms | | HTTP Signatures | superseriousbusiness/httpsig | Draft-cavage-12 style | | ORM | uptrace/bun | Postgres & SQLite support | | SQLite (CGO-free) | modernc.org/sqlite | Pure Go implementation | | Postgres driver | jackc/pgx | Driver + toolkit | | Caching | gruf/go-cache | LRU / TTL caches | | Scheduling | gruf/go-sched | Task scheduler | | Storage (S3/local) | minio/minio-go, gruf/go-storage | Object store abstraction | | Markdown | yuin/goldmark | Rich text parsing | | Sanitization | microcosm-cc/bluemonday | HTML policy | | Blurhash | buckket/go-blurhash | Image placeholder | | EXIF removal | exif-terminator | Privacy | | FFmpeg WASM wrapper | gruf/go-ffmpreg | Embedded media processing fallback | | UUID / ULID | google/uuid, oklog/ulid | ID generation | | Rate limiting | ulule/limiter | IP/route buckets | | Validation | go-playground/validator | Struct and field rules | | Password strength | wagslane/go-password-validator | Entropy estimation | | RSS/Atom | gorilla/feeds | Feed export | | Websocket | gorilla/websocket | Real-time features (if used) | | OpenAPI generation | go-swagger | Swagger spec | | Telemetry | opentelemetry-go, uber-go/automaxprocs | Metrics/tracing + CPU config | ## 3. Proposed Rust Replacements | Purpose | Rust Crate | Justification / Notes | |---------|-----------|-----------------------| | HTTP routing & middleware | axum (+ tower, hyper) | Modern, composable, async; ecosystem friendly | | JSON / serialization | serde, serde_json | De-facto standard | | OpenAPI spec | utoipa or okapi | Derive-based spec generation | | OAuth2 server | oxide-auth or custom minimal (JWT + DB) | Evaluate feature coverage; possible custom for scope control | | OIDC client | openidconnect | Mature OIDC flows | | ActivityPub model | Custom crate (activity-types) + serde | Control over extensions + strongly typed enums; potential reuse of existing apub crates (e.g., activitystreams crate) after assessment | | HTTP Signatures | http-signature-normalization (if stable) or custom | Need cavage spec compliance + canonicalization tests | | Database (Postgres/SQLite) | sqlx | Async, compile-time query checking | | Migrations | sqlx migrate | Native integration | | Caching (in-memory) | moka | High-perf, async-safe; or dashmap + custom LRU | | Scheduling | cronback (lightweight) or tokio_cron_scheduler | Choose minimal dependency, else custom time wheel | | Object Storage (S3) | aws-sdk-s3 (feature trimmed) or s3 crate | Consider complexity vs minio interoperability | | Local FS storage | std::fs + tokio::fs | Simplicity | | Media transcoding | ffmpeg-next (unsafe boundary) or spawn external ffmpeg | Start with external binary invocation for safety; abstract via MediaTranscoder trait | | Image processing | image, img_parts (for metadata), blurhash | Blurhash crate exists; exiftool-rs or rexif for EXIF removal | | Markdown | pulldown-cmark + custom renderer | Fast, safe; apply sanitization after render | | HTML Sanitization | ammonia | Policy-based sanitizer | | ULID | ulid | Sequential-friendly IDs | | UUID | uuid | Only if needed (prefer ULID) | | Rate limiting | governor | Token bucket, async aware | | Validation | validator crate | Derive macros similar to go-playground/validator | | Password strength | zxcvbn | Provide entropy score gating | | RSS/Atom | atom_syndication + rss | Feed generation | | Websocket (if needed) | tokio-tungstenite | Async websockets | | Tracing | tracing, tracing-subscriber, opentelemetry, opentelemetry-otlp | Unified observability | | Metrics | metrics + metrics-exporter-prometheus OR opentelemetry-metrics | Unify with OTEL path | | HTTP Client (federation) | reqwest (rustls-tls) | Robust, async; configure timeouts, TLS policies | | Signature Digest (SHA256) | ring or sha2 | ring for performance; fallback sha2 | | Randomness | rand, getrandom | Standard | | Time | time or chrono (v0.4) | Prefer time crate for performance + formatting | | Concurrent queue | tokio mpsc + persistent DB table | Hybrid durable queue | ## 4. Mapping & Gaps | Go Feature | Rust Plan | Notes | |------------|----------|------| | Embedded WASM ffmpeg | External binary / wrapper first | Defer WASM until needed | | go-swagger codegen | utoipa derive | Might need manual some complex schemas | | Custom Activity vocab (interaction policies) | Enum + serde tag + extension field map | Maintain names for compatibility | | LRU/TTL caches | moka with per-entry TTL | Ensure memory target enforcement via instrumentation | | Domain permission subscription parsing (CSV/JSON/plain) | csv + serde_json + custom plain parser | Simple streaming approach | | BlueMonday sanitizer | ammonia | Align allowed tags list | | Rate limit + throttling combined | governor + tower middleware + concurrency semaphore | Provide 429 & Retry-After semantics | ## 5. Security & Crypto Considerations - Prefer rustls over OpenSSL for TLS (simpler supply chain) unless enterprise needs mandate OpenSSL features. - Use ring for signature hashing + constant-time comparisons. - Argon2id (argon2 crate) or scrypt for password hashing (instead of bcrypt) with tunable params. - Encrypt TOTP secrets using age or libsodium (if integrating) else ring AEAD (chacha20poly1305) with rotated key. ## 6. Optional Future / Stretch Deps | Capability | Candidate | |------------|-----------| | Distributed Cache | redis + redis::aio | | Message Queue | nats, lapin (RabbitMQ), or redis streams | | Full-text search | tantivy or meilisearch integration | | Content classification | onnxruntime for ML filters | | Wasm plugin sandbox | wasmtime | ## 7. Dependency Selection Principles - Minimal unsafe; isolate where unavoidable (media). - Stable, well-maintained crates (recent commits, semver discipline, security posture). - License compatibility (AGPL server code can link to permissive crates; avoid copyleft conflicts beyond AGPL). - Benchmarks where multiple candidates similar (axum vs actix, governor vs tower-ratelimit). ## 8. Versioning & Update Policy - Lock versions via Cargo.lock; Renovate/Dependabot weekly PRs. - Security advisories (cargo audit in CI) gate release. - Semantic import paths not required (Rust) but adopt semver expectations for public crates we publish (e.g., activity types crate). ## 9. Tooling - Formatter: rustfmt (default config) + clippy (deny pedantic selective). - Audit: cargo-audit, cargo-deny (license + bans). - Fuzzing: cargo-fuzz targeting Activity deserializer, signature parser, markdown sanitizer pipeline. ## 10. Open Dependency Questions - Choose between sqlx vs SeaORM? (sqlx chosen for lighter abstraction, compile-time checks.) - Evaluate if external ffmpeg invocation meets performance; fallback to binding only if hot path becomes bottleneck. - Decide on combined metrics path (native metrics crate + OTEL export) vs solely OTEL. --- End Dependencies. This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,127 @@ # Product Requirements Document (PRD) ## 1. Vision Provide a lightweight, privacy-and-safety first, backend‑only ActivityPub social networking server, enabling small to medium communities or single users to self‑host a resilient Fediverse presence with fine‑grained control over federation, interaction, moderation, and resource usage. Rust rewrite aims to improve memory safety, predictable performance, and extensibility, while maintaining Mastodon API compatibility to leverage existing clients. ## 2. Objectives / Success Criteria | Objective | KPI (Indicative) | Rationale | |-----------|------------------|-----------| | Functional parity (core) | 95% of currently implemented Mastodon-compatible endpoints + documented custom extensions | Keep existing client ecosystem usable | | Performance | <=200MiB idle RAM target (single user) ; p95 timeline request < 250ms (SQLite) | Rust efficiency and async model | | Safety & Moderation | All existing moderation workflows (reports, domain permissions, interaction policies) functioning by Beta | Maintain differentiation | | Federation Reliability | 99.5% successful outbound delivery (retry-adjusted) | Trust in federation | | Extensibility | Pluggable storage, queue, media, auth layers via traits | Future proof | | Upgrade UX | Zero manual migration for common configs; automatic DB migrations | Admin happiness | ## 3. Target Users & Personas - Solo Admin / Power User: runs single-user instance; needs low resource footprint, simple backup, reliability. - Small Community Moderator: 10–200 users; needs allow/block lists, report handling, domain permission subscriptions, spam mitigation. - Third-Party Client Developer: depends on stable Mastodon-compatible + documented divergence points. - Integrator / Enterprise Dept: wants OIDC, observability, auditable logs, predictable upgrade path. ## 4. In-Scope Functional Requirements (Rust Rewrite) ### 4.1 Federation & Protocol - ActivityPub server + client-to-server flows for: Follow / Accept / Reject, Create / Update / Delete (statuses, polls, edits), Announce (boost), Like/Favourite, Move, Block, Undo. - HTTP Signatures (draft-cavage-12 semantics) for inbox/outbox delivery; shared inbox optimization. - WebFinger discovery; nodeinfo 2.0/2.1; host-meta. - Instance federation modes: blocklist, allowlist (zero mode reserved/future). - Domain permission system: blocks, allows, drafts, excludes, subscriptions (scheduled fetch/parse, priority override, adoption, retractions, drafts → active promotion). - Spam filter heuristic toggle + hooks for future modular filters. ### 4.2 Posting & Content - Status creation with: visibility (public, unlisted, followers-only, direct), local-only, CWs, Markdown → sanitized HTML, media attachments (images, video, audio), polls (limits), scheduled & backdated statuses (config gated), edits, reply restrictions (interaction policies: who can reply/like/announce/quote). - Rich text pipeline: Markdown parse, sanitization (policy), linkification, media description limits, blurhash generation. - Hashtags indexing + retrieval; filters v2; mute (thread, account, duration optional); block, follow requests, migration (Move activity). ### 4.3 Accounts & Auth - Local signup request workflow (open/closed; reason required; backlog + daily limit; moderation approval queue). - OIDC login (optional) with group-based allow + admin rights; 2FA TOTP for local auth path; password strength validation. - OAuth2 application registration, token management (create/revoke, scopes) Mastodon-compatible. ### 4.4 Moderation & Safety - Report lifecycle (create, view, resolve with note); strict visibility enforcement; remote identity + key re-fetch/invalidation; interaction policy defaults; domain permission subscriptions scheduling + manual review; scraper deterrence (PoW) optional. ### 4.5 Media & Storage - Pluggable storage (local FS, S3-compatible with redirect/proxy modes, key prefixes, cache pruning schedule, remote media cache TTL). Media processing pipeline (ffmpeg pool concurrency, thumbnails, size caps per type, emoji import with size limits). EXIF stripping. ### 4.6 Timelines & Feeds - Home, public (configurable exposure), hashtag, direct conv. view, account statuses, favourites, bookmarks (if retained), RSS opt-in per account. ### 4.7 Configuration & Administration - Strongly typed layered config (env, file, cli); dynamic instance metadata editing; instance stats exposure modes (normal/zero/serve/baffle); CSP management; rate limiting + throttling; request header filtering (block/allow). Scheduled tasks (subscriptions process, media cleanup) with robust scheduler. ### 4.8 Observability - OpenTelemetry traces + metrics (optional); structured logging; request correlation ID; health/status endpoints (internal) for readiness/liveness. ### 4.9 Backup & Migration - Logical DB backup guidance (Postgres & SQLite); storage object copy independence; path toward data export of user data (statuses/media/activity) for future portability. ### 4.10 Extensibility & Customization - Theming (static templates + CSS), user custom CSS (opt-in, limit length). Custom emoji management (local upload, remote import). Future plugin points (content filters, auth strategies, storage, media processors, queue backend). ## 5. Out-of-Scope (Initial Rust MVP) - Fediverse relay support (planned later). - Groups / group activities (wishlist feature). - Advanced reputation-based federation heuristics. - Community decision-making workflows for moderation. - Alternative post rendering templates (blog/gallery) beyond baseline HTML. - Full web client (remains backend-first philosophy). ## 6. Non-Functional Requirements | Category | Requirement | |----------|------------| | Performance | Async delivery pipeline; bounded memory; backpressure on federation queue; predictable latency under 200 concurrent active users. | | Scalability | Horizontal scaling via stateless web + background workers sharing DB + object store; eventual addition of message queue (NATS/Kafka/Redis) as pluggable. | | Security | Strict HTTP signature verification; configurable TLS; safe HTML sanitization; rate limiting + throttling; optional PoW deterrence; secure password policies; secret management (env). | | Privacy | Configurable exposure of stats + peers + block/allow lists; strict scoping of visibility; local-only content stays local. | | Reliability | At-least-once outbound federation with dedupe; idempotent inbox processing; retry with exponential backoff; durable queues. | | Observability | Tracing spans for federation inbound/outbound, DB ops, media pipeline; metrics (latency, queue depth, deliveries, failures, cache hit/miss). | | Upgradeability | Zero downtime migrations for Postgres; gated feature flags; semantic versioning. | | Maintainability | Clear module boundaries; interface-driven adapters; 80%+ doc coverage of public interfaces. | | Accessibility | Markdown alt text enforcement option; consistent alt propagation; keyboard navigable static pages. | | Internationalization | Language tags advertisement; UTF-8 correctness; normalization on hashtags + mentions. | ## 7. Key Domain Model Entities Account, Credential, PublicKey, Status, MediaAttachment, Poll, PollOption, Follow, FollowRequest, InteractionPolicy, Favourite, Announce, Notification, DomainPermission (Block/Allow/Draft/Exclude/Subscription), Report, InstanceSettings, CustomEmoji, OAuthApp, OAuthToken, TimelineEntry, QueueMessage (OutboundDelivery), WebfingerAlias, NodeInfoSnapshot. ## 8. Risks & Mitigations | Risk | Mitigation | |------|------------| | Protocol divergence causing interoperability issues | Conformance test harness vs known servers (Mastodon, Akkoma) + golden Activity JSON fixtures | | Performance regressions vs Go version | Benchmark suite (posting, timeline read, 1k follower fanout) gating PRs | | Async complexity leading to subtle race bugs | Use Rust ownership & Send/Sync boundaries; property tests for queue ordering & dedupe | | Media pipeline complexity (ffmpeg) in Rust | Isolate via wrapper service or leverage existing safe bindings (e.g., `rust-ffmpeg-next`), fallback WASM path later | | DB migration errors | Declarative schema migrations (sqlx or refinery) + integration tests on SQLite/Postgres | | Security regressions | Threat modeling + dependency audit CI; fuzzing for signature + parser components | ## 9. Milestones (Indicative) 1. Core Foundations (Config, DB, Entities, HTTP, Auth base) 6w 2. Federation Inbound/Outbound (Follow/Create/Like/Announce, Signatures) 8w 3. Posting Pipeline + Media + Markdown 6w 4. Timelines + Mastodon API surface (read endpoints) 6w 5. Moderation + Domain Permissions + Reports 5w 6. OIDC + 2FA + OAuth Apps/Tokens 3w 7. Filters / Mutes / Edits / Move 4w 8. Observability + Scaling Hardening 3w 9. Beta Parity Validation + Docs 4w ## 10. Acceptance Criteria (Beta) - All mandatory endpoints respond equivalently (fields subset superset doc'd) vs reference tests. - 100% pass federation conformance scenarios (list maintained internally). - Media uploads, polls, edits, local-only, interaction policies function end-to-end with at least two popular Mastodon API clients. - All moderation workflows executable via API + admin endpoints. - Zero critical memory safety or data corruption issues in 30-day soak. ## 11. Open Questions - Adopt ActivityPub vocabulary custom extensions as-is or rationalize? (Need compatibility matrix.) - Pluggable queue: implement initially with in-process channels or adopt Redis early? - Media transcoding strategy: synchronous vs background job separation? (Likely background + status placeholder.) - Backdating scope: retain or restrict? (Config remains.) ## 12. Glossary - Local-only: Status retained locally; never federated even via Announce. - Draft Domain Permission: Proposed block/allow not yet enforced. - Interaction Policy: Per-status constraints on reply/like/announce/quote. --- End of PRD. This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,138 @@ # Core Workflows ## 1. User Registration (Local Auth Path) 1. Visitor loads /signup (client or web form). 2. System checks config: registration-open & backlog/daily limits. 3. User submits (username, email, password, optional reason / required if configured). 4. Validate: username policy, password strength, uniqueness, rate limit. 5. Persist signup request (pending) + send notification (admin UI / email if configured). 6. Admin reviews queue, approves or rejects. 7. On approval: account provisioned (keys, default interaction policies, default visibility) → welcome notification. 8. User logs in (username + password + optional 2FA) obtaining OAuth token. OIDC Path differences: - Step 3 replaced by OIDC redirect → id token → group mapping → auto approve if allowed group; optional admin elevation if group matches. ## 2. Authentication & Authorization 1. Client registers OAuth application (scopes) or reuses existing. 2. User obtains access token (password grant or OIDC + code exchange). 3. Each request: token introspection (cache), scope check, rate limit bucket (IP + account), build request context. 4. Optional 2FA enforcement at login. ## 3. Posting a Status 1. Client submits POST /api/v1/statuses with text, visibility, media ids, CW, local-only flag, scheduled_at/backdated potential, interaction overrides. 2. Validate characters <= limit; media attachments belong to user & not already attached; poll options within limits; scheduling within allowance. 3. Markdown parse → sanitized HTML; generate summary fields (plain text, hashtags, mentions extraction). 4. Persist status + media linkage transactionally. 5. Build fanout plan: local timelines (author home, public if eligible, hashtag timelines) + remote deliveries (followers, mentioned actors respecting visibility + interaction policies). 6. Enqueue outbound deliveries. 7. Respond with status representation (including media metadata, poll state, interaction policy flags). ## 4. Editing a Status 1. Client sends PUT/PATCH edit endpoint with new text/media changes. 2. Re-parse + sanitize; version history persisted (for federated edit propagation) with updated updated_at. 3. Re-deliver Update activity to followers + mentioned recipients. 4. Return edited status. ## 5. Following an Account 1. User issues POST /follow for remote or local target. 2. If remote: build Follow activity, sign, enqueue delivery to target inbox. 3. Remote may auto-accept (unlocked) or issue Follow Request logic (locked) → Accept/Reject returned inbound. 4. On Accept inbound: mark relationship; schedule backfill (optional) of recent statuses. 5. On Reject: mark state; client notified. ## 6. Handling Inbound Activities (Create Note) 1. Receive signed HTTP POST at inbox/shared inbox. 2. Verify signature (public key fetch or cache); check Content-Type. 3. Parse JSON to Activity; dedupe by activity id + digest. 4. Validate relevance (mentions/follow relationships) + spam heuristics if enabled. 5. Materialize entities (account if unknown, media attachments metadata) → store. 6. Insert into follower timelines as appropriate respecting visibility/local-only semantics (local-only ignored inbound). 7. Acknowledge 2xx; metrics update. ## 7. Interaction Policies (Reply/Like/Announce/Quote Control) 1. On status creation: derive base policy from user defaults; overrides from request. 2. Represent as InteractionPolicy entity with booleans / enumerations. 3. Enforcement points: inbound Like/Announce/Reply activities or API actions check permit; if blocked → rejection activity (optional) or HTTP 403. ## 8. Domain Permission Subscription Processing 1. Scheduler triggers at configured time. 2. For each subscription (ordered by priority desc): fetch list, parse (CSV/JSON/plain), validate entries. 3. For each entry: if existing permission managed by higher priority keep; else create or update draft or active permission (depending config). 4. Handle retractions (remove or orphan per setting), adopt orphan permissions flagged. 5. Emit events for changed counts; store run log (success/failure metrics). 6. Admin UI shows new drafts for approval; on approval convert draft → active block/allow. ## 9. Moderation Report Lifecycle 1. User files report (target account/status refs, reason, forward to remote flags). 2. Persist report; notify local moderators (email/notification) and optionally send federated Flag activity. 3. Moderator views report list → filters by status open/resolved. 4. Moderator resolves (action note optional) and executes actions (suspend, silence, restrict media) using account moderation endpoints. 5. Resolution stored; reporter (local) sees resolution note. ## 10. Media Upload & Processing 1. Client uploads media (multipart) → preflight size/type check. 2. Store temp object; queue processing job. 3. Worker: transcode/resize (respect max dims, generate variants), blurhash compute, EXIF strip, finalize storage (local or S3) with canonical key, sign URL or prepare proxy path. 4. Persist MediaAttachment (state=ready) else (failed) with error for client. 5. Return id + preview metadata; attachment may now be referenced by posting workflow. ## 11. Remote Media Cache Eviction 1. Nightly scheduler enumerates remote media older than TTL. 2. For each candidate: delete storage object; keep DB record flagged evicted. 3. On future access: lazy re-fetch pipeline repopulates. ## 12. Key Rotation / Public Key Expiry 1. Admin triggers key invalidate for domain or scheduled maintenance. 2. System marks remote keys stale; next validation requiring signature triggers fetch. 3. Failure path: if fetch fails, delivery/inbound validation error logged; retry policy engaged. ## 13. Timeline Assembly 1. Read request /api/v1/timelines/home with pagination params since_id / max_id / limit. 2. Query timeline table (denormalized status references) ordered by created_at desc with index. 3. Hydrate statuses (batch fetch accounts, media attachments), apply mutes/filters. 4. Return list + pagination cursors; update last-read position if needed. ## 14. Spam Filter Heuristic (Optional) 1. Inbound candidate passes relevance gates. 2. If spam filter enabled: apply heuristic sequence (follower check, mention count, link/media presence, lock status context, follow request relation). 3. Decision = OK or Spam (drop silently) with metrics increment. ## 15. Scheduled & Backdated Statuses 1. User schedules status (future or past date if allowed) within daily + total caps. 2. Store with scheduled_at and state=pending. 3. Scheduler poll: find due statuses (scheduled_at <= now and state=pending) → run standard posting pipeline fanout. 4. Backdated statuses inserted with historical created_at but still deliver new Create activities (some clients may order by created_at). Document behavior. ## 16. Two-Factor Authentication Enablement 1. User requests enable 2FA → system generates secret + QR (TOTP) + recovery codes. 2. User confirms code; secret persisted (encrypted) + 2FA enabled flag. 3. Login flow requires password + current TOTP; fallback to recovery code rotates used code. ## 17. Instance Statistics Exposure Modes 1. Request to /nodeinfo or /api/v1/instance. 2. StatsService chooses mode: normal/zero/serve/baffle. 3. Assemble values (cached snapshot) or transform (zero-out or randomize) → respond; robots.txt conditional generation hides/allows endpoints. ## 18. Scraper Deterrence (PoW) 1. Anonymous web request to profile/status page. 2. If no valid cookie and deterrence enabled: respond with challenge script + difficulty. 3. Client solves → POST solution; verify; issue cookie valid 1h. 4. Subsequent requests pass through until expiry. ## 19. Account Migration (Move) 1. User initiates move to new server (outbound): send Move activity to followers; update redirect fields. 2. Inbound Move from followed account: update local follow relationships to new actor URI; optionally backfill new profile data. ## 20. Backup & Restore Backup: 1. Pause instance (optional) or run consistent snapshot routine. 2. Dump DB (pg_dump or copy sqlite file) + archive storage objects (rsync / s3 sync) + config file. Restore: 1. Provision DB, import schema + data. 2. Restore storage objects to original key paths. 3. Start Rust server with identical host + account-domain config. 4. Run migration tool (schema diff) + verify counts. --- End Workflows.