graph TB
subgraph DEV_VM["Development VM"]
D_MySQL[(MySQL DB<br/>EXISTING)]
D_Maxwell[Maxwell CDC<br/>EXISTING]
D_CDCPub[CDC Publisher<br/>NEW<br/>Port: 3001]
D_MySQL -->|binlog stream| D_Maxwell
D_Maxwell -->|stdout/JSON| D_CDCPub
end
subgraph STAGING_VM["Staging VM"]
S_MySQL[(MySQL DB<br/>EXISTING)]
S_Maxwell[Maxwell CDC<br/>EXISTING]
S_CDCPub[CDC Publisher<br/>NEW<br/>Port: 3001]
S_MySQL -->|binlog stream| S_Maxwell
S_Maxwell -->|stdout/JSON| S_CDCPub
end
subgraph PROD_VM["Production VM"]
P_MySQL[(MySQL DB<br/>EXISTING)]
P_Maxwell[Maxwell CDC<br/>EXISTING]
P_CDCPub[CDC Publisher<br/>NEW<br/>Port: 3001]
P_MySQL -->|binlog stream| P_Maxwell
P_Maxwell -->|stdout/JSON| P_CDCPub
end
subgraph CENTRAL_VM["Central VM - Event Processing Hub"]
subgraph STORAGE["Storage Layer"]
Redis[(Redis Stack<br/>NEW<br/>Ports: 6379, 8001<br/>- BullMQ Queues<br/>- Job State<br/>- Temp Data)]
Postgres[(PostgreSQL<br/>EXISTING<br/>Port: 5432<br/>- Notifications<br/>- Error Logs<br/>- Job History<br/>- Audit Trail)]
end
subgraph QUEUES["BullMQ Queues in Redis"]
Q1[cdc:dev]
Q2[cdc:staging]
Q3[cdc:prod]
Q4[oss:events]
Q5[jira:create]
Q6[jira:retry<br/>Dead Letter]
Q7[scheduler:oss-check]
end
subgraph PROCESSING["Processing Services"]
OSSListener[OSS Listener<br/>NEW<br/>Port: 3001<br/>- Scheduler Setup<br/>- Bucket Monitor]
Workers[BullMQ Workers<br/>NEW<br/>3 Replicas<br/>- CDC Worker<br/>- OSS Worker<br/>- Jira Worker<br/>- Scheduler Worker]
end
subgraph EXTERNAL_SERVICES["External Services"]
JiraService[Jira Service<br/>NEW<br/>Port: 3000<br/>- API Wrapper<br/>- Circuit Breaker<br/>- Rate Limiter]
LoggerService[Logger Service<br/>NEW<br/>Port: 3002<br/>- Error Handler<br/>- Audit Writer<br/>- Notification Creator]
end
subgraph API_LAYER["API & Dashboard"]
AdminAPI[Admin API<br/>NEW<br/>Port: 4000<br/>- REST Endpoints<br/>- SSE Stream<br/>- Bull Board UI]
VueDash[Vue Dashboard<br/>MODIFIED<br/>Port: 5173<br/>- Real-time Monitor<br/>- Job Inspector<br/>- Error Viewer]
end
OSS_Cloud[(Alibaba OSS<br/>All Buckets)]
Jira_Cloud[Jira Cloud API]
OSS_Cloud -.->|poll every 30s| OSSListener
OSSListener -->|create scheduled jobs| Q7
Q7 -->|trigger check| Workers
Workers -->|file changes detected| Q4
Redis --> QUEUES
QUEUES --> Workers
Workers -->|create ticket request| Q5
Q5 -->|process| Workers
Workers -->|HTTP POST| JiraService
JiraService -->|REST API| Jira_Cloud
Workers -.->|on error/failure| LoggerService
OSSListener -.->|on error| LoggerService
JiraService -.->|on error| LoggerService
LoggerService -->|write logs| Postgres
Redis -.->|queue metrics| AdminAPI
Postgres -.->|historical data| AdminAPI
AdminAPI -->|SSE events| VueDash
AdminAPI -->|REST API| VueDash
end
D_CDCPub -->|network: publish job| Q1
S_CDCPub -->|network: publish job| Q2
P_CDCPub -->|network: publish job| Q3
Internet((Users<br/>Browsers))
Internet -.->|HTTPS| VueDash
Internet -.->|HTTPS| AdminAPI
style DEV_VM fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style STAGING_VM fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style PROD_VM fill:#ffebee,stroke:#c62828,stroke-width:3px
style CENTRAL_VM fill:#e8f5e9,stroke:#388e3c,stroke-width:4px
style STORAGE fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style QUEUES fill:#ff6b6b,color:#fff
style PROCESSING fill:#4ecdc4,color:#fff
style EXTERNAL_SERVICES fill:#ffe66d,color:#000
style API_LAYER fill:#42b883,color:#fff
flowchart TD
subgraph SOURCES["Event Sources"]
DB1[Dev MySQL]
DB2[Staging MySQL]
DB3[Prod MySQL]
OSS[OSS Buckets<br/>All Environments]
end
subgraph PUBLISHERS["Event Publishers"]
PUB1[CDC Publisher<br/>Dev]
PUB2[CDC Publisher<br/>Staging]
PUB3[CDC Publisher<br/>Prod]
OSS_PUB[OSS Listener<br/>+ Scheduler]
end
subgraph QUEUE_LAYER["Queue Layer - Redis"]
direction TB
CDC_QUEUES[CDC Queues<br/>cdc:dev<br/>cdc:staging<br/>cdc:prod]
OSS_QUEUE[OSS Queue<br/>oss:events]
JIRA_QUEUE[Jira Queue<br/>jira:create]
DEAD_LETTER[Retry Queue<br/>jira:retry]
end
subgraph WORKERS["Worker Pool"]
direction LR
W1[CDC Workers<br/>3 instances]
W2[OSS Workers<br/>2 instances]
W3[Jira Workers<br/>2 instances]
end
subgraph ACTIONS["Actions"]
JIRA_SVC[Jira Service]
LOGGER[Logger Service]
JIRA_API[Jira Cloud]
PG[(PostgreSQL)]
end
subgraph MONITORING["Monitoring"]
ADMIN[Admin API]
DASH[Vue Dashboard]
end
DB1 --> PUB1
DB2 --> PUB2
DB3 --> PUB3
OSS --> OSS_PUB
PUB1 -->|publish job| CDC_QUEUES
PUB2 -->|publish job| CDC_QUEUES
PUB3 -->|publish job| CDC_QUEUES
OSS_PUB -->|publish job| OSS_QUEUE
CDC_QUEUES -->|consume| W1
OSS_QUEUE -->|consume| W2
W1 -->|transform event| JIRA_QUEUE
W2 -->|transform event| JIRA_QUEUE
JIRA_QUEUE -->|consume| W3
W3 -->|create ticket| JIRA_SVC
JIRA_SVC -->|API call| JIRA_API
W1 & W2 & W3 -.->|on error| LOGGER
JIRA_SVC -.->|on error| LOGGER
LOGGER -->|persist| PG
W3 -.->|max retries failed| DEAD_LETTER
QUEUE_LAYER -.->|metrics| ADMIN
PG -.->|logs| ADMIN
ADMIN -->|SSE| DASH
style SOURCES fill:#e3f2fd,stroke:#1976d2
style PUBLISHERS fill:#fff3e0,stroke:#f57c00
style QUEUE_LAYER fill:#ff6b6b,color:#fff
style WORKERS fill:#4ecdc4,color:#fff
style ACTIONS fill:#ffe66d,color:#000
style MONITORING fill:#42b883,color:#fff
graph TB
subgraph ENV["Environment VMs"]
Maxwell[Maxwell CDC]
CDCPub[CDC Publisher]
end
subgraph CENTRAL["Central VM"]
Redis[(Redis)]
PG[(PostgreSQL)]
OSSListen[OSS Listener]
Workers[BullMQ Workers]
Jira[Jira Service]
Logger[Logger Service]
Admin[Admin API]
Vue[Vue Dashboard]
end
Maxwell -->|1. Reads binlog| CDCPub
CDCPub -->|2. Publishes job| Redis
OSSListen -->|3. Schedules checks| Redis
Redis -->|4. Triggers scheduled job| Workers
Workers -->|5. Polls OSS| OSS_Cloud[(OSS)]
Workers -->|6. Publishes file change| Redis
Redis -->|7. Distributes jobs| Workers
Workers -->|8. Transforms to Jira request| Redis
Redis -->|9. Delivers to Jira worker| Workers
Workers -->|10. Calls API| Jira
Jira -->|11. Creates ticket| Jira_API[Jira Cloud]
Workers -.->|On error| Logger
Jira -.->|On error| Logger
Logger -.->|Writes| PG
Redis -.->|Metrics| Admin
PG -.->|Historical data| Admin
Admin -->|SSE/REST| Vue
User((User)) -.->|Views| Vue
style ENV fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style CENTRAL fill:#e8f5e9,stroke:#388e3c,stroke-width:3px
graph TB
subgraph HOT_STORAGE["Hot Storage - Redis (Fast Access)"]
direction TB
ActiveJobs[Active Jobs<br/>Currently Processing]
WaitingJobs[Waiting Jobs<br/>In Queue]
RecentComplete[Completed Jobs<br/>Last 100 per queue]
RecentFailed[Failed Jobs<br/>Last 1000 per queue]
JobMeta[Job Metadata<br/>Progress, State]
end
subgraph COLD_STORAGE["Cold Storage - PostgreSQL (Long-term)"]
direction TB
ExistingNotif[Notifications<br/>EXISTING TABLE]
ExistingErrors[Error Logs<br/>EXISTING TABLE]
JobFailures[Job Failures<br/>NEW TABLE<br/>Permanent failures<br/>with details]
AuditTrail[Audit Trail<br/>NEW TABLE<br/>All job events<br/>for compliance]
Statistics[Job Statistics<br/>NEW TABLE<br/>Daily aggregates]
end
Workers[BullMQ Workers]
Logger[Logger Service]
Admin[Admin API]
Dashboard[Vue Dashboard]
Workers -->|Read/Write| HOT_STORAGE
Workers -.->|On error| Logger
Logger -->|Write| COLD_STORAGE
HOT_STORAGE -.->|Auto-cleanup after<br/>24h-7d| Archive[Archive Process]
Archive -.->|Optional: Archive to| COLD_STORAGE
Dashboard -->|Recent data<br/>< 24 hours| HOT_STORAGE
Dashboard -->|Historical data<br/>> 24 hours| COLD_STORAGE
Admin -->|Query both| HOT_STORAGE
Admin -->|Query both| COLD_STORAGE
style HOT_STORAGE fill:#ff6b6b,color:#fff,stroke:#c62828,stroke-width:2px
style COLD_STORAGE fill:#336791,color:#fff,stroke:#1a237e,stroke-width:2px
stateDiagram-v2
[*] --> EventOccurs: Database change or OSS file change
EventOccurs --> Published: CDC Publisher or OSS Listener
Published --> InQueue: Job added to BullMQ queue
InQueue --> Picked: Worker picks up job
Picked --> Processing: Worker processes event
Processing --> TransformSuccess: Event transformed
TransformSuccess --> JiraQueue: Added to jira create queue
JiraQueue --> JiraProcessing: Jira worker picks up
JiraProcessing --> JiraAPICall: Call Jira Service
JiraAPICall --> TicketCreated: Success
JiraAPICall --> JiraError: API failure
JiraError --> Retry: Attempt < Max
Retry --> JiraQueue: Re-queued with backoff
JiraError --> LogFailure: Max attempts reached
LogFailure --> DeadLetter: Move to jira retry queue
LogFailure --> PostgreSQL: Log to database
DeadLetter --> ManualRetry: Manual intervention
ManualRetry --> JiraQueue
TicketCreated --> LogSuccess: Log event
LogSuccess --> PostgreSQL: Audit trail
PostgreSQL --> Dashboard: Display in UI
TicketCreated --> [*]
note right of InQueue
Redis Storage
Fast access
Automatic distribution
end note
note right of PostgreSQL
Long-term storage
Historical analysis
Compliance
end note
sequenceDiagram
participant DevVM as Dev VM
participant StagingVM as Staging VM
participant ProdVM as Prod VM
participant Redis as Central Redis
participant Workers as BullMQ Workers
participant Jira as Jira Service
participant Logger as Logger Service
participant PG as PostgreSQL
participant Dashboard as Vue Dashboard
participant User as User Browser
Note over DevVM,ProdVM: Event Generation (Parallel)
DevVM->>Redis: Publish job to cdc:dev
StagingVM->>Redis: Publish job to cdc:staging
ProdVM->>Redis: Publish job to cdc:prod
Note over Redis: Job Distribution
Redis->>Workers: Worker 1: Pick from cdc:dev
Redis->>Workers: Worker 2: Pick from cdc:staging
Redis->>Workers: Worker 3: Pick from cdc:prod
Note over Workers: Parallel Processing
Workers->>Redis: Add to jira:create queue
Redis->>Workers: Jira workers pick up jobs
Workers->>Jira: POST /tickets
alt Success
Jira-->>Workers: 201 Created
Workers->>Logger: Log success (optional)
Logger->>PG: INSERT audit trail
else Failure
Jira-->>Workers: 500 Error
Workers->>Logger: Log error
Logger->>PG: INSERT job_failures
Workers->>Redis: Re-queue with backoff
end
Note over Dashboard,User: Real-time Monitoring
Dashboard->>Redis: SSE: Subscribe to updates
Redis-->>Dashboard: Stream events
User->>Dashboard: Open dashboard
Dashboard->>Redis: GET recent jobs
Redis-->>Dashboard: Last 24h data
Dashboard->>PG: GET historical data
PG-->>Dashboard: Older data
Dashboard-->>User: Display unified view