docs
Architecture

System Architecture

WP AutoFlow is built on a Microservices-ready architecture using the Producer-Consumer pattern. This ensures the Dashboard remains responsive (UI) even while the system is performing heavy background tasks (Scraping/AI).

High-Level Overview

The system is composed of three main layers:

  1. The Server (Producer): Handles the API, Dashboard, and Scheduler.
  2. The Broker (Redis): Manages the Job Queue.
  3. The Worker (Consumer): Isolated process that executes the heavy lifting.

Core Components

1. The Server (Node.js / Express)

This is the entry point. It serves the React Frontend and runs the Scheduler (Cron).

  • Role: It checks the source sites periodically.
  • Action: When it finds a new post, it doesn't process it immediately. It simply adds a job to the Redis Queue and goes back to sleep.
  • Benefit: This keeps the memory usage low and the UI fast.

2. Redis & BullMQ

Redis acts as the central nervous system. It holds the state of every job.

  • Waiting: Jobs ready to be processed.
  • Active: Jobs currently being processed by a worker.
  • Delayed: Jobs scheduled for the future (e.g., "Retry in 5 minutes").
  • Failed: Jobs that encountered an error (allows for manual retry from Dashboard).

3. The Worker

This is where the magic happens. The worker is a separate process that "listens" to Redis.

  • Concurrency: You can configure how many posts to process in parallel (Default: 5).
  • Isolation: If a worker crashes on one specific tricky post, it doesn't take down the whole server.

4. MongoDB

Used for persistent storage of:

  • Site Configuration: URLs, prompts, schedules.
  • History Logs: What was published and when.
  • User Accounts: Admin authentication.

The Data Flow (Lifecycle)

When a job is processed, it follows a strict pipeline:

  1. Ingestion: Connects to the Source WP REST API (/wp-json/wp/v2/posts) and fetches the raw JSON.
  2. Sanitization: Removes source-specific shortcodes or scripts.
  3. Rewriting: Sends the content structure to the configured AI (OpenAI or DeepSeek) with the system prompt.
  4. Media Processing: Downloads the high-res image from the source URL -> Optimizes -> Uploads to Target Media Library.
  5. Publication: Creates the final post on the target WordPress site via API.
  6. Notification: Triggers Webhooks (if configured) and Pings IndexNow.

Scalability

Because of this architecture, WP AutoFlow scales horizontally.

  • Need more speed? You can increase the CONCURRENCY limit in the settings.
  • Too much load? You can run the Redis/Mongo instances on separate servers.

<Callout type="info"> Docker: In the default Docker setup, all these components run in their own containers but share the same internal network, offering a "Production-in-a-Box" experience. </Callout>

InstallationEnvironment (.env)