Queue & Performance

WP AutoFlow utilizes Redis and BullMQ to manage background jobs. This architecture allows the system to decouple the "discovery" of content (Scheduler) from the "processing" of content (Worker).

This ensures that your server never hangs, even if you trigger an update for 50 sites simultaneously.

Processing Modes (Concurrency)

You can control how the worker processes jobs via the Concurrency setting. This determines how many posts are generated at the exact same time.

1. Sequential Mode (Concurrency = 1)

In this mode, the worker processes one post at a time. Job B only starts after Job A finishes.

• ✅ Best for: Low-resource servers (e.g., 1GB RAM VPS), strict rate-limiting on source sites, or debugging.
• ❌ Cons: Slower throughput.

2. Parallel Mode (Concurrency > 1)

The worker processes multiple posts simultaneously. If set to 5, five different articles will be scraped, rewritten, and uploaded at the same time.

• ✅ Best for: High-performance servers, processing large backlogs quickly.
• ❌ Cons: High CPU/RAM usage. Requires a stable database connection.

<Callout type="warning"> Resource Warning: Each concurrent job launches a headless browser (if Puppeteer is enabled) or heavy AI requests. We recommend 1 concurrent job per 1GB of RAM available on your server. </Callout>

Job Lifecycle

Every article processed goes through a specific state machine in Redis:

1. Waiting: The job is in the queue, waiting for a free worker slot.
2. Active: The worker is currently scraping or rewriting the content.
3. Completed: The post was successfully published to WordPress.
4. Failed: An error occurred (e.g., API timeout, AI error).
5. Delayed: The job failed but is scheduled to retry automatically after a backoff period.

Fault Tolerance & Retries

The queue system is designed to be resilient. Network blips happen, and APIs fail.

• Automatic Retries: If a job fails (e.g., OpenAI is down), the system does not discard it. It moves to a "Delayed" state and tries again automatically (Default: 3 attempts).
• Exponential Backoff: The time between retries increases (1min, 5min, 15min) to give the external service time to recover.

Managing the Queue

You can monitor and manage the queue directly from the Dashboard Home:

• View Counts: See how many jobs are Waiting vs. Active.
• Retry Failed: If jobs are in the "Failed" tab (e.g., due to wrong API Key), you can fix the issue and click "Retry All" to move them back to the Waiting queue.
• Clear Queue: If you made a mistake (e.g., queued the wrong source), you can flush the "Waiting" queue instantly.