Manage message queues

You can manage message queues from the command line using cron jobs or an external process manager to ensure that consumers are retrieving messages.

Process management

Cron jobs are the default mechanism to restart consumers. Processes started by cron consume the specified number of messages and then terminate. Re-running cron restarts the consumer.

The following example shows the Magento crontab configuration for running consumers:

/app/code/Magento/MessageQueue/etc/crontab.xml

1
2
3
4
5
...
<job name="consumers_runner" instance="Magento\MessageQueue\Model\Cron\ConsumersRunner" method="run">
    <schedule>* * * * *</schedule>
</job>
...

How often you check message queues depends on your business logic and available system resources. In general, you’ll probably want to check for newly created customers and send welcome emails more frequently than a more resource intensive process (e.g., updating your catalog). You should define cron schedules according to your business needs.

It can be configured in Admin Panel Stores > Settings > Configuration > Advanced > System > Cron configuration options for group: consumers

See Configure and run cron for more information about using cron with Magento.

You can also use a process manager such as Supervisor to monitor the status of processes. The manager can use the command line to restart the processes as needed.

Configuration

Behavior by default

  • Cron job consumers_runner is enabled
  • Cron job consumers_runner runs all defined consumers
  • Each consumer processes 10000 messages and then terminates

If your Magento Commerce store is hosted on the Cloud platform, use the CRON_CONSUMERS_RUNNER to configure the consumers_runner cron job.

Specific configuration

Edit the /app/etc/env.php file to configure the cron job consumers_runner.

1
2
3
4
5
6
7
8
9
10
...
    'cron_consumers_runner' => array(
        'cron_run' => false,
        'max_messages' => 20000,
        'consumers' => array(
            'consumer1',
            'consumer2',
        )
    ),
...
  • cron_run - A boolean value that enables or disables the consumers_runner cron job (default = true).
  • max_messages - The maximum number of messages each consumer must process before terminating (default = 10000). Although we do not recommend it, you can use 0 to prevent the consumer from terminating.
  • consumers - An array of strings specifying which consumer(s) to run. An empty array runs all consumers.

View a list of available message queue consumers

To view a list of all consumers:

1
bin/magento queue:consumers:list

Start message queue consumers

To start message queue consumers:

1
bin/magento queue:consumers:start [--max-messages=<value>] [--batch-size=<value>] [--single-thread] [--area-code=<value>] <consumer_name>

The following table explains this command’s options, parameters, and values.

Parameter Value Required? Default Value
--max-messages=<value> The maximum number of messages to consume per invocation. If the number of queued messages is less than the specified max, the consumer polls for new messages until it has processed the max. If you don’t specify --max-messages, the process runs continuously. No 0
--consumers-wait-for-messages=<value> Specifies whether consumers should continue polling for messages if the number of processed messages is less than the max-messages value. The default value of 0 prevents stuck deployments caused by long delays in message queue processing. Set the value to 1 to allow consumers to wait for messages.
On Magento Commerce Cloud projects, this variable is set by default using the CONSUMERS_WAIT_FOR_MAX_MESSAGES deploy variable.
No 0
--batch-size=<value> The number of messages to consume per batch. If specified, messages in a queue are consumed in batches of <value> each. This option is applicable for the batch consumer only. If --batch-size is not defined, the batch consumer receives all available messages in a queue. No 0
--pid-file-path=<value> This option is deprecated. Use the single-thread option instead. No  
--single-thread This option prevents running multiple copies of a consumer simultaneously. No  
--area-code=<value> The area code preferred for consumer process. No global
<consumer_name> The consumer to start. Yes  

After consuming all available messages, the command terminates. You can run the command again manually or with a cron job. You can also run multiple instances of the magento queue:consumers:start command to process large message queues. For example, you can append & to the command to run it in the background, return to a prompt, and continue running commands:

1
bin/magento queue:consumers:start <consumer_name> &