Too many queries are queued for execution

{question}

How to troubleshoot the following error:

Error 1958: Too many queries are queued for execution, exceeding the limit of 100. You may try running the query again later. Consider reducing cluster load. The limit can be configured with the workload_management_max_queue_depth variable.

{question}

{answer}

If you have more thanworkload_management_max_queue_depth amount of queries in the queue at once, we raise an error ER_TOO_MANY_QUEUED_QUERIES.

The purpose is to alert the user in situations when the cluster is abnormally overloaded, and queries are piling up in the queue faster than they can be processed. An example is if a clustering operation with a long-held lock is blocking progress on the queries.

It is also necessary to limit the queue depth because each queued query occupies a thread; therefore, the default max_connection_threadsof 192 is a bit higher than the default workload_management_max_queue_depthof 100.

However, in many workloads, it is normal to be running a large number of fully distributed queries at once. Then you can:

Raise max_connection_threads to at least 25% higher than the desired maximum queue depth.

• You will get an error if you try to set  workload_management_max_queue_depth to greater than 80% of max_connection_threads. You need to reserve threads for non-fully-distributed queries that are also running at the same time.
  
  
• max_connection_threads cannot be set higher than 8192.

Raise workload_management_max_queue_depth to the desired maximum queue depth.

• If you increase the value of workload_management_max_queue_depthit is highly recommended that this value be increased gradually and tested to see what threshold works best with the workload. Alternatively, see if the workload is doing more work than expected, causing the workload management queue threshold to be reached.

More information on tuning the workload management can be found in the Using the Workload Manager section of the documentation.

{answer}