Backup was aborted because of concurrent failover.

{question}

How to troubleshoot the following BACKUP DATABASE error:

Error 2270: Leaf Error (leaf01:3306): Backup for database `testdb_2` was aborted because of concurrent failover.

{question}

{answer}

While high availability helps harden the cluster against downtime, if a node were to go offline while a backup is running, the backup would inevitably fail. The backup process requires that all leaves be in an online or replicating state while the backup is running regardless of if the environment is using high availability or not to complete successfully.

Troubleshooting Steps

• If any leaf nodes are experiencing a hardware issue and need to be removed or replaced from the cluster, consult the Dealing with Cluster Failures documentation. Once the cluster is in a healthy state, try running the BACKUP DATABASE command once more.

Throttle Backups

There are two methods with which you can throttle the resources that backups use when they are executed. If backups are the root cause of leaf failovers, throttling the resources, namely CPU usage through resource pools and/or backup thread usage that backups use may help alleviate failovers due to backups exhausting the cluster's resources.

• Option 1 - Create Resource Pool for Backups
  • Create a resource pool specifically for taking backups with CREATE RESOURCE POOL.
  • After creating a resource pool for backups set resource_governor_cpu_limit_mode to Hard.
  • Run the backup command with OPTION (resource_pool = your_backup_pool). Consult the BACKUP DATABASE documentation for the complete backup syntax.
    
    
• Option 2 - Adjust backup_max_threads
  • Adjust the value of backup_max_threadsto throttle backup thread usage. This variable controls the number of concurrent threads used to run backups on leaves. By default, each backup uses one thread per partition.
  • For Example: Each backup uses one thread per partition. When we trigger a backup on a leaf with 4 partitions, there would be 4 concurrent threads running for a backup on a particular leaf node. To control the threads used by the backup, adjust the value of the backup_max_threads variable to less than the number of partitions on a leaf node (i.e., setting it to 2 would throttle backup threads to be a maximum of 2 on the leaf node instead of 4.)
  • You must be using SingleStore version 7.1.19, version 7.3.8, or higher to toggle the backup_max_threads variable.

{answer}