How to troubleshoot the following BACKUP DATABASE error:
Error 2270: Leaf Error (leaf01:3306): Backup for database `testdb_2` was aborted because of concurrent failover.
While high availability helps harden the cluster against downtime, if a node were to go offline while a backup is running, the backup would inevitably fail. The backup process requires that all leaves be in an online or replicating state while the backup is running regardless of if the environment is using high availability or not to complete successfully.
- If any leaf nodes are experiencing a hardware issue and need to be removed or replaced from the cluster, consult the Dealing with Cluster Failures documentation. Once the cluster is in a healthy state, try running the BACKUP DATABASE command once more.
There are two methods with which you can throttle the resources that backups use when they are executed. If backups are the root cause of leaf failovers, throttling the resources, namely CPU usage through resource pools and/or backup thread usage that backups use may help alleviate failovers due to backups exhausting the cluster's resources.
- Option 1 - Create Resource Pool for Backups
- Create a resource pool specifically for taking backups with CREATE RESOURCE POOL.
- After creating a resource pool for backups set
- Run the backup command with
OPTION (resource_pool = your_backup_pool). Consult the BACKUP DATABASE documentation for the complete backup syntax.
- Option 2 - Adjust
- Adjust the value of
backup_max_threadsto throttle backup thread usage. This variable controls the number of concurrent threads used to run backups on leaves. By default, each backup uses one thread per partition.
- For Example: Each backup uses one thread per partition. When we trigger a backup on a leaf with 4 partitions, there would be 4 concurrent threads running for a backup on a particular leaf node. To control the threads used by the backup, adjust the value of the
backup_max_threadsvariable to less than the number of partitions on a leaf node (i.e., setting it to 2 would throttle backup threads to be a maximum of 2 on the leaf node instead of 4.)
- You must be using SingleStore version 7.1.19, version 7.3.8, or higher to toggle the
- Adjust the value of