Restore: Could not perform remote backup. NotFound: Not Found

{question}

How to troubleshoot Restore: Could not perform a remote backup. NotFound: Not Found?

How to troubleshoot the following RESTORE DATABASE error:

memsql> RESTORE DATABASE atp FROM S3 "memsql1-backup/db_backup_20210304_210000" CONFIG '{"region":"us-west-1","endpoint-url":"https://s3-us-west-1.amazonaws.com"}' CREDENTIALS '{"aws_access_key_id":"######","aws_secret_access_key":"########"}';
ERROR 2298 (HY000): Could not perform remote backup. NotFound: Not Found
status code: 404, request id: 5YZZ3ZPHB7K29YAP, host id: HaHapapalupa76WhatKey999PBA0GMnjA5dWAe6qjmJKJwi3qU7KgNzF7l+LCfv3DCYmEmgw=

{question}

{answer}

The 404 error message means that the backup files in the path specified in the restore command are not found for the restore process to use.

Troubleshooting Steps

• Use the following steps when you get the 404 error message: 
  • Check that you have access to the S3 backup files location from ALL cluster hosts (use was s3 ls path-to-the-backup-files). 
  • Check that the user running the restore process has read level access to the s3 path-to-the-backup-files.
  • Check that the path-to-the-backup includes the BACKUP_COMPLETE sentinel file generated once the backup completed successfully.
• If you restore a large database, consider increasing subprocess_io_idle_timeout_ms to a higher value when you get a time-out message while a restore process is running.

memsql-client: [Warning] Using a password on the command line interface can be insecure.
ERROR 1970 (HY000) at line 1: Leaf Error (hostname:3306): Subprocess timed out waiting for a response from the external endpoint. No stderr returned. 

• • The subprocess_io_idle_timeout_msis the maximum amount of time, in milliseconds, the engine waits for or retries a request before timing out and failing the backup when connecting to cloud providers. When you set this variable, its value is propagated to all nodes. The default value is 4 minutes; 4*60*1000=240000.

For example:

memsql> SET GLOBAL subprocess_io_idle_timeout_ms = 1200000;
Query OK, 0 rows affected (0.01 sec)

memsql> SELECT @@subprocess_io_idle_timeout_ms;
+---------------------------------+
| @@subprocess_io_idle_timeout_ms |
+---------------------------------+
| 1200000 |
+---------------------------------+
1 row in set (0.00 sec)

{answer}