{question}
Can pipeline decompress compressed file?
{question}
{answer}
The answer to this question is YES. The pipeline can decompress gzip files. By default, if the pipeline receives GZIP files (files with .gz extension), they will get decompressed and processed.
The disable_gunzip
option disables the decompression of files with the .gz extension. If this option is disabled or missing, files with the .gz extension will be decompressed. Click here to learn about the options with CREATE PIPELINE.
Let's look into an example below using the S3 pipeline, created a simple text file like below:
$ cat pipeline_test.txt
Hi this is test for pipeline decompression
Compressed the file with gzip:
$ gzip -k pipeline_test.txt
$ ls -lrt
-rw-r--r-- 1 admin 80 Sep 6 10:57 pipeline_test.txt.gz
Uploaded the file to the S3 bucket:
Creating and starting an S3 pipeline,
mysql> CREATE PIPELINE mytestpipeline AS
-> LOAD DATA S3 's2dbtestbucket'
-> CREDENTIALS '{"aws_access_key_id": "encrypted", "aws_secret_access_key": "encrypted","aws_session_token":"encrypted"}'
-> INTO TABLE `test`;
Query OK, 0 rows affected (0.18 sec)
mysql> START PIPELINE mytestpipeline;
Query OK, 0 rows affected (0.01 sec)
mysql> select * from test;
+---------------------------------------------+
| col |
+---------------------------------------------+
| Hi this is test for pipeline decompression |
+---------------------------------------------+
1 row in set (0.04 sec)
The Example above clearly shows that the pipeline can decompress .gz files and loads the data into the tables.
Note: Pipeline cannot decompress files with extensions .7z , .zip , .bz2 , .tar.
{answer}