Articles in this section

Replicating MongoDB® Collections via AWS PrivateLink complete guide

{question}

How to replicate MongoDB® Collections via AWS PrivateLink: Complete Guide?

{question}

{answer}

  • Deploy a MongoDB Atlas M10 or above cluster on MongoDB Cloud. These are all paid clusters.

  • After the cluster is deployed and ready, go to Network Access > Private Endpoint > Add Private Endpoint(This service is not available for free clusters)

  • After this a Create Private Endpoint setup opens up please follow the steps below -:
  1. Select the cloud provider(AWS)
  2. Choose Service Endpoint Region (choose region where Helios cluster is deployed)
  3. Wait till the Atlas Endpoint Service shows as Ready as shown in below screenshot.
    Please also request the VPC ID & Subnet IDs from SingleStore Support team where the Helios cluster is deployed.

     

  4. Since the Atlas Endpoint Service shows as Ready copy the Atlas Endpoint Service as shown in example below.

  5. Create an Outbound Private Connection in Helios by selecting the following options
    Create Connection     > Endpoint (SingleStoreDB Endpoint) > Workspaces (select your workspace) > Service Name (Put Atlas Endpoint Service here) > Create Connection

    The connection will be created it might take few seconds for connection to go from pending to active state.

  6. Once the Helios Outbound connection is created click on Actions > three dots > view Connections > copy the VPC Endpoint ID from the VPC Endpoint Service Name as highlighted below in red.
  7. On the MongoDB Atlas Private endpoint side complete the connection by clicking edit > provide the VPC ID & Subnet ID from Helios side provided by SRE team > enter the VPC Endpoint ID we got from Helios outbound connection > click on create connection.

    Once the connection is created Atlas will take few minutes to change the status of endpoint from pending to available.

 

  • After this login to Helios and go to SQL editor and please use the following commands to Create a link to the MongoDB hosts using the "mongodb.connection.string".

  • Please use the following command to create the infer pipelines -:

First using the link

## Creating the link
CREATE LINK mongoatlas AS MONGODB
CONFIG '{"mongodb.connection.string": "mongodb+srv://cluster0.rrb2vgy.mongodb.net/",
"collection.include.list": "source_db.source_collection",
"mongodb.ssl.enabled":"true",
"mongodb.authsource":"admin",
"mongodb.members.auto.discover": "false"}'
CREDENTIALS '{"mongodb.user":"xxxx",
"mongodb.password":"xxxx"}';

## creating the pipelines and procedure using infer statement automatically even if the table exists
CREATE TABLE IF NOT EXISTS subscribers 
AS INFER PIPELINE AS LOAD DATA 
LINK mongoatlas "source_db.source_collection"
FORMAT AVRO;

 

OR

 

Second without using link with MONGODB command

## Creating the pipelines and procedure using infer statement automatically even if the table exists using mongodb and config details


CREATE TABLE IF NOT EXISTS subscribers 
AS INFER PIPELINE AS LOAD DATA 
MONGODB "source_db.source_collection"
CONFIG '{"mongodb.connection.string": "mongodb+srv://cluster0.rrb2vgy.mongodb.net/",
"collection.include.list": "source_db.source_collection",
"mongodb.ssl.enabled":"true",
"mongodb.authsource":"admin",
"mongodb.members.auto.discover": "false"}'
CREDENTIALS '{"mongodb.user":"xxxx",
"mongodb.password":"xxxx"}'
FORMAT AVRO;

Note: Please use the connection string from the mongodb Atlas using the Private Endpoint > Proceed to connect > choose the string from mongosh or compass option.

 

Compass (The connection string will now have a `pl` in it which means private link url)

 

Shell (The connection string will now have a `pl` in it which means private link url)

 

  • Check the pipelines and start the stopped pipelines since by default they are stopped.

-- Checking if the pipelines are created using infer pipeline command 
show pipelines;

-- Starting the stopped pipelines
start all pipelines;

  • To verify the batches are being inserted properly please use the command below

select * from information_schema.pipelines_batches_summary;
  • After this please check that the tables are created using the following command
-- Checking if the tables are created
show tables;
  • To view the data inserted properly in readable format use the following command
-- Verifying if the data is properly present
SELECT _id :> JSON , _more :> JSON FROM <table_name>;

For example -:

{answer}

 
{answer}

Related articles