spoof.dns configuration for Kafka pipelines

{question}

How does the spoof.dns configuration for Kafka pipelines work?

{question}

{answer}

This optional configuration allows Kafka pipelines to essentially work around a network configuration that is otherwise not feasible for the use case.

A common example of this is connecting from a Helios workspace to an organization’s AWS MSK environment via PrivateLink. The Kafka brokers will have some address but in reality the Kafka pipeline should actually use the PrivateLink endpoint.

The following example is only to demonstrate the concept of how this functionality works at a basic level.

• This is not a guide for actually configuring it with specific values in your environment – do not copy/paste from here.
• To simulate the unwanted configuration, /etc/hosts on each node has been modified to direct traffic for MyDemoKafkaBroker to 8.8.8.8 (a real host that is generally reachable, but it is not actually a Kafka broker).
  • This is similar to real scenarios where the Kafka Broker address may be reachable but really the pipeline should be using the PrivateLink endpoint instead.

This is how it looks when it does not work (without spoof.dns):

CREATE PIPELINE `NoSpoofPipeline` AS LOAD DATA KAFKA 'MyDemoKafkaBroker/test-topic' 
INTO TABLE `kafka_messages`;

ERROR 1933 (HY000): Cannot get source metadata for pipeline.  Could not fetch Kafka metadata. Are the Kafka brokers reachable from the Master Aggregator? Kafka error Local: Broker transport failure
Got 2 errors, first (Local: Broker transport failure): MyDemoKafkaBroker:9092/bootstrap: Connection setup timed out in state CONNECT (after 30030ms in state CONNECT)

This is how it looks when it does work (with spoof.dns):

CREATE PIPELINE `SpoofDnsPipeline` AS LOAD DATA KAFKA 'MyDemoKafkaBroker/test-topic'
CONFIG '{ "spoof.dns": {  "8.8.8.8":"MyPrivateLinkEndpoint" } }'
INTO TABLE `kafka_messages`;

Query OK, 0 rows affected (0.04 sec)

And this pipeline can be started and ingests data successfully.

• Why 8.8.8.8?
  • Because that is where MyDemoKafkaBroker points in this environment. This address should be wherever your broker address points, and it can have multiple entries (in JSON formatting).
• Why MyPrivateLinkEndpoint?
  • Because in this environment this points at where we actually want this Kafka pipeline to go. In the case of a Helios workspace connecting to AWS MSK via PrivateLink, this should be your Outbound PrivateLink endpoint.
• What about port numbers?
  • This functionality supports specifying port numbers (the test environment is using default ports for simplicity).

Related documentation

Kafka Configuration

{answer}