{question}
Why sdb-admin list-nodes command doesn't show the Child Aggregator even when the CA is added to the cluster?
{question}
{answer}
When performing the sdb-admin list-nodes operation, the command output does not show the Child Aggregator even when the CA is added to the cluster.
sdb-admin list-nodes output:
+------------+--------+------+------+---------------+--------------+---------+----------------+--------------------+--------------+
| MemSQL ID | Role | Host | Port | Process State | Connectable? | Version | Recovery State | Availability Group | Bind Address |
+------------+--------+------+------+---------------+--------------+---------+----------------+--------------------+--------------+
| 995CDA1259 | Master | master-ip-1.0.1.1 | 3306 | Running | True | 7.8.13 | Online | | 0.0.0.0 |
| F9901EC2FC | Leaf | leaf1-ip-10.0.1.1 | 3306 | Running | True | 7.8.13 | Online | 1 | 0.0.0.0 |
| 0592A42A91 | Leaf | leaf2-ip-10.0.2.1 | 3306 | Running | True | 7.8.13 | Online | 1 | 0.0.0.0 |
+------------+--------+------+------+---------------+--------------+---------+----------------+--------------------+--------------+
The above result shows that the Toolbox didn't identify the CA.
SHOW AGGREGATORS output:
When we execute show aggregators via SQL client, we can find the CA existing on the cluster.
singlestore> show aggregators;
+--------------+------+--------+--------------------+------------------------------+-------------------+--------+
| Host | Port | State | Opened_Connections | Average_Roundtrip_Latency_ms | Master_Aggregator | NodeId |
+--------------+------+--------+--------------------+------------------------------+-------------------+--------+
| 10.1.0.55 | 3306 | online | 3 | NULL | 1 | 1 |
| 10.2.161.145 | 3306 | online | 3 | 0.395 | 0 | 7 |
+--------------+------+--------+--------------------+------------------------------+-------------------+--------+
During the registration of nodes using the Toolbox, either the hostname or the IP address needs to be specified to register the host. If the hostname is used and there is a conflict with the hostname, an error of the following type is displayed:
failed to register host: host already exists:
host CA has the same ssh host keys as MA, toolbox doesn't support registering the same host twice.
Please confirm that those hosts are indeed different and are not using identical SSH host keys; see https://docs.singlestore.com/memsql-deploy-redir/duplicate-host-fingerprints for more info.
If you understand the risks of duplicate hosts and are sure that the above hosts are not the same then you may re-run this command with the --allow-duplicate-host-fingerprints flag.
As stated above, this indicates multiple hosts with the same ssh host keys as the host named "master-ip-1.0.1.1". This suggests the '/etc/hosts' file has duplicate hosts with different hostnames. However, this can be overridden by specifying the --allow-duplicate-host-fingerprints flag.
If the above flag is used, the host key validation is skipped, and the host is registered. Hence, even though the hostnames are different, the underlying IP address of the hosts is the same resulting in a conflict and an incorrect output of the sdb-admin list-nodes command.
The following example can be referred to for a better understanding of this issue:
- The hostnames of these particular hosts have been configured in the '/etc/hosts' as below:
~$ cat /etc/hosts
10.1.0.55 master-ip-1.0.1.1
10.1.0.55 child-ip-10.0.1.15
10.2.131.12 leaf1-ip-10.0.1.1
10.2.122.11 leaf2-ip-10.0.2.1 - It can be observed that for the hostnames of master and child aggregators, the IP address is the same. This results in a conflict while registering the hosts.
- During the registration '--allow-duplicate-host-fingerprints' flag is used, which skips the host validation. Hence, the output of the sdb-admin list-nodes and SHOW AGGREGATORS does not match.
To solve this issue, appropriate IP addresses must be assigned to the hostnames in the host file.
- Edit the /etc/hosts file and update the IP of the particular host. In the above example, IP needed to be updated for CA, which is 10.2.161.145.
~$ cat /etc/hosts
10.1.0.55 master-ip-1.0.1.1
10.2.161.145 child-ip-10.0.1.15
10.2.131.12 leaf1-ip-10.0.1.1
10.2.122.11 leaf2-ip-10.0.2.1
{answer}