Clear Ops state

{question}

The Ops commands are not working; they appear to hang.

What should I do?

{question}

{answer}

This can usually be resolved by just restarting all Ops agents using the following steps:

• SSH into the host with the primary Ops agent (usually the same as the Master Aggregator)

• Stop all agents with the command memsql-ops agent-stop --all

  • If any of the agents does not stop, then SSH into the host with that agent and kill its process

• Start the primary Ops agent with the command memsql-ops start

• Start the rest of the Ops agents with the command memsql-ops agent-start --all

If you're still having issues with Ops commands not working, then you'll want to clear pending activities from the primary Ops agent's underlying sqlite database using the following steps:

Note: The topology.db is "metadata" about the cluster state from the Ops point of view as well as a history of all jobs run by the agents, which in some cases can get large and cause Ops to slow down.

• SSH into the host with the primary Ops agent (usually the same as the Master Aggregator)

• Stop the primary Ops agent with the command memsql-ops stop

• Locate the memsql-ops internal metadata database named topology.db in the primary Ops agent's directory and cd into it. By default, the path will be /var/lib/memsql-ops/data/

• Prior to manually altering it, take a backup of the topology.db file with the command cp topology.db topology.db_bak

• Access topology.db while in primary Ops agent's data directory with the command memsql-ops sqlite topology.db

• Once in the sqlite prompt, run the following commands:

  • delete from job_queue;
  • delete from intentions;
  • delete from intention_states;
  • delete from job_logs;
  • vacuum;
  • .exit

• Start the primary Ops agent with the command memsql-ops start

• Confirm the issue was resolved and MemSQL Ops commands are running normally, then delete the topology backup file rm topology.db_bak

To learn more about Ops to Toolbox migration, click here.

{answer}