docs: add binary api trace replay details
Folks need to know that they MUST carefully control the set of plugins
to avoid feeding messages to the wrong binary API message handlers.
@@ -182,6 +182,80 @@ and `src/vlibapi/api_shared.c
<https://docs.fd.io/vpp/18.11/d6/dd1/api__shared_8c.html>`_. See also
the debug CLI command "api trace"
+API trace replay caveats
+________________________
+
+The vpp instance which replays a binary API trace must have the same
+message-ID numbering space as the vpp instance which captured the
+trace. The replay instance **must** load the same set of plugins as
+the capture instance. Otherwise, API messages will be processed by the
+**wrong** API message handlers!
+
+Always start vpp with command-line arguments which include an
+"api-trace on" stanza, so vpp will start tracing binary API messages
+from the beginning:
+
+.. code-block:: console
+
+ api-trace {
+ on
+ }
+
+Given a binary api trace in /tmp/api_trace, do the following to work
+out the set of plugins:
+
+.. code-block:: console
+
+ DBGvpp# api trace custom-dump /tmp/api_trace
+ vl_api_trace_plugin_msg_ids: abf_54307ba2 first 846 last 855
+ vl_api_trace_plugin_msg_ids: acl_0d7265b0 first 856 last 893
+ vl_api_trace_plugin_msg_ids: cdp_8f707b96 first 894 last 895
+ vl_api_trace_plugin_msg_ids: flowprobe_f2f0286c first 898 last 901
+ <etc>
+
+Here, we see the "abf," "acl," "cdp," and "flowprobe" plugins. Use the
+list of plugins to construct a matching "plugins" command-line argument
+stanza:
+
+.. code-block:: console
+
+ plugins {
+ ## Disable all plugins, selectively enable specific plugins
+ plugin default { disable }
+ plugin abf_plugin.so { enable }
+ plugin acl_plugin.so { enable }
+ plugin cdp_plugin.so { enable }
+ plugin flowprobe_plugin.so { enable }
+ }
+
+To begin with, use the same vpp image that captured a trace to replay
+it. It's perfectly fair to rebuild the vpp replay instance, to add
+scaffolding to facilitate setting gdb breakpoints on complex
+conditions or similar.
+
+API trace interface issues
+__________________________
+
+Along the same lines, it may be necessary to manufacture [simulated]
+physical interfaces so that an API trace will replay correctly. "show
+interface" on the trace origin system can help. An API trace
+"custom-dump" as shown above may make it obvious how many loopback
+interfaces to create. If you see vhost interfaces being created and
+then configured, the first such configuration message in the trace
+will tell you how many physical interfaces were involved.
+
+.. code-block:: console
+
+ SCRIPT: create_vhost_user_if socket /tmp/foosock server
+ SCRIPT: sw_interface_set_flags sw_if_index 3 admin-up
+
+In this case, it's fair to guess that one needs to create two loopback
+interfaces to "help" the trace replay correctly.
+
+These issues can be mitigated to a certain extent by replaying the
+trace on the system which created it, but in a field debug case that's
+not a realistic.
+
Client connection details
_________________________
