Cross-VM Monitoring

One of perfino's core features is the ability to track calls between monitored VMs. No configuration is necessary to enable cross-VM monitoring. As soon as an outgoing and incoming call of a supported subsystem can be matched between two monitored VMs, perfino shows it as a remote call in the call graph and in the call tree.

The following remote call mechanisms are supported:

Call graph

The natural representation for remote calls is the call graph, where each pair of VMs that participate in a remote call are connected by an edge. In addition, two other node types are present in the call graph that usually represent calls to and from external processes:

When a node or an edge is selected in the call graph, a detail panel is shown that holds several tabs with information about the associated transactions. The number and names of the tabs depend on the selected object:

Call tree

In the call tree, you can recognize transactions that cause a remote call by the "show remote call" link that is shown next to the transaction name. When following such a link, it is advisable to select the "Merge" option for the policy split, otherwise the numbers on the remote side may not match the numbers in the source VM. This is because policy splitting is not tracked across VMs. In any case, there may be small discrepancies due to calls that are close to period boundaries which may be assigned differently by source and target VMs.

Clicking on the "show remote call" link opens a new dialog that shows the call tree of transactions in the remote VMs that were started by the selected transaction. At the top-level, the call tree is partitioned into VM groups.

The call tree only shows the transaction in the target VMs. If there are further remote calls, you will see "show remote call" links, just like in the original call tree view. Clicking on those links replaces the call tree in the dialog and shows a Back In History button that can be used to return to the previous level. If the button disappears after clicking it, you are now looking at the first level.

It is also possible to move in the opposite direction and ask which remote calls in other VMs started transactions in the currently selected VMs. First, change the remote origins drop-down to "Split". By default, that drop down is set to "Merge" and you cannot see the remote origins. With the "Split" setting, "Called from ..." nodes appear at the top-level containing transactions that were started by remote calls from other monitored VMs. If you are looking at VM group or VM pool data, a split is performed for each VM group, but not for individual VMs. If the data is not granular enough, consider making your VM group structure more granular. If you are looking at the data for a single named VM, the split is performed for each remote VM.

Similarly to remote calls, clicking on the "show remote origins" link brings up a modal dialog. In the dialog, the count and duration of the selected transaction is attributed to the remote transactions that started it. The data is presented as hotspots with the innermost transactions from the originating VM at the top level. Analogous to the "remote calls" dialog, you can jump further back in a chain of remote calls when you see a "show remote origins" link next to a transaction. The Back In History button brings you closer to the original transaction.

Overload protection

If you monitor many VMs that all call each other, the number of the remote call sites grows with the square of the number of VMs. Remote call sites have a substantial overhead, since the transaction call trees have to be split for each remote call site. If too many distinct remote call sites are being created, perfino's overload protection mechanism prevents excessive resource usage.