perfino is designed to be able to monitor hundreds of VMs with a single server:
You can organize your VMs into hierarchical groups to keep track of related VMs. The perfino UI shows you aggregate data for each VM group and configuration settings are applied to to entire groups rather than to single VMs.
In an example scenario, you might put your frontend VMs into one group and the backend VMs into another group and configure your business transactions differently for both groups. Inside each group, you would add nested groups for first generation and second generation machines, so you could set the warning thresholds differently.
The group of a VM is specified in the -javagent VM parameter for the monitored VMs, so it can easily be changed and new groups can be created on the fly.
Single VMs can be given names, so you can easily find them in the perfino UI and keep track of their historical data across restarts and machine changes.
Like the VM group, the name of a VM is specified in the -javagent VM parameter for the monitored VMs. Adding new VMs does not require any configuration in the perfino UI.
If you have a number of exchangeable VMs with no separate identity, you can set up a VM pool. Instead of specifying group and name in the -javagent VM parameter for the monitored VM, you specify a pool name. In this way, you can use the identical -javagent VM parameter for multiple VMs.
VM groups, VM names and VM pools are set in the -javagent VM parameter for the monitored VM. The perfino collector creates groups and pools as required. You can then edit those groups or pools in the perfino UI to apply your own configuration. If not, the configuration is inherited from the parent group.
When you change recording settings, you do not have to restart the monitored VMs. The new recording settings will be applied on the fly.
perfino instruments selected methods to obtain a maximum of information with a minimum of overhead:
perfino uses a pure Java agent which intercepts class loading to instrument selected methods. Except for the profiling integration, the perfino agent does not load native libraries which could introduce overhead or a general stability risk.
Transactions with the same name are grouped in perfino, for example in the dashboard or in the call tree.
Transactions names are constructed by concatenating a number of name parts. The available parts depend on the transaction type. For example, for web transactions, you can use URL segments, query parameters or even capture regular expressions groups from the URL. For method interceptions, the class and method names as well as the method parameters or arbitrary getter chains are available. Together with fixed text snippets, you can construct intuitive names that capture the essence of an operation.
perfino does not only record top-level business transactions, but each business transaction can have nested transactions, up to an arbitrary depth. The amount of nested transactions depends on your transaction definitions.
For example, a business transaction may created by an URL invocation. Within that transaction, several EJB transactions could recorded. Furthermore, inside those transactions, several POJO transactions would be measured.
When a nested transaction has the same name as the parent transaction, it is not recorded separately. In addition, perfino offers a range of more sophisticated settings to limit nested transactions. You can group certain transactions, avoid nested transactions from the same transaction definition or transaction type, or exclude further nested transactions altogether.
These options allow you to adjust the level of detail that you see in the perfino and exclude transactions that do not add new information.
perfino records all nested transactions into a cumulated call tree. Transactions with the same name are added to the same node in the call tree. perfino keeps track of execution times and invocation counts for each call tree node. The call tree is a view in the perfino UI where you can expand nested transactions.
From the call tree, perfino calculates hot spots. Hot spots are those transactions where most of the time is spent. Each hot spot has a list of cumulated backtraces that show in what ways the hot spot transaction has been called. When looking at the hot spots view in the perfino UI, you can quickly find bottlenecks and their causes.
The smallest resolution for which a call tree is available is one minute. Other available time ranges are 10 minutes, one hour and one day. Each of these intervals can be compared to previous intervals.
In the call tree and hot spot views, you can either compare the entire tree or show a time line for selected transactions.
Call tree data of transactions is not discarded, it is only aggregated to larger time ranges. For example, the one minute granularity is only available for one day, but the one hour granularity is stored for 90 days. The one day granularity is kept indefinitely.
In this way, the size of the database eventually grows at just a small linear rate.
perfino detects business transactions from the following subsystems:
perfino can intercept URL invocations in all application servers that implement the Java servlet API. URL segments, query parameters and regular expression groups can be used to build the transaction name. You can configure different settings for different URLs by filtering with wildcards or regular expressions.
In addition to HTTP server transactions, HTTP client requests are measured by perfino. The following implementations are supported:
EJBs (3.0+) from all applications servers can be intercepted by perfino. All invocations of a public EJB method create a business transaction. For transaction naming, the class and method names are available, as well as arbitrary getter chains on the EJB instance. You can filter EJBs by packages or type (stateless, stateful or message-driven) and configure different settings for them.
Public methods in Spring beans annotated with @Component, @Controller, @Repository or @Server are intercepted by perfino and business transactions are created. For transaction naming, the class and method names are available, as well as arbitrary getter chains on the EJB instance. You can filter Spring beans by packages or annotation and configure different settings for them.
Perfino can create business transactions for each incoming RMI call. You can filter remote interfaces by packages and assign different settings to them. For transaction naming, class and method names are available.
perfino offers several ways to define custom business transactions:
If you use a framework that uses annotations to mark entry points or important methods, you can tell perfino to create business transactions for those methods.
Both annotations that are used on methods and annotations that are used on classes are supported. For classes, all public method invocations create business transactions. Optionally, you can choose to intercept methods in derived classes as well. perfino offers several options to restrict the set of intercepted methods in that case.
You can configure arbitrary methods or classes for which perfino should create business transactions. Classes and methods can conveniently be selected from the connected VMs or from uploaded JAR, WAR or EAR archives.
For classes, all public methods create business transactions. If you chose to intercept derived classes as well, perfino lets you filter derived classes by package names and offers several options to restrict the set of intercepted methods.
For method interceptions, you can use the method parameters for transaction naming, or an arbitrary getter chain on their values. Class and method names as well as getter chains on the instance are also available for transaction naming.
With the same features as the "Pojo invocations" above, the DevOps annotations can be used to define business transactions directly in your own code. Perfino annotations have class retention and are not visible at runtime, so they can be safely added to your project.
The @MethodTransaction and @ClassTransaction annotations designate methods or classes where business transactions should be measured. Transaction naming and inheritance options are fully configured with the annotation parameters.
Policies are frequently changed in operation and are configured in the perfino UI.
perfino can track transactions across VM boundaries:
An invocation from an EJB client to an EJB server is tracked by perfino if both VMs are monitored by perfino and the EJB container is one of the following:
A HTTP request is tracked if both the calling and the serving VM are monitored and one of the following HTTP clients are used:
A web service call is tracked if both the calling and the serving VM are monitored and one of the following web service implementations is used:
RMI calls are tracked, if both the calling and the serving VM are monitored.
The call graph view in the perfino UI shows all interactions from tracked inter-VM calls as well as database calls. In the call graph, you can inspect transactions for selected nodes as well as for selected edges.
Cross-VM tracking is also available in the call tree and in method-level sampling.
perfino offers a number of ways to identify problematic business transactions:
Each transaction definition offers two separate configurable thresholds for identifying slow transactions: "slow" and "very slow". These categories are the basis for further actions and are displayed in all transaction views.
Transactions that do not return after a long time are classified as "overdue". Overdue transactions have their own view in perfino and are a critical condition for firing triggers.
perfino detects errors in transactions in a variety of ways: For web transactions, error status codes are detected. For transactions based on method invocations, runtime exceptions, checked exceptions and errors can optionally mark a transaction as an error.
perfino can intercept selected logging systems and check if an error or a warning has been logged. The following logging systems are supported:
The call tree is split for policy violations (slow, very slow, overdue or error) so you can see the measurements for those transactions as well as their nested transactions in isolation.
If slow, very slow or overdue transactions are detected, perfino can automatically start method-level sampling, so you can investigate the cause of an unexpected slowness without having to bring in a profiler.
Policy definitions are optional for transaction definitions. If a number of transaction definitions require the same policies, you don't have to define them each time, but you can handle them with a "catch-all" entry.
If a single transaction definition results in transactions that require different policies depending on the transaction name, you can add policy specializations to the transaction definition.
Database operations are measured with specific probes for each database type:
JDBC statements are measured independently of the used JDBC driver. Prepared statements are shown without their parameters and optionally unprepared statements can be resolved as well.
JPA/Hibernate queries and entity operations are measured by perfino. The following implementations are supported:
For each JPA/Hibernate operation, the associated JDBC statements are shown.
The following NoSQL databases are supported:
perfino tries to remove values that change frequently, so the hot spots are more significant.
The "probe hot spots" views show the database operations that take most time. Each operation can be expanded to show the cumulated backtraces of business transactions that called the hot spot.
If a business transaction is slow due to a database call, you can see it immediately in the call tree. For each database type where at least one operation was recorded, a child node is added to a business transaction.
Under special circumstances, perfino can record single-threaded method-level call trees on a per-transaction basis:
Transactions are sampled periodically, the period is configurable in the policy settings. In that way, you always have method-level data for recent transactions as a baseline when looking at method-level data from slow transactions.
In the case of certain policy violations (slow, very slow and overdue transactions), perfino can start sampling automatically, as soon as the configured threshold is exceeded. Method-level sampling will help you to find non-database related causes for slow transactions.
perfino can monitor page load times in the browser and correlate them with business transactions in the server:
Even if server response times are acceptable, the page load times in browsers may be not, especially if content is loaded from different sites. perfino can measure page load times for web transactions. For each transaction definition you can specify the percentage for which this kind of measurement should be performed.
End user experience monitoring is supported for the following servers:
perfino knows which business transactions on the server correspond to which page load times. In the "End user monitoring" view, business transactions and their reported page load times are shown and an overhead figure is calculated.
perfino records telemetries from a wide range of scalar data sources:
perfino shows the following VM telemetries:
perfino shows telemetries for the following monitoring statistics:
perfino shows telemetries for the following probe measurements:
For each VM group, custom telemetries can be defined that poll numeric attribute values of MBeans. MBean servers do not have to be exposed with a JMX connection server for this to work, the perfino agent will pick up all MBean servers in the JVM. Each telemetry can be configured with a number of data lines that can optionally by stacked into an area graph. The unit of the measurements, a scaling factory as well as the behavior when aggregating multiple VMs can be configured.
Devops telemetries can be created by annotating a static method with a numeric return value with the @CustomTelemetry annotation. Such methods will be sampled by perfino and telemetries will be created in the perfino UI.
Telemetry data is never discarded, but just aggregated to larger time intervals. For example, the 10 second resolution data is only kept for 6 hours, the aggregated 2 minute resolution data is kept for 10 days. Even further aggregation yields 3 hour resolution data which is kept for one year. At the end, the 12 hour resolution data is kept indefinitely.
For telemetries, acceptable ranges can be defined and monitored:
For each VM group, thresholds can be defined that determine the acceptable limits for any kind of telemetry. You can define optional lower and upper bounds.
To avoid an excessive number of threshold violations, you can configure a minimum time for a threshold violation as well as an inhibition time during which no further threshold violations will be detected.
Threshold violations are counted for each VM group and bubble up to the "All VMs" groups. They do not have directly associated actions, but are the basis for threshold triggers. The threshold violation data is shown in the dashboard and as historical data in the "VM data views".
For policy and threshold violations, a list of actions can be executed:
Policy triggers are fired if a number policy violations occurs in a defined period of time. The relevant transactions for a policy trigger can be filtered with wildcards or regular expressions. An inhibition time can be set to prevent duplicate trigger firings within a certain period.
Threshold triggers react to threshold violations. When a number of threshold violations are fired in a certain period of time, the threshold trigger fires. During the configured inhibition time, no further trigger firings can occur.
With connection count triggers you can ensure that a minimum number of VMs are connected to the perfino collector. The trigger can be armed after the threshold has been reached or after a fixed delay.
Several trigger actions can be used to create notifications on different levels. In escalating order, they are:
A trigger can start more extensive data collection with the following trigger actions:
Alerts are created by triggers with the "Create dashboard alert" trigger action and are overlaid over the transaction time line in the dashboard. Alerts can have different severities with different associated colors.
In addition to the dashboard, alerts are available as historical data in the "VM data views".
perfino offers built-in memory monitoring with very low overhead:
perfino includes a low-overhead memory monitoring technique that does not rely on high-overhead memory snapshots. This yields a list of loaded classes and their instance counts.
perfino records a low overhead memory snapshot periodically, so you always have a baseline for historical comparisons. In addition, you can trigger memory snapshots for single VMs in the "Memory" view.
Memory snapshots can be compared in perfino. You can either compare the entire table or show a time line of the instance counts or shallow sizes of previous snapshots.
When searching for memory leaks, you need a full memory snapshot of the VM. Apart from the trigger action that takes an HPROF memory snapshot, the "VMs" view offers an action for manually taking an HPROF snapshot for a selected VM.
HPROF snapshots can be opened in JProfiler or other Java profilers and memory analyzers. Snapshots are stored in perfino until you delete them from the "Snapshots" view.
With perfino, you can leverage JMX to inspect and control your applications:
Usually, you have to start a JMX connector server to access MBeans from outside a JVM. With perfino, this is not necessary, because the agent accesses MBeans directly. In this way, you avoid the security implications of having to open a JMX port to the outside.
Beside simple types like integers and strings, many MBeans use JMX Open Types with composite and tabular data. perfino shows arrays, composite types and tabular types in a single tree table. Tabular types with simple keys and single values are shown in a compact transposed form, so that maps from MXBeans, like the system properties in the "Runtime" MBean can be viewed as a property list.
Writable MBean attributes with simple types or their array types can be changed from perfino.
MBean operations with simple parameter types or simple array parameter types can be invoked from perfino. The return value is shown in a separate window and displays all MBean Open Types in a tree table.
In the recording settings, telemetries can be defined that poll numeric MBean values. An MBean browser helps in finding the desired MBean and attribute. Monitoring values from an attribute path that crosses composite or tabular data is supported as well.
perfino integrates with JProfiler to facilitate more extensive data collection in case of need:
By default, perfino does not load native JVMTI libraries. However, in case of need, perfino can load JProfiler agent libraries and prepare a JVM so that JProfiler can attach to it remotely.
All you need to do to enable this integration is to extract any version of JProfiler next to the perfino agent JAR file. After that, you can switch a VM to profiling directly in the "VMs" view.
For full method sampling, perfino can activate a specially controlled JVMTI profile where only very few capabilities of the profiling interface are requested. This reduces the overhead and stability risk to an absolute minimum.
For this sampling mode, perfino monitoring and JVMTI method sampling are concurrently available.
The sampling results are recorded as a JProfiler snapshot and delivered to your perfino inbox.
JDK Flight Recorder (JFR) snapshots can be recorded from the VMs view. This is a non-intrusive way to collect more low-level profiling data for a selected JVM. A corresponding trigger action is also available.
perfino can be deployed and configured interactively or in unattended mode:
perfino offers installers for Windows, macOS and Linux/Unix. A service/daemon is set up automatically by the installers. All important connectivity settings for the perfino server are configured directly in the installer.
After the installer completes, you can open your browser (by default on localhost:8020) to complete the installation wizard in the perfino UI.
For deployments in the cloud, perfino can be installed without user interaction. All relevant configuration files can be placed next to the installer and are applied automatically.
When you update the perfino server, you do not have to distribute new perfino monitoring agents to all your monitored VMs. perfino is backwards compatible with older monitoring agents and sends the new agent files to the monitored VMs where they are used when the VM is restarted the next time.
The entire configuration of perfino can be exported to a single file from the perfino UI. Apart from backup purposes, this file enables you to install the perfino server in unattended mode.
perfino comes with its own web server. By default, you don't have to bring in your own application server to deploy perfino.
However, for networks topologies with a DMZ, the perfino collector and the UI may not run in the same network. in that case, you can deploy the perfino UI as a WAR file into another servlet container.
perfino users can have one of three access levels: