perfino is designed to be able to monitor hundreds of VMs with a single server:

• VM groups for organizing VMs Show

  You can organize your VMs into hierarchical groups to keep track of related VMs. The perfino UI shows you aggregate data for each VM group and configuration settings are applied to to entire groups rather than to single VMs.

  In an example scenario, you might put your frontend VMs into one group and the backend VMs into another group and configure your business transactions differently for both groups. Inside each group, you would add nested groups for first generation and second generation machines, so you could set the warning thresholds differently.

  The group of a VM is specified in the -javagent VM parameter for the monitored VMs, so it can easily be changed and new groups can be created on the fly.

• Monitoring of named VMs Show

  Single VMs can be given names, so you can easily find them in the perfino UI and keep track of their historical data across restarts and machine changes.

  Like the VM group, the name of a VM is specified in the -javagent VM parameter for the monitored VMs. Adding new VMs does not require any configuration in the perfino UI.

• Monitoring of VM pools Show

  If you have a number of exchangeable VMs with no separate identity, you can set up a VM pool. Instead of specifying group and name in the -javagent VM parameter for the monitored VM, you specify a pool name. In this way, you can use the identical -javagent VM parameter for multiple VMs.

• No config required for connecting new VMs Show

  VM groups, VM names and VM pools are set in the -javagent VM parameter for the monitored VM. The perfino collector creates groups and pools as required. You can then edit those groups or pools in the perfino UI to apply your own configuration. If not, the configuration is inherited from the parent group.

• No restart required for configuration changes Show

  When you change recording settings, you do not have to restart the monitored VMs. The new recording settings will be applied on the fly.

perfino instruments selected methods to obtain a maximum of information with a minimum of overhead:

• Pure Java agent, no native libraries Show

  perfino uses a pure Java agent which intercepts class loading to instrument selected methods. Except for the profiling integration, the perfino agent does not load native libraries which could introduce overhead or a general stability risk.

• Flexible transaction naming Show

  Transactions with the same name are grouped in perfino, for example in the dashboard or in the call tree.

  Transactions names are constructed by concatenating a number of name parts. The available parts depend on the transaction type. For example, for web transactions, you can use URL segments, query parameters or even capture regular expressions groups from the URL. For method interceptions, the class and method names as well as the method parameters or arbitrary getter chains are available. Together with fixed text snippets, you can construct intuitive names that capture the essence of an operation.

• Nested business transactions Show

  perfino does not only record top-level business transactions, but each business transaction can have nested transactions, up to an arbitrary depth. The amount of nested transactions depends on your transaction definitions.

  For example, a business transaction may created by an URL invocation. Within that transaction, several EJB transactions could recorded. Furthermore, inside those transactions, several POJO transactions would be measured.

• Configurable reentry prevention for nested transactions Show

  When a nested transaction has the same name as the parent transaction, it is not recorded separately. In addition, perfino offers a range of more sophisticated settings to limit nested transactions. You can group certain transactions, avoid nested transactions from the same transaction definition or transaction type, or exclude further nested transactions altogether.

  These options allow you to adjust the level of detail that you see in the perfino and exclude transactions that do not add new information.

• Call tree and hot spots Show

  perfino records all nested transactions into a cumulated call tree. Transactions with the same name are added to the same node in the call tree. perfino keeps track of execution times and invocation counts for each call tree node. The call tree is a view in the perfino UI where you can expand nested transactions.

  From the call tree, perfino calculates hot spots. Hot spots are those transactions where most of the time is spent. Each hot spot has a list of cumulated backtraces that show in what ways the hot spot transaction has been called. When looking at the hot spots view in the perfino UI, you can quickly find bottlenecks and their causes.

• Historical comparisons Show

  The smallest resolution for which a call tree is available is one minute. Other available time ranges are 10 minutes, one hour and one day. Each of these intervals can be compared to previous intervals.

  In the call tree and hot spot views, you can either compare the entire tree or show a time line for selected transactions.

• Indefinite data-retention with automatic aggregation Show

  Call tree data of transactions is not discarded, it is only aggregated to larger time ranges. For example, the one minute granularity is only available for one day, but the one hour granularity is stored for 90 days. The one day granularity is kept indefinitely.

  In this way, the size of the database eventually grows at just a small linear rate.

perfino detects business transactions from the following subsystems:

• URL invocations in servlet containers Show

  perfino can intercept URL invocations in all application servers that implement the Java servlet API. URL segments, query parameters and regular expression groups can be used to build the transaction name. You can configure different settings for different URLs by filtering with wildcards or regular expressions.

  In addition to HTTP server transactions, HTTP client requests are measured by perfino. The following implementations are supported:

  • java.net.URLConnection
  • Java HTTP Client (Java 11+)
  • Apache HttpClient 4+
  • Apache Async HttpClient 4+
  • OkHttp 3.9+
  • Jersey (JAX-RS) Async Client 2.x
  • RestEasy Async Client 3.x
  • CXF Async Client 3.x
• EJB invocations Show

  EJBs (3.0+) from all applications servers can be intercepted by perfino. All invocations of a public EJB method create a business transaction. For transaction naming, the class and method names are available, as well as arbitrary getter chains on the EJB instance. You can filter EJBs by packages or type (stateless, stateful or message-driven) and configure different settings for them.

• Spring service invocations Show

  Public methods in Spring beans annotated with @Component, @Controller, @Repository or @Server are intercepted by perfino and business transactions are created. For transaction naming, the class and method names are available, as well as arbitrary getter chains on the EJB instance. You can filter Spring beans by packages or annotation and configure different settings for them.

• RMI calls Show

  Perfino can create business transactions for each incoming RMI call. You can filter remote interfaces by packages and assign different settings to them. For transaction naming, class and method names are available.

perfino offers several ways to define custom business transactions:

• Custom annotations Show

  If you use a framework that uses annotations to mark entry points or important methods, you can tell perfino to create business transactions for those methods.

  Both annotations that are used on methods and annotations that are used on classes are supported. For classes, all public method invocations create business transactions. Optionally, you can choose to intercept methods in derived classes as well. perfino offers several options to restrict the set of intercepted methods in that case.

• Pojo invocations Show

  You can configure arbitrary methods or classes for which perfino should create business transactions. Classes and methods can conveniently be selected from the connected VMs or from uploaded JAR, WAR or EAR archives.

  For classes, all public methods create business transactions. If you chose to intercept derived classes as well, perfino lets you filter derived classes by package names and offers several options to restrict the set of intercepted methods.

  For method interceptions, you can use the method parameters for transaction naming, or an arbitrary getter chain on their values. Class and method names as well as getter chains on the instance are also available for transaction naming.

• perfino DevOps annotations in your own code Show

  With the same features as the "Pojo invocations" above, the DevOps annotations can be used to define business transactions directly in your own code. Perfino annotations have class retention and are not visible at runtime, so they can be safely added to your project.

  The @MethodTransaction and @ClassTransaction annotations designate methods or classes where business transactions should be measured. Transaction naming and inheritance options are fully configured with the annotation parameters.

  Policies are frequently changed in operation and are configured in the perfino UI.

perfino can track transactions across VM boundaries:

• EJB tracking Show

  An invocation from an EJB client to an EJB server is tracked by perfino if both VMs are monitored by perfino and the EJB container is one of the following:

  • JBoss 7.1+ / Wildfly
  • Weblogic 11+
• HTTP tracking Show

  A HTTP request is tracked if both the calling and the serving VM are monitored and one of the following HTTP clients are used:

  • java.net.URLConnection
  • Java HTTP Client (Java 11+)
  • Apache HttpClient 4+
  • Apache Async HttpClient 4+
  • OkHttp 3.9+
  • Jersey (JAX-RS) Async Client 2.x
  • RestEasy Async Client 3.x
  • CXF Async Client 3.x
• Web service tracking Show

  A web service call is tracked if both the calling and the serving VM are monitored and one of the following web service implementations is used:

  • JAX-WS-RI
  • Apache Axis2
  • Apache CXF
• RMI tracking Show

  RMI calls are tracked, if both the calling and the serving VM are monitored.

• Call graph Show

  The call graph view in the perfino UI shows all interactions from tracked inter-VM calls as well as database calls. In the call graph, you can inspect transactions for selected nodes as well as for selected edges.

  Cross-VM tracking is also available in the call tree and in method-level sampling.

perfino offers a number of ways to identify problematic business transactions:

• Slow and very slow transactions Show

  Each transaction definition offers two separate configurable thresholds for identifying slow transactions: "slow" and "very slow". These categories are the basis for further actions and are displayed in all transaction views.

• Overdue transactions Show

  Transactions that do not return after a long time are classified as "overdue". Overdue transactions have their own view in perfino and are a critical condition for firing triggers.

• Transactions with errors Show

  perfino detects errors in transactions in a variety of ways: For web transactions, error status codes are detected. For transactions based on method invocations, runtime exceptions, checked exceptions and errors can optionally mark a transaction as an error.

  perfino can intercept selected logging systems and check if an error or a warning has been logged. The following logging systems are supported:

  • java.util.logging
  • log4j
  • JBoss logging
  • Logback
• Separate recording for policy violations Show

  The call tree is split for policy violations (slow, very slow, overdue or error) so you can see the measurements for those transactions as well as their nested transactions in isolation.

• Method sampling for policy violations Show

  If slow, very slow or overdue transactions are detected, perfino can automatically start method-level sampling, so you can investigate the cause of an unexpected slowness without having to bring in a profiler.

• Common policy handling Show

  Policy definitions are optional for transaction definitions. If a number of transaction definitions require the same policies, you don't have to define them each time, but you can handle them with a "catch-all" entry.

• Policy specializations based on transaction names Show

  If a single transaction definition results in transactions that require different policies depending on the transaction name, you can add policy specializations to the transaction definition.

Database operations are measured with specific probes for each database type:

• JDBC monitoring Show

  JDBC statements are measured independently of the used JDBC driver. Prepared statements are shown without their parameters and optionally unprepared statements can be resolved as well.

• JPA/Hibernate monitoring Show

  JPA/Hibernate queries and entity operations are measured by perfino. The following implementations are supported:

  • Hibernate 3.x and 4.x
  • openJPA 2.1+
  • eclipselink 2.3+.

  For each JPA/Hibernate operation, the associated JDBC statements are shown.

• NoSQL monitoring Show

  The following NoSQL databases are supported:

  • MongoDB (MongoDB driver starting with version 2.11 and higher)
  • Cassandra 1.2+
  • Apache HBase 0.92+

  perfino tries to remove values that change frequently, so the hot spots are more significant.

• Hot spots of database operations Show

  The "probe hot spots" views show the database operations that take most time. Each operation can be expanded to show the cumulated backtraces of business transactions that called the hot spot.

• Database operations in the call tree Show

  If a business transaction is slow due to a database call, you can see it immediately in the call tree. For each database type where at least one operation was recorded, a child node is added to a business transaction.

Under special circumstances, perfino can record single-threaded method-level call trees on a per-transaction basis:

• Periodic sampling on a per-transaction basis Show

  Transactions are sampled periodically, the period is configurable in the policy settings. In that way, you always have method-level data for recent transactions as a baseline when looking at method-level data from slow transactions.

• Sampling for policy violations Show

  In the case of certain policy violations (slow, very slow and overdue transactions), perfino can start sampling automatically, as soon as the configured threshold is exceeded. Method-level sampling will help you to find non-database related causes for slow transactions.

• Database operations in method-level samples Show
  In addition to the method-level call tree and hot spots, method samples contain all database operations that were executed during the sampling period.
• Cross-VM tracking in method-level samples Show
  Method-level sampling can span several VMs. When a business transaction is tracked across a VM boundary and sampling is in progress, the remote VM is sampled as well. VM nodes in the method sampling call tree show such VM boundaries.
• Averaging of multiple samples Show
  To improve the statistical accuracy of method-level sampling, you can average multiple samples for a selected transaction. This de-emphasizes outliers and produces more reliable hot spots.
• On-demand sampling Show
  When you are looking at a particular transaction in the method sampling view, you can ask perfino to record more samples for that transaction as soon as possible.

perfino can monitor page load times in the browser and correlate them with business transactions in the server:

• Measurement of page load times for a sample of all web requests Show

  Even if server response times are acceptable, the page load times in browsers may be not, especially if content is loaded from different sites. perfino can measure page load times for web transactions. For each transaction definition you can specify the percentage for which this kind of measurement should be performed.

• Support for all major application servers Show

  End user experience monitoring is supported for the following servers:

  • Tomcat 6+
  • Jetty 6+
  • JBoss 6+ / Wildfly
  • Weblogic 11+
  • Glassfish 3+
• Transparent injection of monitoring script into servlet requests Show

  Page load times are measured by injecting a small snippet of Javascript into each HTML page. The code snippet uses the HTML5 Web Performance API to report the real page load time back to a servlet filter that is automatically installed in your application server. The overhead of this operation is very small, since the script does not execute until the page is fully loaded.

• Overhead calculation of page load times Show

  perfino knows which business transactions on the server correspond to which page load times. In the "End user monitoring" view, business transactions and their reported page load times are shown and an overhead figure is calculated.

perfino records telemetries from a wide range of scalar data sources:

• VM telemetries Show

  perfino shows the following VM telemetries:

  • Heap usage
  • CPU load
  • Thread count
  • GC activity
• Monitoring statistics Show

  perfino shows telemetries for the following monitoring statistics:

  • Connection count
  • Transactions
  • Web requests
  • GC activity
• Probe telemetries Show

  perfino shows telemetries for the following probe measurements:

  • JDBC: Executed statements and average statement execution time
  • JPA/Hibernate: Entity operation count, query count and query duration
  • MongoDB: Executed operations and average operation execution time
  • Cassandra: Executed operations and average statement execution time
  • HBase: Executed operations and average operation execution time
  • JMS: Message count
  • JNDI: Query count
• MBean telemetries Show

  For each VM group, custom telemetries can be defined that poll numeric attribute values of MBeans. MBean servers do not have to be exposed with a JMX connection server for this to work, the perfino agent will pick up all MBean servers in the JVM. Each telemetry can be configured with a number of data lines that can optionally by stacked into an area graph. The unit of the measurements, a scaling factory as well as the behavior when aggregating multiple VMs can be configured.

• Devops telemetries Show

  Devops telemetries can be created by annotating a static method with a numeric return value with the @CustomTelemetry annotation. Such methods will be sampled by perfino and telemetries will be created in the perfino UI.

• Indefinite data-retention with automatic aggregation Show

  Telemetry data is never discarded, but just aggregated to larger time intervals. For example, the 10 second resolution data is only kept for 6 hours, the aggregated 2 minute resolution data is kept for 10 days. Even further aggregation yields 3 hour resolution data which is kept for one year. At the end, the 12 hour resolution data is kept indefinitely.

For telemetries, acceptable ranges can be defined and monitored:

• Threshold definitions with upper and lower bounds Show

  For each VM group, thresholds can be defined that determine the acceptable limits for any kind of telemetry. You can define optional lower and upper bounds.

• Options for avoiding duplicate threshold violations Show

  To avoid an excessive number of threshold violations, you can configure a minimum time for a threshold violation as well as an inhibition time during which no further threshold violations will be detected.

• Threshold violation counters for VM groups Show

  Threshold violations are counted for each VM group and bubble up to the "All VMs" groups. They do not have directly associated actions, but are the basis for threshold triggers. The threshold violation data is shown in the dashboard and as historical data in the "VM data views".

For policy and threshold violations, a list of actions can be executed:

• Policy triggers Show

  Policy triggers are fired if a number policy violations occurs in a defined period of time. The relevant transactions for a policy trigger can be filtered with wildcards or regular expressions. An inhibition time can be set to prevent duplicate trigger firings within a certain period.

• Threshold triggers Show

  Threshold triggers react to threshold violations. When a number of threshold violations are fired in a certain period of time, the threshold trigger fires. During the configured inhibition time, no further trigger firings can occur.

• Connection count triggers Show

  With connection count triggers you can ensure that a minimum number of VMs are connected to the perfino collector. The trigger can be armed after the threshold has been reached or after a fixed delay.

• Notification actions Show

  Several trigger actions can be used to create notifications on different levels. In escalating order, they are:

  • Write text to the log file. This can be useful for diagnostic purposes.
  • Add a dashboard alert. Dashboard alerts are shown prominently in the dashboard and are available for historical analysis as well.
  • Write a message to the perfino inbox. Inbox messages have an unread status separately for each user and the unread count is shown prominently in the perfino UI.
  • Send an email.
• Data collection actions Show

  A trigger can start more extensive data collection with the following trigger actions:

  • Saving a memory snapshot is useful to further investigate the reasons for low memory. Heap snapshots are saved in HPROF format and delivered to your perfino inbox. They can be further analyzed with JProfiler or other Java profilers.
  • Saving a thread dump is particularly useful in a policy trigger that fires for overdue transactions. The thread dump is delivered as a text file to your perfino inbox.
  • Recording fine-grained CPU data in profiling mode is useful to get more accurate method-level data for solving performance problems. For example, if the CPU load is very high, you need sampling data for all threads in the VM. CPU snapshots are delivered to your perfino inbox and can be opened with JProfiler.
• Alerts Show

  Alerts are created by triggers with the "Create dashboard alert" trigger action and are overlaid over the transaction time line in the dashboard. Alerts can have different severities with different associated colors.

  In addition to the dashboard, alerts are available as historical data in the "VM data views".

perfino offers built-in memory monitoring with very low overhead:

• Loaded classes and instance counts Show

  perfino includes a low-overhead memory monitoring technique that does not rely on high-overhead memory snapshots. This yields a list of loaded classes and their instance counts.

• Periodic and on-demand snapshots Show

  perfino records a low overhead memory snapshot periodically, so you always have a baseline for historical comparisons. In addition, you can trigger memory snapshots for single VMs in the "Memory" view.

• Historical comparisons Show

  Memory snapshots can be compared in perfino. You can either compare the entire table or show a time line of the instance counts or shallow sizes of previous snapshots.

• HPROF snapshot actions Show

  When searching for memory leaks, you need a full memory snapshot of the VM. Apart from the trigger action that takes an HPROF memory snapshot, the "VMs" view offers an action for manually taking an HPROF snapshot for a selected VM.

  HPROF snapshots can be opened in JProfiler or other Java profilers and memory analyzers. Snapshots are stored in perfino until you delete them from the "Snapshots" view.

With perfino, you can leverage JMX to inspect and control your applications:

• Access MBeans without JMX connector servers Show

  Usually, you have to start a JMX connector server to access MBeans from outside a JVM. With perfino, this is not necessary, because the agent accesses MBeans directly. In this way, you avoid the security implications of having to open a JMX port to the outside.

• Show complex MBean data Show

  Beside simple types like integers and strings, many MBeans use JMX Open Types with composite and tabular data. perfino shows arrays, composite types and tabular types in a single tree table. Tabular types with simple keys and single values are shown in a compact transposed form, so that maps from MXBeans, like the system properties in the "Runtime" MBean can be viewed as a property list.

• Edit MBean attributes Show

  Writable MBean attributes with simple types or their array types can be changed from perfino.

• Invoke MBean operations Show

  MBean operations with simple parameter types or simple array parameter types can be invoked from perfino. The return value is shown in a separate window and displays all MBean Open Types in a tree table.

• Monitor numeric values Show

  In the recording settings, telemetries can be defined that poll numeric MBean values. An MBean browser helps in finding the desired MBean and attribute. Monitoring values from an attribute path that crosses composite or tabular data is supported as well.

perfino integrates with JProfiler to facilitate more extensive data collection in case of need:

• On-demand loading of native profiling agent Show

  By default, perfino does not load native JVMTI libraries. However, in case of need, perfino can load JProfiler agent libraries and prepare a JVM so that JProfiler can attach to it remotely.

  All you need to do to enable this integration is to extract any version of JProfiler next to the perfino agent JAR file. After that, you can switch a VM to profiling directly in the "VMs" view.

• Low-overhead JVMTI sampling mode Show

  For full method sampling, perfino can activate a specially controlled JVMTI profile where only very few capabilities of the profiling interface are requested. This reduces the overhead and stability risk to an absolute minimum.

  For this sampling mode, perfino monitoring and JVMTI method sampling are concurrently available.

  The sampling results are recorded as a JProfiler snapshot and delivered to your perfino inbox.

• JDK Flight Recorder (JFR) support Show

  JDK Flight Recorder (JFR) snapshots can be recorded from the VMs view. This is a non-intrusive way to collect more low-level profiling data for a selected JVM. A corresponding trigger action is also available.

perfino can be deployed and configured interactively or in unattended mode:

• Installers for all major platforms Show

  perfino offers installers for Windows, macOS and Linux/Unix. A service/daemon is set up automatically by the installers. All important connectivity settings for the perfino server are configured directly in the installer.

  After the installer completes, you can open your browser (by default on localhost:8020) to complete the installation wizard in the perfino UI.

• Installation in unattended mode Show

  For deployments in the cloud, perfino can be installed without user interaction. All relevant configuration files can be placed next to the installer and are applied automatically.

• Automatic update of monitoring agents Show

  When you update the perfino server, you do not have to distribute new perfino monitoring agents to all your monitored VMs. perfino is backwards compatible with older monitoring agents and sends the new agent files to the monitored VMs where they are used when the VM is restarted the next time.

• Export and import of entire configuration Show

  The entire configuration of perfino can be exported to a single file from the perfino UI. Apart from backup purposes, this file enables you to install the perfino server in unattended mode.

• Optional separate deployment of collector server and UI server Show

  perfino comes with its own web server. By default, you don't have to bring in your own application server to deploy perfino.

  However, for networks topologies with a DMZ, the perfino collector and the UI may not run in the same network. in that case, you can deploy the perfino UI as a WAR file into another servlet container.

• User management for restricting access to the perfino UI Show

  perfino users can have one of three access levels:

  • Admin: Has all rights including licensing, user management and global configuration
  • Profiler: Can modify recording settings for selected VM groups. The profiling actions work for VMs in those VM groups.
  • Viewer: Cannot modify any settings, but view all recorded data.