In previous article, we explanined how to develop a microservice in SpringBoot that uses infinispan as spring-cache implementation. We exposed the actuator prometheus endpoint to check in our local environment that it was running right. In this post, we are going to use the actuator endpoint to scrape it with prometheus and create a grafana dashboard to monitor the performance.
Initial Set Up in kubernetes
We assume that we have our microservice already deployed on your kubernetes namespace. From now on, in our examples, we will use the namespace
We need a prometheus server to scrape the metrics. At that point, there are several alternatives for this task. In another post about how to deploy and monitor infinispan servers, I used prometheus operator to do the set up.
However, that is an usual approach if you have cluster-admin rights, or you already have the prometheus operator deployed, as installing CRDs requires those privileges.
If you are in this situation, go ahead and use a ServiceMonitor, as it is the easiest approach. However, if you are not able to use the operator, I will give you an alternative at this point.
I took the helm chart from
this project. You can select the version you need. Even this helm chart is usually meant for creating ClusterRole and ClusterRoleBindings. However, if we select
server.useExistingClusterRoleName=false, it will use a serviceAccount without extra privileges. So you can add those by creating a Role and RoleBinding (not cluster) on your namespace.
You need also to configure the rules for scrapping the actuator endpoint. I include an
example with ServiceMonitor and also an example of
values.yaml where the infinispan monitoring has been set up for namespace
customers using the field
To deploy on namespace
customers without cluster-admin privileges, just do:
helm install prometheus prometheus-community/prometheus -n customers -f values.yaml
We deploy grafana with the same procedure as in this previous post. However, as time goes by, I have updated the image version to 8.1.2.
Now that we have the datasource created by the dashboard (review the endpoint of your prometheus server if you are not using
customers namespace), we can import the dashboards. I have updated
this dashboard with the latest changes of the SpringBoot version I am using, removed the HikariCP panel and updated charts to latest (and greatest) version of grafana. I am providing it in case you want to reuse or take it as a base for your own
You only need to import the dashboard and assign the prometheus datasource. Similar to that, I have developed a very simple one for infinispan with the same structure. If you prefer, you may combine both into only one with the infinispan part as a panel. This is my example of Infinispan Embedded dashboard.
Looking at the metrics and dashboard
Micrometer defines a set of metrics that are supposed to be implemented by all cache implementations and and additional way of extending those metrics .
Mapping from internal metrics to micrometer ones is done in this binder. We are grouping metrics to show the common and the particular ones.
The dashboard is split into a set of panels. First panel is related to general cache metrics as defined with micrometer.
Second one is about cache statistics: when cache was started and when statistics have been reset.
And last one is related to performance in the caches. Memory usage and time that takes to do operations in the cache are shown.
As you can see, after first put that warms the microservice, average put time is very low, but the more impressive one is get time, that takes less than a millisecond and so it cannot even be seen.