Hey there! I made my first cloud-native contribution! I added a service discovery for Hetzner Dedicated (robot) and Cloud (hcloud) servers to Prometheus!
So, what does this mean?
In the "old" way you need to specify each target within your Prometheus job configuration:
And you need to do this for each of your servers, it doesn't matter where you host it. But actually, i'm always lazy ;) I wanted to have a way that all my servers are auto-discovered and I don't need to reload/restart Prometheus every time an IP changes. But wait, how is that possible?
It is quite easy, Prometheus offers a function called `Service Discovery`. If you use Kubernetes you might already use the Service Discovery from Prometheus. It tries to find all targets you want to have. In the k8s world, this is mostly done with annotations and Prometheus just "Selects" all pods with a specific label, so Prometheus periodically calls the Kubernetes API with a query and the K8s API responds with all pods, Prometheus then adds the targets into is memory.
After I understood the process, I started working on it. It was quite easy, and the first running version was cobble within under one hour. I asked our DevOps Team @HetznerCloud if they wanted to try it, I got some valuable feedback, and they were happy about it. Sadly, I needed to stop working on it because we wanted to have the Load Balancers general availability. We needed to do some more things for this, but after the Load Balancers went GA and i was back from my small vacation i started working on it again! I polished it a bit, added tests and then asked the Prometheus team if they would accept a new service discovery. They liked the idea and Julien Pivotto jumped onto the train to review my code. We had a few back and forths and in the end it was not only a Hetzner Cloud Service Discovery, it also supported the Hetzner Dedicated (Robot) servers!
Okay, now enough "history", how can you use it?
First of all you need the latest Prometheus Release (At least v2.21.0-rc.0). Then have a look at the following snippet:
And then you need to start Prometheus :)! If your credentials are correct you can now see your servers on your Prometheus Dashboard: http://<your-prometheus>/service-discovery
Look at all of those labels! As you can see, there is a small difference between Hetzner Robot and Hetzner Cloud server labels because the robot web service and the Cloud API don't return the same fields. You can see the availability of the labels within the image of this post or head over to the Prometheus documentation.
When you now add/remove a server within your cloud or robot account, Prometheus will automatically recognize that! Isn't that awesome? No more restarts, no more config reloads, no more: "Oh, I missed to configure the server and didn't noticed it!".
You want to try it out? You could use my Hetzner Cloud Referral link and get 20€ Cloud credits for free!
This was actually my first contribution to a project with the size (and reputation) of Prometheus and my first contribution to a project from the Linux Foundation. I hope I can give more back to the "Open Source World" in the future! A special thank goes out to Julien Pivotto and Brian Brazil who reviewed my MR (and it was a long one :)!) and of course to all the colleagues @HetznerCloud who helped me during the development (and testing everything out!)
Did you know that we have a few open positions @Hetzner Cloud? Have a look at our job listings.