Detect Malicious Behaviour on Kubernetes API Server through gathering Audit Logs by using FluentBit - Part 2
Introduction
In the previous blog post, we had talked about the Audit Logs in more detail, this post is a continuation of the previous blog post, so I suggest you take a look at the previous blog post before continuing reading. The only difference in this blog post is that this time we will use Log Backend mode, which is one of the Audit Backends that kube-apiserver
supports, then we will use some kind of log collector project which is FluentBit
in this case to gather these audit logs and forward them to some HTTP endpoint. You might be thinking why this kind of architecture can be more useful, the answer is that in the previous blog post we used Webhook Backend
mode, so we could send the audit logs to one location only. But with using Log Backend
mode and FluentBit
, we are now able to send audit logs to different locations, so we are now more flexible in this regard.
Here is the high-level overview architecture that shows what we want to achieve at the end of the day:
Prerequisites
We need tools with the following minimum versions to achieve this demo:
minikube
v1.22.0kubectl
v1.21.2helm
v3.6.2+gee407bd
Tutorial
Provision local Kubernetes Cluster
There are various ways to provision a local Kubernetes cluster such as KinD,
k3s
, k0s
, Minikube
etc. We are going to use Minikube
this time in this walkthrough.
There is a detailed guide about configuring Audit Logging in minikube, if you want to learn more about it, please see.
Let's get provision a local Kubernetes cluster:
Enable Kubernetes Audit Logging
Let's connect to the Minikube instance, which runs in VM, and download the Audit Policy file. This file is kind of a filter to decide what should be in the Audit Logs. This Audit Log file can grow very quickly and cause some performance issues if we didn't filter anything.
This is not the recommended Audit Policy file, this is just an example, it would be better to prepare your Audit Policy file that is suitable for your needs.
Let's edit the kube-apiserver, which is deployed as static Pod with the necessary flags and volumes to enable the Audit Logging.
Once you updated the manifest file, the kube-apiserver can't be reachable for a while, so please keep trying until the kube-apiserver becoming healthy.
The audit log file should be generated when the kube-apiserver becomes healthy, so please be sure about this before moving into the next phase.
Install Falco and Falcosidekick with Auditing Feature Enabled
We'll use Helm to install the Falco, so there is an option that we can enable or disable the audit log feature called auditLog.enabled
, once we set the value of the option as true
, the embedded webserver will be started within the Falco to consume audit events from the port 8765
and behind k8s-audit
endpoint.
Firstly, we'll create the namespace that will host both Falco
and Falcosidekick
:
We add the helm
repo:
In a real project, you should get the whole chart with helm pull falcosecurity/falco --untar
and then configure
the values.yaml
. For this tutorial, will try to keep thing as easy as possible and set configs directly
by passing arguments to helm install
command line:
You should get this output:
And you can see your new Falco
and Falcosidekick
pods:
Check your services within the namespace falco
, and you should see that Falco service has 8765
port, let's verify that.
The argument falcosidekick.enabled=true
sets the following settings in Falco
for you:
The arguments --set falco.jsonOutput=true --set falco.httpOutput.enabled=true --set falco.httpOutput.url=http://falco-falcosidekick:2801
are there to configure the format of events and the URL where Falco
will send them. As Falco
and Falcosidekick
will be in the same namespace, it can directly use the name of the service (falco-falcosidekick
) above Falcosidekick
pods.
We check the logs:
WebUI
is displayed as enabled output, everything is good 👍.
Deploy FluentBit
In order to deploy FluentBit, we're going to use its Helm chart with our own configurations.
If you want to get more detail about the Helm chart of the FluentBit
, please see.
Now we are ready to test everything.
Test
There is a rule called Create/Modify Configmap With Private Credentials
in Falco for detecting ConfigMaps that contains private credentials from Kubernetes Audit Logs
. You can see the details of the rule from this link. To test this, we should create a ConfigMap that contains private credentials.
First, use kubectl port-forward
to access the Falco Sidekick UI:
Open http://localhost:2802/ui/ to load the UI (omitting the /ui/
portion of the URL will result in an error)
Now, let's create the ConfigMap.
As soon as you created the ConfigMap, the Audit Log will be generated, then FluentBit will gather the audit log and send it to Falco. In turn, Falco will detect the malicious behavior from the audit logs and send an alert to the Falcosidekick. Finally, Falcosidekick forwards the alert to the Web UI. So, you should see similar screens like the following:
Conclusion
With this really simple example, we only want to show how FluentBit can be useful for us to implement this kind of architecture.
If you would like to find out more about Falco:
- Get started in Falco.org.
- Check out the Falco project in GitHub.
- Get involved in the Falco community.
- Meet the maintainers on the Falco Slack.
- Follow @falco_org on Twitter.