In this blog post, I will show you how easy it is to limit access to your Elasticsearch cluster in Kubernetes when you are using Cilium. As a side note: of course I know that Elastic has its own ways of limiting access, which are not all without a license (fee), so this post is not to be read as “this is how you limit access to Elasticsearch”, but as “Hey this is fun, you can use Cilium” to secure your Elasticsearch cluster.
What is Cilium
Cilium is open source software for transparently securing the network connectivity between application services deployed using Linux container management platforms like Docker and Kubernetes. At the foundation of Cilium is a new Linux kernel technology called eBPF (extended Berkeley Packet Filter), which enables the dynamic insertion of powerful security visibility and control logic within Linux itself. Because eBPF runs inside the Linux kernel, Cilium security policies can be applied and updated without any changes to the application code or container configuration.
What is eBPF
eBPF is a revolutionary technology that can run sandboxed programs in the Linux kernel without changing kernel source code or loading kernel modules. By making the Linux kernel programmable, infrastructure software can leverage existing layers, making them more intelligent and feature-rich without continuing to add additional layers of complexity to the system or compromising execution efficiency and safety.
To dive into eBPF goes beyond the scope of this blog post. If you are into Linux, networking and security, and you have not yet heard of it, please do yourself a favor and spent some time learning at least the basics. In my humble opinion, eBPF will have a great impact on Linux, networking and security in the future, I think for example that it will replace (parts of) iptables and other tools that are used for tracing and profiling and observability and monitoring.
For this blog post, I installed Elasticsearch, Kibana and fluent-bit from Helm charts, all in one namespace on my Kubernetes cluster, which is installed using mosibi-kubernetes.
helm repo add elastic https://helm.elastic.co kubectl create namespace logging helm install --namespace logging --set persistence.enabled=false elasticsearch elastic/elasticsearch helm install --namespace logging kibana elastic/kibana helm repo add fluent https://fluent.github.io/helm-charts helm install --namespace logging fluent-bit fluent/fluent-bit
Kibana and Fluent Bit
Kibana is a web-based graphical user interface for Elasticsearch, which has become the defacto standard to visualize data from Elasticsearch and Fluent Bit is a log processor and forwarder which allows you to collect logs from different sources like containers and send them to multiple destinations. In this lab environment, Fluent Bit reads the logging from containers and sends it to Elasticsearch, which is a very common setup in Kubernetes land.
Although not necessary, I also created the following ingress rules so that I can reach http://kibana.example.com and http://elasticsearch.example.com from my laptop.
apiVersion: traefik.containo.us/v1alpha1 kind: IngressRoute metadata: name: elasticsearch-ingress namespace: logging spec: entryPoints: - web routes: - kind: Rule match: Host(`elasticsearch.example.com`) services: - kind: Service name: elasticsearch-master port: 9200 --- apiVersion: traefik.containo.us/v1alpha1 kind: IngressRoute metadata: name: kibana-ingress namespace: logging spec: entryPoints: - web routes: - kind: Rule match: Host(`kibana.example.com`) services: - kind: Service name: kibana-kibana port: 5601
As I wrote earlier, in this post I will show you how to limit access to Elasticsearch using Cilium, and then specifically using the OSI layer 3 filtering where the filtering is done based on an ip address or Kubernetes label and on layer 7, where I explicitly allow access to certain Elasticsearch endpoints.
The following policies will be created:
- Fluent-bit pods are allowed to inject data into Elasticsearch
- The Kibana deployment which lives in the logging namespace has complete access to Elasticsearch
- Access to the Elasticsearch cluster health API will be allowed from everywhere.
- Access to the logstash index will be allowed from the default namespace. The logstash index is automatically created by Fluent Bit and contains the log output from containers.
When an allow policy is set, all other traffic is not allowed anymore, so I also have to create a policy to allow the Elasticsearch nodes to talk to each other.
Elasticsearch internal traffic
The first policy I need to create, is the one that allows the Elasticsearch nodes to talk to each other, otherwise your cluster will become red very soon!
cat <<EOF | kubectl -n logging apply -f - apiVersion: "cilium.io/v2" kind: CiliumNetworkPolicy metadata: name: "elasticsearch-allow-internal-cluster-traffic" spec: description: "Allow Elasticsearch cluster nodes to talk to each other" endpointSelector: matchLabels: app: elasticsearch-master ingress: - fromEndpoints: - matchLabels: app: elasticsearch-master EOF
This is a simple layer 3 ingress policy that defines, that traffic to the elasticsearch pods is allowed from nodes which have a label app: elasticsearch-master, the ‘to’ part is specified with an endpointSelector. After you apply this policy, every other traffic to the endpoint elasticsearch-master is denied, which in this case is exactly what I want :).
A Cilium endpointSelector selects all endpoints with a matching label and the Cilium terminology specifies that an endpoint is a Kubernetes Pod with an ip address. In other words, common Kubernetes labeling.
An other product from Cilium is Hubble which is a product with a graphical web interface that gives deep visibility into the communication and behavior of services as well as the networking infrastructure. After applying our first policy, it is clearly visible in Hubble that traffic from the fluent-bit pods is dropped immediately, and that the internal traffic between the Elasticsearch pods is allowed (forwarded)
Fluent Bit is installed with the default configuration using a Helm chart. That default configuration reads container logs files from
/var/log/containers/*.log and the log output from systemd unit
kubelet.service and send it to Elasticsearch host
elasticsearch-master, which is a Kubernetes services that load-balances over the Elasticsearch masters, in index
logstash. The following layer 3 policy allows Fluent Bit to talk to the Elasticsearch masters and there are no limitations on the index it wants to write to.
cat <<EOF | kubectl -n logging apply -f - lang=shell apiVersion: "cilium.io/v2" kind: CiliumNetworkPolicy metadata: name: "elasticsearch-allow-fluent-bit" spec: description: "Allow fluent-bit access to Elasticsearch" endpointSelector: matchLabels: app: elasticsearch-master ingress: - fromEndpoints: - matchLabels: "k8s:app.kubernetes.io/name": fluent-bit EOF
After the policy is applied, you should see data coming into the logstash index. You can see that in Kibana, or using a curl command against one of the Elasticsearch masters:
$ kubectl -n logging exec -ti elasticsearch-master-0 -- curl http://elasticsearch-master:9200/_cat/indices?v health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open logstash-2021.01.10 pkKioML8RvahFk1rB_mvNA 1 1 1147 0 1.2mb 750.6kb green open .apm-custom-link jE3ju5xpQYOFV_yMZsrQQg 1 1 0 0 416b 208b green open .kibana_task_manager_1 MAWaGn6rTtm2piukCKQP_g 1 1 5 15 120.5kb 64kb green open .kibana-event-log-7.10.1-000001 IrszLTlcSCerl5RxzBr10w 1 1 1 0 11.1kb 5.5kb green open .apm-agent-configuration BzQZ8pmDRAa_OaR8e9vhsQ 1 1 0 0 416b 208b green open .kibana_1 jnCvQX68TtuO3Tybihk2UQ 1 1 8 0 4.2mb 2.1mb
Kibana can be used to display Elasticsearch data and to manage the Elasticsearch stack, therefor it needs full access. The following layer 3 policy accomplices that.
cat <<EOF | kubectl -n logging apply -f - apiVersion: "cilium.io/v2" kind: CiliumNetworkPolicy metadata: name: "elasticsearch-allow-kibana" spec: description: "Allow kibana access to Elasticsearch" endpointSelector: matchLabels: app: elasticsearch-master ingress: - fromEndpoints: - matchLabels: app: kibana EOF
Elasticsearch cluster health API
The following network policy is a layer 7 policy that allows the HTTP method GET on _/cluster/health.*. By using this wildcard notation, I give access to the bare
health API as well as all methods behind it. For example, all of these curl commands are allowed with this policy:
curl http://elasticsearch.example.com/_cluster/health curl http://elasticsearch.example.com/_cluster/health?pretty curl http://elasticsearch.example.com/_cluster/health/logstash-2021.01.10?level=shards
cat <<EOF | kubectl -n logging apply -f - apiVersion: "cilium.io/v2" kind: CiliumNetworkPolicy metadata: name: "elasticsearch-allow-cluster-endpoint" spec: description: "Allow HTTP access to Elasticsearch _cluster endpoint" endpointSelector: matchLabels: app: elasticsearch-master ingress: - fromEntities: - "all" toPorts: - ports: - port: "9200" protocol: TCP rules: http: - method: "GET" path: "/_cluster/health.*" EOF
In the previous Cilium network policies, I used fromEndpoints together with a Kubernetes label to select from where certain traffic is allowed. In this policy, I use fromEntities which has a list of pre-defined entities that are known by Cilium, for example host, cluster and world. In this policy the pre defined entity all is used:
The “all” entity represents the combination of all known clusters as well world and whitelists all communication.
Access to the logstash indices from the default namespace
Where all the previous policies, are policies that set the base to get a working cluster and allow basic communication with it, this policy allows access from an application living in the default namespace to access data in the logstash indices.
cat <<EOF | kubectl -n default apply -f - apiVersion: "cilium.io/v2" kind: CiliumNetworkPolicy metadata: name: "elasticsearch-allow-access-to-logstash" spec: description: "Allow HTTP access to the Elasticsearch logstash index" endpointSelector: matchLabels: app: elasticsearch-master ingress: - fromEndpoints: - matchLabels: "k8s:io.kubernetes.pod.namespace": default toPorts: - ports: - port: "9200" protocol: TCP rules: http: - method: "GET" path: "/logstash.*" EOF
Although this is a very simple policy that only allows the HTTP method GET, it is of course possible to create more complex rule sets where for example a GET is allowed on logstash., but also a *PUT is allowed on /mydata:
rules: http: - method: "GET" path: "/logstash.*" - method: "PUT" path: "/mydata"
By now, you should have an idea about the power of the Cilium network policies and a basic understanding about how you can use Cilium to secure your applications.