Ingesting via Kubernetes

You can ingest logs from your Kubernetes cluster into Axiom using filebeat.

The following is an example of a DaemonSet configuration to ingest your data logs into Axiom.

Configuration

apiVersion: v1
kind: ServiceAccount
metadata:
name: filebeat
namespace: kube-system
labels:
k8s-app: filebeat
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: filebeat
labels:
k8s-app: filebeat
rules:
- apiGroups: [''] # "" indicates the core API group
resources:
- namespaces
- pods
verbs:
- get
- watch
- list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: filebeat
subjects:
- kind: ServiceAccount
name: filebeat
namespace: kube-system
roleRef:
kind: ClusterRole
name: filebeat
apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
data:
filebeat.yml: |-
filebeat.autodiscover:
providers:
- type: kubernetes
node: ${NODE_NAME}
hints.enabled: true
hints.default_config:
type: container
paths:
- /var/log/containers/*${data.kubernetes.container.id}.log
processors:
- add_cloud_metadata:
output.elasticsearch:
hosts: ['${AXIOM_HOST}/api/v1/datasets/${AXIOM_DATASET_NAME}/elastic']
api_key: 'axiom:${AXIOM_INGEST_TOKEN}'
setup.ilm.enabled: false
kind: ConfigMap
metadata:
annotations: {}
labels:
k8s-app: filebeat
name: filebeat-config
namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
k8s-app: filebeat
name: filebeat
namespace: kube-system
spec:
selector:
matchLabels:
k8s-app: filebeat
template:
metadata:
annotations: {}
labels:
k8s-app: filebeat
spec:
containers:
- args:
- -c
- /etc/filebeat.yml
- -e
env:
- name: AXIOM_HOST
value: http://axiom:80
- name: AXIOM_DATASET_NAME
value: dataset
- name: AXIOM_INGEST_TOKEN
value: xait-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
- name: NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
image: docker.elastic.co/beats/filebeat-oss:7.9.3
imagePullPolicy: IfNotPresent
name: filebeat
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
securityContext:
runAsUser: 0
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/filebeat.yml
name: config
readOnly: true
subPath: filebeat.yml
- mountPath: /usr/share/filebeat/data
name: data
- mountPath: /var/lib/docker/containers
name: varlibdockercontainers
readOnly: true
- mountPath: /var/log
name: varlog
readOnly: true
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: filebeat
serviceAccountName: filebeat
terminationGracePeriodSeconds: 30
volumes:
- configMap:
defaultMode: 416
name: filebeat-config
name: config
- hostPath:
path: /var/lib/docker/containers
type: ''
name: varlibdockercontainers
- hostPath:
path: /var/log
type: ''
name: varlog
- hostPath:
path: /var/lib/filebeat-data
type: ''
name: data
updateStrategy:
rollingUpdate:
maxUnavailable: 1
type: RollingUpdate

Configure env

In the above configuration,

Configure your environment variables

env:
- name: AXIOM_HOST
value: http://axiom:80
- name: AXIOM_DATASET_NAME
value: dataset
- name: AXIOM_INGEST_TOKEN
value: xait-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

Where:

  • AXIOM_HOST is your Axiom deployment URL.

  • Insert the URL and port in the AXIOM_HOST value to match your configuration. For example, https://cloud.axiom.co:443 where https://cloud.axiom.co is ther URL for your Axiom Cloud installation. It is important to specify the port number (443 in the URL for Axiom Cloud or self-host with HTTPS, and port 80 for self-host with plain HTTP). The usage of port 9200 is not supported for Axiom cloud.

  • AXIOM_DATASET_NAME is your dataset name.

  • AXIOM_INGEST_TOKEN This should your ingest token.

The ingest token can be retrieved from the settings > Ingest Token page of your Axiom deployment.


  • After editing your values, apply the changes to your cluster using kubectl apply -f daemonset.yaml