Ingesting via Kubernetes

You can ingest logs from your Kubernetes cluster into Axiom using filebeat.

The following is an example of a DaemonSet configuration to ingest your data logs into Axiom.

Configuration

apiVersion: v1
kind: ServiceAccount
metadata:
name: filebeat
namespace: kube-system
labels:
k8s-app: filebeat
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: filebeat
labels:
k8s-app: filebeat
rules:
- apiGroups: [''] # "" indicates the core API group
resources:
- namespaces
- pods
verbs:
- get
- watch
- list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: filebeat
subjects:
- kind: ServiceAccount
name: filebeat
namespace: kube-system
roleRef:
kind: ClusterRole
name: filebeat
apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
data:
filebeat.yml: |-
filebeat.autodiscover:
providers:
- type: kubernetes
node: ${NODE_NAME}
hints.enabled: true
hints.default_config:
type: container
paths:
- /var/log/containers/*${data.kubernetes.container.id}.log
processors:
- add_cloud_metadata:
output.elasticsearch:
hosts: ['${AXIOM_HOST}/api/v1/datasets/${AXIOM_DATASET_NAME}/elastic']
api_key: 'axiom:${AXIOM_INGEST_TOKEN}'
setup.ilm.enabled: false
kind: ConfigMap
metadata:
annotations: {}
labels:
k8s-app: filebeat
name: filebeat-config
namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
k8s-app: filebeat
name: filebeat
namespace: kube-system
spec:
selector:
matchLabels:
k8s-app: filebeat
template:
metadata:
annotations: {}
labels:
k8s-app: filebeat
spec:
containers:
- args:
- -c
- /etc/filebeat.yml
- -e
env:
- name: AXIOM_HOST
value: http://axiom:80
- name: AXIOM_DATASET_NAME
value: dataset
- name: AXIOM_INGEST_TOKEN
value: xait-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
- name: NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
image: docker.elastic.co/beats/filebeat-oss:7.9.3
imagePullPolicy: IfNotPresent
name: filebeat
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
securityContext:
runAsUser: 0
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/filebeat.yml
name: config
readOnly: true
subPath: filebeat.yml
- mountPath: /usr/share/filebeat/data
name: data
- mountPath: /var/lib/docker/containers
name: varlibdockercontainers
readOnly: true
- mountPath: /var/log
name: varlog
readOnly: true
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: filebeat
serviceAccountName: filebeat
terminationGracePeriodSeconds: 30
volumes:
- configMap:
defaultMode: 416
name: filebeat-config
name: config
- hostPath:
path: /var/lib/docker/containers
type: ''
name: varlibdockercontainers
- hostPath:
path: /var/log
type: ''
name: varlog
- hostPath:
path: /var/lib/filebeat-data
type: ''
name: data
updateStrategy:
rollingUpdate:
maxUnavailable: 1
type: RollingUpdate

Configure env

In the above configuration,

Configure your environment variables

env:
- name: AXIOM_HOST
value: http://axiom:80
- name: AXIOM_DATASET_NAME
value: dataset
- name: AXIOM_INGEST_TOKEN
value: xait-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

Where:

  • AXIOM_HOST is your Axiom deployment URL.

  • Insert the URL and port in the AXIOM_HOST value to match your configuration. For example, https://cloud.axiom.co:443 where https://cloud.axiom.co is ther URL for your Axiom Cloud installation. It is important to specify the port number (443 in the URL for Axiom Cloud or self-host with HTTPS, and port 80 for self-host with plain HTTP). The usage of port 9200 is not supported for Axiom cloud.

  • AXIOM_DATASET_NAME is your dataset name.

  • AXIOM_INGEST_TOKEN This can be either your ingest token or personal token.

The personal token can be retrieved from the users profile page (Setting > Profile > Personal token)

or an ingest token retrieved from the settings > Ingest Token page of your Axiom deployment.

The personal access token grants access to all resources available to the user, while the ingest token just allows ingestion into the datasets the token is configured for.


  • After editing your values, apply the changes to your cluster using kubectl apply -f daemonset.yaml