Ingesting logs from Kubernetes Cluster
You can ingest logs from your Kubernetes cluster into Axiom using filebeat.
The following is an example of a DaemonSet configuration to ingest your data logs into Axiom.
Configuration
apiVersion: v1kind: ServiceAccountmetadata:name: filebeatnamespace: kube-systemlabels:k8s-app: filebeat---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata:name: filebeatlabels:k8s-app: filebeatrules:- apiGroups: [''] # "" indicates the core API groupresources:- namespaces- podsverbs:- get- watch- list---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata:name: filebeatsubjects:- kind: ServiceAccountname: filebeatnamespace: kube-systemroleRef:kind: ClusterRolename: filebeatapiGroup: rbac.authorization.k8s.io---apiVersion: v1data:filebeat.yml: |-filebeat.autodiscover:providers:- type: kubernetesnode: ${NODE_NAME}hints.enabled: truehints.default_config:type: containerpaths:- /var/log/containers/*${data.kubernetes.container.id}.logprocessors:- add_cloud_metadata:output.elasticsearch:hosts: ['${AXIOM_HOST}/api/v1/datasets/${AXIOM_DATASET_NAME}/elastic']api_key: 'axiom:${AXIOM_API_TOKEN}'setup.ilm.enabled: falsekind: ConfigMapmetadata:annotations: {}labels:k8s-app: filebeatname: filebeat-confignamespace: kube-system---apiVersion: apps/v1kind: DaemonSetmetadata:labels:k8s-app: filebeatname: filebeatnamespace: kube-systemspec:selector:matchLabels:k8s-app: filebeattemplate:metadata:annotations: {}labels:k8s-app: filebeatspec:containers:- args:- -c- /etc/filebeat.yml- -eenv:- name: AXIOM_HOSTvalue: http://axiom:80- name: AXIOM_DATASET_NAMEvalue: dataset- name: AXIOM_API_TOKENvalue: xaat-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx- name: NODE_NAMEvalueFrom:fieldRef:apiVersion: v1fieldPath: spec.nodeNameimage: docker.elastic.co/beats/filebeat-oss:7.9.3imagePullPolicy: IfNotPresentname: filebeatresources:limits:memory: 200Mirequests:cpu: 100mmemory: 100MisecurityContext:runAsUser: 0terminationMessagePath: /dev/termination-logterminationMessagePolicy: FilevolumeMounts:- mountPath: /etc/filebeat.ymlname: configreadOnly: truesubPath: filebeat.yml- mountPath: /usr/share/filebeat/dataname: data- mountPath: /var/lib/docker/containersname: varlibdockercontainersreadOnly: true- mountPath: /var/logname: varlogreadOnly: truednsPolicy: ClusterFirstrestartPolicy: AlwaysschedulerName: default-schedulersecurityContext: {}serviceAccount: filebeatserviceAccountName: filebeatterminationGracePeriodSeconds: 30volumes:- configMap:defaultMode: 416name: filebeat-configname: config- hostPath:path: /var/lib/docker/containerstype: ''name: varlibdockercontainers- hostPath:path: /var/logtype: ''name: varlog- hostPath:path: /var/lib/filebeat-datatype: ''name: dataupdateStrategy:rollingUpdate:maxUnavailable: 1type: RollingUpdate
Configure env
In the above configuration,
Configure your environment variables
env:- name: AXIOM_HOSTvalue: http://axiom:80- name: AXIOM_DATASET_NAMEvalue: dataset- name: AXIOM_API_TOKENvalue: xaat-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Where:
AXIOM_HOST is your Axiom deployment URL.
Insert the URL and port in the AXIOM_HOST value to match your configuration. For example,
https://cloud.axiom.co:443
wherehttps://cloud.axiom.co
is ther URL for your Axiom Cloud installation. It is important to specify the port number (443
in the URL for Axiom Cloud or self-host with HTTPS, and port80
for self-host with plain HTTP). The usage of port9200
is not supported for Axiom cloud.AXIOM_DATASET_NAME is your dataset name.
AXIOM_API_TOKEN is your API token.
The API token can be retrieved from the settings > API Token page of your Axiom deployment.
- After editing your values, apply the changes to your cluster using
kubectl apply -f daemonset.yaml