The cluster has about 50 nodes and 700 pods. Were running Filebeat on a Kubernetes cluster with Istio (not sure if relevant). Post that change the stop event will be published immediately if any container within the pod has a non-empty running container status. Hi everyone, So were having this problem for over a week now. Before that change the stop event on pod termination would never be published immediately it was always delayed by the close timeout. Note the stop event being published immediatelyįrom some investigation I think it relates to changes in Refactor kubernetes autodiscover to avoid skipping short-living pods by jsoriano T20:26:10.129Z INFO input/input.go:136 input ticker stopped T20:26:10.128ZĝEBUG autodiscover/autodiscover.go:253 Got a stop event Logs from Filebeat 7.13.0 T20:26:10.123ZĝEBUG kubernetes/pod.go:145 Watcher Pod update Note the 60 second gap between the pod update event being received from Kubernetes as the pod terminates and the stop event being published within filebeat to stop reading the logs autodiscover, remove filebeat.inputs configuration and uncomment this: todiscover. autodiscover: providers: - type: kubernetes templates: time travel curve WebTo deploy Filebeat to Kubernetes, run: kubectl create -f filebeat-kubernetes. ![]() T20:28:52.857Z INFO input/input.go:136 input ticker stopped T20:28:52.856ZĝEBUG autodiscover/autodiscover.go:253 Got a stop event T20:27:52.854ZĝEBUG kubernetes/pod.go:145 Watcher Pod update (terminating) Logs from Filebeat 7.12.1 T20:27:52.120ZĝEBUG kubernetes/pod.go:129 Watcher Pod update for pod: log-provider-0, status: Running That was fixed by the introduction of the close_timeout configuration option which is still in place today.įrom testing different versions it works as expected up to Filebeat 7.12.1, but is not working as expected from 7.13.0 onwards. elastic/beats () but for the kubernetes input.This issue appears to the same as reported in Filebeat autodiscover should not stop docker prospectors until it reads all lines ![]() ![]() When a Kubernetes pod terminates filebeat immediately stops reading log entries which can result in log lines at the end of the logs not being published.įilebeat configuration is setup as todiscover: I am using filebeat with autodiscover within a Kubernetes cluster to capture logs.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |