Skip to content

Instantly share code, notes, and snippets.

@ITBlogger

ITBlogger/filebeat configmaps

Last active December 10, 2019 14:19
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save ITBlogger/f50632d643ec4cb241bdd41355b295ba to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save ITBlogger/f50632d643ec4cb241bdd41355b295ba to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. ITBlogger revised this gist Aug 23, 2018. 1 changed file with 0 additions and 1 deletion.
    1 change: 0 additions & 1 deletion filebeat configmaps
    View file
    Original file line number Diff line number Diff line change
    @@ -12,7 +12,6 @@ data:
    filebeat.autodiscover:
    providers:
    - type: kubernetes
    host: ${NODE_NAME}
    hints.enabled: true
    include_annotations: '*'

  2. ITBlogger revised this gist Aug 23, 2018. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions filebeat configmaps
    View file
    Original file line number Diff line number Diff line change
    @@ -12,6 +12,7 @@ data:
    filebeat.autodiscover:
    providers:
    - type: kubernetes
    host: ${NODE_NAME}
    hints.enabled: true
    include_annotations: '*'

  3. ITBlogger revised this gist Aug 21, 2018. 1 changed file with 3 additions and 3 deletions.
    6 changes: 3 additions & 3 deletions k8s deploy snippet
    View file
    Original file line number Diff line number Diff line change
    @@ -17,6 +17,6 @@ spec:
    service-type: ruby
    annotations:
    co.elastic.logs/exclude_lines: 'GET\s"?/status'
    co.elastic.logs/module: nginx
    co.elastic.logs/fileset.stdout: access
    co.elastic.logs/fileset.stderr: error
    co.elastic.logs/module: 'nginx'
    co.elastic.logs/fileset.stdout: 'access'
    co.elastic.logs/fileset.stderr: 'error'
  4. ITBlogger revised this gist Aug 20, 2018. 1 changed file with 10 additions and 9 deletions.
    19 changes: 10 additions & 9 deletions k8s deploy snippet
    View file
    Original file line number Diff line number Diff line change
    @@ -9,13 +9,14 @@ metadata:
    worker-type: main
    service-type: ruby
    spec:
    metadata:
    labels:
    log-type: puma-log
    worker-type: main
    service-type: ruby
    annotations:
    co.elastic.logs/exclude_lines: 'GET\s"?/status'
    co.elastic.logs/module: nginx
    co.elastic.logs/fileset.stdout: access
    template:
    metadata:
    labels:
    log-type: puma-log
    worker-type: main
    service-type: ruby
    annotations:
    co.elastic.logs/exclude_lines: 'GET\s"?/status'
    co.elastic.logs/module: nginx
    co.elastic.logs/fileset.stdout: access
    co.elastic.logs/fileset.stderr: error
  5. ITBlogger revised this gist Aug 20, 2018. 2 changed files with 12 additions and 17 deletions.
    13 changes: 1 addition & 12 deletions filebeat configmaps
    View file
    Original file line number Diff line number Diff line change
    @@ -13,18 +13,7 @@ data:
    providers:
    - type: kubernetes
    hints.enabled: true
    include_annotations: true

    filebeat.config:
    inputs:
    # Mounted `filebeat-inputs` configmap:
    path: ${path.config}/inputs.d/*.yml
    # Reload inputs configs as they change:
    reload.enabled: false
    modules:
    path: ${path.config}/modules.d/*.yml
    # Reload module configs as they change:
    reload.enabled: false
    include_annotations: '*'

    processors:
    - add_cloud_metadata:
    16 changes: 11 additions & 5 deletions k8s deploy snippet
    View file
    Original file line number Diff line number Diff line change
    @@ -8,8 +8,14 @@ metadata:
    log-type: puma-log
    worker-type: main
    service-type: ruby
    annotations:
    co.elastic.logs/exclude_lines: 'GET\s"?/status'
    co.elastic.logs/module: nginx
    co.elastic.logs/fileset.stdout: access
    co.elastic.logs/fileset.stderr: error
    spec:
    metadata:
    labels:
    log-type: puma-log
    worker-type: main
    service-type: ruby
    annotations:
    co.elastic.logs/exclude_lines: 'GET\s"?/status'
    co.elastic.logs/module: nginx
    co.elastic.logs/fileset.stdout: access
    co.elastic.logs/fileset.stderr: error
  6. ITBlogger revised this gist Aug 20, 2018. 1 changed file with 4 additions and 25 deletions.
    29 changes: 4 additions & 25 deletions filebeat configmaps
    View file
    Original file line number Diff line number Diff line change
    @@ -13,6 +13,7 @@ data:
    providers:
    - type: kubernetes
    hints.enabled: true
    include_annotations: true

    filebeat.config:
    inputs:
    @@ -33,11 +34,11 @@ data:
    xpack.monitoring:
    enabled: true
    elasticsearch:
    hosts: ["<elk-cluster>:9200"]
    hosts: ["<elasticsearch-cluster>:9200"]

    output:
    logstash:
    hosts: ["<elk-cluster>:5044"]
    hosts: ["<elasticsearch-cluster>:5044"]
    ttl: 60
    pipelining: 0
    ssl:
    @@ -46,26 +47,4 @@ data:
    shipper:
    logging:
    files:
    rotateeverybytes: 10485760 # = 100MB
    ---
    apiVersion: v1
    kind: ConfigMap
    metadata:
    name: filebeat-inputs
    namespace: kube-system
    labels:
    k8s-app: filebeat
    kubernetes.io/cluster-service: "true"
    data:
    kubernetes.yml: |-
    - type: docker
    containers.ids:
    - "*"
    fields:
    service: kubernetes
    document_type: kubernetes
    fields_under_root: true
    close_inactive: 1m
    processors:
    - add_kubernetes_metadata:
    in_cluster: true
    rotateeverybytes: 10485760 # = 100MB
  7. ITBlogger created this gist Aug 18, 2018.
    71 changes: 71 additions & 0 deletions filebeat configmaps
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,71 @@
    ---
    apiVersion: v1
    kind: ConfigMap
    metadata:
    name: filebeat-config
    namespace: kube-system
    labels:
    k8s-app: filebeat
    kubernetes.io/cluster-service: "true"
    data:
    filebeat.yml: |-
    filebeat.autodiscover:
    providers:
    - type: kubernetes
    hints.enabled: true

    filebeat.config:
    inputs:
    # Mounted `filebeat-inputs` configmap:
    path: ${path.config}/inputs.d/*.yml
    # Reload inputs configs as they change:
    reload.enabled: false
    modules:
    path: ${path.config}/modules.d/*.yml
    # Reload module configs as they change:
    reload.enabled: false

    processors:
    - add_cloud_metadata:

    tags: [ 'kubernetes', 'Wendimore', 'EKS-worker-nodes' ]

    xpack.monitoring:
    enabled: true
    elasticsearch:
    hosts: ["<elk-cluster>:9200"]

    output:
    logstash:
    hosts: ["<elk-cluster>:5044"]
    ttl: 60
    pipelining: 0
    ssl:
    certificate_authorities: ["/etc/logstash-certs/logstash.crt"]

    shipper:
    logging:
    files:
    rotateeverybytes: 10485760 # = 100MB
    ---
    apiVersion: v1
    kind: ConfigMap
    metadata:
    name: filebeat-inputs
    namespace: kube-system
    labels:
    k8s-app: filebeat
    kubernetes.io/cluster-service: "true"
    data:
    kubernetes.yml: |-
    - type: docker
    containers.ids:
    - "*"
    fields:
    service: kubernetes
    document_type: kubernetes
    fields_under_root: true
    close_inactive: 1m
    processors:
    - add_kubernetes_metadata:
    in_cluster: true
    15 changes: 15 additions & 0 deletions k8s deploy snippet
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,15 @@
    ---
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
    creationTimestamp: null
    name: foo-bar
    labels:
    log-type: puma-log
    worker-type: main
    service-type: ruby
    annotations:
    co.elastic.logs/exclude_lines: 'GET\s"?/status'
    co.elastic.logs/module: nginx
    co.elastic.logs/fileset.stdout: access
    co.elastic.logs/fileset.stderr: error