[elasticsearch] 使用 filebeat 收集 ECS docker log 到 elasticsearch

ECS container 一多，怎麼集中化的觀看 ECS cluster 的 containters 的 logs 呢?
由 filebeat 把 ecs ec2 的 instance log 送出來，

filebeat -> logstash -> elasticsearch -> Kibana

filebeat.yml

filebeat.autodiscover:
# Autodiscover docker containers and parse logs
providers:
  - type: docker
    hints.enabled: true
    containers.ids:
      - "*"

filebeat.inputs:
- type: docker
 containers.ids:
   - "*"
 fields:
   index: "filebeat-docker"
 processors:
   - add_docker_metadata: ~
   - drop_event:
       when:
         or:
           - and:
             - contains:
                 docker.container.image: filebeat
             - contains:
                 message: "Read line error: invalid CRI log"
           - and:
             - contains:
                 docker.container.image: cadvisor
             - contains:
                 message: "Stat fs failed. Error: no such file or directory"

output.logstash:
  enabled: true
  hosts: ["logstash_host:5044"]

# output.console:
#  enabled: true
#  pretty: true


這裡 fileds 加上 index 是由於 elasticsearch 6 之後只支援一個index 一個type,
藉由在 fileds 加上 index 這個欄位 來判斷是哪個 index 。
event 中 使用 contains 來過濾一下條件，過濾掉過過多不必要log，再往 logstash送。
先由 output.console: 驗證是否如需求。

``` filebeat.Dockerfile
FROM docker.elastic.co/beats/filebeat:6.3.1

COPY filebeat.yml /usr/share/filebeat/filebeat.yml
USER root
RUN chmod go-w /usr/share/filebeat/filebeat.yml
```

docker build -t pcfilebeat -f filebeat.Dockerfile .
docker run --user=root --name testd --privileged --rm -v "/var/lib/docker/containers:/var/lib/docker/containers:ro" -v "/var/run/docker.sock:/var/run/docker.sock:ro" pcfilebeat -e -strict.perms=false \n

使用 root 執行 並且 mount /var/run/docker.sock ，
這樣子 filebeat.autodiscover docker 才能偵測到 docker的資訊，

看到類似的 event
```
{
  "@timestamp": "2019-01-10T08:30:55.382Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.3.1"
  },
  "source": "/var/lib/docker/containers/78433ef87b5f0e011c1f6c3363e292044959759f322b76fc986df8ae405f6039/78433ef87b5f0e011c1f6c3363e292044959759f322b76fc986df8ae405f6039-json.log",
  "stream": "stdout",
  "message": "              \"version\": \"1.8.0\",",
  "prospector": {
    "type": "docker"
  },
  "input": {
    "type": "docker"
  },
  "offset": 597893,
  "docker": {
    "container": {
      "id": "78433ef87b5f0e011c1f6c3363e292044959759f322b76fc986df8ae405f6039",
      "name": "testd",
      "image": "pcfilebeat",
      "labels": {
        "org": {
          "label-schema": {
            "name": "filebeat",
            "schema-version": "1.0",
            "url": "https://www.elastic.co/products/beats/filebeat",
            "vcs-url": "https://github.com/elastic/beats-docker",
            "vendor": "Elastic",
            "version": "6.3.1"
          }
        },
        "license": "Elastic License"
      }
    }
  },
  "beat": {
    "name": "78433ef87b5f",
    "hostname": "78433ef87b5f",
    "version": "6.3.1"
  },
  "host": {
    "name": "78433ef87b5f"
  }
}

```

logstash 這裡
output {
   
        elasticsearch {
            hosts => ["esnode"]
            index => "%{[fields][index]}-%{+YYYY.MM.dd}"
        }

}

這樣便可以把 ECS log 收集到 elasticsearch中了


參考連結:
Autodiscover | Filebeat Reference [6.5] | Elastic https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover.html#_docker_2
Running Filebeat on Docker | Filebeat Reference [6.5] | Elastic https://www.elastic.co/guide/en/beats/filebeat/current/running-on-docker.html#_custom_image_configuration