|
|
@@ -0,0 +1,193 @@ |
|
|
# Send docker logs to ELK through gelf log driver |
|
|
|
|
|
There's so many way to send logs to an elk... logspout, filebeat, journalbeat, |
|
|
etc. |
|
|
|
|
|
But docker has a gelf log driver and logstash a gelf input. So here we are. |
|
|
|
|
|
[Here is a docker-compose](docker-compose.yml) to test a full elk with |
|
|
a container sending logs via gelf. |
|
|
|
|
|
## Try |
|
|
|
|
|
### Requirements |
|
|
|
|
|
- docker |
|
|
- docker-compose |
|
|
- 4 Go of ram (elastic is set to take 2 Go, and you will run logstash + kibana) |
|
|
|
|
|
### Versions |
|
|
|
|
|
The `docker-compose.yml` file run, from docker.elastic.co registry: |
|
|
|
|
|
- elasticsearch 5.3.0 |
|
|
- logstash 5.3.0 |
|
|
- kibana 5.3.0 |
|
|
|
|
|
The `docker-compose.old.yml` run, from docker hub: |
|
|
|
|
|
- elasticsearch 2.4 |
|
|
- logstash 2 |
|
|
- kibana 4 |
|
|
|
|
|
### Run |
|
|
|
|
|
1. Run the stack |
|
|
|
|
|
``` |
|
|
docker-compose -f docker-compose.yml up |
|
|
``` |
|
|
|
|
|
1. See in the logstash logs the incoming logs from `plop` container |
|
|
|
|
|
1. Wait for kibana to be up (and wait again if not...) |
|
|
|
|
|
1. Add `logstash-*` as index with `@timestamp` as Time-field name |
|
|
|
|
|
1. Go to `Discover` and see your logs! |
|
|
|
|
|
Even if your output is not well formated, you will see logs with metadata |
|
|
like command, image id, container id, timestamp, container name, etc. |
|
|
|
|
|
```json |
|
|
{ |
|
|
"_index": "logstash-2017.04.27", |
|
|
"_type": "docker", |
|
|
"_id": "AVuucsRtmwIbXgw8iFVb", |
|
|
"_score": null, |
|
|
"_source": { |
|
|
"source_host": "172.18.0.1", |
|
|
"level": 6, |
|
|
"created": "2017-04-26T14:15:52.292252751Z", |
|
|
"message": "My Message Thu Apr 27 08:06:48 UTC 2017", |
|
|
"type": "docker", |
|
|
"version": "1.1", |
|
|
"command": "/bin/sh -c while true; do echo My Message `date`; sleep 1; done;", |
|
|
"image_name": "alpine", |
|
|
"@timestamp": "2017-04-27T08:06:48.676Z", |
|
|
"container_name": "squarescaleweb_plop_1", |
|
|
"host": "plop-xps", |
|
|
"@version": "1", |
|
|
"tag": "", |
|
|
"image_id": "sha256:4a415e3663882fbc554ee830889c68a33b3585503892cc718a4698e91ef2a526", |
|
|
"container_id": "abbdfad9b05a45037327da369f0ae22c3f5477760f0e0d6bb00f796627b32706" |
|
|
}, |
|
|
"fields": { |
|
|
"created": [ |
|
|
1493216152292 |
|
|
], |
|
|
"@timestamp": [ |
|
|
1493280408676 |
|
|
] |
|
|
}, |
|
|
"sort": [ |
|
|
1493280408676 |
|
|
] |
|
|
} |
|
|
``` |
|
|
|
|
|
## Under the hood |
|
|
|
|
|
Logstash [defines](logstash.conf) an input of type `gelf` with port |
|
|
`12201`. |
|
|
|
|
|
It's `udp` port, so don't forget to correctly open it using `12201:12201/udp` |
|
|
in docker settings. |
|
|
|
|
|
And logstash send logs to `stdout` for debug and elasticsearch. |
|
|
|
|
|
The containers we want to see logs should define the logging configuration. |
|
|
In a `docker-compose` file in version 2: |
|
|
|
|
|
```yaml |
|
|
logging: |
|
|
driver: gelf |
|
|
options: |
|
|
gelf-address: udp://localhost:12201 |
|
|
``` |
|
|
|
|
|
Careful, the address to send the log is relative to the docker host, not the |
|
|
container! |
|
|
|
|
|
If you run docker instead of docker-compose: |
|
|
|
|
|
``` |
|
|
docker run --log-driver gelf --log-opt gelf-address=udp://localhost:12201 |
|
|
``` |
|
|
|
|
|
To organize your logs, you can add tag (only one): |
|
|
|
|
|
```yaml |
|
|
logging: |
|
|
options: |
|
|
tag: staging |
|
|
``` |
|
|
|
|
|
or |
|
|
|
|
|
``` |
|
|
docker run --log-opt tag="staging" |
|
|
``` |
|
|
|
|
|
And result is like: |
|
|
|
|
|
```json |
|
|
{ |
|
|
"_index": "logstash-2017.04.27", |
|
|
"_type": "docker", |
|
|
"_id": "AVuuiZbeYg9q2vv-JShe", |
|
|
"_score": null, |
|
|
"_source": { |
|
|
"source_host": "172.18.0.1", |
|
|
"level": 6, |
|
|
"created": "2017-04-27T08:24:45.69023959Z", |
|
|
"message": "My Message Thu Apr 27 08:31:44 UTC 2017", |
|
|
"type": "docker", |
|
|
"version": "1.1", |
|
|
"command": "/bin/sh -c while true; do echo My Message `date`; sleep 1; done;", |
|
|
"image_name": "alpine", |
|
|
"@timestamp": "2017-04-27T08:31:44.338Z", |
|
|
"container_name": "squarescaleweb_plop_1", |
|
|
"host": "plop-xps", |
|
|
"@version": "1", |
|
|
"tag": "staging", |
|
|
"image_id": "sha256:4a415e3663882fbc554ee830889c68a33b3585503892cc718a4698e91ef2a526", |
|
|
"container_id": "12b7bcd3f2f54e017680090d01330f542e629a4528f558323e33f7894ec6be53" |
|
|
}, |
|
|
"fields": { |
|
|
"created": [ |
|
|
1493281485690 |
|
|
], |
|
|
"@timestamp": [ |
|
|
1493281904338 |
|
|
] |
|
|
}, |
|
|
"sort": [ |
|
|
1493281904338 |
|
|
] |
|
|
} |
|
|
``` |
|
|
|
|
|
You can now filter your logs using the tag. |
|
|
|
|
|
## Resources |
|
|
|
|
|
[ADVENTURES IN GELF](https://blog.docker.com/2017/02/adventures-in-gelf/) and |
|
|
[Orchestration workshop](https://jpetazzo.github.io/orchestration-workshop/#248) from [Jérôme Petazzoni](https://twitter.com/jpetazzo). |
|
|
|
|
|
For the fun, just read the headers of the first link: |
|
|
|
|
|
- GELF |
|
|
- Using a logging driver |
|
|
- I would tell you an UDP joke, but |
|
|
- DNS to the rescue |
|
|
- Hmmm … TCP to the rescue, then? |
|
|
- (╯°□°)╯︵ ┻━┻ |
|
|
- Hot potato |
|
|
- Haters gonna hate |
|
|
- Workarounds |
|
|
- Future directions |
|
|
|
|
|
_Future directions_ is where it's written: |
|
|
|
|
|
> If you need better GELF support, I have good news: you can help! I’m not going to tell you “just send us a pull request, ha ha ha!” because I know that only a very small number of people have both the time and expertise to do that — but if you are one of them, then by all means, do it! |
eunomie created this gist