1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
|
Kafka destination
=================
Here is a simple configuration sending the messages on a dedicated
Kafka queue (`syslog-ng`) using Logstash's JSON event layout:
```
source s_system {
system();
};
destination d_kafka {
channel {
rewrite {
set("${HOST}" value(".eventv1.host"));
set("1" value(".eventv1.@version"));
set("${ISODATE}" value(".eventv1.@timestamp") condition("${.eventv1.@timestamp}" eq ""));
set("${MESSAGE}" value(".eventv1.message") condition("${.eventv1.message}" eq ""));
set("${MSG}" value(".eventv1.message") condition("${.eventv1.message}" eq ""));
set("generic" value(".eventv1.type") condition("${.eventv1.type}" eq ""));
};
destination {
kafka-c(config(metadata.broker.list("localhost:9092")
queue.buffering.max.ms("1000"))
topic("test")
message("$(format-json --key .eventv1.* --rekey .eventv1.* --shift 9)"));
};
};
};
log {
source(s_system);
destination(d_kafka);
};
```
Compilation
-----------
You need [librdkafka](https://github.com/edenhill/librdkafka/). Once
installed, compile with `--with-librdkafka`.
Running Kafka
-------------
If you are not too familiar with Kafka, a simple recipe
[Kafka](https://kafka.apache.org/quickstart) can get you started in a minute.
Once the Kafka zookeeper, server are running and topic is created, start Syslog-ng with the above settings in the configuration file. The logs sent by Syslog-ng will be seen in kafka consumer.
Another useful tool is
[kafkacat](https://github.com/edenhill/kafkacat). For example, to look at logs sent to Kafka, use:
```
kafkacat -C -u -b localhost -t syslog-ng
```
|
