Fluent bit回顾
Log Agent - Fluent Bit 简介
Log Agent - Fluent Bit 安装与常见架构模式
Log Agent - Fluent Bit Service配置与内建 API
Log Agent - Fluent Bit Input元件 与 Tail浅谈
Log Agent - Fluent Bit Parser元件

Multiline Parsing

昨天介绍的Regex Parser其实只适用於单行的Log资料.
因为Tail是读取一行就往Parser送.

所以如果Log资料本来就是Multiline多行的.
就需要用Multiline Parser, 对Regex Parser做点处理

内建的Multiline Parsers

针对有些环境所产出的Log, Fluent bit有内建好几个Multiline Parser
就不必自己刻写了

• Docker
• CRI
• Go
• Pythn
• Java

但我们也能自定义Multiline Parser
上一篇提到的Parser, 它都是[Parser]这样作为section name
但Multiline Parser则是[MULTILINE_PARSER]作为section name
然後Parser与MULTILINE_PARSER都建议别直接配置在fluent-bit.conf
需要另外拉一个parsers.conf或者是parsers_multiline.conf
在fluent-bit.conf做import

[SERVICE]
    flush 1
    Daemon off
    log_level info
    parsers_file parsers.conf
    parsers_file parsers_multiline.conf

然後MULTILINE_PARSER需要设定几个properties

• Name
  • 就Multiline Parser的name
• type
  • 设定成regex
• rule
  • 用rule来写regex, 使得让multiline parser知道第一行的样貌跟读到怎样的样貌是结束
  • 下面范例, start_state 就是起始状态的名字 只要符合其regex pattern的就是多行Log的第一行
  • start_state匹配到第一行後, 就看有没有next state, 这里指定下一个state是cont
  • 就继续读下一行, 判断是不是匹配start_state, 不是就拿现在状态的cont的regex patern来继续匹配
  • 这样直到某一行读到, 它是匹配start_state的, 就是下一段的多行了

# rules   |   state name   | regex pattern         | next state name
# --------|----------------|----------------------------------------
    rule     "start_state"   "/(Dec \d+ \d+\:\d+\:\d+)(.*)/"  "cont"
    rule     "cont"          "/^\s+at.*/"                     "cont"

来个范例
fluent-bit.conf
这里input的multiline.parser,
我们测试内建的Go multiline parser和自定义的parser
多个parser用,做分隔就好

[INPUT]    
    Name        tail    
    Path        /var/log/demo/demo.log
    read_from_head   true
    multiline.parser      multiline-regex-test, go

parsers_multiline.conf

[MULTILINE_PARSER]
    name          multiline-regex-test
    type          regex
    rule      "start_state"   "/(Dec \d+ \d+\:\d+\:\d+)(.*)/"  "cont"
    rule      "cont"          "/^\s+at.*/"                     "cont"

测试log

Dec 14 06:41:08 Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
    at com.myproject.module.MyProject.badMethod(MyProject.java:22)
    at com.myproject.module.MyProject.oneMoreMethod(MyProject.java:18)
    at com.myproject.module.MyProject.anotherMethod(MyProject.java:14)
    at com.myproject.module.MyProject.someMethod(MyProject.java:10)
    at com.myproject.module.MyProject.main(MyProject.java:6)

Dec 14 06:41:09 Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
Dec 14 06:41:10 Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
Dec 14 06:41:11 Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
    at com.myproject.module.MyProject.badMethod(MyProject.java:22)
    at com.myproject.module.MyProject.oneMoreMethod(MyProject.java:18)
    at com.myproject.module.MyProject.anotherMethod(MyProject.java:14)
    at com.myproject.module.MyProject.someMethod(MyProject.java:10)
    at com.myproject.module.MyProject.main(MyProject.java:6)
    
panic: my panic

goroutine 4 [running]:
panic(0x45cb40, 0x47ad70)
  /usr/local/go/src/runtime/panic.go:542 +0x46c fp=0xc42003f7b8 sp=0xc42003f710 pc=0x422f7c
main.main.func1(0xc420024120)
  foo.go:6 +0x39 fp=0xc42003f7d8 sp=0xc42003f7b8 pc=0x451339
runtime.goexit()
  /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003f7e0 sp=0xc42003f7d8 pc=0x44b4d1
created by main.main
  foo.go:5 +0x58
panic: my panic

goroutine 4 [running]:
panic(0x45cb40, 0x47ad70)
  /usr/local/go/src/runtime/panic.go:542 +0x46c fp=0xc42003f7b8 sp=0xc42003f710 pc=0x422f7c
main.main.func1(0xc420024120)
  foo.go:6 +0x39 fp=0xc42003f7d8 sp=0xc42003f7b8 pc=0x451339
runtime.goexit()
  /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003f7e0 sp=0xc42003f7d8 pc=0x44b4d1
created by main.main
  foo.go:5 +0x58

output
可以看到[0]这结构化日志, 是多行的, 这麽多行才整理成一笔结构化日志做Output
而不会笨笨的一行就输出一笔出去
[1]、[2]、[3]则是来玩看看, 那两个state状态的匹配顺序与切换
[1]那行匹配到了start_state,
接着下一行[2], 又匹配到了start_state, 就表示[1]能输出了
接着下一行[3],又匹配到了start_state, 就表示[3]能输出了

再来的一行是Go的, 就怎样也不匹配start_state和cont这两个state, 就换注入的下一个parser解析看看

Go的panic log与自定义的Java log都被multiline parser给正确解析

fluentd_1  | [0] tail.0: [1634142611.283200049, {"log"=>"Dec 14 06:41:08 Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
fluentd_1  |     at com.myproject.module.MyProject.badMethod(MyProject.java:22)
fluentd_1  |     at com.myproject.module.MyProject.oneMoreMethod(MyProject.java:18)
fluentd_1  |     at com.myproject.module.MyProject.anotherMethod(MyProject.java:14)
fluentd_1  |     at com.myproject.module.MyProject.someMethod(MyProject.java:10)
fluentd_1  |     at com.myproject.module.MyProject.main(MyProject.java:6)
fluentd_1  | "}]
fluentd_1  | [1] tail.0: [1634142611.283256734, {"log"=>"Dec 14 06:41:09 Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
fluentd_1  | "}]
fluentd_1  | [2] tail.0: [1634142611.283260341, {"log"=>"Dec 14 06:41:10 Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
fluentd_1  | "}]
fluentd_1  | [3] tail.0: [1634142611.283263447, {"log"=>"Dec 14 06:41:11 Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
fluentd_1  |     at com.myproject.module.MyProject.badMethod(MyProject.java:22)
fluentd_1  |     at com.myproject.module.MyProject.oneMoreMethod(MyProject.java:18)
fluentd_1  |     at com.myproject.module.MyProject.anotherMethod(MyProject.java:14)
fluentd_1  |     at com.myproject.module.MyProject.someMethod(MyProject.java:10)
fluentd_1  |     at com.myproject.module.MyProject.main(MyProject.java:6)
fluentd_1  | "}]
fluentd_1  | [4] tail.0: [1634143058.181373085, {"log"=>"panic: my panic
fluentd_1  | 
fluentd_1  | goroutine 4 [running]:
fluentd_1  | panic(0x45cb40, 0x47ad70)
fluentd_1  |   /usr/local/go/src/runtime/panic.go:542 +0x46c fp=0xc42003f7b8 sp=0xc42003f710 pc=0x422f7c
fluentd_1  | main.main.func1(0xc420024120)
fluentd_1  |   foo.go:6 +0x39 fp=0xc42003f7d8 sp=0xc42003f7b8 pc=0x451339
fluentd_1  | runtime.goexit()
fluentd_1  |   /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003f7e0 sp=0xc42003f7d8 pc=0x44b4d1
fluentd_1  | created by main.main
fluentd_1  |   foo.go:5 +0x58
fluentd_1  | "}]

其实也能在Filter再做multiline.parser的设定,
让Input单纯的只做监听与捞取资料