Day05 Filebeat(三) 正则表达式

接下来这一个章节，焦点还是会在filebeat上，通常在收集log，并不是所有资料都需要收集到Elasticsearch，而今日主题将会是如何使用正式表达式，来达收集到真正所需要的资料。

如何使用正则表达式

在filebeat提供下列二种方式来过滤资料

• exclude_lines (支援正则表达式)
  在输入资料中排除符合正则表达式的那些行数。
• include_lines (支援正则表达式)
  在输入资料中符合正则表达式的那些行数。

注: include_lines 执行完成後，才会再执行 exclude_lines

测试用的资料

"GET /navigation.php HTTP/1.1" 200 22486 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.58 Safari/537.36"
"POST /navigation.php HTTP/1.1" 200 22486 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.58 Safari/537.36"
"PUT /navigation.php HTTP/1.1" 200 22486 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.58 Safari/537.36"
"GET /navigation.php HTTP/1.1" 400 22486 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.58 Safari/537.36"
"POST /navigation.php HTTP/1.1" 400 22486 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.58 Safari/537.36"
"GET /navigation.php HTTP/1.1" 200 22486 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.58 Safari/537.36"

范例1

排除所有开头为GET的资料

#filebeat.yml
filebeat.inputs:

# 设定要抓取log的路径
- type: filestream
  enabled: true
  # 排除开头资料为 GET
  exclude_lines: ["^\"GET"]
  paths:
    - ./mylog.log

# 设定kibana
setup.kibana:
  host: "localhost:5601"

# 设定elasticsearch
output.elasticsearch:
  hosts: ["localhost:9200"]
  #设定索引名称
  index: "mylog-%{+yyyy.MM.dd}"

# 设定索引样板资讯
setup.template.name: "mylog"
setup.template.pattern: "mylog-*"
setup.ilm.enabled: false

elk得到资料如下:

Sep 8, 2021 @ 13:33:03.284	"PUT /navigation.php HTTP/1.1" 200 22486 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.58 Safari/537.36"

Sep 8, 2021 @ 13:33:03.284	"POST /navigation.php HTTP/1.1" 400 22486 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.58 Safari/537.36"

Sep 8, 2021 @ 13:33:00.646	"POST /navigation.php HTTP/1.1" 200 22486 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.58 Safari/537.36"

范例2

取出所有包含GET的资料

#filebeat.yml
filebeat.inputs:

# 设定要抓取log的路径
- type: filestream
  enabled: true
  # 资料有包含 GET
  include_lines: ["GET"]
  paths:
    - ./mylog.log

elk得到资料如下:

Sep 8, 2021 @ 13:46:13.559	"GET /navigation.php HTTP/1.1" 400 22486 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.58 Safari/537.36"

Sep 8, 2021 @ 13:46:13.559	"GET /navigation.php HTTP/1.1" 200 22486 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.58 Safari/537.36"

Sep 8, 2021 @ 13:46:13.558	"GET /navigation.php HTTP/1.1" 200 22486 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.58 Safari/537.36"

范例3

取出所有包含GET的资料，并排除400

#filebeat.yml
filebeat.inputs:

# 设定要抓取log的路径
- type: filestream
  enabled: true
  # 排除开头资料为 400
  exclude_lines: ["400"]
  # 资料有包含 GET
  include_lines: ["GET"]
  paths:
    - ./mylog.log

elk得到资料如下:

Sep 8, 2021 @ 13:50:02.269	"GET /navigation.php HTTP/1.1" 200 22486 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.58 Safari/537.36"

Sep 8, 2021 @ 13:49:59.636	"GET /navigation.php HTTP/1.1" 200 22486 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.58 Safari/537.36"