Day06-filebeat，logstash多实例，pipline，ElasticStack项目架构梳理及实战案例

  • 开发
  • 17

Day06-filebeat，logstash多实例，pipline，ElasticStack项目架构梳理及实战案例

0、昨日内容:

  • logstash

    • input:

      用于接收数据:

      • beats

        • kafka
        • stdin

    • filter:

      • date
        处理日期字段。对时间字段进行格式化并转换为date类型。
      • grok
        基于正则匹配文本，将该字段提取出来。
      • geoip
        将公网IP地址进行解析，可以解析经纬度，国家，城市等信息。
      • mutate
      • user_agent
      • json

    • output:

      将数据发送到目的端。

      • elasticsearch
    • stdout

image-20240603164122902

测试数据:
OLDBOYedu2023 教室07

grok自定义正则模式:

[root@elk101.oldboyedu.com ~]# cat oldboyedu-linux85-patterns/jiaoshi07 
YEAR [\d]{4}
CLASSROOMNUMBER [0-9]{2}
TEACHER [A-Z]+
[root@elk101.oldboyedu.com ~]#

1、logstash的单分支和双分支

[root@elk101.oldboyedu.com ~]# cat config/06-tcp-grok_custom_pattern-es.conf 
input { 
  beats {
    port => 8888
    type => "beats"
  }

  tcp {
    port => 9999
    type => "tcp"
  }

  http {
    type => "http"
  }
} 

filter {
  if [type] == "beats" {
      grok {
         match => { "message" => "%{HTTPD_COMBINEDLOG}" }
         remove_field => [ "agent","log","input","host","ecs","tags" ]
      }
geoip {
     source => "clientip"
     add_field => {"custom-type" => "jiaoshi07-beats"}
  }
  
  date {
      match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
      timezone => "Asia/Shanghai"
      target => "oldboyedu-linux85-date"
    }
  }

  if [type] == "tcp" {
     grok {
         # 指定加载pattern匹配模式的目录，可以是相对路径，也可以是绝对路径
         patterns_dir => ["./oldboyedu-linux85-patterns"]
         # 基于指定字段进行匹配
         # match => { "message" => "%{TEACHER:teacher}edu%{YEAR:year} 教室%{CLASSROOMNUMBER:classroom_number}"}
         match => { "message" => "%{TEACHER:teacher}.{3}%{YEAR:year} 教室%{CLASSROOMNUMBER:classroom_number}"}
         add_field => {"custom-type" => "jiaoshi07-tcp"}
    }
  }else {
    mutate {
       add_field => { 
           "school" => "oldboyedu" 
           "class" => "linux85"
           "custom-type" => "jiaoshi07-http"
       } 
    }
  }

}

output { 
 stdout {} 

 #  elasticsearch {
 #    hosts => ["http://localhost:9200"]
 #    index => "oldboyedu-linux85-logstash-nginx"
 #  }
}
[root@elk101.oldboyedu.com ~]#

2、logstash的多分支案例

[root@elk101.oldboyedu.com ~]# cat config/07-tcp-grok_custom_pattern_if-es.conf 
input { 
  beats {
    port => 8888
    type => "beats"
  }

  tcp {
    port => 9999
    type => "tcp"
  }

  http {
    type => "http"
  }
} 

filter {
  if [type] == "beats" {
      grok {
         match => { "message" => "%{HTTPD_COMBINEDLOG}" }
         remove_field => [ "agent","log","input","host","ecs","tags" ]
      }
geoip {
     source => "clientip"
     add_field => {"custom-type" => "jiaoshi07-beats"}
  }
  
  date {
      match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
      timezone => "Asia/Shanghai"
      target => "oldboyedu-linux85-date"
  }
    } else if [type] == "tcp" {
     grok {
         # 指定加载pattern匹配模式的目录，可以是相对路径，也可以是绝对路径
         patterns_dir => ["./oldboyedu-linux85-patterns"]
         # 基于指定字段进行匹配
         # match => { "message" => "%{TEACHER:teacher}edu%{YEAR:year} 教室%{CLASSROOMNUMBER:classroom_number}"}
         match => { "message" => "%{TEACHER:teacher}.{3}%{YEAR:year} 教室%{CLASSROOMNUMBER:classroom_number}"}
         add_field => {"custom-type" => "jiaoshi07-tcp"}
    }
  }else {
    mutate {
       add_field => { 
           "school" => "oldboyedu" 
           "class" => "linux85"
           "custom-type" => "jiaoshi07-http"
       } 
    }
  }

}

output { 
 stdout {} 
 
 #  elasticsearch {
 #    hosts => ["http://localhost:9200"]
 #    index => "oldboyedu-linux85-logstash-nginx"
 #  }
}
[root@elk101.oldboyedu.com ~]#

使用多分支语法分别将"beat，tcp，http"这3个输入类型写入ES集群对应不同的索引:
oldboyedu-linux85-beats
oldboyedu-linux85-tcp
oldboyedu-linux85-http

3、filebeat多实例案例

[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# filebeat -e -c config/01-stdin-to-console.yaml 

[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# filebeat -e -c config/02-tcp-to-console.yaml  --path.data /tmp/oldboyedu-linux85-filebeat

4、logstash多实例

[root@elk101.oldboyedu.com ~]# cat config/09-multiple_instance-beats.conf 
input { 
  beats {
    port => 8888
    type => "beats"
  }
} 


filter {
   grok {
      match => { "message" => "%{HTTPD_COMBINEDLOG}" }
      remove_field => [ "agent","log","input","host","ecs","tags" ]
   }

   geoip {
      source => "clientip"
      add_field => {"custom-type" => "jiaoshi07-beats"}
   }

   date {
       match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
       timezone => "Asia/Shanghai"
       target => "oldboyedu-linux85-date"
   }
}

output { 

 # stdout {} 

  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "oldboyedu-linux85-multiple_instance-beats"
  }
}
[root@elk101.oldboyedu.com ~]# 
[root@elk101.oldboyedu.com ~]# logstash -rf config/09-multiple_instance-beats.conf 


[root@elk101.oldboyedu.com ~]# cat config/09-multiple_instance-http.conf 
input { 
  http {
    type => "http"
  }
} 

filter {
  mutate {
     add_field => { 
         "school" => "oldboyedu" 
         "class" => "linux85"
         "custom-type" => "jiaoshi07-http"
     } 
  }
}

output { 

 # stdout {} 

 elasticsearch {
   hosts => ["http://localhost:9200"]
   index => "oldboyedu-linux85-multiple_instance-http"
 }
}
[root@elk101.oldboyedu.com ~]# 
[root@elk101.oldboyedu.com ~]# logstash -rf config/09-multiple_instance-http.conf --path.data /tmp/oldboyedu-linux85-http

[root@elk101.oldboyedu.com ~]# cat config/09-multiple_instance-tcp.conf 
input { 
  tcp {
    port => 9999
    type => "tcp"
  }
} 

filter {
   grok {
       # 指定加载pattern匹配模式的目录，可以是相对路径，也可以是绝对路径
       patterns_dir => ["./oldboyedu-linux85-patterns"]
       # 基于指定字段进行匹配
       # match => { "message" => "%{TEACHER:teacher}edu%{YEAR:year} 教室%{CLASSROOMNUMBER:classroom_number}"}
       match => { "message" => "%{TEACHER:teacher}.{3}%{YEAR:year} 教室%{CLASSROOMNUMBER:classroom_number}"}
       add_field => {"custom-type" => "jiaoshi07-tcp"}
  }

}

output { 
 # stdout {} 

 elasticsearch {
   hosts => ["http://localhost:9200"]
   index => "oldboyedu-linux85-multiple_instance-tcp"
 }
}
[root@elk101.oldboyedu.com ~]# 
[root@elk101.oldboyedu.com ~]# logstash -rf config/09-multiple_instance-tcp.conf --path.data /tmp/oldboyedu-linux85-tcp

5、logstash的pipline案例

(1)编写配置文件

[root@elk101.oldboyedu.com ~]# cat config/10-pipeline-beats.conf 
input { 
  beats {
    port => 8888
    type => "beats"
  }
} 

filter {
   grok {
      match => { "message" => "%{HTTPD_COMBINEDLOG}" }
      remove_field => [ "agent","log","input","host","ecs","tags" ]
   }

   geoip {
      source => "clientip"
      add_field => {"custom-type" => "jiaoshi07-beats"}
   }

   date {
       match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
       timezone => "Asia/Shanghai"
       target => "oldboyedu-linux85-date"
   }
}

output {
 # stdout {} 

  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "oldboyedu-linux85-pipeline-beats"
  }
}
[root@elk101.oldboyedu.com ~]# 
[root@elk101.oldboyedu.com ~]# 
[root@elk101.oldboyedu.com ~]# cat config/10-pipeline-http.conf 
input { 
  http {
    type => "http"
  }
} 

filter {
  mutate {
     add_field => { 
         "school" => "oldboyedu" 
         "class" => "linux85"
         "custom-type" => "jiaoshi07-http"
     } 
  }
}

output {
 # stdout {} 

 elasticsearch {
   hosts => ["http://localhost:9200"]
   index => "oldboyedu-linux85-pipeline-http"
 }
}
[root@elk101.oldboyedu.com ~]# 
[root@elk101.oldboyedu.com ~]# 
[root@elk101.oldboyedu.com ~]# cat config/10-pipeline-tcp.conf 
input { 
  tcp {
    port => 9999
    type => "tcp"
  }
} 

filter {
   grok {
       # 指定加载pattern匹配模式的目录，可以是相对路径，也可以是绝对路径
       patterns_dir => ["./oldboyedu-linux85-patterns"]
       # 基于指定字段进行匹配
       # match => { "message" => "%{TEACHER:teacher}edu%{YEAR:year} 教室%{CLASSROOMNUMBER:classroom_number}"}
       match => { "message" => "%{TEACHER:teacher}.{3}%{YEAR:year} 教室%{CLASSROOMNUMBER:classroom_number}"}
       add_field => {"custom-type" => "jiaoshi07-tcp"}
  }

}

output { 
 # stdout {} 

 elasticsearch {
   hosts => ["http://localhost:9200"]
   index => "oldboyedu-linux85-pipeline-tcp"
 }
}
[root@elk101.oldboyedu.com ~]#

(2)修改pipline的配置文件

[root@elk101.oldboyedu.com ~]# yy /oldboyedu/softwares/logstash-7.17.5/config/pipelines.yml 
- pipeline.id: oldboyedu-linux85-pipeline-beats
  path.config: "/root/config/10-pipeline-beats.conf"
- pipeline.id: oldboyedu-linux85-pipeline-tcp
  path.config: "/root/config/10-pipeline-tcp.conf"
- pipeline.id: oldboyedu-linux85-pipeline-http
  path.config: "/root/config/10-pipeline-http.conf"

(3)启动logstash实例

logstash

6、logstash的useragent过滤器及kibana出图展示

(1)filebeat采集日志

[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# cat config/20-nginx-to-logstash.yaml 
filebeat.inputs:
- type: log
  paths:
    - /var/log/nginx/access.log*
      json:
      keys_under_root: true
      add_error_key: true
      overwrite_keys: true 

# 将数据输出到logstash中
output.logstash:
  # 指定logstash的主机和端口
  hosts: ["10.0.0.101:8888"]
[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# 
[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# filebeat -e -c config/20-nginx-to-logstash.yaml

(2)logstash分析数据

[root@elk101.oldboyedu.com ~]# cat config/11-beats-grok_geoip_date_useragent-es.conf 
input { 
  beats {
    port => 8888
  }
} 

filter {
   mutate {
      remove_field => [ "agent","log","input","host","ecs","tags" ]
   }

  geoip {
     source => "clientip"
  }

  date {
      match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
      timezone => "Asia/Shanghai"
  }

  # 用于分析客户端设备类型的插件
  useragent {
    # 指定基于哪个字段分析设备
    source => "http_user_agent"
    # 指定将解析的数据放在哪个字段，若不指定，则默认放在顶级字段中
    target => "oldboyedu-linux85-agent"
  }

}

output { 
# stdout {} 

 elasticsearch {
   hosts => ["http://localhost:9200"]
   index => "oldboyedu-linux85-logstash-nginx-useragent"
 }
}
[root@elk101.oldboyedu.com ~]# 
[root@elk101.oldboyedu.com ~]# logstash -rf config/11-beats-grok_geoip_date_useragent-es.conf

(3)kibana出图展示

image-20240604151416591

7、mutate组件数据准备-python脚本

cat > generate_log.py  <<EOF
#!/usr/bin/env python
# -*- coding: UTF-8 -*-
# @author : Jason Yin

import datetime
import random
import logging
import time
import sys

LOG_FORMAT = "%(levelname)s %(asctime)s [com.oldboyedu.%(module)s] - %(message)s "
DATE_FORMAT = "%Y-%m-%d %H:%M:%S"

# 配置root的logging.Logger实例的基本配置

logging.basicConfig(level=logging.INFO, format=LOG_FORMAT, datefmt=DATE_FORMAT, filename=sys.argv[1]
, filemode='a',)
actions = ["浏览页面", "评论商品", "加入收藏", "加入购物车", "提交订单", "使用优惠券", "领取优惠券",
 "搜索", "查看订单", "付款", "清空购物车"]

while True:
    time.sleep(random.randint(1, 5))
    user_id = random.randint(1, 10000)
    # 对生成的浮点数保留2位有效数字.
    price = round(random.uniform(15000, 30000),2)
    action = random.choice(actions)
    svip = random.choice([0,1])
    logging.info("DAU|{0}|{1}|{2}|{3}".format(user_id, action,svip,price))
EOF
nohup python generate_log.py  /tmp/apps.log &>/dev/null &

logstash编写:

[root@elk101.oldboyedu.com ~]# cat config/12-beats-mutate-es.conf 
input { 
  beats {
    port => 9999
  }
} 


filter {
   mutate {
      remove_field => [ "agent","log","input","host","ecs","tags" ]
   }

   mutate {
      # 将message字段使用"|"进行切分
      split => { "message" => "|" }
   }

   mutate {
     add_field => {
        userid => "%{[message][1]}"
        verb => "%{[message][2]}"
        svip => "%{[message][3]}"
        price => "%{[message][4]}"
     }
   }

   mutate {
     rename => {
        "verb" => "action"
     }
   }

   mutate {
     convert => {
       "userid" => "integer"
       "svip" => "boolean"
       "price" => "float"
     }
   }


}

output { 
 # stdout {} 

 elasticsearch {
   hosts => ["http://localhost:9200"]
   index => "oldboyedu-linux85-logstash-nginx-mutate"
 }
}
[root@elk101.oldboyedu.com ~]# 
[root@elk101.oldboyedu.com ~]# logstash -rf config/12-beats-mutate-es.conf

filebeat编写:

[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# cat config/21-apps-to-logstash.yaml
filebeat.inputs:
- type: log
  paths:
    - /tmp/apps.log 

output.logstash:
  hosts: ["10.0.0.101:9999"]
[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# 
[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# filebeat -e -c config/21-apps-to-logstash.yaml

8、将nginx日志分析，通过kibana展示数据，pv，带宽总量，公网IP的Top10统计等信息。

image-20240604160254052

项目案例：
logstash配置文件

[root@elk101.oldboyedu.com ~]# cat config/13-procect.conf 
input { 
  beats {
    port => 7777
  }
} 

filter {
   mutate {
      remove_field => [ "agent","log","input","host","ecs","tags" ]
   }

  geoip {
     source => "clientip"
  }

  date {
      match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
      timezone => "Asia/Shanghai"
  }

  useragent {
    source => "http_user_agent"
    # target => "oldboyedu-linux85-agent"
  }

}

output { 
  # stdout {} 

  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "oldboyedu-linux85-logstash-nginx-project-%{+yyyy.MM.dd}"
  }
}
[root@elk101.oldboyedu.com ~]#

filebeat配置:

[root@elk103.oldboyedu.com ~]# cat /oldboyedu/softwares/filebeat-7.17.5-linux-x86_64/config/22-project.yaml 
filebeat.inputs:
- type: log
  paths:
    - /var/log/nginx/access.log*
  json:
      keys_under_root: true
      add_error_key: true
      overwrite_keys: true 

output.logstash:
  hosts: ["10.0.0.101:7777"]
[root@elk103.oldboyedu.com ~]#

9、地理位置案例

01-创建索引映射

PUT	http://10.0.0.103:9200/oldboyedu-map
{
  "mappings": {
    "properties": {
      "location": { 
        "type": "geo_point"
      }
    }
  }
}

02-写入地理位置-lat代表纬度,lon代表经度

POST http://10.0.0.103:9200/oldboyedu-map/_doc
{
  "location": { 
    "lat": 39.914,
    "lon": 116.386
  }
}

03-批量地理位置

{ "create" : { "_index" : "oldboyedu-map" } }
{ "location": { "lat": 24,"lon": 121 }}
{ "create" : { "_index" : "oldboyedu-map" } }
{ "location": { "lat": 36.61,"lon": 114.488 }}
{ "create" : { "_index" : "oldboyedu-map" } }
{ "location": { "lat": 39.914,"lon": 116.386 }}

10、修复nginx日志解析经纬度问题故障演练

01-修改nginx的索引的地理位置映射

PUT http://10.0.0.103:9200/oldboyedu-linux82-project-nginx
{
    "mappings": {
        "properties": {
            "geoip": {
                "properties": {
                    "location": {
                        "type": "geo_point"
                    }
                }
            }
        }
    }
}

02-批量创建测试地理位置数据

POST	http://10.0.0.103:9200/_bulk
{ "create" : { "_index" : "oldboyedu-linux82-project-nginx" } }
{ "geoip.location": { "lat": 25,"lon": 121 }}
{ "create" : { "_index" : "oldboyedu-linux82-project-nginx" } }
{ "geoip.location": { "lat": 35.61,"lon": 114.488 }}
{ "create" : { "_index" : "oldboyedu-linux82-project-nginx" } }
{ "geoip.location": { "lat": 35.914,"lon": 116.386 }}
{ "create" : { "_index" : "oldboyedu-linux82-project-nginx" } }
{ "geoip.location": { "lat": 45.914,"lon": 118.386 }}
{ "create" : { "_index" : "oldboyedu-linux82-project-nginx" } }
{ "geoip.location": { "lat": 55.914,"lon": 126.386 }}
{ "create" : { "_index" : "oldboyedu-linux82-project-nginx" } }
{ "geoip.location": { "lat": 75.914,"lon": 26.386 }}
{ "create" : { "_index" : "oldboyedu-linux82-project-nginx" } }
{ "geoip.location": { "lat": 85.914,"lon": 16.386 }}

image-20240605174932103

filebeat的modules实战案例-了解

[root@elk103 filebeat-7.17.5-linux-x86_64]# filebeat modules list
Enabled:

Disabled:
activemq
apache
auditd
aws
awsfargate
....

[root@elk103 filebeat-7.17.5-linux-x86_64]# ll modules.d/
总用量 300
-rw-r--r-- 1 root root   484 2022-06-24 07:24 activemq.yml.disabled
-rw-r--r-- 1 root root   476 2022-06-24 07:24 apache.yml.disabled
-rw-r--r-- 1 root root   281 2022-06-24 07:24 auditd.yml.disabled
-rw-r--r-- 1 root root  2112 2022-06-24 07:24 awsfargate.yml.disabled
-rw-r--r-- 1 root root 10575 2022-06-24 07:24 aws.yml.disabled
-rw-r--r-- 1 root root  1707 2022-06-24 07:24 azure.yml.disabled
...

[root@elk103 filebeat-7.17.5-linux-x86_64]# filebeat modules enable nginx tomcat
Enabled nginx
Enabled tomcat
[root@elk103 filebeat-7.17.5-linux-x86_64]# filebeat modules list
Enabled:
nginx
tomcat

Disabled:
activemq
apache
auditd
aws
...
[root@elk103 filebeat-7.17.5-linux-x86_64]# filebeat modules disable tomcat
Disabled tomcat
[root@elk103 filebeat-7.17.5-linux-x86_64]# filebeat modules list
Enabled:
nginx

Disabled:
activemq
apache
auditd
[root@elk103 filebeat-7.17.5-linux-x86_64]# cat modules.d/nginx.yml
# Module: nginx
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.17/filebeat-module-nginx.html

- module: nginx
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/tmp/oldboyedu-linux85/access.log"]

  # Error logs
  error:
    enabled: false

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

  # Ingress-nginx controller logs. This is disabled by default. It could be used in Kubernetes environments to parse ingress-nginx logs
  ingress_controller:
    enabled: false

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

image-20240606092237249
今日作业:
完成的课堂的所有练习并整理思维导图.

阅读全部

相关推荐

  3. 项目是如何实现多租户架构

    2024-07-16 14:12:04       27 阅读

  7. Python实战开发及案例分析（3）——多目标优化

    2024-07-16 14:12:04       36 阅读

最近更新

  3. docker php8.1+nginx base 镜像 dockerfile 配置

    2024-07-16 14:12:04       67 阅读

  4. Could not load dynamic library ‘cudart64_100.dll‘

    2024-07-16 14:12:04       72 阅读

  9. 在Django里面运行非项目文件

    2024-07-16 14:12:04       58 阅读

  19. Python语言-面向对象

    2024-07-16 14:12:04       69 阅读

  20. 如何分清楚常见的 Git 分支管理策略Git Flow、GitHub Flow 和 GitLab Flow

    2024-07-16 14:12:04       64 阅读

热门阅读

  6. ChatGPT对话：如何把Html文件转换为Markdown文件

    2024-07-16 14:12:04       18 阅读

  7. 第2部分：物联网模式在行动

    2024-07-16 14:12:04       19 阅读

  8. AI推介-大语言模型LLMs论文速览（arXiv方向）：2024.07.10-2024.07.15

    2024-07-16 14:12:04       20 阅读

  11. c# 在线程中访问ui元素

    2024-07-16 14:12:04       22 阅读

  14. [240716] AMD 收购 Silo AI 扩展其 AI 解决方案 | 亚马逊云科技推出 Amazon App Studio

    2024-07-16 14:12:04       17 阅读

  15. C语言入门-7.结构体与C++引用

    2024-07-16 14:12:04       23 阅读

  17. Python3 第二十二课 -- 装饰器

    2024-07-16 14:12:04       28 阅读

  23. moment()获取时间

    2024-07-16 14:12:04       21 阅读

  29. 【Vue】 style中的scoped

    2024-07-16 14:12:04       18 阅读