ELK + filebeat 7.x 설치(Ver. Windows)

ELK 다운로드 및 설치(v7.5.2)

• wget 유틸을 사용하여 다운로드 수행
• 압축 해제시 설치 완료됨
• Node1에 ELK 설치하며, web 서버에 filebeat 설치

#elasticsearch windows version download
cmd> wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.5.2-windows-x86_64.zip

#logstash window version 없음
cmd> wget https://artifacts.elastic.co/downloads/logstash/logstash-7.5.2.tar.gz

#kibana windows version download
cmd> wget https://artifacts.elastic.co/downloads/kibana/kibana-7.5.2-windows-x86_64.zip

#filebeat windows version download
cmd> wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.5.2-windows-x86_64.zip

 

 

Elasticsearch 설정(elasticsearch.yml)

• [중요] 단일 호스트 network.host 설정 시 discovery(cluster.initial_master_nodes) 설정 필수

  •  

    기존의 마스터 후보 장비 목록을 설정하던 discovery.zen.ping.unicast.hosts 와 Split Brain 을 막기 위한 discovery.zen.minimum_master_nodes 설정이 없어지고 discovery.seed_hosts 와 cluster.initial_master_nodes 설정이 위 설정들을 대체하게 됨
  • discovery.seed_hosts : 마스터 후보 장비 목록 설정(단일 호스트 경우 설정 필요 없음)
  • cluster.initial_master_nodes : 마스터 선출 가능 목록을 구성하는 설정(단일 호스트 경우 필수 설정)
    
    • split brain : 클러스터 구성에서 네트워크 단절로 인해 여러개의 노드가 서로 마스터로 인식되는 증상
    • 4대 마스터 운영시 최소 마스터 개수 (4/2+1) 3대, 2대가 내려가면 클러스터 중지시켜 split brain 방지 

# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: ECM
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: jacob-pc
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: D:\apps\elk\elasticsearch-7.5.2\data
#
# Path to log files:
#
path.logs: D:\apps\elk\elasticsearch-7.5.2\logs
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: xxx.xxx.xxx.xxx
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
# discovery.seed_hosts: ["xxx.xxx.xxx.xxx"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["xxx.xxx.xxx.xxx"]
#
# For more information, consult the discovery and cluster formation module documentation.
#

## 추가 필수 설정(기동 안됨)
xpack.ml.enabled: false

 

Elasticsearch 기동

cmd> cd %elasticsearch%\bin
cmd> elasticsearch.bat

 

Logstash 설정(logstash-xxx.conf)

• logstash.yml 파일 변경 사항 없음
• logstash-sample.conf 형식으로 conf(logstash-webtob-access-log.conf) 파일 생성
  • (예시)  beats에서 전달된 webtob access log를 파싱하여 elasticsearch 로 전달
  • 참고 URL
    • 정규 표현식 테스트 https://regex101.com
    • grok 테스트 http://grokdebug.herokuapp.com

#################################
#logstash-webtob-access-log.conf#
#################################

# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.

input {
  beats {
    port => 5044
  }
}

filter {
  grok { match => { "message" => [ "%{IPORHOST:[clientip]} \[%{HTTPDATE:[timestamp]}\] \"%{WORD:[method]} %{DATA:[request]} HTTP/%{NUMBER:[http_version]}\" %{NUMBER:[response]} (?:%{NUMBER:[bytes]}|-) %{NUMBER:[elaspsed]}" ] } }
date {
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss +0900" ]
target => "@timestamp"
        }
geoip {
source  => "clientip"
        }
   
ruby { code => "event.set('index_day',event.get('@timestamp').time.localtime('+09:00').strftime('%Y.%m.%d'))" }
mutate { remove_field => [ "host" ] }
}

output {
if "_grokparsefailure" in [tags] {
file { path => "D:\apps\elk\logstash-7.4.0\logs\logstash-webtob-access-log-grokparsefailure-%{index_day}.log" }
}else{
elasticsearch {
hosts => ["http://xxx.xxx.xxx.xxx:9200"]
index => "webtob-access-log-%{index_day}"
manage_template => "false"
}
}
}

 

Logstash 기동

cmd> cd %logstash%
cmd> bin\logstash.bat -f config\logstash-webtob-access-log.conf -r
# -r 옵션 : logstash-webtob-access-log.conf 변경시 자동 재시작

 

Filebeat 설정

• Logstash로 전달할 파일 경로 설정
• CASE 1 or CASE 2 택일하여 설정

#########################################################
# (CASE 1) filebeat.yml  경로 설정(들여쓰기 주의, 기동안됨)
#########################################################
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - Z:\uploadErrLog\2020-01-13\*.txt

###############################################################################################
# (CASE 2) [filebeat_home]/modules.d/scouter.yml  파일 생성 후 경로 설정(들여쓰기 주의, 기동안됨)
###############################################################################################
- module: scouter
  log :
    enabled: true
    var.paths:
    - Z:\uploadErrLog\2020-01-13\*.txt 

• (참고) CASE2 module 적용 방법

./filebeat modules enable scouter

• Filebeat에서 Logstash로 전달하기 위한 output 설정
• filebeat.yml output 변경 : Elasticsearch 비활성, Logstash 활성

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["xxx.xxx.xxx.xxx:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

 

 

Filebeat 기동

cmd> filebeat.exe -e -c filebeat.yml -d "publish"

 

Kibana 설정

• 서버 정보 및 Elasticsearch 설정 값 변경

# Kibana is served by a back end server. This setting specifies the port to use.
#server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "xxx.xxx.xxx.xxx"

# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""

# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
#server.rewriteBasePath: false

# The maximum payload size in bytes for incoming server requests.
# request entity too large 에러 발생시 설정
#server.maxPayloadBytes: 1048576 (default)

# The Kibana server's name.  This is used for display purposes.
#server.name: "your-hostname"

# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["http://xxx.xxx.xxx.xxx:9200"]


... 생략

 

Kibana 기동

cmd> cd %kibana%\bin
cmd> kibana.bat