ELK + filebeat 7.x 설치(Ver. Windows)

    ELK 다운로드 및 설치(v7.5.2)

    • wget 유틸을 사용하여 다운로드 수행
    • 압축 해제시 설치 완료됨
    • Node1에 ELK 설치하며, web 서버에 filebeat 설치
    #elasticsearch windows version download
    cmd> wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.5.2-windows-x86_64.zip
    
    #logstash window version 없음
    cmd> wget https://artifacts.elastic.co/downloads/logstash/logstash-7.5.2.tar.gz
    
    #kibana windows version download
    cmd> wget https://artifacts.elastic.co/downloads/kibana/kibana-7.5.2-windows-x86_64.zip
    
    #filebeat windows version download
    cmd> wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.5.2-windows-x86_64.zip

     

     

    Elasticsearch 설정(elasticsearch.yml)

    • [중요] 단일 호스트 network.host 설정 시 discovery(cluster.initial_master_nodes) 설정 필수
      •  

        기존의 마스터 후보 장비 목록을 설정하던 discovery.zen.ping.unicast.hosts 와 Split Brain 을 막기 위한 discovery.zen.minimum_master_nodes 설정이 없어지고 discovery.seed_hosts 와 cluster.initial_master_nodes 설정이 위 설정들을 대체하게 됨
      • discovery.seed_hosts : 마스터 후보 장비 목록 설정(단일 호스트 경우 설정 필요 없음)
      • cluster.initial_master_nodes : 마스터 선출 가능 목록을 구성하는 설정(단일 호스트 경우 필수 설정)
        • split brain : 클러스터 구성에서 네트워크 단절로 인해 여러개의 노드가 서로 마스터로 인식되는 증상
        • 4대 마스터 운영시 최소 마스터 개수 (4/2+1) 3대, 2대가 내려가면 클러스터 중지시켜 split brain 방지 
    # ---------------------------------- Cluster -----------------------------------
    #
    # Use a descriptive name for your cluster:
    #
    cluster.name: ECM
    #
    # ------------------------------------ Node ------------------------------------
    #
    # Use a descriptive name for the node:
    #
    node.name: jacob-pc
    #
    # Add custom attributes to the node:
    #
    #node.attr.rack: r1
    #
    # ----------------------------------- Paths ------------------------------------
    #
    # Path to directory where to store the data (separate multiple locations by comma):
    #
    path.data: D:\apps\elk\elasticsearch-7.5.2\data
    #
    # Path to log files:
    #
    path.logs: D:\apps\elk\elasticsearch-7.5.2\logs
    #
    # ---------------------------------- Network -----------------------------------
    #
    # Set the bind address to a specific IP (IPv4 or IPv6):
    #
    network.host: xxx.xxx.xxx.xxx
    #
    # Set a custom port for HTTP:
    #
    http.port: 9200
    #
    # For more information, consult the network module documentation.
    #
    # --------------------------------- Discovery ----------------------------------
    #
    # Pass an initial list of hosts to perform discovery when this node is started:
    # The default list of hosts is ["127.0.0.1", "[::1]"]
    #
    # discovery.seed_hosts: ["xxx.xxx.xxx.xxx"]
    #
    # Bootstrap the cluster using an initial set of master-eligible nodes:
    #
    cluster.initial_master_nodes: ["xxx.xxx.xxx.xxx"]
    #
    # For more information, consult the discovery and cluster formation module documentation.
    #
    
    ## 추가 필수 설정(기동 안됨)
    xpack.ml.enabled: false
     

    Elasticsearch 기동

    cmd> cd %elasticsearch%\bin
    cmd> elasticsearch.bat

     

    Logstash 설정(logstash-xxx.conf)

    • logstash.yml 파일 변경 사항 없음
    • logstash-sample.conf 형식으로 conf(logstash-webtob-access-log.conf) 파일 생성
    #################################
    #logstash-webtob-access-log.conf#
    #################################
    
    # Sample Logstash configuration for creating a simple
    # Beats -> Logstash -> Elasticsearch pipeline.
    
    input {
      beats {
        port => 5044
      }
    }
    
    filter {
      grok { match => { "message" => [ "%{IPORHOST:[clientip]} \[%{HTTPDATE:[timestamp]}\] \"%{WORD:[method]} %{DATA:[request]} HTTP/%{NUMBER:[http_version]}\" %{NUMBER:[response]} (?:%{NUMBER:[bytes]}|-) %{NUMBER:[elaspsed]}" ] } }
    date {
    match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss +0900" ]
    target => "@timestamp"
            }
    geoip {
    source  => "clientip"
            }
       
    ruby { code => "event.set('index_day',event.get('@timestamp').time.localtime('+09:00').strftime('%Y.%m.%d'))" }
    mutate { remove_field => [ "host" ] }
    }
    
    output {
    if "_grokparsefailure" in [tags] {
    file { path => "D:\apps\elk\logstash-7.4.0\logs\logstash-webtob-access-log-grokparsefailure-%{index_day}.log" }
    }else{
    elasticsearch {
    hosts => ["http://xxx.xxx.xxx.xxx:9200"]
    index => "webtob-access-log-%{index_day}"
    manage_template => "false"
    }
    }
    }

     

    Logstash 기동

    cmd> cd %logstash%
    cmd> bin\logstash.bat -f config\logstash-webtob-access-log.conf -r
    # -r 옵션 : logstash-webtob-access-log.conf 변경시 자동 재시작

     

    Filebeat 설정

    • Logstash로 전달할 파일 경로 설정
      • CASE 1 or CASE 2 택일하여 설정
    #########################################################
    # (CASE 1) filebeat.yml  경로 설정(들여쓰기 주의, 기동안됨)
    #########################################################
    filebeat.inputs:
    - type: log
      enabled: true
      paths:
        - Z:\uploadErrLog\2020-01-13\*.txt
    
    ###############################################################################################
    # (CASE 2) [filebeat_home]/modules.d/scouter.yml  파일 생성 후 경로 설정(들여쓰기 주의, 기동안됨)
    ###############################################################################################
    - module: scouter
      log :
        enabled: true
        var.paths:
        - Z:\uploadErrLog\2020-01-13\*.txt 
      • (참고) CASE2 module 적용 방법
    ./filebeat modules enable scouter
    • Filebeat에서 Logstash로 전달하기 위한 output 설정
      • filebeat.yml output 변경 : Elasticsearch 비활성, Logstash 활성
    #================================ Outputs =====================================
    
    # Configure what output to use when sending the data collected by the beat.
    
    #-------------------------- Elasticsearch output ------------------------------
    #output.elasticsearch:
      # Array of hosts to connect to.
      #hosts: ["localhost:9200"]
    
      # Optional protocol and basic auth credentials.
      #protocol: "https"
      #username: "elastic"
      #password: "changeme"
    
    #----------------------------- Logstash output --------------------------------
    output.logstash:
      # The Logstash hosts
      hosts: ["xxx.xxx.xxx.xxx:5044"]
    
      # Optional SSL. By default is off.
      # List of root certificates for HTTPS server verifications
      #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
    
      # Certificate for SSL client authentication
      #ssl.certificate: "/etc/pki/client/cert.pem"
    
      # Client Certificate Key
      #ssl.key: "/etc/pki/client/cert.key"

     

     

    Filebeat 기동

    cmd> filebeat.exe -e -c filebeat.yml -d "publish"
     

    Kibana 설정

      • 서버 정보 및 Elasticsearch 설정 값 변경
    # Kibana is served by a back end server. This setting specifies the port to use.
    #server.port: 5601
    
    # Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
    # The default is 'localhost', which usually means remote machines will not be able to connect.
    # To allow connections from remote users, set this parameter to a non-loopback address.
    server.host: "xxx.xxx.xxx.xxx"
    
    # Enables you to specify a path to mount Kibana at if you are running behind a proxy.
    # Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
    # from requests it receives, and to prevent a deprecation warning at startup.
    # This setting cannot end in a slash.
    #server.basePath: ""
    
    # Specifies whether Kibana should rewrite requests that are prefixed with
    # `server.basePath` or require that they are rewritten by your reverse proxy.
    # This setting was effectively always `false` before Kibana 6.3 and will
    # default to `true` starting in Kibana 7.0.
    #server.rewriteBasePath: false
    
    # The maximum payload size in bytes for incoming server requests.
    # request entity too large 에러 발생시 설정
    #server.maxPayloadBytes: 1048576 (default)
    
    # The Kibana server's name.  This is used for display purposes.
    #server.name: "your-hostname"
    
    # The URLs of the Elasticsearch instances to use for all your queries.
    elasticsearch.hosts: ["http://xxx.xxx.xxx.xxx:9200"]
    
    
    ... 생략

     

    Kibana 기동

    cmd> cd %kibana%\bin
    cmd> kibana.bat

    댓글

    Designed by JB FACTORY