安装部署ELK，并收集tomcat，nginx日志

一、ELK部署

准备三台虚拟机

elk-node01、elk-node02、elk-node03

三台机器都是 CentOS 7.6 ，内存 >=3G 1 、设置主机名和 hosts 解析

[root@elk-node01 ~]# tail -3 /etc/hosts 
192.168.163.143 elk-node01 
192.168.163.147 elk-node02 
192.168.163.146 elk-node03

2 、时间同步 3 、部署 jdk

 三台机器都部署jdk，建议内存3G以上

[root@elk-node01 ~]# rpm -ivh jdk-8u144-linux-x64.rpm 
[root@elk-node01 ~]# java -version
java version "1.8.0_144" 
Java(TM) SE Runtime Environment (build 1.8.0_144-b01) 
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

部署ElasticSearch集群环境

[root@elk-node01 ~]# cat /etc/yum.repos.d/elk.repo 
[elk]
name=elk 7.x
baseurl=https://mirrors.tuna.tsinghua.edu.cn/elasticstack/yum/elastic-7.x/ 
gpgcheck=0

 或者离线包安装

[root@elk-node01 ~]# yum install -y elasticsearch-7.2.0

配置Elasticsearch集群

elk-node01 节点的配置

[root@elk-node01 ~]# cp /etc/elasticsearch/elasticsearch.yml{,.bak} 
[root@elk-node01 ~]# grep '^[a-Z]' /etc/elasticsearch/elasticsearch.yml 
cluster.name: my-elk 
node.name: elk-node01 
path.data: /var/lib/elasticsearch 
path.logs: /var/log/elasticsearch 
network.host: 192.168.163.143
http.port: 9200 
discovery.seed_hosts: ["elk-node01", "elk-node02", "elk-node03"] cluster.initial_master_nodes: ["elk-node01"] 
# 主节点相关配置 加在最后
node.master: true 
node.data: false 
node.ingest: false 
node.ml: false 
cluster.remote.connect: false

安装head插件

[root@elk-node01 ~]#  yum install -y nodejs npm

下载head插件

cd /var/lib/elasticsearch/

wget https://github.com/mobz/elasticsearch-head/archive/master.zip

#解压
yum install unzip
unzip master.zip
(3)安装依赖包

yum install openssl bzip2 unzip -y
下载运行head必要的文件（放置在文件夹/tmp下）
cd /tmp 
wget https://npm.taobao.org/mirrors/phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2

用以下命令把下载到的包添加到npm cache目录中 
npm cache add phantomjs

cd elasticsearch-head-master/

npm install -g cnpm --registry=https://registry.npm.taobao.org
# 安装依赖
cnpm install

修改配置文件

vim Gruntfile.js
#找到并修改
options: {
				port: 9100, 
				base: '.', 
				keepalive: true,
				 hostname: '*' 
		 }

修改 elasticsearch-head 默认连接地址，将 localhost 改为本机 IP

# vim _site/app.js 
this.base_uri = this.config.base_uri || this.prefs.get("app-base_uri") || "http:192.168.163.143:9200";

修改elasticSearch配置文件并启动ElasticSearch

//追加下列两行实现跨域访问
http.cors.enabled: true 
http.cors.allow-origin: "*"

[root@elk-node01 ~]systemctl restart elasticsearch
启动插件：
# cd /var/lib/elasticsearch/elasticsearch-head-master/
# nohup ./node_modules/grunt/bin/grunt server &

访问 IP:9100 就能看到我们集群信息

 filebeat收集nginx的json格式日志

1、filebeat配置

[root@node3 filebeat]# cat /etc/filebeat/nginx.yml
filebeat.inputs:
    - type: log
      enabled: true
      json.keys_under_root: true
      json.overwrite_keys: true
      paths:
        - /var/log/nginx/access.log
      fields:
        log_topics: nginx
output.logstash:
    hosts: ["127.0.0.1:10001"]

nginx配置（修改日志格式）

 logstash配置

[root@node3 conf.d]# cat nginx.conf
input {
    beats {
      port=>10001
    }
}
output {
    if [fields][log_topics]=="nginx"{
    elasticsearch {
     hosts=>["192.168.163.143:9200"]
     index=>"nginx-%{+YYYY.MM.dd}"
    }
   }
}

[root@localhost conf.d]# logstash -f nginx.conf
[root@localhost filebeat]# filebeat -e -c nginx_json.yml