ELK 简易 Nginx 日志系统搭建: ElasticSearch+Kibana+Filebeat

> 本文由 [简悦 SimpRead](http://ksria.com/simpread/) 转码， 原文地址 [www.tuicool.com](https://www.tuicool.com/articles/qmmyiu3)

该教程以 ubuntu 16.04 为例, 其他系统类似

官网: [https://www.elastic.co](https://www.elastic.co)

使用 ElasticSearch 和 Kibana, 日志输入采用 Filebeat, 没有使用 Logstash

参考: [https://github.com/shildondu/elkstack_filebeat](https://github.com/shildondu/elkstack_filebeat)

准备阶段
----

1.  一台比较干净的可联外网的 ubuntu 16.04 系统的服务器, 并具备 root 权限
2.  安装 `JDK 1.8` 和 `supervisor` 和 `nginx`

```
# 安装JDK和supervisor和nginx
sudo apt-get update && sudo apt-get install openjdk-8-jdk supervisor nginx
# 启动nginx
nginx -t && nginx
# 测试nginx
curl localhost
```

创建用户与下载安装包
----------

```
# 创建用户elk
sudo useradd -d /home/elk/ -m elk
su elk
cd /home/elk/
# 下载安装包
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.3.tar.gz
wget https://artifacts.elastic.co/downloads/kibana/kibana-6.2.3-linux-x86_64.tar.gz
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.3-amd64.deb
# 解压安装包
tar -xzvf elasticsearch-6.2.3.tar.gz
tar -xzvf kibana-6.2.3-linux-x86_64.tar.gz
# 安装filebeat
dpkg -i filebeat-6.2.3-amd64.deb
```

安装 elasticsearch
----------------

搜索引擎

```
su elk
cd /home/elk/elasticsearch-6.2.3/
bin/elasticsearch-plugin install ingest-geoip
bin/elasticsearch-plugin install ingest-user-agent

# 配置全局访问, 不限制访问者ip
echo "network.host: 0.0.0.0" >> config/elasticsearch.yml
# 启动elasticsearch
bin/elasticsearch -d -p pid.txt
# 测试elasticsearch启动
curl localhost:9200
```

kibana
------

用于搜索以及展示数据

```
su elk
cd /home/elk/kibana-6.2.3-linux-x86_64/

# 修改配置文件,让kibana支持外网访问
echo "server.host: 0.0.0.0" >> config/kibana.yml

bin/kibana
```

filebeat
--------

监控日志文件, 并输出的 ElasticSearch

详见 [https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-nginx.html](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-nginx.html)

```
# 启用filebeat的nginx模块(提前安装好nginx)
filebeat modules enable nginx
 # nginx的模块配置
# vim /etc/filebeat/modules.d/nginx.yml
 # filebeat基本配置, 配置ElasticSearch和Kibana
# vim /etc/filebeat/filebeat.yml
output.elasticsearch:
   hosts: ["xxx.xxx.xxx.xxx:9200"]
 # 设置filebeat
filebeat setup -e
# 启动filebeat
service filebeat start
```

配置 supervisor
-------------

让服务可以后台运行

```
[program:elasticsearch]
directory=/home/elk/elasticsearch-6.2.3/
command=/home/elk/elasticsearch-6.2.3/bin/elasticsearch
autorestart=true
user=elk
stdout_logfile=NONE
 [program:kibana]
directory=/home/elk/kibana-6.2.3-linux-x86_64/
command=/home/elk/kibana-6.2.3-linux-x86_64/bin/kibana
autorestart=true
user=elk
stdout_logfile=NONE
```

结果展示
----

![](https://img0.tuicool.com/nURrInr.png!web)

拓展其他的日志
-------

### 配置 Java 工程

1.  添加依赖 `compile('net.logstash.logback:logstash-logback-encoder:4.11')`
2.  修改 `logback-spring.xml` 配置

```
<?xml version="1.0" encoding="UTF-8"?>
<configuration>

<propertyname="logDir"value="./logs"/>

<appendername="JSON_ERROR_LOG"class="ch.qos.logback.core.rolling.RollingFileAppender">
    <File>${logDir}/json_error.log</File>
    <rollingPolicyclass="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
        <FileNamePattern>${logDir}/json_error-%d{yyyy-MM-dd}-%i.log.gz</FileNamePattern>
        <MaxHistory>2</MaxHistory>
        <TimeBasedFileNamingAndTriggeringPolicyclass="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
            <MaxFileSize>100MB</MaxFileSize>
        </TimeBasedFileNamingAndTriggeringPolicy>
    </rollingPolicy>
    <encoderclass="net.logstash.logback.encoder.LogstashEncoder"/>
    <filterclass="ch.qos.logback.classic.filter.ThresholdFilter">
        <level>WARN</level>
    </filter>
</appender>

<rootlevel="INFO">
     <appender-refref="JSON_ERROR_LOG"/>
</root>

</configuration>
```

### 根据条件输出到不同的 index

`/etc/filebeat/filebeat.yml`

```
filebeat.prospectors:

- type: log
 paths:
 - /my_app/server/logs/json_error.log
  json.keys_under_root: true
  json.add_error_key: true
  fields.log_type: "server-error"

- type: log
 paths:
 - /my_app/shangyuer/server/logs/json.log
  json.keys_under_root: true
  json.add_error_key: true
  fields.log_type: "server-log"

filebeat.config.modules:
 path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 3

output.elasticsearch:
 hosts: ["xxx.xxx.xxx.xxx:9200"]
 indices:
 - index: "nginx-access-%{+yyyy.MM.dd}"
      when.equals:
        fileset.module: "nginx"
        fileset.name: "access"
 - index: "nginx-error-%{+yyyy.MM.dd}"
      when.equals:
        fileset.module: "nginx"
        fileset.name: "error"
 - index: "server-error-%{+yyyy.MM.dd}"
      when.equals:
        fields.log_type: "server-error"
 - index: "server-log-%{+yyyy.MM.dd}"
      when.equals:
        fields.log_type: "server-log"

setup.kibana:
 host: "xxx.xxx.xxx.xxx:5601"

#setup.template.name: "nginx-access"
#setup.template.pattern: "nginx-access-*"
#setup.dashboards.index: "nginx-access-*"
#setup.template.overwrite: true
setup.dashboards.enabled: false
setup.template.enabled: false
```

赏

谢谢你请我吃糖果

![](https://img0.tuicool.com/nUB77fZ.jpg!web)

支付宝

![](https://img0.tuicool.com/MNRJby6.jpg!web)

微信