ELK + Spring Boot: A Guide to Local Configuration
Setting up a centralized logging solution using the ELK stack on your local machine.
Hi Folks,
In this tutorial, I will guide you through the process of setting up a centralized logging solution using the ELK stack on your local machine. While the production-ready configuration may be more complex and different, it is still valuable to familiarize yourself with the components and configurations by setting up the ELK stack locally.
What is ELK?*
ELK is an acronym that stands for Elasticsearch, Logstash, and Kibana. It is a powerful open-source software stack used for centralized logging, log analysis, and visualization. Each component of ELK serves a specific purpose:
- Elasticsearch: It is a distributed search and analytics engine that stores and indexes data. Elasticsearch allows for fast and efficient searching, aggregating, and analyzing of large volumes of data.
- Logstash: It is a data processing pipeline that ingests, filters, and transforms data from various sources before sending it to Elasticsearch for indexing. Logstash can handle different types of data, such as logs, metrics, and other event data.
- Kibana: It is a web-based data visualization and exploration tool that works in conjunction with Elasticsearch. Kibana provides a user-friendly interface for searching, analyzing, and visualizing data stored in Elasticsearch. It offers various visualizations, dashboards, and tools to help gain insights from the data.
Together, these three components form the ELK stack, which is widely used for log management, monitoring, and troubleshooting in various applications and systems.
The following high-level architecture diagram illustrates the placement of components within a system.
1. Build Springboot Application
- Go to Spring initializr and select your preferable Language, Project & Java version (> v1.8 is recommended).
- Add Web dependence to your project.
- Generate the project, in my demo the generated maven project POM file is mentioned below.
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>3.0.6</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId>com.elk.demo</groupId>
<artifactId>com-elk-demo</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>elk-demo</name>
<description>Demo project for Spring Boot</description>
<properties>
<java.version>17</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>4. Since you add the WEB dependency, loggers are pre-configured in your project.
But by default, spring boots are logs only to the console. So you have to add the following configuration in the application.property file to write logs to a file.
logging.file.path=.5. Implement Controller class and create an endpoint to write a log on both the console & log file.
package com.elk.controller;
import org.apache.logging.log4j.Level;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
@RestController
public class Controller {
private static final Logger LOG = LogManager.getLogger(Controller.class);
@GetMapping(value = "/echo")
public String echoMessage() {
LOG.log(Level.INFO, "Echo Triggered");
return "Echo Triggered";
}
}6. spinning up the spring boot application by executing the below command.
mvn clean install spring-boot:run2. Set up Elastic Search
- Download and unzip Elasticsearch.
- Go to
elasticsearch-<version>-<os>-<system-type>\elasticsearch-<version>\bin - Open CMD and navigate to the above-mentioned directory then execute the following command.
elasticsearch.bat4. If you're using Windows OS, you may need to allow access blocked due to Windows Defender Firewall.
5. Scroll down a bit, you can see the Elasticsearch security features have been automatically configured! message on the CMD output.
6. Copy the following values and save them.
- Password for the elastic user
- Enrollment token
To Generate a new enrollment key
1. Go to elasticsearch-<version>-<os>-<system-type>\elasticsearch-<version>\bin
2. Run elasticsearch-create-enrollment-token.bat - url https://localhost:9200 -s kibana
To Reset the elastic user password
1. Go to elasticsearch-<version>-<os>-<system-type>\elasticsearch-<version>\bin
2. Run elasticsearch-reset-password.bat -u elastic7. To check the status of the Elasticsearch cluster connection, hit https://localhost:9200/ on your browser and provide the Username as “elastic” and use the password you saved in step #6.
If your cluster is working properly you should get the below output.
3. Set up Kibana
- Download and unzip Kibana.
- Go to
elasticsearch-<version>-<os>-<system-type>\elasticsearch-<version>\bin. - Open CMD and navigate to step #2 mentioned directory, then execute the following command.
kibana.bat4. Once the Kibana spinning up correctly, the URL should be available on CMD, copy it.
5. Go to the URL copied in step #4.
6. Provide the Enrollment token saved in “Set up Elastic Search” step #6.
7. Click Configure Elastic.
4. Set up Logstash
- Download and unzip Logstash.
- Go to
logstash-<version>-<os>-<system-type>\logstash-<version>\config. - Open the
logstash-sample.conf& replace the input & output plugins as mentioned below.
# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.
input {
file {
path => "<Log File Full Path>"
start_position => "beginning"
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => ["https://localhost:9200"]
ssl_certificate_verification => false
ssl => true
index => "elkdemoindex"
user => "elastic"
password => "<Elastic Search Password>"
}
}4. Replace the <Log File Full Path> with your log file location. (Make sure to use forward slashes).
5. Replace <Elastic Search Password> with the password, you saved in “Set up Elastic Search” step #6.
6. Go to the logstash-<version>-<os>-<system-type>\logstash-<version>\bin directory and spinning up the Logstach using the below command.
logstash.bat -f ./config/logstash-sample.conf5. Verify the ELK + Spring boot Flow
- Check the Logstash to Spring boot Integration
In “Set up Logstash” step #3, we set parameters in the output plugin, stdout { codec => rubydebug }, this will allow us to check the integration between the Springboot application & Logstash.
If both parties are connected as expected, you can see spring boot application logs in JSON format on the Logstash terminal like this.
{
"@version" => "1",
"log" => {
"file" => {
"path" => "D:/Projects/com-elk-demo/com-elk-demo/spring.log"
}
},
"host" => {
"name" => "Denuwan-Laptop"
},
"event" => {
"original" => "2023-05-16T01:34:54.503+05:30 INFO 19908 --- [main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)\r"
},
"@timestamp" => 2023-05-16T05:22:37.881555300Z,
"message" => "2023-05-16T01:34:54.503+05:30 INFO 19908 --- [main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)\r"
}2. Configure Kibana Dashboard
- Go to http://localhost:5601/
- Click the burger button on the left side and go to
Management > Stack Management. - Inside the Management portal, go to
Data > Index Management, if the Logstash to Elastic search connection working properly you should get theelkdemoindexvalue on the indices list.
4. Again click the burger button on the left side and go to Analytics > Discover.
5. You can create a new data view as mentioned below.
6. Once you create a new data view, navigate to the newly created data view and check the logs.
Congratulations! Now you are successfully configured ELK + Spring boot stack on your local.
Happy coding!
* "What is ELK?" the theoretical part of the content is generated by ChatGPT and reviewed by a human.
