Creating a SIEM with Elasticsearch, Kibana, and Auditbeat on Ubuntu

The picture is from: https://canonical.com/blog/a-new-look-for-the-circle-of-friends

Hi everyone,

Welcome back to my blog. Yes, I haven’t written a post in a while. To be honest, job hunting for an entry-level cybersecurity role has been hard. I took a break from writing this blog. I am back now and sharing a new project that I work on with Simon and Guerline: How to create a SIEM tool with ElasticSearch, Kibana, and Auditbeat on Ubuntu.

Elasticsearch stores and search your data, Kibana allows users to visualize data with charts and graphs, and Auditbeat audits the activities of users and processes on your system.

In order to create a SIEM tool, you have to download Ubuntu. I set up Ubuntu on my virtual machine: VMware. The link to download Ubuntu 22.04.1 LTS and the youtube tutorial link.

After Ubuntu is installed, go to edit virtual machine settings. Then, select network adapter to a bridged network. A bridged network connects the virtual machine to a physical network adapter in the host system. It allows the virtual machine to connect to the local area network(LAN) that the host machine uses. Also, the bridged network creates a unique identity on the network for the virtual machine. This unique identity separates the virtual machine from the host system. This allows the virtual machine to access other machines on the network and contact other machines as it is a physical computer on the network. To learn more about a bridged network, you can read this link.

After setting up Ubuntu, you can now start installing Elasticsearch and Kibana by following this tutorial link and this youtube tutorial link. During this process, it wasn’t easy as the tutorials showed it. I had to do a lot of googling in order to fix the errors to connect to the elasticsearch server. Hopefully, my shared tips can ease the process.

Troubleshooting #1: The command: curl -X GET “localhost:9200”. If you get the output, curl: (52) Empty reply from server.

1. Stop the elasticsearch process using the command: sudo systemctl stop elasticsearch.
2. Go back and edit the elasticsearch file using the command: sudo nano /etc/elasticsearch/elasticsearch.yml.
3. Scroll until you find Begin Security Auto Configuration. Under the enable security features, change xpack.security.enabled from true to false.
4. Start the elasticsearch process using the command: sudo systemctl start elasticsearch.
5. Run the command: curl -X GET “localhost:9200” again and you should get a response. The response indicates that you can connect to the Elasticsearch server.

Right now, Kibana and Elasticsearch are running and you can access these servers via your browser. You can now download Auditbeat. Auditbeat is going to act as our SIEM in this project. Auditbeat allows you to monitor user activity and processes and analyze your event data in the elastic stack.

Before downloading Auditbeat, you need to reconfigure the /etc/kibana/kibana.yml, /etc/elasticsearch/elasticsearch.yml files. Why? Because this current configuration is running on a loopback. In order to send logs from an outside source ( meaning outside of the machine), the configuration needs to be changed.

You can follow the youtube tutorial or use the directions below to change the elasticsearch and Kibana configuration files:

Elasticsearch configuration:

1. Use the command: sudo service elasticsearch stop.
2. Use the command: sudo nano /etc/elasticsearch/elasticsearch.yml. Under the network section, uncomment the network.host and change the IP address to your VM’s IP address. You can use the command: ip addr to find it. (Hint: Look for inet: here is your VM’s IP address).
3. Use cmd prompt and use the command: [VM’s IP address:9200]. This command verifies if you can receive logs from an outside source.

Troubleshoot #2: The command: curl [VM’s IP address:9200]. If you get the output, curl: (7) Failed to connect to server.

1. Stop the elasticsearch process using the command: sudo systemctl stop elasticsearch.
2. Go back and edit the elasticsearch file using the command: sudo nano /etc/elasticsearch/elasticsearch.yml.
3. Scroll until you find Begin Security Auto Configuration. Under the allow HTTP API connection, change the http. host to 0.0.0.0.
4. Start the elasticsearch process using the command: sudo systemctl start elasticsearch.
5. Run the command: curl [VM’s IP address]:9200 again and you should get a response. The response indicates that you can receive logs from an outside source.

Kibana configuration:

1. Use the command: sudo service kibana stop.
2. Use the command: sudo nano /etc/kibana/kibana.yml. Under the System section, uncomment the elasticsearch.host and change the IP address to your VM’s IP address.
3. Use the command: sudo service kibana start

Now, you can download Auditbeat on Ubuntu using the command: sudo apt-get install auditbeat. Then, use the command auditbeat -e setup. This command connects to Kibana and uploads the templates to the Kibana dashboard. Type into the browser: http://localhost:5601 and you should see auditbeat logs on the dashboard. An example is shown below.

The picture is from: https://logz.io/blog/linux-auditbeat-elk/

Congratulation, you have successfully created a SIEM using Elasticsearch, Kibana, and Auditbeat. Now, you can look around and see what data your SIEM is collecting.

Thank you for reading my blog post about creating a SIEM using Elasticsearch, Kibana, and Auditbeat on Ubuntu.

Special shout to my two partners on this project: Simon and Guerline. Without them, I would not be able to do this project without them.

Sources used for this project:

1. Install Ubuntu on VMware Workstation: https://www.youtube.com/watch?v=9rUhGWijf9U
2. Understanding a bridged network: https://docs.vmware.com/en/VMware-Workstation-Pro/16.0/com.vmware.ws.using.doc/GUID-BAFA66C3-81F0-4FCA-84C4-D9F7D258A60A.html
3. Install Elasticsearch and Kibana: https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elastic-stack-on-ubuntu-22-04
4. How to create a SIEM using Kibana, Elasticsearch, and Auditbeat: https://www.youtube.com/watch?v=GvzosoaOaIQ&t=1292s