What is ELK & Why ELK?

With millions of downloads for its various components since first being introduced, the ELK Stack is the world’s most popular log management platform. In contrast, Splunk — the historical leader in the space — self-reports 15,000 customers in total.

What exactly is ELK? Why is this software stack seeing such widespread interest and adoption? How do the different components in the stack interact?

In this guide, we will take a comprehensive look at the different components comprising the stack. We will help you understand what role they play in your data pipelines, how to install and configure them, and how best to avoid some common pitfalls along the way.

Additionally, we’ll point out the advantages of using OpenSearch and OpenSearch Dashboards – the open source forked versions of Elasticsearch and Kibana, respectively, launched by AWS together with Logz.io and other community members shortly after Elastic closed sourced the ELK Stack, in an effort to keep the projects open source. 

And lastly, we will reference Logz.io as a solution to some of the challenges discussed in this article – which offers a SaaS logging and observability platform that’s based on these popular open source stacks, while offloading the maintenance tasks required to run your own ELK Stack or OpenSearch.

The latest on the ELK Stack

The ELK Stack grew into the most popular log management and analytics solution in the world as a collection of open source projects maintained by Elastic – whose founders launched the ELK Stack. Since then, Elastic’s relationship with the open source community has grown more complicated.

In early 2021, Elastic announced a bombshell in the open source world: the ELK Stack would no longer be open source, as of version 7.11. The company implemented dual proprietary licenses to govern ELK-related projects – including SSPL and the Elastic license – which includes ambiguous legal language on appropriate usage for the ELK Stack.

Shortly after, AWS announced the launch of OpenSearch and OpenSearch Dashboards, which would fill the role originally held by Elasticsearch and Kibana, respectively, as the leading open source log management platform. 

There are a few capabilities supported by OpenSearch that are only available in the paid versions of ELK:

• OpenSearch includes access controls for centralized management. This is a premium feature in Elasticsearch.
• The OpenSearch community is building an Observability Plugin, which unifies log, metric, and trace analytics in one place. While Elastic has been adding similar capabilities, many of them are not open source.
• OpenSearch has a full suite of security features, including encryption, authentication, access control, and audit logging and compliance. These are premium features in Elasticsearch.
• ML Commons makes it easy to add machine learning features. ML tools are premium features in Elasticsearch.