vovatoolbox.blogg.se - Elasticsearch metabase

#Elasticsearch metabase series

But since these do not scale with increasing data and workload, Elasticsearch is quickly becoming a popular alternative.

#Elasticsearch metabase series

While storing time series based metrics, InfluxDB and Prometheus are some of the popular choices.

Used for large scale stream processing in companies like GoJek, Netflix, Alibaba, Lyft, etc.įlink Dashboard Elasticsearch as Serving Data Store.

In-built support for different telemetry tools for monitoring & alerting – JMX, InfluxDb, Prometheus, etc.

In-built integration support for popular sources (Kafka, AWS Kinesis, RabbitMQ) and sinks (Elasticsearch, Cassandra, Kafka), etc.

Provides multiple deployment options – from single JVM to multi-node cluster, from native deployment to resource manager based deployments (Kubernetes, Mesos, YARN).

It supports storing state data on multiple frameworks – Hadoop HDFS, AWS S3, Google GFS, etc.

Built-in state management for intermediate stream processing data.

Flink’s kernel (core) is a streaming runtime which also provides distributed processing, fault tolerance, etc.

Flink is the true stream processing framework (doesn’t cut stream into micro-batches) and processes events at a consistently high speed with low latency.

It replicates the data based on use cases so that data can be availed easily in case of system failure.Īpache Flink is chosen over other stream processing frameworks like Apache Storm, Apache Spark Streams and Kafka Streams for following reasons:

Kafka provides a high availability and fault tolerance system.

Kafka can handle high-velocity real-time data.

Kafka is a distributed publish-subscribe messaging system.

We decided to choose Kafka for the event store in Velocity due to following reasons:

Kibana/Grafana – Open source visualisation frameworks that connect to the Elasticsearch data store and act as Serving Layers.

Elasticsearch – Open source data store primarily optimised for search, but of late has become very popular as a Serving Layer Store for operational and business metrics.

Apache Flink – Open-source platform that provides data distribution, communication, state management and fault tolerance for distributed computations over data streams.

Apache Kafka – Kafka has become a common denominator in most open-source stream processors as the de-facto Storage Layer for storing and moving potentially large volumes of data in a streaming fashion with low latency.

Velocity comprises of following components: Kibana displays the data through dashboards. Custom jobs deployed on the Flink cluster read the events from Kafka, perform the aggregations and store the result in Elasticsearch indices. AWS Lambda functions are used to enrich this event data (by calling the REST APIs of microservices) and enriched events are pushed into Kafka topics. Architectureĭata from individual microservices are currently published to SNS (a lightweight eventing system between microservices). This drove the need for a separate real-time data pipeline and Velocity was born. Non-availability of data on RedShift (in real-time) was not a concern for reporting but was a major change for the real-time dashboards. Hence, instead of DMS jobs (that copy the data in realtime), the data team planned to use the ETL jobs to copy the data to Redshift in batches (with separation of multiple hours).

When the data team analysed the root cause, we found that RedShift, like most Data Warehouse systems, is optimised for batch updates, especially when the read traffic is less. This started impacting the performance of the RedShift system, along with a substantial increase in query response time. But with an increase in business, we saw our backend data sources increase rapidly. The data in Redshift is used by the operations team for various purposes like BI Reporting (through PowerBI/Looker), real-time monitoring of business metrics (Metabase dashboard) and ad-hoc queries (Metabase Questions). The DMS jobs copied any edited data over to Redshift in real-time. Given how querying data from multiple sources might slow down our operations, we set up our data platform to copy the data from individual databases to AWS Redshift via DMS jobs. But from Halodoc's operations point-of-view, a single transaction by a user might span across multiple business domains. Business domains like Pharmacy Delivery, Tele Consultations, Appointments, and Insurance have their own microservices. These microservices have access to their data but rely on the APIs to retrieve data from other microservices. Halodoc backend consists of multiple microservices that interact with each other via REST APIs.