100 Best Apps for Data Engineers
Data engineering stacks vary by team, architecture, and budget. This opening selection covers widely used platforms and developer tools across warehousing, pipelines, orchestration, processing, and quality.
Snowflake
Snowflake provides a cloud data platform for storing, querying, sharing, and analyzing structured and semi-structured data. It reduces infrastructure administration for teams needing scalable analytics warehouses without managing servers.
Databricks
Databricks combines data engineering, analytics, and machine learning workloads on a managed lakehouse platform. It helps teams process large datasets collaboratively without separately maintaining fragmented compute environments.
Google BigQuery
Google BigQuery is a serverless cloud warehouse for SQL analytics across large datasets. It lets engineers run large analytical queries without provisioning or tuning database infrastructure.
Amazon Redshift
Amazon Redshift is a managed cloud data warehouse designed for analytical queries and reporting workloads. It helps AWS-based teams centralize analytical data without operating warehouse hardware or software.
Azure Synapse Analytics
Azure Synapse Analytics combines data warehousing, big-data processing, and data integration within Microsoft Azure. It helps Azure teams analyze warehouse and lake data through connected services and interfaces.
PostgreSQL
PostgreSQL is an open-source relational database for reliably storing, querying, and managing transactional data. It gives teams a mature SQL database when applications require dependable structured data storage.
Apache Kafka
Apache Kafka is a distributed event-streaming platform for publishing, storing, and processing data streams. It helps engineers move high-volume events between systems without tightly coupling producers and consumers.
Apache Airflow
Apache Airflow orchestrates data workflows as scheduled, monitored, and dependency-aware pipelines defined in Python. It helps teams coordinate multi-step jobs while providing visibility into failures, retries, and dependencies.
dbt
dbt transforms warehouse data using version-controlled SQL models, tests, documentation, and dependency management. It helps analysts and engineers make SQL transformations repeatable, reviewable, and easier to maintain.
Fivetran
Fivetran provides managed connectors that replicate data from business applications into analytical destinations. It reduces custom integration maintenance when teams need regularly updated data from SaaS systems.
Airbyte
Airbyte is a data integration platform with connectors for moving data between sources and destinations. It helps teams build and customize data replication pipelines without writing every connector from scratch.
Dagster
Dagster is a data orchestrator that models pipelines as software-defined assets with observability features. It helps engineers understand data dependencies and diagnose pipeline issues through asset-oriented workflow design.
Prefect
Prefect is a workflow orchestration platform for building, scheduling, monitoring, and running data pipelines. It helps teams manage resilient workflows with retries and monitoring instead of brittle scheduled scripts.
Apache Spark
Apache Spark is a distributed processing engine for large-scale batch analytics, streaming, and machine learning. It helps engineers process datasets too large or slow for single-machine computation.
Apache Flink
Apache Flink is a distributed engine for stateful stream processing and batch data applications. It helps teams process continuously arriving events with stateful computations and low-latency results.
Trino
Trino is a distributed SQL query engine for querying data across multiple sources and formats. It helps analysts query data lakes and databases without first copying everything into one warehouse.
DuckDB
DuckDB is an in-process analytical database designed for fast SQL queries on local data files. It helps engineers explore and transform files locally without deploying a separate database server.
ClickHouse
ClickHouse is a column-oriented database management system built for fast analytical queries on large datasets. It helps teams serve high-volume analytics workloads where conventional row-oriented databases become slow.
MongoDB
MongoDB is a document database for storing and querying flexible, JSON-like application data structures. It helps teams handle evolving application schemas without requiring rigid relational table designs.
Elasticsearch
Elasticsearch is a distributed search and analytics engine for indexing, searching, and aggregating data. It helps teams make logs, documents, and operational data quickly searchable at scale.
Terraform
Terraform defines and provisions cloud infrastructure through declarative configuration files and provider integrations. It helps teams reproduce data infrastructure consistently instead of manually configuring cloud resources.
GitHub
GitHub hosts Git repositories and supports collaborative code review, issue tracking, and automation workflows. It helps data teams review pipeline changes and preserve a shared history of engineering decisions.
Docker
Docker packages applications and dependencies into portable containers that run consistently across computing environments. It reduces environment inconsistencies when pipeline code moves between local development, testing, and production.
Kubernetes
Kubernetes orchestrates containerized applications by scheduling, scaling, networking, and managing their deployment. It helps platform teams operate containerized data services reliably across clusters of machines.
Great Expectations
Great Expectations validates data with documented expectations, tests, and reports for pipeline outputs. It helps teams detect unexpected data changes before flawed datasets reach downstream users and reports.
Confluent
Confluent provides managed Kafka services and tooling for building, operating, and governing streaming data pipelines. It reduces the operational burden of running Kafka clusters while teams publish and consume real-time data.
Apache NiFi
Apache NiFi designs and manages data flows through a visual interface with configurable processors and routing. It helps engineers replace brittle custom ingestion scripts with traceable, reusable flow definitions.
Kafka Connect
Kafka Connect moves data between Kafka and external systems through source and sink connector plugins. It eliminates repeated integration code when teams need to ingest or deliver event data.
Debezium
Debezium captures database changes and streams row-level inserts, updates, and deletions into event platforms. It helps teams keep downstream systems current without repeatedly querying production databases for changes.
Apache Iceberg
Apache Iceberg is a table format for large analytic datasets stored in data lakes. It addresses unreliable lake queries by adding snapshots, schema evolution, and partition management.
Delta Lake
Delta Lake adds transactional tables, schema controls, and versioning to cloud data lake storage. It helps prevent inconsistent lake data when concurrent pipelines read and write shared datasets.
Apache Hudi
Apache Hudi manages large data-lake tables with record-level updates, incremental queries, and transactional consistency. It helps engineers efficiently process changing records instead of rebuilding entire datasets after updates.
Apache Hive
Apache Hive provides SQL-based querying and data warehousing capabilities for distributed storage systems. It gives SQL-oriented teams a familiar way to analyze large files without custom MapReduce code.
Apache HBase
Apache HBase is a distributed, column-oriented database for low-latency access to large sparse tables. It supports workloads needing fast individual record lookups that batch-oriented warehouses cannot efficiently serve.
Apache Cassandra
Apache Cassandra is a distributed NoSQL database designed for high availability across multiple nodes. It helps applications maintain write-heavy operational data across regions without a single central database.
Redis
Redis is an in-memory data store used for caching, queues, streams, and fast key-value access. It reduces application latency by serving frequently requested data without repeatedly querying primary databases.
Neo4j
Neo4j is a graph database that stores connected entities and supports relationship-focused queries. It simplifies analysis of deeply connected data that requires cumbersome joins in relational databases.
Amazon S3
Amazon S3 provides object storage for datasets, backups, logs, and files used by data pipelines. It gives teams durable centralized storage for growing data volumes without managing storage hardware.
Google Cloud Storage
Google Cloud Storage provides object storage for datasets, archives, media, and pipeline staging files. It helps teams store and exchange large pipeline artifacts without maintaining file servers.
Azure Data Lake Storage
Azure Data Lake Storage provides scalable cloud storage optimized for analytics workloads and hierarchical file organization. It helps Azure teams consolidate raw analytical data while retaining folder-like access controls.
AWS Glue
AWS Glue provides managed ETL jobs, a data catalog, and serverless data preparation tools. It reduces infrastructure management for teams building transformations across data stored in AWS services.
Google Cloud Dataflow
Google Cloud Dataflow runs managed Apache Beam pipelines for batch and streaming data processing. It helps engineers deploy scalable Beam workloads without provisioning and operating processing clusters.
Azure Data Factory
Azure Data Factory orchestrates cloud data movement and transformation through pipelines, activities, and connectors. It centralizes scheduled integrations for teams otherwise coordinating transfers across disconnected Azure services.
Matillion
Matillion provides visual data integration and transformation tools for cloud data platforms and warehouses. It helps teams build repeatable warehouse pipelines without writing every orchestration step from scratch.
Talend Data Integration
Talend Data Integration provides graphical tools for connecting, transforming, and delivering data between systems. It reduces manual integration development when organizations must connect many legacy and cloud sources.
Informatica Cloud Data Integration
Informatica Cloud Data Integration builds and runs cloud-based pipelines for data ingestion, transformation, and synchronization. It helps teams standardize data movement across diverse enterprise applications and cloud platforms.
Dataiku
Dataiku provides a collaborative platform for preparing data, building models, and operationalizing analytics workflows. It helps technical and business teams collaborate on governed data projects within one workspace.
Monte Carlo
Monte Carlo monitors data pipelines and tables to detect freshness, volume, schema, and quality anomalies. It helps engineers find unreliable data before downstream dashboards and decisions are affected.
Soda
Soda provides data quality testing and monitoring tools for checking datasets within engineering workflows. It helps teams catch broken data assumptions early through automated checks and observable results.
OpenMetadata
OpenMetadata is an open-source platform for cataloging data assets, lineage, ownership, and governance metadata. It helps teams discover trusted datasets and understand dependencies across fragmented data infrastructure.
Collibra
Collibra provides data governance workflows, cataloging, lineage, and policy management for enterprise data assets. It helps teams resolve unclear ownership and compliance responsibilities by documenting stewards, policies, and definitions.
Alation
Alation offers a data catalog for organizing metadata, search, governance, and collaborative documentation. It reduces time spent locating trustworthy datasets by making context and usage information easier to find.
Atlan
Atlan provides an active metadata platform for cataloging data assets, lineage, governance, and collaboration. It helps engineers understand downstream dependencies before changes by connecting metadata across data systems.
DataHub
DataHub is an open-source metadata platform for data discovery, governance, lineage, and observability. It addresses fragmented dataset knowledge by centralizing technical metadata and ownership information across platforms.
Amundsen
Amundsen is an open-source data discovery and metadata engine originally created at Lyft. It helps analysts find relevant tables faster by providing searchable dataset descriptions, owners, and usage signals.
Secoda
Secoda centralizes data documentation, cataloging, lineage, and search across an organization's data stack. It reduces repeated questions about metrics and datasets by making institutional knowledge easier to discover.
Apache Superset
Apache Superset is an open-source business intelligence platform for exploring data and building dashboards. It helps teams share SQL-based analysis visually without requiring every stakeholder to write queries.
Metabase
Metabase is a business intelligence tool for querying databases and creating interactive dashboards. It makes operational data more accessible by letting nontechnical teammates explore approved datasets through a visual interface.
Looker
Looker is a business intelligence platform that uses semantic modeling to define reusable metrics. It reduces inconsistent reporting by centralizing metric definitions and calculations across dashboards and analyses.
Tableau
Tableau provides visual analytics tools for connecting data sources, exploring information, and publishing dashboards. It helps teams communicate complex patterns by turning query results into interactive visualizations and reports.
Power BI
Power BI is Microsoft's analytics platform for modeling data, building reports, and sharing dashboards. It helps organizations distribute standardized reporting by combining data modeling with familiar Microsoft ecosystem integrations.
Grafana
Grafana visualizes metrics, logs, and traces through configurable dashboards and alerting integrations. It helps engineers monitor pipeline health by bringing operational signals from multiple sources into one view.
Prometheus
Prometheus is an open-source monitoring system that collects time-series metrics and evaluates alert rules. It identifies infrastructure and service issues earlier by storing measurable system behavior for querying and alerting.
Datadog
Datadog is a cloud monitoring platform for infrastructure, applications, logs, traces, and security signals. It reduces troubleshooting context switching by correlating operational telemetry across data services and cloud infrastructure.
New Relic
New Relic provides observability tools for monitoring applications, infrastructure, logs, and distributed transaction performance. It helps engineers isolate slow or failing services by connecting performance data with application traces.
Sentry
Sentry tracks application errors and performance issues, providing stack traces and release context. It speeds debugging of data applications by grouping errors and showing where failures originated.
OpenTelemetry
OpenTelemetry provides open standards and tools for generating, collecting, and exporting telemetry data. It reduces vendor-specific instrumentation work by standardizing how applications emit traces, metrics, and logs.
Splunk
Splunk indexes and analyzes machine-generated data, including logs, events, metrics, and security records. It helps teams investigate incidents by searching large volumes of operational data from distributed systems.
DBeaver
DBeaver is a database management tool for connecting to, querying, and administering many database systems. It reduces the need for separate database clients by providing a unified interface for diverse connections.
DataGrip
DataGrip is a cross-platform SQL IDE for writing queries and managing relational databases. It helps engineers work more efficiently with database schemas through code completion, navigation, and query tools.
JupyterLab
JupyterLab provides a web-based workspace for notebooks, code, terminals, files, and interactive computing. It supports exploratory data work by keeping executable code, outputs, notes, and visualizations together.
Apache Beam
Apache Beam provides a unified programming model for defining batch and streaming data pipelines. It reduces pipeline portability challenges by allowing the same transformations to run on supported execution engines.
Meltano
Meltano is an open-source data integration platform for running and managing ELT pipelines. It helps teams operationalize Singer-based integrations by organizing extraction, loading, and transformation workflows together.
SQLMesh
SQLMesh is a data transformation framework that manages SQL models, environments, and deployment plans. It helps prevent risky transformation changes by planning model impacts before production deployment.
Liquibase
Liquibase manages database schema changes through version-controlled changelogs and automated deployment workflows. It prevents manual schema drift by applying documented database changes consistently across environments.
Apache Pinot
Apache Pinot is a real-time distributed OLAP datastore for low-latency analytics on streaming and batch data. It helps teams query fresh event data quickly when warehouse loading creates unacceptable dashboard delays.
Apache Druid
Apache Druid is a distributed analytics database designed for fast aggregations across event-oriented time-series data. It helps engineers serve interactive operational analytics without repeatedly scanning large raw event datasets.
Starburst
Starburst provides managed and enterprise tooling for querying distributed data sources using the Trino SQL engine. It helps teams analyze data across systems without moving every dataset into one central database.
Materialize
Materialize maintains SQL views incrementally as streaming data changes, enabling continuously updated analytical results. It helps engineers avoid expensive full recomputations when dashboards require constantly refreshed derived data.
Redpanda
Redpanda is a Kafka-compatible streaming data platform for publishing, retaining, and consuming event records. It helps teams adopt event streaming while reducing operational friction around compatible producer and consumer applications.
Apache Pulsar
Apache Pulsar is a distributed messaging and streaming system supporting pub-sub, queues, and persistent topics. It helps engineers handle diverse messaging patterns without operating separate systems for queues and streams.
RabbitMQ
RabbitMQ is a message broker that routes messages between applications through exchanges, queues, and consumer acknowledgments. It helps decouple data-producing services from downstream processing workers that run at different speeds.
CockroachDB
CockroachDB is a distributed SQL database built for transactional workloads across multiple nodes and locations. It helps teams maintain relational data availability when applications need resilience across infrastructure failures.
TimescaleDB
TimescaleDB extends PostgreSQL with time-series storage, querying, compression, and data lifecycle capabilities. It helps engineers manage timestamped metrics and events without abandoning familiar PostgreSQL SQL workflows.
InfluxDB
InfluxDB is a time-series database for storing, querying, and monitoring measurements indexed by time. It helps teams efficiently retain and inspect high-volume infrastructure metrics that grow rapidly over time.
Amazon DynamoDB
Amazon DynamoDB is a managed NoSQL database for key-value and document workloads on AWS. It helps engineers run predictable low-latency application data workloads without managing database servers themselves.
Google Cloud Spanner
Google Cloud Spanner is a managed distributed relational database offering SQL transactions across scalable infrastructure. It helps teams scale transactional relational systems when a single database instance becomes operationally limiting.
Azure Cosmos DB
Azure Cosmos DB is a managed distributed database supporting multiple data models and application programming interfaces. It helps teams serve globally distributed application data without building replication and partitioning systems manually.
MySQL
MySQL is an open-source relational database management system for structured application and operational data. It helps engineers store and query structured records using widely understood SQL and administration practices.
MariaDB
MariaDB is an open-source relational database platform compatible with many MySQL tools and applications. It helps teams operate relational workloads when they need an open-source database with familiar SQL tooling.
Microsoft SQL Server
Microsoft SQL Server is a relational database platform for transactional processing, analytics, administration, and reporting. It helps organizations manage structured business data within Microsoft-oriented infrastructure and development environments.
Oracle Database
Oracle Database is a relational database management system for enterprise transactional, analytical, and operational workloads. It helps teams support complex business data requirements with mature SQL, security, and administration capabilities.
IBM Db2
IBM Db2 is a relational database platform used for transactional processing and analytical data management. It helps enterprises work with established IBM data environments while maintaining relational query and governance controls.
MinIO
MinIO is an object storage server that provides an API compatible with Amazon S3. It helps teams build data lakes on self-managed infrastructure using widely supported object storage interfaces.
Flyway
Flyway manages database schema changes through versioned migration scripts applied consistently across environments. It helps engineers prevent schema drift when multiple environments require repeatable database deployment processes.
pgAdmin
pgAdmin is a graphical administration and query tool for PostgreSQL databases and servers. It helps users inspect PostgreSQL objects and run queries without relying exclusively on command-line tools.
Visual Studio Code
Visual Studio Code is a code editor supporting extensions for SQL, Python, configuration, and version control. It helps data engineers work across pipelines and infrastructure files within one customizable development environment.
Jenkins
Jenkins is an automation server that runs configurable build, test, and deployment pipelines. It helps teams automate repetitive data pipeline checks and releases instead of executing them manually.
GitLab
GitLab is a DevOps platform for source code management, code review, CI/CD, and collaboration. It helps distributed teams connect data project changes with reviews, automated tests, and deployment workflows.
Apache Ranger
Apache Ranger centralizes authorization policies and auditing for data services in Hadoop ecosystem deployments. It helps administrators apply consistent access controls across multiple data platforms instead of configuring each separately.
The remaining entries expand this editorial list with additional tools for ingestion, governance, observability, development, and operational data work.