Amazon Aurora PostgreSQL now supports direct querying of Apache Iceberg and Parquet data in your data lake
AWS has released Aurora PostgreSQL versions 17.11 and 18.6 with a built‑in aurora_analytics extension that embeds the open‑source DuckDB engine. By attaching an IAM role and enabling the extension, developers can create foreign tables that point at Iceberg or Parquet objects stored in Amazon S3 or cataloged in AWS Glue. The engine performs predicate push‑down, column pruning, and caches frequently accessed lake data, allowing a single SQL statement to join fresh transactional rows in Aurora with years‑old historical rows in a Parquet file without moving data. The feature works with Iceberg REST Catalog‑compatible catalogs, supports bulk import of foreign schemas, and reports per‑query metrics via aurora_analytics_stat_statements().
This move pushes AWS further into the “lakehouse” arena where operational databases and analytical data lakes converge. Competitors such as Snowflake, Databricks, and Google BigQuery already expose external tables or native support for Iceberg; Aurora’s new capability narrows that gap by leveraging DuckDB’s in‑process analytics, which is known for low‑overhead columnar scans. By folding lake querying into a familiar PostgreSQL endpoint, AWS reduces the friction for existing Postgres applications that need historical context or AI‑driven enrichment, sidestepping the reverse‑ETL pipelines that many firms have built to feed data warehouses. The integration also signals Amazon’s strategy of absorbing high‑performance open‑source projects (DuckDB) to augment its managed services.
Practically, customers will see a shift in cost and architecture decisions. While the feature is free to enable, charges accrue for Aurora compute cycles and S3 request fees, so workloads with heavy lake scans could affect budgets. Performance will depend on cache hit rates and the ability of DuckDB to prune data; latency‑sensitive queries may still require materializing hot subsets into native Aurora tables. Security hinges on correctly scoped IAM roles and Glue catalog permissions. Watch for future updates that may add support for additional formats (e.g., ORC), tighter integration with AWS Lake Formation, and performance benchmarks that compare Aurora‑DuckDB scans against dedicated data‑warehouse engines.
Key Takeaways
Aurora PostgreSQL now embeds DuckDB, enabling direct SQL queries over Iceberg and Parquet data in S3 via the aurora_analytics extension.
The feature removes the need for reverse ETL pipelines, letting a single query combine live transactional rows with years‑old lake data.
Predicate push‑down, column pruning, and on‑instance caching aim to keep lake scans efficient, but compute and S3 request costs will drive budgeting decisions.
Future enhancements may broaden format support and tighten security integrations, making Aurora a stronger contender in the lakehouse space.
About the Source
This analysis is based on reporting by AWS Blog. Here is a short excerpt for context:
Amazon Aurora PostgreSQL now lets you directly query Apache Iceberg and Parquet data stored in your data lake alongside live operational data—no ETL pipelines required. Powered by DuckDB embedded within Aurora, this capability enables single queries that join transactional and historical data using familiar PostgreSQL syntax. It supports AWS Glue Data Catalog, S3, and S3 Tables, with optimizations like predicate pushdown and caching for efficient performance.Read the original at AWS Blog