The Dremio Blog

Healthy Iceberg monitoring combines event reports with table-state queries. ScanReport shows planning and filtering behavior. CommitReport records write outcomes and snapshot changes. Metadata tables reveal file sizes, delete-file pressure, snapshot growth, and manifest shape. Together they support SLOs for freshness, commit reliability, planning latency, and maintenance debt.

A Polaris realm is a top-level administrative boundary that lets one Polaris service host separate sets of principals, roles, catalogs, and configuration. Realms reduce accidental cross-tenant visibility at the catalog layer, but they do not replace network isolation, separate encryption keys, storage policies, database controls, or resource quotas. Production tenancy needs all of those boundaries to agree.

Apache Polaris can validate OAuth 2.0 access tokens from an external OpenID Connect provider instead of issuing every identity itself. The reliable setup aligns issuer, audience, JWKS, claim mapping, token lifetime, and Polaris role grants. Keycloak and Okta differ in administration, but Polaris sees the same signed token contract.

Polaris generic tables register non-Iceberg assets with a name, location, format, and properties so they can participate in catalog discovery and access workflows. They do not turn Delta Lake or another format into Iceberg, coordinate that format's commits, update its transaction log, or automatically vend credentials. The current feature is beta and should be framed as catalog registration, not full table-format governance.

Apache Polaris policies attach maintenance intent to catalog resources. Built-in policy types cover data compaction, metadata compaction, orphan-file removal, and snapshot expiry. A policy records configuration and inheritance, while a separate execution service must discover, schedule, run, and report the maintenance work. Treat policy storage and task execution as two distinct production components.

Trino connects to Polaris through its Iceberg connector with the REST catalog type. The core settings are the Polaris catalog URI, warehouse name, OAuth credential, token scope, and vended-credential support. Once Polaris grants and storage permissions align, Trino can discover, read, and write the same Iceberg tables used by other REST-compatible engines.

DuckDB and iceberg-rust can both use an Iceberg REST catalog such as Polaris without a JVM. DuckDB offers SQL-first local analytics through its Iceberg extension. iceberg-rust provides native Rust APIs for applications and services. They share the REST contract and object storage, but their feature coverage and operational roles differ enough that compatibility must be tested separately.

An Iceberg Kafka Connect sink writes Kafka records into Iceberg data files and commits them through a catalog. With Polaris, the connector uses the Iceberg REST interface for table metadata and OAuth-based authorization. A production setup must align topic routing, schema evolution, commit intervals, dead-letter handling, storage credentials, and maintenance because connector health alone does not prove that table snapshots are advancing.

Moving from AWS Glue or Amazon S3 Tables to Polaris is a catalog migration, a permissions migration, and sometimes a storage migration. Existing Iceberg data can often remain in place when Polaris can register the table metadata and access its location. S3 Tables require a separate decision because table buckets are managed resources with their own endpoints, permissions, and service behavior. Do not treat the two source systems as one cutover path.

Iceberg v3 support must be evaluated by feature, operation, and engine version. A product may open a v3 table yet reject deletion vectors, geometry, variant values, encryption, or writes. This matrix records documented support as of September 22, 2026 and separates table-level read, table-level write, and individual v3 capabilities so teams can choose the lowest common denominator deliberately.

Vendor lock-in is the cost and risk of changing a provider, not the presence of a contract or a proprietary feature. The Exit Cost Test measures that cost across data format, catalog metadata, identity and policy, SQL and APIs, operations, egress, and organizational retraining. A platform is easier to exit when another engine can read the same bytes, discover the same tables, reproduce controls, and take over writes in a tested amount of time.

A production Polaris deployment needs a durable relational persistence layer, repeatable schema initialization and migration, externalized secrets, TLS, health probes, metrics, backups, and a tested upgrade path. The official Helm chart supplies Kubernetes resources, but production readiness comes from the values, database operations, identity configuration, and failure testing around the chart.

Polaris event listeners turn catalog and security activity into records that an external logging system can retain and analyze. A production design captures who acted, what resource changed, the result, request identifiers, and timing, while excluding credentials and limiting sensitive metadata. Audit value comes from durable export, normalized identity, retention, and tested alerts, not from enabling a listener alone.

Apache Iceberg 1.12 and Iceberg Rust 0.11 are close to the finish line, and neither one is a routine patch. Apache Iceberg 1.12.0 for Java carries 210 merged pull requests on its milestone. Iceberg Rust 0.11.0 packs roughly two months of work, including end-to-end table encryption and Variant support. Both went through release candidate trouble in mid-September 2026, which tells you something about how much changed.

A data lakehouse is an architecture: object storage, an open table format such as Apache Iceberg, a catalog, and multiple engines reading the same tables. Medallion architecture is a layering convention: bronze for raw data, silver for cleaned data, gold for business-facing aggregates. They are not alternatives. You run medallion inside a lakehouse, and the question worth arguing about is not which to pick but how many of those layers need to be physical copies.