Geospatial data lake and query layer

Farmdar's data lives in three places: TB-scale satellite imagery and rasters in S3, a PostGIS database behind every product, and the outputs of analysis notebooks. Before the lakehouse, getting a joined view meant exporting from one and loading into another.

The goal was one SQL entry point that data scientists, analysts and dashboards could use, without copying the production database around or handing out database credentials.

Trino runs on EKS and exposes two kinds of catalog. Apache Iceberg tables live in AWS S3 Tables, with one table bucket per environment and a namespace per product. The PostGIS production database is attached through Trino's PostgreSQL connector, so it is queried live, not replicated.

Sign-in to Trino goes through AWS Cognito. Cognito groups map to catalog and schema permissions, so access is managed in the same place as access to every other Farmdar application. Superset and JupyterHub sit on top.

Pipelines write to the lake through a small Python library in our shared package. It creates partitioned tables through Trino and appends or overwrites data through PyIceberg, with geometries stored as WKB.

• Managed catalog over a self-hosted one

  We first ran Apache Nessie as the Iceberg catalog. Moving to the S3 Tables Iceberg REST endpoint removed a service we had to run ourselves, and S3 Tables handles table maintenance such as compaction.

• Federate the production database, don't copy it

  PostGIS stays the source of truth for fields, subscriptions and results. Trino reads it live through a connector, so there is no sync job to break and no second copy to go stale.

• One library for writing to the lake

  Notebooks and DAGs call a single create, append and overwrite API instead of talking to the catalog directly. Partitioning and geometry encoding are decided once, in code review, not in every notebook.

• Short-lived credentials

  Trino reaches S3 Tables through IAM roles for service accounts, not long-lived access keys.

I designed the lakehouse and led the platform team that built it: S3 Tables and networking in Terraform, Cognito sign-in for Trino, IAM for service accounts, and the shared Python library that pipelines use to write to the lake.

The lakehouse is the newest layer of the platform, and yield estimation is the first production pipeline writing to it.