Satellite imagery pipelines

Farmdar sells crop-health maps and reports built from satellite imagery. The analysis was written by data scientists in Jupyter notebooks and run by hand, so every new customer area meant more manual runs and slower delivery.

The workloads are mixed: light API calls, memory-heavy raster processing and GPU model inference. They need to run on the right machines without paying for idle ones.

Airflow runs on EKS with Celery workers split into queues by resource profile: low, medium, high-memory and GPU. Workers scale on queue depth, and spot nodes are added on demand.

Notebook steps run through a custom operator built on KubernetesPodOperator. Each task gets its own pod that pulls the shared library at start-up and executes the notebook with Papermill, with parameters passed in from the DAG.

For Planet, a DAG creates subscriptions for 3 m, 8-band surface-reflectance imagery that Planet delivers directly to S3. The main pipeline then maps over batches of fields: shortlist, download, plant health and nitrogen maps, database load, scouting points and PDF reports.

• Run notebooks as tasks instead of rewriting them

  Data scientists keep working in notebooks, and the platform turns them into scheduled, retryable, isolated tasks. A new analysis goes to production as a notebook plus a few lines of DAG code.

• Let Planet push to S3

  Subscriptions with S3 delivery replaced searching and downloading scenes ourselves. New imagery arrives without a polling job, and the pipeline starts from files that are already in our bucket.

• Split GPU and CPU stages

  For field delineation, I split the work into two DAGs that hand off through S3, so a GPU node is not held for hours of CPU-bound post-processing.

• Right-size before scaling

  I reduced worker memory requests to what tasks actually used and moved GPU node pools to spot instances, as part of a cost review that cut monthly AWS spend by 62%.

One plant-health run made millions of S3 list requests in a single day. GDAL was listing the whole directory each time it opened a raster. One GDAL setting (GDAL_DISABLE_READDIR_ON_OPEN) fixed it, and it is now a default in our shared configuration.

I lead the pipeline platform. I designed and built the notebook operator and guided the team through its second version (templating and deferrable mode), built the Planet subscription DAG and ingest stage, parallelised batch processing, and led the cost review and right-sizing.