Ten million S3 list requests from one Airflow run

What we saw

One morning the S3 request line on our AWS bill looked wrong. Storage was flat, data transfer was flat, but list requests on the bucket that holds our Planet imagery had jumped. One day accounted for most of the month's S3 request cost.

Tracing the requests led to a single source: the Airflow workers running our productivity-zone DAGs. In one day they had issued roughly ten million ListObjects calls.

Why it happened

Those DAGs read Planet GeoTIFFs directly from S3 with rasterio, which uses GDAL's virtual file system:

with rasterio.open("/vsis3/<bucket>/<field>/<date>.tif") as src:
    ...

By default, GDAL_DISABLE_READDIR_ON_OPEN is NO. Before GDAL opens a file, it lists the directory it lives in to look for sidecar files such as .ovr overviews, .aux.xml metadata and .msk masks. On a local disk that costs nothing. On S3 every open becomes a LIST request on the prefix.

The DAGs open two images per field per date across a three-month window, for every field in a batch. That adds up to millions of opens, and each one was preceded by a list request. Planet delivers plain GeoTIFFs, so there were never any sidecar files to find.

Why we hadn't hit it before

We had already solved this once. Our shared Python package sets the right GDAL options before it reads Sentinel imagery. But that only applies to code that goes through that helper. The Airflow worker pods themselves had no GDAL configuration at all, so any task that called rasterio directly got GDAL's defaults.

The fix

Instead of patching each DAG, I added the settings to the environment secret that every Airflow pod loads. Now every task on the platform gets them, whichever library it uses to read rasters:

GDAL_DISABLE_READDIR_ON_OPEN: "EMPTY_DIR"
CPL_VSIL_CURL_ALLOWED_EXTENSIONS: ".tif,.tiff,.vrt,.json"
GDAL_HTTP_MERGE_CONSECUTIVE_RANGES: "YES"
VSI_CACHE: "TRUE"

• EMPTY_DIR tells GDAL to treat the directory as empty, so it skips the listing.
• ALLOWED_EXTENSIONS stops GDAL from probing for sidecar files with HEAD and GET requests.
• MERGE_CONSECUTIVE_RANGES combines adjacent byte-range reads into one request when reading windows of a raster.
• VSI_CACHE keeps file headers in memory, so repeated reads of the same file don't fetch them again.

I applied the change to the production secret the same day. Workers are autoscaled on queue depth, so they picked up the new environment on their next scale-up without a separate rollout.

What I took from it

• Libraries built for local disks carry defaults that are expensive on object storage. GDAL, HDF5 and anything that walks directories are worth checking.
• Set this kind of configuration at the platform level, not in one helper. If it lives in a library, only the code that remembers to call the library is protected.
• Watch request metrics, not just storage. A bucket that isn't growing can still cost a lot to read.