Every Azure OpenAI Batch Suddenly Failed with 500: the 500-File Limit and the API That Hid the 400

Short version: if your Azure OpenAI Batch code has no expires_after and nothing that deletes files, that feature will eventually stop completely. After this post you can work out when, and decide how to clean up.

At work (a contract management SaaS on Rails) we have a feature that extracts contract fields with AI in bulk. One morning it began failing on the first chunk for every organization. It took nearly a week to fix. This one line was all the Rails error monitor showed.

Batch create failed (500)

A 500, so I assumed our extraction server had died. Wrong.

The real cause was at the bottom layer

The Batch API takes a file full of requests and works through them asynchronously within 24 hours. The chain is Rails Job, then extraction API (Python), then Azure OpenAI, with one batch per chunk (a fixed-size slice of contracts). Each batch uploads a JSONL input file (one request per line) through the Files API and reads an output file at the end. Nobody ever deleted those files.

Azure accepts 500 batch input files per resource when they have no expiry. Past that it returned 400 quotaExceeded, and the extraction API caught the exception and passed Rails a 500.

To see the cause we had to open the errors on the extraction API side. The caller had no trace of a 400 or a quota.

Why it looked like one organization's bad data

The first failure was a run of a few thousand contracts, and it stopped about halfway. The moment the file count hit the limit, the remaining chunks got rejected. Look at that alone and you start suspecting that organization's data.

It was really a counter shared by the whole resource. After that, anyone who clicked got a failure, and people clicked several times. The file list showed 501 input files. I don't know why that's one off from the documented 500.

When will your resource hit it?

The math is simple. One input file per batch, so 500 / batches per month gives you months. We hit it less than two months after launch.

Batches/month   Hits the limit in
  50            10 months
 100             5 months
 300            about 1.7 months

If you split work into chunks, one user run makes several files, so count chunks and not runs. To see where you stand now, list the Files API and count the ones with purpose batch.

Delete vs expire vs periodic sweep

I'd set the expiry first. It's one line and no failure path can skip it. Add deletion if the input is sensitive. Relying on deletion alone makes me nervous: one exception leaves a file behind, and months later that's an outage.

batch_file = client.files.create(
    file=open('chunk.jsonl', 'rb'),
    purpose='batch',
    extra_body={'expires_after': {'seconds': 1209600, 'anchor': 'created_at'}},  # 14 days
)

batch = client.batches.create(
    input_file_id=batch_file.id,
    endpoint='/chat/completions',
    completion_window='24h',
    extra_body={'output_expires_after': {'seconds': 1209600, 'anchor': 'created_at'}},
)

The numbers come from the official docs. seconds must be 1209600 to 2592000 (14 to 30 days). Divide the 10,000 limit by 14 days and you can sustain about 700 files a day. Output and error files don't count toward this limit, but they have no default expiry and stay forever if you leave them. Hence output_expires_after too.

One more thing. Files you already uploaded have no expiry, so fixing the code won't lower the counter (my inference, the docs say nothing about backfilling). You have to delete the old ones yourself.

I pointed the openai SDK 2.48.0 at a local stub server to see the request. extra_body went out as the form field expires_after[seconds], while the curl example in the docs uses expires_after.seconds. If you call HTTP directly, follow the docs. I did not re-run this against real Azure.

Don't flatten 4xx into 500

Cleaning up files is half of it. What actually cost us time was this shape in the middle API.

# Before: anything becomes a 500
try:
    upload()
except Exception:
    raise HTTPException(500, 'Batch create failed')

# After: pass on the upstream status and code
try:
    upload()
except openai.APIStatusError as e:
    raise HTTPException(e.status_code, {'code': e.code, 'message': e.message})

Against a stub with a limit of 3, the first gives HTTP 500 and the second gives 400 quotaExceeded. Had the caller received that one code, the investigation would have taken minutes. Careful with an upstream 401, though: pass it through as is and the caller thinks its own auth broke. Turn those into a 502 and put the original code in the body.

The emergency fix that day

Order matters. First we checked that every batch was completed and nothing in the app was waiting on results. Then we deleted the input files and the output files already applied, a bit over a thousand in total. Outputs from failed extractions stayed for the investigation. Last, we uploaded a test file from a production container, saw it succeed and deleted it.

Easy to miss: extractions that failed during the outage don't re-run by themselves. Someone has to click again, so tell the affected organizations.

I couldn't confirm whether OpenAI itself (not Azure) has the same limit. I couldn't read that docs page, so check it yourself. You can also skip the Files API entirely: Azure documents a Batch option that uses Blob Storage.

If a working AI feature stopped one day and you can't see why, you can ask negabaro. We can go down the layers with you and find the cause.