Python file automation is reliable when every job can be rerun without creating duplicates or half-finished output. That means a clear contract for each incoming file, validation before anything changes, a ledger of what already ran, and outputs that only appear once all checks pass.
Three file jobs make good starting points: checking supplier exports before import, producing recurring reports, and preparing import batches for another system. Each one follows the same pattern. Keep the input, validate it, transform it, reconcile the result, and release it only when the checks pass.
What should a file contract include?
A file landing in a folder does not prove it is ready. An upload may still be running, a name may be reused, or yesterday's export may arrive again. Before writing any code, we agree a readiness signal with the system that sends the file, such as an upload-complete event or a temporary name that changes once the transfer ends.
The contract then lists the filename pattern, source system, business date, encoding, delimiter, required columns, size limit and allowed value ranges. It also answers a question that trips up many teams: does a file with zero rows mean "no activity" or "the export failed"? Someone has to own that answer.
A few details cause many file bugs. Account numbers and product codes are identifiers, not numbers, so leading zeros must survive. Dates should be parsed in the agreed format, not guessed from a computer's regional settings. For money, Python's decimal module gives exact decimal arithmetic, but the business still has to choose its rounding rule.
How do you validate supplier files before import?
Take a nightly supplier export with item codes, quantities, unit prices and a business date. The job should reject a wrong layout before it reads a single row.
Use Python's csv module rather than splitting lines on commas, because quoted fields can contain commas and line breaks. The documentation says to open files with newline='', otherwise line breaks inside quoted fields are not read correctly. DictReader gives you rows by column name, but it does not check that required fields exist. That part is yours to write.
Our validation order is:
- Confirm the file is complete, readable and within the size limit.
- Compare headers with the contract, catching missing and duplicate names.
- Check row shape, required IDs, dates, number formats and allowed ranges.
- Find duplicate business keys inside the batch.
- Reconcile row counts and totals against a supplier manifest, if one exists.
- Produce a clean output or a clear rejection report, never an unclear partial result.
If some rows may pass while others fail, agree that rule in advance and give every rejected row a reference and a reason. Log the run ID, row reference, field and error type, not full customer records.
How do you stop a report from looking complete when it is not?
A polished report can still be missing a whole source. We build a manifest for every run: expected inputs, inputs received, the business period covered, row counts, validation result, code version and output location.
Completeness should be a release condition. If a daily report needs four regional files and only three arrive, either hold the report or publish a clearly labeled partial version somewhere agreed. Never overwrite yesterday's complete report with a partial one under the same name.
Keep business time and run time separate. A Tuesday report may run on Wednesday after a late file, and both timestamps should be stored. If the output will be opened in a spreadsheet, plan for formula injection too. The OWASP CSV Injection page lists the characters that can turn a cell into a formula and notes there is no single fix that works for every spreadsheet app, so test against the one your team uses.
How do you prevent duplicate imports when a job is rerun?
Creating an import file and applying it to a business system are two different steps. When an import changes inventory, invoices or customer records, we keep them apart: first a reviewable file and a summary of intended changes, then an approved submission, then a receipt from the target system that can be matched back to the batch.
A filename is not a safe duplicate key. We use a run ledger that stores the source, business period, a content checksum, the code version and the processing state. A checksum spots identical bytes, but two files with different formatting can describe the same events, so business actions also need a stable record key.
Corrections need their own rule. A corrected file might replace an earlier batch, reverse it or update it. That is a business decision, and a blanket "skip any filename we have seen" rule gets it wrong.
How do you write outputs so a crash leaves nothing half done?
Write to a temporary file on the same filesystem, close it, run the output checks, then move it to the final name. Python's os.replace is documented as atomic when it succeeds, and it may fail when the source and target are on different filesystems. Network drives and cloud storage behave differently, so test the storage you actually use. For object storage, use its own versioning and conditional writes.
Stop two runs from publishing to the same place at once with a lock or a claim, and decide how a stale claim gets cleared. Keep the original input until your retention rule says it can go.
What should an operator recovery checklist cover?
A retry should be a deliberate step, not pressing Run again and hoping. Give each run an ID and track clear states: received, validated, prepared, released, acknowledged and failed. Python's logging module can attach that context to every event.
Before go-live, we test these cases with made-up data:
- A duplicate delivery, and a corrected delivery with the same filename.
- A missing column, an extra column, a bad amount and a quoted multi-line field.
- A valid zero-row file and a late source file.
- A crash before publishing, and a crash after publishing but before acknowledgment.
- Two overlapping runs and an unavailable destination.
For each case we write down the expected outcome: reject, retry, review or succeed. Alerts go to a named owner when a business deadline is missed, not only when a script exits with an error. For document-heavy flows, our invoice-to-ERP automation blueprint applies the same thinking to PDFs and scanned files.
Frequently asked questions
It depends on who owns it and how complex it is. Python suits repeatable parsing, testing and controlled runs. A small task that lives entirely in one spreadsheet may not need a separate tool.
Combine a run ledger with business keys or duplicate protection in the target system. Track acknowledgment separately from submission, because a timeout does not prove the other system rejected the batch.
Only when the fix is explicit and approved. Trimming agreed whitespace is fine. Guessing a missing customer ID is not. Keep the original value and record any change that affects meaning.
Start with a bounded, reversible preparation job whose inputs and output can be checked. Add writes to other systems only after validation, monitoring and recovery have been proven.
Sources
- Python Software Foundation: csv module
- Python Software Foundation: os.replace
- Python Software Foundation: decimal module
- Python Software Foundation: logging module
- OWASP: CSV Injection
Turn a fragile file job into an owned workflow
If a recurring export or import has become a quiet dependency for your business, we can map its inputs, checks, release deadline and recovery steps, then build the automation and monitoring around it. We also connect the result to your other systems through API integration.

