Darwin's Duplicate Image Problem: The Key Decisions Ahead for Territory Archives

NT government agencies are sitting on tens of thousands of duplicate digital images spread across at least three separate records management systems, and the decisions made in the coming months will determine whether years of archival work — including irreplaceable records tied to Aboriginal land claims and remote community programs — survive the cleanup intact.

The issue has sharpened because the Department of Infrastructure, Planning and Logistics is mid-transition toward a consolidated digital asset platform, with a staged rollout that began in the first quarter of 2026. When agencies migrate records and encounter duplicate image files, standard protocol calls for automated deduplication — a process that, without proper oversight, can strip context metadata or delete master copies while retaining compressed derivatives. For collections tied to Native Title determinations or the Northern Land Council's documentation work, that distinction matters enormously.

What's At Stake on the Ground

The practical pressure is visible at two sites. The NT Library and Archives, located on Married Quarters Road in Darwin's parliamentary precinct, holds digitised collections dating to the 1870s. Staff there have flagged that automated deduplication tools cannot reliably distinguish between a true duplicate and a near-identical image taken seconds apart — a distinction that matters for photographic evidence used in heritage assessments. Separately, the Darwin office of the Northern Land Council on Esplanade has raised concerns through internal channels about image records supporting land-use agreements across Arnhem Land being caught in bulk-delete queues during agency migrations.

The Territory Records Office, which sits within the Department of Chief Minister and Cabinet, is the statutory body responsible for setting retention standards. Its existing schedule — Territory Records Act Schedule 11, covering digital assets — does not specifically address image deduplication workflows generated by AI-assisted migration tools, which are newer than the schedule's last substantive revision.

That regulatory gap is the crux of the problem. Agencies are making ad hoc calls. Some are opting for conservative approaches that retain everything, generating storage bills that have climbed sharply as data volumes grow. Others are applying aggressive deduplication that may not survive a future audit or legal challenge.

The Decisions That Will Define the Outcome

Three concrete choices are coming. First, the Territory Records Office must decide by the end of the 2026 financial year whether to issue an emergency interim direction covering AI-assisted deduplication, or to wait for a full schedule review — a process that typically takes 18 months. Second, the Department of Infrastructure, Planning and Logistics must determine which agencies proceed to the next migration phase before that guidance exists. Third, the Northern Land Council and other First Nations organisations with co-management interests in digital records need to decide whether to formally request third-party audit rights over image collections held by government agencies on their behalf.

The stakes are not abstract. Under the Aboriginal Land Rights (Northern Territory) Act 1976, documentary evidence — including photographic records — can be material to compensation and royalty disputes. Royalty payments from the Amadeus Basin and offshore Timor Sea gas fields run through Land Councils that rely partly on historical records to verify traditional ownership continuity. A carelessly deleted image set is not recoverable.

Storage costs provide some context on the financial pressure driving hasty deduplication decisions. Cloud archival pricing for government agencies in the NT, based on published whole-of-government procurement arrangements, sits at roughly $23 per terabyte per month for warm storage. An agency holding 500 terabytes of unrationalised image archives — not an unusual figure for a department managing remote community infrastructure photography dating back two decades — faces bills exceeding $11,500 a month. That pressure is real, but it is not a justification for skipping governance.

The most defensible path forward involves a short moratorium on bulk image deletion across agencies with First Nations-linked collections, combined with a rapid interim directive from the Territory Records Office covering minimum metadata retention standards before deletion is authorised. Agencies proceeding with migration should be required to log every file flagged for deletion to an auditable register accessible to relevant Land Councils. The technology to do this exists. The question is whether the NT Labor government will require agencies to use it before the next migration phase begins, likely in September 2026.