manningtree

DNG Roars

(or Why my digital negatives are DNGs)

Having got back into proper photography I quickly fell slap bang into the middle of the ‘digital negative’ debate. In this digital world, what is the best equivalent of the 35mm; 120 and odd 5×4 negs I have stashed away?

Around the time I started grappling with this, Juergen Specht and others launched openRAW.org with the objective of persuading camera manufacturers to openly publish their camera RAW formats and so avoid the RAW problem. Adobe has also recognised this problem and responded with the DNG format, a common specification for digital negatives, supported by software to convert from proprietary manufacturers formats to DNG.

My first inclination was to store everything as .cr2; the format that my camera provides as camera RAW. That seamed like a good idea until I realised that canon had recently dropped support for camera RAW files from an older camera body in a recent update of their camera RAW processing software. A further read of some of the background on openRAW.org made me nervous of that route to say the least.

Next stop was to consider conversion to something like 16-bit LAB encoded TIFF, using the software supplied by Canon. The problem with this route is that there’s information in the camera RAW data that isn’t retained when you go to TIFF. In the future, it’s possible that improved conversion algorithms could make better use of that RAW data so moving everything to TIFF today didn’t seam sensible.

So what about DNG
? Well, in the end DNG is what I decided to use. Here’s why.

For me, there are two high level requirements for a ‘digital negative’:

1. I want an archival format. If I store image data in a format, I want confidence that someone will be able to access that image in 10, 20, 40 or 80+ years time.

2. I want the format to be complete & comprehensive. It must capture all the raw data from my camera and enable re-interpretation of the image (i.e. re-printing) at a later date. Furthermore, I want a high probability that the format will be able to do the same for other cameras I might use, both current and future models.

These high level requirements boil down to needing a format that:

1. Retains all the data that my camera’s RAW format provides
2. Is likely to be able to retain all the data that an alternative or future camera RAW format provides
3. Will be supported in the future by software that is capable of extracting all the data from the format even when the current hardware and software I’m using is long forgotten

Tough one that.

I could stay with .cr2 with the hope that the openRAW.org initiative succeeds and Canon publish .cr2. But that wouldn’t help me if I borrowed a Nikon or later updated to a RAW capable Sony compact (see my second high level requirement). For some reason many people seam to prefer publication of a proprietary format (like .cr2) over adoption of DNG. However, assuming all manufacturers did openly publish the formats, there’s still a host of problems to overcome. These include:

Ambiguity: it is very difficult to write a specification that is unambiguous and that can be reliably implemented by third parties. It is especially difficult if you are specifying something that is already implemented. Typically, the better specifications are drafted to solve a problem and implementations are then done against the specification (preferably, with the results of initial independent implementations feeding back into the drafting process). If you’re trying to document something you’ve already implemented, it’s very hard to cover everything and to get it right.

Implementation: having a specification doesn’t mean you have access to the image data. To get access someone, somewhere, needs to write code to extract and process the raw data. Either that or you need a very large sheet of graph paper, a sharp pencil and more time than I have.

Intellectual Property: having a specification doesn’t give you the right to implement that specification. There may be intellectual property that you need to license before you can implement. Just as one example, Unisys had patents that covered the use of LZW compression in the Compuserve GIF image format, even though the GIF specification was available on-line.

Trade Secrets: A company may have a clever trick or optimisation that they use that publication of the full format specification would expose. In some cases you might get a partial specification published that omitted details of this trick meaning that although access to the image is possible, fidelity and performance may not be optimal in an independent implementation.

Critical Mass: The more widely used a format is, the more likely you’ll find an application or source code that can access it. Having a well documented but obscure format won’t do you much good in 50 years when it’ll cost a fortune to get someone to implement a reader in whatever programming languages happen to be around then. Conversely, if you use a format that is more widely used, maintaining access to implementations in the future is likely to be easier.

When you look at that list, it’s obvious that having the manufacturers publish the relevant specifications doesn’t solve the problem. Looking at DNG, there’s several advantages over publication of proprietary camera RAW formats.

DNG is published: Pretty obvious, but DNG is published already. You can get the specification from Adobe, and just in case it disappears you can download and save a copy locally or even print a copy out onto that old fashioned paper stuff. [I have a couple of copies stored already - JIC :-)]

Free implementations: There are already free tools able to convert to DNG. Adobe provide the free Adobe DNG Converter for both Windows and Mac and there are also an increasing number of applications that can read DNG.

Widespread implementations: Because the format is published. People other than Adobe are adding support for DNG.

Track record: Adobe have a good track record of publishing and maintaining high quality specifications that third parties can implement. Examples include PostScript, PDF and TIFF (acquired with Aldus). PDF is used as the basis for international standards like PDF/X (ISO 15930-4:2003 & ISO 15930-5:2003) and Adobe is on recor
d
as saying “Ultimately, it may make sense to turn over DNG to an appropriate standards body for further enhancement, so that its evolution can truly be a collaborative effort.”

Unambiguous: I haven’t implemented a DNG reader or a DNG writer so I don’t know how DNG stacks up. But there are independent implementations and critical mass suggests that any ambuguity that might exist will get ironed out pretty quickly.

DNG is comprehensive: it has to be because to support all the different camera RAW formats it needs to provide a superset of the capabilities of any one model. That makes it far more likely to meet my needs for a format that supports other current and future cameras. [Aside: Barry Clive Pearson asked how frequently Adobe expected to rev DNG to maintain compatibility with new camera technology, Thomas Knoll’s response was (to paraphrase) not that often).

Critical mass: because DNG supports the RAW format from multiple cameras it already has critical mass. This makes it far more likely that implementations will exist in the far distant future.

DNG is based on TIFF: like some other RAW formats (but unlike others), DNG is an extension of TIFF. TIFF is widely implemented with high quality source code (like libtiff) available publicly. This makes it easier to implement DNG and adds to confidence in its longevity.

Open Specification: there’s big debate over whether DNG is ‘open’. The problem is all about how you define ‘open’. For me, and the criteria I’m using, DNG is ‘open’. For you, and your criteria, it might not be.

I trust Adobe: This one is purely personal, and probably doesn’t mean anything to anyone else reading this. Adobe does a good job in maintaining open standards that third parties can implement and use to add value both to their own business and the ecosystem around the format. There are whole businesses, paying many people’s mortgages, based on the published PostScript and PDF specifications. There’s also increasing evidence that this model is being adopted by other companies in the computing industry who are seeing their value in the software that manipulates and processes data from open and standard formats, rather than in having a locked and proprietary format (as one piece of evidence, I’ll just point to Microsoft opening up the default formats for the next version of Microsoft Office).

That’s why I’ve gone with DNG, it’s the best available format for my requirements. Does that mean that DNG is the answer to all the issues? I’m afraid not. There’s also the problem of the second, or digital, dark age. Having a published specification and a software implementation doesn’t mean squat if I’ve burnt all the data onto DVDs and no longer have access to a DVD reader. Likewise if the DVDs have deteriorated. Sound silly? Well until very recently I had some early digital pics on 3.5″ floppies. Question: is there a floppy drive on the machine you’re reading this on? I’ll need an ongoing program of transferring content to contemporary media, verifying the copies and maintaining duplicates…..

Footnotes:

For links to related information see these del.icio.us tags:

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>