18th Annual Preservation Conference
18th Annual Preservation Conference
Preservation Reformatting:
Digital Technology vs. Analog Technology
Overview: Analog vs. Digital for Preservation Reformatting
By Steven Puglia
Preservation and Imaging Specialist
U.S. National Archives and Records Administration
8601 Adelphi Road, Room B810
College Park, MD 20740, USA
Phone: 301-837-3616 (Office) or 301-837-3706 (Digital Imaging Lab)
Fax: 301-837-3703 (Office) or 301-837-0958 (Still Photo Lab)
Email: steven.puglia@nara.gov
March 27, 2003
There is no single or simple answer.
- Technology is never “THE ANSWER”.
- Technology is only tools.
- The answer is to select and use your tools wisely.
We have been seduced by and sold on the promise of new technologies.
Our imagination always outpaces our technology.
Jason Ohler, Taming the Beast - Choice and Control in the Electronic Jungle, Technos Press, Bloomington, IN, 1999. Page 11.
Technology is an accelerator of greatness already in place, never the principal cause of greatness or decline.Great companies first build a culture of discipline - disciplined people who engage in disciplined thought and who take disciplined action - and create a business model that fits squarely in the intersection of three circles: what they can be the best in the world at, a deep understanding of their economic engine and the core values they hold with deep passion. They then use technology to enhance these pre-existing variables, never as a replacement.
Jim Collins, How Great Companies Tame Technology,
Newsweek,
April 29, 2002.
New technology often takes us so much by surprise that we feel we have
been ambushed by the future.
Jason Ohler, Taming the Beast - Choice and Control in the Electronic Jungle, Technos Press, Bloomington, IN, 1999. Page 27.
It is the big-picture consideration of technology that allows us to see it in all its horror and glory, in terms of the promising but uncertain future it guarantees us.
Ohler, p. 25.
The possibility of de-technologizing, even if we had an intense desire to do so, is problematic. Technology has a way of ingratiating itself into our existence so thoroughly that we let it hang around, like the guest who comes to dinner and forgets to leave.
Ohler, p. 46.
To overcome our inherent insecurities, many of us spend an inordinate amount of energy trying to limit change. Developing attachments to rapidly evolving technology only amplifies and exacerbates the stress involved in this process.” “Today’s goal is to try to be at peace in the eye of the hurricane and, on a good day, even enjoy the ride.
Ohler, p. 38.
And the ultimate goal is control…which of course, as many philosophers of technology make quite clear, is illusory if not impossible. Yet I suggest that the only illusion more destructive than believing we can control our technology is believing we can’t. So choose your illusion.
Ohler, p. 44.
We can not ignore digital technology, it is not going to go away.
We must learn to:
- Be critical observers of technology.
- Honestly evaluate its strengths and weaknesses.
- Be sophisticated users of all appropriate tools.
Four stages of literacy-
Stage 1. Seeing
Stage 2. Absorbing
Stage 3. Forgetting
Stage 4. Seeing plus evaluating
Ohler, p. 18.
- Static Analog Media
Printed publications, photographs, microfilm, and digital hardcopy output.
- Dynamic Analog Media
Motion pictures and analog video.
- Digital Media
All types.
- Static Analog Media
Chemical and physical stability limits usable life. Dynamic
- Analog Media
Chemical and physical stability and systems obsolescence limits usable life.
- Digital Media
Systems obsolescence is critical factor, but chemical and physical stability can be problematic
Increasing complexity and corresponding risk of loss. Increasing expense to reformat and to preserve.
“…complexity always overwhelms simplicity.”
“…a technological universe absorbs its non-technological opposition.”
Ohler, p. 41.
"The complexity of…systems makes it inevitable that in actual use, some great flaws will appear that were hidden from designers."
"But it is notoriously difficult to predict all problems, or their levels of difficultly, in advance."
"One mark of a newer technology is that while it is cheap in routine operation, it is expensive to correct and modify"
"Once more, the price of protection is chronic vigilance."
Edward Tenner, Why Things Bite Back: Technology and the Revenge of Unintended Consequences.
Considerations for Reformatting:
- Archival considerations
- Costs
- Reproduction quality
- Stability of imaging materials
- Ease of distribution
- Availability of equipment
Cost-Benefit Analysis:
- Reformatting is almost always one of the most expensive options, compared to providing the proper storage for originals to extend the usable life.
- Machine-dependent records often do not benefit as much due to systems obsolescence.
Reformatting is needed for-
- Actively deteriorating materials.
- Frequently requested items to minimize handling and allow originals to stay in proper storage.
- Providing continuing access to records on obsolete formats.
Archival Medium is a recording material that can be expected to retain information forever, so that such information can be retrieved without significant loss when properly stored.
However, there is no such material, and it is not a term to be used to describe material or system specifications in American National Standards.
Life Expectancy (LE) is the length of time that information is predicted to be acceptable in a system at 21°C and 50% RH.
LE Ratings:
Longevity of Digital Optical Media: 30 to 200 years
Digital Data System Obsolescence:
5 to 10 year system life (maybe less) is critical factor.
Paul Conway, Duke University
"Future access depends on an unbroken chain of such migrations frequent enough to prevent media from becoming physically unreadable or obsolete before they are copied."
"A single break in this chain renders digital information inaccessible, short of heroic effort."
Jeff Rothenberg," Ensuring the Longevity of Digital Documents," Scientific American, Jan. 1995.
“One great obstacle is the creeping plague known as ‘digital decay.’ Awareness is spreading…that information stored digitally is much more vulnerable to corruption than people who are rushing to embrace that alternative have assumed.” “The upshot is that while the development of practical digital archiving procedures may be inevitable, it is not imminent.”
David Chute, “Film Preservation at the (Digital) Crossroads,” http://www.geocities.com/Tokyo/Island/3102/f-prez.htm
Systems Perspective:
Managing and preserving digital data/objects/records is different than managing and preserving physical records.
Preservation Microfilm Model:
- Human readable with magnification only.
- Life expectancy of 500 years when stored properly.
Preservation Microfilm Model:
- Duplicate only every 50 years to produce new printing master.
- Lower cost to store.
Digital Preservation Model:
- Machine readable only, system dependant.
- In an unmanaged environment, life expectancy of system only 5 to 10 years.
Digital Preservation Model:
- Constant management.
- Higher cost to maintain data. Need large-scale automated systems to manage digital data and make it cost effective.
In an unmanaged environment, digital has 1/100 the life expectancy of microfilm (5 years compared to 500).
Digital has to be handled more often than microfilm (constant activity).
“Digital preservation is hard, and will consume significant amounts of money, attention, and talent.”
Dale Flecker, Harvard University, “Thinking About a Local Digital Preservation Program,” March 2003.
Textual-
- Nicholson Baker controversy.
- Preserve the original, the artifact?
- Yes, we are replacing bad microfilm with bad digital images.
- Preservation quality reformatting using digital imaging is possible.
- In many cases can do a better job via digital imaging than microfilm - including
the ability to reproduce color.
“Presenting a superior long-term strategy. Longevity. Not something you necessarily associate with technology, right? Consider this: the…DR-5060F simultaneously scans and provides backup on microfilm.
So basically we’re talking about a scanner with a photographic memory: an archival medium that will still be readable in twenty, fifty - even a hundred years.”
“It’s a superior long-term archival strategy because you don’t have to constantly migrate your data files as technology evolves….we’re giving people the know-how to not just scan, but also to preserve records for generations.”
Canon advertisement.
Still photographs-- The market is moving to digital photography.
- Hard to produce a preservation surrogate using digital imaging.
- No defined standards for digital preservation surrogates.
For still photographs, analog storage - keeping the originals and photographic duplication - offers greater longevity and less risk for long-term retention.
The Tools:
- Proper storage for originals- cool and dry. Cold storage for acetate, nitrate and color.
- Photographic duplication for deteriorating originals.
- Digitization to facilitate access and reproduction.
Corbis - Bettman Archive
- Began scanning in 1996 at a cost of $20 per photograph.
- Stopped systematic digitization after scanning approximately 225,000 photographs, out of approximately 7.5 million original images (11 million total).
- Corbis has approximately 2.1 million digital images online for licensing,
out of 65 million total.
“If Corbis had scanned everything it would have taken 25 years to finish. And that was time the pictures did not have.” “The images scanned first were those deemed most valuable, both culturally and commercially.”
“A Century’s Photo History Destined for Life in a Mine” by Sarah Boxer, NY Times, April 2001.
Scanning will continue on demand and at a slower rate, while the original negatives are stored in a new cold storage vault for long term preservation.
Cold vault maintained at minus 20 degrees C (minus 4 degrees F) and 35% relative humidity.
“By housing the collection in a secure preservation environment, by offering so many of the images online, by being set up to digitize and fulfill orders through an on-site lab at the storage facility, and by offering in-person access when appropriate, Corbis is able to achieve it two key goals - preserving the collection and preserving access to it.”
Corbis press release, April 29, 2002.
Technical Issues- Duplicating Still Photo Negatives:
- Method of producing duplicates.
- Tone reproduction approach.
- Objective means of evaluating tone reproduction.
- Variability.
- Choosing an appropriate film.
Method of producing duplicates:
- Print and copy negative of print
- Two-step duplication- Interpositive (IP) and Duplicate Negative (DN)
- One-step duplication- Direct Duplicate Negative (DDN)- Positive acting film or reversal processing
- Contact printed vs. optically printed Proper processing.
Tone reproduction approach- still photo negatives
Traditionally- use film with long straight-line portion of characteristic curve, developed to a contrast of 1.0, and expose so image is rendered on straight-line portion of curve.
Conceptually, dates back to the first sensitometry experiments of Hurter and Driffield.
Approaches to Tone Reproduction:
- Standardized Exposure. -optional shadow mask to extend tonal range (Chicago Albumen Works)
- Shadow Normalization.
- Hybrid Standardized Exposure and Highlight Normalization.
Objective means of evaluating tone reproduction.
Variability:
- Measured the variability of the duplication process.
- Suspect much of the variability is in the measurement process itself.
- Established variability limits for individual IPs/DNs/DDNs and for batches.
Choosing an appropriate film-
- Medium contrast- gamma of 1.0.
- Long tonal scale- can accurately duplicate original negatives with density range of up to 2.0.
- Fine grain and high resolution.
- Polyester base- LE 500.
- Panchromatic emulsion preferred- minimizes effect of staining.
Technical Issues- Digitizing Negatives:
- Approach to digitizing.
- Tone reproduction approach.
- Objective means of evaluating tone reproduction.
- Variability.
- Selection of scanner/camera.
- Image processing techniques.
Digital Resolution for Photographs:
Digital Dynamic Range and Colors for Photographs:
Est. File Sizes for Preservation Quality Scans of Photographs - Negatives or Transparencies:
Approach to digitizing:
- Scan as positive- digital image has negative tone orientation
- Scan as negative- digital image has positive tone orientation
- Use scanner
- Use digital camera
Audio and Video-
- Analog equipment hard to acquire - the market has gone to digital equipment.
- Moving to higher bit depths and sampling frequencies compared to older standards like Audio CD. Trying to make the recordings more like analog.
Motion pictures-
- The market is moving to digital photography.
- Hard to produce a preservation surrogate using digital imaging.
- No defined standards for digital preservation surrogates.
“Ever since I started in the movie business in 1952, I have been hearing that film would not be around very much longer. It was always just about to be replaced with something electronic.”
Roger Mayer, President and COO of Turner Entertainment Company.
“Movie images are already often largely digital. 85% of the shots in The Matrix, and close to 95% of those in The Phantom Menace, have a digital component.”
“With digital restorations like these, Robert Gitt says, ‘the footage comes back looking good, but not quite as good as if you still had the original elements….’ ”
David Chute, “Film Preservation at the (Digital) Crossroads,” http://www.geocities.com/Tokyo/Island/3102/f-prez.htm
“Film as a capture medium is still hard to beat. And it’s still the only archiving format that you would ever trust. The idea of having a file of a movie that you considered to be archival and indestructible is ludicrous.”
Steven Soderbergh in Mike Snider, “The Digital Force is with Big-Time Directors,” USA Today, August 4, 2002.
“It is nevertheless a big leap to the notion that digital media could be the ultimate long term archival platform for visual information. Even the most ardent digital futurists warn that before that can happen, there are major wrinkles to be ironed out. For one thing, visual information is a notorious glutton for storage space. At a conservative estimate, 360 gigabytes of tape or disk capacity would be needed to store all the information on the 150,000 frames of an average-length feature film.”
“ ‘The costs are horrible,’ admits ardent digital advocate Timo Muinonen, Deputy Director/Head of Collections at the Finnish Film Archive. ‘Compare the $1.5 million it cost Disney to digitally restore Snow White to our most expensive duplicates onto safety film: we can copy six SovColor nitrate features for $100,000 and get a dupe-negative, a dupe-positive and a print for each one. But the prices of digital systems continue to get lower, and lab costs continue to rise, so the situation is changing rapidly.’ ”
David Chute, “Film Preservation at the (Digital) Crossroads,” http://www.geocities.com/Tokyo/Island/3102/f-prez.htm
Digital-
- Everyone still trying to determine all that will be necessary to preserve digital data over the long term.
- Often, will need to preserve digitally created materials using digital technology.
- Media reversion is not an option for many types of records, will lose functionality and then it is no longer the record.
- OCLC
- Actively manage risk.
- Emphasize data security, including back-ups, distributed storage of multiple copies, etc., to prevent catastrophic loss of records.
- A hard drive crashes every 15 sec.
- 1 in 5 computers suffer a fatal hard drive crash during their lifetime.
- 25% of lost data is due to the failure of a portable drive.
- 31% of PC users have lost all of their PC files to events beyond their control.
- 32% of data loss is caused by human error.
- 60% of companies that lose their data close down within 6 months of the disaster.
Data Loss Statistics, www.kiesoft.com/eb/crashstat.htm
Cost of Data Loss:
“IT Performance Engineering & Measurement Strategies: Quantifying Performance Loss”, Meta Group, Oct. 2000.
Build appropriate information technology infrastructure, including well designed and robust IT systems.
Be prepare to spend lots of $$$$$$$$$$.
IT systems have a limited life, so be prepared to spend more $$$$$$$$$$.
“Digital preservation programs are hard and costly. There will be few large players.”
Dale Flecker, Harvard University, “Thinking About a Local Digital Preservation Program,” March 2003.
On-Going Costs
Cost for minimal maintenance of one set of the master image files (off-line) and access files (on-line) during the first 10 years likely to be 50% to 100% of initial investment.
Cost for maintaining image files in large-scale automated digital repository during the first 10 years likely to be 10% to 25% of initial investment.
Cost to install, staff and maintain network infra-structure and the digital data for 1st 10 years is up to 5 times the initial investment.
In the IT world, typically the full lifecycle cost (for a 7 to 10 year lifecycle) for a system is 10 times the cost of development.
Conclusion: Use each technology to its best advantage to minimize the disadvantages of the other.
“Never go to excess, but let moderation be your guide.”
Cicero“Moderation is a fatal thing. Nothing succeeds like excess.”
Oscar Wilde