NYPL Digital Preservation

Why DPI

Mon, 03 Oct 2022 14:00:00 +0000

“Images for publication should be at least 300 DPI”

You’ve probably seen something like this on the licensing/reproduction/etc pages of many cultural organizations. And in reality, that sentence doesn’t make much sense. A couple of recent incidents with born-digital images got me to think about this more deeply. In the process, I realized I’m still not sure what would make more sense, but I’m giving it my best go here.

The Incidents

We have the collection of a photographer that shot digitally, and someone wanted to review files for potential reuse in a publication. The library didn’t hold the copyright to these items, so in order to share them for review we needed to decrease the quality below publication levels. Each photo was direct from the camera. Each photo had a DPI of 72.

These were high-quality 40 megapixel images. The 72 DPI meant that none them were “publication reading”. But opening a file made it clear that we needed to significantly down-res the photos before sharing them.

In another collection, we have examples of pixel art which someone wanted to include in a research article. The images were created digitally with a DPI of 72. After sharing the images, the person contacted us to say that they were not high-enough quality for publication. Again, these were the original files. There was no way to get a higher-quality file.

In both cases, the problem was not with the quality of the image. The problem was with the concept of DPI.

Resolution of Display

The ability of the object displaying an image to represent detailed features depends on the maximum density of small units of color that it can represent. In a world of printers, this is called Dots Per Inch (DPI), in reference to the ink droplets that it prints with. In one inch, a 300 DPI printer can lay down 300 evenly-spaced dots of ink. The more DPI, the more resolution, and the clearer the details.

In a world of screens, this is called Pixels Per Inch (PPI), in reference to the pixels in the display. In one inch, a 300 PPI screen can contains 300 evenly-spaced pixels.

However, size is relative. One inch of screen in your VR goggles looks larger than one inch of screen on your phone in your hands versus one inch of screen on you desk monitor. Correspondingly, the manufactures of these devices will use higher PPI screen depending on how close they will be to your eyes.

Similarly, with printed material the DPI used by a desktop printer will probably higher than the DPI used to print posters that are hung on the side of a building.

However, that is just the resolution of the display. The quality of the perceived image will depend on the resolution of what is being displayed.

Resolution of the Digital Image

Digital images, like digital displays, are broken down into pixels. However, unlike digital displays, they don’t have a physical metric. How many inches wide and tall should the 8x16 pixel sprite of Mario be? That really depends on user choice.

You can zoom out until you can’t distinguish Mario’s hat from his boots. You can zoom in until Mario fills the entire screen or page. In both of those states we can calculate another pixel per inch metric. An 13x16 pixel Mario blown up to 8x16 inches tall would be 1 pixel per inch.

Pixel art like Mario is a good kind of image to start with. The relationship between the pixel and the total image is part of the aesthetic. Thinking about other kinds of images, like photos, the relationship get blurrier.

A digital photograph consists pixels as well, although we’d prefer if we only perceived the objects that were photographed instead the pixels. As you zoom out, the pixels become invisible and the objects stand out. As you zoom in, the objects become less clear and the pixels become more obvious. At a very high level of zoom, the photo is an abstract grid of color blocks.

When Image Resolution Meets Display Resolution

Behind the scenes of zooming images in and out, the printer DPI and digital display PPI holds steady. For example, with a 300 DPI printer the 16-inch-tall printed Mario will be 4800 dots of printer ink tall. On a 1170x2532 pixel iPhone screen, the screen-size Mario would be 2532 display pixels tall. In both scenarios, the image is still 16 pixels tall, but each image pixel is represented by 90,000 ink dots (300x300) or about 23,000 display pixels (152x152).

This rendering of Mario’s sprite is represented by some number of pixels on your screen. It’s hard to say how many because that depends on what kind of screen you are using and how much you have zoomed in. And if you want to get technical, the sprite I’ve been using isn’t really 13x16 pixels. It’s been blown up to a 190x220 image, but we can still see the original pixels.

The upshot is that DPI/PPI doesn’t mean anything about objective image quality. What really determines image quality is if the image looks good displayed in context. A low-resolution photo printed large and examined up close will probably look terrible and pixelated. But, so will a high-resolution image printed at a medium size and examined with a magnifying loupe.

What Does it Mean When a Digital Image is 72 DPI?

Nothing.

Absolutely nothing.

If you think about it, how is it possible that so many images all have such neat DPIs, like 300 or 72. Where are the images with 57.897?

When numbers are so nice, it typically means someone was designing the system. In this case, this comes from the 1980s when computer manufacturers were designing printers and computers that would work well together. One of the designs that came out was to lock the digital display resolution to 72 PPI and the printer resolution to 144 DPI. That number stuck around for a few computer generations, and slowly it became part of the folk wisdom of computing. See the usage of .JPG instead of .jpeg or .jpg for a similar technical practice that doesn’t make sense.

When you take a picture with a modern digital camera, more often than not, the resulting JPEG will store a value that says it is 72 DPI. That does not mean that when you open the picture on your computer, that it will be scaled appropriately according to the DPI.

An iPhone 14 can take 8,604x6,048 pixel photos. At 72 DPI, that would be 120x80 inches, or 10x6.5 feet. The iPhone’s screen is 1170x2532 or about 3x6 inches. When you view the image on the phone, it ignores the DPI value and fits the image to the display.

When you open an image in Microsoft Word, it scales the image as well. The default page format in Word is US Legal or A4, about 8 inches wide. If Word respected the DPI when you added your iPhone photo, the image would be 10 times larger than the page you want to put it on.

Instead, Word scales down the image to a size that fits on the page. If you scale the photo across the width of the page, it would be about 1075 pixels per inch (8604/~8 inches). The printer receives those 1075 pixels per inch and interpolates them in 300 dots per inch, or ~3.5 pixels per dot.

That means that the 72 DPI image, which was supposedly not good enough for publication, has more information than the printer can actually represent.

How do I follow digitization guidelines based on DPI?

Many guidelines recommend that certain materials should be captured at a specified DPI. For example, FADGI guidelines for 3-star newspaper digitization is 300 DPI and for 3-star 35mm photographic negative digitization is 3000 DPI. FADGI doesn’t recommend a specific resolution.

This makes sense since any category will contain materials of different size. To capture at the same level of detail, the final resolution of the image will be relative to the physical size of the object. So, standards are defined based on how many pixels you should use to capture a base unit, and you can define the final resolution from there.

Take newspaper capture at 300 DPI. The New York Times is 12x22 inches. A digitized version would have to be at least 3600x6600 to meet the FADGI 3-star threshold. The South China Morning Post is 13x21 inches. A digitized version would have to be at least 3900x6300 to meet the threshold.

To capture both of these with the same camera, the sensor would have to be able to capture a grid of at least 3900x6600, or 25.74 megapixels. Cameras often have a 4x3 height to width ratio, so an actual minimum camera would have a sensor resolution of 4950x6600, or 32.67 megapixels.

Now take a 35mm photographic negative capture at 3000 DPI. It’s dimensions are roughly 1.4x1 inches. A digitized version would be at least 4200x3000 pixels. That’s well within the range of a 32 megapixel camera. We could create a 4714 DPI image with a 32 megapixel camera if we have the negative fill the entire frame.

The camera will blithely store the DPI for any of these images as 72, 180, 300 or some other default value. If you want to test this out, use a tool like to find the DPI and resolution of a preservation file from a still imaging program. I use exiftool -XResolution -YResolution -ImageSize path/to/file. Divide the resolution by the DPI to see what the size of the original object was according to the file metadata. Compare that to the actual size of the object. In many cases they will disagree by a large amount.

As before, the DPI doesn’t matter, the resolution does.

What Should I Do?

The point of this whole article is to say that DPI/PPI are contextual values. As soon as you change the context, those values lose most of their meaning. But there is an expectation to use these values, so each piece of software writes something that kind of makes sense.

If you are digitizing items and want to meet recommendations for various guidelines, do the math to see if your camera or scanner has enough megapixels to capture at the recommended DPI. Enough is contextual, but 50+ megapixels will typically be sufficient for high-end work.

If you are responsible for delivering files for reuse, change the DPI value in the file header to 300 so your users don’t worry about publisher requirements. My personal recipe is exiftool -Xresolution=300 -Yresolution=300 -overwrite-original path/to/folder/of/jpegs.

If you are receiving images with an embedded DPI value from someone else, ignore it. You have to determine whether the image can be displayed with sufficient quality for your own purposes.

exFAT: A File System for Your Portable Drives

Mon, 14 Feb 2022 14:00:00 +0000

exFAT: A File System for Your Portable Drives

We use a lot of portable drives at the library. They range in size from 500 GB solid state disks the size of a credit card to a 126 TB RAID 5 array with its own rollaway case. All of them use the same file system, exFAT. And if you are also using a lot of portable drives, you should consider exFAT as well.

What is a file system?

A file system is the way that data is structured on a storage device. To better understand it, think about some of the things that happens when you open a file on your desktop named video.mov

The computer figures out where the Desktop folder is in the folder hierarchy.
It checks if there is a file in that folder named video.mov.
It checks to see if you have permission to open that file.
It finds out the sector of the device where the first part of that file is written.

The rules that create the architecture are part of the file system. The File Allocation Table (FAT) file system, first published by Microsoft in 1977, is a good example to use for illustration.

Each folder is defined as its own table with a row for every file or folder that is a member of that folder.
The first 11 bytes of the row define the file or folder name, represented as (8 characters).(3 characters).
The 13th byte of the row specifies whether the file requires a password to read, modify, or delete.
The 20th and 21st byte specify which sector of the device the first chunk of the file is stored in.

As long as your computer system knows those rules and can follow them, it can use a storage device formatted with that file system. If it can’t read those rules, your computer will be able to see the storage device but not the contents of it.

What are other examples of file systems?

A file system developed in 1977 will have the biases of a computing in 1977. For example, a 5.25-inch floppy disk only held about 360 thousand bytes, so the designers didn’t want to consume too much space with the file system. With that perspective 11 characters for the file name and 2 bytes for the sector position was fine. On a device with 5 billion bytes of capacity, 11 characters for a file name is miserly, and 2 bytes isn’t enough for the sector positions. FAT12 was extended to deal with some of these issues, first to FAT16 (1984) and then to FAT32 (1996).

But file systems can do even more than just organize a storage device. They can have features like journaling, data integrity checks, on-the-fly compression, and storing metadata about the files. The implementation of these features could be commercially advantageous and vendors created their own file systems to best interact with their systems.

Apple has used HFS (1986), HFS+ (1998), and APFS (2017). Microsoft created NTFS (1993). In Linux, there is ext2 (1993), ext3 (2001), ext4 (2008), and btrfs (still in development). There are many more if you’re curious. (Thanks to Ross Spencer for this link)

Take note of the relationship between operating systems and file systems. Every OS will best support in-house developed file systems. Supporting others’ file systems is not guaranteed. Windows can’t read HFS+. macOS can read from an NTFS-formatted device, but it can’t write to it. Linux can work with nearly everything, if someone released software to do it and you installed it.

Why does this matter?

For those of us that work with old storage media, identifying a file system and having a computer with the appropriate software to read it is important. That’s just another example of needing to maintain an array of options to access data from the harried years of the 80s and 90s.

For those of us writing data to storage devices, we need to choose file systems that meet the needs of our computing environment. A common need is cross-platform compatibility. Because, if my digitization vendor uses Windows workstations and I use Macs, the hard drive we shuttle data with must be compatible with both.

You can install software to add additional file system compatibility. And I do this when I have open-source options available or budget to buy licenses. But, it’s easier if I don’t have to worry about what software is on what workstation. Isn’t there a cross-platform file system?

exFAT: The Cross-Platform Choice

Microsoft released another file system in 2006, the extensible File Allocation Table (exFAT). exFAT addresses FAT32’s critical flaw in modern computing environments. The FAT table records the size of a file using 4 bytes. That means a storage device using FAT can only store files that are 4,294,967,295 bytes (4.29 GB) large. In digitization work, this is smaller than most video files and even some audio files.

exFAT uses 8 bytes to store the size of a file, which means the maximum file size is 1.84 quintillion bytes (1.84 exabytes). At this point, that’s plenty of space for our entire collection, much less a single file.

exFAT is natively supported in Windows, macOS, and Linux. That means no software installation needed.

It also supports all unicode characters, meaning 视频.mov, ویڈیو.mov, this-file-name-is-really-very-long-,-probably-too-long-to-be-useful_.movand 📹 .mov are all legal.

What’s the catch?

Well, exFAT doesn’t have support for useful features like journaling, data integrity checks, and on-the-fly compression. Of these, the lack of journaling is the most important. File systems with journaling keep track of changes in progress. If you’re in the middle of an action like moving a folder when you unplug it or it loses power, the file system could be corrupted. Your computer warns you about unplugging a drive without ejecting it because of that risk. A journaling file system can fix the file system next time it is connected. exFAT can leave you high and dry. (Thanks to Kieran O’Leary for prompting me to learn what journaling is.)

The cross-platform support also isn’t perfect. macOS only supports exFAT drives formatted with a sector size of 128-1024 kiB. Either format the drive on a Mac or be careful when choosing options during formatting on Windows and Linux.

Linux support only started in 2019 with kernel in 5.4, with better support starting in 5.7 (I’m still amazed that I can plug a 100+ TB RAID into an Android phone). However, if you’re using a Linux distro with a pre-5.4 kernel, you need to install exFAT drivers yourself. Most notably in our community, that includes BitCurator 2.x, which is based on Ubuntu 18.04 LTS with the 5.3 kernel. (Thanks to Kieran for this as well.)

To me, those are acceptable compromises when it comes to portable hard drives used to transfer files.

exFAT is an Industry Standard

If you’re looking for more validation, take a look at SD cards. They are the last major removable media format that needs to be accessible on a variety of computers. The default file system for all cards larger than 32 GB is exFAT.

exFAT has proved to be the most reliable solution in our work. It’s worth evaluating in your context if you’re frustrated with platform cross-compatibility.

Sed It and Forget It: Stream Editing and Metadata Repairs

Wed, 30 Jan 2019 14:00:00 +0000

I made a mistake prepping some metadata files the other day. I picked the wrong three-letter code for a division field for use in a spreadsheet, exported the rows of the spreadsheet to 1200 structured data files we use, and then updated those files with technical metadata. So how to repair that mistake?

Option 1: Go back to my spreadsheet, update the columns, update some dependent columns, re-export, and re-update.

Option 2: Copy my Python script to update the technical metadata and rewrite it to update the fields with the typo.

Option 3: Write a sed command.

For me, the right answer was sed.

sed, the stream editor

Here’s the Wikipedia definition: “a Unix utility that parses and transforms text, using a simple, compact programming language.” In other words, sed edits texts as it reads it, according to whatever short editing recipe you give it. For my purposes, the goal is replacing one three-letter string with another three-letter string. Let’s dig in.

# Replace typo string with correct string
# sed, edits a stream of text
# 's/"doh/"div/g', recipe for the edit, '/' divides the parts of the recipe
## s, substitute - replace one string with another
## "doh, string to look for
## "div, string to replace it with
## g, global - perform substitution everywhere it's found, regardless of case
# /path/to/file

> sed 's/"doh/"div/g' /path/to/file

That command will print out the result to your terminal, which is always a good practice as you’re building a recipe.

That recipe performs global replacements, so defining the strings too broadly might lead to problems. For example, if I just looked for ‘doh’, I could overwrite ‘Play-doh’ and ‘Doha Airport’ as ‘Play-div’ and ‘Diva Airport’. I knew my typos always occured with a preceding ‘”’, so I include that ‘”’ in my recipe.

Even more powerfully, sed can immediately write changes back to the original file if you add a specific flag.

# Replace typo string with correct string and rewrite original file
# sed, edits a stream of text
# -i '', save the edited text back to the input file (for Mac)
# 's/"doh/"div/g', recipe for the edit, '/' divides the parts of the recipe
# /path/to/\*.json, path wildcard to call all json files in a directory 

> sed -i '' 's/"doh/"div/g' /path/to/\*json

Once the recipe is performing as expected, the ‘-i’ flag writes the edits back to the file instead of printing it to the terminal. Using the ‘*’ pushes the automation one step further by opening all of files that match the path pattern. In one fell swoop, that’s 2400 corrections across 1200 files for me.

The tool can do much more, but I only really know how to do text substitution with it. If you want to go really deep there’s a grymoire for it.

Structured Data is Nice for Digitization Projects

A side note about metadata editing tools, the format of a metadata container will often define the options for editing.

Using spreadsheets as the primary container for metadata, would mean using spreadsheet software to perform any edits. There’s a longer post to write about comparing spreadsheets and structured data files, but the basic point is that spreadsheets offer enormously powerful tools, within a single spreadsheet. If you have more spreadsheets, you need to perform those actions in each spreadsheet.

On the other had, structured data formats are readable by a number of bash tools and programming languages. And, those tools often have idioms for applying one change to a large number of files. For example, using more wildcards expands my reach from a single directory to multiple.

# Replace typo string with correct string and rewrite original file
# sed, edits a stream of text
# -i '', save the edited text back to the input file (for Mac)
# 's/"doh/"div/g', recipe for the edit, '/' divides the parts of the recipe
# /path/to/project\*/\*.json, path wildcard to call all json files in directory that start with 'project'

> sed -i '' 's/"doh/"div/g' /path/to/project\*/\*json

And the really nice thing about structured data formats is that they are stored as text, so you can use either tools optimized for text processing like sed and tools that can parse the data structure like xml or json.

Conclusions

If you want to see the possibilities, I’d recommend pulling together some XML files in EAD, PBCore, or whatever flavor you tend to use and try out some manipulations.

MaybeBytes: What the '700MB' on a CD-R means

Tue, 08 Jan 2019 14:00:00 +0000

When you look at an old CD, you might see ‘700MB’ printed in large letters on the label. MB is the SI abbreviation for megabytes or 1,000,000 bytes, but that doesn’t mean a ‘700MB’ CD has a maximum capacity of 700,000,000 bytes. It can store 737,280,000 bytes.

And of course that’s not really right either.

Binary and Decimal 1000’s

Computer scientists and engineers love binary math. A bit has 2 potential values (2¹). A byte has 8 potential values (2³). When counting bits and bytes, why not keep using powers of two? That’s why you see those numbers so often. The N64 had a processor 64-bit processor (2⁶). Extended ASCII encodes 256 characters (2⁸). Conveniently 2¹⁰ is 1024, almost a thousand.

And that’s where the problems start. When we started storing lots and lots of bytes, engineers wanted an abbreviation. The SI prefixes for powers of 10, k (thousand, 10³), M (million, 10⁶), G (billion, 10⁹), etc, were familiar. Some thought they should be adopted as is, but others believed binary was a more natural for digital counting. They used the following: k (2¹⁰), M (2²⁰), G (2³⁰), etc.

So now you have two groups of people using the same word for a different. Errors start to build. You buy a hard drive that can store 250 GB (250x10⁹ bytes), but then your operating sytem says it can only hold 233 GB (233x2³⁰ bytes). And it gets worse as you count higher. A decimal k is 2.3% smaller than a binary k. A decimal G is 6.8% smaller than a binary G.

Obviously there was a problem that had to be fixed. The end result is that we have two prefixes: decimal prefixes stayed the same, and binary 1000’s are supposed to use binary prefixes. kiB instead of kB. MiB instead of MB. GiB instead of GB. And binary prefixes are pronounced by trading the last two letters of the decimal prefix for ‘bi’. So ‘kibi’, ‘mebi’, ‘gibi’, ‘tebi’, etc.

And since language is simple, this change happened immediately (it was incorporated into standards starting in 1998 and finishing in 2008), and no one ever makes a mistake. Definitely, there is not a fascinating Wikipedia article.

Back to CD’s

As you can guess, the ‘M’ in ‘700MB’ refers to a mebibyte, not a megabyte. A CD can store 730MB or 700MiB. And you will find CD’s that have ‘730MB’ printed on them. They store the exact same amount of data as those with ‘700MB’.

Problem 1

At the beginning, I said a 700MiB CD can store 737,280,000 bytes, so why 730MB not 737.28 MB? Marketing. Round numbers sound nicer, and it would be false advertising to round up, so 730MB it is.

Problem 2

CD was created as an audio format. Some of the original specifications are for audio playback, like a requirement for 90 seconds of silence so a machine could tell when the CD was finished playing. Some software and drives can write data to these areas, called overburn. That’s good for another few MB of data capacity.

So what is the max capacity of a CD?

Data on a CD is stored as a continuous spiral of dots and dashes. To guide the laser while burning to a writeable disc, the plastic is molded with a spiral called the pre-groove. If you tighten the curve of the spiral, you can actually fit more data onto the disc, as long as your machine is capable of following the tighter groove.

Look around eBay long enough and you’ll find some ‘800MB’ and ‘900MB’ CD-R’s, although not all CD players can play these. And of course the ‘900MB’ CD actually holds about 870MiB which is really 912MB…

There are probably even more asides to this question, but it’s starting to feel like trying to measure the coastline of the Great Britain, so I’m letting it lie.

ePADD and Archivematica

Fri, 02 Nov 2018 14:00:00 +0000

ePADD and Archivematica: IPs, IPs, and IPs

I’m a very big fan of using OAIS to model workflows. Get some SIPs. Make some AIPs. Deliver some DIPs. It’s all so easy, until you start trying to do it with actual tools. While working with ePADD and Archivematica recently, I ran into that theory->practice barrier difficulty again.

If you haven’t heard of ePADD, it’s an amazing tool built at Stanford to work with email archives. The workflow is basically

Load 1 or more email inboxes into the Appraisal module.
Appraise, redact, accession, describe, process correspondence using Appraisal and Processing modules.
Allow users to search, browse, and export processed emails using the Discovery and Delivery modules.

And making all of that possible for inboxes that may contain millions of messages are some nifty tools like entity extractors, PII identifiers, and attachment browsers.

If I were to OAISify ePADD, each email account loaded into Appraisal is a SIP. Sets of correspondence loaded into Discovery and emails exported through Delivery are DIPs. But where’s the AIP?

The Processing module creates a set of files that could be treated as an AIP, but the ePADD system does not have storage functionality built in to do things like monitor fixity or manage location. It also isn’t possible to create a policy for AIP structure, file formats, or accompanying PDI. And that’s perfectly fine since there are other tools that do that. The challenge is how to integrate ePADD with those tools.

For example, if you use Archivematica to produce an AIP, how would handle a processed ePADD package? And how would ePADD load an Archivematica AIP into its Discovery and Delivery modules? The ePADD team was testing those questions out, and NYPL contributed.

OAIS Time

Since everyone uses OAIS a little differently, I’ll define how I use them here at NYPL.

Content Information - the target of preservation
PDI - Preservation Description Information, information the supports the identification, preservation, and understanding of Content Information.
IP - Information Package, a group that includes Content Information and/or PDI
SIP - Submission Information Package, the objects that we receive as part of an archival acquisition, digitization project, or another process. In this case, each inbox is a SIP.
AIP - Archival Information Package, the material that the Library has accessioned and commits to preserve for long-term access. In this case, the package poduced by Archivematica is an AIP.
DIP - Dissemination Information Package, the material that the Library makes available to its users. In this case, ePADD’s Discovery and Delivery modules create DIPs.
Ingest - The set of processes that produces descriptions and AIPs from SIPs. It can include augmenting, combining, splitting, deleting, or transforming one-or-more SIPs. In this case, ePADD’s Appraisal and Proceesing modules and Archivematica are all part of ingest, particularly the Receiving SIP, Generating AIP, and Generating Description functions.

I think it’s important to define my usage of these terms because they can be at odds with how the tools use those definitions. For example, in Archivematica every submitted package is a transfer, every transfer is transformed into one-or-more SIPs, and every SIP is transformed into an AIP. But I think of the original submission to ePADD as a SIP, so going by my tools’ terms a workflow would look like this: SIP -> Transfer -> SIP -> AIP. That’s confusing and limits how I can conceptually link these tools together.

For me, in this case and most other worfklows, Archivematica is receiving an IP that is somewhere between a SIP and an AIP.

ePADD IP and Archivematica Expectations

Back to the integration work. ePADD takes one or more inboxes and turns them into an IP with the following directory structure.

user
- blobs - directory for attachments
- indexes - directory for indexes created to search email and attachments
- lexicons - directory of terms used to classify emails
- sessions - directory of added mappings like VIAF IDs and annotations

We would like to put that IP into Archivematica, but Archivematica has expectations about how any IP is structured.

directory
- objects - digital objects to be preserved
- metadata - metadata about digital objects to be preserved
  - submissionDocumention - any documents about the acquisition of records
- logs - metadata about processes performed on digital objects

In Archivematica, /objects contains Content Information and /metadata and /logs contains PDI. That’s a key piece of information because Archivematica runs some microservices on everything it thinks is Content Information, and generally maintains PDI as is.

For example, if you rename ePADD’s /user as /objects and send it to Archivematica, all of the indexes, lexicons, and sessions will be treated as Content Information. If you apply file naming or normalizing rules to those files, they would be rendered useless for the ePADD Discovery and Delivery modules.

On the other hand, if we treat those directories as PDI describing the context of the emails and attachments, they should be placed in /metadata and would remain untouched by Archivematica microservices. And then moving the /blobs to /objects allows us to apply preservation policies to any attachments.

To do this, all we need is a script that translates the ePADD IP to the Archivematica IP. That looks like this. Note, we were experimenting with bags, so this script also restructures the bag so that Archivematica can read the checksums and validate the Content Information.

Archivematica AIPs and ePADD DIP Expectations

After translating the directory structure, Archivematica processes the ePADD IP into an AIP, and sends it to the Storage Service. That’s great, but only half of the journey. To go from the Storage Service back to ePADD, we need to translate the Archivematica AIP back into an IP that ePADD can understand. Since the Archivematica AIP is similar to its transfer requirements, doing translating back can be accomplished by doing the same steps in reverse, for the most part.

There are several files and directories created by Archivematica that I left in their original position. That’s fine, since ePADD is unaware of what most of those files are. You might even take the step of deleting those files since an end user may have no interest in them. It’s not a permanent deletion since we’re working with a copy of the AIP.

The most important file is the METS file, which records any changes made to the Content Objects by Archivematica. This may include file name change like replacing spaces with ‘_’ or file format changes like converting a PICT image to a BMP.

So what?

So far, these scripts translate from ePADD IP to Archivematica IP and back, but this isn’t just shuffling a deck of card for the fun of it. Archivematica can transform the Content Information in a number of ways, and all of those changes are logged in the Archivematica METS file.

By selectively applying Archivematica to the attachments stored in /blobs, we can address preservation risks in those attachments. The ePADD team has written code that parses out what preservation actions were taken and can then present both the original and normalized files. If you set a policy in Archivematica to migrate a Wordstar document to PDF, when you load the transformed AIP into ePADD, it will update its indexes to display the migrated version of the object. Very cool!

Challenges Remaining

This work is not wrapped up. It only covers the use case where Archivematica is normalizing formats for preservation. I’d like to add the ability to use file normalized for access, but that requires documenting how the Archivematica AIP stores those files.

It also needs some more documentation. Several options in the Archivematica workflow must be configured in specific ways. For example, Archivematica should not unpack any compressed packages, since ePADD doesn’t index the contents of compressed packages. I’ve included a MCP XML file with the correct configuration, but it would be more useful to include which of those choices are required.

There’s also the matter of adding this to automation tools. That would require identifying some heuristics to identify ePADD bags and applying the right scripts.

If you have any questions about this work, please contact me through email or the Github repository. One of the most important things that this work helped me realize is how fixity information can be passed into and out of Archivematica, which is something that I’ll write about in the future.

What Fixity Means

Mon, 27 Aug 2018 14:00:00 +0000

Fixity and Data Loss in Libraries

Last week, a story went out that the Queen Elizabeth II Library at Memorial University in Newfoundland had lost the primary copy of its digital archives in a server crash. The good news is that library has digital preservation practices in place and is restoring the data from backups. Preservation rescues access again!

This and similar stories offer a view into another core digital preservation topic, fixity.

What is Fixity?

Fixity is the property of objects to remain fixed, stable, unaltered, etc. Ideally, preservation keeps objects stable, but all things change. Oxides flake from a VHS tape. Dyes fade on a photograph. Pests eat the pages of a book. Fixity is a useful concept for every kind of collection; however, the term fixity is mostly used within digital preservation.

In digital preservation, fixity is often presented alongside the concept of checksums, an amazing piece of math that lets us create a fairly unique signature for files based on the bytes they contain. This piece of fixity information can be used to detect very small changes in an object.

But fixity information can also be other things like the size and location of a file. And these pieces of fixity information can be used to detect much bigger changed such as has a file been partially or completely lost. And frankly, in my experience and others, complete loss is more common than the slow trickle of bit flips detected by checksums. If you want a physical example, think of how much damage is caused by theft or natural disaster.

Existence Fixity in Practice

Going back to the data loss incident at the Queen Elizabeth II Library, the server failure was likely detected in a fixity check to ensure that the server migration was completed successfully. They may not have called it a fixity check, but from a digital preservation perspective it was. And it’s a useful check to incorporate into your own workflows.

For example, if you use bags in your workflows, many tools have a completeness check that compares the expected list of files in the manifest with the files present on the file system.

In Bagger, it’s the yellow cube icon.

And in bagit-python, it’s the --completeness-only option.

> bagit.py --validate --completeness-only path/to/bag

The benefit of these tools is that checking for the existence of a file is much, much faster than validating the checksums of a file. Because checksums are calculated based on every bit of a bytestream, validation requires reading every bit. On the other hand, completeness only requires reading file directory tables. You can’t detect bit-level changes, but in the face of a disaster, crime, or another event, you can survey the fixity of an entire collection relatively quickly.

For a more general overview of fixity, I recommend the report What is Fixity, and When Should I be Checking It? published by the NDSA.

Bashing out another file format signature

Wed, 31 Jan 2018 14:00:00 +0000

In the last post, I used some bash tools to derive a file format signature for the ACS format used to store Clippy. During that analysis, 32 of the files had an OLE format signature, 0xD0CF11E0A1B11AE1.

Also known as the Compound File Binary Format, OLE was used as a structure for many different formats from software like 97-2003 Microsoft Office, SPSS, Minitab, and Caseware. It was also used to package the animations and character data used for Microsoft Agents.

On this documentation page, you can see the three formats Microsoft used for agents: ACS, ACF, and AAC. The ACS can store the entire character, but Microsoft also wanted to allow people to build websites and network applications with Agents. ACF and AAC made this easier since an application would only have to download the character data and animations it needed. On the other hand, Clippy.ACS took 6+ minutes to download over a 56k modem.

Examples of ACF and AAC are actually difficult to find on the web. Microsoft used to host them at locations like http://agent.microsoft.com/agent2/chars/genie/genie.acf. As far as I can tell, all of these resources are now down and weren’t collected by a web archive. However, the OLE files from http://www.ponx.org/msagent/Acs/ have some promise.

# Show bytes 1152-1212 from wade.acs
# tail, print the end of the data stream
# -c +1152, limits to starting at byte 1152

# head -c 160, print the first 160 bytes of the tail

# xxd show the hexcode of the bytes

> tail -c +1152 wade.acs | head -c 160 | xxd

0000000: 0063 0068 0061 0072 002e 0061 0063 0066  .c.h.a.r...a.c.f
0000010: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000020: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000030: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000040: 0012 0002 00ff ffff ffff ffff ffff ffff  ................
0000050: ff00 0000 0000 0000 0000 0000 0000 0000  ................
0000060: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000070: 0000 0000 0000 0000 00e4 0300 0000 0000  ................
0000080: 0061 006e 0069 006d 0031 002e 0061 0061  .a.n.i.m.1...a.a
0000090: 0066 0000 0000 0000 0000 0000 0000 0000  .f..............

This OLE file has at least one ACF file and AAF file embedded in it. The trick is extracting these files from inside the OLE file.

Collect all the OLE files from ponx

Honestly, I would probably use python for this, but it was a fun exercise to work out as a shell-scripting ‘one-liner’. There should be 33 files in there.

# Make a directory to store the OLE files
mkdir ole

# Find OLE files based on their first 2 bytes
# for ...;, open a for loop
# file in *.[aA][cC][sS], on each loop process one acs or ACS file
# do ...;, the loop
# bytes=$(head -c 2 $file);, store first two bytes in variable, byte
# test $bytes != 'ë', test if first bytes are not ë, which are true ACS files
# && cp $file ole, if first two bytes are not ë, copy file to ole folder
# done, finish loop
>for file in *.[aA][cC][sS]; do bytes=$(head -c 2 $file); test $bytes != 'ë' && cp $file ole; done

# Check out OLE ACS files
> ls ole

Extract ACF and AAC files from OLE

This requires some tools that can read the OLE format. Since it is the basis of the 97-2003 Office formats, there’s already a very good Java library to manipulate the formats, Apache POI. Even better, Ross Spencer wrote a nice command-line tool with the Java library to extract the files.

# Install python java wrapper jython
sudo apt-get install jython

# Download the Apache POI library
wget http://apache.mirrors.ionfish.org/poi/release/bin/poi-bin-3.17-20170915.tar.gz
tar -xzf poi-bin-3.17-20170915.tar.gz

# Download Ross Spencer's OLE tool
git clone git@github.com:exponential-decay/ole2-re-combiner.git

# Extract all files from the ACS files
# jython, run jython
# -Dpython.path=poi-3.17/poi-3.17.jar, using the POI library
# ole2-re-combiner/ole2recombiner.py, and Ross's Python script
# --extract $file, extract files out of OLE
for file in ole/*; do jython -Dpython.path=poi-3.17/poi-3.17.jar ole2-re-combiner/ole2recombiner.py --extract $file; done

Find signatures for ACF and AAC files

The ole folder should be filled with subfolders now, each with the extracted ACF and AAC files. Follow the same process as the post on the ACS files to analyze them for magic numbers. You can use a small tweak to peak into all the subdirectories with one command.

# Display first 16 bytes of each ACF file
# ole/*/*.acf, all files in subdirectories of ole that have the acf extension

> head -c 16 -q ole/*/*.acf | xxd

I ended up with 0x1F000100 and 0x1E000100 for AAF and 0xC1ABCDAB for ACF, although I’m doing a little more research before submitting these to PRONOM.

Bashing out a file format signature

Tue, 30 Jan 2018 14:00:00 +0000

PRONOM is the digital preservation file format registry of choice, but it’s only as good as the file format signatures that we put into it. There are already great resources about how to do this. Jenny Mitcham from the University of York included links to most resources in her blog post about creating a signature and there is a trove OPF blogs on the subject by Ross Spencer, Andrea Byrne, Becky McGuiness, and others.

This post will be a very fast flyover of building signature for Microsoft’s ACS format, aka Clippy’s format.

Gather some sample data.

The more examples of a format you have, the more confidence you have that you’re finding a good signature. Luckily for ACS, the Microsoft Agent tool had a small community that built more of these demonic helpers. I even found one resource with over 300 examples of agents. So let’s download all of them.

# Make a throwaway directory to keep all these files organized.
> mkdir ~/acs
> cd ~/acs

# Download all the files
# wget, downloading program
# -r, to open all the links on the page
# -nH, to not create a directory for www.ponx.org
# -np, to not create sub-directories for msagent and Acs
# --cut-dirs, to not download anything outside the Acs directory

> wget -r -nH -np --cut-dirs 3 http://www.ponx.org/msagent/Acs/

Look around for magic numbers

A lot of formats use a semi-unique series of bytes at the start of the file to identify themselves. We can see if this is the case by printing out the first 16 bytes of all of the ACS files.

# Display first 16 bytes of each file
# head, prints the start of a data stream
# -c 16, limits head to the first 16 bytes
# -q, makes sure the file name is printed
# *.acs, matches all the acs files in the folder
# It doesn't match

# xxd, prints the hexadecimal interpretation
# defaults to 16 bytes per line

> head -c 16 -q *.acs | xxd
0000000: c3ab cdab 33cd 0300 8707 0000 93c9 0300  ....3...........
0000010: c3ab cdab ebbb 2300 ab0a 0000 c3a2 2300  ......#.......#.
0000020: d0cf 11e0 a1b1 1ae1 0000 0000 0000 0000  ................
0000030: c3ab cdab 5e53 0000 4f06 0000 f251 0000  ....^S..O....Q..
0000040: c3ab cdab 017d 0300 a307 0000 697b 0300  .....}......i{..
0000050: c3ab cdab d1dc 0c00 9106 0000 91d9 0c00  ................
0000060: c3ab cdab e628 0a00 eb09 0000 a219 0a00  .....(..........
0000070: c3ab cdab b224 0100 2d06 0000 7823 0100  .....$..-...x#..
0000080: c3ab cdab cefc 0100 e906 0000 0afc 0100  ................
0000090: c3ab cdab 4ee5 3700 e70a 0000 f2c9 3700  ....N.7.......7.
00000a0: c3ab cdab 437e 1f00 8307 0000 7161 1f00  ....C~......qa..
00000b0: c3ab cdab 437e 1f00 2c07 0000 7161 1f00  ....C~..,...qa..
00000c0: c3ab cdab 8a92 6000 ff0b 0000 7e6c 6000  ......`.....~l`.

Test any patterns

The first four bytes (0xC3ABCDAB) look like a good candidate, so let’s see how many files have that as the first 4 bytes.

# Count up how many files start with different 4-byte strings
# cut, extract specified fields from data
# -d " ", fields are delimited by spaces
# -f 2,3, extract the 2nd and 3rd field from xxd output

# sort, alphabetical sort of the extracted lines

# uniq -c, count how many of each line there

> head -c 16 -q *.acs | xxd | cut -d " " -f 2,3 | sort | uniq -c
284 c3ab cdab
22 d0cf 11e0

You can try cutting more or less fields, but 0xC3ABCDAB looks like the longest common byte string. If you search for 0xD0CF11E0, you’ll find that that is the signature for Microsoft OLE Compound File, a general file structure used as the basis for many formats like .doc Word files and .ppt PowerPoint files. These OLE ACS files probably deserve more research.

Research to backup your hypothesis

It’s always good to back up your work with data from other people. One potential source is the TrID format registry. In this case there is a signature for ACS files, and it matches our signature.

There’s even someone that wrote an unofficial spec for the format. Remy Lebeau’s work also match our signature.

Make and test a PRONOM signature file

Ross Spencer built this excellent tool so that you don’t have to write the XML by hand. Both DROID and Siegfried can be loaded with the custom signature for testing

Submit the signature.

Package up any of the information and resources you have found and submit them to PRONOM. The folks at the National Archives will look over the signature and contact you with any comments.

Data Analysis Tools

Sat, 30 Dec 2017 14:00:00 +0000

This is a quick aside from my series on working with Siegfried.

Data is useless unless without tools to analyze that data. To say anything meaningful, we need to do things like group, split, manipulate, and recombine the individual data points. A lot of larger systems have some analysis functions built in, e.g. dashboards or reports. However, when you’re not working within a system, or if that system can’t do the analysis you need, you’ll need a dedicated tool.

The Spreadsheet Options (Excel, Sheets, etc)

The Good: It’s installed, you know how to use it, you can see the data, it has a decent user interface

The Bad: It slows down with too much data, it transforms data without telling you, it lets you fiddle with data

Spreadsheets can take your analysis pretty far pretty quickly thanks to their graphical user interface, and I often use them for fast analysis. Spreadsheets stop being useful when the interface gets in the way of the analysis. A couple of examples:

You have 100 spreadsheets that you want collapsed into a single dataset. Open-copy-paste can get it done, but will take a lot of double-checking to make sure nothing was missed.
Your barcodes are 14-digits long. Your spreadsheet program converts this to a floating-point number (3.43… * 10^13), since 32-bit integers max out at 10 digits.
You need to add address information to 1000 rows of your spreadsheet. Writing the zip code in the first row and dragging down moves the address to Wyoming, one zip code at-a-time.

All of these problems have workarounds, including bruteforce, cell-by-cell manipulation of data. As you do larger amounts of analysis more routinely, that potential solution becomes a problem. What’s to guarantee that someone hasn’t fixed up a “few” cells by hand?

Data scientists advocate for reproducibility, the ability for someone else to get the same results using the same data. As if often the case, the most likely someone else is a future you trying to rerun the analysis in the future. The less manual fiddling that’s needed for analysis, the more reproducible it can be. That includes messing aroud with Open file... menus, as well as cell-by-cell fiddling.

The Scripting Options

To make analysis more reproducible, the first step is to start recording every step of analysis that you do.

Step 1: Load the data
Step 2: Clean/manipulate the data
Step 3: Run the analysis
Step 4: Produce a report/dataset/etc

This looks a bit like a computer program, but for it to be a program you need a language to express it in and software to write it in. The following are my two favorite ways of doing that.

pandas + Jupyter Notebooks

The Good: You may already know Python, you can use the rest of Python alongside pandas, you can process Big Data, you can’t fiddle with data, you can draft code like writing in a notebook

The Bad: The graphing libraries are okay, there can be an overwhelming number of right ways to do something, indexes can become very complicated, notebooks can be very drafty

Pandas and R both use a concept called a dataframe, a 2 (or more) dimensional structure that stores data in columns and rows that are named by indexes. Basically, it’s a spreadsheet. By abstracting away the graphical representation, you don’t have to think about how answers to your questions fit on the sheet.

import pandas as pd

df = pd.read_csv('path/to/siegfried.csv')

# How big are the files in the Siegfried report?
print(df.filesize.sum())

# What 10 file formats have the largest average file size?
print(df.groupby('format').filesize.mean().sort_values(ascending = False).head(10))

# What are the 5 most common file formats with a modified data in the 1980s?
print(df[df['modified']<'1990'].groupby('format').size().sort_values(ascending = False).head(5))

You can write pandas code in program you prefer, but the most common is Jupyter notebook. The examples above can be saved as Python files and run as scripts (e.g. python3 analysis.py), but often analysis requires exploring the data. In the last example, you might want to explore a specific year df['modified']=='1990' or bound your filter (df['modified']<'1990') & df['modified']>'1980' while you explore. Jupyter notebooks let you edit and rerun lines of code quickly without have to rerun everything from scratch.

Another great benefit of pandas is that you can pull use the rest of the Python ecosystem. For example, if you want to combine a large set of CSVs into a single file, you can use the glob module to find the filenames, and then read/combine/export the data with pandas.

import pandas as pd
import glob

# Find all the CSV's in a folder
all_files = glob.glob('path/to/folder/of/*.csv'))

# Read them into pandas and glue them together
df_from_each_file = (pd.read_csv(f) for f in all_files)
all_df = pd.concat(df_from_each_file, ignore_index=True)

# Save to a new file
all_df.to_csv('one_big.csv')

R + RStudio

The Good: The tidyverse grammar is incredibly expressive, ggplot is an amazing visualization library, you develop analyses towards a finished project.

The Bad: You probably need to learn R from scratch, R can be slow

R does not come with the full power of a general scripting language like Python. However, it’s data analysis functions can often feel much more fluid. Here’s the most common file formats from the 1980s question using the piping feature from tidyverse.

library(tidyverse)

df = read_csv('path/to/siegfried.csv')

# What are the 5 most common file formats with a modified data in the 1980s?
df %>%
  filter(modified < '1990-01-01') %>%
  groupby(format) %>%
  count() %>%
  arrange(n) %>%
  head(5)

Personally, that is much simpler to read than the pandas version. You can even pipe results into what many consider R’s best feature, the ggplot2 graphing library.

df %>%
  mutate(year = year(modified)) %>%
  groupby(format, year) %>%
  count() %>% 
  ggplot() +
  geom_line(x = year, y = n, color = format)

And now you have a line graph of files by the year of their last modified date and grouped by their file format. Pretty powerful stuff.

Conclusion

The best tool for any task is the one you can best use. If you don’t have experience with pandas or R, there are plenty of free online course, Q&A communities, and colleagues in the field that can help you learn. On this blog, I’ll be presenting most of my data analysis work in R or pandas. I hope you can find these posts useful in learning or improving your own data toolset.

Troubleshooting a Siegfried Run

Tue, 26 Dec 2017 17:00:00 +0000

In the last post, I talked about getting Siegfried installed on a Raspberry Pi and kicking off a process to identify file formats. Easy peasy, except sometimes things don’t work out perfectly. Here are a few tips on handling problems with a run.

Connecting to a Raspberry Pi from another computer

I have a monitor and keyboard for my Pi, but sometimes it’s easier to use it through my laptop instead of turning on the monitor. For this, you’ll need to turn on SSH (Secure SHell) on the Raspberry Pi.

In order to login to the Pi from another computer, you’ll need to know the Pi’s IP address.

# Show all network connections on the Pi, run this on the Pi
# wlan0 shows the wireless connection, eth0 shows the ethernet connection
# The IP address for the connection is on the line that starts `inet address:`
# The IP address probably looks like 192.###.###.### or 10.###.###.###
sudo ifconfig

Then from the other computer you can login to a command line session on the Pi.

# Connect to pi from another computer
# The default username is `pi`
ssh username@pi_ip_address

Running Siegfried over SSH

Any commands run during an SSH session will typically end when the user ends the session. To keep a long-running process going, use the nohup (NO HangUP) command.

# Make sure a Siegfried doesn't stop when you end the SSH session
nohup sf -csv -f sffiles.txt > sf.csv &

Put it in the background

The command from the end of the last post will automatically run Siegfried in the background thanks to the ‘&’ at the end of the command. If you didn’t add the ‘&’, you can still put the process in the background like this.

Press ctrl + z to suspend the process
Enter the command bg to put the process in the background

You should be able to enter new commands agains.

Restarting a run

If the power turns off, you’re working over a flakey network connection, or for any number of reasons, your Siegfried run might get interrupted. Rather than start from scratch, you can pick up from where you left off.

Update 2018-01-31 Richard Lehane helpfully pointed out a feature in Siegfried to make this simpler. Instead of concatenating the files with >>, you can have Siegfried combine multiple result files.

# Find the name of the last file characterized 
tail -1 sf.csv
# Get the line number of that filename from your list of files
# to figure out how many files you characterized
grep -n 'name_of_the_file' sffiles.txt
# Make a new file list by skipping characterized files
# +, number of lines to skip, use the line number from last command
# if that file is problematic, add one until Siegfried runs again
tail +number sffiles.txt > sffiles2.txt
# Restart Siegfried, adding resuls to your previous output file
# >>, add new output to the end of file instead of overwriting
nohup sf -csv -f sffiles2.txt > sf_2.csv &

Repeat as necessary.

Clean up the output

In case you had to restart Siegfried, you’ll have a couple result files. It’s easier to handle this as one dataset.

Update 2018-01-31 Richard Lehane pointed out a feature in Siegfried to make this simpler. Instead of concatenating the files and removing extra header lines, Siegfried can combine the files cleanly.

# Combine the Siegfried outputs
sf -replay sf.csv sf_2.csv > sf_all.csv

Here’s the strategy I used to use. It’s messier, and it’s probably more error prone.

# Combine multiple result files
cat sf.csv sf_2.csv > sf_all.csv

# Delete any lines that include the header fields, after the first line
# 2, skip the first line
# ${}, edit to execute
# /.../d, delete lines that match the pattern
sed '2,${/filename,filesize,modified,errors,namespace,id,format,version,mime,basis,warning/d}' sf_all.csv > sf_clean.csv

Copying data over the network

If you can use SSH, you can also copy files over SSH using scp (Secure CoPy). That means less fooling around with USB sticks to get data to where you need it.

# Copy Siegfried output from the Pi
# username@pi_ip_address, login details for Pi
# :..., path to file to copy from Pi
# path/to/..., local paths don't need the login details and colon
scp username@pi_ip_address:sf.csv path/to/copy/file/to/

# If you need to copy from your computer to the Pi
scp path/to/a/file.ext username@pi_ip_address:path/to/copy/file/to/

I’ll add more tips as I cause more problems for myself.