Forensic Files, Episode 1: "Fishing for the Truth"
Sara Allain
Ashley Blewer
Code4lib San José
2019-02-19
Introduction
Introduction
Computers!
A refresher
“Ones and Zeros”
Everything digital boils down to a binary system where everything is translated down into 0 or 1, or "no" and "yes" or "off" and "on" or “true/false.”
It's a language somewhat similar to how Morse Code is composed completely of dots and dashes.
Bits and bytes
bit = BInary digiT
byte = 8 bits
A lot of things get sorted out at the byte-level, which is why it is referred to so often.
Endian-ness
When we store a collection of bytes (like a file) in a computer system or transmit a collection of bytes over a network, the bytes are stored or sent in a particular order.
Big endian = bytes are ordered from the most significant bit.
Little endian = bytes are ordered from the least significant bit.
Endian-ness
Big endian = the byte that contains the most significant bit is sent or transmitted first.
Little endian = the byte that contains the least significant bit is sent or transmitted first.�
12345678
Most significant bit
Least significant bit
Encodings
Character encodings help you talk to your computer, or enable your programs to talk to your computer. They are like human-computer translation devices.
ASCII, Hexadecimal, and Unicode are all examples of encodings.
We will talk about Hex more later!
Don’t underestimate the power of encodings!
Don’t underestimate the power of encodings!
A and A and A
“Every platonic letter in every alphabet is assigned a magic number by the Unicode consortium which is written like this: U+0639. This magic number is called a code point. The U+ means “Unicode” and the numbers are hexadecimal. U+0639 is the Arabic letter Ain. The English letter A would be U+0041.”
https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/
Files!
What are they?
Files
A file is an array of bytes that are encoded in a particular way.
Files are things that can be:
File formats
We have many different types of files! These are categorized into different formats.
The format of your file tells your computer what to do with the data inside the file.
File formats living in binary
Magic numbers
What separates a file from the rest of the binary landscape?
Magic numbers!
Magic numbers
Magic numbers are sometimes known as file signatures. They are a short string of hexadecimal code at the beginning and/or end of a file’s byte array that allow a file to identify itself.
Magic numbers
The magic number for the audio format FLAC is 66 4C 61 43 in hexadecimal, which becomes fLaC when translated to text.
Whenever a program sees 66 4C 61 43 at the beginning of the file, it knows that the file is a FLAC.
File formats living in binary
66 4C 61 43
File structure
The data within a file (the byte array) is organized in a particular order:
File format specifications
A specification is a document that defines how a file must be structured to be considered a particular format. It outlines all the rules required to have a valid file - for example, “A FLAC bitstream consists of the "fLaC" marker at the beginning of the stream.” [1]
File format specifications
File format specs can be standardized through a standards body (but don’t have to be). They can be open (anyone can read them) or closed (proprietary, only people in a company/organization/who pay can access them).
Example specification
More complex media
Registries!
People love lists
Registries
PRONOM (UK National Archives)
Format Descriptions (US Library of Congress)
`file` (Unix command)
File Formats Wiki (ArchiveTeam)
Registries
Registries
More about format registries/signatures
My first file format signature - Jenny Mitcham, DPC
Five Star File Format Signature Development - Ross Spencer, Artefactual
Break time!
Hex editor: What am I looking at?
Reading hexadecimal
Humans count in base-10, which means we get to nine and then we go to one-plus-zero, or ten, followed by one-plus-one (11), one-plus-two (12), etc.
Computers count in base-2. Binary is just 0 and 1.
Hexadecimal is base-16, which means it gets to 16 and then starts over. It's confusing to think about. But they are represented in our common human-language as 0-9 and A-F.
One Hexadecimal can represent four bits.
Reading hexadecimal
Hex Fiend (macOS)
HxD (Windows)
https://mh-nexus.de/en/hxd/
Hex editor: What am I looking at?
Try these formats:
File analysis tools!
Awww yeah, now we’re getting to the good stuff.
File analysis tools
Tools that read and analyze digital objects for you!
FIDO
FIDO (Format Identification for Digital Objects) identifies file formats based on their file signature, which it obtains from the PRONOM file signature registry.
Siegfried
Like FIDO, Siegfried is a signature-based file format identification tool, but it’s more robust. It implements:
JHOVE
JHOVE is a tool for format identification, validation, and characterization of digital objects.
Format validation conformance is determined at three levels:
JHOVE Format validation
Exiftool
Exiftool is used for reading, writing and editing metadata.
It’s especially useful for image files, but also works with audio, video, and PDF files.
(has no logo)
MediaInfo & family
MediaInfo extracts metadata, primarily but not exclusively from audiovisual files. It is a characterization tool.
For a video file, this would include information like the format, codec ID, aspect, frame rate, bit rate, color space, etc.
MediaInfo & family
MediaConch is a file format conformance checker (validation tool) for working with FFV1, MKV, and LPCM formats.
It will tell you if your file is well-formed, valid, and consistent with the specification.
MediaInfo & family
MediaConch is also a policy checker, validating a file against an institution-created policy.
You can create a policy that says all videos must have a frame rate of 24 fps and use MediaConch to confirm that all the videos in your collection conform to this policy.
MediaInfo & family
MediaTrace extracts file information at the binary level. It looks at the chunks of data that make up audiovisual files to tell you about the internal structure of a chunk.
(has no logo)
Let’s play!
We will do some demonstrations and answer questions; if you have a laptop, play along or on your own!
Thank you!
Cat pictures: S. Allain and A. Blewer
GIFs: gifcities.org
Forensic Files, Episode 2: "A Purr-fect Match"
Sara Allain
Ashley Blewer
Code4lib San José
2019-02-19
Introduction
Introduction
Digital preservation lifecycle
How do we end up with all this stuff and what do we do with it?
Digital preservation life cycle
Information gathering
Accessioning/Curation
Digitization
Characterization
Metadata extraction
Normalization
Preservation of digital objects
Storage
Storage formats
Accessing stored material
Access to digital objects
Derivative generation
Presentation
Discussion
Break time!
Tools
Preservation planning using a format policy registry
Designing digital preservation workflows
Level 1: Bit-level digital preservation
How much control do we have over our digital objects?
Level 2: Metadata capture
How much can we find out about a given file?
Level 3: Preservation planning
What actions can we take to increase our chances that a file will persist over time?
Level 4: Accessibility
How do we ensure that we have ongoing access to our data?
See also
NDSA Levels of Digital Preservation
Digital Preservation Handbook (Digital Preservation Coalition)
...and lots of other resources!
“Just right” levels of preservation
What is the right fit for you org?
Your institutional mandate / legal requirements are your foundation - build from there.
What do you need to do?�What do you want to do?�What would you like to do in an ideal world?
Discussion
Thank you!
Cat pictures: S. Allain and A. Blewer
GIFs: gifcities.org
Citations