open-workflows

List of open workflows and resources for A/V archiving. Please contribute!

If you have never contributed to GitHub before, here is a guide! You can also file an issue (which is like filling out a form) and someone else will add it.

Sibling repositories: analog-inspection && time-based-media-art

Contents

• Documentation
• Transfer Stations and Digitization Labs
• Resources
• Scripts
• Tools
• Publications
• Administrative metadata

Documentation

• Andy Warhol Museum Digital Strategy: A versioned digital strategy repository for The Andy Warhol Museum.
• archivematica-case-studies: Collection of resources, papers, blog posts, and other documentation around working on and with Archivematica.
• Capturing and Archiving MiniDV Tapes on macOS: Blog post by Leo Bernard on archiving MiniDV.
• Carnegie Hall Web Archiving: "The Carnegie Hall Archives is researching and working on web archiving: identifying possible sites to capture as an expansion to the Carnegie Hall (CH) Digital Collections, establishing description practices, and reviewing access options."
• CCA Digital Archives Processing Manual: The Canadian Centre for Architecture Digital Archives Processing Manual "covers each step of the processing process, including capture of data, arrangement, description, ingest into our Archivematica-based digital repository, and access. It includes general policies and procedures as well as specific information about tools used in this process."
• Community Owned Workflows (COW)
• Converting FFV1/MKV to v210/MOV
• CUNY TV DAT Transfer Workflow: Workflow for transferring DAT tapes using DDS drives and editing created audio files using metadata error logs.
• Digital Processing, Bentley Historical Library: Workflow for ingesting digital objects at the Bentley Historical Library.
• DPX preservation workflow with RAWcooked and fixity checking
• DPX: What Not To Do: Workflow walkthrough by Hirshhorn time-based media conservator Eddy Colloton.
• Educopia Institute - OSSArcFlow: Published research and "as is" workflows for preservation from multiple institutions.
• Emory University: Building A Home Digitization Station: Written by AV Conservator Nina Rao.
• Experiences of an LTO/LTFS beginner: Elaborate blog post by Joanna White on her "LTO journey."
• Format Migrations at Harvard
• Homewood Photography NDSR Final Report (PDF)
• Indigitization: A 'toolkit' of workflows focused on supporting First Nations communities' digitization of cultural resources. Contains detailed walkthroughs for cassette digitization as well as metadata best practices.
• Invisible Defaults and Perceived Limitations: Processing the Juan Gelman Files
• Library Workflow Exchange: A site designed to help librarians share workflows and best practices across institutions.
• Living Content: Digitizing Magnetic Media at NLM: Overview of film/video digitization initiatives from the (U.S.) National Library of Medicine.
• Louisiana Public Broadcasting Digital Preservation Plan: Document summarising LPB's digital preservation practices, policies, and responsibilities. (PDF)
• National Library of the Netherlands: Roll the tape – recovering ’90s data tapes in BitCurator
• NYPL ami-specifications: A repository for NYPL's digital asset technical specifications (audio, moving image, and more).
• NYPL media-ingest workflow: Workflow for NYPL media ingest process.
• NYPL Spotlight on Audio and Moving Image Quality Control: Workflow blog post written by Genevieve Havemeyer-King, Media Preservation Coordinator, Audio and Moving Image (AMI) Preservation Unit.
• An Optical Media Preservation Strategy for New York University’s Fales Library & Special Collections: This report presents an overview of the diverse and unique difficulties posed by optical media, with recommendations and proposed workflows based on local specifications.
• PBCore Audiovisual Tricks: Scripts for PBCore-based preservation workflows.
• Personal Digital Archiving Day Kit: Robust documentation around organizing and hosting a Personal Archiving Day, including A/V resources.
• Protecting a Story’s Future with History and Science: Archival formats and color concerns from the Netflix Technology Blog.
• Recovering '90s Data Tapes - Experiences From the KB Web Archaeology project: An iPres 2019 paper in blog post form by Johan van der Knijff.
• UCLA Metadata Quality Control Workflow: (PDF)
• UCLA International Digital Ephemera Project (IDEP) Toolkit: A super slick, multi-lingual guide to various audiovisual digitization workflows. Particular strengths include: DV, lacquer discs, and 1/4" open reel.
• University of Michigan ICPSR: "This guide describes digital preservation practices at ICPSR, the primary objective of which is to ensure long-term access to the more than 500,000 files in our collection."
• University of Washington Media Center: Step by step documentation of open-reel audio digitization workflow in place at the University of Washington Libraries Media Center.
• VideoPreservation Website: Site to encourage the preservation of historic video using the mature technology of digital capture.
• Washington State University Center for Digital Scholarship and Curation (WSU-CDSC): Documentation for digital workflows, with helpful entries on DV transfer and NDSA Levels of Digital Preservation Surveys.
• WGBH PBS NewsHour Digitization Project Update: Ingest and Digital Preservation Workflows: Blog post detailing digitization, ingest, and QC workflow from WGBH.
• York University Preservation Documentation: Version controlled draft versions of York University Libraries' digital preservation policy and documentation.

Transfer stations and digitization labs

• Brooklyn Public Library Info Commons
• DC Public Library Memory Lab: How to transfer formats (specific to their setup but applicable elsewhere).
• Indiana University Scholar Commons Digitization Lab
• Kalamazoo Public Library The Hub
• Moving Image Preservation of Puget Sound (MiPoPs)
• On Personal Archiving Labs: Blog post by Sam Abrams on setting up a personal archiving lab for Madison Public Library.
• UW Media Arcade: A digital A/V makerspace that supports media creation, preservation and use.
• University of Wisconsin-Madison RADD How-to guides to digitization.
• XFR Collective

Resources

• analog-inspection: List of analog media inspection templates/forms.
• awesome-tdr: "Awesome list" for organizations with Trusted Digital Repository status and their audit reports.
• audio-digitization-toolkit: A list of resources for setting up an audio digitization workflow.
• avpres-training: Educational slidedecks on various audiovisual preservation topics by Ashley Blewer.
• A/V Artifact Atlas: For use in the identification and definition of the technical issues and anomalies that can afflict audio and video signals.
• Building Audio, Video, and Data-Rescue Kits: Guide for building portable kits for audio and video (A/V) and digital data rescue. From Dorothea Salo/iSchool at University of Wisconsin-Madison. (PDF)
• cable-bible: A guide to cables and connectors used for audiovisual tech.
• Cassette Decks: Buying Guide
• Cleaning Digibetas
• Community Owned digital Preservation Tool Registry (z): COPTR describes tools useful for long term digital preservation and acts primarily as a finding and evaluation tool to help practitioners find the tools they need to preserve digital data.
• DMIA: Digital Moving Image Archives: A guide designed for independent filmmakers to provide basic information about preserving digital data and to encourage collaboration with moving image archives.
• Digital POWRR Tool Grid 2.0: An ubergrid of tools useful in digital preservation, and their role in the preservation lifecycle.
• Know Your Carrier: An easy to use website that enables A/V format identification through a progression of prompts and images. It also includes information about preservation and digitization with a public facing writing style. Created by VIAA, it is available in both English and Dutch.
• Libraries Sharing Code: Wiki list of libraries sharing their code on GitHub or on other open repositories.
• Living Will for Home Movies: GoogleDocs version and Home Movie Day blog post with links to additional editable/fillable formats. This is "a series of editable documents that allow home movie owners/creators document their collections and provide provenance, ownership, and stewardship information." From Snowden Becker (2017)
• Matters in Media Art Assessment Workstation
• Mediapedia: Physical Format Carrier Resource: The Mediapedia resource is intended to enable the identification of various physical media carrier types for assisting with collection planning, assessment, documentation, infrastructure and preservation planning for the content they hold.
• media-id-posters: Visualized summaries of physical objects for A/V archivists.
• Nederlands Instituut voor Beeld en Geluid Winter School 2017: Talks featuring explanations of workflows from Tate, WITNESS, and other organizations from the Sound & Vision Winter School for Audiovisual Archiving program.
• No Time To Wait! Symposium #1: Documentation around conference including slides and videos on talks focused around Matroska and FFV1 workflows.
• No Time To Wait! Symposium #2: Conference website.
• No Time To Wait! Symposium #3: Conference website.
• No Time To Wait! Symposium #4: Conference website.
• The Patch Bay Blog: All-around awesome resource from Ethan Gates of Cable Bible fame. Particularly useful entries on BagIt and upgrading Mac video digitization stations.
• QCTools Cheat Sheet
• Sustainable Heritage Network: Collection of workflows, instructional resources and guides with a particular focus on aiding indigenous communities with preserving their cultural heritage. Created by the Center for Digital Scholarship and Curation at Washington State University.
• UW Audio Preservation and Restoration Guide: Large aggregation of Audio resources with descriptions.

Scripts

• Carnegie Hall: Post-digitization quality control workflow documentation and scripts; scripts and overview documentation for generating linked data from event records; and scripts and documentation detailing workflow for batch linking metadata records from disparate CSVs to files.
• CUNY mediamicroservices: Suite of bash scripts for A/V archiving.
• FFmpeg Cookbook for Archivists
• ffmprovisr: Repository of useful FFmpeg command lines for archivists!
• IFIscripts: Scripts for video/DCP/image sequences/fixity for use in the IFI Irish Film Archive.
• MACEscripts: Scripts for video transcoding and other tasks by MACE, the Media Archive for Central England.
• National Library of New Zealand (NLNZ): Scripts for generating METS records, Rosetta-compliant SIPs, etc.
• New York Times video microservices: Documentation and links to services scripts.
• NYPL ami-tools: Python3 scripts and classes to help with managing bags of NYPL AMI files.
• Puget Sound and Vision Scripts: Scripts/documentation used at University of Washington and MIPoPS for automated generation of AIPs containing derivatives and pertinent metadata (preservation and technical). Also contains scripts supporting LTO storage workflows.
• Rockefeller Archive Center Transfer Workflow: Digital media inventory and transfer workflow documentation.
• ucsb-src-microservices: Post-processing scripts we use at UCSB Special Research Collections AVLab.
• U.S. National Archives
• WGBH preservation-workflows: Useful scripts to help automate common WGBH Media Library and Archive digital preservation workflows into a single preserve.rb Ruby script.
• Yale University Libraries: Manuscripts and Archives GitHub.

Tools

• AEO-Light 2: Open source optical sound extraction software.
• Audacity: Free open source digital audio workstation. Supports recording and editing of audio.
• Audiogrep: Transcribes audio files and then creates "audio supercuts" based on search phrases. It uses CMU Pocketsphinx for speech-to-text and pydub to stitch things together.
• Aurora: Aurora is a Django web application that can receive, virus check and validate transfers of digital archival records, and allows archivists to appraise and accession those records.
• BWF MetaEdit: Allows the embedding of BWF metadata in WAV files.
• ClairMeta: a python package for Digital Cinema Package (DCP) probing and checking.
• Digital Media Log: A Rails app that uses ArchivesSpace's API to quickly inventory digital media items and log disk imaging.
• DCP-o-matic: A free, open-source program to create digital cinema packages (DCPs) from videos, images, sound and subtitle files.
• ExifTool: A Perl library plus a command-line application for reading, writing and editing meta information in a wide variety of files, with a focus on still image file formats.
• forced-alignment-tools: A collection of links and notes on forced alignment tools.
• iromlab: Loader software for automated imaging of optical media with Nimbie disc robot.
• TheKeep: A repository-based Django web application for managing digital surrogates or "masters" of archival (and other) source materials.
• MediaConch: Tool for quickly creating and comparing policies for digital media to ensure compliance with chosen standards.
• MediaInfo: Essential tool for quick analysis of A/V file metadata.
• QCTools: Enables extensive analysis of digitized video for quality control purposes.
• reel time: Tool to calculate the duration of open-reel audio.
• salvagedv.py: Scans an input (such as a broken file) for apparent DV frames and extracts them into a new DV file.
• untrunc: Tool to attempt repair of truncated MP4/MOV files. Works by analyzing a similar known good file (such as from the same camera) and attempting to reconstruct metadata in broken file.
• Videogrep: Searches through dialog in video files (using .srt subtitle tracks or pocketsphinx transcriptions) and makes supercuts based on what it finds.
• vrecord: Open source tool for analog video digitization. Includes a variety of virtual scopes. Requires a computer running a recent macOS or Linux and Blackmagic hardware.
• XLD: Tool for lossless transcoding of various audio formats. Useful for error free CD ripping.

Publications

• A Guide to Approaching Audiovisual Digitization for Artists and Arts and Culture Organizations by Morgan Oscar Morel, Bay Area Video Coalition (BAVC). (PDF)
• ARSC Guide: "The ARSC Guide to Audio Preservation is a practical introduction to caring for and preserving audio collections. It is aimed at individuals and institutions that have recorded sound collections but lack the expertise in one or more areas to preserve them."
• Describing Archives: A Content Standard (DACS): Guide to DACS, a commonly used set of rules for the description of archival materials.
• DV-based video cassettes: Draft technical specifications for the transfer to files (PDF): Documentation on transfer of DV video on analog carriers by VIAA and AV-RD
• IASA Publications: Comprehensive works by the International Association of Sound and Audiovisual Archives (IASA). TC-03 Safeguarding the Audio Heritage: Ethics, Principles and Preservation Strategy, TC-04 Guidelines on the Production and Preservation of Digital Audio Objects, TC-05 Handling and Storage of Audio and Video Carriers.
• Memoriav Recommendations: Digital Archiving of Film and Video: Principles and Guidance by Agathe Jarczyk, Reto Kromer, Yves Niederhäuser and David Pfluger.
• Sound Directions: An overview of audio preservation and digitization workflows developed at Harvard and Indiana University. Includes examples of BWF metadata profiles. (PDF)
• The Video Guide: Good primary source for analog video and related technology. Originally published in 1981, permission was granted by author for free access/distribution.
• UC Guidelines for Born-Digital Archival Description: A University of California-wide descriptive standard for born-digital archival material.
• Pragmatic Audiovisual Preservation, October 2020 Ashley Blewer “This report provides a foundation in the basics of audiovisual preservation and delivers practical steps on taking forward the preservation of these materials in typical organisational settings,” explains Paul Wheatley, Head of Research and Practice for the DPC. “I am sure that organizations of all sizes will find it useful, as it sets out guidelines to moving image and sound preservation and features case studies from a wide range of institutions.”
• Preserving Moving Images and Sound, 2nd Ed, March 2020 "This report provides guidance for anyone with responsibility for sound or moving image content and an interest in its preservation. The report covers digitization, encoding, file formats and wrappers, use of compression, obsolescence and the particular digital preservation problems of sound and moving images."

Administrative metadata

Maintainers

For help contributing to this repository or other issues, please contact Ashley Blewer (@ablwr) or Tod Robbins (@todrobbins).

Code of Conduct

Actions taken in this repository, associated pull requests and issue tracker forms should adhere to the Contributor Covenant Code of Conduct guidelines.

Copyright

To the extent possible under law, https://github.com/amiaopensource/open-workflows has waived all copyright and related or neighboring rights to open-workflows.