The Process
How we safe thousands of sata sets and make them publically available
Preface
Organization, inspiration, determination, these are all important to success. So too is process. It’s critical that the work that people devote to data rescue results in data that is trustworthy, readily available, and impactful to people’s lives and livelihoods.
For the organizations that own data records at risk, data rescue efforts are typically a special initiative and not part of their regular data collection process. This is where IEDRO’s experience can be especially valuable, building on industry best practices and its own lessons from helping similar organizations down the same path. They can be that trusted partner, from navigating funding options, setting reasonable goals, and training staff through ensuring data quality, accessibility and analysis.
Become a Part of the Solution
To save the climate data so crucial to all of us, we need you. Become part of IEDRO and help making a difference!
Resources and Standards
Rescued data must be of known quality and reliability if it is to be used effectively. That includes the rescue process itself. To make sure that happens, IEDRO follows best practice guidelines from the World Meteorological Organization for data rescue including those for hydrologic data. The European Union’s Copernicus Climate Change Service also offers its own best practice guidelines for climate data rescue. For example, these standards require that data records are backed up by images of the original paper forms or charts so that digital data can be validated against their source.
SIX Steps to Save the World
How do we rescue data?
1# Planning the Project
IEDRO volunteers meet virtually or in person with the “owners” of the data to be rescued. All parties sign an agreement to allow the rescued and digitized environment data to be freely available through an open and unrestricted database. A tentative implementation schedule is set.
2# Setting up the Site
The IEDRO volunteers assist the country’s national meteorological service or other owners of environmental data to prepare the data rescue facility, purchase and install computer and scanning or camera equipment, and train the data rescue personnel.
3# Inventorying the records
Records at risk are inventoried to identify medium (paper, microfilm, strip charts), types of data, time periods, and measurement locations or stations with an aim to identifying priorities for rescue and setting a scope for the project. Candidate records are compared with archived data to identify which remain unavailable and thus need to be rescued.
4# Imaging the Data
Imaging methods used depend on the medium. Sometimes scanning equipment can be used but often photographing the records is the best approach. In that case, each sheet of observations is oriented on a stand so that the entire page of data is visible to the camera. When the image is in focus, the photograph is taken, with precautions to flatten the sheet. When the camera’s memory is full, the images are downloaded into the computer and saved to portable storage media.
5# Digitizing the Data
After the weather or other environmental data is captured, the images are digitized. Images of line graphs such as strip charts are digitized with a special program designed for graphical images. Alphanumeric data is manually keyed in using either the Weather Wizards web application or a spreadsheet-based process developed by Old Weather. In either case, double entry of data helps ensure the reliability of the data, in concert with other quality assurance steps. Individuals performing the data entry may be “citizen scientist” volunteers, student employees recruited for this project, or staff hired by the data owners themselves.
6# Making the Data accessible to everyone
IEDRO receives the photographic/scanned images of the original environmental data (i.e. alphanumeric weather observations, graphs, photographs, etc.) on a USB drive or via the cloud. We then log the images received and send a complete copy along with the digitized data to the nearest environmental World Data Center – NOAA’s National Centers for Environmental Information (NCEI) in Asheville, North Carolina.