This week’s post is about the digital humanities work—using computers to assist with the research process—I am doing here in Hungary. Since that can be rather technical, I thought I would provide the direct connection up front and then for those interested a description of the technical part follows. Thanks for reading! -PCL

Summary:

A major part of what I am doing in Hungary is digital humanities work on the Magyar Exodus 1956-1957 project. The project aims to gather resources about the 200,000 Hungarians who fled their homeland following the 1956 Uprising and then make those resources available to researchers, genealogists, and the general public. Currently we are working to transform a set of 10,000 refugee index cards into the core of database. That is where you could participate in this project if interested.

We have constructed a web portal where humans can review the computer transcribed documents. The great thing is that no special skills are needed, so anyone can participate. If you go to this page you can read the instruction document and/or watch the training video. There is also a registration form, or you can contact me to get access. If you are interested, we would appreciate your assistance!

Technical:

We have a collection of over 10,000 images of index cards capturing the basic information about the Hungarian refugees assisted by the Jewish Joint Distribution Center, a Jewish relief organization that is still active today. These cards contain wonderful information, but that data is primary of interest in aggregate or in combination with other information about a specific individual. Their sheer number makes aggregation a challenge and the current format is largely unusable for locating specific individuals. That is where digital humanities come in; by leveraging computational tools to process these cards and database can be constructed that is searchable for individuals while also facilitates aggregation. Moreover, that database would then serve as a foundation for future work to incorporate similar collections from other relief organizations.

The problem is these are not computer-age documents, rather they are images of typewritten index cards from the 1950s. This means they are discolored, smudged, or contain handwritten corrections or characters created with worn keys or a combination thereof. Additionally, while the technology for computers to read typed characters, Optical Character Recognition (OCR), has existed for decades, it has never been effective at reading text from images that do not look like a clean sheet of typewritten text. 

In 2020, I attempted to process these cards by leveraging the Baylor University’s high-performance computing cluster (HPC) and the best open-source software to clean and read the images. While the process worked the results were simply unusable. Then in 2022, I met Dr. Tamás Scheibner who is leading the Magyar Exodus project. In our conversations, we learned about our shared interests and frustrations. I also mentioned a new technological development that looked promising, but I had not yet been able to investigate. With our overlapping interests and skills, I joined his research team and applied for a Fulbright grant to collaborate with the team in person.

The new technology is the Amazon Web Service (AWS) Textract which augments traditional OCR with Artificial Intelligence (AI) typically for business uses like automated invoice processing, etc. After arriving in Hungary this fall, I developed a proof-of-concept test using a computer script (Python) to send Textract a few sample images via the Application Programming Interface (API). The results were very promising, so then I needed to develop this into a process that could work at scale.

Fortunately, the HPC team at Baylor also oversaw the university’s collaboration with AWS. After a series of conversations, we identified a process using several AWS offerings to not only “read” the cards but also provide for a human review process allowing for correction of OCR errors. It took about a month of development, but I was able to assemble the process last November.

In late November Dr. Scheibner and I tested the process and then brought in a few other teammates to expand the testing and create the instructional materials. We went live with the public-facing process in early December. 

One part of digital humanities that I find fascinating is the ability to repurpose tools, like AWS, for academic uses. Creating these tools solely for research purposes would be time-consuming and/or expensive, yet by leveraging a “business” tool we can take advantage of this technology for a small fraction of that cost while also providing a different type of societal benefit. If you would like to know more about this process or technology, please let me know.