Mightier Acorns

I find it hard to ask for help.

This is not because I am afraid to admit to not knowing things, but rather because asking people for help means one of two things will happen: either I will end up explaining the same complicated problem to each person I ask, Or, if I cast my question out to a large number of people, few of them will know what I do and don’t already know, none of them will quite understand what I’m asking, and all of them will argue with each other about something unrelated before telling me I should do whatever it is myself.

Which I still can’t do because I need help.

The Right Tool For the Job

The “job” in question at the moment is, “Collecting the records upon which to base a Callan One Name Study.”

In theory, databases and indexes already exist for collections of records. The U.S. Census records are publicly available and are already indexed, so I thought it would be easy to compile a database of people with variations of the Callan surname in the enumerations from 1850-1950.

What I want to do is input the likely spelling variations of “Callan,” and get back a table that lists each person indexed in the Census, with each individual in a row and columns for the information about them – surname, forenames, date of birth, head of household, year of census, census place, and (vitally) the census record where I can verify this information.

I already have a table I have been working on, which anyone should be able to view in my Google Drive: Callan One Name Study. This table started with an export of all WikiTree profiles I could find for “Callan” spelling variations. Next, I folded in a table I had already been building of people with the surname who appeared in the 1901 and 1911 Ireland Census for County Louth, and then added people I had already documented in my Callin Family History research. That table (which only includes people with a Callan surname at birth) has just over 2,500 names, and I have been slowly working my way through it (manually) to deduplicate rows and connect each individual to any existing WikiTree or FamilySearch profiles.

So, getting a list of individuals from the U.S. Census in that format is the goal. How do I accomplish that?

The Input: Where and What

There are two places to go that already have the U.S. Census digitized and indexed: FamilySearch and the U.S. National Archives (nara.gov). I contacted both about what I’m looking for, and neither could help.

FamilySearch said, “We are not aware of a way to accomplish the CSV capture that you are interested in.” NARA told me, “[O]ur office merely preserves and provides access to the census schedules. We unfortunately do not have the technical expertise necessary to advise you on extracting data from the census schedules.” (NARA suggested the FamilySearch Developer Portal, which, it turns out, is not an option for an individual researcher like myself.)

If I can figure out how or where to send my query, the “What to look for” problem is at least manageable. I reviewed the very long list of Soundex variations for Phonetic Soundex Key: C450 (which includes “Callin”) and decided to focus on a subset of that. (Eight variations – swapping out the second vowel for “a,” “e,” “i,” and “o,” and looking for either one or two “L”s. There is simple RegEx language that would cover those variations, but that’s as much as I know about RegEx.)

This is where AI tools usually enter the conversation, only adding to the frustration of trying to acquire this information.

Part of the problem is that few people use the term “AI” correctly, and the vast majority of people who refer to AI are completely unaware of the differences among GenAI, Large Language Models, and basic automation (e.g., Perl or Python scripting). So when most people say, “Have you tried using AI?” they might as well be asking, “Have you tried computer?”

They might as well ask if I have used magic.

A subset of those people are actual data science experts who know what they are talking about, but those people don’t usually know as much about genealogy, and often assume that I have access to a program or know how to program something to automate the task I’m trying to accomplish.

I do not.

What Has Failed

I have just enough understanding of programming and data analysis technology to know that the GenAI chat tools available to me are not capable of doing the kind of recursive search needed to query the publicly available query interface.

I have played with Gemini, because it is free and Google is already sucking up all of the data associated with me, anyway. When I first asked Gemini to try, it assured me that it could do that kind of query, but it lied. The results it gave me were not verifiable. I can query FamilySearch for individual records, and none of the people in the CSV file Gemini generated were in FamilySearch. It quickly became obvious that Gemini was making up tables of names and filling them in with data – which I can’t use for my name study.

Gemini gave me several spreadsheets full of garbage before it admitted that it can’t access an Application Programming Interface (API). NARA doesn’t offer an API interface, and FamilySearch restricts use of theirs to commercial app developers.

I tried to use FamilySearch’s public search tool to see if it was possible to do what I needed as a manual/brute force task. For example, I can choose their Records search and filter to search the 1850 U.S. Census for “Calan or Callan” (they run a “fuzzy” search by default), which returns 37,404 results. I can export each page of results, but I can only view (and thus export) 100 results at a time.

Exporting 100 results to a CSV file is partially useful and gives me a table with some of the columns of information I’m looking for. However, there is an “Other” field in the results which contains “other names in the household,” and this breaks the format so that the information in the fields does not line up – which means I would have to manually error-correct each of the 100 results in each of the 375 CSV tables before manually combining them into one table.

That’s just for the 1850 Census. The 1900 Census gives me 135,544 results; the 1950 Census gives 299,778.

And for those about to ask, no, AI tools are not “good at” cleaning that data or putting it into the format I can use. There are too many unpredictable variables, so writing the rules needed to teach a machine to give me what I want is beyond my abilities.

Asking for Help

Which brings me back to asking for help.

I have put my question out on social media, trying to find out if anyone else who has done a One Name Study has found a program or website where someone has solved this problem. I thought perhaps, since people have been using computers to do genealogical research for decades, somebody might have a basic Census index table.

Social media was not helpful. (The moderators at r/Genealogy on Reddit went so far as to delete my question for “not relating to genealogical research” – after other users yelled at me for even considering AI tools, and one person snarkily asked me, “do you think everyone with the same surname is related?”)

As I said, I have friends in the data science field, friends with a background in programming and data mining, and friends who do genealogical research; even those who belong to two or more of those groups were at a loss.

So after wasting untold hours trying to acquire a list of “Callan” folks, I am back to the old-fashioned methods, adding a family at a time, and hoping that someday I might have as complete a table as possible.

As for AI tools, the only use I have found for them is generating silly pictures – which doesn’t seem worth the damage the industry is doing to the environment or the economy. But what do I know? I’m just a hobbyist chasing dead people.

Gotta catch ’em all.