Race and ethnicity in the US census

Thanks to Donna for informing us of this interesting article in The Washington Post about how the US census has described various races and ethnic groups since 1790.  You might find the descriptions of African Americans especially interesting since we will begin USCT VCC in January.

Hope you enjoy the article.

Servant inferred relationships

We've noticed in the checks that sometimes you're asking the inputter to change a servant's inferred relationship from unknown to non-relative. You don't have to do this. Occasionally servants turn out to be relatives. Enter whichever makes sense for your situation. Checkers, you don't need to comment on it, as it's not that important.

If you have an argument for why it should be one way or the other, please let me know in the comments.

The hardest people you've found

We are going to be participating in a round table discussing both mechanized and human census linking. We like to provide examples of difficult finds that our inputters have made. Please let me know of the find you're most proud of. Who was super-difficult to find, but you still managed to find her/him?

Please provide the RecId number and all relevant information, so that we can use these as examples.

Evidence of how awesome you guys are

At the request of our PI, Chris has been testing a system of mechanized searching and matching of soldiers to the census. Most people who use the census in their research use some form of mechanized linking. We've always maintained that humans are better than machines. It looks like Chris has proved us correct.

He took soldiers from our POW sample and did a mechanized search for them in the 1870 census. I will not explain it, because it doesn't matter, and I don't really understand it.

Chris searched for a subset of our soldiers. That is, he only searched for soldiers we'd already found. The mechanized search "matched" 37% of our soldiers. Unfortunately, 16% of those matches turned out to be incorrect. When we originally searched for these soldiers, we found 80% of them. Even if you just compared 37% with 80% we'd look good. But what that number really means (based on who Chris searched) is that the mechanization only found 30% of the 80%. Hope that is correct and makes sense.

What it amounts to is census inputters rock! This is one job that won't be taken over by robots.

Feedback on the new checking system

Thanks to those of you who provided feedback on our new checking system. Here are some of the issues that were addressed.

• Some of you don't like to do full checks. Sorry, for the time being, we will continue with both full and quick checks just as we've been doing. If you have any confusion about what either of these are, please let me know.

• Some of you found census or death information that the original inputter missed. Good job! Unfortunately, sometimes you did not write your search in the checking document. Please explain how you found the new information. The original inputter should be able to replicate your search and find the same information. This helps everyone improve.

• Some people were really good about getting their checks done in a timely fashion. Others waited until the last minute. Some people were stressed out because their partners didn't send them a soldier to check until the last few days of the month. They were worried that they would not have enough time to complete the assignment. From now on, please send your soldiers out to be checked no later than the 20th of the month. That gives your partner enough time to do the check. If you are unable to do that communicate with your partner about what is going on, so each of you knows what to expect. You might consider sending out soldiers for November and December checks even earlier since there are holidays at the end of both months.

On the whole, I think the system worked pretty well. I think you were all able to get a little more inputting done because you spent less time checking. If you think of anything else you'd like to say about the new system, feel free to contact me. Keep up the good work!

October 2015 checking stats

In October, we checked 40 soldiers under our new system. I've reviewed all of the checks, and I've come up with categories of errors. I've tallied the number of errors. I recognize that some of the "errors" were judgment calls. Sometimes, words are just really difficult to read. Sometimes a family is just difficult or Ancestry is acting up. Every person had errors, and there is always room for improvement. Here are the categories, their definitions, and the total number of errors for each category.

GRID Errors

• MILIN?/MAR? - Forgot to mark the column - 8
• Missing HH member - Forgot to add a person from the manuscript to the Grid - 4
• Duplicate people - An individual was included on the Grid multiple times - 2
• Wrong person - An incorrect person was added to the Grid - 1

Inferred Relationships

• Incorrect relationships - This is an incorrect inferred relationship (do not confuse it with the relationships on the manuscript) - 1

Census Errors

• Name - The name was not changed to match the manuscript - 46
• Typo/Reading/Wrong - This is a typo, a reading error, or some other way that data was entered incorrectly - 85
• State Code - The state or country code was entered incorrectly - 3
• Missing/Wrong URL - The URL for the census manuscript is missing or incorrect - 0 (yippee!)
• Missing data - A field that had information on the manuscript was not entered in the screens - 35
• Additional finds - The checker was able to find decades the original inputter missed (Some inputters sent soldiers they had difficulty with out for checks. That is fine. You can see by this number, that checking helped improve the data.) - 58
• Quality Code - The quality code is incorrect - 16

Death Errors

• Typo/Reading/Wrong - This is a typo, a reading error, or some other way that data was entered incorrectly - 8
• Missing data - A field that had information on the death record was not entered in the screens - 13
• Missing/Wrong URL/Source - The URL or death source is missing or incorrect - 11
• Quality Code - The quality code is incorrect - 9
• Additional finds - The checker was able to find death information the original inputter missed - 22

Tree Errors

• Missing/Incorrect information/relationships - There is wrong information on the tree or you did not include on your tree information that you entered into the screens or that you otherwise used to do the work. This does not include saving all those relationships in the early decades that aren't marked on the manuscript. - 16

The total number of errors for all 40 soldiers is 338. Some errors affect the data more than others. If we checked other pensions, we'd probably find similar errors. To put this in perspective, (provided I did the math correctly) at this rate, the total number of errors for all of Project 1 would be 71,825. Of course we corrected the 338 errors. We plan on checking and correcting about 10% of the sample. This means we will correct about 7,183 errors. At the end of the sample, we will still have 64,642 errors in the data. 

Thank you all for taking this so seriously. Let's set a goal to reduce our errors in November.

Births and deaths within the census year

There is a little confusion about what to do when individuals are born in the census decade, but after the census was taken. There is also confusion about what to do when individuals die in the census decade but before the census was taken.

If the individual was born after the census was taken in a given decade, the system will not add a black dot for that decade. Please make that decade a black X (for not searched). For example, if a person was born in 1880 after the census was enumerated, the system will have black dots for the decades 1850, 1860, and 1870. Mark 1880 with a black X.

If the individual died before the census was taken in a given decade, the system will not add a purple gravestone for that decade. Please make that decade a black X (for not searched). For example, if a person died in 1920 before the census was enumerated, the system will have purple gravestones for the decades 1930 and 1940. Mark 1920 with a black X.

This does not hurt the data. From a data perspective, black dots, purple headstones, and black Xs all mean the same thing. They are just administrative tools to make our jobs easier.