How not to name self-verifying test data
Self-verifying test data makes it easy to determine if it’s right. You don’t need to go check somewhere else. It is its own oracle. Here’s an example:
Address: My last name
Something is wrong. It’s obvious from the data itself. We don’t know exactly where the problem is or how to fix it, but we can tell there’s something funky with either Address or Last Name or both. If you feel like there must be more to it than this, read Noel Nyman’s 13-page paper to see that there isn’t.
Let’s look at a real-life example: I’m writing an automated test for an API. First, I create (POST) an item, then I read (GET) the database to see if it’s there. One of the fields in the body of the POST call is a Universally Unique Identifier, or UUID.
A UUID is a 128-bit identifier; we’re using them as unique keys in our database. Here’s an example of a version 4 UUID: 74ee94d-d32f-4844-9def-81f0d7fea2d8. (If you’re a human that can’t read regex, you might find this useful.) I generated that using the Online UUID Generator Tool.
I wanted to see what would happen if I tried to POST with a UUID that wasn’t valid. If I’d taken my example UUID and removed some characters to make it 74e4d-d32f-4844-9def-81f0ea2d8, it would have been invalid. My test would have behaved as I expected. But I wouldn’t have been able to tell at a glance if this was a valid UUID or not. It wouldn’t have been self-verifying.
I decided to name my UUID This is not a valid UUID.I wanted to easily be able to tell from the GET call if it succeeded, or the error message in the POST call if it failed. It would be clear when running or debugging the test what value was being passed in, why, and to which field it belongs.
Or so I thought.
I ran the test. This was the output.
I sat starting at it for a while. The first line where the red AssertionError starts looks confusingly similar: The left side looks like the right, so the assert should have passed instead of failed. The message below had is not a valid UUID twice. Huh? Finally, I realized what I did, and highlighted the part you now see in blue. I gave my self-verifying test data a name too similar to the error message. Let me boil this down:
Error message I expected: UUID not valid
Error message I got: Published service UUID {{insert UUID here}} is not a valid UUID.
Unfortunately, I’d named my UUID This is not a valid UUID. so the whole invalid input error message was:
Published service UUID This is not a valid UUID. is not a valid UUID.
Fans of 30 Rock may recognize this problem:
My self-verifying test data would have worked fine if the error message was exactly how I expected it. The test would have passed, and I may not have taken a closer look at the text of the error message. But of course, my developers had changed it to be more meaningful and give more context, the bastards. Thus, I uncovered my perfect-turned-nonsensical name. I changed the UUID in my test data to be calledFAKE UUID. It may not be the perfect name, but at least the code is better.
Calling things names related to the thing they are: great!
Calling things names too similar to the thing that you’re trying to test: confusing.
Originally published on Medium.