Podcast: Matching "Unmatchable" Fingerprints

In this episode, we talk about how a driven undergraduate student from Columbia University mapped fingerprints from different fingers of the same person - which was widely known to be impossible by the experts in the field - using publicly available data and machine learning.

This podcast is sponsored by Mouser Electronics. 

EPISODE NOTES

(3:30) - AI Discovers That Not Every Fingerprint Is Unique

This episode was brought to you by Mouser, our favorite place to get electronics parts for any project, whether it be a hobby at home or a prototype for work. Click HERE to learn more about the history and evolution of machine vision!

Transcript

Folks, this is the first time that I wish we had a different theme song because we are talking about fingerprints and I would die if we could have some Law-and-Order theme music roll right now. But alas, buckle up and let's get into it if you're down for the vibes.

I'm Daniel, and I'm Farbod. And this is the NextByte Podcast. Every week, we explore interesting and impactful tech and engineering content from Wevolver.com and deliver it to you in bite sized episodes that are easy to understand, regardless of your background. 

Farbod: All right, folks, as we discussed, we're talking about fingers today. But before we get into the fingers, let's talk about machine vision because it is a fundamental topic for what we're talking about today.

Daniel: It's the foundation for today's topic, if you will.

Farbod: Truly it is. And we're so lucky to have a sponsor like Mouser Electronics. And you should know by now, Mouser is one of the biggest electronic suppliers in the world, which means they know what's going on in academia. They know the cutting edge in industry. And occasionally, they write articles about it. The one they wrote about today, which I highly recommend checking out, is not just, it's called the evolution of machine vision. And it's not just about machine vision, it tells you the history of machine vision. Like all the way back to 1970s, the digital darkroom, 1987, how Macintosh incorporated it. And these different generations and what came with the modern age of this processing technology that we now have and the cameras that we're using to shoot this podcast and the AI that's helping us post-process even more and get extract more data out of the visuals that we're seeing, it's pretty comprehensive. And what I really enjoyed actually at the end, they go into the challenges that this technology faces into the future and the requirements that we want from it to continue this trend of innovation.

Daniel: Well, one thing I'll say is actually at the very end of the article talks about modules and cameras that Mouser has available for purchase, right? Again, we love how they do a really good marriage of, hey, here's the foundational technology, here's the science behind it, if you will, that you need to understand if you're gonna go build one of these projects. And then they go equip you saying, here's the exact part, the exact parts list if you wanna go buy something to get started. They talk about tech specs, they talk about frame rates and which ones are compatible with USB type C so it can go straight into your MacBook. Really legit and integrated details on, hey, here's the foundation, here's the history. Now, if you want to go get started, here's your first push. And obviously, appreciate that as well.

Farbod: Dude, absolutely. And it makes you, I don't know, because if you're reading that article, you're probably already interested in the topic. It gives you that little bit of ignition you need to pick up a weekend project, which is always pretty nice.

Daniel: And we're proud to say that. I would consider ourselves in that weekend maker category, garage hacker category, but also I have the feeling a lot of the folks in our community follow them, would label themselves in that category as well. So, if you're looking to get started with machine vision, understanding the foundations of it, and then actually going and getting started, you should check out the article we're linking in our show notes.

Farbod: Absolutely. And let's jump into today's article. We're going to be going over to Columbia University, so we’re gonna be in New York and it's a collaboration between Columbia actually and the University at Buffalo State University of New York, which is, I never got that. Is it?

Daniel: It's a mouthful.

Farbod: Yeah. I guess they call it SUNY as the short hand.

Daniel: For the state university system.

Farbod: Yeah. And I prefer that. So, let's just go with that. Yeah. But what's interesting about this article is that the lead here is not a professor, it’s not a graduate student. It's actually an undergraduate student, Gabe Guo, who's a senior, and we're gonna get into his work a little bit, but I believe he got the REU grant from NSF, which allows research for undergraduate student, which really hits home for us.

Daniel: Because we applied for that and didn't get it.

Farbod: Didn't get it, but doing undergraduate research is how we met, and we've talked about it so many times, and how great it is, and how students should partake in it. But yeah. This was awesome to see very, I like saying that this was done by undergraduate folks, but what are they doing here? We've set the background enough. The general idea that you need to know is when it comes to fingerprint forensics, like what you're seeing in NCIS or any of these crime shows, the general understanding has always been that different fingers from the same person are unique and therefore unmatchable. It's called intra-person identification, I think. And that's kind of not been ideal, right? Because if there's a set of prints that is not available in the database, like a middle finger or a pinky or whatever, or if there's multiple crimes that happen, but it's not the same fingers that appear that you're getting the scans for, it's really hard to tie them together. But what if our understanding of these fingerprints is actually just completely flawed?

Daniel: Well, and let's just say like in general, right? There's a lot of prevailing wisdom, let's say about fingerprints, right? They're, they've been the gold standard for a long time in forensics and trying to link an individual to a crime scene. Right. That's why they're so important here. Like you said, there was this long-standing belief that if you're trying to link me to a crime scene, there's no possible way that if you have a pinky print that I allegedly stole Farbod's teacup right here and I had. I left a pinky fingerprint on there, but you had only had my thumb and fingerprint on file. The long standing belief had been that there was no possible way for you to link my fingerprint from my pinky to the fingerprint from my thumb and my pointer finger. Basically saying, I feel like it's one of those things that you hear growing up as a kid and obviously hear still colloquially as a saying, every fingerprint is unique. That's like as much common knowledge as like the sky is blue. But I think it's awesome that here now we've got an undergraduate student with no background in forensics.

Farbod: None.

Daniel: With very little background in AI.

Farbod: Yep.

Daniel: Using AI to challenge this longstanding assumption and coming up with some really, really compelling conclusions saying, you know what, actually not every fingerprint is unique. And there is a way that we can link your pinky fingerprint to your thumb or to your pointer finger with relatively high accuracy.

Farbod: And what was the genesis of this idea? Well, Gabe here was like, huh, the National Institute of Science and Technology, NIST, has a database of 60,000 fingerprints that are just publicly available, right? So that's data, a lot of data, that is already cleaned and mapped and whatnot, which could be used for training some sort of artificial intelligence model which has been very popular these days. And it's become a lot easier to do. I would say over the past two years or so.

Daniel: Yeah, it's crazy. Like you said, this is an undergraduate student performing an undergraduate research project of their own interests. And they even encountered a bunch of resistance as well.

Farbod: That's what I was gonna say. They're like taking on a behemoth, which is like the traditional understanding of how fingerprint identification is supposed to work. But before we get into the behemoth, I think we should actually talk about what he did and his approach to it. So, like I mentioned, there's this database of 60,000 fingerprints that Gabe had access to. And as I mentioned, AI seemed like a good way of seeing if it's even possible to say Farbod’s pinky and Farbod’s middle finger, can we match them to see that they're actually from Farbod? So, the artificial intelligence model that they decided to pursue here is the contrastive learning model. And I had not heard of it before, but apparently, it's just like a general, common differentiating model that's used for image analysis. And as the name implies, you're trying to figure out what is supposed to be paired with what and what is supposed to be not paired with what. So positive and negative testing. And the way they trained this was by providing taking half the data, so 30,000 data points for example, and giving in positive data, so again, Farbod’s middle finger, Farbod’s thumb, this is from the same person, find out similarities if you can, and then giving Daniel's pinky and Farbod’s index finger and saying these are not related together whatsoever. And what that essentially created was a black box where they could say, are these two from the same person? Yes or no, right? What they ended up realizing as this thing was trained was I think with a 77% certainty or positive rate, it could tell us that different fingers from the same person belong to the same person, which is pretty impressive, right? Especially when you consider the fact that previous attempts at doing anything like this have resulted in, I think, a roughly 50% success rate, which is a coin toss.

Daniel: Yeah, the long-standing wisdom has been there's no way you can identify that this is Farbod’s based off of you only have his thumbprint on file and we have an unidentified fingerprint from the pointer finger at the site of a crime scene, there's no possible way that you can identify that that's Farbod’s. And like you said, given the best possible attempt so far today, it does it with about 50% accuracy, which is literally a coin toss. There are only two options, yes or no anyway. So, you're not doing that well, but getting to 77% for a single pair is really, really impressive.

Farbod: And it's not enough to get someone on and with absolute certainty to be like, yeah, you are at fault here, right? But it's still pretty impressive. And that's what they submitted to a very well-respected journal to be like, this is our work and this is what we're seeing.

Daniel: And just, it's a really interesting finding, right? Absolutely. 77% of the time, we can take Farbod’s pointer finger and his thumb finger and be like, hey, this might be from the same person. 77% of the time, the model's right. They submitted it to a journal and I'm just going to read the feedback that they got.

Farbod: After months, by the way, after months, this is the feedback they got.

Daniel: “An anonymous expert reviewer and editor concluded that it is well known that every fingerprint is unique.”

Farbod: Thank you, dear expert. I wish I could Google that.

Daniel: “It is not possible to detect similarities even if the fingerprints come from the same person.” So, they rejected these findings from being published in like a really well established. But the team opted to continue pushing, rewrite the manuscript, try and publish it to a more general audience outside the forensics community because they're like, we don't usually, this is another quote here, “we don't usually argue editorial decisions, but this finding was far too important to ignore.”

Farbod: Well, pause. That is actually coming from one of the professors that they work with, right? And that professor saying this was so meaningful that they were just not gonna back down from this. But prior to that, so after they got their first rejection, they're like, no, we're onto something. We're gonna double down, and we're gonna dig into this even more. So, they kept working on the model, and the question they asked is, how does this black box work? Because right now we just know that it can pair, but we don't really understand what's happening in there that this is able to find. Like, what is the distinction that it's able to find? And after doing some more testing and a deeper look at what was going on, that's where they kind of cracked the code a little bit. We gotta take a step back. The way that people understand or…

Daniel: Identify.

Farbod: Identify, like the fingerprint matching is by minutiae? Minutiae? How do you pronounce that? Am I right?

Daniel: Either way.

Farbod: Close enough.

Daniel: Yeah.

Farbod: Something with an N.

Daniel: Tomato, tomato.

Farbod: Is by the location of the starts and stops of the bumps, the ridges in our hands.

Daniel: Yeah, they call them branchings and endpoints.

Farbod: Exactly. So that's how it's typically been.

Daniel: If you've ever watched a crime show online, or you know or on TV, right? And they're like, oh man, let's run this fingerprint through the database. And they're running through and it's flashing a bunch of fingerprints and then identifies, oh, this is a 99% match and it highlights, it's gonna be highlighting these branching and endpoints. The ridges towards the outside of the fingerprint, the understanding the shape and then the size and the angles of the way that the ridges on the outside of the fingerprint mainly are where these branching and endpoints are, where they're able to make really, really high fidelity matches between a fingerprint that's on file and a fingerprint that's found at a crime scene.

Farbod: And what was the approach here that was different? That's the crazy thing. They realized that the model was actually looking at the angles and the curvatures of the swirls, like the patterns that we see at the center, close to the center of each finger, which they call the singularity. So, the patterns of how it's curving and the angles of where it's curving, or the angles that it is curving by, is what the model was focusing on to see if the middle finger and the thumb were from the same person or not.

Daniel: Which is something that has been traditionally ignored because the minutiae toward the outside of the fingerprint are the ones that are really easy to identify and match fingerprint to fingerprint. But it's kind of been overlooked. The center of the finger is actually really, really similar or has characteristic similarities between my pointer finger and my thumb finger and my middle finger and my ring finger that the center, the singularity area. There's actually a lot of patterns there that can indicate if you've got one of my fingerprints, you can tell that if it came from the same person, just based off of these patterns in the center of the finger, which is incredible.

Farbod: And with further refinement and better understanding of what is going on here, so with a single pair of data points, again, left hand pinky, left hand thumb, they were able to match and get 77% success rate, but with more pairings, with more samples, they were able to hit as high as 88%.

Daniel: I think when they say more than one pair, that means they've got a minimum of three fingerprints.

Farbod: Yes.

Daniel: So, you've got one pair between finger one and finger two, and you've got another pair between finger two and finger three. Right, once you've got more than one pair available or more than three fingers available…

Farbod: You have a higher certainty of distinguishing.

Daniel: Jump to 88% success rate, which is incredible.

Farbod: And that's when the professor stepped in and was like, all right, no, I'm not taking these rejections. What you've stumbled on is actually quite impressive and we're gonna make them see that. And they had, I guess, some more back and forth with another journal until they were able to get their work published, which is kind of crazy. I don't understand the resistance to, no, I actually do understand the resistance to general knowledge, right? We like to, and by we, I mean the general scientific community, we like to preach that first principles thinking is important, but when you're challenging someone's understanding that has been running for the past 40 or 50 years to solve crimes, that's difficult to swallow that an undergraduate student who has no forensics background has able to make great strides in this field. And I don't know. It's pretty impressive to me what they've been able to achieve with limited background in AI forensics and whatnot. But this just goes to show that we live in a day and an age where curiosity and genuine drive can get you pretty far with the technologies that are generally available to us.

Daniel: Well, and I, like you said, it's awesome. This is an undergraduate student with literally without even a degree yet. Was able to challenge the establishment, got around a lot of roadblocks, let's say with the support of their professor, but Gabe Guo, huge kudos.

Farbod: We're big fan.

Daniel: Yeah. We're huge fans and you've used technology that's available in a field that you aren't specialized in to challenge this assumption that's existed for decades. And let's talk about the potential impact here, right?

Farbod: Please, yes.

Daniel: So largely, there are a lot of cases that have gone unsolved because potentially you've got, you've only got my thumb and pointer finger on file and maybe if I'm a criminal, you've picked up my pinky finger and my ring finger at a crime scene, but because the ongoing working assumption has been that you can't match those to the other fingerprints of mine that you have on file, because every fingerprint is unique, that that's been the ongoing assumption today, which we know now has been flawed. That case goes cold, right? That's the only lead you had. There was no way for you to solve that case. Now having this new technology available with AI that can help unearth and potentially link criminals to crimes that are unsolved, that's awesome. One of the things that I personally have got an extra passion about is not just reviving cold cases, but also helping acquit innocent people. So, there are times that folks have been convicted on evidence, despite the fact that they weren't able to make a fingerprint match. Maybe there was other circumstantial evidence that that put them in the place and they might insist that they're innocent and they could actually be innocent. And now we've got the ability to use this new type of forensic marker based on the angles and curvatures of the middle of the fingerprint to…

Farbod: Better rule people out.

Daniel: Yeah, better rule innocent people out.

Farbod: Yeah.

Daniel: Better catch the bad guys. And I think it's a significant step forward in the use of AI in forensic science as a whole, right? Because we're now not just looking for direct visual comparisons, fingerprint to fingerprint. We've not got the ability to infer. If I've got fingers one, two, and three on file, if someone uses finger four or five to touch something in a crime scene, there's a high likelihood that I'll be able to catch them, which I guess thanks to Gabe Guo. If you're a criminal, you should be scared.

Farbod: For sure, for sure. And I'll note one more thing. Fortunately, this database had fingerprints from all genders and races, which is awesome. They did note that the performance seems to be the same across the board for data that they have from all the genders and races. But they would like more data points. Just to make sure that it's more robust before actually deploying this and using it at scale.

Daniel: Well, and I think...

Farbod: A good consideration.

Daniel: Yeah. Especially if it's gonna be used to help convict people. You don't want to be a part of the problem and wrongly can have people. But yeah, I think it's awesome. What do you say we wrap it up here man?

Farbod: I'm with you. Basically, the general understanding where it comes to forensic science has been that you cannot match different fingers of the same person because every fingerprint is like a snowflake. It's special. It's one of a kind. It's unique. But turns out that this undergraduate student at Columbia discovered that that is not the case at all. Having no background in forensics or artificial intelligence, he brought those two worlds together using publicly available database of fingerprints and recognized that the way we've been identifying the uniqueness of fingers, which is by the minutiae, the start and the stop of these ridges that we have in our fingers, totally wrong. You can actually just look at the center of each finger, look at the angles, the curvature of those patterns, and be able to match different fingers on the same hand. Now what does that mean? It means that some of these cold cases that we just haven't been able to make movement on, they can potentially get reopened with new insights. It means that cases where the wrong people have been getting convicted, we can overturn those because now we can narrow down the people that are actually at fault. Honestly, it just means a better approach to forensic science and crime solving in general.

Daniel: Now, again, huge kudos to Gabe Guo, man.

Farbod: Absolutely.

Daniel: This is incredible, awesome application of AI, and it's got legs, right? It's got the ability to impact a lot of people, again, especially something I'm passionate about, making sure that innocent people who are in jail have the opportunity to get acquitted of those crimes.

Farbod: For sure, and one thing I'll note, the code and the data used to train this model publicly available.

Daniel: Awesome, we love open source.

Farbod: We really do, we really do. And I think that's about it.

Daniel: Yeah.

Farbod: Folks, thank you so much for listening. And as always, we'll catch you in the next one.

Daniel: Peace.

As always, you can find these and other interesting & impactful engineering articles on Wevolver.com.

To learn more about this show, please visit our shows page. By following the page, you will get automatic updates by email when a new show is published. Be sure to give us a follow and review on Apple podcasts, Spotify, and most of your favorite podcast platforms!

--

The Next Byte: We're two engineers on a mission to simplify complex science & technology, making it easy to understand. In each episode of our show, we dive into world-changing tech (such as AI, robotics, 3D printing, IoT, & much more), all while keeping it entertaining & engaging along the way.