Leave a Reply

Your email address will not be published. Required fields are marked *

15 comments

  1. Boonton says:

    Profiling.

    Doesn’t work here and the reason why it doesn’t work is in the difference between Type I and Type II errors.

    Type I – You think someone isn’t ‘normal’ when he really is.

    Type II – You think someone is normal when he really isn’t.

    Imagine you’re at the border of Mexico heading back into the US after a vacation. There’s a line of cars and a drug sniffing dog is being walked around them. Let’s say out of 100 cars the dog barks at 5 of them and he is always right, the cars he barks always have drugs. Is he 100% perfect? Depends….

    Type I yes he is. He never barks at an innocent car.

    Type II???? Well let’s say out of 100 cars 10 have drugs. In this case he has a pretty high Type II error rate. For every two drug smugglers, he clears one of them as ‘normal’.

    The system we have has very lot Type II errors at the expense of a lot of Type I errors. This is because unlike drug smuggling at the border, we really want zero type II errors. We don’t want some bombs stopped, we want all of them stopped.

    So when people say ‘profile’ what does that mean? If it means do it on top of what we are already doing….well what’s the gain? We already got Type II errors very low if not at zero. There’s not much gain there. So all we’d get is raising our Type I error even higher, why does that help?

    If they mean do profiling rather than screening everyone….well there’s a serious problem there in that you are going to blow Type II errors thru the roof…meaning that terrorists will simply dodge the profile and since everyone is not longer screened once they find someone who doesn’t ‘look like’ a terrorist you’ve given them a red carpet to bringing down a plane. Today if they find someone who doesn’t ‘look like’ a terrorist, the value to them is less since it’s not like being white or a woman or having long term US citizenship let’s you walk onto a plane with a few hand guns and a pound of C4 without anyone stopping you.

  2. Boonton says:

    Sorry my post should have read our system currently has very FEW Type II errors.

  3. Boonton:

    What evidence do you have of that? The TSA as far as I can tell has a 0% success rate on detecting actual threats. They may have “few” type II errors, but that’s only because there aren’t many true positives out there to find. Any idiot can sneak something by the TSA — see every actual test conducted ever by amateur or professional.

  4. Mark says:

    Boonton,
    You use the categories Type I and II like that is some standard notion of types of errors. From whence do you get that terminology.

    JA,
    Uhm, I forget which airport it was (somewhere down south) but there they had a display with “things” confiscated by the TSA. Besides the (likely plethora) of innocent knivery and multi-tools, there were some distinctly non innocent material (a hand grenade was up there I think). What evidence do you have the TSA has a 0% success rate? They don’t arrest you for having a knife or a non-innocent tool. They just take it. Even if someone was detained for a weapon … why do you think it would hit the news?

    The point is your statement of “every test” conducted by amateurs or pros does not necessarily pass. I suspect those who are caught don’t trumpet that the TSA succeeded.

  5. Mark says:

    Boonton,
    When you talk about no profiling. Do you really mean that the TSA should spend as much attention and effort to screen 90 y/o women from Iowa as urban young adults?

    The TSA when they are screening has 0 information concerning who you are, even where you’ve traveled in the past few months.

    If they mean do profiling rather than screening everyone….well there’s a serious problem there in that you are going to blow Type II errors thru the roof…meaning that terrorists will simply dodge the profile and since everyone is not longer screened once they find someone who doesn’t ‘look like’ a terrorist you’ve given them a red carpet to bringing down a plane.

    Again, by what influence and incentive might a terrorist apply to that 90 y/o grandmother from Iowa? Do you really think a “red carpet” spread to the elderly midwestern women raises the risk level substantially?

  6. Boonton says:

    You use the categories Type I and II like that is some standard notion of types of errors. From whence do you get that terminology.

    You never took a course in statistics? The notion comes from hypothesis testing. You begin with a ‘null hypothesis’, say something like people who take new drug A do no better than people who don’t take new drug A. The question then is whether your sample is strong enough to reject that hypothesis. In that case a Type I error would concluding that drug A works when it really doesn’t. (I.e. your sample might have just been lucky people who got better on their own and you were unlucky in that your random sample drew you a bum hand). The flip side is the Type II error. Maybe you see little or no improvement in your drug A sample. It very well may be that drug A works but you failed to pick that up.

    JA
    What evidence do you have of that? The TSA as far as I can tell has a 0% success rate on detecting actual threats.

    A Type II error would be someone getting a bomb or weapon on board a plane so we’d know if something like that happened. (Except….if someone got a bomb on a plane undetected but opted not to use it for whatever reasons. ‘Underwear bomber’ doesn’t count as he boarded the plane outside the US).

    Fair point: If no one has tried to get on board a plane in the US with a bomb since the TSA started screening we don’t really know what the Type II error rate would be.

    Fair counter point: Maybe no one has tried because they figure there’s little hope in getting the weapon or bomb thru.

    Mark
    Again, by what influence and incentive might a terrorist apply to that 90 y/o grandmother from Iowa? Do you really think a “red carpet” spread to the elderly midwestern women raises the risk level substantially?

    Probably not. But then what’s the point? How many people standing on line at any moment are 90 y/o grandmothers? Next time you fly look around, do you think that there’s a certainity that a large portion of the people you are standing with absolutely couldn’t be terrorists or criminals of any type?

    I’m fine with maybe a ‘reverse profile’ where people who are very clearly not a threat (babies, the 90 yr old grandmother) are given very light treatment (although I suspect to a degree that already happens). That won’t do much to lessen lines, though, nor make those who are hyperventilating about ‘pornoscans’ and ‘don’t touch my junk’ feel better.

  7. Type 1 and type 2 is standard terminology. See also precision vs. recall.

    Regarding the TSA’s lack of successes, I meant they haven’t caught any actual terrorists. Obviously they’re extremely successful at taking relatively harmless tools and novelty items from passengers. Even still, you hear all the time of people sneaking on knives and the like, most recently Mythbuster’s Adam Savage, who even went through the new pornoscanner.

    The problem with profiling is as Boonton pointed out. It’s too easy to game. All they need to do is find one old lady willing and able to pass as a grandmother from Iowa (perhaps because she is one!) They don’t even need her cooperation, necessarily. Maybe they sneak something into her bag or coat. Or they con her.

    Note that neither the shoe bomber nor the underwear bomber were even Arabs.

  8. Mark says:

    JA,
    How would you know if they haven’t caught any terrorist. Say that guy with the grenade I saw in that airport display. Terrorist or not? How would you know? Would you expect him to say, “Oh, right that grenade. I’m a terrorist, that’s why I carry it.”? Instead of giving some lame excuse for why he had it.

    precision vs recall? In what context?

    Regarding grandma. So. … you’re a high ranking Jihadist based in the Middle East. You want to recruit a Lutheran 90 y/o Iowan woman to carry a bomb on a plane. Exactly how do you go about doing that?
    Boonton,
    No. I never took any statistics? I was a Physics/Math major in college and a Physics grad student. No statistics except what I figure was (is?) a pretty good familiarity with error analysis and error propagation.

    How is a reverse profile not a profiling?

  9. How would you know if they haven’t caught any terrorist.

    You think they wouldn’t brag about it?? Obviously there’s some possibility that they stopped an actual terrorist and couldn’t/didn’t make it public for whatever reason, so let me rephrase: they have a 0% success rate that we know of.

    precision vs recall? In what context?

    Pattern recognition. Precise means all hits are “good” hits (no junk results) while recall means all potential good hits were found. For some applications (like security) you care a lot more about recall than precision. For others (like pregnancy tests) you care more about precision.

  10. Mark says:

    JA,
    OK. If the TSA catches someone they have a choice, tell or not. Question is there strategic advantage to not telling? I’d offer that there is. And you haven’t responded at all to my suggestion that some those who are caught with “stuff” may in fact be terrorists. Perhaps you missed the point. For as the TSA doctrine seems to be just to confiscate and not detain … would you count a success an actual terrorist who has his weapon confiscated but is not detained a success or not? No attack occurs … but no arrest or anything to report either (except stuff taken that’s actually really dangerous, e.g., a grenade).

    So precision vs recall is just another way of saying no false positives and no false negatives. That is precision = limit false positives and recall = limitfalse negatives.

  11. Boonton says:

    Regarding the TSA’s lack of successes, I meant they haven’t caught any actual terrorists.

    Lack of success here would be letting actual terrorists thru, to date that hasn’t happened. Might they fail spot tests? Yes but it would seem that actual terrorists would rather not try to get through. I mean ok maybe some yokel just likes to carry a grenade around and if he wasn’t caught would have taken it with him on the plane and no one would have ever known about it because nothing would have came of it. Nonetheless, I’d rather even well meaning people not carry grenades on planes.

    Regarding grandma. So. … you’re a high ranking Jihadist based in the Middle East. You want to recruit a Lutheran 90 y/o Iowan woman to carry a bomb on a plane. Exactly how do you go about doing that?

    Well a Jewish grandmother was charged with carrying messages for an Egyptian terrorist. Granted she was a radical lawyer. But the above is not quite impossible. You might approach it by creating a front organization, maybe making her think she is carrying something other than a bomb for a Jihadist group (maybe, say, a package of tightly wrapped pot for which she will get a nice amount of cash when she lands on the other side).

    No. I never took any statistics? I was a Physics/Math major in college and a Physics grad student. No statistics except what I figure was (is?) a pretty good familiarity with error analysis and error propagation.

    And Type I and II errors are really brand new concepts to you? It took a while for me to get them (statistics is really a different world of mathematics it seems).

  12. Boonton says:

    In terms of TSA success, think of it in terms of drug sniffing dogs at the border.

    If, for every 100 cars that come thru 10 have drugs.

    If the dogs pick up 5 cars and all 5 have drugs then we have 100% Type I success (no innocent people bothered) but only 50% Type II success.

    If dogs pick up 20 cars, 10 have drugs, we have 100% Type II success but only 50% Type I (innocents othered).

    We know that no actual terrorist made it thru the TSA and attacked a plane. It’s possible that maybe some terrorist did make it through but for whatever reason got cold feet and didn’t attack even though he could have on the plane. It’s also possible that no terrorist made it thru the TSA simply because terrorists haven’t tried. Maybe they think the TSA is too good or maybe they are too disorganized to make an attempt.

    We’d only know if terrorist start sending people into airports and then we can see how many get caught and how many get through. That’s data, though, that we simply don’t have. I suppose you may be able to simulate it with undercover drills but they aren’t quite the same thing just like a fire drill in your building isn’t quite the same thing as an actual fire.

  13. Mark says:

    Boonton,
    Examine the grenade case. I did see a grenade in the case, yet no terrorist was claimed or caught. Now, the guy might have gone to jail … but what does that mean regarding terrorism? Even if he was a terrorist, he wasn’t going to claim he was. Again, if a terrorist fails to get stuff through and the material is confiscated … what tells you he is a terrorist and not a yokel who likes to carry a (?!) grenade. So let’s say you have a dozen or more people who may be false positives caught with genuinely dangerous stuff a year, some just have their material confiscated others get detained for illegal weapons chargers. None of them are going to be called terrorists however, so the real positives are going to look like false ones.

    What does the Type I/II categorization gain you over false positive/false negative discussions?

    (Updated slightly for clarity)

    Seems to me a little bit of public background information and flight data for the passenger coming into TSA arena’s would be helpful in alerting the inspectors who to look “just a little more” at.

  14. Mark says:

    Boonton & JA,
    HEP data analysis with statistical evaluation of data and creation of triggers. False positive/false negative were the terms used there. What advantage do these other terms have over those? Is there a subtlety that I’m missing?

  15. Boonton says:

    Maybe not, I think Type I & II might be more familiar to people who had to learn the concept at least once in stats class.

    In terms of old people, I give you James Wenneker von Brunn (http://en.wikipedia.org/wiki/James_von_Brunn). The 88 yr old man shot up the United States Holocaust Memorial Museum. While it may be hard to imagine a 90 yr old Iowa farmer getting recruited into Al Qaeda, it’s not unprecedented that one could nonetheless participate in a terrorist attack that may have nothing to do with Islamic extremism. I’ll grant you 90 yr old women have exceptionally low rates of all type of violence but it’s not sufficient to say its so unlikely that it’s unimaginable.

    Again, if a terrorist fails to get stuff through and the material is confiscated … what tells you he is a terrorist and not a yokel who likes to carry a (?!) grenade. So let’s say you have a dozen or more people who may be false positives caught with genuinely dangerous stuff a year,

    Well not quite, if someone is carrying a grenade on a plane and is stopped that, in my book, is not a ‘false positive’. It’s all well and good if his motives were purely innocent but stupid, I still want stupid nice guys carrying grenades stopped from boarding planes as much as evil terrorist guys.

    In terms of JA’s measure of the TSA, I think we agree ‘haven’t stopped a terrorist’ isn’t quite the right metric. The TSA is basically checking everyone. In that case failure is letting a terrorist slip by and so far there’s no evidence that happened (leaving aside the possibility that a terrorist did sneak by but for whatever reason didn’t attack). Since the policy is basically to scan everyone, I wouldn’t consider ‘everyone’ to be a false positive. I’d consider a false positive to be scanning someone, deciding that something ‘looks funny’ and then discovering after an invasive search that there wasn’t anything amiss after all. But if you insist we can call this policy one of maxing out Type I errors (or false positives) in order to make Type II errors (or false negatives) as low as possible.