Man jailed over computer password refusal - <AUTHOR> http://www.bbc.co.uk/news/uk-england-11479831 <comment><USER2> Encryption and password privacy is an entirely unsettled area of US law. The courts can probably compel you to _enter_ your password (to decrypt a drive, or what have you), while you can maintain that the _content_ of your password can be protected under the 5th. So, for instance, say you had encrypted files of plans to build a bomb and detailed schematics of the White House. The judge can order you to decrypt the files without forcing you to reveal that the password was "K1llt3hPr3zn0w!"

As a practical matter, I've wondered what would happen if someone simply claimed they couldn't remember the password. Especially if one could make it look like the encrypted files hadn't been accessed in over a year.

TrueCrypt's Plausible Deniability (<http://www.truecrypt.org/docs/?s=plausible-deniability>) makes these issues even more complicated.

But yeah: by simply refusing, you'd be thrown in jail for contempt and your only way out would be appellate review of the order. You'd have to challenge the contempt citation on the basis that the original order was unlawful.

<comment><USER3> I was a grad student in CS at Cambridge when this law was introduced. A nice man from the police came to lecture us about it. We asked about proving that say the results from a Monte Carlo simulation, or even just a blank disc weren't encrypted - we were told not to worry the law would only be used against terrorists.

Since this was before 911 - the 'terrorists' in question were presumably the IRA, not sure they had much of an online presence back then.

<comment><USER4> > Since this was before 911 - the 'terrorists' in question were presumably the > IRA, not sure they had much of an online presence back then.

You do know that 9/11 was not the first terrorist attack on the US by Islamic extremists, do you?

<comment><USER3> Yes, but I'm assuming the UK didn't draft a law in the mid 90s to prevent Islamic attacks on the US.

The fact that their plans would be written in a foreign language (never mind a foreign alphabet) would have been more than adequate to keep it secret from British intelligence.

ps. You do know that we have been having terrorist attacks for almost a century.

<comment><USER5> >The fact that their plans would be written in a foreign language (never mind a foreign alphabet) would have been more than adequate to keep it secret from British intelligence.

This is one of the stupidest statements I have ever seen.

<comment><USER6> Why? Given that in 2010, nearly a decade after 9/11 and almost 7 years after the invasion of Iraq, the US military, State Dept, domestic law enforcement agencies and presumably intelligence agencies still have a severe shortage of Arabic-speaking staff, why do you find the statement stupid? The UK before 9/11 was much less focused on terrorism than the US is now. Developing robust foreign language capability in large organizations is very hard. That's why we often fail at it.

Plus, in my experience, Arabic is a difficult language for people who grew up speaking romance languages to learn. Perhaps not as difficult as Japanese, but still much more difficult than German or French.

<comment><USER5> Of course most police forces and so on will not have speakers available, but the comment said "British intelligence". Every intelligence agency has translators for all major languages, and have for decades (Arabic was an important language in the Cold War almost from the start). And they have translators for minor languages on tap - I bet they could find a Basque, Lapp, or Chukchi speaker if they needed one quicker than most universities.

<comment><USER7> "I bet they could find a Basque, Lapp, or Chukchi speaker if they needed one quicker than most universities."

That's probably where they get them.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> Just thought of a feature idea for TrueCrypt and other similar packages: encrypted files or partitions can have multiple passwords, which reveal different things. So you could have a password that reveals something embarrassing but not incriminating. If the police or border nazis threaten you with prosecution unless you reveal your password, you give them this one. Meanwhile, you hide anything really confidential behind a password that you never give out.

Or maybe they already have this?

<comment><USER9> TrueCrypt already has this feature: <http://www.truecrypt.org/hiddenvolume>

<comment><USER10> Presumably the UK police are aware of this feature, which could lead to a more interesting situation when you can't prove that you've really unlocked to the deepest level.

<comment><USER11> My understanding of the feature is that is is impossible to verify whether or not you are using a hidden volume within a TC encrypted volume.

 _Although file-hosted TrueCrypt volumes (containers) do not contain any kind of "signature" either (until decrypted, they appear to consist solely of random data), they cannot provide this kind of plausible deniability, because there is practically no plausible explanation for the existence of a file containing solely random data. However, plausible deniability can still be achieved with a file-hosted TrueCrypt volume (container) by creating a hidden volume within it._

<http://www.truecrypt.org/docs/?s=plausible-deniability>

<comment><USER12> Clarification: it's impossible to determine if a hidden volume exists in a TrueCrypt volume. It is trivial to determine whether a given password unlocks the main, hidden, or neither volume.

<comment><USER13> How is it trivial to verify whether a password unlocks something the existence of which is impossible to verify?

<comment><USER14> Data about the hidden volume is encrypted and kept in the second 512 bytes of the volume, where as data about the normal volume is in the first 512 bytes. If there is no hidden volume, the second 512 bytes are purely random data.

It's impossible to tell an encrypted volume header apart from random data. It's very much "try, and if you fail, you either have the wrong key or the volume doesn't exist".

</comment></comment></comment></comment></comment></comment></comment><comment><USER15> It occurs to me that since this part of a child pornography investigation, 16 weeks in jail for not giving them a password might actually be a far lighter sentence than if the key was provided and illicit material was found. I don't condone the porn, but it does seem like a logical trade-off.

<comment><USER16> Wait until they ask him again, 16 weeks from now...

<comment><USER17> Can you be tried for the same crime twice?

<comment><USER18> In the USA you cannot be tried for the same crime twice under double jeopardy laws. From wikipedia:

 _There are three essential protections included in the double jeopardy principle, which are:

\- being tried for the same crime after an acquittal

\- retrial after a conviction, unless the conviction has been reversed, vacated or otherwise nullified

\- being punished multiple times for the same offense_

London and Wales repealed prohibition against double jeopardy in 2003, so if you live there, you can indeed be tried for the same crime twice.

<comment><USER19> _But_...

this wouldn't be double jeopardy. It would be a second instance of him refusing to turn over the passwords. Just as you can be tried twice for murder twice if there are two separate murders, you could be tried twice in this situation.

<comment><USER20> But... it's the same password/encrypted data here.

<comment><USER19> Disclaimer: IANAL. Disclaimer: IANAA (I Am Not An American)

Assaulting the same person twice would still be two different assaults. Stealing a truck, getting caught and punished, and stealing the same truck again would, to my understanding, not be risk-free, legally speaking. I suspect the same would probably apply here, though given how unintuitive the law is, especially in this area, I may well be dead wrong.

Edit: To clarify, my point is that _if_ the law amounts to "Refusal to turn over requested passwords => jail time", this would seemingly constitute a second refusal, even if the requested password was the same.

<comment><USER21> By the same logic, if the authorities had asked him one hundred times in the first interview for his password, and he'd refused one hundred times, then he could be charged with one hundred counts of the offence and put away for 30 years.

The courts aren't run by robots. If it's substantially the same instance of the offence, he couldn't be tried again.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER22> A friend of mine flew back home to Canada. After clearing customs, he was one of the random people chosen to have their luggage inspected. He had his laptop on him and the customs agent booted up the computer, asked him to enter his password and then took his laptop away before bringing it back without telling him anything about it.

I wondered what would have happened if he refused to type in the password.

<comment><USER23> Customs aren't law enforcement, they just like to think so.

The request to enter a password should be treated as a impolite request, which you should politely deny.

<comment><USER24> Not sure why your were upvoted (probably for the sentiment that freely entering one's password without reason should not happen) because that is not correct.

In the US, "customs" typically refer to ICE and TSA, both are enforcement functions of the United States Department of Homeland Security.

<comment><USER23> _checking facts_... You are correct. I incorrectly assumed they were equivalent to UK and Norwegian customs, where customs and border protection are separate entities.

</comment></comment></comment></comment><comment><USER25> In the USA can you be compelled to testify against yourself by being coerced into giving a password to law enforcement?

Has this been tested yet, out of curiosity?

I remember reading they can deny you entry/exit to the USA if customs can't read your laptop but never heard anything like local/FBI.

If not, I hope it doesn't go before _this_ particular supreme court.

<comment><USER26> They can slap an "obstruction of justice" charge on. Or charge you with "contempt of court" and just jail you based on that.

In broader terms yes the system has a way to inflict random punishment on you for disobidience.

In other countries they will just start breaking your fingers, your loved ones fingers, and so on. So the password problem is solved a lot "easier" then.

<comment><USER27> Once they start breaking your fingers, it gets a whole lot harder to enter that 50-character password.

<comment><USER28> Relevant: <http://imgur.com/0YHea>

</comment></comment></comment></comment><comment><USER29> It seems like a bad idea to store anything incriminating on your local hard drive. Why not keep your encrypted files on a flash drive? If the police show up destroy the flash drive using a hammer, ensuring that the flash memory chip is thoroughly pulverized and completely unreadable.

Likewise, if you are going to be using the internet for devious purposes drive around and use a neighbor's open wireless network access point, which highly reduces the chances that anything can be traced back to you. Or set up your own unsecured wireless network point and suggest to officers that illegal use came from an outside source.

Not that I want to condone illegal activities, or condone lying to police officers, but to the hacker in me these seem like simple, sensible steps to take that will be more dependable than even a 50 character password.

<comment><USER2> Intentionally destroying incriminating evidence is probably not something you should ever do. Certainly not in such a way that leaves evidence in the form of pulverized IC remains all over your kitchen counter.

<comment><USER20> But how can they prove that it was _incriminating_ evidence?

<comment><USER30> Usually there'll be a reason for the police to kick your door down. I imagine that plus destroyed evidence would be enough to prove beyond reasonable doubt.

<comment><USER30> Also, I don't think it necessarily needs to be proven that the evidence was incriminating; just destroying evidence is criminal behavior.

Of course, IANAL, but that's how I'd do it if I ruled the world...

<comment><USER31> How do you know it was evidence or related to the bust?

<comment><USER30> There might be all sorts of reasons - if you're being busted for fraud or tax evasion and all of your hard drives are mysteriously blank, or there are empty filing cabinets and a big pile of ash then I would imagine that would go down fairly badly in court.

The modern equivalent is a whole bunch of destroyed media - thumb drives, flash cards or hard drives.

</comment></comment></comment></comment></comment></comment></comment><comment><USER32> "It sends a robust message out to those intent on trying to mask their online criminal activities that they will be taken before the courts with the ultimate sanction, as in this case, being a custodial sentence."

seems like a good reason to have "innocent until proven guilty". drive could contain anything, or nothing.

</comment><comment><USER33> I wonder why he didn't say he "forgot the password". Although it may seem implausible, how could they prove it's not true?

<comment><USER10> I think that the law is worded so that it's an offence to have encrypted files and not be able to decrypt them. Whether it's deliberate or just forgotten isn't relevant (though I'd hope it would make a difference in sentencing).

<comment><USER34> How can you even prove that a file is encrypted? The whole law is baloney.

<comment><USER35> Use TrueCrypt to do whole disk encryption on your Windows XP hard drive. Then boot your computer with a Linux Live CD and dd the first 512 bytes to stdout. This is what you'll see in plain text ASCII:

 _"TrueCrypt Boot Loader"_

No expert is needed to prove that you are using TrueCrypt whole disk encryption. It has a huge stamp right up front.

<comment><USER36> That's certainly true for that particular instance, in that particular implementation. But it's not necessarily the case. Take some random file and encrypt it with gpg; there's nothing obvious in the contents to mark it out as encrypted data.

<comment><USER35> tom@ubuntu:~$ gpg -c flag.png

tom@ubuntu:~$ gpg -v --list-packets flag.png.gpg

       :symkey enc packet: version 4, cipher 3, s2k 3, hash 2

       salt eae60ad4255dc4e2, count 65536 (96)

       gpg: CAST5 encrypted data

OpenPGP encrypted data is easy to find too. It even tells you the algo used. The example is symmetrically encrypted, but it works the same with asymmetric keys. Even shows who it is encrypted for. Edit: formatting.

<comment><USER35> Here's what the gpg data looks like when using someone's public key to encrypt a file... now we know who to go hit over the head with a hammer ;)

tom@ubuntu:~$ gpg -v --list-packets file.gpg

       :pubkey enc packet: version 3, algo 16, keyid 63E6E0BBB9FEE3A5

    	data: [2045 bits]

    	data: [2047 bits]

       gpg: public key is B9FEE3A5

       gpg: using subkey B9FEE3A5 instead of primary key 7A997B0A

<comment><USER37> With option --throw-keyids, you'd have to try your hammer on several heads until one fits.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER38> "50-character encryption password" - nice!

I'm wondering ... Person A refuses for - pure principle (and maybe some ripped DvD's) Person B refuses for - let's say child pornography and a dirty bomb manual

Both will get the same jail time?

<comment><USER39> There are other legitimate reasons to not want to reveal the contents of your hard-drive besides principle or self incrimination. For instance, if you had the private information of any other people. My SO works with HIV, and recently got access to sensitive data that had to be sent on DVD via courier.

Who here trusts the police to not disclose their HIV status?

<comment><USER40> Disclaimer: IANAL

If you're in the Unites States, the data is probably protected by HIPAA, the Health Insurance Portability and Accountability Act[1]. HIPAA includes a clause stating that the Attorney General or their designee may issue a subpoena compelling your SO to disclose that information, but only to someone investigating a Federal health care offense.

I've searched through the rest of HIPAA for keywords such as "law enforcement", "criminal", and "disclosure", but I couldn't find anything about being compelled to disclose HIPAA-protected information to law enforcement in any other circumstance than investigation of a Federal health care offense. However, I did not thoroughly read HIPAA, and there might be something in another section of the US Code that's relevant.

Hopefully someone more knowledgeable about this can let me know if I've missed something.

[1]: <http://www.legalarchiver.org/hipaa.htm>

<comment><USER39> Yup, that's the case. The question is whether or your hard drive is protected from a criminal charges subpoena. The data is kept encrypted on the hard drive, but of course the authorities don't know what's on the hard drive until it's decrypted.

<comment><USER40> Disclaimer: IANAL.

Edit: My understanding of HIPAA was incorrect.

45 C.F.R 164.512:

A covered entity may use or disclose protected health information without the written consent or authorization of the individual... in the situations covered by this section, subject to the applicable requirements of this section.

(a) Standard: Uses and disclosures required by law. (1) A covered entity may use or disclose protected health information to the extent that such use or disclosure is required by law and the use or disclosure complies with and is limited to the relevant requirements of such law.

This would seem to give court orders and criminal subpoenas the power to demand decryption of your hard drive regardless of whatever HIPAA data it contains.

</comment></comment></comment></comment></comment><comment><USER41> As great as hidden volumes are, traces showing the inconsistency between the fake and real volume will be left on your system unless you take heroic measures to erase them.

Things like logs of all the external drives you connect, and links to recently opened files.

<comment><USER12> Your entire operating system can be a hidden volume. I don't know how "heroic" this is.

</comment></comment><comment><USER42> I've been resetting people's 8 character passwords lost due to Post-Vacation- Insomnia for ages, I'd really like to see them expect me to remember a 50 character password under stress conditions.

<comment><USER27> I have a 12 character password and it is a pain in the ass to type it several times a day. I couldn't imagine using a 50 character password.

<comment><USER43> It is probably just a concatenated string of his credit card number or social security number and random words. I wonder if they are currently trying to crack it using some kind of dictionary brute force mechanism, or if there is some kind of lock out enabled after five tries.

<comment><USER44> If they have physical access, then there is no effective lock-out mechanism. Presumably they can determine which encryption software is used, and can use the algorithm as many times as they want.

</comment></comment></comment></comment><comment><USER10> This has happened before, although the details aren't clear. It was reported last year (<http://www.theregister.co.uk/2009/08/11/ripa_iii_figures/>) that two people had been convicted for similar offences. It seems that most people don't comply, but not all of them are charged: "Of the 15 individuals served, 11 did not comply with the notices. Of the 11, seven were charged and two convicted."

</comment><comment><USER45> How do they know it's a 50 character password?

<comment><USER46> He probably volunteered that information.

<comment><USER45> Seems a silly thing to offer

</comment></comment></comment><comment><USER47> I realize this story is in the UK, but could it happen in the US or would the 5th amendment protect you?

As an American, it seems a bit insane to me for someone to be jailed for refusing to help convict himself.

<comment><USER48> I thought remembered a court precedent in the US that ruled being compelled to divulge encryption keys was equivalent to being compelled to hand over keys to locked places (like a safe), which is historically required.

That said, I can't find such an article, so hopefully I'm making it up and we are indeed safe in our minds.

<comment><USER49> No legally in the US you are not required to give the keys to the safe, but they can drop your sentence significantly for working with the investigators. For a safe, if you don't give them the key, the get a grinder and hack saw, for an encrypted hard drive in the US they give it to the NSA (though 50 chars would be tough for even them i'd expect).

<comment><USER50> The NSA doesn't want anyone to know what their capabilities are, but I remember a case where they gave it to a university which threw a cluster at it...

edit: 'they' being 'the authorities', not the NSA.

</comment></comment></comment></comment><comment><USER51> I am surprised that they didn't keylog his machine - as having a warrant to search/seize means a warrant to keylog probably could have been obtained.

The police will learn from this and avoid these 'oh dammit' moments by just keylogging everybody from now (or at least those suspected of having encrypted volumes).

Keylogging is the one real weakness of all the TrueCrypt/other encryption schemes (that and your password is in memory in the clear while the volume is mounted, and even afterwards depending on your settings).

<comment><USER52> Could you defeat that somehow by having rotating keys?

<comment><USER53> Wouldn't that take a very long time to re-encrypt a whole drive with a new key?

<comment><USER54> You wouldn't re-encrypt the whole drive; usually, the master key is stored in the first few sectors of the disk, encrypted with the login key. That way, you only have to re-encrypt a small amount of data to effectively change the entire disk's key.

</comment></comment></comment></comment><comment><USER55> In the US, how does the fifth amendment work in this type of issue? If he complies, then he is testifying against himself.

</comment><comment><USER56> It's about time this happened. In the UK we have a law called the Regulation of Investigatory Powers Act (RIPA) which allows access to certain data held by ISPs or can compel people in certain cases to not only hand over encryption material but prohibits them from acknowledging that they had been charged under such a law.

As you can imagine, that last bit results in some very complicated situations. The laws governing paedophilia are quite different, with paedophiles having to sign a sex offenders register.

In the case of a sex offender being caught, it's easier to just take the RIPA sentence instead. This is what appears to have happened. I hope the guy's password is long enough otherwise regardless of his crime he's in for a world of pain.

</comment><comment><USER57> If they are unable to crack the pass code, does he go free on lack of evidence?

<comment><USER58> If they base the case entirely on one piece of evidence than they have already lost. Through programs like CSI, people are convinced that most of the evidence for a case comes from a lab and all they need is that one piece of evidence.

What really happens is the investigators have to answer the 5-Ws (who,what,etc) and they "build" a case against you. If they have one piece, then they can go fishing for the rest. This is why it's important to shut your month when talking to the police. Anything you say at this point can really open you up to all types of crap later. They'll twist your words around, become your friend, good cop/bad cop, mention friends and family and all other types of tricks to get you to talk. Believe it or not, criminals willingly give up testimony about themselves. Some guy in a lab with half a shoeprint isn't what wins the case, you do.

If all the prosecution has is one piece of evidence then a competent barrister can shred it to pieces. This guy's job is to create FUD, and lots of it. The less evidence the prosecution has the easier his job is. But what he/she can't do is fix anything you say in front of the police while he isn't there. This is why it's important to shut up and ask a lawyer first.

</comment></comment><comment><USER59> I am not sure I understand the basis on which this person was jailed. If the court order to search his computer was provided on the basis of probable cause, what was that probable cause? If law enforcement already had evidence of some kind of misdeed, why do they need access to his computer? If the evidence of his misdeeds is on the computer, wouldn't providing it be a) self- incrimination of some description, as mentioned extensively elsewhere here and b) law enforcement's inability to get access to the information get the case dismissed due to lack of evidence?

</comment><comment><USER60> Can anyone explain how TrueCrypt works for OS X? Could I have my entire home directory on the hidden partition? What about their hidden operating system feature? So I can have my normal OS as the decoy OS and then have a hidden Linux OS (as example) that I use for sys adm type stuff and boot to it when I need to? Can VMWare or Parallels see this partition and create a VM based on it?

<comment><USER61> Essentially, you can't true-crypt anything that can't run TrueCrypt before you need access to it. So you wouldn't be able to run TC and enter your password prior to logging in, which requires your home directory. Perhaps this could be sidestepped, but I'd think it'd be a mega-hack unless you can boot to a USB drive which can decrypt things and _then_ boot OSX. I haven't heard of anyone doing that though, probably because Macs are a bit different with their bootup. You're essentially stuck making a file-as-a-volume or a hidden partition, though I don't know how / how well hidden partitions work in OSX.

Once your file / hidden partition is mounted, it's just another mounted volume. Anything which can read / write to a volume it's not on shouldn't notice a thing.

Also, it looks like it might just be Windows which gets the hidden-OS capability, as it requires a TC boot-loader on-disk or on an external booting device. Which means it should be possible for others as well, but it sounds like they haven't done it yet. <http://www.truecrypt.org/docs/?s=hidden- operating-system>

</comment></comment><comment><USER62> Would it be possible to have one password for accessing the system, and a separate password for permanently wiping sensitive parts (in the background, even)? What would be the legal implications (other than the obvious obstruction of justice charges if the authorities catch on)?

<comment><USER12> The problem there is that any evidence should have been copied and should be accessed from a read-only media at that point.

</comment></comment><comment><USER31> It sounds to me like TrueCrypt needs a new feature: a password that just destroys all data.

<comment><USER63> would be fast too. just zap the headers.

</comment></comment><comment><USER64> They already have this. You can have two levels of passwords: one for the OS and one for a hidden encrypted partition or encrypted file where you keep your truly sensitive info.

</comment><comment><USER65> Not the first one: <http://www.theregister.co.uk/2009/11/24/ripa_jfl/>

</comment><comment><USER66> It's the modern alternative to pulling fingernails.

</comment><comment><USER67> Steganography.

</comment><comment><USER68> If the Police have a warrant to search a safe, presumably you can be required to hand over the combination or be in contempt of court. Even if the combination for the safe 'lives in your brain'.

ISTM a virtual 'locked container of documents' would have the same legal status.

I'm not sure why so many techies go down the Walter-Mittyesque 'Enemy of the State' route when discussing this sort of thing. Mention the police in conjunction with encryption and suddenly everyone's a paranoid compound- dweller...

Let's be clear here - it is unlikely that this guy is making a stand for paranoid techies - it is much more likely that he's got pics and videos on his HDD of kids getting raped that he doesn't want the police to see.

</comment><comment><USER69> This Star Wars quote seems appropriate here, especially given the hidden volume feature of TrueCrypt:

Princess Leia: "The more you tighten your grip, Tarkin, the more star systems will slip through your fingers."</comment>


Wall Street Starts Trading BitCoin on the Sly - <AUTHOR> http://www.betabeat.com/2011/08/02/digital-derivatives-bitcoin-markets-wall-street-bankers/

<comment><USER2> On Sunday night, Betabeat blundered into #bitcoin-pit, an IRC channel "for professional trading," and found a surprising number of Wall Street professionals, including "Bill," who declined to give his name but proved his employment the next day with a furtive call from an empty conference room at Credit Suisse. He had just sold 400 Bitcoins, he told Betabeat, which he planned to buy back in the morning. Given that the third-largest Bitcoin exchange, Bitomat.pl, had just accidentally deleted the 17,000 BTC it had in deposits at the same time another popular Bitcoin service shut down—possibly because its owner decided it was time to take the money and run—Bill was betting the price would be right.

"Bitcoin is like Wall Street was 15 to 20 years ago when I entered," he said. "Fragmented markets, wide spreads, manipulation, the trading stone ages."

<comment><USER3> Which is like a huge warning to get the fuck out: They're just looking for volatility. Not something you want in a currency.

</comment></comment><comment><USER4> "He had just sold 400 Bitcoins, he told Betabeat, which he planned to buy back in the morning."

"You can't do more sophisticated strategies."

Which just shows that the entire finance sector is a side game of the market. Literally a casino with slightly better predictability. They produce nothing. They just move money around to make money. Not surprisingly, they're moving into another side game, less functional and strategic, but less regulated.

<comment><USER5> All true, but nonetheless good news for bitcoin, because the government is less likely to shut it down if their Lords and Masters are making money with it.</comment></comment>


A Look into Machine Learning's First Cheating Scandal - <AUTHOR> http://dswalter.github.io/blog/machine-learnings-first-cheating-scandal/ <comment><USER2> Long story short:

\- The LSVRC "visual recognition" competition has a rule in place, that limits to twice per week how often each contestant can run their entries to the contest against the ImageNet dataset.

\- The Baidu team run their tests much more often, claiming that they understood the limit was placed on a person basis and not per team.

\- This rule is in place because more frequent submissions somehow distort the quality of the test, by switching its emphasis to overlearning (adapting too much to the specific data included in the test dataset) instead of true algorithmic advances, which can influence the whole machine learning discipline given this contest's high profile. ( * )

\- As a result, the whole Baidu company has been banned from the competition for a year.

( * ) ("The Baidu team's oversubmissions tilted the balance of forward progress on the LSVRC from algorithmic advances to hyperparameter optimization.")

<comment><USER3> Do they test against the whole dataset? Wouldn't that be fixed by having a test dataset and a separate, bigger dataset that would be used for the final grading (like on Kaggle)?

<comment><USER4> I should have read the comments, I just asked the same thing. I always wondered how that worked, though - you don't submit an algorithm, do you? Just a spreadsheet with your predictions, in which case how can they evaluate your algorithm on another test set?

<comment><USER5> You get to see the feature vectors of the test set, or at least submit your algorithm to execute upon them. If you can see the features of the test set but not the labels or target variable for each feature vector, then you can't use it to help you (e.g. the problem of modeling the distribution over outcomes for a given feature vector (whose target outcome you don't know) _is_ the problem you're solving).

But if you can repeatedly run different models against the test set _and you do get to see a score_ , you could do something like random parameter searching or other optimization ideas to tune your algorithm to be highly overfitted to the test set.

Another way to avoid this would be to develop several test sets that are roughly "equivalent" in terms of the distributional properties, and then randomly change the test set periodically, or change the test set right after the final submission deadline, to discourage people from pursuing overfitting.

<comment><USER4> That makes sense. I was thinking of the multiple test set approach, but rather than forcing people to re-submit on a new test set near the end (if that's what you meant), the organisers could just return the score based on a random fraction of the test set. 'Cheaters' would then overfit to just half of the actual test set and this would be apparent when the final scores (on the other half) are revealed. I suspect this is what Kaggle do, as they don't make you re-submit at the end (AFAIK).

</comment></comment></comment></comment></comment><comment><USER6> To translate to a more familiar domain, think about the SATs. After a period of study, students take the SAT where they get back the composite score of how they did, but never the actual answers to the questions on the test.

Now imagine a student can take the test repeatedly over the space of a few days, and can use the score to reverse engineer the answers to the questions. They can put in random answers and note which ones cause the score to go up. Of course the real life SATs don't allow this, and they change up the questions to prevent this sort of cheating. If this were possible, our enterprising/cheating student can derive the complete answer key over time, noting the changes in scores for each run. And once they have the key, they can ace the test. No longer is it a test of their aptitude, but rather of their knowledge of the answer key.

This scandal is analogous to this albeit contrived example. With an ML testset, it's not possible to change the data because you want it standardized so you can evaluate improvements that new approaches may bring. It's the only way to have a meaningful yardstick to measure against. Thus, the only way to prevent such gaming is restricting multiple submissions, so that you can't do 'hyperparameter optimization' \- i.e. overlearn on the testset.

That's why it's cheating - it's not a measure of how well your algorithm did, but rather on how well you reverse-engineered the answer key. It's a huge disservice to the field and the people who did this should be ashamed of themselves.

</comment><comment><USER7> Idle speculation:

> "The key sentence here is, 'Please note that you cannot make more than 2 > submissions per week.' It is our understanding that this is to say that one > individual can at most upload twice per week. Otherwise, if the limit was > set for a team, the proper text should be 'your team' instead," Wu wrote.

I wonder whether, to a native Chinese speaker, this really does sound like it's talking about individual people, and saying "you" when one means "your team" seems really bizarre. Can any Chinese speakers weigh in?

(Even stipulating this, the affair still sounds more like malice than incompetence on the part of Wu.)

<comment><USER8> I agree with your last remark. Regardless of the letter of the law, any ML researcher who could possibly win such competition should clearly understand its spirit. They had to know that they were training on the test data, a cardinal sin in machine learning.

<comment><USER9> Exactly. This is machine learning chapter 0 stuff. To think a team of machine learning (most likely) PhDs innocently misinterpreted the rules is naive. More likely, I imagine the researchers felt pressured to obtain results significantly enough to let their morals take the sidecar.

<comment><USER5> Especially when you consider that the first failure mode should be to _ask for clarification_ if you are unsure. It's completely unreasonable for them to claim they read the rules, and upon their singular reading of the rule, they _were completely sure_ that the limit was for individual participants and not whole teams. That there was not even any microscopic amount of doubt that maybe, just maybe, it was for whole teams.

This sounds like they are trying to cover their malicious intent with an "it's better to ask for forgiveness than permission" kind of trick.

</comment></comment></comment></comment><comment><USER4> Congratulations to the author for both of his last 2 posts making it to the HN front page!

This explains the need to drill home the train-test idea from the last post. I hadn't thought about this before but multiple submissions do amount to multiple peeks at your held-out test set, which is a huge ML no-no.

I don't know much about LSVRC, but doesn't the way Kaggle work prevent this? AFAIR you get a "public" test-score which is used for the leaderboards, but once the deadline for submissions is up, each submission is evaluated on a held-out test set giving you a "private" score. Now that I think about it, I'm not sure how that works, I guess the accuracy they show you as your public score is only on part of the submitted rows? Regardless of how that's done, could the LSVRC organisers not do something similar?

</comment><comment><USER10> Couldn't the LSVRC just limit the amount of submissions? Why would you rely on the competence or good intentions for this?

<comment><USER11> > Couldn't the LSVRC just limit the amount of submissions?

One of the article's notes indicates that it most likely does, and that the Baidu team wilfully got around that limit:

> Members of the Baidu team had to create multiple logins in order to > circumvent the "two submissions per week" rule

<comment><USER12> The fact that they created multiple logins was the biggest ethical breach here IMO. It was a willful violation of the contest rules.

The absolute number of submissions is somewhat arbitrary, e.g. why are 40 okay, but 200 are not?

<comment><USER11> There isn't an absolute limit on submissions, only a rate limit. The rate limit is probably somewhat arbitrary but it has a purpose: the aim of the challenge is to measure and compare algorithmic progress in machine learning, that's also why the test dataset is kept secret.

Without that, it becomes possible to pre-tune the algorithm to more closely match or better recognise the specific dataset ("hyperparameter optimisation") but not work any better in the general case, so the submitter does better on the specific challenge, but doesn't actually advance the field.

The limit is there to skew incentives towards algorithmic improvements, the specific rate doesn't really matter as long as it makes hyperparameter overfitting less convenient/efficient than algorithmic work.

</comment></comment></comment></comment><comment><USER13> > there are almost no papers focusing on 3 or 4-layer CNN's these days, for > example

What's the 'best practices' for the number of hidden layers in a CNN? 1 or 2 hidden layers?

<comment><USER14> Certainly more than 6. It depends on what you're going for. If you're just trying out a new technique (e.g. a new activation function, etc.) super-deep nets don't really matter. But if you're trying to get the best performance possible on a task (for a competition, say), you can tack on many more layers (MSR did ~150 layers).

</comment></comment><comment><USER15> So how would a better dataset look like? Does bigger always equal harder? What are the criteria to measure that?

And wouldn't video datasets be somewhat easier to analyse given the fact that you have multiple frames of the same object?

<comment><USER16> It is very difficult to measure data set quality. The goal of a data set to give you insight into how well an approach performs for any given data point (in this case an image and associated label). The problem is that we can not collect all the possible data points, so we have to settle for what we hope to be a fair sample of all data points.

While bigger usually means better, if for example there was a bias in how the data points were selected the data set can actually be worse than a smaller one where there was no such bias. Building a good data set really depends on the current understanding of the nature and difficulty of the task, both on the part of the scientific field and the researchers working on the data set.

The very same problem exists in the medical domain. How do you select the right patients to evaluate for a treatment? How do you know that there is not a specific genetic, racial, gender, etc. trait that leads to side effects? The answer is of course that you can not know with absolute certainty unless you include every human on the planet (and then there is the question as-of-yet unborn humans), but you can use your experience and medical knowledge to try to select patients (your data set) for the medical trial to minimise the possible impact of as-of-yet unknown side effects.

</comment></comment><comment><USER17> Nice article on why it is cheating (from a mathematical sense, independent of language) to get scores from the hold-out leader board too many times (plus some methods to mitigate the effect): [http://arxiv.org/abs/1502.04585](http://arxiv.org/abs/1502.04585) .

</comment><comment><USER18> As long as you make the public leaderboard set small, and the private one shot leaderboard set very large, the number of submissions matters very little in the final rankings. The only real issue is hand labeling the public leaderboard set to augment training data.

</comment><comment><USER19> Meh.. I'm not so sure about this.

Didn't people claim "cheating" back when the first compilers started doing data flow analysis too?

<comment><USER20> You might have a point if there was any sort of parallel between what they are doing and data flow analysis, but there really isn't.

The parallel would be more like adding a detector for certain benchmarks in your compiler, and outputting hand tuned assembly for that case ... except even worse than that, because you'd have to implement the detector and assembly generation in such a way that it made your compiler behave worse on general input.

<comment><USER21> I'm not sure if the analogy was intentional but your comment made me immediately think of the Volkswagen emissions scandal.

<comment><USER20> There is a parallel, but only to the first part. What VW did was cheat on benchmarks, much like certain driver vendors and compilers have been known to do.

But what we're talking about here is much, much worse from a design point of view. It specializing your system for the benchmark, in such a way that you are actually making it worse in general.</comment></comment></comment></comment>


Using AI to extract data from e-sports live streams - <AUTHOR> https://firstname.lastname@example.com/using-ai-to-extract-data-from-e-sports-live-streams-a10c3655ebd4 <comment><USER2> This was a great read. The step by step along with the videos make this super engaging even if this not my area of expertise.</comment>