Blekko donates search data to Common Crawl - <AUTHOR> http://blog.blekko.com/2012/12/17/common-crawl-donation/ <comment><USER2> The point worth note is that it's not an archive of downloaded pages, it's data for the Blekko equivalent of PageRank (i.e. computed relationships, not just the pages). To generate this independently would not only require access to a large crawl, but also robust code and most probably a large cluster to compute it in reasonable time, not to mention legal advice to avoid stepping on Google (et al) patents.

If only certain other companies weren't so precious about their publicly derived data. Fabulous donation.

<comment><USER3> > not to mention legal advice to avoid stepping on Google (et al) patents.

You may be right, you may not be right, but if this is the equivalent of PageRanked data then you may not be in the clear to use this as is. After all if 'PageRank' went into producing it buying it and using it makes you the beneficiary of patent infringement.

Personally I'd say so-su-mi, but I still think that it should be noted that the fact that someone else did the infringing does not put you automatically in the clear when using the end product.

[http://en.wikipedia.org/wiki/Patent_infringement_under_Unite...](http://en.wikipedia.org/wiki/Patent_infringement_under_United_States_law)

Section indirect infringement.

<comment><USER4> No, this is not the equivalent of PageRank. it could be an approximation however, and they are probably calling it a different name.

If that's not legal, then SEOMoz would've been sued a long long time ago (see OpenSiteExplorer, Page Authority, Domain Authority, etc)

<comment><USER5> blekko doesn't compute PageRank, and we don't compute anything similar to it, either. It's highly gamed and less useful than you might think. (The academic equivalent of PageRank for research papers is highly gamed, too, by citation clubs...)

By the way, the original PageRank patent is owned and licensed by Stanford University, not by Google.

<comment><USER2> Can you tell us a little more about what the 'ranking metadata' is, as there's not much to go on from the announcement. It's also not clear whether the data is available only for Common Crawl's operational purposes, or whether it's intended to become an integral part of the public data set.

<comment><USER5> The ranking metadata consists of: domain ranks, url ranks, and booleans for whether blekko considers the domain or url to be webspam or porn. This list will expand in the future.

The data is currently available for Common Crawl's operational purposes, and is eventually going to be part of Common Crawl's public dataset. We're currently ironing out a useful format for making it efficiently accessible, compatible with some other metadata which Common Crawl is planning on making available.

</comment></comment></comment></comment></comment></comment><comment><USER6> This is great, but I'm confused by the part about avoiding porn:

> _Common Crawl will use blekko's metadata to improve its crawl quality, while > avoiding webspam, porn, and the influence of excessive SEO (search engine > optimization)_

Why avoid porn? Millions of people deliberately search for porn on the internet every day. It's hardly less worthy of crawling than any other content. The next sentence goes on to suggest that porn is not "useful to humans", which is obviously false.

If Common Crawl is indeed filtering out content they determine to be pornographic, I hope they are taking care not to also remove information on sexual and relationship health and LGBT rights, which are often collateral damage of porn-blocking systems. And it would be nice to see an open acknowledgement that filtering is going on - I couldn't find any references to this at commoncrawl.org.

<comment><USER7> I work at Blekko and am the primary engineer working on our porn tagger. We include LGBT, reproductive/sexual health, breast cancer, bands like "Pussycat Riot," etc in our training set to make sure these sites do not get hidden from our search results.

We do not have anything against porn. However, when people are not searching for porn, showing them porn results makes for a bad search experience. So identifying porn, and only showing porn on relevant porn results is vitally important to search quality.

<comment><USER3> So tag it but include it.

Your answer is a bit at odds with <http://news.ycombinator.com/item?id=4933437>

<comment><USER8> Jacques, the porn is there, its just identified as such. Whether or not it is included in results is a function on the query.

One of the funny things about language is that there is always a 'pun' or an innuendo which can trigger a hit on a porn site, however if most of what you're looking for isn't porn then the web site has to assume you are _not_ looking for porn and avoid some NSFW link from surfacing into your search results. You could always explicitly ask for it with /porn but then that is a clear signal of what you are looking for.

Part of the crawl data includes an indication as to whether or not the ranker thought the document was 'porn' or 'not porn', so if you're selecting things to return you can ignore that bit, mixing porn with non-porn when someone searches for 'beavers' you get a wider variety of results than you would if you were assuming you meant the furry critters which chew on trees or sports teams and limiting results to those documents.

<comment><USER3> That's actually really useful.

Having it there but tagged is halfway towards being able to use it to filter them out. Not having it means that when you merge it with another set that you're not going to be able to remove the porn.

And it also allows you to use it as a training set for classifiers.

<comment><USER8> "And it also allows you to use it as a training set for classifiers."

One could imagine a project on Common Crawl which auto-generated a list of slang terms for porny things by creating a list of n-grams from the words used in documents tagged as porn.

</comment></comment></comment></comment></comment></comment><comment><USER9> This is going to be invaluable for information retrieval researchers.

Google, MSR, and Yahoo! have an edge on research over universities because of the large amount of data they collect from the users; all the other institutions are left with either small-size benchmark datasets or synthetic data, which are usually not representative of the actual usage scenarios. I myself had to synthetize a query log from the Wikipedia request logs to test some of my data structures on large-scale data.

I expect to see a huge number of papers which will use these data in their experiments in the immediate future. Thanks, Blekko!

<comment><USER10> As far as I can tell, this contribution from Blekko doesn't have any user data/queries in it.

As far as I can tell, this is the best resource for publicly available search engine query logs.

<http://jeffhuang.com/search_query_logs.html>

I don't intend to downplay the contribution, having a large collection of spam/porn classified web docs is still a very nice thing to have for researchers.

<comment><USER5> This is our first donation. We have a lot more we plan on giving, but for user queries, for example, the privacy issues are a lot more difficult to work through. We have no interest in being the next privacy scandal.

</comment></comment></comment><comment><USER11> I am part of Common Crawl and I just wanted to say that we are super excited about blekko's donation! This is yet another demonstration how much blekko values openness and transparency.

</comment><comment><USER5> If you'd like to see some examples of what you can do with Common Crawl data, here are the winning projects from a code contest held last September:

[http://commoncrawl.org/announcing-the-winners-of-the-code- co...](http://commoncrawl.org/announcing-the-winners-of-the-code-contest/)

Some code libraries for using Common Crawl data:

<https://github.com/commoncrawl/>

Some clues for getting started:

<http://commoncrawl.org/get-started/>

<comment><USER12> This is a pretty cool thing that you're doing, Greg.

</comment></comment><comment><USER13> Last I checked Blekko was pretty good, better than DuckDuckGo in search relevancy I thought, but that was like a year ago.

<comment><USER14> Does Blekko also NOT track you, like DuckDuckGo ?

<comment><USER15> The terms seem pretty reasonable, which means they are not as extreme as DDG's. <https://blekko.com/about/privacy-policy>

</comment></comment></comment><comment><USER16> I was at the open data meetup in Mozilla where Common Crawl presented and Blekko's CTO was present. Little did I know that great stuff like this was in the making.

</comment><comment><USER17> Common Crawl has a really neat mission, as there isn't a whole lot of free and open data out in the world right now and they're trying to change that. With this donation it looks like their commons will be augmented with some great stuff and that can only mean awesome things.

</comment><comment><USER18> I've tried Blekko a couple of times, and thought they were okay, but I don't find them so much better than Google to make me switch.

I'm curious, how Blekko can stay in business? Do they get enough traffic and revenue from ads, etc, to maintain some sort of positive cash flow, or are they simply burning through cash from investors?

<comment><USER19> For me, blekko doesn't have to be "so much better" to make me use it over Google, it just has to be just as good or slightly better because I respect and support blekko's philosophy. If two products were of equal quality, wouldn't you rather use the one made by a company that shares your values?

<comment><USER11> Strongly agree! blekko's Bill of Rights is a great expression of their values and of why we should all be using blekko.

blekko Bill of Rights

1\. Search shall be open

2\. Search results shall involve people

3\. Ranking data shall not be kept secret

4\. Web data shall be readily available

5\. There is no one-size-fits-all for search

6\. Advanced search shall be accessible

7\. Search engine tools shall be open to all

8\. Search & community go hand-in-hand

9\. Spam does not belong in search results

10\. Privacy of searchers shall not be violated

<comment><USER20> Although to be fair, you have to pay to get the SEO/Ranking data now. I'm cool with that if it keeps the service open but it should be pointed out.

On a related note www.procog.com has a totally open algorithm.

</comment></comment></comment></comment><comment><USER21> I wondered how Google's foothold in search would ever be overcome. I think this might just be the start of something.

<comment><USER22> Google has had years to tweak and tune things. I think it is practically impossible for anyone to match them, let alone surpass them in index quality.

Google needs to open its index and create a search market place. There are millions of domain/location specific apps that can be built around that index.</comment></comment>


How the Government Killed a Secure E-mail Company - <AUTHOR> http://www.newyorker.com/online/blogs/elements/2013/08/the-government-versus-your-secrets.html <comment><USER2> " _If Lavabit lost its appeal to the F.I.S.C.R., and still refused to coöperate, it would run a serious risk of being found in contempt; that's how most courts punish those who disobey its orders. The FISA court is no different. According to the court's rules of procedure, a party may be held in contempt for defying its orders. The secret court may consider many punishments—secret fines for each day of noncompliance, or even secret jail time for executives. The idea behind civil contempt is that "you hold the key to your own cell." If you comply, the punishment stops. But hold out long enough and your contempt may be criminal, and your compliance will not end the jail sentence or displace the fine._ "

Any claims the FISA court is not a real court fail hard if the above is correct. Which is pretty much has to be for the current system to work.

<comment><USER3> Secret courts

extraordinary rendition

waterboarding

indefinite detention in a location beyond the reach of the judicial system

shoot to kill via drone

spying on the populace on a scale that the stasi could only dream of.

Spying on allies as well as enemies

What kind of a regime is that?

<comment><USER4> USA -> USSR. US govt has decided to follow practices of USSR govt as it considers them to be much more superior than the former USA govt practices.

</comment></comment></comment><comment><USER5> The only way I can imagine secret jail time working is by publicly charging that person with a crime they didn't commit to cover up the actual reason. This is similar to what is alleged with the CEO of Qwest, who would not cooperate with the NSA and was then charged with insider trading and is still in prison.

However, what possible reason would the person have to remain silent about the real reason they're in prison once they're already in prison?

<comment><USER2> " _However, what possible reason would the person have to remain silent about the real reason they 're in prison once they're already in prison_"

Look at Qwest CEO Nacchio's case. He claimed exactly that, see e.g. [http://dailycaller.com/2013/06/13/jailed-qwest-ceo- claimed-t...](http://dailycaller.com/2013/06/13/jailed-qwest-ceo-claimed-that- nsa-retaliated-because-he-wouldnt-participate-in-spy-program/) and what happened?

He's jailed for 6 years, I'm sure most people believe it's because he's eeevil capitalist scum, and whatever the truth, everyone else in his position has to take it as likely they'd suffer the same fate.

Now that this seems to have turned a corner it _might_ play out differently, but "Do you feel lucky, punk?"

</comment></comment><comment><USER6> Also re secret jail time: [http://www.rcfp.org/browse-media-law- resources/news-media-la...](http://www.rcfp.org/browse-media-law- resources/news-media-law/news-media-and-law-winter-2012/silencing-stories)

<comment><USER7> >allows the BOP to monitor client-attorney communications.

here goes another one down the drain.

</comment></comment><comment><USER8> _When in the Course of human events, it becomes necessary for one people to dissolve the political bands which have connected them with another, and to assume among the powers of the earth, the separate and equal station to which the Laws of Nature and of Nature 's God entitle them, a decent respect to the opinions of mankind requires that they should declare the causes which impel them to the separation.

We hold these truths to be self-evident, that all men are created equal, that they are endowed by their Creator with certain unalienable Rights, that among these are Life, Liberty and the pursuit of Happiness.--That TO SECURE THESE RIGHTS, GOVERNMENTS ARE INSTITUTED AMONG MEN, DERIVING THEIR JUST POWERS FROM THE CONSENT OF THE GOVERNED, --THAT WHENEVER ANY FORM OF GOVERNMENT BECOMES DESTRUCTIVE OF THESE ENDS, IT IS THE RIGHT OF THE PEOPLE TO ALTER OR TO ABOLISH IT, AND TO INSTITUTE NEW GOVERNMENT, LAYING ITS FOUNDATION ON SUCH PRINCIPLES AND ORGANIZING ITS POWERS IN SUCH FORM, AS TO THEM SHALL SEEM MOST LIKELY TO EFFECT THEIR SAFETY AND HAPPINESS. Prudence, indeed, will dictate that Governments long established should not be changed for light and transient causes; and accordingly all experience hath shewn, that mankind are more disposed to suffer, while evils are sufferable, than to right themselves by abolishing the forms to which they are accustomed. But when a long train of abuses and usurpations, pursuing invariably the same Object evinces a design to reduce them under absolute Despotism, it is their right, it is their duty, to throw off such Government, and to provide new Guards for their future security.--Such has been the patient sufferance of these Colonies; and such is now the necessity which constrains them to alter their former Systems of Government. The history of the present King of Great Britain is a history of repeated injuries and usurpations, all having in direct object the establishment of an absolute Tyranny over these States. To prove this, let Facts be submitted to a candid world._

(emphasis mine)

[http://en.wikipedia.org/wiki/United_States_Declaration_of_In...](http://en.wikipedia.org/wiki/United_States_Declaration_of_Independence)

Do yourself a favor and take a few minutes to read the full document. Now, if you really care, play with it a little, modify the Indictment to reflect current events and relevant grievances.

Frankly, it scares me. This is a 237 year old document that feels like we might consider writing it almost verbatim today. I have to say that I have never lived in fear of what my country might become. Now, with the actions of this and the prior administration I really don't know what to think. Obama and his regime have accelerated our descent into something indistinguishable from what I remember as the country I genuinely love with the deepest corners of my heart and mind. It really saddens me to the core in a manner that is hard to express with words.

This is now the domain of tears, not words.

Who's fault it this? It's that of the political fundamentalists. Those among us who choose not to think critically and, instead, vote like sheep, year after year, always supporting their ridiculous parties, Democrats and Republicans either out of self-interest or ignorance. This is on you and nobody else.

You ought to be ashamed. All of you. You bunch of assholes. You now have a front seat from which to enjoy the destruction of our Nation from the inside out. This is almost like a slow motion train wreck happening right in front of our collective eyes. Will you do anything to help fix it or will you sit quietly watching the train wreck? Will you break ranks and vote the morons out of office in a united show of political force by the people and for the people. Will you hold all those you supported accountable for their lies and actions? You probably wont. You will probably continue to vote like sheep, supporting your petty fucking causes, missing the forest for the trees.

Fucking cowards.

<comment><USER9> Then realize it was written by a bunch of Aristocratic slave-holding white men, who implicitly meant for the document to apply only to people like themselves.

What the traditionally disenfranchised are thinking is: welcome to the fucking club.

<comment><USER10> Does that invalidate the arguments put forth in the document? Did it not result in greater freedom for all and eventually lead to massive prosperity?

<comment><USER9> It definitely inspired people to resolve its core hypocrisy, which led to positive effects.

</comment></comment></comment></comment><comment><USER11> I'd say that all this gov secrecy is right in the basis of the USA, from its start. Just remember that almost all founding fathers were members of the free masonry secret society.

\-- Oh, he is on conspiracy theory now! Let's ignore him.

</comment><comment><USER12> How is Mega not all over this?

I feel like this would be Kim Dot Com's favorite thing in the world to do (sticking his finger in America's legal eye). Why haven't they rolled out a privacy focused email service. Their tagline is the "the privacy company."

</comment><comment><USER13> This now gives an edge to all the big internet companies that offer messaging services that aren't based on email.

</comment><comment><USER14> Completely aside from the main point, but I am curious as to why the New Yorker chose to use umlauts for all of the instances of the word: "coöperate." It's a latin word so I wouldn't expect to see that.

I wonder if it is a rendering trick or something intentional for tracking.

<comment><USER15> It's called a diaeresis. It marks the start of a new syllable. They have an article discussing its history at the magazine.

[http://www.newyorker.com/online/blogs/culture/2012/04/the- cu...](http://www.newyorker.com/online/blogs/culture/2012/04/the-curse-of-the- diaeresis.html)

<comment><USER14> Ah thanks! Never knew about that.

</comment></comment></comment><comment><USER16> Didn't Lavabit shutdown voluntarily?

<comment><USER17> In self-defense, and the pre-emptive defense of others</comment></comment>


Python insights - <AUTHOR> http://eli.thegreenplace.net/programs-and-code/python-insights/ <comment><USER2> Here's one I only found out about recently.

Multiple assignment in Python is already well known.

      x, y = 10, 50

But this also works when the left or right variable lists are sequences.

      [x,y] = 10,4

      t = (1,2,3)

      a,b,c = t

If you have a loop or a comprehension, and the values are sequences, you can access them like so

      for (left, right) in zip(leftArray,rightArray):
          ...

You can also do variable assignment inside an argument list. To see what I mean, suppose you're writing a 2d geometry library, and you use tuples as points. Ordinarily you might write

      def func(point):
          x, y = point[0], point[1]
          ...

or even

      def func(point):
          x, y = point
          ...

Instead, you can just write

      def func((x,y)):
          ...

This has many advantages. It gets around the problem of having to navigate to some other place to find out what composite data structures _are_. Points are just (x,y) tuples. It makes it easier to write sensible names, in many cases, which is crucial in a dynamic language. It's shorter. You spend less time packing, unpacking, and generally, schlepping your data, and more time _actually doing things._

Unfortunately, it's not the most generic of methods. It fixes you to a particular data representation. This is effective in certain cases, but is otherwise a problem when you need to change things. In practice, though, this can be avoided, and other code would break during such changes too.

<comment><USER3> Interesting. It seems that (according to my Python prompt)

      x, y = 1, 2

and

      [x, y] = 1, 2

and even

      x, y = [1, 2]

are all identical in function. Are there any (subtle) distinctions between the three?

<comment><USER4> The last is different in that it returns a list, not a tuple. This makes a difference if you want to change the object.

      >>> tuple = x, y = 1, 2
      >>> type(tuple)
      <type 'tuple'>
      >>> tuple
      (1, 2)
      >>> tuple[0] = 1
      Traceback (most recent call last):
        File "<stdin>", line 1, in ?
      TypeError: object doesn't support item assignment
      >>> lst = x, y = [1, 2]
      >>> type(lst)
      <type 'list'>
      >>> lst
      [1, 2]
      >>> lst[0] = 3
      >>> lst
      [3, 2]

<comment><USER3> Aha! There's actually no difference (i think) between:

1, 2

and:

(1, 2)

That seems to make sense.

<comment><USER4> I think that is true, but I think that it's only later versions of Python (2.5+, maybe?) that allow you to omit the parenthesis. I've had earlier versions (2.3?) complain of a syntax error when I leave them out.

</comment></comment></comment></comment></comment><comment><USER5> From the article:

 _String reverse

Python doesn't have a built-in reverse method for strings. Luckily, this can be easily done with slices:

      def reverse(str):
          return str[::-1]

_

If you don't like str[::-1] (even though it is a common idiom), you could just use

      "".join(reversed(str))

Is reversing a string really common enough to have a special function for it with such a general name like reverse?

<comment><USER6> There is a list-reverse that changes the list and returns None but maybe the reason there isn't one for strings are because they are immutable while lists are mutable.

<comment><USER5> That's a good point. I was thinking more along the lines, though, that there are some times where you actually need to reverse a list. How often does it make sense to reverse a string? When palindrome testing?</comment></comment></comment>


Cache Poisoned DoS Attack: Shutdown any CDN Website with One HTTP Request - <AUTHOR> https://cpdos.org/ <comment><USER2> So after the authors disclosed this issue to AWS it was fixed and CloudFront no longer caches 400 Bad Request by default, also from the paper linked on the website [0]:

""" Amazon Web Services (AWS). We reported this issue to the AWSSecurity team. They confirmed the vulnerabilities on CloudFront. The AWS-Security team stopped caching error pages with the status code 400 Bad Request by default. However, they took over three months to fix our CPDoS reportings. Unfortunately, the overall disclosure process was characterized by a one-way communication. We periodically asked for the current state, without getting much information back from the AWS-Security team. They never contacted us to keep us up to date with the current process.

"""

[0] - [https://cpdos.org/paper/Your_Cache_Has_Fallen__Cache_Poisone...](https://cpdos.org/paper/Your_Cache_Has_Fallen__Cache_Poisoned_Denial_of_Service_Attack__Preprint_.pdf)

</comment><comment><USER3> It seems like the unstated larger problem here (in the blog article at least, I haven't read the paper) is that cached pages are served that do not match the full HTTP request. The problem isn't just that error pages are being cached, but that certain HTTP headers that should change the resulting page from the server are being ignored when it comes to determining identical requests. Even if no one is maliciously using this feature to cause error pages to be paged, the cache is still breaking the website if it causes a different page to appear than would have been served directly from the server!

<comment><USER4> Actually, the origin is supposed to send a `Vary` header if it changes behavior based on any header.

So, if a client sends a 20kb `X-Oversized-Header`, when the server responds with a 400 -- it might be conceivable that it should include `Vary: X-Oversized-Header`.

Is that "really" the right fix? Probably not. But the HTTP RFCs provide `Vary` for exact this kind of reason within HTTP caching: the origin is varying its response based on a subset of headers.

<comment><USER5> In RFC-world (which may not be the same as the real world), there's no need for `Vary` on a 400 response, because a 400 response isn't cacheable (unless it has a cache-control or expires header).

<comment><USER6> I feel like HTTP response codes are an attempt to squish several layers of Result monads together into one single many-valued one, where each layer of the original nested set of Results has different caching semantics.

Like, in this case, a 400 is really the origin is saying that it's not even _sending_ you a representation of the resource you requested, because you didn't make a request that can be parsed as any particular resource. It's a Left RequestFormatError instead of a Right CachableResourceRepresentation.

And, annoyingly, the codes don't have any line-up with which layer of the result went bad. 4XX is "client error", sure; but 404 isn't really an "error" at all, but an eminently cacheable representation of the representation of the non-existence of a resource.

It'd be neat to see the HTTP codes rearranged into layers by what caching semantics they require of the implementing UA, such that UAs could just attach behavior to status-code _ranges_. Maybe in HTTP/4?

<comment><USER7> Well, HTTP was probably not designed for doing well with caches. You certainly know that the first digit of a HTTP response code tells about some grouping, where jt is

        4xx (Client Error): The request contains bad syntax or cannot be fulfilled

I would state that, since this is client-specific, all 4xx responses should not be cached at some proxy/CDN, since he is not the client. And even the client should not cache a 404. A ressource could just be created the next moment.

<comment><USER8> Could be, but there's a reason to want cachable, proxyable errors here: they're often very expensive for the origin.

<comment><USER7> Actualy, handling an error should be cheaper then handling a proper request. Just because an error most likely means an early exit of the handling server -- which means less time-to-answer, i.e. cheaper. (This does not cover any kind of DoS attack, which is _always_ difficult to handle, regardless of an error or non-error answer)

However, effectively we agree with _derefr_ , saying that HTTP status code design did not have this pecularity of cachable vs. non-cachable errors in mind. This is definetly a shortcoming.

</comment></comment></comment></comment></comment></comment></comment><comment><USER9> _Many intermediate systems such as proxies, load balancers, caches, and firewalls, however, do only support GET and POST. This means that HTTP requests with DELETE and PUT are simply blocked. To circumvent this restriction many REST-based APIs or web frameworks such as the Play Framework 1, provide headers such as X-HTTP-Method-Override, X-HTTP-Method or X-Method- Override for tunnel blocked HTTP methods_

This is so f __*ing scary. Who in their right mind invents such crazy tricks that absolutely circumvent all we know about web API security, implements them in frameworks, and leaves them enabled by default?

<comment><USER10> People who have to deal with end users behind overly restrictive corporate firewall. After you get a few hundred bad reviews telling you your website is broken, it looks very tempting to just fix it on your end.

<comment><USER11> Maybe don't fix it and just accept that organization X doesn't want it's users to access your resource?

Seems more reasonable than trying circumvent a legitimate restriction. Not every block is an adversary which needs defeating.

<comment><USER12> What is the legitimate restriction? You'll let "users" POST but not PUT? In what Mordacian fever-dream is that reasonable?

<comment><USER13> The PUT and DELETE methods can't be used with HTML forms.

[https://softwareengineering.stackexchange.com/q/114156](https://softwareengineering.stackexchange.com/q/114156)

<comment><USER12> Forms aren't the only sources of HTTP requests.

</comment></comment></comment></comment></comment></comment><comment><USER14> It's pretty surprising to see an issue like this. I've spent some time in the past tuning http caches/cdns. One key takeaway I recall is the importance of your "cache key". That is, which fields matter when deciding if requests match. If the cache key of two requests match, the requests are considered identical and thus the desired behavior would be to serve from cache.

Obviously, things like the request method, path and Host header matter a lot. Perhaps if you're A/B testing, the A/B cookie would make sense as part of the cache key too.

This seems like a simple misconfiguration at a very critical location. But an 'exploit' that warrants it's own domain? Hardly. This is a promotion for the authors and their upcoming presentation. It's a very nice gotcha.

<comment><USER15> This is a very easy mistake to make: you configure your cache keys in one place, and process requests in another. With nothing linking them it's not at all surprising for them to be out of sync.

As for whether it requires it's on domain, domains are cheap and publicity gets people to pay attention to problems and fix them.

</comment></comment><comment><USER16> A sane CDN doesn't cache error responses. There is no legitimate reason to cache a non 2xx/3xx response, unless you / your CDN is really pinching pennies.

Caching a 5xx kinda makes sense I guess, but a 4xx client error? That's nuts.

<comment><USER17> Caching a 404 can make sense, however.

For example, every browser requests /favicon.ico by convention. If you don't have one, you wouldn't want every single request reaching your backend.

<comment><USER16> Meh, it's subjective but I'd argue that's a bad idea. A 404 isn't exactly a heavy request; your origin should be able to handle plenty of those. The whole point of CDNs is to serve fat assets (images, videos, js libraries sometimes) not patch holes in your site or bandaid your raspberry-pi origin. Caching error responses, including 404s, can lead to all sorts of trouble and isn't worth it IMO.

<comment><USER18> Actually, generating heavy to process 404s is a fairly common attack, for some kinds of site.

<comment><USER19> Could you maybe provide a practical example of such a "heavy to process" 404? I'm having difficulty to image how a "Not Found" could (or ever should) involve heavy processing. AFAIK, a 404 should be only given for a non-existing resource (e.g. file). That should be straightforward enough. Granted, I've seen some horrible semantic resource pointers/paths in URI's over the years, some of which requiring processing, and some of them generating 404s. However, such contortions are mostly just a testament to horribly bad design. If the request involves processing on the server side (higher up than the web server or cache itself), resulting in a conclusion that a resource is unavailable, should that not return a 5xx response?

<comment><USER20> The origin server might be set up to serve everything through cgi. That would mean loading the entire framework just to spit out a 404 for the favicon. If all the other pages cache just fine, you could hit 100% load on favicons when you'd otherwise be at 10%.

It's not an ideal design but I wouldn't call it 'horrible'.

<comment><USER21> What is CGI in this context? I've only heard of it in a movie special effects context.

<comment><USER20> The web server having to run an external program to handle the request, typically because it's written in a scripting language.

Technically it stands for "common gateway interface" but that doesn't come up very much.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER22> CDNs could whitelist HTTP header keys, constrain HTTP header values, and regenerate a full clean header to the web server as part of their security services.

CDNs already parse headers, and the CDN developers have the knowledge and know how to correctly constrain header values.

There are 100's of different HTTP Header attacks e.g. I like this one that returns a response from a different persons HTTP response into your HTTP response: [https://portswigger.net/blog/http-desync-attacks-request- smu...](https://portswigger.net/blog/http-desync-attacks-request-smuggling- reborn)

<comment><USER23> Well this is the thing I'm confused about, having worked on some cloudfront stuff last month, documents like [https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope...](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/header- caching.html) gave me the clear impression that every header cloudfront passes to the origin will be considered a cache key. In particular if you configure it to "forward all headers", which is what would surely be needed to pass this custom "X-Oversized-Header" onwards, it would effectively disable all caching.

I'm clearly going to have to spend another day staring at the cloudfront documentation.

<comment><USER21> And why weren't all headers considered in the cache key to begin with? The key computation will never not be O(1).

Only thing I can think of is too many cache entries with the same cache values for the case where requests with different headers produce the same API response.

Which could lead to a large cache.

</comment></comment></comment><comment><USER24> Why would you have a CDN configured to cache error responses from the origin? It shouldn't serve those at all.

It's also a good idea to whitelist headers and query params your app uses.

Better put this sounds like an attack that only works on very poorly configured CDNs.

<comment><USER14> A main purpose of a cdn is to protect your resources by using caching. Error pages still require server-side resources to create. Not caching error pages gives attackers an easy way to consume your server's resources a la "DoS".

<comment><USER24> That is if a hacker can find an endpoint which is erroring, which they shouldn't be able to if you've configured your cdn not to serve errors from the origin in the first place. There's obviously other good reasons not to serve error messages out to the public.

Anyway, if you're an attacker it's easier to DDOS against urls that will 404 as those must go back origin anyway typically and likely won't be safe to cache for long even if the cdn is config'd to do so for 4xx responses. To protect against that your cdn provider probably has some sort of ddos protection feature as well though.

<comment><USER25> If the CDN doesnt serve an error, what pray tell should it return when the request triggers an error on the origin?

<comment><USER24> Depends what kind of error you're referring to but if it's a 5xx cdns can be setup to serve a static html. That html can contain as much or as little info about the error (including no info if you think the endpoint is likely to come under attack the page could be disguised as something legit).

Of course you should be monitoring for and logging these error responses from the origin and fixing them as soon as possible. The cdn response is just to provide cover. Again, that is if you need or want it. If you want to expose errors to the public go right ahead nobody is going to stop you.

</comment></comment></comment></comment></comment><comment><USER26> In the text it points out that adhering to the standard regarding what may be cached - i.e. _not_ most kinds of error responses - is a key mitigation, so it's pretty noteworthy that Amazon CloudFront™ seems to be by far the most afflicted CDN type. Also nice to see that Good Old Squid fares exceptionally well.

</comment><comment><USER27> Maybe I'm missing something, but is the exploit somehow generating a cache miss? Otherwise everything that's already in cache isn't vulnerable to this right? Not that it makes it in any way less scary, but slightly more complicated at least..

<comment><USER28> Yeah, that's part of it. The attacker wants to hit a url that isn't already cached by CDN. That might be easy or hard depending on the site. Like if the CDN just has a 30 minute TTL, the attacker will need to be the first request right after the 30 minutes expires.

</comment></comment><comment><USER29> Why would a CDN cache a 400?

<comment><USER30> A CDN that doesn't follow HTTP standards

"One of the main reasons for HHO and HMC CPDoS attacks lies in the fact that a vulnerable cache illicitly stores responses containing error codes such as 400 Bad Request by default. This is not allowed according to the HTTP standard."

<comment><USER26> It seems that it should be feasible to cache more kinds of errors if the request that populated the cache and the subsequent request are identical. These attacks all rely on that not being the case. However, "identity" is a more slippery concept than most might think. Generally it requires putting requests into some canonical form, but defining that canonical form (especially what it excludes) requires making exactly the same kinds of distinctions that were missed to make these attacks possible. It just shifts the problem around, and introduces new potential for breakage. In the end it's no better than just following the darn standard, whose authors probably defined what was cacheable with exactly these concerns in mind.

</comment></comment></comment><comment><USER31> Is that really a legitimate attack vector?

I would expect web servers to just discard weird headers and serve the regular page. And, at the same time, I would also expect CDN to not cache 400 bad requests or 500 server errors.

<comment><USER32> Yes, it is. Many servers/proxies/caches/ad-hoc-header-filters configurations, if they're not handle properly, don't expect that some clients will tamper their headers or people who configure them don't even take them into account, so servers/proxies/caches/ad-hoc-header-filters don't discard them (better approach is to use white-lists but it's not the panacea). CDNs can cache or not, but I have seen a variety of behaviors even for the same CDN, so again I suspect many times this could be caused by misconfiguration.

</comment></comment><comment><USER33> I can think of only rare occasions where any status other than 200 is cached. Usually this is an edge case that has to be configured in the caching tier.

<comment><USER34> Not caching your error pages is a good way to bring down your servers once something goes wrong. We usually have a short cache for errors to make sure there is an upper limit to the amount of requests coming through the cache.

</comment></comment><comment><USER35> One more mitigation idea: set non-trivial expiration times and "prime" your caches. When you update a resource, hit all your CDN endpoints (at least ones which don't follow spec and are known to be exploitable) to force them to download and cache the good copy. Bake it into your CI deployment process.

</comment><comment><USER36> I usually do not let any header through the CDNs. I need to try this attack out on some of our infra, not sure if any of those are affected if you have tight header policy.

</comment><comment><USER37> The title is hyperbolic but the article is clear and well researched. Issue seems to affect Cloud Front more than other CDNs - from the matrix provided</comment>


Declare the Variables First - <AUTHOR> http://blog.acthompson.net/2012/12/declare-variables-first.html

<comment><USER2> My CS teacher also used the gezinta/gezouta box example, even though we were on the West Coast. I didn't understand that he meant goes into, goes out of for a while!

<comment><AUTHOR> The teacher may have spent some time in New York. :-)</comment></comment>


Fishing for graphs in a Hadoop data lake - <AUTHOR> https://www.oreilly.com/ideas/fishing-for-graphs-in-a-hadoop-data-lake <comment><USER2> The article is a nice summary. I think the author missed a key argument for his multi-model case, AWS costs. Short queries via Spark/Hadoop will cost more on AWS than a focused graph model on dedicated graph DB / multi-model DB on AWS.

</comment><comment><USER3> Thanks for the article. Seems like a good appeoach to combine the strenghts of both graph & Hadoop. Wonder which other, in addtion to the described ones, use cases could be suitable here.

Anyone an idea?

<comment><USER4> We get good visibility into what folks do in practice based on their use of Graphistry: we're a DB-agnostic scalable visual graph analytics environment, so we've been seeing (& assisting) what analysts do standalone / what developers build / what data scientists do from notebooks.

1\. Most graphs are small (< 100M nodes & edges, probably even < 1M). So analysts just load a CSV directly into us or dump into pandas and work from there. Most Graphistry users do this. It became so common that we baked in a transform to our library that shortcuts the data wrangling problem of SQL/CSV records -> node table + edge table via our "hypergraph" transform.

2\. Sometimes the data is too big or they want to use a query language they're more comfortable with vs. Pandas. We'll see a bunch of SQL (incl. Spark), Splunk, Elastic, etc. when approach #1 isn't enough. No need for Neo4j/Titan/GraphX for that problem. If they end up doing this a lot, Neo4j ends up being a sensible choice because of the ergonomics of the Cypher query language.

3\. Sometimes, graph queries or analytics _are_ technically critical. We'll see mostly analytics via use of NetworkX or maybe iGraph, such as for slightly better community detection, or something smarter than degrees for node sizes. Sometimes we'll see query langs, probably Neo4j because (I'm guessing) the database is packaged accessibly. For ergonomic reasons, I've been expecting the efforts around OpenCypher for Spark will eventually supplant GraphX for the exploratory case, and we'll start seeing more Janus as it gains more steam.

4\. Even more occasionally, people are building true graph algorithms that cannot be sufficiently approximated with their existing tools. E.g., we're seeing a bunch more in the knowledge graph space (ex: finance), and in security/fraud, we're seeing the bigger enterprises needing the same for correlation work. This gets into powering latency-sensitive ML / detection algorithms, fast analyst experiences, etc. However, stuff like regular SQL & Splunk & Spark still gets _most_ teams mostly there with great scaleout etc., so there's a bit of a problem/time/budget/expertise thing going on.

We've been happy to support all these kinds of projects at Graphistry -- and are often part of the entry into them -- so always happy to chat about it. Likewise, I'm not listing work by good teams like those at Datastax Graph, Blazegraph, and Amazon Neptune -- we see them, just they're used more in specific enterprise/federal scenarios.

<comment><USER5> Author of posted article here: thanks for the additional pointers. It seems that graphistry excels at visualization. Essentially, your offering confirms the main story of the article: make more out of your (graph) data by extracting it from Hadoop to a different tool.

And obviously, one should use the right tool for the purpose . I think graphistry is a good choice for graph visualization, graph databases like ArangoDB or Neo4j will be good at ad hoc traversals. And multi-model databases like ArangoDB or OrientDB will be good at a wide range of ad hoc queries. Anyway, thanks again for the pointers.

<comment><USER4> Yep. Maybe the observation is (1) data has gravity -- it was originally in another non-graph-specific DB -- and (2) the graph structure part is normally small. So we indeed see a lot of extraction into easier-to-use systems.

The nuance being... with stuff like data science notebooks and pandas, the people skilled enough to do extraction are also skilled enough that it's easier to just use pandas. The exception is repeat work or when it is for regular analysts. Friendly query languages like Neo4j's Cypher helps there. Not sure what Arango supports... Gremlin? Proprietary?

Graphistry's environment is agnostic, and _not_ a database, so it'd be wrong of me to advocate teams drop their system of record and use just us ;-) We ended up building a visual "playbook" investigation environment to help teams streamline these scenarios. They run visual playbooks against their legacy db (splunk, elastic, sql, ...) for faux-graph queries, or their new graph db for deeper ones (e.g., path queries). So we're more of the system of record + superpowers for your investigations, kind of like a smarter version of what Tableau/Looker do for SQL.</comment></comment></comment></comment>


Some US scientists consider moving overseas to continue their research - <AUTHOR> http://www.forbes.com/sites/janetraedupree/2013/09/25/us-scientists-are-leaving-the-country-and-taking-the-innovation-economy-with-them/

<comment><USER2> The writing on the wall became apparent to me with the 1993 cancellation of the Superconducting Super Collider, when I was getting my PhD in physics. I didn't know the numbers for a cost-benefit analysis, but I couldn't then nor can I now see cancellation as helpful to the U.S. The U.S. would have stayed way ahead of the rest of the world in particle physics and all the accompanying jobs, discoveries, and innovation. Instead we look to the rest of the world for leadership.

The Forbes article paints a picture of a United States full of ambition and curiosity driving innovation. That part exists, but it has just as long been a country of puritanism, self-righteousness, and fundamentalism.

The article points to politics as a major problem. That politics reflects this tradition among the people. A large fraction of the population here genuinely dislikes science, no matter how much they like the computers and cd players it brought.

I grew up liking idealism because I thought I was right about so many things. I since dropped my need to be right a priori in favor of _what works_ after testing. Politics in this country does have an idealistic view of the individual -- each person should do for him or herself and anyone helping them will decrease their ability to achieve and lower their motivation. Healthcare, welfare, education, and so on are viewed by a large block of voters as detrimental to the nation. We, as a nation if not readers of this site, stick to this principle in the face of nearly the entire world providing evidence to the contrary.

We put down other countries for doing things against our ideals, saying they don't work, even as they pass us in countless measures of quality of life and community.

Choosing idealism in the face of contrary evidence of what works is a recipe for disaster.

<comment> T-A "jobs, dicoveries, and innovation" from particle physics? Really?

I'll go with jobs, but only tax-financed ones. I'll go, tentatively, with discoveries, although everything to come out of the LHC so far is simply confirmation of the same old Standard Model we've known since the 70s. The SSC would have been able to probe higher energies, but as things are standing, there is no indication of new physics; if it exists, it could easily be out of range for both the LHC and the hypothetical SSC.

More to the point, the economic value of those discoveries is almost certain to be 0, for simple reasons of scale.

As for innovation, what has particle physics brought which would not equally well be brought by any number of other large scale technological projects? We all know the "WWW invented at CERN" story, but surely we also all realize that the physics input to the WWW was exactly 0, that any large organization needing to handle large number of documents over a heterogeneous network would have come up with something similar, that others probably did, and that the real reason CERN's solution took off was it's being free open source?

Particle physics is a luxury. At best, it's pursued for reasons of intellectual curiosity. Trying to portray it as an economic engine is not going to convince anyone worth convincing.

<comment><USER3> You know...I just don't think that's the case. We can't predict what products or societal changes particle physics will result in until we actually do the research (the Schrodinger's Cat of research?).

One thing is for sure: if we don't do the research, we won't know any more, and we won't have gained any further understanding of the universe. I just don't think understanding our universe, and the world we live in, is a luxury.

</comment></comment></comment><comment><USER4> This might be as bad as the time Bush was reelected and all the scientists moved to Canada! Or when all those hardworking small business owners fled the country after Obama's election - we still haven't recovered from that one.

In all seriousness, the title should be fixed. Scientists telling a survey they might leave != scientists leaving.

Incidentally, here is data on science funding over time. For anyone wanting to make political points (which appears to be everyone else commenting so far), the data probably doesn't map onto your narrative all that well.

[http://www.aaas.org/spp/rd/fy2013/hist13pDefNon.pdf](http://www.aaas.org/spp/rd/fy2013/hist13pDefNon.pdf)

[http://www.aaas.org/spp/rd/fy2013/hist13pAgcyRes.pdf](http://www.aaas.org/spp/rd/fy2013/hist13pAgcyRes.pdf)

<comment><USER5> _" For anyone wanting to make political points (which appears to be everyone else commenting so far), the data probably doesn't map onto your narrative all that well."_

Right. Because it's not like _you 're_ trying to make a (conservative) political point every time I see one of your comments.

The problem with the data you've cited is that it actually shows the problem you're dismissing: science funding has declined in real terms (let alone inflation-adjusted terms) since the end of the Clinton administration. There's been a huge bloodletting -- across the board -- over the last 13 years. An entire generation of US scientists has been decimated by pointless budget cuts.

If you don't know anything at all about science, you might look at the overall trend of this graph and say: "science funding has gone up since 1976, so shut up and stop whining". This is the simpleton's interpretation of the graph. A more nuanced interpretation is that we invested billions of dollars into training of new scientists starting in the Clinton administration (~1996), and right about the time those new scientists hit the point in their careers when they needed funding to really get their labs started (~2000), we stranded them. There's an entire generation of scientists with structural under- employment because of this phenomenon, and yes, they're leaving the country (I personally know a few who have, and I can't fault their reasoning. You go where you can work. If I were still doing science, I'd probably be gone, too.)

But that's not even the most important story. An _intelligent_ observer looks at this graph and says: the US budget for 2013 is _$3.8 TRILLION_. We spend _$600 BILLION_ on the department of defense, and nearly _$900 BILLION_ on Social Security (aka: old people who contribute almost nothing to the growth of the economy). Yet we're trying to "save money" by shaving a few tens of billions (that's 0.8% of total federal spending, in case you're bad at math) off of the one tiny part of our government that has consistently delivered outsized real returns to the economy. Any economist -- conservative or liberal -- will tell you that you don't make your budget by nibbling at the smallest line items while ignoring big expenses.

This is what we call "cutting off your nose to spite your face". Stupid, pointless, and definitely _not_ conservative.

<comment><USER6> Your response is excellent. I do however have one nitpick. Social Security is not a budget expense. I know it's listed as one but it shouldn't be. Social Security is an actuarially sound self-funded pension system. There is a slight under funding right now however its payments are not budgetary items.

<comment><USER5> A fair point. I just went to the FY2013 federal budget numbers, but didn't think about the fact that social security has a rather large tax of its own, and is therefore (mostly) "funded".

That said, every dollar we spend is a dollar that we could be spending on something else, so I think it's still fair to consider the expense of social security relative to other things we buy. We obviously can't abandon old people, but if someone said to me "we should find a way to cut social security by 0.8% to restore science funding", I'd be inclined to think it was a good idea, all other things being equal.

I'm definitely not a doctrinaire liberal -- I think we should have a social safety net (including things like single-payer health insurance), but I think we have plenty of other places we can look to balance the books before we start cutting into science and technology.

</comment></comment></comment></comment><comment><USER7> The title isn't descriptive of the article. The author discusses serious issues with decreasing funding of science in the US, but there is no exodus of scientists.

Let's hope Mars can be the national (I include Canada because of the close relationship) project that fuels science and innovation for our generation.

<comment><USER8> "there is no exodus of scientists."

If there's one thing American's like its creating their personal identity solely using their job title. I believe the article assumes domestic readers know this peculiarity. The article is full of "Fifty-five percent know a colleague who has lost a job or expects to soon." and "I have just laid off my technician and will lose my postdoc in six months,"

There are no open jobs, and those unemployed workers will have to do "something" if they intend to eat and obtain medical care and shelter. So I would assume once they're hired as day care workers or bartenders or waiters, displacing the mere high school diploma holders, they'll no longer be considered scientists, but be considered taxi drivers or whatever.

There should be data on numbers employed, or fraction of the population employed, or salaries in those fields. For example if there were an extreme shortage of biology grads, you'd expect to see starting salaries for the DNR around $200K or so. Oh wait, most of those grads end up at Starbucks.

</comment></comment><comment><USER9> I'm having a hard time finding any data that support the conclusions of this article.

My mom is an immunologist, and in 1999 my family moved from France to the US due to better opportunities for her as a scientist.

<comment><USER10> Sick and aging people are a growth industry for the next 20-30 years, thanks to the demographic bulge the country experienced following WW2 resulting in a large and wealthy generation known as the 'baby boomers,' who are now retiring _en masse_. Other developed economies had similar demographic bulges, but none large as the US or with the same sort of competitive medical services market. As I pointed out above, funding for health research (through the NIH) has tripled over the last 25 years.

The picture is not as rosy for scientists outside the medical field.

</comment></comment><comment><USER11> It's not just the funding. The politics in science (I don't know about internationally, but definitely here in the US) can be pretty ridiculous.

</comment><comment><USER8> In the old days, as a scientist, if you studied hard and worked hard, before you got downsized for being too old, you might make local management rich. They happen to own the government. No great surprise the feds back student loans and encourage the myth of a STEM shortage, feed more minds into the gristle mill. You don't need some kind of actively managed centrally controlled conspiracy, you just need individual actors following their own pocketbooks.

In the modern era, lets say post 1990, as a scientist, if you studied hard and worked hard, before you got downsized for being too old, you might make some factory owner in China who stole your idea and research, and the Chinese .gov bribe takers, very rich. The local management makes all their money off non- scientific stuff like social networks and chat programs, so they certainly don't need science, they've pivoted out of the physical market and left it for the Chinese. Because the rich guys own the government, the .gov also don't care much anymore.

The best hope for modern scientists is to hope for sponsorship / patronage from the Chinese government and hope they can outperform the more numerous locals. Good Luck, you'll need it.

</comment><comment><USER12> Jiminy cricket, guys, enough with the linkbait on HN already.

1 in 5 scientists have considered leaving the U.S. That's the news here, if there is news. By way of comparison, another recent study shows 9% of the total population has considered leaving. People consider lots of things, and surveys show all kinds of nifty trends, like the percentage of people who believe we have been visited by aliens, or those thinking the moon landing was faked -- both of which, last I checked, had more traction in the public than the idea of leaving the country. People believe weird shit and hold interesting opinions. Film at 11.

So the title doesn't match the essay. But wait! That's not all! The body of the text, supposedly from a non-partisan viewpoint, is pretty much an attack on one of the political parties in the U.S. I won't spoil the surprise for you, but I bet you can guess.

Among the shifting and meandering logic, we get a nice confusion of whether or not funding equals science (the two are not necessarily related) and a nice slam across corporate America's face with the old "They're just in it for the profits" routine.

All fairly normal and pedestrian rhetoric, but nothing much newsworthy, nothing with a supporting argument for the thesis, and nothing, frankly, that interesting. Unless you want to make a call for political action as part of supporting some political message, which this author does. That's great, but I'm not sure what to do with it on HN. Argue politics with all the usual players taking up the usual roles? Not so much. At least for me.</comment>


Without Touching a Line of Code - <AUTHOR> https://mikeylicio.us/blog/without-touching-line-of-code <comment><USER2> I understand what you mean, sometimes I also get frustrated at the companies advertising their codeless service. I think that you have to realize that most likely you are not the target audience for these products. They are simplification tools that tend to break at some point of complexity where you are forced into the nitty gritty anyway. I think this might even be a reaction to the "Everyone should learn to code" type initiatives.

</comment><comment><USER3> ok...?</comment>


Show HN: I'm testing the waters on a short term project matchmaking service - <AUTHOR>

I'm gauging interest for a service that would help connect devs/designers/copywriters with folks who have short term projects (a few days or weeks of work). The projects would range in value, but should ideally be at least a few $100.
Through Forrst, there are already thousands of such people available to work, it's just a matter of if this is interesting to the other side. If it is, would you leave your info? Thanks a ton.
Form: https://kylebragger.wufoo.com/forms/short-term-project-matchmaking/ <comment><AUTHOR> Clickable: [https://kylebragger.wufoo.com/forms/short-term-project- match...](https://kylebragger.wufoo.com/forms/short-term-project-matchmaking/)

</comment><comment><USER2> Are you still accepting new copywriters, and if so, is there a separate form to fill out?

<comment><AUTHOR> Haven't got that far yet :)

But there certainly will be, so long as there is enough interest from the hiring side.

<comment><USER2> Ahhh, well, good luck on it!

</comment></comment></comment><comment><USER3> just signed up... seems like a cool idea.</comment>


A Japanese Love Hotel - <AUTHOR> https://roadsandkingdoms.com/2019/japanese-love-hotels-toko-sekiguchi/ <comment><USER2> These are big in Korea too, and the purpose isn't just infidelity as the article would have you believe. It is common to live with your family until marriage, and many Korean families outwardly hold traditional values of chastity until marriage. Families also tend to have a greater financial involvement (and therefore more influence over) in their childrens ability to rent a nice place because of jeonsei (very high security deposit - like a downpayment - in exchange for reduced rent), so it is in adult childrens best interest to keep up appearances with their parents. The average marriage age is 32 for women, 35 for men. So this means many, many years of adulthood dating where you cannot bring home a date to sleep with because you live with your parents. So love hotels are common in Korean nightlife districts and they are actually quite nice and cheap too. I stayed at some of them years ago and I recall they were around $75 per night, with free pirated DVD movies to borrow.

<comment><USER3> I took a teaching position back in the mid-00s in Korea. Because it was a new school, the housing wasn't ready when we arrived. The school decided to put us up in one because it was cheap. We were there for a month.

Nice? That's not the word I would choose. Random stains on the furniture and carpet. Hairs on the sheets. A vending machine at the end of the hall selling an assortment of toys.

They did have free pirated DVDs on a shelf at the end of the hall. It was all porn.

It did have fantastic sound effects at 2 AM in the morning, though.

Nice is definitely not the word I would choose based on my experience.

<comment><USER4> The place your school put you up in probably didn't charge US$75/night. In my experience (in Taiwan), many decent, modern hotels offer 2- to 4-hour couples packages just to tap that extra segment of the market.

In addition to those, there are definitely seedy places where you can get a room for 3 hours for around US$15.

</comment></comment></comment><comment><USER5> I'm french, and live in France, near Paris.

In France, there is only (AFAIK) _one_ love hotel (inspired by the japanese ones).

It's pretty small and the rooms are not big at all, but okayish (I only visited once).

What's interesting is the history of its creation: from what I've read, it was pretty hard to get the authorizations (which explains why there are not any others). Basically, this is the old B.S. about prostitution. Which is total hogwash because they check your ID when you go in, so it would be impossible for an escort to use it with more than a couple customers (and regular hotels are frequently used for escorting anyway so what's the difference?).

Personally, I think that love hotels are a great thing, much better than a regular hotel, and MANY people would benefit from those. Especially people in alternative sexualities (like myself) for whom finding a place to have sex is often a serious problem: going to one partner's home is often seen as a risk for security & anonymity.

BTW, there has also been recently (one year ago), the opening of a "sex doll brothel" in Paris, and they were also checked by the police... Seriously, a freaking SEX DOLL hotel! It's crazy how anything related to sex in any way creates so much problems.

<comment><USER6> Sadly, I don't think love hotels would work in America. Too many people disrespect and trash public spaces. It's the same reason they don't have net cafes like they do in Japan (where you can informally crash and use the internet).

<comment><USER7> I don't know exactly how common they were, but I've definitely been to an internet café in the US (in a very small town FWIW). AFAICT they're gone because they just don't make economic sense anymore with everyone having a WiFi device already.

<comment><USER8> Also, public libraries have expanded into the space, using taxpayer dollars to provide the same service for free. Very entrepreneurial of them.

</comment></comment></comment></comment><comment><USER9> If you need a hotel in Tokyo you can get a good deal on these love hotels compared to big name hotels. You do not need to go there for sex. They are available on hotels.com and bookings.com and such.

The rooms are good size (in Tokyo hotel rooms tend to be small), you get a very luxurious bath room and bath utilities, you can order a tons of food through TV and of course you get some free condoms.

Do not expect amazing lobby, though. Entrances are more hole-in-a-wall style.

Locals seem to have a cultural taboo with these hotels, but as a Western traveler it is a pleasant and fun experience.

<comment><USER10> There are a couple of catches to using love hotels as regular hotels though:

* If you want a "stay" (overnight) instead of a "rest" (hourly), check-in times are late and check-out times are early, with steep hourly rates slapped on top if you overstay

* It's often not possible to leave the room and come back (insert joke about in-and-out privileges here)

* Rooms are often windowless

All that said, they can be great deal. Here's a mildly tongue-in-cheek comparison of the JW Marriott Seoul vs a random love hotel in Cheonan:

* $400 (rack) vs. $50 (rack)

* Internet access $25/day vs In-room Internet PC for free

* Two chocolates vs. two condoms on your pillow

* Ordinary bathtub vs. heart-shaped jacuzzi in bathroom

* Whisky and cognac vs. beer, soju and dildos in minibar

* Porn costs $18/channel/day vs. three channels of it for free

<comment><USER11> Why would anyone want a cold dildo?

<comment><USER10> This particular love hotel had both a minibar and a (uncooled) toybar. Very mildly NSFW picture: [http://www.patokallio.name/photo/travel/Korea/Suanbo/Motel_T...](http://www.patokallio.name/photo/travel/Korea/Suanbo/Motel_Toys.JPG)

</comment></comment></comment></comment><comment><USER12> In Brazil these things are known as "motels". I still have to think twice when I see a reference to the word in its original context, like "Bates Motel" in Psycho.

<comment><USER13> Is that because people park their motor-vehicles there?

<comment><USER14> I don't know how they started. They may have started as regular "motels" like in the US and then turned into love hotels because they were usually far away from downtown, near highways and that's a better place if you don't want to be seen with someone else? Nowadays "motel" equals "lovel hotel" in Brazil, without exception AFAIK.

Cheap motels (in the US sense) that are near highways and are not "love hotels" will just call themselves "hotels" and avoid the sex-related stuff. There are plenty of those.

I think it's also pretty common place for people who meet at night clubs and don't want to take their partners back home (in order to keep their privacy). They go to the motel and that's it.

Wikipedia doesn't have a lot on this: [https://en.wikipedia.org/wiki/Motel#South_America](https://en.wikipedia.org/wiki/Motel#South_America)

<comment><USER15> Yeah, it's the same in Chile. Cheap hotels go by names that generally translate to "inn".

</comment></comment></comment></comment><comment><USER16> >I have to say: if Japan wasn't already the most neurotically clean country I've ever been to

I wonder if he's been to Singapore though? I felt like Japan was very clean, but Singapora was on a whole different level. And not just the "they banned gum" comments, but that I saw at 11:00pm people disappear and the cleaning crews come out pressure washing sidewalks and anywhere people might have been. Tokyo especially felt a LOT like NYC to me, except with much wider alleys, and cleaner. Singapore in contrast didn't feel 'real' at all, like if Disney made a whole country.

Had a long layover in Japan, tried to stay in a Love Hotel as all we needed was 6 hours, but it wasn't $82, more like $120 and turned out we found a cheaper rate at a 'normal' hotel. Next time though!

<comment><USER17> Japan is clean from end to end, though. I've driven all day across the countryside many times and never seen much more than neatly-stacked supply storage in terms of clutter. Outside of Tokyo and Osaka, the other big cities like Nagoya are clean as a whistle, with every street swept clear every morning and all trash carefully monitored by a cadre of obaasan volunteers. This is the situation in every corner of Japan that I've ever seen. It's truly "bottom-up" clean, which is remarkable and special.

<comment><USER10> So is Singapore. Then again, Singapore is only about 40 km end to end!

</comment></comment></comment><comment><USER18> I little bit of footnote history: back in 2003, in conjunction with the 1st International Moblogging Conference in Tokyo, a group of people (who being armed with pretty much the planets first camphones connected to the internet were also arguably the pioneers of instagram culture) held the 1st International Love Hotel Moblogging Conference.

[http://www.tokyotidbits.com/lovehotel/](http://www.tokyotidbits.com/lovehotel/)

aka we were overexposing on the internet before it was cool ;)

</comment><comment><USER19> I was wondering about the Slack emoji :love_hotel:, and now I know.

<comment><USER20> Its not just part of Slack or Discord's emoji set; its a standard Unicode emoji

[https://emojipedia.org/love-hotel/](https://emojipedia.org/love-hotel/)

<comment><USER21> And before that, it was in SoftBank's nonstandard extension to the Shift JIS encoding standard at code FBA1 [https://ja.wikipedia.org/wiki/SoftBank%E7%B5%B5%E6%96%87%E5%...](https://ja.wikipedia.org/wiki/SoftBank%E7%B5%B5%E6%96%87%E5%AD%97)

</comment></comment></comment><comment><USER22> Love Hotels are a hit in East Asian. So yes, it's popular China as well since at least 2016[1].

[1]: [https://www.theguardian.com/world/2016/jan/13/chinas-love- ho...](https://www.theguardian.com/world/2016/jan/13/chinas-love-hotel-boom- shows-nations-changing-attitudes-to-sex)

</comment><comment><USER23> 'Telos' called here in Argentina.

</comment><comment><USER24> Webpage text doesn't render at all unless you enable some Javascript which is used to load fonts _with no fall-back_ , and the fonts once loaded are also ugly and unreadable.</comment>


My Favorite Programming Problem to Teach: Digit Length - <AUTHOR> https://jstrieb.github.io/posts/digit-length/ <comment><USER2> > As a result, solutions using strings are disallowed on problem sets and > quizzes until they are taught. However, the few students who have prior > Python programming experience may be tempted to find digit length without > loops using a variant of the following (for our purposes) invalid solution.

Wow. this is one of the reasons I hated school. No programmatic reason what given for why a string solution couldnt be used, only an arbitrary reason. Here students may have knowledge from self teaching or whatever, but they are unallowed to use that knowledge because "reasons".

To any teacher that thinks its a good idea to punish students for thinking outside the box: shame on you. All youre going to end up doing is crushing enthusiasm and/or creating drones. Please dont.

<comment><USER3> Strings are disallowed because they are not necessary for this problem and although the solution is shorter, is far more inefficient; it also doesn't demonstrate the algorithmic thinking that the course is obviously trying to teach.

I've taught CS courses before, and have seen plenty of self-proclaimed self- taught know-it-alls who seem to be more stackoverflow-copy-pasters than anything else.

<comment><USER2> > Strings are disallowed because they are not necessary for this problem

the division examples are not necessary either, thats the point. you can solve it different ways, that doesnt mean one way is not necessary, it just means its different. one may be faster, one may be more readable. If you dont allow different solutions you cant explore the tradeoffs between them.

<comment><USER4> Generally, the purposes of the exercises in a class is to reinforce the material that has been taught up to that point and to demonstrate that the student can use it.

For example, if early in an elementary number theory class the student is asked to prove that there are in infinite number of primes of the form 4n+3, a solution that just invokes Dirichlet's theorem on primes in arithmetic progressions would probably not be acceptable. That approach does work to show that there are an infinite number of 4n+3 primes, but completely fails to show that that the student understood the material actually taught in class.

It's the exact same thing with the digit counting problem. Solving it by just invoking the built in string length function does little to demonstrate that the student understands the material taught so far.

</comment></comment></comment></comment><comment><USER5> But the math.log10 solution is unfortunately "wrong" too, at least in my python3 implementation.

        import math

        def digitLengthCorrect(n):
            return len(str(n))

        def digitLengthClever2(n):
            return 1 if n == 0 else (math.floor(math.log10(n)) + 1)

        testcases = (
            [10 ** i for i in range(300)] +
            [(10 ** i) - 1 for i in range(300)]
        )

        for t in testcases:
            a = digitLengthCorrect(t)
            b = digitLengthClever2(t)
            assert a == b, (t, a, b)

<comment><USER6> Good point. To indemnify youself from this type of data loss, change math.log10() to np.log10() and use Decimal() for tasks involving precise number crunching.

EDIT: Setting Decimal module's precision manually via getcontext() may be required if you work with "long" numbers.

        import math
        import numpy as np
        from decimal import getcontext, Decimal

        def digit_length_correct(n):
            return len(str(n))

        def digit_length_clever_2(n):
            if n ==0:
                return 1
            else:
                return math.floor((np.log10(Decimal(n)))) + 1 # You can use np.floor() and eschew the use of math. module altogether, but I left it intact to show the minimal necessary modifications

        def generate_cases(n):
            getcontext().prec = n + 3
            return (
            [10 ** i for i in range(n)] +
            [(10 ** i) - 1 for i in range(n)]
        )

        cases = generate_cases(300)

        for t in cases:
            a = digit_length_correct(t)
            b = digit_length_clever_2(t)
            assert a == b, (t, a, b)

        for t in cases[-2:]:
            print(t)
            print(digit_length_correct(t))
            print(digit_length_clever_2(t))

<comment><USER7> Using arbitrary precision arithmetic kind of defeats the point of not using loops.

</comment></comment></comment><comment><USER8> Another fun calculation: maximum amount of decimal digits a binary number could have.

      digits = ceil(bits * log10(2))

A 10 KiB picture can be thought of as one 24 661 digit number. A 2 GiB video file can be thought of as one 5 171 655 946 digit number. I find this really puts everything in perspective. Every number already exists, digital content creators are just trying to find them. Certain numbers are actually illegal.

A practical application of this: calculating buffer sizes for a function that converts numbers to text.

      /* digits = ceil(bits * log10(2))
       *        = ceil(64   * log10(2))
       *        = 20
       */
      #define DECIMAL_DIGITS_64_BITS 20

      /* digits + sign + NUL */
      char digits[DECIMAL_DIGITS_64_BITS + 1 + 1];

It's possible to handle arbitrary precision numbers by counting the number of bits and calculating the amount of memory that must be allocated for the resulting string.

<comment><USER9> As shown in this thread, you should be very uneasy about relying on this for arbitrary inputs. Non-trivial floating point math (such as log10) + rounding almost surely ends up with off-by-one errors.

</comment></comment><comment><USER10> Are these the typical steps students go through? I would guess that most students know to convert it to a string and count the length, with a smaller group knowing about base 10 logs.

I like the general idea of it--improving the algorithm incrementally, finding exception cases, but I wonder if there is a better example that could be used.

It seems to me that one either knows how to use logarithms or not, and thus students would either skip to the final solution, or be stuck until having the answer given to them.

<comment><AUTHOR> I was writing less about the steps students would go through when solving the problem on their own, and more about points I try to hit when using the problem to teach. At this point in the course, strings haven't been taught, so using them would not be a natural solution for those without prior programming experience. I will also review logarithms if necessary.

</comment></comment><comment><USER11> quite funny reading HN articles sometimes. So...

I read [https://www.netmeister.org/blog/cs- falsehoods.html](https://www.netmeister.org/blog/cs-falsehoods.html) which came from [https://news.ycombinator.com/item?id=21500672](https://news.ycombinator.com/item?id=21500672) in the top HN articles at the moment

Item 27 of that list made me laugh when I read this article :)

<comment><AUTHOR> I definitely designed my site with the stereotypical 90s hacker terminal in- mind :)

</comment></comment><comment><USER12> So in the end the solution proposed by the TA is wrong and he invalidated the good solutions proposed by people with actual python experience.

I hope he learnt more than the students...

</comment><comment><USER13> From my point of view, this isn't programming. This is high level language tricks. It would be much more fun doing it in Assembly language, without any use of any kind of library, expect for input and output. This way student would have learnt so much more, about how integer are manipulated into cpu (just bits in base 2) doing smart math conversion to represent it in base 10. And why not generalise the problem to also compute the size in base 8, or any base N.

I hate those programming class just trying to teach python surface use, while in a programming class you have time to go deeper and learn about how python works, cause basically python use all str to binary and loop for doing all the work requested by the teacher, without student even being aware of how it does it !

<comment><USER14> > _It would be much more fun doing it in Assembly language, without any use of > any kind of library, expect for input and output. This way student would > have learnt so much more, about how integer are manipulated into cpu (just > bits in base 2) doing smart math conversion to represent it in base 10. And > why not generalise the problem to also compute the size in base 8, or any > base N._

One could say as well: this isn't programming, this is low level language tricks.

You don't need to know "how integers are manipulated into cpu" when learning to program, and at an introductory class like that described in the article you shouldn't either. There's a reason SICP at MIT was in Lisp and then Python.

I also very much doubt it would be "much more fun doing it in Assembly language".

The author went at length to explain how this is a useful exercize for programming, as it introduces edge cases, alternative implementations, testing, etc.

<comment><USER13> I understand what you say, but i think you are wrong. Assembly is closest to what a cpu actually does for REAL, so you are programming. When using Python or whatever any advanced language, you are less programming CPU and more relying and tons of layers from other's works which are basically created to never let you understand how to program a CPU. So peoples feels they are programming, while they are just integrating tons of library and set them to do something usefull (which is good too).

But if you want to teach programming, i would follow my path, and provide a deeper understanding at how to control a cpu and how it really works. In order to demystify computer and gives student a real experience of all the hidden works that are done with a 4 lines python code.

and for the fun side, guess why assembly is the second searched language on Stack overflow during weekend : [https://stackoverflow.blog/2017/02/07/what- programming-langu...](https://stackoverflow.blog/2017/02/07/what-programming- languages-weekends/)

i guess people are trying to have more fun on weekend than on boring office project during work days :)

And those numbers show a real interest about Assembly which is usually greatly discarded in any common CS teaching anywere. So teachers decides it's not interesting, while in fact most peoples search about it on weekend...I mean it illustrates a real issue here. May be understanding how to program a cpu at low level is something natural, that only scholar peoples cannot understand, therefore neglecting natural tendency of normal peoples to try to understand how things really work...

And for those really wanting to even dig deeper and understand what is a CPU, i strongly suggest looking for "from nand to tetris" [https://www.nand2tetris.org/](https://www.nand2tetris.org/) wich basically start at nand logical gate, to the extent to create a full working cpu and programming it to play tetris.

<comment><USER15> You can always learn assembly language later. Putting it in the intro class is a bad idea.

<comment><USER13> well can you develop a little bit your answer cause a one sentence opinion is just nothing.

</comment></comment></comment></comment></comment><comment><USER16> digitLength n = if n < 10 then 1 else 1 + digitLength (n / 10)

Avoiding loops by recursion is no worse than avoiding loops by calling a looping library function...

</comment><comment><USER17> You ever get the feeling that companies and programmers who become massively productive and profitable are either finding a narrow path where there are no concerns like the other discussions here, or simply ignoring them and ploughing on with a pile of faulty code that mostly works?

Look how much discussion there is over a few thousand nanoseconds here, edge cases around 16 digit numbers (if my bank account gets that high, and you accidentally round it up to too many digits, we can deal with that when it happens).

It seems inconcievable that the most 'productive' can spend this much effort on everything they do, and 'pick your battles' only goes so far towards explaining it. 'Move fast and break things' goes farther but only in the space of possibilities where correct vs incorrect has approximately no consequences.

"Companies with unicorn valuations are most probably doing things which don't have to be done very well"?

</comment><comment><USER18> Must be a good teaching problem seeing as it's generated such a large amount of interesting discussion.

</comment><comment><USER19> This reminds me of something I enjoyed writing a couple of years ago. I'd decided to use the Damm checksum algorithm[1] for order numbers at work. Every implementation that I could find was turning the number into a string than then check summing it one character at a time. And that approach felt rather suboptimal. So, I decided to write a numeric implementation[2].

[1]: [https://en.m.wikipedia.org/wiki/Damm_algorithm](https://en.m.wikipedia.org/wiki/Damm_algorithm)

[2]: [https://github.com/jeethu/damm](https://github.com/jeethu/damm)

</comment><comment><USER20> By using a do-while loop, you don't have to handle the special case of zero (although one could argue that the do-while loop _is_ the special handling for zero). Python does not have such a loop, so you need to do an infinite loop with an explicit break and I know some people recoil in horror at the sight of such code.

        def digitlength(n):
            digits = 0
            while True:
                digits += 1
                n /= 10
                if n == 0:
                    return digits

<comment> a-nikolaev

        def digitlength(n):
            digits = 1
            while (n > 9):
                digits += 1
                n /= 10
            return digits

Also may want to set n = abs(n) in the beginning, in case n is negative.

<comment><USER20> The problem statement said that the input would be natural numbers.

<comment><USER21> Wait, do natural numbers include zero? _evil grin_

<comment><USER22> I looked up wikipedia which says there's no conclusive definition.

<comment><USER21> The joke is that, if you don't need to check if the input is negative because it is not a natural number, you can also choose a definition of the natural number without zero to avoid special casing it at all.

</comment></comment></comment></comment></comment></comment><comment><USER23> Why not:

      def digit_len(n):
        return len(str(n))

<comment><USER20> For people who, like me, thought that this would be slower than repeated division: my crappy benchmark indicates that allocating a new string and taking its length is faster by a factor of 2–2.5x.

Edit: In C, the version that loops is 15 times faster than the version that allocates a new string. Python is weird.

<comment><USER24> It's a common feature of interpreters that they impose a large slowdown on whatever is done in interpreted code; Python's is about 40×, although better interpreters like GhostScript and Lua are more like 10×. It's surprising the difference isn't bigger in this case, but I can mostly confirm your results, getting 3× except for small numbers:

        In [1]: dl = lambda n: 1 if n < 10 else 1 + dl(n // 10)

        In [3]: map(dl, [0, 1, 9, 10, 99, 1000, 15432, 32, 801])
        Out[3]: [1, 1, 1, 2, 2, 4, 5, 2, 3]

        In [4]: %timeit dl(15322)
        100000 loops, best of 3: 4.64 µs per loop

        In [5]: %timeit len(str(15322))
        1000000 loops, best of 3: 1.41 µs per loop

        In [6]: %timeit dl(0)
        1000000 loops, best of 3: 721 ns per loop

        In [7]: %timeit len(str(0))
        1000000 loops, best of 3: 1.24 µs per loop

<comment><USER24> Upon thinking about this further, I guess it's sort of obvious why this happens. dl(0) does a constant, a function invocation, a comparison, a conditional, and a return of a constant (although if you look at the bytecode you'll see a few more steps in there). len(str(0)) does a constant and two built-in function invocations, which (as it happens) involve lookups in the global namespace at runtime in case you have redefined len or str.

So, basically, it's only interpreting a very small number of interpreter bytecodes either way, so the small number it has to interpret to use the builtins is comparable to the small number it has to interpret to run the recursive definition, and so the interpretive overhead is comparable (and it swamps the overhead of things like allocating a new string).

This machine is kind of slow. This took 54 CPU seconds in LuaJIT:

        > function dl(n) if n < 10 then return 1 else return 1 + dl(n/10) end end
        > for i = 1, 1000*1000*100 do dl(15322) end

That means that approach took 540 ns per invocation rather than Python's 4640 ns --- only about 9× slower instead of the usual 40×. Or maybe this is a case where LuaJIT isn't really coming through the way it usually does.

</comment></comment></comment></comment><comment><USER4>
      def digitLength(n):
          dlen = 1
          high = 9
          while n > high:
              dlen += 1
              high = 10*high + 9
          return dlen

<comment><USER15> One fun enhancement to this is to avoid the n^2 cost it faces with large integers.

        def digit_length(n):
            totlen = 0
            n += not n
            while n > 0:
                klen = 1
                k = 10
                while n > k * k:
                    k = k * k
                    klen *= 2
                n = n // k
                totlen += klen
            return totlen

</comment></comment><comment> 08-15 The log10 method is incorrect: math.log10(999999999999999)==15.0. (That's 15 nines.)

Good job teaching unreliable algorithms.

</comment><comment><USER25>
      def what_a_hack(n:int):
          return len(str(n).strip('-'))

</comment><comment><USER26> What about len(str('12345')) If your teaching python you may as well take advantage of the standard library

<comment><USER27> If you read the article, you will see that this solution was covered at the end.</comment></comment>


Ask HN: How credible is Reinforcement learning in finance(stocks)? - <AUTHOR> Recently came across blog post on reinforcement learning on stock data to trade. However in reinforcement setting agent should be able to play with simulated environment to explore and learn. However there is no way to simulate the price all it does is takes historical fragment of price and replays it.
In my mind it is equivalent to historical data labeled and resampled with redundancy after shuffling. Am I missing something here?
Blog that I was reading was: https://towardsdatascience.com/deep-reinforcement-learning-for-automated-stock-trading-f1dad0126a02 <comment><USER2> > However there is no way to simulate the price

Compared to other real world problem domains which are far less computerised it seems relatively simple to get fresh data for a trading system: let the system execute actual trades and measure the actual response.

It might be difficult to do this without having a budget to burn in running experiments, both in terms of money that is going to be lost while the system makes poor trades, setting up controls to ensure that not too much money can be lost in any experiment, and subscribing to data feeds that give a richer view of how the market responds (e.g. order book info).

In contrast, consider trying to get fresh data in materials science etc where you may need to manufacture small batches of materials and then test them. It might cost $10k in materials and weeks of work with expensive machinery and skilled technicians to generate a dozen or so fresh data points.

</comment><comment><USER3> The problem I see with ML on stock prices is that stock price is not a function of the various available numerical indicators. Markets are non- deterministic and chaotic systems. Contrast that with driving, where the decision to turn left, right, throttle, etc. is largely a function of the features (surroundings, current speed, etc.).

That said, people are using ML to predict future prices, although no hedge fund claiming this has published their returns.

With regards to reinforcement learning, why not just try it? Numer.ai is an ML competition that gives you free obfuscated stock data and rewards you if your model is successful.

<comment><AUTHOR> Yeah thanks, I have tried, I was comparing and found reinforcement learning is not really _reinforcement_ in this setting. As it cannot allow exploring like what if scenario. It is similar to supervised learning setting as we do in labeling large data and trying to classify etc.

In contrast if we take [https://gym.openai.com/envs/Pendulum-v0/](https://gym.openai.com/envs/Pendulum-v0/) Environment allows algorithm to play with it and learn from new conditions and actions, so is reinforcement. To me finance is like teacher trying to tell same story no matter what student does/learns, bit wierd.

</comment></comment><comment><USER2> > In my mind it is equivalent to historical data labeled and resampled with > redundancy after shuffling

It is even more constrained than that: there can be no shuffling along the time dimension, as that would destroy the relationship between time and the price trajectory (e.g. momentum). It is possible they could have sampled subsets of the full universe of 30 stocks to generate different training scenarios, but the blog post doesn't talk about anything related to that.

> agent should be able to play with simulated environment to explore

The work makes the assumption that the environment is not impacted by the agent's actions, see section 3.3:

> Market liquidity: The orders can be rapidly executed at the close price. We > assume that stock market will not be affected by our reinforcement trading > agent.

It's interesting to try to roughly estimate how many non-overlapping windows of market data were used in training the ensemble strategy. What I really want is to estimate how many independent samples of input data there are to train on, but there are probably no truly independent samples, since we're talking about historical trajectories of stock prices over time. For the window size we can try to figure out how much time series data the ensemble model needs to consume to output a single prediction.

The ensemble strategy is described as:

> Step 1. We use a growing window of 𝑛 months to retrain our three agents > concurrently. In this paper, we retrain our three agents at every three > months.

> Step 2. We validate all three agents by using a 3-month validation rolling > window followed by training to pick the best performing agent which has the > highest Sharpe ratio. We also adjust risk-aversion by using turbulence index > in our validation stage.

> Step 3. After validation, we only use the best model with the highest Sharpe > ratio to predict and trade for the next quarter.

Steps 1, 2 and 3 depend on a number of numerical and structural parameters, such as the size of the windows in step 1 and 2, the choice of which metric to pick best agent, the "turbulence index" adjustment, the use of argmax instead of some other approach in step 3. It is possible that the researchers wrote down and pre-commited to exactly these parameters before looking at any data, and never changed them, but it is more likely that these parameters were chosen out of a huge space of alternatives based on what was observed while running experiments, before finally measuring performance on the hold-out data set.

When thinking about automated ensemble strategies, the above 3 steps all need to be executed before the ensemble model can output a single prediction. I don't quite understand the explanation of step 2 but it suggests that the ensemble model needs to consume at least 3 months + 3 months = 6 months of trailing data before it can output a single prediction. In reality it would be worse, many of the input features defined for each stock that are fed as inputs appear to be technical indicators that are some fancy form of a moving average, so to define them they also need to consume some trailing window of price data -- these details aren't described in the blog. If we assume that each of these features needs at most 1 month of trailing price data then that means we need 1 month + 3 months + 3 months = 7 months of trailing price data before the ensemble model can output a single prediction.

> Data from 01/01/2009 to 12/31/2014 is used for training, and the data from > 10/01/2015 to 12/31/2015 is used for validation and tuning of parameters. > [...] we test our agent's performance on trading data, which is the unseen > out-of-sample data from 01/01/2016 to 05/08/2020

So data from 01/01/2009 to 12/31/2014 is used for training, and data from 10/01/2015 to 12/31/2015 was also used to tune parameters, i.e. also used for training. This gives 15 years of data for training, which is enough to generate about 15 years / 6 months = 30 non-overlapping windows of data that could be used to generate outputs from the ensemble model. Then there's enough out-of-sample data for about 4.5 years / 6 months = 27 non-overlapping windows of data to evaluate the model performance.

It seems like this work involves fitting and tuning a very high parameter model using a dataset that only offers a single trajectory (containing all the stocks) of 30 non-overlapping periods of input data to use when training, and 27 non-overlapping periods of data to evaluate.

<comment><AUTHOR> Thanks for your reply</comment></comment>