Mathematics in type theory - <AUTHOR> https://xenaproject.wordpress.com/2020/06/20/mathematics-in-type-theory/ <comment><USER2> For anyone interested in formal verification, there's this super slick project Metamath Zero [0] by Mario Carneiro that is working to implement a verifier that can verify itself, compiled binary and all.

As discussed on the project page, Lean is aimed at being _ergonomic_ but at the price of a large (and somewhat buggy) codebase. On the opposite end of the spectrum is Metamath [1] which aims at being the minimum needed for formal verification, but at the cost of extremely pedantic (albeit interesting!) proofs.

Metamath Zero, attempts to find a happier balance by designing a language which is both ergonomic for the proof-writer as well as minimal enough to allow self-verification.

At the moment, the project is still pretty alpha but Mario is both extremely talented and actively developing. That said, if you are someone like me and just want to dive into the math, definitely check out Metamath [1]. It's library of proofs for standard ZFC foundations, set.mm [2], is seriously impressive. I mean, you can find a proof of the fundamental theorem of calculus and trace it all the way back to axioms if you so please (see theorems ftc1 [3] and ftc2 [4])!

For someone with a background in math, the set.mm [2] database is an absolute blast to rogue around in. It takes some getting used to, but I am starting to prefer set.mm over Wikipedia when applicable.

Anyway, I love this stuff!

[0]:[https://3g2upl4pq6kufc4m.onion/html?q=metamath%20zero](https://3g2upl4pq6kufc4m.onion/html?q=metamath%20zero)

[1]:[http://us.metamath.org/mm.html](http://us.metamath.org/mm.html)

[2]:[https://github.com/metamath/set.mm](https://github.com/metamath/set.mm)

[3]:[http://us.metamath.org/mpeuni/ftc1.html](http://us.metamath.org/mpeuni/ftc1.html)

[4]:[http://us.metamath.org/mpeuni/ftc2.html](http://us.metamath.org/mpeuni/ftc2.html)

</comment><comment><USER2> Not the point of the article at all, but in my experience when an author says "canonical map" it can typically be made precise as a unique natural transformation between whatever functors capture the objects of discussion. I think this is part of what MacLane meant by Category Theory being invented to talk about natural transformations.

</comment><comment><USER3> Avigad, one of the Lean authors has several resources cited elsewhere in Xena Project. These are really useful to learn the necessary background [1,2].

IMHO, logic and type theory are the algebra and calculus of computer science. CS degrees should be designed around some core courses on these, in the same way Math degrees tend to build up on linear algebra and real analysis coursework.

[1] [http://avigad.github.io/logic_and_proof/](http://avigad.github.io/logic_and_proof/)

[2] [https://avigad.github.io/formal_methods_in_education/](https://avigad.github.io/formal_methods_in_education/)

</comment><comment><USER4> Should a language be able to form predicates total over all possible expressions in the language (including these predicate expressions) in a non trivial way, could these be then identified as 'types' in the sense that each can partition the set of all expressions in the language into 'of the type' and 'not of the type'? Is this still type theory, or something else?

<comment><USER5> Usually it's not possible to form such a predicate. If we have impredicative base universe, then it's possible to form a predicate over all types in the base universe (which may or may not be _all_ types in the language), including the predicate itself. However, this feature is rarely used in type theory, because it is incompatible with classical logic.

In programming, this is more commonly available, because we don't necessarily care about propositions, let alone classical logical proofs. The polymorphic identity function type `forall a. a -> a` in System F quantifies over all types including itself.

</comment></comment><comment><USER6> Is it just me, or is "proof" used in two ways also – the logic and the result? To use the given example, do I need to understand the logic of the Bolzano- Weierstrass proof in order to use it?

<comment><USER7> It _is_ both, in the same way as a function is both its implementation and its "extension" (its effect on all inputs). You don't need to know how `sort` is implemented, if you know that it sorts; similarly, you don't need to know how Bolzano-Weierstrass is proved if you want to use it.

<comment><USER8> > It is both, in the same way as a function is both its implementation and its > "extension" (its effect on all inputs).

Note that, according to definition of 'function' most common in math, this is false. A function need not even be implementable.

Aside: As someone trained in math I've always found explanations of type theory "fractally confusing" because of stuff like this: type theorists' words mean something different from my words, and in the explanation of what they _do_ mean, they use words that also mean something different.

<comment><USER9> A function is rather like a group: it's more important as an idea than as a rigorous definition. Working mathematicians may be vaguely aware that a function "is" a set of ordered pairs (just as they may be vaguely aware that a real number "is" a Dedekind cut), but they're unlikely to think about a function in those terms when they're actually working with it (unless they're working specifically on foundational issues). Indeed the set-theoretic model of a function accommodates impossible functions - but that's one of the main problems that type theory exists to solve. The type-theoretic model of a function does not correspond directly to the ZFC model of a function, but the hope of type theory is that it should correspond _more_ closely to the day-to- day working concept of a function (whether you're a mathematician or a programmer).

</comment></comment></comment></comment><comment><USER10> The title "Mathematics in type theory" is like mathematics in real analysis or set theory.

<comment><USER11> It is like mathematics in set theory, as set theory can be used for a foundation of mathematics; however, it is not like real analysis, because real analysis cannot be used as a foundation for mathematics (without torture.)

<comment><USER10> I think you missed my point. The type theory is mathematics. So saying that math in type theory is nonsense.

<comment><USER11> I'm sorry but you misunderstand the situation. Set theory, type theory and category theory can be used for metamathematics [1] whereas real analysis and number theory cannot. They are foundational theories whereas real analysis and number theory are not. The reason for the difference has to do with the expressiveness of the formal languages and how they themselves can be expressed without a prior language other than formal logic itself.

[1]. [https://en.wikipedia.org/wiki/Metamathematics](https://en.wikipedia.org/wiki/Metamathematics)

[2]. [https://en.wikipedia.org/wiki/Foundations_of_mathematics](https://en.wikipedia.org/wiki/Foundations_of_mathematics)

<comment><USER10> I know they are foundations of mathematics. They are elements of mathematical logic. No disagreement on this.

But how they are not mathematics? They are part of mathematical logic and mathematical logic or logic in general is branch of mathematics. Do you want to say they are somehow separated from math?

<comment><USER11> They are mathematics: mathematics about mathematics is what metamathematics means.</comment></comment></comment></comment></comment></comment>


Perseverance and burning out. - <AUTHOR> http://louiebacaj.com/perseverance-and-burning-out/ The difference between constantly improving and burning yourself out. <comment><USER2> I think it's important to differentiate, I work with a lot of people who cant tell the difference. They think working 12-16 hours will get them results and they will get noticed.</comment>


GitHub Has Become a Haven for China's Censored Internet Users - <AUTHOR> https://www.npr.org/2019/04/10/709490855/github-has-become-a-haven-for-chinas-censored-internet-users <comment><USER2> Wouldn't this just add more fuel to the fire for China to crackdown on access to GitHub and ultimately make it even harder for engineers there to use and contribute to projects? Only asking because I recall a time where they did indeed block GitHub because of some of the projects hosted and I'd imagine this might potentially cause a similar course of action.

<comment><USER3> I don't think they could afford to block it because China's tech sector is dependent on lots of American-developed open source projects hosted on it. They could set up their own git service and fork them, but they wouldn't get the benefit of ongoing development; they'd be on their own. It's almost like connecting with the rest of the world has tangible benefits...

<comment><USER4> Yup the PRC's approach so far has been to force the rest of the world of kowtow, company by company, to their ridiculous petulant insecurities and autocratic fascism. That way they get the best of both worlds - a connection to a wide world the paranoid, boot-black politburo need not fear.

<comment><USER5> Is that not what most countries would do if they had the power to?

<comment><USER6> I was recently told on HN that Russia is trying to set up something like that [https://news.ycombinator.com/item?id=19521584](https://news.ycombinator.com/item?id=19521584)

</comment></comment></comment></comment></comment><comment><USER7> One of my long-idle projects is oddly popular in China (it's about 30 lines of shell script total) so I've been adding pro-Tibet and pro-Taiwan messaging. Figured it's worth using it for _some_ good.

Anyone have a suggestion for a good pro-democracy message that could be targeted at Chinese developers?

<comment><USER8> I find it concerning that someone might think it's appropriate to add political messages to software projects. Imagine if some software developer added a "fuck Bernie Sanders!!!" remark to the README of a popular repo. Just my two cents.

<comment><USER7> Well if Bernie Sanders becomes a totalitarian dictator that starts suppressing democracy, I'll be more than happy to add "fuck Bernie Sanders" to all my repos.

<comment><USER9> who defines what a totalitarian dictatorship is? you? Here is an example:

If you ask most christians (or minorities) in Syria what they think of Bashar´s leadership, they´d hail him as their christ and savior. Meanwhile, for us here in the west he´s a dictator and a murderer.

I am not justifying any actions or atrocities committed by such an actor. I am merely point to the fact that if you want to politicize an aspect of our lives, you need to at least have rigid and firm definitions. Otherwise, all you end up doing is giving ammunition to your opposition to do the same. Ex. I can add the same about Trump now into my repos and call him a fascist, or do the same thing when Obama was in office. Or heck, imagine having "LOCK HER UP" at the top of each ReadMe.

That cannot possibly be a healthy way of going about things.

<comment><USER7> I'm more that comfortable with my stance. You're invoking the slippery slope.

<comment><USER10> That's not "the slippery slope", it's a valid point.

Yeesh not everything has to be a fallacy, and honestly if even if it had been he still had a point, a fallacy doesn't magically invalidate the argument being made.

</comment></comment></comment></comment></comment></comment><comment><USER11> Text-only version: [https://text.npr.org/s.php?sId=709490855](https://text.npr.org/s.php?sId=709490855)

Also, previous discussion of 996.icu: [https://news.ycombinator.com/item?id=19507620](https://news.ycombinator.com/item?id=19507620)

and about Chinese-made browsers blocking the GitHub repo (not 996.icu itself): [https://news.ycombinator.com/item?id=19553681](https://news.ycombinator.com/item?id=19553681)

</comment><comment><USER12> I think all this will do is eventually lead to China copying repositories from github that they agree with, make their own internal repo system and block access to the actual github.

</comment><comment><USER13> Github's has also become a haven for starred repositories gamed from China.

<comment><USER14> Could you please explain what you mean? Or what the benefits would be?

<comment><USER15> If you have a 1k+ star repo, you might get a higher salary when finding a new job

</comment></comment></comment><comment><USER16> Sometime in 2018 or '17 I've read that GitHub is again blocked and programmers have to switch jobs because it's pretty hard to do programming without GitHub now. So, _they_ can come after your repos too.

(The source of the info was a Chinese redditor who makes Arduino-like robots, so I hope she knew what she was talking about.)

</comment><comment><AUTHOR> Prior discussions (didn't notice these when I submitted this story):

[https://hn.algolia.com/?query=996](https://hn.algolia.com/?query=996)

</comment><comment><USER17> I'm surprised Microsoft-owned GitHub hasn't started censoring yet. I suspect it's only a matter of time.

</comment><comment><USER18> Headline from next week: "GitHub _Was_ a Haven for China's Censored Internet Users".

</comment><comment><USER19> Microsoft is in a really awkward position with this now, curious how it will play out

</comment><comment><USER20> I subscribe to a couple of newsletters that send the most popular repositories in several languages to my email every week. There's been a trend of increasing Chinese activity for months now. The problem with Chinese programmers is that they very rarely bother to use English, as the rest of the world, rendering their technically "open source" code largely useless for most of us. There's clearly a niche for a Chinese language counterpart of Github, and I suspect it's going to be filled soon.

<comment><USER11> Chinese programmers use English at about the same frequency as programmers from other non-English-speaking countries, but there are so many of them that the minority using Chinese for everything is still very visible.

FWIW, most of the Chinese-only repos I've seen on GitHub are tutorials, other study materials or social gathering places like 996.icu, so I don't think you're actually missing a lot of relevant open source code.

The most likely contender for a Chinese-language GitHub-replacement is probably the hosted GitLab instance offered by Alibaba: [https://code.aliyun.com/explore](https://code.aliyun.com/explore)</comment></comment>


Coronavirus Assistance Site – Looking for Volunteer Firebase, React and Web Devs - <AUTHOR> https://www.cv19assist.com/ <comment><AUTHOR> FYI, there is a project in github where I am starting to add specific asks. Any help would be appreciated! [https://github.com/CV19Assist/app/projects/1](https://github.com/CV19Assist/app/projects/1)

</comment><comment><USER2> Hey, I'd like to help. How can I contact you?

<comment><AUTHOR> OP here, please join our slack at [https://join.slack.com/t/cv19assist/shared_invite/zt- crw9dwq...](https://join.slack.com/t/cv19assist/shared_invite/zt- crw9dwqs-y~VZC6fwXxuqP9ojpF19EA)

<comment><USER2> okay,thanks</comment></comment></comment>


Bitcoin: A call-to-arms for technologists - <AUTHOR> http://www.jperla.com/blog/post/bitcoin-a-call-to-arms-for-technologists <comment><USER2> "Bitcoin is based on math and deep computer science."

The tech certainly is, but the value is not. The value is based on simple supply and demand. Currently, because the price is rocketing, there's huge demand. That _will_ end when people get scared of the bubble bursting. Then the price will tumble. This is Economics 101 stuff.

The difficulty in creating an economy is not tech. That's relatively easy. The hard bit is creating a system that people can trust in the long term. That's not something anyone is going to crack overnight.

<comment><USER3> Exactly. It doesn't matter if it was based on puppies and rainbows, the valuation of it is entirely based on speculation. There's zero intrinsic value to a Bitcoin.

The "math and deep computer science" part is mostly an anti-cheating mechanism. If it was too easy to produce Bitcoins, they'd be worthless. If it was too hard, nobody would care. The mining process is an attempt to balance between these two concerns.

In any event, even debased fiat paper money can be burned to heat your house. What exactly can you do with a mostly random string of data, anyway?

<comment><USER4> Be careful when you use words like 'intrinsic.' It may be argued that fiat- currencies are 'intrinsically' without value either.

This 'anti-cheating' mechanism is a big part of what makes a currency valuable.

<comment><USER5> > Be careful when you use words like 'intrinsic.' It may be argued that fiat- > currencies are 'intrinsically' without value either.

No, that's not arguable, its part of the whole point of fiat currencies is avoiding intrinsic value and replacing it with fiat.

Commodity currencies rely on the intrinsic value of the commodity to provide certainty that the currency will be exchangeable for goods and services (providing a lower limit on exchange value.) They also rely on the inherent supply characteristics of the underlying commodity to control the supply of the currency.

Fiat currencies rely on the issuing government (or participating governments, in the case of something like the Euro) accepting it for taxes, fines, and government services for the same purpose, as paying some or all of these is common enough that it provides demand for the currency, and rely on active management by a central bank to control supply of the currency.

Bitcoin's mechanism for control of the supply of the currency are algorithmic, and much has been said about them, but the value basis is ... less clear.

</comment></comment></comment></comment><comment> Double-O Fortunately or unfortunately, the potency of a legal tender is not a matter of "deep math or deep computer science" but a matter of confidence and scarcity. That is all.

In the past, people bought things with precious metals. These had to be scarce to be of value. This is the basis of modern day "money or currency". Let's take the dollar for example. Because it does not physically exist and is rather made-up, it is scarcer than gold or silver, hence more conceptually sound and more highly prized. And because, somehow, there is broad agreement on the dollar's tenability, it works so well. These two factors -- confidence and scarcity -- are the fundamentals of supply and demand. They are the only things Bitcoin must fulfill to take hold as a broad legal tender.

Today, Bitcoin is rather scarce because its creation is exceedingly computationally expensive and arcane. This meets condition one. Condition two, confidence, is not yet satisfied today by Bitcoin. Broad confidence takes time. Until both are met and sustained for a long and tried period of time, Bitcoin will not gain broad acceptance. This means Bitcoin very well could.

</comment><comment><USER6> In the end it doesn't matter what your currency is made of, people are going to trade on credit mostly. The creation of fiat currency in the first place was less an attempt to scam the world into devaluing its currency than it was to bring the physical medium in line with the way the economy works anyway.

Many say that one of the biggest advantages to bitcoins is the anonymity of the currency, but in the end, it'll probably be its downfall as well. When people can't place a name and location on their debtors, loss aversion tends to kick in.

</comment><comment><USER7> Just because Bitcoin is based on solid cryptographic principles doesn't mean it's based on solid financial principles. This is a common mistake I have been seeing all along.

Your friend is right.

</comment><comment><USER8> one criticism i've heard of bitcoin is of its independence of influence from monetary policy is also its achilles' heel. there's no mechanism to alter supply subjectively and the global supply is fixed to a logistic curve.

a problem arises where when one loses/destroys their bitcoins, there is not enough counter measure to recover the lost supply. it's like burying treasure with a sealed guarantee that the value of it will be gone forever-- an unsoundly characteristic of any currency with high ambition

</comment><comment><USER9> Let's not pretend bitcoin is based on "deep computer science" - it's based on fundamental macroeconomics and our understanding of how currency works.</comment>


Celebration with 10000 Kubernetes Pods - <AUTHOR> https://blog.phuaxueyong.com/post/2020-05-24-kubernetes-celebration-and-learning <comment><AUTHOR> I saw this interesting youtube video of a guy celebrating his milestone. Decided to share some of my learnings from my daily interaction with kubernetes as well. What are some of your learnings or pain points of using this tool?

[https://blog.phuaxueyong.com/post/2020-05-24-kubernetes- cele...](https://blog.phuaxueyong.com/post/2020-05-24-kubernetes-celebration- and-learning)</comment>


Ask HN:  Getting information about a URL or domain from other web services - <AUTHOR>

I'm midway working on a sort of "reverse look-up" tool for domains and URLs.  The idea is: you enter a URL, and then choose from a big list of actions.  These actions should utilize the huge gamut of web services available, and should merely forward you to some other webpage.
For example, a user might type in "news.ycombinator.com" and an action could be "Find similar websites to this domain using Google," which would forward them to http://www.google.com/search?q=related:news.ycombinator.com
I'd like to get some ideas on which categories and/or specific web services you guys think would be useful.  It doesn't have to be particular to any topic at all... anything goes.  It should be something you would personally use or that the general population would find useful.
Here are some example actions I've thought of myself:

  - Quantcast, Alexa, Compete, show stats on that domain
  - Look up that domain on CrunchBase to view information on the start-up behind it
  - social media submission (submit to Digg, Reddit, etc)
  - find highest ranked Dugg/Reddit stories from that domain
  - Find similar webpages to the URL or Domain (using Google 'similar' feature)
  - translate page with Google Translate
  - shorten the URL using bit.ly et al
  - view tweets containing that URL using Twitturly
  - search for blog articles linking to that URL using Sphere, Technorati, SocialMention, et al
  - what was the website built with, using BuiltWith.com
 Hopefully you guys can fill in the blanks.
Thanks! <comment><USER2> Here are the results for this page using the Site Information Tool extension I have in Firefox: [http://www.wmtips.com/tools/info/?url=http%3A%2F%2Fnews.ycom...](http://www.wmtips.com/tools/info/?url=http%3A%2F%2Fnews.ycombinator.com%2Fitem%3Fid%3D464724&src=ff&v=1.2)

<comment><AUTHOR> Thanks. Another great one is DomainTools.com: <http://whois.domaintools.com/news.ycombinator.com>

Domain look-up services I've got pretty well covered. Any other services you use?

</comment></comment><comment><USER3> Reverse IP look up can be helpful, as well as info from the Who Is database. Check out abouthisite.com, they have some functionality along those lines.

I would also recommend using Yahoo Site Explore and the Google link: operator to look at back links.</comment>


Ask HN: Your relationship or your dream? - <AUTHOR>

I created a startup that has been growing for 6+ months. Just voted a top small startup in a major US city, building a respectable customer base, getting press, and we have plenty of non-zero numbers. Myself and 2 team members now. Truly love this and want to do it everyday.
Then last night, my partner of 9 years was given the exciting news he was accepted to medical school across the country. Great school just not in a tech hub or a place the startup can continue to grow. The startup needs to stay in the city I'm at for a while. The team, networks, mentors, local buzz, etc are all here.
We talked about it and are committed to making this relationship work long distance, but we also talked how the odds are not in our favor. I'm worried it will come down to a choice between his dream, mine, or ending our relationship.
Lots of thoughts/emotions going through my head right now, any personal anecdotes are truly appreciated. <comment><USER2> It's hard to be in a relationship with two go-getters. I think you already know what it is that you want to do. Long distance relationships are hard, but if you are committed then it will be okay.

How many years are we talking here?

<comment><AUTHOR> 4 years of medical school, 3 years of residence to follow (likely in another city of the US). For the startup, probably needs another year or two before the team can start working remotely.

</comment></comment><comment><USER3> Long distance relationships are not impossible. I found the key to make a vow of daily contact, even if its for less than a minute.

Best of luck :)

<comment><AUTHOR> That's fair and makes sense. Thank you!</comment></comment>


Goldman Sachs Created a Font, but You're Forbidden by Its License to Critique GS - <AUTHOR> https://www.techdirt.com/articles/20200625/11063644781/goldman-sachs-created-font-you-are-forbidden-license-to-critique-goldman-sachs-using-it.shtml <comment><USER2> This license doesn't work in Japan for the result of font rendering is not protected by the copyright.

It was decided in the court that in order the shape to be recognized as a character, that shape must follow the pattern we generally agree upon. The copyright is granted for the expression that reflect the human thought, if the shape can be recognized as a character, there is limited human thought left.

I think that's a fair argument.

<comment><USER3> Font copyrights are connected to the embedded hinting program.

<comment><USER4> My knowledge in font copyrights is limited, but what if one just removed the embedded hinting programs? Sure, it'd look bad, but would that fix the copyright issue?

<comment><USER5> More or less - this kind of thing does happen, but not by as technical means as extracting the hinting from the font (because you would need to provide new hinting anyway for it to look any good, and you get into a complex knowledge firewall situation doing that).

Instead, it's common for font foundries to design a new font based on the result of an existing one, creating an effective duplicate font divorced of copyright claim. It's pretty common for a few differences to be inserted though, I think less out of concern over copyright and more so that it's possible to determine which font was used for license enforcement of the new one.

This prominently happened with Arial, which is a copy of Helvetica designed by Monotype so that they could stop paying fees to license Helvetica from Linotype as part of their font offerings for early raster printers (back when fonts were embedded in the print engine). There are numerous other examples, to the extent that if one foundry offers a font which is particularly popular, the other major foundries probably also offer their own "version" of it.

</comment></comment></comment></comment><comment><USER6> Did GS pull the page touting their font? Or is my browser acting up?

[https://design.gs.com/d/story/goldman- sans/](https://design.gs.com/d/story/goldman-sans/)

<comment><USER7> > Requests to the server have been blocked by an extension.

This is after disabling all of my extensions, including uBlock Origin.

<comment><USER8> I've come across a couple of instances of this over the past couple of weeks. Not sure what to make of it.

</comment></comment></comment><comment><USER9> > (C)(2)(d) The User may not use the Licensed Font Software to disparage or > suggest any affiliation with or endorsement by Goldman Sachs.

> (E)(2) Further, Goldman Sachs may terminate this License, without notice to > the User, for any reason or no reason at all and at any time, completely at > Goldman Sachs's sole discretion.

<comment><USER10> > The User may not use the Licensed Font Software to disparage or suggest any > affiliation with or endorsement by Goldman Sachs.

I would like to make clear that although I may disparage Goldman Sachs this does not mean I disparage any affiliation with them.

<comment><USER11> I think it's fairly clear that it's

> disparage or (suggest any ((affiliation with) or (endorsement by))) Goldman > Sachs

Not

> (disparage or suggest) any ((affiliation with) or (endorsement by)) Goldman > Sachs

And GS has more lawyers than you.

<comment><USER10> >And GS has more lawyers than you.

yet they're still not able to write a contract punctuated so as to avoid ambiguous wording.

</comment></comment></comment></comment><comment><USER12> The difficult to distinguish 0's and O's are still the worst part of this. It's supposed to be for numbers focused things, come on.

<comment><USER13> > It's supposed to be for numbers focused things, come on.

It's not supposed to be for programming. If the 0 and O glyphs were identical, where would that confuse a reader, in a numbers-focused document?

<comment><USER14> Contracts. I've run into the issue trying to reference paragraph 2O vs paragraph 20. (Paragraph 2 is often definitions and tends to get lengthy.)

<comment><USER13> That's a good example. It makes me curious about such nested references, though -- in my mind, they usually look something like 2.III.O, or 2(III)(O), or the like.

<comment><USER14> That's definitely how they would usually be referenced but when it's crunch time and you're firing emails back and forth trying to hammer out the details that tends to fall by the wayside. Especially on mobile. And 99% of the time it's clear enough. 2(B) or 2B doesn't leave much room for confusion but oh man did those missing parens on the O cause us a lot of confusion about version control.

</comment></comment></comment></comment></comment><comment><USER15> Apart from its strange provenance, I thought it is a beautiful typeface. Something ugly gave birth to something beautiful.

<comment><USER16> and then wrapped it in an ugly license blanket

</comment></comment><comment><USER17> How to get people to use your typeface to disparage your company 101

See also: Striesand Effect

</comment><comment> awinter-py philosophically interesting concept -- a typeface that cannot express certain concepts

GS being boring and doing it w browsewrap, but I wonder if you could go farther and use ligature rules to warp certain words

<comment><USER18> It's been done!

Scunthorpe Sans censors out profanity: [https://vole.wtf/scunthorpe- sans/](https://vole.wtf/scunthorpe-sans/)

Seen censors "spook" words: [http://projectseen.com/](http://projectseen.com/)

<comment><USER19> > Scunthorpe Sans censors out profanity

Failed at the first attempt. You'll need more than that to stop a mischievous primate

<comment><USER20> Read the page before advertising your ignorance of it.

<comment><USER19> Well. The idea is a simple one, No exactly rocket science. I assume that anybody can grasp it even if just don't care about studying the metrics and full specifications manual book.

My bet is that will not work as expected and will be easily abused by humans. Based in my experience tricking the scunthorpe font in the first attempt

What really would work would be stopping overreacting about put-your-fave- letter-here words and make it boring again. The more outrage they cause, the most tempted will be people to use for fun and profit. I understand in any case that some cultures are more direct than other when communicate.

</comment></comment></comment></comment></comment><comment><USER21> [https://news.ycombinator.com/item?id=23616255](https://news.ycombinator.com/item?id=23616255)

</comment><comment><USER22> Does anyone else notice this thread's font sizes are different per post, seemingly at random?

<comment><USER22> Sorry it's a weird render that is only happening in landscape on iOS (XR) Safari

<comment><USER4> This is a known issue with iOS[0]. Basically, to keep you from having to zoom in when mobile browsing first took off, Mobile Safari would make the text bigger.

[0]: [https://stackoverflow.com/questions/3226001/some-font- sizes-...](https://stackoverflow.com/questions/3226001/some-font-sizes- rendered-larger-on-safari-iphone)</comment></comment></comment>


Supreme Court remands raisin growers' case to the Ninth Circuit - <AUTHOR> http://www.law.cornell.edu/supremecourt/text/12-123

<comment><AUTHOR> Previous coverage on HN: [https://news.ycombinator.com/item?id=5476048](https://news.ycombinator.com/item?id=5476048)</comment>


SWiSHzone is now CLOSED - <AUTHOR> https://swishzone.com/ <comment><AUTHOR> So sad, SwishMax was so simple and intuitive alternative to flash, and you can use it in a so big range of tasks. Internet evolution leaves somes deep wounds...</comment>


Manage.py, a human friendly CLI builder for python apps - <AUTHOR> https://github.com/Birdback/manage.py

<comment><USER2> This kind of "hey I will parse your CLI out of some callables' argspecs" has been done _many_ times before:

\- <https://github.com/fritzo/parsable.py>

\- <https://github.com/gissehel/cltools>

\- <https://github.com/piranha/opster>

\- <https://github.com/kennethreitz-archive/argue>

\- <https://github.com/pdubroy/simpleopt>

(all of those found from a 30 second GitHub search, so apologies if I've left one out or if one of those isn't exactly alike).

The thing is, the fact that you cut out some code does not make your thing more Pythonic, or simpler, or easier to use _by itself_. It's definitely a start, but CLIs are hard. And while argparse has its issues, if you try to supplant it, I don't doubt that you can get the simple cases right, but it's the more complex composition of components that is hard to do, and that's why I find anything in this realm hard to latch onto.

docopt is a nice step in a different direction, and I use it occasionally, but it's hard to do this -- for the simple cases argparse _really_ isn't that bad, and I really really don't think that magically introspecting callables is a good approach. You _will_ at some point want something more complicated, or to prevent some magic from doing somethings, and that kind of interface _will_ become awkward, inextensible, and painful. Or at least that has been the case so far for anyone that's tried this.

So, applaud any code that anyone's sharing, apologies for coming down hard, but I don't think this is something anyone should use for anything but a trivial "hey here's my CLI it's a one off, and I don't want to learn argparse [which takes an hour or two]". And even for that, docopt is probably a better option.

<comment><USER3> I think the most popular is <https://pypi.python.org/pypi/Baker/>. And I like automatic introspection of callables very much - no more argument parsing, just a decorator under a function. Sure this is not suitable for polished CLI interfaces, but my use cases are almost only in-house scripts for internal usage.

<comment><USER4> +1 for Baker, it is the first script I import in my git repo when writing a python tool.

A big plus is that it is a single file (baker.py) with no dependencies, which is very useful when I have to use my scripts on machines I don't have root access on. (I know about virtualenv, but it is definitely overkill for most of the things I do)

</comment></comment></comment><comment><USER5> I would suggest that you change the name -- manage.py is associated with django, I think... for most people in python land

<comment> nezza-_- Yes, definitely. People who google for "python manage.py" are looking for django-related stuff.

</comment></comment><comment><USER6> I prefer to use the batteries Python ships with; argparse is NOT that hard to understand. I haven't seen anyone that says otherwise who isn't just trying to pimp their side project.

<comment><USER7> ...and?

You're in the wrong part of the Internet if "I made a tool that builds on something that already exists and is probably good enough for the smart person's needs" makes you shrug.

</comment></comment><comment><USER8> How does this compare to Fabric? Fabric offers a load of extra stuff to do with SSH/Deployment, but just looking at the basic features of running Python tasks from the command line, why should I use this over Fabric?

Also, why use `manager.arg` for documenting arguments, why not just use Python docstrings?

<comment><USER9> I was wondering the same thing. I also strongly associate manage.py with Django, so it confused me a bit when I first read it.

</comment></comment><comment><USER10> Why this? [https://github.com/Birdback/manage.py/blob/master/setup.py#L...](https://github.com/Birdback/manage.py/blob/master/setup.py#L17)

Would be more straight forward if you would remove that and let users just run python manage.py or ./manage.py

<comment><AUTHOR> Then just add:

if __name__ == '__main__': manager.main()

to your manage.py

</comment></comment><comment><USER11> My personal favorite is docopt: <http://docopt.org/>

</comment><comment><USER12> Human friendly and CLI - sounds little bit like a oxymoron.

<comment><USER13> Not when you consider which subset of humans the tool targets.

</comment></comment><comment><USER14> I wrote a little language with a similar goal a while back: <https://github.com/pbiggar/buggery>

</comment><comment><USER15> argparse is pretty easy to use. I was able to build a git-like command line app with subcommands quite easily. Not sure why it needs a replacement.

</comment><comment><USER16> You can do much of this with Fabric, specifying remote and local hosts. Cool piece of code though.</comment>


Bain Capital, Andreessen Horowitz, Polychain Invest $133M into Basecoin - <AUTHOR> http://fortune.com/2017/10/16/cryptocurrency-basecoin-investment/ <comment><USER2> Economist Bernard Lietaer proposed a currency similarly based on a basket of commodities called the Terra back in 2001, but could never get it off the ground. Would be interesting if this turned out to be equivalent.

[https://en.m.wikipedia.org/wiki/Terra_(currency)](https://en.m.wikipedia.org/wiki/Terra_\(currency\))</comment>


Ask HN: What engineer job involves a lot of sales? - <AUTHOR> I'm trying to find a programming job where I can also develop some sales skills (ideally enterprise sales). Are there any such jobs out there? <comment><USER2> Yes, usually they are called pre-sales tech, or solutions architect, integration consultant etc. Solution architect can also be an inside role too so you have to check how that JD is written.

Usually what you want to do is get in to one of the pre-sales teams so you are part of the sales process but can still stay technical. This will let you see the sales process and start developing new skills. This is usually focused on the enterprise market where the pricing justifies a more expensive sales process.

You can also do this at a startup but it won't teach you sales typically. Startups are learning how to sell themselves usually, so they don't have a mature sales process or people so if you are wanting to develop those skills go to one of the big consulting firms, or CRM companies etc.

<comment><AUTHOR> True though a lot of those roles would be a large demotion for me

<comment><USER2> If you are looking to learn new skills usually the fastest and most successful way is to make yourself somewhat uncomfortable which helps motivate you (e.g. take a less senior role, or less money at first). If you want to learn enterprise sales you need to dedicate real time to that process, it can be a super intense and sometimes frustrating process which requires time in the seat, so generally people have to immerse themselves into the process to really learn it. There are other ways to get some basic sales skills, but you asked what roles would help you develop the skills and still write code, these are the best roles for it that I know of which fit your basic criteria.

I am curious why you feel it is a demotion? Money, title, prestige or ? Not judging, honestly curious.

<comment><AUTHOR> Mostly money - don't really care about title or prestige

<comment><USER3> A lot of sales engineer jobs pay well... but often in sectors like electrical engineering or medical.

It's because the skillset is rare; hospitals aren't going to buy some medical equipment from some sweet talker in a suit. They prefer a nerd who understands exactly what they're selling and what the advantages and disadvantages are, maybe even explain training needed. I had one oscilloscope sales job offer, where the boss insists that the hardest part is understanding the tech, and the sales part is easy to train.

Many sales roles give commissions, which can come up to a hell lot.

Software probably doesn't involve the dirty details anymore, but I imagine things like selling a database, AWS-type service, military technology, machine learning, and so on.

</comment></comment></comment></comment></comment><comment><USER4> Yes, become a co-founder or act as one by entirely, completely, unapologetically owning the product by going way beyond the distance regular people go: code, architecture, features, marketing, UX, the business side of things, the pitch, legal, relations with prospects, knowledge management, documentation, building tooling for colleagues, building deployment pipelines, making it easier for co-workers to do their work, disseminating knowledge, living good engineering practices, enabling colleagues in areas where you're better, simplifying onboarding, training documents, sales copy, story of the product, _everything_ and I mean _everything_ down to the minute details legal documents have or the phrasing of responses to non-pursuit towards job applications. Own everything.

As far as I can tell, many things are limited only by the amount of complexity people are willing to handle and the workload they can support. If you're okay pulling a work-day with co-workers and then another one when they leave, then we understand each other. Some people can pull off 16 hour work days or more, even during the week-ends, even during holidays and think about the product _all the time_. Some are great and can do that in 4 hours, I personally can't.

Find things that suck, fix them, institutionalize that knowledge, document everything, prepare for the eventuality of your disappearing, build an organization, train people to think like that, read a whole lot, become good a things, add a lot of value, become someone you would fear if they were a competitor, have very high expectations of yourself and be afraid to disappoint yourself tomorrow, experiment, demonstrate by building things and implementing things, not just as "ideas" but actual outputs, either codes/graphics/copy/documentation/training manuals/videos. Go for it.

</comment><comment><USER5> Sales Engineer is a very real title and role. Basically, you work with salespeople to spec out a system and help the customer understand what would be involved in rollout and how their system would be integrated with yours. This is most common in enterprise sales.

</comment><comment><USER6> Programming is usually very far from sales, I haven't seen such a position yet (well unless you're a founder as somebody else mentioned). However, as others mentioned already, you could be a technical salesperson. It does not involve programming, but you need to understand the tech well enough to be able to talk about it with actual developers on the client's side.

</comment><comment><USER7> Typically, you will look for a "Sales Engineer" role where you are involved in Sales process discussing detailed technical requirements with the prospects.

</comment><comment><USER8> Consultancy work, especially if you are at senior/lead/principal level in many cases involves sales work.

</comment><comment><USER9> Try Searching: jobs, engineering + enterprise sales + Chicago

*Yes, I looked you up. I'm that good.

Promise not a stalker!

<comment><AUTHOR> What did you look up exactly?

<comment><USER10> I imagine he found your tweet to producthunt on 8/27/18 where you say you're in Chicago

</comment></comment></comment><comment><USER11> Founder.

<comment><USER12> I was going to say the same thing - Sales Engineer at a B2B startup is probably the next best thing.

<comment><USER11> Similarly, I think Customer Success is also sales-related and often requires developer skills.</comment></comment></comment>


Train machines like it's 1979 Gesture controlled Atari 2600 - <AUTHOR> https://github.com/nickbild/vectron_ai <comment><USER2> As if repetitive thumb stress wasn't pain enough, now you can have shoulder and arm pain from playing 2600 games!

Gesture systems seem clever until you have to hold your arm up for hours on end to use them.</comment>


Interview with CTO of Buffer, Sunil Sadasivan - <AUTHOR> https://www.bugtrackers.io/interview-buffer-sunil-sadasivan <comment><USER2> He mentioned in Interview that they have only 500,000 access tokens but assuming the user base of buffer it should be more.</comment>


Yes, Python Is Slow, and I Don't Care - <AUTHOR> https://hackernoon.com/yes-python-is-slow-and-i-dont-care-13763980b5a1 <comment><USER2> The idea that a certain programming language could be proven to be more productive than another is laughable. There are so many factors involved in that, I can pretty much guarantee you it will never be studied to an acceptable level of rigor.

The author presents some flimsy "study" about coding string processing algorithms with a straight face, and basically loses all credibility. What about lifetime productivity of a codebase? Speed of refactoring, onboarding, testing etc..

Why don't you just be honest and say you don't care Python is slow because you like using it?

Performance doesn't matter? Of course it does. On the frontend, bad performance has been proven by many eCommerce companies to lead to lost business. On backend data pipelines, bad performance kills your flexibility. The longer it takes you to process N, the less times you can do it. The more times you can do it, the more times you can get it wrong.

Also who says that time-to-market is everything? Tech history has shown that product quality/brand marketing win out in the long term. There are tons of forgotten shitty products that were first. Being first is over-rated.

<comment><USER3> _"The idea that a certain programming language could be proven to be more productive than another is laughable."_

I disagree. In the extreme, take python on the one hand and a variant of brainfuck that requires the hash (could be as simple as CRC-32 or as hard as SHA-256) of every source file to be zero on the other.

I don't think you have to go that far to get provable differences, though. Some features that, I think, can fairly easily be proven to make programmers more productive in large parts of the space of programming tasks:

\- assembler macros

\- variable names that can have more than two characters

\- direct support for floating point

\- support for complex numbers

\- local variables

\- structures

\- support for serialization and deserialization. Compare the way one can make simple structures serializable using Swift's Codable protocol to whatever is 'best' for C (probably something macro-based)

\- generics

\- support for Unicode strings

\- iterators

Of course, "python vs Java" or "python vs Perl" are much closer races where I expect measuring differences is way harder and results will depend on problem domain, experience of the subjects of the study, number of good and bad answers on StackOverflow, etc, but even for 'normal' languages, I expect some conclusions to be possible, given enough effort is spent. Python 1.0 didn't have complex numbers, for example, so if your program needed them, python 1.4 was a huge improvement.

I doubt evaluating productivity of languages is worthwhile, though.

</comment></comment><comment><USER4> But python is both slow and inexpressive. No types, no match statements, no easy FP. Also the language is poorly designed and inconsistent.

I'm practically forced to use python for quant finance at my job because of the network effect: numpy, pandas, cvxpy, etc.

But the language itself is total garbage compared to, say, Scala or OCaml. And both of those languages are also much faster.

<comment><USER5> That is my experience as well - the only redeeming feature of the language is the vast selection of easy-to-use libraries. The speed that the language sacrificed isn't some inherent trade-off, but completely orthogonal to what makes the language useful.

<comment><USER6> You do realize that "vast selection of easy-to-use libraries" is the most important feature of a programming language. Right? It's up there with, "it's not hard to hire programmers for the language", which Python also benefits from.

Like, I'm partial to Haskell, but if I'm actually trying to get work done, I reach for a language with a robust ecosystem. These days I reach for Typescript, but before then, python was my go-to since it's way nicer to program with than java/c++/c/javascript.

<comment><USER7> >You do realize that "vast selection of easy-to-use libraries" is the most important feature of a programming language.

No, it's not. When I switched from Python to OCaml it felt like blessing, since the quality of the libraries felt much higher in OCaml. There are also the projects that you have to build from the ground up without libraries (any mission critical software), and in such cases you don't care about libraries at all, you care about the merits of the language solely.

Python is the BASIC of the XXI century, and it's fine in that role, but not as a general purpose language.

</comment></comment></comment></comment><comment><USER8> One big factor in performance is whether a process can be parallelized. It can be an interesting computational problem, and these techniques has really gained relevance lately now the the barrier to clusters has dropped (cost and technical complexity). A lot of algorithms (say for solving a massive optimization problem) that were 10x slower in total operations but can be split up and solved in arbitrarily small pieces are suddenly immensely valuable. And in this context, the actual speed of the programming language probably isn't terribly important.

</comment><comment><USER9> "Python is more than 2x as productive as Java in the above study."

How about compared to a functional language like Haskell, OCaml, or F#?

I agree that we should choose productivity over performance when possible. However, Java, for example, is extremely verbose.

My impression is that people who use functional languages are quite productive.

Plus, they need fewer unit tests.

<comment><USER10> I'm not sure if people using functional languages are more productive. One of the points Uncle Bob makes in his talk supporting functional programming is that it is harder and slower to write:

[https://youtu.be/7Zlp9rKHGD4](https://youtu.be/7Zlp9rKHGD4)

That said, I have to wonder if these dynamic scripting languages are truly enabling productivity or if it just feels like that because you can write it and try many things quickly. It reminds me of being such in traffic on the highway - sometimes I get off and take back roads even if my GPS tells me it will take longer, because it is more pleasant to be moving than feeling stuck.

That said, I think the biggest benefit of dynamic languages is that they are so easy to refactor. You don't have as rigid a structure, so you can try easily hack together some idea that, done "properly", would require a big refactor. Despite being functional, Haskell's strong type system makes refactoring a huge pain. If course the same things enabling this quick refactoring in the dynamic languages works against you when it comes to speed and reliability.

<comment><USER11> I sit with an FP expert (he might deny it, but he is). What I find, is that he takes a lot longer to get to the solution that I hack out in a day in Python3 (with his assistance most of the time) but his solution has three interesting properties.

1) its usually founded on a very good understanding of the problem, and the abstractions inherent in the problem. If he can re-use the abstractions he has used before he does, to great effect in speed-to-solution

2) it is very usually correct. it is remarkably bug free because the Haskell typing removes a huge amount of ambiguity about the problem going in

3) he has absolutely no problem explaining what it does 3 months or more later.

I value these behaviours immensely. My own Python is usually buggy, often re- produces techniques I did a week ago, badly, and I can't explain why I used that lambda/map function.

<comment><USER12> It is really cool when you can find people like this. He sounds class!

<comment><USER11> He is!

</comment></comment></comment></comment></comment><comment><USER13> Thing is, there are many languages expressive as Python, while providing compilation to native code without having to search for additional tooling.

No need to sacrifice performance for productivity.

<comment><USER14> Care to name some examples?

<comment><USER13> Lisp, Scheme, Dylan[1], Smalltalk, JavaScript, Prolog.

[1] - Yeah, I know it is dead now, just wanted to mention it anyway.

<comment><USER15> Is Prolog known for it's performance?

<comment><USER13> Well, it beats Python, given that most well known implementations do have JIT compilers.

</comment></comment></comment></comment></comment><comment><USER16> Great observation, 100% on point. "Microservices are inherently slow. The very concept of a microservice is to break up a boundary by a network call. This means you are taking what was a function call (a couple cpu cycles) and turning it into a network call. There isn't much you could do that is worse in terms of performance. Network calls are really slow compared to the CPU. But these big companies still choose to use microservices. There really isn't an architecture slower than microservices that I know of. Microservices' biggest con is performance, but greatest pro is time-to-market. By building teams around smaller projects and code bases, a company is able to iterate and innovate at a much faster pace. This just goes to show that very large companies also care about time-to-market, not just startups."

<comment><USER17> Except it doesn't scale. Amazon, Google and Netflix (the companies referenced before this quote) can't run their applications on a single machine. Distributed programming is unavoidable at scale.

Also there's a big difference between throughput and latency. Networking might introduce a longer delay than a CPU call, but if each one of those calls does significantly more work, the overall performance of the system can increase.

</comment></comment><comment><USER18> Whether slow code matters or not depends on the scale, because executing code requires power.

If it's a one off or otherwise rarely executed, yeah, it doesn't matter much. But it starts to matter when it's constantly running on a significant number of systems.

Widely distributed frequently running slow code can waste a lot of energy. Cell phones and laptops heat up and have a shorter battery life. Even desktops require more power.

On the global scale, a large number of such systems can waste several nuclear reactors worth of electricity.

10 times slower code might require 10 times the number of servers. Maybe it doesn't matter much when it's just the difference between one and 10 servers. Or perhaps you don't even consume 10% of a single server. But there's a point when the hardware and power costs start to dominate.

The scale matters.

</comment><comment><USER19> Long story in short description:

Python isn't slow. Programming languages aren't slow or fast but interpreters are.

cpython is slow. PyPy interpreter is reasonably faster, upto 8 times.

Recently pypy 7.0 released which brought more hope.

[http://speed.pypy.org](http://speed.pypy.org)</comment>


Widespread gamma error in image scaling algorithms (2007) - <AUTHOR> http://www.4p8.com/eric.brasseur/gamma.html

<comment><USER2> The primary reason for this problem is twofold:

1\. People writing image scaling libraries are big on speed. Many, like libswscale, are intended for realtime video playback on systems without graphics overlay acceleration. They contain truckloads of shortcuts to cheat wherever possible in the name of performance.

2\. A "correct" method requires one of two things:

A. Performing one LUT per input pixel to the tapfilter, plus one LUT per output pixel (see <http://news.ycombinator.com/item?id=1142084>). This completely eliminates any possibility of SIMD. Other LUT schemes are also possible, like LUT[pixelX][pixelY].

B. Converting the entire image to linear 16-bit colorspace internally, then converting it back. This adds a slow preprocessing (and postprocessing) step and halves the speed of image processing.

Since both have huge speed impacts, most people writing graphics libraries don't like to do them.

Oh, and standard GPU-accelerated overlays can't really do either of these.

<comment><USER3> > Since both have huge speed impacts, most people writing graphics libraries > don't like to do them.

Python Image Library does gamma correct rescale by default. I did not realize this until I wrote my own implementation. I thought it would be several times slower than the built in, but it was only 3% slower. Boy was I surprised when I figured out why.

</comment></comment><comment><USER4> I don't know why, but gamma is one of the most confusing topics in all of image processing. Most software screws it up, and I'm not surprised at the results shown. Unfortunately while I applaud the awareness-raising, the explanation here left my head spinning, despite writing code for one of the apps he highlights as doing it right! If you're in the same boat, I'd recommend Poynton's Gamma FAQ:

<http://www.poynton.com/GammaFAQ.html>

<comment><USER2> _the explanation here left my head spinning_

Then here's a simpler explanation:

We need to blend pixel values X and Y.

Normal image-scaling algorithms perform the operation (X+Y)/2, because this is fast.

However, the correct operation is ToNonLinear((ToLinear(X)+ToLinear(Y))/2), because X and Y represent a nonlinear scale, similar to how adding a 60db and 60db sound doesn't result in a 120db sound.

<comment><USER4> The approach that helped me is thinking in photons. 50% in pixel values is only 25% of the photons that 100% in pixel values represents (approximately). Linear light measures just means converting to photons before you do any operations, and then converting them back to the encoding.

I almost wish gamma was a more _obvious_ encoding process, like gzipping a file, rather than a simple function. That would save a lot of conceptual pain.

</comment></comment></comment><comment><AUTHOR> Although the article is dated 2007, the error it describes is still present in every implementation I've tested. There's a test page for browsers at:

<http://www.4p8.com/eric.brasseur/gamma_dalai_lama.html>

<comment><USER5> Are projects like Gimp and ImageMagick not yet aware of it, don't consider it a bug, just haven't gotten around to fixing it, or what? In some brief trawling through bugtrackers and mailing lists I haven't been able to find any discussion on the subject, but it's somewhat hard to search on variations of the widely used terms "gamma", "scaling", and "bug".

<comment><USER4> Part of the issue is that people have been trained by two decades of Photoshop and expect the results of the 8-bit algorithms. Gaussian blur for example looks very different in linear light, with light areas of the image blooming out over the darker. If you've spent years working with the 8 bit gamma space algorithms, that's a bug. Instead, I've ended up implementing both algorithms, usually with the more correct one labeled as 'defocus' or 'camera blur'.

Rescaling and in general resampling is a lot simpler. These days it's pretty inexcusable not to do it in linear space. Here's my own exploration of that problem from a couple of years back:

[http://petewarden.typepad.com/searchbrowser/2008/08/why- font...](http://petewarden.typepad.com/searchbrowser/2008/08/why-fonts- need.html)

<comment><USER6> Because human visual response is non-linear, doing operations like gaussian blur (which is an artificial operation, unrelated to physical phenomena) in linear light space isn't "more correct" in any absolute sense. Calling a gaussian blur "camera blur" or "defocus" (whatever the color space) is just inaccurate and confusing.

Compositing and interpolating should definitely be done in linear space though.

<comment><USER4> > Calling a gaussian blur "camera blur" or "defocus" (whatever the color > space) is just inaccurate and confusing.

Nonsense. When you defocus a camera, you get an effect that's comparable with a gaussian blur performed in linear space. Blurring is a well-understood operation that's existed for a long time in the analog world, before we ever did any digital processing.

<comment><USER6> Well, "comparable" insofar as you don't care about the details, I suppose; you could also call a square or triangle blur kernel "defocus" if you wanted.

A real lens blur doesn't look much like a gaussian blur for someone who is familiar with both and paying attention.

For explanation, see for instance <http://en.wikipedia.org/wiki/Bokeh>

<comment><USER4> Yep, my 'camera defocus' also included arbitrary kernel geometry as an option! I'm not trying to be obnoxious about this, but I really did spend five years of my life coding this stuff for Apple.

</comment></comment></comment></comment></comment></comment></comment><comment><USER7> The video game industry is aware of this problem. Any pre-process mipmaping is being done is done with gamma-corrected downsampling.

</comment><comment><USER8> What the article does not seem to be aware of is that most Photographic workflow products, such as Aperture and Lightroom, are working in a 32-bit colorspace even if you open an 8-bit JPEG. The reason is simply that to work in RAW consistently, your tools need to be working at a higher bit-depth than the existing RAW images you are trying to use.

Some RAW formats are 14 and some 12. High-end Hasseblads are 16. It is not unlikely that future sensors will go beyond 16 bit. Therefore it makes sense to design the workflow around 32-bit processing. Otherwise you are stuck re- architecting your entire workflow. Witness the number of Photoshop filters which only work at 8-bit.

</comment><comment><USER9> Here's another strange effect: on my monitor if I look at the top picture while standing, it's a grey box. If I move my head down and look up at it, the grey is gone and it's a saturated image. I know my monitor's not great at different viewing angles, but this surprised me.

<comment><USER6> Yes, most LCDs are still pretty bad at keeping colors constant with viewing angle. To see the effects being talked about precisely, get a calibrated CRT. :)

</comment></comment><comment><USER10> Hrm.. I just tested out Gimp-2.6.8 and Eog-2.28.2 - both had wrong, although cool looking results :S

</comment><comment><USER11> Sounds like a job for a photoshop plugin

<comment><USER6> What? Plugins can't modify the behavior of built-in tools, and are therefore absolutely powerless to fix this problem, which affects all of the compositing of layers in Photoshop, and many of its other tools.

(As for scaling in particular: many plugins already exist using better methods than bicubic interpolation, and avoiding problems like the lack of gamma correction, etc.)

<comment><USER11> plugins cannot modify built-in tools, but they can replace/duplicate their functionality

</comment></comment></comment><comment> k-zed Somehow the whole thing reads like pointless wanking to me.

But I'm not a graphics designer or whatever</comment>


Can large, untargeted conditional cash transfers increase graduation rates? [pdf] - <AUTHOR> https://andrewdustan.com/pdf/motivating_bureaucrats.pdf <comment><AUTHOR> > This paper estimates the effects of a massive, minimally targeted > conditional cash transfer program in Mexico City's public high schools on > graduation rates, test scores, and school choice. Using a difference-in- > differences approach that exploits variation ineligibility between students > and cohorts within a high school, I find that this program had no > appreciable effect on high school completion. The results are sufficiently > precise to rule out policy-relevant effect sizes. Null effects persist for > subgroups that could be candidates for a targeted program. End-of-high > school exam scores are apparently unaffected by the program and effects on > high school choices by eligible students are minimal. There is no evidence > for heterogeneous effects with respect to implicit or explicit cost of > attendance, suggesting that liquidity constraints are not a key driver of > high school dropout in this urban setting. These results highlight the > challenges of using cash to improve academic outcomes in cities.

This is the program which was analyzed: [https://www.prepasi.cdmx.gob.mx/](https://www.prepasi.cdmx.gob.mx/)

Its annual report from 2015: [https://www.prepasi.cdmx.gob.mx/storage/app/uploads/public/5...](https://www.prepasi.cdmx.gob.mx/storage/app/uploads/public/58e/2e0/d19/58e2e0d199e2e116301900.pdf)

</comment><comment><USER2> Is the link correct? It goes to a paper "Motivating bureaucrats with non- monetary incentives when state capacity is weak: Evidence from large-scale field experiments in Peru∗".

<comment><AUTHOR> No, it's the wrong URL. Sorry! I can't edit it, but the correct link is [https://andrewdustan.com/pdf/dustan_prepasi.pdf](https://andrewdustan.com/pdf/dustan_prepasi.pdf)</comment></comment>


Lively.next: A personal programming kit - <AUTHOR> https://lively-next.org/ <comment><USER2> Author here. lively.next is a runtime and development environment. It comes with a graphical user interface and programming tools that allow you to program similarly to Smalltalk and Self, i.e. instead of a edit-compile-run cycle you have an ever running "program" and change various parts of it directly. With that you have access to runtime objects that you can inspect, use for code completion, or modify.

Apart from the UI (inspired by Self's Morphic [1]), lively.next implements a meta system on top of normal JavaScript that makes such things possible as to hold on and inspect system state (variables defined inside a module, mechanisms for redefining things etc).

Also, lively.next runs on any JS system and it's various packages can be embedded, e.g. to create a programming environment for node.js and other such things.

The Lively project evolved over time and lively.next is the newest version:

\- [https://www.lively-kernel.org](https://www.lively-kernel.org) is from the Sun Labs / HPI days (check out the ancient [http://sunlabs-kernel.lively- web.org](http://sunlabs-kernel.lively-web.org), fully SVG based rendering)

\- Lively Web: [https://lively-web.org](https://lively-web.org) A live, programmable wiki (2012-2015)

\- lively.next since 2016 [https://lively-next.org](https://lively-next.org).

[1] [http://ftp.squeak.org/docs/Self-4.0-UI- Framework.pdf](http://ftp.squeak.org/docs/Self-4.0-UI-Framework.pdf)

<comment><USER3> This sounds interesting for personal use, but I'm curious about how you share programs with people who aren't running the whole lively environment? What does that look like?

<comment><USER2> There are two parts to this:

a) How to user graphical constructions (we call them parts) outside of the main framework? We currently work on ways to "snapshot" them in a way that they include all their dependencies. So not just the object (graph) state but really all code dependencies. es6 modules are a big help here since they allow to trace dependencies statically. This is work in progress and we will post updates about this.

The state right now is that you can the HTML view of morphs into web pages, e.g. see [2]

b) Several parts of the system can be used standalone, e.g. you can enable support for Lively tooling in node.js processes or arbitrary webpages. See [https://lively-next.org/worlds/lively-2-webpage](https://lively- next.org/worlds/lively-2-webpage) for a bookmarklet that loads the lively.vm [1] and lively-system-interface on a arbitrary web page: [https://imgur.com/a/2foGR](https://imgur.com/a/2foGR). This allows you to then use the workspaces, system browser, inspectors, etc. on remote targets.

[1] [https://livelykernel.github.io/lively.vm/](https://livelykernel.github.io/lively.vm/)

[2] [https://lively-next.org/worlds/lively.next%20website](https://lively- next.org/worlds/lively.next%20website)

</comment></comment></comment><comment><USER4> This appears to be a system in the Smalltalk lineage but based on Node and JavaScript and currently seems only to target MacOS. It uses Self/Smalltalk's UI system and is based on ideas by Alan Kay (of Xerox PARC fame and one of the creators of Smalltalk). The web interface looks and feels similar to a web- capable Smalltalk variant called Lively Kernel[1] but in a less-mature state of development and with more of a focus on text.

1) [https://lively-kernel.org/](https://lively-kernel.org/)

</comment><comment><USER5> Another semi-graphical live programming environment I guess.

I'm still hoping for [http://witheve.com/](http://witheve.com/) to take off :)

Lively reminds me of TouchDesigner (node based programming with a focus on visuals) [http://derivative.ca/events/2017/AlienCovenant/](http://derivative.ca/events/2017/AlienCovenant/)

</comment><comment><USER6> Seeing this brings back the question that always pops up when seeing smalltalk-esque IDEs; why don't these ever take off? Maybe they aren't solving a specific problem someone has and are being too general? Not flexible enough? Maybe someone who has tried out these systems to solve a specific problem can comment?

<comment><USER7> I guess people are just married to their customized Vim, Emacs, Visual Studio etc. setups which they are using for many years. These Smalltalk-related IDEs are too different and break a lot of old habits.

<comment><USER2> I was myself a fairly dedicated emacs user and what I like so much about the Lively UI is that you a) get all the programmability and customizability that you have e.g. with elisp (all the stuff on the screen can be programmed, keybindings, windows, everything down to the characters in texts) while b) get a user interface that is way more powerful than simple text. It's easy to embed images and interactive objects into normal texts allowing things to be more interactive to use. It's easy to plot and visualize data (interactively, not just a static screen), e.g. see the plotly morph in the partsbin. There is a sketch morph in the PartsBin as well, allowing you to sketch and take handwritten notes, I use this in combination with Astropad. Etc.

All that stuff makes the environment much more convenient to use for general content creation, not just programming.

</comment></comment></comment><comment><USER8> Slightly tangential question: why call it "lively.next" ? What is the .next? I understand including a ".tld" in a name when the TLD is part of the domain name, but "lively dot next" doesn't exactly roll off the tongue and .next isn't even an available TLD?

<comment><USER2> Hoping it will be one at some point :)

We first wanted lively.web but then came the .web tld disaster .

.next was chosen b/c there were / are several versions of Lively over the years with version numbers LivelyKernel 2 [1], Lively4 [2], etc. lively.next should simply signal that this is the ongoing and evolving project in the Lively family.

[1] [https://lively-web.org](https://lively-web.org) [2] [https://github.com/LivelyKernel/lively4-core](https://github.com/LivelyKernel/lively4-core)

</comment></comment><comment><USER9> How hard is it to get this to work with something other than Javascript?

<comment><USER2> Not very hard. Two years ago we did something similar for Clojure: [http://cloxp.github.io/cloxp-intro.html](http://cloxp.github.io/cloxp- intro.html) [https://www.youtube.com/watch?v=qQvvgzvPgQI](https://www.youtube.com/watch?v=qQvvgzvPgQI)

We are currently evaluating options for getting this stuff funded and Python would be a another target language / platform.

<comment><USER10> I can see python being very useful for this, but perhaps ruby would be a better fit? I can't help but wonder if lively.next+ruby on the truffle vm might not be able to steal a lot of attention, and kick off a bit of an "old- style" / message-passing object oriented revival... One can hope...

On The other hand, zope/zodb did manage to get pretty far wrt "magically persistent" object graphs for python.

</comment></comment></comment><comment><USER11> FYI: Theres a large margin of empty space on the left on safari OSX, requiring horizontal scroll to see the site.

<comment><USER2> Thanks, will be fixed!

</comment></comment><comment><USER12> I really want to try this, but it doesn't progress from "loading" on my iPad.

<comment><USER2> Will look into it. For now, most of the UI is geared towards mouse and keyboard, however. I can recommend Astropad [1] or some similar app, I regularly use it and Lively for sketching and notes.

[1] [https://astropad.com/](https://astropad.com/)

</comment></comment><comment><USER13> Can this be run on a non-connected chromebook?

<comment><USER2> We have not added support for this yet but there is no reason why it shouldn't be possible. Generally, the serialized objects can be synchronized into the browser's indexdb (we use pouchdb in the background so this is pretty straightforward) and together with a service worker and some caching this is doable.

</comment></comment><comment><USER14> That frontpage uses alot of words to say absolutely nothing...

Someone care to tl;dr what this is?</comment>


Cleveland.com's 'right to be forgotten' removes names from some crime stories - <AUTHOR> https://www.imediaethics.org/cleveland-coms-new-right-to-be-forgotten-program-removes-names-from-some-expunged-crime-stories/ <comment><USER3> Former news editor here.

Perhaps the biggest danger in this and other similar policies is not how it is applied today -- removing the names of people accused of minor crimes that have had their records expunged seems like a compassionate policy, given the effects it can have on things like employment prospects -- but in how it may be applied in the future.

The idea that a newspaper archive is a collection of living documents, capable of being amended, rather than fixed snapshots of the past, is a fundamental shift.

My concern is that down the line, more alterations will be permitted to be made, and potentially for less reasonable reasons, and then it's only a matter of time before archival information cannot be trusted, because there may be no way to determine what, if anything, has changed.

That didn't used to be the case. When print copies of newspapers were used for library archives, it was practically impossible to alter all extant hard copies.

But nowadays, unless this content gets indexed and preserved by numerous outside entities, it would be very easy for a newspaper to silently modify its own archives.

I would be more comfortable if the newspaper included, as part of its policy, a requirement that a versioning ledger be published along with any altered archival stories, such as:

* On 2018-10-19, this story was edited to remove the name of the suspect, under our "Right to Be Forgotten" policy. A copy of the original story is available for inspection at our office.

That, at least, would allow people perusing the archives to know whether a story had been altered long after publication, and it would give people who _really_ need the information to acquire it (albeit not easily).

<comment><USER4> Paranoid response: if such a versioning ledger were to exist, how could you trust it? The feasibility problem still remains (it is feasible to alter archives that were formerly infeasible) and you now rely on trust that (1) all alterations are accompanied by a versioning notice, and (2) the summary presented in the notice is accurate and complete.

<comment><USER5> >Paranoid response: if such a versioning ledger were to exist, how could you trust it?

Blockchain to the rescue (for real this time, I think, I'm not an expert or a blockchain fanboy). A publicly ledger that anyone can verify hasn't been tampered with is the whole point.

<comment><USER6> Why not just provide a hash for each version of the document?

<comment><USER7> But that just changes the problem to one of archiving the hashes immutably.

<comment><USER8> Which is what blockchain does. We're experimenting with using it to create audit logs of data, which can't be tampered with.

</comment></comment></comment></comment></comment></comment><comment><USER9> The 'right to be forgotten' is backwards. Much better would be a 'right to remember'. The right to be forgotten is a positive right, in that it requires other people to take action (e.g. delist a story from Google) to fulfill it. A right to remember would be a negative right. Like a freedom of the press doesn't require the government to subsidize a printing press or equivalent, a right to remember would just mean that the state can't compel anyone to delist a story, etc.

Positive rights are more open to abuse, and better serve the centralization of power.

A right to be forgotten cannot be applied without infringing on someone else's right to speak about what must be forgotten, as in the case of forcing Google to delist something. A right to remember only infringes on the very dubious "right" to control someone else's speech and/or memory prosthetics, e.g search engines.

<comment><USER10> Yes, this is always my argument against this sort of thing. Enforcing one person's right to be forgotten is infringing on someone else's right to remember.

Say, for example, I get beat up by someone. They are convicted and go to jail, but then in 10 years time they get their record expunged. Am I no longer allowed to tell people about what happened? I should be allowed to tell my story. I should be allowed to tell people what that guy did to me. If I want to write about it, I should be allowed to do that.

It is my right to decide how long I want to remember, and who I want to tell. If I don't want to forgive, and I want to tell people "This guy beat me up 10 years ago, I don't think you should have them around", that is my right, too.

The government doesn't get to decide how long I can hold that grudge.

<comment><USER11> It's not your right to dispense such justice--that in fact belongs to the state in a civilized society. I think this example proves the exact opposite of what you desire. Someone who has spent time in jail and had their record expunged 10 years later has been deemed by the state to have satisfied the burden of their offense, and, yes, their right to have their actions forgiven and forgotten should be at its zenith.

[Edit: Downvotes? How about a response then? What's the purpose of jail if (1) we know it's not rehabilitation and (2) the convicted's debt to society is not satisfied upon serving the sentence?]

<comment><USER12> Where are you getting that serving your time in jail gives you the right to have your actions forgotten? Society 'forgives' you when you serve your full sentence, but there is nothing stated anywhere that your victims need to forget what you did once your sentence is up.

<comment><USER13> The OP article explains the concept of "expungement"

<comment><USER12> Yes - it is one thing to remove it from official records - it is a whole other thing to ask victims to not remember, or at least not bring up, what a person did.

<comment><USER14> If "bring up" means putting it online where everyone with a name and 20 seconds on google will find it, then there isn't really any point to removing something from official records.

That said people wanting to be forgotten tend to just ask for search engines to ensure that their name will no longer be connected to certain pages rather than having the information removed entirely.

</comment></comment></comment></comment></comment></comment></comment><comment><USER15> this is important in the united states, as the criminal justice system is often used as a blacklist. Once convicted of a DUI or drug possession, this criminal record can be used to bar employment, rescind voting rights, and legally practice housing discrimination.

American criminal justice is often a precursor for lasting biblical retribution. For example, sex offenders are often branded with scarlet letter for life even after having been incarcerated and released, presumably rehabilitated and having paid restitution to society. Criminal records are made public and posted to websites. disclosure is often required by law for prior felons.

<comment><USER16> Even more so than the sex offender registry, I think US news reporting is more problematic. I wish the US were more like Germany (I think?) in the regard that they don't identify people by name until conviction. By that time, there's less interest in sensationalizing a story.

<comment><USER14> In the Netherlands it is customary (but not mandatory) for the media to include only the first letter of the last name for people who are not yet convicted. So when John Smith is in the news as the prime suspect of a murder, he will be referred to as John S. There are some exceptions, e.g. when someone is already a public figure, or when someone actually seeks the media attention. Sometimes the last-name omission looks a bit silly because many other countries (such as the USA) will publish the full name, so for high profile cases one can generally find the full names just by searching online.

That said the system works pretty well because people generally don't bother looking up the full names.

</comment></comment></comment><comment><USER17> > the site would remove names from older stories if their records were > expunged.

That seems reasonable. If the record is "expunged", you can _legally_ say that you have not been convicted of that crime. So it can't legally be used against you (like on a job application)... so references to that crime should be removed from public view.

<comment><USER18> Yes, but a news article saying you were arrested for something still happened, and is still factual.

<comment><USER17> The problem is you can _legally_ say you were never convicted of that crime if it's expunged. If a background check is performed, it will not show up on your record, which will be contradictory if it's still in a newspaper article (or some other public source). It's a catch-22. An employer cannot legally use it against you. If they see it in print they can discriminate (illegally, but who would know) against you. If you can legally claim "I have not been convicted of a crime," then it shouldn't be in print showing you have been. It should go away, just like the criminal record, and just like if someone were to conduct a bg check.

<comment><USER19> You don't see a problem that the government is infringing on my right to post literal facts online?

<comment><USER18> It's not a fact, though. Expungements are the legally way of saying "this should have never happened and legally it did not" for criminal convictions. So anything saying that someone was convicted of Crime X when it was expunged is by definition _not_ a factual article.

</comment></comment></comment></comment></comment><comment><USER20> This is not the strangest self-censorship cleveland.com has engaged in. The strangest I'm aware of is publishing otherwise full physical descriptions of wanted criminal suspects while omitting their race.

<comment><USER21> If you have a full physical description then isn't race redundant?

<comment><USER12> I'm guessing they meant traits like hair color, eye color, height, distinguishing tattoos or marks, gender, but not skin color.

<comment><USER20> Precisely. Hence my use of the phrase " _otherwise_ full physical descriptions."

cleveland.com meant well and thought that this omission would help fix a problem they had, a high volume of blatantly racist postings in the largely unmoderated comments section of their paper in reaction to stories about crime. This solution did not work. It's as if racists aren't driven by sober and precise readings of the newspaper text, or something...

</comment></comment></comment></comment><comment><USER22> An additional solution would be to stop publishing names for suspects entirely. Wait until the person is convicted to release the name.

<comment><USER17> They have already been prosecuted and convicted. You can't "expunge" a crime off your criminal record unless you've been convicted of it. It's ok to publish people who have been convicted of a crime. That's public info. It's NOT ok, to still have it published when it's _expunged_ from your record. The article is specifically discussing expunged crimes.

<comment><AUTHOR> That said, u/nostromo raises a good tangent. When I worked at my college newspaper, which was before Google had become so dominant a fact of day-to-day life, there was no second thought about publishing the name, age, and address of suspects in the police blotter (the daily listing of suspects arrested, mostly for public intox). The age and address were part of the arrest record, and important to disambiguate between similarly-named people. Back then, it hadn't yet occurred to society how Google effectively made info permanently, globally, and efficiently searchable.

Nowadays, I think the status quo for college newspapers is to not print names: [https://www.poynter.org/news/college-papers-dropping- arreste...](https://www.poynter.org/news/college-papers-dropping-arrestee- names-crime-blotters)

</comment></comment></comment><comment><USER23> In New Zealand we have strong name suppression laws - if the court orders it, a defendants name cannot be published in any media (even a tweet naming the person would be illegal, and they do prosecute). AFAIK, in most cases you can still go attend the hearing in the court and see who the defendant is - <USER2> it's not really secret, but difficult to find out without going to some effort.

The theory behind it is that the public shaming (particularly for public figures) from being named would be a punishment, and you shouldn't be punished for a crime without having been found guilty. There's been quite a few high profile abuse cases with this recently where rumours spread around around which 'high profile sportsman' it is.

These days with blogs and social media it usually just turns into a rumor mill, and someone outside NZ jurisdiction can name the person legally. It's also a pretty clear violation of free speech, and judges seem to be very willing to give name suppression, and sometimes maintain it in place even if there is a plea deal and the person is convicted - There's a solid argument that it results in abusers being back out in the public, without people even being warned. So it's not exactly my favourite bit of NZ law :)

</comment><comment><USER24> I'd like to see a codified system where elsctronic docs and records must have timestamps and certain kinds of content must get "forgotten" de-indexed, over time. Midemeanor mentions get de-indexed afer x-years, felonies y-years, murders and the sort, never.

Obviously this would require refinement. That said, someone could resurrect one of these records and reset the clock, but being derivative, the click would have to run much faster. Also it should not be automated (as that would subvert the while idea).

However, in practice this msy be unworkable without draconian enforcement.

<comment><USER10> Not even sure how draconian enforcement would work, unless you forced every country to do the same. I could just set up a service that makes a copy of every document/record that is set to be deleted, and host it in a country that doesn't have that rule.

<comment><USER25> You could do that, and US lawmakers will likely get lobbied to just block your site. Which is fair.

</comment></comment></comment><comment><USER26> Rehabilitation doesn't always work. I doubt a sexual degenerate can be "cured". It's okay if they stay in prison just to be punished and are taken apart from society. It's okay if they can never be around children.

<comment><USER16> "The evidence suggests otherwise. Sex crimes researchers R. Karl Hanson and Kelly E. Morton-Bourgon of Public Safety Canada conducted a large-scale meta- analysis (quantitative review) of recidivism rates among adult sex offenders. They found a rate of 14 percent over a period averaging five to six years. Recidivism rates increased over time, reaching 24 percent by 15 years. The figures are clearly out of alignment with the public's more dire expectations."

[https://www.scientificamerican.com/article/misunderstood- cri...](https://www.scientificamerican.com/article/misunderstood-crimes/)

<comment><USER27> Sorry but 25% of offenders committing another sex crime in the next 15 years more then enough for me to agree with the fact that they need to be identified and tracked.

<comment><USER28> Do you support tracking non-sex crime criminals that have been released given they have a higher rate of reconviction than sex offenders?

"Recidivism rates for sex offenders are lower than for the general criminal population. For example, a Bureau of Justice Statistics study of 108,580 non- sex criminals released from prisons in 11 states in 1983 found that nearly 63 percent were rearrested for a non-sexual felony or serious misdemeanor within three years of their release from incarceration; 47 percent were reconvicted; and 41 percent were ultimately returned to prison or jail."[1]

[1] - [https://abcnews.go.com/US/story?id=90200&page=1](https://abcnews.go.com/US/story?id=90200&page=1)

</comment></comment></comment></comment><comment><USER29> So strange to see so many people celebrating 1984 style censorship.

What the hell is "right to be forgotten". That isn't a right.

Media gets to be less and less reliable and less truthful.

If people want to be "forgotten", let them change their name/identity.

</comment><comment><USER30> I wonder what would happen if a world leader, like the president of France, invoked his EU-given right to be forgotten.

Imagine the chaos, not only on web sites, but across all of society.

Would schoolbooks have to be re-written? What would teachers say when children asked who the xth president of the republic was?

<comment><USER31> The President of France does not have a right to be forgotten, as people of public interest are exempt.

<comment><USER30> _people of public interest are exempt_

Sounds like a new badge of honor for minor "celebrities" and people who are self-important.

Are there codified specifications for "people of public interest?"</comment></comment></comment>


Ask HN: Techniques to go around IDS systems? - <AUTHOR>

For background about my site http://www.classhunt.com, read this: http://news.ycombinator.com/item?id=600478
Since I posted that, I had a meeting with the Vice Chancellor of IT. At the end, he basically said there is nothing he could do to keep my site functioning. I told him I will continue to find ways around them blocking me even though I don't enjoy playing cat and mouse.
Also, the site has become a huge hit amongst incoming freshmen. Back when I posted about it here, we had few hundred users. Today we have few thousand students that are relying on it. This has furthered my motivation to keep the site alive.
Here's the deal: the python bot that scrapes latest scheduling data off of uni's server keeps getting blocked by the uni's IDS system. I've put random delays and intervals but haven't had much luck. After a day or so, the IDS removes the block only to put it back later.
The IDS blocks my personal uni account, not the IP. When I first made the app the data was public. But then the uni, in an attempt to block the site, removed the data from being public. So I had to modify the POST string and go through my uni username/password to retrieve the data.
What I am doing right now:
- randomizing # of threads between 2-4
- randomizing the pause between new requests between 1-3 seconds
- randomizing how often my bot runs between 3-7 minutes
This is what I get when the IDS blocks it: To protect the univeristy from a denial of service attack, script blockers have been enabled. Your transaction has been blocked besause you have tried to login too many times in a short period of time. Please wait and try again soon.
How do I fool this sucker? <comment><USER2> If other students are relying on it, why not have them pass their credentials into the app, then have your app use their login to acquire the data? If you are secure about it, you should be able to avoid the uni's wrath while still keeping your service alive.

<comment><AUTHOR> That's an option. Though asking users for their username/pass will cause significant overhead on my part in terms of security as well as turn away many of the users IMO.

</comment></comment><comment><AUTHOR> Now that I think about it, currently each time my bot starts(every few mins), it logs in and gets a new sessionID which I then use to retrieve the data. I'm going to take the error message to heart and save the session ID between bot runs. Not getting a new sessionID should technically mean that I am not logging in many times.

Will try and report back soon as they unblock me!

</comment><comment><USER3> _How do I fool this sucker?_

Don't. Attempting to circumvent access control measures could not get you thrown in jail.

<comment><USER4> _Attempting to circumvent access control measures could not get you thrown in jail._

If you don't have to worry about getting thrown in jail, I wonder why more people aren't doing it? ;)

Anyway, to answer the OP's question...is there some combination of POST options that loads the entire class schedule at once? I know my school's (U of Alabama) system had a form where you could select all of the course departments and load the entire schedule on one page.

Also, do you really need to reload the data so often? If the IT department is worried about server load, perhaps you could arrange to get a data feed once or twice per day (overnight would probably be best).

<comment><AUTHOR> Yeah, the function of the site is to alert you the moment a class opens up. So I need to get the data as often as I can.

 _If the IT department is worried about server load, perhaps you could arrange to get a data feed once or twice per day (overnight would probably be best)._

Tried that. All they would have to do is put all the data on one page instead of me having to make 130 requests(one for each department). They refused saying they don't have resources. I've tried to be as nice as I possibly can be and they've been as dick-like as they can be.

My last resort would be to ask our user-base to call the IT office and voice their support.

<comment><USER5> > My last resort would be to ask our user-base to call the IT office and voice > their support.

Why isn't that your first resort?

<comment><USER6> This seems like a great idea to me, as it could get you a job, and make your service much faster, in that you could build your notification system into the class scheduling system, which would mean you wouldn't need to poll their system every 3-7 minutes - it could be event-driven instead. (Dropping a class could trigger a check of who's watching the class and notify recipients that way.)

Writing in that line or two of code would also take care of the school's strange DoS concerns.

Of course, if profit is the goal, this isn't an option (though it doesn't look like you're trying to monetize this... kudos for that).

Also, this removes the exclusivity from classhunt (as any student would see this while browsing the schedule online). I'm sure you can make more inferences about this option. (Less social capital in the business world as you have nothing to point at and say "I did that," etc.)</comment></comment></comment></comment></comment>


Infochimps.org: Free Redistributable Rich Data Sets - <AUTHOR> http://infochimps.org/ <comment><USER2> Hey! That's me &co., yay. Our goal is to build the best free repository of data on the web, like a 'flickr for datasets' or the almanac to wikipedia's encyclopedia. There's some similar, excellent work being done by numbrary.com, swivel.com and Freebase, and we intend to complement those projects (as well as to fuel and inspire the projects you all are building). The difference between infochimps and those sites is we're 'messy' -- we'll take the data as it stands, and we'll give it to you with no restrictions and no sandbox, to play with on your machine with your tools. As people find various datasets interesting or useful and thus choose to enhance them (better formats, more metadata), we'll make it easy to share those contributions back.

If you're excited about this project and want to help it grow, please (<http://help.infochimps.org/help/show/Contact>) contact us. The site's still in rough shape, but there's a lot there to love. You can also follow along with the development at our blog -- <http://blog.infochimps.org/></comment>