Hover on iPhone/iPad - <AUTHOR> http://trentwalton.com/2010/07/05/non-hover/ <comment><USER2> Let's ask ourselves why hovering to expose interface elements is so very prevalent in web interfaces, when it's all but absent in native computer software. The answer is that classic web interfaces are static rendered pages spruced up with some CSS and javascript interactivity patches, while native software is dynamically interactive. As web apps move to being AJAX-y or HTML5 interactive javascript experiences instead of a collection of static pages, then the interface conventions of native software will become more prevalent. When you want to edit something in a list, you hit the 'edit' button, and your whole interface goes into 'edit' mode. No more needing to patch fake interactivity onto a static canvas. No more worrying about users being afraid to click on a link or button, for fear of not making it back to their current status, or of being lost while they wait for new content to load. The interactive nature of emerging web standards renders the lack of hovering irrelevant. Current interfaces were all going out the window anyway; touch devices just bring the problem to the forefront.

</comment><comment><USER3> So far, the only place I really, really miss hover is xkcd.

<comment><USER4> Do you know <http://m.xkcd.com> ? No hover needed.

<comment><USER5> I had no idea this existed! Thanks for the link!

</comment></comment></comment><comment><USER6> I'd like to see a 3-fingered magnify gesture, which would double as a hover indicator.

3 points define a triangle, which can be circumscribed by exactly one circle. Associate the 3 fingers in a triangle gesture with a reticle/magnifying glass. Initial placement of the 3 fingers defines the "magnifying glass." Expansion or contraction of the circle defined by the fingers does not change the size of the reticle but instead controls the relative zoom. The cross-hairs in the reticle can also be hovered over GUI items to give a hover functionality.

This would have utility beyond enabling hover. (Sometimes it is more convenient to momentarily magnify a portion of the screen than to zoom the whole screen. One example is Google Maps, where a zoom-in/zoom-out sequence can cause unwanted downloading of map tiles.)

<comment><USER7> I see more than one problem with this approach. First, I cringe at the idea of trying to explain this finger manipulation pattern to my mom. Second, placing three fingers from the same hand on the touch screen would block the place on the screen I am trying to magnify/manipulate from my view. Using two hands to perform this task would be very cumbersome on the larger tablets like an iPad (I just tried without putting my ipad down, didn't go so well). Third, there are already use cases where three fingered swipes are used, increasing the possibility of performing the wrong action.

Hover is a way of hiding complexity in a design. Eliminating that complexity is the best way to hide it.

<comment><USER6> > First, I cringe at the idea of trying to explain this finger manipulation > pattern to my mom.

Just have her put her 3 fingers down on the iPad. No words necessary!

> Second, placing three fingers from the same hand on the touch screen would > block the place on the screen I am trying to magnify/manipulate from my > view.

Wrong. Only certain triangular placements will do this and users will easily learn not to do that. Very flat triangles will have the hand off to the side, just as if you were holding a magnifying glass with no handle. With the right hand, you can do this by having the index finger and thumb extended, with the middle finger bent to sit just below and to the right of the index finger, like so:

                     x
                       x

              x

Since your hand is forming a backwards L, less than 1/4th of the magnifier circle is obscured by your hand. There are many other possible positionings which result in an unobscured view of the reticle.

EDIT: This gesture would only work on the iPad. The iPhone would be too small to accommodate it.

</comment></comment></comment><comment><USER8> I thought about this a bit and I think there is a valid "hover" motion for the brave new world of touch, but it's not available on the iPad yet.

Once the iPad has a front facing camera, in theory eye tracking can serve as a hover. It would be modal, so that you'd have to be holding down a button (perhaps on the edge of the iPad's bezel) to enable "hoverability", but it would probably work well. If you want to see more detail on a certain DVD in Netflix, look at it, and hit the button on the side of the iPad. This will require less physical motion on behalf of the user than even a hover, and much less than a click.

<comment><USER9> Wow, this sounds over-engineered.

<comment><USER8> Eh, not really. Imagine that there are four buttons on the iPad instead of just one. They're each at the midpoint of the bezel on each size. Depending on your rotation, the bottom one acts as home, and the left and right act as buttons that act on what you're looking at. (not sure about the top one :))

</comment></comment></comment><comment><USER10> _Utilize tap to reveal a hover state._

I like this approach, but as noted, often the problem is "users get no cue that tapping... is even an option".

I think a suitable solution is to start with one item 'pre-tapped', with strong cues that it is 'selected' (via highlighting, intensified colors, icons, etc.). That provides a strong clue that its peer items could be tapped to show the same 'hover' controls.

</comment><comment><USER11> The technology to enable a hover-state in a touch input device is certainly there. Hopefully in the near future OEM's will create a UI that allow you to see a hover state as your finger approaches a target element. Ideally, you'd get three states: hover distance, touch, and touch-pressure.

This would then allow the hover state to live long and prosper.

</comment><comment><USER12> Thoughts on hovering: My laptop has a capacitive Wacom panel which can detect the presence of the pen about an inch above the screen. You can track the mouse around the screen without ever actually touching the screen. This allows for full hovering ability. Why not implement this for smaller devices like phones?

<comment><USER13> There's a lot of technology in a Wacom pen that is missing from a finger. I'm not aware of a practical way to identify and measure the location of a fingertip that's any significant distance from the screen, except by using cameras.

Ultimately, I think any tablet made for serious work will have to implement Wacom pen tech and multitouch.

<comment><USER14> With a capacitive device, it's possible to detect things like fingers some distance away from the surface, but the signal is so noisy that you get a lot of false positives.

The endpoint of all this stuff seems to be the Minority Report UI ([http://www.ted.com/talks/john_underkoffler_drive_3d_data_wit...](http://www.ted.com/talks/john_underkoffler_drive_3d_data_with_a_gesture.html)) or Kinect, where the computer is detecting what you're doing with your body in a holistic way; it can infer intent, rather than measuring specific points of data without knowledge of where they came from.

Of course, that requires bigger and more expensive sensors than you're likely to find in a $499 device today.

</comment></comment></comment><comment><USER15> Why is it okay to design layouts that require a pointing device, yet it's not okay to design for hover? How does one draw this line?

I think this is just a matter of designing for your audience, not a matter of "moving forward".

<comment><USER16> You're welcome to design solely for an audience that has hover support, just as you're welcome to design solely for an audience that has IE installed.

Perhaps you should also put up a popup: "This site best experienced with a computer that has a mouse. If you are visiting with a phone or touch-screen tablet, please consider buying a computer, or visiting one of our many fine competitors."

Remember when lots of websites had messages urging you to visit with specific browsers? Remember how much fun that was?

<comment><USER17> I don't think that's unreasonable if you're reasonably sure your audience is close to entirely IE.

When I make websites I have no time for ideals or philosophical statements, I look for where my market is now and where it's going in the short term, and make my decisions based on that.

<comment><USER18> I've seen many applications that were created under that assumption ("they're 'intranet' apps, and IE is our corporate browser") only to become extranet applications quite suddenly - undermining that assumption by merely replacing "in" with "ex" in the functional spec.

It's not just a philosophical statement. It's a practical matter. Unless your plan is to high-tail it out of there and leave it to someone else to maintain your mess, that is.

<comment><USER16> The difference between philosophical statements and practical matters is surprisingly difficult to articulate. You can rarely point to a single moment when the one becomes the other.

Every practical situation starts out life as a philosophical position. The atomic bomb was once a napkin doodle.

</comment></comment></comment></comment></comment><comment><USER19> I guess I want to blame web apps in general - our hands are tied until only the simplest controls and the least helplful experience is possible.</comment>


Show HN: Parity Bar – Not everyone earns a US salary - <AUTHOR> https://paritybar.com/ <comment><AUTHOR> Hey HN!

Yesterday I set myself a challenge (after being dared by a friend). I wanted to create a startup in 24 hours.

I got inspired by Wes Bos his Pricing Power Parity he used for his courses. What is PPP you're asking?

"Purchasing power parity (PPP) is a measurement of prices in different countries that uses the prices of specific goods to compare the absolute purchasing power of the countries' currencies."

Offering discounts for lower index countries gives them a chance to also move forward with their business. It's unfair that everyone has to pay the same price for a product.

I'm currently sitting out corona in Bali. What I never realized is, that the average salary in Bali, Indonesia is $133 per month . They have no fair chance online with salary gaps this big!

So, offering PPP discounts is an amazing way to help people out. But, offering these kinds of discounts can be quite hard, so I set myself out to make it easier. It also seemed like a small enough scope to build in one day.

Parity Bar makes it super easy to give discounts to customers around the world, based on their location.

1\. You add your discounts 2\. Style the banners 3\. Add the script to the website 4\. And the tool does the magic!

It's free to use up to 10.000 visitors. After this I gotta charge a little bit, to pay for all the APIs and server costs.</comment>


Military Supercomputer Surpasses Petaflop Milestone - <AUTHOR> http://www.nytimes.com/2008/06/09/technology/09petaflops.html?partner=rssnyt&emc=rss

<comment><USER2> now who wouldnt want to run their app on that.</comment>


Airbnb to lay off between 1k and 2k employees - <AUTHOR> https://www.theinformation.com/articles/airbnb-plans-significant-layoffs <comment><USER2> Primary thread: [https://news.ycombinator.com/item?id=23083692](https://news.ycombinator.com/item?id=23083692)</comment>


Sorkin vs. Zuckerberg: Lawrence Lessig on "The Social Network" - <AUTHOR> http://www.tnr.com/article/books-and-arts/78081/sorkin-zuckerberg-the-social-network?page=0,0

<comment><USER2> This is a very thoughtful article, it's a shame it didn't get more votes.</comment>


A Slow-Books Manifesto - <AUTHOR> http://www.theatlantic.com/entertainment/archive/2012/03/a-slow-books-manifesto/254884/ <comment><USER2> I like this idea on the surface, as with many of the other (mentioned) "slow" movements; what worries me is these movements seek to divide, to categorize - some to the good, some to the bad. With respect to "Slow-Books" who is the arbiter of what constitutes Literature and Non-Literature?</comment>


Tesla's New Gigafactory: Stats in the Construction Permits - <AUTHOR> https://www.inverse.com/article/17433-tesla-s-new-gigafactory-10-mind-boggling-stats-in-the-construction-permits <comment><USER2> > The factory itself is 5.8 million square-feet, which, if my math is correct, > means building the damn thing is costing Tesla $55.6 million per square > foot.

Your math is incorrect, $332.6 million divided by 5.8 million sqft is $55.60/sqft.

<comment><USER3> Well, they fixed the math but now unfortunately the sentence is now ridiculous:

> if my math is correct, means building the damn thing is costing Tesla $55.6 > per square foot. And that's just the estimated cost of the contracts so far.

Based on a quick look[1] that's a totally average price per square foot for an industrial building. _AMAZING_.

[1][http://www.brownwegher.com/cost-of-construction- chart/](http://www.brownwegher.com/cost-of-construction-chart/)

</comment></comment><comment><USER4> Some hilariously bad math right here:

 _> 9\. The whole thing has already cost $322.6 million._

 _> The total cost of the contracts awarded is skyrocketing toward half a billion dollars (though it has a ways to go to meet that mark). The factory itself is 5.8 million square-feet, which, if my math is correct, means building the damn thing is costing Tesla $55.6 million per square foot. And that's just the estimated cost of the contracts so far._

$322.6 million / 5.8 million sq-ft => $55.6 per sq-ft -_-

//edit all 3 of the first comments were about the math. Love you guys!

</comment><comment><USER5> "The factory itself is 5.8 million square-feet, which, if my math is correct, means building the damn thing is costing Tesla $55.6 _million_ per square foot."

Sigh, I hope this was a joke.

</comment><comment><USER6> Really an poor article that just fails on too many levels.

That it should have sufficient solar power to drive production is commendable but I am curious if they are going to have some onsite storage mechanism as I assume this is a 24x7 facility.

The amount of materials that have to be brought in by rail road would a statistic I would find mind boggling. How much water does the plant consume?

<comment><USER7> It is a battery factory. They will definitely have some on-site storage mechanism. :)

<comment><USER8> plus Musk wants to buy solarcity so planning to build the power system internally. I wonder if they have taken into account the total covering of the giga factory roof with solar panlels. Actually might produce more electricity than they need even if they have storage system for 24hr usage

<comment><USER7> I doubt it. They will have to charge the batteries they produce, which will be a lot of energy going out the door.

</comment></comment></comment></comment><comment><USER9> >batteries, which are more efficient at lower temperatures.

Shouldn't that read less efficient?

<comment><USER10> Taken in the context of the article, I believe they're talking about how the chiller room will be used to develop batteries that are more efficient at lower temperatures.

</comment></comment><comment><USER11> This is a LOT of debt to put on the balance sheet of what is (still) a young company facing new entrants into the luxury-electric vehicle market segment.

<comment><USER12> The thing though is Tesla is also aiming at the energy storage part of the market with has the potential to be much larger.

Assuming their battery technology is competitive there's going to be no shortage of companies interested in their batteries.

<comment><USER13> True there is a huge potential for growth in the battery market, the electric car market is likely to grow 10X in five to ten years. From 1% to 10% of total cars produced. If Tesla is profitable with the factory in Nevada, then they are setup to build more factories in other locations as well. Quick estimation, 10 million electric cars per year, $5000 per car in batteries, $50 billion annual sales.

Also the Tesla Model 3 at $35k (before tax incentives, etc) is barely in luxury vehicle category.</comment></comment></comment>


Requests-HTML: HTML Parsing for Humans - <AUTHOR> https://github.com/kennethreitz/requests-html <comment><USER2> +1 to make nicer APIs! It is always good to have more high-quality API designs to look at.

That said, it looks more like an API experiment, not a practical solution for a day job, at least in its current state:

* response body encoding detection is wrong, as it doesn't take meta tags or BOM into account;

* base url detection is wrong, as it doesn't take <base> tag in account;

* URL parsing (joining, etc.) is implemented using string operations instead of stdlib, so a careful inspection is required to make sure it works in edge cases. For example, I can see right away that .absolute_links is wrong for protocol-related urls (e.g. "//ajax.microsoft.com/ajax/jquery/jquery-1.3.2.min.js")

* html2text used for .markdown is GPL - I know people have different opinion on this, but in my book if you import from a GPL package, your package becomes GPL as well;

* each .xpath call parses a chnk of HTML again, even if a tree is already present

* shortcuts are opinionated and with no clear behavior, e.g. .links deduplicates URLs by default, it deduplicates them using string matches (so e.g. different order of GET arguments => URLs are considering unique); it checks for '.startswith('#")' which looks arbitrary (what if these links are used in a headless browser? what if someone wants to fetch them using _escaped_fragment which many sites still support? why filter out such URLs if they are relative, but not if they are absolute?)

<comment><USER3> Would you mind posting these as issues?

<comment><USER2> TBH I don't see myself using this package: in its current stage it is very little code, and almost every method has an issue either with edge cases or with API; also, it is tied to requests library, unnecessarily IMHO, and in my opinion it is GPL even if setup.py says it is MIT.

Because there is nothing usable code-wise in requests-html from my point of view (it is no better than existing alternatives), I don't feel like raising these issues, advocating for fixing them, discussing alternative solutions with a goal of improving requests-html. Of course, everyone is free to raise these issues in a repo.

I appreciate the work put into requests-html API design, the design is very nice overall. This might be a way to go: create a nice API design, attract people, fix implementation over time, but this battle is not mine, sorry :(

<comment><USER4> GPL dependency removed.

All of these improvements I'd like to be made to the software. It's all about getting a nice API in place first, then making it perfect second.

<comment><USER4> I addressed most of your issues, like not using urlparse in the latest release.

With libraries like these, it's all about getting the API right first, them optimizing for perfection second. :)

<comment><USER2> :thumbs up:

A second iteration of review:

* encoding detection from <meta> tags doesn't normalize encodings - Python doesn't use the same names as HTML;

* I'm still not sure encoding detection is correct, as it is unclear what are priorities in the current implementation. It should be 1) Content-Type header; 2) BOM marks; 3) encoding in meta tags (or xml declared encoding if you support it); 4) content-based guessing - chardet, etc., or just a default value. I.e. encoding in meta should have less priority than Content-Type header, but more priority than chardet, and if I understand it properly, response.text is decoded both using Content-Type header and chardet.

* lxml's fromstring handles XML (XHTML) encoding declarations, and it may fail in case of unicode data ([http://lxml.de/parsing.html#python-unicode-strings](http://lxml.de/parsing.html#python-unicode-strings)), so passing response.text to fromstring is not good. At the same time, relying on lxml to detect encoding is not enough, as http headers should have a higher priority. In parsel we're re-encoding text to utf8, and forcing utf8 parser for lxml to solve it: [https://github.com/scrapy/parsel/blob/f6103c8808170546ecf046...](https://github.com/scrapy/parsel/blob/f6103c8808170546ecf046b9f4ea6dead94de189/parsel/selector.py#L38).

* when extracting links, it is not enough to use raw @href attribute values, as they are allowed to have leading and trailing whitespaces (see [https://github.com/scrapy/w3lib/blob/c1a030582ec30423c40215f...](https://github.com/scrapy/w3lib/blob/c1a030582ec30423c40215fcd159bc951c851ed7/w3lib/html.py#L325))

* absolute_links doesn't look correct for base urls which contain path. It also may have issues with urls like tel:1122333, or mailto:.

For encoding detection we're using [https://github.com/scrapy/w3lib/blob/c1a030582ec30423c40215f...](https://github.com/scrapy/w3lib/blob/c1a030582ec30423c40215fcd159bc951c851ed7/w3lib/encoding.py#L187) in Scrapy. It works well overall; its weakness is that it doesn't require a HTML tree, and doesn't parse it, extracting meta information only from first 4Kb using a regex (4Kb limit is not good). Other than that, it does all the right things AFAIK.

<comment><USER4> Thanks for the feedback, integrated w3lib!

</comment></comment></comment></comment></comment></comment></comment><comment><USER5> Oooo... I just finished writing a script with BeautifulSoup. While it wasn't all that bad (it works ;) ), I'm sure the "Kenneth Reitz experience" would be much better. I won't be rewriting the script now, I can't wait to find an excuse to try this. :)

EDIT: first commit 22 hours ago - goes to show that when you have thought about the idea and know what you're doing it doesn't take long to produce the first version. :)

<comment><USER6> Would not recommend BeautifulSoup for this type of thing.

lxml.html is much better in my experience. If you want to use CSS selectors, there's pyquery.

<comment><USER7> I remember years ago I needed to parse some html(which was about 2-3 million characters) and after a fair bit of time, I had it up and running with beautifulsoup. Now, my use case was likely quite atypical to most html parsing, but my god was it ever slow! I forget the exact numbers, but I think it was taking about 150 seconds to complete. So, then I wrote it using lxml, which was an improvement, but that was still taking around 100 seconds.

Now, I very rarely have any need to scrape and parse html data, and I was scratching my head at how it was taking these parsers so long to parse a 3.5 mib html page. I mean, it should be able to go through that and get what I want in under a second right?

So, I said screw it and wrote some regexes. 10-15 seconds was how long it was now taking to parse that html. It actually took 1.5 or so seconds to parse the html; the rest was waiting for it to download the webpage.

Ironically, implementing the regexes was actually quicker than figuring out how to use those html parsers and write the code. Of course, that's assuming you know how to craft regexes. Since it was set-up to run every 5 minutes, I wanted something that could do it without spending 1/2 the time parsing the data(amongst other tasks the processors were needed for).

YMMV

<comment><USER8> You're using regex to parse html? Have you not read: [https://stackoverflow.com/a/1732454/1090568](https://stackoverflow.com/a/1732454/1090568) ?

<comment><USER9> If you have an HTML document you want to extract information from, regexes are fast and easy.

It's when you don't have guarantees about the structure of the html you are working with, that regex will come up short.

</comment></comment></comment></comment></comment><comment><USER10> Really looking forward to using this. BeautifulSoup is great, but it's counterintuitive (I always need to refer to documentation even for very basic aspects that I've used dozens of times before) and it's often slow and has some really weird xml bugs)

Also check out newspaper3k if you haven't seen it. It is high level, but really useful for a bunch of simple scraping related use cases

<comment><USER11> > _BeautifulSoup is great, but it 's counterintuitive (I always need to refer > to documentation even for very basic aspects that I've used dozens of times > before) and it's often slow and has some really weird xml bugs)_

I won't argue the slowness and the occasional bugs, but unlike you I find b4 to be very intuitive. And this is mainly why I use it, despite its faults. Maybe our use case are different, but with a basic knowledge of html, I only rarely find myself reading the documentation. Care to give some examples of what you find counter-intuitive?

</comment></comment><comment><USER12> A similar library is [https://github.com/tryolabs/requestium](https://github.com/tryolabs/requestium)

Though it adds parsel as a parser (which has a really nice api) to requests. It also integrates with selenium.

</comment><comment><USER13> > Render an Element as Markdown:

I prefer my dependencies to be orthogonal and lightweight i.e. do one thing well. Maybe this is better for interactive use in the REPL.

<comment><USER4> I consider it a nice-to-have, but we can definitely remove it if deemed unnecessary. Want to open an issue about it?

<comment><USER13> That's just my preference. Keep the features that are best for your specific use cases. It's your project after all. I think that's better than design by committee. Linus didn't write Linux for other people.

<comment><USER14> If you're doing markdown conversion well, keep it. Apis are not about doing the smallest thing, it's about designing a clean abstraction of a problem you're trying to solve.

<comment><USER13> Sure, if APIs existed within a vacuum. In the real world we have to worry about ease of packaging, ease of maintenance, and various other practical costs.

</comment></comment></comment></comment></comment><comment><USER15> Nice. It would be pretty cool to have an "run javascript and wait for network idle" optional for scraping js-requiring websites. Can selenium do this? Headless chrome?

I personally use bs4 for web scraping and it works pretty well, but if there was an option to also do js with a sane API, I'd switch in a heartbeat.

<comment><USER14> We use puppeteer for our smoke tests. Ensure all network requests load, no JavaScript errors, take screenshot of page and simple validations so we know our deploys aren't borked.

I really love puppeteer over selenium. Much deeper control.

</comment></comment><comment><USER16> The output of _r.html.absolute_links_ on the home page looks like it contains errors. For example,

      'https://www.python.org//docs.python.org/3/tutorial/'

      'https://www.python.org//docs.python.org/3/tutorial/controlflow.html#defining-functions'

I think it's important to remember for every single new library that comes out, that you are trading apparent usability for unknown issues that haven't surfaced yet because the library is so new.

<comment><USER4> That bug has been fixed, but the docs hadn't been. Fixed now :)

<comment><USER4> Looks like there's still some room for improvement though!

<comment><USER16> There is always room for improvement. I think it's easy to underestimate the time required to _dwell_ in a thing before we really understand it. This is true for runtimes, libraries, even entire programming paradigms. We want to take shortcuts using abstract reasoning, and that works well usually, but sometimes you just gotta use something for _years_. (Alas, one lifetime may be too short to dwell in all the various paradigms the way they each require. But I digress...)

</comment></comment></comment></comment><comment><USER17> Interesting! How does this compare with MechanicalSoup (which seems to be the current best-in-class solution for scraping in Python)?

<comment><USER4> Very different use cases, imo. MechanicalSoup emulates a web browser experience. This is more for scraping.

</comment></comment><comment><USER18> I love Kenneth's work.

He's absolutely focused on user experience – in this case the developer experience – and it absolutely shows.

I'm sure I'll end up using this utility at some point in the future.

</comment><comment> Const-me Same functionality for .NET: [http://html-agility-pack.net/](http://html- agility-pack.net/)

<comment><USER19> There is overlap, but this isn't the same as Requests-HTML. It is nice that this library has been developed further over the years.

</comment></comment><comment><USER20> > Select an element with a jQuery selector.

What is a 'jQuery' selector? Is it the same as CSS selectors, or does jQuery support non-standard syntax?

<comment><USER21> For what it's worth, yes, JQuery does support non-standard syntax/selectors

</comment></comment><comment><USER22> Kenneth Reitz comes out with yet another good UI (in terms of library) to accomplish daily mundane tasks, with joy.

</comment><comment><USER23> That's a neat little library. The problem is that the web is rapidly moving away from having HTML as the main information carrier to HTML merely being an envelope to deliver a bunch of JavaScript (and if you're even more unlucky: WebAsm).

Scraping the web will become a lot harder in the future.

<comment><USER21> In saying that, Javascript is just as parsable, and these 'javascript sites' are probably loading structured data via a JSON API which will probably be easier to scrape than a bunch of layout HTML.

</comment></comment><comment><USER24> Looks like this is built on top of lxml and parse. I built an adapter over the bs4 interface in lxml, which was much faster than using bs4 with an lxml backend.

This is great news, as this space was dominated by bs4 in the Python ecosystem.

Can't wait to use this in the future :)

</comment><comment><USER25> Also in PHP with Symfony Dom Crawler: [https://symfony.com/doc/current/components/dom_crawler.html](https://symfony.com/doc/current/components/dom_crawler.html) or Goutte for an easy to use web scraper [https://github.com/FriendsOfPHP/Goutte](https://github.com/FriendsOfPHP/Goutte) that uses Dom Crawler

</comment><comment><USER26> This is a nice wrapper around requests, pyquery [https://github.com/gawel/pyquery/](https://github.com/gawel/pyquery/) and parse [https://github.com/r1chardj0n3s/parse](https://github.com/r1chardj0n3s/parse) of which only requests is Kenneth Reitz. Let's give credit where it's due.

<comment><USER27> To give even more credit where it's due, requests is a nice wrapper around urllib3, which is the work of Andrey Petrov, Cory Benfield and contributors. While requests provides good user-friendly defaults and API semantics, urllib3 does a lot of the heavy lifting.

<comment><USER28> Which is written in python, so better give credit for that.

Wow and now python is in C so line up your credit books, we are in for a long night tonight.

Thoughts & prays for all contributors

<comment><USER29> ...I know you're trolling, but I seriously wonder where it would stop if we were to go down all that way.

<comment><USER30> Gotta give credit to whichever cavemen/cavewomen discovered fire, and all the civilizations that invented the wheel, Ben Franklin for harnessing the power of electricity, and then maybe Ken Thompson and Dennis Ritchie for inventing Unix.

</comment></comment></comment></comment></comment><comment><USER31> Is there a shell interface, or do I have to call it through python? It would be nice to be able to use it in a Ruby project.

</comment><comment><USER32> I don't get the point, it's just a nicer alternative to beautifulsoup. Html pages are always changing, instead of writing the parsing logic in code, I think we should put xpath or css expressions in some config files.

<comment><USER33> I'm sure some pages could be represented as css expressions in config files, but in general it seems like scraping is about working around unanticipated idiosyncrasies(e.g. X% of the pages have a different structure for mysterious reasons).

I haven't used xpath much though (and it seems pretty beefy!).

</comment></comment><comment><USER34> So stoked this popped up. Literally woke up this morning debating about moving away from bs4 and wrapping the functionality I needed in lxml. I was just thinking I wish there was a requests equivalent for parsing...

<comment><USER6> Is there any useful feature in bs4 that is not available in lxml.html?

</comment></comment><comment><USER35> This looks awesome, can't wait to try it. Last time I used pyquery though it was considerably more limited than CSS and jQuery syntax, and I reverted to XPath. Has it improved recently?

</comment><comment><USER36> HTML requests are: plain text, self-explanatory ("Content-length", "charset", etc.). What exactly is unhuman about that?

<comment><USER37> That's HTTP, not HTML.

<comment><USER36> Yikes, major misread on my part. Whoops.</comment></comment></comment>


Take Me Cooler: Enter a desired temp - we'll tell you where to go to get cool - <AUTHOR> http://pressbin.com/takemecooler

<comment><USER2> I tried a few zips around the country. All it ever did was tell me to go to San Francisco. Is there something more I'm supposed to be seeing here?

</comment><comment><USER3> I put in a zipcode and it just took me to the same page I was on. Pretty page, but it didn't seem to do anything.</comment>


Ask HN: How are you protecting your privacy? - <AUTHOR> I'm continually trying to improve how I handle my privacy online.
I've slowly been migrating off all Google services with the intention of deleting my entire account (got Youtube and Google Maps to tackle next).
DDG for primary search (but unfortunately have to fall back on Google a lot)
I'm using Fastmail for email, caldav and carddav.
I use Private Internet Access as an always-on VPN.
I use Firefox with uBlock and Privacy Badger enabled. I try to use containers for Amazon and Google.
I use Resilio Sync for syncing files between devices.
I use Timemachine for backups and have x2 NASs at home and off-site. I'm planning on removing my Backblaze cloud backup soon.
I'm planning on swapping my Android phone for an iPhone on the basis that at least I know what Apple are trying to sell me.
How do you handle your privacy? <comment><USER2> searx or startpage are search engines that show google results so you could try them as a fall back before you finally try google. !searx and !sp are the bangs you can put anywhere in the DuckDuckGo query

for maps, im really liking maps.me on mobile (its called "maps" in the fdroid store). its a good simple app that lets you add bookmarks and group them in folders, and its good for navigation since it has a 3d view. osmAnd is another great app but its more of a poweruser app that does a lot of things so there is a lot going on in the UI.

i switched to protonVPN last year from PIA and im especially glad after seeing the smear campaign PIA were part of a few months ago where they were trying to stir up rumours about other VPN services. (one of them forget to remove their google account profile picture from one of the screenshots and was caught out that way). they provide a good service but whether you want to give your money to a shady company like that is up to you

i use resilio sync myself. syncthing is open source so is probably the better choice for privacy, but im not in any rush to switch since nothing is being stored online.

i have a few GBs of files on backblaze but havnt really started using it yet. is there anything shady about backblaze that i dont know about or are you just keeping things offline just to be safe?

</comment><comment><USER3> Running pretty similar setup too. I use synching and borgbackup though for sync and backups and mailbox.org as mail/caldav service

Possible additions: \- a free DNS service \- signal or xmpp for messaging \- open source password manager with self-hosted sync, I use password-store \- maybe trivial: use different, strong passwords and 2-factor-auth where possible \- use different mail-aliases for every service you register. Mailbox.org allows me to receiver any mail send to email@example.com, that way if I get spam or my address is leaked I can know how gave it away in many cases \- use lineageOS instead of stock android \- install apps from f-droid when possible

I have not found a good replacement for Google maps yet, especially on the phone when I need navigation.

</comment><comment><USER4> You can buy a cheap android phone and install miniG to get rid of all the google shite, but you will need to use newpipe for google (it wont cast to chromecast), and I recommend waze for maps.

Why not setup something like zsync then encrypt the files and upload them to backblaze, cheap encrypted backups.

</comment><comment><USER3> P.s Https everywhere add-on</comment>


Coinbase lets you convert your tokens into gift cards - <AUTHOR> https://techcrunch.com/2018/07/25/coinbase-lets-you-convert-your-tokens-into-gift-cards/ <comment><USER2> > "Gift cards aren't as convenient as receiving money on your bank account or > a debit card. But they're a great way to avoid telling your bank that you > made money by speculating on cryptocurrencies. Many banks directly report > data on their users to local tax authorities. But don't forget that Coinbase > can track all your withdrawal events and notify tax authorities too."

Sigh. Always comes down to tax evasion or illegal transactions.

<comment><USER3> > _tax evasion_

Not only tax evasion. Stupid tax evasion. Tax evasion that leaves a screaming paper trail. None of this is being enforced right now. But in a downturn? Particularly if the offender has assets? Enormously stupid way to do an already stupidly-risky thing.

<comment><USER4> In the US anyway you can report someone for tax evasion and receive a portion of what they recapture. All these people will get caught eventually.

</comment></comment></comment><comment><USER5> "Gift cards aren't as convenient as receiving money on your bank account or a debit card. But they're a great way to avoid telling your bank that you made money by speculating on cryptocurrencies. Many banks directly report data on their users to local tax authorities. But don't forget that Coinbase can track all your withdrawal events and notify tax authorities too."

So this "journalist" is telling people it's a great idea to evade tax... hmmmm...

<comment><USER6> Sounds like the opposite to me or is my legalese off? It reads more like a warning than anything.

<comment><USER7> Maybe...but it sounds like the "warnings" that grape juice manufacturers placed on their products during Prohibition, to prevent turning them into wine accidentally by letting them ferment.

<comment><USER8> Do you know examples of such labels? I have grapes :-)

</comment></comment></comment></comment><comment><USER9> Couldn't someone create a 'Credits' card that converts crypto to 'Credits' and get the top 100 stores on board, through some form of incentivisation (i.e no processing fees, like there are with CC and Debit cards) therefore bypassing Fiat, but do it in the form of a 'Gift Card'?

It seems you can bypass some regulations by doing this. How long would it take until the regulatory board starts banging down your door.

<comment><USER10> They could consider using "dollars" which conveniently are accepted means of transaction by literally everyone.

<comment><USER9> Oh yes Dollars, how could I forget! You forgot to mention the other upsides; they are subject to taxes when converting from crypto, you can literally have them lose value in your pocket via deflationary monetary policy, the work only with the confines of a nations borders, the list goes on and on!

<comment><USER3> > _they are subject to taxes when converting from crypto_

Using a Bitcoin gift card to purchase goods is the same as selling it for cash.

> _you can literally have them lose value in your pocket via deflationary > monetary policy_

Deflation means your money _gains_ value doing nothing. The dollar is inflationary. It loses a little bit of value if it sits around doing nothing. (If you have lots of money losing money to inflation, you're hiding illegally- obtained currency or doing something stupid.)

Moreover, this is an odd complaint to make of the dollar relative to cryptocurrencies, which have a habit of suddenly losing lots of value.

> _the work only with the confines of a nations borders_

Dollars are relatively internationalized. And again, an odd complaint to make of the dollar relative to cryptocurrencies.

 _Disclaimer: I am neither a lawyer nor a CPA. This is not legal nor tax advice._

<comment><USER11> > _Deflation means your money gains value doing nothing. You 're criticizing > an aspect of inflationary monetary policy._

What a surprise, the person railing against fiat currency and trying to reinvent credit cards "but with bitcoin, and without fees!" has no idea what they're talking about.

And I think you only need that disclaimer if you _are_ a lawyer or CPA and what you're saying could reasonably be construed as legal or tax advice.

</comment></comment></comment></comment></comment><comment><USER12> Cool, so it looks like they're going all-in on the international money laundering clearing house thing. I'm sure it will be quite profitable.

</comment><comment><USER13> > Gift cards aren't as convenient as receiving money on your bank account or a > debit card. But they're a great way to avoid telling your bank that you made > money by speculating on cryptocurrencies. Many banks directly report data on > their users to local tax authorities. But don't forget that Coinbase can > track all your withdrawal events and notify tax authorities too.

Does TechCrunch have any actual editorial review process or do "journalists" publish whatever tax evasion / money laundering 101 instructional article they want?

<comment><USER14> They just publish whatever companies pay then to publish.

</comment></comment><comment><USER15> FYI BitPay offers buying Amazon gift cards with Bitcoins since 2017 [https://blog.bitpay.com/amazon-gift-cards/](https://blog.bitpay.com/amazon- gift-cards/)

You can buy until $ 2000 daily.</comment>


Favorite Language/Editor Features - <AUTHOR>

(Originally posted on the TopCoder forums)
What are your favorite language or editor features, or what would you love to be able to use?
Some of my favorites are closures, dynamic typing, mix-ins, and functions as first order objects. Originally these were just neat features, but I've found myself using them as the best way to express my ideas in code, even in a contest situation. Actually, C#'s anonymous delegates are the reason I use it in contests, despite generics being quite a bit weaker than templates (superficially they seemed as powerful, but as a scientist the first thing I would want to do is code numerical routines in a generic fashion, and guess what -- you can't even ensure that a certain class can use regular operators. This really bugs me.)
I'll also admit to being swayed by Paul Graham's Essays, and I'd love to try out some of the interesting macro hacks that he speaks of, but so far I haven't had many opportunities to do so.
There are a few dream features I'd love to use, even though I'm not sure if they'd be altogether useful. I've just never seen them before, and I figure that language/editor design is something fun enough that I'd like to throw them out there to see what people have to say.
There's a pattern running through most of these requests, namely that programming languages seemed to have evolved and entrenched themselves at a time when terminal capabilities enforced a text-only approach to programming. Now we have real-time 3D games producing life-like imagery, but for some reason, programming has only incorporated such features into a layer on top of the languages (like the facilities in ReSharper, which has made my programming much more enjoyable). I want to know how much farther you can push things.
1. Support for different fonts and symbols, native to the code, and extremely accessible to the editor.
This might just be my background in math, but I'd love to declare my variables using Greek letters, declare a function using a big beautiful, curly, math blackboard font F, write case statements using a curly brace { and conditionals on the right side, access array fields with subscript notation -- or tensors using super/subscript notation, and have all of that shown to me graphically.
Again, I'm not sure if it's just my background, but when I see people sketching out a program on a blackboard, or on scratch paper, or formatting it for publication, it doesn't look like code, but much more like formulas. They don't use massive variable names -- they declare these once, like in documentation, and then proceed to think using this extremely compressed language, with one letter identifiers and mathematical notation. I don't know why languages haven't done this yet. (Edit: actually this might be a good jumping off point. Scoped aliases, which would work like c-macros, not references, but they'd be scoped)
2. Support for mathematical notation like summation (the Sigma symbol), or products, or set operations, and so forth, in the code. I also think a facility to distinguish between enumerable, comparable, and hashable objects and data structures might be a win, but it seems quite heavy-weight.
3. Easy and straightforward access to random numbers and characters within a range, in the code. It's no secret that random numbers are useful for a wide range of problems. They're so useful that it seems silly that every time I want one, just any old random number, that I must declare a randomization object, then type Rand.Next(0,100) or whatever. Why can't I write ~(0,100) or something?
4. A graphical representation of state machines. Over and over again, state machines seem to drastically simplify a large number of otherwise difficult problems. We can work these out on paper, but then we can't ship them with the code. Or we can use some external program to draw things for us, and then ship the doc with the code, hoping that people will read it, and not let it get out of sync. Why not have simple facilities for writing or detecting state machines, and then graphical representations of them? With facilities like Yield, Iterators, and object wide booleans, we're halfway there.
5. Interactive compilation. I don't just mean like a regular interpreter, I mean something more like when in Mathematica or Maple, you type in a command, hit the enter button, and it produces something that the program compiles down to. "What's the derivative of some long expression?" Write it, take the derivative, and it tells me. I want to be able to ask, "What code is produced when I pass this closure to this function?" Write it, hit enter, and it should be able to say.
Do you guys have any other dream features? Anything so good that it would compel you to try a new language? Why? <comment><USER2> 1\. Didn't APL try something like this? Special characters sound like a win. Super/subscript notation sounds like a lose (syntactic regularity, readability).

2\. The first part sounds like a win for readability, but I think this might be a net lose because it could be harder to navigate with the keyboard,but maybe I'm just not using my imagination. Worse is the addition of syntactic complexity, which makes metaprogramming harder. It wouldn't be a problem if it was just a macro that compiles down to regular syntax. I'm not sure what you're talking about in the second part - probably due to my lack of experience with Java and C#. Are you saying you want the editor to display these objects in a special way?

3.

        (set-macro-character #\~
    		     #'(lambda (stream char)
    			 (random (read stream t nil t))))
        CL-USER> ~40
        25
        CL-USER> ~40
        18

Support for ranges is left as an exercise for the reader.

4\. Detecting and modeling arbitrary, informally specified state machines in code would be hard. Creating a library that provides a DSL for making state machines and displaying visual representations sounds fairly easy.

5\. I'm not sure if you're asking for something like Common Lisp's macroexpand function, or Slime's disassemble command. I'm pretty sure it's one or the other - maybe both.

The notion of language support for provably correct programs (the type systems in ML and Haskell are a step in this direction) is very interesting to me, but I'm not sure it's ideal for a production language.

<comment><AUTHOR> Thanks for your comments! To continue the discussion:

1\. A[i] and Ai (that should be subscript) seem pretty similar to me, but I like Ai more, it just seems cleaner. In C# we can write A[i,j,k] for multidimensional indexers, but tensor script seems cleaner still. Again, I think this really depends on background -- someone who actually deals with tensors might find my proposal more attractive. I'm just driven up the wall when I have to write A[i][j][k]. It's so wasteful.

2\. I'm not sure if it would be harder to navigate with a keyboard -- it seems like something that you'd have to build first and get used to. The idea of implementing these things as macros which compile down to regular syntax really appeals to me, in fact it's what I had in mind. (C++ has awkward features to do this in the STL with templates, and numerous TopCoders include C macros to expand summation, product, or accumulator functions. If they're useful in the heat of the moment, why not in regular usage?)

3\. Nice. Thanks. At some point I'll have to take on another big product in LISP, I haven't really touched it since I went into scientific computing, and I think my opinion of it was colored by the bare-bones Scheme I was using.

4\. "Detecting... would be hard" -- it doesn't have to be perfect. I mean, if it was perfect, implicitly it would mean 'graphically model arbitrary code'. I'm thinking there are a collection of schemes which might find maybe 80% of what people would call state machines which aren't so bad. Maybe 80% isn't satisfying, but it's a start. Like bug catching codes in compilers, it won't work all of the time, but maybe it's not necessary.

It might be easier to reach at it via making a creation and display layer though, you're right.

5\. It seems like macroexpand is basically what I'm looking for -- at least it's powerful enough to do what I want it to do without too much pain. I'm unfamiliar with Slime's disassemble command, and the first Google page was unhelpful...

As for language support for provably correct programs, they sound interesting to me, too. They should definitely not encompass the whole language though. To be honest I'm not really sure that the mathematics is where it needs to be. So many proofs rely on obvious things, not 'obvious' in the naive sense, but actually obvious, and it seems like such a pain to program these in that provably correct systems are impossible to use, at the moment. Yet, Doron Zeilberger seems to think otherwise.

I don't know how to escape the problem of infinite regress, or, set differently, Godel's incompleteness theorem. Any sufficiently powerful proving system we have is unable to prove it's own consistency, so, as someone who is admittedly unfamiliar with this territory, I don't see where the starting point is for so called provably correct programs.

What would you guess the _next_ step in the right direction is?

<comment><USER2> 1, 2. I think it would be nice if editor support made it easy to switch back and forth between math notation and standard syntax.

3\. Lisp is nice, but the implementations have issues. I'm mostly satisfied with SBCL at the moment, but I keep writing utility functions to get around the warts in the language. Languages seem to be an instance of the "all X sucks - some just sucks less" pattern.

4\. I think providing explicit support for state machines would be a Good Thing even without the visualizer.

5\. Something that lets you look at an expression as you step through evaluation and substitution would be useful. Macroexpand does this at compile- time, but I want something similar at runtime for parts of the expression that aren't macros. I suspect something like this exists, but I haven't seen it yet. Slime's disassemble takes the name of a function and displays the corresponding assembly code in a buffer.

I agree that it's not practical at this point to try to completely prove the correctness of programs; I think it's more useful to prove specific aspects of a program as an alternative or supplement to tests. It's also useful to make assumptions about things you can't prove in the mathematical sense but can rely on when writing or running a program.

What's next? I've been meaning to download and try out Qi, a language that builds on top of Common Lisp and has a type system based on sequent calculus. It looks like Qi allows users to start with fairly general types and narrow them down as the program develops.

I didn't enjoy OCaml when I spent a couple days trying to learn it. The restrictions of the type system were frustrating, though I did get the feeling that it's a much better way to keep people from breaking stuff in a "software engineering" situation than using Blub.

</comment></comment></comment><comment><USER3> Graphical programming would be a 'killer app' for any language that can implement it first. Reasoning about code visually will not only make programming more accessible, but it will allow us to apply spatial reasoning skills to the task of writing applications. Flow-based programming seems to have the best chance of getting this sort of thing off the ground: <http://www.jpaulmorrison.com/fbp/>

<comment><USER4> Sorry to sour your future-casting, but this "killer app" has been a non- starter for many many years. Except for some niches where the flow of data is the _primary_ thing you want to express, visual languages are too cumbersome. See <http://en.wikipedia.org/wiki/Visual_programming_language>

</comment></comment><comment><USER5> My favorite feature is "i", followed by ":w".</comment>


Should the government know less than Google? - <AUTHOR> http://www.economist.com/blogs/democracyinamerica/2013/06/surveillance-0?fsrc=scn%2Ftw_ec%2Fshould_the_government_know_less_than_google

<comment><USER2> Most of the private data Google collects about individuals is provided freely by the people. (Perhaps not all, and maybe that should change, but the point still stands.) Google and others use it to target advertising. The government uses it to monitor people for perceived wrongdoing and imprison them. There's a difference.

Privacy isn't necessarily about keeping everything private from everyone, it's also about choosing who you reveal information to. It is perfectly legitimate to be willing to reveal private information to Google, but wishing to prevent the government from being able to warrantlessly scoop it up as well.

<comment><USER3> Agreed. However I think there's also a certain agreement in place with companies like Google. I willing accept to share my locational data with you in exchange for you telling me about a new coffee shop or local offer. I can weigh the pros/cons of that to ultimately decide if I opt-in.

I don't have a similar (or any) agreement with the government.

<comment><USER2> And furthermore, if the government had had its way, we still wouldn't know they were doing this. (And they haven't admitted to everything they're apparently doing, judging by more recent reports.)

</comment></comment></comment><comment><USER4> It bears repeating: "The problem isn't so much that we haven't set up a legal architecture to preserve our online privacy from the government; it's that we haven't set up a legal architecture to preserve our online privacy from anyone at all. If we don't have laws and regulations that create meaningful zones of online privacy from corporations, the attempt to create online privacy from the government will be an absurdity."

There's also the perverse result that government has little incentive to pass laws protecting personal privacy from corporations.</comment>


Google's Public DNS intercepted in Turkey - <AUTHOR> http://googleonlinesecurity.blogspot.com/2014/03/googles-public-dns-intercepted-in-turkey.html <comment><USER2> If anyone is in Turkey and can configure their system to use Google Public DNS IPs / OpenDNS IPs as their resolvers, I'd be interested to learn what you see on: [http://whatsmyresolver.stdlib.net/](http://whatsmyresolver.stdlib.net/)

It's clear that addresses such as 126.96.36.199 are being intercepted. But it's unclear if the interceptor is passing on the uninteresting queries to the "real" Google Public DNS / OpenDNS, or if they are acting as standalone resolvers. The above website records what IP address the resolver uses to communicate with authoritative DNS servers.

<comment><USER3> 188.8.131.52

<comment><USER2> Thanks! that looks like they are running standalone resolvers:

        inetnum:        18.104.22.168 - 184.108.40.206
        netname:        TR-TELEKOM-960902
        descr:          Turk Telekomunikasyon Anonim Sirketi

in that case, each ISP is probably running their own standalone servers.

<comment><USER3> Nope, this is not an ISP. Turk Telekom is stated-owned broadband internet (infrastructure) company. All of the ISPs are having the same peering. I checked your link through TTNet. We also have Superonline and Smile as main ISPs.

<comment><USER4> Turk Telekom is not state-owned, not for the last 8 years or so.

<comment><USER5> Details courtesy of Wikipedia: Privatized in 2005. "55% of the shares of Turk Telekom belongs to Oger Telekomünikasyon A.Ş. and 30% of the shares belongs to Undersecretariat of Treasure of Turkey. The remaining 15% of shares has been offered to the public." Oger Telekom is apparently 100% privately owned by the Hariri family.

So, still 30% state owned. As one point of reference, the German government still holds 15% directly plus 17% indirectly of Deutsche Telekom. France and Orange (nee France Telecom) are similar.

</comment></comment></comment></comment></comment></comment><comment><USER6> This is bad if true, but there aren't any technical details concerning the nature of the attack or how they discovered it.

Is it possible to claim that you are 22.214.171.124 from within Turkey using some sort of BGP wizardry?

<comment><USER7> They are using an BGP hijack attack: [https://twitter.com/TelecomixTurkey/statuses/449924292914839...](https://twitter.com/TelecomixTurkey/statuses/449924292914839554)

<comment><USER7> Looks like this is actually being done through IGP: [http://www.bortzmeyer.org/dns-routing-hijack- turkey.html](http://www.bortzmeyer.org/dns-routing-hijack-turkey.html)

</comment></comment></comment><comment><USER8> Only thing they've achieved is starting a unwinnable game of public DNS whack- a-mole.

<comment><USER9> considering that 188.8.131.52 was widely graffiti'd across Turkey, it stands to reason that informing people about alternative dnss might be a labourious task.

</comment></comment><comment><USER10> I'm curious about how all this seams reactive and not creative… Isn't the reason for this an audio recording of officials suggesting to make-up a _casus belli_? Sounds easy enough to share as an e-mail attachement, is it? Or USB memory dongle, or in pirate radios…

I don't mean to downplay the overall technical side, it's… well quite exactly it is on of Jon Zittrain's nightmare happening live, so that's wonderful in an incredibly scary and nerdy way, but… My question is: Why isn't the cat out of the bag a thousand times, yet?

<comment><USER11> The turkish people already know the government is corrupt so the cat is out of the bag in that sense (they didn't need these recordings to know that).

But twitter etc allow much easier spreading of information and organising protests than do email, memory sticks etc

</comment></comment><comment><USER12> Let me confess upfront to being ignorant when it comes to demographics and statistics about Turkey... but it looks like TT is a pretty big player WRT Internet in Turkey.

As of a few moments ago, I see ~5,363 prefixes in BGP originating from TT or one of their downstream peers, of which there appear to be ~307 (my numbers are approximate and almost certainly not exact, due to the way I came up with them). That's a whole lotta people that are affected by this.

</comment><comment><USER13> It's fucked up in a morbidly fascinating way how everyone here is just going through the technical details of how this particular variety of oppression works, instead of seeing the oppression itself.

What's going on is that _a government 's subjects_ are saying things it doesn't want them to say, and the government has decided to "prevent" it by force. You're right in that it's a bit of a losing game.. up to the point where the oppression gets so severe that the population stops resisting out of fear.

The root problem here, again, is that people believe they should have rulers. They should not, and Turkey is helpfully providing yet another example of why.

<comment><USER7> I'm not sure it is a losing game. It depends on the goals the government wishes to achieve.

Certainly this filtering has made access more difficult for segments of the population, it has sent a message that the government is willing to employ outright censorship. If I worked at a newspaper or ran a social network inside Turkish boarders I would be very worried.

Governments use censorship because they believe it will allow them to achieve their goals, often they are right. Consider: "If Erdogan's party manages to sustain its early lead as the ballot count continues, it would suggest such troubles have been largely shrugged off by many of Turkey's over 50 million eligible voters." [http://news.yahoo.com/turkeys-embattled-pm-faces-key-test- lo...](http://news.yahoo.com/turkeys-embattled-pm-faces-key-test-local- elections-025354614.html)</comment></comment>


Probabilistic Assertions: Crashing When Something Feels Wrong - <AUTHOR> http://www.spiteful.com/2008/04/14/crashing-when-something-feels-wrong/ <comment><USER2> Yay for automated monitoring software. Nagios (<http://www.nagios.org/>) does this for networks (and is extendable for some other things). At my old job we used Hobbit (<http://hobbitmon.sourceforge.net/>) to watch our Java server instances (memory usage, etc.). There's no reason why these monitoring programs couldn't be used to monitor internal program statistics, as long as those stats were made available.

Generally you monitor from your internal network, and then provide some hook for the monitor to get information that's only accessible from there. (SSH or a limited-access URL, etc.)

Monitoring programs are super-powerful and generally complex. Check them out — it's a good skill to have when working with production software.

(I also posted this on the article.)

</comment><comment><USER3> I've worked with threshold logic like that for collecting and analyzing traffic on telephone switches where an alarm or notification would be generated if the threshold was broken.

Personally I would never want to debug something like that using a statistical probability that something _might_ have gone wrong. Better to fail gracefully with something like multiple chains so that when a request chain goes down it gets logged, cleaned up, and recreated.

Worst case scenario they get a request timeout warning.

</comment><comment><USER4> This sounds like circuit breakers for software. Instead of an over current condition you've got excessive busynesses.

</comment><comment><USER5> Maybe frequent backups would be a better solution?

<comment><USER6> The "delete all of a user's files" example was just one of many. It is sometimes valid to force a crash when the code detects an impossible case (or when you choose not to recover from that case for simplicity). If each component either functions correctly or the program crashes, that can significantly reduce debugging time. There aren't any silent failures from components that simply ignore invalid input.

That said, with procedural languages I try to write code that recovers from all reasonable cases and assert the rest.

<comment><USER5> What if the validation code itself contains the errors. It might be a fallacy to assume problems can be solved in that way.

<comment><USER6> "Solved" certainly shouldn't mean "avoided entirely". One way to write good software is to quickly recover from mistakes. So instead of staring at your code while you triple-check each corner case, you approach the problem from the other direction by testing each code path until you're confident that the code runs as it should for the inputs that matter. (Be sure to assert for all other inputs).

In that light, it is easy to see why it can be good to crash for invalid cases. Since you don't handle a bunch of failure cases, you end up writing less code. And since less time is spent in the debugger, more time is focused on the correct task: achieving architectural goals rather than solving structural problems.

After you acquire a deeper understanding of the architecture, it is best to redesign (throw away) your previous attempt. After the second and especially the third iteration you will have written a solid and elegant program in a relatively small amount of time. Programming is about trusting yourself to make decent decisions based on your current knowledge.

As with anything, there is only a finite amount of time to solve a problem. So if you don't have a lot of time then don't fret if your code isn't perfect (or reusable) as long as it works.</comment></comment></comment></comment>