Ask YC: Any ideas about intelligent crawlers :) - <AUTHOR> Hi everyone. Not sure if I should be posting this to a forum or not. Curious to see what peoples answers are here as I read this site a lot
I'm thinking of creating an intelligent crawler in Python. I have a project with a friend where we'd like to crawl a few specific car-related websites, grab some of the info and look for new entries.  I am wondering if there is any existing technology out there where a crawler is sent to a site and either trained (visually?) or which can understand repeating information like tables that we could use to create a proof of concept.  I'd appreciate any critique of my ideas which is:
1. create a visual tool - probably windows/mac based which uses the browser to navigate a site and to highlight elements that we would like to capture, such as car name, description, price. This would also have to be able to automatically/manually work out repeating elements
2. this tool would create some kind of file (xml?) which would then be used by the main crawler to understand how to navigate the site
3. The crawler, which we'd write in python would visit the site every week to look for new information
Am I going about this the right way or does anyone have any ideas
One point, we would seek permission from the sites before crawling - it would be to their benefit as we're looking to push people their way.
Appreciate any thoughts anyone might have
All the best
John <comment><USER2> I wanted to do this same thing a while ago and have done a lot of research and reading in this area. Here are some search terms that will likely help you:

* automatic wrapper generation

* information extraction

* removing noisy information from Web pages

* template detection

* wrapper induction

"Wrapper" is a fancy computer-science term for "scraper."

I wrote some Python code that does this -- given X sample documents, detect the differences between them and automatically create a scraper tailored to those documents. I released the first version open source -- it's called templatemaker: <http://code.google.com/p/templatemaker/> .

But that version of templatemaker is quite brittle, because it was designed to work on plain text as much as on HTML. I've since written an HTML-aware version of templatemaker that is really frikkin' awesome (if I may say!) and beats the pants off the old one. I don't know if I'm going to open-source it, as it's quite valuable to my own startup.

Hope this helps!

</comment><comment><USER3> Don't write your own crawler. Use nutch.

It is designed to scale and do mapreduce kind of parallel processing. I would strongly recommend you to take a look before writing your own.

<http://lucene.apache.org/nutch/>

<comment><USER4> mapreduce? Just how many requests will you be making to third-party sites at once? Sounds like a good way to get blocked fast.

</comment></comment><comment><USER5> I think you are in over your head, but it's a great way to learn about the plumbing and underbelly of the Web.

This visual tool is basically what a company called onDisplay was doing back in 1999, before they were bought by consulting firm Vignette for an obscene amount of money. But scraping against the html structure is a losing battle.

A better approach is to use clues in the information itself to guess its content: something with a "$" is a price. Something containing "toyota" is probably a name, "blue" a color, more than 20 words containing "good", "v8" is a description, etc. That way your scraper is resistant to structure changes.

All that is separate from the problem of a crawler. It takes a long time and a lot of effort to convince content sites that what you are doing is a) helpful to them and b) something they should not be doing themselves.

It's like jumping on stage with the band and starting to play. You better be really good and friendly and prepared to get the crap beaten out of you.

</comment><comment><AUTHOR> Not sure if this is appropriate - as not responding to one particular person - but I'd like to send a BIG Thank You to everyone out there for the advice, encouragement and sometimes the reality "keep your feet on the ground" type stuff. This has inspired me to move forward with this. To those who have done stuff like this before, thanks for the links and I'm grateful for you sharing your experience.

If its OK I'd like to let people know about my experiences. Oh, if anyone is interested in collaborating or just sharing ideas then I'd be happy to do likewise

All the best

<comment><USER6> Good idea. Add an email or website at <http://news.ycombinator.com/user?id=franklymydear> and then you can exchange private messages by multicast rather than broadcast.

Feel free to ask me more questions by email. I spend a fair bit of time thinking about html parsers.

<comment><AUTHOR> Thanks! I've just done this. Going to do some research into this area and then make a plan to start in the next week.

</comment></comment></comment><comment><USER6> For 1 you mean you want to build a _parser_ for arbitrary html that your crawler returns. Hard problem, as others have said. My advice:

1\. Use an html parsing library. Beautiful soup (python) or hpricot (ruby) are good building blocks.

2\. Practice manually building parsers for a few sites, then see if it leads you to any insights about how to generalize the process.

3\. Ignore everything else until you do 2. Just use wget as your crawler. Skip the visual interface for now; just parsing arbitrary pages is a hard enough problem to bite off.

</comment><comment><USER7> Someone mentioned dapper.net and I upmodded it, but I think it will get lost in the noise.

As far as I understand, they are very close to what you are trying to do, so study them carefully as a competitor.

<comment><AUTHOR> Yes, this is close to what I want to do in terms of functionalioty. My idea was to use a wizard-like approach to record the elements of a page that we need to capture and how to navigate through a specific site. They appear to be doing this, though the system has failed a few times on me - they're doing it in a browser, whereas I'd planned to create an app. Very interesting though. Anyone who knows of anything similar or who is interested in building something like this, get in touch

<comment><USER7> Actually I might be interested - please leave an email.

</comment></comment></comment><comment><USER8> I know in php it's possible to load an html document and parse the DOM tree using XPath expressions, presumably that capability exists in python.

So i guess in theory you could write a frontend (firefox extension?) where you could highlight / select a screen area (webdeveloper already does this), then pass it's DOM information (i.e. #body table tr td#username ) to your backend, which would then scrape that field(s) from any applicable site pages.

This of course assumes that 1) The website(s) are well formed enough for your parser and 2) Well programmed enough that the same info is in the same place in the DOM tree, and preferably ID'd, which are pretty HUGE assumptions, but could be worked around if you were determined enough.

Not sure if this is what you're looking for, and it seems a bit circuitous, but it's a plausible idea anyway.

</comment><comment><USER9> I have built up something similar to what you are describing and it was a fun project. My first reaction to #1 is that if the information you want to repeatably in the same place you are probably better off just doing things manually and not going to the trouble of building a visual tool. In my experience the interesting pieces of data tend to move around and something like regex is the best way to handle this. I used wget to grab data because it was quick and easy. I then did the post processing in the background, separating out the grabbing of data and the interpretation of data.

</comment><comment><USER6> I remember seeing a screencast of some startup that did something like this. It was maybe a year ago. You click on elements, it shows you other hypotheses, you correct if necessary, and you get an RSS feed compiled from the page structure.

Anybody remember this?

<comment><USER10> dapper.net

<comment><USER6> Yes!

</comment></comment></comment><comment><USER11> What you are talking about is the deep web I think and I don't think anyone has managed this yet! Essentially you want a system that can fill in forms and pull back results. I think it needs to be done on a per-site basis

</comment><comment><USER12> John,

One thing that I would point out about the script you intend to write is that it requires an awful lot of maintenance (when sites change layout) and is frequently not very reusable. One solution that I tried is Mozenda (<http://www.mozenda.com>). The have all the stuff you're looking for (i.e. visual, browser-based tool, writing to XML) but also have error handling and notifications, so that if an agent breaks you'll know and be able to fix things inside the visual tool.

</comment><comment><USER4> It's called "scraping" and I've done it lots of times with python, very easy. Don't bother with those other specialized, non-python, frameworks that people are suggesting.

<http://wwwsearch.sourceforge.net/mechanize/>

And if you need to do complicated html parsing in combination with that:

<http://www.crummy.com/software/BeautifulSoup/>

From there, it's cake.

</comment><comment><USER13> I know this is ruby but might be worth a look. Will help you no end and no need for a web browser.

<http://mechanize.rubyforge.org/mechanize/>

might be worth a look <http://www.crummy.com/software/BeautifulSoup/>

</comment><comment><USER6> Another talk I saw at shdh in october (<http://superhappydevhouse.org/SuperHappyDevHouse20>):

<http://tagtheplanet.net>

They seem to be attempting an intelligent crawler as well.

</comment><comment><USER14> You ought to check out screen-scraper (<http://www.screen-scraper.com/>). It's a commercial app, but the best I've used for this kind of thing. They also offer a freeware version.

</comment><comment><USER15> What do you want to do?

At least in Germany, there exists a few solutions which do exactly that. If a person puts his car on sale (a bargain) on one of the car related websites, he get's the first call in about 20 seconds from someone using these programs.

</comment><comment><USER16> And still one thing .. if you want to ask the site owner for permissions, why not ask them to produce some specific xml file for you?

<comment><USER6> Because granting permission is easy. Why would they go to more effort than that for random people?

<comment><USER16> Why thousands of job sites produce custom xml output for simplyhired or indeed ??

<comment><USER4> They like buzzwords?

</comment></comment></comment></comment><comment><USER17> webharvest ?

</comment><comment><USER18> Does anyone want to write a search engine? Python and C++. I thought we could analyze the links between pages and come up some kind of ranking algorhythm. We'd have to make the system widely parallel - but I think we could be breaking some new ground here.

Any takers?</comment>


Appsdeck, EU-based PaaS, is now in open beta - <AUTHOR> http://blog.appsdeck.eu/post/84518081058/appsdeck-is-now-in-open-beta

<comment><USER2> Honestly, I don't get the pricing. (this applies to Heroku too, but they use Amazon). A bare metal server like this[1] priced at 130 EU/month should easily host (let's be generous and not oversell the CPUs) at least 100 containers at 20 EU/month = 2000 EU/month. Even with all the convenience of a PaaS, this doesn't seem like a good deal.

[1] [http://www.hetzner.de/en/hosting/produkte_rootserver/px120](http://www.hetzner.de/en/hosting/produkte_rootserver/px120)

<comment><USER3> Founder here.

1\. when we'll charge a credit card (which should happen in the coming weeks as we're incorporating the company), we'll be already cheaper than Heroku in a price/performance ratio 2\. to be honest we're more pessimistic than you about the number of containers that we can host on a single server like the one you cited (at Appsdeck we use only SSD equipped servers with more cores than in the machine you cited)

That said, our price could change a little bit depending on the real workload that our cluster can handle in real life.

<comment><USER4> As a happy (small time) Hetzner customer, I concur, I don't think it's quite so rosy as parent hints at. The most crucial part: look at the bandwidth. Peek at 1Gbps (probably more like 5-600 after overhead), guaranteed 200 Mbps. If you divide that by 100, that's just 2Mbps guaranteed bandwidth per container! Sure, you can oversell, but then you're in a different game.

</comment></comment></comment><comment><USER5> No imprint, no names, anonymized whois, no contact details except the mail address, blog hosted at tumblr, main site hosted at a bulk web hosting company called [http://www.devclic.fr/](http://www.devclic.fr/)

<comment><USER6> Related Github profile lists location as "La Plage Digitale, Strasbourg". Googled that + Appsdeck, discovered La Plage Digitale is a coworking space. Found a couple of articles tagged Appsdeck, which led me to Yann Klis:

[https://twitter.com/yannski](https://twitter.com/yannski)

It shouldn't be that hard. An imprint / contact details page is a minimum requirement for anyone offering a service such as this for money.

<comment><USER3> Imprint and terms of service is the minimum and we know that. Technically we don't charge for money ATM.

</comment></comment></comment><comment><USER7> Congrats on launching the beta.

Some suggestions for the home page though. Nothing above the fold tells me what AppsDeck is. The title and content are talking about how you're no longer in beta. For someone who didn't know you existed until 5 minutes ago you lost the chance of telling me what AppsDeck does.

Best of luck with the service and kudos for putting it out there.

<comment><USER3> Thanks for the suggestion.

</comment></comment><comment><USER8> Does anyone know other EU-based PaaS providers? One that comes into my mind is CloudControl [1]

[1] [https://www.cloudcontrol.com/](https://www.cloudcontrol.com/)

<comment> vertex-four How about EU IaaS? I'd love to see something that's not under US jurisdiction, but provides, say, an OpenStack service (and potentially value-adds).

<comment><USER3> Check [https://www.runabove.com](https://www.runabove.com)

</comment></comment></comment><comment><USER9> Was in the private beta, have been using this one for the last 4 months on a customer project. Quite good, everything you would expect from Herok-eu.

</comment><comment><USER10> Hey guys, cool project! I always love seeing new Docker-based project emerge. I'd love to talk more - get in touch: firstname.lastname@example.com

</comment><comment><USER11> Already use this service for several projects, works really well and the tech support is always available ;)

</comment><comment><USER12> Congrats on launching ! :) #FrenchTech!

<comment><USER3> Thanks buddy!</comment></comment>


World's steepest funicular rail line opens - <AUTHOR> https://www.theguardian.com/world/2017/dec/15/world-steepest-funicular-rail-line-open-switzerland <comment><USER2> _A traditional aerial gondola funicular was ruled out because it would have had to pass through an active shooting range._

So...couldn't have been cheaper to buyout (and close) the shooting range and use traditional gondola design

</comment><comment><USER3> It's a marvel of engineering, no doubt, but that seems like an awful lot of money to spend on a system to serve a village with only 100 people.</comment>


Visualize Your Data with Google Public Data Explorer - <AUTHOR> http://www.readwriteweb.com/archives/visualize_your_data_with_google_public_data_explor.php?sms_ss=hackernews&at_xt=4d5c3360e988f8aa%2C0 <comment><USER2> There is a cool startup on this space called findthebest.com - clearly, there is a lot of interest in making all sorts of public data available in a more "browsable" format. I'd like to know how other people feel about startups like factual or findTheBest.</comment>


Nokia Asha 501: Impressive Low-end Touchscreen Phone for $99 - <AUTHOR> http://techpp.com/2013/05/09/nokia-asha-501/ <comment><USER2> Hardly impressive - fairly pathetic really. Observe:

[http://androidandme.com/2013/04/smartphones-2/archos- reveals...](http://androidandme.com/2013/04/smartphones-2/archos-reveals- stock-android-phones-starting-at-99/)

[http://www.androidforcheap.com/cell-phone-android- phone-c-58...](http://www.androidforcheap.com/cell-phone-android- phone-c-58_53.html)

[http://www.pandawill.com/mobile-phone-c1/android-os- phone-c5...](http://www.pandawill.com/mobile-phone-c1/android-os- phone-c551.html)

etc. Just Google around for tons more.

Generally better in just about every way, and running a real OS that can run real apps.

<comment><USER3> I think there was a quad core 1080p Chinese phone for $160, too. Considering these are running Symbian, that's a lot less value for buck compared to an Android one, even if the price is "low".

EDIT: This one iOcean X7 - quad core 1.2 Cortex A7, 1080p screen, PowerVR 5XT (1080p video support), 2 GB of RAM, 13 MP camera. Now _that's_ impressive. And it doesn't look too bad either:

<http://android-sale.com/iocean-x7-phone.html>

<comment><USER4> These are not running Symbian. From the article:

> Nokia Asha 501 represents the first fruit after Nokia bought Smarterphone, > back in November 2012. And maybe thanks to the Smarterphone software, Nokia > has managed to pack so much inside a tiny, 3inch touchscreen device.

As for the iOcean, I can't find a review about it anywhere on the web in English. Kind of risky.

<comment><USER5> Indeed, it is running on a modified version of the S40 platform.

<comment><USER6> That's correct. That means the "app" runtime on Asha is J2ME plus some Nokia- specific classes. Only pre-installed software can access the S40 OS directly.

That's not competitive with Android, Sailfish, or Firefox OS, all of which are already or will soon be in comparably priced devices in the developing world markets.

Nokia should have kept Meego and/or Meltemi to remain competitive in this segment. They also could have sold Android alongside Windows Phone, as other OEMs do.

</comment></comment></comment></comment></comment><comment><USER7> This "used-car salesman" undertone, wordily mulling over the same points, and over the top praise are very typical of most Indian magazines and newspapers even. I'm not trying to be nasty here, even I am Indian and not a native speaker of English. I just find this style of writing very amusing and can recognize it anywhere. :-)

EDIT: I was wrong about 'recognizing it anywhere', this article isn't from an Indian publication. No offense.

<comment><AUTHOR> It is, but I don't get your point. Though the writer is from Romania

<comment><USER8> By "the writer", you mean yourself, right? I think the GP is just pointing out how unusual in style such articles are compared to, say, US tech websites. I very much doubt you'd see a sentence like "There are many things that might steal your heart away at this device" there.

A more cynical man than I might label it a gushing puff piece garnered by some expenses paid Nokia PR event, GP points out cultural differences that might belay such an accusation.

</comment></comment></comment><comment><USER9> For all the non-impressed people. It was unveiled in New Delhi, meaning it focus group isn't western world. It is not meant for you, who already own a smartphone.

And for its focus group, it is pretty darn impressive. Battery life for people who don't have toilet, not mentioning wall outlets.

edit: I think it would be perfect phone for kids first phone. Simple, cheap, good battery, durable.

</comment><comment><USER10> On a conceptual level, I like the approach of building up a mini-smartphone from the bare essentials. A few years back Nokia acquired a company called Smarterphone that was doing this[1], and this updated Asha platform is apparently the first fruits of the acquisition.

The alternative is to take the "kitchen-sink" platform of Android which is actively developed for quad-core phones and full-HD displays, and try to pare it down to run on low-end hardware that's an order of magnitude slower and also has an order of magnitude less pixels to work with... While still preserving compatibility with half a million existing apps.

That's of course pretty much impossible, so instead the super-cheap Android devices are running old software from several years ago with extra limitations imposed by the low-end platform.

The difference in the two approaches is analoguous to Windows RT vs. iOS on tablets -- the other party tries to cut down the full system for cheaper devices, while the other builds up from software that's been proven on smaller devices.

\--

[1] According to previous reports, the Smarterphone OS had an app platform written in Scheme! <http://en.wikipedia.org/wiki/Smarterphone>

...I doubt anything of that survives in the Asha platform, but it would be cool to do low-end smartphone apps in Lisp :)

</comment><comment><USER11> If this is a return to the good old days of being able to throw your nokia down a flight of stairs without even dropping the call, then count me in.

<comment><USER12> Wait, you can make calls with it?

</comment></comment><comment><USER13> Interesting to see lots of influence from the N9. The interface seems quite good on this one, there's a little demonstration by Peter Skillman, starting at about 2:00: <http://www.youtube.com/watch?v=D6iMsePF0kk>

</comment><comment><USER14> If by "free access to facebook" they mean no data charge for facebook access for the lifetime of the phone, I wouldn't be surprised by this being a hit with the college crowd... especially if the camera is decent.

<comment><USER15> A lot of carriers provide "free" access no matter what phone you have <https://www.facebook.com/blog/blog.php?post=391295167130>

The catch is the videos/photos are not free, nor are the links to other websites of course.

</comment></comment><comment><USER16> ZTE produces tons of Android devices for this amount or slightly less. The ZTE Blade being one example: which has been out three or four years.

This seems less "impressive" and more "impressive marketing." Why are we reading about this? How does it advance anything? A worse phone for slightly more money, because it has "Nokia" on the front?

<comment><AUTHOR> Did you read the entire article?

</comment></comment><comment><USER17> A nice, cheap, small phone with dual-sim? Isn't this just like every other Asha phone they make?

<comment><USER18> Awesome build quality and impressive design. The screen size could've been bigger though

</comment></comment><comment><USER19> This is awesome and may well be my next phone! Going to wait for the 3G version, though.

</comment><comment><USER20> Seems like a better "Facebook phone" than the one on Android.

</comment><comment><USER21> So, who will write J2ME apps for them?

</comment><comment><USER22> Can I put Android on it?

<comment><AUTHOR> why?</comment></comment>


5 reasons tech startups are like indie bands – but with better hygiene - <AUTHOR> http://blog.battletrip.co/2013/06/23/5-reasons-tech-startups-are-like-struggling-indie-bands-but-with-better-hygiene

<comment><AUTHOR> I think the "with better hygiene" is highly debatable.</comment>


Singletons are Pathological Liars (2008) - <AUTHOR> http://misko.hevery.com/2008/08/17/singletons-are-pathological-liars

<comment><USER2> I'm not one to defend singletons (been burned by them myself), but this is a pretty poor example of why they suck. Any singleton that requires an outside .init() call is a pretty poor singleton. The whole point vs. globals is that they are lazy loaded. Why not:

      getInstance() {
         if(_instance == null) { init(); }

         return _instance;
      }</comment>


Hong Kong closed schools. Singapore did not. hardly any diff in transmission - <AUTHOR> https://www.washingtonpost.com/opinions/2020/03/21/facing-covid-19-reality-national-lockdown-is-no-cure/ <comment><USER2> I haven't consented to the WaPo site so not read the article.

I live in Sweden, with preschoolers. Unlike in neighboring Norway and Denmark, the preschools and elementary schools have not closed down.

According to [https://www.folkhalsomyndigheten.se/smittskydd- beredskap/utb...](https://www.folkhalsomyndigheten.se/smittskydd- beredskap/utbrott/aktuella-utbrott/covid-19/fragor-och-svar/) under "Varför stänger vi inte för- och grundskolorna i Sverige?" (Why don't we close pre- and elementary schools in Sweden?)

> We believe that it is not at present a meaningful measure to close all > schools in Sweden. There are no scientific studies showing that such a > measure would have any significant significance. Also, there are no data > pointing to any major spread of covid-19 in schools anywhere in the world.

> Closing preschools and primary schools would also have negative consequences > for the rest of society. For example, many parents who work in healthcare > and other socially important functions would need to stay home with their > children. It could also lead to vulnerable groups, such as grandparents, > helping to care for the children.

That said, the interview with Christian Drosten in Die Zeit (linked from HN) at [https://www.zeit.de/wissen/gesundheit/2020-03/christian- dros...](https://www.zeit.de/wissen/gesundheit/2020-03/christian-drosten- coronavirus-pandemic-germany-virologist-charite/komplettansicht) points out:

> ZEIT ONLINE: The country is grinding to a halt. Daycare centers and schools > have been closed nationwide as of this week. Is the government reacting > appropriately at the moment?

> Drosten: Few of the decisions made in recent days were purely evidence- > based. Many were political, and certainly correct. ...

> ZEIT ONLINE: It is said that you played a central consultative role in the > decision to close schools. Was it the correct decision to do so now?

> Drosten: I don't know. It probably won't be clear until later whether it was > the right time to do this. ... I don't see my job as condensing the truth, > but as explaining aspects of the truth and allowing for uncertainties and > saying, 'We don't know' – which makes a political decision necessary. And as > long as it's communicated as a political decision, I think that's fine.

</comment><comment><USER3> The title of the article is "Facing covid-19 reality: A national lockdown is no cure"</comment>


ShadowCrypt – automated text field encryption for websites - <AUTHOR> http://shadowcrypt-release.weebly.com/

<comment><USER2> Wow, it's not frequently that i learn of such a tool elsewhere (in this case, Nonsense NYC) and only then see it on HN! :)

Looks interesting/legit, i'll let a few early adopters play with it first.</comment>


Ask HN: Which AI initiatives are ethical and morally sound? - <AUTHOR> Elon Musk is drawing attention to this excellent documentary on the ongoing AI revolution, a documentary that brings it all together:
https://twitter.com/elonmusk/status/982119546420002817
(Free to watch until Sunday night EST, April 8)
The gist: AI is just a technology that can be used for good and for bad purposes.  But the AI field is mostly (entirely?) unregulated.
The AI revolution will have an exponentially bigger impact on humanity than tech revolutions that came before it.
Question: Given the huge implications, are there any (large) AI projects that take the ethical and moral aspects into account in their development? <comment><AUTHOR> Wow. 2 hours in the air and zero upvotes. Because AI is popular, I can only conclude that 'ethics' and 'morals' are not :(

<comment><USER2> That is the only conclusion.</comment></comment>


What It's Like to Face a 150 M.P.H. Tennis Serve [video] - <AUTHOR> http://www.nytimes.com/interactive/2015/07/10/sports/tennis/facing-a-150-mph-tennis-serve.html <comment><USER2> This is pretty interesting. It shows how much specialized a human can get at doing something. It truly is the closest thing we've got to superheroes.

</comment><comment><USER3> There was a post a month ago on how to write an app to measure serve speed using computer vision:

[https://news.ycombinator.com/item?id=9611298](https://news.ycombinator.com/item?id=9611298)

</comment><comment><USER4> That's great! What's changed in the last 30 years that serves are so much faster? Racket design? Balls? How we measure the serve?

<comment><AUTHOR> Several factors:

Rackets - they got more forgiving and powerful. I think even Mac now serves faster with a modern graphite racket than he did 30 years ago with a wooden one.

Strings - with the invention of polyester strings which snap back after "gripping" the ball during the stroke, the amount of top spin applied to the ball has gone through the roof. That means players can string their rackets at lower tensions, allowing even more power on their strokes (including the serve).

Overall physical fitness and training standards of pro tennis players - today's top tennis players are simply better athletes.

<comment><USER5> Poly definitely changed the game in many ways, but not as much when it comes to _fast_ serves. Even strung in the 40lb range, poly strings generate much less power than gut. You'll find that most of the big servers still use gut in the crosses (and they'd probably use all gut if they could just hit serves and no groundstrokes).

<comment><AUTHOR> The idea is that because of the poly in the mains you can generally string your racket at a lower tension (including the gut in the crosses), which increases power (without the gripping poly you couldn't use low tensions because the ball would fly on you on anything but the first serve).

<comment><USER5> The advantage of poly isn't that it grips, but that it's elastic enough to deform and snap back quickly. The old idea of string having 'bite' has been mostly disproven with the advent of high speed video to show what's really happening in the fraction of a second during impact. Those geometric extrusion strings are more marketing than anything else (and shred your stringer's fingers).

I play with full poly at 48#, but have experimented with gut hybrids and full gut at times. Even comparing 48# poly to 60# gut, the gut is like a rocket launcher compared to the poly. The amount of power it generates, at any tension, is unreal. You can hit huge flat serves with it, but the lack of spin isn't worth the extra pace.

<comment><AUTHOR> > The advantage of poly isn't that it grips, but that it's elastic enough to > deform and snap back quickly.

Sure, I should've put "gripping" in quotes - what I meant is the top spin potential, not the actual physics behind it.

> Even comparing 48# poly to 60# gut, the gut is like a rocket launcher > compared to the poly.

How about 60# full gut vs 50# hybrid (gut in the crosses)? Would the former still be a lot more powerful than the latter?

<comment><USER5> > How about 60# full gut vs 50# hybrid (gut in the crosses)? Would the former > still be a lot more powerful than the latter?

The hybrid setups I tested were usually in the ballpark of 46-48# poly mains with 50-52# gut crosses. My experience was that the hybrid setups were (obviously) more powerful than full poly, but still very noticeably less powerful than full gut at any tension. Without poly's spin/control, the full gut setups were a big disadvantage from the baseline, but I could hit some amazing (for me) first serves.

I didn't experiment too much though because I was breaking every gut hybrid I tried in a few hours, which got expensive quick.

</comment></comment></comment></comment></comment></comment></comment><comment><USER6> My nephew played in a minor tournament that Sam Groth won a couple of years ago. He said Groth did a second serve kicker that bounced over his opponent and continued over the back fence.

</comment><comment><USER7> I just watched 7 days in hell where he serves 174 mph. Just incredible!</comment>


Nonlinear Computation in Deep Linear Networks - <AUTHOR> https://blog.openai.com/nonlinear-computation-in-linear-networks/ <comment><USER3> ? It sounds like the author is ignoring denormals?

-edit-

Yes, the author is ignoring gradual underflow and the resulting denormal numbers.

So as you move from one binate to the next, the spacing between floating point numbers doubles or halves depending on whether you are increasing or decreasing the exponent. When you reach the binate with the most negative possible exponent, you have two choices: a) round toward zero, which leads to a huge non-monotonic jump in the spacing of numbers on the floating point number line. This is a great annoyance to numerical analysts and leads to convergence instabilities. That is why any modern computer used for numerical work incorporates choice b) gradual underflow, which implies that you must allow non-normalized numbers in the two binates (the two being + and - <USER2> bit) of the most negative exponent, which has the effect of creating another pair of binates around zero. This keeps the spacing of numbers on the floating point number line the same in the four binates around zero. Numerical algorithms are then much more stable.

I haven't looked at what GPU's do, I strongly suspect that they round toward zero, because first of all it doesn't matter much to graphics applications, and secondly, the typical method of handling denormals is to take a trap and drop into software emulated floating point because the cost of the additional hardware to handle denormals is very large and the hardware complexity is crazy-making. A GPU isn't going to want to break the pipeline for a denormal.

<comment><USER4> It looks like nvidia GPUs treat denormals as zeros for single-precision floating point math: [http://developer.download.nvidia.com/assets/cuda/files/NVIDI...](http://developer.download.nvidia.com/assets/cuda/files/NVIDIA- CUDA-Floating-Point.pdf) (sections 4.1 and 4.2)

<comment><USER3> In the context of graphics processing that trade-off totally makes sense.

Thanks for doing the homework that I was too lazy to do :)

It seems to me that in the context of NN computations, using the lack of gradual underflow as a non-linear element is going to severely limit the dynamic range of the neurons. On the plus side, the non-linear element is a computational freebie. But in addition to limited dynamic range, it makes the NN ridiculously non-portable across hardware implementations.

<comment> scott-gray Actually if you read section 4.6 of that paper you'll see that denormals are the default on sm_20 and above. But you can see in that same section this this can easily be disabled with the ftz flag.

I had to give Jakob custom gemm kernels to do this research. Not sure why the denormal point was left out of this blog as it's pretty critical to the whole experiment.

<comment> scott-gray So a minor correction here. We did explore placing ftz on various instructions inside the matmul ops, but it turns out you don't need anything more than what is already baked into tf by default. All tf gpu primitives are built with -nvcc_options=ftz=true. This means you have an implicit non-linearity after any non-matmul op (provided the scale of computation is near 1e-38). Matmul ops are called through cublas and have denormals enabled.

</comment></comment></comment></comment></comment><comment><USER5> These findings seem to be at odds. The former says that deep linear nets are useful, non-linear and trainable with gradient descent. The latter says that the non-linearity only exists due to quirks in floating point and that evolutionary strategies must be use to find extremely small activations that can exploit the non-linearities in floating point.

Exact solutions to the nonlinear dynamics of learning in deep linear neural networks

[https://arxiv.org/abs/1312.6120](https://arxiv.org/abs/1312.6120)

"We attempt to bridge the gap between the theory and practice of deep learning by systematically analyzing learning dynamics for the restricted case of deep linear neural networks. Despite the linearity of their input-output map, such networks have nonlinear gradient descent dynamics on weights that change with the addition of each new hidden layer. We show that deep linear networks exhibit nonlinear learning phenomena similar to those seen in simulations of nonlinear networks, including long plateaus followed by rapid transitions to lower error solutions, and faster convergence from greedy unsupervised pretraining initial conditions than from random initial conditions."

Nonlinear Computation in Deep Linear Networks

[https://blog.openai.com/nonlinear-computation-in-linear- netw...](https://blog.openai.com/nonlinear-computation-in-linear-networks/)

"Neural networks consist of stacks of a linear layer followed by a nonlinearity like tanh or rectified linear unit. Without the nonlinearity, consecutive linear layers would be in theory mathematically equivalent to a single linear layer. So it's a surprise that floating point arithmetic is nonlinear enough to yield trainable deep networks."

<comment><USER6> The arxiv paper here is analyzing the nonlinearities in a network's learning dynamics; exploring why training time / error rates are not do not vary linearly throughout the the training process.

They note: "Here we provide an exact analytical theory of learning in deep linear neural networks that quantitatively answers these questions for this restricted setting. Because of its linearity, the input-output map of a deep linear network can always be rewritten as a shallow network."

</comment></comment><comment><USER7> It's super interesting to think that _any_ non-linearity at all can make it work. This particular non-linearity is surprising since it's clamping to zero at the _center_ of the response curve. I'd have thought that's right where you want the linear response, and that clamping in the middle would cause bad things to happen. Sigmoid and RelU (and others) clamp at the foot/shoulder. Perhaps this network just learns negative weights, compared to the traditional activation functions??

<comment><USER8> there's a theorem that any nonlinearity works (for sufficiently sized networks).

<comment><USER9> The universal approximation theorem actually assumes that the nonlinearity is monotonically increasing, nonconstant and continuous. I don't think floating point nonlinearities technically satisfy that.

<comment><USER8> 0) nonconstant. Yes, for most cases the floating point nonlinearities map x => x, so it is not a constant.

1) bounded. Yes, the nonlinearites are bounded by the range of the FP.

2) monotonically-increasing. Yes. Consider a + b, where fp(a + b) < a + b, in other words, it's been rounded down. examine fp(a + (b - db)), cannot be rounded up to a number higher than fp(a + b), so the the floating point rounding functional fp must be monotonic for the operation +, a similar argument applies for multiply, and thus for any linear function.

3) continuous function. No. Well, you can't win at everything, no computer representation can be truly continuous, but it's reasonable approximation of the approximation theory, otherwise ML on computers _in general_ would be hopeless.

</comment></comment></comment></comment><comment><USER10> So this exploits the fact that floating point numbers have finite precision (and perhaps uneven spacing) to generate non-linear operations?

That's actually a really cool usage of the specification!

<comment><USER3> Well, except that the author misunderstands the specification and how it is typically implemented on modern computers.

<comment><USER11> (I work at OpenAI.)

TensorFlow by default is built with denormals off (ftz=true), so denormals aren't relevant for the applications we're interested in. We have updated the post to indicate this — thanks for the feedback!

</comment></comment></comment><comment><USER12> This is one cool hack.

You can't construct deep neural network from only linear parts because consecutive layers can be always combined into single transformation matrix. That's why you need alternating linear and nonlinear operations.

I wonder if it's possible to design special purpose low resolution floating point circuit that maximizes this effect while preserving enough linearity. Then you have fast DNN network pipeline constructed from just summation and addition.

<comment><USER13> It's also possible to binarize DNNs to use the faster bitwise operations

[https://arxiv.org/pdf/1602.02830.pdf](https://arxiv.org/pdf/1602.02830.pdf)

</comment></comment><comment><USER14> This is cool, but it seems like a crazy hack with no benefit besides "because it's there". Relu is already such a cheap function to compute.

</comment><comment><USER15> I didn't understand how the gradients are produced to honor this underflow behavior. Is that the reason why they use "ES" instead of symbolic (or probably they meant automatic) differentiation?

<comment><USER8> correct.

</comment></comment><comment><USER16> Well, that is cool and not what I would expect. That said, for practical applications make training easier using leaky-relu, tanh, etc. activation functions.

</comment><comment><USER17> I wonder if this / gradient descent would work with integers, and with mod 2^n as the nonlinearity.

</comment><comment><USER18> I guess the hope is that a round of computation can be shaved off.

</comment><comment><USER19> Wasn't this already obvious from simple networks which implement e.g. the XOR function?

<comment><USER20> Simple network without non linear function (eg sigmoid) can't learn XOR</comment></comment>


10 Considerations for the Startup planning to offer to the Enterprise (and why many will fail) - <AUTHOR> http://www.web-strategist.com/blog/2008/01/14/10-considerations-for-the-startup-planning-to-become-enterprise-20/

<comment><USER2> Check out point #8 in his list. As the comments suggest, this ones a killer. It took us nearly 8 months and we had to do a few free trials before we got it done finally. It also took weaks just to negotiate all of the contract points.

The upside? 3 year $1mil contract. They are hard but if you get them, the corporate contracts are sweet deals.

The secret is to offer technology and more importanty attentiveness that they just can't get from a more off-the-shelf solution.

</comment><comment><USER3> 5) Build a robust system that won't fail from heavy enterprise use, sadly, Twitter would never make it.

Huh?

<comment><USER2> Twitter failed spectacularly during macworld today. Specifically, right during SJ's keynote. Many people are fairly put out about it all.

<comment><USER3> I know - I actually meant that a popular consumer app probably has way worse scaling issues than a white-label enterprise system would have.</comment></comment></comment>


Ask HN: How would you make Emacs do this? - <AUTHOR>

For a long while now I've only used two applications almost all the time - a web browser, and Emacs.  For reasons I can't quite explain but feel strongly about, I'd love to get that down to just one.
So I have the idea of being able to have a web site that's actually just a gateway to an Emacs instance running on one of my servers.
I don't have a problem imagining how I might do the web app itself - it would probably use html/javascript for presentation, and then a small piece of Flash code to give me a real tcp socket that I can use to talk to Emacs.  No problem - I can do these kinds of things easily.
It would be useful to open and save-to local files in addition to ones already on the server.  Again, using a little bit of Flash 10, I can make that happen easily.
Now the frontend needs to tell Emacs, via this socket, whenever the user presses a key.  And Emacs in turn needs to tell the web app to update the display.  I could easily decide on a simple socket protocol for these things.
But on the backend I don't have much idea at all - I've modified my own .emacs a bunch, but I've never actually hacked Emacs.
Does anyone have any pointers?  Should I model this after an existing display toolkit (like motiff, gtk, etc)?  Or would a different approach work better?  Should I start writing this in ELisp, or am I very likely to actually need to drop down to C for some reason? <comment><USER2> I don't see why it would really be beneficial - People who want to connect with a remote Emacs session just use ssh + screen or tmux, which is stable and quick to set up. If you really want to implement Emacs-over-the-web, though, familiarity with these might help:

distel - distributed elisp, via integration with Erlang (<http://fresh.homeunix.net/~luke/distel/>)

TRAMP - remote file editing (See e.g. <http://www.emacswiki.org/emacs/TrampMode>)

Also, there are decent web browsers for Emacs. I like emacs-w3m. (<http://emacs-w3m.namazu.org/>)

</comment><comment><USER3> How about the opposite: browse the Web from within Emacs? <http://www.emacswiki.org/emacs/emacs-w3m>

[Edit: link to W3M, not the older W3]

</comment><comment><USER4> I've been thinking about this too ([http://blog.ianbicking.org/2010/05/04/browser-desktop- develo...](http://blog.ianbicking.org/2010/05/04/browser-desktop-developer- tools/)) -- but I figured I'd just let go of Emacs in the process, presumably to replace it with Bespin: <http://mozillalabs.com/bespin> \-- it is purported to have some Emacs keybindings (I haven't been able to enable them though).

</comment><comment><USER5> Hmm. Perhaps one of these would make a decent starting point?

<http://antony.lesuisse.org/software/ajaxterm/>

<http://anyterm.org/>

<http://testape.com/webtty_sample.php>

</comment><comment><USER6> You could look at Web Sockets (<http://en.wikipedia.org/wiki/Web_Sockets>). I know Chrome supports it, not sure about other browsers. A friend made a very cool interactive multi-user app using this and was quite impressed at the responsiveness.

</comment><comment><USER7> You can control Firefox from emacs using MozRepl. This extension allows you execute code on the browser. For example, you can tell emacs to refresh a specific tab when an html file is saved. MozRepl is not the only extension doing that. There are 4 or 5 similar projects.

More info <http://www.emacswiki.org/emacs/MozRepl>

</comment><comment><USER8> I think it would be useful as, say, a chrome plugin. A hotkey to open up a new emacs tab right next to my webpage tabs would be nice.

</comment><comment><USER9> How about something like this? <http://nomachine.com/web-player.php> Saving files locally might not be straight forward though.

</comment><comment><USER10> For editing remote files, I find that sshfs is extremely useful: <http://en.wikipedia.org/wiki/SSHFS>

<comment><AUTHOR> To clarify, I don't have a problem editing remote files - I use TRAMP right now (haven't tried sshfs, but it looks interesting thanks!).

The problem I'm trying to solve is that right now I need to install Emacs on any computer I want to do work on. I'd like to be able to jump onto a random friend's laptop and open Emacs without _any_ setup cost.</comment></comment>


Ask HN: Any good online CS / CE / etc. bachelors that's arent scams? - non-entity A lot of online schools / programs seem to fall under one or more of the following categories<p>- For profit online scams (University of Phoenix,  Full Sail, etc.)<p>- No online undergrad degrees or useless online undergrad degrees (it seems that the few reputable schools with online programs fall under this one)<p>- Not flat out scams, but the education seems subpar and I don't trust them after a cursory look? (SNHU, WGV, etc. A lot of these almost seem like slightly better bootcamps)<p>My question is, is there any decent online bachelors programs for fields like CS? <comment><AUTHOR> I'm a current student in the Harvard Extension School concentrating in CS and can answer any questions you have.

Edit: Should say for the record the Harvard program is not a scam, highly reputable (Harvard), classes / professors are the best I've ever encountered, students range from duds to awe inspiring as most places do.

<comment><USER2> Are you doing the undergrad or graduate program and do you live in the Cambridge area?

I am in the graduate program and agree that it has been excellent so far (about halfway done). In my most recent course I ended up presenting my final project at a well-known conference and have been working with the professor on another research project over the summer. Despite being in my mid-40s I am now considering a doctoral degree.

I think it shows that you are not a "lesser" student that snuck in the back door, you are a Harvard student with the same potential and ability to achieve if you work hard and want to make the most of the experience.

<comment><USER3> That's pretty cool! I'm also going through the graduate program. I'm two classes in and I'm taking two this semester. Any chance you'd want to grab coffee and talk about the program? I'd love to hear about your experiences and the project you presented.

<comment><AUTHOR> Hey Bob I'm local. Not going through the grad program (undergrad) but would still be down to grab a coffee and chat. Shoot me an email: whoback [at] g.harvard.edu

</comment></comment></comment></comment><comment><USER4> University of London's BSc in CS should be relatively reputable - [https://london.ac.uk/courses/computer- science](https://london.ac.uk/courses/computer-science)

</comment><comment><USER3> I would start with these: [https://thebestschools.org/rankings/best-online- bachelors-co...](https://thebestschools.org/rankings/best-online-bachelors- computer-science/)

Scroll past the ads that mention the "schools" you mentioned, and take a look at the list that starts with the University of Florida's program at number one.

I'm not personally familiar with the schools on the list above; however, I am currently going through Harvard's extension school for a graduate degree in software engineering. This has worked out well for me thus far. They also offer an undergraduate degree. You can find information about that one here [https://www.extension.harvard.edu/academics/bachelor- liberal...](https://www.extension.harvard.edu/academics/bachelor-liberal-arts- degree/degree-requirements)

</comment><comment><USER5> Oregon State has one, geared toward a second bachelor's:

[https://ecampus.oregonstate.edu/online- degrees/undergraduate...](https://ecampus.oregonstate.edu/online- degrees/undergraduate/computer-science/)

</comment><comment><USER6> The University of Missouri Rolla is a highly regarded engineering school. They have a distance graduate program:

[https://dce.mst.edu/credit/degrees/computerscience/index.htm...](https://dce.mst.edu/credit/degrees/computerscience/index.html)

For what it's worth their graduate engineering programs are rated #22 on US News:

[https://www.usnews.com/education/online- education/missouri-u...](https://www.usnews.com/education/online- education/missouri-university-of-science--technology-OENG0076/engineering)

</comment><comment><USER7> WGU has been recommended a few times on HN. Worth checking out.

<comment><USER2> I've known a few people that attended WGU.

Like all online options, you need to be strongly self-motivated to succeed. Their education model is different from most universities that you will encounter. A class more or less takes as long as it needs. Once you have demonstrated that you can satisfy the requirements for passing, you pass and move on. If you have a lot of experience in a certain area, this can be a wonderful way to accumulate credits.

If you are looking for the "college experience", you'd probably want to look elsewhere, but if you have a need for a degree for career advancement or grad school, it appears to be very worthwhile. Keep in mind that it is a real public university accredited by the same board as Washington and Oregon. It's not a joke by any means.

</comment></comment><comment><USER8> Graduated from Oregon state CS online not too long ago, not the best but not a scam either.

</comment><comment><USER9> [https://csc.uis.edu/online-undergraduate- admission](https://csc.uis.edu/online-undergraduate-admission) I think the actual degree doesn't have an online designation

</comment><comment><USER10> >any decent online bachelors programs for fields like CS

Are you looking for a degree or a job?

If looking for a job, take a look at: [https://lambdaschool.com/](https://lambdaschool.com/)

</comment><comment><USER11> Harvard Extension. If you keep over a 3.5 you have the option to take the same classes as Harvard College Students for your entire time at Harvard.</comment>


Apply HN: DevJoy-Developer Sourcing Tool with Focus on Minimising Recruiter Spam - <AUTHOR> DevJoy scans the web, and maintains a database of ~3 million developer profiles. It provides powerful tools to developers to minimise recruiter spam, and to recruiters to search and filter the ideal candidates for their company.
(Github, Bitbucket, Gitlab, Stackoverflow, Twitter, LinkedIn)
For developers (devs)
===
We allow devs to see and update their profile.
We are annoyed by recruiter spam ourselves, and are building tools to give devs the power to minimise recruiter spam:
Spam Lists:
- DevJoy maintains a global spam list of recruiting agencies
- Each dev can add to / update the spam list locally for his/her profile
- A recruiter who is on the spam lists (global + local profile) would not see the dev's profile in the search results
- We also plan on creating plugins so that the devs can sync their DevJoy spam lists with their email service (gmail, outlook). This will help avoid any recruiter spam from DevJoy as well as from all other sources.
Job Search Status:
- We allow devs to set their status as 'Actively looking' / 'Passively looking' / 'Not looking'.
- Devs whose status is 'Not looking' will not show up in any search results.
===
For recruiters
===
- DevJoy provides a wide range of options to  filter developer profiles, such as:
-- contributions to specific open-source projects on Github / Bitbucket / Gitlab with relevant language / stack
-- devs connected to any of your team members (for referrals)
-- location specific search
-- contributions on specific topics on Stackoverflow
-- packages on NPM, RubyGems and other package managers
- Referral recruiting - you can uncover  if any candidates are connected with your team members on twitter, linkedin
- Tools to collaborate internally with your team: create shortlists, internal notes
===
We are working on the prototype and will have something to show the community soon!
We encourage you to put your criticism and feedback below. <comment><USER2> What is the monetization strategy?</comment>


How to build a startup – without quitting your day job - <AUTHOR> https://medium.com/swlh/how-to-build-a-startup-without-quitting-your-day-job-9f5bbf7a1a2b <comment><AUTHOR> I started building products as a hobby in college. While working full-time as a programmer, I patched together pockets of time to water and trim my fledgling products. It wasn't until five years later that I quit my day job and went all-in.

Back then in my dorm room, I could never have predicted that one of my products, JotForm, would have 3.2 million users and 100 employees — without receiving a single dime of outside funding.

<comment><USER2> Thanks for sharing your story.</comment></comment>


TileDB closes $15M Series A for universal data engine - k-rus https://tiledb.com/blog/tiledb-closes-15m-series-a-for-industry-s-first-universal-data-engine-2020-07-14 <comment><AUTHOR> Congrats on the raise!

Meta comment: The confusion we see in this thread is what happens when a startup tries to create a new category -- a strategy known as "category creation." Founders imagine everyone will jump onboard with this category and run to them as the de facto leader of that category.

In reality, it just creates another point of confusion. Whereas before you had to explain just one thing, now you have to explain two things: What your product does, _and_ what the category means.

There are many existing subcategories within the "database" category of products. Pick the subcategory where you want to compete, and market your innovation to stand out and win over new or existing customers in that category. Timeseries, in-memory, data lake, RDBS, ... Lots to choose from.

This goes for TileDB and any startup founder reading this who's about to launch something like a "Intra-Terrestrial Data Pipeline Miracle" or "Middle- Out Machine Learning Capacitor" or "The First Cloud Fog Edge Dew Platform" or whatever.

</comment><comment><USER2> It's difficult to tell from the landing page, but what exactly is this? It says a lot of things that it's not, but that leaves it sounding like it's an object store with a tightly coupled map reduce framework allowing the "pluggable computer layer".

How wrong is this assessment?

<comment><USER3> Seth from TileDB here.

The foundational invention is the TileDB universal storage engine based on dense and sparse multi-dimensional arrays. Genomic variants are 2D sparse arrays. Images and video are dense arrays. KV are interestingly sparse arrays as well[1]. TileDB is the first solution that can efficiently model all data with a single powerful data structure. TileDB universal storage engine natively supports cloud object stores, as well as local filesystems and we natively handle the eventual consistency through our MVCC design.

On top of the TileDB storage engine we have built and worked on integrations with a range of existing computation tools and frameworks. Our goal is to make the computation pluggable and provide fast and efficient (zero-copy or apache arrow where possible) integrations to allow you to continue to use your favorite computation tools in your favorite language. A few examples are our integrations with numpy/Pandas, Spark, MariaDB, gdal and more.

In addition, we built TileDB Cloud on our storage engine, which has two more innovations: (1) easy data sharing and logging at global scale (beyond organizations), and (2) a complete serverless infrastructure for scaling out compute, very similar to the DAG approach of dask.delayed.

[1] [https://docs.tiledb.com/main/handling-key-value- stores](https://docs.tiledb.com/main/handling-key-value-stores)

[2] [https://docs.tiledb.com/cloud/client-api/task- graphs](https://docs.tiledb.com/cloud/client-api/task-graphs)

<comment><USER4> Seth, congrats on the funding.

> The foundational invention is the TileDB universal storage engine based on > dense and sparse multi-dimensional arrays.

Want some advice? Find a way to explain TileDB not to sound super intelligent, but to help readers understand. You will be much more successful if you do just that.

Added bonus: start with what problem TileDB solves, that current solutions don't solve.

Edit: 1) scalability for complex data; and 2) deployment; seem to be the problems you solve. Is that correct?

<comment><USER2> I'm not even sure this is the problem. For the target customer, this language is fine (sparse arrays being notoriously difficult).

It's the rest of the language that seems to allude to some sort of innovation/breakthrough but stops short of explaining that.

</comment></comment></comment></comment><comment><USER5> Yet Another Database.... As Shelnutt2 states below, its success will depend on how quickly TileDB can be integrated into other tools.

However, I don't see any benefit in TileDB supporting fast and efficient updates (and duplicates) of time series data. Time series should be immutable, and only in rare occasions require updating.

<comment><USER3> Time series data can vary and I'd agree that most time series is immutable. There are however use cases in which updates can happen. For instance, at my previous job before TileDB Inc, we had a case where 99% of our data was immutable and never updated but a very small amount of data could be updated if there were late arriving parts. In order to get near-realtime data we accepted that sometimes some columns might not be available within the window the datapoint represents. In that case that record might reappear at a later time with the complete and correct values.

Of course there are trade offs, we could have forced a longer waiting period until we were confident the data was finalized. We also could have ignored the updates. In the end we used a system of staging tables for loading the last 24 hours of data before merging out into a more finalized table. This kept the load of updating records in the database down, and still allowed us to achieve our goals.

At the time, several years ago, I was not aware of TileDB else we would have considered it instead of a more traditional database vendor.

</comment></comment><comment><USER6> I've looked at tiledb a few times. I think it would make a lot of sense to use it as a serialization format for legion. [https://legion.stanford.edu/](https://legion.stanford.edu/)

Its been on my todo list to try it out for a while. Maybe my next weekend project.

</comment><comment><USER7> I'm looking at some of the comments and still having a little bit of trouble understanding what makes the data engine "universal."

I see reference to "universal storage" in some areas, but keep landing on a multi-dimensional array structure for data storage - and this seems kind of at odds. Maybe I'm missing something, but isn't specifying the structure of the data inherently not universal?

I'm relatively shielded in my databases knowledge, though - having only worked with "traditional" tools. If I'm missing something definitely let me know!

</comment><comment><USER8> Seeing the number of customer testimonials on their announcement makes me once again wonder how brand-new products get traction with customers (who really serve as guinea pigs). I'd love to hear (and learn) how startups have successfully gotten their foot in the door with technologies like this.

For context, I spent a bit shy of a year a while back developing a middleware idea, and severely struggled to get anyone to try it. Friends suggested open- sourcing it, but even that would have been a struggle, I'm sure.

<comment> k-rus I believe TileDB had some customers when it was framed as the product. According to its website: "TileDB, Inc. is a data management company spun out of Intel Labs and MIT"

</comment></comment><comment><USER9> I couldn't find a nice example of using TileDB together with data frames. From the documentation, I understand this is one of the uses of TileDB. Is someone aware of a performance oriented blog post about TileDB and data frames?

</comment><comment><USER10> If anyone from TileDB is reading this thread, the link for "Geospatial" under Applications in the main page's footer points to the wrong link.

<comment><USER3> Seth from TileDB here. Thanks for reporting this, we've fixed the incorrect link.

</comment></comment><comment><USER11> Is this solving "where are my HDFS files" ?

<comment><USER12> Can you elaborate? I can't figure out what you're referring to with just that, but I wonder whether it's a silly nickname for a common limitation of databases?

<comment><USER13> HDFS is the Hadoop distributed file system, but maybe they meant HDF5 files, which is a format the Pandas library saves in. That makes use of Numpy's n-dimensional arrays, so it's related to what these guys are working on.

<comment><USER11> Yeah my typo - I was trying to understand with limited time budget TileDB and their marketing pitch - they mentioned something like "the problem of knowing where to keep data was solved by traditional databases, now its solved by us"

On a dumb level I was just wondering if they are tracking the data sets people create - it does seem more like a data policy than a product. But I may not understsnd it well

<comment><USER14> TileDB Embedded is a storage engine like HDF5, with the following differentiators: (1) it is cloud-native, (2) it supports also sparse arrays, (3) it offers rapid updates, (4) it supports data versioning and time traveling built into its format. TileDB Cloud (our cloud SaaS solution) further allows you to see which arrays you own in the cloud and which ones you share with others, along with full access logs. You can also attach arbitrary descriptions and metadata that can search on, even find and access public datasets posted by you or others.

</comment></comment></comment></comment></comment><comment><USER15> Seems similar to pilosa... What would the differences between these two be conceptually?</comment>


SymphonyOS – Light, Friendly, and Useful - <AUTHOR> http://www.symphonyos.com/ <comment><USER2> I'm downloading it to try in a VM, but this website has almost 0 information on it. Where are the screenshots? FAQ? Where is any information about what this is or why I should use it?

The amount of time it would take to add this info must be a pittance compared to the dev hours they put into the OS!

<comment><USER3> There's a little more in the wiki entry for Mezzo: [https://en.wikipedia.org/wiki/Mezzo_%28desktop_environment%2...](https://en.wikipedia.org/wiki/Mezzo_%28desktop_environment%29)

Looks like it challenges elements of the desktop metaphor. The white paper (from 2005) on the ideas behind Mezzo explains a bit more: ftp://ftp.netlabs.org/pub/voyager/docs/MezzoGreypaper.pdf

<comment><USER4> I remember Mezzo from that time. I found the ideas interesting but I couldn't understand why they couldn't plug them into an existing WM via plug-ins.

</comment></comment></comment><comment> david-given tl;dr: it's a Debian derivative. (I could be wrong. The wikipedia page is very mangled, and the website is entirely content free.)

It looks like it mainly exists as a platform for Mezzo, a desktop environment. I skimmed the design document; it's painfully opinionated, and keeps repeating the tired old cliché that users can't handle configuration options and we should make their lives easier but not letting them change anything...

<comment><USER5> The design document is 11 years old. Symphony has been a solo side project for the last 6-7 years with a release once or twice a year. There are a few more recent updates on the project's facebook page including screenshots of the new build in progress. [https://www.facebook.com/SymphonyOS/](https://www.facebook.com/SymphonyOS/)

<comment> david-given Are there any videos of the new Mezzo in action?

It occurs to me that my post was unduly dismissive. I totally believe that most modern UIs are horribly broken, and I completely sympathise with the 'burn it all down!' approach. But forcing everybody to change their workflow because reasons is, I believe, not it. (Though I'm willing to be persuaded otherwise.)

<comment><USER5> Many of the concepts we built into Symphony have since become common in major desktop and mobile environments such as fullscreen menus and hot corners. The project started in 2004 when I as a web developer wanted to start playing in the Desktop space and decided to give it a shot with the tools I knew. These days it's a personal side project that I wish I had more time for. I do not have any videos of the new build in action, just the screenshots on the project's facebook page: [https://www.facebook.com/SymphonyOS/](https://www.facebook.com/SymphonyOS/) I'll see what I can do about that this weekend when I am back into it. The new build is coming together but I am not trying to convince anyone that this is what they should use. It's a fun personal hobby at least until more developers get involved.

<comment> david-given Mmm.

I loathe hot corners; every day or so I accidentally lock the screen on my work macbook because when I fling the mouse pointer out of the way of what I'm looking at it hits a corner.

I also loathe full-screen menus. Screens are big, and having the entire contents of my screen be replaced with an information-heavy overlay is a non- trivial perceptual context switch which instantly causes me to lose my place in what I was doing. You underestimate how cheap a right-click menu selection can be; press, drag a small distance down, let go. And if the app's designed properly you can right click _anywhere_.

(Which, to be honest, most apps aren't. I grew up on RISC OS, which was driven entirely by context menus, and did this stuff right --- each app was a single context target. Modern apps have different contexts for every tiny UI element. Web browsers are particularly bad; I can't count the number of time I've done 'Open in new tab' rather than 'Back' because the mouse just happened to be on a link rather than a text area.)

My point is: one size doesn't fit all.

<comment><USER6> You know you can change the function of the hot corner (or simply disable it), right?

(In case you don't know how: go to System Preferences → Desktop & Screen Saver → Screen Saver → Hot Corners…)

</comment></comment></comment></comment></comment></comment><comment><USER7> Wow, I remember hearing about this project over a decade ago, and liking some of the UI concepts. I followed the progress on the website for a little while, and then it seemed to stall, so I mostly forgot about it. As others have mentioned here, the website is unclear. What's going on with this? Is it progressing again?

<comment><USER5> I am the sole developer/maintainer of SymphonyOS these days and it showing up on HN today is a bit of a suprise since it's been several months since a public release.

Development is progressing albeit slowly since my day job takes up most of my time. I am currently in the middle of a rewrite of the desktop environment making changes like moving from FVWM to openbox and rewriting some Perl code that has been around in one form or another since 2004 in Ruby. I am hoping to have a new release out in January :)

</comment></comment><comment><USER8> I was very interested in this back in the day, in fact, I wrote a HOWTO to get the desktop bit run under Ubuntu: [https://fak3r.com/2006/01/31/howto-mezzo- desktop-on-ubuntu/](https://fak3r.com/2006/01/31/howto-mezzo-desktop-on- ubuntu/)

I'm sure that's not going to work anymore, but I got a lot of questions and comments about that post, there was significant interest. Here's hoping a new release comes out in January and people can try it out.

</comment><comment><USER9> I fail to see how a desktop UI based on HTML5 can be marketed as "light".

<comment><USER10> Absolutely. If anything, it should be tight, native code with little bells and whistles and tuned for speed. HTML5, its DOM model and Javascript are hardly fit for embedded / old / constrained environments.

</comment></comment><comment><USER11> It's a little light on details.

Is this related to the symphony framework in any way, or is it just a naming coincidence?

<comment><USER12> Name collision I believe, there's a russian language demo on youtube. It seems to me as a tiny fltk based WM to fit the current UX trends without having a "heavy" DE like Gnome3.

</comment></comment><comment><USER13> I work with the maintainer of Symphony and it is still "alive".

<comment><USER14> Perhaps you could find a volunteer with a little time to just add a list of packages and functions available from the live image to the Web site?

That would help the distro collectors to decide if this is interesting or not.

<comment><USER5> Distrowatch has a partial list here: [http://distrowatch.com/table.php?distribution=symphony](http://distrowatch.com/table.php?distribution=symphony)

Symphony showing up on the front page of HN was a total surprise to me today. As a sole developer my focus right now is on the new build and next month's release. I agree that the site is way to sparse and could be much better (it was thrown together in one evening) but focusing on the site would only take away from the limited time I have available to work on the distro itself. If after the January release I am able to get a few more developers involved I would love to improve the site, documentation and community around the project.

<comment><USER14> Get a volunteer on it. Someone who wants visibility and to develop some documentation skills. You stay with the main focus. Your HN exposure has convinced me to try the iso!!

<comment><USER5> Absolutely. Right now I am heads down to get the new rewrite released in January. Then the focus will be back on building the community, bringing on some more volunteer developers and putting together good documentation.

</comment></comment></comment></comment></comment><comment><USER15> Any torrents we can use? I'd rather not get it through a huge HTTP download.

<comment><USER16> use Burnbit.com

</comment></comment><comment><USER17> There is remarkably little information on the website. I don't really want to download it without at least some indication that it's going to be worth my time. Is it just a modified fvwm running in Debian?

<comment><USER5> The currently available release is Debian based utilizing FVWM along with tint2 and a custom launcher written using HTML5 and WebKit. It's worth trying out in a VM or as a LiveDVD. I am currently in the process of doing a complete rewrite of most of the Desktop environment, replacing some very old Perl pieces with shiny new Ruby ones.

</comment></comment><comment><USER18> If it's as friendly as the website, there is still plenty to do there.

<comment><USER5> Unfortunately SymphonyOS is a one-person project right now and has been for a long time. My focus has been on the codebase and the website which I will admit is crap was put together in one evening. Currently I am in the middle of a complete rewrite of the desktop environment with a new release coming in January. Personally, I would not have submitted it to HN with the website in it's current state but was surprised this morning when a co-worker told me it was on the front page

</comment></comment><comment><USER19> Please not another Linux distro.

<comment><USER20> Which distro, in your opinion, is 100% perfect and can't be improved on?

<comment><USER10> FreeBSD :)

<comment><USER21> [https://www.dragonflybsd.org/](https://www.dragonflybsd.org/) those folks would disagree.

<comment><USER10> If it must be Linux, there's always [https://crux.nu](https://crux.nu)

</comment></comment></comment></comment></comment><comment><USER22> Is this based on one of the popular linux distributions? If yes, which?

<comment><USER14> Currently Debian (Wheezy according to author of distro in this discussion) in future Ubuntu

</comment></comment><comment><USER23> Yay! Another F*cking linux distro to learn about! Linux distros may not be as common as JS frameworks, but they're almost there. Just install Arch and have done with it, people. It's like having your own distro, but without all the annoying bits.

<comment><USER14> Quote from Wikipedia page about the OS

 _" At first, Symphony OS was based on Knoppix. Since its May 2006 release it is no longer based on Knoppix, but rather on Debian unstable, and features a functional hard drive installer."_

Debian Sid is viable as the basis for desktop use, but there can be _issues_ now and again and they can persist for a week or two. Just wondering at the target market for this one.

Always fun to do a dist-upgrade on anything Debian derived to see what happens...

<comment><USER5> The current release available on the website is based on Debian Wheezy and was released in January. A new release, with a completely re-written desktop, based on Ubuntu will be coming out next month.

The wikipedia page is out of date in many places

<comment><USER14> Thanks for clarification. Wheezy is mega-stable at present. Good luck with the transition to Ubuntu!! Quite a job.</comment></comment></comment></comment>


Self-learning flight search creates itineraries using only one-ways - <AUTHOR> http://questorganizer.com/

<comment><AUTHOR> If anyone would like me to use some advanced searching to find them an epic flight, let me know your orign, destination, and travel dates.

-Doug (co-founder)</comment>


Robin Hood Hashing – Programming.Guide - <AUTHOR> https://programming.guide/robin-hood-hashing.html <comment><AUTHOR> Just published this article! Happy to hear your thoughts, especially if you have ideas for improvements.</comment>


Rare Beasts, Birds, and the Calaboose: A Brief History of London's Tower Menagerie - <AUTHOR> http://www.theparisreview.org/blog/2016/09/22/rare-beasts-birds-calaboose/ <comment><USER2> Thanks for the post! It's always fun to see a few of these ( not exactly tech news) articles to help round out our worlds.</comment>


Freakonomics Author, Stephen Dubner on Bug Bounties and Rat Farming - <AUTHOR> http://threatpost.com/en_us/blogs/how-bug-bounties-are-rat-farming-092011

<comment><USER2> The guy usually makes sense, but he doesn't here.

You can't breed bugs in programs. You can't earn more money by creating more bugs in your browser. You simply don't have access.

<comment><USER3> He wasnt making that point. He was making the point that rat farming was bad but software bug farming was good.

<comment><USER2> Why bring it up at all, then? It didn't serve to illuminate anything. He just brings up a bad analogy, then ... Shoots down his own analogy? Why bother?

</comment></comment></comment><comment><USER4> That's a remarkably silly analogy. Researchers do not introduce vulnerabilities themselves, they are not "farming bugs" to game the system.

<comment><USER5> Yup. I clicked through, expecting to read an article with some evidence (however flimsy) that developers were intentionally introducing bugs in their software so that their partner could report the bug and claim a bounty. What I actually read was just weak.

The common term for this is the Cobra Effect <http://en.wikipedia.org/wiki/Cobra_effect> but a quick google doesn't bring up any leads on the veracity of his South Africa / rat-farming variant.

EDIT: more googling doesn't find any source for this other than Dubner himself. Looks like he might have just made it up.

</comment></comment><comment><USER6> Either he's explained something really badly or someone has this exactly 180 degrees wrong.</comment>


Two Women Fell Sick from the Coronavirus. One Survived - <AUTHOR> https://www.nytimes.com/interactive/2020/03/13/world/asia/coronavirus-death-life.html <comment><USER2> This is a tough essay to read but beautifully written. I didn't notice how the background was changing as I read it till I got to the end. The fade to black hit me like a ton of bricks.</comment>


Ask HN: Does anyone dislike public code review of coworkers' code on GitHub? - <AUTHOR> I don't believe that I was uniquely taught that when critiquing someone it is best to do it privately.
DownSides: a) Potentially embarrasses the person b) Can create animosity towards co-workers c) Difficult for introverts regarding reviewing & critiquing
Alternative: Private code reviews
Any suggestions? <comment><USER2> A few thoughts:

1) you are not your code. When you're doing a code review, you're critiquing the code, not the person.

2) critique is not the same thing as criticism. Yes, I was taught to criticize privately, but critique is frequently public (within a small group). Artists have critiqued publicly for centuries. The reason is that it's not only the creator who can learn from the critique.

3) critique doesn't have to come across as harsh. Instead of "you should have done Foo using Bar instead of Baz", try "why did you decide to use Bar instead of Baz for Foo?". Instead of "this code is badly written", try "do you think this might be simpler/clearer if you did it this way instead?" Personally, I've learned a ton by asking questions like these and paying attention to the answers.

It's possible (preferable, IMO) to critique with empathy. The nice thing about code review is that it's an asynchronous medium where you can think through how you phrase things, which can be good practice for learning to interact empathetically in other potentially confrontational contexts.

<comment><USER3> > Personally, I've learned a ton by asking questions like these and paying > attention to the answers.

Yes good code reviews can be a learning experience for both reviewer and reviewee.

At work I lurk on a number of PRs that I'm directly involved with just so I can learn from the comments.

Maybe I'm lucky to be part of a team with a healthy code review culture.

</comment></comment><comment><USER4> I feel like if you're embarrassing other people in code reviews, you have a culture problem that probably won't be resolved by doing private code reviews.

</comment><comment><USER5> If your code review makes the coder feel personally attacked or embarrassed, you're not doing something right. Code reviews are meant for 2 (or more) way communications and discussions in order to hold quality of the code up to a standard, not a opportunity for the reviewer to feel superior and attack others. Commenting things like "this line is poorly written" is not helpful to anyone, instead try "do you x would be a better solution? because <insert reason>".

I'm sorry but if you're a functioning adult working in a professional environment, you should expect to receive critiques and feedback on your work. If someone is so sensitive that they cannot accept constructive criticism on their work, I don't know how they're ever going to learn or improve.

</comment><comment><USER6> Just for clarification, when you say public do you mean open to everyone (the internet) to make a code review or open for everyone in the team to see and comment on.

Personally I find the first idea fairly bad since the amount of constructive criticisms you might get from people far outside of the problem realm seems limited. Having them open to the entire team on the other hand seems like a good way to spread knowledge and to get input from different sources

</comment><comment><USER7> You lose the ability to bring in additional co-workers into the discussion for important decisions by making it private.

</comment><comment><USER8> You basically want to take a Postel's law type position, where you're quick to accept others' feedback and make the changes they request, while being conservative in the feedback and requests for changes you emit. Keep in mind that you're trying to get stuff done, not win debates with developers, who can be unbelievably persnickety and shortsighted.

If a reviewer asks you to change something and you believe your version is just as good as what they suggest, change it. If they suggest something they claim is "totally optional", do it. Remember, we're getting stuff done. This is faster than arguing (over stupid shit that isn't going to matter anyway).

On the other side, if you ask for something in a review and they're not gonna do it, bias toward letting that slide unless it's a real problem. If you think a piece of feedback will seem embarrassing, mention it in a private chat or something first so they have an opportunity to address it.

Code-style issues should not come up in reviews. Set up a linter, fail builds on lint errors, and forget about it.

Maybe it is difficult for introverts, but you know, a lot of things are going to be difficult. Sorry about it.</comment>


Bullet-proof Arch Linux install process - <AUTHOR> https://wiki.archlinux.org/index.php/User:Altercation/Bullet_Proof_Arch_Install <comment><USER2> As a developer that always tries to make software that "just works" for the users, I always wondered if the Arch devs are masochists or sadists.

<comment><USER3> I use Ubuntu instead of Arch. When I run into a problem or want to configure something, I wind up on Arch Wiki. I've considered using Arch because using Ubuntu always makes me hope that there is a simple GUI solution and waste time looking for it instead of just acknowledging that Linux is Linux and the efficient way to fix problems looks like Arch.

Still haven't switched because I don't know enough Linux. But at some point I might.

<comment><USER4> I feel like waiting until you know enough to switch is like the developer's Zeno's Paradox.

If you want to switch to Arch, find an afternoon and switch. The wiki has a good starters guide. After you've run through it once wipe it and run through it again with everything you learned the first time through.

<comment><USER3> Alas, the concept that Ubuntu isn't a free lunch cuts both ways. I'm using the Ubuntu flavor of Arch, so to speak. I even use the wipe and reload technique.

One of the reasons I settled on Ubuntu is AskUbuntu. It has a different standard toward questions, more like HN or StackOverflow, than the Arch community has traditionally been known for.</comment></comment></comment></comment>


Nearly 1/3 of tech workers will delete Facebook accounts amid scandal, study - <AUTHOR> https://www.pcauthority.com.au/news/nearly-13-of-tech-workers-will-delete-facebook-accounts-amid-scandal-study-487886 <comment><USER2> "Nearly 1/3 of meth heads will stop smoking rocks after finding out meth is bad for you."

See how crazy that sounds?

</comment><comment><USER3> I wouldn't be surprised if a lot of tech workers never had a Facebook account in the first place.

<comment><AUTHOR> Like Steve Jobs didn't give iPhones to his kids.</comment></comment>


What are the Options for Storing Hierarchical Data in a Relational Database? - <AUTHOR> http://stackoverflow.com/questions/4048151/what-are-the-options-for-storing-hierarchical-data-in-a-relational-database <comment><USER2> Sometimes you might not need 'relational' in the SQL sense of the word, you might find the way a 3D renderer, e.g. in a games engine, has a better way of representing hierarchical data. Even if you are not displaying the data you can query it and do things with it on a GPU.</comment>


Show HN: The Codex – a graph database project for the digital humanities - <AUTHOR> http://the-codex.net <comment><AUTHOR> In 2015 I quit my full-time job as an ASP.NET developer to build what I think of as an "atlas of history" for the Italian Renaissance. It is a semantic-web style database build in Neo4j, .NET MVC, and KnockoutJS, and is an attempt to build a map of historical events and personalities for the digital humanities.

[http://the-codex.net](http://the-codex.net)

I am currently the sole developer, product designer, and researcher on the project -- but I am looking for collaborators who would be willing to help me take this further.

As an "atlas of history" is a broad concept I decided to give the project clarity by focusing primary source documents from the Italian Renaissance. The two main sources at present are the 'Florentine Diary' of Luca Landucci and the letters of Michelangelo. I have entered about 40 years worth of entries from Landucci's diary and a good portion of Michelangelo's early letters from his Roman period. In the process I have added hundreds of historical personalities, places, artworks, etc., in order to give the user real data to work with. I have also built various screens with data visualisation tools to mine the historical events. And of course I have built an extensive back-end for managing the data and relationships.

Is anyone interested in helping me out? I'd love input from anyone with an interest in art history and graphic design, or data visualisation tool, Neo4j, or anyone who wants to help me research and enter data.

Feel free to email me any time at: firstname.lastname@example.com

In the meantime, why not check out the Control Panel on Leonardo da Vinci's dataset. Clicking any links in the text will load the datasets for those entities; or you can search for them by name. Why not try adding Michelangelo's dataset to the mix? You can then switch between the three data- vis modes at the bottom, fiddle with the date filters, etc.

[http://the-codex.net/Time/ControlPanel](http://the- codex.net/Time/ControlPanel)

Many thanks, Iian

<comment><USER2> I appreciate what you've built and I'm interested in helping out. What is the way going forward for the project? Do you plan on monetizing it? Is code eventually gonna be public?

<comment><AUTHOR> Hi,

Thanks for your interest! I have no plans for monetizing the project, partly because I cannot see a meaningful way to do that, but mostly because it is a research endeavour. The code is currently public on my BitBucket repo, and I'll shoot through the URL as soon as I can.

Basically, I want to explore the limits of how graph databases can be applied to the visualisation of history. My immediate goals are to:

(A) expand the range of datasets that are collected; (B) improve the tools for entering and annotating the data to make the process quicker and more automated; (C) to add more powerful data visualisations; and (D) to complete the input of the primary source datasets mentioned and add more.

I want to build a graph database time machine of the Italian Renaissance so you can pick a day in history and see what was happening across Italy with various sources, see events plotted or even animated on a map (e.g., watch a battle unfold or even weather events), etc. The system is underpinned by subject tags which are taxonomically organised in a 'is-a' hierarchy, which I hope will provide a 'semantic zoom in/out' functionality.

I understand that the current platform of C#, ASP.NET MVC, KnockoutJS may be a turn off ... And I am open to other technologies if that is crucial. But conversion to another platform would certainly take a lot of time and manpower. But to be honest I think the hard work is mainly in the product design, the visualisations, the data-entry, and the Neo4j Cypher queries.

What kind of collaboration did you have in mind?

<comment><USER2> I could whip up an alternative frontend implementation if you'd like, I've been looking for a project where to flex my ClojureScript/Om Next (based on React) muscles. I'm very experienced both with Clojure and with the frontend, and with interface design. I actually worked with KnockoutJS (in a C# shop I used to work at) many years ago, but yeah I don't have fond memories of MVC/MVVM for large projects/efforts, the React model speaks more to me.

<comment><AUTHOR> ClojureScript and React sound interesting to me, too, and might be a good fit for the rich UI Control Panel page, which will need to handle large JS data- sets and update visualisations quickly. I've been finding that KnockoutJS is a bit sluggish when wiring up the filters as computed function observables. It could be something in my filter code, also, but I have seen benchmarks where KnockoutJS doesn't perform well in.

We should talk further ... drop me a line at my email? email@example.com

</comment></comment></comment></comment></comment><comment><USER3> Hi Iian, great work!

I built recruitment CRM with cached 1,000,000+ nodes and can imagine how much time and efforts you invested in your project.

I assume that .NET MVC and KnockoutJS was your first choice as you get used to it ... and it's fine.

I personally found a lot of frustration dealing with Neo4J full-text search in multiple languages English, Russian and Japanese. There was no clear guideline on Lucene integration at the time.

How is your experience with Neo4J? Do you have to sync Neo4J data with traditional databases like M$ SQL?

Do you want to make your project as community efforts or see it as potential business?

The last question is much more important. I assume that a lot of developers won't be excited about .NET or KnockoutJS staff, however they might consider to share some crawling data, D3.js graph visualizations code, etc with you.

<comment><AUTHOR> Hi Ilya,

Thank you for your kind words and your excellent feedback re: the chosen platform and the OSS community. To be honest I hadn't thought about the language and platform angle until you mentioned it, but it is definitely something I may need to consider. C# is a mature language comparable to most other quasi-imperative/quasi-functional ones so conversion I don't think would be hard so much as time consuming. The brain of the system is ultimately in the data structures, the Neo4j Cypher queries, and the product design/architecture.

Can I ask what technologies you think would be most likely to attract community involvement, but wouldn't sacrifice static typing? Perhaps some Java framework for the back end, Angular or React for the clientside framework? I would not be able to immediately convert the project to those technologies, though, given the code count, but something to keep in mind ...

Many thanks, Iian

</comment></comment><comment><AUTHOR> I also recorded a screenshare presentation of how The Codex works:

[https://www.youtube.com/watch?v=_R0ESfLBuHo](https://www.youtube.com/watch?v=_R0ESfLBuHo)

</comment><comment><USER4> Hey, Iian. Not sure if this will be useful to you, but you might consider checking out [http://endlessorigins.com/](http://endlessorigins.com/) (the largest structured collection of human events, available for download as a single TSV file). And good luck with your project! :)

<comment><AUTHOR> Hi Tuvalle, thank you for your encouragement and for the link to this fascinating resource! Can't wait to open up the dataset and check it out soon. May be possible to import this into the Codex if the data structures are broadly compatible ... :-)</comment></comment>


NYT paywall on April 4 - <AUTHOR> http://scripting.com/stories/2011/04/04/nytPaywallOnApril4.html <comment><USER2> I'm running chrome in incognito mode without being logged in, and oddly enough I'm neither being prompted to login, nor subjected (so far) to any article limitations.

I'm not sure whether to feel guilty or not. I _am_ finding myself using my WSJ subscription (which I do pay for) a whole lot more from my iPad as I run into article limitations on the NYT there.

<comment><USER3> Yeah, according to my dad, apparently clearing your cookies works just as well (which incognito mode does whenever you close the window). They should have used that perma-cookie JS library that uses like 10 different methods of fingerprinting a computer.

<comment><USER4> "They should have used that perma-cookie JS library that uses like 10 different methods of fingerprinting a computer."

Yeah, but give them a break, they only had $40M to build the paywall with so they made do with what they had.

</comment></comment></comment><comment><USER5> I think the most interesting piece of the paywall is that you are connected with the actual value you find from the site. If you never hit the paywall, it's not worth much to you. If you hit it within a week, it lets you know that you care about it much more than you thought.

I found myself thinking, "do I really care about this enough to take a hit on the paywall?" today. Perhaps that means I'm more interested in the journalism than I thought.

</comment><comment><USER6> Think the bigger question here is, how long till you (we) end up paying to bypass the wall.

If the Times does have content worth paying for, then perhaps we'll stop freeloading and pay for it, right? :)

<comment><USER7> Or if they adopt a payment model that makes sense and readers will buy into. Reminds me of a recent HN submission - <http://news.ycombinator.com/item?id=2388064></comment></comment>