Ask HN: Is anyone using Watson in their business? - <AUTHOR> Analytics, or language, or anything Watson <comment><USER2> What exactly is Watson except a marketing term? I mean, as far as I understood it, it's Apache UIMA with some bare bones NLP bolted on, using RDF ontologies to power some inference. But I've never seen it being used in the wild except as a tool to win over people to the IBM-side, raking in large contracts ultimately doomed to fail due to lack of domain knowledge. It all seems so ... familiar. Now everyone is rushing into Deep Learning for NLP, but apart from things like sentiment analysis on synthetic data sets ... it's still /hard/ and needs constant attention. I really don't see how you can "IBM Watson" stuff unless you're already heavily invested on integrating NLP/ML into your workflow ... and that's tricker than it looks. The stuff on BlueMix seems nice and all, but far from a silver bullet or golden hammer when it gets down to the nitty gritty of real world data.

</comment><comment><USER3> "IBM Watson" is a group of consultants and a brand. It's not an algorithm or a service or an "artificial intelligence". Don't fall for the marketing.

[https://news.ycombinator.com/item?id=14766793](https://news.ycombinator.com/item?id=14766793)

[https://news.ycombinator.com/item?id=14682816](https://news.ycombinator.com/item?id=14682816)

[https://news.ycombinator.com/item?id=14767407](https://news.ycombinator.com/item?id=14767407)

<comment><USER4> [https://console.bluemix.net/catalog/?category=watson](https://console.bluemix.net/catalog/?category=watson)

<comment><USER5> The real question may be, "How many IBM-managed software consultant projects brand their software with the Watson label AND actually use Watson services as described here: [https://www.ibm.com/watson/products- services/?"](https://www.ibm.com/watson/products-services/?") One thing is for sure: Watson branding != Watson services.

</comment></comment></comment><comment><USER6> I'd give some insights, but I don't want the stock price to go down before I liquidate. (Or get sued) Hoping Q3 earnings are good b/c of mainframe sales, fingers crossed!

I was hired to work on a 8+ digit contract to apply Watson, office mate and I (only engineers on the team) were laid off 2 months into our new jobs. Do what that company did, put cancellation clauses in any contract should IBM not deliver.

</comment><comment><USER7> Yes. We use the Natural Language Understanding service at www.bustle.com for doing categorization of content. It maps decently to IAB categories: [https://www.iab.com/guidelines/iab-quality-assurance- guideli...](https://www.iab.com/guidelines/iab-quality-assurance-guidelines- qag-taxonomy/)

<comment><USER8> Thanks for giving an actual example. Looks like it's a tiered model, first classing into broad categories and then the specific labels? What was integration like: fairly straightforward or was there a long setup/tuning process?

<comment><USER7> Very straightforward. We send text, they send back categories. We then map those to IAB categories in our own code

</comment></comment></comment><comment><USER9> I was loosely involved with a project at a very large company where they were using Watson (all on-premise, no cloud) to classify customer feedback (verbatims) into an existing taxonomy. This was actual customer feedback that came through service channels, not social media chatter.

The taxonomy basically determined the escalation process within the company. As an example, think about issues with a cell phone. "Sometimes my phone volume goes down automatically" isn't as critical as "Sometimes my phone gets really really hot in my pocket."

The Good: They felt Watson was doing a good job classifying the customer feedback. Overall, they felt they could rely on Watson and could defend using Watson instead of human raters in a legal situation.

The Bad: The amount of time/resource investment required to get it up and running had been way beyond what they initially expected. They also felt various parts of the system could have been more open (e.g. Watson's back end database) to inter-operate with downstream systems. They also were exploring if other solutions (e.g. Data Scientists + Python) could do this and had started to see positive results.

<comment><USER10> Sounds like the perfect fit for machine learning. Let people continue classifying the input text, store the input text and classification, keep doing this for a while and you get a large training set. Regularly run ML on it and compare results. Switch to ML when its results are good enough.

</comment></comment><comment><USER11> I'm familiar with 2 large public companies that got the sales pitch, bought in and results were underwhelming or failed. From an inside sales person "we are great at selling vision and lackluster at delivering results".

<comment><USER2> Not by any chance in the pharmaceutical industry? Because that story seems eerily familiar.

<comment><USER12> Has to be the financial industry... because it is also eerily familiar.

</comment></comment></comment><comment><USER13> Watson is probably not for companies whose core competency is software. There's plenty of less glitzy and more functional machine learning tools out there. Have you tried scikit-learn and nltk?

Watson seems to me like it's mostly leadgen to sell IBM consulting services to companies that don't know how to build software in-house.

<comment><USER14> >Watson seems to me like it's mostly leadgen to sell IBM consulting services to companies that don't know how to build software in-house.

This is exactly what it is. It's a marketing product designed to help IBM sell the company overall as a big data company. The only people the commercials are targeted at are C-level execs who don't know the first thing about data science but know they have to invest in it to stay competitive. I don't work for IBM, but I'd wager a lot that the idea for it was created by a sales team and not an engineer.

That said, IBM is certainly not the only business to do this and it's not even clear that it's not a smart business strategy. I'd wager that most enterprise software sales use deception as a major part of sales strategy.

</comment></comment><comment><USER15> Thank you for asking this question.

I work at a large bank and we've seen the Watson sales pitch every year for four years now. Every time they visit we ask them this question.

It would be great to hear from folks using it.

<comment><USER16> What do they say when you ask them that question?

<comment><USER17> Since it's IBM, they probably have duped a few logos into buying a trial subscription as part of another sale of a more traditional product.

Voila, now everybody is "using" it

where "using" = sitting dormant.

</comment></comment></comment><comment><USER18> I worked with a large company that was one of IBM's 'partners' for watson. They mostly used the partnership for the free publicity, but complained about how terrible all of the disparate watson services were. In the end they ended up replacing most of the services with google ones, but maintained the public partnership.

</comment><comment><USER19> I use it to transcribe voice messages that come in through our PBX. The PBX sends the voicemail mp3 to Watson, it sends back the transcription along with a guess (%) on how accurate it is (then the PBX emails the transcript attempt and voicemail attachment to whoever). It's almost always good enough to know what the message is about, sometimes perfect, often hilarious.

<comment><USER8> Does it return a confidence % for the whole transcription or does it rate each word/phrase?

<comment><USER19> The PBX usually chunks the send, Watson returns a % for each chunk it processes and then the overall average upon completion.

</comment></comment></comment><comment><USER20> With Watson, IBM appear to be trying to stop "nobody ever got fired for buying IBM" from being true any more.

</comment><comment><USER21> I have a friend that works at a famous credit checking company in their data research team, here in Latin America.

He told me that IBM's data service (ie Watson) was not at all impressive. At the time (about a year ago) their service was not worth the pay. He told me all the solutions they provided, and the techniques they used (in terms of algorithms and infrastructure) were very easy to implement themselves (I mean, it was an internal data research team).

But, in our conversation, he told me that there were reasons to go for some AWS dervices (such as integration and provided apis) or Google (the amount of data their stuff in trained on, the infrastructure they have) could be pointed out as reasons to hire AWS or Google. He couldn't find any that suggested hiring Watson.

</comment><comment><USER22> Watson was much hyped for Cancer research and it has remarkably failed, there was a news article a long time back. I remember that everyone in my company was like "Watson Watson" but then the AI craze stopped. Good day are back!

<comment><USER5> "Big Data Bust: MD Anderson-Watson Project Dies: Top Cancer Center Spent $62M" [http://www.medscape.com/viewarticle/876070](http://www.medscape.com/viewarticle/876070)

<comment><USER23> There is lots of blame to go around on why that project failed - IBM only shoulders some of it.

</comment></comment></comment><comment><USER24> Here's an example: Shimmy Technologies is basing their company on Watson API. They make custom swimsuits using Watson's Speech to Text "recognizeMicrophone" so buyers can say their measurements which then gets converted into CAD files and Shimmy produces the customized clothing from there. [http://www.techrepublic.com/article/stop-the-hype-the- real-v...](http://www.techrepublic.com/article/stop-the-hype-the-real-value- of-ibm-watson-is-driving-small-incremental-business-value/)

</comment><comment> martin-adams I saw an advert this morning on Twitter for Watson Content Hub. I checked it out[1]. Honestly it looked underwhelming as the only part that Watson came into play was auto-tagging the contents of an image when uploading. Using the Watson branding for such a narrow aspect of the product seemed more like marketing wishes than actual innovation.

[1] [https://www.ibm.com/products/watson-content- hub](https://www.ibm.com/products/watson-content-hub)

<comment><USER25> I believe Fark uses it to auto-generate story tags when they go green.

I'm not actually sure if this is a compliment, now that I think about it.

</comment></comment><comment><USER26> I had worked with IBM until earlier last year as Watson architect.

Watson is:

1\. A brand - almost everything that has anything to do with analytics, gets labelled Watson. From cloud services to healthcare co-developments, to statistical software (i.e. SPSS) to anything (e.g. an e-commerce suite). 1.1 There also is "Watson Machine Learning" which is nothing more than SPSS in the cloud. (But of course you can use it for doing ML stuff)

2\. A research project, which developed Watson as the system for the Jeopardy gameshow event and is still continuing to develop this system/technology further

3\. There is a Watson core system, nowadays codenamed "Watson Discovery Advisor", which is architecturally pretty close to what was the original research system. There are only a handful of systems built on this architecture. It has nothing to do with the cloud services on bluemix, but is a massive parallel pipline in which pretty decent NLP gets combined with a set of hypothesis evidence scorer (various algorithms, from grammatical reasoning like simple LAT to ontological reasoning, lookups, tuple- and triple searches, indri, fulltext and other search approaches). This architecture is used for example, to power the Watson Cancer product. Every project like this is super expensive... speaking of 20 million +

4\. A set of APIs in IBM's cloud platform ("Bluemix").

4.1 There we have NLC, a semantic text classifier (I personally find it quite good), a chatbot-system which pretty much is NLC with continuous training plus a handwritten dialog tree ("Conversations")

4.2 a machine-learning-ranking based search system ("R&R", "Discovery"), which combines an Elastic Search with a query language, paragraph handling, an NLP pipeline (see 4.3) and a supervised machine learning based ranker. The ranker is used to train the search to find relevant answers for a given natural language question (i.e. long tail question answering)

4.3. A NLP pipeline, which is NOT UIMA based, but based on the AlchemyAPI aquisition (which brought in the public taxonomy) and a SIRE based statistical machine learning relationship extraction approach. There is a training environment called "Watson Knowledge Studio" which lets you either program (dictionaries, rules) or train (click instances on sample texts) the NLP to your type system. This is running on Bluemix as "NLU" or in "Discovery"

4.4 And other stuff like image classifiers (party brought through the AlchemyAPI acquisition, partly from IBM's research in Almaden), Speech 2 Text, Text 2 Speech, Translation

6\. An on-premise software called "Watson Explorer" which is a mixture of the Vivissimo aquisition (a search engine + portal + data integration) and IBM's homegrown UIMA pipeline and a nice analytical interface (data science tool for textual analytics - correlational analysis on text, NLP features and metadata with deviations, timelines, etc). This pipeline can be programmed with a rule- and dictionary based dev environment AND with the supervised machine learning based training environment Watson Knowledge Studio (->5)

7\. Very few "products" like Watson for Oncology or ICPA, which are based on a variation of the above technologies and sold through specific channels

8\. A Watson Health brand, dedicated to applying the Watson ideas to the healthcare market. They "own" Watson for oncology. They also have a big set of custom cloud APIs (some quite good and complex), like body part and drug tagging, adverse event detection, ICD10 coding, etc.

There is a huge sales folk in IBM, who does not understand the complexity and the breadth of portfolio the company has. Very often this results in overselling. The big Watson projects are pitched as a "this is what you get" plus the Watson APIs on the Bluemix cloud are shown as a "this is how easy and cheap you can start"... sales "forgets" to mention, that there is a huge gap between playing around with APIs and developing large systems like Watson for oncology. Or they are pitching the big Watson story and then selling Watson Explorer, just because this is, what they get their sales-quota for.

Also the stuff requires really skilled people to actually forge a customer specific Watson system. I have seen many projects gone bad, just because some product managers think "this is my product and we solve the problem with just this" \- instead you would have needed a few good techies and an skilled architect to develop a solution, leveraging multiple different APIs and a combination of cloud and on-prem (Watson Explorer) to achieve a good result for the customer.

If you know what the technology can do, do not oversell it, and have skilled people working on it, you can built great things. I myself have lead the implementation of a good dozens of pilots and a handful of production projects (which are still active!)

</comment><comment><USER27> I'm not using it in my business yet, but I created an IoT project to test the Watson text to speech API for the Renesas IoT Sandbox. See this. [http://learn.iotcommunity.io/t/using-speech-in-your-iot- proj...](http://learn.iotcommunity.io/t/using-speech-in-your-iot- projects/1368)

It's actually easy to use the API and I'm hoping to use more features for the my next project, which is a mobile app to get voice alerts of distributed IoT data from the cloud. Not sure if this answers the question, but I think the API is usable by businesses.

</comment><comment><USER8> Why does IBM keep separate branding for Watson and Bluemix AI? It seems like most "Watson-in-the-field" deployments are leveraging the Bluemix pre-baked models for specific workflows like document classification, transcription, etc.

Seems like they'd better align the marketing and the offering if Bluemix was the "Watson gateway" so to speak. They could still sell and market Watson as the custom consulting they do, with Bluemix branded as an entry point to basic Watson functions. There's some speak of this in their product copy, but it could probably be more obvious.

</comment><comment><USER28> It is glorified robbery. My experience is that they are smooth talkers, roll deep, excellent sales people, and the execution and product are lackluster. Try to roll your own where possible or find another vendor.

</comment><comment><USER29> I've been using Watson's speech-to-text libraries to automate the transcription and tagging of videos of our usability tests.

Mixed results, so far on how accurate it is, but I also recognize that there are likely a number of improvements that can be made to the parameters that I'm setting for each processing run. I also have only started to play with the custom modeling.

Right now it's a toy, but promising for our small automation needs.

<comment><USER30> If you are processing big volumes of speech data its much better to setup Kaldi ASR on your own infrastructure. Same or better accuracy, ability to train and many other nice things.

</comment></comment><comment><USER31> As a contract developer on a social media project we started to use Watson for analysis of posts about companies (that were clients of our client) and we had to go another route due to the time constraints of using Watson as well as the fact there were easier ways to use existing libraries that did "one" ML / algorithmic task to accomplish what we were looking to do (sentiment mostly).

<comment><USER32> Isn't that particular offering basically just the Alchemy API they acquired and rolled into the platform?

</comment></comment><comment><USER33> I believe Watson, the AI engine, is still an early concept for healthcare and the reason I say this is that my friend is working on a healthcare startup that collects a vast amount of information. Their partnership with IBM means they are required to share the information that they're collecting, possibly to provide Watson with training datasets in the healthcare space.

</comment><comment><USER34> We brought the Watson consultants on for the pitch, total cost for our org was going to literally be in the millions and it was just going to be simple NLP and integration with a couple systems. The audio NLP seemed cool, the text NLP seemed easily replicated with modern libraries.

</comment><comment><USER35> [https://global.rakuten.com/corp/news/press/2017/0426_01.html](https://global.rakuten.com/corp/news/press/2017/0426_01.html)

</comment><comment><USER36> Watson has a classifier, NLC. Anyone that knows what type of classifier they are using in NLC? Take long time to train, but the respons are quite fast.

</comment><comment><USER37> I thought IBM was a blockchain on mainframe company?

Really though, I don't think bluemix/watson has anything GCP/AWS/Azure don't have at this point. Plus it looks like a bunch of their machine learning research got the axe.

</comment><comment><USER38> How would a solo dev use it if they were interested?

<comment><USER39> Apart from the free trial mentioned by sqrt17 (orange "Get started free" button at [http://bluemix.com/](http://bluemix.com/)), you can find open source (Apache 2.0) SDKs for different languages.

The one I used for NodeJS: [https://github.com/watson-developer-cloud/node- sdk](https://github.com/watson-developer-cloud/node-sdk)

It was useful to read the source code to figure out some aspects I didn't find in the documentation.

</comment></comment><comment><USER40> No.

<comment><USER41> Come on! Be fairer answering a question.</comment></comment>


Philadelphia Just Banned Most Cashless Stores - <AUTHOR> https://gizmodo.com/philadelphia-just-banned-most-cashless-stores-a-first-1833145078 <comment><USER2> Merge with yesterday's discussion?

[https://news.ycombinator.com/item?id=19328547](https://news.ycombinator.com/item?id=19328547)

<comment><USER3> Done. Thanks!

</comment></comment><comment><USER4> Cashless comes with problems that cannot be avoided. It's easy to trace cashless. What with more and more entities interested in an individual's purchases (insurance companies car/health/house), being able to buy things in cash is important.

There are certain things I will only pay cash for to avoid the stigma, insurance issues, and some others: booze, fast food, firearms (face-to-face private transaction, which is legal in TX), and tobacco. All of these are paid for with cash.

There is an increasing interest from various companies to learn who buys this stuff. Everyone is trawling the databases to learn more about you. I pay cash at the supermarket for everything, because your purchase history is sold. I know a couple of people in marketing and a couple in the insurance industry, and these people have talked with me about this stuff.

Hospitals and insurance companies are now trawling social media and other sources to get dirt on people and to see how their incomes and habits stack up for a variety of reasons. Articles abound on this behavior. I'd rather not be on the radar of anyone for my purchase habits. Cash, for now, allows people to enjoy their peccadilloes without fear of their insurance going up or other bad stigmas. I have "nothing to hide" as the saying goes, but I'm private and none of this is anyone's business.

</comment><comment><USER5> Source article: [https://www.wsj.com/articles/philadelphia-is-first-u-s- city-...](https://www.wsj.com/articles/philadelphia-is-first-u-s-city-to-ban- cashless-stores-11551967201)

</comment><comment><USER6> One of the stupidest things I've ever heard.

<comment><USER3> Ok, but please don't post unsubstantive comments here.</comment></comment>


Ask HN: How do you analyze logs? - <AUTHOR> Analyzing logs is huge part of a software problem solver's job. What kind of tools and techniques do you use to be more effective in your job? <comment><USER2> One thing we've done that's greatly improved the effectiveness of our API logs is creating a unique context identifier string for each api call and passing it back in the response headers of a call. This allows you to copy the string from the browser and grep the logs to immediately find the call in question, and if an error occurred.

</comment><comment><USER3> When I cared about logs on individual servers, I wrote a program to parse them to make it easy to find what I want, and I wrote it as a command line utility so that it could be used in combo with cat, grep, awk, sed, etc. It's on my github: [https://github.com/jedberg/quickparse](https://github.com/jedberg/quickparse)

</comment><comment><USER4> Splunk/ELK. I have used Splunk since its inception (I was one of the first paying customers) and enjoy its many features and integrations (e.g. AWS Cloudtrail, Nagios, and anomaly detection). In the past few years, I've come to know and like ELK, ElasticSerch, Kibana, and Logstash. The open source approach has some nice properties as well. It's on you to get the logs ingested but generally that is remote syslog (rsyslog,syslog-ng, or equiv) and logstash with redis. The docs on this integration are not super strong but with a little hacking you can get it working. The feature set is less in this stack and the UI is not nearly as nice. The savings are huge though especially as log volume goes up. Splunk also has a free service targeted at developers called Spluk Storm. Good for proof of concept and easy to setup without any hardware requirements as it runs on AWS.

<comment><USER5> Some people find EFK (Elasticsearch Fluentd Kibana) to be another compelling alternative to Splunk (Disclaimer: I am one of the maintainers of Fluentd)

[http://docs.fluentd.org/articles/free-alternative-to- splunk-...](http://docs.fluentd.org/articles/free-alternative-to-splunk-by- fluentd)

<comment><USER6> What makes Fluentd better than syslog-ng?

<comment><USER5> It is not "better" but different.

1\. Easier to extend that syslog-ng if you have a modest knowledge of Ruby

2\. Easy to configure file- and memory- based buffering and failover.

3\. Advanced filtering out of the box.

4\. Rich plugin ecosystem with 300+ plugins.

At least that's what I've heard from the users who switched from syslog-ng to Fluentd. I am happy to learn more about what makes syslog-ng great since I've never used it seriously myself =)

</comment></comment></comment></comment><comment><USER7> Once Splunk was about to break the bank, we abandoned it and started looking for something in the open-source world.

We've toyed with and pretty much failed using Graylog2. Although it has been coming along steadily in features and stability, we just found that the interface-although pretty-was not intuitive to us: lots of links and multi- click scenarios to get to what you want; and creating filters and streams was difficult and prone to failure.

After watching a couple of very compelling presentations by Jordan Sissel (Logstash founder), we decided to test it out. Once I realized that creating a filter (Grok rocks!) that searched for a term and reorganized the log to my liking only took a couple hours, I was sold.

Another selling point for us was that Logstash has over 2 dozen ways to suck logs in, including the usual suspects - syslog, files, tcp, udp and *mq. You can also perform a bunch of log parsing on the client (i.e. the servers with the logs) before sending them to your central ELK server/cluster.

At the end of the day, there is nothing magical about any of these systems. You alone know your logs best and have to figure out how to read/parse/search them. Our switch to Logstash from Graylog2 was our failing, not Graylog2's.

</comment><comment><USER8> We use the heck out of GrayLog2. We have a dual datacenter setup with multiple elastic search instances. A coworker of mine did a fantastic job writing up our setup:

[http://secopsmonkey.com/migrating- graylog2-servers.html](http://secopsmonkey.com/migrating- graylog2-servers.html) [http://secopsmonkey.com/migrating-graylog2-servers- part-2.ht...](http://secopsmonkey.com/migrating-graylog2-servers-part-2.html) [http://secopsmonkey.com/migrating-graylog2-servers- part-3.ht...](http://secopsmonkey.com/migrating-graylog2-servers-part-3.html) [http://secopsmonkey.com/migrating-graylog2-servers- part-4.ht...](http://secopsmonkey.com/migrating-graylog2-servers-part-4.html) [http://secopsmonkey.com/migrating-graylog2-servers- part-5.ht...](http://secopsmonkey.com/migrating-graylog2-servers-part-5.html) [http://secopsmonkey.com/migrating-graylog2-servers- part-6-le...](http://secopsmonkey.com/migrating-graylog2-servers- part-6-lessons-learned.html)

</comment><comment><USER9> I've only maintained a small number of servers, but I've found a good solution in what I'll call the GEL (Graylog2 - Elasticsearch - Logstash) stack. It's been some time since I last used Graylog2, and I can recall that it was somewhat lacking in the pretty charts and graphs department, though that may have improved recently, and the search functioned beautifully.

<comment><USER10> I'm surprised Graylog2 isn't mentioned more here. We've got tens of terabytes of logs in Graylog2, and I couldn't imagine not using its streams, alerts, and search functionality. It's become a core part of our alerting an monitoring infrastructure.

</comment></comment><comment><USER11> logstash + elasticsearch + kibana

That's for personal projects, the CloudFlare setup is a little more complex and perhaps one of the data team would be best answering that... if you're interested then I can ping them to see if there's a volunteer for a blog post describing how we do logs at scale.

<comment><USER12> An ELK stack is definitely a lot easier and straight forward to setup. Since we need to support a dozen or so teams with varying performance considerations we found that logstash left a lot to be desired. In the end fluentd performed well and gave us a lot of flexibility.

<comment><USER13> I'd like your view from the trenches regarding Logstash vs Fluentd, specifically why it's better for 12+ teams? I'm having to make a similar decision myself and would enjoy your insight! Thanks

<comment><USER5> (Disclaimer: I am a Fluentd maintainer)

Here are some comparisons out in the wild (Note: neither of them is a Logstash or Fluentd maintainer afaik)

\- [http://jasonwilder.com/blog/2013/11/19/fluentd-vs- logstash/](http://jasonwilder.com/blog/2013/11/19/fluentd-vs-logstash/)

\- [https://blog.deimos.fr/2014/05/13/logstash-vs- fluentd/](https://blog.deimos.fr/2014/05/13/logstash-vs-fluentd/)

Also, if you have any question or doubt about Fluentd, please feel free to email me at firstname.lastname@example.com

</comment></comment></comment></comment><comment><USER14> [https://papertrailapp.com/](https://papertrailapp.com/)

</comment><comment><USER15> [http://lnav.org](http://lnav.org) \-- A tool I wrote and use everyday to view/analyze the logs of the software I work on.

</comment><comment><USER16> Ozge, co-founder of topLog (toplog.io) here. Perhaps my opinion is slightly biased but here is what we have learnt for the last couple of years.

\- ELK is great (in fact, we use E+L under the hood) however you really need to know what you're doing with it and you need to spend some time configuring things while putting it together. Do you have that kind of time? Maybe, maybe not...

\- Every available tool lets you search and create alerts for monitoring. So, the analysis is always on you. This still takes a lot of manual search time during troubleshooting.

\- What if pattern and behaviour detection on your logs can be done automatically? Well, it can be. And that saves you some good amount of time instead of you creating regexs and following the trails to find the root- cause.

I would love to hear your thoughts on automated analysis and anomaly detection on logs. If you're giving a keyword and a specific time frame to search for anomalies, is it real anomaly detection? or a just an improved search?

<comment><USER17> You should give Echofish atry. It's made wonders on our network with its "whitelisting of normal behaviour". You wont beleive the things you ll discover with this approach.

EDIT: The most fascinating aspect for me is that echofish is more geared towards the actual log entries, rather than statistical analysis, in order to automatically detect anomalies in your logs activity.

<comment><USER18> Is echofish geared towards network activity norm / abnorm or general logs (syslog, app / dev logs, etc)?

Sounds cool.

<comment><USER17> Well, its approach (quoting its project page) is pretty simple:

Echofish is a purpose-built solution for filtering & monitoring of syslog activity. By whitelisting regular messages through the web UI, the administrator can instruct the log processing mechanism to create alerts only for anomalies (irregular messages).

...and actually, it can do lots more once you read the built-in help (such as distribution (using BGP) of IP blacklists, consisting of IP addresses collected through syslog activity).

TLDR; It's gearred towards filtering noise from logs. This also means you can possibly have another daemon reporting network activity through syslog, while echofish can act as your noise-filter.

</comment></comment></comment></comment><comment><USER19> Did monitoring logs with Bayesian filters ever catch on? They are very good and finding things that are "off"

<comment><USER20> By "monitoring" do you mean anomaly detection?

<comment><USER18> thats part of how our engine works.. part (full disclosure, co-founder of toplog.io)

</comment></comment></comment><comment><USER21> To the people suggesting ELK i just want to ask if you have actually used it in production? Like for real bughunting and investigating support requests?

As much as we absolutely love ElasticSearch for our other indexing needs, we find it quite hard to get the LK-part of the stack to deliver as promised. Kibana may serve up nice graphs and charts, but when you need to drill down into a large amount of log data, we often feel like loosing both overview _and_ detail.

It might very well be that we are to blame, and that we are just doing it wrong (tm) - but I would love to hear how other people are leveraging the ELK stack in production environments?

<comment><USER22> We use ElasticSearch and Kibana in production for real bughunting and support requests. Logstash was too frustrating to deal with so we wrote our own simple wrapper around an open source ElasticSearch client library to log ourselves.

We log every request (everything but the body usually) and response. If an error occurs, its logged as part of the request. We can practically replay actions taken by users and easily drill down to the exact requests pertaining to an error.

</comment></comment><comment><USER23> [http://static.googleusercontent.com/media/research.google.co...](http://static.googleusercontent.com/media/research.google.com/en/us/pubs/archive/36632.pdf)

</comment><comment><USER24> awk, grep, sed, ag, cut, sort,etc for most of the stuff. Elasticsearch, logstash, kibana for more complex queries

<comment><USER25> We use splunk but there's something to be said for the low-tech approach for ad-hoc queries.

A great little tool is 'since' \- a stateful tail. [http://welz.org.za/projects/since](http://welz.org.za/projects/since)

      since is a unix utility similar to tail.
      Unlike tail, since only shows the lines
      appended since the last time. It is useful
      to monitor growing log files.

It's in the usual yum/apt repos as well as homebrew for Mac.

It's a bit hard to websearch for because 'since' is a common word.

</comment></comment><comment><USER26> I regularly search logs across a huge fleet of hosts (thousands). I have a script that will send an arbitrary command in parallel to each host and return the output in a JSON file.

Once I get the logs I'm interested in, it's usually a straightforward combination of jq, grep, sed, awk, cut, sort, uniq, xargs, etc. If I need to do some fancy queries on the data, I have another script that will parse the logs and load them into a SQLite db.

<comment> brown9-2 What are you using to execute commands on thousands of hosts in parallel?

<comment><USER26> Just forking off processes that SSH into the hosts and write to temp files, which are picked up the by the main process. It's gotten pretty elaborate, but the basics are pretty simple.

</comment></comment></comment><comment><USER27> As others have said, the classics awk, grep, cut, sort, uniq, and scripting languages (perl/ruby for on the fly one-liners) are great for log analysis. One additional tool I've found particularly useful (three actually) is zgrep/zcat/zless. You'll often be searching through archived gzipped logs, so it's nice being able to work with the files without needing to pipe everything through tar.

</comment><comment><USER28> Lots of companies use SumoLogic [https://www.sumologic.com/signup/](https://www.sumologic.com/signup/). The UI is very similar to Splunk. But having used both Splunk and SumoLogic, I personally think that Sumo is superior in many ways.

1\. It requires very little hacking and setup to work (cough ELK cough Splunk) since Sumo is completely cloud-based. It literally takes 2 minutes to sign up, 5 minutes to download and configure some log collectors, then voila you're ready to send data and search. Also I believe you can tell Sumo Logic to grab logs directly from S3 for example if you're running everything on AWS.

2\. SumoLogic is pretty easy to use (it's basically cloud-based grep/awk) and has some really cool features that makes Splunk feel clunky in comparison. Parsing and transposing data for graphing is really simple. Also little things like auto-suggesting sources/hosts while you're typing a query makes the experience much smoother than jumping around tabs copy/pasting shit.

3\. If you start to generate a lot of logs, and I mean a metric fuckton of logs from 1000s of servers, Splunk Storm will most definitely not be able to help you. In-house Splunk / ELK clusters will need to be carefully sized (just google ELK sizing).

As software developers, we have enough on our plates that it really pays to use tools that help rather than make you wanna throw up your fists and curse. KISS

<comment><USER29> Coincidentally, I had a demo call with Sumo today. I was very impressed, except for one point that makes Splunk the clear winner at the moment: I can send structured logs to Splunk and it automatically finds the keys and allows me to query on them immediately.

Meaning, I can send: 2015-02-02 01:00:00 event="Product sold" price=5

And with zero configuration in Splunk, I can now query: event="Product *" price>2 | stats sum(price)

And in the next iteration of my app, I could add 30 more key/value pairs to that message and could query on the rest of them just the same, no configuration. It makes development incredibly rapid to be able to instantly report on any metric anyone on my team logs out, debug-related or otherwise, without having to maintain some master list of every key in every log message in every service we write.

I was floored in my Sumo call today when I was told that wasn't the case in that product yet. It seems like such a basic feature-- and is why many products have switched entirely to JSON-based logs. Have you discovered a workaround, or find that to be as cumbersome as I'm anticipating?

<comment><USER30> Sumo Logic currently has the ability to extract known fields on ingest, making them available for searches, much like the Splunk query provided above. Dynamic fields, such as new KVPs that are logged out are able to be pulled out in the query with one extra step, as follows:

| kv infer "event","price" | sum(price) by event | where price >2

The kv operator refers to key value pairs. There is also a json operator which functions the same way.

</comment></comment></comment><comment><USER31> Personally i use Emacs with occur-mode for filtering text also when it finds the pattern it is able to switch from instances quickly. Sometimes i also use regex with occur to find multiple items.

Also Notepad++ has a Analyze plugin which i recommend for complex stuff and if you dont like emacs.

<comment><USER32> I second emacs occur.

</comment></comment><comment><USER33> Kibana provide an excellent way to track log.

Currently handle +100gb/day of heavy document (more or less 100 items per document), on our current setup. And probably designed to handle way mooore.

Dashboard are constantly opened on +50 screens and we use them also to track MySQL, mailing, internal stats...

</comment><comment><USER34> Used papertrailapp.com but as the number of servers grew, this became expensive.

Moved to EL + Kibana, but not liking the interface yet and it doesn't seem to have 'tail -f' kindof functionality.

<comment><USER35> We use paper trail. I suppose expense is relative, but it's absolutely worth it to us. They've really nailed log search, which is what is really important to us.

</comment></comment><comment><USER36> Kafka for log aggregation. sed,awk,grep,tail and other bash utilities for analysis. Extremely fast at over 5k lines per second and totally maintenance free running on a 2G AWS instance

</comment><comment><USER37> ELK is easy to setup but when you really need to analyse logs it can be painful, I think the real problem is Kibana which is just not good enough for that task.

\- no logs coloration

\- a lot of bugs / weird refresh behavior

\- no auth

<comment><USER13> Do you know of a better alternative to Kibana for a EL_ stack?

</comment></comment><comment><USER38> At Mozilla Opsec, we use (and write) MozDef: [https://github.com/jeffbryner/MozDef/](https://github.com/jeffbryner/MozDef/)

<comment><USER39> Amazing name

</comment></comment><comment><USER40> Splunk. Here's my latest Splunk log analytics app:

[http://www.mensk.com/#prettyPhoto/0/](http://www.mensk.com/#prettyPhoto/0/)

</comment><comment><USER22> Ship JSON to ElasticSearch, visualize with Kibana, and backup to S3/glacier.

We use found.on and qbox.io for managed hosting of ElasticSearch clusters.

</comment><comment><USER41> graylog2. they have worked around the shortcomings of elasticsearch for log management (in the end it is a lucene based full text search engine for general purpose tasks) and the 1.0 that is about to go GA soon has crazy stability

[https://www.graylog2.org/](https://www.graylog2.org/)

</comment><comment><USER42> LogEntries.com

<comment><USER43> We tried logentries, but their agent was a terrible java-application, that was hard to get working right.

The list of files was saved on their service (rather than in a text-file on the server), and the name of our servers was also guessed by their servers which made it hard for us to add and maintain servers.

I think they should build a better agent that embraces UNIX more, and can be configured through a local configuration. Their platform seems nice, but we weren't able to use it, sadly.

<comment><USER44> Hey - You might want to check the agent docs here which have a local configuration file: [https://github.com/logentries/le#configuration](https://github.com/logentries/le#configuration)

So the logs being followed are actually configured in a text file on the servers. This makes it super simple for deploying via chef/puppet in large scale environments.

</comment></comment></comment><comment><USER45> Grep, awk, sed for the easy stuff, logastash/elasticsearch/kibana for the harder stuff.

</comment><comment><USER46> logstash

</comment><comment><USER47> fluentd -> elasticsearch/kibana

Works pretty damn well.

</comment><comment><USER48> www.sumologic.com

<comment><USER49> My work recently switched to sumo and I love it. I've you've used splunk, logstash, elastic search + kibana previously and think sumo is the best. It is the right amount of power and simplicity plus great documentation.

</comment></comment><comment> bra-ket grep

</comment><comment><USER20> How many logs are we talking about, and produced at what rate? These are key questions. If you have tons of logs and you might have to make a full access pass over them, the last thing you want to do is to centralize them on some kind of logs host with lots of disks and few CPUs. But if you have little or moderate amounts of logs you may be able to get away with a single host and some xargs -P grep type of thing.

</comment><comment><USER50> Logstash/kibana (free, open source)

Splunk (good, but ridiculously expensive)

<comment><USER51> You can do quite a lot with Splunk Storm[0] for free.

[0] [https://www.splunkstorm.com/](https://www.splunkstorm.com/)

<comment><USER52> And if you prefer a non-cloud option, there's a Splunk Free license [1] which allows for indexing of up to 500MB logs per day.

[1] [http://docs.splunk.com/Documentation/Splunk/6.2.1/Admin/More...](http://docs.splunk.com/Documentation/Splunk/6.2.1/Admin/MoreaboutSplunkFree)

</comment></comment></comment><comment><USER53> splunk</comment>


Show HN: Android game discovery website that I built a few years ago - <AUTHOR> http://playworthy.io <comment><AUTHOR> edit: To find more about me and motivation behind the project, please read my about page [https://www.playworthy.io/info/about/](https://www.playworthy.io/info/about/)

I needed to learn Django a few years back and was aware that the play store was actually an awful place for discovering high quality games that were worth my time and that's how this project came to be. It didn't receive much love on reddit and it never took off. Inspired by a similar project posted here a couple of days ago that I just saw ([https://news.ycombinator.com/item?id=22063744](https://news.ycombinator.com/item?id=22063744)) I decided to show my project as well and got some feedback.</comment>


Going back to school for programming? Yes? No? Are you crazy? - <AUTHOR>

After learning about YCombinator almost a month ago I ended up applying for my project that I've been working on. As I've read more and more though I know I face an uphill battle. A) I'm not a programmer so I'm trying to work with a company. B) I'm founding my project solo.<p>Truth is I have an economics and finance background. I am absolutely fascinated with the possibilities of the internet over the past few years and have see the proverbial "light" or I keep leaving my MAC on at night. I'm considering going back to school for a postbac in computer science. Should I go this route or are there other alternatives? At the moment I have an enormous amount of student loan debt and I am nervous about adding to what seems like an already insurmountable amount but I believe I can be successful in this field. But do I need another degree to be so? <comment><USER2> A common theme here is that computer science doesn't teach you to be a programmer. You learn to be a programmer through practice and experience. Also, learning a scripting language for web development is relatively easy. Rather than spend time in class, take some time to go to a coding cave and learn the basics of one of the popular scripting languages. Take some time to practice every day. Build some applications. Read Hacker News to see what people have answered for similar questions.

Personally, I'm not sure this is the best route to go. Bootstrapping by learning web development is fine, but keep in mind there are still more roles that you may not be able to fill. For example, people who spend most of their days writing code generally aren't great designers. Designers who can make things pretty may not be the best user interface people. A great development team still needs people who can make sales and know how to run a business well. So, you could spend a lot of time learning to be a coder and still have big gaps to fill depending on what you are building.

</comment><comment><USER3> If you're thinking of going back to school strictly just to learn how to program in a "structured" way that you do in school, I would suggest checking out the free courses that MIT offers online. ([http://ocw.mit.edu/courses/#electrical-engineering-and- compu...](http://ocw.mit.edu/courses/#electrical-engineering-and-computer- science))

And it will give you a taste if learning programming in a school like environment is right for you.

Good luck!

</comment><comment><USER4> You can be up and running in just a few days to weeks if you just start doing it on your own. You dont need to go to school. There are some fundamentals of computer science that would definitely be useful like basic algorithms and data structures that may take longer to learn but are still doable. Taking a single language course at a community college might not be a bad way to start though.

</comment><comment><USER5> You don't need no formal education, in my humble opinion.

Instead, practise, practise, practise.

One good starting point: <http://docs.djangoproject.com/en/1.3/intro/tutorial01/></comment>


The future of electronics based on memristive systems - <AUTHOR> https://www.nature.com/articles/s41928-017-0006-8 <comment><USER2> I've just started a PhD on neuromorphic memristive Systems at EPFL, so if someone wants to ask questions/chat about this I will do my best to serve. Here or via the contact info in my profile:)

<comment><USER3> I'm finishing a PhD on neuromorphic computing. If _you_ want to ask me about it I will do my best to serve :)

<comment><USER2> Heh cool, what did you work on? What did you find the hardest? What do you regret not having time for/would you like to add to what you did?

<comment><USER3> I've worked on HW/SW codesign, for mixed signal chips, initially for convnets and image recognition, and later as a more general DL architecture. My advisor is the guy who "found the missing memristor" at HP in 2008.

Looking back now I regret going into HW field. I should have applied to CS rather than ECE, and focus on DL algorithms (especially RL), or maybe even something like what Numenta is doing, because my primary interest is AI, not hardware to run AI.

The hardest part has been being the only DL/ML specialist in my group. Everyone else here is more HW-oriented. I only understood this when I did an internship where I worked with people I could learn from/discuss ideas with.

I looked at your CV, and my guess is that for you, the hardest part will be to focus on one thing for the next 4 years.

<comment><USER2> Thanks for sharing and this:

>I looked at your CV, and my guess is that for you, the hardest part will be to focus on one thing for the next 4 years.

Is an astute observation although I've found more than enough things in my lab to keep my interest so far^^"

Could you please link or mail me some of your publications? I'd greatly appreciate it

<comment><USER3> Our group page: [https://sites.google.com/site/strukov](https://sites.google.com/site/strukov)

If you like to chat more, send me an email.

</comment></comment></comment></comment></comment></comment><comment><USER4> The last paragraph is everything!

"We conclude by noting that biology has always served and will continue to serve as a great inspiration to develop methods for achieving lower-power and real-time learning systems. However, just as birds in nature may have inspired modern aeronautics technology, we eventually moved in new directions and capabilities for faster travel, larger carrying capacities and entirely different fuelling requirements. Similarly, in computing, modern application needs to go beyond those faced in nature, such as searching large databases, efficiently scheduling resources or solving highly coupled sets of differential equations. Interestingly, some of the observed characteristics in memristors may similarly provide 'beyond biology' opportunities in computing, taking advantage of the novel device dynamical behaviour and the network topology inspired by biology. In this regard, concepts such as the memory processing unit represent truly exciting opportunities down the road. To achieve these and other new computing systems of the future will require persistent and creative research that goes beyond any single discipline, and must include insights from neuroscience, physics, chemistry, computer science, and electrical and computer engineering, among others."

<comment><USER5> I don't understand. Can someone please dumb it down one setting on the dial? What makes this obviously superbiologically capable?

<comment> T-A The obvious things are size, speed and integration with existing computers.

A neuron is measured in micrometers, a memristor in nanometers (and the paper suggests even single-atom devices may be possible).

A neuron can fire at most a couple hundred times per second; a memristor has "subnanosecond switching speed", i.e. it operates in the usual GHz operating frequency range of modern computers.

And you can integrate them at the circuit level, so they can draw on the superbiological capabilities of existing computers at native speed.

<comment><USER2> Downsides:

* routing is a BITCH

* cooling is a BITCH

* noise is a slightly nicer BITCH

* we are very early in building real stuff with this

But yeah, it's cool, I hope it works out (otherwise I chose a deadend/false start for my PhD :-)

</comment></comment></comment></comment><comment><USER6> Beyond replacing current transistor logic, I find it theoretically pleasing that memristors fill the gap to complete the linking of current, voltage, charge, and flux with the existing 3 fundamental passive devices. What would basic circuit analysis lessons look like 10 years after they become commonplace?

<comment><USER7> > What would basic circuit analysis lessons look like 10 years after they > become commonplace?

Exactly the same as they do now. Memristors aren't linear time-invariant, so it's extremely unlikely that they would show up in a basic circuit analysis lesson.

<comment><USER8> Not even inductors or caps are in a basic circuit analysis course. But this quadriga of passive components was at least mentioned in the very beginning of the first ee semester, components class which focused the physical effects (and was notoriously hard).

</comment></comment></comment><comment><USER9> The "future" should be in quotes. Articles have been selling these BS for 20 years now --- and we're gonna get the same articles for the next 30 years at least....</comment>


How to hire unicorns (aka tech-savvy marketers) - <AUTHOR> http://venturebeat.com/2014/09/11/how-to-hire-unicorns-aka-tech-savvy-marketers/

<comment><USER2> IMO a unicorn is a full stack developer that also has design / marketing knowledge. the thing is about these guys, is most of them are savvy enough to create their own products, apps or startups and if you do find them it is a matter of time before they break off and do something on their own.

<comment><USER3> I don't think that's always the case. I ran my own company for years but honestly running a company gets exhausting. After years of doing it I came to the realization that what I really like doing is growing companies, not running them.

Once I realized that, it was an easy transition back to being an in-house marketer.</comment></comment>


Interactive Course about Fractals (From Mathigon) - <AUTHOR> https://www.youtube.com/watch?v=ohygDnKg5mg <comment><USER2> This is just a trailer video, but you can access the course itself at [https://mathigon.org/course/fractals](https://mathigon.org/course/fractals)

</comment><comment><USER3> Wow... so beautiful!</comment>


Ask HN: InfoSec questionnaire – what to do when customer wants one completed? - <AUTHOR> A colleague asked me today if I would recommend they fill out an infosec questionnaire to secure a high-profile question. I'm stumped and maybe some of you have some practical advice. The questions are, basically, do you have bank-level security, and if not, in what way - in quite great detail. Has anyone here run into this situation? If the
correctly-answered questionnaire got into the hands of the wrong person, they would have a lot of information to infiltrate the system.
Example questions:
"Are you able to detect and protect accounts that may have been compromised?"
"Do you allow users to change their passwords more than once in a 24-hour time period?"
Completely valid but how many small businesses without a security expert on board, have these in place? And why would it be beneficial to tell a customer about this? How could a small business deny completing this questionnaire but still get the customer on board? <comment><USER2> Properly constructed information security questionnaires enable a business partner to conduct a high-level, non-intrusive assessment of your organization. You have a right to be concerned about the sensitivity of the data, but if you distrust your business partner that much, then don't do business with them. Your org is not a special snowflake and this is a very common practice by organizations of all shapes and sizes with organizations of all shapes and sizes. They are not the be-all/end-all of information sources (orgs regularly fib on these forms) and you do have a similar right to ask the inquisitors how they will protect your form data (that will also partoy show them you at least give lip-service to data security). It's also far more likely that your organization is going to get pwnd via phishing that is completely unrelated to the potential loss of confidentiality of this document. I say that as someone who has formally studied cybersecurity breaches for years.

Also, as cyberinsurance increasingly becomes "a thing", you're going to see this questionnaire situation increase in frequency. Your org should consider creating a pre-composed (and regularly updated) SSAE 16 ([https://en.wikipedia.org/wiki/SSAE_16](https://en.wikipedia.org/wiki/SSAE_16)) to avoid having to fill out unique assessment questionnaires for every request that comes in. It'll save you time and — unless you "go N/A crazy" on the SSAE 16 — should be accepted by any firm worth doing business with.

</comment><comment><USER3> >Completely valid but how many small businesses without a security expert on board, have these in place?

Sorry to say this, but none of your examples require a security expert to implement. And frankly, they are both what I would consider low hanging security fruit. If you balk at implementing these, I question whether your product is ready for the enterprise.

>And why would it be beneficial to tell a customer about this?

This is risk management and honestly, if you don't understand risk management, you'll have a lot of trouble serving enterprise customers.

It's beneficial because in the company's mind, a legal action is a big enough risk that they want to invest resources in showing they were diligent before picking a partner.

>How could a small business deny completing this questionnaire but still get the customer on board?

There is almost no chance unless you have strong political ties to a major decision maker at that company. And, even if you do have that kind of connection, this form isn't asking for the keys to the castle.

Can your company provide a decent level of data security? How do you know?

If the answer to question #1 is not a resounding yes, you have no business trying to serve an enterprise.

</comment><comment><USER4> In my experience you won't need to answer yes to every question in order to "pass" the info sec, but their team will want to be able to understand the risk of doing business with you. Depending on exactly what it is you'll be doing together, will determine how much risk they are willing to take on.

<comment><AUTHOR> Most of the questions should not be answered, in my opinion - if the document got into the hands of a bad-actor it could be abused. I'll recommend they liberally use the N/A option and try and have a discussion with the customer security team.

<comment><USER3> Based on the example questions you posted, you are being paranoid. Neither one of those questions are terribly sensitive and it doesn't matter if the answers fall into the wrong hands. Your company isn't that special - many companies want to know the same things before they work with someone.

Three things:

1.) Security through obscurity never works.

2.) Based on the examples you posted, they aren't asking for anything special. In fact, they both seem like about the base level of security I would expect an enterprise ready company to provide. If you want to liberally enter n/a to cover up that you aren't big enough/don't have enough people/haven't implemented what they ask, that is dishonest.

3.) If questions like this cause you so much trouble, you need to seriously ask yourself whether working for a startup is for you. Due diligence processes (either initiated by an investor who wants to fund you, or a body that wants to acquire you) should be expected to go much deeper.

<comment><USER3> It's too late to edit this response, but I wanted to add something.

In my original, I said that security through obscurity never works. That isn't entirely true, because it might work great. The problem is that it makes it much harder to defend yourself after a breach has happened.</comment></comment></comment></comment>


Life After StrongLoop – A New Beginning with LunchBadger - <AUTHOR> In February of this year, I left StrongLoop. Want to know another little known fact? I was never part of IBM. That's right - I never signed any IBM papers continuing post acquisition and remained the sole StrongLoop employee till the dissolution of our corporate entity.
Why? I didn't agree with the direction - simply put. And from what I've seen so far, I would add "strongly" in front of "disagree".
I was in a post acquisition mess. It got so bad that I even hired an attorney.
"You should write a book…", my attorney said.
God forbid that he would ever be able to - being bound by client attorney privilege.
"It's so sensational, everyone would think it's fiction."
Maybe...some day. The truth of the matter is dealing with acquisitions and transitions is not fun, even for a founder - especially when there is a lot riding on the line for your career choices.  Unfortunately, it happens more often than not here in Silicon Valley.
On the brighter side, four years later - I learned a lot.  Especially, from you my colleagues, who dared to dream of better software that really challenges the status quo and offers simple value by making the day to day life easier for all those who deal in technology.
What I took away from StrongLoop was an unanswered call to action - a challenge of sorts in the form of questions like…
- Can cloud software that solves hard problems still be easy to use? - Digital transformation???  How about the rest of the us who are already born in the cloud and not worried about dinosaur concerns what do we use? - How does the API economy, serverless, composable enterprises transform how we build the businesses of tomorrow?
Today, we launch LunchBadger- https://www.lunchbadger.com. We're looking for some brave souls willing to indulge an attentive ear and open mind on what APIs could mean for a cloud native world.
Thanks for your time! <comment><USER2> I think you should focus on something like Docker and Swarm providing IaaS and everything being fully open-source. No one is going to want to get their hands dirty on this one without knowing what's within it.. Good luck, best wishes!

<comment><AUTHOR> Thanks! That's great point - thanks for pointing out that I need to clarify - that's exactly what we've done.

Built on everything on top of open source and package it up as extensible service or run entire on-premise on your cloud. \- The container based runtime is built on top of Kubernetes. \- The SCM CI/CD integration and config as code is done via git \- The serverless models and functions we're looking to use open source Node.js framework I help create - LoopBack.io, back at StrongLoop. \- The GUI is entirely extensible via React and webpack.

The best part is - we've abstracted our interfaces so that your underlying services can be wrapped up in JavaScript no matter what language, OSS framework you have etc - bring it and we'll plug on top of it to help with reducing complexity and saving time.

</comment></comment><comment><USER3> Although I came from larger enterprises, I am being reborn on the cloud. Sounds exciting ! Could you shed light on the serverless part ? Is this like Lamda ?

<comment><AUTHOR> we're about providing a serverless experience to everyone - regardless of what cloud you're using, the trick though is provide a serverless experience to the devs while giving devops full visibility of what's going on in the runtime

</comment></comment><comment><USER4> Same question about Lambda

<comment><AUTHOR> we're looking to support AWS Lambda, as well as other IaaS serverless implementations - but also give you "AWS Lambda in a box" for developer experience - but with devops transparency and control

<comment><USER3> Private Lamda is great...your video shows predictive modeling. what algo are you using ? Can I bring my own ?

<comment><AUTHOR> auto-regression, cyclical, and yes - apply your own via machine learned or manually specified</comment></comment></comment></comment>


Ask HN: Legality of using M Turk to filter content - <AUTHOR>

Hi!
Perhaps I should ask a lawyer this, but I'm sure someone on HN must have already faced this issue and figured it out. Here it is:
Suppose I'm running a social network site and users are allowed to upload photos. The site needs to filter for adult content, but we lack the resources to do so ourselves. This seems like a perfect application for M. Turk. Now, are there any potential legal issues for this (since the photos presented to the M. Turk-ers are user uploaded photos.)
Thanks! <comment><USER2> It's almost certainly not illegal so long as your TOS is worded right. The only time I could think of it being illegal is if your TOS promised never to provide the pictures to anyone besides their friends. But that's unreasonable anyway because you need some kind of admins.

Now, it's still tricky, because if the solution allows anyone to go to a link you post on m-turk and see users' content, users will get very mad. So you need a secure interface that only lets them see the specific pictures. But it could still get problematic.

Better might be attempting to hire a person or a few people for very cheap (possibly overseas). The would work some real amount of time for you, and they'd have to sign all kinds of things saying they won't do anything with the content. Then no one could get mad because theyr'e an employee, not anyone with an m-turk account.

<comment><USER3> I think there might also be an issue with the turkers, since they might be exposed to all kinds of "illegal" stuff that users uploaded.</comment></comment>


Why the Mythbusters won't do RFID (2008) - <AUTHOR> http://www.youtube.com/watch?v=X034R3yzDhw <comment><USER2> Actually Adam did a hasty follow-up to this when the video came out to say something to the effect of 'Hmmm, I may have embellished the story - and um that didn't happen' (BTW, thats me doing some heavy me para-phrasing, not a quote)

Here's the link:

<http://news.cnet.com/8301-13772_3-10031601-52.html>

September 3, 2008 10:59 AM PDT

'MythBusters' co-host backpedals on RFID kerfuffle

<comment><USER3> Wonderful. Which account do we believe now?

<comment><USER4> This was discussed on No Agenda recently as an example of why corporate advertising is bad for media that does this kind of work (although Mythbusters rarely broaches subjects that run against corporate culture).

It seems odd Adam would have made up his original account, but only he knows the real story at this point.

<comment><USER5> Found this: [http://www.youtube.com/watch?v=vmajlKJlT3U&feature=relat...](http://www.youtube.com/watch?v=vmajlKJlT3U&feature=related)

<comment><USER6> I don't really understand the problem to be honest. Maybe someone can explain it to me.

Sure it's mildly inconvenient if your CC number gets stolen. But the CC company is the one that foots the bill. In that sense, they are the ones with the best incentive to keep the number secure. If fraudulent transactions instigated by RFID-scanning thieves ever gets to the point where it is a serious concern, I am certain that the companies will act in their own best interest to curb the behavior. In the mean time, who cares if they lose some money?

<comment><USER7> If they can convince the judges that the cards cannot be skimmed, than the very existence of a record of a transaction with a skimmed RFID is legal proof that you did in fact authorize that transaction with your authentic card. There is absolutely no risk for the card issuer involved.

It is like it was with debit card PINs here in germany, the banks convinced the judges that the cards are absolutely secure so that any fraud was in fact to blame on the card holder who either didn't protect his PIN or was actively trying to defraud the bank.

<comment><USER8> [citation needed] (out of personal interest)

<comment><USER7> Here is a citation (in german legalese, sorry for that): <http://www.jurpc.de/rechtspr/20000026.htm>

<comment><USER8> Even though I am German, thanks for the warning ;)

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER9> This is just begging for a little unaffiliated team to do a professional investigation along with a good amateur video producer.

<comment><USER10> ...and decent legal counsel.

<comment><USER11> What if they're outside North America or Western Europe? Some place with weaker laws?

<comment><USER12> Personally I would call weaker the laws that don't allow you to produce a video like this.

</comment></comment></comment></comment><comment><USER13> Well Adam has stated in at least one interview that he reads Hacker News regularly so maybe we'll have a nice anonymous reply :)

<comment><USER14> I was curious about the validity of your statement so I did a little digging. It's true, he even mentions that he reads it on a daily basis.

Source: <http://www.youtube.com/watch?v=J8jqea8R-bE>

_[edit]_ if you don't want to watch the whole 3 part video: <http://www.youtube.com/watch?v=fFcVaFhKd_4#t=08m32s>

<comment><USER15> You could use (also on HN front page) <http://speakertext.com> for linking to that. Just thought it was a neat instance!

</comment></comment></comment><comment><USER16> I'm the founder of a company that sells RFID blocking wallets and passport cases <http://www.difrwear.com>. I met with Adam briefly back in 2008 at HOPE when he gave this talk and gave him one of our wallets.

I ended up quite dissaponited they couldn't air the show. Would have brought a lot of awareness to the issue. It is really easy to copy RFID credit cards... all you need to do is go buy a point of sale terminal from eBay, poke the little speaker that beeps when it reads a card with a needle and then plug it into a laptop and you've got a skimmer.....

<comment><USER17> I totally want to buy one of your wallets but you are currently out of stock on everything except the garish pink ones..

</comment></comment><comment><USER18> Can someone that knows about this stuff explain exactly what it is the CC companies don't want us to know?

<comment><USER19> I'm guessing it's:

a) RFID is readable from further away than they'd like you to think.

b) You don't know when your RFID card is being read.

c) Points a and b make tracking you really easy... for anyone to do.

d) The only thing that _should_ (ideally) be stored on any RFID chip is a unique number... not any history (recent transactions), personal data (name/phone/picture), or payment system (think public transport) where the actual info about how much money is on the card is stored on the card itself... but that's exactly the type of information which is stored on these cards.

e) Nearly all encryption mechanisms are shoddy, either because they're poorly implemented open standards, or developed in-house by the vendor (security through obscurity). Cards that make use of real encryption would be (are?) expensive to make.

Here in the Netherlands the entire public transit system is being switched to an RFID-based system, and even to a non-security expert (me) it's clear that the system is based on an insecure premise (d), and would be very vulnerable to unknown scanning by someone wishing to track you from a decent distance (a-b-c).

I was interested in the security of this system, and found this video ([http://events.ccc.de/congress/2007/Fahrplan/events/2378.en.h...](http://events.ccc.de/congress/2007/Fahrplan/events/2378.en.html)) of some hackers who did an amazing job tearing it to shreds. They're pretty adiment that nobody is doing adequate encryption on RFID cards. If you're interested in this at all it's an amazing hack, involving dissolving the cards layer by layer to see the code.

<comment><USER20> a) it's a radio signal. However low power it is, it gets transmitted _huge_ distances while still being detectable (especially if you capture it multiple times to read through noise). I'd love to take a massive dish (say, 20 foot diameter) & see how many can be captured from _inside_ a neighboring building.

b) I have yet to hear of a single RFID card which has a _switch_ on it to address this. It's a _big_ security problem. I saw one hobbyist hook up an OLED pixel, but that's it.

e) I've heard of a couple, _very_ expensive, challenge-response and public-key RFID systems. _That_ is acceptable for authentication, but I've _never_ heard of them actually being used, and one or two were only proofs-of-concept, IIRC. Many (I'd say easily _most_ in use, from what I gather) simply transmit a unique ID, that never changes, which is used to perform X, which is ridiculously insecure.

<comment><USER21>
      > I'd love to take a massive dish (say, 20 foot
      > diameter) & see how many can be captured from
      > inside a neighboring building.

Are you talking about active or passive RFID? I was under the impression that most RFID in use is passive. In that case, you'd have to transmit something to get a response, unless you're talking about camping out in an area where lots of cards are going be activated by various things other than yourself (e.g. entrance to the transit system). But even then the transmitting power of the RFID chip is proportional (?) to the power used to activate it, so something that only expects to read it from 2 feet away isn't going to blast it with enough power to be reliably read from 100 feet away, unless I'm misunderstanding how people do those long distance RFID reading records...

<comment><USER22> So say a 5-foot range. Find a group of employees out for lunch together and I walk past the table with a backpack on. Hardly suspicious, and I've probably got most of their building access cards.

<comment><USER21> I was responding to someone to someone talking about a 20-foot dish though. That's not something you stuff in a backpack. I was commenting on his desire to listen with a huge dish at a distance.

</comment></comment></comment></comment></comment></comment><comment><USER23> Johns Hopkins University researchers did a video where they got the RFID code from one of those gasoline auto-passes that you put on your keychain. It was a video where they sit next to someone with the RFID pass in their pocket, scan it with their laptop and then use the code at a gas station. I am at work and the link it blocked, but I do believe this here is the video and information. <http://rfidanalysis.org/>

<comment><USER24> The link is now squatted but still available in archive.org :

[http://web.archive.org/web/20061109232923/http://www.rfidana...](http://web.archive.org/web/20061109232923/http://www.rfidanalysis.org/)

</comment></comment><comment><USER25> Thankfully with a community named 'HackerNews' hopefully somebody here will be inspired to look into it deeper and see if they can do the show that networks can't do.

What's required to figure out how to hack these chips which are clearly readily available?

<comment><USER26> This is actually an active research area so a google scholar search can turn up interesting stuff from various security conferences. Here's a summary of what I've read and heard about:

If a chip has unecrypted personal data stored on it an attacker can easily gain access to it by stealing the device. If encryption is used throughout the chip then side channel attacks can usually break the encryption. This requires something like an oscilloscope, some resistors, and a soldering iron. The danger of this attack to a consumer depends upon what's stored on the RFID chip since the consumer will notice if someone has stolen their device and will have it disabled in short order.

To clone a tag that doesn't use encryption, for instance a tag that just sends an ID, you'd need a reader to query the tags and some device to copy the responses. This is probably the easiest attack but the reader, which needs to transmit a strong radio pulse and then listen for a response, either needs to be very large or in very close proximity and you could protect a card in your wallet by surrounding it in a metal mesh (which forms a faraday cage) so it's not clear how dangerous this could be in the wild.

If the communcation channel is encrypted then an attacker could listen to the query and response from a legitimate reader and RFID tag and could then replicate the legitimate response later. However, if there is any timestamp or counter involved this won't work.

</comment></comment><comment><USER27> Primarily about sniping bluetooth, but some basic discussion on reading RFID tags at a distance in the comments on this thread: [http://www.schneier.com/blog/archives/2005/04/bluetooth_snip...](http://www.schneier.com/blog/archives/2005/04/bluetooth_snipe.html)

</comment><comment><USER28> Let HN users create our own RFID MythBusters?

</comment><comment><USER29> Ok, so how hackable/trackable is RFID?

</comment><comment><USER30> Site is 404-ing currently...</comment>