More Data Is Not Better and Machine Learning Is a Grind - <AUTHOR> https://scm.ncsu.edu/scm-articles/article/more-data-is-not-better-and-machine-learning-is-a-grind-just-ask-amazon <comment><USER4> More data is better.

You can reduce it via PCA one of the many techniques in multivariate statistic.

You can do anova to select your predictors.

In general you can use a subset of it using the tools that statistic have provided.

Complaining about messy data... welcome to the real world. As for complaining about non-reproducible models , choose a reproducible ones. I've only done mostly statistical models and forest base algorithms and they're all reproducible.

All I see in this post is complaints and no real solutions. The solution that's given is what? Have less data?

> The results were consistent with asymptotic theory (central limit theorem) > that predicts that more data has diminishing returns

CLT talks about sampling from the population infinitely. It doesn't say anything about diminishing returns. I don't get how you go from sampling infinitely to diminishing returns.

<comment><USER5> >> All I see in this post is complaints and no real solutions. The solution that's given is what? Have less data?

The solution is to direct research effort towards learning algorithms that generalise well from few examples.

Don't expect the industry to lead this effort, though. The industry sees the reliance on large datasets as something to be exploited for a competitive advantage.

>> You can reduce it via PCA one of the many techniques in multivariate statistic.

PCA is a dimensionality reduction technique. It reduces the number of featuers required to learn. It doesn't do anything about the number of examples that are needed to guarantee good performance. The article is addressing the need for more _examples_ , not more _features_.

<comment><USER6> >>>Don't expect the industry to lead this effort, though. The industry sees the reliance on large datasets as something to be exploited for a competitive advantage.

This is only true for the Facebooks and Googles of the world. There are definitely small companies (like the one I work for) trying very hard to figure out how to build models that use less data because we don't have access to those large datasets.

The industry is larger than just the Big N.

<comment><USER5> Btw, if you have relational data and a few good people with strong computer science backgrounds rather than statisticians or mathematicians, have a look at Inductive Logic Programming. ILP is a set of machine learning techniques that learn logic programs from logic programs. The sample efficiency is on a class of its own and it generalises robustly from very little data[1].

I study ILP algorithms for my PhD. My research group has recently developed a new technique, Meta Interpretive Learning. Its canonical implementation is Metagol:

[https://github.com/metagol/metagol](https://github.com/metagol/metagol)

Please feel free to email me if you need more details. My address is in my profile.

___________________

[1] As a source of this claim I always quote this DeepMind paper where Metagol is compared to the authors' own system (which is itself an ILP system, but using a deep neural net):

[https://arxiv.org/abs/1711.04574](https://arxiv.org/abs/1711.04574)

 _ILP has a number of appealing features. First, the learned program is an explicit symbolic structure that can be inspected, understood, and verified. Second, ILP systems tend to be impressively data-efficient, able to generalise well from a small handful of examples. The reason for this data-efficiency is that ILP imposes a strong language bias on the sorts of programs that can be learned: a short general program will be preferred to a program consisting of a large number of special-case ad-hoc rules that happen to cover the training data. Third, ILP systems support continual and transfer learning. The program learned in one training session, being declarative and free of side-effects, can be copied and pasted into the knowledge base before the next training session, providing an economical way of storing learned knowledge._

<comment><USER6> Ah yes I am very familiar with ILP - thanks for sending these references!

<comment><USER5> You're welcome, and what a pleasant surprise, it's rare to find people who know about ILP in the industry :)

</comment></comment></comment></comment></comment></comment><comment><USER7> In image recognition, Google asserted in 2017 that they were unable to find decreasing returns -- "Performance increases logarithmically based on volume of training data." Maybe for supply-chain and regression models there is a limit but it seems in deep neural nets maybe there's a different answer.

Blog: [https://ai.googleblog.com/2017/07/revisiting-unreasonable- ef...](https://ai.googleblog.com/2017/07/revisiting-unreasonable- effectiveness.html)

Paper: [https://arxiv.org/abs/1707.02968](https://arxiv.org/abs/1707.02968)

<comment><USER8> Isn't that the very definition of diminishing returns?

<comment><USER9> Yes, precisely. Logarithmic growth asymptotically decreases - it's the definitional opposite of exponential growth. This is clearly illustrated in typical graphs depicting logarithmic growth: [https://jamesclear.com/wp- content/uploads/2015/04/logarithmi...](https://jamesclear.com/wp- content/uploads/2015/04/logarithmic-growth-curve-1200x800.jpg)

So in point of fact, according to the cited paper Google is asserting there _are_ diminishing returns to increasing the volume of data.

<comment><USER10> The two of you are asserting different hypotheses than the OP presented: diminishing returns != decreasing returns. The Google paper found that increased amounts of data always improved performance, but did so at a lower _rate_ the more data that had already been provided. First derivatives vs. second.

The headline of this article is "More data is not better", which is a stronger claim than diminishing returns - it's neutral or negative returns.

<comment><USER5> >> The headline of this article is "More data is not better", which is a stronger claim than diminishing returns - it's neutral or negative returns.

Well, if I was paying $10,000 for 10,000 examples (to collect, cleanup, process, train with, etc), getting 90% accuracy and making $90,000 from the training model, and now I'm paying $10,000,000 for 10,000,000 examples, getting 91% accuracy and making $91,000 from the trained model, I'm losing money where before I was making some. That's "not better".

<comment><USER10> Few big tech companies pay for training data - it usually arises organically out of usage data from their products. You need to collect this anyway for product function / business metrics / abuse prevention, and build the cleaning & processing pipelines. So the only marginal cost of feeding more training data into your machine learning pipeline is the computational cost of training it, which is usually tiny fractions of a penny per sample.

<comment><USER5> If it was so simple to collect, process and train with (very) large amounts of data, everyone woudl be doing it. Instead, it's just a few very large companies that can do that, Google, Facebook et al.

Anyway, the cost per example doesn't have to be astronomical. If you need a few millinos of those, you can pay a fraction of a penny and still have a big black hole in your budget, unless you can significantly improve performance.

<comment><USER10> Only a few big companies can do it because there's a bootstrapping problem. To get large amounts of virtually free data, you need lots of users who have signed up for giving you their data in exchange for a useful service. This was much easier to achieve for companies started between 1995-2005, when the web was young, because the Internet was such a huge leap forwards over what came before it. Existing startups now have to compete with the products of these giants, many of which have been enhanced by years of machine learning. That's challenging.

To give you a sense of how cheap computing power is, my startup regularly processes roughly 2B webpages with some complicated algorithms that need to go node-by-node over the whole DOM tree. That's roughly 77TB of (gzipped) data, and around 100 trillion nodes. It costs me a few hundred bucks of AWS time. That's a rounding error for a big corp; a single data scientist's salary for one day will run you around that much.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER11> My limited opinion: The Right Data is better than Big Data, if you can get it.

There was a push to get data out of app dbs and into big data repositories. But then no one could use the big data because it made no sense. So then ML?

But if you already know what it means in the app db, just make it available in a sensible format.

<comment><USER12> I think you can have both situations. I still remember a talk from a Google person maybe around 2010 where he showed clearly that if you try to make the data "better" but in general have way too little data points, then it can't be useful either.

It's just now that we had almost a decade of pushing towards more IoT and more Big Data, that many companies have huge data lakes that they don't know how to make use off.

So instead of applying one of these lessons it's probably best to see where one is lacking (quantity or quality) and work on resolving that specific problem accordingly.

</comment></comment><comment><USER13> See the "Law of large numbers" \- [https://en.wikipedia.org/wiki/Law_of_large_numbers](https://en.wikipedia.org/wiki/Law_of_large_numbers)

Saying you need more data is like saying you need to flip a quarter 500 million times to get a better percentage estimate of heads vs. tails compares to 1 million coin flips. After a certain point, having more data only helps with identifying outliers and changes in behavior over time (when dealing with human/natural data).

<comment><USER14> You're using low-dimensional intuition. If your data has a non-linear relationship between 1000 features (or more), then 500 million samples is quantifiably better than 1 million samples. If these are boolean features, you have 2^1000 possible data points. Your samples are _extremely_ sparse.

The current state of the art of ML assumes nonlinear relationships between all parameters. It can't assume simpler & reasonable models, and therefore it can't extrapolate easily with reduced data.

<comment><USER15> _> The current state of the art of ML assumes nonlinear relationships between all parameters. It can't assume simpler & reasonable models, and therefore it can't extrapolate easily with reduced data._

I'm not really sure what "low-dimensional intuition" means, but I pretty regularly build models that do not "assume nonlinear relationships between all parameters".

<comment><USER16> As the dimension d goes to infinity, if you sample integers from d-multivariate normal distribution, your samples will collect near the unit d-sphere; whereas in low dimensions they'd collect near the origin. This is because as d goes to infinity, your samples become more and more far apart. This is a topological intuition of why higher dimensions don't work like lower dimensions.

<comment><USER17> I reserve the possibility that I am completely mistaken, in which case I apologize in advance, on the condition that you refer me to an actual calculation or derivation (not just a quote in the same spirit) in a paper or text or textbook.

I have seen such insinuation multiple times, that samples "collect" near the unit d-sphere when samples are drawn from the unit d-sphere in high d dimensions.

From a physics perspective this is very familiar to me, but not in the sense of fact, but in the sense of misinterpretation.

I do believe the observation is very useful in the educational sense as long as it is pointed out as being a paradoxical _illusion_. In this sense I can appreciate (even encourage) a professor or a TA showing this phenomenon, on the condition they finish up by explaining why this _seems_ to be the case, but is nevertheless a misinterpretation, and they should make sure the connection with Jacobian determinants etc are made clear.

Consider a normal distribution of any dimension (as high as you want), but I will showcase the phenomenon even with low dimensions (here merely d=3) to illustrate this has nothing to do with high dimensions.

Clearly the probability density is maximal in the center of the distribution.

In computer processing of data points, we typically loop over points, calculate some hopefully interesting function on each sample, and then plot the samples say by binning with equal bin sizes. A programmer typically disregards transformation properties like the Jacobian determinant. Suppose the value we calculate for each data point is the absolute length or distance from the center. The further we go from the center the smaller the probability density of the normal distribution becomes... _but the larger the volume of a shell of radius r!_

Since we are binning with equal bin sizes (equal length intervals per bin), for small lengths, then even though the actual probability density is highest near the center, we will get relatively few samples because the volume under consideration is small compared to the volume under consideration for a shell of a larger radius of equal thickness (area of a sphere grows quadratically with radius). However for even larger distances, the exponential decay of the normal distribution dominates and the number of samples in highest radius bins will decrease again. So in between there will be a peak.

This explains the _fact_ that [ the probability density of [ the absolute distance from the center over [ the sample points ] ] has a peak at some non- zero length.

But it is a conceptual mistake to interpret this as if those sample points in the original d-dimensional space form a dense shell on some "unit sphere" ... This is a pure illusion which illustrates the interpreter is not familiar with jacobian determinants etcetera.

Consider volume or triple integrals over some volume element dx dy dz and for symmetry you prefer integrating in a spherically symmetric coordinate system, theta,phi,r then you can not simply replace dx dy dz with dtheta dphi dr, you need to use dV = dx dy dz = r^2 sin( phi ) dtheta dphi dr.

It is this necessary factor that is ignored when processing sample by sample and causing this illusion in the AI community. I did not follow conventional machine learning courses, but given the learned language used whenever I see statements to the effect of samples lying near the unit sphere in high dimensions, I can only conclude it has its origins in 1) direct observation or experience of plotting in bins of the length of the vector without guidance in interpretation; or 2) guidance during education, where the phenomenon is shown, and the origin of the paradoxical illusion or confusion adequately explained and then the illusory nature subsequently forgotten or 3) a teachers assistant having gone through 2) and showing the phenomenon to students without emphasizing the illusory nature of the misinterpretation.

But perhaps I am wrong, and the normal distribution in high dimensions actually has a higher probability density near its "unit sphere" if the dimension is beyond some critical dimension d_c... but again, I'd like to see a derivation showing it :)

EDIT: For more precise language: they do "collect" (reach a peak) at a certain non-zero length or distance, but they do _not_ collect to a unit sphere of such radius in the original space!

<comment><USER18> It seems you're mostly arguing with the description of "collection", because to you it implies high density in the full n-dimensional space. But the point of the curse of dimensionality is very much that most points do _not_ lie in regions of high density, because regions of low but still non-negligible density are so much larger. If you prefer, you can say that they are "close" to the surface of a sphere, without implying high density.

There are also descriptions of the curse that do not involve spatial analogies at all. Assume that the data is independently identically distributed along each dimension and is an outlier if it's sufficiently far along one dimension, which happens with probability _p_ in the one-dimensional case. Then in _n_ dimensions, the proportion of outliers is 1 - (1 - _p_ )^n -> 1 for _n_ to infinity. Most points are outliers along at least one dimension.

<comment><USER17> I am _not_ arguing with the word "collect", as it can be interpreted in best faith.

I DO argue against the "collect _near the unit d-sphere_ ", if it does not come with an explicit pointer or reference to the explanation of this illusion, I don't care if one points to Maxwell Boltzmann speeds vs velocity vector distribution, or Jacobian determinant, but one should point to _something_ else communication makes no sense. We communicate to teacch and learn. Only when explaining why there _appears_ to be a sphere in the higher dimension, and how this illusion arises is communicating about this pseudosphere justified. The unit d-sphere makes no sense on the 1-dimensional axis of absolute length on which we project the samples. A reference to a "unit d-sphere" only makes sense as residing in the original d-dimensional sample space. But in _that_ space there is absolutely no packing of samples near the peak radius as it _appears_ in the distribution of lengths.

I was not responding to the "curse of dimensionality". I show that this effect already exists at low dimensions, and physicists are very acquainted with it because within their first years of university study they get drilled in 1) jacobians for non-linear coordinate transformations 2) the velocity distribution of molecules in the kinetic theory of gases, where there is a similar plot for the absolute speed (Maxwell-Boltzmann) distribution [0] showing a peak at a non-zero speed, even though the _velocity vector distribution_ is a normal distribution... Every physicist worth his / her salt, immediately recognizes the phenomenon as relating to the Jacobian determinant, that this has nothing to do with velocities aggregating to some sphere of non-zero radius, and that this is a misinterpretation of magnitude distribution plot...

Clearly in this physics example d=3. Would you say d=3 already shows the curse of dimensionality? I call bullocks, and suspect a misinterpretation of the magnitude plot...

Again the peak in the magnitude plot is very real, any reference to a sphere with radius of the peak in the magnitude plot and residing in the original d-dimensional space is purely imaginary!

Of course the person with simultaneously avera height, avg weight, avg income, avg capital, avg age, avg ... is very rare, ... but _less rare_ than a similarily accurately specified person with _non-average_ weight, but all other variables still average...

[0] [https://en.wikipedia.org/wiki/Maxwell- Boltzmann_distribution...](https://en.wikipedia.org/wiki/Maxwell- Boltzmann_distribution#Derivation_and_related_distributions)

see the last 2 sections on velocity vector distribution, and speed distribution...

</comment></comment></comment></comment></comment></comment></comment><comment> rv-de > The results were consistent with asymptotic theory (central limit theorem) > that predicts that more data has diminishing returns.

That's not asymptotic theory or central limit theorem. It's a meek proposition that you learn to prove in first semester when studying math. A bounded and monotonous function will always converge.

> Higher velocity data does not improve percentage accuracy and makes accuracy > levels worse!

What on earth is this supposed to mean? An ML algorithm doesn't care about how fast data "arrives". "Higher velocity" is marketing lingo related to their Kinesis Data Streams service. But I assume that he refers to the usual loss of performance observed with all online-learning algos. Feeding knowledge one by one will mostly just overwrite learned generalizations. Or maybe he is indicating that they are compromising algo quality for faster execution.

> It is important to pick a single metric to improve, even if it is not > perfect, but to use it as the basis for measuring performance improvement.

I think that is wrong. Single metrics never capture complex behavior well and will lead to distortions if fed back to the system. I mean it's normal to use one metric for the error. But saying this is the real deal sounds ridiculous to me as this is being done most of the time anyway and probably is going to change in the future. Backprop only works with one metric at the moment - that's just the fact.

> Pat noted that improvement and learning is often very slow – sort of like a > slow weight loss program, where you lose weight very slowly. Processes may > only be improving by 20 basis points a quarter, or 80 basis points a year. > That isn't a lot, but over a decade, it really makes a difference.

Now he's contradicting himself as he gives a reason for why big data _is_ beneficial.

> His final word of advice – students should be broad in their knowledge of a > lot of things, but need to be very deep in one area.

And again he is contradicting himself. Because if you make an analogy from student to a learning algorithm he now gives TWO orthogonal metrics to optimize for.

<comment><USER19> > What on earth is this supposed to mean? An ML algorithm doesn't care about > how fast data "arrives".

My guess is that he's referring to the resolution of a time series. E.g. Going from months to weeks is 'more' data, but makes your models worse.

<comment><USER9> Good point. If I retrofit the term "resolution" in place of "velocity", the author's statement does make a lot more sense. If that's what was intended they really should have used that terminology, because I was similarly confused at the term "velocity."

Whenever I've worked with time series data I've always referred to the granularity of the time dimension as its "resolution" \- this is also common in geospatial data. I don't think (but am happy to be corrected) that "velocity" is a term of art in timeseries analysis.

<comment><USER19> Ya I was confused by it at first too, it definitely could have been more clear. But it's also been my experience with time series that more resolution isn't always better.

</comment></comment></comment></comment><comment><USER20> Parts of this piece appear to me as evidence for the importance of basic-mid level classes / foundations in statistics, probability, and econometrics for machine learning. Learning to build a model doesn't mean you can always use it well.

<comment><USER21> Indeed. The first one, for example, reads to me as a dead ringer for a case where someone didn't bother to think about ecological validity until _after_ they started having problems in production.

</comment></comment><comment><USER22> How and where data gets scrubbed will have significant consequences. While clean data is more valuable than dirty data (the mass of raw data collected by sensors), there is hidden value in those large masses of data. They can show broad trends and patterns that are not obvious in clean data, which is the modern equivalent of not seeing the forest for the trees [https://semiengineering.com/data-vs- physics/](https://semiengineering.com/data-vs-physics/)

<comment><USER23> This may be a bit elementary for this crowd but regarding the balance of data cost vs capturing the most significant features. We use a simple decision tree as a significance cluster and optimize data munging around these clusters.

On some levels it is anti-diversity but given real world constraints it has yielded the best results. Any thoughts or links regarding this topic would be appreciated.

</comment></comment><comment><USER5> >> A model is now running in production but not producing the same results (or at the same level of accuracy) as what was demonstrated during experimentation… no one knows why

The simplest explanation is that the training sample (meaning the entire dataset; not just the training partition in cross-validation) was not drawn from the same distribution as the distribution of the data that is being processed in production.

Machine learning is guaranteed some performance bounds under PAC learning, assuming that the training dataset and unseen data, on which the training model will be used, come from the same distribution. Absent this, performance cannot be predicted. You might as well classify stuff by throwing a bunch of dice.

Unfortunately, this assumption, that we're representing the real-world distribution in our training dataset, cannot be justified as long as we don't know the ground truth in the real world. Which is most of the time.

Essentially, there's no way to know for sure that a model that has performed very well in experiments will continue to do so once it's deployed in production.

<comment><USER14> I agree with the intuition, but not sure that I agree with the takeaways (in particular the lack of knowledge of the ground truth). I mean it _is_ a problem, but there are bigger, more practical mechanisms at work tool.

In terms of sample distribution - let's say that you have an online model serving traffic (and the outcomes of that traffic are logged as your training & holdout data). Then, when you train a wildly different model, it may start picking other things to serve that the original model never served - <USER3> therefore, there was no training data. This is a pretty fundamental and hard problem.

Second, funny thing, but sometimes you can't use the same metric for training as you do for evaluating your model online. I don't want to (can't) get into details, but it's also a pretty fundamental and hard problem.

Last, the production traffic always shifts. Using historical data, given how much current models are "reactive" and "compression-like", as opposed to true generalization - they perform worse given fresh traffic that changed slightly from e.g. 5 days ago. If your training data is "day -10 to day -3", your holdout is "day -2 to day 0" \- models will likely perform worse on holdout than pure overfit theories would have you assume (mind you, still plenty enough to have a ton of value), but when you launch it on day 1, it will perform worse still - as day 1 is different from days -10 to day -3.

I haven't done the analysis, but I'd assume that for non-historical models where you don't need to structure your holdout data to be from the "future" \- that they'd perform better when you first launch them online.

<comment><USER5> >> In terms of sample distribution - let's say that you have an online model serving traffic (and the outcomes of that traffic are logged as your training & holdout data). Then, when you train a wildly different model, it may start picking other things to serve that the original model never served - <USER3> therefore, there was no training data. This is a pretty fundamental and hard problem.

I think we're talking about the same thing here, that the real world can be very different than your training sample. Sorry, I think I contacted the jargon flu this week :)

</comment></comment></comment><comment><USER24> Statistically they must be right - most of the statistical improvement by an ML system is provided by the first "large" "good" data set, after that gains are sure to be marginal. But, a lot of problems aren't really statistical in nature; they are scientific in that if you can discover a new, better, theory and use that then you can catch a few cases that you may often do the right thing on without knowing why, and sort those out. For example - you may be learning to diagnose a disease with one common driver, there may be a rare disease that responds to the same treatment but has a slightly different presentation, you often treat these patients because of false positives, but if you learn to properly classify them then you can almost always catch them; the statistical gain is negligible, the value is tangible.

<comment><USER24> I should say that above is an example, I am very cautious about using ML for medical diagnosis.

</comment></comment><comment><USER25> After I collect a dataset I save duplicate entries with the text reversed. More data is good data. Except now my machine learning robot is dyslexic.

<comment><USER26> Of course you're joking here, but for images people do reverse the image to enhance the output of their models and this has shown to be beneficial. There are other augmentation you can do as well. In total they're called "Synthetic Data".

<comment><USER16> This is not surprising because if image A is an apple, its mirror image A' is also an apple. Adding A' to your dataset is just plain ol' regularization. It is meant to lower bias, to prevent overfitting.

<comment><USER27> It will also make the flag of Côte d'Ivoire be perceived as that of Ireland.

</comment></comment></comment></comment><comment><USER28> Title was editorialized from "More Data is Not Better and Machine Learning is a Grind…. Just Ask Amazon"

Was it the ellipsis or the reference to the employer of the lecturers? Either way, please don't change titles needlessly. Thanks!

<comment><USER29> Disagree. "...Just ask Amazon" is needless clickbaity extra words. Ultimately the opinion is that of the writer.

<comment><USER30> He was writing a summary of a talk from an Amazon exec. I suppose "per Amazon exec" would have been more exact, but are we just not allowed to have tone anymore? Style is out?

</comment></comment></comment><comment><USER31> Speak not untrue generalizations is better. Not that the opposite is true, by no means. Generalizations are extremely rarely found to be true.

</comment><comment><USER32> As it turns out, more data is a far second to doing anything at all.

</comment><comment><USER33> Diversity in data is better.</comment>


Django with Visual Studio - <AUTHOR> http://www.windowsazure.com/en-us/develop/python/tutorials/django-with-visual-studio/

<comment><USER2> What's so perverse about that? Everybody I know who's tried it claims that VS with PTVS (<http://pytools.codeplex.com/>) is an excellent python dev environment.

edit: original title was editorialized to to claim that doing this was a perversion

<comment><USER3> Are you going to deploy Django on Windows? You would not be wise to do so. So why choose such a profoundly different development environment?

<comment><USER4> Kind of taking some liberties with the definition of the world 'profoundly', aren't you?

If someone works quicker and better in a given dev environment, power to 'em.

<comment><USER5> I don't think "profoundly different" is an overstatement when comparing Windows to anything Linuxy.

If someone works quite a bit better in Windows, OK... but I think it's a far better idea to mirror your deployment environment as closely as possible, not just in staging/testing, but on your development machine as well.

That's not always practical, but working in an environment you should _know_ doesn't even vaguely approximate any deployment scenario means you _will_ miss any platform-specific issue that would have been obvious if you were staying closer to your deployment platform. At best it will be found during testing/staging or by another dev who's using a more appropriate environment, but that's several extra steps and extra time that wouldn't have otherwise been necessary.

<comment><USER6> I'd want to add that working in a windows for dev, linux for deploy environment will eventually make you constrain yourself to using libraries which work on both platforms.

<comment><USER7> Which, unfortunately, is a big gap in Python. We had this developer using Windows who wouldn't want to install Visual Studio. So we tried finding binaries for stuff like LXML, and the newest versions we could find where a good two or three years old. Eventually we hit a wall with one of the packages (I can't recall exactly which one, but one related to cryptography I think.) We couldn't find a binary, and installing Visual Studio + configuring to work from the command line, etc. took way more time than setting up VirtualBox with an Ubuntu machine.

A massive waste of time.

<comment><USER2> I agree that the whole situation with C based python packages on Windows is really quite broken, but until they get around to actually fixing the situation every python programmer who has to touch windows should have <http://www.lfd.uci.edu/~gohlke/pythonlibs/> bookmarked.

</comment></comment></comment></comment></comment></comment></comment><comment><USER8> The true perversion is that all the screen shots have the Solution Explorer docked to the righthand side of the window. It's hard to describe how fundamentally wrong that feels.

<comment><USER9> meh , VS devs get used to that , like the horrible theme colors in the latest VS. I said good for microsoft if they offer developers choice. And it means opensource is kinda winning. But i'm waiting for the day Windows is POSIX to go back to that os , windows is too crappy for Open-source devs even with ming installed...

<comment><USER10> Visual Studio color schemes <http://studiostyl.es/>

</comment></comment></comment><comment><USER11> I think Visual Studio is the best IDE there, but I'm not sure I'd be more productive switching from a _nix based environment just for the IDE. Would anyone whose gone from_ nix to windows for python web development be able to comment on the experience and tradeoffs?

<comment><USER12> IntelliJ is the best IDE :).

Eclipse Python support is lacking.

NetBeans seems to come along pretty well, I just hope that they have more support.

<comment><USER13> Does the community edition support Python?

<comment><USER14> According to this matrix[1], no. I use PHP, Python and Ruby on a day-to-day basis and really need an IDE that can support all of these. Eclipse is flaking out on me now as well.

Anyone know if IDEA really the best multi-language Eclipse alternative, or is there something better?

[1]: [http://www.jetbrains.com/idea/features/editions_comparison_m...](http://www.jetbrains.com/idea/features/editions_comparison_matrix.html)

<comment><USER15> I like it well enough for python, php, javascript, css/html and android/java. Used it once or twice for ruby, but not enough to know how good it was as I have never used ruby otherwise. Intellij is also its own code base, not another fork of eclipse, like most IDEs out there. Intelij also has built in support for git, hg, github/bitbucket, pretty much any SQL dialect including SQLite (as well as SQL syntax checking ERD diagramming, and table editing). Unit testing integration for python, php, javascript, java. Django support (and for templates), Google App Engine (python and java), jQuery support, javadoc support, epydoc/restructured support, phpdoc support, jsdoc support, and some of the popular php frameworks. I could go on, but you can see it pretty much comes with everything one needs (and what one does not need can be disabled from plugins). If it does not come with it, open a feature request and they'll probably add it later.

Other IDEs I have used:

\- Aptana (PHP, html/js, css, python). It's really a fork of eclipse with some additional features. If you want a free IDE, it's better than eclipse, but not quite as good as intellij.

\- Visual Studio (2005, 2010, 2012). Great if you are doing C# or F# and combine it with Jetbrain's Resharper. Not as fond of it for other languages though. I've heard JS support is also very good now, but my web development is all scripting languages like Python and PHP so it does not help me much.

\- NuSphere's PHPed. A good natively compiled IDE for PHP and web, but development kind of lags behind other IDEs and a few bugs that were never fixed when I used it would constantly annoy me. Also, only Windows unless you do WINE (it worked well enough on WINE though).

\- Eclipse. Eclipse interface is clunky and honestly, it's slower than Intellij or any of its derivative cousins. Aptana improves on it though. Used Eclipse for a couple of years before finding Aptana and then moving onto Intellij.

It's not just about the IDE though, jetbrains in general is a great company. Despite English not being their first language for many employees, they have awesome customer service and are very up front about their road map and interact with their users directly.

I've used some commercial IDEs in the past, such as NuSphere's PHPed (which was a decent IDE at the time, not sure how it is now). PHPed's developers did not speak English as their primary language either, but they did not have anywhere close to the same level of interaction and support. Jetbrain's bug tracker is open for anyone to report on and they actually fix stuff you report (filed a few reports over the past few years, all were eventually fixed).

Maybe I'm just getting overly excited over what any decent company should be doing in the first place. However, many IDE vendors do not seem to do that overly well. Having confidence in the developers of your IDE is important (at least to me) and that they are listening to their customers/users.

</comment></comment></comment></comment></comment><comment><USER16> I'm a .NET dev and I literally just gave this a try last week. I've been toying with Python on my Ubuntu laptop but I really wanted to see how this worked on Windows too. Visual Studio is by far my favorite IDE and being able to write Python in VS was pretty amazing!

</comment><comment> X-Istence Why are Visual Studio's menu's yelling at me?

<comment><USER17> VS2012 did a lot of visual changes, many of them controversial. The guise was to make it "more metro" despite Windows 8 not taking the same approach (like all caps).

The VS 2012 betas tried putting all caps on various parts of the UI. They got tons of pushback on that, so they eventually just made it the menu so they could still claim they kept the principle idea alive. In the final release, there's an option to turn it off.

A lot of VS devs are happy that they added a proper dark mode, and there's a plugin to autohide the menus so it's less of an issue that it appears at first. The biggest usability issue is the single-colour icons.

There's probably amazing internal threads and all sorts of political issues surrounding the design. In the end, I think the champions of the change count on wearing you down, making their opponents feel silly for arguing over such petty things as menu styles.

</comment></comment><comment><USER18> "To complete this tutorial, you need a Windows Azure account"

yeah, right.

<comment><USER19> You only need an azure account if you want to deploy your app, the last step. If you just want to run locally and check out python in VS, you don't need an account.

</comment></comment><comment><USER20> I'm curious to know how VS compares to PyCharm in terms of how rapidly the Python environment is able to restart after a code refresh.

It's been my personal experience that PyCharm's Run/Debug configuration is fast to restart in Run mode, but considerably slower in Debug mode, taking 5-10 seconds to restart and attach the debugger. So while I'd like to be able to work with breakpoints and inspection always available, in practice those tools have been harmful to my productivity.

If VS is better or faster in this regard, it might be worth a try.

<comment> dav-id In my personal experience with working in .NET MVC within VS 2012 it is noticeably slower working with Debug always on as you cannot simply build and refresh the code, each time you need to load the debugger, test drop out of debug, fix, repeat.

I prefer to do as much without debugger and then when I need to inspect variables and step through code drop into debug mode.

With Python this may be different but I doubt it. I expect you will have to wait while the debugger loads up. If you can edit code whilst debugger is running then I can see that being a big plus and a reason to keep it on.

</comment></comment><comment><USER21> Running a Hello World is easy

Maybe they got it to be easier in Azure, still, here's a couple of things they didn't show how to do and is usually a pain:

\- serving static resources (easier with newer Django versions, still)

\- serving in a subaddress (like serving it from mycloud.com/myapp)

\- Using MSSQL with it (there are Django adaptors, but last time I tried there were some issues)

</comment><comment><AUTHOR> Someone changed the title. Any reason for that apart of some people not agreeing with my world views?

<comment><USER22> Sensationalism is a type of editorial bias in mass media in which events and topics in news stories and pieces are over-hyped to increase viewership or readership numbers. [..] Some tactics include being deliberately obtuse, appealing to emotions, being controversial, intentionally omitting facts and information, being loud, self-centered and acting to obtain attention. Trivial information and events are sometimes misrepresented and exaggerated as important or significant, and often includes stories about the actions of individuals and small groups of people, the content of which is often insignificant and irrelevant relative to the macro-level day-to-day events that occur globally.

<http://en.wikipedia.org/wiki/Sensationalism>

</comment></comment><comment><USER23> Just go download Komodo.

I regularly use Komodo Edit (100% Free) and it is pretty good, the IDE is even better if you want a more involved tool. This is probably my favourite for a "windows like" experience.

</comment><comment><USER24> To think that people use an IDE for Python is sad news for me.

<comment><USER25> Real programmers use cat.

<comment><USER26> Real Real programmers use echo

<comment><USER27> real programmers use butterflies

</comment></comment></comment></comment><comment><USER28> Coming Soon - How to develop Ruby on Rails apps on an Etch-a-sketch

<comment><USER28> I knew this was going to get downvotes, but come on, doesn't this seem to anyone else just a little bit desperate on Microsoft's part?

They're giving out bad advice to sell bad tools. I'm a little disheartened it's got so much attention on here.

<comment><USER29> Have you used VS? What's your basis for dismissing it as a "bad tool"?

<comment><USER28> Yes, I spent a year using VS full-time for .NET development, and I think it's a ridiculously contrived way of developing Django applications.

Hence the promotional 'articles' like this from Microsoft.

<comment><USER30> My goodness, a whole year.

People bitch and moan about Microsoft being closed off from the rest of the development community, then when they offer the rest of the development community a way to use VS to write in their language of choice, they get this crap.

A lot of what Microsoft does is shit, but the constant and pointless anti-MS circle jerking is ridiculous.

<comment><USER28> Grow up. I'm not bitching and moaning, nor am I anti-MS. I told you I worked with MS languages.

I just think this is a crap way of doing Django. Just like developing Silverlight apps in VIM would be.</comment></comment></comment></comment></comment></comment>


HOList: An Environment for Machine Learning of Higher-Order Theorem Proving - <AUTHOR> https://arxiv.org/abs/1904.03241 <comment><USER2> The title of the article this points to seems to be very different from the title of this submission ... Probably pointing to wrong paper?

<comment><USER3> This seems to be the correct paper: [https://arxiv.org/abs/1904.03241](https://arxiv.org/abs/1904.03241)

<comment><USER4> Changed to that from [https://arxiv.org/abs/1904.0324](https://arxiv.org/abs/1904.0324). Thanks!</comment></comment></comment>


Intelligence Startup Goes Behind Enemy Lines to Get Ahead of Hackers - <AUTHOR> http://www.nytimes.com/2015/09/14/technology/intelligence-start-up-goes-behind-enemy-lines-to-get-ahead-of-hackers.html?_r=0 <comment><USER2> Should this be a government function? It seems that it could easily involve intrusions into others' computer systems ... And if that's allowed for a private company, for this reason, then it can be allowed for other reasons.</comment>


Isaac Asimov on How to Be Prolific - <AUTHOR> http://briangroat.com/2016/10/26/asimov-on-how-to-be-prolific/ <comment><USER2> I must say "Thank you". Thank you, Asimov, for my language skills.

I started reading Asimov in my native language (Latvian) until I run out of it. It was late 80s. Then I had to work on my Russian reading skills in order to read everything else USSR had to provide from Asimov. Then I read everything libraries in Riga had on sci-fi.

And when 90s arrived, I accidentally got a hold of few Asimov books in English. I remember vividly the nights I spent with his book on the left side and English dictionary on the right one. Underlining and writing words and page numbers of words I coulnd't find just to return to them later when context would be much cleaner. Ever so funny struggle with idioms and cultural references I couldn't possibly understand. There was no internet to look that stuff up.

So, yes. Thank you, sci-fi in general and Asimov in particular for motivation and opportunity to learn one of basic life skills - ability to read and understand foreign language better than school could teach.

<comment><USER3> > I remember vividly the nights I spent with his book on the left side and > English dictionary on the right one. Underlining and writing words and page > numbers of words I coulnd't find just to return to them later when context > would be much cleaner. Ever so funny struggle with idioms and cultural > references I couldn't possibly understand. There was no internet to look > that stuff up.

I salute your perseverance and skill - that's incredible.

We take for granted how easily we now obtain the information and literature we want today (tailored to our preferences at that!) thanks to the internet.

In some ways I feel we had to use our brains more before technology and data became so universal and cheap!

<comment><USER4> > _In some ways I feel we had to use our brains more before technology and > data became so universal and cheap!_

Maybe. Funny thing is, we have more opportunity to use our brains better now, and yet it seems we're not taking it.

For instance, books. Many people say that they don't read much books anymore - even though today you have access to much _better_ selection of books than before. Issues like being an adult = lots of bullshit errands aside, it's easy to be distracted by blogs and other short-form texts today. I have to constantly remind myself to keep reading actual books. Today you can literally just Google "best book on X", browse several discussion threads and pick out what is possibly the best resource whole humanity has on topic X. It's that easy. But for some reason, it's not an obvious thing to do. It wasn't for me until recently.

</comment></comment></comment><comment><USER5> It's interesting that Asimov offered advice that suggests writing without concern for whether everything fit together; just getting the first draft out was the goal. This somewhat conflicts with Asimov's own style of writing (which he talked about in at least one of his three memoirs, as well as other places), which was effectively to write the final draft the first time. He was not prone to heavy editing after writing, and was also not prone to editing while writing. So, he was basically just writing and handing it off to his editor/publisher (and I think it's also been established that his editors also had a soft touch with his work).

So, maybe he was just being kind enough to not hold everyone to his own standards. And, maybe he developed his skill for writing without a net through tremendous amounts of practice (more practice than almost every other human who ever lived...he's among the most prolific book authors the world has seen).

Anyway, I read everything about Asimov I come across. But, this is kinda blogspammy. Asimov's own memoirs are worth a read. I have not read Janet Asimov's books about Asimov, yet, but surely will eventually.

Also, this site breaks the back button for me in Firefox. That's gross, and I literally hate every person who's ever intentionally broken basic browser functionality for profit, ego, or whatever reason they thought made it OK.

<comment><USER6> I agree the site is blogspammy, but is the back button a Firefox issue? I tried the site in Safari & Chrome as well and there's no issues with the back button there, only in Firefox.

<comment><USER5> The back button in Firefox works fine on sites that don't fuck with the back button. So...no, I don't think we can reasonably blame Firefox.

Firefox has a reasonable implementation of the History API, so it isn't that Firefox doesn't support some feature that a single page app would need in order to make stuff like URLs work right. And, it's not like Firefox is some obscure browser with 1% of market share.

</comment></comment></comment><comment><USER7> Something to keep in mind is that Asimov was very much a product of the age of pulp magazines, so early on in his career he was paid by the word for his work.

A side geek note: If you were active in NYC fandom from say the 50s on chances were very good that you would actually get to see Isaac in person at many local science fiction conventions. For some reason he disliked air travel, and would show up at almost any local convention that he was invited to.

If you want to read a really good book on being a geek from the depression era to about the 50s I would highly recommend reading The Way the Future Was by rederik Pohl.

<comment><USER8> He had fear of flying, I remember reading that in one of his autobiographical books. He was aware of that and wrote a lot about space travel, on the cave of steel he depicted humans as being agarophobic

<comment><USER9> That's "agoraphobic".

</comment></comment></comment><comment><USER10> A lot of us are prolific makers, we're just not prolific sharers. I've amassed a ton of bitbucket repos the world has never seen. Now, I'm learning to share my work and focus on marketing in conjunction with development.

I wrote a piece last night about it:

[https://blog.oldgeekjobs.com/be-a-prolific- sharer-31bfdfb50a...](https://blog.oldgeekjobs.com/be-a-prolific- sharer-31bfdfb50a40)

It's about not being afraid to share the things you make and tuning them until they resonate with your audience.

<comment><USER11> That's kind of funny you should say that. A book I'm currently reading advocates that our best writers of today's day and age won't be those on the internet that share their work easily but instead hidden away, released to small groups of audiences. One such forum referenced was TOR and Darknet.

<comment><USER10> That sounds interesting. How does the book argue that quality of work correlates with availability and audience size? What book is it?

<comment><USER12> It's a small book for a select audience :)

<comment><USER13> I think hes talking about "But What If We're Wrong?" by Chuck Klosterman. Sorry to spoil the secret!

edit: Its highly recommended. Also, its not an argument about quality. The author speculates about what kind of art produced in the present day that people in the future will value.

<comment><USER11> Yep, that's the book! Good find.

</comment></comment></comment></comment></comment></comment><comment><USER14> With regards to mortality as a motivator, I often find the opposite to be true. When I am reminded of death or illness, I usually end up falling into a spiral of procrastination -- anything to avoid the thought of oblivion. I find it to be quite a major source of time wastage and I suspect I'm not alone in this.

When Asimov was unwell following bypass surgery, James Randi wrote that Asimov fell into a depression and gave up writing altogether [1], presumably for the same reason.

[1] [http://www.skepticfiles.org/atheist/asimovob.htm](http://www.skepticfiles.org/atheist/asimovob.htm)

<comment><USER14> I'd be interested to know if anyone has advice on how to break out of these thought patterns.

<comment><USER15> Set goals and go about pursuing them. It requires discipline but once you surpass the initial blocks it becomes second nature. Also, don't forget of yourself. Do some form of physical activity and set some goals for that too. Rinse and repeat.

<comment><USER16> Regarding goals, I've personally found them to be counterproductive unless they're simply achieved and a matter of habit. For instance, "go to the gym and do nothing" became "go to the gym and walk on the treadmill" when I became habituated to showing up at the gym. Now I'm lifting regularly and I feel off if I don't get a chance to go work out.

Audacious goals, on the other hand, are demotivating because there's too much chance of failure. If I hold myself to lifting certain weights, I may even end up getting injured. But now I'm happy simply showing up, doing what I have the capacity for, whether it's walking, stretching, or lifting. I can (and will) come back if I have an off day.

Scott Adams talks about something similar in Goals vs Systems[1]. A goal of writing a book is much harder to achieve than a goal of typing a little bit in the morning. Ironically, the latter may be the best route to the former...as long as you don't think about it too much :)

[1]: [http://blog.dilbert.com/post/102964992706/goals-vs- systems](http://blog.dilbert.com/post/102964992706/goals-vs-systems)

</comment></comment></comment></comment><comment><USER17> I stand by the idea proposed by Cal Newport that the best way to do great work is to do "Deep Work"—large (but not all day) amounts of no-distraction, in- the-zone work. Harder to do than ever, but when I can do it I'm amazed.

<comment><USER18> Deep work isn't a new concept. I just finished Cal's book and, while it's a decent read, I didn't learn anything.

Maybe because I read a lot of productivity-based articles and blog posts?

<comment><USER17> How about trying to actually do it? I agree self-help books are mostly fluff that can be said in half the length, but I do think "Deep Work" is a useful technique.

One problem I did have with the book was that Deep Work might not be useful for someone whose work is largely mechanical or mundane, though in that case I'd apply Deep Work to learn skills to into more interesting jobs if one wishes.

</comment></comment></comment><comment><USER19> FTA:-

      "So what if it limps. Its purpose is to get you into the next stage of the
      story and you take off from there. Time enough when you go through the novel
      again to correct the transition. For all you know, the material that you will
      write much later in the novel will make it plain to you exactly how the
      transition ought to have been. No amount of rewriting and repolishing now will
      get it right in the absence of knowledge of the course of the entire book. So
      let it limp and get on with it … Think of yourself as an artist making a sketch
      to get the composition clear in his mind, the blocks of color, the balance, and
      the rest. With that done, you can worry about the fine points."
      – Asimov's advice, in a letter, about 'limping transitions' in first drafts

This would seem to be good advice in any creative endeavour and particularly relevant to startup culture.

</comment><comment><USER20> His older novels such as the later Foundation series were crap. Fad idea of the day (Gaia) instead of exploring a topic a bit more in depth (advantages of large populations in combating stasis). His best work was his early short stories (three laws of robotics, ultimate question). IMHO, his quality gradually went down before falling off a cliff. Traded quantity for quality?

<comment><USER21> I'm as big an Asimov fan as anyone, but I generally agree that his later novels were weaker. He set himself the challenge of merging the Robot universe with the Foundation universe and I'm not sure it was a good idea. I have always said that his short stories were his best work. I recently heard the radio adaptation of "Hostess" and was reminded was a great story that is.

I was a fan of his science essays before I read his fiction. I taught myself physics via those essays before I learned it in High School. But even more importantly, I learned how to think like a scientist. I owe him a huge debt of gratitude for that, which, fortunately, I was able to express to him once in person.

<comment><USER22> Yes, although he wrote over 400 books, most of the good ones were in the first 100.[1] There were a lot of junk books in the later years. That happens to working writers.

At least Asimov didn't get stuck writing movie tie-ins. Some good SF writers have had to write movie tie-in novels to pay the rent. Alan Dean Foster, who wrote some excellent novels in his own universes, has ground out too many Star Dreck, Star Wars, Alien, Terminator, and Transformers tie-ins.

[1] [http://www.asimovonline.com/oldsite/asimov_titles.html](http://www.asimovonline.com/oldsite/asimov_titles.html)

</comment></comment></comment><comment><USER23> 90,000 letters is equal to 4 a day for 61 years. That's a superhuman output

<comment><USER24> It's amazing what people can get up to if they're not being distracted by the internet or other popular media.

<comment><USER25> That's just an excuse. Most people back in the good old days didn't get up to much either.

<comment><USER24> Most people, sure. But they weren't going to do great works anyways.

It's the exceptional ones are more likely these days to be eaten up by transitory media along with everyone else.

But who knows. It's all speculation from me. There's no ethical way to run the experiment.

</comment></comment></comment></comment><comment><USER26> Cached version here: [http://webcache.googleusercontent.com/search?q=cache:VMbrnBW...](http://webcache.googleusercontent.com/search?q=cache:VMbrnBWdtncJ:briangroat.com/2016/10/26/asimov- on-how-to-be-prolific/+&cd=1&hl=en&ct=clnk&gl=us)

</comment><comment><USER27> Issac Asimov's later works are known to have been ghostwritten. Hiring other writers to write under your byline is the easiest way to be prolific.

</comment><comment><USER28> The "Candy Store" this article refers to (and Asimov in the quote) is in my neighborhood. Hilariously I discovered this through Pokémon Go.

<comment><USER28> Its no longer a candy store of course.

</comment></comment><comment><USER29> > expanding a short story about a collapsing universe, and expand it into a > novel.

What story is this? When I google it I get non-fiction links.

<comment><USER30> I believe it's "The Gods Themselves", though Wikipedia doesn't have anything about it starting as a short story (at least skimming on my phone)

<comment><USER31> I was thinking that's exactly what it sounded like, which is my hands-down favourite Asimov book.

<comment><USER32> That's because it's the _best_ Asimov book. :P

<comment><USER33> Pfff, The End of Eternity is obviously the best Asimov book :)

edit: and probably the best time travel novel as well

</comment></comment></comment></comment></comment><comment><USER34> I can't quite put my finger on why, but I suspect that this guy wants people to subscribe to his newsletter.

<comment><USER35> I'll pass on the newsletter signup. His writing is less than stellar; there are all kinds of fragment sentences which make the article difficult to read.

<comment><USER36> Maybe his other stuff isn't as bad.

... Nope. No other stuff. This is literally the only thing on the site and there are three "Join now!" boxes.

<comment><USER37> Based on the topic of his first post, I am sure we can expect many, many more to come.

</comment></comment></comment></comment><comment><USER38> step one: grope everyone you see

<comment><USER39> You're thinking of Arthur C. Clarke (pedo).

<comment><USER38> thats wrong. but asimov was a notorious lech at conventions

<comment><USER39> In his own words:

"I have never had the slightest interest in children – boys or girls. They should be treated in the same way. But once they have reached the age of puberty, then it is OK," Mr Clarke was quoted as saying in the Sunday Mirror. "If the kids enjoy it and don't mind it doesn't do any harm … there is a hysteria about the whole thing in the West."

Mr Clarke subsequently denied he was a paedophile, saying: "The allegations are wholly denied." But he never sued the Sunday Mirror and died aged 90 at his Sri Lanka home in 2008.

Source: [http://www.independent.co.uk/news/media/press/notw-editor- sp...](http://www.independent.co.uk/news/media/press/notw-editor-spiked- paedophilia-scoop-on-arthur-c-clarke-for-fear-of-murdoch-7920816.html)

"I am trying to think of the youngest boy I have ever had because, of course, you can't tell it here. I think most of the damage comes from the fuss made by hysterical parents afterwards. If the kids don't mind, fair enough,'' he was reported to have said in the interview which was conducted at his house.</comment></comment></comment></comment>


Show HN: Graph Your Twitter Network with Neo4j, Powered by Docker - <AUTHOR> http://network.graphdemos.com/ <comment><USER2> You don't need the two follows pointers. That's an anti-pattern and solved by the right query syntax.

<comment><USER3> That only applies when the relationship implies bi-directionality. On Twitter, me following you doesn't necessarily mean that you follow me. Thus both directions are persisted in the model.

If this were on Facebook, however, you would be right. (:User)-[:FRIENDS_WITH]-(:User) would only need to be persisted once and with an arbitrary direction.

</comment></comment><comment><USER4> Couldn't load the Neo4j instance. Refreshed 3x. Latest Chrome + uBlock Origin + Privacy Badger.

<comment><USER4> disabled those extensions. ctrl+f5 refresh. still not loading.

<comment><AUTHOR> Sorry, we hit EC2 instance limits. Now fixed. Try again :-)</comment></comment></comment>


FARK has gone white to "support" SOPA/PIPA - <AUTHOR> http://www.fark.com/

<comment><USER2> Aside from the humor of this, Fark probably has the best annotated bibliography of links to check out. A number of other sites have more links, but Fark's are actually curated and more worthwhile overall.

</comment><comment><USER3> [TheDailyWTF.com](<http://thedailywtf.com/>) does something similar.

<comment><USER4> I believe the traditional reply is: SOPA is the real wtf.

</comment></comment><comment><USER5> Truly awesome. While I've fallen out of visiting Fark every day (which happens when you're not on salary and instead self-employed) this hilarious stunt makes me want to add it back to my daily slack rotation of websites.

</comment><comment><USER6> Okay, I lol'd. They got me!

</comment><comment><USER7> I hope they can claim Fair Use of the soundbyte from The Price Is Right!</comment>


Ask HN: How Do You Find/Curate Academic Papers? - <AUTHOR> After discovering Sci-Hub I am excitedly diving back into reading academic papers. Fellow HN denizens, what do you use for finding papers that interest you?
I have used sciencedaily.com for general news that sources directly from articles and journals, is there anything similar around for other areas, or niches, maybe something like a website or newsletter with articles of interest? For instance I enjoy reading research papers around political science and international relations, but also would love to explore things around UI/UX.
Pray tell, what do you use? <comment><USER2> ACM digital library is great to browse (and search) [1], its free at the moment - they have SIGs(Special Interest groups) for currently active research areas - SIGCHI might be the one you're after for UI/UX [2]

In my brief stint in molecular biology I found review articles great for an overview of a topic, though I haven't seen these type of articles as much in CS.

[1] [https://dl.acm.org](https://dl.acm.org)

[2] [https://www.acm.org/special-interest- groups/sigs/sigchi](https://www.acm.org/special-interest-groups/sigs/sigchi)

<comment><AUTHOR> This looks fantastic and exactly what I am looking for, thank you!

</comment></comment><comment><AUTHOR> I spent the afternoon working it out myself and cataloging places to find papers, a combination of the following works amazingly well.

This chrome extension: [https://github.com/allanino/sci-hub- fy](https://github.com/allanino/sci-hub-fy)

Track and catalog what I have read (à la GoodReads): [https://www.zotero.org/](https://www.zotero.org/)

and Zoteros chrome extension for easy logging (works from within a PDF too): [https://chrome.google.com/webstore/detail/zotero- connector/e...](https://chrome.google.com/webstore/detail/zotero- connector/ekhagklcjbdpajgpjgmbionohlpdbjgc)

which I sync to the web library (must create a free account and login to see) rather than using the desktop client: [https://www.zotero.org/user/login/](https://www.zotero.org/user/login/)

</comment><comment><USER3> [https://www.meta.org/](https://www.meta.org/)

<comment><AUTHOR> That's awesome! Is there anything like 'meta' but more general? Or for other fields?</comment></comment>


Ask HN: Slow engineers, did you get faster? How? - <AUTHOR> After four years, I left my first team in March to join a new team at the same company. Now it's performance review season, so my old manager forwarded some notes to my current manager about my prior work. Basically it said that I had solid results, wrote some really stable code, "championed" testing and testability for the team, and that people liked working with me. I haven't seen it yet, but apparently it was a lengthy essay.
But he also added that the one thing that had been keeping me from moving from SDE I to SDE II was my velocity at turning out deliverables. I'm basically the last one of my peers who hasn't made the jump, and that was the ongoing reason from my old manager. I'm getting passed by newer engineers pretty regularly now, and I've been at my current level for about three years.
In the last couple weeks since my new manager got that letter, she's seemed really concerned about my velocity. It wasn't until this Friday that she had read anything about it. Yet in the same time span, the new-guy-on-a-new-team feeling was pretty much gone, and productivity was up. I got put on a pretty high-visibility, high-impact feature for this month. Before, I think my manager saw me as ready. Now I think she sees me as a liability.
I was starting to get worried that I'd never meet this speed expectation at my old job; but now I'm extra nervous that if I slip on any deadline even a little bit, I'll seal this reputation with my new team and make it my label from here on out. The Slow Guy. Sure he's nice, and he writes maintainable, reliable code... but if we need it done soon, might as well give it to an intern!
Q: Did anyone else shake a reputation for being slow? Is there any hope I can do to put this velocity thing to rest and catch up with my peers? Or am I just a B engineer who peaked right out of college?
PS: I'm getting married next month. Will I have to choose between my marriage and permanent 12 hour days? <comment><USER2> I would echo the perfection/artist thing, I've known a lot of engineers and ones who are perceived as slow, are often also setting a really high bar for their code. The learning for those engineers is to develop an understanding of how good is necessary, and how good is ideal.

If you find yourself rewriting your code a lot, consider using a technique that Pixar uses for making digital films which is they work on them on sections until they are "good enough" and they put that part aside with the label "Good enough, could use improvement", until they get everything in the can, now the film is completely "done" but all of the sections can use improvement. They order those sections from most impact to least and start working through them. At any point they could just print the film and ship it, but the longer they work the better it will get. Often there are other parts of the pipeline that have to be aligned and so they will get a chance to knock off a bunch of improvements before it actually ships.

<comment><USER3> I generally agree that it is important for a software engineer to be able to identify the quality/speed desired by the client/manager. However, I think this advice needs to be applied cautiously:

> If you find yourself rewriting your code a lot, consider using a technique > that Pixar uses for making digital films which is they work on them on > sections until they are "good enough"

This approach works for features where side-effects are isolated to a small part of the system (eg. UI changes). For anything major, it is worth doing right the first time. eg. A library on which other software is constructed. The mastery is in weighing up the scope and likelihood of rework that might be required in the future.

</comment></comment><comment><USER4> I've only seen advice so far on how you can "get faster", but I believe the real issue here is that you are being _perceived_ as being slow by a manager who might not know the value of what you are doing.

I had a manager tell me I was slow at one of my last jobs as well.

A few months later, I was having trouble convincing her to give me a promotion/raise. I knew I was doing the same amount of work as everyone else, if not more, and I was fixing a lot of bugs caused by developers 2 or 3 levels above me. Our 1 on 1 meetings consisted of me listing the reasons why I deserved a raise, and my manager telling me no, but never a good reason why. She even read me the guidelines for what the company looks for in engineers at each level, and I was doing all of it and more, but still no raise.

I ended up quitting (not just because of the pay), and taking my skills to an awesome new job where I was considered a top engineer, and I'm really proud of everything I got done there. The new company gave me a raise when I asked for it because they valued me more than my previous company.

My personal advice is to look for a new job where you can apply your skills where the company needs them, and make a good impression from the beginning. The added bonus being that you'll be able to negotiate a higher salary with the new job as well.

<comment><USER5> Indeed; one of the very most frustrating jobs I ever had, my last at a Lisp company, ended up "pitting" me against a very junior programmer, who was a flurry of obvious activity, including lots of debugging. I put in less hours (but still a lot, it was a pre-launch startup), often took a nap in the afternoon, was much quieter in a way, only asking questions about the code base and e.g. asking for one procedure to be coded in assembler to make the editor we were working on fast enough, etc.

My output of _working_ code and functionality was a _lot_ higher, I removed entropy from the code base (the other guy at best didn't add a lot), and ended up being demoted and then quitting in disgust. In vague mitigation we were reporting to the 3 founders and friends who'd all gone to the same school and this was their first attempt to manage others, but....

I don't know if your situation is akin, but it does sound like there's, realistic or not, an unbridgeable gap between you and your manager(s), and its time to find another job with a better fit before the company takes worse actions.

</comment></comment><comment><USER6> First, never sacrifice personal satisfaction and happiness for a job. It will never work out well in the end and you will regret it later in life. Jobs come and go (there is very little company loyalty in either direction now a days), family shouldn't.

Second, I will admit to being a douche earlier in my career when managing projects and wanting faster production. What changed it for me was an older engineer who was SLOW; he made no apologies for it and kept pointing out to me that yes, he took more time than many others. But many of them also had to come back 4-5 times to fix issues in the same code while in his the amount of rework was minimal if at all. So my perception was he was slower, but reality was he was faster because in the end he spent fewer man hours on any given problem.

Once I learned that, mostly thanks to him helping show me, I changed how I viewed everyones productivity. His approach was not defensive, not belittling, not condescending and not ego driven. Instead he approached me and said he understood my concerns but thought it would help me to understand the man hours being spent versus just the deadline. Not that the deadlines weren't important, but if you make a deadline and have to rework code constantly for the next 2 weeks to fix preventable errors then did you really meet the deadline?

So your answer out of this, is try to change the perception from your slow, to you spend less man hours (time doing rework) being more productive and writing better code. It won't always work, but frankly if it doesn't that isn't a team or company you want to be apart of. If they value speed over reliability and more man hours over fewer then they will just work people to death and treat them as disposable. One other point, if the team or company is a point, shoot, aim type this argument won't work and they will only value speed not accuracy or reliability. In that case, look for other work or another team within the company that has different values.

</comment><comment><USER7> This might be a sign that you have a tendency towards perfectionism. This is not the worst problem you can have and is manageable.

I suggest working on new features as WIP pull requests and have others review the code when it's feature complete. Not when you deem it ready. Then you can gage from others where you are spending unnecessarily much time, using quick learning cycles that will teach you to be faster without sacrificing quality to a large degree.

</comment><comment> a-saleh In bullet points because I have almost the same problem. Here is what helped me:

1\. talking with my manager more

We have perf reviews every 3 months, and the review has 1-2 month delay. I recieved in spring a report that my perf in winter was mediocre. And then I recieved in summer that my perf in spring is worse. And I didn't think it detiorated, so I came to him and asked "Hey, how did it get worse past quarter, because I thought I am getting better at this." And he told me about concrete project that he thought shold have taken a week or two and took month. And that after I finally finished it, the rest of my work-pace this quarter looks better.

2\. identify the problem with your manager

For me it was, that I am able to get stuck on a task for a several just reading all the error logs without asking others for help. I know that some other team-mates had similar problem, so we introduced daily standups and more over I do my best to track whether I am trying to get unstuck more than one hour and if so, ask around for help.

3\. I kinda embraced it

Everybody on the team knows that I am the one with wife and a kind that gets stuck on his work often :-) So, on meetings I allways ask after I have new task "Who should I bug if I get stuck?" and they know, that I am at work at 8 am sharp, but hell or high water at 5 pm I am already playing at home with my 1y old :-)

So, just talk about this with your manager. And try to fix your biggest pain points (like I try to to with me getting stuck :)

</comment><comment><USER8> Every hear of "fast, good, cheap: pick two" ? [https://en.wikipedia.org/wiki/Project_triangle](https://en.wikipedia.org/wiki/Project_triangle)

You manager won't tell you directly, but they're asking you for crappier code, faster. Give them what they want but also cover yourself by sprinkling the codebase with TODOs. And in your updates on tasks when you're done mention areas for future work not yet done.

Most developers who churn out code tend to churn out bad code.

Also I hate to say it but politics and image is a large part of any job. My honest advice would just be to find a new job where you're appreciated.

</comment><comment><USER9> Are you actually slower than coworkers in an apples to apples comparison? Normally when I see things like this, it's usually comparing developers having a different definition of "complete". One will write the code, the unit tests, the e2e tests, and the documentation before saying the task it complete. The other, trumpets completion once the code is done.

If you're actually slower, I'd start keeping track of where my time is going. Are you spending too long imagining all possible scenarios of a feature? Taking two hour lunches and coffee breaks every hour? Many hours spent in meetings? Wasting time building a "super generic" solution that isn't necessary? Spending three days writing tests for a feature that took 5 hours?

If you don't feel like you're actually slow, it's just a problem of perception. I'd take some higher visibility features and crank them out, working the extra hours if necessary. I'd rather kick the stigma than deal with that anchor around my neck. Tread carefully with this strategy as it can be a double-edged sword and increase expectations and workload.

</comment><comment><USER10> There are definitely ways to get faster. As long as you are paying attention and care about your work, you'll naturally get faster as your career progresses. But making a concerted effort pays off as well.

One thing that would eat up a lot of my time is trying to automate every last thing, and make it reproducible. I would do it with Python. I still believe in this goal, but I do things with shell and a smattering of Python now, which is way faster and actually more elegant. Shell is an unbelievably productive tool! Though it takes time and effort to learn properly.

Another thing I do is write down everything I plan do in a Wiki -- down to a very granular level. Always keep the end goal in mind when writing out the steps. It helps think of short hacky strategy first, in addition to the "right" strategy. Frequently the shortcut will give you some new information that changes the "right" strategy, or it might be good enough by itself.

Since it sounds like you have the tendency to do things right (like me), you won't fall into the trap of shipping sloppy and broken code like others (i.e. the state of the software industry). This is a trick to balance your natural tendency. You want to learn about "unknowns" earlier to reduce risk.

The third thing major is unit testing, though it sounds like you're already doing that. For me, it makes things faster, because I can switch between many projects at once. I think of it like putting state in your brain down into executable code. It takes me very little time to pick up where I left off on old code now, whereas I see a lot of people struggling to load the context into the brains. However, I also tend to write tests in shell now, which is way faster than lots of complicated unit test frameworks.

<comment><USER11> Great advice here! One thing (slightly OT) - what do you use for unit testing in bash? That sounds like it could be really cool, though I currently write my tests in Coffee so I'm a bit skeptical as to the productivity gains.

<comment><USER12> Technically couldn't you just write a separate bash script that tests the exit status or return of anything you are scripting in bash? Feels like bringing in any other "framework" would possibly just get in the way?

<comment><USER11> I think the GP was talking about tests written in Bash, not tests for Bash code.

</comment></comment></comment></comment><comment><USER13> Were the deliverables fixed by the management or you were responsible for the estimate? If you are the one who consistently underestimates the time, maybe that's a legitimate point.

Ultimately, it is stupid if they'd favor giving a project to an intern over you despite your results/writing maintainable code etc. The company will pay for shit code accrued over time.

</comment><comment><USER14> I don't know enough to truly speak on this, but it honestly sounds to me as if you are someone who is very rational and thinks before acting. If this is a detriment to your employer...I don't know what to tell you.

</comment><comment><USER15> Slow guy here.

First thing to check: do you want to make it more beautifull than you should? Sounds to me you want to release good products. But in my experience this can also lead to scope creep.

Then there is stable code. I also am one of the slowest engeneers but my manager knows this will result in less work after the deadline.

So my advice: stick to what should be done to keep the deadline and prove your code saves time after the deadline.

</comment><comment><USER16> It's not good to work for people who see you as a liability. Consider working for someone else.

</comment><comment><USER17> If you are working 12 hour days you might want to get out of there, unless if you are making 150k then stay. If you are making around $80k ish then run.

Another option is if you are making enough, you might want to outsource some of your work. Get multiple people working on any tricky problems so you can tie up the loose ends.

</comment><comment><USER18> The horrible truth is that regularly writing crap code that bursts into flames and then making flashy, heroic efforts to put those fires out is the fast track to the top. Taking the time to produce quality artifacts feels really good, but it is a rare company indeed that can recognize the value in this.

I'll echo other commenters and say that if you have this stink on you and/or you're company isn't appreciating your work, it's probably time to move on (esp after four years). You can't fix companies--all you can do is find a better match.

As for myself, I'm a slow engineer. In my decades, if anything I've gotten somewhat slower. At the same time, my quality has risen dramatically. I write bugs rarely enough these days that I've gotten quite rusty on debuggers; usually a few judicious print statements (at most) will point out the problem.

The key is that I design and write code that has a "shallow" bug surface. When bugs happen, they're almost never those deep-dive kind that lead to a week- long bug hunts.

Don't sweat it too much. Keep sharpening your axe, and if you go six to twelve months without feeling your work is appreciated, run.

</comment><comment><USER11> I don't know if I've ever been "slow", but I've definitely gotten faster over the years. My personal measure of how experienced I've become as a developer is the number of problems that I treat as a hash table lookup (vs those I need to actively debug). Given that, there are a few possible causes for your so- called "slowness".

If you spent 4 years on the same team, it's quite possible that rather than getting 4 years' worth of experience, you got one year of experience 4 times over - by not constantly being exposed to new classes of problems, you haven't actually expanded your ability to solve problems by a significant amount.

Another cause is that you may not put enough thought into the type of work you find yourself doing. There's a fabulous talk by Rich Hickey ([http://www.infoq.com/presentations/Design-Composition- Perfor...](http://www.infoq.com/presentations/Design-Composition-Performance)) where he compares software developers to improvisational jazz artists. The gist is that Coltrane didn't just get up on stage and play music that magically popped into his head: he practiced, and tried stuff out, and then on stage he composed little things that he already knew worked into a single performance. Software is much the same way - by playing with new things in your spare time, reading about different patterns and libraries, and thinking about the types of problems you face day-to-day, you're building up a huge library of small solutions that you'll compose into software at work.

Reading HN every day is a good start. I like to use HN as a source of inspiration and a place to discover things to learn about. Were it not for browsing HN pretty much every day, I wouldn't have discovered most of the things to which I attribute to my growth as an engineer over the past few years. Certainly, I wouldn't have grown much if I didn't actually investigate those things, but it's great to have a constant source of new "known unknowns".

The degree to which you can put any of this advice to use, is of course, entirely dependent on how much time you're willing to devote to getting better at building software. Aside from spending time with my fiancée and son, I don't do much other than what I've described above. As a result, I'm fairly one-dimensional, but I'm more interested in writing great software than being interesting at dinner parties, so it's an acceptable tradeoff for me (the other result being that I'm told I'm pretty damn fast at writing good software).

TL;DR practice, read, practice, read, practice.

<comment><USER19> > Reading HN every day is a good start.

Just be careful with that. It can really suck the time. I have a problem that I am reading too much and writing/working less since I discovered HN. I am almost seeing it as addiction.

<comment><USER11> It absolutely is. If you're interested in why, I recommend Nir Eyal's (great) book, Hooked: How to Build Habit-Forming Products. Basically, our brain gets addicted to "variable reward systems" (Facebook, slot machines, news websites) because it gets excited when it finds something interesting/entertaining. Not everything is though (this is the key part), so when you finally do find something good your brain gives you a dopamine rush, which addicts you. I won't pretend that I don't struggle with this addiction myself, but I've found having genuinely interesting work really helps - I read HN at work quite rarely at my new job.

</comment></comment></comment><comment><USER20> I don't know enough about your situation, but you should know that A) managers are just people and make errors in judgements and B) sometimes managers exaggerate stuff like that to get more from employees (12h days for instance).

For A, you could pull a bunny out of a hat - finish a project early (as early as possible) and make sure it is noticed (but don't brag about it). For B - sometimes you have to compromise, and sometimes you don't have to. Can you find another job? Do you want to? Can you stand up to them? Depends on your situation...

Last but not least, "slow" is not the same as "not meeting the deadlines". Are you meeting them? Make sure this is noticed! I would rather have a "slow" member in my team who can reliably deliver good results in time than some "hotshot" who can delivers tons of (unusable) code in very fast time...

Bottom line: in companies what should matter is how much you contribute. Just make sure others notice it too.

</comment><comment><USER21> > Before, I think my manager saw me as ready. Now I think she sees me as a > liability.

Did you talk to her about this? Did you ask her what can be done? This entire situation is painted that you're the one to blame. There are 2 sides to every story, your company can set realistic expectations. It sounds like your work is solid and that people respect you, but that your management has unrealistic expectations about your velocity. A compromise can be reached.

> PS: I'm getting married next month. Will I have to choose between my > marriage and permanent 12 hour days?

Yes, and your marriage is more important. Unless your job is incredibly fulfilling and meaningful, it's not worth working 12 hours a day.

</comment><comment><USER22> The high value project is a great place to find the parts that require the least amount of effort on your part but will have the highest value to the company and deliver those first. As an engineer this feels like "cheating". It's not, you are being prudent.

I just finished reading The Effective Engineer by Ed Lau and highly recommend it( [https://www.theeffectiveengineer.com/book](https://www.theeffectiveengineer.com/book) ). It covers both concrete (like keep a daily log) and more general ideas(e.g. your work is measured by the value it produces for the company, not the effort you put into it).

<comment><USER23> Sorry, but I just can't take this book seriously.

The author uses this incredibly annoying marketing copy associated with get rich quick schemes. It comes across as manipulative and patronizing. He even sells optional pro and master packages with the book.

</comment></comment><comment><USER24> The two biggest things that jump out at me here:

\- you feel like you're performing well, but are being told you're not up-to- snuff

\- you're already worried about working 12 hour days and having to sacrifice your work/life balance with your spouse-to-be

Aside from valid comment about learning to move quicker, perfection is the enemy of good, etc etc.. it seems like you need to evaluate your working situation as a whole.

You believe that you're performing well. Your code is stable, there's top-down acknowledgement that you're pushing towards tested and maintainable code. Yet, you're being told you're... slow. Now, maybe that's true. All engineers operate at different speeds. With a proper manager, though, assuming you're not going amazingly slow, they should see this and understand it. If one of your best qualities is being pushed as one of your worst, then management is doing you a disservice, instead of being able to position you get better.

Also, consider that you're working 12 hour days. I could, with a little hyperbole, say that you're already married to your job. :) This is already a weird situation. Are you part of an on-call rotation? Is it a busy season? Does your company give out bonuses frequently to acknowledge this extra work being put in? Consider the psychological and physiological effects of being worried about what your manager thinks of you, getting married and, potentially, the traditional planning rigamarole that goes along with it, along with physically sitting at a computer for 12 hours, all while dealing with the other things normal humans might have to deal with on a day-to-day basis. It's a corrosive situation. It's a negative feedback loop.

What I'm saying, if anything, is that it sounds like maybe you're ready to explore other opportunities. Changing jobs isn't simply for making a salary jump. There are a lot of great companies out there who value the qualities you (not the general "you", _you_ ) possess and have the right management to make sure that your greatest strengths stay that way, and are applied beneficially without being perceived as a risk or liability. There are also companies where working a normal day, without extended hours, is not only normal, but jokingly enforced... because many companies do actually respect work/life balance. :)

</comment><comment><USER25> QA: practice, practice, practice.

PSA: no - You should NEVER choose work over your private life. If Your current boss ask You to choose between it - change the job.

</comment><comment><USER26> Could you describe on the technical capabilities of your current and past managers. How the team members view your managers? Does the managers are only interested in getting the work done and not bothered to look at implementation ?

</comment><comment><USER27> Collaborate with your manager on more detailed work schedules. Ex: if you do week-to-week deliverables, switch to day-to-day. They'll have a better notion of expected cadence.

</comment><comment><USER28> First step is acknowledgement. Seems like if you figure that out you can figure out how to speed up a bit (without compromising quality too much).

</comment><comment><USER29> SDE I, SDE II... Microsoft?

<comment><USER30> same denomination is used in Amazon as well, and probably other (big) companies as well. Not sure if it's indicative of Msft.</comment></comment>


Turn Gtalk Into A Command Line for the Web With Zapier - <AUTHOR> https://zapier.com/blog/2013/04/29/turn-gtalk-into-a-command-line-for-anyone/

<comment><USER2> Brian, whenever we bill the monthly subscription fee to our users for our saas app, we send them an email with their username and the amount. I get a copy of each such email in my gmail inbox. Is there a way I can use Zapier to parse the amounts into a google spreadsheet so I know the total amount billed for the day? (Currently I have to do this manually by opening each email and copying the username and amount inot a google spreadsheet).

I should have our dev build this report, but don't get me started on that rant :-)

<comment><AUTHOR> Interesting use case!

Unfortunately we don't have that clever of a parser. We've definitely talked about it but aren't 100% sold on any sort of internal implementation.

We've thought about a "reverse template" where you would mark blocks of text as "variables" across a few examples and we'd parse it out. Not sure if that is feasible though...

Any HN'ers done this before and have ideas or tips?

<comment><USER3> I think postmark and mandrill (and maybe others) do inbound email processing and then fire a webhook. Maybe hooking those together via zapier could provide a solution? I haven't actually used any inbound mail processing yet (or I'm ashamed to admit, zapier), but it sounds doable.

p.s. Hey Bryan!

<comment><AUTHOR> We definitely support catching inbound email (<https://zapier.com/zapbook/email/> and <https://zapier.com/zapbook/mailgun/> comes to mind). I'm curious if there is a clever parsing solution...

</comment></comment></comment></comment><comment><USER4> Dumb question: let's say I want to link gchat with twitter, and all the text after "tweet " I send to the zapier bot I want to be tweeted. Naively, I set things up (pretty easily, too!) to tweet ALL of the text I sent to the zapier bot. Is there an easy way to be able to tweet just parts of text? (maybe following a regexp, altho that's already getting too sophisticated for something so simple, especially since you guys just posted into turning gchat into a command line)

<comment><USER5> Yup there is. Check out our variable syntax: [https://zapier.com/support/questions/119/passing-named- varia...](https://zapier.com/support/questions/119/passing-named-variables- through-to-writes-advanced-tip/)

It's certainly much simpler than a regex but imposes more structure. We're always open to suggestions, so let me know if you've got any cool ideas on how we can parse smarter.

</comment></comment><comment><USER6> This reminds me of a side project I've been working: <http://autobot.io>. Idea is to have a chat bot in your email that can handle mundane tasks. I started with scheduling as that's a particularly acute pain-point for me.

</comment><comment><USER7> Awesome idea guys! We've been doing some of this with Hipmob - giving users commands to interact with all their integrations but we're purely CRM/Chat - this is an insanely useful tool.

</comment><comment><AUTHOR> We were largely inspired by GitHub's Hubot in writing this list of examples. Additionally, it's fairly trivial to use our developer platform to spin up new, private actions for custom services and APIs.

Happy to answer questions!

</comment><comment><USER8> Can you provide a bit more explanation on to set up something like this? Is it a Zapier feature? The article just lists 5 faves and skips the "turning Gtalk into" part.

<comment><USER5> Sure, I can elaborate. The article assumed a little bit of familiarity with Zapier. The basic mechanics are creating a new Zap on Zapier (which is a pair of one trigger and one action).

For the trigger, choose "GTalk" and "New Message". Specifically, the Zap will trigger whenever you send a message to our always-on bot.

You can pick any action that suites you, see the article for ideas.

We'll automatically perform the action whenever the trigger occurs.

There is a lot of flexibility in the system but that's the 500 foot view.

<comment><USER9> That's really cool -- certainly very nifty for Zapier users.

The title, though, implies a service (or hack) that isn't dependent on any other system -- maybe it might be better titled "Turn Gtalk into a Command Line for the Web using Zapier" or something?

<comment><AUTHOR> Good call! I've updated it.

</comment></comment></comment></comment><comment><USER10> I've been doing this for a while with Ifttt, which has some instant hooks and no stupid limits.</comment>


Uhh dude… you got hacked - <AUTHOR> https://medium.com/@meuspartum/uhh-dude-you-got-hacked-249d2c08d7ed

<comment><USER2> Author here -

The purpose of going through and removing the malicious scripts was to learn more about server security. I figured if I had been compromised I might as well learn what was causing it and how to fix it.

I have since toasted the server and set it back up with all of the things I mention in the article. I wrote the piece because I felt that there might be others out there who can learn from my mistakes and to also have a record for myself.

We are always learning and sometimes we need to learn the hard way, I got lucky in that this was on a side project and not any real client work.

</comment><comment><USER3> Ow man that is bad!

Thanks for sharing. Lessons shared from this post:

        1- Lock down ALL ports you are not using with either iptables or ufw.

        2- Be sure to always run system updates,

even with doing this the exploit still made it in, but the updates can help.

        3- Use tools available to you such as iptraf, nethogs, and iftop.

        4- Watch your cronjobs and strange startup files

I think locking down the ports that are not required is a very important thing to do.

I have faced such stuff, but with lower traffic luckily ended up creating a whole instance and server and ripping a part old configs and setup.

</comment><comment><USER4> When someone compromises your system, you can't know _everything_ that has been done. The only truly safe thing to do is reinstall everything from scratch, backing up configs of course.

</comment><comment><USER5> How did they get in,in the first place?

<comment><USER2> Author here.

It was a brute force attack on my root. I added an edit for clarification on the article. Basically I use another user for deployments and server work that has a pub key and didn't remove root login permissions.</comment></comment>


Fedora is moving to Git - <AUTHOR> http://jkeating.livejournal.com/76407.html

<comment><USER2> "To facilitate that I want to make it easy to discover our modifications and pass them along to upstream and/or other consumers of a given upstream. One way to force this is to continue working with a upstream archive (tarball release, maybe a snapshot) and our modifications as patches."

It is interesting that Debian faces the same issues and has come up with the idea of using Git as a source package format: <http://wiki.debian.org/GitSrc></comment>