Ruby is too slow for programming competitions - <AUTHOR> http://blog.clifreeder.com/blog/2013/04/21/ruby-is-too-slow-for-programming-competitions/ <comment><USER2> A lot of people are talking about the problem itself - which is interesting - but very often you run into real problems which cannot be simplified so easily (NP hard problems, many different other cases). In addition, when you need to do important calculations on a server for your product and you're using Ruby, you may very well end up requiring 10x as many servers. This is pretty massive when you consider an AWS server can be $350/month with Go (1 large instance), and then $3,500/month with Ruby - if you can even parallelize your problem that well - generally you can't. That's a lot of startup runway being eaten for not that much benefit.

Of course, most products don't really need to do calculations outside of a database and this is why Ruby has taken off so well - but it's still important to realize that choosing Ruby over PyPy/Go/Whatever really can be a very expensive choice in the long run when your product suddenly relies on some unique math solutions, and having half your product in Ruby and half in C is awful to have to deal with.

The solution to this is a project like PyPy for Ruby, but it doesn't seem to be coming...

<comment><USER3> > having half your product in Ruby and half in C is awful to have to deal > with.

Why? In many cases, only a tiny fraction of your code needs to be fast. In my experience, it is very reasonable to code that part in some fast language while delegating the bigger (and often more complex) part in a higher level language such as Ruby.

<comment><USER4> Everyone always agrees with this and, in my experience, has never really tried it. It works better in theory than in practice (or, at a minimum, it works only on a small subset of the types of problems that a naive view might otherwise lead you to believe). The problem is the data isn't organized in a way to be computation efficient, and all the C code in the world isn't going to make chasing pointers to pointers to pointers to ints fast.

A ton of work has to go into computation-friendly data organization (see all of numpy, scipy, etc) before you can ever actually optimize effectively.

EDIT: since all of my responses are of the form: "oh no I did it once with great results", I'll retract the part where I said "have never really tried it". I'm not arguing it's never worked. I'm arguing that the set of likely performance problems and the set of easy-to-optimize-with-C problems don't overlap as much as people would like to believe.

<comment><USER5> I have tried it. My MSc dissertation required a lot of computation (weeks of runtime _after_ I'd rewritten critical portions in C++), and I'd do it that way again.

For many types of problems, putting the data into an efficient format before passing it to your C/C++ code is trivial in terms of both difficulty and time spent.

I prototyped everything in Ruby, profiled, and then rewrote a handful of the critical parts in C and C++.

<comment><USER6> I've done it too, and I think most of the responses to the GP are missing the point. Can it be done? Sure. But it's a pain in the ass. It's not nearly as trivial as people suggest. It's basically only worthwhile when you know you have the resources to build an infrastructure of re-usuable fast modules that get coupled with a scripting language interface.

Even when you give up on writing true "hybrid" code (i.e. "just write modules for the slow parts in C++", which is a _gigantic_ pain), and instead try to write driver code that calls standalone compiled applications, you find yourself having to write a lot of duplicate code to parse/validate I/O on both sides of the language boundary. It sucks, and there's no way to make it not suck.

I'd go so far as to say that for problems where processing speed is a known requirement from the start, you should just give up on interpreted languages. Fast interpreted languages have been right around the corner for long as I've been writing code, but they sure seem to be taking their time.

<comment><USER5> I didn't find it hard at all, nor a pain in the ass. I wrote my code. Profiled it. Found the few, small bits that needed to be fast and treated the Ruby code as an executable spec for the C, and rewrote it very quickly.

There is SWIG support for Ruby, as well as the Ruby FFI implementation and RubyInline, offering various degrees of simplicity, but none of them are hard. The Ruby extension interface is more work, but even that is fairly simple to work with. I've taken C libraries and wrapped them in Ruby in minutes using SWIG for example.

Overall I often these days choose to write in Ruby first and replace _even for applications that I intend to write the finished version of entirely in C_. Prototyping in Ruby and rewriting in C once I have a working version is faster to me than iterating over a C version.

<comment><USER6> _"I wrote my code. Profiled it. Found the few, small bits that needed to be fast and treated the Ruby code as an executable spec for the C, and rewrote it very quickly."_

The fact that the speed-critical parts of your problem could be reduced to a _"few, small bits"_ indicates that we're talking about entirely different things.

I was in a problem domain where writing the time-critical pieces in a different language meant ensuring that huge amounts of working-set data were available to that optimized code. Efficient access to that data meant storage in native format, which meant that supporting two languages would require a nasty I/O layer for the interpreted language. Not worth it.

And that's just for the problems where it was possible to consider using Perl/Python at all. For some problems, the memory overhead of Perl/Python data structures ruled out use on all but the smallest problems. Again, not worth it.

It's nice that the speed-critical parts of your problem were reducible to a few lines of code, but that's not often the case. For most real problems, I/O is a huge concern, and can't be dismissed just by pre-processing the input.

<comment><USER7> Also, it's a weird wonderland in which you know exactly which parts need to be fast, reimplement those, and then never need to touch those bits again. In my experience, that's not the case - you're going to want to tune and alter your algorithm, and if it's API needs to go through several languages, it's going to be a pain. Every time you decide you need a slightly different input format or data store or set of hyperparameters, you'll likely be changing many more files than you would in a one-language solution, and in a way that's hard to unit-test and hard to statically type.

It's certainly doable, and definitely the right way to go if you want number crunching in a high-level language, but it's not a free lunch. It's certainly not a quick one-time conversion and then you can forget about it.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> _Sure, I knew Ruby wasn't going to be zomg fast, but I always assumed that if I chose the right solution and wrote in an efficient manner (memoizing, storing values/lookups that would be used later, limiting search spaces, etc), my ability to write code quickly and concisely mattered more than sheer processing speed._

 _I was wrong._

Sure, the author is wrong, but not because Ruby is slow (it is, though). He's wrong, because he did not find the right solution for the problem. Instead, he wrote a brute-force solution with a simple optimization, and did not realize that this is the real cause of his code under-performing, blaming the Ruby's slowness for it, when he really should have blamed the Dunning-Kruger effect.

<comment><USER9> Even if he did brute force it, he did the same thing with Go and it was much faster. So by comparison Ruby is slow and his conclusion is sound. Yes he could have optimized it, but that just means he could of optimized the Go solution too.

<comment><USER10> Let's consider his conclusions:

        ruby is slow:                         sound
        ruby's slowness is his main problem:  not-sound

He's not going to win programming contests with brute-force solutions. Even his brute-force method makes unnecessary computations - why keep running after n^2 is out of range?

> memoizing, storing values/lookups that would be used later, limiting search > spaces, etc

He did none of that. Kudos to OP for benchmarking & profiling though -- the speed of Go is impressive.

</comment></comment></comment><comment><USER11> I also come to the conclusion that Ruby-like languages are just not the right tool for this kind of problem, but it should be pointed out that you would have been able to solve the 10^14 data set easily, even with Ruby.

I was solving that problem with PHP and got it running reasonably fast (1s per range) for ranges up to around 10^60. PHP is probably faster than Ruby, but it's still in the same area.

The first trick you can use is that you don't have to go through all numbers between start and end, rather you can directly only traverse the palindromes (i.e. you will only have to increment one half of the integer).

The second trick is that only numbers consisting of 0s and 1s (and a 2 at the start/end or the middle) can satisfy the condition (exception: the number "3"). This further massively reduces the search space.

There are more criteria that you can use to narrow down the numbers to consider (e.g. the number of non-zero digits can't be greater than 9). But in the end, cracking the 10^100 range is probably not possible in a language like Ruby or PHP. Using something like C++ on the other hand it becomes a triviality.

<comment><USER12> Yep, in a competition you want a language that had as little magic as possible. Languages like Ruby (not necessarily dynamic languages!) are just not really suitable for it

<comment><USER13> This seems like such a vague, insubstantial piece of advice. I've seen it before, but never in a more concrete form. I don't see any "magic" in Ruby. How do I determine if a language is magic? Usually, the best I can figure is that they're either confusing syntax with semantics or they're actually thinking of some library like ActiveRecord rather than Ruby.

Really, for pretty much anything I could conceivably think of to call "magic" in Ruby (e.g. method_missing), there's equivalent "magic" in Python (e.g. __get_attr__). And yet PyPy is pretty darned fast.

<comment><USER14> I think people are referring to language features that are very powerful and concise but have horrible performance when you get large data sets like this.

Something like the inject method, at least in my limited use of ruby with these kind of Google Code questions, seemed problematic.

</comment></comment></comment></comment><comment><USER15> For comparison, I translated it pretty directly to Python:

        import sys
        from math import sqrt

        def string_palindrome(num):
            s = str(num)
            return s == s[::-1]

        sys.stdin.readline() # throw away first line (number of cases)

        for count, line in enumerate(sys.stdin):
            found = 0
            start, finish = [int(num) for num in line.split(" ")]
            sqrt_start  = int(sqrt(start ))
            sqrt_finish = int(sqrt(finish))

            for x in xrange(sqrt_start, sqrt_finish+1):
                if string_palindrome(x):
                    square = x * x
                    if string_palindrome(square) and start <= square <= finish:
                        found += 1

            print("Case #%d: %d" % (count, found))

For the first 15 test cases I get the following run times:

        pypy 2.0-beta2: 1.6 seconds
        pypy 1.9:       2.1 seconds
        ruby 1.9.3p194: 3.7 seconds
        python 3.3.0:   5.9 seconds
        python 2.7.1:   6.2 seconds

The integer version for checking palindromes was also much slower in python.

Edit: Even though the interger version of _palindrome is slower in pypy, there's another optimization that works there, but is slow in cpython. It brings the run time down to 0.6 seconds!

        def half_string_palindrome(num):
            num_str = str(num)
            num_len = len(num_str)

            for i in xrange(0, num_len // 2):
                if num_str[i] != num_str[num_len - i - 1]:
                    return False

            return True

<comment><USER16> Check with PyPy if you can

<comment><USER17> I just tested, and it should be about 6x faster than Python 2.7. About 1.0 second for the first fifteen cases.

</comment></comment></comment><comment><USER18> The following site allows you to search for people's solutions by language:

<http://www.go-hero.net/jam/13/solutions>

There are actually several people who solved this problem (including both large inputs) using Ruby.

And for those asking about Python, most Code Jam problems are apparently calibrated to be solvable using Python. Keep in mind that it's one of the languages Google uses internally and Guido used to work there, so they probably want to support it. From <https://code.google.com/codejam/problem- preparation.html>:

"Where possible, if you allow multiple programming languages, make it so that all languages are fast enough to solve the problem. Occasionally that's impossible; in Code Jam we generally calibrate by making sure a somewhat optimized Python program can solve a problem. Sometimes, though, there's just no way to have Python succeed with the best algorithm and C++ fail with a worse one. In those cases we'll typically decide that sometimes in life, you just need to use a faster programming language. We try to avoid making that decision if we can."

</comment><comment><USER19> First of all, there are plenty of people who solved both large inputs with Python/Ruby (with and without pre-computing all the fair and square numbers).[0][1]

As others have stated, the contests are designed such that language shouldn't matter and problem sets are deemed solvable with Python as a baseline. For this particular question, the official contest analysis goes into detail on how to reduce the solution space:

[https://code.google.com/codejam/contest/2270488/dashboard#s=...](https://code.google.com/codejam/contest/2270488/dashboard#s=a&a=2)

Now the issue becomes the trade off between development speed vs runtime performance.

Is it possible that you might solve the problem with a worse solution in a better performing language? Yes.

Is it possible you might finish a solution within the time constraints in a more expressive language? Yes.

If something is CPU bound in production, then going with a better performing language is a viable consideration.[2] However, _my_ goal in programming contests is often to become a better programmer in the process. As a result, I choose to focus more on improving algorithmic complexity which is language agnostic.

[0]: <http://www.go-hero.net/jam/13/solutions/0/3/Ruby>

[1]: <http://www.go-hero.net/jam/13/solutions/0/3/Python>

[2]: For the pedantic, I understand that language performance is different from implementation performance.

Edit: Corrected links, thanks victorhn.

<comment><USER20> Correct Link for [0] is

<http://www.go-hero.net/jam/13/solutions/0/3/Ruby>

And the most difficult version (L2) was solved by 19 contestants.

</comment></comment><comment><USER21> Wow, 5 minutes in Ruby, and less than a second in Go? I know scripting languages are slow, but it's easy to forget just how much slower. (EDIT: Corrected misread number thanks to dljsjr).

I don't want to detract from the article's main point with a cliche "algorithmic optimization beats fine tuning", but I think it's worth mentioning. Unless I'm mistaken, this particular problem can be solved about 10,000,000x more efficiently.

        > Given two numbers X and Y, return how many numbers in that range
        > (inclusive) are palindromes, and also the square of a palindrome.

I would just count the digits of X and Y, then ranging between those counts, apply the following algorithm: For N digits, enumerate all numbers of N/2 digits satisfying the bounds X and Y. If N is even then simply mirror the first N/2 digits. If N is odd, mirror the first N/2-1. Finally, check the bounds of the resulting number against X and Y to filter out edge cases.

If the original algorithm runs in 10^N time, this would be 10^(N/2). For N=14 as mentioned, that's roughly a speedup of 10,000,000.

Edit: It seems I misinterpreted the question (which sounds like you're supposed to enumerate all palindrome numbers X>=N>=Y, and also print out the squares of those numbers). This is actually asking for a particular type of palindromes -- those which are palindromic squares of palindromes. As macobo helpfully points out, you can drastically optimize mathematically from this additional filter.

<comment><AUTHOR> To clarify, the Go program (which took under a second to run) was doing the same processing as the full Ruby program that took 50 minutes to run.

<comment><USER22> Others have asserted that they aren't doing the same processing and the GO program is not correct.

<https://news.ycombinator.com/item?id=5586152>

<comment><USER23> Go is not an acronym

Go is not an acronym

Go is not an acronym

See also: "ham" (amateur radio) is not an acronym

</comment></comment></comment></comment><comment><USER24> I'm not a rubyist but it appears the OP was doing something like this:

      for each range (start to sqr(end))

        for each number in the range

          check if the number is fair and square

If you do this you're checking the same numbers many times (total count of numbers checked is over 3 hundred million calls...)

The right approach is to first compute _all_ fair and square numbers between 0-10^14, store the result, and then simply check how many are found in each range.

It turns out there are only 39 fair and square numbers between 0 and 10^14.

I participated in Code Jam 2013 and passed the qualification round using Python; after precomputing the fair and square numbers, checking ranges was pretty much instantaneous (for large input 1).

<comment><USER25> Sounds right to me as well. But hey, isn't it much easier to blame the language and its VM? Or interpreted languages in general?

</comment></comment><comment><USER26> The Go solution is incorrect. It's not using int64, and most of the finishes are bigger than a 32bit int. If you run it on the following input: """ 2 8 10000200002 1 500 """ You will see that the loop that runs from start to finish is not running on the first case, because it sees the finish as 0. Haven't tested it using int64, but I would imagine it would be much slower(Still a decent bit faster than Ruby presumably), because any heavy computation for this problem was removed.

<comment><USER27> I noticed that as well [1], and rewriting the code to use int64 (and thus cover the entire range) makes the Go code run in 2m40s. Still faster than Ruby, but not as incredible as the author thought it was.

[1] <https://news.ycombinator.com/item?id=5586152>

</comment></comment><comment><USER28> These sort of contests are not tests of speed of computation, they are tests of algorithmic knowledge. The final data sets will (usually) all be designed to make even the most optimized implementations of the wrong algorithm fail.

</comment><comment><USER27> I'm sorry, but is the Go program correct? I ran it on my machine, and for Case #1, it reports a solution of 1, while there are actually 19.

EDIT: I modified the Go program to use int64 instead of int (the upper bound of the first case is not a valid 32-bit integer), and the execution time is now much higher: 2 minutes 41 seconds on my laptop.

<comment><USER29> Go 1.1 made int default to int64, so it may depend on how up-to-date your particular version is. God knows what the blog poster uses, though.

<comment><USER27> I'm still on Go 1.0.2. However, I strongly doubt that having 1.1 yields the correct result in the amount of time that the article boasts; I rewrote the program in C (direct translation), and with both clang and gcc, I get timings of around 2m05s. It's hard to believe that for a CPU-bound task, a Go program would be 200x faster than a C program.

<comment><USER29> I'd agree it's dubious, but I was mostly commenting on the "correctness" of the code. It'd work on Go 1.1, so it's technically "correct". Sometimes. :)

<comment><USER27> Fair enough.

</comment></comment></comment></comment></comment><comment><USER30> It is true that Ruby tends to be slow for programming competitions, especially if the computation is done serverside such as Codeforces.com.

Notice that your solution runs in O(sqrt(N)) time. If N were up to 10^14, with 10^3 test cases, then your solution could have to deal with up to 10^10 iterations... which is dangerous even for a fast language (since your CPU can only do about 10^9 things per second). As a rule of thumb, one should only ever deal with at most 10^8 things. Even just incrementing an integer 10^10 times can take some time (fortunately, not every one of the test cases were 10^14 so your Go solution took under a second).

However, Problem C of Google Code Jam qualification 2013 can be easily solved, even with a slow language, by making some observations:

1) Instead of iterating through numbers and checking if they are palindromic, you can just iterate through the numbers with half the number of digits and generate the palindrome by mirroring the half. A naive solution that uses this would run in O(N^(1/4) log(N)^1.6) instead of your O(N^(1/2) log(N)^1.6) solution. (The log(N)^1.6 is due to the time taken for multiplication, assuming the Karatsuba method).

2) With a small amount of math you can figure out that for any fair and square number N = M _M, then for any digit s in M, we must have s_ (sum of all digits in M) < 10\. Thus solutions are of the form:

case 1) Contains up to nine digits 1. (e.g. 101, 111111111)

case 2) Contains up one digit 2 and up to four digits 1. (e.g. 1002001, 2)

case 3) Contains up to two digits 2 and up to one digit 1. (e.g. 200010002, 202)

case 4) Contains one three. (i.e., just 3)

A naive solution using observation 2 may run in O(log(N)^5.6) time.

3) Use combinatorics to figure out count the ways to arrange the digits in the 4 cases shown above, rather than iterating through them. As it turns out, it is unnecessary to do this since even using an O(log(N)^6) solution I passed the 10^100 input. Since there are only about 40,000 fair and square numbers from 1 to 10^100, you can easily precompute them and find them by binary search.

</comment><comment><USER31> This is a problem which is CPU-bound rather than IO-bound. Ruby, Python and the like won't typically show a great difference in speed when the majority of time is spent in IO operations (like web crawling, or text processing, running web apps, querying databases, etc), because the bottleneck on performance comes mostly from the slow reading of files and network sockets. In this case, though, the operations are mostly dependent on just doing computation, so a high-level interpreted language like Ruby really shows its limitations.

This is one of the (many) reasons why although languages like Ruby and Python are great for rapid development, on-the-fly updates and other reasons, it can pay dividends to be fluent in a high-performance language. It wouldn't even need to be C (although this wouldn't be hard to do in C) -- a language like D would give you excellent performance and nearly as easy development as Ruby.

</comment><comment><AUTHOR> Hey everyone, thanks for all the input on this. As I mentioned in the post, programming competitions aren't my forte, and I have a lot to learn in that arena. All the feedback on how this should have been done more efficiently is great.

In the future, I obviously need to work on coming up with better algorithms for solving these kinds of problems, but I'll probably continue to try and use Go because it's nice to have that fallback to sheer speed when my algorithm isn't great.

<comment><USER32> Just a suggestion, but I would also give some thought to efficient coding style, not just algorithms. For example, you could have speeded up your I/O dramatically.

<comment><USER32> Correction:

I benchmarked the I/O improvement I was thinking about. It actually only resulted in an improvement of about 30%. Worthwhile, probably, but not as "dramatic" as I thought it would be. Mea culpa.

<comment><USER32> hsitz:

It was only an EXAMPLE. It might be irrelevant to this particular issue, but look at the context of my comment: it isn't irrelevant to the subject of learning to code more efficiently.

<comment><USER33> No problem. When I read your post I was confused whether you were suggesting that he could have gotten a 30% speed up in overall time by making i/o more efficient. Now I understand you weren't.

But I would still suggest that, within the context of programming competitions, a certain kind of "programming efficiency" is close to irrelevant. A sloppily coded solution that nevertheless implements the proper algorithm can be many orders of magnitude faster than an efficiently coded implementation of a slow algorithm. Programming contests are more about having the insight to implement an algorithm having low big O, not so much about efficient coding practices. This isn't to say that efficient coding practices aren't important generally, just that there are much more important things to focus on in programming competitions.

</comment></comment></comment></comment></comment><comment><USER34> * competitions based on speed of computation.

I'm sure if the competition was based on speed of implementation there would be different conclusions.

<comment><USER35> Given two developers, one proficient in Ruby and one in Go there would be probably no difference in implementation time for such a trivial problem.

</comment></comment><comment><USER36> I participated in this contest, and solved this problem. The problem with the post is that it's basically saying "This correct ruby program is way slower than this _incorrect_ go program, therefore ruby is bad for programming competitions". Obviously not a valid argument.

</comment><comment><USER37> I am risking myself getting insanely downvoted for writing this. But I hope it at least gets some of the attention.

I really hope the Ruby Communities do actually admit Ruby is Slow. Because until you actually admit there is a problem, there will never be any notion of wanting to fix it.

Most of the time, they will come up with Rails Examples or serving dynamic web pages, and the bottleneck being in Database. And the higher amount of request you get, you can solve it with caching.

This is, in my view a very limited scope of Ruby usage.

They deny any wrong doing of Ruby, suggesting it is a dynamic languages, and it's wrong comparing to a compiled languages, or the algorithm are done wrong.

They keep suggesting under _xxx_ usage, Ruby is fast enough to get the job done.

Then there is the argument of throwing more money and hardware to grow and scale. But the truth is most of time Ruby will require more then a few dedicated servers to keep going. While it is not as drastic as the iron.io 28 to 1, but for startup lowering running cost is important too.

Then there is this arguments comes up which i hated most. You are not going to worry about the need of that many servers, or scaling that needs to change to another languages unless you are AirBnb, Twitter etc..... I mean what? I am sorry i may not have worded it correctly but most of the time those read like you are never going to be successfully that you are very unlikely to have to worry about.

Performance is important!. And for most this isn't about pushing it to Javascript V8 or insane 1000x programmer Mike did with LuaJIT. It is just Ruby should have a more respectable performance VM. While Ruby is designed to make programmers happy, I am pretty sure there are many aren't happy with its basic performance.

<comment><USER38> PHP and PYTHON are slow too. Like them Ruby can take advantage ) of C extensions. You cant write everything in ruby like PHP developpers use native drivers for database communication , compression , image manipulation ,etc ... that's normal.

Scripted languages should be used for what they are for , a thin layer on the top of other more performant languages.

Like Twitter , Ruby/Rails allowed them to bootstrap their idea ,"probe" a market, make it work , then they thought about performance and scaling. That's what scripted languages are for.

<comment><USER39> Yes, but I think what ksec was driving at is there isn't an intrinsic reason what you said should be true.

 _Why_ should Ruby (and scripting languages) be used for RAD and not performance? That's because implementations range from really slow to somewhat slow. Self, LuaJIT, V8 and IonMonkey, and increasingly PyPy are ample evidence that this need not be the case, and that's just listing the dynamically-typed languages.

The bar has been raised the past few years. Increasingly we can pick three of: productivity, libraries, performance. Ruby lacks the last, which makes it a less interesting prospect for future greenfield projects compared to its competition.

</comment></comment></comment><comment><USER40> Even using the same algorithm implementation, there is a huge speed processing difference between languages.

For example the implementation of the Mandelbrot algorithm in RUBY takes 47 MINUTES to complete, while it takes LESS THAN 30 SECONDS when doing the same computation using much faster languages such as JAVA, SCALA, C or FORTRAN. According to this performance test, those languages are more than 100 times faster than Ruby, Python or Perl. Source: Computer Language Benchmarks [http://benchmarksgame.alioth.debian.org/u32q/performance.php...](http://benchmarksgame.alioth.debian.org/u32q/performance.php?test=mandelbrot) Other benchmark speed comparisons between programming languages, showing similar results: <http://benchmarksgame.alioth.debian.org/>

<comment><USER41> > between languages

Between specific programs, using specific _programming language implementations_.

</comment></comment><comment><USER42> This whole, "Ruby is slow," diatribe is just ridic.

Everyone here that has done any C/C++ or Ruby development would probably agree that if you're looking for number smashing, Ruby is just not cutting it unless you're into rockin' C extensions and getting what you want... for now.

Ruby, within a few years, will probably have some VM action, whether the JVM with JRuby or Rubinius wins out, who cares.

In any case, in like 5 or 6 years when we all have 50/100 core machines, Ruby's gonna be just fine for almost everything. Maybe by then we will be done arguing and just agree that Ruby is the nicest language to look at and develop in, even if it has subpar performance. Because if you're arguing something different, you've probably never gone from Ruby to C or Java or even Python... just so you all know, it blows.

<comment><USER43> Tim Sweeney predicted that we'd all have 20 core machines with 80 hardware threads by 2009. Not really what happened. The whole point is that while Ruby is nice, it's just too slow. Someone is better off investing time learning a faster compiled/JIT language.

<comment><USER44> Too slow for what? Certainly not all classes of problems. Heavy mathematical work (every number in Ruby is an object! That's going to be much, much slower than a C primitive!), sure, but there are lots of folks using it quite happily to solve a wide range of problems.

<comment><USER45> How does Ruby scale?

\- Not-a-Rubyista

</comment></comment></comment></comment><comment><USER46> Many people seem to be asserting just because the OP's algorithm was sub- optimal, that it means his Ruby code running slow vs Go is a non-issue. The problem is even well written Ruby vs Go has the same kind of performance divide:

[http://benchmarksgame.alioth.debian.org/u64/benchmark.php?te...](http://benchmarksgame.alioth.debian.org/u64/benchmark.php?test=all&lang=go&lang2=yarv&data=u64) (All the usual benchmark disclaimers apply, your mileage may very, your app is always the best bench, etc etc.)

<comment><USER47> Ruby is said to be useful for text processing and such. In Japan they had a need for the language to deal with text files specified in a Japanese codeset. Some of the flexibility to support different codesets cost Ruby a little. For Ruby 1.9 and 2.0, Matz took a while in the transition to "Unicode" support in order to keep some flexibility, even though many people preferred the old way of UTF8 and so on.

Ruby is best when it's used for the things it was meant for, but that hasn't kept people from trying to use it for more stuff. That's how Ruby found a good use in servicing web apps, first with CGI and then with dedicated instances in Rails and the like.

Now with Dart I understand that people demand languages be architected in different ways to extract more performance from them. Dart doesn't have "eval", for example. Whereas Ruby and Javascript do.

The question though is one of a divide, between the people who would never use Ruby or JavaScript and those who do use and love those languages that come with a lot of flexibility from the core. Python is like Ruby and JavaScript, even if the Python syntax could be borrowed for different languages that are more restricted in what they offer.

<comment><USER48> I don't see the point with eval. Every lisp has a eval, and most AOT Lisps are faster than ruby.

Not including eval seems to be a "discipline and bondage" fetish of language designers.

Yes, you can write pretty awful code with eval. But it allows metaprogramming aswell. I'd like to judge if something is awful or awesome.

<comment><USER47> With eval language designers might have a delayed execution that could make compile-time optimizations less straightforward. Part of the toolset could have to be present during deployment time to be able to handle eval which happens at runtime. And yes, security concerns could also play a hand.

In Dart the code declaration is taken very seriously in many ways, and avoiding eval buys them more compile-time, loading-time and perhaps runtime optimization opportunities. I find that more than the language designers themselves, it's the end-users or developers that ask for more "bondage and discipline" from their tools. People who come from Java and C++ backgrounds, for example. They just can't have enough compile-time stuff aided by an IDE.

</comment></comment></comment></comment><comment><USER49> > TL;DR. Ruby is optimized for developer happiness, not machine happiness.

sounds good . I am ruby happy developer .

</comment><comment><USER50> There is a huge difference between Ruby and Go, in terms of performance. I wonder if anyone has any experience with Python, considering that both are interpreted programming languages.

<comment><USER51> I've has decent experiences with Python in Google Code Jam, with two exceptions. (1) Python can't handle large numbers, so the mere existence of huge integer inputs for the large prime number problem blew up my code since Python could not convert numbers that large. (2) Python is fine with math, but at a certain point of data accumulation, performance just __plummets __, no matter how good your algorithm is. Overall, though, I haven't had the experience the parent comment did. Either Python works beautifully or it literally just doesn't work, rather than take an hour.

I treat it as a challenge, though, as it forces me to be smarter about my implementation.

<comment><USER52> _Python can't handle large numbers_

Are you sure? Can you post a number of which you think Python can't handle?

        >>> int("983728338427643876432784367834678326438763278463276543675324675327867498274982787463273647235347652673547623548628347892374982374862384676235478632894790238409382984782768347683274983274964826487264826482264862384628648723643874268426482634826438742648276487264823764873627463252642654")

        983728338427643876432784367834678326438763278463276543675324675327867498274982787463273647235347652673547623548628347892374982374862384676235478632894790238409382984782768347683274983274964826487264826482264862384628648723643874268426482634826438742648276487264823764873627463252642654L
        >>>

</comment></comment></comment><comment><USER53> Ruby is too slow for brute force solutions for programming competitions. If you look at the third set of cases, the inputs could have a range 0..10^100. Even with the square root, any solution that is going to check 10^50 numbers/strings is going to take a very long time in any language.

</comment><comment><USER54> Idiomatic Go tip: you should use more type inference via := instead of declaring all variables at the beginning of a function. And use gofmt :)

</comment><comment><USER55> From the post...

Running time:

Ruby: 3180 seconds

Ruby (profiled and optimized): 342 seconds

Go (first attempt): 0.7 seconds

</comment><comment><USER56> He did everything right. Tried it in ruby, if it would have been fast enough this article wouldn't exist. Instead he applied good software engineering and found the right tool for the job.

This doesnt say "don't use ruby it's too slow". It's says prototype in your comfort zone and optimize. Isn't this what we all agree on?

Edit: auto correct fail.

</comment><comment><USER57> most numbers are not palindromes. so generate palindromes, and then test.

<comment><USER58> yes exactly, this is how a friend did it in jruby (he got a perfect score)

</comment></comment><comment><USER59> I have noticed a similar thing recently. A programming challenge with a 10s max time restriction, and just reading the input and storing it in an array takes over 7 seconds.

I allocate the array ahead of time, as soon as I know the size.

The equivalent C/C++ version took under a second.

<comment><USER60> How did you read the input? All at once or one line at a time?

<comment><USER59> One line at a time. Is that significantly slower for stdin? I would assume that this wouldn't make a difference, since we're talking about simple memory operations, and 20k method invocations shouldn't be that hard on a modern system.

<comment><USER60> stdin? No, that shouldn't matter then, but it would if you were reading from file.

</comment></comment></comment></comment><comment><USER61> In this context, no it's not.

One thing that Code Jam does differently (and imho better) is that they give you an input set and you have 8 minutes (or so) to upload the solution. This is expected to give more than enough space for language speed variance.

Instead of execution time, coding time is much more precious in such competitions, and

Micro-optimizing has it's place, but it's usually not in a programming competition. that's what using ruby instead of C gives you leverage for that, as you can express more complex ideas and algorithms more easily (which should be the focus point).

This doesn't mean you shouldn't know the cost of operations.

<comment><USER62> "Instead of execution time, coding time is much more precious in such competitions"

Unfortunately, in such competitions you'll be competing against people who write C++ algos faster than normal people writing Python.

<comment><USER61> That's because most contests (but not all) are limited to C, C++ and Java and they've practiced hundreds if not thousands of hours on solving those kinds of problems. Of course they will be more efficient in c++ than a person who is spending most of their time wondering how costly string operations are.

That doesn't mean that there aren't also people who have done the same and still prefer python (or some other language) not for execution, but development speed (and bignum, larger standard library...)

</comment></comment></comment><comment><USER63> So you wrote a brute-force algorithm in a slow, interpreted implementation of a dynamic language, and you were surprised that your solution took a long time to run?

Using a faster language implementation could have allowed you to use the same brute-force algorithm and come in under the time limit, but your real problem is your algorithm, not your language implementation.

You can write fast code in a fast language and get very fast running time.

You can also write slow code in a fast language, or fast code in a slow language, and still have acceptably fast running time.

But you wrote slow code in a slow language, the worst of both worlds.

</comment><comment><USER64> Maybe the result would vary with JRuby or Rubinius (I'd be interested in benchmarks) because not only is Ruby itself not the fastest language, but its main implementation (MRI) is also not the best for speed.

<comment><AUTHOR> This is a good point that I didn't consider. Running the 'just iterate over the range version' of this code with jruby:

        [master] clifff@fair_and_square: ruby -v
        jruby 1.7.3 (1.9.3p385) 2013-02-21 dac429b on Java HotSpot(TM) 64-Bit Server VM 1.6.0_43-b01-447-11M4203 [darwin-x86_64]
        [master] clifff@fair_and_square: time ruby fair_and_square.rb C-large-1.in.bak

        real	6m39.105s
        user	6m37.762s
        sys	0m19.009s

<comment><USER65> What is sys time? The amount of time it's actually run on the processor?

<comment><USER66> No, it is the amount of time your program spent in system calls: [http://stackoverflow.com/questions/556405/what-do-real- user-...](http://stackoverflow.com/questions/556405/what-do-real-user-and-sys- mean-in-the-output-of-time1)

</comment></comment></comment></comment><comment><USER67> Did you need to check `&& (start..finish).cover?(square)` on Line 17?

<comment><USER12> nope, iterating up to the square root of finish covers that

</comment></comment><comment><USER68> The article doesn't actually say that performance was one of the judging criteria, and I'm unable to find anything in the code jam rules about this. Can anyone here clarify?

<comment><USER58> Performance is not technically a criteria but you have eight minutes from the time you download the input data until you have to be done uploading the output data. So if you spend 50 minutes incrementing integers, then you'll get 0 points no matter what :)

<comment><USER68> Thanks!

</comment></comment></comment><comment><USER69> If you precompute the palindromes in the range, store a table of prefix sums, and calculate each of the 10,000 cases in constant time it will probably pass (in ruby).

Edit: as electrograv pointed out, another optimization is to generate palindromes from sqrt(start) to sqrt(end) rather than testing each number in the range for palindrome-ness. This saves you a factor of 10^3.5.

</comment><comment><USER70> From the edits: "If you are like me and don't always come up with the best solutions, you may want to consider other languages too."

\--> "You may always want to consider other languages too."

No one should ever agree with an ultimatum on language choice: if you need hardcore speed, use a lower-level language, not an interpreted garbage- collected scripting language. Duh.

</comment><comment><USER28> Also, if you ever think you need built-in support for large numbers in your programming language during one of these contests, then you're using the wrong algorithm. These contests are typically designed to be doable with very, very simple code using primitive data types.

</comment><comment><USER71> Remember PAPP[1]

[1] Richard Buckland's awesomeness - <http://www.youtube.com/watch?v=JTAkUs- NjxU#t=44m21s>

</comment><comment><USER72> Isn't it obvious that there must be a better solution to the problem then your solution? You shouldn't try to find the palindromes, but rather generate them.

</comment><comment><USER73> <https://news.ycombinator.com/item?id=5304873>

</comment><comment><USER74> Couldn't someone implement a Ruby CLANG dynamic runtime recompiler?

</comment><comment><USER75> Ruby is uber fast for hipsters..

</comment><comment><USER76> EventMachine helps with performance critical Ruby code

<comment><USER77> You're right that EventMachine helps when you're dealing with IO-limitations (like those commonly found in web applications), but for CPU-limited functions like the ones that you see in programming competitions, it's not going to provide much of a benefit.

</comment></comment><comment><USER78> Great, so stop dicking around with some zero-sum game ( <http://en.wikipedia.org/wiki/Zero%E2%80%93sum_game> \- a competition may only have one winner ) and go build a business, where no one cares what language you use as long as the solution works for your customers - and there can be more than one winner in many cases.

Or even if you don't have a side project or startup or something, go build something cool and open source it and do that, rather than working on some artificial problem.

You can click the little down arrow all you want, but it won't keep me from thinking you're mostly wasting your time with these competitions, and it won't put any money in your pocket.

<comment><USER79> Curious, would you call University education dicking around as well?

Seems to me challenging the brain is never a waste of time. We would not have some of the medical technologies we have today if it were not for massively parallel GPUs, which of course were developed for the sole purpose of playing games.

<comment><USER78> > would you call University education dicking around as well?

That's a big debate! For a lot of people, I would, though, yes. There are serious economists who posit that universities are mostly about signalling, rather than actually learning much. I don't think I agree with that entirely, and believe it depends a lot on what's studied. It also depends on the opportunity costs, which may not be that high for many people that age.

> GPUs, which of course were developed for the sole purpose of playing games.

I don't believe the creation of games is a zero-sum game at all; there can be many winners, and, as you say, there are other side effects that are beneficial. The creation of games also falls into one of business, side- project or open source project as mentioned in my other post, no?

Artificial coding competitions aren't really about creating much of anything though, from what I can tell. A bunch of people solve the same thing and then throw away what they made.

I agree that challenging your brain isn't a waste, but it seems that in a world that is desperate for coders, there are more productive ways of challenging your brain in a way that's perhaps even more satisfying, because of the positive feeling of having helped to create something useful.

<comment><USER79> > That's a big debate! For a lot of people, I would, though, yes. There are > serious economists who posit that universities are mostly about signalling, > rather than actually learning much. I don't think I agree with that > entirely, and believe it depends a lot on what's studied. It also depends on > the opportunity costs, which may not be that high for many people that age.

Fair enough. For myself, it was what lead me to my career. I found a large number exams followed a similar format to this very coding competition. Many times on the job I have thought back on exam questions for inspiration for the problems I was solving. In that sense, I most definitely do not classify my education as "dicking around".

It brings be to my original point; when video games were being developed, these admitted benefits were certainly not obvious at the time. We simply wanted to render things faster, and at higher resolutions. Now other applications of the same technology is ubiquitous in many regards.

As a final tangential point, I don't think the code competition is wasteful because it does not have any immediate intrinsic value. See exams example above, and really, the discussion here. Also, perhaps this was his so called "downtime". Many people would argue that not all facets of our life should be strictly geared towards productivity. He may already run a business or hold a day job (maybe his blog gives clues otherwise, I don't know).

<comment><USER78> There are plenty of things that aren't "productive", but strike me as still being more creative/interesting/useful than churning out the same answer as everyone else to a made-up problem. Making your own programming language or OS is not 'productive', but a great learning experience, creative, and lots of fun. And who knows, maybe someone will get some use out of it. I once made a fairly simple programming language, and it did end up seeing some niche adoption and use in industry, which was cool.</comment></comment></comment></comment></comment>


Linux Kernel Release Model - <AUTHOR> http://kroah.com/log/blog/2018/02/05/linux-kernel-release-model/ <comment><USER2> _" At the end of 2017 the size of the Linux kernel was just over 61 thousand files consisting of 25 million lines of code...The Linux kernel contains the code for all of the different chip architectures and hardware drivers...Because of this, an individual system only runs a fraction of the whole codebase. An average laptop uses around 2 million lines of kernel code from 5 thousand files to function properly, while the Pixel phone uses 3.2 million lines of kernel code from 6 thousand files due to the increased complexity of a SoC"_

Interesting snippet. In hindsight, I suppose it's obvious, but I hadn't considered that my particular instance of a kernel might use less than 8 percent of the source code.

</comment><comment><USER3> I do appreciate the perspective on bug fixes. A bug is a bug - fix it. This does not mean that security issues are not taken seriously, but that all bugs are taken seriously.

And in return, you should be able to apply any patches coming from upstream without fear of breaking anything. Having had to deal with patch management on Windows, I _really_ appreciate that.</comment>


Downside of Lower Manhattan's Boom: It's Too Crowded - <AUTHOR> http://www.nytimes.com/2016/12/02/nyregion/lower-manhattan-crowding.html <comment><USER2> > Just to get to his apartment in the financial district, he has to contend > with hordes of commuters and selfie-snapping tourists clogging narrow > sidewalks.

So make the sidewalks wider. I can think of nowhere in the US more appropriate to allocate more space to sidewalks than Manhattan.

> Ms. Starr, for one, has all but stopped using the Citi Bike system to > commute. She used to ride from her apartment in TriBeCa to her office on > Maiden Lane in the financial district nearly every day. But as more > pedestrians and cyclists filled the streets, she had to concentrate to avoid > running into anyone or being run into.

Sounds like they need separated bike lanes then.

Manhattan is the densest major city* in the country. The amount of space allocated to cars on the roads should be the bare minimum: enough for emergency vehicles, business deliveries, surface transit, the handicapped, a limited number of taxis. With only a tiny amount of space per person, you really have to push for denser forms of transportation. Personal vehicle use should be at a minimum in Manhattan.

> In Lower Manhattan, people are not the only obstacle. Construction on > streets and buildings is everywhere. A labyrinth of imposing metal > scaffolding hems in available walkways and forces pedestrians closer > together, or into the street.

The solution here is to cut into space on the road to provide more walking space.

* I know it's only part of a city but you know what I mean

<comment><USER3> Have you seen the streets in lower Manhattan? If they get any narrower there wouldnt be any room for cars. There are also already tons of bike lanes. Most New Yorker's don't drive - they walk, bike, take the train, or taxi/uber. Don't you think if the answers were this simple someone would have done this already?

<comment><USER4> Getting rid of cars is a feature, not a bug.

<comment><USER5> That can be hard to accomplish without also making life more difficult for ambulances, police, and fire departments.

<comment><USER6> Not at all. In Jerusalem, Yafo Street is now pedestrian and light-rail only. Even still, police vehicles routinely drive on Yafo Street, and coexist with the light rail by not parking on the tracks. When police vehicles have their sirens on, pedestrians get out of the way. Deliveries occur on side streets and alleyways.

The Jerusalem municipality closed literally the busiest street in Jerusalem to private vehicular traffic and the results could not be better. It turns out, "induced demand" isn't just an idea which explains why you get more traffic when you build more highway lanes, it can also explain how to get more foot traffic, too. So business is booming, and the area is vitalizing even more.

<comment><USER5> Manhattan has 4x the population density of Jerusalem (twice the population in half the space). On top of that, it's an island so alternate traffic routes sometimes don't exist.

Is only Yafo blocked? Blocking a street is one thing, but eliminating cars from Lower Manhattan is another.

</comment></comment></comment></comment></comment></comment><comment><AUTHOR> I commute from Tribeca to the WeWork Charging Bull via Citibike along the busiest [0] and one of the best separated bike lanes in America. I do have to deal with lots of tourists walking on the bike lanes (which might be improved with better signage), but I still feel incredibly privileged to have such a pleasant commute.

Within the Financial District, the solution is to de-prioritize cars and make more room for people. The initiative of closing off FiDi to all through traffic and sharing streets equally between people, bikes, and cars [1] is incredibly promising, but was poorly marketed and executed. I rode along the streets with my 7 and 9yo sons and we had cars honking at us, even as we moved at a reasonable speed.

[0] [http://www.streetsblog.org/2010/10/19/at-riverside-park- look...](http://www.streetsblog.org/2010/10/19/at-riverside-park-looking-to- more-bike-lanes-to-soothe-bikeped-conflict/) [1] [http://www.streetsblog.org/2016/06/16/new-twist-for- summer-s...](http://www.streetsblog.org/2016/06/16/new-twist-for-summer- streets-this-year-a-low-car-zone-way-downtown/)

</comment><comment><USER7> This is mainly a problem during rush hour, when traffic is always a problem everywhere. It might be exacerbated in that area but so much so that an average resident feels like they should be complaining about it? If your commute is from TriBeCa to FiDi (adjacent neighborhoods) and you're upset about losing 10 minutes to 'tourists' (even though, more often than not, people holding me up are clearly people who live here who don't know what common courtesy is) then talk to my coworker who lives on Staten Island and has to traverse water and take four forms of transportation to get to work, or my old boss who comes down from the tippy top of Washington Heights through some of the biggest tourist traps in the city and has to deal with more stops than you can count on your fingers.

I live in the Lower East Side and we like to go to the Financial District every now and then because NO ONE IS THERE. Go there for dinner or drinks on a Friday night or go walk around and make your way toward Battery Park on a nice Saturday afternoon. It's almost creepy, like a ghost town.

> It means never getting that babysitter on a Saturday night, or abandoning > hope of ever getting tickets for Shakespeare in the Park.

Ok then don't have kids? Move to Jersey? Well-to-do people are displacing less fortunate people all over the city, talking down to them about not being able to afford to rent or buy groceries, and then they're complaining about problems like this?

I also used to ride CitiBike and yes there are people but there are also people if you drive a car, there are bikes and cars if you walk. This is a commute not a Sunday stroll or trail ride. You're participating in traffic, it can be dangerous and you need to pay attention which is true everywhere in the world.

Obviously there are crowding problems in New York City, I take the train like four stops to work and I absolutely hate the traffic as much as anyone, but the anecdotes in this article are pathetic relative to some of the other living issues people are facing here.

</comment><comment><USER8> Barcelona has the answer: close many blocks to cars.

[https://www.theguardian.com/cities/2016/may/17/superblocks-r...](https://www.theguardian.com/cities/2016/may/17/superblocks- rescue-barcelona-spain-plan-give-streets-back-residents)

<comment><USER9> That's not the problem. The FiDi has no throughways, only local traffic. The only vehicles down there are taxis doing pickups/dropoffs and delivery trucks. It's just massively more dense than Barcelona.

<comment><USER10> No more taxis, then, deliveries and access-a-ride ( _shudder_ ) only (and deliveries only at night).

The bankers can walk.

</comment></comment></comment><comment><USER11> I live in the area. Right outside my building is one of the heaviest trafficked areas in the city. A building across the street is being demolished, which means that side's sidewalk is closed off. On this side, a wheel chair ramp in front of a coffee shop eats up half the sidewalk. So on trash days, there is only enough space one person to pass through the bottleneck.

In other places, you have very precious sidewalk space taken up by signs in front of restaurants, sidewalk sheds, lamp posts, panhandlers, mailboxes, people smoking cigarettes, and of course, piles of trash.

Don't get me wrong, it's a great place to live, but there definitely needs to be some big changes. I think we should remove street parking to expand the sidewalks.

</comment><comment><USER12> "Nobody goes there any more - it's too crowded" \- Yogi Berra.

<comment><USER13> it becomes non-paradoxical if you tweak it to:

"Nobody goes there _by choice_ any more - it's too crowded"

Obviously it's still not "nobody" for LM - or anywhere - but things change and people get stuck with the new reality sometimes.

<comment><USER12> I don't think anyone's ever gone to Manhattan with the expectation that it would be a staid, static and relaxed kind of place. Change is a constant in big cities like NY.

</comment></comment></comment><comment><USER14> Not a New Yorker, but living in the middle of the East Coast I've been to, done business in, and toured NYC many times.

The Financial District is cool, but why anybody would want to live there who doesn't work there is beyond me. It's a short walk from any of the residential areas to any of the major financial firms, it's a miserable slog anywhere north of Chinatown.

And it's not like it's cheap, 800sq ft 1 bedrooms run well above a million dollars.

There's other problems which, at this point, seem to matters of inertia rather than planning:

\- NYC is one of the few, dense, major cities without service alleys for trash pickup and delivery. A casual observer might notice that NYC blocks are solid, not hollow, the middle is still full of building. In other cities, with other designs, an access alley allows for service vehicles to enter a mid-block courtyard to provide service functions. As a result both delivery vehicles and waste vehicles block the street with both their presence and the materiel they move in and out.

\- The regulations around inspection scaffolding are absolutely out of control rent-seeking. I would be comfortable in saying that there isn't a highrise block of the city that's doesn't have this scaffolding present on it. While it does provide some minimal protection from the elements, it's more like walking through a construction site than a an arcade. The regulations should either be changed to get rid of this nonsense, or to make formal arcades a requirements, thus presenting new facades to the city. I'd actually be in favor of the later since NYC suffers from pretty much the worst of all seasons: cold and wet in the winter, and hot and miserable in the summer.

I love love love NYC, but it's not without its imperfections.

<comment><USER15> The requirements around scaffolding are kind of an annoying middle ground of rent seeking. You are required to put them up, but then you aren't charged enough for them while they are up to incentivize anyone to take them down. In some cases the scaffolding goes up and the company that owns it goes out of business or the contractor goes bankrupt and it just gets left there.

<comment><USER16> who is doing the rent seeking though?

For years they would leave the scaffolding up so they sell advertising on it and the fine was laughably small compared to the revenue generated from selling ads. That was outlawed a few years back though.

This is an interesting look at the phenomenon:

[http://www.crainsnewyork.com/article/20160124/REAL_ESTATE/16...](http://www.crainsnewyork.com/article/20160124/REAL_ESTATE/160129960/scaffolds- are-everywhere-in-new-york-city)

</comment></comment></comment><comment><USER17> I suggest that we build elevated walkways covering every busy street in (lower) Manhattan. You know it's going to happen _someday_ , so why not get it over with. And then we'll be one step closer to Coruscant.

I'm envisioning something like an extrawide catwalk built over some of the busier streets with either ladders or stairs down to the next floor of the city. They'll either attract tourists or they'll become the pedestrian superhighways of the city.

<comment><USER18> And truck/delivery/car service entrances working at ground level. There is a location or two where that could work. I also have been thinking of the same.

<comment><USER17> I have a feeling the stores on 5th Ave would be less optimistic. Like I said - I think this idea is an eventuality in such dense urban centers.

</comment></comment></comment><comment><USER19> One obvious answer here: makes far more sense to increase population of NYC outside of Manhattan. Brooklyn, Queens, and the Bronx combined (excluding Staten Island since it's not accessible by subway) are TEN TIMES the size of Manhattan. Not all this space is a quick commute to Manhattan…but there are plenty of neighborhoods in these boroughs that are nowhere near as densely populated as lower Manhattan, and are still within a ~30 minute commute to downtown.

</comment><comment><USER20> Another less sensational, but lengthier and more informative read on that: [http://www1.nyc.gov/assets/planning/download/pdf/plans/trans...](http://www1.nyc.gov/assets/planning/download/pdf/plans/transportation/td_fulllowermanhattan.pdf)

<comment><USER21> It's also a 20 year old read.

Edit: Many of these proposals in that have either been followed or exceeded, largely as a result of the September 11th attacks. Most notably Broad street is completely a pedestrian plaza, closed off to traffic.

</comment></comment><comment><USER22> Nobody goes there anymore.

</comment><comment><USER23> Megalopolises continue to be hellish to actually live in, news at 11.

</comment><comment><USER24> If where you live changes you have the choice of either staying and accepting the change or leaving to live somewhere else. Trying to hold back progress (for some value of 'progress') is futile in the extreme.

<comment><USER25> > _you have the choice of either staying and accepting the change or leaving > to live somewhere else_

The third option is trying to effect another change.

</comment></comment><comment><USER26> We never had these problems before the disneyfication of Manhattan. This city wasn't designed to be a tourist destination. Manhattan was already over burdened by sucking in most of the other boroughs' residents and the bridge and tunnel crown in for work. Add to that the hordes (literally) of tourists who walk around as if this _working city_ is a park or a shopping mall and you get what we have now.

[p.s. For years I heard the old saw about "loving Paris and hating the French" and wondered as a (yep) tourist why do the Parisians hate us so much? :) Since the past decade I have nothing but sympathy for Londoners and Parisians. Now I get it.]

<comment><USER16> Oh please, Manhattan has always been a tourist destination. That is not new.

The Brooklyn Bridge, the Empire State Building, Grand Central, The Statue of Liberty, Central Park, 5th Ave, Rock Center. The high concentration of tourist sites predates the "Disney-fication" of Times Square which was too seedy for many years to be a tourist destination.

That old trope about French hating Americans is a bad stereotype. As a long time visitor I can tell you this is simply not true. People in London, Paris and New York get annoyed by inconsiderate people whether they be local inconsiderate people or inconsiderate people from abroad. And there is no shortage of inconsiderate residents in any of those places.

<comment><USER26> > The Brooklyn Bridge, the Empire State Building, Grand Central, The Statue of > Liberty, Central Park, 5th Ave, Rock Center.

Exactly. Designated tourist areas. The rest of the island was ours.</comment></comment></comment>


The Apple Mafia - <AUTHOR> http://www.forbes.com/2009/02/18/apple-silicon-valley-technology-enterprise-tech_apple.html <comment><USER2> from the title, i thought this was going to be about apple using its clout to bully others into doing its bidding. it's actually about former apple employees who took apple's seeds of greatness with them into other tech companies.

<comment><USER3> It's playing off of other "Mafias", notably the Stanford one and the PayPal one.

</comment></comment><comment><USER4> Correct me if I'm wrong, but I think the only two guys I know of who worked at Hewlett Packard's calculator division are Jeff Hawkins and Woz.</comment>


The Internet of Pointless Things - <AUTHOR> http://www.forbes.com/sites/theopriestley/2015/09/21/consumers-prepare-for-an-internet-of-very-pointless-things/ <comment><USER2> Yes. Another angle...

Just looking around me there are 45 wireless SSIDs, crazy interference and nothing but crap flying around my airspace. Half the time I can see people's bluetooth devices wide open in nearby houses as well.

Add to that the impending 9000 useless little devices that will be broken and this is the RF equivalent of space debris. It's got to the point I can barely keep a WiFi connection up for more than a few minutes already. Even on my 4G connection I'm lucky to get 4mbits due to the contention locally. I can upload at 40mbits though which is crazy

So literally last weekend I cabled my house with 5 runs of 100 meg ethernet and a couple of long patch leads in the living room and stuffed in a cheap 8 port netgear switch from ebay. Plugged the whole thing into my router and turned off the WiFi entirely just using it as a switch + NAT combo.

I can now transfer files between my desktop and laptop at 10Mbytes/second rather than 500k/second with dropouts. It is simply bliss. Even my wife is no longer annoyed about the prospect of dangling wires because stuff is working again.

It has got to the point that I'd rather have a broken CAT5 plug that falls out every 5 minutes than WiFi. I really don't know how all this tech is going to work when there is so much contention and noise already. I think it will just contribute to the pain.

<comment><USER3> I just don't understand this wireless craze. I get that normal consumers buy into it, but for anyone mildly technically inclined, nobody should be using wireless anywhere they can avoid it. Wireless is just a messy solution. The only wireless devices I use at home are my phones.

<comment><USER4> Do you use laptops on couches at home? Cables seem hasslesome there.

<comment><USER2> That's what I'm now doing. I duct taped a couple of ethernet cables to the back of the couch and you just stick your hand down and grab one if you need it. I terminated the cables slightly earlier and used an inline coupler so you can replace the last 1m when the plastic clip inevitably breaks off after a couple of months without having to rerun 25m of cable.

I mostly use a desktop so this is primarily for the wife and kids.

<comment><USER5> This is crazy. Duct taping cables to the couch? Too funny.

What about tablets/iPads for wife and kids?

I don't know why you were getting so many dropouts but it's not because "wifi is bad". I live in a house with lots of noise from neighbours wifi, but no problems here. Wifi will be solid for months at a time.

I must have a good wifi router (Billion 7800NL). Not all routers are created equal, I've gone through a couple of duds before the Billion. Some deal with interference better than others. Let the router choose the channel automatically too, very important.

My house is old, quite large, thick walls and so on. But everyone uses wi-fi here from different rooms without any issues. A lot of simultaneous connections going on too.

</comment></comment></comment></comment></comment><comment><USER6> I am from the industry on the B2B side and this article rings even more true and much more strongly so on the B2B - industrial internet side of things. Go to any 'messe' (as they call it in Germany) or salon or expo and you see tons of connected/connectable industrial products but when you dig deeper into how those machines/components are going to be integrated, installed, commissioned, used and maintained, it becomes quickly evident that the vendors haven't thought those things through.

The dominant discourse when you talk to many OEMs is that this is a defensive play to be prepared for all those secretive inventions that Google and Apple are making in their X-Labs. They don't know what's coming so they are throwing everything at it to "cloud-wash" and "IoT-wash" their offerings even when their customers are justifiably scared about the risks of cyber-attacks that connected machines bring into their factories and plants.

In short, not enough RoI evident for the investments and changes in processes IoT mandates in the enterprise - especially in the production plants. However, based on my discussion with all the different actors, its likely that we'll find use cases in either Operations Optimization or Asset Optimization.

<comment><USER7> I actually think the industrial side of the story is where real value will be created. I'm also working on an industrial iot application, and would really like to chat! Is this your company?

[http://www.zeefaxcms.com/images/data_sheets/zeefax_data_shee...](http://www.zeefaxcms.com/images/data_sheets/zeefax_data_sheets/ZFX_MachIQ%20Data%20Sheet.pdf)

<comment><USER6> No it's not my company - thanks for pointing it out though - I am cofounder at a startup and we call ourselves MachIQ. Looks like a Trademark purchase will have tp be made eventually :-)

you can reach out to me via LinkedIn/Google - rchikballapur.

Cheers!

</comment></comment></comment><comment><USER8> As a meta comment, I can't get to the forbes story on a direct link because I adblock and scriptblock. I googled the title and got in via the google search free pass and I now see I'm blocking 19 social scripts/ads which provide zero to negative value for me. The point is no one wants spam or pointless interruptions or endless big brother tracking. The future of "smart devices" is your overhead lightbulb connecting to an absolute minimum of 19 social media and tracking sites and government offices to permanently record your every activity and no one wants that except spies and corrupt people.

If there's no end user benefit unless they can get the government to force the population to use them, most "smart" devices will end up doing the equivalent of the 80s/90s VCR clock flashing 12:00. Maybe the strategy to work around that is light bulbs of the future will refuse to turn on unless you give the home factory in China full access to your complete facebook, twitter, and linkedin profiles and perhaps your credit card number so you can purchase "upgrades" to run beyond 100 hours or beyond 50% brightness levels.

I'll be using dumb bulbs and laughing at people stuck with inferior smart bulbs.

<comment><USER9> I've considered going back to candles, honestly. This stuff is SO out of control.

<comment><USER10> Careful. If you don't source your candles properly you might end up with lead wicks, even though they've (shockingly recently) been outlawed in the US.

I've also got concerns about the material used to make the candles themselves. I started burning beeswax candles, and now when I'm around the more common sort it smells like factory waste, or like someone took all the chemicals under a sink and mixed them together. Unpleasant and disquieting.

<comment><USER11> Not to mention the fire hazard that they pose..

</comment></comment></comment></comment><comment><USER12> It's sort of getting old.

Create a closed/proprietary platform, lock people in, cash out.

Create sloppy "security", sell the solution for it, cash out.

We really don't need this stuff. It exists because money is being spent on talking us into spending money on it.

</comment><comment><USER13> I take a contrarian view on this. Such advancements / toys / useless things are vitally important to forward progress. What this article fails to recognize is that innovation is often built on the current "pareto (efficient) frontier" ... that is, it is built around the universe of available devices/applications/protocols/technologies/useless-things ... rearranged and consumed in new and novel ways. While the final application might not be a smart oven, as it might indeed be a useless item, that final application could very well be an application that requires the availability of a smart oven from which to build on, or one whose innovation was triggered by the availability of a smart oven. In other words, a smart oven plays a role in innovation and society as a whole, and shouldn't be dissed.

<comment><USER14> I agree - with the caveat that it should be "done responsibly" (which is a tough nut to crack in this context).

I'd love a "smart house" \- the fewer things I have to do when I get home, the more time I have to relax or do my own thing! I know the promise of the smart house and the execution of the smart house are years apart, but I feel we are just now getting to the "promise" part.

</comment></comment><comment><USER15> obligatory: [https://mobile.twitter.com/internetofshit](https://mobile.twitter.com/internetofshit)

<comment><USER16> It's funny how you get redirected to a mobile version of a website when you're on the phone, but not to a desktop version when you're on the computer.

</comment></comment><comment><USER17> Remember in the 90s when everyone said virtual reality and smart TVs were the next wave of high tech? And how all the big companies started putting money into R&D for said technologies? Remember how it never materialized? (Well, it's being realized now, 2 decades late.)

The "internet of things" is the same shit. Something people like to mention at TED conferences. But do we want it? Is the market ready for it? No one's asking that. If you ask me, we're not ready. At the moment, the tech is too expensive and too useless. A few things will win out with affluent customers (thermostats, audio systems). But putting chips in everything we own is a far off dream.

<comment><USER18> Funny how these things are. My reading of the last 20 years is the exact opposite.

We've gone from talking about how devices will change how we live and work to having devices that are changing how we live and work. Ok they weren't VR or TVs, they were sensor heavy mobile devices. But still.

And they're not completely pointless. They may not have solved world peace but they've changed comms in quite positive ways; I have to travel for work and video chat to my family is a wonderful thing.

Also, over the last year or so, I've seen a noticeable increase in people taking an interest with their health due to Fitbits and the like. Having that feedback loop, although a small thing, can make quite a difference.

I can see this sort of thing having logical extensions into the home because we now carry sophisticated control systems with us. It wouldn't surprise me in the slightest if we don't start getting QR codes on menus/food packs with nutrient content, for example.

Don't get me wrong, I fully appreciate that what I'm talking about here are all first world solutions to first world problems. But the idea of connected stuff affecting ones life isn't in the future.

<comment><USER17> Yeah but in order to reach that conclusion you have to abstract too much. We can all agree that devices are and will change how we live. But now we're not talking about VR, Smart TVs, or the Internet of Things anymore. Because the truth is, we never know what form the next wave of technology will take. And that's why I'd rather experiment than prophesize.

</comment></comment></comment><comment><USER19> See: [https://www.kickstarter.com/projects/883092715/smart-froc- th...](https://www.kickstarter.com/projects/883092715/smart-froc-the-worlds- first-smart-highchair)

<comment><USER20> As a coherent story this chair is terrible. The parents start off saying they couldn't find a good chair for their kids. So they designed a decent looking chair and it seemed like a great idea that it would work for kids until they were about 10 years old. That's a great solution, adult chairs can be pretty bad for kids at the dinner table. Anyway.

Then inexplicably, halfway through the video this crap about measuring the child's weight every time they use the chair. Nothing in the original problem says anything about tracking weight? Ridiculous leap from "can't find a good high chair" to "high chair with built in wireless scales".

Is there something in the kickstarter ts&cs that says "projects must have wireless connectivity and an app" that I'm missing?

<comment><USER19> It's one more ridiculous example of the 'Internet of Shit'.

If it's not secure, I could imagine people fiddling with the weights just to scare the crap out of parents. It's something my younger self would do.

</comment></comment></comment><comment><USER21> Here's my litmus test for taking an IoT thing seriously: If nowhere in the entire ecosystem of that thing does anyone try to replace light switches with taking out my phone and fiddling with an app, I'll at least listen.

It demonstrates to me how fundamentally unserious the IoT is that people keep coming back to trying to replace a light switch. Light switches are really good, guys. They're fast, everyone understands them, and they're reliable. Don't get me wrong, anything can be improved, but:

1\. Every IoT lightbulb thing I've ever seen has been strictly inferior to the light switch.

2\. Even if you did manage to actually improve it, how _much_ are you improving it, at what _cost_? I'm not going to pay $100 to make it very very slightly more convenient to turn on my lights.

</comment><comment><USER22> As a consumer, why would I ever buy a "smart device" from a startup that may not exist in 2 years?

<comment><USER23> Many of them aren't as 'smart' as they pretend to be, and can easily be MitM'ed. No subscription fees, local data storage. Depends on the functionality of whatever server it is connecting to, of course. I did this with a body scale - the online component is only storage.

Then again, most of this 'connected' crap is good as a hobby only. I like playing with it, but the actual added value is very low.

</comment></comment><comment><USER24> There are many pointless devices. But there are also devices that actually solve peoples problems. They will be the ones that emerge as real companies.

</comment><comment><USER25> Smart Pipe

[https://www.youtube.com/watch?v=DJklHwoYgBQ](https://www.youtube.com/watch?v=DJklHwoYgBQ)

</comment><comment><USER26> I'm pretty sure whoever wrote this article doesn't cook.

I'm a huge believer in data and its ability to make things better, whether for companies (seen for years with Business Intelligence software and whatnot) and now, with IoT, for individuals.

Problem is, now that we can gather thousands of data points about an individual (what exact time I woke up this morning, how well I slept, how long I brushed my teeth, the calories in my breakfast, what I had for breakfast, what time the sun came up, my self-reported mood, the length of my commute, how many stop lights I encountered...) we have no idea what to DO with that data. The state of IoT is ridiculous right now because every company is trying to cash in on it using the old model of marketing ("Surely if we know enough about our customers we can create ads that they will want to action on!") and this old model is 95% dead now that the majority of people get their content served up via the internet instead of newspaper/magazine and streaming instead of cable.

I believe that once the marketing field catches up with the way people are actually living and works in conjunction with the makers in technology companies coming up with products and solutions that actually make people's lives better then IoT will start to make sense. I have hope that this will happen... but am pessimistic about the timeline.

IoT means data, and to me "more data always means more better", but data is only as good as the questions you ask it and right now we're really, really bad at asking the right questions.

</comment><comment><USER27> For me the environmental impact is also a factor.I don't need a bunch of items sitting around wasting energy because they're all pointlessly wi-fi connected, have sensors, or are "always on and ready to go". And with more complicated parts things are more likely to break, so that's more likely to go in a landfill.

</comment><comment><USER28> I think what we need is a nice standard icon representing clearly that a product does NOT communicate wirelessly at all. Then let consumers decide which devices are useful to have connected as part of an IOT and which they would rather not.

</comment><comment><USER29> Needs JavaScript to read article text.

<comment><USER30> I also had to whitelist the site on ghostery. I'm not sure which one it was depending on.</comment></comment>


The Cedar Choppers - <AUTHOR> https://www.texasmonthly.com/the-culture/cedar-choppers-once-ruled-texas-hill-country/ <comment><USER2> The "cedar" in the Hill Country is actually the Ashe Juniper tree and it thrives in the area. It is drought tolerant in an area of frequent and prolonged drought and the hungry deer population won't touch the saplings while they obliterate the competing saplings from most other tree species that they are able to get to.

My parents live in the Hill Country and when I go to visit, I end up spending some time in the back of the property, barefoot with a brush hand saw, taking down large bottom branches from the Junipers. I do this for recreation and also to divert the growing energy of the tree upwards as it matures.

The aroma from the cut branches is quite pleasing and the systematic work of turning the branches into posts is satisfying. Usually, the long bottom branches are cut first, so much of the time, the work is in the shade of the upper branches, which can also transpire significantly on a hot day, with a prominent cooling effect. I can understand how getting paid for doing this might be an attractive life.

<comment><USER3> _barefoot with a brush hand saw_

Oh man that made me cringe. I've spent a lot of time on Texas Hill Country land (Kerville etc) and between rattlesnakes and chiggers you are one brave soul.

<comment><USER2> Something else is even worse. Fire ant mounds hidden under thick grass. If the ground feels a little softer than it should, hop along extra quick!

</comment></comment></comment><comment><USER4> My Dad lived in Austin in the 50s. He used to call poor, uneducated rural people "cedar choppers" and until today I had no idea where that came from, or that it was an Austin thing. It seemed -- weird -- because I never heard Dad say a perjorative word about any other group of people.

</comment><comment><USER5> The tree pollinates during a 2 to 3 month period during the winter months, and for many people, the allergies it causes are horrendous. If you're allergic to it, you're basically screwed from December - February. The pollen is so thick in the area that your car will covered with orange dust every morning.

<comment><USER6> From [https://nosneezes.com/resources/pollen- information/](https://nosneezes.com/resources/pollen-information/) :

"The highest pollen counts ever recorded anywhere in the world occur annually when these trees cut loose."

That source doesn't give any further information to back this up, but it doesn't seem too hard to believe.

<comment><USER7> what would be a definition of "cut loose"?

If it means pruned, sounds logical to me.

<comment><USER6> It's a seasonal allergy. I'm pretty sure that "cut loose" in this case just means when the time of year comes that the trees naturally start releasing their pollen.

I'm not a botanist by any stretch of the imagination, but I believe that once pollen is on the trees, the wind is the final thing that shakes it loose.

Here's a video showing what I'm talking about: [https://www.youtube.com/watch?v=QEYcBnE- IPA](https://www.youtube.com/watch?v=QEYcBnE-IPA)

</comment></comment></comment></comment><comment><USER8> > "They didn't care about possessions," Roberts says, "and they lived in > housing fit for animals."

I find it interesting how disrespectfully nose up folks talk about humble people like the Cedar Choppers, when if we all continued to live like they did, we wouldn't be in this climate mess we are in.

It's caring about possessions and extravagant homes ( _un_ fit for animals?) that has destroyed the planet for _all_ animals.

<comment><USER9> If we lived like they did, we'd probably also have half the life expectancy and way more deaths during childbirth, just to name a couple of differences.

<comment><USER10> That's an extreme of course, but we have examples from other small communities which shun luxuries to some extent like mennonite and Amish communities. In any event we don't need everyone to live barebones but even 30-40% volunteering or seeking this lifestyle could help against our mass consumption society. At least a significant portion of people wouldn't live connected lives 24x7. Maybe 12x7. That could reduce energy consumption.

<comment> 52-6F-62 I think the movement's already started to some effect with modern homesteaders.

[https://psmag.com/environment/opting-homesteading-matter- tod...](https://psmag.com/environment/opting-homesteading-matter- today-77062#.c4pghodw2)

[https://www.literallydarling.com/blog/2016/11/04/millennials...](https://www.literallydarling.com/blog/2016/11/04/millennials- find-home-homesteading-movement/)

<comment><USER11> That sounds good, but realistically, and perhaps somewhat differently from what most people imagine, denser city living (as opposed to suburban sprawl) is probably the best bet for the environment for most people who can't go live in the woods.

<comment><USER12> There's also a growing urban homesteading movement that combines the benefits of density with some of the benefits of self-sufficiency, a reduced environmental footprint, and the added bonus of potentially living in a city filled with green stuff!</comment></comment></comment></comment></comment></comment>


The Enemy Within: Male Fish Dislike Their Reflections More Than Competitors - <AUTHOR> http://www.wired.com/wiredscience/2010/05/cichlid-mirror/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+wired%2Findex+%28Wired%3A+Index+3+%28Top+Stories+2%29%29&utm_content=My+Yahoo

<comment><USER2> Couldn't this have more to do with the behavior of the reflection mirroring the fish's mental state? Perhaps 50% of the time when one fish shows aggression by flapping a fin or something the other fish makes a motion that indicates it will move on. That would diffuse the situation and lead to an instance where the fish don't attack each other. In the mirror, the reflection would mimic the aggression and would cause an attack more often.

Or perhaps fish know how to size each other up to avoid an attack and when they're the exact same size they attack to figure out who is dominant.

Calling it "the enemy within" just seems like we're trying to give the fish a very human-like trait without evidence.</comment>


Facebook in Silicon Valley talent battle - <AUTHOR> http://www.ft.com/intl/cms/s/0/228f5a74-2373-11e2-bb86-00144feabdc0.html#axzz2Ao8E73nP <comment><USER2> TIL ye olde Financial Times now has a cool web pricing table.</comment>


GitHub deplatforms Code of Merit Author Roberto Rosario - <AUTHOR> Rosario's account had been in a sort of limited state for years which didn't allow people to follow him on GitHub. According to sources close to GitHub, the platform had enough but had to wait until the Linux Kernel adoption of a code of conduct story died down to take down the Code of Merit repository. Terms of service violations were cited as the reason to take down the repositories and the account. In addition to the Code of Merit the repositories and organizations for Awesome Django and Mayan EDMS were taken down too. In total these three projects alone had more than 11,000 stars and 3,000 forks. Awesome Django and Mayan EDMS are two of the most influential and well known Django projects and second in popularity to Django itself. GitHub censorship is affecting thousands of developers that rely on Awesome Django for their work.
Only a copy of Awesome Django survives in the service named GitLab. As can be seen this is the 5th most popular repository in the entire site, and ranks higher than even some of GitLab's own products: https://gitlab.com/explore/projects/starred
GitHub joins the Python Software Foundation, the Django Software Foundation and the International Games Developer Association in their effort to silence Rosario and his movement against code of conducts in free open source projects. All three organizations had already removed or force Rosario to resign from positions in their organizations and have blacklisted him from their events.
Rosario's last appearance at a Django/Python event was in 2016 for the talk about his free open source heart monitoring system using Arduino and Django. At the beginning of the talk he confirms that talk will be his last public appearance at a Django/Python event, either as speaker or audience: https://www.youtube.com/watch?v=rubzEAojf-k
http://code-of-merit.org/ <comment><USER2> I would also like to know. I have watches the video and read the manifesto on GitLab. Was there some backlash?</comment>


N.Y.P.D. Plans Initiatives to Fight Terrorism and Improve Community Relations - <AUTHOR> http://www.nytimes.com/2015/01/30/nyregion/nypd-plans-initiatives-to-fight-terrorism-and-improve-community-relations.html

<comment><USER2> " (...) partol officers (...) more time to visit with community members (...) also include more high-powered weapons (...)"

Interesting to mention those two facts in one sentence.</comment>


Most powerful open letter from a gay pop star – Gay Star News - <AUTHOR> http://www.gaystarnews.com/article/this-is-the-most-powerful-open-letter-you-might-ever-read-from-a-gay-pop-star/?fb_ref=8a04021dc71346259e034fa20da85c99-Hackernews <comment><USER2> Powerful? I'm not sure. It needs an edit for grammar and clarity.

From what I can tell it is mostly trite "tumblr quality" writing.

> So yeah, gay people have sex, and it's not just gay people, it's all kinds > of people! All these non-straight people, they're out there, having sex! > Sex, between two consenting adults, can be a healthy, positive, safe and > enjoyable thing!

There is nothing new or interesting here. I remember er the same writing from the likes of boy George and George Michael.

Before that I'm sure it was the likes of Betty Friedan and the 60's and 70's feminists.

There's nothing powerful about this.

He can press his genitals against anything he wants (*with all appropriate caveats)</comment>