Ask HN: what features do you want in a geocoder? - <AUTHOR> Hi,
we're building a public facing geocoding service (forward and reverse) on top of our own technology, OpenStreetMap, and various others open geo services. What features would make such a service compelling for developers? What is your wishlist?
Thanks for taking the time to answer. <comment><USER2> Depends on what your target customers are doing!

For a delivery company, inaccurate results can send drivers to the wrong place, so it's important to get the best accuracy available.

You want it to work way out in the country, where buildings are few and far between; and with named buildings as well as numbered ones. So a search like [1] should come back accurate to less than 100m.

Fuzzy/imprecise matching should be used with care. If there's a search for Manor Close in London, it should ask which of the four Manor Closes you mean [2]. If the only part of the address that matches is London, that's not enough information to send a delivery driver - the address should be rejected.

If there are parts of the address you can't match that's sometimes a problem - you don't want to map 1 Hopton Parade, Streatham High Road to 1 Streatham High Road. But you do want to map Some Company Ltd, 1 Streatham High Road to the latter.

On the other hand, if your target users are dating websites wanting to show rough distances between members, just matching city might be plenty accurate enough; property search engines like Zoopla will show any shitty approximation on their maps if they don't recognise a street or postcode.

[1] [https://maps.google.co.uk/maps?q=Paradise+Wildlife+Park,+Whi...](https://maps.google.co.uk/maps?q=Paradise+Wildlife+Park,+White+Stubbs+Lane,+Broxbourne) [2] [https://maps.google.co.uk/maps?q=from:E17+5RT+to:NW7+3NG+to:...](https://maps.google.co.uk/maps?q=from:E17+5RT+to:NW7+3NG+to:NW9+9HD+to:SE28+8EY)

</comment><comment><USER3> A downloadable bulk geocoding service. Some address databases are not licensed for exposure to 3rd parties, but geocoding is very interesting.

If you added CASS or address validation (deliverability) service it would increase the value even more.

Having recently installed PostGIS and imported loads of TIGER data, it would be useful if you provided some discussion about data sets backing your geocoder, especially if you do better than just TIGER.

I am only interested in U.S./Canada addresses (thus the mentions of CASS/verification) but I understand OSM is a global project.

<comment><AUTHOR> thanks for the feedback. If I may ask, there are several people who supply exactly what you're asking for. Why are you not using them?

To be open with you, the North American market is well served and highly competitive, many other parts of the world are the opposite. Thus the US/CA probably won't be our area of focus. That being said, very keen to learn what isn't working for you in current solutions.

<comment><USER3> I am building a PostGIS/TIGER/PAGC/other data engine to improve address data. It comes in dirty, I want to improve it as much as possible: Normalize, geocode, verify, correct. Correction is basically a pipe dream for bulk operation but it's needed for point-fixes. Everything else is feasible in bulk. I haven't found all the features under one hood yet.

It has to be downloadable because I am not allowed to send my data out to third parties. It has to be free because I have a LOT of data. The North American markets for shipping packages are well served by multiple products in this space; efforts to contextualize/augment poor quality North American address data are not well served.

Hope that helps!

</comment></comment></comment><comment><USER4> Ability to report inaccurate addresses without just telling the customer to go to OpenStreetMap and edit it.

We have a lot of problems with Google Maps not knowing about odd addresses like 235B Whatever St. or 113-76 XYZ street. This is perhaps because its a new construction and the address isn't in the database yet.

Town names in the Hamptons are quite contentious. What the post office knows it as is not what the locals call it, and nothing like what the real estate agents call it (but they make up their own and they lie about the locations).

But primarily you need to parse native language and messy addresses. 2 Ave, First Street, Fifth Ave, Madison Ave (same as 5th), CPW (central park west)

This is where google wins.

<comment><AUTHOR> It's funny you mention that, as our main business is the real estate search engine Nestoria [http://www.nestoria.com](http://www.nestoria.com) We parse about listings 15M addresses in 9 different countries every day (though not the US). We work in pretty chaotic markets like India and Brazil. The world is a very diverse place, but there is one constant - agents do not feel the need to let themselves be bound by the "on the ground" truth of where a listing is.

Regarding Google, you are right, they do a good job, especially in the US. Full credit to them. The problem is the cost and usage restrictions.

Coming back to your initial point of reporting inaccuracies. What would be your preferred way to report problems? some sort of API you could automate? Would you just tell us there is a problem, or want to also tell us the solution?

<comment><USER4> If you are using OSM then it would be for you to take the change request and then perform it yourself. Don't plan to do this at scale - just do it so that your client thinks they are getting full service. They wouldn't bother to do it very often.

But the real estate agents do get frustrated when its a real mapping error.

India - in Chennai they have 2 numbers on each house : old numbering and new numbering. So addresses are shown as 17 : 34

and the rickshaw drivers have never seen a map in their lives. I hold up a google map on the phone and they stare fascinated at it for minutes, but it really has nothing to do with the reality of getting there. Many people don't even know the names of the streets and they change the names all the time. Its done by corners and what landmarks are there.

curious - how did you come up with the name nestoria ?

<comment><AUTHOR> Regarding the name, we just did some brainstorming and a member of the team just came up with the name, nothing fancier than that. Our requirements were

\- domain available \- written the way it is spoken \- shorter is better

in our old logo we used to use the "nest" image, but that only makes sense in some languages. Last year we went with a more modern look.

</comment></comment></comment></comment><comment><USER5> One thing that I make use of but don't see too many services providing is some kind of 'match level' \- where the geocoder returns a code indicating how confident it is about the quality of it's result. A result of 1 might mean a building level match, while 100 might be street level etc.

IIRC Google's geocoder does something like this, but it's pretty inaccurate, overstating it's match level consistently.

As others have said, geocoding is very hard to do well, but I commend the efforts being made with Nominatim and komoot/photon.

<comment><AUTHOR> agree, a simple to understand confidence score is critical.

Also agree nominatim and photon are impressive

<comment><USER3> FWIW I ran millions of addresses through PostGIS geocoders (using both TIGER as well as PAGC's normalizer functions) and found that MOST addresses geocoded with confidence level 0 or 1. 60% were 0 or 1, the other 40% were spread across ratings 2 - 100.

I don't have a great way to characterize the geographic coverage or data quality of the geocoder, but it is clear that it has a data set which must be maintained to support geocoding into the future. Soon I'll have to start figuring out how long my current data is useful, and how long it will be before the next update from the census bureau.

I'm starting to think that it's crazy for so many businesses to need reliable GIS data and have so few sources to go for it. With the right organizational structure, we could be croudsourcing it it daily.

But I digress.

</comment></comment></comment><comment><USER6> One thing that our application (processing real-estate data feeds) needs is the ability to figure out an approximate location if the address is a little vague.

For example, when we a vague address such as "Ichobod Crane Circle", we still want to get a position because that road is very short. However, if the address is something like "Sleepy Hollow Road" or "Murders Kill Road" (Coxsackie sure has some weird names), those are very long roads, and placing a marker anywhere on them would be meaningless.

Google solves this for us by providing, in the results, the bounding box of the result. When the match is not "street_address" but something else such as "route", "premise", "point_of_interest" or the like, what we do is take the bounding_box, calculate the area, and use the area if it's less than 500x500 meters. It's not optimal, but it's better than having no location at all.

Another thing that Google does semi-well is constrain the search to a specific area, like a country or a state. Unfortunately, Google doesn't let you pass in more than one state, but other than that, it works well. Some of the addresses we get are so vague that they would geocode to other countries (Oxford, England instead of Oxford, NY) if it were not for this filtering ability.

<comment><AUTHOR> we have a long history in real estate, very familiar with exactly the problems you describe.

</comment></comment><comment><USER7> Awesome. We need bulk requests (one or more lat/lng), and reverse geocoding with locale components (state, county, city, neighborhood) Extending tzaman's localization request, a globally unique identifier, e.g. ISO code, for every piece of locale when reverse geocoding is critical for us. When storing reverse geocoded points in our own database I want to key off the unique values but lookup the locale specific versions later on client devices (ideally via REST or an offline API if possible).

/geocode?latlng=47.639548,-122.356957&language=en,fr

{ "ISO":{ Country:"US", Administrative:"WA", SubAdministrative:"King", Locality:"Seattle", SubLocality:"Queen Anne" }, "fr":{ Country:"Etas-Unis", Administrative:"Washington", SubAdministrative:"Roi County", Locality:"Seattle", SubLocality:"Renne Anne" } "en":{ Country:"US", Administrative:"Washington", SubAdministrative:"King County", Locality:"Seattle", SubLocality:"Queen Anne" } }

<comment><USER8> Can I dig a little deeper into this as your example has me indulging in some furious head scratching.

Providing language synonyms makes perfect sense where these exist (cf: London in English, Londra in Italian, Londres in French).

But your example implies translation of place names into their language specific equivalent. Kings County in Washington state is, unless I'm mistaken, Kings County in all other languages. Although the local residents may disagree, this county isn't blessed with a language synonym as it doesn't fall into the (ill defined) category of "well known place with a language variant".

Unless you're suggesting that if, say, French is requested as a language, a geocoder should translate place names so "Kings County" would (maybe) be "Comté Roi" in French. Although this approach sounds odd to me as (AFAIK) no one else refers to this place in this way?

<comment><USER7> Sorry for the confusion, let's see if I can clear this up. We'd like to see these locales translated to the same names that the native map program on a device would show (which is what I rely on now). For example, on my iPhone, when I switch to French and I'm in the Mission District of SF it says "Etats- Unis" / "Quartier de la Mission". Actually, that's a bad/rare example, a better one is a multi-lingual country like Switzerland where you can have 3 languages at once for some cities/neighborhoods. I want to pass the locale of the speaker to the API and get back what they'd expect in their local dialect.

Make sense?

<comment><USER8> Makes a lot more sense, yes. Thanks for this.

I do have to ask how realistic this is. It makes sense for places that do have multi language versions. So "Etats-Unis" for the US, when in French, makes total sense.

But does translating the Mission to "Quartier de la Mission" make sense? It makes me go "errrr?".

To put it another way, taking the ever present UK "High Street" as an example. I'd expect to see this as "High Street" regardess of language and not as "Grande Rue" because no one ever says this of uses it.

So yes, I think passing the locale to the service makes a lot of sense. Supporting those places which have multi language versions makes a lot of sense too. Translating all place names to a specific language makes less sense.

Or am I missing the point? It's always probable.

</comment></comment></comment></comment><comment><USER9> God, no. Everyone is trying to build a Geocoder and everyone is failing, because no one is actually realizing that geocoding is probably the most complex topic in GIS.

"Yeah, let's put some Elasticsearch and PostgreSQL and it will work out fine." No, it won't - you have no idea. And of course you won't believe me, but let me list some problems you will have that you don't realize right now:

* There is a lot of different charsets. Latin, kyrillic, there are umlauts, RTL, weird abbreviations, language standards that you don't know, because you don't know enough about foreign cultures.

* It's a shitload of data: OpenStreetMap is expanded about 700 GB large (not including history). And you will want to have autocompletition or autosuggestion, so response times will have to be < 100 ms.

* Ranking. Your user types "Tokyo". Is it the restaurant next to the user, is it the capital of Japan or is it some village next to Shitfuckistan?

No matter what, it will take you about a year to get any usable result. So I suggest you to look into Nominatim (the standard geocoder of OpenStreetMap which has actually got a lot better) or Photon (a geocoder based on the Nominatim DB, but with auto suggestion).

<comment><USER10> > or is it some village next to Shitfuckistan?

Wow, where did that come from?

<comment><USER11> I'm going to guess it comes from the frustration of getting dumb results rankings that don't take into account the likelihood that the result is actually something you wanted.

</comment></comment></comment><comment><USER12> I've used pretty much all of the big geocoding services, and here are problems I've ran into.

1\. Rate limiting: I get it, you have to make money and/or limit your freeloading, but rate limiting has killed things I've built in the past, especially Google's hard rate limit. A soft rate limit, or an alternate way to monetize, would be huge.

2\. Accuracy: MapBox's geocoder is not good. Aside from inaccurate map tiles, their geocoder misses entire US zip codes. PLEASE at least include helpful error messages and a path to report incorrect results.

3\. A solution for shared IPs and rate limiting. I have helped several small websites that do not come close to approaching Google's daily rate limit, but because their IP was used by someone else, they are not allowed to make geocoding calls. This forced us to use a different service.

Honorable mention: It would be nice to be able to specify what data I get back from a call. If all I need is lat/lng, I don't need another kilobyte of neighborhood/city/time zone info in my result.

Hope this helps.

<comment><AUTHOR> It helps a lot, thanks.

re: alternate ways to monetize, what do you propose?

<comment><USER13> Two options the way I see it:

* Offer unlimited access for more money (popular) * Offer a cheap, simple way to batch requests of very large and/or unlimited sizes (e.g., CSV upload)

The latter is nice, because it's not worth it for me to upload a CSV file with 1 address in it -- I might as well use your regular API. And it's not worth it for just 25 addresses, either. There's some threshold where it becomes more useful for me to submit my addresses in bulk, and that's where the CSV files come in. It should be way cheaper for you to process a file with 1 million rows than it would be to process 1 million API requests, so if you were to do that, it would be a gold mine for businesses like mine that require geocoding capabilities for millions of addresses at a time.

<comment><USER14> I really don't want to hijack the thread, but I couldn't help notice that the company I work for[1] recently added a lot of these features which might be of interest.

We have support for both bulk CSV upload and an API endpoint for batch geocoding. We are also starting to introduce unlimited access for a flat monthly fee (with no limits to requests per sec), please contact us if you're interested [2].

[1] [http://geocod.io](http://geocod.io) [2] firstname.lastname@example.com

<comment><AUTHOR> Hi Mathias,

big fan of your company's approach, congrats on your progress. The one weakness is that it's limited to just the US.

</comment></comment></comment></comment></comment><comment><USER15> An understanding of colloquial geography, such as:

\- informal place names

\- boundaries of neighbourhoods

\- nesting of those things within administrative boundaries

Yahoo's Where On Earth database had a lot of this, but it doesn't seem to be available to download any more, and they didn't accept updates. GeoNames is pretty messy and inaccurate, and the copyright status has never been cleared up.

<comment><AUTHOR> Hi Tom,

thanks for commenting. Big fan of your work on flickr neighbourhood boundaries.

We hear you and are on it, which doesn't mean we'll be perfect of course, but definitely aware of this issue.

<comment><USER13> Also, beware of places like Carmel-by-the-sea, CA, where there are no street addresses! None of the houses in downtown Carmel have mailboxes -- they all have P.O. boxes at the post office downtown. If you try and geocode these, you wind up with the post office lat/lng and not the house! Frustrating as hell...

</comment></comment></comment><comment><USER16> * Support for and intelligent detection of a variety of coordinate formats including LAt/Lon, utm, and township and range. This would be really useful when dealing with old well or surveyors logs etc.

* Better support for terrain features. Google is getting better here but for a while "Mount Rainier" was sending you to the parks business office.

* Better support for localized search. This ties into the last one, a frequent use for me is to be zoomed in on a general area and want to find an obscure creek or peak.

* Better support for non driving use cases. Google has a nasty habit of resolving things like unquoted locations to the nearest drivable street address which is really stupid when you are using it to find a wilderness lake or something.

* Finer grained search by type.

(FWIW I run hillmap.com so most of my desires spring from the needs of a service targeted at hikers and backcountry skiers.)

<comment><AUTHOR> thanks, good feedback

</comment></comment><comment><USER17> Simple.

I've worked in GIS for a number of years. I've worked on marine and scientific data management on top of GIS support. From google maps/earth to ArcGIS and pulling data from KML to OGC services.

If a service takes me nearly a month to learn to use, I'm going to push adamantly to use something else.

</comment><comment><USER18> * API client with batching and parallelization built in (100 queries in a single request, multiple requests run in parallel, etc.)

* robustness in the face of bad street suffixes (for example, in Burlington, VT, you may find data with "CR" meaning "CIRCLE" instead of the official USPS "CREEK")

* fuzzy street name matching (PAKCER -> PACKER)

* accurate geocoding in rural United States

* fuzzy international place matching (like "ST PANCRAS ST STATION" in London)

<comment><AUTHOR> In your experience is the rural US geocoding problem a software problem or a lack of underlying data?

</comment></comment><comment><USER19> Don't forget desktop and mobile applications. Most mapping and geocoding services do.

\- I have to ship my API keys to end users. Someone could grab it and repurpose it \- Rate limiting by API key penalizes one end customer for the other's misbehaving

I would love an API that is aware of the end user. Applies rate limiting on a user basis. Allows for anonymized user-based usage report. E.g. number of end users, average number of API calls by users, …

<comment><AUTHOR> your comments generated a lot of discussion for us here, thanks. Our conclusion: if you build an app (mobile, desktop, whatever) that becomes popular and depends on a third party service, in our case a geocoder, it generates real costs for the thidd party service. So there are three potential groups who can pay the cost

a. the third party service provider can just provide service for free. We can't, at least not indefinitely.

b. the end consumer can somehow be billed by the third party service. Feels complicated, especially as the use of the service may be deep in the internals and behind the scenes of the app. The consumer may well have no idea it is being used

c. the application developer can pay. Either directly or via billing the end consumer.

Option c. feels like the only sustainable one. Happy to hear your thoughts on it though.

<comment><USER19> Option c is the way to go. Up to the app developer to see how to monetize the app.

Bonus points if the end user can be identified. E.g. if the app can pass an opaque token to the web service. Reporting / billing from the web service provider groups usage by token.

</comment></comment></comment><comment><AUTHOR> Thanks everyone for the feedback, very useful. Please keep it coming. I need to be offline for a bit, but will check in later. If you're interested in learning more about our progress please follow us on twitter. ta.

[https://twitter.com/opencagedata](https://twitter.com/opencagedata)

</comment><comment><USER20> Apart from the most obvious (being accurate) I would say well documented API and properly localised results

<comment><AUTHOR> thanks for commenting. If you don't mind, what exactly do you mean by "well localised"? Can you give me an example, ideally via a service you're currently using that is doing it badly. Cheers.

<comment><USER21> The Google geocoding service returns a normalized address in addition to the lat/lon. There are times I need that address localized for the area it is in, and other times I need it localized for another culture. It would be very useful to be able to specify which locale the return value should be.

<comment><AUTHOR> makes sense, can you give me a specific example?

<comment><USER11> In China all the maps are in Chinese. There are times when I want them in English. I assume this is what the poster means.

<comment><USER20> Partially yes, plus the formatting, for example, some locales have zip in front of the city while some do it the other way around.

</comment></comment></comment></comment></comment></comment><comment><USER22> Get the timezone (and related information: utc offset, local time, etc) from a lat, lng point or address

<comment><USER14> Hi Hernantz,

Shameless plug, but this is something we recently started offering [1] for our geocoding service. I'm happy to help if you have any questions.

[1] [http://geocod.io/docs/#toc_21](http://geocod.io/docs/#toc_21)

</comment></comment><comment><USER23> Quick and cheap daily batch geocoding with any type of export option, from CSV to JSON to XML.

<comment><AUTHOR> Not sure what your definitions are of cheap or quick, nr of what country your data is in, but there are lots of people who do bulk geocoding. Why don't you use them?

<comment><USER23> I do use others. The op here was asking what we want in a geocoding service so I told him.

But to elaborate on what I mean by quick and cheap:

Quick means I don't have to wait for an email notifying me that it's done, and I don't want my requests queued for a couple hours. I want them upon request.

Cheap means better than the average price of the competitors. It would be nice to maybe pay $100 and be able to geocode 100K addresses, for example (but I don't really have a huge sample of competitor prices).

<comment><AUTHOR> relax dude, I am the OP. Just trying to get specifics not general terms like "cheap". For one person $100 is rounding error, for the next it's a meaningful chunk of project budget. Thanks for clarifying.

</comment></comment></comment></comment><comment><USER24> Publicly visible and flexible pricing. Not $10k+ per year as Google Maps API.

</comment><comment><USER25> Fast for single queries, not for batch geocoding?

So a web query can come from a user of a web service, the external(?) geocoding API can be called -- and the reply can go back to user [applying lon/lat processing] without waiting too long.

(I don't do anything like this for a while, so please apply NaCl.)</comment>


Uniform Structured Syntax, Metaprogramming and Run-Time Compilation - <AUTHOR> https://m00natic.github.io/lisp/manual-jit.html <comment><USER3> This is very impressive in only the way that lisp can be. But part of why it works is because SQL is a well defined interface. There was a comment on ltu a good while back:

\</comment><comment></comment><comment>

"I would say that a system that allowed other metathings to be done in the ordinary course of programming (like changing what inheritance means, or what is an instance) is a bad design. (I believe that systems should allow these things, but the design should be such that there are clear fences that have to be crossed when serious extensions are made.)"

The fact Kay realized this fine point of design in the late '60s (according to him) is why he is a Turing Award Winner.

I know Lisp programmers who even today don't understand this point - <USER2> code is succinct but the API has a massively unnecessary learning curve due to unclear boundaries.

Sometimes my coworkers object to me paying extraordinary attention to detail about what the boundaries are. However, if we don't pay attention to boundaries, we may as well all be Netron Fusion COBOL programmers munging VSAM records and EDI data formats. By Z-Bo at Wed, 2009-04-15

[http://lambda-the-ultimate.org/node/3265#comment-48165](http://lambda-the- ultimate.org/node/3265#comment-48165)

<comment><USER4> I think more analogous to the examples Kay was giving would be for a single program to quietly change the rules for function application, or to quietly change the behavior of CLOS.

I think of an embedded DSL, based on a macro like it is here, as a bit different than that, because it's not doing anything quietly -- it applies only to the parenthesized extent of the code that begins with your macro name:

      (my-dsl-macro my dsl fills the rest of the parentheses)

People who don't already know what `my-dsl-macro` is see the documentation or code that says it's a macro. Even were it a normal function, people reading the code should probably know what the function does, so maybe they have to glance at where the IDE automatically displayed the syntax or documentation for them, anyway.

Ideally, your editor will also color the macro names differently, especially for readers who don't know what's a macro or a function. (Rust adds an exclamation mark to the macro name for a use, which is a good idea when you don't already know what's macros, but maybe a bit annoying to have all those exclamation marks when you do know that, say, `println` is a macro.) But even if you don't know the name is a macro, you'll be clued in if the text within it doesn't look like your top-level language, which it often doesn't (e.g., SQL in s-expressions doesn't look like base CL).

A key is to use DSLs judiciously -- for improved readability (for your base language programmer, or domain experts), maintainability, and/or performance. Maybe not for, say, a convenience for the sake of minor code terseness improvement only.

Of course, SQL is a whopper of a language, and perhaps overkill if you invented it as a DSL for this particular application. (More suspicious would be to invent your own relational query language that seems gratuitously different than SQL, when SQL already existed.)

As for changes more like I think Kay was talking about, in the Racket (Lisp family) universe, outside of a macro, normally you wouldn't do that, but when you have a good reason -- say, you want to prototype a lazy language, or implement a specialized language for a GPU programming backend, or a DSL for your domain experts -- you can. In that case, you'd have a `#lang` line at the very top of the file that tells you what different language this is, instead of `#lang racket`. You might make that produce modules that can interoperate with modules of other `#lang`s, but you're not doing anything sneaky that breaks the language of those other modules.

<comment><USER5> You still have the problem that the `my-dsl-macro` is a black box.

That macro can do anything it wants to the sub-AST. The problem is not about hygiene re: variables but about the meaning of symbols. A function application within it that doesn't obviously have anything to do with the DSL is also subject to the macro's will to have its meaning changed. This can make composition difficult. Who guarantees that if you combine `my-dsl-1-macro` with `my-dsl-2-macro` or simply with general purpose code, that they don't interfere in odd ways?

Granted, this is not a big concern for a well-designed and widely used macro. However, the bottom line is that the inherent freedom of the abstraction allows for issues to arise whose debugging require a deep understanding of the involved macros and their implementations.

You don't really have the same problem if there is only procedural abstraction. The boundaries are clearer there. Even if you do something like the Interpreter Pattern or Haskell style AST-constructing EDSLs, you have guarantees that whatever AST is constructed cannot inspect whatever non-DSL code you combined with it.

<comment><USER6> > Who guarantees that if you combine `my-dsl-1-macro` with `my-dsl-2-macro` or > simply with general purpose code, that they don't interfere in odd ways?

Nobody does. But that's where the power lies. You can't have unrestricted power with restricted features. Lisp gives you unrestricted power. It's up to you to use it correctly, for w/e definition of correct _your_ context makes sense.

<comment><USER5> Yes, but the argument was rather along the lines of "here's why I think (some people think) we shouldn't have unrestricted features" or "I don't want ... because...". Languages don't necessarily _need_ macros (many don't have them) so arguing on the basis that the dangers of macros are an inevitable cost for having them is correct but it works under the assumption that macros are indeed desired by everybody.

It's a bit like arguing about pointers and memory safety.

I do think there is a lot of value to look into these things and research alternatives to macros. For instance the problem I have described could be addressed by a new macro-like feature that has to respect certain boundaries.

</comment></comment></comment></comment></comment><comment> m-felleisen Does it matter that McCarthy won the Turing Award before Alan Kay? What a silly point.

Is it awful that functions are black boxes? That they are bundled in libraries?

Does it matter that for loops don't leave a stack trace? How can poor programmers debug them w/o a stack trace?

Can you imagine that overriding a method completely changes what a bundle of methods does in a class?

It is so sad that every time a new idea comes out, the "resistance" (programmers who often lack experience with it or experienced an inferior implementation of the idea) are the loudest to complain and hold sw dev back.

</comment><comment><USER7> This is talking about evaluating dynamic expressions over (semi-?)-structured row-oriented data, for the purpose of filtering.

It contrasts a tree interpreter in C++ with a JITted dynamically generated Lisp expression, with some hand-waving away of what the equivalent JIT in C++ would be, seemingly dismissing it as taking too long (is that what "unpause cosmic time" alludes to? I'm not sure).

The tree interpreter is a little unorthodox - it isn't how I'd write an AST- walking interpreter - and other interpreter techniques like generating linear programs for a simple virtual machine aren't considered. These can be pretty fast, especially with some use of implementation-specific computed goto, available in gcc and clang. It would get rid of the author's worries about recursion and lack of TCO, increase locality and decrease cache usage.

But of course there's not much need to write such an interpreter. Why not use a JIT framework for C++? Depending on the library, it wouldn't be a whole lot more complex than a traversal of the AST.

And the next question is, if the problem is querying plain-text databases, why not use Apache Impala? It's written in C++, and uses LLVM to compile SQL expressions into native code, and can evaluate filters (but not just filters, the full power of SQL) over CSV text.

Maybe Impala and its dependencies is too big, but if that's the case then your data is small and a simple interpreter would be plenty fast enough.

<comment><USER6> Unless you provide one of

\- a simple VM implementation \- code using a JIT framework \- using Apache Impala on this particular dataset

which performs on par _and_ didn't take astonishingly long to write then these are just vague claims and hypotheses.

<comment><USER8> A lot of people have experience with simple VMs. There's nothing super unique about this problem that requires every claim about it to be accompanied by a proof by construction.

<comment><USER6> Either you missed the context or are pulling a straw man. Building a simple VM is not unique, but can you build one that will be competitive with the other's in a reasonable timeframe?

<comment><USER8> Yes, I could. Sorry, I wasn't trying to fight a straw man. I sincerely thought you were saying you wouldn't accept estimates of the project's difficulty in lieu of proof.

</comment></comment></comment></comment></comment><comment><USER9> I'm not sure if I'm missing something, or if this is meant to be allegorical for some other, more difficult problem, but the argument here seems very strange. For sure, C++'s string handling is an awful sight, but the jump to DSLs seems unmotivated. This issue can be handled with simple, traditional helper functions.

        select(recordS5, [](cxr, subcode, commercial_name, date_disc) {
            return cxr.like("YY|XX") &&
                   ...etc;
        });

<comment><USER10> It's allegorical in the sense that this is general Lisp technique useful not just in this case. The DSL is targeted at (non programmer) end users and supposed to be fired through a REPL. Wouldn't want to make them write C++ with lambdas, semi-columns and whatever syntax traps of the latest standard.

<comment><USER9> That doesn't support the post's argument, that it's _not_ about aesthetics, but about qualitatively simpler solutions.

> Often times I hear the claim that (programming language) syntax doesn't > matter or if it does, it's only to provide some subjective readability > aesthetics. It's somewhat surprising to me how large majority of supposedly > rational people like programmers/computer scientists would throw out > objectivity with such confidence. So let me provide a simple real world case > where uniform syntax enables out of the box solution which is qualitatively > simpler.

That making non-programmers use complex syntax and type semicolons is bad is fair, but it's a rather different claim than the post's.

<comment><USER10> Well, dynamic queries by end users is the main goal here. Your static helper functions are completely unusable in that context. Analysing a query and generating code at runtime is easy and idiomatic with uniform syntax (and accompanying language support) and the claim is that solution is not only speedier but implementation in CL is qualitatively simpler than the alternatives.

<comment><USER9> The point my first comment made is that you don't need to analyse a query at runtime, you just need to provide functions. I will agree again that Lisp makes REPLs easier to write than does C++, but a CL with Lua syntax could still, just as easily, provide functions and expose a REPL. It's the same solution, just without the unjustified AST transformations.

<comment><USER10> How can you provide a REPL language without analyzing it at runtime? Write Lua random syntax in the REPL? Not a great improvement over C++. Not to mention that you'll probably use something like `eval` which is not compilation thus inferior.

By the way, even your original example - I can't see how it can work honestly. How can you identify fields through (lambda) parameter names only (no mention of types either)? Probably the least boilerplate-heavy solution would be stringly typed.

<comment><USER9> A Lua REPL is hardly worse than a Lisp one.

> something like `eval` which is not compilation thus inferior

I honestly don't know what that means. Turning text into code _is_ compilation; there is no difference between the two in that regard, except perhaps that in the Lisp DSL case it's more manual.

> How can you identify fields through (lambda) parameter names only (no > mention of types either)?

Not familiar enough with Lua, but in Python you just use keyword arguments.

</comment></comment></comment></comment></comment></comment></comment><comment><USER11> This looks super interesting, but very difficult to read on mobile with Chrome (Android, Samsung S10e):

[https://i.imgur.com/bJw95FB.jpg](https://i.imgur.com/bJw95FB.jpg)

Rotating sideways helped to a degree, but in hopes the author sees this thread..

Update:

Sadly, Firefox isn't any better:

[https://i.imgur.com/ONswSNb.jpg](https://i.imgur.com/ONswSNb.jpg)

I guess this means this instance can't chalked up to "Chrome is becoming the new IE6".

 _grin_

<comment><USER12> True.

Fortunately, there's a workaround - "request desktop site" in Firefox (and Chrome) menu.

<comment><USER11> Oh, wow, this is excellent, and fixes it!

I never knew about this option. Thank you, sincerely, TeMPOral!

<comment><USER12> You're welcome. It's a life-saver, in cases like this, or when a mobile site disables the ability to pinch-zoom.

<comment><USER13> Pinch zooming can also be force-enabled for all sites in the browser's accessibility settings.

</comment></comment></comment></comment></comment><comment><USER14> If I remember correctly, Postgres was initially developed in Lisp, but then rewritten. Was that a mistake, or evidence against the thesis of this article?

<comment><USER15> Not really, the original Postgres was developed in a mix of 17000 lines of Lisp and 63000 lines of C. This was difficult to develop/debug at that time. Probably still would be.

It had a 'gigantic' memory footprint of 4 MB - the all-in-C version only used 1 MB. The Lisp version was also slower and they didn't use features like GC...

[http://db.cs.berkeley.edu/papers/ERL-M90-34.pdf](http://db.cs.berkeley.edu/papers/ERL-M90-34.pdf)

<comment><USER14> > By a process of elimination, we decided to try writing POSTGRES in LISP. We > expected that it would be especially easy to write the optimizer and > inference engine in LISP, since both are mostly tree processing modules. > Moreover, we were seduced by AI claims of high programmer productivity for > applications written in LISP.

Yes, that's what I remembered, they started out using LISP.

> Our feeling is that the use of LISP has been a terrible mistake for several > reasons.

"Terrible mistake" is pretty unambiguous language.

<comment><USER15> > Yes, that's what I remembered, they started out using LISP.

Only for the optimizer and the inference engine.

The authors also had no prior experience developing an application in a hybrid of C and Lisp.

> "Terrible mistake" is pretty unambiguous language.

4MB memory footprint was a terrible mistake at that time.

<comment><USER14> They obviously started writing Postgres in LISP, because _" we soon realized that parts of the system were more easily coded in C"_ wouldn't make sense if writing a hybrid had been the initial plan.

They tried going all LISP at first, and failed. Was it them, or was it LISP?

<comment><USER15> Since they had no experience in Lisp programming, they chose the wrong language just for 'doing something different'.

Writing a database in a performant way isn't something for a Lisp newbie.

"By the time Version 1 was operational, it contained about 17000 lines of LISP and about 63000 lines of C".

Version 1 was written in a mix of C and Lisp.

That's also not surprising, since that would have been a common approach for some technical reasons. But it's a bit difficult to do - again, especially as a newbie.

> Was it them, or was it LISP?

Their lack of experience, their approach, the LISP implementation they were using, the hardware constraints (4 MB footprint was not acceptable to them), ... A conservative approach using a lower-level systems programming language like C was a good choice at that time and they were much more successful with that approach.

There were/are a bunch of databases written in Lisp and even in a mix of C and Lisp: Statice (Symbolics), Zeitgeist (TI), Orion/Itasca, AllegroStore (Franz), ... But they were written by Lisp experts.</comment></comment></comment></comment></comment></comment>


Engineer spends 15 years fireproofing his house and saves it - <AUTHOR> https://www.businessinsider.com/california-fire-fireproof-home-sonoma-county-kincade-2020-2#but-for-arai-making-it-through-the-fire-was-bittersweet-as-many-of-his-neighbors-were-forced-to-move-when-their-homes-were-destroyed-im-happy-to-have-survived-it-arai-said-but-you-know-im-very-unhappy-that-i-lost-all-my-neighbors-16 <comment><USER2> That's an interesting paragraph. I wonder why they didn't write an article about it?</comment>


Laziness Impatience Hubris - <AUTHOR> http://wiki.c2.com/?LazinessImpatienceHubris <comment><USER2> "I divide my officers into four classes as follows: The clever, the industrious, the lazy, and the stupid. Each officer always possesses two of these qualities. Those who are clever and industrious I appoint to the General Staff. Use can under certain circumstances be made of those who are stupid and lazy. The man who is clever and lazy qualifies for the highest leadership posts. He has the requisite nerves and the mental clarity for difficult decisions. But whoever is stupid and industrious must be got rid of, for he is too dangerous."

General Kurt von Hammerstein-Equord, probably

<comment><AUTHOR> "Gilbreth studied the methods of various bricklayers—the poor workmen and the best ones, and he stumbled upon an astonishing fact of great importance and significance. He found that he could learn most from the lazy man!"

"Most of the chance improvements in human motions that eliminate unnecessary movement and reduce fatigue have been hit upon, Gilbreth thinks, by men who were lazy—so lazy that every needless step counted."

"Another important thing Gilbreth noted was that the so-called expert factory workers are often the most wasteful of their motions and strength. Because of their energy and ability to work at high speed, such men may be able to produce a large quantity of good work, and thus qualify as experts, but they tire themselves out of all proportion to the amount of work done."

\- Popular Science, 1920, [https://quoteinvestigator.com/2014/02/26/lazy- job/](https://quoteinvestigator.com/2014/02/26/lazy-job/)

<comment><USER3> I got an interesting personal education on this working on some volunteer construction projects down in Mexico. The locals all move with a very deliberate slow pace. Us gringos tried to be more gung ho, and by 11 am the heat plus our effort had destroyed us. Meanwhile the local folks just kept on truckin' until sunset.

<comment><USER4> Sustainable Pace is one of those good buzz-terms to throw around - sure you can be fast, but is it sustainable?

It's a good attitude to have in life and work as well - sure, you can do 16 hour binge working days, but for how long can you sustain that? Can you keep that up until retirement, whenever that may be? Is it worth it?

</comment></comment></comment></comment><comment><USER5> I think laziness and impatience have always been listed in a sort of tongue in cheek way. Patience is probably the single most important attribute a programmer can have, many '10x' engineers are just the people who have an unshakeable ability to step through boring code for 4 hours until the problem reveals itself.

Laziness pays off for me when I argue for simpler requirements and paring down of features on things. It results in a better end product but most of my motivation was that I just didn't want to build all the stuff!

<comment><USER6> Laziness to me is thinking "Eh this doesn't need tests" or "It might be possible to break it this way but I doubt anyone will try that"

What people seem to mean by it is the ability to stop and think "This task is repetitive and tedious, we are passing the threshold where writing an automatic solution is faster than manually completing the task" I'm not sure what word would really describe this though "Efficient" probably.

<comment><USER7> I think it is laziness, but the appropriate kind. I am willing to work harder now, in order to be lazy later. I am so lazy, I do not ever want to manually test a function. Therefore, I write automated tests to enable my laziness. I am so lazy, I do not want to remember caveats on how a function works. Therefore, I do my best to make an interface be hard to use incorrectly.

<comment><USER8> Well this kind of laziness seldom comes at an advantage in the modern workplace? I mean there's always more work so working hard now doesn't equate to working less later. Unless you're an entrepreneur building your own product.

<comment><USER9> Sure it does. If you work with people who don't automate then every time you finish a piece of automation you can slow down your pace a little and still be faster than the people doing stuff manually.

<comment><USER10> The reward of work well done is more work and shorter deadlines, something well known as early as 1987.

At some point you stop trusting that you can spend time on making better tools and just surrender to the tedium.

</comment></comment></comment></comment></comment></comment><comment><USER11> "Programming Perl" is absolutely chock full of great quotes. I think one of the things that attracted many programmers to Perl was how genuinely witty Larry Wall was in his writings/talks. Add to that his background in linguistics and he comes across as a pretty hip/zany/brilliant character.

I think lots of Perl programmers probably thought of themselves as "being in on" his creation and therefore being a bit like him, themselves. That was certainly one of the upshots for me back in the late 90s early 2000s.

<comment><USER12> IMHO famous software people have amazing quotes, right from the start, with:

"On two occasions I have been asked, 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question." \- Babbage

"If I had a little less brains, I should & would be a good Catholic, & cling to that certainty which I do long for. However I don't wish to be without my brains, tho' they doubtless interfere with a blind faith which would be very comfortable." \- Ada Lovelace

"I do not think that this argument is sufficiently substantial to require refutation. Consolation would be more appropriate..." \- Turing

<comment><USER13> >"On two occasions I have been asked, 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question." \- Babbage

To me, this belongs into the same category as Henry Ford's "If I'd asked people what they wanted, they would have asked for a better horse":

Google search does exactly that, trying to offer the right answer, no matter the question.

One day, they will link the user tracking and the search engine, and when you hesitate for a moment, google will tell you the missing information that you are lacking. Thus, the right answer no matter the input. But of course, what they tell you to buy when you are making a purchasing decision will be up to the highest bidder.

As to Henry Ford, self driving cars are faster horses. Seems like people don't buy cars because they want to steer by themselves all the time, they just want to indicate the general direction.

</comment></comment></comment><comment><USER14> The discussions about the meaning of "hubris" in Greek mythology are slightly off. Hubris means to insult _the gods_.

So, while some of the mythological examples in the first long-ish discussion are good ones (e.g. Arachne), it is wrong to say that Leto commits hubris by killing Niobe's children (i.e. an "insult"). The gods cannot commit hubris. Hubris is only for us, mortals. It's to not know our place, that the gods are our masters and we shouldn't piss them off, or they will do something really nasty, and cruel, and unstoppable and extremely unfair- like Apollo and Artemis killing Niobe's children, to punish Niobe (I mean, what did the poor children do?).

Hubris is often glossed as "arrogance", but arrogance itself is not hubris- it's the _cause_ of hubris (though not the only one).

Also, Odysseus did not commit hubris- he pissed of Poseidon by killing his son, Polyphemus the Cyclops. That did not "insult" the god- it pissed him off and he took revenge, but note that he didn't outright smite Odysseus, just made his life misery. Because, after all, Polyphemus had tried to eat Odysseus (and ate some of his comrades) so it was a fair fight. The "fight" with Poseidon of course was not "fair"\- but then, Odysseus had Athena to watch over him.

Prometheus also did not commit hubris- hubris is for the mortals and Prometheus was an immortal, a titan. Prometheus was punished for going against the authority of Zeus, but again that was fair and square because Zeus himself had detrhoned the titans to become the greatest of the gods.

Source: Greek, with Greek schooling and upbringing and a lot of reading of Greek mythology as a kid.

</comment><comment><USER15> Previously on Hacker News:

4 years ago, 72 comments: [https://news.ycombinator.com/item?id=11516215](https://news.ycombinator.com/item?id=11516215)

5 years ago, 59 comments: [https://news.ycombinator.com/item?id=9788088](https://news.ycombinator.com/item?id=9788088)

</comment><comment><USER16> Also the three great virtues of a programmer: Persistence, Patience, and Humbleness.

<comment><USER17> so a total of 6 great virtues for every programmer as two sets of 3 conflicting pairs of virtues

<comment><USER18> One is operational expenditures of programmer time, the other is for capital expenditures of time.

(Save yourself for the yak shave!)

</comment></comment></comment><comment><USER19> I think it's something of a sign that we're all terrible at our jobs that this resonates with us. If you tend to be wrong about things, sometimes it cancels out with your vices. It's not good to do the wrong thing, but if you must, let's hope you're lazy and impatient about it to limit your progress so you can't do much damage.

IDK guys, just found out last night a project I've been working on for months should have gone in a totally different direction, this stuff is hard.

<comment><USER20> If you need some encouragement: i agree that this is hard. But experiences like this are the most valuable. Many advances in our field came from a place like this.

Sometimes it means throwing away stuff and starting from scratch. This isn't giving up, quite the opposite.

</comment></comment><comment><USER21> This is no doubt due to the fact that I am both lazy and impatient, but I find it very hard to figure out who's saying what in this post.

</comment><comment><USER22> Laziness is the avoidance of work, not necessarily the minimization of work. A truly lazy person may not look for effort-saving solutions, but rather avoid providing a solution altogether by compromising requirements, making excuses or having someone else do the work.

Impatience will make you avoid tasks that take "too long", like reading documentation and taking the time to understand how things work. It will make you approach problems by trial and error. By acting in this way, your solutions will likely be suboptimal and your understanding will be superficial and incomplete.

Hubris will make you avoid asking for help or code reviews from people that might have something legitimate to say about your code, eventually wasting everyone's time including your own.

So I disagree. I know the source of these is a rather popular book written by smart people, and that those paragraphs make sense. But there might be better words to represent those paragraphs.

<comment><USER23> I would say that really depends on the person - you can be lazy in a "good faith" or "bad faith". "Bad faith" is just like you described - let someone else deal with that.

It also depends on organization itself, because in some cases the reward for being efficient is more work.

It all applies to every other "virtue" too.

<comment><USER22> Lazy is a word that has a pejorative connotation. Nobody will reply "thank you" when they're told they are lazy.

<comment><USER23> I disagree - i take pride in my laziness.

Although i try to minimize the whole group's effort, not mine.

</comment></comment></comment></comment><comment><USER24> I get that this idea is appealing because it's quirky and counter-intuitive, and self-deprecating as well as self-aggrandizing.

But it's not true, obviously. Truly "lazy" people (i. e. those suffering from depression) will be trying to hack their motivation and beating themselves up over repeatedly failing. It's something that comes up in an Ask HR at least every two months.

Impatience & hubris are what gets "tech bros" the reputation of shallow arrogance. These traits optimize for Uber-like outcomes, in both scale and completely lack of decency. But going for the 1-in-100000 outcome is rarely a good _personal_ strategy, and some marginal improvement doesn't change that.

<comment><USER25> > Ask HE

Intentional? :)

</comment></comment><comment><USER26> I thought for sure hubris would be something about "oh, that'll be easy to automate!" I'm not sure if it's what makes a _great_ programmer but I do think there's some sort of Paradox of Programmers where we can somehow entirely underestimate the "Big Problem" while also being completely undeterred - defiant even - once the "Big Problem" reveals its true size.

</comment><comment><USER27> I think arrogance is very underrated and should be included. A degree of arrogance makes you take on tasks that you are unqualified for; this aids in growth. It also gives you an inflated trust in your own opinions, and hence promotes exposing them to challenge and scrutiny. Furthermore, it encourages risk-taking, which is usually good for society even though it's often bad for the individual.

<comment><USER28> Why would you expose your opinions to challenge if you know they are right?

Also why would you need to take any risks if you think you know everything already?

I don't think you know many arrogant people, I deal with people at my company that are both stubborn and usually wrong, which is a lethal combo. You have to workaround these people or else the project won't get done.

<comment><USER27> Sure, but all of those "virtues" have good and bad manifestations. I agree that arrogance can be terrible.

> Why would you expose your opinions to challenge if you know they are right?

You expose them to challenge _because_ you know they're right; you don't act to protect them by keeping them hidden and not testing them. You "put them under load." This is basically mental and rhetorical dogfooding.

edit: I think the difference is between "I think I'm right" and "I want to believe I'm right." If you want to protect your sense that you're right, you'll be both hesitant to test your beliefs and vicious in their defense. Conversely, if you're attached to the actual state of being right, you'll trial your beliefs and abandon them if they fail you. Arrogance, or at least a moderate degree of it, can help you act like the latter by making ideas seem more reliable than they deserve, creating natural trials.

edit: I guess the common thread is that arrogance promotes exploration by suppressing fear of loss.

</comment></comment></comment><comment><USER29> I will always have a soft spot for Perl. I got my first real big kid job writing Perl and it was a wonderful experience. The Swiss Army Chainsaw. I came close on a few occasions but I never did cut off (my own) limbs.

</comment><comment><USER30> I think, mostly, if you didn't automate yourself out of your job, you didn't embody the programmer.

If they need you to make every little change, if they still need you, you weren't the programmer.

It's a tragic job ultimately, if you do it right.

</comment><comment><USER31> I hope this post stays at the number one spot on the front page for long enough that a certain satirical HN-review page to pick up, and comment on. :)

</comment><comment><USER22> It depends: are you developing, a prototype or the early iteration of a product, or a mature product?

The "Maximizing the work not done" mantra... really depends on what time-frame you pick. Work not done over a day, a week, a year, a decade? Different time windows will lead to different decision-making processes.

<comment><USER32> True, but as the timeframe go up, so do the uncertainty of the decisions.

It also depends on the field too. Almost nobody made good/relevant "maximising work not done" decisions a decade ago about how their mobile apps should be built - who even knew Swift or Kotlin (or React of Flutter) we're on the horizon? And nobody cares, because mobile apps from 10 years back are mostly irrelevant today.

It's very easy/common to make errors in those sorts of decisions, by worrying about things too early. I know many many startup stories who ran out of runway while building Google/Facebook scale infrastructure architectures with no more than family and friends sized user bases. Projects that probably could have been live a year or two earlier using WordPress and a few custom plugins as a backend, and scaled at least to 100 engineers worth of revenue before needing heroic internet-scale design from the two or three person founding team.

<comment><USER22> I have also known, and seen first-hand, companies that neglect their code bases for a long time, and then have to let go entire teams once the quality bar is raised, or once they fail to adopt a proactive attitude towards defects.

Startups are not small companies. Startups are companies created with exponential growth in mind, and if your product is not prepared for that kind of growth, you do not need startup scale funding. You could do fine with small business funding. Unless you can find some chump that will buy your small company at a startup price, usually by inflating the payroll with a lot of redundant employees right before selling.

It turns out building scalable software most of the time is not about making the kind of investments you talked about. It's about designing each feature with scale in mind. Then, unless the startups you talk bought multiple geographically distributed datacenters and deployed their own submarine cables, they were not working at that scale. Don't exagerate.

<comment><USER32> > Then, unless the startups you talk bought multiple geographically > distributed datacenters and deployed their own submarine cables, they were > not working at that scale. Don't exagerate.

No, you're right. Totally exaggerating, but that was seriously their mindset. "We can't use an SQL database, it won't scale to a billion users!" I strongly suspect the team talked about where they'd deploy their data enters and the risks of using other people's cables - only _partly_ in jest.

The one I had in mind specifically while writing that were fighting with sharded MongoDB clusters, eventual consistency, and auto scaling app server fleets, while not having 1% of the user base I support with a more complex app on a handful of t3.medium spot and on demand instances, and a dB.t3.large rds aurora database. I know it's always hard to reliably forecast when your current solution's vertical scaling capability will runout and you need something more scalable than a stateless monolith with a vertically scaling SQL database behind them, but they were easily three maybe four or more orders of magnitude short of that inflection point. I have zero doubt their kinda good idea could have scaled to high seven or low eight digits of revenue before needing any heroic platform architecture. Well before that became a necessary problem to solve they could have had a 50 person engineering team with leadership who'd done it before to solve that problem for them.

And I totally agree with your "neglected codebase" problem too, I've spent the last 18 months or more on a "rewrite the old Grails backend code in Java" journey, which has gone about as smoothly as they always do /\/\/\/\\... We're now finally in a place where the Java platform is as good as the old one in most areas, and betting in some important ones. But of course we now have a bunch of "legacy clients" who are not (and may no ever be) migrated of the Grails platform, and I'm fighting very hard to avoid the worst of the "neglected codebase" problems that's gonna throw me over then next few years while we solve the legacy client problem (and COVID isn't making it any more likely that some of out government clients will upgrade is the need to pay to do so...)

</comment></comment></comment></comment><comment><USER33> I like to think of the first virtue as Proactive Laziness. I'll work hard right now (on something that's fun, automating tasks!) to avoid doing a lot more hard work later that's boring (doing the same tedious task manually over and over).

<comment><USER34> Doing something tedious and manual is a good way to let your brain unwind and kind of meditate. After a bit you get all sorts of insights into how to make the process work better. Or if there's nothing at all worthwhile about the process, your subconscious can still be solving other problems in the background.

One of the problems I find with impressing people with your technical abilities is that they start piling projects on you and you stop doing the tedious stuff that teaches you what needs to be done and how.

The greatest frustration in my career has been that everybody wants to do grandiose automation projects without being willing to assign any proportional effort and talent towards understanding the existing processes.

<comment><USER35> Amen, I've seen so many failed attempt to automate a process that hadn't even been attempted the "slow" "unscalable" way first. Doing a task manually a few (dozen?) times will reveal a disproportionate amount of information about what actually needs to be done.

</comment></comment></comment><comment><USER36> 'Conscientious Laziness' as in, if you don't make a huge mess then you don't have to clean it up, and the room remains clean just the same.

Or, 'I don't make the feature unless it's needed'.

<comment><USER4> The best code is no code. Better to spend an hour googling than a week coding.

</comment></comment><comment><USER37> I get it, it's cheeky to refer to someone who's very keen on automating as "lazy". That being said I'm really not a big fan of using "lazy" as a quality for a good programmer. It's pejorative and it should stay that way. I've never met a "lazy" programmer who I enjoyed working with. Lazy people cut corners because it's less work in the short term.

Being someone who finishes and _completes_ automation is antithetical to lazy. It takes _MUCH_ effort to take things from 80% to 100%.

<comment><USER38> "Keen on automating" is one of the few definitions of lazy that actually make sense.

If we need to get rid of anything, it's the pejorative meaning. Using the word that way is simply a way to get out of having to deal with the complex system that is human motivation.

If someone is consistently lacking in motivation, there are two responses:

1\. Figure out why and be able to do something about it; or

2\. Call them "lazy" and hope the problem fixes itself.

As a general rule, option 2 almost never works out productively. It just ascribes a systematic problem to a single person and nothing improves. If anything, things get worse because people's attitudes change, further reducing motivation.

</comment></comment><comment><USER39> Bill Gates said he only hired lazy programmers because they automate stuff in code and take shortcuts to speed it up.

<comment><USER40> He probably didn't[1], but it's a good idea any way!

[1] [https://quoteinvestigator.com/2014/02/26/lazy- job/](https://quoteinvestigator.com/2014/02/26/lazy-job/)

</comment></comment><comment><USER41> [http://archive.is/FPv9D](http://archive.is/FPv9D)

</comment><comment><USER42> i don't agree for the most part. perl is cool :)

<comment><USER29> See "Perl is worse than Python because people wanted it worse." also by Larry Wall.

</comment></comment><comment><USER43> What is: Boris Johnson?

</comment><comment><USER44> Diligence, Patience, and Humility

[https://www.oreilly.com/library/view/open- sources/1565925823...](https://www.oreilly.com/library/view/open- sources/1565925823/ch11.html)

<comment><USER16> Great find - Larry Wall writes:

> Three great virtues of programming are laziness, impatience, and hubris. > Great Perl programmers embrace those virtues. So do Open Source developers.

> But here I'm going to talk about some other virtues: diligence, patience, > and humility.

> If you think these sound like the opposite, you're right. If you think a > single community can't embrace opposing values, then you should spend more > time with Perl. After all, there's more than one way to do it.

</comment></comment><comment> non-entity I've always heard these as good traits for a programmer, but I must have got a cursed version of them. As to laziness and impatience particularly: I'll sooner decide not to do a task at all rather than somehow automate it.

<comment><USER45> You don't have quite enough hubris to think the automating will be easy.

</comment></comment><comment><USER46> No. Hubris is wearing a tshirt to an interview

<comment><USER47> > _Also the quality that makes you write (and maintain) programs that other > people won 't want to say bad things about._

In this context, I took hubris as paradoxical since the end result is a good quality (and well maintained) software program.

<comment><USER46> Exactly bro! Now keep on rocking in the free world and understand that hubris alone does not a good programmer make but a good programmer is hubristic. Therefore hubris is a poor heuristic no matter what any tech bro tells you.

Therefore you should at least make an effort to dress up for an interview because you never know if it's your last chance to make any sort of impression.

And let your hubris shine that way instead of choosing the casual path that every other one will certainly choose.

Edit: of course this argument is lost on all the beautiful minds who spare no effort at shabby chic

<comment><USER40> This reminds me of [https://www.folklore.org/StoryView.py?story=Gobble_Gobble_Go...](https://www.folklore.org/StoryView.py?story=Gobble_Gobble_Gobble.txt)

Relevant quote: "As soon as the guy walked into the room, I knew it was going to be problematic, because he seemed extremely straight-laced and uptight, dressing more like an insurance salesman than a technologist."

Jobs was an ass, but Andy had a point: dressing correctly depends on the context. The idea of dressing "up" or "down" as if it was a single axis was never correct, and is even less nowadays.</comment></comment></comment></comment>


Joxa – A concurrent, distributed Lisp - <AUTHOR> http://joxa.org/ <comment><USER2> How does this compare with Lisp-Flavored Erlang ([http://lfe.io/](http://lfe.io/))? Any particular advantages or disadvantages?

EDIT: answering my own question: [http://www.ericbmerritt.com/2012/02/21/differences- between-j...](http://www.ericbmerritt.com/2012/02/21/differences-between-joxa- and-lfe.html)

<comment><USER3> Not arguing about Lisp 1 vs Lisp 2, but the example seems broken:

        (defun hello-world ()
           'hello-world)

        (: hello-world)

I freshly compiled LFE to test that and what is written above return an error. Besides, you can already call (hello-world) without problem.

</comment></comment><comment><USER4> +1 for better title.

So I know about Joxa from another HN post related to BEAM based languages. I think that post better represents context for this Lispy-not-Erlang.

I have to say, I really like it, I plan to play more with it, but I think BEAM is solid platform and something I am not jaded with like with JVM, which probably is also excellent platform, just dealing with it /w JRuby soured me for life.

Anyhow, excellent effort, the other Lisp on BEAM LFE is great as well. Off to play with Joxa.

<comment><USER5> Why are you jaded with the JVM? Would that lead you to choose Joxa over Clojure?

<comment><USER4> I think I referred of joy of using JRuby, nominally it is much faster, but problems with it are just not worth it. It is very hard to compare Joxa vs. Clojure, but I would definitely appreciate BEAM way more, especially as I like Elixir and Erlang a lot.

</comment></comment></comment><comment><USER6> I am interested in the "concurrent, distributed" part.

But the web site does not provide any examples, or explanation.

Do you guys plan to add more info on that?

<comment><USER7> Check out the joxa-otp* source[0]. If you aren't familiar with Erlang, they map up to its OTP concepts[1].

[0] [https://github.com/joxa/joxa/tree/master/src](https://github.com/joxa/joxa/tree/master/src)

[1] [http://learnyousomeerlang.com/what-is- otp](http://learnyousomeerlang.com/what-is-otp)

</comment></comment><comment><USER8> Can this please have a more informative title?

Joxa - Yet Another Lisp /s

Joxa - Concurrent Distributed Lisp

<comment><USER9> Joxa - A Lisp on the Erlang VM

<comment><USER10> I would have found that title most informative.

</comment></comment></comment><comment><USER11> Neat little project

</comment><comment><USER12> Lisp is very weird. I was watching a web development video in Lisp ( [https://www.liveedu.tv/foamy125/videos/K0QWA-a-simple- landin...](https://www.liveedu.tv/foamy125/videos/K0QWA-a-simple-landing-page- in-common-lisp-3) ) and now I understand why people prefer PHP.</comment>


Telecom Regulatory Authority of India: Letter to Facebook [pdf] - <AUTHOR> http://trai.gov.in/WriteReadData/PressRealease/Document/PR-12012016.pdf <comment><AUTHOR> Main Points \-----

• First, Facebook tried to mislead it's users into sending emails to TRAI which just said "We support Free Basics".

• After receiving hordes of such emails, TRAI picked on Facebook to communicate with the senders in order to advise them to send emails that contained answers to the consultation. Accordingly, TRAI extended the consultation period by 7 days.

• Facebook responded towards the end of the period that they were unable to reach users. Apparently, they claimed in their letter to TRAI that peoples' support for Free Basics in the emails (somehow) translated to answers to TRAI's questions.

• Facebook claimed that 11 million emails were sent to TRAI through their platform. TRAI says it only received 1.89 million.

• TRAI has declared that it has not received any revised responses.

<comment><USER2> I'd also like to note that TRAI has usually been on the side of user's and not usually caved into lobbying. Over the years they've made sure of:

* Free incoming calls

* state-level number portability so you can switch carriers and hang on to the old number

* national level number portability

* compensation to callers for dropped calls -- an incentive for telecoms to uphold a certain level of service quality

* and now, a stern reply to FB's misleading campaign on free basics.

India as a result has very healthy competition in the telecom space, and the lowest rates in the world.

Kudos, TRAI.

<comment><USER3> > I'd also like to note that TRAI has usually been on the side of user's

What about when it goes against users ? Even now, it has gone against some users. Why do you think its a good thing for <1000 people (TRAI) to affect 1.25 billion ?

The goverment intervention was absolutely unnecessary. They should have left it to market to judge.

<comment><USER4> > The goverment intervention was absolutely unnecessary. They should have left > it to market to judge.

Because that's what I want. Multi-national tech companies going around the world raping economies because "we'll let the market judge".

I'll take regulators please. Slower progress is better sliding backwards.

<comment><USER3> What if I dont want the regulation as a citizen ? What then ? There are so many issues, so many opinions in the world today. I bet diversity of opinions has never been highest. How should we move as a democracy when everyone has different direction in mind ? Free market is the only solution I can think of.

I acknowledge that not everyone can think/worry about everything. Hence why there would be always need of regulatory/look-out-for-me authorities. However, in this fast paced and diverse world, I propose that they should be opt- in/opt-out.

<comment><USER4> > What if I dont want the regulation as a citizen ?

You move.

</comment></comment></comment></comment></comment></comment><comment><USER5> Weird how official letters to Facebook were sent to a hotel. I'm sure they have a permanent office in India.

</comment><comment><USER6> Hard to tell if this is the usual gymnastics from bureaucrats trying to cover their butts or if TRAI is really annoyed that Facebook is using their platform dominance to lobby favorable government policy.

</comment><comment><USER7> Hope TRAI scraps it and go to favor of net neutrality

</comment><comment><USER8> seems like net neutrality is going to win

</comment><comment><USER9> good move to finally unmask and reveal fbs real face. hypocrisy at its peak. supporting diff pricing and the only internet firm to side with telcos on breaking net neutrality

</comment><comment><USER10> Please do not editorialize the titles of stories you submit to HN. It breaks the site guidelines to do so. Users are asked to change the title only when it is misleading or linkbait, and only to something accurate and neutral.

The submitted title was "[India] TRAI is going hammer and tongs after Facebook [pdf]". If someone suggests a better title, we can change it again.</comment>


The Mathematical Opinions of Dr. Doron Zeilberger - <AUTHOR> http://www.math.rutgers.edu/~zeilberg/OPINIONS.html

<comment><USER2> Yes, he is quite the character and a legitimate mathmetician^. I also looked into him and questioned his validity after first reading his opinions.

I read a paper by him on ultrafinitism^^ a few years ago and was intrigued and entertained.

^ <http://en.wikipedia.org/wiki/Doron_Zeilberger>

^^ [http://www.math.rutgers.edu/~zeilberg/mamarim/mamarimPDF/rea...](http://www.math.rutgers.edu/~zeilberg/mamarim/mamarimPDF/real.pdf)

</comment><comment><USER3> As I'm not versed in math, I wonder, are his opinions valid?

<comment><USER4> I am versed in mathematics, and I am finding these opinions, as they are no doubt intended to be, bloody infuriating. There are truths, no doubt, but I think these are doing far more harm than good.

I'd like to see a computer proof of the emergence of the giant component in a random graph process. I'd like to see a computer proof that the TSP is equivalent to 3-coloring a graph. I'd like to see a computer proof of the Banach-Tarski paradox.

No doubt many experiments would be shown and then the conclusions drawn without actual proof. There is a place for experiment to create and guide intuition, and a place for computer assistance in complex manipulations, but to pretend that mathematicians are "clinging to pencil and paper" instead of simply learning to use computers is laughable.

If only he weren't so eloquent and persuasive I'd be less angry.

It is also true that many things are currently done badly by people who could do better if they were trained in skills they currently lack. There is an application of sharpening the saw, but his implications of incompetence are unfounded and distasteful.

<comment><USER5> _I'd like to see a computer proof of the emergence of the giant component in a random graph process._

Maybe this will help: <http://arxiv.org/abs/0808.1549>

<comment><USER4> OK, I've spent literally two minutes as I'm in the middle of something urgent, and I'll come back to it later, but my impression is that while the result is interesting, and the authors may have used computer experiments to gain intuition about what was happening, it's not a computer proof. It's a tradtional proof of something that's relevant to computers.

If I'm wrong please provide a better summary so I can read it with your insights to guide me.</comment></comment></comment></comment>


Independent Filmmakers Distribute on Their Own  - <AUTHOR> http://www.nytimes.com/2009/08/13/business/media/13independent.html?_r=1&hp

<comment><USER2> "Mr. Gervasi, whose studio writing credits include "The Terminal, directed by Steven Spielberg". Top marks for getting his, or any sort of movie made, but he could hardly be described as an indy filmmaker in the true sense of the word - an outsider creating their work without the backup, and especially the contacts, from within the mainstream industry.

This does show how the movie industry is changing beyond the comprehension of most people working within it. I say this after working in features production for nearly 20 years.</comment>


Ask HN: How to convince a fellow programmer to join your startup - <AUTHOR>

The question says it all. Assume that the programmer is good but isn't that brave enough to take the risks and has a good day job. How do founders convince people they know into joining their companies? <comment><USER2> As Charlie Munger says _"If you would persuade, appeal to interest not to reason."_

Address what the worst case scenario could be, in most startups that would be going out of business. Highlight that it would be a better experience in the same period working for a startup rather than said current company.

It could also be financial, maybe he/she has stock options & trigers that would vest in a few months.

<comment><USER3> That's an old Ben Franklin quote.

<comment><USER2> Thanks, didn't know it was Ben.

</comment></comment></comment><comment><USER4> Honestly I don't try to persuade people. I want them to know what they're getting into and be willing to take that risk with us. I feel like if you lure someone away from a steady job with pie-in-the-sky logic, you're setting yourself up to lose a friend. When I got my co-founder in on things, we talked very seriously about the realistic chance of success, how much stress it would be, and well, how much we'd learn along the way and that we might even have a little fun. Expectation management seems to be important.

</comment><comment><USER5> Here's an argument from an older, successful corporate guy, which I found very compelling:

Throughout my life I have taken many opportunities, and not all of them went well. But the only ones I truly regret are the ones which I didn't take.

</comment><comment><USER6> _Assume that the programmer is good but isn't that brave enough to take the risks_

I don't think that "someone who isn't brave enough to take the risks" is really someone you should want joining your startup, personally.

<comment><USER7> Not necessarily, especially if the talent is great. Woz didn't want to leave HP at one point.

<comment><USER8> _"... Woz didn't want to leave HP at one point. ..."_

Woz also probably wanted to build stuff for himself that HP refused to take up. So creating Apple was a logical step. I think the real convincing was via the _"Reality Distortion Field"_ ~ [http://folklore.org/StoryView.py?project=Macintosh&story...](http://folklore.org/StoryView.py?project=Macintosh&story=Reality_Distortion_Field.txt&topic=Reality%20Distortion&sortOrder=Sort%20by%20Date&detail=medium)

<comment><AUTHOR> Yeah. That is what I wanted to know. What do people who don't have the Reality Distortion Field do to persuade people?

The programmers I know are pretty good but they also have PHB (pointy haired bosses) and also hate some things of working in a big corporate.

<comment><USER9> _What do people who don't have the Reality Distortion Field do to persuade people?_

Learn to develop one. It doesn't need to be as good as Steve Job's, but being able to persuade people that you're onto something is a pretty essential skill as a start-up founder. In fact, I wouldn't start up unless you are able to convince a few people that your proposed business is really good and worth backing. Consider it a first hurdle.

If you can't convince people to join up with you, either your pitching skills are not good enough (so improve them) or your idea sucks (so iterate through that).

And, as others have said, if that person is not really interested in the start-up life, don't push them... find someone else (unless that person is a genius).

</comment></comment></comment></comment></comment><comment><USER8> _"... Assume that the programmer is good but isn't that brave enough to take the risks and has a good day job ..."_

Are you a born salesman? Do you have great persuasion skills and killer smile that can motivate others to do what you want? Are you capable of identifying the targets motivation and swaying them over? If you did you wouldn't be asking this question.

Find another.

A programmer is not by default an ideal Startup Founder. Finding someone who is technical and who compliments your skillset is important. Someone who can build stuff. But if someone has to be convinced and not brave enough to leave what they are doing. When things get tough guess who will leave? Are they motivated enough to stick out the marathon? I've often thought to myself, _"what is the best measure of a potential co-founder?"_. Is it a combination of raw ability, skill & determination? Could the "X" factor be a fierce desire to do something for themselves? Build something (product) and profiting from it?

If the fire is not there, a programmer might make a good employee. But Co- founder?

</comment><comment><USER10> You have to turn on the charm, find out what the guys fears or objections are and remove/counter them. Emphasise the positives.

DONT offer equity to someone who isn't brave enough to take the risks. I've told prospective hires (and current hires) that providing equity at this stage would cause them more problems than it solves, but then again I'm not looking to cash in my chips in the next 6 months.

</comment><comment><USER11> the fact of the matter is that there is inherent risk in working a startup. or starting one. it takes a certain type of person to be willing to do this. not everyone has the ability to do it. for example, a guy supporting a family with house and car payments is more likely to want a steady corporate job.

to get people, you need to offer a competitive salary. everyone has to pay the bills. after that, its the perks. if working at a company is fun, who wouldn't want to work there?

</comment><comment><USER12> DON'T!

To be sucessful at a startup, the cofounders really have to believe in it. If you explain your project to someone, and they don't buy it, and you sell hard and convince them, sooner or later they are going to have second thoughts. And that's going to doom you.

</comment><comment><USER13> Sometimes its what you're working on that matters. If its something worthwhile like kiva.com or world vision, I would go for it even if I got to take a pay cut. You got to persuade using the intangibles.

</comment><comment><USER14> I think it is worth the effort to convince him if you are convinced that he will bring the startup forward. Convincing someone is not something to be done in 5 minutes. Ask him the question: what would be the worst case scenario you could face taking the risk? Show him what he might win - I am not talking about money, but experience, fun, responsibility, reputation, stories for his grand-children. Give him some articles of Paul Graham to read, e.g. "Why not to start a startup".

</comment><comment><USER15> If you really want him badly, I would say do whatever you can to convince him. Show the advantages of working on this start up and disadvantages of his day job. Sooner or later, he will have a bad day at work and he will surely think about quitting the day job. That is one when you will become an alternative.

</comment><comment><USER16> Is gaining this person vital to completing the project? If not do it on your own, until you are far enough to be able to invite the person at the later stage.

If yes, you are going to need to pony up the cash, have enough to be able to provide for both of you to survive on ramen noodles for 6 months.

</comment><comment><USER17> Try to get him to go part-time, even just a few hours a week, so that he can see why your start-up is so great.

<comment><AUTHOR> Infact this is one thing which really does work well with most people. Since good programmers are already working part time on projects which interests them, it becomes really easy to allow them to moonlight and join as full time a few months later.

</comment></comment><comment><USER18> Is another programmer your startup's top need or your top need?</comment>


Ember.js 1.0 RC6 released - <AUTHOR> https://github.com/emberjs/ember.js/tree/v1.0.0-rc.6

<comment><USER2> Changelog: [https://github.com/emberjs/ember.js/blob/v1.0.0-rc.6/CHANGEL...](https://github.com/emberjs/ember.js/blob/v1.0.0-rc.6/CHANGELOG)

Lots of cool stuff in this release, but the best of it is Alex Matchneer's excellent "Router facelift" patch, which standardizes and promise-ifies the router (while not changing the public API, unlike the last router update). He wrote a guide to it here: [https://gist.github.com/machty/5723945](https://gist.github.com/machty/5723945)

<comment><USER3> Looks like you forgot to attach the link?

<comment><USER2> Thanks, fixed!

</comment></comment></comment><comment><USER4> I am amazed at how a minor RC release of this framework gets voted to the front page every couple of weeks. Whoever is making it happen has some serious HN-foo, or knows how to avoid tripping the voting ring detector.

<comment><USER5> Its not a scam, Ember.js is the shizzle.

<comment><USER6> It certainly was the shizzle until AngularJS came out.

Now, these constant HN front pages for every RC that comes out every other week certainly look a lot more like astroturfing than genuine evidence of popularity.

<comment><USER7> Angular was released in 2009, Ember in 2011 (SproutCore in 2010)

<comment><USER8> First release was on Oct 20, 2010 [http://en.wikipedia.org/wiki/AngularJS#Releases](http://en.wikipedia.org/wiki/AngularJS#Releases)

</comment></comment></comment></comment></comment><comment><USER9> Does anyone have a good example app which is more involved than a single or couple of models and uses ember data?

Hacking this past weekend it seems harder to find much in the way of solid guidance past very simple apps.

<comment><USER10> I'm working with Balanced right now to build a full dashboard for their payments product using Ember.js. It's open-source and being used by real users. We started with Ember when it went RC for version 1.0 and haven't had any problems so far. Not using Ember Data (didn't play well with our API), but a good app to look at for regular Ember stuff.

[https://dashboard.balancedpayments.com](https://dashboard.balancedpayments.com)

[https://github.com/balanced/balanced- dashboard](https://github.com/balanced/balanced-dashboard)

<comment><USER9> Thanks for that, will check it out.

</comment></comment></comment><comment><USER11> I added this Open Letter [https://news.ycombinator.com/item?id=5932981](https://news.ycombinator.com/item?id=5932981)

I think that guy has a point with: [https://github.com/emberjs/ember.js/commits/master/ember.jso...](https://github.com/emberjs/ember.js/commits/master/ember.jso..). "Ember is released in "revisions", because it's not production ready and not ready to be 1.0. Hence the core team chose to rather call breaking changes a new "revision". Well guess what dudes, that's exactly what versions are for. Why couldn't you just like normal people release 0.4, 0.5? Why do you need to hit 1.0 directly?"

</comment><comment><USER11> version 1.0pre is released 11 months ago. [https://github.com/emberjs/ember.js/commit/b1e7bb99746bd5916...](https://github.com/emberjs/ember.js/commit/b1e7bb99746bd5916528cdf1b0d0334fa8fa0259)

I think that now is about time to have version 2.0pre</comment>


Exit Interview: Founders look back at acquisitions by Google, AOL, Microsoft - <AUTHOR> http://37signals.com/svn/posts/2942-exit-interview-founders-look-back-at-acquisitions-by-google-aol-microsoft-and-more <comment><USER2> It was a tough decision whether or not to sell Intrinsa to Microsoft back in 1999. I was the only one of the founders still at the company at the time, and the board was split. In retrospect, it couldn't have worked out any better. Combining our PREfix team with some really good people from Microsoft Research moved the technology forward a lot faster than we would have otherwise -- and tech transferring the stripped-down version of PREfast into Visual Studio put it in the hands of a lot of users across the world. At the same time, we found a good way to support our existing UNIX customers; and as expected, once we had validated the market segment, other competitors emerged, and Coverity has now taken things way farther than we ever did. From a career perspective, it worked out well for me and the others involved.

One of the key things we did before deciding to sell was look a lot at the political environment we were getting into. An acquisition's success depends a lot on its champions' political fortunes and how the "not invented here" antibodies get dealt with. Also, we were working with some great people who we clicked well with. And the problem we were addressing was very strategic to the company, so we got a lot of attention. So these are all things I'd consider in any future acquisition situations.

</comment><comment><USER3> Anyone else feel that these quotes have been cherry-picked to support 37 signals business philosophy?

<comment><USER4> Completely agree. Other than the quote about Grand Central ("At the end of the day, more users were definitely better off due to the acquisition."), all of them paint the acquisition in negative manner. But I guess it's not surprising. There are far more stories about acquisition gone wrong instead of acquisition gone right.

<comment><USER5> mediabistro

"I consider myself lucky. That was not the case with our new owner. Alan Meckler is very much an entrepreneur and he told me early on he learned the hard way to listen to the instincts of the founding entrepreneur after an acquisition. "

</comment></comment></comment><comment><USER6> This is getting old. Why is 37signals so bitter?

<comment><USER7> Not enough magazine covers.

</comment></comment><comment><USER8> Normally I'd rant about 37s' rhetoric getting old. But with the Groupon IPO on the horizon and the overbearing feeling that there's some kind of startup bubble about to burst, I think I'll cut them a little slack this time.</comment>


Redirecting Stdio is Hard - <AUTHOR> http://blog.httrack.com/blog/2014/12/20/redirecting-stdio-is-hard/ <comment><USER2> One FD_CLOEXEC-related bug that got us in Chrome: on Linux, while we were careful to close any open fds when spawning a subprocess, the GTK IME ("input method editor", modules to facilitate typing Japanese etc.) could itself spawn subprocesses in response to innocuous calls (like user keystrokes). Those subprocesses could then inherit (and keep open) fds we wanted to be closed.

<comment><USER3> Oh hey, any chance you could explain this bit of the /usr/bin/google-chrome script I've always wondered about? (Sadly, the bug is RVG.)

       # Sanitize std{in,out,err} because they'll be shared with untrusted child
       # processes (http://crbug.com/376567).
       exec < /dev/null
       exec > >(exec cat)
       exec 2> >(exec cat >&2)

Somehow child processes can abuse stdin/stderr/stdout in ... creative ways?

<comment><USER4> Funny. /usr/bin/chromium shows nothing like that.

I'm wondering why these launchers differ so much.

<comment><USER2> /usr/bin/google-chrome is written by Google and analyzed by the security team. /usr/bin/chromium is written by enthusiasts (the distro's package author).

</comment></comment></comment></comment><comment><USER5> _The thing is that spawned processed do not give a damn about already opened file descriptors (except for stdio). But the standard is broken for historical reasons, very unfortunately._

It would be nice to have more detail here, at least why this is "historical" and "unfortunate". A whole slew of programs wouldn't be possible and capabilities wouldn't exist without this, both older and modern. Two examples are inetd's wait functionality (letting the spawned program accept(2) further connections on the inherited listening socket) and systemd's, ahem, socket activation.

 _main program sens[sic] commands to remote location and closes the socket_

If you intend the socket connection to be ended, you should be calling shutdown(2) on it, not close(2). close(2) only asserts that the calling process is done with it. The child process would end up with a dead socket fd (although, there are bugs related to improperly handling this, here's one that bit me: [https://github.com/brianmario/mysql2/issues/516](https://github.com/brianmario/mysql2/issues/516) ). This is why the failure cases on syscalls should always be examined and there should be a sane, default failure case. In an ideal world, the leaked file descriptor would merely be just that, and a properly written program would ignore (or close) any file descriptors it wasn't explicitly designed/meant to use.

<comment><USER6> _It would be nice to have more detail here, at least why this is "historical" and "unfortunate"._

I think a lot of people feel that way about Unix features that interact awkwardly with threads. I tend to feel that threads are the unfortunate part, but that seems to be minority opinion.

</comment></comment><comment><USER7> I had a really "fun" redirection bug. My library was appening a log file to stderr, but for various convenience reasons, it opened /dev/stderr (instead of just writing to fd #2).

We got a bug report that if you ran a program that used the library by doing:

        prog >& /tmp/log

then stderr from the program before my library started logging would disappear. If you did this intead, it would work fine:

        prog |& tee /tmp/log

It turned out (obvious in hindsight, not at the time) that we opened /dev/stderr with O_TRUNC, causing /tmp/log to be truncated and earlier logs to be lost.

TLDR: /dev/stderr can be truncated, which was unexpected.

<comment><USER8> /dev/stderr wasn't being truncated. The redirect caused the options (e.g. O_TRUNC) to be inherited by the file it was redirected to.

</comment></comment><comment><USER9> Say I have a command that I want to read both a piped in stdin _and_ interactive input from the user.

Is there a saner way to facilitate this than what I have done? Which is to run it like this from the shell:

        something | mycmd 10<&1

and in mycmd read stdin as normal and interactive input from /dev/fd/10

<comment><USER10> Read from stdin as normal, and read interactive input from /dev/tty.

<comment><USER9> That's the ticket!

</comment></comment></comment><comment><USER11> Great post, and can I also plug the fact that the parent website / app (HTTRACK) is an excellent app and something I use or have used on a daily basis for a few years now.</comment>