Proof-Of-Work is a Decentralized Clock - <AUTHOR> https://grisha.org/blog/2018/01/23/explaining-proof-of-work/ <comment><USER2> > The Bitcoin Difficulty adjusts dynamically so that a proper hash is found on > average once every ten minutes.

So here, the bitcoin network needs to defer to the actual time. The difficulty is adjusted every 2016 blocks. Then, the time it took to create those blocks is determined by looking at actual time stamps of the blocks. That is, time stamps that purport to be the time in UTC when the block was created.

I never understood how these time stamps are trusted and reliable. If I recall correctly, a block with a creation time that is 'too far of' is supposed to be rejected by well behaving nodes. Yet, getting accurate time-stamps is the problem bitcoin is trying to solve.

I suppose that for almost anyone, it is feasible to get time accurate down to 60s and that might be enough that the difficulty calculations aren't affect as much. Especially due to the averaging over 2016 blocks.

edit:

I found this link en.bitcoin.it/wiki/Block_timestamp it states that:

> A timestamp is accepted as valid if it is greater than the median timestamp > of previous 11 blocks, and less than the network-adjusted time + 2 hours. > "Network-adjusted time" is the median of the timestamps returned by all > nodes connected to you.

So the limits are quite loose.

(An interesting footnote from memory)

The difficulty calculation contains a bug where it calculates the average block time ignoring either the first or last block. Thus, the block time is slightly wrong. However, changing this would be a hard-fork because 'the running concensus code is the spec'. That is, if you fix this bug everyone has to fix it at the same time.

<comment><USER3> If you choose a time in the future, other nodes could reject you.

If you choose a time obscenely short (say, 1 second after the last block), presumably the difficulty is an average of many created blocks, and it would take a lot of malicious actors to make this valid.

Plus, if Block A is found at 10:00, and I find Block B at 10:10 but say it was 10:01... and then Block C is found at 10:20, the average of these three is ~10 minutes still.

<comment><USER4> There is no requirement that timestamps be nondecreasing. A block at 10:00 can be followed by one at 9:59.

What is required however it that a timestamp be at least the median of the last 11 blocks.

</comment></comment></comment><comment><USER2> I think some of this is kind of wrong.

Mostly the claim that "The Difficulty is Intergalactic" is just flat wrong. Consider a miner on mars with 10% of the hash-rate on earth. Lets say the light delay from earth to mars is 10 minutes (it is 14 on average).

Now, suppose mars has last seen block B_0 and it was mined on earth (as would happen most often due to 90% of the hash rate being there). We will call E_1 the next block found if only earth were to mine, and M_1 the block for mars. Now, mars has two really big dis-advantages. First of all, earth gets to see B_0 ten minutes before mars does. So, earth can start mining a lot earlier. Second of all, when mars does find a block, but earth finds a block 10 minutes later, earth is going to orphan the martian block.

So, if mars has fewer than 20 minutes of advantage over earth, there is a very good chance any block they find won't matter.

Now, I suppose if the hash-rate is more spread-out over multiple planets, and one planet isn't dominating things might work out. But we might also see consensus break down. I might sit down and do the rigorous math on that tommorow, just to see what happens.

(Other minor nitpicks)

> SHA is Memoryless and Progress-Free

Technically false, because memory-less processes have positive probabilty of yielding no block after 2^256 tries, whereas for sha-256 using brute force, this probability is totally 0. I don't think this matters at the current hash- rate though. (our current hashrate covers about (2.5 * 10^-53)% of the entire output space every 10 minutes.)

> Trying a SHA Makes You a Participant

Only if you would actually submit your hash if you found a match. The same holds for the prime-factorization example. If I happen to factor a huge number and tell no-one I haven't contributed.

<comment><AUTHOR> >> Mostly the claim that "The Difficulty is Intergalactic" is just flat wrong.

(I wrote the article) - I see your point, but I think the article is still correct. Now _communicating_ the winning block is a problem, as you pointed out, so as a miner on Mars you're at a disadvantage, but that statistically the probability of solving the puzzle remains same regardless of your location in the universe is still true.

As someone else pointed out - for the intergalactic version we'd need an interval much longer than 10 minutes.

This is why 10 minutes was chosen by Satoshi probably - it's plenty of time for everyone to communicate the solved block, yet not too terribly long in "human time".

<comment><USER2> Hi, I hope my tone wasn't too grating.

I stand by my point though, for the purpose of totally ordering blocks, the difficulty is not intergalactic because a solution on mars is much less useful than a solution on earth. The problem here is time-delay preventing simultaneity. This is the same problem that block chain time-stamping hopes to solve. Thus it seems unfair to me to ignore this problem.

I wonder how tight the 10 minute mark is at the moment. I imagine there might be nodes that are 5 seconds apart on the network. Maybe even more when we have backhoe outages. How much lower could the 10 minutes be before we start seeing to many orphans even on earth?

<comment><USER5> If there is no double spend between the later arriving block from mars and the block earth already mined the next block could acknowledge both. Basically making the blockchain a blocklatice. If I haven't missed anything the only reason it wasn't done like this from the beginning is that when the orphan rate is low it' not worth the additional code complexity.

<comment><USER2> Interesting! A block lattice seems like a nice thing.

It would be problematic with regards to money creation though. As we get more blocks per 10 minutes, we get more block-rewards (new bitcoin) per minute.

It also seems weird to 'amend' history by saying some other stuff also happened.

Thinking of this, could you maybe use this to somehow block a payment by adding a 'past block' that already spends the bitcoins in your payment?

</comment></comment></comment></comment></comment><comment><USER6> This analysis is obviously wrong, if the important thing was establishing a global clock, you could just use GPS receivers. Ordering transactions works also without proof of work, just including a hash of an existing transaction or block in a new transaction or block and this proofs the order of those transactions or blocks relative to each other. No clock required at all. And also exactly what Bitcoin does but which is totally unrelated to the proof of work.

I guess this has been said countless times before, but the role of proof of work is to establish a somewhat strange kind of identity among anonymous participants. If you had a trusted list of all participants, you could simply grant everyone one vote per block and decide by majority whether to accept or reject a block and get rid of the proof of work altogether. Of course using some cryptography to establish authenticity of votes and such.

But because Bitcoin is anonymous - or pseudonymous if you insist - there is no such list of participants and you need a mechanism to prevent participants from casting an arbitrary number of votes by essentially inventing identities. And this is what proof of work does, it limits your ability to invent identities and cast as many votes as you would like by making casting a vote really hard respectively expensive. The capital costs and the energy consumption of your warehouse full of ASIC miners become a proxy for your identity via their hashing power.

And that's it, nothing more, nothing less. The rest follows from here, for example a 51 % attack is just someone using a lot of money to buy more than half of all the available identities in the Bitcoin system granting him the majority of the available votes.

<comment><USER7> > This analysis is obviously wrong, if the important thing was establishing a > global clock, you could just use GPS receivers

Is it so obviously wrong? From the first paragraph of the introduction to Satoshi's Bitcoin Paper: "In this paper, we propose a solution to the double- spending problem using a peer-to-peer distributed timestamp server to generate computational proof of the chronological order of transactions".

GPS is a trusted system, whereas bitcoin is designed to be completely trust- less. The reason why timestamping is a difficult problem is that if I publish two transactions spending the same money, the network must agree on the order of events (which is to say: must assign different timestamps to the two transactions) so that one is considered valid, and the other invalid. Doing this without proof-of-work leaves you vulnerable to Sybil attacks, where I can pretend to have an overwhelming amount of nodes in the consensus, and vote for the more favourable (to me) transaction to be considered the canonical one.

<comment><USER8> Couldn't we save a lot of energy and have a distributed stores of identities (public keys) with ledgers, something like a git repo per identity?

Whenever I want to spend money, I just commit a changelog to my repo, I want to transfer some credit to identity X, signed with my identity (private key). I sign the ledger with my private key and commit the entry to the distributed store. Push out the changes to anyone who will listen. If you want to forget my identity and transactions, you may, but if I'm credit worthy, my identity will probably be remembered and followed.

Github could remember a repo for each of us, but users of the network would want to mirror the repos, index the repos, and keep an eye out for modified histories.

I imagine there are likely problem(s) with the idea, can someone explain the problem?

<comment><USER6> The problem is, where do your coins come from? Who or what is allowed to bring coins into existence and what prevents them from just creating as many as they like? The second problem is, how do you resolve conflicting transactions spending the same coins?

<comment><USER8> There wouldn't be any coins, instead double entry accounting I (pubkey:ae32...) owe (pubkey:af41...) 10 US dollars. Medium of exchange is agreed upon by the two parties, USD is one option.

If you're my friend or a business, and you see my balance is deeply negative, maybe you don't give some service.

If your my friend or a business, and my credit is not too bad, you follow my repo and give me some service. You commit a change that says, (pubkey:ae32...) paid (pubkey:af41...) 10 US dollars and reference my repo and commit.

I don't think there could be double spending because the receiver of the money acknowledges the credit offer on record (the git repo that paid).

If there are identities and entries in my ledger that are questionable, maybe you don't count them. For example, you might say credit only counts from those identities that are willing to answer a two-factor sms challenge. Or, if my git repo is credited by a google account, that's something worthwhile that could count toward my balance.

<comment><USER6> So the entire thing starts with everyone having a balance of zero. Then A signs a transaction that he gives $1000 to B and we broadcast that transaction, new balances A $-1000, B $+1000. Now B wants to buy something for $500 from C and signs a matching transaction. What does C do? Is A a bank that provided a $1000 loan and therefore C will be able to actually get $500 from A? Or are A and B one and the same poor guy who just created two keys and invented $1000 out of thin air?

Maybe I am missing something but I have no idea how that could ever work unless this is essentially just the current banking system were I know that A is a trustworthy bank and I will be able to redeem your $500 check. Augmented with the anti-feature that we are now all broadcasting our balances and all our transactions to everyone.

<comment><USER8> The important part is A signed over to B $1000. B and C may be the same person. B wanting to do business as C.

If A, B and C are all the same person and there wasn't an identity attached to any account, outsiders wouldn't value the record much, especially if they were transactions of significant value.

C could redeem the value with Western Union (or similar agent). They might say you have to agree to sign the closure of the account to no longer modify the balance of the account. They'd basically be like a collections agency, buying debt.

You'd be able to have similar plausible deniability as Bitcoin. Coinbase sees the transaction flows and the ledger is public. You could do business as a pseudo identity ( Do business as C, if you're B )

<comment><USER9> Thing is, why would Western Union want to buy the debt unless they knew who A is and if they have any hope of collecting from them? This would probably only work if A was a known entity, and generally trusted to have the money they signed over. But if they are, why do we need Western Union at all? Just have C go to A to ask for the money. And if A is an publicly-known entity that exchanges signed amounts for the underlying currency, they're essentially an Exchange.

Combining this, you essentially get GNU Taler: [https://taler.net/](https://taler.net/)

No proof-of-work, just a way for people to get digital money (representing USD, gold, or anything else) issued by A (Exchanges). And rather than regular signatories they use blind ones, so that A doesn't even know that B spent its coins on C.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER10> We are building a ledger based on Verifiable Delay Functions. It's like a high resolution decentralized clock.

[https://github.com/solana-labs/solana](https://github.com/solana-labs/solana)

</comment><comment><USER11> [https://muratbuffalo.blogspot.com/2018/03/anatomical- similar...](https://muratbuffalo.blogspot.com/2018/03/anatomical-similarities- and-differences.html)

Proof of work is the leader election phase of consensus. It is followed by the accept phase, where the leader broadcasts the decision to be accepted. The commit comes eventually and as probabilistic.

<comment><AUTHOR> > Proof of work is the leader election phase of consensus.

This is a common misconception stemming from looking at PoW through the Paxos/RAFT prism. In Paxos a leader is elected, then the leader decides the order of events. But that's not a valid comparison, because in PoW the supposed "leader" does not get to decide anything at all - the block has to be put together prior to "winning the election" (because the block is the _input_ to the SHA).

PoW is a radically different approach - there is no leader, but there exist indisputable points in "blockchain time", and the order of blocks is determined by the longest chain, which is essentially a matter of chance.

Another related misconception is that PoW is good, but proof-of-stake is better. In fact, PoW is the evolution of PoS, not the other way around. A PoS based digital money was described 1998 by Wei Dai [1], but that approach had flaws that would only be overcome by a system described Nakomoto's Bitcoin paper.

1\. [http://www.weidai.com/bmoney.txt](http://www.weidai.com/bmoney.txt) (B-Money, Wei Dai, 1998)

      "Each server is required to deposit a certain amount of
       money in a special account to be used as potential fines
       or rewards for proof of misconduct."

<comment><USER11> Leaders don't decide. They propose values (such as a block of transactions). There may be multiple leaders with different proposals and participants get to a consensus about which proposal gets accepted for a given slot.

You speak authoritatively, saying "this is a common misconception", but blockchain is a distributed consensus protocol and I don't think you are familiar with multileader and leaderless flavors of distributed consensus.

</comment></comment></comment><comment><USER12> The article references "orphans" in blockchain, what becomes of those orphans? My understanding is that nodes compete to solve the hash - then the "winning" node has its transactions from the preceding 10 minutes added to the blockchain. What about all the other (thousands?) of nodes that didn't win - their transactions must not vanish into the ether, but what becomes of them? They must get added to the blockchain at some point otherwise something like bitcoin would collapse on itself, but i've never been able to wrap my head around how that happens.

<comment><USER13> All the nodes have all the transactions (roughly). When you say the winning node puts "its" transactions in the block, that is really the transactions that it is currently aware of, and that fit in the block. All the losing nodes have the same transactions as well.

When orphan blocks are identified and dropped, any unique transactions in those blocks are added to the pool of transactions that have the opportunity to be added to the next block

<comment><USER9> Well, most of the transactions will be the same, but they _can_ have some transactions the other doesn't, depending on what transactions they received and then chose from their own mempool. It's just that those transactions who got inserted into the "losing" block - but not in the "winning" one - will be kept around to be added in a new block in the future.

</comment></comment></comment><comment><USER14> While the analysis contained in this article is rather arbitrary, blustery and imprecise, there is a mechanism by which blockchains can serve as 'partially synchronized clocks.' One interesting facet about Bitcoin is that the model under which you analyze the effect of network delays can give you different results about how unsynchronized local copies of the blockchain can become. In the paper, "Analysis of Blockchains in Asynchronous Networks" [0], a certain (and more realistic) model is used in which adversarial miners are allowed to delay forwarding of blocks that they receive. The goal of this model is to describe how networking can account for changes to the probability of selfish mining [1]. The authors of [0] prove that for Bitcoin, under some assumptions about the maximum delay an adversary can use and the block production rate, is consistent because it has both a lower bound on how much chains between different participants can differ as well as an _upper bound_ about how far ahead a single participant can grow their chain. Note that all of these statements have to be interpreted as statements "with high probability" relative to security parameters.

In [0], the authors argue that Bitcoin, in particular, is a partially- synchronized global clock, and they take advantage of it in [2] to make a hybrid blockchain-DB system that provides more traditional eventual consistency guarantees.

NOTE: I'm not an author of any of these papers, just a casual observer

[0] [https://eprint.iacr.org/2016/454.pdf](https://eprint.iacr.org/2016/454.pdf)

[1] [https://arxiv.org/abs/1311.0243](https://arxiv.org/abs/1311.0243)

[2] [https://eprint.iacr.org/2016/917.pdf](https://eprint.iacr.org/2016/917.pdf)

</comment><comment><USER15> Can someone who knows the technical details of bitcoin answer the following questions? Suppose initially A owns 1BTC and B owns 0BTC. Suppose A publishes a signed transaction that would give 1BTC to B, and simultaneously B publishes a signed transaction that would give 1BTC to C. If a miner includes both transactions in a block, is the block valid? Does it depend on the order of the transactions within the block (A's transfer to B preceding B's transfer to C)?

<comment><USER2> Yes, and this actually has a use.

You can use this to increase the fee of a transaction after the fact. So, suppose A has a fee of 0.001 in the first transaction, and the current fee needs to be 0.01 . Then, B has this transaction, but no-one will put it into a block, so B hasn't really received his money yet. A could sign another transaction with a higher fee that would send the money back to A. This could happen until the original transaction was actually deep enough in the chain.

But using your situation, B can do the same thing as A. B can create a transaction sending his own 0.999 BTC to another address of his, and include a fee here of 0.019 . In this fee, there is 0.01 to fund this transaction, and another 0.009 to fund the fee A forgot to include. Miners will see this, and include both transactions, and B is guaranteed he has his money.

<comment><USER16> > _increase the fee of a transaction after the fact_

Someone should have volunteered to help the Pineapple Fund do this instead of switching to Bitcoin Cash (or maybe this is what "Child-Pays-For-Parent" is, and it cost too much). [https://news.ycombinator.com/item?id=15995391](https://news.ycombinator.com/item?id=15995391)

>> _Since this created a series of unconfirmed transactions, we had to do something drastic: use Child-Pays-For-Parent with a very significant fee._

<comment><USER2> > maybe this is what "Child-Pays-For-Parent" is

Yep, this is what that means. The downside is that the 'child' transaction doesn't just pay for the 'parent' transaction, but also for the 'child' transaction itself. If you make the child transaction a really complex one that draws from many parents, that transaction becomes larger (in terms of bytes) and thus becomes more expensive.

</comment></comment></comment></comment><comment><USER17> > There is a separate consensus in a rare but common case of two consecutive > ticks being associated with conflicting blocks. The conflict is resolved by > what block will be associated with the next tick, rendering one of the > disputed blocks "orphan". How the chain will continue is a matter of chance

It being a "matter of chance" may be false and thus the thesis that this clock is "decentralized" should be questioned. There are situations where there will be conflicting blocks and no block gets orphaned unless a central authority commands it to be so. This was seen in the 2013 Bitcoin fork, when Pieter and Luke ordered the miners to move away from the majority chain (0.8) onto the minority chain (0.7) and this was also seen in 2016 when Vitalik ordered that a permanent chain divergence (ie. fork) happen. In both cases, the blockchain split and a central authority was necessary to resolve the conflict. It was not a matter of chance. The fact that this clock can have permanently divergent chains and that a central authority is required to say which chain should be chosen as the right one means that this clock is not decentralized.

<comment><USER18> Fascinating, I found this article by Vitalik that explains what happened. In v0.8 they swapped from BerkleyDB to LevelDB but due to a bug the two chains started diverting.

[https://bitcoinmagazine.com/articles/bitcoin-network- shaken-...](https://bitcoinmagazine.com/articles/bitcoin-network-shaken-by- blockchain-fork-1363144448/)

<comment><USER17> Yup, we'll never know if the bug was intentional or not, and that's why the fix being centrally coordinated is scary. Maybe it was a legitimate mistake, but maybe one of the developers planned this and executed large double-spends on the 0.8 branch because he knew 0.7 would be ultimately chosen. The fact that nothing prevents this type of attack is worrisome.

You can see how the developers behaved like a central authority in the IRC logs of the bitcoin-dev channel. More info here: [https://freedom-to- tinker.com/2015/07/28/analyzing-the-2013-...](https://freedom-to- tinker.com/2015/07/28/analyzing-the-2013-bitcoin-fork-centralized-decision- making-saved-the-day/)

</comment></comment></comment><comment><USER19> Looks like this article describes a system that's very similar to [http://solana.io](http://solana.io)

</comment><comment><USER20> > _The Bitcoin blockchain Proof-of-Work is simply a distributed, decentralized > clock._

While this is a cute way of looking at the issue it also isn't true. The bitcoin blockchain gets updated on average every 10 minutes with the data produced by proof of work.

The author is saying that the cart pulls the horse, because they are both moving.

</comment><comment><USER21> To be clear, we're talking about Logical Clocks [0], not what humans usually think of as a clock or its purpose.

      [0] https://en.wikipedia.org/wiki/Logical_clock

</comment><comment><USER22> Wouldn't a proof-of-time scheme have a bootstrapping problem? You need some way to unforgeably (and fairly) generate your Bitcoins before anyone will care to trade them with you.

</comment><comment><USER23> On the last section, is a better name for it "Proof-of-Time" or "Proof-of- Tick" (not that we can likely change the popular name, the cat's out of the bag)?

<comment><AUTHOR> I like Proof-of-Tick :)

</comment></comment><comment><USER24> But it isn't a clock because it doesn't tell us how much time has passed. PoW allows for the creation of an ordering. I'm not sure these metaphors are helpful.

<comment><USER25> I believe the reference is to a vector clock [0]

[0] [https://en.wikipedia.org/wiki/Vector_clock](https://en.wikipedia.org/wiki/Vector_clock)

<comment><USER24> That makes a lot more sense, thanks.

</comment></comment></comment><comment><USER26> Decentralized TimeStamp Server... This is clearly wrote in the bitcoin white paper. Why is this becoming a news?

<comment><USER27> It's just a blog post; not necessarily 'news'.

</comment></comment><comment><USER28> Well put, sir!

</comment><comment><USER29> > Nothing Happens Between Blocks

Actually, this is only partially true. With Lightning, things can most certainly "happen" in between blocks. Granted, some things that "happen" may never make it to the main chain, but many things will get rolled up and committed for all to "see".

This is, ironically, how reality works.</comment>


SecondMarket is launching a bitcoin fund - <AUTHOR> https://angel.co/bitcoin-opportunity-fund

<comment><AUTHOR> There's also a bitcoin information page on [https://www.secondmarket.com/education/learn/bitcoin- educati...](https://www.secondmarket.com/education/learn/bitcoin-education- center)

The evidence seems pretty solid.</comment>


Browser Pong - <AUTHOR> http://stewd.io/pong/

<comment><USER2> The first game on HN which I actually competed. Great fun.

Had a rather fantastic come back from 1 - 9 down to win 12 - 10 against the computer. Fairly beatable once you get the technique down.

My heart stopped at 8 - 9 when I dropped a point and thought I had lost. The shot was a horizontal slow shot from left to right and for some reason I moved out the way..

Never found pong so thrilling. Must play it more often.

</comment><comment><USER3> Cool. But it's pretty easy to cheat, just resize your paddle to take up the entire screen and you auto-win.

<comment><USER4> This can lead to a bug, if you play human vs human and resize both paddles, the "ball" speed up and eventually go through one of the paddles

</comment></comment><comment><USER5> I don't understand what it's supposed to do, it just spawned three windows, one plain yellow and the two others with 3 stripes and an increasing counter. Is that a bug or did I miss something? (I tested with up-to-date firefox and chromium, same thing)

<comment><USER6> If the windows are not moving, then it is a bug (although, check that you haven't clicked pause on the game). Otherwise, they should be moving and recreating the game pong. The plain yellow is the ball, the striped are the paddles for the characters (controlled by A&Z for the left one and UP&DOWN for the right one). Tick one of the boxes as non human to play against a computer.

<comment><USER5> Oh, the windows are supposed to be the game! Ok, then it's not a bug (per se) but I'm using a tiling wm, it's preventing the windows from moving. Thanks for the explanation.

<comment><USER7> I had the same problem

</comment></comment></comment></comment><comment><USER8> Pretty good. The source code [1] is also clean and commented. The randomized increasing velocity was a nice touch.

[1] [http://stewd.io/pong/lib/pong.js](http://stewd.io/pong/lib/pong.js)

<comment><USER9> Can anyone explain this a quirk line of the source code to me? (In the framebuster portion at the top of the file):

window.location.href = 'ht' \+ 'tp' \+ ':' \+ '//stewd'\+ '.io/' \+ 'pong';

Why has he written 'ht' \+ 'tp' \+ ':' etc instead of just the URL as one string? My suspicion is something to do with preventing framebuster-busters from working, but I'm not sure. I've never seen that before.

<comment><USER10> There are sites out there that attempt to offer a service that protects your privacy by fetching websites for you through their servers and then pass you the content afterward—so your IP isn't part of the request chain. That sounds like a nice service. Unfortunately I found there were 3 problems with a particular company offering this service:

1\. From what I've read these guys just pass your info on to interested agencies anyway, entirely defeating the purpose. 2\. In order to offer the service for free they insert adverts into the content—I assume to cover their operational costs. 3\. This particular agency was able to get a pretty high search engine result—rivaling my own URLs—by just copying my stuff, Browser Pong being a prime example.

At the time this meant if someone liked what I did and searched for it there was a good chance they'd end up visiting a boxed in version of Pong with adverts on it. Aside from some tongue-in-cheek exceptions I've been firm about not putting ads on my sites. (And for the most part this doesn't matter because they don't get a ton of hits, but when something blows up like Pong or Jed there's you know ... like maybe a whole $29.95 to be made there.) I made these things for the enjoyment of it and didn't want to dilute them with "a word from our [randomly selected] sponsor." So I started experimenting and figured out how this company was parsing the HTML, CSS, and JavaScript to patch the links so everything would go through their servers. The code snippet you pointed out was just a quick way to hide the URL from their parser and get them out of my hair.

I don't have anything against ad-based models or referencing someone else's work (obviously). But in this particular case I felt it crossed a line so I pulled the trigger. I hope that answers your question ;)

</comment></comment></comment><comment><USER11> Just FYI.. It didn't work for me on Google Chrome (stable) or Chromium on Ubuntu 13.04.. It did work in Firefox. Pretty crazy use of windows. Also, I tried my best to beat the AI, but couldn't! That's a hard game

<comment><USER12> > It didn't work for me on Google Chrome (stable) or Chromium on Ubuntu 13.04.

Did you disable popup blocker?

<comment><USER11> Oh yeah! That was probably what was stopping it from working. It did open up the opponents window though. I think the score raised to 2 after some moving around and then stopped (I guess it was my turn then).

</comment></comment></comment><comment><USER10> I feel compelled to point out that all of Browser Pong's sounds were composed specifically for it by a Mr. Dominic Matar who is not only a multi-talented musician, earnest with feedback, and strikingly handsome—he's also a breeze to collaborate with. So if you're looking for that sort of person for a project do check out his stuff and drop him a line: [http://dominicmatar.com](http://dominicmatar.com)

</comment><comment><USER13> I honestly thought that moving a browser window with javascript had been impossible since early IE.

<comment><USER14> Generally speaking, you're allowed to manipulate windows as long as _you_ spawned them

</comment></comment><comment><USER15> Stewart Smith has a presentation in which he discusses Browser Pong along with some other projects. It's a very enjoyable presentation to watch!

[http://vimeo.com/22433439](http://vimeo.com/22433439)

</comment><comment><USER16> I have two monitors (Google Chrome on Ubuntu). The paddles appeared on one while the ball appeared on the other. It was still playable, but was rather odd lining up the paddle against the ball on a different monitor.

</comment><comment><USER17> Wasn't this one of the original "Chrome Experiments"? I remember there being quite a lot of them, most of which were fun to play with.

</comment><comment><USER18> Using 'A' and 'Z' for up and down is always great fun on a German keyboard where 'Y' and 'Z' are switched.

</comment><comment><USER19> A much older implementation:

[http://www.project-euh.com/pong/](http://www.project-euh.com/pong/)

</comment><comment><USER20> This is the most awesome use of browser windows ever.

</comment><comment><USER21> Just delete the opponent window, easy shutout</comment>


OneSentence.Me. Get your friends to describe you in One Sentence. - <AUTHOR> http://OneSentence.Me

<comment><USER2> This seems like a rip-off of threewords.me here let me help you for your next idea fiveideas.me maybe some of your friends will give you better ideas through this site.

<comment><AUTHOR> Try it out and compare. Implementation and improvement of similar and new ideas is what we humans have been doing since the beginning. That's why today we have many similar and better wheels.</comment></comment>


Ask HN: When should one leave a company based on learning stagnation? - <AUTHOR>

I had quit a corporate business analyst job and joined a tech startup to pursue programming full time instead of hobby. Joined as front-end engineer (as first few employees), within a few weeks due to other backend guys not joining picked up backend and wrote the whole web app from scratch. Now have experience with full-stack, deciding architecture of the app and other devops stuff. It's been great learning experience for almost 2 years now.
I see 2 red flags now:
1. I am not seeing people smarter than me joining (startup got $1M+ funding) i.e. I end up teaching stuff instead of learning stuff from others (I am pretty sure I don't follow all the best practices, so have a lot to learn). 2. recently learned the equity vesting will start few months later than what was initially agreed upon... so even after 2 years, i will have much lesser vested stock :( Also strategies around growth by other team members haven't been much successful as yet either... so can't evaluate how valuable the stock will be eventually...
Would appreciate if you could share opinions if this is normal in most startups or if I should actually start looking out specially from the perspective of learning opportunities in the early career years...
PS: I am single and in 2nd half of 20s. This is just my 2nd company out of college. <comment><USER2> 1\. One way of learning is to apprentice to a master craftsman. Another way is to figure out things on your own. Yet another way is to take stuff you think you already know and try to explain it to people that knows less than you. None is better than the other, and each has it's unique insights. Maybe you currently don't have some greyback to follow and learn from, but when you find your teacher, you will be able to pick up more of what he says by virtue of having experienced first hand what he's talking about.

2\. Kid, life's a marathon, not a sprint. It is pretty much irrelevant if your options will vest now or in a couple of months. What you really need to be in the look out is for dishonest leadership. If this is a one off event, all is OK. But if it is a trend of the founders failing to respect previous commitments to employees... you don't want to waste your time around that kind of people.

3\. Assuming healthy leadership, it sounds like you have earned a key role in that company. You probably want to part in good terms and maybe work with those people again, so please speak with your manager about your concerns. Maybe something can be arranged, maybe not. Maybe you can negotiate a better deal if you feel your responsibilities have grown faster than your compensation. But whatever is the end result, your creed as a professional will be much better if you speak beforehand than if the first time your boss hears about your dissatisfaction is the same day you give notice.

<comment><AUTHOR> Thanks for addressing my concerns point wise! I have been a little worried about the stock option move ever since I have come to know about it. It has been the only sort of red flag as of yet about the leadership, but I will keep your words in mind and see if there is a trend. You have given very actionable advice. I guess talking to them about this should be my first step in any scenario I proceed with.

<comment><USER2> Glad that you find my opinion helpful.

One more piece of advice that came to mind. Maybe you can negotiate that part of your current responsibilities be offloaded to a more junior engineer with the right balance of knowledge and attitude. This would serve the dual purpose of releasing some of your time for more "research oriented" activities while simultaneously providing you with the intellectual challenge of having someone to mentor for the next 3-4 months. Hopefully after that time you will have a better idea of what you want to do next in your career.

This all of course depends on you having a boss that actually listens and is interested in having the company's personnel grown into the professionals they are meant to be, vs extracting the maximum amount of labor each quarter.

</comment></comment></comment><comment><USER3> I'm in the same situation, and to be honest, I'm not really sure.

First thing first, for all practical purposes, startup stock is useless -- and especially if you're already thinking of quitting.

In my situation, I joined the startup right after college, and it has been almost 2 years for me as well. I was the first technical employee, and except one other employee (the second hire), I'm not seeing other people better than me joining either. That said, there are a few things to keep in mind:

\- Startup is NOT the place to be learning technical skill, especially early stage startup. The big 4 will serve you much better than that (and I know this going in). However, one of the things I've learned over the last year is that there are more to it than just technical skill. I have no doubt that I haven't been learning anywhere as fast as I could have in term of tech skill, but I've learned other stuffs that might just be as valuable: from dealing with (less than ideal) team member, dealing with expectation (mine and others), being resilient, and a gazillion things that probably make me a better human. Startup is emotional, for better or worse.

\- I remember reading a quote that if you can only learn from people better than you, it's gonna be awfully hard to learn anything. As you eventually get better, you will have to learn from everyone around you.

\- Since both of the previous points are extremely personal and subjective, I think that the one most important point to consider would be this: do you think the startup have any fighting chance? If the startup grows, both of those previous problems will fix itself. If the startup have no chance, well why would you stay there anyway? In my case, I can't answer that question with either a straight yes or no, so I can't decide.

<comment><USER4> I am also struggling with whether it is time for me to move or not. But I think your point about only learning from people better than you... that is a interesting perspective.

Like the OP, I think we consider ourselves to be at the "hyper-growth-learning phase", where we can really accelerate our potential by working with really talented people. But yeah, eventually you're going to be that "Senior Developer," and it'll be up to you to continue improving and helping others improve.

I've only been in the industry for 2 years as well.

Right now, I feel like I'm the only person on my team that is continually self-improving and trying out new things. It's always me bringing back new things and sharing with the team. Sometimes, I feel like I could benefit way more - like 2-3x - if I was on a team where everyone was self-improving and we could just all feed off that energy.

I have an extraordinary amount of autonomy. My year end review only consisted of, how I knocked everything out of the park and something along the lines of helping get into more of a leadership role.

Sometimes, I think - am I doing so well because everyone around me is just average?

Do I wait around to be promoted into leadership? I mean I've done fine self- learning and keeping myself up to date. But honestly, I think would hate to end up being in some sort of leadership position just because I've been with the company for a long time and have delivered consistent results, and then end up with mediocre tech skills.

Or do I say its time to part ways and focus on becoming a really good engineer and join a team that can continue to accelerate my skills?

For me, I am leaning toward the latter. When I feel I hit that "Senior Developer" phase, that's where I'll re-evaluate where I want to go next.

<comment><USER3> The autonomy part you mentioned is something that I forgot about, but that's definitely a very important point as well: from what I've known, there is a huge limit on what you're allowed to do in a bigger environment. There are a risk of learning things that you might not like -- of course, I've never worked in one, so takes it with a cup of salt.

Personally for me, the other thing is that I like to learn a lot of things, and I just figure that whether now or a few years from now (when I'm "Senior Developer"), there would be just as much things to learn about. I figure it's just better to learn how to learn and get people to teach you stuffs without being in the same work place.

Another thing is that leadership skill is different than tech skill, which is also different than "technical leadership" skill, and this is back to the whole autonomy thing: I got a chance to try my ability to cram the idea that "not everything in a language need to be used" to the js guy, or telling the Java guy "may be we will just decide to rewrite our ORM with another library and putting up 5 layer of abstractions could potentially help us do that" is not a sound idea. Engineering skill set seems to be distinct from coding skill, which I'm not sure will qualify in the usual sense of "technical skill" or not, since you can't put "good variable and function naming" on your CV. Believe it or not, working with really bad code forced me to come back to first principle and ask question like "why should we use function", which I don't think you will have the chance to do in a bigger code base?

There are also benefits from working closer to the business: you learn what actually is important, and what is crap.

Of course, all of this would apply if you join another startup ... and I don't mean for any of this to be an advice or anything, more of a commiserating comment.

I can't really put my email here. But since you and some others are on their main account, do you mind if I have a way to contact you? I just want to follow up a few months from now to see your decisions.

<comment><USER4> For sure, I've added my email in my profile.

</comment></comment></comment></comment><comment><USER5> I have been in a similar situation. I was a co-founder at a company, I was not joining with technical skill. I had more marketing and business development background. I had some knowledge but was just starting out.

The opportunity was fantastic and I learned a lot (things todo and not todo). However, as you stated I felt the exact same things you did:

A. Learning Was beginning to stagnate - I was no longer able to continually learn and develop.

B. Company was not growing and my confidence in the rest of the team plummeted. We were focused on the wrong things and promising things to customers and partners that we knew we could not deliver on.

These two things alone where enough for me to know that my time was done. There is a post on Medium, "Rate-of-learning: the most valuable startup compensation" (link at the end) this is why I had to move on. I knew that for me to continue progressing and learning the best option for me was to leave.

In regards to equity and raising capital. I learned two things (1) people can become extremely greedy when equity and raising capital come about and (2) greed can easily lead to ethics and values being compromised. I don't know your exact situation but I would keep these things in mind.

Hopefully this helps. Best of luck!

Link: [https://medium.com/@KyleTibbitts/rate-of-learning-the- most-v...](https://medium.com/@KyleTibbitts/rate-of-learning-the-most- valuable-startup-compensation-56dddc17fa42)

PS: This is another post that may help you by Ryan Hoover. Great post! [https://medium.com/@rrhoover/knowing-when-its-time-to- move-o...](https://medium.com/@rrhoover/knowing-when-its-time-to-move- on-6c5b8f6e6908)

<comment><AUTHOR> Thanks for sharing your experience. I hadn't read the post by Ryan. Thanks. This does look really encouraging and helpful at this point.

</comment></comment><comment><USER6> I was on the same boat. After working for 2 years delivering over 5 projects, I decided it was time to move on. Not to sound too arrogant but I was fed up teaching and guiding others. I was doing over 80% of work by myself and it was a hectic period. Since then I have moved to a different company. So far the experience has been really positive.

<comment><AUTHOR> I was just wondering how did you deal with other senior team members that still stayed back. Hope there was no bad blood. I am curious as it seems you leaving would have been a big blow to them and they could have reacted a little adversely maybe out of self-interest...

<comment><USER6> The whole team relied on me so it was a hard decision. I gave them a month notice and slowly transitioned myself out of the meetings, code reviews and architectural decisions. I felt bad but it was the right decision for me and my career.</comment></comment></comment>


Show HN: Choose – An alternative to cut and sometimes awk - <AUTHOR> https://github.com/theryangeary/choose <comment><USER2> I think it is very cool that Rust has led to a renaissance of rewriting classic Unix tools to make them fit more with current use. Unix was never meant to stand still. It just happened that AT&T broke up and it took a while for Linux to catch up, and by then people got used to the idea of a fixed set of POSIX utilities. But their CLIs are often quite bad, and security was never a consideration in the olden days, so it's good see them re-evaluated.

<comment><USER3> > I think it is very cool that Rust has led to a renaissance of rewriting > classic Unix tools to make them fit more with current use. Unix was never > meant to stand still.

Technically it has not, even for the core tools they've been getting extended, usually incompatibly, in both GNU and BSD lineages. Though it's pretty funny how much the rust community has been taken up by providing alternatives and replacements for "classic" (POSIX) utilities.

<comment><USER4> While development has not stopped, there haven't really been many advances in improving the syntax. Many of these tools are stuck in awkard, unintuitive syntax, which is cumbersome unless you use them frequently. I feel like the renaissance has lately been one of usability, which I personally really appreciate. Obviously, hard-core daily users would disagree, but considering I use `awk` at most once every 6 months, I hate that I need to spend 20 minutes re-learning how to use it every single time, particularly for basic purposes.

</comment></comment></comment><comment><USER5> Hmm, so compared to cut this

1\. saves -f because it doesn't support cut's -b and -c modes (edit: actually -c is supported, I just didn't see it);

2\. Uses -f instead of -d, making it rather confusing for cut users;

3\. Uses : instead of - for range specifications;

4\. Offers an exclusive indexing mode;

5\. Misses a bunch of other cut features (assuming coreutils cut).

Not sure I see much appeal...

Edit: Another thing I missed: regex separator instead of just character list.

<comment><USER3> > Not sure I see much appeal…

The appeal is the same as replacing grep with a fancier searcher:

1\. it has good and sensible defaults (field mode, also I'd have to check but hopefully and _unlike cut_ it doesn't print the entire line when it's unhappy with the selection you asked for, that's worse error handling than ed) (edit: confirmed, if you give `choose` nonsensical selection _it doesn 't print anything_ e.g. if you ask `cut` for columns 10-15 of data with 3 columns it's going to print the source as-is, choose is properly going to print a bunch of empty lines, that alone makes it better than cut)

2\. It works better on actual data, which is generally whitespace-separated rather than tab-separated, meaning cut requires preprocessing before it'll do anything of use

Can you massage cut or the data to fit? Yes, in the same way you can massage grep or your data to fit. That you don't have to and the utility behaves sensibly by default is appealing. This exact thing is one I've been thinking about for some time now, I'm glad somebody else agreed and did the legwork.

</comment></comment><comment><USER6> I can't understand why you are mentioning awk. Cut or choose cannot be compared to awk, awk is a programming language.

Also I don't think that it's so much easier to use than cut. On the other hand every *nix system has cut so if you make scripts with it they are portable.

<comment><USER7> > I can't understand why you are mentioning awk. Cut or choose cannot be > compared to awk, awk is a programming language.

Because 99% of awk IRL use is just as a fancier cut.

It's very rare someone even sets a variable using awk. If you do it, you are a statistical rarity.

> Also I don't think that it's so much easier to use than cut. On the other > hand every *nix system has cut so if you make scripts with it they are > portable.

I, for one, never remember the syntax for cut. If "choose" gets a deb, I'll use it: Python slicing is something familiar to me.

I don't care if cut is on every unix system: if I have the possibility to install things on the machine, then I'll just install what I need. I have a script for that. If I don't, I'll google/man/\--help GNU commands as usual.

And as for writing shell scripts, I use Python anyway.

<comment><USER3> > Because 99% of awk IRL use is just a as fancier cut.

You say "fancier", I say "working": since cut can't work on general whitespace without a pre-processing phase (e.g. tr), it simply doesn't work for the vast majority of the things I try to shove into it, and I pretty much always end up using awk instead.

Choose means my awk use will fall down by 99% or so.

<comment><USER7> Agreed.

In fact, anybody promoting cut, please give me the cut version of:

        echo -e "foo   bar   baz" | choose -1 -2

It should work on an arbitrary number of spaces, and fields.

The oneliner is going to be... interesting.

Now you can do it with awk using:

       echo -e "foo bar baz" | awk '{ print $NF " " $(NF-1)}'

But it's neither easy to type, nor to remember.

Choose is what cut should have been.

<comment><USER8> By using only basic functionality that's easy enough to remember I guess I'd go with something like

`echo -e "foo bar baz" | tr -s ' ' | rev | cut -d ' ' -f 1-2 | rev | awk '{print $2 " " $1}`

Everything except the awk part is something that I use all the time and is easy to type & remember.

To be honest I'd use `choose` if it was available everywhere, but for string manipulation I can't justify using nonstandard tools since they aren't always available.

Every now and then there are some new ones I actually start to use. For example `ripgrep` mostly replaced `grep -R` for me some time ago, a lot of it has to do with the fact that if `rg` is not found I can fallback to normal grep and get the same result, just a bit slower.

I guess my point is that while I do appreciate innovation & making better tooling, the hard part always is getting the tool where it's most needed.

</comment></comment></comment></comment></comment><comment><USER9> Not sure why it is even compared to awk instead of just cut. It could've been introduced as cut-like command with regex input field separator. Or at least not say things like:

>However, the awk command is not ideal for rapid shell use

And

>cut is far from ideal for rapid shell use, because of its confusing syntax

anything new is confusing until you learn enough to be comfortable

>ranges are just plain difficult to get right on the first try

and how does choose become easy to use with ':' character instead of '-'

Is this a typo or does inclusive/exclusive depend on whether first number is specified?

>choose 2:5 # print everything from the 2nd to 5th item on the line, _inclusive_ of the 5th

>choose :3 # print the beginning of the line to the 3rd item _exclusive_

<comment><USER7> I heard those arguments when ag went out as an alternative to grep, and ffind as an alternative to find.

But now, I install their successors, ripgrep and fdfind, on all my machines. Including the windows ones.

<comment><USER5> Tools like rg offer additional features and/or save you from tediously specifying many options. This one gives you -f for free, that's about it. Detailed comparison: [https://news.ycombinator.com/edit?id=23445931](https://news.ycombinator.com/edit?id=23445931)

<comment><USER7> \- it supports regex separators. That's a great feature to me.

\- the default separator is "\s", like python's split(). Just for that I will adopt it: not having to care about tabs/spaces/mixes is a much better experience.

\- it has negative indexes, again like python. Getting the last field, or the last nth field, is something common enought. I don't want to rewrite the thing with a twisted double "rev" with proper index. And I don't want to have to google it.

\- plus the syntax is just must easier to remember to me. When I use cut, I always try: "echo 'foo bar baz' | cut 2", just to realize that I need to pass '-f', then I do "cut -f 2", and get stump, and google it, to then remember I need to pass the delimiter explicitly even if it's a space.

\- it works the same on windows. I dual boot.

Compare:

        echo -e "foo bar baz" | choose -1

To:

        echo -e "foo bar baz" | rev | cut -d ' ' -f 1 | rev

cut is, to me, the opposite of a friendly API.

Something so basic in the Unix world should have sane default.

Default are not sane if I have to google it once out of two.

<comment><USER3> One more great feature of choose which cut doesn't have: not returning garbage output on garbage input.

If you give cut columns which don't exist, it's going to output the _entire_ source as-is.

</comment></comment></comment></comment></comment><comment><USER10> Can it output fields in an order other than their input order? That's the one thing I regularly wish cut could do. I would like the output of the second cut below to be "3,1", not "1,3".

      $ echo "1,2,3,4,5" | cut -d , -f 1,3
      1,3
      $ echo "1,2,3,4,5" | cut -d , -f 3,1
      1,3

<comment><USER11> Yes:

      $ echo "1,2,3,4,5" | choose -f , 2 0
      3 1
      $ echo "1,2,3,4,5" | choose -f , 2:0
      3 2 1

Note that the indexing starts with 0, "-d" is "-f", and a range is denoted by ":" instead of "-" which is used for indexing from the end.

</comment></comment><comment><USER12> I love my coreutils replacements...not that they're in Rust but because they're generally faster and easier to use. fd, rg, bat and now i shall use choose! I _almost_ always have to lookup awk's syntax but the defaults in choose seem trivial to remember. Thanks for making this!

</comment><comment><USER13> I am not sure why being zero-indexed is considered a feature. I have no problem using a zero-indexed system, but I've never really thought of it as a feature. Is there something I'm missing that makes zero-indexed systems faster, easier to use or otherwise better than one-indexed system?

<comment><USER14> There's a better reason than this that I'm forgetting, but never underestimate the power of being the same as what people are already familiar with. Every time I have to write lua or read some Matlab, the mental overhead of having to remember everything is one-indexed is just incredibly annoying.

<comment><USER15> Anyone used to command-line tools is used to fields being 1-indexed.

awk uses $0 as the whole line, and $1 as the first field. cut uses -f1 as the first field $1 is the first argument to a posix shell script /1 is the first matched reference in a sed $1 is the first regex match in perl

A command-line tool being 0-indexed breaks from expectation of what everybody is used to using on the command line.

<comment><USER16> Anyone is a generalization based on a narrow viewpoint. I would say I am pretty used to command-line tools at this point, at least enough to be using linux as a daily driver comfortably. And I frankly I didn't know 1-indexed fields were the norm, even though I knew $1 is the first argument to a poxis shell script (I always assumed $0 referred to the script or command itself).

<comment><USER15> That's fair, I overgeneralized.

</comment></comment></comment></comment></comment><comment><USER11> Inspired by a remark about Python's default split behavior in comment [https://news.ycombinator.com/item?id=23446146](https://news.ycombinator.com/item?id=23446146) I wrote a Python oneliner for field selection works similar to "choose" but throws exceptions when the field cannot be found:

      $ echo " a    b c" | choose 1 2
      b c
      $ echo " a    b c" | python3 -c 'import sys; [print(f[1], f[2]) for line in sys.stdin if (f := line.split()) or True]'
      b c

</comment><comment><USER7> Someone should make a bundle installer with this, bat, fdfind and ripgrep. I do enjoy those alternative to GNU, and install as many as I can: they are easier to use, usually faster, and just make more sense to my brain.

This pain is real: [https://xkcd.com/1168/](https://xkcd.com/1168/)

<comment><USER9> There is [https://github.com/uutils/coreutils](https://github.com/uutils/coreutils) implemented in Rust

<comment><USER7> Yes but their goal seems to be API compatible, which I understand the point of, but is not useful to me.

</comment></comment></comment><comment><USER17> What does this do that cut can't?

<comment><USER18> This is a poor question as it invokes the Turing tarpit. (Why using any language that is higher level than machine code?)

If it is more comfortable to use for some people then it's a great invention.

</comment></comment><comment><USER19> Very cool. An inverse mode to suppress matched fields might be a neat feature.

</comment><comment><USER20> Very nice. Good work.</comment>


Professional Software Development - iris-digital http://mixmastamyk.bitbucket.org/pro_soft_dev/ <comment><AUTHOR> I like this book as a means to present the ways people describe how software is developed. However, as I read I'm itching to annotate it with notes on "what really happens". I've worked in this business for a few decades now, in various locations and various sizes of companies, including a few that are household names. I have NEVER seen software built on the ground in exactly the ways one sees documented (documented anywhere, not just in this book).

A couple of things I don't see mentioned (apologies if they're in there, I haven't read every word):

1\. Process does (and should) vary tremendously depending on factors including the organization size, organization maturity, market maturity, experience level of the people involved, budget, etc. The book seems to suggest that there's a one-process-to-rule-them-all.

2\. Often there are significant unknowns about a project : unknown technologies, unknown market needs, unknown requirements. Being able to accommodate the unknowns, which can mean not expending effort trying to know something unknowable, is important. The book I think gives an impression of quite confident smooth progress toward project completion that I personally have never observed.

Also: The value of Rubber Chickens is not mentioned...

<comment><USER2> Yes, it does mention these things. Perhaps not early enough? The "really happens"/disasters are towards the end of Part I, and there are few project cartoons giving hints.

There's so much to cover, I started out with the formal stuff first. It seemed like a good order, but perhaps it is a bit off-putting at the beginning of the book.

<comment><USER3> Why are "really happens" and "disasters" associated together? I assume by "really happens" you mean the ways reality departs from what's in the book -- but when process varies over time, adapts to different personnel, business circumstances, tech capabilities, etc., instead of remaining rigid, like ironically rigid adherence to Agile despite variation in the above-mentioned characteristics, _that_ is what leads to disaster.

I'd see strict adherence to any sort of one-size-fits-all recipe for workflow management as a greater sign of disaster than healthy compromise, accommodation for individual working styles, and respect for the human need for autonomy and self-direction.

<comment><AUTHOR> Right. I'm not talking about bad outcomes in the grandparent. Far from it. I'm more interested in the fact that project participants will for example fabricate a fake manifestation of a process (fake[1] project plans, fake task lists, fake design documents) for the benefit of external folks asking them to "follow process". Meanwhile they get on with building the product "The Way we Actually Do it[2]", successfully. This phenomenon can be confusing because outsiders might conclude that the process led to success when in fact it was the presence of people who knew how to build software successfully that led to success and the perception of process was a mirage.

[1] By fake I don't mean fraudulent but rather something that has as its only purpose to meet the needs of the process enforcement. For example a requirements document that nobody reads after it has been written, or a design document written before the manner in which the problem will be solved is well understood (and is therefore never read). Project plans that somehow turn out to have underestimated effort by 3X...

[2] TWWADI(tm)

<comment><USER3> > This phenomenon can be confusing because outsiders might conclude that the > process led to success when in fact it was the presence of people who knew > how to build software successfully that led to success and the perception of > process was a mirage.

See, I think this is actually intentional. The "process" itself exists to give management a surface area of metrics they can datamine for their own political needs, more or less independent from the actual success of the project or team.

I liken it to creating Dutch books. With each layer of management you go up, the pressure to create Dutch books that provide you with blame insurance and protect your bonus -- crucially, diversified across _all_ different business outcomes -- gets more intense, and the people who pull this off get compensated hugely and can often make leaps into C-level executive teams.

Lower level managers might not consciously know about this, and on one hand may actually think there's some truly redeeming reason to care about the frivolous, irrelevant metrics (e.g. Agile burndown). But the higher up you go, the more directly and unabashedly people openly function _honestly_ \-- they know the metrics are bullshit; they know that you know that they know the metrics are bullshit ... they don't care ... they are just getting on with their own business of political games to diversify away bonus risk and arrange for metrics-based blame stories for scapegoats.

It's not that it's confusing where the credit lies. It's that people _know_ the credit does not lie with the formalized process _at all_ but that they need that excuse in order to politically manage how they get their share of the credit.

Unfortunately, I think despite the OP's strong efforts to present the topic in a useful, taxonomic way, the sad thing is that _treating formalized process with this degree of veneration at all simply perpetuates the political system that needs inexperienced developers to roll over and play dead to the system._

</comment></comment></comment></comment></comment><comment><USER2> Author here, writing this book was one of the hardest things I've ever done, and there is seemingly no end to the small issues I've faced (both writing and technical). Would appreciate some feedback to make it as good as it can be, AMA thanks.

<comment><USER4> The separation of design and construction into phases is a hangover from civil engineering. It has the baked in assumption that the design phase is relatively cheap, short and somewhat unpredictable the construction phase is expensive, long and predictable. The root problem is the assumption that specifications can be validated for correctness, like a blueprint for a bridge can. Nothing could be further from the truth. This is a persistent myth in software development.

<comment><USER5> > The root problem is the assumption that specifications can be validated for > correctness, like a blueprint for a bridge can.

We can validate specifications for correctness with automated theorem proving! The least we can do is model-checking. At least for the hard algorithms and core system interactions. We've had this ability for decades.

The problem with software development is that we get all hand-wavy about "architecture," and "design." I try not to physically groan when someone starts sketching out a box diagram. They're fancy pictures, sure, and useful at a conceptual level... but the reality is that the software will look nothing like that diagram. And there's no way to check that the diagram is even correct! Useless!

It doesn't have to be painfully slow, burdensome, and expensive to employ these tools either. Contrary to popular belief it doesn't take a team of highly trained PhD physicists to build a formal mathematics for modelling computations these days. There are plenty of open-source tools for using languages like TLA+ that work great besides! Ask Amazon -- they published a paper about it.

Watch _Leslie Lamport - Who Builds a Skyscraper without Drawing Blueprints?_ : [https://www.youtube.com/watch?v=iCRqE59VXT0](https://www.youtube.com/watch?v=iCRqE59VXT0)

<comment><USER6> I draw pictures because I think in pictures.

I just took a short look at TLA+ and read a tutorial.

Don't think I could use it without drawing a picture.

<comment><USER5> Drawing a picture is a perfectly valid and legitimate start. However I think there are many problems where a picture is not sufficient and it's necessary to have good, reliable mathematics to sketch out our designs and prove properties of them.

A sketch is still useful the the absence of nothing else. But we don't build bridges and skyscrapers that way so why is it good enough for our data-centres and applications?

I don't think lives have to be on the line in order for software to be considered harmful if not built correctly. There are plenty of problems where having a thoroughly checked design will save you plenty of headaches and afford you many more opportunities.

<comment><USER7> > However I think there are many problems where a picture is not sufficient > and it's necessary to have good, reliable mathematics to sketch out our > designs and prove properties of them.

It's necessary to have a formal language to describe this design. Such a formal language needs to have one and only one interpretation. We should be able to take a document written with that formal language and verify that it meets our assumptions. We should also be able to verify that the final implementation is isomorphic to the design.

Luckily such tools are ubiquitous. You can use whatever programming language you like to specify your design. In the case where the programming language is defined poorly, then you must also decide what build system and run time environments you will support ahead of time.

You can document your assumptions using the same programming language. This is made easier if you use a "test framework", but if needed you can also roll your own. You can validate your design by running the tests against your design. Normally it is best to break up your design into "units" that make it more clear how to validate your assumptions. You can then add some extra validation that the composition of "units" transitively validates your assumptions.

You can verify that your final produce adheres to that design because there will be a 1:1 mapping from the design to the final product (i.e. the final product will be implemented using the same code as your design). There are wonderful tools that will tell you the "coverage" of code used in the final product that has been validated against the assumptions.

Finally, you can even specify your requirements using the same formal language and verify that the design meets the requirements. Normally you do this by validating that the "integration" of "units" meets your assumptions. This should not be done without individually validating the assumptions on the "units", though, because the "integration" can lead to exponentially growing alternatives. Normally it is infeasible to validate each alternative.

Yes, this reply is tongue in cheek, but I am not ignorant of formal specification methods. They have their place. That place is currently not in a professional software development shop. We have better methods at the moment. Possibly formal specifications methods will improve to the point where we can reasonably use them, but we aren't there yet.

<comment><USER5> > Possibly formal specifications methods will improve to the point where we > can reasonably use them, but we aren't there yet.

I disagree. Amazon has had great success employing TLA+ in finding bugs, testing design changes, and chasing aggressive optimizations [0].

Perhaps it is because there are myths that are still floating around regarding formal methods that still make developers cringe when they hear mention of them [1].

None the less I couldn't find reference to it in the book... did I miss it?

And besides... unit tests, I'm sure you are aware, aren't good enough alone. They can only increase your confidence in an implementation but they prove nothing.

If we want to start calling ourselves _engineers_ I think we better start looking to how the other engineering disciplines operate. I don't think it will be long before an insurance company comes along and finds actuaries capable of monitoring risk in software development projects and liability becomes a necessary concern.

[0] [http://research.microsoft.com/en- us/um/people/lamport/tla/fo...](http://research.microsoft.com/en- us/um/people/lamport/tla/formal-methods-amazon.pdf)

[1] [http://www.fm4industry.org/index.php/G-HM-2](http://www.fm4industry.org/index.php/G-HM-2)

<comment><USER6> > I disagree. Amazon has had great success employing TLA+ in finding bugs, > testing design changes, and chasing aggressive optimizations [0].

I googled TLA+ examples and tutorials and I am wondering how I would apply this to an already existing application?

<comment><USER5> I look for the critical bottlenecks in the application where the cost of failure or mistakes is fairly high. This is where I feel I will get the most "bang for the buck" so to speak.

You don't have to specify the entire application to get the benefit of high level specifications. Even specifying the protocol between two communicating channels can bring benefits.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> In a book like this, I'd like to see cost estimation mentioned earlier than Chapter 8. I'd argue that in nearly all software development projects (particularly the ones where this sort of book applies) the cost of implementation is one of the key factors in evaluating requirements. If you put together your requirements without considering their cost, you end up with a project that's too expensive to build, and you want to know that during the Requirements Gathering phase, not later on during construction. Even Agile projects need to start out with a rough idea of what's being built and a rough idea of what it'll cost to build it. Otherwise how does the person funding the project decide whether or not to approve it?

Of course, as we all know providing estimates during the requirements phase is very difficult, especially if they're treated as hard commitments rather than rough ballparks. Chapter 2 mentions that getting requirements wrong is a key factor in causing software projects to fail; I'd say that the reason for that is usually because of the implementation costs of the known requirements that were estimated inaccurately or not at all, and the implementation costs of requirements that aren't discovered until later. It always comes down to cost; I think it's relatively rare for a software project to fail because a requirement turned out to be impossible to implement. (Unless it involves AI. You always have to watch for people trying to sneak in an AI requirement.)

<comment><USER2> Great high-level feedback, thanks. I'll add a cost/estimates component to Requirements.

<comment><USER2> I've added a stub section on the subject that I will expand as I learn more: [http://mixmastamyk.bitbucket.org/pro_soft_dev/sdlc_2_req.htm...](http://mixmastamyk.bitbucket.org/pro_soft_dev/sdlc_2_req.html#cost- estimation)

</comment></comment></comment><comment><USER9> Thanks to the author -- it does like like your labour of love and written with some personality. (That's a good thing!) Attractive layout, lots of insets/quotes/diagrams, and sources. Will read more later.

The "Models & Methodologies" chapters looks great. It may become my new "here, read this!" when people ask me "what's agile?" or "how else?"

[http://mixmastamyk.bitbucket.org/pro_soft_dev/models.html](http://mixmastamyk.bitbucket.org/pro_soft_dev/models.html)

<comment><USER2> Thank you, I've tried to do exactly that, give new people a good overview with not too much information at once, and encourage further study with copious references.

</comment></comment><comment><USER10> I'm disappointed to see a book aimed at "professional" developers continue to spread outdated and debunked information, such as the NIST "study" adduced as evidence for the imperative necessity of "defect cost containment". See my post on the topic: [https://plus.google.com/+LaurentBossavit/posts/8QLBPXA9miZ](https://plus.google.com/+LaurentBossavit/posts/8QLBPXA9miZ)

Or again the Wikipedia page on the history of software engineering, which is frightfully inadequate. [https://plus.google.com/+LaurentBossavit/posts/gpSwoWn4CBK](https://plus.google.com/+LaurentBossavit/posts/gpSwoWn4CBK)

I'll add my voice to those that have already stated such a book shouldn't start by assuming the SDLC as a reference model: it embodies too many of those outdated assumptions. More in that vein in my own book [http://leanpub.com/leprechauns](http://leanpub.com/leprechauns)

<comment><USER5> I haven't read all of it yet but it also seems to miss the vein of development that has led to movements such as _Correct by Construction_ et al. The idea that we should model our computations in a high-level specification and check that those designs meet our goals and hold the invariants we press upon them.

Instead there's Agile and the idea that we can throw together something that roughly works and iterate until our confidence is enough such that we can release it. The so-called, _beta-driven-development_. (Perhaps a vestigial remnant of the unix philosophy?).

I'm not arguing that formal methods should be used for every software project. I think Carmack was right to point out that if all software was written like the software at JPL we'd be decades behind where we are now. However I do think that it should be a part of the experience of becoming a programmer so that when we encounter hard problems we have the correct instincts to rely on mathematics to help us.

<comment><USER2> This is a beginners book, and keeping in mind we can't teach them everything in the first "semester," do you still think this topic should be included?

<comment><USER5> Definitely think the groundwork can be laid out without covering the entire breadth of formal methods. Even a history lesson with Djiskstra, Lamport, Mizra, et al can be illuminating. It's good to lay the foundation so that when a beginner encounters a problem they have the instincts to know that they should look up these tools in order to help them.

I found it a shame that I didn't encounter these tools only until very recently. It's a well kept secret in academia that I think should be shared.

</comment></comment></comment></comment><comment><USER11> Offtopic: How do I get that github.io type hosting for my own bitbucket account? I didn't know it was possible.

Edit: It's as mixmastamyk says. I created a repo called 'achshar.bitbucket.org' and it works.

<comment><USER12> You can use the Aerobatic add-on for Bitbucket

[https://www.aerobatic.com/](https://www.aerobatic.com/)

SSL, custom domains, static generator auto-builds, etc.

disclaimer: co-founder of Aerobatic

<comment><USER13> I'm an Atlassian engineer & heavy Aerobatic user. It's very nicely integrated with Bitbucket, and has great support for Jekyll, Hugo, and arbitrary npm builds. It also has some advantages over github.io like being able to deploy multiple feature branches from the same repository to separate sites, so you can have separate "staging" and "production" versions.

It is a paid offering, though you get two repositories free, and is very pretty reasonably priced beyond that.

</comment></comment></comment><comment><USER14> Definitely touch on a lot of topics here. And all looks well organized. It also does look like a laundry list of buzzwords and techspeak.

Obviously this wouldn't be used to teach anyone any particular topic in detail but to get them familiar with the general concepts/steps involved in software dev.

Edit:

The two books I read that I thought covered these ideas well were Code Complete and Code Craft. But it's been about a decade now. Perhaps they're too dated.

<comment><USER2> > It also does look like a laundry list of buzzwords and techspeak.

Thanks, interesting. I've tried to define difficult terms, and it is aimed at a technical audience, but there is definitely room for improvement. If there are any readers having trouble, I'd appreciate hearing where. Will take a look myself as well.

<comment><USER14> I picked the Design section to read first. Perhaps that one isn't as detailed as some of the others.

I'm not trying to knock you here as I'm not exactly the audience for a publication of this type anyway.

It does give a good idea about what you should know/be familiar with and a bunch of link outs to other sources on the web.

<comment><USER2> Yes, I think you're right. Design is such a huge topic and hard to condense into one chapter (one of the shorter). I will pull more details into it and perhaps others, thanks.

</comment></comment></comment></comment><comment><USER15> Why are you using Joy Division's Unknown Pleasures album as the cover of the book?

<comment><USER2> It is a scientific image, I believe public domain. Happen to love music and you will find that reading the book. The cover is the first clue. ;)

(More information is at the bottom of the intro/title page under Acknowledgements).

<comment><USER16> I guess, but it's lacking the dyadic relationship between the image and the music (or in your case: content).

That's what made that album cover so great.

<comment><USER17> Are you drunk? You think a data plot has an intimate relationship with music but not software? Sounds like someone is romanticising. ;)

</comment></comment></comment></comment><comment><USER18> Kudos to the author for tackling this huge and controversial topic.

Read first several chapters and picked up the impression that the author doesn't put enough effort to point out how the processes/practices can (and should) be completely different depending on the circumstances. If a startup tries to use the same processes as Google or Facebook, it'll be dead in the water. If SpaceX engineers write software the same way as SnapChat does it, we will never see their rockets leaving launch pads.

<comment><USER2> Thank you. I'm going to beef that up in the introduction, due to several comments such as yours. May I quote you?

</comment></comment><comment><USER19> Interesting book. I'll take some time to do a read through. A few things that bother me though off the bat:

1\. Anything that uses the word 'protip' can not be taken seriously. I think this needs a law. 'The Law Of Silly Programming Memes - Anything using the word 'protip' cannot be taken seriously'

2\. The 800px width format in the world of responsive design gives me pause. Basically this says 'I'm for mobile - screw you'. I would hope that a better format for this would be chosen in the future.

<comment><USER2> Oh, always read "protip" on reddit with a smile. It is aimed at the young, sounds like you're being a bit chatinho.

Also thought it was recognized that narrow columns are easier to read, such as in a newspaper. It uses the well-regarded "read the docs" theme. Maybe zoom would help?

</comment></comment><comment><USER20> Here's some for you collection given I see some omissions. Cleanroom, always omitted (sighs), is a big one as it was doing agile-like development in 80's with code so reliable it was sometimes warrantied. Also one of first, formal methods that didn't require a mathematician to use. Fagan's Software Inspection Process came before that in the 70's. I throw in Praxis and 001 for good measure as they're _engineered_ software methods with better results than Cleanroom albeit at higher cost. Leave off plenty of others too constrained for most software development but did prove out in smaller projects. The B Method & Chlipala's Certified Programming in Coq are examples if you want to Google around.

[http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.88....](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.88.2667&rep=rep1&type=pdf)

Note: An academic recently combined Cleanroom with Python for some nice results given how high-level Python is. I thought Haskell would be more ideal.

[https://en.wikipedia.org/wiki/Major_Defect](https://en.wikipedia.org/wiki/Major_Defect)

Note: Describes Fagan process with relevant links.

[http://www.sis.pitt.edu/jjoshi/Devsec/CorrectnessByConstruct...](http://www.sis.pitt.edu/jjoshi/Devsec/CorrectnessByConstruction.pdf)

Note: Altran/Praxis Correct by Construction is a modern high-assurance method with numerous successes. Cost a 50% premium for nearly defect-free systems. SPARK Ada is GPL these days.

[http://htius.com/Articles/articles.htm](http://htius.com/Articles/articles.htm)

Note: Margaret Hamilton, who helped invent software engineering on Apollo mission, deserves mention for the first tool that automated most of software process. You can spec a whole system... one company specified their whole factory haha... then it semi-automates design then automatically does code, testing, portability, requirements traces, and so on. Guarantees no interface errors, which are 80+% of software faults. Today's tools have better notations & performance but still can't do all that for general-purpose systems: always a niche.

[https://www.eiffel.com/values/design-by- contract/introductio...](https://www.eiffel.com/values/design-by- contract/introduction/)

Note: Added Eiffel method to make up for fact that I have little to nothing on OOP given I don't use OOP. Meyer et al get credit for a powerful combo of language features and methodology in Eiffel platform with huge impact on software. Specifically, Design-by-Contract has so many benefits that even SPARK and Ada both added it to their languages. Just knocks out all kinds of problems plus can support automated generation of tests and such.

So, there's you some reading on methods of making robust software that might fit into your book or something else you do. :)

<comment><USER21> How often is formal correctness actually used in industry applications? Is this common outside of high risk things like aerospace?

Thanks a lot for sharing this, by the way

<comment><USER20> It's not common. The problem is a combination of user demand, vendor incentives, social issues, and no legal liability. User demand kept pushing for more features, faster, and cheaper price at all cost. Quality, security, and maintenance are the consistent costs. Vendors have been pushing broken stuff intentionally since early days to make immediate profit on cost savings then more as they supply patches. They also try to ship as fast as possible to get First Mover advantage even though medium assurance methods still work with that. They spend a little time upfront to knock out lots of debugging time. Gabriel's Worse is Better essay and how much people hold onto C are examples of the social aspect where something spreads like wildfire. They then justify its failings or keep them for convenience. Lastly, the areas most prone to high assurance have regulations, policies, or legal liabilities that encourage its use whereas most vendors immunize themselves against liability with EULA's. Worse, customers seeing unreliable, insecure systems everywhere are conditioned to think that's inevitable rather than artificial and subject to change.

Note: After a big recall, the hardware field is the one exception where they have all kinds of formal verification and testing. They're big on that stuff. Not same tools as software, though, for the most part.

Far as those using it, it helps to look at what products are available and who vendors say are their customers. Look at high-assurnace plus medium as many former customers of high-assurance do medium these days due to above reasons. Even most vendors in the niche are saying "F __* it... " since demand is so low. So, you get especially high-security defense, a few in private security, some banking, aerospace, trains/railways, medical, critical industrial (esp factories or SCADA), firms protecting sensitive I.P., and some randoms. The suppliers are usually defense contractors (BAE's XTS-400, Rockwell-Collins AAMP7G); small teams in some big companies (eg IBM Caernarvon, Microsoft VerveOS); and small firms differentiating with quality & security (Altran/Praxis, Galois, Sentinel HYDRA, Secure64's SourceT).

Here's some examples. Some have marketing teams in overdrive. Just ignore it for use-cases, customers, and technical aspects. ;) Altran comes first as they focus on high quality or effectiveness for premium pay, with some high- assurance. Probably a model company for this sort of thing. AdaCore lists lots of examples which are _actual_ customers. Esterel has a DSL with certified, code generator that has plenty uptake. INTEGRITY links show common industries & specific solutions that keep popping up on security side. NonStop is highly- assured for availability with reading materials probably having customer info. Last one is a railway showing B-method, most common in that domain, doing its job. Hope this list is helpful. I can't do much better in a hurry since the field is so scattered and with little self-reporting.

[http://www.altran.com/](http://www.altran.com/)

[http://www.adacore.com/customers](http://www.adacore.com/customers)

[http://www.esterel-technologies.com/success-stories/](http://www.esterel- technologies.com/success-stories/)

[http://integrityglobalsecurity.com/pages/commercial.html](http://integrityglobalsecurity.com/pages/commercial.html)

[http://integrityglobalsecurity.com/pages/otherSolutions.html](http://integrityglobalsecurity.com/pages/otherSolutions.html)

[http://www8.hp.com/us/en/products/servers/integrity/nonstop/...](http://www8.hp.com/us/en/products/servers/integrity/nonstop/index.html)

[http://www.methode-b.com/wp- content/uploads/sites/7/dl/thier...](http://www.methode-b.com/wp- content/uploads/sites/7/dl/thierry_lecomte/Formal_methods_in_safety_critical_railway_systems.pdf)

</comment></comment></comment><comment><USER2> Thanks all for the great discussion and feedback, I've already incorporated much of it and will continue to improve the weaker sections.

</comment><comment><USER22> The double icon for external links to youtube|wikipedia|twitter etc. seems superfluous and makes things harder to read.

<comment><USER2> I'll take a look at toning that down tomorrow.</comment></comment>