What is "segregated witness" and how can it improve network scalability? I've just heard about it, and it is claimed to be a "turning point" in the block-size debate.

What is "segregated witness?" how does it work?

How can it be used for improving network scalability?

Is this the answer to all of network's scalability concerns?
It's not a silver bullet solution, but it's a really good start.  

As Gavin Andresen has said, Segregated Witness is a poor name. The 'segregated' part of the name is there to denote that there is separation being done. The 'witness' portion of the name comes from the fact that digital signatures are often time called witnesses. 

Segregated witness splits up transactions into different parts that can be handled separately instead of the single chunk of data as they are now. Specifically, it takes the digital signatures out of transactions and puts them in a separate merkle tree that has the same structure as the transaction merkle tree. So, if fully implemented, to check that an input legally spends its previous output, you would get the signature from the signature tree, instead of the standard scriptSig field.

These are a few of the benefits of this idea:


Since signature data (witness data) is stored outside the transaction (and outside the standard block), it means that that data doesn't have to be counted towards the block size. Pieter Wuille is proposing a 75% discount on space taken up by signature data, meaning that you can fit 4x as much signature data into blocks. This effectively results in a soft fork increase to the block size.
Completely solves malleability issues. Using transactions with the signature data outside the transaction means that TXIDs don't hash the signature data, which means that they're not malleable (assuming you're using the standard SIGHASH flag). Technically, the signatures are still malleable, it's just that modifying them doesn't invalidate chains of transactions because the signatures don't sign the modifiable parts.
Allows for a slow upgrade. Software has to opt in to using Segregated Witness after it has been fully deployed to the network, but in the meantime (and afterward) transactions can still be made as usual without segregated witness.
All future Script updates become soft forks. When segregated witness gets fully implemented, it will have a version byte in outputs for what version of Script it is using. And the behaviour for clients that see a script with a non recognized version number is that they treat it as an 'anyone can spend' output. 
Signatures only prove that a transaction is authorized, it doesn't describe where funds are going or where they came from. So, after they're checked they can be discarded. Putting the signatures in a separate data structure makes it much easier to prune that data, which results in much less blockchain data needing to be stored on your hard drive.


However, this doesn't completely increase the block size, it just increases the amount of signature data that a block can store. Since transactions are roughly 60% made up of signature data, this is still a pretty big gain.

The downside is that it's a non-trivial upgrade for the network to start using segregated witness. The transaction serialization format is different, and code everywhere that makes bitcoin transactions needs to be updated. Since it's an opt-in process, though, upgrading can be done slowly over time.

You can watch Pieter Wuille's great talk on Segregated Witness at:

https://www.youtube.com/watch?v=fst1IK_mrng&amp;feature=youtu.be&amp;t=37m12s.