Write a Shell in C (2015) - <AUTHOR> https://brennan.io/2015/01/16/write-a-shell-in-c/ <comment><USER2> If you want to see a pretty complete shell in a higher level language (Python, 11K lines of code), check out:

[https://github.com/oilshell/oil/](https://github.com/oilshell/oil/)

And a blog here: [http://www.oilshell.org/](http://www.oilshell.org/) (index: [http://www.oilshell.org/blog/2016/11/20.html](http://www.oilshell.org/blog/2016/11/20.html))

This article is good in that it shows that starting processes boils down to the system calls fork(), exec(), wait(), etc.

My shell has function calls, loops, conditionals, pipelines, subshells, command substitution, redirects, here docs, etc. (The code has some messiness, but the architecture is clean IMO.)

It turns out that these features require just a few more system calls -- dup2() and fcntl() to manipulate file descriptors, pipe(), and open() / read() / write() / close(). It's a good reminder of how simple Unix is. The complexity of the shell is basically all in string processing, and not in interfacing with the kernel (although the interactive parts of the shell do complicate things).

These system calls are all available in Python, so the code ends up being comparable to what you would write in C (and I'm in the process of porting to C++ now that the architecture has settled.)

<comment><USER3> Microsoft PowerShell is pretty complete in a higher level langage (C#, mostly) and is open source on Github here:

[https://github.com/PowerShell/PowerShell](https://github.com/PowerShell/PowerShell)

<comment><USER4> ...and absurdly complex.

One of the quick tests I like to do to ascertain the complexity of a codebase is "how long does it take to find the entrypoint" and "how long does it take to find the core functionality"?

In this case, I have been clicking around through nearly empty directories for 15+ minutes and haven't seen anything that looks like an entrypoint yet, nevermind the main loop.

In contrast, I downloaded the Bash source and within 5 minutes found the main loop (eval.c) and entrypoint (shell.c).

Now I'm curious to know what the cmd.exe source is like, but MS seems to have no intention of releasing that.

<comment><USER5> Some hand-wavily relevant comments:

\- I had the exact same problem - "how do I get at the entrypoint and follow this" \- only earlier today, with Chromium. I was trying to figure out why sync was broken and thought I'd try and follow the (C++) breadcrumbs. It didn't go very well. [https://bugs.chromium.org/p/chromium/issues/detail?id=671498](https://bugs.chromium.org/p/chromium/issues/detail?id=671498)

\- AFAIK, cmd.exe is basically a horrible, horrible hack. I'm not 100% sure, but I think an instance of cmd.exe is the only way you can run things with a stdout, as in Windows has no concept of TTYs - the cmd.exe process uses a bunch of undocumented kernel interface APIs to "make it work™". Related: [https://github.com/Microsoft/BashOnWindows/issues/111#issuec...](https://github.com/Microsoft/BashOnWindows/issues/111#issuecomment-238302654)

<comment><USER5> Edit/Update: PSA, if you use a sync passphrase with Chrome your history from other devices doesn't show up in chrome://history. Known bug, will hopefully be rearchitected long-term.

I have to say though, the writeup on that Chromium ticket I filed was absolutely awesome. Did someone here notice...? :P

</comment></comment></comment></comment></comment><comment><USER6> This is very well written, and it seems the author has already used the great code reviewing powers of the Internet (there are references to improvements from Reddit comments).

I suffer from compulsory C review disorder, so just a few quick points:

\- The pattern "x = realloc(x, ...)" is a bad idea, it leaks the original memory on failure. Since the code seems to car about its memory management (and since it's a tutorial), I think this is worth mentioning (and fixing).

\- I generally recommend against scaling string allocations by sizeof (char); that's always 1 so it doesn't _do_ anything. Some people seem to feel that it's part of a general usage pattern that makes the intent more clear, though.

\- I would recommend more "const" and "static", but some people seem to not care.

\- I would use more size_t rather than int for things that are sizes. This is more clear, and also safer since it's unsigned.

\- Using strtok() is not always the best choice, it has issues, but here it's basically doing what it was designed to do I guess.

<comment><USER7> > The pattern "x = realloc(x, ...)" is a bad idea, it leaks the original > memory on failure. Since the code seems to car about its memory management > (and since it's a tutorial), I think this is worth mentioning (and fixing).

What would the alternative be here? Allocate a new region, copy everything over and free the other region? Or is there something better?

> I generally recommend against scaling string allocations by sizeof (char); > that's always 1 so it doesn't do anything. Some people seem to feel that > it's part of a general usage pattern that makes the intent more clear, > though.

Isn't this just a habit born out of portability concerns?

<comment><USER6> The alternative is to use a separate variable:

        y = realloc(x, ...);

Then if it fails, you can still free(x) before continuing. Of course, if you're going to exit() due to the failure it doesn't matter and perhaps that's why the original code looks that way. I still think it's worth pointing out, though.

About sizeof (char), there can be no "portability concerns", and that very thing is what I think gives rise to a lot of (in my opinion, bad) cargo cult programming. The value of "sizeof (char)" in C is 1. Note that I didn't say "using GCC x.y.z on x86". It's true for all conforming compilers, on all platforms, and on all architectures. It's defined by the language.

<comment><USER8>
      > The value of "sizeof (char)" in C is 1.

For those who might not be aware, this _does not_ mean that a char is always 8 bits.

<comment><USER9> According to C99 spec, the maximum number of bits for the smallest object that is not a bit-field is 8 bits. See "CHAR_BIT" in the C99 spec.

[1] [http://www.open- std.org/jtc1/sc22/wg14/www/docs/n1256.pdf](http://www.open- std.org/jtc1/sc22/wg14/www/docs/n1256.pdf)

<comment><USER10> Smallest. It might be larger (and infact it is in a lot of DSP toolchains)

</comment></comment></comment></comment></comment></comment><comment><USER11> My university has a project where you implement a shell in c from scratch with most of the commonly known features of bash -- scripting, file redirection, command substitution, the whole nine yards. Needless to say it's a mess by the end.

<comment><USER7> Does the scripting part utilize ready made libraries or is it implemented from scratch?

<comment><USER11> By scripting I mean if, while, and friends (maybe for, I don't remember)

It's also from scratch (no libraries).

The professor provides an extensive test suite to check your implementation.

</comment></comment></comment><comment><USER12> A nice complement to this article is the MIT 6.828 shell exercise. It provides the command loop and parsing for you, and then has you implement command execution, redirection, and pipes.

[https://pdos.csail.mit.edu/6.828/2016/homework/xv6-shell.htm...](https://pdos.csail.mit.edu/6.828/2016/homework/xv6-shell.html)

</comment><comment><USER13> The classic for me of an exposition of writing a shell is Advanced UNIX Programming by Marc Rochkind (Bell labs guy and author of SCCS) (not to be confused with W Richard Steven's text which is also excellent)

[http://basepath.com/aup/](http://basepath.com/aup/)

First published in 1985; for me ranks up there with K&R The C Programming Language and K&P The UNIX Programming Environment.

<comment><USER2> Where does it talk about writing a shell?

As far as I can tell, it's about using the kernel programming interface, which is of course very relevant to writing a shell. But I don't see anything about the shell specifically here:

[http://basepath.com/aup/toc.htm](http://basepath.com/aup/toc.htm)

<comment><USER13> Starting from section 5.4, Implementing a shell, version 1

As it suggests, he adds features to the shell as he introduces the system calls needed. You can see the source here

[http://basepath.com/aup/ex/group__AUP2ex.html#Chap](http://basepath.com/aup/ex/group__AUP2ex.html#Chap). 5

See files sh0.c (for versions 1 2 3) and sh3.c for version 4.

<comment><USER2> Yes this looks nice. xv6 has a tiny shell as well:

[https://pdos.csail.mit.edu/6.828/2016/xv6.html](https://pdos.csail.mit.edu/6.828/2016/xv6.html)

</comment></comment></comment></comment><comment><USER14> I tried to implement a shell in C once, just to see how to do it. Until I realized, I was basically just writting a shell language interpreter, and thus implementing a lexer and parser for that language manually (which it seems is exactly what happens in the article).

Which made the whole project boring to me, since it would've just come down to 3 well-defined, already solved problems: 1\. write shell language grammar file 2.Use something like flex/bison (nowadays maybe antlr, or whatever) to generate the shell language parser 3\. Implement the behaviour of the shell as reactions to parser events

At that wasn't interesting because: 1\. I wanted to be POSIX compatible, the POSIX shell langauge is well defined, writing the grammar for that would've been boring 2\. Boring by design, the whole point of using parser generators is making compiler/interpreter writing an easy, boring task 3\. Arguably, this would've been interesting, to see how to do it. But if you already have a deep understandidg how a *nix shell does what it does, not that interesting anymore...

<comment><USER2> A shell is indeed an interpreter for a programming language, and you do need a lexer and a parser (actually 4 parsers, and my lexer requires 13 modes).

But that's not what's happening in the article -- he is showing a simple REPL and fork() exec(), which is about as much as you can expect to do in an article that long. lsh_split_line() in no way resembles a real shell lexer, and there is no parser, since there are no programming language features like function calls, loops, and conditionals. Not to mention pipelines, subshells, and redirects.

I think you're overestimating the power of flex, bison, and ANTLR. I actually ported the POSIX shell grammar to ANTLR -- it's not usable as the basis for a shell parser. The POSIX grammar also only covers 1 of 4 sublanguages in shell, and it's a significantly smaller language than bash.

Bash uses bison/yacc, and maintainer Chet Ramey talks about how it was a mistake here:

[http://www.aosabook.org/en/bash.html](http://www.aosabook.org/en/bash.html)

My blog is basically about parsing bash, and I discovered a lot of interesting things:

[http://www.oilshell.org/blog/2016/10/20.html](http://www.oilshell.org/blog/2016/10/20.html)

[http://www.oilshell.org/blog/2016/11/01.html](http://www.oilshell.org/blog/2016/11/01.html)

[http://www.oilshell.org/blog/2016/10/17.html](http://www.oilshell.org/blog/2016/10/17.html)

<comment><USER14> Well, very interesting indeed! Cool that you actually went through with writing a complete shell :)

I guess, I might've been too optimistic about the power of parser generators. Still, I think if someone were to start implementing a shell, I'd advise them to start with the grammer/lexer/parser part, either implementing them themselves or trying some of the generators. I think it's the right way to go about it (just wasn't what I thought I'd be doing when implementing a shell, at the time)

I did do a few things with parser generators though. Granted, nothing (yet) as complex as bash or even just POSIX, but still, they each had their own little trickinesses. E.g.:

[https://github.com/BugsBunnySan/edl](https://github.com/BugsBunnySan/edl) (ANTLR4 / Python)

[https://github.com/BugsBunnySan/Phat- Agnus](https://github.com/BugsBunnySan/Phat-Agnus) (YAPP / Perl)

</comment></comment></comment><comment><USER15> He completely misses a very important aspect: Signal handling. Fortunately, there is a page covering this complex topic well:

[https://www.cons.org/cracauer/sigint.html](https://www.cons.org/cracauer/sigint.html)

<comment><USER4> Best read in combination with [http://www.linusakesson.net/programming/tty/](http://www.linusakesson.net/programming/tty/)

</comment></comment><comment><USER16> This is a such a great tutorial. I don't code in C at all and I hardly understand the language. Yet, I was able to follow this and understand.

I mostly write Python and if you are looking for a shell written in Python, check xon.sh [0]

[0] - [http://xon.sh](http://xon.sh)

</comment><comment><USER17> Read uClibc and Busybox code. They're both fun platforms to hack on and the people who wrote them know a little bit about C.

<comment><USER18> As someone who still considers himself a beginner at C, busybox source code is fun as hell to read. So many "Oh, so that's how you do that" moments.

</comment></comment><comment><USER19> It's not a shell up to late 1980's standards until it handles POSIX job control (Ctrl-Z suspend, move another job from background to foreground) and traps Ctrl-C interrupts to return to the prompt.

Adding some of these requirements as an afterthought into a command interpreter will likely be a pain in the butt. Ctrl-C is usually done with a longjmp(). Unplanned introductions of setjmp/longjmp into a code base could wind up with leaks or stability issues. (Side note: don't write a shell in C: pick some open source managed language in which it is safe to throw an exception out of a signal handler.)

Another requirement is arranging pipes, and handling out-of-order termination of the pipeline elements; a shell must juggle multiple child processes in order to support pipes. Pipes are put into process groups called jobs, etc.

Without all these features, you don't have a Unix shell; just an interpreter for a simple command language.

<comment><USER2> For Ctrl-C and longjmp(), are you talking about builtin shell commands? I think it will "just work" for external processes. They receive SIGINT and terminate, and then the shell gets control again.

I guess I need to do a test of doing some expensive computation in bash, and then hitting Ctrl-C. Like doing the mandelbrot set or something.

<comment><USER19> Hello, World test case:

      bash$ while true ; do : ; done
      ^C
      bash$

<comment><USER2> Good tip, thanks!

I think you can use any of C++ exceptions, explicit returns, or setlongjmp(). setlongjmp() doesn't work well with C++ destructors. The interpreter loop just has to check for a flag set by the signal handler and then pop back to the interactive loop.

I noticed that the Lua parser uses setlongjmp() when compiled as C, but C++ exceptions when compiled as C++.

</comment></comment></comment></comment><comment><USER20> Also, you can check the original sh by Stephen R. Bourne (Bell Labs, [1]). Or the original csh by Bill Joy (BSD, [2]). In Github there is a huge repository with a collage of Unix History, where is possible to check how shells have been evolved ([3]).

[1] [https://github.com/dspinellis/unix-history-repo/tree/Bell- Re...](https://github.com/dspinellis/unix-history-repo/tree/Bell- Release/usr/src/cmd/sh)

[2] [https://github.com/dspinellis/unix-history- repo/tree/BSD-2-S...](https://github.com/dspinellis/unix-history- repo/tree/BSD-2-Snapshot-Development/src/csh)

[3] [https://github.com/dspinellis/unix-history- repo](https://github.com/dspinellis/unix-history-repo)

</comment><comment><USER5> Hi.

I'll just leave this here.

[http://unix.derkeiler.com/Newsgroups/comp.unix.shell/2009-10...](http://unix.derkeiler.com/Newsgroups/comp.unix.shell/2009-10/msg00324.html)

Like shells? This is the link you didn't know you were looking for, but were hoping to find anyway.

<comment><USER21> Great links. Would have come in helpful a year ago. I built most of (d)ash clone in Go as part of my UnderGrad dissertation. [https://github.com/danwakefield/gosh](https://github.com/danwakefield/gosh)

</comment></comment><comment><USER22> This is definitely a cool little tutorial, but I'd strongly caution anyone who's writing string-handling software (and what is a shell _but_ string- handling software: the input and the output are all strings!) not to write it in C. There are higher-level languages with good string support which support fork(), exec(), wait() & friends, and are far less susceptible to string- handling and memory errors.

That said, a shell's remarkably limited scope is actually something C _is_ reasonably suited for.

<comment><USER2> I don't think there are that many languages that are good for writing a shell. There seem to be a few shells written in Go, but Go only reluctantly supports fork(), because it interacts with its threaded runtime. Python is actually closer to what you want than Go, but I think it will end up having problems with signal handling, because the Python interpreter does its own signal handling. The prototype of my shell is written in Python [1].

Garbage collection almost always interacts poorly with signal handling. You can't interrupt a garbage collector at an arbitrary point.

I wanted to bootstrap my shell [1] with Lisp -- and I hacked on femtolisp because Julia is bootstrapped in exactly this manner. And for awhile I thought about using an OCaml front end.

But in the end I settled on C++ (writing 3K lines of code, which I plan to return to after my prototype is done.) C++ lets you do fork(), exec() and handle signals exactly as in C, but it actually has useful string abstractions!

C++ has a lot of problems, but it appears to be the best tool for this job.

[1] [https://github.com/oilshell/oil/](https://github.com/oilshell/oil/)

<comment><USER19> I think that a shell is doable using my TXR Lisp as a base.

As far as signal handling goes, it's in the box. See this Rosetta Code example under the task "Find limits of recursion" where we catch a SIGSEGV using a TXR Lisp lambda (running on an alternate stack via sigaltstack):

[https://rosettacode.org/wiki/Find_limit_of_recursion#TXR](https://rosettacode.org/wiki/Find_limit_of_recursion#TXR)

Handling a SIGINT:

[https://rosettacode.org/wiki/Handle_a_signal#TXR](https://rosettacode.org/wiki/Handle_a_signal#TXR)

Programming reference:

[http://www.nongnu.org/txr/txr- manpage.html#N-010CFD89](http://www.nongnu.org/txr/txr- manpage.html#N-010CFD89)

You don't have to worry about signal handlers and garbage collection.

If you need to do something in C++, the TXR code base will support you. Though it is C, it all compiles in C++, which I check for regressions every release. (At this time, I won't accept patches which break away from C compatibility and cause C++ to be _required_ , but for experimenting and forking off your own thing, there it is.) Just ./configure cc=g++ and off you go.

TXR also integrates a nicely hacked version of Linenoise that could be used to bootstrap the shell command prompt. I rewrote the multi-line support so that it is excellent. There is visual cut/copy/paste, paren matching, undo, and more. (It's the only part of TXR without proper Unicode support, unfortunately: in the TODO list.)

<comment><USER2> Yes, interesting project, and I think it would work. But femtolisp would work fine too... In the end I decided not to use a Lisp because it wasn't making me more productive. It felt a bit unstructured.

Instead, it looks like I will generate a lot of C++ from Python to control the complexity (and line count).

For example, right now I'm working with Zephyr ASDL, which is basically a DSL for algebraic data types which can bridge Python and C++. You can do this in Lisps too of course, but Python works just as well in this case. Julia is interesting because the lexer and parser are in femtolisp, and that enables Julia macros.

If I'm reading your page right, Python handles signals the same way -- it receives the signal in C, and then runs a Python handler later on the main interpreter loop. I think you pretty much have to do it that way.

(But Python has some logic about turning certain signals into exceptions on the main thread, which I don't want to bother with.)

I wrote on some related topics here:

[http://www.oilshell.org/blog/2016/12/05.html](http://www.oilshell.org/blog/2016/12/05.html)

<comment><USER19> TXR has deferred signals as well as async ones. Both the Rosetta examples show async signals going off: lambda being called in the signal handler context. For the SIGSEGV catch, this must be so: the lambda executes on the alternate stack arranged with sigaltstack.

There is a flag which enables or disables async signals. In the case of a CPU exception signal (SEGV, BUS, ILL, FPE, ...), we ignore this flag and just do async. Other signals respect the async flag. The async flag is almost always off; async signals are enabled in various places in the library when blocking system calls are being made. So for instance an async signal won't go off during garbage collection; the internal handler will see that async signaling is disabled, and arrange for a deferred call at the next opportunity (unless it is a CPU exception).

</comment></comment></comment></comment></comment><comment><USER23> Very cool, thanks for sharing. I'm a go person so I tried to "port" this over to go: [https://github.com/esell/eshell](https://github.com/esell/eshell)

Will be a great base to start with and should be a great learning experience trying to make it more functional.

</comment><comment><USER24> Lots of projects need their own CLI for administration purposes, here there are ready tools for command line parsing - like GNU readline (or editline/libedit if you can't work with the GPL), these are very useful as they do such things as tab completion, etc.

</comment><comment><USER25> Here is a lib I wrote for shell experiments in PHP: [https://github.com/grothkopp/PHPCliWrapper](https://github.com/grothkopp/PHPCliWrapper)

</comment><comment><USER26> That's fantastic.

</comment><comment><USER27> Nice, love it

</comment><comment><USER28> aah, implementing a shell in C.

sweet, sweet nightmares of yonder.</comment>


Modular Code and Lazy Evaluation in Haskell - <AUTHOR> https://hackhands.com/modular-code-lazy-evaluation-haskell/

<comment><USER2> The rule of thumb of how i personally handle laziness is "lazy in the large, strict in the small". That is, I compose systems in a lazy way, and my inner loops are strict!

Its not a perfect approach, but its a default approach that avoids a lot of style gotchas from being too strict or too lazy globally. I also spend way too much time thinking about how to make the perfect mutable multi dim array libraries for haskell, so I also tend to have a funny perspective :)</comment>


Fierce and furry: Protective feline demons of ancient Egypt - <AUTHOR> http://www.demonthings.com/fierce-furry-protective-feline/ <comment><USER2> Symbolic stuff is very hard to interpret from outside the context of a culture.

Take european gargoyles, with an archeo-anthropologist eye. They are all over churches built during a 500-1000 year span. The most important cathedrals tend to have more gargoyles. They're the most obviously supernatural element....

What role do gargoyles play in European religion or culture? Pretty much none. There are obviously some myths and a Disney movie, but they're just not as important as you would think.

<comment><USER3> >They're the most obviously supernatural element....

Gargoyles are really just rain spouts to protect masonry work. Obviously under different name they date back at least as far as ancient Egyptians where they were in the form of lion heads and those were copied in Ancient Greece. At some point they were converted to "grotesques" (the supernatural creatures you reference) mostly winged dragon like creatures.

The real irony is on many of the churches they lost their actual function (diverting rain water). I think that is the real cultural reference of the time, the people lost knowledge and took something that had practical function and converted it to just decorations that had no real function (except to ward off spirits...supernatural as you say).

So as you say in a vacuum it's nearly impossible to understand certain cultural representations in art, because who knows what practical function these representations may have had in another culture predating the Egyptians, the Egyptians just copied because of supernatural reasons. That said obviously domesticated cats provided real function back then and killing snakes was a big part of that, without needing to interprete their legends and artistic depictions of the same.</comment></comment>


Freeing yourself from Facebook's new web trackers - <AUTHOR> http://www.zdnet.com/freeing-yourself-from-facebooks-new-web-trackers-7000030597/

<comment><USER2> Ghostery plugin is excellent for that

<comment><USER3> I'm using Disconnect since Ghostery sells data to advertisers.

<comment><USER4> Ghostery does ask permission from the user.</comment></comment></comment>


The War on Drugs and Prison Growth: Limited importance and legislative options [pdf] - <AUTHOR> http://harvardjol.com/wp-content/uploads/2015/06/HLL104_crop1.pdf <comment><USER2> These two paragraphs lead me to believe that if the author had considered the rise of mandatory minimum sentences part of the war on drugs the headline of the paper might be different.

 _" The five means by which the War on Drugs can drive up incarceration rates (or punishment more generally) considered in Part II are (1) the direct incarceration of drug offenders, (2) the re-incarceration of all types of offenders due to drug-related parole violations, (3) the impact of drug incarcerations on prison admissions instead of prison populations, (4) the extent to which prior drug offenses trigger repeat-offender enhancement, even for non-drug crimes, and (5) the effects of large-scale drug arrests and incarcerations on neighborhood social cohesion, and the connections between social stability and incarceration. "_

 _" The connection here between the War on Drugs, longer criminal records, and increased prosecutorial aggressiveness is fairly straightforward. Increased drug enforcement results in defendants with longer felony records and prosecutors may be more aggressive against such defendants. They may be less willing to plead down felonies to misdemeanor, or to drop cases altogether; to divert to an alternative program, or to drop more serious charges. They may also be more willing to select charges that carry mandatory minimums even when there are viable alternate charges that carry no minimum. Such harshness could reflect increasingly punitive attitudes on the part of prosecutors, perhaps in response to rising crime rates from the 1960s to the 1990s, or to other political and social factors. Or it could be that prosecutors have maintained a relatively constant approach toward charging repeat offenders, but the number of arrestees with long records has grown, thanks in part to drug- related convictions. Note, too, that prosecutors need not be more aggressive just toward those with more convictions, but perhaps also toward those only with more prior arrests, even if some of those arrests never resulted in convictions."_

<comment><USER3> Mandatory minimums aren't a part of the drug war. They're a response to the huge spike in crime that started in the late 1960's: [http://www.americanthinker.com/legacy_assets/articles/assets...](http://www.americanthinker.com/legacy_assets/articles/assets/Murders%201.bmp).

Note that the Sentencing Reform Act of 1984 (which imposed mandatory sentencing guidelines), and the Comprehensive Crime Control Act of 1984 (which eliminated federal parole), came two years _before_ legislation to increase punishments for drug-related crimes.

Similarly, it wasn't the drug war that led the way when states passed reforms limiting/eliminating parole or imposing life sentences for three strikes. In fact, most states don't include drug crimes within the scope of their three strikes laws.

<comment><USER4> So what caused the bump in crime? Boredom? Perhaps fun-loving kids just looking to have a good time? Or, I dunno, maybe............ drugs?

<comment><USER5> The prison population isn't a function of a bump in crime. The prison population shot up as the crime rate fell through the 90s. The paper concludes by asking a series of questions about why prosecutors got harsher even as the crime rate dropped.

<comment><USER4> Well that's a nonsensical question to pose. First of all, harsh is subjective. Prosecutors don't dole out "harsh" sentences, they dole out longer or more restrictive sentences. To them, the sentence is not harsh because they're keeping innocent civilians safer by keeping criminals behind bars longer, or by making it easier to send potentially harmful people back to prison. Prison is no longer seen as a way to be penitent about one's actions: prison is now seen as a way to keep the bad away from the good.

The rate of crime may have dropped, but the _kind_ of crime did not, and prison gangs certainly haven't diminished in size or scope. People seem to forget that going to prison is a way of life to many people. Keeping those people 'off the streets' is basically the prosecutor's primary role.

<comment><USER5> Huh? Harsh is not only objective but trivially easy to measure. As crime rates fell, the likelihood than an _arrest_ would convert to a _felony conviction_ rose. That's the opposite of the causality you'd want or expect: as the community gets safer, the criminal justice system should become less intrusive, not more.

<comment><USER6> There's an alternate explanation for that: that the harsher sentences made that population unable to commit further crimes by leaving them in prison. This could also be correlated with the rise in prison violence.

That would tend to make sense given that prison is not a deterrent, in the sense that studies have shown doesn't seem to make people less willing to commit crimes, whereas the state of being in prison certainly does restrict their ability to do so.

<comment><USER5> How does that square with the evidence that the overwhelming majority of prison inmates serve (a) short sentences (b) only 1-2 times in their life?

<comment><USER6> That would seem to make sense if the three strikes laws are as significant a factor in causing the increase in felony convictions as I've been lead to believe.

<comment><USER5> According to this research, those laws are not a significant factor in the prison population.

(of course, that doesn't make them a good idea!)

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER7> Overzealous charging and prosecution are also a major part of the equation. They charge seriously felonies now days in what should be minor misdemeanor cases.

If you saw someone who had been charged with kidnapping you would probably think oh they must have tied someone up and taken them someplace. Maybe even taken a kid?

I know someone that was charged with kidnapping when someone tried to leave the room with his phone. He would not let them leave with his phone. Arrested for kidnapping (yes the exact same charge someone who takes a kid from a playground gets) with a massive bail. Lost job etc.

<comment><USER5> I had that reaction too. The author claims that the role of long sentences is overblown, since most offenders serve relatively short sentences (and don't subsequently reoffend). But the availability of long sentences to prosecutors might convert a lot of cases that wouldn't involve custodial time into felonies.

From the paper:

 _The primary engine of prison growth, at least since crime began its decline in the early 1990s, has been an increased willingness on the part of district attorneys to file felony charges against arrestees_

<comment><USER8> Part of the reason for _that_ is legislatures have given them more options in that department. The money crimes that were created to give cops an oblique way to go after drug dealers apply to pretty much _any_ crime, i.e. if you rob a bank it's likely they'll be able to charge with with money laundering, structuring, currency smuggling, wire fraud, or any of a dozen of those types of crime. Plus, conspiracy goes with just about everything.

Prosecutors don't have to take much to trial any more, since instead of charging you with just the crime people recognize (say, bank robbery or drug dealing), they can add up a bunch of lower level 3-5 crimes and have you looking at 300 years in jail unless you take the plea bargain. 98% of federal prosecutions never go to trial - we used to laugh at the commies for that sort of thing.

</comment></comment></comment><comment><USER3> It's useful to juxtapose Figure 1 on 174 with Figure 2 on 181.

As the article explains, the causation narrative is all wrong. The state prison population started ticking up in the 1970's, but drug prisoners started ticking up in the 1980's. From 1990 to 2010, the state prison population doubled, while the percentage of those who were drug offenders peaked in 1990 and declined thereafter.

The real explanation for skyrocketing prison populations is high crime combined with sentencing reforms (limiting judicial discretion) and parole reforms (prisoners in many states used to serve just 30-35% of their sentences before discretion was taken away from parole boards).

Interestingly, the U.S. has much higher prison populations than Europe, but would not be in the lead in terms of number of custodial sentences imposed: [http://www.nytimes.com/2008/04/23/world/americas/23iht-23pri...](http://www.nytimes.com/2008/04/23/world/americas/23iht-23prison.12253738.html?pagewanted=all) ("Still, it is the length of sentences that truly distinguishes American prison policy. Indeed, the mere number of sentences imposed here would not place the United States at the top of the incarceration lists. If lists were compiled based on annual admissions to prison per capita, several European countries would outpace the United States. But American prison stays are much longer, so the total incarceration rate is higher.")

<comment><USER5> It looks like the author does not believe sentences are directly responsible for the prison population, as most are short and most offenders are not re- incarcerated.

<comment><USER3> Interesting. Will have to read the rest of the paper. It'd be pretty bleak if you couldn't blame the government one way or the other for the level of incarceration.

<comment><USER5> It does blame the government!

</comment></comment></comment></comment><comment><USER9> Footnote 9 seems like a glaring issue with this analysis:

 _9 Technically speaking, the claim that "a large majority of those admitted to prison never serve time for a drug charge" means that for that majority of inmates the most serious charge was never a drug charge. Many inmates are convicted of multiple charges, and someone convicted of a violent or property offense along with a drug charge will be classified as a "violent" or "property" offender, not a "drug" offender._

Opponents of the War on Drugs believe that a significant portion of violent and property crimes are causally linked to strict enforcement of prohibition. Prohibition gives an implicit monopoly to criminal organizations, which causes violent crimes in the form of territorial disputes. This monopoly also increases the price of drugs for addicted users, which drives property crime, e.g. stripping copper to sell for scrap. An analysis that categorizes a heroin dealer that shoots a victim as a "Violent" offender but not a "Drug" offender seems flawed.

<comment><USER10> The criminals have lots of violent disputes, not just over territory, and the users commit more than just property crime, e.g. robbery. People up and down the supply chain try to rip each other off. It seems that the author did not expend much imagination in looking for prohibition-driven crime.

<comment><USER5> Unless you think a violent drug dealer is _less likely_ than a nonviolent offender to _ever_ have been charged with a drug offense, the data doesn't bear this out. According to the author's data, it seems that most violent offenders have never been sentenced for a drug charge.

<comment><USER9> If a violent offender that was also charged with a drug offense is classified in the "Violent" category, how can you draw any conclusions about how likely they are to have ever been charged with a drug offense? The data seem to only show that you are unlikely to end up in prison with a drug offense as your most serious crime.

This jives with David Simon's description of the prevailing attitude of enforcement in _The Corner_. The justice system is too overwhelmed to press charges for mere drug offenses, so you are less likely to end up with a conviction without a more serious crime attached.

<comment><USER5> This is discussed towards the end of Section 2, where he attempts to establish that the set of inmates charged with violent offenses is mostly disjoint with the set of inmates charged with drug offenses.

<comment><USER9> Thanks for the pointer. Are you referring to Sections II C&D? Tables 4A and 4B seem like they would have the same categorization issue I described above. As I read it, the "Never drugs" column refers to a given offender never having a drug offense as the most serious offense for a conviction. This doesn't mean that the offender was never _charged_ with a drug offense.

</comment></comment></comment></comment></comment></comment><comment><USER11> tl;dr: "Taken together, these findings suggest that the effects of the War on Drugs are often relatively slight compared to other causes, and that they are certainly not as big as many often assert. In reality, a majority of prison growth has come from locking up violent offenders, and a large majority of those admitted to prison never serve time for a drug charge, at least not as their "primary" charge. These results pose a challenge to those who wish to aggressively scale back incarceration, since the current politics of reducing sanctions for drug offenders is less complicated than that for reducing punishment for violent or property offenders. Reforming drug statutes is easier, but doing so will likely not effect significant change in the overall incarceration rate." p.6

<comment><USER11> "These results pose a challenge to those who wish to aggressively scale back incarceration, since the current politics of reducing sanctions for drug offenders is less complicated than that for reducing punishment for violent or property offenders."

I think there is an important difference between someone who is violent against other people versus someone who uses drugs (and might be violent in a sense to him/herself but no one else). So where "reducing sanctions for drug offenders" might be a novel thing to do, "reducing punishment for violent or property offenders" might not be. Apart from which one is "less complicated" or more efficient to the goal of scaling back incarceration.

<comment><USER5> You cut the author off before he could finish his point, just a few words later, that many violent offenders also don't belong in prison.

<comment><AUTHOR> Could you quote this in the text (just enough so I can \C-s to the right place)? I'm having a hard time finding this point in the rather large document.

<comment><USER5> _" Not all violent offenders are murderers and serious assaulters, and many would likely be better served by not being incarcerated, but the optics and political risks of violent-offender decarceration are much tougher to navigate than those for drug offenses."_

The Big 4 violent crimes are murder/manslaughter, rape, robbery and ag. assault. Of those 4, it's easy to get charged with the last two. For instance: I was "gently mugged" at night a few years back on my block; 3 teenagers jumped me and demanded my phone, which I gave them, and my bag, which I refused to give them, at which point they ran. They were charged with robbery (I didn't bother showing up to their trial, which I understands means they were acquitted, which seems like a decent outcome).

<comment><USER12> Just to clarify, by "jump" do you mean "surprised", "hit", or "beat the s __* out of "? I wasn't familiar with a usage of "jumped" that didn't include at least being hit (if not beat up), and Urban dictionary seems to agree with that interpretation: [http://www.urbandictionary.com/define.php?term=jumped](http://www.urbandictionary.com/define.php?term=jumped)

</comment></comment></comment></comment></comment></comment><comment><USER5> _The five means by which the War on Drugs can drive up incarceration rates (or punishment more generally) considered in Part II are (1) the direct incarceration of drug offenders, (2) the re-incarceration of all types of offenders due to drug-related parole violations, (3) the impact of drug incarcerations on prison admissions instead of prison populations, (4) the extent to which prior drug offenses trigger repeat-offender enhancement, even for non-drug crimes, and (5) the effects of large-scale drug arrests and incarcerations on neighborhood social cohesion, and the connections between social stability and incarceration._

<comment><USER13> And most notably ignores the question of whether the war on drugs drives up violent crime.

<comment><USER5> Not only does it not ignore that question, it addresses it empirically. You should read past the 14th footnote, where you say you stopped; this is addressed towards the end of section 2, about halfway through.

<comment><USER13> In which table does it empirically address this question?

</comment></comment></comment></comment><comment><USER14> Part of the problem is probably that in the USA you can be locked up for sneezing. It's common for normal people to have been jailed for something at some point in their lives, who would never have had any brush with the law in another developed country.

Another contributor is this: [https://en.wikipedia.org/wiki/Poverty_in_the_United_States](https://en.wikipedia.org/wiki/Poverty_in_the_United_States)

There are relationships between incarceration, poverty, and class and race struggles.

(How about the fact that something like 40% of the prison population consists of blacks? Looks like the effect of "War on Black" more than "War on Drugs".)

<comment><USER5> Unless your claim is that sneezing is classified as a "violent crime", the paper appears to refute this point.

<comment><USER15> > Unless your claim is that sneezing is classified as a "violent crime", the > paper appears to refute this point.

A number of people in the US have been arrested for assault for farting on a police officer. At least one was arrested for farting not on a police officer, but for fanning the fumes from his fart towards a police officer.

</comment></comment></comment><comment><USER5> Page 194/22 makes an interesting point:

 _Drug inmates' share of unique offenders is basically the same as their share of inmates: about 20%, or 26% if we count those convicted of both drug and non-drug offenses as "drug offenders," compared to 17% of the total stock_

When you look at state and local prison statistics, or at least the ones I've looked at during message board arguments, you quickly see that drug crimes do not dominate; in the midwest, the dominating crime appears to be domestic violence.

A common rebuttal to that point is that at any point in time, there may be more domestic violence inmates in prison, but the overall flow of drug offenders is greater: they serve shorter but more frequent sentences; the "installment plan", as the author puts it.

The statistics don't appear to bear that out. The tables that precede this point are interesting, too. To the extent that prison is a "revolving door" for inmates, that too does not appear to be the explaining factor for mass incarceration.

<comment><USER10> Three quarters of US murder victims are male, and more than half are black. The same goes for US murderers, to which we can add that more than half are under 30, and more than half use handguns (not an inexpensive tool). I suspect that the circumstances more often involve drug gangs than domestic violence. It seems unlikely to me that Drug Prohibition could be such a big factor in murder but not in other crime or incarceration.

<comment><USER5> I wasn't providing a narrative, I was citing statistics you can get from prisons.

</comment></comment></comment><comment><AUTHOR> Also, let me emphasize that this does NOT suggest that we shouldn't end the war on drugs. Circa 2009, there were 134k people in state prison for drug crimes. We should let them out immediately.

This will be a wonderful thing for 134k people who've harmed no one and have had their freedom stolen by an oppressive government.

But we shouldn't expect it to significantly reduce incarceration in the US. According to this article, the best way to do that would be to somehow convince Americans to beat, rape and murder people less.

<comment><USER5> That's not what the article says. It is in fact at pains to point out that violent offenders are not necessarily rapists and murderers, and concludes by discussing --- at length --- the empirically observable increase in prosecutorial aggression, which it terms "ill-advised", and the data we'd need to start ascertaining the reasons why prosecutors are charging more people with felonies even as crime drops.

</comment></comment><comment><USER16> I look at it from the following perspective. There are two intertwined economies. The formal economy and the underground economy. Because of the war on drugs, the underground economy is booming. It's counter-intuitive because you'd expect it to shrink because of the war on drugs. Except the war on drugs moved drug trade from the formal economy to the underground economy. Add to that the fact that when drugs became illegal it made them more scarce and as a result more expensive. Most (if not all) incarcerated people took part in this underground economy, willingly or unwillingly, by faith or by choice. Human trafficking, prostitution, drug abuse, and violence are all part of the underground economy and they are increasing because this economy is growing, enabled by the war on drugs.

</comment><comment><USER17> Where is the 17% number coming from? A quick search turns up the official stats from the Federal Bureau of Prisons showing 48% of Federal incarcerations are due to drugs[1]. And using slightly older data, but still more recent than the author's 2010 numbers, it looks like over 20% of incarcerations at a combined State and Federal level are for drugs[2].

Also, the paper focuses on policies and enforcement, but seems to mostly ignore the social effects of broken drug policies in creating criminals. How many people are robbing or burgling (accounting for over 20% of incarcerations again) in order to support drug habits? How much gang-related violent crime leading to incarceration is funded by drugs?

You can't just cherry pick the easy numbers and say "look, drug policies aren't that bad for incarceration!"

[1] [http://www.bop.gov/about/statistics/statistics_inmate_offens...](http://www.bop.gov/about/statistics/statistics_inmate_offenses.jsp) [2] [http://felonvoting.procon.org/view.resource.php?resourceID=0...](http://felonvoting.procon.org/view.resource.php?resourceID=004339)

<comment><AUTHOR> If you read the words surrounding the number, you might observe that "drug offenders comprise only 17% of _state prison populations_ ". Thus, observing that 48% of _incarcerations_ at the _federal level_ are due to drugs is, well, a different matter entirely.

Also, an important distinction between _incarcerations_ and _prisoners_ is explained on the fourth page of the article. Tl;dr; consider a 2 man prison, with 1 murderer in jail for 20 years and a rotating cast of heroin users serving 1 year sentences. Drug users represent 95% of _incarcerations_ and 50% of _prisoners_.

<comment><USER17> Agreed, but data point two in my first statement is for prisoners as well, not incarcerations. Ignoring Federal prisoners entirely is disingenuous since the total percentage across all prisons for drug offenders is above 20%. That's a lot more than 17%, and it's rising.

</comment></comment></comment><comment><USER18> tl;dr: verbatim from the linked pdf

 _Decades of stable incarceration ended suddenly in the mid-1970s, as the U.S. prison population soared from about 300,000 to 1.6 million inmates, and the incarceration rate from 100 per 100,000 to over 500 per 100,000. (...) [T]he United States is now the world's largest jailer, both in absolute numbers and in rate_

 _Table 1B: various offenses' contribution to state prison growth, 1980–1990, 1990–2009_

                  1980      1990      2009        % 1980-1990   %1990-2009
        Total     294,000   681,400   1,362,000
        Violent   173,300   316,600   724,300     36%           60%
        Property  89,300    173,700   261,200     22%           13%
        Drug      19,000    148,600   242,200     33%           14%
        Other     12,400    45,500    134,500      9%           13%

_Drug offenses are still not the dominant contributor to prison growth, even during the first stage of rising incarceration, but their role in the 1980s is on a par with the locking up of violent offenders.

When the crime drop begins, however — which is when one might expect drug offenses to become more important, since they are more discretionary — the importance of drug offenses declines precipitously, and the incarceration of violent offenders dominates.

In other words, whatever the historical importance of drug offenses to prison growth, the incarceration of drug offenders is not a central causal factor today._

And in the conclusion, also verbatim:

 _(...) [i]f legislators decide that reducing drug enforcement is still a net social good, regardless of its impact on prison populations, the tools at their disposal are limited.

Criminal justice enforcement in the United States is highly disaggregated across a wide range of institutions operating relatively independently of each other. At least right now, prison growth is driven by prosecutorial aggressiveness, and legislatures have little control over locally elected, locally funded prosecutorial offices.

Legislative success may require unconventional yet viable approaches, such as adopting charging or pleading guidelines or making efforts to push the cost of felony incarceration onto county budgets_

<comment><USER11> Don't like the 10 vs. 20 year jump there, it would have been nice if they had added statistics for 2000 as well.

<comment><USER19> The other two columns are percentages, so the interval size doesn't matter. In addition, 1990 is an important point because of the notable shift in overall crime rates that occurred there (crime went up until 1991, then declined dramatically and continued falling as the prison population increased).

</comment></comment></comment><comment><USER20> A strange paper, difficult to follow, not well organized.

Perhaps not surprisingly, the author is an economist.

I do find it extremely odd when a scholar uses an informal term like "Drug inmate" (for example, p. 180)

<comment><USER5> No, the author is criminal justice professor famous in academia for empirical study of mass incarceration.

<comment><USER20> Not sure what "No" refers to.

I found this listed under Education at [http://www.fordham.edu/info/23171/john_pfaff](http://www.fordham.edu/info/23171/john_pfaff) : University of Chicago: BA, 1997; JD, 2003; PhD (Economics), 2005

</comment></comment></comment><comment><USER5> The HN reaction to this paper is disappointing, most notably because so few of the people with strident opinions (against _or_ for its conclusions) appear to have read it. I feel qualified to judge the prevailing sentiment on HN about criminal justice, and based on that: I feel like HN should be all over this paper; it should confirm all their biases except for one.

Here are some Cliff's Notes:

First, the author: John Pfaff is an internationally recognized legal scholar specializing in the dynamics of mass incarceration. This paper isn't a drive- by by a random economist; he's cited widely on the subject, including in lots of very readable blogs. I lost an hour today reading things about his work. It's interesting.

Next, on the drug war: the author appears to be _against the drug war_ and _against mass incarceration_. Some commenters are responding to this paper as if it was an appeal to continue the drug war. That's the wrong way to read it. The author sets out a simple problem statement: incarceration is increasing even as the crime rate falls. That can't be right! The point of the paper is simple: "end the drug war" is a bromide for mass incarceration. If we're really to make a dent in our prison population, more radical interventions are needed.

Third, and the reason I'm so irritated at HN's response: Pfaff agrees with HN about criminal justice in the US! The problem, as he sees it, is _prosecutorial discretion_. As crime has fallen, the likelihood than a US arrest will convert to a felony charge has drastically increased. That's so counterintuitive that the author is moved to hypothesize about why it's happening. Perhaps crime is falling, but neighborhoods are continuing to decline, and frustrated prosecutors are overcharging accused people as a sort of "quality of life" tactic. Or perhaps the incentives are all screwy: maybe we have too many prosecutors with too few budget constraints, and, as crime falls, they have to scrape the bottom of the criminal barrel to justify their paychecks.

This paper is fascinating, carefully laid out, and full of counterintuitive observations. Most of the places you'd look for simple answers to US prison population aren't actually in the evidence. You think it's revolving-door sentences, or "three strikes laws"? Nope! Most offenders aren't reincarcerated at all, and of those that are, there's a sharp drop-off after 2 prison stays. Think it's long sentences? Nope! Most offenders serve short sentences (the author notes the impact that nominal sentences have on prosecutor incentives, though). Think it's parole and drug-related parole violations? Nope, that's not in the evidence either; in fact, parole strictness doesn't seem to be a major driver at all.

Again: the point of this paper is not that prison is full of violent offenders and so mass incarceration isn't a problem. Even among the cohort of violent offenders, the author is concerned about prison population; as he points out, many _violent_ offenders don't belong in prison either. The author is discomfited by the fact that prison population and the budget drain of our criminal justice process are increasing even as crime falls. Mass incarceration is a problem. If you're serious about wanting to end it, this paper is good news: it strongly suggests that letting college kids buy weed at Walgreens isn't nearly enough to allow us to write that problem off.

<comment><USER21> There's been a big move toward overcharging to force a plea rather than a trial.

<comment><USER5> Since most criminal defendants are in fact guilty, perhaps the logic here could be something like: "prosecutors have made the system of adjudicating crimes so efficient using plea bargaining rules that sentences need to be ratcheted down sharply, both to account for the increased likelihood that the accused will go to prison and as a check on prosecutorial power".

<comment><USER22> There is also a question of whether the crimes defendants are in fact guilty of should even be on the books.

There has been a trend in legislation. X is bad but is already illegal, and some people who do X use Y, so Y is made illegal too. Some gangsters use large sums of cash so let's ban large sums of cash. Some terrorists use strong encryption so let's ban strong encryption. Some spammers use whois privacy so let's ban whois privacy. Some drug dealers use scales so let's ban scales. Some bootleggers break DRM so let's ban breaking DRM.

This happens many times over until most of the people who are not gangsters or terrorists or spammers or drug dealers or bootleggers can nonetheless be charged because they harmlessly use cash or strong encryption or privacy services or digital scales or DeCSS. And charged with "money laundering" or "arms dealing" or other incredibly serious crimes.

Which may at least partially explain why so many people are locked up for "violent crime" \-- it isn't that so many people (outside of gangs) are spilling blood, it's that we've allowed "violent crime" to encompass things like weapons possession that don't inherently involve violence or harm to anyone.

<comment><USER5> I do not believe unlawful possession of a firearm is classified as a "violent crime"; for instance:

 _" Crime of violence" does not include the offense of unlawful possession of a firearm by a felon_ (US code).

Possessing a firearm while committing a robbery is a violent crime, whether you use it or not. But the robbery itself is also classified as a violent crime, with or without the gun.

<comment><USER22> > _" Crime of violence" does not include the offense of unlawful possession of > a firearm by a felon (US code)._

That's at the federal level, which constitutes a smaller percentage of total inmates, more of which are in for non-violent drug offenses, than the states. And for example:

"Although the statutes allow for varying degrees and punishments, the crime [Criminal Possession of a Weapon] is generally considered a violent offense in New York with mandatory minimum terms of incarceration."

[http://www.new-york-lawyers.org/weapon-crimes.html](http://www.new-york- lawyers.org/weapon-crimes.html)

But we don't need to get into the whole gun control debate, it's only one example. The problem of criminal laws expanding to encompass more than what a normal person would consider to be criminal behavior is _prolific_ and by no means limited to nonviolent offenses. The instance elsewhere in the comments of someone being arrested for "kidnapping" for refusing to allow someone else to abscond with his phone, for example.

</comment></comment></comment></comment></comment></comment><comment><USER13> This seems to ignore the question of whether the 'war on drugs' contributes to violent crime, and what fraction of criminals classified as violent should therefore be included in the 'war on drugs'.

<comment><USER14> It's not ignored; the paper discusses the "collateral effects" of the WOD, in contrast with the strict definition of a "drug offender". P 220:

 _In other words, defining "drug offenders" solely as those convicted of drug crimes is not as objectively correct as it might initially seem. Although certainly a valid and useful definition, there are other ways to think about what counts as the product of drug enforcement that may alter estimates of the impact of the War on Drugs. Furthermore, this section should establish that scaling back the War on Drugs can have collateral effects — such as changing how society manages addiction or how enforcement interacts with drug-market violence — that are not immediately apparent when simply look- ing at arrest or incarceration rates._

<comment><USER13> Thanks for pulling out that quote. To me, this section invalidates the whole paper: I think this paper would have been much more valuable had this been one of the 5 questions/mechanisms examined.

I suspect though that had it been one of the questions it _would_ have shown a bigger link between the war on drugs and incarceration, and thus not been sufficiently interesting to be worthy of publication. Academic click-bait!

<comment><USER5> Wait, did you _read_ the paper, or did you read the abstract, wait for someone to quote something from the paper, and then use that quote as an argument against the whole paper?

If you have read the paper, can you give a serious coherent series of arguments about how its analysis is broken? Every third page is a simple presentation of statistics and then an analysis of those stats. Which of them are wrong?

I'm sorry to ask the question, since "did you actually read" is borderline uncivil, but when someone quotes something and your reaction is "aha!", the clear suggestion is that you yourself hadn't read far enough to see that quote yourself.

It's fine not to read the paper, but it's less fine to suggest that a tiny quote could invalidate an entire paper that you haven't read.

<comment><USER13> Of course I didn't read the whole paper! I stopped reading the paper when I recognized the fundamental flaws:

1) the hypotheses examined were very narrowly framed

2) Footnotes 9 and 13 admit that drug-related offenders may in fact be present in the statistics as 'violent crime', yet the paper does not consider this further.

After that I skim-read to ensure there flaws weren't addressed. If I missed the place where these flaws are addressed, I would appreciate your pointing it out.

I am glad you read the whole paper, but I would caution you against believing you got any closer to the whole truth by doing so.

<comment><USER5> For those playing along at home: this is a paper with 124 footnotes.

<comment><USER13> If I missed the place where these flaws are addressed, I would appreciate your pointing it out.

</comment></comment></comment></comment></comment></comment></comment><comment><USER23> Sure it is.

Let's see a measurement of the total eco-system (cultural, economic, etc) destruction that the war on drugs has caused to, eg the black community since 1965.

Let's see a proper measurement of all the fallout and spin-off effects, ranging from from repeat offenders, to fatherless households, to education effects, to tens of thousands that have been killed as a direct consequence.

Let's measure the land value collapse that went with the war on drugs, which practically robbed an entire race of people of a prime source of savings and wealth. To have those potential assets be replaced by completely run down housing.

Then let's see how the failed policies on drugs, having decimated the black community in just 20 years, then led to fears about skyrocketing crime rates and violence that directly came out of that 'drug war' \- including the implementation of mandatory minimum sentencing, which was a reaction by the white community, by politicians looking to placate their voters, to the crime that was caused by the war on drugs.

This article tries to pretend the war on drugs occurred in a semi-vacuum, when its effects were extraordinarily wide spread, touching every possible aspect of society. It is precisely the _root_ cause of the extreme rise in the prison population, all the other causes extended from it.

<comment><USER24> Searched the article for the word "gang". No results.

There's quite a lot of violent crime that's _directly_ related to and driven by the illegal drug trade.

<comment><USER23> Makes sense. To stick with the false premise core to the article, one would have to avoid much discussion of the organized crime aspects of the truly massive black market that the war on drugs created. How that organized crime affects communities through non-stop fear and intimidation, how it sucks in desperate people, how it perpetuates an endless cycle of crime and poverty that is centered around the profit system of the illegal drug trade.

Of course this is already a well understood effect. Which makes the article's intentional evasion of it that much more blatant. Whether we were to discuss the effects of organized crime around alcohol prohibition, or the Mexican cartels. One need look no further than what the war on drugs has done to hundreds of Mexican towns and cities, to understand what it has done to the black communities in the US in the last ~50 years.

</comment></comment></comment><comment><USER14> > _Only 17% of all prisoners are serving time for drug offenses._

Yeah, but 17% is still a large percentage. A 17% reduction would make a difference.

<comment><USER5> This is the wrong way to read the paper. The author isn't making a case against decriminalizing drugs. He's saying that even if you decriminalize drugs, you will still need other stronger interventions to work against mass incarceration and, for that matter, the racial disparities in incarceration.

<comment><USER14> The specific argument is made that if you decriminalize marijuana, it won't make a difference because there aren't that many incarcerations for marijuana specifically.

<comment><USER5> Which is an argument so obviously true as to be banal; even the marijuana decriminalization campaign in Colorado pointed it out.

Marijuana is going to be legal in the US. It's not a live issue. We don't have to import it into every other social science inquiry.

</comment></comment></comment></comment><comment><USER25> Many crimes are driven by poverty. Looking at only drug-related charges is short-sighted. How many crimes are committed by young men whose fathers were ever incarcerated for a drug-related charge?

</comment><comment><USER26> There seem to be many flaws, but rather than go through them now, I'll assume that only 20% of incarceration is drug-war based.

 _Great_ , that means if we end the drug war, we'd get _20%_ reduction in our world-record incarcerated population. A _half-million more people_ (roughly) will be freed to get treatment and attempt to contribute to the economy and society.

That sounds like a great start to me.

</comment><comment><USER27> As an aside, the author of the study clerked for a Reagan appointee (who was also a Hoover Institute fellow), is from an especially conservative law school, and seems to have published so far almost exclusively on this particular topic.

I would personally want to do more research on this person and his writings before taking his entire argument at face value.

<comment><AUTHOR> His arguments are laid out in detail. His data is available for inspection. Could you explain the relevance of his conservative bonafides? If he were a more typical liberal professor, would that also be relevant?

<comment><USER27> What I see the author doing as I read his work is finding a lot of ways to use different definitions of terms to determine which offenders count as "drug offenders". His assumptions are notably different from the assumptions of some other researchers. It seems possible to me, at least on first skimming of these articles, that a lot of the differences in the data that Pfaff uses can be accounted for by these alternative definitions of terms.

For instance, I grant that a "violent offender" who is in jail on a principal charge of shooting a police officer while in the process of being arrested on a drug charge or during a drug raid was indeed violent, and is rightly viewed differently than a "non-violent drug offender".

But we still have to acknowledge that if there were no War On Drugs, then there wouldn't be as many of those drug raids and drug arrests to begin with. The circumstances for many of those violent crimes simply would not occur.

So, the process by which you sort the offenders into batches seems to have a strong impact on the conclusions you make.

</comment></comment></comment><comment><USER28> If you don't have time to read the whole thing. There is a summarization on pages 29 & 30.

This is an interesting read on the whole situation. Curious to see where it leads. And hopeful that more research will make it into decisions. Seems too much of that space (legislation) is fueled by dogma and priviledged ignorance.

<comment><USER14> > _There is a summarization on pages 29 & 30._

... corresponding to logical pages 201 and 202.

</comment></comment><comment><USER29> Little known fact; the prisons are primarily filled with dangerous criminals.

<comment><USER30> and how many of them were dangerous criminals BEFORE we locked them up?

<comment><USER29> According to this study, most of them.

<comment><USER31> It does make you wonder why the US has so many more "violent criminals" compared to other similar Western countries though...

<comment><USER29> If you look at the violent crime offenders in the USA, more than 3/4's of them are locked up for "Aggravated Assault".

According to Wikipedia, the definition of Aggravated Assault in the US is:

-an attempt to cause or purposely, knowingly, or recklessly causing bodily injury to another negligently causing bodily injury to another with a dangerous weapon.[20]

-causing bodily harm by reckless operation of a motor vehicle (vehicular assault).[21]

-threatening another in a menacing manner.[22]

-knowingly causing physical contact with another person knowing the other person will regard the contact as offensive or provocative[23]

-causing stupor, unconsciousness or physical injury by intentionally administering a drug or controlled substance without consent[24]

-purposely or knowingly causing reasonable apprehension of bodily injury in another[25]

-any act which is intended to place another in fear of immediate physical contact which will be painful, injurious, insulting, or offensive, coupled with the apparent ability to execute the act.[26]

If you want to investigate the difference, I suspect you should start there, since that seems to make up the bulk of our incarcerated.

</comment></comment></comment></comment></comment><comment><USER10> Completely missing from the article is an estimate of the coupling between drug and non-drug crime (e.g. prostitution or robbery to finance a drug habit, violent conflict vertically and horizontally along the supply chain, tax avoidance, and corruption of police and other government employees). There are also tertiary effects, like drug gangs engaging in additional illegal activities supported by the core competencies of trafficking and violence. This merits an additional study.

<comment><USER5> Not only is that not completely missing from the paper, it's addressed empirically. The author is particularly concerned with the affect of prior drug _arrests_ on felony charging, with previous drug convictions as a predicate for three-strikes laws, and with the relationship between violent offenders and drug charges --- they turn out to be mostly disjoint cohorts.

The author agrees with you that further study is merited.

<comment><USER10> That's a nice start, but many members of a drug gang might not ever touch drugs, at least if I can believe The Wire's portrayal of drug organizations and their division of labor. I am also skeptical of the correlation between arrests for drug offenses (where there is no victim to complain) and violent crimes.

<comment><USER5> The author would be the first to agree that there are questions raised by drug prohibition that we don't have enough data to answer. You should give the paper a close read. It's hard to come away from it thinking that he's blasé about prohibition.

</comment></comment></comment></comment><comment><USER14> Author should define an abbreviation for "War on Drugs" early in the paper instead of repeating the full phrase.

</comment><comment><USER32> Either individuals own themselves or they don't. And if we don't, who does? Where do they get that authority?

</comment><comment><USER33> I didn't think that the direct effects of policy of the drug war were considered the primary driver of prison population growth. The abstract (because I don't have time to read the whole thing) doesn't mention how much more _money_ police departments have received, and how their funding rates are directly tied to how much 'crime' they 'fight'. The author lists what he sees as the ways drug war policy could impact prison population growth but the list is meager and does not integrate drug war policy into the broader social and political context.

The drug war was just a subset of a systematic antagonism by the state against certain population segments. The same communities that were ravaged by crime and police alike were pushed into calamity by social policies that lifted up _ahem_ privileged segments of the population while leaving others in the dust with inadequate social and economic resources.

The only connection between conviction rates and the drug war the author considers are drug related convictions. That is, frankly, stupid. It is the increased presence of police brought about by the drug war by increased funding, bloating of federal agencies, and escalated enforcement practices that links the policy to the effect. If a cop is in the neighborhood because of the drug war and decides to rough someone up and arrest them for 'quality of life' offenses or 'resisting arrest', that is an effect of policy, of which the drug war is a major component.

Normally I wouldn't feel so skeptical of an expert's findings on a highly complicated subject, but there's also the fact that this publication claims it is nonintuitive and runs counter to the conventional thoughts of his peers. I think the lines of reasoning I tried to point out are well demonstrated in the first source he cites, Michelle Alexander's fantastic book. I think the author can only achieve the results he can by drawing a small box around some policies and calling that the drug war when what other people call the drug war is far larger than that.

P.S. Another comment mentioned gang violence, which is strongly linked to inadequate social and economic resources _and_ a large illegal drug trade.

P.P.S. When the police and prosecution get to decide who is a violent offender, you can't use measurements of who is a violent offender in order to establish whether or not police policy is effective and just. Furthermore, why do people become violent offenders? The higher order effects of a policy that treats communities as disposable, unwanted, and antagonistic certainly feed into this.

</comment><comment><USER34> Can someone write a tl;dr? Many a day on HN has been spent ranting about the war on drugs and the prison population.

e.g. [https://news.ycombinator.com/item?id=9490310](https://news.ycombinator.com/item?id=9490310)

<comment><USER29> TLDR: Only 17% of the prison population is made up of drug offenders, making up only 20% of the 5-fold increase in the prison population in the last half century, and the types of legislation available to us would likely not lower that population measurably.

<comment><USER23> And how many of the other 80% were a result of the collapsed economic and cultural sphere that the war on drugs caused for millions of black families? (emphasis there, because such a large percentage of that huge increase in prison population has been black)

None of that happens in a vacuum. If you lock up a parent for drugs, and his son is raised in a broken household, in a broken neighborhood overflowing with the effects of failed war on drugs policies, what's the likely outcome to that situation? You don't have to take my word for it, read any interview with any black person that has ever made it out of that situation. Read about what their lives were like, what the culture was like, what their households were like.

That's why this article's premise is so obviously far off. The war on drugs touches everything.

<comment><USER5> I don't know which article you're arguing against, but it isn't this one. This isn't a case against decriminalization. It's a case that decriminalization of drugs is _not enough_ to end mass incarceration.

If you believe what you appear to believe about the injustice of mass incarceration and its racial disparities, as I do, you should be celebrating this article, not excoriating it.

</comment></comment></comment></comment><comment><USER35> Just skimmed the PDF. I believe the war on drugs is responsible for the majority of the offenses reported in the Violent and Property categories. And let's finally face it: had psychedelic therapy options been available, literally no one would get stuck in hard addiction drugs including alcohol. And had cannabis been legal and used to treat depression, large part of the US population would not be psychotic and intermittently suicidal zombies. It also reminds me of the JRE podcast with an ex-cop where he says that 90% of arrests are due to the war on drugs: [https://youtu.be/lHb23-puvLI?t=739](https://youtu.be/lHb23-puvLI?t=739)</comment>


TinyChat – Lightweight Chat application - <AUTHOR> http://tiny.chat <comment><USER2> How? Especially the less battery part. How? Will i miss out on live notifications?

<comment><AUTHOR> You get an option to use SMS for notifications

<comment><USER2> Does that seriously safe battery? I would have thought push notifications are the most optimized thing ever.

<comment><AUTHOR> SMS is one of the options(If you are in power savings mode or switched off your mobile data because if low battery)

Less battery is used because there is no active service in background that keeps connected to server.Like WhatsApp or FB does.

Hence using less data and less battery</comment></comment></comment></comment>