Ask HN: I have to analyze 100M lines of Java – where do I start? - <AUTHOR> As part of a huge "let's see what's going on here and re-build this from scratch" they dumped the whole code repository on me and my team.
We've started parsing it and tried to work on extracting abstract syntax trees and all that.
Any idea would help us a great deal.
Thanks. <comment><USER3> For rewrite from scratch projects, I always start by identifying the use cases covered by the application. You don't need the code for that. Just run the application and identify what it is that it does. Then, work backwards. For each use case, use the existing code as specification of the use case behavior.

At 100 million lines, I'd suspect this is either an extremely large project, where a rewrite from scratch is inadvisable, or that there is a code generator at work. If it is the latter, you want to analyze the code generating source, not the end result.

Anyhow, generically, for a first contact with a new code base, code coverage tools are a good start, as is a call graph debug run of the project. It'll let you spot dead code as well as hot code (code being called at every run of the application). It'll highlight the important and non-important code parts, allowing you to read less code and get a grasp on the architecture.

<comment><USER4> Yup, start the profiler up, use the application, check the call graph.

At 100 MLOC the code base is probably a complete mess.

Also, simian is probably your friend as it will identify large chunks of duplicate code.

As well source control can be your friend, the older the source is the more likely it is to contain useful code. The files with the most changes will usually be where the bugs are.

<comment><USER5> Why did this get downvoted? It's actionable and on topic.

</comment></comment></comment><comment><USER6> 10 years ago I worked on a large project to re-write a code base written in C. Our approach was to forget about the code and document everything it did from the user's perspective. Once everything was mapped out we decided on what we were going to keep, modify or remove, and then started building everything from scratch. You can always go back to the original code to see how a particular feature was implemented and perhaps re-use the same logic.

<comment><USER7> Having done projects like this before as well, this is the best method. Knowing WHAT needs to be done today and tomorrow is far more important that knowing HOW it was done before. The how is only important once you know what you need to do.

<comment><USER8> lol, so true...

Funny thing is, often the users tell you "I used it for XYZ" and it was...

...never been written for XYZ

...never DID XYZ, all the results/numbers were trash, but no one noticed

</comment></comment></comment><comment><USER9> With a codebase like that, it's better to look at it through the users' eyes, rather than trying to reverse engineer the business from the code. Things that look like bugs in the code may actually be features for the users, or may have been absorbed so long ago that they've fundamentally changed the nature of the business.

You don't need to understand the whole codebase. It will take years. Best to focus on what the users need and analyze small chunks. If it's truly 100M lines, there's not going to be any semblance of consistency in the code.

You can also slap New Relic on it and you may be amazed at what you learn, right away.

Don't waste too much time trying to understand all the code. Focus on a couple issues first, make some hypothesis, and then see how well your understanding of the code fits the bigger picture. Refactor and repeat.

<comment><AUTHOR> Any human line-by-line/application-by-application analysis is (for this particular discussion) out of the scope.

The size of the thing and the way we thought we were going to work is quite different.

For instance, suppose we produce AST for all the routines/pieces of logic/you_name_it we wanted to then find similar patterns or clusters that would give us hint to then work on a "pareto-like" way.

As already stated it's not ONE project, it's an old (but still running), poorly-documented codebase produce in decades around this big firm we work for.

<comment><USER10> Don't try to figure out how the code does what does yet. Figure out what systems exists inside it:

      1.  What kind of modules?
      2.  Which servers/hardware?
      3.  Which databases/datastores?
      4.  What systems talk to what?
      5.  What test systems exist or existed?
      6.  Which api/frameworks where used?
      7.  Who is currently working on them/maintaining it?
      8.  Is anyone left who used to?
      9.  Why is a rewrite on the table?
      10. Is there any way you can work on smaller pieces at a time?
      11. What are the pain points of the current users (will tell you what area to focus on)?
      12. Can you document what comes in and out?

In my experience with such large code bases, there is never one way to do things. i.e. I once worked on a smaller system with 4 ways to talk to the same database. On one with 100 million lines I would expect even more ways to rome ;)

If you do want to go down the static analysis path, start with existing tools before trying to build your own. If needed get external help for this.

A 100 Million lines of code is not so bizarre. The project I work on is currently about 300,000 lines and a project some 300 times larger is quite imaginable for me.

<comment><USER5> Code complexity does not increase linearly.

100 M lines is stupendous.

<comment><USER10> Not really, my project is 3 FTE for 8 years. Double it to 16 years and then multiple the number of developers by 100.

Consider a large enterprise having 300 developers in multiple teams, I am not at all surprised that they can manage to write 100 million lines or so. Also I think this is really a system of systems, and in my experience probably has large parts developed by the lowest bidding firms. Which when software is developed for 10 years or more means more than one way to do the same thing.

Also having to deal with lots off ancient systems and working around weird bugs probably fixed years ago. You know things like bugs in Java 1.2 on HP-UX and stuff like that, or errors in Oracle 7i etc...

Plus functional duplication because team A did not know subteam C2 build the same thing...

Editing my comment instead of replying to the excellent comment by @jacquesm as hn does not allow me to reply to the reply.

Actually completing the transfer of a codebase like that is unlikely to a new team without much much more of a handover. But some high/middle frustrated with the current system manager asking a team to start rebuilding before it gets shut down a few months/years later is very possible. An other plausible option is a corporate take over... But then I would expect a very experienced team to work on it who do not need to ask HN for this kind of thing.

I personally have been in a situation where code moved between companies and no documentation or old developers where available. Not as large as this only a 1 or 2 million lines of code/xml. But I am no longer surprised by the stupid acts that large corporations can perform.

And this must be systems not just one. You can't build a single jar file of 50+ millions of lines of code than could have loaded in a JVM around 1.3.1 on even high end hardware for the time.

<comment><USER5> How realistic does it sound to you that a codebase that size would be transferred to a new team without any of the old team, outside of a hack of a bank or a reversal of some outsourcing decision or something like that?

Typically the value of such a codebase is determined by the quality of the team maintaining it and the degree to which it is documented.

Complexity of software constructs is not linear and enough books have been written about simply multiplying and dividing manyears and lines of code that I don't think we need to hash that out all over again. See 'the mythical man- month' and many similar books and articles.

</comment></comment></comment></comment></comment></comment><comment><USER11> Source to UML: [http://www.architexa.com/](http://www.architexa.com/)

Getting call paths: [https://github.com/gousiosg/java- callgraph](https://github.com/gousiosg/java-callgraph)

Line coverage from instrumented jars: [http://emma.sourceforge.net/](http://emma.sourceforge.net/)

For this type of request, I'd push back and say, let's identify very small parts of this and begin rewriting those one at a time in an isolated project. Kind of an agile rewrite that will combine the legacy project with the slowly rewritten one. Use the tools to identify parts of the project than can be isolated. Build new interfaces or services to let the old project communicate with the new one. Get a history of the source repository to see where recent edits are and prioritize those to be rewritten first (presuming they want a rewrite to lower maintenance costs).

<comment><USER12> nice links! :)

</comment></comment><comment><USER13> You haven't really described your goals: What do you want to extract from your analysis? Metrics to tell you what's "wrong" with the existing code base? Some sort of model of the system's semantics?

<comment><AUTHOR> We'd love to know what these lines do. For example what part of this codebase deals with the DB and what part does not. And then go deeper.

The final goal is to re-do what these lines do :(

<comment> grey-area 100, 000, 000 lines of code is a huge amount and would take you over 1000 days just to read at 1 line per second, and 1000 man-years to fully understand. If your final goal is to rewrite all of it you are probably doomed to fail. You should first ask yourself (and your clients) some simple questions about why this insane project has been dumped on you and what the goal is:

What is the order of priority of services - which services/apps are critical, and which are not very important?

Which services actually need to be rewritten and which are working just fine?

Which services have a clearly defined interface and can be rewritten?

Which tests are in place to test the existing services, and which will you have to write?

I wouldn't touch the code till you have answered those questions, and once you have those answers, having some sort of overview of code coverage etc is going to seem less important, because it will become obvious which bits need to be touched first (the ones that are both mission critical and broken), and which bits you can easily isolate.

You will find it very very hard to show concrete progress if you try to change all of this code at once, in a global way (for example by tidying up every single reference to a db to use a new db interface, or things like that). If you do, you'll never reach your final goal, and end up spending months tidying up without actually delivering value to the business.

<comment><USER14> Yes, a "rewrite" is the wrong way to think about this project. A typical programmer might be able to produce 10K lines of production code in a year, which means it would take 10,000 engineer-years to rewrite it all.

The OP needs to think in terms of stewardship, not complete reconstruction, and make improvements by small steps as a gradual process.

I talked about a similar problem here: [http://short- sharp.blogspot.ca/2012/08/fixing-broken-codebas...](http://short- sharp.blogspot.ca/2012/08/fixing-broken-codebase-part-i.html)

</comment></comment></comment></comment><comment><USER15> What type of application has 100 million loc's? Windows 7 has 40 million lines of codes so I'm wondering why type of application/software it is.

<comment><USER16> I have a quite strong feeling that certain complex automated systems within financial services / insurances domain could reach those LOC levels. Including all the frontend side, internal backend logic, possible web services, internal tools, tests, tens to hundreds of interfaces to different kinds of external services, report generation, libraries, etc.

<comment><AUTHOR> BINGO!

<comment><USER5> BINGO all you want but if you're at liberty to disclose such things (or to confirm them) you should have included it in your original write up.

<comment><USER17> Sure, 'cause if he/she had, you would already have found a solution! No need to be rude nor arrogant, even more if pointlessly so.

<comment><USER5> No, it's just that such information makes a huge difference to those that try to make sense of the question, especially a poorly defined and somewhat confusing question as posed here.

<comment><USER18> How exactly does knowing it's an insurance/finance application _magically_ change the answer to "how to get my head around 100m lines of Java code"? Is there some "financial java code analysis" tool or technique that's so completely distinct from "engineering java code analysis" or "healthcare java code analysis" that it deserves a snotty, condescending retort?

<comment><USER5> An 'Ask HN' with this level of input is not a guessing game where the people that try to help have to play 'bingo' with the asker for information that may or may not be important for the answer. If there is context available and the asker is free to talk about that context then it should be supplied up-front.

There is nothing snotty or condescending about that, but there is something very weird about this whole thread, I wished I could put my finger on it.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER19> Before rebuilding anything piece of software from scratch, I would give a serious look at this amazing bunch of wisdom: [http://www.joelonsoftware.com/articles/fog0000000069.html](http://www.joelonsoftware.com/articles/fog0000000069.html)

<comment><USER20> Great article, it hits at the core point. There's an old saying, don't throw the baby out with the bathwater.

</comment></comment><comment><USER21> 1) Focus on the functional use-cases and not code.

2) Identify integration points to other systems and ask why they are there

3) Realize that a "big-bang" rebuild never works and that it's better to break up the system into smaller pieces and replace them piece by piece.

</comment><comment><USER22> Funny, I have 50,000 individual Java apps to analyze. I started with a copy/paste detector. Pmd has a free one. Good luck!

<comment><USER23> Wow, a blast from my distant past... a link to the copy/paste detector:

[http://pmd.sourceforge.net/pmd-5.1.3/cpd- usage.html](http://pmd.sourceforge.net/pmd-5.1.3/cpd-usage.html)

</comment></comment><comment><USER24> As a first pass, try deleting as much code as possible :) If there are files or whole projects that aren't needed anymore, they're just slowing down your analysis. Also some dead-code analysis could be helpful, at least in broad strokes. You could instrument the code with a test coverage tool, then run the code instead of the tests to see what code gets reached.

Edit: You could also look for duplicated code, and quickly refactor that to just be in one place.

</comment><comment><USER25> Do you and your team have experience of Java development?

Your question sounds like something that someone with either no real experience and/or no experience of an object-oriented language would ask.

100 million lines is a lot of code. Why do you need to "parse it to extra the AST"? That's crazy.

Do you have the original design documents and architectural documentation? If you do, read it.

<comment><USER26> Downvoted.

Even if the design docs would exist, it would take months to read them, without any guarantee that they correspond to the reality.

Meanwhile, automated analysis of actual code can give you at least high-level overview of the codebase and maybe a hint where to start digging. Getting AST is a first step required for most automated tools to do their work.

EDIT: I acted too rashly and downvoted your post before I realized what are we really talking about. Sorry about this. I still am convinced that automatic, static analysis of the code is the way to go, but you obviously don't deserve a downvote for having different opinion. I'll try to make it up to you by being more careful in the future :)

<comment><USER25> Hahaha, no problem.

Static Analysis would be my second step, but first I'd have a look at the architectural documentation. I can't imagine that a project of this size wouldn't at least have a Powerpoint explaining the structure and concepts of the code.

Then it's time to start using tools.

</comment></comment></comment><comment><USER27> Here's a suggestion I haven't seen: Unless you have full management support, a skilled team, valid business reasons for this conversion, and expectations of succeeding, consider moving to another company/job.

You've been given the task of digital archeology/septic cleanup. Unless you like the tedium and stank, it's not going to bode well...

</comment><comment><USER28> Understanding the "shape" of a codebase is something I've always been interested in and I started building a tool to help me understand and traverse code here:

[http://sherlockcode.com/](http://sherlockcode.com/)

However I don't think it would scale to 100M lines of code. I have run Linux through it and it was acceptable (both in run times and browse times). At 100M lines of code you need some way to see an overall "map" of the codebase and then drill in to the bits you are interested in. Just linking via symbols like SherlockCode does is too micro of a view.

There are a lot of interesting visualization tools out there both commercial and academic. I don't have any Java specific ones to recommend but a quick Google search for "java code visualization tools" shows a lot of promise.

<comment><USER29> This thing seems like a good start but I have a bug report for you. For me at least, it's a deal-breaker. When browsing a source file, (firefox 32.0 on macos) pageup/pagedown/spacebar and up/down arrows do not scroll the code, even when the code pane has focus. Pressing any of these give focus to the search box. I need to be able to use keyboard navigation at least for scrolling.

</comment></comment><comment><USER30> A very rough estimate: assuming you have 10 experienced developers on the team, each can read and comprehend 1000 lines of code per hour. Given a 10-hour workday, the team can digest 100000 lines of code per day. To finish just reviewing the code, it will take 1000 days, about 2 years and 8 months. Not sure how much time you have and when you would expect to deliver the final product. On the other hand, if you can find out the use cases and even take a look at the current product, you then may not have to review the source code but just go ahead implementing the features.

</comment><comment><USER31> I don't think you can just do a cold re-write of that size without domain knowledge. I would first try to refactor the existing system just to reduce the code size. That big a system probably has horrific code and you can easily shrink it quickly. Just finding duplicate code will have an impact. Pulling out to open source systems like file utilities based code.

Basically I would first try to reduce the size of the problem while trying to get domain expertise. I wouldn't consider a rewrite at this stage...

</comment><comment><USER32> 1\. Configure Jenkins builds 2\. Add PMD - code analyzing 3\. Add Sonar - <USER2> analyzing (they has different rules than PMD) 4\. Use Archeology 3d > [https://github.com/pslusarz/archeology3d](https://github.com/pslusarz/archeology3d) to visualizing your code stats.

But before you start just pray to Omnissiah ([http://warhammer40k.wikia.com/wiki/Machine_God](http://warhammer40k.wikia.com/wiki/Machine_God)).

</comment><comment><USER5> Callgraph. Then document the larger chunks, working your way down.

It's like having a map versus having no map at all.

And 100M lines? Are you sure there is no code generator at work here?

<comment><AUTHOR> It's code that's been developed and it's been running for decades, I'm afraid.

<comment><USER33> developed in java and running for decades? how long has the language been around?

<comment><USER34> I started using Java in early '95 (having begged a copy from someone at Sun) and I seemed to be one of the first people writing stuff outside of Sun.

<comment><USER20> I started in August 1995 when I first heard about it, which I thought was when they released it but it may have had a variety of trickle releases.

<comment><USER34> I was working on something _slightly_ similar (embedding a VM into a browser) during '94/'95 and was a bit miffed when I first heard of Java....

However, I did think Java was rather good and when I co-founded a start-up in mid '95 we positioned ourselves as a "Java company" \- which was no bad thing in the long term as we were in a reasonable position when Netscape, Novell and IBM later decided they wanted to support it. Indeed our 2 round of VC investment was led by Novell - quite unusual for a UK company at the time...

<comment><USER35> I was doing Java in a very mainstream UK company in 1998; but we were leading edge and it became "standard" in 2000ish. By 2003 there were 100's of developers using it where I worked.

</comment></comment></comment></comment></comment></comment></comment><comment><USER35> First think is to go find the key users (start from the CEO and work down) and find out what is important that it does to them. Map that.

Find anyone technical who is still around and can talk sensibly about it and find out what they think is important. Map that.

Use anything automated to map what it's up too (calling...) and find out where the core of it is.

You may know what is important by this time, you will be able to make some sort of start...

</comment><comment><USER36> Large projects tend to accumulate lots of unused cruft. Coverage tools like emma/jacoco but I have successfully used UCDetector [1]. It's not bullet proof but it helped me lot when analyzing code to remove the unused parts, even if there are false positives sometimes. [1] [http://www.ucdetector.org/](http://www.ucdetector.org/)

</comment><comment><USER37> The hardest part is often figuring out what the inner loop actually looks like. The best way to find it is to hook up a profiler, and look at a bunch of stack traces. That'll let you find the most common entry points and calling patterns, which will go a long way towards understanding it.

</comment><comment><USER38> Read this, then try to convince them not to re-build from scratch: [http://www.informit.com/articles/article.aspx?p=1235624&seqN...](http://www.informit.com/articles/article.aspx?p=1235624&seqNum=3)

</comment><comment><USER39> A few ideas that have worked for me in the past:

Map the control flow. This code/app/whatever is doing something in production right now. What tells it to start? How does the control flow from the start point to the stuff that takes in data to the stuff that writes the output or does whatever this app does? Whatever the options for how it works are, where are they set, how do they make it into the core of the application to affect whatever it does?

Map the data flow. Input must be coming into this thing somewhere. Find where it reads it in, where it writes it out, and how it gets from one to the other, what data structures and methods it passes through on the way.

</comment><comment><USER40> 1) determine the use cases of the different applications involved (ie, what is each one used for, how does it fit into the company's workflow)

2) treat each app as a black box, understand the major data flows involved (which data sources is it interacting with? is it doing reads or writes? which tables)

3) treat each app as a black box, and try to understand the interactions between each app and any external components (other apps, web services, etc)

4) identify the overall architecture, determine the class hierarchy for each app, identify the major classes and functionality

By this stage you should be ready for a rewrite. At no point do you need to go into the code in any great depth.

</comment><comment><USER41> There is no programmatic way of doing this. You need to have guys with domain expertise to help you through. Obviously, the effort to fix/migrate will always be proportional to the time it took to create such a mess.

This is 100MM lines, it never will be easy. I would take some time to create the tooling to do this. Say, create a tool to add some bytecode to generate a pretty callgraph. Then, I'd run the use cases or functionalities individually, and save the callgraph somewhere. But in the end, you will always need domain knowledge expertise to guide you through the logic of it.

</comment><comment><USER42> From the "decentralized web"/Agile spirit: Keep the original app online, separate it in several functional domains, and replace them progressively, month after month. This way, each iteration is a small manageable chunk, functional experts can have a complete understanding of their own scope, and the result is a set of independant scalable webapps with a clearly defined scope.

... assuming you have webapps.

</comment><comment><USER43> Start at the main function. See how things get setup and walk through the code from there. Keep notes on the structure and flow of things (if any). There isn't really an easy way to do this unless it had been documented properly before.

AstroGrep is a good Windows based tool that allows you to search within file so you could use it to find which files spit out a particular output to screen.

Not sure what you mean by using ASTs though.

<comment><USER5> AST = abstract syntax tree.

With 100M lines your process will take many years.

<comment><USER44> Am I missing something or are you saying that making an AST like a compiler will help you understand a huge codebase better and faster?

<comment><USER26> "making an AST like a compiler" will give you a semantic - as contrasted with textual - understanding of the code. Especially with how much data Java encodes in its source, this makes it a very good base for running automated analysis and visualisation tools, perhaps written by yourself.

In general having AST is always better than having a plain text file, unless you want to read it. But then you can easily dump AST back to text whenever you want.

Yeah, making AST will help you analyse your codebase programmatically which in turn will let you understand the codebase better and faster. This is some very basic programming knowledge, I think. Or is it not? Some comments here don't know what AST even is - is this the state of PL knowledge in the mainstream? Lisp and Smalltalk people would be very, very sad if it was so.

<comment><USER5> I disagree an AST would help with a project this size, it's just unmanageable.

You'd be better off to start with just the build scripts and build tools.

ASTs are great for increasing understanding of _much_ smaller projects but for something this size you'd likely end up with very little to show for your effort except the crashlogs of your tools.

You need to go 'coarse' before you can go 'fine' on something this magnitude.

This is not a 3 week project, just mapping the thing properly will take (man)years.

<comment><USER26> > something this magnitude

Yeah, I started commenting before the realization of how HUGE this thing would be hit me, sorry :)

<comment><USER5> np.

I think your sorry would be better directed at this guy:

[https://news.ycombinator.com/item?id=8257519](https://news.ycombinator.com/item?id=8257519)

</comment></comment></comment></comment></comment></comment></comment><comment><USER45> Divide and conquer.

Find a way to split it into something like 10 pieces of 10m LOC each in a way where you can understand and [re-]document the data and control flow between them.

Repeat with further subdivision, as much as you have people.

Then, if you really need to re-build this from scratch, do it per component - first, make automated integration tests for its functionality, and only then attempt to rebuild that part of the system.

</comment><comment><USER46> I don't know if you want to re-write it. Having to identify all the use cases for a system that large will be horrific. Especially since people have build onto the broken ways. And a large customer will require that it be exactly like it was before so you have to recreate the old broken way and the new shiny way. Who decided it needed to be re-written?

</comment><comment><USER47> Please elaborate. Why is there need for extracting ASTs? Is it a single 100M line of Java source file? AFAIK Java has limitations on method size. If code is already organized into file and methods try to come up with some sort of UML representation. (I am assuming you are trying to understand the code base, not profiling or doing code analysis.)

<comment><USER48> Seconded.

> I have to analyse [...]

> We've started parsing it and tried to work on extracting abstract syntax > trees and all that.

Why? How will this help? What are you really after?

<comment><AUTHOR> Not a single line. The whole Java codebase.

<comment><USER48> That doesn't answer my question.

How does parsing ASTs help accomplish the goal? What _is_ the actual goal here? What is meant by 'analyse'?

</comment></comment></comment></comment><comment><USER49> Several years ago I saw an impressive demo of an analysis and refactoring tool for large Java codebases called SonarJ (now Sonargraph) by hello2morrow. There are a few other tools in this category (jdepend, agilej, jarchitect). They can give you dependency graph visualizations to help untangle the spaghetti and grok the higher-level structure.

</comment><comment> V-2 As for your first step - with a project that size (and thus of substantial age, I assume), there's surely tons and tons of dead code. I'd throw it away first. Slim the thing down. The very process of identifying unused code will already familiarize you roughly with the conceptual "shape" of the application

</comment><comment><USER12> If you're a UML guy here's a great option [http://www.altova.com/umodel/uml- reverse-engineering.html](http://www.altova.com/umodel/uml-reverse- engineering.html)

</comment><comment><USER50> What are you actually trying to accomplish? "Analyze" is a broad term.

</comment><comment><USER51> with 100 million lines of code I would: 1\. Find out what is still used, remove the rest. 2\. Split code into stand alone supportable units. Applications/Libraries etc.. 3\. Rank units in order of new requirements and what code will need to be changed. 4\. Divide code between teams. 5\. Get code to build, pass any tests and match the last released versions. 5\. go back to management and get them to let you hire lots of people. A person per million lines would be very low... 6\. Learn code in order of need.

</comment><comment><USER52> Start with a static analysis tool - it will find lots of small possible bugs, by fixing them you will get good coverage of the code and insight into the structure + it will be better at the end.

</comment><comment><USER53> The code length has to be overstated, by including libraries, generated files, or data files. Is any real code that long?

I bet the core java code the team actually wrote is 2 orders of magnitude smaller.

<comment><USER54> Yeah, the LOCs in one of the biggest banks in the world is close to 10M (including: Trading[eq & ficc] , Credit, Mortgages, Wholesale, Wealth Management, Quant, Risk, and also UIs, tests , reports , recons, interfaces) , I don't think a task like that was given to single person, usually with that base of code a big IT consulting company is hired.

</comment></comment><comment><USER55> Look into Structure101 ([http://structure101.com/](http://structure101.com/)) to use static analysis to see the structure of the application.

</comment><comment><USER56> if you can run the program, i highly recommend searching HN for strace.

eg: "whats that program actually doing. start with strace" [https://blogs.oracle.com/ksplice/entry/strace_the_sysadmin_s...](https://blogs.oracle.com/ksplice/entry/strace_the_sysadmin_s_microscope)

i've successfully reverse engineered messaging protocols, written drivers for a different language, and ported large projects just by trying to see what it does over the file system, network.

~B

</comment><comment><USER57> Just curious: what application needs 100M lines of anything?

<comment><AUTHOR> It's not just one application. It's many of them.

<comment><USER58> I agree with kubiiii.

Since a rewrite is in the card (hopefully) there is something wrong with the entire system.

* Identify where the applications interact with each other.

* Identify the most problematic applications.

* Rewrite those (starting with the smallest) while trying to keep the interfaces between applications constant.

And involve end users as much as possible.

</comment></comment></comment><comment><USER12> On Eclipse [http://www.nwiresoftware.com/products/nwire- java](http://www.nwiresoftware.com/products/nwire-java)

</comment><comment><USER20> jvisialvm comes with the JDK, I'd start there with profiling:

[http://visualvm.java.net/profiler.html](http://visualvm.java.net/profiler.html)

Edit: Adding, I'd set some judicial breakpoints in the hot spot areas identified through profiling along with some System.out.println's (or better dump to a flat file database, SQL can be used to work wonders for analysis even for flat file data).

</comment><comment><USER59> You need to have a clear understanding of the point of the analysis before you analyze anything. What, specifically, does your team have to produce? How much time and how many people do you have to complete the work? If you're the leading edge of an effort to rewrite 100MLoc, my presumption is that your deliverable is mainly a 'gross anatomy' of the system... a basic description of the major structural components and how they interact with each other. If that's the case, I'd start by looking at the build scripts and the modules they build. Try to make a comprehensive list of major components. You'll get it wrong initially, but you'll need a starting point.

The next thing I'd do is take the top level list of modules and start assigning it to individual people within the team. Their responsibility is to produce some kind of top level description of how the individual modules work. A big part of this phase of the effort should be meetings or informal conversations as the per-module analysis progresses. As your team talks among itself, you should be able to find commonality between modules, communication links, etc. The key at this point is to keep it high level, and avoid getting too bogged down in the details. With this much code, there are plenty of details to get bogged down in. As a result, you'll probably have some mysteries about how the code actually works beneath various abstraction layers. Make and update a list of these 'mysteries' and keep it next to your team's list of modules. As you work through the list of modules, some of these will solve themselves, and some will be so obviously important that it's worth a detailed deep dive to really understand what's happening. Either way, there will be times that you have no idea what's going on in the codebase and you'll just have to trust that you'll figure it out later.

One final comment I'd like to make is that, as silly as SLoC is as a measure of the size of a software system, you're looking at a large software package. (Bigger than Windows, Facebook, Linux, OSX, etc.) If you take each line of code to have cost $5-10, then the system arguably cost $1B to build in the first place.

Because of the size of the system, you shouldn't expect your analysis work to be easy, fast, or cheap. Buy the tools you need to do the work. This means technical and domain training, software, hardware, process development, new staff,... basically whatever you need to make the work happen. You're at the point where long term investments are highly likely to pay off, because your scope is so large and your timeline is entirely in front of you.

I'd also highly recommend working this problem from two angles. You can understand the existing system by looking at the code, but you also need to clearly understand the system requirements from the 'business' point of view. If you're doing bottom-up analysis, then some other group needs to be doing top-down. Along those lines, you should also start to thinking about deployment strategies. I highly recommend avoiding a big bang deployment of that large of a system, so there will be some period of time when you're liable to be running both the 'old world' and the 'new world' systems at the same time. Think about how you want to do that...

There is lots to think about here, because this is a complex problem. Hopefully, I've given you at least a little bit to think about. Good luck.

</comment><comment><USER60> rebuild 100M lines from scratch? sounds impossible to me.

You can re implement the applications that are causing problems maybe one at a time maybe.

I don't understand what you want to get syntax trees for, but it sounds like you are gonna need to store them in a database and do queries on it if there really is info that you need.

</comment><comment><USER61> why would you need the ast?

if given a large chunk of code to maintain i'll usually run doyxgen on it to generate the xmlish kind of chart that it makes. at least it gives me a roadmap to start, but it's not super great.

<comment><USER20> I used this on some projects, works well. Building with the Dot diagrams is a good visualization, in the final HTML documents choosing Classes | Class Hierarchy shows them.

</comment></comment><comment><USER62> Read "Working Effectively with Legacy Code".

</comment><comment><USER35> Also - what is the history of getting into this mess?

</comment><comment><USER63> check out [https://github.com/facebook/pfff](https://github.com/facebook/pfff)

</comment><comment><USER64> I would build a general profile of the application and then drill in as needed rather than try to grok the whole buffet at once.

If the idea is to rebuild the application I would start at the beginning: what is the input and output? What does the user see? What are the various service hooks? How are they called? When are they called? Why are they called?

Then I would look at how the overall code is organized. What modules are there? Are there core utility modules that seem to be called by everything else? What are those doing? What are the most used business function modules?

Then I would look at the build process. What external dependencies are there? What are they used for? Are there modern alternatives? What about internal dependencies? Does the build process look organized and sane or a chaotic mess cobbled together over the years?

Do you have logs? What is the most utilized part of the application?

Then I would look at the database. What tables seem to be the most important (if you could get usage stats from a running and used application that could help, but otherwise you could look at which tables are keyed off of the most)? What data is most critical? What modules interact with that data? What tables are essential for supporting this data?

Answering these questions will start to fill out a nice 30,000 ft view of the application and how it is actually used.

You are going to get the most bang for your re-implementation buck by identifying and replacing often used utilities (especially if they are custom built or built before a good de-facto standard was formed for that particular task) with modern, well known, alternatives. Then follow the execution path of the most often used modules and the modules that work with the most critical data and work down the list.

With a 100 million line application, you are looking at many years to understand all of it and many years to re-implement. To get anything useful in a reasonable amount of time you are going to have to boil it down as much as possible, then break what's left down into independent functional areas and tackle it an area at a time.

The code is important, but if it were me, I'd try to analyze how the users and processes work before I'd dig into the nitty gritty of the code too much if at all possible. I'd build the smallest functional unit from what I deem to be the most important and critical module(s) trying to cut as much cruft from the application and database as possible. I'd get users and processes to start banging on the new app as soon as possible. I'd keep the old application up and running and available to analyze (not for the users but for the developers and analysts) as the team works down the most often used parts. I would not try to analyze the whole mess in one go beyond finding waypoints as described above. If possible I'd also try to get users to understand that the old way is not necessarily the right way. Much pain has been caused trying to make new systems work exactly like the old systems when the new systems don't face the same constraints. It is just too tempting to say 'make it work like it did'.

</comment><comment><USER65> box

</comment><comment><USER66> You could try to reduce the lines of code by removing alle the useless whitespaces.</comment>


Dracula Graph Library - <AUTHOR> https://www.graphdracula.net/ <comment><USER2> Very interesting. However, I've yet to see a graph library that effectively handles large graphs/communities.

It's very difficult to automate displaying more than about 10-15 connections in a meaningful way. I think a solution to this probably might be extremely far away though, as I think it requires making assumptions about meaning and where summarization would be appropriate.</comment>


Building a 10BASE5 "Thick Ethernet" Network (2012) - <AUTHOR> http://tech.mattmillman.com/projects/10base5/ <comment><USER3> I am _so_ happy that coax bus based ethernet died. One of the hardest to track bugs for me involved a machine named 'chopper', an SGI box set up to monitor a bunch of other machines. Every now and then chopper would report one of the other machines on the network as down, and send out a page to an operator, which got quite annoying. We tried everything we could think of to trace the fault, replacing all the bits and pieces, including entire servers to figure out what was causing the problem.

Eventually the only thing left that we had not swapped out was a BNC 'T' connector at the back of the MAU of chopper itself, the one that was used to put it on the coax bus.

My buddy Jasper and I both looked at each other at the same moment around 4 am or so, bleary from sleep 'that can't possibly be it', let's swap it out. And sure enough that was it.

Apparently that particular 'T' connector had electrical capabilities unlike any other 'T' connector produced before or after. It managed to filter out 'ping' reply packets from the IP address of the server reported as down...

<comment><USER4> As someone who does temporary ad-hoc (read event) networking, having a coax standard for high speed, long point to point runs would be awesome - <USER2> that, 100 or 1000baseT 2 port POE repeaters (read hub) would also be a boon - you could get nearly 1000 feet by powering from each end of the span.

Why not fiber? Its fragile, expensive, hard to pull, and not well suited to temporary situations, precut armoured fiber is very expensive - coax is fragile, but cheap and easy to terminate.

<comment><USER5> FYI there are a variety of MoCA products out there that support 175Mbps Ethernet over coax up to 300':

[http://www.amazon.com/gp/aw/d/B008EQ4BQG?pc_redir=1414247414...](http://www.amazon.com/gp/aw/d/B008EQ4BQG?pc_redir=1414247414&robot_redir=1)

<comment><USER4> Thats actually kinda cool!

</comment></comment></comment></comment><comment><USER6> It's pretty interesting to read into the original "broadcast" ethernet specs.

The "thick yellow wire" and the "coax" both had a single broadcast domain for all devices on the segment. You then linked segments with a bridge (so you could scale networks).

Some things I find interesting about them:

1) The idea is based on radio broadcast (ALOHANET). Since you're using radio broadcast there, a single collision domain makes sense. [http://en.wikipedia.org/wiki/ALOHAnet](http://en.wikipedia.org/wiki/ALOHAnet). The protocol is best understood as "room full of people shouting at each other". Don't start talking if something else is, and if two people start at the same time, you both back off. Simple as that.

2) If two nodes try to transmit at the same time, you get a collision and both retry. The spec has a random retransmit timer to avoid immediate (and perpetual) recollision. Requires fair nodes...

3) The max cable length and packet size and related via the speed of light (in copper): [http://www.wildpackets.com/resources/compendium/ethernet/pro...](http://www.wildpackets.com/resources/compendium/ethernet/propagation_delay)

4) It was (semi-)reasonable to use a "vampire tap" connector to dig through the insulation and into the core of a the "thick yellow" copper cable. Just don't wiggle it and cut the copper (or introduce too much of an edge and get signal reflection and hence more collisions).

5) When you had a copper connection between all the ports on your network, an etherkiller really _meant something_. (A cable, mains plug on one end, coax connecter on the other. Plug in and fry every network card on the segment). RJ-45 etherkillers are single-target. Pah.

<comment><USER7> Ethernet is still "broadcast" (CSMA/CD). It's just that now, with switched ethernet, the broadcast domain is reduced to the switch port and the station.

</comment></comment><comment><USER8> One network which I have a particular fondness for is Arcnet. It also used "thick ethernet" (93Ω) cabling, but all stations were connected to a central hub. It was remarkably resilient.

I used it a lot for gaming (LAN parties), back in the days when Ethernet (that's thin coax ethernet) cards were still expensive. Arcnet equipment could be obtained for cheap (or even free), the speeds were enough for gaming (Doom, Quake, Descent), the DOS drivers were great, and overall it was a great solution which Just Worked. Most importantly, it didn't have the problem of one person bringing down the entire network just by disconnecting the cable at his station, which thin ethernet had.

See [http://en.wikipedia.org/wiki/ARCNET](http://en.wikipedia.org/wiki/ARCNET) for more details.

<comment><USER9> Points for mentioning Descent. A decent remake would be excellent... the extra resolution of its successors somehow failed to maintain the grit of the original. We used to play over serial cables before we had ethernet. Incidentally, another favourite at the time was Heretic. For no apparent reason my system used to work much faster at Heretic than anyone else's, showing excellent screen refresh rates despite an equivalent or lower CPU clock. Never quite figured that one out, perhaps the (Oak?) video card, the motherboard, or the RAM clock speed. Ahh, the days before pervasive internet... where social networking meant plugging cables.

<comment><USER10> Descent came up 4 months ago on HN: [https://news.ycombinator.com/item?id=7948262](https://news.ycombinator.com/item?id=7948262)

[http://descentchampions.org/new_player.php](http://descentchampions.org/new_player.php)

</comment></comment></comment><comment><USER11> Cached link - [http://webcache.googleusercontent.com/search?q=cache:_rBACik...](http://webcache.googleusercontent.com/search?q=cache:_rBACikqzuQJ:tech.mattmillman.com/projects/10base5/+&cd=1&hl=en&ct=clnk&gl=uk)

</comment><comment><USER12> "Error establishing a database connection"

10 Mbit/s doesn't cut it apparantly.

<comment><USER3> How do you know his database server is on the other end of a 10 Mbit link? What if the datbase and the web server are on the same box and the outside link is doing just fine but his db is simply overloaded?

I don't think you can take it as given that the network is the problem when you see an 'error establishing a database connection'. I'd start with looking at the database.

<comment><USER13> woosh. He was clearly joking.

On another note... this is clearly MySQL. I recall this error from the 1990s

</comment></comment></comment><comment><USER14> Wow, this is a blast from the past, circa 1992-1998. For my sins I used to maintain a fairly large Thick Ethernet network for a large textile manufacturing company in the UK (working for a contractor that basically had very few clues about employee health and safety, hindsight is a wonderful thing).

A fair chunk of this network ran through their factory up in the factory roofspace. I used to climb these ridiculously high step ladders to reach up into the factory roof steelwork to install "vampire taps".

It used to scare the bejesus out of me, mostly due to an aversion to heights, the other being that the factories produced cotton thread, which during one of the processes requires the application of a fine mist of wax or light grease, I don't remember why. But every high up surface was deceivingly lubricated with a fine layer of wax or grease as a result of being kinetically atomised into the air by high speed thread spinners, rollers, spoolers and guides. To further focus your mind, there was also an opportunity of being impaled on these massive crane lifted pointy multi-spool holder things should you put a foot wrong (they were used to lift large bulk spools of rough spun cotton into even larger dying machines).

Although we had a proper coring tool my biggest fear was of creating a short between the shield and core and unf*cking that up a 30 ft step ladder in that environment would have ruined my day (we had to install these things on live segments, there was no downtime permitted). Fortunately the quality and accuracy of the tooling was good, the customer used the correct cable (not some cheap imitation) and my karma and anxiety levels could return to normal once the transceiver lights flickered into life.

Just thought I'd share my lasting memory and experience of 10BASE5 hands on work.

</comment><comment><USER15> The most amazing thing here is the USB-to-AUI converter. Wish I could reach the cached page for that.

<comment><USER16> here you go.

[http://tech.mattmillman.com/wp- content/files/axaui_m1.pdf](http://tech.mattmillman.com/wp- content/files/axaui_m1.pdf)

[http://webcache.googleusercontent.com/search?q=cache:_rBACik...](http://webcache.googleusercontent.com/search?q=cache:_rBACikqzuQJ:tech.mattmillman.com/projects/10base5/+&cd=1&hl=de&ct=clnk&gl=de&client=firefox-a)

</comment></comment><comment><USER17> He doubts network gaming would have taken off?!

I spent many hours playing Doom and Warcraft 2 on exactly this setup! (Feeling a bit old now...)

</comment><comment><USER18> Wow. I admire this amount of attention to detail (with the custom USB to AUI interface) for a "fun" project. Nice work.

</comment><comment><USER19> I hope he enjoys going round and working out which ethernet connector has fallen out, because of those useless "slide locks". I definitely don't miss thick ethernet.

</comment><comment><USER20> I actually have a couple of old PCI ethernet cards that have both standard RJ45 connectors and BNC coax connectors. Does that mean I actually have something rare?

<comment><USER7> Not as rare as the handful of PCI ethernet cards I have with a RJ45 and a _DB9_ connector. 100BASE-CX, ethernet over STP, used for a very brief period of time so people could reuse their Token Ring wiring plant for Ethernet.

<comment><USER21> Is it DB9, or AUI? (AUI is 15-pin, and is the card required for this story)

<comment><USER7> DB9. As I said, it's so customers with a wiring plant based on Token Ring STP (aka Type 1) could use ethernet at the end points without running new wiring.

</comment></comment></comment></comment><comment><USER22> I remember when I first mount a network on my house to play with my brother DooM , with coax Ethernet and with a few ISA network cards that I got for free...

</comment><comment><USER23> next you have to try the old standard of terminating either or both ends of your thick ether with thin ether (and then real terminators at the far ends of that)

(and yes thick ether standardly came with connectors on the end, I still have a roll somewhere, I have no idea why)</comment>


A central extended amygdala circuit that modulates anxiety - <AUTHOR> http://www.jneurosci.org/content/early/2018/05/29/JNEUROSCI.0705-18.2018 <comment><USER2> Why is this here? If interested in the circuitry of fear and the amygdala these are better reviews:

[https://www.ncbi.nlm.nih.gov/pubmed/20393190](https://www.ncbi.nlm.nih.gov/pubmed/20393190)

[https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4565157/](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4565157/)

</comment><comment><USER3> Whilst this is quite interesting, I struggle to see how the animal suffering and the human brainpower + effort + resources required to do this work is justified..

<comment><USER4> People living with anxiety disorders justify it.

<comment><USER3> Knowing more detail about anxiety circuits in mice is very very far from helping people living with anxiety disorders.. More research on improving existing human treatments that are known to work well for many people seems to be more justifiable..

<comment><USER4> More research on improving internal combustion engines that were known to work well seemed more justifiable than pouring money into electric vehicle research.

I'm pretty sure there was a time when improving steam engines was more "justifiable" than figuring out how to make internal combustion engines too.

Or breeding stronger horses instead of researching steam engines.

So no, that's not how science works.</comment></comment></comment></comment>


Ethics: A company goes back on its promise -- revisited - <AUTHOR>

This post is a follow through post for http://news.ycombinator.com/item?id=1355550
---
Quick recap: A company I recently interviewed for, has been giving me pains in reimbursing my money they had promised to me before the interview. I have their emails and it has been two months.
---
Folks here at HN have been really helpful. Thanks a lot for your comments.
So, I have called the company and they say they are working on it. Meanwhile, I have another questions: Is it ok if I write a blog post on my personal blog about my experience in that company? Can they sue me if I do that? <comment><USER2> Truthfully, I wouldn't recommend it. Hiring managers have a very unhealthy knack for dissecting everything you say as means of determining what kind of employee you'll be.

Rattling the cage in _any_ fashion in this situation is only going to hurt your chances of finding another job if it gets enough attention. That's not your fault, because I'm sure you're capable of writing an excellent entry on this. Sadly, hiring managers are too superficial.

</comment><comment><USER3> _"Can they sue me if I do that?"_

Ask an attorney or for self-help try something like Nolo press' web site.</comment>


The Open Standard, a news site about open, transparent, collaborative systems - <AUTHOR> https://openstandard.mozilla.org/

<comment><USER2> Often I see good critiques and discussion of Mozilla software endeavors here on HN. I rarely see much discussion about the Mozilla community at large. There was a discussion during the Eich appointment scandal, but I didn't see as much as I had expected.

Mozilla is an extraordinary organization. I would love to see more discussion and critique of the Foundation's non-software community work.</comment>


We made a HUGE mistake - <AUTHOR> https://medium.com/inside-floown/we-made-a-huge-mistake-f4ff3069d558 <comment><USER2> I thought naming it "floown" was the mistake. Apparently the mistake was actually not firing everyone involved and starting over.

"our platform that helps organizations always know who's available to work."

What does that even mean? Is it a calendar? Ugh this .com stuff drives me up the wall...

<comment><USER3> There's also this beauty:

>Still, we pulled through and re-shifted our focus to our original vision: to build an on-demand versatile team that truly believes in a future of work where people have multiple clients or employers.

That one has so many buzz words I'm not even sure what it was they were trying to say. Luckily they clarify it in the next sentence:

>A team that believes in the platform we're trying to build, and more importantly, believes in the roadmap, decisions and actions of the founders.

Which I presume means that some of the initial people they hired had their doubts about the product and the competence of its founders. Since those founders responded by firing them all and hiring new people, I'm starting to have doubts as well. Maybe I'm reading too much into it, but if they expect the people they hire to have full confidence in their product and its founders then I don't think I meet their hiring criteria.</comment></comment>


I made pagination suck less - <AUTHOR> http://programmerstrouble.blogspot.com/2011/03/design-pattern-pagination-with-useful.html It certainly seems to make pagination less of a pain, so +1 <comment><AUTHOR> Has anyone seen this before on a website? It seems so simple, I find it hard to believe that this is the first implementation.

<comment><USER2> It ceases to solve a problem when the user controls sorting.

<comment><AUTHOR> I believe it still works. When the user sorts on another field, the hint shows that (for strings, the first 3 characters).</comment></comment></comment>


Some Nice Features of the Objective-C Language - <AUTHOR> http://pmougin.wordpress.com/2008/03/13/some-nice-features-of-the-objective-c-language/

<comment><USER2> Ahh, Objective-C. Such an under appreciated language. Perhaps not for much longer though.</comment>


Ask HN: What exception tracking service do you use? - <AUTHOR> I've been looking at Honeybadger, Airbrake and Rollbar.  I've noticed that this same question was asked on HN three years ago, but I expect the feature sets have changed since then. I'm looking at this for a Java/Spring web application. <comment><USER2> We use Honeybadger at Honeybadger (I'm one of the cofounders), and think it's pretty sweet. :)

Our Java client supports Spring: [https://github.com/honeybadger- io/honeybadger-java#spring-fr...](https://github.com/honeybadger- io/honeybadger-java#spring-framework-usage)

</comment><comment><USER3> Sentry -- can't say I've used the others you've mentioned.</comment>


In Minneapolis, Armed Residents Set Up Patrols Amid Calls to Defund the Police - <AUTHOR> https://www.marketscreener.com/news/In-Minneapolis-Armed-Residents-Set-Up-Patrols-Amid-Calls-to-Defund-the-Police--30991764/?countview=0 <comment><USER2> Welcome to unintended consequences of ineffectual governance by reactivity to systematic selective amplification of vocal minorities without counterbalance. (Doesn't matter which side you're talking about; if you can't acknowledge a middle-ground and commit to all or nothing, destabilization is what you get)

People _will_ self organize in the abscence of an official, agreed upon need, whether an official body funds it or not. The major difference is, what gets coordinated will lack thorough integration with the rest of the system, accountability or auditability will greatly suffer, and everything will become less transparent.

One of three responses I see as likely:

A) The government realizes the magnitude of oops, and works to figure out what they need to do to get things back to the point everyone is willing to delegate and accept the official edifice again. (One would hope this would be the outcome from a Statesmanship perspective. Rollback and try again in programming parlance)

B) The government doubles down and tries to clamp down on the ability of private citizens to organize their own protections given their lack of faith in the official institution. (This road leads to serious risk of State legitimacy issues or further destabilization; shotgun debug or rearchitecture in programming parlance; who is doing the shotgunning is left as an exercise for the reader's imagination)

C) Deadlock occurs, no official measure for change to compensate locally can be realized, and the complexity bubbles up a layer of abstraction until more remote systems come into play to neutralize/compensate for the change. (Exception raises in programming parlance)

Either way; hell of a ride in progress.</comment>


Ask HN: Watch people write code? - <AUTHOR>

Does anyone know of events where i can watch someone else write code. Not virtual, but actually sitting next to the person and just watching them code in real time.
Don't know why, but i find it relaxing. I am talking about really mind-calming, zen-like relaxing.
Does anything like that exist in real life? (Damn, i sound like a creep, but my question is serious) <comment><USER2> I've thought about putting up youtube videos where I go through a project, start to finish (including mistakes and all), but I couldn't imagine it being educational or entertaining. Also I almost never finish anything :/

I kind of understand though - since I do get into a zen state just _looking_ at code sometimes. Just kind of the shape of it sometimes is calming.

</comment><comment><USER3> In any software development team in almost all company?

There are also events like: [http://princetonacm.com/code-at- night/](http://princetonacm.com/code-at-night/) but I wouldn't say it's how code is written usually. Those are exceptional situations.

</comment><comment><USER4> There seems to be quite a bit of existing content on this topic: [https://www.google.com/search?q=watch+people+code](https://www.google.com/search?q=watch+people+code)

</comment><comment><USER5> Totally agree, you sound like a creep. J/K. I too love watching others write code, and usually learn a lot in the process. Try YouTube. I learned PHP just watching YouTube video's.

</comment><comment><USER6> Actually I think this is a pretty good idea. I know I have learned a lot when doing some pair programming watching other people code.</comment>


How We Learned to Love the Bill of Rights - <AUTHOR> https://www.bunkhistory.org/resources/1837?related=388&relationship_name=RELATED <comment><USER2> Reading the Soviet Bill of Rights is pretty instructive because it'll show you how much such a thing is worth on its own.

Here is just a subset of the rights guaranteed by the USSR Constitution: speech; press; religion (both to profess and refuse to profess); assembly (including a specific right to demonstrate); participation in lawmaking; inviolability of the person (from warrantless arrest) and of the home (from unlawful entry or search); privacy of all correspondence (including phone conversations); right to challenge actions of government officials and bodies in court; right to a job; work week limited to 40 hours; free medical care; welfare payments in old age, sickness, or disability; free housing; free education; academic freedom for scientists.

It's a very good list, and what I listed above isn't even half of it [1]. Precisely none of those were actually respected by the government. Having a Bill of Rights is great, but having a tradition of limited government, checks and balances, and rule of law is what gives those guarantees any weight.

[1] [https://www.departments.bucknell.edu/russian/const/77cons02....](https://www.departments.bucknell.edu/russian/const/77cons02.html)

<comment><USER3> This is what scares me about the current attitude (nationally) towards gun control. The right to bear arms is a basic civil right, unambiguously encoded in the Bill of Rights alongside speech, assembly, religion, due process, etc. Throughout history, taking away weapons has always been an early step down the slippery slope to totalitarianism and stripping the populous of their other rights. Just because it's unlikely to happen here -- that we _think_ it can't happen here -- doesn't mean it will not. That's what everyone in all of history has thought, but in fact by the time anyone realizes what's happening, it's usually too late.

"All political power grows out of the barrel of a gun. The communist party must command all the guns, that way, no guns can ever be used to command the party." \-- Mao Zedong

<comment><USER4> Let's talk in practical terms for a second, though. Do you truly believe that gun ownership in the US could result in a bunch of citizens overthrowing the US government should it become tyrannical? I just don't see it happening. The US military is far too well-trained and -armed, and of _course_ a hypothetical tyrannical US government would turn the military against its citizens if there was an armed uprising. And they would win so laughably easily.

Now sure, there's the possibility that factions in the military would turn against this tyrannical government and save the day, but you don't need the 2nd Amendment for that to work out.

<comment><USER5> Afghanistan. 16 years, still haven't won.

And us citizens have more weapons than the taliban; longer harder to protect borders than Afghanistan; and a population that would be sympathetic to the cause. I wouldn't be so sure the us military would win.

According to current military doctrine[0], counterinsurgency requires a minimum of 1 counterinsurgent per 50 people in the population. If you assume ALL us military personnel can serve in combat simultaneously (ie: no troops required for supply lines, paperwork, management, etc), and you included all police forces in the US, you would only get to 1 per 100.

So you say they would "win so laughably easily".. but according to the military itself, it does not have enough troops to handle an insurgency in the US.

0\. As written in the Army and Marine Corps field manual (and this doctrine was based on research of past (us and non-us) conflicts)

<comment><USER6> Also, 99 percent of the US military will defect to protect their own country against a rogue federal government. These are normal, good people who signed up to protect us.

<comment><USER7> Are you sure about that?

History shows us that typically, soldiers will fight for whoever guarantees their pay.

Combined with this;

[https://en.wikipedia.org/wiki/Milgram_experiment](https://en.wikipedia.org/wiki/Milgram_experiment)

it shows that money, orders and chain of command are powerful motivators.

 _These are normal, good people who signed up to protect us._

I wouldn't put that number anywhere near 99%. Probably closer to 50%. Maybe even lower.

I'm sure the people in any given totalitarian regime once thought the same way about their armed forces.

</comment></comment></comment></comment></comment></comment><comment><USER8> Unfortunately, pretty much all of the BoR is regularly under attack. Those rights have to be constantly reaffirmed.

<comment><USER9> On their defense, people mean well, but 'the road to hell did paved with good intentions' as they say.

1st Amendment is being attacked by liberals, with pronoun laws, hate speech regulations, equal opportunity laws, forced cake baking, many other things.

The 2nd is also being attacked from a variety of ways by liberals, which as said before, is well intentioned, but ultimately destructive.

Thank goodness the 3rd isn't an issue!

The 4th amendment is being attacked by both conservatives and liberals in a variety of ways. From NSA data collections on Americans under Obama, to DEA/BATF sponsored civil asset forfeiture under Bush, and unlawful searches by ICE under the current administration, this Amendment is very important but is being chipped away by both authoritarian ideologies.

Ah, our 5th amendment has been weakened by both liberals and conservatives. From No Fly lists and travel bans you can't get out of, to waiting periods on firearms, we've truly let this very important amendment become weakened.

I like _all_ of the amendments. I don't want to see any single one of them weakened any more and some of our rights restored.

<comment><USER8> > I like all of the amendments.

I'm not an NRA member, but the idea of repealing the 2nd Amendment, or worse, interpreting it out of existence has bothered me in ways I couldn't quite put my finger on. I finally realized what it was. If we decide one of the rights is "archaic", "obsolete", "insane", or any of the epithets routinely leveled at it, we set a terrible precedent. How long will it be before the rest fall when they are inevitably inconvenient?

We need to stand for _all_ the BoR. Not whatever is convenient.

<comment><USER10> Scott Greenfield covered your exact point pretty well over at his "Simple Justice" law blog [https://blog.simplejustice.us/2015/10/06/ten-favs-from- the-b...](https://blog.simplejustice.us/2015/10/06/ten-favs-from-the-bill-of- rights/)

<comment><USER11> This amendment feels so random, tho. Why firearms and not other weapons? Why not general means of defense?

There are also consistency issues, the prohibition was an amendment, what made that thing negociable?

Other fundamental rules are completely inconsistent like being 21 to drink while the majority is 18.

<comment><USER12> > Why firearms and not other weapons?

Something something bringing a knife to a gun fight...

The amendment doesn't specify firearms; it just so happens that firearms are the most contentious weapons protected under it, for obvious reasons: firearms are among the most deadly weapons in common use, but hoplophilic citizens do not want to be denied access to weapons their enemies have access to. Nevertheless, there was a Second Amendment case brought to the Supremes ( _Caetano v. Massachusetts_ ) about a woman's right to buy a stun gun, which was illegal in Massachusetts. The Supremes found in her favor.

As for why the Second is considered "special" \-- it's part of the Bill of Rights. There's nothing stopping anybody from repealing it, but the ten amendments in the Bill of Rights are considered special in American legal tradition because the promise of their passage was essential to the ratification of the Constitution itself. So it would take much more effort to gather support for repeal of a Bill of Rights amendment than the Eighteenth Amendment (which was largely passed because of the temporary political vogue of the temperance movement).

</comment></comment></comment></comment></comment></comment><comment><USER13> I am no American, but I think Americans are wise to revere their Bill of Rights partly because it is a reasonably good model for rights that everyone should have but also because Americans have lived for a long time with those particular and have made them mostly work. And as the article makes clear, the Bill of Rights didn't really just happen in 1789, it has evolved and (mostly) improved over 200+ years.

And that's a good reason to be skeptical of worries about "fetshising" it and dreams of how you could "remake the legal and political world". Big changes happened after the Revolution and the Civil War, how could they not? But evolution has happened at at all other times.

</comment><comment><USER14> _"...And he worries that this process of reinvention may have stalled, with Americans now fetishizing the Bill of Rights, having lost sight of their own ability to remake the legal and political world..."_

And I'm beginning to seriously doubt my time commitment to this article.

The actual history of the Constitution and Bill of Rights is quite interesting and involved. It covers everything from classical literature and history to The Enlightenment and the Thirty Years' War. Just talking about how some folks revere the docs so much now and didn't in the past is breathtakingly shallow. As an example of how short-sighted this is, note that the BoR wasn't supposed to say anything that needed saying. Some questioned the need for them at all. The government wasn't constructed to get that overpowering and intrusive into all of our lives. It's made not to do that. Why should we start making lists of things it can't do? It obviously can't do anything we didn't tell it to do.

The founders in the U.S. dispassionately threw together a huge chunk of natural science, philosophy, and history, made a ton of compromises, and did the best they could. You don't have to worship or deify the folks or the documents they made. Just understand what was done and why. That's much more useful than going on at length about how unhealthy some segments of the population are being.

This article is a political polemic, not a history lesson (unless you consider it a lesson on how things get more and less popular over time, which says absolutely nothing about the BoR) I love political essays that show how public thought changes over time. These kinds of essays tend to be difficult to discuss in large diverse online groups -- and many times, sadly, their purpose is to create false, simplistic narratives that can take the place of understanding. Best consumed in small doses and with lots of context.

</comment><comment><USER15> In the post, Magglioca says that the bill of rights was not what people of the time thought a bill of rights looked like. Well I find it is not accurate to say that since many ideas were lifted from the English Bill of Rights 1689. No standing armies in peacetime, freedom of speech of parliament, peasants bearing arms, no cruel unusual punishment, no excessive bails, no taxes without parliament. I hope that I am missing something in translation, because it is unsettling that he would overlook this. The Virginian Bill of Rights mentioned in the story took from this as well. In other words, it set a standard.

</comment><comment><USER16> In New Zealand we don't have a constitution. But rather we have a Bill of Rights defined in law, which can be changed by an act of parliament, and a series of laws defined around this bill of rights.

It's not the constitution that makes a country free, but the people fighting for their rights. New Zealand gave women the vote before any other country in the world, and we didn't need a constitution to do it, but rather the willpower of the people.

</comment><comment><USER17> This article is critical of the "cult" of the bill of rights. But look at China the other day and how their constitution was changed on a whim. Look at Russia which has Putin despite of a fairly modern and standard constitution.

Constitutions and laws are mere pieces of paper until the society within which they live endow them with sacred status. Without the "cult" and the moral indignation at violations surely civil liberties would erode at a much faster rate. As another comment says "Unfortunately, pretty much all of the BoR is regularly under attack." If the BoR didn't have the same nearly religious status one of those attacks would have succeeded by now.

<comment><USER18> > If the BoR didn't have the same nearly religious status one of those attacks > would have succeeded by now.

I don't think this follows at all from the article. After all, one of the main points is that there WASN'T a cult until the 1940s...yet those rights were able to stand for 160 years without a religious status supporting them.

Your argument that the Bill of Rights had a religious status for all 200+ years of its history seems a direct contradiction of one of the book's central claims and I don't see that you've presented any arguments in support of your counter-claim.

If the Bill of Rights didn't have religious status for those 200+ years then your subsequent claim that the religious status is a necessary part of keeping it intact would seem to wither.

<comment><USER19> Perhaps the religious status evolved in response to increasing attacks on the BoR?

<comment><USER18> Sure, that's a possibility but is yet another variant that wasn't claimed and provided with no backing evidence.

</comment></comment></comment></comment><comment><USER20> I ctrl+f'd "natural rights" and came up with nothing, which is how I knew from the start this wasn't a very good article on the subject.

Look, here is the thing that people have forgotten, or were never taught (due to a failing civics education system):

People have natural rights, independent of any government, and governments are formed among the people in order to protect those rights. _The government does not grant rights!_ _It can not take them without due process!_

This is the fundamental principle of what I consider real American exceptionalism, and why American government at least in principle is the most free system of government in existence. (despite it's failings to live up those principles often) It's why Christopher Hitchens became an American and called it "the last revolution that stands a chance".

This was the original debate about the Bill of Rights in the first place. That by listing a few important rights the government would then think anything not listed would be allowed for the government to violate. Hence the "right to wear a hat" argument. That's not how things work though, and that's not what our government was founded on.

I'll give you a good example. Locke calls the right to self defense the first natural right. I agree. The second amendment is about defense against a tyrannic government. You could take it away and I would still have a right to bear arms independent of the (theoretically) absent second amendment.

Both parties hate stating this sort of thing out loud though because what they both are is authoritarian. For the conservatives, it would undermine all kinds of their moves like the drug war (unconstitutional abuse of commerce clause, states rights), and the same can be said of the left. This is why I get tired of all the cries anytime someone sees through the bullshit and calls them both equally corrupt, which they are, and are immediately hit with logically fallacious half-retorts of "whataboutism" and other rebuttle du jours.

I swore an oath to the Constitution, and I stand on those principles even when they make me uncomfortable. I'll give you another example. I always thought FDR's New Deal was one of the best things ever, a brilliant move we should seek to emulate in the modern day. When I dug into it though, I have to admit I have seen enough evidence that many of the New Deal policies were unconstitutional that I have an entire reading list prepped just to dig into the matter. So you can see how for someone who really wants more wealth equality how difficult it is to potentially admit minimum wage just might be unconstiutional. At least I am meeting those hard questions head on though, instead of burying my head in the sand.

<comment><USER4> > _People have natural rights, independent of any government, and governments > are formed among the people in order to protect those rights._

Not really. We've come up with that fiction because we believe -- and we're probably right -- that it gives us the best chance at living together peacefully and without interfering too much with each people's lives.

But it really is a fiction. I personally believe in what you call a person's natural rights, but that's all it is, just a belief. It takes all of us to band together and form structures to enforce and protect those rights for people to be able to enjoy them.

> _The second amendment is about defense against a tyrannic government. You > could take it away and I would still have a right to bear arms independent > of the (theoretically) absent second amendment._

I mean, sure, if you define "right" in that way, then you can say you have the "right" to literally anything. If the 2nd Amendment were repealed, that imagined "natural right" won't help you much when the government comes and takes your guns away, or jails or kills you for having them when you try to fight back.

There is no absolute moral truth here. It's only what we've collectively decided on, and we still have a long way to go before everyone in the world is treated equally and with dignity. We look at the world 500 years ago and think of how uncivilized everything was, and how poorly people treated each other. In another 500 years they'll be saying the same about us, assuming we don't destroy ourselves before then.

<comment><USER21> "Natural rights" don't come naturally. Something is a "natural right" not because laws of physics dictate they are inviolable but because they are essential to humanity and peaceful existence, and every human is born into them.

The "natural rights" to life, liberty, pursuit of happiness, assembly, association, religion, arms, etc. are the closest we've come as a race to "absolute moral truth".

</comment></comment></comment><comment><USER22> Why does the headline say "the Bill the Rights" instead of "the Bill of Rights"? Is this the glaringly obvious typo it appears to be or am I missing something?

This is the actual headline of the article, not just the HN headline.</comment>


Secrets Behind Managing Techies - <AUTHOR> http://www.computerworld.com/s/article/300537/Career_Watch

<comment><USER2> Wow, very good, not the usual BS at all.

Read it to find out why having HR send out a memo that everyone wear shoes was _an entirely good thing_ ^_^.</comment>


Kim Dotcom: "I'm broke" (german article) - <AUTHOR> https://translate.google.com/translate?sl=auto&tl=en&js=y&prev=_t&hl=en&ie=UTF-8&u=http%3A%2F%2Fwww.heise.de%2Fnewsticker%2Fmeldung%2FKim-Dotcom-Ich-bin-pleite-2465551.html&edit-text= <comment><USER2> Kim Dotcom is in jail because he thumbed his nose at the US entertainment/enforcement cartel.

It's absurd that his file sharing service (which functionally is no different than Dropbox or Google Drive) gets shut down, but start-ups like Uber and AirBnb - which actually endanger people's lives and cause very real damage to property, are allowed to continue operating because their VCs are politically connected.

I used to be in the restaurant/hospitality business - you know how awesome it would be if I could just set up a restaurant, hotel or yes BnB in my house without going through the proper permits? In the part of Canada I live in, a BnB requires no fewer than 7 permits (3 municipal, 3 provincial, and 1 federal).

Edit - if you're in Canada, here's a useful tool to search for some of the permits and licenses your future business will require: [http://www.canadabusiness.ca/eng/search/stp1](http://www.canadabusiness.ca/eng/search/stp1)

Here you have 2 US-based businesses thumbing their nose at local laws, as well as the laws of other countries, however because they're connected (and there's a lot of money invested) they get away with it (in the US anyway, and they try their luck elsewhere).

To say the Kim Dotcom debacle is anything but political is absurd...

<comment><USER3> The charge against Megaupload was not that they hosted copyrighted content but that they knowingly did so and encouraged it through their affiliate program.

<comment><USER4> Youtube actively does the same thing with people directly making money using 'pirated' content.

Note: They currently remove specific content when requested and even automate the process. But, you can still find plenty of 'pirated' content where the owner has yet to complain.

<comment><USER3> In what way? This was true when YouTube first launched (I've heard stories that the founders uploaded copyrighted movies and tv shows themselves) but YouTube has a system in place now that removes copyrighted content automatically and it works pretty well (some say too well).

<comment><USER5> Type in the search terms "full album" and then tell me their automatic removal of copyrighted material is working pretty well.

<comment><USER6> "copyrighted" != "cannot be shown"

copyrighted means that the owner of the copyright is the only one with the right to decide when/where/how it gets shown. But if that owner is cool with it being on Youtube, there's no reason for it to be taken down.

</comment></comment></comment></comment></comment></comment><comment><USER7> Tweet from 9hrs ago.

>I guess I'm going back to court soon to get some of my assets unfrozen for legal fees and living expenses. To be continued...

He still have a bunch of assets that are outside his control.

He also tweeted that his rent is paid until halfway to next year.

His wife is suing him for 20+ million.

Dude still has money, he just can't get access to large amounts of it to stir stuff up.

Also, I imagine there is quite a bit this guy can do once he is able to move his focus away from fighting huge organizations (including the U.S. gov) to stay out of jail.

<comment><USER8> > His wife is suing him for 20+ million.

Talk about adding insult to injury.

<comment><USER9> Actually it could be a smart way of getting money for his wife and kids while they appear to be apart. When one gets dragged into courts its often advantageous to endup with whoever wants money from you to be at the end of a long list of people you owe money to.

<comment><USER8> Interesting strategy, you could be right.

<comment><USER10> It's more than possible. The article I read last night said his wife is full owner of Mega (the follow up to Megaupload).

</comment></comment></comment></comment></comment><comment><USER11> Everytime I see a headline about Kim, I wonder why its interesting to people. Is it the equivalent of internet rubber-necking? He did some bad things, made a lot of money, then profited from redistribution of copyrighted material - why is he internet famous?

<comment><USER12> Ok, controversial opinion here: he made downloading/sharing files easy, Dropbox-style. Had his "startup" been in Silicon Valley, things would have turned out _very_ differently; starting with less emphasis on copyrighted material piracy(maybe), but also more political weight (hello, VC connections), and probably not a raid at its house while its US competitors are safe.

<comment><USER13> Had his "startup" been in Sillicon Valley, he had been in jail a long time ago. I don't see how there would have been less emphasis on copyrighted material, as that is and has always been the focus of megaupload and mega.

What US competitors are safe? You mean I can easily share copyrighted movies with strangers on dropbox? I've never seen anyone do that.

<comment><USER14> > Had his "startup" been in Sillicon Valley, he had been in jail a long time > ago.

I'm hoping the founders of Napster get early release from jail. They've done their time, surely.

<comment><USER13> "Napster agreed to pay music creators and copyright owners a $26 million settlement for past, unauthorized uses of music, as well as an advance against future licensing royalties of $10 million"

Have you ever seen Kim engage in talks for a settlement? Would it even make sense for the industry to settle at this point? You think Sean Parker would still be a free man if he would just continued running Napster as it was?

(The Dread Pirate Roberts was also running an innovative business where users could exchange government restricted goods through a P2P system with a centralized registry, look where that got him)

</comment></comment></comment></comment></comment><comment><USER15> with every dotcom headline i'm thinking back of his days during the '99 boom where he was a "prominent" figure in german television with his hacking skills and millionaire-lifestyle which was all just for show.

he's only really clever at self-promotion and this is just the latest chapter, either to hide money from governments/plaintiffs, his ex-wife or swindle some of his followers out of theirs.

can we not just ignore this guy finally; he's not some kind of internet robin hood / freedom fighter.

<comment><USER16> Seems to me to be more Mark Karpeles than Edward Snowden.

</comment></comment><comment><USER17> The original video where he talks about it might be more interesting than the translated article:

[https://www.youtube.com/watch?v=68-PYA7uuGI](https://www.youtube.com/watch?v=68-PYA7uuGI)

<comment><USER18> I'm seeing a re-occurring theme of big names talking about how much lawyers make. I guess you never really fully fathom the expense until you get really involved with a legal proceeding.

</comment></comment><comment><USER19> He is not broke He hid his money, including giving control of Mega to his wife which is still worth millions.

<comment><AUTHOR> According to them, they split. Shortly before, she was stripped of company control. Not sure how far she still is involved though.

[http://tvnz.co.nz/national-news/kim-dotcom-s-wife-left- compa...](http://tvnz.co.nz/national-news/kim-dotcom-s-wife-left-companies- ahead-split-5975515)

</comment></comment><comment><USER20> for anyone interested a doc done by Vice interviewing Kim.

[https://www.youtube.com/watch?v=gMxhIfG0MpY](https://www.youtube.com/watch?v=gMxhIfG0MpY)

</comment><comment><USER21> No! End of an era

</comment><comment><USER22> Could Kimble be the Fluffy Bunny?</comment>


Show HN: The Gaming Pub – A curated gaming content newsletter - <AUTHOR> https://www.thegamingpub.com/ <comment><AUTHOR> Hey guys,

I'm the guy responsible for The Gaming Pub newsletter, a weekly hand-curated gaming content newsletter. It's heavily inspired by the Hacker News newsletter.

If you like to play games, it's a good source to know what's happening in the gaming world without having to read multiple sources. Every Friday I sent a summary with all the most important news, discussions, videos and features of the week.

If you like to see some older issues, you can take a look here [https://www.thegamingpub.com/past-issues/](https://www.thegamingpub.com/past- issues/)

Any feedback is appreciated. Thanks!</comment>


Good New for Solidity Developer - <AUTHOR> It is new development environment framework for Etheruem smart contract. check it out here https://github.com/cobraframework/cobra . Feel free to open an issue if you find a problem, or a pull request if you've solved an issue. Thanks! <comment><USER2> How did this get to the top of Ask HN with 2 points in 24 minutes? What is going on here?

<comment><USER3> Just fewer users than usual online these summer days and so a reduced number of upvotes is enough to push posts to the top.

</comment></comment><comment><USER4> What are the advantages of Cobra framework compared to Brownie ([https://github.com/iamdefinitelyahuman/brownie](https://github.com/iamdefinitelyahuman/brownie))?</comment>


Show HN: A simple way to browse the most popular products on Amazon - <AUTHOR> https://www.bestmeta.com/ <comment><AUTHOR> Hi there, I'm the creator of this webapp.

I made BestMeta because I like to buy the best of something when I pony up the money.

I'm sure that makes me frugal or cheap - but I just think it's smart to buy something that's been highly reviewed or a favorite for a category. I hate buying something and then finding out that I should have just got "The other version that doesn't have X issue". I love reading reviews and finding the best of the best.

I got tired of scanning for the small yellow "Best Seller" badge when searching on Amazon - so I figured - what if I could only see the best selling/most popular items during my browsing?

Being a full stack dev - that's what I did. I signed up for API access (which is its own story ha) and got cracking. The result is BestMeta.

Let me know if you have any feedback if possible. Thank you.</comment>


While True: learn() – Simulator of a machine learning specialist - <AUTHOR> https://luden.io/wtl/ <comment><AUTHOR> I found a cool game that teaches you through the history of machine learning techniques via visual programming.</comment>