Ask HN: What's the largest amount of bad code you have ever seen work? - <AUTHOR> I think I've broken my own record with this one ~2500 lines of incoherent JavaScript/C#. Works though. <comment><USER5> I am maintaining one application in construction industry space. That application was created 25 years ago by construction worker that never wrote single line of code before, but because he caused a lot of problems on construction site they give him Programming 101 book and let him build it.

15 years later the app was close to half milion lines long of huge bowl of spaghetti code. Only comments in whole codebase were timestamps. I don't know why he dated his code, but I find it fascinating: he never deleted basically anything so you can find different timeframes of when he discovered various concepts. There is use-exception-instead-of-if period, there is time when he discovered design patterns, there is time before he learnt SQL so all the database queries was done by iterating over whole tables and such. I am sure I will find commented Hello world somewhere in the code someday.

I am working on this codebase for 10 years. Code quality improved and major issues get fixed, but there is not enough budget to actually rewrote whole system, so after all it is more or less huge spaghetti monster and I get used to it.

<comment><USER6> I have to salute this construction worker for building a solution that is apparently so valuable for the business that they can't simply replace or rewrite it. This probably means that it solves a real problem for them, and adding 30.000 lines of code per year without any formal training or much tooling is no small feat either. I understand the criticisms and laughs here from the "real" software developers, but damn it's just impressive what people can create on their own given enough time and motivation.

<comment><USER5> It is impressive indeed. The coding started just in time when Windows 95 were released. There was no Stackoverflow and they don't even really have internet back then. The programmer (as far as I know) didn't even speak English so he has access to book or two in German language and code snipets in help section of Delphi. At the same time creating applications with UI just started, so there was very little experience available, espcially in rural Austria.

Company did tried to migrate to other software few times, but the software is just too specific for given industry and legislation of small country that the companies who tried to create similar software usually went bankrupt soon.

<comment><USER7> Rural Austria? Please provide a company name ... or at least first and last character :)

</comment></comment></comment></comment><comment><USER8> Years ago as an intern at Microsoft, I had code go into the Excel, PowerPoint, Word, Outlook, and shared Office code.

Excel is an incomprehensible maze of #defines and macros, PowerPoint is a Golden Temple of overly-object-oriented insanity, and Word is just so old and brittle you'd expect it to turn to dust by committing. There are "don't touch this!"-like messages left near the main loop _since roughly 1990_.

I had a bug in some IME code causing a popup to draw behind a window occasionally, and a Windows guru had to come in to figure it out by remote- debugging the windows draw code with no symbols.

I learned there that people can make enormous and powerful castles from, well, shit.

<comment><USER9> "Don't touch this" around the main loop can mean being able to make promises about responsiveness, reliability, etc.

Frequently there are critical code sections where it is much easier to tell people "don't touch it" rather than training people how to work on it safely.

<comment><USER10> When that is the case, would it not also be a really good place to explain why not, or provide a link to the place where such an explanation is provided?

<comment><USER11> In reality there often isn't time.

Getting it done > Getting it done properly as far a management is concerned.

<comment><USER12> Ever had this discussion with a coworker?

Coworker: "I hadn't enough time to do it right"

You: "Given enough time, how would you do it differently?"

Coworker: "............" (crickets)

IMHO it's not related to deadlines only ; the "not enough time" argument is often a comfortable fallacy, keeping us from facing the limits of our current skills.

I found it to be especially true with testing. I've lost the count of how many times I heard "we didn't had time to write (more) tests". But testing is hard. And when given enough time, these developers don't magically start doing it "right" overnight.

<comment><USER13> >I found it to be especially true with testing. I've lost the count of how many times I heard "we didn't had time to write (more) tests". But testing is hard. And when given enough time, these developers don't magically start doing it "right" overnight.

Bingo. It's not necessarily only skills though. It can be myriad reasons and "no time" is just the easiest excuse they can think of. In big companies I've often seen the company process prescribing such bad tools that a good TDD testing strategy is impossible to do with those tools, but they won't move away from them, because somebody in purchasing already bought 10,000 licenses for this bad tool (which is often just a bad GUI, which doesn't really help, except for selling the thing).

The worst tool was a bad GUI where you couldn't even define functions you want to use, that had slow (>1h), non-deterministic, test execution, for a unit test.

<comment><USER11> To even write unit tests effectively you need to write your code in a certain differently.

In C# this normally means using IOC + DI.

Also almost nobody I know does proper TDD. I know it is very convincing when one of the TDD evangelists shows you how to write something like a method to work out the nth number in a fibonacci sequence using nothing but just writing tests.

In reality most 95% of developers that even write tests write the logic and write the test afterwards to check the logic does what it should.

<comment><USER13> >To even write unit tests effectively you need to write your code in a certain differently.

>In C# this normally means using IOC + DI.

I've become quite partial to functional programming in the last few years. Side effect free functions with functions as interfaces for DI lend themselves perfectly to TDD and data parallel async without worrying too much.

C# is now slowly taking over most of the good features from F#, but I think the culture won't transform so easily.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER14> The worst program I ever worked on was something I was asked to maintain once. It consisted of two parts. The first was a web application writen in ASP. The second portion was essentially Microsoft Reporting Services implemented in 80,000 lines of VB.NET.

The first thing I did was chuck it into VS2010 and run some code metrics on it. The results were, 10 or so Methods had 2000+ lines of code. The maintainability index was 0 (number between 0 and 100 where 0 is unmaintainable). The worst function had a cyclomatic complexity of 2700 (the worst I have ever seen on a function before was 750 odd). It was full of nested in-line dynamic SQL all of which referred to tables with 100+ columns, which had helpful names like sdf_324. There were about 5000 stored procedures of which most were 90% similar to other ones with a similar naming scheme. There were no foreign key constraints in the database. Every query including updates, inserts and deletes used NOLOCK (so no data integrity). It all lived in a single 80,000 line file, which crashed VS every time you tried to do a simple edit.

I essentially told my boss I would quit over it as there was no way I could support it without other aspects of work suffering. Thankfully it was put in the too hard basket and nobody else had to endure my pain. I ended up peer reviewing the changes the guy made some time later and a single column update touched in the order of 500 lines of code.

There was one interesting thing I found with it however, there was so much repeated/nested if code in methods you could hold down page down and it would look like the page was moving the other way, similar to how a wheel on TV looks like its spinning the other way.

<comment><USER15> When I do code necromancy, instead of understanding the code, I try to understand what it does, how it interacts with the world, and then recreate that behavior.

Meaning I capture all the I/O and recreate it. SQL, HTML, PDF, CSV, whatever. Serialize everything, before and after. And then do diffs on the outputs to see if new code reproduces expected behavior.

Much easier than dead code removal, code deduping, incremental changes, backfilling tests, etc.

Once someone captures, documents what the code is actually doing, the real refactoring begins. Removing unnecessary queries. Grooming data. Simplifying schemas. Aligning the app with the biz. Etc.

<comment><USER16> This is often the only sane thing to do.

It's my default approach with thorny scientific code: get everyone to agree on what output should be for a bunch of relevant inputs, get the original systems output, hash it, and then write a bunch of tests in a new project that all assert that these inputs produce outputs whose hashes are as follows..

then never look at the guts of the horror again.

</comment></comment></comment><comment><USER17> At my first gig I teamed up with a guy responsible for a gigantic monolith written in Lua. Originally, the project started as a little script running in Nginx. Over the course of several years, it organically grew to epic proportions, by consuming and replacing every piece of software that it interfaced with - including Nginx.

There were two ingredients in the recipe for disaster. The first is that Lua comes "batteries excluded": the standard library is minimalist and the community and set of available packages out there is small. That's typically not an issue, as long as one uses Lua in the intended way: small scripts that extend existing programs with custom user logic (e.g. Nginx, Vim, World of Warcraft). The second is that Lua is a dynamic language: it's dynamically typed, and practically everything can be overridden, monkey patched and hacked, down to the fundamental iterators that allow you to traverse data structures.

This was the playground for the guy to create his own reality.

Lacking a serious standard library, he crafted his own. Where a normal world e.g. file rename function would either do the job or return a error to the caller, he chose a different approach. Functions were autonomous highly intelligent pieces of code that tried to resolve every possible problem, entangled with external logic, so grokking the behaviour of the most fundamental things was challenging - let alone understanding fragments of code composed of library calls.

Lacking a OO model in Lua, he built his own. I can spend a lot of time describing with what was wrong with it, but it suffices to say that each object had SIX different 'self' or 'this' pointers, each with slightly different semantics. And highly entangled with external unrelated logic of course.

I'll save the stories about the scheduler and time series database he built for another time.

<comment><USER18> I've seen personal reality building happen in Lua several times already. It's very seductive to intelligent solo artists who are given a lot of freedom.

To be fair, I've also seen it happen in C, C++, and JavaScript.

<comment><USER19> "personal reality building" is the greatest quote in this whole thread, it's perfect. Will be using.

</comment></comment></comment><comment><USER16> Behold, academia.

The only maintainers of this code, ever, have been grad students and postdocs. I estimate there have been about 12-15 generations worth. This code has supported hundreds of publications in its lifespan.

A codebase that began life in 1987, in C. First ported to matlab in 1999. First source control was added (as SVN) in 2015. Between 2015 and 2018, there were 6 commits total, yet 3 people graduated out of the lab from it. Probably 100,000 loc total, of which I estimate maybe a third is ever used. 1400-line matlab functions are normal-ish. I've found loops nested 11 levels deep.

It's a series of psychophysical experiments. Each experiment exists in at least 4 different versions side by side in source, each named slightly different, often by incorrect datestamp of last modification. Version control across machines is not well maintained, so you have to diff everything before you can copy or move files lest you accidentally blow something away completely.

Oh, and it's mexed and wrapped for use on a mac on exactly one snow leopard machine, hardware from 2007.

edit: I think this counts as a job, not a student experience, because I am not a student. I just have to clean this mess up once in a while.

<comment><USER20> I think it is a problem in general with code for experiments. You just need to change a tiny bit for new experiment and you don't want to ruin earlier experiment.

<comment><USER21> There's a very simple solution: guard your tiny bit of code with e.g. a command line flag that defaults to "off", and commit that.

It's a little more work sometimes, especially when your experiment changes things structurally, but it pays off over and over.

<comment><USER22> Isn't this what git branches are for?

<comment><USER23> No!

There's a great article somewhere about how the normal version control flow doesn't really work for this style of computing.

You want to keep both "versions" of code live and active in the same place at the same time (often in the same notebook).

People end up with methods named methodName, methodName2 etc, which isn't very good. But once you see the workflow you understand why normal version control doesn't work either.

There should be a solution to this, but AFAIK there isn't yet.

</comment></comment></comment></comment></comment><comment><USER24> A customer-facing dashboard. Yes, a dashboard. How bad can a dashboard be bad you ask? Well, for one, the dashboard had tabs, and each tab was a separate webapp hosted on a separate server. And each team was responsible for developing and maintaining the webapp that their team was incharge of (i.e the User team in charge of Users webapp, Feature1 team incharge of Feature1 webapp).

Now add on the fact that different teams had varying levels of frontend competence. This led to some webapps being in (badly written) React, some in Angular, some in JQuery and one in Angular2 as well. Some were java API backed, some were NodeJS backed and one used Ruby in the backend. Oh yeah, and each had different datastores as well.

Now alongside this, there's no central auth framework, so each webapp had their own way of how to determine user auth (there was a shared cookie thankfully on the *.company.com domain), so there were 6-7 possible login and logout pages as well.

When the company had a brand redesign and needed all customer facing stuff to re-align with new design, we had to literally re-design 6 dashboards which have used different paradigms and different tech stacks.

To this date, the dashboard still exists and is used by customers (Main dashboard for a company valued at $500M+) and the most common issues are issues related to Auth (eg. random logouts when switching tabs), data inconsistency (They have crons which update data from one DB to another, but it's not immediate) and an inconsistent design and UI behaviour (since JS is also different for each app) which pisses off many users.

Till date, I'm not sure who signed off on this pointless dashboard design.

<comment><USER25> To some extent this sounds like Spotify. I have been told it's basically Iframes stitched together, and each frame is owned by a team [0] , I do think Spotify has some better auth communication though.

0: [https://www.quora.com/How-is-JavaScript-used-within-the- Spot...](https://www.quora.com/How-is-JavaScript-used-within-the-Spotify- desktop-application-Is-it-packaged-up-and-run-locally-only-retrieving-the- assets-as-and-when-needed-What-JavaScript-VM-is-used)

<comment><USER26> Aha! This is the Spotify model!

</comment></comment></comment><comment><USER27> We have absolutely no idea how to write code. I always wonder if it's like this for other branches of engineering too? I wonder if engineers who designed my elevator or airplane had "ok it's very surprising that it's working, let's not touch this" moments. Or chemical engineers synthesize medicines in way nobody but a rockstar guru understands but everyone changes all the time. I wonder if my cellphone is made by machines designed in early 1990s because nobody was able to figure out what that one cog is doing.

Software is a mess. I've seen some freakishly smart people capable of solving very hard problems writing code that literally changes the world at this very moment. But the code itself is, well, a castle of shit. Why? Is it because our tools (programming languages, compilers etc) are still stone age technology? Is it because software is inherently a harder problem than say machines or chemical processes for the human brain? Is it because software engineers are less educated than other engineers? .....?

<comment><USER28> It's like that in other fields of engineering too, _when they are making something they haven 't built before_. That's the essential part: for example, a lot of construction is really just rebuilding the same thing that's already been built 100,000 times in the past 100 years.

When they attempt to build something new, it often ends up like software – tremendous overruns in both cost and schedule.

C.f. the only new nuclear power plant being built in the West, which is $4 billion over budget and ten years late: [https://en.wikipedia.org/wiki/Olkiluoto_Nuclear_Power_Plant#...](https://en.wikipedia.org/wiki/Olkiluoto_Nuclear_Power_Plant#Unit_3)

The reason why this feels more commonplace in software is that we're usually designing something new. Software has essentially no reproduction costs, so there's no reason for anybody to design software that's a carbon copy of something you can already download or buy off-the-shelf. That's not the case in engineering of physical products or works. New buildings are needed all the time, even if they're performing exactly the same function as the building in the adjacent lot.

<comment><USER29> > It's like that in other fields of engineering too, when they are making > something they haven't built before.

Exactly this. I like to use an analogy of building bridges :)

      - OK, we have this valley and we want to drive cars over it. What do we
        do?

      - Hmm, we could just make the road along the bottom of the valley

      - Wouldn't that cause the cars to fall because of the steep angle?

      - Good point. Maybe if we built the road with a lot of twist and turns

      - Then it's a slow and long drive.

      - Maybe we could build some sort of catapult setup to throw the cars
        across the valley

      - That would make safety a concern though. Is there a way we could use
        helicopter rotors to suspend a road in mid air over the valley?

      - Or what if we attach the road to each side of the valley and make a
        road that's strong enough to not crack in the middle under its own weight?

      - Yeah, that sounds like a good idea. How do we make sure it can hold its
        own weight, though?

      - .....etc

<comment><USER30> > Maybe we could build some sort of catapult setup to throw the cars across > the valley

This was played for laughs by German satire news website "Der Postillon" (like The Onion, but in German): "Department of Traffic to replace ramshackle bridges by jumping hills"

Nicely photoshopped picture at: [https://www.der-postillon.com/2014/11/lander- ersetzen-marode...](https://www.der-postillon.com/2014/11/lander-ersetzen- marode-brucken-durch.html)

</comment></comment></comment></comment><comment><USER31> At one stage a company I worked for was considering licensing the code for a school time-tabling application, rather than paying the company to do the (fairly minor) changes we required to meet (non-USA) state requirements. The company was started by a couple of teachers, the same people who wrote their product. It was 10s of ks of Pascal code, but with not a single variable name or function name that made any sense; everything was A, AA, AA1, X, XX, XX2 etc. I spent a few days looking at it, then recommended we keep paying the somewhat steep cost for the modifications. Then at least if anything broke it was on them to fix it.

Incidentally we had a small falling out with this company, and they were refusing to update their executable until this issue was resolved. This looked like affecting some hundreds of schools and their timetables. I did some checking, and it turned out their 'non-updated' executable was doing a simple date check on the local PC; if it was past a certain date, the executable refused to run. So I did a quick hack in our application that involved: \- setting the local PC date to prior to the 'cutoff date' \- running their executable with the required parameters, and grabbing the results \- setting the local PC date back correctly

This led to interesting negotiations as they were puzzled why their 'gun to our heads' no longer appeared to be working, and things were resolved to the benefits of both parties soon after.

<comment><USER7> Code obfuscation + blackmail negotiations ... awesome company to do business with.

</comment></comment><comment><USER32> I took over a Perl project where every SQL call was an exec to a java program which would make the query.

The largest madness was a J2EE mess where persistence was achieved by taking your current object, sending a message bean to the server, it would touch the database and return the result which was being polled for (making it synchronous). The amazing thing is that the client and the server were the same J2EE instance. So Class A could have just called Class B. Instead it was A -> turn it into a message bean -> send it to the "server" (same machine) -> unwrap it into A again -> transform it into B -> message bean it back to client -> unwrap into B.

Literally three months of 8 people ripping all of that out and replacing it with A.transform() // returns B

Oh, and at the same time, none of this was in source control. It was being worked on by 12 people up until that point. They didn't know who had the current files which were in production. So my first job was taking everyone's files and seeing which ones would compile into the classes being used in production, then checking those into source control. Up until then, they just kept mental track of who is working on which files.

<comment><USER33> Was there a reason they did not use DBI to call the code ?

<comment><USER32> I had been told he bragged about it being "job security" since nobody could understand what he had written. I never spoke to him directly as I was his replacement...

The java mess was a pattern copied from within the company where it was used correctly. The main product of the company was built around some very complex scheduling software. It was a black box and communication was entirely through inter-server beans. This was copied to intra-server, which makes no sense at all, and it was used everywhere.

<comment><USER34> "job security" coupled with "I was his replacement" says that his plan worked about as well as his code...

<comment><USER32> Indeed. It was quite the entry to the company for me as well. "What are you working on?" "Fixing Dave's code." "Good luck."

But my most auspicious entry to a company was where the guy I was replacing had been arrested for stealing data from a competitor. My first two days were spent recovering logs, downloads and other traces of what he had done and handing it over to the lawyers. (the circumstances were not mentioned in the hiring process)

</comment></comment></comment></comment></comment><comment><USER35> My first job was sysadmin of a third tier ISP back in the dialup days. The account management and provisioning system that ran EVERYTHING was probably close to 100k lines of csh. Everything was done via a UI that the shell generated as a sort of curses-style interface.

What was horrible about it was that it controlled everything from who got a website, active domains, what POPs users could dial into, metered billing, you name it. And it did all of this by manipulating flat files of pipe-delimited data on a central server, then rcp'ing those files to the various machines, then rsh'ing to the various machines and kicking off THEIR scripts, which parsed the source files and generated their own files, which called another set of scripts that parsed THOSE files and generated the software config files.

This included doing things like updating init scripts so that new IPs got added to interfaces, and what email server a user was provisioned on, so it had to generate new exim configr with routing rules.

All this to say that it all worked, but I dreaded having to go in to manipulate anything. Adding a server at least had a dedicated procedure so that was fine, but anything else was a nightmare.

Case in point - as part of a gradual plan to remove this nightmare, I swapped out the radius server that they were using for one that could support a database backend, and modified the local config generator script to make a new config for the new software as a stopgap until I could get it into a database.

The config file had a series of fields that just had numbers in them, and after much digging, it seemed like that controlled whether a terminal dial in user was presented with a menu of options, and what options. I had to reimplement that logic for the new software, made a mistake, and accidentally removed the option for UUCP for the 10 customers that were still using UUCP. One of them was on an ISDN line and their mailer decided to continuously redial looking for the UUCP, tacking up thousands of dollars in carrier rate charges for the weekend that it took anyone to notice something was broken.

<comment><USER36> Wow this is legendary. I'd love to direct a short film or tv series that revolves around an IT/software team using a massive csh codebase like this. I'd love to generate some training montage / diagram sequence shots of the system being built by the characters maybe make some cool blender / adobe premiere overlay screen splits of the high level architecture as the team references certain aspects of the system

<comment><USER37> Make it like the IT crowd, but for a software team. That would be golden.

<comment><USER38> I'd take something less fictionally dramatic and more along the lines of reality TV (ala home makeovers/Kitchen Nightmares/Bar Rescue): a team of crack engineers untangling the mess and laying out the best practices for future development. The concept is even ripe for booze sponsorship.

</comment></comment></comment></comment><comment><USER39> The project around 10 million lines of code. Some of it was written in a very specific version of Fortran that was a PITA to compile. One fun experience was opening a file and seeing it was created on my birthday. Not in the sense just matching the day and month of my birtdhay. The code was literally written on the day I was born.

<comment><USER40> I recall rewriting a several thousand line program that was written in a preprocessing language that compiled to fortran that was homerolled by some random guy that was just so ahead of the game at the time that he basically wrote an extension of fortran with all the macros he made and rolled into one thing.

The rewrite was 100 lines long.

</comment></comment><comment><USER41> Ten years ago I was called in to remediate a new web application which had been subcontracted to an Indian development company. The PHP developers who'd put it together evidently didn't know about classes, and each page in the application was hundreds, sometimes thousands, of lines of spaghetti code, most containing the same duplicated (but subtly changed) blocks providing database connectivity etc. Security had not been a concern either; passwords and credit card details were stored unencrypted in database text fields.

I was called in because, while most of the application worked, some of the requested features were not yet complete. When I made my initial recommendation (scrap the whole thing and start again) I was told the client's board would not agree to that because of the money already invested and the fact that the board had seen a demonstration proving that "most of it worked".

It took two developers eighteen months to beat this sorry mess into a maintainable state while ensuring it remained "usable". It would have taken one third of that time to rewrite it from scratch.

<comment><USER42> 10 years ago PHP didn't have classes

<comment><USER43> PHP 5 was released 14 years ago in July, 2004. One of its new features was OOP. Classes were supported at that time. In fact, a comment from 14 years ago shows an example class [1].

[1]: [http://php.net/manual/en/language.oop5.php#46290](http://php.net/manual/en/language.oop5.php#46290)

</comment></comment></comment><comment><USER44> I use to work on hospital lab software.

* It was over 20 years old by the time I started

* It was written in Fortran

* Variable names were single and double digits

* Each fortran program would run in isolation but had a shared memory process

* It was formally a terminal program but a weird Java frontend was created so everything looked like Windows GUI

* All program names were four letter acronyms

* All data was stored in fixed width binary "flat" files

* It previously was under CVS version control, but each install slowly drifted apart, so each site had it own unique features and bugs.

* I once had to move a significant feature from one install to another using only patch files generated from the work done on the original install.

</comment><comment><USER45> The worst for me was a rescue project: a site for a US tech sector Public- private partnership. Nothing too complicated: recurring donations, paid events, a small members area. They had sent it to an Indian firm to build in Drupal 7 - not a lightweight system to begin with.

I would like to say "cue the stereotypes for Indian developers" and we could all have a good laugh. But no. This is more like Heart of Darkness. They must have traveled to the darkest corners of the subcontinent to find a mind capable of the eldritch horrors we found there. We started keeping a wiki of design patterns, to save us WTF time. Here are a few.

* Memory management. What's that? This site required 16 GIGABYTES as a PHP memory limit in order to load the front page for an anonymous user.

* Security. What's that? Part of the reason it required so much memory, is that it would include a complete debug log of the site's entire transaction history, including PII and credit card numbers, in every session. Meaning any vulnerability was a critical vulnerability.

* They would arbitrarily include the entirety of Zend framework to access a single simple function. This happened several places in the codebase, with different versions of Zend committed.

* Can't reach the ERP to get the price for an event? Let's set it to $999,999.00 and proceed with the transaction.

* Invoice numbers were random numbers between 1-1000. Clashing numbers meant a fatal exception that would fail to store the invoice or payment record... But not until after payment had been processed. Birthday paradox means this happened a lot.

* The developers used arcane bits of Drupal API to do totally mundane things. Like, if you know about hook_page_alter, you know there's a setting in the UI for the frontpage. But we'll just use hook_page_alter instead.

* Write queries using Drupal Views, rewrite the query in code, override the views query with their custom (identical) version using an unusual API hook, just to add a sort.

I could go on, but I think you get the picture. Eldritch horror of a codebase.

<comment><USER46> Holy crap. You already earned yourself an upvote for this sentence:

> * Memory management. What's that? This site required 16 GIGABYTES as a PHP > memory limit in order to load the front page for an anonymous user.

<comment><USER47> I'm tempted to log out and created 5 new accounts to upvote for each bullet point :-D

</comment></comment></comment><comment><USER48> I used to own a C++ application that was a morass of abstractions and indirections so it was impossible to reason about. It took a number of hours to compile.

On one infamous occasion we were making a, relatively small, patch release. The debug version worked fine but the release version crashed systematically. Even when we backed out _all_ the changes we had the same behaviour. We were screwed.

Until one of the team had a bright idea. She stripped strings from the debug build and tested it. To our surprise it not only worked, it was only slightly bigger than the previous release version and it was also slightly faster! We shipped.

This experience was the trigger to make me go all-in on a full re-write that I had been contemplating. One of only a couple of times in my career that I've made that decision on a major piece of software.

The re-write was a huge success. It was also about 10% of the original in terms of LoC. The day our testing finished, we held a ceremony where we deleted all the old code from the current version.

This caused a slightly different issue. At the time, code metrics were starting to get fashionable but LoC wasn't yet the pariah it became.

So, a couple of days later I got a concerned call from the metrics guys. Apparently, we had deleted more code than all the other teams combined had added in the previous measurement period. This caused their metric calculation to barf. Their solution? We should add all the code back in! This led to a somewhat heated argument that ended up with me persuading them that deleting code was good and they should, at least, abs(LoC) it. It didn't make the metrics any more useful but meant that we had an application we could reason about. Happy days.

<comment><USER49> > Apparently, we had deleted more code than all the other teams combined had > added in the previous measurement period. This caused their metric > calculation to barf. Their solution? We should add all the code back in!

ah yes, when bad metrics become targets

</comment></comment><comment><USER50> From 10+ years ago when I worked at PayPay, webscr (the frontend at the time, where you'd log in) was a total of 2GB of C++ that would get compiled into a 1GB CGI executable, deployed over some ~700+ web servers.

Debug versions would never get compiled, as I'm told the resulting file was too large for the filesystem to handle.

Apparently a great deal of the code was actually inline XML.

They _knew_ this was a bad pile of technical debt, too: at one point, a senior/staff engineer gave a company presentation where they brought a fanfold printout of the main header file for this monstrosity, and literally unrolled it across the entire stage.

<comment><USER51> I think you meant to say PayPal?

<comment><USER50> Oh whoops, thank you, yes!

</comment></comment></comment><comment><USER11> I've worked on a CMS that was partially done in .NET, Iron Python and used an XSLT templating system to generate HTML for the front end.

The architecture looked like something from the Early Java days.

The system used Iron Python / C# in the following way.

1\. A web request would hit the CMS 2\. There was a massive switch statement to work out how the query would be rewritten 3\. If it the url was prefixed with processor it would attempt to find the processor in the db. 4\. The code would then find the python script associated with the processor. 5\. The processor would then spin up a command line instance in a hidden command window on windows server. 6\. The processor would have to return XML that had to be built up using strings (not element tree for you). 7\. This would return the XML to the C# which would then try to render into the XSL Transform.

If at any time this failed. Silent failure. There was no way to debug easily (There was a magic set of flags that had to be set in Visual Studio or wise you couldn't debug the python scripts).

To get the software to build on a new machine. It took a contract 4 months to reverse engineer an XP machine. None of it was documented anywhere.

It used ImageMagick to generate thumbnails on the fly which doesn't work to well with windows server.

The lead engineer was an alcoholic. He used to go to the local pub for 4 hours in the middle of the day and come back smelling like a brewery.

<comment><USER52> I actually laughed while reading this. I've encountered something eerily similar, but it was a Ruby app. Almost exactly your steps 1-3, however, what was returned from the db were ruby method names, which were then invoking other pieces of the code through some deep, twisted ruby magic.

Uncovering what the hell was actually happening was like peering into the mind of a psychopath.

Once myself and 3 other engineers spent 4.5 hours trying to figure out how to send an email from this treacherous app (a feature which had stopped working months before we showed up). After those 4.5 hours, none of us even came close.

To top the whole thing off, page load times were in the minutes. The standard joke from the users was that they "get to take lots of coffee breaks."

Much to the (new) manager's credit (and my sanity), we got buy in to just build a new version and let the old one die. I left the team shortly after the 2.0 beta launch, but God I wish I would have stuck around a little longer so I could have seen the official end of life for that calamity of an app.

<comment><USER11> In a weird way I kinda like these totally mental systems. It really ups your skill level for debugging if nothing else.

I've worked with lots of proprietary CMS systems and I kinda got into a groove with working with them. I knew exactly how to manipulate the system within the parameters of said system.

I kinda found it challenging. When a system is well built. I find it boring.

</comment></comment></comment><comment><USER53> I wanted to tackle a rewrite of a legacy LISP program, used by AutoCAD, to take in a list of 3D coordinate points, and output a DWG of a 2D projection (from which to create a jig on which to assemble a prototype). Cool stuff, right? My intent was to create a "proper" script in AutoCAD to actually create 3D objects from the input data, and let the program do the projection to a 2D drawing. (I'm still not sure if that would have been possible; I never got that far.)

I had never written in LISP before, so I bought a book!

I was having trouble reading the program on the computer, so I printed out the roughly 15,000 lines. Not a lot, I know, but it was about an inch and a half thick stack of paper. I started going through it.

It consisted of LOTS of subroutines. Thousands. Each one neatly formed; no more than a couple hundred lines. It gave me hope.

It read in the text file, created a blob of a string, and then passed that blob to the first subroutine. Then it passed to the next subroutine. And then the next, and so on. As far as I could tell, it never called a subroutine a second time, and it never returned to the starting method. Given the strangeness of LISP, I couldn't figure out what it was doing, or why.

The guy who wrote it had retired, and we didn't get along anyway, so I didn't try to chase down what his thinking was.

I gave up.

To my knowledge, they're still using that program 17 years later.

</comment><comment> le-mark Bad is not a good measure. But taking 'bad' to mean large, convoluted, zero documentation; I was once at a financial services company that had a 25 million+ LOC mainframe cobol application that had been under active development since 1969. This was a batch and CICs system. It was spaghetti on every level; database (db2, vsam, isam), the screens of the app, the batch jobs, the cobol. It was truly astounding. It was also the source of about $500 million in revenue. They were doing software as a service in the 1970's. It's still going today. Customers in that space don't have many options.

<comment><USER54> Sounds like a market opportunity. What vertical market is this, more specifically?

<comment><USER55> Market opportunity? Sounds more like the best way to throw away your sanity. Some things are just beyond repair.

<comment><USER54> Since when is a greenfield app on a new product addressing a market with only old entrenched players "throwing away your sanity?"

<comment><USER55> Sorry, I did not realize you meant to create a new software from scratch. I thought you were talking about supporting/extending/refactoring this old pile of madness. Thanks for the clarification!

I suppose you were referring to the statement "Customers in that space don't have many options". Your statement is reasonable and there might be market opportunities to create new software. But I somehow have the feeling that it would take a tremendous amount of time to recreate something that checks the same boxes as the old system. But I do not work in that sector so my judgement might be totally off.

<comment><USER54> Thing is, a competing product does not (at launch) have to match the entrenched one feature-for-feature. It just has to tick the MOST IMPORTANT boxes while being advantageous in other ways (such as being built on FAR newer tech, more reliable etc.)

<comment><USER56> Being built on newer tech isn't an advantage from a customer POV. It's only an advantage when the newer tech is better, and tbh I have a hard time believing that you'd match the performance and security of a mainframe that easily.

<comment> le-mark Being an order of magnitude cheaper would be compelling to these same companies though.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER57> In my first year of university, I wrote a Java swing game which had classes for literally every part of the GUI (I was young, carefree and unwilling to yield on the principals of OO I'd been taught earlier in the year. I think it had somewhere in the region of 100 classes. Of which only about 3 had the real logic in them.

Now I've put it in perspective, I went on to an intern position between years 2 and 3 of uni. I was handed a _lovely_ piece of code which had:

\- Around 300 classes \- 3 or 4 layers of nested generics \- Factories, factory factories, generator factory factories \- 90% of the parameters were passed in from the build engine running the code, so it was impossible to run locally, ever. \- 0 tests \- Some 100 pages of documentation, which had been lost until I was about halfway through my placement (and mostly documented how to set up and run it, not how to maintain it)

Seriously, this thing was designed to the extreme, made to be generic to every single scenario in existence.

So what did it do? It took items from a customer facing system, transferred it onto the internal work tracking system. Then when they were updated in the modern system, mirrored the relevant updates back to the legacy.

The best part? Every time the internal work tracking system updated (once every 6 months), this thing broke horribly and it was practically impossible to fix. Even if you managed to set up stuff so you could work in a development environment, it still connected to the customer facing system, so you had to be incredibly careful what you did during testing.

It wasn't the biggest in terms of LOC, but it astounded me just how much effort (apparently the guy who wrote it squirreled away for a year to write it, then moved to Canada, and was famous in the department for having one too many beers during a outing to a local Indian) went into designing this behemoth.

I still have the occasional nightmare about it!

</comment><comment><USER58> I once had to look at a client's code to determine if/how we'd go about taking over their application. Their only developer threatened to quit and this is when they realised it would be best to outsource this and reduce the bus factor.

It was a huge _folder_ (not repo - and there were zip files of different "versions" of the code in there). The main monster was a huge Visual Studio solution with hundreds of targets, one would be an application for entering some data, the other was for entering data from a hardware device (a scale if I remember right), etc.

The main source of truth was an MSSQL database to which all these apps would connect _as root_. There is no backend as such to ensure access control & consistency, and any misbehaving app could essentially trash the entire DB.

Database credentials were hardcoded in every app's main entrypoint, with earlier "versions" of the credentials commented out below.

I thought that surely these must be either staging DBs or at the very least there would be network-level access control meaning the DB wasn't accessible from outside... but no - I managed to connect to their production DB _as root_ from a random, untrusted location. I do not know if MSSQL uses encryption by default but I would bet good money there was none and they were essentially connecting to their DB as root, over plaintext, from hundreds of different locations across the country without any kind of VPN.

In terms of code you obviously have your standard & expected "spaghetti monster" with UI & business logic scattered everywhere. What struck me the most was an _empty_ exception handler around the main entrypoint.

In the same folder there was also source for an iOS app. Didn't look at it but I don't see _any_ valid reason why this should be in the same place as the Windows apps.

Thankfully I no longer work there and even if I were I had no major C# experience (which gives me a very convenient excuse not to touch this mess).

<comment><USER59> Ha!

Are you me? :-)

Had almost the same experience, minus the database. Friend of the owner wanted to buy a company and asked us to evaluate their code to see if it was maintainable enough to add new features. I got a zip of hundreds of firmware projects each representing a different version. They were all on the same basic platform but with different hardware features #ifdef'd, or customized for a particular customer. The code itself wasn't _that_ bad (not that good either!), but their developer clearly had no idea what Version Control meant.

In the end I gave the thumbs up and he bought the company, then ended up having to redesign the product from scratch since much of the originally designed-in components were no longer available. He did his Due Diligence for the software, but ignored the hardware side!

</comment></comment><comment><USER60> I worked for a company that makes sortation devices (conveyors etc), and i inherited an old product from a previous colleague.

The software was largely a standard base with modifications done for the individual customer to suit their business needs.

In this particular project, it was a batch sortation, meaning we received a large batch file from their mainframe, which would then be parsed and executed by the controller software.

Everybody feared this particular project, and estimates on new functionality were sky high, but I didn't think much of it until i had to modify the batch parsing code. I was met with 22000 lines of C code in a single function. This single function was modified multiple times each year, usually adding more code in the process.

It took me the better part of a month to refactor into "manageable" chunk sizes, and in the end i was left with a 1700 LOC function that was still "too big", but nobody really understood how it worked, and we couldn't test it, so i just left it at that.

After my refactor could implement new functionality somewhat faster, but in the end it was still a very complex algorithm, so despite being in smaller functions, you still had to be very careful when modifying it.

</comment><comment><USER61> (I am a regular of HN. I don't want this attributed to my acct.)

Previous job. Was sysad. This code runs most of the academic US internet.

Everything was Perl5. There were over 150 different major applications written with it, ranging from 40 lines to 500k lines. The older the recent commit, the worse. Touching any of this would cause errors, either in itself OR in associated applications! You'd be working on thing A, get it working well, and 4 weeks later thing B would fail horrendously.

The worst was a tie between a variable that was a 6 line long SQL query, which was packed to the brim with function calls that ended up expanding a query to something like 50 lines long.

The tie was a gem in code that wasn't touched for 7 years. This wasn't at the top, or even in a config file. It was hardcoded in the middle of the perl5 program...

         $db_server = (servername);
         $db_user = root;
         $db_password = (password);

Other dishonorable mentions are as follows:

1\. No primary keys for the main database....

2\. Goober had the idea of storing pictures in said MySQL database. 70 GB of pics...

3\. Redhat 4 still in production, along with RH 5.

4\. Its everyone for themselves. The goal is you hobble it along enough for the next oncall. Let them get hit with it.

5\. Running iron from 10 years ago. Contracts pull in $$$, but you're dealing with paleo-datacenter crap

6\. Just retired LTO3 tapes. Now they have "shiny" LTO5....

<comment><USER62> You know what's terrible? This could almost be any one of about three of the Perl contracts I've had in the last decade (apart from the US part, obvs.)

<comment><USER61> Yeah, the more I dig around how perl shops work, this seems endemic.

Now, I work in a heterogenous Windows/Linux shop with non-paleo hardware. We're even deploying on AWS in a limited fashion.

This place has its warts too, but everywhere has _something_. But that previous place... I'm surprised it still hasn't crashed and burned. Their networking was solid tho. Just anything with Linux was a tire-fire.

And as one more gem, there was a program that terrified the fuck out of me. A fellow engineer showed me this update tool that would update a router remotely. Great. Well, if you add the -n (customer) flag with the customer number, it would update __all __the routers for that customer!

It was spectacular, and terrifying at the same time. I asked them for their testing procedure, and it was a -t (customer-testing). If they forgot the -testing, well.....

</comment></comment></comment><comment><USER63> At a former employer someone created a framework to try and create a generic process to process and load data (i.e. ETL). It was written with the best of intentions, but was terrible.

This framework was given to an offshore team, and they were told to use it to farm out hundreds of requests. The framework was inflexible enough that they each started adding snippets here and there to make it work for those projects, with no code review.

When I joined there were well over a hundred different projects, all using this framework at the core, most having little bespoke tweaks to make the code work. Every failure was essentially a new failure in a new way.

It was a useful experience - it's one of those experiences that teaches me via a negative example. This was the worst example of how "roll your own" resulted in incredible technical debt I've seen.

<comment><USER64> Had something similar happen to me. I gave a new guy a task of 'figure out luigi or airflow to help us process nightly data better than crontab'.

He came back with a custom python/c++ framework where literally every function was called handle(), and in c++ it used type inference to figure out which handle() to call.

Apparently this is what he wrote for his last firm.

We quickly found something else for him to do but he didn't last much longer after that anyways.

</comment></comment><comment><USER65> My own! For a university assignment we had to make a simple Excel clone as a group.

We divided the work and I ended up working on the formula parser. I spent a week thinking about it and couldn't figure it out (I wanted to work it out from scratch). Eventually I had a flash of insight: I know how to parse simple formulas, so I can use string replacement to recursively rewrite a formula until I can parse it.

By the time I had written all of it, I didn't understand how it worked anymore, but it did work!

FormulaParser ended up being longer than the rest of the codebase combined, and I eventually learned the other groups did it with a regex and ~50 LoC...

</comment><comment><USER66> Most large enterprise systems that are more than a few years old tend to be pretty bad, and almost all of them work just fine, as long as you respect the 'dragons be here' signs and you don't attempt to fix that which isn't broken.

Millions of lines of code (or even 10's of millions of lines) are really not all that exceptional. The original programmers are typically in some home or are pushing up the daisies.

</comment><comment><USER67> I came here was about to complain about some projects of my previous companies. But reading this thread makes me feel ashamed for getting angry with them, compare to how much more pain you guys went through.

Thanks for all the sharing. Young devs (like me) should really read and appreciate this thread.

</comment><comment><USER68> My previous job. More than a million lines of code for a glorified CRUD app, with more than a $7 million annual budget.

Accruing technical debt was a process feature. More bad code that everyone is afraid to touch means more budget for terrified developers and testers, and insane networked database design means more budget for servers and sysops. The fear leads to meetings, the meetings lead to suffering, and suffering leads to the dark side. It still works according to spec, and is human-fixed quickly whenever it doesn't, but the poor quality of the codebase is likely costing at least $2 million per year.

<comment><USER36> Read that last paragraph in the emperor's voice from star wars episode 6.

</comment></comment><comment><USER69> I'm guessing by bad, you mean ugly. I'm fairly convinced there is no "good" code, or that it's such a minuscule subset that you'll never likely encounter it in your career. Every year I look back on last years stuff I've written, and I can find ten ways to clean things up.

In my opinion, if it works, and provides the utility it was designed to bring, then it doesn't matter. If it makes money, then who really cares!

<comment><USER70> > In my opinion, if it works, and provides the utility it was designed to > bring, then it doesn't matter. If it makes money, then who really cares!

This makes sense. However, if the code 1) is hard to understand (according to the developers available) and change and 2) needs to be changed, it costs money.

Eg, read about the expense banks are now incurring trying to maintain COBOL systems. Whether the code is "bad" is debatable. But the fact is that they have a hard time finding people who can work on it.

<comment><USER69> Sure, but you're acting under the premise that "if we just did it 'right' the first time, we wouldn't have this mess". What I'm saying is that only under very few circumstances does it ever workout that way. Particularly with long standing systems and their software. It just builds over years, nothing you can really do about it.

<comment><USER70> > Sure, but you're acting under the premise that "if we just did it 'right' > the first time, we wouldn't have this mess".

I think you're right that every long-lived code base will have warts. And I don't think that means that the original builders were wrong-headed.

But if you've got a decades-old system that nobody understands anymore, you've got a huge liability. You can't ship features to compete, you can't fix bugs, you can't comply with new regulations. You can't even rewrite confidently because you don't know what the old system does.

There must be things you can do as a code base ages to keep it maintainable, allow incremental rewrites, etc.

<comment><USER71> You may not want to reinvent the wheel, but you have to change the tires.

</comment></comment></comment></comment></comment><comment><USER72> 300,000 lines of Python…

The company had bought Ellington, a Django based CMS, but the team basically rewrote the entire thing using multi table inheritance (unintentionally), so everything in the database had two copies, and we had over 70 tables, hundreds of gigabytes, disasters every week and tons of bugs... I discovered this more than a year later and nobody was even aware. Even the DBA wasn't aware the tables were duplicates.

</comment><comment><USER73> Not the largest, but the most insane.

At an old employer there were 15,000 lines of batch script across 14 .bat files on a Windows laptop. Old director of IT used it to onboard new customers. It basically copied a DB and turned "CHANGE ME" in some columns to the client's name.

It had it all. 5k lines of date validation, 3k lines of "UI", 400 goto statements, hard-coded passwords, versioning by incrementing the file names (leading to a bunch of code that was never called), and to top it off a static IP granted to the laptop that used as a part of authentication.

Took me two weeks to unravel it and replace with ~20 lines of Ruby.

Later, all of my complaining on Facebook led an old professor to invite me back to give a talk on the importance of code quality!

</comment><comment><USER74> More than a million lines with a single file containing more than 100,000 lines of spaghetti code consisting of macros and badly indented C code in a single function! This powered the GUI for an entire generation of low cost phones (pre Android era).

</comment><comment><USER75> A while ago, I was tasked with maintaining production code written in R by an enthusiastic junior developer. He loved R so much that it blinded his ability to use the right tool for the job.

Instead, he wrote web applications in R, instead of Python or Ruby, which my company had many developers who had expertise in, and eventually handed it over to to me. He even persuaded our bosses to invest into R Studio Server and had an instance installed in one of our machines. It's not only the choice of the programming language that made me furious, it's also the quality of code. He also mixed up snake case and camel cased variables all over the code. In addition, the same name would refer to different things, eg. `abc` and `Abc` and `a_bc` would mean totally different things. And stuff that could be written in a simple Sinatra or Flask application were written in R Shiny.

As a non-R person, I quickly learnt the language, (while mentally cursing it all the way for the bad choices it had made and the terrible inexplicable syntax it had) but getting used to this bad code was quite a challenge. We had several top tier clients whose reports were critical and reliant on this R code and it would frequently, randomly fail while maxing out on memory, no matter how much you threw at it. Debugging was another issue and I struggled with this codebase for 8 months while this junior developer had moved on to other technologies.

Eventually, my main role _almost_ switched to devops which I hated, because I enjoyed writing web applications and good code that doesn't require maintenance nor devops much. Eventually, I realized I couldn't take responsibility for this anymore as it would cost me my reputation and I really didn't like the way the company handled the situation as well. They were quite supportive of the junior dev encouraging him to move on to newer technologies while he half-assed everything and threw it on other people's heads who already had other responsibilities. They did this so that they could show off at meetups "We use the latest tech stack..blah blah" while adding 0 value for clients.

So, I quit the company, along with a dozen others and never looked back. But, I did learn quite a lot..my my.

<comment><USER76> > In addition, the same name would refer to different things, eg. `abc` and > `Abc` and `a_bc` would mean totally different things.

That's not too unusual. In Java `Camel` would usually be a class and `camel` an object, and in Prolog `Camel` is a variable wheras `camel` is an atom. Not sure about R though..?

> And stuff that could be written in a simple Sinatra or Flask application > were written in R Shiny.

I don't know R Shiny, but the examples looks neat and simple.[0] Are you sure this is not just a case of "I don't like X" rather than the code being bad?

[0]: [https://shiny.rstudio.com/articles/basics.html](https://shiny.rstudio.com/articles/basics.html)

<comment><USER75> Well, that nomenclature works well for classes and objects, but if you use mix up cases within an object, I think we call can agree it's really bad code?

As for R Shiny, those examples all look fine, but wasn't the case with my code.

</comment></comment></comment><comment><USER54> Here's a metaquestion:

Is it possible for _any_ codebase to _NOT_ eventually (given enough time) become a crufty pile of garbage?

I suspect (but have no real evidence... yet) that SOME of this spaghetti garbage is due to the traits of procedural, OOP, mutable languages. But this would then imply that things like functional-language codebases have much longer lifespans... and I don't have evidence for that... but I'm hoping someone can chime in

<comment><USER77> Linux kernel is massive and has been around for decades now and it's probably millions of lines.

It certainly has a lot of cognitive overhead since you need to always keep in mind the context in which the code you are writing will be running (to reason about concurrency etc.), but it's relatively easy to understand and well written.

<comment> yjftsjthsd-h Yeah, I think some FOSS projects qualify; I'd say OpendBSD.

<comment><USER78> OpenBSD has a very sane culture in which deleting code, or reformatting code without the compiler object change, are good things.

Most organizations don't see the value in that.

</comment></comment></comment></comment><comment><USER79> The worst I've personally seen was a many tens of millions of lines C++ Qt GUI app. Won't mention the name to protect the guilty but it's something some people here might recognize.

It didn't really need to be that big. A lot of its size was a result of pathological over-engineering. Apparently a previous (of course) engineer who built much of the original app thought boost::bind and functional binding was awesome. He also absolutely loved template meta-programming.

The code base was full of templates that contained templates that contained templates that contained... layers upon layers upon layers of templates, generics, bind, and so on. The horror. The compile times were awesome too. Local builds on an 8-core machine took 15-20 minutes.

I once made a commit to that code base where I replaced two entire subdirectories of files and tens of thousands of lines of code with one function. It took me weeks to understand and then finally to realize that _none of it was necessary at all_. It was all over-engineering. The function contained a case statement that did the job of layers of boost::bind and other cruft.

I definitely had a net negative line count at that job. The experience helped to solidify my loathing of unnecessary complexity.

It also made me respect languages like Go that purposely do not give the programmer tools like complex templating systems, dynamic language syntax, etc. It's not that these features have no uses, but honestly their uses are few. I've used Go for a while now and have found maybe two instances in tens of thousands of lines where I missed generics. I imagine I'd miss operator overloading in heavy math code, but that's about it. The problem is that these features are dangerous in the hands of "insufficiently lazy" programmers that love to over-engineer. I'd rather not have them and have to kludge just a little than to deal with code bases like the one I described above ever again.

</comment><comment><USER80> I never actually saw the code that makes this work (thank Christ) but at the previous place I worked, we did everything in SQL. Believing SQL Agent to be 'too limited', my boss had taken it upon himself to reimplement it... in SQL...

Just let that sink in for a moment.

There were 20+ tables all modelled after how SQL Agent does its own scheduling, 30+ stored procedures for interacting with it, and it was all intended for use with a GUI that was (naturally) never written. A relationship diagram of the tables sat on my personal Wall of Shame board for most of the time I was there. And yes, I had to use it from time to time. It was installed in every database for 200+ clients, in production, UAT, development, you name it.

The most delicious irony is, the only way such a thing could run automatically... was to use a SQL Agent job to poll it once a minute.

<comment><USER80> And yes, I'm aware you can programmatically create SQL Agent jobs in SQL, so he couldn't even use that excuse. The only thing it had over SQL Agent was that it could be 'event driven'. All that ultimately meant was that the job would only run if a condition was met.

Yeah...

</comment></comment><comment><USER81> As many experienced commenters below have already noted, nearly any piece of substantial, revenue-generating, long-lived code will ascend to a place full of dragons and mysticism. The more interesting debates revolve around:

1) How did it get that way, and what can we learn from that? 2) Is it inevitable that all software will end up like that? 3) How can an organization ever successfully sunset or move to a more maintainable system? Or should they even aspire to that?

<comment><USER70> [https://www.reuters.com/article/us-usa-banks-cobol/banks- scr...](https://www.reuters.com/article/us-usa-banks-cobol/banks-scramble-to- fix-old-systems-as-it-cowboys-ride-into-sunset-idUSKBN17C0D8) describe banks struggling to maintain ancient COBOL systems as the developers who understand them are dying off.

I don't know exactly what they should have done and when, but it seems like rewrites are going to be necessary, and it sure would be nice to start rewriting a system while you still have someone who can explain what it does.

<comment><USER81> In the post-Y2K years, I heard a lot about how the next order of business was going to be to replace all those systems in health care, banking, etc, especially as a generation went into retirement. But, here it is 2018 and there are still articles Food for thought.

</comment></comment></comment><comment><USER82> My worst experience was with a PHP app (no framework), 60K LOC per file, 6 dimension arrays, variables and comments in 3 different languages, elseif statements spread over 2-3k LOC, no kind of separation of concerns (HTML mixed with php, db calls, etc).

It was a 3 months nightmare with an arrogant as fudge client/developer.

</comment><comment><USER83> I have seen a complete project. The project was maintained by one person. Former developer left and transferred his work to the next.

It was a C project, but the second developer was coming from a Java background. So his half is all written with Java notation and naming conventions.

The thing was maths-heavy and had no comments. Instead they were consulting to a documentation book which also had file names and approximated line numbers.

The error tracing was done with function call stacks, so every function pushed some data into a global stack before a critical operation and, popped the same data if everything went without any problems.

Some libraries they were using were ad-hoc patched to taste, and grafted into the code tree. So, debugging was 5x harder and longer.

The functions were not divided into headers in a logical way, they were ad- hoc. So finding something was dependent on an source indexer or a grep sprint.

Last but not the least, the developer was so arrogant that the bugs were resolved by threatening to force him to clone his code and sent it out to another developer to debug and clean. Otherwise he pretended that the bugs were not present, because it was running on his test bed.

... and yes. It was in production.

</comment><comment><USER84> A nested if statement approximately 75 levels deep all because the author didn't understand that an ID can be unique. So he manually checked the value (which meant it could never be changed without a code change).

He did't understand the concept of a join. So he'd nest queries in VBScript with join key supplied from the outer query to the inner. Row by row. Essentially, a manual cursor.

Same programmer wrote an ASP portal app. The login of which got most of its security because they didn't know how to iterate over a returned dataset. Same code would set a cookie for access IN THE PRESENCE of a password. It could be wrong, you would still get access. Worse, the logout function didn't delete the access cookie, it just redirected you to the login page. Meaning you could impersonate anybody if knew thier username. Included admin.

I once corrected bug, by using a view. I sent him the view. He had no concept what a view was "That's like a stored procedure right?'. I'm shocked he knew what a stored procedure was.

He's still in business and the software is deployed worldwide. He refuses to fix it. He's a multi-millionaire.

</comment><comment><USER85> Worked on a file-sync system written in Python 2.6/7 by B-players in a hurry.

Lots of microservices, before it was cool, that was fine. Shitty code everywhere, commented code, dead code, 5 blank lines here and there. Many lines over 100 chars, lots pushing 150 or 200+. Didn't understand how to use argparse or logging but tried. Crazy mixed-case WTFExtremelyLongSillyNamesEverwhereSendThis at the command-line interface, instead of verbs like send.

Had a custom ORM that didn't want to look like one and took 5 times the code to do similar things. Little handling of exceptions, things like wrong permissions or IO like tar file creation might cause a 6 hour job to crash.

Daemons couldn't be shutdown gracefully, had to tail their logs until they paused for a moment, cross fingers, and kill process, often kill -9. Old daemontools made it more difficult. Bad timing could mean you are in for 3 to 6 hours of manual job cleanup work. Would happen a few times a week anyway, cutting into dev time. Still you could count on it to work about 90% of the time.

Token test suite and docs. Embarrassing web interface that would look amateurish in the 90's. Original developer made us do a standup everyday at 10:30am just when getting into the zone. They felt worthless for a while, and then it dawned on me why, we were all working on different projects.

The punchline: spent three months of 60 hour weeks taking out the trash, writing tests, paying down debt. Spent the next month or two with another dev designing/writing a vastly improved V2 with graceful shutdown, Django-style ORM, and quality as headline features.

A few weeks before we're about to knock it out of the park and deliver, the old author of V1 comes back in a panic, says we need to finish at end of month as a huge project is finishing. Doesn't seem to make sense, big changes at end are a bad idea. Takes over control of project, designs/implements V1.1, pushing aside our improvements and whips it up in a few weeks while I sit there with nothing to do. 6 months work of 60 hour weeks flushed down toilet.

After picking jaw off floor and offering a few choice words I left the job by mutual agreement a few weeks later and didn't look back. Good times.

<comment><USER86> My only code style rule is don't write lines more than 100 char which usually means "one statement per line". It floors me how people think it's acceptable to write code that you have to scroll not one but two axises to try and comprehend. Reading code should be like reading a book. And forcing yourself to write code that grows vertically instead of horrizontally usually solves all the problems and more that people dumping countless hours into auto- formatters are trying to grapple with.

<comment><USER85> He'd code fullscreen with a 16:9 monitor, so felt "it's not a problem."

</comment></comment></comment><comment><USER87> Not exactly code but people would be scared if they found out how much of the Internet, at OSI layers 1, 2 and 3, is held together by the metaphorical equivalent of duct tape and twine. At OSI layer 1, sometimes almost literally. Major ISP outages that have been caused by somebody tripping over the $5 power strip that was powering the authoritative nameserver.

<comment> yjftsjthsd-h > duct tape and twine. At OSI layer 1, sometimes almost literally

How about barbed wire? [https://news.ycombinator.com/item?id=15910263](https://news.ycombinator.com/item?id=15910263)

</comment></comment><comment><USER88> I worked on the DMS switch in Nortel (if PRI broke for you, I'm sorry, I tried my best!). 31 millions lines of absolutely horrific code. You'd think _somewhere_ in that mess there would be some redeeming code, but if there was I never found it.

To give you some examples, I originally came on as a contractor because they had some refactoring they wanted done. The entire system was home built (including the programming language) and there was a file size limit of 32,767 lines. They had many _functions_ that were approaching this limit and they didn't know what to do, so they hired me. Probably you can imagine what I did.

One time I went to a code review. They were writing a lot of data into some pointers. I asked, "Where do you allocate the memory"? The response was, "We don't have to allocate memory. We ran it in the lab and it didn't crash, proving that allocating memory is a waste of time". No matter how much I tried reasoning with them, I couldn't convince them. The code shipped like that.

One of my more amusing anecdotes is that when I worked there the release life cycle was _five years long_. The developers would work on features for 3 years. The developers were responsible for testing that their own code worked. There was no QA. After 3 years, we would ship the code to the telcos (telephone companies) and they would test it for acceptance for 2 years. We would fix the bugs that they found.

I started working there at the end of a release cycle, so people were only fixing bugs. I got an interesting bug in that I couldn't find any code that implemented the feature. The feature had apparently been implemented at the beginning of the cycle (so around 4 years before), by someone who was now my C level manager. I started looking at the other features that person had implemented. There was no code. It seems that this enterprising person had started work and realised that nobody would check his code for 3 whole years. He just checked off all his work as done without actually doing anything. Since he was an order of magnitude faster than everybody else, he was instantly promoted into management. When I reported my findings to my manager, he made it clear I wasn't to tell anybody else ;-)

Such a messed up place. _But_ the switch worked! It had an audit process that went around in the background fixing up the state of all the processes that ended up in weird states. In fact, when I worked there, nobody I worked with knew how to programmatically hang up a call. If you were using a feature like 3 way call, etc, they would just leave one side up. Within 3 minutes, the audit process would come by and hang up the phone. Tones features "worked" that way -- by putting the call into weird states and waiting for the audit process to put it back again. You could often hang up after a 3 way call, pick up the phone and still be connected to the call.

Most people don't know it, but because of some strangeness with some of the protocols, telcos used to "ring" their main switches with Nortel DMS switches. This would essentially fix the protocols so that everything could talk to everything. So, if you ever made a long distance telephone call 20 or 30 years ago, it almost certainly went through a DMS switch. The damn thing worked. Somehow. I have no idea how, though ;-)

<comment><AUTHOR> This one might take the cake. Thanks for sharing.

<comment><USER89> I agree. It's like a buffet. Multiple stories, each frightening in their own way.

</comment></comment></comment><comment><USER90> Nobody mentioned Wordpress yet? Last time I checked it was a terrible pile of procedural code

<comment><USER91> Without a doubt. I've run into so many little "glitches" in WP that it really soured me to CMS backed websites. The next site I run is going to be 100% hand-coded.

... and it'll probably load a lot faster than the PHP+CSS nightmare that is WP.

</comment></comment><comment><USER92> A web application written in C++ before there was the STL or many libraries written for C++... in 2007 or so. It was probably 200k LOC or so of CGI application code, libraries to render PDFs, manage a filesystem of XML files, an XML parser... none of it documented or tested.

The developer who was leaving the company dropped a copy of Michael Feathers' _Working Effectively with Legacy Code_. There was a small amount of Python to wrap the API to the C++ code using Boost against which a small suite of unit and functional tests were being developed. I learned a lot on that project.

I never fully understood how the C++ code all worked but having that API interface in Python helped to grok parts of it (and eventually replace it with a few hundred lines of Python code at a time).

</comment><comment><USER93> Actually, if you want to see clean code that works well, try the traditional ex/vi source. It's something like 20K lines, but very neat functional programming all the way down.

<comment><USER94> I just want you to know that through rabbit holing 'traditional ex/vi source', the following happened...

1) Read a Wikipedia page for vi.

2) Noticed that Bill Joy used a Lear Siegler ADM-3A terminal.

3) Discovered that Lear Siegler was the result of a merger between Siegler Corporation and Lear, Inc.

4) I know that Lear, Inc. was founded by William 'Bill' Lear.

So, not only do we have Bill Lear to thank in a way for vi, but your username is oddly familiar...

</comment></comment><comment><USER95> 12k line code-behind for an ASP.NET page that duplicates PDF files, driven off of a spreadsheet, and adds some additional information at the top of each page. This has generated the majority of a major state's ballots for at least the past 4 elections.

</comment><comment> cat-turner In my past life I was an oceanographer at a consulting company. Essentially we wrote Matlab code that processed data we get from ocean and meteorological models. Many of these models, used by NOAA, are all written in Fortran. While most were well documented, one thing you could not get around is the use of parameters for these models ( think of global variables used everywhere) and a never ending game of go-to statements. I had to figure out how this one flood model worked... literally had to read 5,000 lines of Fortran. This is what we use to determine potential flood impact from hurricanes, today. Software written for oceanography is a whole other beast.

</comment><comment><USER96> I was given the opportunity to work with iOS at one of my former employers and I quickly agreed (I was a backend developer back then). It was a rather popular application at a rather large Austrian media company, with around 50k unique monthly users.

Boy did I regret that decision!

\- 100KLOC

\- Initial development outsourced to India. Comments, variable names in Indian.

\- Subsequent development outsourced to Belarus. Add comments, variable names in Russian.

\- "Why use ObjC OO features when we can write buggy and incoherent C"?

\- Global, implicit state everywhere. Tapped on something? Hope it didn't mess up the state you are relying on.

\- Obviously no tests, and no testers.

\- Inheritance chains of up to 20 classes.

\- iOS kindly forces MVC, but you can obviously write empty controllers and all spaghetti-logic into your views. Needless to say, that was what they did.

\- Complete lack of proper structure. Several views iterated up their parent controllers to the desired one, grabbed its views, iterated over them until the right view was found and something was done to that view.

\- Building and running the application was controlled by 10+ env variables (the other dev was fired after he pushed a dev build to the App store, which mysteriously passed review. "Whoopsie, forgot to set one of the env variables correctly").

\- about 80% of the logic was copy-pasted for the iPad build instead of reusing anything. It was not a different target, it was a separate project.

<comment><USER97> You were "given the opportunity" huh? :)

How long did you work on that before you threw in the towel?

<comment><USER96> About six months. I managed to actually get a new feature into that hot mess while fixing bugs that would take days to weeks to track and handling compatibility with a new iOS major version, but then I got an offer to work in the space industry and quickly left.

A while later I heard they threw everything away and started an in-house rewrite in Swift. Some people do seem to learn from their mistakes.

EDIT: clarification on the opening statement.

</comment></comment></comment><comment><USER98> Two case:

1\. Millions lines of messy Java code with annotations and dependency- injection form a multi-shard distributed job with tens of distributed downstream backends, some of which provide fake always-success synchronous RPCs hiding the fact that the underlying operations are actually asynchronous and may fail very often. What makes it worse to work on these code is it was built with a single transactional underlying database but then due to reliability issues of the single database, data are now across at least three different transactional databases. This causes endless race conditions and concurrency issues to fire everywhere. The production release of this distributed job used to be once every week, now multi-months is normal, and quarter rollback is not a surprise to people.

2\. Almost million lines of messy C++ code with several .cc/.cpp files containing tens of thousands C++ code, some class implementations are across multiple .cc/.cpp files. I have always been scared when touching some of these giant .cc/.cpp files. People who have been working on the code for years can still easily make ignorant mistakes when adding/modifying a small feature (with so-called fully unit-test overage of course). There are multi-million lines of testing code, which is almost 10x of the code be tested, but most of them are bogus and almost test nothing, hence silly mistakes are everywhere, everyday. Even the original author of the code base needs 5 follow-up fixes in order to make a 10-line behavior change work.

For both code bases and the jobs running these code, people are now talking about breaking them into microservices, by converting function calls in the existing code bases into RPCs. I can foresee a tremendous number of service outages are coming...

</comment><comment><USER47> I might have a slightly different record, if you count lines in a single source file...

One client wanted to hire me to continue developing his AutoCAD plugin written in AutoLisp (a cut down version of lisp for writing macros in AutoCAD). He had all his code in a single file which was around 63,000 lines long. (I calculated this as 504 meters of screen/paper top to bottom).

This was for a product that was in production, used commercially, and he'd been going for 15 years, adding bits as he went. There were loops hitting 300 lines, well over a hundred global variables, no consistency, and duplication like you wouldn't believe because he hadn't grasped the idea of moving code out to reusable subroutines.

I asked him what he did when his clients found a bug. The answer was "knuckle down for a few weeks".

I just couldn't believe that this was his daily work for 15 years, and he never thought to learn anything about programming other than what was immediately required to solve the problem in front of him...

Another unbelievable part of this, is that AutoCAD has an absolutely superb IDE built into it (i.e. it's fricking free!) which has features like the ability select & run code in current scope with minimal clicks which make for the fastest development environment I have ever played with, as in, you have no idea how much functionality you can churn out in a day. But he didn't like it, so he edited his code in....wait for it... WordPad! (That's a Windows built in Rich text editor where double clicking on an underscored_word only selects up to the underscore because it's not made for code, and that on its own makes it impossible to work with).

I tried to modify his code for several hours but had to stop myself because it was madness. So I told him the only way forward would be to rewrite everything from scratch after which point I could make it do anything he wanted. I reckoned it would only take 4 weeks to rebuild his 15 years of mess - <USER2> because the IDE makes development so damn quick, and was even willing to do it on a fixed price, but he declined, which I think was utter madness.

The thing is, he had 15 high paying clients, and two other part time employees helping with other aspects of the business, and drove a brand new Audi A3, which means he probably holds a world record for highest ratio of money earned to quality of code written, at least in the CAD subcategory :-D

</comment><comment> a-dub NaN kloc.

Nobody could even describe definitively where all the code could be found in version control. The best part was that nobody trusted version control either, so even if you did find something that looked relevant, there was a good chance it was dead and only served the purpose of confusing you or forcing you to memorize unstructured structure just to be able to get around.

It was madness inducing.

</comment><comment><USER99> Here's a story I haven't shared in awhile.

One of my first gigs actually getting paid to code was getting hired on as a last ditch effort to save what was (unknown to me at the time) a failing business. It was a company that basically relisted real-estate auctions on their own site, coded entirely in PHP, by a single developer who had read "How to Code PHP in 24 Hours". They had one client that was basically keeping them afloat. It was so disorganized that at one point I was tasked with making a quick YouTube commercial in Adobe Premier for a client, showing off the features of our whitelabel product with their logos, edited from a stock template. I do not know Adobe Premier. I digress.

The main PHP file (yes) was 20,000 lines of code. Want to add a new feature? Copy that file into a new file and save it as newfeature.php. Database operations weren't transactional, there was no change management, and for about a week we were using production systems to code until development environments were made for us.

There was other shady stuff going on too, like using over a hundred proxy accounts to scrape content from other listing sites. I refused to touch or even look at that logic in the codebase, and it was always talked about in kind of a hushed way. I was young, didn't know any better, would nope the eff out if a similar opportunity came along at this stage in my career.

They folded shortly after laying off pretty much everyone but the CEO and the lone coder. Dumpsterfire would be an understatement, but my coworkers were chill and helped make the best out of a bad situation.

EDIT: Oh yeah! I forgot to mention the hardcoded password the was site wide that we used as a sort of "impersonation" feature. You could type in any user account and this password, and it would log you in no problem. No, we did not have auditing controls.

</comment><comment><USER100> When I took over my current job a outsourcing company who employed developers based in Islamabad (Pakistan) wrote most of the code. It's C#, they didn't understand the concept of null, by ref or by value.

It was interesting. The database was Mongo and they had some horrible Entity Framework port that they forked on Github and used which wasn't maintained and at the time in 2015, 3 years old.

The data layer had business logic. Let's say they were getting a user record from the database but they didn't want to include the first and last name they'd write a method called

GetUserWithoutFirstAndLastName();

That would be in the User repository. Another requirement would come up to get the user but not include the user's language for instance and they'd create another method

GetUserWithoutFirstAndLastNameAndLanguage();

Ended up with about 70 or so methods which basically gave different levels of hydration for the User object.

The frontend was written in Extjs which took about 25 seconds to load.

They had no tests.

<comment><USER36> C# 8 just announced optional nullability as a feature

</comment></comment><comment><USER101> All code is bad and Google puts almost everything in one repository, so I'm going to say that ;-).

For more traditionally defined bad code, I worked with a team that rewrote an existing service from scratch using with 40000 lines of spaghetti. The service that it replaced was 80000 lines, and was replaced despite working perfectly fine because it was a total clusterfuck with dependencies interwoven across every service anyone had ever heard of. This was a payments system for a major online retailer. All of this code has been removed since.

I also used to participate in Java4k, a competition to make games fit in 4 kilobyte jar files. Writing almost everything inside a while loop in a single function is basically table stakes for that.

<comment><USER102> > I worked with a team that rewrote an existing service from scratch using > with 40000 lines of spaghetti. The service that it replaced was 80000 lines, > and was replaced despite working perfectly fine because it was a total > clusterfuck with dependencies interwoven across every service anyone had > ever heard of.

Your post should be at the top of the thread. Rewriting projects from scratch tend to do more harm than good, and the only reason this problem isn't addressed very often os that the people invested in reinventing the wheel don't admit to having caused more problems than they solved, and the ones who developed the old systems have moved on and thus are unable to say anything in their defense.

</comment></comment><comment><USER103> Any large company. I'm being serious. As a development coach I see a lot of code and I must say the DEFAULT is "how does any of this even work?"

<comment><USER104> I've been working on fixing some old C++ code at my company so that it compiles with the most recent version of the compiler. To my surprise, the errors that the new compiler gives me are very reasonable. But I always ask myself, "how on earth did this ever compile in the first place?"

The crazy part is going into the commit logs and seeing that a lot of the people who wrote this have been very successful. They've become vice presidents, retired rich, and written influential papers.

<comment><USER105> I've been doing the same, porting (getting to compile and seemingly run) an old c and c++ codebase from 32bit solaris to 64bit linux. For a while I was spending my days alternating between "how did this ever compile" and "how did this ever run" when debugging. It took a lot of modifications to get it to compile on a now 6 year old version of GCC, then I got to debug a lot of issues with type sizes, endianess and posix differences. The team responsible for it is still in favor of keeping it 32bit unfortunately, so we can't even do a desperately needed valgrind run. I wish I could look at the history but I don't have access to the clear case repository I think they're still using.

We've got another codebase of similar vintage that is preventing us from deploying spectre patches...

</comment></comment></comment><comment><USER106> I'd like to hear people describe why the code they've seen is bad.

<comment><USER88> In my experience (despite what you might imagine) the biggest factor has been lack of experience for the developers. You can get some pretty weird ideas when you first start out. Those weird ideas can gain traction in a group and then it becomes "the way" to do it. Over time, the system gains cruft and there is no practical way of addressing the problems. It takes _really_ experienced people who specialise in legacy code to systematically improve large systems -- and it takes a donkey's age. It's just not a sustainable enterprise.

We tend to think that harsh deadlines and unthinking managers are to blame, but writing good code over the long term is incredibly difficult. Group dynamics are hard to deal with and as you add (or replace) people on the team, you are bound to eventually go off the rails even if you started off well. Which is not to say that you shouldn't try, but our industry is really immature. Most developers are quite young and even when you have a couple of older people on the team, they may or may not have the skills for long term design evolution. Maybe 50 years from now it will be the norm to write good code, but I think we'll still be writing legacy messes for a while.

I should point out that even the worst code I've seen written this decade is at least an order of magnitude better than the average code I saw when I started my career. As an industry we _are_ improving!

</comment></comment><comment><USER107> A 10,000-line C++ module for transmitting hundreds of different packets to a radio module from the driver, all cut-pasted copies of the same code with different structure name. Bugridden (bad copy-paste-edit bugs).

I replaced it with an 11-line template.

</comment><comment><USER108> I worked on an Excel\vba swat team at an investment bank. Whenever someone in the various business units got themselves into more trouble than they could handle with an Excel model, we'd try to set things right. We saw a lot of wonderfully creative vba code from industrious programming beginners (or, as we liked to call them, "Macro Recording Cowboys". The most amusing I saw was a model where the vba code used a Greek mythology variable naming convention:

Dim Hermes As String, Artemis As Long, Odin As Range

</comment><comment><USER109> A few weeks ago I had to reproduce one of the recent papers on deep learning theory. It had ~3000 lines of code over around 10 files, but when I just removed stale and unused calls and spaghetti (just like static analysis does), it reduced to ~1000 lines with identical functionality.

Original code also computed the most time-consuming routine twice to thrice each run and had many many typos, e. g. "calculte_infomration". Imagine this, but in every second function/variable.

However, it worked.

</comment><comment><USER110> Three years ago I was hired as a Business Analyst, with 0 background in anything related to IT. Our team was responsible to maintain a legacy base in COBOL.

As time went by, everything started to make sense and I was able to grasp almost everything about the environment and the tech stack. Even perform a little bit of system analysis in COBOL to try and identify some gaps in the code base.

But what always intrigued me is that even those senior developers in the team with 25+ years of experience with COBOL wouldn't ever touch this one program. The program responsible for 90% of the logic of the product of that company. Every now and then, this program would ABEND for whatever reason and sometime, no one could figure out why. They respected (cute way of saying they were afraid) this program so much that, instead of refactoring it, they would just throw an if statement and let it run.

This program was built in the 60s and had I don't know how many hundreds of thousands of lines of code. It is still running to this day.

Now I don't have the expertise to say if that was bad code, and even if I had, I didn't deal with this program enough to say this anyway, but I was very intrigued why would this particular program ABEND out of nowhere, and no of these super experienced developers would have the guts to touch it.

</comment><comment><USER111> There should be an award. Like the Turing award, but not quite.

The trophy should be a plate of spaghetti, in gold.

<comment><USER112> Hell yeah! Like the Golden Raspberry Awards or the Bad Sex Awards, but, you know, for code!

</comment></comment><comment><USER113> I worked in investment banking as a quant. Once had the misfortune of working with a regulator-approved custom built market risk system from the 90s. It had code in every language you can imagine, from cryptic Perl, to Java, to C. The thing worked, but development cycles were a year long, primarily to allow for regression testing. It was a monster to debug, as very few people have the brain space to simultaneously comprehend that many moving parts.

</comment><comment><USER114> My first job out of university was to program IVR systems for a telecom company using an in-house developed framework.

An IVR application would typically play a recording to the user, then wait for the DTMF signal, then maybe take the user to a sub-menu, prompt the user to choose another option and do a database query and then play another recording and so on.

The IVRs had to repeat a menu if the user made an invalid selection, and "press 'star' to return to the main menu" and so on.

So most of the applications I maintained were thousand line C++ functions that looked something like this (paraphrased):

      void ivr_main(int ch) {
      main_menu:
        play("mainmenu.vox");
        d = get_dtmf(ch);
        if(user_hung_up(ch)) return;
        switch(d) {
        case '1' : goto menu_1;
        case '2' : goto menu_2;
        case '3' : goto menu_3;
        case '4' : goto menu_4;
        }
        goto main_menu;
      menu_1:
        play("menu1.vox");
        d = get_dtmf(ch);
        if(user_hung_up(ch)) return;
        switch(d) {
        case '1': goto menu_1_1;
        case '2': goto menu_1_2;
        case '*': goto main_menu;
        }
        goto menu_1;
      // etc...
      }

</comment><comment><USER115> After many years as a software engineer, I can only say that every large old application is a pile of messy junk. Whether it's database software at Microsoft, visualization software at (another company), a database at a third company, another database at a 4th company, another database at a 5th company, they are all filled with junk and crap.

</comment><comment><USER116> The one I am currently working with \- Multiple calls to the database (a lot of times in a loop) \- Rather than reference, or variable passing, DB id is passed as a parameter to functions, so multiple calls to the database for a single object per API call \- Database design was normalized initially, for "optimization" most of the database has been denormalized. \- Queries without index \- If we were to create an index, probably have to create index on every column \- one of the function has more than 36 if else (including nested if else) \- Does not work with database partitioned or multiple databases \- OLAP like queries running wild on OLTP systems, just because we can \- Fastest API call - 1 second, slowest - <i dare no say> It started as an outsourced project and the same codebase is being dragged on, with features being added every day and with no coherent plan.

</comment><comment><USER117> I've been patching a configuration user interface web application that uses the same codebase for a bunch of different product categories. Some 40 files totaling 15000 lines of HTML/js combining filepp preprocessing (a C-like build time pre-processor) and SSI (server-side includes and conditional rendering with directives snuck into HTML comments) split over several repositories that are all tightly interdependent. Adding to that, obviously no one wants to claim ownership over this codebase and changes are frequently made by people that understandably don't know exactly what they're doing and reviewed by other people shooting from their hips. Any change to it has to be considered for every product that uses it, which I've never seen a full list of.

Not too many LOC but the badness/LOC ratio is terribly high.

</comment><comment><USER118> Microsoft Biztalk. By "work", I mean it didn't crash. It also didn't do anything useful.

<comment><USER119> If SAP is how Lucifer interacts with our world then Biztalk is how Cthulhu seeks to enter our world by contaminating the minds of innocents with its eldritch horrors.

Mind you, it may have got better than my experiences with it, which as you can probably guess weren't very positive.

</comment></comment><comment><USER120> If you search youtube for something like "pure data dance music" you'll find some excellent spaghetti diagrams that produce music/video.

Since the environment provides realtime feedback, it's a drag to try to refactor the current diagram into a reusable abstraction. Instead, many users optimize their creative time and just keep adding functionality to the diagram in a single graphical window. By the time they are done there is text overlapping other text and a bunch of lines obscuring most of the diagram.

Even with the ones that have a simple set of controls for a sequencer or whatever, there is usually a "guts" module that hides all the spaghetti.

Also, for any running program you can instantly make it twice as spaghetti by doing "select all" and "duplicate." :)

</comment><comment><USER121> ADP Retirement Services first generation of PES (Plan Entry System) program. It was/is an internal tool with all the horror stories that you hear about internal tools. The people who made it wrote obscure logic with encrypted variable naming with the main purpose of having job security, it's a core tool in their business model but treated like gutter trash, built with visual basic and embedded angular code to make it look more modern, the supposed rewrite talk went on for 8 years, and to put icing on the top it was QAd/maintained by underqualified offshore old-school contractors (you cant even call this group devs/engineers).

</comment><comment><USER122> The leaked Windows source some years back was pretty damn ugly.

And it was only a small portion of the OS.

</comment><comment><USER123> Microsoft Word back in '99 -- can't really estimate the amount of code, I didn't have the experience to handle something like that. What I saw of it and what I worked on, that alone made me reconsider my career path.

</comment><comment><USER124> If Foo1.php, Foo2.php and Foo3.php are all copies of the 1000-line Foo.php, each with a different smattering of bug fixes, each included by a different 10 - 20% of the overall codebase... am I allowed to count all 4 separately?

</comment><comment><USER125> I worked for a cell phone company. They had a large piece of software that is used by all customer-facing employees to manage accounts (billing, network, provisioning). Over its life, the software had been rewritten in many languages. Prior to my joining the team, someone thought to take a .NET application and rebuild it as a web application (Spring MVC and ExtJS4).

The first day on the team, before even looking at the code base, I ask our development lead what unit testing framework we are using for the Java code and what we are using for the front end code. He gives me this funny look and tells me to speak the guy onboarding me. I of course go and ask the lead onboarding me and he gives me an answer that turned my world upside down.

"Automated testing is a waste of time. You will spend too much time writing test then developing code and delivering stories to the business"

I need to point out that this guy eventually goes on to an executive level position. proving that it is not the quality of work you deliver, but the optics of delivering quality software that counts in large corporations. I digress.

I receive my SVN credentials later that day and come to the realization that I have made a very poor career choice. The front end code alone is over 5k Javascript files with functions spanning thousands of lines long all full with 100's of nested asynchronous callbacks. Not only is the code crap, but the tools that we had were not used correctly. For example comments.

//01/01/2018 - Fix bug - Begin //01/01/2018 - Fix bug - <USER3>

The code base was riddled with these. What story was this for? would have been useful to pull the Jira story and see what you were trying to do. I guess I'll spend the day going through your 5k line function that does everything, but nothing. Or, better yet, let me check the SVN commit date. Maybe they committed some good notes for me there.

01/01/2018 - Fix bug

Damnit!

I did my year, which is the minimum you can be in a job before posting out. I left without a second thought. When I left the business was complaining because the testing team was larger than the development team. I wonder why?

</comment><comment><USER126> Bad code is a relative term. It might look awful but the design might be decent and of course it works or it wouldn't qualify.

I'm maintaining two semi-large applications right now that I wish I had the time to fix.

The bad stuff is basically no documentation, no attempt at pep8 compliance, written before decorators existed in python and very dependent on Python 2 syntax.

But it works and it's actually a very good distributed monitoring system that was ahead of its time. Closed source but it would remind people of prometheus if it was released, yet it was several years ahead of that product.

</comment><comment><USER127> The ugliest, most incomprehensible code I've ever see was when I had to mess with RedHat's version of Anaconda (for building distros) back in the early 2000s. It was worse than the multi-thousand line TCL screen-scraper that I had to occasionally fix in 1999. It was the first code I'd even seen written in Python. You couldn't really trace anything, it was all so nested and idiomatic. It made the Perl code I inherited from a long-time Perl guru downright comprehensible. I've looked askance at Python since.

</comment><comment><USER128> Had to maintain a web code from the pre JSP era,i.e, servlets and writing UI code in Java classes (think writing html in s.o.p). The beauty of it all was that there were only two classes for the entire application: one was the servlet, and the other held all the UI code for all the pages. Requests bounced between the both of them, held together and dictated by a series of "if" conditions checking for "string equals" in a method with 50 variables, all strings, named s, s1, s2..s49. And no comments anywhere.

</comment><comment><USER129> Magento.

<comment><USER50> ... which was theoretically better than OSCommerce :/

<comment><USER130> ...or Websphere

</comment></comment></comment><comment><USER131> I was in an outsourcing team working for Panasonic door-phone system, the source code was big, but what my team 's done was to develop some more functionalities including video call. It was about ~20 KLOC written in C for the UI & stuff (what we were ordered to do, since the behind layer was not disclosed to us). The source code was a mess since it's written by low- experienced programmers and even intern students. Later I quit that job :)

</comment><comment><USER132> I once had to wade into a PHP app for a government procurement website that was just horrifyingly messy. I used to maintain records for the highest line number on which the opening <html> tag appeared in a template (well into the thousands) and the highest number of opening/closing PHP tags (hundreds in a single file). There was no structure beyond bunging stuff in the session and hoping it all hung together.

The app ran on Mac servers.

</comment><comment><USER133> A million+ lines of Java code for a product we (in a previous company) had; it started out nice and design pattern etc but after 5 years with ~200 different coders adding/changing things and no time to refactor, it became bad enough to trigger a phase out of the product and replacement by another product. I actually moved roles to figure out how to not make the same mistakes: those lessons work for them until today.

</comment><comment><USER134> Two projects with 300000 lines of Python code each at a small company handling physical access control systems with smartcards and stuff. Once I looked the most critical part; the logic which decides if a door should be opened or not and it was implemented with the assumption that in Python, the bitwise operators work the same way as in C (they doesn't at all).

</comment><comment><USER135> An Angular app with ~50k LOC.

Barely any tests and several hand-rolled components, some of which were merged only because the author managed to implore somebody to give them an R+ after a few sprints of the branch just sitting there.

A total of 21 people were involved in creating this system and for some insane reason we still had daily standups with almost full attendance.

<comment><USER136> What's the problem with "hand-rolled components"? Isn't that pretty much the essence of building non-trivial frontend stuff? Or does this have some special meaning in the Angular world?

<comment><USER135> Not in the case of components which could easily be replaced with a well- tested and proven solution.

Notable among the ones in this project was the date picker - there's a decent number of those available and yet somebody made the decision to hand-roll it.

The result was a mess that for some reason had a 300 LOC service as a part of it. Needless to say minimal tests.

It was a waste of man-days in a project that was already over budget.

<comment><USER137> There are a lot of date pickers, however if you have a specific UI and UX to implement, it might not be worth trying wrangle one of them into doing what you need it to do.

There's so many integration points (business logic, i18n/l10n, styling, keyboard navigation, accessibility, etc) that I think it's often easier to write your own. Hopefully now that most browsers support <input type="date" />, we can just use that most of the time.

</comment></comment></comment></comment><comment><USER138> My codebase I'm working on right now. Went from a proof of concept demo, to small mini game, to a bigger mini game, to full game, to SAS product. With no time(budget) to rewrite it up until delivery to a customer. Its pretty frustrating, as its my fault, but budget and tight deadlines forced my hand.

</comment><comment><USER139> So since we are on this subject, does anyone have advice for wrangling spaghetti QT apps into submission? We have two of them here and of course their creators aren't here anymore and we find it very difficult for developers that weren't involved in their development to grok all the shared state.

</comment><comment><USER140> Server code written in C++ that called into a lower level C library. There was a 4000 line case statement that handled message replay. I am not sure why it wasn't broken out into functions, but it was "sensitive" code and refactoring it was considered a no go.

</comment><comment><USER141> Drupal 8

<comment><USER141> I never worked in Drupal 7, but I can't image it was much better.

<comment><USER26> Drupal 6 was actually a pretty consistent hooks-based system that exploited the fact that PHP has a fast and feature-rich implementation of the function lookup table.

Drupal 7 started the migration to object-oriented code and was halfway through the messy rewrite when it was released. Drupal 8 finished where Drupal 7 left. That's where the majority of the developers left as well (pun intended).

</comment></comment></comment><comment><USER142> My first PHP app was an invoicing application. It is ~3500 lines of php in a single file. Most of which is a few heredocs with javascript to be in lined on the client side. I haven't touched the code in 10 years. I still use it for my freelance invoicing.

</comment><comment><USER143> As a budding programmer (hobbyist), what can I do not to fall into the mistakes others talk about in the comments? Is there any particular set of mental habits that could lessen the chance of spaghetti occurring, or is it just a matter of when rather than if?

<comment><USER144> All these comments are good. Also ALWAYS watch out for 'Quick Win': When you do something quick and dirty, mark it up for 'refactoring' (i.e. Tech Debt). When your list of quick and dirty builds up, go and clean it. Do not accumulate tech debt, try to keep it low otherwise you will pay full price later (examples up here).

Another great tip that saved me from tons of refactoring: "The wrong abstraction is worse code duplication". Meaning sometimes duplication of code is better than trying to create the wrong architecture (as long as you mark it in your quick and dirty list).

<comment><USER145> You don't just pay full price on technical debt, you pay it back with interest!

</comment></comment></comment><comment><USER146> Windows Vista. It is practically a virus, and ran ~5x slower than XP on the same hardware.

<comment><USER119> It wasn't even consistent - a colleague had weird errors with networking on Vista and I didn't even though we had identical laptops.

Microsoft support was basically "yes it does that sometimes".

</comment></comment><comment><USER147> Largest amount? 13k lines in a single Java file, none of which was boilerplate. Who knows how many lines were in the whole project... a million seems about right.

</comment><comment><USER148> C functions that span over 10k lines with 46 parameters.

</comment><comment><USER40> Not as big as some of the examples here, but still kind of interesting. I worked for a city government, and the entire backend code for our work-bids site was a few hundred files of PHP written with some sort of Dreamweaver framework that was deprecated years ago. Old enough that the entire site had to be run on PHP 5.7. I ended up rewriting the entire thing with Laravel (the framework of choice for the other webdev that was in the IT dept). It really didn't take that long; he just had too much on his plate to worry about it before I showed up.

</comment><comment><USER149> has to be itunes. What a bloated pile of crap that is.

<comment><USER150> any entertaining examples from your time working on it?

</comment></comment><comment><USER151> Once upon a time, there was a search product and one of the data sources that it could search was a Solr/Lucene database. This should be no problem, since search is what Solr does. It should be as simple as passing the user's query through to Solr and then reading the response. The problem was, it was important to know exactly which parts of any matched records were relevant to the search.

The Guy Before Me™ decided that the best way to implement this would be to split the user's search into individual words, perform a _separate_ search query through Solr's HTTP API for each _individual_ word, and then do a bunch of very clever and complex post-processing on the result sets to combine them into a single set of results.

This led to endless headaches due to horrible performance. Imagine if you wanted to implement web search this way. How would you synthesize the results for the search "boston plumbers" given the search results for "boston" and the search results for "plumbers?" You would need tens of thousands of results for each search term to find even one match that applies to both terms. Now scale this to getting hundreds of results to present to the user. Now scale this to n search terms.

I was tasked with making this take less than 8,000ms for a simple query. I spent a while getting to understand how this code worked and building out performance tests so that we could determine how it would behave under load (we didn't have any users yet). The results were pretty grim. I presented two possible options for moving forward:

1\. Move this crazy result-set-intersection logic closer to the data. I could build a custom Solr plugin to do this stuff inside the Solr server so that we didn't need to copy gigantic result sets across the network from Solr to the application server for every query.

2\. Delete ALL of this nonsense because _literally exactly_ what this whole mess of code was meant to accomplish is already implemented in Solr. They call it highlighting. It's one of the marquee features of the program. I can't stress enough that this is precisely, perfectly, unequivocally, the exact thing that all of this complexity was meant to accomplish.

My manager thought it would be a shame to throw away all of that very expensive code and lose the flexibility of an in-house solution. So we went with option one. I spent the next month writing a Solr plugin that reproduced the original logic. It was still slow as mud so I sharded the data across multiple Lucene servers and distributed the algorithm across them with a map/reduce sort of scheme.

In the end, it all worked great. It was fully ten times slower than the solution already built into Solr, but it worked.

The startup later ran out of runway trying to build a big-data-sized in-memory distributed database from scratch to speed up search. The founder (also the lead developer while I was there) insisted that everyone use raw C-style arrays and a custom in-house hash table implementation because he thought STL was too slow. Basically, "not invented here" was in the DNA of that company. I'm surprised we even used commodity hardware and didn't design some kind of in-house search coprocessor that would do everything in silicon.

<comment><USER89> ... it all "worked great". Wow, what a mess.

</comment></comment><comment><USER83> Obligatory XKCD: [https://xkcd.com/303/](https://xkcd.com/303/)

<comment><USER89> There's an xkcd comic for everything.

<comment><USER83> Absolutely.

Maybe, if we pitch the idea to Randall, he may prove it. :)

<comment><USER152> Ha ha Turing Complete xkcd...

</comment></comment></comment></comment><comment><USER153> Huge ecommerce company. Code written from the 90s in .NET was ported into PHP using an auto-library... nothing else to say

</comment><comment><USER154> ah my godness that brings back bad memories... :|

I was at a project were every new developer said "i have never seen such bad code". Srsly! 3 new People said it and me as well.

\- Bad testability \- no tests \- hidden 3 bad bugs surfaced in just one year \- ...

I had very little trust in that code. And it was not much fun. On Upside: Cleaning it up was a great feeling.

</comment><comment><USER155> Oracle Database 12.2.

It is close to 25 million lines of C code.

What an unimaginable horror! You can't change a single line of code in the product without breaking 1000s of existing tests. Generations of programmers have worked on that code under difficult deadlines and filled the code with all kinds of crap.

Very complex pieces of logic, memory management, context switching, etc. are all held together with thousands of flags. The whole code is ridden with mysterious macros that one cannot decipher without picking a notebook and expanding relevant pats of the macros by hand. It can take a day to two days to really understand what a macro does.

Sometimes one needs to understand the values and the effects of 20 different flag to predict how the code would behave in different situations. Sometimes 100s too! I am not exaggerating.

The only reason why this product is still surviving and still works is due to literally millions of tests!

Here is how the life of an Oracle Database developer is:

\- Start working on a new bug.

\- Spend two weeks trying to understand the 20 different flags that interact in mysterious ways to cause this bag.

\- Add one more flag to handle the new special scenario. Add a few more lines of code that checks this flag and works around the problematic situation and avoids the bug.

\- Submit the changes to a test farm consisting of about 100 to 200 servers that would compile the code, build a new Oracle DB, and run the millions of tests in a distributed fashion.

\- Go home. Come the next day and work on something else. The tests can take 20 hours to 30 hours to complete.

\- Go home. Come the next day and check your farm test results. On a good day, there would be about 100 failing tests. On a bad day, there would be about 1000 failing tests. Pick some of these tests randomly and try to understand what went wrong with your assumptions. Maybe there are some 10 more flags to consider to truly understand the nature of the bug.

\- Add a few more flags in an attempt to fix the issue. Submit the changes again for testing. Wait another 20 to 30 hours.

\- Rinse and repeat for another two weeks until you get the mysterious incantation of the combination of flags right.

\- Finally one fine day you would succeed with 0 tests failing.

\- Add a hundred more tests for your new change to ensure that the next developer who has the misfortune of touching this new piece of code never ends up breaking your fix.

\- Submit the work for one final round of testing. Then submit it for review. The review itself may take another 2 weeks to 2 months. So now move on to the next bug to work on.

\- After 2 weeks to 2 months, when everything is complete, the code would be finally merged into the main branch.

The above is a non-exaggerated description of the life of a programmer in Oracle fixing a bug. Now imagine what horror it is going to be to develop a new feature. It takes 6 months to a year (sometimes two years!) to develop a single small feature (say something like adding a new mode of authentication like support for AD authentication).

The fact that this product even works is nothing short of a miracle!

I don't work for Oracle anymore. Will never work for Oracle again!

<comment><USER156> A sentiment among members of a former team was that automated tests meant you didn't need to write understandable code - let the tests do the thinking for you.

This, and stuff like your story, are why I don't trust people who promote test-driven development as the best way to write clean APIs.

<comment><USER157> TDD need to die. This is a curse.

There should be integration tests along with some property based tests and fuzzy tests. Usually catches a lot of things.Invest in monitoring and alerts too.

TDD is like relying on debugger to solve your problem. Is debugger a good tool? yes,it is a great tool. But using it as an excuse to avoid understanding what happens under the hood is plain wrong.

The problem lies in industry where software engineering is not given any value but whoteboarding and solving puzzles is.

Software engineering is a craft honed over years of making mistakes and learning from them. You want code asap, kick experience engineers get codemonkeys in and get a MVP.

Quality is not clever algorithm, but clear conscise logic. Code should follow the logic, not the other way around.

Clear > clever.

<comment><USER158> I think TDD is the best way to develop (yet). Obviously tests are code, and if you write crappy highly-coupled tests you will end up with only much more messy code. This is a clear example of bad testing. The greatest advantage of TDD is in design, everything should be modular and easy to unit test, so you could:

\- reproduce bug and verify your bugfix in matter of ms with proper unit test

\- understand what code does

\- change and refactor code whenever you want

You can tell from what is written that they are not following TDD. Redesign that codebase in an easy and clean to test design would require an exponential effort and time compared to have it done step by step, but it would be worth it

<comment><USER159> A unit test is the least useful kind of test. It requires your design to be "easy to unit test" instead of simple, and if you change something and have to rewrite the test you might miss some logic in both pieces.

Plus the tests never break on their own because they're modular, and each time you run a test that was obviously going to pass, you've wasted your time.

As long as you have code coverage, better to have lots of asserts and real- world integration tests.

<comment><USER158> Integration tests are much slower usually, and you are testing tons of things at the same time. Something breaks (like in that example) and you have no idea of what and why went wrong.

If you unit test _properly_ you are unit testing the business logic, that you have to properly divide and write in a modular fashion. If you want to test a more complex scenario, just add initial conditions or behaviors. If you can't do that or don't know how to do that, then you don't know what your code is doing or your code is bad designed. And that may be the case we read above.

Tests rarely break because they help you not breaking the code and functionalities, and they are so fast and efficient on making you realizing that that you don't feel the pain of it.

I can't imagine any example where "easy to unit test" != simple

<comment><USER160> in my work with python, easy to unit tests usually makes things a bit harder. You want functional methods , not mega classes with 100s of class variables , and each class method operates on some portion of those class variables. It makes it impossible to truly isolate functionality and test it. While coding though, it is very easy to make a class and throw god knows what into the class variable space and access those variables whenever... However if we have staticmethods , not reliant on any class , just the arguments provided, and it doesnt modify any class state, the test are great. We can change/refactor our models with confidence knowing the results are all the same.

</comment></comment></comment></comment></comment></comment></comment><comment><USER161> Thank you for the question it gave us so much knowledge in the replies.

Wish you all the best

</comment><comment><USER162> A million lines of mostly 90s style C++ codebase - works like a charm :)

</comment><comment><USER163> One of the startups I worked at had 3 frameworks in PHP - all grown, all messed up in all the wrong ways - and that was just the backend. We had a homegrown framework written in JQuery for the frontend as well. never Again.

<comment><USER164> And probably making a ton of money so biz didn't want to replace it.

</comment></comment><comment><USER165> Every line of code I've written over the past 30 years.

</comment><comment><USER166> 100k lines of semi generated, semi handrolled java/c

</comment><comment><USER167> I made scones with the wrong flour once. ( _¿_ )

</comment><comment><USER168> Amavisd. About 11,000 lines of crufty perl.

</comment><comment><USER169> The most insidious cases are those where bad processes are developed around bad software, leading to a vicious cycle of co-dependency.

I once worked for a reasonably large business that ran all of their invoicing and stock management through an MS Access project. The whole thing was wacky. half the business logic was implemented in VBA. The other half was stored procedures, but there was no obvious pattern (predictably, anything that involved a task the DB was optimised for was written in VBA). "Deployment" consisted of saving to a network drive and waiting until everyone opened it again in the morning. Data integrity seemed optional and inconsistent. The symbol naming convention could be described as cryptographic. The only documentation was a comment over each function stating the original developer's name and a timestamp. It had a proud splash screen stating "Developed by Dave Davidson" that was shown for 5 seconds on startup - <USER4> was of course completely simulated and there was no reason to have a splash screen. I could never quite fathom the magnitude of this guys delusions of grandeur or why they would want to put their name to it.

The worst part about it all was that for the most part, it worked. The parts that didn't work were well known to the people using it and worked around. So processes were developed around it, and over time these became so deeply ingrained in the teams using it that they couldn't imagine working any other way. Most of this consisted of taking telephone orders, printing off the invoices that were generating in Access, and then rekeying that information into an accounting system (for efficiency purposes of course, these printed copies were passed to another team member with notes scribbled on the original document. Mistakes were commonplace and accepted as a CoB).

Part of my role was to implement a web based ordering system. We did a reasonably good job but of course had plenty of our own WTFs. The biggest pain was integrating this with a particular team. They could not imagine a process that did not involved printing off orders and rekeying. After a while we realised that the reason we got so much push back was that once fully integrated, our system would make half of the team members redundant.

With support from management we went ahead, and over time the wrongs were righted. When I left there was still a deep level of distrust in the new system. Mistakes that were daily occurrences in the old world were "proof that the new system won't work". Orders were still printed "just to make sure I don't lose it". New bugs were treated as if the sky were falling. I would spend more time managing expectations than writing code. But we got there in the end.

Buggy software can be fixed. Buggy humans are an entirely different kettle of fish.

</comment><comment><USER170> Windows.

</comment><comment><USER171> SAS Viya. Enough said.

</comment><comment><USER172> git ls-files | xargs wc -l

<puts face in palms, starts weeping gently>

<comment><USER173> cloc .

</comment></comment><comment><USER174> The Internet.

</comment><comment><USER175> While not related to LOC, warranted nonetheless: [https://xkcd.com/2030/](https://xkcd.com/2030/)

</comment><comment><USER176> The entire Athenahealth stack.

<comment><USER177> Perl, right?

<comment><USER176> The whole damn thing.

</comment></comment></comment><comment><USER178> if it works, it's not bad code

<comment><USER179> I don't think I want to work with you, ever :)

</comment></comment><comment><USER180> My own CMS. Here's the function signature of the heart of it, which depending on circumstance may call itself recursively.

        function a_nodes_list_trees(
        	$site_url,
        	$base_url,
        	$mode,
        	$site_id,
        	$max_subtree_depth,
        	$nlevels,
        	$node_id = 0,
        	$tree_id = 0,
        	$nleft = 0,
        	$nright = 0,
        	$nlevel = 0,
        	$current_subtree_depth = 0,
        	$flags = 0,
        	$order = 'tree',
        	$inline = false, /* needed when loading stuff via ajax*/
        	$skipped_types = [],
        	$items_per_page = 10,
        	$min_tag_count = 2,
        	$min_author_count = 2,
        	$is_site_root = false,
        	$skip_pagination = false,
        	$ignore_404 = false,
        	$skip_display_options_and_batch_ops = false,
        	$display_bottom_pagination = true)

As you can tell from the default values, it started out having 6 arguments. And it has things like this in it:

        	$mysql_the_rest = 'FROM
        			'.DBTP.'node n
        				LEFT JOIN
        					'.DBTP.'node n2
        						ON
        							n.tree_id = n2.id
        						AND
        							n2.perm_view & ' . A_PERMS . '
        			'.$mysql_join.'
        		WHERE
        			'.$mysql_where2.'
        			'.$mysql_where.'
        			n.perm_view & ' . A_PERMS . '
        		AND
        			n.site_id = ' . $site_id . '
        		';
        	$total_count = $A->db->fetch_one_column('c', ' SELECT COUNT(*) c '.$mysql_the_rest);

.. you know? The whole CMS is 20k lines of PHP, with HTML, PHP, and MySQL all happily living together in the same files (it's not that I don't have templates, I just have plenty HTML in the PHP, too)

Yet, it works like a charm, PHP updates made it faster even, and I can use it for everything I needed so far, and use its output in a variety of ways. I still want to rewrite it, but it seems a lot of work to just shave off a few ms and have nicer code, with the same result for the visitor, and also having to write something that migrates the content. I suspect with enough content, it will slow down, and then I'll think about the next iteration. But it's still a mixture of pride, plain being happy to have it, and groaning whenever I fix a bug or add a feature.

<comment><USER181> An easy refactor for your function is to use a "Builder" for your params, i.e.:

options = NodeBuilder .author({ min: 2 }) .flag({ .. }) .pagitation({ .. })

function a_nodes_list_trees(options) { ... }

Because in the end, if your function is fast and working correctly, it's fine even though it's a little bit messy. The problem is more all the code calling this function and having to pass dozens of params in the right order, and then it's a pain to start adding/changing parameters. Also, when calling this function, you probably need a bunch of temporary variables to "build" all the params; all those temporary variables could instead live inside that builder object.

<comment><USER180> I think I'll just use arrays, one for the caller, and one globally that has the default values. Fortunately that function isn't called in that many places.

Making and then really using the CMS helped me with knowing what I would want in my next CMS, and just keeping the whole in mind from the start would probably make it a lot better by itself. My thinking is that the longer I put that off, the more languages improved and the more I hopefully learned in the meantime ^^

</comment></comment></comment><comment><USER182> Windows</comment>