Ask HN: As a CTO, what is your most frustrating problem with technical debt? - <AUTHOR> _or as a person in a CTO-like role_
Ah, the technical debt.
While the concept is sweet from the business perspective, the result of leveraging technical debt, most often, is a big mess.
The technical debt is accruing every week, after each iteration, sprint, what have you. Your developers are whining about it, and they want to do a significant rewrite or solve it with microservices altogether.
And you know this rewrite, like others in the past, will be a significant sink of time, and likely a catastrophe.
This is how I see it from my humble developer/team lead perspective. I'm really curious to know how people in CTO roles perceive it.
I agree that this is a useful quality lever that can get the whole company out of a bind, or could let business leverage a market opportunity before the competitors do.
What I've found is that people in various project-manager-like roles tend to overuse this lever. I believe this lever should be used only when there is a significant gain for the business, and the debt should be paid shortly after that.
What I've personally seen, and what I've been told by my fellow developers, is that it is common to use the technical debt lever just to:
- reach the "sprint commitment,"
- meet the artificial deadline,
- increase "developer's productivity,"
- show manager's power.
These don't sound like critical high-value goals to reach, given the price of the Technical Debt.
That is my perspective. Now, what do you have to say?
As a CTO (or in a CTO-like role), what was your most consistent pain, frustration or problem with Technical Debt in your company in the last 2 years?
Have you ever had any?
Thank you for your time reading this and giving a thoughtful response. :) <comment><USER2> I think a lot of the time when a developer shouts "technical debt" what they are really shouting is "code someone else wrote that I'd rather rewrite than understand". (The rest of the time is the same but they've understood it enough to think it's a disaster area.)

I have found it's best to not take tech debt complaints very seriously and instead look at actual success metrics. For example if every change to a bit of code introduces new bugs then that might be a reason to tidy it up.

<comment><USER3> > "code someone else wrote that I'd rather rewrite than understand"

I'm not disagreeing, but there is something to be said for writing code that any idiot can understand so that future developers don't even _want_ to rewrite it, and can maintain it.

I've landed in code bases on both ends of the spectrum, and I never asked management to rewrite the well-written ones.

<comment><USER4> In a wicked way, people who write that kind of code never get credit for it, and often lose leverage of their job.

I am not saying that writing convoluted code is good, no, but I can tell, even if based on personal experience, that barely anyone talks about "wow, this is such a straight forward code" or gets promoted for "everyone can understand your code!".

It is a weird dynamic that gets worst by having non-technical managers, and the reason why I believe team leaders should be hands-on in software development and involved with code review.

<comment><AUTHOR> Let me get this straight:

Person A, a programmer, who writes easy-to-understand code, but takes a bit more time to finish their tasks

Person B, a programmer, who writes dirty code, but takes much less time to finish their tasks

The current non-technical manager will see only one part of the picture and promote Person B. Moreover, Person A might get in trouble, and pressure to "compromise on quality."

Within a short-term, it is understandable.

Within a long-term, this is a disaster scenario!

So should the "code quality" be part of a performance portfolio of the developer when it gets to promotion time? And how do we get there?

<comment><USER5> I've also seen

Programmer A writes straightforward code that's easy to change and understand.

Programmer B writes convoluted "clever" code that completely falls apart when requirements change, or even just in production because their clever solution didn't account for all of the edge cases.

Now Programmer B has to save the day consistently, and therefore is seen as the person saving the company, when the problems were all of Programmer B's construction to begin with.

<comment><USER6> I would expand upon this and say that Progammer B doesn't ship the code at all, it wouldn't pass code review muster.

Unless we're talking about a seriously silo'ed org, or Programmer B is actually "Team B"

<comment><USER5> A lot of teams are a cult of personality around one developer. Particularly if there are a bunch of juniors on the rest of the team, Programmer B can usually get anything through code review with the 'ol "of course it's complicated, that's why _I_ am the one working on it. It's not for mortals to understand".

<comment><USER7> Treat every question about your code as feedback. If multiple people ask the same question you have a problem that needs to be fixed. Documenting the gotcha should be your last resort. Implying incompetence isn't even in the board.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> I've been the CTO at two tech startups. Here's how I think about it.

"Debt" is a good metaphor because by accumulating it, you get advantages right now in speed and ability to focus on other things, in exchange for having to pay it off later. You can either pay it off by stopping what you're doing (uncommon) and fixing stuff, or you can do lifetime installment payments of more complexity, bugs, etc.

Tech debt is just one aspect a CTO has to manage. Time to market is pretty damn important. Features are important. Many other things too. This is not to say that tech debt should be ignored, but elegance of the system is not the paramount concern. Just as startups borrow money and take financing to move quickly, so too a responsible amount of tech debt can be a good thing.

In engineering we tend to over-emphasize how important the technical qualities of the system are because that's what we see every day. There are a great many things that are also important, and various business scenarios where they become MORE important than tech debt. But if you're in the stage of trying to find product-market fit, you probably have bigger fish to fry.

The biggest tech debt I've struggled with always comes from the same patterned source. You think a system is intended to have a certain feature set for a certain user base, and so you make a set of architectural decisions to match. The world moves on quickly, you adapt, pivot, whatever you want to call it, and suddenly your carefully made architectural decisions are no longer correct or wise for your new scenario, but there is no time to go back and have a "do- over".

So in essence, failure to be able to tell the future, paired with a constant need to move forward, is the source of the problem. Both of those things are going to keep happening to startups forever.

<comment><USER9> In other words, it's not a problem to be avoided, but rather a problem to be embraced.

If this is indeed true, how does one fully embrace it? Optimize for easy refactoring? What does this look like in practice?

<comment><USER8> No, I wouldn't put it that way. It's not a problem to be embraced, it's just another variable to manage.

Just as there is the trilemma of "better, faster, cheaper: pick 2" there are others as well. Speed and tech debt are balanced against one another. It's not that I think we should embrace tech debt. I actually rather hate it. But sometimes I'm willing to trade to get to a business objective that is not related to technical elegance.

In terms of how to pay it off, it may sound like a cop-out but it just depends. The core problem is that things are always changing and we can't predict the future. So staying supple and flexible I think is the way to go. Pay off as much of it as you can when you can, but I think the main point for me is to keep the eyes on some business objective (where tech debt appears as only one variable in the equation) -- the main point is __not __to keep your eyes on the tech debt all the time.

</comment></comment></comment><comment><USER10> If you believe technical debt is developers wanting the code to be "perfect", you completely misunderstand what technical debt is.

Technical debt is a collection of choices you and your team make that can have real world impacts on your companies ability to deliver features in the future, can open your company up to hacking, and could make your software less competitive than the rest of the market.

Technical debt is not a loan, it is an anchor. Instead of deferring the debt payment until later, you pay for it with every new feature and eventually your development team will slow to a crawl or the product your building will no longer be cost effective due to rising maintenance costs.

Technical debt is a unknown risk. Look up any "X company was hacked" story and they have all come down to bad choices made by the product team. It could be a developer on your team left there machine unsecured or put in some bad code, or it could be a deferred decision to fix that technical debt until later.

Technical debt confines the space in which you can operate. If the market pivots, but your software is too rigid to keep up with the changing shape of the market you will lose market share or the ability to capitalize on new things.

All companies will have to make hard choices when it comes to technical debt, and no product can be debt-free. But understanding that technical debt is more than your developers whining about code is paramount to being an effective leader in your organization.

<comment><USER11> > Technical debt is not a loan, it is an anchor.

This is a great analogy. You can still sail a ship with one anchor, maybe two, but eventually you are going to have so many anchors that ship isn't going to ever get to its destination.

</comment></comment><comment><USER12> I try to use an economic model for all software. All code is a loan against taken against entropy (a liability).

Best solution is no code or a simplification of the problem to existing solutions. This results in zero liability, while the revenue the solution generates is pure assets.

When a code liability does need to be incurred, consider both the amount of code written (the principal of the loan) and the architecture of the solution (the rate of compound interest). A good programmer will work to reduce the principal, an excellent one will work to reduce the rate of compound interest with a simpler architecture.

The architecture that is does more by adding complexity and conditionals / code contains a horribly high rate of compound interest, even if it seems easier to do at first.

Architectures which do more / everything by virtue of being simple may be more difficult to implement and understand (especially without some background knowledge) but can have a vanishingly small rate of interest. Double entry bookkeeping, graph theory, pipelines, normalisation/denormalization, map/reduce, unidirectional data flow etc.

For any serious business investment (not an experiment or proof of concept), architecture as rate of interest always trumps quantity of code as principal.

Order of priority is 1\. Don't take loans 2\. If you must, design until you have a low and manageable rate of interest. Revisit often. 3\. Reduce amount of code.

<comment><USER13> > Best solution is no code or a simplification of the problem to existing > solutions. This results in zero liability, while the revenue the solution > generates is pure assets.

Not exactly _zero_ liability, but the source of the liability is pushed outside the bounds of your own code, such that the burden is shared among far more people than you can employ on your own. Point being, you may still find yourself contributing code to these shared solutions, but that's a good thing, because such external work will have a wider-reaching impact than anything you build internally.

</comment></comment><comment><USER14> I run a bunch of websites on top of popular, open-source CMS software like Wordpress or Drupal.

An under-appreciated source of technical debt in those sites are plugin/module upgrades--the ones that are not security patches. Most developers and agencies seem to take the approach "if it's working for us, and not a security issue, leave it alone."

That approach works fine until there _is_ a security patch. Then you have to apply it fast, and guess what--it comes with all the previous code changes you chose to avoid. Now you're getting all those changes all at once, and under time pressure.

Keep your plugins and modules up to date! It might mean some level of ongoing development to deal with changes in functionality. That's a good thing. That's a muscle you need to flex in order to keep it in shape. If you only update modules when there's a scary security issue, some of that technical debt is going to come due when you least want it to.

Another underappreciated source of technical debt is the differing preferences among developers and agencies on how to use the CMS + plugins/modules to build a site.

Ok you're building a Drupal site... do you build your layouts with custom template files? Context module? Panels and Panelizer? Paragraphs module? Do you handle embedded media in fields or WYSIWYG? Do build your sidebar with blocks or with Views + nodes + Display Suite?

You can build the same website different ways, and it will work fine. But a legitimate choice can often look like awful technical debt to someone else, who would have made different choices. You have to take this into consideration when hiring new developers or agencies to work on an existing site. Otherwise you can waste a lot of resources rebuilding functionality with no discernible improvement to the end user.

<comment><USER15> That's what backports are for.

[https://en.wikipedia.org/wiki/Backporting](https://en.wikipedia.org/wiki/Backporting)

<comment><USER14> Backports are very rarely provided by the community for older plugin/module versions.

Backporting by hand, on your site, makes technical debt worse. Now, not only are you not up to date with the community code, but you've got extra custom code that will have to be integrated or torn out to catch up.

In terms of competencies: if a team is struggling to integrate and deploy community-provided code, how good a job are they likely to do writing, testing, and deploying custom code? Especially in a hurry.

That's what I mean by flexing the muscle. A team that updates their site's code frequently is going to be a team that understands their application better, has more complete test coverage, and deploys with more confidence. And if custom code is required (like a backport), they're going to be better equipped for that too.

</comment></comment></comment><comment><USER16> There are a few side-affects of technical debt that drive me nuts. These come from companies where developers are autonomous, self-choosing of time investment:

* continuous complaint without action (i.e. pretending to be a victim of the system you've built)

* lack of measurements for a system (technical debt or not, how is it working? Are you improving your measurement system after each failure to further understand the system?)

* unbuilt is better than unmaintained (this builds off the previous two — tell me what we learned from our previous attempts. If you didn't understand the quality of the previous attempt, then how do we know if we actually improved?)

I'm down with fixing technical debt, just show me some metrics that prove that it was an improvement.

<comment><AUTHOR> > pretending to be a victim of the system you've built

Nicely put! I haven't thought of it this way.

> Are you improving your measurement system after each failure to further > understand the system?

Oh my god. Yes. This is a quote that made my day.

So far, it feels like we need to:

1\. Have a concept of area (and maybe sub-area) of the code-base

2\. Do frequent checks how painful it is to work with a certain area. Maybe, after every feature (ticket, what have you), make developers record how they felt about the areas that they've touched, and why. For example, this was really painful because this class is just too coupled to this other one or something like that. Track this information.

3\. Track frequency and size of changes required to different areas in the code right now (+ anticipation in the next period, whatever it is).

4\. Regularly have a technical retro, where you look at the areas, and prioritize areas that:

      * give the most pain AND
      * need to be frequently changed now and in the future

5\. Then go through the list of problems and choose the most horrible ones to resolve

6\. Schedule time to resolve these, just like we do with features/tickets/what have you.

This is just a draft of what comes to my mind.

<comment><USER17> This is very similar to what my startup is looking to help with.

</comment></comment></comment><comment><USER18> For the last 5 years I've worked primarily with startup CTOs and enterprise directors/VPs to build teams practices XP, lean product development, and user- centered design.

My advice to them, and my philosophy on this topic, is that technical debt is just that- debt. It's not a dirty four-letter word to be avoided, it's something to be managed the way you manage actual financial debt. There are some projects where the software you build will be self-contained (no dependencies), will require minimal changes in the future, or it may have a short shelf life (e.g. it's a stopgap before you cut over to a new system). In these scenarios, accruing technical debt isn't the worst thing in the world if it allows you to reap business value faster.

When you're building software that's going to live long-term, or that drives core parts of your business (and therefore will need to change frequently as your business changes), technical debt needs to be tackled early and often. It's ok to accrue technical debt in the short term ONLY if you make that decision consciously knowing that you'll need to address it later (e.g. I had a client that needed to release their software for a trade show, so we accrued tech debt and created tasks to track every piece of debt we'd want to tackle later, and then prioritized that work immediately after the trade show).

That said, it's also important to be clear about what constitutes tech debt vs what is over-optimization. Outside of libraries and frameworks, I don't really need the software to be "perfect". I need it to be clear enough, tested thoroughly enough, and easy enough to change to maintain a high velocity. If a part of the code has become spaghetti-like, or if we see code duplicated more than 3 times, or if we have disparate code that's too closely coupled to easily make changes, that's technical debt worth tackling.

The vast majority of organizations I've worked with typically think much shorter term, which is why they find themselves repeatedly accruing technical debt to the point where changes to the system occur at a snail's pace. It takes a lot of discipline, and coordination between IT and business, to practice what I'm describing, but that's the ideal that I strive for and advise my clients to strive for.

<comment><USER19> " _It 's not a dirty four-letter word to be avoided, it's something to be managed the way you manage actual financial debt._"

As others have pointed out, technical debt operates the same way an unhedged option works. It can go catastrophically debt. Compare that to issuing bonds. If you issue $10 million in bonds, you know exactly how much you will have to repay, and you know the interest, and you know when you will have to repay it. With technical debt, you don't know when you'll have to repay it, and you don't know how much it'll cost you. You just go along knowing that someday something catastrophic might happen. See this previous discussion:

[https://news.ycombinator.com/item?id=8777237](https://news.ycombinator.com/item?id=8777237)

<comment><AUTHOR> Technical debt is very similar to financial debt.

Don't get fooled. There is very small, and very important difference:

With financial debt, the business takes on it in hopes of delivering more value quicker that will pay for interest and principal, and have a lot of (monetary) value left after.

Financial debt, if anything, is speeding the process of delivery of value up.

On the other hand, technical debt is slowing the delivery down. And this delivery is what business hopes to speed up.

Well, maybe it makes this one iteration/release quicker, but it slows down the next one, and next one after that, and so on.

So, it seems that they are deceivingly similar, but you need to adapt the tactics and strategies.

<comment><USER20> I fail to see the difference. Financial debt is also a brake on future activities, in the sense that you have to pay back with interest, so less funds are available for other activities.

So in your words: it makes this one iteration quicker, but slows down the next one...

<comment><AUTHOR> I see what you mean, and I agree partially.

Follow my thoughts (and please do poke holes in them!):

Financial debt allows you to deliver more -> sell more -> pay down debt quicker -> deliver more (if you got it right and haven't failed).

So the money itself becomes a mechanism by which you deliver faster, and, as a result of faster delivery, earn more money to deliver even more and faster.

Now, the technical debt.

Earning more money from more feature delivered in current iteration might result in more revenue NOW. But does it result in more/faster delivery after that revenue happened?

Not really.

You can potentially hire more people. But they will only slow the team down at first. There is a delay between "more people" => "faster delivery."

So financial debt can enable a positive reinforcing loop. Can technical debt do that?

</comment></comment></comment></comment></comment><comment><USER21> The most frustrating part is explaining technical debt. From a business perspective you have a running software. You can sell it - it has all the features needed. Now you come along and tell me you need to fix something, that no client can see, that does not add any features and it might even lead to bugs. And you need two people and x weeks. I can imagine how "technical debt" is perceived as "snake oil" or "developer fun time" for management.

To avoid frustration it is best to fix technical debt in small steps, mix it in with others open issues. Devoting developer time for a big technical debt rewrite is hard to communicate.

<comment><USER22> The way that I've explained it with higher degrees of success is: "Feature X is used by a high percentage of users and also contributes to a high amount of bugs. We can invest two people and x weeks to bring those bug numbers down and product quality up."

<comment><USER21> Totally agree with you. As other people commented here. The key to communicating about technical debt is numbers. You need some metric to at least show that you have a problem and a number to show that you improved quality after reducing technical debt.

</comment></comment></comment><comment><USER23> Speaking of technical debt, you might want to remember the following 'handy rule' from the book Team Geek[1], chapter "Offensive" Versus "Defensive" Work:

 _[...] After this bad experience, Ben began to categorize all work as either "offensive" or "defensive." Offensive work is typically effort toward new user-visible features—shiny things that are easy to show outsiders and get them excited about, or things that noticeably advance the sexiness of a product (e.g., improved UI, speed, or interoperability). Defensive work is effort aimed at the long-term health of a product (e.g., code refactoring, feature rewrites, schema changes, data migra- tion, or improved emergency monitoring). Defensive activities make the product more maintainable, stable, and reliable. And yet, despite the fact that they're absolutely critical, you get no political credit for doing them. If you spend all your time on them, people perceive your product as holding still. And to make wordplay on an old maxim: "Perception is nine-tenths of the law."

We now have a handy rule we live by: a team should never spend more than one- third to one-half of its time and energy on defensive work, no matter how much technical debt there is. Any more time spent is a recipe for political suicide._

\- - -

Previously discussed here: [https://news.ycombinator.com/item?id=16810092](https://news.ycombinator.com/item?id=16810092) ("A Taxonomy of Technical Debt")

</comment><comment><USER24> I'm one step below the CTO, but here's my perspective for what it is worth. It really depends on the stage of development your company is in. When you're first starting out, writing "good quality code" is about the worst thing you can invest in. The fact is, the code you're writing is almost certainly the wrong thing. A new startup is a hypothesis on a business model, and the first bit of code is to make the most basic tests of the hypothesis. It's VERY rare you get it right your first try. Writing good quality code is a long term investment, but if there's no one willing to buy it, it's a bad investment that will never have an ROI. On the other hand, if you're rewriting the system because now you know the market, and you know what it wants and you're scaling to meet it, now tech debt is bad. I think it's critical to understand the first system can and probably will be thrown away, so it makes little sense to build it to "scale".

<comment><USER25> Given the uncertain business direction, you will definitely need to make the code easy to change. Low quality code inhibits that.

</comment></comment><comment><USER26> Since there are already quite a lot of responses to your original question I'll instead address a specific aspect of your post: when you have an old legacy system that you don't want to build on any more it is almost _never_ the right decision to try a one-shot migration or rewrite.

I have seen so many people try and solve technical debt with a full rewrite and it just never works. The better way of solving this problem is to gradually peel off individual use cases and API calls. It doesn't matter if your legacy system is riddled with bugs and is impossible to build on; it's at least a known quantity. Your new system will likely have less bugs in it, but until it's been used in the wild for a while you won't know where those bugs are. If your new system has a systematic problem, you are in some serious trouble, but if you migrated over slowly at least it will be manageable.

</comment><comment><USER27> People here talk about technical debt as equivalent to financial debt, but I think it is wrong. Differences:

1.) Financial debt does not slow you down, technical debt does slow down work. Depending on how messy the system is, it can kick as soon as 3 months into development. It can make project fail or slow to develop and it is insidious - it is sometimes hard to see that things could be faster from CTO position.

2.) If you have money it is easy to pay financial debt and it happens instantly. Even if you have time, it may be hard to impossible to fix technical debt. The engineers might just create bigger mess in their attempt to clean things up, they may cause bugs, miss whole features etc. But most often, they just don't end with all that much cleaner system, because the mess is result of broken process or other systematic reasons or missing knowledge.

3.) It is easy to know when you have financial debt. It is harder to know (for CTO) when you have technical debt. Some engineers complain about pretty much everything someone else wrote. Many decisions are actually subjective calls where there are multiple valid options and the issue is not so much "it is bad", but "I dont agree with decision". They will never all agree. Some technical debt complains are attempts to make oneself look like more attentive coder, to excuse perceived length of development or part of power struggle between engineers.

<comment><USER28> > Financial debt does not slow you down, technical debt does slow down work.

It slows down your spending, or it ought to.

Instead of spending money, technical debt is spending time.

<comment><AUTHOR> Interesting.

Then, in one month of time, you can potentially have 5000$ or 5M$ in finances available.

In time though, in one month, you can have only one month.

It is true, it is spending your time. But it seems like this time is much more precious resource than money.

What do you think?

</comment></comment></comment><comment><USER29> The Effective Engineer[0] has a chapter on technical debt, where he goes into how a lot of crappy code decisions are there for a reason, and how rewrites can be chaotic. If you have the time, I'd recommend giving it a read.

I frankly got a lot more pragmatic about asking for rewrites after reading it, and I feel it helped me to grow (mature?) as a developer.

Either way, I think there's a need to balance the need for employee self actualization needs which they often push as rewrite requests "Oh since we're rewriting this, we should (do it in)/use/etc X instead". I have often realized that a lot of requests to rewrite something are really tinkering desires camouflaged as a business related request, which is not to say that the code that does exist could have problems, it could, but having a period of debt repayment would improve it as well. So finding a way to allow your employees to tinker without letting their desires torpedo your products would be positive.

Either way, it's a complex subject, and I don't really think there's a single "right" response to it. Best of luck!

0: [https://www.amazon.com/Effective-Engineer-Engineering- Dispro...](https://www.amazon.com/Effective-Engineer-Engineering- Disproportionate-Meaningful/dp/0996128107)

EDIT: I'm not a CTO, I'm a developer.

</comment><comment><USER30> I notice at my job there were a lot of i need feature x to which we commonly replied i would like to build x but the horrible old y system is stopping me or we would just make things much worse to try and meld feature x into legacy/messy y system. I was a dev dealing with the legacy systems at my company for years and then i took over management and said we need to remove or clean up these legacy weak points so we can build features faster and more maintainable, so yes we started at the first day of this year cleaning up from the worst / weakest links first on. In 6 months we are done using 2 new hires to complete new work while 2 senior devs did clean up. 6 months compared to years of spinning tires was well worth it. Eventually the CEO gets sick of hearing excuses of why we cant build this or that or are wasting 50% of our time fixing bugs. Productivity is soaring now for multiple reasons at my company, testing, documentation, removing old systems and even though we have a small team every senior dev is leading a junior dev. Side note this is the first time we have hired junior developers at this company and its been a big pay off giving a lot more free time to the senior guys to work on the most important things.

</comment><comment><USER31> I don't have much frustration with tech debt itself. It's a normal, acknowledged, often understood and predictable side effect of moving fast OR weighing tradeoffs.

I actually get pretty annoyed when it's conflated with product debt. I don't know if that has a real definition, but for me that's where your market/audience has grown to need things you did not envision/never built. This is often labeled as tech debt, even though it has nothing to do with your coders. This is an organizational issue, and one I haven't fully conquered yet.

Tech debt should not be accruing every week, it should be a conscious decision. MVP's are riddled with tech debt, intentionally. You want to get the thing out there and start gathering feedback/information, you acknowledge you don't know the right thing to build, and will do so once you have more info.

I'm also annoyed at the junior dev that doesn't take time to understand a system, and shouts that it needs to be re-written. Most times that's just because the dev is junior, or lazy. It's actual tech debt when the system is too convoluted to understand, or you haven't documented.

I have no idea what "sprint commitment" or "manager's power" are supposed to mean. They both sounds kinda sick.

If you know a rewrite is going to be a time sink and catastrophe, why on earth would you do it? Also sounds sick.

Your org should be focused on business results. If you're focused on optics, things are not healthy.

You are not in a race with your competitors unless you're in a market that is itself racing to a commodity. This is a business strategy problem vs an engineering one.

Technical debt seems far less a problem to me if you have a moderately capable software engineering team. It happens, yeah. But as your experience grows you know the proper balance of paying it down vs new work. It comes down to measuring quality of your services from an end user perspective vs how much work on engineering and support teams to maintain the desired quality.

~7-8 years cto'in

<comment><USER32> > I'm also annoyed at the junior dev that doesn't take time to understand a > system, and shouts that it needs to be re-written.

That's something that your senior devs should be correcting through training and mentoring. Junior means he doesn't know any better, after all.

Speaking as a senior dev, one great method to impart this wisdom is to let the junior dev try and do it and see for himself what happens.

</comment></comment><comment><USER33> One big issue I have (and I'm trying various solutions around it) is code upgrade.

Let's say we are doing something one way - step 0, and we realize "hey, guess what' there's a better way" \- let's call it step 1). Now going forward all new code is step 1, and touches to old code is step 0. Later you realize - there's an even better way - let's call this step 2. Now going forward you have step 2, step 1 and step 0. This process continues - but people who worked on this leave - leading to technical debt. This carries on for a few iterations and now you are on step 50 (49, 48, ...).

I have no idea how to deal with this problem.

I personally feel more micro-services would be better - but I also don't want to jump the gun since we were not too big a company at that point (and being profitable with our runaway took all priority)

<comment><USER34> It's a hard problem to deal with. I try to make sure the "better" way is "better _enough_ " to warrant a change, knowing the technical debt that will ensue.

</comment></comment><comment><USER35> The biggest problem is that Product and/or Sales are looking for constant growth via constant new features. It is impossible to get them to truly understand that by meeting some arbitrary deadline that you've had to take shortcuts. To their mind, done is done and now we need the next Y so they get their Q3 bonus.

The next problem is that too often the tech team wants to do a complete rewrite every 12 months because they want to use hindsight to remake last year's system/feature today. The issue is that in another 12 months this rewrite suffers from the same problem and now has several more things to consider. Some technical debt will always exist and I would argue always necessary. Perfect is the enemy of good enough and software that isn't live isn't adding value to the customers or company.

My perspective for this was to treat some percentage of work each quarter to resolving technical debt but do this in an agile prioritized way. Just like always focusing on the feature that will add the most value to the next release, also pick the biggest manageable issue that will reduce the debt the most to the next release. In this way, you know certain technical debt will never disappear but you have a ranked list to always decide the things that are most urgent and correct them before the house falls down. If your team is large enough, it is also helpful to have a person or standing task to update and document this list with WAG options and costs. Of course, as in all things, ymmv.

</comment><comment><USER36> Technical debt should be quarantined, not structural. That's the one, most important thing to take away. A really messy view with tangled if statements and hacks to get things to work isn't great, but it's leagues better than a really messy controller because it isn't going to introduce ACL errors. A really messy controller is leagues better than a fucked schema / data model because it doesn't spread the poison across every endpoint in your application.

Some easy wins are to default to making something a "many", even if it is only a "one" or a "one or none" because it makes adding a second case much easier. For example, a user should have multiple roles at multiple projects by default, so business users can do fine grain control over permissions.

Another useful trick is to have default behaviour, but introduce a set of tables for custom settings. This makes it easier to make changes because you don't need to run a migration to add a column to a table, you just write a new record to the "user_overrides" table.

But at the very end of the day, taking on technical debt of any kind can be warranted. It all depends on the stakes. If you need to get something done in a week to close a company-saving deal, then fucking do it. Fix it later. The implied interest rate might be 1000%, but if it saves the company it is worth it.

<comment><USER37> One of my friends came up with a tongue-in-cheek name for your "default to many"-advice: [https://theharemrule.com](https://theharemrule.com)

After being given the advice couple years ago, I found it to be applicable to quite a number of scenarios me and my colleagues faced. In my experience its pretty solid!

<comment><USER36> Yeah, it's one of those tricks that I find a lot of developers in their 20s haven't learned.

There is another trick that I use when I don't control the dev team. For example, say the company has outsourced a pricing app to a third party dev shop, in this case any decision I can make that will make the app more "table-y" I make. So instead of defining control flow to determine price for any given object, what I do is I get them to make a "rules engine" that can operate on the major variables that are typically used to make decisions. That way fine tuning changes can be done on our side, and it doesn't rack up billables. I think this is called "table based design" but Googling for it isn't showing anything.

</comment></comment></comment><comment><USER38> I've been thinking how technical debt can be flagged / measured, and then managed. That is, a tool that can do a combination of:

Manual: Perhaps the person doing the code review can negotiate with the developer and agree that the code needs to be cleaned up at some point soon, but needs to be merged as is right now due to business needs, with some sensible score for the magnitude of the debt. When the debt does get paid later (stop laughing), the actual effort can be correlated with the debt score.

Semi-Automatic: The tool can take diffs from a pull request and show it to some other developers in the company and poll them on some subjective attributes like clarity etc. The score will be normalized based on other scores given by the person. Once in a while the app can even show nonsensical code in this poll to see how that is scored.

Automatic: 1. Flag the files that change frequently with issues, other pull requests, or 2. when there are long conversations and further commits to a file before a pull request affecting some files are merged (while accounting for some reviewers approving anything vs some reviewers nitpicking everything)... I have several more ideas like that.

Would you use a tool that does this? Do you already have something like that? Please feel free to email me your feedback, if you'd rather do that: email@example.com

<comment><USER39> We treat technical debt the same as any other issue. Put an issue on the board, decide on timing, assign it to a release, etc. I think the problem for some is that they don't realize they are creating technical debt.

</comment></comment><comment><USER40> Technical debt is inevitable. If you ignore it, it becomes worse. I've never been on a project without technical debt.

Of course not every little problem is technical debt. There are plenty of nice to haves, we could do X, etc. style improvements that aren't really that critical. My advice for those: just apply the boy-scout rule and improve things when you work on them. For the bigger stuff, somebody needs to step up and take decisions. Ultimately for big chunks of work related to technical debt, that's a CTO decision that involves taking into account the interests of both tech and business. A CTO must have the power to do this and be wise enough to use that power responsibly.

Stuff to watch out for as a CTO is when seemingly simple stories snowball into a lot of firefighting, bugs, or deployment issues. That's usually a good sign something is wrong. If that happens a lot, you are definitely experiencing technical debt. Firefighting eats away development budget. If your team spends half their sprint diagnosing weird issues instead of getting stuff done, cleaning that stuff up is probably time well spent.

</comment><comment><USER41> "Tech Debt" and "Code Quality" are myths. I know that is an unpopular perspective, but allow me to explain. To start with, for me everything is just a problem described in terms of its properties.

What are some examples of problems I have encountered that were worth solving?

1\. We have a server rendered site and an API. Changes to the server rendered site require matching changes in the API code effectively duplicating our efforts. If we re-implement the site as a single page app consuming the API we can kill two birds with one stone.

2\. The lacking test coverage of a critical section of code matched against the frequency of changes to that code results in a frequent regressions and delays.

3\. Unifying two divergent implementations will allow us realize a shared, multi-tenant deployment model for our partners. The load characteristics will give us the same point of presence for half the hardware.

Why are these worth solving? Because they have objective rationales. The third embodies a strategic direction with real bottom line consequences. The case can be made "If you want this outcome here are the steps" Then the only question becomes "Do we want this outcome?"

Many times "tech debt" and "code quality" are tossed around as easy justifications. They lack the intellectual rigor or organizational context to make a compelling case that its worth the opportunity cost. The person using those terms generally has strong feelings that its the right path, but can't really articulate why. In many cases there is no "why" and its more a of a "just because"

In my experience "Tech Debt" projects that are actually undertaken or not called "Tech Debt" they are called "Lets do this because its important to the business and here is why" projects.

</comment><comment><USER42> The primary source of our technical debt is launch impatience.

Specifically, that most stakeholders in a feature or project accept "mostly works, with the exception of rare edge cases" as sufficient cause for launch. This usually takes the form of "get it out the door as-is so we can promote it, and then refactor later".

This is actually somewhat reasonable in terms of balancing business and technical interests, and I can see the case being made for doing it.

The problem, of course, is that "refactor later" never happens because there is _always_ another project or feature demanding to be worked on. These same stakeholders might allow a day or two after launch to refactor, but then promptly decide that starting on Project B takes higher priority than refactoring Project A (which, in their mind, "works").

The above scenario covers _at least_ 95% of our existing technical debt.

</comment><comment><USER39> The core trade-off is "what does the business need now" vs "what are the long term goals". I only consider the technical debt that will block us in the future.

At any given decision point, you need to understand when technical debt is accrued and track it like any other issue. If you don't actively acknowledge technical debt from the onset, you place your business goals at risk.

In general, I see that the most politically charged projects are also accruing the most technical debt. They have high pressure, short time-frames, and scope creep out the wazoo. Those take the most of my time, and are extremely prone to failure unless there's a human shield to keep the heat off.

</comment><comment><USER43> >> what is your most frustrating problem with technical debt?

Actually using the product that has a strong scent of technical debt behind it.

Vendor provides updates (could call them minor or major) which are essentially colored bandaid patches on top of most critical issues without any significant improvements.

Likely caused by waves of temporary hit-and-run consultants hired to "save the day" and each of them done that to minimize personal effort before collecting the paycheck and leave.

Eventually the product needs to be completely rewritten or risk to become obsolete due to healthy competition.

</comment><comment><USER44> Poor or incomplete abstractions and code duplication. They usually go hand in hand. What typically happens is that one developer needs to solve a problem. He creates some abstraction (one or more classes) to help him solve the problem. He tries to make his abstraction reusable, but it's hard to do if there is only one use case for it. So the abstraction ends up with some essential functionality missing. Or worse, the abstraction may be poorly designed, so it's not reusable at all.

The second developer comes along with his own problem to solve. He looks around and finds that abstraction that solves a similar problem. He tries to use it, and it doesn't quite work for him. He could fix the abstraction, but that would require changing code where it's used. He doesn't want to touch someone else's code because he just wants to solve his problem. Or perhaps he just didn't realize how to use it correctly (because it wasn't well designed in the first place). So he goes ahead and creates a copy of the abstraction, modifies it slightly to fit his needs and voila - his problem is solved.

Couple of more iterations like this and you get poorly designed code being copied all over the place with slight variations. If there is a critical flaw discovered in all these copies, it becomes a major pain to fix. So naturally nobody wants to touch such code with a ten foot pole. In the mean time, the bad code gets duplicated more and more like cancer.

</comment><comment><USER45> In startups a lot of technical debt is more organisational debt and lack of role specific hiring and poor decisions at a higher level early on when you do have to move quickly. For every "full stack engineer" you want a real systems architect, a technical writer, a testing manager/writer, a db administrator/architect, a UX designer. Plus at least one person for serious taking care of early stage devops/sysadmin/IT so that whatever you have isn't fully evolved out of the side 5 minutes someone gives it. You don't want to give the most agency, to the most opinionated and outspoken developer, but the least experienced. Especially if they argue for heavy handed engineering solutions they think are "correct" and this is almost the first ever real job they have had post university. But everything they say sounds great, and its the "right" thing you read online so they must be a good engineer, unlike these other silently working peons. You almost always end up with something brittle everyone else has to work around, leaving a pile of adhoc architecture and code that cannot be reused. The original hotshot "architect" you promoted above the others then gets annoyed and loses a sense of ownership because their "vision" has not been met by all of these other insufficient "full stacks". This is the recipe of every toxic BS startup out there. The solutions are simple, know what the fuck you are doing building a team and dealing with software. Have people who are head of software who have done more than a masters in writing a ToDo app in Java one time.

</comment><comment><USER46> I have seen some very bad technical debt. Everyone talks about rewriting it, but what ends up happening is the company will just throw more people on it.

In a more perfect scenario, something like TDD would have been used and there would be a large test suite. There would also be some really nice well maintained requirements and technical specifications. Under this scenario it would be much easier to redesign or refactor the system. But given the reality of business, this is never the case.

</comment><comment><USER47> As a senior sysadmin who has frequently been in the role of CTO because businesses often don't even have a CTO... my number one pain was always when the technical debt requires more personnel in order to manage properly but upper-management refuses to hire more people for the role. I know your question seems more oriented towards the SV-startup/dev cycle, so my answer is more for general business (even ones without programmers).

My pet peeve though has always been structured cabling. I don't know if yall understand just how bad the cabling situation is is many businesses, but it's bad and for some reason they really don't like spending money on that particular thing even though it's one of the most important things for keeping a business up with the times (and the data rates that match).

So, in general, it has been a frustration with the communication gap between the technical team and the C-level and the board. Too often you either get a CTO who wants to "program with you" or one who is too much MBA and not enough tech.

As a sysadmin I have realized my number one failure was not working on my MBA- side more and thinking purely technical proposals and solutions would win the day. AKA I should have spent more time golfing and going to lunch so I had more power with the Cs.

</comment><comment><USER48> There are lots of reasons that TD happens but regardless of what they are, the easiest way to deal with it is to be open with each other. If the COO made a bad call on something, tell the technical team and work out the best way forwards. Same if a Contractor wasn't as good as you thought or a new framework promised lots but wasn't actually good at some of the detail.

When you can be honest about mistakes, tech people are generally more pragmatic than non-techies might assume.

In the other direction, I clearly communicate the "cost" of short deadlines or the need to finish something that is in a bad state so that the decision can be made with eyes open. If a quick fix now leaves us in a poor state to add new features then the CEO or whoever can accept that so that when they ask for something new, "sorry, system is too brittle".

In reality, there are many reasons things get rewritten so I also try and assure my devs that we don't live with these things forever, eventualy we remove the problems and create a load of new ones! I've rewritten our basic system 3 times in 5 years and only one of those was mainly down to tech debt (although we took the opportunity to update to latest tech etc.)

</comment><comment><USER49> As a developer, I see Sprint ends as an artificial deadline that people end up strictly meeting to the bare minimum. I find technical debt accrues most quickly with smaller stories.

Agile coaches will blast you for estimating workload too high and blame you for technical debt and rushing. They will say we need to refactor as we go along. And that stories will need to be even smaller.

Quite frankly, bigger changes are easier to review as a whole and there's less wasted time updating the test automation after each small change. Personally, I like to take a few days to review a change and I like to review a whole feature at once to see how it all fits together. It can be hard to see that until all the pieces are ready. Polishing it up before merging is also nice.

In the business world we are led by timelines and milestones and competitor products. We have fixed dates we need to get things out by. Ive not seen a business ever run on the agile principal of "take your time and release when ready and don't commit to dates but commit to features blah blah". And I've heard from a few coaches now that the whole organization needs to be agile for it to work because the business can't be waterfall while engineering is agile.

Small agile rant aside..

I see technical debt as natural and acceues because we are given limited time to do something that keeps our paycheques coming. It's normal for engineering to whine and for managers to push back. This conflict keeps both sides in check. Reality would stink if either side dominated the business. So don't feel bad about telling your engineers "no", but also give them enough independance to manage themselves as responsible professionals. Find that balance point.

</comment><comment><USER50> Not a CTO, but in an architect role and spend my life cleaning up tech debt or building guard rails so it's harder to accumulate.

The most important thing is to teach decision makers (be they product or other senior engineers) that tech debt is a downstream result of the decisions they make. That's as simple as keeping them accountable for their actions instead of hiding the mess under the rug.

Example:

If a colleague uses the word 'later' in a product discussion, pretend they said 'never' and speak up if it sounds weird.

"We'll get the new feature working with our most popular product areas and circle back for the others later."

"The feature is ready on time, we'll fix those P3 bugs later."

If you don't call out the use of later->never here, they'll get away with it and you'll accrue tech debt for no reason. If people want to defer part of their implementation plan, that decision needs to be on the books somewhere with their name next to it. If management doesn't see it, it isn't happening.

My most frustrating problem with technical debt is how difficult it is to shoehorn this bookkeeping into whatever process management or task tracking system the company uses.

<comment><USER51> > If a colleague uses the word 'later' in a product discussion, pretend they > said 'never' and speak up if it sounds weird.

I can't decide if this is really clever, or really misguided. Often "later" does mean "never", and sometimes that's ok, sometimes it's bad.

However, you can't do everything at once, and there are plenty of things that you really will do later, because you know that as soon as you implement one feature, a customer will request the logical followup, and you will do it.

</comment></comment><comment><USER52> I can't answer as a CTO-like person, but I'm in an Architect-type role and deal with our CTO-like person on a daily basis.

The high-level management cares only about "The Big Rewrite" because it is the only thing that gets visibility at that level of the organization and it comes with a large budget that everyone knows will be replenished for years as it gets spent. Our CTO-like person has fully bought into this idea.

We have too many projects and too few developers, so incurring technical debt is an everyday thing. It will be dealt with in "Phase 2" of the project, which all the developers know is something that never happens. From my vantage point - having one foot on the developer side and one foot on the management side, I see that the management thinks the developers are incompetent and the developers think the management is incompetent.

It's not a good position to be in, yet at the same time we have a pretty laid back culture, work less than 40 hours per week, great benefits, etc. So turnover is fairly low. Most of the developers have either a side gig or are going to school part time, or else they'd probably go insane.

</comment><comment><USER53> Ironically to me, the most apt application of "debt" to any source you do is not the hack or shortcut code, but the libraries you bring in. You are literally borrowing against the knowledge and expertise of someone else.

The _vast_ majority of the time, this is the correct thing to do. But _if_ you find yourself scaling to absurd levels, this is the code you will have to worry about. You will find that you almost certainly will start to care about all of the details of the network that you were able to ignore in the past. Connection and thread pools you were able to ignore, now need you to invest directly in.

The hacks? Just make sure you aren't worried about aesthetic changes. Keep people on point solving problems, but don't berate them for wanting things to look at certain way. Developer happiness is important. Often that rewrite of a module that you thought would take too long, can help morale so that folks are moving faster by the end of it. Just make sure to keep it goal oriented and for the love of all things good, don't let them get away without keeping feature parity for your customers.

</comment><comment><USER54> Not a CTO and I've never come close to having a conversation with one. As a developer, though, I've been burned by tech debt too many times to feel comfortable with it.

The biggest problem with technical debt is that everyone has a different tolerance for it. Some even have an appetite for it. This is a breeding ground for conflict.

The next biggest problem is that the people who take it on are rarely the same people who end up paying it off. The former tend to get showered with kudos whilst the latter experience stress and poor performance reviews.

IMO the worst kind of technical debt is that which is taken on to route around a business process ("Governance"). So it, sort of, contributes to faster delivery but the conditions which drive the choice tend to persist until a regime change so the compromise sticks around forever.

In an ideal world, you should only agree to take on technical debt if you are also presented with a repayment plan. And that leads to my final problem with technical debt: most people tend to assume that that last part will take care of itself.

<comment><USER55> > _IMO the worst kind of technical debt is that which is taken on to route > around a business process ( "Governance"). So it, sort of, contributes to > faster delivery but the conditions which drive the choice tend to persist > until a regime change so the compromise sticks around forever._

Can you give an example?

<comment><USER54> Client access to database: too much effort to get stakeholder sign off for api changes and regression testing so we'll just grant the client full access to the database schema. Now we can't change the schema without breaking zero, one or many clients.

DBA deployment processes: insist on taking system offline for cold backup. Every. Single. Time. Solution: make application run dynamic code which is stored in a database table. Effect releases by issuing UPDATE statements. It's hard to argue that an Oracle database can't handle one of these reliably so the DBAs don't require cold backups for these. Still, we no longer have compiler support so our tests had better be solid. Bonus con: the DBAs cottoned on and started doing the cold backup thing for these updates so we lost business agility too.

Finally: a legacy web service. No toolchain but we do have WSDL and source. Ported to Weblogic in 30 minutes. Fits in with current infrastructure, more performant and gives company an opportunity to decommission the old box. But PM hasn't got a Jira for this work and we're already into integration testing so throws it out. The architect resigns.

Governance. A.k.a. Insurance-Against-Corporate-Liability. Responsible for at least 3 fucked up systems during my tenure there.

</comment></comment></comment><comment><USER56> IMHexperience... There's no complete escape from technical debt, there's only reshaping / transforming and moving it elsewhere where it does not hurt (well, defects/bugs do not count as technical debt).

In most cases this fix is by rising complexity - as number of (meta-)levels of the system (mind you, that is always humans+machines system, stretched in time). So what is being used everyday by everybody becomes a piece of cake.. and noone is to touch the magic interpreter that lets it happen. And funny(or sad), that magic thing could also be done perfectly by itself.. so it looks like there's no technical debt... Just the experience-power needed to understand it, is like 100x what is needed for the easy stuff it has made possible. So, essentialy, technical debt has turned into "experience" debt. a Futures contract of a kind... or maybe i'm just too pessimistic..

</comment><comment><USER57> Tech debt, whether creating it or paying it is a critical function of the business. Where this gets messy is when an engineering organization feels they own or somehow should prioritize the debt. This is especially true of paying it.

In every case that I've been involved in or had explicit oversight in creating tech debt has been when customer and revenue obligations are needed to be met. It's very rare in startups to have artificial or made up deadlines. I'm sure they exist at larger orgs, but as a startup executive, I can say with confidence that everything is constantly on the line, each day is critical, and so shipping product is paramount.

Of course, you have to pay that tech debt back at some point, but this is a business decision. It may turn out the business needs to kill that product or alter it in such a way that paying back debt is useless.

</comment><comment><USER58> Technical debt should be paid one commit at a time.

You cannot pay it at once. Just make many small continuous improvements.

</comment><comment><USER59> Slight tangent, but I personally don't like the term debt.

It's an attractive analogy but it doesn't go very far; worse, it leads people to think of it as easy to reason about.

The biggest issue by far around tech debt, which has been covered thoroughly throughout this thread, is that measuring it is tricky.

Measuring financial debt is trivial, you can put a dollar amount on it. Measuring tech debt is very subjective, as it depends on the level of seniority of your team, their ability/authorization to make changes (in banking, you can change nothing, in a startup, you can change everything), attrition rate, etc...

It's such a complicated equation, that using the term debt tricks upper management (non technical) to think it's easy to predict: it really isn't at all.

</comment><comment><USER60> As the team gets larger AND/OR the product attains its necessary features, "technical debt" becomes more of a problem. I would love to address it at every stage of a product, but it is simply not possible, many times during early development our team is just figuring things out, seeing what works and what doesn't. If we were to analyze future development for each iteration during this stage, it would be a nightmare. I like to trust in my team that they have a good understanding of best practice and choosing between tough work and future work. When the product "feels" at a good stage, say a few bug fixes after release, we spend much more time refactoring.

</comment><comment><USER61> Personally I find that the term tech debt gets overused. I've often heard it applied to relatively fresh projects for features/projects that just haven't been done yet, or to pitch the architecture of the day.

The times when I have seen a real breakdown where the term tech debt is legitimately used is when a scrum process breaks down deliverables below their smallest atom. E.g. different stories to write the code and unit tests, projects spread across devs such that no individual has ownership of any portion of the product etc.

In such situations we're not talking about debt as much as we're talking about junk.

<comment><USER2> Yeah I always laugh when there's a project with no code written yet, but with plans for how we will "pay off the debt" that will inevitably be created.

It doesn't need to be like that if you run your projects well.

</comment></comment><comment><USER62> Technical debt is code that makes everybody waste time (eg. because of its complexity), or that creates problems in production (eg. performance issues) BUT that you cannot easily replace because it is too tightly knit to the rest of the architecture, or it would be very long/risky. So you have to live with and pay its dividends every year.

The most frustrating thing for me is that you cannot throw it away and start from scratch (if you want to keep your users). You have to bite the bullet and find ways to make that ugly/smelly code go away in very small increments. This process can take years.

</comment><comment><USER63> I think it is important to understand that technical debt is very difficult to define and quantify. Depending on your role, responsibility and skillset you will consider technical debt to be different things, and often attribute different (negative) impact of it.

Technical debt can cripple a development team, but I have also seen it being used as a continuous argument to invest into system development that really had zero positive business impact at the end. I am not saying to ignore it, but I also know that as a developer you tend to overvalue the things that are just in front of you.

</comment><comment><USER64> I find that it doesn't get too frustrating if you don't let it get out of hand.

Routinely prioritize keeping dependencies up to date, clearly depreciate "old" / bad code, and focus on coming up with good product designs that have reasonable compromises between time-to-market and high quality engineering.

You just can't let stuff slip. Don't let 6 months go by without at least _trying_ to upgrade to the new version of whatever or without making some _some_ progress moving from the "old way" to the "new way" for some component.

</comment><comment><USER65> I believe people are mis-framing the tech debt debate or at least missing part of it.

Tech debt has to be managed and triaged, this is true. Tech debt slows down product development and degrades stability, also true.

But tech debt is also a talent retaining and hiring tool. No one enjoys to work on a project that gets in your way for reasons that can be resolved. You need to pay down tech debt when people ask in a reasonable fashion otherwise, simply put, they'll leave.

</comment><comment><USER66> As a follow up, what are the most useful methods for managing technical debt without incurring too high of a cost to team/product/project velocity?

<comment><USER67> Address issues as they are raised. Don't wait too long to get rid of debt, and most certainly don't rely on fixed time cycles of fixing things. A team has good sense of the issues they deal with, and you can capture their view in retrospective meetings. Depending on how bad the debt is, you should allocate time to improve. Velocity will only get worse as debt grows, so there is a clear business case for such stories: the more debt, the slower the velocity and the lower the quality. Spend time now, to speed things up. But just don't do it from a theoretical perspective. Address real issues, set real goals and measure them.

</comment></comment><comment><USER67> "And you know this rewrite, like others in the past, will be a significant sink of time, and likely a catastrophe."

To avoid this establish clear goals for the rewrite phase and measure results. Don't just rewrite for the sake of rewriting. Have a session where you identify issues and how to tackle them.

From a CTO-Like perspective, getting rid of technical debt should happen frequently, while keeping business goals in mind.

</comment><comment><USER68> I think technical debt should be called "management debt" because management is the primary source of such debts. Technical debt implies it something engineers forgot to do or were just lazy to do it. While it maybe the case in some places , mostly its Agile/Project planning that lacks any appropriate place for fixing technical debt and refactoring.

</comment><comment><USER69> I never liked this term because debt as we commonly relate to it, especially in business, is a quantifiable obligation that can be discharged.

In contrast "tech debt" is not a thing with a quantity, no one can know when or even if it will come due.

</comment><comment><USER70> Legacy unsupported open source projects with outdated dependencies. (We work on a very large Java application (1.8M lOC) and were acquired by large financial institution that has a lot of policy around open source libraries and support.)

</comment><comment><USER71> Not a CTO.

1\. Not only legacy code eats resources that can be thrown on new developments. While legacy code is most troublesome, it still will be at maximum only twice as bad resource-wise, than new and pretty code.

2\. As company grows, it may well be that a piece of software being used by by less than 5 people at the company, but require an own developer team to continue actively support it.

3\. Maintenance resources vs resources spend on development is a never ending drama.

4\. You should proactively think how to cull you internal projects, and be able to dictate to the rest of the company to stop using it.

5\. Custom software for "business guys" \- an easy way to multiply IT support budget. Hiring technically illiterate analysts, finance people, supply chain people costs a lot, a great great lot.

6\. Last point. THERE SHOULD BE AN IRON WALL in between people making "money making code" and all other tech at the company. Can't stress this more.

<comment> frant-hartm Could you elaborate on the last point?

<comment><USER71> Well, the most extreme case will be for example: in an online ads company, somebody disrupts the team responsible for bidding software to do somebody's pet project, do a UI job, or making them do software for internal IT needs.

</comment></comment></comment><comment><USER72> A point I often make about technical debt:

[http://codemonkeyism.com/ceo-to-cto-what-is-your- rewriterati...](http://codemonkeyism.com/ceo-to-cto-what-is-your- rewriteratio/)

</comment><comment><USER73> By biggest frustration with technical debt is that, at some point, it becomes so great, that the developers are more busy maintaining the existing code than they are building new, business critical functionality.

</comment><comment><USER74> Given a software product with a modicum of market fit and or an established user community, AND a competent engineering organization, there is no reason to have technical debt in 2018.

</comment><comment><USER75> Nine times out of ten the issues are being exaggerated and can simply be ignored. That's not what the engineers want to hear of course. They are eager to fix things, to stop their daily grind.

It's very rare to actually get stuck due to technical debt. The times I've seen it happen, we grabbed all the (relevant) developers, came up with a solution, and fixed it quickly.

As long as you don't have high turnover, technical debt is fine. You have a team who is aware of all the quirks of the codebase(s) and knows how to deal with it.

<comment><USER55> > _As long as you don 't have high turnover, technical debt is fine._

"If you tell them to shut up enough times, eventually they will."

> _They are eager to fix things, to stop their daily grind._

Yes, because being ground down daily will eventually wear you down.

> _You have a team who is aware of all the quirks of the codebase(s) and knows > how to deal with it._

But that's not a good thing. If your technical debt adds an hour to a dev's day, every day, you're running at ~85% capacity. If you have tests that need to be babysat, or code that takes a long time to add new features to (or fix bugs in) because of tech debt, you're losing valuable time.

Even if it's not necessarily the sort of thing that brings a codebase to its knees, or the kind of thing that sinks a business, but it is the equivalent of refusing to tie your shoes and tripping and falling as you keep trying to walk to your destination.

<comment><USER76> > Yes, because being ground down daily will eventually wear you down.

This is true. If too much technical debt accrues unchecked over time, it can have a real effect on the quality of life and morale of your engineering team - which eventually manifests as high turnover, which in turn leads to more tech debt, in a vicious feedback loop.

I've worked at such a place, and the only relief from that cycle came in the form of lost clients. But then again, despite death spiral, the company was bought out, and the owners made out pretty well.

I think maybe the old adage about outrunning a cheetah is apt here... to outrun one, you only have to run faster than the guy next to you.

<comment><USER55> > _which eventually manifests as high turnover_

Or, arguably worse, a crew of beaten-down 9-5 lifers who don't particularly care about their job.

</comment></comment></comment></comment><comment><USER77> I question the motive to complain about tech debt with the "solution" already. For example, microservices will fix it.

</comment><comment><USER19> " _And you know this rewrite, like others in the past, will be a significant sink of time, and likely a catastrophe._ "

This statement can only be true if you're using a very specific definition of "microservices". The first time I wrote about small services, in 2013 [1], I had not heard the phrase "microservices" so I used the phrase "an architecture of small apps". I was thinking of apps that talked to each via Redis, or HTTP, or didn't talk to each other and pushed everything into a database -- my idea was broad. Then in March of 2014, Martin Fowler wrote his essay on Microservices [2], which brought the phrase into widespread use. Soon after that there arose a number of startups that were promoting microservices, and they were promoting a very specific ideal: a system of apps that spoke to each other using HTTP, finding each other via auto-discovery, and offering data via a RESTful interface, with Oauth for authentication. A large number of people then began to use "microservices" for this specific set of interlocking ideas. That's fine, but it is a bit rigid. Something so comprehensive can demand a full re-write. But there is the more general idea, of slowly breaking up a monolith, and replacing small pieces with independent apps -- that does not demand a full re-write. That can be done slowly, and you have the option to keep the original monolith in a limited role, often to handle the frontend. Martin Fowler said all this in his essay, but some of the more extreme proponents of the idea have ignored the nuances that people like Fowler or myself have written about. I myself used the phrase "microservices" for a few years and now I've given up on it as too many people are associating with a very specific set of ideas, instead of the general idea of small apps cooperating with each other.

All the same, the idea can be powerful. Check out "Two months early. 300k under budget" [3]

[1] [http://www.smashcompany.com/technology/an-architecture-of- sm...](http://www.smashcompany.com/technology/an-architecture-of-small-apps)

[2] [https://martinfowler.com/articles/microservices.html](https://martinfowler.com/articles/microservices.html)

[3] [http://thoughtworks.github.io/p2/issue09/two-months- early/](http://thoughtworks.github.io/p2/issue09/two-months-early/)

</comment><comment><USER78> As with others, not a CTO, but an architect for many years, and I know the perspective I'd start out with, which is that the biggest problem at the CTO level is the mismatch between the people making decisions about what we work on next, which once you are no longer a 3-5 person start up is not the developers, and the people who understand where the technical debt is and what could be done about it.

Transferring the knowledge across that gap is very difficult, perhaps virtually impossible, because both sides of that transaction have a lot of incentives to mangle the message in the process.

The manager who is deciding what to do, and likely incentivized one way or another based on features, uptake, performance, etc. is going to always be very biased in favor of providing more to the customer, and will want the developers always visibly working on something that provides visibly more to the customers. They don't want to hear about why we need to rewrite this thing for no customer-visible change except maybe "fewer bugs", and probably don't want to hear about why the project is slower than expected ("just fix it! but without any additional resources") or what could speed it up.

The engineers have even more complicated problems. First, the engineer may not be able to correctly determine what technical debt even _matters_ if they are not properly kept in the loop about where the product is going. That is, bad code on a feature we're about to dump or obsolete is way less important than on a core feature. Keeping engineers in the loop about future product directions seems to be a big challenge. (I mean, I've seen it done successfully, but it takes a lot of effort. It's just so easy for the manager to go to some four hour meeting with the CEO and the board about the product and all that ever gets back to the team is "We need feature #64." I mean, OK, great for distilling those 4 hours down to the next action item, but at least your senior engineers really could use a bit more guidance about where the product is going.) Second, the engineer is likely to have a personal connection to the code, which can have any number of effects ranging from being defensive about any suggestion that the code is less than perfect, to believing that everything, everywhere is "technical debt" and really needs to be rewritten.

To properly gauge the technical debt on a project takes an almost perfect merger of both the business side of the project and full engineering knowledge of the project. It's harder than even spec'ing out a new feature by far. I don't know that I've ever seen a great solution for this; the requisite bits of knowledge are structurally very hard to bring together.

(This is written from the assumption that technical debt is in fact a big problem. There's a distinct possibility that it really isn't, but I'm answering the question as written. In general if I was starting a new CTO job at anything but a very established software company, I would be surprised and pleased if "technical debt" really was my biggest problem, and not flawed processes, broken cross-team communication, or goodness forbid but all too likely, _internal_ team communication.)

</comment><comment><USER79> The idea of Technical Debt, though prevalent, is fallacious, because implicit in the idea is the belief that time-to-delivery is somehow _necessarily_ a function of code quality.

But this is not true. Here's why:

The first thing to realize is that code quality is an objective quality; it is the degree to which your code is decoupled, the narrowness of the interfaces between components and the relative size of each component. (This is objective because you can take two systems that deliver the exact same business function and compare them wrt their degree of coupling, cohesion, and interface width.) The first fallacy is to think that code quality is somehow subjective; it is not: not at least not for the sense of code quality that matters--which is how much _necessary_ business cost there is to evolve the code. (Note that _necessary_ is the key word; obviously it will always cost an inexperienced/novice a lot of time to contend with a code base regardless of its structural qualities.)

Another thing that gets conflated when talking about Technical Debt is the "flexibility" or generality of the system that is built. It is a mistake to consider a specific (ie, non-generalized) system as having higher technical debt. That would be like saying Gmail has high technical debt because it doesn't have Kanban features. A high-quality systems (low technical debt, so to speak) can be a system that is quite specific, quite hard-coded, etc.

These are two fallacies. Now that they have been called out, the last thing to consider is whether or not there is some _necessary_ cost savings in delivering a system with "higher debt" (ie, a system with more complexity).

As an experienced practitioner who has worked with many other experienced practitioners in large and small companies alike, I can say confidently that one's ability to deliver a simple system is a function of skill set not time. In other words, a skilled practitioner will take a no extra time to deliver a simple system albeit perhaps a specific one. There is a necessary time trade off if we are talking about making a system with more generalized capabilities --but at this point we are talking about features and priority and managing feature creep and the like.

This is where I think many startups fail when they think that 5 mid-to-junior engineers are better than 1 qualified* senior. The only caveat is that there are many (and, perhaps, most) "seniors" out there who have a not all cultivated the relevant skill set. Who have spent their careers leaning on this false notion of Technical Debt and the idea that "I simply didn't have the time to make a good system"\--in this line of thinking, there will be no growth because implicit in it is the idea that there really is nothing new, no now skill set, to learn, that all there is just a shortage of time.

A tennis player would never think, "I would played a better game had I had more time." As if somehow there is a world in which tennis could be played in slow motion. No: the tennis player practices and practices until her muscle memory is tuned to play a competitive game in real time.

Likewise, a developer must _specifically_ train herself to play a good game in real time. So that the ability to deliver quality is part of their muscle memory. At that point it is actually harder -- takes longer! -- for this developer to build a complex system. Decoupling becomes instinct. Much like a skilled tennis player would have to go out of their way to play badly, with bad form, etc.

Unfortunately I don't see this training happening in either academia or in the practitioner's world. Everything is too optimized for immediate delivery and so people spend their careers learning how to hack it through and then only _after the fact_ come up with this idea that Technical Debt is somehow a necessary thing. The idea of Technical Debt is alluring because it is much easier to lean on than the idea that you have a serious lacuna in your skill set.

Now one might say I'm not being convincing because you still have to take the time to "learn" the skill set. This is true. But this is not change the above reasoning at all. Unless one thinks that learning and delivering can happen together in some optimal fashion. I don't believe that they can. I do not want the person building my home to be learning on the job, that is for sure.

</comment><comment><USER80> As a dev, "or solve it with microservices altogether" hahaha. Good one!

</comment><comment><USER81> I've been CTO (well, IT Director) for 3 years and I saw no particular problem with technical debt. I just handled it the same way I would handle financial debt. Repaying during good times and taking on debt in projects with high stakes and fixed deadlines.

It all depends on the industry as well. My CTO experience was in News and Media, where software is rewritten quite often and does not have any business value by itself.

I am now in a consulting position in the banking industry. Things are different there. None is willing to take on the risks associated with technical debt repayment. It accumulates endlessly. Once it reaches a point where no local developer would work on the project, the project is handed over to Asian sweatshops which support it indefinitely.

<comment><AUTHOR> > Repaying during good times and taking on debt in projects with high stakes > and fixed deadlines.

I agree this is an amazing strategy when you can do it!

> None is willing to take on the risks associated with technical debt > repayment.

What do you think they are afraid of? What risks are there? Is it possible to mitigate these risks in some way?

<comment><USER82> >What do you think they are afraid of? What risks are there? Is it possible to mitigate these risks in some way?

Not OP, but usually there are no (trusted) unit tests.

<comment><USER81> IMO, unit tests, integration tests, TDD, BDD, etc do not matter as much as the size of the codebase. If your system has millions of lines of code, you are doomed and no amount of unit tests can help you.

<comment><USER83> I believe thats why modularization was invented?

<comment><USER81> Modularization per se does not work. What works is administrative boundaries. Once a product reach a certain size, the team should be split up.

<comment><USER83> "Once a product reach a certain size, the team should be split up."

That is also called modularization.

</comment></comment></comment></comment></comment></comment></comment><comment><USER84> Refactor everything all the time, duh</comment>


FactoryPal: New Scala framework for creating objects as test data - <AUTHOR> http://www.blogeek.com.ar/2012/12/19/factorypal-new-scala-framework-for-creating-objects-as-test-data-say-no-to-fixtures-and-mocks/

<comment><AUTHOR> Hey, if you could give me your opinions about it, I'd appreciate it!

Thanks!

</comment><comment><AUTHOR> I hope you like it :)</comment>


How to Fold a Julia Fractal - <AUTHOR> http://acko.net/blog/how-to-fold-a-julia-fractal/ <comment><USER2> More than being an intro to Julia fractals, I think this post is a great introduction to complex numbers and functions of the complex plane[1].

The way this is presented is very similar to how most math-folk I know picture these concepts in their head. This is probably one of the toughest things for beginners, who don't understand that (most) math-folk think in pictures like this and not in symbols.

For example, starting at around Slide 29 in the first visualization, the author actually paints a picture of a branch cut[2] without using that term.

Likewise, starting at around Slide 12 of the last visualization, the author hints at the special relationship between complex numbers and differentiation in the complex plane. The jargon-y stuff involved here are holomorphic functions[3], the Cauchy-Riemann equations[4], and the very surprising-but- central theorem of complex analysis: Cauchy's integral theorem[5].

      [1]: Functions from ℂ to ℂ are "hard" to reason about because there
           are 4 dimensions involved, at least if you're picturing ℂ as a
           2-dimensional plane.
      [2]: https://en.wikipedia.org/wiki/Branch_point#Branch_cuts
      [3]: https://en.wikipedia.org/wiki/Holomorphic_function
      [4]: https://en.wikipedia.org/wiki/Cauchy-Riemann_equations
      [5]: https://en.wikipedia.org/wiki/Cauchy%27s_integral_theorem

</comment><comment><USER3> What a wonderful post this is! I first encountered Julia fractals while taking a masters class in chaos theory, technically a physics class. It was a masters of liberal arts program so you got a little bit of everything... I hadn't taken math since high school, really, and the class utterly annihilated my conception of the world.

I remember the Julia fractal in particular because it was so beautiful, and it was around this part of the course -- maybe 75% of the way through -- that the fractals and like topics started to blow my mind. Our professor showed us this video that zoomed in on a Julia fractal, something like this, [https://www.youtube.com/watch?v=gruJ0S3TTtI](https://www.youtube.com/watch?v=gruJ0S3TTtI), and I remember watching it all day at work the next day. I also searched for images of the most beautiful Julias to make as my desktop background, of course.

Not only were they beautiful but so symbolic, as this article captures: Julia fractals are part of chaos theory, which holds that even determinate, logical systems can nevertheless manifest completely unpredictable and nonrecurrent behavior. It's a straightforward equation that gets you these beautiful -- and utterly terrifying, ceaseless, dreamlike -- images, when mapped in a certain way. For me, that's a really beautiful concept because with "Enlightenment" mathematics, Newton and Leibniz and co, you got this concept of a determinate universe, which could therefore also be known and predicated in advance. Yet chaos theory shows that even determinate systems can be impossible to know, refusing to allow the complexity and variety of that which exists to boil down into a boring pattern of predictable and even controllable outcomes.

</comment><comment><USER4> Nice to read this again.

Not nearly as pretty, but if you'd like to read more about complex numbers, here's a slide show I put together a while back:

How to explain Euler's identity using triangles and spirals [https://docs.google.com/presentation/d/1oMNjkDp- LieSGnZEwNpc...](https://docs.google.com/presentation/d/1oMNjkDp- LieSGnZEwNpceG8KTIvbnus9olu3KqnM5bg/edit#slide=id.i0)

<comment><AUTHOR> That's awesome, thank you! It really did "connect the dots" in my head :)

</comment></comment><comment><USER5> Previous discussion: [https://news.ycombinator.com/item?id=7898883](https://news.ycombinator.com/item?id=7898883)

<comment><USER6> Older discussions:

[https://news.ycombinator.com/item?id=5017078](https://news.ycombinator.com/item?id=5017078) (4 comments)

[https://news.ycombinator.com/item?id=5036235](https://news.ycombinator.com/item?id=5036235) (50 comments)

</comment></comment><comment><USER7> WOW! The masthead alone is worth a scroll, you even get an achievement badge. LOL

<comment><USER8> You would like the article on how they made it

[http://acko.net/blog/zero-to-sixty-in-one-second/](http://acko.net/blog/zero- to-sixty-in-one-second/)

All of their MathBox-powered articles are wonderfull really.

<comment><USER9> nitpick: _He_ not _they_. [0] Steven is also the person who developed MathBox. [1]

[0] [http://acko.net/about/](http://acko.net/about/)

[1] [http://acko.net/blog/making-mathbox/](http://acko.net/blog/making- mathbox/)

<comment><USER10> Thanks for adding more information, but that's a perfectly cromulent use of the word they.

<comment><USER9> Actually I think the correctness of using a singular they is disputed but I am not a grammaticist, however the word _cromulent_ which you used is not even a real word but I didn't mention the correction to do all that nitpicking, I just thought since the person's information is accessible, it'd be nice to refer to him correctly that's all.

<comment><USER11> > the word cromulent ... is not even a real word

Only to a prescriptivist[1] trying to keep the language static. "Cromulent" is slowly making its way into descriptivist[1] dictionaries and is recognized about as often as any other new word, so it just as much a "real" word as other new words ("email", "google" (transitive verb), "truthiness").

> the correctness of using a singular they

The alternative is gendered pronouns which have several problems[2].

/* I'm not trying to nitpick your post; I just thought these two Tom Scott clips were fun and relevant to these grammar issues. */

[1] [https://www.youtube.com/watch?v=2qT8ZYewYEY](https://www.youtube.com/watch?v=2qT8ZYewYEY)

[2] [https://www.youtube.com/watch?v=46ehrFk- gLk](https://www.youtube.com/watch?v=46ehrFk-gLk)

<comment><USER9> That was kinda my point, I was not nitpicking -relative OP's- grammatical usage, I just thought it was convenient to refer to the author in a more specific manner since his details were obvious. I had no idea whether _their_ -relative OP- :) intention was to use "their" in a singular or plural manner. Heck, one of my favorite Stephen Fry videos [0] talks exactly about this!

Also, the videos you referenced are great :)

[0] [https://www.youtube.com/watch?v=J7E-aoXLZGY](https://www.youtube.com/watch?v=J7E-aoXLZGY)

</comment></comment></comment></comment></comment></comment></comment><comment><USER12> This page was very useful when I was teaching a (particularly clever) 12 year old kid about complex numbers. He really wanted to render Mandelbrot/Julia fractals (using Processing), had done some googling on the subject of complex numbers, but most of the articles he found were ever-so-slightly above the level of math he had learned in school (turned out he hadn't yet learned about the distributive rule for multiplication, that (a+b)*(c+d) = ac + ad + bc + bd, which is kinda important if you want to work out (x + iy)^2 given that i^2 = -1).

I was lucky that someone explained me complex numbers when I was 15 (I also had wanted to plot Mandelbrot fractals for a long time, but back then I didn't even have the Internet to help me), using a very visual approach similar to the featured article. That is, multiplication by -1 is the same as a 180 degree rotation around the zero ... so what would happen if we decided we could rotate by 90 degrees?

So I took a similar approach. Then I remembered this article about "folding Julia fractals", the visualizations in this article were a great supplement to the graphs and scribbles we made on paper, exploring the weird world of complex numbers.

I did a little video interview with him to show off his work (cause, you know, I was kinda proud): [https://www.youtube.com/watch?v=rR6klRdtjsg](https://www.youtube.com/watch?v=rR6klRdtjsg) \-- It's in Dutch and I'm not a very good interviewer, also no editing (and yes I should've held my phone horizontally, sorry).

But the best part was two weeks later, I kinda feared I had dumped too much information onto him at once (especially given I also had to explain the distributive rule), I asked him if he had made any improvements or additions; "Yeah, I had to wait at the dentist's this week, and I had Processing for Android (APDE) on my phone, so I wrote the Julia version of the Mandelbrot zoomer" ... Oh, if only I had have a powerful pocket computer when I was 12!!! (so jealous!)

</comment><comment><USER13> Reading this gave me a moderate ASMR response. Anyone else have that occasionally from reading about math?

</comment><comment><USER14> What brilliant article and website design!

I feel stupid for not being able to visualize complex numbers before.

</comment><comment><USER15> This would have helped me tremendously when I was studying complex numbers for signal processing. I guess I'm a visual learner. I find that to be helpful in some field in math, but held me back when I was studying statistics.

</comment><comment><USER16> There's not a whole lot of material about the Julia set itself.

When he talks about "folding" the Julia set, I immediately thought of this picture of some fractions [http://i.imgur.com/wxz2a3t.png](http://i.imgur.com/wxz2a3t.png) from this paper: [http://arxiv.org/abs/1201.4225v1](http://arxiv.org/abs/1201.4225v1)

</comment><comment><USER17> OWAW OWAW! I am starting to understand complex number, which I had to "flirt with" a few times working on signal processing.

Thanks, amazing blogpost / webapp :)

</comment><comment><USER18> Thank you ! I've been so looking for this article, I was totally amazed when I first read it (back in 2013, I guess). I think it's one of the best math explanations (or rather, visualisations) that I've ever seen.

</comment><comment><USER19> [Slightly OT] Crashes my firefox everytime (FF 40 on OSX).

<comment><USER12> I know. My low-powered laptop can barely handle it either. But this particular page, is worth trying on a slightly more powerful computer :-)

</comment></comment><comment><USER20> The article is interesting, but it's very difficult to read with that background. Also, my slow netbook becomes very slow rendering it.

<comment><USER5> The Julia set slideshow/animation (36 steps) is really the best part, the animation of the set "folding" (square every point) is great.</comment></comment>


Why Elite:Dangerous isn't just another computer game - <AUTHOR> http://ollieclark.wordpress.com/2012/12/

<comment><USER2> I played a bit of elite Back In The Day (speccy, got to Deadly). I always assumed that Eve online was pretty much exactly what the blogger described. Spaceships, freedom, trading.

Is there something I'm missing?

<comment><USER3> Actually Eve is pretty close, you're right. It owes a huge debt to the original Elite. I've never actually played Eve but I believe the combat is based much more on tactics and strategy rather than skill and reaction time. Nothing wrong with that but it's quite a different experience.

</comment></comment><comment><USER4> _If there's a game that comes close to this depth, I've yet to see it._

You may want to check out <http://www.eveonline.com> .

</comment><comment><USER5> Can't wait for this - 20yrs on the back burner and it's finally happening :)

</comment><comment><USER6> OMG... I logged 1000+ hours with the Amiga 500 on this back in the day.</comment>