Big companies, like Google & Facebook, store all their code in a single monolithic repository or monorepo... but why? Learn how to use tools like NPM or Yarn workspaces, Learna, Nx, and Turborepo to scale your codebase https://fireship.io/pro 

#software #js #compsci

🔗 Resources

Turborepo https://turborepo.org/
Nx https://nx.dev/
Lerna https://lerna.js.org/
Google's Monorepo https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext

📚 Chapters

00:00 The World’s Largest Codebase
00:49 Benefits of Monorepos
01:50 The Problem with Monorepos
02:18 Yarn & NPM Workspaces
02:48 Lerna
03:30 PNPM
03:48 Nx vs Turborepo
06:10 Turborepo Tutorial

🔥 Get More Content - Upgrade to PRO

Upgrade to Fireship PRO at https://fireship.io/pro
Use code lORhwXd2 for 25% off your first payment. 

🎨 My Editor Settings

- Atom One Dark 
- vscode-icons
- Fira Code Font

🔖 Topics Covered

-  Monorepo Pros and Cons
-  What is a monorepo?
-  Who uses Monorepos?
-  Should I use a monorepo?
-  Nx compared to Turborepo
-  How Google's Monorepo works.
2024-02-04T13:50:43.017Z
nova-2
2-general-nova
2024-01-09.29447
30089e05-99d1-4376-b32e-c263170674af
c5adbc63-c50d-4192-b940-b5a44c9c7767
f6b133721880efadb221015458473de5ed3befe69e60e42bf97754ba3ba06975
67875a7f-c9c4-48a0-aa55-5bdb8a91c34a
deprecated
A software company like Google maintains a lot of code.
Like, seriously, a lot.
Like, over 2,000,000,000 lines of code and nearly a 100 terabytes of data to go along with it.
It's a scale that's hard to even comprehend, like all the stars in the universe or all the universes in the multiverse.
And you have thousands of around the world working on it simultaneously.
But get this, they store all their code in a single monolithic repository and they've been doing since the very beginning.
Today, Google's mono repo is likely the largest code base in the world, but it takes an extraordinary effort to scale.
They have their own home grown version control system and a highly advanced build tool called Bazel, which goes by the much cooler name of Blaze internally.
In today's video, you'll learn everything you ever wanted to know data out mono repos.
And how you, the humble JavaScript developer, can build a high performance mono repo in your garage.
If you're new here, like and subscribe.
We're so close to 1,000,000, and if You hit the button right now, we might even make it there by Christmas.
You may have heard some exciting news last week that Vercel acquired a company called Turborepo.
Data.
It's a build tool written in Go that makes it really easy to manage multiple apps and packages in a single Git repository.
But first, let's answer the question of why would I want to use a monorepo?
There are many reasons, but at the highest level, it gives you visibility of your company's entire code base without the need to track down and clone a bunch of different repos.
In addition, it provides consistency because you can share things like your ESLint config, a UI library of web components for your design system, utility libraries, documentation, and so on.
The real power though comes in the form of dependency management.
Imagine somebody makes a breaking change to a shared library.
All affected applications will know instantly, and monorepo tools can actually help you visualize the entire dependency graph of your software.
When it comes to third party a monorepo can dedupe packages that are used in multiple apps.
A monorepo is also ideal for continuous integration and automation, because your code is already unified by default, making it much easier to build and test everything together.
But there is one big problem with monorepos and that's the fact that they're big.
As the monorepo becomes larger, there are more things to test, more things to build and a lot more artifacts to store as a disk.
Versus Code will lag trying to process the massive git history and you'll need a 20 minute smoke break waiting for everything to run on the CI server after every commit.
To operate a monorepo at scale, it's absolutely essential to have the right tooling.
That's why Facebook created Buck, Microsoft created Rush, dang Google created Bazel.
You just need a PhD in order to use it.
Luckily, there are other options out there.
The most basic approach is to use your package manager, like darn or NPM to define workspaces.
These tools basically configure your project with a root level package JSON, which then has nested workspaces like apps and packages that are linked back to the root level project.
A cool thing about this is that it will dedupe your node modules, which means if you have the same package installed in multiple apps, it will only be installed once.
It also allows you to orchestrate scripts, like, if you want to build or test all of your apps at the same time.
That's a good easy place to start, but if you're building an open source project data that publishes a bunch of different packages, then you'll likely want to look into a tool called Lerna, which is a tool that can optimize the workflow of a multi package repo.
Here's an example.
Turf JS is a geospatial library that has a ton of different packages that are essentially helper functions for working with geolocation data.
Each one of them can be installed as its own package and lives in its own subdirectory here on the repo.
Lerna is the tool that it uses to help manage its workflow efficiently.
Most importantly, it allows you to publish all of your packages to digits to NPM with a single command.
These tools are great at configuring monorepos, but they still suffer from the same problem I mentioned earlier.
They become really slow and difficult to work with as they grow larger.
One problem is the installation of dependencies.
If you're looking to improve your install speed, an easy optimization to make is to replace NPM with PNPM.
It's a drop in replacement that will install your dependencies globally and sim link them, and that can make your install speeds up to 3 times faster.
That's a nice upgrade, but what really makes a monorepo slow is the constant need to recompile, rebuild, and retest everything.
And that brings us to the fun part of the video where we talk about an entirely different class of tools that can make your monorepo operate at the speed of Google.
The tools I'll be comparing are n x and turbo repo.
They both operate as what I would as a smart build system.
What I mean by that is they create a dependency tree between all of your apps and packages, which allows the tooling to understand what data to be tested and what needs to be rebuilt whenever there's a change to the code base.
They cache any files or artifacts that have already been built and can also run jobs in parallel to execute everything much faster.
We'll look at some example code in just a second, but first, what is the difference between NX and Turborepo?
NX has been around for about 5 years and was created by 2 ex Googlers.
Turborepo, on the other hand, was just open source the other day.
It was created by Jared Palmer, who you might know from the React ecosystem system with packages like Formic.
At this point, Turborepo is a lot more minimal than nx.
Nx can do everything that Turborepo does and has a lot of features beyond like a CLI that can automatically generate boilerplate code for you, a plug in ecosystem, a Versus Code extension, and something called distributed task execution, DIN, which allows you to distribute work across multiple continuous integration servers.
It's a pretty amazing feature from a technical standpoint and was inspired by Bazel.
On one hand, it's great to have all these features, But on the other hand, it might be a little more than you actually need.
NX has been criticized for requiring too much configuration, although, I don't necessarily agree with data.
If you're only using the core NX features, that configuration is actually very minimal.
That being said, Turborepo feels like a tool that goes out of its way data to get out of your way.
For example, if you're already using yarn or npm workspaces, all you have to do is add a few lines of configuration to your package JSON, and you now have a magically super fast monorepo.
When it comes to dependencies, both tools visualize the dependency graph, although nx does a much better job at this.
But again, it's had a lot more time to bake in the data.
Another cool thing that both of them do is remote caching.
With Turborepo, when it caches something, it can also cache those files on Vercel.
That means if Bob builds the duplication, then Alice checks it out on a different computer, the entire cache can be downloaded remotely instead of wastefully rebuilding and recomputing everything in the monorepo.
And that can save a huge amount of time for a large organization.
The final comparison point I wanna make is that Turborepo is written in Go, while n x is written in TypeScript.
In theory, this could result in startup times when running Turborepo, but the underlying language is not where the performance gains come from.
It depends on when the computations take place and how they're d cache, but I'd be interested to see some benchmarks between these two tools.
Now it's time for you to build your first monorepo.
I already have a video on n x if you wanna check that out, but today we're going to take day.
1st look at Turborepo.
You can add it to an existing monorepo where they also have a template to start from scratch.
Run npx create turbo from the command line, and that should bring up a menu with some gradient text that says Turborepo.
It'll give you the option for a package manager and I'll go with yarn as the default.
When we open up the project in Versus Code, you'll see that we have 2 root level directories for apps and packages, where apps are the actual applications you deploy, like a Next.
Js or React app for example.
And packages are the different configurations, utilities, and UI libraries that those apps depend on.
Now if we open up the root level package JSON, you'll notice we have work databases set up for apps and packages there as well.
Then towards the bottom, you'll notice the turbo configuration.
This is where you configure a pipeline to run tasks efficiently.
Normally, in a monorepo, everything runs 1 by 1.
You do all your linting first, then you do all your builds, then your testing, and finally your deployments.
Disk.
With a pipeline, you can explicitly tell turbo what a task depends on.
Like, you can run Lint whenever, but to run your builds, you'll want to make sure that all the dependencies have been built first.
The caret symbol refers to dependencies.
Then to test the application, we'll want to make sure that that build has completed and to deploy, we'll ensure that build, test, and have all been completed.
A pipeline like this allows you to condense the timeline and utilize more CPU resources to deliver faster builds and that's pretty much all it takes to get started.
We data.
Now build or develop all of our apps in parallel.
If we run yarn run dev, it serves both of the Next.
Js applications in this repo at the same time.
Time.
In the terminal, you can see where it's using the cache.
In the browser, you get 2 apps running at the same time without having to deal with multiple processes.
An interesting thing to notice here is that if we go in data package.
Json in one of the applications, you'll notice that it has dependencies like UI that are linked to a star character, and the same for config d and tsconfig.
These are dependencies that live in the monorepo under the packages directory.
What's cool about this is that if we go into our UI package and make a change, the change will be reflected instantly in both applications at the same time.
There's no need to recompile and reinstall a dependency.
Everything is continuously integrated into the entire codebase.
And and the awesome thing about turbo is that everything becomes much faster by default.
Like, if we run the build command, the first time around might take a couple minutes, but when we run it a second time Everything will be cached and that run might only take a few 100 milliseconds, at least in theory.
There's a bug on Windows right now that's ruining my video, but Normally, this build will be done in a matter of milliseconds.
I blame myself for not using Linux.
To summarize, monorepos can be an awesome tool when you have a large complex project.
The great thing about tools like turbo and n x is that they dramatically lower the barrier of entry to scale a massive code base.
A mono repo might be overkill for your blog.
But, if you're the technical co founder of a company, it's definitely an approach to building software that you want to be aware of.
I'm gonna go ahead and wrap things up there.
If you want access to more advanced JavaScript development content, consider becoming a pro member at Fireship IO.
Thanks for watching, and I will see you in the next one.

Speaker 0: A software company like Google maintains a lot of code. Like, seriously, a lot. Like, over 2,000,000,000 lines of code and nearly a 100 terabytes of data to go along with it. It's a scale that's hard to even comprehend, like all the stars in the universe or all the universes in the multiverse. And you have thousands of around the world working on it simultaneously.

But get this, they store all their code in a single monolithic repository and they've been doing since the very beginning. Today, Google's mono repo is likely the largest code base in the world, but it takes an extraordinary effort to scale. They have their own home grown version control system and a highly advanced build tool called Bazel, which goes by the much cooler name of Blaze internally. In today's video, you'll learn everything you ever wanted to know data out mono repos. And how you, the humble JavaScript developer, can build a high performance mono repo in your garage.

If you're new here, like and subscribe. We're so close to 1,000,000, and if You hit the button right now, we might even make it there by Christmas. You may have heard some exciting news last week that Vercel acquired a company called Turborepo. Data. It's a build tool written in Go that makes it really easy to manage multiple apps and packages in a single Git repository.

But first, let's answer the question of why would I want to use a monorepo? There are many reasons, but at the highest level, it gives you visibility of your company's entire code base without the need to track down and clone a bunch of different repos. In addition, it provides consistency because you can share things like your ESLint config, a UI library of web components for your design system, utility libraries, documentation, and so on. The real power though comes in the form of dependency management. Imagine somebody makes a breaking change to a shared library.

All affected applications will know instantly, and monorepo tools can actually help you visualize the entire dependency graph of your software. When it comes to third party a monorepo can dedupe packages that are used in multiple apps. A monorepo is also ideal for continuous integration and automation, because your code is already unified by default, making it much easier to build and test everything together. But there is one big problem with monorepos and that's the fact that they're big. As the monorepo becomes larger, there are more things to test, more things to build and a lot more artifacts to store as a disk.

Versus Code will lag trying to process the massive git history and you'll need a 20 minute smoke break waiting for everything to run on the CI server after every commit. To operate a monorepo at scale, it's absolutely essential to have the right tooling. That's why Facebook created Buck, Microsoft created Rush, dang Google created Bazel. You just need a PhD in order to use it. Luckily, there are other options out there.

The most basic approach is to use your package manager, like darn or NPM to define workspaces. These tools basically configure your project with a root level package JSON, which then has nested workspaces like apps and packages that are linked back to the root level project. A cool thing about this is that it will dedupe your node modules, which means if you have the same package installed in multiple apps, it will only be installed once. It also allows you to orchestrate scripts, like, if you want to build or test all of your apps at the same time. That's a good easy place to start, but if you're building an open source project data that publishes a bunch of different packages, then you'll likely want to look into a tool called Lerna, which is a tool that can optimize the workflow of a multi package repo.

Here's an example. Turf JS is a geospatial library that has a ton of different packages that are essentially helper functions for working with geolocation data. Each one of them can be installed as its own package and lives in its own subdirectory here on the repo. Lerna is the tool that it uses to help manage its workflow efficiently. Most importantly, it allows you to publish all of your packages to digits to NPM with a single command.

These tools are great at configuring monorepos, but they still suffer from the same problem I mentioned earlier. They become really slow and difficult to work with as they grow larger. One problem is the installation of dependencies. If you're looking to improve your install speed, an easy optimization to make is to replace NPM with PNPM. It's a drop in replacement that will install your dependencies globally and sim link them, and that can make your install speeds up to 3 times faster.

That's a nice upgrade, but what really makes a monorepo slow is the constant need to recompile, rebuild, and retest everything. And that brings us to the fun part of the video where we talk about an entirely different class of tools that can make your monorepo operate at the speed of Google. The tools I'll be comparing are n x and turbo repo. They both operate as what I would as a smart build system. What I mean by that is they create a dependency tree between all of your apps and packages, which allows the tooling to understand what data to be tested and what needs to be rebuilt whenever there's a change to the code base.

They cache any files or artifacts that have already been built and can also run jobs in parallel to execute everything much faster. We'll look at some example code in just a second, but first, what is the difference between NX and Turborepo? NX has been around for about 5 years and was created by 2 ex Googlers. Turborepo, on the other hand, was just open source the other day. It was created by Jared Palmer, who you might know from the React ecosystem system with packages like Formic.

At this point, Turborepo is a lot more minimal than nx. Nx can do everything that Turborepo does and has a lot of features beyond like a CLI that can automatically generate boilerplate code for you, a plug in ecosystem, a Versus Code extension, and something called distributed task execution, DIN, which allows you to distribute work across multiple continuous integration servers. It's a pretty amazing feature from a technical standpoint and was inspired by Bazel. On one hand, it's great to have all these features, But on the other hand, it might be a little more than you actually need. NX has been criticized for requiring too much configuration, although, I don't necessarily agree with data.

If you're only using the core NX features, that configuration is actually very minimal. That being said, Turborepo feels like a tool that goes out of its way data to get out of your way. For example, if you're already using yarn or npm workspaces, all you have to do is add a few lines of configuration to your package JSON, and you now have a magically super fast monorepo. When it comes to dependencies, both tools visualize the dependency graph, although nx does a much better job at this. But again, it's had a lot more time to bake in the data.

Another cool thing that both of them do is remote caching. With Turborepo, when it caches something, it can also cache those files on Vercel. That means if Bob builds the duplication, then Alice checks it out on a different computer, the entire cache can be downloaded remotely instead of wastefully rebuilding and recomputing everything in the monorepo. And that can save a huge amount of time for a large organization. The final comparison point I wanna make is that Turborepo is written in Go, while n x is written in TypeScript.

In theory, this could result in startup times when running Turborepo, but the underlying language is not where the performance gains come from. It depends on when the computations take place and how they're d cache, but I'd be interested to see some benchmarks between these two tools. Now it's time for you to build your first monorepo. I already have a video on n x if you wanna check that out, but today we're going to take day. 1st look at Turborepo.

You can add it to an existing monorepo where they also have a template to start from scratch. Run npx create turbo from the command line, and that should bring up a menu with some gradient text that says Turborepo. It'll give you the option for a package manager and I'll go with yarn as the default. When we open up the project in Versus Code, you'll see that we have 2 root level directories for apps and packages, where apps are the actual applications you deploy, like a Next. Js or React app for example.

And packages are the different configurations, utilities, and UI libraries that those apps depend on. Now if we open up the root level package JSON, you'll notice we have work databases set up for apps and packages there as well. Then towards the bottom, you'll notice the turbo configuration. This is where you configure a pipeline to run tasks efficiently. Normally, in a monorepo, everything runs 1 by 1.

You do all your linting first, then you do all your builds, then your testing, and finally your deployments. Disk. With a pipeline, you can explicitly tell turbo what a task depends on. Like, you can run Lint whenever, but to run your builds, you'll want to make sure that all the dependencies have been built first. The caret symbol refers to dependencies.

Then to test the application, we'll want to make sure that that build has completed and to deploy, we'll ensure that build, test, and have all been completed. A pipeline like this allows you to condense the timeline and utilize more CPU resources to deliver faster builds and that's pretty much all it takes to get started. We data. Now build or develop all of our apps in parallel. If we run yarn run dev, it serves both of the Next.

Js applications in this repo at the same time. Time. In the terminal, you can see where it's using the cache. In the browser, you get 2 apps running at the same time without having to deal with multiple processes. An interesting thing to notice here is that if we go in data package.

Json in one of the applications, you'll notice that it has dependencies like UI that are linked to a star character, and the same for config d and tsconfig. These are dependencies that live in the monorepo under the packages directory. What's cool about this is that if we go into our UI package and make a change, the change will be reflected instantly in both applications at the same time. There's no need to recompile and reinstall a dependency. Everything is continuously integrated into the entire codebase.

And and the awesome thing about turbo is that everything becomes much faster by default. Like, if we run the build command, the first time around might take a couple minutes, but when we run it a second time Everything will be cached and that run might only take a few 100 milliseconds, at least in theory. There's a bug on Windows right now that's ruining my video, but Normally, this build will be done in a matter of milliseconds. I blame myself for not using Linux. To summarize, monorepos can be an awesome tool when you have a large complex project.

The great thing about tools like turbo and n x is that they dramatically lower the barrier of entry to scale a massive code base. A mono repo might be overkill for your blog. But, if you're the technical co founder of a company, it's definitely an approach to building software that you want to be aware of. I'm gonna go ahead and wrap things up there. If you want access to more advanced JavaScript development content, consider becoming a pro member at Fireship IO.

Thanks for watching, and I will see you in the next one.
A software company like Google maintains a lot of code. Like, seriously, a lot. Like, over 2,000,000,000 lines of code and nearly a 100 terabytes of data to go along with it. It's a scale that's hard to even comprehend, like all the stars in the universe or all the universes in the multiverse. And you have thousands of around the world working on it simultaneously. But get this, they store all their code in a single monolithic repository and they've been doing since the very beginning.
software
cosmology
google
Today, Google's mono repo is likely the largest code base in the world, but it takes an extraordinary effort to scale. They have their own home grown version control system and a highly advanced build tool called Bazel, which goes by the much cooler name of Blaze internally. In today's video, you'll learn everything you ever wanted to know data out mono repos. And how you, the humble JavaScript developer, can build a high performance mono repo in your garage. If you're new here, like and subscribe. We're so close to 1,000,000, and if You hit the button right now, we might even make it there by Christmas. You may have heard some exciting news last week that Vercel acquired a company called Turborepo. Data. It's a build tool written in Go that makes it really easy to manage multiple apps and packages in a single Git repository. But first, let's answer the question of why would I want to use a monorepo? There are many reasons, but at the highest level, it gives you visibility of your company's entire code base without the need to track down and clone a bunch of different repos.
In addition, it provides consistency because you can share things like your ESLint config, a UI library of web components for your design system, utility libraries, documentation, and so on. The real power though comes in the form of dependency management. Imagine somebody makes a breaking change to a shared library. All affected applications will know instantly, and monorepo tools can actually help you visualize the entire dependency graph of your software. When it comes to third party a monorepo can dedupe packages that are used in multiple apps.
software
systems
software development
infrastructure
A monorepo is also ideal for continuous integration and automation, because your code is already unified by default, making it much easier to build and test everything together. But there is one big problem with monorepos and that's the fact that they're big. As the monorepo becomes larger, there are more things to test, more things to build and a lot more artifacts to store as a disk. Versus Code will lag trying to process the massive git history and you'll need a 20 minute smoke break waiting for everything to run on the CI server after every commit. To operate a monorepo at scale, it's absolutely essential to have the right tooling. That's why Facebook created Buck, Microsoft created Rush, dang Google created Bazel.
You just need a PhD in order to use it. Luckily, there are other options out there. The most basic approach is to use your package manager, like darn or NPM to define workspaces. These tools basically configure your project with a root level package JSON, which then has nested workspaces like apps and packages that are linked back to the root level project.
software
software development
A cool thing about this is that it will dedupe your node modules, which means if you have the same package installed in multiple apps, it will only be installed once. It also allows you to orchestrate scripts, like, if you want to build or test all of your apps at the same time. That's a good easy place to start, but if you're building an open source project data that publishes a bunch of different packages, then you'll likely want to look into a tool called Lerna, which is a tool that can optimize the workflow of a multi package repo.
software development
software
Here's an example. Turf JS is a geospatial library that has a ton of different packages that are essentially helper functions for working with geolocation data.
Each one of them can be installed as its own package and lives in its own subdirectory here on the repo. Lerna is the tool that it uses to help manage its workflow efficiently. Most importantly, it allows you to publish all of your packages to digits to NPM with a single command.
software
software development
These tools are great at configuring monorepos, but they still suffer from the same problem I mentioned earlier. They become really slow and difficult to work with as they grow larger. One problem is the installation of dependencies. If you're looking to improve your install speed, an easy optimization to make is to replace NPM with PNPM. It's a drop in replacement that will install your dependencies globally and sim link them, and that can make your install speeds up to 3 times faster. That's a nice upgrade, but what really makes a monorepo slow is the constant need to recompile, rebuild, and retest everything. And that brings us to the fun part of the video where we talk about an entirely different class of tools that can make your monorepo operate at the speed of Google.
The tools I'll be comparing are n x and turbo repo. They both operate as what I would as a smart build system. What I mean by that is they create a dependency tree between all of your apps and packages, which allows the tooling to understand what data to be tested and what needs to be rebuilt whenever there's a change to the code base.
software development
software
infrastructure
They cache any files or artifacts that have already been built and can also run jobs in parallel to execute everything much faster. We'll look at some example code in just a second, but first, what is the difference between NX and Turborepo? NX has been around for about 5 years and was created by 2 ex Googlers. Turborepo, on the other hand, was just open source the other day.
It was created by Jared Palmer, who you might know from the React ecosystem system with packages like Formic. At this point, Turborepo is a lot more minimal than nx. Nx can do everything that Turborepo does and has a lot of features beyond like a CLI that can automatically generate boilerplate code for you, a plug in ecosystem, a Versus Code extension, and something called distributed task execution, DIN, which allows you to distribute work across multiple continuous integration servers. It's a pretty amazing feature from a technical standpoint and was inspired by Bazel. On one hand, it's great to have all these features, But on the other hand, it might be a little more than you actually need. NX has been criticized for requiring too much configuration, although, I don't necessarily agree with data.
programming
graphic design
software development
computer science
software
If you're only using the core NX features, that configuration is actually very minimal. That being said, Turborepo feels like a tool that goes out of its way data to get out of your way. For example, if you're already using yarn or npm workspaces, all you have to do is add a few lines of configuration to your package JSON, and you now have a magically super fast monorepo.
When it comes to dependencies, both tools visualize the dependency graph, although nx does a much better job at this. But again, it's had a lot more time to bake in the data. Another cool thing that both of them do is remote caching. With Turborepo, when it caches something, it can also cache those files on Vercel.
baking
That means if Bob builds the duplication, then Alice checks it out on a different computer, the entire cache can be downloaded remotely instead of wastefully rebuilding and recomputing everything in the monorepo. And that can save a huge amount of time for a large organization. The final comparison point I wanna make is that Turborepo is written in Go, while n x is written in TypeScript. In theory, this could result in startup times when running Turborepo, but the underlying language is not where the performance gains come from. It depends on when the computations take place and how they're d cache, but I'd be interested to see some benchmarks between these two tools. Now it's time for you to build your first monorepo. I already have a video on n x if you wanna check that out, but today we're going to take day. 1st look at Turborepo. You can add it to an existing monorepo where they also have a template to start from scratch. Run npx create turbo from the command line, and that should bring up a menu with some gradient text that says Turborepo.
It'll give you the option for a package manager and I'll go with yarn as the default. When we open up the project in Versus Code, you'll see that we have 2 root level directories for apps and packages, where apps are the actual applications you deploy, like a Next. Js or React app for example. And packages are the different configurations, utilities, and UI libraries that those apps depend on. Now if we open up the root level package JSON, you'll notice we have work databases set up for apps and packages there as well. Then towards the bottom, you'll notice the turbo configuration. This is where you configure a pipeline to run tasks efficiently. Normally, in a monorepo, everything runs 1 by 1. You do all your linting first, then you do all your builds, then your testing, and finally your deployments. Disk. With a pipeline, you can explicitly tell turbo what a task depends on.
software
software development
programming
Like, you can run Lint whenever, but to run your builds, you'll want to make sure that all the dependencies have been built first. The caret symbol refers to dependencies.
Then to test the application, we'll want to make sure that that build has completed and to deploy, we'll ensure that build, test, and have all been completed. A pipeline like this allows you to condense the timeline and utilize more CPU resources to deliver faster builds and that's pretty much all it takes to get started. We data. Now build or develop all of our apps in parallel. If we run yarn run dev, it serves both of the Next. Js applications in this repo at the same time. Time. In the terminal, you can see where it's using the cache. In the browser, you get 2 apps running at the same time without having to deal with multiple processes.
programming
infrastructure
software development
An interesting thing to notice here is that if we go in data package. Json in one of the applications, you'll notice that it has dependencies like UI that are linked to a star character, and the same for config d and tsconfig. These are dependencies that live in the monorepo under the packages directory. What's cool about this is that if we go into our UI package and make a change, the change will be reflected instantly in both applications at the same time. There's no need to recompile and reinstall a dependency. Everything is continuously integrated into the entire codebase. And and the awesome thing about turbo is that everything becomes much faster by default. Like, if we run the build command, the first time around might take a couple minutes, but when we run it a second time Everything will be cached and that run might only take a few 100 milliseconds, at least in theory.
There's a bug on Windows right now that's ruining my video, but Normally, this build will be done in a matter of milliseconds. I blame myself for not using Linux. To summarize, monorepos can be an awesome tool when you have a large complex project. The great thing about tools like turbo and n x is that they dramatically lower the barrier of entry to scale a massive code base.
video gaming
A mono repo might be overkill for your blog. But, if you're the technical co founder of a company, it's definitely an approach to building software that you want to be aware of. I'm gonna go ahead and wrap things up there. If you want access to more advanced JavaScript development content, consider becoming a pro member at Fireship IO. Thanks for watching, and I will see you in the next one.
A software company like Google maintains a lot of code. Like, seriously, a lot. Like, over 2,000,000,000 lines of code and nearly a 100 terabytes of data to go along with it. It's a scale that's hard to even comprehend, like all the stars in the universe or all the universes in the multiverse. And you have thousands of around the world working on it simultaneously. But get this, they store all their code in a single monolithic repository and they've been doing since the very beginning. Today, Google's mono repo is likely the largest code base in the world, but it takes an extraordinary effort to scale. They have their own home grown version control system and a highly advanced build tool called Bazel, which goes by the much cooler name of Blaze internally. In today's video, you'll learn everything you ever wanted to know data out mono repos. And how you, the humble JavaScript developer, can build a high performance mono repo in your garage. If you're new here, like and subscribe. We're so close to 1,000,000, and if You hit the button right now, we might even make it there by Christmas. You may have heard some exciting news last week that Vercel acquired a company called Turborepo. Data. It's a build tool written in Go that makes it really easy to manage multiple apps and packages in a single Git repository. But first, let's answer the question of why would I want to use a monorepo? There are many reasons, but at the highest level, it gives you visibility of your company's entire code base without the need to track down and clone a bunch of different repos. In addition, it provides consistency because you can share things like your ESLint config, a UI library of web components for your design system, utility libraries, documentation, and so on. The real power though comes in the form of dependency management. Imagine somebody makes a breaking change to a shared library. All affected applications will know instantly, and monorepo tools can actually help you visualize the entire dependency graph of your software. When it comes to third party a monorepo can dedupe packages that are used in multiple apps. A monorepo is also ideal for continuous integration and automation, because your code is already unified by default, making it much easier to build and test everything together. But there is one big problem with monorepos and that's the fact that they're big. As the monorepo becomes larger, there are more things to test, more things to build and a lot more artifacts to store as a disk. Versus Code will lag trying to process the massive git history and you'll need a 20 minute smoke break waiting for everything to run on the CI server after every commit. To operate a monorepo at scale, it's absolutely essential to have the right tooling. That's why Facebook created Buck, Microsoft created Rush, dang Google created Bazel. You just need a PhD in order to use it. Luckily, there are other options out there. The most basic approach is to use your package manager, like darn or NPM to define workspaces. These tools basically configure your project with a root level package JSON, which then has nested workspaces like apps and packages that are linked back to the root level project. A cool thing about this is that it will dedupe your node modules, which means if you have the same package installed in multiple apps, it will only be installed once. It also allows you to orchestrate scripts, like, if you want to build or test all of your apps at the same time. That's a good easy place to start, but if you're building an open source project data that publishes a bunch of different packages, then you'll likely want to look into a tool called Lerna, which is a tool that can optimize the workflow of a multi package repo. Here's an example. Turf JS is a geospatial library that has a ton of different packages that are essentially helper functions for working with geolocation data. Each one of them can be installed as its own package and lives in its own subdirectory here on the repo. Lerna is the tool that it uses to help manage its workflow efficiently. Most importantly, it allows you to publish all of your packages to digits to NPM with a single command. These tools are great at configuring monorepos, but they still suffer from the same problem I mentioned earlier. They become really slow and difficult to work with as they grow larger. One problem is the installation of dependencies. If you're looking to improve your install speed, an easy optimization to make is to replace NPM with PNPM. It's a drop in replacement that will install your dependencies globally and sim link them, and that can make your install speeds up to 3 times faster. That's a nice upgrade, but what really makes a monorepo slow is the constant need to recompile, rebuild, and retest everything. And that brings us to the fun part of the video where we talk about an entirely different class of tools that can make your monorepo operate at the speed of Google. The tools I'll be comparing are n x and turbo repo. They both operate as what I would as a smart build system. What I mean by that is they create a dependency tree between all of your apps and packages, which allows the tooling to understand what data to be tested and what needs to be rebuilt whenever there's a change to the code base. They cache any files or artifacts that have already been built and can also run jobs in parallel to execute everything much faster. We'll look at some example code in just a second, but first, what is the difference between NX and Turborepo? NX has been around for about 5 years and was created by 2 ex Googlers. Turborepo, on the other hand, was just open source the other day. It was created by Jared Palmer, who you might know from the React ecosystem system with packages like Formic. At this point, Turborepo is a lot more minimal than nx. Nx can do everything that Turborepo does and has a lot of features beyond like a CLI that can automatically generate boilerplate code for you, a plug in ecosystem, a Versus Code extension, and something called distributed task execution, DIN, which allows you to distribute work across multiple continuous integration servers. It's a pretty amazing feature from a technical standpoint and was inspired by Bazel. On one hand, it's great to have all these features, But on the other hand, it might be a little more than you actually need. NX has been criticized for requiring too much configuration, although, I don't necessarily agree with data. If you're only using the core NX features, that configuration is actually very minimal. That being said, Turborepo feels like a tool that goes out of its way data to get out of your way. For example, if you're already using yarn or npm workspaces, all you have to do is add a few lines of configuration to your package JSON, and you now have a magically super fast monorepo. When it comes to dependencies, both tools visualize the dependency graph, although nx does a much better job at this. But again, it's had a lot more time to bake in the data. Another cool thing that both of them do is remote caching. With Turborepo, when it caches something, it can also cache those files on Vercel. That means if Bob builds the duplication, then Alice checks it out on a different computer, the entire cache can be downloaded remotely instead of wastefully rebuilding and recomputing everything in the monorepo. And that can save a huge amount of time for a large organization. The final comparison point I wanna make is that Turborepo is written in Go, while n x is written in TypeScript. In theory, this could result in startup times when running Turborepo, but the underlying language is not where the performance gains come from. It depends on when the computations take place and how they're d cache, but I'd be interested to see some benchmarks between these two tools. Now it's time for you to build your first monorepo. I already have a video on n x if you wanna check that out, but today we're going to take day. 1st look at Turborepo. You can add it to an existing monorepo where they also have a template to start from scratch. Run npx create turbo from the command line, and that should bring up a menu with some gradient text that says Turborepo. It'll give you the option for a package manager and I'll go with yarn as the default. When we open up the project in Versus Code, you'll see that we have 2 root level directories for apps and packages, where apps are the actual applications you deploy, like a Next. Js or React app for example. And packages are the different configurations, utilities, and UI libraries that those apps depend on. Now if we open up the root level package JSON, you'll notice we have work databases set up for apps and packages there as well. Then towards the bottom, you'll notice the turbo configuration. This is where you configure a pipeline to run tasks efficiently. Normally, in a monorepo, everything runs 1 by 1. You do all your linting first, then you do all your builds, then your testing, and finally your deployments. Disk. With a pipeline, you can explicitly tell turbo what a task depends on. Like, you can run Lint whenever, but to run your builds, you'll want to make sure that all the dependencies have been built first. The caret symbol refers to dependencies. Then to test the application, we'll want to make sure that that build has completed and to deploy, we'll ensure that build, test, and have all been completed. A pipeline like this allows you to condense the timeline and utilize more CPU resources to deliver faster builds and that's pretty much all it takes to get started. We data. Now build or develop all of our apps in parallel. If we run yarn run dev, it serves both of the Next. Js applications in this repo at the same time. Time. In the terminal, you can see where it's using the cache. In the browser, you get 2 apps running at the same time without having to deal with multiple processes. An interesting thing to notice here is that if we go in data package. Json in one of the applications, you'll notice that it has dependencies like UI that are linked to a star character, and the same for config d and tsconfig. These are dependencies that live in the monorepo under the packages directory. What's cool about this is that if we go into our UI package and make a change, the change will be reflected instantly in both applications at the same time. There's no need to recompile and reinstall a dependency. Everything is continuously integrated into the entire codebase. And and the awesome thing about turbo is that everything becomes much faster by default. Like, if we run the build command, the first time around might take a couple minutes, but when we run it a second time Everything will be cached and that run might only take a few 100 milliseconds, at least in theory. There's a bug on Windows right now that's ruining my video, but Normally, this build will be done in a matter of milliseconds. I blame myself for not using Linux. To summarize, monorepos can be an awesome tool when you have a large complex project. The great thing about tools like turbo and n x is that they dramatically lower the barrier of entry to scale a massive code base. A mono repo might be overkill for your blog. But, if you're the technical co founder of a company, it's definitely an approach to building software that you want to be aware of. I'm gonna go ahead and wrap things up there. If you want access to more advanced JavaScript development content, consider becoming a pro member at Fireship IO. Thanks for watching, and I will see you in the next one.
A
a
software
software
company
company
like
like
Google
google
maintains
maintains
a
a
lot
lot
of
of
code.
code
Like,
like
seriously,
seriously
a
a
lot.
lot
Like,
like
over
over
2,000,000,000
2,000,000,000
lines
lines
of
of
code
code
and
and
nearly
nearly
a
a
100
100
terabytes
terabytes
of
of
data
data
to
to
go
go
along
along
with
with
it.
it
It's
it's
a
a
scale
scale
that's
that's
hard
hard
to
to
even
even
comprehend,
comprehend
like
like
all
all
the
the
stars
stars
in
in
the
the
universe
universe
or
or
all
all
the
the
universes
universes
in
in
the
the
multiverse.
multiverse
And
and
you
you
have
have
thousands
thousands
of
of
around
around
the
the
world
world
working
working
on
on
it
it
simultaneously.
simultaneously
But
but
get
get
this,
this
they
they
store
store
all
all
their
their
code
code
in
in
a
a
single
single
monolithic
monolithic
repository
repository
and
and
they've
they've
been
been
doing
doing
since
since
the
the
very
very
beginning.
beginning
Today,
today
Google's
google's
mono
mono
repo
repo
is
is
likely
likely
the
the
largest
largest
code
code
base
base
in
in
the
the
world,
world
but
but
it
it
takes
takes
an
an
extraordinary
extraordinary
effort
effort
to
to
scale.
scale
They
they
have
have
their
their
own
own
home
home
grown
grown
version
version
control
control
system
system
and
and
a
a
highly
highly
advanced
advanced
build
build
tool
tool
called
called
Bazel,
bazel
which
which
goes
goes
by
by
the
the
much
much
cooler
cooler
name
name
of
of
Blaze
blaze
internally.
internally
In
in
today's
today's
video,
video
you'll
you'll
learn
learn
everything
everything
you
you
ever
ever
wanted
wanted
to
to
know
know
data
data
out
out
mono
mono
repos.
repos
And
and
how
how
you,
you
the
the
humble
humble
JavaScript
javascript
developer,
developer
can
can
build
build
a
a
high
high
performance
performance
mono
mono
repo
repo
in
in
your
your
garage.
garage
If
if
you're
you're
new
new
here,
here
like
like
and
and
subscribe.
subscribe
We're
we're
so
so
close
close
to
to
1,000,000,
1,000,000
and
and
if
if
You
you
hit
hit
the
the
button
button
right
right
now,
now
we
we
might
might
even
even
make
make
it
it
there
there
by
by
Christmas.
christmas
You
you
may
may
have
have
heard
heard
some
some
exciting
exciting
news
news
last
last
week
week
that
that
Vercel
vercel
acquired
acquired
a
a
company
company
called
called
Turborepo.
turborepo
Data.
data
It's
it's
a
a
build
build
tool
tool
written
written
in
in
Go
go
that
that
makes
makes
it
it
really
really
easy
easy
to
to
manage
manage
multiple
multiple
apps
apps
and
and
packages
packages
in
in
a
a
single
single
Git
git
repository.
repository
But
but
first,
first
let's
let's
answer
answer
the
the
question
question
of
of
why
why
would
would
I
i
want
want
to
to
use
use
a
a
monorepo?
monorepo
There
there
are
are
many
many
reasons,
reasons
but
but
at
at
the
the
highest
highest
level,
level
it
it
gives
gives
you
you
visibility
visibility
of
of
your
your
company's
company's
entire
entire
code
code
base
base
without
without
the
the
need
need
to
to
track
track
down
down
and
and
clone
clone
a
a
bunch
bunch
of
of
different
different
repos.
repos
In
in
addition,
addition
it
it
provides
provides
consistency
consistency
because
because
you
you
can
can
share
share
things
things
like
like
your
your
ESLint
eslint
config,
config
a
a
UI
ui
library
library
of
of
web
web
components
components
for
for
your
your
design
design
system,
system
utility
utility
libraries,
libraries
documentation,
documentation
and
and
so
so
on.
on
The
the
real
real
power
power
though
though
comes
comes
in
in
the
the
form
form
of
of
dependency
dependency
management.
management
Imagine
imagine
somebody
somebody
makes
makes
a
a
breaking
breaking
change
change
to
to
a
a
shared
shared
library.
library
All
all
affected
affected
applications
applications
will
will
know
know
instantly,
instantly
and
and
monorepo
monorepo
tools
tools
can
can
actually
actually
help
help
you
you
visualize
visualize
the
the
entire
entire
dependency
dependency
graph
graph
of
of
your
your
software.
software
When
when
it
it
comes
comes
to
to
third
third
party
party
a
a
monorepo
monorepo
can
can
dedupe
dedupe
packages
packages
that
that
are
are
used
used
in
in
multiple
multiple
apps.
apps
A
a
monorepo
monorepo
is
is
also
also
ideal
ideal
for
for
continuous
continuous
integration
integration
and
and
automation,
automation
because
because
your
your
code
code
is
is
already
already
unified
unified
by
by
default,
default
making
making
it
it
much
much
easier
easier
to
to
build
build
and
and
test
test
everything
everything
together.
together
But
but
there
there
is
is
one
one
big
big
problem
problem
with
with
monorepos
monorepos
and
and
that's
that's
the
the
fact
fact
that
that
they're
they're
big.
big
As
as
the
the
monorepo
monorepo
becomes
becomes
larger,
larger
there
there
are
are
more
more
things
things
to
to
test,
test
more
more
things
things
to
to
build
build
and
and
a
a
lot
lot
more
more
artifacts
artifacts
to
to
store
store
as
as
a
a
disk.
disk
Versus
versus
Code
code
will
will
lag
lag
trying
trying
to
to
process
process
the
the
massive
massive
git
git
history
history
and
and
you'll
you'll
need
need
a
a
20
20
minute
minute
smoke
smoke
break
break
waiting
waiting
for
for
everything
everything
to
to
run
run
on
on
the
the
CI
ci
server
server
after
after
every
every
commit.
commit
To
to
operate
operate
a
a
monorepo
monorepo
at
at
scale,
scale
it's
it's
absolutely
absolutely
essential
essential
to
to
have
have
the
the
right
right
tooling.
tooling
That's
that's
why
why
Facebook
facebook
created
created
Buck,
buck
Microsoft
microsoft
created
created
Rush,
rush
dang
dang
Google
google
created
created
Bazel.
bazel
You
you
just
just
need
need
a
a
PhD
phd
in
in
order
order
to
to
use
use
it.
it
Luckily,
luckily
there
there
are
are
other
other
options
options
out
out
there.
there
The
the
most
most
basic
basic
approach
approach
is
is
to
to
use
use
your
your
package
package
manager,
manager
like
like
darn
darn
or
or
NPM
npm
to
to
define
define
workspaces.
workspaces
These
these
tools
tools
basically
basically
configure
configure
your
your
project
project
with
with
a
a
root
root
level
level
package
package
JSON,
json
which
which
then
then
has
has
nested
nested
workspaces
workspaces
like
like
apps
apps
and
and
packages
packages
that
that
are
are
linked
linked
back
back
to
to
the
the
root
root
level
level
project.
project
A
a
cool
cool
thing
thing
about
about
this
this
is
is
that
that
it
it
will
will
dedupe
dedupe
your
your
node
node
modules,
modules
which
which
means
means
if
if
you
you
have
have
the
the
same
same
package
package
installed
installed
in
in
multiple
multiple
apps,
apps
it
it
will
will
only
only
be
be
installed
installed
once.
once
It
it
also
also
allows
allows
you
you
to
to
orchestrate
orchestrate
scripts,
scripts
like,
like
if
if
you
you
want
want
to
to
build
build
or
or
test
test
all
all
of
of
your
your
apps
apps
at
at
the
the
same
same
time.
time
That's
that's
a
a
good
good
easy
easy
place
place
to
to
start,
start
but
but
if
if
you're
you're
building
building
an
an
open
open
source
source
project
project
data
data
that
that
publishes
publishes
a
a
bunch
bunch
of
of
different
different
packages,
packages
then
then
you'll
you'll
likely
likely
want
want
to
to
look
look
into
into
a
a
tool
tool
called
called
Lerna,
lerna
which
which
is
is
a
a
tool
tool
that
that
can
can
optimize
optimize
the
the
workflow
workflow
of
of
a
a
multi
multi
package
package
repo.
repo
Here's
here's
an
an
example.
example
Turf
turf
JS
js
is
is
a
a
geospatial
geospatial
library
library
that
that
has
has
a
a
ton
ton
of
of
different
different
packages
packages
that
that
are
are
essentially
essentially
helper
helper
functions
functions
for
for
working
working
with
with
geolocation
geolocation
data.
data
Each
each
one
one
of
of
them
them
can
can
be
be
installed
installed
as
as
its
its
own
own
package
package
and
and
lives
lives
in
in
its
its
own
own
subdirectory
subdirectory
here
here
on
on
the
the
repo.
repo
Lerna
lerna
is
is
the
the
tool
tool
that
that
it
it
uses
uses
to
to
help
help
manage
manage
its
its
workflow
workflow
efficiently.
efficiently
Most
most
importantly,
importantly
it
it
allows
allows
you
you
to
to
publish
publish
all
all
of
of
your
your
packages
packages
to
to
digits
digits
to
to
NPM
npm
with
with
a
a
single
single
command.
command
These
these
tools
tools
are
are
great
great
at
at
configuring
configuring
monorepos,
monorepos
but
but
they
they
still
still
suffer
suffer
from
from
the
the
same
same
problem
problem
I
i
mentioned
mentioned
earlier.
earlier
They
they
become
become
really
really
slow
slow
and
and
difficult
difficult
to
to
work
work
with
with
as
as
they
they
grow
grow
larger.
larger
One
one
problem
problem
is
is
the
the
installation
installation
of
of
dependencies.
dependencies
If
if
you're
you're
looking
looking
to
to
improve
improve
your
your
install
install
speed,
speed
an
an
easy
easy
optimization
optimization
to
to
make
make
is
is
to
to
replace
replace
NPM
npm
with
with
PNPM.
pnpm
It's
it's
a
a
drop
drop
in
in
replacement
replacement
that
that
will
will
install
install
your
your
dependencies
dependencies
globally
globally
and
and
sim
sim
link
link
them,
them
and
and
that
that
can
can
make
make
your
your
install
install
speeds
speeds
up
up
to
to
3
3
times
times
faster.
faster
That's
that's
a
a
nice
nice
upgrade,
upgrade
but
but
what
what
really
really
makes
makes
a
a
monorepo
monorepo
slow
slow
is
is
the
the
constant
constant
need
need
to
to
recompile,
recompile
rebuild,
rebuild
and
and
retest
retest
everything.
everything
And
and
that
that
brings
brings
us
us
to
to
the
the
fun
fun
part
part
of
of
the
the
video
video
where
where
we
we
talk
talk
about
about
an
an
entirely
entirely
different
different
class
class
of
of
tools
tools
that
that
can
can
make
make
your
your
monorepo
monorepo
operate
operate
at
at
the
the
speed
speed
of
of
Google.
google
The
the
tools
tools
I'll
i'll
be
be
comparing
comparing
are
are
n
n
x
x
and
and
turbo
turbo
repo.
repo
They
they
both
both
operate
operate
as
as
what
what
I
i
would
would
as
as
a
a
smart
smart
build
build
system.
system
What
what
I
i
mean
mean
by
by
that
that
is
is
they
they
create
create
a
a
dependency
dependency
tree
tree
between
between
all
all
of
of
your
your
apps
apps
and
and
packages,
packages
which
which
allows
allows
the
the
tooling
tooling
to
to
understand
understand
what
what
data
data
to
to
be
be
tested
tested
and
and
what
what
needs
needs
to
to
be
be
rebuilt
rebuilt
whenever
whenever
there's
there's
a
a
change
change
to
to
the
the
code
code
base.
base
They
they
cache
cache
any
any
files
files
or
or
artifacts
artifacts
that
that
have
have
already
already
been
been
built
built
and
and
can
can
also
also
run
run
jobs
jobs
in
in
parallel
parallel
to
to
execute
execute
everything
everything
much
much
faster.
faster
We'll
we'll
look
look
at
at
some
some
example
example
code
code
in
in
just
just
a
a
second,
second
but
but
first,
first
what
what
is
is
the
the
difference
difference
between
between
NX
nx
and
and
Turborepo?
turborepo
NX
nx
has
has
been
been
around
around
for
for
about
about
5
5
years
years
and
and
was
was
created
created
by
by
2
2
ex
ex
Googlers.
googlers
Turborepo,
turborepo
on
on
the
the
other
other
hand,
hand
was
was
just
just
open
open
source
source
the
the
other
other
day.
day
It
it
was
was
created
created
by
by
Jared
jared
Palmer,
palmer
who
who
you
you
might
might
know
know
from
from
the
the
React
react
ecosystem
ecosystem
system
system
with
with
packages
packages
like
like
Formic.
formic
At
at
this
this
point,
point
Turborepo
turborepo
is
is
a
a
lot
lot
more
more
minimal
minimal
than
than
nx.
nx
Nx
nx
can
can
do
do
everything
everything
that
that
Turborepo
turborepo
does
does
and
and
has
has
a
a
lot
lot
of
of
features
features
beyond
beyond
like
like
a
a
CLI
cli
that
that
can
can
automatically
automatically
generate
generate
boilerplate
boilerplate
code
code
for
for
you,
you
a
a
plug
plug
in
in
ecosystem,
ecosystem
a
a
Versus
versus
Code
code
extension,
extension
and
and
something
something
called
called
distributed
distributed
task
task
execution,
execution
DIN,
din
which
which
allows
allows
you
you
to
to
distribute
distribute
work
work
across
across
multiple
multiple
continuous
continuous
integration
integration
servers.
servers
It's
it's
a
a
pretty
pretty
amazing
amazing
feature
feature
from
from
a
a
technical
technical
standpoint
standpoint
and
and
was
was
inspired
inspired
by
by
Bazel.
bazel
On
on
one
one
hand,
hand
it's
it's
great
great
to
to
have
have
all
all
these
these
features,
features
But
but
on
on
the
the
other
other
hand,
hand
it
it
might
might
be
be
a
a
little
little
more
more
than
than
you
you
actually
actually
need.
need
NX
nx
has
has
been
been
criticized
criticized
for
for
requiring
requiring
too
too
much
much
configuration,
configuration
although,
although
I
i
don't
don't
necessarily
necessarily
agree
agree
with
with
data.
data
If
if
you're
you're
only
only
using
using
the
the
core
core
NX
nx
features,
features
that
that
configuration
configuration
is
is
actually
actually
very
very
minimal.
minimal
That
that
being
being
said,
said
Turborepo
turborepo
feels
feels
like
like
a
a
tool
tool
that
that
goes
goes
out
out
of
of
its
its
way
way
data
data
to
to
get
get
out
out
of
of
your
your
way.
way
For
for
example,
example
if
if
you're
you're
already
already
using
using
yarn
yarn
or
or
npm
npm
workspaces,
workspaces
all
all
you
you
have
have
to
to
do
do
is
is
add
add
a
a
few
few
lines
lines
of
of
configuration
configuration
to
to
your
your
package
package
JSON,
json
and
and
you
you
now
now
have
have
a
a
magically
magically
super
super
fast
fast
monorepo.
monorepo
When
when
it
it
comes
comes
to
to
dependencies,
dependencies
both
both
tools
tools
visualize
visualize
the
the
dependency
dependency
graph,
graph
although
although
nx
nx
does
does
a
a
much
much
better
better
job
job
at
at
this.
this
But
but
again,
again
it's
it's
had
had
a
a
lot
lot
more
more
time
time
to
to
bake
bake
in
in
the
the
data.
data
Another
another
cool
cool
thing
thing
that
that
both
both
of
of
them
them
do
do
is
is
remote
remote
caching.
caching
With
with
Turborepo,
turborepo
when
when
it
it
caches
caches
something,
something
it
it
can
can
also
also
cache
cache
those
those
files
files
on
on
Vercel.
vercel
That
that
means
means
if
if
Bob
bob
builds
builds
the
the
duplication,
duplication
then
then
Alice
alice
checks
checks
it
it
out
out
on
on
a
a
different
different
computer,
computer
the
the
entire
entire
cache
cache
can
can
be
be
downloaded
downloaded
remotely
remotely
instead
instead
of
of
wastefully
wastefully
rebuilding
rebuilding
and
and
recomputing
recomputing
everything
everything
in
in
the
the
monorepo.
monorepo
And
and
that
that
can
can
save
save
a
a
huge
huge
amount
amount
of
of
time
time
for
for
a
a
large
large
organization.
organization
The
the
final
final
comparison
comparison
point
point
I
i
wanna
wanna
make
make
is
is
that
that
Turborepo
turborepo
is
is
written
written
in
in
Go,
go
while
while
n
n
x
x
is
is
written
written
in
in
TypeScript.
typescript
In
in
theory,
theory
this
this
could
could
result
result
in
in
startup
startup
times
times
when
when
running
running
Turborepo,
turborepo
but
but
the
the
underlying
underlying
language
language
is
is
not
not
where
where
the
the
performance
performance
gains
gains
come
come
from.
from
It
it
depends
depends
on
on
when
when
the
the
computations
computations
take
take
place
place
and
and
how
how
they're
they're
d
d
cache,
cache
but
but
I'd
i'd
be
be
interested
interested
to
to
see
see
some
some
benchmarks
benchmarks
between
between
these
these
two
two
tools.
tools
Now
now
it's
it's
time
time
for
for
you
you
to
to
build
build
your
your
first
first
monorepo.
monorepo
I
i
already
already
have
have
a
a
video
video
on
on
n
n
x
x
if
if
you
you
wanna
wanna
check
check
that
that
out,
out
but
but
today
today
we're
we're
going
going
to
to
take
take
day.
day
1st
1st
look
look
at
at
Turborepo.
turborepo
You
you
can
can
add
add
it
it
to
to
an
an
existing
existing
monorepo
monorepo
where
where
they
they
also
also
have
have
a
a
template
template
to
to
start
start
from
from
scratch.
scratch
Run
run
npx
npx
create
create
turbo
turbo
from
from
the
the
command
command
line,
line
and
and
that
that
should
should
bring
bring
up
up
a
a
menu
menu
with
with
some
some
gradient
gradient
text
text
that
that
says
says
Turborepo.
turborepo
It'll
it'll
give
give
you
you
the
the
option
option
for
for
a
a
package
package
manager
manager
and
and
I'll
i'll
go
go
with
with
yarn
yarn
as
as
the
the
default.
default
When
when
we
we
open
open
up
up
the
the
project
project
in
in
Versus
versus
Code,
code
you'll
you'll
see
see
that
that
we
we
have
have
2
2
root
root
level
level
directories
directories
for
for
apps
apps
and
and
packages,
packages
where
where
apps
apps
are
are
the
the
actual
actual
applications
applications
you
you
deploy,
deploy
like
like
a
a
Next.
next
Js
js
or
or
React
react
app
app
for
for
example.
example
And
and
packages
packages
are
are
the
the
different
different
configurations,
configurations
utilities,
utilities
and
and
UI
ui
libraries
libraries
that
that
those
those
apps
apps
depend
depend
on.
on
Now
now
if
if
we
we
open
open
up
up
the
the
root
root
level
level
package
package
JSON,
json
you'll
you'll
notice
notice
we
we
have
have
work
work
databases
databases
set
set
up
up
for
for
apps
apps
and
and
packages
packages
there
there
as
as
well.
well
Then
then
towards
towards
the
the
bottom,
bottom
you'll
you'll
notice
notice
the
the
turbo
turbo
configuration.
configuration
This
this
is
is
where
where
you
you
configure
configure
a
a
pipeline
pipeline
to
to
run
run
tasks
tasks
efficiently.
efficiently
Normally,
normally
in
in
a
a
monorepo,
monorepo
everything
everything
runs
runs
1
1
by
by
1.
1
You
you
do
do
all
all
your
your
linting
linting
first,
first
then
then
you
you
do
do
all
all
your
your
builds,
builds
then
then
your
your
testing,
testing
and
and
finally
finally
your
your
deployments.
deployments
Disk.
disk
With
with
a
a
pipeline,
pipeline
you
you
can
can
explicitly
explicitly
tell
tell
turbo
turbo
what
what
a
a
task
task
depends
depends
on.
on
Like,
like
you
you
can
can
run
run
Lint
lint
whenever,
whenever
but
but
to
to
run
run
your
your
builds,
builds
you'll
you'll
want
want
to
to
make
make
sure
sure
that
that
all
all
the
the
dependencies
dependencies
have
have
been
been
built
built
first.
first
The
the
caret
caret
symbol
symbol
refers
refers
to
to
dependencies.
dependencies
Then
then
to
to
test
test
the
the
application,
application
we'll
we'll
want
want
to
to
make
make
sure
sure
that
that
that
that
build
build
has
has
completed
completed
and
and
to
to
deploy,
deploy
we'll
we'll
ensure
ensure
that
that
build,
build
test,
test
and
and
have
have
all
all
been
been
completed.
completed
A
a
pipeline
pipeline
like
like
this
this
allows
allows
you
you
to
to
condense
condense
the
the
timeline
timeline
and
and
utilize
utilize
more
more
CPU
cpu
resources
resources
to
to
deliver
deliver
faster
faster
builds
builds
and
and
that's
that's
pretty
pretty
much
much
all
all
it
it
takes
takes
to
to
get
get
started.
started
We
we
data.
data
Now
now
build
build
or
or
develop
develop
all
all
of
of
our
our
apps
apps
in
in
parallel.
parallel
If
if
we
we
run
run
yarn
yarn
run
run
dev,
dev
it
it
serves
serves
both
both
of
of
the
the
Next.
next
Js
js
applications
applications
in
in
this
this
repo
repo
at
at
the
the
same
same
time.
time
Time.
time
In
in
the
the
terminal,
terminal
you
you
can
can
see
see
where
where
it's
it's
using
using
the
the
cache.
cache
In
in
the
the
browser,
browser
you
you
get
get
2
2
apps
apps
running
running
at
at
the
the
same
same
time
time
without
without
having
having
to
to
deal
deal
with
with
multiple
multiple
processes.
processes
An
an
interesting
interesting
thing
thing
to
to
notice
notice
here
here
is
is
that
that
if
if
we
we
go
go
in
in
data
data
package.
package
Json
json
in
in
one
one
of
of
the
the
applications,
applications
you'll
you'll
notice
notice
that
that
it
it
has
has
dependencies
dependencies
like
like
UI
ui
that
that
are
are
linked
linked
to
to
a
a
star
star
character,
character
and
and
the
the
same
same
for
for
config
config
d
d
and
and
tsconfig.
tsconfig
These
these
are
are
dependencies
dependencies
that
that
live
live
in
in
the
the
monorepo
monorepo
under
under
the
the
packages
packages
directory.
directory
What's
what's
cool
cool
about
about
this
this
is
is
that
that
if
if
we
we
go
go
into
into
our
our
UI
ui
package
package
and
and
make
make
a
a
change,
change
the
the
change
change
will
will
be
be
reflected
reflected
instantly
instantly
in
in
both
both
applications
applications
at
at
the
the
same
same
time.
time
There's
there's
no
no
need
need
to
to
recompile
recompile
and
and
reinstall
reinstall
a
a
dependency.
dependency
Everything
everything
is
is
continuously
continuously
integrated
integrated
into
into
the
the
entire
entire
codebase.
codebase
And
and
and
and
the
the
awesome
awesome
thing
thing
about
about
turbo
turbo
is
is
that
that
everything
everything
becomes
becomes
much
much
faster
faster
by
by
default.
default
Like,
like
if
if
we
we
run
run
the
the
build
build
command,
command
the
the
first
first
time
time
around
around
might
might
take
take
a
a
couple
couple
minutes,
minutes
but
but
when
when
we
we
run
run
it
it
a
a
second
second
time
time
Everything
everything
will
will
be
be
cached
cached
and
and
that
that
run
run
might
might
only
only
take
take
a
a
few
few
100
100
milliseconds,
milliseconds
at
at
least
least
in
in
theory.
theory
There's
there's
a
a
bug
bug
on
on
Windows
windows
right
right
now
now
that's
that's
ruining
ruining
my
my
video,
video
but
but
Normally,
normally
this
this
build
build
will
will
be
be
done
done
in
in
a
a
matter
matter
of
of
milliseconds.
milliseconds
I
i
blame
blame
myself
myself
for
for
not
not
using
using
Linux.
linux
To
to
summarize,
summarize
monorepos
monorepos
can
can
be
be
an
an
awesome
awesome
tool
tool
when
when
you
you
have
have
a
a
large
large
complex
complex
project.
project
The
the
great
great
thing
thing
about
about
tools
tools
like
like
turbo
turbo
and
and
n
n
x
x
is
is
that
that
they
they
dramatically
dramatically
lower
lower
the
the
barrier
barrier
of
of
entry
entry
to
to
scale
scale
a
a
massive
massive
code
code
base.
base
A
a
mono
mono
repo
repo
might
might
be
be
overkill
overkill
for
for
your
your
blog.
blog
But,
but
if
if
you're
you're
the
the
technical
technical
co
co
founder
founder
of
of
a
a
company,
company
it's
it's
definitely
definitely
an
an
approach
approach
to
to
building
building
software
software
that
that
you
you
want
want
to
to
be
be
aware
aware
of.
of
I'm
i'm
gonna
gonna
go
go
ahead
ahead
and
and
wrap
wrap
things
things
up
up
there.
there
If
if
you
you
want
want
access
access
to
to
more
more
advanced
advanced
JavaScript
javascript
development
development
content,
content
consider
consider
becoming
becoming
a
a
pro
pro
member
member
at
at
Fireship
fireship
IO.
io
Thanks
thanks
for
for
watching,
watching
and
and
I
i
will
will
see
see
you
you
in
in
the
the
next
next
one.
one
success
The speakers discuss how Google's software company maintains a large code base and uses multi-year stacks to store all of their code. They also explain why a mon Republic repo is best for continuous integration and automation, as it provides visibility of the entire code base and consistency. The benefits of using a monor repo for faster development and efficient installation are discussed, along with benchmarks for the first monor repo. The process for building a monor repo is also mentioned, along with the use of a command line for building monoraps and the benefits of using a single repo for multiple projects. Finally, the speakers encourage attendees to consider becoming a pro member at Fireship IO.
Monorepos - How the Pros Scale Huge Software Projects // Turborepo vs Nx
A software company like Google maintains a lot of code. Like, seriously, a lot. Like, over 2,000,000,000 lines of code and nearly a 100 terabytes of data to go along with it. It's a scale that's hard to even comprehend, like all the stars in the universe or all the universes in the multiverse. And you have thousands of around the world working on it simultaneously. But get this, they store all their code in a single monolithic repository and they've been doing since the very beginning. Today, Google's mono repo is likely the largest code base in the world, but it takes an extraordinary effort to scale. They have their own home grown version control system and a highly advanced build tool called Bazel, which goes by the much cooler name of Blaze internally. In today's video, you'll learn everything you ever wanted to know data out mono repos. And how you, the humble JavaScript developer, can build a high performance mono repo in your garage. If you're new here, like and subscribe. We're so close to 1,000,000, and if You hit the button right now, we might even make it there by Christmas. You may have heard some exciting news last week that Vercel acquired a company called Turborepo. Data. It's a build tool written in Go that makes it really easy to manage multiple apps and packages in a single Git repository. But first, let's answer the question of why would I want to use a monorepo? There are many reasons, but at the highest level, it gives you visibility of your company's entire code base without the need to track down and clone a bunch of different repos. In addition, it provides consistency because you can share things like your ESLint config, a UI library of web components for your design system, utility libraries, documentation, and so on. The real power though comes in the form of dependency management. Imagine somebody makes a breaking change to a shared library. All affected applications will know instantly, and monorepo tools can actually help you visualize the entire dependency graph of your software. When it comes to third party a monorepo can dedupe packages that are used in multiple apps. A monorepo is also ideal for continuous integration and automation, because your code is already unified by default, making it much easier to build and test everything together. But there is one big problem with monorepos and that's the fact that they're big. As the monorepo becomes larger, there are more things to test, more things to build and a lot more artifacts to store as a disk. Versus Code will lag trying to process the massive git history and you'll need a 20 minute smoke break waiting for everything to run on the CI server after every commit. To operate a monorepo at scale, it's absolutely essential to have the right tooling. That's why Facebook created Buck, Microsoft created Rush, dang Google created Bazel. You just need a PhD in order to use it. Luckily, there are other options out there. The most basic approach is to use your package manager, like darn or NPM to define workspaces. These tools basically configure your project with a root level package JSON, which then has nested workspaces like apps and packages that are linked back to the root level project. A cool thing about this is that it will dedupe your node modules, which means if you have the same package installed in multiple apps, it will only be installed once. It also allows you to orchestrate scripts, like, if you want to build or test all of your apps at the same time. That's a good easy place to start, but if you're building an open source project data that publishes a bunch of different packages, then you'll likely want to look into a tool called Lerna, which is a tool that can optimize the workflow of a multi package repo. Here's an example. Turf JS is a geospatial library that has a ton of different packages that are essentially helper functions for working with geolocation data. Each one of them can be installed as its own package and lives in its own subdirectory here on the repo. Lerna is the tool that it uses to help manage its workflow efficiently. Most importantly, it allows you to publish all of your packages to digits to NPM with a single command. These tools are great at configuring monorepos, but they still suffer from the same problem I mentioned earlier. They become really slow and difficult to work with as they grow larger. One problem is the installation of dependencies. If you're looking to improve your install speed, an easy optimization to make is to replace NPM with PNPM. It's a drop in replacement that will install your dependencies globally and sim link them, and that can make your install speeds up to 3 times faster. That's a nice upgrade, but what really makes a monorepo slow is the constant need to recompile, rebuild, and retest everything. And that brings us to the fun part of the video where we talk about an entirely different class of tools that can make your monorepo operate at the speed of Google. The tools I'll be comparing are n x and turbo repo. They both operate as what I would as a smart build system. What I mean by that is they create a dependency tree between all of your apps and packages, which allows the tooling to understand what data to be tested and what needs to be rebuilt whenever there's a change to the code base. They cache any files or artifacts that have already been built and can also run jobs in parallel to execute everything much faster. We'll look at some example code in just a second, but first, what is the difference between NX and Turborepo? NX has been around for about 5 years and was created by 2 ex Googlers. Turborepo, on the other hand, was just open source the other day. It was created by Jared Palmer, who you might know from the React ecosystem system with packages like Formic. At this point, Turborepo is a lot more minimal than nx. Nx can do everything that Turborepo does and has a lot of features beyond like a CLI that can automatically generate boilerplate code for you, a plug in ecosystem, a Versus Code extension, and something called distributed task execution, DIN, which allows you to distribute work across multiple continuous integration servers. It's a pretty amazing feature from a technical standpoint and was inspired by Bazel. On one hand, it's great to have all these features, But on the other hand, it might be a little more than you actually need. NX has been criticized for requiring too much configuration, although, I don't necessarily agree with data. If you're only using the core NX features, that configuration is actually very minimal. That being said, Turborepo feels like a tool that goes out of its way data to get out of your way. For example, if you're already using yarn or npm workspaces, all you have to do is add a few lines of configuration to your package JSON, and you now have a magically super fast monorepo. When it comes to dependencies, both tools visualize the dependency graph, although nx does a much better job at this. But again, it's had a lot more time to bake in the data. Another cool thing that both of them do is remote caching. With Turborepo, when it caches something, it can also cache those files on Vercel. That means if Bob builds the duplication, then Alice checks it out on a different computer, the entire cache can be downloaded remotely instead of wastefully rebuilding and recomputing everything in the monorepo. And that can save a huge amount of time for a large organization. The final comparison point I wanna make is that Turborepo is written in Go, while n x is written in TypeScript. In theory, this could result in startup times when running Turborepo, but the underlying language is not where the performance gains come from. It depends on when the computations take place and how they're d cache, but I'd be interested to see some benchmarks between these two tools. Now it's time for you to build your first monorepo. I already have a video on n x if you wanna check that out, but today we're going to take day. 1st look at Turborepo. You can add it to an existing monorepo where they also have a template to start from scratch. Run npx create turbo from the command line, and that should bring up a menu with some gradient text that says Turborepo. It'll give you the option for a package manager and I'll go with yarn as the default. When we open up the project in Versus Code, you'll see that we have 2 root level directories for apps and packages, where apps are the actual applications you deploy, like a Next. Js or React app for example. And packages are the different configurations, utilities, and UI libraries that those apps depend on. Now if we open up the root level package JSON, you'll notice we have work databases set up for apps and packages there as well. Then towards the bottom, you'll notice the turbo configuration. This is where you configure a pipeline to run tasks efficiently. Normally, in a monorepo, everything runs 1 by 1. You do all your linting first, then you do all your builds, then your testing, and finally your deployments. Disk. With a pipeline, you can explicitly tell turbo what a task depends on. Like, you can run Lint whenever, but to run your builds, you'll want to make sure that all the dependencies have been built first. The caret symbol refers to dependencies. Then to test the application, we'll want to make sure that that build has completed and to deploy, we'll ensure that build, test, and have all been completed. A pipeline like this allows you to condense the timeline and utilize more CPU resources to deliver faster builds and that's pretty much all it takes to get started. We data. Now build or develop all of our apps in parallel. If we run yarn run dev, it serves both of the Next. Js applications in this repo at the same time. Time. In the terminal, you can see where it's using the cache. In the browser, you get 2 apps running at the same time without having to deal with multiple processes. An interesting thing to notice here is that if we go in data package. Json in one of the applications, you'll notice that it has dependencies like UI that are linked to a star character, and the same for config d and tsconfig. These are dependencies that live in the monorepo under the packages directory. What's cool about this is that if we go into our UI package and make a change, the change will be reflected instantly in both applications at the same time. There's no need to recompile and reinstall a dependency. Everything is continuously integrated into the entire codebase. And and the awesome thing about turbo is that everything becomes much faster by default. Like, if we run the build command, the first time around might take a couple minutes, but when we run it a second time Everything will be cached and that run might only take a few 100 milliseconds, at least in theory. There's a bug on Windows right now that's ruining my video, but Normally, this build will be done in a matter of milliseconds. I blame myself for not using Linux. To summarize, monorepos can be an awesome tool when you have a large complex project. The great thing about tools like turbo and n x is that they dramatically lower the barrier of entry to scale a massive code base. A mono repo might be overkill for your blog. But, if you're the technical co founder of a company, it's definitely an approach to building software that you want to be aware of. I'm gonna go ahead and wrap things up there. If you want access to more advanced JavaScript development content, consider becoming a pro member at Fireship IO. Thanks for watching, and I will see you in the next one.
The speakers discuss how Google's software company maintains a large code base and uses multi-year stacks to store all of their code. They also explain why a mon Republic repo is best for continuous integration and automation, as it provides visibility of the entire code base and consistency. The benefits of using a monor repo for faster development and efficient installation are discussed, along with benchmarks for the first monor repo. The process for building a monor repo is also mentioned, along with the use of a command line for building monoraps and the benefits of using a single repo for multiple projects. Finally, the speakers encourage attendees to consider becoming a pro member at Fireship IO.