text: Jekyll2020-12-19T14:21:17+00:00 DonohueTom D's blog, sharing tutorials, opinions, and stuff I'm working on. Tom DonohueWhy I’ve launched a new website2020-12-19T12:00:00+00:002020-12-19T14:21:14+00:00<p>I’ve been writing articles on this website (on this, and previous domains!) for over <strong>five years</strong> now. Yikes, that is a long time.</p> <p>Along the way, I’ve learned a lot about blogging.</p> <h2 id="i-just-want-to-create">I just want to create</h2> <p>During my blogging journey, I’ve learned that I love writing articles for you lot. I enjoy learning and explaining things.</p> <p>I also like sharing my knowledge and teaching others.</p> <p><strong>But more recently, I’ve begun to realise that there is a wealth of tech topics which I want to write about, and the content doesn’t really fit a “personal” website anymore.</strong></p> <p>In 2019, I bought the domain <em>tomd.xyz</em> and moved everything under this domain, to make all my content “personal”.</p> <p>But I’ve since realised that writing all my articles under the name “Tom Donohue’s Blog” is a bit restrictive. For example, how do I separate tech content, from personal blog posts, like this one?</p> <p>So, a bit of a back-track – or pivot, you could say.</p> <p>I’m launching a new site for my tech content. I want to let the new site speak for itself, and be a general hub for tech tutorials.</p> <p>In short: I’m changing things up a bit.</p> <h2 id="the-new-site">The new site</h2> <p>Just a few weeks ago, November 2020, I launched a new website, called <a href="">Tutorial Works</a>:</p> <figure class=""> <img src="/assets/img/tutorialworks-2020.webp" alt="Screenshot from Tutorial Works website" /> <figcaption> Tutorial Works - my new website </figcaption></figure> <p>Tutorial Works is all about tech content, tutorials and blog articles. Actionable stuff that you can use as a software engineer or architect.</p> <p>It’s still in its early stages right now, but I’ve got a lot of knowledge that I want to share on there.</p> <p>Tutorial Works will be similar to my blog, but it’ll be covering a wider range of topics, and going much wider and deeper.</p> <p>I’ll be writing about things like DevOps, Java, Kubernetes and containers, as I do now. And, I’ll also be writing about cloud providers, serverless, tech culture and much more.</p> <h2 id="the-challenges-ahead">The challenges ahead</h2> <p><del>Starting a new blog is difficult these days.</del> Getting traffic to a new blog is difficult these days.</p> <figure class="align-right"> <img src="" alt="GIF of Judge Judy saying Yes" /> <figcaption> Google decides…. </figcaption></figure> <p>Google – the ultimate judge and jury of websites, in many ways – doesn’t just index your content, and then throw you onto the first page of the search results the next day.</p> <p>Instead, Google prefers to <em>wait and see</em> whether your new website is any good. This is the so-called “Google sandbox” effect. It can sometimes take several months before Google decides that a new website should earn a place in the Google search results.</p> <p><strong>If you’re just starting your own developer blog, then persevere during this stage! It can be disheartening when it seems like Google doesn’t want to rank the content that you’ve poured hours and energy into writing.</strong></p> <p>In fact, right now, the site doesn’t have many visitors from Google at all:</p> <figure class="align-right"> <img src="/assets/img/tutorialworks-searchconsole.png" alt="Screenshot of Google Search Console results - all-time high of 2 clicks per day" /> <figcaption> The early days for a new blog - all-time high of 2 clicks in one day!! </figcaption></figure> <p>Given the effects of the Google sandbox, I’m not expecting much traffic. At the moment, I’m in the content “seeding” phase. If I have an idea for a new blog post, I’m currently writing it for Tutorial Works, not here.</p> <p>This means there’s already a good selection of content over on the new site which I think you’ll find useful.</p> <p>For example, in my most recent post, <a href="">Kubernetes Learning Resources</a>, I do a rundown of all my favourite tools for learning Kubernetes. The list includes all the resources that I’ve been recommending to people over the past few months.</p> <p>So it’s a long road, and this is the beginning.</p> <h2 id="creating-a-little-utopia">Creating a little utopia</h2> <p>“Utopia?!” CRINGE. OK, bear with me.</p> <p>I wrote an article earlier this year about <a href="/taking-action/">taking action</a>. Experimentation and trying new things are two of the best ways to learn.</p> <p>You can <strong>read</strong> all of the content and blogs that you like, about creating. But, if you really want to find out whether something will work, the only way forward is to <strong>actually create it.</strong> Start something today.</p> <p>The great thing about the internet is that you can start a website for free. You can write content and hit publish. Fortunately, big tech hasn’t managed to take this right away from us, yet.</p> <p>I could continue writing content for this personal blog, or I could exercise my right, and try creating something new and see what happens.</p> <p>It might succeed, it might not. I’m probably going to be publishing in the dark for months.</p> <p>But, as with this blog, I want to create the kind of website that I would find useful. As Derek Sivers says in one of my favourite books, <a href="">Anything You Want</a>:</p> <blockquote> <p>“When you make a business, you get to make a little universe where you control all the laws. This is your utopia.”</p> </blockquote> <p>Substitute the word “business” for “website”, and that’s what I’m aiming for.</p> <h2 id="so-whats-going-to-change">So what’s going to change?</h2> <p>Very little is going to change around here. I’ll still be publishing articles on this blog, and answering your comments and questions.</p> <p>My Camel articles are not going away. 😄</p> <p>But, I’m going to be writing a lot more, and most of my new content will be going onto <em>Tutorial Works</em> first.</p> <p>Firstly, I’m going to be writing more articles over the Christmas holidays (2020) and publishing a lot of this content over there in the coming weeks.</p> <p>And I’ll be using this, my personal blog, to be sharing more info. in case it inspires you to start your own blogging journey.</p> <p>I’m really excited to see what happens with the new website and I hope you can join me there!</p> <p>Want to read my latest content? Head on over to <a href="">Tutorial Works</a> now.</p> <p>Thanks for reading this. If you’ve got any questions or it’s sparked any creativity in you, I’d love to hear your comments.</p> <p>PS. Also, you should totally create your own website. :)</p>Tom DonohueI've started a new blog for my tech articles. Here's why.When should you use Apache Camel?2020-11-06T12:00:00+00:002020-12-19T14:21:14+00:00<p>Someone asked me a question the other day. They had heard about Apache Camel, but weren’t sure whether they should be using it. The person asked me “When should you use Apache Camel?”. What a great question! So I wanted to summarise what I know: when you should use Camel? When should you <strong>not</strong> use it? And how do people and companies use Apache Camel in the real world?</p> <p><strong>Apache Camel is an integration framework for Java. It’s most suited for situations where you want to fetch data from files or applications, process and combine it with other data, and then move it to another application.</strong></p> <aside> <nav> <header><h4></h4></header> <ul class="toc__menu" id="markdown-toc"> <li><a href="#what-is-apache-camel" id="markdown-toc-what-is-apache-camel">What is Apache Camel?</a></li> <li><a href="#when-should-you-use-apache-camel" id="markdown-toc-when-should-you-use-apache-camel">When should you use Apache Camel?</a></li> <li><a href="#how-do-people-use-camel-in-the-real-world" id="markdown-toc-how-do-people-use-camel-in-the-real-world">How do people use Camel in the real world?</a></li> <li><a href="#when-shouldnt-you-use-it" id="markdown-toc-when-shouldnt-you-use-it">When shouldn’t you use it?</a></li> <li><a href="#how-do-you-create-apps-with-apache-camel" id="markdown-toc-how-do-you-create-apps-with-apache-camel">How do you create apps with Apache Camel?</a></li> <li><a href="#final-thoughts-on-using-camel" id="markdown-toc-final-thoughts-on-using-camel">Final thoughts on using Camel</a></li> </ul> </nav> </aside> <p>Apache Camel is a great choice when you’re working with data that needs to be shared between systems. This happens when you have data stored in different applications. For example, personnel files might be stored in an HR system, but need need to be shared with Finance to be able to process the monthly payroll. Camel acts as this kind of programmatic glue between applications, implemented in Java.</p> <p>Are you wondering whether you should adopt Apache Camel in your own projects? Read on to find out more, and which kind of projects it’s right for.</p> <h2 id="what-is-apache-camel">What is Apache Camel?</h2> <aside class="context"> <header> <p>More on <strong>Apache Camel</strong></p> </header> <ul> <li class=""><p><a href="">Camel Tutorial</a></p></li> <li class="active"><p><a href="">When to use Apache Camel</a></p></li> <li class=""><p><a href="">What are routes in Apache Camel?</a></p></li> <li class=""><p><a href="">Create a REST service in Camel</a></p></li> <li class=""><p><a href="">Content-based routing in Camel</a></p></li> <li class=""><p><a href="">Testing Camel routes</a></p></li> </ul> </aside> <p><a href="">Apache Camel</a> is an integration toolkit or framework, written in Java.</p> <p>Camel comes as a set of libraries and components, and a language (called a <a href="">DSL</a>) for describing how you want to move data between applications. You can add it to your existing Java application or you can run it in a standalone Java application.</p> <p>Whenever you need to pull data from an application, remix it, merge it, and route it somewhere else, you can use Camel. Camel does this by providing:</p> <ul> <li> <p>a <strong>language</strong> (DSL) for writing your data flows, which are called <a href="/camel-routes/">routes</a> in Camel</p> </li> <li> <p>a set of <strong>patterns</strong> for implementing common things like error handling and transformation, which can be added to your Camel routes</p> </li> <li> <p>a <a href="">set of 300+ <strong>components</strong></a> for connecting to hundreds of different applications and protocols</p> </li> <li> <p>an embeddable <strong>runtime</strong> (the “Camel context”) which runs your integrations</p> </li> </ul> <p>Camel is distributed as a set of libraries (JAR files) and is released under the open source Apache License, under the Apache Software Foundation.</p> <h2 id="when-should-you-use-apache-camel">When should you use Apache Camel?</h2> <p>I found there are a few major drivers when you would want to use Apache Camel:</p> <figure class="small-right"> <img src="/assets/img/camel-toolkit.png" alt="The Apache Camel toolkit" /> <figcaption> Riding with the Camel </figcaption></figure> <ul class="checklist"> <li> <p><strong>Integrating applications together:</strong> Camel is intended for situations where you need to move data between different protocols and applications (like files, emails, APIs or web apps).</p> <p>You can use Apache Camel when you want to move data between any one of the apps and protocols supported by its 200+ components. Components in Camel generally all work in a similar way. So, once you understand how to use one component, you will find it easier to use others. Camel includes components for many different apps, from Facebook and Twitter, to Salesforce and Workday. You can also write your own custom component.</p> </li> <li> <p><strong>Pattern-based development:</strong> Many frequent requirements for integration – like support for transactions, or transformation – would usually be complicated to plan and write in code. But Camel provides many of these, and can often be enabled with just the flick of a switch (OK, by just changing a variable!). Camel provides patterns and functionality for things like:</p> <ul> <li> <p>routing data based on its content, using Camel’s <a href="/camel-choice-when/">content-based routing</a></p> </li> <li> <p>handling errors, transactions and rollbacks</p> </li> <li> <p>transforming data</p> </li> <li> <p>caching frequently-accessed data</p> </li> <li> <p>encryption and authentication</p> </li> </ul> <p>These are examples of just <strong>some</strong> of the things that Camel can do.</p> <p>These requirements are made easier in Camel, because it provides these features as a set of <a href="">patterns</a>, called enterprise integration patterns (after the <a href="/enterprise-integration-patterns-review/">book</a> of the same name). You can pull any of these enterprise integration patterns “off the shelf”, and use them in your code, without having to write your own solution every time you need these capabilities.</p> <p>This toolbox of patterns can make Camel a very productive way to write integration glue code, when you need to connect distributed systems.</p> <figure class="small-right"> <img src="/assets/img/highway.jpg" alt="A four-lane motor highway" /> <figcaption> Once you’ve learned to drive… </figcaption></figure> </li> <li> <p><strong>One high-level style for many integrations:</strong> Once you’ve learned the basic patterns, and how to work with Camel components, you’ll find that it becomes easy to churn out many integrations in Camel.</p> <p>This is an advantage of Camel: the ability to create many integrations fairly quickly. Camel is ideal if you are developing a set of integrations, and you would like them all to developed in a similar way. This can be really attractive option in larger companies where it helps to pick one approach which is shared and understood by the development team.</p> </li> <li> <p><strong>Working with data, and especially Java objects:</strong> As it’s a Java framework, it’s especially good at working with Java objects. So if you’re working with a file format that can be de-serialised into a Java object (many file formats can be converted into Java objects, like XML, JSON….) then it will be handled easily by Camel.</p> </li> </ul> <h2 id="how-do-people-use-camel-in-the-real-world">How do people use Camel in the real world?</h2> <p>What sort of real-world examples are there? How is Camel used in the wild?</p> <p>Here’s just a smattering of projects that I’ve either seen or been involved with. In each of these projects, companies are using Camel to achieve a certain goal. I’ve also included the related technologies in brackets:</p> <figure class="medium"> <img src="/assets/img/camel-world.png" alt="Illustration of a globe with some camels attached to it" /> <figcaption> The wide world of Camel </figcaption></figure> <ul> <li> <p><strong>Process and route data:</strong> Process customer orders and route them on to a database (Camel on Spring Boot with ActiveMQ Artemis and Qpid Dispatch Router)</p> </li> <li> <p><strong>Process web-submitted data:</strong> Receive and process/transform agricultural survey forms, and then insert into a database (Camel on Apache Karaf)</p> </li> <li> <p><strong>Financial transaction processing using message queues:</strong> Process financial transactions and route them to the correct department (Camel on Apache Karaf)</p> </li> <li> <p><strong>Put a gateway in front of your APIs:</strong> Implement a lightweight API Gateway, which authenticates and routes messages to the right API (Apache Camel on Spring Boot)</p> </li> <li> <p><strong>Data distribution:</strong> Poll for changes in HR data in a SaaS application, and distribute the changes to many downstream apps and files (Apache Camel on Spring Boot on Kubernetes)</p> </li> <li> <p><strong>Back-end order processing:</strong> Process car orders and car service requests (Apache Camel on Spring Boot, on Kubernetes)</p> </li> <li> <p><strong>Modernising legacy APIs:</strong> Expose data from a legacy ERP system as a REST API, so that clients can consume it (Apache Camel on Spring Boot, on Kubernetes)</p> </li> <li> <p><strong>Ad hoc data processing:</strong> Generate a sales report every day from data in a database (Apache Camel on Spring Boot)</p> </li> </ul> <p>Generally the best use cases for Camel are where you have a <strong>source</strong> of data that you want to consume from – e.g. incoming messages on a queue, or fetching data from an API – and a target, where you want to send the data to.<="when-shouldnt-you-use-it">When shouldn’t you use it?</h2> <p>So when shouldn’t you use Camel? There’s a time and a place for everything, and I think Camel is great but it can’t do it all:</p> <ul class="crosslist"> <li> <p><strong>For heavy data transformation:</strong> Although Camel is great at connecting to lots of different applications, it’s not designed for heavy data transformation and analysis. If you have a really data-heavy workflow, where you need to do lots of intensive merging and processing of data – for example like batch processing or ETL – then I think there are other tools which are better for the job.</p> <p>But, Camel is still great at orchestrating several steps together into a <em>flow</em>. So you might still consider using Camel as the orchestrator for your data transformation processes.</p> </li> <li> <p><strong>If you only need to write one single integration ‘flow’..:</strong> Camel is fairly lightweight, and it has got even more lightweight in recent releases (3.x onwards). But it’s probably not worth learning all the patterns and the whole Camel approach to integration development, if you just need it for writing one integration. If you just need to write one integration flow, you’re maybe better off writing your integration code yourself.</p> </li> <li> <p><strong>If you don’t have any Java skills in the team:</strong> As Camel is a Java framework, it requires some Java knowledge to use it. It doesn’t require much if you use the XML based DSL (language) for configuring it, but it still helps to have some Java knowledge to understand the concepts, because it usually runs on a Java Virtual Machine (JVM). This knowledge is especially useful so that you can troubleshoot when things go wrong: knowing how to read a stack trace always comes in handy!</p> </li> </ul> <h2 id="how-do-you-create-apps-with-apache-camel">How do you create apps with Apache Camel?</h2> <p>Apache Camel is free and open source. Getting started with Camel usually involves either creating a new Java application containing Camel, or adding it to your existing Java application.</p> <p>Camel can be deployed in lots of different ways:</p> <ul> <li> <p>Embedded in Spring Boot applications and microservices</p> </li> <li> <p>Embedded in Quarkus applications, for serverless and containers</p> </li> <li> <p>Embedded in Apache Tomcat</p> </li> <li> <p>Deployed into Apache Karaf, the OSGi container</p> </li> <li> <p>Deployed into WildFly, the Java application server</p> </li> <li> <p>Run natively on Kubernetes using <a href="">Camel-K</a></p> </li> <li> <p>As a simple, <a href="/camel-standalone-example/">standalone Java app</a> with a <code class="language-plaintext highlighter-rouge">main()</code> method</p> </li> </ul> <p>To get started, you add the Apache Camel dependencies to your application, create a Camel Context, write your integration <em>routes</em> in Camel’s DSL (either Java or XML), and then start the Context.</p> <p>A good way to create and run a simple Camel application is to read <a href="/camel-tutorial/">my Camel tutorial</a>, and then check out the <a href="">Camel examples</a> repository on GitHub, which is packed with useful example apps for you to learn from.</p> <p>Once you’ve developed your Camel-based integration app, you can deploy it to your server, virtual machine or a cloud environment.</p> <h2 id="final-thoughts-on-using-camel">Final thoughts on using Camel</h2> <p>Camel is a fantastic integration framework that I honestly love working with. It’s especially suited to application integration – that means it’s good if you need to consume, transform and route data between applications.</p> <p>Apache Camel is even more relevant today in the cloud era, where the number of applications in a typical enterprise just seems to be growing bigger and bigger.</p> <p><strong>So I hope you’ve found this guide useful, and now you know when to use Apache Camel!</strong> When you need to think about how to integrate and process data from applications, why not give Camel a try?</p>Tom DonohueSee the use cases for Apache Camel, and when you might think about using this Java integration framework.What are Camel routes?2020-10-16T10:00:00+01:002020-12-19T14:21:14+00:00<p>If you’re still learning the basics of Camel then you might have come across the term <strong>route</strong>.</p> <p>But what exactly is a Camel route?</p> <p><strong>A route in Apache Camel is a sequence of steps, executed in order by Camel, that consume and process a message. A Camel route starts with a consumer, and is followed by a chain of endpoints and processors.</strong></p> <p.</p> <h2 id="how-do-you-create-a-route-in-camel">How do you create a route in Camel?</h2> <p>To create a route in Camel, you first define it in code. This is called a <strong>route definition</strong>, and it is usually written in Java or XML.</p> <p>Then, you start Camel, passing it your route definition. Camel reads the route definition and creates the route inside the <em>Camel Context</em>. The Camel Context is an engine which is part of Camel, and which runs your routes.</p> <p>If you’re using some frameworks (like Spring Boot), Camel will try to discover your routes automatically.</p> <p>Once the Camel Context has started, you can <a href="/camel-monitoring-with-rest/">inspect it</a> to see all of the routes that are running.</p> <h2 id="an-example-camel-route">An example Camel route</h2> <p>To see what a route actually looks like, here’s an example.</p> <p>First, this route is written in the Java syntax or DSL (Domain-Specific Language). In Java, routes go inside a <code class="language-plaintext highlighter-rouge">RouteBuilder</code> class, which has a <code class="language-plaintext highlighter-rouge">configure()</code> method that you add your route code into:</p> <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">org.apache.camel.builder.RouteBuilder</span><span class="o">;</span> <span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyFirstRouteBuilder</span> <span class="kd">extends</span> <span class="nc">RouteBuilder</span> <span class="o">{</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">configure</span><span class="o">()</span> <span class="kd">throws</span> <span class="nc">Exception</span> <span class="o">{</span> <span class="n">from</span><span class="o">(</span><span class="s">"jms:queue:HELLO.WORLD"</span><span class="o">)</span> <span class="o">.</span><span class="na">log</span><span class="o">(</span><span class="s">"Received message - ${body}"</span><span class="o">);</span> <span class="o">}</span> <span class="o">}</span> </code></pre></div></div> <p>This route uses the JMS component, to receive a message from a queue called <em>HELLO.WORLD</em>. Then, it writes a log message to the console, containing the body of the message (<code class="language-plaintext highlighter-rouge">${body}</code>).<="camel-route-and-components-explained-infographic">Camel route and components explained (Infographic)</h2> <p>Sometimes it can be difficult to understand exactly what a route does, and how it works. So I created this infographic to show how the different concepts in Camel come together in a <strong>route</strong>.</p> <p>In this diagram, we have one route. We use the File component to read and write files. We also use a <a href="/camel-choice-when/">content-based router</a> to filter files based on their filename. The File component helps us by adding headers to the Exchange, which we can use in a Predicate expression.</p> <figure class=""> <img src="/assets/img/camel-route-explained.webp" alt="Infographic showing a Camel route, featuring components, endpoint URI, EIP, Predicate, content-based router and a Camel route" /> <figcaption> Routes in Apache Camel explained </figcaption></figure> <h2 id="how-do-you-call-a-camel-route-from-java-code">How do you call a Camel route from Java code?</h2> <p>If you’ve already embedded Camel in your application, then you can send a message to a Camel endpoint by using the Camel API.</p> <p>To send a message to a Camel route from your code, grab the <code class="language-plaintext highlighter-rouge">ProducerTemplate</code> object. Then:</p> <ul> <li>to send a message asynchronously, use one of the <code class="language-plaintext highlighter-rouge">sendXXX()</code> methods</li> <li>to send a message synchronously (and use the response), use one of the <code class="language-plaintext highlighter-rouge">requestXXX()</code> methods</li> </ul> <p>For example, if you were using Spring you might do something like this to send a message asynchronously (fire-and-forget) to an endpoint:</p> <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">org.apache.camel.ProducerTemplate</span><span class="o">;</span> <span class="kd">public</span> <span class="kd">class</span> <span class="nc">InvokeCamelRoute</span> <span class="o">{</span> <span class="nd">@Autowired</span> <span class="kd">protected</span> <span class="nc">ProducerTemplate</span> <span class="n">template</span><span class="o">;</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">invokeRoute</span><span class="o">()</span> <span class="o">{</span> <span class="n">template</span><span class="o">.</span><span class="na">sendBody</span><span class="o">(</span><span class="s">"direct:start"</span><span class="o">,</span> <span class="s">"Your message body goes here"</span><span class="o">);</span> <span class="o">}</span> <span class="o">}</span> </code></pre></div></div>Tom DonohueUnderstanding routes and how to use them in Apache CamelEnterprise Integration Patterns: Book Review2020-10-12T10:00:00+01:002020-12-19T14:21:14+00:00<p>The book “Enterprise Integration Patterns” was published in 2004. Written by Gregor Hohpe and Bobby Woolf, it was initially conceived as an academic paper. It was then refined and expanded into a very large, hardback book.</p> <p><strong>Enterprise integration</strong> is the sharing of data or commands between different applications. In some enterprises, there might be dozens, if not hundreds of applications, that all must be connected, for the business to operate - HR systems, finance databases, just to name a couple.</p> <p>The idea of the book is that enterprise integration problems can be solved using messaging.</p> <p>By using asynchronous messaging as a backbone or “channel”, integration becomes a series of asynchronous messages between applications.</p> <p>These messages can be used to trigger events, track processes or transform data.</p> <p>But the book is over 15 years old now. Is it still useful today?</p> <h2 id="why-messaging">Why messaging?</h2> <blockquote class="pullquote"> <p>“there are no simple answers for enterprise integration”</p> </blockquote> <p>The premise of the book is covered in the first chapter: <em>Solving Integration Problems Using Patterns</em>.</p> <p>Hohpe and Woolf saw the big ball of spaghetti. Even back in the early 2000s they understood that most companies run many applications and need to connect them together, and that it’s tough to do this.</p> <aside class="pull-right"> <p>See the <a href="">original academic paper</a> that led to <strong>Enterprise Integration Patterns</strong>.</p> </aside> <p>Fast-forward to today, and companies are are running ever more applications, connecting to cloud services and storing data in SaaS products. In other words, the problem still exists, and is growing.</p> <p>The book suggests that when we connect applications in a synchronous way, such as by using technologies like Web services, we enforce tight coupling between them.</p> <p>This means that a synchronous interaction can bind two components so closely together, that a failure in one application will cause a severe problem for the other. These tightly-coupled interactions make things, according to the book, “brittle, hard-to-maintain and poorly scalable”.</p> <p>Instead, asynchronous interactions, the book argues, are more flexible and loose. It talks about the advantages and disadvantages of using this approach, and then suggests a set of patterns that you can apply to the problems you’re trying to solve.</p> <figure class="medium"> <img src="/assets/img/eip-book-page.jpg" alt="A page from the Enterprise Integration Patterns book on Canonical Data Model" /> <figcaption> The infamous Canonical Data Model pattern </figcaption></figure> <h2 id="learning-the-background">Learning the background</h2> <p>One of the best parts of the book is how it sets the scene first.</p> <p.</p> <figure class="medium-right"> <img src="/assets/img/patch-cables.jpg" alt="Patch cables" /> <figcaption> Integration….it’s easy </figcaption></figure> <p>This is one of the strengths of this book.</p> <p>It starts with a few introductory chapters on integration and messaging, and then the remainder of the book is largely presented as a list of patterns. So you get to first understand the problem, before diving into the solutions.</p> <p>The first sections give a good introduction, along with an business scenario as an example, which explains the problems around enterprise integration. It also covers the possible approaches to integration (File Transfer, Shared Database, RPC, Messaging) before settling on messaging for the rest of the book.</p> <p>Later, when each pattern is introduced, the authors explain the problem it’s trying to solve, giving a detailed explanation and then some sample code. (But remember this book was published in 2004, so don’t expect any modern Java, like Spring!)</p> <h2 id="enterprise-integration-patterns">Enterprise Integration Patterns</h2> <p>And then, the book introduces patterns, one-by-one.</p> <p>An enterprise integration pattern is an abstract solution which can be applied to the problem of how to connect two applications together.</p> <figure class=""> <img src="/assets/img/eip-book-ref.jpg" alt="The inside back cover of Enterprise Integration Patterns" /> <figcaption> The back of the book includes a pattern reference guide </figcaption></figure> <p>In the book, you learn basic patterns about messaging itself, like:</p> <ul> <li> <p><strong>Message Channel</strong> - this is the “thing” that carries messages. For example: an address or queue on a message broker, like <a href="/camel-activemq/">ActiveMQ Artemis</a>, or a Destination in JMS.</p> </li> <li> <p><strong>Message</strong> - a way of wrapping a unit of data to send between applications. For example: a message that you place on a <a href="/jms/">JMS queue</a>.</p> </li> </ul> <p>And then you learn about patterns for processing messages, like:</p> <ul> <li> <p><strong>Splitter</strong> - a way of processing a message that contains multiple elements. For example: processing multiple lines in a CSV using <a href="">Apache Camel’s Splitter component</a>.</p> </li> <li> <p><strong>Message Translator</strong> - how to allow systems to communicate using messaging, even when the systems use different data formats. For example: mapping between Java POJOs using <a href="/dozer-camel-languages/">Apache Camel’s Dozer component</a>.</p> </li> </ul> <p>In all, there are over 60 patterns in the book.</p> <h2 id="who-is-this-book-for">Who is this book for?</h2> <p>I think that you will get a lot of value out of this book if:</p> <ul> <li> <p>You’re a developer who wants to learn about integration or retrain into the area, especially if you are learning Apache Camel</p> </li> <li> <p>You’re working with, or <a href="/camel-tutorial/">learning Apache Camel</a> (or Red Hat Fuse, or Talend ESB), or a message broker like ActiveMQ or RabbitMQ</p> </li> <li> <p>You are an application architect, looking at how to share data between two applications</p> </li> <li> <p>You are new to messaging, and you want to understand what’s possible, and why people would choose to use message queues</p> </li> <li> <p>You are an enterprise architect, looking to understand how other people have solved the same problems that you have</p> </li> <li> <p>You are a systems administrator, managing message brokers and infrastructure, and you want to understand how and why messaging is used</p> </li> </ul> <h2 id="the-good-parts">The good parts</h2> <p>So, onto the pros and cons of <em>Enterprise Integration Patterns</em>. Why should you buy it?</p> <ul class="checklist"> <li> <p>HTTP/synchronous style was popular for a while, but now messaging is finding favour again (Kafka, Event-Driven Architecture). The techniques in this book are semi-timeless and can be applied to problems, 15 years on from publication.</p> </li> <li> <p>You should absolutely buy this book if you’re going to be working with Apache Camel. You will appreciate much more how Camel works, and what it’s capable of.</p> </li> <li> <p>The book encourages loose coupling, which is a great school of thought for application architecture, and there are lots of patterns in the book to help you achieve this.</p> </li> <li> <p>It’s a good introduction to messaging and thinking asynchronously. It will be especially useful if you’re a new enterprise developer, and you might not have covered messaging in your university degree.</p> </li> </ul> <h2 id="the-not-so-good-parts">The not so good parts</h2> <ul class="crosslist"> <li> <p <code class="language-plaintext highlighter-rouge">WS-ReliableMessaging</code>, which we probably wouldn’t think of implementing now.</p> </li> <li> <p>The book mandates messaging. This might be obvious, given the subtitle of the book is <em>“Designing, Building, and Deploying Messaging Solutions”</em>. But modern integration isn’t just about asynchronous messaging. We have REST and gRPC APIs now, which aren’t the solution to everything, but are a firm part of the integration mix.</p> </li> <li> <p>Canonical Data Model - this is mentioned throughout. Whoops! This was a bit of an architects’ dream, from the days of centrally-planned architectures. In my experience, it was never practically implemented. How do you make updates to a canonical schema without upsetting everybody? See: <a href="">Conway’s Law</a>.</p> </li> </ul> <h2 id="verdict">Verdict</h2> <p><strong>Score: ⭐⭐⭐⭐ (4 out of 5)</strong></p> <aside class="pull-right"> <p><a target="_blank" href=""><img border="0" src="//ws-na.amazon-adsystem.com/widgets/q?_encoding=UTF8&MarketPlace=US&ASIN=0321200683&ServiceVersion=20070822&ID=AsinImage&WS=1&Format=_SL160_&tag=tomd1-20" /><.com<.co.uk</a></p> </aside> <p>If you are an integration architect, especially if you are architecting solutions with Apache Camel, I strongly recommend this book.</p> <p>Although you probably won’t find the code samples useful, the patterns in this book are close to timeless, and can still be applied to problems over 15 years later.</p> <p>A word of warning though: This book carries a hefty price tag!</p> <p>But make no mistake, this is <strong>a comprehensive book.</strong> In my edition, it runs to almost 700 pages. It’s not a book to read cover-to-cover, but a book to dip into and use as a reference.</p> <p>It sits in a prime position on my bookshelf and I think it’s essential!</p> <p><strong>Have you read the book? What did you think? Feel free to drop your comments below.</strong></p>Tom DonohueIt's a classic software architecture book. Is it worth a buy?Red Hat Fuse on OpenShift Tutorial [video]2020-09-20T09:00:00+01:002020-12-19T14:21:14+00:00<p>Let’s talk about Fuse.</p> <p>Red Hat Fuse (formerly JBoss Fuse) is Red Hat’s enterprise integration product. It’s a packaging of integration projects like Camel and CXF, on top of a runtime. In Fuse 6.x, the common runtime for Fuse used to be Apache Karaf. But these days, the common way of running Fuse is to deploy it onto OpenShift.</p> <aside> <h2 id="fuse-apache-camel-whats-the-difference">Fuse? Apache Camel? What’s the difference?</h2> <p>Fuse is a Red Hat <strong>product</strong> which includes various upstream <strong>projects</strong> like Apache Camel, Apache CXF and Spring Boot. In practice, companies use Fuse as a way of developing integration applications with these projects with Red Hat support. Check out more at the <a href="">Fuse Developer page</a>.</p> </aside> <h2 id="red-hat-fuse-on-openshift---developer-tutorial">Red Hat Fuse on OpenShift - Developer tutorial</h2> <p>This video is my short tutorial on how to get started with Red Hat Fuse on OpenShift. If you want to know how to run Apache Camel on OpenShift, then this one is for you! You’ll need a Red Hat account (you can get a <a href="">free developer account</a>) and access to an OpenShift cluster.</p> <!-- --> <p>This video is the steps I take when I install Fuse and create a new project for OpenShift. In the video, I cover:</p> <ul> <li>Looking at the <a href="">Fuse on OpenShift Guide</a> documentation</li> <li>Reminder to check out my <a href="/openshift-red-hat-registry/">blog on setting up Red Hat registry authentication for OpenShift</a></li> <li>Installing the Fuse image streams and templates</li> <li>Creating a new Fuse project with the Launcher</li> <li>Having a peek at the project and Camel routes in an IDE and making a code change</li> <li>Deploying it to an OpenShift cluster using the Fabric8 Maven Plugin</li> <li>Inspecting and testing the Fuse application in OpenShift</li> </ul> <!-- Courtesy of embedresponsively.com //--> <div class="responsive-video-container"> <iframe src="" frameborder="0" allowfullscreen=""></iframe> </div> <p>If you liked the video, please give it a thumbs up! I’d like to make more videos like this, so <strong>please let me know what you think in the comments below.</strong></p>Tom DonohueGetting started with Fuse? Follow this short video, where I create a Fuse app and deploy it onto OpenShift.Pulling images into OpenShift from the Red Hat registry2020-09-06T09:00:00+01:002020-12-19T14:21:14+00:00<p>How do you deploy Red Hat containers onto OpenShift? Containers for products like Fuse, AMQ and JBoss EAP?</p> <p>Red Hat publishes a <strong>lot</strong> of images for products in its own <a href="">Docker registry</a> at <code class="language-plaintext highlighter-rouge">registry.redhat.io</code>..</p> <p>And sometimes, it seems that OpenShift deploys stuff fine from the Red Hat registry without needing a password, but… sometimes it complains that you’re unauthorised! Why?</p> <p>Confused? I hear you. This article is especially for you, my friend.</p> <p><strong>🔥 Are you running OpenShift 4.5?</strong> There’s a special surprise for you <a href="#enhancement">at the end of this article</a>.</p> <h2 id="getting-knocked-back-from-the-registry">Getting knocked-back from the registry</h2> <p>Let’s start with a typical scene. A situation like this may have brought you here:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code" </code></pre></div></div> <p>It seems that OpenShift doesn’t want to pull an image from the Red Hat registry (<code class="language-plaintext highlighter-rouge">registry.redhat.io</code>) because the registry is private, and OpenShift doesn’t know how to authenticate to it.</p> <p>Seriously, what a drama. 🙄</p> <figure class="medium"> <img src="" alt="Drama queen" /> <figcaption> Please just deploy Fuse already </figcaption></figure> <p>To move forward, OpenShift needs credentials to authenticate to the registry.</p> <h2 id="do-i-need-to-provide-registry-credentials-myself-or-not">Do I need to provide registry credentials myself, or not?</h2> <p><strong>No, but yes. Here’s what you need to know.</strong></p> <p>An OpenShift cluster which been installed correctly has credentials for the Red Hat registry <a href="">installed onto each Node</a>. This allows you to pull images from the Red Hat registry on any Node. This means that you can do things like create a DeploymentConfig, or use <code class="language-plaintext highlighter-rouge">oc run</code> to deploy a Docker image directly.</p> <p>As an example, this is exactly what the <a href="">template for AMQ Broker</a> does. It deploys the image <code class="language-plaintext highlighter-rouge">registry.redhat.io/amq7/amq-broker:7.7</code>). I’ve snipped the relevant parts of the YAML here:</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">DeploymentConfig</span> <span class="na">spec</span><span class="pi">:</span> <span class="na">template</span><span class="pi">:</span> <span class="na">spec</span><span class="pi">:</span> <span class="na">containers</span><span class="pi">:</span> <span class="na">image</span><span class="pi">:</span> <span class="s">${IMAGE}</span> <span class="nn">...</span> <span class="na">parameters</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">description</span><span class="pi">:</span> <span class="s">Broker Image</span> <span class="na">displayName</span><span class="pi">:</span> <span class="s">Image</span> <span class="na">name</span><span class="pi">:</span> <span class="s">IMAGE</span> <span class="na">required</span><span class="pi">:</span> <span class="no">true</span> <span class="na">value</span><span class="pi">:</span> <span class="s">registry.redhat.io/amq7/amq-broker:7.7</span> </code></pre></div></div> <p><strong>But image streams and builds don’t work the same way.</strong>.</p> <p>Image streams and builds need to have registry credentials provided via a Secret. This Secret must be linked to the Service Account which will pull the images to run or build.</p> <h3 id="what-about-image-streams-in-the-openshift-project-they-seem-to-work">What about image streams in the OpenShift project? They seem to work!</h3> <p>Image streams in the <code class="language-plaintext highlighter-rouge">openshift</code> Project are, errrrr, special. They will work magically, because there’s usually <a href="">a Secret in there on OpenShift 4.x which contains credentials</a> (or see the <a href="">3.11 equivalent</a>) to authenticate to the Red Hat Container Registry at <code class="language-plaintext highlighter-rouge">registry.redhat.io</code>. Party time. Job done.</p> <p>But, as a developer, you almost certainly won’t have permissions to modify stuff in the <code class="language-plaintext highlighter-rouge">openshift</code> Project, like creating new image streams. This Project is fairly important and typically only administrators can do stuff inside it. So you’ll need to either have a friendly administrator, or work inside your own project.</p> <!-- --> <h2 id="creating-a-secret-to-authenticate-to-the-red-hat-registry">Creating a secret to authenticate to the Red Hat registry</h2> <p>So let’s create a secret which allows image streams and builds to pull images from the Red Hat registry. These steps will create a secret to allow OpenShift to pull images from the protected Red Hat registry <em>registry.redhat.io</em>, as required for image streams or builds.</p> <p>Registry credentials usually consist of a username and a password. This can either be your Red Hat username and password, or a specially generated username and token, which is the preferred way. We’ll do this in the steps below:</p> <ol class="tech-steps"> <li> <p>Grab your Red Hat login details, go to <a href=""></a> and create a new <em>Registry Service Account</em>.</p> <p>(If you’re not a Red Hat customer, and you’re just learning OpenShift, you can use your free Red Hat Developer account. This allows you to authenticate to the Red Hat Container Registry. <a href="">Join Red Hat Developer</a>.)</p> </li> <li> <p>Download the JSON file given on the page, or copy the credentials and create the new secret yourself:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>oc create secret docker-registry redhat-registry-secret \ --oc new-build</code>, etc.), link the Secret to the <code class="language-plaintext highlighter-rouge">builder</code> Service Account:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>oc secrets link builder redhat-registry-secret </code></pre></div> </div> <p>To make the secret available for deployments, link the Secret to the <code class="language-plaintext highlighter-rouge">default</code> Service Account, and add <code class="language-plaintext highlighter-rouge">for=pull</code>:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>oc secrets link default redhat-registry-secret --for=pull </code></pre></div> </div> </li> </ol> <h2 id="recap-setting-up-authentication-to-the-red-hat-registry">Recap: setting up authentication to the Red Hat registry</h2> <p>If you just want to deploy Docker images from the Red Hat registry (without using image streams), you shouldn’t need to do anything.</p> <p>If you want to create image streams or builds which use images from the Red Hat registry, you need to either:</p> <ul> <li> <p>Get your cluster administrator to create an image stream in the <code class="language-plaintext highlighter-rouge">openshift</code> project for the Red Hat registry image that you want to use. For example, your administrator can create an image stream for the <a href="">Red Hat Fuse</a> image using <code class="language-plaintext highlighter-rouge">oc import-image</code>:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>oc import-image fuse-java-openshift:1.7 \ --from=registry.redhat.io/fuse7/fuse-java-openshift:1.7 \ -n openshift --confirm </code></pre></div> </div> <p>✅ This is the ideal option, because you don’t have to worry about providing any credentials yourself. Yay.</p> </li> <li> <p>OR, if “administrator-says-no”, and you can only work with image streams in your own project, create a Secret containing your credentials for the Red Hat registry, and link it to a Service Account.</p> </li> </ul> <p>Now go and enjoy all those sweet, sweet container images.</p> <h2 id="enhancement">Epilogue: for OpenShift 4.5+ clusters <img src="/assets/img/new.gif" alt="new" /></h2> <p>In Steve Jobs style, here’s “one more thing”. If you’re running OpenShift 4.5, the good news is <strong>you don’t need to worry about any of this anymore.</strong> A new enhancement (<a href="">see it here</a>) has given OpenShift the ability to use the Node’s registry credentials during a build or image stream import.</p> <p>This means that from OpenShift 4.5, you no longer need to create your own Secret if you want to create an image stream or run a build with images from <code class="language-plaintext highlighter-rouge">registry.redhat.io</code>.</p> <p>So here’s me doing the same on an OCP 4.5 cluster:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ </code></pre></div></div> <p>Asciinema screen cast if you’re interested:</p> <asciinema-player</asciinema-player> <script src="/assets/js/asciinema-player.js"></script> <p>So all of that stuff further above applies to earlier releases of OCP 4.x, and 3.x, but you can forget it if you’re on 4.5 or greater. 💁 Thanks to Ricardo Maraschini for pointing this out.</p>Tom DonohueHow to use all those sweet images from the Red Hat registry, without nasty authorisation errorsA Brief Guide to DevOps2020-07-23T09:00:00+01:002020-12-19T14:21:14+00:00<p>DevOps. It’s a confusing topic, isn’t it? I’ve been asked a few times about it, and I wanted to take the time to write down how I see it.</p> ,</p> <p>So I hope you enjoy this brief introduction to DevOps for busy people. Or, if you’re too busy to read now, and you want a book to read on the go, then I’ve made a PDF of this which you can download.</p> <p><a href=""><b>Download the ebook version of this article (PDF)</b></a></p> <p>Let me know if you agree/disagree in the comments!</p> <aside> <nav> <header><h4></h4></header> <ul class="toc__menu" id="markdown-toc"> <li><a href="#what-is-devops" id="markdown-toc-what-is-devops">What is DevOps?</a></li> <li><a href="#the-three-ways-of-devops" id="markdown-toc-the-three-ways-of-devops">The Three Ways of DevOps</a></li> <li><a href="#the-devops-loop" id="markdown-toc-the-devops-loop">The DevOps Loop</a></li> <li><a href="#what-problem-does-devops-try-to-solve" id="markdown-toc-what-problem-does-devops-try-to-solve">What problem does DevOps try to solve?</a></li> <li><a href="#what-are-some-examples-of-devops" id="markdown-toc-what-are-some-examples-of-devops">What are some examples of DevOps?</a></li> <li><a href="#whats-not-good-devops" id="markdown-toc-whats-not-good-devops">What’s not good DevOps?</a></li> <li><a href="#what-techy-things-do-you-need-to-know" id="markdown-toc-what-techy-things-do-you-need-to-know">What techy things do you need to know?</a></li> <li><a href="#how-do-i-get-started-with-devops" id="markdown-toc-how-do-i-get-started-with-devops">How do I get started with DevOps?</a></li> <li><a href="#conclusion" id="markdown-toc-conclusion">Conclusion</a></li> </ul> </nav> </aside> <h2 id="what-is-devops">What is DevOps?</h2> <p>What exactly is DevOps? You might have heard people say things like:</p> <blockquote> <p>DevOps is a mindset! … It’s a way of working! … It’s culture!</p> </blockquote> <p>So what actually is it?! Is there a true definition?<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup></p> <p>To me, DevOps is a way of thinking about how companies can deliver better software, quicker.</p> <p>It promotes the idea that development and operations teams need to work more closely together, in order to make reliable software.</p> <p>A <strong>DevOps mindset</strong> is about improving developers’ ability to get quality software into production, and increasing developers’ involvement in how their applications are run.</p> <p>It’s also about using data and metrics to improve the quality of software, and sour ability to recover from faults. And it’s also about how to help operations teams handle applications that need to be released ever more frequently, sometimes several times a day.</p> <p>DevOps is about thinking how we can deliver software more quickly, by valuing open culture, collaboration and constant improvement.</p> <h2 id="the-three-ways-of-devops">The Three Ways of DevOps</h2> <p>It’s a bit of a fuzzy concept, and I’ve not found a definition which everyone agrees with.</p> <p>But, fortunately, there are three guiding principles of DevOps, to help us figure it out. These were initially <a href="">devised by Gene Kim</a>. They help define what DevOps is all about:</p> <ol class="tech-steps"> <li> <p>Improve <em>Flow</em> from business to customer</p> <p>How can we improve the <em>Flow</em> of our work, so that value is delivered to customers faster and smoother? (In this context, value means useful software features.)</p> <p>Can we break down large, risky chunks of work into smaller pieces, so they can be delivered to the customer more often? Or could we automate some tasks that we do repeatedly?</p> </li> <li> <p>Create and use feedback loops</p> <p?</p> </li> <li> <p>Learn and experiment continuously.</p> <p>This means making time and space for learning, and using the things that we learn to make the system better overall. That might mean improvements to how we build software, or how we can make the app better for users.</p> </li> </ol> <h2 id="the-devops-loop">The DevOps Loop</h2> <p>You might have seen people talk about DevOps and use a diagram like this. It’s sometimes called the “DevOps Loop”:</p> <figure class="medium"> <img src="/assets/img/devops-loop.png" alt="The DevOps Loop - Code, Build, Test, Release, Deploy, Operate, Monitor" /> <figcaption> The DevOps Loop </figcaption></figure> <p>This picture is a way of showing that building software isn’t a linear process, it’s a constant loop.</p> <p>When the first version of an application is released by developers, the work doesn’t just stop there! The output from development goes into operations. And then any learnings and output from operations can go back into development. And so we go round the loop again…</p> <p>So we can see that our job is never done. By going round this loop, we can make our product better and deliver more value.</p> <h2 id="what-problem-does-devops-try-to-solve">What problem does DevOps try to solve?</h2> <p>The problem DevOps tries to solve is <strong>slowness to change.</strong></p> <p>Remember the recent spike in online grocery shopping, due to the coronavirus pandemic? Businesses need to adapt fast to change.</p> <p>But to adapt, companies usually need to change parts of the software that runs their business. For example, they might need to add more servers to cope with demand.</p> <p>This might seem easy: just tell the development and operations teams to do it!</p> <p>But, there’s a problem: IT teams in companies are traditionally focused on providing stability (protecting the castle), not quick delivery of new features.</p> <p>Operations teams want to keep applications stable and up-and-running (maintaining profit).</p> <p>But development teams want to deliver new features quickly for the business (making more profit).</p> <p>These can seem like opposing goals.</p> <p>The result is that delivering new features can often be slow and expensive. By the time that a new feature is ready, the business opportunity might have been missed.</p> <p>This is a very simple example, but the principle applies across the board. Most software is created or changed for a business need, whether that’s increasing capacity to be able to take more orders, or something like making changes to the payroll system.</p> <p>DevOps tries to help companies deliver software that users need, faster. In this context, a user could be an employee or a customer - they’re both users.</p> <p>DevOps recognises the idea that development and operations teams are very important in providing value to the business. So the right culture needs to be in place to allow those teams to deliver that value sooner, and more predictably.</p> <!-- --> <h2 id="what-are-some-examples-of-devops">What are some examples of DevOps?</h2> <p>DevOps is about adopting culture and mindset, and not a specific set of rules. But if a company is adopting DevOps practices, you should be able to see some evidence of it, like these:</p> <ul> <li> <p>Development and operations teams working closely together</p> </li> <li> <p>New versions of applications are released several times a month, or even several times a day</p> </li> <li> <p>Developers’ code is automatically built and deployed, using scripts or pipelines</p> </li> <li> <p>Servers and infrastructure is built using automation scripts</p> </li> <li> <p>Applications are monitored by tools that collect and report information about application health</p> </li> <li> <p>Automated quality checks are run whenever new code is committed</p> </li> <li> <p>New features can move from idea into production very quickly</p> </li> <li> <p>Developers have more ownership of how their software is deployed and operated</p> </li> </ul> <p>There are many more examples than this! You’ll be able to spot DevOps when you see it.</p> <h2 id="whats-not-good-devops">What’s not good DevOps?</h2> <p>If a company isn’t adopting DevOps fully, you might see that development and operations teams that don’t work well together, and are siloed.</p> <p>You might also see that features can take a very long time to go from idea into production, because they have to run through many stages, or are bounced from team to team.</p> <p>If you see an operations team which has simply been renamed to the “DevOps team” - well, that just doesn’t make any sense, and is the opposite of what DevOps is about! It’s about better collaboration between existing teams, not just creating new ones.</p> <p>Suspicion between teams, or a lack of knowledge-sharing, are also signs that DevOps culture isn’t there yet.</p> <h2 id="what-techy-things-do-you-need-to-know">What techy things do you need to know?</h2> <p>You’ve noticed I’ve taken quite a while to talk about technology. This is because DevOps is not primarily about technology. It’s about people. It’s about enabling them to work together, and removing the barriers that stop them from doing so.</p> <p>But now, the word DevOps is just as much associated with specific tools, as it is with culture. Lots of job ads today use the title “DevOps Engineer” and simply list a set of tools.</p> <figure class=""> <img src="/assets/img/devops-loop-with-tools.png" alt="DevOps loop with software tools" /> <figcaption> DevOps Loop - with software and tools </figcaption></figure> .</p> <p>These are tools you might be interested in learning if you’re joining an organisation that uses DevOps, or you want to become a DevOps engineer.</p> <h3 class="no_toc" id="project-management-and-collaboration">Project management and collaboration</h3> <p.</p> <ul> <li>Examples: JIRA, Trello, Slack</li> </ul> <h3 class="no_toc" id="code-writing-quality-software">Code: writing quality software</h3> <p>Developers need tools that allow them to work on code together, without having to copy it onto memory sticks or send over email! They also need features that allow them to work on the same code at the same time.</p> <ul> <li>Examples: GitHub, Bitbucket, GitLab</li> </ul> <h3 class="no_toc" id="build-continuous-integration--cicd">Build: continuous integration & CI/CD</h3> <p>Good continuous integration is essential for DevOps. These tools are used to automate the process of compile code, run tests, and deploy the application. This means that the same process can be run dozens of times per day without effort.</p> <ul> <li>Examples: Jenkins, Travis, GitLab, Circle CI, Azure DevOps</li> </ul> <h3 class="no_toc" id="deploy-configuration-management--provisioning">Deploy: configuration management & provisioning</h3> <p>These tools are used to create the environments where applications will be deployed. They might configure servers, install software, or create objects in the cloud. They are controlled using code. This means that environments can be created at the touch of a button.</p> <ul> <li>Examples: Ansible, Terraform</li> </ul> <h3 class="no_toc" id="operate-run-the-application">Operate: run the application</h3> <p>When an application is built, it is then deployed onto a server or a cloud platform. These platforms run the applications.</p> <ul> <li>Examples: Kubernetes, Amazon EC2</li> </ul> <h3 class="no_toc" id="monitor-and-measure">Monitor and measure</h3> <p).</p> <ul> <li>Examples: Prometheus, Splunk, Grafana</li> </ul> <h2 id="how-do-i-get-started-with-devops">How do I get started with DevOps?</h2> <p>DevOps isn’t a tool, a methodology, or a product, or a service that you can buy. It’s not really meant to be a job title, or a team either!</p> <p>It’s a mindset, or a way of thinking about how teams can collaborate and work together. You might be doing this already without having “DevOps” in your job title or team name.</p> <p>You can start doing DevOps by doing things like:</p> <ul> <li> <p>Collaborating more openly with other teams, sharing your successes and failures so that others can improve</p> </li> <li> <p>Looking at your current software development process, finding the bottleneck (the part of the process that is difficult or takes a long time) and trying to improve it.</p> </li> <li> <p>Automating manual tasks, to give more confidence in your deployments and server builds</p> </li> <li> <p>Writing tests to check the quality of your code, so that it can be measured and continually improved</p> </li> <li> <p>Using a tool to automatically compile and test your code, like Jenkins</p> </li> <li> <p>Gather feedback from end users about the application, so that their ideas can go back into the loop and be developed as new features</p> </li> </ul> <h2 id="conclusion">Conclusion</h2> <p.</p> <hr /> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1" role="doc-endnote"> <p>It’s also hard to find a good definition, because the term has been adopted by tech companies to sell their products! <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p> </li> </ol> </div>Tom DonohueA brief introduction to the ideas of DevOpsEvent-driven integration on Kubernetes with Camel & KEDA2020-07-14T11:00:00+01:002020-12-19T14:21:14+00:00<p>I’ve been working with a team this week who are investigating how they could use <a href="">AWS Lambda</a> for data processing. The process is vaguely something like this:</p> <ol> <li> <p>write some data processing logic</p> </li> <li> <p>spin it up when an event happens (e.g. a message arrives on an SQS queue)</p> </li> <li> <p>terminate when finished.</p> </li> </ol> <p>So they want to use AWS Lambda for this, to make it scalable and serverless. These are going to create lightweight bits of data transformation logic, driven by events, that use minimal compute power. Sounds good so far.</p> <p.</p> <h2 id="keda-kubernetes-event-driven-autoscaling">KEDA: Kubernetes Event-Driven Autoscaling</h2> <p>After digging around for a little while, I found <a href="">KEDA - Kubernetes Event-Driven Autoscaling</a>. This is quite new and backed by Microsoft (Azure).</p> <p>KEDA is a way of auto-scaling applications in Kubernetes based on an external metric. The metrics are collected by a set of <strong>Scalers</strong> which support things like:</p> <ul> <li> <p>ActiveMQ Artemis</p> </li> <li> <p>Apache Kafka</p> </li> <li> <p>Amazon SQS</p> </li> <li> <p>Azure Service Bus</p> </li> <li> <p>and lots more….</p> </li> </ul> <p>The basic idea is that a KEDA Scaler monitors one of these objects for a metric. The metric is usually something like the number of messages on a queue.</p> <p>When the metric goes above a certain threshold, KEDA can scale up a Deployment automatically (called “Scaling Deployments”), or create a Job (called “Scaling Jobs”). It can also scale down Deployments when the metric goes down.</p> <p>It does this by creating a Horizontal Pod Autoscaler (HPA).</p> <p.</p> <p>And when I think of messaging, of course I immediately think of <a href="/camel-activemq/">ActiveMQ and Camel</a> and how KEDA could be used. I think I feel an example coming on….</p> <h2 id="keda-example-apache-camel-and-activemq">KEDA example: Apache Camel and ActiveMQ</h2> <p.</p> <figure class=""> <img src="/assets/img/keda-example-app.png" alt="Architecture diagram of KEDA Camel example application" /> <figcaption> A basic architecture for a KEDA-scaled Camel/ActiveMQ app </figcaption></figure> <h3 id="about-the-demo-app">About the demo app</h3> <p>I’ve created an example Camel app which uses Quarkus as the runtime. I’ve published the image to Docker Hub and I use that in the steps further below. But if you’re interested in how it was created, read on.</p> <p><a href=""><b>Get the code on GitHub</b></a> <a href=""><b>Get the image on Docker Hub</b></a></p> <p>I decided to use Quarkus because it boasts super-fast startup times, way faster than Spring Boot. When we’re reacting to events, we want to be able to start up quickly and not wait 30-60 seconds for the app to start.</p> <p>To create the app, I used the <a href="">Quarkus app generator</a>.</p> <p>As Quarkus is configured using extensions, I needed to find a Quarkus extension which would help me create a connection factory to talk to ActiveMQ Artemis. So I’m using the <a href="">Qpid JMS Extension for Quarkus</a>, which wraps up the <a href="">Apache Qpid JMS</a> client for Quarkus applications. This allows me to talk to ActiveMQ Artemis using the nice, open AMQP 1.0 protocol.</p> <p>The Qpid JMS extension creates a connection factory to ActiveMQ when it finds certain config properties. You only need to set the properties <code class="language-plaintext highlighter-rouge">quarkus.qpid-jms.url</code>, <code class="language-plaintext highlighter-rouge">quarkus.qpid-jms.username</code> and <code class="language-plaintext highlighter-rouge">quarkus.qpid-jms.password</code>. The Extension will do the rest automatically, as it says in the docs:</p> <figure class=""> <img src="/assets/img/keda-amqphub-quarkus.png" alt="Table showing the config properties required for Qpid JMS Quarkus extension" /> <figcaption> Checking out how to configure the Qpid JMS extension for Quarkus </figcaption></figure> <p>Then, I use Camel’s AMQP component to actually consume the messages. This will detect and use the connection factory created by the extension.</p> <p>I’ve compiled and packaged the application into a native binary, not a JAR. This will help it to start up very fast. You need GraalVM to be able to do this.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./mvnw package -Pnative </code></pre></div></div> <p>Or, if you don’t want to install GraalVM, you can tell Quarkus to use a helper <a href="">container with GraalVM baked in</a>, in order to build the native image. You’ll need Docker running for this, of course:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./mvnw package -Pnative -Dquarkus.native.container-build=true </code></pre></div></div> <p>The output from this is a native binary which should start up faster than a typical JVM-based application. Nice. Good for rapid scale-up when we receive a message!</p> <p>Finally, I built a container image with Docker and pushed it up to Docker Hub. There’s a Dockerfile provided with the Quarkus quickstart to do the build, and then it’s an easy <code class="language-plaintext highlighter-rouge">docker push</code>:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker build -f src/main/docker/Dockerfile.native -t monodot/camel-amqp-quarkus . docker push monodot/camel-amqp-quarkus </code></pre></div></div> <p>Now we’re ready to deploy the app, deploy KEDA and configure it to auto-scale the app.</p> <!-- --> <h3 id="deploying-keda-and-the-demo-app">Deploying KEDA and the demo app</h3> <ol class="tech-steps"> <li> <p>First, install KEDA on your Kubernetes cluster. You’ll probably need to have cluster-admin permissions to be able to do this.</p> <p><em>If you need a Kubernetes cluster of your own, you can use Minikube or a cloud offering like Amazon’s EKS. <a href="/aws-kubernetes/">Read about my experiences with EKS on AWS</a>).</em></p> <p>To install KEDA, you should probably follow the <a href="">instructions on the KEDA web site</a>, but I installed it with <a href="">Helm</a> like this:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ helm repo add kedacore $ helm repo update $ kubectl create namespace keda $ helm install keda kedacore/keda --namespace keda </code></pre></div> </div> </li> <li> <p>Create a namespace for our demo.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kubectl create namespace keda-demo </code></pre></div> </div> </li> <li> <p>Now we need to deploy an ActiveMQ Artemis message broker.</p> <p>Here’s some YAML to create a Service and Deployment for it in Kubernetes. It uses the <a href=""><code class="language-plaintext highlighter-rouge">vromero/activemq-artemis</code> community image of Artemis</a> on Docker Hub, and exposes its console and amqp ports. I’m customising it by adding a ConfigMap which:</p> <ul> <li> <p>Changes the internal name of the broker to a static name: <code class="language-plaintext highlighter-rouge">keda-demo-broker</code></p> </li> <li> <p>Defines one queue, called <code class="language-plaintext highlighter-rouge">ALEX.HONKING</code>. If we don’t do this, then the queue will be created when a consumer connects to it, but it will be removed again when the consumer disappears, and so KEDA will just get confused. So we define the queue first.</p> </li> </ul> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ </code></pre></div> </div> </li> <li> <p>Next, we deploy the demo Camel Quarkus AMQP consumer application, and add some configuration.</p> <p>So we create a Deployment. I’m deploying my demo image <code class="language-plaintext highlighter-rouge">monodot/camel-amqp-quarkus</code> from Docker Hub. You can also deploy my image, or you can build and deploy your own image if you want.</p> <p>We use the environment variables <code class="language-plaintext highlighter-rouge">QUARKUS_QPID_JMS_*</code> to set the URL, username and password for the ActiveMQ Artemis broker. These will override the properties <code class="language-plaintext highlighter-rouge">quarkus.qpid-jms.*</code> in my application’s properties file:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"> </code></pre></div> </div> </li> <li> <p>Now we tell KEDA to scale the pod down when there are no messages, and back up when there are messages.</p> <p>We do this by creating a <code class="language-plaintext highlighter-rouge">ScaledObject</code>. This tells KEDA which Deployment to scale, and when to scale it.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ </code></pre></div> </div> <p>By the way, to get the credentials to use the Artemis API, KEDA will look for any environment variables on the Deployment pods of the Camel app This means you don’t have to specify the credentials twice :-) So here, I’m using <code class="language-plaintext highlighter-rouge">QUARKUS_QPID_JMS_USERNAME</code> and <code class="language-plaintext highlighter-rouge">_PASSWORD</code>. They reference the environment variables on the demo app’s Deployment.</p> </li> <li> <p>Now let’s put some test messages onto the queue.</p> <p>You can do this in a couple of different ways: either point and click using the Artemis web console, or use the Jolokia REST API.</p> <p>Either way, we need to be able to reach the <code class="language-plaintext highlighter-rouge">artemis</code> K <code class="language-plaintext highlighter-rouge">localhost</code> port <code class="language-plaintext highlighter-rouge">8161</code>:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kubectl port-forward -n keda-demo svc/artemis 8161:8161 </code></pre></div> </div> <p>Leave that running in the background.</p> <p>Now, in a different terminal, hit the Artemis Jolokia API with <code class="language-plaintext highlighter-rouge">curl</code>, via the kubectl port-forwarding proxy. We want to send a message to an Artemis queue called ALEX.HONKING.</p> <p>This part requires a ridiculously long API call, so I’ve added some line breaks here to make it easier to read. This uses ActiveMQ’s Jolokia REST API to put a message in the Artemis queue:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code\"]}" </code></pre></div> </div> <p>(If you have any issues with this, just use the Artemis web UI to send a message, it’s at <code class="language-plaintext highlighter-rouge"></code>)</p> <p>All good!</p> </li> </ol> <p>You put messages in the queue, you should see the Camel app pod starting up and consuming the messages.</p> <figure class=""> <img src="/assets/img/keda-scaled-dashboard.jpg" alt="Kubernetes Dashboard showing Camel and Artemis pod" /> <figcaption> KEDA scaled up the demo app (camel-amqp-quarkus) when it noticed a message </figcaption></figure> <p>After all messages are consumed, there will be no messages left on the queue. KEDA waits for the cooldown period (in this demo I’ve used 30 seconds as an example), and then scales down the deployment back to zero, so there are no pods running.</p> <p>This is autoscaling… in action!</p> <h2 id="epilogue-you-forgot-knative">Epilogue: You forgot Knative</h2> <figure class="small-right"> <img src="/assets/img/knative.png" alt="Knative logo" /> <figcaption> Knative </figcaption></figure> <p>So you’ve probably noticed that I didn’t mention Knative.</p> <p>I started my research by looking at Knative. Knative is a beast of a project. It was announced a couple of years ago and is backed by Google.</p> <p>There are a couple of major parts to Knative, but the interesting ones to me are:</p> <ul> <li> <p><strong>Knative Serving</strong> -.</p> </li> <li> <p><strong>Knative Eventing</strong> - is the other half of the project, which is about making “events” a native concept in Kubernetes, and decoupling producers and consumers. You can then write apps which respond to these events.</p> </li> </ul> <p:</p> <figure class=""> <img src="/assets/img/knative-eventing-wip.png" alt="Screenshot of Knative Eventing open issue" /> <figcaption> Autoscaling for Knative Eventing is still WIP </figcaption></figure> <p>I think that means that Knative is out of the running. For now….</p>Tom DonohueCan we develop apps in Kubernetes that autoscale based on events? Perhaps, with this example using KEDA, ActiveMQ and Apache CamelKubernetes on AWS: a developer’s perspective2020-07-07T12:00:00+01:002020-12-19T14:21:14+00:00<p>It’s a sad fact that it’s taking more and more computing power to learn new things in programming these days!</p> <p>I love my ThinkPad X1 Carbon, but it’s not powerful enough to be running workloads worthy of any envy.</p> <p>Take Kubernetes, for example, or its vendor distributions like OpenShift. There are quite a few moving parts in it. Even a fairly lightweight Kubernetes cluster can require a lot of memory and CPU overhead.<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote">1</a></sup></p> <p>Not to mention, Kubernetes requires a fair amount of confidence to deploy and configure it on your own.</p> <figure class=""> <img src="/assets/img/weak-man.jpg" alt="Weak sports man" /> <figcaption> My laptop running Kubernetes with more than 1 Pod </figcaption></figure> <p.</p> <p>And then on top of that, there’s the extra overhead of things like a Service Mesh, serverless frameworks, <a href="/container-logs/">container log</a> aggregators and observability tools. Now you can’t hope to run all of these on your laptop, so you have to find other ways.</p> <p>This makes me quite sad. It makes the barrier-to-entry <strong>very high</strong> for new engineers, because you have to either have money for decent computing power, or whip out your credit card and spend on cloud computing.</p> <p>Anyway this is a rant for another day, perhaps.</p> <p>In this article I’m talking specifically about one way to get some beefy Kubernetes power on demand when you’re learning or hacking.</p> <h2 id="why-would-i-want-to-run-kubernetes">Why would I want to run Kubernetes?</h2> <p>First up, I should clarify something. I’m not talking about setting up a working production-like cluster. I’m talking about a cluster for a Kubernetes developer.</p> <figure class="small-right"> <img src="/assets/img/yak.jpg" alt="Illustration of a Yak with the letter 'Y'" /> <figcaption> A Yak, yesterday </figcaption></figure> <p!</p> <p>When I’m learning or hacking, there’s nothing worse than having to do a whole load of <strong>yak shaving</strong>. Task after task, just to get going. It really bursts my balloon, and makes me wish I’d never bothered!</p> <p>(reaches for a strong yet fruity drink 🍹)</p> <p>I would like an easy way to play with Kubernetes. I can get a Kubernetes developer cluster by using <a href="">Minishift</a>, or <a href="">Minikube</a>. But they come with the overhead of running a virtual machine on my laptop. Because of that, I quickly hit a ceiling for the workloads that I can run on my laptop.</p> <p>I am hungry…. for power.</p> <p>Some of my wonderful friends run their own OpenShift labs at home (<a href="">Exhibit 1</a>, <a href="">Exhibit 2</a>, <a href="">Exhibit 3</a>) but my flat is pretty small and there’s not enough room for a small server farm just now.</p> <p>So I thought, why not try running my own pet Kubernetes cluster in the cloud! It could be a good way to learn and run bigger workloads (and grant myself system admin privileges, thankyou very much…)</p> <p>So today I’ve been checking out AWS’s Kubernetes service, which is called <a href="">Elastic Kubernetes Service or EKS</a>.</p> <h2 id="what-is-eks">What is EKS?</h2> <p.</p> <figure class=""> <img src="/assets/img/eks-architecture.png" alt="Diagram showing process of creating an EKS cluster on AWS" /> <figcaption> EKS in a nutshell (credit: AWS) </figcaption></figure> <p>EKS is not to be confused with ECS, which is Amazon’s <strong>non-Kubernetes</strong> way of running containers in the cloud.</p> <p>I’m sure ECS is great and everything, but I’d rather use EKS, so I can be learning my Kubernetes primitives, and having a great time running around the <a href="">cloud-native sweet shop</a>, and installing the latest trendy software onto my cluster.</p> <figure class="medium"> <img src="/assets/img/pick-n-mix.jpg" alt="Pick'n'Mix Stall" /> <figcaption> The Cloud-Native Landscape, yesterday (picture now out of date) </figcaption></figure> <p>EKS effectively offers you a Kubernetes cluster with a single click, with everything configured for you. There is no Kubernetes Dashboard built in, but you can easily <a href="">install one</a>. It integrates with AWS IAM, so you use your AWS identity to authenticate to the cluster.</p> <p>There are more details on AWS’s <a href="">EKS landing page</a>, where my favourite sentence is:</p> <blockquote> <p>“EKS is the best place to run Kubernetes for several reasons.”</p> </blockquote> <figure class="threequarter-width"> <img src="/assets/img/eks-is-the-best.png" alt="Screenshot of AWS EKS sales copy" /> <figcaption> EKS is the best!!!!!!!! 4EVAA IDST </figcaption></figure> <p>I like this sentence, because it sounds like the kind of sentence I would have written at school when I was 12. Excerpt I would probably have written something like: “Computers are the best for several reasons.”</p> <h2 id="creating-an-eks-cluster-on-aws">Creating an EKS cluster on AWS</h2> <p>I was pleased at how easily I was able to set EKS up.</p> <p><a href="">I followed the getting started guide on the AWS docs</a>. I decided to point-and-click my way through the web UI (AWS Management Console). But you can use the <strong>eksctl</strong> command-line utility if you prefer (more on that below).</p> <p><em>Another tip if you’re a beginner and want to avoid unnecessary diversions, is to limit the amount of new tools you first need to understand to do the thing you want to do.</em></p> <p>It’s good to know that the control plane, which is where all the Kubernetes infrastructure lives (like the Kubernetes API server, etcd data store, and so on), seems to be hidden away from me, so I can’t break it.</p> <!-- --> <p>To create my cluster, I did these things:</p> <ul class="checklist"> <li> <p>Create an EKS cluster using the web UI - and you can give it a name, as if it was your own pet! I named mine <em>cillablack</em> of course:</p> <figure class="medium"> <img src="/assets/img/eks-clusters-cillablack.png" alt="Screenshot of EKS console showing 1 cluster" /> <figcaption> My first EKS cluster </figcaption></figure> </li> <li> <p>Add worker nodes to the cluster. These are added in things called Node Groups. These will create the nodes that actually <strong>run</strong> your containers. You need at least 1 of these to do anything useful. The “master” nodes, or “control plane”, where the Kubernetes API server runs, is not configured here.</p> <p.</p> </li> <li> <p>Grab credentials using the AWS CLI, and access the cluster using <code class="language-plaintext highlighter-rouge">kubectl</code>. This part involves a couple of fiddly steps, as I needed to upgrade to the latest version of the AWS CLI, and install the <code class="language-plaintext highlighter-rouge">aws-iam-authenticator</code> tool, which provides auth to the Kubernetes cluster.</p> </li> </ul> <p>But then, it was working! And without much effort, too.</p> <figure class=""> <img src="/assets/img/eks-kubectl.png" alt="Authenticating to an AWS EKS cluster" /> <figcaption> Authenticating to my AWS EKS cluster and testing </figcaption></figure> <aside class="notice notice--accent"> <p><strong>Note:</strong> if you’re creating an EKS cluster using the management console (web UI), make sure you also use that <strong>same</strong> user when you access it for the first time using the command line! You need to authenticate to the cluster as your web user (perhaps it’s your AWS root user), and not a different, or temporary, IAM user.</p> </aside> <h3 id="spin-up-a-cluster-fast-with-eksctl">Spin up a cluster fast with eksctl</h3> <p><strong>If you want to create a cluster even faster, you can use ‘eksctl’.</strong> eksctl is an “official” command-line utility for creating AWS EKS clusters. It is excellent. <a href="">Grab eksctl from here.</a></p> <p>Run <code class="language-plaintext highlighter-rouge">eksctl create cluster</code>? :-)</p> <p>Then at the end of your day, use eksctl to delete your cluster again, using <code class="language-plaintext highlighter-rouge">eksctl delete cluster <cluster name></code>. Magic.</p> <h2 id="where-are-the-worker-nodes">Where are the worker nodes?</h2> <p>When you create a Kubernetes cluster in AWS, the main thing to note is that the control plane is basically hidden from you. Here’s my list of EC2 instances in the AWS management console:</p> <figure class=""> <img src="/assets/img/eks-ec2-instances.png" alt="Screenshot of EC2 console showing 1 running instance" /> <figcaption> My single worker node in my EKS cluster </figcaption></figure> <p>I added 1 worker node to my EKS cluster. This can be seen here as an EC2 instance. AWS configures this for you and joins it to the EKS cluster.</p> <p)</p> <h2 id="testing-the-cluster">Testing the cluster</h2> <p>So how to test a fresh Kubernetes install? By running a Pod, of course.</p> <p>I used the <code class="language-plaintext highlighter-rouge">hello-world</code> image from Docker Hub. It just prints out a hello message.</p> <figure class=""> <img src="/assets/img/dockerhub-hello-world.png" alt="Screenshot of Docker Hub showing hello-world image" /> <figcaption> hello-world on Docker Hub - an easy test container </figcaption></figure> <p>You can run this against the new cluster. This command will create a Pod which runs the <code class="language-plaintext highlighter-rouge">hello-world</code> container:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ kubectl run hello-world --restart=Never --generator=run-pod/v1 --image=hello-world </code></pre></div></div> <p>Then check the logs and there should be a nice message from the container - it ran for me, and produced this output:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ kubectl logs pod/hello-world Hello from Docker! This message shows that your installation appears to be working correctly. </code></pre></div></div> <h2 id="its-just-kubernetes-but-thats-ok">It’s “just” Kubernetes… but that’s OK</h2> <p>The downside of EKS is that I’m “just” getting plain Kubernetes, of course. I don’t get any of the nice features that I’m used to with OpenShift, like <a href="/openshift-s2i-example/">container image builds</a>, an image registry and so on.</p> <p>Instead, I have to build my images myself and push them to Amazon ECR or to Docker Hub.</p> <p>It all takes a bit of getting used to, especially if you’ve mostly just been used to OpenShift.</p> <h2 id="open-thy-wallet-lets-talk-about-the-cost">Open thy wallet…. let’s talk about the cost</h2> <figure class="small-right"> <img src="/assets/img/thief.jpg" alt="Illustration of a thief stealing a purse from a handbag" /> <figcaption> Amazon, take my money </figcaption></figure> <p>If you want to start learning Kubernetes then you could do worse than spin up your own cluster on AWS. It doesn’t cost too much, as long as you delete clusters when you’re not using them.</p> <p>So you want to know the exact costs. Well, at the time of writing (Jul 2020), I’m paying about $0.10 per hour for my EKS cluster, <strong>plus</strong> the cost of my nodes. These are charged at Amazon’s usual EC2 instance prices, plus storage fees.</p> <p>I started with 1 node. So if I forget to spin a cluster down, it will cost $2.40 per day for the EKS cluster <strong>plus</strong> the cost of 1 x t3.small EC2 instance for 24 hours. This is a drop in the ocean for big companies, but for individual developers it can add up quickly. So definitely delete the cluster when you’re not using it.</p> <p>These would be my rough estimated costs to run EKS for 1 month with one t3.small node:</p> <ul> <li> <p>1 x node (EC2 t3.small) = $0.022 per hour = $15.84</p> </li> <li> <p>1 x EKS cluster = $0.10 per hour = $72</p> </li> <li> <p>20GB EBS storage = $3.30</p> </li> <li> <p><mark>Total: $91.14</mark></p> </li> </ul> <p>A lot of money for a very small cluster! I wouldn’t leave it running all the time, it’s just not worth it. So I’ll create it and shut it down when needed. My next step will probably be to create some automation to do that. <a href="/automation-worth-effort/">Automation, yaaaassssss!!!</a></p> <p>But would I recommend EKS for hacking around on Kubernetes? Yes! And it’s even easier if you use <strong>eksctl</strong> to create and delete your cluster.</p> <p>Right, now I’m off to shut down my cluster before my credit card self-destructs….</p> <h2 id="further-reading--links">Further reading & links</h2> <ul> <li> <p><a href="">My notes on Kubernetes</a></p> </li> <li> <p><a href="">Amazon Elastic Kubernetes Service - landing page</a></p> </li> <li> <p><a href="">Getting started with EKS using the AWS Management Console</a></p> </li> <li> <p><a href="">Install Kubernetes Dashboard onto EKS</a></p> </li> <li> <p><a href="">Amazon EKS Workshop</a></p> </li> <li> <p><a href="">Cloud-Native Computing Federation Landscape</a></p> </li> <li> <p><a href="">Learn Kubernetes basics</a></p> </li> <li> <p><a href="">Anatomy of my Kubernetes Cluster (Antonin Stefanutti)</a></p> </li> </ul> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1" role="doc-endnote"> <p>I’m not forgetting the <a href="">k3s</a> project, which is Kubernetes “lite” for Internet-of-Things (IoT) devices and folks who have smaller computing requirements. But I hope to cover that in another article someday. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p> </li> </ol> </div>Tom DonohuePlaying around and learning Kubernetes? I tried creating an EKS clusterIntroducing… Safety Curtain2020-07-03T12:00:00+01:002020-12-19T14:21:14+00:00<p>I launched a new micro-product this week. It’s called <a href="">Safety Curtain</a>! It’s a website where you can display a message of your choice, over a nice background. You can see it here:</p> <p><a href="" class="button btn btn--info btn--large">Open Safety Curtain</a></p> <p>It looks a bit like this:</p> <figure class=""> <img src="/assets/img/safetycurtain-boiledegg.jpg" alt="Screenshot of Safety Curtain website with message 'Gone for a boiled egg'" /> <figcaption> Safety Curtain </figcaption></figure> <p>OK, so you might be thinking, “What the heck is this, Tom? Is this something you found in a beginners’ JavaScript tutorial?” 😆 And to be honest, you may be right.</p> <p>But there is a story behind it! It’s a tiny product, made mostly for fun, to learn and experiment.</p> <p>Read on to find out more.</p> <h2 id="it-began-in-lockdown">It began… in lockdown</h2> <p>At the beginning of lockdown, I did a couple of remote consulting engagements, where I was doing code-pairing sessions with customers, using screen-sharing (we use <a href="">Bluejeans</a>).</p> <p>They were pretty long days, and we had a lot of work to do, so naturally we would break every so often for coffee or fresh air.</p> <p>To keep the video call open during breaks, I left my screen share on, showing a message in a word processor, a bit like a status message:</p> <figure class="threequarter-width"> <img src="/assets/img/libreoffice-back-at-2pm.png" alt="Screenshot of LibreOffice showing the words 'Back at 2PM'" /> <figcaption> The kind of message that I would leave on a call </figcaption></figure> <p>But, it is ugly. I thought it would be cool if there was a way to display this in a nicer way – with all the nice typography <strong>and colour</strong> of the web. And everybody has a web browser, so surely it could be web-based.</p> <p>That’s when I came up with the initial idea for <a href="">Safety Curtain</a>: a quick tool to display a message, whether you’re on a video call or a presentation. I added it to my <em>ideas.txt</em> list.</p> <p>. . .</p> <p>A couple of months passed (COVID-19 lockdowns have flown by very fast), and I still hadn’t made a start on the idea. But it was still floating around in the back of my mind.</p> <p>Why hadn’t I done any work on the idea? Because I was blocked, stuck in the daydreaming phase, and not taking any action.</p> <p>But, in recent weeks, I’ve absorbed a few thoughts on action that encouraged me to just do it. I’ve <a href="/taking-action/">documented these in a separate blog post here</a>.</p> <p>So this week I went from concept to “live” for a tiny product in 24 hours. Not a great feat for some, but for me that’s a big deal.</p> <!-- --> <h2 id="how-i-built-the-product">How I built the product</h2> <p>I knew it could all be done in JavaScript - but how? My knowledge of web development is pretty crap.</p> <p>The extent of my “development” knowledge is confined to the back-end (Java, Spring, <a href="/camel-tutorial/">Camel</a>, etc). And even within my domain, I’m also not a hardcore developer. I sit somewhere in the middle, sort of “DevOpsy-coder-lite”. :-)</p> <p!</p> <p>My product idea was just to create a simple web page, with some editable text that looks nice. I didn’t want to spend time learning a new framework for such a simple idea, so I tried to just stick to plain (“vanilla”) JavaScript.</p> <p>Some of <a href="">Chris Ferdinandi’s articles</a> were hugely helpful for this! It was good to read Chris’s page on <a href="">how to show and hide elements with vanilla JavaScript</a>.</p> <p>I also knew that I would need to know how to trigger actions when a link is clicked (e.g. on the Customise form, or the Full-Screen option), so I also delved a little deeper on event handlers in JavaScript.</p> <p>I also took some help with the design. Earlier this year I saw Andy Brewer’s <a href="">MVP.css make the front page on Hacker News</a>. I bookmarked it when I saw it on Hacker News, in case I needed to remember it later. (Aside: I strongly recommend checking Hacker News every so often, there are usually a few things on there each week that I bookmark for later)</p> <figure class="threequarter-width"> <img src="/assets/img/mvpcss-hn.png" alt="Extract from MVP.css posting on Hacker News" /> <figcaption> The item on Hacker News which helped me find MVP.css </figcaption></figure> <p>CSS is something I struggle with. Even popular frameworks like Bootstrap and Tailwind require a learning curve.</p> <p>But MVP.css doesn’t require much learning at all. It just adds styles to some basic HTML elements. It doesn’t introduce any complicated new classes or grid systems, and you can create something nice-looking with little effort:</p> <figure class=""> <img src="/assets/img/mvp-css.jpg" alt="MVP.css website" /> <figcaption> MVP.css simple CSS stylesheet </figcaption></figure> <p>I wanted to get building quickly, to capitalise on my initial enthusiasm. So finding MVP.css was great timing.</p> <p>With the JavaScript sorted, and the design, the site started to come together. I also found a very useful tool for <a href="">creating CSS gradient backgrounds</a> which allowed me to create the first colour scheme. And that was it.</p> <h2 id="publishing-and-launching">Publishing and launching</h2> <p>The site was essentially finished, but I was drawn to tweaking and fiddling with it.</p> <p>This final step is the hardest for me: polishing the product, adding a logo or brand, sharing it on social media… I tend to lose confidence at this stage.</p> <p>But, this time I pushed through, and forced myself to launch and share it on the same day.</p> <p>I published the site.</p> <p>I submitted it to <a href="">Product Hunt</a>. I submitted it at 8pm GMT, and received just 1 upvote, which I assume was mine! But it was still exciting to go through the process of submitting it and putting the product out there.</p> <p>I also <a href="">tweeted</a> about it:</p> <figure class="half-width"> <img src="/assets/img/tweet-safetycurtain.png" alt="Screenshot of my tweet launching Safety Curtain" /> <figcaption> Launching the product on Twitter </figcaption></figure> <p>So I went from writing the first line of code, to having the site up and running in just over 24 hours, using only my free time outside of work. It may not look like much, but for me, it feels like a big achievement to see a <strong>working product</strong> online in such a short space of time.</p> <p>And yes it’s a working product. You could use it on a conference call next week with anyone.</p> <h2 id="learnings--tricks-to-get-unstuck">Learnings + tricks to get unstuck</h2> <p>So, I shipped something! I got stuck, but not too much.</p> <p>Here are the three things I did which I think helped me finished something before I had a chance to get bored or stuck:</p> <ul> <li> <p><strong>I didn’t use any new frameworks or tools.</strong>.</p> <p>This project was the perfect opportunity to learn some core JavaScript and build a simple, lightweight website.</p> </li> <li> <p><strong>I avoided the thing that I’m not good at (design).</strong>.</p> <p>But not this time! I used the excellent <a href="">MVP.css bare-bones CSS stylesheet</a> by <a href="">Andy Brewer</a>. This simple stylesheet got me going. I still got stuck trying to vertically center some text on the page, but I’ve had that same problem for about 20 years…</p> </li> <li> <p><strong>I didn’t set up a domain name.</strong> For previous side-projects, I would have spent ages poring over the perfect domain name (2-3 days), then had to register it, set up nameservers and wait while it propagated (another 1-2 days).</p> <p>Instead, I used the first name I thought of. I set up the website on a subdomain of this site (<strong>curtain.tomd.xyz</strong>). I use <a href="">Linode</a> to host my sites, so I was quickly able to set up the new subdomain in my Linode Manager, configure Nginx and publish. You could have done this quicker than me if you use something like <a href="">Netlify</a>.</p> </li> </ul> <h2 id="the-next-steps">The next steps</h2> <p>What are my next plans? Well, Safety Curtain is a finished product now. It’s usable. But I’m always thinking about what to do next. So:</p> <p><strong>Add more features…</strong>.</p> <p><strong>To code or not to code…</strong>.</p> <p>That’s it for my intro to my latest micro-product. Thanks for reading! Any feedback/questions? Let me know in the comments below.</p>Tom DonohueFrom lockdown idea to MVP: I launched a new tiny product this week