text:
Read more on the Go Challenge website
1 2 3 4 5 6 7 8 9 |
The challenge is going on for 2 weeks until March 15, a new challenge will be posted on April first. Prizes and conditions available on the website. Comments and discussions available on HackerNews
Note that I also wrote another challenge that we usually give our frontend candidates. If you're not into Go and prefer JS, you might like this challenge better.
]]>To illustrate this pretty dry topic, I told the story of Alice and Bob, two entrepreneurs building a succesful web application and facing some serious challenges.
I gave a talk at Go SummerFest and then later on another one during I/O. For more information about the events, see this blog post from the Go team.
My talk was recorded but it is not yet online.
]]>Very much like with many other programming languages, a challenging part of Go is to learn how to write idiomatic code. The good news is that Go makes refactoring easy (and already has a lot of conventions). I strongly recommend this post from Peter Bourgon about Go at SoundCloud and the extra conventions they follow (Splice also follows the same conventions).
One of my favorite Go projects is the gobot project by HybridGroup.
The Gobot project is pretty young and I noticed a few things that could be improved so I offered my help to Ron, Adrian and the rest of the team. Our discussion quickly turned into a fun group refactoring session (featuring @kytrinyx, @deadprogram, @codegangsta, @jnbeck, @adzankich )
Gobot is split into multiple packages, a core and a few other packages. The gobot team, out of habit chose to put a package per repo. After further discussions, we chose to bring all official packages inside the same repo to keep things easier and to keep the import paths clean and logical.
So instead of having:
1 2 3 |
All the none-core packages are moved to subdirectories:
1 2 3 |
This also allowed us to fix the package names
is now simply sphero
Which also allowed us to simplify the following code:
1 2 3 4 |
1 2 3 4 |
We did that with a few other types and methods all over the packages.
We had a discussion about what lead to the multiple repos vs one repo. There are legitimate cases for both approaches but in this situation, the decision was based on a misunderstanding. The author thought that by importing the top package, all sub packages would also be somewhat included in the build, making the binary bigger than needed. Since Go only compiles and links packages imported, moving all packages within the same repo wouldn't change the binary output. Note that this is not because in this specific case we have all packages in the same repo that this is the right thing to do every single time.
By conventions, package should contain a doc.go
file that contains
an overview of the package and often some information so the developer
trying to use the library can find the right entry points.
As usual, the standard libraries are a good example,
here is the net/http doc.go
We spent some time refactoring master.go
which is the file implementing
the code handling one or multiple robots (which can each have multiple devices).
The original function code looked like this:
1 2 3 4 5 |
There are a few things that aren't really idiomatic in this code.
The first thing is that by convention, constructors are usually called New<Type>
Secondly, the community seems to follow the following stylistic choice:
only use new
and make
when you need to set the capacity (make([]string,3)
Finally we don't need to allocate a variable. Here is the refactored code:
1 2 3 |
In the original code, we had a variable called trap
which was
a function living at the top level of the package:
1 2 3 |
The func was then used to handle signals. The author
chose to use a variable so he could mutate it in the test suite and
avoid sending an interrupt when testing.
We realized we could avoid having this function variable at the top of the package by moving
it as a field on the Master
type and setting the default func in the constructor.
1 2 3 4 5 6 7 8 |
The code still behaves the same and we can still overwrite the trap function in our tests (since the tests are part of the same packge, the non exported field is available) but we got rid of a top level var.
The following code was ranging over a predefined channel (c
) of signals.
and when a signal would arrive, all robots belonging to the master
would be halted and disconnected.
1 2 3 4 5 6 7 |
The code above works well but could be cleaned up a little:
1 2 3 4 5 6 |
This code does the same thing but simpler. We are trying to read from the channel which will block (we don't care about the result so we don't capture or could have used an underscore). Then we loop through each robot and stop them. We managed to remove a for loop on the channel (with an odd break) and made the code intent clearer.
Next, we tackled the following method:
1 2 3 4 5 6 7 |
The funny thing about this method is that it's not needed. We could get the same result by calling:
When I said that, someone suggested that it might be a bad idea
since FindRobot()
might return nil
and now we would be calling
on nil
and bad things would happen.
Looking at the code, it was actually easy to fix.
Here is the original code:
1 2 3 4 5 6 7 8 |
Here is the refactored version:
1 2 3 4 5 6 7 8 9 10 11 |
Did you spot the difference? We just added a check to see if the pointer (r
was nil, if it is, we just return nil
When I added the code above, the person who was worried
about calling GetDevice()
on nil
was scratching his head.
Golang does something very interesting (and a bit surprising if you come from a dynamic language), it returns a nil pointer of the type we defined as return type. Let's walk through the code by rewriting it slightly differently:
1 2 |
At this point if FindRobot()
didn't find a robot, bot
is still
of type *Robot
but the pointer is nil.
Because we defined a method GetDevice()
on *Robot
, we
can call:
The GetDevice()
method will execute and will return nil
away because we check if the pointer is nil
The fact that nil pointers have types has 2 important implications,
the first one is that you can nicely chain methods without
checking at the caller site if the returned value is nil
The second is that your methods should expect to be potentially
called on a nil pointer and should properly handle such cases.
Note: Go team member Andrew Gerrand
suggested on Hacker News
to name the method Device
instead of GetDevice
. The word Get
is almost always redundant.
In the same chain of thoughts, maybe we should rename FindRobot
just Robot
I'm writing this post on my way back from GopherCon and there was one more thing I wanted to clean up and share with you. This is a nice pattern I use often to simplify my code.
Our Robot
type has a connections
field and a devices
1 2 3 4 |
To avoid always having to manually loop through the slice, a method is defined on
pointers to Robot
. This method iterates over
the devices and halts them:
1 2 3 4 5 |
This code is totally fine but from an API design perspective, wouldn't it be nicer to use?:
One of the nice things with this approach is that the concept of halting, which
really belongs to the devices, doesn't need to leak into the Robot
To implement the suggested API change, we need to define a type alias:
We can now define methods on our new type:
1 2 3 4 5 |
We then need to update our Robot
1 2 3 4 |
And we are done with our refactoring.
One last note, since we might need to call different methods on our collection we could create an iterator method.
1 2 3 4 5 6 7 8 9 10 |
Needless to say that we had fun. The refactoring went much further and we removed the use of reflections, some sleeps and much more. The code is going through a nice cleanup before reaching 1.0 and I can only encourage everybody to play with Gobot, there are very few things as fun as Go and Robots! (The code is open sourced, look at it, add new drivers, send PRs!)
I'd like to thank Ron Evans and the Hybrid Group for open sourcing their code and sharing the fun with all of us. I can't wait for the next LA Go + Robot hack night.
Finally, Splice is hiring, our stack uses a lot of different technologies but our backend is all in Go and we are always looking for talented engineers. Drop me a line if interested.
]]>A big thanks to all the people who recommended these books, I'm looking forward to 2014!
If you want more, you can check the previous book recommendation list
]]>Since I own both apps, I thought it would be as simple as sharing the secret session key and re-implementing Rails crypto process in Go. It turned out to be a lot more interesting.
In a nutshell, here is what I discovered:
Because Rails serializes and deserializes the session and any encrypted/signed cookies using Ruby's Marshal library, someone with the app secret can wreak havoc. They can embed arbitrary Ruby code into the cookie, submit it with a request, and the server-side deserialization will execute that code without you noticing. Granted, this requires the attacker to have the app secret, but since 99% of the apps out there have the shared secret in their source code, anyone with access to the source code has this data. It's not data you can easily rotate when employees leave or when you are done working with contractors. Anybody with the shared secret is a potential attacker. Start by moving this data out of the code base and into an environment variable.
Rails doesn't let you change the default serializer directly. But Rails relies on ActiveSupport for its crypto work and AS supports swapping the serializer. Some people in the community are aware of this issue and monkey patch Rails to serialize their sessions using JSON or another alternative. Here is an Airbnb article and Rails 3 patch. Here is my Rails 4 monkey patch to switch the serialization to JSON. I'm using it in production with Rails 4, but it's untested on Rails 3.
You can modify either solution to use MessagePack instead of JSON if you want to fit more data in the 4K cookie size.
Once I addressed the serialization issue, I had to reimplement the crypto work done by Rails to encode and/or sign the data.
Most of us just rely on our frameworks/libraries to do the right thing, but we rarely look under the hood. I ported the logic to Golang which has an amazing support for crypto (albeit lower level than Ruby). My Go package contains an explanation of the code logic and the examples needed to decode/verify as well as encode/sign sessions that are compatible with Rails.
Here is a high level summary of what Rails does when it encodes and signs your session data:
1 2 3 4 5 6 |
The session_content
string is then set as the session cookie value.
Note that you could do that in any Ruby app using ActiveSupport
, making it easy to share sessions between Ruby applications (like Rails & Sinatra).
Technically, there are a lot of things going on. To avoid using the same secret to sign and encode data, Rails relies on derived keys using PBKDF2 (password based key derivation function). It treats the app secret as a password and applies a pseudorandom function 1000 times (Rails default) using a default salt. The result is a derived key so the original password isn't shared. The derived key can be regenerated identically if the salt and secret are known (because the function is pseudorandom).
The two derived keys are then passed to the MessageEncryptor
class which uses MessageVerifier
to do the signing. The generated keys are 64 bytes long. One key goes to the encryptor while the other goes to the verifier.
The verification is done via HMAC (SHA1) and it uses the full 64 byte key. The encoding is done via AES 256 CBC only using the first 32 bytes of the encryption derived key. Rails will only generate a 32-byte key since that's the expected key length.
The session data is serialized (using Marshal by default) then encoded via AES. Both the encoded string and the IV are encoded using base64 and joined in a string using a predefined format.
At this point, the session is encoded but it could be tampered with. To avoid that, Rails signs the encoded data using the verifier (HMAC) and appends the base 64 encoded signature to the encoded data.
To decode and verify the data, Rails repeats the process in reverse using the serializer to deserialize the data.
Note that you can also rely on the the same crypto process to safely encode/sign any data you want to share. If you're ok with the data being user-readable, sign it to make sure it isn't tampered with along the way. If you don't want it to be user-readable, encrypt it first then sign the encrypted data.
Many apps are moving to an SOA approach. That often means multiple languages living together in production. Sharing a web session can be very useful, especially until you switch to a SSO solution.
The key is to start by having the session data serialized in a format that is available in all your relevent languages. JSON, XML MessagePack, and protobuf are good examples.
The second step is to reimplement the crypto dance I just explained above. The good news is that I've already done it for Go. Using that example, you should be able to port it to other languages (Node, Scala/Clojure/Java, Rust, Elixir, Python or whatever you fancy).
Even though the test suite isn't perfect (yet), it should greatly help you through the porting process. To be honest the hardest part was understanding the process, not writing the code. Most languages have decent crypto libraries to do the hard parts for you. But for Go I had to implement lower level pieces like the PKCS7 padding for the AES CBC encryption/decryption.
Hopefully this article was helpful and you now better understand how Rails does its session encryption. Once you understand the process Rails uses, you can implement it in any language.
Finally, if you interested in working on interesting and challenging problems like these ones, consider joining the Splice team!
]]>The format was a bit different than usual, I had a 40 minute slot and divided in 2, I made my points for 20 minutes and invited two special guests on stage to discuss the topic. The reason for this format is because I think we all learn better by looking at things from different perspectives. I can't thank enough Sandi Metz and Katrina Owen for their contributions.
Here are a few points that I think are interesting and that we discussed during the talk.
Bad code doesn't exist, you have code that can be interpreted/compiled and code that doesn't. "Good" and "bad" are moralistic designations, not scientific ones. Let's try to stop using these terms to refer to talk. Let's be more precise when arguing about code, "what do you mean by bad?" Is it hard to maintain, hard to understand, slow etc.. ? Always refer to the context in which the code was written. Don't use it as an excuse to defend anybody's potentially hurt ego, but instead to explain why the code was written caring about certain values instead of others. Most code turns emo after a little while, being able to understand the context, helps a lot the devision process when facing such code.
As developers, we aren't paid to write code, we are paid to build products, to convert ideas into something "concrete". When I use my favorite app, I don't care that the code is beautiful, I care that it works, that it's stable and provides me with what I need. If the code is written in Go, Pascal, Erlang, VB or Ruby doesn't matter at all. That the code has full test coverage and was written in an agile manner using weekly scrums and TDD is probably as important as knowing the Pantone color of a company's logo: it only matters to the people deeply involved.
Don't focus on how to build, focus on why you build things. Then the how will come as you learn from others, experiment and discover "how" to build the "why". The "why" is often the constant, the "how" keeps on changing as we collect more information.
When you learn, rules are easier. Katrina made a very good point during our discussion. When she teaches, she needs to have very strong rules that she can justify. That said, her rules can change, and as you become better, you can start challenging the rules. Rules are somewhere between training wheels and a guard-rail. They are very useful but shouldn't be used to attack other people.
Ruby has a language doesn't enforce many rules, as a matter of fact, Matz designed a language so you can set your own rules, or barely have any rules. This fact probably explains why so many people are after solid rules they can rely on. Rules are easy to follow and are a good reference. The Ruby language doesn't have enough rules to some, so the community is helping by coming together to define them. This is by the way, the biggest difference with Python and Go where these 2 languages explicitly want to only have 1 way of doing one thing.
The key to building a good product (which is what we are paid to do) is communication. Unfortunately, us developers, are on average, pretty terrible at that. We need to do a better job communicating with the rest of the organization (i.e: anybody who's not an engineer). It usually starts by the designers. The expected outcome of the product should be clear, well understood by all and easily evaluable. Within the engineering team, coding values should be set explicitly. Not everyone will agree but when we write code, we should all care about the same overall values so our work is consistent. When something happens and we have a disagreement, it's easier to refer to our values to decide what solution to pick.
I'm starting this new project, what technology do you think I should use?
Often these people fit in one of two categories:
At the end of the day, I doubt that many of these people actually cared about my answers. They probably just wanted to know if we were on the same page or wanted to be reassured.
To be honest, as an engineer, I believe a great idea can be built with almost any technology. They all have their pros/cons. No matter what stack you choose, you'll pay a certain price for whatever advantages it offers. But really, the success or failure of your project has more to do with vision, leadership, execution, and market than technological choices.
Now that I'm an entrepreneur, I make technical decisions daily. When I choose a specific technology, I need to be able to justify the decision to myself, my partners/employees and potential investors. I make technical choices based on the project and company's vision.
For a project to be successful you must have a strong vision. If you can convert your vision into a set of values to benchmark every decision, your path will be clear and it'll be easier to find the right people to join you.
Besides the vision, a lot of startups focus on culture. It is commonly said that the culture is defined by the founders, the first few employees, and the product itself. However, what isn't often mentioned is that the technical decisions will have a direct effect on the company culture.
Whether your new startup is based on J2EE/Oracle, Perl, PHP, Rails, Node.js or .NET, the team's engineers will have different expectations, different values, and different concerns. None of these technologies are intrinsically bad. Great things have been built with each. But they do come with a culture.
A couple years ago, I met an entrepreneur who chose to build his application in Node.js. Curious, I asked why he chose Node. The response was simple: * smart engineers are excited about it so I can more easily recruit * people are willing to contribute for free because it builds their experience
This decision clearly set the engineering culture and defined the team of people who could work or be interested in working on the project.
So maybe instead of asking what technology I should use, we should ask ourselves:
Does this technology fit my company's core values?
That's a much harder question because you need to actually understand your core values. That understanding is key to building a successful product.
You can't blindly copy a tech stack in the same way you can't copy a business plan. It's a part of your company's identity. Your core values, your objectives, your team and your expectations are different.
The whole "it worked for X" argument is rarely valid. Look, Facebook uses PHP, it "worked for them". Does that mean we should all use PHP?
Characterizing communities is difficult, but I'll share with you the impressions and perspectives that I have on various options. Feel free to use the comments to share your own perspective and cover other communities.
Here are some of the "classics": languages that have been used for a while and have proven their values. They're widespread, but don't inspire much passion anymore.
Note: I omitted Perl because I personally don't know any new startups building their core technology in Perl (6?).
Common use case: (as of mid-2013)
Personal opinion:
PHP had its days of glory. It really made web development easy and accessible. However, probably due to the really large amount of new programmers who started with PHP and a not so opinionated community, very few people can write good PHP.
Good idiomatic code examples are hard to find and I'm not even sure there is such as as idiomatic PHP. The result is a community known for poor code quality, lack of tests, security nightmares and an after taste of the early 2000s.
Strong PHP teams with well established conventions, processes and guidelines can accomplish great things, but such teams are rare.
Personal Opinion:
Java is quite interesting. A few years ago a lot of developers got tired of Java and explored other lands. They often switched to interpreted languages such as PHP, Python, Ruby or more esoteric languages like Erlang.
However, Google via Android was able to show that Java in itself isn't as terrible as we remembered (as long as you don't have to use J2EE or Swing). There is also a "hipsterish" trend that seems to indicate that Java is actually cool again. A lot of that has to do with two things:
That said, for a lot of us, writing Java all day long doesn't sound appealing. If you are going to rely on the Java stack, there is long list of other JVM languages which are mature and play well with Java libs (i.e: Scala, Groovy, JRuby, Clojure). You can always to mix and match.
Hiring Java developers isn't too hard since most students coming out of school learned Java, but finding great early-stage startup engineers who want to write Java is quite challenging.
Side note: If you are targeting Android, keep it simple, stay with the official stack even if you might fancy another JVM language better.
There are still many reasons to use Java's technology for your new startup, but you might also consider using a more "rapid/flexible" solution in parallel (Ruby, Python, Node…). A multilingual environment brings a lot of value to both the company and the engineers, which is something the Java community seems to be slowly but surely discovering.
Java mainly attracts more classically trained engineers looking for comfortable, repetitive, well known patterns. They will be used to the language, its tools and its natural rhythm. They might not be the most curious developers but they are reliable (if you pick the right ones obviously).
Personal Opinion:
I went back and looked at C# when C# 5 was released and I have to say that I was really impressed by some of the new language features. From a purely language design perspective, C# is quite a bit ahead of Java. I was also surprised by how pleasant it was to write Javascript in Visual Studio (I really didn't expect that since my experience with VS was mainly around C++).
Another thing that really impressed me: the quality level of the available documentation is outstanding! But the fact that C# isn't open source, that Visual Studio + MSDN is so expensive and the whole environment reeks of licenses and costs, is bit of a turn off.
Microsoft is slowly opening up to open source and more open solutions like Azure. But as a community, .NET is still quite Microsoft-centered. As a startup entrepreneur, you should consider how you feel about open source vs enterprise backed cultures.
C# mainly attracts a variant of the Java crowd: engineers seeking stability and a support contract over open source. And they can tolerate IIS!
Over the years, two dynamic languages became cherished by startups: Python and Ruby. The two languages are actually quite similar. Nowadays Python is quite popular for backend apps (NLP, biotech, APIs, SOA elements) while Ruby is more popular for consumer-facing apps. Both of these languages suffer from the same limitations (mainly performance and concurrency) but their core values and communities have different focuses.
Personal Opinion:
As someone who chose Ruby over Python, I often envy the quality of the documentation you find in Python projects. I also have a love/hate relationship with the fact that Python is designed to give you just one right way. This is often great for teams, but it can also be frustrating.
In some areas, Python has some of the best libraries out there, and depending on the problems you are tackling, Python might be the right choice. Python developers know how to communicate about their code. They document what they do and are process oriented while being pragmatic about their approaches.
But Python was created way before the internet became popular and if concurrency and high throughput is a concern for you, a dynamic, interpreted language with poor concurrency might not be the right choice.
Python mainly attracts more pragmatic, experienced, full-stack developers wanting a modern but well-proven language.
Personal opinion:
As far as I'm concerned, Ruby has been my go-to language for years. You will find an incredible, sometimes overwhelming amount of Ruby open source code. Rails is really an amazing web framework making most web projects easy to implement if you know how to use the tool.
But the flexibility and rapid development cycle also have downsides. Be ready to invest a large chunk of your time keeping your code base up to date and migrating away from abandoned libraries. If you can't rely on caching, the throughput of a successful app will often be limited by the lack of good concurrency support.
Ruby developers are mainly Rails developers and a great majority might have a hard time being able to identify core language features versus framework features. They are often curious, opportunistic (in a good way), somewhat pragmatic and care about code quality/structure and test coverage. Rails developers are typically early adopters due to the fact that the framework itself uses some new technologies by default (coffeescript, turbolinks, CSS pre-processors…).
Ruby and Rails mainly attract developers wanting to get things done quickly but elegantly. These developers are often product-oriented and care more about the purpose and customer value than the lower-level computational details.
These are the languages/technologies that get people excited. They represent the new wave of programming languages designed to run in "the cloud".
Node.js isn't a programming language but it's the most popular way to run JS server side. The same way most of my comments about Ruby were about Rails, I'll focus on Node more than JS itself.
Personal Opinion:
I find Node.js interesting. Technically there isn't much new with Node. Python has Tornado/Twisted, Ruby has EventMachine, and C had libevent.
Event-driven frameworks have been used for a while but Node has two major advantages: * most JS libs are non-blocking * most web developers have to write some JS anyway
The idea of using the same programming language both in the front end and the back end appeals to many, but the value is still unproven.
Node offers great throughput (as long as you stick to IO operations), is easy to get started, and is fun to write.
Due to the nature of event-based programming, debugging and testing is challenging. Dealing with callbacks can be maintenance hell. I hope that Node will adopt an official future/promise solution. And documentation is typically spotty making jumping on an existing project difficult.
Node developers are definitely early adopters and comfortable creating a custom structure/pattern rather than following convention.It attracts developers wanting to use a known language (JS) to handle high levels of concurrency. Node as a framework is lower level than the classical MVCs which is a plus for hackers. Node developers also really like the idea of using the same programming language on both server and client.
Personal Opinion:
What I like the most about Clojure is the lisp spirit. Once you get past the parenthesis and the operator/argument order, Clojure challenges you to entirely rethink the way you architect your code. It's really good and efficient at processing data and pushes you to keep your code short.
My problem with Clojure is that I'm not smart enough to write a lot of it. My brain quickly stack overflows trying to follow the data. Exceptions are often meaningless and trying to debug someone else's code is challenging since the language itself is complex and it can be extended by macros. Finally, the Clojure community isn't really web-oriented, most of the work done in Clojure seems to be data-centric.
Clojure mainly attracts more fringe, language-curious, data-oriented programmers. If you are looking for data scientists with a programming language fetish, Clojure is a good way to attract them.
Personal Opinion:
Scala is currently my language of choice when targeting the JVM. The learning curve is steep. Knowing when to use FP vs OOP can be tricky and so is dealing with the language syntax itself.
That said, getting the benefits of using FP, while still keeping OOP when needed, is very useful. Once you "get" the language idioms, writing Scala is actually pleasant and the community is quite nice.
The Play framework is really good and offers a good alternative to Rails, especially for API development. Twitter's engineering team offers a lot of resources and open source code.
Using Scala is a pretty safe bet at this point. Java developers feel confortable and get to try a more "modern" language. Dynamic language developers don't feel too far from home but get the Java ecosystem, the performance boost, concurrency and immutability. The tooling and convetions make using Scala on a growing team quite nice, if the compilation time doesn't get you down.
Like Ruby, though, the Scala community isn't big on documentation. I really hope the API doc will be rewritten to be more intuitive and overall more useful. But to be fair there are a lot of great resources out there such as Twitter's Scala school and Coursera's FP in Scala class given by Martin Odersky (Scala's creator).
Scala mainly attracts curious Java developers wanting something more modern as well as Ruby/Python developers wanting a more scalable version of their language. Scala is a good way to attract great developers who want to push the boundaries of their existing dev environment as well as developers being able to leverage the duality of the language.
Personal Opinion:
I really like Go (aka Golang). After playing with it for years, I chose to use it to develop the APIs of my own startup. Go might sound boring to some, but its simplicity and efficiency just work.
Go forces you to think a bit more about how you structure your data/behavior because you can't just stick to the usual OO patterns. I've found that my code ends up being easier to follow and simpler in structure, yet sometimes a bit more repetitive (ex: error handling).
Concurrency can't get much easier than Go. While it is compiled, your code compiles and boots in less time than a Rails server starts up. Go supports some form of duck typing making the transition from Ruby (for instance) quite easy. The production performance is quite amazing when coming from a scripting language and the memory footprint stays small.
Go is designed so a single user or a big team can work on the same codebase and the tooling around the language is really great.
However, it's not a perfect language. 3rd party dependency management can be tricky at times. The code can feel too low-level when you're used to high-level programming languages. And some of the language design decisions can cause confusion at times (ex: interacting with interfaces vs structs).
Go seems to become quite popular within the startup scene when performance and concurrency matters. I've seen a good number of startups migrating from Node to Go and others simply extending their stack by adding small Go apps.
The Go community seems to be a mix of old school hackers coming from C/C++ and a younger crowd enjoying a lower-level language. The language and the community leaders are opinionated which makes understanding their vision and approach easy. It also allows you to quickly evaluate how comfortable you are with their philosophy and see if it matches your expectations.
Go mainly attracts performance/architecture oriented developers. They want easy concurrency, the execution speed of C with the development speed of Python/Ruby. They don't look for a new fun language, they look for a solid compromise.
Technical decisions have cultural impact. Think clearly and carefully about how your technologies align with your company's core values. Make the right choices and you'll spend less time fighting about technical details and more time building a great business.
And if you miss those arguments, there's always hackernews.
Update: Speaking of HN, here is the thread for this post
]]>I'm not yet super familiar with all the std lib packages, so when I wanted to test my Go web services, I wrote a few lines of code to create a multipart file upload function that was building the body from scratch. Once I was done messing with the various headers, boundary protocol etc.. I started testing some edge cases, I found some bugs in my code. Looking at Go's packages, I realized that all the tools were already available for me to use. I was just lacking a good example. Walking through the test suite I finally figured out how to write a simple multipart file upload example with some extra query params.
Hopefully this example will be helpful to some of you.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 |
Example's source code on GitHub
All the work is done in the newfileUploadRequest
function and
really, the mime/multipart
package hides all the complexity of
creating a multipart request.
The key is to set a new multipart.Writer
The writer will do all the work and will write directly to our body (which itself is a buffer of bytes).
We then create a part for the file form entry with the name of the file
param and the name of the file (that we extracted using the path/filepath
We need to add the content of the file to the file part, we use the
to do so. In the first version of this article, I had used
to read the content of the file (see code here).
However a few readers rightfully mentioned that I should instead copy
content from the file to the part instead of temporarily loading the content of
the file in memory. Here is an
even more optimized version using goroutine to stream the data, and
here is the full example using a pipe.
1 2 |
The multipart.Writer
takes care of setting the boundary and formating
the form data for us, nice isn't it?!
Then for any extra params passed as a map of string keys to string
value, we use another function of the multipart.Writer
Once again, the writer takes care of creating the right headers, and to add the passed value.
At this point, we just need to close our writer and use our body to create a new request.
1 2 |
One last thing before triggering our request, we need to set the header that contains the content type including the boundary being used. Once again, the Go lib has us covered:
As a reference, here is the generated body:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
Golang might not be as high level as Ruby or Python, but it's not too far off and it certainly comes with some great std libs. I know I recently caught myself writing a lot of small scripts in Go, something I used to do in Ruby. I think this is mainly due to the fact that Go is compiled, designed for concurrency, has great std libs and is quite easy to write.
Hopefully this code sample illustrates how easy Go can be and can also serve as a reference point if you are looking for a way to do multipart upload.
]]>Before optimizing, we need to measure.
Without a solid baseline, how can you say that the time you invested in making things better wasn't a total waste?
True refactoring is done with a solid test suite in place. Developers know that their code behavior didn't change while they cleaned things up. Performance optimization is the same thing: we need a good set of metrics before changing anything.
There are plenty of monitoring tools out there, each with its own pros and cons. The point of this article isn't to argue about which one you should use, but instead to give you the some practical knowledge about Graphite.
Graphite is used to store and render time-series data. In other words, you collect metrics and Graphite allows you to create pretty graphs easily.
During my time at LivingSocial, I relied on Graphite to understand trends, issues and optimize performance. As my coworkers and I were discussing my recently announced departure, I asked them how I could help them during the transition period. Someone mentioned creating a Graphite cheatsheet. The cheatsheet turned into something much bigger than I expected and LivingSocial was nice enough to let me publicly publish this short guide.
For a more in depth dive into the statsd/graphite features, look at this blog post
There are many ways to feed Graphite, I personally used Etsy's statsd (node.js daemon) which was being fed via the statsd RubyGem. The gem allows developers to push recorded metrics to a statsd server via UDP. Using UDP instead of TCP makes the metrics collection operation non blocking which means that while you might theoretically lose a few samples, your instrumented code performance shouldn't be affected. (Read Etsy's blog post to know more about why they chose UDP).
Tip : Doing DNS resolution on each call can be a bit expensive (a few ms), target your statsd server using its ip or use Ruby's resolv standard library to only do the lookup once at initialization.
Note: I'm skipping the config settings about storage retention, resolution etc.. see the manual for more info.
Always namespace your collected data, even if you only have one app for now. If your app does two things at the same time like serving HTML and providing an API, you might want to create two clients which you would namespace differently.
Properly naming your metrics is critical to avoid conflicts, confusing data and potentially wrong interpretation later on. I like to organize metrics using the following schema:
1 2 3 |
I use nouns to define the target and past tense verbs to define the action. This becomes a useful convention when you need to nest metrics. In the above example, let's say I want to monitor the reasons for the failed password authentications. Here is how I would organize the extra stats:
1 2 3 |
As you can see, I used failure
instead of failed
in the stat name.
The main reason is to avoid conflicting data. failed
is an action and
already has a data series allocated, if I were to add nested data using
, the data would be collected but the result would be confusing.
The other reason is because when we will graph the data, we will often
want to use a wildcard *
to collect all nested data in a series.
Graphite wild card usage example on counters:
This should give us the same value as accounts.authentication.password.failed
so really, we should just collect the more detailed version and get rid
of accounts.authentication.password.failed
Following this naming convention should really help your data stay clean and easy to manage.
StatsD lets you record different types of metrics as illustrated here.
This article will focus on the 2 main types:
Use counters for metrics when you don't care about how long the code your are instrumenting takes to run. Usually counters are used for data that have more of a direct business value. Examples include sales, authentication, signups, etc.
Timers are more powerful because they can be used to analyze the time spent in a piece of code but also be used as a counters. Most of my work involves timers because I want to detect system anomalies including performance changes and trends in the way code is being used.
I usually use timers in a nested manner, starting when a request comes into the system, through each of the various datastores, and ending with the response.
It's a well known fact that the response time of your application will both affect the user's emotional experience and their likelihood of completing a transactin. However understanding where time is being spent within a request is hard, especially when the problems aren't obvious. Tools like NewRelic will often get you a good overview of how your system behave but they also lack the granularity you might need. For instance NewRelic aggregates and averageses the data client side before sending it to their servers. While this is fine in a lot of cases, if you care about more than averages and want more detailed metrics, you probably need to run your own solution such as statsd + graphite.
I build most of my web-based APIs on wd_sinatra which
has a pre_dispatch_hook
method which method is executed before a
request is dispatched.
I use this hook to both set the "Stats context" in the current thread and extract the client name based on HTTP headers. If you don't use WD, I'll show how to do the same thing in a Rack middleware.
1 2 3 4 5 |
Then using Sinatra's global before/after filters, we set a unique request id and start a timer that we stop and report in the after filter. If we were using Rails we'd get the unique identifier generated automatically.
Before filter:
1 2 3 4 5 6 7 |
After filter:
1 2 3 4 |
Note that this could, and probably should, be done in a Rack middleware like this (untested, YMMV):
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
Note that the stats are organized slightly differently and will read like that:
The dots in the stats name will be used to create subfolders in graphite.
By using such a segmented stats name, we will be able to use *
wildcards to analyze how an old version of an API compares against a
newer one, which clients still talk to the old APIs, compare response
times, etc.
We're collecting stats on every request so we can see request counts and median average response times. But wouldn't be better if we could measure the time spent in specific parts of our code base and compare that to the overall time spent in the request?
We could, for instance, compare the time spent in the DB vs Redis vs Memcached vs the framework. And what's nice is that we could do that per API endpoint and per API client. In a simpler case, you might decide to monitor mobile vs desktop. The principle is the same.
Let's hook into ActiveRecord's query generation to track the time spent in AR within each request:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 |
This code might not be pretty but it works (or should work).
We subscribe to ActiveSupport::Notifications
for sql.active_record
and we extract the info we need. Then we use the stats context set in
the thread and report the stats by appending
The final stats entry could look like this:
As you can see, we are getting granular data. Depending on how you setup statsd/graphite, you could have access to the following timer data for each stat (and more):
Instrumenting Redis is easy too:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
Using Ruby's alias method chain, we inject our instrumentation into the Redis client so we can track the time spent there.
Applying the same approach, we can instrument the Ruby memcached gem:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
We now have collected and organized our stats. Let's talk about how to use Graphite to display all this data in a valuable way.
When looking at timer data series, the first thing we want to do is create an overall represention. Your first inclination is probably an average.
The problem with the mean is that it's the sum of all data points divided by the number of data points. It can thus be significantly affected by a small number of outliers.
The median value is the number found in the center of the sorted list of collected data points. The problem in this case is that based on your data set, the median value might not well represent the real overall experience.
Neither median nor mean can summarize the whole story of your system's behavior. Instead I prefer to use a 5-95 span (thanks Steve Akers for showing me this metric and most of what I know about Graphite). A 5-95 span means that we cut off the extreme outliers above 95% and below 5%.
Here is a comparison showing how the graphs can be different for the same data based on what metric you use:
Of course the span graph looks much worse than the other two, but it's also more representative of the real user experience and thus more valuable. Here is how you would write the graphite function to get this data.
Given that we are tracking the following data-series:
The function would be:
1 2 |
If you try that function, the graph legend will show the entire function, which really doesn't look great. To simplify things, you can use an alias like I did in the graph above:
1 2 3 |
Aliases are very useful, especially when you share your dashboards with others.
Another neat feature you might add to your graph is a threshold. A threshold is a visual representation of expectations. Say, for example, that your web service shouldn't be slower than 60ms server side. Let's add a threshold for that:
and here's how it would look in a graph:
Another useful trick is to change the render options of a
graph to draw null values as zero.
Open the graph panel, click on Render Options
, then Line Mode
and check
the Draw Null as Zero
Here is a graph tracking a webservice that isn't getting a lot of traffic:
You can see that the line is discontinued, that's because the API
doesn't constantly receive traffic. If your data series gets only very
few entries, you might not even see a line. This is why you want to
enable the Draw Null as Zero
By default graphite shows data at a 10 second interval. But often you want to see less granular data, like the quantity of requests per second.
Let's say we didn't use a counter for the amount of requests, but because we used the middleware I described earlier, we are timing all responses. Graphite keeps a count of the timers we used, so we can use this count value with a wildcard:
If we were to render a graph for this stat we would see a graph per
client. Right now we only care about showing the total amount of requests.
To do that, we'll use the sumSeries
The graph looks pretty but it's hard to understand what kind of request volume we are getting. We can summarize this data to show 1 min summaries instead:
We can now see the quantity of requests per minute. You could do the same to resolve by hour, day, etc.
Graphite has the ability to compare a given metric across two different time spans. For instance, let's compare today's quantity of logins vs those from last weeks.
To generate today's graph:
Then we use the timeShift
function to get last week's data:
Graphing both series in the same graph will give us that:
Wow, it looks like last week we had an authentication peek for a few hours. Why? It would be interesting to graph our promos and sales in the same graph to see if we can find any correlations.
Depending on your domain, you might want to compare against different
time slices. Just change the second timeShift
Another technique is to compare the percentage growth since last week. Let's imagine we are looking at sales or signup numbers. We could graph today's sales per minute vs those from last week.
To do that, Graphite has the asPercent
function. This function
takes a series representing 100% and second to compare against.
The function call looks a bit scary so let me try to break it down over
multiple lines:
1 2 3 4 |
The first argument is the summarized RPMs (requests per minute) and the second is last week's summarized RPMs.
Here is how the graph looks:
Based on all the data we collect, we can now graph something like that:
This graph is basically the same as the one above, but we used the overall response time as the 100% value and we graphed all the different monitored sections of our code base.
You can now build some really advanced tools that look at trends, check pre- and post-deployment measurements, trigger alerts, and help you refactor your code.
Maybe you suspect that your app has a chokepoint at the database level. You can track the query types and the targeted tables per API endpoint. You can see where you spend most of the time and which code path is responsible for it. You can quickly see if adding indicies or other database-level techniques actually make a difference.
Campfire and many other chat tools offer image preview as long as they detect that the url has an image extension. Unfortunately, Graphite's graph urls look more like this:
To get a preview, just append the with: &.jpg
You might want to do something fancy with the data like
create alerts. For that you can ask Graphite for a json representation
of the data by adding &format=json
to the URL.
1 2 3 4 5 6 7 8 |
The data points are the timestamped value of each graphed point. Note that you can also ask for the CSV version of the data then pass it on to some poor bastard using Excel.
Let say that you are graphing the response time of all your APIs. The amount of displayed graphs can be overwhelming.
To limit the displayed graphs, use one of the filters. For instance the currentAbove
filters that can help you only display web services with
more than X RPMs for instance. Using filters can be very useful to find
Hopefully this guide will help and inspire you to start using Graphite to easily collect and analyze your metrics. I'm sure there are great tricks I forgot to mention, please add your favorites in the comments.
Thanks to Jeff Casimir for reviewing this post before its publication!
]]>I therefore wrote the same web API in Ruby (using Rails) and in Go (at first using Revel and then rewriting it without a framework since Go's std lib have everything one might need). The API spec was simple: * extract an authorization token contained in the request header * use the token to query a MySQL database * respond by sending back the MySQL row in json format * return 401 if the token isn't value
I didn't try to optimize the Ruby code, nor the Go code. The idea wasn't to get precise benchmark results, the goal was to get an idea of how much faster Go was in a real life situation. The other goal was to evaluate the amount of work needed to write web APIs in Go for someone who already knows the language.
At the end of the day the API implemented in Go is more than 50x faster than the Ruby version. Interesting enough, writing the code and tests for the Go API was pretty close to the Ruby experience (more on that later). 50X performance gain, including high concurrency support might be a very good argument to start using some Go when it makes sense.
I documented my experiment on Google+, click the following screenshot to read more.
essentially allows you to hook into Ruby's events and
listen for events.
Being curious and since I just started a brand new Rails 4/Ruby 2 app, I decided to write a little middleware and see what Rails is up to when handling incoming requests.
Here is my TracePoint Rack Middleware.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
(the gist shows a modified version so I could dump to disk the json representation of the calls)
I then inserted the middleware in Rails:
1 2 |
I saved the output in json format for the curious: click here
On average, in production mode, using Ruby 2.0 and Puma on my laptop, my hello world index page takes 5ms.
To render my page, Rails uses (more or less):
Here is a small selection of some of the methods dispatched:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
is a great new addition and I hope to see some new crazy
tools being developed (production dead-code analyzer, deprecation code path
finder anyone?)
To end, this short post, here is an interesting quote from Chad Fowler Berliner by adoption:
]]>Abstractions are expensive. The cost increases exponentially as you add them to a codebase. Chad Fowler
Turns out it wasn't that hard, but the lack of documentation made me lost a couple hours. I therefore updated OmniAuth's wiki and wrote this quick post so hopefully you won't waste time looking for simple details.
You actually only need to add 2 gems to your Gemfile:
1 2 |
Now, the second gem is the one I didn't know about. The gem is actually provided by Google itself. It turns out, Google Apps use a custom discovery protocol. They monkey patched the popular OpenID Ruby libraries so you can just drop in their gem and their discovery system will magically work.
You need to require 4 files to get everything setup properly:
1 2 3 4 |
The first one is the omniauth extension for OpenID, the second one is the main Ruby OpenID library (needed so we can set our SSL cert). The third one allows us to store temporary data on disk instead of keeping it in memory (optional). And finally, the last one is Google's magical gem to get their discovery system working.
Because we are going to communicate via SSL, we want to make sure that the OpenID library uses our certs to verify the SSL communications:
(Obviously, you need to change the path to your own cert)
We are almost done with the setup, we just need two more things:
I'm using Sinatra, so I'll load the Rack::Session
middleware before I
set Omniauth:
(Rails turns that option by default, so you don't need to worry about it)
Then I can finally setup OmniAuth:
1 2 3 4 5 |
There are two important things to notice. First, because I set the
provider's name to be 'admin', the magical paths provided by OmiAuth
will use that name (/auth/admin
). Secondly, and more importantly, notice how I added
the name of my Google Apps domain at the end of the identifier:
In Sinatra, your just need to define the routes OmniAuth would use (same goes for Rails, just use the router for that).
By default, omniauth now offers you a /auth/admin
endpoint that will
push the user through Google Apps authentication.
Once the authentication is over, the user will be redirected to the
following endpoint:
1 2 3 4 5 6 |
You can access the authentication details from request.env['omniauth.auth']
and redirect the user to another page, like the admin landing page for
1 2 3 4 5 6 7 |
On the landing page, you need to verify that the user is logged in, in this case, during the previous step, we added the email of the user to her session. We can therefore verify the presence of that information to confirm the authentication status. If the user is authenticated, then we'll render an ERB template otherwise we'll redirect her back to the login page.
We should also provide an endpoint in case the authentication failed:
1 2 3 4 |
Note that by default, in dev mode, Omniauth won't redirect the user there. To enable this behavior, use the following snippet (works with any rack app, Rails, Sinatra or whatever):
1 2 3 |
Using Google Apps for authentication with OmniAuth is trivial as long you know two things:
'https://www.google.com/accounts/o8/site-xrds?hd=' + your_domain_name
This blog post was written using omniauth 1.1.1
, omniauth-openid 1.0.1
, rack-openid 1.3.1
and ruby-openid-apps-discovery 1.2.0
. This might not apply to you if you come from the future :)
Today we are going to look at Google's Go language. Go approaches concurrency from an interesting view point. But instead of digging into the history and reasoning which led to this approach, I'd like to show you the language constructs by actually writing real life code.
The following example is taken from my recent presentation Ruby vs. the World. I explored a few programming languages and showed how they changed my Ruby.
To show how Go addresses concurrency, I decided to build a program which would concurrently fetch various web resources, wait for all of them to be fetched, then process them all at once. In other programming languages, we could have used threads) and a semaphore), actors or callbacks. Go's approach is slightly different, let's walk through the code together.
The first part of our code gets us setup:
1 2 3 4 5 6 7 8 9 10 11 12 13 |
The code above names our package then imports a few standard libraries that we are going to need. It then defines an array/slice of strings representing the urls we are going to fetch.
Next we define a type we will use a bit later:
1 2 3 4 5 |
You can think of a struct type as a simple representation of a class. Technically, we are defining a structure with some typed attributes. We will later on, create instances of this defined type.
Go implements OOP slightly differently than other languages.
Methods in Go are more general than in C++, Java: they can be defined for any sort of data, even built-in types such as plain, "unboxed" integers. They are not restricted to structs (classes).
We can therefore define methods/functions for any type of data, including "any/all" types. This approach to types is called structural typing.
Here is the code:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
This is the meat of our application. And there is quite a lot of going on in just a few lines. Assuming you aren't familiar with Go, I'll walk though the code.
Let's start with the signature:
which is an "array" of strings (I used quotes around the word
array because it's technically what Go calls a slice)HttpResponse
pointersThen in the function body:
We start by creating an instance of a channel
and assigning it to the
variable name. Think of a channel as a pipe like in unix. We can write to and read from that channel.
In the next line we create an empty instance of a slice containing pointers to
Then, using the for range
language construct, we iterate through our urls
, storing the current value being used into the scoped variable url
. The url
is then available within the block/lambda/closure marked by the curly braces.
Now this is where the async construct comes in. Using the go
keyword, we define an anonymous function that takes a string argument representing a
The function prints this string, then uses the net/http
library to fetch the web resource. We use the returned data to create an
instance of our HttpResponse
type and send it to the channel.
This part gets a bit confusing because I reused the name url
. We call this
anonymous function right away passing it the url
variable set
by the loop.
You might wonder why we bother to create an anonymous function and
call it right away instead of just executing the code directly.
The go
keyword executes the code that is passed in as a goroutine which is well explained here
A goroutine is a function executing concurrently with other goroutines in the same address space. It is lightweight, costing little more than the allocation of stack space. And the stacks start small, so they are cheap, and grow by allocating (and freeing) heap storage as required.
In other words, you start a goroutine and you let the "system" handle how it wants to deal with the low level details. Technically, goroutines might run in one or multiple threads, but you don't need to know. We trigger each http fetch in a separate goroutine and then each response is pushed down the channel.
The second block of code begins with another for
loop containing a switch/case statement.
The case statement checks if something is
in the channel. If there is something, we…
variableIf the length of the array is the same as the length of all urls we want to fetch, we are done fetching all our resources and can return. While still waiting for responses, we print a dot every 50ms.
Update: In the first version of this blog post I had used a 'default' case statement to print the dot and sleep for 50ms so the loop wouldn't be too tight and the concurrency effect was more obvious. But some HN comments pointed out that it wasn't needed and I shouldn't block. For reference here is what I had before (don't use this code):
1 2 3 4 5 6 7 8 9 10 11 12 13 |
Thank you HackerNews.
With that code constructed, our main
can make use of it like this:
1 2 3 4 5 6 7 |
Compiling and running the code look like this:
1 2 3 4 5 6 7 8 9 10 |
As you can see from the print statements, the 3 urls are triggered in a sequential way, but the responses come back in different orders due to different server latencies and response transfer time.
Go was designed to make concurrency easy for developers. It is a very well documented language and you will find on this page a lot of information about its concurrency philosophy and details about each available constructs works.
I like that the language is very simple and the constructs explicit. If you want to write concurrent code, Go pushes you to do it in a specific style. That style is clear and comfortable for me. My code stays simple, I don't go crazy with callbacks, and the conventions make it simple for everyone else to understand my code.
Whether or not Go appeals to you stylistically, but clearly the designers stayed close to the goal of developing to a 21st century C with a special focus on concurrency with the unix approach.
]]>I'll show you why we are lucky bastards, why we aren't so good at picking the right jobs and some hints on how to solve this issue.
I remember a family friend telling me when I was a kid that computers are going to be the future and that there will be a lot of jobs in this field. I also remember that the idea of sitting all day, alone, in front of a minitel-like computer scared the hell out of me. But he was right and I now work from home, spending 12+ hours in front of a monitor. I get emails and phone calls from many people reaching out to me to help them find software engineers.
At least three things make "now" the best time to be a software engineer:
There is a huge demand for engineers. There are many more job openings than candidates. But this is much better than in late 90s/early 2000 when anyone who could write a line of HTML would get a job. Now the projects people are building are way more interesting and have a real potential to change lives. Before, you had to work for a giant company to have a chance to do that. But now, with internet and smart mobile devices everywhere, almost any startup (fancy name for small company) can have a huge impact – look at Twitter for instance.
Lastly, being a geek is cool. Movies, cartoons, TV Shows now have geek heroes (granted they usually don't represent real geeks, but hey it's better than nothing).
Most software engineers I know, are really bad at choosing the right job for themselves. They don't design a career. Engineers are good at solving technical problems in an objective way, but when it comes to our jobs and future, we seem to struggle.
I'm not an expert but I have a few guesses I'd like to share with you:
Most modern companies need solid engineers to be relevant in the short/medium term and they know it. Most engineers have no idea how their talent and dedication converts into real business value. Without that appreciation, they can't easily estimate how much they are worth. The salary scale for software engineers is dramatically different from other jobs/industries. To your business, you might be worth twice or three times the salary of someone like a teacher. This is probably not fair because of the social value a teacher offers, but that's the way the law of demand works in our society. Knowing how much you are worth to a company and how much to ask is critical to properly negotiate or renegotiate a contract.
I remember when I moved to America and was thankful to have a job. I had no idea that at $45k/year with basically no health coverage and no vacation, I was grossly underpaid and could have been paid twice that amount down the street. (Note: the average software engineer salary in the US is at $89,000 according to indeed.com). The average salary for a high school teacher is $47k/year so I wasn't complaining. As a matter of fact, I didn't leave this job because of the salary. I truly believe that "Money doesn't buy happiness" and it shouldn't be your primary reason to accept or leave a job. Money is nice and often makes life easier. But the point is that you have to understand how much you're worth, so you can get paid and focus on your work.
John Allspaw wrote a great blog post about what it means to be a senior engineer. I strongly recommend you read it. I often look back at it and pick up a couple points I need to focus on myself. John wrote a book which is a collection of essays and interviews regarding tech/web ops.
Here is a very interesting quote:
Not everyone can be senior. If, after five years, you are senior, are you at the peak of your game? After five more years will you not have accrued more invaluable experience? What then? "Super engineer"? Five more years? "Super-duper engineer." I blame the youth of our discipline for this affliction. […] Given the dynamics of our industry many elected to move on to managerial positions or risk an entrepreneurial run at things."
There are two very strong points in this quote:
I might be a bit radical – but the day I stop learning/improving will be the day that I will quit, change jobs or careers. John's post has great pointers to help us improve our skills. But the question I'm trying to raise is:
what do we want from our career?
"Career" sounds like a dirty word to many of us. It has a corporate, sleazy, back stabbing connotation. When I hear it, I picture a cliché stock photography of a bunch of smiling people wearing 80's suits. In the context on this post, let's take the Oxford English Dictionary definition: "course or progress through life (or a distinct portion of life)". The word comes from from French via the Old Occitan word: "carriera" which means "street".
A better word for career might be "path".
I think we have a hard time knowing what kind of path we want to be on. When faced with the question, a lot of us answer: "solving problems", "having fun", "changing the world". All these answers sound good, but they aren't paths, they are just attributes.
I have to admit that I'm still struggling with this question and probably will for a while. I'm pretty good at defining short term goals but I have a hard time seeing the long term path. As a matter of fact, a little while back I was seated in front of Chad Fowler in his office in Washington, DC. Chad wrote a great book called "The Passionate Programmer: Creating a Remarkable Career in Software Development".
I've read Chad's book (which inspired me in many ways) and have known Chad for years. The point of our meeting was to decide what I was going to work on next. Unconsciously, I expected Chad to just tell me what I should be doing. I trusted him to pick the right "path" for me. But I was surprised when Chad told me: "you're the kind of engineer who can do anything. You're a generalist who can pick a topic and become a specialist. So what do you want to do?"
I wasn't sure how to take it, it sounded like a compliment but, at the same time, the fact that Chad didn't have a solution to my problem bothered me. I remember thinking, wait, he's the expert and he's deflecting the situation by asking me the question I came to ask him. Sure, the compliment was nice but that wouldn't solve anything. What does that mean about me? If an expert can't figure out what I should do, I might be screwed.
Then on my way back home, I realized that it didn't matter how well Chad knew me, he couldn't guess what I even didn't know about myself.
My long term happiness depends on me finding a direction I want my professional life to take. In Chad's book, there is a strong focus on finding a market, understanding it, developing skills and marketing yourself. However there was something I had missed.
The goal-oriented, destination-focused thinking that you usually do leads only from one goal to the next. It has no logical end. What most of us fail to realize is that the path is the end.
I've always known that the journey is more important than the destination in itself, but what I had missed is that you still need to define a destination or maybe more precisely a direction, a vector. My problem is that my path was just a bunch of scattered dots. Hopping from one to the other, I was hoping it was going to make a pretty drawing. The challenge is when I got to a spot, I was stuck not knowing what to do next. I ended up picking another short term goal/destination based on the opportunities available at that time.
What I should do instead was to define a general direction and then learn through the process. I believe this will help me enjoy my job more than running after goals. It will allow me to see the world differently and will help me make the right career choices when the time is right. To be honest, I think that's the only way I can build endurance and not burn out in 5 years. That said, I'll still have goals, deadlines and the usual – but they won't define my own personal progress.
Changing jobs is a pain. As an engineer I weigh the pros and cons and try to logically pick the right choice – at least in theory. In practice I avoid dealing with questions that might result in challenging consequences.
Quiting a job is tough. You have to tell your current employer and your colleagues that you are leaving them for something you think is better for you. The nicer the people you work with, the harder it is. The better you are paid, the harder it is. If you work with nice people and you're well paid, leaving is really hard (take note if you run a team).
In our profession, changing jobs isn't usually seen as something bad. Recruiters might pressure you to take a new, better job. Beware impulsive changes though. Recruiters want their commissions so they'll do anything they can do make you switch. They'll try to convince you that the grass is greener on the other side. Maybe that's only "bad" recruiters.
There are some recruiters who care about people and companies. Recruiters who will help you find the right job for you. However, they won't be able to help you if you don't know what direction you want to go to.
I remember being stuck in a pretty terrible work environment, being underpaid and the projects I was working on weren't going anywhere. A friend gave me a Seth Godin's book called "The Dip: A Little Book That Teaches You When to Quit (and When to Stick)".
I'm not a big fan of self-help/business books, but this book raised a very good and simple question: how do the efforts compare to the returns? Which situation are you in:
Think about it. Will the effort you put into your work pay off? If you don't think it will, then you should quit right away.
The logic is pretty simple but requires you to forecast. For that you need some sort of metrics helping you to see if you are getting closer or further from the direction you set for yourself.
Trust is the key element of any relationship. In The Five Dysfunctions of a Team, Patrick Lencioni explains that the base of management dysfunctions is absence of trust:
Turns out it's the same thing for our careers. We need a team of people to help us define a vision/direction and keep us honest and accountable.
Find people who you can trust to talk to about your professional goals, your progress, failures and doubts. People who will be honest with you and tell you things you might not want to hear. Find mentors and honest people. These people don't have to be working in the industry. They just have to be able to listen and care.
People like that are extremely hard to find, but so are good executives. I believe that having trustworthy friends (partners/family members..) who care is a big part of what makes someone successful.
As I explained earlier, I'm no expert and I also struggle with the issues I described. However, I'd be glad to provide a bit of my free time to help you think through these issues.
A lot of you are doing a great job without the rest of us noticing. If you don't have a popular twitter account, blog, open source projects, published books or given talks at conferences, it might be hard to get yourself noticed or even know how much you're worth. Trust is a big deal and if you are considering moving on, you probably don't want your boss and your colleagues to know. You probably also don't know good recruiters you can trust. You might not even be sure it's worth investing too much time.
Most of you probably won't consider it, but I'd like to offer my help if you'd like it. I promise full anonymity and no strings attached. Just
After exchanging a few emails , I'll try to make good use of my network to find you a way to move in your desired direction. Or, if you work for an interesting company with current openings, feel free to contact me too.
Update: My inbox is overflowing with emails and I already spent literally days replying to as many people as possible. I'm sorry but at this time I can't reply to any new enquiries. I'll write a follow up blog post that covers what I learned from my interactions with so many readers.
I have no idea how this will turn out. Maybe I'll get a couple of emails, zero, or way too many to handle – but it's worth a try. I do have a full time job, so please don't expect me to reply to your emails within the hour.
I finally took the time to enable the comments in this blog. Feel free to leave advice or feedback.
]]>Ruby is an awesome programming language, it's so pleasing you probably haven't seriously looked at other languages since you switched. The programming world is evolving fast, new languages are created daily, new trends are emerging. Let's take some time to look at a few languages from a Ruby developer perspective.
The slides are available on [Matt's SpeakerDeck page]https://speakerdeck.com/matt_aimonetti/ruby-vs-the-world) and can be downloaded here.
Matt's presentation was filmed by Confreaks and posted here.
For more RubyConf 2012 talks, go there.
]]>mruby is Matz' new Ruby implementation, it's not cooler than node.js, it doesn't natively support Hypstermedia, it looks just like the good old Ruby. So why should we, as a community care?
Matt's talk is divided in two parts, an introduction of mruby (embedded Ruby) and and revisiting Ruby.
The slides are available on [Matt's SpeakerDeck page]https://speakerdeck.com/matt_aimonetti/mmmm-dot-mruby-everywhere-and-revisiting-ruby) and can be downloaded here.
Matt's presentation was filmed by Confreaks and posted here.
]]>Programming languages affect the way one looks and solves problems. But comparing programming languages isn't as simple as drawing a table comparing features.
In his talk, Matt shows what he likes, dislikes, the philosophy and concrete example of how to use 7 programming languages:
The slides are available on Matt's SpeakerDeck and can be downloaded here.
]]>Instead of arguing the pros and cons of OOP vs FP and how each of the previously mentioned languages handle being OOP and FP, I'd like to introduce a very powerful Scala idiom: pattern matching. Note that pattern matching isn't something Scala invented nor that it only exists in Scala. Pattern matching can be achieved many different ways. However, the majority of the popular languages don't put this concept at the center of their language. A few languages way before Scala rested heavily on pattern matching such as Erlang, Haskell but that's a different story. How does Scala offers Pattern Matching, what is it and finally why is it valuable?
As its name indicates, pattern matching is used to detect patterns. Here is an example that covers a few interesting cases:
1 2 3 4 5 6 7 |
If you've never seen any Scala that probably looks like gibberish to you. Let me break it down:
I define a new function called listAnalysis
which takes an argument
named list
which is of type List
(this list could contain any kind
of elements).
The implementation of this function is a pattern match on the list
The body of this 'pattern match' looks like a classical switch statement.
But it's actually much more than a simple switch statement. Surely it
could be used like one, but as we will see, it can do much more.
Note that you can apply a pattern match against more than one object at once as shown a bit later.
Let's look at the statements inside the function.
In this case we are checking that the list is empty or nil. If that's the case, the statement on the other side of the "fat arrow" is executed. In this case, we return a string but we could have called another function or so whatever.
Now the second statement is much more complex and much more powerful:
Remember that we are doing pattern matching against our list object.
What we are doing here is use the ::
operator (aka cons operator) to
extract the head and the rest of the list and then we match the head
against the 'a'
This statement could have been written different ways:
In that case we named the tail
of the list rest
, but really we don't
care how it's called or its value, so the sensitive thing to do is to
rewrite that statement like that:
This basically says we are looking for a list that starts by 'a'
(and we
don't care about the rest).
Another statement:
In this case we type match the first element of the list and check that we have an integer. Note that using the cons operator in the match cases doesn't seem to affect performance. It would seem that at compilation, the statement are rewritten to avoid creating uneeded objects (List also implements structural sharing of the tail list). I'm not a Scala expert so someone with more experience might be able to confirm/clarify.
Now let's look at a variant of this statement:
This is the same statement as above, but we are adding an extra condition after the match. This is quite useful when simple matching doesn't cut it.
Finally we have a fallback statement:
For more information about the cons operator, read about the extractor objects and what they can do.
Here is the result of calling our function with different lists:
1 2 3 4 5 6 |
Here is another example using 2 items for the match:
1 2 3 4 5 6 7 8 9 10 11 12 13 |
In short, pattern matching allows the developer to deconstruct a structure to find specific elements, in other words the pattern, needed to then constuct an object/structure or trigger a function.
It's the opposite process of calling a method on an object. Here we start from a structure (instead of the instance of an object), this structure is just a basic struct and based on a found pattern, we then trigger a function (with access to the data if we need it). When you have a stable and known data structure, it's often very interesting to use the pattern matching approach because you can easily expand the operations you can execute. However, if your operations are stable but the data changes, then the Object Oriented approach seems more adequate.
Besides that, pattern matching will often make your code clearer than using if/else statements. Especially in a language like Scala where you can define pattern matching function within a function and you can also pass pattern matching functions around. Like eveything else, it needs to be used with caution so the intend of the code is still understandable. That said it's a great tool to have handy and I've had a lot of fun rewriting my newbie Scala code using a more idiomatic approach based on pattern matching.
I hope you enjoyed this quick introduction. You can read more about pattern matching in Scala in the following articles: (note: Ikai's post on how he uses regexps with pattern matching is a fun read.)
An example of something that might seem simple is inheritance. Ruby, unlike C++, doesn't support multiple inheritance. What that means is that a Ruby class can only have 1 parent class (superclass). However multiple inheritance can be achieved via modules used as a mixins. That's a very common pattern, people put some code in a module and then mix it in/include it in a bunch of classes. The problem I see though, is that people abuse these concepts and don't respect the difference between a class, a module and a module used a mixin.
Object Oriented Programming 101:
"In object-oriented programming, a class is a construct that is used to create instances of itself – referred to as class instances, class objects, instance objects or simply objects. A class defines constituent members which enable its instances to have state and behavior."
So if you create a class and you don't create instances, you are using the wrong construct. Here is an example of what I often see:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
Which can be used as such:
1 2 3 |
(granted this isn't a great example since we could have used a subclass
of Hash
but just bear with me)
Actually, the developer who wrote the code above would probably also use
some Ruby magic like method_missing
to provide a more laxed API and allow for "nicer" getters
such as Settings.secret
and Settings['secret']
. I have my
own thoughts on the topic but it's
an entirely different subject.
Note also that the way class level methods are defined can also be different depending on who wrote the code, you might see the following variations (and other more esoteric ones):
1 2 3 4 5 6 7 8 9 10 |
The Settings
code above works, the code is simple, yet I will argue one thing: it's
an abuse of the class construct. We're breaking the #1 rule of classes:
"create instances of self".
It's easy, whenever you don't create instances of a class, please don't use a class.
That's also true for slightly different examples such as:
1 2 3 4 5 6 7 |
There is no need whatsoever to create an instance of API
, using a class is
picking the wrong construct. Also, I don't care if you use the Singleton
module to only allow 1 instance of the class, you still shouldn't use a
class in the above example.
In Ruby's object hierarchy, the Class object actually inherits from the Module object.
1 2 3 |
As per Ruby's source code defintion:
"A Module is a collection of methods and constants."
I like to think of modules as namespaced methods and constants. Whenever you want code that logically belongs together but that won't require that you create instances of a 'model', then a module is the right construct to use.
As a matter of fact, the two examples above are great cases where a module should have been used.
The confusing bit is that modules can have module level methods but also instance level methods. Here is an example:
1 2 3 4 5 6 7 |
This is a module level function, it could also be written like that:
1 2 3 4 5 6 7 8 9 |
And be used like that:
Until now, it makes sense. The weird thing is that even though, modules unlike classes aren't meant to create instances, we have the possibility to define module instance methods.
1 2 3 4 5 6 7 8 9 10 11 12 13 |
Great, but we can't actually use these methods since they are instance methods and we don't create instances of modules. Well, that isn't quite true, there is a way to access them and that's by using a module as a mixin. What that means is that we inject/copy the module code inside a class or an(other) object. Example in code:
1 2 3 4 |
Or we can add the code to a class so the instances of this class can access our module instance methods:
1 2 3 4 5 6 |
There a plenty of resources online about the many ways to use mixins in Ruby to achieve multiple inheritance and do cool stuff. But the point of this article is to try to demonstrate that mixins shouldn't be abused.
My problem with the above example is that by mixing in the Settings
module inside our Foo
class, we created an uneeded, confusing extra
level of abstraction. Instances of Foo
now have access to two
methods/objects: repository
and secret_key
. These methods or the
objects they refer to don't belong to the Foo
class, but it seems
convenient to not have to type Settings.repository
so we mixed things
in. Plus, a lot of Ruby developers seem to dislike adding class/module
level methods so they feel that this approach 'feels better'.
Here is the thing, the convenience of typing a few less characters isn't
worth it. Next time you or someone else will look at an instance of the
class calling repository
, finding where it is defined is going
to be a pain. That's especially true if you have many mixins in your
class. Settings
will also probably grow and you will end up with a
bunch of methods that have nothing to do with your class instances.
In this case, I will call the use of a mixin, an abuse of construct.
Sure, Ruby allows you to do it, but that doesn't mean it's the right
thing to do. In Ruby, unlike in Python, there are 101 ways to do a
simple thing. It doesn't mean that the 101 ways are good, it just means
that Matz wasn't sure how people would use his programming language and
chose to give us more freedom to messup/doing it our own way.
I have my own rule: use mixins whenever you need to share behaviors between different classes.
In the above example, we weren't sharing behaviors, we were sharing objects, there was no need to actually use a mixin.
That said, rules aren't rules without exceptions. A good example of this
exception would be the Math
module from the standard library.
This module offers trigonometric and transcendental functions. You might
think that this module would be designed to be a mixin so you can get
, cos
, exp
and friends available in your math related classes.
It turns out, all Math's methods are defined a module functions meaning
that they are meant to be called from the Math
module directly.
However, Ruby allows you to mixin module functions, but these functions
become private. If you do include the module inside your class, your instance methods
will be able to call hypot(x,y)
directly, but these methods won't be
available from the outside (Foo.new.log(42)
would raise an
To conclude with mixins: mixins are great but don't abuse them or you
will endup with so much abstraction that your coworkers will secretely
call you Kandinsky.
Stick to simple mixins allowing you to share behaviors between at least
a couple classes. See DataMapper
for a great way to use mixins.
I have to say that I do like functional programming. The idea of having functions not mutating the states of things around them pleases me. It just seems clean, you feed data to a function and you get another piece of data. No states were changed, maybe some temporary variables were allocated to process the data, but the only thing that matters is the input and the output. Easy to grasp, easy to follow, no magical states being changed by some code fairies.
The good news is that Ruby allows us to write code like that. And this
is where modules are great. Very much like the Math
module we
discussed above, there are many cases where you want to have a bunch of
functions that process an input and provide an output without keeping
any states. A good example of such a module would be a param
verification filter. The filter takes an input, takes some rules and
verifies that the input matches the rules.
Surely, we could create an instance for each verification, this would
allow to keep states in our class and do the usual OOP things. But we
could also simply use a module with a bunch of module (level) functions
that would pass to each other the input they need to not need to keep
states. The end result will be faster, nicer on the GC and easy to
Mixing OOP and functional programming isn't new, ask Scala developers! If done right, by adopting this approach we can simply our code base, make it faster, easier to maintain and not losing the chance to also use OOP paradigms when needed.
As shown earlier, modules and classes have pros and cons. Classes are
however much more natural to use for Object Oriented Programming. A
compromise about the Settings
examples was suggested by Evan Phoenix.
The solution is elegant and simple. Use a class and an instance.
Here is an implementation based on his suggestion:
1 2 3 4 5 6 7 8 |
The point here is not about the class implementation but that fact that we use a class and associate an instance of this class to a constant so it can be shared all over the place. Wow, a constant, this is so nasty you might think. Classes are constants too and so are modules, here we just allocate an instance of a class to a constant. Surpisingly simple and efficient.


text:
Buy Tickets The Returned
Show From Series
Sunday, June 15
Monday, June 16
Tuesday, June 17
Wednesday, June 18
Thursday, June 19
Friday, June 20
Saturday, June 21
The Returned
November 23, 2013
In an idyllic French mountain town, a seemingly random collection of people find themselves in a state of confusion as they attempt to return to their homes. What they do not yet know is that they have been dead for several years, and no one is expecting them back. Read more »

subset: commoncrawl


text:
International Negotiations and U.S. Climate Legislation
May. 21, 2009 / Posted by: EBast / Categories: Blog
The nations of the world expect U.S. leadership this year in international climate negotiations, which will culminate in the Conference of the Parties of the UN Framework Convention on Climate Change in Copenhagen, Denmark in December. These negot...
Read More »


text:
- Your file is being downloaded. It will remain available for download from the "My Files" section of your member area for 72 hours.
#3378076 -
mr. dead
© Konstantin Sutyagin - See portfolioPricing
Author | Konstantin SutyaginSee portfolio |
Image reference | 3378076 |
Country | United States of America |
Descriptive Category | People On Location Performance/Circus |
Conceptual | Lifestyles Celebrations Halloween |
Keywords |
["actor","cabaret","cane","character","creep","creepy","curtain","curtains","dark","dead","drama","dramatic","eye contact","face","halloween","hat","horizontal","horror","makeup","man","mister","movie","mr","one","people","performer","person","photo","play","portrait","posing","rags","scary","single","smile","smiling","stage","stare","staring","studio","suit","surreal","theater","theatre","wide","wide-eyed"]
actor cabaret cane character creep creepy curtain curtains dark dead drama dramatic eye contact face halloween hat horizontal horror makeup man mister movie mr one people performer person photo play portrait posing rags scary single smile smiling stage stare staring studio suit surreal theater theatre wide wide-eyed
See all keywords
The photo "mr. dead" from Konstantin Sutyagin is available on Fotolia under a Royalty Free license from 1 credit (Credit from $0.74).
Available in JPEG format, this photo may be downloaded for all kinds of professional uses and in different resolutions (up to 3,504 x 2,336 pixels in 300 DPI).
To download this photo, you can buy Fotolia credits or purchase a subscription plan and benefit from the amazing price of $0.19 per image.


text:
Great page and research! I was a Donny fan once upon a time...Like in the 60's and 70's. Oooops, show my age again :D!
Printable View
Great page and research! I was a Donny fan once upon a time...Like in the 60's and 70's. Oooops, show my age again :D!
here's my page - all about 1964 when us brits inflicted the beatles on the usa :D
ps i spelt sullivan wrong....apologies
Just to help you out if Fiona doesn't mind..I found this site and it gives you great info on what happened on your birthday... http://www.dmarie.com/timecap/step1.asp Just type in your birthday and have fun with what you find..Real interesting...
*going to work on my page now...* :D
Love these pages girls!
Love your page Fiona! Wolvsie, that is a great site thank you! Still have to do mine.
umm you can still be a fan now lol....they are all still going strong and donny does the most amazing fantastic concerts, over here he feels out arenas and comes over every 18 mths or so....poor guy knows me by name now after I travelled to australia and nz to see him...shared a lift alone with him and sat in front of him for 3 hours on a flight lol.....bless he recalled me when I had a VIP in october and 3 of my photos made it into the uk tour programme :)
My page. The year was 1953
Used Fiona's freebie newspaper for background and the rest is just paperscraps
Here is my page for the year 1959. I used Fiona's newspaper background and some 1950's clip art from the forum embellishment section. Thank you to whomever put those embellishments in that section. (I'm not sure of the name.) My mother didn't have any newborn photos taken of me so this is the best I could find. :) I just noticed that the Game of Life was cut off of the original for the toy listing. And gas is 30 cents per gallon not $30 per gallon. :OI'll need to fix that!
these are all so fabulous, love reading all the info ....hasn't everything gotten so expensive!
Here's mine. I used Ma3's Celebrate America kit available in the freebies section.


text:
• depth of field •
About This Zine
A zine of art photography. News, art criticism, camera/equipment reviews and links are welcome, too. Your images can speak for themselves, but if you'd like to say something about the story behind your photograph, that would be welcome. > more aboutLATEST ACTIVITY ▸
Sunset Bridge Ananasa .com
Who can resist the gorgeous sunsets regardless of where you are in the world....
The Desert Journey Ananasa .com
From Bedoiuns to travelers, a scene that captures the details of the sky amongst an open desert at sunset before…
Into the Savannah Ananasa .com
A lovely pair of gazelles grazing in the savannah, capturing nature at it's best with it's grazers and natural beauty.
The Victim Of Social Media Irvin A Kelly
Art For Sale The Victim Of Social Media 27.38 X 48.00 Picture Rag Limited Edition 3/3 Copies $420.00 Find me…
One year passes... Cliff Furnald
...another begins. I wish you all a fine new year. As Woody says, "There's a better world a-comin'."
Would you like to join "• depth of field •"?
• depth of field •
can submit immediately
editors and members can submit
• depth of field •
can submit immediately
editors and members can submit