A blog by Ninefold and friends.

Ninefold Uptime

A new Ninefold Australian-based customer recently asked about how our uptime compared to competitors.

A few of us hadn’t looked at our APAC uptime for a while but we had a feeling it was good; we couldn’t immediately remember any incidents. This peaked our interest, so we did some analysis thanks to our friends at Cloud Harmony:

logo

What immediately jumped out is Ninefold had the lowest downtime over the last year across both our APAC DCs except for AWS*. We were one of only 4 providers to provide 4 Nines uptime measured over the last year.

Why the asterisk for AWS? AWS claims to have 2 standalone deployments in AU, yet they only show up as a merged uptime measure. We are not sure what this means specifically, but it doesn’t seem to capture their October outage.

So, to answer the question, we compare pretty well and the data proves it. Market leading might be a good term. And a call out to our ops team who deliver this every day.

Of course, we always make this information available through:

Total Cost of Hosting Ownership (TCHO): What Goes Into It?

It costs money to provide online servers to host any app, website, or other product. In most cases, though, analysis of these costs begins and ends with an online calculator, if the cost issue is even raised at all. And because cloud servers are relatively cheap and can easily be spun-up and spun-down on demand, most companies don’t look any farther than that.

Unfortunately, it’s usually not as simple as that. Thinking of hosting or cloud service costs as no more than a monthly line item on your credit card misses many of the true costs of hosting. For a realistic picture, you need to think about the Total Cost of Hosting Ownership, or TCHO.

shutterstock_124697734

Your TCHO is made up of four factors:

  1. The actual cost of hosting. This is the easiest to identify and compare, but may be the smallest component of the total cost.
  2. The performance of the app, which can impact customer experience and ultimately revenue. This depends on how much revenue you generate through your app.
  3. The internal cost of deploying your code into production. This is the opportunity cost of the time it takes to actually get your code into production.
  4. The internal cost of monitoring and managing your production environment. This hidden cost covers making sure the networking is sound, the servers are working, and that customers can actually use your app. You know, ops.

To help explain the four components of TCHO, let’s look at the fictitious ABC Corp., an app-based, online-only, Software-as-a-Service provider earning $2 million yearly revenue. ABC Corp. has a CTO/CEO, five full-time developers, one testing/release manager, a support tech, a COO who looks after finance as well, a marketing manager, a community engagement manager, and a marketing systems admin. Let’s assume the 11 staffers earn an average of $150,000 for a wage bill of $1.65 million a year. The company also pays for a variety of systems (Jira, GitHub/BitBucket, BaseCamp, CRM, CMS, MarComms automation, billing, and more) that adds another $200,000 in costs, and ABC Corp. pays $50,000 in annual rent. That’s a total of $1.9 million in costs leaving $100,000 in profit … before we get to hosting.

1. Actual cost of hosting

The actual cost of hosting is the easiest to quantify, but relying solely on this figure leaves out many important expenses.

The choice of a hosting company is typically made by the developer team or the COO. Developers usually select a vendor they’ve used before, or perhaps a hot new vendor they’ve recently heard about. The COO looks at some calculators online, pretends to listen to the dev team, then chooses what he thinks is cheapest.

Lets visit our friends at ABC Corp., who currently leverage a basic Infrastructure-as-a-Service (IaaS) vendor for hosting–roll your own servers, no bells and whistles. Their hosting requirements run $2,000 a month ($24,000 a year) including testing and staging environments, and all the usage costs. ABC has also tested a Platform-as-a-Service (PaaS) offering that would cost $3,500 a month ($42,000 per year). They went with the IaaS vendor hoping that the $18,000 yearly price difference would drop right to the bottom line as profit.

Did ABC Corp. make the right decision?

2. App performance

The performance of an app can be measured in both page load time as well as the consistency of this performance. Both measurements matter in very different ways. Here’s some of the easiest ways to improve customer conversion and revenue with simple app performance improvements.

Faster page load speeds have long been linked to customer retention and sales. As quoted by CNET, Google’s data suggests a 500ms erosion in load time resulted in a 20 percent reduction in click through. Amazon reportedly provides an even easier metric: for every 100ms improvement in page load, revenue increased 1 percent.

Does 1 percent really matter? Well, for ABC Corp., every 1 percent represents $20,000 in additional annual revenue. If a different hosting provider could improve ABC’s average page load time by 300ms, that could mean $60,000 in additional revenue. If the PaaS offering delivered that improved performance out of the box, its additional cost of $18,000 would be more than offset by the additional revenue, resulting in an extra $42,000 in profit.

Consistency of app performance is even more important. Every interaction each user makes (typically a click) impacts their experience of your app or site. Their experience is not just the first page load, but their entire journey, which involves multiple clicks. This is something that we have tested extensively at Ninefold.

What does that mean for hosting and revenue? Let’s make some assumptions:

  1. 5 percent of clicks—1 in 20—are inconsistent
  2. 80 percent of the affected users will continue blithely along even after a slow page load—a very optimistic estimate
  3. The typical transaction path has 13 clicks.

Why is the number of clicks important? Because 1 in 20 clicks is not the same as 1 in 20 users. To calculate the number of users affected, remember Statistics 101 in college. The chance of each click not being affected is 95 percent. However, each click on the journey is subject to this same risk. Of course, not every click has the same impact, but to simplify the math, let’s assume they’re all equal.

The chart below shows the chance a user will face at least one “inconsistent click” based on the number of clicks in the process:

TCHO chart

With 13 clicks, one of every two visitors will experience the “inconsistent click” phenomenon.

What does this mean for ABC Corp.? If 50 percent of its users experience the inconsistent performance, and of those 20 percent leave and do not transact, the company is forfeiting 10 percent of its transaction funnel. On $2 million in revenue, that works out to $200,000 of potential revenue that could be lost due to inconsistent app performance.

3. Deployment costs

These are the costs for your team to get every release pushed and stable. How much can this cost? ABC Corp. has six people in its dev/release team, for a total of $900,000 in staff costs. The team runs on three-week sprints, with 17 releases a year.

If we assume that a PaaS deployment solution would save an average of half a day for the entire team for each release compared to staying with the IaaS provider. That’s a total of 8.5 workdays per team member, about half a sprint. Looked at another way, it 3.4 percent of the 250 available workdays for the entire year. At our assumed rate of $150,000, that comes out to $30,600 of wages.

4. Monitoring and managing your production environment

Given the importance of performance, it’s clear that you also have to devote resources to deployment, to keeping your app running and your systems flowing, and your networking operating smoothly. That typically means employing a SysAdmin or instituting a DevOps structure.

So how much would a DevOps approach cost? Let’s assume that each dev team member spends one week on rotation monitoring the environment. Further, let’s assume that each week a developer is on DevOps watch she loses 50 percent of her development time if ABC Corp. is using an IaaS solution vs. just 25 percent of her time with a PaaS solution. For ABC Corp., with a $150,000 developer cost, DevOps costs $75,000 in an IaaS world but just $37,500 in a PaaS world.

Adding up the Total Cost of Hosting Ownership

Let’s review the TCHO situation for ABC Corp. It currently spends $24,000 per year for basic Infrastructure-as-a-Service hosting. Going to a more comprehensive Platform-as-a-Service hosting offering would cost $42,000, an $18,000 a year difference.

But according to our calculations and assumptions, the PaaS solution would have the following effects:

  • Improving App Performance would bring in $60,000 in additional revenue
  • Making App Performance more consistent would bring in $200,000 in revenue now being lost
  • Improved deployments procedures frees up $30,600 of developer time
  • Reducing DevOps commitments for developers would release another $37,500 worth of developer time

That’s a total of $328,100 in annual revenue gains and productivity savings. Subtract the $18,000 price differential and you get $310,100 in net benefits for ABC Corp. That’s enough to hire two more developers!

Of course, these figures and assumptions may not directly match your company’s situation, but the principle is clear. Considering only the actual costs of hosting ignores important factors potentially worth far more than any out-of-pocket savings.

Spreadsheet image courtesy Shutterstock.com.

The Vibe

The Vibe: “Northern Holiday edition”

I have been on holiday at the beach. I highly recommend it. I swam a lot. I saw a whale. And, to be honest, I kind-of forgot all about the Vibe. Sorry about that.

Scriptster - Making your Ruby scripts all hipster

Scriptster is a simple boiler plate for using Ruby as a (shell) scripting system.

Proxygen - HTTP framework

Facebook released an interesting C++ HTTP framework for building high performance HTTP servers. An excellent reference for modern C++ in action.

Coordination Avoidance in Database Systems

Summary of a rather deep paper pdf on coordination in db systems

Serializing transactions is sufficient for correctness, but it is not necessary for all operations of all applications. The downside of serialization is that it kills scalability and is overkill in many cases.

Playing with the Elixir and Go concurency models

Using Elixir to model Go’s Channels. An excellent dive into what makes these two languages such solid choices for concurrency.

Microsoft announces Remote IE; run IE on your Mac almost Natively

What it says on the box: run IE on OSX. Very very handy.

DevArcs - On the philosophical ramifications of microservice architectures

Microservices pull software architecture from the high-level and abstract to a concrete and emergent concern of a truly cross-functional development team.

8 Docker development patterns

Here I will outline some patterns that have started to show up repeatedly in my use of Docker. Very useful if you are just starting to learn Docker. Which is so hot right now. As you may have already heard.

Raptor

Raptor is “a radically new Ruby web server”. Which seems only slightly hyperbolic. Appears to have some sort of caching built-in.

Distributed Data Types – Riak 2.0

One of Riak 2.0’s most exciting features is support for Commutative Replicated Data Types or CRDTs, including Counters, Flags, Sets,Registers & Maps.

RefluxJS

RefluxJS is a simple library for unidirectional dataflow architecture inspired by ReactJS Flux.

State machines in Ruby

I’m not a fan of the currently popular Ruby state machine gems (state_machine, AASM). State machines are simple, and these are complicated gems with large APIs.

Internet Arcade

The Internet Arcade is a web-based library of arcade (coin-operated) video games from the 1970s through to the 1990s.

The Hammer-Feather Drop in the world’s biggest vacuum chamber

The world is amazing


The Vibe is an eventually consistent weekly collection of interesting links and … stuff.

It’s Ruby, it’s Rails, it’s programming, it’s databases, it’s the vibe, and, uh … No, that’s it.

It’s the vibe[^vibe]

[^vibe]: Confused? Watch this clip from The Castle and you might understand. Watch the whole film and you definitely will.

Threads in Ruby (part one)

threads

Purpose of this post

I recently gave a talk about Threads in Ruby but since a few people believed this was a hardcore topic for a 15-minute talk, I thought I would elaborate in this post.

Why do I need to know about threads?

My main purpose to discuss threads is not for the audience to start writing multithreaded code but to have a good idea of what is happening to their application when they are using multithreaded gems in their codebase.

Nowadays a good number of us are using Sidekiq for background processing or Puma as an efficient web server and I’m sure there will be more of such gems soon enough so I believe this makes it essential for us to know how our applications behave when we use such gems and how we can get the most out of them.

Threads vs Process Forks

Fork means that a process creates a copy of itself. Threads have one parent process and share memory space.

A good example is the difference between Resque and Sidekiq. Resque uses forks whereas Sidekiq uses threads instead.

To fork a process, the operating system will create a separate address space and copy all the memory segments of the parent process. This also makes it more expensive for forked processes to switch context since OS needs to save and reload everything. Also, if the parent dies while there are children that haven’t exited, then those children become zombies.

zombie processes

Threads on the other hand, have considerably less overhead since they share address space, memory, and allow easier communication. Also, with threading you do not have to worry about zombie processes since all threads die when the process dies, avoiding the issue of zombies.

Concurrency vs. Parallelism

Rob Pike explains the difference between these two concepts very nicely in this video.

In a nutshell: * Concurrency means dealing with a lot of things at once * Parallelism means doing a lot of things at once

This is a very important difference. I give you an example: Imagine, you have 2 threads that are doing the exact sequence of operations (A, B and C). Operation B involves some IO.

Now thread 1 uses CPU to execute A while thread 2 is waiting. Thread 1 then switches to B and does some IO stuff. Meanwhile thread 2 uses CPU to do A. This process includes Concurrency meaning threads are doing different things concurrently and minimizing the time that CPU is idle.
CPU is still used by only one thread at each point of time.

Parallelism on the other hand means that thread 1 and thread 2 are using CPU at once.
Multiple threads use CPU at the same time.

Threads make Concurrency as well as Parallelism possible.

Performance

Concurrency alone can increase performance even without Parallelism. If the workload of a thread blocks on I/O, Ruby can context-switch to other threads and do other work until the I/O finishes. This can happen when the workload reaches out to an external API or is accessing the file system.

Memory

Threads are more memory efficient compared to processes. Main reasons are the following: * They share the same address space. * They do not need a separate Virtual memory allocation * Context switching that is the process of storing and restoring the state (context) of a process or thread so that execution can be resumed from the same point at a later time is cheaper with threads.

Threadsafety

Threadsafety tries to solve the problem of multiple threads that share variables try to modify them at the same time. There are a few solutions for this problem one of which is Lock/Semaphore/Mutex.

Mutex(Mutual exclusion) is used to synchronize access to a section of a code so only one thread at a time can access it. This is the way Ruby handles threadsafety. We’ll look into that a bit later.

Threadsafety in Ruby

Ruby is not threadsafe. A good example is data structures such as Array and Hash. However different Implementations of ruby behave differently:

The difference between Thread in MRI vs. JRuby and Rubinius:

  • MRI: It has GVL(Giant VM Lock) or GIL (Global Interpreter Lock) which is wraps all your code in a Mutex and therefore prevents Parallelism except for one case: If you have one thread that’s waiting on IO (i.e. waiting for a response from the DB), MRI will allow another thread to run in parallel. In MRI even when writing multi-threaded Ruby only a single thread is on-CPU at a point in time.

  • JRuby and Rubinius: Rather than a global lock, key parts of the runtimes are protected with fine-grained locks so that their internals are thread-safe which allows your code to be truly parallel.

This means that if you run a multithreaded code such as this:

array   = []
threads = []

20.times do
  threads << Thread.new do
    15.times do
      array << 'blah'
    end
  end
end

threads.map(&:join)

puts array.size

The result in MRI is consistent but in JRuby and Rubinius it changes every time.

Rails however is different. The vast majority of normal, day-to-day Rails programming is inherently threadsafe. There’s an old but informative article on this topic that I encourage you to read. Since Rails 4 threadsafety is enabled by default.

Next post

This is the end of part one. In the next post I’ll dig into the things you need to watch for in order to have a threadsafe code. I’ll also talk about the future of threading in Ruby and where it is heading. See you soon!

The slides for my talk exists here
The video of the talk exists here kindly added by codr.tv

The Vibe

The Vibe: “Business as Usual Edition”

This week/s I have nothing exciting to report.

I’ve been experimenting with deis, which has come a long way in a very short time. Trying to track the explosion of tech in what I will loosely call ‘Dockerland’ is impossible. So. Much. Stuff. I have no idea what I am doing.

The Path to Full Time Open Source

Mike Perham, creator of the excellent Sidekiq talks about going full-time developing Open Source. Living the dream.

Plan out how you can make it better: simplify it, discard superfluous functionality, add useful functionality. Divide the functionality into open source and commercial parts.

Also, if you are using Sidekiq in production at any sort of scale, Sidekiq Pro is excellent.

A microservices implementation retrospective

Actual real-world usage of everyone’s favourite architecural trend. Microservices: not just for Zombie Architectural Astronauts.

Our design used microservices in 3 different roles: 1. Stable interfaces – in front of each application we put a service that exposed a RESTful API for the underlying domain objects. This minimised the amount of coupling between the internals of the application and the other services. 2. Event feeds – each “change” to the domain objects that we cared about within the third party applications was exposed by an event feed service. 3. Synchronisers – the “sync” services ran at regular intervals, reading from the event feeds, then using the stable interfaces to make the appropriate data updates.

revisit.io

Revisit is an awesome minimalist bookmark & discovery tool. Highly recommended. Built with Ember and Elixir too.

Spotify Engineering Culture

Part 2 of an animated video series Spotify’s engineering culture.

MySQL HTTP Plugin

MySQL listens to HTTP and speaks JSON. Boom!

Aphex Twin’s List of Gear Used on Syro

Aphex Twin has a new album after a decade hiatus. Turns out he was just collecting gear. Incredible list. And the album isn’t even that great. But still, Aphex Twin!

ActiveRecord Eager Loading with Query Objects and Decorators

Great pattern for extending Active Record to allow use of Postgres Window Functions.

spec-me-maybe

Are your tests order-dependent? Tired of all those randomly failing specs? Can’t be bothered to use Timecop? Just give up and surrender. But at least use a proper syntax.

Introducing the maybe syntax for RSpec.

maybe(author.name).will eq 'Toby Hede'

Genius.

How platforms attract developers

Some insights into how Docker is conquering the world.

Rich Hickey on Transducers at StrangeLoop

Rich Hickey, the author of Clojure presents a mind-bending dive into Transducers.

People keep implementing map, filter and other fundamental algorithms in different contexts. This talk will describe transducers, a new library feature for Clojure (but of interest to other languages) that emphasizes composable, context-free, intermediate-free notions like 'mapping’ and 'filtering’ and their concrete reuse across all of the contexts above.

I have no idea what it’s about, but it’s awesome.

Optimizing Redis Storage

Some excellent tricks for optimizing Redis.

Redis Cluster is coming very soon, which could be a game changer.

Redis: the swiss army knife of storage.

An Unconventional Review of React

React is so hot right now.

TL;DR: Recommended.

Spaceship!

And last but not least. Spaceships!


The Vibe is an eventually consistent weekly collection of interesting links and … stuff.

It’s Ruby, it’s Rails, it’s programming, it’s databases, it’s the vibe, and, uh … No, that’s it.

It’s the vibe[^vibe]

[^vibe]: Confused? Watch this clip from The Castle and you might understand. Watch the whole film and you definitely will.

Tell us what's what.