David Codes

AWS costs every programmer should know

Sun, 09 Jun 2019 00:00:00 +0000

The title for this blog post is a direct reference to Latency Numbers Every Programmer Should Know. There are several versions of those numbers available now, and I could not find the original author with certainty. Some people attribute the original numbers to Jeff Dean.

When working on a project that will reach a certain scale, you need to balance several concerns. What assumptions am I making and how do I confirm them? How can I get to market quickly? Will my design support the expected scale?

One of the issues associated with scale, is the cost of your infrastructure. Cloud providers allow you to provisions thousands of CPUs and store terabytes of data at the snap of a finger. But that comes at a cost, and what is negligible for a few thousand users might become a burning hole in your budget when you reach millions of users.

In this article, I’m going to list reference numbers I find useful to keep in mind when considering an architecture. Those numbers are not meant for accurate budget estimation. They’re here to help you decide if your design makes sense or is beyond what you could ever afford. So consider the orders of magnitude and relative values rather than the absolute values.

Also consider that your company may get discounts from AWS, and those can make a massive difference.

Compute

What’s the cost of a CPU these days? I used the wonderful ec2instances.info interface to extract the median price of a vCPU.

You can get the source data out of their Github repo. I copied it and processed it using a python script that you can find on Github. All prices are for the eu-west-1 region.

	Median monthly cost
1 modern vCPU (4 AWS ECUs)	58 $/month
With 1 year convertible reservation (all up front)	43 $/month
With 3 years convertible reservation (all up front)	30 $/month
With spot pricing (estimated)	30 $/month

I estimated spot pricing based on anecdotal data I got from various sources. As the prices vary within a day and I could not find a reliable data source for it.

AWS represents the computing power of its machines in Elastic Compute Units, and 4 ECUs represent more or less the power of a modern CPU. So the prices above are for one CPU or core, not one instance.

Here’s the price of 1 ECU in $ per hour across all instance types I looked at:

And here’s how on-demand compares with 1 year and 3 year reservations (both convertible, upfront payments):

Storage

So you want low latency, high throughput and are planning to store everything in Redis? Then on top of those CPU costs, you’ll need to pay for RAM.

I used the same approach to extract the median price of 1GB of RAM on EC2. Elasticache is more or less twice as expensive for on-demand but prices drop quite quickly when looking at reserved instances.

	Median monthly cost
1 GB RAM	10 $/month
1 GB RAM 1 year convertible reservation (all up front)	8 $/month
1 GB RAM 3 years convertible reservation (all up front)	5 $/month
SSD	0.11 $/month
Hard Disk	0.05 $/month
S3	0.02 $/month
S3 Glacier	0.004 $/month

While this is the pure storage cost, you also need to look at the usage patterns for your data. How much CPU will you need to run that in-memory database 24/7?

Same for S3: how much will you pay in writing/reading requests? I’ve seen workloads where the storage cost on S3 was negligible but the cost of writing a lot of objects in S3 made the team write their own filesystem on top of S3.

Bandwidth

A few comments on HackerNews pointed out that I left the bandwidth costs out. Indeed if you are serving data to end users, or need cross-region replication, you need to look into those costs as well.

Type of data transfer	Cost of transferring 1GB
EU/US region to any other region	0.02 $/GB
APAC region to any other region	0.09 $/GB
EU/US region to Internet	0.05 $/GB
APAC region to Internet	0.08 $/GB
Between two AZs in the same region	0.01 $/GB
Inside the same AZ	Free

Advent of Code 2018

Wed, 19 Dec 2018 20:38:00 +0000

As a kid, my parents would buy advent calendars for my siblings and I before Christmas. If you have never seen one, an advent calendar counts down the days from December 1st to Christmas day or Christmas eve.

A typical advent calendar would countain a chocolate per day (bear with me, we’ll be talking about code soon). Although these days you could get anything in those calendars: perfume, protein balls, cereal bars…

This year in addition to that typical advent calendar, I am taking part in Advent Of Code. Created and run by Eric Wastl, Advent of Code has been running since 2015. The website offers a new coding problem each day over the 25 days from December 1st to Christmas.

Each problem follows the same general structure :

a description of the problem, with some background story (usually around helping Santa’s elves to save Christmas)
some examples of input data and expected result
an input file that needs to be parsed and processed to get the answer
a second part for the problem, that you can only access when you’ve solved the first part. Very often, the second part looks like let's make the input data 100 times bigger

Here’s what a problem looks like:

 Below the message, the device shows a sequence of changes in frequency (your puzzle input).
 A value like +6 means the current frequency increases by 6;
 a value like -3 means the current frequency decreases by 3.

For example, if the device displays frequency changes of +1, -2, +3, +1, 
then starting from a frequency of zero, the following changes would occur:

Current frequency  0, change of +1; resulting frequency  1.
Current frequency  1, change of -2; resulting frequency -1.
Current frequency -1, change of +3; resulting frequency  2.
Current frequency  2, change of +1; resulting frequency  3.

In this example, the resulting frequency is 3.

Here are other example situations:

+1, +1, +1 results in  3
+1, +1, -2 results in  0
-1, -2, -3 results in -6

Starting with a frequency of zero, what is the resulting frequency 
after all of the changes in frequency have been applied?

This is the first problem of the advent of code 2018, and also the easiest. Subsequent problems get harder.

The input data is different for each participant. This amazes me. Eric had to come up with an original problem, and then write a generator to provide different input to each participant (70 000 people solved the first problem)!

Clearly Advent of Code is a successful formula. To the point that Eric had to ask people to stop sending spammy requests to track their leaderboard.

Personally I like it because it allows me to play with technologies that I don’t get to use every day. This year I tried to solve it in Scala, although I might finish it in Go. It’s also a good way to review concepts such as algorithmic complexity, breadth-first versus depth-first or Dijkstra’s algorithm.

Reading other people’s point of views on Reddit or on my company Slack, we all have different reasons to take part, and enjoy different types of problems. Some of my colleagues enjoy the parsing of the input file, while like me enjoy when the problem challenges me to think about algorithmic complexity and find clever ways to solve issues with large datasets.

If you’re bored during the winter holidays, or want to try out a new language, do check out Advent of Code. And if you’re curious, my solutions to the problems is on Github.

Postmortems: failure done right

Sun, 05 Mar 2017 10:10:00 +0000

A postmortem is a document in which a team reflects on a recent outage and think of ways to prevent it from happening again. After an outage or security breach, the team gathers and writes a document discussing the event and what they’ve learnt from it.

I am fascinated by these documents and read any postmortem that I encounter. They are a great way for a team to reflect on their practices and improve upon them. Because postmortems address the failure of a system managed by the team, writing them is also an exercise in humility.

Every time I read a postmortem, I think “can something similar happen in my project?”. There is always something to learn from other people’s postmortems. We all have been in situations where we forgot something or made mistakes that we would have never thought of. It is also a good reminder that we are fallible. By reading other postmortems we learn from other people’s mistakes and we reflect on ours.

Some companies use the name RCA, or Root Cause Analysis for such documents. I prefer the term “postmortem”, because RCA sounds like the work stops at identifying a cause. The value of the process is rather in discussing the solutions and lessons learned rather than focusing on the cause of the issue.

How to write a postmortem document

Gather the team

After an outage, gather the team in a room to edit the postmortem together. Make sure you include everyone who was involved in detecting the problem and implementing the solution. If there were a lot of teams involved, get at least a representative of each team.

If you cannot gather everyone together (remote workers, people too busy), you can collaboratively edit a document on Google Docs or something similar.

Describe the outage

Start with an introduction describing in one paragraph what the issue was, what its impact has been, and when it occurred.

Then write up a precise timeline of the events. Include any meaningful even such as:

An alert was received
An action was taken
An action was completed
An engineer made an important realisation on the problem at hand
Something was communicated externally
Impact started/stopped being noticeable by users

Discuss causes and learnings

Once this is done, you have a good basis to discuss causes and learnings. Start by having a look at what went right. Did your escalation process work correctly? Did an engineer follow one of your operations playbooks with success? If so, do mention it here. It is an acknowledgement of how much the team has already accomplished to improve the system’s resiliency.

Then have a look at what did not go well. Explore each issue, and look for the causes. What made things difficult? Why did this system fail, or why was this alert ignored? There is usually more than one cause, so do not stop at the first one you find. Iterate on those causes to get a better understanding of what caused the problem. Use a technique such as 5 Why’s to achieve this.

Agree on actions

Finally, list out all the actions that need to be taken to prevent similar problems from occurring. You will probably want to patch your code to fix the issue you found, but you should consider other actions, for example:

Improving documentation on solving this kind of issue
Adding an alert to surface an issue earlier
Improving the incident response process if necessary
Communicating the postmortem to other teams so that they learn from it
Reviewing the system for similar vulnerabilities that would not have been found yet
Planning for a repetition of the incident to ensure that the steps taken work

Like any action plan, name an owner for each of the actions, to make sure they’re dealt with. You should also agree on a date to check on those actions’ progress.

What is a good postmortem?

Precise

The most important quality of a good postmortem is being factual. In particular, the timeline should be as precise as possible and describe all the events from the first symptoms to the resolution. Be concise and stay away from vagueness. Take a statement such as:

the latency increased after deploying the new version

It should be replaced with a more precise version, such as:

the 99th percentile latency grew from 300ms to 1.2s within 3 minutes of the deployment, and stayed that way during 12 minutes before going down to the nominal value of 300-400ms

Blameless

Second, a postmortem is blameless. It does not judge, it describes the events and the actions taken to solve it. If a postmortem is not blameless, people cannot be candid about the errors they made. With a blameful postmortem, you will not reach the right conclusion. This is a tricky exercise. By the time the outage is over, people are exhausted. To make sure that a postmortem does not blame anybody, it is better to write it after everyone had a good night’s sleep.

Facilitated

An external facilitator can be very helpful. They will gather the facts and ask questions to people involved in the outage. This will result in a more factual, blameless postmortem. The worst option is for the manager of the people involved to lead the postmortem meeting! For a lot of people, it is harder to discuss openly a mistake they made in front of the person who assesses their performance and will decide on their salary bump in a couple of months. Postmortem meetings lead by managers will always be less transparent that ones led by independent facilitators.

Constructive

Postmortems and their quality are a reflection of your team and company’s dynamic. A team who feels safe owning their mistakes will write candid, transparent postmortems. A team worried about deflecting blame and securing their next performance review will not. Teams where members do not feel safe to speak their mind will provide a biased version of the postmortem and will avoid addressing the actual causes. Fix the team issues, and the postmortems will improve by themselves.

Conclusion

Today, more companies are transparent and publish their postmortems on their blog. This is an extraordinary occasion to see the challenges faced by other engineers. You can see how Google, Facebook, Microsoft, or smaller startups address issues. We should acclaim companies and engineers who publish such documents. It takes a lot of humility to admit failure and expose mistakes for the world to see.

The resource section below contains examples of interesting postmortems published by famous companies. Take the time to read a couple of these, they read like a thriller and I guarantee you will learn a lot!

I hope this article was helpful to you. Do get in touch at @dhatanian if you want to discuss systems resiliency or share interesting postmortems with me!

Some examples of postmortems

Additional resources on postmortems:

What I wish I knew when I started as a software developer

Mon, 03 Oct 2016 10:10:00 +0000

Having worked in the software engineering field for 6 years, I sometimes pause and consider how much I have learned since I started my career. One thing that strikes me is how many things I would want to tell just-out-of-college me if I had the occasion.

While I don’t have that capability, I hope that the few items below will be useful to people on their way out of college and into the software industry.

Keep on learning

You just obtained your diploma, are overwhelmed by a sense of accomplishment and now feel the desire to prove yourself to your new colleagues? That is great, congratulations and good luck, but keep in mind that this is just the beginning of your learning.

You will find soon that you need to keep learning new technologies, new methods, new techniques, new languages. Start as soon as possible, because learning in an unsupervised way (no syllabus, no teacher, no final exam) is a skill that you need to hone.

Learn things outside your comfort zone. If you work on a dynamic language, learn a statically typed one. Use VIM or Emacs if you are used to an IDE, etc.

Find a mentor

I just said that there are no teachers after college, but that is not entirely true. Nobody will sit in a classroom with you and go over a predefined lesson. But it is still extremely beneficial to find one more experienced person that can guide you.

Sandro’s book “The Software Craftsman” describes how a mentor is beneficial to a new software developer at great lengths.

Depending on the company you are working with, you might be provided with a mentor, or someone will come to you and spontaneously offer their help. Sometimes you will need to look for them. If you cannot find them amongst your colleagues, look for them in community events for example.

Ask questions

I did not know what a framework was when I got out of college. Looking up the definition of Wikipedia did not help either, and for several long months I was too afraid to show my ignorance and ask.

Do not repeat my mistake. You are not expected to arrive at your first job knowing everything. Ask, don’t be afraid to show you do not know something.

Most of the time, you will find out that either at least one other person in the team had the same question, or people have slightly different definitions of the term you are asking about. Even if you are the only one asking the question, you will not be the only one learning something in return.

Get good practices early

These good practices will be the foundation upon which you will build your career. Get them as early as possible.

Leverage existing libraries and frameworks

I barely knew how to properly use a library when I got out of college. As I said earlier I did not even know what a framework was.

As juniors, we have a tendency to re-invent the wheel. Our custom-made solutions will usually be of less quality than a library on which hundreds people have collaborated over several years. Try to re-use what is already available.

On the other hand, understand what are the tradeoffs of a given library. Is it still maintained? It is easy to use? Does it require huge changes in your application’s architecture? Make sure that you isolate that library, so that replacing it does not mean making changes all over your code.

But still understand the fundamentals

Libraries are convenient, because they remove complexity. However, take the time to understand a bit of how they work under the hood.

While your web framework will handle HTTP requests for you, you need to get what HTTP is and how to conveys information between two computers. Similarly, take the time to understand the basis of DNS, TLS, and other low-level protocols. If your application uses OAuth or session-based authentication, you need to know it and you need to know what the consequences are. This page is a good starting point.

Assume the mistake is yours

You will face unexplainable bugs. Sometimes even heisenbugs, that disappear when you debug them. In despair, you might be tempted to reject the fault to something else than your code. Always start with the assumption that it is an issue with your code, not the library, the OS or anything else.

This is a lesson in humility. There is a much higher chance that your barely tested, 2 days old code is buggy than a battle-hardened library that has been around for years. As your debugging skills and your bug-hunter instinct improve over time, you will come to that conclusion anyway.

Sometimes, it is indeed a bug in another software component than yours. However flagrant the bug, however careless the offending code seemed, remember of how many bugs your code was littered to being with, and don’t hold it against the authors of the faulty code.

Test, test, test

I think that automated testing and test-driven development are not practiced enough in software engineering courses. Those are amongst the most important skills in software development today, and I wish I was made aware of that earlier in my career.

Learn how to write tests and how to maintain them. Pair with someone who can teach you test driven development. If you cannot find someone like that amongst your colleagues look at a software craftsmanship community next to where you live.

Clean code principles

Learn how to write clean code from the start. The book by Robert C Martin is an excellent way to start. There is also plenty of content online (just check other articles on the Codurance blog). Take the time to read some of that content and apply it. Ask your mentor or colleagues for feedback.

Depending how much programming practice you’ve had before starting your career, you might still believe that a programmer’s day is spent sitting alone in front of their editor. You need to understand as soon as possible how much you win by working with others.

Pair programming

This is a very important skill to develop. Get used to coding with someone else, passing the keyboard from one person to the other from time to time. Practice this both with people more skilled than you, and with beginners.

Being good at writing code is not even half the job

You need to learn how to work with users or Product Owners, ask questions about features and manage expectations.

You will discover that sometimes, the answer will not be to add more code to your project: either the feature is not needed, or it can be implemented with the current version of the app, or there is another application that provides 80% of the functionality.

Don’t be “clever”

Being clever is rewarded in college. You may have been encouraged to optimize a for loop, or write one less line of code by replacing i=i+1 with i++ in the line before.

Being creative and clever to solve a hard, complex problem is good. But do not use clever micro-optimizations that make your code less readable.

One of the greatest challenges as a beginner software engineer is to make your code readable. Your teammates need to be able to understand your code and evolve it without your help. Clever optimizations often go against that by making the code’s intent less obvious. Unlearn this “academic” reflex.

But understand it is a tradeoff, and readable code is more important that clever code. Boring code is good.

We are hiring software craftspeople and apprentices! More information here.

Isolating integration tests and mocking dependencies with Spring Boot

Sun, 31 Jan 2016 00:00:00 +0000

Integration tests can be slow and unreliable because they depend on too many components in the system. Up to a certain point, this is unavoidable: integration tests are here to validate how each part of your system plays with other internal or external components.

We can, however, improve some integration tests by only spinning up the required dependencies, instead of the whole system. Let’s imagine an application that depends on a database, a third-party REST API and a message queue:

Assume now that we would like our integration test to validate a behavior that only includes calls to the REST API but no call to the database or the message queue. To give a concrete example, let’s assume we want to check that our REST client is correctly configured to time out after 3 seconds.

All we need for this is a small Controller that will mock the REST API by waiting before returning an answer to the REST client. The wait time will be passed as a parameter in the query string.

@Profile("restTemplateTimeout")
@RestController
@RequestMapping(value = "/test")
public class DelayedWebServerController {

  @RequestMapping(value = "/delayRestTemplate", method = GET)
  public String answerWithDelay(@RequestParam Long waitTimeMs) {

    if (waitTimeMs > 0) {
      try {
        Thread.sleep(waitTimeMs);
      } catch (InterruptedException e) {
        throw new RuntimeException(e);
      }
    }

    return "Delayed Result";
  }

}

What is the @Profile annotation used for? If we inject this controller into our standard application context, this has several drawbacks:

The test will be slow: we only need to start one controller, not the whole thing
Our controller will be picked up by Spring and injected into every other integration test, slowing down each integration test and maybe stepping on another test’s toes

A better alternative would be to spin up a minimal Spring Boot application exposing only our DelayedWebServerController. We will also tell Spring Boot to scan only the packages we are interested in, and to exclude persistence-related auto-configuration since we do not need it to spin up a controller. This is done in a Configuration class like this one:

@Profile("restTemplateTimeout")
@Configuration
@EnableAutoConfiguration(
    exclude = {DataSourceAutoConfiguration.class, HibernateJpaAutoConfiguration.class})
@ComponentScan(basePackages = "my.application.resttemplate.timeout")
public class DelayedWebServerConfiguration {
    //The class is empty and only used to support the annotations
}

The Spring context configuration can get quite confusing, let us look at the annotations one after the other:

@Profile: This tells Spring that this configuration should only be used then the restTemplateTimeout profile is active. Further in this article, we will see how we enable this profile for a specific integration test. It is this annotation that prevents the configuration to be picked up by other unrelated integration tests. Note that our DelayedWebServerController is identically annotated.
@Configuration: Standard annotation to tell Spring that this is a context configuration class.
@EnableAutoConfiguration: Here we disable some of the Spring Boot “magic” that we do not need for our specific test
@ComponentScan: We speed up the Spring Boot application startup by only scanning one package instead of the whole project. Any Spring-annotated class that is outside of this package will not be picked up by Spring.

Here is how the integration test looks like:

@RunWith(SpringJUnit4ClassRunner.class)
@WebIntegrationTest("server.port:0")
@SpringApplicationConfiguration(classes = DelayedWebServerConfiguration.class)
@ActiveProfiles("restTemplateTimeout")
public class RestTemplateShould {

  @Rule
  public ExpectedException thrown = none();

  @Value("${local.server.port}")
  private int port;

  @Autowired
  private RestTemplate restTemplate;

  @Test
  public void throw_timeout_if_response_lasts_more_than_two_seconds() {
    thrown.expect(ResourceAccessException.class);
    thrown.expectCause(instanceOf(SocketTimeoutException.class));

    callEndpointWithDelay(3000);
  }

  @Test
  public void do_not_throw_timeout_if_response_lasts_less_than_two_seconds() {
    callEndpointWithDelay(10);
  }

  private void callEndpointWithDelay(long delayMs) {
    restTemplate.getForObject(
        "http://localhost:" + port + "/test/delayRestTemplate?waitTimeMs=" + delayMs, String.class);
  }
}

Of course, all those classes are stored in our test source folder (usually src/test/java) since they are not required for production.

Let us have a look again at the annotations:

@RunWith: The test will use the Spring Junit runner who will take care of creating the Spring context for us.
@WebIntegrationTest: Tells Spring that this is an integration test running a web application, otherwise by default Spring will not run an HTTP server in test mode. We also set the server.port to a value of 0 so that Spring Boot choose a random port for the HTTP server to listen to. This allows to have several tests running in parallel, or to have another version of the application running in the background.
@SpringApplicationConfiguration: We tell Spring where it will find the DelayedWebServerConfiguration class we created before.
@ActiveProfiles: Enables the restTemplateTimeout profile, otherwise the Controller and the Configuration will be filtered out.

We now have an integration test running with a limited set of dependencies instead of the whole application. What if we wanted to go further and add mocks into the game? This may be required when a dependency does not have a dev environment or that it is too complicated to call from a developer’s workstation. In that case, we can add those mocks to the Configuration class and they will be injected into the test’s Spring context.

Here is a Configuration example where we inject a custom CustomerService mocked by Mockito instead of the default one:

@Profile("validationTests")
@Configuration
@EnableAutoConfiguration(
    exclude = {DataSourceAutoConfiguration.class, HibernateJpaAutoConfiguration.class})
@ComponentScan(basePackages = {"my.application.controller",
    "my.application.actions"})
public class ValidationEndToEndConfiguration {
    @Bean
  public CustomerService customerService() {
    return Mockito.mock(CustomerService.class);
  }
}

With this approach, we can make our integration tests more resilient. For slow or unreliable dependencies, it is more efficient to have the developers run their integration tests against a mocked version. However, do not forget that in the end your application will have to integrate with the real system, not the mocked one. For this reason, it makes sense to have the continuous integration server run the tests against the real system at the very least every day.

New job at Codurance

Fri, 20 Nov 2015 00:00:00 +0000

I started on October 2015 a new job at Codurance in London. Codurance is a software consultancy centered around the principles of Software Craftsmanship.

The co-founders are Sandro Mancuso and Mashooq Badar, both very active in the Software Craftsmanship community. Sandro even wrote a book, The Software Craftsman : Professionalism, Pragmatism and Pride. It highlights his views on how a good software engineer should take his career in his hands. I highly recommend it to anyone who works in software engineering.

I first heard of Codurance through Sandro’s conference at Devoxx France 2014. I then read his book and found that a lot of the principles embodied in his view of the software craftsman resonated very strongly in me. I applied during the summer to work at Codurance and went through the hiring process.

I was already very happy to receive an email from Sandro and Mashooq saying that my résumé had passed the first filter and that I was invited to write a simple program to show my development skills. I wrote the program, paying as much attention as I could to tests and design, and submitted my code. Two weeks later, I received a review of my code, along with an invitation for a Skype call with Mashooq and Sandro. Happy, I proceeded to read the code review, and I became blank : the review had 4 positive items and 25 items were listed in the “improvements” section. I honestly thought my submission was the best work I could have done, and they found 25 things that I missed ! Even worse, I could not find a single improvement I disagreed with.

Clearly, the bar at Codurance was higher than I expected, and I dreaded that the Skype interview would show that I did not have the required skills. One week later, I had this Skype call with Mash and Sandro, and they made me immediately comfortable. We had a talk about design principles, they asked questions about SOLID and checked if I understood all their feedback. I was a bit stressed but eventually the interview went well, and I was extremely happy to receive an offer from Codurance later in the day.

I have now been working at Codurace for one month, and I can say the atmosphere is really different there. People here are passionate about software and good design, they take pleasure in sharing their findings with their colleagues and we gather quite often to talk about anything related to software. I do not spend a day without learning something new.

Codurance is a new step for me, and I’m proud to be part of such a great company.

Configuring an enterprise app sucks

Thu, 19 Nov 2015 09:00:00 +0000

There has been quite a lot of discussion over the last years about how configuration should be managed as code, be it software settings or infrastructure configuration:

For infrastructure, all the cool kids now use Docker, Vagrant or some other tool that allows you to store your infra’s configuration as text files. You can find more on this blog post
For software settings, it is now commonly accepted that anything that is not specific to the environment (secrets, IP addresses, ports) should be stored along with the code in the source code repository. Some teams even store environment-specific settings in the same repository.

But when one configures an enterprise application, be it an open-source one or something purchased from a vendor, or even a Software As A Service solution, one is guaranteed to end up with all those levers, buttons and knobs to adjust and fiddle with. But that makes sense, enterprise applications need to be tweaked according to each company’s specificities. So the best option is to shove all the configuration possibilities in an maze of screens, tabs, checkboxes and helpful tooltips, right?

I think that is wrong. This way of configuring software is a guarantee that something bad will happen at some point. Even collaboration tools such as Google Apps, who thrive on usability, provide administrators with overly complex settings options hidden deep withing a web interface. Identity and Access management tools such as OpenAM, IBM Tivoli Identity Manager and others suffer from the same plague.

Here’s what the advanced configuration screen for Google Apps looks like. All those tabs and options are only for the GMail section, which gives you an idea of the overall complexity:

And this how you configure a token-based authentication on OpenAM. An admin needs to go through dozens of those screens filled with fields and not make a single mistake. We are talking about an application that centralizes authentication, mistakes can make all the applications in your information system inaccessible:

Those configuration screens are wrong on a lot of levels:

It is hard to look for a specific setting. You often have to search several screens and sections, each of those potentially relevant to your subject. Some configuration interfaces provide a search tool to allow you to find the screen where you can configure a given setting, which gives you an idea of how hard it can be to navigate the interface in the first place.
Keeping track of the changes performed on the configuration is complicated. Any serious enterprise application will provide an audit track, unfortunately the track is usually a log of atomic events such as “Administrator Michael changed setting X to value A”. There rarely is a way to have an overview of the complete configuration at the moment of this change. Did this change make sense at the time? If some setting Y was dependent on the setting X, was Y also correctly set?
For any use case but the most trivials, the administrators of those applications will want to setup and maintain several environments. At the very least there will be one pre-production environment where admins test new settings before enabling them in production. Usually, the configuration is done in the test environment through a mazelike user interface, and each step is documented along with screenshots in a change document. Once the change is validated (i.e. when the business is happy with the feature), the change document is used to reproduce the same steps in production. This means that:
- It is hard to keep environments in sync. If an environment’s configuration drifts from the configuration, there’s no easy way to detect it, and it can invalidate your future tests.
- The process of moving from one environment to another is error prone. An admin who fails to check a box to check will break production without knowing it.
- The process is slow. Updating all the settings for a single change can take from several minutes up to hours for very complex change. During that time, the application is in some kind of untested, probably-still-works state. Or it was taken offline for the time of the maintenance, which means that user are now blocked and waiting for the change to complete.
- Since the process is error prone, you can be sure that at some point, something will go wrong and you will need to rollback your change. Guess what? Rolling back is at least as slow as performing the change, if not worst. So the application remains available for a longer time.

We need to configure our enterprise applications the same way we manage other software settings or modern infrastructure: we need to move from an imperative configuration administration composed of manual steps like this:

Change setting A to value X

to a declarative style that basically says:

Here's a complete description of the configuration I expect, now make it happen

The good news is that we already have the tools for this: uploading the configuration as a text file in a simple human-readable format such as json or yaml would be enough. This solves all the issues we’ve seen:

Looking for a specific setting? Use the search feature of your preferred text editor, or even the grep command line tool!
The application can let administrators track changes on its configuration by providing a source control system similar to Git. You would know who changed a given line, when, and could in one click have an immediate view of the resulting configuration after this change. Here is how a change on a configuration file would be viewed in GitHub:

Moving from one environment to another becomes trivial: it is a matter of transferring the configuration file to the new environment, which is much faster and less error prone. Since the settings history is kept, rolling back to the previous state becomes trivial: just tell the application to revert back to the previous version of the settings.
As a plus, your configuration is easy to export and backup. This would allow competing platforms to let you import the configuration from your previous solution when you migrate. Want to migrate from CloudFileServer™ to SAASEnterpriseFileManager™? Just import your old configuration and be ready to start in minutes!

As software engineers, we’ve progressed a lot in the field of ad hoc developments, we have industrialized the way we deploy our applications and manage our servers, we keep most of the configuration safely stored and tracked in source repositories. But when we provide platforms to our customers in the form of enterprise and/or SAAS applications, we still let then manage those like we’re in the 90’s. It is time to provide them with a proper way to manage their IT solutions.

Falsehood articles - everything you have been told is a lie

Thu, 30 Jul 2015 00:00:00 +0000

I wrote last year an article about dates, and how complicated it can get. I have updated it recently to included two excellent articles from Noah Sussman, a tester at Etsy who explains “falsehoods” about dates : thing we might hold for true, but that aren’t. For example :

Years have 365 days.

Yeah this one is easy, we have leap years with one more day.

A week (or a month) always begins and ends in the same year.

OK, another obvious one, but I’ve seen this error before

Human-readable dates can be specified in universally understood formats such as 05/07/11

This one is interesting. In France it would translate as the 5th of July while in the US it would be the 7th of May.

One minute on the system clock has exactly the same duration as one minute on any other clock

Wrong, as can be expected

Ok, but the duration of one minute on the system clock will be pretty close to the duration of one minute on most other clocks.

There are cases where clocks can go really adrift. But it’s weird to imagine that even the duration of a minute could be very different from one clock to another. But the coup de grâce comes right after.

Fine, but the duration of one minute on the system clock would never be more than an hour.

This one puzzled me. I couldn’t imagine how a clock would suddenly make minutes last hours. Or, as Noah writes it :

You can’t be serious.

He then gives an example of a clock for which some minutes would actually last hours :

There was a fascinating bug in older versions of KVM on CentOS. Specifically, a KVM virtual machine had no awareness that it was not running on physical hardware. This meant that if the host OS put the VM into a suspended state, the virtualized system clock would retain the time that it had had when it was suspended. E.g. if the VM was suspended at 13:00 and then brought back to an active state two hours later (at 15:00), the system clock on the VM would still reflect a local time of 13:00. The result was that every time a KVM VM went idle, the host OS would put it into a suspended state and the VM’s system clock would start to drift away from reality, sometimes by a large margin depending on how long the VM had remained idle.

There was a cron job that could be installed to keep the virtualized system clock in line with the host OS’s hardware clock. But it was easy to forget to do this on new VMs and failure to do so led to much hilarity. The bug has been fixed in more recent versions.

Funny right ? I find that kind of stuff very interesting. There are other such falsehood posts on other subjects, which I recommend reading :

I think this is the oldest one, Patrick McKenzie’s Falsehoods Programmers Believe About Names
The one which is quoted at the beginning of this post, Noah Sussman’s Falsehoods programmers believe about time
A lot of people wanted to contribute to this list, so Noah compiled their falsehoods in a second post : More falsehoods programmers believe about time; “wisdom of the crowd” edition
Matthias Wesmann’s Falsehoods programmers believe about geography
C. Scyphers’ Falsehoods Programmers Believe About Gender

Using HTTP sessions with Google Cloud Endpoints

Sat, 06 Jun 2015 00:00:00 +0000

I like how easy Google Cloud Endpoints makes it to create a REST API on Google App Engine. Granted, there are other solutions in the Java world (I also used RestX and Spring) but I still like the simplicity of Google Cloud Endpoints.

However, one problem with Cloud Endpoints is the authentication : by default it only supports Google’s OAuth2 authentication protocol. Because my application uses sessions for other purposes, I wanted to authenticate the user based on his session. Endpoints authentication is actually extensible through an almost undocumented class called Authenticator.

public interface Authenticator {
  User authenticate(HttpServletRequest request);
}

Another issue is that this authenticator does not have access to the session either… The Endpoints infrastructure strips off the cookies and session information before injecting the HttpServletRequest in Authenticator.authenticate().

The solution I found is to send the session id in a header. In App Engine, sessions are stored in memcache and the datastore, so I can extract those sessions easily. Here’s how it works :

public class OAuth2EndpointsAuthenticator implements Authenticator {
    private static final Logger log = Logger.getLogger(OAuth2EndpointsAuthenticator.class.getName());
    private DatastoreService datastoreService = DatastoreServiceFactory.getDatastoreService();
    private MemcacheService memcacheService = MemcacheServiceFactory.getMemcacheService();

    @Override
    public User authenticate(HttpServletRequest httpServletRequest) {
        HttpSession session = httpServletRequest.getSession(false);
        if (session == null) {
            String jSessionId = httpServletRequest.getHeader("X-MYAPP-JSESSIONID");
            if (jSessionId == null) {
                return null;
            }
            MyAppUser user = getUserFromSessionInDatastore(jSessionId);
            return toEndpointsUser(user);
        } else {
            Object user = session.getAttribute(OAuth2UserService.USER_SESSION_PROPERTY);
            if (user != null && user instanceof MyAppUser) {
                return toEndpointsUser((MyAppUser) user);
            } else {
                return null;
            }
        }
    }

    private MyAppUser getUserFromSessionInDatastore(String jSessionId) {
        String entityId = "_ahs" + jSessionId;
        String memcacheId = "cacheduser/" + entityId;
        SessionDataInMemcache sessionDataInMemcache = (SessionDataInMemcache) memcacheService.get(memcacheId);
        if (sessionDataInMemcache == null || sessionDataInMemcache.isExpired()) {
            Entity session;
            try {
                session = datastoreService.get(KeyFactory.createKey("_ah_SESSION", entityId));
            } catch (EntityNotFoundException e) {
                return null;
            }
            Long expires = (Long) session.getProperty("_expires");
            if (expires < System.currentTimeMillis()) {
                return null;
            }
            Blob values = (Blob) session.getProperty("_values");
            Map valuesMap;
            try (ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(values.getBytes())) {
                try (ObjectInputStream objectInputStream = new ObjectInputStream(byteArrayInputStream)) {
                    valuesMap = (Map) objectInputStream.readObject();
                } catch (ClassNotFoundException e) {
                    throw new RuntimeException(e);
                }
            } catch (IOException e) {
                throw new RuntimeException(e);
            }
            MyAppUser user = (MyAppUser) valuesMap.get(OAuth2UserService.USER_SESSION_PROPERTY);
            sessionDataInMemcache = new SessionDataInMemcache();
            sessionDataInMemcache.setExpirationTime(expires);
            sessionDataInMemcache.setUser(user);
            memcacheService.put(memcacheId, sessionDataInMemcache);
            return user;
        } else {
            return sessionDataInMemcache.getUser();
        }
    }

    public static class EndpointUser extends User {

        public EndpointUser(String email) {
            super(email);
        }

		//You can put here additional attributes if you wish to

    }

	private User toEndpointsUser(MyAppUser currentUser) {
        if (currentUser == null) {
            return null;
        }
        EndpointUser user = new EndpointUser(currentUser.getEmail());
        //If you have extra attributes in your session, put them here
        return user;
    }

	public static class SessionDataInMemcache implements Serializable {
	    private User user;
	    private long expirationTime;

	    public User getUser() {
	        return user;
	    }

	    public void setUser(User user) {
	        this.user = user;
	    }

	    public long getExpirationTime() {
	        return expirationTime;
	    }

	    public void setExpirationTime(long expirationTime) {
	        this.expirationTime = expirationTime;
	    }

	    public boolean isExpired() {
	        return expirationTime <= System.currentTimeMillis();
	    }
	}
}

In this code, the Authenticator gets the session id from the X-MYAPP-JSESSIONID HTTP header. It then tries to fetch and deserialize the session info from memcache. If this fails, the session is retrieved from the datastore.

Important : this only works correctly if you invalidate the session when the user logs out and then logs back in. If one given session can be associated with more than one MyAppUser, or if the MyAppUser object changes during the session’s life (for example if you update attributes) then the SessionDataInMemcache object will be out of sync. This was not a problem for my application.

Then all I need to do is configure jQuery to include the session ID as a header when it makes AJAX calls, here’s how it’s done :

var sessionId = Cookies.get('JSESSIONID');
$.ajaxSetup({headers: {"X-MYAPP-JSESSIONID": sessionId}})

The Cookies class here is provided by a small Cookies.js library.

All we have to do now is to configure our Cloud Endpoints API. Note the authenticators parameter on the @Api annotation.

@Api(name = "myapi", version = "v1", authenticators = {OAuth2EndpointsAuthenticator.class})
public class LandingPage {
    @ApiMethod
    public HelloMessage sayHello(User endpointUser) throws ForbiddenException {
        OAuth2EndpointsAuthenticator.EndpointUser user = checkUser(endpointUser);
        return new HelloMessage("Hello, "+user.getEmail());
    }

    private OAuth2EndpointsAuthenticator.EndpointUser checkUser(User endpointUser) throws ForbiddenException {
        if (endpointUser == null) {
            throw new ForbiddenException("User must be authenticated");
        }
        if (!(endpointUser instanceof OAuth2EndpointsAuthenticator.EndpointUser)) {
            throw new ForbiddenException("Expected user with opco information");
        }
        return (OAuth2EndpointsAuthenticator.EndpointUser) endpointUser;
    }
}

That’s it ! We are now ready to roll with a Google Cloud Endpoints service protected by Java’s standard HTTP sessions.

New blog on Jekyll and Github pages

Sat, 16 May 2015 00:00:00 +0000

I finally got tired of Blogger’s very slow and overcomplicated interface, so when I bought the hatanian.com domain name I decided to also migrate to something lighter.

Setting up

I wanted something that would accept markdown, display code nicely, accept git versioning. Jekyll was a great fit, and since it works well on Github Pages, well I got hosting for free !

I won’t explain how Jekyll and Github Pages work together, since Github has a very clear tutorial already.

By now, my new and empty Jekyll blog was hosted on dhatanian.github.io which is the default domain for Github Pages. I wanted it to be hosted on my new hatanian.com domain, so I first followed Github’s instructions on how to use a custom domain. Then it hit me : how will I add TLS, since this is not supported by Github Pages ?

Luckily Cloudflare has a free plan that includes TLS (which they call SSL, weirdly). This was really a two-clicks setup, Cloudflare is perfectly integrated with Godaddy and in a few minutes I had a blog serving its (still empty) content over HTTPS. It is not perfect as the connection is only encrypted between the browser and Cloudflare, but not between Cloudflare and Github Pages. However it’s good enough for a private blog !

It felt a bit strange to simply give Cloudflare access to my DNS settings on Godaddy and see that they were able to generate a certificate for my domain. Then again it’s logical : a lot of certificate authorities check the domain ownership through the DNS entries.

First tweaks

Anyway I now had a fresh new blog that I needed to fill in. I fixed up the default theme a little bit by jutifying the text by default and moving the navigation bar to the left of the screen.

Jekyll’s default index.html is the list of blog posts. I don’t think that’s a great landing page and I prefer people to arrive on my latest post, so I used this great hack by nimbupani to get it right :

Migrating Blogger’s content

This was quite a piece of cake. Blogger allows you to export a XML file that contains all your blog posts and the associated comments (we’ll talk about comments later).

I then used Jekyll’s Blogger import gem, and it generated each blog post in html format, even keeping some code formatting.

The <img> tags where still linking to Blogger though, so I downloaded them manually (there were’nt so many) and put them on Github Pages along with the articles.

For some reason, Blogger set fixed width and height attributes for the images. This resulting in weird-looking image that were shrinked when viewed on small screens. Removing the width and height attributes from the <img> tags fixed the issue.

Setting up a workflow

When I started working with images, I was faced with a problem : where do I put images associated with a given blog post ? I would prefer to store them next to the post, so I started creating dedicated folders for each post, with each time the html/md file along with the associated images. But Jekyll started to complain :

Error:  invalid byte sequence in UTF8

Jekyll actually tries to process every file that is in the _posts folder, even ones that are clearly binaries. There is no way do define exclusions. The Jekyll documentation recommends that images and other assets be placed in a dedicated assets directory, but I did not want images from all my posts mixed together.

I eventually settled for an intermediate solution : each post will have an associated folder in the assets directory. My folder structure now looks like this :

\david-codes
|--\_posts
|  |--- 2015-05-16-new-blog-on-jekyll-and-github-pages.md
|
|--\assets
   |--\posts
      |--\2015-05-16-new-blog-on-jekyll-and-github-pages.md
	     |--- image.png

It’s not super convenient since images and posts are separated, but at least it’s easy to find all the images for a given post.

Inside the post, a link to an image would look like this :

<img src="assets/posts/2015-05-16-new-blog-on-jekyll-and-github-pages.md/image.png"/>

Which obviously is not really convenient to type and depends too much on the article’s name. So I used a liquid tag :

<img src="/assets/{{ page.path | replace:'\_posts','posts' }}/image.png"/>

(ignore the \ character which is here to fix a very annoying issue in the atom editor)

I then created an atom snippet, that’s a piece of code that I can generate quickly. The Cson configuration is like this :

'.text.html, .source.gfm':
  'jekyll.asset':
    'prefix': 'img'
    'body': '/assets/ {{ page.path | replace:\'\_posts\',\'posts\' }} /${1:image}.png$2'

Now when I type img-TAB the full liquid tag is automatically generated for me.

Finally, I wrote a quick shell script to create an article and the associated assets folder :

title=$1
slug=`echo $title | tr '[:upper:]' '[:lower:]' | sed -E 's/[^a-z0-9]+/-/g'`
date_slug=`date +"%Y-%m-%d"`
file="$date_slug-$slug.md"

echo "Creating post $file"
post_file="\_posts/$file"
touch post_file
mkdir assets/posts/$file
echo "---" >> $post_file
echo "layout: post" >> $post_file
echo "title: $title" >> $post_file
echo "author: David Hatanian" >> $post_file
echo "---" >> $post_file
echo "" >> $post_file

atom $post_file

Adding comments

Adding comments is absolutely trivial with Disqus. This free service generates by default a comment thread per URL of your blog. You just have to insert some HTML and Javascript at the end of your page.

Because my posts can have several URLs (the latest post is also the root of the site), I slightly modified the Disqus initialization code to include a custom identifier, which is the post id extracted by a Liquide tag :

<div id="disqus_thread"></div>
<script type="text/javascript">

var disqus_shortname = 'david-codes';
var disqus_identifier = '{{ page.id }}'; //Here's the trick

(function() {
	var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
	dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js';
	(document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
})();
</script>

Now I needed to import comments from Blogger to Disqus. Disqus has a very nice import system but it required comments in the wxr format (from Wordpress). Fortunately, a kind soul wrote a Google App Engine appengine blogger2wordpress service to convert the Blogger format to wxr.

The first import went well, but because the URLs had changed from Blogger to Github Pages, Disqus wouldn’t load the old comment threads in my blog. I reuploaded the wxr after manually specifying the disqus_identifier on each thread :

<item>
  <title>How to provide seamless Single Sign On on Google Apps</title>
  <link>http://david-codes.blogspot.com/2014/07/how-to-provide-seamless-single-sign-on.html</link>
  <!-- Add this to identify the Disqus thread -->
  <dsq:thread_identifier>/2014/07/03/how-to-provide-seamless-single-sign-on</dsq:thread_identifier>
  <pubDate>Thu, 03 Jul 2014 04:18:00 -0700</pubDate>
  <dc:creator>David Hatanian</dc:creator>

After that, I had a nice commenting interface, with my old comments loaded !

Redirecting from Blogger to Github Pages

This post by Suriya Prakash explains well how to setup a redirection from Blogger.

I adapted it that way :

<b:if cond='!data:mobile'>
    <!-- posts -->
    <div class='blog-posts hfeed'>

      <b:include data='top' name='status-message'/>

      <data:defaultAdStart/>
      <b:loop values='data:posts' var='post'>
		<b:if cond='data:numPosts != 1'>
          <script>location.href=&quot;https://david-codes.hatanian.com/allposts&quot;</script>
        </b:if>
       <script>
           var d=&#39;<data:blog.url/>&#39;;
           d=d.replace(/.*\/\/[^\/]*/, &#39;&#39;);
			var postPage = d.replace(/\/201[3-5]\/[0-1][0-9]\//,&#39;&#39;);
           var timestamp=&#39;<data:post.timestampISO8601/>&#39;;
         console.log(&quot;Post page : &quot;+postPage);
         console.log(&quot;Timestamp : &quot;+timestamp);
		var dateUrl = timestamp.replace(/T.+/,&#39;&#39;).replace(/-/g,&#39;/&#39;);
         var redirectUrl = &quot;https://david-codes.hatanian.com/&quot;+dateUrl+&quot;/&quot;+postPage;
		location.href = redirectUrl

		</script>

I also had to :

disable the mobile-specific theme
change my theme to the usual static one because otherwise it would always redirect to the latest post. Note : change the theme before inserting custom code.

One small issue was that the post files name were not exactly right after migration from Blogger. In Jekyll, a post starts with a date in YYYY-mm-dd format, for example this post is :

2015-05-16-new-blog-on-jekyll-and-github-pages.md

This post has the following URL :

2015/05/16/new-blog-on-jekyll-and-github-pages.md

But for some posts the date prefix was incorrect. For example the file name would be :

2014-07-22-google-apis-checking-scopes-contained.html

But the URL would be off by one day :

2014/07/21/google-apis-checking-scopes-contained.html

This was not on all posts. Because I mainly blogged from Vietnam and India, I guess this was a timezone issue. The solution was to remove the date header from the migrated articles :

---
layout: post
title: '"Cloud Platform Orchestrator" - An App Engine/Compute Engine demo'
date: '2014-05-06T05:05:00.004-07:00' --> REMOVE THIS
author: David Hatanian
tags:
modified_time: '2014-05-06T05:05:55.659-07:00'
thumbnail: http://4.bp.blogspot.com/--WuW40qT3vM/U2jQBN5c0lI/AAAAAAAAInU/BIKePHu2JTk/s72-c/createexecution.png
blogger_id: tag:blogger.com,1999:blog-5219665179084602082.post-8680951713691172637
blogger_orig_url: http://david-codes.blogspot.com/2014/05/cloud-platform-orchestrator-app.html
---

Conclusion#

That’s all what is needed to move to Jekyll on Github pages ! The process was quite smooth, now I need to tweak a bit this default theme :-)