Orr Sella

Summary of Brendan Gregg's Linux Performance Analysis Tools

Fri, 11 Nov 2016 00:00:00 -0800

Brendan Gregg wrote a great post on his first 60,000 milliseconds when trying to diagnose a server with a performance issue. If you haven’t yet, you should go read his post.

There are many great tips there, and I really wish I could remember all of them. Unfortunately I don’t get to use most of these day-to-day, but only every once in a while. And when I do – they come in very handy.

To help me remember these, and not have to open up his blog post every time, I added a handy alias to my dot files, which I take with me everywhere. This alias prints a summary of all the tools, with brief descriptions for each tool based on Brendan’s advise.

Here’s what it looks like when running it:

$ perf-tools
Linux Performance Analysis in 60,000 Milliseconds
Source: http://techblog.netflix.com/2015/11/linux-performance-analysis-in-60s.html
Even more tools: http://techblog.netflix.com/2015/08/netflix-at-velocity-2015-linux.html

 1) uptime              Load averages, indicate the number of tasks (processes) wanting to run (CPU and I/O).
 2) dmesg | tail        Last 10 system messages (if there are any).
 3) vmstat 1            Virtual memory stat, 1 second summaries. CPU stats are on average, across all CPUs:
                          - r          Number of processes running on CPU and waiting for a turn (CPU only, no I/O). Value > cores = saturation
                          - free       Free memory in kilobytes; see (7) for more info on free mem
                          - si/so      Swap-ins and swap-outs; if these are non-zero, you\'re out of memory
                          - us         User time
                          - sy         System time (kernel), necessary for I/O processing
                          - id         Idle
                          - wa         Wait I/O (like idle for I/O reason), constant value points to a disk bottleneck
                          - st         Stolen time
 4) mpstat -P ALL 1     CPU time breakdowns per CPU, allows to check for an imbalance (a single hot CPU can be evidence of a single-threaded application).
 5) pidstat 1           Per-process summary, useful for watching patterns over time. CPU column is the total across all CPUs (cores).
 6) iostat -xz 1        Workload and performance of block devices:
                          - r/s        Reads per second
                          - w/s        Writes per second
                          - rkB/s      Read Kbytes per second
                          - wkB/s      Write Kbytes per second
                          - await      The average time for the I/O in milliseconds (time the application suffers, as it includes both time queued and time being serviced)
                          - avgqu-sz   The average number of requests issued to the device; values greater than 1 can be evidence of saturation
                          - %util      Device utilization during internal (1 second in this case)
 7) free -m             Memory stats in Mbytes. Cached memory can be reclaimed quickly if apps need it, so it should be considered free (-/+ buffers/cache line). Also buffers and cached shouldn't be near-zero in size.
 8) sar -n DEV 1        Network interface throughput:
                          - rxkB/s     Received Kbytes per second (x8 for Kbits)
                          - txkB/s     Transmitted Kbytes per second (x8 for Kbits)
                          - %ifutil    Device utilization (max of both directions for full duplex)
 9) sar -n TCP,ETCP 1   Summarized view of some key TCP metrics:
                          - active/s   Number of locally-initiated TCP connections per second
                          - passive/s  Number of remotely-initiated TCP connections per second
                          - retrans/s  Number of TCP retransmits per second (sign of a network or server issue)
10) top/htop            Non-rolling system overview (which makes it hard to see patterns over time). Pro-tip: Ctrl-S to pause, Ctrl-Q to continue.

Want to add this to your dot files? Just copy the perf-tools function from here.

Thanks Brendan for your great tips, as always!

Managing Local and Remote Dot Files

Sat, 29 Oct 2016 00:00:00 -0700

This post describes how I like to think about and manage the dot files on both my local macOS dev machine, as well as any remote Linux server I use regularly – in one single setup.

TL;DR: Check out my dot files repo for the code. To get started you can just clone/fork and run one of the setup scripts.

My Dot Files

We all love dot files. A lot has been written about them, and there are many popular GitHub repos with cool examples. What makes mine different is the fact that I like to take them with me everywhere I go, including when SSH-ing into any server – both for work and personal use – even for the ones I’ve never used before!

They serve two purposes for me:

Not being attached to any particular dev machine – I rely on my dot files and the various setup scripts for a very quick ramp-up when moving to a new Mac.
Having maximum convenience and productivity anywhere possible.

My dot files include a lot of the typical shell/tools setup:

Shell¹
- Config
- Prompt
- Environment variables
- Aliases
- Handy utility functions
Vim
Tmux/Screen
Git/Mercurial

I also store the config files for some of my native Mac apps, including:

Alfred
Sublime Text
IntelliJ
iTerm

How It Works

My setup includes careful integration of the following:

Using git to store and version all files.
Precisely managing which dot file goes where.
Supporting both local (macOS) and remote (Linux) dot files, while also reusing the common parts.
Using the great sshrc script.
Configuring tmux/screen to work with sshrc.

Setup includes cloning the repo locally² and symlinking all files listed in links.

When SSH-ing somewhere, I use the sshrc command, which zips all files found in the ~/.sshrc.d directory (also managed by links), and extracts them to a temporary directory on the destination server. It then sources .sshrc to configure everything.

Lastly, I use my custom screenrc and tmuxrc commands on the server to open new sessions. These make sure that new sessions are initialized with everything extracted by sshrc.

Conclusion

This might sound complicated, but it works seamlessly and effortlessly. I no longer have to worry about keeping the various snowflake instances of my dot files on the various servers I use in sync, and I get to share a lot of the code that makes sense with my local Mac setup. I also don’t litter any server I connect to with local copies of my personal files since sshrc uses tmp files and cleans up after itself.

Interested? Take it for a spin: https://github.com/orrsella/dotfiles

Have a cool addition? Fork and add! Send a pull request if you think I should include in my setup as well.

I only use bash. I find comfort in knowing that any environment I work with has bash, without relying a specific configuration – that’s out of my control – having other shells like zsh, fish, etc. I feel like my dot files make it so I get a lot of the convenience of using other modern shells while having a consistent experience everywhere. This doesn’t mean you have to do the same. ↩
I like to use ~/.dotfiles. ↩

Preparing For a System Architecture Interview

Sat, 28 May 2016 00:00:00 -0700

This is the second part of my previous post on Preparing for a Facebook/Google Software Engineer Interview. If you haven’t yet, I suggest you go read it and then come back.

Disclaimer 1: Obviously, opinions are my own.

Disclaimer 2: None of the things I’m sharing here are things I’ve learned after starting my job or from the interviews themselves (like every other candidate I’ve signed NDAs).

System Architecture Interviews

Software engineering interviews with Facebook/Google (and similar companies) usually entail 1-2 sessions of system architecture questions. These questions tend to be more abstract – usually with little to no coding – and are trickier to prepare for. They are much easier if you have actual experience with system architecture¹ or are well versed in the topic.

In addition to seeing how experienced you are with large-scale – usually distributed – systems, these questions examine, among other things, how you:

Articulate complex and abstract ideas.
Think on your feet and tackle unknown obstacles, while asking the right questions.
Zoom out to examine a system as a whole.
Make real-world assumptions and balance trade-offs.

Additionally, system architecture questions are sometimes used to gauge how experienced a candidate is (along with their past work experience). Your performance in these interview sessions might impact the seniority level assigned to you as a candidate.

Wait, Is This a System Architecture Question?

Or, how to know you’re being asked a system architecture question.

Architecture questions are often presented in a very vague and under-specified manner. This is routinely intentional, as it enables the interviewer to discern how you “peel-off” an unknown assignment, which clarifying questions you ask, and assumptions you make. This is true for both “regular” coding questions and system architecture ones, but is more crucial in the latter as these tend to be very open-ended, and can be addressed in totally different ways.

One side-effect of presenting questions this way is that you might not realize you’re being asked a system architecture question. For example, consider the following:

How can we repeatedly find a string in a large dataset?

An eager candidate might immediately jump to the conclusion that having a large dataset searched repeatedly means it must be sorted and then binary-searched in log(n), and will start writing the code for mergesort on the whiteboard. All while the interviewer is thinking of a large document corpus and trying to get them to design a search engine.

Unless you ask some clarifying questions, like: “How big is the data set?”, “What are its characteristics?”, “How do we obtain it?”, you might not realize what you’re up against.

If you’re uncertain, a good trick here is to try and frame the question as a method signature. If it’s a system architecture question, it will force the interviewer to steer you away from it and think in more abstract terms.

Concepts You Should Know

Here’s a list of concepts/considerations/tools that you should have in your tool belt when approaching system architecture questions. In no particular order, and without being exhaustive:

Amount of data (disk)
Amount of RAM (memory)
- Does everything need to be in RAM? (often times – yes)
RAID configurations
Requests/sec
Request time
Latency
Data transfer rates
Network limits within a single data center
Geographically separated locations (multiple data centers)
Sharding/Partitioning (by user/date/content-type/alphabetically)
- How do you re-balance?
- Consistent hashing for sharding/partitioning
CAP theorem
MapReduce
Distributed hash tables
Difference between frontend/backend
SOA (service oriented architectures, also “micro-services”)
Backups
- Backup a single user on how many servers?
Caching
- Many requests for the same data/lots of updates – should be in cache
Distribute so that the network is negligible
Parallelizing network requests
Fan-out (scatter and gather)
Load-balancing
High availability
CDNs (content delivery networks)
POPs (points of presence)
Queues
Failures
- Timeouts
- Fault tolerance
- Failing fast
- Circuit breakers
- Throttling
Consensus (e.g. Paxos, Raft) and membership protocols (heartbeats, SWIM)
Compression
Possible to work harder on writes in order to make reads easier
Operations should almost never be more than n*log(n), preferably n

See the System Architecture page of my interview prep notes for more examples, explanations, and links for further reading.

Numbers

Having your “numbers” down is crucial for tackling these questions. What does this mean? It means that you should know by heart how long it takes to make an L1 cache reference, what reading from HDD/SSD means, and how many seconds are in a day, for example. You should also be very comfortable with order-of-magnitude calculations. Without thinking about it too much, you should know how much data is 10^12 bytes, or estimate how much memory is needed to store a trillion web addresses (also RAM vs disk; does it matter?).

Finally, you’ll be better off if you can easily convert between powers of two and ten. For example, know that 2^20 is roughly 10^6 (and also 1 MB, or more precisely 1 MiB).

Doing these calculations correctly and fast sends a very positive signal. In contrast, not being able to make these under pressure can derail your interview into an arithmetic duel, of which rarely can you come out victorious (even if you’ll end up being right).

Check out my summary of the Numbers you should memorize.

Designing Platforms

System architecture questions are usually to solve a specific problem or design a product. Rarely you might be asked to design a more broad platform or a lower-level service. These questions try to gauge what you think good platforms should have, and the types of services/feature you expect to have and use in a distributed environment.

If tasked with this challenge, here are some features of modern platforms you can throw up on the whiteboard and design with:

Monitoring
- Applications (response time, throughput, errors)
- Operating systems
- Hardware
Alerting
Metrics (counters)
Logging
Distributed tracing (cross-service)
Datastores
Caching
Queues
Service discovery
Configuration
Debugging, profiling
ALM (application lifecycle management) and deployment
- Continuous deployment
- Blue-green deployment
- Rollbacks
- Canary changes
- Shadow traffic
Auto scaling
Load balancing, traffic splitting/control
KPIs

Approaching a Question

I like the following approach to solving these questions:

Begin by asking qualifying questions to understand the system you want to design. It’s a good idea to get up to the whiteboard as early as you can, and start writing some features/assumptions you’re making at the edge of the board. Write down the 1-3 key features of the system. What are the most important features this system should have in your opinion?
Throw out some numbers to gauge the size of the problem. Give rough estimates to the amount of data/users/requests you’ll need to handle, and deduce the number of servers/HDD/datacenters you’ll need to implement it. Explain the rationale for your assumptions and write them on the board.
If you and your interviewer agree on the problem and its size, start a very high level design of the system. This should probably be some rectangles with arrows between them. For example: We’ll have these modules, these types of workers, this queue, that datastore, etc. As you make this high-level design, keep writing things you’ll need to consider or come back to at the edge of the board. Use tools from “Concepts You Should Know” above.
Get a sense of what interests your interviewer, which parts they think are key, and dive into them in greater resolution. Do we want to focus on data input? Efficiency? Error handling? Backup? The question can go in so many places, so make sure you’re addressing the most important and relevant parts.

Examples

Finally, you should run through a few examples while preparing for your interview. There are many examples for system architecture questions in sites like Glassdoor. Checkout github.com/checkcheckzz/system-design-interview for some nice ones.

These questions are typically asked in interviews of more experienced candidates. New-grads aren’t usually expected to be able to solve them. Instead, some companies ask a lighter variation which mostly revolves around designing APIs for a client application. If you’re not sure what’s expected of you – ask your recruiter! ↩

Preparing for a Facebook/Google Software Engineer Interview

Sat, 14 May 2016 00:00:00 -0700

TL;DR: This is the theoretical stuff I think you should know for an interview with Facebook/Google. Read on for some background and tips.

In late 2015 I interviewed at both Facebook and Google for a software engineering position. I received offers from both, and eventually decided to take Facebook’s one. In the months leading up to the interviews I prepared a lot. In this post I’d like to share how I think one should prepare for such an interview and give some tips.

Disclaimer 1: Obviously, opinions are my own.

Disclaimer 2: None of the things I’m sharing here are things I’ve learned after starting my job or from the interviews themselves (like every other candidate I’ve signed NDAs).

Your Job

Or, what an interview is and isn’t.

Before we get to business, here’s my take on how you should view an interview with Facebook/Google.¹ A lot has been written about how a typical interview with these companies is structured (phone screen, on-site, types of sessions, lengths, whiteboard/paper/computer coding, etc.). If you haven’t already, go read up on Glassdoor/other sites to get familiar with the format. It’s very important that you know what to expect. The more you know beforehand, the easier it will be and more comfortable you’ll feel.

Regardless of the technicalities – which again, are important – here’s how I think you should look at this interview. Let’s assume that you’ve already made up your mind that you’d like to work at these companies.² From your point of view, the purpose of the interview is to help you get an offer. That’s it. It is not to showcase your impressive résumé, or make you feel good about yourself, or to show how smart you are. You want to get that offer.

Your task in an interview with these companies is obviously to successfully solve the questions you’ll be presented with, but more importantly do it while showing your interviewer how you think and approach problem solving. Your goal is to send the right signal that your interviewer is looking to pick up. He or she will most likely be interviewing anywhere between 2 to 5 candidates a week. Remember that and think how you can help them see that you’re a good fit for the company. See more tips below on how to achieve this.

How To Prepare

I like to separate preparation to three parts:

Theory
Coding Problems
System Architecture Questions

Theory

This includes everything you’re expected to know as a software engineer.³ Things like: complexity analysis, data structures, algorithms, bit manipulation, operating systems, multi-threading, system architecture, numbers,⁴ how hardware works, and networking – to name a few. These are a mix of things that you’ll learn in school, from work experience, and by reading books/blogs/research/etc.

This is the basis for any discussions you’ll have during your interviews, for the code you’ll write, and for the system architecture solutions you’ll draw on the whiteboard. It’s essential that you have your theory nailed-down, and there’s no reason not to.

In my preparation, I summarized everything I thought I should’ve known by heart when coming for the interviews. Once summarized, I read it a few times during the weeks leading up to the interviews and one last time 2-3 days before each of the sessions.

You can find all my notes here:

https://github.com/orrsella/soft-eng-interview-prep

or in GitBook format here. It shouldn’t be a very long read – a few hours should suffice.

To compile these I read thru Cormen and Skiena, as well as countless other blog posts and Wikipedia articles (some are linked from the notes). It contains both explanations and code examples for what I think is important.

Coding Problems

Most of what you’ll do during an interview is write code to solve various problems (other things will be technical discussions, system architecture questions, and your own questions). They will usually require at most a few dozens of lines (it’s hard to fit a lot more than that on a whiteboard in ~40 minutes). The best way to prepare for these is to simply practice and solve as many as you can.

The best resources I used to practice coding questions are:

My recommendation would be to get at least one of these books and solve it cover-to-cover. There are lots of other online tools like LeetCode and various question banks. Quora also has some gems, if you search for something more specific.

After you feel comfortable with solving these types of questions, run thru some actual questions that were recently asked during interviews at the company you’re interviewing for, by searching Glassdoor.

In general – unsurprisingly – the more code you’ll write and problems you’ll encounter, the less likely it is that you’ll be surprised when you get to the interview. At some point questions start to repeat themselves, and you’ll recognize the patterns rather quickly.

System Architecture Questions

These are a completely different beast than coding questions. They are much more nuanced, and involve a lot more talking/explaining and almost no actual code. Preparing for these is harder and less straight-forward, especially if you haven’t worked on distributed systems before. See my post on how to prepare for these.

You can also read these parts of my notes:

System Architecture (specifically the Concepts section)
System Architecture Examples

And these helpful resources:

Do’s and Dont’s

Some tips, in no particular order:

Write actual code that works, not pseudocode
Opt for a mainstream programming language (Java, C/C++, Python, JavaScript)⁵
Practice coding on a whiteboard⁶
Practice coding on paper (preferably blank, without lines)
Write clean code:
- Use good names (classes/functions/variables)
- Show good modularity (classes/functions)
- Leave time for error checking and edge cases, even better to start with them (or add TODOs for yourself)
- If you’re cutting corners state that out loud, and say what you would do if you had more time (e.g., in Java: I’m using public fields for brevity here, but would otherwise use getters/setters)
Use proper algorithms and data structures and make sure to state Big-O for all of them
When first approaching a problem:
- Make sure you understand the task. Repeat it to show that to yourself and the interviewer
- Don’t begin by writing code immediately!
- Ask questions about the task, inputs, assumptions, formats; most questions are under specified on purpose
- Assume nothing! Or state you’re doing so
- Think out loud, share what you’re thinking (brainstorm)
- Try to show how you think thru the issue
- Silence isn’t good, long ones are bad
- Make sure the interviewer has a clear idea of how you’re doing (so they can help!)
- Start with a simple example, later add detailed ones
- Clarify the function signature of the problem early on. This will help focus your thoughts, and possibly invite subtle guidance from your interviewer
- Start with a simple inefficient solution (simple != easy)
- Improve the inefficient one
- Break down the problem to parts
Questions are in-depth, usually don’t have an easy solution (can be simple tho)
Think about input validation, constraints
Think about test cases, run thru them to make sure code is correct (but don’t assume it’s correct, really check as if someone else wrote it)
If the interviewer gave examples/hints use them
When done ask if you can refine the code, improving variable names, extracting other methods, etc.
Leave out trivial parts as functions that you need to implement, and only go back to those if you have time
Upon completion of first solution, either try to improve it, or try to come up with a different solution that is better (e.g.: recursion vs. imperative code)
Assumptions might now change, and so the solution needs to be adapted
Glossing over standard APIs if you’re not sure is OK, make sure you state that and give a reasonable API to work with

Ask Questions

At the end of each interview session, your interviewer will usually leave a couple of minutes for your questions. Use this time to ask meaningful questions. This both helps you know the company you’re interviewing for, but also sends a good signal about you. You should come prepared with a few questions, that can range from technical to organizational. Not having any questions isn’t a good sign, so make sure you’re covered.

Good Luck!

This, and a lot of other things from this post, also apply to other tech companies – YMMV. ↩
I recognize that the hiring process by these companies is controversial, and that many people don’t think that it really evaluates your expertise as a software engineer. I’m deliberately ignoring this, and assuming that this is what you want. If you want to get an offer from these companies then this is the “game” you’ll have to play. ↩
I’m thinking about this from the point of view of an engineer with 3-10-ish years of experience. I assume that if you’re a new grad, expectations are a bit different. ↩
You’d be surprised how handy these could be. ↩
You can most likely program in any language you want, but you’re better off if your interviewer knows your language of choice and can help if you get stuck. These choices increase the odds of that. ↩
Seriously, buy a whiteboard for home to practice on. It will make you feel much more comfortable with it, and you’ll learn how to better manage the space on it. ↩

Getting Started With Pants Build for Scala Projects

Sun, 31 May 2015 00:00:00 -0700

I’ve recently been examining a new build tool for large JVM projects (specifically tools aimed at monorepos). There is actually a surprising amount of options for such a tool: Google’s Bazel, Facebook’s Buck, and Pants, which is developed by Twitter, Foursquare, Square and others – are among the most popular.

When you start reading the docs for each of them, you get a sense of Déjà vu. And it’s for good reason – all three were modeled after Google’s internal build tool Blaze.

From Bazel’s FAQ:

What’s up with the word “Blaze” in the codebase?

This is an internal name for the tool. Please refer to Bazel as Bazel.

and Pants’ overview:

[…] Pants models code modules (known as “targets”) and their dependencies in BUILD files—in a manner similar to Google’s internal build system. […]

After going through quite a lot of the docs, I decided to give Pants a try. Besides having a decent amount of documentation, a deployable you can simply install (and not have to build from source), and being actively developed, Pants supports Scala out-of-the-box, which is exactly what I’m looking for.

After throughly reading the Overview and Tutorial, I got to Pants’ Installation page, which begins with the following statement:

As of September 2014, alas, Pants is not something you can just install and use. To be precise: you can install it, but unless you’ve also Set up your code workspace to work with Pants, it won’t work. You can use it in a workspace in which some Pants expert has set it up.

We’re fixing this problem, but we’re not done yet.

If want to try out Pants and no Pants expert has set it up for you, you might try https://github.com/twitter/commons.

[They are not exaggerating – it really won’t work ;)]

So that’s exactly what I did – used twitter/commons and other existing projects already configured by “pants experts” to understand the minimal setup required for a Pants Build project. It wasn’t easy, but eventually payed off. I reduced everything into one small project, which can be found here: pants-getting-started.

This simple project, which has one code file, and two tests, requires the following:

BUILD.tools
pants.ini
ivysettings.xml
7 3rdparty dependencies
10 BUILD files

Aside from the cumbersome bootstrapping experience, I’m really enjoying working with Pants so far, and would definitely recommend that you check it out. Want to get started? Use this quick template as a starting point: https://github.com/orrsella/pants-getting-started.

TL;DR

$ pip install pantsbuild.pants
$ git clone git@github.com:orrsella/pants-getting-started.git
$ cd pants-getting-started/
$ pants test example:

Init.d and Start Scripts for Scala/Java Server Apps

Thu, 06 Nov 2014 00:00:00 -0800

I like to use two scripts for running and managing the lifecycle of server applications: a start script and an init.d script.

The start script does the actual job of launching the application. In the case of Scala or Java it fires up the JVM and adds all the necessary arguments (memory, GC, etc.). It should also provide the basic configuration for your application: main entry point, classpath, config file, logging config file, etc. You should be able to easily run the application by simply launching the start script; however this is not something you’re most likely ever do to – for that you have the init.d script.

The init.d script is used to control the application’s process state and lifecycle: start and stop it, check whether it’s running, and restart it if necessary. It should also manage the application’s pid file. It could be used for event monitoring, like letting an external system know that the service’s state has changed. The init.d script doesn’t interact with the app itself, but rather with its start script. This allows you to decouple application specific configuration from the lifecycle and state of the operating system process.

Below are the two scripts that I use. These are actually jinja2 templates that I use in conjunction with Ansible to deploy multiple services. They assume that your application is deployed as a thin jar in BIN_DIR, with all of its dependencies in the same directory (as can be achieved by using standalone Ivy, Maven and its dependency:get plugin, or simply by exploding a fat jar downloaded on the server).

Start Script

#!/bin/bash

MAIN_CLASS={{ main_class }}
APP_CONFIG={{ app_config }}
LOG_CONFIG={{ log_config }}
BIN_DIR={{ bin_dir }}

# ***********************************************
# ***********************************************

ARGS="-Dconfig.file=${APP_CONFIG} -Dlogback.configurationFile=${LOG_CONFIG}"

exec java $ARGS -cp "$BIN_DIR/*" $MAIN_CLASS

(Gist found here)

(For application configuration I’m using Typesafe config with the config.file parameter, and Logback with the logback.configurationFile parameter for logging, but that shouldn’t matter at all. Customize as needed.)

I usually like to place the start script right next to the BIN_DIR in a script directory. Here’s my typical structure:

$ tree /opt/myapp
├── bin
│   ├── com.example.myapp-1.0.0.jar
│   └── ... # all your dependency jars
├── conf
│   ├── application.conf
│   └── logback.xml
└── script
    └── start

All of these pieces are put in-place by the CM tool. Launching the start script will run the application interactively in the logged in session, displaying stdout in the terminal. This could be useful for debugging, but should generally be avoided – you should start the app from the init.d script.

Init.d Script

#!/bin/bash

START_SCRIPT={{ start_script }}
PID_FILE={{ pid_file }}

# ***********************************************
# ***********************************************

ARGS="" # optional start script arguments
DAEMON=$START_SCRIPT

# colors
red='\e[0;31m'
green='\e[0;32m'
yellow='\e[0;33m'
reset='\e[0m'

echoRed() { echo -e "${red}$1${reset}"; }
echoGreen() { echo -e "${green}$1${reset}"; }
echoYellow() { echo -e "${yellow}$1${reset}"; }

start() {
  PID=`$DAEMON $ARGS > /dev/null 2>&1 & echo $!`
}

case "$1" in
start)
    if [ -f $PID_FILE ]; then
        PID=`cat $PID_FILE`
        if [ -z "`ps axf | grep -w ${PID} | grep -v grep`" ]; then
            start
        else
            echoYellow "Already running [$PID]"
            exit 0
        fi
    else
        start
    fi

    if [ -z $PID ]; then
        echoRed "Failed starting"
        exit 3
    else
        echo $PID > $PID_FILE
        echoGreen "Started [$PID]"
        exit 0
    fi
;;

status)
    if [ -f $PID_FILE ]; then
        PID=`cat $PID_FILE`
        if [ -z "`ps axf | grep -w ${PID} | grep -v grep`" ]; then
            echoRed "Not running (process dead but pidfile exists)"
            exit 1
        else
            echoGreen "Running [$PID]"
            exit 0
        fi
    else
        echoRed "Not running"
        exit 3
    fi
;;

stop)
    if [ -f $PID_FILE ]; then
        PID=`cat $PID_FILE`
        if [ -z "`ps axf | grep -w ${PID} | grep -v grep`" ]; then
            echoRed "Not running (process dead but pidfile exists)"
            exit 1
        else
            PID=`cat $PID_FILE`
            kill -HUP $PID
            echoGreen "Stopped [$PID]"
            rm -f $PID_FILE
            exit 0
        fi
    else
        echoRed "Not running (pid not found)"
        exit 3
    fi
;;

restart)
    $0 stop
    $0 start
;;

*)
    echo "Usage: $0 {status|start|stop|restart}"
    exit 1
esac

(Gist found here)

This script is a lot more complicated. Let’s break it down:

You would normally put this script in /etc/init.d/myapp.
The script accepts the following commands: status, start, stop and restart.
START_SCRIPT should be the location of the start script.
PID_FILE is where you want your pid file saved, usually: /var/run/myapp.pid.
The script is intelligent enough not to let you start two instances of the app if it’s already running.
Standard script exit codes:
- 0 – program is running or service is OK
- 1 – program is dead and /var/run pid file exists
- 3 – program is not running
Running each of the commands will output a colored status message with the result. For example:

$ sudo /etc/init.d/myapp start
Started [27882] # <= trust me, this is green

The combination of these two scripts offers a robust way to manage the lifecycle of the app. I think they provide a good separation of concerns, and allow the init.d script to be reusable for multiple types of applications, provided that they have a start script. Only the start script intimately knows the app and that it’s is JVM-based, and it does not bother itself with environment concerns like process ids and such.

If you’re interested in the rest of the deployment code I use for such server apps you can check out this Ansible role.

Have a comment or a suggestion for a better way to do any of this? Please let me know!

Embedded Elasticsearch Server for Scala Integration Tests

Tue, 28 Oct 2014 00:00:00 -0700

Integration tests are used to check how our code works against code we can’t change. As described in GOOS:

We use the term integration tests to refer to the tests that check how some of our code works with code from outside the team that we can’t change. It might be a public framework, such as a persistence mapper, or a library from another team within our organization. The distinction is that integration tests make sure that any abstractions we build over third-party code work as we expect.

A classic example for an Integration Test is for a Dao class. Let’s assume that we have some abstract PersonDao trait, that defines how we store, fetch, and search for Persons. A hypothetical application could have an implementation of that trait: ElasticsearchPersonDao which abstracts over Elasticsearch.

To make sure that our Dao implementation works with Elasticsearch as intended and that it fulfills the PersonDao contract, we should have an Integration Test. During that test we want to run against an actual instance of Elasticsearch, as close to the real thing as possible. One way to achieve this is to have our test bring-up an embedded server version of Elasticsearch, have the Dao work against that server, and then shut it down.

Doing so for Elasticsearch is particularly easy since Elasticsearch is written in Java. Let’s see how to do this:

package com.example

import java.nio.file.Files
import org.apache.commons.io.FileUtils
import org.elasticsearch.client.Client
import org.elasticsearch.common.settings.ImmutableSettings
import org.elasticsearch.node.NodeBuilder._

class ElasticsearchServer {

  private val clusterName = "elasticsearch"
  private val dataDir = Files.createTempDirectory("elasticsearch_data_").toFile
  private val settings = ImmutableSettings.settingsBuilder
    .put("path.data", dataDir.toString)
    .put("cluster.name", clusterName)
    .build

  private lazy val node = nodeBuilder().local(true).settings(settings).build
  def client: Client = node.client

  def start(): Unit = {
    node.start()
  }

  def stop(): Unit = {
    node.close()

    try {
      FileUtils.forceDelete(dataDir)
    } catch {
      case e: Exception => // dataDir cleanup failed
    }
  }

  def createAndWaitForIndex(index: String): Unit = {
    client.admin.indices.prepareCreate(index).execute.actionGet()
    client.admin.cluster.prepareHealth(index).setWaitForActiveShards(1).execute.actionGet()
  }
}

Let’s see what’s going on here. First of all this code assumes that you have Elasticsearch as a dependency (which you probably do since you’re using it as your datastore…). It also uses apache commons-io FileUtils (version 2.4) for performing directory cleanup. If this is a problem you can easily write this cleanup code yourself.

After creating the settings, a nodeBuilder is used to create a local Elasticsearch Node. A Node can be used as a way to connect to an existing Elasticsearch cluster, or it can form a cluster. To start the Node we use the start/stop methods. Notice that the node is a lazy val, which means it isn’t initialized until accessed. After starting the node, the client can be passed to the Dao object, which wouldn’t be able to tell if it’s talking to a remote cluster or a local node – exactly what we want.

The createAndWaitForIndex convenience method helps us create the index we’re going to use during the test setup, otherwise an IndexMissingException will be thrown and fail the test.

Let’s see how a test would use the ElasticsearchServer to exercise the Dao (this example uses the specs2 test library):

package com.example

import java.util.UUID
import org.specs2.mutable.Specification
import org.specs2.specification.Scope

class ElasticsearchPersonDaoIntegrationTest
  extends Specification
  with PersonMatchers {

  val server = new ElasticsearchServer

  trait Context extends Scope {
    val dao = new ElasticsearchPersonDao(server.client)
  }

  step {
    server.start()
    server.createAndWaitForIndex("persons")
  }

  "Elasticsearch person dao" should {
    "return None for a non-existing person" in new Context {
      val person: Option[Person] = dao.get(UUID.randomUUID)
      person must beNone
    }

    "create and then return a person" in new Context {
      val id: UUID = dao.add(name = "John")
      val person = dao.get(id)
      person must beSome[Person]
      person must haveName("John")
    }
  }

  step {
    server.stop()
  }
}

A few notes:

We are using specs2’s steps for bringing-up and tearing-down our embedded server.
The use of Context provides us with variable isolation between every test case. A new Dao object is created every time, but it always uses the same client to connect to our Node.
Our second test case checks both the add and get functions of the Dao.
We’re using custom PersonMatchers to make the second test assertion a little nicer. These could go a long way for complex/nested data objects.

That’s all you need to start creating Elasticsearch Integration Tests. All that’s left is implementing the Dao itself. Having comprehensive datastore ITs will enable you seamlessly swipe out your datastore for another one, assuming your Integration Test passes for all of them. The exact same test could be used to test a hypothetical MySqlPersonDao, only with a different embedded server. If such a scenario arises, you should make the test abstract and run it for both Daos.

HTTP Request Diagnostics With cURL

Mon, 06 Oct 2014 00:00:00 -0700

I had recently needed to diagnose some HTTP requests to understand where time was spent, download speeds, etc. This great post by Joseph Scott shows how to get timing info for a request using curl. I expanded on it by adding all available curl --write-out variables and HTTP headers (sample response below).

Here’s what you need to do:

Create a file for the curl formatting (named curl-format in our case).
Add the following to curl-format:

\n\n
      content_type: %{content_type}\n
filename_effective: %{filename_effective}\n
    ftp_entry_path: %{ftp_entry_path}\n
         http_code: %{http_code}\n
      http_connect: %{http_connect}\n
          local_ip: %{local_ip}\n
        local_port: %{local_port}\n
      num_connects: %{num_connects}\n
     num_redirects: %{num_redirects}\n
      redirect_url: %{redirect_url}\n
         remote_ip: %{remote_ip}\n
       remote_port: %{remote_port}\n
     size_download: %{size_download}\n
       size_header: %{size_header}\n
      size_request: %{size_request}\n
       size_upload: %{size_upload}\n
    speed_download: %{speed_download}\n
      speed_upload: %{speed_upload}\n
 ssl_verify_result: %{ssl_verify_result}\n
     url_effective: %{url_effective}\n
\n\n
   time_namelookup: %{time_namelookup}\n
      time_connect: %{time_connect}\n
   time_appconnect: %{time_appconnect}\n
  time_pretransfer: %{time_pretransfer}\n
     time_redirect: %{time_redirect}\n
time_starttransfer: %{time_starttransfer}\n
                   -------\n
        time_total: %{time_total}\n
\n

Then make a request with the format file:

$ curl -v -w "@curl-format" -o /dev/null -s http://www.orrsella.com

Example response:

* Adding handle: conn: 0x7f8b3b803600
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x7f8b3b803600) send_pipe: 1, recv_pipe: 0
* About to connect() to www.orrsella.com port 80 (#0)
*   Trying 104.131.121.209...
* Connected to www.orrsella.com (104.131.121.209) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.30.0
> Host: www.orrsella.com
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
* Server nginx/1.4.6 (Ubuntu) is not blacklisted
< Server: nginx/1.4.6 (Ubuntu)
< Date: Mon, 06 Oct 2014 13:42:42 GMT
< Content-Type: text/html
< Content-Length: 193
< Connection: keep-alive
< Location: http://orrsella.com/
<
{ [data not shown]
* Connection #0 to host www.orrsella.com left intact


      content_type: text/html
filename_effective: /dev/null
    ftp_entry_path:
         http_code: 301
      http_connect: 000
          local_ip: 192.168.28.249
        local_port: 58388
      num_connects: 1
     num_redirects: 0
      redirect_url: http://orrsella.com/
         remote_ip: 104.131.121.209
       remote_port: 80
     size_download: 193
       size_header: 203
      size_request: 80
       size_upload: 0
    speed_download: 689.000
      speed_upload: 0.000
 ssl_verify_result: 0
     url_effective: http://www.orrsella.com/


   time_namelookup: 0.001
      time_connect: 0.141
   time_appconnect: 0.000
  time_pretransfer: 0.141
     time_redirect: 0.000
time_starttransfer: 0.280
                   -------
        time_total: 0.280

Check out the curl man page for a detailed explanation of each of the used %{variable}s.

True End-to-End Testing in Scala: Scalapeño 2014 Talk

Wed, 24 Sep 2014 00:00:00 -0700

Earlier this week I gave the following talk at Scalapeño 2014. Below are the slides for your convenience. The video recording can be found on the conference’s Parleys channel. The talk is also accompanied by a very detailed code example that shows how to implements all the discussed concepts.

In this talk we will show how we can take our unit/integration tests a step further, and see how to test our applications end-to-end. Scala is a prime candidate for end-to-end testing, having all the right pieces in place: an easily extendable build tool (SBT), fluent DSL testing libraries (Specs2/ScalaTest), and the power/tooling of the JVM behind it.

We will see how to approach testing our application from the outside, and touch on concepts such as: the Test Harness, abstracting our SUT (System Under Test) by using test Drivers, using Fakes for testing against external dependencies, and more. We will put everything together by using tools such as: SBT, Ansible and Vagrant.

Integration and End-to-End Test Configurations in SBT

Wed, 24 Sep 2014 00:00:00 -0700

It is common to have three levels of tests:

Unit Tests – Checking if your objects do the right thing, if they work correctly. Usually tests methods/classes/small clusters of classes.
Integration Tests – Testing if your code works against code that you can’t change (3rd party or internal code that you can’t influence). Are your abstractions over that 3rd party code correct and used as intended?
End-to-End Tests – Checking if the whole system works and composes correctly.

When developing using TDD, I find it very useful to be able to choose which type of tests to run. Each kind of testing has its own trade-offs, and I sometimes find myself wanting to run a specific group of tests (for example: only quickly running unit tests and not end-to-end tests, which take far longer to complete).

To achieve this, here is how I like to structure my projects:

src
├── e2e
├── it
├── main
├── test
└── ...

This is the typical Maven directory layout , with the additional testing source roots. I place my unit tests under the test/ source directory and name them *Test, my integration tests under it/ and named *IntegrationTest, and – unsurprisingly – end-to-end tests under e2e/ and named *EndToEndTest.

This separation clearly divides the types of tests and allows granularity in running them. Here is how to configure this in SBT using .scala build configuration:

// project/Configs.scala

import sbt._

object Configs {
  val IntegrationTest = config("it") extend(Runtime)
  val EndToEndTest = config("e2e") extend(Runtime)
  val all = Seq(IntegrationTest, EndToEndTest)
}

// project/Testing.scala

object Testing {
  lazy val testAll = TaskKey[Unit]("test-all")

  private lazy val itSettings =
    inConfig(IntegrationTest)(Defaults.testSettings) ++
    Seq(
      fork in IntegrationTest := false,
      parallelExecution in IntegrationTest := false,
      scalaSource in IntegrationTest := baseDirectory.value / "src/it/scala")

  private lazy val e2eSettings =
    inConfig(EndToEndTest)(Defaults.testSettings) ++
    Seq(
      fork in EndToEndTest := false,
      parallelExecution in EndToEndTest := false,
      scalaSource in EndToEndTest := baseDirectory.value / "src/e2e/scala")

  lazy val settings = itSettings ++ e2eSettings ++ Seq(
    testAll <<= (test in EndToEndTest).dependsOn((test in IntegrationTest).dependsOn(test in Test))
  )
}

When defining your projects, add the new configs and settings:

import sbt._
import sbt.Keys._

object TheBuild extends Build {
  lazy val root = Project("root", file("."))
    .configs(Configs.all: _*)
    .settings(Testing.settings: _*)
}

Then to run only a specific test configuration (end-to-end for example):

$ sbt> e2e:test

We also added the testAll task, so we can run all tests together:

$ sbt> test-all

If you need more examples you can check out my scala-e2e-testing sample project, which shows how to test your Scala apps end-to-end, and uses this project structure.