CI Using Sauce Labs and Travis CI

I’ve been meaning to set up a build / integration server for the past year but haven’t gotten around to it for a myriad of reasons. Last week, I had enough with the

  • Features breaking every time a new feature is released (regression)
  • Manually smoke testing URLs
  • Having no structure for developing/testing new features

So, I decided to setup a Continuous Integration system for Tint! Here’s some notes on what I found as I navigated the confusing waters of setting up a build server.

Initial research

Continuous integration: the practice, in software engineering, of merging all developer working copies with a shared mainline several times a day.

Outline what your needs are with the build server. For example, my needs were:

  • Run selenium tests, preferably in parallel
  • Be triggered by Github pull requests and git pushes
  • Have an easy to use UI to see breaking builds
  • Have easy integration to email, HipChat, and Github

Tools:

  • Travis CI - I used them over CircleCI because Travis seemed to have more industry adoption and also had a better UI and documentation.
  • Sauce Labs - The leader in Selenium Grid SAAS, they also do a lot of active development on open source Selenium projects such as Selenium Builder, which is cool.
  • Ruby/Rspec/Rake - Wanted to use a language that had strong automation tools around it and was low on verbosity, yet still easily readable, so we went with Ruby and company.
  • Of course, there are many, many alternatives to Travis and Sauce (I actually started this project using CircleCI and BrowserStack), however, I chose Travis and Sauce in the end because they had more documentation and were easier to use.

Setup

  • (Optional) Use The Selenium Guidebook to setup a framework for your Selenium Tests, and to learn how to write maintainable tests. Make sure all the tests run on your Cloud Selenium tool you plan on using (Sauce or BrowserStack) before going forward.
  • Make an account with TravisCI, and turn on the repo that you want to set up a build server for.
  • TravisCI will automatically hook into Github events so that it will trigger a build on pull requests and pushes. It uses a .travis.yml file in the root of your directory to figure out how to create your build server.
  • Configure your .travis.yml so it builds your server properly. For example, in our travis.yml file, we clone our puppet manifest and then use puppet to create our webapp server and handle package dependencies, virtualhost files, random config files, and starting the services.
  • Once the server is built properly, you can use Sauce Lab’s Connect feature to run Selenium Webdriver test on Sauce against your build server. Unfortunately, how this all works is not adequately covered in the Sauce Labs documentation, so bear with the magic (this is literally what Sauce Labs gives you for documentation).

Debugging Your TravisCI Build Server

If you’re having trouble creating a working build server, you can email support@travis-ci.com nicely and they will allow you to setup a debug build server for you to log into to test things out. However, according to their support team, running a Vagrant VM using the default precise32 ubuntu image is very close to their current setup, so consider that as an option as well.

I highly recommend using Puppet to simplify your build process. Puppet is also useful outside of setting up CI as it allows you to easily configure many servers quickly (for example, adding a new virtualhost file to 10 servers in one shot), and does it in a way that is maintainable and version controlled.

Even with all of these useful tools, it took me a good couple of days to get the build up and running, so don’t be discouraged by how tedious it may seem. The only way to really debug a build server at the moment seems to be to make changes, rebuild the server, see if the build goes farther, and then repeat until everything is working as expected.

Weird things I ran into

When sauce labs tells you to insert the following into your travis.yml: curl https://gist.github.com/santiycr/5139565/raw/sauce_connect_setup.sh | bash It should actually be curl https://gist.githubusercontent.com/santiycr/5139565/raw/sauce_connect_setup.sh | bash Otherwise, you’ll get a strange “Connection refused” error.

Trying to load a second private repo in your Travis build server will result in a “Repository not found” or “Authentication failed” error which can only be fixed using this obscure support article.

Tuning Nginx and PHP-FPM

I was running into the issue where our CPU on our nginx webapp servers was not being fully utilized, and caused timeouts whenever CPU went above about 10% and memory was hardly being used. I had tried changing the configurations for nginx in the past with no success, so things were getting out of hand. When our traffic spiked yesterday morning due to Google Cloud Developers Conference, Where they are using Tint, we went down, and I had to increase our server count to 20 8GB 8vCPU servers.

Twenty servers to handle 20RPS just seemed ridiculous to me since nginx can handle thousands of RPS on a tuned machine. So I spent a couple of hours yesterday formulating a process to guess and check the effects of the server configurations in order to find out what was causing the issue.

The Process

  1. Isolate a single production server by removing it from all load balancers.
  2. Set up a Blitz.io account and validate the server in step 1 using the various methods outlined within blitz.
  3. Load test the server to see its performance.
  4. Shell into the server and change the server configuration, I was experimenting with /etc/nginx/nginx.conf and /etc/php5/fpm/pool.d/www.conf (don’t forget to restart the server)
  5. Load test the server while running ‘top’ and see if the performance changed.

Those 5 steps allowed me to finally figure out a combination of settings that allowed nginx and PHP to better utilize the CPU.

Test Results

Before

Image of pm.max_children = 5 test results

After

Image of pm.max_children = 375 test results

Server Configuration Changes

/etc/nginx/nginx.conf:

what I changed in nginx.conf

/etc/php5/fpm/pool.d/www.conf:

I changed pm.max_children = 5 to pm.max_children = 375

See the links below for more details on what these settings mean.

Additional Findings

average day * All of our traffic (~1600 concurrent users on Google Analytics realtime overview) can be handled by a single server with these new configurations. CPU of the single server handling all of our traffic was ~40%. * 6 of these servers behind a load balancer, an average of 53RPS could be handled while keeping response time less than 1s. Usually our RPS is around 5-15.

How To Create an Apple Developer Account Anonymously

Masked Credit Card

Yesterday, we ran into an issue where we were trying to release our white-label app in the Apple App Store. We wanted to submit it under a pseudonym or anonymously, however the App Store requires you to use a credit card with a name attached to it, and uses that name when you publish in the App Store. This is obviously unacceptable for a white label app, since searching any of our teammate’s names will quickly lead back to our company, and it put us in a bind because creating a credit card under a pseudonym is a pain in the ass.

So, after some sleuthing around, we found a credit card masking service called Do Not Track Me run by a company called Abine. Some further sleuthing revealed that this is in fact a legitimate company as cited by Forbes. We signed up for the premium monthly service, created a masked card, and voila! We were able to sign up to an Apple Developer Account under a pseudonym.

The Evolution of Your Deployment System

Stage 1

You wake up one day and decide to build an app. You spin up a micro instance and start developing. Deployment is simple. Just git pull and you’re done. When you expect more people to start loading the site, you spin up a couple more micros and put them behind a load balancer. This is the first time you’ve ever used these fancy load balancers, and everything seems to be humming along. Although, it gets annoying that you have to shell into each machine to update your code.

Stage 2

A couple weeks set in and it gets annoying to shell into each and every server to tweak anything. You do some sleuthing online and you read about Capistrano. So you set it up and after a half a day of being stuck on an SSH key issue, and another half a day tweaking the ruby code (while learning ruby at the same time), you can deploy with a single command. Yay.

Stage 3

Except, the only hitch is that the site is down for 30 seconds to a minute while the deployment happens. Not a big deal when your site has a handful of users on it concurrently. But as you grow, more and more users notice when you have downtime and you cannot ignore the issue any longer. You can’t figure out why Capistrano is causing the downtime anywhere on the web. Sigh. So, you decide to write a script that interacts with AWS to stagger Capistrano deploy commands and remove servers from the load balancer before deploying. After learning the AWS CLI, you cobble together a python script that results in zero downtime deployments.

Stage 4

Except, the script takes about 1 minute to run. Then, you have to double the number of servers since more and more people are using your service. That doubles deployment time to 2 minutes. Although it isn’t impossible, waiting 2 minutes for your code to deploy begins to irritate you. But your coworker mentions dploy.io and you install it in 30 minutes. Suddenly you have a scalable deployment method that deploys in less than 10 seconds and even has an intuitive web interface. Lovely. Why didn’t you figure this out earlier?

How to Find a Technical Cofounder

During my 4 years as an undergraduate computer science student at USC, I received hundreds of recruiting emails from business students looking for a technical co-founder or somebody to build their idea. Many of them were undeniably bad. Although I applaud all of these entrepreneurs for making an effort (however small), it bothers me that so many ideas die on the vine due to awful recruiting emails.

In order to help future entrepreneurs seeking technical help, here is a rough primer on how not to sound awful.

Let’s start by examining the average recruitment email:

LOOKING FOR A STUDENT TO WORK ON APP DESIGN PROJECT

Must be able to create a fully functional social app from scratch

Position responsibilities: will initially include building the front end and back end of our beta site. Must have a hacker attitude. If you are a Rockstar with machine learning, natural language, and data aggregation experience, you are amazing. Alright, we will talk to you if you could do only one of the three.

Compensation: Will be negotiable but ideally we are looking to bring someone into our team with programming expertise who really believes in our idea/business model. Candidates must sign an NDA before discussing the idea.

Duration: The initial duration will basically be as long as it takes to build the beta site with further opportunities available down the line.

Skills required: We believe they will need skills in PHP, MySQL, Ruby on Rails, Backbone.js, Angular, Databases, and Cloud Computing

Although initially the email looks organized and thorough, a more careful reading shows some glaring issues that immediately qualifies it as garbage.

The mistakes:

  • Mystery Product - There are exactly zero details on what the idea or project is about, resulting in exactly zero interest being sparked.
  • The NDA - Do not ask people to sign an NDA. Ideas are a dime a dozen, it disrespects whoever you are sending the email to, and starts the conversation in a distrustful tone.
  • “Hacker attitude / Rockstar / Amazing” - These buzzwords illustrate an ‘us’ vs ‘them’ attitude towards people with technical skills. Nobody in CS calls themselves a rockstar hacker.
  • Compensation - The explicitly stated negotiation makes the relationship transactional, signifying the business student’s belief that technical skill is a mere commodity to be traded for money and equity, neither of which are incredibly satisfying on their own.
  • Skills Required - Name-dropping frameworks and languages without any context demonstrates a total unfamiliarity with anything technical. Who would want to write software with someone who knows zilch about software?

Contrast this with an email that successfully recruited me to join a startup that would later be known as Tint

###The Job We are seeking an experienced back-end web developer to join our web startup. Our main shortage in talent is back-end development, and if you are passionate about making web applications and are looking for real-world experience, we are looking for you.

###About Hypemarks Hypemarks is an incubator-backed startup temporarily based in New York, started by five USC students. We see a huge problem of the amount of sites appearing everyday. With so many sites, how do you know which ones are relevant to you or worth your time checking out? Simply put: your friends.

We are a social bookmarking platform that collects all the sites you like and organizes them so you can share with your friends easily. Our vision is to allow people to define themselves based on their online experience when they share it with their friends. See an attached mockup to get a clearer idea of our vision: http://bit.ly/jYAJlg.

###Ideal Profile You are an experienced back-end developer not only excited to be part of a small startup co-founding team, but also motivated by personal ambition. You are flexible and learn quickly. As a co-founder, you will have tremendous ownership and responsibility on development. Communication will be key since we will be working remotely for the summer. We are a team of diverse USC students and alumni who are laid-back, yet focused on working to build something big and hope you will believe in our vision as well.

###Desirable Skills

Passion for building web apps Desire to learn a web application framework (or have knowledge in web application frameworks) CS Fundamentals Interest in start-up culture Strong database skills, preferably with MySQL Experience in building web applications

What got me to respond to this email?

  • Clear Product - Giving a clear picture of what the product vision is conveys both honesty and confidence in the idea. It also demonstrates that the business guy is capable of contributing to the project. The more links to prototypes / product decks, the better!
  • No Buzz Words - Clear, down to earth language that is conversational, but professional. Increases the trust factor.
  • Impact & Ownership - A clear overview of what is being offered, and an understanding that most technical people are interested in making an impact and taking an ownership. Being able to identify what motivates the person on the other end of the email is key to success.
  • Technical Understanding - The “Desired Skills” section shows a good understanding of the general knowledge necessary without resorting to name-dropping frameworks or languages. Clearly the writer did the necessary homework to learn a bit about web application programming, an excellent sign.

Being able to clearly convey value proposition doesn’t just help sell your product, it helps build your team. At the early stages, it’s easy to think of finding a team as an afterthought to your brilliant startup idea. However, what I’ve learned in building Tint is that team is everything. Put as much effort into finding quality people as you would in finding a potential mate. If all goes well, your email might be the first step to building a lifelong work relationship.

Additional Reading

My friend and CEO of Tint, Tim, has a good blog post from the other side of the table about how to convince developers to join your startup. Check it out.

Also, Drew Houston, Founder of Dropbox, has a few words to say on the topic of how to find a technical co-founder.

And a brief survey by Andrew Chen explaining what developers are looking for when choosing to work as a technical cofounder. Hint: It’s traction.

Empathy is Underrated

Our most recent hire, Brandon Ancier, said something at our last meeting that I had never considered before: “What I like about working here is that everyone one this team has business sense.” And as flattering as that is, I have to concede that he is right :) Looking back on how much our team has been able to achieve, the most important skill all of us have gained so far has been this “business sense”, which I define as the ability to identify optimal business decisions in a ambiguous, unknown environment. What are some of the perplexing decisions our team has to make on a day to day basis?

  1. What product features should I build next?
  2. Should the feature be self-service or custom built?
  3. How should I phrase the value-add on this product tour page?
  4. Should I answer the support emails first or the sales emails?

When questions like this come up, a good business sense allows us to come up with answers such as:

  1. Let’s write a feature story and send a survey out to the customers that would use this feature.
  2. Depends on what the survey says.
  3. Let’s get the Sales Team’s input on this. What language has worked for them in their pitches?
  4. Probably sales emails first, but let’s make it a priority to answer every email promptly even if we don’t know the answer. Let’s make sure they know we care.

See the pattern? The answer lies in empathy, being able to perceive the world though other people’s perspectives. If there were a single largest mistake I’ve seen cripple teams and throw wrenches in business processes, it’s lack of empathy.

The paradox is that in order to have [emotional] influence, you have to be influenced - The 7 Habits of Highly Effective People

Simply put, there’s no way of providing for a customers correctly without getting their input. Although Steve Jobs may have famously said, “Customers don’t know what they want”, consider that most of Apple’s products redefined entire product categories, and that’s not the domain of most businesses. If you’re like any regular business, you will want to constantly improve on your empathy. How?

What we do to improve customer empathy:

Give out your Cell Phone Number

Screenshot of Tint support email

Put your personal phone number in your communications with your most important customers. Encourage them to give you a ring! We’ve found great success in giving our largest customers multiple avenues to reach us. Although this might not scale in the future, we think it’s worth it.

Add customers to your instant messanger

Screenshot of customer instant messenger

Add your customers via gchat or skype and build a relationship. We do this with a few of our agencies, and it makes it incredibly easy to get instant feedback on something when you think to yourself, “I wish I could ask a customer right now!” Having customers know your company on a first name basis starts with knowing your customers on a first name basis.

Send nicely-crafted surveys

Screenshot of Tint customer feedback survey

I’m surprised at how few companies can put together a survey that is nicely-crafted in that it asks the right questions and establishes a personal connection.

When we write surveys, we write it with the intention of nurturing a relationship with the customer who is gracious enough to give us feedback and we frequently follow up on our survey participants to go deeper into their needs. The people who are going to fill out your surveys are going to be your most vocal customers, a fantastic opportunity to nurture your business’ brand!

No one cares how much you know, until they know how much you care - Theodore Roosevelt

The best teams have highly attuned senses of empathy. The empathy concept goes beyond customer-facing decisions.

When it comes to building a positive workplace and striving for better employee retention, seeing decisions from your co-workers’ perspective is essential. Open communication is the only way to build lasting, meaningful work relationships. Teams that can overcome the emotional roller coaster of building a business need to trust that everyone stands behind each other. And empathy is the foundation upon all communication and trust rests. How does Team Tint do this?

What we do to improve team empathy:

Transparent Compensation

Screenshot of compensation google doc

All of our compensation structure is made public within the company so that everyone can have a full picture and make sure that they feel that all team members are being compensated fairly in relation to each other. Making salaries public fosters empathy because it forces other people to consider the perspective of others when discussing compensation.

Encourage Sales/Engineering cross communication

Photo of team tint

Ever since the beginnings of the sales team (with the hiring of Zen, our Director of Sales), the development team (which was Nik and I) have always strived to make sure communication between Sales and Development was tight. We want to make sure that either team is 100% comfortable talking to the other team, and this communication fosters inter-departmental empathy that manifests itself in a number of positive ways including:

  • Less animosity between teams (fewer salespeople annoyed at bugs in the product and fewer developers annoyed of salespeople making promises the developers can’t fulfill)
  • Practical ideas from Sales being implemented by the Devs, such as automating key parts of their workflow
  • Efficient prioritization of inter-departmental tasks made possible by openly communicated cost/benefit.

We have each other’s backs

Photo of team tint

We understand that anyone can make a mistake, even our own teammates. By discouraging a blame-based culture, we build stronger bonds as well as encouraging people to speak the truth, to admit things that they feel are going wrong, and to work together to find solutions to inadequacies. Sure, blaming someone might be easier, but in the long run, a team that blames among itself is not a team, but rather a collection of disgruntled individuals.

In the end, cultivating empathy within yourself and among a team takes time, focus, and effort and often can’t be quantified. In an environment full of deadlines, it can be easy to be lazy and not think about the customers or teammates carrying you forward. We hope that sharing our steps to improve empathy can inspire you to improve your business and make choices to spend the resources to cultivate empathy. Let me know if you agree/disagree in the comments :)

Startups need role models too

Entrepreneurs are too obsessed with being unique. Whether it be through patents, a strange business model, or a weird UI, everyone seems to be affected by this unfortunate startup variant of Not Invented Here Syndrome. Everyone is ready to answer the question “How are you different?”

However, it’s more productive to also consider the question “How are you the same”? What is it about different companies that make them successful? It isn’t necessarily bad to be like other companies, even if they are your competition.

That’s why we aren’t bashful when we explain our company’s culture/vision/tech in terms of the companies we admire. It isn’t a matter of simply copycatting the hottest new thing. It’s understanding why companies choose to use a certain technology or adopt a certain organizational policy. It’s about picking and choosing the parts of an organization that fit your needs and wants.

We believe that it is critical to have organizational role models, much in the same way that it is critical to have personal role models. Without someone to follow, it is easier to lose sight of where you want to be, your organization’s goals and aspirations. A team’s vision and values can be written out in words, but far more powerful is watching another team execute them.

Why are organizational role models important?

  • They are a target: a concrete and visible example of where a group wants to be.
  • They are inspirational: Increasing morale, and give the entire organization hope. Role models serve as proof that certain goals are attainable.
  • They are insightful: Role models can make mistakes too, and learning from theirs can help prevent your own.

What are important characteristics to consider in choosing an organizational role model?

  • One Step Ahead: It helps if the organization is where you want your organization to be in the next 1-5 years. Too different and it’s hard to achieve the same
  • Open and Introspective: Find a role model that has a lot of content on their own processes or behavior. It’s hard to learn anything going on behind closed doors.
  • Similar Goals: The core of what makes someone a role model is that they are actively achieving something that you want to achieve. This one is tricky because only you can answer the question: What do you want to achieve? More revenue? Happy customers? Happy employees?

Who are our role models?

37 Signals - Make Money

In David’s video, he explains that a startup is a business, and a business is all about making money. So instead of ignoring the issue like many startups do, David encourages startups to start charging early. We agree with his strategies because our own success came from early monetization, and his talks inspired us to take the leap of faith!

Github - Keep it Flat

Github has changed over the years but their emphasis on keeping a flat organizational hierarchy is reflected in their avoidance of the word “Manager”, instead preferring “Primarily Responsible Person (PRP)”. It can be argued that this distinction is purely etymological. However, their policies go beyond words, giving everyone the authority to make relevant decisions when it comes to how things are implemented. For example, how customer support processes are organized. As we grow, we want to make sure to follow their advice and hire “managers of one” and strive for as much individual autonomy as possible.

Etsy - Engineering for Happiness

Etsy has engineered itself for developer happiness, and that’s something that we admire because a large part of day to day life involves the tedious things, including deploying code. Etsy’s investments in their 1 click deploy process and DevOps automation has led to numerous trickle-down improvements in their employees’ day to day lives. Less time waiting for scripts to run. Less time worrying about if code has made it to every server. More time doing what engineers love doing, making things. This is something we hope to have at Tint someday.

Buffer - Default to Transparency

Buffer’s claim to fame is Transparency. And not just regular transparency, but a transparency that goes above and beyond what a regular company is comfortable with. They’ve published the formula they use to calculate their salary and also the salaries of their entire team. Unlike a traditional company that hides salary information so their employees stay in the dark, adopting the policy of “ignorance is bliss”, Buffer embraces transparency, addressing issues of perceived under/overcompensation as they come up. Being open about salary forces a company to be as fair as possible.

Let us know who your organization’s role models are in the comments! Or if you don’t have any, start reading about what other organizations in your industry are doing, what parts of them are wonderful, and how you want to be like them!