May 3, 2014
6 months ago, Nikhil and I were the only developers at our 4 person startup. With business growing steadily, we were so spread thin that there was no hope of improving our product if we didn’t bring more help onto the team. So, Nik and I put our recruiting hats and began our journey to find talented engineers to join us. Fast forward 6 months to the present, and our Engineering team is about to grow to 7 (including 2 interns!), and I can safely say that I’ve learned a whole lot in the process:
Never consider recruitment work “a waste of time”
Time spent finding the right people for your team lays the foundation for everything else at a startup. A great product starts with a great team. So no matter how disheartening it feels to comb through resume after resume and still not find the right fit, always remember that recruitment work is as important as building a new feature or optimizing a process. So do yourself a favor and put quality time into doing the following:
Work your network
Our second engineering hire, Brett, came from Nikhil’s extended network and not from any job board or recruiting company. You never know who’s looking and with social media, it’s easier than ever to let all of your friends know that you’re looking to hire. It’s also easier to bring someone onto the team if they’re vetted by a friend than if they are a stranger. Not only do you feel like you can trust their competency, they can also better trust your competency!
Post a Quality Job Post
Know what people are looking for in their next job.
Hint: it’s probably Mastery, Autonomy, and Purpose
And know why your job is what people are looking for.
Your job post should highlight your strengths.
For example, our strength is our company culture. Our mission is to build a company culture that champions transparency, fairness, happiness, and sustainability. And we make sure to highlight that in our job posting:
- Profit Sharing - We split 20% of all revenue made over 100k and distribute it evenly among the team.
- Team Transparency - We calculate compensation based on a formula that we all agree on. Cap table is made available to all employees. Business financials are known by all teammates.
- Personal Autonomy / Consensus Driven Culture - We foster consensus-driven rather than top-down decision making when it comes to important business decisions. From what features to build next to what furniture to buy for the office, we believe it’s the fairest way of making decisions.
- Customer Driven Culture - We’re very in-tune with our customers and they love us. For example, we decide what features to build based on surveys we send directly to customers. Check out this one that we sent out last year to decide what we would build this past quarter.
- Personal Development Stipend - A monthly stipend designed for self-improvement. Whether it’s books, yoga classes, or a fitness tracker, we want our teammates to improve themselves.
Send Quality Emails
Quality recruiting emails are emails that recognize and understand the candidate. Here are some tips to add some empathy to your correspondance, embedded in a sample Tint recruitment email:
Thanks for scheduling a time with me! To prep for our interview, I’d recommend reading up on our company, getting familiar with what we do, and coming up with a few questions to ask us.
Here are some helpful links to peruse:
- Give the candidate a small assignment to assess their interest in the listing.
- Arm your candidate with the basic knowledge you expect them to know so you can have a productive discussion.
- Give the candidate the motivation they need to get excited about the opportunity.
Protip: Use Assistant.to to schedule your interviews. It’s a Gmail extension that allows you to easily give candidates a way to instantly book a meeting with you and have it show up in your calendar!
Protip 2: Use Yesware to create templates for your common recruiting emails, saving you further time.
Use a CRM
Handing resumes manually through email is incredibly time consuming. Use one of thousands of resume tracking Applicant Tracking Software (ATS in recruiter lingo) such as Resumator, Jobvite, or JobScore to simplify your life.
Find creative places to post to
A job listing link can travel far! But it’s your job to take it there. Consider the following places we posted our link to:
- Craigslist - We posted our listing on 10 major metro tech centers advertising paid relocation and had some success on attracting some good candidates. At $25 a posting, it was an affordable way to reach attractive candidates in markets that have much less competition than San Francisco.
- Hacker News - We found some quality candidates (including one of our interns) from posting our listing as a comment within the monthly “Who’s Hiring” thread. It gets posted on the 1st of every month, so don’t miss out!
- Indeed/Careers/Monster - Surprisingly, these mainstream job boards are frequented by talented people too! Most ATS systems will post to these major boards automatically, so be sure to configure your system to do that.
- Github Jobs - We found some alright leads from this paid posting, fewer applicants but the average quality was higher.
- StackOverflow Careers - We paid to run a campaign on StackOverflow but found that all of the submissions were overseas Java developers at big corporations looking for visa sponsorships. Maybe we were doing something wrong, but we ended up asking for a refund.
- Reddit - Plenty of subreddits to explore if you’re looking to find a community of people who you think would be a good fit. Think /r/bigdatajobs or /r/sysadminjobs
If you’re looking to expand your team, you have to recruit like a pro. It’s better to do things thoroughly from the get-go than to lukewarmly recruit for a longer period of time. Follow the tips above, and finding an engineer in San Francisco shouldn’t be as impossible as everyone says.
April 8, 2014
I’ve been meaning to set up a build / integration server for the past year but haven’t gotten around to it for a myriad of reasons. Last week, I had enough with the
- Features breaking every time a new feature is released (regression)
- Manually smoke testing URLs
- Having no structure for developing/testing new features
So, I decided to setup a Continuous Integration system for Tint! Here’s some notes on what I found as I navigated the confusing waters of setting up a build server.
Continuous integration: the practice, in software engineering, of merging all developer working copies with a shared mainline several times a day.
Outline what your needs are with the build server. For example, my needs were:
- Run selenium tests, preferably in parallel
- Be triggered by Github pull requests and git pushes
- Have an easy to use UI to see breaking builds
- Have easy integration to email, HipChat, and Github
- Travis CI - I used them over CircleCI because Travis seemed to have more industry adoption and also had a better UI and documentation.
- Sauce Labs - The leader in Selenium Grid SAAS, they also do a lot of active development on open source Selenium projects such as Selenium Builder, which is cool.
- Ruby/Rspec/Rake - Wanted to use a language that had strong automation tools around it and was low on verbosity, yet still easily readable, so we went with Ruby and company.
- Of course, there are many, many alternatives to Travis and Sauce (I actually started this project using CircleCI and BrowserStack), however, I chose Travis and Sauce in the end because they had more documentation and were easier to use.
- (Optional) Use The Selenium Guidebook to setup a framework for your Selenium Tests, and to learn how to write maintainable tests. Make sure all the tests run on your Cloud Selenium tool you plan on using (Sauce or BrowserStack) before going forward.
- Make an account with TravisCI, and turn on the repo that you want to set up a build server for.
- TravisCI will automatically hook into Github events so that it will trigger a build on pull requests and pushes. It uses a .travis.yml file in the root of your directory to figure out how to create your build server.
- Configure your .travis.yml so it builds your server properly. For example, in our travis.yml file, we clone our puppet manifest and then use puppet to create our webapp server and handle package dependencies, virtualhost files, random config files, and starting the services.
- Once the server is built properly, you can use Sauce Lab’s Connect feature to run Selenium Webdriver test on Sauce against your build server. Unfortunately, how this all works is not adequately covered in the Sauce Labs documentation, so bear with the magic (this is literally what Sauce Labs gives you for documentation).
Debugging Your TravisCI Build Server
If you’re having trouble creating a working build server, you can email firstname.lastname@example.org nicely and they will allow you to setup a debug build server for you to log into to test things out. However, according to their support team, running a Vagrant VM using the default precise32 ubuntu image is very close to their current setup, so consider that as an option as well.
I highly recommend using Puppet to simplify your build process. Puppet is also useful outside of setting up CI as it allows you to easily configure many servers quickly (for example, adding a new virtualhost file to 10 servers in one shot), and does it in a way that is maintainable and version controlled.
Even with all of these useful tools, it took me a good couple of days to get the build up and running, so don’t be discouraged by how tedious it may seem. The only way to really debug a build server at the moment seems to be to make changes, rebuild the server, see if the build goes farther, and then repeat until everything is working as expected.
Weird things I ran into
When sauce labs tells you to insert the following into your travis.yml:
curl https://gist.github.com/santiycr/5139565/raw/sauce_connect_setup.sh | bash
It should actually be
curl https://gist.githubusercontent.com/santiycr/5139565/raw/sauce_connect_setup.sh | bash
Otherwise, you’ll get a strange “Connection refused” error.
Trying to load a second private repo in your Travis build server will result in a “Repository not found” or “Authentication failed” error which can only be fixed using this obscure support article.
March 26, 2014
I was running into the issue where our CPU on our nginx webapp servers was not being fully utilized, and caused timeouts whenever CPU went above about 10% and memory was hardly being used. I had tried changing the configurations for nginx in the past with no success, so things were getting out of hand. When our traffic spiked yesterday morning due to Google Cloud Developers Conference, Where they are using Tint, we went down, and I had to increase our server count to 20 8GB 8vCPU servers.
Twenty servers to handle 20RPS just seemed ridiculous to me since nginx can handle thousands of RPS on a tuned machine. So I spent a couple of hours yesterday formulating a process to guess and check the effects of the server configurations in order to find out what was causing the issue.
- Isolate a single production server by removing it from all load balancers.
- Set up a Blitz.io account and validate the server in step 1 using the various methods outlined within blitz.
- Load test the server to see its performance.
- Shell into the server and change the server configuration, I was experimenting with /etc/nginx/nginx.conf and /etc/php5/fpm/pool.d/www.conf (don’t forget to restart the server)
- Load test the server while running ‘top’ and see if the performance changed.
Those 5 steps allowed me to finally figure out a combination of settings that allowed nginx and PHP to better utilize the CPU.
Server Configuration Changes
I changed pm.max_children = 5 to pm.max_children = 375
See the links below for more details on what these settings mean.
- All of our traffic (~1600 concurrent users on Google Analytics realtime overview) can be handled by a single server with these new configurations. CPU of the single server handling all of our traffic was ~40%.
- 6 of these servers behind a load balancer, an average of 53RPS could be handled while keeping response time less than 1s. Usually our RPS is around 5-15.
March 12, 2014
Yesterday, we ran into an issue where we were trying to release our white-label app in the Apple App Store. We wanted to submit it under a pseudonym or anonymously, however the App Store requires you to use a credit card with a name attached to it, and uses that name when you publish in the App Store. This is obviously unacceptable for a white label app, since searching any of our teammate’s names will quickly lead back to our company, and it put us in a bind because creating a
credit card under a pseudonym is a pain in the ass.
So, after some sleuthing around, we found a credit card masking service called Do Not Track Me run by a company called Abine. Some further sleuthing revealed that this is in fact a legitimate company as cited by Forbes. We signed up for the premium monthly service, created a masked card, and voila! We were able to sign up to an Apple
Developer Account under a pseudonym.
February 20, 2014
You wake up one day and decide to build an app. You spin up a micro instance and start developing. Deployment is simple. Just git pull and you’re done. When you expect more people to start loading the site, you spin up a couple more micros and put them behind a load balancer. This is the first time you’ve ever used these fancy load balancers, and everything seems to be humming along. Although, it gets annoying that you have to shell into each machine to update your code.
A couple weeks set in and it gets annoying to shell into each and every server to tweak anything. You do some sleuthing online and you read about Capistrano. So you set it up and after a half a day of being stuck on an SSH key issue, and another half a day tweaking the ruby code (while learning ruby at the same time), you can deploy with a single command. Yay.
Except, the only hitch is that the site is down for 30 seconds to a minute while the deployment happens. Not a big deal when your site has a handful of users on it concurrently. But as you grow, more and more users notice when you have downtime and you cannot ignore the issue any longer. You can’t figure out why Capistrano is causing the downtime anywhere on the web. Sigh. So, you decide to write a script that interacts with AWS to stagger Capistrano deploy commands and remove servers from the load balancer before deploying. After learning the AWS CLI, you cobble together a python script that results in zero downtime deployments.
Except, the script takes about 1 minute to run. Then, you have to double the number of servers since more and more people are using your service. That doubles deployment time to 2 minutes. Although it isn’t impossible, waiting 2 minutes for your code to deploy begins to irritate you. But your coworker mentions dploy.io and you install it in 30 minutes. Suddenly you have a scalable deployment method that deploys in less than 10 seconds and even has an intuitive web interface. Lovely. Why didn’t you figure this out earlier?
February 4, 2014
During my 4 years as an undergraduate computer science student at USC, I received hundreds of recruiting emails from business students looking for a technical co-founder or somebody to build their idea. Many of them were undeniably bad. Although I applaud all of these entrepreneurs for making an effort (however small), it bothers me that so many ideas die on the vine due to awful recruiting emails.
In order to help future entrepreneurs seeking technical help, here is a rough primer on how not to sound awful.
Let’s start by examining the average recruitment email:
LOOKING FOR A STUDENT TO WORK ON APP DESIGN PROJECT
Must be able to create a fully functional social app from scratch
Position responsibilities: will initially include building the front end and back end of our beta site. Must have a hacker attitude. If you are a Rockstar with machine learning, natural language, and data aggregation experience, you are amazing. Alright, we will talk to you if you could do only one of the three.
Compensation: Will be negotiable but ideally we are looking to bring someone into our team with programming expertise who really believes in our idea/business model. Candidates must sign an NDA before discussing the idea.
Duration: The initial duration will basically be as long as it takes to build the beta site with further opportunities available down the line.
Skills required: We believe they will need skills in PHP, MySQL, Ruby on Rails, Backbone.js, Angular, Databases, and Cloud Computing
Although initially the email looks organized and thorough, a more careful reading shows some glaring issues that immediately qualifies it as garbage.
- Mystery Product - There are exactly zero details on what the idea or project is about, resulting in exactly zero interest being sparked.
- The NDA - Do not ask people to sign an NDA. Ideas are a dime a dozen, it disrespects whoever you are sending the email to, and starts the conversation in a distrustful tone.
- “Hacker attitude / Rockstar / Amazing” - These buzzwords illustrate an ‘us’ vs ‘them’ attitude towards people with technical skills. Nobody in CS calls themselves a rockstar hacker.
- Compensation - The explicitly stated negotiation makes the relationship transactional, signifying the business student’s belief that technical skill is a mere commodity to be traded for money and equity, neither of which are incredibly satisfying on their own.
- Skills Required - Name-dropping frameworks and languages without any context demonstrates a total unfamiliarity with anything technical. Who would want to write software with someone who knows zilch about software?
Contrast this with an email that successfully recruited me to join a startup that would later be known as Tint
We are seeking an experienced back-end web developer to join our web startup. Our main shortage in talent is back-end development, and if you are passionate about making web applications and are looking for real-world experience, we are looking for you.
Hypemarks is an incubator-backed startup temporarily based in New York, started by five USC students. We see a huge problem of the amount of sites appearing everyday. With so many sites, how do you know which ones are relevant to you or worth your time checking out? Simply put: your friends.
We are a social bookmarking platform that collects all the sites you like and organizes them so you can share with your friends easily. Our vision is to allow people to define themselves based on their online experience when they share it with their friends. See an attached mockup to get a clearer idea of our vision: http://bit.ly/jYAJlg.
You are an experienced back-end developer not only excited to be part of a small startup co-founding team, but also motivated by personal ambition. You are flexible and learn quickly. As a co-founder, you will have tremendous ownership and responsibility on development. Communication will be key since we will be working remotely for the summer. We are a team of diverse USC students and alumni who are laid-back, yet focused on working to build something big and hope you will believe in our vision as well.
Passion for building web apps
Desire to learn a web application framework (or have knowledge in web application frameworks)
Interest in start-up culture
Strong database skills, preferably with MySQL
Experience in building web applications
What got me to respond to this email?
- Clear Product - Giving a clear picture of what the product vision is conveys both honesty and confidence in the idea. It also demonstrates that the business guy is capable of contributing to the project. The more links to prototypes / product decks, the better!
- No Buzz Words - Clear, down to earth language that is conversational, but professional. Increases the trust factor.
- Impact & Ownership - A clear overview of what is being offered, and an understanding that most technical people are interested in making an impact and taking an ownership. Being able to identify what motivates the person on the other end of the email is key to success.
- Technical Understanding - The “Desired Skills” section shows a good understanding of the general knowledge necessary without resorting to name-dropping frameworks or languages. Clearly the writer did the necessary homework to learn a bit about web application programming, an excellent sign.
Being able to clearly convey value proposition doesn’t just help sell your product, it helps build your team. At the early stages, it’s easy to think of finding a team as an afterthought to your brilliant startup idea. However, what I’ve learned in building Tint is that team is everything. Put as much effort into finding quality people as you would in finding a potential mate. If all goes well, your email might be the first step to building a lifelong work relationship.
My friend and CEO of Tint, Tim, has a good blog post from the other side of the table about how to convince developers to join your startup. Check it out.
Also, Drew Houston, Founder of Dropbox, has a few words to say on the topic of how to find a technical co-founder.
And a brief survey by Andrew Chen explaining what developers are looking for when choosing to work as a technical cofounder. Hint: It’s traction.
January 27, 2014
Our most recent hire, Brandon Ancier, said something at our last meeting that I had never considered before: “What I like about working here is that everyone one this team has business sense.” And as flattering as that is, I have to concede that he is right :) Looking back on how much our team has been able to achieve, the most important skill all of us have gained so far has been this “business sense”, which I define as the ability to identify optimal business decisions in a ambiguous, unknown environment. What are some of the perplexing decisions our team has to make on a day to day basis?
- What product features should I build next?
- Should the feature be self-service or custom built?
- How should I phrase the value-add on this product tour page?
- Should I answer the support emails first or the sales emails?
When questions like this come up, a good business sense allows us to come up with answers such as:
- Let’s write a feature story and send a survey out to the customers that would use this feature.
- Depends on what the survey says.
- Let’s get the Sales Team’s input on this. What language has worked for them in their pitches?
- Probably sales emails first, but let’s make it a priority to answer every email promptly even if we don’t know the answer. Let’s make sure they know we care.
See the pattern? The answer lies in empathy, being able to perceive the world though other people’s perspectives. If there were a single largest mistake I’ve seen cripple teams and throw wrenches in business processes, it’s lack of empathy.
The paradox is that in order to have [emotional] influence, you have to be influenced - The 7 Habits of Highly Effective People
Simply put, there’s no way of providing for a customers correctly without getting their input. Although Steve Jobs may have famously said, “Customers don’t know what they want”, consider that most of Apple’s products redefined entire product categories, and that’s not the domain of most businesses. If you’re like any regular business, you will want to constantly improve on your empathy. How?
What we do to improve customer empathy:
Give out your Cell Phone Number
Put your personal phone number in your communications with your most important customers. Encourage them to give you a ring! We’ve found great success in giving our largest customers multiple avenues to reach us. Although this might not scale in the future, we think it’s worth it.
Add customers to your instant messanger
Add your customers via gchat or skype and build a relationship. We do this with a few of our agencies, and it makes it incredibly easy to get instant feedback on something when you think to yourself, “I wish I could ask a customer right now!” Having customers know your company on a first name basis starts with knowing your customers on a first name basis.
Send nicely-crafted surveys
I’m surprised at how few companies can put together a survey that is nicely-crafted in that it asks the right questions and establishes a personal connection.
When we write surveys, we write it with the intention of nurturing a relationship with the customer who is gracious enough to give us feedback and we frequently follow up on our survey participants to go deeper into their needs. The people who are going to fill out your surveys are going to be your most vocal customers, a fantastic opportunity to nurture your business’ brand!
No one cares how much you know, until they know how much you care - Theodore Roosevelt
The best teams have highly attuned senses of empathy. The empathy concept goes beyond customer-facing decisions.
When it comes to building a positive workplace and striving for better employee retention, seeing decisions from your co-workers’ perspective is essential. Open communication is the only way to build lasting, meaningful work relationships. Teams that can overcome the emotional roller coaster of building a business need to trust that everyone stands behind each other. And empathy is the foundation upon all communication and trust rests. How does Team Tint do this?
What we do to improve team empathy:
All of our compensation structure is made public within the company so that everyone can have a full picture and make sure that they feel that all team members are being compensated fairly in relation to each other. Making salaries public fosters empathy because it forces other people to consider the perspective of others when discussing compensation.
Encourage Sales/Engineering cross communication
Ever since the beginnings of the sales team (with the hiring of Zen, our Director of Sales), the development team (which was Nik and I) have always strived to make sure communication between Sales and Development was tight. We want to make sure that either team is 100% comfortable talking to the other team, and this communication fosters inter-departmental empathy that manifests itself in a number of positive ways including:
- Less animosity between teams (fewer salespeople annoyed at bugs in the product and fewer developers annoyed of salespeople making promises the developers can’t fulfill)
- Practical ideas from Sales being implemented by the Devs, such as automating key parts of their workflow
- Efficient prioritization of inter-departmental tasks made possible by openly communicated cost/benefit.
We have each other’s backs
We understand that anyone can make a mistake, even our own teammates. By discouraging a blame-based culture, we build stronger bonds as well as encouraging people to speak the truth, to admit things that they feel are going wrong, and to work together to find solutions to inadequacies. Sure, blaming someone might be easier, but in the long run, a team that blames among itself is not a team, but rather a collection of disgruntled individuals.
In the end, cultivating empathy within yourself and among a team takes time, focus, and effort and often can’t be quantified. In an environment full of deadlines, it can be easy to be lazy and not think about the customers or teammates carrying you forward. We hope that sharing our steps to improve empathy can inspire you to improve your business and make choices to spend the resources to cultivate empathy. Let me know if you agree/disagree in the comments :)