August 14, 2014
An open letter to the tech workers of san francisco
Dear fellow tech workers,
“We interviewed a senior being evicted from their home in the Mission who said, ‘Google is Hitler’. What would you say to that?”
An interviewer from TechCrunch asked me this question a month ago. The question didn’t surprise me, even though it should have. It seems like that’s all that’s been in the news lately.
The same week, I went to a Youth Speaks poetry slam with Monica. This was the first poetry slam that I’ve ever been to, and I was excited to hear youth speaking out about the issues they hold dearest. The event was fantastic and it inspired me to see youth cultivating their creativity.
But it wasn’t long until a slam came up accusing “toxic” tech workers of ruining the city:
Link to video
On valencia now that's all you see. It's spreading. Like an airborne toxicity. And that's exactly what I mean, it's a toxic city. So they force us out. Both young and old. Raised up the cost of living, no rent control. So if we can't afford to live our only option is to die or move out to Tracy or Antioch like a couple of my guys. While I'm in my city, they're out in the burbs. Not to mention that Twitter and Google are too strung up for words. They're speechless. Denying the fact that the only ones who can afford to live here now are the ones that are Google bussed in. Like they're employees from the mystical wonderland called the valley of silicon. It's really damn sickening, and I'm a 19 year old mother f**cking San Franciscan, damn. - Jerome Robles-Reyes "In My City"
It seems, from all fronts, that the city hates tech workers. Even SF Streetsblog, a blog I hold near and dear as a daily cyclist, declares the tech community as a monoculture that “blames those less wealthy for their own problems”.
Monocultures serve no one, including those whose culture takes over. - Fran Taylor [SF Streetsblog](http://bit.ly/1Ad0qEo)
From these articles, I should be ashamed. I should move back to where I came from. I guess that would be Indiana.
But I’m staying in San Francisco. The solution to evictions is building more housing. But building more housing isn’t going to conquer the root problem which is the animosity many native SF’ers have against people who work in software.
Instead of leaving, I’m going to see all the hate as a challenge to become a better member of the local San Francisco community. I think as tech workers we can make a big difference in public perception with consistent, everyday steps that any techie is capable of doing. You don’t need to be a community organizer to make things happen. A community is just a bunch of ordinary folks having relationships with each other.
I did some research, and apparently there are 20 ways to not be a gentrifier as described by local paper Oakland Local. It inspired me to make a list of my own:
Go get a haircut at a local barbershop or hairdresser (price must be < $15 (guys) or < $30 (gals)). Talk to your hairdresser. Talk about the car accident that happened down the block last weekend. Talk about the traffic issues from Outside Lands. And listen. Learn what’s on the mind of folks in the community.
Read and talk about local news. Be aware of the pulse of the city and about what’s affecting everyone, not just the software industry.
Get involved in local volunteerism. - This summer I helped Doug, a local SFUSD high school teacher, in an externship hosted at Tint. He learned technical skills with us that he can bring to the classroom in the upcoming school year. This fall, I hope to mentor local high school students so they too can learn how to write code. There are lots of resources for you out there, you just have to look! For starters, check out SF Citi or Mission Bit.
Participate in local art. It could be as simple as going to a poetry slam or an art walk, or go even further! My friend and colleague Brandon is a great example for this. He’s working with a local organization called Clittorati on the Vulvatron. What could be more SF than a visually iconic mobile art piece, empowering women, goddesses, and the feminine identity?
Don’t talk down to people less fortunate than you - I once met a fellow tech worker who condescendingly referred to the 38 as the ‘dirty eight’. As someone who rides the 38 every day, it made my blood boil to hear that comment. I finally knew how it felt to hate techie outsiders. Don’t reinforce negative stereotypes.
These are just a small subset of the many things that can be done to cultivate a community and dismantle the image of the evil techie outsider. But, the biggest change that anyone can make is to treat everyone from all walks of life with respect. Even with the fairest of intentions, it’s easy to seem condescending to outsiders, so it’s our responsibility to think and act actively to participate in the community.
May 3, 2014
6 months ago, Nikhil and I were the only developers at our 4 person startup. With business growing steadily, we were so spread thin that there was no hope of improving our product if we didn’t bring more help onto the team. So, Nik and I put our recruiting hats and began our journey to find talented engineers to join us. Fast forward 6 months to the present, and our Engineering team is about to grow to 7 (including 2 interns!), and I can safely say that I’ve learned a whole lot in the process:
Never consider recruitment work “a waste of time”
Time spent finding the right people for your team lays the foundation for everything else at a startup. A great product starts with a great team. So no matter how disheartening it feels to comb through resume after resume and still not find the right fit, always remember that recruitment work is as important as building a new feature or optimizing a process. So do yourself a favor and put quality time into doing the following:
Work your network
Our second engineering hire, Brett, came from Nikhil’s extended network and not from any job board or recruiting company. You never know who’s looking and with social media, it’s easier than ever to let all of your friends know that you’re looking to hire. It’s also easier to bring someone onto the team if they’re vetted by a friend than if they are a stranger. Not only do you feel like you can trust their competency, they can also better trust your competency!
Post a Quality Job Post
Know what people are looking for in their next job.
Hint: it’s probably Mastery, Autonomy, and Purpose
And know why your job is what people are looking for.
Your job post should highlight your strengths.
For example, our strength is our company culture. Our mission is to build a company culture that champions transparency, fairness, happiness, and sustainability. And we make sure to highlight that in our job posting:
- Profit Sharing - We split 20% of all revenue made over 100k and distribute it evenly among the team.
- Team Transparency - We calculate compensation based on a formula that we all agree on. Cap table is made available to all employees. Business financials are known by all teammates.
- Personal Autonomy / Consensus Driven Culture - We foster consensus-driven rather than top-down decision making when it comes to important business decisions. From what features to build next to what furniture to buy for the office, we believe it’s the fairest way of making decisions.
- Customer Driven Culture - We’re very in-tune with our customers and they love us. For example, we decide what features to build based on surveys we send directly to customers. Check out this one that we sent out last year to decide what we would build this past quarter.
- Personal Development Stipend - A monthly stipend designed for self-improvement. Whether it’s books, yoga classes, or a fitness tracker, we want our teammates to improve themselves.
Send Quality Emails
Quality recruiting emails are emails that recognize and understand the candidate. Here are some tips to add some empathy to your correspondance, embedded in a sample Tint recruitment email:
Thanks for scheduling a time with me! To prep for our interview, I’d recommend reading up on our company, getting familiar with what we do, and coming up with a few questions to ask us.
Here are some helpful links to peruse:
- Give the candidate a small assignment to assess their interest in the listing.
- Arm your candidate with the basic knowledge you expect them to know so you can have a productive discussion.
- Give the candidate the motivation they need to get excited about the opportunity.
Protip: Use Assistant.to to schedule your interviews. It’s a Gmail extension that allows you to easily give candidates a way to instantly book a meeting with you and have it show up in your calendar!
Protip 2: Use Yesware to create templates for your common recruiting emails, saving you further time.
Use a CRM
Handing resumes manually through email is incredibly time consuming. Use one of thousands of resume tracking Applicant Tracking Software (ATS in recruiter lingo) such as Resumator, Jobvite, or JobScore to simplify your life.
Find creative places to post to
A job listing link can travel far! But it’s your job to take it there. Consider the following places we posted our link to:
- Craigslist - We posted our listing on 10 major metro tech centers advertising paid relocation and had some success on attracting some good candidates. At $25 a posting, it was an affordable way to reach attractive candidates in markets that have much less competition than San Francisco.
- Hacker News - We found some quality candidates (including one of our interns) from posting our listing as a comment within the monthly “Who’s Hiring” thread. It gets posted on the 1st of every month, so don’t miss out!
- Indeed/Careers/Monster - Surprisingly, these mainstream job boards are frequented by talented people too! Most ATS systems will post to these major boards automatically, so be sure to configure your system to do that.
- Github Jobs - We found some alright leads from this paid posting, fewer applicants but the average quality was higher.
- StackOverflow Careers - We paid to run a campaign on StackOverflow but found that all of the submissions were overseas Java developers at big corporations looking for visa sponsorships. Maybe we were doing something wrong, but we ended up asking for a refund.
- Reddit - Plenty of subreddits to explore if you’re looking to find a community of people who you think would be a good fit. Think /r/bigdatajobs or /r/sysadminjobs
If you’re looking to expand your team, you have to recruit like a pro. It’s better to do things thoroughly from the get-go than to lukewarmly recruit for a longer period of time. Follow the tips above, and finding an engineer in San Francisco shouldn’t be as impossible as everyone says.
April 8, 2014
I’ve been meaning to set up a build / integration server for the past year but haven’t gotten around to it for a myriad of reasons. Last week, I had enough with the
- Features breaking every time a new feature is released (regression)
- Manually smoke testing URLs
- Having no structure for developing/testing new features
So, I decided to setup a Continuous Integration system for Tint! Here’s some notes on what I found as I navigated the confusing waters of setting up a build server.
Continuous integration: the practice, in software engineering, of merging all developer working copies with a shared mainline several times a day.
Outline what your needs are with the build server. For example, my needs were:
- Run selenium tests, preferably in parallel
- Be triggered by Github pull requests and git pushes
- Have an easy to use UI to see breaking builds
- Have easy integration to email, HipChat, and Github
- Travis CI - I used them over CircleCI because Travis seemed to have more industry adoption and also had a better UI and documentation.
- Sauce Labs - The leader in Selenium Grid SAAS, they also do a lot of active development on open source Selenium projects such as Selenium Builder, which is cool.
- Ruby/Rspec/Rake - Wanted to use a language that had strong automation tools around it and was low on verbosity, yet still easily readable, so we went with Ruby and company.
- Of course, there are many, many alternatives to Travis and Sauce (I actually started this project using CircleCI and BrowserStack), however, I chose Travis and Sauce in the end because they had more documentation and were easier to use.
- (Optional) Use The Selenium Guidebook to setup a framework for your Selenium Tests, and to learn how to write maintainable tests. Make sure all the tests run on your Cloud Selenium tool you plan on using (Sauce or BrowserStack) before going forward.
- Make an account with TravisCI, and turn on the repo that you want to set up a build server for.
- TravisCI will automatically hook into Github events so that it will trigger a build on pull requests and pushes. It uses a .travis.yml file in the root of your directory to figure out how to create your build server.
- Configure your .travis.yml so it builds your server properly. For example, in our travis.yml file, we clone our puppet manifest and then use puppet to create our webapp server and handle package dependencies, virtualhost files, random config files, and starting the services.
- Once the server is built properly, you can use Sauce Lab’s Connect feature to run Selenium Webdriver test on Sauce against your build server. Unfortunately, how this all works is not adequately covered in the Sauce Labs documentation, so bear with the magic (this is literally what Sauce Labs gives you for documentation).
Debugging Your TravisCI Build Server
If you’re having trouble creating a working build server, you can email firstname.lastname@example.org nicely and they will allow you to setup a debug build server for you to log into to test things out. However, according to their support team, running a Vagrant VM using the default precise32 ubuntu image is very close to their current setup, so consider that as an option as well.
I highly recommend using Puppet to simplify your build process. Puppet is also useful outside of setting up CI as it allows you to easily configure many servers quickly (for example, adding a new virtualhost file to 10 servers in one shot), and does it in a way that is maintainable and version controlled.
Even with all of these useful tools, it took me a good couple of days to get the build up and running, so don’t be discouraged by how tedious it may seem. The only way to really debug a build server at the moment seems to be to make changes, rebuild the server, see if the build goes farther, and then repeat until everything is working as expected.
Weird things I ran into
When sauce labs tells you to insert the following into your travis.yml:
curl https://gist.github.com/santiycr/5139565/raw/sauce_connect_setup.sh | bash
It should actually be
curl https://gist.githubusercontent.com/santiycr/5139565/raw/sauce_connect_setup.sh | bash
Otherwise, you’ll get a strange “Connection refused” error.
Trying to load a second private repo in your Travis build server will result in a “Repository not found” or “Authentication failed” error which can only be fixed using this obscure support article.
March 26, 2014
I was running into the issue where our CPU on our nginx webapp servers was not being fully utilized, and caused timeouts whenever CPU went above about 10% and memory was hardly being used. I had tried changing the configurations for nginx in the past with no success, so things were getting out of hand. When our traffic spiked yesterday morning due to Google Cloud Developers Conference, Where they are using Tint, we went down, and I had to increase our server count to 20 8GB 8vCPU servers.
Twenty servers to handle 20RPS just seemed ridiculous to me since nginx can handle thousands of RPS on a tuned machine. So I spent a couple of hours yesterday formulating a process to guess and check the effects of the server configurations in order to find out what was causing the issue.
- Isolate a single production server by removing it from all load balancers.
- Set up a Blitz.io account and validate the server in step 1 using the various methods outlined within blitz.
- Load test the server to see its performance.
- Shell into the server and change the server configuration, I was experimenting with /etc/nginx/nginx.conf and /etc/php5/fpm/pool.d/www.conf (don’t forget to restart the server)
- Load test the server while running ‘top’ and see if the performance changed.
Those 5 steps allowed me to finally figure out a combination of settings that allowed nginx and PHP to better utilize the CPU.
Server Configuration Changes
I changed pm.max_children = 5 to pm.max_children = 375
See the links below for more details on what these settings mean.
- All of our traffic (~1600 concurrent users on Google Analytics realtime overview) can be handled by a single server with these new configurations. CPU of the single server handling all of our traffic was ~40%.
- 6 of these servers behind a load balancer, an average of 53RPS could be handled while keeping response time less than 1s. Usually our RPS is around 5-15.
March 12, 2014
Yesterday, we ran into an issue where we were trying to release our white-label app in the Apple App Store. We wanted to submit it under a pseudonym or anonymously, however the App Store requires you to use a credit card with a name attached to it, and uses that name when you publish in the App Store. This is obviously unacceptable for a white label app, since searching any of our teammate’s names will quickly lead back to our company, and it put us in a bind because creating a
credit card under a pseudonym is a pain in the ass.
So, after some sleuthing around, we found a credit card masking service called Do Not Track Me run by a company called Abine. Some further sleuthing revealed that this is in fact a legitimate company as cited by Forbes. We signed up for the premium monthly service, created a masked card, and voila! We were able to sign up to an Apple
Developer Account under a pseudonym.
February 20, 2014
You wake up one day and decide to build an app. You spin up a micro instance and start developing. Deployment is simple. Just git pull and you’re done. When you expect more people to start loading the site, you spin up a couple more micros and put them behind a load balancer. This is the first time you’ve ever used these fancy load balancers, and everything seems to be humming along. Although, it gets annoying that you have to shell into each machine to update your code.
A couple weeks set in and it gets annoying to shell into each and every server to tweak anything. You do some sleuthing online and you read about Capistrano. So you set it up and after a half a day of being stuck on an SSH key issue, and another half a day tweaking the ruby code (while learning ruby at the same time), you can deploy with a single command. Yay.
Except, the only hitch is that the site is down for 30 seconds to a minute while the deployment happens. Not a big deal when your site has a handful of users on it concurrently. But as you grow, more and more users notice when you have downtime and you cannot ignore the issue any longer. You can’t figure out why Capistrano is causing the downtime anywhere on the web. Sigh. So, you decide to write a script that interacts with AWS to stagger Capistrano deploy commands and remove servers from the load balancer before deploying. After learning the AWS CLI, you cobble together a python script that results in zero downtime deployments.
Except, the script takes about 1 minute to run. Then, you have to double the number of servers since more and more people are using your service. That doubles deployment time to 2 minutes. Although it isn’t impossible, waiting 2 minutes for your code to deploy begins to irritate you. But your coworker mentions dploy.io and you install it in 30 minutes. Suddenly you have a scalable deployment method that deploys in less than 10 seconds and even has an intuitive web interface. Lovely. Why didn’t you figure this out earlier?
February 4, 2014
During my 4 years as an undergraduate computer science student at USC, I received hundreds of recruiting emails from business students looking for a technical co-founder or somebody to build their idea. Many of them were undeniably bad. Although I applaud all of these entrepreneurs for making an effort (however small), it bothers me that so many ideas die on the vine due to awful recruiting emails.
In order to help future entrepreneurs seeking technical help, here is a rough primer on how not to sound awful.
Let’s start by examining the average recruitment email:
LOOKING FOR A STUDENT TO WORK ON APP DESIGN PROJECT
Must be able to create a fully functional social app from scratch
Position responsibilities: will initially include building the front end and back end of our beta site. Must have a hacker attitude. If you are a Rockstar with machine learning, natural language, and data aggregation experience, you are amazing. Alright, we will talk to you if you could do only one of the three.
Compensation: Will be negotiable but ideally we are looking to bring someone into our team with programming expertise who really believes in our idea/business model. Candidates must sign an NDA before discussing the idea.
Duration: The initial duration will basically be as long as it takes to build the beta site with further opportunities available down the line.
Skills required: We believe they will need skills in PHP, MySQL, Ruby on Rails, Backbone.js, Angular, Databases, and Cloud Computing
Although initially the email looks organized and thorough, a more careful reading shows some glaring issues that immediately qualifies it as garbage.
- Mystery Product - There are exactly zero details on what the idea or project is about, resulting in exactly zero interest being sparked.
- The NDA - Do not ask people to sign an NDA. Ideas are a dime a dozen, it disrespects whoever you are sending the email to, and starts the conversation in a distrustful tone.
- “Hacker attitude / Rockstar / Amazing” - These buzzwords illustrate an ‘us’ vs ‘them’ attitude towards people with technical skills. Nobody in CS calls themselves a rockstar hacker.
- Compensation - The explicitly stated negotiation makes the relationship transactional, signifying the business student’s belief that technical skill is a mere commodity to be traded for money and equity, neither of which are incredibly satisfying on their own.
- Skills Required - Name-dropping frameworks and languages without any context demonstrates a total unfamiliarity with anything technical. Who would want to write software with someone who knows zilch about software?
Contrast this with an email that successfully recruited me to join a startup that would later be known as Tint
We are seeking an experienced back-end web developer to join our web startup. Our main shortage in talent is back-end development, and if you are passionate about making web applications and are looking for real-world experience, we are looking for you.
Hypemarks is an incubator-backed startup temporarily based in New York, started by five USC students. We see a huge problem of the amount of sites appearing everyday. With so many sites, how do you know which ones are relevant to you or worth your time checking out? Simply put: your friends.
We are a social bookmarking platform that collects all the sites you like and organizes them so you can share with your friends easily. Our vision is to allow people to define themselves based on their online experience when they share it with their friends. See an attached mockup to get a clearer idea of our vision: http://bit.ly/jYAJlg.
You are an experienced back-end developer not only excited to be part of a small startup co-founding team, but also motivated by personal ambition. You are flexible and learn quickly. As a co-founder, you will have tremendous ownership and responsibility on development. Communication will be key since we will be working remotely for the summer. We are a team of diverse USC students and alumni who are laid-back, yet focused on working to build something big and hope you will believe in our vision as well.
Passion for building web apps
Desire to learn a web application framework (or have knowledge in web application frameworks)
Interest in start-up culture
Strong database skills, preferably with MySQL
Experience in building web applications
What got me to respond to this email?
- Clear Product - Giving a clear picture of what the product vision is conveys both honesty and confidence in the idea. It also demonstrates that the business guy is capable of contributing to the project. The more links to prototypes / product decks, the better!
- No Buzz Words - Clear, down to earth language that is conversational, but professional. Increases the trust factor.
- Impact & Ownership - A clear overview of what is being offered, and an understanding that most technical people are interested in making an impact and taking an ownership. Being able to identify what motivates the person on the other end of the email is key to success.
- Technical Understanding - The “Desired Skills” section shows a good understanding of the general knowledge necessary without resorting to name-dropping frameworks or languages. Clearly the writer did the necessary homework to learn a bit about web application programming, an excellent sign.
Being able to clearly convey value proposition doesn’t just help sell your product, it helps build your team. At the early stages, it’s easy to think of finding a team as an afterthought to your brilliant startup idea. However, what I’ve learned in building Tint is that team is everything. Put as much effort into finding quality people as you would in finding a potential mate. If all goes well, your email might be the first step to building a lifelong work relationship.
My friend and CEO of Tint, Tim, has a good blog post from the other side of the table about how to convince developers to join your startup. Check it out.
Also, Drew Houston, Founder of Dropbox, has a few words to say on the topic of how to find a technical co-founder.
And a brief survey by Andrew Chen explaining what developers are looking for when choosing to work as a technical cofounder. Hint: It’s traction.