Hi, my name is Ryo and I am a Cofounder and Co-CTO at TINT, a 35 person company that has a green selenium build. I’ve been a big advocate of selenium at our company and I want to tell you how we went from no test coverage to green selenium build and almost gave up on the way.
So, let me give you a quick background. At TINT, we make social media display software. A year and a half ago, we were 5 engineers and were closing deals with our first large customers. This is a picture of one of our first, the mall of Dubai!
The problem was, our software failed on a weekly basis due to our everyday deployments with no test coverage. You can guess that it resulted in A LOT of angry phone calls. With our product, you can only make a couple mistakes before your customers leave. Our company was teetering between failure and success, and instability was a make-it-or-break-it concern. This is when I started on our yearlong journey to a green selenium build.
When we started, our team had no resources or domain experience with selenium. We started out writing Selenium IDE tests, but were not able to integrate them into our workflow because they required so much time to supervise. We switched to ruby selenium webdriver and rspec, and were able to make progress, but we still felt like were not doing things the right way.
We almost gave up. Maybe selenium wasn’t worth all of the trouble after all.
We started asking questions, and here is what got us over the hump. First, we found The Selenium Guidebook / Elemental Selenium by Dave Haeffner, we read this and it allowed us to organize our tests, but more importantly, help us feel confident that what we were doing wasn’t foolish. Secondly, we met Neil Manvar at Sauce Labs through a mutual friend. Neil brought a lot of domain knowledge and more importantly provided emotional support in letting us know that selenium was not flaky and in fact we had written our tests poorly. It just goes to show the value of working your network!
We had no CI server, and were running tests manually. When that got too painful, I embarked on setting up a CI server. For about a month I spent time trying to get our unique Ruby on Rails / PHP environment working on Travis, Circle, and Codeship CI. I failed every time trying to get both apps running on the same server on these CI as a service solutions.
We almost gave up. Maybe selenium wasn’t worth all of the trouble after all.
Luckily, we discovered a couple things that got us over the hump. We stepped back and realized we were using the wrong tool for the job. Instead of using CI as a service solutions, we set up a TeamCity server that allowed us to customize how our environments are set up. Once the build was up and running, we noticed that it was still flaky. We wrote a custom re-run script that re-runs tests that have failed. This change took our build from an unusable 50% flakiness to a usable 10% flakiness. Once we had a consistently green build, we noticed that developer’s weren’t paying attention to it, so we integrated github with team city so that selenium builds would kick off automatically and developers would be reminded that they need to have a green build before merging any pull requests. Turning this on took developer participation from about 30% to 100% instantly. All of these tasks were small things we put off for a long time, but when we finally got around to implementing them, we wondered why we hadn’t done them sooner because of how much value they added for us. And although they don’t relate directly to selenium, they were essential in getting us to use it.
When we started, we had very little developer participation in writing selenium tests. Nobody remembered to write selenium tests with this PR’s, and setting up your environment to run a test was cumbersome. After all, in order to write or debug a selenium test, we would have to shell into a machine, set up a sauce labs tunnel, run our tests, and watch it on the sauce labs viewer. Debugging a selenium test on sauce labs was nearly impossible. We reached a point where maybe there was too much pushback on the team to make adequate integration test coverage a reality.
We almost gave up.
But, we discovered a few key things that got us over the hump. I started nudging people in code reviews and letting them know PRs would not be accepted without integration tests.Others started to enforce this rule and we began to build up coverage. We wrote a script to make it easy to run a selenium test locally with a single command. We set up an unlimited sauce plan to run concurrent tests without worrying about utilization costs. These small behavioral nudges added up in making writing selenium tests a habit rather than a chore.
I want to share a quote from my best friend from high school who is now a software developer at Slalom Consulting. We were hanging out and we naturally started talking about Selenium, and he said “We had a selenium suite, but it was flaky so we stopped using it.” They gave up.
We almost gave up. But we didn’t! We spent a year working on our build on-and-off, almost giving up. And now we’re able to reap the benefits of our green selenium build. The best practices might sound like it wouldn’t make a big difference. My one message to you is: Don’t give up.
The problem: Employees who stay with your company deserve to be rewarded with more salary and equity. How do you design a system that is fair, easy to understand, and easy to implement?
Our process: We asked friends in the industry to explain how promotion process works in their company and researched online to understand the status-quo for the tech industry. Based on our findings, we identified parts of other companies’ processes that we liked. Finally, we assembled a policy that felt the most in-line with the company culture.
Our first step was to ask our friends. Here’s what they had to say (I highlight the parts interesting to me):
It all depends on performance, very heavily weighted. The most obvious thing they do is to have a set of responsibilities for each level and see quantitative proof that the person meets the expectations of the next level. They do this by discussing among the next to next level guys. The most realistic estimate is 1-1.5 years at lower levels. Hike is subjective, again, by performance. They have a committee that decides the hike. it’s a combination of max pay per level, how good this person is, and how much he gets. As a common trend, they give more bonuses and fewer hikes. Because bonus is a one-time thing and hike is a commitment. I don’t know the qualitative number for hike, but bonus is around 10-12% for normal performance
Compensation is reevaluated every year according to market standards. Promotions are totally merit-based, performance-based, so it depends on the level you are operating on, consistently, for at least 6 months. We want people to be consistently performing at the next level for at least 6 months before promotion. Every 6 months, we have a review cycle. Managers get peer feedback on what the engineer accomplished, and managers from different orgs will review the anonymized engineer, and compare engineers at the same rank to calibrate for impact across orgs.
The employee who thinks he/she is eligible for promotion applies to their manager. Along with this application he/she gets recommendation from other engineers they worked with during the last 6 months. These peer employees write a single paragraph describing what they worked on how that employee did a good job. Once the application is in, the manger sits with the promotion committee and determines the case. A promotion committee is usually 2-3 top senior engineers who evaluate the work and give their opinion. Ultimately its the manager’s decision to promote that employee.
At Salesforce, my title transitions were: AMTS -> MTS: 13 months, 10% increase MTS -> SMTS: 15 months, 8.77% increase Here’s what I think is standard: AMTS -> MTS: 11-18 months MTS -> SMTS: 24 months SMTS -> LMTS: 24-36 months LMTS -> PMTS: 36+ months.
Based on the above accounts and other research we did, we picked out the patterns that stood out to us:
Performance Reviews - Everybody does them. People hate them. It seems that they are imperfect in many ways, but are often the best that can be done in tackling the impossible challenge of gauging how an employee has performed. Performance reviews are inherently challenging because:
Performance is subjective: People can easily have a different idea of how an individual is performing based on how much they like that person, their visibility of that role, and other non-objective factors. This is especially the case in startups where employees where multiple hats and are continuously redefining their role.
Criteria is subjective: It is difficult to set an expectation on what behavior results in a promotion when . Especially in a startup that does not have other employees to serve as data points in a compensation function.
Power Dynamics: It leads to a power dynamic where those conducting the performance review have power over the one being evaluated. Which is especially challenging for a startup which values collaboration over competition.
Stack Ranking - Also known as a vitality curve, this is the management technique of grading employees based on their individual productivity. Many large organizations such as Yahoo, Amazon, and Microsoft have implemented stack ranking with mixed results.
Peter Principle and The Law of Crappy People - The Law of Crappy People (TLCP) states: “For any title level in a large organization, the talent on that level will eventually converge to the crappiest person with the title.” Ben Horowitz has a good segment on mitigating TLCP that boils down to being very disciplined about defining skills at each level.
After absorbing the above research, we finally came up with a rough guideline to handle promotions: Promotions are done every quarter and generally the expectation for promotions are as follows, broken down by the frequency of promotions depending on performance. These frequencies are relative to start date or date of last promotion. Keep in mind that these are guidelines, not hard rules, and does not mean that circumstances outside of the guidelines will not be considered: 3 months - a promotion can be given in 3 months after an employee’s start date or after any promotion if evaluations show that the company would be in a more fair state if the employee was set at a higher multiplier. Promotions do not necessarily mean that your next promotion is delayed by 3 months. 6 months - an exceptional promotion given very rarely, to employees whose performance greatly exceeds expectations 9 months - a great promotion, given infrequently to employees whose performance exceeds expectations 12 months - a good promotion given regularly to employees whose performance meets expectations
Role Differences: The compensation vs. time (assuming constant performance) curves for sales and engineering are different when considering market rate and as such the guidelines may be applied differently depending on the role. See graphic below from https://www.wealthfront.com/tools/startup-salary-equity-compensation Seniority: The seniority of positions affects how frequently promotions are given. It is common for promotion frequency to decrease as the position becomes higher, and this will affect how the guidelines will be applied. Expectations Work smarter, not harder (aka work more efficiently, not more hours). There is no expectation for work outside of regular office hours, and you will not be held back for promotion because you didn’t work outside office hours.
ISSUES WITH OUR SYSTEM
Our system isn’t perfect. After having implemented it and running it for the past two quarters, we have identified challenges that we will want to solve as we grow:
No crisp definitions of skill levels at different skill multipliers. This has left us vulnerable to The Law of Crappy People. As the roles in our organization become more specialized, this issue will become easier to resolve.
No crisp definitions of responsibilities at different skill multipliers. This has caused employees to feel that there is no clear long-term path for career advancement in the company.
High overhead process: Currently the way we decide who gets promoted involves a lengthy representative nomination process with multiple handoffs and a committee meeting that needs to be scheduled around everyone’s busy schedules. This happens every quarter, and takes up a lot of time to organize.
Responsibility bottleneck: Right now the cofounders are the ones ultimately making the decision on promotions. This is a problem because as the team is growing, the cofounders are becoming more and more distant from the day-to-day tasks of everyone and less qualified to make judgements on compensation. In addition, our time just does not scale.
If you have any questions or feedback for us on what you’ve read or our promotion guidelines, feel free to tweet me @ryochiba and @tint . Would love to hear your ideas and share what we have learned!
A good partner understands pairing is a difficult skill that takes time and effort to cultivate. A bad partner expects others to be great pairs right off the bat.
A good partner seeks to understand their own bad pairing habits. A bad pair fails to think about their own pairing habits and only blames others.
A good partner brings up mistakes that they make and that others make in a way that turns a mistake into an improvement opportunity. A bad partner bottles up issues they have while pairing and allows the issues to harm relationships and their quality of life at work.
A good partner frequently syncs up on schedule and goals and is not afraid to adjust the schedule to make it more realistic. A bad partner does not talk about schedule or goals and avoids adjusting the schedule until the last minute, causing confusion and mismatched expectations.
A good partner sets small achievable goals and celebrates achieving them. A bad partner sets one big (likely unrealistic) goal and stresses out about achieving it.
A good partner clearly communicates availability. A bad partner disappears for unpredictable periods of time with no notice leaving the other partner to work alone and confused.
A good partner uses 2 input devices and 1 computer and avoids using 2 computers unless necessary. A bad partner defaults to using 2 computers and gets distracted / unpaired.
A good partner balances give and take in decision making. A bad partner will not be aware of the decision making balance and either be a bunny or Alpha Male.
A good partner is aware of skill differences and works to rebalance them by making the effort to be a mentor or a student.
A good partner will continue to help with an epic all the way to the finish line and after with deployment tasks and bug fixes. A bad partner will stop contributing to the epic once the feature is accepted and not help their pair get the feature stable and in production.
A good partner focuses on cultivating the following pairing roles
Mentor - explains concepts and reasoning with minimal judgement
Moleskine - maintains state and checks for edge cases and unexpected side effects
Captain - cultivates their own soft skills necessary in effective pairing and cultivating their partners as well
And a bad partner falls into the following bad pairing roles
Alpha Male - mainly takes in a give and take discussion
Superman - grabs keyboard and starts coding quickly and silently
Bunny - mainly gives in a give and take discussion
Rodolfo Valentino - never disagrees with their partner’s decision
Monk - does not acknowledge the validity of other’s ideas
Puppet Master - uses their influence to control the driver’s coding
Backseat Driver - uses their voice to control the driver’s coding
Auditor - nitpicks small, trivial decisions at the expense of larger and more important decisions
A good driver frequently asks for agreement on ambiguous choices. A bad driver makes difficult choices silently and does not ask the navigator for input.
A good driver makes easy decisions fast. A bad driver constantly asks for input on decisions that do not require input.
A good driver is aware of the navigator’s focus and will ask for input or suggest trading off if they notice the navigator is distracted.
A good driver thinks out loud and communicates intentions and reasons.
A good navigator is continuously sanity checking code. A bad navigator is checking their phone for text messages.
A good navigator anticipates next steps and maintains the overall state of the system, reminding the driver about necessary changes that may have been overlooked. A bad navigator is working on a bug while simultaneously attempting to pair, but not contributing to the pair.
A good navigator is aware of their own focus, and suggests switching roles if they notice that their focus is slipping. A bad navigator allows their focus to deteriorate and is afraid of suggesting to switch roles.
A good pair has built the rapport between them to share a few laughs while pairing and have fun.
I was having lunch with my friend in South Park. Earlier that week, he texted me about how we calculate our team’s salaries at TINT. He wasn’t sure if he was getting paid fairly at his startup and wanted to use TINT’s formula as a data point to figure it out. His questions sounded familiar because they were the same questions we asked ourselves when we set up our salary formula! The most important question being: “What is fair compensation for working at my startup?” As cofounders of TINT who’ve created a transparent salary structure that is now implemented for more than 20 employees, Tim, Nik, and I have collectively spent hundreds of hours discussing the answer. Here are my thoughts:
Before we start, we should address the question that was the first question my friend asked: “Is it fair to even ask?” e.g. Is it immoral to be that guy, the squeaky wheel that gets the grease? At TINT, we’ve solved this problem by being transparent, but my friend made me realize that not everyone has that luxury. From stories I’ve heard from other friends, it’s important to ask the question, especially if you don’t have transparency.
Another friend who works at a large undisclosed company in San Francisco told me about how her manager secretly offered an intern a higher salary than hers, even though she’s been working with the company for the past 2 years. Remember that asking the difficult questions about compensation may be the only way to move up. Don’t feel guilty about negotiating.
Fair compensation depends on 3 main factors, which are:
Is it fair in relation to other people on your immediate team?
Is it close to market rate?
Is it fair to YOU?
Is it fair in relation to other people on your immediate team?
At companies with closed salaries, the answer is usually off the table, and even asking the question can feel inappropriate. However, if you can find a way to ask your colleagues in a socially appropriate manner, you should learn more about how others are being compensated. Afraid of repercussions? In the US, the National Labor Relations Act makes it illegal to fire anyone because they shared their compensation within the organization.
Managers have long used salary information asymmetry to try to keep salaries as low as possible, hoping that employees will keep their heads in the sand. But salaries never stay secret forever. It’s a liability for organizations that hide salaries from employees, which makes it even more mysterious why the practice is mainstream. If you are still feeling weird about asking, read these two comment threads on Hacker News here and here.
Is it close to market rate?
Luckily, answering the second part of the fairness equation is easier than the first. There are a variety of resources to give you a general estimate and range.
The challenge in figuring out your market rate is that every company is slightly different, and every role varies as well. Your own market rate will likely be based on:
Company size/risk - Riskier businesses will usually involve sacrificing salary for equity. But you’ll have to judge whether the potential equity value, increased ownership and other perks of working on a smaller team will outweigh the sacrifice.
The role’s growth opportunity - If the role could be a valuable stepping stone in your career, it could possibly be worth a sacrifice in salary.
Job perks - Free lunch, 401k matching, mandatory vacation policy, and other perks should be included in your calculations regarding the total compensation package.
Is it fair to YOU?
The third part of the fairness equation is the most interesting because it is both the most personal and also the most important component of the equation. It encapsulates your current situation as well as intangible factors that can make the biggest impact in your day-to-day work life:
“The Happiness Factor” - A totally subjective value that depends on you. It represents how happy you would be working on this team because of the company’s mission, the people on the team, or the flexibility it’s structure allows. For many people, this is the most important factor.
Your potential growth opportunity - If you could grow into a role at an accelerated pace, management may be willing to pay more.
Your skills - Do you bring anything special to the table that few others could? List them out, and don’t be shy about bringing them up. An angel dies every time a talented but quiet employee gets screwed by their management.
How do I figure it out?
The resources below will give you a range and you will tweak that range yourself based on the conditions above.
The strong point of this chart is in how it visualizes the ranges of compensation and the ways it allows you to filter down into its dataset via role and company size. Just keep in mind that sometimes the data available is small, and that many other factors such as the Happiness Factor are not included in the compensation.
This is a great article if you’re wanting to convince management to adopt a more transparent approach to compensation. It reveals that many large companies do attempt to make some effort to make their compensation transparent, and that even the biggest names in the startup world recognize the fact that good compensation structures are founded on openness and formula.
Did you know that Wolfram Alpha can give you a cost of living index between two locations and visualize it? Pretty handy to figure out how location influences your salary range. Sometimes it can make a huge difference!
The worst thing you can do to yourself is to voluntarily remain ignorant about how fair your salary is. I guarantee that if you remain ignorant in a closed-salary organization, your salary will end up being unfair, maybe even less than the intern’s. Unfortunately, most organizations would prefer you to stay ignorant. Educate yourself and fight back against information asymmetry. Or even better, work for a company like TINT that prides itself on fair compensation and implements innovative compensation structures such as monthly company-wide bonuses. Your most valuable asset is yourself, so don’t let people have it for less than you deserve.
“Move fast and Break things” (MF&BT) The company mantra is as common in the Bay Area startup scene as a Chrome backpack. Why? It captures the essence of why startups run by 20-somethings in hoodies can make headway into markets monopolized by Fortune 500 behemoths. More generally, it describes why the Silicon Valley tech industry has thrived in the last half century. If you are involved in tech, it defines the culture here: Take risks. However, the phrase has become so common that seeing a startup poster with MF&BT has become the equivalent of a boat of rowers paddling above “TEAMWORK”. Plenty an engineer has worked with a team that espoused the notion without success. The thing is, it’s not enough to tell employees to MF&BT and expect a faster MVP in the same way that it is meaningless and silly to tell people to increase their “TEAMWORK”. These phrases become vapid in the mouths of management who have forgotten the hard work involved in creating human and computer systems that naturally embody these ideals. The mistake is in thinking that these behaviors are the root of why some startups are successful and some are not. It is a mistake to aspire to these ideals without understanding that teams MF&BT not because of a poster in the break room, or because management told them so, but because that is what naturally comes to them. Some teams just naturally MF&BT, others don’t. But why? Teams that MF&BT:
Are not afraid of repercussions when they break things
Quickly learn lessons when things break
Can “unbreak” their systems with minimal effort
To make this happen, they are usually good at creating policies and cultivating workplace habits that focus on feedback, learning from failure, and fault tolerance. Let’s examine these in more detail:
Employee feedback systems are boring, unappreciated, and hard. Creating a culture that encourages everyone to give feedback everyday is even harder. But doing so ensures that everyone can feel safe when they break things and know that it is sanctioned within the group.
When someone has a great idea, let them know! Feedback doesn’t just mean negative feedback. Positive feedback increases morale and helps communicate what everyone wants to see more of.
Cultivate your ability to give constructive criticism. This is a difficult skill that can take a lifetime to perfect, so start today! When having to give criticism, approach it as a challenge to improve your skills instead of an uncomfortable task to get over with quickly. There is nothing that you can read that will improve your skill here. Instead, find people who are known for giving good feedback and watch them. Then, find every opportunity you can to practice what they preach. I’ve found that people who are good at constructive criticism are great listeners, empathizers, and instinctively know how to walk the fine line between being realistic and demoralizing.
Make feedback a regular process. At Tint, we’ve instituted monthly feedbacks where every employee requests feedback from at least 3 others in the company, and we take company time to introspect and write thoughtful feedback about what we like and what we want to see improved professionally.
2. Learning from failure
Creating policies that allow people to examine failures without blame.
After every failure whether it’s a missed sales quota or system downtime, gather the team and figure out what went wrong, without blame, without shame. Etsy has a great blog post on how they conduct their blameless postmortems. The most important thing is to NOT ignore failure and to NOT point fingers.
Write down what went wrong, why it went wrong, and what was learned, and email it out to everyone on the team. It helps everyone understand that failures are okay as long as we learn from them and helps spread knowledge.
Write blog posts about your most notable failures. That way the community at large can see how you fixed the problem. You’re probably not the only one that failed!
Create a fail wall at the office! You can see from the photo above the Fail Wall created by Spotify engineers. They have a great video series on their engineering culture that describes how they promote failure.
3. Fault Tolerance
Being able to easily tell when things go wrong and being okay with changing direction quickly is easier said than done. For some occupations, it’s not even possible. Luckily, it’s probably possible for you.
Aspire to having great continuous integration - This means having high test coverage, fast build times, and as much automation as possible. Obviously, this takes a lot of work. But the faster it is to test code, the faster it is to write code.
If your team is large enough, aspire to have innovative continuous integration. Github’s ChatOps is a great example of how far operations can be taken.
Be disciplined about ROI. At TINT, we used to not formally measure the impact of going to social media conferences. However, we realized that without measuring impact, we can’t discern whether it was worth it or not.
Leadership should encourage experimentation. If someone has an idea that they think could improve the process, but involves some risk, they should be given the freedom to try it out! Teams that experiment naturally fail more (in a good way), and failing more will make a team naturally more fault tolerant.
None of these techniques make for a sexy inspiration poster, but iterating on HR surveys for employee feedback are just as important, if not more important, as how many features get shipped every month. Ironically, MF&BT requires the opposite of recklessness. It requires attention to detail and more importantly, discipline.
After taking income and subtracting savings, travel, transportation, and unexpected large costs, the remainder comes out to about $876/mo, which is $29.2 a day in food and discretionary spending. If I were to reduce this by 30% and save it, this would come to around three thousand dollars, which would increase my savings rate by only 4%, and given that a 30% cut in discretionary income would involve changes in my lifestyle, I would say that I did a good job this year of managing my money
appropriately. What this analysis says is essentially that my food and discretionary (not including travel) spend comes to use 13% of my annual income, which I am comfortable with.
Total travel spend this year comes down to approximately $3.5k, which is about 5% of my income, which I see as too little. In 2015 I want to travel more, and aim to increase that percentage to 7% which would correspond with a large trip every quarter.
As for savings goals, my goal is to continue to avoid lifestyle inflation and exceed a savings rate of 55% for the next year. If no large unexpected big costs occur, then both my travel and savings goals should be attainable. If some unexpected costs come up as they always do, I will probably cut into my savings rate rather than travel just because I may only have the next decade to travel with true freedom. Another goal for this year is to find a way to put more time into volunteering and philanthropy as I know many people don’t have the opportunity
to pursue financial independence.
reddit, HN, medium, vanity fair, chow, sfgate, nytimes: Top distracting sites in October
10 hours -> 4 hours: 60% reduction in time spent on distracting websites from October to November
63 hours -> 64 hours: negligible change in time spent doing software development
So over the course of the month, I added 6 more hours to my life by just reducing the time I vegetate!
The hardest thing was to stop the habit of opening a new reddit or hacker news tab during any downtime. It was practically muscle memory! I found myself opening and immediately closing tabs many times a day for the first week. However, the knowledge that my performance was being monitored by RescueTime helped keep me going.
One thing I anticipated was that I would be less up-to-date both in local and technical news. However, I was surprised to find the funniest or most important content being shared in the company chatroom anyway, filtered and curated by my friends. This didn’t end up being as large of a problem as I originally anticipated.
In the end, I felt more focused and able to direct my energy toward more difficult media consumption goals. I got halfway through a book that I would not have picked up if I didn’t do this challenge. I believe that it’s worth continuing this challenge into the future and hopefully I’ll have the discipline to do so.
Write down a list of alternate content to consume in advance.
Download books that you’ve been meaning to read and on your phone so you’re not languishing in the
Track yourself using RescueTime both as a motivation tool and to measure your performance change.