Two clueless consultants

How I hire a DBA or Database Developer

Livedrive is currently looking to hire a rock star DBA and developers with a strong understanding of databases. We have 100TB of mySQL data online and a SQL server mirror running at 15k tx/sec non stop (peaking at 200K) with a nice little 3TB OLTP system.

And my goodness – rock star DBAs are hard to find. For those of you looking for one – or thinking you are one – I wanted to write up my advice.

Here is how I hire a DBA:

Initial Screening and Recruiters

We rely on internal HR and personal connections to find CVs. Every day – I get 3-4 invitations on LinkedIn from recruiters. Half of these just invite me to connect with the default LinkedIn message. Those get rejected right away. The other half write some MBA-style empty talk about partnering with us and how they can help our business do blah blah blah (the word: “Value” sprinkled generously into the paragraph). Those get laughed at, then discarded. Only once have I gotten a nice letter from a recruiter with details of the candidates he has available showing some understanding of what we are actually looking for. That gets forwarded to our internal HR. Most recruiters, give recruiters a bad name.

Only twice have I recieved a CV directly from a real candidate. Why do job seeking candidates  bother with recruiters?

Phone interview

After perusing the CVs we receive, we discard about 75% of them outright. The ones we like – we arrange for a short phone call with the candidate. This call typically lasts between 15-20 min.

The interview is conducted by the COO and myself.

The purpose of the initial call is to understand what motivates the candidate to move to a new job. We also try to understand what the person actually did in her/his previous positions – especially what they are proud of or feel they learned from. If a CV has been “pimped” we will nearly always discover obvious flaws in the person’s skills at this point. Candidates who are obnoxious (no matter how skilled) also get discarded during this interview. We don’t hire jerks – it’s just not worth the time.

What positive signs do I look for during this phase?

First of all, passion. People who are passionate about their job will nearly always learn what they need to learn. We are not your run-of-the-mill installation – and we don’t expect people to hit the ground running. We have some really smart people on board that the candidate can learn from, if they have the passion to do so. It doesn’t matter if you previously failed miserably – as long as you have learned from it.

Second, courage and intelligence. A candidate who has learned a great many skills is highly rated with us. It doesn’t really matter if you don’t know MySQL, if you have already acquired MSSQL, Oracle and MongoDB (to take an example) skills instead. Similarly – we are a mixed tool shop (OSS and MS) – so we try to filter out obvious Linux bigots and MS worshippers.

We once interviewed a candidate who knew none of the skills we required – yet spoke 8 languages fluently and had changed career twice. We did not hire this person (there was too large a skill gap) – but this is someone who will do well and get past our initial filters.

We also answer any questions the candidate has. Even if we don’t like the candidate – we still sell ourselves to them. They will tell their friends – and we may hire those friends one day (if you must – think of this like good karma)

During this initial interview – we discard another 2/3 of the candidates. The last ones – we invite on site.

On site interview

The final stage our our process is the on site interview. There are a lot of things you can’t tell about a candidate until you meet them in person. Additionally – because of the way I do techical testing – I need them in front of a whiteboard. We tell the candidate to reserve half a day for this interview.

As the candidate arrives – we offer then a cup of coffee or tea and walk them to our meeting room set up for the purpose. This room is equipped with a table, a few chairs and the most important furniture: a whiteboard.

The first phase of the interview is conducted by the same two people who did the phone interview – the COO and me. We want the candidate to feel comfortable at this point and will start the meeting with some small talk, asking a few questions about them and who they are. Anyone who lied during the phone interview – we catch at this point (as sad as this is – this happened twice).

Once the candidate has gotten to know us – the grilling begins!

I first tell the candidate that we like to understand how they think about problem solving. I say – and this is true – that I am not looking for specific answers – just a discussion. I leaned this intro from reading Joel Spolsky.

Depending on my initial impression of the candidate’s skill level – I pick a “toy problem” that is not too hard and put it on the whiteboard. If the candidate appears nervous – we start slow. If the candidate is full of confidence – I hit them with a harder problem.

The purpose of the discussion that follows is:

  1. To find out how the person thinks about problem solving
  2. To show the person some real problems they will be dealing with (good candidates will love the challenges)
  3. To bring the person out of their technical comfort zone and see how they react

The last may seem cruel. But, I can assure you that until you see people admit they don’t know things, you don’t really know them. Nobody is perfect and reality will sometime hit you with unexpected situations – I need to know that I can rely on them to think straight when faced with a problem they don’t know the solution to.

After presenting the toy problem, I ramp up the difficulty of questions – typically using very open ended problems that allow the discussion to flow.

Good candidates exhibit some very specific traits during this phase.

First, they will eagerly engage in the discussion. Some people are shy and will remain seated at the table, but others will jump to the whiteboard and start asking clarifying questions. The good candidate will seize the change to prove him/herself.

Second, the candidate will look for paths to solutions. It is not uncommon to find candidates asking a lot of clarifying questions (which we encourage), but not really moving towards a solution while asking those questions. While it is admirable to seek understanding of a problem before jumping to conclusions; there is a point where you have to take and stand and put forth a straw man solution – even if it may be wrong. As an example, we once interviewed a really promising candidate who asked a lot of intelligent questions – but never really had the courage to propose a solution based on the answers. Such people can spend a lot of time in meetings, but they wont produce much code.

Finally, we are looking for people who rule out bad solutions when they see them. It has been said that experts are not better at solving things, they are just better at filtering obviously bad ideas. You will too frequently find candidates who will strongly defend the most obscure solutions to a problem by inventing non existing (or non important) issues that are addressed by those solutions. We need programmers, not politicians.

A Sample Toy Problem

The toy problems we give candidates are designed to move them a bit out of their “best practice and canned solutions” comfort zone. Any fool can read blog entries and whitepapers and follow the cooking recipes there, it takes skills to go beyond that. We are looking for rock stars after all.

To give you an idea of the flavour of problem I pick, here is one example (that mimics what we have in production):

“In our systems, we store metadata about a user’s files. The data is stored in a relational database for quick retrieval.

There are two ways to retrieve a file: by it’s name and by it’s unique id. In both cases, the account-id of the user is also passed to the database.

To make the example concrete, consider this schema:

CREATE TABLE FileMeta
(
Id [Some Data Type] NOT NULL
, ParentId [Some Data Type] NOT NULL
, Type CHAR(1) NOT NULL /* F = File or D = Directory */
, AccountId [Some Data Type] NOT NULL
, Name VARCHAR(255)
)

These two queries are the most frequently run (90% of the workload):

SELECT …
FROM FileMeta
WHERE AccountId = [Foo]
AND Id = [Bar]

and:

SELECT …
FROM FileMeta
WHERE AccountId = [Foo]
AND Name = ‘[Some Name]’

My question is: How would you finish the design of this table and it’s indexes?”

This apparently  simple question hides some quite complex considerations. Depending on how the candidate fares, I can take the questions in different directions and up the game. Examples:

  • What data type will you pick for Id?
  • Should you use cluster indexes or regular indexes?
  • What about the hierarchy implied by ParentId? Are there better solutions?
  • How will you store the Name to support multiple languages?
  • What about index sizes, if this table is large (we have 60TB of this stuff) – could you be smart about the indexing structures to locate the Name column?
  • Is the Type column the right way to model this problem?
  • If you were to implement file versioning, how would you extend this model?
  • With the indexes you proposed, could you draw me the expected query plan of this statement?
  • If I wanted to introduce a query that retrieves ALL the files from a user, would your indexing strategy then change? If yes, how? If no, what would the impact be?
  • Could you estimate the storage cost/row with your current solution?
  • What is the expected runtime of this query? What will it depend on and how will you size a server for it?
  • If this table grew very large, how would you partition it, and if you do why do it like that?
  • If you were to shard the workload, how would you approach this? Will your indexing strategy change based on that?

There are multiple right answers to these questions. The good candidate will be able to think of tradeoffs and list them out. I will then ask them to pick between the alternatives – allowing them to ask me clarifying questions to reach a conclusion (again, we try to filter out the empty “it depends” statements; you must take a stand). Sometimes, I will throw a spanner in the works and propose a really silly solution and see if the candidate has the chops to shoot it down.

Sometimes, I suspect a candidate is spewing best practices at me or they have read this blog and come up with pre-canned answers. If this is the case, I dig in to test their understanding. For example, if the person says that the Name should be Unicode to support international characters, I will ask them how they will determine the encoding to use for it if we were to support all of Mac, Linux and Windows. If the candidate offers platform specific solutions (for example the hierarchy data type for SQL Server) – I will ask them why that solution is better than a self join (or some of the OTHER alternatives).

Team interview

If the candidate does well during the grilling session, we invite their future team members to speak with them and introduce the candidate to the people they might end up working with. Typically, we will have 2-3 interview sessions where neither me, not the COO are present (to avoid influencing the decisions).

After each session, we debrief with the interviewer. If more than one person rejects the candidate – we do not proceed.

Final Phase

Once a candidate has been through the grilling session and team interview, we have enough information to make a choice. At this point, the candidate is most likely very tired. We thank them for their time and inform them of our decisions.

After this, salary negotiations begin and we hopefully reach a mutually agreeable solution.