Candidate telemetry diagnostic, error autopsy, and step-by-step query construction walkthrough.
Live aggregated metrics across candidate sandbox attempts
28 solved
First attempt fail
Evaluated submissions
Median time to solve
Unlocked answer
Before trusting any A/B test result, an analyst must confirm the traffic split actually landed close to what was configured. Your experimentation platform was configured for a 50/50 control/treatment split. After two weeks you observe:
Run a chi-square goodness-of-fit test against the expected 50/50 allocation and flag a Sample Ratio Mismatch (SRM) if the p-value is below 0.01 (the stricter threshold conventionally used for SRM, since even a small allocation bug can silently invalidate every downstream metric).
Return a dict result with keys chi2_stat (rounded to 4 decimals), p_value (unrounded), and srm_detected (bool).
Operating on a DataFrame slice without `.copy()`, producing `SettingWithCopyWarning`, or using `.apply(axis=1)` with custom Python functions when vectorized NumPy / pandas column operations could run 50x faster with 0 memory overhead.
Interviewers assess whether you write idiomatic, vectorized pandas code instead of slow row-by-row procedural Python loops.
Construct the solution logically from first principles to avoid typical edge case pitfalls.
Use boolean masks with `.loc[row_mask, col_list]` to avoid chained assignment warnings.
subset = df.loc[df['status'] == 'completed'].copy()
Use numpy/pandas native operations like `.groupby()`, `.transform()`, or column arithmetic.
df['rev_share'] = df['revenue'] / df.groupby('country')['revenue'].transform('sum')Sort values and reset index to match the required evaluation output contract.
result = df.sort_values('rev_share', ascending=False).reset_index(drop=True)import math
n_control_observed = 24100
n_treatment_observed = 25900
total_users = n_control_observed + n_treatment_observed
expected = total_users / 2
chi2_stat = (n_control_observed - expected) ** 2 / expected + (n_treatment_observed - expected) ** 2 / expected
# Chi-square with 1 degree of freedom: P(X > x) = erfc(sqrt(x/2)) -- exact
# closed form via the standard normal (matches scipy.stats.chisquare's
# p-value for a 2-category goodness-of-fit test without needing scipy).
p_value = math.erfc(math.sqrt(chi2_stat / 2))
result = {
"chi2_stat": round(float(chi2_stat), 4),
"p_value": float(p_value),
"srm_detected": bool(p_value < 0.01),
}
Real code patterns candidates submit that fail the grading suite.
results = []
for i, row in df.iterrows():
results.append(row['val'] * 2)Three recurring syntax and semantic traps relevant to this problem domain.
Modifying a filtered slice like df[df.age > 30]['salary'] = 5000 modifies an ephemeral copy instead of the underlying dataframe.
df[df['active'] == True]['status'] = 'verified' # ❌ SettingWithCopyWarning
df.loc[df['active'] == True, 'status'] = 'verified' # ✅ Idiomatic inplace assignment
Calling df.groupby('category').agg(...) places 'category' into the MultiIndex/Index, making subsequent column references fail.
res = df.groupby('dept')['salary'].mean(); print(res['dept']) # ❌ KeyErrorres = df.groupby('dept', as_index=False)['salary'].mean() # ✅ Keeps column intactIterating rows with for index, row in df.iterrows() is an anti-pattern in data science interviews.
for i, r in df.iterrows(): df.at[i, 'total'] = r['price'] * r['qty'] # ❌ Slow
df['total'] = df['price'] * df['qty'] # ✅ C-speed vectorization
Launch our in-browser coding environment. Run queries, view execution plans, and get instant comparative diff grading with no setup.