Dan Patiño
Experimentation is where analytics stops being descriptive and starts being worth real money, and it is also where most self-taught analysts have the biggest hole. Running a test is easy. Sizing it correctly, resisting the urge to peek, and knowing when a result does not support the decision someone wants to make is the actual skill.
What should an A/B testing course cover?
Statistical significance is the easy part and most courses stop there. Look for power analysis and minimum detectable effect, because that is what tells you whether a test is worth running at all. Then the failure modes: peeking, multiple comparisons, novelty effects and sample ratio mismatch. A course that does not distinguish correlation from causation is teaching a calculator, not a method.
Coderhouse Data Analytics
Coderhouse
Eleven weeks of live, practitioner-led classes covering SQL through window functions, Python and pandas, visualization and data storytelling, statistics, A/B testing and causal thinking, plus a module on working with ChatGPT and Claude while validating their output. Ends in a capstone on an ambiguous business problem, assessed by experts.
Live online classes
11 weeks (one live class per week plus 6 to 8 hours of practice)
Beginner (no prior analytics, SQL or programming experience required)
$595 (launch price, full course)
Yes, an industry-recognized certificate earned by passing an expert-assessed capstone; curriculum reviewed by Google, Microsoft and MongoDB
People who want analytics taught live by practitioners and are ready to put in 6 to 8 hours a week of real practice.
No job guarantee or recruiting team. It teaches SQL and Python rather than Excel or Power BI, and the weekly workload is high for a part-time course.
A/B Testing
Udacity (course by Google)
The long-standing reference free course on online experiments, built by Google: metric choice, variability, sizing, and the traps that invalidate a test.
Self-paced, pre-recorded
About 4 weeks of self-paced study
Intermediate
Free
No
Anyone who wants the standard reference treatment of online experiment design for free.
Dated in its examples and tooling, and there is no instructor or feedback on your work.
Statistics with Python Specialization
Coursera (University of Michigan)
A three-course university specialization covering distributions, inference and fitting statistical models, taught through Python notebooks.
Self-paced, pre-recorded
Self-paced, commonly 2 to 4 months
Beginner to intermediate
About $49/month on Coursera
Yes, university-issued certificate
People who want statistical inference taught rigorously, with Python as the vehicle.
Academic in framing: strong on inference, light on how these decisions get made inside a company.
DataCamp Data Analyst Career Track
DataCamp
Short video plus an in-browser code window with hints, repeated at high volume. Genuinely good at building syntax fluency in SQL and Python, and one of the cheapest options here.
Self-paced, in-browser exercises
Self-paced
Beginner to intermediate
About $12 to $33/month
Yes, platform certificate
People who want to drill SQL and Python syntax cheaply and already have structure in their week.
Exercises are short and heavily scaffolded; you rarely have to decide the approach yourself.
Google Data Analytics Professional Certificate
Coursera (Google)
The best-known entry point into analytics: eight courses covering spreadsheets, SQL, R and Tableau, with a capstone case study. Content quality is high and the brand carries weight with recruiters.
Self-paced, pre-recorded
Self-paced, commonly 4 to 6 months
Beginner
About $49/month on Coursera
Yes, Google-issued certificate
Beginners who want a recognizable brand on their resume and can hold a schedule on their own.
No live instruction and no deadline anyone enforces, which is where most self-paced learners stall.
How much do experimentation courses cost?
Udacity's A/B Testing course, built by Google, is free and still one of the clearest treatments available. The Michigan statistics specialization runs about $49 a month on Coursera and DataCamp $12 to $33. Coderhouse covers experiment design, power, MDE and causal thinking across two modules of its 11-week course at $595.
What do you learn in an experimentation course?
Framing a hypothesis and choosing a primary metric with guardrails around it, calculating power and minimum detectable effect before launching, running the test without peeking, interpreting a result honestly, and recognizing the pitfalls that invalidate conclusions. The best courses go on to causal inference for the many questions you cannot answer with a randomized test.
Do experimentation skills pay more?
The BLS does not publish a 'data analyst' occupation. The closest published code is Data Scientists (15-2051), with a median wage of $120,230 in May 2025, a 10th percentile of $67,240 that is the realistic entry-level reference, and projected growth of 35% through 2035. Operations Research Analysts (15-2031), at a $88,940 median, is the closer match for day-to-day analyst work. Experimentation is one of the clearer differentiators between reporting analysts and the product and growth analytics roles that sit nearer the upper percentiles.
Which A/B testing course should you take?
Udacity's free course is the best standalone introduction and has been for years, though its examples have aged. Michigan's specialization is the right pick if the gap you are closing is statistical inference rather than experiment practice. Coderhouse is the better choice if you want experimentation taught as part of how analysis actually gets done, with metric trees, cohorts and causal thinking around it, live over 11 weeks for $595.
What is the difference between A/B testing and experimentation?
A/B testing is one method; experimentation is the broader discipline of designing studies that support causal claims, including quasi-experimental approaches when randomization is not possible.
How much statistics do I need?
Enough to reason about sampling, variability, confidence intervals and hypothesis testing. You need the intuition and the ability to spot a bad conclusion far more than the derivations.
What is minimum detectable effect?
The smallest true difference a test can reliably detect given your sample size. Calculating it first tells you whether the test can answer your question at all, which is why it comes before launch.
Why is peeking at results a problem?
Checking repeatedly and stopping when the result looks significant inflates false positives substantially. Either fix the duration in advance or use a method designed for sequential testing.
Can you prove causation without an A/B test?
Not with the same confidence, but causal inference methods can get you closer when randomization is impossible, which covers a large share of real business questions.
![](data:image/svg+xml,<svg display="block" id="329064441" role="presentation" viewBox="0 0 336 42" xmlns="http://www.w3.org/2000/svg"><path d="M 19.855 0 C 30.245 0 38.207 7.461 39.519 17.181 L 27.893 17.181 C 26.795 13.733 23.708 11.404 19.855 11.404 C 14.96 11.404 11.33 15.076 11.33 20.028 C 11.33 24.979 14.96 28.651 19.855 28.651 C 23.73 28.651 26.828 26.297 27.91 22.817 L 39.525 22.817 C 38.232 32.516 30.261 40 19.855 40 C 8.415 40 0 31.043 0 20.028 C 0 8.957 8.415 0 19.855 0 Z M 60.648 0 C 72.033 0 80.503 8.957 80.503 20.028 C 80.503 31.043 72.033 40 60.648 40 C 49.208 40 40.794 31.043 40.794 20.028 C 40.794 8.957 49.208 0 60.648 0 Z M 60.648 28.651 C 65.488 28.651 69.118 24.979 69.118 20.028 C 69.118 15.076 65.488 11.405 60.648 11.405 C 55.754 11.405 52.124 15.076 52.124 20.028 C 52.124 24.979 55.754 28.651 60.648 28.651 Z M 97.886 0.723 C 109.326 0.723 117.411 8.957 117.411 20.028 C 117.411 30.876 109.161 39.277 97.721 39.277 L 81.772 39.277 L 81.772 0.723 Z M 97.721 28.261 C 102.782 28.261 106.081 24.979 106.081 20.028 C 106.081 15.076 102.782 11.794 97.721 11.794 L 93.047 11.794 L 93.047 28.261 Z M 144.017 11.905 L 129.718 11.905 L 129.718 15.577 L 144.017 15.577 L 144.017 24.534 L 129.718 24.534 L 129.718 27.983 L 144.017 27.983 L 144.017 39.277 L 118.443 39.277 L 118.443 0.723 L 144.017 0.723 Z M 168.076 23.977 L 175.663 39.277 L 161.638 39.277 L 158.598 25.035 L 156.798 25.035 L 156.798 39.277 L 145.524 39.277 L 145.524 0.723 L 161.143 0.723 C 168.293 0.723 174.123 5.786 174.123 13.185 C 174.123 17.858 171.156 22.086 168.076 23.977 Z M 159.989 17.024 C 161.748 17.024 162.848 16.022 162.848 14.298 C 162.848 12.629 161.748 11.516 159.989 11.516 L 156.798 11.516 L 156.798 17.024 Z M 197.667 0.723 L 208.942 0.723 L 208.942 39.277 L 197.667 39.277 L 197.667 24.701 L 188.207 24.701 L 188.207 39.277 L 176.932 39.277 L 176.932 0.723 L 188.207 0.723 L 188.207 14.576 L 197.667 14.576 Z M 237.822 38.49 C 235.466 39.463 232.853 40 230.065 40 C 227.264 40 224.644 39.463 222.285 38.49 Z M 243.933 34.472 C 243.343 35.046 242.716 35.583 242.055 36.079 L 218.052 36.079 C 217.391 35.583 216.765 35.046 216.176 34.472 Z M 247.096 30.454 C 246.767 31.008 246.412 31.544 246.031 32.061 L 214.082 32.061 C 213.703 31.544 213.349 31.008 213.021 30.454 Z M 224.216 26.436 C 224.951 27.119 225.815 27.664 226.778 28.043 L 211.808 28.043 C 211.587 27.519 211.387 26.983 211.21 26.436 Z M 248.915 26.436 C 248.737 26.983 248.536 27.519 248.314 28.043 L 233.321 28.043 C 234.277 27.664 235.135 27.119 235.867 26.436 Z M 221.84 22.418 C 221.988 22.983 222.187 23.52 222.433 24.025 L 210.59 24.025 C 210.488 23.497 210.407 22.961 210.345 22.418 Z M 249.784 22.418 C 249.722 22.961 249.64 23.497 249.538 24.025 L 237.643 24.025 C 237.889 23.52 238.088 22.983 238.235 22.418 Z M 230.065 0 C 241.443 0 249.909 8.946 249.919 20.008 L 238.534 20.008 C 238.525 15.066 234.898 11.404 230.065 11.404 C 225.177 11.404 221.551 15.066 221.541 20.008 L 210.21 20.008 C 210.221 8.946 218.632 0 230.065 0 Z M 267.193 0.362 L 278.633 0.362 L 278.633 26.064 C 278.633 33.574 273.628 39.638 264.883 39.638 C 256.193 39.638 251.189 33.574 251.189 26.064 L 251.189 0.362 L 262.628 0.362 L 262.628 25.229 C 262.628 26.676 263.288 27.955 264.883 27.955 C 266.533 27.955 267.193 26.676 267.193 25.229 Z M 299.849 16.94 C 303.149 18.609 306.394 21.836 306.394 26.843 C 306.394 34.353 301.004 39.973 292.919 39.973 C 284.889 39.973 279.665 34.131 279.665 26.843 L 279.665 25.563 L 289.785 25.563 L 289.785 26.286 C 289.785 28.234 290.774 29.847 292.699 29.847 C 294.239 29.847 295.229 28.901 295.229 27.51 C 295.229 26.453 294.679 25.73 293.744 25.229 L 286.429 21.502 C 283.074 19.833 280.105 15.939 280.105 11.321 C 280.105 4.979 285.164 0.028 293.029 0.028 C 301.004 0.028 305.734 5.146 305.734 11.099 L 305.734 12.601 L 296.329 12.601 L 296.329 12.156 C 296.329 10.654 295.229 9.152 293.359 9.152 C 291.819 9.152 290.994 10.042 290.994 11.21 C 290.994 12.489 291.874 12.935 292.644 13.268 Z M 318.7 11.905 L 333 11.905 L 333 0.723 L 307.426 0.723 L 307.426 39.277 L 333 39.277 L 333 27.983 L 318.7 27.983 L 318.7 24.534 L 333 24.534 L 333 15.577 L 318.7 15.577 Z" fill="rgb(23, 23, 23)" height="40.00000265962436px" id="sLX_26zbq" transform="translate(1.593 1.059)" width="332.99999286102025px"/></svg>)