Lessons learned from 127,000 experiments
CRO weekly #183
Hi,
Welcome to the first CRO weekly newsletter from 2024 with an overview of interesting reads and jobs you might have missed.
In 2024 CRO weekly is back to a weekly newsletter instead of biweekly.
Have a nice week!
Tom
Fundamentals of product experimentation for beginners
In this product experimentation handbook, Convert shows you how to start, what metrics matter, and how to use a product experimentation framework effectively.
So, product managers, growth marketers, developers, founders, or anyone looking to enhance user experience and boost product adoption and retention, hang on to this
Read the full article here from Convert
CRO weekly sponsor
Thanks to our partner Convert for their support
Interesting reads
Lessons learned from running 127,000 experiments
The way you run your A/B testing and experimentation program has a huge effect on what you get — both in terms of optimizing experiences and ROI. This report is packed with data from 127k experiments — revealing insights, techniques, and examples for turning practitioners into champions..
Link to the article
Beware of the Bayesian imposter
Sometimes, statistical guarantees are not what they seem. Here, we discuss the implications of a classic work that demonstrates a paradox with the Bayesian approach to experiment analysis: when not careful, the experimenter runs the risk of running a frequentist analysis without realizing it.
Link to the article
Career interview with Shiva Manjunath
Learn from Shiva Manjunath. He is currently Senior Web Product Manager for CRO at Motive. This is one of his tips:
“Don’t think your experimentation program sucks … All programs start somewhere – keep on grinding to build the absolute best program you can!”
If you want to read more career interviews check this link
Is not AB testing everything fine?
Leading voices in experimentation suggest that you test everything. Some inconvenient truths about A/B testing suggest it’s better not to.
Link to the article
Best A/B testing talks
Most viewed talks in A/B Testing, or Online Controlled Experiments (not statistics or broader topics, over 100 thumbs up/year, over 10 minutes). These are hand-curated, so probably there are talks missing.
Link to the article
Job opportunities
Looking for a new challenge in experimentation? Find 100+ experimentation related jobs on ExperimentationJobs.com. These jobs are from all over the world, on-site, fully remote or hybrid. Take a look and start pivoting your career.
This week’s featured roles:
Principal Product Manager – Product Analytics & Experimentation at Amazon (Seattle, USA)
Senior Data Engineer – Experimentation Platform at Zalando (Berlin, Germany)
Product Manager – Growth and Conversion at Aira (Stockholm, Sweden)
Staff Engineer, Billing Growth Engineering at Stripe (USA or Canada)
Senior Experimentation Program Manager at HP (Multiple locations, APAC)
Are you hiring? Submit here
Last year I started to experiment with AI images. This one is created in Canva with this prompt:
”Lessons learned from running 127,000 experiments in digital art”
Are you not subscribed yet to this newsletter? Click on the button below.