Notifications
Project Submitted for Job: Product Manager emp0302 2 by testkrish 0602
×
Advanced screening Submitted for Job: Product Manager emp0302 2 by testkrish 0602
×
Job Application for Job: Product Manager emp0302 2 by testkrish 0602
×
Advanced screening Submitted for Job: Ats Product Manager US - United States by testkrish 2201
×
Job Application for Job: Ats Product Manager US - United States by testkrish 2201
×
Mark all as read

Project Payments, Tech Innovation, and A/B Testing for PMs

Welcome to our Product Newsletter, a biweekly email highlighting top discussions, and learning resources for product managers.

What We Will Cover In This Edition:-

Top Discussions: 

1) Company paying for a take-home project. That’s news to me

2) Tech companies must keep shipping new things or die. Your thoughts on this

3) PMs who once had no previous A/B Testing experience – how did you learn product experimentation fundamentals?

Top Learning Resources:

1. The first principles of product management

2. I asked 52 product managers what does it take to be great PM. Here’s their responses

3. Symptoms of a broken product culture — Part 1

__________________________

Top Discussions

Question 1) Company paying for a take-home project. That’s news to me

After applying for so many PM jobs, this is the first company I’ve seen paying applicants to complete a take-home project. More of this!

The ad read:

Our virtual interview process takes about two to three weeks, and it involves 9 hours of interviewing and paid project work.

  • Application (allow for 1-2 weeks response time)
  • Phone interview with our People & Culture Team (30 minutes)
  • Zoom Hiring Manager Interview with our Head of Digital Product Team (1 hour)
  • Zoom Peer interview with our Senior Product Managers (1 hour)
  • Zoom Homework Collaboration & Presentation with our Digital Product Growth, CX and Creative teams (1 hour collaborating, 1 hour presenting, plus 4 hours of paid project time over 1 week)
  • Zoom Executive meeting with our Senior Leadership (30 minutes)

Why you should work with us

Every minute you spend working with us, you are making a difference, the more we grow, the bigger our impact. Sell your soul no longer! We think that’s pretty great, but we don’t stop there… The job is amazingly flexible, something you are working hard to hold on to even as we grow quickly. Our benefits are designed to support and be responsible responsive to our healthy happy and driving team.

What is your say on this? Have anyone of you come across such a company? Would love to hear from you.

Thanks in advance.

– Albert Chappel

Discussion

A] They may be underestimating the time commitment, which worries me. Though I don’t have anything against take-home homework, I do think the finest employers can push you during the interview while also being considerate of your time. I spent 20 hours getting ready for my current job interviews, but that was just general preparation (reviewing market sizing, product craft, finding stories for behavioral interviews etc.).

Therefore, rather than being work necessary only for that post at that business, all the preparation would be important in subsequent interviews at different organizations.

– Yuri Roman

B] I’ve had one of these interviews. At the end of roughly 10-12 1-hour interviews, I was given a presentation to prepare on a “hypothetical” business problem. I spent roughly a week preparing this presentation during which I found out this was an actual product they were building, and this was essentially free consulting work. Very frustrating and had I known beforehand I would have declined.

– Pauline Francis

C] I never gave it much thought, but I’m having trouble understanding why having an assignment is a bad thing. I can see how that may be an issue if it takes several days, but is 4 hours too long?

Our hiring procedure includes a 4-hour assignment as well, which is a crucial stage. Additionally, we don’t view it as a way to obtain free labour, and I’d be astonished if anyone does. It seems like a terrible technique to complete work because most of the time it is useless and outsiders, especially those in PM jobs, lack the necessary context to produce something worthwhile in four hours faster than those inside. However, it’s a great approach to assess the attitude and abilities needed for the position.

If the project requires several days and is on a live project or problem, I believe it is acceptable to request payment. I’ve come across businesses that require applicants to work a free week of work as a condition of employment. But is four hours excessive? What then accomplishes the desired result without being unfair?

–  Nathan Endicott

__________________________

Question 2) Tech companies must keep shipping new things or die. Your thoughts on this

Paul Graham says, “Tech companies must keep shipping cool new things or die. Not because customers require it, but because the best employees do. If your best programmers stop being able to ship new things, they’ll leave work somewhere they can.” I would like to know your thoughts on this.

Thanks in advance.

– Carolyn Miles

Discussion

A] Paul also implies that the PMs of IT companies and non-tech companies are essentially distinct from one another. I had a job as a program/product manager in the financial sector.

Yes, it is annoying that there are many hoops to go through in order to accomplish a task, but people don’t leave because of it. In the end, a PM is accountable for millions of clients, daily transactions totaling billions of dollars, system and client integrity, etc. That provides plenty incentive to stay.

People depart because of poor management and a toxic culture. There are many IT firms that are doing a lot of wonderful things, but employees continue to leave because of poor cultures.

– Marco Silva

B] Paul is a smart man who approaches problems from an engineer’s point of view. Although shipping to ship is unquestionably the incorrect business strategy, he is not necessarily wrong. One of our apps reached a point where it required more users in order for us to verify that it was functioning properly and resolving the issue, and everyone wanted additional features in order for it to attract more users. It turned out that we declined. But this makes sense because our head mobile developer left.

– Jesus Rojas

C] Paul has a typical rich man’s thinking; he no longer comprehends what drives real people. It’s fine for many outstanding people in their late thirties and early forties to deliver respectable goods and iterate with a few fascinating ideas, but not always act crazily. Nobody desires to work more than 70 hours per week. Burnout exists.

It would be fantastic if we stopped idolizing the extravagantly wealthy and began to give more weight to the opinions that matter.

– Vlad Podpoly

__________________________

Question 3) PMs who once had no previous A/B Testing experience – how did you learn product experimentation fundamentals?

Having been a B2B PM all by career, I am an A/B testing dunce. I know a few concepts, but I’d really like to learn this properly (and quickly, in preparation for a new role)

  1. What are some helpful resources that explain foundational product experimentation concepts well?
  2. Are there tools or websites that let you run your own tests to facilitate learning?

Much Thanks.

– Risa Butler

Discussion

A] Think of it like a science experiment. You have the control and the test group(s). Each of the test groups will be compared to the control and that helps say whether that individual experience is better than control and should be pushed to production / to all users.

Keep in mind there are lots of metrics you look at to determine if something should GA. It shouldn’t not only make sense holistically , but also should align with the product / company strategy.

FYI the bucket sizes should all be the same size. You will want a size that will give you a statistically significant results within a few weeks. If you have large audience, you need a smaller size. You will need either software or data scientist or you need to know a bit of statistics to say if the results are significant (i.e. not a change in metrics by chance)

– Nathan Endicott

B] Start at Aristotle, Plato and Karl Popper. This is when ideas started to really be documented on the scientific method.

Then work your way up to randomized control trials from the medical community.

That way, you have a proper base and you will know how the “method” really works and why; falsification, testing and why you do randomization, what is statistical significance and how you can arrive to N for your hypothesis and product.

Keywords for YouTube videos: experimental design courses, research methods, hypothesis testing, falsification, randomized control trials, p-hacking (good to know this one), segmentation, statistical significance.

– Samantha Yuan

C] I’ve been doing A/B testing for pretty much my whole career. Have done both in-house and at experimentation focused agencies. My focus has been mainly on the web but the foundation is still the same.

Ron Kohavi who has done testing at Airbnb, Microsoft, and Amazon has a great book called Trustworthy Online Controlled Experiments

If you are a visual/auditory learner like myself, this video 1 is extremely helpful for you to see the big picture as well as some specific examples of being a Growth PM, which is a term that some companies use these days.

Also as someone had mentioned, Georgi Georgiev is great and knows his stuff. He devised an agile calculator (cheap but not free) for testing that helps to make faster business decisions compared to other testing calculators out there. Not sure how much traffic your product gets but low traffic is always a hurdle for testing.

As others have also said, CXL is useful for quick learning and inspiration.

My tip is, don’t get caught up in the process too much. I made the mistake of spending too much time doing preparation for testing and not enough on actually getting tests out earlier in my career.

It’s important to have the knowledge but actually being effective in running tests is a whole another skillset. Testing velocity is arguably the most meaningful measure of an effective experimentation program. Your manager, his/her boss, and your peers will want to hear about the tests you ran, what you learned from it and what was gained from it.

Of course many will want to see positive results on the KPIs but I’ve always emphasized the learning aspect more because the learnings can scale across the org. If you focus only on the key metrics being measured, you are also risking your performance to be reliant on your test wins, which will no doubt have its ups and downs.

– Marco Silva

__________________________

Top Learning Resources

The first principles of product management

First principle thinking helps PMs because as companies scale, communicating the rationale behind historical, current, and future decisions can be simplified in a way that their team and stakeholders can rally around. This enables people around the PM to move quickly in the same direction, decouple, and make smart trade offs without their presence.

I asked 52 product managers what does it take to be great PM. Here’s their responses

Prioritization is priceless — with the overwhelming number of issues a PM faces daily, prioritization is key. My favorite quote on this is by Yee Jie from Rakuten Viki, “A good product manager puts out fires. A great product manager lets fires burn and prioritize from there”

Symptoms of a broken product culture — Part 1

In every company, there are invariably more problems than people have the time to deal with. At best, this results in circumstances where minor issues are overlooked. At worst, ongoing firefighting depletes the resources available to an operation. Managers and engineers jump from one assignment to the next without finishing the previous one. Trying to solve problems becomes worthless patchwork. The team’s productivity plummets. Managing turns into a perpetual game of juggling which fire to put off for now while deciding where to put overworked people.

__________________________

If you enjoyed this newsletter, please consider sharing it with a friend by asking them to sign up here.

Who’s Prowess? We are optimist product managers, engineers, and educators working on creating a world where merit meets opportunity. On Prowess, aspiring and experienced product managers showcase skills, learn from the community, and connect with employers to advance their careers.

How can you grow with Prowess?

    1. Learn from curated learning resources and community
    2. Work on curated projects or join expert-guided group projects
    3. Receive personalized feedback from product leaders
    4. Build a portfolio to stand out as a product manager
    5. Access and apply to curated jobs
    6. Prepare for interviews with Q&A from top companies

Latest Posts