It’s been exactly a year since I started writing The Next Iteration. In case you’re curious, a few quick stats:

• 23 articles published
• 16 subscribers for the first email
• Nearly 400 subscribers for this email (wow! Thank you!)
• My most popular articles (by web traffic) have been:
  • ​Set new hire product managers on the path to success​
  • ​How to build a customer-obsessed culture: product operations in action​
  • ​From chaos to focus: product operations team models​
• I launched (and am almost done with) the first cohort of the Product Ops Impact Accelerator​
• I’ve had meetings with more than 50 product ops professionals this year (👋 if you're one of them)

Today's article is inspired by a great conversation that Graham Reed and Holly Holbrook recently led. We started talking about how to understand product teams in your first few weeks on the job, and there were a ton of questions around how to use surveys.

Thank you for joining me on this journey. Looking forward to what this newsletter looks like in the next iteration and see you in the new year!

Jenny

The Best Survey Strategies To Measure Product Ops Success

As a consultant, I have to be able to quickly jump in to understand how a product team is functioning. Just like any product manager, I do my user research before I start making feature changes. I combine in-depth interviews, surveys, and document reviews to get a full picture. One technique I use that’s been the subject of a few product ops discussions recently has been the survey.

Surveys have some advantages:

• Easy to run (though hard to design)
• Can provide data over time to show progress
• Allow you to hear from a large group of people

I never run these surveys in a vacuum; just like with product research I use many methods to get as complete a picture as possible. The survey data gives me one set of data points that I always supplement with interview data.

At the end of an engagement, I run the survey again to show how I've been able to support a product team's growth. It's great to be able to demonstrate that I've had a positive impact on their goals.

Why it's so hard to measure product operations

Within product ops, there are currently no agreed-upon methods to measure our impact. That’s because the outcome of product ops is a better product culture and culture is notoriously hard to measure. We can measure behavior. Behavior drives culture, but it provides an incomplete picture.

The HR world has been struggling to measure culture for many years. They've established a few limited ways to get a sense of company culture – surveys, interviews, behavior tracking, and following some metrics that can be gleaned from management systems (retention and referrals, for example). We have access to these same tools in product ops, and in this article, I’m going to dive into one method in particular – the survey.

A survey is going to be an imperfect measure. They aren’t great for giving the context behind answers, and subtleties get missed on multiple-choice questions. Despite its imperfections, it’s the best way to get a quantitative perspective on our qualitative work. We must take a product mindset to our work in product ops – which includes measuring our outcomes, just as we expect product managers to do. If we decline to try and measure our work at all, we run the risk of being categorized as ineffective or unnecessary.

In the rest of the article, we'll cover best practices for running product ops surveys.

Survey design should focus on the target culture

It's quite easy to end up with a survey that asks about every element of your product culture. While it is tempting to baseline everything, proceed carefully. A long survey will have lower response rates and many of the results will likely sit around unaddressed. That could create frustration for some people who feel they are not being listened to.

Focus your survey questions on the type of company culture you're trying to build. Work with your leadership to define the key characteristics that your team values and measure for those. Don't waste your time measuring things that you aren't trying to change.

For instance, let's say that your focus is on improving the product team's use of product analytics. Focus your questions on how frequently they're referencing the data, how easy it is to access, and what they're doing to communicate findings. Since use of data will probably not correlate to release notes and documentation, don't ask about those. On the other hand, use of data relates to how a product team prioritizes their roadmap, so you may want to ask about roadmapping.

Follow best practices around what good survey design looks like. Work with experts in survey design at your company – you likely have someone in UX who is trained in this. Test your survey questions by running the survey as an interview with 2-3 people to make sure questions are being interpreted appropriately. Ask someone else to edit your survey before it goes live.

How and when to run a product operations survey

Product operations doesn't impact product management alone, but touches many other departments. Therefore, run the survey with all relevant colleagues – product development and go-to-market teams. Use question logic to show the applicable questions to the correct audience to keep the survey as short as possible for everyone.

To get strong engagement with the survey, think about how you will advertise it. Use techniques that will get you the highest engagement levels. I always ask my client, who is usually the head of product, to send out an email in advance of my sending the survey. The email lets everyone know to expect an email from me, a stranger. The announcement from leadership also encourages people to participate. In companies that have a strong Slack culture, I schedule a few posts directing folks to the survey so they get notified of it via multiple channels. I also mention in meetings that the survey is running and encourage everyone to do it. For best results, leave 5 minutes at the end of a meeting for everyone to take it then and there.

In terms of frequency, send the survey out on a regular cadence, changing as little as possible each time. This way you get valuable data to see where you're improving. Send the survey to the same teams each time, making sure to add new hires and remove people who have changed roles. You want to keep elements constant.

To decide the optimal cadence, run the survey when you think you'll see new information emerge. If you are rolling out a new roadmapping process, but only do roadmapping once a quarter, I would expect there to be useful feedback from the survey every 6 months or so – once the team has had a chance to try the new process out twice. If you are doing monthly roadmapping, you could start to see the effects of your work every quarter and so that would be a good cadence.

I don't generally recommend doing a survey more frequently than quarterly. The feedback is less likely to have changed and you run the risk of creating survey fatigue by asking the same questions too frequently.

Questions to ask

Choose the questions that work best for what you're trying to achieve and learn. I start each survey by collecting demographic data – where the employee works, their department, their tenure with the company, and whether they're an individual contributor or a manager. I then dive into general sentiment about the product development process.

After that, the survey splits depending on whether they're part of the product development team or not. The product development team gets asked about whether they feel their ideas are valued, if they feel like they contribute to the team, and questions about communication patterns.

The go-to-market teams get asked about what type of communication they receive, if they know what product is doing, and if they feel like they're being listened to. You can find inspiration for the types of questions to ask in this article, where I go into detail about five of my favorite questions.

What gets asked beyond that depends on what is getting measured. It's useful to use the pillars of product operations to make sure you're covering all that you need. At the end of each section, I leave an open-ended option in case someone feels inspired to share more information.

Interpreting and using the survey data

In general, you want to look for places where there's strong agreement or disagreement. That indicates topics where your survey is more likely to have actionable insights.

After reviewing everything at a high level, I always slice the data by role. For instance, with one client I found out that everyone thought the product development process was sub-par except for engineering, which loved it. Other times I'll find that GTM teams feel left out of the loop, but product development feels like things are headed in the right direction.

Other dimensions to use to interpret the results include by tenure at the company and by whether they're a manager or IC. Finally, if the company is split across multiple locations I check if there are differences between those near HQ and elsewhere. For instance, with one client they had the majority of their engineering in India. The survey results indicated communication issues due to the time zones.

As soon as analysis is complete, create a quick report on it and record a video walkthrough. Send the video to everyone who was asked to take the survey. By sending out the results of the survey quickly, you promote a culture of transparency and indicate that the survey data is being used. This helps get a strong response rate for the follow-up surveys since everyone knows it won't be a total black hole. If you make changes based on what you learned in a survey, be sure to communicate that out.

Don't overcomplicate it

In terms of tools, whatever your company already uses for surveys should suffice. The most important feature is the ability to add skip logic so you're asking the right people the right questions. For analysis, I usually import the data into a spreadsheet and pivot table so I can dissect the data in various ways.

To summarize:

• Focus your questions on the culture you’re trying to build
• Include colleagues from across the company
• Keep it as short as reasonable
• Run it on a regular cadence
• Market your survey strategically
• Use skip logic to keep it relevant
• Analyze it across multiple dimensions

If you want extra help in survey development or execution advice, drop me a line about coaching opportunities. If you'd like to run a larger baseline assessment and are looking for a neutral outside perspective, I can conduct a product operations assessment for you.

The sooner you start taking your baseline, the sooner you'll be able to measure your progress. Keep it simple, use best practices, and get your survey out the door. You'll be excited to see the results roll in.

Thank you for the quality feedback from Adam Kecskes, Michelle Harris, and Miriam Karlin.