In this week’s interview, we talk incrementality with Shoji Ueki, Sr. Director of Growth Marketing at SeatGeek.
Could give me a little background to SeatGeek, how long you’ve been there, and what your role is overall?
SeatGeek is a marketplace for buying and selling tickets to live events — sports, concerts, theatre, etc. We’re basically in the same space as Ticketmaster and StubHub. I’ve been at SeatGeek for a bit over a year and lead the acquisition side of the marketing team. My team is primarily focused on acquiring users to the platform across channels like SEO, SEM, paid social, affiliate partnerships, and retargeting.
How does your role fit into the overall eCommerce landscape? As in, what’s the relationship between the customer acquisition element and the rest of the marketing team?
You can broadly split the marketing team into three sub-teams. We have a brand/creative team that’s generally focused on our creative strategy and driving brand awareness. The acquisitions team is focused on acquiring new users and getting them to their first purchase. And our CRM is team is focused on driving repeat purchases to maximize long-term value from each customer.
And are you guys in touch every day? Are the teams siloed or do you talk a lot?
We definitely talk. A recent emphasis has been on having more consistent brand messaging across our various channels. We’ve also been experimenting with coordinated, cross-channel campaigns targeting specific cities.
Can you talk about a recent project that you’ve worked on?
Something that we’ve been working on recently and that is still ongoing is around refining how we measure ROI for our upper-funnel brand and reengagement campaigns.
Why is this a priority right now? And who is behind it?
Our marketing strategy has traditionally been very direct response oriented. Going back a few years, the biggest initial marketing channels at SeatGeek were search and affiliate partnerships. Given the very DR nature of these channels, it was a bit more straightforward to measure the ROI of our efforts. We’ve, of course, expanded to other channels over the years, such as paid social, podcasts and influencer marketing.
As we’ve continued to grow, we’ve increasingly realized that we need to focus more on upper funnel brand oriented channels to raise our brand awareness. Two of our biggest competitors – StubHub and Ticketmaster – have near-universal name recognition among our target market. This gives them a huge leg up across our DR channels. Brand awareness – at least positive brand awareness – also helps build trust. Purchases are typically in the hundreds of dollars and for something like the Super Bowl can be several thousand dollars. Especially given that our industry is known for potential sketchiness, we need consumers to trust us to give us a chance. We view our brand marketing efforts as a way to build awareness and trust so that we have a chance to later drive purchases. In order to really lean into brand-oriented campaigns, though, we need to increase our confidence in the ROI of these efforts and have a more refined understanding of true incrementality.
It’s similar on the reengagement side. Some portion of users would’ve made another purchase anyway, so it’s important to tease our what portion of those conversions were truly incremental. In both of these cases – brand spend and re-engagement spend – we believe there’s an opportunity to scale them significantly. And one of the things that will enable that for us is increasing our understanding of the ROI, so that we can invest in it with confidence.
How were you measuring ROI before and why has that suddenly changed?
I don’t think our approach has necessarily changed. It’s more that we’re increasingly moving beyond DR channels that can be measured in more straightforward way, towards top-of-funnel channels where a DR framework doesn’t accurately capture the impact.
With SEM, for example, if someone searches for “Knicks tickets”, clicks on our ad, and buys tickets, it’s relatively straightforward to assume that SEM played a big role in driving that purchase. On the other hand, when we’re running a brand awareness-focused digital video campaign on YouTube, we don’t necessarily expect a direct response. Focusing on just click-through and view-through conversions can significantly over- or under-state the true impact. And so, we need a more refined way of measuring the ROI.
How are you tackling this and what results have you seen so far?
I think it’s really important to take a step-by-step approach to this. Our approach has been to start with reasonable assumptions, refine those assumptions based on the data we have available, and then run a number of controlled lift tests to continually move us towards that right answer. I’ll break down each of those steps.
The first step was to make rough assumptions. For our paid re-engagement campaigns, for example, we made an assumption regarding how much credit we’d give to click-through and view-through conversions. We decided to give only a fraction of credit to view-throughs because we knew that, given the nature of whom we were targeting, some portion of those users would’ve converted anyway. So, based on those assumptions, we had an initial ROI model that we knew was somewhat wrong, but it was a starting point.
The second step was to refine those assumptions with the data we had. One type of data we analyzed was the distribution of time between the click or view of an ad and the conversion. So we knew how many people converted five minutes or five hours after view click of view. For example, if the view-to-convert distribution was completely random, this probably implies that re-engagement campaign isn’t driving the conversions (i.e., the users would’ve purchased anyway). Of course, if all the conversion happened literally a second after the click or view, it was probably some kind of fraud. The reality was somewhere in between, and we used this to infer the incrementality of the spend. We also used data from our post-purchase survey and overlapping claims across channels to further refine these assumptions.
The third step is where we’ve been spending most of our time. It’s an ongoing step. We’ve been running a number of controlled incrementality tests to further vet the assumptions we made in step 2. For example, we’re running lift tests with Facebook and our retargeting vendor to measure how much our campaigns are truly increasing conversions. And we’ve run tests with AdWords to measure how much our YouTube campaigns are increasing brand awareness, leading to more branded searches, and increasing search CTRs. We’ve also been testing coordinated, cross-channel market saturation campaigns, and we’ve used a similar incrementality framework (along with pair markets) to measure the impact of these efforts. Each of these incrementality tests allows us to further refine our assumptions and, over time, get closer and closer to the right answer.
Thanks for outlining that step-by-step. Was there anything that didn’t work as you thought it was going to or that held you up in a certain stage of the process?
I think there have been times when we’ve tried to jump too many steps ahead to the “perfect” solution, which has led to some wasted effort and solutions that weren’t really practical. The blog Occam’s Razor has a great post about attribution where he talks about approaching attribution in an evolutionary rather than revolutionary manner because you need the people, culture and processes to evolve with the technology. I think that’s very true. It’s tough because when you approach it in an evolutionary way you end up having to make assumptions at times that you know aren’t completely correct. But that’s okay as long as you’re continually evolving in the right direction.
In this week’s interview, we talk incrementality with Shoji Ueki, Sr. Director of Growth Marketing at SeatGeek.
Could give me a little background to SeatGeek, how long you’ve been there, and what your role is overall?
SeatGeek is a marketplace for buying and selling tickets to live events — sports, concerts, theatre, etc. We’re basically in the same space as Ticketmaster and StubHub. I’ve been at SeatGeek for a bit over a year and lead the acquisition side of the marketing team. My team is primarily focused on acquiring users to the platform across channels like SEO, SEM, paid social, affiliate partnerships, and retargeting.
How does your role fit into the overall eCommerce landscape? As in, what’s the relationship between the customer acquisition element and the rest of the marketing team?
You can broadly split the marketing team into three sub-teams. We have a brand/creative team that’s generally focused on our creative strategy and driving brand awareness. The acquisitions team is focused on acquiring new users and getting them to their first purchase. And our CRM is team is focused on driving repeat purchases to maximize long-term value from each customer.
And are you guys in touch every day? Are the teams siloed or do you talk a lot?
We definitely talk. A recent emphasis has been on having more consistent brand messaging across our various channels. We’ve also been experimenting with coordinated, cross-channel campaigns targeting specific cities.
Can you talk about a recent project that you’ve worked on?
Something that we’ve been working on recently and that is still ongoing is around refining how we measure ROI for our upper-funnel brand and reengagement campaigns.
Why is this a priority right now? And who is behind it?
Our marketing strategy has traditionally been very direct response oriented. Going back a few years, the biggest initial marketing channels at SeatGeek were search and affiliate partnerships. Given the very DR nature of these channels, it was a bit more straightforward to measure the ROI of our efforts. We’ve, of course, expanded to other channels over the years, such as paid social, podcasts and influencer marketing.
As we’ve continued to grow, we’ve increasingly realized that we need to focus more on upper funnel brand oriented channels to raise our brand awareness. Two of our biggest competitors – StubHub and Ticketmaster – have near-universal name recognition among our target market. This gives them a huge leg up across our DR channels. Brand awareness – at least positive brand awareness – also helps build trust. Purchases are typically in the hundreds of dollars and for something like the Super Bowl can be several thousand dollars. Especially given that our industry is known for potential sketchiness, we need consumers to trust us to give us a chance. We view our brand marketing efforts as a way to build awareness and trust so that we have a chance to later drive purchases. In order to really lean into brand-oriented campaigns, though, we need to increase our confidence in the ROI of these efforts and have a more refined understanding of true incrementality.
It’s similar on the reengagement side. Some portion of users would’ve made another purchase anyway, so it’s important to tease our what portion of those conversions were truly incremental. In both of these cases – brand spend and re-engagement spend – we believe there’s an opportunity to scale them significantly. And one of the things that will enable that for us is increasing our understanding of the ROI, so that we can invest in it with confidence.
How were you measuring ROI before and why has that suddenly changed?
I don’t think our approach has necessarily changed. It’s more that we’re increasingly moving beyond DR channels that can be measured in more straightforward way, towards top-of-funnel channels where a DR framework doesn’t accurately capture the impact.
With SEM, for example, if someone searches for “Knicks tickets”, clicks on our ad, and buys tickets, it’s relatively straightforward to assume that SEM played a big role in driving that purchase. On the other hand, when we’re running a brand awareness-focused digital video campaign on YouTube, we don’t necessarily expect a direct response. Focusing on just click-through and view-through conversions can significantly over- or under-state the true impact. And so, we need a more refined way of measuring the ROI.
How are you tackling this and what results have you seen so far?
I think it’s really important to take a step-by-step approach to this. Our approach has been to start with reasonable assumptions, refine those assumptions based on the data we have available, and then run a number of controlled lift tests to continually move us towards that right answer. I’ll break down each of those steps.
The first step was to make rough assumptions. For our paid re-engagement campaigns, for example, we made an assumption regarding how much credit we’d give to click-through and view-through conversions. We decided to give only a fraction of credit to view-throughs because we knew that, given the nature of whom we were targeting, some portion of those users would’ve converted anyway. So, based on those assumptions, we had an initial ROI model that we knew was somewhat wrong, but it was a starting point.
The second step was to refine those assumptions with the data we had. One type of data we analyzed was the distribution of time between the click or view of an ad and the conversion. So we knew how many people converted five minutes or five hours after view click of view. For example, if the view-to-convert distribution was completely random, this probably implies that re-engagement campaign isn’t driving the conversions (i.e., the users would’ve purchased anyway). Of course, if all the conversion happened literally a second after the click or view, it was probably some kind of fraud. The reality was somewhere in between, and we used this to infer the incrementality of the spend. We also used data from our post-purchase survey and overlapping claims across channels to further refine these assumptions.
The third step is where we’ve been spending most of our time. It’s an ongoing step. We’ve been running a number of controlled incrementality tests to further vet the assumptions we made in step 2. For example, we’re running lift tests with Facebook and our retargeting vendor to measure how much our campaigns are truly increasing conversions. And we’ve run tests with AdWords to measure how much our YouTube campaigns are increasing brand awareness, leading to more branded searches, and increasing search CTRs. We’ve also been testing coordinated, cross-channel market saturation campaigns, and we’ve used a similar incrementality framework (along with pair markets) to measure the impact of these efforts. Each of these incrementality tests allows us to further refine our assumptions and, over time, get closer and closer to the right answer.
Thanks for outlining that step-by-step. Was there anything that didn’t work as you thought it was going to or that held you up in a certain stage of the process?
I think there have been times when we’ve tried to jump too many steps ahead to the “perfect” solution, which has led to some wasted effort and solutions that weren’t really practical. The blog Occam’s Razor has a great post about attribution where he talks about approaching attribution in an evolutionary rather than revolutionary manner because you need the people, culture and processes to evolve with the technology. I think that’s very true. It’s tough because when you approach it in an evolutionary way you end up having to make assumptions at times that you know aren’t completely correct. But that’s okay as long as you’re continually evolving in the right direction.