Click to Preview
Designing leverage points
User research, UX, UI
Focus:
Timeline:
Fall 2018
After Mode transitioned to a freemium model for the first time, I needed to design a solution that would improve new user retention, increase our ROI, and encourage users to write their very first query
Overview:
In 2018 I had the honor of giving a talk at the Design for Business Impact leadership event series in San Francisco, and wanted to document the case study I presented there.
Before coming to Mode, I worked for a screen capture and video software company that made tools like Camtasia to really help users share their story through video. I quickly became captivated by this idea of taking something that many found to be really inaccessible and intimidating and make it as approachable as possible. Mode drew me in a similar fashion.
We’re an analytics platform that aims to help every analyst find success and share their stories through analysis. I currently work on the Advanced Analytics and Visual Analytics product teams, helping both our core users find success as well as thinking of ways we can really expand the tool to reach an audience that may not be as comfortable writing code. That said, today our primary tool for doing analysis is very much SQL, a query language used to help people talk to their databases. Like many, I was completely new to query writing when I started at Mode and I’m still learning plenty of new things every day.
Before I dive into the project that we worked on, I want to give a little more context around what was happening at Mode at the time. On April of 2018 we did something pretty bold. We launched a free version of our product for the first time, Mode Studio. Rarely does a company introduce a freemium model this late in the game, and especially after having such success with their paid offerings, but we saw a huge opportunity to make Mode truly ubiquitous among analysts, to increase our funnel, and frankly, to completely transform our strategy. Just to reiterate, we were going to take our product, that our users were already paying us for, and give most of it away, for free.
The launch dates were aggressive. We pushed ourselves really hard to get Mode Studio out into the wild as quickly as possible. We redesigned our marketing website. We changed the way we talked about the product. All of these little pieces had to fall into place for the launch to truly be successful. And afterwards, momentum was really high. Signups were looking great, the design team was making all of this new swag to celebrate. Things were good. And then the data started coming back…
One day retention rates had dropped by ten percent. Similarly, users were significantly less active. The majority of new accounts ran only a single query in their first seven days. Now, we knew that this seven day retention metric was critical. Not only was it one of our biggest predictors of future success in Mode Studio, it was also a huge predictor of the likelihood of them eventually becoming paid customers and sticking around for the long haul.
Internal roadblocks & what was at stake
With concern from leadership, Brian (a PM) and myself were tasked with answering these two questions:
If the tool is now free, why are people still leaving?
What can we do about it?
Unfortunately at this time, there was no clear sense of where or why the drop-off was occurring, we now had to convince these churned users to come back and talk to us, and we were still unsure how this transition to a freemium model would impact the business long term. Another thing to keep in mind is that this drop-off wasn’t something happening early in the funnel. These users had gone through all of the trouble to create accounts and connect to their private data, which was a pretty intimidating and cumbersome step in a signup flow. So they had personal, meaningful data in the tool to work with.
A lot of the product team wanted to jump to what we could do about it and begin to dive into iterating on solutions. With a desire to fix things, it was easy to start throwing out ideas. And while it was tempting to start attacking certain pain points or even suggest these solutions to churned customers, I fought to keep things really open ended to first understand the why before jumping to the what.
Despite pushback that research would take too much time, we all agreed that we needed more actionable insight. Design argued to start by sending out a quick, open-ended survey via email. After introducing myself as a member of the design team I would ask users one simple question, “What made you stop using Mode?” We spent countless iterations trying to come up with the wittiest subject line that would ensure the highest possible open rate and encourage people to respond, but the numbers were painfully low and the results inconclusive.
So we went back to the drawing board. With an improved subject line in hand and Amazon gift cards, we began to ask folks if they’d be wiling to get on the phone with us for 15 minutes in exchange for a $50 gift card. We were quickly able to book twenty slots, and received over double the response rate from interested participants as we had in the initial survey.
Now as I mentioned before, a lot of members of the product team already had ideas about what was causing the drop-off and areas we could improve. However, theres a huge difference between users bringing up not knowing how to write SQL on their own, and us guiding them there by asking if they had trouble. We were really cautious of any leading questions that would bias these results. If there’s one thing we can rally behind, it’s the importance of data integrity. This is just as important with design research as it is with a trusted dashboard.
Instead of trying to get feedback on an idea or asking about something specific, we went with a very simple set of consistent, open-ended questions and ended each call with space for the person to share any general feedback they had:
What brought you to Mode? / What were you looking for?
Did it meet those needs and expectations?
What were (are) the blockers for you moving forward?
What might Mode do to make it a better solution for you?
What did we learn
After we finished our interviews, one takeaway really stood out to us. Users didn’t know where to start when landing in the query editor for the first time. Half of the candidates mentioned not having a particular query in mind and as a result, not knowing what to do next.
On the quantitative side, the data from the analytics team showed an extremely high error rate for new SQL writers. Over ⅔ of new users were experiencing some kind of SQL error in their first 24 hours of connecting to their database.
If we hadn’t done the design research and had only looked at the quantitate metrics, we likely would’ve worked on building features that helped users write and debug real, meaningful work…when in reality, users just wanted to evaluate the tool without needing to have something to write.
The challenge here was that before doing anything else with the tool, before really evaluating any other features, you had to first have a successful query. This was our opportunity. We had heard firsthand from users that after getting their first successful results back and doing something valuable with them, like building their first chart, was really when they had that “aha moment”. This was something we could improve. But before building anything new, we wanted to start by assessing some existing areas for opportunity.
First, there was really a lack of hierarchy between portions of the editor, which made it tough to know where to focus. Second, the data that I mentioned users had gone through all the trouble to connect wasn’t particularly discoverable or even exposed by default. Generally, the page didn’t inspire a ton of action. They often say a blank page is the most intimidating step for a writer. We just needed to help them get that first sentence on paper - imperfect, small, but a starting point. So we got to work.
We were all quickly drawn to this idea of previewing data. Many products, especially desktop SQL editors, already offered some way to preview data, whether those results would open in a light box or a new view all together. If we could show users what data they had to work with, maybe they would feel more inclined to start writing queries. But was it enough? Would it inspire action? And most importantly, could we get buy in on the idea?
Getting the green light
I shared designs early and often. We were constantly iterating with our internal analytics team, and we were able to share some early mockups with a few external customers. The feedback was all the same. While seeing their data was valuable, we needed to make those results more actionable
And then it clicked. What if we could help users understand their data and write their first query at the same time?
So we made some updates to the plan. Instead of displaying the data in a light box, which took you out of the editor and wasn’t editable, we’d allow users to click a play button on any table and Mode would populate a simple query for you that would return a set of data. That query could then be used and edited after the fact. We called it Click to Preview.
During initial feedback we also learned that this wasn’t only valuable for new and inexperienced users, but also for our power users. Data is constantly changing and evolving all the time, and analysts need to understand it as it comes in so that they’re able to work with it. This led to a new debate. How could we create a workflow that was user friendly and easy to understand for new users that wasn’t patronizing or in the way for power users?
There was intense discussion between the design team and the analytics team. On the design side, we wanted to make sure this workflow felt as seamless, lightweight, and performant as possible. If we were going to be generating all of these objects for users, we didn’t want to overwhelm them by creating a lot of unnecessary cruft. The analysts, however, were really concerned with how it would feel to use within existing work, and were worried we might blow away code they had already written or interrupt their workflow.
There were a lot of perception and interaction fears on both sides, so we agreed to get it into staging to actually validate whether these assumptions were true. If you didn’t have anything written in the editor, it didn’t really matter where or how we generated the SQL code. But to make it work for all use cases, I went with the design decision to inject the code at the top and run only that portion. This prevented us from breaking or overriding any existing code, and afterwards you could simply hit backspace to remove it. This quickly alleviated most of the concerns on the analytics team, and actually opened up a lot of new contextual workflows for them.
Putting it all together, compared to the intimidating blank page on the left users could now instantly get results and start making charts with the click of a play button. In addition to this new CTA, we also improved some of the flaws we documented at the beginning to add better hierarchy and put a lot more emphasis on the users’ data.
Results
Since the launch, seven day retention rates have increased by 50% for new editors that use Click to Preview within their first 24 hours. Now, if we assume that translates into 50% more people using the product for longer, that leads to 50% more users hitting paywalls, 50% more sales calls, and 50% more sales opportunities from Mode Studio. Okay…so maybe you’re thinking “carrying 50% all the way through feels a little generous.”
But even if this increase in retention only leads to 5% more customers, the ROI is still 200-400% after only a year, and the quarterly return of that increase is 5-8x higher than the cost of developing the feature. Lastly, this didn’t only provide value to new users. Within weeks, existing customers were already using the feature up to 30x a day.
Takeaways
I’d highly encourage you to get your whole product team in on research calls as much as possible
Improvements for new and inexperienced users can often be just as valuable and impactful for power users
Qualitative research and quantitative analysis are extremely powerful when combined. It helps give you a really holistic view and avoids limiting your scope of ideas to your product teams’ go-to solutions
Small features can have a really big impact
Today this concept helps a user write their first query. In the future it could help them create their first chart. These kind of business metrics are great at informing discrete solutions, but the thing that is often overlooked is that they’re also impactful in helping us change the way that we think about our mental models and where we can go in the future.