Concierge Testing versus Wizard of Oz Prototyping. What to Choose?

However, there are multiple types of MVP approach you can take, and these different methods can produce different results depending on your business goal. In this article, we will be comparing two of the main types of MVP: Concierge testing and Wizard of Oz prototyping.

We will explain what each one is, the pros and cons and different uses for them. By the end of this article, you will understand how and when to use a Concierge MVP or a Wizard of Oz MVP, and ultimately be able to apply these methods to help make your own product development more successful.

What is Concierge testing?

Concierge testing for MVPs involves going directly to your potential customers to manually help them accomplish a goal or task, and is used as a means of validating whether or not there is a market for what product or service you’re offering.

With this method, you don’t actually need to build a product - Food on the Table - a tailored shopping list generator - is one good example. When Manuel Rosso founded the service, he didn’t have a website or an app. Instead he sold the service in person and generated the lists as he accompanied shoppers around the store.

Pros of Concierge

There are a number of different advantages to the Concierge MVP approach. It is generally used for generating new ideas around your potential product or solution, identifying obstacles to implementing that solution and communicating directly with your target customer. Pros include:

• You don’t need to actually build a product or website which, of course, saves you time and money in development. All you have to do is ask your customer: ‘Would you X service so you can achieve Y benefit?’. The whole thing is transparent to the customer and you will learn what works and doesn’t work without spending time and money making something first.
• You get to interact directly with customers in a natural environment where they would actually use the product. This is called ethnographic interaction and not only does it allow you to generate feedback on your product but it also allows you to see how it might work in practice.
• By interacting in real time with consumers, you will be able to generate new ideas about your potential product. For example, Manual learned about how Food on the Table could incorporate diet preferences, allergies and customer health goals by sharing the generated shopping lists with customers and gauging their reactions in person.

Cons of Concierge

While Concierge testing is beneficial for the reasons above, there are also some disadvantages to this method, such as sampling bias or false positive bias. Let’s take a look at those in more detail:

• Since a concierge service is one delivered by an individual in a personal setting, the results of any Concierge MVP test can be skewed by the likability of the concierge, and therefore not accurately reflect the usefulness of the underlying product idea. As such, you might receive false positive feedback (or even false negative feedback), so you need to be cautious about using this method on its own.
• As the Concierge method is performed manually, it can be hard to find a bias. On the one hand, you need a large enough sample for the results to be reflective of your audience, but on the other too large a sample of customers can be overwhelming for your product team. They may spend so much time collecting the sample data that they have too little time to actually analyze it effectively. The risk here is that the collected samples are not representative of the market you are trying to target.

What is Wizard of Oz prototyping?

The Wizard of Oz prototyping method takes its name from the famous film, in which (spoiler alert!) an old man pulls levers from behind a curtain, influencing the actions of certain characters who seem, on the surface, to be autonomous.

Wizard of Oz prototyping is a method in which potential customers interact with a system, product or service that they believe to be autonomous and fully formed, but which is actually controlled by an unseen human operator observing from elsewhere. Aardvark, a question and answer service which routes customer enquiries to other - expert - users, is a great example of this.

When Aardvark was first founded, they conducted Wizard of Oz prototyping, manually posting customer questions to online users to see who would answer, and then simply pasting those answers back to the initial user who asked the questions.

There was no algorithm involved in this process, but to users on the outside it appeared that the Aardvark MVP was a fully functioning system. Later, of course, automated algorithms were built into the product as expected, but at first the developers were very much like the old man in Wizard of Oz, pulling levers from behind a screen.

As we can see, this is quite a different method from Concierge testing - in fact, they are essentially the opposite. In Concierge MVPs, there is direct and obvious interaction with the customer, but it Wizard of Oz prototyping, the customer is not aware that there is manual intervention in the system.

There are a number of advantages that this gives to Wizard of Oz, but also some disadvantages.

Pros of Wizard of Oz prototyping

You might have already guessed some of the Wizard of Oz benefits, but there are a number of different advantages you can consider when choosing which MVP testing method to use.

Whether you’re looking to receive quick feedback at a low cost or test the functionality of your product without committing to extensive coding and development, this MVP method can help. Let’s take a look at the Wizard of Oz benefits in more detail:

• When you are just starting out in the prototyping phase, it can save you a lot of money to have humans mimicking a fully-functioning system, rather than spending time and resources building that system before testing
• You can get an early look at how a full system would behave, meaning it takes less time to receive feedback, make changes and learn what your customers want and need. While your algorithms and back-end systems might take months or years to build, the Wizard of Oz MVP allows you to get feedback in much less time.
• This system is less prone to bias than the Concierge system, because users are interacting with a system they believe to be fully-functioning, without human intervention. As such, they will interact with it more naturally, and the the data you generate from the tests will be more reliable.
• This method also has benefits when it comes to User Experience (UX) design - you can see in real-time where and how your potential customers have issues with the design of your product or service, and make changes to improve it.
  
  

Cons of Wizard of Oz prototyping

While Wizard of Oz offers a lot of benefits in terms of the way customers interact with and respond to your prototype, it can create some difficulties on the other side, as you need to train your ‘Wizard’ who will be behind the curtain, and it can be difficult for them to respond consistently like a machine would do if you had built an algorithm for it.

The success of a Wizard of Oz MVP relies on the human behind the screen knowing their job and doing it well; it is therefore more open to human error than other MVP types.

It might also be more difficult to identify errors that occur as a result of system performance this way - errors might be confused for something else or simply not spotted. As such, it’s vital that if you are going to use the Wizard of Oz method, the ‘Wizard’ is domain expert and well trained in what they need to do to get accurate data and feedback.

When should you use Concierge testing?

So now we’ve listed all the benefits and disadvantages of Concierge vs Wizard of Oz prototyping, let’s take a look at when you should think about using each one.

First, Concierge. The Concierge MVP testing method should be used when you need to get in touch directly with your customers and work out how to solve short-term problems in real time. The main point of this method is to maximize learning around your product and mitigate the risks involved with releasing a version to market.

So if you’re very early in your product development process and you need to better understand your customers, the Concierge method is extremely useful. You can test the hypotheses of your product idea without building it - especially useful if you don’t have the time or resources to spend on building something.

When is Wizard of Oz prototyping a better choice?

The Wizard of Oz MVP more accurately reflects what the final user experience will be when you eventually build the product. The customer doesn’t really care whether the back-end of your service is run by a human or an algorithm - they only care about the end result.

That makes Wizard of Oz a better choice when you want to capture accurate data which faithfully represents how you customers will interact with the final product. They won’t know that a human is involved and the way they use and interact with the service will be the same as how they would use a fully finished product.

To summarize the last two sections:

• Use the Concierge MVP test when you need deeper customer interaction to understand their problems
• Use the Wizard of Oz prototype when you want to assess how your customers will interact with your end product

Choosing the right MVP type

Whatever type of product you are developing, using an MVP (Minimum Viable Product) is a great way to test it and learn how to improve it without spending all the time and money needed to develop a full product first.

As we have seen, however, there are different types of MVP which can help you in different stages of the product development process. The Concierge MVP is great for getting directly in touch with your customers and the Wizard of Oz MVP is better for seeing how they naturally interact with the product.

Both of them enable you to save time and resources versus fully developing a product, so you need to choose which one matches your product goals and the stage you’re at with developing it.

Now you know the different pros and cons of each method, you should be able to apply them successfully to your own product development. However, if you need more advice, speak to a Netguru product development expert today - we’re happy to help!