Our Review Process

Reviewing products is hard. Especially when you’re expected to declare one product as the best in its category.

There are far too many variables to factor in. Some users will prioritize price over everything else. Some others will pick the better looking product over a more competent one. And some will focus exclusively on features, price and looks be darned.

Then there’s also the fact that preferences vary by category and product. Reviewers might wax on and on about a headphone’s “balanced sound” and “rich warmth”, but customers might simply care about how good the headphones look and what brand they belong to.

Which is why we have a slightly different approach to reviews.

The Roger Ebert Approach to Reviews

The late movie critic Roger Ebert was arguably the most well-known critic in America. He was certainly my favorite reviewers, bringing a sense of balance to a job that was usually filled with extreme opinions.

One of the things that made Ebert’s reviews approachable for most people was his ability to evaluate movies within their own category. While other critics would only laud cinematic masterpieces, Ebert understood that movies have to be compared to their in-genre peers, not across genres. This is why he was okay scoring You Don’t Mess With the Zohan – hardly a high-brow comedy – a solid 3/4 stars. Because he compared it to other in-genre comedies, not Citizen Kane.

In reviewing products on MIDINation, I’ve attempted to follow the same advice. It makes no sense for me to tell a prospective buyer about the best headphones that cost $200 when the buyer can only afford $50 and his frame of reference is the latest Beats. A starting producer looking to make his first beats isn’t going to find much utility from a $500 Akai Advance Pro.

Thus, every recommended product you see on this website is reviewed in context of a) its peers, and b) its category requirements.

If you want to understand why I recommended a 3-star product over a 4.5-star one, just take a closer look at the article title (i.e. its category) and the product’s peers within its price/feature range.

 

The Shortlisting Approach

Outside of a few categories (such as headphones), the music gear market is fairly small. If you’re in the market for a new MIDI keyboard, you typically have about a dozen brands to choose from. Same with something like digital pianos. Sure, there might be dozens of brands on Amazon (cheap Alibaba knockoffs), but most people will prefer to buy Yamaha, Casio and Roland.

This makes the shortlisting criteria for most product categories o this site fairly straightforward. Here’s my step-by-step approach for creating my initial shortlist:

  1. Shortlist all products I have used in the past
  2. Ask for recommendations from musician friends
  3. Browse the catalogs of known brands for products
  4. Survey the market for new products and brands

It almost never comes down to #4 in categories like DJ controllers/MIDI keyboards/digital pianos that are filled with entrenched players. In categories like headphones, however, new entrants keep popping up every few weeks so a market survey is important.

This initial shortlist gives me a good number of products for the first review draft. In this draft, I weed out products that don’t meet my selection criteria. This criteria is based on the following factors:

  • The product must have at least average reviews. I can’t consider a product that is obviously faulty
  • The product must be purchasable commercially from established retailers. There are plenty of small makers creating boutique gear such as amps, but these are hard to buy for most consumers and are, hence, not considered.
  • The product must have a reputable brand with established support channels behind it. I can’t review or recommend a product from a fly-by-night operator that might disappear within a year. You don’t buy a MIDI keyboard or audio interface every few months, after all.

This selection criteria weeds out a large number of knock-off brands and poor quality offerings.

For the rest, I use the review approach outlined below.

 

Talking Numbers

At the end of the day, a review is nothing but an attempt to quantify a subjective experience.

While every product is different, I use the following general weighted template for quantifying review scores:

Let’s break each of these down into more detail:

  • Category affinity: This is the “in-context” reviewing approach – aka the Roger Ebert approach – that I talked about above. Simply put, it means how well a particular product performs when compared to other products in the same category.
  • Price: Affordability and value for money are, after all, top factors in any product selection.
  • Features & Performance: How well the product performs is the biggest factor in recommending a product.
  • Build quality: Music gear usually takes a beating. This criteria prioritizes equipment that is well built and will last a while.
  • Design: Design not only includes aesthetics, but also how well the instrument is laid out. Usability is as much a part of ‘design’ as appearance.
  • Integrations & Support: Finally, our review covers how well the gear works with popular software stacks and should you run into any problems, how easy it is to get support.

The weight for each criteria will vary from review to review and article to article. If the article is about the “best budget keyboard“, value for money (i.e., price) will have the biggest weight. Whereas if the article is about the “best keyboard for pros“, the weight will be skewed in favor of features and performance.

The broad approach, however, will be the same and follow the guidelines above.

Keep this in mind as you read our reviews.

 

For any questions or doubts about our review process, feel free to contact me here.