Wednesday, August 29, 2012

On the blind tasting bandwagon

Yesterday, W. Blake Gray (why the W, Blake?) wrote a blog post about blind tasting. The piece was presented as a review of his two-part interview with New York Times wine critic Eric Asimov. In the interview, Asimov stated that blind tasting is infantilizing and "dumbed down way of looking at wine." Gray strongly disagreed with Asimov on the issue of blind tasting. His post was well presented and provided a very interesting discussion on the merits of reviewing wines via blind and non-blind protocols. However, one little line sparked a bit of consternation from a few of his readers (including yours truly). Gray's statement, "Wine Spectator says they blind taste, but nobody believes them" even elicited a response from Wine Spectator's executive editor. I don't know if Blake truly believes his own statement (he said he'll post on the issue soon), but blind tasting methods seem to be a bit controversial and thus deserve a more attention.

If you happen to also read Steve Heimoff's eponymous wine blog, you may have noticed that Steve often claims that he tastes and scores California wines in a blind setting; in accordance with Wine Enthusiast's tasting policy. I've confronted Steve a few times on the issue and he has mostly resorted to ignoring my comments. Personally, I couldn't care less if a critic reviews wines with the labels visible or with no information except what's in the glass in front of him or her. There are advantages and disadvantages to both techniques. Honestly, I think a critic must taste in both settings to successfully evaluate wines.

Now, what I do care about is when stated policies are not followed, misleading or falsely attacked. I have no reason to doubt that Wine Spectator indeed rates wines in a blind setting. My only beef with Wine Spectator is that tasting flights are organized in such narrow terms that preconceptions may influence scores. If you knew that you were tasting a group of wines that cost more than $100 from a prestigious region, you might tend to give higher scores based on expectations. The same can be said about if you were tasting wines from "other" wine regions such as Colorado, New York or Virginia. Ben O'Donnell had a great article on the perceptions many consumers have about American wines not from California, Oregon or Washington. Why should critics be excluded from having the same perceptions? Knowing that you are tasting such wines might affect critical analysis. Now if a flight of wines was not qualified by price or region but by only one factor, such as American red Bordeaux blends, the results *might* be much different. I've seen it happen many times when a Colorado wine outshines more prestigious and expensive counterparts. I've also seen Colorado wines go down in flames against much better competition. Can't win them all... unless you stack the deck.

Steve Heimoff's approach is a bit different. He organizes his own tastings by placing bottles into brown bags, closing his eyes and magically forgetting which wines he is tasting. Now, I like reading Steve's blog and I admit that I sometimes taste wines in the same way. But the difference between Steve and me is that I'm not awarding wines with high scores and claim to move markets. Steve's level of influence (and he is a very influential and respected critic) is in a state of disaccord with his methods. I would think that someone that sees Antonio Galloni and James Laube as peers would have a bit more of a rigorous tasting technique.

Now, you may ask why is it ok for me to occasionally "blind" taste the same way? I don't write short reviews on thousands of wines as if they existed in a vacuum. I don't have a databse of all of my tasting notes. Most of my reviews are focused on one, or just a few wines and how I experience them. I don't focus solely on the subjective traits of a wine, but more often than not write about the story behind the wine or something more personal. I would love to taste wine blind more often, but because I am not a professional critic it is not that easy for me to do so. Tasting with the label visible is also ok, because that is how almost all consumers experience wine. In fact, I mostly just open a bottle of wine to drink with dinner with my lovely wife. This way, I am able to experience wine a normal consumer would. I just pay more attention to the qualities and how the wine changes through the evening and then write about it.

So if consumers don't drink wine in a blind setting, why and how does one drink blind? Gray proclaimed three reasons, with which I don't disagree. Really though, they all boil down to one reason: to eliminate bias. If there is a bottle of 1992 Screaming Eagle sitting in front of you, you have information that can affect your assessment of the wine. You might expect it to be a perfect wine and score it according to expectations. If you are let down by the wine you might score it lower. But of course you can't possibly score it below 90 points because it is worth thousands of dollars after all and nothing that expensive could possibly be less than 90 points. Aren't all 89-pt wines worse than feed lot runoff? If you don't know what you are drinking you are able to evaluate a wine purely on its qualities presented in the glass. You don't know if you've had a bad experience with the wine before. You don't know if it is from a region that you may think can't produce excellent wine. And finally, you won't be swayed by a wine's $500 or $5 price tag.

A blind experimental design is done by concealing information from participants that could introduce bias or somehow skew the results.  Blind wine tasting isn't exactly conducted according to scientific standards, so the terms single-blind and double-blind tasting mean different things than to academics. In a single-blind tasting the critic should be blind to all but one piece of information. You are tasting white Burgundy. You don't know if there is a Montrachet or a Pouilly-Vinzelles in the lineup. This way, you have some knowledge on what you should be looking for in the wines. If one of the wines tasted like grass and grapefruit, you'd know something was wrong. In a double-blind tasting a reviewer shouldn't know anything about what they are tasting. Often times a mix of different varietal wines and blends are lumped together. This method is more directed at assessing the critic than the wines, and is really only useful for parlor tricks or Master of Wine exams.

Does Steve Heimoff taste according to either of these designs? No. Does Wine Spectator? Yes, but with a bit more detail than just variety or region. Where Steve has intimate knowledge of what is in each bottle (he may or may not know which bottle is in which bag, but he has admitted that he can sometimes instantly tell which wines are which), Wine Spectator tasting coordinators set up the tasting so that the editors do not know which specific wines are on the table. Though by setting up flight of Oakville cabernet sauvignon that cost more than $75, the editor probably has a good guess which wines might be involved.

Now, if I were a professional critic, this is how I would set up a blind tasting. I would pick a broad, yet specific, range of wines (how's that for contradictory?). Perhaps I'd select rieslings from a recent vintage. I'd have my tasting coordinator (congratulations honey, you've got the job!) select a handful wines from a variety of regions and a range of price points. I'd ideally like to have California, Washington, Michigan, New York, Colorado, France and Germany represented. I wouldn't need Austria because we know that no Americans drink Austrian wine ;). This might require two or more flights. I'd have the wines poured in identical glasses, but I'd make sure not to see or touch any of the bottles. Who knows, maybe a heavy bottle would give away the producer or insert a bit of bias with regards to the quality of wine contained within. I'd spend a few minutes with each wine, getting to know it before moving on to the next. I'd also go back and forth amongst the wines to be able to compare and contrast the best that I could. I'd describe the characteristics I find in each wine and how those wines made me feel. After all isn't that why people drink wine; to feel something in their mouths and their minds (even if that feeling is numbness).

Despite my complaints, I truly believe that there is no perfect way to evaluate wine. As long as methods are clear and truthful, consumers can make determine which critic is best suited for them. However, people should always trust their own palate and never blindly follow even the most highly esteemed wine writer. The most important thing that Blake's post did yesterday is increase the dialogue. Wine is too often seen as an elitist commodity because people are sometimes afraid to talk about wine. Let's more talking about wine (anyone who has lived in Japan will get it....).


  1. Kyle: Why not? It's my name.

    Let's W.

  2. So your first name is W? Do you go by W or do you go by Blake. If you go by Blake, why keep the W? Does the W stand for Willard? Please tell me it does! W. Mitt Romney. W. Blake Gray. Hmmm. Thanks for reading and thanks for commenting! I'm looking forward to your WS blind tasting tome.

  3. hmmm.. I like to try wines from different places that I have not yet tasted. We just came back from a Wine Bloggers Conference where we had a "Night of Many Bottles". People brought wine from all over. I did bring 4 Colorado Wines. I did get one person say he really enjoyed them.. He probably did what I did and that was pick wines from other states other than California. Not to say California wines are not good, but let's face it... I really wanted to see what Florida had to offer. Sometimes you will be surprised by a little treasure you find. Don't go in with preconceived notions about something and you can only be surprised.
    Price has no bearing on whether the wine is good or bad.. Too bad that the only way Colorado Wines can win a competition is to be blind tasted.. Give them a try as well as other states... You will be pleasantly surprised.

  4. Both blind and categorized tastings have their place.

    To apply a permanent set of rules to tasting and scoring wines does not make any sense. If a wine costs $100.00 dollars and tastes like a $50.00 wine, then I want to know that. The resulting scores are OPINIONS of people who are supposed to be professional enough to be able to also state what styles they prefer and why.

    Being a wine writer or critic does not place that person in a category with scientific truth or biblical prophecy.


  5. Blind tasting as per a WSET Diploma or MW program is difficult and a challenge but fair. The single blind method as you described isn’t blind it’s a dog and pony show, and bringing the WA guys into it is odd, as often they are sitting in the Lafite/Mouton/ Petrus wine tasting rooms, so they know where they are and what they are tasting. I like Dr. B. Lewin MW attitude, he scores wines very fairly, none blind but prefers as you do to taste them with a meal to really get a sense of the wine as it evolves from opening over the time the meal takes, as you know this is the way to score wines ;)

  6. Lee, I never brought the WA guys into the conversation (I assume you meant WS). Obviously, blind tasting only one winery's wines at the winery is not a rigorous method for giving scores. It might make sense for ascertaining vintage characteristics, but you do know what you are tasting, just not a vintage or vineyard/blend depending on the winery. Again, I have no problem tasting wines in a non-blind setting. My issue is when one says they're tasting blind and are actually not doing so. Credibility is the issue, but both methods are useful. Please read Dave McIntyre's comment on Blake's post. I really like his distinction between writers and critics. Stories vs scores.

    1. Sorry I meant WS not WA (I think it was a spell check issue) and you are spot on as to the “blind” tasting of say 10 $100+ Napa wines not being blind ;)

  7. Kyle,

    As you point out in this articulate column, there are many ways to taste, depending on your goals, and there are differing approaches to tasting "blind."

    At Wine Spectator, we believe, as you do, that "blinding" the taster to producer and price is crucial to eliminating the most powerful sources of bias. But we also believe that typicity is an important element in wine quality, so we do reveal vintage, appellation and, where relevant, grape variety. Clearly, these decisions can be debated, but they are based on long experience. (I would note, however, that we never organize a flight of wines "over $75" or in any stated price range.)

    There is definitely a debate going on over the merits of differing tasting approaches. It would seem to me that advocates of blind tasting would applaud our methodology. For some reason, that doesn't always seem to be the case.

    As for "stories versus scores," why does it have to be either/or? We publish both wine reviews and reported stories about winemakers, regions and styles. Connecting the wine in the glass to its origins is one of the glories of wine itself.

    Thanks for tackling this important subject.

    Thomas Matthews
    Executive editor
    Wine Spectator

  8. Thomas, thank you for adding to the discussion. I absolutely agree that vintage and variety can often be valuable information for a "blinded" taster. Appellation is a bit trickier. If classified and non-classified Bordeaux are tasted together (as you've noted elsewhere), then this makes sense. But if only First Growths and maybe "Super Seconds" are tasted together (and made known to the critic), too much bias is introduced into the system. The same can be said when knowingly tasting only less heralded regions/appellations, such as with Colorado or Long Island.

    As for the "stories versus scores," it most definitely doesn't have to be either/or! In the current state of our industry, both are important and valid approaches. I happen to personally gravitate to the story end of the spectrum, but there are many people who feel that the number is the most important characteristic. I think that WS does a good job of balance both and publishing stories on many different regions. I enjoyed Ben O'Donnell's piece on Long Island wines. I hope to post my thoughts later this week. As you say, "Connecting the wine in the glass to its origins is one of the glories of wine itself." I'd add that thoughtfully discussing the qualities of wine is another. Thanks again for sharing!