Who will hire the Zagat editors if this works?
Do these stars tell the whole story? Should you buy a product based solely on a star rating?
The answer is No.
The star rating needs backup for sure. Jean, one of our product managers, tells her story about finding the best garlic press. So why talk about stars….well, yesterday I read on TechCrunch that the Pluribo folks have come up with their own solution to this problem. The idea is to “read” all the underlying reviews with software and regurg some interesting phrases and also give a graph of sentiment. It’s an interesting idea and certainly whoever cracks this nut will make the legions of Zagat editors a little nervous. (Zagat is my favorite summarization “engine” as everybody at Buzzillions knows too well).
We have our own solution which is to ask consumers while they are reviewing the product to tag it with Pros, Cons and Best Uses (in addition to writing general comments). See screen shot:
Then we summarize these tags above the fold for an at-a-glance, wisdom of the crowd view of the product. Here, you can quickly grok the salient points summarized from the underlying reviews which definitely coincide with my anecdotal knowledge about the iPod (think screen fragility, simple controls, good for commuting).
Let’s hope for the Zagat editors sake that the Opentable and Pluribo folks don’t hook up anytime soon.
I tried Pluribo. It’s intriguing, but it’s certainly not a complete solution. I see two main concerns.
First, its current implementation is limited to a small set of product categories, and it’s not clear to me how much work is required on their part to add a new category–or what is their granularity of product categories.
Second, while the summaries are a great starting point, they are a bit too lossy for me. They don’t give me a holistic picture of the product, but instead make me feel like I’m just seeing a random pro or con.
Compare them to, say, the sets of scores used to rate products on Circuit City, where each product type typically is rated along 4 dimensions. There, users explicitly assign scores for each dimension. Or compare them to the Buzzilions approach of summarizing reviews as weighted lists of pros and cons–again soliciting them from users rather than trying to extract them automatically from text.
Finally, I love playing with NLP algorithms as much as the next computer science PhD, and we do our share at Endeca. But information extraction algorithms (which should really be called heuristics) will never be 100% accurate. Sometimes they are the best option we have. But, in this case, why guess when you can just ask the user?