Blog

We’ve fielded a lot of great user questions since launch, and the number one area we’ve fielded them in is sentiment. This may be too much information for some of you, but if you really want the details, read on!

The sentiment feature in the Scout Labs application is the ability for the machine to judge whether or not the author of a story is expressing a positive or negative attitude towards a specific word or phrase. For those companies with only a few posts per day that they can judge for themselves, this feature is a nice to have. But for brand and product marketers looking at a significant volume of posts, this feature is essential to understanding changes in consumer opinion.

So how do we do it? How accurate is it? And how should you use it?

How we do it

Scout Labs’ sentiment is “entity specific”. What some products do when they produce “machine generated sentiment” is that they count happy words vs. sad words in a news article. The “tone” of the article is shown by the happy word count. Consider “I love baseball. My happiest memories in life are from sitting in the bleachers at Fenway. It’s the greatest game on earth. But guys like Bonds and A-Rod are bringing it down.” Despite the high “happy” word count, this does not express a positive opinion about Barry Bonds or Alex Rodriguez.

In the Scout Labs application, we don’t count happy words. We evaluate sentiment for each particular word or phrase you search for. We can tell that the sentiment for baseball is positive but negative for Bonds and Rodriguez. This is done via part of speech tagging: parsing the underlying semantic structure of a sentence and determining which emotion words apply to the key word. Emotion words come from dictionaries of standard English words and have been augmented with phrases and slang to better map to the world of social media. So Scout Labs’ sentiment is entity-specific, which is very important.

Scout Labs’ sentiment can be changed by users. We use confidence intervals to decide whether something is positive or negative, but if we get it wrong (more on that below), you can change the score, immediately updating that item for yourself and the rest of your team. Charts and graphs update immediately as well. And the really cool part is that every time a user changes a sentiment value, that item becomes a labeled piece of data that we can use to abstract out additional rules and add words and phrases for our dictionary. So our ability to detect sentiment just gets better over time.

Scout Labs can “backfill” sentiment data for the previous 3 months in less than a day. We have 3 months of live data in our app for our users right now (6 months soon). We can go backward and score all the posts from the last 3 months in less than 24 hours. So you will have complete sentiment trend for everything going forward and going backward within less than a day from creating a search (All other graphs — buzz, share of voice, etc.) are real time and have no lag time at all).

Does it work?

Yes. We have done extensive human vs. machine testing and it’s accurate in the 70-80% range, meaning our algorithm agrees with humans’ scores 70-80% of the time. This is only slightly less than humans agree with each other. Some other insights and findings from our testing:

• College educated people with business experience agree on the sentiment ratings for a blog post about 85% of the time. Using less qualified people, such as you might find in a random Mechanical Turk experiment, produces lower rates of agreement. We were surprised that we couldn’t get that rate higher. Some of the discrepancy stems from the human tendency to equate negative opinions and negative information: “I hate Coke” is a negative opinion; “Merrill Lynch just downgraded Coca-Cola” is negative information.
• The Scout Labs sentiment feature agrees with college educated people about 75% of the time. We try to pad that a little by being conservative about what we call positive or negative — we call things neutral if they’re borderline.
• The Scout Labs sentiment feature sucks at detecting irony and sarcasm. Posts that are heavy on the irony often end up classed as “neutral” because the machine can’t even guess. Consider “Another winner from the almighty Microsoft.” That’s a tough one.
• Machines don’t understand business context. Perhaps you work for Apple and every mention of an unlocked iPhone is negative because people shouldn’t unlock their iPhones. An algorithm that uses grammar and vocabulary based rules cannot classify this post as negative about iPhone: “I love my iphone. My boyfriend unlocked it for me last night.”

So the Sentiment feature produces a pretty good guess, about what you’d get using if you got a half dozen ratings from Mechanical Turk and chose the rating the most humans agreed on. (See this useful paper from the Dolores Labs blog about how to use Mechanical Turk to get reliable human judgments). And our best guess plus your teams’ efforts to quickly change the things we miss or get wrong means really high accuracy levels for you and your team with a minimum amount of work and expense.

How you should use the sentiment feature

• To find the top positive and negative posts. Click on “Sentiment” and filter for positive or negative posts. You’ll get immediate insight into some forceful opinions about what is wrong — or what is right — about the product or brand you are searching for.
• As a starting point for your own sentiment analysis. Any user can change the sentiment rating for any post. If you work for Apple and you want all those unlocked iPhone posts marked “negative,” you can do that. Just click on the sentiment icon and make the change. These changes will carry through to all graph data, so you can create accurate data sets to view in the application or export data for. We use your rating changes as machine learning inputs, but your specific ratings are proprietary and confidential to your workspace.
• To get insight into consumer opinion via alerts. When you set up a daily, weekly or monthly alert for a search, you’ll get buzz, top news, new words, recent tweets, and the top positive and negative posts pushed to your inbox via a text email. It’s a great way to stay informed and know when to invest more attention.
• To compare sentiment between brands or products. Do consumers like Symantec or Norton? The Lebron 6 or the KD1? Embarq or Comcast? Sentiment Trend graphs can help you see trends, spikes, and make comparisons.

We have heard over and over again from our users that an affordable, reliable way to assess sentiment, with user override built in, is critical to getting insight into social media, so we continue to work on this feature. We hope you’ll let us know how you want it to evolve in the future. We’ve already got a slew of new feature requests to work on, including more metrics, visualizations, and customizations. Get your ideas into the mix at support <at> scoutlabs <dot> com.

My rocket-obsessed 4-year-old was SO excited that mommy was “launching” Scout Labs last week. He kept asking me if I was going into space. Well, things were ‘getting off the ground’ I guess.

But I couldn’t help but notice on launch day how much it actually felt like Mission Control in the Scout Labs office. I thought I share with you some of the ways we monitored and directed launch day.

Of course, we were logged into our Scout Labs workspace, watching and responding to blogs, tweets, news and comments in real-time. We kept refreshing the page to see the newest posts of the minute, which was easy enough, as we were logged in all day long, but we definitely felt the need to have our aggregated search results available via RSS. (To those of you who have requested it, that is officially prioritized on our roadmap now). From the Scout Labs dashboard, we bookmarked Tweets and posts with tags like influencer, event, launch and feature_idea. We emailed and assigned things to each other, but there was also plenty of “Margaret, will you take that one?” and “Jenny, you on this one?” yelled across the room as well.

Another thing that we are all tuned in to (because it is projected on our wall to a size of about 15-feet square) is a streaming list of real-time searches by users inside our app. Every time a user hits “Save” on a search, we see the search terms requested, how many fractions of a second (or seconds, in a few cases) each search took us to return the data. You can imagine how fun this is to watch - people’s interests flickering on our wall in real-time, sometimes hundreds per minute.

We’ve got application metrics as well, with by account and totals for number of workspaces, users, invites sent, bookmarks created, sentiment values overridden, discussions started, etc. We’ve got back-end monitors for how long sentiment takes to process, how long it takes for features to populate and everything else you can imagine, all wired up to pagers so that we are immediately notified if something goes down or stops working. In fact, the only thing that was uneventful on launch day was our systems! No pagers. No lags. No problems. (OK, I lied. Apologies to Sean Power who was transferred to the SF fire department when Julie tried to transfer him to me! Thanks for calling back, Sean.)

It’s not quite as cool as sending a rocket to space, but a launch of any kind is an amazing thing to be a part of. Especially when you’ve got the right tools in place and a rock star team on the job.

I’m happy to announce that Scout Labs is open for business today at www.scoutlabs.com.

Our deepest thanks go to our several hundred very savvy beta companies who have been both brutally honest and wonderfully encouraging. We stayed in beta longer than most because we believed that if our product could earn their business, we’d be poised for success. We are proud that many are now clients, so it’s time to launch.

Scout Labs is a web-based app that finds signals in the noise of social media so that companies can build better products and stronger relationships with customers. Everyone seems to be talking about social media analysis these days I’d like to just cut to the chase and tell you what makes us different:

1. Scout Labs is true software to support your own listening efforts. Analyses are automated and real time. We employ no analysts to generate PowerPoint reports. We want to teach you and your extended team to listen — not listen for you.
2. We know that brands are innovated and protected in “packs”, so Scout Labs is for teams. All plans are for unlimited users. And we offer many ways to interact with the data, to add insights, to collaborate with each other, and to engage with customers.
3. We are committed to delivering qualitative insight as well as quantitative metrics. You can use our app to track when buzz goes up and down. But we also want you to understand what the buzz is about. What are customers ranting about? What are they raving about? We use NLP (natural language processing) and other text mining techniques to deliver insight into what customers are saying. Today that means features like automated sentiment detection, conversation digests and the identification of emerging conversations. Tomorrow, that means so much more. But it is our goal and the direction we will continue to push.
4. Thanks to our strong focus on automation and collaboration (and infrastructure that is ready to scale), we offer this advanced suite of analysis tools, including real-time sentiment detection, at a very low price point. All companies need to be listening to and analyzing social media. Now they can afford to.

We think our product speaks for itself, which is why we are offering a 30-day free trial with all plans. Give us a try. Reach out if you need help. Tell us what you think and what you want to see next.

Thank you (and woohoo!) from the entire Scout Labs Team.

Photo source: Mathieu Thouvenin

The other day an analyst asked me about the different ways companies use Scout Labs — why they start using it and all the ways they end up using it. It got me thinking about the evolution I see when companies decide to use a service to help them tune in to customers across the Internet. In fact, it reminds me of Maslow’s Hierarchy of Needs. In 1943, Abraham Maslow developed his Hierarchy of Needs, a theory to explain human behavior. Maslow suggested that psychological needs are hierarchical, and you can’t move up the pyramid until the underlying psychological needs are met. Many have since challenged his strict hierarchy, pointing out several exceptions to the rule. But as a pop-culture metaphor, it helps describe the various ways companies use Scout Labs.

Here is Scout Labs’ Social Media version of the Hierarchy of Needs:

1. Find and fight fires (or CYA)

Many organizations get interested in consumer-generated media (CGM for short) because they want to find fires—exploding laptops, rants from prominent bloggers, a rumor leak, a copyright violation—and fight them as quickly as possible. This is an absolutely essential way to apply a service like Scout Labs, and the reason why whatever service you select needs to:

• Be real-time (so you can find those fires as they happen).
• Find the fires for you, prioritizing what’s important / worth paying attention to right now.
• Have alert capabilities built in (email, SMS, RSS).
• Help you quickly mobilize your response and take action.

In this time of highly vocal customers who are connected to each other real-time and whose seething blog posts about you can show up within the first few results on Google, companies worry about the sheer volume of CGM and their lack of visibility into it. There are huge dollars at stake. Finding and fighting fires are fundamental reasons to scout the Internet, and your organization must become good at it. This means you need to have confidence not only in the tool you use, but in your team’s judgment and ability to take action in time of threat.

2. Build relationships with customers

Only after a company is a good listener can it jump in and start building relationships with customers. If you’re new to the social media scene, watch for a while. Learn who the key influencers are. Get a feel for the language, the concerns, and the issues. See which companies enjoy a good rapport online and which get flamed. And when you’re ready… jump in. Be part of the conversation. Answer questions, ask questions, and inform the best you can.

I like to think of it like a schoolyard game of jump rope. The customer conversation has its rhythm, pattern and players, and you don’t want to barge right in before getting the lay of the land—you’ll just get all tangled up. (Scout Labs lore: At one point, many moons ago, we toyed with the idea of naming the company Double Dutch!)

Of course, building relationships with customers being key to business success is nothing new. That idea is as old as commerce itself. But what is new is the accessibility of customers (thank you web 2.0!) and the availability of solutions like Scout Labs to actually build relationships on a mass scale. It’s not easy to evolve to this place. You have to really understand your customers and their communities before you can be welcomed in. AND you have to trust your employees to have these conversations and build these relationships. But if you can get there, the rewards are great.

3. Seek out feedback on products and marketing

Once an organization gets good at finding and fighting fires and starts to engage in dialogues with their customers, it’s going to start hearing what customers are actually saying ;-) I joke, but in the early phases of a company’s social media evolution, a person talking about your brand is often a “mention”—an event to log and tally. But pretty soon you start reading the content of those mentions and you see a rich tapestry of feedback—rants about customer service issues, wishes for features the product is lacking, competing products they are considering switching to, people with problems (that lo and behold, your product can solve).

Rather than monitoring for the huge PR nightmares, companies ask lots of its employees to listen every day, seeking little insight they can use to improve products and marketing. Product managers, research groups and marketing managers love using Scout Labs in this way.

4. Be a customer-centric organization

The apex for an individual, according to Maslow, is self-actualization—making the most of your abilities and striving to be the best you can be. For an organization that wants to listen to customers, the pinnacle is being a truly customer-centric organization. For a company at this stage of evolution, customers are partners. Listening to customers and engaging with them to build better products and sell more is a strategic priority and part of a company’s culture. Everyone—from the CEO to customer service, from product to PR—is tuned in to what customers are talking about, coming up with new, customer-inspired ideas, jumping into conversations to build relationships, and truly innovating.

Being a customer-centric organization is more than a nice-sounding aspiration. We believe it’s a strategic competitive advantage. Whoever listens better, innovates faster and builds stronger relationships, wins.

Note: this is an update to a post I did a year ago.

You all know Heelys — the roller skate / shoe hybrids that are beloved by “kids” of all ages. Well, Heelys and their savvy agency partner, 358 Communications, are using Scout Labs to get a real-time beat on who’s wearing Heelys and what product or communications issues might exist they can solve. For instance, they noticed a number of people asking whether the shoes come in adult sizes (which they do!), so Heelys is now looking at campaigns to get that message out. They have also used Scout Labs to find the all-important celebrity sightings. Right now Heelys is seeking endorsements and partnerships with a number of celebrity Heelys fans.

“We have a small marketing team here at Jott, which makes it all the more important to have efficiency tools like Scout Labs to manage web communications. Making sure we are on the pulse of what our customers are saying helps us be more responsive in everything we do, from product development to pricing.”

Jott Networks operates a voice to text service that helps users capture notes, set reminders and calendar appointments, and interact with web sites and services. Since fall of 2008, the company had been offering both free and paid service levels to users, with about 30% of users converting from the free to the paid service.

In January 2009, Jott announced an end to its free service and the introduction of a new voicemail to text service. Jott has an active community of users who blog and tweet about the convenience and ease of the service and the company was not expecting all of them to welcome the news. As an internet startup, the marketing team is small and Doug Aley, the VP of Marketing, was handling public perception of the announcement and launch pretty much solo:

“We have a small marketing team here at Jott, which makes it all the more important to have efficiency tools like Scout Labs to manage web communications. Making sure we are on the pulse of what our customers are saying helps us be more responsive in everything we do, from product development to pricing.” — Doug Aley - VP Marketing and Business Development, Jott Networks, Inc.

Scout Labs helped Jott to find all the mentions of Jott that came up during the critical time period, prioritize items for response, and track which posts had been responded to. The Scout Labs application is still helping Jott do the work of converting users of a free service to paying customers by helping them find instances of social media they need to respond to. Jott also continues to use Scout Labs to monitor key competitors and provide insight into the personal productivity and voice to text space.

“Scout Labs has a really fresh and engaging approach to exploring online conversations. Our team is able to use the product in collaboration with our clients, and we are achieving real insights into markets — not just collections of data.”

“Scout Labs provides an intuitive and elegant interface for managing a wealth of conversations across the web. It makes social media monitoring dead-simple.”

“There’s one reason why we use Scout Labs: it helps us turn conversations in to action. Buzz monitoring is one thing, but the ability to collaborate with our clients in real-time to reach far across the social web to engage our customer communities is the ultimate capability. Our clients are genuinely excited by the ability to stimulate word of mouth among their influencers and track our programs’ effects over time using a smart system like this one.”

OpenDNS, a leading provider of free recursive Domain Name Service (DNS) with Web content filtering and phishing protection, has some very savvy social media folks on board. They are very active in the blogosphere and have an avid (and vocal) fan base.

They have been using Scout Labs for the last few months and have really been enjoying the comprehensive email alerts that we push to them daily.

“This is fantastic!” — Allison Rhodes, Director of Marketing at OpenDNS

OpenDNS is getting creative with its searches, as well. In addition to tracking OpenDNS and its DNS-guru and founder David Ulevitch, the team is looking for people out across the Internet who are having problems with web content filtering or those looking for a better solution. Helpful tips and information from proactive OpenDNS team members are delighting potential customers.

It’s been been less than 24 hours since the most exciting Superbowl game in recent history has ended, and buzz is off the charts for the Steelers — and the Cardinals — and Santonio Holmes — and Larry Fitzerald. Looking at the graphs, you can barely tell who even won:

This kind of buzz is great if you’re a smaller brand and you bought that spot specifically to drive awareness — look at Cash4Gold:

But while graphs and charts can do a good job of event detection — something happened! — they don’t give brand or product managers insight into what, exactly, just happened, or inform what they should do about it. There’s no doubt that major marketing events like superbowl ads have some, perhaps enduring, perhaps fleeting, influence on brand perception. Witness responses to the Doritos ad from Twitter:

@dmgerbino Are you not “aware” of Doritos? What about launching new flavors or packaging, some reason to buy?

I have watched the doritos crystal ball commercial like 5 times now but I still won’t buy chips.

Or GoDaddy, which so alienated both male and female customers that by midnight Sunday their one competitor was offering a “DumpGoDaddy” coupon to anyone who wanted to switch hosting companies, and that competitors 1and1 and Register are getting awareness bumps:

@kdpaine oppty for someone to provide a classy, female friendly web host alternative to GoDaddy I’d switch now.

Whatever the fallout from the Superbowl ad spot, it’s the flavor acquired over time that lingers on the palate. Think about Steelers fans. I dare you to walk into the Steelers bar in your city or town (and chances are that you already know which one it is) and talk smack about Roethlisberger or Holmes. At any point in the next 25 years. You do not, if you follow sports AT ALL, need to know that this would be a bad idea. I could almost hear them in Florida from here in San Francisco. Yeah their buzz is up- the most Superbowl wins ever!- but the basic Steelers identity remains unchanged: tough, blue collar, fanatic, an American legacy, the beloved flagship franchise of benighted post-industrial Pittsburgh. That’s the Steelers brand. Never seen an ad about it.

For those professionals who need to manage their brands and companies in the everyday, in the here and now, with few Superbowl moments to provide glory, there aren’t always cues that say “Look here. Look at me now. Pay attention to my opinions” and if there are, they’re likely to be PR disasters (how familiar were you with the Peanut Corporation of America before the salmonella outbreak?). The best way to do conversational marketing, to mine the world of consumer opinion and stay on top of customer zeitgeist, is not to be event driven. It’s to be everyday insight driven. A smattering of relevant blogs and tweets for Superbowl advertisers:

i just saw a kid crush up 2 bags of doritos and pour them over his salad.

Nice, gas light just came on at 450 miles on the tank! #hyundai

Toyota working on solar-powered cars. Will it work in Seattle? Doubtful.

How much more interesting than ad ranking is the insight that people smash up Doritos and use them as toppings on other foods? Could that become a product extension or a marketing campagn? How interesting that people are noticing miles per tank for their cars — talk about a viral promotion idea! How interesting that there are doubts about the efficacy of solar powered vehicles in cloudy climates. Is this a real product deficiency or a marketing challenge? These are the customer insights, this is the context that product people and marketing people and customer service people need to have in order to understand, to evolve, to compete. In marketing, every day is game day.