Monitoring & Analyzing Social Media

How many results will there be? How big an account do I need?

These questions are understandably common given that our pricing is based on total number of results in your account. I hear them everyday as I work with prospective customers who are doing proposals for social media monitoring (yes, I do help with developing your business model around SM2- shoot me a note if you’d like to know more).

Given that even a moderately well-known brand can generate large numbers of results, it is important to have some idea of what you’re getting into before you commit to a plan. We have developed a tool to help with this.

First let me define ’search results’ within SM2. Search results are any mention of your keyword(s) in social media. They might be a blog post, a Tweet, a comment on a wall, a forum post, a wiki entry, etc. However, a search result is much more than that. It is also a set of data points that we collect that are associated with that result. These include all kinds of things including any demographic info, tags and categories, Alexa, Technorati and PageRank data, geo-location, etc. Up to 35 fields depending on what’s available in publically accessible places (we don’t violate EULAs).

This is important because we use that data to build our analysis of your results.

Back to the estimator tool. When you’re planning a social media monitoring project you can contact us and we’ll run your keyword phrases through the tool (it’s internal) to get you a ballpark estimate of how many results we have in our ever-expanding database aka the ’social media warehouse’. This gives you, and your client or team, an idea of the expense required to accurately measure the full response in social media- before you commit to a plan.

SM2 is not a search engine

We had an interesting conversation here at Techrigy headquarters about our technology with a guest who came by to learn about what we do. And we ran into a common misconception about SM2: That it is some kind of search engine. While it has some search capabilities, search is not its primary function.

Search solves a problem: finding specific answers to queries.

SM2 solves different problems: Finding references to specific terms across social media conversations and content and understanding who is having those conversations, what they’re saying and why. Unlike search, which seeks to supply the best answer(s), SM2 offers up all the possible results and then provides tools for organizing and understanding all of the relevant results.

SM2 has two components. There is a collection system that goes out and collects new social media results on an ongoing basis and stores those results in our Social Media Warehouse. Each result, which might be a blog post, a Tweet, meta data from a YouTube video, etc., is parsed for various data within that result. This data includes public information about the person who created the result such as location, gender, age, etc, any tagging or categorization the user has provided, things like DNS records and IP addresses, URLs, Alexa and Technorati data, etc. Each result can have up to 30 or more data fields in SM2.

The warehouse grows every day. We recently bypassed the 500 million results level and will rapidly hit one billion results as social media participation explodes. Each of those billion results will have multiple data fields which SM2 users can access. Obviously understanding all of this is a challenge. That’s where the second component of SM2 comes in.

The SM2 application front end is a set of tools for discovering conversations and understanding them without having to manually go through them one by one. Set-up gives you the ability to tailor the things you wish to monitor by using keyword phrases, excludes (phrases you do not want to find), whitelists (sources you specifically wish to monitor), etc. Once you run your search, SM2 goes into the warehouse and brings back all the results it finds that match your set-up and analyzes them. After the initial search it continues to bring back results as they are added to the warehouse until you terminate the search.

The analysis tools SM2 provides are extensive, comparable in some ways to web metrics tools like Google Analytics- except that they are very focused on the humans behind the conversations rather than traffic sources and patterns. They look at sentiment, gender, age, location, popularity, trends and themes, categories, etc.

For people who are very used to the search engine model, these differences are a little challenging to grasp. The key lies in understanding how social media differs from the traditional or web 1.0 Internet. Once you grasp that social media is primarily about communication the difference becomes easier to understand.

It’s not Big Brother, It’s public and recorded

When I tell people we monitor conversations in social media the more idealistic ones scrunch up their face and give me the big brother look as in ‘Big Brother Is Watching’. The implication is that there is something unsavory about peering into people’s private thoughts…wait a minute. Social media that is not password protected is not private.

Let me repeat that:

Social media that is not password protected is not private.

When you post, Tweet, comment, write on a wall, etc. you are essentially standing on that proverbial soapbox in the park and shouting to a crowd with one big difference:

What you are saying is being recorded, preserved and, presumably, indexed for future reference.

This is a big differentiator from private correspondence like email or phone calls that require a warrant to be monitored and/or recorded (though this administration apparently thinks otherwise for those conversations too- another subject for another place).

Social media is social and, as such, not private. So if you’re saying something you don’t want to be reminded of forever don’t say it here and cry later.

There are no take-backs in social media.

Twitter backfire

I’m not a big Twitter user nor do I follow a lot of people. However I’m noticing that those who are out there Twittering like crazy might want to rethink things a bit. Just as Googling a person became a requirement for HR people vetting resumes a few years ago, following a prospect on Twitter is surely the next reference source.

There’s a lot of Twittering going on- as I mentioned recently we collected over 45 million Tweets in the last few months and those were only Tweets with keyword phrases our SM2 users were searching on. I think that having a record of constant and often mindless Twittering could be something that could come back to haunt one in the future.

The important consideration are frequency (if he has time to do this all day, is he unemployed or wasting his employer’s time?) and subject matter (if he is doing this all day is he actually doing something productive or simply killing time?). Either negative would be a red flag for me if I was considering a candidate.

Before we had monitoring and search tools tools like our SM2 and Twitter’s own Summize, you could Tweet along all day without considering these real world consequences. Now, as micro-blogs start to have real world impact and attract analytical scrutiny, we need to consider the long term affects of telling the world that it’s Tuesday afternoon and you’re throwing them back in the neighborhood watering hole…

Social media is a public conversation that is being recorded. More on that in my next post…

Ning Thing

Ning is Marc Andreesson’s white label social network platform. It’s super simple to use which means anyone who wants to build a social network around any subject can do so in literally a few minutes. Our Techrigy network for SM2 users, Conversation, is built with Ning.

I spent a little time recently building one for a place me and my friends like to go to for live music and libations (a bar in other words) . It is really interesting watching it grow.

These things are so simple and so feature-rich that I think they change the dynamic of the Internet. Just as blogs are rapidly replacing conventional static websites, these social network platforms are replacing portals for any kind of subject.

Of course we index Ning sites in SM2.

541 million social media search results and counting

Our social media warehouse, the place where we store and index the collective search results brought back from SM2 searches, has reached a milestone: over one half a billion results collected. Not surprisingly blogs are the biggest source with over 400 million blog posts collected. Twitter, though added after our launch late last year, accounts for 43 million results, dwarfing those from any other microblog.

We save the results of user searches and store them in our ‘warehouse’ to create a historical reference in addition to our real time discovery results. Our analysis tools have also indexed all of these for the individual data fields we provide (as many as 35 per result) which include things like demographics, indications of sentiment and location, trends, authority, etc. Any SM2 user can tap into this data.

I suspect we’re going to hit the billion mark very quickly as we continually add Freemium users and as more professional power users enter the system. Anyone who thinks you can ignore the power of social media should take heed: These are just results from very specific brand and reputation-focused searches. As such they represent the tip of the iceberg in social media activity.

Is your brand being hijacked in social media?

Exxon Mobil’s was…

Thanks to Andrew from Newsvetter. BTW, if you haven’t checked out Newsvetter you should!

Three months in the social

While I’ve been a participant in social media for a long time including a personal blog over three years old, I only started thinking about it as a marketing tool when I did a product launch a few years ago- and found that reaching out to relevant blogs was the most effective activity I pursued, more effective than any of the traditional PR and advertising we did at the same time. It didn’t hurt that it was a software as a service product right at the onset of web 2.0.

Fast forward a few years and social media is my primary marketing tool but in ways that have evolved considerably. When I started working with Techrigy three months ago I was still in SEM/SEO mode. That went away fast though we do use these techniques quite effectively (please don’t send me SEO analyses of our sites- we’re quite aware of where they stand). I simply dove in to start learning about the wonderful world of social media monitoring and how people were trying to use it. I intentionally use the word ‘trying’ because it rapidly became apparent that we’re all in a continuous learning curve (and will be forever I think).

Those who dropped pre-conceptions the earliest have, IMHO, become the default thought leaders. I’ve gotten so I laugh when I read about social media ‘campaigns’, products to automate pushing messages out to social media and other broadcast mentality approaches to spreading the word or effecting change in social media. This is not the model, again IMHO!

The reason we constantly see and talk to people trying to retain this model is that the available alternatives are freaking them out:

” I’m supposed to read blogs and twitter all day and add-in stuff?”

“I don’t don’t have the time or the bandwidth for that!”

“What good is this stuff? What’s the ROI?”

Etc.

With broadcast you reach millions of people, 99.999% of whom frankly don’t give a sh*t. You spend money to reach that fragment who do. With search you focus more but you’re still seeking true intent through relevant placement. A lot better.

In social media, if you do your job and participate and carefully build a reputation you become a member of the inner circle. This membership is precarious and precious but incredibly egalalitarian. An ambitious or enthusiastic intern or career-changer can join if they prove themselves and affect the positioning of their product or service in game-changing ways- because that inner circle is incredibly influential.

Why are they so powerful? Because of the network effect. It used to be said that an angry consumer would tell ten others about their negative experience while a happy one would only tell three. With social media both can reach hundreds or thousands who in turn can influence untold thousands more. So, IMHO (again), you cannot afford to ignore or minimalize social media as a marketing tool. It’s word of mouth on steroids.

That my lesson from three months of working to build awareness of a brand and product in social media. The potential is explosively more powerful than what came before.

And equally risk-laden for those venture away from 100% honesty and transparency. Why? Because it is self-regulated by those who you are marketing to. That’s another game-changing element in this new world.

What social media sources do we index?

I’ve been asked several times for a list of what SM2 covers in its social media discovery process. The problem (and it’s not really a problem) is that we are constantly adding new sources. For example we recently added FriendFeed, Identi.ca, Pownce and Plurk.

Here’s a quick overview of what we index:

• blogs
• comments
• wikis
• forums
• public content on social networks
• meta-content on user-generated media like YouTube, Flickr, etc.
• micro-blogs like Twitter and those mentioned above

Pretty much everything we can hook into in social media. We respect end-user license agreements (EULAs) unlike some of the aggregation sites that appear to monitor social media (I am not referring to any of our legit competitors).

We also provide analysis tools to help sort through the results including:

• Sentiment- an indicator only but you can drill down to read and mark a result for accuracy
• Gender
• Age
• location
• trends and trend comparisons by date ranges, keywords and categories
• author categories- how did a social media participant categorize their conversation?
• themes- cool charts that show relationships between people, ideas and your brands and reputations
• Authority Rank
• results from Top 100 and Top 1000 blog

All of these things can be customized with rules, we offer extensive chart customization capabilities, we do exports with user-configurable fields, offer custom reporting via email and RSS and more.

I’ll be updating this list frequently as we are on a constant improvement path with SM2.

Rules of Engagement V.1, social media style

The Times today has an article about Comcast’s social media monitoring and engagement guy, Frank Eliason. Frank apparently follows comments, mostly negative, and actively responds to complaints, usually surprising people in the process. The piece is a bit overly cheery but it brings out some basic points for an engagement plan:

• You have to dedicate resources, in this case a full time community manager who is practically 24/7
• They have to be empowered to make promises and successfully execute on them
• There will be negative responses to being monitored, in this case some found it creepy
• People warm up considerably when they realize a real human is listening to their rants
• This is a very time-consuming role that requires a commitment of resources

The ROI for this kind of thing is not easily quantifiable. In this case a relatively positive article in the Times seems a pretty big win on its own.