Posted by joehall

This post was originally in YOUmoz, and was promoted to the main blog because it provides great value and interest to our community. The author’s views are entirely his or her own and may not reflect the views of SEOmoz, Inc.

About two weeks ago I gave a talk at Pubcon titled, “Global Social Media Signals For SEO“. During the end of the talk I briefly mentioned several techniques used to “game” social signals. Afterwards it was apparent that many were interested to learn more about gaming these signals. So without further ado, I would like to talk more about how to game social media signals.

Oh wait a second! It looks like I got a little ahead of myself, before we go any further I think we should try to answer one question:

Why should anyone try to game social media signals?

If you are a giant dork (like moi), then gaming social media signals can be fun! However, if you have friends in the real world then I would suggest you try and identify actionable goals with the signals you are gaming, otherwise this might turn into a waste of time. Some actionable goals would be to push content with in networks, increase user “authority”, or even game outside channels like search results.

Quick note about, gaming search results with social signals: Social media signals for search are still in its infancy. Over the last several years we have seen the engines employ a variety of different techniques to integrate social signals. However, each of these have had fluctuating significance. Therefore, I would advise that you integrate social signals as a part of a much larger marketing plan that also includes all of the fundamentals of SEO. Social media is not replacing SEO, it’s making it better.

Get Inside Your Target’s Head (psychology)

Your target here is your audience, and it’s important that you understand what they are motivated by to get them to act. Most economists will call these “incentives”. However, most of the time we aren’t using direct calls to action in social media, so these are more likely akin to dangling a worm in-front of a fish. We aren’t telling the fish to bite; we are hoping that the worm looks good enough for the fish to bite on its own.

One of the most well known psychologists of the 20th century on the topic of motivation was Abraham Maslow. Maslow is widely known for his “hierarchy of needs” which he first published in a paper in 1943. Here you can see that Maslow has classified needs into 5 sections: Physiological, Safety, Love/belonging, Esteem, and Self-actualization. Despite the fact that each of these sections clearly articulate an individual need within the human experience, I tend to disagree with the concept a hierarchy because it implies that each are separate of the others. For example I might want self-actualization and crave a cheeseburger at the same time. However, understanding Maslow’s theory can help you position your content and your sharing strategy in a way that aligns with an individual’s basic needs.

A much more recent study [PDF] published by the Journal of Marketing Research looked at what types of content are more likely to go “viral”. They concluded that content that invoked high physiological arousal was more likely to be shared or engaged.

Leveraging The Right Content

Recently Matt McGee showed us data that reveals that images draw more Facebook interactions than any other posting type. Remember that interactions aren’t the same as content consumption. For example Its clear that video consumption on the web is constantly rising. But the nature of social media relies on metrics that tend to have quick shelf life. In other words, users might watch a video or read a blog post, but when done they are ready to move on to the next piece of content, not click “like” or +1. Images are quick and can be consumed without a click through, which keeps the user close to the “share”, “retweet”, “like” buttons.

While images are ruling Facebook, animated images are killing it on Google+. Some of you might find them annoying, but it has become increasingly apparent that these zany GIFs are very popular with “shares” and “+1″‘s. Just a few days ago I got over 200 shares in less than 12 hours with an animated GIF. Check out GIFBin and browse the top rated or top tagged for animated GIFs that will work well with in Google+.

If blog posts are your cup of tea, you should learn to embed viral factors to increase on page sharing. Things like uniform image sizes and emphasizing white space can motivate more social clicks.

Images (both animated and not) can get a lot of traction, but other content types can be just as successful. The trick is finding (or creating) what works. What I typically do is troll other networks like reddit or digg looking for content that is already trending. Then I re-post on other networks. I used to call this “Retweet Bait”, but it can be applied anywhere that inner-network sharing is available.

Identifying The Right Signals

Share/Spread Signals - Gaming these signals can help not only distribute content to the widest audience, but will also put your name in-front of other users, increasing relationship metrics and improving authority. If you are a “breaking” news publisher you are going to want to focus on these metrics to influence Google’s recent freshness update and trip the query deserves freshness signal. For best results you are going to want to use clear straightforward calls to action like, please retweet, “please share”, or any other appropriate variant.

Simple Quantitative Signals – These have got to be the simplest social signals available. They are great for measuring content quality and act as a baseline for other metrics. You can easily game these by including clear straight forward calls to action. “Please support us by liking this post!” If you are using WordPress, you are going to want to check out the WP Greet Box it allows you to include a custom call to action above or below the post based on the referring URL. So if a user comes to your blog from Google+ you can include a call to action similar to: “Hey there! If you enjoy this post please +1 it!”

Authority Signals – Search engines and social networks are constantly trying to judge authority. These signals are vital to having strength in social media. The most obvious authority metric is relationship ratios on asymmetrical networks like Twitter. With asymmetrical networks we can judge authority [PDF via] by looking at the ratio to following and followers. Facebook recently added the subscribe feature which will give the ability to use this same type of analysis. Authority can also be weighted by inter-network mentions and even inbound activity.

Discussion Signals – When users comment on your content they are effectively sending a signal that your content holds value. Content that starts dialog generally also gets shared. To game this signal you are going to want to ask open-ended questions that inspire debate or dialog.

Make The Signals Pop

Getting a handful of Facebook likes or +1′s can be a good start. But to see real traction you have to make the signals “pop”. It’s hard to tell exactly what is needed to get each signal to register on the various networks, but one starting place is taking a second look at Facebook’s EdgeRank. EdgeRank is responsible for pushing the most popular content with in each Facebook social stream. From what we know, Facebook is using three main factors to influence EdgeRank:

• Affinity Score – This is a relationship metric that measures how close you are to others. If a user visits another’s page often, or sends them multiple messages the score is higher. If a user has a high affinity score with another they are more likely to show up in their social stream. You can game this score by getting users to regularly visit your Facebook page.
• Edge weight – Every time a user engages content with in the social stream, the content is given an “edge” over other less popular content. Comments, likes, and shares all count towards “edge”. You can game this metric [PDF] by asking open ended questions with an inherent bias. Here’s an example: “How badly do you think the republicans will do in this election?” Democrats will “like” (or +1) this question because of the inherent bias. Republicans will comment on it, because of the inherent bias. As a result we are gaming two of the needed metrics to influence Edge weight.
• Time decay – Fresher content is more likely to be included in the social stream. The best way to game this metric is to develop evergreen content that you can re-share periodically.

While EdgeRank is exclusive to Facebook, other networks have similar systems of ranking internal content. It is clear that Google+ is using something analogous to EdgeRank, but with two main differences: Google+ doesn’t filter content in the social stream, it just reorders it. Also, it appears that Google+ doesn’t put as much weight on affinity as Facebook does.

Kitchen Sink Strategy

In marketing (and life) I often execute what I call the kitchen sink strategy. Basically this entails throwing “everything but the kitchen sink” at a problem and seeing what works. Gaming social signals are no different. Which is why when I promote content I try to include as many of the tactics described above as possible.

For example, not long ago we launched a small site to test various marketing strategies. When it came time to test Google+ I wanted to attack the signals from all corners. Therefore, I embedded the OpenGraph image meta tag to pull a large version of our logo into the social stream. Then I designed a question with direct calls to action embedded into multiple choice answers. Coaxing the user to engage with multiple choice questions is an example of manipulating inherent cognitive biases. The result? We get basic feedback about a design and gamed Google+’s social stream with all the elements needed to make the signals “pop”.

Keep Pushing The Limits

Social media signals are gaining significance every day. However, as information sharing changes and the various social channels rise and fall in popularity, there is no set methodology you should follow now or in the future. Instead, it is important to constantly test new strategies and ideas. Good luck gaming!

Do you like this post? Yes No

Read other posts by this Author: joehall

Posted by iPullRank

Bing, Yahoo!, members of the Microsoft family, Tim, distinguished guests, inbound marketers and fellow Mozzers today we say goodbye to competitive link intelligence as offered by one of the Big 3 Search Engines. The SEO world will remember the brainchild of Tim Mayer, Yahoo! Site Explorer, as the first comprehensive tool that allowed users to find out which sites and pages were indexed, inbound links to any site and submit and track XML feeds. Yahoo, the search engine that could — and did invent the precursor to not only Bing and Google’s Webmaster Tools but also link indices such as LinkScape/Open Site Explorer and MajesticSEO. Yahoo Site Explorer was born September 29^th, 2005 and has been laid to rest November 21^st, 2011; you had an amazing run.

Yahoo! Site Explorer or YSE as you were known to those closest to you, we will remember your sleek easy to use “Don’t Make Me Think” user interface.  It was very clear what you wanted us to do and once we did it you generously shared your data with us as though we were family. You asked no questions of us unless of course we were trying to change your mind. Only then did you require us to prove ourselves. 

We will remember you for allowing us to submit feeds, URLs, sitemaps to “Tell us what we don’t know. If you don’t find a URL that you expect to be in the index, use free submit. In case you hadn’t heard, we are also accepting lists of URLs, so you don’t have to provide us one URL at a time” as Yahoo! declared at your birth.

YSE we will remember you for charming ability to show us more about ourselves and where we could improve to be better in the eyes of the Internet. You shared what information you knew about us almost as fast as you could collect it. 

YSE we will remember you for your special ability to tell us about everyone else that knew about us and where they’d shared it on the web, before you there was no comprehensive way to do that. Sure Google had the “link:” operator but it was never as forthcoming as you were.

Most importantly YSE we will remember you for telling us who linked to our competitors. This is how you truly changed the world.  We respect you and commend for all your efforts and the API that once fed a variety of tools such as BackLinkWatch and the SEOBook Link Tool Suite. Your knowledge, speed and freshness will be missed.

Many Search Marketers will pause and reflect on Yahoo Site Explorer with feelings of inspiration and serenity:

“Y!SE was the inspiration for me to raise capital and build the Linkscape index w/ Nick+Ben. I felt the web's link graph should be a resource that's available to anyone, and that's why we always made sure that users of OSE could get the same functionality Y!SE offered for free (up to 1,000 links, unlimited runs, etc.)” – Rand Fishkin (CEO & Co-Founder of SEOmoz)

“YahooSiteExplorer, to me, has become like a well-loved piece of furniture, think Grandma's old couch, that served its purpose at one point, but now sits forgotten in the parlor. Much like Grandma's old couch, I have not used it much in the past year, since I have had full access to OpenSiteExplorer and MajesticSEO, but YahooSiteExplorer will always be remembered fondly." –John Doherty

 “Feels like I'm one of the few that will miss it! I used to love some of the combinations of operators you could use on it” – Paddy Moogan  

 “cos 'All good things are wild, and free' – Henry David” –Himanshu Sharma

"Free backlink research on your competitors will never be the same" – Dennis Goedegebuure

“YSE was simple to use and a great way to find links to competitors other tools might miss.” –Joe Youngblood

"I 'grew up' with it as an SEO, had a nice little link from my desktop, was easy and convenient to access." – Carla Marshall

“No alternative is as fast as YSE. Gone are the days of same-week link reports. We'll have to get smarter (as always in SEO).” – Tre Jones

“I will miss it because it was a very fast way to see if a link was there. And because it was my 1st SEO tool (with other)” –Gianluca Fiorelli

"Even after I switched to OSE for everyday use, it was always strangely comforting to know that YSE was there. It was kind of like that friend you rarely talk to, but you know will be on the other end of the phone if you need them." -Dr. Pete

While others will dance on your grave and celebrate your death:

 “YSE was an inspired flank-attack by Yahoo on Google at the time and was the SEO tool of choice for years. But its time has come.” – Dixon Jones (Marketing Director of MajesticSEO)

“I actually won't miss it! Sorry YSE!, it was good while it lasted.” –Richard Baxter

“I won't miss YSE. There's really been little to no value in site explorer since Yahoo gave up on being a search engine.” –Bill Slawski

“I won't miss it, it was a brain melt of information with no logical organisation. There are plenty of tools that give better data” –Wayne Barker

“I agree with Bill Slawski I quit using YSE 2 years ago.” –Joe Hall

Do you like this post? Yes No

Read other posts by this Author: iPullRank

Posted by Kenny Martin

When keyword targeting is approached separately from a content creation strategy, the concocted results can often leave us scratching our heads and pointing fingers at the malformed “Frankenpages.” By fostering a more cohesive relationship between these traditionally detached endeavors, we can greatly enhance our results and deliver considerable value to our audience.

This week Rand shows us how we can move past conventional keyword targeting practices and generate web pages that won’t leave us “running for the hills.”

Video Transcription

Howdy SEOmoz fans! Welcome to another edition of Whiteboard Friday. Thrilled to have you with us. Today we’re talking about mapping keywords to content for maximum impact.

Now the problem is that a lot of folks think about the world of keyword research and keyword targeting separately from the worlds of content creation. This can happen a lot of the times because the SEO person is not always involved in the design of the content strategy or what’s going to go on the website. They’re brought in after the fact, maybe in an internal role or in an external consulting role. That can be super frustrating. Let me show you, give you an example of, sort of the traditional keyword targeting process and why this is so bad.

So here’s Mr. Biz Owner, and he would like to rank well for oven mitts. A perfectly reasonable request, want to rank for oven mitts. Great. All right. So the SEO person is brought in, and the SEO person goes, “Well, you know, I want to be able to make some changes. I need to add some content to your website.” The business owner is like, “No, no, no, no, no. I already have a page. I just want it to rank for oven mitts.” Well, okay. Let’s chose the best page you’ve got for oven mitts and we’ll try to make that one rank better. The business owner is like, “All right. All right. Good job. Good job. I appreciate that. You did good work. Now I want to rank well for heat retardant oven mitts.” The SEO is like, “Well, okay. You know what? We can modify that page again and target that particular phrase.”

But this cycle goes on and on and on. Soon enough you’ll have Frankenpage, ooh, super scary. He’s trying to target ridiculous terms like “advanced kid- friendly oven mitts for hardcore baked lentils.” You’re like, “How did this happen? How did this Frankenpage get here?”

Well, it got there because of this process, this broken process of the SEO not being the person with the authority or the influence to be able to choose what content needs to be existing on the website and what content needs to be targeting which keywords. This happens all over the Web. You can click on tons of search results in all sorts of verticals and sort of be like, “What were they thinking when they made this page?” It’s not that the website is all that bad or they have done something terrible in SEO. It’s just that it is not strategic. It is a very tactical approach to SEO, and that tends to lose out over time to pages that are built specifically for users searching for those things that deliver everything they want in the content.

So, let’s talk about a strategy to do exactly that. Over here we have a better process. No Frankenpages.

Step one: Establish the full list of keywords. Rather than going sort of one by one and saying, oh, we want to target this, we want to target that, it’s nice to be able to start with that full list of keywords. As you refine, if you need to refine that keyword list, beginning again with this process and making sure that the new keywords that you need to be targeting work into the process in this way. We’ve got our full list of keywords to target. Hopefully, we’ve figured out how valuable and important they are so we have our spreadsheet. We say, “Well, these are the top converting keywords. These are the ones that send the most traffic, and these are the ones with the lowest difficulty. So based on those three factors, this is how we want to target them.” Then we’ll map the keywords to existing content based on their relevance. So this means does the page’s content actually serve the needs of a keyword phrase that they are targeting? So, if you have a heat-retardant oven mitts page, does that actually contain heat-retardant oven mitts? Is that a full category page? Is it a subcategory page? Is it a single item that happens to be the most heat- retardant oven mitts? Is it a brand page? What is it? We make sure that it is relevant.

Second, we’re going to target user intent. This means not just thinking about whether the page is relevant for the keyword, but thinking about, “What does the user want when he gets to this page?” If I am searching for heat-retardant oven mitts, I probably want a bunch of information about why it’s heat retardant, what it’s made from, explaining to me what kind of temperatures it can handle. I want to know information about where I can buy these magical oven mitts, what the sources are, what the different brands are. I’d like to be able to filter on that data. Maybe I even want tutorials and demos on like, oh, well, this is the kinds of things that you could cook with them. Cool.

Then you can think about yourself, about conversion goals. So you make them happy and they’ll make you happy. The conversion goal can be we want them to sign up for an email, we want them to click on a button, we want them to add this to their cart, we want them to convert out of the store. Great. Whatever that is, fine, super.

Then we have step two and a half, which is sort of an interim here. The reason we’ve got it is because a lot of the times when you’re mapping keywords to content, it is not a 1:1 ratio. This again can make for Frankenpages unless you’re careful. So, you want to be selecting is this a multiple or a singular keyword page focus. Meaning for the oven mitts, for just that broad keyword phrase, I might suggest, in fact, I’d probably be very strongly suggesting to a business owner who has a website about oven mitts, that that should be one page in and of itself. We should not try to make this a multiple keyword targeting page because we don’t know what the user intent is. Someone who has that broad of a phrase is going to need to do a lot of research and discover whether they want heat retardant ones or they want ones for grills or pit fires, or they’re looking for a certain material, they want it to withstand certain temperatures, they’re looking for kid-friendly gloves, they want gloves for certain sizes, they want gloves with fingers on them or gloves that are just the classic mitt form. Whatever that is, we need to be providing them with a ton of different sorts of data. So, this page is going to have all sorts of selections and things. That has to map to A, B, and C here, or we’re going to lose out and that’s why I wouldn’t try to get a bunch of different phrases ranking for this.

You could conceivably, maybe it’s possible that you would have a page for oven mitts and oven gloves and target both on the same one. So oven mitts and gloves could be a page title, could be the target. But I don’t know. I think gloves specifies fingers and mitts specifies just like this, and then they’re the hybrid ones that has the one finger. I don’t know where those go. Kitchen people will figure that out. Don’t worry.

Then you have things like, oh, well, this page, oven mitts for kids, that can target lots of keywords like child-friendly oven mitts or kid-friendly oven mitts or children’s sizes, oven mitts in children’s sizes. So you take the user intent and the relevance of the keyword and you add those onto the page and then you can figure out what are all the pages that the kid- friendly one should target. We’ll make the most important ones in the title. We’ll put maybe the secondary ones in the body content. We’ll try and make that page work for that combination because we don’t want to build one that’s child friendly and one that’s kid friendly when they are exactly the same page just to be able to target different keywords. That generally makes no sense, because again, the link equity gets split up and Google does a lot of things with topic modeling anyway to figure out that those two are probably really similar. So that doesn’t make good sense. We can do this. So I’ll draw a tiny little oven glove right there. Oh adorable, for kids.

Then you have high-temperature oven mitts. These are, oh, they’re big and strong. They can handle a bunch of high temperatures. Oh, look at all that heat they can take. The high-temperature oven mitts could be ones that include phrases like heat resistant, heat retardant, for advanced chefs, for foodies, whatever it is. Those high-temperature oven gloves, they can target a bunch of phrases as well, but we have to go back to relevance and user intent for those.

Then finally, maybe we’ll have something in the longer tail, like pit fire mitts or pit fire gloves, and those for people who need to dig around in coals or who are doing the fancy smoking in a backyard barbecue. Whatever it is. Professional grade stuff. Fine. Cool. I don’t know. I’ll put a hammer there to indicate they’re, like, hardcore professionals. I’m not sure why.

Once you have done this process, you can then take the map of keywords that you created to content and actually go build that content to make searchers happy. This works so much better than the Frankenpage approach. I can’t even describe to you how well this will work. It doesn’t have to be right from the start. You can take an existing site right now, run through this process, and have just a huge win both in terms of your ability to target searches and rank for those keywords as well as your ability to better convert those visitors because of how you’ve targeted the relevance and the user intent.

I hope you’ve enjoyed this edition of Whiteboard Friday. We’ll see you again next week. Take care.

Video transcription by Speechpad.com

Do you like this post? Yes No

Read other posts by this Author: Kenny Martin

Posted by Dr. Pete

“No one saw the panda uprising coming. One day, they were frolicking in our zoos. The next, they were frolicking in our entrails. They came for the identical twins first, then the gingers, and then the rest of us. I finally trapped one and asked him the question burning in all of our souls – 'Why?!' He just smiled and said ‘You humans all look alike to me.’”

- Sgt. Jericho “Bamboo” Jackson

Ok, maybe we’re starting to get a bit melodramatic about this whole Panda thing. While it’s true that Panda didn’t change everything about SEO, I think it has been a wake-up call about SEO issues we’ve been ignoring for too long.

One of those issues is duplicate content. While duplicate content as an SEO problem has been around for years, the way Google handles it has evolved dramatically and seems to only get more complicated with every update. Panda has upped the ante even more.

So, I thought it was a good time to cover the topic of duplicate content, as it stands in 2011, in depth. This is designed to be a comprehensive resource – a complete discussion of what duplicate content is, how it happens, how to diagnose it, and how to fix it. Maybe we’ll even round up a few rogue pandas along the way.

Let’s start with the basics. Duplicate content exists when any two (or more) pages share the same content. If you’re a visual learner, here’s an illustration for you:

Easy enough, right? So, why does such a simple concept cause so much difficulty? One problem is that people often make the mistake of thinking that a “page” is a file or document sitting on their web server. To a crawler (like Googlebot), a page is any unique URL it happens to find, usually through internal or external links. Especially on large, dynamic sites, creating two URLs that land on the same content is surprisingly easy (and often unintentional).

II. Why Do Duplicates Matter?

Duplicate content as an SEO issue was around long before the Panda update, and has taken many forms as the algorithm has changed. Here’s a brief look at some major issues with duplicate content over the years…

The Supplemental Index

In the early days of Google, just indexing the web was a massive computational challenge. To deal with this challenge, some pages that were seen as duplicates or just very low quality were stored in a secondary index called the “supplemental” index. These pages automatically became 2nd-class citizens, from an SEO perspective, and lost any competitive ranking ability.

Around late 2006, Google integrated supplemental results back into the main index, but those results were still often filtered out. You know you’ve hit filtered results anytime you see this warning at the bottom of a Google SERP:

Even though the index was unified, results were still “omitted”, with obvious consequences for SEO. Of course, in many cases, these pages really were duplicates or had very little search value, and the practical SEO impact was negligible, but not always.

The Crawl “Budget”

It’s always tough to talk limits when it comes to Google, because people want to hear an absolute number. There is no absolute crawl budget or fixed number of pages that Google will crawl on a site. There is, however, a point at which Google may give up crawling your site for a while, especially if you keep sending spiders down winding paths.

Although the “budget” isn’t absolute, even for a given site, you can get a sense of Google’s crawl allocation for your site in Google Webmaster Tools (under “Diagnostics” > “Crawl Stats”):

So, what happens when Google hits so many duplicate paths and pages that it gives up for the day? Practically, the pages you want indexed may not get crawled. At best, they probably won’t be crawled as often.

The Indexation “Cap”

Similarly, there’s no set “cap” to how many pages of a site Google will index. There does seem to be a dynamic limit, though, and that limit is relative to the authority of the site. If you fill up your index with useless, duplicate pages, you may push out more important, deeper pages. For example, if you load up on 1000s of internal search results, Google may not index all of your product pages. Many people make the mistake of thinking that more indexed pages is better. I’ve seen too many situations where the opposite was true. All else being equal, bloated indexes dilute your ranking ability.

The Penalty Debate

Long before Panda, a debate would erupt every few months over whether or not there was a duplicate content penalty. While these debates raised valid points, they often focused on semantics – whether or not duplicate content caused a Capital-P Penalty. While I think the conceptual difference between penalties and filters is important, the upshot for a site owner is often the same. If a page isn’t ranking (or even indexed) because of duplicate content, then you’ve got a problem, no matter what you call it.

The Panda Update

Since Panda (starting in February 2011), the impact of duplicate content has become much more severe in some cases. It used to be that duplicate content could only harm that content itself. If you had a duplicate, it might go supplemental or get filtered out. Usually, that was ok. In extreme cases, a large number of duplicates could bloat your index or cause crawl problems and start impacting other pages.

Panda made duplicate content part of a broader quality equation – now, a duplicate content problem can impact your entire site. If you’re hit by Panda, non-duplicate pages may lose ranking power, stop ranking altogether, or even fall out of the index. Duplicate content is no longer an isolated problem.

III. Three Kinds of Duplicates

Before we dive into examples of duplicate content and the tools for dealing with them, I’d like to cover 3 broad categories of duplicates. They are: (1) True Duplicates, (2) Near Duplicates, and (3) Cross-domain Duplicates. I’ll be referencing these 3 main types in the examples later in the post.

(1) True Duplicates

A true duplicate is any page that is 100% identical (in content) to another page. These pages only differ by the URL:

(2) Near Duplicates

A near duplicate differs from another page (or pages) by a very small amount – it could be a block of text, an image, or even the order of the content:

An exact definition of “near” is tough to pin down, but I’ll discuss some examples in detail later.

(3) Cross-domain Duplicates

A cross-domain duplicate occurs when two websites share the same piece of content:

These duplicates could be either “true” or “near” duplicates. Contrary to what some people believe, cross-domain duplicates can be a problem even for legitimate, syndicated content.

IV. Tools for Fixing Duplicates

This may seem out of order, but I want to discuss the tools for dealing with duplicates before I dive into specific examples. That way, I can recommend the appropriate tools to fix each example without confusing anyone.

(1) 404 (Not Found)

Of course, the simplest way to deal with duplicate content is to just remove it and return a 404 error. If the content really has no value to visitors or search, and if it has no significant inbound links or traffic, then total removal is a perfectly valid option.

(2) 301 Redirect

Another way to remove a page is via a 301-redirect. Unlike a 404, the 301 tells visitors (humans and bots) that the page has permanently moved to another location. Human visitors seamlessly arrive at the new page. From an SEO perspective, most of the inbound link authority is also passed to the new page. If your duplicate content has a clear canonical URL, and the duplicate has traffic or inbound links, then a 301-redirect may be a good option.

(3) Robots.txt

Another option is to leave the duplicate content available for human visitors, but block it for search crawlers. The oldest and probably still easiest way to do this is with a robots.txt file (generally located in your root directory). It looks something like this:

One advantage of robots.txt is that it’s relatively easy to block entire folders or even URL parameters. The disadvantage is that it’s an extreme and sometimes unreliable solution. While robots.txt is effective for blocking uncrawled content, it’s not great for removing content already in the index. The major search engines also seem to frown on its overuse, and don’t generally recommend robots.txt for duplicate content.

(4) Meta Robots

You can also control the behavior of search bots at the page level, with a header-level directive known as the “Meta Robots” tag (or sometimes “Meta Noindex”). In its simplest form, the tag looks something like this:

This directive tells search bots not to index this particular page or follow links on it. Anecdotally, I find it a bit more SEO-friendly than Robots.txt, and because the tag can be created dynamically with code, it can often be more flexible.

The other common variant for Meta Robots is the content value “NOINDEX, FOLLOW”, which allows bots to crawl the paths on the page without adding the page to the search index. This can be useful for pages like internal search results, where you may want to block certain variations (I’ll discuss this more later) but still follow the paths to product pages.

One quick note: there is no need to ever add a Meta Robots tag with “INDEX, FOLLOW” to a page. All pages are indexed and followed by default (unless blocked by other means).

(5) Rel=Canonical

In 2009, the search engines banded together to create the Rel=Canonical directive, sometimes called just “Rel-canonical” or the “Canonical Tag”. This allows webmasters to specify a canonical version for any page. The tag goes in the page header (like Meta Robots), and a simple example looks like this:

When search engines arrive on a page with a canonical tag, they attribute the page to the canonical URL, regardless of the URL they used to reach the page. So, for example, if a bot reached the above page using the URL “www.example.com/index.html”, the search engine would not index the additional, non-canonical URL. Typically, it seems that inbound link-juice is also passed through the canonical tag.

It’s important to note that you need to clearly understand what the proper canonical page is for any given website template. Canonicalizing your entire site to just one page or the wrong pages can be catastrophic.

(6) Google URL Removal

In Google Webmaster Tools (GWT), you can request that an individual page (or directory) be manually removed from the index. Click on “Site configuration” > “Crawler access”, and you’ll see a series of 3 tabs. Click on the 3rd tab, “Remove URL”, to get this:

Since this tool only removes one URL or path at a time and is completely at Google’s discretion, it’s usually a last-ditch approach to duplicate content. I just want to be thorough, though, and cover all of your options. An important technical note: you need to 404, Robots.txt block or Meta Noindex the page before requesting removal. Removal via GWT is primarily a last defense when Google is being stubborn.

Update: In the comments, Taylor pointed out that Google lifted the requirement that you have to first block the page to request removal. Removal requests can be done without blocking via other means now, but the removals only last 90 days.

(7) Google Parameter Blocking

You can also use GWT to specify URL parameters that you want Google to ignore (which essentially blocks indexation of pages with those parameters). If you click on “Site Configuration” > “URL parameters”, you’ll get a list something like this:

This list shows URL parameters that Google has detected, as well as the settings for how those parameters should be crawled. Keep in mind that the “Let Googlebot decide” setting doesn’t reflect other blocking tactics, like Robots.txt or Meta Robots. If you click on “Edit”, you’ll get the following options:

Google changed these recently, and I find the new version a bit confusing, but essentially “Yes” means the parameter is important and should be indexed, while “No” means the parameter indicates a duplicate. The GWT tool seems to be effective (and can be fast), but I don’t usually recommend it as a first line of defense. It won’t impact other search engines, and it can’t be read by SEO tools and monitoring software. It could also be modified by Google at any time.

(8) Bing URL Removal

Bing Webmaster Center (BWC) has tools very similar to GWT’s options above. Actually, I think the Bing parameter blocking tool came before Google’s version. To request a URL removal in Bing, click on the “Index” tab and then “Block URLs” > “Block URL and Cache”. You’ll get a pop-up like this:

BWC actually gives you a wider range of options, including blocking a directory and your entire site. Obviously, that last one usually isn’t a good idea.

(9) Bing Parameter Blocking

In the same section of BWC (“Index”), there’s an option called “URL Normalization”. The name implies Bing treats this more like canonicalization, but there’s only one option – “ignore”. Like Google, you get a list of auto-detected parameters and can add or modify them:

As with the GWT tools, I’d consider the Bing versions to be a last resort. Generally, I’d only use these tools if other methods have failed, and one search engine is just giving you grief.

(10) Rel=Prev & Rel=Next

Just this year (September 2011), Google gave us a new tool for fighting a particular form of near-duplicate content – paginated search results. I’ll describe the problem in more detail in the next section, but essentially paginated results are any searches where the results are broken up into chunks, with each chunk (say, 10 results) having its own page/URL.

You can now tell Google how paginated content connects by using a pair of tags much like Rel-Canonical. They’re called Rel-Prev and Rel-Next. Implementation is a bit tricky, but here’s a simple example:

In this example, the search bot has landed on page 3 of search results, so you need two tags: (1) a Rel-Prev pointing to page 2, and (2) a Rel-Next pointing to page 4. Where it gets tricky is that you’re almost always going to have to generate these tags dynamically, as your search results are probably driven by one template.

While initial results suggest these tags do work, they’re not currently honored by Bing, and we really don’t have much data on their effectiveness. I’ll briefly discuss other methods for dealing with paginated content in the next section.

(11) Syndication-Source

In November of 2010, Google introduced a set of tags for publishers of syndicated content. The Meta Syndication-Source directive can be used to indicate the original source of a republished article, as follows:

Even Google’s own advice on when to use this tag and when to use a cross-domain canonical tag are a little bit unclear. Google launched this tag as “experimental”, and I’m not sure they’ve publicly announced a status change. It’s something to watch, but don’t rely on it.

Update (11/21/11): For even more confusion, Google has recently added the "standout" tag. This is supposed to be used when you break a news story, but the interplay between it and syndication-source is unclear. Again, I wouldn't rely on these tags for now. Thanks to SEO Workers for pointing this out in the comments.

(12) Internal Linking

It’s important to remember that your best tool for dealing with duplicate content is to not create it in the first place. Granted, that’s not always possible, but if you find yourself having to patch dozens of problems, you may need to re-examine your internal linking structure and site architecture.

When you do correct a duplication problem, such as with a 301-redirect or the canonical tag, it’s also important to make your other site cues reflect that change. It’s amazing how often I see someone set a 301 or canonical to one version of a page, and then continue to link internally to the non-canonical version and fill their XML sitemap with non-canonical URLs. Internal links are strong signals, and sending mixed signals will only cause you problems.

(13) Don’t Do Anything

Finally, you can let the search engines sort it out. This is what Google recommended you do for years, actually. Unfortunately, in my experience, especially for large sites, this is almost always a bad idea. It’s important to note, though, that not all duplicate content is a disaster, and Google certainly can filter some of it out without huge consequences. If you only have a few isolated duplicates floating around, leaving them alone is a perfectly valid option.

V. Examples of Duplicate Content

So, now that we’ve worked backwards and sorted out the tools for fixing duplicate content, what does it actually look like in the wild? I’m going to cover a wide range of examples that represent the issues you can expect on a real website. Throughout this section, I’ll reference the solutions listed in Section IV – for example, a reference to a 301-redirect will cite (IV-2).

(1) “www” vs. Non-www

For sitewide duplicate content, this is probably the biggest culprit. Whether you’ve got bad internal paths or have attracted links and social mentions to the wrong URL, you’ve got both the”www” version and non-www (root domain) version of your URLs indexed:

Most of the time, a 301-redirect (IV-2) is your best choice here. This is a common problem, and Google is good about honoring redirects for cases like these.

You may also want to set your preferred address in Google Webmaster Tools. Under “Site Configuration” > “Settings”, you should see a section called “Preferred domain”:

There’s a quirk in GWT where, to set a preferred domain, you may have to create GWT profiles for both your “www” and non-www versions of the site. While this is annoying, it won’t cause any harm. If you’re having major canonicalization issues, I’d recommend it. If you’re not, then you can leave well enough alone and let Google determine the preferred domain.

(2) Staging Servers

While much less common than (1), this problem is often also caused by subdomains. In a typical scenario, you’re working on a new site design for a relaunch, your dev team sets up a subdomain with the new site, and they accidentally leave it open to crawlers. What you end up with is two sets of indexed URLS that look something like this:

Your best bet is to prevent this problem before it happens, by blocking the staging site with Robots.txt (IV-3). If you find your staging site indexed, though, you’ll probably need to 301-redirect (IV-2) those pages or Meta Noindex them (IV-4).

(3) Trailing Slashes ("/")

This is a problem people often have questions about, although it's less of an SEO issue than it once was. Technically, in the original HTTP protocol, a URL with a trailing slash and one without it were different URLs. Here's a simple example:

These days, almost all browsers automatically add the trailing slash behind the scenes and resolve both versions the same way. Matt Cutts did a recent video suggesting that Google automatically canonicalizes these URLs in "the vast majority of cases".

(4) Secure (https) Pages

If your site has secure pages (designated by the “https:” protocol), you may find that both secure and non-secure versions are getting indexed. This most frequently happens when navigation links from secure pages – like shopping cart pages – also end up secured, usually due to relative paths, creating variants like this:

Ideally, these problems are solved by the site-architecture itself. In many cases, it’s best to Noindex (IV-4) secure pages – shopping cart and check-out pages have no place in the search index. After the fact, though, your best option is a 301-redirect (IV-2). Be cautious with any sitewide solutions – if you 301-redirect all “https:” pages to their “http:” versions, you could end up removing security entirely. This is a tricky problem to solve and should be handled carefully.

(5) Home-page Duplicates

While problems (1)-(3) can all create home-page duplicates, the home-page has a couple unique problems of its own. The most typical problem is that both the root domain and the actual home-page document name get indexed. For example:

Although this problem can be solved with a 301-redirect (IV-2), it’s often a good idea to put a canonical tag on your home-page (IV-5). Home pages are uniquely afflicted by duplicates, and a proactive canonical tag can prevent a lot of problems.

Of course, it’s important to also be consistent with your internal paths (IV-12). If you want the root version of the URL to be canonical, but then link to “/index.htm” in your navigation, you’re sending mixed signals to Google every time the crawlers visit.

(6) Session IDs

Some websites (especially e-commerce platforms) tag each new visitor with a tracking parameter. On occasion, that parameter ends up in the URL and gets indexed, creating something like this:

That image really doesn’t do the problem justice, because in reality you can end up with a duplicate for every single session ID and page combination that gets indexed. Session IDs in the URL can easily add 1000s of duplicate pages to your index.

The best option, if possible on your site/platform, is to remove the session ID from the URL altogether and store it in a cookie. There are very few good reasons to create these URLs, and no reason to let bots crawl them. If that’s not feasible, implementing the canonical tag (IV-5) sitewide is a good bet. If you really get stuck, you can block the parameter in Google Webmaster Tools (IV-7) and Bing Webmaster Central (IV-9).

(7) Affiliate Tracking

This problem looks a lot like (6) and happens when sites provide a tracking variable to their affiliates. This variable is typically appended to landing page URLs, like so:

The damage is usually a bit less extreme than (5), but it can still cause large-scale duplication. The solutions are similar to session IDs. Ideally, you can capture the affiliate ID in a cookie and 301-redirect (IV-3) to the canonical version of the page. Otherwise, you’ll probably either need to use canonical tags (IV-5) or block the affiliate URL parameter.

(8) Duplicate Paths

Having duplicate paths to a page is perfectly fine, but when duplicate paths generate duplicate URLs, then you’ve got a problem. Let’s say a product page can be reached one of 3 ways:

Here, the iPad2 product page can be reached by 2 categories and a user-generated tag. User-generated tags are especially problematic, because they can theoretically spawn unlimited versions of a page.

Ideally, these path-based URLs shouldn’t be created at all. However a page is navigated to, it should only have one URL for SEO purposes. Some will argue that including navigation paths in the URL is a positive cue for site visitors, but even as someone with a usability background, I think the cons almost always outweigh the pros here.

If you already have variations indexed, then a 301-redirect (IV-2) or canonical tag (IV-5) are probably your best options. In many cases, implementing the canonical tag will be easier, since there may be too many variations to easily redirect. Long-term, though, you’ll need to re-evaluate your site architecture.

(9) Functional Parameters

Functional parameters are URL parameters that change a page slightly but have no value for search and are essentially duplicates. For example, let’s say that all of your product pages have a printable version, and that version has its own URL:

Here, the “print=1” URL variable indicates a printable version, which normally would have the same content but a modified template. Your best bet is to not index these at all, with something like a Meta Noindex (IV-4), but you could also use a canonical tag (IV-5) to consolidate these pages.

(10) International Duplicates

These duplicates occur when you have content for different countries which share the same language, all hosted on the same root domain (it could be subfolders or subdomains). For example, you may have an English version of your product pages for the US, UK, and Australia:

Unfortunately, this one’s a bit tough – in some cases, Google will handle it perfectly well and rank the appropriate content in the appropriate countries. In other cases, even with proper geo-targeting, they won’t. It’s often better to target the language itself than the country, but there are legitimate reasons to split off country-specific content, such as pricing.

If your international content does get treated as duplicate content, there’s no easy answer. If you 301-redirect, you lose the page for visitors. If you use the canonical tag, then Google will only rank one version of the page. The “right” solution can be highly situational and really depends on the risk-reward tradeoff (and the scope of the filter/penalty).

(11) Search Sorts

So far, all of the examples I’ve given have been true duplicates. I’d like to dive into a few examples of “near” duplicates, since that concept is a bit fuzzy. A few common examples pop up with internal search engines, which tend to spin off many variants – sortable results, filters, and paginated results being the most frequent problems.

Search sort duplicates pop up whenever a sort (ascending/descending) creates a separate URL. While the two sorted results are technically different pages, they add no additional value to the search index and contain the same content, just in a different order. URLs might look like:

In most cases, it’s best just to block the sortable versions completely, usually by adding a Meta Noindex (IV-4) selectively to pages called with that parameter. In a pinch, you could block the sort parameter in Google Webmaster Tools (IV-7) and Bing Webmaster Central (IV-9).

(12) Search Filters

Search filters are used to narrow an internal search – it could be price, color, features, etc. Filters are very common on e-commerce sites that sell a wide variety of products. Search filter URLs look a lot like search sorts, in many cases:

The solution here is similar to (11) – don’t index the filters. As long as Google has a clear path to products, indexing every variant usually causes more harm than good.

(13) Search Pagination

Pagination is an easy problem to describe and an incredibly difficult one to solve. Any time you split internal search results into separate pages, you have paginated content. The URLs are easy enough to visualize:

Of course, over 100s of results, one search can easily spin out dozens of near duplicates. While the results themselves differ, many important features of the pages (Titles, Meta Descriptions, Headers, copy, template, etc.) are identical. Add to that the problem that Google isn’t a big fan of “search within search” (having their search pages land on yours).

In the past, Google has said to let them sort pagination out – problem is, they haven’t done it very well. Recently, Google introduced Rel=Prev and Rel=Next (IV-10). Initial data suggests these tags work, but we don’t have much data, they’re difficult to implement, and Bing doesn’t currently support them.

You have 3 other, viable options (in my opinion), although how and when they’re viable depends a lot on the situation:

1. You can Meta Noindex,Follow pages 2+ of search results. Let Google crawl the paginated content but don’t let them index it.
2. You can create a “View All” page that links to all search results at one URL, and let Google auto-detect it. This seems to be Google’s other preferred option.
3. You can create a “View All” page and set the canonical tag of paginated results back to that page. This is unofficially endorsed, but the pages aren’t really duplicates in the traditional sense, so some claim it violates the intent of Rel-canonical.

Adam Audette has a recent, in-depth discussion of search pagination that I highly recommend. Pagination for SEO is a very difficult topic and well beyond the scope of this post.

(14) Product Variations

Product variant pages are pages that branch off from the main product page and only differ by one feature or option. For example, you might have a page for each color a product comes in:

It can be tempting to want to index every color variation, hoping it pops up in search results, but in most cases I think the cons outweigh the pros. If you have a handful of product variations and are talking about dozens of pages, fine. If product variations spin out into 100s or 1000s, though, it’s best to consolidate. Although these pages aren’t technically true duplicates, I think it’s ok to Rel-canonical (IV-5) the options back up to the main product page.

One site note: I purposely used “static” URLs in this example to demonstrate a point. Just because a URL doesn’t have parameters, that doesn’t make it immune to duplication. Static URLs (parameter-free) may look prettier, but they can be duplicates just as easily as dynamic URLs.

(15) Geo-keyword Variations

Once upon a time, “local SEO” meant just copying all of your pages 100s of times, adding a city name to the URL, and swapping out that city in the page copy. It created URLs like these:

In 2011, not only is local SEO a lot more sophisticated, but these pages are almost always going to look like near-duplicates. If you have any chance of ranking, you’re going to need to invest in legitimate, unique content for every geographic region you spin out. If you aren’t willing to make that investment, then don’t create the pages. They’ll probably backfire.

(16) Other “Thin” Content

This isn’t really an example, but I wanted to stop and explain a word we throw around a lot when it comes to content: “thin”. While thin content can mean a variety of things, I think many examples of thin content are near-duplicates like (14) above. Whenever you have pages that vary by only a tiny percentage of content, you risk those pages looking low-value to Google. If those pages are heavy on ads (with more ads than unique content), you’re at even more risk. When too much of your site is thin, it’s time to revisit your content strategy.

(17) Syndicated Content

These last 3 examples all relate to cross-domain content. Here, the URLs don’t really matter – they could be wildly different. Examples (17) and (18) only differ by intent. Syndicated content is any content you use with permission from another site. However you retrieve and integrate it, that content is available on another site (and, often, many sites).

While syndication is legitimate, it’s still likely that one or more copies will get filtered out of search results. You could roll the dice and see what happens (IV-13), but conventional SEO wisdom says that you should link back to the source and probably set up a cross-domain canonical tag (IV-5). A cross-domain canonical looks just like a regular canonical, but with a reference to someone else’s domain.

Of course, a cross-domain canonical tag means that, assuming Google honors the tag, your page won’t get indexed or rank. In some cases, that’s fine – you’re using the content for its value to visitors. Practically, I think it depends on the scope. If you occasionally syndicate content to beef up your own offerings but also have plenty of unique material, then link back and leave it alone. If a larger part of your site is syndicated content, then you could find yourself running into trouble. Unfortunately, using the canonical tag (IV-5) means you'll lose the ranking ability of that content, but it could keep you from getting penalized or having Panda-related problems.

(18) Scraped Content

Scraped content is just like syndicated content, except that you didn’t ask permission (and might even be breaking the law). The best solution: QUIT BREAKING THE LAW!

Seriously, no de-duping solution is going to satisfy the scrapers among you, because most solutions will knock your content out of ranking contention. The best you can do is pad the scraped content with as much of your own, unique content as possible.

(19) Cross-ccTLD Duplicates

Finally, it’s possible to run into trouble when you copy same-language content across countries – see example (9) above – even with separate Top-Level Domains (TLDs). Fortunately, this problem is fairly rare, but we see it with English-language content and even with some European languages. For example, I frequently see questions about Dutch content on Dutch and Belgian domains ranking improperly.

Unfortunately, there’s no easy answer here, and most of the solutions aren’t traditional duplicate-content approaches. In most cases, you need to work on your targeting factors and clearly show Google that the domain is tied to the country in question.

VI. Which URL Is Canonical?

I’d like to take a quick detour to discuss an important question – whether you use a 301-redirect or a canonical tag, how do you know which URL is actually canonical? I often see people making a mistake like this:

The problem is that “product.php” is just a template – you’ve now collapsed all of your products down to a single page (that probably doesn’t even display a product). In this case, the canonical version probably includes a parameter, like “id=1234”.

The canonical page isn’t always the simplest version of the URL – it’s the simplest version of the URL that generates UNIQUE content. Let’s say you have these 3 URLs that all generate the same product page:

Two of these versions are essentially duplicates, and the “print” and “session” parameters represent variations on the main product page that should be de-duped. The “id” parameter is essential to the content, though – it determines which product is actually being displayed.

So, consider yourself warned. As much trouble as rampant duplicates can be, bad canonicalization can cause even more damage in some cases. Plan carefully, and make absolutely sure you select the correct canonical versions of your pages before consolidating them.

VII. Tools for Diagnosing Duplicates

So, now that you recognize what duplicate content looks like, how do you go about finding it on your own site? Here are a few tools to get you started – I won’t claim it’s a complete list, but it covers the bases:

(1) Google Webmaster Tools

In Google Webmaster Tools, you can pull up a list of duplicate TITLE tags and Meta Descriptions Google has crawled. While these don’t tell the whole story, they’re a good starting point. Many URL-based duplicates will naturally generate identical Meta data. In your GWT account, go to “Diagnostics” > “HTML Suggestions”, and you’ll see a table like this:

You can click on “Duplicate meta descriptions” and “Duplicate title tags” to pull up a list of the duplicates. This is a great first stop for finding your trouble-spots.

(2) Google’s Site: Command

When you already have a sense of where you might be running into trouble and need to take a deeper dive, Google’s “site:” command is a very powerful and flexible tool. What really makes “site:” powerful is that you can use it in conjunction with other search operators.

Let’s say, for example, that you’re worried about home-page duplicates. To find out if Google has indexed any copies of your home-page, you could use the “site:” command with the “intitle:” operator, like this:

Put the title in quotes to capture the full phrase, and always use the root domain (leave off “www”) when making a wide sweep for duplicate content. This will detect both “www” and non-www versions.

Another powerful combination is “site:” plus the “inurl:” operator. You could use this to detect parameters, such as the search-sort problem mentioned above:

The “inurl:” operator can also detect the protocol used, which is handy for finding out whether any secure (https:) copies of your pages have been indexed:

You can also combine the “site:” operator with regular search text, to find near-duplicates (such as blocks of repeated content). To search for a block of content across your site, just include it in quotes:

I should also mention that searching for a unique block of content in quotes is a cheap and easy way to find out if people have been scraping your site. Just leave off the “site:” operator and search for a long or unique block entirely in quotes.

Of course, these are just a few examples, but if you really need to dig deep, these simple tools can be used in powerful ways. Ultimately, the best way to tell if you have a duplicate content problem is to see what Google sees.

(3) SEOmoz Campaign Manager

If you’re an SEOmoz PRO member, you have access to some additional tools for spotting duplicates in your Campaigns. In addition to duplicate page titles, the Campaign manager will detect duplicate content on the pages themselves. You can see duplicate pages we’ve detected from the Campaign Overview screen:

Click on the “Duplicate Page Content” link and you’ll not only see a list of potential duplicates, but you’ll get a graph of how your duplicate count has changed over time:

The historical graph can be very useful for determining if any recent changes you’ve made have created (or resolved) duplicate content issues.

Just a technical note, since it comes up a lot in Q&A – Our system currently uses a threshold of 95% to determine whether content is duplicated. This is based on the source code (not the text copy), so the amount of actual duplicate content may vary depending on the code/content ratio.

(4) Your Own Brain

Finally, it’s important to remember to use your own brain. Finding duplicate content often requires some detective work, and over-relying on tools can leave some gaps in what you find. One critical step is to systematically navigate your site to find where duplicates are being created. For example, does your internal search have sorts and filters? Do those sorts and filters get translated into URL variables, and are they crawlable? If they are, you can use the “site:” command to dig deeper. Even finding a handful of trouble spots using your own sleuthing skills can end up revealing 1000s of duplicate pages, in my experience.

I Hope That Covers It

If you’ve made it this far: congratulations – you’re probably as exhausted as I am. I hope that covers everything you’d want to know about the state of duplicate content in 2011, but if not, I’d be happy to answer questions in the comments. Dissenting opinions are welcome, too. Some of these topics, like pagination, are extremely tricky in practice, and there’s often not one “right” answer. Finally, if you liked my panda mini-poster, here’s a link to a larger version of Pandas Take No Prisoners.

Update: Post-publication, a handful of people requested a stand-alone PDF version of the post. You can download it here (22 pages, 560KB).

Do you like this post? Yes No

Read other posts by this Author: Dr. Pete

Posted by russvirante

This post was originally in YOUmoz, and was promoted to the main blog because it provides great value and interest to our community. The author’s views are entirely his or her own and may not reflect the views of SEOmoz, Inc.

Howdy, Mozzers. This is Russ Jones (@rjonesx) from Virante, Inc. I recently spoke at the Search Exchange conference in Charlotte, NC on the topic of programmatic, automated SEO solutions and realized that it could probably be more valuable in front of a larger audience. Of course, the attendees have a head start, so you better get to work.

I have a confession to make. I love infomercials. In fact, I would probably call myself an infomercial elitist / hipster. I liked infomercials before they were cool; before the Billy Mays and Slap Chop Guy made their way into internet memes. I pledge my allegiance to the godfather of infomercials, Ron Popeil, while guys like Anthony Sullivan weep at his alter, asking forgiveness for their sub-par jobs as pitchmen. OK, maybe I take it a little too seriously – I do happen to have a DVR full of Gator Grip, Ginsu Knives, and Flowbees – but I believe there is something extremely motivating about this type of advertising. And Ron Popeil hit it on the head over and over again: Set It and Forget It.

This was the tag line for the Ronco Showtime Rotisserie, an amazing success for infomercials. You see, there is an innate desire for us to find solutions to common, everyday problems that do not require our attention. These nagging, annoying problems like making dinner, cleaning up, and in our industry – SEO tedium – tend to suck up our time and attention while bringing only marginal improvements. 

Unfortunately, there is this perception, almost bias, against automation in our space: a misbelief that there is nothing that we can set and forget in SEO. Well, I am here today to free you from the reigns of some of your daily miseries of  SEO, all for the incredible price of free. 

Strategy 1: Real Time Referrer Indexing

We often joke that "Google knows everything." While we can lament the loss of privacy and liberty, there is one thing that I do want Google to know about – my links. I want them to know about as many links pointing to my site as possible. Unfortunately, Google misses out on a good portion of the web. Well, what if you could find links that Google hasn't necessarily found, and then make sure that Google does index them and count them? Introducing Real Time Referrer Indexing:

If you were go into your Google Analytics right now and export all of the pages that have sent visitors to your site since your website's inception, what percentage of them do you think will have been indexed by Google? 90%, 95%, 99%? Sure, it will probably vary from site to site, especially given how many different sites out there have sent traffic to you, but there are likely to be a handful that Google never got around to crawling. Our goal with this first set-it and forget-it tactic is to find the pages that refer traffic to your site on-the-fly and make sure if they have a link, that Google knows about it.

Ideally, our automated solution would work like this…

1. The script would record every referrer from other sites.
2. The script would spider that site to see if it actually has a real, followed link.
3. The script would check to see if Google had cached that referring page with the followed link.
4. The script would coax Google to reindex that page if it had not yet found the link.
5. The script would continue to check to see if Google had cached the referring page.

This is actually quite easy to accomplish programmatically. The first three steps are done every day by tools regularly used by SEOs.The only difficult part is finding a way to encourage Google to visit the referring pages it has not yet indexed. We can solve this by simply having a widget on the page that displays those referrers, essentially an "As Seen On" bulleted list of pages that had linked to your site, but had not yet been indexed. 

Well, I have a treat for those of you who are or know someone with some half way decent programming skills. Here is sample code that does just this on your typical LAMP (Linux, Apache, MySQL, PHP) installation. A word of warning – it is highly likely that this code is buggy. Make sure that you check it and make modifications before running it on production. All you need to do is install the script on any pages of your site for which you would like to perform real time referrer indexing.

This is exactly the type of set-it-and-forget-it SEO that I love. Simple techniques, simple solutions, long-term results.

So let's move on to another set-it-and-forget-it technique.

Strategy 2: On-the-Fly PageRank Recovery

Alright, so if you haven't heard of PageRank Recovery before, you are going to need a quick little lesson. Whenever someone links to your site, but screws up the URL, the PageRank that flows through that link essentially evaporates. I am pretty sure that it ends up in Matt Cutt's personal PageRank stash, which he has learned to convert into a powerful foodstuff that he consumes prior to mountain climbing and running marathons. But I digress, if you can find where those broken links point to on your site, then 301 those URLs to a real page, you can "recover" that PageRank. Virante created a tool to do just that based on SEOMoz's Site Intelligence API which Rand highlighted a little while ago, but it still requires you spend time going and running the tool regularly. I want to be lazy and have my site recover PageRank for me while I watch The Facts of Life dressed in a Snuggie and downing 5 hour energy shots. So here is how it would work:

Ideally, our program would do the following…

1. The script sits in your CMS right before a 404 is fired. If you don't have a CMS, you would direct your HTACCESS file to pass all 404 traffic through it first.
2. The script captures the URL that the visitor or GoogleBot tried to visit.
3. The script somehow magically knows what URL you MEANT to visit.
4. The script 301 redirects you there.

What's that you say? "But Russ, our programmers don't know magic. They are all muggles. And even if they did know magic, I can't find a USB powered wand anywhere these days." Well, I am bringing you good news from some friends: Mr. XML Sitemap and Ms. Levenshtein. 

If you were paying attention to countless blog posts in the SEO world, you should have an XML Sitemap which keeps record of all the URLs on your site. This is a good start to the magic that is On-The-Fly PageRank Recovery, because now we know all the possible URLs your visitor or GoogleBot may have been trying to reach. Now, we simply have to find the most similar URL to the one the visitor came to. How do we accomplish this? Levenshtein Distance.

Levenshtein Distance, also known as the Edit Distance, is a measurement of the minimum number of changes necessary to convert one piece of text into another by adding a letter, removing a letter, or substituting a letter. For example, the Levenshtein Distance between the words "Rock" and "Russ" is 3, because we will have to substitute the O, C, and K with U, S, and S. Below is an example of how Levenshtein Distance could be used to find two similar URLs:

So, the way On-the-Fly PageRank Recovery works is by reading all the URLs in your sitemap and then comparing the Edit Distance between those URLs and the URL your visitor entered. If the server finds a close match, we then 301 redirect rather than show a 404 error. Subsequently, when a Googlebot tries to visit those previously 404 pages, it will instead find that 301 redirect and appropriately pass the PageRank through to the intended page. Plus, On-the-Fly PageRank Recovery is a huge usability win for visitors who now don't have to try and search your site to find the correct page.

Want to give it a test drive? Try any one of these broken links back to Virante and my blog, TheGoogleCache

• Virante's Tool Page: http://www.virante.com/se9-toolz
• Second Page Poaching: notice the dollar sign in the url

Now, It would be hypocritical of me to talk about setting it and forgetting it, and then make you go out and do all the work yourself to get it up and running. So, in the spirit of laziness, I have included a couple of options for you to use as well. Of course, double-check everything before you go into production with any code you ever get on the internet, regardless of whether or not it is on a trusted site like SEOmoz.

• WordPress Plugin
• Drupal Module
• Generic PHP Code

Final Thoughts

There are incredible opportunities in the world of Search Engine Optimization that we have only begun to address. So much more can be done in terms of describing, detecting, and repairing SEO issues all in a programatic, automated fashion. These are just two of them. Good luck, and keep inventing!

Do you like this post? Yes No

Read other posts by this Author: russvirante

Posted by Cyrus Shepard

SEOs don't talk about advertising much, perhaps because it's the conceptual opposite of “great content.” The truth is, advertising is the gasoline that runs much of the web. Without ad revenue, great sites we love like Search Engine Land, Smashing Magazine, and even Wired might cease to exist.

Ads are great, but as SEOs we need to present them as the commercials that they are, not the main show.

Optimizing for CTR the Old Way

Not long ago, it was common to see sites like this dominating the SERPs.

(Thanks to Michael Gray for the lead)

When Panda struck, sites like this got hit hard, time and time again. Even websites with superior content were penalized if they contained over-aggressive ads above the fold. I don't know if the site above was penalized by Panda, but I'm guessing their traffic is not as healthy as it could be, and a simple layout change would help significantly.

1. Ads as a Ranking Factor

The 2011 Ranking Factors showed a slight negative correlation between rankings and the amount of Adsense on a page.

Several Panda updates have rolled out since this data was collected, and I would expect the relationship today to be even more negative.

Although Adsense isn't the only game on the market, it's the one ad network SEOs get the most information from. Matt Cutts has said that his team sends one way messages to the Adsense team in order to help webmasters comply with Google quality guidelines.

In April, after Panda hit, Adsense changed how they advocate best practices for ad placement. Gone (or at least tucked away) were the old heat maps.

2. Panda Friendly Layouts

The new layouts specifically advocate for ads that do not push content below the fold.

These are the types of layouts that should be safe no matter what kind of ads you run. You can see earlier versions in their one-click optimizer, but these older layouts don't go very far in placing content first. Use at your own risk.

3. Balance Your Template Footprint

Ads are a component your template footprint. A template footprint is any non-unique content that appears on every page, as opposed to content that makes the page unique.

It's best to keep your ratio of unique content to footprint as high as possible. If you can't reduce your template footprint, at least place your content in the highest, most prominent place possible in order to stay out of the penalty zone.

4. Future Proof Your Ads

The new Adsense recommendations are great for this round of Panda, but what about next year? In my opinion, they represent the minimum of what you should do to avoid a penalty.

The New York Times does a good job of balancing ads against content. Their strategy neither ignores users nor puts them at risk for near-future algorithm changes.

Aggressive ads tend to alienate users, which can effect your bounce rate, time on site, pageviews and other user engagement metrics. All of these can have undesirable long term consequences. For publishers dedicated to long term profits, there is a better approach.

5. Beyond CTR – Smart Ways to Increase Ad Revenue

It's true that higher click-through rates give webmasters incentive to place ads above content. But CTR isn't the only way to increase earnings. You can optimize several other factors to your long term advantage. If you are an Adsense publisher, you are familiar with these concepts.

1. Coverage
2. Cost-Per-Click (CPC)
3. Cost Per Impression (CPI or CPM)
4. Impressions

All of these can be optimized for higher earnings. Number 4, impressions, is the most actionable from an SEO point of view. If you're producing great content and promoting it the right way, then your pageviews will soar. Here in the States, the SuperBowl will always make more in ad revenue than reruns of Murder, She Wrote.

If you sell ads, be the SuperBowl of content publishers. Produce the best content you can, and you can sell your premium ad space for top dollar.

Do you like this post? Yes No

Read other posts by this Author: Cyrus Shepard

Posted by Stephen

This post was originally in YOUmoz, and was promoted to the main blog because it provides great value and interest to our community. The author’s views are entirely his or her own and may not reflect the views of SEOmoz, Inc.

Illustrated London News has a 170 year history as a content and print company. Recently we made the obvious move to bring one of our print publications online – PODIUM an intelligent view of sport.

Our major articles are in-depth interviews with sports stars, our commentary is from globally renowned pundits and we often do our own photo shoots. Cover stars so far have been Usain Bolt and Frankie Dettorri. Other Articles have focused on Golf, F1, The All Blacks, horse racing and the 2012 Olympics.

On the face of it, that sounds like an SEO's dream – rich, unique content that search engines will love. It's a pity that does not help us one little bit.

Am I going to get Search Engine traffic for "Usain Bolt" on a brand new website? To beat his own website, his sponsor Puma, the BBC or The Guardian? Not a chance.

So I decided to abandon the un-loyal scraps of long-tail search and to Design for Social

This is the tale of what I did to grow traffic for www.thepodiummagazine.com/ when SEO wasn't a viable option. There is still a lot more to do, but I think we have learned enough that it's worth sharing, for the attitude, strategy and outcomes so far.

By engaging people in a social context, we can keep them coming back with each release and create own our own traffic streams and marketing channels outside of Google and search engines.

WHAT DOES DESIGNING FOR SOCIAL MEAN?

PODIUM Version 1 with Facebook given equal weight as a web article
 

We had a clear strategy to leverage Social from Day One

1. Only the best goes online

I chose to put less online. Only the best articles, and pieces we thought would spark debate, made it online. This means we didn't water down the user experience – readers only get the good stuff. Where Search would say "Stick everything online and pray for longtail" I believe the mantra for Social is "Don't bore me. Blow me away!".

2. Twitter is for the Insider's View

Twitter would be owned by the Print Editor, (Andy @sportingpodium). This means we have a highly knowledgeable sporting writer who is able to engage online with the people we cover. When you cover Usain Bolt and your writers are guys like F1's David croft, getting them to retweet your coverage of them IS your Twitter strategy.

This is a virtuous circle of promotion, everybody taking part wins. Build this into your products and you have a marketing beast.

3. Facebook is for Debate

For Facebook, we decided to pick the most contentious article each edition and put it behind a like wall on Facebook.

The article would sit on the site homepage, looking like an article, but when clicked on, would take people to the Facebook page. Our hope is that people will debate these articles on our Facebook page, thereby taking advantage of Facebook Edgerank, to make these articles pop up into everyone's feeds.

4. It's all one product concept = better use of time and energy

This meant we could cut down our energy expenditure on where we were trying to funnel people. Instead of diluting our energy trying to get people to Facebook, Twitter and the Website, we focus on the website and allow the strategy and mechanisms built into our use of Twitter and Facebook to naturally accrue users on those platforms.
 

WHAT WE LEARNT

PODIUM Version 2 with Twitter and Facebook taking pride of place

5. Do more of what is successful

By the time we got round to Version 2, we found that Twitter was a steady audience builder. We wanted to promote Twitter in the same way as Facebook. We didn't do that by slapping a Twitter button onto the webpage.

In the new design, we made Twitter and a Facebook a living part of the website. Social is not an afterthought, the website is now a Social Content Delivery Mechanic.

6. Not everything succeeds

Facebook is hard for us and we haven't cracked creating the conversation there yet. This is partly due to the exciting rigours of turning print writers into digital writers. But each success that we do make, in traffic spikes and twitter followers, builds a stronger and stronger internal business case to pursue this route with other titles.

7. Jump on every opportunity

This means monitor your Analytics daily! We need to turn every scrap of attention into engagement. To do this you need to react quickly.

The debate piece

The first spike was a forum that had picked up our F1 piece, which certainly provoked some controversy.

I read their discussion and realised I could add something to it, so joined the forum and posted. This engagement kept the debate going, drove more traffic to the site and means we can go back in future and promote other F1 stories.

Note: I posted openly and clearly as Podium, clarifying a point without appearing spammy.

The Wow! Piece

I saw this traffic spike in the Analytics and tracked it back to the artist's Dribbble page. He had created a Pixorama for us to illustrate a story and linked to us from his Dribble account to say it was going to be in the Magazine shortly

This was not going to appear online but when I saw it and the traffic it was generating to our site from Dribbble, I knew it had to go on the site. I'm sure it is going to turn into awesome linkbait. 

8. Partnerships = Win

This comes in two forms

Firstly, competitions for partners help us drive traffic but also allow us to make connection with brands in a scenario in which our first interaction is that we help them

Secondly, as we write our own, exclusive content with major stars, we can share some of it with another website. They will send traffic and link to us to read the full article. Sharing your unique content is something that would be hard to do if you had your SEO hat on, but in the social world, its fine.

As long as you have mechanisms set up to capture that visiting traffic on Facebook and Twitter, you are building your long term marketing channel. 

9. Tools help

We learnt what generated buzz and discussion and tried to work that into our future thinking. F1 has a crazy community!

A tool like Followerwonk would be heresy to traditional print journalists as a means for deciding who is newsworthy, but it can now become part of our process in choosing who to cover, and who to talk to about specific sporting articles. 

Followerwonk allows you to see the most influential people who have a keyword in their bios.
 

IN CONCLUSION

We still have a ton to do. But we have great, unique content at our disposal and real subject matter experts to create our conversations.

Long term, I am sure (and relieved) that the traffic we have seen and the community we are building will be a much better investment of our time and scarce resources than pure SEO.

Do you like this post? Yes No

Read other posts by this Author: Stephen

Posted by randfish

For many SEO professionals, on-page optimization is back to basics. But sadly, there seem to be a lot of us who still make some very basic mistakes. In this post, I’ll try to add on to my previous writing on perfecting on-page optimization by sharing some visuals that can hopefully help to hammer key points of the practice home.

#1: The Value of Optimization and the Danger of Overdoing It

I generally abide by the 80/20 rule when it comes to keyword use. 80% of the value to be had comes from 20% of the effort. Nail the title, the headline and make sure the phrase is on the page (and the page is actually on the subject of the keyword) and you’ve done your job. The additional impact on rankings to be gained from perfectly calculating the number of repetitions or ensuring every paragraph fits into the “theme” of the keyword and document is likely to be a waste of time better spent on other priorities. That’s what this graph tries to illustrate:

Nothing in the on-page world is going to provide exceptional ranking influence, but getting perfect is often only marginally better than just nailing the title and headline. If you’re spending a ton of bandwidth on the last 80% of work (providing 20% of value), I might re-consider your to-do list.

#2: On-Page SEO is More than Keywords

When I first got into SEO in the early 2000′s, the search engines seemed to have a fairly naive algorithm for content analysis, which led to SEOs adopting equally naive tactics for on-page optimization. Years later, these tendencies, sadly, still persist.

Yes, it’s essential to effectively target your keywords in your page titles, headlines (or early in the body copy), URLs, etc. But content analysis has become far more sophisticated with engines “reading” pages almost the way humans do and pattern-matching good content, design, layout and usability. There may even be some elements of on-page analysis that look at the authenticity and passion of the written word (or something that approximates it).

In the graphics below, I’ve tried to illustrate this:

I’m not suggesting one shouldn’t optimize for keywords or that using terms and phrases that stay relevant and on-topic won’t help. I’m merely noting that optimizing for the experience real, human users have and the value they derive from your work can produce outsized returns to simple, classic on-page optimization.

#3: How Search Engines Can Measure a Page’s Value

This one’s less of an illustration and more of a text-based diagram. I wanted to help explain all the signals Google can measure from their many sources of information, and how this can potentially affect SEO:

Google’s tremendous reach across the Internet, measuring nearly everything and every way people interact with web pages brings with it powerful data. That data is likely used to improve the quality of search results by helping the stuff that appears authentic, editorial and high quality to rise up while the junk falls to the bottom (at least when the data+algorithms work properly).

For on-page optimization, this means we can’t merely focus on keyword targeting. We need truly great content.

#4: Consolidation vs. Multi-Page Targeting

The question of whether to target two keyword terms/phrases on the same page or build individual pages for each seems to be a consistent struggle for SEOs. I field a question like this almost every week, and in 9/10 cases, the following flowchart would provide the right answer:

It’s not complex – and that’s the beauty. When targeting similar phrases or phrases that can work together and target the same intent for most users, a single page should suffice. When the phrases cannot logically work together in a title/headline or when the intents don’t have a high liklihood for overlap, it’s time to build different pages and target the keywords separately.

Feel free to use these in your presentations, websites and internal/client documents (though a source credit is much appreciated). And best of luck with your keyword targeting + on-page efforts!

Do you like this post? Yes No

Read other posts by this Author: randfish

Posted by Stephanie Chang

Distilled’s Tom Critchlow recently gave a presentation on SEO Ranking Factors at Digital East. One of the key points he made is how brand links could potentially be affected by search volume. The higher the search volume for your brand, the more likely you are to appear in the brand links.

What is a Brand Link?

In May of 2010, Google announced via their blog that they had made brand refinements for product information in Google. Google’s intent was to help inform users who are looking for information about a product, but are also unfamiliar with the brands that are associated with that product. The purpose of the brand links is to make it easier for users to find "popular" brands. According to their post, these "popular" brands are determined algorithmically.

For some head terms, these brand links would appear right below the paid search ads. These are especially influential for companies in the e-commerce space because it always helps to be associated as a "popular" brand with any product. 

Many of the brand links pertain to articles of clothing, but they can also pertain to accessories (handbags).

It can even include equipment, such as bikes.

Or something as simple as office supplies, like pencils.

It’s uncertain which factors are involved in identifying "popular" brands for head terms. However, there is strong evidence to suggest that each brand’s corporate website and how optimized they are for keywords is not enough to result in a brand link. This is because many of the websites that do have brand links actually do not employ SEO best practices, such as using targeted keywords on the website.

For instance, I took a look at American Eagle to evaluate how optimized their site is for the head term "jackets", since they appear in Google’s brand links for this term. Using SEOmoz’s on-page report card tool, it was determined that American Eagle actually has an F grade for "jackets" optimization.

For the head term "jackets", American Eagle isn’t even ranked in the first 3 pages of the SERPs. Thus, what are some possible reasons why American Eagle would receive a brand link for "jackets"?

Keyword Search Volume and Its Impact on Brand Links

One possible reason is due to increased search volume for the brand "American Eagle" in conjunction with the keyword "jacket". This was identified using Google’s Insights for Search, which provides information about search volume trends. Thus, it is possible that Google is taking notice of which head terms are being associated with which brands, particularly in search queries. The rising searches below appear to be very similar to the type of brands that have brand links for the head term "jackets" (Hollister and Gap along with American Eagle).

This type of scenario also applies for other head terms, such as "sneakers".

Below is a breakdown of the totals for all the brand links + the keyword sneakers using the Google Insights for Search tool. According to Google, the totals numbers represent the number of searches conducted for a relative term compared to the number of searches on Google overtime. There were a significant number of searches for many of the brands as shown below.

Other Potential Factors

Upon further investigation, such as observing the rising search trends, there is evidence indicating that search volume is not the only factor influencing brand links. If that was the case, then "Adidas" sneakers should also have a brand link.

Although, it’s not entirely clear why "Adidas" was left out of the brand links, a couple of hypotheses can be construed. For example, perhaps Google is taking into account the number of news mentions for a brand or even the number of social media mentions.

I tested out some examples of news mentions. In this case, Adidas sneakers received 96 Google News mentions.

On the other hand, New Balance sneakers, which has a brand link, received about 202 news mentions.

This one scenario is not definitive by any means and it’s evident that more research would need to be conducted.

Similarly to news mentions, it is also possible that there might be some correlation between brand and social mentions (although it definitely does not appear to be a primary factor). To identify and quantify social mentions of handbag brands, Topsy’s analytics were used. This chart compares brand mentions of Louis Vuitton handbags, Gucci handbags, and Guess handbags. From the graph, it shows that there has been significantly and consistently more social mentions for Louis Vuitton and Gucci handbags over the past month in comparison to Guess handbags social mentions. Perhaps, this is one of many reasons why Louis Vuitton and Gucci have brand links for handbags, whereas Guess does not.

It’d be interesting to compare current handbag brand links (or any other type of brand link) month by month. Which brands retained their brand link and which ones have lost them? Perhaps, after isolating different scenarios and noticing changes overtime, a better understanding of these links could be developed.

I’m interested to know the SEO community’s experience with brand links and any observations that they have made on how these type of links have been acquired. I’m also curious to hear if any SEOs have any stories on how brand links affected brand searches/conversions and whether they are effective in creating brand awareness/increasing the amount of organic traffic or not. Please share your thoughts below; I’d love to hear from you.

Do you like this post? Yes No

Read other posts by this Author: Stephanie Chang

I’m not sure what other writers are calling Google’s algorithm update, as we’ve only had it for a day. I’m staying with the KISS (keep it simple stupid) method, and calling it “Google Fresh“. No doubt, SEO experts, webmasters, and …

Read other posts by this Author: John Vinson