Okay so you’re looking for some up-to-date best practices when it comes to doing keyword research for your next blog post. I have some good news for you: in a lot of ways, it’s much easier than it used to be.

However, the end goals have changed somewhat, and it’s important to understand the value you should expect to get from your effort.

The kind of keyword research you might be thinking of—where you assemble huge spreadsheets of keywords, and sort them by monthly search volume and difficulty—is not the most productive use of your time in 2018.

To understand why, let’s take a step back to understand how we got here, how modern search engines treat keywords, and what value we’re trying to get out of these keywords in the first place.

Old-School Keyword Research

Back in the olden days, search engines relied on literal keywords to link a search query to a web page. So, if you searched for “best machine learning courses” you would only see results which had those keywords somewhere on the page. There was no wiggle room—if you wanted to rank for that search, you needed to have it on your page.

In fact, if you wanted to have the most success, you would set up your entire page to target that specific keyword phrase. So, your title tag, meta description, h1 tag, and URL would all contain that key phrase.

In extreme cases people would buy the domain name of the search they were targeting. This practice was quickly put to a halt with the Exact Match Domain Update of September, 2012.

As you can imagine, in such a search environment it was critical to know and target every variation of a keyword that you could manage. So what has changed?

A Matter of Semantics

In August of 2013 Google released the Hummingbird algorithm update. The update allowed Google to understand the meanings behind the words rather than simply matching literal keywords as it had done in the past. This is referred to as semantic search or conversational search.

So, now when we type in “best machine learning courses” we’re going to see a much more varied set of results. As of this writing the number one organic result does not contain that key phrase:

Instead of matching exact keywords to phrases, Google is attempting to understand the intent of the searcher, and dish out pages that help fulfill that intent.

It follows that the goal of our keyword research is now to discover the most pressing needs of our potential customers so that we can be the ones who provide the answer they’re looking for.

So, if we hope to rank, we should be seeking the satisfy the users’s intent.

It doesn’t matter if a searcher types in “best machine learning courses” or “top machine learning class”. The top result is the same in both cases. Google has identified which pages are most likely to solve the user’s problem.

The Competitive Landscape

Gone are the days when you can throw some keywords on a page, get a few backlinks, and hope to rank. The level of competition has become fierce.

If you want a given piece of content to rank, you gotta beat out the competition. What’s the first step to beating the competition? Knowing the competition.

If you know the enemy and know yourself, you need not fear the result of a hundred battles.

Sun Tzu, The Art of War

Let’s review what we know so far:

• We don’t need to target exact keywords to rank for them. We must instead target customer intent.
• We’re going to have to do better than our competitors at fulfilling that intent.

So how should we approach keyword research in a way that sets us up for success?

Competitive Research > Keyword Research

Rather than spending a lot of time using keyword valuation tools, we’re going to spend our time where our users will one day find us – in the search engine results pages. There is no substitute for spending time in the SERPs.

Here’s the thing: we’re not doing keyword research. We’re doing competitive research, and we’re doing content research. We’re learning what the search landscape looks like for our user so that we can improve that landscape through our content.

Start with the most obvious search query you can think of that your potential visitors might use. Let’s stick with the example we’ve been using, “best machine learning courses”.

If we look at the SERP for that search, we see that the results are not dominated by mega-brands and/or something like Wikipedia. This is good- it tells us that we have a chance of ranking if we can create a better piece than what’s out there already.

Let’s look at the number one organic result, Every Single Machine Learning Course on the Internet, Ranked by Your Reviews.

Click on it. Scroll through it. What makes this article tick? Is the site friendly to use? Is the site bombarding us with ads?

This article succeeds because it completely and thoroughly satisfies the user’s need while providing a positive experience along the way. The article is long (more than 5,000 words), it includes embedded video, images, quotes, and links. The writing is clear and relatable, and the blog’s interface lets the content take center stage.

As a secondary effect—because the article was well-written and thorough—there is a substantial discussion in the comments section which helps to further cement the post’s number one ranking. Additionally, it has garnered many inbound links and social shares.

Continue clicking through the search results, noting the quality of the competition. Take notes, save links, and click on until you have a clear understanding of what’s out there. This is not a job for robots or automated tools; you’re going to have to engage your brain here.

Think in Buckets of Intent

While we’re spending all this time in the search results pages, we’ll encounter some really useful information from the most reliable source we have—the search engine itself.

Toward the bottom of the results you’ll see this gem:

This tells us at a minimum what topics we should consider covering in our super-amazing and thorough blog post. It doesn’t matter what the relative ranking difficulty of each key phrase is because we’re going write about all of them.

Click through to these related searches to get a feel for what kinds of articles are ranking there. In many cases you’ll see the same top content showing up for a wide variety of related searches.

The end result of this is that you’ll have a list of distinct user needs, and for each of those you’ll know what level of content will be required to make a dent in the rankings. Additionally you’ll have a list of sub-topics to cover for each of those in order to be as thorough as possible.

As always take notes, save links, drink coffee, use your brain. There is no silver bullet or magical formula to substitute for creativity and diligence.

What About Other Tools?

Although I’ve made the argument that the most useful tool is the search engine itself, there’s nothing stopping you from taking advantage of other resources. Sometimes you just want to know which of two keywords is used more, or which keyword is trending up versus down.

I regularly use Google Trends to see if a particular topic is on its way up or on its way down, and to compare it with related topics.

AnswerThePublic.com is a useful way to discover questions people are asking about a given subject. This can help us hone in a bit on what people’s needs are. If people are asking a particular question, but there are no good resources answering the question, then we’ve found a content gap waiting to be filled.

It can still be helpful to use tools like the AdWords Keyword Planner or Moz Keyword Explorer. However, I have personally found that my time is better spent just doing some searches and seeing what’s out there.

The More Things Change, the More They Stay the Same

As the field of SEO has evolved, it has become less and less about stuffing keywords on a page, and more about understanding our users and how to fulfill their needs. This is not a new trend, but rather has been the inevitable direction from the beginning. Skilled marketers have been doing this since the early days of human history.

However, many people still think of SEO as the art of the stuffed keyword, when in fact it’s much closer to marketing. When someone asks us which keywords need to go into a piece for it to rank, it’s up to us to gently inform them that they may be operating with a false premise.

In fact, the real question is what content can we create, and what form does it need to take, so that we can fully satisfy the intent of our users.