How Content Marketing Can Impact Crawl Frequency
Comparing Apples To, Well, Faster Growing Apples
We’ve often speculated about how consistent content marketing would affect how often Google crawls a site (more on that later), and hypothesized the benefits it would provide, specifically to organic traffic. Sadly, we hadn’t had the opportunity to test a true apples-to-apples comparison . . . that is, until now. Recently, Strathcom—but specifically, myself—had the good fortune of launching two content marketing packages, simultaneously, for two sites with very similar starting points and business goals. Now, while these two apples may not have fallen from the same tree, they were close enough (think a Fuji and a Red Delicious).
In fact, it would be hard to find a better side-by-side comparison than this one: it was the same dealer group, with the same amount of content on their sites (next to nothing), the same content marketing package (so the same frequency of content), and the same content strategist working on the packages (quite the handsome and talented devil, I’m told). If that weren’t enough, they both represented import OEMs in the same city. One big difference: one of the sites had been up for a while, while the other site was being re-launched at the same time as it started its content marketing journey.
Now, we understand that most dealers live in a bubble of immediacy (which is a nice way of saying you demand immediate results; or, are impatient), and content marketing is all about thinking long-term. With that being said, the benefits we saw to the site that started creating content when it relaunched were substantial.
You’ve Gotta Be Crawled, Before You Can Walk
First, I suppose we should provide a little background to any readers who are unsure of what we’re talking about:
Content Marketing = excellent content creation and overarching SEO strategy to drive organic traffic to your site
SEO = search engine optimization (optimizing your content to end up in the SERP)
SERP = search engine results page (where you want to be)
Crawling (in relation to SEO) = the process in which search engines scan a website, collecting data about what’s on the site and its individual pages (so it can show up in the SERP)
Crawl frequency = the rate at which, and amount of times, your website is crawled in a given period
Google = a search engine (easily the most important one at that)
Red Delicious = a type of apple
Fuji = a mountain and, in this instance, a type of apple
So, if it weren’t clear enough, if you’re producing fresh, valuable, and unique content for your website (and you should be), the frequency at which Google crawls your website is incredibly important to said content being visible and delivered as a result in a SERP. Of course, if you’re not putting any content on your site for Google to crawl, it doesn’t really matter—just don’t expect to show up in the SERP as frequently, either. But, if you’re regularly updating content on your site, you’re giving Google a reason to crawl it more frequently (since you’re providing them with answers that they can deliver a potential user), which gives them a reason to put you in the SERP.
The Importance Of Being Crawled
Excuse the clumsy analogy, and think about your website’s content as a baby for a moment. A baby usually begins to crawl by accident, falling forward while reaching toward the shiny beer apple you left on the coffee table. As arm and leg strength builds, the baby gains mobility. But beyond mobility, it helps build the child’s motor skills (and it probably does something good for the brain, too). Now, ideally, all that work and motor skill development results in them becoming a star athlete and competing for gold on the highest stage while you cheer loudly from the sidelines—and not just because you’re living vicariously through them (thanks a lot, dad).
Now, your baby—er, website’s content will invariably be crawled by Google periodically (you can manually request crawls, but again, if you’re not creating content there’s no point, and Google will penalize you if you do it too frequently). While there’s guesses out there as to how often your site is crawled, there’s no real concrete timeline laid out by Google. But, without being crawled, your content won’t rise through the ranks and compete for the top position in the SERP—just like you won’t make it to the Olympics if your dad doesn’t take you to the meets.
There you have it, the importance of being crawled (or crawling). If you’re a baby, it could mean becoming an Olympic gold medal-winning athlete, and all the trappings that come along with it: fame, adoration, the cover of the Wheaties box. If you’re a website, it can mean becoming a favourite of Google’s, and all that entails: lower cost-per-click in your digital advertising by improving your site’s overall quality score, appearing in the SERP for users to click on, and even showing up in a Google SERP feature (the search equivalent of the Wheaties box).
You can find all sorts of information on our blog as to the importance of showing up in Google, but do we really need to tell you this? It’s how people intake information, and answer questions. Gone are the days of the Guinness Book of Records getting trotted out while trying to figure out the weight of the fattest baby ever born, replaced by smartphones and Google SERPs. If you want to sell cars, you have to show up on Google. If you want to show up on Google, you need to have a website. But it isn’t that simple anymore: you need to have an online strategy that includes digital advertising and content marketing, working in conjunction, and being crawled by Google. If you have that, you will sell more cars.
An Apples-to-Apples Comparison
Setting Up Content Marketing on Both Sites
Now that we’ve got all that out of the way, and you’re all caught up on what needs to be done (and why it’s important), let’s go back to our first comparison. When we launched their content marketing packages, we produced two blogs and two landing pages a week for both of these dealership sites—sites that had little to no content on them to begin with. Starting with a steady diet of model landing pages and related blogs, we built up a strong base of content for the sites. This was content that showed Google these two dealerships were experts in their field, showcased a clear path for the user to move through the website (thereby demonstrating its purpose and value), reinforced their respective locations through smart keyword usage, and basically said, “Hey, we’re the best place to buy X vehicles in the city of Y!”
Monitoring the Crawl Rate
In addition to creating the content, we monitored when Google crawled the sites. While we intentionally didn’t request a crawl on either site when publishing said content marketing pieces, in the process of relaunching Site 1 we had to submit a sitemap—which automatically requests a crawl of the new site. Because of this, Site 1 was crawled quite quickly, whereas Site 2 was crawled days later. Thanks to the additional content on the site between the initial re-launch crawl and the subsequent “check back” crawl (i.e. the model landing pages), Google decided to come back and crawl Site 1 again shortly after because, in their eyes, the site was including content that users would find relevant and interesting. It’s worth noting, that when we say additional content we mean more than just the standards: homepage, New Vehicle Inventory, Used Inventory, Finance, Contact Us, etc. All of these pages have a purpose, yes, but they’re not providing additional benefit to users over and above what they can find on a competitor’s website. This is especially true when you consider that Google views their Google My Business (GMB) listings as most relevant to users, since they contain all the information a customer might need to know about what your business does, why they should choose your business, and how they can find or contact your business (assuming your GMB listing is up to date and accurate). This is why when you search up a business’ phone number, you’ll see the GMB listing before you ever see a Contact Us page (but that’s a blog for another day).
While Site 2 was producing similar content (more new vehicle landing pages and related blogs), it had been a long time—if ever—that they had done so. Google wasn’t interested in Site 2 as a site that created relevant or interesting information, because historically they hadn’t. Whereas Site 1, which had just been relaunched, appeared as a new site with additional content that was, you guessed it, relevant and interesting. We continued to monitor, and saw that Site 1’s crawl rate remained more consistent than Site 2’s, at least for the first few weeks.
Eventually, Site 2’s crawl rate caught up to and mirrored that of Site 1 (with both sites maintaining a consistent and identical frequency of published content). However, the performance of Site 2 seemed to lag behind when it came to organic traffic. That’s not to suggest that Site 2 didn’t see excellent improvements in organic traffic, because they did. Only Site 1’s performance far exceeded that of Site 2.
As mentioned earlier, you can manually request a crawl on your site when you start creating and publishing content. However, if your site has been around for any extended period of time, without offering anything above and beyond what the majority of the sites in your vertical are doing, Google has less impetus to increase their crawl frequency. Also, Google factors in your habits and history. Just because you’ve created some informative content recently, there’s a precedent that stretches months (maybe even years) that says you don’t regularly create or update your content. So, exercise caution when it comes to manually requesting a crawl from Google, since doing it too often (especially without substantial updates to your content) can lead to Google penalizing you.
Google is a big company, and they’re very busy. You can request a crawl and they’ll happily oblige, but if they imagine you’re abusing their system, those sadists in Mountain View will happily punish you, too. With a shoot first, ask questions later mentality, the onus of proving that you were working within their rules is on you—and even if you they do decide that you didn’t deserve to be punished, there is nothing that can be done retroactively to your site’s performance, or the impact on your organic traffic.
How Big Are We Talkin’ Here? (Online Performance)
Whether you’re comparing apples, babies, or organic traffic, the thought process is probably “the bigger the better” (well, maybe not when it comes to babies). But we finally had a chance to compare two incredibly close cases, and the results were pretty stark. While Site 1 saw an immediate drop in user numbers, this is to be expected with a relaunched site; however, the important behaviour metrics we look at, like Pageviews, Pages per Session, and Average Session Duration, all took a large step forward. Site 2 saw slight increases across the board, with the exception of Bounce Rate—but that’s also another topic for another day, as sometimes Analytics data can be deceiving on the surface. Below are the organic traffic numbers for Sites 1 and 2, comparing the month prior to launching content marketing, as well as the three months after the package was launched (keep in mind, the package was essentially launched in the middle of month 1).
Local Organic Traffic Month-to-Month (W/ Total Percentage Changes) & Average Change
Site 1
Site 2
While it’s unlikely anyone would be tripping over themselves trying to spread the word of either sites’ performance after the first month, keep in mind that there was only about 8 pieces of content published per site, and the sites had just been crawled. Looking at the sites’ organic traffic the following two months is where we see the real leap.
Comparing the Two Sites’ Performances
Seriously though—in the second month Site 1 saw an over 50% increase in all user metrics, and large increases across the behaviour metrics, including a 96.34% increase in Pageviews. Those are the types of huge leaps that propel sites into the primetime, and onto Wheaties boxes. Let’s use organic Users as our example here. Site 1 saw a jump from 2,741 organic users to 4,191 organic users. Now, the organic portion(s) of that sentence are incredibly important. That doesn’t mean users to the website, or people who’ve clicked on their paid ads. That’s a 52.9% increase in users who’ve found Site 1 through the SERP. That’s 1,450 users who are somewhere in their car-buying journey (probably near the start), and are searching terms related to Site 1’s OEM. That’s a massive improvement in online visibility, and a major step toward directing early funnel traffic directly into your dealership.
There was nothing wrong with Site 2’s performance—though, when you’re in that bubble of immediacy, it might not seem like much. Content marketing is a long term play, one that requires time, patience, and sustained effort. If you wanted to bake an apple pie, you wouldn’t plant an apple tree and expect to harvest the fruit a month later. But, if you planted the tree, let it grow strong roots, nurtured it, and continued to provide some upkeep over time, you would reap the fruits of your labour for a long, long time.
How Relaunching the Site Impacted Crawl Rate
Launching the content marketing packages for both these sites at the same time, while relaunching Site 1, confirmed a long-held suspicion of ours: that an increased crawl rate alongside consistently content production would lead to major improvements in organic traffic. After it was relaunched, Google viewed it as a fresh site with supplementary content (that was both relevant and interesting to users). Because of this, they crawled it more frequently. Now, this combined with the fact that we were populating the site with well-researched, keyword-dense content, led to Site 1 appearing in more SERPs, and getting more organic traffic. While Site 2’s crawl rate improved over time, thanks to the frequency of content, it neither appeared in as many SERPs, nor as quickly as Site 1. With time, the performance of Site 2 would probably mimic that of Site 1; but for all those living in the bubble of immediacy might not have the patience to wait for that to happen.
If we just look at the averages of the metrics in the first three months (found below), we see a huge difference in organic traffic between sites 1 and 2. The biggest difference between the two sites? Site 1 was relaunched, and was immediately crawled more frequently.
Don’t Buy Green Bananas Unripe Apples
If we mention a publishing frequency of four times a week and you start to imagine your marketing budget disappearing at an alarming rate, keep in mind that by creating a strong base of content (i.e. planting the tree and nurturing the roots) you will need to do less work (or, spend less money) in the future. For the above sites, we made the suggestion of actually lowering the frequency of content we wrote and published; if we look back at our apple tree, the amount of work subsequent to planting it is pretty minimal. By regularly publishing even one blog or landing page a week, we will keep the forward momentum going.
Of course, healthy apple trees don’t grow in a bubble of immediacy. Impatience is a killer, and by not keeping that forward momentum going there’s a great chance all the work will be for naught. Knowing what we know about the decay of organic traffic after engaging in—then cancelling—content marketing, a huge chunk of the progress and improvements seen on any site will soon go by the wayside (we’ve got proof). In the case of these two sites, we planted the trees, tended to the roots, and started to see apples (though, more on Site 1). Without continuing to tend to the trees, pruning away the dead branches or supporting those that need support, they’ll just wither and die. If you want to make a pie once, you’re better off just going to the grocery store and buying the apples needed for the pie. If you want to continue to make pies for a long time, you’d be better off planting the tree. Keep in mind that tree can provide a nice bit of shade in the backyard, meaning it does more than just give you free apples. Again, your online advertising cost is associated with your website’s quality score, which will improve thanks to new, relevant, or updated content. In other words, you could end up saving money in the long term. But you’ve got to stick with it.
What Does All This Mean For Your Site?
Relaunching a site alongside a content marketing package because you want to see rapid improvement probably isn’t in the cards for all dealerships. In fact, it wasn’t the plan here, but to quote the late Bob Ross (and definitely not my parents), it was quite the “happy little accident.” However, if you are looking to launch or relaunch a site, now would be the time to leap headfirst into the world of content marketing, and get while the gettin’s good.
Of course, we’ve only found one “bad time” to start content marketing, which is tomorrow, or the day after. That’s because, as a long-term strategy, content marketing is a smart way to build your online relevance and presence in the SERP, alongside a number of other benefits. But it is a long-term strategy, one that requires commitment and patience.
Here at Strathcom, we don’t just talk the talk, we walk the walk, and stand by the effectiveness of our content marketing packages. Feel free to email our SEO team at seo@strathcom.com, and we’d gladly walk you through just what our content marketing packages entail. If baseball great Yogi Berra were still with us, and interested in digital marketing, he’d surely agree: sometimes you’ve gotta walk before you can get crawled.
The More You Know: An Addendum of Sorts
I have since learned that a singular apple tree will not bear fruit, and it needs a second tree nearby to cross-pollinate—but the analogy still stands. I’ve also learned that you need two people to make a baby as well, though this portion of the analogy won’t stand until somewhere between 6 and 9 months old. Also, this means that your baby is even more mobile than it was when it was crawling, which is a terrifying proposition.
Remember, you don’t need an orchard worth of trees to have an effective online marketing strategy. But a content marketing tree, alongside a paid online advertising tree could have substantial impact on your gard—er, your online presence.
Some More Fun Facts About Apples
- Archaeologists have evidence of people eating apples since 6,500 B.C.
- There are more than 8,000 varieties of apples
- Apple trees are deciduous
- Apple trees have a life expectancy of around 100 years
- Apples contain no sodium, fat, or cholesterol
- Apples come in two different quantities: pecks, and bushels
- Around 36 apples go into one gallon of apple cider (keep that in mind when you’re slugging back the Magners’ at the bar this weekend)
- George Washington pruned his apple trees as a hobby (what a nerd!)
- It takes about 10 years for an apple seed to grow into the apple you know and love—so be patient
- Stock in Apple started at $22/share (which adjusts to about $0.39/share in 2019) and created 300 millionaires in its first day being public
Content Marketing, Google Analytics, Google My Business (GMB), Performance Metrics
