Why I don't trust SEO studies

I bet you read a lot of SEO content related to studies and analysis. I do it all the time. Many reputable sources run tests to know how to do a better SEO. That is not considering how many tests are being made at any given time, but unpublished because the results weren't exciting enough or because they are private intellectual property. Hell, I have done my share of such tests myself.

Given this abundance of interest in testing, we could fall in the tempations of believing that including SEO as a scientific discipline is possible.

But it is not.

We can apply a scientific method in SEO, but it is even less reliable than using the same approach for social sciences. There are a few reasons for this. I don't want to argue that such tests shouldn't be done, but professionals in this industry will do a favor to stop believing that a test of this nature is scientific and, therefore, true.

Not source data

Very few studies present the data clearly and understandably. They offer their results, not the data itself. This makes everything difficult to verify. We share links with exciting titles such as "How we improve SEO results for an ecommerce using Schema and structured data", but without any reliable way to verify any of its claims. We do because we trust the source, confirmation bias, or because it helps us to make a business case. But how many studies share their source data, and for the very few that do, have you ever looked at it?

Not peer revisions

Of course, these studies are published as is, without any actual proof, supervision, or any resemblance of verification. If you believe them, it is because you trust them, not for the data. SEO is one of the less regulated industries, so nobody gives a fuck. Confirmation bias, conveniently blurred data (for privacy reasons, of course), and sometimes just the exciting approach is enough to belive such studies.

Planet SEO has an ever-changing physics

In Planet SEO, the laws of physics are not constant. If that were the case, we would have broken the SEO code years ago. But it does not happen that way. Gravity changes five times a year, and the Plank constant is everything but constant. We can not find the truth because the truth is changing precisely to be never wholly understood.

Not meta-studies

Take a look at studies made a few years ago. Some of them seem helpful now, but many are not. In the worst case, they could be misleading and can make your efforts useless. One of the main components of the scientific method is to review not only individual studies but many of them to see patterns and reduce noise. Try to do that with SEO studies, and you will see more dispersion than throwing darts with the eyes closed.

No replicability

Replicability in scientific studies refers to the ability of researchers to reproduce the results of a study when they conduct the same experiment or follow the same methodology independently. It is a fundamental principle of scientific research and essential for establishing scientific findings' credibility and validity. Replicability helps ensure that the results of a study are not simply due to chance, bias, or unique circumstances, and it promotes the accumulation of reliable knowledge in the scientific community.

Does anything of that sound remotely close to any SEO study you have read lately?

I guess not.

So, what to do?

I wrote this because I see a lot of SEO analysts sharing things and placing their entire strategy/livelihood/trust in stuff like "a super study says Google is rewriting titles over 671.5 pixels" or "we let ChatGPT write our robots.txt", and that is fucking dangerous.

We run an eternal race, but we are doomed to fail because it is changing.
Take studies as a tendency, not a final truth. If you can think of an element that can be analyzed, search engineers have already thought of it, and Google is probably doing such an analysis at a massive scale. Think that machine learning will refine everything to an atom level, so even the best SEO study could have zero value to apply to any other website/industry/technology.

So, this is not a call to not read those posts or to disregard them completely, but to use whatever you have between your ears and critically think about them.