Most SEO work is pattern work. You redirect a thousand old product URLs after a migration, block an entire folder of PDFs from being crawled, or hunt down every 404 that came from a single campaign. Doing any of that one URL at a time eats hours you will never get back. Regular expressions, universally shortened to regex, let you describe the pattern once and handle every matching URL with a single rule.
Regex has a reputation for being cryptic, and the syntax does look intimidating at first glance. The good news is that you do not need to learn all of it. A small handful of characters covers nearly everything an SEO will ever use. In this guide we will learn exactly those characters, then apply them to redirects, robots.txt rules, Search Console filters, and log file analysis, with examples you can adapt to your own site.
A regular expression is a way of describing a pattern of text instead of one literal string. The pattern /blog/ matches only that exact path. The pattern /blog/.* matches every URL that starts with /blog/. That tiny change is the difference between writing one rule and writing five hundred individual lines.
A regex engine reads your pattern and checks strings against it, answering one simple question: does this text match? If it does, the rule applies. If not, the rule is skipped. Nearly every SEO tool that filters or rewrites URLs, from Apache and nginx to Screaming Frog and Google Search Console, understands the same core syntax, which makes this a skill that transfers everywhere you work.
The fastest way to get comfortable is to experiment live. Keep our free regex tester open in a tab while you read, and paste in some of your own URLs to try each example as you go.
There are dozens of regex features, but these are the ones that show up in day-to-day SEO work. Learn this short list and you can read and write almost any rule you will encounter:
. matches any single character: a letter, a digit, a slash, anything.* matches zero or more of the character before it.+ matches one or more of the character before it.? makes the previous character optional, or tames a greedy match.^ anchors the match to the start of the string.$ anchors the match to the end of the string.\ escapes the next character, so \. matches a literal dot.[abc] matches any single character inside the brackets.[^abc] matches any single character that is not inside the brackets.(a|b) matches either a or b.(...) groups characters and captures what it matched for reuse.\d matches any digit, \w any word character, \s any whitespace.The classic SEO gotcha is the unescaped dot. In regex, . means "any character", so the pattern example.com also matches exampleXcom or example-com. When you mean a literal dot in a domain or file extension, always write \. instead. This single habit will prevent more broken rules than anything else you learn here.
Redirects are where regex pays for itself immediately. Imagine a site migration that moved every post from /old-blog/slug to /blog/slug. Without regex you would write one redirect per post. With it, you write one line:
RedirectMatch 301 ^/old-blog/(.*)$ /blog/$1
The ^ and $ lock the pattern to the full path, (.*) captures whatever comes after /old-blog/, and $1 reuses that captured slug in the new URL. That one line redirects /old-blog/seo-tips to /blog/seo-tips, /old-blog/anything-else to /blog/anything-else, and every other post the same way.
If hand-writing server syntax makes you nervous, our htaccess redirect generator builds the exact rule for you, including the 301 status code and the L flag, so you can paste it straight into your file without worrying about a typo.
Two warnings before you deploy. First, keep redirect chains short: pointing A to B to C wastes crawl budget and leaks link equity at every hop, so always redirect straight to the final destination. Second, test every pattern against a real list of your URLs, because a single bad rule can redirect your entire site to one page.
Google's robots.txt dialect supports only a small slice of regex: the * wildcard and the $ end anchor. That is still enough to express most crawl rules you actually need:
User-agent: *
Disallow: /*.pdf$
Disallow: /*?sort=
Allow: /assets/*.css$
The first line blocks every PDF on the site, the second blocks every URL containing ?sort=, and the third explicitly allows CSS files inside the assets folder. Notice how *.pdf$ simply reads as "anything ending in .pdf".
These rules are deceptively easy to get subtly wrong, and a misplaced wildcard can block your whole site from Google. Build them with a robots.txt generator rather than typing them freehand, then verify the result in Search Console before you rely on it.
Google Search Console's page filter accepts full regex, which turns the Performance report into a precision instrument. Want to see only blog posts that are not tutorials? The pattern ^/blog/(?!.*tutorial).*$ uses a negative lookahead to exclude any blog URL containing the word "tutorial". Want every AMP page at a glance? Try /amp$.
Server logs are the other big use case. A single grep command can answer questions that would take an hour in a spreadsheet. This one lists every document a crawler downloaded from your site:
grep -E '"GET [^"]+\.(pdf|xlsx|docx)' access.log | grep -i 'bot'
It matches any GET request for a PDF, spreadsheet, or Word file, then keeps only the lines that mention a bot. Patterns like this are how you audit exactly what search engines spend their time downloading, which is a core part of managing your crawl budget on larger sites.
Regex is unforgiving, and the same small errors trip up almost everyone at first:
example.com matches exampleXcom. Write example\.com.blog matches /blog/, but it also matches /my-blogroll/ and /reblog/. Use ^/blog/ when you mean the start of the path..*: .* grabs as much as it possibly can. Combined with anchors this is usually fine, but in the middle of a pattern it can swallow more than you expect. .*? matches as little as possible./Blog/ does not match /blog/ unless you enable case-insensitive matching, with the NC flag in Apache or (?i) at the start of the pattern.(.*)* can grind a regex engine to a halt on long URLs, a problem called catastrophic backtracking. Keep patterns simple and flat.Reading about regex is not enough; you learn it by writing it. Try these three, then check your answers with a tester:
1. Match every URL under /products/ that ends in .html. Answer: ^/products/.*\.html$ 2. Match paginated pages like /shoes/page/2/ but not /shoes/page/about/. Answer: ^/shoes/page/\d+/$ 3. Match any URL containing a utm_campaign parameter, anywhere in the string. Answer: utm_campaign=
If all three made sense on the first read, you already know more regex than most SEOs.
Regex is one of those skills that keeps paying off. The ten minutes you spend learning .* and $ today will save you hours on every future migration, audit, and crawl analysis. It turns repetitive URL chores into one-liners and gives you surgical control over tools that otherwise only offer blunt filters. Start small, test everything, and keep a library of patterns that worked: you will reuse them constantly.
Ready to practice? Paste your own URLs into our free regex tester and watch your patterns match in real time.
.* mean in regex?.* combines two characters: the dot, which matches any single character, and the star, which means "zero or more of the previous character". Together they match any sequence of characters of any length, including an empty string. In SEO it is most often used with anchors, like ^/blog/.*$, to match everything under a path.
The core syntax is the same almost everywhere, but each tool supports a different subset. Apache and Search Console support full regex, while robots.txt only supports the * wildcard and the $ anchor. Always check what your specific tool supports before borrowing a pattern from somewhere else.
Because in regex a plain dot means "any character", not a literal dot. The pattern example.com will also match exampleXcom, example-com, or example/com. Writing example\.com tells the engine you mean an actual dot, which is what you want in domains, file extensions, and URL paths.
* and +?The star means "zero or more" and the plus means "one or more" of the previous character. So a* matches an empty string as well as aaa, while a+ requires at least one a. In URL patterns, .* is common for "anything here", while \d+ is better for "at least one digit here".
Yes, badly. A wrong pattern in robots.txt can block your entire site from being crawled, a bad redirect rule can send every page to one URL, and a sloppy Search Console filter can hide the data you need. Always test patterns against real URLs, including ones that should not match, before deploying them.
