This article explains what regex (or regular expressions) is and how to get started using it with web crawlers, plus some interesting real-world use cases.

String manipulation is always resource-intensive, but compiled regex in Python and Java can help keep performance issues at bay.