The mental model
A regular expression is a description of a set of strings, evaluated left to right, taking as much as it can at each step and backing off when the rest fails to match.
Almost every surprising result comes from that greediness. If a pattern matches far more than you intended, the quantifier is the first place to look.
The constructs that cover most work
Character classes in square brackets, and the shorthands for digits, word characters and whitespace. Quantifiers: optional, one or more, zero or more, and explicit counts in braces.
Anchors for start and end of string, groups in parentheses for capturing, the alternation bar for either-or, and non-greedy quantifiers when you want the shortest match.
That set handles the overwhelming majority of practical extraction and validation tasks.
Build patterns incrementally
Start with the simplest pattern that matches one example, test it, then extend. Writing a long pattern in one go and debugging it afterwards is far slower.
Always test against strings that should not match. A pattern that matches everything you throw at it is usually broken, not excellent.
Comment complex patterns, or use extended mode where the language supports it. A regular expression you cannot read in six months is a liability.
What not to match with regex
HTML and XML. Use a parser. Nested structures are outside what regular expressions describe.
Email addresses, beyond a loose sanity check. The full specification is enormous; check for an at sign and a dot, then verify by sending a message.
Anything where a dedicated parser exists: URLs, dates, CSV with quoted fields, and JSON.