How to Clean Up Messy Pasted Text

A repeatable order of operations for text pasted out of PDFs, emails and web pages.

4 min read

Work in a fixed order

Cleaning in a haphazard order creates new problems. Do it in this sequence: remove invisible characters, normalise line breaks, fix punctuation, then fix case and spacing.

Keep a copy of the original before you start. Every cleaning step is destructive.

Invisible characters first

Text copied from web pages and word processors carries non-breaking spaces, zero-width characters and stray tabs. These break search, alignment and later find-and-replace steps.

Replace non-breaking spaces with ordinary spaces, strip zero-width characters, and convert tabs to spaces unless you are working with delimited data.

Line breaks and paragraphs

Text from PDFs typically has a break at the end of every visual line. Rejoin them, then reinstate real paragraph breaks where a blank line or an indent existed.

Remove trailing spaces at the end of lines; they cause double spaces after rejoining.

Collapse runs of three or more blank lines to one.

Punctuation, case and duplicates

Decide between curly and straight quotes and apply one consistently. Mixed quotes look careless and break code and CSV files.

Fix double spaces after full stops if your style calls for single, and normalise dashes to one style.

Finally, sort and de-duplicate if you are working with a list. Doing this before the earlier steps means near-identical lines survive as separate entries.

Frequently asked questions

Tool for this

TextToolsCount, clean, convert and de-duplicate text.

Open the TextTools page

More in Text Utilities & Processing