PottyMouth

© 2007 Matt Chisholm
matt dash pottymouth at mosuki dot com

What does it do?

PottyMouth transforms completely unstructured and untrusted text to valid, nice-looking, safe HTML.

PottyMouth is designed to handle input text from non-technical, potentially careless or malicious users. It produces HTML that is completely safe, programmatically and visually, to include on any web page. And you don’t need to make your users read any instructions before they start typing. They don’t even need to know that PottyMouth is being used.

What is it for?

PottyMouth is ideal for displaying blog comments, text email bodies in a web mail application or mailing list web archive, or any text fields on any site with user input text, such as a social networking, dating, or community site. In short, any input which is displayed in HTML and is input as text by a non-technical and/or untrusted user. It has been in use on mosuki.com since January 2007.

What is it not for?

PottyMouth is not intended for HTML page generation, such as writing blog entries, where the author is an authorized and trusted user who may want to exert more control over the content of his or her post. Markdown and SmartyPants together are good solutions for this.

PottyMouth is also not intended for wikis, where the text is more heavily structured and where poorly formatted or malicious input can be quickly corrected by another user. There are many good wiki packages out there; this is not one of them.

Why should I care about…?

…unstructured text input?

The average, non-technical user doesn’t care about formatting syntax and won’t take the time to learn it. PottyMouth lets your website display any user input without having to make your users learn anything. The only “syntax” that PottyMouth uses are conventions that are ubiquitous on-line. If your site displays text input via external programs, third-party sites, or other sources like email, you can't rely on your users to know about your site's text formatting conventions.

…layout-safe HTML?

You want to allow your users the freedom to put whatever they want on your site. But you don’t want badly formatted text to make that text look ugly, or to screw up the layout of other elements on the page.

…untrusted text input?

If it’s possible for an untrusted or anonymous user to input text that gets inserted in HTML on your site, you need to process that text to make sure it cannot cause problems for other visitors. If your site displays text input via external programs, third-party sites, or other sources like email, you can't control or check that text.

…secure HTML?

Allowing anyone to insert raw or even limited HTML into your site is dangerous. If an attacker can insert JavaScript, media, or malicious links into your site, he or she can cause a user or their browser to perform malicious actions or send spam, on your site or third party sites, or they can insert DHTML id attributes or JavaScript to break your DHTML/JavaScript application. If an attacker can insert CSS into your site, they can hide or override advertisements, warnings, or instructions with their own content.

What does it prevent?

PottyMouth prevents against a wide range of potential problems:

Although the problems above could be solved by simply allowing a short white-list of HTML tags and no HTML attributes whatsoever, inserting raw HTML tags is a feature that non-technical users don't need. And PottyMouth automatically detects most of the instances where the average user would want HTML tags.

PottyMouth syntax

Although PottyMouth has no syntax that users must learn, it does parse input text to transform it to HTML. It relies on some ubiquitous text formatting conventions to do the best formatting job possible.

Paragraphs, newlines, and ad-hoc lists

PottyMouth intelligently identifies paragraph breaks, newlines, and ad-hoc lists. A sequence of more than one blank line is turned into a paragraph break. Within a single paragraph, PottyMouth distinguishes between “short” and “long” lines and treats them differently.

Extensive testing has shown that fifty characters is a good threshold between short and long lines (this threshold is configurable if your data differs, however). Here’s an example:

This is some text that has been through a bunch of broken email 
programs
and got hard line wrapped really badly at some point because a programmer
was lazy. 

here's some more text where someone decided to list their favorite things:
raspberries
pink hair
devil ducks
science fiction
And that list goes right in the middle of a paragraph, but does it screw up
potty mouth? Nope.

becomes:

This is some text that has been through a bunch of broken email programs and got hard line wrapped really badly at some point because a programmer was lazy.

here’s some more text where someone decided to list their favorite things:
raspberries
pink hair
devil ducks
science fiction
And that list goes right in the middle of a paragraph, but does it screw up potty mouth? Nope.

Detecting very long sequences of non-breaking characters and inserting a soft hyphen (­) character that will cause natural text wrapping is planned for a future release.

List syntax

Support for literal lists denoted by lines beginning with 1., *, # or • (bullet, •) is planned for a future release.

Block quotes

PottyMouth identifies sequences of lines beginning with one or more > and groups them into nested sequences of <blockquote> and <p> tags. In other words, input like this:

A reply to a reply
More of the reply to the reply
> A reply to a message
> More of the reply to the message
>> The original message
>> More of the original message
> Last line of the reply
Last line of the reply to the reply

is rendered like this:

A reply to a reply
More of the reply to the reply

A reply to a message
More of the reply to the message

The original message
More of the original message

Last line of the reply

Last line of the reply to the reply

Hyperlinks

PottyMouth identifies hyperlinks beginning with these protocols: http, https, webcal, feed, ftp, news, and nntp, and ending in a valid URL. Adding new protocols is trivial.

When using PottyMouth to generate content for a web application, it expects you to provide it with a list of one or more domain names for the site. Unless you explicitly leave this name blank, PottyMouth will only hyperlink links that point to other sites.

If you want PottyMouth to hyperlink site-internal links, you must also provide it with a white-list of regular expressions that match allowed site-internal links. This allows you to denote the site-internal URLs that users can include that will become hyperlinked, and other URLs will remain as text.

For example, if you were using PottyMouth on http://www.mysite.com, and chose to allow links to posts, you would use a whitelist like https?://(www\.)?mysite\.com/viewpost\?id=\d+. Then, these URLs would get hyperlinked:

But these URLs, which might just be mis-typed, mis-encoded, or might be malicious URLs, would not get hyperlinked:

While PottyMouth should not be considered a substitute for correctly protecting against the latter two types of malicious links in your software, preventing them from being automatically hyperlinked on your site raises the bar significantly for these types of attacks.

Embedded media

PottyMouth optionally allows embedded media. URLs ending in .JPG, .JPEG, .GIF, and .PNG are considered to be embedded images, and are included as <img> tags. It also detects links to YouTube videos and embeds them using YouTube’s standard embedding syntax.

The embedded media feature is disabled by default, because it does somewhat compromise the safety of the generated HTML. Embedded media could be used to launch cross-site scripting attacks on another site, if an attacker can generate a malicious URL to the remote site that ends in JPG, GIF, or PNG. However, protecting against cross-site scripting attacks is really the responsibility of the target site, not you.

Embedded media could also be used as web bugs by a third party to collect IP addresses of visitors to your site. This could only be mitigated by running a cache which served the third party content to your site visitors, and appending the target URL onto the cache service. Adding configuration options for this is planned for a future release.

Embedded media is still relatively safe, for the following reasons:

Bold and italic

PottyMouth identifies balanced sets of * and _ and turns them into bold (<b>) and italic (<i>) tags. This support was added because this shorthand is extremely common in text input, even from non-technical users. Bold and italic can be nested; however, they cannot be overlapped and neither can be nested, at any depth, inside itself. Un-balanced * and _ are rendered literally. 

this is *bold _and italic_* or _italic *and bold*_ or just *one* or the _other_
but *I dunno _what* this_ is *supposed to* be.

produces:

this is bold and italic or italic and bold or just one or the other but I dunno _ what this _ is supposed to be.

Support for other shorthands, such as - for strikeout and = for monospaced text is a possibility, but unlikely as it requires some user knowledge, is much rarer than * and _, and would likely interfere with the normal use of those characters.

Special characters

PottyMouth renders single and double quotes, backticks, ellipsis and double-dashes into the appropriate HTML entities:

Single dashes are not converted into dash, hyphen, minus or emdash, as it is not possible to reliably detect what is the correct character to use from context. See The Trouble With EM ’n EN for more information. Because PottyMouth is intended for non-technical, novice users, there is no syntax for distinguishing these characters.

All characters that are not valid HTML, including <, >, and &, are escaped in the output.

Support for smilies and additional special characters is a future possibility.

Usage

PottyMouth is written as a Python module. To use it, first instantiate a parser and tell it what domain it’s going to be used on:

from PottyMouth import PottyMouth
pm = PottyMouth(url_check_domains=('www.mysite.com', 'mysite.com'),
                url_white_lists=('https?://www\.mysite\.com/allowed/url\?id=\d+',),
                allow_media=True)

The parse() method returns a list of PottyMouth.Node objects representing <p> nodes:

p_nodes = pm.parse(string_to_parse)

You can then iterate over the PottyMouth.Node objects and stringify them with repr():

output_string = ""
for p in p_nodes:
    output_string += repr(p)
print output_string

PottyMouth.Node objects inherit from native Python lists, so you may also iterate over their contents and convert them to whatever native XHTML objects that your application requires.

Download

PottyMouth is licensed under the BSD License. It requires Python, version 2.4 or newer, and is available in source or as .deb packages:

Demonstration

Potty input: (1000 characters maximum.)

Potty output: