Delimiter

This article is about delimiters in computing. For delimiters in written human languages, see interword separation.

File:Csv delimited000.svg

A stylistic depiction of a fragment from a CSV-formatted text file. The commas (shown in red) are used as field delimiters.

A delimiter is a sequence of one or more characters used to specify the boundary between separate, independent regions in plain text or other data streams. An example of a delimiter is the comma character, which acts as a field delimiter in a sequence of comma-separated values.

Delimiters represent one of various means to specify boundaries in a data stream. Declarative notation, for example, is an alternate method that uses a length field at the start of a data stream to specify the number of characters that the data stream contains.

Overview

Delimiters can be broken down into:

• Field and record delimiters; and
• Bracket delimiters.

Field and record delimiters

Field delimiters separate data fields. Record delimiters separate groups of fields.

For example, the CSV file format uses a comma as the delimiter between fields, and an end-of-line indicator as the delimiter between records. For instance:

    
fname,lname,age,salary
nancy,davolio,33,$30000
erin,borakova,28,$25250
tony,raphael,35,$28700

specifies a simple flat file database table using the CSV file format.

Conventions

Computing platforms historically use certain delimiters by convention. The following tables depict just a few examples for comparison.

Programming languages (See also, Comparison of programming languages (syntax)).

	String Literal	End of Statement
Pascal	singlequote	semicolon
C	doublequote, singlequote	semicolon

Field and Record delimiters (See also, ASCII, Control character).

	End of Field	End of Record	End of File
Unix, Mac OS X, Amiga OS	Tab	LF	none
Windows, MS-DOS, OS/2, CP/M	Tab	CRLF	Control-Z (optional)
Classic Mac OS, AppleDOS, ProDOS, GS/OS	Tab	CR	none
ASCII/Unicode	UNIT SEPARATOR Position 31 (U+001F)	RECORD SEPARATOR Position 30 (U+001E)	FILE SEPARATOR Position 28 (U+001C)

Bracket delimiters

Bracket delimiters (also block delimiters, region delimiters or balanced delimiters) mark both the start and end of a region of text. They are used in almost all programming languages, including Wikicode.

Common examples of bracket delimiters include:

Delimiters	Description
( and )	Parentheses. The Lisp programming language syntax is cited as recognizable primarily by its use of parentheses.
{ and }	Curly brackets.
[ and ]	Square brackets.
< and >	Angle brackets.
" and "	commonly used to denote string literals.
' and '	commonly used to denote string literals.
<? and ?>	used to indicate XML processing instructions.
/* and */	used to denote comments in some programming languages.
{| and |}	used to indicate a table in Wikicode.
<% and %>	used in some web templates to specify language boundaries. These are also called template delimiters.

Delimiter collision

Delimiter collision is a problem that occurs when an author or programmer introduces delimiters into text without actually intending them to be interpreted as boundaries between separate regions. In the case of XML, for example, this can occur whenever an author attempts to specify an angle bracket character. In most file types there is both a field delimiter and a record delimiter, both of which are subject to collision. In the case of comma-separated values files, for example, field collision can occur whenever an author attempts to include a comma as part of a field value (e.g., salary = "$30,000"), and record delimiter collision would occur whenever a field contained multiple lines. Both record and field delimiter collision occur frequently in text files.

In some contexts, a malicious user or attacker may seek to exploit this problem intentionally. Consequently, delimiter collision can be the source of security vulnerabilities and exploits. Malicious users can take advantage of delimiter collision in languages such as SQL and HTML to deploy such well-known attacks as SQL injection and cross-site scripting, respectively.

Solutions

Because delimiter collision is a very common problem, various methods for avoiding it have been invented. Some authors may attempt to avoid the problem by choosing a delimiter character (or sequence of characters) that is not likely to appear in the data stream itself. This ad hoc approach may be suitable, but it necessarily depends on a correct guess of what will appear in the data stream. Other, more formal conventions are therefore applied as well.

ASCII delimited text

The ASCII and Unicode character sets were designed to solve this problem by the provision of non-printing characters that can be used as delimiters. These are the range from ASCII 28 File Separator to ASCII 31 Unit Separator. The use of ASCII 31 Unit separator as a field separator and ASCII 30 Record separator solves the problem of both field and record delimiters that appear in a text data stream.

Escape character

One method for avoiding delimiter collision is to use escape characters. From a language design standpoint, these are adequate, but they have drawbacks:

• text can be rendered unreadable when littered with numerous escape characters;
• text becomes difficult to parse through regular expression
• they require a mechanism to "escape the escapes" when not intended as escape characters; and
• although easy to type, they can be cryptic to someone unfamiliar with the language.

Escape sequence

Escape sequences are similar to escape characters, except they usually consist of some kind of mnemonic instead of just a single character. One use is in string literals that include a doublequote (") character. For example in Perl, the code:

print "Nancy said \x22Hello World!\x22 to the crowd.";  ### use \x22

produces the same output as:

print "Nancy said \"Hello World!\" to the crowd.";      ### use escape char

One drawback of escape sequences, when used by people, is the need to memorize the codes that represent individual characters (see also: character entity reference, numeric character reference).

Dual quoting delimiters

In contrast to escape sequences and escape characters, dual delimiters provide yet another way to avoid delimiter collision. Some languages, for example, allow the use of either a single quote (') or a double quote (") to specify a string literal. For example in Perl:

print 'Nancy said "Hello World!" to the crowd.';

produces the desired output without requiring escapes. This approach, however, only works when the string does not contain both types of quotation marks.

Padding quoting delimiters

In contrast to escape sequences and escape characters, padding delimiters provide yet another way to avoid delimiter collision. Visual Basic, for example, uses double quotes as delimiters. This is similar to escaping the delimiter.

print "Nancy said ""Hello World!"" to the crowd."

produces the desired output without requiring escapes. Like regular escaping it can, however, become confusing when many quotes are used. The code to print the above source code would look more confusing:

print "print ""Nancy said """"Hello World!"""" to the crowd."""

Multiple quoting delimiters

In contrast to dual delimiters, multiple delimiters are even more flexible for avoiding delimiter collision.

For example in Perl:

print qq^Nancy doesn't want to say "Hello World!" anymore.^;
print qq@Nancy doesn't want to say "Hello World!" anymore.@;
print qq§Nancy doesn't want to say "Hello World!" anymore.§;

all produce the desired output through use of the quotelike operator, which allows any convenient character to act as a delimiter. Although this method is more flexible, few languages support it. Perl and Ruby are two that do.

Content boundary

A content boundary is a special type of delimiter that is specifically designed to resist delimiter collision. It works by allowing the author to specify a long sequence of characters that is guaranteed to always indicate a boundary between parts in a multi-part message, with no other possible interpretation.

This is usually done by specifying a random sequence of characters followed by an identifying mark such as a UUID, a timestamp, or some other distinguishing mark. (See e.g., MIME, Here documents).

Whitespace or indentation

Some programming and computer languages allow the use of whitespace delimiters or indentation as a means of specifying boundaries between independent regions in text. Python and YAML are prominent examples.

Regular expression syntax

In specifying a regular expression, alternate delimiters may also be used to simplify the syntax for match and substitution operations in Perl.

For example, a simple match operation may be specified in Perl with the following syntax:

$string1 = 'Nancy said "Hello World!" to the crowd.';    # specify a target string
print $string1 =~ m/[aeiou]+/;                           # match one or more vowels

The syntax is flexible enough to specify match operations with alternate delimiters, making it easy to avoid delimiter collision:

$string1 = 'Nancy said "httq://Hello/World.htm" is not a valid address.'; # target string
   
print $string1 =~ m@httq://@;       # match using alternate regular expression delimiter
print $string1 =~ m{httq://};       # same as previous, but different delimiter
print $string1 =~ m!httq://!;       # same as previous, but different delimiter

ASCII armor

Although principally used as a mechanism for text encoding of binary data, "ASCII armoring" is a programming and systems administration technique that also helps to avoid delimiter collision in some circumstances. This technique is contrasted from the other approaches described above because it is more complicated, and therefore not suitable for small applications and simple data storage formats. The technique employs a special encoding scheme, such as base64, to ensure that delimiter characters do not appear in transmitted data.

This technique is used, for example, in Microsoft's ASP.NET web development technology, and is closely associated with the "VIEWSTATE" component of that system.

Example

The following simplified example demonstrates how this technique works in practice.

The first code fragment shows a simple HTML tag in which the VIEWSTATE value contains characters that are incompatible with the delimiters of the HTML tag itself:

<input type="hidden" __VIEWSTATE="BookTitle:Nancy doesn't say "Hello World!" anymore." />

This first code fragment is not well-formed, and would therefore not work properly in a "real world" deployed system.

In contrast, the second code fragment shows the same HTML tag, except this time incompatible characters in the VIEWSTATE value are removed through the application of base64 encoding:

<input type="hidden" __VIEWSTATE="Qm9va1RpdGxlOk5hbmN5IGRvZXNuJ3Qgc2F5ICJIZWxsbyBXb3JsZCEiIGFueW1vcmUu" />

This prevents delimiter collision and ensures that incompatible characters will not appear inside the HTML code, regardless of what characters appear in the original (decoded) text.

See also

• Delimiter-separated values
• String literal
• CamelCase (used in WikiWikiWeb as an alternate method of link creation that does not require delimiters to indicate links)
• Federal Standard 1037C (contains a simple definition for "delimiter")
• Naming collision
• Sigil