Newline
From Wikipedia, the free encyclopedia
- For other uses, see New Line (disambiguation)
In computing, a newline is a special character or sequence of characters signifying the end of a line of text. The name comes from the fact that the next character after the newline will appear on a new line—that is, on the next line below the text immediately preceding the newline. The actual codes representing a newline vary between hardware platforms and operating systems, which can be problematic when exchanging data between systems of different types.
There is also some confusion as to whether newlines terminate or separate lines. If a newline is considered a separator, there will be no newline after the last line of a file. The general convention on most systems is to add a newline even after the last line, i.e. to treat newline as a line terminator. Some programs have problems processing the last line of a file if it isn't newline terminated. Conversely, programs that expect newline to be used as a separator will interpret a final newline as starting a new (empty) line. This can result in a different line count being reported for the file, but is generally harmless otherwise.
Newlines are sometimes also called line breaks and sometimes refered to as end-of-line (EOL).
Contents |
[edit] Representations
Software applications and operating systems usually represent the newline with one or two control characters:
- Systems based on ASCII or a compatible character set use either LF (Line Feed, 0x0A) or CR (Carriage Return, 0x0D) individually, or CR followed by LF (CR+LF, 0x0D 0x0A); see below for the historical reason for the CR+LF convention.
- EBCDIC systems—mainly IBM mainframe systems, including z/OS (OS/390), i5/OS (OS/400)—use NEL (Next Line, 0x15) as the newline character. Note that EBCDIC also has control characters called CR and LF, but the numerical value of LF differs from the one used by ASCII. Additionally, there are some EBCDIC variants that also use NEL but assign a different numeric code to the character.
- OpenVMS uses a record-based file system and stores text files as one record per line. No line terminators are actually stored, but the system can transparently add a terminator to each line when it is retrieved by an application.
Most textual Internet protocols (including HTTP, SMTP, FTP, IRC and many others) mandate the use of ASCII CR+LF (0x0D 0x0A) on the protocol level, but recommend that tolerant applications recognize lone LF as well. In practice, there are many applications that erroneously use the C newline character '\n' instead (see section Newline in programming languages below). This leads to problems when trying to communicate with systems adhering to a stricter interpretation of the standards; one such system is the qmail MTA (not to be confused with Gmail) that actively refuses to accept messages from systems that send bare LF instead of the required CR+LF.
[edit] Unicode
The Unicode standard addresses the problem by defining a large number of characters that conforming applications should recognize as line terminators:
LF: Line Feed, u000A
CR: Carriage Return, u000D
CR+LF: CR followed by LF, u000D followed by u000A
NEL: Next Line, u0085
FF: Form Feed, u000C
LS: Line Separator, u2028
PS: Paragraph Separator, u2029
This may seem overly complicated compared to a simple approach like converting all line terminators to a single character, for example LF. The simple approach breaks down, however, when trying to convert a text file from an encoding like EBCDIC to Unicode and back. When converting to Unicode, NEL would have to be replaced by LF, but when converting back it would be impossible to decide if a LF should be mapped to an EBCDIC LF or NEL. The approach taken in the Unicode standard allows this transformation to be information-preserving while still enabling applications to recognize all possible types of line terminators.
[edit] History
ASCII was developed simultaneously by the ISO and the ASA, the predecessor organization to ANSI. During the period of 1963–1968, the ISO draft standards supported the use of either CR+LF or LF alone as a newline, while the ASA drafts supported only CR+LF. The Multics operating system began development in 1964 and used LF alone as its newline. Unix followed the Multics practice, and later systems followed Unix.
The sequence CR+LF was in common use on many early computer systems that had adapted teletype machines, typically an ASR33, as a console device, because this sequence was required to position those printers at the start of a new line. On these systems text was often routinely composed to be compatible with these printers, since the concept of device drivers hiding such hardware details from the application was not yet well developed; applications had to talk directly to the teletype machine and follow its conventions. The separation of the two functions concealed the fact that the print head could not return from the far right to the beginning of the next line in one-character time. That is why the sequence was always sent with the CR first. In fact, it was often necessary to send CR+LF+NUL (ending with the control character indicating "do nothing") or CR+CR+LF (sending CR twice) to be sure that the print head had stopped bouncing. Once these mechanical systems were replaced, the two-character sequence had no functional significance, but it has persisted in some systems anyway.
It has been speculated that QDOS (which Microsoft purchased and renamed MS-DOS) adopted CR+LF to copy the implementation used by CP/M. Further speculation indicates that CP/M chose CR+LF to introduce a deliberate incompatibility with Unix to mitigate a possible lawsuit by AT&T/Bell over violating their Unix copyrights as CP/M was (according to this theory) loosely modeled on UNIX. Others believe this to be unlikely, arguing that CP/M resembles DEC operating systems such as RT-11 more closely than UNIX, and that it was originally designed with teletype rather than CRT operation in mind. Whatever its source, this convention was inherited by Microsoft's later Windows operating system.
[edit] Newline in programming languages
To facilitate creation of portable programs, programming languages provide some abstractions to deal with the different types of newline sequences used in different environments.
The C programming language provides the escape sequence '\n' (newline) and '\r' (carriage return). However, contrary to popular belief, these are in fact not generally equivalent to the ASCII LF and CR control characters. The C standard only guarantees two things:
- Each of these escape sequences maps to a unique implementation-defined number that can be stored in a single char value.
- When writing a file in text mode, '\n' is transparently translated to the native newline sequence used by the system, which may be longer than one character (Note: A C implementation is allowed to not even store the newline characters at all. For example, a system that stores the lines of a text file as rows of a SQL table or one that prefixes each line with the length of the line). When reading in text mode, the native newline sequence is translated back to '\n'. In binary mode, the second mode of I/O supported by the C library, no translation is performed, and the internal representation of any escape sequence is output directly.
On Unix platforms, where C originated, the native newline sequence is ASCII LF (0x0A), so '\n' was simply defined to be that value. With the internal and external representation being identical, the translation performed in text mode effectively turns into a no-op, making text mode and binary mode behave the same. This has caused many programmers who developed their software on Unix systems to simply ignore the distinction completely, resulting in code that is not portable to different platforms.
Another common problem is the use of '\n' when communicating using an Internet protocol that mandates the use of ASCII CR+LF for ending lines. Writing '\n' to a text mode stream works correctly on Windows systems, but produces only LF on Unix, and something completely different on more exotic systems. Using "\r\n" in binary mode is slightly better, as it works on many ASCII-compatible systems, but still fails miserably in the general case. One approach is to use binary mode while directly specifying the desired numerical values of the ASCII control sequence, "\x0D\x0A".
Java also provides '\n' and '\r' escape sequences. In contrast to C, these are guaranteed to represent the values 0x0A and 0x0D, respectively. The Java I/O libraries do not transparently translate these into platform dependent newline sequences on input or output. Instead, they provide functions for writing a full line that automatically add the native newline sequence, and functions for reading lines that accept any of CR, LF or CR+LF as a line terminator (see BufferedReader.readLine()).
[edit] Common problems
The different newline conventions often cause text files that have been transferred between systems of different types to be displayed incorrectly. For example, files originating on Unix or Apple Macintosh systems may appear as a single long line on a Windows system. Conversely, when viewing a file from a Windows computer on a Unix system, the extra CR may be displayed as ^M at the end of each line or as a second line break.
The problem can be hard to spot if some programs handle the foreign newlines properly while others don't. For example, a compiler may fail with obscure syntax errors even though the source file looks correct when displayed on the console or in an editor. Modern text editors generally recognize all flavours of CR / LF newlines and allow the user to convert between the different standards. Web browsers are usually also capable of displaying text files of different types.
The File Transfer Protocol can automatically convert newlines in files being transferred between systems with different newline representations when the transfer is done in ASCII mode. However, transferring binary files in this mode usually has disastrous results: Any occurrence of the newline byte sequence—which does not have line terminator semantics in this context, but is just a normal sequence of bytes—will be translated to whatever newline representation the other system uses, effectively corrupting the file. FTP clients often employ some heuristics (for example inspection of filename extensions) to automatically select either binary or ASCII mode, but in the end it is up to the user to make sure his files are transferred in the correct mode. If there is any doubt as to the correct mode, binary mode should be used, as then no files will be altered by FTP, though they may display incorrectly.
[edit] Conversion utilities
Generally, using a text editor is the simplest and most convenient way of converting a text file between different newline formats; most modern editors can read and write files using at least the different ASCII CR/LF conventions. Unfortunately, the standard Windows editor Notepad is not one of them, though Wordpad is.
On Windows systems without a better editor, the old MS-DOS editor EDIT that still ships with modern Windows versions can be used to convert a Unix text file to DOS/Windows newlines. A simple way of doing this is by creating a shortcut to EDIT on the desktop (context menu / New / Shortcut / "edit" / Next / Finish), dragging the text file in question onto it, and then saving the file again (File / Save).
You can convert with editors relatively small files. For larger files on Windows NT/2000/XP you can use the following command:
TYPE unix_file | FIND "" /V > dos_file
On many Unix systems, the dos2unix and unix2dos commands can be used to translate between ASCII CR+LF (DOS/Windows) and LF (Unix) newlines. However, there are different versions of dos2unix around that vary slightly in their command syntax. The tr command is available on virtually every Unix-like system and can be used to perform arbitrary replacement operations on single characters. A DOS/Windows text file can be converted to Unix format by simply removing all ASCII CR characters with
tr -d '\r' < inputfile > outputfile
or, if the DOS/Windows text has only CR's, by converting CRs to LFs with
tr '\r' '\n' < inputfile > outputfile
In the other direction, a conversion from Unix to DOS format can be performed with sed:
sed -e 's/$/\r/' inputfile > outputfile
The Perl alternative would allow the conversion of UNIX to DOS and vice versa to be performed on other OSs and not just UNIX:
perl -p -e 's/\n/\r\n/' inputfile > outputfile # UNIX to DOS perl -p -e 's/\r\n/\n/' inputfile > outputfile # DOS to UNIX
On Unix systems the file utility provides a convenient way of identifying what type of line breaks a text file contains.
[edit] Code samples
Detecting and replacing new line characters in C:
/* for each line in the file referred to by FILE *fp: */
while ( fgets( buf, sizeof(buf), fp ) ) {
/* if there is a line separator it would be at the end */
char * last = strchr(buf, '\0');
/* look for LF & replace it */
if (last - 1 >= buf && *--last == '\n')
*last = '\0';
/* look for CR & replace it */
if (last - 1 >= buf && *--last == '\r')
*last = '\0';
/* now do something useful with newline-free string in buf */
}
[edit] External links
- The Unicode reference, see paragraph 5.8 in Chapter 5 of the Unicode 4.0 standard
- The End-of-Line Story
- The [NEL] Newline Character
- How to convert a group of mixed text files to the preferred line ending all at once using vim
