Rich Text Format (often abbreviated as RTF), to surprise of many, is quite complex text data format. During its long history RTF bought a lot of add-ons that disturb the process of getting "clean" text. Let's try to solve that...
A little theory
At first let’s look into the RTF file
An RTF file consists of unformatted text, control words and groups.
A control word is a specially formatted command that RTF uses to mark printer control codes and information that applications use to manage documents. A control word is made up of lowercase alphabetic characters between "a" and "z". Each control word begins with a backslash (\) and ends with one of the following:
- A space. In this case, the space is part of the control word.
- A numeric parameter that can be a positive or a negative number.
- Any character other than a letter or a digit.
Therefore, the character string \rtf1\ansi\ansicpg1251 easily can be divided into three control words: rtf with parameter 1 (the major format version), ansi (the current encoding) and ansicpg with parameter 1251 (the current code page number 1251).
A group consists of text and control words enclosed in braces ({}). Control words defined within a group affect only the text inside this group and all nested subgroups. In order to know which control words are active now we will use the control words stack. When reading the opening brace ({) we will add new array stack element and write the data from previous stack element to it. When reading the closing brace (}) – remove top stack element.
Also we need to mention that some control words may be turned off not by closing the group but adding parameter 0 to the control word. For example, strings This is {\b bold} text and This is \b bold \b0 text give us the same result This is bold text.
Now we can come to the conclusion that all characters in an RTF file that are not control words or braces are plain text.
How to get "clean" text?
Even if we already know how to distinguish plain text from the control words, we need to discuss the characters encoding question.
RTF is an 8-bit format. That would limit it to ASCII, but RTF can encode characters beyond ASCII by escape sequences. The character escapes are of two types: code page escapes and Unicode escapes.
In a code page escape, two hexadecimal digits following an apostrophe (\'hh) are used for denoting a character taken from a Windows code page. The current code page is specified by control word \ansicpg. For example, if/ansicpg1256 is present, the sequence \'c8 will encode the Arabic letter beh (ب).
If a Unicode escape is required, the control word \u is used, followed by a 16-bit signed decimal integer giving the Unicode codepoint number. For the benefit of programs without Unicode support, this must be followed by the nearest representation of this character in the specified code page. For example, \u1576? would give the Arabic letter beh, specifying that older programs which do not have Unicode support should render it as a question mark instead.
The control word \uc0 can be used to indicate that subsequent Unicode escape sequences within the current group do not specify a substitution character.
Let’s read!
Now we have enough theoretical knowledge to start reading our first .rtf files:
<?php
// Function that checks whether the data are the on-screen text.
// It works in the following way:
// an array arrfailAt stores the control words for the current state of the stack, which show that
// input data are something else than plain text.
// For example, there may be a description of font or color palette etc.
function rtf_isPlainText($s) {
$arrfailAt = array("*", "fonttbl", "colortbl", "datastore", "themedata");
for ($i = 0; $i < count($arrfailAt); $i++)
if (!empty($s[$arrfailAt[$i]])) return false;
return true;
}
function rtf2text($filename) {
// Read the data from the input file.
$text = file_get_contents($filename);
if (!strlen($text))
return "";
// Create empty stack array.
$document = "";
$stack = array();
$j = -1;
// Read the data character-by- character…
for ($i = 0, $len = strlen($text); $i < $len; $i++) {
$c = $text[$i];
// Depending on current character select the further actions.
switch ($c) {
// the most important key word backslash
case "\\":
// read next character
$nc = $text[$i + 1];
// If it is another backslash or nonbreaking space or hyphen,
// then the character is plain text and add it to the output stream.
if ($nc == '\\' && rtf_isPlainText($stack[$j])) $document .= '\\';
elseif ($nc == '~' && rtf_isPlainText($stack[$j])) $document .= ' ';
elseif ($nc == '_' && rtf_isPlainText($stack[$j])) $document .= '-';
// If it is an asterisk mark, add it to the stack.
elseif ($nc == '*') $stack[$j]["*"] = true;
// If it is a single quote, read next two characters that are the hexadecimal notation
// of a character we should add to the output stream.
elseif ($nc == "'") {
$hex = substr($text, $i + 2, 2);
if (rtf_isPlainText($stack[$j]))
$document .= html_entity_decode("&#".hexdec($hex).";");
//Shift the pointer.
$i += 2;
// Since, we’ve found the alphabetic character, the next characters are control word
// and, possibly, some digit parameter.
} elseif ($nc >= 'a' && $nc <= 'z' || $nc >= 'A' && $nc <= 'Z') {
$word = "";
$param = null;
// Start reading characters after the backslash.
for ($k = $i + 1, $m = 0; $k < strlen($text); $k++, $m++) {
$nc = $text[$k];
// If the current character is a letter and there were no digits before it,
// then we’re still reading the control word. If there were digits, we should stop
// since we reach the end of the control word.
if ($nc >= 'a' && $nc <= 'z' || $nc >= 'A' && $nc <= 'Z') {
if (empty($param))
$word .= $nc;
else
break;
// If it is a digit, store the parameter.
} elseif ($nc >= '0' && $nc <= '9')
$param .= $nc;
// Since minus sign may occur only before a digit parameter, check whether
// $param is empty. Otherwise, we reach the end of the control word.
elseif ($nc == '-') {
if (empty($param))
$param .= $nc;
else
break;
} else
break;
}
// Shift the pointer on the number of read characters.
$i += $m - 1;
// Start analyzing what we’ve read. We are interested mostly in control words.
$toText = "";
switch (strtolower($word)) {
// If the control word is "u", then its parameter is the decimal notation of the
// Unicode character that should be added to the output stream.
// We need to check whether the stack contains \ucN control word. If it does,
// we should remove the N characters from the output stream.
case "u":
$toText .= html_entity_decode("&#x".dechex($param).";");
$ucDelta = @$stack[$j]["uc"];
if ($ucDelta > 0)
$i += $ucDelta;
break;
// Select line feeds, spaces and tabs.
case "par": case "page": case "column": case "line": case "lbr":
$toText .= "\n";
break;
case "emspace": case "enspace": case "qmspace":
$toText .= " ";
break;
case "tab": $toText .= "\t"; break;
// Add current date and time instead of corresponding labels.
case "chdate": $toText .= date("m.d.Y"); break;
case "chdpl": $toText .= date("l, j F Y"); break;
case "chdpa": $toText .= date("D, j M Y"); break;
case "chtime": $toText .= date("H:i:s"); break;
// Replace some reserved characters to their html analogs.
case "emdash": $toText .= html_entity_decode("—"); break;
case "endash": $toText .= html_entity_decode("–"); break;
case "bullet": $toText .= html_entity_decode("•"); break;
case "lquote": $toText .= html_entity_decode("‘"); break;
case "rquote": $toText .= html_entity_decode("’"); break;
case "ldblquote": $toText .= html_entity_decode("«"); break;
case "rdblquote": $toText .= html_entity_decode("»"); break;
// Add all other to the control words stack. If a control word
// does not include parameters, set ¶m to true.
default:
$stack[$j][strtolower($word)] = empty($param) ? true : $param;
break;
}
// Add data to the output stream if required.
if (rtf_isPlainText($stack[$j]))
$document .= $toText;
}
$i++;
break;
// If we read the opening brace {, then new subgroup starts and we add
// new array stack element and write the data from previous stack element to it.
case "{":
array_push($stack, $stack[$j++]);
break;
// If we read the closing brace }, then we reach the end of subgroup and should remove
// the last stack element.
case "}":
array_pop($stack);
$j--;
break;
// Skip “trash”.
case '\0': case '\r': case '\f': case '\n': break;
// Add other data to the output stream if required.
default:
if (rtf_isPlainText($stack[$j]))
$document .= $c;
break;
}
}
// Return result.
return $document;
}
?>
Conclusion
This code will cope correctly with the majority of .rtf files. However, there are several ways to improve it. First, you can add the additional checks for non-textual data. Given code will cut off only the fonts, colors, theme design, binary data and everything that is marked as "do not read me if you cannot" (text marked with \ *). Second, you may add the further encoding and code page parsing to reflect the \ 'hh keywords more accurately.
0 comments:
Post a Comment