(flex.info)Multiple Input Buffers
11 Multiple Input Buffers
*************************
Some scanners (such as those which support "include" files) require
reading from several input streams. As 'flex' scanners do a large
amount of buffering, one cannot control where the next input will be
read from by simply writing a 'YY_INPUT()' which is sensitive to the
scanning context. 'YY_INPUT()' is only called when the scanner reaches
the end of its buffer, which may be a long time after scanning a
statement such as an 'include' statement which requires switching the
input source.
To negotiate these sorts of problems, 'flex' provides a mechanism for
creating and switching between multiple input buffers. An input buffer
is created by using:
-- Function: YY_BUFFER_STATE yy_create_buffer ( FILE *file, int size )
which takes a 'FILE' pointer and a size and creates a buffer
associated with the given file and large enough to hold 'size'
characters (when in doubt, use 'YY_BUF_SIZE' for the size). It returns
a 'YY_BUFFER_STATE' handle, which may then be passed to other routines
(see below). The 'YY_BUFFER_STATE' type is a pointer to an opaque
'struct yy_buffer_state' structure, so you may safely initialize
'YY_BUFFER_STATE' variables to '((YY_BUFFER_STATE) 0)' if you wish, and
also refer to the opaque structure in order to correctly declare input
buffers in source files other than that of your scanner. Note that the
'FILE' pointer in the call to 'yy_create_buffer' is only used as the
value of 'yyin' seen by 'YY_INPUT'. If you redefine 'YY_INPUT()' so it
no longer uses 'yyin', then you can safely pass a NULL 'FILE' pointer to
'yy_create_buffer'. You select a particular buffer to scan from using:
-- Function: void yy_switch_to_buffer ( YY_BUFFER_STATE new_buffer )
The above function switches the scanner's input buffer so subsequent
tokens will come from 'new_buffer'. Note that 'yy_switch_to_buffer()'
may be used by 'yywrap()' to set things up for continued scanning,
instead of opening a new file and pointing 'yyin' at it. If you are
looking for a stack of input buffers, then you want to use
'yypush_buffer_state()' instead of this function. Note also that
switching input sources via either 'yy_switch_to_buffer()' or 'yywrap()'
does _not_ change the start condition.
-- Function: void yy_delete_buffer ( YY_BUFFER_STATE buffer )
is used to reclaim the storage associated with a buffer. ('buffer'
can be NULL, in which case the routine does nothing.) You can also
clear the current contents of a buffer using:
-- Function: void yypush_buffer_state ( YY_BUFFER_STATE buffer )
This function pushes the new buffer state onto an internal stack.
The pushed state becomes the new current state. The stack is maintained
by flex and will grow as required. This function is intended to be used
instead of 'yy_switch_to_buffer', when you want to change states, but
preserve the current state for later use.
-- Function: void yypop_buffer_state ( )
This function removes the current state from the top of the stack,
and deletes it by calling 'yy_delete_buffer'. The next state on the
stack, if any, becomes the new current state.
-- Function: void yy_flush_buffer ( YY_BUFFER_STATE buffer )
This function discards the buffer's contents, so the next time the
scanner attempts to match a token from the buffer, it will first fill
the buffer anew using 'YY_INPUT()'.
-- Function: YY_BUFFER_STATE yy_new_buffer ( FILE *file, int size )
is an alias for 'yy_create_buffer()', provided for compatibility with
the C++ use of 'new' and 'delete' for creating and destroying dynamic
objects.
'YY_CURRENT_BUFFER' macro returns a 'YY_BUFFER_STATE' handle to the
current buffer. It should not be used as an lvalue.
Here are two examples of using these features for writing a scanner
which expands include files (the '<<EOF>>' feature is discussed below).
This first example uses yypush_buffer_state and yypop_buffer_state.
Flex maintains the stack internally.
/* the "incl" state is used for picking up the name
* of an include file
*/
%x incl
%%
include BEGIN(incl);
[a-z]+ ECHO;
[^a-z\n]*\n? ECHO;
<incl>[ \t]* /* eat the whitespace */
<incl>[^ \t\n]+ { /* got the include file name */
yyin = fopen( yytext, "r" );
if ( ! yyin )
error( ... );
yypush_buffer_state(yy_create_buffer( yyin, YY_BUF_SIZE ));
BEGIN(INITIAL);
}
<<EOF>> {
yypop_buffer_state();
if ( !YY_CURRENT_BUFFER )
{
yyterminate();
}
}
The second example, below, does the same thing as the previous
example did, but manages its own input buffer stack manually (instead of
letting flex do it).
/* the "incl" state is used for picking up the name
* of an include file
*/
%x incl
%{
#define MAX_INCLUDE_DEPTH 10
YY_BUFFER_STATE include_stack[MAX_INCLUDE_DEPTH];
int include_stack_ptr = 0;
%}
%%
include BEGIN(incl);
[a-z]+ ECHO;
[^a-z\n]*\n? ECHO;
<incl>[ \t]* /* eat the whitespace */
<incl>[^ \t\n]+ { /* got the include file name */
if ( include_stack_ptr >= MAX_INCLUDE_DEPTH )
{
fprintf( stderr, "Includes nested too deeply" );
exit( 1 );
}
include_stack[include_stack_ptr++] =
YY_CURRENT_BUFFER;
yyin = fopen( yytext, "r" );
if ( ! yyin )
error( ... );
yy_switch_to_buffer(
yy_create_buffer( yyin, YY_BUF_SIZE ) );
BEGIN(INITIAL);
}
<<EOF>> {
if ( --include_stack_ptr == 0 )
{
yyterminate();
}
else
{
yy_delete_buffer( YY_CURRENT_BUFFER );
yy_switch_to_buffer(
include_stack[include_stack_ptr] );
}
}
The following routines are available for setting up input buffers for
scanning in-memory strings instead of files. All of them create a new
input buffer for scanning the string, and return a corresponding
'YY_BUFFER_STATE' handle (which you should delete with
'yy_delete_buffer()' when done with it). They also switch to the new
buffer using 'yy_switch_to_buffer()', so the next call to 'yylex()' will
start scanning the string.
-- Function: YY_BUFFER_STATE yy_scan_string ( const char *str )
scans a NUL-terminated string.
-- Function: YY_BUFFER_STATE yy_scan_bytes ( const char *bytes, int len
)
scans 'len' bytes (including possibly 'NUL's) starting at location
'bytes'.
Note that both of these functions create and scan a _copy_ of the
string or bytes. (This may be desirable, since 'yylex()' modifies the
contents of the buffer it is scanning.) You can avoid the copy by
using:
-- Function: YY_BUFFER_STATE yy_scan_buffer (char *base, yy_size_t
size)
which scans in place the buffer starting at 'base', consisting of
'size' bytes, the last two bytes of which _must_ be
'YY_END_OF_BUFFER_CHAR' (ASCII NUL). These last two bytes are not
scanned; thus, scanning consists of 'base[0]' through
'base[size-2]', inclusive.
If you fail to set up 'base' in this manner (i.e., forget the final
two 'YY_END_OF_BUFFER_CHAR' bytes), then 'yy_scan_buffer()' returns a
NULL pointer instead of creating a new input buffer.
-- Data type: yy_size_t
is an integral type to which you can cast an integer expression
reflecting the size of the buffer.
automatically generated by info2www version 1.2.2.9