Diff for /gforth/doc/vmgen.texi between versions 1.6 and 1.11

version 1.6, 2002/08/07 22:19:44 version 1.11, 2002/08/14 09:00:22
Line 1 Line 1
   \input texinfo    @c -*-texinfo-*-
   @comment %**start of header
   @setfilename vmgen.info
 @include version.texi  @include version.texi
   @settitle Vmgen (Gforth @value{VERSION})
   @c @syncodeindex pg cp
   @comment %**end of header
   @copying
   This manual is for Vmgen
   (version @value{VERSION}, @value{UPDATED}),
   the virtual machine interpreter generator
   
   Copyright @copyright{} 2002 Free Software Foundation, Inc.
   
   @quotation
   Permission is granted to copy, distribute and/or modify this document
   under the terms of the GNU Free Documentation License, Version 1.1 or
   any later version published by the Free Software Foundation; with no
   Invariant Sections, with the Front-Cover texts being ``A GNU Manual,''
   and with the Back-Cover Texts as in (a) below.  A copy of the
   license is included in the section entitled ``GNU Free Documentation
   License.''
   
   (a) The FSF's Back-Cover Text is: ``You have freedom to copy and modify
   this GNU Manual, like GNU software.  Copies published by the Free
   Software Foundation raise funds for GNU development.''
   @end quotation
   @end copying
   
   @dircategory GNU programming tools
   @direntry
   * Vmgen: (vmgen).               Interpreter generator
   @end direntry
   
   @titlepage
   @title Vmgen
   @subtitle for Gforth version @value{VERSION}, @value{UPDATED}
   @author M. Anton Ertl (@email{anton@@mips.complang.tuwien.ac.at})
   @page
   @vskip 0pt plus 1filll
   @insertcopying
   @end titlepage
   
   @contents
   
   @ifnottex
   @node Top, Introduction, (dir), (dir)
   @top Vmgen
   
   @insertcopying
   @end ifnottex
   
   @menu
   * Introduction::                What can Vmgen do for you?
   * Why interpreters?::           Advantages and disadvantages
   * Concepts::                    VM interpreter background
   * Invoking Vmgen::              
   * Example::                     
   * Input File Format::           
   * Using the generated code::    
   * Changes::                     from earlier versions
   * Contact::                     Bug reporting etc.
   * Copying This Manual::         Manual License
   * Index::                       
   
   @detailmenu
    --- The Detailed Node Listing ---
   
   Concepts
   
   * Front end and VM interpreter::  Modularizing an interpretive system
   * Data handling::               Stacks, registers, immediate arguments
   * Dispatch::                    From one VM instruction to the next
   
   Example
   
   * Example overview::            
   * Using profiling to create superinstructions::  
   
   Input File Format
   
   * Input File Grammar::          
   * Simple instructions::         
   * Superinstructions::           
   * Register Machines::           How to define register VM instructions
   
   Simple instructions
   
   * C Code Macros::               Macros recognized by Vmgen
   * C Code restrictions::         Vmgen makes assumptions about C code
   
   Using the generated code
   
   * VM engine::                   Executing VM code
   * VM instruction table::        
   * VM code generation::          Creating VM code (in the front-end)
   * Peephole optimization::       Creating VM superinstructions
   * VM disassembler::             for debugging the front end
   * VM profiler::                 for finding worthwhile superinstructions
   
   Copying This Manual
   
   * GNU Free Documentation License::  License for copying this manual.
   
   @end detailmenu
   @end menu
   
 @c @ifnottex  @c @ifnottex
 This file documents vmgen (Gforth @value{VERSION}).  @c This file documents Vmgen (Gforth @value{VERSION}).
   
   @c ************************************************************
   @node Introduction, Why interpreters?, Top, Top
 @chapter Introduction  @chapter Introduction
   
 Vmgen is a tool for writing efficient interpreters.  It takes a simple  Vmgen is a tool for writing efficient interpreters.  It takes a simple
Line 12  it).  The run-time efficiency of the res Line 119  it).  The run-time efficiency of the res
 within a factor of 10 of machine code produced by an optimizing  within a factor of 10 of machine code produced by an optimizing
 compiler.  compiler.
   
 The interpreter design strategy supported by vmgen is to divide the  The interpreter design strategy supported by Vmgen is to divide the
 interpreter into two parts:  interpreter into two parts:
   
 @itemize @bullet  @itemize @bullet
Line 39  A @emph{virtual machine} (VM) represents Line 146  A @emph{virtual machine} (VM) represents
 machine code.  Control flow occurs through VM branch instructions, like  machine code.  Control flow occurs through VM branch instructions, like
 in a real machine.  in a real machine.
   
 In this setup, vmgen can generate most of the code dealing with virtual  In this setup, Vmgen can generate most of the code dealing with virtual
 machine instructions from a simple description of the virtual machine  machine instructions from a simple description of the virtual machine
 instructions (@pxref...), in particular:  instructions (@pxref{Input File Format}), in particular:
   
 @table @emph  @table @asis
   
 @item VM instruction execution  @item VM instruction execution
   
Line 60  source level. Line 167  source level.
   
 @item VM code profiling  @item VM code profiling
 Useful for optimizing the VM insterpreter with superinstructions  Useful for optimizing the VM insterpreter with superinstructions
 (@pxref...).  (@pxref{VM profiler}).
   
 @end table  @end table
   
 VMgen supports efficient interpreters though various optimizations, in  @noindent
   Vmgen supports efficient interpreters though various optimizations, in
 particular  particular
   
 @itemize  @itemize @bullet
   
 @item Threaded code  @item Threaded code
   
Line 81  Replicating VM (super)instructions for b Line 189  Replicating VM (super)instructions for b
   
 @end itemize  @end itemize
   
 As a result, vmgen-based interpreters are only about an order of  As a result, Vmgen-based interpreters are only about an order of
 magintude slower than native code from an optimizing C compiler on small  magnitude slower than native code from an optimizing C compiler on small
 benchmarks; on large benchmarks, which spend more time in the run-time  benchmarks; on large benchmarks, which spend more time in the run-time
 system, the slowdown is often less (e.g., the slowdown of a  system, the slowdown is often less (e.g., the slowdown of a
 Vmgen-generated JVM interpreter over the best JVM JIT compiler we  Vmgen-generated JVM interpreter over the best JVM JIT compiler we
Line 91  and all other interpreters we looked at Line 199  and all other interpreters we looked at
 interpreter).  interpreter).
   
 VMs are usually designed as stack machines (passing data between VM  VMs are usually designed as stack machines (passing data between VM
 instructions on a stack), and vmgen supports such designs especially  instructions on a stack), and Vmgen supports such designs especially
 well; however, you can also use vmgen for implementing a register VM and  well; however, you can also use Vmgen for implementing a register VM and
 still benefit from most of the advantages offered by vmgen.  still benefit from most of the advantages offered by Vmgen.
   
 There are many potential uses of the instruction descriptions that are  There are many potential uses of the instruction descriptions that are
 not implemented at the moment, but we are open for feature requests, and  not implemented at the moment, but we are open for feature requests, and
Line 101  we will implement new features if someon Line 209  we will implement new features if someon
 list above is not exhaustive.  list above is not exhaustive.
   
 @c *********************************************************************  @c *********************************************************************
   @node Why interpreters?, Concepts, Introduction, Top
 @chapter Why interpreters?  @chapter Why interpreters?
   
 Interpreters are a popular language implementation technique because  Interpreters are a popular language implementation technique because
 they combine all three of the following advantages:  they combine all three of the following advantages:
   
 @itemize  @itemize @bullet
   
 @item Ease of implementation  @item Ease of implementation
   
Line 129  Vmgen makes it even easier to implement Line 238  Vmgen makes it even easier to implement
 techniques for building efficient interpreters.  techniques for building efficient interpreters.
   
 @c ********************************************************************  @c ********************************************************************
   @node Concepts, Invoking Vmgen, Why interpreters?, Top
 @chapter Concepts  @chapter Concepts
   
   @menu
   * Front end and VM interpreter::  Modularizing an interpretive system
   * Data handling::               Stacks, registers, immediate arguments
   * Dispatch::                    From one VM instruction to the next
   @end menu
   
 @c --------------------------------------------------------------------  @c --------------------------------------------------------------------
 @section Front-end and virtual machine interpreter  @node Front end and VM interpreter, Data handling, Concepts, Concepts
   @section Front end and VM interpreter
   
 @cindex front-end  @cindex front-end
 Interpretive systems are typically divided into a @emph{front end} that  Interpretive systems are typically divided into a @emph{front end} that
Line 151  interpreter, except for VM branch instru Line 268  interpreter, except for VM branch instru
 structures.  The conceptual similarity to real machine code results in  structures.  The conceptual similarity to real machine code results in
 the name @emph{virtual machine}.  the name @emph{virtual machine}.
   
 In this framework, vmgen supports building the VM interpreter and any  In this framework, Vmgen supports building the VM interpreter and any
 other component dealing with VM instructions.  It does not have any  other component dealing with VM instructions.  It does not have any
 support for the front end, apart from VM code generation support.  The  support for the front end, apart from VM code generation support.  The
 front end can be implemented with classical compiler front-end  front end can be implemented with classical compiler front-end
Line 164  Vmgen currently has no special support f Line 281  Vmgen currently has no special support f
 information in the instruction descriptions can be helpful, and we are  information in the instruction descriptions can be helpful, and we are
 open for feature requests and suggestions.  open for feature requests and suggestions.
   
   @c --------------------------------------------------------------------
   @node Data handling, Dispatch, Front end and VM interpreter, Concepts
 @section Data handling  @section Data handling
   
 @cindex stack machine  @cindex stack machine
Line 176  significantly more complex to implement Line 295  significantly more complex to implement
 Vmgen has special support and optimizations for stack VMs, making their  Vmgen has special support and optimizations for stack VMs, making their
 implementation easy and efficient.  implementation easy and efficient.
   
 You can also implement a register VM with vmgen (@pxref{Register  You can also implement a register VM with Vmgen (@pxref{Register
 Machines}), and you will still profit from most vmgen features.  Machines}), and you will still profit from most Vmgen features.
   
 @cindex stack item size  @cindex stack item size
 @cindex size, stack items  @cindex size, stack items
Line 191  the data on the stack. Line 310  the data on the stack.
 @cindex immediate arguments  @cindex immediate arguments
 Another source of data is immediate arguments VM instructions (in the VM  Another source of data is immediate arguments VM instructions (in the VM
 instruction stream).  The VM instruction stream is handled similar to a  instruction stream).  The VM instruction stream is handled similar to a
 stack in vmgen.  stack in Vmgen.
   
 @cindex garbage collection  @cindex garbage collection
 @cindex reference counting  @cindex reference counting
Line 202  harder, but might be possible (contact u Line 321  harder, but might be possible (contact u
 @c reference counting might be possible by including counting code in   @c reference counting might be possible by including counting code in 
 @c the conversion macros.  @c the conversion macros.
   
   @c --------------------------------------------------------------------
   @node Dispatch,  , Data handling, Concepts
 @section Dispatch  @section Dispatch
   
 Understanding this section is probably not necessary for using vmgen,  Understanding this section is probably not necessary for using Vmgen,
 but it may help.  You may want to skip it now, and read it if you find statements about dispatch methods confusing.  but it may help.  You may want to skip it now, and read it if you find statements about dispatch methods confusing.
   
 After executing one VM instruction, the VM interpreter has to dispatch  After executing one VM instruction, the VM interpreter has to dispatch
 the next VM instruction (vmgen calls the dispatch routine @samp{NEXT}).  the next VM instruction (Vmgen calls the dispatch routine @samp{NEXT}).
 Vmgen supports two methods of dispatch:  Vmgen supports two methods of dispatch:
   
 @table  @table @asis
   
 @item switch dispatch  @item switch dispatch
 In this method the VM interpreter contains a giant @code{switch}  In this method the VM interpreter contains a giant @code{switch}
Line 229  Dispatch consists of loading this addres Line 350  Dispatch consists of loading this addres
 incrementing the VM instruction pointer.  Typically the threaded-code  incrementing the VM instruction pointer.  Typically the threaded-code
 dispatch code is appended directly to the code for executing the VM  dispatch code is appended directly to the code for executing the VM
 instruction.  Threaded code cannot be implemented in ANSI C, but it can  instruction.  Threaded code cannot be implemented in ANSI C, but it can
 be implemented using GNU C's labels-as-values extension (@pxref{labels  be implemented using GNU C's labels-as-values extension (@pxref{Labels
 as values}).  as Values, , Labels as Values, gcc.info, GNU C Manual}).
   
 @end table  @end table
   
 @c *************************************************************  @c *************************************************************
 @chapter Invoking vmgen  @node Invoking Vmgen, Example, Concepts, Top
   @chapter Invoking Vmgen
   
 The usual way to invoke vmgen is as follows:  The usual way to invoke Vmgen is as follows:
   
 @example  @example
 vmgen @var{infile}  vmgen @var{infile}
Line 251  current working directory) and replacing Line 373  current working directory) and replacing
 and @file{-peephole.i}.  E.g., @command{bison hack/foo.vmg} will create  and @file{-peephole.i}.  E.g., @command{bison hack/foo.vmg} will create
 @file{foo-vm.i} etc.  @file{foo-vm.i} etc.
   
 The command-line options supported by vmgen are  The command-line options supported by Vmgen are
   
 @table @option  @table @option
   
Line 271  Print version and exit Line 393  Print version and exit
 @c env vars GFORTHDIR GFORTHDATADIR  @c env vars GFORTHDIR GFORTHDATADIR
   
 @c ****************************************************************  @c ****************************************************************
   @node Example, Input File Format, Invoking Vmgen, Top
 @chapter Example  @chapter Example
   
   @menu
   * Example overview::            
   * Using profiling to create superinstructions::  
   @end menu
   
   @c --------------------------------------------------------------------
   @node Example overview, Using profiling to create superinstructions, Example, Example
 @section Example overview  @section Example overview
   
 There are two versions of the same example for using vmgen:  There are two versions of the same example for using Vmgen:
 @file{vmgen-ex} and @file{vmgen-ex2} (you can also see Gforth as  @file{vmgen-ex} and @file{vmgen-ex2} (you can also see Gforth as
 example, but it uses additional (undocumented) features, and also  example, but it uses additional (undocumented) features, and also
 differs in some other respects).  The example implements @emph{mini}, a  differs in some other respects).  The example implements @emph{mini}, a
Line 320  stat.awk           script for aggregatin Line 450  stat.awk           script for aggregatin
 seq2rule.awk       script for creating superinstructions  seq2rule.awk       script for creating superinstructions
 @end example  @end example
   
   @noindent
 You would typically change much in or replace the following files:  You would typically change much in or replace the following files:
   
 @example  @example
Line 342  check}.  You can run run mini programs l Line 473  check}.  You can run run mini programs l
   
 To learn about the options, type @samp{./mini -h}.  To learn about the options, type @samp{./mini -h}.
   
   @c --------------------------------------------------------------------
   @node Using profiling to create superinstructions,  , Example overview, Example
 @section Using profiling to create superinstructions  @section Using profiling to create superinstructions
   
 I have not added rules for this in the @file{Makefile} (there are many  I have not added rules for this in the @file{Makefile} (there are many
Line 392  preceed larger superinstructions. Line 525  preceed larger superinstructions.
 Now you can create a version of mini with superinstructions by just  Now you can create a version of mini with superinstructions by just
 saying @samp{make}  saying @samp{make}
   
   
 @c ***************************************************************  @c ***************************************************************
   @node Input File Format, Using the generated code, Example, Top
 @chapter Input File Format  @chapter Input File Format
   
 Vmgen takes as input a file containing specifications of virtual machine  Vmgen takes as input a file containing specifications of virtual machine
Line 400  instructions.  This file usually has a n Line 535  instructions.  This file usually has a n
   
 Most examples are taken from the example in @file{vmgen-ex}.  Most examples are taken from the example in @file{vmgen-ex}.
   
   @menu
   * Input File Grammar::          
   * Simple instructions::         
   * Superinstructions::           
   * Register Machines::           How to define register VM instructions
   @end menu
   
   @c --------------------------------------------------------------------
   @node Input File Grammar, Simple instructions, Input File Format, Input File Format
 @section Input File Grammar  @section Input File Grammar
   
 The grammar is in EBNF format, with @code{@var{a}|@var{b}} meaning  The grammar is in EBNF format, with @code{@var{a}|@var{b}} meaning
Line 411  spaces and especially newlines; it's not Line 555  spaces and especially newlines; it's not
 any sequence of spaces and tabs is equivalent to a single space.  any sequence of spaces and tabs is equivalent to a single space.
   
 @example  @example
 description: {instruction|comment|eval-escape}  description: @{instruction|comment|eval-escape@}
   
 instruction: simple-inst|superinst  instruction: simple-inst|superinst
   
 simple-inst: ident " (" stack-effect " )" newline c-code newline newline  simple-inst: ident " (" stack-effect " )" newline c-code newline newline
   
 stack-effect: {ident} " --" {ident}  stack-effect: @{ident@} " --" @{ident@}
   
 super-inst: ident " =" ident {ident}    super-inst: ident " =" ident @{ident@}  
   
 comment:      "\ "  text newline  comment:      "\ "  text newline
   
Line 431  Note that the @code{\}s in this grammar Line 575  Note that the @code{\}s in this grammar
 C-style encodings for non-printable characters.  C-style encodings for non-printable characters.
   
 The C code in @code{simple-inst} must not contain empty lines (because  The C code in @code{simple-inst} must not contain empty lines (because
 vmgen would mistake that as the end of the simple-inst.  The text in  Vmgen would mistake that as the end of the simple-inst.  The text in
 @code{comment} and @code{eval-escape} must not contain a newline.  @code{comment} and @code{eval-escape} must not contain a newline.
 @code{Ident} must conform to the usual conventions of C identifiers  @code{Ident} must conform to the usual conventions of C identifiers
 (otherwise the C compiler would choke on the vmgen output).  (otherwise the C compiler would choke on the Vmgen output).
   
 Vmgen understands a few extensions beyond the grammar given here, but  Vmgen understands a few extensions beyond the grammar given here, but
 these extensions are only useful for building Gforth.  You can find a  these extensions are only useful for building Gforth.  You can find a
 description of the format used for Gforth in @file{prim}.  description of the format used for Gforth in @file{prim}.
   
 @subsection  @subsection Eval escapes
 @c woanders?  @c woanders?
 The text in @code{eval-escape} is Forth code that is evaluated when  The text in @code{eval-escape} is Forth code that is evaluated when
 vmgen reads the line.  If you do not know (and do not want to learn)  Vmgen reads the line.  If you do not know (and do not want to learn)
 Forth, you can build the text according to the following grammar; these  Forth, you can build the text according to the following grammar; these
 rules are normally all Forth you need for using vmgen:  rules are normally all Forth you need for using Vmgen:
   
 @example  @example
 text: stack-decl|type-prefix-decl|stack-prefix-decl  text: stack-decl|type-prefix-decl|stack-prefix-decl
Line 472  stack-prefix ( stack "prefix" -- ) Line 616  stack-prefix ( stack "prefix" -- )
 @end example  @end example
   
   
   @c --------------------------------------------------------------------
   @node Simple instructions, Superinstructions, Input File Grammar, Input File Format
 @section Simple instructions  @section Simple instructions
   
 We will use the following simple VM instruction description as example:  We will use the following simple VM instruction description as example:
Line 489  just plain C code. Line 635  just plain C code.
 The stack effect specifies that @code{sub} pulls two integers from the  The stack effect specifies that @code{sub} pulls two integers from the
 data stack and puts them in the C variables @code{i1} and @code{i2} (with  data stack and puts them in the C variables @code{i1} and @code{i2} (with
 the rightmost item (@code{i2}) taken from the top of stack) and later  the rightmost item (@code{i2}) taken from the top of stack) and later
 pushes one integer (@code{i)) on the data stack (the rightmost item is  pushes one integer (@code{i}) on the data stack (the rightmost item is
 on the top afterwards).  on the top afterwards).
   
 How do we know the type and stack of the stack items?  Vmgen uses  How do we know the type and stack of the stack items?  Vmgen uses
Line 516  This line defines the stack @code{data-s Line 662  This line defines the stack @code{data-s
 pointer @code{sp}, and each item has the basic type @code{Cell}; other  pointer @code{sp}, and each item has the basic type @code{Cell}; other
 types have to fit into one or two @code{Cell}s (depending on whether the  types have to fit into one or two @code{Cell}s (depending on whether the
 type is @code{single} or @code{double} wide), and are converted from and  type is @code{single} or @code{double} wide), and are converted from and
 to Cells on accessing the @code{data-stack) with conversion macros  to Cells on accessing the @code{data-stack} with conversion macros
 (@pxref{Conversion macros}).  Stacks grow towards lower addresses in  (@pxref{VM engine}).  Stacks grow towards lower addresses in
 vmgen-erated interpreters.  Vmgen-erated interpreters.
   
 We can override the default stack of a stack item by using a stack  We can override the default stack of a stack item by using a stack
 prefix.  E.g., consider the following instruction:  prefix.  E.g., consider the following instruction:
Line 548  arguments can only appear to the left of Line 694  arguments can only appear to the left of
 If there are multiple instruction stream arguments, the leftmost is the  If there are multiple instruction stream arguments, the leftmost is the
 first one (just as the intuition suggests).  first one (just as the intuition suggests).
   
 @subsubsection C Code Macros  @menu
   * C Code Macros::               Macros recognized by Vmgen
   * C Code restrictions::         Vmgen makes assumptions about C code
   @end menu
   
   @c --------------------------------------------------------------------
   @node C Code Macros, C Code restrictions, Simple instructions, Simple instructions
   @subsection C Code Macros
   
 Vmgen recognizes the following strings in the C code part of simple  Vmgen recognizes the following strings in the C code part of simple
 instructions:  instructions:
Line 556  instructions: Line 709  instructions:
 @table @samp  @table @samp
   
 @item SET_IP  @item SET_IP
 As far as vmgen is concerned, a VM instruction containing this ends a VM  As far as Vmgen is concerned, a VM instruction containing this ends a VM
 basic block (used in profiling to delimit profiled sequences).  On the C  basic block (used in profiling to delimit profiled sequences).  On the C
 level, this also sets the instruction pointer.  level, this also sets the instruction pointer.
   
Line 571  middle of the C code, you need to use @s Line 724  middle of the C code, you need to use @s
 is a conditional VM branch:  is a conditional VM branch:
   
 @example  @example
 if (branch_condition) {  if (branch_condition) @{
   SET_IP(target); TAIL;    SET_IP(target); TAIL;
 }  @}
 /* implicit tail follows here */  /* implicit tail follows here */
 @end example  @end example
   
Line 589  is not yet implemented in the vmgen-ex c Line 742  is not yet implemented in the vmgen-ex c
 typical application is in conditional VM branches:  typical application is in conditional VM branches:
   
 @example  @example
 if (branch_condition) {  if (branch_condition) @{
   SET_IP(target); TAIL; /* now this TAIL is necessary */    SET_IP(target); TAIL; /* now this TAIL is necessary */
 }  @}
 SUPER_CONTINUE;  SUPER_CONTINUE;
 @end example  @end example
   
 @end table  @end table
   
 Note that vmgen is not smart about C-level tokenization, comments,  Note that Vmgen is not smart about C-level tokenization, comments,
 strings, or conditional compilation, so it will interpret even a  strings, or conditional compilation, so it will interpret even a
 commented-out SUPER_END as ending a basic block (or, e.g.,  commented-out SUPER_END as ending a basic block (or, e.g.,
 @samp{RETAIL;} as @samp{TAIL;}).  Conversely, vmgen requires the literal  @samp{RETAIL;} as @samp{TAIL;}).  Conversely, Vmgen requires the literal
 presence of these strings; vmgen will not see them if they are hiding in  presence of these strings; Vmgen will not see them if they are hiding in
 a C preprocessor macro.  a C preprocessor macro.
   
   
 @subsubsection C Code restrictions  @c --------------------------------------------------------------------
   @node C Code restrictions,  , C Code Macros, Simple instructions
   @subsection C Code restrictions
   
 Vmgen generates code and performs some optimizations under the  Vmgen generates code and performs some optimizations under the
 assumption that the user-supplied C code does not access the stack  assumption that the user-supplied C code does not access the stack
Line 617  the following. Line 772  the following.
 Accessing a stack or stack pointer directly can be a problem for several  Accessing a stack or stack pointer directly can be a problem for several
 reasons:   reasons: 
   
 @itemize  @itemize @bullet
   
 @item  @item
 You may cache the top-of-stack item in a local variable (that is  You may cache the top-of-stack item in a local variable (that is
Line 633  automatically by mentioning a special st Line 788  automatically by mentioning a special st
 @c sometimes flushing and/or reloading unnecessary  @c sometimes flushing and/or reloading unnecessary
   
 @item  @item
 The vmgen-erated code loads the stack items from stack-pointer-indexed  The Vmgen-erated code loads the stack items from stack-pointer-indexed
 memory into variables before the user-supplied C code, and stores them  memory into variables before the user-supplied C code, and stores them
 from variables to stack-pointer-indexed memory afterwards.  If you do  from variables to stack-pointer-indexed memory afterwards.  If you do
 any writes to the stack through its stack pointer in your C code, it  any writes to the stack through its stack pointer in your C code, it
Line 656  macros can be implemented in several way Line 811  macros can be implemented in several way
 contents.  contents.
   
   
   @c --------------------------------------------------------------------
   @node Superinstructions, Register Machines, Simple instructions, Input File Format
 @section Superinstructions  @section Superinstructions
   
   Note: don't invest too much work in (static) superinstructions; a future
   version of Vmgen will support dynamic superinstructions (see Ian
   Piumarta and Fabio Riccardi, @cite{Optimizing Direct Threaded Code by
   Selective Inlining}, PLDI'98), and static superinstructions have much
   less benefit in that context.
   
 Here is an example of a superinstruction definition:  Here is an example of a superinstruction definition:
   
 @example  @example
Line 668  lit_sub = lit sub Line 831  lit_sub = lit sub
 @code{sub} are its components.  This superinstruction performs the same  @code{sub} are its components.  This superinstruction performs the same
 action as the sequence @code{lit} and @code{sub}.  It is generated  action as the sequence @code{lit} and @code{sub}.  It is generated
 automatically by the VM code generation functions whenever that sequence  automatically by the VM code generation functions whenever that sequence
 occurs, so you only need to add this definition if you want to use this  occurs, so if you want to use this superinstruction, you just need to
 superinstruction (and even that can be partially automatized,  add this definition (and even that can be partially automatized,
 @pxref{...}).  @pxref{VM profiler}).
   
 Vmgen requires that the component instructions are simple instructions  Vmgen requires that the component instructions are simple instructions
 defined before superinstructions using the components.  Currently, vmgen  defined before superinstructions using the components.  Currently, Vmgen
 also requires that all the subsequences at the start of a  also requires that all the subsequences at the start of a
 superinstruction (prefixes) must be defined as superinstruction before  superinstruction (prefixes) must be defined as superinstruction before
 the superinstruction.  I.e., if you want to define a superinstruction  the superinstruction.  I.e., if you want to define a superinstruction
Line 695  sumof4 = add add add Line 858  sumof4 = add add add
 Here, @code{sumof4} is the longest prefix of @code{sumof5}, and @code{sumof3}  Here, @code{sumof4} is the longest prefix of @code{sumof5}, and @code{sumof3}
 is the longest prefix of @code{sumof4}.  is the longest prefix of @code{sumof4}.
   
 Note that vmgen assumes that only the code it generates accesses stack  Note that Vmgen assumes that only the code it generates accesses stack
 pointers, the instruction pointer, and various stack items, and it  pointers, the instruction pointer, and various stack items, and it
 performs optimizations based on this assumption.  Therefore, VM  performs optimizations based on this assumption.  Therefore, VM
 instructions that change the instruction pointer should only be used as  instructions that change the instruction pointer should only be used as
Line 703  last component; a VM instruction that ac Line 866  last component; a VM instruction that ac
 not be used as component at all.  Vmgen does not check these  not be used as component at all.  Vmgen does not check these
 restrictions, they just result in bugs in your interpreter.  restrictions, they just result in bugs in your interpreter.
   
   @node Register Machines,  , Superinstructions, Input File Format
   @section Register Machines
   
   If you want to implement a register VM rather than a stack VM with
   Vmgen, there are two ways to do it: Directly and through
   superinstructions.
   
   If you use the direct way, you define instructions that take the
   register numbers as immediate arguments, like this:
   
   @example
   add3 ( #src1 #src2 #dest -- )
   reg[dest] = reg[src1]+reg[src2];
   @end example
   
   If you use superinstructions to define a register VM, you define simple
   instructions that use a stack, and then define superinstructions that
   have no overall stack effect, like this:
   
   @example
   loadreg ( #src -- n )
   n = reg[src];
   
   storereg ( n #dest -- )
   reg[dest] = n;
   
   adds ( n1 n2 -- n )
   n = n1+n2;
   
   add3 = loadreg loadreg adds storereg
   @end example
   
   An advantage of this method is that you see the values and not just the
   register numbers in tracing (actually, with an appropriate definition of
   @code{printarg_src} (@pxref{VM engine}), you can print the values of the
   source registers on entry, but you cannot print the value of the
   destination register on exit.  A disadvantage of this method is that
   currently you cannot generate superinstructions directly, but only
   through generating a sequence of simple instructions (we might change
   this in the future if there is demand).
   
   Could the register VM support be improved, apart from the issues
   mentioned above?  It is hard to see how to do it in a general way,
   because there are a number of different designs that different people
   mean when they use the term @emph{register machine} in connection with
   VM interpreters.  However, if you have ideas or requests in that
   direction, please let me know (@pxref{Contact}).
   
 @c ********************************************************************  @c ********************************************************************
   @node Using the generated code, Changes, Input File Format, Top
 @chapter Using the generated code  @chapter Using the generated code
   
 The easiest way to create a working VM interpreter with vmgen is  The easiest way to create a working VM interpreter with Vmgen is
 probably to start with one of the examples, and modify it for your  probably to start with one of the examples, and modify it for your
 purposes.  This chapter is just the reference manual for the macros  purposes.  This chapter is just the reference manual for the macros
 etc. used by the generated code, and the other context expected by the  etc. used by the generated code, the other context expected by the
 generated code, and what you can do with the various generated files.  generated code, and what you can do with the various generated files.
   
   @menu
   * VM engine::                   Executing VM code
   * VM instruction table::        
   * VM code generation::          Creating VM code (in the front-end)
   * Peephole optimization::       Creating VM superinstructions
   * VM disassembler::             for debugging the front end
   * VM profiler::                 for finding worthwhile superinstructions
   @end menu
   
   @c --------------------------------------------------------------------
   @node VM engine, VM instruction table, Using the generated code, Using the generated code
 @section VM engine  @section VM engine
   
 The VM engine is the VM interpreter that executes the VM code.  It is  The VM engine is the VM interpreter that executes the VM code.  It is
Line 737  The following macros and variables are u Line 959  The following macros and variables are u
   
 @item LABEL(@var{inst_name})  @item LABEL(@var{inst_name})
 This is used just before each VM instruction to provide a jump or  This is used just before each VM instruction to provide a jump or
 @code{switch} label (the @samp{:} is provided by vmgen).  For switch  @code{switch} label (the @samp{:} is provided by Vmgen).  For switch
 dispatch this should expand to @samp{case @var{label}}; for  dispatch this should expand to @samp{case @var{label}}; for
 threaded-code dispatch this should just expand to @samp{case  threaded-code dispatch this should just expand to @samp{case
 @var{label}}.  In either case @var{label} is usually the @var{inst_name}  @var{label}}.  In either case @var{label} is usually the @var{inst_name}
 with some prefix or suffix to avoid naming conflicts.  with some prefix or suffix to avoid naming conflicts.
   
   @item LABEL2(@var{inst_name})
   This will be used for dynamic superinstructions; at the moment, this
   should expand to nothing.
   
 @item NAME(@var{inst_name_string})  @item NAME(@var{inst_name_string})
 Called on entering a VM instruction with a string containing the name of  Called on entering a VM instruction with a string containing the name of
 the VM instruction as parameter.  In normal execution this should be a  the VM instruction as parameter.  In normal execution this should be a
Line 770  macros do nothing.  Then also related ma Line 996  macros do nothing.  Then also related ma
 straightforward to define.  For switch dispatch this code consists just  straightforward to define.  For switch dispatch this code consists just
 of a jump to the dispatch code (@samp{goto next_inst;} in our example;  of a jump to the dispatch code (@samp{goto next_inst;} in our example;
 for direct threaded code it consists of something like  for direct threaded code it consists of something like
 @samp{({cfa=*ip++; goto *cfa;})}.  @samp{(@{cfa=*ip++; goto *cfa;@})}.
   
 Pulling code (usually the @samp{cfa=*ip;}) up into @samp{NEXT_P1}  Pulling code (usually the @samp{cfa=*ip;}) up into @samp{NEXT_P1}
 usually does not cause problems, but pulling things up into  usually does not cause problems, but pulling things up into
Line 781  extreme variant is to pull code up even Line 1007  extreme variant is to pull code up even
 the previous VM instruction (prefetching, useful on PowerPCs).  the previous VM instruction (prefetching, useful on PowerPCs).
   
 @item INC_IP(@var{n})  @item INC_IP(@var{n})
 This increments IP by @var{n}.  This increments @code{IP} by @var{n}.
   
   @item SET_IP(@var{target})
   This sets @code{IP} to @var{target}.
   
 @item vm_@var{A}2@var{B}(a,b)  @item vm_@var{A}2@var{B}(a,b)
 Type casting macro that assigns @samp{a} (of type @var{A}) to @samp{b}  Type casting macro that assigns @samp{a} (of type @var{A}) to @samp{b}
Line 831  Macro for executing @var{expr}, if top-o Line 1060  Macro for executing @var{expr}, if top-o
 top-of-stack caching for @var{stackpointer}; otherwise it should do  top-of-stack caching for @var{stackpointer}; otherwise it should do
 nothing.  nothing.
   
   @item SUPER_END
   This is used by the VM profiler (@pxref{VM profiler}); it should not do
   anything in normal operation, and call @code{vm_count_block(IP)} for
   profiling.
   
   @item SUPER_CONTINUE
   This is just a hint to Vmgen and does nothing at the C level.
   
 @item VM_DEBUG  @item VM_DEBUG
 If this is defined, the tracing code will be compiled in (slower  If this is defined, the tracing code will be compiled in (slower
 interpretation, but better debugging).  Our example compiles two  interpretation, but better debugging).  Our example compiles two
Line 857  basic type of the stack. Line 1094  basic type of the stack.
 @end table  @end table
   
   
 @section{VM instruction table}  @c --------------------------------------------------------------------
   @node VM instruction table, VM code generation, VM engine, Using the generated code
   @section VM instruction table
   
 For threaded code we also need to produce a table containing the labels  For threaded code we also need to produce a table containing the labels
 of all VM instructions.  This is needed for VM code generation  of all VM instructions.  This is needed for VM code generation
Line 882  For switch dispatch, we also need to def Line 1121  For switch dispatch, we also need to def
 used as case labels in an @code{enum}.  used as case labels in an @code{enum}.
   
 For both purposes (VM instruction table, and enum), the file  For both purposes (VM instruction table, and enum), the file
 @file{@var{name}-labels.i} is generated by vmgen.  You have to define  @file{@var{name}-labels.i} is generated by Vmgen.  You have to define
 the following macro used in this file:  the following macro used in this file:
   
 @table @samp  @table @samp
Line 892  For switch dispatch, this is just the na Line 1131  For switch dispatch, this is just the na
 name as used in @samp{LABEL(@var{inst_name})}), for both uses of  name as used in @samp{LABEL(@var{inst_name})}), for both uses of
 @file{@var{name}-labels.i}.  For threaded-code dispatch, this is the  @file{@var{name}-labels.i}.  For threaded-code dispatch, this is the
 address of the label defined in @samp{LABEL(@var{inst_name})}); the  address of the label defined in @samp{LABEL(@var{inst_name})}); the
 address is taken with @samp{&&} (@pxref{labels-as-values}).  address is taken with @samp{&&} (@pxref{Labels as Values, , Labels as
   Values, gcc.info, GNU C Manual}).
   
 @end table  @end table
   
   
   @c --------------------------------------------------------------------
   @node VM code generation, Peephole optimization, VM instruction table, Using the generated code
 @section VM code generation  @section VM code generation
   
 Vmgen generates VM code generation functions in @file{@var{name}-gen.i}  Vmgen generates VM code generation functions in @file{@var{name}-gen.i}
 that the front end can call to generate VM code.  This is essential for  that the front end can call to generate VM code.  This is essential for
 an interpretive system.  an interpretive system.
   
 For a VM instruction @samp{x ( #a b #c -- d )}, vmgen generates a  For a VM instruction @samp{x ( #a b #c -- d )}, Vmgen generates a
 function with the prototype  function with the prototype
   
 @example  @example
Line 951  every type that you use as immediate arg Line 1193  every type that you use as immediate arg
 In addition to using these functions to generate code, you should call  In addition to using these functions to generate code, you should call
 @code{BB_BOUNDARY} at every basic block entry point if you ever want to  @code{BB_BOUNDARY} at every basic block entry point if you ever want to
 use superinstructions (or if you want to use the profiling supported by  use superinstructions (or if you want to use the profiling supported by
 vmgen; however, this is mainly useful for selecting superinstructions).  Vmgen; however, this is mainly useful for selecting superinstructions).
 If you use @code{BB_BOUNDARY}, you should also define it (take a look at  If you use @code{BB_BOUNDARY}, you should also define it (take a look at
 its definition in @file{vmgen-ex/mini.y}).  its definition in @file{vmgen-ex/mini.y}).
   
Line 962  superinstruction at the branch), and bec Line 1204  superinstruction at the branch), and bec
 themselves to the profiler.  themselves to the profiler.
   
   
   @c --------------------------------------------------------------------
   @node Peephole optimization, VM disassembler, VM code generation, Using the generated code
 @section Peephole optimization  @section Peephole optimization
   
 You need peephole optimization only if you want to use  You need peephole optimization only if you want to use
Line 990  instruction to branch to), you have to c Line 1234  instruction to branch to), you have to c
 (@pxref{VM code generation}) at branch targets.  (@pxref{VM code generation}) at branch targets.
   
   
   @c --------------------------------------------------------------------
   @node VM disassembler, VM profiler, Peephole optimization, Using the generated code
 @section VM disassembler  @section VM disassembler
   
 A VM code disassembler is optional for an interpretive system, but  A VM code disassembler is optional for an interpretive system, but
Line 1023  VM instruction table. Line 1269  VM instruction table.
 @end table  @end table
   
   
   @c --------------------------------------------------------------------
   @node VM profiler,  , VM disassembler, Using the generated code
   @section VM profiler
   
   The VM profiler is designed for getting execution and occurence counts
   for VM instruction sequences, and these counts can then be used for
   selecting sequences as superinstructions.  The VM profiler is probably
   not useful as profiling tool for the interpretive system.  I.e., the VM
   profiler is useful for the developers, but not the users of the
   interpretive system.
   
   The output of the profiler is: for each basic block (executed at least
   once), it produces the dynamic execution count of that basic block and
   all its subsequences; e.g.,
   
   @example
          9227465  lit storelocal 
          9227465  storelocal branch 
          9227465  lit storelocal branch 
   @end example
   
   I.e., a basic block consisting of @samp{lit storelocal branch} is
   executed 9227465 times.
   
   This output can be combined in various ways.  E.g.,
   @file{vmgen/stat.awk} adds up the occurences of a given sequence wrt
   dynamic execution, static occurence, and per-program occurence.  E.g.,
   
   @example
         2      16        36910041 loadlocal lit 
   @end example
   
   indicates that the sequence @samp{loadlocal lit} occurs in 2 programs,
   in 16 places, and has been executed 36910041 times.  Now you can select
   superinstructions in any way you like (note that compile time and space
   typically limit the number of superinstructions to 100--1000).  After
   you have done that, @file{vmgen/seq2rule.awk} turns lines of the form
   above into rules for inclusion in a Vmgen input file.  Note that this
   script does not ensure that all prefixes are defined, so you have to do
   that in other ways.  So, an overall script for turning profiles into
   superinstructions can look like this:
   
   @example
   awk -f stat.awk fib.prof test.prof|
   awk '$3>=10000'|                #select sequences
   fgrep -v -f peephole-blacklist| #eliminate wrong instructions
   awk -f seq2rule.awk|            #turn into superinstructions
   sort -k 3 >mini-super.vmg       #sort sequences
   @end example
   
   Here the dynamic count is used for selecting sequences (preliminary
   results indicate that the static count gives better results, though);
   the third line eliminats sequences containing instructions that must not
   occur in a superinstruction, because they access a stack directly.  The
   dynamic count selection ensures that all subsequences (including
   prefixes) of longer sequences occur (because subsequences have at least
   the same count as the longer sequences); the sort in the last line
   ensures that longer superinstructions occur after their prefixes.
   
   But before using it, you have to have the profiler.  Vmgen supports its
   creation by generating @file{@var{file}-profile.i}; you also need the
   wrapper file @file{vmgen-ex/profile.c} that you can use almost verbatim.
   
   The profiler works by recording the targets of all VM control flow
   changes (through @code{SUPER_END} during execution, and through
   @code{BB_BOUNDARY} in the front end), and counting (through
   @code{SUPER_END}) how often they were targeted.  After the program run,
   the numbers are corrected such that each VM basic block has the correct
   count (originally entering a block without executing a branch does not
   increase the count), then the subsequences of all basic blocks are
   printed.  To get all this, you just have to define @code{SUPER_END} (and
   @code{BB_BOUNDARY}) appropriately, and call @code{vm_print_profile(FILE
   *file)} when you want to output the profile on @code{file}.
   
   The @file{@var{file}-profile.i} is simular to the disassembler file, and
   it uses variables and functions defined in @file{vmgen-ex/profile.c},
   plus @code{VM_IS_INST} already defined for the VM disassembler
   (@pxref{VM disassembler}).
   
   
   @c **********************************************************
   @node Changes, Contact, Using the generated code, Top
   @chapter Changes
   
   Users of the gforth-0.5.9-20010501 version of Vmgen need to change
   several things in their source code to use the current version.  I
   recommend keeping the gforth-0.5.9-20010501 version until you have
   completed the change (note that you can have several versions of Gforth
   installed at the same time).  I hope to avoid such incompatible changes
   in the future.
   
   The required changes are:
   
   @table @code
   
   @item vm_@var{A}2@var{B}
   now takes two arguments.
   
   @item vm_two@var{A}2@var{B}(b,a1,a2);
   changed to vm_two@var{A}2@var{B}(a1,a2,b) (note the absence of the @samp{;}).
   
   @end table
   
   Also some new macros have to be defined, e.g., @code{INST_ADDR}, and
   @code{LABEL}; some macros have to be defined in new contexts, e.g.,
   @code{VM_IS_INST} is now also needed in the disassembler.
   
   @node Contact, Copying This Manual, Changes, Top
   @chapter Contact
   
   @node Copying This Manual, Index, Contact, Top
   @appendix Copying This Manual
   
 Invocation  @menu
   * GNU Free Documentation License::  License for copying this manual.
   @end menu
   
 Input Syntax  @include fdl.texi
   
 Concepts: Front end, VM, Stacks,  Types, input stream  
   
 Contact  @node Index,  , Copying This Manual, Top
   @unnumbered Index
   
   @printindex cp
   
 Required changes:  @bye
 vm_...2... -> two arguments  
 "vm_two...2...(arg1,arg2,arg3);" -> "vm_two...2...(arg3,arg1,arg2)" (no ";").  
 define INST_ADDR and LABEL  
 define VM_IS_INST also for disassembler  

Removed from v.1.6  
changed lines
  Added in v.1.11


FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>