Exercise 1

Introduction

This exercise is designed to give you some experience parsing text into lexemes using Java's StreamTokenizer class. We have to be careful about terminology because StreamTokenizer is really a "lexemizer" rather than a "tokenizer" according to the text's use of those terms. The idea is to identify and print each lexeme in a source file.

I have chosen to have your program analyze Java programs, but once you have completed this assignment, you should be able to use the StreamTokenizer class to analyze source code for any programmng language.

Assignment

Write a Java application which does the following for each file specified on the java command line:

If the file name does not end with ".java" print an error message and ignore the file.
Use the file name to construct a BufferedReader, and use the BufferedReader to construct a StreamTokenizer. If either of these operations fails, print an error message, and ignore the file.
Modify the StreamTokenizer so that it will recognize the lexemes of the Java language.
The steps to do this include:
- Make '/' an ordinary character, and make the StreamTokenizer recognize both C and C++ style comments.
- Make '.' an ordinary character, and make the digits and underscore characters word characters.
- Make end of line significant to the StreamTokenizer. This has more to do with formatting the output of your program than with the syntax of the Java language, which does not attach any significance to line boundaries except when recognizing string literals.
For each statement in the source file, print the Statement number and the line number on which it occurs, followed by each lexeme that makes up the statement. We will use the heuristic of saying that each semicolon ends a statement. (How accurate is this?) For this assignment you are to separate the lexemes by single spaces. An optional extension to this project is to print the token type for each lexeme; in this case, you should print each lexeme on a separate line.

Sample Output

If your working directory contains a file named Sample.java and your application's main class is called PrintLexemes, the following could represent an interaction you could have with your program:

cs300> java PrintLexemes junk.java Sample.java tower.lsp

Unable to read junk.java

File: Sample.java
  Statement 1 (line 2)
     import java . io . * ;
  Statement 2 (line 4)
     public class Sample
     {
     public static void main ( String [ ] argv )
     {
     int count , nextToken ;
  Statement 3 (line 28)
     for ( int i = 0.0 ;
  Statement 4 (line 29)
     i < argv . length ;
  Statement 5 (line 29)
     i + + )
     {
     try
     {
     FileReader fr = new FileReader ( argv [ i ] ) ;
  Statement 6 (line 34)
     BufferedReader br = new BufferedReader ( fr ) ;
  Statement 7 (line 35)
     StreamTokenizer st = new StreamTokenizer ( br ) ;
  Statement 8 (line 36)
     count = 0.0 ;
  Statement 9 (line 37)
     nextToken = StreamTokenizer . TT_WORD ;
  Statement 10 (line 38)
     while ( StreamTokenizer . TT_EOF ! =
     ( nextToken = st . nextToken ( ) ) )
     {
     count + + ;
  Statement 11 (line 42)
     }
     System . out . println ( argv [ i ] + " has " + count +
     " lexemes." ) ;
  Statement 12 (line 45)
     }
     catch ( IOException ioe )
     {
     System . err . println ( ioe . getMessage ( ) ) ;
  Statement 13 (line 49)
     }
     }
     }
     }

tower.lsp is not a .java file.
cs300>

Here is the program Sample.java that was used as input to the program to generate the output above. It doesn't do what your program is supposed to do, but I used it for sample input because it should help you get started on your own Java application.

//  Sample.java

import java.io.*;

//  Class Sample
//  ------------------------------------------------------------------
/**
  *     Sample application to use for testing StreamTokenizer exercise
  *     for CS-300.
  *
  *     @author   C. Vickery
  *               999-99-9999
  *     @version  1.0 - February, 2000
  */
  public class Sample
  {
    //  Method main()
    //  --------------------------------------------------------------
    /**
      *   Uses the default settings for a StreamTokenizer to count the
      *   number of lexemes in a file.
      *
      *   @param argv List of files to be processed.
      *   @return     Void
      */
      public static void main( String[] argv )
      {
      int   count, nextToken;

        for ( int i=0; i < argv.length; i++ )
        {
          try
          {
            FileReader      fr = new FileReader( argv[i] );
            BufferedReader  br = new BufferedReader( fr );
            StreamTokenizer st = new StreamTokenizer( br );
            count = 0;
            nextToken = StreamTokenizer.TT_WORD; // Anything but EOF
            while ( StreamTokenizer.TT_EOF !=
                                      ( nextToken = st.nextToken() ) )
            {
              count++;
            }
            System.out.println( argv[i] + " has " + count +
                                                        " lexemes." );
          }
          catch ( IOException ioe )
          {
            System.err.println( ioe.getMessage() );
          }
        }
      }
  }

Developing and Submitting the Assignment

You may work on this assignment either individually or in groups of up to three people. You are free to draw on any resources you wish to do this assignment, including friends, relatives, and strangers. Most importantly, feel free to ask me for help if you need it, either in class (if you think your questions are possibly of general interest) or by email (typically for debugging questions). You will not actually receive any credit for doing the assignment, but you will be tested on the concepts involved in doing it on your first exam in the course.

Submit the assignment to me using the following procedure:

Create a zip file that contains your .java file (not your .class file(s)). You can do this using WinZip or the jar command that comes with the JDK. For example, you can type:
```
        jar cvf Exercise_1.zip PrintLexemes.java
```
This will create a zip file named Exercise_1.zip that contains a file named PrintLexemes.java.
Attach the zip file to an email message and send it to me (vickery@forbin.qc.edu).

If you work in a group, submit just one copy of the program to me, but be sure to put an @author tag for each memeber of the group in the comments at the beginning of the program. (Put each @author tag on a separate line containing just an optional asterisk followed by the @author tag and your name. Do not include your ID number or anything else on the same line as the @author tag. Instead, put your ID number on the line below the @author tag, as shown in Sample.java.

If you really want to do this project in C or C++ instead of Java (some transfer students are more comfortable with those languages than with Java), you may do so provided you use only standard library functions (plus any you write yourself). The program must run from either a Unix or DOS command line; it must not be dependent on any Integrated Development Environment or Windows for me to be able to compile and test it. I will use GNU development tools to compile your C/C++ code; check with me if you don't know how to do that yourself on forbin.