- separated out token definition from scanner
- cleaned up token and scanner implementation
- added comments to exported interfaces

R=r
OCL=25665
CL=25665