KiCad PCB EDA Suite
DSNLEXER Class Reference

Implement a lexical analyzer for the SPECCTRA DSN file format. More...

#include <dsnlexer.h>

Public Member Functions

 DSNLEXER (const KEYWORD *aKeywordTable, unsigned aKeywordCount, FILE *aFile, const wxString &aFileName)
 Initialize a DSN lexer and prepares to read from aFile which is already open and has aFilename. More...
 
 DSNLEXER (const KEYWORD *aKeywordTable, unsigned aKeywordCount, const std::string &aSExpression, const wxString &aSource=wxEmptyString)
 Initialize a DSN lexer and prepares to read from aSExpression. More...
 
 DSNLEXER (const std::string &aSExpression, const wxString &aSource=wxEmptyString)
 Initialize a DSN lexer and prepares to read from aSExpression. More...
 
 DSNLEXER (const KEYWORD *aKeywordTable, unsigned aKeywordCount, LINE_READER *aLineReader=nullptr)
 Initialize a DSN lexer and prepares to read from aLineReader which is already open, and may be in use by other DSNLEXERs also. More...
 
virtual ~DSNLEXER ()
 
void InitParserState ()
 Reinit variables used during parsing, to ensure od states are not used in a new parsing must be called before parsing a new file after parsing an old file to avoid starting with some variables in a non initial state. More...
 
bool SyncLineReaderWith (DSNLEXER &aLexer)
 Usable only for DSN lexers which share the same LINE_READER. More...
 
void SetSpecctraMode (bool aMode)
 Change the behavior of this lexer into or out of "specctra mode". More...
 
void PushReader (LINE_READER *aLineReader)
 Manage a stack of LINE_READERs in order to handle nested file inclusion. More...
 
LINE_READERPopReader ()
 Delete the top most LINE_READER from an internal stack of LINE_READERs and in the case of FILE_LINE_READER this means the associated FILE is closed. More...
 
int NextTok ()
 Return the next token found in the input file or DSN_EOF when reaching the end of file. More...
 
int NeedSYMBOL ()
 Call NextTok() and then verifies that the token read in satisfies IsSymbol(). More...
 
int NeedSYMBOLorNUMBER ()
 Call NextTok() and then verifies that the token read in satisfies bool IsSymbol() or the next token is DSN_NUMBER. More...
 
int NeedNUMBER (const char *aExpectation)
 Call NextTok() and then verifies that the token read is type DSN_NUMBER. More...
 
int CurTok () const
 Return whatever NextTok() returned the last time it was called. More...
 
int PrevTok () const
 Return whatever NextTok() returned the 2nd to last time it was called. More...
 
int GetCurStrAsToken () const
 Used to support "loose" matches (quoted tokens). More...
 
char SetStringDelimiter (char aStringDelimiter)
 Change the string delimiter from the default " to some other character and return the old value. More...
 
bool SetSpaceInQuotedTokens (bool val)
 Change the setting controlling whether a space in a quoted string isa terminator. More...
 
bool SetCommentsAreTokens (bool val)
 Change the handling of comments. More...
 
wxArrayString * ReadCommentLines ()
 Check the next sequence of tokens and reads them into a wxArrayString if they are comments. More...
 
void Expecting (int aTok) const
 Throw an IO_ERROR exception with an input file specific error message. More...
 
void Expecting (const char *aTokenList) const
 Throw an IO_ERROR exception with an input file specific error message. More...
 
void Unexpected (int aTok) const
 Throw an IO_ERROR exception with an input file specific error message. More...
 
void Unexpected (const char *aToken) const
 Throw an IO_ERROR exception with an input file specific error message. More...
 
void Duplicate (int aTok)
 Throw an IO_ERROR exception with a message saying specifically that aTok is a duplicate of one already seen in current context. More...
 
void NeedLEFT ()
 Call NextTok() and then verifies that the token read in is a DSN_LEFT. More...
 
void NeedRIGHT ()
 Call NextTok() and then verifies that the token read in is a DSN_RIGHT. More...
 
const char * GetTokenText (int aTok) const
 Return the C string representation of a DSN_T value. More...
 
wxString GetTokenString (int aTok) const
 Return a quote wrapped wxString representation of a token value. More...
 
const char * CurText () const
 Return a pointer to the current token's text. More...
 
const std::string & CurStr () const
 Return a reference to current token in std::string form. More...
 
wxString FromUTF8 () const
 Return the current token text as a wxString, assuming that the input byte stream is UTF8 encoded. More...
 
int CurLineNumber () const
 Return the current line number within my LINE_READER. More...
 
const char * CurLine () const
 Return the current line of text from which the CurText() would return its token. More...
 
const wxString & CurSource () const
 Return the current LINE_READER source. More...
 
int CurOffset () const
 Return the byte offset within the current line, using a 1 based index. More...
 

Static Public Member Functions

static bool IsSymbol (int aTok)
 Test a token to see if it is a symbol. More...
 
static const char * Syntax (int aTok)
 

Protected Types

typedef std::vector< LINE_READER * > READER_STACK
 

Protected Member Functions

void init ()
 
int readLine ()
 
int findToken (const std::string &aToken) const
 Take aToken string and looks up the string in the keywords table. More...
 
bool isStringTerminator (char cc) const
 

Protected Attributes

bool iOwnReaders
 on readerStack, should I delete them? More...
 
const char * start
 
const char * next
 
const char * limit
 
char dummy [1]
 when there is no reader. More...
 
READER_STACK readerStack
 all the LINE_READERs by pointer. More...
 
LINE_READERreader
 
bool specctraMode
 if true, then: 1) stringDelimiter can be changed 2) Kicad quoting protocol is not in effect 3) space_in_quoted_tokens is functional else not. More...
 
char stringDelimiter
 
bool space_in_quoted_tokens
 blank spaces within quoted strings More...
 
bool commentsAreTokens
 true if should return comments as tokens More...
 
int prevTok
 curTok from previous NextTok() call. More...
 
int curOffset
 offset within current line of the current token More...
 
int curTok
 the current token obtained on last NextTok() More...
 
std::string curText
 the text of the current token More...
 
const KEYWORDkeywords
 table sorted by CMake for bsearch() More...
 
unsigned keywordCount
 count of keywords table More...
 
KEYWORD_MAP keyword_hash
 fast, specialized "C string" hashtable More...
 

Detailed Description

Implement a lexical analyzer for the SPECCTRA DSN file format.

It reads lexical tokens from the current LINE_READER through the NextTok() function.

Definition at line 78 of file dsnlexer.h.

Member Typedef Documentation

◆ READER_STACK

typedef std::vector<LINE_READER*> DSNLEXER::READER_STACK
protected

Definition at line 507 of file dsnlexer.h.

Constructor & Destructor Documentation

◆ DSNLEXER() [1/4]

DSNLEXER::DSNLEXER ( const KEYWORD aKeywordTable,
unsigned  aKeywordCount,
FILE *  aFile,
const wxString &  aFileName 
)

Initialize a DSN lexer and prepares to read from aFile which is already open and has aFilename.

Parameters
aKeywordTableis an array of KEYWORDS holding aKeywordCount. This token table need not contain the lexer separators such as '(' ')', etc.
aKeywordCountis the count of tokens in aKeywordTable.
aFileis an open file, which will be closed when this is destructed.
aFileNameis the name of the file

Definition at line 72 of file dsnlexer.cpp.

73  :
74  iOwnReaders( true ),
75  start( nullptr ),
76  next( nullptr ),
77  limit( nullptr ),
78  reader( nullptr ),
79  keywords( aKeywordTable ),
80  keywordCount( aKeywordCount )
81 {
82  FILE_LINE_READER* fileReader = new FILE_LINE_READER( aFile, aFilename );
83  PushReader( fileReader );
84  init();
85 }
void PushReader(LINE_READER *aLineReader)
Manage a stack of LINE_READERs in order to handle nested file inclusion.
Definition: dsnlexer.cpp:199
const KEYWORD * keywords
table sorted by CMake for bsearch()
Definition: dsnlexer.h:531
const char * next
Definition: dsnlexer.h:503
A LINE_READER that reads from an open file.
Definition: richio.h:172
const char * limit
Definition: dsnlexer.h:504
bool iOwnReaders
on readerStack, should I delete them?
Definition: dsnlexer.h:501
LINE_READER * reader
Definition: dsnlexer.h:512
const char * start
Definition: dsnlexer.h:502
unsigned keywordCount
count of keywords table
Definition: dsnlexer.h:532
void init()
Definition: dsnlexer.cpp:40

References init(), and PushReader().

◆ DSNLEXER() [2/4]

DSNLEXER::DSNLEXER ( const KEYWORD aKeywordTable,
unsigned  aKeywordCount,
const std::string &  aSExpression,
const wxString &  aSource = wxEmptyString 
)

Initialize a DSN lexer and prepares to read from aSExpression.

Parameters
aKeywordTableis an array of KEYWORDS holding aKeywordCount. This token table need not contain the lexer separators such as '(' ')', etc.
aKeywordCountis the count of tokens in aKeywordTable.
aSExpressionis text to feed through a STRING_LINE_READER
aSourceis a description of aSExpression, used for error reporting.

Definition at line 88 of file dsnlexer.cpp.

89  :
90  iOwnReaders( true ),
91  start( nullptr ),
92  next( nullptr ),
93  limit( nullptr ),
94  reader( nullptr ),
95  keywords( aKeywordTable ),
96  keywordCount( aKeywordCount )
97 {
98  STRING_LINE_READER* stringReader = new STRING_LINE_READER( aClipboardTxt, aSource.IsEmpty() ?
99  wxString( FMT_CLIPBOARD ) : aSource );
100  PushReader( stringReader );
101  init();
102 }
void PushReader(LINE_READER *aLineReader)
Manage a stack of LINE_READERs in order to handle nested file inclusion.
Definition: dsnlexer.cpp:199
const KEYWORD * keywords
table sorted by CMake for bsearch()
Definition: dsnlexer.h:531
const char * next
Definition: dsnlexer.h:503
const char * limit
Definition: dsnlexer.h:504
bool iOwnReaders
on readerStack, should I delete them?
Definition: dsnlexer.h:501
LINE_READER * reader
Definition: dsnlexer.h:512
const char * start
Definition: dsnlexer.h:502
#define FMT_CLIPBOARD
Definition: dsnlexer.cpp:35
unsigned keywordCount
count of keywords table
Definition: dsnlexer.h:532
Is a LINE_READER that reads from a multiline 8 bit wide std::string.
Definition: richio.h:240
void init()
Definition: dsnlexer.cpp:40

References FMT_CLIPBOARD, init(), and PushReader().

◆ DSNLEXER() [3/4]

DSNLEXER::DSNLEXER ( const std::string &  aSExpression,
const wxString &  aSource = wxEmptyString 
)

Initialize a DSN lexer and prepares to read from aSExpression.

Use this one without a keyword table with the DOM parser in ptree.h.

Parameters
aSExpressionis text to feed through a STRING_LINE_READER
aSourceis a description of aSExpression, used for error reporting.

Definition at line 123 of file dsnlexer.cpp.

123  :
124  iOwnReaders( true ),
125  start( nullptr ),
126  next( nullptr ),
127  limit( nullptr ),
128  reader( nullptr ),
130  keywordCount( 0 )
131 {
132  STRING_LINE_READER* stringReader = new STRING_LINE_READER( aSExpression, aSource.IsEmpty() ?
133  wxString( FMT_CLIPBOARD ) : aSource );
134  PushReader( stringReader );
135  init();
136 }
void PushReader(LINE_READER *aLineReader)
Manage a stack of LINE_READERs in order to handle nested file inclusion.
Definition: dsnlexer.cpp:199
const KEYWORD * keywords
table sorted by CMake for bsearch()
Definition: dsnlexer.h:531
const char * next
Definition: dsnlexer.h:503
const char * limit
Definition: dsnlexer.h:504
bool iOwnReaders
on readerStack, should I delete them?
Definition: dsnlexer.h:501
static const KEYWORD empty_keywords[1]
Definition: dsnlexer.cpp:121
LINE_READER * reader
Definition: dsnlexer.h:512
const char * start
Definition: dsnlexer.h:502
#define FMT_CLIPBOARD
Definition: dsnlexer.cpp:35
unsigned keywordCount
count of keywords table
Definition: dsnlexer.h:532
Is a LINE_READER that reads from a multiline 8 bit wide std::string.
Definition: richio.h:240
void init()
Definition: dsnlexer.cpp:40

References FMT_CLIPBOARD, init(), and PushReader().

◆ DSNLEXER() [4/4]

DSNLEXER::DSNLEXER ( const KEYWORD aKeywordTable,
unsigned  aKeywordCount,
LINE_READER aLineReader = nullptr 
)

Initialize a DSN lexer and prepares to read from aLineReader which is already open, and may be in use by other DSNLEXERs also.

No ownership is taken of aLineReader. This enables it to be used by other DSNLEXERs.

Parameters
aKeywordTableis an array of #KEYWORDS holding aKeywordCount. This token table need not contain the lexer separators such as '(' ')', etc.
aKeywordCountis the count of tokens in aKeywordTable.
aLineReaderis any subclassed instance of LINE_READER, such as STRING_LINE_READER or FILE_LINE_READER. No ownership is taken.

Definition at line 105 of file dsnlexer.cpp.

106  :
107  iOwnReaders( false ),
108  start( nullptr ),
109  next( nullptr ),
110  limit( nullptr ),
111  reader( nullptr ),
112  keywords( aKeywordTable ),
113  keywordCount( aKeywordCount )
114 {
115  if( aLineReader )
116  PushReader( aLineReader );
117  init();
118 }
void PushReader(LINE_READER *aLineReader)
Manage a stack of LINE_READERs in order to handle nested file inclusion.
Definition: dsnlexer.cpp:199
const KEYWORD * keywords
table sorted by CMake for bsearch()
Definition: dsnlexer.h:531
const char * next
Definition: dsnlexer.h:503
const char * limit
Definition: dsnlexer.h:504
bool iOwnReaders
on readerStack, should I delete them?
Definition: dsnlexer.h:501
LINE_READER * reader
Definition: dsnlexer.h:512
const char * start
Definition: dsnlexer.h:502
unsigned keywordCount
count of keywords table
Definition: dsnlexer.h:532
void init()
Definition: dsnlexer.cpp:40

References init(), and PushReader().

◆ ~DSNLEXER()

DSNLEXER::~DSNLEXER ( )
virtual

Definition at line 139 of file dsnlexer.cpp.

140 {
141  if( iOwnReaders )
142  {
143  // delete the LINE_READERs from the stack, since I own them.
144  for( READER_STACK::iterator it = readerStack.begin(); it!=readerStack.end(); ++it )
145  delete *it;
146  }
147 }
bool iOwnReaders
on readerStack, should I delete them?
Definition: dsnlexer.h:501
READER_STACK readerStack
all the LINE_READERs by pointer.
Definition: dsnlexer.h:509

References iOwnReaders, and readerStack.

Member Function Documentation

◆ CurLine()

const char* DSNLEXER::CurLine ( ) const
inline

Return the current line of text from which the CurText() would return its token.

Definition at line 431 of file dsnlexer.h.

432  {
433  return (const char*)(*reader);
434  }

Referenced by Duplicate(), Expecting(), NeedNUMBER(), NextTok(), and Unexpected().

◆ CurLineNumber()

int DSNLEXER::CurLineNumber ( ) const
inline

Return the current line number within my LINE_READER.

Definition at line 423 of file dsnlexer.h.

424  {
425  return reader->LineNumber();
426  }
virtual unsigned LineNumber() const
Return the line number of the last line read from this LINE_READER.
Definition: richio.h:135
LINE_READER * reader
Definition: dsnlexer.h:512

References LINE_READER::LineNumber(), and reader.

Referenced by Duplicate(), Expecting(), NeedNUMBER(), NextTok(), and Unexpected().

◆ CurOffset()

int DSNLEXER::CurOffset ( ) const
inline

Return the byte offset within the current line, using a 1 based index.

Returns
a one based index into the current line.

Definition at line 451 of file dsnlexer.h.

452  {
453  return curOffset + 1;
454  }
int curOffset
offset within current line of the current token
Definition: dsnlexer.h:526

References curOffset.

Referenced by Duplicate(), Expecting(), NeedNUMBER(), NextTok(), and Unexpected().

◆ CurSource()

const wxString& DSNLEXER::CurSource ( ) const
inline

Return the current LINE_READER source.

Returns
source of the lines of text, e.g. a filename or "clipboard".

Definition at line 441 of file dsnlexer.h.

442  {
443  return reader->GetSource();
444  }
virtual const wxString & GetSource() const
Returns the name of the source of the lines in an abstract sense.
Definition: richio.h:109
LINE_READER * reader
Definition: dsnlexer.h:512

References LINE_READER::GetSource(), and reader.

Referenced by Duplicate(), Expecting(), NeedNUMBER(), NextTok(), and Unexpected().

◆ CurStr()

const std::string& DSNLEXER::CurStr ( ) const
inline

Return a reference to current token in std::string form.

Definition at line 406 of file dsnlexer.h.

407  {
408  return curText;
409  }
std::string curText
the text of the current token
Definition: dsnlexer.h:529

References curText.

◆ CurText()

const char* DSNLEXER::CurText ( ) const
inline

Return a pointer to the current token's text.

Definition at line 398 of file dsnlexer.h.

399  {
400  return curText.c_str();
401  }
std::string curText
the text of the current token
Definition: dsnlexer.h:529

References curText.

Referenced by CADSTAR_ARCHIVE_PARSER::LoadArchiveFile(), PCAD2KICAD::LoadInputFile(), scanAtom(), and scanList().

◆ CurTok()

int DSNLEXER::CurTok ( ) const
inline

Return whatever NextTok() returned the last time it was called.

Definition at line 237 of file dsnlexer.h.

238  {
239  return curTok;
240  }
int curTok
the current token obtained on last NextTok()
Definition: dsnlexer.h:528

References curTok.

Referenced by Scan(), and scanList().

◆ Duplicate()

void DSNLEXER::Duplicate ( int  aTok)

Throw an IO_ERROR exception with a message saying specifically that aTok is a duplicate of one already seen in current context.

Parameters
aTokis the token/keyword type which was not expected at the current input location.
Exceptions
IO_ERRORwith the location within the input file of the problem.

Definition at line 357 of file dsnlexer.cpp.

358 {
359  wxString errText = wxString::Format(
360  _("%s is a duplicate"), GetTokenString( aTok ).GetData() );
362 }
int CurLineNumber() const
Return the current line number within my LINE_READER.
Definition: dsnlexer.h:423
wxString GetTokenString(int aTok) const
Return a quote wrapped wxString representation of a token value.
Definition: dsnlexer.cpp:314
#define THROW_PARSE_ERROR(aProblem, aSource, aInputLine, aLineNumber, aByteIndex)
Definition: ki_exception.h:164
#define _(s)
void Format(OUTPUTFORMATTER *out, int aNestLevel, int aCtl, const CPTREE &aTree)
Output a PTREE into s-expression format via an OUTPUTFORMATTER derivative.
Definition: ptree.cpp:200
int CurOffset() const
Return the byte offset within the current line, using a 1 based index.
Definition: dsnlexer.h:451
const char * CurLine() const
Return the current line of text from which the CurText() would return its token.
Definition: dsnlexer.h:431
const wxString & CurSource() const
Return the current LINE_READER source.
Definition: dsnlexer.h:441

References _, CurLine(), CurLineNumber(), CurOffset(), CurSource(), Format(), GetTokenString(), and THROW_PARSE_ERROR.

◆ Expecting() [1/2]

void DSNLEXER::Expecting ( int  aTok) const

Throw an IO_ERROR exception with an input file specific error message.

Parameters
aTokis the token/keyword type which was expected at the current input location.
Exceptions
IO_ERRORwith the location within the input file of the problem.

Definition at line 333 of file dsnlexer.cpp.

334 {
335  wxString errText = wxString::Format(
336  _( "Expecting %s" ), GetTokenString( aTok ) );
338 }
int CurLineNumber() const
Return the current line number within my LINE_READER.
Definition: dsnlexer.h:423
wxString GetTokenString(int aTok) const
Return a quote wrapped wxString representation of a token value.
Definition: dsnlexer.cpp:314
#define THROW_PARSE_ERROR(aProblem, aSource, aInputLine, aLineNumber, aByteIndex)
Definition: ki_exception.h:164
#define _(s)
void Format(OUTPUTFORMATTER *out, int aNestLevel, int aCtl, const CPTREE &aTree)
Output a PTREE into s-expression format via an OUTPUTFORMATTER derivative.
Definition: ptree.cpp:200
int CurOffset() const
Return the byte offset within the current line, using a 1 based index.
Definition: dsnlexer.h:451
const char * CurLine() const
Return the current line of text from which the CurText() would return its token.
Definition: dsnlexer.h:431
const wxString & CurSource() const
Return the current LINE_READER source.
Definition: dsnlexer.h:441

References _, CurLine(), CurLineNumber(), CurOffset(), CurSource(), Format(), GetTokenString(), and THROW_PARSE_ERROR.

Referenced by NeedLEFT(), NeedRIGHT(), NeedSYMBOL(), and NeedSYMBOLorNUMBER().

◆ Expecting() [2/2]

void DSNLEXER::Expecting ( const char *  aTokenList) const

Throw an IO_ERROR exception with an input file specific error message.

Parameters
aTokenListis the token/keyword type which was expected at the current input location, e.g. "pin|graphic|property".
Exceptions
IO_ERRORwith the location within the input file of the problem.

Definition at line 341 of file dsnlexer.cpp.

342 {
343  wxString errText = wxString::Format(
344  _( "Expecting '%s'" ), wxString::FromUTF8( text ) );
346 }
int CurLineNumber() const
Return the current line number within my LINE_READER.
Definition: dsnlexer.h:423
#define THROW_PARSE_ERROR(aProblem, aSource, aInputLine, aLineNumber, aByteIndex)
Definition: ki_exception.h:164
#define _(s)
void Format(OUTPUTFORMATTER *out, int aNestLevel, int aCtl, const CPTREE &aTree)
Output a PTREE into s-expression format via an OUTPUTFORMATTER derivative.
Definition: ptree.cpp:200
int CurOffset() const
Return the byte offset within the current line, using a 1 based index.
Definition: dsnlexer.h:451
const char * CurLine() const
Return the current line of text from which the CurText() would return its token.
Definition: dsnlexer.h:431
const wxString & CurSource() const
Return the current LINE_READER source.
Definition: dsnlexer.h:441

References _, CurLine(), CurLineNumber(), CurOffset(), CurSource(), Format(), text, and THROW_PARSE_ERROR.

◆ findToken()

int DSNLEXER::findToken ( const std::string &  aToken) const
protected

Take aToken string and looks up the string in the keywords table.

Parameters
aTokenis a string to lookup in the keywords table.
Returns
with a value from the enum DSN_T matching the keyword text, or DSN_SYMBOL if aToken is not in the keywords table.

Definition at line 240 of file dsnlexer.cpp.

241 {
242  KEYWORD_MAP::const_iterator it = keyword_hash.find( tok.c_str() );
243 
244  if( it != keyword_hash.end() )
245  return it->second;
246 
247  return DSN_SYMBOL; // not a keyword, some arbitrary symbol.
248 }
KEYWORD_MAP keyword_hash
fast, specialized "C string" hashtable
Definition: dsnlexer.h:533

References DSN_SYMBOL, and keyword_hash.

Referenced by GetCurStrAsToken(), and NextTok().

◆ FromUTF8()

wxString DSNLEXER::FromUTF8 ( ) const
inline

Return the current token text as a wxString, assuming that the input byte stream is UTF8 encoded.

Definition at line 415 of file dsnlexer.h.

416  {
417  return wxString::FromUTF8( curText.c_str() );
418  }
std::string curText
the text of the current token
Definition: dsnlexer.h:529

References curText.

Referenced by ReadCommentLines().

◆ GetCurStrAsToken()

int DSNLEXER::GetCurStrAsToken ( ) const
inline

Used to support "loose" matches (quoted tokens).

Definition at line 253 of file dsnlexer.h.

254  {
255  return findToken( curText );
256  }
int findToken(const std::string &aToken) const
Take aToken string and looks up the string in the keywords table.
Definition: dsnlexer.cpp:240
std::string curText
the text of the current token
Definition: dsnlexer.h:529

References curText, and findToken().

◆ GetTokenString()

wxString DSNLEXER::GetTokenString ( int  aTok) const

Return a quote wrapped wxString representation of a token value.

Definition at line 314 of file dsnlexer.cpp.

315 {
316  wxString ret;
317 
318  ret << wxT("'") << wxString::FromUTF8( GetTokenText(aTok) ) << wxT("'");
319 
320  return ret;
321 }
const char * GetTokenText(int aTok) const
Return the C string representation of a DSN_T value.
Definition: dsnlexer.cpp:295

References GetTokenText().

Referenced by Duplicate(), Expecting(), and Unexpected().

◆ GetTokenText()

const char * DSNLEXER::GetTokenText ( int  aTok) const

Return the C string representation of a DSN_T value.

Definition at line 295 of file dsnlexer.cpp.

296 {
297  const char* ret;
298 
299  if( aTok < 0 )
300  {
301  return Syntax( aTok );
302  }
303  else if( (unsigned) aTok < keywordCount )
304  {
305  ret = keywords[aTok].name;
306  }
307  else
308  ret = "token too big";
309 
310  return ret;
311 }
const KEYWORD * keywords
table sorted by CMake for bsearch()
Definition: dsnlexer.h:531
const char * name
unique keyword.
Definition: dsnlexer.h:41
unsigned keywordCount
count of keywords table
Definition: dsnlexer.h:532
static const char * Syntax(int aTok)
Definition: dsnlexer.cpp:251

References keywordCount, keywords, KEYWORD::name, and Syntax().

Referenced by GetTokenString().

◆ init()

void DSNLEXER::init ( )
protected

Definition at line 40 of file dsnlexer.cpp.

41 {
42  curTok = DSN_NONE;
43  prevTok = DSN_NONE;
44 
45  stringDelimiter = '"';
46 
47  specctraMode = false;
48  space_in_quoted_tokens = false;
49  commentsAreTokens = false;
50 
51  curOffset = 0;
52 
53 #if 1
54  if( keywordCount > 11 )
55  {
56  // resize the hashtable bucket count
57  keyword_hash.reserve( keywordCount );
58  }
59 
60  // fill the specialized "C string" hashtable from keywords[]
61  const KEYWORD* it = keywords;
62  const KEYWORD* end = it + keywordCount;
63 
64  for( ; it < end; ++it )
65  {
66  keyword_hash[it->name] = it->token;
67  }
68 #endif
69 }
int curOffset
offset within current line of the current token
Definition: dsnlexer.h:526
bool commentsAreTokens
true if should return comments as tokens
Definition: dsnlexer.h:523
const KEYWORD * keywords
table sorted by CMake for bsearch()
Definition: dsnlexer.h:531
const char * name
unique keyword.
Definition: dsnlexer.h:41
KEYWORD_MAP keyword_hash
fast, specialized "C string" hashtable
Definition: dsnlexer.h:533
bool space_in_quoted_tokens
blank spaces within quoted strings
Definition: dsnlexer.h:521
bool specctraMode
if true, then: 1) stringDelimiter can be changed 2) Kicad quoting protocol is not in effect 3) space_...
Definition: dsnlexer.h:514
int prevTok
curTok from previous NextTok() call.
Definition: dsnlexer.h:525
unsigned keywordCount
count of keywords table
Definition: dsnlexer.h:532
int curTok
the current token obtained on last NextTok()
Definition: dsnlexer.h:528
Hold a keyword string and its unique integer token.
Definition: dsnlexer.h:39
char stringDelimiter
Definition: dsnlexer.h:520
int token
a zero based index into an array of KEYWORDs
Definition: dsnlexer.h:42

References commentsAreTokens, curOffset, curTok, DSN_NONE, keyword_hash, keywordCount, keywords, KEYWORD::name, prevTok, space_in_quoted_tokens, specctraMode, stringDelimiter, and KEYWORD::token.

Referenced by DSNLEXER().

◆ InitParserState()

void DSNLEXER::InitParserState ( )

Reinit variables used during parsing, to ensure od states are not used in a new parsing must be called before parsing a new file after parsing an old file to avoid starting with some variables in a non initial state.

Definition at line 166 of file dsnlexer.cpp.

167 {
168  curTok = DSN_NONE;
169  prevTok = DSN_NONE;
170  commentsAreTokens = false;
171 
172  curOffset = 0;
173 }
int curOffset
offset within current line of the current token
Definition: dsnlexer.h:526
bool commentsAreTokens
true if should return comments as tokens
Definition: dsnlexer.h:523
int prevTok
curTok from previous NextTok() call.
Definition: dsnlexer.h:525
int curTok
the current token obtained on last NextTok()
Definition: dsnlexer.h:528

References commentsAreTokens, curOffset, curTok, DSN_NONE, and prevTok.

◆ isStringTerminator()

bool DSNLEXER::isStringTerminator ( char  cc) const
inlineprotected

Definition at line 490 of file dsnlexer.h.

491  {
492  if( !space_in_quoted_tokens && cc == ' ' )
493  return true;
494 
495  if( cc == stringDelimiter )
496  return true;
497 
498  return false;
499  }
bool space_in_quoted_tokens
blank spaces within quoted strings
Definition: dsnlexer.h:521
char stringDelimiter
Definition: dsnlexer.h:520

References space_in_quoted_tokens, and stringDelimiter.

Referenced by NextTok().

◆ IsSymbol()

bool DSNLEXER::IsSymbol ( int  aTok)
static

Test a token to see if it is a symbol.

This means it cannot be a special delimiter character such as DSN_LEFT, DSN_RIGHT, #DSN_QUOTE, etc. It may however, coincidentally match a keyword and still be a symbol.

Definition at line 324 of file dsnlexer.cpp.

325 {
326  // This is static and not inline to reduce code space.
327 
328  // if aTok is >= 0, then it is a coincidental match to a keyword.
329  return aTok == DSN_SYMBOL || aTok == DSN_STRING || aTok >= 0;
330 }

References DSN_STRING, and DSN_SYMBOL.

Referenced by NeedSYMBOL(), and NeedSYMBOLorNUMBER().

◆ NeedLEFT()

void DSNLEXER::NeedLEFT ( )

Call NextTok() and then verifies that the token read in is a DSN_LEFT.

Exceptions
IO_ERRORif the next token is not a DSN_LEFT

Definition at line 373 of file dsnlexer.cpp.

374 {
375  int tok = NextTok();
376  if( tok != DSN_LEFT )
377  Expecting( DSN_LEFT );
378 }
int NextTok()
Return the next token found in the input file or DSN_EOF when reaching the end of file.
Definition: dsnlexer.cpp:526
void Expecting(int aTok) const
Throw an IO_ERROR exception with an input file specific error message.
Definition: dsnlexer.cpp:333

References DSN_LEFT, Expecting(), and NextTok().

◆ NeedNUMBER()

int DSNLEXER::NeedNUMBER ( const char *  aExpectation)

Call NextTok() and then verifies that the token read is type DSN_NUMBER.

Returns
the actual token read in.
Exceptions
IO_ERRORif the next token does not satisfy the above test.

Definition at line 412 of file dsnlexer.cpp.

413 {
414  int tok = NextTok();
415 
416  if( tok != DSN_NUMBER )
417  {
418  wxString errText = wxString::Format( _( "need a number for '%s'" ),
419  wxString::FromUTF8( aExpectation ).GetData() );
421  }
422 
423  return tok;
424 }
int CurLineNumber() const
Return the current line number within my LINE_READER.
Definition: dsnlexer.h:423
#define THROW_PARSE_ERROR(aProblem, aSource, aInputLine, aLineNumber, aByteIndex)
Definition: ki_exception.h:164
#define _(s)
void Format(OUTPUTFORMATTER *out, int aNestLevel, int aCtl, const CPTREE &aTree)
Output a PTREE into s-expression format via an OUTPUTFORMATTER derivative.
Definition: ptree.cpp:200
int CurOffset() const
Return the byte offset within the current line, using a 1 based index.
Definition: dsnlexer.h:451
int NextTok()
Return the next token found in the input file or DSN_EOF when reaching the end of file.
Definition: dsnlexer.cpp:526
const char * CurLine() const
Return the current line of text from which the CurText() would return its token.
Definition: dsnlexer.h:431
const wxString & CurSource() const
Return the current LINE_READER source.
Definition: dsnlexer.h:441

References _, CurLine(), CurLineNumber(), CurOffset(), CurSource(), DSN_NUMBER, Format(), NextTok(), and THROW_PARSE_ERROR.

◆ NeedRIGHT()

void DSNLEXER::NeedRIGHT ( )

Call NextTok() and then verifies that the token read in is a DSN_RIGHT.

Exceptions
IO_ERRORif the next token is not a DSN_RIGHT

Definition at line 381 of file dsnlexer.cpp.

382 {
383  int tok = NextTok();
384 
385  if( tok != DSN_RIGHT )
386  Expecting( DSN_RIGHT );
387 }
int NextTok()
Return the next token found in the input file or DSN_EOF when reaching the end of file.
Definition: dsnlexer.cpp:526
void Expecting(int aTok) const
Throw an IO_ERROR exception with an input file specific error message.
Definition: dsnlexer.cpp:333

References DSN_RIGHT, Expecting(), and NextTok().

◆ NeedSYMBOL()

int DSNLEXER::NeedSYMBOL ( )

Call NextTok() and then verifies that the token read in satisfies IsSymbol().

Returns
the actual token read in.
Exceptions
IO_ERRORif the next token does not satisfy IsSymbol().

Definition at line 390 of file dsnlexer.cpp.

391 {
392  int tok = NextTok();
393 
394  if( !IsSymbol( tok ) )
396 
397  return tok;
398 }
static bool IsSymbol(int aTok)
Test a token to see if it is a symbol.
Definition: dsnlexer.cpp:324
int NextTok()
Return the next token found in the input file or DSN_EOF when reaching the end of file.
Definition: dsnlexer.cpp:526
void Expecting(int aTok) const
Throw an IO_ERROR exception with an input file specific error message.
Definition: dsnlexer.cpp:333

References DSN_SYMBOL, Expecting(), IsSymbol(), and NextTok().

◆ NeedSYMBOLorNUMBER()

int DSNLEXER::NeedSYMBOLorNUMBER ( )

Call NextTok() and then verifies that the token read in satisfies bool IsSymbol() or the next token is DSN_NUMBER.

Returns
the actual token read in.
Exceptions
IO_ERRORif the next token does not satisfy the above test.

Definition at line 401 of file dsnlexer.cpp.

402 {
403  int tok = NextTok();
404 
405  if( !IsSymbol( tok ) && tok!=DSN_NUMBER )
406  Expecting( "a symbol or number" );
407 
408  return tok;
409 }
static bool IsSymbol(int aTok)
Test a token to see if it is a symbol.
Definition: dsnlexer.cpp:324
int NextTok()
Return the next token found in the input file or DSN_EOF when reaching the end of file.
Definition: dsnlexer.cpp:526
void Expecting(int aTok) const
Throw an IO_ERROR exception with an input file specific error message.
Definition: dsnlexer.cpp:333

References DSN_NUMBER, Expecting(), IsSymbol(), and NextTok().

◆ NextTok()

int DSNLEXER::NextTok ( )

Return the next token found in the input file or DSN_EOF when reaching the end of file.

Users should wrap this function to return an enum to aid in grammar debugging while running under a debugger, but leave this lower level function returning an int (so the enum does not collide with another usage).

Returns
the type of token found next.
Exceptions
IO_ERRORonly if the LINE_READER throws it.

Definition at line 526 of file dsnlexer.cpp.

527 {
528  const char* cur = next;
529  const char* head = cur;
530 
531  prevTok = curTok;
532 
533  if( curTok == DSN_EOF )
534  goto exit;
535 
536  if( cur >= limit )
537  {
538 L_read:
539  // blank lines are returned as "\n" and will have a len of 1.
540  // EOF will have a len of 0 and so is detectable.
541  int len = readLine();
542 
543  if( len == 0 )
544  {
545  cur = start; // after readLine(), since start can change, set cur offset to start
546  curTok = DSN_EOF;
547  goto exit;
548  }
549 
550  cur = start; // after readLine() since start can change.
551 
552  // skip leading whitespace
553  while( cur < limit && isSpace( *cur ) )
554  ++cur;
555 
556  // If the first non-blank character is #, this line is a comment.
557  // Comments cannot follow any other token on the same line.
558  if( cur<limit && *cur=='#' )
559  {
560  if( commentsAreTokens )
561  {
562  // Grab the entire current line [excluding end of line char(s)] as the
563  // current token. The '#' character may not be at offset zero.
564 
565  while( limit[-1] == '\n' || limit[-1] == '\r' )
566  --limit;
567 
568  curText.clear();
569  curText.append( start, limit );
570 
571  cur = start; // ensure a good curOffset below
573  head = limit; // do a readLine() on next call in here.
574  goto exit;
575  }
576  else
577  {
578  goto L_read;
579  }
580  }
581  }
582  else
583  {
584  // skip leading whitespace
585  while( cur < limit && isSpace( *cur ) )
586  ++cur;
587  }
588 
589  if( cur >= limit )
590  goto L_read;
591 
592  if( *cur == '(' )
593  {
594  curText = *cur;
595  curTok = DSN_LEFT;
596  head = cur+1;
597  goto exit;
598  }
599 
600  if( *cur == ')' )
601  {
602  curText = *cur;
603  curTok = DSN_RIGHT;
604  head = cur+1;
605  goto exit;
606  }
607 
608  // Non-specctraMode, understands and deciphers escaped \, \r, \n, and \".
609  // Strips off leading and trailing double quotes
610  if( !specctraMode )
611  {
612  // a quoted string, will return DSN_STRING
613  if( *cur == stringDelimiter )
614  {
615  // copy the token, character by character so we can remove doubled up quotes.
616  curText.clear();
617 
618  ++cur; // skip over the leading delimiter, which is always " in non-specctraMode
619 
620  head = cur;
621 
622  while( head<limit )
623  {
624  // ESCAPE SEQUENCES:
625  if( *head =='\\' )
626  {
627  char tbuf[8];
628  char c;
629  int i;
630 
631  if( ++head >= limit )
632  break; // throw exception at L_unterminated
633 
634  switch( *head++ )
635  {
636  case '"':
637  case '\\': c = head[-1]; break;
638  case 'a': c = '\x07'; break;
639  case 'b': c = '\x08'; break;
640  case 'f': c = '\x0c'; break;
641  case 'n': c = '\n'; break;
642  case 'r': c = '\r'; break;
643  case 't': c = '\x09'; break;
644  case 'v': c = '\x0b'; break;
645 
646  case 'x': // 1 or 2 byte hex escape sequence
647  for( i = 0; i < 2; ++i )
648  {
649  if( !isxdigit( head[i] ) )
650  break;
651 
652  tbuf[i] = head[i];
653  }
654 
655  tbuf[i] = '\0';
656 
657  if( i > 0 )
658  c = (char) strtoul( tbuf, nullptr, 16 );
659  else
660  c = 'x'; // a goofed hex escape sequence, interpret as 'x'
661 
662  head += i;
663  break;
664 
665  default: // 1-3 byte octal escape sequence
666  --head;
667 
668  for( i=0; i<3; ++i )
669  {
670  if( head[i] < '0' || head[i] > '7' )
671  break;
672 
673  tbuf[i] = head[i];
674  }
675 
676  tbuf[i] = '\0';
677 
678  if( i > 0 )
679  c = (char) strtoul( tbuf, nullptr, 8 );
680  else
681  c = '\\'; // a goofed octal escape sequence, interpret as '\'
682 
683  head += i;
684  break;
685  }
686 
687  curText += c;
688  }
689 
690  else if( *head == '"' ) // end of the non-specctraMode DSN_STRING
691  {
692  curTok = DSN_STRING;
693  ++head; // omit this trailing double quote
694  goto exit;
695  }
696 
697  else
698  curText += *head++;
699 
700  } // while
701 
702  // L_unterminated:
703  wxString errtxt( _( "Un-terminated delimited string" ) );
705  cur - start + curText.length() );
706  }
707  }
708  else // is specctraMode, tests in this block should not occur in KiCad mode.
709  {
710  /* get the dash out of a <pin_reference> which is embedded for example
711  like: U2-14 or "U2"-"14"
712  This is detectable by a non-space immediately preceding the dash.
713  */
714  if( *cur == '-' && cur>start && !isSpace( cur[-1] ) )
715  {
716  curText = '-';
717  curTok = DSN_DASH;
718  head = cur+1;
719  goto exit;
720  }
721 
722  // switching the string_quote character
723  if( prevTok == DSN_STRING_QUOTE )
724  {
725  static const wxString errtxt( _("String delimiter must be a single character of "
726  "', \", or $") );
727 
728  char cc = *cur;
729  switch( cc )
730  {
731  case '\'':
732  case '$':
733  case '"':
734  break;
735  default:
737  }
738 
739  curText = cc;
740 
741  head = cur+1;
742 
743  if( head<limit && !isSep( *head ) )
744  {
746  }
747 
749  goto exit;
750  }
751 
752  // specctraMode DSN_STRING
753  if( *cur == stringDelimiter )
754  {
755  ++cur; // skip over the leading delimiter: ",', or $
756 
757  head = cur;
758 
759  while( head<limit && !isStringTerminator( *head ) )
760  ++head;
761 
762  if( head >= limit )
763  {
764  wxString errtxt( _( "Un-terminated delimited string" ) );
766  }
767 
768  curText.clear();
769  curText.append( cur, head );
770 
771  ++head; // skip over the trailing delimiter
772 
773  curTok = DSN_STRING;
774  goto exit;
775  }
776  } // specctraMode
777 
778  // non-quoted token, read it into curText.
779  curText.clear();
780 
781  head = cur;
782  while( head<limit && !isSep( *head ) )
783  curText += *head++;
784 
785  if( isNumber( curText.c_str(), curText.c_str() + curText.size() ) )
786  {
787  curTok = DSN_NUMBER;
788  goto exit;
789  }
790 
791  if( specctraMode && curText == "string_quote" )
792  {
794  goto exit;
795  }
796 
797  curTok = findToken( curText );
798 
799 exit: // single point of exit, no returns elsewhere please.
800 
801  curOffset = cur - start;
802 
803  next = head;
804 
805  return curTok;
806 }
int curOffset
offset within current line of the current token
Definition: dsnlexer.h:526
int CurLineNumber() const
Return the current line number within my LINE_READER.
Definition: dsnlexer.h:423
bool commentsAreTokens
true if should return comments as tokens
Definition: dsnlexer.h:523
bool isSep(char cc)
Definition: dsnlexer.cpp:461
bool isStringTerminator(char cc) const
Definition: dsnlexer.h:490
static bool isNumber(const char *cp, const char *limit)
Return true if the next sequence of text is a number: either an integer, fixed point,...
Definition: dsnlexer.cpp:476
int findToken(const std::string &aToken) const
Take aToken string and looks up the string in the keywords table.
Definition: dsnlexer.cpp:240
const char * next
Definition: dsnlexer.h:503
#define THROW_PARSE_ERROR(aProblem, aSource, aInputLine, aLineNumber, aByteIndex)
Definition: ki_exception.h:164
const char * limit
Definition: dsnlexer.h:504
static bool isSpace(char cc)
Test for whitespace.
Definition: dsnlexer.cpp:433
#define _(s)
bool specctraMode
if true, then: 1) stringDelimiter can be changed 2) Kicad quoting protocol is not in effect 3) space_...
Definition: dsnlexer.h:514
int prevTok
curTok from previous NextTok() call.
Definition: dsnlexer.h:525
int readLine()
Definition: dsnlexer.h:461
int CurOffset() const
Return the byte offset within the current line, using a 1 based index.
Definition: dsnlexer.h:451
const char * start
Definition: dsnlexer.h:502
std::string curText
the text of the current token
Definition: dsnlexer.h:529
const char * CurLine() const
Return the current line of text from which the CurText() would return its token.
Definition: dsnlexer.h:431
const wxString & CurSource() const
Return the current LINE_READER source.
Definition: dsnlexer.h:441
int curTok
the current token obtained on last NextTok()
Definition: dsnlexer.h:528
char stringDelimiter
Definition: dsnlexer.h:520

References _, commentsAreTokens, CurLine(), CurLineNumber(), CurOffset(), curOffset, CurSource(), curText, curTok, DSN_COMMENT, DSN_DASH, DSN_EOF, DSN_LEFT, DSN_NUMBER, DSN_QUOTE_DEF, DSN_RIGHT, DSN_STRING, DSN_STRING_QUOTE, exit, findToken(), isNumber(), isSep(), isSpace(), isStringTerminator(), limit, next, prevTok, readLine(), specctraMode, start, stringDelimiter, and THROW_PARSE_ERROR.

Referenced by CADSTAR_ARCHIVE_PARSER::LoadArchiveFile(), PCAD2KICAD::LoadInputFile(), NeedLEFT(), NeedNUMBER(), NeedRIGHT(), NeedSYMBOL(), NeedSYMBOLorNUMBER(), ReadCommentLines(), Scan(), and scanList().

◆ PopReader()

LINE_READER * DSNLEXER::PopReader ( )

Delete the top most LINE_READER from an internal stack of LINE_READERs and in the case of FILE_LINE_READER this means the associated FILE is closed.

The most recently used former LINE_READER on the stack becomes the current LINE_READER and its previous position in its input stream and the its latest line number should pertain. PopReader always starts reading from a new line upon returning to the previous LINE_READER. A pop is only possible if there are at least 2 #LINE_READERs on the stack, since popping the last one is not supported.

Returns
the LINE_READER that was in use before the pop, or NULL if there was not at least two readers on the stack and therefore the pop failed.

Definition at line 211 of file dsnlexer.cpp.

212 {
213  LINE_READER* ret = nullptr;
214 
215  if( readerStack.size() )
216  {
217  ret = reader;
218  readerStack.pop_back();
219 
220  if( readerStack.size() )
221  {
222  reader = readerStack.back();
223  start = reader->Line();
224 
225  // force a new readLine() as first thing.
226  limit = start;
227  next = start;
228  }
229  else
230  {
231  reader = nullptr;
232  start = dummy;
233  limit = dummy;
234  }
235  }
236  return ret;
237 }
An abstract class from which implementation specific LINE_READERs may be derived to read single lines...
Definition: richio.h:80
char * Line() const
Return a pointer to the last line that was read in.
Definition: richio.h:117
const char * next
Definition: dsnlexer.h:503
const char * limit
Definition: dsnlexer.h:504
READER_STACK readerStack
all the LINE_READERs by pointer.
Definition: dsnlexer.h:509
LINE_READER * reader
Definition: dsnlexer.h:512
const char * start
Definition: dsnlexer.h:502
char dummy[1]
when there is no reader.
Definition: dsnlexer.h:505

References dummy, limit, LINE_READER::Line(), next, reader, readerStack, and start.

◆ PrevTok()

int DSNLEXER::PrevTok ( ) const
inline

Return whatever NextTok() returned the 2nd to last time it was called.

Definition at line 245 of file dsnlexer.h.

246  {
247  return prevTok;
248  }
int prevTok
curTok from previous NextTok() call.
Definition: dsnlexer.h:525

References prevTok.

◆ PushReader()

void DSNLEXER::PushReader ( LINE_READER aLineReader)

Manage a stack of LINE_READERs in order to handle nested file inclusion.

This function pushes aLineReader onto the top of a stack of LINE_READERs and makes it the current LINE_READER with its own #GetSource(), line number and line text. A grammar must be designed such that the "include" token (whatever its various names), and any of its parameters are not followed by anything on that same line, because PopReader always starts reading from a new line upon returning to the original LINE_READER.

Definition at line 199 of file dsnlexer.cpp.

200 {
201  readerStack.push_back( aLineReader );
202  reader = aLineReader;
203  start = (const char*) (*reader);
204 
205  // force a new readLine() as first thing.
206  limit = start;
207  next = start;
208 }
const char * next
Definition: dsnlexer.h:503
const char * limit
Definition: dsnlexer.h:504
READER_STACK readerStack
all the LINE_READERs by pointer.
Definition: dsnlexer.h:509
LINE_READER * reader
Definition: dsnlexer.h:512
const char * start
Definition: dsnlexer.h:502

References limit, next, reader, readerStack, and start.

Referenced by DSNLEXER().

◆ ReadCommentLines()

wxArrayString * DSNLEXER::ReadCommentLines ( )

Check the next sequence of tokens and reads them into a wxArrayString if they are comments.

Reading continues until a non-comment token is encountered, and such last read token remains as CurTok() and as CurText(). No push back or "un get" mechanism is used for this support. Upon return you simply avoid calling NextTok() for the next token, but rather CurTok().

Returns
Heap allocated block of comments or NULL if none. The caller owns the allocation and must delete if not NULL.

Definition at line 809 of file dsnlexer.cpp.

810 {
811  wxArrayString* ret = nullptr;
812  bool cmt_setting = SetCommentsAreTokens( true );
813  int tok = NextTok();
814 
815  if( tok == DSN_COMMENT )
816  {
817  ret = new wxArrayString();
818 
819  do
820  {
821  ret->Add( FromUTF8() );
822  }
823  while( ( tok = NextTok() ) == DSN_COMMENT );
824  }
825 
826  SetCommentsAreTokens( cmt_setting );
827 
828  return ret;
829 }
bool SetCommentsAreTokens(bool val)
Change the handling of comments.
Definition: dsnlexer.h:296
int NextTok()
Return the next token found in the input file or DSN_EOF when reaching the end of file.
Definition: dsnlexer.cpp:526
wxString FromUTF8() const
Return the current token text as a wxString, assuming that the input byte stream is UTF8 encoded.
Definition: dsnlexer.h:415

References DSN_COMMENT, FromUTF8(), NextTok(), and SetCommentsAreTokens().

◆ readLine()

int DSNLEXER::readLine ( )
inlineprotected

Definition at line 461 of file dsnlexer.h.

462  {
463  if( reader )
464  {
465  reader->ReadLine();
466 
467  unsigned len = reader->Length();
468 
469  // start may have changed in ReadLine(), which can resize and
470  // relocate reader's line buffer.
471  start = reader->Line();
472 
473  next = start;
474  limit = next + len;
475 
476  return len;
477  }
478  return 0;
479  }
char * Line() const
Return a pointer to the last line that was read in.
Definition: richio.h:117
const char * next
Definition: dsnlexer.h:503
const char * limit
Definition: dsnlexer.h:504
LINE_READER * reader
Definition: dsnlexer.h:512
unsigned Length() const
Return the number of bytes in the last line read from this LINE_READER.
Definition: richio.h:143
const char * start
Definition: dsnlexer.h:502
virtual char * ReadLine()=0
Read a line of text into the buffer and increments the line number counter.

References LINE_READER::Length(), limit, LINE_READER::Line(), next, reader, LINE_READER::ReadLine(), and start.

Referenced by NextTok().

◆ SetCommentsAreTokens()

bool DSNLEXER::SetCommentsAreTokens ( bool  val)
inline

Change the handling of comments.

If set true, comments are returned as single line strings with a terminating newline. Otherwise they are consumed by the lexer and not returned.

Definition at line 296 of file dsnlexer.h.

297  {
298  bool old = commentsAreTokens;
299  commentsAreTokens = val;
300  return old;
301  }
bool commentsAreTokens
true if should return comments as tokens
Definition: dsnlexer.h:523

References commentsAreTokens.

Referenced by ReadCommentLines().

◆ SetSpaceInQuotedTokens()

bool DSNLEXER::SetSpaceInQuotedTokens ( bool  val)
inline

Change the setting controlling whether a space in a quoted string isa terminator.

Parameters
valIf true, means

Definition at line 280 of file dsnlexer.h.

281  {
282  bool old = space_in_quoted_tokens;
283 
284  if( specctraMode )
286 
287  return old;
288  }
bool space_in_quoted_tokens
blank spaces within quoted strings
Definition: dsnlexer.h:521
bool specctraMode
if true, then: 1) stringDelimiter can be changed 2) Kicad quoting protocol is not in effect 3) space_...
Definition: dsnlexer.h:514

References space_in_quoted_tokens, and specctraMode.

◆ SetSpecctraMode()

void DSNLEXER::SetSpecctraMode ( bool  aMode)

Change the behavior of this lexer into or out of "specctra mode".

If specctra mode, then: -#) stringDelimiter can be changed. -#) KiCad quoting protocol is not in effect. -#) space_in_quoted_tokens is functional else none of the above are true.

The default mode is non-specctra mode, meaning: -#) stringDelimiter cannot be changed. -#) KiCad quoting protocol is in effect. -#) space_in_quoted_tokens is not functional.

Definition at line 150 of file dsnlexer.cpp.

151 {
152  specctraMode = aMode;
153  if( aMode )
154  {
155  // specctra mode defaults, some of which can still be changed in this mode.
156  space_in_quoted_tokens = true;
157  }
158  else
159  {
160  space_in_quoted_tokens = false;
161  stringDelimiter = '"';
162  }
163 }
bool space_in_quoted_tokens
blank spaces within quoted strings
Definition: dsnlexer.h:521
bool specctraMode
if true, then: 1) stringDelimiter can be changed 2) Kicad quoting protocol is not in effect 3) space_...
Definition: dsnlexer.h:514
char stringDelimiter
Definition: dsnlexer.h:520

References space_in_quoted_tokens, specctraMode, and stringDelimiter.

◆ SetStringDelimiter()

char DSNLEXER::SetStringDelimiter ( char  aStringDelimiter)
inline

Change the string delimiter from the default " to some other character and return the old value.

Parameters
aStringDelimiterThe character in lowest 8 bits.
Returns
The old delimiter in the lowest 8 bits.

Definition at line 265 of file dsnlexer.h.

266  {
267  int old = stringDelimiter;
268 
269  if( specctraMode )
270  stringDelimiter = aStringDelimiter;
271 
272  return old;
273  }
bool specctraMode
if true, then: 1) stringDelimiter can be changed 2) Kicad quoting protocol is not in effect 3) space_...
Definition: dsnlexer.h:514
char stringDelimiter
Definition: dsnlexer.h:520

References specctraMode, and stringDelimiter.

◆ SyncLineReaderWith()

bool DSNLEXER::SyncLineReaderWith ( DSNLEXER aLexer)

Usable only for DSN lexers which share the same LINE_READER.

Synchronizes the pointers handling the data read by the LINE_READER. Allows 2 #DNSLEXER objects to share the same current line, when switching from a #DNSLEXER to another #DNSLEXER

Parameters
aLexerthe model.
Returns
true if the sync can be made ( at least the same line reader ).

Definition at line 176 of file dsnlexer.cpp.

177 {
178  // Synchronize the pointers handling the data read by the LINE_READER
179  // only if aLexer shares the same LINE_READER, because only in this case
180  // the char buffer is be common
181 
182  if( reader != aLexer.reader )
183  return false;
184 
185  // We can synchronize the pointers which handle the data currently read
186  start = aLexer.start;
187  next = aLexer.next;
188  limit = aLexer.limit;
189 
190  // Sync these parameters is not mandatory, but could help
191  // for instance in debug
192  curText = aLexer.curText;
193  curOffset = aLexer.curOffset;
194 
195  return true;
196 }
int curOffset
offset within current line of the current token
Definition: dsnlexer.h:526
const char * next
Definition: dsnlexer.h:503
const char * limit
Definition: dsnlexer.h:504
LINE_READER * reader
Definition: dsnlexer.h:512
const char * start
Definition: dsnlexer.h:502
std::string curText
the text of the current token
Definition: dsnlexer.h:529

References curOffset, curText, limit, next, reader, and start.

◆ Syntax()

const char * DSNLEXER::Syntax ( int  aTok)
static

Definition at line 251 of file dsnlexer.cpp.

252 {
253  const char* ret;
254 
255  switch( aTok )
256  {
257  case DSN_NONE:
258  ret = "NONE";
259  break;
260  case DSN_STRING_QUOTE:
261  ret = "string_quote"; // a special DSN syntax token, see specctra spec.
262  break;
263  case DSN_QUOTE_DEF:
264  ret = "quoted text delimiter";
265  break;
266  case DSN_DASH:
267  ret = "-";
268  break;
269  case DSN_SYMBOL:
270  ret = "symbol";
271  break;
272  case DSN_NUMBER:
273  ret = "number";
274  break;
275  case DSN_RIGHT:
276  ret = ")";
277  break;
278  case DSN_LEFT:
279  ret = "(";
280  break;
281  case DSN_STRING:
282  ret = "quoted string";
283  break;
284  case DSN_EOF:
285  ret = "end of input";
286  break;
287  default:
288  ret = "???";
289  }
290 
291  return ret;
292 }

References DSN_DASH, DSN_EOF, DSN_LEFT, DSN_NONE, DSN_NUMBER, DSN_QUOTE_DEF, DSN_RIGHT, DSN_STRING, DSN_STRING_QUOTE, and DSN_SYMBOL.

Referenced by GetTokenText().

◆ Unexpected() [1/2]

void DSNLEXER::Unexpected ( int  aTok) const

Throw an IO_ERROR exception with an input file specific error message.

Parameters
aTokis the token/keyword type which was not expected at the current input location.
Exceptions
IO_ERRORwith the location within the input file of the problem.

Definition at line 349 of file dsnlexer.cpp.

350 {
351  wxString errText = wxString::Format(
352  _( "Unexpected %s" ), GetTokenString( aTok ) );
354 }
int CurLineNumber() const
Return the current line number within my LINE_READER.
Definition: dsnlexer.h:423
wxString GetTokenString(int aTok) const
Return a quote wrapped wxString representation of a token value.
Definition: dsnlexer.cpp:314
#define THROW_PARSE_ERROR(aProblem, aSource, aInputLine, aLineNumber, aByteIndex)
Definition: ki_exception.h:164
#define _(s)
void Format(OUTPUTFORMATTER *out, int aNestLevel, int aCtl, const CPTREE &aTree)
Output a PTREE into s-expression format via an OUTPUTFORMATTER derivative.
Definition: ptree.cpp:200
int CurOffset() const
Return the byte offset within the current line, using a 1 based index.
Definition: dsnlexer.h:451
const char * CurLine() const
Return the current line of text from which the CurText() would return its token.
Definition: dsnlexer.h:431
const wxString & CurSource() const
Return the current LINE_READER source.
Definition: dsnlexer.h:441

References _, CurLine(), CurLineNumber(), CurOffset(), CurSource(), Format(), GetTokenString(), and THROW_PARSE_ERROR.

Referenced by Scan(), and scanList().

◆ Unexpected() [2/2]

void DSNLEXER::Unexpected ( const char *  aToken) const

Throw an IO_ERROR exception with an input file specific error message.

Parameters
aTokenis the token which was not expected at the current input location.
Exceptions
IO_ERRORwith the location within the input file of the problem.

Definition at line 365 of file dsnlexer.cpp.

366 {
367  wxString errText = wxString::Format(
368  _( "Unexpected '%s'" ), wxString::FromUTF8( text ) );
370 }
int CurLineNumber() const
Return the current line number within my LINE_READER.
Definition: dsnlexer.h:423
#define THROW_PARSE_ERROR(aProblem, aSource, aInputLine, aLineNumber, aByteIndex)
Definition: ki_exception.h:164
#define _(s)
void Format(OUTPUTFORMATTER *out, int aNestLevel, int aCtl, const CPTREE &aTree)
Output a PTREE into s-expression format via an OUTPUTFORMATTER derivative.
Definition: ptree.cpp:200
int CurOffset() const
Return the byte offset within the current line, using a 1 based index.
Definition: dsnlexer.h:451
const char * CurLine() const
Return the current line of text from which the CurText() would return its token.
Definition: dsnlexer.h:431
const wxString & CurSource() const
Return the current LINE_READER source.
Definition: dsnlexer.h:441

References _, CurLine(), CurLineNumber(), CurOffset(), CurSource(), Format(), text, and THROW_PARSE_ERROR.

Member Data Documentation

◆ commentsAreTokens

bool DSNLEXER::commentsAreTokens
protected

true if should return comments as tokens

Definition at line 523 of file dsnlexer.h.

Referenced by init(), InitParserState(), NextTok(), and SetCommentsAreTokens().

◆ curOffset

int DSNLEXER::curOffset
protected

offset within current line of the current token

Definition at line 526 of file dsnlexer.h.

Referenced by CurOffset(), init(), InitParserState(), NextTok(), and SyncLineReaderWith().

◆ curText

std::string DSNLEXER::curText
protected

the text of the current token

Definition at line 529 of file dsnlexer.h.

Referenced by CurStr(), CurText(), FromUTF8(), GetCurStrAsToken(), NextTok(), and SyncLineReaderWith().

◆ curTok

int DSNLEXER::curTok
protected

the current token obtained on last NextTok()

Definition at line 528 of file dsnlexer.h.

Referenced by CurTok(), init(), InitParserState(), and NextTok().

◆ dummy

char DSNLEXER::dummy[1]
protected

when there is no reader.

Definition at line 505 of file dsnlexer.h.

Referenced by PopReader().

◆ iOwnReaders

bool DSNLEXER::iOwnReaders
protected

on readerStack, should I delete them?

Definition at line 501 of file dsnlexer.h.

Referenced by ~DSNLEXER().

◆ keyword_hash

KEYWORD_MAP DSNLEXER::keyword_hash
protected

fast, specialized "C string" hashtable

Definition at line 533 of file dsnlexer.h.

Referenced by findToken(), and init().

◆ keywordCount

unsigned DSNLEXER::keywordCount
protected

count of keywords table

Definition at line 532 of file dsnlexer.h.

Referenced by GetTokenText(), and init().

◆ keywords

const KEYWORD* DSNLEXER::keywords
protected

table sorted by CMake for bsearch()

Definition at line 531 of file dsnlexer.h.

Referenced by GetTokenText(), and init().

◆ limit

const char* DSNLEXER::limit
protected

Definition at line 504 of file dsnlexer.h.

Referenced by NextTok(), PopReader(), PushReader(), readLine(), and SyncLineReaderWith().

◆ next

const char* DSNLEXER::next
protected

Definition at line 503 of file dsnlexer.h.

Referenced by NextTok(), PopReader(), PushReader(), readLine(), and SyncLineReaderWith().

◆ prevTok

int DSNLEXER::prevTok
protected

curTok from previous NextTok() call.

Definition at line 525 of file dsnlexer.h.

Referenced by init(), InitParserState(), NextTok(), and PrevTok().

◆ reader

LINE_READER* DSNLEXER::reader
protected

◆ readerStack

READER_STACK DSNLEXER::readerStack
protected

all the LINE_READERs by pointer.

no ownership. ownership is via readerStack, maybe, if iOwnReaders

Definition at line 509 of file dsnlexer.h.

Referenced by PopReader(), PushReader(), and ~DSNLEXER().

◆ space_in_quoted_tokens

bool DSNLEXER::space_in_quoted_tokens
protected

blank spaces within quoted strings

Definition at line 521 of file dsnlexer.h.

Referenced by init(), isStringTerminator(), SetSpaceInQuotedTokens(), and SetSpecctraMode().

◆ specctraMode

bool DSNLEXER::specctraMode
protected

if true, then: 1) stringDelimiter can be changed 2) Kicad quoting protocol is not in effect 3) space_in_quoted_tokens is functional else not.

Definition at line 514 of file dsnlexer.h.

Referenced by init(), NextTok(), SetSpaceInQuotedTokens(), SetSpecctraMode(), and SetStringDelimiter().

◆ start

const char* DSNLEXER::start
protected

Definition at line 502 of file dsnlexer.h.

Referenced by NextTok(), PopReader(), PushReader(), readLine(), and SyncLineReaderWith().

◆ stringDelimiter

char DSNLEXER::stringDelimiter
protected

Definition at line 520 of file dsnlexer.h.

Referenced by init(), isStringTerminator(), NextTok(), SetSpecctraMode(), and SetStringDelimiter().


The documentation for this class was generated from the following files: