Tokenization



Download the document from here. Use the tokenizer with ID the last digit of your student's registration number. Describe briefly the prons and cons of the tokenizer. Evaluate the tokenizer filling the table.
  1. Lucene English Analyzer
  2. Lucene Greek Analyzer
  3. NLTK tokenizer
  4. OpenNLP tokenizer
  5. LT TTT tokenizer
  6. Stanford POS tagger
  7. SPECIALIST NLP tokenizer
  8. MedPost tokenizer
  9. Brillís POS tagger
  10. weka