package info (click to toggle)
tokenizers 0.20.3%2Bdfsg-1
  • links: PTS, VCS
  • area: main
  • in suites: experimental
  • size: 5,480 kB
  • sloc: python: 4,499; javascript: 419; makefile: 124

Folder: implementations

d .. (parent)
- - rw-r--r-- 0 __init__.py
- - rw-r--r-- 1,456 test_base_tokenizer.py
- - rw-r--r-- 1,905 test_bert_wordpiece.py
- - rw-r--r-- 3,039 test_byte_level_bpe.py
- - rw-r--r-- 2,301 test_char_bpe.py
- - rw-r--r-- 2,664 test_sentencepiece.py