konlp.tokenize package

Submodules

konlp.tokenize.api module

Korean Natural Language Toolkit tonkenizer interface

class konlp.tokenize.api.SimpleTokenizer[source]

Bases: konlp.tokenize.api.TokenizerI

For an example about how to inherit the class above

tokenize(string)[source]

Simple string tokenizer by white-space character

Parameters:string (str) – String to tokenize
Returns:Tokenized tokens
Return type:str
class konlp.tokenize.api.TokenizerI[source]

Bases: object

Tokenizer Interface

tokenize(string)[source]

Return a tokenized copy of string.

Parameters:string (str) – String to tokenize
Returns:Tokenized tokens
Return type:list(str)
Raises:NotImplementedError – If not implement this method on a class that extends this class

Module contents