Python nltk tokenize

>>> import nltk
>>> sentence = """At eight o'clock on Thursday morning
... Arthur didn't feel very good."""
>>> tokens = nltk.word_tokenize(sentence)
>>> tokens
['At', 'eight', "o'clock", 'on', 'Thursday', 'morning',
'Arthur', 'did', "n't", 'feel', 'very', 'good', '.']

Source: stackoverflow.com

Import word_tokenize

import nltk
from nltk import word_tokenize

Similar codes

3 votes240 views2 answers
Django radio button
Python Django
0 vote250 views2 answers
Python replace matching string
Python
0 vote215 views1 answer
Django form is onvalid
Python Django
4 votes244 views3 answers
How to remove numbers from string in python dataframe
Python
0 vote239 views1 answer
Python if in list multiple
Python
0 vote243 views1 answer
Pyqt disable maximize button
Python
0 vote227 views1 answer
Dependency inversion
Python
1 vote219 views1 answer
How to filter queryset with foreign key in django
Python
1 vote250 views1 answer
How to append to every second item in list python
Python
2 votes235 views1 answer
Python api define bearer token
Python