/Python/How to add special token to bert tokenizer

How to add special token to bert tokenizer

special_tokens_dict = {'additional_special_tokens': ['[C1]','[C2]','[C3]','[C4]']}
num_added_toks = tokenizer.add_special_tokens(special_tokens_dict)
model.resize_token_embeddings(len(tokenizer))

Similar codes

5 votes252 views3 answers
How to print multiple lines in one line python
Python
0 vote244 views1 answer
Access dynamicall to name attribute python
Python
1 vote244 views1 answer
Python docker api example
Python
12 votes249 views4 answers
How to know if a input is a interger in python
Python
89 votes196 views6 answers
Int to string python
Python
1 vote246 views1 answer
Python convert to percentage
Python
8 votes234 views2 answers
Python count files directory
Python
6 votes231 views1 answer
Pip freeze requirements.txt no weird path
Python
28 votes250 views4 answers
Pygame scale image
Python
6 votes294 views3 answers
Cosine similarity python
Python