/Python/How to add special token to bert tokenizer

How to add special token to bert tokenizer

special_tokens_dict = {'additional_special_tokens': ['[C1]','[C2]','[C3]','[C4]']}
num_added_toks = tokenizer.add_special_tokens(special_tokens_dict)
model.resize_token_embeddings(len(tokenizer))

Similar codes

8 votes233 views2 answers
How do I set Conda to activate the base environment by default?
Python
0 vote216 views1 answer
How to call the tkinter insert command from another class
Python
10 votes234 views2 answers
Python get absolute path of file
Python
0 vote239 views1 answer
Django cms api
Python
0 vote208 views10 answers
Convert c code to python online free
Python
7 votes259 views3 answers
Linux uninstall python
Python
9 votes225 views2 answers
How to connecct id's in the models of jdango
Python
0 vote246 views1 answer
Export a Pandas dataframe as a table image
Python
0 vote258 views1 answer
Django swagger settings https
Python Django
1 vote229 views1 answer
2d array pytho
Python