/Python/How to add special token to bert tokenizer

How to add special token to bert tokenizer

special_tokens_dict = {'additional_special_tokens': ['[C1]','[C2]','[C3]','[C4]']}
num_added_toks = tokenizer.add_special_tokens(special_tokens_dict)
model.resize_token_embeddings(len(tokenizer))

Similar codes

2 votes335 views1 answer
Python pandas cumulative return
Python
0 vote285 views2 answers
How to add space in python
Python
0 vote287 views1 answer
Django update field after save
Python
2 votes308 views2 answers
Python asterisk as generator
Python
3 votes303 views1 answer
Python get item from queue
Python
0 vote300 views1 answer
ModuleNotFoundError: No module named 'html5lib'
Python
1 vote326 views4 answers
Python 2.7 venv
Python
0 vote311 views1 answer
Merge multile dict
Python
0 vote513 views1 answer
Python devops modules
Python
1 vote313 views3 answers
Copy a list to another list python
Python