/Python/How to add special token to bert tokenizer

How to add special token to bert tokenizer

special_tokens_dict = {'additional_special_tokens': ['[C1]','[C2]','[C3]','[C4]']}
num_added_toks = tokenizer.add_special_tokens(special_tokens_dict)
model.resize_token_embeddings(len(tokenizer))

Similar codes

21 votes166 views1 answer
How to plot a graph using matplotlib
Python
0 vote157 views1 answer
Python set workspace dir
Python
155 votes170 views6 answers
Python check if file exists
Python
0 vote161 views1 answer
Python create tuple from input
Python
49 votes178 views6 answers
Python run command
Python
0 vote156 views1 answer
Python length checker/fill
Python
0 vote185 views1 answer
Python save .mat
Python
1 vote169 views1 answer
Animations text terminal python
Python
2 votes167 views1 answer
Validate IP Address
Python
0 vote155 views1 answer
Run all python files in a directory in bash
Python