Data Science
Predicting The Next Word using “BERT”
An open-source machine learning framework for natural language processing (NLP)
Welcome Learners! In this tutorial, I shall show you how to make a web app that can Predict the next word using pretrained state of art NLP model BERT.
If you don’t know what is BERT then you can have a look here. The motive of this tutorial is to create a web app not explain BERT. So it is strongly recommended to take a look at the given link.
This project has been developed using Pytorch and Streamlit. So without wasting time let’s move on.
Step 1) Load Model and Tokenizer
First of all Load BERT model. Here I have used uncased bert which means it is case-insensitive and the point to remember is that model will be in eval mode because we are not training model but using pretrained model. Also Load Bert tokenizer that tokenizes the text.
Step 2) Add mask at the end
As we want to predict the last word in given text so it is required to add a mask token at the end of input text becasue BERT requires input to be preprocessed in this way.
Step 3) Encode and Decode
In this step input text is encoded with bert tokenizer . Here I have used add_special_tokens = True because I want to encode out-of-vocabulary words aka UNK with special token that BERT uses. Then, when tokenizer encodes the input text it returns input_ids . After that get mask index (mask_idx) that is the place where mask has been added.
When input_ids is returned by tokenizer then that is passed into bert model and after that model predicts new words in encoded format (shown in step 4).
So now last step is to decode it and then select top_k words via removing some tokens that are not required as they are not valid words and get readable words from there.
Step 4) Wrapper function for encoder and decoder
Both of the above mentioned functions have been wrapped in the following function.
Steps 5) Interface
Here is some extra code that has been used for developing streamlit webapp interface.
Here is Github link for this project. Feel free to fork there.
