=================================================================================
BERT is a deep neural network trained to predict the probability of a word to appear in a given context, represented by a sequence of words. [2] BERT and other LMs can be used in different neural networks
as an embedding layer, i.e., a layer that represents sequences of
tokenized words by vectors. A trained LM usually maps words
that appear in similar contexts to vectors, which are similar with
respect to some measure, like cosine or Euclidian distance. This
vectorized representation is then provided to other layers of a
network, e.g., classification layers.
BERTScore:
i) Is the abbreviation of Bidirectional Encoder Representations from Transformer, which is a deep bidirectional, unsupervised language representation, pre-trained using a plain text corpus.
ii) Is an automatic evaluation metric used for testing the goodness of text generation systems.
iii) Focuses on computing semantic (contextual) similarity between the tokens of candidate sentence and the reference sentence.
As a result, the pre-trained BERT model can be fineturned with just an additional output layer to create state-of-the-art models for a wide range of tasks such as:
i)
Question answering.
ii)
Language inference.
Figure 2408 shows the illustration of the computation of the recall metric RBERT.
Figure 2408. Illustration of the computation of the recall metric RBERT. Given the reference and
candidate, we compute BERT embeddings and pairwise cosine similarity. We highlight the greedy
matching in red, and include the optional idf importance weighting. [1] |
Table 2408a lists some standard machine learning algorithms to choose.
Table 2408a. Some "standard" machine learning algorithms to choose.
| ML task |
Standard algorithms |
Description |
| Image classification |
ResNet (originally by Microsoft Research, and implementation open-sourced by Google) |
ResNet, which stands for Residual Network, is a type of convolutional neural network (CNN) that introduced the concept of "residual learning" to ease the training of networks that are substantially deeper than those used previously. This architecture has become a foundational model for many computer vision tasks. |
| Text classification |
FastText (open-sourced by Facebook Research) |
FastText is an algorithm that extends the Word2Vec model to consider subword information, making it especially effective for languages with rich morphology and for handling rare words in large corpora. It’s primarily used for text classification, benefiting from its speed and efficiency in training and prediction. |
| Text summarization |
Tansformer and BERT (open-sourced by Google) |
The Transformer model introduces an architecture that relies solely on attention mechanisms, dispensing with recurrence and convolutions entirely. BERT (Bidirectional Encoder Representations from Transformers) builds upon Transformer by pre-training on a large corpus of text and then fine-tuning for specific tasks. Both are effective for complex language understanding tasks, including summarization. |
| Image generation |
GANs or Conditional GANs |
GANs consist of two neural networks, a generator and a discriminator, which compete against each other, thus improving their capabilities. Conditional GANs extend this concept by conditioning the generation process on additional information, such as class labels or data from other modalities, allowing more control over the generated outputs. This methodology has been revolutionary in generating realistic images and other types of data. |
============================================
[1] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi, BERTScore Evaluating Text Generation with BERT, ICLR Conference, 2020.
[2] Corinna Grabner, Anna Safont-Andreu, Christian Burmer, Konstantin Schekotihin, A BERT-based Report Classification for Semiconductor Failure Analysis, ISTFATM 2022: Conference Proceedings from the 48th International, https://doi.org/10.31399/asm.cp.istfa2022p0028, Pasedena, California, USA.
|
