=================================================================================

PyTorch is introduced by Facebook in 2017. It is a library for Python programs that facilitates building deep learning projects, which gives the user a blend of 2 high-level features – Tensor computation (like NumPy) with strong GPU acceleration and developing Deep Neural Networks on a tape-based auto diff system. PyTorch provides a great platform to execute Deep Learning models with increased flexibility and speed built to be integrated deeply with Python. In fact, PyTorch is a deep learning library based on Torch and developed by Facebook. It is relatively easy to use, and thus has recently gained a lot of popularity. It will automatically select a GPU, if one is available, reverting to the CPU otherwise.

Example of training and testing of the model, based on WM-811K wafer dataset, were carried out on a DELL T7920 workstation (Round Rock, Texas, USA). [1] The main hardware configuration was twoGeForce RTX 2080TI graphics cards and a 64 GB memory. The software environment isUbuntu 18.04 and was implemented based on the PyTorch deep learning framework. The cross-entropy loss was used for model training, and the initial learning rate was set at 0.01, which was reduced by a factor of ten when the number of iterations reached half of the total number. In the representation learning stage, ResNet-18 integrated with the improved CBAM algorithm was trained for 100 epochs. In the classifier fine-tuning stage, the model based on the previous stage was fine-tuned for 25 epochs at a learning rate of 0.001.

PyTorch is a popular deep learning framework that provides a flexible and efficient platform for implementing various self-attention mechanisms, including those mentioned above such as scaled dot-product attention, multi-head attention, additive attention, relative positional encoding, and causal attention. PyTorch offers a dynamic computation graph, which is particularly beneficial when working with attention mechanisms and variable-length sequences. We can define and customize our attention modules using PyTorch's tensor operations and automatic differentiation capabilities.  

============================================

[1] Qiao Xu, Naigong Yu and Firdaous Essaf, Improved Wafer Map Inspection Using Attention Mechanism and Cosine Normalization, Machines, https://doi.org/10.3390/machines10020146, 10, 146, 2022.