=================================================================================

The keras modules which are needed for Model.ipynb and required to quickly implement high-performance neural networks with the TensorFlow backend, are:
         import numpy as np
         import keras
         from keras.models import Sequential
         from keras.layers import Dense, Dropout, Flatten, Activation
         from keras.layers import Conv2D, MaxPooling2D
         from keras import backend as K
         from keras.layers.normalization import BatchNormalization

The tf.keras.layers.normalization(axis = -1, mean = None, variance = None, **kwargs) is used for feature-wise normalization of the data (input features).

"from sklearn.model_selection import train_test_split" can be used used to split a dataset into training and testing sets (refer to page4312). In addition, to set up a neural network using TensorFlow's Keras API for binary classification, we need:

1. Create a Sequential model:

   model = tf.keras.models.Sequential()  

   The Sequential model in TensorFlow's Keras API is a linear stack of layers that allows for easy building and training of neural networks. It represents a simple linear stack of layers where the output of one layer is used as the input to the next. This is suitable for a plain stack of layers, where each layer has exactly one input tensor and one output tensor.

2. Add the first hidden layer with 8 units and ReLU activation: 

   model.add(tf.keras.layers.Dense(8, input_shape=(4,), activation="relu"))  

   We can add layers to the model using the add method. Each layer has specific configuration parameters like the number of units, activation function, input shape, etc. This layer has 8 units, and the input_shape=(4,) specifies that the input data has 4 features. 

3. Add the output layer with 1 unit and sigmoid activation (for binary classification):

   model.add(tf.keras.layers.Dense(1, activation="sigmoid")) 

In addition to the "sigmoid" activation function, there are several other activation functions we can choose from. Some commonly used activation functions include: 

• ReLU (Rectified Linear Unit): 

  activation="relu" 

• TanH (Hyperbolic Tangent):

  activation="tanh" 

• Softmax: This activation function is often used in the output layer for multi-class classification problems. It converts the raw scores (logits) into probabilities.

  activation="softmax"

• Linear (Identity): This is a simple identity activation function, often used in regression problems.

  activation="linear"

• Leaky ReLU: An alternative to ReLU that allows a small, positive gradient when the input is negative, which can help with training deep neural networks.

  activation=tf.keras.layers.LeakyReLU(alpha=0.01)

• Exponential Linear Unit (ELU): Another variant of ReLU that aims to mitigate the vanishing gradient problem.

  activation=tf.keras.layers.ELU(alpha=1.0)

• Parametric ReLU (PReLU): Similar to Leaky ReLU, but with the slope parameter learned during training.

  activation=tf.keras.layers.PReLU()

4. Compile the model:

   model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"]) 

   Here,

   optimizer="adam": Adam is an optimization algorithm. 

   loss="binary_crossentropy": This is the appropriate loss function for binary classification problems. 

   metrics=["accuracy"]: The model will be evaluated based on accuracy during training. 

   After adding the layers, we need to compile the model. This involves specifying the optimizer, loss function, and metrics to be used during training. 

5. Training: 

   Once compiled, you can train the model using the fit method on your training data. 

   model.fit(X_train, y_train, epochs=10, batch_size=32, validation_data=(X_val, y_val)) 

6. Evaluation and Prediction: 

   After training, you can evaluate the model on new data or make predictions using the evaluate and predict methods, respectively. 

   loss, accuracy = model.evaluate(X_test, y_test) 

   predictions = model.predict(X_new_data)  

Discretization, normalization and StringLookup are trainable layers. Regularization can help build generalizable models by adding dropout layers to neural networks.

An example of deep neural networks can be:
          (X_train, y_train) , (X_test, y_test) = keras.datasets.mnist.load_data()
          # Define a model
          myModel = tf.keras.models.Sequential([
          tf.keras.layers.Flatten(),
          tf.keras.layers.Dense(128, activation = "relu"),
          tf.keras.layers.Dense(128, activation = "relu"),
          tf.keras.layers.Dense(128, activation = "relu"),
          tf.keras.layers.Dense(10, activation = "softmax")
          ])

The learners can use TensorFlow Playground to understand the basics of neural networks and how different parameters and architectures can affect their performance.  Table 4132a lists some examples of case studies of classification, and Table 4132b lists some outputs of different cases listed in Table 4132a. In this example, we try to learn the decision boundary in the particular form shown in Case 1, where shows the original data with two classes (orange points and blue points), and then to separate the two classes with machine learning. In Case 2, we are not able to separate the two classes with an 10k epoch and 2 features (X₁ and X₂).  The two features are colored in Table 4132b.

To improve the learning, in Case 3, 1 hidden layer has been added, then we can do the learning slightly better: one area with orange points is marked in orange, and one area with blue points is marked in blue.  The outputs of the two features are shown in the column named as Neural networks in Table 4132b. That is, each neuron is learning its own boundaries, for instance, shown in Figure 4132a.  

 Figure 4132a shows the separation boundaries learnt by Neuron 1 and Neural 2, respectively.

(a)	(b)

Figure 4132a. Separation boundaries learnt by Neuron 1 (a) and Neural 2 (b). 

In Case 4, we added another neuron so that we have 3 neuron in total. Then, the separation between the two classes is much better. That is, only by adding more neurons, the separation boundaries can be much more accuracte, which is also shown in Case 5.       

Table 4132a. Examples of case studies of classification. 

Table 4132b. Outputs of different cases listed in Table 4132a.  

Case	Neural networks	Output
1	None
2
3
4
5

============================================