Regression Using MLP

MLPs can be used for regression tasks. If you want to predict a single value (e.g., the price of a house, given many of its features), then you just need a single output neuron: its output is the predicted value. For multivariate regression (i.e., to predict multiple values at once), you need one output neuron per output dimension. 

For example, to locate the center of an object in an image, you need to predict 2D coordinates, so you need two output neurons. If you also want to place a bounding box around the object, then you need two more numbers: the width and the height of the object. So, you end up with four output neurons.

In general, when building an MLP for regression, you do not want to use any activation function for the output neurons, so they are free to output any range of values. If you want to guarantee that the output will always be positive, then you can use the ReLU activation function in the output layer. Alternatively, you can use the softplus activation function, which is a smooth variant of ReLU: softplus(z) = log(1 + exp(z)).It is close to 0 when z is negative, and close to z when z is positive. Finally, if you want to guarantee that the predictions will fall within a given range of values, then you can use the logistic function or the hyperbolic tangent, and then scale the labels to the appropriate range: 0 to 1 for the logistic function and –1 to 1 for the hyperbolic tangent.

The loss function to use during training is typically the mean squared error, but if you have a lot of outliers in the training set, you may prefer to use the mean absolute error instead. Alternatively, you can use the Huber loss, which is a combination of both.

Example:Multi-Layer Perceptron (MLP) for regression tasks. 

First, a random dataset will be generated using NumPy to observe how the MLP learns non-linear patterns. 

Building dataset using NumPy

from sklearn.datasets import make_regression
import numpy as np
import matplotlib.pyplot as plt

X_train = np.linspace(-10, 10, 1000)
y_train = np.sin(X_train) + np.random.normal(0, 0.2, size=X_train.shape)

X_test = np.linspace(-10, 10, 500)
y_test = np.sin(X_test) + np.random.normal(0, 0.2, size=X_test.shape)

Now, if you plot the dataset, you'll see something like this,

plt.scatter(X_train, y_train)
plt.show()

Creating and Training MLP Model

Let's create our Neural Network using  the TensorFlow version of Keras to create the network. You can use the standard Keras version without any problem.

import tensorflow as tf

model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Dense(62, activation='relu', input_dim=1))
model.add(tf.keras.layers.Dense(62, activation='relu'))
model.add(tf.keras.layers.Dense(1, activation='linear'))

We created a model which has 62 neurons with input dimension 1 .The  second hidden layers have 62 neurons each. Well, it is completely up to you to select the number of neurons. Here I selected 62 because it performs well. now let's compile & train the model.Final output layer has 1 neuron.

model.compile(loss='mse', optimizer='adam', metrics=['mae'])
history = model.fit(X_train, y_train, epochs=500)

Plotting the Regression Curve
Here is how you plot the Regression Curve formed when training the model,

plt.scatter(X_train, y_train)
plt.plot(X_train, model.predict(X_train), color='red')
plt.show()

The red line curve in the graph is known as the regression curve formed by our MLP. The network actually finds the wavey pattern of the dataset, and this is how basically regression works with MLP. Now, let's plot the testing data as well

pred = model.predict(X_test)

plt.scatter(X_test, y_test)
plt.plot(X_test, pred, color='red')
plt.show()

Plotting is a great way to evaluate the performance of the model, but it alone doesn't give a good idea about the efficiency of regression models, so, it is good the check the MSE on testing the model.

from sklearn.metrics import mean_squared_error
pred = model.predict(X_test)
mean_squared_error(pred, y_test)

---------

0.04593161449409425