[Learn about machine learning from the Keras] - 1. Dense Layer operation process
Keras provides several Layer classes for use, but let’s first look at how the most commonly used classes work initially, and then look at other applications by analogy.
Let’s take a look at the example first:
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
model = Sequential()
model.add(Dense(units=64, activation='relu'))
model.add(Dense(units=10, activation='softmax'))
model.compile(loss='categorical_crossentropy',
optimizer='sgd',
metrics=['accuracy'])
model.build(input_shape=(None, 3))In this way, a neural network can be initially constructed.
In this section, we need to first understand the layer and what it did at the beginning.
The class “keras.engine.base_layer “ plays an important role in building the model, that is, the model is composed of these classes.
When new object entities are added at the beginning, the focus is on the setting part. The constructor containing the parent class is also setting related properties. Getting to know each other here is like meeting a customer first and then having an in-depth chat.
These settings will affect how subsequent training will work. Therefore, the initial operation of constructing class entities focuses on parameter setting. In terms of sequence, just know the operation first. At the beginning, you can directly look at what each parameter does. In fact, it is impossible to understand it. If you look at it at the beginning, you have not looked at it. So just understand the operation and purpose first, and you will repeat the steps later. It is more efficient to look back and find out what these settings are supposed to do.
The key points of operation highlighted here are:
keras.layers.core.dense inherits keras.engine.base_layer.Layer. For Dense Layer, take a look at what its class content looks like:
First, go to the constructor method __new__ of the parent class “keras.engine.base_layer.Layer” of keras.layers.core.dense.
The constructor mainly takes out the parameter names of the init method of keras.layers.core.dense:
[‘self’, ‘units’, ‘activation’, ‘use_bias’, ‘kernel_initializer’, ‘bias_initializer’, ‘kernel_regularizer’, ‘bias_regularizer’, ‘activity_regularizer’, ‘kernel_constraint’, ‘bias_constraint’]
These parameters are composed into a dictionary corresponding to the actual passed values, and updated into kwargs for use in subsequent initialization.
In this case, layers.Dense(units=64, activation=’relu’), the parameter value 64 will correspond to ‘units’, and {‘units’: 64} will be updated to **kwargs to become kwargs = {‘activation ‘: ‘relu’, ‘units’: 64} .
Next, execute the init method of keras.layers.core.dense. This method is decorated with @utils.allow_initializer_layout, which will be described later. In this method, the following fields will be initialized:
(1) keras.layers.core.dense.Dense.units = int(units):
is the dimension value of the output tensor. The parameter value in the sample program is 64.
(2) keras.layers.core.dense.Dense.activation = activations.get({activation_name}):
Set activation function.
{activation_name} is the activation=”relu” specified by the parameter, which will be passed through the keras.activations
The activations.get method in the activations.get method obtains the function body from the keras.layers.activation module defined in this module.
The callable object is obtained. For this example, activation=”relu” will be searched in keras.activations. finally revealed
The keras.saving.serialization_lib.deserialize_keras_object function will correspond to types.FunctionType
The relu function body is returned.
(3) keras.layers.core.dense.Dense.use_bias = True:
Whether to use bias vector, default True.
(4) keras.layers.core.dense.Dense.kernel_initializer = initializers.get({kernel_initializer}):
The keras file states that this parameter is: Initializer for the kernel weights matrix.
Get the instance through keras.initializers.get. keras.initializers.get passes in presets via
{kernel_initializer}=’glorot_uniform’ string, and then get the
The instance object produced by keras.saving.legacy.serialization.deserialize_keras_object. Will come back here
Pass keras.initializers.initializers.GlorotUniform.
(If it is not a legacy module, it will be determined by the keras.saving.serialization_lib.deserialize_keras_object function.
Returns an instance object, similar to how the activations.get function is implemented. )
(5) keras.layers.core.dense.Dense.bias_initializer = initializers.get({bias_initializer}):
The keras file states that this parameter is: Initializer for the bias vector.
Get the instance through keras.initializers.get. Because {bias_initializer} = None is defaulted, no parameters are specified.
If so, None will be returned.
(6) keras.layers.core.dense.Dense.kernel_regularizer = regularizers.get({kernel_regularizer}):
The keras file states that this parameter is: Regularizer function applied to the kernel weights matrix.
Get the instance through keras.initializers.get. Because {kernel_regularizer} = None is defaulted, no parameters are specified.
If so, None will be returned.
(7) keras.layers.core.dense.Dense.bias_regularizer = regularizers.get({bias_regularizer}):
The keras file states that this parameter is: Regularizer function applied to the bias vector.
Get the instance through keras.initializers.get. Because {bias_regularizer} = None is defaulted, no parameters are specified.
The response is None.
(8) keras.layers.core.dense.Dense.kernel_constraint = constraints.get({kernel_constraint}):
The keras file states that this parameter is: Constraint function applied to the kernel weights matrix.
Get the instance through keras.initializers.get. Because {kernel_constraint} = None is defaulted, no parameters are specified.
If so, None will be returned.
(9) keras.layers.core.dense.Dense.bias_constraint = constraints.get({bias_constraint}):
The keras file states that this parameter is: Constraint function applied to the bias vector.
Get the instance through keras.initializers.get. Since {bias_constraint} = None is defaulted, if no parameters are specified
It also returns None.
(10)keras.layers.core.dense.Dense.input_spec = InputSpec(min_ndim=2):
Specify the rank, dtype, and shape of each input tensor via the keras.engine.input_spec class.
(11)keras.layers.core.dense.Dense.supports_masking = True:
Set the mechanism to ignore sequence data when it is lost.
After the init method of keras.layers.core.dense is executed, because the @utils.allow_initializer_layout modifier decorates this __init__ method, the content of utils.allow_initializer_layout will be executed:
Check whether the parameters of keras.layers.core.dense match
[
“alpha_initializer”,”beta_initializer”,”bias_initializer”,”depthwise_initializer”,”embeddings_initializer”,”gamma_initializer”,”kernel_initializer”,”moving_mean_initializer”,”moving_variance_initializer”,”pointwise_initializer”,”recurrent_initializer”,
]
Among the parameter names, if they match and there is an instance object, directly set the corresponding name attribute (if not, add an attribute) to the keras.layers.core.dense instance.
After each keras.layers.core.dense is constructed through the build function, the model will call the call function one by one to perform related operations.
The above is a record of the initialization process using the keras.layers.core.dense class when constructing the model, and the content used to combine the model in the future.
