Network Architectures

Neural network architectures are foundational frameworks designed to tackle diverse problems in artificial intelligence and machine learning. Each architecture is structured to optimize learning and performance for specific types of data and tasks, ranging from simple classification problems to complex sequence generation challenges. This guide explores the various architectures employed in neural networks, providing insights into how they are constructed, their applications, and why certain architectures are preferred for particular tasks.

The architecture of a neural network dictates how information flows and is processed. It determines the arrangement and connectivity of layers, the type of data processing that occurs, and how input data is ultimately transformed into outputs. The choice of a suitable architecture is crucial because it impacts the efficiency, accuracy, and feasibility of training models on given datasets.

Neural Network Architectures & Deep Learning

Feedforward Neural Networks (FNNs)

A basic neural network architecture where data flows only in one direction, from input layer to output layer, without any feedback loops. Feedforward Neural Networks are the simplest type of neural network architecture where connections between the nodes do not form a cycle. This is ideal for problems where the output is directly mapped from the input.

  • Usage: Image classification, regression, function approximation
  • Strengths: Simple to implement, computationally efficient
  • Caveats: Limited capacity to model complex relationships, prone to overfitting

FNN input_layer Input Layer hidden_layer1 Hidden Layer 1 input_layer->hidden_layer1 hidden_layer2 Hidden Layer 2 hidden_layer1->hidden_layer2 output_layer Output Layer hidden_layer2->output_layer

Input Layer: This layer represents the initial data that is fed into the network. Each node in this layer typically corresponds to a feature in the input dataset.

Hidden Layers: These are intermediary layers between the input and output layers. Hidden layers allow the network to learn complex patterns in the data. They are called “hidden” because they are not directly exposed to the input or output.

Output Layer: The final layer that produces the network’s predictions. The function of this layer can vary depending on the specific application — for example, it might use a softmax activation function for classification tasks or a linear activation for regression tasks.

Edges: Represent the connections between neurons in consecutive layers. In feedforward networks, every neuron in one layer connects to every neuron in the next layer. These connections are weighted, and these weights are adjusted during training to minimize error.

Convolutional Neural Networks (CNNs)

A neural network architecture that uses convolutional and pooling layers to extract features from images. CNNs are highly effective at processing data that has a grid-like topology, such as images, due to their ability to exploit spatial hierarchies and structures within the data.

  • Usage: Image classification, object detection, image segmentation
  • Strengths: Excellent performance on image-related tasks, robust to image transformations
  • Caveats: Computationally expensive, require large datasets

CNN input_image Input Image conv1 Convolution Layer 1 ReLU input_image->conv1 pool1 Pooling Layer 1 conv1->pool1 conv2 Convolution Layer 2 ReLU pool1->conv2 pool2 Pooling Layer 2 conv2->pool2 fully_connected Fully Connected Layer pool2->fully_connected output Output (Classification) fully_connected->output

Input Image: The initial input where images are fed into the network.

Convolution Layer 1 and 2: These layers apply a set of filters to the input image to create feature maps. These filters are designed to detect spatial hierarchies such as edges, colors, gradients, and more complex patterns as the network deepens. Each convolution layer is typically followed by a non-linear activation function like ReLU (Rectified Linear Unit).

Pooling Layer 1 and 2: These layers reduce the spatial size of the feature maps to decrease the amount of computation and weights in the network. Pooling (often max pooling) helps make the detection of features invariant to scale and orientation changes.

Fully Connected Layer: This layer takes the flattened output of the last pooling layer and performs classification based on the features extracted by the convolutional and pooling layers.

Output: The final output layer, which classifies the input image into categories based on the training dataset.

Recurrent Neural Networks (RNNs)

A neural network architecture that uses feedback connections to model sequential data. RNNs are capable of processing sequences of data by maintaining a state that acts as a memory. They are particularly useful for applications where the context or sequence of data points is important.

  • Usage: Natural Language Processing (NLP), sequence prediction, time series forecasting
  • Strengths: Excellent performance on sequential data, can model long-term dependencies
  • Caveats: Suffer from vanishing gradients, difficult to train

RNN input_seq Input Sequence rnn_cell RNN Cell input_seq->rnn_cell hidden_state Hidden State rnn_cell->hidden_state Next output_seq Output Sequence rnn_cell->output_seq rnn_cell->output_seq Each timestep hidden_state->rnn_cell Recurrence

Input Sequence: Represents the sequence of data being fed into the RNN, such as a sentence or time-series data.

RNN Cell: This is the core of an RNN, where the computation happens. It takes input from the current element of the sequence and combines it with the hidden state from the previous element of the sequence.

Hidden State: This node represents the memory of the network, carrying information from one element of the sequence to the next. The hidden state is updated continuously as the sequence is processed.

Output Sequence: The RNN can produce an output at each timestep, depending on the task. For example, in sequence labeling, there might be an output corresponding to each input timestep.

Long Short-Term Memory (LSTM) Networks

A type of RNN that uses memory cells to learn long-term dependencies. LSTM networks are designed to avoid the long-term dependency problem, making them effective at tasks where the context can extend over longer sequences.

  • Usage: NLP, sequence prediction, time series forecasting
  • Strengths: Excellent performance on sequential data, can model long-term dependencies
  • Caveats: Computationally expensive, require large datasets

LSTM input_seq Input Sequence lstm_cell LSTM Cell input_seq->lstm_cell cell_state Cell State lstm_cell->cell_state Update hidden_state Hidden State lstm_cell->hidden_state Update output_seq Output Sequence lstm_cell->output_seq lstm_cell->output_seq Each timestep cell_state->lstm_cell Recurrence hidden_state->lstm_cell Recurrence

Input Sequence: Represents the sequential data input, such as a series of words or time-series data points.

LSTM Cell: The core unit in an LSTM network that processes input data one element at a time. It interacts intricately with both the cell state and the hidden state to manage and preserve information over long periods.

Cell State: A “long-term” memory component of the LSTM cell. It carries relevant information throughout the processing of the sequence, with the ability to add or remove information via gates (not explicitly shown here).

Hidden State: A “short-term” memory component that also transfers information to the next time step but is more sensitive and responsive to recent inputs than the cell state.

Output Sequence: Depending on the task, LSTMs can output at each timestep (for tasks like sequence labeling) or after processing the entire sequence (like sentiment analysis).

Transformers

A neural network architecture that uses self-attention mechanisms to model relationships between input sequences. Transformers are particularly effective in NLP tasks due to their ability to handle sequences in parallel and consider all parts of the input at once.

  • Usage: NLP, machine translation, language modeling
  • Strengths: Excellent performance on sequential data, parallelizable, can handle long-range dependencies
  • Caveats: Computationally expensive, require large datasets

Transformer input_tokens Input Tokens embedding_layer Embedding Layer input_tokens->embedding_layer positional_encoding Add Positional Encoding embedding_layer->positional_encoding encoder Encoder Stack positional_encoding->encoder decoder Decoder Stack encoder->decoder encoder_output Encoder Output encoder->encoder_output self_attention Self-Attention encoder->self_attention output_tokens Output Tokens decoder->output_tokens decoder->self_attention cross_attention Cross-Attention decoder->cross_attention decoder_input Decoder Input decoder_input->decoder cross_attention->encoder_output

Input Tokens: Represents the initial sequence of tokens (e.g., words in a sentence) that are fed into the Transformer.

Embedding Layer: Converts tokens into vectors that the model can process. Each token is mapped to a unique vector.

Positional Encoding: Adds information about the position of each token in the sequence to the embeddings, which is crucial as Transformers do not inherently process sequential data.

Encoder Stack: A series of encoder layers that process the input. Each layer uses self-attention mechanisms to consider all parts of the input simultaneously.

Decoder Stack: A series of decoder layers that generate the output sequence step by step. Each layer uses both self-attention mechanisms to attend to its own output so far, and cross-attention mechanisms to focus on the output from the encoder.

Output Tokens: The final output sequence generated by the Transformer, such as a translated sentence or the continuation of an input text.

Encoder Output and Decoder Input: Not actual data flow, but illustrate how information is transferred from the encoder to the decoder.

Self-Attention and Cross-Attention: These mechanisms are core features of Transformer models. Self-attention allows layers to consider other parts of the input or output at each step, while cross-attention allows the decoder to focus on relevant parts of the input sequence.

Autoencoders

A neural network architecture that learns to compress and reconstruct input data. Autoencoders are typically used for dimensionality reduction tasks, as they learn to encode the essential aspects of the data in a smaller representation.

  • Usage: Dimensionality reduction, anomaly detection, generative modeling
  • Strengths: Excellent performance on dimensionality reduction, can learn robust representations
  • Caveats: May not perform well on complex data distributions

Autoencoder input_data Input Data encoder Encoder input_data->encoder latent_space Latent Space encoder->latent_space decoder Decoder latent_space->decoder reconstructed_output Reconstructed Output decoder->reconstructed_output

Input Data: Represents the data that is fed into the Autoencoder. This could be any kind of data, such as images, text, or sound.

Encoder: The first part of the Autoencoder that processes the input data and compresses it into a smaller, dense representation. This part typically consists of several layers that gradually reduce the dimensionality of the input.

Latent Space: Also known as the “encoded” state or “bottleneck”. This is a lower-dimensional representation of the input data and serves as the compressed “code” that the decoder will use to reconstruct the input.

Decoder: Mirrors the structure of the encoder but in reverse. It takes the encoded data from the latent space and reconstructs the original data as closely as possible. This part typically consists of layers that gradually increase in dimensionality to match the original input size.

Reconstructed Output: The final output of the Autoencoder. This is the reconstruction of the original input data based on the compressed code stored in the latent space. The quality of this reconstruction is often a measure of the Autoencoder’s performance.

Generative Adversarial Networks (GANs)

A neural network architecture that consists of a generator and discriminator, which compete to generate realistic data. GANs are highly effective at generating new data that mimics the input data, often used in image generation and editing.

  • Usage: Generative modeling, data augmentation, style transfer
  • Strengths: Excellent performance on generative tasks, can generate realistic data
  • Caveats: Training can be unstable, require careful tuning of hyperparameters

GAN noise Noise vector (z) generator Generator (G) noise->generator generated_image Generated image (G(z)) generator->generated_image discriminator Discriminator (D) generated_image->discriminator Fake real_image Real image (x) real_image->discriminator Real D_output_fake D(G(z)) discriminator->D_output_fake Output for fake D_output_real D(x) discriminator->D_output_real Output for real

Noise vector (z): Represents the random noise input to the generator.

Generator (G): The model that learns to generate new data with the same statistics as the training set from the noise vector.

Generated image (G(z)): The fake data produced by the generator.

Real image (x): Actual data samples from the training dataset.

Discriminator (D): The model that learns to distinguish between real data and synthetic data generated by the Generator.

D(G(z)) and D(x): Outputs of the Discriminator when evaluating fake data and real data, respectively.

The Noise vector feeds into the Generator.

The Generator outputs a Generated image, which is input to the Discriminator labeled as “Fake”.

The Real image also feeds into the Discriminator but is labeled as “Real”.

The Discriminator outputs evaluations for both fake and real inputs.

Residual Networks (ResNets)

A neural network architecture that uses residual connections to ease training. ResNets are particularly effective for very deep networks, as they allow for training deeper networks by providing pathways for gradients to flow through.

  • Usage: Image classification, object detection
  • Strengths: Excellent performance on image-related tasks, ease of training
  • Caveats: May not perform well on sequential data

ResNet input Input Image conv1 Initial Conv + BN + ReLU input->conv1 resblock1 ResBlock + ReLU conv1->resblock1:f0 conv1->resblock1:f1 resblock2 ResBlock + ReLU resblock1:f2->resblock2:f0 resblock1:f1->resblock2:f1 resblock3 ResBlock + ReLU resblock2:f2->resblock3:f0 resblock2:f1->resblock3:f1 avgpool Average Pooling resblock3:f2->avgpool fc Fully Connected Layer avgpool->fc output Output fc->output

Input Image: The initial input layer where images are fed into the network.

Initial Conv + BN + ReLU: Represents an initial convolutional layer followed by batch normalization and a ReLU activation function to prepare the data for residual blocks.

ResBlock: These are the residual blocks that define the ResNet architecture. Each block contains two parts: a sequence of convolutional layers and a skip connection that adds the input of the block to its output.

Average Pooling: This layer averages the feature maps spatially to reduce their dimensions before passing to a fully connected layer.

Fully Connected Layer: This layer maps the feature representations to the final output classes.

Output: The final prediction of the network.

U-Net

A neural network architecture that uses an encoder-decoder structure with skip connections. U-Net is designed primarily for biomedical image segmentation, where it is crucial to localize objects precisely within an image.

  • Usage: Image segmentation, object detection
  • Strengths: Excellent performance on image segmentation tasks, fast training
  • Caveats: May not perform well on sequential data

UNet input Input Image conv1 Conv + ReLU Downsampling input->conv1 conv2 Conv + ReLU Downsampling conv1->conv2 concat1 Concatenate conv1->concat1 Copy crop bottom Conv + ReLU conv2->bottom concat2 Concatenate conv2->concat2 Copy crop upconv1 UpConv + ReLU Upsampling bottom->upconv1 upconv1->concat1 upconv2 UpConv + ReLU Upsampling concat1->upconv2 upconv2->concat2 finalconv Conv + ReLU 1x1 Conv concat2->finalconv output Output Segmentation Map finalconv->output

Input Image: The initial input layer where images are fed into the network.

Conv + ReLU / Downsampling: These blocks represent convolutional operations followed by a ReLU activation function. The “Downsampling” indicates that each block reduces the spatial dimensions of the input.

Bottom: This is the lowest part of the U, consisting of convolutional layers without downsampling, positioned before the upsampling starts.

UpConv + ReLU / Upsampling: These blocks perform transposed convolutions (or up-convolutions) that increase the resolution of the feature maps.

Concatenate: These layers concatenate feature maps from the downsampling pathway with the upsampled feature maps to preserve high-resolution features for precise localization.

Final Conv: This typically includes a 1x1 convolution to map the deep feature representations to the desired number of classes for segmentation.

Output / Segmentation Map: The final output layer which produces the segmented image.

Attention-based Models

A neural network architecture that uses attention mechanisms to focus on relevant input regions. Attention-based models are particularly effective for tasks that require understanding of complex relationships within the data, such as interpreting a document or translating a sentence.

  • Usage: NLP, machine translation, question answering
  • Strengths: Excellent performance on sequential data, can model long-range dependencies
  • Caveats: Require careful tuning of hyperparameters

AttentionBasedModels input Input Sequence embedding Embedding Layer input->embedding positional Add Positional Encoding embedding->positional multihead Multi-Head Attention positional->multihead addnorm1 Add & Norm positional->addnorm1 Skip Connection multihead->addnorm1 feedforward Feedforward Network addnorm1->feedforward addnorm2 Add & Norm addnorm1->addnorm2 Skip Connection feedforward->addnorm2 output Output Sequence addnorm2->output

Input Sequence: Initial data input, typically a sequence of tokens.

Embedding Layer: Converts tokens into vectors that the model can process.

Add Positional Encoding: Incorporates information about the position of tokens in the sequence into their embeddings, which is crucial since attention mechanisms do not inherently process sequential data.

Multi-Head Attention: Allows the model to focus on different parts of the sequence for different representations, facilitating better understanding and processing of the input.

Add & Norm: A layer that combines residuals (from skip connections) with the output of the attention or feedforward layers, followed by layer normalization.

Feedforward Network: A dense neural network that processes the sequence after attention has been applied.

Output Sequence: The final processed sequence output by the model, often used for tasks like translation, text generation, or classification.

Skip Connections: Dashed lines represent skip connections that help to alleviate the vanishing gradient problem by allowing gradients to flow through the network directly. They also help the model to learn an identity function which ensures that the model does not lose information throughout the layers.

Graph Neural Networks (GNNs)

A neural network architecture that uses graph structures to model relationships between nodes. GNNs are effective for data that can be represented as graphs, such as social networks or molecules, as they capture the relationships between entities.

  • Usage: Graph-based data, social network analysis, recommendation systems
  • Strengths: Excellent performance on graph-based data, can model complex relationships
  • Caveats: Computationally expensive, require large datasets

GNN input_graph Input Graph node_features Node Features input_graph->node_features edge_features Edge Features input_graph->edge_features gnn_layers GNN Layers node_features->gnn_layers edge_features->gnn_layers aggregate Aggregate Messages gnn_layers->aggregate update Update States aggregate->update readout Graph-level Readout update->readout output Output readout->output

Input Graph: The initial graph input containing nodes and edges.

Node Features: Processes the features associated with each node. These can include node labels, attributes, or other data.

Edge Features: Processes features associated with edges in the graph, which might include types of relationships, weights, or other characteristics.

GNN Layers: A series of graph neural network layers that apply convolution-like operations over the graph. These layers can involve message passing between nodes, where a node’s new state is determined based on its neighbors.

Aggregate Messages: Combines the information (messages) received from neighboring nodes into a single unified message. Aggregation functions can include sums, averages, or max operations.

Update States: Updates the states of the nodes based on aggregated messages, typically using some form of neural network or transformation.

Graph-level Readout: Aggregates node states into a graph-level representation, which can be used for tasks that require a holistic view of the graph (e.g., determining the properties of a molecule).

Output: The final output, which can vary depending on the specific application (node classification, link prediction, graph classification, etc.).

Reinforcement Learning (RL) Architectures

A neural network architecture that uses reinforcement learning to learn from interactions with an environment. RL architectures are highly effective for sequential decision-making tasks, such as playing games or navigating environments.

  • Usage: Game playing, robotics, autonomous systems
  • Strengths: Excellent performance on sequential decision-making tasks, can learn complex policies
  • Caveats: Require large datasets, can be slow to train

RL environment Environment state State environment->state reward Reward environment->reward updated_state Updated State environment->updated_state Feedback Loop agent Agent state->agent action Action agent->action action->environment reward->agent updated_state->state New State

Environment: This is where the agent operates. It defines the dynamics of the system including how the states transition and how rewards are assigned for actions.

State: Represents the current situation or condition in which the agent finds itself. It is the information that the environment provides to the agent, which then bases its decisions on this data.

Agent: This is the decision-maker. It uses a strategy, which may involve a neural network or another function approximator, to decide what actions to take based on the state it perceives.

Action: The decision taken by the agent, which will affect the environment.

Reward: After taking an action, the agent receives a reward (or penalty) from the environment. This reward is an indication of how good the action was in terms of achieving the goal.

Updated State: After an action is taken, the environment transitions to a new state. This new state and the reward feedback are then used by the agent to learn and refine its strategy.

Evolutionary Neural Networks (ENNs)

A neural network architecture that uses evolutionary principles to evolve neural networks. Evolutionary Neural Networks are particularly effective for optimization problems, where they can evolve solutions over generations.

  • Usage: Neuroevolution, optimization problems
  • Strengths: Excellent performance on optimization problems, can learn complex policies
  • Caveats: Computationally expensive, require large datasets

ENN population Initial Population (Neural Networks) selection Selection population->selection crossover Crossover selection->crossover mutation Mutation crossover->mutation fitness Fitness Evaluation mutation->fitness new_population New Generation fitness->new_population best_network Best Performing Network fitness->best_network If Optimal new_population->selection Next Generation best_network->new_population Update Population

Initial Population: This represents the initial set of neural networks. These networks might differ in architecture, weights, or hyperparameters.

Selection: Part of the evolutionary process where individual networks are selected based on their performance, often using a fitness function.

Crossover: A genetic operation used to combine features from two or more parent neural networks to create offspring. This simulates sexual reproduction.

Mutation: Introduces random variations to the offspring, potentially leading to new neural network configurations. This step enhances diversity within the population.

Fitness Evaluation: Each network in the population is evaluated based on how well it performs the given task. The fitness often determines which networks survive and reproduce.

New Generation: After selection, crossover, mutation, and evaluation, a new generation of neural networks is formed. This generation forms the new population for further evolution.

Best Performing Network: Out of all generations, the network that performs best on the task.

Feedback Loops:

  • Next Generation: The cycle from selection to fitness evaluation and then back to selection with the new generation is a loop that continues until a satisfactory solution (network) is found.

  • If Optimal: If during any fitness evaluation a network meets the predefined criteria or optimality, it may be selected as the final model.

Spiking Neural Networks (SNNs)

A neural network architecture that uses spiking neurons to process data. SNNs are particularly effective for neuromorphic computing applications, where they can operate in energy-efficient ways.

  • Usage: Neuromorphic computing, edge AI
  • Strengths: Excellent performance on edge AI applications, energy-efficient
  • Caveats: Limited software support, require specialized hardware

SNN input_neurons Input Neurons synaptic_layers Synaptic Layers (Weighted Connections) input_neurons->synaptic_layers spiking_neurons Spiking Neurons synaptic_layers->spiking_neurons threshold_mechanism Threshold Mechanism synaptic_layers->threshold_mechanism Dynamic Weights spiking_neurons->threshold_mechanism output_neurons Output Neurons spike_train Spike Train Output output_neurons->spike_train threshold_mechanism->output_neurons

Input Neurons: These neurons receive the initial input signals, which could be any time-varying signal or a pattern encoded in the timing of spikes.

Synaptic Layers: Represents the connections between neurons. In SNNs, these connections are often dynamic, changing over time based on the activity of the network (Hebbian learning principles).

Spiking Neurons: Neurons that operate using spikes, which are brief and discrete events typically caused by reaching a certain threshold in the neuron’s membrane potential.

Threshold Mechanism: A critical component in SNNs that determines when a neuron should fire based on its membrane potential. This mechanism can adapt based on the history of spikes and neuronal activity.

Output Neurons: Neurons that produce the final output of the network. These may also operate using spikes, especially in SNNs designed for specific tasks like motor control or sensory processing.

Spike Train Output: The output from the network is often in the form of a spike train, representing the timing and sequence of spikes from the output neurons.

Dynamic Weights: Indicates that the synaptic weights are not static and can change based on the spike timing differences between pre- and post-synaptic neurons (STDP - Spike-Timing-Dependent Plasticity).

Conditional Random Fields (CRFs)

A probabilistic model that uses graphical models to model sequential data. CRFs are particularly effective for sequence labeling tasks, where they can model complex relationships between labels in a sequence.

  • Usage: NLP, sequence labeling, information extraction
  • Strengths: Excellent performance on sequential data, can model complex relationships
  • Caveats: Computationally expensive, require large datasets

CRF input_sequence Input Sequence feature_extraction Feature Extraction input_sequence->feature_extraction crf_layer CRF Layer feature_extraction->crf_layer state_transition State Transition Features feature_extraction->state_transition output_labels Output Labels crf_layer->output_labels state_transition->crf_layer

Input Sequence: Represents the raw data input, such as sentences in text or other sequential data.

Feature Extraction: Processes the input data to extract features that are relevant for making predictions. This could include lexical features, part-of-speech tags, or contextual information in a natural language processing application.

CRF Layer: The core of the CRF model where the actual conditional random field is applied. This layer models the dependencies between labels in the sequence, considering both the input features and the labels of neighboring items in the sequence.

Output Labels: The final output of the CRF, which provides a label for each element in the input sequence. In the context of NLP, these might be tags for named entity recognition, part-of-speech tags, etc.

State Transition Features: This represents how CRFs utilize state transition features to model the relationships and dependencies between different labels in the sequence. These are not actual data flow but indicate the type of information that influences the CRF layer’s decisions.

Mixture of Experts (MoE)

A neural network architecture that consists of multiple expert networks (submodels), each specialized in different parts of the data or tasks. A gating network determines which expert(s) are most relevant for a given input. MoE is particularly effective for large-scale machine learning models, where it can dynamically route tasks to the most appropriate experts.

  • Usage: Large-scale machine learning models, task-specific adaptations, dynamic routing of tasks
  • Strengths: Highly scalable, capable of handling diverse tasks simultaneously, efficient use of resources by activating only relevant experts for each input.
  • Caveats: Complex to implement and train, requires careful tuning to balance the load across experts and avoid overfitting in individual experts.

MoE input_data Input Data gating_network Gating Network input_data->gating_network expert1 Expert 1 gating_network->expert1 Weight expert2 Expert 2 gating_network->expert2 Weight expert3 Expert 3 gating_network->expert3 Weight combined_output Combined Output gating_network->combined_output Decision Weights expert1->combined_output Output 1 expert2->combined_output Output 2 expert3->combined_output Output 3

Input Data: Represents the data being fed into the model. This could be anything from images, text, to structured data.

Gating Network: A crucial component that dynamically determines which expert model should handle the given input. It evaluates the input data and allocates weights to different experts based on their relevance to the current data point.

Experts: These are specialized models (expert1, expert2, expert3) that are trained on subsets of the data or specific types of tasks. Each expert processes the input independently.

Combined Output: The final output of the MoE model, which typically involves aggregating the outputs of the experts weighted by the gating network’s decisions.

Weights: These edges show how the gating network influences the contribution of each expert to the final decision. The weights are not fixed but are determined dynamically based on each input.

Output 1, 2, 3: These labels on the edges from experts to the combined output represent the contribution of each expert to the final model output. Each expert contributes its processed output, which is then combined based on the weights provided by the gating network.