A Gentle Introduction to Deep Learning - Basic Concepts
English (current) | 简体中文
Preface
Recently, I had the opportunity to gain a shallow but systematic understanding of the deep learning process, from network design and quantization to FPGA deployment. Therefore, I’m writing this article to record my knowledge, organize my thoughts and questions, and hopefully help those who are new to this field.
Since my background is in Computer Systems rather than AI, there may be oversights and inaccuracies. Please bear with me.
Key Metrics
Let’s start with three important metrics: Parameters, FLOPs (Floating Point Operations), and Memory Usage. It’s worth noting that in today’s deep learning, FLOPs has become the standard metric for evaluating computational complexity.
The following table lists the metrics of several classic networks:
| Model | Parameters (M) | FLOPs (M) | Memory (MB) |
|---|---|---|---|
| AlexNet | 61.1 | 710 | 233.1 |
| VGG-16 | 138 | 15,470 | 527.8 |
| GoogleNet | 6.6 | 1,500 | 49.7 |
| Inception-v3 | 27.2 | 5,710 | 103.9 |
These metrics only indicate model complexity, not performance. To evaluate model performance, we need to focus on metrics like Accuracy, Precision, Recall, and F1 Score.
If you look up the specific applications of these models, you’ll find that most of them are image-related models for feature recognition. In the following discussion, I will mainly focus on deep learning for image recognition.
Training Process
Training refers to the process of optimizing network parameters for specific tasks, such as image classification and natural language processing.
In neural networks, each circle represents a neuron, and the connections between neurons contain parameters like weights and biases.
The training process includes two key steps:
Forward Propagation
For input images, multiple layers of neurons sequentially extract features at different levels:
- Shallow layers: Basic features like edges and textures
- Deep layers: High-level features like shapes and semantics
- Usually processed separately for RGB color channels
The extracted features are stored as tensors in multi-dimensional arrays.
Backward Propagation
The loss function evaluates the difference between predicted results and true labels, and optimization algorithms like gradient descent are used to update network parameters.
One complete forward and backward propagation is called a batch, and multiple batches form an epoch. Models need to go through multiple training epochs to achieve expected performance.
Model Optimization Techniques
Modern deep learning models can have millions or even billions of parameters. How to deploy these models on resource-constrained hardware (such as embedded devices and FPGAs) is an urgent problem to solve.
Pruning
Remove parameters that have little impact on model performance, thereby reducing the overall parameter count.
Structured Pruning
Removes entire neurons or channels, maintaining regular network structure, easy to accelerate on various hardware.
Unstructured Pruning
Removes arbitrary parameters, achieving higher compression rates but requiring special hardware support.
Quantization
Map floating-point parameters to low-precision integer representations.
Quantization Types
Common quantization approaches include:
- INT8 Quantization: From 32-bit floats to 8-bit integers
- INT4 Quantization: More aggressive quantization for further storage and computation reduction
Here’s a simple quantization example:
Original weight: 0.847 (32-bit float)
Quantized: 217 (8-bit int)Quantization can significantly reduce model size and inference time, particularly suitable for hardware platforms like FPGAs.
Summary
This article introduces some fundamental concepts of deep learning, covering key points from model evaluation metrics to training processes and model optimization. Of course, complete deep learning engineering also includes data preprocessing, model validation, hyperparameter tuning, and other aspects.
In subsequent articles, we will use specific network architectures (such as AlexNet) as examples to explore the practical applications of these concepts in depth.