There are three main variants of Gradient Descent, differing in how much data is used to compute the gradient.<\/span><\/p>\n(1) Batch Gradient Descent<\/b><\/p>\n\n- Uses the\u00a0<\/span>entire training dataset<\/b>\u00a0to compute the gradient.<\/span><\/li>\n
- Pros:<\/b>\u00a0Stable convergence, accurate updates.<\/span><\/li>\n
- Cons:<\/b>\u00a0Computationally expensive for large datasets.<\/span><\/li>\n<\/ul>\n
(2) Stochastic Gradient Descent (SGD)<\/b><\/p>\n\n- Uses\u00a0<\/span>one random training example<\/b>\u00a0per iteration.<\/span><\/li>\n
- Pros:<\/b>\u00a0Faster updates, can escape local minima.<\/span><\/li>\n
- Cons:<\/b>\u00a0Noisy updates, may not converge smoothly.<\/span><\/li>\n<\/ul>\n
(3) Mini-Batch Gradient Descent<\/b><\/p>\n\n- Uses a\u00a0<\/span>small batch of samples<\/b>\u00a0(e.g., 32, 64, 128) per iteration.<\/span><\/li>\n
- Pros:<\/b>\u00a0Balances speed and stability (most commonly used in practice).<\/span><\/li>\n
- Cons:<\/b>\u00a0Requires tuning batch size.<\/span><\/li>\n<\/ul>\n
\n- Challenges & Improvements<\/b><\/li>\n<\/ol>\n
Common Challenges:<\/b><\/p>\n\n- Learning Rate Selection:<\/b>\n
\n- Too small \u2192 Slow convergence.<\/span><\/li>\n
- Too large \u2192 Overshooting, divergence.<\/span><\/li>\n<\/ul>\n<\/li>\n
- Local Minima & Saddle Points:<\/b>\n
\n- The algorithm may get stuck in suboptimal points.<\/span><\/li>\n<\/ul>\n<\/li>\n
- Noisy Updates (in SGD):<\/b>\n
\n- High variance in parameter updates.<\/span><\/li>\n<\/ul>\n<\/li>\n<\/ol>\n
Improvements & Optimizers:<\/b><\/p>\n
To address these issues, several advanced optimizers have been developed:<\/span><\/p>\n\n\n\n| Optimizer<\/b><\/th>\n | Key Idea<\/b><\/th>\n | Advantage<\/b><\/th>\n<\/tr>\n<\/thead>\n\n\n| Momentum<\/b><\/td>\n | Adds a fraction of the previous update to current gradient.<\/span><\/td>\n | Reduces oscillations.<\/span><\/td>\n<\/tr>\n\n| Nesterov Accelerated Gradient (NAG)<\/b><\/td>\n | Improves Momentum by looking ahead before updating.<\/span><\/td>\n | Better convergence.<\/span><\/td>\n<\/tr>\n\n| AdaGrad<\/b><\/td>\n | Adapts learning rates per parameter.<\/span><\/td>\n | Works well for sparse data.<\/span><\/td>\n<\/tr>\n\n| RMSProp<\/b><\/td>\n | Improves AdaGrad by using an exponentially decaying average.<\/span><\/td>\n | Handles non-convex optimization better.<\/span><\/td>\n<\/tr>\n\n| Adam (Adaptive Moment Estimation)<\/b><\/td>\n | Combines Momentum and RMSProp.<\/span><\/td>\n | Most popular, works well in practice.<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n- Practical Considerations<\/b><\/li>\n<\/ol>\n
Choosing the Learning Rate<\/b><\/p>\n\n- Use\u00a0<\/span>learning rate scheduling<\/b>\u00a0(e.g., reducing\u00a0\u03b1<\/span>\u03b1<\/span><\/i>\u00a0over time).<\/span><\/li>\n
- Try\u00a0<\/span>adaptive optimizers<\/b>\u00a0(Adam, RMSProp).<\/span><\/li>\n<\/ul>\n
Monitoring Convergence<\/b><\/p>\n\n- Plot the\u00a0<\/span>cost vs. iterations<\/b>\u00a0(should decrease over time).<\/span><\/li>\n
- Use\u00a0<\/span>early stopping<\/b>\u00a0if the validation error stops improving.<\/span><\/li>\n<\/ul>\n
Feature Scaling<\/b><\/p>\n\n- Gradient Descent works better when features are\u00a0<\/span>normalized<\/b>\u00a0(e.g., using StandardScaler).<\/span><\/li>\n<\/ul>\n
\n- Conclusion<\/b><\/li>\n<\/ol>\n
Gradient Descent is a powerful optimization algorithm that drives most machine learning models. Understanding its variants, challenges, and improvements is crucial for training efficient models.<\/span><\/p>\nKey Takeaways:<\/b><\/p>\n \u2714<\/span> Gradient Descent minimizes the cost function by following the negative gradient.<\/span>
\n<\/span>\u2714<\/span> Batch, Stochastic, and Mini-Batch are the main variants.<\/span>
\n<\/span>\u2714<\/span> Advanced optimizers (Adam, RMSProp) improve convergence.<\/span> | | | | | |
![https:\/\/miro.medium.com\/max\/1400\/1*N5F9JZ6sf6N2XyQnQ6QNqw.gif<\/span><\/a><\/p>\n\n Types of Gradient Descent<\/b><\/li>\n<\/ol>\nThere are three main variants of Gradient Descent, differing in how much data is used to compute the gradient.<\/span><\/p>\n(1) Batch Gradient Descent<\/b><\/p>\n\nUses the\u00a0<\/span>entire training dataset<\/b>\u00a0to compute the gradient.<\/span><\/li>\nPros:<\/b>\u00a0Stable convergence, accurate updates.<\/span><\/li>\nCons:<\/b>\u00a0Computationally expensive for large datasets.<\/span><\/li>\n<\/ul>\n(2) Stochastic Gradient Descent (SGD)<\/b><\/p>\n\nUses\u00a0<\/span>one random training example<\/b>\u00a0per iteration.<\/span><\/li>\nPros:<\/b>\u00a0Faster updates, can escape local minima.<\/span><\/li>\nCons:<\/b>\u00a0Noisy updates, may not converge smoothly.<\/span><\/li>\n<\/ul>\n(3) Mini-Batch Gradient Descent<\/b><\/p>\n\nUses a\u00a0<\/span>small batch of samples<\/b>\u00a0(e.g., 32, 64, 128) per iteration.<\/span><\/li>\nPros:<\/b>\u00a0Balances speed and stability (most commonly used in practice).<\/span><\/li>\nCons:<\/b>\u00a0Requires tuning batch size.<\/span><\/li>\n<\/ul>\n\n Challenges & Improvements<\/b><\/li>\n<\/ol>\nCommon Challenges:<\/b><\/p>\n\nLearning Rate Selection:<\/b>\n\nToo small \u2192 Slow convergence.<\/span><\/li>\nToo large \u2192 Overshooting, divergence.<\/span><\/li>\n<\/ul>\n<\/li>\nLocal Minima & Saddle Points:<\/b>\n\nThe algorithm may get stuck in suboptimal points.<\/span><\/li>\n<\/ul>\n<\/li>\nNoisy Updates (in SGD):<\/b>\n\nHigh variance in parameter updates.<\/span><\/li>\n<\/ul>\n<\/li>\n<\/ol>\nImprovements & Optimizers:<\/b><\/p>\nTo address these issues, several advanced optimizers have been developed:<\/span><\/p>\n\n\n\nOptimizer<\/b><\/th>\nKey Idea<\/b><\/th>\nAdvantage<\/b><\/th>\n<\/tr>\n<\/thead>\n\n\nMomentum<\/b><\/td>\nAdds a fraction of the previous update to current gradient.<\/span><\/td>\nReduces oscillations.<\/span><\/td>\n<\/tr>\n\nNesterov Accelerated Gradient (NAG)<\/b><\/td>\nImproves Momentum by looking ahead before updating.<\/span><\/td>\nBetter convergence.<\/span><\/td>\n<\/tr>\n\nAdaGrad<\/b><\/td>\nAdapts learning rates per parameter.<\/span><\/td>\nWorks well for sparse data.<\/span><\/td>\n<\/tr>\n\nRMSProp<\/b><\/td>\nImproves AdaGrad by using an exponentially decaying average.<\/span><\/td>\nHandles non-convex optimization better.<\/span><\/td>\n<\/tr>\n\nAdam (Adaptive Moment Estimation)<\/b><\/td>\nCombines Momentum and RMSProp.<\/span><\/td>\nMost popular, works well in practice.<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n Practical Considerations<\/b><\/li>\n<\/ol>\nChoosing the Learning Rate<\/b><\/p>\n\nUse\u00a0<\/span>learning rate scheduling<\/b>\u00a0(e.g., reducing\u00a0\u03b1<\/span>\u03b1<\/span><\/i>\u00a0over time).<\/span><\/li>\nTry\u00a0<\/span>adaptive optimizers<\/b>\u00a0(Adam, RMSProp).<\/span><\/li>\n<\/ul>\nMonitoring Convergence<\/b><\/p>\n\nPlot the\u00a0<\/span>cost vs. iterations<\/b>\u00a0(should decrease over time).<\/span><\/li>\nUse\u00a0<\/span>early stopping<\/b>\u00a0if the validation error stops improving.<\/span><\/li>\n<\/ul>\nFeature Scaling<\/b><\/p>\n\nGradient Descent works better when features are\u00a0<\/span>normalized<\/b>\u00a0(e.g., using StandardScaler).<\/span><\/li>\n<\/ul>\n\n Conclusion<\/b><\/li>\n<\/ol>\nGradient Descent is a powerful optimization algorithm that drives most machine learning models. Understanding its variants, challenges, and improvements is crucial for training efficient models.<\/span><\/p>\nKey Takeaways:<\/b><\/p>\n\u2714<\/span> Gradient Descent minimizes the cost function by following the negative gradient.<\/span>\n<\/span>\u2714<\/span> Batch, Stochastic, and Mini-Batch are the main variants.<\/span>\n<\/span>\u2714<\/span> Advanced optimizers (Adam, RMSProp) improve convergence.<\/span>](\"https:\/\/miro.medium.com\/max\/1400\/1*N5F9JZ6sf6N2XyQnQ6QNqw.gif\"){kind=link}