Understanding and Coding Self-Attention, Multi-Head Attention, Cross-Attention, and Causal-Attention in LLMs
FIXEDD This article will teach you about self-attention mechanisms used in transformer architectures and large language models (LLMs) such…
ai
Research Papers in Jan 2024: Model Merging, Mixtures of Experts, and Towards Smaller LLMs
2023 was the year when the potential and complexity of Large Language Models (LLMs) were growing rapidly. Looking…
Improving LoRA: Implementing Weight-Decomposed Low-Rank Adaptation (DoRA) from Scratch
Low-rank adaptation (LoRA) is a machine learning technique that modifies a pretrained model (for example, an LLM or…
Research Papers in February 2024: A LoRA Successor, Small Finetuned LLMs Vs Generalist LLMs, and Transparent LLM Research
Once again, this has been an exciting month in AI research. This month, I’m covering two new openly…
Tips for LLM Pretraining and Evaluating Reward Models
It’s another month in AI research, and it’s hard to pick favorites. Besides new research, there have also…
Unveiling Key Trends: Events Shaping Artificial Intelligence in 2024
5. AI for Climate Action: Addressing climate change requires innovative solutions, and AI is playing a crucial role…