Practical Tips for Finetuning LLMs Using LoRA (Low-Rank Adaptation)
FIXEDD Low-rank adaptation (LoRA) is among the most widely used and effective techniques for efficiently training custom LLMs. For…
April 2024
Research Papers in Nov 2023: Tackling Hallucinations, Boosting Reasoning Abilities, and New Insights into the Transformer Architecture
This month, I want to focus on three papers that address three distinct problem categories of Large Language…
Ten Noteworthy AI Research Papers of 2023
This year has felt distinctly different. I’ve been working in, on, and with machine learning and AI for…
Understanding and Coding Self-Attention, Multi-Head Attention, Cross-Attention, and Causal-Attention in LLMs
This article will teach you about self-attention mechanisms used in transformer architectures and large language models (LLMs) such…
Research Papers in Jan 2024: Model Merging, Mixtures of Experts, and Towards Smaller LLMs
2023 was the year when the potential and complexity of Large Language Models (LLMs) were growing rapidly. Looking…
Improving LoRA: Implementing Weight-Decomposed Low-Rank Adaptation (DoRA) from Scratch
Low-rank adaptation (LoRA) is a machine learning technique that modifies a pretrained model (for example, an LLM or…
Research Papers in February 2024: A LoRA Successor, Small Finetuned LLMs Vs Generalist LLMs, and Transparent LLM Research
Once again, this has been an exciting month in AI research. This month, I’m covering two new openly…
Tips for LLM Pretraining and Evaluating Reward Models
It’s another month in AI research, and it’s hard to pick favorites. Besides new research, there have also…
Learning from Crossrail’s information management challenge
Nearly a month ago, BIMplus highlighted how the Crossrail project team learned the hard way the trouble associated…