We all think we need to learn something that can be useful at some point in our career. But what I realized lately is the term useful is very subjective and unpredictable as we don't know the future. So, it is always very important to stay curious to progress professionally and in other aspects as well. Curiosity and learning things out of comfort zone helps us to stay sharp and connect the dots in the grand scale.
As part of that, I plan to document whatever I learn in this repository. Daily learning signifies that the learning efforts should be consistent and deep reads signifies that we need to understand the concepts in-depth.
Whether it be reading a new blog post, a Youtube tutorial or a lecture video; if I feel I've learned something meaningful, I will add it here. I can also add a list of to-read's that can enhance my learning.
I believe most of the content that comes to this repo will be technical but not exactly related to my job.
So, to sum it up - Be curious, don't stop learning and document.
Adding new resources that I might read. Will just add whatever I find interesting in the meantime.
- Debugging Distributed Codebase - https://youtu.be/_8xlRgFY_-g?feature=shared
- Stanford lecture on scaling laws https://youtu.be/9vM4p9NN0Ts?feature=shared (refer
blog_reads_notes/Yann_StanfordLecture_LLMintro.pdf) - CMU lecture on Advanced NLP - https://www.youtube.com/watch?v=MM48kc5Zq8A&list=PL8PYTP1V4I8D4BeyjwWczukWq9d8PNyZp
- UMass lecture - https://people.cs.umass.edu/~miyyer/cs685/schedule.html
- MLSys Stanford series - https://mlsys.stanford.edu/
- Industry related best practices - https://parlance-labs.com/education/
- TinyML - https://hanlab.mit.edu/courses/2024-fall-65940
- Speculative Decoding - https://speculative-decoding.github.io/
Learn some basics of engineering systems, databases and OS and memory stuff. This will help to better design software.
- OS - https://pages.cs.wisc.edu/~remzi/Classes/537/Fall2021/
- Parallel Computing - https://youtu.be/V1tINV2-9p4?feature=shared
- What every programmer should know about memory? - https://people.freebsd.org/~lstewart/articles/cpumemory.pdf
- Performance Engineering Optimization - https://ocw.mit.edu/courses/6-172-performance-engineering-of-software-systems-fall-2018/video_galleries/lecture-videos/
- CUDA Mode - https://www.youtube.com/@GPUMODE (only after completing some OS and PMPP I can understand this better I believe!)
- Blogs - https://www.evidentlyai.com/ml-system-design
- HuggingFace notebook https://huggingface.co/spaces/nanotron/ultrascale-playbook
- GPUFryer - https://github.com/huggingface/gpu-fryer
- DSPy Usecases and tutorials - https://dspy.ai/dspy-usecases/
- Llama from scratch - https://blog.briankitano.com/llama-from-scratch/
- Distributed finetuning - https://sumanthrh.com/post/distributed-and-efficient-finetuning/
- How do ViTs work? - https://github.com/xxxnell/how-do-vits-work
- Tensor puzzles - https://github.com/srush/Tensor-Puzzles
- GPU puzzles - https://github.com/srush/GPU-Puzzles
- Transformer puzzles - https://github.com/srush/transformer-puzzles
- Understanding details of inference - https://kipp.ly/transformer-inference-arithmetic/
- KV cache https://dipkumar.dev/becoming-the-unbeatable/posts/gpt-kvcache/
- Mamba - https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-mamba-and-state
- GPUs brrr - https://horace.io/brrr_intro.html
- GEMM history - https://petewarden.com/2015/04/20/why-gemm-is-at-the-heart-of-deep-learning/
- Quantization indepth - https://mohitmayank.com/a_lazy_data_science_guide/machine_learning/model_compression_quant/
- https://github.com/gkamradt/LLMTest_NeedleInAHaystack?tab=readme-ov-file
- https://github.com/huggingface/search-and-learn
As a sidenote, I've refactored my old repository (ML CodeBase) to expand the learning materials and scope thus the commits can be much older. This is part of recycling efforts (haha!) Old ReadME - Learning how to code in Pytorch. I try to code NN, ML, DL stuff from various sources as part of my learning. This replication helps me to revise easily in future and also it's a great practice to learn DL.
- Transformers from scratch - https://www.youtube.com/watch?v=U0s0f995w14
- ViT from scratch - https://www.youtube.com/watch?v=ovB0ddFtzzA
- Hugging face course - https://huggingface.co/course/chapter1/1