Recurse Center - Batch 2 - Cycle 20241012-20241014 - Normalization


Normalization

This cycle was mostly spent with ARENA and NNFS. I've been spending a lot of time thinking about what to do post-ARENA, which lead me to think a lot about why I'm even doing ARENA in the first place. I dug around a lot more into AI safety, and I think I'm motivated by interpretability and observability of models. I'm interested how some of this work will translate to voice based models and agents, as that really feels like an under-explored area that I would have a lot of fun figuring out with opportunities to make novel contributions.

Day 1

I built an implementation of ResNet34 from scratch, and it was super exciting to have it all come together.

I also just had tons of fun debugging the network, seeing how many parameters it contained, and delightfully moving through the code in order to get it to work correctly. I found a lot of joy in that kind of low-level ML engineering, and could see myself having lots of fun doing that kind of work professionally.

Here is what my ResNet looked like with some layers missing

I had a really subtle bug where I was returning a list to my BlockGroup class that didn't match the size of other params that I was zipping up in order to determine the number of layers in my BlockGroup.

After a bit of debugging I realized that my in_featers_per_group was missing two entries. It turns out that I was returning just the first two elements of the list by saying self.out features per group[:1], but I just wanted to get a list without the last element, in which case I meant to write [:-1] instead of [:1]. So subtle yet so crucial!

I made the fix and got the right size so that my zip function would create four BlockGroup objects instead of just two.

And here's what it looked like once I properly got those layers networked together.

After finishing this, I started to realize that I now have enough skills to start building my own speech classifier, and that would be a really fun and exciting thing to work on as we finish up all of these pre-requisite, foundational excercises for AREA.

Day 2

Today I started working on the Optimizers chapter in AREA, and also started watching videos related to AI safety, just to get a better sense of the field.

I also made some much-needed improvements to Macaw, including adding paper authors to the show description.

Day 3:

Today I worked more on Optimizers and paired with a Recurser on Ansible setup on Heap :)

Things for next cycle