107MB of cPython defeated

Go to try for self

Step 1 download 2.4GB of CUDA

The size of CUDA really is astonishing. Any chance someone might figure out how to slim that down?

Taking a peek inside the package it seems to mostly be the libraries - CuFFT alone is about 350MB for example, twice over for the debug and release versions. I'm guessing those are probably fat binaries pre-compiled for every generation of Nvidia hardware rather than just the PTX bytecode, which would help to speed up fresh builds, at the expense of being huge.

Here’s a blog that breaks down how large different pieces of CUDA are:

https://carlpearson.net/post/20231023-cuda-releases/

Talking directly to the kernel / driver / firmware.

As others have said, George Hotz is doing his best in reverse-engineering and skipping layers.

Raise your voice on their forum: https://forums.developer.nvidia.com/t/how-to-overcome-the-hu... Tried my luck 2 years ago but it keeps increasing.

Nvidia is the only one who could, since they own it.

I don't think it's about the byte size, but the inherent complexity of the implementation. 1000 lines of C code is extremely simple by any standard. Whereas a sundry collection of Python and PyTorch libraries is anything but.

A bunch of install methods for torch via pip include ~1.5GB of lib/ because of CUDA. libtorch_cuda.so is like 800MB on its own

I mean, being fair, the 2.4GB CUDA SDK is absolutely required for the cPython implementation as well

Python has been popular for this because it’s convenient to quickly hack on and experiment with, not because it’s the most efficient thing.

The overhead really isn't that bad is it? Since the the python code is mostly about saying multiply matrix A with matrix B, and then that actual computation is done by optimized low level code.

It depends on how you define overhead. Runtime overhead and memory usage is absolutely marginal, and the tightest, most perfect implementation will have trouble beating it.

Instead people are trying to optimize install size of dependencies, which while maybe a fun hacking project...who really cares?

I suspect that this has a high chance of running afoul of Ahmdal’s Law. Even if you can parallelise the bulk of the computation, the serial parts remain single-threaded and start to dominate the total runtime.

I don’t think the serial parts of ML training are Python’s fault, are they? It’s all “operation B depends on the output of operation A”.

For that stuff, yeah you're correct.

What I've seen is issues with the implementation of those libraries in a project.

I don't remember exactly, but I was playing with someone's wrapper for some kind of machine learning snake game and it was taking way longer than it should have on back of the napkin math.

The issue was using either a dict or a list in a hot loop and changing it to the other sped it up like 1000x.

So it's easy to think "yeah this library is optimized" but then you build something on top of it that is not obviously going to slow it down.

But, that's the Python tradeoff.

> The issue was using either a dict or a list in a hot loop and changing it to the other sped it up like 1000x.

The programmer using the wrong data structure is not a problem with the language.

Kinda. I guess my native tongue is C/C++ and I wouldn't expect such a huge performance difference when using an array vs a linked list or something.

It's not like I had millions of items in that structure either, it was like 100. I think it contained the batch training data from each round. I tried to find the project but couldn't.

I was just shocked that there was such a huge difference between primitive data structures. In that situation, I wouldn't have guessed it would make a difference.

It really is with Python. There are simply too many containers and container-like concepts. Lists, arrays, sets, dicts...

What modern language doesn’t have those?

Go kind of cheats and has maps play double duty as sets.

That sounds irrelevant to Python and just a matter of slow code cropping up in libraries until someone runs a profiler.

But then again if your program have places where choosing the right Python primitive is important for performance, then using python is affecting performance here since even the best algorithm in Python would be slower than the equivalent C.

Most of the time it doesn't matter because there's nothing hoy on the Python side, but if there is, then Python is going to be slowing your stuff down.