No matter if it takes you a single run or ten thousand runs; qualified manpower to code and supervise it costs more than $12/run, so with prices like that the limiting factor is how much people-time you can devote to it.
I disagree. I've participated and even won a few of those "ML in sciences" competitions in the past, and all other things being equal, the one with the most compute wins. 12$/hour sounds doable, but as soon as you need to grid over more than 4 hyperparameters (and that's small), you're looking at 100s to 1000s of runs in parallel. And you'll iterate over that process several times. Eg It's no coincidence that deepmind (with all of alphabets compute power) won CASP with alphafold. Yes they have a lot of very smart people, but so does everyone else. But no-one has the power to experiment with such large models so extensively for so long.
If you're doing grid search on more than 4 hyperparameters, that probably explains why you think computation wins. Even random search with a constrained budget will do pretty well without breaking the bank.
If you're not running standard models (where you only have learning rate and regularization to grid over) but you're trying more specialized stuff/developing new models, 4 hyperparameters are not too many.
