> 1. Why does AI need that folder structure? Why not a flat list of files and let the AI agent explore with BM25 / grep, etc.

Two reasons I think:

Coding agents simulate similar things to what they have been trained on. Familiarity matters.

And they tend to do much better the more obvious and clear a task is. The more they have to use tools or "thinking", the less reliable they get.