The former is the right answer. Now, there are open source options, e.g. Nouveau for Nvidia originally for Linux, but has been ported to FreeBSD.

Gallium should make it easier to do what you wanted as well - providing the glue between a video driver and APIs such as OpenGL / DirectX ( - there is a gallium implementation of DirectX 9 for Wine).