Short answer: yes.
In fact, it's a little more complicated than that. The device manufacturer or OEM buys different components (chips and sensors) from other manufacturers. Some of those components (such as the SoC or chipset) in turn contain hardware designed by other companies. For example, an LG phone might contain a SoC manufactured by Qualcomm, which in turn contains a GPU designed by ARM.
Just like how the hardware contains components from different vendors, so does the software. In the example, when ARM licenses (sells) the GPU design to Qualcomm, they also supply the source code for a reference driver for the GPU design. (They'd include source for whatever platforms Qualcomm is interested in, including Android.) Qualcomm would adjust all the parameters of the driver (such as the memory map, which you mentioned) for their SoC, and put it together with drivers for all the other components of the SoC.
Then, when LG buys the SoC from Qualcomm, Qualcomm would supply this combined driver to LG. LG puts this together with the drivers for other parts of the phone (such as the touch screen), adds their UI customizations, and builds a ROM image that they can flash onto the phone during manufacture.
There's sometimes an extra step, too: the carrier might change the ROM to include more pre-installed apps, a different boot animation, or other customizations.
So, although in general each component manufacturer or designer is responsible for the driver for their own components, that driver is changed further down the line.