Okay I found the solution.
In HAProxy backend for the server that is affected you set:
send-proxy
In the “Per server pass thru” box under Advanced Settings.
Then in your Apache site config you have to include:
RemoteIPProxyProtocol On
As well as enabling the mod_remoteip module.
Hopefully this is helpful to anyone else finding themselves with this type of configuration.
