Heroku push failed due to g++ not finding library

I've got an Node js web app using binding.gyp to compile a C++ standalone program. The standalone program requires libmagic.a and libz.a. Deployment failed with these errors

g++: error: /app/.apt/usr/lib/x86_64-linux-gnu/libmagic.a: No such file or directory

g++: error: /app/.apt/usr/lib/x86_64-linux-gnu/libz.a: No such file or directory

These two libraries are installed by apt buildpack at

/app/.apt/usr/lib/x86_64-linux-gnu/

with permission (shown through commands on heroku bash):

-rw------- 1 u28074 dyno 233490 Nov 20 2015 /app/.apt/usr/lib/x86_64-linux-gnu/libmagic.a

-rw------- 1 u28074 dyno 143722 Mar 3 17:52 /app/.apt/usr/lib/x86_64-linux-gnu/libz.a

I try to set full permission to those files through heroku bash but it is reset back to what it was once I exit the bash. And the push keeps failing. What should I do to make the library visible?

via gunner308