Test non pkg-config based linking in ci

[codyps]

I've broken this before. We should test it in ci #149 #150
Also broken in the past: detecting that we didn't have the library at all (2d4c186)

Adding checks for these to the ci would be good to do.