Poor mans nitric acid would be ok for gold, the sulfate salts will not be a problem with gold, but if your nitrate salt was high in chlorides it could dissolve your gold when you do not want it to, the chlorides can be removed easily from your nitric acid see below.
With poor mans nitric acid the problem of using it with silver is not the small amount of chlorides the nitrate salt may have contained, as these can be removed easily.
The chloride ions can be removed from the nitric acid, by using silver nitrate solution, forming a precipitant of silver chloride; decanting the silver chloride would leave nitric acid chloride free.
When you make poor mans nitric acid using a nitrate salt and sulfuric acid, you make nitric acid mixed with a sodium or potassium sulfate salt, it is the Na2SO4, or K2SO4 that you try to precipitate out of solution, although chilling the solution you can remove more of these salts, you cannot remove them all without distilling your home made nitric acid, it is the sulfate salts that become a problem with your nitric when working with silver, this will form silver sulfate which is not very soluble, and the silver sulfate is harder to recover silver later (melting with iron).
You can use your homemade nitric acid for silver if you distill the nitric acid, for a more pure nitric, leaving the sulfate salt behind in the reaction vessel, the chlorides carried over in the distillation is cleaned up from the nitric with silver nitrate solution, as stated above, this way you can end up with a nitric acid solution free of sulfates or chlorides, which will make it easier to use on silver, or for parting your in-quartered gold where you wish to recover the silver.