The flame or fire can be oxidizing or reducing depending on the air or oxygen content, so depending on the flame silver can actually gather oxygen from your torch of furnace flame, adjusting your torch to a more carbonizing flame or reducing flame can help as then the fire will take up the involved oxygen, and providing a more reducing environment for the silver.
Silver when heated to the melting point has a unique behavior with oxygen, when pure silver is melted in ambient air can absorb about ten times its volume, or about 0.3% of its weight of oxygen. On cooling to a few degrees above solidification, it abruptly releases most of its oxygen spitting out the oxygen, this oxygen can come from the air, or from your torch, starving the environment from air can help, as well as a carbon flux cover while melting.
Nitrogen does not burn, but some of it can combine with oxygen at these high temperatures forming some NOx gases.