It would reduce the opportunities for corruption in government. It's money that could be spent on revitalising industry where its sorely needed – a straightforward doing away with bad practices and encouraging good ones. It would also signal to the world that we might not be the two-faced hypocrites that they believe us to be.
The export credit guarantee fund ensures that British arms manufacturers get paid even if a foreign client reneges on payment. In other words when a client state agrees to pay £10 billion for a defence package that client state may decide after it has received the goods that it can't afford to pay for them so the British taxpayer pays. Originally this was designed to protect our arms industry but it now seems to have become a byword for robbery and corruption. Clients agree to a sum knowing full well they will never pay more than half and the British taxpayer will cover the outstanding balance. Anonymous Brits in suits give their blessing to arms deals and receive briefcases full of money. What's a few mill when you're getting a 50% reduction on 10 Billion?