G H Newbery & Son was fined £12,000 last month for depositing controlled waste including asbestos on the unlicensed land in Devon.
Other metal recyclers in the west country spoke out against the Exeter-based company, saying it had scarred their reputation.
But G H Newbery owner Russel Toghill has insisted that the firm that took the waste from his site did so without his permission.
Toghill said: I think there has been a bit of an over-reaction in some parts overall we have an excellent reputation with the Environment Agency.
I employed an outside contractor, John Brown Recycling, to take away soil and rubble waste, but it took all the other stuff, which is what got us in trouble.
I accept that I did not oversee it properly, and I have not complained about the fine. But I would have expected the contractor to know what he was taking for a start he would have charged about 10 times more to take controlled waste.
I am talking to my solicitor with a view to seeking damages from John Brown Recycling.
A west country scrap merchant said after G H Newberry was fined in June: The feeling in the region is that this company has given our trade a bad name. Its actions are bad for the whole of recycling. Its too low a fine.
But Toghill added: I think it was a substantial fine, and it was made in relation to our turnover. There is no doubt I will be asking to see licences in the future.