I've spent the last hour reading VOSA and DVLA documents and it seems farmers are ok to do pretty much whatever they want. They can haul building materials as long as it's for use on their property and they can transport their stock to market or waste to a disposal facility. Contractors can also haul goods from one part of a farm to another as long as it's for the same land owner and and the materials being hauled are either needed for the job they're doing (ie muck spreading) or are the product of the work going back to another part of the land for the same land owner (ie bales, grain etc).
Red diesel cannot be used in an agricultural tractor used for tree surgery or mowing unless the trees/verges are bordering public roads or are on land designated as forestry. So going to someones garden to cut their tree down or cut their lawn (and possibly paddock, not sure) is excluded unless the vehicle to be used on the property is trailered to the site and isn't used on the road, ie compact tractor mower, digger etc.
Whether the vehicle needs to be taxed other than agricultural I'm not sure. But it looks like the use of red is a no go for me, even when on site in domestic cases.