The quickest way will be by train. You need to go from London Liverpool Street station, and starting from anywhere in Buckinghamshire, the most practical way IS to go into London and out again. Going across in a straight line through Hertfordshire etc really isn't sensibly possible by public transport, and even driving, it would all be slow little roads. Everything seems to fan out from London. No wonder your family said it's too far to drive! It certainly would be tiring for them.
You say in a comment that your friend lives in Diss, which is on the main line from London to Norwich. Only fast trains stop there so you're in luck! - it's a direct train from London, and you can look up times and prices, and book tickets, at
https://www.greateranglia.co.uk/ There are two trains per hour and it takes about an hour and a half.
So talk it over with your friend. Is it worth going to Diss, or could you meet up in London and have some fun together there? Diss isn't very big, and it's your friend you want to see rather than Diss. It would give you longer together to split the difference and both travel to meet half way.
I can't help remembering John Betjeman's poem:
"Dear Mary, yes it would be bliss
To go with you by train to Diss...
We'll meet, my sweet, at Liverpool Street..."
Now there's an idea for where to meet each other if you decide on a day together in London!