I've just picked up a new, old car.

When I drive it over bumps, the left hand passenger side wheel area or suspension makes a rumbling, sort-of knocking noise.

I've had the wheel off (and swapped it for the wheel from my old car due to the tryes being better). But that hasn't made a difference.

Anyone any idea what this would be?

The car's a Peugeot 405 estate. A stately 1995 diesel.

could be worn wheel bearings,
To check a wheel bearing, jack the wheel up off the ground, grasp the tire at the 12 and 6 o’clock positions and attempt to rock the tire. As a rule, there should be zero play on most FWD cars, but up to .010 inch of play in the front bearings may be acceptable on RWD applications that have adjustable bearings. Also rotate the tire by hand. Any roughness or noise from the bearings would also tell you the bearings are worn or damaged and need to be replaced.
http://en.wikipedia.org/wiki/Wheel_bearing

just one possibility

And I did start asking myself the other night if it was wandering a little (though there were quite high winds on the A1) that night.

However, it does fit in with the rest of the bearing play section of that link.

Thanks scottie!

Hmm, possibilty of it being a ball joint/rubber bushes in suspension struts.

As scott says grasp the tire/wheel at the 12 and 6 o’clock positions and attempt to rock the tire/wheel spin the wheel, always it spin it in the direction of rotation.

If you want to stop it spinning using your hands, never grasp the tire across it's width, you may cut your hands on a stone or similar stuck in the tread.

Thanks for all the suggestions (and tips for investigating). My local Peugeot mechanic says:

front nearside top suspension bushing is gone
front nearside pea bushing? is gone
sub-chassis bushing front and back are gone

Other pals say: they are exploiting your ignorance of cars, be careful.

However, it's just money... this is the sort of problem money was made for spending on!

I think

Just over 300. Exact price tba, depending on time required to extract rusted-in bolts etc.

Well... it is less than I paid for the car. Just!

Well I couldnt really tell you if thats reasonable or not, as I havent been to a garage in twenty years. But definatly the faults theyve said and your description seems to fit.
If its anything like my last two Peugeots it will be worth it. Shame you don't know any one thats handy with a spanner could save you a lot of dosh, cos the parts mentioned arent expensive.