Bring a large saucepan of water to the boil. Add the potatoes and boil for about 15 mins or until tender. Transfer to a colander and drain well, then return to the pan and set over a very low heat for 2 mins to dry completely.
Heat the milk and butter in a small pan, then pour over the potatoes. Remove pan from the heat, then mash potatoes using an electric hand whisk or potato masher. Tip in the créme fraîche and beat with a wooden spoon until smooth and creamy. Season with pepper and a pinch of salt.