Admittedly, he spotted Crabbe and Goyle moving around the castle on their own more often than usual, sometimes remaining stationary in deserted corridors, but at these times Malfoy was not only nowhere near them, but impossible to locate on the map at all.