Ben Garrity scored the only goal of the game five minutes from time to give the hosts a fifth victory in their past six matches.
Vale were the dominant team for most of the encounter, but their clear-cut chances were kept to a minimum by a strong defensive performance.
That was until Garrity let fly to break the Cobblers' hearts, and it left Brady despondent.
"It was a tough afternoon's work," said the Town boss.
"Port Vale played really well, and we weren't at our best on the ball today, and they suffocated us in the end.
"It was hard to get out and we had to work hard out of possession, and we did work really hard, and their best chance falls 19 or 20 yards out to Garrity who scores from a hell of a strike, although we should have defended that better."
Even more frustrating for Brady was the fact, just minutes before the match-winning goal, Sam Hoskins had a great chance to fire the visitors in front, only to prod his effort from close range over the bar.
"I think we had the best chance of the game just before that, and usually Sam puts that away for us," said the Cobblers boss.
"Even if the ball goes by him, it goes through to Mitch (Pinnock) who taps it in, so it is frustrating."
And he added: "Sometimes you are going to have to come away at this level, and we are going to have to accept we are not going to dominate the ball.
"I wouldn't say we were great on the ball today, we weren't at our fluid best, but that is a compliment to them on the way they suffocated us on our shape.
"Look, we need to be better all round."