The 60-year-old, who led the Owls and Boro out in two cup finals apiece in the 1990s, had spent two and a half years in charge of Bristol City.
The former Hull City and Leicester City manager led his Robins to a 2-1 victory at Rotherham United earlier this month but won only one of the next four games, including a defeat at Leeds United.
Chairman Jon Lansdown admitted it has been a difficult decision but believes now is the right time to make a change, with the club 15th in the table.
He told the club’s website: “We all wanted Nigel to achieve our ambition to be promoted but, with our recent results, feel that now is the time to make a change to give the club the best possible chance of success.
“Nigel is highly respected by the staff and players and has been very supportive of the Academy structure and our Women’s team.
“We all wanted Nigel to achieve our ambition to be promoted but, with our recent results, feel that now is the time to make a change to give the club the best possible chance of success.”
First team coach Jason Euell and head of medical performance Dave Rennie have also left the club.
Assistant manager Curtis Fleming will take on the role of interim head coach, supported by Ali Hines and Kalifa Cissé as the club seek a new head coach.