In a public letter, Mr Ballmer said he had become "very busy" since he quit the top job and that it would be "impractical to continue".
His decision to leave follows his recent purchase of the Los Angeles Clippers basketball team.
"I see a combination of the Clippers, civic contribution, teaching and study taking a lot of my time," he wrote.
He said his departure would be immediate due to a "hectic" autumn including both the start of the NBA basketball season and his teaching of a new class.
Mr Ballmer left the chief executive role in February after more than 14 years at the helm, and has been involved in the company for more than three decades.
He still holds more Microsoft shares than any other individual, and on Tuesday he pledged to hold onto them for the "foreseeable future".
"I bleed Microsoft - have for 34 years and I always will," he wrote.