Mr Rajan had been “open” to continuing after his initial three-year term, but has made his move after “due reflection and consultation with the government”.
The former IMF economist, who was one of the few to predict the 2008 financial crisis, has been praised for his handling of the Indian economy. It is now the fastest growing major economy in the world.
However he has faced pressure from a faction within Prime Minister Narendra Modi’s ruling party for keeping interest rates high. There has also been criticism that he had begun to stray into politics.
Analysis: Simon Atkinson, India Business Editor, Mumbai
Raghuram Rajan’s exit (or Rexit as it has been labelled here) has thrown up more questions than answers. And the biggest of all: did he walk or was he pushed?
Certainly the comment that he’d been open to staying but made his decision “after due reflection and in consultation with the government” is leading many here to conclude he’d been told to go quietly.
The RBI governor has had plenty of public detractors – including one recently from a senior figure in the ruling BJP party that he was “mentally not fully Indian” because he has a US green card.
But while not pleasing all politicians – Mr Rajan’s economic track record means he’s hugely respected by investors both at home and abroad.
And with India relying heavily on foreign investment to keep its growth going, there is a lingering worry that the country’s economic credibility overseas could now be called into question by the decision not to make sure the man nicknamed the “rockstar” banker was talked into staying for another term.
In a letter to Reserve Bank of India (RBI) workers, Mr Rajan remarked that two of his major policy ideas – the creation of a monetary policy committee to set interest rates and reform of the heavily indebted banking sector – had not been completed.
He also made passing reference to the referendum in the UK next week on EU membership.
“Colleagues, we have worked with the government over the last three years to create a platform of macroeconomic and institutional stability,” he said.
“I am sure the work we have done will enable us to ride out imminent sources of market volatility like the threat of Brexit.”
His term of office ends on 4 September.
It will be the first time in 24 years that an RBI chief has departed after a single three-year term.