President Obama and Japanese Prime Minister Shinzo Abe said the plutonium and highly enriched uranium would be disposed of in the US.
The nuclear fuel could potentially be used to make some 50 nuclear weapons.
The US administration is trying to secure worldwide supplies to stop them falling into the hands of terrorists.
Japan will still retain large quantities of additional plutonium in spent fuel from its nuclear power industry.
US Energy Secretary Ernest Moniz described the deal to return some 300kg of plutonium as a very significant nuclear security pledge.
The material was originally bought from the US in the 1960s for what were described as research purposes.
The Abe administration was reported to have resisted initial approaches from the US to hand it back.
That led to expressions of concern by China at Japan's nuclear intentions.
Japan has a long-standing policy not to develop nuclear weapons.
But experts say its remaining stockpile of lower-grade plutonium contained in waste fuel could potentially be reprocessed as fissile material for an extremely large nuclear arsenal.
Japan is planning to open a nuclear reprocessing plant this year in the north of the country, which could turn out large quantities of additional plutonium.
The plutonium was intended to be reused as fuel in Japan's nuclear power stations, but they have been standing idle for three years in the aftermath of the disaster and meltdown at the Fukushima nuclear plant in 2011.
In February, China expressed grave concern at what it called Japan's possession of weapons-grade nuclear materials.
State media in China said that Japan needed to explain why it was stockpiling so much plutonium.
Some experts believe that Japan has the technology to build a nuclear weapon within a year should the government abandon its longstanding opposition to nuclear weapons.
Japanese public opinion, conditioned by the atomic attacks on Hiroshima and Nagasaki, remains hostile to such a move.
China has accused Mr Abe of taking Japan down a more nationalistic path as he resists Chinese pressure over disputed islands administered by Japan in the East China Sea.
BBC
