Révocat Karemangingo had been a lieutenant in the Rwandan army that was overthrown in 1994 by forces led by President Paul Kagame.
In Mozambique he became a businessman and was not involved in politics, the leader of the refugee community said.
The Rwandan government has been accused of targeting opponents living abroad.
It has consistently denied the allegation.
Mr Karemangingo, who was the treasurer of the Rwandan refugee association, was driving to his home near the Mozambican capital, Maputo, when his car was ambushed on Monday.
He was shot nine times, police said.
No-one has been arrested so far and the police do not yet have a motive for the killing.
Mr Karemangingo had warned the authorities that there were people with links to Rwanda who were plotting to kill him, refugee association chairman Cléophas Habiyaremye told the BBC.
The Rwandan high commissioner in Mozambique was not immediately available for comment.
A previous plan to kill him in 2016 was foiled as he had unexpectedly changed the vehicle he was travelling in, Mr Habiyaremye added.
Mr Karemangingo was among the suspects in the 2019 killing of another Rwandan refugee, Louis Baziga, who was at the time the leader of the Rwandan community in Mozambique.
But he was cleared of involvement in that killing and played no role in Rwandan politics, Mr Habiyaremye said.
The death of Mr Karemangingo comes three months after the arrest by Mozambican police of Cassien Ntamuhanga, a former journalist turned opposition activist who had escaped from prison in Rwanda. He has since been held incommunicado.
Three weeks ago there was an attempt on the life of the secretary of the refugee association, the chairman said.
Human rights organisations have consistently accused Rwanda of targeting or killing opposition members abroad who are involved in - or perceived to be involved in - activities against the government.
The Rwandan government has also accused some Rwandan refugees of carrying out or financing activities aimed at destabilising the country.
In neighbouring South Africa, a leading Rwandan opposition politician Seif Bamporiki was shot dead in February.
Rwanda's former intelligence chief Patrick Karegeya was murdered in a hotel room in Johannesburg in 2014.
Former army chief Gen Faustin Kayumba Nyamwasa was shot and wounded in the same city in 2010.
Rwanda's government has always dismissed suggestions that it was involved in any of these killings or attempted killings.
