Senior figures from each religion will discuss details of the planned talks, due to involve Pope Benedict XVI.
Catholic-Muslim relations soured after a 2006 speech in Germany, in which the Pope quoted a 14th Century Byzantine emperor’s criticisms of Islam.
The Regensburg speech provoked Muslim fury and triggered protests worldwide.
But it also prompted 138 Muslim scholars to launch an appeal to the Pope for greater theological dialogue, called the Common Word.
Since then the number of signatories to the appeal has grown to over 200.
Terrorism issue
This week’s two-day talks mark the start of a process which, it is hoped, will lead to the larger meeting, which is likely to take place either at the Vatican before the summer, or at the Pope’s summer residence, south of Rome, in August.
A representative of the Italian Islamic Religious Community said that terrorism would be one of the subjects to be discussed at that meeting.
Although Pope Benedict repeatedly expressed regret for the reaction to his speech in Regensburg, he stopped short of the clear apology sought by Muslims.
But the BBC’s Frances Kennedy in Rome says the Vatican is now clearly convinced of the need for a wider, if more difficult, dialogue with Islam.