Mr Lubbers emerged from the "frank" talks to deny accusations that he engaged last year in a pattern of harassment, and insisted he would see out the last year of his term.
"The secretary general didn't ask me to resign," Mr Lubbers told reporters. But Mr Annan made clear that the accusations were causing inconvenience to the UN. Officials had sought to portray yesterday's meeting as routine, but a statement from the secretary general's office indicated otherwise.
"While some other issues were discussed between the secretary general and high commissioner for refugees, Ruud Lubbers, today, their main focus was Mr Lubbers's future in the organisation, particularly in the light of recent developments - including today's press coverage," the statement said. "At this stage, the secretary general does not want to elaborate further."
The main allegation came after a female employee at the UNHCR, the principal UN agency for dealing with refugees, complained in April 2004 that Mr Lubbers had sexually harassed her. The unnamed woman said that at the end of a meeting on December 18 2003 Mr Lubbers asked her to stay behind, placed his hands on her waist, pulled her towards him and pressed his groin against her.
Mr Lubbers vehemently denied the allegations. "That's made up," he said. "There were two witnesses in the room who very clearly saw that I ushered the lady out of the room with my hand on her back, and that was all.
"I call it familiar but certainly not sexual harassment."
Other employees later came forward to suggest that Mr Lubbers, the longest-serving Dutch prime minister, had also made unwanted sexual advances towards them.
Excerpts of a report by the UN's office for internal oversight services (OIOS), published in the Independent this week, concluded: "Mr Lubbers did engage in unwanted physical contact with a subordinate female staff member. New allegations that came to the OIOS's attention during the investigation were also examined and indicate a pattern of sexual harassment by Mr Lubbers."
The episode has started to have a serious effect on staff morale at the UNHCR. Ron Redmond, a spokesman for Mr Lubbers, told reporters in Geneva yesterday: "No organisation can go through something like this and not find it difficult, especially when it in volves the leader. Of course it's not easy."
A UN insider added yesterday: "It's like a black cloud that follows him around wherever he goes. He wants to talk about the work and people keep asking about this. Of course it has an impact."
Mr Redmond said Mr Lubbers continues to deny the charges and that the findings of the report by themselves gave a biased account of events.