The Cabinet Secretary prepared a paper earlier in the year on the transition periods between governments, including the process for a hung parliament – it’s been published here.
The key quote is: “Where an election does not result in a clear majority for a single party, the incumbent Government remains in office unless and until the Prime Minister tenders his and the Government’s resignation to the Monarch. … As long as there is significant doubt whether the Government has the confidence of the House of Commons, it would be prudent for it to observe discretion about taking significant decisions, as per the preelection period. The normal and essential business of government at all levels, however, will need to be carried out.”
In other words: whatever happens, short of a clear majority, Brown remains in power until the dust settles.
As to just how the dust settles, this summary points out an interesting wrinkle: the Lib Dems have a system in place to control how a coalition is decided upon. Clegg would need the explicit support of three-quarters of the MPs and of the party executive; failing that, he could appeal to a special conference, and need a two-thirds majority there; failing that, he could take it to a ballot of all party members. I can’t see it ever dragging on that long, but it is interesting to know that it’s not simply a matter of the party heads making a decision and then moving swiftly on to government.