A leading member of London's successful bid for the 2012 Olympic Games said the legacy promises made by Sebastian Coe would be reduced to rubble if Tottenham Hotspur take over the main stadium.