Here is my take on the problem, without the restriction on adding attendees to the speaker list if they are already on it.
(That is easy to add.)
Selection of an attendee from the Attendees lpk automatically adds him to the end of the speakers list, but he can be removed after selection from the speakers (actually speaker order) list.
I added options to rearrange the speaker order.