Koh Samet is 7 kilometers off the mainland coast in Rayong province Thailand.
There are public ferries that leave every hour from 8am - 6pm. A ticket on the public ferry costs 50 Baht one way. The public ferry will take around 25 minutes to make the trip across to Samed Port pier.
Samed Port pier is only 8 minutes walk across the island to the centre of Kai Saew Beach, maybe 15 minutes if you are lugging your luggage.
Share speedboat 800 baht return trip. They can drop you directly to your beach side resort. The trip from the mainland takes around 10 minutes to the Samed Port or 15 minutes around to the other side of the island where Kai Saew Beach is.
If you go with the shared speedboat option you may have to wait for them to get some more people to share the boat with you. In peak times this wont take time, 10 - 15 minutes. I have gone across on a shared speedboat with only 3 other people so they don't wait to fill it.
You can hire a personal speed boat for 3500 Baht return trip for up to 10 people.
Car from Bangkok will take around 3 hours to drive to Banphe Pier Terminal. Then add the time for the public ferry or shared speedboat. There are also mini-bus services which leave from Bangkok's Eastern Bus Terminal in Ekkaimai which take around 3.5 to 4 hours to get to Banphe Pier.
Car from Pattaya will take just over 1 hour.
If you want to travel further east you can get to the ferry port that crosses over to Koh Chang in around 2.5 hours.