Good afternoon Chief, you ask "What size wire should I use?"
Lets do the math. A 5 HP 240 VAC Single Phase motor will draw in the vicinity of around 28 FLA which 8 Gauge Copper wire will handle HOWEVER that doesn't account for Voltage Drop. Therefore, I would use AT THE MINIMUM 6 Gauge THWN Copper. Bigger wire would yield less voltage drop and/or to allow for future expansion.
To feed say a 60 Amp 120/240 Volt Single Phase Three Wire SUB PANEL equipped with two pole Main Breaker in that remote outbuilding, you need to run FOUR Conductors out. Two Ungrounded Hots (L1 & L2),,,,,,,,,,One GrounDED Conductor Neutral,,,,,,,,,,,,,One Equipment GroundING Conductor. You will need to install and bond a Grounding Electrode (earth driven copper rod or rods or other suitable electrodes, check local authority or NEC) at the outbuildings electrical service. While the exact rating of the home panels circuit breaker which feeds the outbuilding depends on the wire size and type and enclosure, a 60 amp will likely suffice. DONT FORGET IN THE REMOTE PANEL THE NEUTRAL AND GROUND BUSSES (unlike in your homes main panel) ARE TO BE KEPT SEPARATE AND ISOLATED. The tubs case/frame is bonded to the Equipment Ground Buss but NOT the Neutral.
You could enclose the individual THWN conductors in buried PVC Conduit (so long as NOT under driveway and if so concrete encased or Rigid),,,,,,,,,,,,,,,,,If they make it???? you could direct bury 4 Conductor UF Underground Feeder Cable,,,,,,,,, You might get by direct burying Underground Service Entrance (USE) Cable if its manufactured with 4 Conductors (its typically 3 like to my home from the transformer). NOTE if you were to enclose an Equipment GroundING Conductor in metallic conduit, it needs to be insulated NOT bare as I recall. Heck you could use aerial cable if you like. I LIKE THE IDEA OF BURYING PVC CONDUIT and the use of individual THWN Conductors myself.
John T Longgggggggg retired Electrical Engineer and rusty on the latest codes SO NO WARRANTY but believe this is most likely correct, check with local authority or Utility providers and the NEC if adopted and it applies, versus anything posted here (where's Billy Bob and Bubba lol) is my best free advice.