During high workload night single pilot helicopter hoist flight an error in automation selection was not detected until a passenger warned the aircraft had descended to just 59 ft.