The “Wonder Sea” vessel was carrying 48 passengers and five crew members when it capsized because of sudden heavy rain, the VNExpress news site said.
Most of those on board were families visiting from the capital Hanoi, with more than 20 children among the passengers, it said.
Border guards had rescued 11 people and recovered 34 bodies by Saturday evening, it added.
Overnight three crew members’ bodies were found in the cabin and rescue efforts continued into Sunday morning to find the five people still missing.
One of the rescued, a 10-year-old boy, told state media outlet VietnamNet: “I took a deep breath, swam through a gap, dived then swam up, I even shouted for help, then I was pulled up by a boat with soldiers on”.
Prime Minister Pham Minh Chinh sent his condolences on Saturday to the families of the deceased and called on the defense and public security ministries to conduct urgent search and rescue.
Authorities would “investigate and clarify the cause of the incident and strictly handle violations”, a government statement said.
Tran Trong Hung, a resident in the Ha Long Bay area, told AFP: “The sky turned dark at around 2:00 pm.”
There were “hailstones as big as toes with torrential rain, thunderstorm and lightning”, he said.
Torrential rain also lashed northern Hanoi, Thai Nguyen and Bac Ninh provinces on Saturday.
Several trees were knocked down in the capital by strong winds.
The storm followed three days of intense heat, with the mercury hitting 37 degrees Celsius in some areas.
Mai Van Khiem, director of the National Center for Hydrometeorological Forecasting, was quoted in VNExpress as saying that the thunderstorms in northern Vietnam were not caused by the influence of Tropical Storm Wipha in the South China Sea.
Wipha entered the South China Sea on Sunday gaining strength, and is on course to make landfall in Vietnam early next week.
Ha Long Bay is one of Vietnam’s most popular tourist destinations, with millions of people visiting its blue-green waters and rainforest-topped limestone islands each year.
Last year, 30 vessels sank at boat lock areas in coastal Quang Ninh province along Ha Long Bay after Typhoon Yagi brought strong wind and waves.
And this month, a ferry sank off the popular Indonesian resort island of Bali, killing at least 18 people.
© Agence France-Presse
