Of the 43 people on board there were 24 survivors, according to operator Precision Air.
The two pilots initially survived and managed to speak to local officials from the cockpit but the prime minister says they may have since died.
The plane crashed near the shore at the end of the Bukoba airport runway.
Rescuers were able to wade through the water to bring some of those still inside the plane to safety.
Abdul Nuri was at the airport waiting for the return flight to Tanzania's biggest city, Dar es Salaam, when he saw the plane plunge into Africa's largest lake.
"We were really shocked. People panicked and some started crying and shouting," he told the BBC.
"At the arrivals gate people panicked as well - most of them were waiting to welcome their relatives."
He has spoken to the fishermen who were first on the scene. They told him they managed to get into the plane to rescue people after a flight attendant opened the rear door after the plane had crashed.
The tragedy, which occurred around 08:50 local time (05:50 GMT), has been blamed on bad weather.
Emergency workers have used ropes to pull the ATR-42 aircraft even closer to the shore, and some of the plane's body is now above the water.
Immediately after the crash, the plane was almost completely submerged with only the brown and green tail fin visible.
Richard Komba, a survivor of the crash, told the BBC that when the plane was about to land, the weather had deteriorated, forcing the pilot to reroute the aircraft.
"We were then informed that we would be landing shortly, but there was heavy turbulence. We later found ourselves in the lake," Mr Komba said.
"Water then entered the plane and those sitting near the front were covered by it. I was in the back seat and most of us in the back of the plane struggled to get out."
He said one cabin crew member was struggling to open the aircraft's door, but he was eventually able to be freed.
"When we got out, no boat was there - it took quite long to get rescued but the boat that came was not so good, it was a canoe."
He said the survivors were "scared" because of the number of people trying to fit in the one vessel, but he said other rescue boats arrived minutes later.
Tanzania's Prime Minister Kassim Majaliwa said the rescued passengers were in hospital and were not seriously injured, "but they are shocked and worried".
Earlier, Mr Majaliwa went to the scene and said an extensive investigation would be carried out to determine the full causes of the crash.
"We are still identifying the bodies, but it is highly likely that the pilots might have perished," he said.
The airport has been closed until further notice.
President Samia Suluhu Hassan has expressed her condolences to those affected and called for calm as the rescue operation continues.
The plane was flying from Dar es Salaam to Bukoba via Mwanza.
Precision Air is Tanzania's largest private airline and is partly owned by Kenya Airways. It was founded in 1993 and operates domestic and regional flights.