It takes a certain kind of brave (and perhaps foolish) person to join Starfleet and venture into the great unknown, and most of the stories in the Star Trek universe are about an adventure gone wrong. Plenty of brave souls never return from these missions, and considering how dangerous it can be out there, it's amazing that anyone ever comes home.
Famous last words often include, "a mission to a mysterious planet" and the location turns out to be deadly for visitors and residents. There is no shortage of locations in the heavens that nobody ever wants to visit, and a few of them are planets such as these.
7 Ceti Alpha 5
The planet of Ceti Alpha 5 first appeared in The Original Series episode "Space Seed," when Kirk marooned Khan and his followers on what seemed like a relatively safe and even thriving planet, Ceti Alpha 5. Alas, an unlucky celestial event knocked the planet out of orbit and destroyed almost every living thing on it, other than a disgusting little brain-eating space worm.
Khan and his team managed to live for years on this ruined and dangerous place until a Starfleet crew showed up looking for Ceti Alpha 6, a nearby planet that had been destroyed in the galactic cataclysm that turned Ceti Alpha 5 into a wasteland. They would find out that with Khan on the planet, Ceti Alpha 5 was even more dangerous.
6 Qo'noS
There might be some incentive to see the Klingon homeworld for the adventurous and curious, but depending on the timeline this could be one of the most dangerous planets in the universe. For any Federation-race members, this planet is perpetually hostile just from a social point of view. Meanwhile, those that have been conquered and are now part of the Empire would always be regulated to subordinate roles.
The real danger to life and limb on Qo'noS actually comes from its moon, Praxis. In the year 2239, careless mining practices would cause it to explode, and the atmosphere was so badly damaged that the Klingon homeworld was given only 50 more years of habitability. These events formed the storyline of the sixth Star Trek movie, The Undiscovered Country.
5 Genesis
"Genesis allowed is not! Is planet forbidden!" These are the cries of the black-market pilot offering an agitated Dr. McCoy a ride. Considering this shady character has been to some dangerous places, it's chilling that going to Genesis for them is unthinkable.
The Genesis planet only lasted a short time. It was an experiment that was being conducted by Carol Marcus and her son David with the support of Starfleet Command and the Corps of Engineers. The hypothesis was that it could make lifeless planets flush with life, and it seemed to work at first, but David's use of proto-matter caused the planet to tear itself apart.
4 Murder Planet
As if the name wasn't obvious enough, Murder Planet is like a nightmare version of the amusement park-like world that the original crew experienced in the TOS episode, "Shore Leave." That planet also responds to the secret desires of the visitors, but in a positive and playful way that's not dangerous.
According to the Prodigy episodes that feature Murder Planet, "Terror Firma" and "Dream Catcher," this is the only class-M planet in the Hirogen star system. However, it's not really a planet, but a sentient life form that lures in prey using hopes and dreams. This name isn't a joke: this planet is actually trying to kill visitors, and more specifically, eat them.
3 Megas-Tu
This planet only exists in a parallel universe that appeared for an episode on an episode of Star Trek: The Animated Series and it's one of the strangest along with being the most dangerous. Its exact location is unknown, save that it's in a "non-space" area in the heart of either the Alpha or Beta quadrant.
The Megans are a race that relies on magic as opposed to science, which made sense to them because the non-space part of the galaxy doesn't have the same laws governing physics or matter. For those that understand anything about how magic works, however, that can be a perilous planet indeed, especially for non-magic users.
2 Talos 4
There was a time when Captain Christopher Pike just couldn't get a break. He had just finished a dangerous mission on one planet just to land on another even worse one, a place where the worst nightmares are made real along with the wildest dreams.
The Talosians are an advanced race and can read minds and create illusions that seem real. They use these powers to lure unsuspecting travelers to their planet so that they can study them, imprison them, and even breed them. After Pike's survival and escape, this planet was marked as a no-go zone for Federation ships, and for good reason.
1 Rura Penthe
Strictly speaking, this prison planet is an oversized asteroid with only one season: extreme, debilitating cold. This is where Kirk and McCoy were sent after being convicted of the assassination of Chancellor Gorkon in the movie Star Trek V: The Undiscovered Country.
This environment is so deadly that this galactic prison did not need walls, guards, or even a surveillance system. Punishment for anything was always exile to the outside of the prison, which was certain death within minutes.