As we understand the word," said the old Doctor. "Though, mind you, there may be a third factor. But, in my heart, I believe that it is a matter of chemistry; Conditions and a suitable medium; but given the Conditions, the Brute is so almighty that it will seize upon anything through which to manifest itself. It is a Force generated by Conditions; but nevertheless this does not bring us one iota nearer to its explanation, any more than to the explanation of Electricity or Fire. They are, all three, of the Outer Forces—Monsters of the Void....
- William Hope Hodgson, "The Derelict"
I've been thinking about a Spelljammer recently that keeps the basic concept but utterly jettisons the feel or flavor. Spelljammer has never felt me to be about exploration, rather the vessels flying through spaces seem a means to an end. There's nothing wrong with that, but plenty of science fiction literature paints space as a place for confronting the unknown. This is really a perfect fit for Spelljammer where its pre-modern, "magical" spacecraft put the stars within reach but not the science to understand any of it. Not that there is necessarily science as we know it to understand, in any case.
I think I would look to the horror/adventure stories of William Hope Hodgson, specifically his nautical yarns like The Boats of the Glen Carrig, "The Voice in the Night," "A Tropical Horror," and "Demons of the Sea." A little pseudo-science borrowed from his Carnacki stories could only help.
The characters are competent space-hands, perhaps mildly colorful rogues like Howard's Wild Bill Clanton or just working stiffs like the crew of the Nostromo in Alien, not bold explorers or science fantasy swashbucklers. Their jobs involving them going through places that are not (usually) inhabited by hostile species of space orcs or the like, but are instead fundamentally almost wild, always strange. Weird danger can rear it's head at any time, and your vessel is just another ship that disappeared in the Void.
Weird phenomena should be encountered as frequently as monsters, I think. Monsters, when they do show up should be unfamilar, and probably not seen enough to become mundane.
Beyond the stories of Hodgson and Alien, other potential sources of inspiration could be the comic series Outer Darkness, the science fiction stories of Clark Ashton Smith, Poe's Narrative of Arthur Gordon Pym, and of course, Moby Dick.