Area X - the Southern Reach Trilogy by Jeff VanderMeer

This trilogy would qualify, I suppose, as Science Fiction, since it features aliens, or as Horror, since some deadly entities are bent on harm. But there's far more to it - this is almost a treatise on the natural world, and how the decay of human habitation opens the way for vegetation and wildlife to reassert themselves in territory formerly their haven.

Book I - Annihilation concerns the Twelfth Expedition into Area X, a place formerly a coastal human community: a lighthouse and a couple of villages. But Something has instigated change - humans are gone, and the rate of decay and fecundity of nature have accelerated.  Previous expeditions have ended badly - the venturers do not return, or die of aggressive cancers soon after emerging, or go insane - in many cases, they kill each other.

A few characters stand out: the Biologist, whose name we do not learn, a tough resourceful woman whose husband accompanied a previous expedition but returned so changed she wants to go, possibly for vengeance, or at least to understand what happened to him; and the Director, billing herself as the Psychologist, who uses hypnosis and drugs to control the rest of the crew - except the Biologist, who has made herself immune.

What they find in their explorations is a teeming beautiful wilderness run rampant, as they come up against the limits of their capabilities.

Book II - Authority brings into focus Southern Reach, headquarters of the organization dedicated to understanding and containing Area X. Here our third main character shows up: Control, son of high-up functionaries at Central (think CIA), recruited by his cutthroat mother in her last push to make a success of him. He's assigned to support the Director, and also to figure out why this group isn't accomplishing its goals. The employees are, variously: mad, prone to peculiar habits, hyper-aggressive, reclusive, obsessed, or in zombie-like states of confused stasis. Even the building has an "off" personality. It should - it stands near the Border, on the other side of which Area X flourishes despite their efforts. In this book we delve further into the Biologist, who having returned in an altered state from the Twelfth Expedition, is imprisoned at Southern Reach while administrators, including Control and the Director/Psychologist, attempt to probe her mind.

But chaos descends: leaks over from Area X. Eventually the organization cannot function. Control and the Biologist flee up the coast.

Book III - Acceptance takes us back into Area X, providing history of the place pre-invasion as well as insight into other, more secret, attempts to contact and direct the alien presence. And we learn more about the forces that continue to transform land, air and water.

These books are evocatively written:
"The wind picked up, and it began to rain. I saw each drop fall as a perfect, faceted liquid diamond, refracting light even in the gloom, and I could smell the sea and picture the roiling waves. The wind was like something alive; it entered every pore of me and it, too, had a smell, carrying with it the earthiness of the marsh reeds." 
and
"Control  still couldn't tell from his examination of the records... if the iterations of [the actually 38 expeditions numbered up to Eleven] had started out as a clerical error and become codified as process (unlikely) or been initiated as a conscious decision by the director, sneakily enacted... as if always there. A need to somehow act as if they weren't as far along without concrete results or answers. Or the need to describe a story arc for each set of expeditions that didn't give away how futile it was fast becoming." 

In this time of humans trampling the natural world as if intent on destroying what is actually our only home, this horror story/ science fiction/what-have-you gives a voice to Earth, gives it a means to push us back and renew its dominion.