The Manor

by Mac Griswold

  The author and publisher have provided this e-book to you for your personal use only. You may not make this e-book publicly available in any way. Copyright infringement is against the law. If you believe the copy of this e-book you are reading infringes on the author’s copyright, please notify the publisher at: us.macmillanusa.com/piracy.

  To Emma Tara Johnston Lee

  Contents

  Frontispiece

  Title Page

  Copyright Notice

  Dedication

  Map of Sylvester Manor

  Family Tree

  Preface

  1. The Discovery

  2. Living with the Indians

  3. Amsterdam

  4. The Other Island: Barbados

  5. Nathaniel’s Middle Passage

  6. Before the Whirlwind

  7. The World Turns Upside Down

  8. “Time of Longing”

  9. Where They Lived

  10. How They Lived

  11. In the Ground

  12. “Oppression upon the Mind”

  13. Quaker Martyrs, Quaker Peace

  14. “A Duchman in His Hartt”

  15. “Children of the Founders”

  16. Illusion and Reality

  17. The Doors

  18. Family and Slavery

  19. Summer Colony

  20. Ladies of the Manor

  The Immigrants

  Sylvester Manor Time Line

  Time Line of World Events

  Photographs

  Notes

  Bibliography

  Acknowledgments

  Index

  Illustration Credits

  Also by Mac Griswold

  Copyright

  Preface

  When I began my research for this book in 1997, I became the tourist that the cultural geographer J. B. Jackson described as “the solitary, uninformed traveler, setting out, hardly knowing why, in search of a new kind of pleasure and a new kind of knowledge.” I found both at Sylvester Manor on isolated Shelter Island, set between the North and South Forks of Long Island. It is the only former slaveholder’s plantation north of the Mason-Dixon line that still exists with papers, architecture, and landscape all in one place to tell its story. The property remains in the hands of the eleventh generation of the European colonists who settled there in 1651. The graves of Africans and Manhansett Indians lie quietly under the pines.

  Though I have walked through dozens of historic landscapes, some abandoned and others restored, I never felt the immediacy of history so intensely as at Sylvester Manor. What sounds did these Shelter Islanders hear? What did they eat, how did they talk, what did they smell (and smell like)? Which cultural traditions, social ambitions, and political forces had shaped their experiences? I needed to get a clear picture of how oppression, war, religious beliefs, and a morality that accepted chattel slavery had operated at Sylvester Manor. At first I saw only that a line ran from the canals of Amsterdam to this handsome, smallish house and to scientists and poets in nineteenth-century Cambridge, Massachusetts. Soon the manor landscape swelled outward to include the Atlantic World as far as the West Indies and the slave-trade castles of the African Gold Coast. What I saw at the manor was connected to the flow of people, ideas, plants, and animals from a dozen other landscapes on four continents.

  The written history of the manor has lasted for more than three and a half centuries, but half of this book is devoted to its first eighty years, because, as Richard Rabinowitz, who in 2005 mounted the first exhibition about the pervasive presence of slavery in New York, said, “The founding of Sylvester Manor is as exciting as the moment when people stepped across the Bering Strait. Here African, European, and Native American people came together to build an economic enterprise that was rooted in an ecology, a technology, and a set of racial hierarchies that continue to mark our lives today.”

  I made many discoveries. To find out where slaves such as Reuben and Chloe slept, I tiptoed through an ancient probate inventory, reconstructing the path of the appraisers through each nook and cranny of the house. A man at the Plimoth Plantation museum, near where the Pilgrims landed in 1620, sang a psalm tune for me that Nathaniel Sylvester had sung as a child in 1620s Amsterdam. A snowfall picked out in white a lost eighteenth-century entrance drive. I learned that the most lethal malarial mosquito in colonial Jamestown (where Nathaniel spent almost a year in 1644) had striped legs. On a June morning, I opened the door to the secret vault where priceless family documents were kept and smelled rotting paper (the antique plumbing above had burst). We saved them.

  One midnight I asked myself, Why do we say “slave plantation”? It is, as one friend said, “as if they were growing slaves” like cotton or sugar or tobacco. It’s a dehumanizing phrase we’ve come to live with, to romanticize. When I started to write, I was aware that my family tree, like that of many white Americans, includes generations of slaveholders who migrated across the South from Virginia to Texas. As my understanding of manor life deepened, I was seduced by the beauties and terrors of the place. By the time I finished writing about this Northern plantation, my bones had been rattled by the everydayness of slavery and its long legacy in our country.

  I have written this study of a single piece of land and its inhabitants expecting that it will light up a long stretch of our American history. I hope my sensory impressions, combined with the historical facts—the moral paradoxes, combined with my guesses as a writer—remain as fresh and startling for the reader as they were when I gathered them.

  1

  THE DISCOVERY

  Boxwood

  It has taken us about twenty minutes to get into Gardiners Creek from a mooring in the town harbor of Shelter Island, set snugly between two peninsulas, the North and South Forks of Long Island, New York, whose tips stretch out to touch the western edge of the Atlantic Ocean, where the Gulf Stream runs close to the continent as it flows north and east toward Europe.

  The tide is full as we ease the dinghy through a big pipe that supports a bridge, a bridge so low to the water that we have to flatten ourselves on the thwarts to get through without banging our heads. The pipe acts as an echo chamber, even for a whisper. It is too narrow for us to use the oars, so we brace our hands on the curve of the low ceiling and push. A friend has brought me here to see “something,” but he won’t say what. It is a summer day in 1984.

  Finally out in the light, we see nothing but woods looming down to the water. Then, about a half mile off, at what seems to be the end of the inlet we have found our way into, phragmites and cattails fringe the shore. On our right are a few roofs, half hidden in the trees. On our left, toward the east, lies only a salt marsh where white egrets stalk in the long grasses. No houses. Not even a dock, a boat, or a mooring. Gulls wheel above a low hill covered with large trees: oak, hickory, walnut. Turning east means seeing land set back in time, so far back it looks as if it had never been inhabited. We blink, feeling tension rise between the modern world we’ve left behind so abruptly and the past we are rowing into.

  A mudbank lies ahead, lurking under shallow water, and we get stuck, briefly. It is only when we steer into the tide channel, stirring up silty brown clouds in the water as we pole ourselves with the oars, that we first see the big yellow house. From its hip roof and big brick chimneys to its well-proportioned bulk, the house quietly acknowledges its eighteenth-century origins. I’m in a time warp.

  As we cautiously approach, rowing as quietly as we can, hulking blackish-green boxwoods suddenly loom above the corner of a porch. Buxus sempervirens, or common box, these shrubs look to be an astounding twelve feet tall. Is this an illusion created by looking up from such a small boat? No, it isn’t. They’re gigantic. We’re used to seeing boxwoods as frilly edging around flower beds, or modest gree
n bosses set to either side of a front door. Slow-growing boxwoods do well in this moderate climate, tempered by the surrounding waters of the Atlantic, but they seldom exceed eight feet north of the Mason-Dixon Line. As a landscape historian and a gardener, I feel that these linebacker giants must be very, very old. Boxwoods, one of the few shrubs that in optimum conditions can live for hundreds of years, are like the guardians of history. The dense evergreen foliage of specimens like these seem to hold memory in each small, shining leaf.

  Boxwoods, seen here in a family photograph taken before 1908, have guarded the central garden path at Sylvester Manor for at least two hundred years.

  The reflection of the house in the glassy water doesn’t tremble. No wind. I hold my breath too, as if the building itself would disappear if the water moved.

  My friend idles his oars. As we pull closer to shore, we see what looks like a place to land. Apparently a crude boulder wall, it has a few stone steps rising from the water. Nobody is around. Can’t resist. I climb the steps; my friend waits, amused but concerned that I’m going to get caught trespassing. However, he’s used to my marauding tactics in deserted gardens.

  Once up the steps, I see that the wall is actually a land bridge carpeted with grass. Wonderfully strange. On the far side of the bridge, another part of the inlet we’ve ventured into continues. A track runs down from the woods we just rowed past, heading for the house. I follow it. I knock softly at the front door, then louder. I call. Still no answer, nobody home. Great. So I check out the gangling, rusty windmill and water tank on stilts near a big barn, and an old cannon facing the water, sitting on a crude wooden carriage. Where the heavy lower branches of a gigantic copper beech have reached the ground, they have taken root and sprung up into a copse surrounding the mother tree. The tree bulks almost as large as the long, elegant house that was clearly built over many generations, lying composedly in the sun on a summer afternoon.

  On the east side of the house, away from the water, white pickets protect a rambling flower garden. The big boxwoods I glimpsed from the boat flank the garden gate. Now that I am close to them, I see they are indeed twelve feet tall, and fifteen feet broad. Inside, the garden is cut by a central path running straight and narrow through two lines of more boxwoods. The far end of the path telescopes to a distant gate, a view that seems to stretch back at least two hundred years. Grand gardens were built like this then, fenced rectangles on axis with the main house.

  I climb back into the rowboat, stunned. This place isn’t self-consciously “historic”; it’s not restored in any sense. It has simply been here, waiting for time to pass. Waiting for me.

  Suddenly we hear water moving. The Atlantic tide begins to empty itself in a thin silver coil from the inlet above the stone bridge. Twice a day, every day, for how many years? This place stands still, outside any ordinary dimension of time or space, but tide and time move through it. We row out of Gardiners Creek, moving with the tide.

  * * *

  Curious to find out exactly where I’d been in the dinghy, I checked a map of the island to get my bearings, and asked the owner of the local sandwich joint if he knew who owned the place: an Andrew Fiske, and his wife, Alice. I wrote asking to meet them. I had trouble with my letter, trying out various versions of “Who are you and what is this place?” At last I settled on something economical and true, but pitifully inadequate to express the curiosity I felt. “Dear Mr. and Mrs. Fiske,” I wrote, “I rowed into the creek below your house and could not resist walking around your lawns and garden,” and I finished by saying I was a garden historian and would like to meet them and learn about the history of their house. Several months and three letters later, I received a reply, and an invitation to visit.

  I pulled through the white-painted cement gates opposite Shelter Island’s lone supermarket and rattled down the long wooded driveway, two worn sandy tracks separated by a grass ridge. Not much traffic here, that was clear. I wondered whether the couple I would meet could possibly match the magic of my waterside introduction.

  To the Staircase

  Two small figures stand under the front portico, one in a wrinkled linen jacket and open-necked checked shirt, the other in skirt and blouse, with a chintz mobcap topping the ensemble. Sensible shoes on both, and an air of the fifties about them. Pleasant greetings, a swift assessment from Andrew’s pale blue eyes, a big lipsticked smile from Alice. As they invite me into the darkened front hall, light hits the polished brass doorknob. A large key gleams in the heavy, square lock. I step across the threshold and into a history project spanning eleven generations, three and a half centuries, and four continents. Here, where the Hamptons, jittery playground of the rich and famous, are only eight miles away, I am astonished to learn I’m meeting a member of a family that has lived on this tranquil-seeming property in an unbroken line since 1652.

  Alice, clearly a character and perhaps the boss, departs for the garden. Andy asks whether I’d like to see the house. As we continue to stand in the front hall, he regales me with his romantic version of a family story passed on to him and shaped mostly during America’s Colonial Revival, between 1876 and the twenties, by his great-grandfather Eben Norton Horsford and his youngest daughter, Cornelia Horsford. I would review and reassemble this history many times.

  Eventually I combined the results of a decade of research into Andy’s documents with information from oral and traditional histories; intellectual, economic, agricultural, and architectural history; and archaeology, dowsing, and dendrochronological analysis of the timbers of the house. This book offers an interpretation of the Sylvester Manor site that is more startling, more full of gaps, and more complex and paradoxical than Andy’s tale—and it still leaves the place and its history open to further study. The colorful Andy version rings with a certainty that my version doesn’t possess, but I would discover many disconcerting errors and gaps in his tale.

  His account runs like this: In 1652, the dashing Nathaniel Sylvester, son of an English merchant family, sailed north from Barbados, where his family owned sugar plantations that depended on the labor of hundreds of Africans. With him came his teenaged bride, Grizzell Brinley, daughter of Charles I’s court auditor, Thomas Brinley. After a dramatic shipwreck, the newlyweds landed on Shelter Island, where a large and comfortable house (built previously by Nathaniel and his servants) stood ready to receive the young couple. Nathaniel and his brother Constant and two other Barbadian planters, Thomas Middleton and Thomas Rous, had bought the island from Stephen Goodyear, the deputy governor of New Haven Colony, who had bought it as a speculative venture in 1651 from the estate of the Earl of Stirling, its first English owner, for 1,600 pounds of the unrefined brown sugar known as muscovado. Andy carefully explained that even though Nathaniel and Grizzell were not Quakers themselves, a sense of noblesse oblige moved them to offer their isolated island as a sanctuary for the earliest Friends fleeing the savage persecution of Boston Puritans. The young couple invited George Fox, the Quaker founder, to visit in 1673 and preach the Inner Light. Because Grizzell’s father, a high civil servant who served in the Royal Exchequer, had the ear and the gratitude of the king, Charles II, the persecution stopped. The Sylvesters enjoyed good relations with the Indians and respectfully purchased the island again from the local Manhansett chief, Youghco. They hobnobbed with, and married into, New England’s ruling elite. Last but not least, Grizzell bore twelve children, eleven of whom lived to maturity, thereby begetting the long line that led to Andy Fiske.

  Andy fast-forwards two generations to Nathaniel’s grandson, the fashionable Brinley Sylvester, born in Newport, who inherited in 1733. To Brinley, the ramshackle eighty-year-old family mansion seemed very out of date; he tore it down and built this house. We are standing in the east parlor as Andy tells me this, a room of extraordinary beauty and strangeness. Beautiful because the proportions and the fully paneled walls are exquisite, and all the more so for being so very early Georgian high style in this now remote corner of the world, once part of a th
riving maritime economy. Strange, because there are only two coats of paint on these walls, through which the silvery old wood shows in patches. The bottom coat is that acid blue-green so fashionable in the mid-eighteenth century. The top coat is a modest biscuit color, emblematic of good taste, applied sometime in the 1840s, just before wall colors turned dark and Victorian. How peculiar and lucky it is that no one put on a third coat, I think to myself.

  On one wall hangs a dingy portrait of a solemn man with a very large nose, Andy’s notable ancestor Ezra L’Hommedieu, Nathaniel Sylvester’s great-grandson, an American statesman who died in 1811. As we turn away from the painting, I am startled to see the same nose confronting me in the flesh—on Andy’s face. The doubling of past and present doesn’t stop with the portrait. Andy gently coaxes me to notice that the same blue and white pearlware basket depicted in the portrait of a little girl in red over the mantel now sits safely on a shelf behind the parlor door. Despite the past, how strangely lived-in this room feels, with its big bunch of plastic daffodils, box of mah-jongg tiles, and folding card table, vintage 1950.

  It hits me that Andy and Alice are living in a place that is hidden away from the outside world—but in plain sight. Everything is simultaneously ghostly and absolutely present. I’ve lost my bearings and have no idea how to deal with the vast world laid out before me. But because I’m a landscape historian, and landscape historians always look out the window to see what’s there, I peer through the east window. Yes, the axial garden path that marches through the giant boxwoods does align exactly with the view from this window, meaning there is a good chance that the builder of the house knowingly connected the two. I feel I could follow this slim lead on the ground and backward into history. We head into the library, dimmed by half-drawn blinds, where the rare histories of rail that are Andy’s passion stand shelved beneath a sepia photograph of a fur-hatted “Papa” Horsford, as Professor Eben Norton Horsford, Andy’s great-grandfather, is called by the family. The personal fortune of this self-made man gave the manor a sorely needed boost in the 1850s. I’m told that a framed crewelwork strip hanging beside the photo came from a set of eighteenth-century bed hangings. Andy says that the bed once furnished a room on nearby Gardiners Island, a sister manor first settled in 1635 by Lion Gardiner, a Dutch-trained English fortifications engineer. A Gardiner descendant married the girl in the red dress, a Sylvester descendant. I nod. Of course.