The Manor

by Mac Griswold

  From atop a wagonload of hay, a young woman in a straw boater, perhaps Cornelia Horsford, watches the farm crew pitch up the last forkfuls. The far horse wears muslin caps on its ears to keep the flies off.

  The manor’s guest roster also reflects the handoff from theology to science as the great preoccupation of nineteenth-century America, particularly in New England and above all at Harvard. Down through the manor generations, beginning with George Fox, the most notable visitors had been preachers. By contrast, the Horsfords collected an exceedingly clubbable fraternity of scientists. Asa Gray, “the father of American botany” and Darwin’s champion, was the best known and most distinguished. During his thirty years at Harvard he became the de facto coordinator of American botany, receiving specimens from all over the nation and corresponding with colleagues worldwide. His Manual of Botany survives as a standard reference work, and his many textbooks gave two generations of American schoolchildren their first glimpse of natural history. Sylvester Manor tradition pictures him arriving with a sapling copper beech in 1877 and planting it himself. The huge tree—the tree that so amazed me on my first visit—stands at the water’s edge today. Gray also took an interest in the younger Horsfords: in the vault, a tiny yet heavy metal spirit level, whose green bubble still doggedly floats to dead center in the glass, bears the label “To Cornelia Horsford from Asa Gray.” It’s as if the botanist recognized Cornelia not only for her interest in gardens and houses, which had become the American woman’s domain, but also for an ability to appraise her physical surroundings with a precision any man might emulate. Cornelia, who stuck the red-edged schoolroom label to the level, clearly treasured this acknowledgment.

  Meaning and Memory

  Except for Cornelia, who entered the world in September 1861, the Horsford girls were born in the run-up to the Civil War. All of them spent their early childhood in the anguished atmosphere of wartime Cambridge. As children, their playmates and schoolfellows (both boys and girls) at local private elementary schools included many of the offspring of Cambridge and Harvard luminaries. Growing up together, the children became lifelong friends, forming a close-knit second generation of Cambridge society. In the fall of 1861, Cambridge’s women and girls, including Longfellows, Danas, and Horsfords, formed a sewing circle like many others in the North, where they made clothes and bandages for soldiers.

  As young adults visiting Shelter Island in the 1870s and ’80s, some of the girls’ closest friends would have gazed with special intensity at the somber burying ground near the big white oak (which still stands) silhouetted against the farm barns. (No white pine plantation had yet been planted to enclose the graveyard in its own private shield of darkness.) For Richard Henry Dana III and his sisters, Rosamund and Lily, and for Benjamin Robbins Curtis Jr., the cemetery and its memorial embodied what the historian Michael Kammen has called “the imperative of memory,” as opposed to “the comfort of amnesia.”

  The Danas’ father, Richard Henry Jr., a founder of the antislavery Free-Soil Party, had eloquently—and futilely—defended Anthony Burns, who fled slavery in Virginia only to be put on trial in Boston in 1854 under the Fugitive Slave Act. The elder Dana subsequently watched as the entire Boston militia was called out to prevent the outraged citizenry from freeing Burns as he was marched, surrounded by guards armed with swords and revolvers, to the ship taking him back to Virginia. All three Dana children went to elementary school in the Longfellow House on Brattle Street, which neighborhood children of both sexes attended. Rosamund and Lily Dana were among the Horsford girls’ closest friends and schoolmates.

  Benjamin Robbins Curtis Jr., who married Mary Gardiner, the Horsfords’ eldest daughter, in 1877, was the son of a Supreme Court justice who had filed one of the two dissents in the infamous Dred Scott case twenty years before. Scott, a slave, sued for his freedom in 1847 on the grounds that he had lived both in Illinois, a free state, and in Wisconsin, then part of federal territories and not yet a state. Ten years later, the issues before the Supreme Court were whether it had jurisdiction to hear the case and whether Scott could qualify as a citizen of the United States. (The great unspoken issue was, of course, the spread and the legitimacy of slavery.) Chief Justice Roger B. Taney, in a manifestly pro-Southern majority decision, stated that the framers of the Constitution viewed blacks as “a subordinate and inferior class of beings who … had no rights or privileges but such as those who held the power and the Government might choose to grant them.” The decision also overturned the Missouri Compromise of 1820, by whose terms Congress in 1850 had limited slavery’s westward expansion in the territories that had not achieved statehood.

  The senior Curtis, a legal conservative—and no abolitionist—first noted in his dissent that at the time the Constitution was ratified, some African Americans had been recognized as citizens in states both North and South and thereby had automatically enjoyed federal citizenship, and that therefore Taney’s “jurisdictional” objection was ill-founded. Curtis went on to state that since Congress on a number of occasions had legislated with respect to slavery in the territories, the Missouri Compromise was a valid exercise of Congress’s authority. He thereby held that Scott’s residence in Wisconsin, not then a state, had made him a free man. (Curtis’s views on slavery, property rights, and legal process were complex: he had once argued that a slave owner could restrain his human property when in a free state, and in 1862 he attacked Lincoln for issuing the Emancipation Proclamation on the grounds of abuse of executive power.)

  Curtis released his opinion, which was published at once; Taney withheld his opinion “for revision” and refused to let Curtis see it for a month, claiming that Curtis’s intention was to discredit him. The Court’s decision, although hardly the cause of the Civil War, further inflamed passions on both sides of the political divide. The personal breach between the justices over the case became so rancorous that Curtis soon resigned and returned to the practice of law in Massachusetts. Scott was manumitted by his owners’ sons shortly after the decision came down. He died nine months later. The Thirteenth Amendment, passed by Congress with difficulty in 1865, abolished slavery; the Fourteenth, passed in 1868, made all people born within the United States citizens of the nation.

  Many Cambridge men of the elder generation, such as Oliver Wendell Holmes and Wilkie James, William and Henry James’s brother, had gone to war to save the Union, not to free the slaves. Nonetheless, they had taken part in the terrible slaughter. At Sylvester Manor, the children of that generation stood in one of the few places where the fact of slavery in the North—obliterated almost everywhere else—was inescapable. By then in New England and New York, African Americans were a visible part of society, particularly in the cities. However, the manor memorial brought sharply into focus not only the slaves who had lived in the North for two centuries, but also generations of those who had kept them in bondage.

  The North, as it retreated into an idealized New England past, seldom acknowledged the apparition of Northern slavery except as it was displayed in iron hitching post figures the size of small children, often disfigured as dwarves, often in livery or comic dress, with distorted faces like gargoyles. At most eighteenth-century New England properties where slaves had created the original basis of prosperity, once the system was abolished, the entire visible apparatus—the garrets and slave kitchens and other occasional housing, the ephemeral gardens—simply fell down, grew up into weeds, or became appropriated for different purposes, just as happened on Shelter Island.

  The Horsfords—whatever their private ruminations and reservations—wished to do justice somehow to the complex conflicting memories clustered around their place. They inscribed the big glacial boulder near the burying ground with the words THE COLORED PEOPLE. Today, this phrase is so linked with segregation that it raises our hackles. For the Horsfords in 1884, however, the words managed to confer a dignity and humanity that “slaves” would not have done. Moreover, referring to “colored people” rather than “Negroes” descr
ibed the reality of the site, a reality in which we now can read the long, mingled history of blacks and Native Americans.

  The business of commemorating the Sylvester family swung into high gear in 1884. On a beautiful July afternoon, the Horsfords, along with their relations and friends (including many from Cambridge) and the general population of Shelter Island, trooped to the head of Gardiners Creek. There, in the small eighteenth-century cemetery that has come to be called the Quaker Graveyard, stood a freshly carved table monument to Nathaniel Sylvester, surrounded by a motley collection of old grave markers recently fenced for the unveiling. The top of the brownstone tablet on its stout legs memorialized Nathaniel, by then dead for some two centuries, as the Quaker Protector. The steps that surround it are dedicated to the early Friends who visited Shelter Island. Inscriptions on the treads detail the executions, scourges, maiming, mutilations, and imprisonments that Quakers endured at the hands of Boston Puritans. The first part of the inscription on the tabletop displays the Brinley crest and an encomium of Nathaniel as a faithful, intrepid, and hospitable Englishman. Beneath the table, the supporting base lists the succession of Shelter Island’s proprietors, beginning with the Manhansetts, and traces a descent through the female line from Anne Wase Brinley, Auditor Thomas’s wife. This inscription recognizes the “Daughters of Mary & Phoebe Gardiner Horsford, Descendants of Patience, daughter of Nathaniel Sylvester” who have set up “with Reverence and Affection … a memorial to the good name of their ancestor.”

  A big crowd, dressed in their best, came to the old cemetery at the head of Gardiners Creek to celebrate the 1884 dedication of the monument to the Sylvester family and the Quakers.

  Only one other living woman was publicly recognized at the creekside graveyard. After the minister blessed the monument and Eben spoke at length, and after descendants of Mary Dyer and the Southwicks read poems, including stanzas that James Greenleaf Whittier had composed especially for the occasion, Eben called on Julia Havens to rise in the audience. He then addressed her in what a local newspaper called “one of the pleasantest incidents” of the proceeding. “This venerable colored woman, now in the neighborhood of 80, has been a faithful servitor … For the last sixty-five years, there has not been an event of sorrow or joy in these three families in which she has not participated.”

  Julia had sold the last of her land to Eben almost twenty years before. The 1880 census finds her in Greenport, home to a growing black population. It is quite likely she lived in the manor house as custodian from time to time, because Eben Case, that revered twentieth-century repository of island memory, handed down a description of her presence there as “fierce and feared.” In a supreme irony, she had effectively become mistress of the empty manor during the nine months of the year when the Horsfords were in Cambridge.

  The Picture of the Past

  Like her sisters, Cornelia studied drawing and painting. More than proficient, she produced landscapes, still lifes, and architectural compositions. The history of Sylvester Manor could be similarly “composed.” Her canvas was the manor itself, where she amassed and sorted objects—Queen Anne chairs and Georgian silver with the marks of renowned colonial smiths—as well as legends, hearsay, and fragments of history. Noble Indians? Youghco and Wyandanch could be conjured up by the waterside shell mounds and the stone points spaded up by ethnologist Cushing on the North Peninsula. Loyal African slaves? They spiced Cornelia’s sagas of sugar, rum, and tropical mahogany, of brave mariners, bold entrepreneurs, and dashing planters. No question about who suffered for their faith—somewhere on the property the Quakers Lawrence and Cassandra Southwick rest in peace. A handsome horseman in lace and velvet? Brinley Sylvester fit the bill. Ladies in panniers and mobcaps? The Horsford collection of heirloom satin-stitched samplers evoked delicate fingers plying their needles. Heroes of a young republic? Ezra L’Hommedieu, Thomas Dering, and General Sylvester Dering led the charge. Horticultural talismans of antiquity? Gnarled boxwoods hulked along the garden path. Venerable customs? To the amusement of island natives, Cornelia devised folk “traditions” such as the annual Baptism of the Calves, at which the young animals were garlanded with flowers on the manor house lawn by the local minister while their disconsolate mothers bawled in a nearby field.

  “Being a custodian of the past is primarily its own reward,” writes Michael Kammen, and Cornelia, unmarried at a time when spinsterhood could be as incapacitating as a broken leg, rewarded herself well. She married her house, a serviceable, if sometimes lonely, union. Emboldened by a father who played fast and loose with history, and by the ample supply of relics surrounding her, Cornelia freewheeled through the generations. Like many nineteenth-century women of her class with limited education, she seems to have had few outlets for her curiosity, intelligence, and energy. She chose historical fantasy. She concocted the unlikely romance of a barge rowed by slaves (quite probably a detail borrowed from her father’s trip up the James River to the Tylers’) carrying the Sylvester girls to the Congregational church in Southold, where, as Quakers, they would probably not have been welcomed. And she conjured, among other fantasies, a grateful Charles II bestowing an inlaid traveling knife and fork on Auditor Brinley; a tubercular Latimer Sampson nobly relinquishing his betrothed (Nathaniel’s eldest daughter) and sailing away, never to return; and a modest Mary Burroughs Sylvester telling the local minister that she prized her sewing skills more than all her wealth and standing.

  Most of Cornelia’s yarns have some historical basis, however, and all have survived and thrived into the present day in print and online. To her credit, she put enough stock in the documented past to obtain a copy of Nathaniel’s 1680 probate inventory and other records from surrogate court archives. In England she visited Datchet and the Sylvester tombs in Burford, and she hired a professional genealogist to research the Brinleys (his report rather savagely debunked some of her assertions). Cornelia also directed her architect, Henry Bacon, to draw up detailed plans of the existing manor house before starting renovations. And in 1915 she agitated for an official marker to commemorate the spot revered by Native Americans as the place where Youghco’s funeral procession rested for the night on its way to Montauk in 1652.

  Cornelia squeezed every drop of memory from the place as if it made her more real to herself, which, of course, it did. For her, the past was always as present as the here and now, and, as often happens with lonely people, her retelling of “memories”—even when she had only read about an event—grew ritualistic and ever more elaborate. A remarkably detailed letter she wrote when she was seventy-three lays out her obsessive preparations for East Hampton visitors. As part of their visit, she wrote, the guests would be asked to pass the family’s seventeenth-century silver tankard from hand to hand to drink the health of “the loved and the absent.”

  She cherished even her smallest adventures. Early one spring, Cornelia made a foray to the island from Cambridge, bringing pine tree seedlings—probably the pines that shade the burying ground today, which Alice said were planted in 1900. In a letter to her mother she described how, aboard the sailboat ferry from Greenport, she saw the low wooded shore by Dyd’s Creek where “the cherry & shad [blow] trees were white, looking like clouds caught among the trees.” The island oaks were in flower, she wrote. “The box borders [were] covered with tiny light green leaves … the asparagus was delicious … Violets by the roadside … it is raining but warm.” She closed with the news that “Julia is poorly I have not seen her yet.”

  Cornelia didn’t date her letter, but it can be bracketed by the age of the pines and the death of her mother in 1903. Julia, in her mideighties or perhaps even ninety, was living somewhere on the island. Before 1907 she moved to Sag Harbor, where she had relatives, and there, apparently, she died. An anonymous letter printed in an issue of The Friend, a Quaker publication, mentions the Burying Ground of the Colored People as part of the manor’s history. Dated August 28, 1908, and possibly the work of Cornelia’s sister Katherine, the letter states that Ju
lia “died at Sag Harbour eighteen months ago; by her request her remains were brought here to be laid with those of her forebears.” Julia left no known descendants. Her grave is unmarked. Recognized in life as a “faithful servitor,” in death she remains anonymous, like all the rest of those interred there.

  Both Cornelia and her sister Lilian occasionally felt burdened by the weight of history, and by a disquieting suspicion that some things might not have been exactly as they imagined. They felt the sorrow of learning about the instability that lies at the heart of all things. Such emotions must have ghosted up in stray remarks and slight gestures, stirring the supernatural, magnetic strangeness that every visitor to Sylvester Manor feels. Lilian, who observed, recorded, and recalled more objectively than her sister (her lucid pencil sketches contrast markedly with Cornelia’s impressionistic watercolors) wrote a memoir that she read aloud at a Shelter Island Historical Society meeting. She told of her fear as an eight- or nine-year-old—as real as childhood can make such fears—when she threw a pebble against a glacial boulder in a field, a spark flashed, and then she sniffed a “slight smell—of what? We thought it was brimstone.” Her memoir went on, “We thought the flash was the fire of Hell and that we had discovered an entrance … We never played there again. We never spoke of it again.”

  Alice

  Alice and I are looking at the portrait of Cornelia as a little girl that hangs at the bottom of the front stair. Cornelia’s eyes do not waver. They appear to be fixed on the gauzy middle distance so beloved of Victorian portrait painters. But Cornelia’s gaze is not gauzy—never mind that she is wearing a little pink frock with a frilled neckline, or that she is only about ten years old. If this girl turned her head, she would stare through Alice and me, backward through history. As an adult and with nineteenth-century certitude, Cornelia had firmly reshaped the house and gardens to suit herself and her ideas of what the place had been.