The Black History of the White House

by Clarence Lusane

  Author Jesse J. Holland believes that Reed’s contribution went beyond simply disassembling the plaster model of Freedom. He argues that the anonymous “black master-builder” described in a New York Tribune article as having worked together with Clark Mills to assemble Freedom “joint to joint, piece by piece, till they blended into the majestic ‘Freedom’ who to-day lifts her head in the blue clouds above Washington” was Philip Reed.46 In light of the aforementioned scenario, Holland is likely correct.

  * * *

  The 1790s were pivotal years not only for the official launch of the nation’s new capital, but also for consolidating the hierarchy of race relations that would persist for the next two centuries. Despite the brilliant wording of the Constitution and other revolutionary documents, the nation was born agonizing over the unresolved issue of slavery. Constantly challenging the Founding Fathers’ rhetoric and soaring words of liberation and democracy was a rising backlash against their racism—a backlash that would not go away.

  The White House would be more or less finished by the date set by President Washington, and at 1:00 p.m. on November 1, 1800, the second president of the United States, John Adams, entered the premises as its first official resident. The labor of Negro Peter and the other enslaved black workers, as well as that of the rest of the multiracial workforce, had completed a home for the nation’s highest and most symbolic political office. It would be the site of growing influence and prestige not just for the president and the first family, but for the nation as a whole. With time, the White House would become one of the most famous buildings in Western civilization, an enduring symbol of U.S. power, and forever linked to the American people’s internal struggles for freedom, equality, and justice.

  After spending his first night there, President John Adams gave a benediction and blessing to his new home, “I pray Heaven bestow the best of the blessings on this house, and on all that shall hereafter inhabit it. May none but honest and wise men ever rule under this roof.”47 He failed to acknowledge the black people whose years of work went into constructing the house.

  CHAPTER 4

  Burning all Illusions: The White House, Black Slavery, and the Rising Wave of Disunion

  Prologue: Paul Jennings’s White House Story

  The first person to write a memoir of a firsthand experience working in the White House was Paul Jennings, a black man. In 1865, more than half a century after being enslaved by the fourth president of the United States, Jennings published a nineteen-page book titled, A Colored Man’s Reminiscences of James Madison.1 Jennings’s text was first published two years earlier in the January 1863 issue of The Historical Magazine, and Notes and Queries Concerning the Antiquities, History, and Biography of America. Short in length but rich in historical details, Jennings provides critical insights into the lives of free and enslaved blacks in President Madison’s White House and beyond.

  Father of the U.S. Constitution and the Bill of Rights, James Madison was a towering intellectual figure in the early life of the nation, but not quite as successful as its president. Jennings makes no direct statement on Madison’s politics, but he does share personal encounters that are valuable, unique, and notable. Jennings writes about race, war, charity, slavery, and even his presence with the Madisons as the former president took his last breaths. Perhaps no other person was as close to Madison for as long as Paul Jennings, not even Madison’s wife, Dolley.

  What Jennings does not do in his memoir is discuss his role in one of the most ambitious and daring efforts to free enslaved blacks conceived during the antebellum period. His central involvement in what can perhaps be called the “Great Escape from Washington, D.C.” came to light after his death in 1874, and showed a radical side of Jennings that James and Dolley Madison had certainly not witnessed or even suspected, and fortunately for Jennings, not many others had either. His involvement in the clandestine plan to liberate dozens of black people from their white enslavers demonstrates the complicated and multitextured lives that African Americans led during the era of legal slavery. The impact of the great escape and its aftermath had wide-ranging consequences that affected the racial politics of the White House for both Presidents James Polk and Millard Fillmore.

  Jennings’s articulate and astute observations, as well as his later activism, confute notions that enslaved people were passive and disengaged as they carried out their duties and labor. In fact, as his life experiences show, despite the racist brutality of enslavement and white people’s attempt to keep blacks uneducated and illiterate, critical minds were at work. Jennings’s story also intimates that the sort of role reversal in which a former slave takes charge and the once high and mighty fall may have occurred to a much greater degree than the records acknowledge. Ironically, it would be Jennings who would take care of his former enslaver—the widow of a U.S. president no less—lending her small amounts of hard-earned money when she had hit bottom and few others were there for her.

  Jennings was born into slavery on Madison’s farm in 1799. His mother was of mixed African and American Indian heritage and enslaved to the Madisons; his father was a white man, an English trader.2 By the age of ten, Jennings was working at the White House as a footman, and when he was older as a valet. He described Washington, D.C. as “a dreary place” that “was always in an awful condition from either mud or dust,” and he noted that the East Room of the White House was still unfinished.3 Jennings continued to serve Madison until the very end. “I was present when he died,” Jennings wrote in his memoir.4

  The famous painting of George Washington that was saved from the White House just before the British army sacked and burned it in 1814.

  Madison’s presidency is most noted for the launching of an ill-advised war against England. During the course of that war the British invaded and captured Washington, D.C., then sacked and burned the White House on August 24, 1814. The first family and the entire White House staff fled just before the British arrived, and the Madisons, along with Jennings, never returned to the White House during the three years that remained in Madison’s presidency. Jennings was not just an eyewitness to history but a participant in events as they unfolded. It is because of Jennings that important details about the British invasion of 1814 are known and that one of the great stories of White House history was exposed as a myth.

  The first lady and several servants and slaves were at the White House on the day charging British forces captured a mostly abandoned Washington, D.C. As the frightful word came that the British were near, Dolley ordered an evacuation and told the staff to gather whatever could be preserved. The panicking servants and staff grabbed what they could and fled. She later claimed that she personally saved the famous Gilbert Stuart painting of George Washington, which some consider to be among the White House’s most valuable historical objects.5 While he does not directly accuse Dolley of making the claim, in response to media reports along those lines, Jennings states, “This is totally false. She had no time for doing it.”6 He writes that the French doorkeeper John Susé and the president’s gardener Magraw were the ones who “took it down and sent it off on a wagon, with some large silver urns and such other valuables as could be hastily got hold of.”7 This debunking of the Dolley Madison myth happened decades after her death in 1849.

  Jennings’s version also spotlights his complicated and highly personal relationship with the president’s widow. On the one hand, he praises her and holds her in high regard. “She was a remarkably fine woman,” he asserts. “She was beloved by everybody in Washington, white and colored.”8 But clearly, not everyone loved her. He also reports a story told him by Sukey, a slave woman traveling with the first family. At one point, Dolley Madison and several servants were separated from James Madison after the White House evacuation. Dolley spent one night at the home of a Mrs. Love, who lived two or three miles across the river from Washington, D.C., and the next night attempted to stay at the home of a different woman. When the woman found out her lodger’s identity, she becam
e very upset and screamed at Dolley, “Miss Madison! If that’s you, come down and go out! Your husband has got mine out fighting, and d— you, you shan’t stay in my house; so get out!”9 At that time, accentuated by the British occupation of Washington, D.C., disaffection with the war and the Madisons ran high.

  Beyond his dubious praises, Jennings had some legitimate grievances toward Dolley Madison. His work for James and then for Dolley kept him away from his beloved wife, Fanny. He had married her in 1822 while she was enslaved, and she was forced to stay on another Virginia plantation. They were unable to live together and were reduced to seeing each other only on late Saturday nights or Sundays.10 Like married but physically separated slave couples everywhere, Paul engaged in “nightwalking,” i.e., the all-night walks that husbands undertook to be with their spouse. It was reported, moreover, that the Madisons treated him less than honorably. Dolley was accused of hiring out Jennings and keeping “the last red cent” of his earnings, leaving him “to get his clothes by presents, night work, or as he might.”11

  His biggest gripe, however, had to be the way she treated his aspiration to be free. In 1841, Dolley had written in her will, “I give to my mulatto man Paul his freedom,” leaving the impression that she would not sell Jennings as long as she was alive.12 Reneging on her self-imposed promise, Dolley sold Jennings for two hundred dollars to Pollard Webb, a Washington, D.C. insurance agent. The low price for someone as valuable and experienced as Jennings may have been because Jennings was in the process of buying his freedom, and the sum may have amounted to the balance that Jennings owed to Madison. In his memoir, Jennings states that he “had years before bought my freedom of her,” referring to Dolley.13 Some have interpreted that statement to mean that he lied to protect her reputation, but more likely it was a poorly worded way of saying that he had—at least in part—been paying her for his freedom at the time he was sold. It would also explain the even lower sum that U.S. Senator Daniel Webster paid to buy Jennings ten months later—$120. Jennings was freed by Webster after negotiating a contract with him on March 19, 1847, with the stipulation that Jennings “agrees to work out the sum [of his purchase price] at $8.00 a month.”14 Jennings paid off Webster as promised and closed out his debt.

  In the years following her husband’s death, Dolley Madison fell into deep poverty and was largely abandoned by both family members and friends. Few offered help in her state of destitution and despair, but one who did was the enslaved man she had once exploited, Jennings. Clearly in a forgiving mode, he reports that he “occasionally gave her small sums from my own pocket.”15

  For the most part, Jennings never publicly discussed the politics of the White House or what life was like for the blacks whom the president enslaved there. However, Jennings does note that after the British burned the White House and the Capitol, there were efforts to try to move the seat of government to a new location in the North, “but the southern members kept it” in Washington, D.C.16

  In his memoirs, Jennings avoids any mention of his role in the massive attempted slave escape that shook Washington, D.C. to its slaveholding core in 1848, just one year after he gained his freedom. There were at least three known free black men who were involved in organizing the escape: Paul Jennings, Daniel Bell, and Samuel Edmonson. Bell worked as a butler for a powerful attorney in the city, Joseph Bradley. Edmondson was employed as a blacksmith at the Navy Yard.17 Jennings’s involvement, rooted no doubt in his desire to liberate as many of his people as possible, may have also been driven by the fact that one of the women involved was Mary Ellen Stewart, a black woman who had escaped from Dolley Madison and who was well known to Jennings.18 In any case, Jennings has been called a “leading plotter” in the escape19 and is said to have “formed a pack with Daniel Drayton” to execute the plan after meeting him by chance in Baltimore in March 1848.20

  The week of April 10 was projected to be a busy one in Washington, D.C. Some city leaders were planning to celebrate the recent fall of the French monarchy. In February 1848, King Louis-Philippe abdicated the throne, and shortly thereafter, on February 26, the Second Republic was declared. The era of the French monarchy was over. In the year that Karl Marx published the Communist Manifesto, revolutions were occurring in other parts of Europe as well, including Germany, Italy, Hungary, Switzerland, and elsewhere.21 Many in the United States felt that these revolutions of rural and urban working classes echoed the sentiments and spirit of the American Revolution. Yet most failed to identify their cheering efforts to overthrow oppressive regimes in Europe as being in contradiction with their support of the ever-expanding role of slavery and the racial subjugation of blacks and Native Americans in the United States. In one outrageous statement of obtuseness, slaveholder and slavery-defending Mississippi Senator Henry Foote, speaking about the abdication of the French king, proclaimed that “the age of tyrants and slavery was rapidly drawing to a close.”22

  While the escape was planned long before the dates of the celebration events could have been known (and its exact timing determined by such factors as availability of a large boat), it is notable that Washington, D.C.’s black community—enslaved and free—was simultaneously organizing its own rebellion, highlighting the schism between an avowed love for liberty and its true actualization. In one broad step, black Washington, D.C. was about to secede from white Washington, D.C.

  On Saturday evening, April 15, 1848, as parties celebrating French liberty were being held across the city, more than seventy blacks, most enslaved, made their way to the docks where a ship named the Pearl was waiting for them. In small groups, they slowly boarded the ship where Captains Edward Sayres and Daniel Drayton were ready and waiting. Sayres was the Pearl’s regular captain, and Drayton had been contracted by Washington’s Underground Railroad cell to pilot the ship. The plan was to leave late Saturday night, because Sunday was usually a no-work day for most slaves and their white owners would not miss them until it was far too late. Between seventy-four and seventy-seven blacks were on board, representing the “property” of forty-one white enslavers from the area. When the local whites discovered what was happening, it set off what researcher John Paynter called something “approaching a panic.”23

  Unfortunately, two unexpected events foiled the well-planned plot. First, after traveling about 150 miles, the boat ran into bad weather and was forced to anchor at Cornfield Harbor, near the end of the Maryland peninsula. The delay was not welcome, but even under those circumstances the Pearl had enough of a head start that a few hours should not have made a difference as long as the escape and its intended route remained undiscovered.

  However, a betrayal of epic proportion was unfolding back in Washington, D.C. Although contemporary reports mentioned other individuals, most later researchers believe that Judson Diggs, a free black man, had informed a posse of slave hunters that the escaping blacks were on the Pearl and which direction it had headed. Whether it was out of fear, revenge against a woman on the boat who reportedly had jilted him, or for money—all three reasons have been suggested—Diggs’s treachery goes down in history as one of the worst betrayals of the antebellum period. Apparently, slave catchers initially believed that the slaves had escaped on foot and were weaving their way north, the exact opposite direction from the way the Pearl was headed.24 Soon, on the information provided by Diggs, a steamboat set out and seized the Pearl and everyone on board while the boat was still anchored.

  Drayton and Sayres were charged with larceny, and all the black individuals— enslaved and free—were unceremoniously marched through the streets of Washington, D.C. and caged in slave pens. White enslavers and their supporters in Washington, D.C. were so incensed that they wanted to drag Drayton and Sayres out of jail and have them executed on the spot. While a few of the captured blacks were taken back by their owners, most were being prepared to be sold South, where the merciless cruelty of slave breakers was renowned. Of the eleven members of the Bell family that were captured on the Pearl, only Mary and her youngest son
Thomas were purchased by family and supporters, the rest were bought and re-enslaved. The six Edmonson siblings were all bought, re-enslaved and sent South. “Among those sold were people who were legally free,” writes G. Franklin Edwards and Michael R. Winston in an article featured on the White House Historical Society website.25

  The White House of President James Polk found itself in a tight spot, endeavoring to prevent riots from breaking out as news of the white-assisted escape spread. More out of political self-interest—it would have been disastrous to have a race riot in Washington, D.C. during an election year—but his sympathies lying with his fellow white enslavers, President Polk sought to quell the mobs in the streets and reluctantly sent troops to protect the offices of the abolitionist National Era newspaper from being destroyed.26 One ironic link is that in 1845, just two years before Paul Jennings achieved his liberation, Dolley Madison had earned herself a little money by renting him out to President Polk.27

  Though the escape of 1848 was foiled, it generated a nationwide movement to defend and free both the blacks and the whites who were involved. Abolitionist forces mobilized education campaigns, raised funds, and put together a legal team that included Horace Mann, then a congressman who had succeeded relentless antislavery advocate John Quincy Adams and later became famous as an educator. They also received support from Uncle Tom’s Cabin author Harriet Beecher Stowe, and a massive effort was mounted to obtain a pardon from President Millard Fillmore.

  For their part in the attempted escape, Drayton and Sayres spent four years in prison before receiving an executive pardon from President Fillmore on August 11, 1852. Larceny charges had been dropped, but Sayres was convicted on the charge of illegal importation of slaves.28 Drayton remained defiant, however, even after being released from prison. Criticizing the racism of slavery, he wrote: