James and Dolley Madison

by Bruce Chadwick

  The revived tobacco business did not prosper, though, since in 1837 America entered one of her worst recessions, an economic downturn that affected the country for years. The Panic of 1837 was caused by land speculators in the west who borrowed enormous sums of money and could not pay it back, the instability of American banks, and the general failure of the wheat crop throughout the country. Grain prices were so high that mobs in New York attacked grain houses and looted them. A collapse in British banking that same year caused large banks in London to call in foreign loans to America and other countries to reestablish their credibility. And a recession in Europe caused a dramatic drop in the prices of American goods shipped there.

  The panic hurt Montpelier, and each year Dolley had the additional costs of the clothing and the feeding of the seventy-one new slaves she had purchased before the economy wobbled. Her debts mounted. Her son's wild entrepreneurial dreams did not help, either. By 1840, Payne, who had failed at everything he had tried, opened up a marble quarry on his farm. The land contained some marble beneath it, and he planned to mine it and produce marble in the shape of steps and other configurations. Geology was one of Payne's hobbies, and he had read several dozen books on marble, making him, he told his mother, an expert. People purchased marble from Europe, so why not sell the marble cheaper at Montpelier? He needed help, though, and so his mother, yet again certain that her son had finally found a successful business, gave him a dozen of her slaves, and then added more to her farms.

  Earlier, she had given him slaves to work and live at his farm, Toddsberth, and help him increase the size of his ramshackle home there. With this new labor, all paid on Dolley's books, Toddsberth grew. There was a main building and several cottages that surrounded it. Payne's outlandish architecture made Toddsberth the talk of Orange County. In addition to some small cottages, it had a large tower-shaped main house with a dining room and spacious ballroom for dancing and parties. He wanted his mother to move there into one of the cottages. He did not want her to have to walk around the building on her way from Montpelier to get into the cottage, though, so he made one of the first-floor windows a large window/doorway for her to enter through. It was the joke of the county. Dolley never did move there. She did give her son all the furniture he desired from the mansion, and he looted the rest of the home. He took two large feather beds, chairs, mirrors, half a dozen chairs, a dining-room table, a dozen paintings, dishes, glasses, shelves, and decanters. He planned to make Toddsberth larger and larger as his marble business boomed, but, like all of his other ventures, it eventually died.

  No one liked Toddsberth. Mary Cutts told a friend that it was the residence of an eccentric and that she and members of her family thought the entire complex looked ridiculous. She frowned at Payne's contention that his mother would one day live there with him.

  In the Civil War twenty years later, a soldier, H. H. Chamberlayne, who stumbled upon Toddsberth, wrote that “the marks of the poor spendthrift [Payne] are still to be seen, walks that he began, never to finish, an attempted ice house turned into a stately pleasure dome, like Kubla Khan's, quarries opened for marble which was not there.”18

  A few years later, Payne stunned his mother with an odd request. The federal government's consul to Liverpool, England, had resigned, and Payne wanted the job. He knew that his mother was a close friend of President John Tyler and asked her to get him the post. It was a well-paying job and he told Dolley he would give her $5,000 a year out of his salary, which would stabilize her always-unsteady financial situation. She went to Tyler, who said he would take it under advisement. He asked for a background check on Payne and then called Dolley back to the White House. Diplomatically, and very gently, he said that “Mr. Todd is not fitted for the office.”19

  He was, in fact, unsuited for any job, something that friends told Dolley throughout her retirement, in a soft way, but she never paid attention to them. When the president died, Payne returned to Montpelier and tried to help his mother sort out the terms of the will. He butted in, overriding lawyers, and wrote dozens of legal-like letters himself to various relatives to whom Madison had bequeathed something. It did not help his mother at all.20

  Everyone in Dolley's family, neighbors, friends, and even strangers, knew how reckless and unfeeling Payne Todd was and denounced him in scathing terms when talking to each other about the Madisons’ son. James Paulding, who knew Payne well, wrote Jared Sparks in 1836 that Dolley was going to let Payne sell the Madison papers to publishers. “If you are acquainted with him, you need not be told that he is the last man in the world to compass such a business,” he wrote, and he offered to help Payne to prevent a debacle. What he did not know, though, was that Dolley had asked friend George Tucker to negotiate with publishers for her. Tucker met with executives at Harper Brothers and reached an agreement. A few days later, though, Tucker met Payne Todd on a New York street. Payne heard what he was in town to do and abruptly informed him that his mother had just sent him to New York to do the same thing. He told Tucker to cease talking to anyone and go home. “I thought that any further efforts on my part were unnecessary and my continued inquiries were consequently suspended,” Tucker wrote Dolley. Friend William Rives always referred to Payne as “the prodigal son.”21

  Dolley's prime job after her husband's death was to sell his papers to a New York publisher. Unfortunately, she entrusted the task to her son, Payne, who told her he knew people in New York, whom he had visited often, and could get a good price for them. She could not have picked a worse agent. Payne did little, was a poor representative, and, after a few weeks of talks with publishers such as Harper Brothers, was back home without any money.

  Dolley believed Payne's story about the lack of interest in her husband's papers, despite Tucker's success, and then decided to stop seeking a publisher and to sell them instead to the federal government, completely ignoring further attempts at New York publishers just because Payne said so. She wrote President Jackson and asked him to recommend that Congress buy the papers.

  Congress was surprised at Dolley's insistence that they pay her $100,000 for the papers; the government had paid George Washington only $25,000 for his papers. Representatives told her brother that the fee was way too high; Dolley backed off and said she would accept $50,000, but that was rejected, too.22 Congress did buy the papers, but for just $30,000. Dolley, who needed the money, accepted the offer. Congress also extended her free mailing privileges, as her husband had, to help keep her living costs down.23

  It took Congress more than two years to buy the papers, though, and nearly four to publish them after politicians convinced President Martin Van Buren to pocket veto them for a congressional term. Dolley had to hold off her creditors and wait for Congress to act. Later, she decided to sell volume 4 of her husband's papers and this time went herself to see the editors at Harper Brothers, who had turned her down on the first three volumes. They again backed off on a deal because Dolley was unwilling to agree to their terms, which were pretty standard for authors. She never engaged anyone in the publishing industry, who knew what they were doing, to represent her as an agent or even just an adviser. She had no idea what she was doing and always turned to Payne for advice, and Payne had no idea what he was doing, either.

  After she failed with the publishers on volume 4, she asked Payne, who had mishandled the first three volumes, to represent her again. He told her that the problem was not the publishers; it was her. “I am to confer with the Harpers as soon as I can see him about a different in balance in your favor and an advance on books and money. Your writing would not be understood, and might embarrass my obtaining any for you,” he told his mother bluntly.24

  Instead of rebuking him and finding someone else, Dolley did what he told her. This deal fell through, too, though. An author of a book about Harpers said that “the answer may be that Mrs. Madison would not accept their terms—presumably a half profits arrangement.”25

  The former First Lady then decided to sell volume 4 to Congr
ess, too, since Payne had said it was impossible to sell the work to any commercial publisher. Having watched Payne fail miserably dealing with Harper Brothers, Dolley incredibly retained him again to represent her in the deal with Congress, too. She begged him to do it. “If you love me…tell me when you will come to offer the papers to Congress and to do something with the 4th volume…. Oh, my son, I am too unhappy not to have you with me and not to have even your opinions and directions, what do I do myself or what individual to engage and at what time.”26

  Payne did not come to Washington but told his mother to keep him apprised of any talks with Congress through the mail. She soon had a meeting with two congressmen, both friends of hers who would do anything to help her, who told her they could get Congress to buy volume 4 for the $30,000 for which they had purchased the first three volumes. It was still short of Madison's suggested price of $50,000. Dolley did not accept the offer or even name her price when they asked her what it was. She told them they had to deal with her son, Payne, far away at Montpelier. “Now my dear Son, will you say at once what you think best to these particular questions. They seem to dwell on the $30,000 as if that were the proper sum, but I must speak now as they are impatient. Oh, that you, my beloved, were fixed in all things, to cooperate with me, I will not say to act solely for me, because I have become the object of interest, and less would be done without me. I want you to reply in a few days.”27

  The papers of her husband were special not only to her, but, she believed, to the whole country as well. They told the story of the Constitution, his life as secretary of state and two terms as president. They were, Dolly told President Jackson, “his legacy.”28

  And what did her beloved son do in this critical matter? Nothing. Dolley never heard from him and, on her own, agreed to Congress's $30,000. Then Congress adjourned without technically agreeing to the deal. Payne, furious, jumped into Dolley's business again and went to New York with an old note from his mother authorizing him to act on her behalf and tried to revive the deal with Harpers. This time he succeeded in selling the book, but only at a low price. Payne then decided to squeeze Congress to get more money out of them by telling them of his contract with Harpers. Congress did not care, but Harpers did. Their executives felt they had been manipulated by Payne and pulled out of the contract. This left Congress as the only buyer and, knowing all about the Harpers dispute, it made a deal with Dolley for the low, $30,000 amount. Payne had made a mess with not just one publisher, but two, and at the same time. And then, on top of all that, Congress did not act on the purchase for another year, leaving Mrs. Madison desperate to meet her bills. These, of course, included more loans, which were never repaid, to her son. The money Dolley sent to Payne constituted an endless river. In the fall of 1836, her niece said she sent him $100 one day, $100 the next day, and then another $100 two days later. “[She] wishes you to tell her how much more you want that she may endeavor to send it soon,” he niece added.29

  She turned to the rather useless Payne at a time when her debts had mounted considerably as the recession of 1837 droned on. She was borrowing money in Washington to pay her monthly bills and annual debts while waiting for the $30,000 from Congress that was continually delayed. She told her son that “we are without funds and those we owe are impatient” and that she needed to sell volume 4 and earn money so that she could maintain her “respectable standing before the world.”30

  In Washington, she asked Payne to bring all of Madison's papers to her so she could turn them over to Congress in order to be paid the $30,000. He did not. “I cannot understand, of course, without explanation, the refusal to accept & take charge of the original manuscripts. Money will not be paid without these being first delivered,” she wrote him, exasperated.31

  Her costs were high and some of her relatives did not help. Her brother-in-law William Madison, who lived nearby, sued her for $2,000 that he insisted the president owed him. She also found herself in court just because she could not easily reach all the dozens of people to whom her husband had bequeathed money in his will. Dolley had to get a lawyer to file a suit against all of them to get them to make their whereabouts known and then pursue them so she could give them the money. It was tedious, time-consuming, and expensive. She had to pay all the taxes on her business and assets, had little revenue coming in from the plantation due to the small workforce and stagnant farming business, and had practically no income from the financial holdings of the Madisons, which had been relentlessly depleted over the years by payments to cover their reckless son's debts. Following a $2,000 gift to the American Colonization Society, a $1,000 gift to Princeton University, a gift to the University of Virginia, and other gifts to relatives in her husband's will, the Montpelier estate and Madison finances were actually $9,000 in debt.

  All was not over when Dolley finally received her $30,000 from Congress. Her friends in Washington were terrified that Payne Todd, who had her power of attorney, would abscond with the money and leave her penniless. “Her son Todd is here playing the fool in high style as far as I can hear. It will never do to let him get hold of this money, or to have anything to do with it. The man is deranged if he ever had any sense…what orders she has given her son I know not, but if she has given him any to pay her debts with it [money] will certainly be squandered,” John Campbell, the treasurer of the United States, wrote. Another public official, George Featherstonhaugh, was absolutely convinced that Payne Todd would loot Dolley's accounts. “It is, however, too well known that his intimates are the Blacklegs and Gamblers of Washington. Their gigs and flasks stand at his door and he appears to be in their hands. Mr. Todd's habits and facilities of temper will probably make him their victim and this money will in all likelihood be lost to Mrs. Madison,” he said. To prevent that, Congress authorized payment of the $30,000 over several years.32

  All of this stress often made Dolley sick. Her eye problems returned. Payne sent her books written by oculists and doctors applied leeches and blisters to the backs of her ears and even bled her several times. She bathed her eyes with milk and water plus sassafras tea. Dr. Physick, who treated her for her knee thirty years earlier, sent a lotion to be applied to her eyes. Local doctors visited her and gave her herbal medicines. All of this did little good. “My general health has suffered much,” she said in the autumn of 1836.33

  She looked and felt terrible. Her brother said that “the inflammation around the eyes is a little abated, but not so the soreness and rawness, and the discharge is unchanged in consistence, and perhaps increased in quantity, and the white balls are slightly inflamed; one more so than the other. The itching and burning sensations of the lids the same.”34

  Her illnesses often caused her to cancel visits to friends, such as one in the spring of 1838. “I feel too dull and dismal to appear before you this evening,” she wrote her host.35

  Her relatives advised her to sell all of Montpelier and move to Richard Cutts's home, a two-story brick house on Lafayette Square, in Washington, that she and her husband owned and lent to Cutts and his family and which was now occupied by the family of Senator William Preston. The Prestons could move elsewhere, and she would be independent and back in Washington with her lifelong collection of friends. The house was a handsome, two-story brick home with dormer windows on the third floor. It overlooked a square full of trees that was populated by birds. Washingtonians enjoyed walking there. In Washington, she would also be free of the burdens of running and paying for Montpelier.

  Before he died, Dolley and her husband had been following events in Washington, particularly the social whirl, at a distance at Montpelier. Dolley could never completely detach herself from the social life of the capital. “I hope you will soon be at the parties and will give me a detail of what is going forward, amidst the various characters in Washington,” she wrote her niece in 1831. A friend wrote Dolley breathlessly that 2,800 people were invited to a party President Adams gave at the White House for Andrew Jackson. “It was really a very brilliant party & ad
mirably well arranged. The ladies climbed the chairs & benches to see General Jackson and Mrs. Adams with him, which gratified the general curiosity.”36

  Dolley's friend Judith Walker Rives went to a wedding in the capital and joined over one thousand other guests. “Everybody appeared in fine humor and finer dress, but, alas, such crushing of flowers, feathers, silks, satins, crape never was witnessed before. I am sure the rooms after the route was over must have resembled a [mess].” Rives added, a glint in her eye, that “the spirit of dissipation appears now to be reviving.”37

  It was not just the parties that the Madisons longed to hear about from friends but also the city itself, which had grown so quickly during their sixteen years there and continued to expand. Friend Judith Walker Rives wrote in 1829 of “the broad bosom of the Potomac, sparkling in the sun and covered with little vessels with their white sails swelling in the breeze. Beneath our feet as it were lies the whole city, sometimes blazing in the beams of the rising or setting sun and sometimes so completely enveloped in mist that the few stately looking buildings on the opposite heights appear like castles in the air.”38

  Edward Coles, Madison's former secretary, who was carving out a nice life for himself in Illinois, filled his letters with hot rumors he had heard out of Washington, such as one that a bank in London had forgiven James Monroe a large debt because of some secret services Monroe rendered to them to help in business with the French government. He told her of government officials in New York who were fired for taking city money and spending it on themselves, of boarding-school heads forced out of town by rising housing costs. He thrived on news of secret organizations, such as “The Club,” a group of powerful Washingtonians who met at each other's homes every Friday night. The Madisons devoured the news and gossip while in retirement at Montpelier.