A lot of perks come with being U.S. president. You’re whisked here and there so that your feet barely touch ground. Heads of state pay their respects. There’s no such thing as a dropped call. You have parties and someone else cleans up.

But there’s one perk that almost everyone else gets that you don’t: a chance to relax. Presidents take vacations at great political peril. There’s never a good time to go, as President Barack Obama learned once again as he headed off with his family for a 10-day break in Martha’s Vineyard, Massachusetts.

How dare he go off and leave behind 9 percent unemployment, a plunging stock market (the Dow Jones index was down more than 400 points the day he left Washington), a global economy on the precipice and living hell in three countries where the U.S. military is mired. Peppered with questions about the president’s trip ― most along the lines of “How could he?” ― White House press secretary Jay Carney dredged up the old saw about how “There is no such thing as a presidential vacation” because the job follows the man.

That happens to be true, of course. In December 2009, Obama was singing Christmas carols with his family in Kaneohe Bay, Hawaii, when he got word of an attempted terrorist attack aboard a plane en route to Detroit. Fa la la la la.

In 2010 Obama tried to inoculate himself against the inevitable complaints by visiting the not-yet-soiled shores of Florida to encourage tourism there after the BP oil spill. He was criticized when he left Florida to vacation in Martha’s Vineyard. If he had stayed in Florida, naturally he would have been criticized for showboating. And if he had stayed home, he would have been criticized for living in a Washington cocoon far removed from average Americans.

Obama’s flurry of visits to heartland states last week, where he shook hands with “real” people as opposed to the wax figures he ostensibly encounters on the East Coast, probably won’t help him. Every photo of him in Martha’s Vineyard wearing dorky jeans and riding a bike or eating ice cream with his daughters will be cited as evidence that he’s not minding the store.

“If you’re the president of the United States, and the nation is in crisis ― and we’re in a jobs crisis right now ― then you shouldn’t be out vacationing,” Republican candidate Mitt Romney said. Romney himself takes “working” vacations, a task made easier by the fact that the multimillionaire Romney owns multiple getaways, one of which is strategically located on a lake in New Hampshire, home to the first 2012 primary.

Like former President Bill Clinton, Obama suffers from a lack of a family compound or second home. We seem to prefer landed gentry, not middle-class hoi polloi, for presidents, and we identify them with their retreats. Franklin Roosevelt had Campobello Island, off the Maine coast, and Warm Springs, Georgia. Dwight Eisenhower had his Gettysburg, Pennsylvania, farm. John Kennedy had Hyannis, Massachusetts, Richard Nixon went to San Clemente, California, and George H.W. Bush was always at home in waspy Kennebunkport, Maine. Jimmy Carter’s retreat, like him, was deemed unpresidential by a press corps that found no romance in a Georgia peanut farm.

Since Nixon, Republicans have taken to affixing “western” before “White House,” underscoring the point that all presidential vacations are working ones. To add gravity to the dusty environs of Crawford, Texas, where he spent his vacations, George W. Bush had a presidential seal plastered on the lectern there.

As a clue to presidential character, Martha’s Vineyard may not tell us much ― at least about this insular president. The Clintons went to Martha’s Vineyard not to get rested, but to get ahead. Theirs wasn’t a working vacation so much as a networking one, punctuated by parties with Jackie O, Beverly Sills, Katharine Graham and James Taylor. Many evenings were devoted to fundraising ― soaking the rich as the rich soaked up the sun. (Even more money can be found among the canapes in New York’s Hamptons, but no president dares go there.)

Obama may feel at home among the Harvard law professors and NPR listeners of Martha’s Vineyard in the same way that Bush was culturally attuned to Crawford and that Nixon was quintessentially Nixonian impressing his wingtips on the soft sand of San Clemente. But Obama is no more readable for being among his fellow liberal strivers. In photographs of the president, you can see a light in his face when he’s horsing around with his daughters and wife; you don’t see it otherwise. He should probably golf more with the likes of Speaker John Boehner, but Obama just doesn’t seem to mix business and pleasure like other pols. That reticence may keep him whole emotionally; it doesn’t help him politically.

According to Mark Knoller, CBS News’ expert numbers keeper, Obama has taken 61 days off in 31 months in office. Ronald Reagan managed to spend 349 days at his Santa Barbara ranch, seemingly spending more time with his horse than with his chief of staff. George W. Bush logged an astonishing 490 days at his ranch in Crawford, where he went to great lengths to be seen working. No matter. The Republican National Committee has built a Web site called ObamaGetAway.com with pictures of Obama eating shave ice and frolicking in the surf.

There is an honest defense against the vacation indictment ― if only we could bear to hear it. The truth is, a president can’t solve most of the problems on his desk even if he never leaves the Oval Office. He might as well take a few days off. 

By Margaret Carlson, Bloomberg

Margaret Carlson is a Bloomberg View columnist. The opinions expressed are her own. ― Ed.