Philibert and Margaret continued their tour of the duchy, and returned to Bourg in April 1503, when they took up their residence at the castle of Pont d'Ain, where the happiest years of Margaret's short married life were passed.
From this favourite castle of the Dukes of Savoy on the river Ain, there is a splendid view of the undulating country, distant hills and forests, which in the days of Philibert were well stocked with game. It would be hard to find a more beautiful spot, and it is no wonder that Margaret loved it and spent most of her time there.
When Philibert succeeded to the dukedom after his father's death, his first act had been to give an appanage to his natural brother René. He bestowed upon him the county of Villars, the castle of Apremont, and the Seignory of Gourdans. This brother, who was known as the Bastard of Savoy, was of an ambitious and grasping nature. Knowing that Philibert hated business and preferred spending his time in hunting and warlike sports, René worked on his indolence until he practically had the management of the duchy in his own hands. He persuaded Philibert to grant him an act of legitimacy and also to give him the title of Lieutenant-General of the States of Savoy. When Louis XII. wished to pass through the duchy to reach Milan he communicated with René. The French monarch made him many promises, which were mentioned in the treaty concluded at Château-Renard with the Cardinal d'Amboise. Duke Philibert, in virtue of this treaty, allowed the passage of the French troops, received Louis XII. at Turin, displayed an extraordinary magnificence, and even accompanied the king to Milan with two hundred men-at-arms. In return for his civility Louis granted him an annual pension of 20,000 golden crowns from the revenues of this duchy.
René's influence over his half-brother was put to a hard test when Margaret became Philibert's wife. The young couple truly loved each other, but the princess could not brook this divided authority. She did all in her power to get rid of René, whom she heartily disliked. The struggle was keen but decisive. Margaret made use of her father's authority, who as the Duke of Savoy's suzerain nullified the deed of René's legitimisation. She also had recourse to religious intervention to accuse him of extortion. At her instigation Friar Malet, the Court preacher, drew a picture of the people's misery and sufferings in a sermon. Addressing Philibert, he exhorted him to 'drive out the thieves who were in his household, who,' he said, 'were leeches sucking the blood of his unhappy subjects.' René was not long in perceiving that his credit at the Court of Savoy was gone. He came to his brother and asked permission to retire to his property. 'I wish,' Philibert answered, 'that you would not only retire from my Court, but also from my State, and that within two days on pain of death.' René took refuge at the Court of France, but even there Margaret's dislike followed him, and all his goods were confiscated after a mock trial.
Philibert had only changed his Prime Minister. After René's departure Margaret took up the reins of government and ruled Savoy and Bresse unhindered. She obtained many privileges from her father, amongst others the temporal jurisdiction over all the bishoprics of Savoy, Piedmont, Bugey, and the provinces of Geneva and Vaud. This concession extended Savoy's right of sovereignty over all lands east of the river Saône, which is still called locally 'the side of the Empire.'
In April 1503 the Archduke Philip paid his sister a visit at Bourg on his return from Spain, where he had been to take possession of the crown of Castile, which through the death of Queen Isabella had descended to his wife Joanna. A grand tournament was held on the Place des Lices in honour of his visit. Philip was then escorted by his sister and her husband to the castle of Pont d'Ain, where fresh festivities were prepared. The nobility of Bresse and Bugey flocked there to welcome the royal guests, and there is even a tradition that the 'Holy Shroud,' usually kept at Turin, and which had long been in the possession of the House of Savoy, was there exposed for the archduke's veneration.
During the next few years the peace of Europe was unbroken, and Philibert was unable to satisfy his warlike inclinations. His exuberant spirits found an outlet in hunting, jousts, and tournaments. He loved splendid armour, gorgeous apparel, and brilliant fêtes. A contemporary chronicler has left an account of the entertainments given by the Court of Savoy in 1504 on the occasion of the marriage of Laurent de Gorrevod (who later became Governor of Bresse and Count of Pont-de-Vaux) with the daughter of Hugues de la Pallu, Count of Varax, Marshal of Savoy. All the nobility of Piedmont and Savoy were assembled at the castle of Carignan on the 18th of February, Shrove Tuesday, where a tournament took place in the presence of Philibert, 'Madam Margaret of Austria, Madame Blanche, Dowager of Savoy, and many other young and beautiful ladies, as much to pass the time as to please the ladies.'
A long and wearisome description of the tournament is given, in which Philibert and his brother Charles carried off several prizes. Such were the duke's favourite pastimes, whether at Turin, Carignan, or at Bourg, where the lists were opened under the castle walls.
Philibert had inherited his passion for hunting from a long line of ancestors who were all devoted to this sport. The castle of Pont d'Ain, standing high on a hill overlooking Bresse and Bugey, with the river Ain flowing at its feet well stocked with fish, and its plains and vast forests abounding with game, was an ideal home for a sportsman like Philibert. Here he and Margaret enjoyed the pleasures of a country life. Accompanied by their nobles and friends the duke and duchess often started at dawn of day on their hunting excursions, returning with the last rays of the evening sun. We are told by Jean le Maire that one day Margaret had an accident which might have proved very serious. When she and her husband were hunting in the fields near the town of Quier in Piedmont, the powerful horse on which she was mounted became quite unmanageable, and kicking and plunging, threw her violently to the ground. She fell under its feet, the iron-shod hoofs trampling on her dress, disarranging her hair, and breaking a thick golden chain which hung from her neck. All those who witnessed the accident were paralysed with terror, believing the duchess could not escape alive, and recalling a similar accident in which her mother, Mary of Burgundy, had lost her life. But Margaret had a miraculous escape, and got up without any harm beyond a severe shaking.
One morning, early in September 1504, Philibert went out hunting, leaving Margaret at Pont d'Ain, and though the weather was extremely hot, followed a wild boar for several hours. All his followers were left behind, and his horses having succumbed to the heat and hard riding, he descended a narrow valley about midday on foot, and at last arrived breathless and bathed in perspiration at Saint Vulbas' fountain. Delighted with the freshness of the spot, he ordered his meal to be served in a shady grove; but before long he was seized with a sudden chill, and pressing his hand to his side in great pain, mounted a horse which was brought to him, and with difficulty rode back to Pont d'Ain, his nobles and huntsmen sadly following. On arriving at the castle the duke threw himself heavily on a bed, and Margaret was immediately summoned. She tried by all means in her power to relieve him, sending in great haste for the doctors. When they came she gave them her precious pearls to grind to powder, and watched them make an elixir with these jewels which she hoped would save the duke's life. She made many vows, and sent offerings to distant shrines, invoking the help of heaven by her prayers. But Philibert was seized with pleurisy; his vigorous constitution resisted the violence of the attack for some days. The physicians bled him, but all their doctoring was in vain, and soon they had to confess that they could do nothing more. 'He himself feeling his end approaching got up, and wished to go and say an eternal farewell to his very dear companion, embracing her closely. After having asked for the last sacraments, and by many acts of faith and devotion shown his love for the holy Christian faith, Duke Philibert expired in Margaret's arms on the 10th of September 1504, at nine o'clock in the morning, in the twenty-fourth year of his age, in the same room in the castle of Pont d'Ain where he had first seen the light.' Margaret's grief was heart-rending: we are told that her sobs and cries echoed through the castle. The whole duchy of Savoy mourned with her for the gallant young prince, so suddenly cut off in the flower of his age.