The Earldom was once held by the Barons of Silverlode (and indeed still own the eponymous sword), but about 400 years ago, King Vincent IV of Vistria stripped the Elstinas of the Earldom and instead gave it to the Baron of Step.
The problem began when Earl Marljotur d'Elstina created a taskforce to tackle the growing threat of organised crime under the leadership of Sir Smith DuLac, his loyal troubleshooter. However, after only a few weeks Sir Smith was killed and his second-in-command, Ander Brolen, was appointed in bis place. Within a year, Brolen managed almost to completely eliminate organised crime from Sword, and for this he was knighted.
However, in truth, Sir Ander had not defeated the gangsters so much as recruited them, bringing them into his own organisation, which simply preyed on other Earldoms outside of his purview. He also managed to take over security for the caravans taking Sword's tax payments to the Duke of Newland, using them to launder counterfeit coinage by swapping them in for legitimate coins. Initially these counterfeit coins were still 90% silver, but in time Ander managed to get an enchanter into the delivery team and his ability to direct inspectors to a small sample of legitimate coins meant that the rest could be swapped out for counterfeits of only 60% or even 50% purity.
Eventually, this came to the attention of Sir Jessica Proulx of Aundar in the course of her own investigation into local highwaymen. Upon trailing some suspects back to their hideout, she discovered the Troll Fursch. She barely escaped with her life, but with the aid of her party she soon discovered that Fursch and Sir Ander were partners, with Fursch assassinating those members of Sir Smith's original taskforce who would not sign on with Brolen's brigands.
However, Marljotur refused to accept this and not only did he not move against Brolen, he forced Sir Jessica to take him with her to battle Fursch. Sir Ander turned on Fursch to cover up his own crimes, leading the investigators to their hideout under the pretense of finding leads and even joining in battle against his former partner, although he paid dearly for this betrayal, being shredded alive by his erstwhile accomplice. Sir Jessica and her team slew Fursch, burned his body to ash and buried his jaws, theoretically preventing him from returning.
With the late Sir Ander no longer able to talk his way out of suspicion, his crimes came to light and Vincent IV and the Duke of Newland agreed that d'Elstina, while not complicit in these crimes, had shown such gross incompetence that his position was untenable. The Earldom of Sword was taken from him and instead given to the Baron of Step, while the Barony of Silverlode was given to Marljotur's daughter, Maria d'Elstina. The Duke decreed that they could retain the Earldom's eponymous sword, since it had been a personal gift from King Vistria and it remains the centrepiece of Baron Silverlode's dining table to this day.