Simple: they are money donated to an institution for use for a specific purpose. If someone donates a million dollars and restricts that for the use of the music program, then the income generated by that money can only be spent on the music program. If the music program doesn’t need the money, it doesn’t matter; the money can only be used on the music program. If the math department is on the verge of collapse due to a lack of funds, it doesn’t matter; the money can only be used for the music program.
Most colleges and universities have enough sources of support that some restricted money is acceptable. On occasion, though, it becomes a problem. The large number of restricted funds at Sweet Briar limited the flexibility that the administration had to change the college.
Restrictions can be removed with the approval of the donor or heirs, with probate court approval, or if they become illegal (for example, a gift to build a dorm for white students). It’s not impossible, but it can take a while to build a case and have the changes approved. It costs money, too. As a result, non-profit institutions discourage restricted gifts or try to add time limits to the restrictions. (If the institution can’t meet legal restrictions, it may have to turn the money over to another charity that can. The money is not going back to the donors, though.)
Donors, on the other hand, like restrictions. Sometimes, it’s because they like calling the shots; that may be how the donor made the money in the first place. Sometimes, the donor has a genuine and sincere interest in meeting an identified need, even if that need eventually becomes obsolete. And often, the donor wants an element of governance attached to the funds to prevent the institution’s administration from wasting the money.
There’s no right answer here, but expect the issue to surface more as more campuses face financial issues.