I let my mother's affair children go into foster care instead of raising them after their parents died

My mother cheated on my father throughout his 10 month battle with cancer. He found out a week before he died and kicked her out. I was a teenager at the time and witnessed everything. I found out a few weeks later that she was 5 months pregnant with her affair partner's child and they had married. We didn't speak after my dad died because his family and I told her that her affair partner would be removed if she took him to the funeral so she didn't come at all. They had three children together and were still married when they died in a car crash earlier this year. My mother's only other living family member was her mom and she's in a nursing home, so I was contacted and asked if I would take in her three children. They were hoping for a family placement for them and I said no. I was even aware that I was the only option left because they had already asked the kids' paternal relatives and all had said no for one reason or another. I was given the option to meet and have a relationship with those children as well but I told them I wasn't interested and wanted no further contact. The social worker tried like hell to convince me otherwise but she left it alone when I hung up and ignored further attempts to make contact.