I have a pretty heavy building sim, with a 9/11 style dust layer forming as it collapses. Without holdouts the render acts as expected for a heavy dust sim @ 1 hour per frame. After adding my packed alembic destruction layers as forced matte, however, the render time increases to up to 6 hours per frame depending on the frame. I must be doing something very wrong, but google hasn't been nice to me on this topic . I cannot post the file because it is work, so I am hoping someone can point me in the right direction.

Brahman35
After adding my packed alembic destruction layers as forced matte, however, the render time increases to up to 6 hours per frame depending on the frame

it almost sounds like the holdout geo (or dust?) has a dependency on a live sim, do you have sim in the same file?
if so try a clean file with just your smoke cache loaded and your packed alembic holdout to see if you have the same issue