Drag and Drop seems to be enable on all records and not just the selected one
I enabled File Drag and Drop on our account, and created 5 record types it can be used on. It appears that it is allowing it on any record types and just not the 5. I am not sure what is causing that issue.
And just as I post the question I found the answer. I had a global folder, which enables all records, along with sub folders for each record type. I removed the global folder and now it is just on the needed record types.