With the USB Device plugged in browse to the downloaded device metadata folder:
C:UsersYOUR_USER_NAMEAppDataLocalMicrosoftDevice Metadatadmrccacheen-US
Browse the folders in there until you find the little rainbow hat guy icon. Delete the entire folder that you found it in (my folder name was "d42472c3-a173-455f-8fc5-640e344597b1"). Then I just ejected the device and plugged it in again and the problem was solved. I actually only had one folder in the en-US folder and it was the one I needed to delete.