I'm quite sure you cannot do this. The VA will need to have a record to manage the context of the conversation...the interaction record is supporting the feature in that capacity. If you were to change this, I would think that you would need to build custom infrastructure to take it's place...which would affect every other feature of VA & Interaction Management.
Rather, I would recommend updating the filters/links/lists for where you don't want to see interactions that are being worked by VA.
