The DAT file thing.
And do I understand correctly that was also the reason why you have to copy a perfectly valid WAV file to XX folder and name it UnicodeTrackxxx.etc? (because if there is no 8.3 filename other process potentially can't find it?)
It seems a long way around, but basically that is what it comes down to, yes. This is not all though, and part of the problem is displaying it correctly. So, it is unlike you said in the beginning that this is about playing the music only. It is also nice to see what you're playing.
Remember, the trick is that from an 8.3 name the original can be obtained, and while (code page) conversions don't work, this works.
If so, have you looked at symbolic links in Vista?
Yes, I have. But I can't recall anymore why they couldn't do the job. Maybe something with rights, maybe something I could solve today because of more knowledge ... but back then, it was useless.
Alternatively, perhaps an even better way would be to use inter-process communication between GUI & Engine process?
In theory yes, but I think you didn't get that there is no GUI at all. There's just nothing. Also under the hood ... nothing.
You mentioned that tracking *.dat files in memory would take too much resources - it seems they are always around 350 bytes per file which even for 10,000 tracks would result in only 3.5MB RAM which, if I look at task manager, is, at this moment in time, less then 1% of RAM XX uses. As those apparently need to be updated if user changes volume it would seem to be much faster doing it in memory than updating 10,000 files on disk.
I don't recall I said something like that, although I will have said something that makes you think this. Anyway :
Same problem as before : there is nothing in memory to change, because ... well, there is nothing in memory. I'm afraid you try to understand something which isn't even there. Besides that, it is irrelevant. Also the context of 10,000 files is not "justified", as talked about before.
but the principles should still apply in general case and would make XX more responsive and far less I/O hungry which, if I understand correctly is always an enemy of SQ right?
Ehhm, wrong ?

I think if you want to talk in terms of "I/O hungry" it is time to tell, well, don't talk in suggestive terms while you actually don't know. Strange though, because you were able to explain something about 350 bytes. So you do know, which leaves "suggestive";
But ok, here are your numbers :
A coverart picture is some 100KB. Better throw those out, because already one of them is 286 times larger than those 350 bytes.
Next is the track to play, which 1:1 goes along with that 350 bytes. I advice you to make a 64 mp3 of everything, because such a (5 minute) 50MB track really is 142857 times larger than those 350 bytes. Forget about hires files at all.
Now, shall we stop this subject so I can continue with useful things ? (btw, go ahead if you like; I really don't mind. What I do mind is justifying about things which really don't make sense).
One of them might be examining those symbolic links again. Note though that this won't help me a bit for those who have FLACed everything. It also doesn't help when you denote "Copy to XX-Drive" which for sure you will do once you have an SS disk.
Peter