Ol, let's try ...
First off, an album would copy in 7-8 seconds, assumed optimal speed (which would be (e)SATAII). Converting from FLAC does not take all that longer.
Creating 1000s of those dat files ... a few seconds at most.
There are many reasons complete files will need a copy, and some are beyond what you'll expect. One of them will be the Unicode thing (diacritical marks etc.), so if they are in the path or file name, a copy already takes place. Another is one type of header format (IIRC WMP can create that), and the whole file needs to be recreated in order to play well.
Whichever reason there is, if the file is copied, converted or recreated, it will turn into a "UnicodeTrack" file. They too show up in the XX folder, but are hidden.
If it takes tens of minutes to let XXHighEnd finish the preparations concerned, you must assume that at least many files will be copied etc. But you can check it, by looking for those UnicodeTrack files (UnicodeTrack0001.wav, UnicodeTrack0002.wav, etc.).
A next subject might be why to load so many tracks in the Playlist in the first place;
Well, you might say "because I have saved Playlists that contain so many". So yes, a good reason. But now you must be careful with the further approach, which is just subject to the good sound thing, and the preparation and time that takes. This a.o. means that those conversions should not take place during playback, and indeed when those preparations are going on, SQ should be less. This is what the checkbox (Start playback during conversion) is for. You could untick it for better SQ in the beginning.
All implies that one should just not hand a 1000 etc. tracks for playack, also "knowing" that usually these files must be converted, assuming that we all store the files in a compressed format (FLAC). Not necessarily of course, but I must assume it, because you just might (and I myself do to btw). This implies for myself :
I never use saved Playlists. Note that the saved Playlists where there right from the start, and later the Galleries emerged;
A Gallery does just the same, but in a more convenient way, and it resides at a, say, higher level. You can randomize them, pick albums (or tracks, whatever you saved in them) by means of coverart, use them for physically copy (or convert from - to) and much more. Also, they challenge you more to play some kind of selection from it.
The latter is important when you use the saved Playlists anyway. So, they are there, and they are there to be used of course, but, they tend to be large(r), and there you have the perceived problem. However, in the Playlist area you can select too. This is just a matter of windows ctrl-click etc. Thus, when you have this 1000 (or more) tracks in there, just don't play them all. Select the first 100 or whatever you think is suitable for the session and press Play. Or a second batch of 100, or the last etc.
Now you will see where the Gallery is more convenient. I mean, you can select tracks in the Playlist Area, but you can't see what you're actually doing. In a Gallery you can, because you will be selecting by means of coverart. And note that if 1000 individual tracks from different albums are in a Gallery, they all carry the original coverart.
Now, I have no clue why XX needs all those small pesky dat files but I assume you have a good reason for it so np: I _could_ live with that provided it's done only once but it seems that even if I 'save' playlist and then exit XX it will delete all dat files and the re-create them next time I play that same list?
So yes, those dat files are recreated always. Even when you higher the volume by means of Alt-u during Unattended Playback, they are. But trust me, this is not what takes the time, and it will go unnoticed, even with a 1000 of them. It will be the UnicodeTrack files that bother you. However ...
*Those* files are not recreated at all. This is a rather complex mechanism, and although you might see their names change at changing volume or press Next (Alt-n), they are not recreated as long as you go down the list. So for example, press Prev, and all indeed has to be recreated.
The dat files contain the information for XXEngine3 to work with. If you change the volume or whatever it is, it is in there, and XXEngine3 takes that information for playback. So again the example : change the volume, and the volume data in the dat files will be changed for all the remaining dat files to play.
All that data indeed could be in one file, but then the problem (of resources needed) would be bigger. For each track to play the right spot has to be found in those files, and instead of picking the right file by ID, the file with tens of thousand of records has to be scanned. And no, this is not a sequential process because of the things you might do and change during playback (I don't think it is necessary to explain that, but trust me on it).
The most important is and remains : maintaining those dat files does not take the time. Unless something is wrong in your system of course, and I think it can. If you look at the latest posts around JohanZ you'll see it can.
Wouldn't it also be better if dat files would respect XX setting for 'Data folder' instead of being created in same folder where XX exe is?
Yes, but merely for "IT" reasons. And for those who try to follow what's happening of course, and made the hidden files visible.
In the beginning I did not anticipate on this (and the Data folder wasn't there yet), and right now many things have to be changed in the code to let it work. The decision to have those files just in the middle of the install was explicit, but wrong (as how I look at it now).
And wouldn't it be better if dat files would not have to be created every time but rather saved together with playlist?
As said, they contain real time data (like volume), so no. But again, they shouldn't bother.
I realize this only applies to WAV files as other types need converting but since XX only wants WAV in the end anyway, then at least I am OK with creating a separate folder where all music has already been converted from FLAC etc to WAV - it sure is extra work and it would be nice if XX would respect that and start playing with minimum of fuss
I understand what you mean, but if this requires your own control, this is not what you want. In the very beginning I had to work like this myself, and you'll get crazy of it.
Thinking of the "Copy to XX-Drive" feature, indeed there is a possibility to keep those files like a kind of cache. However, besides it will take time to create it and I don't want to put the priorities to that right now, it occurs to me this just isn't necessary. However, this takes the experience mostly related to the beginning of this post. So it can be done, it will be beneficial to some, but it won't be in soon anyway.
Well, I hope this gives you some insight. Don't hesitate to ask or suggest more.
Peter