XXHighEnd - The Ultra HighEnd Audio Player
September 20, 2019, 01:46:32 pm *
Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
News: August 6, 2017 : Phasure Webshop open ! Go to the Shop
Search current board structure only !!  
   Home   Help Search Login Register  
Pages: 1 [2] 3  All
  Print  
Author Topic: SFS Solved  (Read 18904 times)
0 Members and 0 Guests are viewing this topic.
Josef
Audio Loudspeaker
*
Offline Offline

Posts: 157



View Profile
« Reply #15 on: October 22, 2010, 12:05:18 am »

Quote
You post before that one ... NO ... not familar with that.  So, will investigate it. Sounds good to say the least !

I'm not an expert but looking at this it seems rather simple to implement:
http://www.developer.com/net/article.php/3828586/Using-Memory-Mapped-Files-in-NET-40.htm
Logged
Josef
Audio Loudspeaker
*
Offline Offline

Posts: 157



View Profile
« Reply #16 on: October 22, 2010, 12:35:17 am »

Quote
Josef, your last post ... all correct, but completely solved.

Great!
Although, reading what you wrote today let me reserve some doubt before I see it Happy
Here's why: Looking at current version, I noticed rather strange memory pattern usage and I'm wondering if you got rid of it completely in next version?

For example, if I set SFS to 100MB I (turns out naively) expected to see a 100MB buffer somewhere. Oh yeah, it is there. But there is another 100MB one. Hmmm..., ok.  And then, there is yet another one. And, finally, one more, yet this time 'only' 50MB in size.
So, 100MB SFS setting causes XX to allocate 350MB? Sounds like either a bug or some heavy tricks are going on? Happy


Logged
PeterSt
Administrator
High Grade Audiophile
*****
Offline Offline

Posts: 15507



View Profile Email
« Reply #17 on: October 22, 2010, 12:41:39 am »

Of course I can't tell whether you see "live" memory of dead, but a native track of 16/44.1 :

- has to be read into memory;
- has to expand to whatever it is (like 24/192);
- has to have a "cyclic" counterpart (Gapless, doubles the latter);
- has to be stuffed into the device's buffer (also cyclic, but this is only a fraction of it all).

If you see more, and it's not dead, please let me know !
Peter

Logged

For the Stealth III LPS PC :
W10-14393.0 - Oct 31, 2018 (2.10)
XXHighEnd Mach III Stealth LPS PC -> Xeon Scalable 14/28 core with Hyperthreading On (set to 10/20 cores) @~660MHz, 48GB, Windows 10 Pro 64 bit build 14393.0 from RAM, music on LAN / Engine#4 Adaptive Mode / Q1/-/3/4/5 = 30/-/1/1/1/ Q1Factor = 10 / Dev.Buffer = 4096 / ClockRes = 15ms / Memory = Straight Contiguous / Include Garbage Collect / SFS = 140.19  (max 140.19) / not Invert / Phase Alignment Off / Playerprio = Low / ThreadPrio = Realtime / Scheme = Core 3-5 / Not Switch Processors during Playback = Off/ Playback Drive none (see OS from RAM) / UnAttended (Just Start) / Always Copy to XX Drive (see OS from RAM) / Stop Desktop, Remaining, WASAPI and W10 services / Use Remote Desktop / Keep LAN - Not Persist / WallPaper On / OSD Off (!) / Running Time Off / Minimize OS / XTweaks : Balanced Load = 35 / Nervous Rate = 10 / Cool when Idle = n.a / Provide Stable Power = 0 / Utilize Cores always = 1 / Time Performance Index = Optimal / Time Stability = Stable / *Arc Prediction Filtering (16x)* / Always Clear Proxy before Playback = On -> USB3 from MoBo -> Lush^2*A:B-W-Y-R, B:B-W-R* USB 1m00 -> Phisolator 24/768 Phasure NOS1a/G3 75B (BNC Out) async USB DAC, Driver v1.0.4b (16ms) -> B'ASS Current Amplifier -> *Blaxius^2 A:B-R, B:B-R* Interlink -> Orelo MKII Active Open Baffle Horn Speakers.
Removed Switching Supplies from everywhere (also from the PC).

For a general PC :
W10-10586.0 - May 2016 (2.05+)
*XXHighEnd PC -> I7 3930k with Hyperthreading On (12 cores)* @~500MHz, 16GB, Windows 10 Pro 64 bit build 10586.0 from RAM, music on LAN / Engine#4 Adaptive Mode / Q1/-/3/4/5 = 14/-/1/1/1 / Q1Factor = 1 / Dev.Buffer = 4096 / ClockRes = 1ms / Memory = Straight Contiguous / Include Garbage Collect / SFS = 0.10  (max 60) / not Invert / Phase Alignment Off / Playerprio = Low / ThreadPrio = Realtime / Scheme = Core 3-5 / Not Switch Processors during Playback = Off/ Playback Drive none (see OS from RAM) / UnAttended (Just Start) / Always Copy to XX Drive (see OS from RAM) / All Services Off / Keep LAN - Not Persist / WallPaper On / OSD On / Running Time Off / Minimize OS / XTweaks : Balanced Load = *43* / Nervous Rate = 1 / Cool when Idle = 1 / Provide Stable Power = 1 / Utilize Cores always = 1 / Time Performance Index = *Optimal* / Time Stability = *Stable* / Custom Filter *Low* 705600 / -> USB3 *from MoBo* -> Clairixa USB 15cm -> Intona Isolator -> Clairixa USB 1m80 -> 24/768 Phasure NOS1a 75B (BNC Out) async USB DAC, Driver v1.0.4b (4ms) -> Blaxius BNC interlink *-> B'ASS Current Amplifier /w Level4 -> Blaxius Interlink* -> Orelo MKII Active Open Baffle Horn Speakers.
Removed Switching Supplies from everywhere.

Global Moderator
Josef
Audio Loudspeaker
*
Offline Offline

Posts: 157



View Profile
« Reply #18 on: October 22, 2010, 01:39:58 am »

Not sure what is your definition of 'live' but it sure seems 'live' to me as it is owned by XXEngine3 process and definitely taking up RAM space Happy
Check out below those LargeObjects and total stats.


* Capture.PNG (9.68 KB, 974x65 - viewed 559 times.)

* Capture2.PNG (12.51 KB, 816x115 - viewed 598 times.)
Logged
PeterSt
Administrator
High Grade Audiophile
*****
Offline Offline

Posts: 15507



View Profile Email
« Reply #19 on: October 22, 2010, 09:14:31 am »

Hi Josef, thanks.

But isn't this (see below) exactly what I described ? The only difference is that you upsample 2x (if I interpret it correctly).
Also, be careful when to look, because in your version things can change along the way, even at an "unlucky" trackboundary (may grow again). This has all been eliminated.

Peter


* LargeObjects01.png (1.29 KB, 57x62 - viewed 566 times.)
Logged

For the Stealth III LPS PC :
W10-14393.0 - Oct 31, 2018 (2.10)
XXHighEnd Mach III Stealth LPS PC -> Xeon Scalable 14/28 core with Hyperthreading On (set to 10/20 cores) @~660MHz, 48GB, Windows 10 Pro 64 bit build 14393.0 from RAM, music on LAN / Engine#4 Adaptive Mode / Q1/-/3/4/5 = 30/-/1/1/1/ Q1Factor = 10 / Dev.Buffer = 4096 / ClockRes = 15ms / Memory = Straight Contiguous / Include Garbage Collect / SFS = 140.19  (max 140.19) / not Invert / Phase Alignment Off / Playerprio = Low / ThreadPrio = Realtime / Scheme = Core 3-5 / Not Switch Processors during Playback = Off/ Playback Drive none (see OS from RAM) / UnAttended (Just Start) / Always Copy to XX Drive (see OS from RAM) / Stop Desktop, Remaining, WASAPI and W10 services / Use Remote Desktop / Keep LAN - Not Persist / WallPaper On / OSD Off (!) / Running Time Off / Minimize OS / XTweaks : Balanced Load = 35 / Nervous Rate = 10 / Cool when Idle = n.a / Provide Stable Power = 0 / Utilize Cores always = 1 / Time Performance Index = Optimal / Time Stability = Stable / *Arc Prediction Filtering (16x)* / Always Clear Proxy before Playback = On -> USB3 from MoBo -> Lush^2*A:B-W-Y-R, B:B-W-R* USB 1m00 -> Phisolator 24/768 Phasure NOS1a/G3 75B (BNC Out) async USB DAC, Driver v1.0.4b (16ms) -> B'ASS Current Amplifier -> *Blaxius^2 A:B-R, B:B-R* Interlink -> Orelo MKII Active Open Baffle Horn Speakers.
Removed Switching Supplies from everywhere (also from the PC).

For a general PC :
W10-10586.0 - May 2016 (2.05+)
*XXHighEnd PC -> I7 3930k with Hyperthreading On (12 cores)* @~500MHz, 16GB, Windows 10 Pro 64 bit build 10586.0 from RAM, music on LAN / Engine#4 Adaptive Mode / Q1/-/3/4/5 = 14/-/1/1/1 / Q1Factor = 1 / Dev.Buffer = 4096 / ClockRes = 1ms / Memory = Straight Contiguous / Include Garbage Collect / SFS = 0.10  (max 60) / not Invert / Phase Alignment Off / Playerprio = Low / ThreadPrio = Realtime / Scheme = Core 3-5 / Not Switch Processors during Playback = Off/ Playback Drive none (see OS from RAM) / UnAttended (Just Start) / Always Copy to XX Drive (see OS from RAM) / All Services Off / Keep LAN - Not Persist / WallPaper On / OSD On / Running Time Off / Minimize OS / XTweaks : Balanced Load = *43* / Nervous Rate = 1 / Cool when Idle = 1 / Provide Stable Power = 1 / Utilize Cores always = 1 / Time Performance Index = *Optimal* / Time Stability = *Stable* / Custom Filter *Low* 705600 / -> USB3 *from MoBo* -> Clairixa USB 15cm -> Intona Isolator -> Clairixa USB 1m80 -> 24/768 Phasure NOS1a 75B (BNC Out) async USB DAC, Driver v1.0.4b (4ms) -> Blaxius BNC interlink *-> B'ASS Current Amplifier /w Level4 -> Blaxius Interlink* -> Orelo MKII Active Open Baffle Horn Speakers.
Removed Switching Supplies from everywhere.

Global Moderator
Flecko
Audio Enthusiast
**
Offline Offline

Posts: 474


View Profile
« Reply #20 on: October 22, 2010, 10:13:25 am »

Quote
First thing to notice is loads of punch and better bass. More deep, more roaring. The freshness is of a level unheard
Yeehaa!  yahoo

You mentioned that you use all four cores with the appointment schemes. I just have two unhappy Do you think it could be worth to invest in a Quadcore?
I need more Ram too, so it might be a new machine Happy
Logged

Software: Windows7 Ultimatex64SP1 | XXHighend 9z9b
Hardware: | Gigabyte X79-UD3 | i7-3820 | 16 GB DDR3 | OS on 128 GB Samsung SSD 830  | Music on 2TB WD Caviar Green | Seasonic X-660

XXHE Settings: | Engine 4 | Adaptive | Buffer=1024 | Q12345=[14,0,0,0,0] | xQ1=1 | Q5=3 | Scheme=3 | Mixed Contiguous with SFS=12 | 176.4kHz32bit | ArcPred + Peakextend | Clock=1ms |
Josef
Audio Loudspeaker
*
Offline Offline

Posts: 157



View Profile
« Reply #21 on: October 22, 2010, 12:16:13 pm »

Quote
But isn't this (see below) exactly whay I described ? The only difference is that you upsample 2x (if I interpret it correctly).

Nope: no upsampling, no funny business: just a single 'straight' 16/44 WAV track.

Quote
- has to be read into memory;
- has to expand to whatever it is (like 24/192);
- has to have a "cyclic" counterpart (Gapless, doubles the latter);

So you're saying that's where those 3x100MB buffers come from i.e. 3.5xSFS memory is needed 'by design'?

If so OK, I could see 2xSFS being needed for gapless as in SFS1 holding current track and SFS2 holding next (or, if SFS is small, SFS1 holding current 'piece' and SFS2 holding next 'piece').

But if that's how it works then buffer #1 above just screams to be eliminated as it seems relatively easy to do any conversion as file is being read instead of reading the whole file and then converting the whole thing. Sounds like perfect application for memory-mapped files approach...?
(btw: does 'conversion' mean e.g. x-upsampling/Arc prediction stuff? I'd sure like my WAVs _not_ to be 'converted' Happy as I don't use any of that stuff...)

Come to think of it, do I understand correctly that this 'conversion' is happening _during_ music playback? (as I can see that next track (or 'track piece') is being loaded by 2nd thread during playback of current track ('piece') just a little before it ends)?

Wouldn't it be an idea to instead do this 'conversion' when 'Copy to XX' is ticked in parent process?

Then all file copying/conversions (i.e. really nasty torrent of I/Os & CPU cache-killing memory copies) would be done _before_ music starts playing (assuming 'Start XXEngine during conversion' is not checked).
Wouldn't that mean you'd both need far less memory and have a lot less 'system disturbance' during playback?

Or have I misunderstood completely? Happy
Logged
PeterSt
Administrator
High Grade Audiophile
*****
Offline Offline

Posts: 15507



View Profile Email
« Reply #22 on: October 22, 2010, 02:02:15 pm »

Quote
Or have I misunderstood completely? Happy

A little, yes. Haha. You wouldn't want to wait for that all and you'd need an 8x (etc. etc.) larger RAMDisk. But you may try AI filtering to get the idea.

About the memory sizes ... The tracks are always upgraded to 32 bits if you are using a more than 16 bits DAC. Even if you are not, the memory is reserved like that.

If this ends up in well meant (!) advices how I should organize the program, please let's stop. I mean, I have better things to do than explaining to you (or anyone) why it works like it works.


About the memory mapped thing (I'm fully open to that) ...

I don't see the benefits and only the contrary. But maybe I see it wrongly;
Remember : this is all not about saving memory or anything; if that were the case XX wouldn't be a memory player.
So what happens with the MMF's is that we can treat a file as if it were memory, while in the mean time the portion needed is read into memory. Then what ? then we have the same as we have now; it could have been nice or maybe helpful if I didn't make all this stuff already, but I did - and so now it will be the other way around : I am (we are) in the hands of Bill Gates how this all works and what it implies. No thank you !

If it were so that nothing's read into memory and a disk could be treated like memory, *then* we'd have another case. But it doesn't work like that (I'd like to say it works the other way around, but that is not really the situation because in fact the former is true, but only "logically"; not physically).
You can see how ridiculous it is (ok, through my eyes) at the examples of inter process communication. So, we write something to memory (which we do, but which ends up on the "mapped disk"), and now another can read from the same memory address and communicate. swoonswoonswoonswoonswoon
Just write some ".dat" files (you know), and be normal !

So, in the end we deal with memory but read/write from/to disk (RAMDisk of course), and next we think we'll have a good response ? yea, maybe, if I first load the part of the file concerned into memory (which, I think, can't even be avoided for *any* response). But if I'd behave transparently (for myself) and operate at the byte level, knowing that disk transfers go at the block level, what do you think the result will be ?
The only thing I don't know - hence couldn't find out/proove, is at what level transfers go to the RAMDisk, because remember, what ever cluster size I defined, writing to the RAMDisk didn't matter a single fraction of a second. So, supposed this happens at the byte level anyway, *then* there's a chance for this. But I won't believe this, because e.g. copying to the RAMDisk just takes way longer than copying an internal array (writing to it) which easily goes within 0.1sec for 500MB or so.

No, the real idea would be the opposite of being a memory player to "solve" this. Ehh, solve what ?
Happy
Peter
Logged

For the Stealth III LPS PC :
W10-14393.0 - Oct 31, 2018 (2.10)
XXHighEnd Mach III Stealth LPS PC -> Xeon Scalable 14/28 core with Hyperthreading On (set to 10/20 cores) @~660MHz, 48GB, Windows 10 Pro 64 bit build 14393.0 from RAM, music on LAN / Engine#4 Adaptive Mode / Q1/-/3/4/5 = 30/-/1/1/1/ Q1Factor = 10 / Dev.Buffer = 4096 / ClockRes = 15ms / Memory = Straight Contiguous / Include Garbage Collect / SFS = 140.19  (max 140.19) / not Invert / Phase Alignment Off / Playerprio = Low / ThreadPrio = Realtime / Scheme = Core 3-5 / Not Switch Processors during Playback = Off/ Playback Drive none (see OS from RAM) / UnAttended (Just Start) / Always Copy to XX Drive (see OS from RAM) / Stop Desktop, Remaining, WASAPI and W10 services / Use Remote Desktop / Keep LAN - Not Persist / WallPaper On / OSD Off (!) / Running Time Off / Minimize OS / XTweaks : Balanced Load = 35 / Nervous Rate = 10 / Cool when Idle = n.a / Provide Stable Power = 0 / Utilize Cores always = 1 / Time Performance Index = Optimal / Time Stability = Stable / *Arc Prediction Filtering (16x)* / Always Clear Proxy before Playback = On -> USB3 from MoBo -> Lush^2*A:B-W-Y-R, B:B-W-R* USB 1m00 -> Phisolator 24/768 Phasure NOS1a/G3 75B (BNC Out) async USB DAC, Driver v1.0.4b (16ms) -> B'ASS Current Amplifier -> *Blaxius^2 A:B-R, B:B-R* Interlink -> Orelo MKII Active Open Baffle Horn Speakers.
Removed Switching Supplies from everywhere (also from the PC).

For a general PC :
W10-10586.0 - May 2016 (2.05+)
*XXHighEnd PC -> I7 3930k with Hyperthreading On (12 cores)* @~500MHz, 16GB, Windows 10 Pro 64 bit build 10586.0 from RAM, music on LAN / Engine#4 Adaptive Mode / Q1/-/3/4/5 = 14/-/1/1/1 / Q1Factor = 1 / Dev.Buffer = 4096 / ClockRes = 1ms / Memory = Straight Contiguous / Include Garbage Collect / SFS = 0.10  (max 60) / not Invert / Phase Alignment Off / Playerprio = Low / ThreadPrio = Realtime / Scheme = Core 3-5 / Not Switch Processors during Playback = Off/ Playback Drive none (see OS from RAM) / UnAttended (Just Start) / Always Copy to XX Drive (see OS from RAM) / All Services Off / Keep LAN - Not Persist / WallPaper On / OSD On / Running Time Off / Minimize OS / XTweaks : Balanced Load = *43* / Nervous Rate = 1 / Cool when Idle = 1 / Provide Stable Power = 1 / Utilize Cores always = 1 / Time Performance Index = *Optimal* / Time Stability = *Stable* / Custom Filter *Low* 705600 / -> USB3 *from MoBo* -> Clairixa USB 15cm -> Intona Isolator -> Clairixa USB 1m80 -> 24/768 Phasure NOS1a 75B (BNC Out) async USB DAC, Driver v1.0.4b (4ms) -> Blaxius BNC interlink *-> B'ASS Current Amplifier /w Level4 -> Blaxius Interlink* -> Orelo MKII Active Open Baffle Horn Speakers.
Removed Switching Supplies from everywhere.

Global Moderator
Josef
Audio Loudspeaker
*
Offline Offline

Posts: 157



View Profile
« Reply #23 on: October 22, 2010, 02:31:15 pm »

And, since I'm probably _really_ getting on your nerves by now (hope you can see it's with seriously good intentions although my tone sometimes may not show it Happy )  may I suggest that you do _not_ kill XXEngine process every time playlist is done?

Why? Well, now that you seem to have solved your .NET memory management issues (I'm guessing by sticking to blittable types and pinning down SFS buffers?) and seeing (hearing) tremendous SQ improvements (as you reported) I am sure you will be able to see the value of not having to 'recreate the universe' every time a new playlist is started:
By having XXEngine 'stick' around with initially allocated RAM you will be helping OS minimize memory starvation issues by simple fact of not working against it! (which I'm afraid you are doing at the moment by killing & restarting Engine with every playlist which inevitably increases memory fragmentation....)

 
Logged
PeterSt
Administrator
High Grade Audiophile
*****
Offline Offline

Posts: 15507



View Profile Email
« Reply #24 on: October 22, 2010, 03:42:33 pm »

Hmm ...

This requires a bit of upside down thinking from my side;

Technically this would be easy to make, but it would require something like "as soon as the OS is up" and always before XXHighEnd starts. So, then XXHighEnd will start on top of it, instead of "under" (memory wise). Should be better, especially when you use an SFS near the memory limit. Also -and as I found by now- the few rounds I need to get everything out of the way (GC), are not needed at all when using a low SFS. This (to me) means that when the several rounds are needed, or no possibility exists for freeing memory at that combination of things, or (more realistically) the dead memory generations (0, 1, 2) are just needed to promote. But for sure XXHighEnd.exe will be in some small space somewhere when it starts first, with XXEngine3 on top of it, leaving that small space of XXHE uncontigious at that moment, which the OS won't like.

I'm blahblahing a bit, but I think your suggestion makes sense (or at least has a good chance of getting it better).

Yes, I will try that.
Peter

Logged

For the Stealth III LPS PC :
W10-14393.0 - Oct 31, 2018 (2.10)
XXHighEnd Mach III Stealth LPS PC -> Xeon Scalable 14/28 core with Hyperthreading On (set to 10/20 cores) @~660MHz, 48GB, Windows 10 Pro 64 bit build 14393.0 from RAM, music on LAN / Engine#4 Adaptive Mode / Q1/-/3/4/5 = 30/-/1/1/1/ Q1Factor = 10 / Dev.Buffer = 4096 / ClockRes = 15ms / Memory = Straight Contiguous / Include Garbage Collect / SFS = 140.19  (max 140.19) / not Invert / Phase Alignment Off / Playerprio = Low / ThreadPrio = Realtime / Scheme = Core 3-5 / Not Switch Processors during Playback = Off/ Playback Drive none (see OS from RAM) / UnAttended (Just Start) / Always Copy to XX Drive (see OS from RAM) / Stop Desktop, Remaining, WASAPI and W10 services / Use Remote Desktop / Keep LAN - Not Persist / WallPaper On / OSD Off (!) / Running Time Off / Minimize OS / XTweaks : Balanced Load = 35 / Nervous Rate = 10 / Cool when Idle = n.a / Provide Stable Power = 0 / Utilize Cores always = 1 / Time Performance Index = Optimal / Time Stability = Stable / *Arc Prediction Filtering (16x)* / Always Clear Proxy before Playback = On -> USB3 from MoBo -> Lush^2*A:B-W-Y-R, B:B-W-R* USB 1m00 -> Phisolator 24/768 Phasure NOS1a/G3 75B (BNC Out) async USB DAC, Driver v1.0.4b (16ms) -> B'ASS Current Amplifier -> *Blaxius^2 A:B-R, B:B-R* Interlink -> Orelo MKII Active Open Baffle Horn Speakers.
Removed Switching Supplies from everywhere (also from the PC).

For a general PC :
W10-10586.0 - May 2016 (2.05+)
*XXHighEnd PC -> I7 3930k with Hyperthreading On (12 cores)* @~500MHz, 16GB, Windows 10 Pro 64 bit build 10586.0 from RAM, music on LAN / Engine#4 Adaptive Mode / Q1/-/3/4/5 = 14/-/1/1/1 / Q1Factor = 1 / Dev.Buffer = 4096 / ClockRes = 1ms / Memory = Straight Contiguous / Include Garbage Collect / SFS = 0.10  (max 60) / not Invert / Phase Alignment Off / Playerprio = Low / ThreadPrio = Realtime / Scheme = Core 3-5 / Not Switch Processors during Playback = Off/ Playback Drive none (see OS from RAM) / UnAttended (Just Start) / Always Copy to XX Drive (see OS from RAM) / All Services Off / Keep LAN - Not Persist / WallPaper On / OSD On / Running Time Off / Minimize OS / XTweaks : Balanced Load = *43* / Nervous Rate = 1 / Cool when Idle = 1 / Provide Stable Power = 1 / Utilize Cores always = 1 / Time Performance Index = *Optimal* / Time Stability = *Stable* / Custom Filter *Low* 705600 / -> USB3 *from MoBo* -> Clairixa USB 15cm -> Intona Isolator -> Clairixa USB 1m80 -> 24/768 Phasure NOS1a 75B (BNC Out) async USB DAC, Driver v1.0.4b (4ms) -> Blaxius BNC interlink *-> B'ASS Current Amplifier /w Level4 -> Blaxius Interlink* -> Orelo MKII Active Open Baffle Horn Speakers.
Removed Switching Supplies from everywhere.

Global Moderator
Josef
Audio Loudspeaker
*
Offline Offline

Posts: 157



View Profile
« Reply #25 on: October 22, 2010, 04:20:16 pm »

I already sent previous post without seeing your latest:
Quote
If this ends up in well meant (!) advices how I should organize the program, please let's stop. I mean, I have better things to do than explaining to you (or anyone) why it works like it works.

I'm sad to hear you saying you're not open to ‘well meaning advices’ but it's your forum and you make the rules here so I guess I'll have to shut up.

I hope it's fair though to let me respond to your latest post?
Quote
it could have been nice or maybe helpful if I didn't make all this stuff already, but I did - and so now it will be the other way around : I am (we are) in the hands of Bill Gates how this all works and what it implies. No thank you !

Fair enough, it's your product.
Considering the ease with which this can be tested I'd personally at least try it: even if it does not work in terms of SQ it might reveal something interesting that was not known previously which just might be useful.... But then, that's just me being curious. I guess I'd have to do it myself in order to find out, LOL...

Quote
If it were so that nothing's read into memory and a disk could be treated like memory, *then* we'd have another case.

That is the point I was hoping you would catch on and IMHO a single most important improvement to XX (or, rather, to audio players in general) with potentially greatest benefits:
Look at your RAMDisk. It is nothing but software. A device driver to be exact. And it sits right alongside OS. In fact, it is on 'per tu' basis with OS Kernel.
Compare that with current situation of XX being a 'user' process (as opposed to system) running in what is essentially a virtual machine on top of OS (.NET).
What if XX could be written as a device driver (a la RAMDisk) but with a twist: It 'knows' about its contents not being 'virtual disk files' but sound data that can be directly accessed without having to go through OS file I/O layer at all?

And by virtue of, effectively being a part of OS, what other layers have also been removed? (not having to deal with VM much less garbage collection etc etc.). And because it sits there (like RAMDdisk) it is 'stable' and forever unchanging (except for contents, of course Happy )

Now, how much closer is that to ‘bare metal’, what new possibilities would that open and how far more could XX go and sound then?
Puzzles the mind....

Quote
You can see how ridiculous it is (ok, through my eyes) at the examples of inter process communication. So, we write something to memory (which we do, but which ends up on the "mapped disk"), and now another can read from the same memory address and communicate. Just write some ".dat" files (you know), and be normal !

I see that you have caught on this one too: I was hoping you'd see the possibility of using MMF to 'replace' *.dat & *.dao files.
I don't, however, agree that it is 'ridiculous' at least as a matter of principle:
I believe most people here can agree that introducing RAMDisk brought positive changes in SQ. And, as you said yourself, it is not clear to you (not to anyone else, me included) just WHY is this so.
Is it a crazy notion to suppose that it has at least something to do with eliminating SDD/HDD disk-induced I/O interrupts as such do not exist with RAMDisk?
And if this at least sounds plausible, wouldn't using MMF instead of writing/reading *.dat/*.dao to disk be beneficial purely on principle as it eliminates disk I/O? (Yes, I know it may be a moot point if dat/dao are being written to RAMDisk but it still _has to_ make a difference as path through OS is very different. Maybe that difference turns out to be small so it's not worth it but would that make it 'ridiculous'?)

BTW - I may be wrong (I don't have logs with me) but if I remember correctly when you do either write or read of dat/dao files they had 'non-cacheable' flags set. Those files are so small they can fit quite nicely in OS cache and, as free bonus, you are likely to avoid file I/O interrupts when reading them back in from the Engine as there is high probability they'll be served from cache. I.e. you may get MMF benefits (at least in this case) without having to use MMF (since you seem to have a particular dislike for them.)

Quote
But if I'd behave transparently (for myself) and operate at the byte level, knowing that disk transfers go at the block level, what do you think the result will be ?

I don't know! And you don’t either! That's the whole point, LOL! Happy
What I was (obviously unsuccessfully) trying to communicate is that if there is an interesting approach that has not been tried before it might be worthwhile to check it out (if it does not require too much time). Obviously, a proposal mentioned above, to write XX as music-player-device-driver is a LOT of work but trying out MMF really isn't. (BTW If you interested in pursuing device-driver approach maybe some people would be willing to help just because they find it interesting – for example maybe it could even be me (God forbid) ....)

But to try to answer your question in a more concrete way: I speculate you would _not_ have to worry about speed. I have read about projects using MMF to provide type-ahead suggestions (you know, like when you start typing on Google and it provides suggestions in drop-down as you type) on very, very large files: order of magnitude larger than any sound file you will EVER have - bigger than 32 bits 384kHz Beethoven's 9th gapless LOL Happy And they had to do dynamic binary search on that file too, so they were jumping back & forth around that mega-giga-tera file without problems, where, in contrast, you only have to move sequentially forward.
And OS is prepared for it, so it will preload SFS sections before you need them - OS is sometimes not that bad you know and they have removed most of code Bill Gates wrote in his days....  Happy  

Quote
So, supposed this happens at the byte level anyway, *then* there's a chance for this. But I won't believe this, because e.g. copying to the RAMDisk just takes way longer than copying an internal array (writing to it) which easily goes within 0.1sec for 500MB or so.

Yes, it _has_ to be slower as you are going through whole OS file I/O stack which can easily be seen in debugger: there is _no_ difference compared to SDD/HDD disk access so it can never do 500MB in 0.1 sec but why would you need that?
Sound card doesn’t need nowhere near that amount of data in that time period even with 8x oversampling etc. And you can ask MMF to open a view on whole file instead of SFS chunks - As mentioned, it would read in as you move forward and you would not need to manage memory because MMF memory _does not count_ in your process set! So we could have even bigger RAMDisks as XX would be using only couple MBs Happy

But OK, I guess I misunderstood the purpose of this forum: I foolishly believed it was a place to discuss possible improvements to XX in particular and explore new boundaries in computer-based audio reproduction in general. Seems I got it all wrong so I apologize and promise to shut up.
Logged
PeterSt
Administrator
High Grade Audiophile
*****
Offline Offline

Posts: 15507



View Profile Email
« Reply #26 on: October 22, 2010, 05:52:31 pm »

Quote
But OK, I guess I misunderstood the purpose of this forum: I foolishly believed it was a place to discuss possible improvements to XX in particular and explore new boundaries in computer-based audio reproduction in general. Seems I got it all wrong so I apologize and promise to shut up.

So ... You put a post A here to which I respond for a first half indicating that you don't need to interfere with some things because it only takes me the time to explain to you that you're not right. Next I spend time on your good ideas from the same post A in a second half - which is exacly what you last post E is about, we both have a nice post C and D in between, and now you suddenly end with this ? what happened to you ?
Maybe you read the gueste of the sequence again and come to the conclusion that you read only half of both mine and your posts ?
I really don't get it.

About the .dat files ... oh dear, I better had not bring that up. Not in front of your eyes (yes, I remember now Happy).

So, I suggest not to act foolish (me included), get back your good mood, or reverse whatever it is that needs reversing. Ok ?


Let's go on.

Quote
I don't know! And you don’t either!

Wrong, I do know. It will be way to slow as I explained. Please don't get me wrong now : The fact that you don't know while I do, should not lead to any strange remark about it being my forum and my rules (I say this just in case). The point is, it will (apparently) need explanation I don't feel the need to. This is not your fault, but the coincidence of me having an advantage knowing what is all happening in there. Look :

Quote
there is _no_ difference compared to SDD/HDD disk access so it can never do 500MB in 0.1 sec but why would you need that?

How do you think the expanded / filtered / upsampled etc. track emerge ?
Must I explain this ? isn't it the most obvious ?


Please read back the last couple of posts, and see how I appreciate your ideas which are unrelated to my way of working - and how I proceeded on that. And still will, whether you (by now) like it or not. Hahaha.
And oh, I didn't read it back, but I don't think I said I wouldn't try the RAMDisk thing either. I only predicted (or at least wanted to) that it will cause the opposite of what's generally needed : less interference of the OS. If you can explain that to me otherwise, I really (!) like to hear it, but I think you just extensively agreed by your own outlay. Or ?

Having it all at some driver level ... also nice. But (obviously) the next thing what will happen is that Engine3 runs as a service, so it's already closer to that. Next ? next I don't see the difference (with driver level) at this moment, but possibly this difference will be there because of OS priorities etc.; I don't know yet.


Allright, let's end this kind of silly post with : if you don't stop smoking I'll start drinking. It's time for that anyway, that latter.
Again, thanks, and I mean that.
Peter


PS:
Quote
so I apologize and promise to shut up.
I just dragged you in. fool
Happy

Logged

For the Stealth III LPS PC :
W10-14393.0 - Oct 31, 2018 (2.10)
XXHighEnd Mach III Stealth LPS PC -> Xeon Scalable 14/28 core with Hyperthreading On (set to 10/20 cores) @~660MHz, 48GB, Windows 10 Pro 64 bit build 14393.0 from RAM, music on LAN / Engine#4 Adaptive Mode / Q1/-/3/4/5 = 30/-/1/1/1/ Q1Factor = 10 / Dev.Buffer = 4096 / ClockRes = 15ms / Memory = Straight Contiguous / Include Garbage Collect / SFS = 140.19  (max 140.19) / not Invert / Phase Alignment Off / Playerprio = Low / ThreadPrio = Realtime / Scheme = Core 3-5 / Not Switch Processors during Playback = Off/ Playback Drive none (see OS from RAM) / UnAttended (Just Start) / Always Copy to XX Drive (see OS from RAM) / Stop Desktop, Remaining, WASAPI and W10 services / Use Remote Desktop / Keep LAN - Not Persist / WallPaper On / OSD Off (!) / Running Time Off / Minimize OS / XTweaks : Balanced Load = 35 / Nervous Rate = 10 / Cool when Idle = n.a / Provide Stable Power = 0 / Utilize Cores always = 1 / Time Performance Index = Optimal / Time Stability = Stable / *Arc Prediction Filtering (16x)* / Always Clear Proxy before Playback = On -> USB3 from MoBo -> Lush^2*A:B-W-Y-R, B:B-W-R* USB 1m00 -> Phisolator 24/768 Phasure NOS1a/G3 75B (BNC Out) async USB DAC, Driver v1.0.4b (16ms) -> B'ASS Current Amplifier -> *Blaxius^2 A:B-R, B:B-R* Interlink -> Orelo MKII Active Open Baffle Horn Speakers.
Removed Switching Supplies from everywhere (also from the PC).

For a general PC :
W10-10586.0 - May 2016 (2.05+)
*XXHighEnd PC -> I7 3930k with Hyperthreading On (12 cores)* @~500MHz, 16GB, Windows 10 Pro 64 bit build 10586.0 from RAM, music on LAN / Engine#4 Adaptive Mode / Q1/-/3/4/5 = 14/-/1/1/1 / Q1Factor = 1 / Dev.Buffer = 4096 / ClockRes = 1ms / Memory = Straight Contiguous / Include Garbage Collect / SFS = 0.10  (max 60) / not Invert / Phase Alignment Off / Playerprio = Low / ThreadPrio = Realtime / Scheme = Core 3-5 / Not Switch Processors during Playback = Off/ Playback Drive none (see OS from RAM) / UnAttended (Just Start) / Always Copy to XX Drive (see OS from RAM) / All Services Off / Keep LAN - Not Persist / WallPaper On / OSD On / Running Time Off / Minimize OS / XTweaks : Balanced Load = *43* / Nervous Rate = 1 / Cool when Idle = 1 / Provide Stable Power = 1 / Utilize Cores always = 1 / Time Performance Index = *Optimal* / Time Stability = *Stable* / Custom Filter *Low* 705600 / -> USB3 *from MoBo* -> Clairixa USB 15cm -> Intona Isolator -> Clairixa USB 1m80 -> 24/768 Phasure NOS1a 75B (BNC Out) async USB DAC, Driver v1.0.4b (4ms) -> Blaxius BNC interlink *-> B'ASS Current Amplifier /w Level4 -> Blaxius Interlink* -> Orelo MKII Active Open Baffle Horn Speakers.
Removed Switching Supplies from everywhere.

Global Moderator
PeterSt
Administrator
High Grade Audiophile
*****
Offline Offline

Posts: 15507



View Profile Email
« Reply #27 on: October 22, 2010, 06:14:33 pm »

Quote
What if XX could be written as a device driver (a la RAMDisk) but with a twist: It 'knows' about its contents not being 'virtual disk files' but sound data that can be directly accessed without having to go through OS file I/O layer at all?

FYI : For KS I had that running. Really. In an indirect way, but still the net effect was just like that. But, it can only work for certain devices, on XP it wouldn't work at all, etc. At the time I didn't want to dive into "interference" at the driver level, which actually would be needed. Today still it could be a tad "too far" for me, but I really like the idea even better than before (knowing more what's happening etc.).

grazy
Logged

For the Stealth III LPS PC :
W10-14393.0 - Oct 31, 2018 (2.10)
XXHighEnd Mach III Stealth LPS PC -> Xeon Scalable 14/28 core with Hyperthreading On (set to 10/20 cores) @~660MHz, 48GB, Windows 10 Pro 64 bit build 14393.0 from RAM, music on LAN / Engine#4 Adaptive Mode / Q1/-/3/4/5 = 30/-/1/1/1/ Q1Factor = 10 / Dev.Buffer = 4096 / ClockRes = 15ms / Memory = Straight Contiguous / Include Garbage Collect / SFS = 140.19  (max 140.19) / not Invert / Phase Alignment Off / Playerprio = Low / ThreadPrio = Realtime / Scheme = Core 3-5 / Not Switch Processors during Playback = Off/ Playback Drive none (see OS from RAM) / UnAttended (Just Start) / Always Copy to XX Drive (see OS from RAM) / Stop Desktop, Remaining, WASAPI and W10 services / Use Remote Desktop / Keep LAN - Not Persist / WallPaper On / OSD Off (!) / Running Time Off / Minimize OS / XTweaks : Balanced Load = 35 / Nervous Rate = 10 / Cool when Idle = n.a / Provide Stable Power = 0 / Utilize Cores always = 1 / Time Performance Index = Optimal / Time Stability = Stable / *Arc Prediction Filtering (16x)* / Always Clear Proxy before Playback = On -> USB3 from MoBo -> Lush^2*A:B-W-Y-R, B:B-W-R* USB 1m00 -> Phisolator 24/768 Phasure NOS1a/G3 75B (BNC Out) async USB DAC, Driver v1.0.4b (16ms) -> B'ASS Current Amplifier -> *Blaxius^2 A:B-R, B:B-R* Interlink -> Orelo MKII Active Open Baffle Horn Speakers.
Removed Switching Supplies from everywhere (also from the PC).

For a general PC :
W10-10586.0 - May 2016 (2.05+)
*XXHighEnd PC -> I7 3930k with Hyperthreading On (12 cores)* @~500MHz, 16GB, Windows 10 Pro 64 bit build 10586.0 from RAM, music on LAN / Engine#4 Adaptive Mode / Q1/-/3/4/5 = 14/-/1/1/1 / Q1Factor = 1 / Dev.Buffer = 4096 / ClockRes = 1ms / Memory = Straight Contiguous / Include Garbage Collect / SFS = 0.10  (max 60) / not Invert / Phase Alignment Off / Playerprio = Low / ThreadPrio = Realtime / Scheme = Core 3-5 / Not Switch Processors during Playback = Off/ Playback Drive none (see OS from RAM) / UnAttended (Just Start) / Always Copy to XX Drive (see OS from RAM) / All Services Off / Keep LAN - Not Persist / WallPaper On / OSD On / Running Time Off / Minimize OS / XTweaks : Balanced Load = *43* / Nervous Rate = 1 / Cool when Idle = 1 / Provide Stable Power = 1 / Utilize Cores always = 1 / Time Performance Index = *Optimal* / Time Stability = *Stable* / Custom Filter *Low* 705600 / -> USB3 *from MoBo* -> Clairixa USB 15cm -> Intona Isolator -> Clairixa USB 1m80 -> 24/768 Phasure NOS1a 75B (BNC Out) async USB DAC, Driver v1.0.4b (4ms) -> Blaxius BNC interlink *-> B'ASS Current Amplifier /w Level4 -> Blaxius Interlink* -> Orelo MKII Active Open Baffle Horn Speakers.
Removed Switching Supplies from everywhere.

Global Moderator
Telstar
Audio Addict
***
Offline Offline

Posts: 729


View Profile
« Reply #28 on: October 22, 2010, 09:30:35 pm »

At the time I didn't want to dive into "interference" at the driver level, which actually would be needed. Today still it could be a tad "too far" for me, but I really like the idea even better than before (knowing more what's happening etc.).

grazy

I dont have the technical background to judge this, but I would do an attempt of the engine working at driver level and see if it changes the SQ or not (while still using the ramdisk of course).
Logged

(2nd Apr 2018)
Software:
W10 14393 Pro x64 | XXHE 2.10 | MinOS | Q=14x1/0/0/0/0 | SFS 5,19 mixed contiguous | Nervous rate 1 | 4096k buffer |

Hardware:
OrigenAE H5 case | E5300 fanless |  8GB RAM | Winmate DC-DC fanless PSU | OS on SSD | Renesas USB3 pcie card | Belden highspeed usb cable | Audio-gd dac19 NOS with sigxer F1 | My_ref_FE mono amps | Albedo Apex speakers
Josef
Audio Loudspeaker
*
Offline Offline

Posts: 157



View Profile
« Reply #29 on: October 22, 2010, 10:45:33 pm »

Quote
So, I suggest not to act foolish (me included), get back your good mood, or reverse whatever it is that needs reversing. Ok ?

Only if I get an invitation to DAC audition and get a glass of nice red wine too!

Quote
How do you think the expanded / filtered / upsampled etc. track emerge ?
Must I explain this ? isn't it the most obvious ?

Believe it or not NO but I see now what you mean and I'm afraid this time you misunderstood me (for a change).

I can see you want ultra speed to write to memory so this CPU intensive 'stressful' task of converting a track (well only if you use 8x Arc etc otherwise it's no so stressful but OK lets assume worst case) can be done as quickly as possible - but you don't, repeat, don't need same speed for _reading_ the file from disk - in fact, you cannot have it even if you wanted it. What?
Well, as we seem to misunderstand each other frequently let me try to explain using an example:

- Suppose we have XX written as a device driver (_not_ a 'RAMDisk' - let's call it 'Xtra Xeroxed audio' driver or 'XX driver' for short Happy ).
- XX driver has a defined RAM size that it uses as a music data buffer - just like you would set up your RAMDisk size now
- Unlike RAMDisk however, XX driver does NOT appear as a disk D: E: F: whatever, because it is NOT an implementation of disk I/O interface (it's an 'audio playback' driver after all - just repeating to make sure I'm obvious)
- access to XX driver is controlled and only possible via XX GUI. Otherwise it is completely invisible (except that your usable RAM is visibly reduced by XX driver's buffer size also just like with RAMDisk now)

Ok, that was easy - But how does it work?
In 6 steps:

Step 1: GUI starts working just like it does now: prepares *.dat/*.dao instruction files and sends them to XX driver (You'd have to use a proprietary API or whatever method you like (oh yes, MMFs could work too, LOL) as you won't be able to 'just copy a file' because XX driver does not have a file I/O interface at all!)

Step 1 addendum: BUT, when 'Copy to XX' is checked it does NOTHING! (apart from noting the setting with all others in dat/dao instruction files it just sent to XX driver)

Step 1 addendum 2: XX driver (which replaces what is now XXEngine3.exe) has full, absolute, unlimited control over its allocated buffer! It does NOT have to go thru OS file I/O stack to write to its RAM! Hence, it can do ultra-fast conversion in its own buffer which is only limited by speed of the computer. IMHO This is not really that important (see Step 3) but I'm making a note as you seem to think it is.

Step 2: So, our perky XX driver opens tracks in playlist (specified by dat/dao) via MMFs! (I can hear you: 'Oh my God here he goes again I just told him it's baaaaad, noooooooo......')

Ahem, why MMFs? Because that way XX driver needs 0, repeat, zero memory to read _all_ tracks regardless of their size! So we can have XX driver set to use 3GB RAM if we wanted to, and we do, so damn the RAM we want more music! (notice, we do NOT have nor will ever need a RAMDisk!). And if we had 8GB RAM we'd make a bloody 7GB XX driver because we're audio freaks and have dedicated audio computer damn it etc etc. you get a picture....(not that it wouldn't work on a 2GB RAM Intel Atom notebook but our XX driver buffer would naturally have to be proportionally smaller....)
 
Step 3: As XX reads in tracks one by one (for sake of argument, let's say it reads 10MB 'chunks' at a time - it really could be anything, does not matter as you are limited by HDD/SDD disk speed anyhow) each 'chunk' gets 'converted' right there 'in-flight' and bytes fresh & ready for sound card are put in XX driver's internal buffer starting from offset 0.

Step 4. Playback thread starts playing music as soon as first chunk is 'converted' OR as soon as buffer is completely full (similar to current 'start engine while converting' setting so it's under user control)
In any case, loading thread stops loading/converting when buffer is full.

Step 5. Music plays, and plays, and plays and generally absolutely nothing interesting happens....track 1, gapless into track 2, I did it myyyy waaayyy, track 3, she loves you, yeah, yeah, yeah, track 4....track 7?8?9?

Step 6. Ooops, XX driver sees there's only, say, 20 seconds of music left in the buffer - what do we do? Panic! Or maybe not: Go to Step 3, repeat until playlist exhausted.

When playlist is done (or if user stops playback) XX GUI gets reactivated same as now.

Notice when you start new playlist absolutely _nothing_ has changed in memory layout: XX driver is there all the time with its ultra fast ultra simple (single!) buffer and will just write new playlist content over previous....no allocations, deallocations, garbage collections, VM swapping, no nothing.....just music playing forever, undisturbed............






 
Logged
Pages: 1 [2] 3  All
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1 RC2 | SMF © 2001-2005, Lewis Media Valid XHTML 1.0! Valid CSS!
Page created in 0.087 seconds with 19 queries.