DSP Robotics Support

Hi gang,

over Christmas I did some reading to catch up on KG's Frequency Delay Line implementation. The general idea of partitioned convolutions is to partition the IR into segments, convolve the input with each segment separately, and combine the results using an apropriate schedule. This can be done entirely in the frequency domain, reusing previous input FFT blocks for convolutions with later IR segments, and adding everything before IFFT.

For long convolutions it is best to use multiple FDLs with different FFT sizes: long-sized FFTs to efficiently
implement long convolutions, and short-sized FFT to achieve low latency. I think the original work for this insight is due to Guillermo Garcia.

So I have designed a convolution reverb demo based on these principles. In order to address the issues mentioned in my earlier post, I had to recreate the FFT/IFFT code and stuff it together with the convolutions in one ASM code block (per FDL). Many technical details there, heavy use of SIMD to speed up complex multiplies, etc. Anyway, the result so far is not so bad:

- zero latency
- fully meshed stereo
- 131k IR size (3 seconds @44.1 kHz)
- 20% CPU (Athlon II X4 640 @3 MHz)

I am supplying the fsm together with a Lexicon 480L impulse response "Large Hall" (thanks Spogg for the link!). You have to extract the IR before loading it (blame it on the 2 MB forum limit ), however you can just as well load your favorite IR. Note that audio is interrupted and the code is recompiled upon IR loading.

Give it a test and come back with feedback, please. There more explanation and further reading inside the fsm.

If you think this has potential we could join forces, make a nice GUI and release it as a Flowstone User Group Plugin on KVR.

Absolutely wonderful!

I tried it with my microphone and had some fun in the "hall"!
On my core i7 32 bit system running Win 7 Flowstone showed a CPU meter varying between 5 and 6.5%. The variations seemed random.

The sheer cleverness of this work just astonishes me Martin; I wish I had just 10% of your abilities!

I was a bit concerned when I saw the limit was 3 seconds but, on this demo, it proved to be sufficient with a long, smooth and accurate tail.
A very minor point but one which puzzled me: when I operated the interuptor module it killed the echo of course but I found that when I turned it back on the tail was distorted. It seemed to be based on the point in the tail where I turned it off. I don't imagine that a final implementation would use this method anyway; probably a dry/wet mix would be better.
I imagine this has something to do with interrupting the stream. I've found that using selectors to mute mono streams often gives weird and bad results, so much so that I avoid it at all times now.

I can't think of any other improvements other than a dry/wet knob, maybe an input level control and a bar graph. Collect a good range of licence-free impulses, get tulamide to create the GUI then...Job done!

Bravo!!

Spogg

Thanks for the feedback Spogg! The CPU readings fluctuate because there are spikes every 300 ms where an FFT/IFFT is performed. There is no easy way to distribute this load over the time interval because it is a block operation. The real issue is if the operation cannot be completed before the next sample, there will be clicks and pops in the audio streams. You might have to increase the audio adapter buffer size in that case.

The interrupt module is really only there to load and correctly initialize a new IR (the sequence in the background being: disable audio - disconnect module - load IR - reconnect module - enable audio). It is certainly not meant as a dry/wet switch. It should probably not even be visible in the final version. Actually there is a volume control in the processed signal path which you can use to set the amount of reverbration.

If we really want to take this further we would need help from someone who has successfully released a couple of audio plugins. Also input regarding desirable features and usability aspects from people who work with such plugins in DAWs would be welcome.

I waited as long as possible to give others a chance to reply. But since I'm about to pee my pants from all the excitement, I have to answer.

I was for superlatives. But I couldn't find words that respect what you've done here. And that's the main issue I think. People don't realize that it is like a quantum leap for Flowstone. And so they don't give you the respect you deserve!

This is the highest art of dsp coding you can currently get, guys. Don't dare to ignore this thread! Show some respect!

About the convolution engine:
- If it works, it sounds as expected.
- What happens to files, that represent a longer tail than 3s? This is an important question, before I can report certain issues.
- Sometimes after selecting another IR, the engine is not accurate. It sounds as if the file is played one octave lower.
- I didn't test this version in Reaper yet, but Flowstone reports a total CPU load of about 9% in the task manager, which is extra-ordinary for my pc!

From me as a heavy user of the convolution engine ReaVerb in Reaper:
Here are screenshots of ReaVerb.
1) https://www.dropbox.com/s/65mpxgkzdnix4qk/ReaVerb01.png?dl=0
2) https://www.dropbox.com/s/z5w21ae29lezzb2/ReaVerb02.png?dl=0
3) https://www.dropbox.com/s/ifgc8eo1l7duq08/ReaVerb03.png?dl=0
4) https://www.dropbox.com/s/p9ik2o8vh785jkt/ReaVerb04.png?dl=0

On the second screen, an option is activated "Apply -18dB gain". As you can see, this is applied to the IR file. There are additional wet and dry signal volume sliders. I don't know if this is just attenuating the file, but this function is like magic. If you use it, you mostly don't need to touch any of the various signal gain controls.

On the third screen you see a lowpass and highpass filter. I use them a lot to get the right tone of the IR. Especially when using it for amp-simulations or other stuff. This could be applied to the file in RAM (or a copy of the file in RAM). No need for realtime filtering!

On the fourth screen you see Trim/Gain/Stretch applied. Again, these are all functions applied to the IR, no need for realtime processing. I use these a lot to get the timing, tone and shape of the IR right.

Those would be features I'd really like to see in a Flowstone Community Convolution Engine. The difference between working with and without them is similar to working with a full featured sampler and a "rompler" (a sample player without synthesis functions). If nothing else can be realized at least the highpass and lowpass filters should be in. Then panning (very important for odd microphone settings when the IR was recorded), then width (which runs from stereo to mono), then start position and finally stretch, to show the order of importance for me.

@Spogg
Free available IR files doesn't really mean free. All of them come with a license of some sort, mostly disallowing commercial use. If we want to sell this, we can't use those IRs, which is why I asked for help.

If someone has a microphone, an active speaker and a mobile device that can record and play at the same time (= all current mobile devices down to smartphones), please consider visiting interesting places (a church, a gym, an old warehouse, your living room, whatever) and recording a sine sweep. I can convert those into IRs, we just need to use the same sine sweep (it has to go all the range from 20Hz to 22kHz over some seconds)

Alternatively we could offer the convolution engine for free, in which case there a quite a few IRs that allow to be bundled. We would still have the attention effect for Flowstone, but would of course make no money to invest in Flowstone's audio/dsp part development.

And of course I'll be doing a GUI. I'll work out drafts that we then use to decide which direction to go.

Hi tulamide

I'm a bit puzzled by the frequency sweep thing. I know nothing of the theory but all the IR wavs I've heard seem to be based on a click with echo, like the demo one that Martin used.
I have a portable stereo recorder but I'm not sure why I can't just record a tail. The question I have in that regard would be what do I use as an impulse? Could I just strike something? If so what could I hit?

Also, if using a sweep, wouldn't the results be affected by the speaker/enclosure quailty?

Sorry for my naive questions...

Cheers

Spogg

Spogg wrote:Hi tulamide

I'm a bit puzzled by the frequency sweep thing. I know nothing of the theory but all the IR wavs I've heard seem to be based on a click with echo, like the demo one that Martin used.
I have a portable stereo recorder but I'm not sure why I can't just record a tail. The question I have in that regard would be what do I use as an impulse? Could I just strike something? If so what could I hit?

Also, if using a sweep, wouldn't the results be affected by the speaker/enclosure quailty?

Sorry for my naive questions...

Cheers

Spogg

What you hear might remind you of a click with echo, but that's what is the desired result (hence impulse response). This sound is the result of a process called deconvolution. What you want is a "blueprint" of a reverb that carries all informations for the whole audible frequency range. Therefore you need to record something that contains all those frequencies in the room whose reverb you want to get. In a postprocess you take this recording and calculate out the signal you used, and so only the reverb share is left.

In the early days, people recorded just something noisy, because noise naturally contains a lot of frequencies. Some recorded a handclap, others white noise, etc. But today, it is common consent that using a sine sweep yields the best results. That's because a sine is the purest and most natural of all sound waves and a sweep makes sure that you don't rely on randomness (like with noise), but exactly define the range of frequencies that will be recorded.

I bet you already guessed by now that for the deconvolving FFT is used once again. But don't ask me how exactly it works. It has to do with transferring time to frequency domain, doing calculations and then transferring frequency back to time domain.

I wonder if Jean Baptiste Joseph Fourier was aware of what he found out when he came up with his Fourier analysis? I quote wikipedia, because I even don't understand that simple sentence: "Fourier's observation that some discontinuous functions are the sum of infinite series was a breakthrough."

I was talking about frequencies, so you might already see that each and every IR is always just a footprint of the used hardware. That doesn't mean that the result are better or worse - they are just different depending on the hardware. Of course, if your speaker doesn't swing at 30 Hz, the recording won't include the frequency. And if the speaker does, but the microphone is weak at low frequencies, it won't be recorded either. But that's just the extremes. 99% of the times you will use frequencies way above and below the limits (My bassdrums have a core frequency of 90 to 100 Hz, for example, a double bass around 60 Hz, and I don't think I ever used any sound that has significant frequencies above 18kHz.

In the end we would have to test it. If you want to give it a try (in one of your rooms maybe), I'll prepare the sine sweep for you.

Thanks Tulamide, your input is very instructive. I was happy to see that neither you nor Spogg reported clicks, dropouts, or even crashes. That's good. Another good thing is the apparently modest CPU load, which leaves us some headroom for longer tails. I could make the convolution engine adapt to the loaded IR size so it would use less CPU for short tails and vice versa. What would be the longest IR we should be targeting?

On the other hand, you both experienced problems after loading a new IR so that's something I need to look into.

The controls on the screenshots make sense to me (most of them anyway). It will be tricky, though, to edit the IR while it is playing. Hmmm...

But hey, we do not want to just reproduce what already exists, do we? So my question is: what would be really cool to have in a convolution reverb? I can think of synthesized IRs to create virtual, if not impossible spaces.

martinvicanek wrote:problems after loading a new IR so that's something I need to look into.

Yup, there is a problem with the memins. It seems that once you start the stream you can not load another IR, not even with the interrupt sequence. Hell it's so frustrating, I just had a blue screen.

tulamide wrote:In the end we would have to test it. If you want to give it a try (in one of your rooms maybe), I'll prepare the sine sweep for you.

Go ahead with this then please!
I don't have a suitable portable setup for playing back a sweep in a public place, plus I think that such an activity, for example in a church, might need permission and an appointment and maybe even a donation!
In the meantime I'll experiment with a clap at the top of our stairs since there's a decent short and damped tail there. I'll upload when I've made and tested it...

@martin
I like the idea of the code adapting to the length of the loaded file and if this helps with CPU that's a big bonus. Otherwise I would vote for 5 seconds max since any more than this would be of little use in practice in my view, even for Enya.

I didn't actually test for loading a new IR. My distortion came when I operated the interuptor switch off/on. I'll try this with my new IR wav when I've done it and report back.

Great project

Cheers

Spogg

@Spogg
Here's an article about the whole process of creating IRs, including pictures of a recording session to get the IR from the inside of a bottle: http://designingsound.org/2012/12/recording-impulse-responses/
And here is an article describing an user's way of recording IRs and his experiences: http://www.creativefieldrecording.com/2014/03/19/the-quick-easy-way-to-create-impulse-responses/
Since he used the vst plugin Altiverb for the deconvolving, you can ignore everything that points to Altiverb, but both articles are very good and I recommend reading them in detail.

@Martin
The memin issues you encountered remind me a lot of the ones I had when we made that Ruby to mem solution. I needed to reset the float array inputs, because otherwise they (which means the stored old data) were always used instead of the new data. A very strange issue, that I could only resolve with that module with the four switches, if you remember. Also, the mem blocks needed to be free from access. In conclusion I would blame the green float array outputs, not the memins. Maybe that helps?