Tamura Kent formally unveils his plans to add new functionality to NetBSD's audio framework (audio(9) and audio(4)) - specifically, the addition of an audio converter pipeline and in-kernel mixing. These additions along with audio device cloning would make it possible to natively support hardware mixing without the use of a software based soundserver. However, Kent's initial plans are to support software mixing first, with hardware supported mixing being considered a "nice-to-have".

Tamura explains:

"I guess it is hard work to identify a common representation of mulit-voice hardwares and change the existing
drivers. The hardware-mixing era already ended, and there is little possibility that new hardware-mixing audio controller is introduced."

Read on for the original post and abbreviated thread. Please follow the archive link for the full thread.

From: TAMURA Kent [email blocked]
To: netbsd-tech-kern
Subject: Ideas on the audio framework
Date: 2004-12-03 14:36:18

I'm planning to add the following capabilities to the audio
framework:
 A. Audio converter pipeline
 B. In-kernel audio mixing


A. Audio converter pipeline

The goals are:
* add capability to run multiple audio stream converters between
  read(2)/write(2) and the hardware buffer
* unify interfaces of playing converters and recording converters
* represent aurateconv as a converter

In the current framework, a hardware driver can set just one
converter to audio_params::sw_code.  So, a sw_code
implementation have to do everything of required conversions.

We have six mulaw_to_something() functions in sys/dev/mulaw.c.
	mulaw_to_ulinear16_le()
	mulaw_to_ulinear16_be()
	mulaw_to_slinear16_le()
	mulaw_to_slinear16_be()
	mulaw_to_ulinear8()
	mulaw_to_slinear8()
We can get the same results as theirs by combinations of
mulaw_to_slinear16_le(), swap_bytes(), change_sign16_le(), and
linear16_to_linear8_le().  We can reduce such redundant code by
introducing the audio converter pipeline.

The current sw_code interface is curious.  The third parameter
is the output size for playing converters, and the input size
for recording converters.  Of course, playing converters can not
be used for recording.  A hardware driver needs to set correct
values to audio_params::factor and factor_denom though they are
fixed values for a converter.

If aurateconv is repreesnted as a converter and the converter
pipeline is implemented, we can easily realize sampling rate
conversion for ADPCM-only hardware like sys/arch/x68k/dev/vs.c.
	ADPCM stream
	  |  [ADPCM -> PCM converter]
	PCM stream
	  |  [aurateconv]
	PCM stream
	  |  [PCM -> ADPCM converter]
	ADPCM stream
	  |
	hardware

What is required?
  - define new converter interface
  - rewrite all sw_code implementations for the new interface
  - parameters of audio_hw_if::set_params() may be changed
  - ...


B. In-kernel audio mixing

The goals are:
* /dev/audioN can be opened by multiple processes, and
* audio streams written by the processes are mixed in the kernel
  and played with a single hardware.

It would not be hard to realize the audio mixing if we had the
audio converter pipeline and the device cloning.  To mix audio
streams, they must be the single PCM format and the single
sampling rate.  The audio converter pipeline would help it.

----------------------------------------------------------------

Any comments?

I have a concrete design of the audio converter pipeline, and
will start implementation soon.

-- 
TAMURA Kent


From: Ben Collver [email blocked]
Subject: Re: Ideas on the audio framework
Date: 2004-12-03 15:24:02

On Fri, Dec 03, 2004 at 11:36:18PM +0900, TAMURA Kent wrote:
> I'm planning to add the following capabilities to the audio
> framework:
>  A. Audio converter pipeline
>  B. In-kernel audio mixing
> [...]
> Any comments?

Woohoo!  I'd be happy to test this.

Ben


From: Sascha Retzki [email blocked]
Subject: Re: Ideas on the audio framework
Date: 2004-12-03 16:18:46

On Fri, Dec 03, 2004 at 11:36:18PM +0900, TAMURA Kent wrote:
> Any comments?

Without really knowing what has to be done implementation-wise, from a user
and a user of the audio(4) API, I'd *love* that :)


With best regards,


Sascha Retzki


From: Nathan J. Williams [email blocked]
Subject: Re: Ideas on the audio framework
Date: 2004-12-03 17:14:38

TAMURA Kent writes:

> B. In-kernel audio mixing
> 
> The goals are:
> * /dev/audioN can be opened by multiple processes, and
> * audio streams written by the processes are mixed in the kernel
>   and played with a single hardware.
> 
> It would not be hard to realize the audio mixing if we had the
> audio converter pipeline and the device cloning.  To mix audio
> streams, they must be the single PCM format and the single
> sampling rate.  The audio converter pipeline would help it.

How will this work for hardware that natively supports different
formats (sample rates, say)? If multiple streams at different rates
are mixed, will the lower-rate ones all have to be resampled? Or will
it resample both to some (maximum?) rate?

        - Nathan


From: Pavel Cahyna [email blocked]
Subject: Re: Ideas on the audio framework
Date: 2004-12-03 19:18:21

Hello,

On Fri, 03 Dec 2004 14:36:18 +0000, TAMURA Kent wrote:

> B. In-kernel audio mixing
> 
> The goals are:
> * /dev/audioN can be opened by multiple processes, and

This shounds great. Here is a hopefully simple suggestion:
currently, two processes can't open the same /dev/audioN device, even if
one wants to play only and the other to record only. Could your future
modifications allow that, please? It even might be simpler to implement
that the full in-kernel mixing, and sometimes also very useful.

Thanks and bye	Pavel


From: Jason Thorpe [email blocked]
Subject: Re: Ideas on the audio framework
Date: 2004-12-03 20:16:08

On Dec 3, 2004, at 9:14 AM, Nathan J. Williams wrote:

> How will this work for hardware that natively supports different
> formats (sample rates, say)? If multiple streams at different rates
> are mixed, will the lower-rate ones all have to be resampled? Or will
> it resample both to some (maximum?) rate?

Not use multiple formats... but some modern audio chips have multiple 
voices and can do the mixing in hardware... we should support that in 
the same interface, as well.

         -- Jason R. Thorpe


From: Wolfgang S. Rupprecht [email blocked]
Subject: Re: Ideas on the audio framework
Date: 2004-12-03 20:22:01

(TAMURA Kent) writes:
> We have six mulaw_to_something() functions in sys/dev/mulaw.c.
> 	mulaw_to_ulinear16_le()
> 	mulaw_to_ulinear16_be()
> 	mulaw_to_slinear16_le()
> 	mulaw_to_slinear16_be()
> 	mulaw_to_ulinear8()
> 	mulaw_to_slinear8()

Any chance of adding the framework for 32-bit samples?  There are
quite a few low-cost sound cards that now support >16bit samples.
Some on-disk formats allow storage of >16bit samples.  It seems like a
shame not to allow producer and consumer to communicate.

-wolfgang

PS. Even 24-bit sound cards aren't that expensive any more.  $100 buys
    you an 8-channel 24bit/channel "M-Audio Revolution 7.1".
-- 
Wolfgang S. Rupprecht


From: Rui Paulo [email blocked]
Subject: Re: Ideas on the audio framework
Date: 2004-12-03 20:31:12

On 2004.12.03 23:36:18 +0000, TAMURA Kent wrote:
> B. In-kernel audio mixing
> 
> The goals are:
> * /dev/audioN can be opened by multiple processes, and
> * audio streams written by the processes are mixed in the kernel
>   and played with a single hardware.

Some cards can do this in hardware (SB Live!, for example), but they
have a "limited" (64 IIRC) channels available for mixing. Will your
implementation be based on software (like ESD) or hardware, or even
both ?

        -- Rui Paulo

• Archive of the Above Thread