audio: fix saturation nonlinearity in clip_* functions

The current positive limit for the saturation nonlinearity is
only correct if the type of the result has 8 bits or less.

Signed-off-by: Volker Rümelin <vr_qemu@t-online.de>
Message-id: 20200308193321.20668-5-vr_qemu@t-online.de
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
master
Volker Rümelin 2020-03-08 20:33:20 +01:00 committed by Gerd Hoffmann
parent 4218fdd77f
commit 194bdf5069
1 changed files with 2 additions and 3 deletions

View File

@ -83,10 +83,9 @@ static inline int64_t glue (conv_, ET) (IN_T v)
static inline IN_T glue (clip_, ET) (int64_t v)
{
if (v >= 0x7f000000) {
if (v >= 0x7fffffffLL) {
return IN_MAX;
}
else if (v < -2147483648LL) {
} else if (v < -2147483648LL) {
return IN_MIN;
}