I'm developing Flutter plugin which is targeting only Android for now. It's kind of synthesis thing; Users can load audio file into memory, and they can adjust pitch (not pitch shift) and play multiple sound with the least delay using audio library called Oboe.
I managed to get PCM data from audio files which MediaCodec class supports, and also succeeded to handle pitch by manipulating playback via accessing PCM array manually too.
This PCM array is stored as float array, ranging from -1.0 to 1.0. I now want to support panning feature, just like what internal Android class such as SoundPool. I'm planning to follow how SoundPool is handling panning. There are 2 values I have to pass to SoundPool when performing panning effect : left, and right. These 2 values are float, and must range from 0.0 to 1.0.
For example, if I pass (1.0F, 0.0F), then users can hear sound only by left ear. (1.0F, 1.0F) will be normal (center). Panning wasn't problem... until I encountered handling stereo sounds. I know what to do to perform panning with stereo PCM data, but I don't know how to perform natural panning.
If I try to shift all sound to left side, then right channel of sound must be played in left side. In opposite, if I try to shift all sound to right side, then left channel of sound must be played in right side. I also noticed that there is thing called Panning Rule, which means that sound must be a little bit louder when it's shifted to side (about +3dB). I tried to find a way to perform natural panning effect, but I really couldn't find algorithm or reference of it.
Below is structure of float stereo PCM array, I actually didn't modify array when decoding audio files, so it should be common structure
[left_channel_sample_0, right_channel_sample_0, left_channel_sample_1, right_channel_sample_1,
...,
left_channel_sample_n, right_channel_sample_n]
and I have to pass this PCM array to audio stream like c++ code below
void PlayerQueue::renderStereo(float * audioData, int32_t numFrames) {
for(int i = 0; i < numFrames; i++) {
//When audio file is stereo...
if(player->isStereo) {
if((offset + i) * 2 + 1 < player->data.size()) {
audioData[i * 2] += player->data.at((offset + i) * 2);
audioData[i * 2 + 1] += player->data.at((offset + i) * 2 + 1);
} else {
//PCM data reached end
break;
}
} else {
//When audio file is mono...
if(offset + i < player->data.size()) {
audioData[i * 2] += player->data.at(offset + i);
audioData[i * 2 + 1] += player->data.at(offset + i);
} else {
//PCM data reached end
break;
}
}
//Prevent overflow
if(audioData[i * 2] > 1.0)
audioData[i * 2] = 1.0;
else if(audioData[i * 2] < -1.0)
audioData[i * 2] = -1.0;
if(audioData[i * 2 + 1] > 1.0)
audioData[i * 2 + 1] = 1.0;
else if(audioData[i * 2 + 1] < -1.0)
audioData[i * 2 + 1] = -1.0;
}
//Add numFrames to offset, so it can continue playing PCM data in next session
offset += numFrames;
if(offset >= player->data.size()) {
offset = 0;
queueEnded = true;
}
}
I excluded calculation of playback manipulating to simplify code. As you can see, I have to manually pass PCM data to audioData float array. I'm adding PCM data to perform mixing multiple sounds including same sound too.
How to perform panning effect with this PCM array? It will be good if we can follow mechanisms of SoundPool, but it will be fine as long as I can perform panning effect properly. (EX: pan value can be just -1.0 to 1.0, 0 will mean centered)
When applying Panning Rule, what is relationship between PCM and decibel? I know how to make sound louder, but I don't know how to make sound louder with exact decibel. Are there any formula for this?
Pan rules or pan laws are implemented a bit different from manufacturer to manufacturer.
One implementation that is frequently used is that when sounds are panned fully to one side, that side is played at full volume, where as the other side is attenuated fully. if the sound is played at center, both sides are attenuated by roughly 3 decibels.
to do this you can multiply the sound source by the calculated amplitude. e.g. (untested pseudo code)
player->data.at((offset + i) * 2) * 1.0; // left signal at full volume
player->data.at((offset + i) * 2 + 1) * 0.0; // right signal fully attenuated
To get the desired amplitudes you can use the sin function for the left channel and the cos function for the right channel.
notice that when the input to sin and cos is pi/4, that the amplitude is 0.707 on both sides. This will give you your attenuation on both sides of around 3 decibels.
So all that is left to do is to map the range [-1, 1] to the range [0, pi/2]
e.g. assuming you have a value for pan which is in the range [-1, 1]. (untested pseudo code)
pan_mapped = ((pan + 1) / 2.0) * (Math.pi / 2.0);
left_amplitude = sin(pan_mapped);
right_amplitude = cos(pan_mapped);
UPDATE:
Another option frequently used (e.g. ProTools DAW) is to have a pan setting on each side. effectively treating the stereo source as 2 mono sources. This allows you to place the left source freely in the stereo field without affecting the right source.
To do this you would: (untested pseudo code)
left_output += left_source(i) * sin(left_pan)
right_output += left_source(i) * cos(left_pan)
left_output += right_source(i) * sin(right_pan)
right_output += right_source(i) * cos(right_pan)
The setting of these 2 pans are are up to the operator and depend on the recording and desired effect.
How you want to map this to a single pan control is up to you. I would just advise that when the pan is 0 (centred) that the left channel is played only on the left side and the right channel is only played on the right side. Else you would interfere with the original stereo recording.
One possibility would be that the segment [-1, 0) controls the right pan, leaving the left side untouched. and vice versa for [0, 1].
hPi = math.pi / 2.0
def stereoPan(x):
if (x < 0.0):
print("left source:")
print(1.0) # amplitude to left channel
print(0.0) # amplitude to right channel
print("right source:")
print(math.sin(abs(x) * hPi)) # amplitude to left channel
print(math.cos(abs(x) * hPi)) # amplitude to right channel
else:
print("left source:")
print(math.cos(x * hPi)) # amplitude to left channel
print(math.sin(x * hPi)) # amplitude to right channel
print("right source:")
print(0.0) # amplitude to left channel
print(1.0) # amplitude to right channel
The following is not meant to contradict anything in the excellent answer given by #ruff09. I'm just going to add some thoughts and theory that I think is relevant when trying to emulate panning.
I'd like to point out that simply using volume differences has a couple drawbacks. First off, it doesn't match the real world phenomenon. Imagine you are walking down a sidewalk and immediately there on the street, on your right, is a worker with a jackhammer. We could make the sound 100% volume on the right and 0% on the left. But in reality much of what we hear from that source is also coming in the left ear, drowning out other sounds.
If you omit left-ear volume for the jackhammer to obtain maximum right-pan, then even quiet sounds on the left will be audible (which is absurd), since they will not be competing with jackhammer content on that left track. If you do have left-ear volume for the jackhammer, then the volume-based panning effect will swing the location more towards the center. Dilemma!
How do our ears differentiate locations in such situations? I know of two processes that potentially can be incorporated to the panning algorithm to make the panning more "natural." One is a filtering component. High frequencies that match wavelengths that are smaller than the width of our head get attenuated. So, you could add some differential low-pass filtering to your sounds. Another aspect is that in our scenario, the jackhammer sounds reach the right ear a few milliseconds before they reach the left. Thus, you could also add a bit of delay to based on the panning angle. The time-based panning effect works most clearly with frequency content that has wave lengths that are larger than our heads (e.g., some high-pass filtering would also be a component).
There has also been a great deal of work on how the shapes of our ears have differential filtering effects on sounds. I think that we learn to use this as we grow up by subconsciously associating different timbres with different locations (especially pertains to altitude and front vs. back stereo issues).
There are big computation costs, though. So simplifications such as sticking with purely amplitude-based panning is the norm. Thus, for sounds in a 3D world, it is probably best to prefer mono source content for items that need dynamic location changes, and only use stereo content for background music or ambient content that doesn't need dynamic panning based on player location.
I want to do some more experimenting with dynamic time-based panning combined with a bit of amplitude, to see if this can be used effectively with stereo cues. Implementing a dynamic delay is a little tricky, but not as costly as filtering. I'm wondering if there might be ways to record a sound source (preprocess it) to make it more amenable to incorporating real-time filter- and time-based manipulation that result in effective panning.
I'm working on kinect v1 depth images.
How do I find the distance in meters for the corresponding depth intensity value of each pixel?
The intensity value ranges from 0-255 since it is a grayscale image and i don't have the raw depth data.
I've tried various ways to get the distance, such as using the following formulas:
- 1.0 / (raw_depth * -0.0030711016 + 3.3309495161)
- 0.1236 * tan(rawDisparity / 2842.5 + 1.1863)
I've also tried to get the raw data by using:
raw = (255 - depthIntensity)/256*2047
How do i solve this problem?
The Kinect actually sends something akin to a disparity image over USB. Both OpenNI and libfreenect are capable of converting that to a depth image using parameters reported by the device (baseline, focal length, and distance to reference plane, IIRC). e.g CV_CAP_PROP_OPENNI_BASELINE
In math form below, it is how we find the depth based on disparity
Depth = Baseline * focal length / disparity. The depth should be corresponding to the Z axis of current image frame.
I've got a blob with the shape n * t * h * w (Batch number, features, height, width). Within a Caffe layer I want to do an L1 normalization along the t axis, i.e. for fixed n, h and w the sum of values along t should be equal to 1. Normally this would be no big deal, but since it's within a Caffe layer it should happen very quickly, preferably through the Caffe math functions (based on BLAS). Is there a way to achieve this in an efficient manner?
I unfortunately can't change the order of the shape parameters due to later processing, but I can remove the batch number (have a vector of blobs with just t * h * w) or I could convert the blob to an OpenCV Mat, if it makes things easier.
Edit 1: I'm starting to suspect I might be able to solve my task with the help of caffe_gpu_gemm, where I'd multiply a vector of ones of length t with a blob from one batch of shape t * h * w, which should theoretically give me the sums along the feature axis. I'll update if I figure out the next step.
I have geometries in a Postgis database with GeoDjango's default SRID, WGS84, and have found lookups directly in degrees to be much faster than in kilometres, because the database can skip the projections I would think.
Basically, Place.objects.filter(location__distance__lte=(point, D(km=10))) is several orders of magnitude slower than Place.objects.filter(location__dwithin=(point, 10)) as the first query produces a full scan of the table. But sometimes I need to lookup places with a distance threshold in kilometres.
Is there a somewhat precise way to convert the 10 km to degrees for the query?
Maybe another equivalent lookup with the same performance that I should be using instead?
You have several approaches to deal with your problem, here are two of them:
If you do not care much about precision you could use dwithin and use a naive meter to degree conversion degree(x meters) -> x / 40000000 * 360. You would have nearly exact results nearby the equator, but as you go north or south your distance would get shrink (shit we are living on a sphere). Imagine a region that is a circle in the beginning and shrinks to a infinite narrow elipse approaching one of the poles.
If you care about precision you can use:
max_distance = 10000 # distance in meter
buffer_width = max_distance / 40000000. * 360. / math.cos(point.y / 360. * math.pi)
buffered_point = point.buffer(buffer_width)
Place.objects.filter(
location__distance__lte=(point, D(m=max_distance)),
location__overlaps=buffered_point
)
The basic idea is to query for a all points that are within a circle around your point in degree. This part is very performant as the circle is in degreee and the geo index can be used. But the circle is sometimes a bit too big, so we left the filter in meters to filter out places that may be a bit farer away than the allowed max_distance.
A small update to the answer of frankV.
max_distance = 10000 # distance in meter
buffer_width = max_distance / 40000000. * 360. / math.cos(point.y / 360. * math.pi)
buffered_point = point.buffer(buffer_width)
Place.objects.filter(
location__distance__lte=(point, D(m=max_distance)),
location__intersects=buffered_point
)
I found the __overlaps doesn't work with postgresql and a point, but __intersects does.
To be sure it helps your query to speed-up, check the explain plan of the query (queryset.query to get the used query.)
I have started coding for Google Glass. I am developing a sample application which will detect whether a user is nodding or not. More preciously, I want to detect whether it is nod "Yes" or nod "NO" so that I can perform some action.
What's the best way of detecting it since ORIENTATION_SENSOR is deprecated?
I would suggest taking a look at the Level or Compass samples that make use of the sensors.
The Level sample uses the GRAVITY_SENSOR to compute the head orientation on the desired axis. The logic resides in the LevelRenderer class:
/**
* Compute the orientation angle.
*
* #param event Gravity values.
*/
private void computeOrientation(SensorEvent event) {
float angle = (float) -Math.atan(event.values[0]
/ Math.sqrt(event.values[1] * event.values[1] + event.values[2] * event.values[2]));
mLevelView.setAngle(angle);
}
Simply change the vectors used in the angle computation to change axis.