Data transfer via Audio channel-Coding

Interface:
The interface was very simple just a bunch of labels and round buttons.
There is one thing I want to point out. The label(in my program) was updated by the model directly, I guess it is not a right way of doing it(according to stanford CS193P tutorial online). Some kind of notification should be used, i guess. But I don't have time to look into the threads and add in notifications, so I just let the model manipulated that directly.The good side is, the frequency of the operation is not so fast(about 100 times/sec), so I thought it's okay. It's running fine, on simulator.


Controller
There is not really a controller, just have to link up the functions to the right button.
Very basic.


Model
As compared to the two categories above, model is a bit more complex, it involves 2 main portions:

• Getting audio Data
• I am using the AudioQueueService
• In order to use a AudioQueue we need to:
• Set up a Audio Session and change the configuration of the audio session to the correct setting(in my case, i used recording, in which input is enabled)
• Implement the call back function that the audioQueue would calls on when a buffer is filled
• This is the most critical part
• We get and process audio data via this audio queue call back function
• Create the audioQueue,initialize, get it proper setting, link it up with the call-back function, allocate the queue some buffer and it's good to go

• Apple AudioQueueProgramingGuide(CookBook) and AudioSessionProgramingGuide is quite good. 
• The sample code SpeakHere, by apple does almost the same thing for the audioQueue part. can make use of that with minor modification.
• Data processing
• a data queue was created. The incoming frequency and it's duration is stored into the queue.
• The algorithm would combine the frequencies within a threshold range.
• Some noise rejection code was added in as well, so that stand-alone noise frequency will not appear in the final result.

Potential Problems:

now, how we compute frequency is by counting the number of times that the lPCM data crosses 0. and divided by the length of the PCM data. This might arise a few problems:

1. Now, the logic is based on comparison operation. I am not sure, whether this operation is optimized for iPhone platform. If not, it might be slow and might not keep up with sampling rate.
• So far, on simulator, it's working properly
2. Noise might have significant effect on the computed frequency, especially when the samples are of low frequency
• Yes, this method does not work for low frequency. Get to use FFT I guess.
• But the cause might not be cause, I guess. It's the resolution! when the frequency resolution gets down to 500HZ, it's of no points saying the frequency is 1kHZ.
3. only one fundamental frequency can be extracted(all the harmonics/weaker frequency would be lost)

Therefore, it would be good if we can replace this method by FFT, which would also come down to the availability of computing power on the iPhone device.

Coding Process:

Notification need to be sent from model to view controller.

So Far Achieved(Tested):

Creating a UI-a simple UI to achieve the purpose of testing

Create an Audio Queue-the queue is set up and initialized, need to look into whether or not do we need to get the proper setting for the audio session object as well.

Compute the frequency-from the data obtained from audio queue

To be implemented:

<Done>Work out the call back function, to decode the signal(Compute the frequency of a buffer)

Further analyze the data, figure out the transition states and eliminate the effect of that.

Achieve speed/accuracy

Post a notification to UI to inform the UI about the updates of the data(*Optional, Can be replaced with a button that manually updates the contents of the updated data)

Data transfer via Audio channel-Testing

I am using a Arduino and a buzzer to test the accuracy of the data transfer quality.

It seems, if the phone can handle the amount of computation, smaller buffer size would be a better choice. 

A smaller buffer size means more buffers generated per second and less samples per buffer. These would result in a lower frequency resolution. Take 0.001 buffer duration, for example, the frequency resolution becomes 1/0.001=1KHz. That means the smallest difference that can be told from a buffer would be 1KHz(if using this 0-crossing counting algorithm).

This is good. the parts where the tone is changing frequency is more clear-cut. And the latency is lower(as we would need to wait for shorter time before a buffer is filled). Of course, the price we pay is the resolution becomes not so good. we can't really tell the difference between 5KHz and 5.5kHZ, if we use a buffer duration of 0.001sec.

Testing Procedure:

I am using the Arduino and the following code to generate the frequency that is fed into a buffer and a low pass filter.

void setup() {              

  // initialize the digital pin as an output.

  // Pin 13 has an LED connected on most Arduino boards:

  pinMode(9, OUTPUT);  

}

void loop() {

  for(int i=1000;i<9000;i+=500)

  {

  tone(9,i);   // set the LED off

  delay(20);

  tone(9,10000);   // set the LED off

  delay(50);

  }

}

We would expect a increasing tone alternated by a fixed pitch at 10KHz.

And this is the graph I obtained by getting the computed frequency for each buffer from NSLog, than plot it using EXCEL:

Buffer Duration of 0.005 sec:

Buffer Duration of 0.002 sec:

Buffer Duration of 0.001 sec:

Data transfer via Audio channel-miscellaneous

Possible ways the application could be coded:

• I/o Unit Directly
• AVRecorder
• OpenAL
• (it's too complicated, it's like a independent system and I don't have time to look into that.It's more meant for gaming, where you place a moving sound source into a stereo space and take into account of effects like echo and dopler effect and all that.)
• AudioQueueServices

By Making use of AudioQueueCallBack Function(AudioQueueService)
Audio Queu Service might be a better choice if we are not trying to implement bi-directional communication.
The Build in API would take care of the hardware connections automatically. The incoming audio data would stored in buffers, which are queued up. Once a buffer is full, the AudioQueuCallBack function is invoked, to deal with the data inside the full buffer, and clear the buffer for further use.

• Advantage of audioQueueService.
• It's one level up from handling i/o unit directly, there is less coding required.(memory management, signal roughing is all taken care of by API)
• The adjustable buffer size would enables us to find a balance between the latency of  processed signal and overhead of repeated function call.
• Disadvantage of AudioQueueService:
• the latency would make it impossible for us to implement handshaking signal.
• As according to apple documentation, a bufferQueueCallBack is typically invoked every half second to every a few seconds
• even if we shrink the buffer size to reduce the interval between the callBacks, it won't be possible to be invoked like render callback, 

By Making Use of RenderCallBack Function:(i/p Unit Directly)


RenderCallBack function is invoked when a slice of output signal is needed(a pull approach, initiated by output). The input audio data is passed into the renderCallBack function and it would do something with the input and pass it on to the output.
In our case, however, there won't be a audio output, as we are trying to transmit data through audio channel. But we can still make use of the renderCallBack function to have a doSomething function synchronized to input data.

• Core of this idea is that we can configure a dummy output device, with a renderCallBack Function. 
• If the dummy output device and the input device is configured at exactly the same sampling rate. the renderCallBack function would be called synchronized to arrival of every sample we get from the audio input.
• The advantage of this method would be:
• We might have better battery consummations, as the processing is carried out for each of the samples exactly once, and there is no constant monitoring going on in this approach.
• There is possibility for implementing hand-shadking signal for data integrity or two way realtime communication, as the latency for this process is very low. As claimed by apple documentation. 
• The constrains would be:
• The renderCallBack function must be simple enough. Otherwise, we might lose samples.

Notes taking as going along documentations:

• in Multimedia Programing guide
• Audio Session:
• need to set Sample rate, Num of channels and enables inputs
• category, interruption handling and route change is not really applicable
• Playing Audio
• Access iPod library with mediaPicker/ mediaQuery
• Vibration and system sound with systemSoundService
• AVAudioPlayer
• AudioQueueServices
• need to read AudioQueueServiceProgrammingGuide
• OpenAL need to check openAL website
• Recording Audio
• Recorde with AVRecorder
• Recorder with AudioQueueService

Resource would be useful:
Librarys need to be studied:
Audio Unit Component Service
Audio Component Service
Audio Unit Hosting Guide for iOS
Guides:

• AudioProgramingGuide:
• iPhone Recording App:
• Audio Session Programing Guide:

As we are concerning real-time communication. The recorder class won't be able to serve this purpose.

Libraries might be useful:

• Framework: 
• AVFundation
• Classes:
• AVAudioPlayer
• AVAudioRecorder
• AVAudioSessionClass
• AVAudioRecorderDeligate Protocol

For our project:

• AVAudioRecorer and its delicate can be used to perform the recording task
• AVAudioPlayer can be used to playback the recorded track, if needed
• AVAudioSessionClass is needed to get the proper audio related system configurations for the software to run on.

iPhone simulator testing v.s. on-device testing:


In short, we would be able to simulate the software on iPhone simulator, but we won't be able to settle the more 'hardware- related' issue such as when the user plug/unplug a earphone/mic.
Therefore, simulator can be used in the initial phase of the development. And we will have to settle the rest by simulating on a actually device.


I have tried a downloaded project done by someone else on a simulator. The simulator would be able to access the mic on Mac and would be able to replay the recorded sound.


However, there are certain sinarios that cannot be tested on iPhone simulator according to the Developer's Guide:

When running in the simulator, you cannot:

• Invoke an interruption
• Change the setting of the Silent switch
• Simulate screen lock
• Simulate the plugging in or unplugging of a headset
• Query audio route information or test audio session category behavior
• Test audio mixing behavior—that is, playing your audio along with audio from another application (such as the iPod)

Hours at KUTE Centre

• 23/12/11
• 10am-5pm
• Test the app using Arduino driven buzzer
• Improve on the parameters of the application
• 22/12/11
• 1pm-5pm
• Try to improve on the accuracy
• Trying to improve on the interface
• 21/12/11
• 10am-5pm
• Debug......
• Made the program running
• 20/12/11
• 10am-5pm
• Continue coding the program, writing call back functions
• Finished most of the model coding
• link up model and controller, can start testing after that=D
• 19/12/11
• 12.30pm-5pm
• Start coding the program
• Built a simple interface
• adapting the sample code to fit what we need.
• 15/12/11
• 10.30am-4.30pm
• Reading documentation on AudioQueueService.
• Trying to understand sample code relavant to audioQueuService.
• 14/12/11
• 10am-5pm
• Research on possible way that can have the audio data with low latency
• Study documentation regarding AudioQueuService
• Continue with Xcode Tutorial on iTuneU
• Get the calculator project working
• 13/12/11
• 10am-5pm
• Finding out how to get real-time PCM data out from audio input.
• Digging through apple documentation to find out how to interact with audio data on real-time bases
• To-Dos tml:
• Continue with this link
• 12/12/11
• 10.30am-5.30pm
• Studying IOS API and ocjective C
• Trying to find out whether iPhone simulator is able to simulate audio activities(Yes, but partially, see link for more)
• Going through audio related documentation to gain knowledge in handling audio in IOS
• 9/12/11
• 10am-5pm
• Studying Xcode4 and Objective c using Stanford course (fall 11) on iTunes u
• Try to code the calculator example in the demo video
• 7/12/11
• 10am-5pm
• Patent Search -wearable gaming vest 
• Read papers- Tactile Gloves
• Research on how to how to handle audio signal on iOS
• 6/12/11
• 1pm-6pm
• Modify Yvonne's Schematic design
• Background research on Gaming devices using haptic technology
• Tried to figure out how RN41 Bluetooth device can be configured and connected(part of Schematics )
• 5/12/11
• 4pm-6pm
• Getting started on Yvonne's Schematic design

Haptic Background research

7/12/11
Gloves proposed/Built by researchers:

A data-glove with vibro-tactile stimulation for virtual social interaction and rehabilitation

• Serve as a data gloves that measures the angles of the fingers' bending with vibration feed back
• Goniometric sensor
• No a libration needed for different finger length.
• built using chip components like plastic, steel cable, permanent magnet and hall effect sensor.
• translate the bending motion of dinger to a sliding motion of permanent magnet w.r.t Hall Effect sensor
• vibro-tactile Actuator
• Vibration alarm unit in handphone is used for lower cost
• Frequency adjustable to create stimulation to four different tactile unit in human skin

Wearable device that vibrates fingertip could improve one's sense of touch

• Done by Minoru Shinohara from Georgia Tech(http://www.ap.gatech.edu/Shino/index.php)
• a glove with vibrator at fingertip was built
• High frequency vibration at fingertip can improve tactile sensitivity (http://gtresearchnews.gatech.edu/sensory-glove/)

Available commercial applications
peregrine Game Glove

• It's more like a wearable programmable keyboard, able to trigger different key strokes based on the position of the touch
• Used in gaming for more convenient controlling

Essential Reality P5 Gaming gloves

• Able to sense 3D motions of fingers 

Vests
US Patent (US 7,967,679 B2)--Tactile Wearable Gaming Device

• Separate recording and playback unit
• Recording unit to be used when creating the Tactile library
• Playback unit to be worn by client when playing video games
• Gaming vest using arrays of pneumatic bladders
• claimed being able to provide feelings of:
• Impact-by inflate part of the arrays of badders
• Right/Left turn- by inflating the right/left of the vest
• Acceleration/deceleration-by inflating the back/front(I don't think this is possible, the counter force would come in, when you inflate the back, the front would be compressed due to counter force) 

Product by TNGames(tngames.com)

• 3RD Space Gaming Vest(us$139)
• Real world application to the patents above. Actually same bunch of people
• Able to simulate impact of
• knife
• piston
• machine gun
• bomb explosion 
• In action games (bio-shock, call of duty, etc)

06/12/11
Haptic technology
Early applications:

• On planes, to feedback on the pilots once the plane is steered beyond the critical angle, which is dangerous.
• On teleoperators, used to remove radioactive wastes and other dangerous operations.

Gaming application:
product by "Novint"

• Falcon
• a hand grip with force feed-backed by three controlling arms
• arms driven by motors with refresh rate of 1 KHz
• able to keep track of the position up to sub-millimeter resolution
• http://www.novint.com/index.php/novintfalcon
• Xio (coming soon)
• arm motion detector with force feedback
• http://www.novint.com/index.php/novintxio

Other gaming pad/joysticks using haptic technology

• Wii
• Rumble pak
• DualShock
• All vibration based