Download Project Proposal Presentation

DJ Spatial Tracking and Gesture
Recognition for Audio Effects and Mixing
Project Proposal
Andrew Hamblin, Evan Leong, and Theo Wiersema
Dr. José Sanchez
Bradley University ECE
October 6, 2015
Problem Statement
• Disconnect for disc jockeys (DJ)
• Complexity of DJ equipment
• Lack of natural connection
Fig. 1. DJ cats [1]
Fig. 2. DJ board
2
Past Solutions
• Past gesture recognition systems
• Voltage sensors [2]
• Light refraction along fingers [2]
• Image recognition of light-emitting diodes (LED) on
gloves [3]
• Infrared sensors [4]
• Algorithms
• Hidden Markov model (HMM) [5]
• Dynamic time warping (DTW) [6]
3
Goals
• Glove with tri-color LEDs
• Acquire and recognize gesture
• Seamless communication
• Real-time dynamic effects
4
Fig. 3. System Block Diagram
5
Constraints
• Tri-color LEDs
• Real-time execution
• Max weight of glove less than 0.45 kg (1 lb)
• One camera
6
Scope
• In scope
• Predefined gestures
• 2D image acquisition
• Out of scope
• User-defined gestures
• 3D image acquisition
7
Fig. 4. Four Step Flowchart
8
Proposed Solution
• LED glove
• Pixy camera
• Raspberry Pi
• HMM algorithm
• Mixxx software
Fig. 5. System diagram visual (glove)
9
Glove
• Black for contrast
• Aesthetically pleasing
• Sewing-friendly fabric
Fig. 6. Glove [7]
10
Tri-colored LEDs
• 4 leads (RGB and ground)
• Duty cycles determine color
• LEDs on glove
Fig. 7. Tri-color LED [8]
Fig. 8. Tri-colored LED [9]
11
Adafruit Trinket Pro
• Small form factor
• 3 pulse-width modulated
(PWM) outputs
Fig. 9. Trinket size comparison [10]
12
Proposed Solution
• LED glove
• Pixy camera
• Raspberry Pi
• HMM algorithm
• Mixxx software
Fig. 10. System diagram visual (Pixy)
13
Pixy Camera and Processing System
• On-board image processing
• Color-based object detection
•
•
Object (x,y) coordinates
Object size
Fig. 11. Pixy camera [11]
Fig. 12. Pixy in action
14
Proposed Solution
• LED glove
• Pixy camera
• Raspberry Pi
• HMM algorithm
• Mixxx software
Fig. 13. System diagram visual (Raspberry Pi) 15
Raspberry Pi
• Runs gesture recognition
algorithm
• Cost-effective
• Small form factor
• Existing application
programming interface
(API) with Pixy
Fig. 14. Raspberry Pi [12]
16
Proposed Solution
• LED glove
• Pixy camera
• Raspberry Pi
• HMM algorithm
• Mixxx software
Fig. 15. System diagram visual (Raspberry Pi) 17
Introduction to HMM
Dwight
Fig. 16. Gumball example
18
Introduction to HMM Cont.
Fig. 17. Gumball machine emission probabilities
19
Introduction to HMM Cont.
Fig. 18. Gumball state diagram
20
Introduction to HMM Cont.
Fig. 19. Gumball example
21
Introduction to HMM Cont.
Fig. 20. Possible outcomes [13][14][15]
22
How do Gumballs Relate to HMM?
• Gumballs → observations
• Gumballs on conveyor belt → observation
sequence
• Gumball machines → states
• Succession of machines dropping gumballs →
sequence of states
• Food → result of observation
23
How HMM Relates to Gesture Recognition
• Observations → angles
• Sequence of observations → trajectory of glove
• States → hidden, abstract representation of
angles
• Sequence of states → abstract representation of
gesture
• Result of observation → audio effect applied
24
What is the HMM?
• States transition with time
• Goal is to estimate state sequence
• States are always hidden
• Correlate observations with state sequence
25
Hidden Markov Model (HMM)
Consists of 3 matrices:
• Transition matrix, A: state transition probabilities
• Emission matrix, B: output probabilities
• Initial condition, 𝜋: initial state distribution
These are trained beforehand.
26
Hidden Markov Model (HMM)
x - states
y - observations
a - state transition
probabilities
b - output
probabilities
Fig. 21. HMM example (states)
27
Hidden Markov Model (HMM)
x - states
y - observations
a - state transition
probabilities
b - output
probabilities
Fig. 22. HMM example (observations)
28
Hidden Markov Model (HMM)
x - states
y - observations
a - state transition
probabilities
b - output
probabilities
Fig. 23. HMM example (state transition probabilities)
29
Hidden Markov Model (HMM)
x - states
y - observations
a - state transition
probabilities
b - output
probabilities
Fig. 24. HMM example (output probabilities)
30
Angle Quantization
• Divided among “bins” [16]
• Angles are rounded to
the nearest bin
Fig. 25. Quantized angle bins
31
Training with Test Gestures
Fig. 26. Test gestures
32
Angles of Test Gestures
Fig. 27. Gesture angles
33
Quantized Gesture Angles
Fig. 28. Quantized gesture angles
34
3 Problems of HMMs
1. Classifying - probability of observing sequence of
observations
1. Decoding - what is best sequence of states that
explains observed sequence of observations
1. Training - how to learn parameters from
observations
35
3 Problems of HMM Cont.
• Classifying
• Forward algorithm [17]
• Backward algorithm [17]
• Decoding
• Viterbi algorithm [18]
• Training
• Expectation/maximization algorithm [19]
• Threshold model [19]
36
Threshold Model
Fig. 29. Threshold model example
37
Proposed Solution
• LED glove
• Pixy camera
• Raspberry Pi
• HMM algorithm
• Mixxx software
Fig. 30. System diagram visual (PC/Mac & speakers)
38
Mixxx DJ Software
• Open source
• User-friendly interface
• Variety of effects
• Plug-in capability
•
Javascript
Fig. 31. Mixxx logo [20]
39
Communication Protocols
• Pixy to Raspberry Pi: serial peripheral interface
(SPI) communication
• Raspberry Pi to PC/Mac: universal serial bus
(USB) to transistor-transistor logic (TTL) serial
cable
40
Specifications
• Display predefined color schemes through tricolor LEDs
• Acquire user’s gestures
• Recognize user’s gestures
• Trigger sound effects specified by gesture and
LED color combination
41
Specifications cont.
• Predefined color schemes
• Tri-color LEDs (red, green, blue)
• Glove subsystem used to meet specifications
42
Specifications cont.
• Acquire user’s gestures
•
•
•
•
10 frames per second
400 x 240 pixel resolution
85% success rate
Ambient light > 250 candela
• Camera subsystem used to meet specifications
43
Specifications cont.
• Recognize user’s gestures
•
•
75% success rate
160 ms latency
• HMM algorithm used to meet specifications
44
Specifications cont.
• Trigger sound effects
• Effects correspond to gesture and LED color
combination
• Effects mapped with musical instrument device
interface (MIDI) signals
• DJ software used to meet specifications
45
Testing the LED Glove
1. Oscilloscope PWM analysis
1.
2.
3.
4.
Send PWM
Measure frequency
Measure voltage level
Match emitted color with PWM signal
1. Power consumption
1. Send PWM from Trinket
2. Measure current to collector of transistor
3. Measure entire current draw of circuit
4. Calculate battery life
5. Measure current through each diode
6. Confirm current through each diode is < 30 mA
46
Alternative Solutions: Hardware
• TMS320C6657 digital signal processor (DSP)
• Video graphics array (VGA) camera
• BeagleBone Black linux computer
Fig. 32. TMS320C6657 DSP [21]
Fig. 33. VGA camera [22]
Fig. 34. BeagleBone Black [23]
47
Alternative Solution: Software
• Dynamic time warping
• Custom image processing
• Custom audio effects
48
Division of Labor
Andrew Hamblin
Evan Leong
Theo Wiersema
Mixxx plugin
Glove circuitry
HMM simulation
Pixy testing
Pixy-Raspberry Pi
communication
Raspberry
Pi-computer
communication
Mixxx testing
Glove testing
HMM testing
Fig. 35. High-level division of labor
49
Cost of Materials
Primary solution
Pixy
Raspberry Pi
Glove
Trinket Pro
Tri-color LEDs
Miscellaneous
Mixxx software
$69.00
$35.00
$17.98
$9.95
$14.90
$40.00
$0
Total
$171.93
50
Fig. 36. Deadlines
51
Summary
• LED glove
• Pixy camera
• Raspberry Pi
• HMM algorithm
• Mixxx software
Fig. 37. System diagram visual
52
DJ Spatial Tracking and Gesture
Recognition for Audio Effects and Mixing
Project Proposal
Andrew Hamblin, Evan Leong, and Theo Wiersema
Dr. José Sanchez
Bradley University ECE
October 6, 2015
Fig. 38. Gantt Chart
54
Fig. 39. Top-Level State Diagram
55
Fig. 40. Glove State Diagram
56
Pixy Considerations
• Detecting small LED size
• Detecting LED color
Fig. 41. Pixy color detection
57
LED Considerations
• Parallel LED burnout
• 5 active diodes
• Power consumption
• Trinket
• LEDs
• Transistors
Fig. 42. Tri-color LED [8]
58
Testing the Pixy
1. LED color signature detection
1.
2.
3.
4.
5.
Emit one color from LED
Use Pixymon to train signature
Assess detection of color signature
Vary sensitivities and camera brightness
Reassess detection of color signature
1. Track LED trajectory throughout gesture
59
Testing the Raspberry Pi
1. Validate trajectory calculation
1. Obtain (x,y) coordinates from one frame to the next
2. Perform angle calculation
3. Verify the Raspberry Pi output
1. Evaluate computation time
1. Set flag “high” when entering trajectory calculation
process
2. Set flag “low” when exiting
3. Use oscilloscope to measure length of pulse
1. Analyze MIDI signal output
60
Testing the HMM
1. Trajectory observations
1. Baum-Welch algorithm [17]
1. Update transition and emission matrices
1. Viterbi algorithm [18]
1. Optimal state sequence
1. Candidate end/start point calculation
1. Threshold analysis
61
Testing the DJ Software
1. Effect execution
1. Observe MIDI signal input
2. Validate mapping
3. Observe audio effect applied
1. Processing time
1. Set flag “high” when entering effect execution process
2. Set flag “low” when exiting
3. Use oscilloscope to measure length of pulse
62
Cost of Materials
Alternative solution
TMS320C6657 DSP
BeagleBone Black
VGA camera
Glove
Trinket Pro
Tri-color LEDs
Miscellaneous
Mixxx software
$399.00
$35.00
$63.00
$17.98
$9.95
$14.90
$40.00
$0
Total
$579.83
63
References
[1] http://www.autostraddle.com/its-here-international-autostraddle-meet-up-week-starts-now-201261/
[2] http://www.rroij.com/open-access/hand-gesture-recognition--analysis-ofvarious-techniques-methods-and-theiralgorithms.pdf
[3] http://research.microsoft.com/en-us/um/people/bibuxton/buxtoncollection/a/pdf/cg%20a%20glove%20survey.pdf
[4] https://apps.leapmotion.com/apps/geco-midi/osx
[5] http://www.cs.sjsu.edu/~stamp/RUA/HMM.pdf
[6] http://www.psb.ugent.be/cbd/papers/gentxwarper/DTWalgorithm.htm
[7] http://sixsixone.com/recon-glove-black
[8] https://upload.wikimedia.org/wikipedia/commons/f/f1/RGB_LED.jpg
[9] https://arduino-info.wikispaces.com/file/view/RGB-animation.gif/264579766/RGB-animation.gif
[10] https://www.adafruit.com/products/2010
[11] http://charmedlabs.com/default/
[12] https://www.raspberrypi.org/
[13] http://www.clipartbest.com/hamburger-black-and-white-clipart
[14] http://www.clipartpanda.com/clipart_images/pizza-coloring-pages-1-33796411
[15] http://www.clipartbest.com/clipart-RTA6qk8nc
[16] http://www.iis.ee.ic.ac.uk/ComputerVision/docs/pubs/John_Report_2014.pdf
[17] https://en.wikipedia.org/wiki/Baum%E2%80%93Welch_algorithm#Algorithm
[18] https://en.wikipedia.org/wiki/Viterbi_algorithm#Algorithm
[19] http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=799904&tag=1
[20] http://mixxx.org
[21] http://www.element14.com/community/docs/DOC-67655/l/tms320c6657-lite-evaluation-module
[22] http://www.robotshop.com/en/vc0706-uart-vga-camera.html
[23] http://beagleboard.org/black
64
System Diagram Visual
Fig. 43. “Crude” visual
65