I have been wanting to build a game that builds worlds/enemies/properties off a given music file for some time. There are some decent games out there that are making such strides, one of which is Tube Rockers. I like that game because it involves people making custom tracks to music videos found on You Tube. Pretty cool.
However, with Flash 10 we have the ability, with the users permission, to read and write to their hard drive. Finally. I know I can say this for every Flash/Flex developer out there that we have been waiting for this since… well… Flash was created really. And now without round tripping to the server! Yay!
Analyzing sound in Flash has been around since SoundMixer.computeSpectrum() was introduced in Flash 8. Using this, along with the read capabilities of Flash 10 we can bring in any mp3 a user would like to play in a game. This is where the idea stems from. Try Catch Games (www.trycatchgames.com) my casual game company is currently concepting such a music game, more on that to come.
Now to talk about sound. Here are some stats:
Human Hearing: 20 to 20,000 Hz
Bass: 20 Hz - 250 Hz Mids: 250 Hz - 6 kHz Highs: 5 kHz - 20 kHz
Bass Drum: 50 Hz - 5.5 kHz Thump: 50 Hz - 100 Hz Punch/Slap: 2.5 kHz - 5 kHz
Cymbals: 300 Hz - 17 kHz Presence: 10 kHz - 14 kHz
Snare: 100 Hz - 12 kHz Center: 1 kHz Tight: 5 kHz - 6 kHz Crack: 8 kHz - 10 kHz
Toms: 4 kHz - 5.5 kHz and 9 kHz - 10 kHz
***
I am going to assume that readers of this blog know a thing or two about sound. If not, check out a wiki and come back.
A lot of you are probably asking “What does this guy know about Flash that I don’t to analyze sound?”. Really the answer is, nothing. I just want to give out some information that I have found out to further your journey into building better Flash games or apps. Let’s jump in.
Checking out the computeSpectrum method in the SoundMixer object you will find a few params:
computeSpectrum(outputArray:ByteArray, FFTMode:Boolean = false, stretchFactor:int = 0)
The params I am going to focus on are the last two and what to do with that data. First, for the data that I want I set FFTMode to true because I am interested in the frequency spectrum instead of the raw sound wave. With this frequency data I want to filter out (as best as possible) values that tell me when something is happening. This “something” would be a beat, vocals or guitar. Now I have to say from the onset that my filtering isn’t really that great…yet. But I believe with the capabilities of extracting raw sound data from an MP3 and computeSpectrum (or one or the other) will help me accomplish my goal more accurately. For now, I am analyzing on-the-fly and averaging values based on what I think a beat is.
The next param is the stretchFactor. I believe this to be important to be tweaked as we can drill down to the frequencies that are interesting to us. In my experiment I am using the values of 3 and 5, which give us a stretch to the frequencies of 5512 and 1378 respectively. Now I obviously use the 1378 frequency stretch for testing for sound data in the lower spectrum (which covers a lot of sound actually) which is mostly beats, bass guitar and some voice depending on the music. The higher one, 5512, would be used for higher frequencies like violin and female voice.
For some simple formulas in finding these frequency ranges I have hard coded x and y point values. This can be found by the frequency max limit, which for the lower one would be 1378, then divide that by 512, as this is the number of channels. You will find that each “slice” will be about 2.69. This means that within each of these slices it will cover a particular frequency range. In the case of covering 0Hz - 100Hz you would divide 100 by 2.69 to find how many slices it occupies. In my experiment I would sometimes give or take a few slices in some cases.
I know none of this is ground breaking, but I did my fair share of searching and no one has talked about it. My experiment that I did off of this research is a simple on-the-fly averaging music program that does a little filtering and displays a square when the grouped slices go over the average. Currently, I am not going to share the source code as its a work in progress but if you would like to check it out contact me directly: nate |at| pacyga |dot| com
If you would like to check out my experiment do so here.