Synthesia Clone "Piano Hero": Creating the Roll

Summer Project Numero Uno: Synthesia Clone for Android
In the last post, I explained very badly how to convert MidiEvents in Java to timed data which you can use to create notes. Now I am going to show you some code on how to get a roll using the times in your notes. The basic idea is to have a thread that continuously redraws while keeping track of how much time has passed.

public void drawRoll(Graphics2D g, ArrayList notes) {
A:   int key_width = 16;
     int key_height = 100;

     // Scale falling notes so that a note of length (show_duration)
     // will stretch to fit the entire space available
B:   double scale_factor = (double)window_height / show_duration;
     boolean drawing_black_keys = false;

     for(int i = 0; i < 2; i++) {
          for(int j = 0; j < notes.size(); j++) {
               Note note = notes.get(j);
C:             if ( note.end_time < current_time ) { notes.remove(i); continue;}
D:             if ( note.start_time > current_time + show_duration) break;

E:             if(notes.is_black_key != drawing_black_keys) continue;
F:             int x = NoteOffset[note.note_number] * key_width;
G:             int y_start = roll_height - (int)((note.start_time - current_time) * scale_factor);
               int y_end = roll_height - (int)((note.end_time - current_time) * scale_factor);
               int height = y_start - y_end;
               if(height == 0) height = 5;
               // Remember that positive and negative y is backwards
H:             if(note.is_black_key) drawBlackNote(g, x, y_end, key_width, height);
               else {drawWhiteNote(g, x, y_end, key_width, height); }
          }
          drawing_black_keys = true;
     }
}

(A) - You should make other methods to draw black keys and white keys and then change the appearance of the keys inside of those.
(B) - window_height is the size of my jframe, change as desired
(C) - Here, we remove notes that we have already passed
(D) - If we have found a note that isn't in view yet, we are done with this loop
(E) - Here, we skip the black keys on the first loop so that they are drawn on top of the white keys
(F) - NoteOffset is actually an Double[], so you should cast the whole line to an int, NoteOffset contains values for each key which contains the number or white keys plus one half times the number of black keys
(G) - role_height is window_height - keyboard_height
(H) - Again, you should make your own methods for drawing keys however you like

I have found it better to draw the static images before hand by creating a BufferedImage and then drawing your notes to the BufferedImage. This way, you don't need to redraw everything in real time and can simply blit pixels to the screen to scroll. To do this:

BufferedImage r = new BufferedImage(width, height, BufferedImage.TYPE_4BYTE_ABGR);
Graphics2D g1 = r.createGraphics();
drawBufferedImages(g1)

Where drawBufferedImages() draws my keyboard, the background, and the guides. I tried drawing everything before hand to one big BufferedImage only to have Java run out of memory for any song that was longer than one and a half minutes. So I don't recommend doing that.


Then to draw everything, overload paint:

public void paint(Graphics g) {
     g1.drawImage( background,  0, 0, background_width, background_height, 0, 0,
          background_width, background_height, null);     
     drawNotePass(g1, active_list);  
     g1.drawImage( keyboard, 0, background_height, keyboard_width, background_height
          + keyboard_height, 0, 0, keyboard_width, keyboard_height, null);
}

where background and keyboard are my BufferedImage's that have been drawn earlier.

For getting good looking notes and other stuff, use gradients. To make a simple horizontal gradient for a vertical note, set x1 so the left most x-coordinate and x2 the right most x-coordinate. (I recommend not choosing two colors that are extremely different :P )

Color one = new Color(  54, 161, 201); // Random color :D
Color two = new Color( 143,  91, 56 ); // Random color :D
GradientPaint fill = new GradientPaint( x1, 0, one , x2, 0, two );
g.setPaint( fill ); // g is Graphics2D

References: Graphics, BufferedImage, GradientPaint

Synthesia Clone "Piano Hero": Parsing Midi Files

Summer Project Numero Uno: Creating a Synthesia Clone for Android

Background: Synthesia is a piano game and trainer written in C++ that builds a piano roll out of a Midi file. Synthesia was also originally named "Piano Hero" before Activision sent a cease and desist letter telling them to change their name.

Synthesia is extremely helpful for learning new songs quickly (especially if you're slow at reading sheet music like me). However, finding a decent position for a computer near/ontop of your keyboard is very troublesome. And with the recent hype over tablet computers, most of which run android?, getting Synthesia to fit on (the thingy that holds sheet music) is a must.

Midi Files: Midi files are composed of MidiEvents, which generally represents an action such as a Note On, and are organized into tracks, which represent separate streams of MidiEvents. Every event has an associated delta-time stamp, measured in ticks, which determines when it should occur relative to the previous event. In order to convert ticks to seconds, we need to know two more things: the resolution and tempo. The resolution is the number of ticks per quarter note, which I kind of think of as the quality of the midi, and can be found in the file header. The tempo is number of microseconds per quarter note, but most people appear to convert this to beats per minute. The tempo is a little more difficult as it can change during as song. Once we have all of this, converting is some pretty straight forward algebra:

ppqn = 480                   // ticks per quarter note, get from file header
bpm  = 60000000 /tempo;      // quarter notes per minute, get tempo from MidiEvents
mspt = 60000 /( bpm * ppqn ) // milliseconds per tick

Working with ticks in Java is a little different, because Java automatically converts the delta ticks to cumulative ticks. So events having the following ticks 10, 10, 10 respectively would become 0, 10, 20. Now here's some half-pseudo-code for parsing a single track in a Midi file in Java:

int bmp = 120; // default is 120
int tempo = 0;
int ppqn = 480; // get from file header
int last_tick = 0;

double ct = 0; // the cumulative time
double mspt = 60000.0 / ( (double)bpm * (double)ppqn );

for(int i = 0; i < track.size(); i++) {
     MidiEvent event = track.get(i);
     MidiMessage msg = event.getMessage();

     if(msg instanceof ShortMessage)
          switch( ((ShortMessage)msg).getCommand() )
               case NOTE_ON:
                    ct += mstp * (event.getTick() - last_tick);
                    last_tick = event.getTick();
               case NOTE_OFF:
                    ct += mstp * (event.getTick() - last_tick);
                    last_tick = event.getTick();
     else if(msg instanceof MetaMessage)
          switch( ((MetaMessage)msg).getType() )
               case 0x51:
                    ct += mstp * (event.getTick() - last_tick);
                    last_tick = event.getTick();
                    tempo = getIntFromByteArray(msg.getData());
                    bpm = 60000000 / tempo
                    mstp = 60000.0 / ((double)bpm * (double)ppqn);

For the sake of a piano roll, we only need to worry about these three types of messages. Notice that NOTE_ON and NOTE_OFF are two separate events. This means that if you want to create some kind of Note object, you need to either keep an array of half complete notes or look ahead for the next NOTE_OFF event with the same key number.

One last precaution! The first track in Type 1 Midi files contain all of the tempo events for all of the other tracks and is called the tempo map.

There are three types of Midi files:

Type 0: Everything is saved in one track.
Type 1: Multiple tracks with individual parts on separate tracks.
Type 2: Multiple tracks which represent different patterns. (Not commonly found)

So what I did was to go through the first track and find all of the tempo events and create duplicate events in the rest of the tracks.

Drawing Amoebae Part 2

Now that we have points which move in an amoeba-like pattern we need to be able to draw a smooth and continuous curve through these points so that our amoeba's outline looks realistic. To do this we will use the built in quadratic curve primitives provided by java. A quadratic curve takes three points. A beginning point, an end point, and a control point which will cause the curve to be stretched in its direction. It works exactly the same way as the curved line tool in MS Paint.


For the amoeba, these beginning and end points will not be the nodes themselves, but rather the midpoints between each node. Then we will use each node as a control point stretching the line between each midpoint to it's left and right. In the following picture, nodes are the large circles and midpoints are the small circles.

You can see that the cubic curves become continuous when you use midpoints as the beginning and end points. Using this approach we obtained a very good looking amoeba outline.

Drawing Amoebae Part 1

We've just started working on a simple Android game related to "the cell." We only have a vague idea, at this point, of what the general premise of the game is, but one of the graphics requirements is the ability to draw a dynamically moving and undulating amoeba with smooth graphics. So this is how we went about doing it.

To start, the outline of the amoeba consists of 20 or so evenly spaced nodes shaped as a circle. Here we will only show a few for simplicity. To get the locations for x number of nodes around a circle we use basic trigonometry.

increment = 2PI / numPoints;

For each point calculate the angle:

theta = i * increment;

To get the x and y locations from this angle use:

x = radius * cos(theta);

y = radius * sin(theta);

In order to keep the amoeba at a generally-similar size and shape it will be bound by somewhat of an exoskeleton. So at each of the previously calculated locations we create a node representing a piece of the exoskeleton which will not move or be visible (dotted) and an actual node mass which moves and makes up the shape of the amoeba (solid). These nodes are assigned random momenta so they start of moving in random directions.

If we just let the masses move off in their random directions decided in the previous step this wouldn't look like an amoeba at all. We now need to make the the amoeba undulate. We do this by simulating the physics which would happen if there were a spring between each node and a strong spring between each node and its corresponding exoskeleton point. This will keep all of the nodes close together so they are similar to the original circular shape, but allow them to undulate forming the basic outline of a moving amoeba.

Putting all of these steps together we get a very amoeba-like set of ~20 points, which undulates to varying degrees depending on the various constants we set. We are able to modify the spring constants, the masses of the nodes, and the number of nodes in order to experiment with various effects. Here is the result of our first few steps: