Software gives human touch to computer-generated music

Software gives human touch to computer-generated music
CAMBRIDGE, England — Performing music, like playing chess and creating art, was once solely the domain of humans. But new software from Sibelius Software Ltd. demonstrates that computers, which have succeeded in creating art and beating chess champions, are in hot pursuit of duplicating human musical virtuosity as well.

"The newest version of Sibelius . . . sounds so much like human performers that some TV producers are using it instead of hiring musicians," said Ben Finn, cofounder and managing director at Sibelius, based here. The emotional rendering of a piece of written music has long separated live performances from computer-generated music. Most music sequencers play back musical scores perfectly as written, which makes them sound mechanical. Even as simple a musical task as a drum roll is beyond the capabilities of conventional sequencers, which render a sound similar to a machine gun when given the score for a drum roll.

Human performances always contain slight variations from lockstep rhythm. A detailed analysis of a drum roll, for instance, reveals slightly different emphases in both amplitude and timing that look "off beat," but which respond to higher-order patterns. Attempts to add randomness help only slightly. Sibelius takes a different tack and interprets those subtle higher-order patterns, such as emphasizing the first beat of each measure in the drum roll, by playing it both louder and slightly ahead of the beat.

"Sibelius does add some randomness into a performance, but most of the expression it adds comes from its knowledge base of music rules — in fact originally, we called it the Sibelius Expert System," said Finn.

The Sibelius Expert System was born circa 1987 from machine code for the now-defunct British Acorn computer, and was then subsequently rewritten for Windows and Macintosh platforms. Ben Finn and his twin Jonathan crafted the code to include all the intricate knowledge that a musical "engraver" uses to hand-fit musical scores to the house rules of major publishers.

If a computer blindly puts down the musical notes where they should be mathematically located on the staff, it is impossible for humans to read it because the notational flags will be on top of one another. On the other hand, moving the notes so that their flags don't overlap will mislocate the accompanying notations, governing, for instance, how loud the notes should be played. Subsequent adjustments to those notations will then cause the notes to once again overlap.

Instead of waiting weeks for a human engraver to solve those problems in a written score, Sibelius performs the task of an expert engraver in real-time as the score is input and edited. Notation never obscures a Sibelius score, regardless of the order in which it is transcribed, because the software responds to all combinations of notational conventions, very much as would an expert musical engraver.

"We have a very methodical set of rules governing how music is made to appear well on a score, but the most intelligent part of our program is the way it plays music like a human performer," said Finn.

To give a convincing human performance, Sibelius' internal expert system has rules that control increases and decreases in loudness (crescendos and diminuendos), phrasing, note groupings and 3-D spatial imaging. Each instrument in a score is processed separately, and its performance is independently modified to mirror the "humanness" of each unique performer.

"We do independent phrasing for each instrument, just like real performers would do, and we have a very elaborate algorithm to do convincing crescendos and diminuendos," said Finn.

3-D sonic image added

After the software separately shapes the timing and amplitude envelopes of each instrument, a second level of processing creates a 3-D sonic image of the instruments as they would be heard in a live performance. For instance, if the score is for an orchestra, then the violins are front left in the 3-D image, with very little reverberation due to their distance from the back wall. On the other hand, the timpani drums, located on the back wall, reverberate strongly due to their distant location.

"We place each instrument in 3-D space by using more reverb for faraway positions," said Finn. The phrasing for each instrument — that is, the way in which the notes are grouped together — takes the most computing horsepower in Sibelius' performing capabilities. The only visible notations indicating phrasing patterns are the bars that sometimes connect adjacent notes. Thus, four notes in a row might or might not have a bar connecting their flags. If so, then they are phrased together and played in lockstep. If not, they are phrased together with other adjacent notes.

"We found that good phrasing requires an intricate understanding of the notes that have just been played, but it doesn't require any looking ahead to the next notes to be played. Luckily, the ear 'hears' good phrasing more loudly and just skips over nearby lackluster or even inappropriate phrasing — the ear is very forgiving," said Finn.

If all else fails in rendering a convincing performance, Sibelius will "record" a live human performance and notate it appropriately. Called "flex time" in the program, it enables a human performer to provide a live example of how a notated phrase should be played. The flex-time recorder locks onto the performance by adjusting its clock to speed up and slow down with the speed of the performer.

"Flex time . . . allows you to preset the types of patterns contained in the performance, so that it discounts unlikely rhythmic patterns and instead interprets them as expressive elements of the music," said Finn.

So far, users can only access the rules Sibelius uses to format scores. They can bring up a dialog box that allows them to customize the presentation of almost every aspect of a score. But for now, the rules governing Sibelius' performance remain hidden. That will change in the future, according to Finn.

"Sibelius interprets every notational mark on a score — how it formats that notation can be fine-tuned by directly accessing the rules it uses internally. For now, you can only access score formatting rules, but future versions will allow you to edit the musical performance rules too," said Finn.

Eventually, the Finns plan to make Sibelius intelligent enough to mimic the performances of famous musicians. For instance, imagine the Keith Jarret performance of a Bach organ piece, or the Hendrix performance of the "Star Spangled Banner." By creating sets of rules that effectively "clone" a musician's style, Sibelius may revolutionize the future of professional music production.