There are already methods to create a synthesized voice, with Stephan Hawking as the best example. For Ramsey, this is not a solution, the way the synthesized voice is acquired also relies on muscle movement. Fortunately methods are improving and there are some revolutionary new ways to obtain patterns out of neurons from the brain, this is what Dr. Philip Kennedy did. A very small electrode is placed inside the skull of Erik Ramsey. This electrode is wired to a computer and the patterns of Ramsey are studied for more than two years. After intense training they are able to distinguish vowels, that Ramsey wants to express. The classification of vowels is already at 75%, this is a very good rating for a system like this. Of course there are a lot of things that need to be improved, they are already experimenting with wireless control. It is still very new, but in the near future this can be fine tuned and Erik Ramsey could be able to convert his brain signals into a computer voice!

Explanation about this short movie:  It consists out of two parts, Ramsey listens to the vowel sound that the computer is making (Listen), after that he tries to reacreate this vowel sound, just by thinking about it and the computer translates the brain signal into vowel sounds (Speak). He is not controlling the cursor by thinking about location, he is really thinking about speaking the vowel sounds!

Read more about Erik Ramsey.

Why is this used?
Basically everything that you do on the internet is controlled by sockets. When visiting a website, your PC sends a request to a server to access the page. The server sends something back, for example the website.

Why do we need it for Brains on Fire?
Network sockets can be used for a diverse range of applications, it is needed to communicate between two PCs or programs. We are going to use two systems, one is running the Brains on Fire software, the other will run our Brain Computer Interface. This will be our new input device that will operate through a network socket. The Brains on Fire software will “listen” to the commands that the BCI software will send via the socket.

OK, Cool! But how can we make a network socket?
First you can create a very basic network socket between two small programs ran on a single PC. One of the programs will run as a server and the other obviously as the client. We will be using Python, because or other software is also written in Python and it is a very robust programming language and easy to set op a network socket.

server.py:

# this is the server

import socket
import sys

# create a socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# associate the socket with a port
host = "" # can leave this blank on the server side, it will take your IP address
port = int(2010) # you can choose your own port number, use 4 numbers to be safe
s.bind((host, port))

# accept "call" from client
s.listen(1)
conn, addr = s.accept()
print "client is at", addr

# read string from client (assumed here to be so short that one call to
# recv() is enough), and make multiple copies (to show the need for the
# "while" loop on the client side)

data = conn.recv(1000000)
data = 10000 * data # concatenate data with itself 999 times

# wait for the go-ahead signal from the keyboard (to demonstrate that
# recv() at the client will block until server sends)
z = raw_input()

# now send
conn.send(data)

# close the connection

conn.close()

client.py:

# this is the client

import socket
import sys

# create a socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# connect to server
host = Server-IP-Address # server address with quotation marks, for example "195.169.211.136"
port = int(2010) # server port
s.connect((host, port))

s.send(sys.argv[3]) # send test string

# read echo
i = 0
while(1):
      data = s.recv(1000000) # read up to 1000000 bytes
      i += 1
      if (i < 5): # look only at the first part of the message
           print data
      if not data: # if end of data, leave loop
            break
      print "received", len(data), "bytes"

# close the connection

Sweet! How can I run this?

To run the server in linux:

python server.py 2000

If that runs, open up a new terminal and run:

python client.py server_machine_name 2000 abc

For windows just open to cmd terminals and do the same, only leave python out of it.

This will create a very simple network socket. It will wait until you send a command from the server. From here you can try and build a more elaborate network socket program. What we need to do is create a simple network socket that listens to the buttons “pushed” by the BCI and send it to Brains on Fire. Let’s make this work!

The original control flow chart of Frets on Fire looks like this:

My new control flow chart of Brains on fire looks like this:

You can see that I use two buttons that are BCI controlled, and one simple key to pause or quit the game. This allows you to fully operate the menu and the game.

Of course we need songs that only use two frets. I edited a standard song with the Song Editor of Frets on Fire. By deleting all the notes on the other frets, I got a nice simple song.

The next thing that needed to be modified was the rhythm/pick button. We don’t want to use this within our BCI game, because it will be very hard to control two buttons at the same time. Also precise timing is a very difficult with the use of a BCI. I modified the source code in GuitarScene.py, which I showed one of my previous posts. For a detailed report of the changes that I made to the source code, you can send me a message. It finally works very well and you can play the game without a rhythm button.

I already stated that precise timing and BCI is a difficult to achieve. The thing I wanted to do, was change the span of the notes. This means that hitting a button half a second earlier or later is allowed. The lag between the BCI and the game software can be two seconds. The problem is that I couldn’t find an easy way to program this within Frets on Fire. I decided doing this by making all notes very long and allowing the players to start the notes in, for instance, the middle of the note.

With these simplifications Frets on Fire is ready to be changed in Brains on Fire. We still have to to connect the BCI to the software. There are a couple of things that we have to do before this will work.

• Step one: Allow a new device to connect with Brains on Fire
• Step two: Make a working network socket that listens to commands from another computer in Python.
• Step three: Implement the network socket into Brains on Fire
• Step four: Change the new device into the BCI
• Step five: Experiment with the BCI and the game

The next post will be about network sockets, what is this and how can you use this?

An open source program which allows you to use this way of developing is Subversion (SVN). You can store your original source code as a local repository, but you can also put it online for a wider use. It is also possible to use repositories already created by other people. All revisions are stored and you will be allowed to go back to any version that SVN saved.
It is especially very useful when you work together with other people and you can share your updates online. Everyone can see the changes you made with respect to to previous version.

But it is easy to use for every program that you will write, even for your web pages or documentation! A good recommendation for any developer.

The thing I have been thinking of is using so called “hammer-ons” or “tappable” notes. Hammer-ons are notes played directly after a strummed or picked note, therefore it doesn’t need to be picked again. If we can make all the notes act as hammer-ons we have eliminated the rhythm button. But how can we do this? We have to dig through the Python code.

After a while I found this in the GuitarScene.py file:

def songLoaded(self, song):

song.difficulty = self.player.difficulty
self.delay += song.info.delay

# If tapping is disabled, remove the tapping indicators

if not self.engine.config.get("game", "tapping"):

for time, event in self.song.track.getAllEvents():

if isinstance(event, Note):event.tappable = False

This code loads the song and its notes. When a note is not tappable they turn it to False and the note has to be picked. If we simply delete this small part of the code (last lines) then all notes will be tappable. We can also change event.tappable from False to True and see if that works.

There are some other lines in this file that could be changed to reach the same result, but we have to test whether it works this way. One problem I for see is that you always have to start with a strummed note before you can hammer-on. I know that because of my experience with the game, of you miss one note, and the next one is a hammer-on, you still have to pick it.

This can be done by changing in the same file this short code:

def keyReleased(self, key):

if self.controls.keyReleased(key) in KEYS and self.song:

# Check whether we can tap the currently required notes

pos   = self.getSongPosition()

notes = self.guitar.getRequiredNotes(self.song, pos)

if self.player.streak > 0 and \

self.guitar.areNotesTappable(notes) and \

self.guitar.controlsMatchNotes(self.controls, notes):

self.doPick()

# Otherwise we end the pick if the notes have been playing long enough

elif self.lastPickPos is not None and pos - self.lastPickPos > self.song.period / 2:

self.endPick()

The thing that is important here is the self.player.streak > 0. This means you have to have a streak of more than 0 hit notes. Else you can’t hammer-on and you have to pick. By simply removing this line you can hammer-on even if you missed the previous note.

Some other things I’ve been thinking about is creating simple songs in Frets on Fire. There is a song editor build in, but it’s not very good, one of the problems is that you can’t create hammer-ons by yourself and that it is frustrating to work with. By importing self-made midi files, Frets on Fire can create the notes for the game. This could be an interesting way to create some easy songs.

The first thing I have to solve is running it on a Linux machine. I’m currently playing Frets on Fire on a Windows machine, the problem here is that the source code is compiled in an other way, all the source files are *.pyd instead of *.py. The *.py files are the real source files, you can edit them if you like, and can be compiled every moment. The *.pyd files can’t be edited easily, these files are comparable to *.DLL files. So, at the moment I can only edit the Linux source, but I can’t play it on a Linux machine, because it gives me problems with OpenGL and some other packages. Hopefully in two weeks I have got another PC with Linux that will be able to play it, so that I can test my modifications!

In a nice looking tube, with lights and sounds, is a ping pong ball. The goal of the game is to elevate the ball to a certain height and keep it their. You get instructions from Yoda and after a while you can be a real Jedi.

For average people it is stunning, but we know it is relatively simple to create a BCI like this. The toys use an adapted form of the EEG technology to measure the brain activity. You already know a lot about this, because I described it in my earlier posts.

A small computer is connected to the fan, which blows the ball up in the air. The computer transforms the brain activity to fan intensity. So, you need to create a lot of brain activity to lift the ball to the highest platform. It is fun to control the ball with your thoughts, but it doesn’t matter if you really think about lifting the ball, a difficult mathematical problem or how you can steal the cookies out of your mother’s jar. The game is all about creating enough brain activity to control the fan and therefore the ball.

What is nice about this invention is that they created a cheap EEG system that can be used for commercial goals. I don’t think this system can be very precise, but it is a good development for BCI. The Force Trainer is available for $75 and is one of the most wanted Christmas gifts of 2009! A lot of people want these kind of “gadgets”, because of their coolness. Controlling a ball with your mind, just sounds very cool, doesn’t it? The success formula!

There is a toolbox for MATLAB called FieldTrip that is designed for MEG and EEG. It has been developed at the Centre for Cognitive Neuroimaging of the Donders Institute for Brain, Cognition and Behaviour in Nijmegen. The software includes algorithms for simple and advanced analysis of MEG and EEG data, such as time-frequency analysis, source reconstruction using dipoles, distributed sources and beamformers and non-parametric statistical testing. It supports the data formats of all major MEG systems and of the most popular EEG systems, new formats can be added easily. FieldTrip contains high-level functions that you can use to construct your own analysis protocols in MATLAB. Furthermore, it easily allows developers to incorporate low-level algorithms for new analysis methods.

Our goal is to develop systems which completely paralysed people (such as late-stage suffers of Amyotropic Lateral Sclerosis, ALS) could use to communicate. We are focussing on the brain, this means we can’t use muscles, peripheral nerves and machine artefacts. The problem is that these sources generate much stronger signals than the brain, thus we must remove them to be sure our BCI has a B in it.

To do this these possible artefacts need to be recorded, detected, rejected from the corrupted data and removed whilst leaving the data intact. That are a bunch of things that need to be done to get a clean signal and some of these steps can be very hard, especially detecting the artefacts en removing them.

Because datasets are usually very large it would be easy to have automatic artefact rejection, but this is very hard. The human eye is very good at noticing patterns, but to program this in software is difficult. Some artefacts that can be seen in data are eye movement (a), eye blink (b) and muscle tension (c). It could also be that a channel is bad or the baseline is drifted in time.

EEG data is very high dimensional (e.g. 64 electrodes * 3000 samples * 600 trials), this makes it hard to see what’s going on, therefore FieldTrip gives us some easy visualisation options. There are different ways to view BCI data in FieldTrip, but the most common are the multi channel view and the topoplot (d). FieldTrip can average over channels or samples, to make a more interesting averaged plot. Still to detect the artefacts we need to see all samples manually to make the data clean, with this amount of data it will be a never ending job.

Fortunately some of these artefacts have a particular shape, this shape can be used as a template to detect other artefacts that are alike. We can automate this and it is called Template Matching Detection. We can make our own template by detecting some artefacts by eye and average these shapes to one shape, this new shape can be used as a template. You can also separately record these artefacts, like record a blinking eye a couple of times. This saves time of searching for the artefacts in the data. There are more ways to detect the artefacts and even more advanced ones than described here.

After detection there are two things that can be done with this “infected” data, either throw the data away or attempt to clean the artefacts from the data. In a nutshell it will look like a simple subtraction, subtract the clean (recorded) eye blink from the data and the other data will remain intact.

There can be software written for this artefact detection and removal, but it will always be good to do some routine manual checks. After this artefact removal the data is clean enough to move on to the next step: classification and eventually play Brains on Fire!

Thanks to Jason Farquhar for some information on this one!

MATLAB is an interactive computer program that serves as a convenient “laboratory” for computations involving matrices. It provides easy access to matrix software developed by the LINPACK and EISPACK projects. The program is written in Fortran and is designed to be readily installed under any operating system which permits interactive execution of Fortran programs.

But in a couple of years MATLAB evolved and it became the standard instructional tool for introductory and advanced courses in mathematics, engineering and science. In industry, MATLAB is the tool of choice for high-productivity research, development and analysis.

After these years of expanding, MATLAB consists of five main parts:

• the language itself
• the working environment
• the graphic system
• the mathematical function library
• the application program interface (API)

By combining these parts you are able to build programs, output a visualisation of your functions, compare your variables and even write C and Fortran programs that can interact with MATLAB. There are a lot of specialized functions, also called Toolboxes, to use within MATLAB and make your analysis, computation or visualisation easier and better.

A downside is that the software is not free! There are some alternatives, like Ocatve and Scilab, but they aren’t as good and complete as the real deal.

This is a very general description of MATLAB, in the next post I will explain why this software is essential for my project and almost for every BCI project!

As you can see in the movie the application uses a 6×6 grid of all possible letters, some numbers, a white space and the submit/twit button. The rows and columns are flashed one by one, the order can be varied and adjusted by the frequency of use per letter to maximize spelling speed. Also the order of the letters in the grid could be changed, according to frequency of the letters and the language.

Like I described in my previous post it uses EEG to pick up the brainwaves. When the letter, the user wants to spell, is in the flashed row or column the user recognizes his letter and produces a P300 brainwave that is picked up by the software. The combination of one row and one column will leave one option, the right letter.

Unfortunately it still takes a lot of time to type a message, but people are able to do up to ten characters per minute with the right training. You learn it by training, just like the first time you texted on your cell phone.

This application deserved a lot of media attention and it is even in the CNN/Time’s “Best 50 Inventions of 2009″. This is one of the first examples where a BCI would be useful for a much larger community of people with neurological deficits. These people are able to send short tweets to their family and friends and can communicate more easily about the state they are in. They will feel not that much different from everybody else in the online world.