Are there any EEG procedures currently available/in development that can formulate letters/words from deliberate, internal monologue?

Question

I am thinking of the combination of an EEG-like device and software to monitor and analyze (very) deliberate thoughts of letters/words.

The input would be in the form of raw electrical brain activity from the EEG and the output from the software analysis would be in the form of text (the pattern-matched words/letters).

The software would be trained to recognize specific patterns of electrical impulses and form conclusions on what letter/word matches the best.

I have been interested in the idea of an input-less and effort-less HID such as this for a while now, mainly for personal enjoyment--I've entertained thoughts of working on documents with vim on the bed with a projector aimed at the ceiling--but realize that something like this would be of immense help to the disabled too.

I'm confident that teams are working right now on such problems, but my searches have yielded no results.

Here are some of my questions.

• Can anyone direct me to some relevant articles and papers?

• What is the current state of progress regarding the formulation of useful information from neural firing patterns and raw brain activity?

• Are current EEG and other-brain scanning devices available right now capable enough to detect and form distinctions between impulses such as the internalization of, say, letters "K" vs "F"?.

  • Can something like the OpenBCI do this?
  • Can something like the Emotiv system do this? They have two models, the "EPOC / EPOC+" and "Insight". The former offers 14 EEG channels while the latter offers only 5. Is 5 sufficient for such purposes?

• Is it a problem of insufficient hardware sensing resolution or the inability to pattern match such noisy data?

I looked at this post too but it did not answer my questions: Can we draw conclusions about content of thoughts from neural firing patterns?

---

Found some interesting links

• OpenEEG, but it looks dead now
• Very promising and exactly what I was looking for: OpenBCI, An Open Source Brain-Computer Interface For Makers
• More EEG info: Selecting an EEG Device
• Emotiv EEGs, also looks promising: Are recent affordable EEG devices any good?
• More discussion in a reddit thread here

Answer 1

This is a really fascinating possibility, but to be honest right now we are not even close to having consumer-based technology like this. The challenge comes down to a few things:

1. The signal-to-noise of EEG is really, really bad
2. It's unclear how to map features of brain activity into linguistic features like phonemes/words/etc
3. Time is a crucial aspect of brain activity, and a crucial aspect of speech, however we're not sure how to map the "timescale" of the brain onto the "timescale" of a person speaking. (e.g., think of the sentence "all dogs go to heaven" and then say it out loud. It probably took you a different amount of time in each case).

The best example of state-of-the-art for EEG decoding is probably the P300 speller. Basically, this takes advantage of a signal that pops up in your EEG only when you see something that you were already attending to. They give you a block of letters, tell you attend to one of them, and then start flashing letters in succession. When the letter you were attending to gets flashed, your brain has a special signal that the P300 speller picks up on. However, this types at the rate of only a few characters per minute, so it is far from the lucid stream of words/thoughts that we'd like to have.

The work from Gallant's lab is really cool, but far from a feasible decoder of brain activity. If I remember correctly, their movie decoder was actually selecting from a bunch of movies they had in a database. They'd take the top 10/20/whatever movies that the decoder selected, and then average them together. This is a really clever idea, but isn't the full-fledged decoder that would let them decode arbitrary thoughts etc.

There is some work in electrocorticography (a version of EEG where the electrodes are placed right on the person's brain instead of on their scalp) that has interesting results, but they are still far, far, far from a consumer product. E.g. see this paper from the Stephanie Martin / Brian Pasley for an attempt to decode acoustic features from imagined speech. The ECoG signal is a much higher-quality signal than EEG, and they definitely show an improved ability to decode, but it still leaves much to be desired. (full disclosure, I'm a co-author on that paper)

so tl;dr - these ideas are really fascinating, and hopefully they'll be a part of our society in the future. However, before we get to that point, we have major problems to solve such as finding a brain signal with a better SNR, creating more clever language models, and figuring out how time-scales are represented differently in the brain vs. during speech. Lots of progress yet to be made, but such is the nature of science :)

ps: I didn't mention any commercial brain decoding systems because these are all almost laughably bad right now. I don't know of any researchers that really believe that a system like Emotive actually records abstract states like "attention" "arousal" etc. Be wary of companies that are trying to make money by capitalizing on people's inherent enthusiasm for neuroscience. Sorry if that makes me sound like a crotchety scientist.

Answer 2

What you are looking for is Imagined / Silent Speech Classification (my best guess). I'm interested in this for a few years (exactly to work with Vim) now and worked with some of the cheap EEG headsets like Emotiv EPOC and Neurosky MindWave Mobile.

A paper on it: http://www.scipublish.com/journals/ABSE/papers/1021

Patent applications: http://www.freepatentsonline.com/
To the above link, add these at the end:
WO2014066855A1.html
WO2014130571A1.html

(Due to a lack of reputation, I cannot post more than 2 links.)

The commercial and inexpensive headsets are good enough to detect what their APIs support: Neurosky: Attention, Meditation and Eye blink. EPOC: Please refer to their Cognitive Suit documentation. You can get 1-2 sets of detections done quickly. Good luck with detecting 4 states.

Search around for Silent Speech Classification. You should be able to find more details. There was a Japanese demo somewhere that uses classification on EEG or EMG to speak. I will post it if I can find it.

Answer 3

I can respond a little to your second question, in particular as it related to EEG: What is the current state of progress regarding the formulation of useful information from neural firing patterns and raw brain activity?

As far as I know, although there are groups working on this general idea (decoding from brain activity) using a number of neuroscientific techniques. In terms of EEG, the data has a very low signal-to-noise rate on single trials (this is far worse in commercial systems), although a search of "single-trial EEG/ERP analysis" would yield some papers, especially recently, largely on binomial classification problems.

At the level of individual words/phrases, it is extremely difficult, if not impossible, to use EEG to decode accurately. It is likely the case that there some evidence for a few broad classifications (car vs face) using EEG data, but as far as I know the best use of EEG for decoding "language" is using the decision-related P300 component in clinical settings (search "P300 speller", as well as the work by Emanuel Donchin, for more info).

Outside of EEG, I know of a number of labs who are using Magnetencephalography (MEG) to try to decode characters of conceptual and linguistic thought. For example, Tom Mitchell's lab is doing some work using individuals' reading entire passages of Harry Potter in an MEG machine, and then looking at ability to decode characters, characters' personality states and action, settings, etc. I know there are some interesting findings, but even with a $2MM machine it is pretty limited. (And, even if you were a research billionaire, you couldn't exactly walk around with a MEG on your head.)

Mitchell's work builds on work done using functional magnetic imaging (fMRI) with Marcel Just. This ended up on 60 minutes with Lezley Stahl and is on youtube. They were able to predict the category of items individuals were thinking of to a relatively high level after training the decoder. The coolest working in this area (IMHO) is in Jack Gallant's lab in Stanford?. Gallant et al have some ridiculous whole brain semantic mapping generated by having individuals watch movies that poor doctoral students coded scene by scene for everything. These maps show how the brain changes its receptivity/tuning/representational nature/who knows dynamically as we watch natural videos (e.g., people in a scene radically changes the brain's conceptual state, if that makes any sense). My point with Gallant's work is that we may first be better able decode concepts and categories at a fine level before we drop down to sentences and words, at least in any non-strongly-constrained system (one with 5 words).

But, I don't know anything, maybe next year they will pop a chip in the ventral pathways and start reading words like a typewriter...

Answer 4

Your thought has been first tested by Larry Pinneo (1975), he got absolutely nowhere in the DARPA Biocybernetics program. In 1981, Russian scientists have found specific waves for specific meanings, "for example, that waves for concepts such as chair, desk, and table are all overlapped by another wave that corresponds to the word furniture.” Check out this article, "Mind reading computer" by Staff Writer. https://sites.google.com/site/mcrais1/pinneo

• Pinneo, L. R. (1975). Persistent EEG Patterns Associated with Overt and Covert Speech. Bulletin of the Human Factors Society, 18(2), 1-2.

Answer 5

Unfortunately, EEG just isn't that high-quality of a signal. But electrocorticography, or basically EEG collected on the surface of the brain, is much better. Robert Knight's lab came out with a paper in (or around) 2012 that had single-trial generation of speech from brain activity.

Citation: http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1001251
Reconstructing Speech from Human Auditory Cortex
Brian N. Pasley , Stephen V. David, Nima Mesgarani, Adeen Flinker, Shihab A. Shamma, Nathan E. Crone, Robert T. Knight, Edward F. Chang