I am a student in the Linguistics M.A. program at Brigham Young University. I work for Dr. Deryle Lonsdale as a research assistant in NLSoar and PSST and as teaching assistant for LingC 361. This semester I am also teaching the Internet Publishing course, CHum 250.

My academic endeavors have focused on research in cognitive modeling, speech recognition and machine learning. I have also done work in other areas of computational linguistics including: second language assessment, working memory, bi-lingualism and animal [bio-acoustic] communication (more specifics on the research groups and current projects below). In addition to my experience with computational linguistics I have worked as a web developer for many years and enjoy learning new technologies (R, ruby and erlang currently top the list). My work experience is also summarized below.

Last updated: May 24th, 2011
Scroll down or click on the links to find out more. Drop me line here.

Current Projects

1. XNLSoar

   Implementing minimalist syntax in SOAR

   In this project we are working to create a system for sentence processing using the Minimalist Program for the underlying syntax. We have made good progress and currently have a functioning prototype. We are now adding additional functionality to the system to handle more linguistic constructions and address other issues, including word sense disambiguation. We will be presenting on this research at the upcoming BRIMS conference.

2. Elicited Imitation

   Using elicited imitation for automatic assesment of oral language proficiency

   We have shown elicited imitation to be a consistent and effective technique for estimating the oral language proficiency of second language learners of English. Our 2008 LREC paper gives a good overview of the techniques used. More recently, in a paper published by SLRF, we explored the effect of a person's native language on their test performance. The PSST group has also been successful in using the elicited imitation framework with other languages including: Japanese, Spanish, French and Korean. We will be presenting on the Spanish effort at LTRC 2011 and on some of the technical details of the system at AALL III. We are also preparing on a paper for the BEA workshop at ACL. The ultimate goal of this project is to create a fast, low-cost and fully automatic test to estimate a student's performance on a high stakes examination.

Research Groups

1. NLSoar Research Group

   Natural Language Processing in the SOAR cognitive architecture

   Details:

   The aim of the BYU Soar Research Group is the processing and modeling of linguistic data within the Soar agent framework, and to investigate related areas such as language/task integration, psycholinguistic modeling, and language learning. We are actively developing and maintaining two Soar systems:

   1. NL-Soar, a cognitive modeling system for natural language use
   2. LG-Soar, a Soar-based information extraction system using the Link Grammar parser

   For even more info on our research, presentations and publications check out our wiki.

2. PSST Research Group

   Pedagogical Software and Speech Technology

   Details:

   The principal focus of this group is to explore and expand the use of speech technology in language learning. We work with existing speech technologies, examine pedagogical needs for language learning, and design and develop improved technological tools (including speech applications) for language learning and testing.

   Besides this principal focus, our interests have expanded to include such areas as: automatic assessment of oral language proficiency, simultaneous interpreter training, text-based essay scoring, speech interfaces to robots, vocal attractiveness, and using speech recognition in animal communication.

   For even more info on our research, presentations and publications check out our wiki.

3. AM Research Group

   Analogical Modeling of Language

   Details:

   Analogical modeling is an exemplar-based (or instance-based) approach to predicting phenomena. It has been used in a wide range of applications including modeling how we use human language. The system works by assigning one or more possible outcomes to a test instance by comparing it to a set of previously supplied, labeled instances.

   The system uses a unique processing algorithm that does not use rule-based, connectionist, or nearest-neighbor metrics. Instead, its decision procedures and measures directly use analogy to measure the degree of (dis)similarity between items.

Publications

1. Elicited Imitation for Prediction of OPI Test Scores. (2011)

   Proceedings of the Sixth Workshop on Innovative Use of NLP for Building Educational Applications.

   Forthcoming Summary:

   Automated testing of spoken language is the subject of much current research. Elicited Imitation (EI), or sentence repetition, is well suited for automated scoring, but does not directly test a broad range of speech communication skills. An Oral Pro?ciency Interview (OPI) tests a broad range of skills, but is not as well suited for automated scoring. Some have suggested that EI can be used as a predictor of more general speech communication abilities. We examine EI for this purpose. A fully automated EI test is used to predict OPI scores. Experiments show strong correlation between predicted and actual OPI scores. Effectiveness of OPI score prediction depends upon at least two important design decisions. One of these decisions is to base prediction primarily on acoustic measures, rather than on transcription. The other of these decisions is the choice of sentences, or EI test items, to be repeated. It is shown that both of these design decisions can greatly impact performance. It is also shown that the effectiveness of individual test items can be predicted.

2. Resolving a syntactic ambiguity type with semantic memory. (2011)

   Proceedings of the Twentieth Conference on Behavior Representation in Modeling and Simulation.

   Full Paper Summary:

   This paper inspects the linguistic ambiguity of prepositional phrase attachment using natural language comprehension in the SOAR architecture. These ambiguities include sentences like, "I saw a man with a telescope." Compare the previous sentence to, "I saw the man with a moustache." We leverage a corpus into SOAR's semantic memory module to show how it can inform parsing decisions in this type of syntactic ambiguity resolution.

3. Variability in L2 acquisition across L1 backgrounds. (2011)

   Selected Proceedings of the 2009 Second Language Research Forum.

   Forthcoming Abstract:

   Elicited imitation (EI) has provided helpful insight into the process of language acquisition and language testing. EI is an oral test in which the subject hears a string of words, forms a cognitive representation, and then produces a string of words according to that representation (Bley-Vroman & Chaudron 1994, 245). EI theory asserts that a person’s ability to reproduce a sentence is connected with that person’s knowledge of the structure of the language. Similarity between English and a learner’s L1 differs greatly depending on the L1, and there is still much research to be done on specific English structures that are most problematic for different L1 speakers. Odlin (1989) explains that similarities and differences between an L1 and L2 affect the acquisition of the L2. There is research on general structures that are problematic for specific L1s, such as plurality and negation (Romaine, 2003), but precise syntactic and morphological structures of English most problematic for different L1 language families have not been compared and contrasted through one test.

   Syntactic and morphological structures are highly problematic for English language learners (DeKeyser & Goldschneider, 2005). This paper reports on efforts to analyze EI data from over 1000 ESL participants to determine the most problematic syntactic and morphological structures for each group. Each participant has taken at least one EI test, which focused on syntactic and morphological structures such as tense, aspect, modality, and phrasal structure. We grouped participants according to their L1 language family (Britannica, 2008), and discovered different syntactic and morphological structures that were most problematic for each group. We compared the most problematic and simple structures from each group to discover differences and consistencies across L1 language families.

4. The Role of Lexical Choice in Elicited Imitation Item Difficulty. (2010)

   Selected Proceedings of the 2008 Second Language Research Forum.

   Full Paper Abstract:

   In recent years, elicited imitation (EI) has become a viable way to measure language proficiency (Vinther, 2002, Chaudron, Nguyen, Prior, & Kozok, 2005). But up to the present selection of items to be used in these tests has been based largely on sentence length and syntactic complexity. There has been little investigation into the effects of lexical choice on EI test items.

   The purpose of this study was to investigate the effects of lexical density and lexical difficulty on the difficulty of EI items. Lexical density is the ratio of content words to the total words in the sentence (McCarthy, 1990). There are several ways of estimating lexical difficulty. The most common of these and the ones that we have chosen to use are frequency and morphological complexity.(Nation, 1991; Gardner, 2007).

   The EI instrument used in the study consisted of 60 sentences, ten sets of six sentences each, varying in length from four syllables to 22 syllables. In each set of six sentences, length and syntactic complexity was kept constant while vocabulary difficulty was varied from easy to difficult. The instrument, along with a previously calibrated set of items, was administered to 185 adult ESL learners from a variety of native language backgrounds and English proficiency levels. Item difficulty was calculated using IRT analyses and a multiple linear regression analysis was used to examine the relative contribution of lexical density and lexical difficulty to item difficulty.

   Some complex relationships were discovered between item difficulty and lexical density and difficulty. Implications of these findings as they relate to the design of EI tests will be discussed.

5. What makes an item difficult?
   A syntactic, lexical, and morphological study of Elicited Imitation test items. (2010)

   Selected Proceedings of the 2008 Second Language Research Forum

   Full Paper Abstract:

   In recent years there has been a renewed interest in using elicited imitation as a tool for examining oral language skills in second language learners (Chaudron, Prior, & Kozok, 2005; Erlam 2006). Some attention has been paid to the role of syntax in determining the degree to which learners’ errors during sentence imitation mirror those which occur during spontaneous speech. However, in the use of elicited imitation as a tool for measuring language proficiency, little work has been done to examine variables which may affect test design. Some of these include the mode of administration of items (delay or no delay; test for comprehension or not; timed or open-ended responses), the methods of scoring (four point scale, counting correct syllables; holistic scoring of quality of utterances; fluency measures), and the nature of the items themselves (item length, lexical choices, morphological forms, syntactic patterns). The focus of this paper will be on examining the nature of item difficulty—specifically, what are the factors which contribute to the degree to which learners can successfully repeat items.

   In the current paper we report the results of a principal components analysis of approximately 150 Elicited imitation items administered to approximately 380 adult learners of English as a second language in an intensive English program. Item analyses were performed on these scores and reliability coefficients ranging from .96 to .98 were found. Using item response theory (IRT) analyses of the items we determined item difficulty scores for each item. We then identified the lexical, morphological, and syntactic features of each sentence, along with general measures of sentence length and complexity and performed a multiple linear regression analysis on the data to determine which factors were contributing to item difficulty. Implications of the results for test design will be discussed.

6. Incremental processing and research usage. (2009)

   9th International Confernce on Cognitive Modeling

   Full Paper Summary:

   This paper showed that using different syntactic stragies in parsing sentences entailed different resource usage by the SOAR cognitive architecture. The different strategies used are outlined as well as the resource usage reults.

7. Elicited Imitation as an Oral Proficiency Measure with ASR Scoring. (2008)

   Proceedings of the Sixth International Conference
   on Language Resouces and Evaluation

   Full Paper Abstract:

   This paper discusses development and evaluation of a practical, valid and reliable instrument for evaluating the spoken language abilities of second-language (L2) learners of English. First we sketch the theory and history behind elicited imitation (EI) tests and the renewed interest in them. Then we present how we developed a new test based on various language resources, and administered it to a few hundred students of varying levels. The students were also scored using standard evaluation techniques, and the EI results were compared to more traditionally derived scores. We also sketch how we developed a new integrated tool that allows the session recordings of the EI data to be analyzed with a widely used automatic speech recognition (ASR) engine. We discuss the promising results of the ASR engine’s processing of these files and how they correlated with human scoring of the same items. We indicate how the integrated tool will be used in the future. Further development plans and prospects for follow-on work round out the discussion.

8. Learning to Recognize Vervet Monkey Calls. (2007)

   Brigham Young University 2009 Journal of Undergraduate Research

   Full Paper Abstract:

   The use of computational approaches for speech recognition and speech processing of human language is well documented. Less well-known are recent efforts to apply these language understanding techniques and tools to animal communication for mammals (Cetacean project, n.d.), birds (Fox et al. 2006), and insects (Potamitis et al. 2006). Vervet monkeys establish elaborate social groups and communicate extensively with each other (Seyfarth 1990). Expert ethologists can differentiate many of the vervet monkeys' calls expressing "speech acts," many of which include semantic-like properties. Even artificially generated calls elicit an appropriate response from vervets. Previous work has aimed at an acoustic and statistical description of these calls and their differentiation (Owren 1988). The work discussed in this paper is the first to perform machine learning in discriminating some of these call types. Extensive vervet call data was collected during field observation. This data has been disseminated under the title "TalkBank Ethology Data: Field Recordings of Vervet Monkey Calls". Each call was subsequently annotated by experts with information identifying the individual monkeys involved, the call type, and other possibly relevant details. We will briefly present sample entries and calls from the corpus relating to our work. We will next discuss how we extracted features for automatic categorization of these calls. From the corpus we carried out an analysis of the acoustical properties of several dozen calls using a public-domain speech analysis tool. Specifically, we chose to address two types of calls—those warning about the approach of a snake or of an eagle—that have unique acoustic properties. Features deemed the most appropriate for differentiating these call types were converted to training and testing data for machine learning. Evaluations were carried out on the performance of a memory-based learning system which took as input these putative discriminative features and attempted to classify the call type. Finally, we will discuss and analyze the results of our machine learning paradigm and hypothesize how we may be able extend our analysis to include other types of vervet calls from the corpus.

Experience

Technical Skills

• PHP, XHTML, CSS, JavaScript, MySQL
• PERL, Visual Basic, C, Java
• XML, ProLog, TCL, Shell scripting

Linguistic Competencies

• Natural Language Processing
• Speech Recognition
• Machine Learning
• Theoretical Syntax
• Morphology, Phonology, Phonetics
• Dialogue, Discourse, Lexicography

Job History

Research AssistantMar 2007 - Present

• Brigham Young University, Provo, Utah
  As a member of the nlSoar Research Group explored the use of cognitive architecture in NLP, specifically language processing.



Research AssistantJul 2004 - Present

• Brigham Young University, Provo, Utah
  As a member of the Pedagogical Software and Speech Technology Research Group utilized programs to explore the use and implementation of speech technology in language learning.



Research InternMay 2010 – Aug 2010

• Air Force Research Lab, Mesa, AZ
  Performance and Learning Models Team
  Created visualization tools for syntactic hierarchy used by cognitive model to process natural language. Also created a PERL pipeline for visualizing large semantic lexicon utilized by the same cognitive model of language.



Computational LinguistAug 2007 – April 2010

• Global Translation Inc., Wheaton, MD
  Review code-base for speech recognition system, review and generate linguistic rules to improve machine translation, phonetic and word level transcriptions for use in training system.



Web DeveloperMay 2006 – Nov 2007

• Prudential Utah Real Estate, Orem, Utah
  Create and update company and agent websites primarily using PHP and MySQL, developing interfaces with different APIs using AJAX, XML and Javascript.



Web DeveloperJul 2005 - Apr 2006

• Hobbytron.com, Orem, Utah
  Responsible for web maintenance and updates. Customer service and administration.

Curriculum Vitae

Education

Master of Arts - Linguistics

• Emphasis: Computational Approaches
  Brigham Young University, Provo, UT.
  Projected Graduation: April 2011




Bachelor of Arts - Linguistics

• Minor: Computing and Information
  Brigham Young University, Provo, UT.
  Graduation: April 2008

Teaching Experience

• Instructor. CHum 250. Internet Publishing. Winter 2011
• Teaching Assistant. CHum 361. Speech Processing. Winter 2011
• Teaching Assistant. CHum 250. Internet Publishing. Winter 2010
• Teaching Assistant. CHum 250. Internet Publishing. Fall 2009
• Teaching Assistant. Ling 330. Introduction to Linguistics. Fall 2008
• Teaching Assistant. CHum 345. Web Databases. Fall 2008
• Teaching Assistant. Ling 330. Introduction to Linguistics. Winter 2008
• Teaching Assistant. Ling 330. Introduction to Linguistics. Winter 2007
• Teaching Assistant. CHum 250. Internet Publishing. Fall 2006

Presentations

• Elicited Imitation for Prediction of OPI Test Scores.
  Sixth Workshop on Innovative Use of NLP for Building Educational Applications (BEA-6 at ACL-HLT 2011); Portland, OR; June 2011.
• Using Elicited Imitation Tests to Test Spanish Oral Proficiency.
  33rd Language Testing Research Colloquium (LTRC 2011); Ann Arbor, MI; June 2011.
• NLP and Elicited Imitation in the Assessment of Leaner Oral Proficiency.
  Automatic Analysis of Learner Language (AALL III)
  The Computer Assisted Language Instruction Consortium (CALICO); Victoria, BC, Canada; May 2011.
• Resolving a syntactic ambiguity type with semantic memory.
  20th Conference on Behavior Representation in Modeling and Simulation (BRiMS 2011); March 2011
• Effects of Multiple Exposure on Elicited Imitation Test Administration.
  American Association for Applied Linguistics (AAAL 2010); Atlanta, GA; March 2010.
• Variability in L2 Acquisition Across L1 Language Families.
  Second Language Research Forum (SLRF 2009); Michigan State University, Flint, MI; October 2009.
• Improving automated oral testing: identifying features and enhancing speech recognition
  Linguistic Association of the Southwest (LASSO 2009); Brigham Young University, Provo, UT; September 2009.
• L2 Morpheme Acquisition Order Across L1 Language Families.
  Linguistic Association of the Southwest (LASSO 2009); Brigham Young University, Provo, UT; September 2009.
• Incremental processing and resource usage.
  9th International Conference on Cognitive Modeling (ICCM 2009); Manchester, UK; July 2009.
• Update on SOAR 9 and Sentence Processing.
  29th Soar Workshop (2009); University of Michigan, Ann Arbor, MI; June 2009.
• Methods of Scoring Elicited Imitation Items: an Empirical Study.
  American Association for Applied Linguistics (AAAL 2009); Denver, CO; March 2009.
• Improving Automated Oral Testing: Identifying Features and Enhancing Speech Recognition.
  Automatic Analysis of Learner Language (AALL '09)
  The Computer Assisted Language Instruction Consortium (CALICO); Arizona State University, Phoenix, AZ; March 2009.
• The Role of Lexical Choice in Elicited Imitation.
  Second Language Research Forum (SLRF 2008); University of Hawaii, Laie, HI, October 2008.
• What makes test items difficult? - A syntactic, lexical, and morphological study of Elicited Imitation test items.
  Second Language Research Forum (SLRF 2008); University of Hawaii, Laie, HI, October 2008.
• Elicited Imitation as an Oral Proficiency Measure with ASR Scoring.
  Proceedings of the Sixth International Conference on Language Resources and Evaluation. (LREC 2008); Marakesh, Morocco, July 2008.
• Scoring an Oral Language Test Using Automatic Speech Recognition.
  Workshop on Automatic Analysis of Learner Language (AALL '08)
  The Computer Assisted Language Instruction Consortium and The International Association for Language Learning Technology (CALICO with IALLT); San Francisco, CA. March 2008.
• Learning to Recognize Vervet Monkey Calls.
  Linguistic Association of Canada and the United States (LACUS 2007); Eastern Kentucky University, Richmond KY; July 2007.
• The Use of Speech Technologies in Developing Communicative Learning Activities.
  15th International Symposium and Book Fair on English Teaching; Taipei, Taiwan. November 2006.

Drop Me A Line

To contact me, email me at jeremiah.mcghee[at]byu.edu or jeremiah.mcghee[at]gmail.com.