Have a good summer, and see you in the fall!

Thanks so much to everyone who was able to join us for our lively discussion today, and to everyone who's joined us this past academic year!

The CoLa Reading Group will be on hiatus this summer, and we'll resume again in the fall quarter.  As always, feel free to send me suggestions of articles you're interested in reading, especially if you happen across something particularly interesting!

Some thoughts on Kwiatkowski et al 2012

One of the things I really enjoyed about this paper was that it was a much fuller syntax & semantics system than anything I've seen in awhile, which means we get to see the nitty gritty in the assumptions that are required to make it all work. Having seen the assumptions, though, I did find it a little unfair for the authors to claim that no language-specific knowledge was required - as far as I could tell, the "language-universal" rules between syntax and semantics at the very least seem to be a language-specific kind of knowledge (in the sense of domain-specific vs. domain-general). In this respect, whatever learning algorithms they might explore, the overall approach seems similar to other learning models I've seen that are predicated on very precise theoretical linguistic knowledge (e.g., the parameter-setting systems of Yang 2002, Sakas & Fodor 2001, Niyogi & Berwick 1996, Gibson & Wexler 1994, among others.) It just so happens here that CCG assumes different primitives/principles than those other systems - but domain-specific primitives/principles are still there a priori.

Getting back to the semantic learning - I'm a big fan of them learning words besides nouns, and connecting with the language acquisition behavioral literature on syntactic bootstrapping and fast mapping.  That being said, the actual semantics they seemed to learn was a bit different than what I think the fast mapping people generally intend.  In particular, if we look at Figure 5, while three different quantifier meanings are learned, it's more about the form the meaning takes, rather than the actual lexical meaning of the word (i.e., the form for a, another, and any looks identical, so any differences in meaning are not recognized, even though these words clearly do differ in meaning). I think lexical meaning is what people are generally talking about for fast mapping, though. What this seems like is almost grammatical categorization, where knowing the grammatical category means you know the general form the meaning will have (due to those linking rules between syntactic category and semantic form) rather than the precise meaning - that's very in line with syntactic bootstrapping, where the syntactic context might point you towards verb-y meanings or preposition-y meanings, for example.

More specific thoughts:

I found it interesting that the authors wanted to explicitly respond to a criticism that statistical learning models can't generate sudden step-like behavior changes.  I think it's certainly an unspoken view by many in linguistics that statistical learning implies more gradual learning (which was usually seen as a bonus, from what I understood, given how noisy data are). It's also unclear to me that the data taken as evidence for step-wise changes really reflect a step-wise change or instead only seem to be step-wise because of how often the samples were taken and how much learning happened in between.  It's interesting that the model here can generate it for learning word order (in Figure 6), though I think the only case that really stands out for me is the 5 meaning example, around 400 utterances.

I could have used a bit more unpacking of the CCG framework in Figure 2. I know there were space limitations, but the translation from semantic type to the example logical form wasn't always obvious to me. For example, the first and last examples (S_dcl and PP) have the same semantic type but not the same lambda calculus form. Is the semantic type what's linked to the syntactic category (presumably), and then there are additional rules for how to generate the lambda form for any given semantic type?

This provides a nice example where the information that's easily available in dependency structures appears more useful, since the authors describe (in section 6) how they created a deterministic procedure for using the primitive labels in the dependency structures to create the lambda forms. (Though as a side note, I was surprised how this mapping only worked for a third of the child-directed speech examples, leaving out not only fragments but also imperatives and nouns with prepositional phrase modifiers. I guess it's not unreasonable to try to first get your system working on a constrained subset of the data, though.)

I wish they had told us a bit more about the guessing procedure they used for parsing unseen utterances, since it had a clear beneficial impact throughout the learning period. Was it random (and so guessing at all was better than not, since sometimes you'd be right as opposed to always being penalized for not having a representation for a given word)?  Was it some kind of probabilistic sampling?  Or maybe just always picking the most probable hypothesis?

Some thoughts on Frank et al. 2010

So what I liked most about this article was the way in which they chose to explore the space of possibilities in a very computational-level way. I think this is a great example of what I'd like to see more of. As someone also interested in cross-linguistic viability for our models, I have to also commend them for testing on not just one foreign language, but on three.

So there were a number of aspects of the model which I think could have been more clearly specified. For instance, I don't believe they ever explicitly say that the model presumes knowledge of the number of states to be learned. Actual infants don't have the benefit of the doubt in this regard, so it would be nice to know what would happen if you inferred that from the data. It turns out there's a well specified model to do that, but I'll get to that later. Another problem with their description of the model has to do with how their hyperparameters are sampled. They apparently simplify the process by resampling only once per iteration of the Gibbs sampler. I'm happy with this although I'm going to assume that it was a typo that they say they run their model for 2000 iterations (Goldwater seems to prefer 20,000). Gibbs samplers tend to converge more slowly on time-dependent models so it would be nice to have some evidence that the sampler has actually converged. Splitting the data by sentence type seems to increase the size of their confidence intervals by quite a lot, which may be an artifact of having less data per parameter, but could also be due to a lack of convergence.

Typically I have to chastise modelers who attempt to use VI or V-measure, but fortunately they are not doing anything technically wrong here. They are correct in that comparing these scores across corpora is hazardous at best. Both of these measures are biased, VI prefers small numbers of tags and V-measure prefers large numbers of tags (they claim at some point that it is "invariant" to different numbers of tags, this is however not true!). It turns out that a measure, V-beta, is more useful than either of these two in that it is unbiased for the number of categories. So there's my rant about the wonders of V-beta.

What I really would have liked to see would be an infinite HMM for this data, which is a well-specified, very similar model which can infer the number of grammatical categories in the data. It has an efficient sampler (as of 2008) so there's no reason they couldn't run that model over their corpus. It's very useful for us to know what the space of possibilities is, but to what extent would their results change if they gave up the assumption that you knew from the get-go how many categories there were? There's really no reason they couldn't run it and I'd be excited to see how well it performed.

The one problem with the models they show here as well as the IHMM is that neither allows for there to be shared information about transition probabilities or emission probabilities (depending on the model) across sentence types. They're treated as entirely different. They mention this in their conclusion, but I wonder if there's any way to share that information in a useful way without hand coding it somehow.

Overall, I'm really happy someone is doing this. I liked the use of some very salient information to help tackle a hard problem, but I would've liked to have seen it a little more realistic by inferring the number of grammatical categories. I might've also liked to have seen better evidence of convergence (perhaps a beam sampler instead of Gibbs, at the very least I hope they ran it for more than 2000 iterations).

Next time on 5/28/13 @ 2pm in SBSG 2200 = Kwiatkowski et al. 2012

Thanks to everyone who joined our meeting this week, where we had a very thoughtful discussion about the experimental design for investigating "less is more" and the implications of the computational modeling in Perfors 2012.  Next time on Tuesday May 28 @ 2pm in SBSG 2200, we'll be looking at an article that presents an incremental learning model that incorporates both syntactic and semantic information during learning:

Kwiatkowski, T., Goldwater, S., Zettlemoyer, L., & Steedman, M. 2012. A Probabilistic Model of Syntactic and Semantic Acquisition from Child-Directed Utterances and their Meanings. Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics.

http://www.socsci.uci.edu/~lpearl/colareadinggroup/readings/KwiatkowskiEtAl2012_SynSemAcq.pdf

See you then!

-Lisa

Some thoughts on Perfors 2012 (JML)

One of the things I quite liked about this paper was the description of the intuitions behind the different model parameters and capacity limitations. As a computational modeler who's seen ideal Bayesian learners before, could I have just as easily decoded this from a standard graphical model representation? Sure.  Did I like to have the intuitions laid out for me anyway?  You bet. Moreover, if we want these kind of models to be recognized and used within language research, it's good to know how to explain them like this. On a related note, I also appreciated that Perfors explicitly recognized the potential issues involved in extending her results to actual language learning. As with most models, hers is a simplification, but it may be a useful simplification, and there are probably useful ways to un-simplify it.

It was also good to see the discussion of the relationship between the representations this model used for memory and the existing memory literature. (Given the publication venue, this probably isn't so surprising, but given that my knowledge of memory models is fairly limited, it was helpful to see this spelled out.)

I think the most surprising thing for me was how much memory loss was required for the regularization bias to be able to come into play and allow the model to show regularization. Do we really think children only remember 10-20% of what they hear? (Maybe they do, though, especially in more realistic scenarios.)

More specific thoughts:

Intro: I found the distinctions made between different "less is more" hypothesis variants to be helpful, in particular the difference between a "starting small" version that imposes explicit restrictions on the input (because of attention, memory, etc.) to identify useful units in the input vs. a general regularization tendency (which may be the byproduct of cognitive limitations, but isn't specifically about ignoring some of the input) which is about "smoothing" the input in some sense.

Section 2.1.2: The particular task Perfors chooses to investigate experimentally is based on previous tasks that have been done with children and adults to test regularization, but I wonder what kind of task it seemed like to the adult subjects. Since the stimuli were presented orally, did the subjects think of each one as a single word that had some internal inconsistency (and so might be treating the variable part as morphology tacked onto a noun) or would they have thought of each one as one consistent word plus a separate determiner-like thing (making this more of a combinatorial syntax task)?  I guess it doesn't really matter for the purposes of regularization - if children can regularize syntax (creoles, Nicaraguan sign language, Simon), then presumably they regularize morphology (e.g., children's overregularization of the past tense in English, like goed), and it's not an unreasonable assumption that the same regularization process would apply to both. Perfors touches again on the issue of how adults perceived the task a little in the discussion (p.40) - she mentions that mutual exclusivity might come into play if adults viewed this as a word learning task, and cause more of a bias for regularization.  Whether it's a morphology task or a combinatorial syntax task, I'm not sure I agree with that - mutual exclusivity seems like it would only apply if adults assumed the entire word was the name of the object (as opposed to the determiner-thing being an actual determiner like the or a or morphology like -ed or -ing). Because only a piece of the entire "word" would change with each presentation of the object, it doesn't seem like adults would make that assumption.

Section 3.0.6: For the Prior bias, it seems like prior is constructed from the global frequency of the determiner (based on the CRP). This seems reasonable, but I wonder if it would matter any to have a lexical-item-based prior (maybe in addition to the global prior)? I could imagine that the forgotten data for any individual item might be quite high (even if others are low) when memory loss is less than 80-90% globally, which might allow the regularization effects to show up without needing to forget 80-90% of all the data.

Section 4: It's an interesting observation that the previous experiments that found regularization effects conducted the experiment over multiple days, where consolidation during sleep would have presumably occurred. Perfors mentions this as a potential memory distortion that doesn't occur during encoding itself, or retrieval, but rather with the processes of memory maintenance. If this is true, running the experiments again with adults, but over multiple days, should presumably allow this effect to show up.

Next time on 5/14 @ 2pm in SBSG 2200 = Perfors 2012 JML

Thanks to everyone who joined our meeting this week, where we had a very spirited and enlightening discussion about Lignos 2012 and the ideas behind it! Next time on Tuesday May 14 @ 2pm in SBSG 2200, we'll be looking at an article that investigates the interplay between memory limitations and overregularization behavior in learners, providing both experimental and computational modeling results:

Perfors, A. 2012. When do memory limitations lead to regularization? An experimental and computational investigation. Journal of Memory and Language, 67, 486-506.

See you then!

Some thoughts on Lignos 2012

I found the simplicity of the proposed algorithm in this paper very attractive (especially when compared to some of the more technically involved papers we've read that come from the machine learning literature). The goal of connecting to known experimental and developmental data of course warmed my cognitive modeler's heart, and I certainly sympathized with the aim of pushing the algorithm to be more cognitively plausible.  I did think some of the criticisms of previous approaches were a touch harsh, given what's actually implemented here (more on this below), but that may be more of a subjective interpretation thing.  I did find it curious that the evaluation metrics chosen were about word boundary identification, rather than about lexicon items (in particular, measuring boundary accuracy and word token accuracy, but not lexicon accuracy).  Given the emphasis on building a quality lexicon (which seems absolutely right to me if we're talking about the goal of word segmentation), why not have lexicon item scores as well to get a sense of how good a lexicon this strategy can create?

Some more specific thoughts:

Section 2.1, discussing the 9-month-old English-learning infants who couldn't segment Italian words from transitional probabilities alone unless they had already been presented with words in isolation: Lignos is using this to argue against transitional probabilities as a useful metric at all, but isn't another way to interpret it simply that transitional probabilities (TPs) can't do it all on their own?  That is, if you initialize a proto-lexicon with a few words, TPs would work alright - they just can't work right off the bat with no information.  Relatedly, the discussion of the Shukla et al. 2011 (apparently 6-month-old) infants who couldn't use TPs unless they were aligned with a prosodic boundary made me think more that TPs are useful, just not useful in isolation.  They need to be layered on top of some existing knowledge (however small that knowledge might be).  But I think it just may be Lignos's stance that TPs aren't that useful - they seem to be left out as something a model of word segmentation should pay attention to in section 2.4.

Of course, I (and I'm assuming Lawrence as well, given Phillips & Pearl 2012) was completely sympathetic to the criticism in section 2.3 about how phonemes aren't the right unit of perception for the initial stages of word segmentation. They may be quite appropriate if you're talking about 10-month-olds, though - of course, at that point, infants probably have a much better proto-lexicon, not to mention other cues (e.g., word stress). I was a little less clear about the criticism (of Johnson & Goldwater) about using collocations as a level of representation.  Even though this doesn't necessarily connect to adult knowledge of grammatical categories and phrases, there doesn't seem anything inherently wrong with assuming infants initially learn chunks that span categories and phrases, like "thatsa" or "couldI". They would have to fix them later, but that doesn't seem unreasonable.

One nice aspect of the Lignos strategy is that it's incremental, rather than a batch algorithm.  However, I think it's more a modeling decision rather than an empirical fact to not allow memory of recent utterances to affect the segmentation of the current utterance (section 3 Intro).  It may well turn out to be right, but it's not obviously true at this point that this is how kids are constrained.  On a related note, the implementation of considering multiple segmentations seems a bit more memory-intensive, so what's the principled reason for allowing memory for that but not allowing memory for recent utterances? Conceptually, I understand the motivation for wanting to explore multiple segmentations (and I think it's a good idea - I'm actually not sure why the algorithm here is limited to 2) - I'm just not sure it's quite fair to criticize other models for essentially allowing more memory for one thing when the model here allows more memory for another.

I was a little confused about how the greedy subtractive segmentation worked in section 3.2.  At first, I thought it was an incremental greedy thing - so if your utterance was "syl1 syl2 syl3", you would start with "syl1" and see if that's in your lexicon; if not, try "syl1 syl2", and so on. But this wouldn't run into ambiguity then: "...whenever multiple words in the lexicon could be subtracted from an utterance, the entry with the highest score will be deterministically used". So something else must be meant. Later on when the beam search is described, it makes sense that there would be ambiguity - but I thought ambiguity was supposed to be present even without multiple hypotheses being considered.

The "Trust" feature described in 3.3 seemed like an extra type of knowledge that might be more easily integrated into the existing counts, rather than added on as an additional binary feature.  I get that the idea was to basically use it to select the subset of words to add to the lexicon, but couldn't a more gradient version of this implemented, where the count for words at utterance boundaries gets increased by 1, while the count for words that are internal gets increased by less than 1? I guess you could make an argument either way about which approach is more naturally intuitive (i.e., just ignore words not at utterance boundaries vs. be less confident about words not at utterance boundaries).

I think footnote 7 is probably the first argument I'm seen in favor of using orthographic words as the target state, instead of an apology for not having prosodic words as the target state. I appreciate the viewpoint, but I'm not quite convinced that prosodic words wouldn't be useful as proto-lexicon items (ex: "thatsa" and "couldI" come to mind). Of course, these would have to be segmented further eventually, but they're probably not completely destructive to have in the proto-lexicon (and do feel more intuitively plausible as an infant's target state).

In Table 1, it seems like we see a good example of why precision and recall may be better than hit (H) rate and false alarm (FA) rate: The Syllable learner (which puts a boundary at every syllable) clearly oversegments and does not achieve the target state, but you would never know that from the H and FA scores.  Do we get additional information from H & FA that we don't get from precision and recall? (I guess it would have to be mostly from the FA rate, since H = recall?)

I thought seeing the error analyses in Tables 2 and 3 was helpful, though I was a little surprised Table 3 didn't show the breakdown between undersegmentation and oversegmentation errors, in addition to the breakdown between function and content words.  (Or maybe I just would have liked to have seen that, given the claim that early errors should mostly be undersegmentations. We see plenty of function words as errors, but how many of them are already oversegmentations?)

Next time on 4/30/13 @ 2pm in SBSG 2200 = Lignos 2012

Thanks to everyone who joined our meeting this week, where we had a very helpful discussion about the empirical basis and learning model in Martin 2011, as well as some ideas for how to extend this model in interesting ways. Next time on Tuesday April 30 @ 2pm in SBSG 2200, we'll be looking at an article that develops an algorithmic model of word segmentation, using experimental evidence from infant learning to ground itself:

Lignos, C. 2012. Infant Word Segmentation: An Incremental, Integrated Model. Proceedings of the 30th West Coast Conference on Formal Linguistics, ed. Nathan Arnett and Ryan Bennett, 237-247. Somerville, MA: Cascadilla Proceedings Project.

http://www.socsci.uci.edu/~lpearl/colareadinggroup/readings/Lignos2012_InfantWordSeg.pdf

See you then!

-Lisa

Some thoughts on Martin (2011)

I really liked how compact this paper was - there was quite a bit of material included without it feeling like a part of the discussion was missing. I appreciated the connections made between the implementation of the model and the cognitive learning biases that implementation represented.

As a researcher with a soft spot for empirically-grounded modeling, I was also pleased to see the connections to English and Navajo phonotactic variation. (I admit, I would have liked a bit less abstraction for some of the modeling demonstrations once the basic principle had been illustrated, but that's probably why it was a 20 page paper instead of a 40 page paper.)  One of the things that really struck me was how much the MaxEnt framework discussed seemed similar to hierarchical Bayesian models (HBMs) - I kept wanting to map the different frameworks to each other (prior = prefer simpler grammars, likelihood = maximize probability of input data, etc.). It seemed like the MaxEnt framework included an overhypothesis (dislike geminate consonants in general [structure-blind]), and then some more specific instantiations (dislike them within words, but don't care about them as much across words [structure-sensitive]).  This would be the "leaking" that the title refers to - the leaking of specific constraints back up to the overhypothesis. This also ties into the idea on p.763 where Martin mentions that structure-blind constraints may be a hold-over from very early learning (Perfors, Tenenbaum and colleagues often talk about the "blessing of abstraction" for overhypotheses, where the more abstract thing can be learned earlier because it's instantiated in so many things. And so perhaps the overhypothesis is reinforced more than any individual instantiation of it, making it more resistant to change later on.) But instead of having them arranged in this kind of hierarchy (or maybe it's more like two factors interacting - (1) geminate preference + (2) within vs. across words?), the constraints were specified explicitly by the modeler. This is great first step to show that all of these constraints are needed, but it does feel like some more-general representation is missing.

I also thought it was a very interesting hypothesis that marked forms (i.e., geminates across word boundaries in compounds) persist because new compounds are formed that are not drawn from the existing phonotactic distribution of geminates.  Martin suggests this is because semantic factors play a role in compound formation, and they have nothing to do with phonotactics. This seems reasonable, but really, the main empirical finding is simply that something besides the existing phonotactic distribution matters.  Something I would have liked to have seen was how far away the new-compound-formation distribution has to be from the existing distribution in order for these forms to persist - in the demonstration Martin does, this distribution is simply 0.5 (half the time new compounds contain geminates).  But one might easily imagine that new compounds are formed from the existing words in the lexicon, and this might be less than 0.5, depending on the actual words in the lexicon.  Do these forms persist if the new-compound-formation distribution is 0.25 geminates, for instance?

Specific comments:

Section 4: I was unsure how to map the learning model to Universal Grammar (UG), especially since Martin makes it a point to connect the model to UG in the first paragraph here. I think he's saying that the "entanglement" of the constraints (which reads to me like overhypothesis + more specific constraints) is not part of UG.  This is fine, if we think about the structure of overhypotheses as general not being a UG thing. But what does seem to then be a UG thing is what the overhypothesis actually is - in this case, it's knowing that geminates are a thing to pay attention to, and that word structure may matter for them. (In the same way, if we think of UG parameters as overhypotheses, the UG part is what the content of the overhypothesis/parameter is, not the fact that there is actually an overhypothesis.) So would Martin be happy to claim that both the "entanglement" structure and the content of the constraints themselves aren't part of UG?  If so, where does the focus on geminates and word structure come from?  Does the attention to geminates and word structure logically arise in some way?

Section 4.2, p.760, discussing the tradeoff between modeling the data as accurately as possible and having as general a grammar as possible: This tradeoff is completely fine, of course, as that's exactly the sort of thing Bayesian models do.  But Martin also equates a "general" grammar to a uniform distribution grammar - I was trying to think if that's the right connection to draw. In one sense, it may be, if we think about how much data each grammar is compatible with - a grammar with a uniform distribution doesn't really give much importance to any of the constraints (if I'm understanding this correctly), so it would presumably be fine with the entire set of input data. This then makes it more general than grammars that do place priority on some constraints, and so don't allow in some of the data.

Section 4.2, p.760: The learning described, where the constraints are assigned arbitrary weights, and then the constraint weights are updated using the SGA update rule, reminds me a lot of neural net updating.  How similar are these? On a more specific note, I was trying to figure out how to interpret C_i(x) and C_i(y) in the rule in (7) - are these simply binary (1 or 0)? (This would make sense, since the constraints themselves are things like "allow geminates".)

Some thoughts on Mohamed et al. (2011)

This brief article focuses on the principles of how deep belief networks (DBN) achieve good speech recognition performance, while glossing over many of the details. Therefore, it seems to me that this article can be approached with two levels of rigor. For the novice with a more leisurely approach, the article provides some very clear and concise descriptions of what a DBN model has that sets it apart from other types of competing models. For the experimentalist who wants to replicate the actual models used in the paper, good luck. Nevertheless, there are more extensive treatments of the technical details elsewhere in the literature, and even the novice will probably wish to consult some of these sources to appreciate the nuances in the method that receive short shrift here.

The three main things that make DBNs an attractive modeling choice:

1) They are neural networks. Neural networks are an efficient way to estimate the states of hidden Markov models (HMM), compared to mixture of Gaussians.

2) They are deep. More hidden layers allows for more complicated correlations between the input and the model states, so more structure can be extracted from the data.

3) They are generatively pre-trained. This is a neat pre-optimization algorithm that places the model in a good starting point for back-propagation to discover local maxima. Without this pre-optimization, models with many hidden layers are unlikely to converge on a good solution.

The idea of using a "generative" procedure to pre-optimize a system seems like it may have immediate applicability for psychologists and linguists who also study "generative" phenomena. After all, the training algorithm is even called the "wake-sleep" algorithm, where the model generates "fantasies" during its pre-training. While the parallels are certainly interesting, without appreciating the details of the algorithm, it's difficult to know how deep these similarities actually are. In his IPAM lecture, Hinton notes that while some neuroscientists such as Friston do believe the model is directly applicable to the brain, he remains skeptical.

Ignoring psychological applications for the moment, I'm still left wondering about how "good" DBNs actually perform. The best performing model in this paper still only achieves a Phoneme Error Rate of 20%, and the variability attributable to feature types, number of hidden layers, or pre-training appears small, affecting performance by only a few percentage points. Again, the evaluation procedure is not entirely clear to me, so it's difficult to know how these values translate into real-world performance. I would believe that current voice-recognition technology does much better than 80%, and in far more adverse conditions than those tested here. It was also interesting to note that DBNs appear to have a problem with ignoring irrelevant input.

The multidimensional reduction visualization (t-SNE) was pretty cool, plotting data points that are near to each other in high-dimensional space close together in 2-dimensional space. It would be nice to have some way to quantify the revealed structures using this visualization technique. The distinctions between Figs 3-4 and 7-8 are visually obvious, but I think we just have to take the authors' at their word when they describe differences in Figs 5-6. Perhaps another way to visualize the hidden structure in the model, particularly comparing different individual hidden layers as in Figs 7-8, would be to provide dendograms that cluster inputs based on the hidden vectors that are generated.

Overall, DBNs seem like they can do quite a bit of work for speech recognition systems, and the psychological implications of these models seem to be promising avenues for research. It would be really nice to see some more elaborate demonstrations of DBNs in action.

Next time on 4/16/13 @ 2pm in SBSG 2200 = Martin 2011

For this quarter, our reading group will be coinciding with Psych245A, the graduate course "Computational Models of Language Learning", and so we'll be meeting Tuesdays at 2pm on SBSG 2200.  Our complete reading group schedule (with specific dates) is now available at 

http://www.socsci.uci.edu/~lpearl/colareadinggroup/schedule.html

So, let's get kicking!  For our first meeting on Tuesday April 16 @ 2pm in SBSG 2200, we'll be looking at an article that investigates how biases in the word forms of a language could result from properties of the algorithms used by individual learners, with concrete examples from English and Navajo:

Martin, A. 2011. Grammars leak: Modeling how phonotactic generalizations interact within the grammar. Language, 87(4), 751-770.

http://www.socsci.uci.edu/~lpearl/colareadinggroup/readings/Martin2011_PhonotacticGramLeak.pdf

See you then!

See you in the spring!

Thanks so much to everyone who was able to join us for our feisty and fantastic discussion today, and to everyone who's joined us throughout the winter quarter! The CoLa Reading Group will resume again in the spring quarter, where it will coincide timing-wise with the seminar course "Computational Models of Language Learning", taught by me and Mark Steyvers. 

See you then!

Some thoughts on Frank et al. (2012)

I definitely appreciate that the authors are trying to explore provocative ideas - specifically, it seems like they want to claim that hierarchical structure isn't required for language use (defined as production, comprehension, and acquisition). However, from what I can tell, the evidence they present is more about how they can lessen the amount of hierarchical structure required for any given aspect of language use, rather than eliminate it altogether (e.g., section 3b: "...evidence for the primacy of sequential processing...", not "sequential processing is the only type of processing going on"; section 4c: "...while a syntactic structure is only assigned at a later stage...", not "while a syntactic structure is never assigned"; section 5a: "...reanalyses that deemphasize hierarchical structure...", not "reanalyses that eliminate hierarchical structure"). This then seems to play into the continuing debate going on about exactly what kind of structure is required for language (for example, generativist representations vs. constructionist representations). And this debate isn't particularly new, as far as I'm aware.

The basic issue that kept occurring to me as I read this was that hierarchy, in its most basic conception, is the idea that you have units that are made out of other units (which can be made out of other units, etc.). Constructing these hierarchical units (or constituents, if you prefer) is one idea of how you derive meaning from a sequence of word-units, for example.  As far as I can tell, I don't think the authors would argue against this view of hierarchical structure. (And if they did, it's unclear to me what alternative they would propose to replace it.)

Also, the authors don't appear to be unhappy with the idea that hierarchy is part of the complete knowledge representation that's built for language (and so would therefore be the target of acquisition). Their claim seems to be more about how we don't need to use all that hierarchical knowledge all the time when we're producing or comprehending language (they try to claim this for acquisition as well, but that seems more tenuous if we think the point of acquisition is to acquire the target knowledge representation).  If we focus just on production and comprehension, I think they still need to be more explicit about how to get from a sequence of word units to the complex knowledge representation an entire sentence corresponds to (they say something like this in section 5c: "...if subjects are motivated to read for comprehension, if sentence meaning depends on the precise (hierarchical) sentence structure.."). They present a kind of idea about this with the parallel streams in Figure 1, but I think this doesn't really take care of the underlying problem of constructing compositional (and non-compositional) meaning (more on this below).

Some more targeted thoughts:

The issue of translating between linear pieces and the entire meaning of a sentence appears right at the beginning, with example 2 in particular. While it's true that the pieces can be chunked this way, if you don't have some kind of additional relationship between "sentences" and "can be analysed" (for example, IP if we think of these pieces as NP and VP), how do you know how to put them together to get the larger meaning corresponding to "sentences can be analysed"? And if you do have that relationship somewhere, isn't that equivalent to having hierarchical structure, since these two pieces are subsumed under a larger unit (called IP above)?

In section 2, they mention the idea that "the mechanisms employed for language learning and use are likely to be fundamentally sequential in nature, rather than hierarchical". I have no problem with talking about the mechanism this way - in fact, that makes perfect sense (incremental processing, etc.).  But isn't the mechanism distinct from the knowledge representations being manipulated?  And that's the part whose structure people generally argue about?

In section 3c, where they talk about some of the models that can learn different aspects of syntax by just using sequential information, do they believe that the target knowledge for these structures doesn't involve hierarchy at all? If they believe the target knowledge does in fact involve hierarchy, then this falls back onto the mechanism vs. knowledge distinction I mentioned above.  If they instead think there's no hierarchy even in the target knowledge, then I think they run into the basic problem of how you map words to sentential meaning without hierarchy (or dependency relations, etc.).  I think they're aiming towards the former idea where hierarchy is present in the target knowledge (section 4b on combining constructions: "...seems intuitive to regard a combination of constructions as a part-whole relation...").

This then brings me to the parallel sequential streams idea presented in Figure 1.  The fact that the pieces combine into a whole seems to be exactly what hierarchy accomplishes (part-whole relationships, etc).  Beyond this, it seems one thing to slot pieces together in a parallel stream, and another to create a mental model from this (i.e., get the interpretation of the whole meaning once the pieces are composed together in particular ways).


~~~
Also, here's an excellent blog post by someone very knowledgeable who read this article last year, and had some very specific (and similar) things to say about it:
Norbert Hornstein's Faculty of Language: Three psychologists walk into a bar...

Next time on 3/11 @ 2:15 pm in SBSG 2221 = Frank et al. (2012)

Thanks to everyone who joined our meeting this week, where we had a very helpful discussion about some of the ideas in Thiessen & Pavlik (2012 forthcoming)! Next time on Monday March 11 @ 2:15pm in SBSG 2221, we'll be looking at an article that investigates the necessity of hierarchical structure in language comprehension, production, and acquisition:

Frank, S., Bod, R., & Christiansen, M. 2012. How hierarchical is language use? Proc. R. Soc. B, published online 12 September 2012. doi: 10.1098/rspb.2012.1741.

http://www.socsci.uci.edu/~lpearl/colareadinggroup/readings/FrankEtAl2012_HierarchicalLangUse.pdf

See you then!

-Lisa

Some thoughts on Thiessen & Pavlik 2013

I found this paper a very enjoyable read, and I like very much that it's looking at the building blocks of distributional learning.  This seems like the next step forward - we want to know not just that statistical/distributional learning works, but also what the underlying cognitive pieces are that make it work. It's a nice demonstration of a particular story of how cognitive pieces could fit together and make distributional learning work for a few different language acquisition tasks, and it definitely aims to be an algorithmic-level ("mechanistic") account of this process. One of the things that was really good is how clear the authors are that this is only one story - that is, it's an existence proof that this account could work.  It doesn't preclude other accounts, but it does shore up support for this account by showing that it does, indeed, work.

The model they propose seems like it can be very prone to initial snowballing, where small initial errors persist and cause larger errors later on.  (This may or may not be a bad thing, if we're concerned with actual human learning.) For example, in the first simulation, they mention how sometimes their bimodal input resulted in a unimodal representation, due to exactly this kind of thing. Also, it did seem like there were a fair number of free parameters involved - of course, the nice thing is that some of those parameters have explanatory power, since we can manipulate them to get different qualitative learning effects. Aiming for qualitative patterns rather than exact behavioral matches seems exactly right, though - there are other factors contributing to the observed output behavior, and the authors (quite reasonably) only modeling some of them.

Something else notable about this model is that it's geared only towards tasks that involve abstraction.  Now, of course, many acquisition tasks are about some kind of abstraction, but some aren't (like word segmentation) - so it's worth remembering that even if this is how (some kinds of) distributional learning are implemented, we still need some explanation for how other non-abstraction tasks are accomplished.   I also like how much they grounded the underlying cognitive pieces of their model in existing models of (long-term) memory - this does my empirical heart good.

Some more targeted thoughts:

I like how they pointed out on p.3 that statistical learning doesn't just have to be about transitional probabilities. Sometimes, these really get equated, and it's a little unfair to the enterprise of statistical learning to talk about it as if it's just dealing with conditional relations. (Of course, much of the experimental work looking at children's inference capabilities involve testing conditional relationships, and many computational models assume conditional relationship tracking abilities.)

The discussion of making inferences from exemplars on p.4 seemed a little simplified to me.  For example, while I can imagine that it's often the case that exemplars occurring more frequently will be weighted more than exemplars occurring rarely, it's not obvious to me that this is always the case. Instead, it seems like it would depend on the learner's hypothesis space.  For instance, in a subset-superset hypothesis space, one counter-example seems like it could be very heavily weighted, even if it occurs rarely.  As another example, the authors talk about how exemplar similarity depends "at least in part upon the variability of the exemplars in the input set". I could imagine that this is true, but I think it also depends on the learner's biases about the hypothesis space - in effect, learner-subjective variability rather than objective variability.

The authors mention on p.7 that they selected the specific linguistic tasks they did because language is a domain where domain-specific mechanisms have often been argued to be at work. I wonder if domain-specific mechanisms have been proposed for the specific linguistic tasks they chose, though - it seems like the type of mechanism proposed depends very much on the task.  So, if the authors want to argue that their results show domain-specific mechanisms aren't needed, they do need to address the specific problems where domain-specific mechanisms have been proposed. It wasn't clear to me that this was done, which makes that argument a little weaker to me.

I thought the ability to explain why variable contexts facilitate the learning of phonetic distinctions (basically, due to having a holistic representation of input exemplars) was really excellent. In effect, the "irrelevant" part of the representation helps keep the "relevant" part distinct. This really argues for not just context-sensitive storage of data from the input, but holistic storage. And this also ties into the idea that minimal pairs are probably helpful to linguists, but not to children.

The basic components of the distributional statistical learning process seem quite reasonable: similarity-based activation of prior memories, strength-based learning of features, abstraction of irrelevant features, and memory decay.  The second and third components do implicitly assume that the learner has a reasonable set of features to begin with, though. This is a non-trivial assumption, especially when you start thinking about the hypothesis space of possible features.  For example, this shows up in simulation 1, where only certain phonetic features are picked out as even in the hypothesis space to begin with.

The effectiveness of the learner really comes from being able to compare across exemplars, which means particular modeling assumptions - such as assuming the learner is memoryless or that the learner is limited to one exemplar at a time - become not so harmless.

I thought it was slightly unfair on p.38 to differentiate the current model from prior models by saying prior models "have been focused on acquiring relatively domain-specific kinds of knowledge...meaning they are not easily applied to other domains". It seemed to me that the current model can only be applied to different domains because the domain-specific knowledge has been built in as part of the feature descriptions. So maybe the point was simply that prior models didn't try to separate out the more-general components from the task-specific components.

I really appreciated the discussion on p.40 about what different parameter values for different learning tasks might actually imply for children. I don't think modelers are always so careful in evaluating what the model parameters & parameter values mean.

A nice aspect of this model discussed in Appendix A is how it can basically recover from spurious examples in the input. Because actual exemplars are kept around (in addition to increasingly more abstract interpretations), one spurious example and its created interpretation can be overrun by lots of non-spurious (i.e., good) exemplars.

Next time on 2/25/13 @ 2:15pm in SBSG 2221 = Thiessen & Pavlik (forthcoming)

Thanks to everyone who joined out meeting this week, where we had a very enlightening discussion about some of the ideas in Stabler (2009b)! Next time on Monday February 25 @ 2:15pm in SBSG 2221, we'll be looking at an article that investigates a single computational learning framework (and general distributional learning strategy) for multiple language learning tasks:

Thiessen, E., & Pavlik, P. 2012 forthcoming. iMinerva: A Mathematical Model of Distributional Statistical Learning. Cognitive Science. 

http://www.socsci.uci.edu/~lpearl/colareadinggroup/readings/ThiessenPavlik2012_iMinerva.pdf

See you then!

Some thoughts on Stabler (2009b)

One of the things I really appreciated about this article was the clear intention to connect the kind of computational models & problems learnability researchers typically worry about with the kind of realistic language acquisition and language use problems that linguistic & psychology researchers typically worry about. A nice example of this was the connection to syntactic bootstrapping, which showed up in some of the later sections.  I also found myself thinking a few times about the connection between some of these ideas and the issue of language evolution (more on this below), though I suspect this often comes up whenever language universals are discussed.

More targeted thoughts:

The connection with language evolution: I first thought about this in the introduction, where Stabler talks about the "special restrictions on the range of structural options" and the idea that some of the language universals "may guarantee that the whole class of languages with such properties is 'learnable' in a relevant sense." The basic thought was that if the universals didn't help language be learned, they probably wouldn't have survived through the generations of language speakers.  This could be because those universals take advantage of already existing cognitive biases humans have for learning, for example.

In section 1, Stabler mentions that it would be useful to care about the universals that apply before more complex abstract notions like "subject" are available. I can see the value of this, but I think most ideas about Universal Grammar (UG) that I'm aware of involve exactly these kind of abstract concepts/symbols.  And this makes a little more sense once we remember that UG is meant to be (innate) language-specific learning biases, which would therefore involve symbols that only exist when we're talking about language. So maybe Stabler's point is more that language universals that apply to less abstract (and more perceptible) symbols are not necessarily based on UG biases.  They just happen to be used for language learning (and again, contributed to how languages evolved to take the shape that they do).

I'm very sympathetic to the view Stabler mentions at the end of section 1 which is concerned with how to connect computational description results to human languages, given the idealized/simplified languages for which those results are shown.

I like Stabler's point in section 2 about the utility of learnability results, specifically when talking about how a learner realizes that finite data does not mean that the language itself is finite. This connects very well to what I know about the human brain's tendency towards generalization (especially young human brains).

Later on in section 2, I think Stabler does a nice job of explaining why we should care about results that deal with properties in languages like reversibility (e.g., if it's known that the language has that property, the hypothesis space of possible languages is constrained - coupled with a bias for compact representations, this can really winnow the hypothesis space). My take away from that was that these kind of results can tell us about what kind of knowledge is necessary to converge on one answer/representation, which is good. (The downside, of course, is that we can only use this new information if human languages actually have the properties that were explored.)  However, it seems like languages might have some of these properties, if we look in the domain of phonotactics.  And that makes this feel much more relevant to researchers interested in human language learning.

In section 3, where Stabler is discussing PAC learning, there's some mention of the time taken to converge on a language (i.e., whether the learner is "efficient").  One formal measure of this that's mentioned is polynomial time. I'm wondering how this connects to notions of a reasonable learning period for human language acquisition. (Maybe it doesn't, but it's a first pass attempt to distinguish "wow, totally beyond human capability" from "not".)

I really liked the exploration of the link between syntax and semantics in section 4. One takeaway point for me was evidence in the formal learnability domain for the utility of multiple sources of information (multiple cues). I wonder if there's any analog for solving multiple problems (i.e., learning multiple aspects of language) simultaneously (e.g., identifying individual words and grammatical categories at the same time, etc.). The potential existence of universal links between syntax and semantics again got me thinking about language evolution, too. Basically, if certain links are known, learning both syntax and semantics is much easier, so maybe these links take advantage of existing cognitive biases. That would then be why languages evolved to capitalize on these links, and how languages with these links got transmitted through the generations.

I also liked the discussion of syntactic bootstrapping in section 4, and the sort of "top-down" approach of inferring semantics, instead of always using the compositional bottom-up approach where you know the pieces before you understand the thing they make up. This seems right, given what we know about children's chunking and initial language productions.

Next time on 1/28/13 @ 2:15pm in SBSG 2221 = Stabler 2009b

Thanks to everyone who joined out meeting this week, where we had a very interesting discussion about some of the ideas in Stabler (2009)! Next time on  Monday January 28 @ 2:15pm in SBSG 2221, we'll be looking at another article by Stabler. This time, it's one that reviews computational approaches to understanding language universals:

Stabler, E. 2009b. Computational models of language universals: Expressiveness, learnability and consequences. Revised version appears in M. H. Christiansen, C. Collins, and S. Edelman, eds., Language Universals, Oxford: Oxford University Press, 200-223. Note: Because this is a non-final version, please do not cite without permission from Ed Stabler.

http://www.socsci.uci.edu/~lpearl/colareadinggroup/readings/Stabler2009_CompModLangUni.pdf

See you then!
-Lisa

Some thoughts on Stabler (2009)

One of the things I most enjoyed about this paper was the way Stabler gives the intuitions behind the different approaches - in many cases, these are some of the most lucid descriptions I've seen about these different mathematical techniques. I also really appreciated the discussion about model selection - it certainly seems true to me that model selection is what many theoretical linguists are thinking about when they discuss different knowledge representations. Of course, this isn't to say that parameter setting once you know the model isn't worthy of investigation (I worry a lot about it myself!). But I also think it's easier to use existing mathematical techniques to investigate parameter setting (and model selection, when the models are known), as compared to model generation.


Some more targeted thoughts below:

I really liked the initial discussion of "abstraction from irrelevant factors", which is getting at the idealizations that we (as language science researchers) make. I don't think anyone would argue that it's necessary to do that to get anything done, but the fights break out when we start talking about the specifics of what's irrelevant. A simple example would be frequency - I think some linguists would assume that frequency's not part of the linguistic knowledge that's relevant for talking about linguistic competence, while others would say that frequency is inherently part of that knowledge since linguistic knowledge includes how often various units are used.

I thought Stabler made very good points about the contributions from both the nativist and the empiricist perspectives (basically, constrained hypothesis spaces for the model types but also impressive rational learning abilities) - and he did it in multiple places, highlighting that both sides have very reasonable claims.

The example in the HMM section with the discovery of implicit syllable structure reminded me very much of UG parameter setting.  In particular, while it's true that the learner in this example has to discover the particulars of the unobserved syllable structure, there's still knowledge already (by the nature of the hidden units in the HMM) that there is hidden structure to be discovered (and perhaps even more specific, hidden syllabic structure).  I guess the real question is how much has to be specified in the hidden structure for the learner to succeed at discovering the correct syllable structure - is it enough to know that there's a level above consonants & vowel?  Or do the hidden units need to specify that this hidden structure is about syllables, and then it's just a question of figuring out exactly what about syllables is true for this language?

I was struck by Stabler's comment about whether it's methodologically appropriate for linguists to seek grammar formalisms that guarantee that human learners can, from any point on the hypothesis space, always reach the global optimum by using some sort of gradient descent. This reminds me very much of the tension between the complexity of language and the sophistication of language learning. First, if language isn't that complex, then the hypothesis space de facto probably can be traversed by some good domain-general learning algorithms. If, however, language is complex, the hypothesis space may not be so cleanly structured.  But, if children have innate learning biases that guide them through this "bumpy" hypothesis space, effectively restructuring the hypothesis space to become smooth, then this works out. So it wouldn't be so much that the hypothesis space must be smoothly structured on its own, but rather that it can be perceived as being smoothly structured, given the right learning biases. (This is the basic linguistic nativist tenet about UG, I think - UG are the biases that allow swift traversal of the "bumpy" hypothesis space.)

I also got to thinking about the idea mentioned in the section on perceptrons about how there are many facts about language that don't seem to naturally be Boolean (and so wouldn't lend themselves well to being learned by a perceptron). In a way, anything can be made into a Boolean - this is the basis of binary decomposition in categorization problems.  (If you have 10 categories, you first ask if it's category 1 or not, then category 2 or not, etc.) What you do need is a lot of knowledge about the space of possibilities so you know what yes or no questions to ask - and this reminds me of (binary) parameter setting, as it's usually discussed by linguists. The child has a lot of knowledge about the hypothesis space of language, and is making decisions about each parameter (effectively solving a categorizing problem for each parameter - is it value a or value b?, etc.). So I guess the upshot of my thought stream was that perceptrons could be used to learn language, but at the level of implementing the actual parameter setting.

It was very useful to be reminded that the representation of the problem and the initial values for neural networks are crucial for learning success. This of course implies that the correct structure and values for whatever language learning problem must be known a priori (which is effectively a nativist claim, and if these values are specific to language learning, then a linguistic nativist claim). So, the fight between those who use neural networks to explain language learning behavior and those who hold the classic ideas about what's in UG isn't about whether there are some innate biases, or even if those biases are language-specific - it may just be about whether the biases are about the learning mechanism (values in neural networks, for example) or about the knowledge representation (traditional UG biases, but also potentially about network structure for neural nets).

Alas, the one part where I failed to get the intuition that Stabler offered was in the section on support vector machines.  This is probably due to my own inadequate knowledge of SVMs, but given how marvelous the other sections were with their intuitions, I really found myself struggling with this one.

Stabler notes in the section on model selection that model fit cannot be the only criterion for modeling success, since larger models tend to fit the data (and perhaps overfit the data) better than simpler models. MDL seems like one good attempt to deal with this, since it has a simple encoding length metric which it uses to compare models -  encoding not just the data, based on the model, but also the model itself. So, while a larger model may have a more compact data encoding, its larger size counts against it.  In this way, you get some of that nice balance between model complexity and data fit.

Winter meeting time set & Jan 14 = Stabler 2009 @ 2:15pm in SBSG 2221

Based on the responses, it seems like Mondays at 2:15pm will work best for everyone's schedules this quarter. Our complete schedule (with specific dates) can now be seen at

http://www.socsci.uci.edu/~lpearl/colareadinggroup/schedule.html

So, let's get kicking!  For our first meeting on Monday January 14 @ 2:15pm in SBSG 2221, we'll be looking at an article that surveys several mathematical approaches to language learning, as well as the assumptions inherent in these various approaches.

Stabler, E. 2009. Mathematics of language learning. Revised version appears in Histoire, Epistemologie, Langage, 31, 1, 127-145. Note: Since this a non-final version, please do not cite without permission from Ed Stabler.

http://www.socsci.uci.edu/~lpearl/colareadinggroup/readings/Stabler2009_MathLangLearn.pdf

See you then!