Diplomarbeit Nr. 3076 The Morphological Component of a Joint Morphological-Distributional Class Language Model Thomas Müller Studiengang: Prüfer: begonnen am: beendet am: CR-Klassifikation: Betreuer: Informatik Hinrich Schütze, Bernhard Mitschang Hinrich Schütze 01.09.2010 03.03.2011 I.2.7, I.5.3 Institut für Maschinelle Sprachverarbeitung Universität Stuttgart Universitätsstraße 38 D - 70569 Stuttgart Abteilung Theoretische Computerlinguistik Erklärung Hiermit versichere ich, diese Arbeit selbstständig verfasst und nur die angegebenen Quellen benutzt zu haben. Stuttgart, 03.03.2011 Unterschrift: Abstract Modeling of out-of-vocabulary (OOV) words, i.e. words that do not occur in the training corpus but in the natural language processing (NLP) task at hand, is a challenging problem of statistical language modeling. We empiri- cally investigate the relation between word context, word class and word mor- phology in English and present a class-based language model, which groups rare words of similar morphology together. The model improves the predic- tion of words after histories containing out-of-vocabulary words. The mor- phological features used are obtained without the use of labeled data, but produce a number of syntactically and even semantically related clusters. The overall perplexity improvement achieved by our model is 4% compared to a state of the art Kneser-Ney model and 81% on unknown histories. We conclude that the usage of morphological features in English language mod- eling is worthwhile. Contents 1 Introduction 8 2 Related Work 11 3 Feature Design and Clustering 14 3.1 Morphological Features . . . . . . . . . . . . . . . . . . . . . . 14 3.1.1 The Reports Algorithm . . . . . . . . . . . . . . . . . . 14 3.1.2 Affixation-based Features . . . . . . . . . . . . . . . . 16 3.1.3 Stem-based Features . . . . . . . . . . . . . . . . . . . 17 3.2 Shape-like Features . . . . . . . . . . . . . . . . . . . . . . . . 17 3.2.1 Capitalization . . . . . . . . . . . . . . . . . . . . . . . 17 3.2.2 Special Characters . . . . . . . . . . . . . . . . . . . . 18 3.2.3 Word Length . . . . . . . . . . . . . . . . . . . . . . . 19 3.3 K-Means Clustering . . . . . . . . . . . . . . . . . . . . . . . . 19 3.4 Evaluation Criteria . . . . . . . . . . . . . . . . . . . . . . . . 22 3.4.1 Pair-based Criteria . . . . . . . . . . . . . . . . . . . . 23 3.4.2 Information-Theory-based Criteria . . . . . . . . . . . 25 3.4.3 Limitations of Contingency-Table-based Criteria . . . . 29 3.5 Experimental Setup and Results . . . . . . . . . . . . . . . . . 29 4 A Morphological Language Model 33 4.1 Foundations of Language Modeling . . . . . . . . . . . . . . . 33 4.1.1 The Maximum Likelihood Estimate . . . . . . . . . . . 34 4.1.2 Kneser-Ney Discounting . . . . . . . . . . . . . . . . . 34 4.1.3 Class-based Models . . . . . . . . . . . . . . . . . . . . 36 4.1.4 Linear Interpolation . . . . . . . . . . . . . . . . . . . 36 4.1.5 Language Model Evaluation . . . . . . . . . . . . . . . 39 4.2 Design of a Morphological Language Model . . . . . . . . . . . 39 4.2.1 A Preliminary Model . . . . . . . . . . . . . . . . . . . 39 4.2.2 Evaluation of the Preliminary Model . . . . . . . . . . 41 4.2.3 The Final Model . . . . . . . . . . . . . . . . . . . . . 42 4 4.2.4 Evaluation of the Final Model . . . . . . . . . . . . . . 44 5 Conclusion 53 6 Future Work 54 Appendix – Proof N I = V 1 55 Appendix – The Penn Treebank Tagset 56 5 List of Figures 3.1 Example of the transitional probabilities in the word refund- ings indicating “-ing” and “-s” as good suffix candidates. . . 15 3.2 An hierarchical clustering of the training set of section 4. The edge weights correspond to the number of word types of the cluster the edge is pointing to. . . . . . . . . . . . . . . . . . . 21 3.3 The three uppermost layers of the same clustering as in 3.2. 21 3.4 Subtree of the right uppermost child node of the graph shown in 3.2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 3.5 An example demonstrating that the NVI can be greater than 1 for decent clusterings. . . . . . . . . . . . . . . . . . . . . . 28 4.1 Implementation of Bahl’s algorithm. The parameter is the desired maximal distance to the true optimum. . . . . . . . . 38 4.2 A block diagram of the training procedure of the final inter- polated model. Gray nodes are part of the cluster process. . . 43 6 List of Tables 3.1 Performance of the Length feature. . . . . . . . . . . . . . . . 30 3.2 Performance of the feature P 1 . . . . . . . . . . . . . . . . . . 31 3.3 Performance of the stem feature S 2 . . . . . . . . . . . . . . . 31 3.4 Performance of the stem feature P 2 . . . . . . . . . . . . . . . 31 4.1 Segmentation of the WSJ corpus . . . . . . . . . . . . . . . . 41 4.2 Perplexity results for the preliminary model. . . . . . . . . . . 41 4.3 Proportion of dominant POS for types with training set fre- quencies f ∈ { 0 , 1 } and for tokens. V* consists of all verb POS tags. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 4.4 Perplexities over all events of valtst. . . . . . . . . . . . . . . . 44 4.5 Perplexities over all events of the test set. . . . . . . . . . . . 45 4.6 Perplexities over all events of valtst, where w 1 or w 2 are un- known. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 4.7 Perplexities over all events of the test set, where w 1 or w 2 are unknown. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 4.8 Perplexities over histories of valtst that are continued by OOV words for different thresholds. . . . . . . . . . . . . . . . . . . 47 4.9 Perplexities over histories of tst that are continued by OOV words for different thresholds. . . . . . . . . . . . . . . . . . . 47 4.10 List of the 20 best POS-classes for the c P OS model at θ = 1000. 49 4.11 The 20 best clusters of the c ∞ model at θ = 1000, using only the suffix features. . . . . . . . . . . . . . . . . . . . . . . . . 50 4.12 List of more common suffixes for the same model as in 4.11. 51 4.13 The 20 best clusters of the c ∞ model at θ = 1000, using the entire feature vector. . . . . . . . . . . . . . . . . . . . . . . . 52 7 Chapter 1 Introduction Statistical language models are an important part of modern solutions to language processing problems, such as statistical machine translation (SMT), speech recognition (SR) and handwritten text recognition (HTR). Language modeling is tightly related to the question of how much text of a certain language is required to learn and reproduce its general structure. First sta- tistical models were merely able to create sentences that consisted at least in parts of events that were seen before, i.e. occurred in the training text. In 1992, Brown et al. [4] proposed a system that grouped words of similar con- texts and treated them equally. This approach is called class-based language modeling and follows the assumption that small differences between those words are noise in the training text, which should be ignored by the model. In a version of such a system [25], only the left half of the word contexts is used to create this word clustering. In this work, we present a model that could complement the half-context model under the usage of morphological information. Our interest is both of theoretical and practical nature. The theoretical question is, how strongly re- lated are morphology and word context and if the morphological information can be used to improve the performance of a language model by building con- textually related classes. In particular it is investigated, if this can be done, even if the morphological decomposition is learned and performed completely unsupervised and thus is not nearly as accurate as the analysis of a human expert. However, not all our features are morphological. Shape-like features as e.g. capitalization and the occurrence of special characters are also used. The practical problem we attempt to solve is related to words that ap- pear in the language processing task at hand, but not in the training set, so called out-of-vocabulary (OOV) words. Especially for productive languages it is often necessary to at least reduce the number of OOVs. Previous work on morphological classes in English has not been able to show noticeable im- 8 provements in language model performance. In this work, class models based on the ideas of Brown et al. are used to tackle the problem. Context-based class models cannot naturally integrate OOVs, as their context is unknown and they therefore cannot be assigned to the word classes. We propose a pure morphological clustering, which can be easily extended to unknown words. A critical remaining question is, what should be considered a good clus- tering? Of course, the best clustering is the one that – integrated in a class- based language model – yields the highest performance in a language pro- cessing task. However, this is a very indirect method, as this performance depends on other components and parameters of the language model and the entire system. Furthermore, it is an unfeasible way of evaluating the cluster quality. From the point of view of linguistics one could say that the goal of a language model is to produce syntactically and semantically correct sen- tences. Traditionally, words are assigned to word classes (part-of-speech (POS) classes) which are then used as tokens in formal grammars. So by assigning words to classes, such as verbs, nouns, adjectives and so on, one represents this syntactical concept in the language model. But is grouping the words corresponding to their POS classes the most promising approach? No, it is not. We already said that the semantical correctness of a sentence is also important. That is, grouping e.g. all past participles into one cluster is an over-generalization, as most verbs only occur with a small number of nouns. However, we have to keep in mind what kind of information we have available. And we assume that shape and morphology are mostly syntax- related. A further aspect is that we evaluate our model on a big corpus with a high number of OOV words. It is possible to accurately POS-tag such a corpus, but there are no precise tools available to do a semantic classifica- tion by word. Thus, throughout the feature design we use the similarity to POS-classes as an indicator of cluster quality. However, it is important to point out that unsupervised POS clustering using only morphological and no form of contextual information is still a virtually impossible task. There is no way to learn that “the” and “a” are similar or that “kid” and “kids” share a similar relation as “child” and “children”. So we have to assume that for a large number of the word types of a corpus no correct clustering will be achieved. In chapter 3 we develop a number of morphological and shape-like fea- tures, which we later use to calculate word clusterings and evaluate by com- parison with POS classes. As cluster evaluation is an important and recently controversially discussed topic [17,21,22] we also compare and discuss a num- ber of older and newer cluster criteria. In chapter 4, we use the generated morphological classes to define and 9 evaluate a morphological language model and discuss particularly the per- formance on OOV words. The rest of this diploma thesis is structured as follows: in chapter 2 we discuss other morphological language models that can be found in the related literature. Chapter 5 summarizes our conclusions and in chapter 6 we discuss some ideas for future research. 10 Chapter 2 Related Work A group of morphological language models presented in recent papers can be summarized under the name of factored language models [3]. Factored language models replace words by sequences of factors or features and ap- ply statistical language modeling to these feature sequences. Especially for highly inflecting language as Finnish [7], Turkish [7, 30] and Arabic [7, 28] or compounding languages like German [2] this principle has been applied to re- duce the number of out-of-vocabulary words by replacing words by sequences of morphemes. Usually the order of the used n-grams has to be increased to compensate the larger sequences, which increases the number of parameters of the model. However, logistical knowledge can be used to reduce the num- ber of dependencies and parameters. Ghaoui et al. [10] for example divide every word w into a root r and a derivation rule g . This procedure leads to a statistical model of the form 1 : P ( w n | w 1 . . . w n − 1 ) = P (( r n , g n ) | ( r 1 , g 1 ) . . . ( r n − 1 , g n − 1 )) = P ( g n | r 1 g 1 . . . r n − 1 g n − 1 r n ) · P ( r n | r 1 g 1 . . . r n − 1 g n − 1 ) They then apply two constraints to drastically reduce the number of param- eters, namely that the root r n is independent of the preceding rules and that the derivation rule g n only depends on the types T of the preceding roots. By strictly dividing semantic and syntactic concepts, the statistical model can be approximated as: P ( w n | w 1 . . . w n − 1 ) ≈ P ( g n | T ( r 1 ) g 1 . . . T ( r n − 1 ) g n − 1 r n ) · P ( r n | r 1 . . . r n − 1 ) This even reduces the amount of parameters in comparison with a word-based model. 1 This chapter requires knowledge of the basics of language modeling. A short intro- duction is given in section 4.1. 11 Other factored language models use improved back-off techniques like the generalized parallel back-off (GPB) [3] to process the feature sequences more effectively. A standard back-off model uses a temporal back-off order. Consider a trigram model P ( w 3 | w 1 w 2 ) the temporal back-off sequences for this model is P ( w 3 | w 1 w 2 ), P ( w 3 | w 2 ), P ( w 3 ). That means if the frequency of occurrences of w 1 w 2 w 3 is below a certain threshold τ , the model backs off to the sequence w 2 w 3 and potentially to the unigram count of w 3 . However, as in a factored model different features are generated from the same word there is no such natural order. Therefore, a GPB model follows and evaluates different back-off paths and chooses a selection of them to calculate the actual probability. A so called back-off function estimates the best path l i = l 1 · · · l n Bilmes and Kirchhoff present 32 different functions to perform a GPB of a trigram model P ( f | f 1 , f 2 , f 3 ). Two examples are: the path yielding the highest probability (a rather ad-hock approach not further explained by the authors): g 1 ( f 1 , f 2 , f 3 , f ) = argmax ( m 1 ,m 2 ) ∈{ (1 , 2) , (1 , 3) , (1 , 2) } P GBO ( f | f l 1 , f l 2 ) and the path with the highest ratio between the count of the specific event and the number of possibilities to continue the back-off path: g 2 ( f 1 , f 2 , f 3 , f ) = argmax ( m 1 ,m 2 ) ∈{ (1 , 2) , (1 , 3) , (1 , 2) } C ( f m 1 f m 2 f ) |{ f ′ | C ( f m 1 f m 2 f ′ ) > 0 }| The last being related to the Kneser-Ney approach as described in section 4.1.2. Of course, some adequate normalization parameter has to be found to yield a consistent statistical model. However, any further discussion of the topic lies out of our scope. Another branch of methods is more similar to the approach described in this work and founded on class-based language models as introduced by Brown et al. [4]. The original class-based model groups n-grams of similar contexts in order to replace the probability of a certain word transition with the product of a class transition and the word emission probability: P ( w n | w 1 . . . w n − 1 ) = P ( c n | c 1 . . . c n − 1 ) · P ( w n | c n ) (2.1) The emission probability P ( w | c ) can be estimated from a training corpus by dividing the number of tokens of word w by the number of tokens of all words of class c . This estimation corresponds to a maximum likelihood approach. Maltese et al. [15] use a supervised tagger and lemmatizer to define three back-off trigram models on words, part of speech tags (POS) and lemmas. They argue that the tag and lemma models only outperform the word model 12 in cases of bad statistics. Therefore they group the interpolation parameters depending on the order of the n-gram that is used by the word model to estimate the specific sequence. They expect a dominance of the word model for trigrams and a higher contribution of the tag and lemma models, if the word model has to back-off to bi- or even unigrams. The parameters are estimated using the expectation maximization algorithm. In the performed experiments on Italian corpora of different domains, the achieved improve- ment in perplexity 2 on a pure word model drops with a rising number of tokens from -8.8% (1 million tokens) to - 5.6% (107 million tokens). Crespo et al. [6] use a similar approach to build the language model for a Spanish speech recognition system. The language model interpolates between a word model and a pseudo word class model, defined over a knowledge- engineered automatic tagger. They claim to reduce the word error rate (WER) on a Spanish speech recognition task by 26%. Uebler et al. [27] use prefixes and suffixes to define morphological word classes. They apply a morphological decomposition algorithm (MALAGA) that is enhanced by a number of thresholds to segment words into prefix, stem and suffix. In a speech recognition task of German and Italian phrases an improvement of WER by 60% is achieved, compared to a word based bigram approach. However, they do not compare their results with a state-of-the-art language model and omit a discussion of the handling of out-of-vocabulary words and n-gram discounting. 2 Perplexity is a standard measure of language model performance. Consult section 4.1.5 for a definition. 13 Chapter 3 Feature Design and Clustering In this chapter we define a set of features to map words to real number vectors, which then can be clustered by a general clustering algorithm. In this work, the k-means algorithm is used. Different combinations of feature vectors and their corresponding cluster solutions are then evaluated against a reference classification. As there is no single accepted cluster evaluation measure - like for example perplexity in language modeling - we use a couple of criteria, which are also defined and discussed in this chapter. The goal of the features is to group words of a vocabulary into a partition similar to their part-of-speech (POS) classes. In this work, the Penn treebank tagset is used. A short description can be found in appendix B. Our features are divided into morphological and shape-like features. 3.1 Morphological Features Our morphological features are based on flat morphological word decomposi- tions. These decompositions are learned by an unsupervised general purpose algorithm called Reports. 3.1.1 The Reports Algorithm The Reports algorithm [13] learns affixes from an unannotated corpus and segments words into a form suitable for the extraction of affix- and stem- based features 3.1.2. It was designed to be simple and almost parameter-less. While it works decently for languages with a simple morphology (particularly English), modifications are necessary to make it suitable for more compli- cated and productive languages like German, Finnish and Turkish. Such modifications were proposed by Demberg [8]. The basic algorithm works as 14 - refun 1637 - d 1637 - i 612 - n 612 - g 612 - s 12 Figure 3.1: Example of the transitional probabilities in the word refundings indicating “-ing” and “-s” as good suffix candidates. follows. A list of suffix candidates is estimated using the probability that a certain character succeeds a certain string. Low transitional probabilities are good indicators for morpheme borders. A suffix s = xu is a candidate, if in at least 5% of its occurrences as the last part of a word w = vyxu : • P ( x | vy ) < 1 • P ( y | v ) ∼ 1 • The frequency of vy in the corpus is higher than a specific threshold As shown in figure 3.1, “-ing” is a good suffix regarding the word “re- funding”, as the probability P ( i | ref und ) is lesser than one, the probability P ( d | ref un ) is one and “refund” occurs quite often in the corpus. In a prun- ing step all suffixes which are concatenations of two other suffix candidates are removed from the list. In this work, we further reduce the list, to the 100 most frequent candidates. Prefixes are treated in an analog way. The strongest assumption the Report algorithm makes and the reason it performs purely on morphological complex languages is the third condition from the list above, i.e. that the word stem is a regular word of the target language. The segmentation uses a rather rough heuristic: of all possible suffixes of a word, the one minimizing the probability P ( x | wy ) is chosen. There is no mechanism to inhibit the clipping of a suffix. So if the word at hand was “for” and “or” was in the list of suffixes, then the segmentation would be f + or 1 . To reduce the number of conflicts between prefixes and suffixes, in 1 Unless the suffix “r” was in the list and had a lower transition probability 15 the segmentation of a word of the length n , suffixes can only be cut from the n/ 2 + 1 back most characters of the word and prefixes accordingly from the n/ 2+1 foremost characters. This convention allows “being” to be segmented into “be+ing”, but also yields segmentations like “sur+f+ace”, where “ace” is not even in the suffix list, but “-face” is. The Reports algorithm uses a weak definition of affix, e.g., a prefix is defined as a morpheme which occurs at the beginning of a number of words. “work”, for example, most probably appeared in the prefix and the suffix list (consider “workbench” and “homework”) and both cases would be correct, regarding this softer definition of an affix. For the definition of four affixation-based features, we need to segment every word into prefix, stem and suffix, where prefix and suffix are allowed to be the empty word , but the stem is not. From the discussion of the Reports algorithm in the last chapter arise some issues we solve in a clean and simple way. The two problematic cases are inconsistent segmentations like “sur+f+ace” and segmentations with an empty stem like “home+work”. In both cases we prefer the suffix over the prefix and define the none-suffix part of the word as the stem. We do so because we regard the suffix as the most important part of the segmentation. 3.1.2 Affixation-based Features Having a robust and unsupervised segmentation algorithm, we define two in- dependent feature groups Prefix ( P 1 ) and Suffix ( S 1 ) over the abstract feature group Affix as an integer vector of the length 101, with: index( w ) := { 0 , if affix( w ) = index(affixes , affix( w )) , else Affix( w )[ i ] := { 1 , if i = index 0 , else (3.1) With affixes as the list of the 100 most frequent affixes and index( l, x ) as the index of x in the list l Thus, the feature is a mapping of a nominal attribute to a Boolean vector. As an example: given the suffix list: , “-s”, “-ing”, “-ed”, · · · , we would get: Suffix(“book”) = (1, 0, 0, 0, · · · ) Suffix(“booking”) = Suffix(“rebooking”) = (0, 0, 1, 0, · · · ) 16 From a linguistic-morphological point of view, the prefix of an English word gives little information about its word type. Nonetheless, we add the Prefix feature to the feature vector to check if there are any useful statistical correlations. The last suffix of a word, however, determines the type of a large group of words in English and other languages, so we expect it to be rather important. 3.1.3 Stem-based Features To separate ambiguous affixes, like “-s” or “-ing”, we introduce another fea- ture to estimate which word type the stem of a word belongs to. Again we define two features Prefix Stem( P 2 ) and Suffix Stem( S 2 ) with an abstract feature Affix Stem Affix Stem( w )[0] := |{ w ′ | stem( w ′ ) = stem( w ) (3.2) ∧ affix( w ′ ) = }| Affix Stem( w )[ i > 0] := |{ w ′ | stem(w’) = stem(w) (3.3) ∧ affix( w ′ ) = affixes[ i ] }| Thus, the affix stem feature is a vector, where every component corresponds to a specific affix and denotes how many word types are decomposed into the particular stem and affix. The first component belongs to the empty affix. Thus, given the suffix list from above: , “-s”, “-ing”, “-ed”, · · · and with only “book”, “book+s”, “book+ing” and “re+book+ing” in the vocabulary we get Suffix Stem(“book”) = Suffix Stem(“booking”) = (1, 1, 2, 0, · · · ) 3.2 Shape-like Features Shape-like features can be found in natural language processing applications such as named entity recognition. Most of the features are designed to iden- tify proper nouns and cardinal numbers. 3.2.1 Capitalization The Capital Group and Special Group are vectors of Boolean predicates. They are transformed into real vectors by mapping the truth values false 17 and true to 0 and 1, respectively. is capital( w ) := first character of w is an uppercase letter is all capital( w ) := ∀ c ∈ w : c is an uppercase letter capital character( w ) := ∃ c ∈ w : c is an uppercase letter appears in lowercase( w ) := ¬ capital character( w ) ∨ w ′ ∈ Σ T Where w ′ is obtained from w by changing all uppercase letters to their low- ercase counterparts and Σ T is the vocabulary of a training corpus. Thus, with “book”, “Book”, “Rebook” and “REBOOK” but not “rebook” in the vocabulary we would get: Capital Group(“book”) = (0, 0, 0, 1) Capital Group(“Book”) = (1, 0, 0, 1) Capital Group(“REBOOK”) = (1, 1, 1, 0) 3.2.2 Special Characters Along the lines of the Capital Group we define the Special Group using three further predicates: special character( w ) := ∃ c ∈ w : c is a special character digit( w ) := ∃ c ∈ w : c is a digit is number( w ) := w ∈ L ([+ − ][0 − 9] (([ ., ][0 − 9]) | [0 − 9]) ∗ ) With L [ expr ] as the language generated by the regular expression expr . As the resulting vectors have to be normalizable, i.e. different from the null vector, we add the predicate not special , defined as the negated conjunction of the three original special group predicates, to the special group. Note, that nothing alike is necessary for the Capital Group, as a word violating the first three predicates must fulfill the last one. While many languages use the Hindu-Arabic numerals to represent a majority of the numbers in the written language, the almost exclusive usage of capitalization to mark proper nouns is seen more rarely, therefore, the Capital Group features may not work for languages not having this property. Independently of the vocabulary we get: Special Group(“book”) = (0, 0, 0, 1) Special Group(“AT&T”) = (1, 0, 0, 0) Special Group(“C$1.5”) = (1, 1, 0,0) Special Group(“100,000”) = (1, 1, 1, 0) Special Group(“42”) = (0, 1, 1, 0) 18 3.2.3 Word Length As mentioned above, an important part of the word types cannot be clustered satisfactorily, as the words of these groups do not share sufficient morpho- logical information. Representative of these types are articles, pronouns, in- terjections, verbs and a some nouns. Using the fact that their average length differs, we try to reduce the similarity between the smaller word types, like pronouns and larger ones like nouns and verbs. That is, we use the fact that very small words have a special word distribution. With n as the length of the word w , the feature Length is defined as: Length ( w )[ i ] := min( n, 4) = i (3.4) For example: Length(“a”) = (1, 0, 0, 0) Length(“be”) = (0, 1, 0, 0) Length(“sea”) = (0, 0, 1,0) Length(“deal”) = (0, 0, 0, 1) Length(“eagle”) = (0, 0, 0, 1) 3.3 K-Means Clustering We cluster using the bisecting k-means algorithm, a variation of k-means. The k-means algorithm estimates for a sequence of data points D = { ~ x i } a vector of centroids ~ v = ( ~ v 1 , . . . , ~ v k ), approximately minimizing the overall distance between every data point and its closest centroid, which is the vector ~ v minimizing the function: f [ D ]( ~ v ) = ∑ i min { ( ~ v j − ~ x i ) 2 | 1 ≤ j ≤ k } (3.5) The standard form of the algorithm was stated in [14] by Stuart Lloyd and mainly consists of alternately calculating centroids and clustering from each other: 1. Start with an initial vector ~ v ′ (e.g. k random elements of ~ x i ) 2. Calculate ~ c = ( c 1 , . . . , c k ), where c i is the set of points closest to ~ v i 3. Calculate ~ v as the centroids of ~ c 19 4. Repeat 2. and 3. until a stopping criterion is reached Lloyd’s algorithm only estimates a local minimum of f and depends heav- ily on the initial centroids v ′ . Steinbach et al. [26] introduced a procedure to find effective initialization vectors, called bisecting k-means. Starting with one cluster holding all data points, the initial vector is estimated by repeating the following scheme k times: 1. Select a cluster to split 2. Find 2 sub-clusters using the basic k-means algorithm. 3. Repeat 2. a couple of times and take the split with the highest overall similarity. Steinbach et al. propose a number of selecting criteria including the largest cluster or the one with the least overall similarity. In this work, the largest cluster is split. The bisecting k-means algorithm can be used to generate hierarchical and flat clusterings. As an example we look at a hierarchical clustering that was calculated using the features Capital Group, Special Group, Length and Suffix. The resulting clustering can be seen in figure 3.2, 3.3 and 3.4. Every leave of the complete tree in 3.2 corresponds to one cluster of the flat clustering we use in the evaluation and in the language model. For the sake of simplicity every flat cluster is represented by one of its words. From the figure we can derive a precedence of our features. The first branching isolates words without special characters that also occur uncapital- ized (cf. figure 3.2 and 3.3). The next couple of splits (with some exceptions) extract the words matching the is number format (cf. figure 3.4). In the later evaluation and the entire language model we only use flat clusterings. 20