SEPARATING DOCUMENTS BY LEXICAL REPETITION VIA YULE'S K AND K-MEANS: VALIDATION THROUGH MONTE CARLO SIMULATION

Resumo

This work proposes and evaluates, through Monte Carlo simulation, a simple methodology for separating textual documents into two groups, high lexical repetition and low lexical repetition, based on Yule's K, a classic measure of vocabulary concentration. The methodology consists of computing Yule's K for each document in a corpus and applying k-means (k = 2) to these values to partition the corpus into the two groups. For each of nine fixed values of the true proportion  (from 0.1 to 0.9), we ran 500 Monte Carlo simulations in which the degree of lexical separation between the two classes is drawn randomly at each repetition, totaling 4,500 simulated corpora. The results show that the method recovers the true proportion with good precision across the whole range of  (mean absolute error of 0.008 and mean classification accuracy of 98.9%), but the error and the variability of the estimate increase systematically at the extreme values of  (close to 0.1 or 0.9), where the minority class is small and more sensitive to scenarios of low separation between the K distributions.