Diff for "Stats"

I just rewrote the stats tools. These are the stats output by the new tools. They differ slightly from the old stats because the rewrite deals with a few edge cases differently. The differences are within a margin of error, or when the difference is big, the old stats tools are probably doing something wrong.

If you got linked here for a statistic that isn't here (like the Core 6k stats), look at the Old Stats page instead.

For now, see the Old Stats page for more details.

If you want to add a script, post it to the Requests page.

See also Changes.

You can get the new analysis tools here (you need this too). They're easier to use than the previous ones, but they require python 3.6 or newer, and if you're not on Windows, you need to patch and recompile mecab yourself to fix a bug (instructions included) (you CANNOT use mecab as compiled by your distribution's maintainer).

Higher is easier: Hayashi; custom metric

Lower is easier: kanji (unique); kanji(2+); chars/line; chars/sentence; freqlistX% Target

The "Hayashi" metric are an estimate of structural complexity. The Hayashi metric is inherently flawed and should only be used as a general guideline.

The "custom metric" is like "Hayashi", and uses similar data, but it's designed to correlate well with "freqlist92.5% target". This is another estimate of structural complexity, but because it estimates the "freqlist92.5% target" metric, it's arbitrary.

The "freqlistX% Target" metrics are an estimate of lexical complexity. High measurements mean that the script uses uncommon words more frequently, or common words less frequently. Commonness is based on every long-enough script combined, not on general purpose frequency lists or the frequency list of the single script in question.

The hours estimate is kanji/18000 + hiragana/31500 + katakana/61400. It doesn't take any real complexity differences into account and it's based on VNDB's inaccurate length categories, take it with a huge grain of salt; also, it assumes a ~475 characters/minute reading speed, not like 300 or 200. It's calibrated based on a bunch of vndb length ratings with "very short" assigned to 1 hour and "very long" assigned to 60 hours. Example of calibrating with only character count as a source variable: https://i.imgur.com/tqTGVlB.png

You can compare the Hayashi, custom metric a, custom metric b, and freqlist 92.5% target stats with this interactive graph.

The Hayashi metric is derived from: chars per sentence; commas per periods*; average consecutive length of each of kanji/hiragana/katakana/other

* sentence endings imply periods

The custom metrics are derived from:

   1 runlen_han   = counts[0]/runs[0]
   2 runlen_hira  = counts[1]/runs[1]
   3 runlen_kata  = counts[2]/runs[2]
   4 
   5 avg_counts = sum(counts[:3])/3
   6 prop_han   = counts[0]/avg_counts
   7 prop_hira  = counts[1]/avg_counts
   8 prop_kata  = counts[2]/avg_counts
   9 
  10 avg_runs = sum(runs[:3])/3
  11 prop_runs_han   = runs[0]/avg_runs
  12 prop_runs_hira  = runs[1]/avg_runs
  13 prop_runs_kata  = runs[2]/avg_runs
  14 
  15 norm_han   = runlen_han *prop_han
  16 norm_hira  = runlen_hira*prop_hira
  17 norm_kata  = runlen_kata*prop_kata
  18 
  19 sentence_han   = counts[0]/sentences
  20 sentence_hira  = counts[1]/sentences
  21 sentence_kata  = counts[2]/sentences
  22 
  23 runs_sentence = sum(runs[:3])/sentences
  24 sentencelen = sentencechars/sentences
  25 
  26 # ....
  27 
  28 properties_a = [
  29     math.log(prop_han),
  30     math.log(prop_hira),
  31     math.log(prop_kata),
  32     
  33     math.log(prop_runs_han),
  34     math.log(prop_runs_hira),
  35     math.log(prop_runs_kata),
  36     
  37     norm_han,
  38     norm_hira,
  39     norm_kata,
  40     
  41     sentence_han,
  42     sentence_hira,
  43     sentence_kata,
  44     
  45     1/runs_sentence,
  46     math.log(sentencelen)
  47 ]
  48 weights_a = [
  49 -2.6283512  ,
  50 -0.8434388  ,
  51 -0.2378658  ,
  52  0.37981617 ,
  53 -0.855799   ,
  54  0.0530583  ,
  55  1.442638   ,
  56 -0.05952703 ,
  57  0.45643884 ,
  58  0.09780279 ,
  59  0.0371693  ,
  60 -0.03599532 ,
  61 -2.5243423  ,
  62 -1.1433551  ,
  63  1.6064633
  64 ]
  65 properties_b = [
  66     runlen_han,
  67     runlen_hira,
  68     runlen_kata,
  69     
  70     math.log(prop_han),
  71     math.log(prop_hira),
  72     math.log(prop_kata),
  73     
  74     math.log(prop_runs_han),
  75     math.log(prop_runs_hira),
  76     math.log(prop_runs_kata),
  77 ]
  78 weights_b = [
  79  0.62152517 ,
  80  0.0021763  ,
  81  0.06648773 ,
  82 -1.611983   ,
  83 -5.896055   ,
  84 -0.32356036 ,
  85  0.37851688 ,
  86  0.63546646 ,
  87  0.04297037 ,
  88  2.2276359
  89 ]
  90 
  91 nothayashi_a = weights_a[-1] + sum(properties_a[i]*weights_a[i] for i in range(len(properties_a)))
  92 nothayashi_a = (1-nothayashi_a)*50+50
  93 nothayashi_b = weights_b[-1] + sum(properties_b[i]*weights_b[i] for i in range(len(properties_b)))
  94 nothayashi_b = (1-nothayashi_b)*50+50

where "counts" stores the number of times a writing system occurs, and "runs" stores the number of runs there are of that writing system. The weights were derived from multiple regression with keras.