Diff for "Stats"

These stats differ from the Old Stats because the new stats tools deal with several few edge cases differently. This is especially true of all the analysis-heavy stats, like the frequency list related stuff.

The freqlistX% Target metrics changed recently. Read the freqlistX% Target paragraph if you're confused about the new numbers being lower.

If you got linked here for a statistic that isn't here (like the Core 6k stats), look at the Old Stats page instead.

For now, see the Old Stats page for more details.

If you want to add a script, post it to the Requests page.

See also Changes.

You can get the new analysis tools here (you need this too). They're easier to use than the previous ones, but they require python 3.6 or newer, and if you're not on Windows, you need to patch and recompile mecab yourself to fix a bug (instructions included) (you CANNOT use mecab as compiled by your distribution's maintainer).

Higher is easier: Hayashi; custom metric

Lower is easier: kanji (unique); kanji(2+); chars/line; chars/sentence; freqlistX% Target

The "Hayashi" metric are an estimate of structural complexity. The Hayashi metric is inherently flawed and should only be used as a general guideline.

The "custom metric" is like "Hayashi", and uses similar data, but it's designed to correlate well with "freqlist92.5% target". This is another estimate of structural complexity, but because it estimates the "freqlist92.5% target" metric, it's arbitrary.

The "freqlistX% Target" metrics are an estimate of lexical complexity. High measurements mean that the script uses uncommon words more frequently, or common words less frequently. Commonness is based on every long-enough script combined, not on general purpose frequency lists or the frequency list of the single script in question. However, the most common uncovered 20 words in the script are ignored, because they're probably names or setting jargon, or even parsing errors.

The hours estimate is kanji/18000 + hiragana/31500 + katakana/61400. It doesn't take any real complexity differences into account and it's based on VNDB's inaccurate length categories, take it with a huge grain of salt; also, it assumes a ~475 characters/minute reading speed, not like 300 or 200. It's calibrated based on a bunch of vndb length ratings with "very short" assigned to 1 hour and "very long" assigned to 60 hours. Example of calibrating with only character count as a source variable: https://i.imgur.com/tqTGVlB.png

In particular, the hours estimate should be taken as complete guesswork for any game that has extended interactions outside of ADV or NVL sequences, like minigames or general gameplay, and it's also not going to take into account any differences in structure or readability caused by complex grammar or rare words.

You can compare the Hayashi, custom metric a, custom metric b, and freqlist 92.5% target stats with this interactive graph.

The Hayashi metric is derived from: chars per sentence; commas per periods*; average consecutive length of each of kanji/hiragana/katakana/other

* sentence endings imply periods

The custom metrics are derived from:

   1 runlen_han   = counts[0]/runs[0]
   2 runlen_hira  = counts[1]/runs[1]
   3 runlen_kata  = counts[2]/runs[2]
   4 
   5 avg_counts = sum(counts[:3])/3
   6 prop_han   = counts[0]/avg_counts
   7 prop_hira  = counts[1]/avg_counts
   8 prop_kata  = counts[2]/avg_counts
   9 
  10 avg_runs = sum(runs[:3])/3
  11 prop_runs_han   = runs[0]/avg_runs
  12 prop_runs_hira  = runs[1]/avg_runs
  13 prop_runs_kata  = runs[2]/avg_runs
  14 
  15 norm_han   = runlen_han *prop_han
  16 norm_hira  = runlen_hira*prop_hira
  17 norm_kata  = runlen_kata*prop_kata
  18 
  19 sentence_han   = counts[0]/sentences
  20 sentence_hira  = counts[1]/sentences
  21 sentence_kata  = counts[2]/sentences
  22 
  23 runs_sentence = sum(runs[:3])/sentences
  24 sentencelen = sentencechars/sentences
  25 
  26 # ....
  27 
  28 properties_a = [
  29     math.log(prop_han),
  30     math.log(prop_hira),
  31     math.log(prop_kata),
  32     
  33     math.log(prop_runs_han),
  34     math.log(prop_runs_hira),
  35     math.log(prop_runs_kata),
  36     
  37     norm_han,
  38     norm_hira,
  39     norm_kata,
  40     
  41     sentence_han,
  42     sentence_hira,
  43     sentence_kata,
  44     
  45     1/runs_sentence,
  46     math.log(sentencelen)
  47 ]
  48 weights_a = [
  49 -3.4920757 ,
  50 -0.6549337 ,
  51 -0.17232664,
  52  1.0730087 ,
  53 -1.1602864 ,
  54  0.02242127,
  55  1.764859  ,
  56 -0.10058399,
  57  0.23199975,
  58  0.0919582 ,
  59  0.01803382,
  60  0.04476589,
  61 -1.3525765 ,
  62 -0.76003593,
  63  0.20918524
  64 ]
  65 properties_b = [
  66     runlen_han,
  67     runlen_hira,
  68     runlen_kata,
  69     
  70     math.log(prop_han),
  71     math.log(prop_hira),
  72     math.log(prop_kata),
  73     
  74     math.log(prop_runs_han),
  75     math.log(prop_runs_hira),
  76     math.log(prop_runs_kata),
  77 ]
  78 weights_b = [
  79  1.253433  ,
  80 -0.14630389,
  81  0.00329586,
  82 -2.4908898 ,
  83 -5.5842404 ,
  84 -0.14600338,
  85  1.5507061 ,
  86  0.13142511,
  87 -0.10953176,
  88  1.4638389
  89 ]
  90 
  91 nothayashi_a = weights_a[-1] + sum(properties_a[i]*weights_a[i] for i in range(len(properties_a)))
  92 nothayashi_a = (1-nothayashi_a)*50+50
  93 nothayashi_b = weights_b[-1] + sum(properties_b[i]*weights_b[i] for i in range(len(properties_b)))
  94 nothayashi_b = (1-nothayashi_b)*50+50

where "counts" stores the number of times a writing system occurs, and "runs" stores the number of runs there are of that writing system. The weights were derived from multiple regression with keras.