F-점수와 정확도

From Eyewire
Revision as of 18:15, 30 December 2015 by Pilnpat (Talk | contribs)

Jump to: navigation, search

아이와이어를 잘 하려면

아이와이어 오른쪽 상단에서 사용자명을 클릭하면 프로필과 정확도(accuracy)를 알 수 있습니다. 정확도는 F-Score라고 하고 색상이 너무 많이 추가된 색상 과다(over coloring)와 조각이 사라진 색상 부족(under coloring)의 2개 부분으로 구성됩니다. Accuracy 바에서 마우스를 움직이면 개인별 맵핑 통계를 포함한 드롭다운(dropdown)을 볼 수 있습니다.

\"색상 과다(Over Coloring)\"는 추적하는 세포 외부에 추가된 전체 부피의 비율을 계산한 것으로 Precision이라고 불리기도 합니다.
\"색상 부족(Under Coloring)\"은 맵핑 시 누락된 전체 가지의 비율을 계산한 것으로 Recall이라고 불리기도 합니다.

F-Score가 높을수록(Precision과 Recall이 높을수록) 아이와이어를 더 잘하는 것이고 또한, 과학에 더 많이 이바지하는 것입니다. 더 많은 과학발전과 더 많은 영광으로 이어집니다. 물론, 개척자(trailblazer)를 플레이할 수 있는 기회도 더 많아집니다.

Error creating thumbnail: Unable to save thumbnail to destination

Before we can calculate the final F-score first we must calculate your individual precision and recall. When a player does a cube there are four possible outcomes for every segment in that cube: a true positive result, a false positive result, a false negative result and a true negative result. A true positive (tp) result is when a player adds a segment that should be added. A false positive (fp) is when a player adds a segment that should not be added. A false negative (fn) is when a player misses a segment they should have added. A true negative (tn) is when a player correctly leaves out a segment that does not belong. In the figure below you can see an example of false negative and of false positive.


NewFScoreEyeWire.png
To the left is an example of a branch submitted by a player. In this example the red and the green segments are what the player submitted, while the purple segment was left out.


The red segment here is a false positive and the purple segment is a false negative. The player mistakenly added the red segment when they should have added the purple segment instead. The green segment is correct.


This brings us to precision; precision is how much of a volume was added correctly. For example if Player A has a precision 0.9221 that means about 92% of what Player A added was correct and about 8% of what Player A added should not have been added. To determine a player’s precision we use their true positive (tp) results, correctly added, and their false positive (fp) results, incorrectly added, in this formula:
Error creating thumbnail: Unable to save thumbnail to destination


Recall measures how much of the volume was missed. Let’s say Player A has a recall of 0.9409. That means that Player A missed about 6% of the correct segments in the cubes Player A worked on. To determine a player’s recall we use their true positive (tp) results, correctly added, and false negative (fn) results, incorrectly missed, in this formula:
Error creating thumbnail: Unable to save thumbnail to destination


Now we would take the results from both of those formulas and plug them into the formula above to get a player’s F-score. Another way to look at it is we take the harmonic mean of a player’s precision and recall to get their overall accuracy rating.

How Accurate are F-Scores?

One question we a get a lot is how do we know what is correct and what isn’t? What is correct is determined by combining the GrimReaper’s corrections with the EyeWirer consensus. If a cube does not have a GrimReaper correction we just use the EyeWirer consensus. EyeWire consensuses have proven to be quite accurate. However, there is still a small chance that a consensus may contain a wrong piece. This means that F-scores cannot prove user accuracy 100% of the time. However, they are accurate enough that we feel confident using them as a player guide.