PHM10 Data Challenge – Scoring

This page describes how the score is calculated for challenge entries.

Your task is to estimate the maximum number of cuts one could “safely” make for a given wear limit. By “safely”, we mean that the maximum wear of any flute does not exceed the wear limit (not the average wear across the flutes). For example, Figure 1 is the wear for all three flutes in case one. Figure 2 highlights the maximum wear for any of the three flutes.

Figure 1: Wear across all three flutes.

Figure 2: Wear across all three flutes with maximum wear highlighted.

The end user of your PHM system can then take a cut schedule and apply it to a piece, stopping at your estimated number of cuts. If you underestimate, that is a little bad – you waste tool life by stopping too soon. If you overestimate, that is a really bad – you maximize tool life, but you mess up the expensive piece being manufactured, and remedial steps have to be taken. So the scoring function is asymmetrical to reflect that (see below).

You will be asked to make an estimate of the max safe cuts at integer values over the range 66 to 165 (Figure 3) – i.e., you are to pretend the wear limit is 66, 67…, 164, 165. This is unusual – normally a component has one life limit and you estimate RUL to that – but makes for a better competition.

Figure 3: the range over which the competition is scored.

Figure 4 is an example of an estimate you might submit. Of course, you don’t know the maximum wear value, but for each integer value between 66 and 165, you can estimate how many cuts it might take to get there (“max cut estimate @ wear limit”). Note that there may (should) be large jumps in your estimate occosionally – some cutting patterns produce very little wear. For example, in Figure 4, the max wear ground truth exhibits little wear between cuts 75 and 125; the estimated number of safe cuts plotted shows that the model used expects very little wear caused by cuts 125 to 200.

Figure 4: an example of a submission (“max cut estimate @ wear limit”) and the underlying ground truth (“maximum wear”).

Figure 5 is an example of what we mean by over and underestimating life. If your tool wear limit was 100, the model here predicts about 85 cuts could be made for a given cut schedule, when actually about 150 cuts could be made, representing 65 cuts worth of lost flute life if the flute were discarded at 85 cuts. For a wear limit of 120, the model overestimates the flute life for the given cut schedule, resulting in ~40 cuts with a flute that is past its rated life, likely gouging the part.

Figure 5: Over and under estimates illustrated.

Figure 6 shows how the scoring is done – at each step in wear limit, the difference between ground truth and estimate is calculated.

Figure 6: Calculating over and under estimates at each wear limit.

The function used for scoring is:
d = estimated_cuts_at_wear – actual_cuts_at_wear;
score(d<0) = exp(-d(d<0)/10)-1; score(d>=0) = exp(d(d>=0)/4.5)-1;
score = sum(score)

Figure 7 illustrates some interesting features of the scoring function. The top frame is a plot of the scoring function itself. Note as discussed above that it is quite asymmetrical – e.g., the middle frame shows that for small errors, the penalty is almost the same, but for large errors, the penalty is much greater for overestimating the maximum number of cuts that can be safely made. The bottom frame shows the ratio for a particular error – e.g., it is about five times worse to overestimate than underestimate by 10 cuts.

Figure 7: Scoring function (top); penalty for an over or under estimate as a function of absolute error (middle); ratio of over to underestimate (bottom)

Submitting Results
Results must be submitted as a comma separated value (CSV) file, with exactly 101 rows. Row one should be:
your_username, your_password
Rows 2-101 should be your estimated maximum safe cut estimate for wear of 66 to 165. Column one should be the estimate for case two; column two should be the estimate for case three; column three should be the estimate for case five. An example of a properly formatted scoring file can be found here.