Which sensors carry real signal about lot failure?

A wafer fab runs 590 in-line sensors. This analysis screens all of them and finds a six-sensor watch-list. On this data, a predictive lot screen did not work forward in time. The screen caught no failures on a time-ordered split.

This is a public 2008 dataset, analysed as a demonstration. The data is UCI SECOM, real production data from one semiconductor fab in 2008. The sensor identities are anonymised. No sensor number maps to a named process step, so this analysis cannot reach a physical root cause. The work demonstrates the method on real data. It does not validate a live fab process.

Read the base rate first

104 lots fail out of 1,567. The base failure rate is 6.64%.

A rule that says "pass" for every lot is 93.36% accurate. That same rule finds zero failures. Accuracy is therefore not the metric here.

Judge every result below against that zero. A model must find failures to be worth anything. A high accuracy score alone proves nothing on this data.

The engineering problem

A fab measures each lot with 590 sensors. A small share of lots fail final test. An engineer cannot watch 590 charts.

So the engineer needs a short watch-list. Three questions follow.

Which sensors track failure?
Does any sensor drift out of control over time?
Can a model predict which lot fails before final test?

This analysis answers all three questions. The first two answers are useful. The third answer is negative, and that negative answer is the most important result on this page.

Lots 1,567 89 days, 2008-07-19 to 2008-10-17

Sensors 590 raw, before you clean the data

Failures 104 6.64% base rate

Watch-list 6 sensors that survive both screens

The data carries 590 sensors, and not the 591 that secom.names states. Every row of secom.data has exactly 590 fields, and the UCI CSV export names Attribute 1 through Attribute 590. The data is treated as authoritative. See change record PC-001.

Data cleaning

Cleaning runs on the training rows only. Two rules drop a sensor. The first rule drops a near-constant sensor, because a sensor that does not move carries no information. The second rule drops a sensor with too many missing readings.

Cleaning step	Sensors
Raw sensors	590
Dropped, near-constant	122
Dropped, high missingness	32
Retained for screening	436

Cleaning drops 122 near-constant sensors and 32 high-missingness sensors. 436 sensors remain. Correlation clustering at absolute r above 0.95 then finds 103 clusters. Those clusters cover 276 of the 436 retained sensors, and the largest cluster holds 6 sensors.

Many retained sensors are near-duplicates of each other. A correlation cluster is a group of sensors that move together. Treat each cluster as one measurement. If you count each sensor in a cluster separately, you count the same evidence more than once.

Per-sensor missingness across all 590 sensors, sorted descending, with the 45 percent drop threshold marked. — Figure 1 Look at the steep left edge. A small group of sensors loses most of its readings, and the curve then falls close to zero. The dashed line is the 45% drop threshold. Only 32 sensors sit above that line. Missing data is concentrated in a few sensors, and it is not spread evenly across all 590.

The screening collapse

This is the most important statistical finding in the analysis.

The screen tests 436 sensors, one test per retained sensor. 63 sensors look significant at a raw p below 0.05. Only 8 sensors survive the Benjamini-Hochberg correction.

Test 436 sensors at p < 0.05 and chance alone produces about 21.8 apparent hits. The raw count is 63. So a large part of the raw count is noise.

The Benjamini-Hochberg procedure controls the false discovery rate at q = 0.05. After the correction, 8 sensors survive.

Most of what looks like signal in a 436-way comparison is not signal. An analysis that reports the raw count reports mostly noise.

Comparisons 436 one test per retained sensor

Raw significant 63 at p below 0.05, uncorrected

Expected by chance 21.8 436 tests times 0.05

Survive FDR 8 Benjamini-Hochberg at q = 0.05

Volcano plot of effect size against Benjamini-Hochberg adjusted significance for 436 sensor comparisons. — Figure 2 Look above the dashed q = 0.05 line. Only eight dark points clear it, and six of those carry a label. The large grey cloud below the line is the rest of the 436 sensors. The cloud shows what a 436-way comparison looks like when you correct it properly.

The six-sensor watch-list

8 sensors survive the FDR screen. 6 of those also survive a second, independent screen: cross-validated L1 selection stability. The watch-list is those 6 sensors. Each one passed two screens that use different evidence.

The watch-list was not padded. The frozen analysis plan suggested a target of 10 to 20 sensors. Six sensors survive both screens. The list stays at six. If you pad a watch-list, you put unvalidated sensors in front of an operator, and the operator then charts noise.

Sensor	Direction in fail group	FDR p	Effect size	L1 selection
sensor_060	runs high	6.18e-05	+0.3683	1.0
sensor_511	runs high	4.35e-03	+0.2921	0.8
sensor_206	runs high	3.57e-02	+0.2448	0.8
sensor_029	runs low	3.57e-02	-0.2439	0.6
sensor_130	runs high	4.08e-02	+0.2382	1.0
sensor_317	runs low	4.36e-02	-0.2346	0.6

L1 selection is the share of the 5 cross-validation folds in which the L1 model kept the sensor. Effect size is the univariate pass against fail separation. A positive effect size means the fail group runs high. A negative effect size means the fail group runs low.

Watch-list sensors with univariate effect size on the left and L1 selection frequency on the right. — Figure 3 Look for sensors that are strong on both panels. sensor_060 has the largest effect size and the model keeps it in all 5 folds. Two sensors run to the left of zero on the left panel. The fail group runs low on those two. A sensor strong on one panel only is weaker evidence.

Statistical process control

The weekly p-chart covers 13 bins. It flags exactly 1 out-of-control point: the week of 2008-07-28. That week ran 10 failures in 48 lots, which is 20.83%. The upper control limit for that subgroup size is 17.42%. So the point is a genuine signal, and it is not small-sample noise.

The p-chart uses stepped control limits, because the number of lots changes from week to week. One small bin held 13 lots, which is below the 20-lot minimum. That bin merged forward into the next week. The merge is recorded and it does not affect the excursion week.

Yield loss over time across 13 weekly bins, plain view without control limits. — Figure 4 Look at the shape over the 13 weeks. The failure rate starts high, then falls below the centre line from mid-August onward. This plain view has no control limits, so no point here is in or out of control. The next figure adds the limits that decide that question.

p-chart of weekly fraction defective over 13 bins with stepped control limits and one flagged week. — Figure 5 Look at the single red point. That week is the only one of 13 above its upper control limit. The limits are dashed and stepped, because each week holds a different number of lots. A week with few lots gets a wider limit. Every other point sits inside its limits, so the rest of the period is in statistical control.

The sensor_060 coincidence

In that same week, sensor_060 ran at a mean of 14.79. Outside that week the same sensor ran at 2.58. The shift is 1.28 standard deviations. sensor_060 is also the top watch-list sensor.

A time coincidence is not proven cause. sensor_060 moved in the same week the failure rate rose. Nothing in this data shows that the quantity sensor_060 measures caused the failures. The shift could be a cause. The shift could also be a symptom, a common-cause co-response, or a coincidence. The data can point at the week. Only fab records can say what happened in that week.

Individuals and moving range control charts for sensor_060 across the full period. — Figure 6 Look at how many red points appear on the left half and how few appear on the right. sensor_060 has 129 out-of-control points on the individuals chart across the whole period. This sensor is unstable in general, and the excursion week is an extreme instance rather than the only event.

Individuals and moving range control charts for sensor_511. — Figure 7 Look at the vertical dashed line, which marks the excursion week. sensor_511 is the second watch-list sensor. Compare its behaviour in that window against sensor_060 in Figure 6.

Individuals and moving range control charts for sensor_206. — Figure 8 Look at the spread of the points across time. sensor_206 is the third watch-list sensor. These three I-MR charts show why each watch-list sensor needs its own limits, frozen from a stable baseline period.

The headline finding: prediction does not carry forward in time

On a time-ordered split, the model finds 0 of 26 failures. Recall is 0.000, with a Wilson 95% confidence interval of 0.000 to 0.129.

On a stratified split, the same model finds 17 of 31 failures. Recall is 0.548, with a Wilson 95% confidence interval of 0.378 to 0.708.

The time-ordered number is the honest one. A stratified split mixes later lots into the training data. A fab cannot do that, because a fab must predict a lot it has not made yet. The stratified recall of 0.548 is not the result of this analysis. The result is the collapse to 0.000.

Read the intervals, not the point estimates. The positive class holds 104 failures, which is small. The time-ordered upper bound is 0.129, so the data cannot rule out a small forward recall. The data does rule out a useful one.

The memo recommends against a predictive lot screen. At 0 of 26 failures found forward in time, a screening model is a false-alarm generator. Deploy the watch-list charts instead.

The two splits, side by side

Preprocessing is refit inside every outer split and every inner cross-validation fold. So no held-out lot informs the transform applied to it. The two splits differ only in how the lots are divided.

Split and model	Recall	Wilson 95% CI	Found	Precision	PR-AUC
Time-ordered, ensemble	0.000	0.000 to 0.129	0 of 26	0.000	0.063
Time-ordered, logistic	0.038	0.007 to 0.189	1 of 26	0.036	0.061
Stratified, ensemble	0.548	0.378 to 0.708	17 of 31	0.218	0.283
Stratified, logistic	0.581	0.408 to 0.736	18 of 31	0.122	0.151

Recall is the share of failures the model finds. Precision is the share of flagged lots that truly fail. PR-AUC is the area under the precision-recall curve on the test set. The trivial always-pass rule scores a recall of 0.000 on both splits.

The 10:1 escape:false-alarm cost ratio is a stated assumption, not a measured cost. A real fab would substitute its own ratio; the sensitivity of the chosen threshold to 5:1 and 20:1 is reported.

Precision-recall curves on the test set for both splits and both models. — Figure 9 Compare the two panels. On the left, the stratified curves rise well above the dashed trivial baseline. On the right, the time-ordered curves sit close to that baseline across the whole range. The right panel is the forward estimate, and it shows the model barely beats the baseline.

Confusion matrices at the tuned thresholds for both models and both splits. — Figure 10 Look at the true-positive cell in each of the four panels. The two top panels are the stratified split, and they find 18 and 17 failures. The two bottom panels are the time-ordered split, and they find 1 and 0. The bottom right panel is the honest forward result.

The two split halves carry different class balances. The earlier portion runs 7.11% failures (78 of 1,097). The later portion runs 5.53% (26 of 470). That difference is NOT statistically significant (two-proportion z = 1.15, p = 0.25; Fisher exact p = 0.27). So it is consistent with ordinary variation, and this analysis does not claim the process changed. It is reported because it explains the different class balances, and not as a finding.

The quality documentation

Three documents turn the statistics into actions. Each one states its own evidence boundary.

PFMEA

The PFMEA covers the failure modes the watch-list suggests. It names no tool, no chamber, and no process step, because the sensor identities are anonymised. Each row is framed by process area archetype instead. An archetype describes the kind of behaviour the data shows, such as a drifting sensor or a shifted fail-group mean.

The severity, occurrence, and detection ratings are engineering judgment under stated assumptions. The run measured none of them. So the RPN values are prioritisation aids, and they are not data. A fab that adopts this PFMEA must re-rate every row with its own process knowledge.

Root cause analysis

The RCA covers the one dated excursion. It runs a 5-Why analysis, and it stops at the second Why. The label is final-test pass or fail only, and it carries no failure mode detail. So the data cannot say why those lots failed.

The RCA separates three categories. The data establishes the excursion and the sensor_060 shift. The link between the two is a temporal association, and it is not a proven cause. The physical step, the failure mode, and the maintenance history are not determinable from the data.

Control plan

The control plan charts the six watch-list sensors on I-MR charts and the weekly failure rate on a p-chart. Every chart has a written reaction plan, an owner, and an escalation path. A chart signal always produces a named action by a named role within a stated time.

Charts detect, and charts do not adjust. No process change follows a chart signal without engineering review. The plan adds one more chart for the missing-data rate, because a silent sensor is not a passing sensor.

The plan contains no capability study. SECOM publishes no spec limits, so the control limits here are statistical limits and they are not specifications.

Limitations

These limits are firm. Read them as part of the result, and not as a disclaimer.

The sensors are anonymised. The dataset names them sensor_001 through sensor_590. No public mapping ties a sensor number to a physical process step. So sensor_060 is a number, and it is not a chamber. Root cause cannot reach a real process step from this data alone.
There are no published spec limits. So no capability study is possible, and none was invented. Every limit on this page is a statistical voice-of-process limit. A Cp or Cpk value would need invented spec limits, which would be fabrication.
The label is final pass or fail only. The data carries no failure mode detail. So the analysis cannot say why any lot failed.
The positive class is small. The dataset holds 104 failures. Every recall figure therefore carries a wide confidence interval. Treat the intervals as the answer, and not the point estimates.
The cost ratio is an assumption. The threshold uses an assumed 10:1 escape-to-false-alarm cost ratio. Nobody measured that ratio. A fab must substitute its own ratio, and the sensitivity at 5:1 and 20:1 is reported in the run.
The monitoring floor is thin. The dataset holds 41,951 missing sensor readings, and 538 of 590 sensors have at least one gap. Any monitoring built on this data needs a missing-data alarm, or the monitoring fails silently.
This is one fab, in 2008. The realism is why the data is worth analysing. The same realism is why these numbers do not transfer to another fab or another year.