May 1, 2007

*Columnists Howard Mark and Jerome Workman, Jr. respond to reader feedback regarding their 14-part column on the analysis of noise in spectroscopy by presenting another approach to analyzing the situation.*

We occasionally get feedback from our readers. Often, the feedback is positive, letting us know that they find reading these columns interesting, enjoyable, and most importantly, useful. Getting responses like that also makes writing these columns interesting and enjoyable, and makes us feel useful, too. Occasionally, one or more readers finds an error (horrors!) that requires a correction (more horrors!), or presents an alternative approach to, or interpretation of, our discussion. When we wrote our 14-part column on the analysis of noise in spectroscopy, a few readers wrote in, letting us know that they felt that some parts of the derivations were weak, and perhaps less than perfectly rigorous. We took their comments to heart and here present the first of a two-part discussion incorporating the comments. This could be considered a (hide your eyes!) correction, or perhaps, simply another approach to analyzing the situation.

Some time ago, we published a subseries of columns within this series (1–14) giving rigorous derivations for the expressions relating the effect of instrument noise and other types of noise to their effects on the spectra we observe. During and after the publication of those columns, we received several comments discussing various aspects, including some errors that crept in. Most of those were minor and can be ignored (for example, a graph axis labeled "Transmission" when the axis went from 0–100%, and therefore, the axis should have been labeled "% Transmission").

Jerome Workman, Jr.

One comment, however, was a little more significant, and therefore, we need to take cognizance of it. One of our respondents noted that the analysis performed could be done in a different way, a way that might be superior to the way we did it. Normally, if we agree with someone who takes issue with our work we would simply publish a correction (assuming their comments are persuasive). In this case, however, that seems inappropriate for several reasons. First, the original analysis was published a long time ago and cannot be dispensed with easily. Second, we're not convinced that our original approach is wrong; therefore, it is not clear that a correction is warranted. Third, some of our readers might wish to compare the two approaches for themselves to decide if the original one is actually wrong or simply not as good, or whether, in fact, the new analysis is better. Therefore, we present a new, alternate analysis, along the lines recommended by our respondent.

Howard Mark

The point of departure is from our original column (5), which dealt with the effect of random, normally distributed noise whose magnitude (in terms of standard deviation) is independent of the strength of the optical signal. Here we present the revised analysis of this situation of the effect on the expected noise level of the computed transmittance when the signal noise is not small compared to the signal level *E _{r}*.

Before we proceed, however, there is a technical point we need to clear up, and that is the numbering of the equations in this column series. Ordinarily, when continuing a subject through several columns, we simply continue the numbering of equations as though there were no break. The column representing our point of departure ended with equation 63. Therefore, it is appropriate to begin this alternate analysis (5) with equation 64, as we would normally. However, equation number 64 (and subsequent numbers) was already used; therefore, we cannot simply repeat using the same equation numbers that we used already. Neither can we simply continue from the last number used in the original analysis (5) because they would also conflict with the equation numbers already used for other purposes.

We resolved this dilemma by adding the suffix "a" to the equation numbers used in this column. Therefore, the first new equation we introduce here will be equation 64a. Fortunately, none of the equations developed in the column with the original analysis, nor the figures, used any suffix, as was done occasionally in other columns (we will copy equation 52b from the previous column, but the "b" suffix does not signify a new equation because it is the equation used previously; also, a "b" suffix is not indicative of a copy of an equation number here, only an "a" suffix). Therefore, we can distinguish the numbering of any equations or other numbered entities in this section by appending the suffix "a" to the number without causing confusion with other corresponding entities.

Now we are ready to proceed.

We reached this point from the discussion just before equation 64 (5). A reader of the original column felt that equation 64 was being used incorrectly. Equation 64, of course, is a fundamental equation of elementary calculus and is itself correct. The problem pointed out was that the use of the derivative terms in equation 64 implicitly indicates that we are using the small-noise model, and especially when changing the differentials to finite differences in equation 65, results in incorrect equations.

In our previous column (4), we had created an expression for *T* + Δ*T* (as equation 51) and separated out an expression for Δ*T* (as equation 52b). We present these two equations here:

will vary due to the presence of the Δ*E _{r}* in the denominator, and using only the second term would ignore the influence of the variability of the first term in equation 51 (4) and not take its contribution to the variance into proper account. Therefore, the expression for Δ

to include the variability of the first term, also.

This, however, leads to another problem: subtracting equation 64a from equation 51 leaves us with the result that *T* = 0. Furthermore, the definition of *T* gives us the result that *E _{s}* is zero, and that therefore, Δ

Our conclusion is that the original separation of equation 51 into two equations, while it served us well for computing *T _{M}* and

This also raises a further question: the analysis of equation 52a by itself served us well, as we noted, but was it proper, or should we have maintained all of equation 51, as we find we must do here? The answer is yes, it was correct, and the justification is given toward the end of the earlier column (4). The symmetry of the expression when averaged over values of *E _{s}* means that the average will be zero for each value of

Therefore, we conclude that the best way to maintain the entire expression is to go back to a still earlier step and note that the ultimate source of equation 51 was equation 5 (2):

Because *E _{s}*/

that would not be correct because the two variances that we want to add have a common term (*E _{r}* + Δ

Thus, we conclude that we must compute the variance of Δ*T* directly from equation 68a and the definition of variance:

and again using the definition of variance:

where we note that the limit of *n*/(*n* - 1)→ 1 as *n* becomes indefinitely large. Of course, the noise level we want will be the square root of equation 75a.

We have seen previously, in equation 77 (5), that the variance term in equation 75a diverges, and clearly, as Δ*E _{r}* → –

To recapitulate some of that here, it was a matter of noting two points: first, that as *E _{r}* gets further and further away from zero (in terms of standard deviation), it becomes increasingly unlikely that any given value of Δ

We now consider how to compute the variance of Δ*T*, according to equation 68a. Ordinarily, we would first discuss converting the summations of finite differences to integrals, as we did previously, but we will forbear that, leaving it as an exercise for the reader. Instead, we will go directly to consideration of the numerical evaluation of equation 68a, because a conversion to an integral would require a back-conversion to finite differences in order to perform the calculations.

We wish to evaluate equation 68A for different values of *E _{s}* and

There are still a variety of ways we can approach the calculations. We could assume that *E _{s}* or

What we will actually do here, however, is all of these. First, we will assume that the ratio of *E _{s}*/

To do the computations, we again use the random number generator of MATLAB to produce normally distributed random numbers with unity variance to represent the noise. Values of *E _{r}* will then directly represent the S/N of the data being evaluated. For the computations reported here, we use 100,000 synthetic values of the expression on the right-hand side of equation 76a to calculate the variance, for each combination of conditions we investigate.

Figure 1

A graph of the transmittance noise as a function of the reference S/N is presented in Figure 1a and the expanded portion of Figure 1a, shown in Figure 1b. The "true" transmittance, *E _{s}*/

The inevitable existence of a limit on the value of *T _{M}*, as described in the section following equation 75a, was examined in Figure 1a by performing the computations for two values of that limit, by setting the limit value (somewhat arbitrarily, to be sure) to 1000 and 10,000, corresponding to the lower and upper curves, respectively.

Note that there are effectively two regimes in Figure 1a, with the transition between regimes occurring when the value of S/N equals approximately 4. When the value of *E _{r}* was greater than approximately 4 — that is, S/N was greater than four — the curves were smooth and appear to be well behaved. When

The "high-noise" regime seen in Figure 1a is the range of values of S/N in which the computed standard deviation is grossly affected by the closeness of the approach of individual values of Δ*E _{r}* to

Changing the number of values of (*E _{s}* + Δ

Figure 2 shows the graph of transmittance noise computed empirically from equation 76a, compared to the transmittance noise computed from the theory of the low-noise approximation, as per equation 19 (2) and the approach, under question, of using equation 52b. We see that there is a third regime, in which the difference between the actual noise level and the low-noise approximation is noticeable, but the computed noise has not yet become subject to the extreme fluctuations engendered by the too-close approach of Δ*E _{r}* to

Figure 2

We also observe that the approximation of using equation 52b gives a proper qualitative description to the behavior of the noise, but in the low-noise regime, a quantitative assessment appears to give noise values that are low by a factor of roughly two.

We will continue our discussion in the next column.

**Jerome Workman, Jr. **serves on the Editorial Advisory Board of *Spectroscopy* and is director of research and technology for the Molecular Spectroscopy & Microanalysis division of Thermo Fisher Scientific. He can be reached by e-mail at: jerry.workman@thermo.com

**Howard Mark **serves on the Editorial Advisory Board of *Spectroscopy* and runs a consulting service, Mark Electronics (Suffern, NY). He can be reached via e-mail: hlmark@prodigy.net

(1) H. Mark and J. Workman, *Spectroscopy***15**(10), 24–25 (2000).

(2) H. Mark and J. Workman, *Spectroscopy***15**(11), 20–23 (2000).

(3) H. Mark and J. Workman, *Spectroscopy***15**(12), 14–17 (2000).

(4) H. Mark and J. Workman, *Spectroscopy***16**(2), 44–52 (2001).

(5) H. Mark and J. Workman, *Spectroscopy***16**(4), 34–37 (2001).

(6) H. Mark and J. Workman, *Spectroscopy***16**(5), 20–24 (2001).

(7) H. Mark and J. Workman, *Spectroscopy***16**(7), 36–40 (2001).

(8) H. Mark and J. Workman, *Spectroscopy***16**(11), 36–40 (2001).

(9) H. Mark and J. Workman, *Spectroscopy***16**(12), 23–26 (2001).

(10) H. Mark and J. Workman, *Spectroscopy***17**(1), 42–49 (2002).

(11) H. Mark and J. Workman, *Spectroscopy***17**(6), 24–25 (2002).

(12) H. Mark and J. Workman, *Spectroscopy***17**(10), 38–41, 56 (2002).

(13) H. Mark and J. Workman, *Spectroscopy***17**(12), 123–125 (2002).

(14) H. Mark and J. Workman, *Spectroscopy***18**(1), 38–43 (2003).