Chapter 2 - Comparing Model and Experiment

2.1Overview¶

In this chapter, we will learn about the process of doing science and lay the foundations for developing skills that will be of use throughout your scientific careers. In particular, we will start to learn how to test a model with an experiment, as well as learn to estimate whether a given result or model makes sense.

2.2Orders of magnitude¶

Although you should try to fight intuition when building a model to describe a particular phenomenon, you should not abandon critical thinking and should always ask if a prediction from your model makes sense. One of the most straightforward ways to estimate if a model makes sense is to ask whether it predicts the correct order of magnitude for a quantity. Usually, the order of magnitude for a quantity can be determined by making a very simple model, ideally one that you can work through in your head. When we say that a prediction gives the right “order of magnitude”, we usually mean that the prediction is within a factor of “a few” (up to a factor of 10) of the correct answer. For example, if a measurement gives a value of 2000, then we would consider that a model prediction of 8000 gave the right order of magnitude (it differs from the correct answer by a factor of 4), whereas a prediction of 24000 would not (it differs by a factor of 12).

2.3Units and dimensions¶

In 1999, the NASA Mars Climate Orbiter disintegrated in the Martian atmosphere because of a mixup in the units used to calculate the thrust needed to slow the probe and place it in orbit about Mars. A computer program provided by a private manufacturer used units of pounds seconds to calculate the change in momentum of the probe instead of the Newton seconds expected by NASA. As a result, the probe was slowed down too much and disintegrated in the Martian atmosphere. This example illustrates the need for us to use and specify units when we describe the properties of a physical quantity, and it also demonstrates the difference between a dimension and a unit.

“Dimensions” can be thought of as types of measurements. For example, length and time are both dimensions. A unit is the standard that we choose to quantify a dimension. For example, meters and feet are both units for the dimension of length, whereas seconds and jiffys^[2] are units for the dimension of time.

When we compare two numbers, for example a prediction from a model and a measurement, it is important that both quantities have the same dimension \textit{and} be expressed in the same units.

2.3.1Base dimensions and their SI units¶

In order to facilitate communication of scientific information, the International System of units (SI for the french, Système International d’unités) was developed. This allows us to use a well-defined convention for which units to use when describing quantities. For example, the SI unit for the dimension of length is the meter and the SI unit for the dimension of time is the second.

In order to simplify the SI unit system, a fundamental (base) set of dimensions was chosen and the SI units were defined for those dimensions. Any other dimension can always be re-expressed in terms of the base dimensions shown in Table 3 and its units in terms of the corresponding combination of the base SI units.

From the base dimensions, one can obtain “derived” dimensions such as “speed” which is a measure of how fast an object is moving. The dimension of speed is $L/T$ (length over time) and the corresponding SI unit is m/s (meters per second)^[3]. Many of the derived dimension have corresponding derived SI units which can be expressed in terms of the base SI units. Table 4 shows a few derived dimensions and their corresponding SI units and how those SI units are obtained from the base SI units.

By convention, we can indicate the dimension of a quantity, $X$, by writing it in square brackets, $[X]$. For example, $[X]=I$, would mean that the quantity $X$ has the dimension $I$, so it has the dimension of electric current. Similarly, we can indicate the SI units of $X$ with $SI[X]$. Referring to Table 3, since $X$ has the dimension of current, $SI[X]=A$.

2.3.2Dimensional analysis¶

We call “dimensional analysis” the process of working out the dimensions of a quantity in terms of the base dimensions and a model prediction for that quantity. A few simple rules allow us to easily work out the dimensions of a derived quantity. Suppose that we have two quantities, $X$ and $Y$, both with dimensions. We then have the following rules to find the dimension of a quantity that depends on $X$ and $Y$:

1. Addition/Subtraction: You can only add or subtract two quantities if they have the same dimension: $[X+Y]=[X]=[Y]$
2. Multiplication: The dimension of the product, $[XY]$ , is the product of the dimensions: $[XY]=[X]\cdot[Y]$
3. Division: The dimension of the ratio, $[X/Y]$, is the ratio of the dimensions: $[X/Y]=[X]/[Y]$

The next two examples show how to apply dimensional analysis to obtain the unit or dimension of a derived quantity.

When you build a model to predict the value of a physical quantity, you should always use dimensional analysis to ensure that the dimension of the quantity your model predicts is correct.

Dimensional analysis can also be used to determine formulas (usually to within an order of magnitude). One famous example of this is when a British physicist named G.I. Taylor was able to determine a formula that showed how the blast radius of an atomic bomb scaled with time. Using pictures of the first atomic bomb explosion, he was able to determine the amount of energy released in the explosion, which was classified information at the time.

2.4Making measurements¶

Having introduced some tools for the modelling aspect of physics, we now address the other side of physics, namely performing experiments. Since the goal of developing theories and models is to describe the real world, we need to understand how to make meaningful measurements that test our theories and models.

Suppose that we wish to test Chloe’s theory of falling objects from Section 1.1:

which states that the time, $t$, for any object to fall a distance, $x$, near the surface of the Earth is given by the above relation. The theory assumes that Chloe’s constant, $k$, is the same for any object falling any distance on the surface of the Earth.

One possible way to test Chloe’s theory of falling objects is to measure $k$ for different drop heights to see if we always obtain the same value. Results of such an experiment are presented in Table 5, where the time, $t$, was measured for a bowling ball to fall distances of $x$ between $1 {\rm m}$ and $5 {\rm m}$. The table also shows the values computed for $\sqrt x$ and the corresponding value of $k=t/\sqrt x$:

When looking at Table 5, it is clear that each drop height gave a different value of $k$, so at face value, we would claim that Chloe’s theory is incorrect, as there does not seem to be a value of $k$ that applies to all situations. However, we would be incorrect in doing so unless we understood the precision of the measurements that we made. Suppose that we repeated the measurement multiple times at a fixed drop height of $x=3 {\rm m}$, and obtained the values in Table 6.

This simple example highlights the critical aspect of making any measurement: it is impossible to make a measurement with infinite precision. The values in Table 6 show that if we repeat the exact same experiment, we are likely to measure different values for a single quantity. In this case, for a fixed drop height, $x=3 {\rm m}$, we obtained a spread in values of the drop time, $t$, between roughly $0.6 {\rm s}$ and $1.0 {\rm s}$. Does this mean that it is hopeless to do science, since we can never repeat measurements? Thankfully, no! It does however require that we deal with the inherent imprecision of measurements in a formal manner.

2.4.1Measurement uncertainties¶

The values in Table 6 show that for a fixed experimental setup (a drop height of $3 {\rm m}$), we are likely to measure a spread in the values of a quantity (the time to drop). We can quantify this “uncertainty” in the value of the measured time by quoting the measured value of $t$ by providing a “central value” and an “uncertainty”:

where $0.76 {\rm s}$ is called the “central value” and $0.15 {\rm s}$ the “uncertainty” or the “error’. Note that we use the word error as a synonym for uncertainty, not ``mistake”. When we present a number with an uncertainty, we mean that we are “pretty certain” that the true value is in the range that we quote. In this case, the range that we quote is that $t$ is between $0.61 {\rm s}$ and $0.91 {\rm s}$ (given by $0.76 {\rm s} - 0.15 {\rm s}$ and $0.76 {\rm s} + 0.15 {\rm s}$). When we say that we are “pretty sure” that the value is within the quoted range, we usually mean that there is a 68% chance of this being true and allow for the possibility that the true value is actually outside the range that we quoted. The value of 68% comes from statistics and the normal distribution.

2.4.2Determining the central value and uncertainty¶

The tricky part when performing a measurement is to decide how to assign a central value and an uncertainty. For example, how did we come up with $t=0.76 \pm 0.15 {\rm s}$ from the values in Table 6?

Determining the uncertainty and central value on a measurement is greatly simplified when one can repeat the same measurement multiple times, as we did in Table 6. With repeatable measurements, a reasonable choice for the central value and uncertainty is to use the mean and standard deviation of the measurements, respectively.

If we have $N$ measurements of some quantity $t$, $\{t_1, t_2, t_3, \dots t_N\}$, then the mean, $\bar t$, and standard deviation, $\sigma_t$, are defined as:

The mean is just the arithmetic average of the values, and the standard deviation, $\sigma_t$, requires one to first calculate the mean, then the variance ($\sigma^2_t$, the square of the standard deviation). You should also note that for the variance, we divide by $N-1$ instead of $N$. The standard deviation and variance are quantities that come from statistics and are a good measure of how spread out the values of $t$ are about their mean, and are thus a good measure of the uncertainty.

Any value that we measure will always have an uncertainty. In the case where we can easily repeat the measurement, we should do so to evaluate how reproducible it is, and the standard deviation of those values is usually a good first estimate of the uncertainty in a value^[4]. Sometimes, the measurements cannot easily be reproduced; in that case, it is still important to determine a reasonable uncertainty, but in this case, it usually has to be estimated. Table 7 shows a few common types of measurements and how to determine the uncertainties in those measurements.

For example, we would quote the length of the grey object in Figure 2 to be $L=2.8\pm0.05 {\rm cm}$ based on the rules in Table 7, since $2.8 {\rm cm}$ is the closest value on the ruler that matches the length of the object and $0.5 {\rm mm}$ is half of the smallest division on the ruler. Using half of the smallest division of the ruler means that our uncertainty range covers one full division. Note that it is usually better to reproduce a measurement to evaluate the uncertainty instead of using half of the smallest division, although half of the smallest division should be the lower limit on the uncertainty. That is, by repeating the measurements and obtaining the standard deviation, you should see if the uncertainty is larger than half of the of the smallest division, not smaller.

The relative uncertainty in a measured value is given by dividing the uncertainty by the central value, and expressing the result as a percent. For example, the relative uncertainty in $t=0.76\pm 0.15 {\rm s}$ is given by $0.15/0.76=20\%$. The relative uncertainty gives an idea of how precisely a value was determined. Typically, a value above 10% means that it was not a very precise measurement, and we would generally consider a value smaller than 1% to correspond to quite a precise measurement.

2.4.2.1Random and systematic sources of error/uncertainty¶

It is important to note that there are two possible sources of uncertainty in a measurement. The first is called “statistical” or “random” and occurs because it is impossible to exactly reproduce a measurement. For example, every time you lay down a ruler to measure something, you might shift it slightly one way or the other which will affect your measurement. The important property of random sources of uncertainty is that if you reproduce the measurement many times, these will tend to cancel out and the mean can usually be determined to high precision with enough measurements.

The other source of uncertainty is called “systematic”. Systematic uncertainties are much more difficult to detect and to estimate. One example would be trying to measure something with a scale that was not properly tarred (where the 0 weight was not set). You may end up with very small random errors when measuring the weights of object (very repeatable measurements), but you would have a hard time noticing that all of your weights were offset by a certain amount unless you had access to a second scale. Some common examples of systematic uncertainties are: incorrectly calibrated equipment, parallax error when measuring distance, reaction times when measuring time, effects of temperature on materials, etc.

As a reminder, we want to emphasize the difference between “error” and “mistake” in the context of making measurements. “Uncertainty” or “error” in a measurement comes from the fact that it is impossible to measure anything to infinite accuracy. A “mistake” also affects a measurement, but is preventable. If a “mistake” occurs in physics, the experiment is generally re-done and the previous data are discarded. The term “human error” should never be used in a lab report as it implies that a mistake was made. Instead, if you think that you measured time imprecisely, for example, refer to “human reaction time”, not “human error”.

Table 8 shows examples of sources of error that students often call “human error” but that should, instead, be described more precisely.

2.4.2.2Propagating uncertainties¶

Going back to the data in Table 6, we found that for a known drop height of $x=3 {\rm m}$, we measured different values of the drop time, which we found to be $t=0.76 \pm 0.15 {\rm s}$ (using the mean and standard deviation). We also calculated a value of $k$ corresponding to each value of $t$, and found $k=0.44 \pm 0.09{\rm s.m^{-\frac{1}{2}}}$ (Example 2.6).

Suppose that we did not have access to the individual values of $t$, but only to the value of $t=0.76 \pm 0.15 {\rm s}$ with uncertainty. How do we calculate a value for $k$ with uncertainty? In order to answer this question, we need to know how to “propagate” the uncertainties in a measured value to the uncertainty in a value derived the measured value. We briefly present different methods for propagating uncertainties, before advocating for the use of computers to do the calculations for you.

1. Estimate using relative uncertainties The relative uncertainty in a measurement gives us an idea of how precisely a value was determined. Any quantity that depends on that measurement should have a precision that is similar; that is, we expect the relative uncertainty in $k$ to be similar to that in $t$. For $t$, we saw that the relative uncertainty was approximately 20%. If we take the central value of $k$ to be the central value of $t$ divided by $\sqrt x$, we find:

Since we expect the relative uncertainty in $k$ to be approximately 20%, then the absolute uncertainty is given by:

which is close to the value obtained by averaging the five values of $k$ in Table 6.

2. The Min-Max method A pedagogical way to determine $k$ and its uncertainty is to use the “Min-Max method”. Since $k=t/\sqrt x$, $k$ will be the biggest when $t$ is the biggest, and the smallest when $t$ is the smallest. We can thus determine “minimum” and “maximum” values of $k$ corresponding to the minimum value of $t$, $t^{min}=0.61 {\rm s}$ and the maximum value of $t$, $t^{max}=0.91 {\rm s}$:

This gives us the range of values of $k$ that correspond to the range of values of $t$. We can choose the middle of the range as the central value of $k$ and half of the range as the uncertainty:

which, in this case, gives the same value as that obtained by averaging the individual values of $k$. While the Min-Max method is useful for illustrating the concept of propagating uncertainties, we usually do not use it in practice as it tends to overestimate the uncertainty.

3. The derivative method In the example above, we assumed that the value of $x$ was known precisely (and we chose 3,m), which of course is not realistic. Let us suppose that we have measured $x$ to within $1 {\rm cm}$ so that $x=3.00 \pm 0.01 {\rm m}$. The task is now to calculate $k=\frac{t}{\sqrt{x}}$ when both $x$ and $t$ have uncertainties.

The derivative method lets us propagate the uncertainty in a general way, so long as the relative uncertainties on all quantities are “small” (less than 10-20%). If we have a function, $F(x,y)$ that depends on multiple variables with uncertainties (e.g. $x\pm\sigma_x$, $y\pm\sigma_y$), then the central value and uncertainty in $F(x,y)$ are given by:

That is, the central value of the function $F$ is found by evaluating the function at the central values of $x$ and $y$. The uncertainty in $F$, $\sigma_F$, is found by taking the quadrature sum of the partial derivatives of $F$ evaluated at the central values of $x$ and $y$ multiplied by the uncertainties in the corresponding variables that $F$ depends on. The uncertainty will contain one term in the sum per variable that $F$ depends on.

In Appendix D, we will show you how to calculate this easily with a computer, so do not worry about getting comfortable with partial derivatives (yet!). Note that the partial derivative, $\frac{dF}{dx}$, is simply the derivative of $F(x,y)$ relative to $x$ evaluated as if $y$ were a constant. Also, when we say “add in quadrature”, we mean square the quantities, add them, and then take the square root (same as you would do to calculate the hypotenuse of a right-angle triangle).

The derivative method leads to a few simple short cuts when propagating the uncertainties for simple operations, as shown in Table 9. A few rules to note:

1. Uncertainties should be combined in quadrature
2. For addition and subtraction, add the absolute uncertainties in quadrature
3. For multiplication and division, add the relative uncertainties in quadrature

2.4.3Using graphs to visualize and analyse data¶

Table 10 below reproduces our measurements of how long it took ($t$) for an object to drop a certain distance, $x$. Chloe’s Theory of gravity predicted that the data should be described by the following model:

where $k$ was an undetermined constant of proportionality.

The easiest way to visualize and analyse these data is to plot them on a graph. In particular, if we plot (graph) $t$ versus $\sqrt{x}$, we expect that the points will fall on a straight line that goes through zero, with a slope of $k$ (if the data are described by Chloe’s Theory). In Appendix D, we show you how you can plot these data using the Python programming language as well as find the slope and offset of the line that best fits the data, as show in Figure 3.

When plotting data and fitting them to a straight line (or other function), it is important to make sure that the experimental values have at least an uncertainty in the quantity that is being plotted on the $y$ axis. In this case, we have assumed that all of the measurements of time have an uncertainty of $0.15 {\rm s}$ and that the measurements of the distance have no (or negligible) uncertainties.

Since we expect the slope of the data to be $k$, finding the line of best fit provides us a with method to determine $k$ by using all of the data points. In this case, we find that $k=0.61\pm 0.13 {\rm s.m^{-\frac{1}{2}}}$. Performing a linear fit of the data is the best way to determine a constant of proportionality between the measurements. Note that we expect the intercept to be equal to zero according to our model, but the best fit line has an intercept of $-0.24\pm 0.22 {\rm s}$, which is slightly below, but consistent, with zero. From these data, we would conclude that our measurements are consistent with Chloe’s Theory. Again, remember that we can never confirm a theory, we can only exclude it; in this case, we cannot exclude Chloe’s Theory with these data. If all of the data points did not sit along a straight line, we would conclude that either Chloe’s Theory is invalidated, or that some form of error in the data was not taken into account.

2.4.4Reporting measured values¶

Now that you know how to attribute an uncertainty to a measured quantity and then propagate that uncertainty to a derived quantity, you are ready to present your measurement to the world. In order to conduct “good science”, your measurements should be reproducible, clearly presented, and precisely described. Here are general rules to follow when reporting a measured number:

1. Indicate the units, preferably SI units (use derived SI units, such as newtons, when appropriate).
2. Include a description of how the uncertainty was determined (if it is a direct measurement, how did you choose the uncertainty? If it is a derived quantity, how did you propagate the uncertainty?).
3. Show no more than 2 “significant digits”^[5] in the uncertainty and format the central value to the same decimal as the uncertainty.
4. Use scientific notation when appropriate (usually numbers bigger than 1000 or smaller than 0.01).
5. Factor out the power 10 from the central value and uncertainty (e.g. $10123\pm 310 {\rm m}$ would be better presented as $10.12\pm 0.31 \times 10^3 {\rm m}$ or $101.2\pm 3.1 \times 10^2 {\rm m}$).

2.4.5Comparing model and measurement - discussing a result¶

In order to advance science, we make measurements and compare them to a theory or model prediction. We thus need a precise and consistent way to compare measurements with each other and with predictions. Suppose that we have measured a value for Chloe’s constant $k= 0.44 \pm 0.09 {\rm s.m^{-\frac{1}{2}}}$. Of course, Chloe’s theory does not predict a value for $k$, only that fall time is proportional to the square root of the distance fallen. Isaac Newton’s Universal Theory of Gravity does predict a value for $k$ of $0.45 {\rm s.m^{-\frac{1}{2}}}$ with negligible uncertainty. In this case, since the model (theoretical) value easily falls within the range given by our uncertainty, we would say that our measurement is consistent (or compatible) with the theoretical prediction.

Suppose that, instead, we had measured $k=0.55 \pm 0.08 {\rm s.m^{-\frac{1}{2}}}$ so that the lowest value compatible with our measurement, $k=0.55 {\rm s.m^{-\frac{1}{2}}}-0.08 {\rm s.m^{-\frac{1}{2}}}= 0.47 {\rm s.m^{-\frac{1}{2}}}$, is not compatible with Newton’s prediction. Would we conclude that our measurement invalidates Newton’s theory? The answer is: it depends... What “it depends on” should always be discussed any time that you present a measurement (even if your measurement is compatible with a prediction - maybe that was a fluke). Below, we list a few common points that should be addressed when presenting a measurement that will guide you into deciding whether your measurement is consistent with a prediction:

• How was the uncertainty determined and/or propagated? Was this reasonable?
• Are there systematic effects that were not taken into account when determining the uncertainty? (e.g. reaction time, parallax, something difficult to reproduce).
• Are the relative uncertainties reasonable based on the precision that you would reasonably expect?
• What assumptions were made in calculating your measured value?
• What assumptions were made in determining the model prediction?

In the above, our value of $k= 0.55 \pm 0.08 {\rm s.m^{-\frac{1}{2}}}$ is the result of propagating the uncertainty in $t$. It is thus conceivable that the true value of $t$, and therefore of $k$, is outside the range that we determined. Since our value of $k$ is still quite close to the predicted value, we would not claim to have invalidated Newton’s theory with this measurement. The difference between our measurement and the predicted value is only 1.25 times $\sigma_k$, so very close to the value of the uncertainty.

In a similar way, we can discuss whether two different measurements, each with an uncertainty, are compatible with each other. If the ranges given by uncertainties in two values overlap, then they are clearly compatible. If, on the other hand, the ranges do not overlap, they could be inconsistent or the discrepancy might instead be the result of how the uncertainties were determined and the measurements could still be considered consistent with each other.

2.5Summary¶

Measurable quantities have dimensions and units. A physical quantity should always be reported with units, preferably SI units.

When you build a model to predict a physical quantity, you should always ask if the prediction makes sense (Does it have a reasonable order of magnitude? Does it have the right dimensions?).

Any quantity that you measure will have an uncertainty. Almost any quantity that you determine from a model or theory will also have an uncertainty.

The best way to determine an uncertainty is to repeat the measurement and use the mean and standard deviation of the measurements as the central value and uncertainty. If we have $N$ measurements of some quantity $t$, $\{t_1, t_2, t_3, \dots t_N\}$, then the mean, $\bar t$, and standard deviation, $\sigma_t$, are defined as:

You have to pay special attention to systematic uncertainties, which are difficult to determine. You should always think of ways that your measured values could be wrong, even after repeated measurements. Relative uncertainties tell you whether your measurement is precise.

There are multiple ways to propagate uncertainties. You can estimate the uncertainty using relative uncertainties or use the Min-Max method, which tends to overestimate the uncertainties. The preferred way to propagate uncertainties is with the derivative method, which you can use so long as the relative uncertainties on the measurements are small. If we have a function, $F(x,y)$ that depends on multiple variables with uncertainties (e.g. $x\pm\sigma_x$, $y\pm\sigma_y$), then the central value and uncertainty in $F(x,y)$ are given by:

This can be easily calculated using a computer.

If you expect two measured quantities to be linearly related (one is proportional to the other), plot them to find out! Use a computer to do so!

2.6Thinking about the material¶

2.7Sample problems and solutions¶

2.7.1Problems¶

2.7.2Solutions¶

Solution 2.2

We start by estimating the volume of Chelsea’s closet as well as that of a pair of shoes. Chelsea’s closet is a ``walk-in closet’ with two double doors. If we know the dimensions of the door, we can estimate the width and height of the closet. Estimating the average size of a large door to be $1 {\rm m}\times 2 {\rm m}$, one face of the close will have an area of $4 {\rm m^2}$. If we estimate the depth of Chelsea’s closet to be about $3 {\rm m}$, the volume of her closet is $12 {\rm m^3}$

Next, we can estimate the size of an average pair of shoes, by modelling a shoe as a rectangular box. A single shoe has a height and width of about $5 {\rm cm}$ and a length of about $25 {\rm cm}$. A pair of shoes will thus be equivalent to box with dimensions $5 {\rm cm} \times 10 {\rm cm} \times 25 {\rm cm} = 1250 {\rm cm^3}$. This is equivalent to $0.00125 {\rm m^3}$. We can now determine how many pairs of shoes, $N$, would fit in the closet:

$$N=\frac{(12 {\rm m^3})}{(0.00125 {\rm m^3})}= 9600\approx 10,000$$

(40)

We find that Chelsea can buy about 10,000 new pairs of shoes on her trip, and still fit them all into her closet. Time to get shopping, Chelsea!