Optimal design of experiments

In research and product development, you often face the challenge of understanding complex biological systems with many influencing factors – without carrying out unnecessary experimental runs. Statistical design of experiments (DoE) gives you the mathematical lever to reduce the number of required experiments to a data-driven minimum. In this way, you secure the statistical robustness of your studies and clinical sample size planning, uncover critical interactions and reach reliable, publication-ready results faster.

(Status: September 2021)

Application example

The following application example illustrates the benefit of experimental design as part of sample size planning and thus as preparation for data analysis: a manufacturer of a skin cream wants to find out in a study how their product, taking into account additional factors, affects the reduction of age spots. On the one hand, the study should yield statistically robust results; on the other hand, only as many subjects as necessary should be included.

It is assumed that, in addition to the cream, age, sex, skin type and nutritional supplements are influencing factors. Treatment success is defined as a desired alignment of skin tone within a predefined treatment period. The task is to plan this study with an adequate number of subjects in different combinations of factors and group sizes.

Below, we briefly introduce the methodological background for solving this purely data-driven task and then apply it to the example.

Methodological background

In many areas of research, development and operation (production and administration), not only is knowledge about system behaviour insufficient when innovations are introduced, there is also a constant desire to optimize products and processes. One option in such cases is to perform experiments to collect measurement data for analysis and thus characterize these systems.

Statistical design of experiments (DoE) comprises a wide range of methods that support a systematic planning of experiments for analysing specific relationships or for optimization. This also means that the experimental design is influenced by assumptions about the system and by the subsequent data analysis of the experiments. At the same time, another task is to optimize the experimental design itself in order to reduce the number of experiments for various reasons. In the experiments, the assumed input variables, also called “factors”, are set to specific “levels”, and the resulting output variables are measured.

The specific method used depends on the current state of knowledge and may be applied iteratively. Depending on available information and the objective, the following (non-exhaustive) classes of methods can be mentioned: (1) factor screening (with submethods (i) one factor at a time, (ii) full factorial design and (iii) fractional factorial design); (2) more complex models and optimization of target variables (interactions, blocking and nonlinearities, especially response surface methods); and (3) reduction of variance (in particular Taguchi-type methods).

For the example introduced above, experimental design using methods from category (1) is considered. “One factor at a time” has, similar to its non-systematic counterpart “trial and error”, the disadvantages of imbalance and inefficiency (more single experiments for the same statistical properties, or more replicates for an apparently smaller number of experiments). In addition, this approach ignores interaction effects between inputs and outputs.

The full combinatorics of a complete factorial design, i.e. all combinations of all levels of all factors, typically leads to an infeasibly large number of experiments. Therefore, in the continued application example, the frequently used fractional factorial design will be presented. This approach can drastically reduce the number of experiments required – at the cost of confounding between main effects and interactions.

Continuation of the application example

For the example above, the factor “age” will be excluded during experimental planning: on the one hand, it is not straightforward to discretize this continuous variable into a small number of levels, and on the other hand, this may simplify the recruitment of study participants. Four factors remain, for which the client suggests the following levels: cream (yes/no), sex (m/f), six different skin types (according to Thomas Fitzpatrick, 1975) and nutritional supplements (none, selenium, vitamin E). The full factorial design results in a total number of experiments of \(\,N = 2 \times 2 \times 6 \times 3 = 72\,\) (view here).

These settings already allow a reduction to a fractional factorial design in the form of an orthogonal array, as published in an overview by Kuhfeld and implemented in the R package DoE.base.

The number of necessary experiments is thereby reduced from 72 to 36 (view here). A further useful effect of this method, based on number-theoretic properties, can be exploited: increasing the number of levels leads to a larger full factorial design, but can result in fewer experiments in the reduced fractional design. In this scenario, an additional nutritional supplement can be included: astaxanthin. The full factorial design would now require \(\,N = 2 \times 2 \times 6 \times 4 = 96\,\) experiments (view here). In the reduced fractional factorial design (orthogonal array), however, the following 24 experimental runs are sufficient:

Even with this orthogonal array, all relevant information is collected: all levels occur equally often and all pairwise level combinations are represented. Based on this design, sample size planning with replicates can be performed, followed by a time-to-event analysis to evaluate the effect of the cream.