Analyzing Families of Experiments in SE: A Systematic Mapping Study
share
Context: Families of experiments -groups of interrelated experiments with the same goal- are on the rise in Software Engineering (SE). The analysis techniques used to aggregate experiments' results within families may impact the reliability of the joint findings. Objectives: Identifying the techniques used to aggregate experiments' results within families. Raising awareness on the importance of applying suitable analysis techniques to reach valid and reliable conclusions within families. Method: A systematic mapping study (SMS) is conducted with the aim of identifying the techniques used to analyze SE families and their conditions of applicability. A series of recommendations to analyze and report families are derived from families' limitations with regard to the data analysis of joint results. Results: Several techniques have been used in SE to analyze families: Narrative synthesis, Aggregated Data (AD), Individual Participant Data (IPD) mega-trial or stratified, and Aggregation of p-values. Ad-hoc procedures are followed to select aggregation techniques within families. AD or IPD stratified are the most appropriate techniques to analyze SE families. Conclusion: Analysis techniques' selection shall be motivated within research articles reporting families. Data analysis' reporting practices shall be improved to facilitate the assessment of the appropriateness of the aggregation technique used to analyze the family.
READ FULL TEXT