A | Reproduceable experiments and objective manner to analyse results |
---|---|

B | Formal computation model and asymptotic complexity results |

C | Targeted observations that allow to distinguish between possible realities |

While the engineering paradigm **A** and mathematical paradigm **B** may achieve a scientific appearance due to plots and formulas, only the science paradigm **C** specifically targets observations (e.g., empirical and theoretical results) that build conclusive knowledge about computational realities
(more details at the bottom of this page).

A hopeful quote from over 15 years ago:

The science paradigm has not been part of the mainstream perception of computer science. But soon it will be. — Former ACM President Peter J. Denning [source]

**Asymptotic computational complexity:**Even average-case/smoothed complexities can be a very poor predictor for real-world behaviour, because for finite problem instances asymptotic formulas are just course approximations.**Handpicked experiments:**A handful cherrypicked problem instances are typically not approriate for generalisations.**Disingenous/sensationalist presentation:**Colourful narratives are very exciting, but without clear and rigorous evidence it misleads readers and creates unrealistic expectations.

**Established Benchmarks:**Not available for most problems. Counteracts cherrypicked experiments, but invites cherrypicked approaches aimed to only perform well for the reference benchmark.**Reproducibility efforts [SIGMOD] [ML/NLP]:**Counteracts misreported experiments. Publicly available code facilitates follow-up studies, but does not affect any incentive structures (e.g., peer-review).

**Predictive theory [more details]:**- Theory that is either guaranteed or demonstrated to bound or predict real-world behaviour.
- Worst-case/average-case/smoothed complexities are supplemented by some study of hidden constants, or even non-asymptotic formulas.
- When approriate, theoretical models are devised for sets of problem instances

**Antagonistic experimental design [more details]:**Alongside representative problem instances, picking some challenging problem instances that are likely harder than most problem instances.**Honest presentation [more details]:**Clearly presenting novelties, but also critically discussing limitations and exploring both strengths and weaknesses in the experiments.

Computer Science | Medical/Social Sciences | |
---|---|---|

Population | Problem instances | Humans |

Sample | Problem instances used in experiments | Humans participating in experiments |

Sample Size | Typically, handful of problem instances (N < 10) | Typically, hundreds of participants (N > 100) |

Sample Type | Typically a non-random, handpicked sample | Typically a randomised convenience sample |

Sample publicly available | Typically yes | Typically no |

Theory | e.g., asymptotic complexity | predictive/descriptive theory |

Independent Variable | old approaches vs new approach | control group vs intervention |

Dependent Variable | performance measures | various measures |

Study Design | within-subject/repeated-measures experiment (without order effects) | various |

Reproducibility | Limited to "sample" | See replication crisis. |

Analysis | Description of effect sizes | Statistical/effect size analysis |

Generalisability | Completely subjective and often overstated | Critically discussed and investigated |

Scenarios and hypotheses | Which hypothetical scenarios must be considered? How can they be grouped into hypotheses? |
---|---|

Predictions | Which observations are expected in each scenario? |

Methodology | Which observations can reliably distinguish the considered scenarios? |

Replicability | How can similar observations be replicated? |

Testing | What are the observations and is it possible to repeat them? |

Analysis | Which considered scenarios can be ruled out? |

**Predictive power**for computer system models is crucial for the progression of technology in an increasingly computerised world.**Cognitive biases**even amongst elites are proven to cloud people's judgements and scientific methods aim to minimise their impact.**Carefully designed experiments**aim to disprove previous explanations for phenomena beyond reasonable doubt, establishing reliable knowledge.**Objective truths**sought after by scientific inquiry are a universal destination for people from any part of the world, culture or school of thought.