Machine learning is the use and development of computer systems that are able to learn and adapt without following explicit instructions through the use of algorithms and statistical models to analyze and draw inferences from patterns in data. Nowadays, machine learning is widely used, such as in the YouTube algorithm to give the user videos that they would likely watch, or to help with the trading of stocks by suggesting certain trades. Recently however, machine learning tools have been used to increase the power of hypothesis testing.
Hypothesis testing is a common statistical method for testing an assumption, for example seeing if changing the color of your room would cause you to be able to focus better. Amazon has recently presented a new method of online hypothesis-testing that factors side information without requiring dozens of different experiments for dozens of control groups.
The way they did this was using a contextual vector, a common machine learning idea, to capture information about the context from where the experiment is taking place. They observed that if the “contextual vectors contained real information about the experimental context, their addition increases the statistical power of the hypothesis-testing method, or its ability to identify true phenomena”(Amazon science blog). Additionally, they were able to prove that even with contextual vectors being set in place, they can still enforce a predetermined limit on the hypothesis-testing false-discovery rate (the frequency in which the method accepts a false hypothesis as true).
The reason behind why this context is so important is that the context vector can be used to adjust the false-discovery budget available for each hypothesis. The degree of adjustment is determined by a function with a turnable parameter that depends on the results of previous hypothesis tests, since the function learns to make more useful adjustments as testing progresses. What this means is that the accuracy of the AI’s statistical power of a hypothesis directly correlates to the quality of the context given.
Source: https://www.amazon.science/blog/machine-learning-tools-increase-power-of-hypothesis-testing