【算法分析】多个对比算法的统计检验方法

一、几种检验方法

（1）方差分析（Analysis Of Variance，ANOVA）——推荐1

（在两组和多组比较中，方差齐性的意思是比较各组的方差大小，看看各组的方差是不是差不多大小，如果差别太大，就认为是方差不齐，或方差不等。）

二、Holm's method详解

Where:

• Target alpha level = overall alpha level (usually .05),
• n = number of tests.

Question: Use the Holm-Bonferroni method to test the following four hypotheses and their associated p-values at an alpha level of .05:

• H1 = 0.01.
• H2 = 0.04
• H3 = 0.03
• H4 = 0.005

Note: we already know the p-values associated with each hypothesis. If you don’t know the p-values, run a test for each hypothesis before attempting to adjust FWER using the Holm-Bonferroni method. （使用之前，先计算p值）

Step 1: Order the p-values from smallest to greatest:  （p值从小到大排序）

• H4 = 0.005
• H1 = 0.01
• H3 = 0.03
• H2 = 0.04

Step 2: Work the Holm-Bonferroni formula for the first rank: （对第一个rank运行Holm-Bonferroni公式）
HB = Target α / (n – rank + 1)
HB = .05 / (4 – 1 + 1) = .05 / 4 = .0125.

Step 3: Compare the first-ranked (smallest) p-value from Step 1 to the alpha level calculated in Step 2:
Smallest p-value, in Step 1 (H4 = 0.005) < Alpha level in Step 2 (.0125).
If the p-value is smaller, reject the null hypothesis for this individual test.
(p值小于alpha，表明有显著差异)

The p-value of .005 is less than .0125, so the null hypothesis for H4 is rejected.  （第一步的最小p值和第二步的alpha值比较，前者小于后者，则拒绝空假设）

Step 4: Repeat the HB formula for the second rank （重复上述步骤给第二个rank）
HB = Target α / (n – rank + 1)
HB = .05 / (4 – 2 + 1) = .05 / 3 = .0167

Step 5: Compare the result from the HB formula in Step 4 to the second-ranked p-value:
Second ranked p-value, in Step 1 (H1 = 0.01) < Alpha level in Step 2 (.0167).
The p-value of .01 is less than .0167, so the null hypothesis for H1 is rejected as well.

Step 6: Repeat the HB formula for the third rank.（重复上述步骤给第三个rank）
HB = Target α / (n – rank + 1)
HB = .05 / (4 – 3 + 1) = .05 / 2 = .025

Step 7: Compare the result from the HB formula in Step 6 to the third-ranked p-value:
Third ranked p-value, in Step 1 (H3 = 0.03) > Alpha level in Step 6 (.025).
The p-value of .03 is greater than .025, so the null hypothesis for H3 is not rejected.

The testing stops when you reach the first non-rejected hypothesis. All subsequent hypotheses are non-significant (i.e. not rejected).   （当找到第一个不拒绝的假设时，步骤终止；所有后续假设失效）

Reference:
Holm, S. 1979. A simple sequential rejective multiple test procedure. Scandinavian Journal of Statistics 6:65-70

python代码：

# -*- coding: utf-8 -*-

# Import standard packages
import numpy as np
from scipy import stats
import pandas as pd
import os

# Other required packages
from statsmodels.stats.multicomp import (pairwise_tukeyhsd,
MultiComparison)
from statsmodels.formula.api import ols
from statsmodels.stats.anova import anova_lm

#数据excel名
excel="sample.xlsx"
#读取数据
#获取第一组数据，结构为列表
group_mental=list(df.StressReduction[(df.Treatment=="mental")])
group_physical=list(df.StressReduction[(df.Treatment=="physical")])
group_medical=list(df.StressReduction[(df.Treatment=="medical")])

multiComp = MultiComparison(df['StressReduction'], df['Treatment'])

def Holm_Bonferroni(multiComp):
''' Instead of the Tukey's test, we can do pairwise t-test
通过均分a=0.05,矫正a,得到更小a'''

# First, with the "Holm" correction
rtp = multiComp.allpairtest(stats.ttest_rel, method='Holm')
print((rtp[0]))

# and then with the Bonferroni correction
print((multiComp.allpairtest(stats.ttest_rel, method='b')[0]))

# Any value, for testing the program for correct execution
checkVal = rtp[1][0][0,0]
return checkVal

Holm_Bonferroni(multiComp)

三、补充知识

1. M个算法，每个运行N次得到数据集怎么称呼？

microarray data，微阵列数据。每个算法运行N次运行结果为一行（或一列）。

gene list，基因列表。每条基因是一个算法运行N次的结果。

2. 为什么要用Holm’s method？

Multiple testing corrections adjust p-values derived from multiple statistical tests to correct for occurrence of false positives. In microarray data analysis, false positives are genes that are found to be statistically different between conditions, but are not in reality.

The p-value of each gene is multiplied by the number of genes in the gene list （一个算法重复运行多次得到的一组数据叫做一个基因，每次运行得到基因上的一个基因项）. If the
corrected p-value is still below the error rate, the gene will be significant:
Corrected P-value= p-value * n (number of genes in test) <0.05
As a consequence, if testing 1000 genes at a time, the highest accepted individual pvalue
is 0.00005, making the correction very stringent. With a Family-wise error rate
of 0.05 (i.e., the probability of at least one error in the family), the expected number
of false positives will be 0.05.

3. 为什么p值小于0.05很重要？

H0：不同供应区域的商品单价没有差异。（零假设）

H1：不同供应区域的商品单价有显著差异，至少有一个供应区域有不同的商品单价。（非零假设）

if p<0.05:
print('There is a significant difference.')
else:
print('No significant difference.')

©️2019 CSDN 皮肤主题: 编程工作室 设计师: CSDN官方博客