Q4.1 improvements to statistical testing

by Matilda 22. February 2011 03:11

More sophisticated methods are now used in most statistical tests that are performed when data is weighted.  In particular, Taylor series linearization is used when computing the standard errors.   This results in a change in automatic statistical tests (i.e., the tests that determine whether cells on tables are highlighted as being significant or not).  The Q4.1 Reference Manual contains more information on this.  The key benefit of this change to the testing is that tests are more powerful when the weight is correlated with one or more of the variables used to construct the table (i.e., selected in the blue and brown drop-down menus).  There is an additional benefit for agencies that work with government clients: the testing will generally be more consistent with the way that testing is conducted by government statistical agencies.

There have been two additional changes to statistical testing.   It is now possible to specify a minimum sample size used in testing (in Edit | Project Options... | Statistical Assumptions).  And, corrections for multiple comparisons now ignore any cells in a table where no p-value could be computed (most commonly, this is for columns containing no data).

Bookmark and Share

Comments

3/19/2011 4:21:02 AM #

Scott

"The default min sample size for sig testing appears to be n=2. I
> have no problem with that, but are the tests modified in such
> cases, e.g. when n < 30 the t distribution would be used instead of
> its normal approximation ?
>
> Further to this, there is a common (mis)apprehension in the market
> place that no result is "reliable" if the column n < 30 approx ...
> I don't subscribe to that view myself (i.e. I believe that _all_
> results are reliable, to a greater or lesser degree). Nonetheless,
> it would be good to have some discussion/clarification on the
> point."

Scott

3/30/2011 2:11:24 AM #

Tim Bock

Normal approximations to t-distributions are not employed (i.e., if the t is appropriate, it is used regardles of the sample size).

I entirely agree with your point about sample size and reliability!  Smile

Tim Bock

Comments are closed