Reliability and validity

Since its creation, the Octogram® has gone through 4 versions. Each revision was made to increase the accuracy of the test and make it more consistent.

When looking at the reliability of a psychometric test, factor analysis is used to determine just how well the test performs. (For an in-depth review of just what that is, read the Wikipedia article on factor analysis.)

Another test for reliability is to test subjects with several “known” instruments and then compare how those results compare with the new instrument. Tests used for this cross-reference analysis included the Big 5, the SOST communication style inventory, the Schein career anchors inventory, and the THBT career interests profile.

The following information is simplified for understandability. The original information is available in Dutch, if you are interested, and can be provided on request.

First, we need to look at how well the traits remain “internally consistent”. As each person takes the test, do they tend to rank items relating to, for example, “Pioneer” at the same level consistently? There is actually a number for this called Cronbach's alpha. An alpha of at least .70 is thought to be “adequate”. So, how does the Octogram® measure up?

(data calculated from 843 candidates)

From the above table, you can see that all of the traits meet the .70 rule and some, such as “Anchor”, far exceed it. As a side note, the “Symbol” column actually discloses this test's Dutch origins: the Dutch word for anchor is verankeraar. Hence the VA.

Each trait seems to be measuring only one thing (the alpha is above .70), and it seems to be measuring what we think it's measuring because we have compared the results against other, previously proven instruments.

The Octogram® is a “circumplex model”. A circumplex is a way of mapping the results of a personality test into a visual structure. In a circumplex, the shape of the graph is usually represented as a circle or polygon. Correlations of personality scores should resemble a simplex form where opposing types have low correlation and close types have a high correlation. The next step is to see if we have the traits of the Octogram® arranged correctly. We expect that Pioneer and Networker would have a good relation with one another; we would also expect these scores to have a negative relation with traits on the other side (distal) of the graph, such as Anchor and Analyst.

Let's take a look at the correlations:

Don't be scared. Here is a summary of what that all means. We expect some traits to have a high positive correlation. The correlation between Pioneer and Networker is .47 (that's good), the correlation between Strategist and Achiever is .14 (surprisingly low), the correlation between Anchor and Analyst is .42 (that's good too), and the correlation between Helper and Team Player is .45 (that's good).

Strategist and Achiever do not show much relationship to each other, which was not expected. This does not seem to affect the validity of the test or the arrangement in the circumplex because the other condition is that traits on the opposite side of the graph should have a low correlation. Let's look at those:

So, things that are close have a positive relationship (except Strategist and Achiever) and things that are far away are very negative, so we can reliably say that we have good traits, they are measuring what we think they measure, and that the graph is an accurate representation of how the traits relate to each other.