In New Zealand, the end of the current six year period for the Performance-Based Research Fund (PBRF) is fast approaching, with its publication cut-off at the end of 2011. In Australia, there is currently a journal re-ranking exercise through the Australian Research Council (ARC). Such events impinge not just on authors submitting papers, but on journals too through submission rates, impact factors, and A* and A journal ratings. For further details, see, for example:
http://www.tec.govt.nz/Funding/Fund-finder/Performance-Based-Research-Fund-PBRF-/
http://www.arc.gov.au/era/tiers_ranking.htm
http://www.arc.gov.au/era/era_journal_list.htm

These measures create a framework in which performance both of authors and journals is assessed. The measures are all internal in the sense that it is agreement within a profession or discipline rather than reference to consequence outside this arena that determines "success".

But is this all that is needed?

To use a yacht racing analogy, rating rules determine boat design, but do not guarantee sufficiently strong links to the external world. The Fastnet tragedy in 1979, where 23 yachts were lost or abandoned in appalling conditions because seaworthiness was not a sufficiently explicit part of the rating requirements, provides a tragic example.

Of course, the consequences in academic publishing are not so obviously dire, but the question of external validation, and even validation outside a particular discipline when there is overlap with another, still remains.

Journals are ranked using various measures. One of the most common is impact factor (IF), determined by the number of citations of papers published in a journal within some specified time window. In Statistics, there is a longer time lag than in many other disciplines because published papers take longer to absorb, and new ones longer to write and be reviewed. IF is a fickle beast, with marked annual fluctuations very much affected by the presence or absence of the comparatively few papers that get high numbers of citations (at least for journals like ANZJS that publishes relatively few papers). For an interesting discussion of this and related issues, see Van Nierop, E. (2009) Why do statistics journals have low impact factors? Statistica Neerlandica 63, 52-62. There are other internal measures such as eigen analysis, but these are no more comprehensive, even given their additional complexity.

During assessment exercises for individuals, such as the PBRF, the journal rankings derived from IF and categorisation into A*, A, B and C journals are perhaps too often interpreted to apply to all papers published in a given journal. There is also a regrettable absence of external validation or even any obvious interest in considering external measures.

This is particularly difficult for applied statistics papers, the best of which may have considerable and even immediate practical consequence, even if they do not always develop substantive new statistical theory. At the other end of the spectrum, theoretical papers may take considerable time to be recognised as important. Student's seminal paper on the t-statistic provides a well-known historical example. For further detail on Student see Boland, P.J. (2000). William Sealy Gosset (Student) 1876-1937. In: Ken Houston (eds). Creators of Mathematics: The Irish Connection. Dublin: UCD Press.

Nevertheless, despite the difficulties of measurement, recognition of impact needs to be extended beyond internal measures. Sound assessment measures also need to acknowledge that not all papers in an A* journal are excellent, nor all papers in a B journal mediocre. In essence the problem in using journal rankings to assess individual papers is statistical - measuring the importance of a paper only by the journal in which it is published is to use a population measure of impact without proper recognition of the large variation in importance between published articles even in the same journal.

However, as authors and editors, we currently need to work within the established system. So, as its managing editor, I encourage you to support ANZJS by submitting good papers soon. We can currently offer fast publication for accepted papers. We want to publish more, relevant local material. We currently receive very few PhD student submissions. We want to encourage these (although they will of course still have to meet the necessary standard to be published).

Addendum – Information received from Wiley-Blackwell, 30 June 2011:

The 2010 Impact Factor for the Australian & New Zealand Journal of Statistics, which was released by Thomson Reuters last night. The Impact Factor has dropped by 25% from 0.821 to 0.618. The journal is now ranked 78/110 in the Statistics & Probability category. The 5-year Impact Factor has also decreased, but only slightly (from 0.843 to 0.811).