Author: Ben Wylie
Using statistical evidence to judge defensive performance
Nick Bishop’s recent article on the apparent growing divergence between analysis and statistics was surprising and disappointing. The Leinster analyst – who also worked with Stuart Lancaster for England – uses a single metric to support this, and the article seems to suggest only that there is a growing divergence between analysis and the use of tackle percentage as a viable metric of defensive performance. Indeed, some of Bishop’s own work is testament to how using statistics alongside technical and visual analysis can be persuasive and effective.
Bishop’s piece is a classic example of sporting scepticism of statistical study, and is of a piece with articles by mainstream journalists citing a number to support their case and feeling the need to include the standard disclaimer ‘statistics don’t tell the whole story’. A particular instance of poor statistical analysis does not invalidate the entire field, in the same way that a particular instance of poor tactical analysis does not invalidate that field. If anything, Bishop’s piece serves as an excellent piece of statistical analysis itself, disputing any presumption of correlation between defensive performance and tackle completion percentage by using evidence to support his argument. The next step is to work out what are effective metrics for describing different types of defence, and the 2016 Hurricanes are a good case study.
Tactical analysis: what rugby can learn from football
Technical and tactical analysis has come to flourish in rugby over the past few seasons, but is still far behind many leading team sports in terms of its detail and clarity. This is partly due to the tactical complexity of rugby relative to sports such as basketball and football: its multiple methods of scoring points bring with them numerous possible strategies to attempt to ‘optimise’ performance, and this adds layers of noise for an analyst trying to establish the signal.
Football’s single mode of scoring allows an analyst to establish a clear framework through which each player’s actions and decisions can be judged: an action’s worth can be determined by the extent to which it has increased its own team’s chances of creating a shot on goal in a high quality location, or decreased the opposition’s chances of doing so. The basis of football strategy is essentially shot creation and shot prevention, but rugby strategy cannot be condensed in the same fashion; rugby’s multiple modes of scoring – and multiple methods of achieving the same scoring action – bring with them more strategic options. One coach may decide that the most efficient way of optimising his team’s performance is by gaining territorial advantage through their open-play kicking game, while another will opt to try and break through the opposition’s defensive line with passing skills and good spacing; a player’s actions and decisions must be judged with reference to this tactical framework to a greater degree than in other sports.
As a result of football’s relative simplicity in this regard, statistical and tactical analysis of the sport has flourished. A player’s passes can be judged based on the quality of shot that they result in, and a team’s attacking shape can be appraised based on the efficiency with which it allows progression of the ball into areas of the pitch where high-quality shots can be taken.
Statistical analysis in rugby is struggling to get past the stage of ‘Player X played well because he carried the ball Y times & made Z tackles’, but potential is there at a team-tactical level: Murray Kinsella’s detailed try analyses show this, as they attempt to take the key elements of a team’s attack and use data to assess them. At a tactical level, there are very few occasions on which we state definitively ‘Team A’s gameplan was to do X and Y in order to cause outcome Z’, using visual evidence to support the thesis, in the manner that sites such as Spielverlagerung can with football.
Luck, legacy and the development of Super Rugby
When Robbie Deans crossed the Tasman at the end of the 2008 Super Rugby season, he left his successor a legacy which was impossible to supersede: five competition titles, two second-place finishes and the creation of one of rugby’s great dynasties. After eight seasons attempting to emulate his former coach, Todd Blackadder has now left his own mark on the franchise he first represented as a player in 1996 – but his trophyless tenure has been judged in a markedly different fashion.
Blackadder is a great example of the fine margins which disproportionately affect the reputations of head coaches: in the four seasons between 2011 and 2014, his Crusaders teams lost two semi-finals and two grand finals by an average of 2.5 points. (For comparison, across twenty-one seasons of Super Rugby, the grand final has been decided by an average of 12.3 points per game.) In order to fill out a profile which tallies with the assumption that something inherent to Blackadder underlies these outcomes, ambiguous qualities such as a lack of ‘tactical nous‘ are retroactively attributed – in much the same way that Arsène Wenger’s time at Arsenal cannot outrun the fallacy of the manager with the longest unbeaten streak in English football history lacking a winning mindset and ‘hard edge’, when tactical analysts have been pointing out on-field evidence for his team’s relative struggles for a number of seasons. This is not to exonerate Blackadder completely for these defeats, but simply to point out that perspective is necessary, chance is unavoidable and that criticism of a coach needs to be evidence-based and specific for it to be robust. The Crusaders in this period have been consistently among the strongest teams in Super Rugby, and fortune is intertwined with the nature of a hybrid league-playoff competition.
Blackadder’s tenure also tells us something about how Super Rugby has developed as a competition as the professional game has developed. In the eleven seasons between 1996 and 2006, the tournament saw twelve different franchises reach the semi-final stages but only three (Blues, Crusaders and Brumbies) win the title; in the ten seasons since 2007, by contrast, seven different winners have emerged from an equal-sized field of twelve semi-finalists. This evidence suggests that, in the early life of a competition, organisations that are able to implement strong performance cultures are able to gain a huge relative advantage over other their competitors who lag behind. However, as time passes more and more teams are able to replicate these successful habits, and in Super Rugby the relative advantage for a franchise like the Crusaders over its opponents diminished as rugby moved into its second decade of professionalism – even as their own methods of maximising performance were likely improving. This is similar to the manner in which the distance between the All Blacks and their international competitors has been growing less significant over time – not due to any lowering of their standards, but because the development of professionalism has pushed more and more teams into the successful development of high performance environments – and is more important context to bear in mind when we judge the legacies of coaches like Blackadder.
The tangible impact of skills coaching
Identifying and accurately charting the effectiveness of a particular coach is one of the most difficult parts of the analysis of any sport. Too often correlation between the identity of a certain individual and results – either positive or negative – is interpreted as causation, belying the unique nature of every professional team and the complexity of factors which contribute to performance. The corollary of this is not that causation cannot be attributed in this way, but that it is important to be careful in the way we attribute it and to place reliance on on-field evidence when doing so.
One aspect of rugby where conventional wisdom and on-field evidence align is the belief that regular, top-class skills coaching can have a substantial impact on a team’s performance, and for the 2017 season the Blues have recruited one of the best in the business.
How we talk about team culture
“If you say rugby unifies New Zealand, if you say better people make better players and if we are going to pay young men six times the average wage when they are 21, we have to accept criticism if we dip below the moral line.”
Discussion of the concept of team culture in rugby is dominated by the All Blacks’ coaching and playing environment. This environment is lauded for the level of on-pitch performance it produces, but is often derided by detractors for its apparent hypocrisy: Steve Hansen declaring that “good people make good All Blacks” is juxtaposed with numerous negative stories involving symbols of this culture such as Aaron Smith and Dan Carter. This reading stems from the way in which sporting behaviour is traditionally couched in moral terms: we talk about a player showing integrity, bravery and heart and extrapolate from this an individual moral judgement, when in reality all we can really deduce is a professional’s aptitude and willingness to work hard to achieve a desired goal.
Parsing Hansen’s comments on his team environment, we can see that he too is guilty of this elision – it is the latter which he essentially interprets as the key to his “good people”, “no dickheads” policy: “For us we are trying to find people of good character…because if they have got good character they will have good character on the track under pressure. Invariably the people who are dickheads off the track are the ones who wilt when they are on it.”
Hansen’s dealings with the media are instructive in the way that they prioritise the team environment above all else: we can see this in his treatment of the Owen Franks case last summer, and his circling of the wagons in the wake of Steven Luatua’s signing with Bristol. Gregor Townsend’s handling of the controversy surrounding Conor Murray’s standing leg is in the same vein: a statement designed to address the events not in their wider context (player safety in the case of Murray and Franks, and a professional player’s freedom of choice in the case of Luatua), but in the context of their effect on the team environment.
Technical analysis of the activity of NZ’s ball-carriers after the tackle, looking at their first 6 games of 2016:
The respective Autumn campaigns of Ireland and Australia sat at opposite extremes of the stylistic spectrum: Michael Cheika’s side alternated between incisive wide attack and moments of frustrating indiscipline and poor decision-making, while Joe Schmidt’s carried powerfully and was exceptionally accurate at tackle and breakdown. Despite claiming wins over New Zealand and Australia, a number of issues in attack which have been present throughout Schmidt’s tenure recurred in the 12 point loss to the All Blacks in Dublin, during which they spent long periods deep inside opposition territory and came away with no tries. This performance comes with the caveat that Jonny Sexton, Robbie Henshaw and CJ Stander – three key figures in their attack in Chicago – had all left the field within half an hour, but we have seen enough of Ireland’s attack with those players present over the past three seasons to tentatively assume that scoring five tries against the world champions was the exception that proves the rule: in order to be so ruthlessly successful, the attacking system they operate requires accuracy and intensity which is unsustainable over a series, season or World Cup cycle.
Contrast this to Australia, whose 1-3-3-1 system looks to have been enhanced by the addition of former New Zealand skills coach Mick Byrne to the coaching staff; in amongst the senseless infringements and maddening turnovers, they have been able to create wide line breaks regularly and finish these chances rather clinically. Their offloading game has clearly developed as the season has progressed, but Reece Hodge’s two ill-advised efforts in the first half are evidence that their decision-making and execution in this area still needs work. Sefa Naivalu’s effort on the left wing in the last five minutes of the test is another excellent example of their required technical progress: after manipulating his arms above the tackler he attempts to transfer the ball without readjusting the height of his hand, and the ball follows a path from high to low which is impossible for his teammate to regather. (Compare this to the supreme work of Anton Lienert-Brown – as Will Greenwood highlights here – who excels at lifting the ball into space for his team-mates from low to high.) As these skills continue to develop, however, Australia’s wide phase attack – which is already one of the best in the world – will continue to grow, and gives Michael Cheika a clear focus around which to build his gameplan.
The diagrams below (inspired by the stellar work of Dutch football blog 11tegen11.net) represent an attempt to visualise both teams’ attacking styles by mapping common passing combinations from November’s test match. Each line between players represents two or more passes between these players (with passer and recipient indicated by the direction of the arrow), and the width of each line corresponds to the number of passes. Where a line emanates from one player but does not have a direct recipient, it indicates that the player made two or more passes from that position to a multitude of players but no more than one to any single player. The diagram does not attempt to map the x- and y-coordinates of each pass; arrows pointed in the direction of the right touchline indicate passing combinations made when the team was playing from left to right, and vice versa.