Future role of rural professionals in a big data world

The impact of new and emerging technologies likely to influence New Zealand's primary industry is getting a lot of coverage within the media and at various conferences at the moment. There is a constant stream of commentators providing their views on the impact of disruptive technologies on the production, processing and marketing of our primary products - ranging from a business as usual approach with nothing really changing other than gradual incrementalism, through to elevated hysteria of apocalyptic proportions whereby everything from farm to consumption is up for systemic and irrevocable change.

Regardless of where you may sit along the spectrum, there is no doubt we are on the cusp of significant technological change within the agricultural and horticultural sectors with an expanding range of technology based tools targeted at improving on-farm productivity and sustainability through enhanced decision-making capabilities.

As part of my Winston Churchill Memorial Fellowship I have been considering the impact of new and emerging technologies occurring on-farm and across the value system, as well as the future role of rural professionals in the provision of professional advice and services to their farming clients.

So what might the future look like where farmers have access to more information and data than their rural professionals?

Farmers already have access to a wide range of on-farm information and data through a number of technology platforms including; cloud-based accounting packages, farm management and production software, data sensory technology and climatic information data. We can also expect that the amount of data available to farmers and rural professionals will continue to grow with the explosion of social networking sites, search and retrieval engines, data networks, media sharing sites, stock trading sites, news sources and so on.

We are entering an era of big data - data sets that are characterised by huge volumes of both structured and unstructured data, received from multiple sources, at ultra-high velocity and variety on a day-to-day basis. The role of the future rural professional in this big data world could be even more important as they work with their clients in assessing the quality and relevancy of data to use from the enormous amount of information available from a wide variety of sources.

Big data provides opportunities for rural professionals to analyse and mine data sources to explore patterns and relationships hidden in large volumes of raw data, and in doing so providing valuable insights and analysis that could ultimately lead to better farm management decisions.

This will require rural professionals to develop their data mining capability and become familiar and proficient with technology and machine learning platforms required to mine big data. Rural professionals will need to be competent in interpreting data and understand how this integrates within the farm system to be able to challenge information and assumptions made or being considered by their farming clients. Importantly, they will need to understand the impact of biological variability on precision measurement and decision consequences of using precision measurement on biological production systems.

This may well give rise to a new group of rural professionals, including data analysts, information technology consultants and data modellers, which is consistent with findings of a recent IBM report The Quant Crunch - How the Demand for Data Science Skills is Disrupting the Job Market. The report identifies that machine learning, big data and data science skills is a projected growth area and there was a strong need for new training programmes in these fields.

A limiting factor of the wider rollout of big data in the primary industry is the sheer velocity of information generated, and the computing power required to quickly process high volumes and variety of data particularly in areas with limited broadband access. But given how quickly technology is developing, this may have a short lag time.

Ultimately, the value and effectiveness of much big data depends on the human operators tasked with the role of understanding it and formulating the proper queries in extracting relevant information available from big data sources - the nexus between biological variability and precision measurement.

Rural professionals will have an important role in providing intuition, curiosity and depth of industry knowledge in the world of big data, without which we run the risk of myopic on-farm solutions and accepting that 'correlation is good enough'. In this new world it will be important for rural professionals to develop an expanded understanding of big data and machine learning to guide and motivate their clients in the use of information generated to improve on-farm decision making processes.