Tuesday, April 23, 2019

Diving down a rabbit hole...

Over the past couple of weeks, I've been trying to track down data to understand how agricultural crops have changed over the last 40 years or so - and largely this has been rather frustrating.

I wanted to obtain annual regional estimates of the winter crop area of oilseed rape (which the swans love), barley and wheat between the 1970s and 2010s - and that doesn't sound too hard when you know that each year this information is collected from farmers across the country in the June Agricultural Census...

My knight in shining armour - the June Agricultural Census:
The June Agricultural Census was administered by the Ministry of Agriculture, Fisheries and Food Statistics Census from the 1970s. So each June, farmers across the country report the area of each crop on their farm (e.g. rape grown for oilseed) and this data summarised by Parishes. Then the MAFF produced 10km² summaries from these.

So I went on a mad google for this data and after a couple of days it appeared that this data was kept in the National Archives. I was ecstatic and started planning a spontaneous trip to London to visit the Archives.

Freedom of Information Request needed to get these data...
But then, closer inspection of the webpages told me that this data wasn't actually freely available to the public, even if I went down to the Archives.  I needed to submit a Freedom of Information (FOI) request, detailing my project's research aims and methodology, for them to reconsider opening up the records again for the public - so of course, I quickly submitted the FOI requests for the records I needed but it could be up to 20 days before anyone would get back to me about this.

The duration for this project is really only about 4 months, which is really short for a research project. I couldn't just sit and hope that in 20 business days the FOI request would be approved (basically a month!). So I begun looking for other resources that might have summarised data from these records, such as books and online data repositories.

There are Agricultural Statistics in the basement of the library!!!
You would not believe my excitement when I found the volumes - Agricultural Statistics: England and Wales; Agricultural Statistics: United Kingdom; and The Digest of Agricultural Statistics; in the basement of the UEA Library. 😉 Here there was data, from the June Agricultural Census, on the area of each crop summaried by counties. They didn't cover all the years I needed but this was a START!

Following 5 days:
She frantically copies the area of each crop within each region from the volumes - almost as though these volumes (which are really rarely sought) will be taken from her at a moment's notice, and emails libraries - here there and everywhere, in search for the years in which volumes are missing from the UEA collection. 

Down a rabbit hole...
But then it turns out that the county and region boundaries used in these volumes have changed a lot over the years (an example below) and I can't actually use these data as it doesn't match with the region boundaries of the other external factors (e.g. water quality) we are measuring 😔
→ Back to searching for ways to access the June Agricultural Census data I go...

Map of England and Wales with Region Boundaries, taken from The Digest of Agricultural Statistics, 1991
Map of England and Wales with Region Boundaries, taken from The Digest of Agricultural Statistics, 1996
Final note:
It is extremely important to think ideas through and play around with preliminary datasets before diving straight in. Thinking through what results are needed to answer your question, and therefore what data you need to collect or find, and whether obtaining this data is feasible - this planning process is vital to ensuring that time and resources are used effectively. If anyone is interested, there is a fantastic book out there - Ecological Census Techniques - where Bill highlights his Reverse Planning Technique.