Did the 2011 Australian Census get religion right?

The results for the 2011 Australian census came out today.  We think the results of the religion question were (unintentionally) distorted by the design of the question.  Specifically, with believe that the number of Australians who are not religious is greater than the number counted as ‘no religion’. 

Counting correctly is important because census results will be used, at least to some degree, when making public policy decisions where religion is involved.  The question was formatted like this:

The Australian Bureau of Statistics (ABS) reported the religious affiliation results in a table.  On the surface, it seems that ‘no religion’ continued to get a bump.  However, there was something hidden.  The bottom of the table listed 100 percent.  However, it had an annotation that the total…

"…includes inadequately described (supplementary codes) religions and people who did not state a religion."

Strangely, there is not a row in the table that lists how large this group is.  We ran the numbers.  Turns out it’s almost one in ten people and the fourth largest group:

If approximately a quarter of those people were actually not religious for the first time ‘no religion’ would be the largest group on the Australian Census.  If you’re a strong adherent to a religion we think it would be unlikely you’d leave that question blank.

 We’ve criticised this design before.  The core of our argument was that the design reduces two questions into one and hides the ‘no religion’ question down the bottom.  This could partially explain the large ‘inadequate’ and ‘blank’ responses. It was for this reason that we proposed some alternate designs which we put to a public vote.  Our final proposed redesign is revealed for the first time below:  

This is not about activism, but appropriate measurement. Accurate measurement in the census is important.  We think there is no malice involved from the ABS, but their design has promoted an error that works against the ‘no religion’ option. Australia has changed and the design of the census does not reflect that.

Keep reading to hear our full story and proposal to the ABS.

Hit us on twitter @humansindesign or in the comments with your response.

_______________________________

Form effects behavior.  Even when it’s the form of a form.

This is not up for debate.  It’s a fact.  You can debate ‘how much’ it effects people’s answers, not that it does.  Sometimes this will be through promoting error, other times it will be through framing the question in a particular way.

The goal of form design should be to eliminate distortions to and reveal only true answers.  An extreme case where this is import is the census.  The purpose of the census is to accurately gather information from the population for the government, and other agencies, to make policy decisions.  You have to assume that any question included will be used in this manner.

A question about religion is included on the Australian census. So, you have to assume that it will be used, at least to some degree, when making public policy decisions where religion is involved.  We are aware this is touchy subject, so we will not comment on any of the issues.  Our only goal here is to gain accurate reporting of information regarding religion. 

The design used on the 2011 form can be seen below:

We have previously criticised the design of this question.  This promoted enough debate that we proposed some re-designs and put them to an unscientific public vote. To summarise, we believe the form or it’s online equivalent, which has ‘no-religion’ at the bottom,  has the potential to promote error in three ways: 

  1. People miss ‘no religion’ simply because it’s hidden under the other box.
  2. People answer ‘Other’ before considering ‘no religion’.  The census itself actually has ‘humanism’ as a suggested ‘other’.  This is an ideology or philosophy rather than a religion.
  3. People answer a religion because they listed first and frames ‘no religion’ as a direct alternative to the listed religion. We imagine this will be the case for many people who have some affiliation to a region in their past, but no longer practice.  Examples are, persons who were baptised Anglican but no longer or never practised the religion.

The extent to which it promotes error is unknown.  However, Australia has changed and the census question design does not reflect that.  Historically it was almost a given that everyone would note a major religion.  Now the rate of ‘no religion’ is climbing and the ‘not stated/inadequately described’ number bobbles around the 10% mark.  This year was no different, as the chart above shows.

Our original designs all included moving items around in the same question.  The most popular option was to move ‘no religion’ to the post preference and change the wording to ‘not religious’.

But, as pointed out to us by Jessica Enders, probably Australia’s leading expert in form design, this single question is actually asking two logically sequential questions.  By asking it in this order the question is framed differently and, we think, appropriately and fairly.

  1. Are you religious?
  2. If yes, what religion?

Jessica suggested going for broke in the next census by breaking it into two questions and, in the first year, have no pre-set response option for a religion.  Simply ask people to list their religion.  Following that she suggested listing any religion that, on the previous result, reached an certain level be listed as an option.  Essentially splitting the question and pushing the reset button.

We are sympathetic to this view, but wonder how hard it will be for the Australian Bureau of Statistics (ABS) to sort and code these responses into something useful.  Therefore, we propose the keeping the religion options as currently designed and simply splitting the question: 

Note, the wording is open to debate, but we felt using ‘identify’ is soft and easily understood.  

It’s a small change, but we believe that it would increase the people answering ‘no religion’ in two ways.  Firstly it would pick up people from the ‘not adequately described’ and ‘blank’ boxes.  Secondly, it makes making people stop and think, ‘am I really religious?’ before answering.  

The above essentially has no different answer possibilities to the the question on the 2011 census.  If form design does not matter then you’d expect exactly the same response with the re-design as with the question that was asked. I bet you don’t think that’s true.

One line of response from the ABS will probably be that changing the question makes it inconsistent with other years and consequently harder to draw trends. This is true, but because the population is changing we question the accuracy of the trend it’s drawing.  Better to start being right, than be consistently and increasingly wrong. 

Our final proposal to the ABS and Australian Government is this: test it.  The next census is not scheduled until 2016.  In the meantime it would be pretty easy to test the effect of our proposed redesign, and others, using randomised control trials.

There is no agenda in this other than accurate measurement.  It’s important.  We believe the current design promotes inaccurate reporting.  

If we’re wrong, fine.  But if we are right, it’s important.

It would also reduce people doing stupid things like this:

- Tristan @humansindesign

&

- Tom @tmnlsn

Related Posts Plugin for WordPress, Blogger...