Why free ICAS past papers don’t exist: Behind the scenes of creating an ICAS question

The ICAS competition has been running for more than 40 years. If you’re reading this, it’s likely that it has in some way been a milestone for your schooling life, or for your children’s school life.

For generations now, ICAS has been part of the fabric of the education year, challenging bright young minds and recognising and rewarding the academic champions among students. Throughout the years, schools, teachers and parents celebrate these internationally recognised competitions as an opportunity for students to explore their strengths and passions and flex their academic talent.

Achievement in ICAS is often used to assist with placement into gifted programs, identify gifted and talented students, and raise the bar of academic recognition. As such, and in a fast-changing world, ICAS tests are continually evolving to stay relevant to educational and cultural conditions.

The questions in each ICAS paper are recognised among educators for stretching students’ applied knowledge and problem-solving skills to the very limits. Indeed, solving an ICAS question often proves just as fun a challenge for adults!

To make all this possible, there’s an incredible team working behind the scenes to achieve this level of academic rigour and creative flair. Over several months, they analyse and apply psychometric data, map out rubrics, add a twist of the unexpected, and write these remarkable questions.

If your child is preparing to take ICAS, the first time you may ever encounter an ICAS question is before the test itself: within an ICAS practice paper.

Free ICAS past papers: Our biggest FAQ

For those who are keen to get a sense of the style and language of ICAS questions before competition day, ICAS makes past test papers available for purchase on the competition website. The ICAS team is often asked whether practice test papers are free. And when we reply that they’re available at a price, our next most-asked question is why.

To answer this, we’re going give you a rare glimpse of what goes on behind the scenes with these remarkable facts:

It takes a year for an ICAS question to go from concept to completion

Yes, you read that right.

At ICAS there are no fewer than 30 test developers, plus consultant academics, designers, and analysts, all working together to devise ICAS questions. It’s a process that begins 12 months in advance of the next competition, which stands to reason when you consider that the team is producing:

• About 10 papers per subject
• each requiring between 30 to 60 questions

That’s 300 to 500 questions!

Writing ICAS questions is an art that requires question writers to use clear language with a specific outcome in mind. They must also be proficient storytellers since it is important to ICAS that questions not only test skills but are also interesting and enjoyable.

Perhaps the most challenging competency that ICAS writers need is the ability to take feedback from several inputs and act on it without losing sight of the question’s original objective.

“Test questions should remain true to their original objectives as they undergo countless rounds of feedback and editing,” says Kerry Tremaine, Head of Assessment Content – ICAS Assessments. “That’s a skill which takes time to develop.”

As well as the writing stage, each question needs to travel through five comprehensive review stages. “We look at something different in each stage,” says Kerry. “We check for the correct level and area of skill testing, difficulty, language clarity, spelling and grammar, and cultural sensitivity, to name a handful of the reviews that take place.”

Questions must also be cross-checked with others, to mitigate the risk of inadvertently hinting at another answer. “A classic example of this is when you have a very easy math question that says, What’s the name of this shape? and another question that says: Find the area of this hexagon?, both with a picture of a hexagon. Those are reasonable questions, but you can’t have both of them in the same test,” Kerry says.

Only about half of the ICAS questions we write actually make the cut

So high are the standards for an ICAS question and so rigorous the review process that, for a test of 40 questions, ICAS Assessments test developers need to write up to 80; each accompanied by supporting items such as graphs, secondary texts, and interactive graphics. That is a whole lot of writing! But it is necessary to ensure the best questions make it into the final test, while the ones that do not pass the rigorous review process remain “on the cutting room floor”.

Kerry says: “A question might be perfectly good but you find out during review that it is testing the same skill as another question. And so, one of them needs to be replaced.”

It can take a year to become a fully-fledged test developer

The skill involved in developing the top-level academic tests in ICAS is considerable. Even the seasoned former educators and academics who join the ICAS team need to expect to be honing their initial specialist test development skills for up to 12 months.

There are many nuanced aspects that need to come together to create the ideal test question. Understanding and mastering the process takes time and requires teaching within the team. Chelsea Bernal, Senior Test Development Manager – ICAS Assessments, calls this “an immersion process” in which experienced test developers continually nurture newcomers.

In addition, question creators need a strong grasp of analytics and how the performance data of past test questions could be used to inform current ones. This historical data provides valuable feedback on how students responded to the test, including whether questions were clearly understood, and attained the correct degree of difficulty.  “Even In your second year, you’re still getting help from your manager and from the rest of your team,” says Chelsea.

We test our own tests with as much rigour as we do our ICAS participants!

You might think that each year, the only thing that the ICAS test development team needs to focus their entire time on is looking forward, and producing the set of questions for the next ICAS competition that lies ahead. However, the organisation is deeply committed to its pursuit of continuous improvement— and this means first reflecting back on the past year’s papers.

Nick Connolly, Head of Analysis and Reporting – ICAS Assessments explains the ways in which the team “tests the test”:

“We look for behavioural anomalies. There will invariably be five or six questions with response patterns that are different to what we expected. So we ask ourselves: Why did the kids not respond as expected to this one? Data can tell us whether the question was confusing therefore misread, or too difficult, for example.”

He elaborates that back in the day before the team carried out its work on computers, in order to find anomalies they would have been looking for a paper misprint. But now that the competition is delivered to students online, and marked, recorded and analysed digitally, analysts can now pinpoint contextual misunderstandings through data.

Getting the question difficulty “just right”: The Goldilocks zone of test development

The “sweet spot” for an average test score across all test takers [for an ICAS paper] is about 60 per cent. If they achieve this average, and an appropriate range of scores, this indicates to the team that the questions met their objectives.

When the analytics team investigates scoring patterns, it is looking for the average scores, distribution of scores (test targeting), the spread of difficulty, and the matching of difficulty to the ability of students. This last one tells the team whether they are “pointing the questions to the right places.”

Why ICAS past papers are so valuable to those who purchase them

From what you’ve seen of the test team’s commitment to quality, it may come as no surprise to know that the ICAS practice papers come under as much scrutiny and quality control as the competitions themselves. Although they are made up of past ICAS questions – which have already passed the processes – the team still need to do more work to ensure that the questions are still relevant, and accurately reflect the current student experience of taking an ICAS test.

”That’s why parents are happy to invest what they see as a relatively small cost for setting their child up for success,” says Kerry. “Particularly those who understand the journey of a test question and what it takes to bring them to life. Many parents let us know how helpful these practice test papers are in preparing their school-age child for academic competition.”

Practice makes perfect

Having walked through the process with us, you’ve now seen how developing ICAS questions is a thorough, deeply-considered process that requires a large volume of expertise. But it’s a formula that works.

How do we know? We get to sit back and enjoy the success stories, year after year.

Visit the the shop to buy your ICAS past papers now.