Assessment & judging panels – what is involved?

I’ve previously written how-to posts to help newcomers into public speaking and judging photography competitions. This post on assessment & judging panels especially relates to my fields – drones/technology, agriculture and the arts (not just photography) – however the principles apply to all industries.

I wrote this when I realised more people would take up requests to help assess grants, scholarships, bursaries and awards IF they had a clear idea upfront of what is typically entailed. Without info, many mistakenly feel like they’re volunteering for a difficult-to-leave committee that may turn out to be a massive time & energy sapper.

Some of what this covers:

  • How time consuming might it be?
  • How is assessment or judging usually done?
  • Lists of considerations and a plan/system
  • Pros and cons for all involved
  • What attributes does a good assessor or judge need?

Summary of my typical process:

  1. Receive the qualified & sorted applications by email
  2. Read all the application info that applicants received so it is clear what they were expected to do, and most importantly – exactly why the primary grant/scholarship/award exists. The ultimate purpose must remain top of mind.
  3. Either judge online or print & score with a pen as they’re read. I find handwriting notes on paper applications is massively more efficient & each can be easily rearranged into 3 initial main score groups: definitely not, probably yes, and the majority as ‘possibly’.
  4. Next the essential – go right back through them all and adjust scores/result order because our scoring changes incrementally between the start and finish, as only once we’ve read the whole lot do we know how each compares to all the others.
  5. Then enter scores online, plus notes – if applicable (especially if there’s a particular point that other assessors may not pick up on). Or better still – I type an email list of ‘yes’ applications, with comments. Sometimes a few ‘possibly’ additions (IE what do other judges think?) – and if there’s any that I think others are likely to think are shoo-ins but that didn’t stand up to scrutiny I list them and what researched revealed that resulted in a ‘no’.
  6. If you encouraged people to enter or gave someone info that is already publicly available online, no problem assessing those. But do explain this to the admin/organiser. If you helped someone write an application (read it before submission and/or suggested the words to use) then explain this to the admin/organiser and disqualify yourself from assessing that particular application – leave it to the other panellists. Of course the bottom line is: when in doubt – explain clearly in writing. Needless to say, if you have previously judged the award/grant/scholarship, then don’t give any later applicants any advantageous information that only a judge/assessor would know – as nepotism isn’t fair on anyone and it trashes trustworthiness.
  7. The admin/organiser then collates the assessment results – ideally sharing with all assessors three lists of applicants – the people everyone agreed on as yes, and no; plus those who received varying assessments.
  8. Ideally panellists would then have an online meeting to discuss the applicants who receiving differing assessments. If the panel includes healthy diversity, some differences are likely due to an assortment of expertise and background. Hearing the reasoning of others can change final results. Unfortunately for expediency this is sometimes not done – but for all significant opportunities, it should be. I’ve had my mind changed, and changed the minds of other judges – and that is exactly how the best award, grant and scholarship judging is run. A group discussion also helps organisers aren’t at risk of complaints from dissatisfied applicants as it demonstrates thoroughness and a united front.

Why should *you* be an assessor?

  • Assessors/judges never need formal qualifications however a thorough knowledge of the specific field is vital. You need to understand ‘industry speak’, have a good idea of what the current situation is and general life experience is something I view as essential – partly to ensure application pork pies and hyperbole are easily identified. If money and/or public recognition is involved there are people who won’t hesitate to lie.
  • The number one consideration regarding taking up an assessment opportunity: why you? If you ‘don’t look like’ the other people involved it’s likely you’ll be a valuable addition as your views are likely to differ in useful ways. If you ‘don’t think you’ll fit’ you’re probably an ideal person to include. Examples of the likely personal differences that matter, due to common under-representation (amongst assessors or recipients):
  • Regional or remote resident (or solid lived experience [not just a tourist, FIFO or temp])
  • Much older or younger
  • A woman (or man, on rare occasions where the percentage is reversed)
  • Neurodiverse (ADHD or autistic or both)
  • Different cultural or socioeconomic background
  • Much longer and/or more varied experience (EG across different industries, even if seemingly unrelated; or in different regions)
  • If ONE OR MORE OF THESE THINGS APPLY – it’s highly likely your views will be an invaluable addition to the assessment panel. If on the other hand you are almost identical to everyone else then to bluntly sum up: you are not such an asset (no panel needs more of the same).
  • Lastly – if you tick some of those boxes but are nervous, remember: ‘if not me then who, if not now then when’. Give it a go. People with trepidation are often the most needed. If you get into it and realise it’s really not for you – of course you can apologise and bow out (just don’t wait until the judging results deadline – speak up early on.)

The first judging/assessment hurdle:

An admin beaver *should* weed out every application that does not meet essential, clearly stated criteria – this saves assessors wasting their time individually checking and second guessing. (Citizenship, for example, should be a requirement for all taxpayer money gifted via grants although apparently not the case, weirdly.)

But – sometimes the basic checks aren’t done by admin/organisers so if you spot any essential omissions speak up immediately, so the other assessors can be told whether or not to include the iffy application/s. It gets messy if this isn’t done & some people may argue (unreasonably) for favourite applicants who actually don’t qualify. Bending entry rules is undeniably unfair to others & this is one hill I’m prepared to die on.

Occasionally organisers will give applicants a chance to rectify omissions but this is usually not a fair thing to do, when other applicants have been thorough. It’s obviously a good idea if specifically seeking people who have never applied for anything before or with less formal education; but not a good idea if it’s a big dollar sophisticated project and/or there are a huge number of applicants. Horses for courses: what is the ultimate aim? Applications are collated then emailed to assessors, with a firm results deadline.

What do assessors consider?

  • THE ULTIMATE ESSENTIAL: the real aim of the grant, scholarship or award – as publicly stated by the funding body and/or organiser/s. (Note – if they haven’t stated this clearly and publicly then raise it with them straight away and ensure it is in future, and check for guidance re assessing current applications; with this to be shared with other assessors, for consistency.)
  • Any application that does not meet the fundamental aim of the whole show, should be culled immediately. As an example – if assessing a bursary expressly intended to increase diversity: all applicants who fit into the overwhelming already-existing majority should be culled. It doesn’t matter if they’ve invented the cure for cancer or lead a flight to the moon; if their application does not meet the aims it is not relevant. (Frustratingly, quite often, assessors are dazzled by achievements or fame and totally fail to cull at this first essential.) If the aim is to identify genuine innovators – check that what the applicant has done is actually THEIR genuine creativity and NOT something copied from someone else (which happens every second of every day. If only I had a dollar for every wheel-reinventing project gifted tens of thousands of taxpayer funding for ‘innovation’, by state and federal governments!) Solid industry knowledge PLUS an effective google search reveals uniqueness or not (eg I know of someone gifted $10k by govt to invent a delivery drone, years after a plethora of overseas companies already had tried & tested models working commercially. To cap it off, they had never worked in any career remotely related to any tech industries or any kind of creativity). Clearly the assessors were absolutely clueless potatoes. Asking other experts for their confidential thoughts may be advisable too; ‘do you know of anyone who has designed xyz?’ If assessors have a solid LinkedIn network, it’s brilliant for this. Know what you know – and know who to ask about the rest!
  • READING THE RESPONSES TO QUESTIONS:
  • Have the questions been answered thoughtfully (but succinctly) and appear to be open and honest? Or –
  • Yes/no bare minimum responses?
  • Fashionable jargon – all the latest buzzwords – but lacking convincing substance? AI or handmade plagiarism?
  • Lots of waffle to hide a lack of relevance?
  • Super slick? Staff from large companies sometimes submit applications written by other staff members who have marketing/communication degrees. Beautifully written, designed to sell, and you won’t even realise you’ve been sold to. But closer scrutiny reveals a lack of heart; no gritty character or coalface experience.
  • NEVER FORGET it’s not a creative writing or journalism test. You are to judge real-world substance *not* eloquence. It’s heartbreaking to think on how many of the world’s most creative people have missed out on what they deserved because of their spelling and grammatical errors, or just absence of slick salesmanship of unashamed FIGJAMness. Because so many of the world’s most creative thinkers and problem solvers are not good at writing. The most unimpressively written applications may well be the most deserving and worthwhile recipients.

Assess thoroughly:

Don’t believe the application. Use Google and/or LinkedIn for checks. It’s usually quick to see if the application gells with what already exists online – or not:

  • Longevity in the one industry/dedication through tough patches
  • If a school leaver – what subjects did they study? Any extra curricular activities? Or have they clearly explained their aspirations and plans? They may have little evidence but have their heart set on it – or – just applying because a teacher suggested it. Read between the lines; look for heart.
  • Philanthropic/altruistic personalities – any evidence of them helping others in the past, *without* personal reward? (Always the best people to give opportunities to, as they already have a history of ‘playing it forward’ – so the grant/scholarship ROI is greatly amplified, and long term)
  • Career track record – a diverse career may well make them better at what they do, especially if any kind of creativity is involved. ADHD personalities have often had laps doing all kinds of things. But if they’ve worked in a stack of sales positions then moved into the drone industry, for example, there’s a 90%+ chance they’ll abandon ship as soon as a better cash cow walks past. IE their heart is in sales and if grannies are in short supply they’ll be flogging them off to the highest bidder via a roadside stall, and leave the drone industry eating dust.
  • Work related posts they’ve written (are they savvy thinkers/thought provoking?)
  • Media mentions or their website, etc. It’s usually quick to see if the application agrees with what’s online. If someone has been around for a while but there’s nothing at all online about them/what they do for a living – that’s a ‘storytime’ red flag.

You know some of the applicants?

  • The following applies when it’s a panel. If you’re the sole assessor and you have some personal connections who applied mixed with strangers then the task is best handballed on to someone else. This ensures there’s no bias or later accusations of it. But if it is a panel of 3 or more:
  • The type of people who volunteer to help assess applications are of course often the same people who encourage other people to apply. How to ensure assessment is 100% above board?
  • Firstly – encouraging someone to enter and/or giving advice that is very general in nature or info that is freely available on the application pages – unlikely to give anyone a personal advantage; it just persuaded them to make the effort to enter. Mention a personal connection to organisers but it should be fine to assess what they’ve written, unless you think you can’t be objective
  • However – if you helped word an application and/or read it before submission and/or gave them very specific information – notify organisers and the usual is to just assess all the others except those who received this individualised assistance from you.
  • I’ve sat on in-person assessment panels and when someone has a personal or business connection with an applicant, they just step out of the room during discussion/decision making re that specific application. (Similar to how local govt is meant to operate.) 
  • Factors to remember:
    1) A (diverse) panel is designed to even out various points of view & accidental favouritism or unconscious bias
  • 2) The pressure isn’t as great if there’s various categories or enough funding/awards for few people, not just one or two
  • 3) If the grant/scholarship or award is annual, applicants can usually reapply

Why being conscientious matters:

  • I’ve seen way too much feral assessment and judging by people who did not take it seriously enough. Either do it properly or bow out. Why?
  • Receiving a grant or scholarship is NOT just about the money, IF it is given to the right person. For the right person – it is a fabulous confidence boost, a public sign of recognition and faith – it can be a catalyst that helps launch a fabulous world-influencing career, help re-launch a stalled career or embark on a career change, or just enable the recipient to finally step up to the next level, when feeling frustrated and stalled. It can help priceless but unrecognised mentors and role models play it forward so far into the future that the benefits are immeasurable. This IS what it means to the *right* recipients. To those for whom ‘getting lucky’ doesn’t mean much – they are the wrong recipients. I’ve read far too many applications from people who just saw an opportunity to grab some free and easy money, or who are already either earning a stonking salary, can easily claim what they’ve asked for as a tax deduction (affordability isn’t an issue), or in a very high profile position. Or people who feign to help others who in reality never do.
  • For the above reasons it continues to frustrate me that almost NO organisers/administrators of funding opportunities of all kinds, include a vital question for *all* applicants: ‘list details of other money/in kind benefits you have received – year, amount, who from, for what (etc).’ NOT because people who have already received ‘are proven to be reliable so a safe bet for more $’ – quite the reverse. Opportunities are always thin on the ground so my personal preference is always to look for people who’ve got the goods but not been gifted a chance to speed ahead. This makes eminent sense and these egalitarian principles are what all Australian opportunities should be based on, but unfortunately often the opposite is true. The already fortunate receive further gifts while others remain stalled and unfairly unrecognised. This happens again and again. Of course if it is an award for previous achievements (usually with little or no money attached, but much appreciated public recognition), it is legit runs on the board that matters (not what someone is ‘gunna’ do in future).

Attributes of a good assessor/judge:

  • Care – about the applicants, the industry/subject, fairness and the future
  • Prepared to invest some thinking time (& same as everything – the more judging you do the easier it is)
  • Enough knowledge and/or quality contacts to accurately and objectively assess
  • Not someone who only does things for personal benefits (in cash or kind)

Summary – my method & time spent:

  1. I first ensure the organiser/funders aims are clear and publicly stated (if not I run screaming, as results are at the mercy of organiser or funder nonsense whims – IE unfair to entrants)
  2. Depending on the judging assessment sheets and the quantity – I prefer to print all applications & quickly write notes as each is read.
  3. Then I sort them into piles (definite yes, probably yes, probably no and definitely not)
  4. I read back through them all, checking scores are in sync & rejigging
  5. I google check vital points – especially that got the successful over the line, to ensure they’re legit (being a touch typist on desktop computer means this is relatively quick and easy for me, but can take time). Story tellers are demoted and under-sellers are promoted.
  6. Then if unavoidable I enter results onto a spreadsheet. But if there’s a lot of people, I email the list of definites (in order, if it’s awards), plus a list of also-rans and the definite no’s. It’s usually a waste of time tapping the whole lot into silly spreadsheet squares.
  7. I hope for a panellist discussion re any discrepancies, but if not, just wait to hear the end results; but also email organisers with any suggested improvements to ensure the best results & smoothest judging, next time.
  8. This all typically takes me between a few hours to a long half day, but I spread it over a few days for best results. Read & sort, next day more scrutiny, third day type them up while checking any points I missed. If it’s a lot of people, lots of money involved, or detailed/long applications – the total time is usually 1 day in total.
  9. Printed applications are then shredded, for privacy reasons.
  10. NOTE: Typical organiser advice includes: strict word limits for every response, encouraging applicant to use dot points and provide a list of online links eg to LinkedIn, their website, any other social media accounts; plus media stories etc. Saves assessors a heap of time! Lastly – ask applicants *why*.)

Why I like being involved:

  • Always interesting! I learn from the applicants and the process. Judging/assessing can be fun. It’s the admin person doing the real grunt work behind the scenes.
  • It forces you to do the kind of useful cogitating that we otherwise don’t get around to – eg re the state of the industry, what matters and what doesn’t
  • I find it rewarding to help others in the industries I work in and very satisfying to help ensure the genuinely best applicants are recognised. When assessors are able to have a chinwag it’s great to discuss ideas and that can mean changing results for the better.
  • I’m often the ‘odd one out’ in a group situation and that = a healthy diversity of ideas and opinions.
  • While assessment and judging positions are usually voluntary (unpaid) & there’s not a plethora of hands up, it is recognition of being thought of as experienced and reliable enough to be included.
  • I’ve particularly enjoyed helping to assess the US-based Women and Drones annual awards and the International Federation of Agricultural Journalists Star Prize for Photography. But I’ve also enjoyed local photography competition judging just as much. All just human beings doing their best, that thoughtful comments and encouragement can make a positive difference to. We should all chip in to help others in whatever way we can and never forget that countless other people have helped pave the way for us. It’s also about making Australia a better place in future, for me.
2023 Women and Drones Presentation at CES, the world's largest electronics expo, in Las Vegas (USA). Award winners from all over the world.
2023 Women and Drones Presentation at CES, the world’s largest electronics expo, in Las Vegas (USA). Award winners from all over the world.

PS:

Like all posts, as soon as I click ‘publish’, additions and clarifications spring to mind – so this will be improved and added to in future.

Feel free to share with anyone – but by linking to this whole post; not copying any of this text or quoting it without a reference to the source. While I write posts like this to help others, I chase thieves with a cudgel.

Tags: , ,