For the past 4 years I have had the honor of being on the SPA Fellows Nomination Committee. For the last two years I have been the chair. The process can be quite intense, but we take our job very very seriously. What happens on our committee, and then on the Union Fellows Nomination Committee, at times seems like a black box. Today I hope to make that box a bit more transparent. One thing to note before we go further is that each committee is different. Each set of people find their own way towards trying to come to a consensus. What I'll describe below is how our committee has run the last two years. As I will not be the chair, or even on the committee next year I can not say if it will continue with this format. What I can say is that in all the years that I have served, and with all the other committee chairs that I have talked to, everyone takes this processes very thoughtfully.
The process for the 2021 nominations is starting now. A few years ago it was called out and acknowledged that our fellows were not representative of our community. There continues to be an over representation of white men and people from the US and Europe compared to the demographics of those in SPA. Now once the committee meets, we can only consider the nominations that have been put forward, so if no women are nominated (as happened the first year I was on the committee), then no women can become a fellow that year. Liz MacDonald and others started a new committee with the goal of making sure qualified women, minorities, and other underrepresented groups we nominated through the development of the nomination task force (website currently under construction). While I am highly disappointed in my field for this stark bias, I am also proud of the steps that we have taken to fix it. Climate and culture is unfortunately slow to change, but it is good that we are starting the process. And yes we should do more.
So what kind of achievements can someone be nominated for to become an AGU fellow? We have three selection criteria.
1. breakthrough or discovery;
2. innovation in disciplinary science, cross-disciplinary science, instrument development, or methods development; and/or
3. sustained scientific impact
The nominee only needs to have one of these selection criteria in order to become a fellow. However, often a nomination package will show how the nominee fits two if not all three selection criteria. The committee does not take into consideration the number of breakthroughs or innovations etc. But I think there is some possible implicit bias that may occur.
The nomination package consists of a two page nomination letter, a two page CV, a two page bibliography, and three support letters each two page long. These can all help the committee really understand the impact that the nominee had on the field. A good guide for a top nomination package are: 1) The nomination letter provides a high level overview of what selection criteria the nominee is being nominated for and what supporting info for that is provided in the rest of the package. 2) The CV is a full two pages and highlights the work towards the selection criteria, any service activities, and potentially any other awards won (see identified biases below). 3) The bibliography includes the works which showcase how the nominee's work has changed and impacted the field. A few really great ones highlight these papers and underneath the citation provide a sentence or two stating how it changed the thinking in the field. 4) The supporting letters each cover in more detail one or maybe a couple of aspects highlighted in the nomination letter. Making sure the nomination package is easy to digest is incredibly important as you will see.
So, once all of the packages have been submitted, AGU goes through and makes sure they are all compliant. Usually by this time the committee has been formed.
You might think that we had covered all of our basis, but there was one more area where representation was necessary. One area of bias that we also found in the fellows was with respect to subfield. Specifically, our ionospheric and atmospheric colleagues were disproportionately under represented. Therefore we made sure that there were an equal number of committee members from each of the subfields.
In short, ensuring representation on our committee had an impact - as did the nomination task force.
Now we have our committee. We started with perhaps an unusual first step. During our initial telecon, and periodically after, we started with a discussion about potential biases we may bring to the discussion and eventual to the ranking of the nomination packages. The list of biases which our committee identified and strived to mitigate and call out when observed were:
- Gender
- Career level (retired/senior/expert vs mid or even mid/expert/senior )
- nationality/race
- Extrovert vs introvert (speaks more at conferences vs doesn’t speak up at conferences)
- Well funded institute/country vs not (e.g. able to be seen at conferences and visit other scientists vs can’t afford to travel as much)
- Large Mission participation vs smaller projects such as CubeSats, rockets, balloons ect. (more time, funding, and potential collaborations leading to more papers and co-authorships)
- Experimentalists vs theorist
- Dependance on short cut metrics (e.g. h-index which moves away from discussing the substance of the publications). - sometimes things like indices or data sets aren’t always cited properly once they become standards and are “always there” and “owned by the community”.
- Bias towards our own subfields
- People who publish/work in a small group and/or often the first author vs those who work more in large collaborative groups and/or mentor others to be first authors/PIs.
- The Matthew Effect (A paper or result being attributed to the largest name, not the person who necessarily did the work or the first author. e.g. http://www.garfield.library.upenn.edu/merton/matthew1.pdf)
- The Matthew/Matilda affect (Where men tend to get the credit or more credit than women who did just as much or more of the work. E.g. https://journals.sagepub.com/doi/10.1177/030631293023002004 )
This step was taken and found to be important for the evaluation process as it brought forward and made present the biased issues we have faced in past committees. It led to acknowledgments and mitigations throughout the evaluation process about biases within the nomination packages, and within our own discussions. We feel that this step has led us to put forward our most deserving and strongest nominees to become AGU fellows over the past few years.
The discussion about our biases was then followed by the discussion and creation of our evaluation criteria, and further discussion and mitigation about the biases which may be within our evaluation criteria. As you saw above, there is a bit of subjectiveness to them. We tried to identify and come to a consensus on what these criteria meant to us. The selection criteria was predetermined prior to looking at the nominations. We also routinely discussed the definitions and what accomplishments counted in the different categories. +No predetermined order or weight to the bulleted definitions/criteria.
What constitutes making a breakthrough and/or discovery:
- Breakthrough: an idea that once accepted, allows others to frame ideas/approach problems differently and more effectively than before.
- enabling collaborations across many subfields
- development of new instruments that have been successful in the field and lead to new* understandings
- development of new methods that other scientists have adopted and have lead to new* understandings within the field.
- Produced a data product or a method that is used on a routine basis even if not correctly cited. (Has become so routine, people have forgotten that this is either produced by someone or was not a standard product previously.)
What constitutes sustained scientific impact:
- Something that has changed the way other scientists approach a problem, perhaps on a smaller scope but cumulatively changes people’s perceptions over time.
- Enabled long-lasting collaborations leading to either significant impact within the field (one of the above criteria)
- Mentor a significant number of collaborators/scientists/students, enabling their development as researchers.
- Produced continued excellent research over the course of their career (a lifetime achievement award so to speak? )
Now that we have our selection criteria better defined we can start looking at the packages. This year we took the the following steps. Step 1) Discuss the different types of biases which we may have, and were identified in the previous year’s committee’s Step 2) Agree on evaluation criteria and process Step 3) Rank packages into the top, middle, and bottom thirds and discuss the averaged order Step 4) Initial discussion about first few packages - a) talk about how the evaluation process applies to the package b) make any changes to the evaluation definitions/process Step 5) Finish discussing packages Step 6) Rank the packages 1 - N Step 7) Discuss and finalize rankings Step 8) Go through and write up top 7 selections Step 9) Go through and write up selections. Step 10) Finalize criticism/write-ups of the rest of the packages so they can be improved for next year. All of this had to be done in the space of about a month and a half. In SPA we often have 20 - 30 nominations. Thus we end up having about two 2-hour meetings a week to go through all of the packages.
So what happens then. Our committee writes up a report. In this report we provide summaries of the nomination packages and our support of them. This report goes up to the union level. At that point it's a bit of a black box to me. I believe they follow a similar process to what we do. Then, a few months later AGU makes the announcement of who is going to be inducted into that year's class of fellows.