HOW DO WE MEASURE SUCCESS? Thoughts on Soccer Goals, Social Impact, and Surgical Errors

This question is so fundamental, and yet too often it’s not even asked by those funding and working towards social impact.

We assume it was asked and answered by someone else at some earlier time, or that this initiative with its holistic approach to the whole person can’t be reduced to a single or even set of metrics. Or maybe it’s that we simply don’t have the time and resources to build that data-driven culture that allows us to adjust our strategy and actions based on whether our indicators of success are flashing green or red.

Let me give a simple, personal example of a soccer team I coached a few years back. If you measured success for those seven-year-old girls based on how many games we won (the traditional metric of success in youth sports, even for younger children), then I failed miserably as a coach as I think we managed to win a single game all season. If you measured success by how many of those girls enjoyed soccer and coming out to compete and play as a team every week – and ultimately how many chose to return to our recreational team the next season after only winning one game – then one could argue I was successful.

You can’t make the argument one way or the other though without answering the question as to why you signed up your daughter for soccer in the first place – what is the purpose of her being on that field?

TO WHAT END?

It all starts with what you’re trying to achieve. To what end? For your nonprofit or team, for your foundation or funding initiative or program – what are you doing, and why? Then how are you defining success? And then how are you measuring that?

Now I do strongly agree with the maxim from William Bruce Cameron that not everything that can be counted counts, and that not everything that counts can be counted.

But there is a definite role for embedding a data-driven culture into our systems, organizations, programs, practices and teams at all levels, allowing for a strong continuous flow of information between our goals, strategies, desired outcomes and shorter-term indicators. This type of culture ensures everyone is focused on the north star for your organization, but just as importantly it allows an organization to stay nimble, to adapt, and to learn. A learning organization allows for the same flow of information as our veins and arteries allow for the blood that courses through our body, where a blockage anywhere can be just as harmful.

Teams that run organizations and programs need timely and relevant feedback to adapt to a rapidly changing environment or simply put they won’t survive. Whether you want to call it a theory of change or a logic model or whatever the latest evaluation jargon is in the world of social impact, it’s critical to put yourself and your organization in strong position for self-learning and data-driven decision-making.

DELOITTE, MEASURING IMPACT, AND AMGEN SCHOLARS

I appreciated the opportunity to speak to Deloitte for their new report on strategic corporate philanthropy in the healthcare industry, and that they chose to call out the Amgen Scholars Program as a strong example for measuring impact. With Scholars – and anytime you have multiple sites running similar programs with the same desired outcomes – you have a wonderful opportunity to use comparative performance to call out data-based best practices (in our case by having all participants – regardless of host institution or country – complete the same anonymous survey.)

Critically, my conversations with the evaluator and select host institutions shaped the design of the evaluation and the desired program outcomes at the beginning of the initiative, creating the system and culture of data-driven program improvement still in place thirteen years later.

Then you’re in position at post-program leadership convenings, based on the data, to elevate the specific practice that rose to the top as most consequential for a specific metric, and to explore as a group exactly why that one institution did so well on that one item (e.g. through this process, we learned why one institution had the most Rhodes Scholars, why another had the greatest student satisfaction with their assigned faculty mentors, and why yet another enhanced students’ science communication skills most effectively).

In its ideal form with multi-year programs, this type of assessment and reflection can lead to replication of that select practice at other institutions the following year (where it aligns with the other institution’s culture); this engenders in the program community a healthy competition that leads to cooperation. Such data that benchmarks you against your peers in a non-threatening, transparent and rigorous manner can be invaluable to grantees, just as it can be for funders themselves through surveys such as the Center for Effective Philanthropy’s Grantee Perception Report.

For more on what we’ve learned from the data about running successful undergraduate research programs, check out this book on best practices across Amgen Scholars. Another key to understanding the success of a program is to build a culture and system that allows you to track longer-term outcomes, which lets us today point (for alumni whose current status is known) to 1,000 students pursuing graduate degrees in scientific fields, 280 with a PhD or MD-PhD, over 500 employed in scientific fields across 33 countries, and still others selected to such lists as the Forbes 30 Under 30 in Healthcare.

Ultimately, the right culture and system for evaluation – including how you define and measure success – should lead to more success for more beneficiaries, helping to create the empowering and supportive environment and opportunities that allow great talent to rise.

SUCCESS IS UNIQUE TO WHO, HOW, WHAT AND WHY

With respect to our lab program at the high school level – the Amgen Biotech Experience (ABE) – our latest evaluation results are here on this page on the program website. Critically, circling back to the importance of measuring success, you’ll notice that we define success differently for our undergraduate initiative than for our K-12 initiatives.

Many often assume that everything we do in science education is to cultivate future scientists. While that’s a worthy and important goal – particularly for our undergraduate initiative – if you look at the above link to our high school program evaluation you’ll notice that we’re intentionally defining and measuring success quite differently than a career in science. Interest in science and biotechnology, attitude towards science, understanding and appreciating science and the role it plays in students’ lives, gains in laboratory skills, knowledge of science. Success is defined consistent with the population served, the type and length of the intervention, and the learning outcomes we hope to achieve.

For the 90,000 high school biology students we annually reach across our 20 regions worldwide, we’re thinking about all facets of scientific literacy – as well as getting students to see themselves as a scientist. And while we are indeed preparing students to successfully enter a lab, we’re also thinking about the competencies that teachers aim to instill in all students for success in college or career, regardless of whether the path they choose is a science one. There are also great benefits in simply putting professional industry scientists and professional science teachers in the same room, as this interview of an Amgen scientist by an ABE teacher will attest to (Amgen scientist Luke Ward provides a resource for biology teachers interested in discussing race, ethnicity, genetics and personalized medicine with their students.)

Virtual initiatives such as LabXchange and Khan Academy also open an entirely new world as it relates to data analytics and measuring success, potentially allowing for real-time data-driven adjustments that can meet an individual learner exactly where they are (check out Khan Academy’s Missions today; future versions of LabXchange should also include adaptive learning features offering students different material based on their demonstrated skill level, as noted in this new article). 

And finally, though defining and measuring success is quite unique to an initiative, one can also leverage the commonalities that have emerged from similar projects for similar populations through important efforts such as the Impact Genome.

SIMPLICITY CAN BE KEY TO EFFECTIVENESS

I’ll end with one last example: Atul Guwande’s book on the importance of implementing a safe surgery checklist. Who would have thought that a simple 19-item checklist – what many of us use to remember the right groceries – could have such an impact on reducing human errors during surgery to save so many lives across the world. Guwande was clear on what he did and why. Success was clearly defined and measured. The impact is significant.

So what are you doing, and why? Yes – discuss that. Whether you’re thinking about surgery, social impact, or you’re just signing your child up for soccer.

And coalesce with your stakeholders (your peers, your grantees, the coach) around a shared understand of how to define success, and then how to measure it to inform practice and shape a learning culture for growth. Neglect that part at your own peril.

Otherwise you could be swimming – with great form and speed and resolve – but in the wrong lane.

***

The Amgen Foundation, the main philanthropic arm of the biotechnology company Amgen, is deeply committed to cultivating future scientists and scientifically literate citizens, as well as strengthening the communities where Amgen staff live and work. 

One way the Foundation achieves this is by building a data-driven culture across its portfolio and in partnership with like-minded organizations. Another is through a clear and steadfast focus on providing more young people with the opportunity (and preparation and mindset and community) to succeed in science – whoever they are, and wherever they reside. For their sake, and for ours.

To learn more about the Amgen Foundation and the $300 million it has committed to date, go to www.AmgenInspires.com and follow @AmgenFoundation on Twitter. You can also find me on Twitter at @heimlich_scott.