The non-profit sector plays a substantial role in providing programs and services that contribute to Canada’s social, economic, and civic health. The role of non-profit organizations in collecting and mobilizing evidence deserves more attention because the sector is a key pillar in building the evidence base for policy.
What is evidence-informed policymaking? This approach to policymaking uses the best available research and information on program results to guide decisions at all stages of the policy process.[1] Properly integrated, evidence can be used effectively to identify what works, highlight knowledge gaps, track outcomes across programs/services and effectively allocate resources in budget processes.
While the upcoming Innovation in Evidence conference[2] will touch on these topics in greater detail, select conference speakers have been asked in this piece to focus their expertise on answering this vital question: What are the paths forward for creating an enabling environment for the sector to support evidence-informed policymaking and service delivery?
— Lisa Lalande
The building blocks of evidence: moving towards an integrated data ecosystem
Tris Lumley
The non-profit sector has a wealth of data on social issues such as homelessness, poverty reduction, criminal justice, and mental health. Yet, it is important to acknowledge that data, by itself, is often not meaningful until it is analyzed and understood within a “data ecosystem”[3] context. This means that data must be processed, analyzed, contextualized, and translated so that it is useful to both policymakers and practitioners. In other words, it should be used to identify unmet needs, shape program and policy development, and understand more broadly what is working and what isn’t.
In addition to the non-profit sector, the data ecosystem includes individuals, governments, funders, businesses, and academic institutions. Working together, these stakeholders gather, use, and share data in an efficient and ethical way to realize community and systems objectives.
Robust organizational data management practices are essential for organizations and the sector to flourish in this ecosystem. While individual boards need to prioritize data capacity building within their organizations, we cannot underestimate the size of the sector’s data challenge. Poor data literacy and a lack of technical skillsets limit opportunities for organizations to engage with data-related issues in a strategic way. It is not feasible for every non-profit to have its own data analyst as the required skills are not easy to find, train, and develop. They also cost money, and many organizations struggle to find the resources to invest in this skillset. Funders are therefore needed to help charities offset the costs related to these efforts.
To realize the benefits of the data ecosystem, we need smarter responses, shared resources, and platforms that allow everyone to benefit from data and analysis. Non-profit organizations need to have clear paths to contribute to shared datasets and the benefits for their participation need to be made explicit. Solutions like the Ministry of Justice Data Lab take a resource that exists and build on it for everyone’s benefit.
Barriers to building the evidence base
Chad Nilson
Canada’s non-profit sector represents an open laboratory bursting at the seams with measurement possibility. The proximity of organizations to social problems and solutions, the ability of their staff to engage Canada’s most vulnerable, and their inherent spirit to “do better,” makes this sector a necessary cohort to involve in evidence-based policymaking and service delivery.
While I firmly believe that interest in contributing to the evidence-base is high, the reality is that few organizations hold the technical knowledge or wherewithal to do so. The harsh reality of our sector is that many organizations operate with the focus of simply staying alive. As funding for this sector continues to be limited to “programs and services” it is becoming more difficult for organizations to find operational dollars — let alone the capital to hire an evidence gatherer. This leaves two considerations: (a) fund organizations to operate — therefore allowing them to look more closely at their own role in evidence; or (b) fund capacity-building within organizations so that they can gather the evidence required to stay operational.
Another challenge facing this sector is that even when an organization is committed to building evidence, they rarely get enough time to do so. Each year, organizations implement programs and initiatives stamped with an expiry date. No matter how successful the program is, funders tell them to build a sustainability plan, seek ownership in the community, and find funding elsewhere. Within the common three- to five-year funding periods provided, there is seldom an opportunity to collect adequate evidence required to properly inform policy or future service delivery. As a solution, funders should commit to longer-term program and service investments supported by proper performance monitoring, ongoing reporting, and effective measurement practices. Furthermore, funders should not be afraid of funding a program/initiative more than once.
A third challenge is the large and heavily-barricaded gap between organizations and decisionmakers (especially in government). The reality is that a long line of funding officers and agents sit between these two, filtering communication in both directions. To improve and deepen non-profit contributions to the evidence-base, the flow of knowledge and experience — combined with good data and lessons learned — must occur unfettered.
What Works Centres from philanthropy’s perspective
James Hughes
The UK’s What Works Centres[4] make research knowledge readily understandable and useable. Government administrators in fields ranging from education to criminal justice are taking advantage of the work product of these new evidence institutions that minimally clarify and translate existing research into accessible knowledge and create new knowledge that is relevant to government and other leaders. Why is philanthropy looking at this exciting new development?
Philanthropic organizations are becoming increasingly interested in understanding the existing evidence base. This interest is due in large part to wanting and needing to ensure that their granting and investment decisions align with the best evidence available. They want to understand the level of risk they are assuming when making a decision, appreciating that where there is little or no evidence in a given area the risk of failure increases significantly. Those working at philanthropic organizations are in the business of risk-taking but, to be responsible with their resources, they want to know as much as possible about those risks.
There is another more important reason. In the age of systemic-thinking and acting, it is essential for philanthropic organizations to understand the context in which they are working. System mapping, ethnographic analysis, and public opinion polling are a few of the tools philanthropy is deploying to better comprehend the barriers to improving individual and community wellbeing. Evidence is another such component. Evidence provides philanthropic organizations with the intellectual capital to support change agendas with government, private and non-profit partners, and affected parties.
Enter What Works Centres. They hold the promise of providing philanthropy (and others) with the deep understanding of past successes and failures relating to relevant interventions and, in doing so, equip intervenors with the language, research results, evaluation frameworks, and economic realities to guide systemically-oriented conversations and relationships. As such, philanthropic organizations have a public interest in making the evidence base as accessible as possible to changemakers who need to be sure they are spending valuable financial and political capital in the most effective ways.
Philanthropy is uniquely positioned to promote What Work Centres as the linchpin in this effort to bring evidence to bear in systemic change processes. Philanthropy can assist in co-sponsoring the launch of What Work Centres in their areas of interest, provide convening resources to ensure Centre objectives flow from the priority needs of key actors and stakeholders within the relevant sector and, finally, disseminate and deploy Centre work products to changemakers who are working in key areas of reform.
The What Works movement is in its infancy in Canada and philanthropy has several reasons to get behind it, not least of which is improved decision-making by all levels of government.
Establishing integrated social practice hubs to achieve better outcomes
Cal Corley and Dale McFee
Across Canada, the non-profit sector plays a significant role in supporting governments in helping solve complex social problems. In fact, non-profits are the program and service delivery partners of choice for all levels of government in our country. But together, we must all do better. Do we know what types of social programming – or which interventions – actually work? We need to strive to measure and evaluate impact rather than focus on outputs. In other words, it is the improvement in people’s lives that matters.
The winds of change are upon us. Governments and other funders are renewing their focus to improve social outcomes, and the bottom line is that they cannot afford not to. Working with evidence will have significant implications for non-profit organizations who want to stay in the game over the long term.
Governments are increasingly concerned about the ever-rising cost to provide human services, while at the same time they are not seeing corresponding improvements in social outcomes at individual or community levels. Saskatchewan, for example, has one of the highest expenditure levels of government programs per capita of any province. In parallel, it is experiencing significant social issues that have not improved despite the level of expenditures (e.g. family instability, lower rates of educational attainment, youth suicide). Community organizations undertake much of this work on the government’s behalf. Last year in Saskatchewan, government distributed approximately $500 million through contracts to the non-profit sector to undertake this work.
So why is Saskatchewan not seeing substantial improvements in social outcomes? A multitude of factors may explain this, and Saskatchewan is not alone in facing these challenges. For example, there is little independent evidence about which types of human service programming or interventions actually work. The non-profit sector generally does not have access to government-held social data. The ability for government to experiment or prototype is limited; and measurement and evaluation of social programming and interventions are typically focused on measuring inputs and outputs rather than impact and outcomes.
Increasingly, governments and funders want to support and invest in social and human service interventions that measurably contribute to improved outcomes. There is also a growing understanding that if the non-profit sector is to continue to be their army on the ground, a system-wide approach will be needed to build outcomes capacity among non-profits. Organizations should not, and cannot, be expected to go down this road alone – nor can we have them go in another direction that is not aligned to data-driven local need.
Such an “ecosystem” approach must draw upon the best that the public, private, non-profit, and philanthropic sectors can contribute to improve and align the system to achieve better social outcomes. This systems approach must be data-centred and evidence-based, co-designed with shared outcomes and an emphasis on co-measurement. These efforts would be complemented by new data sharing legislation and supporting policies, which enable social data to be shared across the ecosystem while at the same time protecting privacy and shifting toward impact and outcomes measurement. Establishing an Integrated Social Policy and Practices Evidence Hub[5] – taking the best of the UK What Works model – will provide necessary support across the system.
Notwithstanding a long-standing culture of organizations working in relative isolation from one another, the efforts in transitioning to an ecosystem approach will yield many opportunities across the sectors. But most importantly, this systems approach holds the best chance we have of reducing system demand and improving outcomes at the individual, family, and community levels.
Evidence beyond impact
Emma Taylor-Collins
Last month in The Philanthropist, Senators Ratna Omidvar and Terry Mercer called for better support for and more data on Canada’s third sector.[6] Data provides a valuable source of knowledge to help us understand the challenges and opportunities faced by charities and identify future trends. But what about the data and evidence charities themselves produce and use?
At the Alliance for Useful Evidence, an open network of more than 4,000 individuals and organizations committed to promoting the use of evidence in decision-making, we think it is crucial that charities have the support they need to embed evidence in their work. Much of what we hear in the UK about how charities use evidence relates to impact: charities are coming under increasing pressure to demonstrate their impact[7] and improve transparency because of the scarcity of resources. As a result, there is a lot of advice out there to help charities with impact measurement, from organizations such as New Philanthropy Capital (NPC) and Evaluation Support Scotland.
But charities can also use evidence in other ways, as we found in our recent report, Evidence for Good.[8] We heard from the loneliness charity The Cares Family, which used open data to identify the extent of loneliness in the UK; homelessness charity Llamau, which used evidence produced by universities, at low-cost, to improve their staff training; and Scottish Citizens Advice Bureaux, which researched the impact of changes to the welfare system using the findings to inform Scottish government policy. The charities we spoke with noted that evidence can be useful — not just in proving their impact, but also in improving the way they work. What’s more, evidence doesn’t have to be costly – several charities we spoke with made use of existing evidence. This path can be more efficient, cheaper, and robust than charities producing new evidence themselves.
We think that supporting charities to use evidence in these different ways is vital for a thriving sector.
What comes next?
The Innovation in Evidence conference is meant to chart a path forward for building Canada’s evidence ecosystem, with a strong focus on the role of the non-profit sector in collecting, mobilizing and moving evidence forward.
The speakers featured in this paper, among others, will explore the following themes at the Innovation in Evidence conference:[9]
- Generating evidence (hierarchies and standards of evidence, evidence supply and demand);
- Translating evidence (communicating findings to end users);
- Adopting evidence (incentivizing adoption, creating feedback loops);
- Funding evidence (identifying sustainable funding sources and scaling up);
- Enabling the non-profit sector (building capacity, exploring outcomes-based funding arrangements); and
- Transitioning to a What Works Centre (selecting organizational models, funding structures).
—
[1] Pew-MacArthur Results First Initiative (2014). “Evidence-Based Policymaking: A guide for Effective Government.” Washington: Pew Charitable Trusts.
[2] The Innovation in Evidence Conference is an upcoming event on improving evidence-informed policymaking that will be held in Regina, Saskatchewan, in November 2018. Co-hosted by The Community Safety Knowledge Alliance and The Mowat Centre’s Not-for-Profit Research Hub (Mowat NFP), the Innovation in Evidence Conference will bring together international and domestic experts in evidence-informed policymaking to share emerging trends, discuss lessons learned, and provide fresh insight into the challenges facing policymakers, practitioners, researchers, and academics in their quest to determine “what works.”
[3] For more information about the “data ecosystem” concept, please see Cave, J., Gyateng, T., Lalande, L., & Lumley, T. (2018). “Collaborating for Greater Impact: Building an Integrated Data Ecosystem. Toronto: Mowat NFP. Available at: https://mowatcentre.ca/collaborating-for-greater-impact/
[4] What Works Centres are one organizational approach to building capacity for evidence-informed policymaking and practice. More on What Works Centres https://www.gov.uk/guidance/what-works-network
[5] The Hub is a model of integrated practice originating in Saskatchewan. The model identifies individuals or families with acutely elevated risk factors that cannot be addressed by a single agency alone and mobilizes human service agencies toward a targeted and timely response specific to individual/family need.
[6] Omidvar, R. & Mercer, T. (2018). “Government Has a Role in Strengthening Canada’s Charities and Non-Profits: Senators Mercer, Omidvar.” The Philanthropist. Retrieved from https://thephilanthropist.ca/2018/10/government-has-a-role-in-strengthening-canadas-charities-and-non-profits-senators-mercer-omidvar/
[7] The Select Committee on Charities. (2017). “Stronger charities for a stronger society.” UK: House of Lords. Retrieved from https://publications.parliament.uk/pa/ld201617/ldselect/ldchar/133/133.pdf
[8] Alliance for Useful Evidence. (2018). “Evidence for good: How charities use evidence to boost their influence and impact.” Retrieved from: https://www.alliance4usefulevidence.org/assets/2018/07/Evidence-for-good-AfUE-v4-.pdf
[9] Lalande, L. & Cave, J. (2018). “Innovation in Evidence Background Paper: Strengthening Canadian Efforts to Identify What Works (and What Doesn’t) in Canadian Social Policy”. Toronto: Mowat NFP. https://mowatcentre.ca/wp-content/uploads/publications/174_IinE_background_paper.pdf.
