Chapter 1 for Tackling Wicked Problems in Complex Ecologies

1

Tackling Wicked Problems in Complex Evaluation Ecologies

Rodney Hopson and Fiona Cram

Technologies live in complex ecologies.
The meaning of any one depends on what others are available.

—Turkle, 2011: 188

“Complexity” is taken to mean complicated when actually it means that simple interactions among different agents can produce emergent order.

—Agar, 2004a: 412

Recent global, national, and local shifts suggest our world is growing more intolerant, bitter, and divisive as multiple sectors and stakeholders scramble for identity, purpose, and control. With the rise of global terrorism and discrimination, the massive effects of climate change, and epidemic proportions of poverty and greed around the world, questions are being raised about the recognition of and support for peoples who are already underserved and underrepresented. For some peoples this is new, but for many it is just a continuation of decades, if not centuries, of entrapment in ostracized spaces and places. Vulnerable peoples can be found in every society, no matter what religions dominate, in developing and developed countries, in the global north and the global south, and under every political regime. Indigenous peoples, women and girls, LGBTI+ peoples, dissident peoples, to name but a few, have been pushed to the margins of their home places because they are not quite the right “fit” and are somehow less acceptable or desirable. If this marginalization spans generations, it is likely that people will have internalized the trauma of wars or enslavement that first led to their marginalization. It is also likely that many will have internalized a belief that their current place is the natural order of things, believing the mythology of their deficits.

Wars and dictatorships have also forced people to leave their homes to travel to presumed safer, kinder places. However, the bodies of children washing up on beaches signals the danger of these attempts, along with the grief and loss of parents who cannot ensure a future for their offspring. If these displaced people survive their journeys across water they may find themselves in a “no-man’s-land,” interned in makeshift refugee camps waiting, waiting, waiting for an invitation to move forward. For some, self-immolation is the only form of protest they have left amid the anger, sadness, and hopelessness that descend on them as months in limbo turn into years. Even staying in a home place can be difficult as drought and famines bring hunger, malnutrition, and death.

Those who live in first world comfort have become somewhat inured to the images on their television screens of the despair of others in far-flung places. Even so, their home place is no guarantee of well-being. In their own backyard, early preventable death (for example, by heart disease, by suicide) is a symptom of their home place’s tolerance for inequity, racism (and other “isms”), anxiety, isolation, and loneliness. Humanity and our social relations are in crisis.

And amid humanity’s crisis, the natural environment is steadily being degraded and lost because nation-states are driven by imperatives other than sustainability (for example, profit, development). Sometimes these pressures are from without, as superpower countries manipulate political climates and corrupt leaders to ensure that minerals and oils are mined and exported. These same superpowers are known to force local farmers off their traditional lands as they buy up great swaths of food-producing land to ensure freedom from hunger for their own people. Even if a country is committed to environmental protection and sustainability, global water and air currents can undermine these efforts as pollutants produced by other nations have an impact on their home place. Our natural environment is as much in crisis as our humanity, and the butterfly effect of globalization is real: “intensification of worldwide social relations which link distant localities in such a way that local happenings are shaped by events occurring many miles away and vice versa” (Arnove et al. 2013: 1).

This introductory chapter has three goals. First, the chapter will expand on the definition of wicked problems in complex ecologies of evaluation. Second, this chapter will situate responsibilities, relationships, and relevance as core evaluation themes central to the study and practice of evaluation, building off both knowns and unknowns in our field. Finally, the chapter provides a forecast of the chapters in the book, including setting the context for each chapter and contributor. As such, each chapter will be grounded in real case examples that illustrate how evaluators in complex ecologies of practice, policy, and praxis address wicked problems and reflect on the core themes of the book.

Wicked Problems

Over four decades ago, Rittel and Webber (1973) contrasted “wicked” problems with tame problems, indicating that the former are less easily defined and cannot be treated, as tame problems can, using linear analytics. Rather, wicked problems are socially complex, multicausal, and highly resistant to resolution. Although they were referring to social planning problems, the term wicked problem has since been applied in many contexts where indigenous and other minoritized groups experience injustice and inequities. Wicked problems drive housing segregation, racial and gendered stereotypes and discrimination, disintegration of schools, rising crime rates, and decreased access to healthy food and its related health benefits, to name just a few consequences. They are inherent in the alarming trends of wealth being concentrated among increasingly fewer and fewer people across the world. The wicked problems of humanity also affect the natural system and have deleterious consequences for the fabric of our global welfare. As Thompson Klein (2004: 517) describes, these problems “are emergent phenomena with non-linear dynamics, uncertainties, and high political stakes in decision making.”

The call for action that “wicked problems” might have once prompted has faded somewhat, though there are increasing signs that these wicked problem notions are reappearing (Kolko 2012; Wilber and Watkins 2015; Williams and van’t Hof 2016). In this volume, Mertens and Boland (Chapter 5) introduce us to “super wicked problems.” These issues are characterized by four key features that stress the urgency of evidence-informed understandings of these problems and evidence-informed solutions; namely, “Time is running out; those who cause the problem also seek to provide a solution; the central authority needed to address it is weak or non-existent; and, partly as a result, policy responses discount the future irrationally” (Levin et al. 2012: 123). Jahn (2008: 3) writes that responding in an informed way to these problems will be possible only “if society’s capacity for taking action is . . . increased in a sustainable manner and its knowledge base is deepened and broadened.” Rather than be silent and reactionary to the set of shifts and dynamics that reverberate in multiple sectors—education, health, social service, and others—evaluators have a role to play in these times, redefining and representing key principles, ideas, and evidence to inform solutions across a plethora of globally connected arenas. What should the responsibilities of evaluators be? How will evaluators forge relationships with key stakeholders amid this changing sociopolitical milieu? And, what is the relevance of evaluation now and in preparation for the decades ahead?

What Role Do Evaluators Play in Complex Ecologies?

When “complex ecologies” was introduced as a conference theme at the 2012 American Evaluation Association (AEA) conference, several evaluators wondered about the choice of the word ecologies in the conference title. They questioned whether systems or context would be more appropriate terms within the current field of practice. We argued then that ecology offers a broader and deeper lens to capture the nuance, meaning, connectivity, consequences, and potential future implications of wicked problems. And although our colleagues’ terms were not far off, their understanding of ecology in evaluation was less considered, and they were a little resistant to its introduction. None of our colleagues, however, questioned the notion of complexity, even though evaluation has been a late adopter of the term (Bamberger et al. 2016; Byrne 2013; Forss et al. 2011; Nwake, 2013; Rogers 2008; Wolf-Branigin 2013).

We argue that our adoption and championing of the term ecology offers a broader and deeper lens for informing evaluative thinking and analysis about intervening in the complex problems facing our world. Although the concept of ecology is not new in evaluation, its use in this book expands the concept beyond a strict environmental focus to encompass the multilayered complexities that define our current existence. Thus, opening evaluation to ecologies encourages and facilitates a discussion throughout the book of wicked problems, of inequity and inequality and their impacts on change initiatives, and of our roles as complexity, contextually, and culturally responsive evaluators of these initiatives.

Evaluation in complex ecologies spans government, nongovernment, philanthropic, tribal, and community settings. It is essential within sectors that address “wicked” and “superwicked” problems. Evaluation has the potential to add immense value to unpackaging, understanding, and responding to such problems where and when we confront them around the world. However, as a key component of accountability, evidence-based work, and intervention strategies, we evaluators have all too often focused narrowly on methods, tools, and theories. This approach has prevented evaluation from being at the table to inform solutions to real-world problems. Now, more than ever, there are community-level, tribal, national, and international stages for evaluators to contribute to understanding the complex ecologies of inequality and inequity before it is too late. Complex ecologies demand that evaluators be concerned with relationships, attend to responsibilities, and focus on the relevance of their work (see the following discussion).

Ecology, like complexity, is not a concept that evaluators tend to use. Ecology represents the diverse natural, physical, and organizational realities and settings of projects, programs, and policies. The physical aspect of an ecology recognizes our built environment and opens an ecology to a critique based on principles of universal design; that is, whether physical spaces (for example, accommodation, business places) are accessible and usable by all peoples regardless of their age, ability, disability, or size (Human Rights Commission 2010; UniversalDesign.com 2016). From an evaluation perspective, there is an additional inquiry about whether, and how, physical spaces facilitate and support initiative success. Within health this form of inquiry dates to Hippocrates in 400 BCE, with a recent review finding that the built environment of health care providers can contribute to patient safety (Huisman et al. 2012).

The “organizational” aspect of an ecology is akin to the human or social system or interactions between and among people. This aspect includes political, patriarchal, and other hierarchies that at a company, community, or country level work to contain and constrain people within what is (often inaccurately) considered to be their “proper place.” Capra (1997: 6) writes that it is necessary to see “patriarchy, imperialism, capitalism, and racism [as] examples of social domination that are exploitative and antiecological.”

When social systems are studied the role of the natural system is often minimized or viewed as a constant (Liu, Dietz, et al. 2007). The natural realities of ecology are captured for instance in indigenous perspectives that recognize the interconnected nature of humanity with the natural world. Hawaiian Hannah Kihalani Springer (1997, in Meyer 2008: 219) says, “I am shaped by my geography.” Manulani Meyer (2008: 216) adds, “Land is more than a physical place. It is an idea that engages knowledge and contextualizes knowing.” Similarly, Cajete (2000) writes about Indigenous communities living a “symbiotic” life in which the natural world is a cocreator of that life and its accompanying “symbolic” culture. These interdependencies, among people and between people and the natural environment, ensure well-being and survival.

Evaluators are encouraged to lift their heads up from decisions about method to look about and think about place and space, to contemplate the land on which they stand and the cultural contexts in which they work. Meyer (2008: 224) writes, “Our senses are culturally shaped.” They are therefore influenced by what we have each uniquely experienced by growing up and being in place, in an ecology. Meyer (2008: 217) calls this our “sensual history” and describes the land as “an epistemological cornerstone”:

Indigenous people are all about place. Land/aina, defined as “that which feeds,” is the everything of our sense of love, joy and nourishment. Land is our mother. This is not a metaphor . . . Consideration of our place, our mother, is the point here. And she is more than beautiful, or not. She is your mother. (Meyer 2008: 216; original emphases)

Other disciples have embraced ecological thinking for some years. In this volume, we highlight more deliberately how the ecological paradigm has manifested. A key figure in Robin Lin Miller’s (Chapter 4) disciplinary background of community psychology, James G. Kelly, and his colleagues write, “The essence of the ecological perspective is to construct an understanding of interrelationships of social structures and social processes of the groups, organizations, and communities in which we live and work” (Kelly et al. 2000: 133). Kelly (2006) has also written autobiographically about the influence of place on his own identity: “My identity was bounded by terrain as well as experience.” His expression of identity here resonates with that of Meyer earlier. Invoking ecologies therefore extends to and recognizes the tangible and intangible resources and materials available (or not) in settings where evaluators do their work. A key thread of this book suggests adapting an ecological framing to enhance evaluation methods, approaches, analysis, and meaning making.

Ecologies are complex because wicked problems contribute to the marginalization of peoples based on their race/ethnicity/color, sex/sexuality/gender, age, religion, or other points and intersections of difference from a cisgender, white, male, able-bodied norm. In this context, “Wickedness isn’t a degree of difficulty . . . A wicked problem has innumerable causes, is tough to describe, and doesn’t have a right answer” (Camillus 2008: 100). Wicked problems are intensely political because ameliorating a wicked problem is often about the redistribution of goods and services to those who may be “less deserving” in the common sense of their wider society. Or a solution may mean putting the needs of social systems aside so that the problems of the natural system can be prioritized and fully attended to.

Our rationale for this volume is that, by deferring to the complex ecologies in which we work, we capture more insight into vulnerability and marginalization, as well as into power and privilege. The views of those who can speak out loud are heard, as well as the voices and stories of those who are frequently silenced. We can come to grips with hierarchies, patriarchies, capitalism, and other “isms” that oppress, harm, and kill people and the environment (Bookchin 1982). This meaning making about a complex ecology can then inform our evaluative thinking, our analysis of complex solutions to wicked problems, and our ability to represent all players within that ecology. This complex ecology perspective additionally offers expanding disciplinary interest in evaluation in ways that strengthen understanding of evaluation in less discussed and frequented topical areas and sectors. This is essential, as segmented, disciplinary-bound thinking and problem solving is limited in its ability to tackle the problems of society (Thompson Klein 2004). Jahn (2008), for example, maintains that his call for broader, deeper knowledge (see earlier discussion) will only happen through the orderly transcendence of disciplinary boundaries. At the same time, we should not be naïve to the fact that evaluation in complex ecologies requires good resourcing.

The 3Rs: Relationships, Responsibilities, and Relevance

Evaluation in complex ecologies needs to attend to the 3Rs: relationships, responsibilities, and relevance. Critical to increasing our usefulness in complex ecologies are the relationships that enhance our ability to connect with and transform communities, the responsibilities we have to ensure that our work meets the diverse and dynamic needs of stakeholders and other interested groups, and a more grounded, functional, and integrated understanding of the relevance of our work. When we gain this understanding as a profession, evaluation can and does inform policy, strategy, programs, and projects.

The theme of relationships involves the call for evaluators to identify the key interests, interactions, variables, and stakeholders amid dynamic and complex issues at program, policy, and project levels. Kushner’s (2000) “personalizing evaluation” stance has all to do with relationships when he writes:

I want to look more closely at what educational programs represent and how we, as evaluators, relate to them. To do so, I will come at the issues from a rather oblique angle—through a consideration of mortality. This is not merely an artifice, a playful disguise, as it were, for a more serious argument. It is, rather, based on a view that our sense of morality and our fears of it are an understated but significant theme in our continuing struggle to construct a democratic society. (26)

Evaluation is personal. We ponder ways in which speech, acts, behaviors, interactions take place and with whom, controlled by and affected by other variables, power relations, and contexts. Evaluators who practice in complex ecologies value the relationships they build or strengthen in the process of evaluating. As Kushner (and we authors of the book) recognizes, how we understand programs, policies, and innovations through individuals and groups and communities of individuals is the crux of the evaluation matter.

Questions that drive the relationships dimension include:

• What key interactions, variables, or stakeholders do we need to attend to (or not)?
• Whose interests and what decisions and relationships drive the context of the work?
• How can we attend to important interactions amid competing interests and values through innovative methodologies, procedures, and processes?

The theme of responsibilities is about propriety. At a glance, it involves evaluators recognizing the complex and complicated nature of the third aspect of the Joint Committee on Standards of Educational Evaluation. The authors write that propriety refers to “what is proper, fair, right, and just” in evaluation (Yarbrough et al. 2001: 106). The sum of those standards, however, portrays a more delicate evaluation, especially when it means developing agreements, acknowledging and addressing conflicts, and understanding rights and responsibilities among all in the evaluation process. Evaluators who work with wicked problems in complex ecologies must attend deliberately and intentionally to appropriate standards and guidelines that promote best practices and principles. It means being technically and ethically sound in the process, calling truth to power in ways uncomfortable and dissenting. Questions that drive the responsibilities dimension include:

• What responsibilities, inclusive of and beyond the technical, do we evaluators have in carrying out our evaluations?
• In what ways might evaluation design, implementation, and utilization be responsible to issues pertinent to our general and social welfare?
• How can evaluators ensure our work is responsive, responsible, ethical, equitable, and/or transparent for stakeholders and key users of evaluations?

The theme of relevance has to do with being accurate and meaningful. Although notions of accuracy imply precision, they do not necessarily imply meaningfulness or transparency. Understanding relevance extends more than through assignment of p or t values but implies understanding the cultural. Relevance drives the evaluators’ intent to be culturally, contextually, and technically precise, accurate, and meaningful in measures, questions, indicators, and variables. Questions that drive the relevance dimension include:

• What relevance do our evaluations have in complex social, environmental, fiscal, institutional, and/or programmatic ecologies?
• How can evaluators ensure that their decisions, findings, and insights are meaningful to diverse communities, contexts, and cultures?
• What strategies exist for evaluators, considering our transdisciplinary backgrounds, to convey relevant evaluation processes, practices, and procedures?

What Is Known about Evaluation in Complex Ecologies

When most evaluators think of complexity, before the most recent splurge of texts and articles focused on evaluation and complexity, Michael Patton’s (2011) book and work in developmental evaluation pointed us in the right direction. The developmental evaluation approach is not business as usual in traditional and conventional evaluations with formative and summative expectations. Patton describes a developmental approach to evaluation more suited for change, turbulence, and adaptation at program levels rather than conventional approaches to the field that identify clear, specific, and measureable outcomes at the outset of the evaluation.

To address what might be most appropriately framed within an “ecological context,” Patton discusses what might be more than evaluation of methodological approaches. Much too often our evaluation colleagues invest a great deal of time and energy selecting and justifying the appropriateness of one or the other evaluation method for addressing particular (often very complex) problems without having invested sufficient time and effort in considering the “ecology of the place,” context, and drivers of presenting problems.

We interpret Patton’s concern regarding this often unexamined application of evaluation methods as the tendency to give short shrift to context and place in problem definition, framing, and understanding. Arguing for his developmental evaluation approach, Patton suggests that the approach is both appropriate for complex situations and, in contrast to traditional program evaluation tendencies, offers different ways of understanding the purpose and situation in which the evaluation occurs; the focus and target of evaluation, modeling, and methods; evaluator roles and relationships; evaluation results and impact; approaches to complexity; and professional qualities (see Exhibit 1.2 in Patton 2011, 23).

This volume, without taking a specifically developmental approach (or any approach, for that matter) to the topic of complex ecologies in evaluation, incorporates how evaluators with complex, contextual, and culturally responsive lenses take multiple evaluation approaches to help solve problems and challenges in diverse ecological contexts of inequity and inequality. In these ways, this volume advances Patton’s notions of complexity without presuming that a developmental evaluation approach is the only way to understand and address recurring issues of inequality and inequity.

Quite typically, what evaluators do when confronted with complexity is to find linear graphic conceptual models (logic models, theories of action or change, or some variation thereof) to depict or describe links between what programs assume their activities are accomplishing and what happens at each step of the way (Weiss 2000). Our way of revealing underlying assumptions is a tried-and-true craft of developing program theory (Rogers et al. 2000). Although our attention to graphic conceptual models has broad applicability across program design, implementation, and evaluation cycles and proves them useful in multiple evaluation approaches for acting as guiding framework and serving as a point of synergy across stakeholders, these same graphic conceptual models tend to oversimplify complex evaluation settings, rarely capture generative or adaptive program events, and/or deemphasize possibilities for multiple pathways (Lawrenz and Huffman 2006).

This volume recognizes seminal contributions from an interdisciplinary and international collection of thought on complex ecologies, such as those who use graphic conceptual models effectively (Funnell and Rogers 2011), realist principles (Pawson 2013), systems concepts and thinking (Williams and Hummelbrunner 2010). Systems thinking, for instance, has proven useful to unpacking social and economic problems that drive the vexing issues of “inequalities,” and inefficiencies that our profession strive to address (Williams and van’t Hof 2016). Indeed, considerable attention has been given in our practice literature to “system thinking,” along with the importance of “interdisciplinary” and “context,” as important factors in calibrating effective methodological evaluation approaches to addressing complex problems (Rog et al. 2012; Thomas 2004). However, we argue that a more comprehensive and granular lens for discerning and addressing “complexity” is required; specifically, one that incorporates both systems and context considerations in ways that inform and encourage more cooperative ways to understand and define the problems we address and the methodologies that might be more appropriate in addressing them.

Already, evaluators in one of the official journals of the field, New Directions for Evaluations, have advanced notions of complex evaluation ecologies (without referring to them as such). Their special issues in the last fifteen years have lifted topics for understanding settings and dynamics in ways unique to the field. They include special issues on evaluation as a democratic process (Ryan and DeStefano 2000), responding to sponsors and stakeholders (Mohan et al. 2002), frameworks for understanding context (Rog et al. 2012), and program evaluation in cooperative extension organizational systems (Braverman et al. 2008).

In addition, recent evaluation books focus on complex ecologies from within a particular discipline (for example, health, social services). Although acknowledging complexity within their discipline, many of these volumes only touch lightly on the importance of relationships, responsibility, and relevance. Some of those more concerned with relationships and responsibility examine evaluation from the point of decision making about the care of individuals. For instance, Stahl’s (2011) book focuses evaluations of complex family situations to make the best decisions for children. Although lifting important understandings of the field, the special issues and books in evaluation do not directly highlight the three subthemes of relationships, responsibilities, and relevance.

Addressing equitable and democratic evaluation is also relatively new in the field. Donna Mertens’s book, Transformative Research and Evaluation (2009), is comparable to the present volume. To Mertens, transformative researchers and evaluators acknowledge the need for systems and ecologies to change in order to facilitate the well-being of communities and individuals. Relationship and a respect for people are highlighted, along with practitioners’ social justice responsibilities. Equity focused evaluation has been the subject of recent monographs sponsored by the UNICEF Evaluation Office (Segone 2012) and web-based material through EVALPartners, a consortium of professional organizations in evaluation around the world (2014). The basis of equity-focused evaluation involves making judgments and assessments and providing strategic lessons of “policies, programmes, and projects concerned with achieving equitable development results” (Segone 2012: 7). In looking at equity dimensions of interventions, equity-focused evaluations rely on identifying gaps between privileged groups and those marginalized (Dean-Coffey et.al, 2014).

Despite the recent surge of interests in equity-focused evaluation, Jennifer Greene (2006) reminds evaluators that there is a legacy of evaluation theorists and practitioners committed to democratic social justice, equality, empowerment, and emancipation. This volume, much like Greene’s analysis of the elements of democratically oriented evaluation theories, takes the position that these same evaluators inevitably raise questions about responsibility, relationships with critical stakeholders and brokers in the evaluation, and relevance of the evaluation to those most challenged or vulnerable. The democratic, deliberative, participatory, critical, cultural, contextual, and indigenous evaluation knowledge frameworks advanced by Greene (2006) are implicitly and explicitly acknowledged by authors in this book.

What We Can Learn from Other Disciplines and Applied Orientations

Another key thrust of this book is that we believe that our evaluation toolbox of approaches to understanding and addressing complex social and economic problems can and should be enhanced by learning and incorporating more from other disciplines that have focused on the issues of “complexities.” In this section, we draw on lessons learned from other disciplines and applied orientations that may be applied to evaluation.

Lessons from Ecological Studies and Indigenous Peoples

Within indigenous ontologies (that is, nature of being) the natural system has cosmological or spiritual significance (Howitt and Suchet-Pearson 2003). Ecological and spiritual caretaking responsibilities and decision making are therefore often intertwined and not disconnected from social and natural systems. The world/ecology of indigenous peoples is relationships—with one another, with the environment, and with the cosmos. Henry and Pene (2001) describe this as supporting an “economy of affection.” Relevancy is about upholding and possibly fighting or advocating for indigenous sovereignty and the right to self-determination (Smith 2006):

Relationship is the cornerstone of tribal community, and the nature and expression of community is the foundation of tribal identity. . . . The community . . . is the context in which the person comes to know relationship, responsibility, and participation in the life of one’s people.

Spirituality is integral to the lives of indigenous peoples as they relate to one another, to their world, and to the universe (Smith 2012). The incorporation of spirituality in one’s worldview and way of knowing is not unique to indigenous peoples. For example, when a Royal Commission of Inquiry into Genetic Modification was held in Aotearoa, New Zealand, in 2000, Māori found themselves objecting to genetic engineering alongside churches and other citizen groups concerned about the spiritual ramifications of this technology (Royal Commission on Genetic Modification 2001). The spiritual system is explicitly added to the understanding of ecology developed here.

Lessons from Ethnographic Complexity in Anthropology

Michael Agar’s (2004a, 2004b) work explores how evaluators might gain lessons in consideration of how ethnography might be understood through a complex adaptive system lens. Within his description of ethnographic complexity lies a rich vocabulary of emergence, landscapes, and patterns. In linking anthropology to complexity, he settles on narratives of connections and contingencies through time to explain drug epidemics specifically and to describe how any study is done. For applied work, like evaluation, Agar (2004a: 416) calls for us evaluators to relinquish the use of one particular technique to describe “the complicated ecology of organizations and ideologies.” Instead, it is for us to use our eclectic skill sets to practice evaluation rather than overreliance on one particular method, model, or framework to answer complicated issues that surround the ecology of evaluands that make up social programs and policies. Practicing ethnographers and anthropologists who carry out evaluations do so recognizing that evaluation operates as a systemic study of a cultural system, and we bring our own “preadaptive traits” into our practice and study (Camino 1997; Copeland-Carson 2005).

Lessons from Ecological Complexities in Environmental Policy

In Marian Chertow and Daniel Esty’s 1997 book, Thinking Ecologically: The Next Generation of Environmental Policy, the policies and practices of that emergent field (environmental policy) were not “confrontational but cooperative, less fragmented and more comprehensive, not inflexible but rather capable of being tailored to fit varying circumstances” (1997: 4). Their call for a more systems-based and interdisciplinary approach, with appreciation for context, is at the core of our tenet on how evaluation practice could be enhanced by conceptualizing “complexities” within an ecological framework to the study of wicked problems in the field. In ushering new ideas of ecological thinking in environmentalism, the Chertow and Esty book recognizes the interrelated and interconnected nature of environmental policy with particular focus on the role of human element in this complex web, not unlike the role of Kushner’s ideas of personalizing evaluation (2000).

Lessons from Ecologies of Freedom

In literature on ecology of freedom, specifically Murray Bookchin’s (1982) book with the same title, Bookchin outlines concepts of social ecology and exposes hierarchy, patriarchy, and capitalism. He explains through the book that hierarchy is exclusively a human phenomenon, and he finds examples of people resisting and overturning hierarchies. Ultimately, Bookchin believes we can create a world based on social equality, direct democracy, and ecological sustainability. The notion of ecology of evaluation reveals larger structural issues and concepts that contribute to program improvement and social betterment, raising the bar on issues of equity and equality as fundamental to notions of values and valuing. The hope of human possibility explored in the area of ecology of freedom extends to the hope in the social betterment and democratic role and tradition of evaluation, one that has played roles in social advocacy and agenda setting in the field (Stufflebeam 2001).

Lessons from Ecologies of Language or Linguistics

Scholars in ecology of language or ecolinguistics have created a subfield of linguistics that emphasizes linguistic nonlinearity and concepts of sustainability and of indigenous and cultural diversity. According to Language and Ecology Research Forum,

Ecolinguistics examines the influence of language on the life-sustaining relationships of humans with each other, with other organisms and with the natural environment. Research ranges from the impact of advertising discourse in encouraging ecologically damaging consumption—to the power of nature poetry to encourage respect for the natural world.

Beyond a sociolinguistic perspective of the field with its focus on social factors of language, an ecological perspective gives more deliberate attention to relationships, cultural diversity, and sustainability.

Building off the contributions of ecological thinking in other applied orientations and disciplines, this volume features applications from thought leaders and practitioners on tackling wicked problems in complex ecologies of evaluation practice. What follows is the logic, organization, and layout of the book with specific attention to how the core elements of this practice with focus on the 3Rs (relationships, responsibilities, and relevance) manifest in the dynamic and unique contexts at multiple program, policy, and project levels.

Overview of the Volume

The logic and organization of the volume reflects attention to a couple of important considerations in the development of the book. First, each chapter reflects the definitions inherent in this initial chapter around complex ecologies in evaluation and the three Rs (relationships, responsibilities, relevance). Although the definitions and themes are consistent, they are applied in unique ecologies and in disciplinary and applied orientations and contexts, adding to the richness and integrative nature of the volume.

Second, the volume is organized into four parts where two or three chapters are located in each part, described in more detail in the following subsections. Chapters within each part highlight relevant cases along a spectrum of complex ecologies from international development focusing on women and disabilities to food systems and from health equity policy to restorative justice and international aid, indigenous populations and those affected by HIV. A final chapter does more than summarize the book chapters; it extends lessons learned to push the field forward in its conceptualization and understanding of evaluation in complex ecologies.

Organization of the Volume

In the first part, “Foundations,” in addition to this initial chapter, two more chapters outline the section focused on elaborating on core terms and concepts of the book. Andy Rowe’s chapter, “Ecological Thinking as a Route to Sustainability in Evaluation” argues that evaluation is currently ill equipped to respond to ecological thinking and therefore lacking in capacity to undertake sustainability-ready evaluations. He introduces us to a coupled human and natural systems (CHANS) approach that bridges the human system, where much of our evaluation practice is currently based, and the natural system. This approach calls on us to engage in ecosystem thinking, in particular to consider the implications of social initiatives for the natural environment, including ways to introduce evaluation to natural systems and natural system science to evaluation.

Linda Tuhiwai Smith’s chapter is grounded in understanding concepts that undergird the complexity of the experience of indigenous peoples. The chapter both speaks to the complexity and diversity of the indigenous experience and focuses on the complex ecologies of the interface between indigenous and nonindigenous worlds, in particular the indigenous and the evaluation world. Her chapter argues for deeper and more authentic engagements and relationships between evaluators and indigenous ecologies in preparation for the development of quality and credible work including appropriate recommendations for improvement.

>Robin Miller’s chapter on evaluating HIV practices and evidence-supported programs in community-based AIDS programs begins the second part, “Vulnerabilities in Diverse Evaluation Ecologies.” The second part is intended to focus on groups most vulnerable and marginalized and their complex ecologies, including ways that evaluators find meaning in working in these settings. Miller focuses on her community psychology work within the ecology of AIDS-related evaluations and her own practice rooted in the United States and Eastern Caribbean; young black gay, bisexual, and lesbian communities; and AIDS-related community-based organizations. By pointing out the limited nature of evaluation, community-based or otherwise, including the lack of longer-term implementation evaluations of programs in AIDS community-based settings, Miller’s chapter highlights challenges and opportunities facing evaluators who, like her, intend to be champions of people’s health.

The following chapter in this section, by Donna Mertens and Arlinda Boland, highlights the difficulties of attending to the disparities experienced by women and by people with disabilities if these two groups are not consciously included in development agendas and the complex ecologies (including cultural, religious, and political attitudes and beliefs) that maintain their marginalization are not recognized. This includes people whose identities or subjectivities place them at the intersections of marginalization and often reinforce their invisibility within a society (for example, deaf women who have sex with women in a homophobic society, indigenous women who are feminists within a patriarchal society). The rationale provided by Mertens and Boland for the transformative paradigm in evaluation is the need to respond to marginalized groups whose worlds have too often remained unseen and whose living conditions have failed to improve. Initiatives and evaluations that are done “to” them rather than “with” or “by” them are at risk of maintaining or even strengthening their oppression. Mertens and Boland write that the transformative paradigm addresses these issues head on by repositioning the role of the evaluator as a supporter of transformation and a steward of courage and justice.

The final chapter in this part, by Jill Anne Chouinard and Ayesha Boyce, proposes a restorative justice approach to evaluation by bringing together primary program stakeholders to restore voice and generate understanding between those previously incarcerated, especially sex offenders, and community stakeholders. By exploring the traditions of criminal justice evaluation and the roots of justice in evaluation, the authors build on the justice-oriented approaches in evaluation to situate restorative justice principles and approaches within complex ecologies of sexual offenders. In extending the 3Rs to a fourth, Chouinard and Boyce suggest that a restorative approach to evaluation has potential to improve quality of life for communities traditionally marginalized or oppressed and requires methods that stimulate healing and reconciliation among victims, offenders, the community, and society as a whole.

The chapters in the Part III, “System and Policy Responsiveness to Complex Evaluation Ecologies,” address ways evaluators use their understandings of complex ecologies to inform systems and policies of change. The chapter by Oran Hesterman and Ricardo Millett focuses on changing the complex ecology of the food system, especially broken elements to present ideas about how evaluation and evaluators can play a role in fixing the food system. Based on their content expert and evaluation expert roles at a philanthropic agency in the U.S. Midwest, the authors rely on key questions to raise questions regarding the social, psychological, political, and environmental drivers of our society that give rise to the vexing problems of the seeming permanence and resiliency of systemic racism, inequality, and marginalization. In their chapter they focus on the implementation and evaluation of SNAP (Supplemental Nutrition Assistance Program), providing insights into the relationships established with providers for the delivery of SNAP and the additional interrogation of the food system undertaken with key stakeholders. The evaluation methodology then builds a data collection system in collaboration with provider organizations, so that consistent data can be used to influence program funding and food system policy.

In the second and final chapter, a team of colleagues working in health equity policy focus their attention to developing relevant and responsible recommendations in health policy and implementation. Much has been written about the need to include recommendations as part of evaluation reporting, but there is relatively little guidance about how these recommendations are developed so they are rigorous and useful and influence both practice and policy. Crystal Barksdale and her colleagues use their evaluation of the implementation of a national health policy initiative, the enhanced National Standards for Culturally and Linguistically Appropriate Services (CLAS) in Health and Health Care as a case example for developing meaningful and relevant recommendations by focusing on questions such as: For whom are recommendations developed? What is the purpose of recommendations? How are recommendation sources and perspectives balanced?

The final part of the book, “Toward Solutions,” includes two chapters designed to push the envelope even further toward solutions and implications of evaluation in complex ecologies. In Michael Quinn Patton’s chapter on principles driving reform in the complex global system of development aid, he focuses on the Paris Declaration Principles on Aid Effectiveness. Through a nonlinear trail of relationships built on shared values, Patton was asked to be the external metaevaluator of the Paris Declaration on Aid Effectiveness. This role complemented the internal metaevaluation role that saw a skilled evaluation team synthesize and communicate country evaluations and study reports. Patton was able to observe this process in real time, talk to key players, and read documentation. The result is the insight he brings to this chapter about why the internal metaevaluation worked and subsequently received the American Evaluation Association’s Outstanding Evaluation Award in 2012.

The final chapter of the book, by Fiona Cram and Rodney Hopson, provides implications for evaluation in complex ecologies of inequities and inequalities. More specifically, the final chapter weaves together the wisdom of the contributors to this volume so that those interested in evaluation (for example, governments, funders, practitioners, those being evaluated), and those who strive to be responsive to the wicked problems within their own complex ecologies can think along with these expert evaluation practitioners.

Ultimately, the book is intended to set us forth on a journey only newly chartered, to contemplate and interrogate how to tackle wicked problems in evaluation. We have assembled a stellar cast of travelers and contributors to think through what this means for those who work locally or internationally and who concern themselves with basic living and sustainability issues such as food, life, health, equity, and dignity so that generations to follow might reap from the benefits of our practice in the days and decades to come.

References

Agar, Michael. 2004a. An anthropological problem, a complex solution. Human Organization 63(4): 411–418.

————. 2004b. We have met the other and we’re all nonlinear: Ethnography as a nonlinear dynamic system. Complexity 10(2): 16–24.

Arnove, Robert F., Carlos A. Torres, and Stephen Franz, Stephen (Eds.). 2013. Comparative education: The dialectic of the global and local. Lanham, MD: Rowman & Littlefield Publishers.

Bamberger, Michael, Jos Vaessen, and Estelle Raimondo. 2016. Dealing with complexity in development evaluation: A practical approach. Los Angeles, CA: Sage.

Bookchin, Murray. 1982. The ecology of freedom: The emergence and dissolution of hierarchy. Palo Alto, CA: Cheshire Books.

Braverman, Marc T., Molly Engle, Mary E. Arnold, and Roger A. Rennekemp (Eds.). 2008. Program evaluation in a complex organizational system, New Directions for Evaluation 120.

Byrne, David. 2013. Evaluating complex social interventions in a complex world. Evaluation 19: 217–228.

Cajete, Greg. 2000. Native science: Natural laws of interdependence. Santa Fe, NM: Clear Light Publishers.

Camino, L. A. 1997. What can anthropologists offer ethnographic program evaluation? NAPA Bulletin 17: 41–57.

Camillus, John C. 2008. Strategy as a wicked problem. Harvard Business Review, May. Retrieved in August 2012 from http://hbr.org/2008/05/strategy-as-a-wicked-problem/ar/1.

Capra, Fritjof. 1997. The web of life. New York: Anchor Books.

Chertow, Marian R., and Daniel Esty. 1997. Thinking ecologically the next generation of environmental policy. New Haven, CT: Yale University Press.

Copeland-Carson, J. 2005. “Theory-building” evaluation anthropology. NAPA Bulletin 24: 7–16.

Dean-Coffey, Jara, Jill Casey, and Leon D. Caldwell. 2014. Raising the bar: Integrating cultural competence and equity: Equitable evaluation. The Foundation Review 6(2).

EvalPartners. 2014. How to design and manage equity-focused evaluations. Available at www.mymande.org/?q=virtual.

Forss, Kim, Mita Marra, and Robert Schwartz (Eds.). 2011. Evaluating the complex: Attribution, contribution, and beyond. New Brunswick, NJ: Transaction.

Funnell, Sue C., and Patricia J. Rogers. 2011. Purposeful program theory: Effective use of theories of change and logic models. San Francisco: Jossey-Bass.

Greene, Jennifer C. 2006. Toward a methodology of mixed methods social inquiry. Research in the Schools 13(1): 93–98.

Henry, Ella, and Hone Pene. 2001. Kaupapa Māori: Locating Indigenous ontology, epistemology and methodology within the academy. Organisation 8: 234–242.

Howitt, Richie, and Sandie Suchet-Pearson. 2003. Changing country, telling stories: Research ethics, methods and empowerment—Working with Aboriginal women. Presentation at the “Fluid Bonds: Gender in Water Resource Management” Conference, Canberra, October 13, 2003. Canberra, October 13.

Human Rights Commission. 2010. New Zealand Human Rights Commission’s major report card on the state of human rights in New Zealand. Wellington: Human Rights Commission.

Huisman, Emelieke R. C. M., Ernesto Morales, Joost van Hoof, and Helianthe Kort. 2012. Healing environment: A review of the impact of physical environmental factors on users. Building and Environment 58: 70–80.

Jahn, Thomas. 2008. Trandisciplarity in the practice of research. In Transdisziplinäre Forschung. Integrative Forschungsprozesse verstehen und bewerten, edited by Matthias Bergmann and Engelbert Schramm, 21–37. Frankfurt and New York: Campus Verlag.

Kelly, James G. 2006. Becoming ecological: An expedition into community psychology. Oxford, UK: Oxford University Press.

Kelly, James G., Ann Marie Ryan, B. Eileen Altman, and Stephen P. Stelzner. 2000. Understanding and changing social systems: An ecological view. In Handbook of community psychology, edited by Julian Rappaport and Edward Seidman, 133–159. New York: Kluwer Academic/Plenum Publishers.

Kolko, J. 2012. Wicked problems: Problems worth solving: A handbook & a call to action. Austin, TX: AC4D.

Kushner, Saville. 2000. Personalizing evaluation. London: Sage.

Language and Ecology Forum, International Ecolinguistics Association. Available at www.ecoling.net/.

Lawrenz, Frances, and Douglas Huffman. 2006. Methodological pluralism: The gold standard of STEM evaluation. New Directions for Evaluation 2006(109): 19–34.

Levin, Kelly, Benjamin Cashore, Steven Bernstein, and Graeme Auld. 2012. Overcoming the tragedy of super wicked problems: Constraining our future selves to ameliorate global climate change. Policy Sciences 45: 123–152. doi:10.1007/s11077-012-9151-0.

Liu, Jianguo, Thomas Dietz, Stephen R Carpenter, Carl Folke, Marina Alberti, Charles L Redman, Stephen H Schneider, et al. 2007. Coupled human and natural systems. Ambio 36(8): 639–649.

Mertens, Donna M. 2009. Transformative research and evaluation. New York: Guilford Press.

Meyer, Manulani Aluli. 2008. Indigenous and authentic: Hawaiian epistemology and the triangulation of meaning. In Handbook of critical and indigenous methodologies, 217–232. Thousand Oaks, CA: Sage.

Mohan, Rakesh, David J. Bernstein, and Maria D. Whitsett (Eds.). 2002. Responding to sponsors and stakeholders in complex evaluation environments, New Directions for Evaluation 95(7).

Nkwake, Apollo M. 2013. Working with assumptions in international development program evaluation. New York: Springer.

Patton, Michael Q. 2011. Developmental evaluation: Applying complexity concepts to enhance innovation and use. New York: Guilford Press.

Pawson, Ray. 2013. The science of evaluation: A realist manifesto. Thousand Oaks, CA: Sage.

Rittel, Horst W. J., and Melvin M. Webber. 1973. Dilemmas in a general theory of planning. Policy Sciences 4: 155–169.

Rog, Debra J., Jody L. Fitzpatrick, and Ross F. Conner. 2012. Context: A framework for its influence on evaluation practice. New Directions for Evaluation 135.

Rogers, Patricia J. 2008. Using programme theory to evaluate complicated and complex aspects of interventions. Evaluation 13(1): 29–48.

Rogers, Patricia J., Anthony Petrosino, Tracy A. Huebner, and Timothy A. Hacsi. 2000. Program theory evaluation: Practice, promise, and problems. New Directions for Evaluation. 2000(87): 5–13.

Royal Commission on Genetic Modification. 2001. Report of the Royal Commission on Genetic Modification: Report and recommendations. Wellington: Royal Commission on Genetic Modification.

Ryan, Katherine E., and Lizanne DeStefano (Eds.). 2000. Evaluation as a democratic process: Promoting inclusion, dialogue, and deliberation. New Directions for Evaluation 85(49).

Segone, Marco. 2012. Evaluation for equitable development results. New York: UNICEF in partnership with Coneval, IDEAS, IDRC, ILO, IOCE, UNDP, and UN Women.

Smith, Linda Tuhiwai. 2012. Decolonizing methodologies: Research and indigenous peoples, 2nd edition. London and New York: Zed Books.

Stahl, Phillip M. 2011. Conducting child custody evaluations: From basic to complex issues. Thousand Oaks, CA: Sage.

Stufflebeam, Daniel. 2001. Evaluation models. New Directions for Evaluation 89: 7–98.

Thomas, Veronica G. 2004. Building a contextually responsive evaluation framework: Lessons from working with urban school interventions. New Directions for Evaluation. 2004(101): 3–23.

Thompson Klein, Julie. 2004. Prospects for transdisciplinarity. Futures 36: 515–526.

Turkle, Sherry. 2011. Alone together: Why we expect more from technology and less from each other. New York: Basic Books.

UniversalDesign.com. 2016. What is universal design? Retrieved on January 15, 2016, from www.universaldesign.com/about-universal-design.html.

Weiss, Carol H. 2000. Which links in which theories shall we evaluate? New Directions for Evaluation 2000(87): 35–45

Watkins, Alan, and Ken Wilber. 2015. Wicked & wise: How to solve the world’s toughest problems. Available at www.overdrive.com/search?q=1C7B0A05-2598-4225-B183-A66EF7A1494F.

Williams, Bob, and Richard Hummelbrunner. 2010. Systems concepts in action: A practitioner’s toolkit. Stanford, CA: Stanford Business Schools.

Williams, Bob and Sjon van’t Hof. 2016. Wicked solutions: A systems approach to complex problems. SI: Bob Williams.

Wolf-Branigin, Michael. 2013. Using complexity theory for research and program evaluation. Oxford, UK: Oxford University Press.