Thursday, May 27, 2010

Reflection Assignment #5 - Evaluation Reports

There are two main "takeaways" for me after reading the report.

The first is stated in the introductory section of this chapter.  It states "Designing, preparing, and presenting an evaluation report is the very essence of evaluation, but if the report does not prove useful or influential, then the 'essence' quickly evaporates into an empty exercise."

As I reflect on this thought it solidifies my preconception that the content and presentation of the evaluation results to the audience is critical to whether the evaluation has any value.  As the chapter points out, a careful and engaging report (both the printed version, and any oral presentations made) will bring about real affect of the evaluation in changes that may be required in the evaluation object.  It is also essential that these reports be useful and reliable for the profession to continue to justify the funds that are required to complete this work.

If evaluations are viewed as bureaucratic "hoops" rather than being useful to key stakeholders in improving their programs, then the value of effective evaluation will be lessened and opportunities for real improvement may be pushed aside.  Perhaps the greatest danger would be for evaluation to move from being an objective means of improving and strengthening programs to becoming an arrow in a quiver of those seeking to fight or support programs.  While this may create more opportunities for professional evaluators, it could taint the supposed value of evaluation, fair and independent assessment of the evaluation object.

Evaluation reports need to keep this issue in mind.  In addition to being engaging and comprehensible, it is important that they be fair and accurate.  As the chapter points out, the best evaluations will result in a fair description and assessment of both the strengths and weaknesses of the object being evaluated.  This means that here will be both positive and negative feelings about the evaluation from the different audiences to the report.  I have observed the point made in this chapter that mentioning of weaknesses will bring about defensiveness.  I have also observed that this is just a natural reaction and that part of the value of the evaluation team is to help the stakeholder work through this defensiveness to help them see the weakness and begin to make improvements.  This requires the right mix of people skills and strategic thinking on the part of the evaluator in order to "add value" to cement the effectiveness of the evaluation.

The second "takeaway" was not really covered in the chapter but was an inference I understood more clearly.   Once you express yourself in writing on paper, particularly in high-stakes evaluations, the words live forever in the context in which the ink exists on the paper.  It is impossible to prevent others from repeating their words and applying their own interpretations and context.  I was even more impressed by the need to be clear and concise in expression and to be economical in the choice of words.

While the chapter would remind us that it is important to be engaging in our prose and to use graphics and other methods of breaking up the monotony of the report, these can diffuse the clarity of the report and lead to vagueness that could be construed according to the desired objectives of the user or reporter of the information.  I also realized that this risk cannot be the overriding concern of the evaluator or the report will become too stale to have any benefit to the user.  There is a balance that must be accounted for so that the user engages, but that the evaluation report is clear in context and interpretation.

The last point that was helpful to me was the discussion of the influence, intention, and time frame postulated by Kirkhart.  The evaluation report will take on a life of its own once it is in the realm of publication.  The words will live forever and can be accessed at multiple points in the immediate, mid, and long terms.  The findings and influence of the report will live on not only in the implications for the immediate evaluation object, but also as it may be inferred to similar objects, and in the lives and experiences of the evaluators and the stakeholders involved with the particular evaluation.  The effects of change and influence from one evaluation will extend through the experiences and findings to other contexts, objects, and some elements of society through the learning that occurs and the influences on the stakeholders.  That is a valuable and astute observation.  In essence, each evaluation will change something, even if it is not the immediate object of the evaluation.

Wednesday, May 12, 2010

Reflections Paper #4 - Design and collection methods

These two chapters relate a common theme as they are both focused on the collection of data for the evaluation.  The greatest common theme is that the data collected, and the design of the method for collecting the data, need to be aligned with the evaluation question to be answered.  The selection of the method to design the data collection, and then the method used to collect the data, can lead to inefficient use of resources and leave the evaluation question unanswered.

Basic rules for the validity of the data compare closely with those with which I am familiar in designing research and statistical projects.  However, the evaluation use of the data allows more freedom in selection of data, the approach, and the method because we are more concerned with the evaluation question pertaining to the particular instance of the program than we are in inferring the result to all instances of the program.  This lends itself more readily to qualitative and mixed-method planning, design, and data collection than might be required under a more stringent standard associated with straight research.

I was grateful to be reminded of the lack of transferability of certain research principles and designs to education and other social evaluation projects.  The example they use of the experimental designs familiar to us in science and medicine and the inability to transfer this same design to social policy situations is enlightening.  The essence of the design and data collection choices we make is that each situation and evaluation is unique.  We have to identify the evaluation question(s) as precisely as we can and then align the design and methods of collection so we can best answer the question(s).

One final highlight for me was that we did not need to choose methods and designs free of bias in order to have a meaningful evaluation.  Rather, understanding the biases present on the various methods is critical to correctly answering the evaluation question.  The thought that the best practice is to mix the design and the data collection methods with an eye towards the various biases so that the findings triangulate and measure the construct validity.

From these readings my biggest "take-away" related to our project is the importance of identifying the critical evaluation question(s) as precisely as we can.  The randomization of the students into the groups for the evaluation will strengthen the validity of the observation, but the groups are small enough that we wouldn't need to consider sampling for some of the analyses we may perform.  We have talked about the possibility of an experimental design using pre-post to assess the impact of the game on learning the 12 times tables.  My concern is that the groups we are using are already fairly experienced with their times tables and the game.  I still think that the pre-post design will be a good approach, but the measurements of impact may be skewed by prior knowledge and we will be starting from a substantially higher baseline than that to which the game is geared.  It is hard to show a difference when many of the subjects have already achieved mastery in a major portion of the skill being evaluated.

Monday, May 10, 2010

Reflections Paper #3

The two major areas of impact to me from the readings for today are that questions are very important to the planning and process of the evaluation; and that stakeholders and interested audiences must be consulted early in the process, and may need to be consulted often as the evaluation identifies pertinent areas for question in the evaluation project.

Page 200 of the text notes that "an evaluation is only adequate if it collects information from and reports information to all legitimate evaluation audiences."  The process of identifying audiences can be driven towards those audiences that are the primary audiences (sponsor, client) or those that are the "squeaky wheel."  Such narrowing of scope in audience may either create an inadequate evaluation, or lead to discounting the evaluation itself when one of these ignored audiences may be impacted by an evaluation in which their concerns or needs were not consulted.  To help identify all audiences that may need to be considered, there is a helpful matrix on page 202 of the text that may help an evaluation team identify pertinent audiences.

As audiences are identified, the complexity and scope of the evaluation could increase to a point where an economically viable evaluation (in terms of financial, human resources, or time) is not possible.  The object of the evaluation must be carefully aligned with the questions to be addressed in order to have a meaningful evaluation.  The questions on pp. 204-205 seemed extremely helpful to identifying the critical object of the evaluation.  The program theory defined by Chen helps phrase the evaluation question.  Program theory helps the evaluator identify the normative theory (how the program should be) and compare it to the causative theory which identifies the likely outcomes given the inputs of the characteristics of the client and their actions.  This moves the evaluation object from the "black box" of sterile objects and plans, to the reality of the impacts of people, circumstances, organization, etc. that may lead to an evaluation of the program's success within the context of the organizational setting.

After carefully defining the audiences and the scope of the evaluation, the evaluator then moves through a series of questions to guide the evaluation itself.  A two part process of identifying divergent questions will help capture pertinent concerns of the audiences and stakeholders.  By meeting with the different stakeholders and asking them to express all of their concerns through questions that help identify the questions, concerns, and values of the stakeholders the evaluation team can identify possible needs to address in the evaluation.

After casting a wide net in the divergent phase, the evaluation team will then move to the convergent phase which identifies the questions that are truly pertinent to the evaluation and that can be reasonably addressed with resources available.  Primary drivers include:
  1. Who needs or wants the information?
  2. Would the answer to the question under consideration provide information not now available?
  3. Would the answer to the question yield important information?
  4. Is the question a matter of passing interest or does it focus on critical dimensions of continuing interest?
  5. Would the scope or comprehensiveness be seriously limited if this question were dropped?
  6. are resources sufficient to make answering this question feasible?
The purpose of the audience consultation and the development of these questions is to limit the scope and give the evaluation the proper focus.  It also helps identify the criteria that determines if the program is successful or not.  It may be successful at some objectives, but that may not be the objectives that matter to primary stakeholders or audiences.

The impact on our project was demonstrated when we met with the teachers today.  The questions that we asked were an attempt to see what the audience of the teacher would desire from this program evaluation.  We will also consult with the client and ask questions that will focus the data collection and help identify key metrics that are important to the analysis.  These questions should continue as we collect data and prepare the analysis.

Thursday, May 6, 2010

Reflection Paper #2 - pp. 129-151

The readings on Participant Oriented Evaluation were a little troubling to me internally.  As I stated in the last reflection paper, I have a background in auditing so I am very comfortable with the concept of evaluations by an independent evaluator using methodologies that identify objectives and assess performance based on the objectives, even if that objective is to inform third parties (consumers, managers, researchers, etc.).  The eliminations of biases through the approaches taken, and the independence of the evaluator seem critical to my training in providing evaluations that can be relied upon.

The ideas behind the various approaches and methodologies for participant oriented evaluations at first do not seem to be evaluation in the sense of my preconceived notions.  But the text identifies an area of evaluation that is an important window into the "effectiveness" of programs.  An evaluation that "experiences" the program in the role of participant may actually be more effective at determining the value of the program than any measurement of objectives.  The fact that some forms of this approach focus on the evaluator as the learner the subjects of the program being the informers or teachers of the evaluator.  This allows the evaluator to understand the real impact of the program and to accurately report these impacts to the stakeholders.

Perhaps the most interesting approach in the participant model is that biases and advocacy are an acceptable input for the evaluation.  In fact, the evaluator is free to pursue evaluations as an independent entity, not upon specific engagement.  This movement from evaluation as a source of information to stakeholders, to evaluation as a force of change upon stakeholders is troubling to me.  Advocacy evaluation seems to be a route to undercut the good-faith and trust of the public as it relates to evaluation.  Since many of the evaluations that are engaged are brought forward to inform the public of the effectiveness of the use of their funds through the programs that are offered, the loss of faith and trust, or the rejection of aggressive forms of advocacy through evaluation, may undermine the ability of providers of services to garner that trust.

We need to be mindful of the participant experiences in our project.  In fact, this may be the easiest thing to evaluate in the limited time we have available.  There may not be time to formulate testable objectives and collect the data in support of any meaningful conclusions.  But it seems that we could readily assess the student experience and report the affects of the game on their attitudes and levels of engagement.

Monday, May 3, 2010

Reading Assignment Week One - Chapters 4-7

This course is my first exposure to evaluation as it relates to the social sciences, particularly education.  My previous academic training and professional experiences have focused on evaluation as it relates to auditing and financial strategic decision-making.  While the subject matter related to evaluation in this text differs from the business training I have previously received, the fundamental principles and concerns are the same in this approach to evaluation as they are in the business approaches that have been my career experiences.

Auditing and business strategy analysis share many concerns in common with evaluation in the context of this text.  Among these similarities are the principle concerns regarding:
  1. The independence of the evaluator/auditor
  2. The role of internal vs external evaluators/auditors and their biases
  3. The role of the audit/evaluation as a report to inform, or a tool to influence
  4. The standards and rules governing the evaluation and the evaluation purpose
  5. The interplay between quantitative and qualitative measurements and their appropriateness for the evaluation need
  6. The objectivity or subjectivity of the observations
I feel very comfortable with the concepts I have read this week because it relates so closely to my previous academic and professional training.

So what did I learn this week?  Primarily I learned the classification of the different evaluation approaches.
  1. Objectives-Oriented Evaluation - the purpose is to identify the extent to which specific program objectives were obtained.  This can be done by collecting data and analyzing the results to see if program goals are achieved.  The most interesting part of this approach for me was the idea of goal free evaluation.  I see many applications of this approach to identify what was actually accomplished without being anchored to the goals as they were outlined.  I have encountered unintended consequences and unanticipated variances so often in my career that I see real potential in this approach to evaluating the accomplishments of the program.
  2. Management-Oriented Evaluation - this is the type of evaluation that I am most familiar with.  Its purpose is to treat the decision maker as the primary stake holder and help them refine and identify the decisions that need to be made and then scope the data collection and analysis to supply the data needed to inform that decision.  This approach reports the facts and its influence on the decision made is largely neutral.  There is danger in trying to anticipate the decision or influence the decision.  This bias may limit the factual credibility of the evaluation.
  3. Consumer-Oriented Evaluation - This approach uses checklists or other criteria in hopes of providing useful information to inform consumers.  While a desire to remain objective is important, there are come who would argue that subjective concerns may benefit society as they lead consumers to more effective alternatives that will lead to change in society.  The major growth in this area cam largely from the expansion in federal funding since the 1960s.
  4. Expertise-Oriented Evaluation - This approach uses an expert, a panel of experts, a formal, or an informal approach to evaluate programs.  A most common evaluation event using this approach is accreditation of educational, medical, and other institutions.  This is also the idea behind peer-reviewed refereed journals.  The experts render an opinion on the effectiveness or validity of the object or program, particularly related to objectives, standards, or societal expectations.
The implications for our class project indicate that our client hopes to show the benefit of their learning game.  This could mean an analysis of objectives to determine the success of learning improvement goals or even information for decisions about the development of the game in the future.  I am sure there will be other possible implications regarding the type, nature, and scope of the evaluation as I learn more.
    Darin