Warren+Report+3

1. Response to comments from Report 2 (previously completed) 2. Instructional analysis as page in Wiki (Report 2) (previously completed) ** 3. **** Describe materials, procedures to be used in one to ones ** ** a. **** Materials: ** i. Private meeting room removed from distractions ii. Instructional analysis iii. Performance Objectives iv. Instructional materials v. Tests and other assessment instruments vi. OPR software is installed on SME’s computer vii. Formative Evaluation Guide -01 (see attachment 500_Formative Evaluation Guide.doc) viii. Assessment tools (Attitude Questionnaire ) designed for the small group formative evaluation ** b. **** Procedure ** i. Meet with Subject Matter Expert and explain Purpose of the meeting. 1. Explain to the SME that a new set of instructional materials has been developed and his/her reaction to them is desired. 2. Explain that the SME’s reactions to instructor directions and rubrics are also desired. 3. Explain that the materials are in formative stage and need feedback on how they can be improved a. Explain that scope of the input is to identify and remove the most obvious errors in instruction 4. Explain why SME was selected to help with the formative evaluation (see 04 below) 5. Explain that the assessment tools designed for the small group evaluation need the SME’s input as well 6. Convince SME is OK to be critical of what is presented to them ii. Provide SME with materials and explain their purpose. 1. Provide Instructional analysis to the SME 2. Provide Performance Objectives to the SME 3. Provide the instructional materials to the SME 4. Provide Tests and other assessment instruments to the SME 5. Ensure the OPR software is installed on SME’s computer and the SME can access 6. Ask the SME to take 1 week to review the materials provided above including accuracy and currency, appropriateness, clarity, and sequence. a. Note: ensure the SME understands the materials provided i.e. understand their purpose. 7. To further facilitate and define the review process, provide the SME with Formative Evaluation Guide -01, explain the definitions of each column, and request the SME use the guide to direct their assessment. a. Note: the instructional materials themselves are the primary data gathering aid for the SME. The SME should circle words or sentences and/or write comments directly on the materials provided. Formative Evaluation Guide -01 is provided merely as a tool to help guide the SME’s thinking. 8. Explain that qualitative feedback is desired. i.e. both positive and negative feedback and rationale. 9. Explain that any input is valued including: a. Typographical errors b. Omissions of content c. Missing labels 10. Review with the SME the type of information desired. Including: a. Adequacy of overall delivery system b. Adequacy of pre instructional Items c. Appropriateness of size/scope of each learning Unit / Cluster d. Adequacy of student groupings e. Adequacy of media selection for delivering learning components f. Valid vs. faulty instructional analysis g. Valid vs. faulty judgments about learner entry skills h. Valid vs. unclear/inappropriate objectives i. Adequate vs. inadequate test questions and/or directions j. Adequate vs. inadequate presentation including: i. Graphs ii. Volume of information at one time iii. Sequence of information presented iv. Clarity of examples k. Adequacy of assessment tools designed for the small group formative evaluation 11. Schedule 1-2 hr meeting (1 week later) with SME to review their assessment 12. Ask SME to evaluate the assessment tools designed for the small group formative evaluation (Attitude Questionnaire) a. Objective here is to identify not only inadequacies but the reasons for the inadequacies. Need to ensure assessment tools are adequate before move on to small group assessment. iii. Second Meeting with SME (1 week after first meeting) - Ensure the one to one process is iterative / interactive: 1. Ensure all following materials are available for the meeting: a. Instructional analysis b. Performance Objectives c. Instructional materials d. Tests and other assessment instruments e. OPR software is installed on SME’s computer f. Formative Evaluation Guide -01 g. All notes created by SME during their initial review 2. Encourage dialogue. Ensure the SME is relaxed and is ready to freely discuss their conclusions. 3. Interview the SME using Formative Evaluation Guide -01 to guide and to document the interview. a. Ensure, at a minimum, the following areas are discussed and documented. i. Adequacy of overall delivery system ii. Adequacy of pre instructional Items iii. Appropriateness of size/scope of each learning Unit / Cluster iv. Adequacy of student groupings v. Adequacy of media selection for delivering learning components vi. Valid vs. faulty instructional analysis vii. Valid vs. faulty judgments about learner entry skills viii. Valid vs. unclear/inappropriate objectives ix. Adequate vs. inadequate test questions and/or directions x. Adequate vs. inadequate presentation including: 1. Graphs 2. Volume of information at one time 3. Sequence of information presented 4. Clarity of examples xi. Adequacy of assessment tools (Attitude Questionnaire )designed for the small group formative evaluation 4. For each item above, ask the SME why he/she drew the conclusions they did. a. This will identify not only inadequacies but the reasons for the inadequacies. ** 4. **** Describe the Subject Matter Expert and why you selected the person to be one to one evaluator ** ** a. **** A. Knowledgeable about the content area of the instruction or is working within the content area ** i. The Director, Planning & Projects - ((OF WHAT COMPANY OR ORGANIZATION... I need you to envision an exact person)) is a manager of project managers i.e. a very experienced and successful senior project manager. The particular individual I’m considering has over 20 years planning and managing projects. Project risk management (identifying, assessing, mitigating, and monitoring project risk to ensure objectives are delivered within time, cost and quality constraints.) is a major role for the Director, Planning & Projects. This real world risk management experience provides the Director, Planning & Projects with the technical expertise needed for the formative evaluation. Having successfully run projects with real world cost and time constraints the Director is qualified to comment on the accuracy, currency, and cost effectiveness of the instruction. ** b. **** Knowledgeable about the target population of learners. ** i. The Director, Planning & Projects, is also an experienced communicator, team leader, and trainer. Business Analysts (Target Learners) were often assigned to the Director’s projects so, when providing input for the formative evaluation, the Director can call upon a wealth of experience teaching, training, and working with representatives from the target population. In other words, the Director, Planning & Projects can empathize with the learners and understands their performance context. This means the Director is very qualified to evaluate the appropriateness of the training materials for eventual use within the performance context

5. Instructional Materials and assessments (previously completed)

** 6. **** Describe characteristics of small group learners (8-20) ** a. A sample of between eight to twenty of the target learners will be used for the small group assessment. An attempt will be mad to include representatives from subgroups within the population. i. In general, the following guide will be used: 1. Low, average and high achievement 2. Low, moderate and high familiarity with subject mater 3. 0-2 years experience, 3-5 years experience, over 5 years experience as business analysts ii. Other / more specific criteria used to ensure most subgroups within the learner population are represented: 1. Age: (20+, 30+, 40+, 50+) 2. Male & Female 3. Jr. & Sr. Business Analysts 4. Novice & expert project management skills 5. Novice & expert experience conducting project risk management 6. Average & strong oral and written communication skills 7. Average and strong negotiation skills. 8. Average and strong analytical skills 9. Strong customer focus 10. Average and strong meeting facilitation skills 11. Average and strong consensus building skills 12. Novice & expert experience conducting group problem solving 13. Novice & expert experience conducting brainstorming sessions 14. Low, average and strong EXCEL skills 15. Low, average and strong grasp of statistical concepts: (Averages, Normal Distributions, Bell Shaped Curves) 16. Graduate and post graduate education ** 7. **** Describe the materials and instruments used in the small group evaluation ** ** a. **** Materials: ** i. PC Lab. Each learner will have access to desktop computer with MS Office/ EXCEL and ORP software installed ii. Instructor’s desktop will be connected to projector iii. Whiteboards and/or flipchart with dry erase markers iv. 3x5 note cards and pens v. Instructional materials vi. Tests and other assessment instruments vii. Assessment tools designed for the small group formative evaluation 1. Attitude Questionnaire (completed via pencil and paper Scale 0-5 0=not at all…5= very much) (See attachment 500_Attitude_Questionnaire.doc) a. Was the instruction interesting? b. Did you understand upfront what you were supposed to learn? c. Were the materials relevant and related to the objectives? d. Did the practice exercises help you learn? Were there too many/too few? e. Did you receive useful feedback on your practice exercises? f. Did the final exercise really measure your knowledge of the objective g. Were you confident in your ability to do the final exercise? h. Do you think the skills you learned will transfer to your real world job? i. Will what you learned help you be more successful in your role as a Business Analyst? j. OPEN ENDED QUESTIONS: i. What did you like most? ii. What did you like least? iii. What part(s) of the instruction gave you difficulty? How should that be fixed? iv. Was the instruction too long or too short? v. Was the instruction too difficult or to easy? vi. How will you use the new skills you learned? vii. What potentially stands in the way of using your using the new skills on the job? viii. What would enhance your ability to use these new skills? ix. How might the training be refined or improved? x. Would there be a better way to teach this material? What are your suggestions? 2. Formative Evaluation Guide -01 (for use by the designer to guide in-depth debriefings with some of the learners in the group). The debriefings would include requesting details and/or reasoning behind attitude questionnaire results and open ended questions /discussion guided by Formative evaluation guide 01 ** 8. **** Describe the procedures in small group evaluation ** ** a. **** Procedure Note: Every attempt will be made to ensure the training that is conducted for the small group evaluation emulates the proposed production training environment as much as possible.** i. Identify 8- 20 learners that represents the target population ii. One to two weeks before the small group evaluation, the designer selects a qualified instructor and verifies the instructor is willing to participate in the evaluation. 1. Provide the instructional materials to the Instructor. 2. Provide Tests and other assessment instruments to the Instructor 3. Ensure the ORP software is installed on Instructor’s computer and the instructor can access 4. Ask the instructor to take 1 week to review the materials. 5. Review with the instructor that after the small group evaluation is completed, the instructors opinion on the following will be requested: a. Adequacy of overall delivery system b. Adequacy of pre instructional Items c. Appropriateness of size/scope of each learning Unit / Cluster d. Adequacy of student groupings e. Adequacy of media selection for delivering learning components f. Valid vs. unclear/inappropriate objectives g. Volume of information at one time h. Sequence of information presented iii. Make scheduling arrangements for the Instructor, learners, and meeting facilities. iv. Reserve and set up the PC lab precisely as it would be expected to be set up for production training to include installing ORP software on individual PCs. v. Meet with mall group learners and explain purpose of the meeting. 1. Explain to the small group learners that a new set of instructional materials has been developed and their reaction to them is desired. 2. Explain that the small group learners’ reactions to instructor directions are also desired. 3. Explain that the materials are in formative stage and need feedback on how they can be improved 4. Convince small group learners is OK to be critical of what is presented to them 5. Explain that qualitative feedback is desired. i.e. both positive and negative feedback and rationale. 6. Explain that any input is valued including: a. Typographical errors b. Omissions of content c. Missing labels d. Valid vs. unclear/inappropriate objectives e. Adequate vs. inadequate test questions and/or directions f. Adequate vs. inadequate presentation including: i. Graphs ii. Volume of information at one time iii. Sequence of information presented iv. Clarity of examples vi. Instruction proceeds as intended to be implemented in final form. From pre-tests to post test. Note: Designer will be present in PC lab but be ‘behind the scenes’ and should intervene as little as possible. Only in cases where equipment/software fails and/or learner cannot continue should the designer intervene. Note: Learner difficulty during instruction and solution should be noted by the designer as input for potential revision to instruction 1. Instructor administers the Pretest. a. Learners are asked to circle any vocabulary that is unclear to them b. Learners are asked mark any directions that are unclear c. Designer notes elapsed time to complete the pretest 2. Instructional materials are administered (Every attempt will be made to ensure the training that is conducted for the small group evaluation emulates the proposed production training environment as much as possible) a. Learners are asked to document any vocabulary that is unclear to them b. Learners are asked document any directions that are unclear c. Designer notes elapsed time to complete each unit of instruction 3. Instructor administers the Post Test. a. Learners are asked to circle any vocabulary that is unclear to them b. Learners are asked mark any directions that are unclear c. Learners annotate any ‘guessing’ d. Designer notes elapsed time to complete the posttest vii. After Instruction is completed (including post test), the Attitude Questionnaire is distributed to each of the learners and they are given time to individually complete and hand in the questionnaire. viii. After all Attitude Questionnaires are received, the designer asks group for and documents any comments offered by the group ix. Designer debriefs the instructor to receives the instructor’s input regarding: a. Adequacy of overall delivery system b. Adequacy of pre instructional Items c. Appropriateness of size/scope of each learning Unit / Cluster d. Adequacy of student groupings e. Adequacy of media selection for delivering learning components f. Valid vs. unclear/inappropriate objectives g. Volume of information at one time h. Sequence of information presented x. Designer compares each learner’s pre and post tests and identifies areas where learning seems inadequate xi. Designer meets with learners 1. As soon as feasible after the training is completed, the designer contacts and meets with as many learner’s as feasible and 2. Designer encourage dialogue. Ensures the learner is relaxed and is ready to freely discuss their conclusions 3. Review with each learner that it is OK to be critical of what was presented to them 4. Designer conducts in-depth debriefings with each using their Attitude Questionnaires and the Formative Evaluation Guide -01 to guide the sessions. The debriefings would include requesting details and/or reasoning behind attitude questionnaire results and open ended questions /discussion guided by Formative evaluation guide 01 i. Objective here is to identify not only inadequacies but the reasons for the inadequacies. 9. Describe what data your small group would create about your project a. The data can show what problems and reactions the small group learners had. Data (including timing as applicable) will be collected on: i. Motivational / Introductory materials ii. Pretest iii. Training 1. Information Presentation 2. Materials 3. Learner participation iv. Posttest v. Attitudes vi. Transfer to performance context b. Reasons for individual item evaluations will also be collected. This data will enhance the designers ability to design the best change required for meeting the instructional goal c. In addition to the verbal input collected through surveys and debriefings. Assessment Item Performance by Objective will be captured and analyzed. (See attached sample for Objective #2….500_Item_Objective_Analysis.xls) to measure item quality and learner performance. 10. Describe how you would implement revisions to instruction and assessment a. The quantitative Assessment Item Performance by Objective analysis, timings, and qualitative data gathered through direct observation, feedback and surveys will be summarized and documented on an Instructional Revision Analysis Form (see sample below). The completed form will serve as a checklist for required revisions.


 * **Instructional Revision Analysis Form** ||
 * **Instructional Strategy Source** ||  **Problem / Reasons**  ||  **Proposed Change in Instruction**  ||  **Evidence and Source**  ||
 * Motivational / Introductory materials ||  ||   ||   ||
 * Pretest ||  ||   ||   ||
 * Information Presentation ||  ||   ||   ||
 * Materials ||  ||   ||   ||
 * Learner participation ||  ||   ||   ||
 * Posttest ||  ||   ||   ||
 * Attitudes ||  ||   ||   ||
 * Transfer to performance context ||  ||   ||   ||