Evaluator’s note on the course and the project
The course Towards Better Employment Projects took place in spring 2021. From the available feedback of the participants it is obvious that the project team used any additional time for preparation of a thorough learning tool, furthermore well-tailored to the uneasy conditions we are still living in. The course participants praised the toolkit, e-learning and workshops and explicitly stressed how well prepared it was despite the online form. The stories of change are collected 6 months after the completion of the course (or other kind of support) and this is the minimum time required for evaluation after the last support takes place.
Self-rated development and knowledge growth
The participants of the course “Towards Better Youth Employment Projects” took part in a survey on their level of experience in various evaluation areas, personal and organizational attitudes towards evaluation, and expectations of the course. The questionnaire was administered to the participants before the course (1 – 30 March 2021) and after the course (27. April – 26. May 2021). The ex-post questionnaire contained additional questions concerning their satisfaction with the course. In the entrance survey, 78 participants responded. In the ex-post survey, 48 participants responded. In total, there were 43 respondents in whose cases it was possible to match their full initial and final answers in both entrance and ex-post surveys.
First, participants were asked to assess their proficiency in various areas of evaluation. Before the course, most of the participants identified themselves as beginners (1 or 2 out of 5) in the majority of the areas. After the course, their self-assessment rose on average by at least one level in most cases (just in data analysis the difference before and after was only 0,9).
The subjective self-assessment growth is supported by a more objective measurement. In the e-learning environment, the participants were obliged to take the entry and final test. The entrance test measured their initial knowledge in various areas reflecting particular modules of the course. Passing the final test (at least 60 % out of 100 points) was also a condition for receiving a certificate. Since the test at the beginning of the course and the one at the end of the course were identical, the difference between the results of the entrance test and the final test reflects the knowledge growth of the participants. While before the course only one third of the participants reached at least 60 % (overall 52 % on average), at the end of the course just two participants didn’t reach the 60 % threshold (and in both cases they didn’t even use all their tries). And on average, participants scored 91 % in the final test.
Figure 2 Average results of participants in the entrance and final tests (out of 100, N = 51)
Change in attitudes and personal capacities
According to the theory of the change of the project, increasing the capacity to successfully conduct evaluation depends not only on personal knowledge, skills and attitudes but also on the personal networks of these evaluators. Courses are not only places of acquiring new knowledge and training new skills but of meeting new people that can help us in the future as well. The results of the survey suggest that after the completion of the course, the participants are now more certain in where to seek missing information on evaluation or whom to ask for help.
Change in the self-assessed organizational capacity
Another aspect of the capacity building is enhancing the evaluation capacity of the organizations the course participants work for. Internal procedures, organizational attitudes and networks. For this reason, the participants were asked about the attitudes of their organizations towards evaluation in the survey before and after the course. However, it can’t be expected that a short course for a few participants, often just one person representing an organization, could change an entire organization so fast. The purpose was rather to create positive cores in these organizations on which the change could accumulate like a snowball. In the comparison of the answers before and after the course there is a visible consistent difference yet too small to be considered a major shift.
Feedback on the course and the toolkit
In the survey after the course, the participants were asked whether the course met their expectations in various areas. It definitely met their expectations in learning about evaluation in general and about specific evaluation methods in particular.
Figure 5 Expectations of the participants and how they were met (N = 43)
The participants seemed to be quite satisfied with the course.
Figure 6 Feedback on the course (N = 43)
At the end of the ex-post survey, participants were asked to write some feedback on the course (“Please, sum up in several words your experience with the course. We also appreciate any feedback that would help us to improve the course in the future”). Although the question was not mandatory, participants left more than five full pages of positive feedback and words of appreciation. Most frequently, the participants valued informational wealth, good organization and friendly spirit.
“The course was organized and conducted at the highest level. The commitment shown by the lecturers encouraged the participants to further deepen the secrets of evaluation. Their genuine sincerity and willingness to help during the analysis of the tools prepared by the participants made them want to take part in such a course again. The period of the pandemic and remote learning do not help with learning such a difficult and complicated material as evaluation, which is why the words of appreciation for the ability to transfer knowledge by these two lecturers are even more desirable. Thanks to this, they managed to build such closeness with the participants despite the distance.”
Students appreciated how well organized and lectured the workshops were despite the online form.
The implementation team also collected feedback on the e-learning course and on the toolkit. The questions were:
- What are your feelings after you read the toolkit? / What are your feelings about the course?
- Are there any elements of the toolkit/course you find useful? Which?
- Are there elements which need to be better explained? Which?
- Is there anything else that could be improved in the toolkit/course? What?
- Do you feel that anything is missing in the toolkit/course? Please specify.
Toolkit
The feedback on the toolkit was overall positive and it was considered everything was comprehensible and clear. Especially appreciated were the parts of the toolkit concerning evaluation methods, data collection, how to ask evaluation questions or how to use the theory of change as a basis for the next steps in evaluation.
Course
Opinions on the course were all positive. They appreciated that the course was comprehensible, clear, related to the toolkit and the tests helped to revise the content of the modules. As particularly useful elements of the course were considered tools and methods (especially the summary of the methods), videos, homework commented by the lecturers during the workshops, downloadable PDF files or the part on the data collection.
by Jakub Vrobel
