The current global economic situation and the wide and complex view featuring Occupational Training as a pillar for the economic revitalization affect the way of carrying out their assessment. Several authors (Gelpi, 1987; De Miguel, 2000) emphasize that the evaluation is essentially a social, cultural fact and not merely technical and methodological. Flor Cabrera (2000) states that the evaluation has become a really complex activity but, in turn, a powerful tool that cares for the quality of educational activities. The evaluation of training is also a tool for quality.  

In this context a set of factors has been identified (Perales, 2000) that describe characteristics of the evaluation in this field within the plurality of approaches and evaluation proposals. Among these the following are included. 

1 -Assessment of Occupational Training has received scant attention from the theoretical and research field, although in recent years has arisen a significant number of working groups on the topic that provide interesting proposals.

2 -The dynamic nature of this type of training leads to a lack of stability required to a proper care for the evaluation processes.

3 -Its link to the active employment policy confers on it a role and specific evaluation targets, not exclusively educational.

4 -It is a highly complex area of study because of the diversity of programs that gathers in order to accommodate a wide variety of groups and audiences involved, which can lead, sometimes to evaluation approaches with statements even found.

   Considering these characteristics that affect the field of occupational training, you can classify and identify different types of evaluation of these programs. 

A. Assessments that establish as reference the unit (person, company, partnership of training institution, etc ...) on which occur the effects of training (Jornet and others, 2001).

B. Assessments that establish as a criterion a certain way to operationalize the concept of quality (quality of program, service, etc ...) (Perales, 2000).

C. Assessments which refer to a number of specific criteria (subject to evaluate, purpose of the evaluation and control) to occupational training plans (Jornet, González and Pérez Such Carbonell, 1996; Jornet, 2000; Perales, 2000). 

The study described in this article is part of the second type proposed, taken as a evaluation criterion the Quality of Service, performed through perception and satisfaction that users have of the adaptation and development of both theoretical and practical training program. 

In the national context are diverse initiatives that seek directly or combined with other functions of the evaluation the analysis of users’ perception as an object of study, which is one of the common objectives in the evaluation of Occupational Training Programs. In relation to this approach, and without wanting to be exclusive, we emphasize in the national context, the evaluation proposals of Tejada (1992), the Evaluation Model of Occupational Training in the Autonomous Community of Valencia (Model EFO, 1996), the Evaluation Model of Basque Country (EGAILAN, 1999), as well as the Assessment System of Training for Employment of the Autonomous Community of Andalusia. The difference in evaluative approaches and the lack of public availability of data has not allowed a comparison between the results of the various initiatives. 

In the case of Asturias quality assessment from the perspective of users is embedded within a broader assessment model that combines three approaches (context analysis, quality of service and quality of the program), as reflected in the Figure 1.

Figure 1: Global Model of Evaluation and Improvement of the Training for Employment in Asturias

    The work reported in this study is inserted within the scope of the Quality of Service, assessing the development of the program through the satisfaction of the students that have participated in training activities. 

3. OBJECTIVES OF THE STUDY 

By 2000 the Principality of Asturias, through the Ministry of Labour and Employment, takes over the functions and services related to the management of Vocational Training for Employment[iii]. This transfer of powers to the Autonomous Community expressed the need to systematically evaluate Vocational Training for Employment which until then had been managed and supervised centrally from Madrid by the old National Employment Institute (INEM), whose evaluation was limited to some general aspects that were considered jointly in the rest of the Spanish territory. 

For this purpose, the same year 2000, a Framework Agreement between the Government of the Principality of Asturias and the Research Group "Canella" at the University of Oviedo was established, to carry out a project aimed at evaluation of Vocational Training Employment, following the model set (Figure 1), in order to improve the quality of this type of training. Among its objectives are: 

1.Centers/Associates of FPE Plan Accreditation.

2.The evaluation of satisfaction of participants in the courses of FPE[iv] Plan.

3.The evaluation of Labor Insertion of students trained in the courses of FPE Plan. 

  The purpose of this agreement is to provide the Government of the Principality of Asturias updated and reliable information about the quality of the Centers / Institutions collaborating with the FPE Plan and actions developed through this program so that it can be used to plan a quality training offer and to make decisions that help to optimize it. 

The research described in this paper aims to evaluate the degree of satisfaction of participants in training activities aimed mainly at the unemployed, based on the following specific objectives: 

●   Estimate the quality of the training in its different aspects (pedagogical, organizational, educational, etc.) through the assessment made by the students about the training.

●   Carry out a comparative analysis of the results obtained over the years under study (2002-2008), in order to know the evolution of this type of training across time.

●   Provide all audiences involved (directors, centers, trainers and technicians) and especially the Administration, relevant information about how the various aspects of educational processes are assessed.

 

4. METHODOLOGY 

4.1. Design 

  From the methodological point of view, this research is an observational study based on surveys, as has been the use of questionnaires the system used to collect the necessary information. The design made can be typified as preordained, selective and of repeated measures. It is preordained as have been previously planned in detail all phases and elements of the research process. It is selective because the methodological strategy used to collect information allows to detect the variability between subjects and to determine differences between groups. Finally, we consider repeated measures since the assessment has been conducted seven times -with the same parameters but with different populations - between 2002 and 2008. The replication of this design for a period of years and the amplitude of the samples used -as it is stated later- are two clear strengths of the study to provide the research with consistency and the research results with validity. 

4.2 Instruments / variables 

The assessment of the satisfaction consisted of a semistructured questionnaire addressed to students of training actions of the Subsystem for Employment Training. For its design were organized several working group sessions with expert staff in the area of vocational training, in order to discuss and establish the structure and content of the instrument to use. Representatives from the different sectors involved participated in these sessions: management, associates and staff of training areas. This system of consultation with experts and the annual reviews carried out permitted to provide the questionnaire with a high degree of validity in relation to the purpose of the survey. Given the size of the instrument designed in this paper is attached (Appendix I). The following is a summary of the resulting structure.

Table 1. Structure of the Questionnaire

    We have also analyzed the reliability and metric validity of the instrument implemented, through the Cronbach Alpha coefficient and factorial analyzes. Referring to reliability, the Cronbach Alpha scores obtained each year range from 0,942 and 0,956 which allows us to state that the instrument has a very good internal consistency. For construct validity we performed principal components analysis with varimax rotation, removing from the resulting component matrix values that have a lower saturation level of 0,35, on the samples for the years 2004 and 2008. From this analysis we obtained similar matrices in both samples, both the variance explained and the number and types of factors extracted. However there is a significant improvement in the variance between the year 2004 (61,290) and 2008 (63,813) and a different range between factors 2 and 3, probably due to the changes introduced to improve the questionnaire.

In both tests (2004 and 2008) the greater weight of the explanation is associated with student satisfaction related to quality of teachers and teaching activities (dimensions C and D of the questionnaire - Table 1). In second place are two other factors with lower weight concerned with the utility of the training activities for access to employment and organizational matters of the courses, which are structured on the basis of integrated variables in the dimensions E, F and B of the questionnaire. In summary, in view of the data obtained in this analysis we can conclude that the construct validity of the questionnaire is high, given the similarity between its dimensional structure and matrices obtained. 

4.3. Procedure / Application 

The figure below shows the different actors involved in the methodological process followed for the assessment of student satisfaction in training activities aimed at unemployed workers  of the Subsystem of Employment Training, as well as information flows established between them. 

Figure 2. Agents involved in the evaluation and information flows established between them 

    The Vocational Training Department performs an annual call for training activities in which the Collaborating Entities participate in a competitive concurrency based on evaluation criteria. Once requests are estimated, an announcement is published assigning the training activities to the Entities with better scores on the scale laid down. Entities start the training and it is administration staff (Training Area) which is responsible for monitoring the action, checking to see its proper development. In the final phase of training this technical staff implements the questionnaire, which is by them previously known since they have participated in its elaboration. Once filled in by the student is sent to the Investigation Team Canella for its treatment and analysis by methodologically differentiated procedures based on the information gathered. 

For quantitative items was accordingly decided to machine-readable by optical reader to perform descriptive and differential analysis later. For open questions in the questionnaire was made a dump of the qualitative information, organized in terms of dimensions and categories subsequently established. The attached table lists the frequencies and percentages of the many records obtained. 

Table 2. Dimensions, categories and frequencies for qualitative analysis

 4.4. Participants 

The study would cover the total population of students participating in training activities offered by the Principality of Asturias aimed mainly at unemployed people during the period between January 2002 and December 2008. The total number of participants amounts to 87.298[v]. The initial objective of this study was to reach the total number of students trained annually, although for reasons specific to the dispersion of the training was not always possible to achieve this goal, but it can learn highly representative sample figures in relation to the different variables analyzed (age , sex, geographic area, profesional family and training specialty). The total number of subjects interviewed in this period was 49.485 which represents 56.69%. The following table specifies the population data and sample over the years.

Table 3. Population and study sample

     Analyzed the sample according to the variables sex and age, there is a predominance of female participation (56,03%) over males (41,47%), also highlights the age range between 25 and 34 (39,06 %) as the dominant category in the age of the respondents. The attached graphic specifies the data dichotomized according to age and sex.

Chart 1. Sample distribution by age and sex

    Based on academic qualifications held by the students participating in the training activities can be noted that the highest percentage belongs to those who have completed the Graduado Escolar (Certificate in Basic Schooling)/ EGB (Elementary Education Level) / ESO (Compulsory Secondary Education)/ High School Graduated (20,80%). Another significant percentage is made up of those with Baccalaureate (16,60%). Third, there are those who have obtained a Higher Level Training Cycle (15,52%) and then those who have completed a University Diploma (12,10%), as reflected in the graphic that follows. 

Chart 2. Sample distribution by academic qualification and sex

    Crossing level of qualification and sex, we observed that women are the majority in the medium and higher degrees, while men predominate in lower training levels, with higher figures in Graduado Escolar (Certificate in Basic Schooling)/ EGB (Elementary Education Level) / ESO (Compulsory Secondary Education)/ High School Graduated, in Certificate of Primary Education, in the Incomplete Primary Schooling, in the Social Guarantee Programmes and between those who are Out of school.  

Analysis of the responses to other issues incorporated into the questionnaire should be noted that 43,63% of students had already participated in other training activities aimed at unemployed workers in the last 5 years, most of them in actions of the FIP Plan itself. Likewise we find that 16,44% of students has received another kind of training simultaneously. In any case, the choice of the training has had on a preferential basis in the 84,47% of cases. 

The two main reasons expressed by the participants to enroll in the course are to increase professional knowledge (8,71) and facilitate their integration into the labor market (8,44). Here are other reasons with less weight, as stated in the following table.

Table 4. Reasons for enrolling in the training

5. ANALYSIS AND RESULTS 

This section shows the results obtained from analysis of data collected. For their exhibition we have followed the dimensional structure of the questionnaire, by mapping the dimensions used in the analysis of quantitative and qualitative data, as shown in the attached table.

Table 5. Correspondence between quantitative and qualitative dimensions

5.1. Satisfaction of participants: global data 

A. In terms of planning and management of the course, students have felt that this was appropriate with an average score of 7,87 on the scale. Looking at each of the items separately, which has achieved a higher score refers to the Security measures the courses have been developed, being Schedule and duration of the course the least satisfactory.

Table 6. Course Planning and Management

  In the qualitative data, the observations related to this dimension (n = 2009) in which more emphasis is made by students who participated in training activities are related to the Uniform selection of students (34,35%) and Transport grant (24,79%).

The following are by way of example, some of the comments most representative of these categories:

“The screening test that we were made to participate in the course seemed appropriate, but should also evaluate criteria such as vocation, interest and willingness to work”. 

“… The selection of students should be "refined" to choose the people concerned and with similar levels…”. 

“It's good that we are paid for transport but also should charge a monthly training grant, since all students are unemployed”. 

“They should pay the money prior to transport tickets, as many times (almost always), the payment is made with 2-month delay”.

   One of the issues that are particularly important in this dimension is the practical work in companies, which participants place a high importance (67,92%). However, we found that the interest expressed by students do not always correspond to what is really done, because despite a 67,92% demand for internships in companies, only 27,50% say he has done or plans to do internships in companies related to the course. 

It was also noted that a high percentage of trainees claims not having received information in relation to the placement (34,38%) and most of them (73,98%) have not previously had any work contact in the training specialty chosen. 

In regard to the evolution over the years of this dimension, we see that the highest scores are achieved in the period between 2004-2007. However, focusing on the data referring to 2008, we see that the scores do not overcome those achieved in 2007, maintaining a clear decreasing trend, except for items concerning the Safety measures with which courses have been developed and the Explanation of course objectives and its organization in the beginning, which get higher values. The following graph shows the evolution of the scores analyzed.

Chart 3. Evolution of scores in dimension - "Training Planning"

B. Material and human resources receive a very positive assessment (8,25). The highest scores in this category are related to teacher competence, both theoretically and practically. 

Chart 4. Human and Material Resources

Qualitative information on this dimension shows a large number of frequencies (n= 2891), most of which refer to the lack of material resources (62,44%), as reflected in the following typical phrases: 

“Greater supply of material for practice, since it is insufficient”  

“The computer equipment should be updated and ready to be used” 

If we compare the values obtained over the years under study in the dimension of "Human and Material Resources" we face two extremes. On one hand, the items concerning the Professional experience of teachers, the principal's attitude to the problems and physical conditions of the classroom and the center have had the best scores in 2002; while items which value the Faculty of theory and practice achieved the top scores in 2007. In view of the comparison, we observe that the results for this dimension in 2008 have fallen across the board compared to previous years. 

We must take into account the fact that, again, we face items that could not be compared with the first two years due to changes in the questionnaire. And, if in 2002 and 2003 the Domain of teachers in the subjects they taught was assessed, from 2004 the faculty of theory and practice is valued separately. The data evolution is represented in the chart below.

Chart 5. Evolution of scores in dimension - “Human and Material Resources”

C. The largest bloc of the questionnaire refers to the methodology and learning climate, whose scores are included in the table below. 

Table 7. Methodology and Learning Climate 

   As you can see, the most valued variable refers to the relationship with faculty (8,91). Also, clarification of doubts by the teacher and peer relationships, achieve equally high scores, 8,71 and 8,61 respectively. By contrast, the lowest score refers to the implementation of external actions during the course to expand the training of participants, through visits to companies, exhibitions, fairs, etc. with an average of 4,57. This variable gets the lowest score of all that make up the questionnaire. 

In the qualitative data of this dimension (n = 9.544) clearly highlight the observations related to the Course Duration  (27,28%), and the Practices Management (17,01%). Here are some of the most representative comments: 

“Students should been told they can make practices, and which procedures they have to perform to it, in addition, companies should provide for practice and not that we have to look for it”. 

“Give higher job prospects for courses with internships”. 

“Good teachers, highly involved in their work”. 

“It would be required more control of teachers quality, as they know very well the teaching but not the specific topic of the course, they are not specialists in the subject they teach and it shows…”.

Regarding the procedures for evaluation of courses by the teachers, the 33,92% of the students have stated that the evaluation was carried out using real practice. Also, the written exams as a way of evaluation has been targeted by a high number of participants (30,98%). By contrast, the least used is the interview with the teachers (2,71%), followed by the self assessment (4,75%) and individual monitoring sheets (6,69%). 3,45% state not having been evaluated.

Table 8. Assessment of learning by teaching staff

    Considering the comparative analysis (2002-2008), it appears that most of the scores on this dimension have been declining compared to 2002. If in the first year the students were more motivated and interested, the valuation of 2008 has decreased slightly from 8,88 to 8,20, although it remains significantly positive. The same happens, among others, with the Level of participation of students in the group, which descends from 8,24 to 8,00. It is logical to assume that both items could be related. 

The decline in scores deserves reflection on each of the items assessed, since we know that student learning and motivation and interest shown to the content are factors that are influenced by the type of dynamics used by teachers and the atmosphere established in the classroom between teacher-student and between the students themselves. 

Due to changes in the questionnaire it has not been possible to compare the items related to the Interest shown by students in the course and the Unequal level of knowledge among participants. The data evolution is shown in the following chart.

Chart 6. Evolution of scores in dimension - "Methodology and learning environment"

D. Overall assessment of the training 

In addition to the differential aspects taken into account when assessing each of the dimensions analyzed in the study, students were asked to conduct an overall assessment of the training, whose results reflects the following table.

Table 9. General evaluation of the training

    As we can see, although the overall assessment of the actions is good, it is remarkable that the item referring to the support or guidance from the schools in terms of labor insertion, get a slightly lower average then the other items , which significantly influences the overall average of the dimension. This means that students not only expect to acquire theoretical and practical knowledge around a particular subject, but also want to receive guidance for, once the training has finished, conducting an active job search. 

From 2005 on, with the aim of comparing the overall assessment more accurately, a Global Index of Satisfaction has been built, with the average of the scores of all questionnaire items, excluding which refers to the Global Rating of the course. This index is 7,78, and shows that there are no significant differences, with the overall score for this dimension (7,84). 

Concerning the analysis of qualitative data (n = 1.717), observations in which students more emphasis made relate to the General satisfaction with the learning experience (53,39%) and with the influence on the employment situation (41,95%), as shown by the following representative comments: 

“In general I think the course has been positive in all respects. Personally, I am very happy”. 

“The training received has been exceptional. Thanks to the professionalism of the center and the faculty, the learning has been a pleasant proccess, with a very satisfactory outcome and beneficial to the student”. 

“I think it was a very useful course if we refer to the training aspect, but it will not facilitate my access to labor market, as it is not approved“. 

“I found the course interesting and well used, but regarding the job prospects look pretty bad". 

Another way of overall assessment of the training has been through the impact estimates issued by the participants. In the attached chart (Chart 7) are specified these estimates over the years. As we can see, the best aspect considered by the students has been the expectation in the Improvement of their general training, so that is valued not only to obtain a professional qualification in a particular specialty, but also the acquisition of general knowledge.

Chart 7. Evolution of scores in dimension - "Expected Impact"

  Finally we performed diachronic analysis on the overall assessment of the training, whose data are shown in Table 10. If we compare the scores from 2002 to 2008, we see that until 2007 have progressively improved in regard to the Interest generated by the course and the Overall rating that is given to it. However, in 2008 there has been a general decline for both these two items and for the one related to Support to the professional integration by the Center. In it was the best score in 2002, resulting from this year on fluctuations in scores, to finally arrive in 2008 to the lowest score. Given the decreasing trend in the ratings it would be important for centers/entities that provide this type of training to improve this aspect because, despite among the benefits expected by the students are to improve their general education, professional qualification and to achieve the labor market, lacks adequate labor orientation to advise them about entering the market.

Table 10. General rating of the training activities (2002-2008)

 5.3. Results by Professional Branches[viii]

Training activities are structured according to the Professional Branches that bring together different sectors of productive activity. The attached table reflects the Branches with higher and lower satisfaction ratings each year.

Table 11. Professional categories with higher and lower scores (2002-2008)

    Professional Branches which over the years have obtained a higher score are Transport and Vehicle Maintenance, Arts and Crafts, Chemistry and Health. Those who have obtained the lowest scores are Energy and Water and Information and Artistic Expression.

 However, these results should be interpreted with caution. If we perform the analysis individually for each year under review we can find Branches who are in a global analysis with a high score, and yet, in a particular year have a low valuation, such as "Chemistry[ix]”.

 5.4. Results by Training Specialties 

Apart from Professional Branches, the training activities are organized by Specialties that refer to specific work activities. The level of satisfaction in this case allows to detect the assessment made by students in specific courses scheduled for a particular job position. The attached table outlines the scores for the Training Specialties more and less valued in each of the years under study. 

When it comes to analyzing the results by Training Specialties shows that there is no correspondence from them (nor those who scored higher or lower) over the years under study. This is because on one hand, the list of Training Specialty is very wide and, secondly, every year are scheduled training activities linked to competencies required in the labor market, causing that both specialties and their scores vary greatly from year to year.

Table 12. Training specialties with the highest and lowest score (2002-2008)

6. SUMMARY AND FINAL VALUATION

6.1. Summary 

   We present a summary of the results obtained attending to the main dimensions studied. 

●       Participants Profile

●  Of the total population of students participating in training activities aimed mainly at unemployed people (N=87.298) was evaluated satisfaction in the 56,68% (n=49.485). Within this population highlights a preponderance of women (56,03%), compared with 41,47% of men.

●  The age group most representative among the participants is in the range 25 to 34 years, both women and men. The most frequent academic qualifications are: Graduado Escolar (Graduate in Basic Schooling)/ EGB (Elementary Education Level) / ESO (Compulsory Secondary Education)/High School Graduated (20,80%), Baccalaureate (16,60%), Higher Level Training Cycles  (15,52%) and University Diploma (12,10%).

●  In 84,47% of cases the choice of the training has had a preferential basis. The main reasons why the students argue for the training in which they participated were: to expand their professional knowledge and facilitate their entry into the labor market. 

●       Satisfaction Results 

A.     Course Planning and Management. 

●  This dimension has achieved an average score of 7,87. Within it, the most valued issue has been the security measures with which teaching have been developed (8,57), beeing the schedule and duration of the course the least satisfactory (7,68 and 6,95 respectively). With regard to qualitative information, suggestions of the participants relate mostly with the homogeneous selection of students (690) and the transport grant (498).

●  The 67,92% (n=33.608) of the students demand internships in companies, compared to 27,05% (n= 3.387) who is not interested in them. Regardless of possible interest in an internship, the 61,77% (n=30.566) states that the centers have provided them information about the possibility of performing them.

●  Only 21,59% (n=10.683) of the students had previously worked in a job related to the specialty in which has been trained, compared to 73,98% (n=36.609) that have not done.

●  The scores were better in the years 2004, 2006 and 2007. However, if we focus on the data referring to 2008, scores fail to exceed those in 2007, maintaining a clear decreasing trend. 

B.     Material and Human Resources 

●  This dimension has been the most valued of all that make up the instrument, receiving an average score of 8,25. Within it, the most valued aspect has been the experience of teachers (8,88), whereas the worst value refers to the physical conditions of the classroom and the center (7,72). In the analysis of qualitative data on this dimension highlight the contributions related to the lack of material resources (1.805), asking for new material or assessing their condition.

●  It has obtained the top scores in 2002 and 2007. However, in view of the comparison, we observe that the results for this dimension in 2008 have fallen across the board compared to previous years. 

C.     Methodology and Learning Climate 

●  This dimension has achieved an average score 7,85 and the best valued issue has been the good relationship with faculty (8,91). However, the one that has reached the lowest score is related to the implementation of external actions (company visits, fairs, exhibitions, etc..) (4,57). In the qualitative information, highlight observations related to internships management (1.623) and teachers (1.332).

●  In 33,92% of cases (n=26.025), assessment of learning has been carried out using real practice, followed by written examinations (30,98% - n=23.766).

●  It is observed that most of the scores have been declining compared to 2002. The only item which reflects a slight improvement is the one that value if the faculty has clarified the doubts, which has increased from 8,32 to 8,67 in 2008. 

D.     Overall Rating of Training 

●  The overall rate of satisfaction (2005-08) has achieved an average score of 7,78. If we compare with other variables we find that women are engaged in a more positive assessment (7,93), versus men, who give an average score of 7,75. By age, the population over 55 years gives the best rating to the actions (8,07), while the age group that gives a lower score is the range between 25 and 34 years (7,74). By academic qualifications, those with Primary School Certificate give the highest rating with 8,17, while those who held a university degree offer the lowest score (7,48).

●  Following the evaluation of training performed by Training Areas [x] in Asturias, we observed that most rural areas are those that provide a higher score: Southwest Area (8,63), followed by the East Area (8,17) and Northwest (8,06). Lower average scores have been provided by Gijón and Oviedo areas with 7,84 and 7,89 points respectively. 

E.      Expected Impact 

●  The most remarkable aspect has been the improvement of supplementary general education, with an average score of 8,14. The improvement of employment guidance has been the item that has received the lowest score (7,12). In the analysis of qualitative data, observations which students highlight are related to their overall satisfaction with the learning experience (938) and with the influence on the employment situation (737). 

As summary, a table which shows the average score in each of the dimensions studied is included: 

Table 13. Average rating for the dimensions of the questionnaire