Transnational European Evaluation Project II (TEEP II)

TEEP II

Self-evaluation manual

1. Background
1.1 Aim of the project
The current project forms the second stage of the ENQA Transnational European Evaluation Project (TEEP I) carried out in 2002-2003 . It aims to contribute to the development of a method for the evaluation of joint programmes and to the process of development of joint degrees in the European context. It does so by evaluating the organisation and management, level and content, and quality assurance systems of three European joint Masters programmes.
1.2 Methodology
The project is based on an internationally recognised evaluation model and is undertaken in dialogue with higher education institutions. It involves:
1. The testing of a common methodology and common criteria;
2. The selection of three joint Masters programmes wishing to participate in the project;
3. A self-evaluation exercise by each of the programme teams;
4. The preparation of a self-evaluation report by each of the programme teams;
5. A visit by an international panel of experts (including both subject area and quality assurance experts and a student) to discuss the self-evaluation report and gather additional information;
6. The preparation of an evaluation report by each of the panels and feedback from each of the programme consortia;
7. The preparation of a summary report on the methodology used and lessons learned.
8. A contribution to the establishment of a methodology shared at the European level.
1.3 Anticipated benefits for participating programme consortia
The likely benefits from participating in the project include:
 An opportunity to share experiences with other programmes, networks and peers, in order to assure continuous improvement of the programme quality and quality assurance;
 The development of criteria that are commonly agreed and that have been tested and offer a dimension of transparency;
 A contribution to the development of the quality assurance of joint degree programmes on the basis of the recommendations from the experts, and identification of good practice for comparable programmes and networks;
 The opportunity to obtain feedback, which may help in identifying opportunities for improvement including the quality assurance ‘culture’;
 The opportunity to promote the institutions, programmes and networks.

2. Framework for evaluation
This section sets out the framework for the evaluation of joint Masters programmes, including some criteria. In establishing the framework, the following have been considered:
 The criteria used in TEEP I;
 The generic reference points for Masters degrees suggested by the Joint Quality Initiative (so called Dublin descriptors);
 The “Golden Rules” for new joint Masters programmes established by the European University Association;
 The generic competencies developed within the TUNING project.

Criteria and regulations that exist within national contexts are also taken into consideration.

As the focus of the project is on the particularity of joint degrees at Masters level, the criteria have been adjusted to suit this purpose. Special account has also been taken of the interdisciplinary nature of the programmes.

While content is an important aspect, the emphasis in the project is on the joint delivery of the programme and the quality assurance system attached to this.

The criteria should not be interpreted as detailed prescriptions but rather as pointers towards the further development of programmes. The project is not intended to lead to either implicit or explicit ranking.
2.1 Organisation and management
Criteria under this heading could include:

a. The aims of the programme are clearly defined.

b. The processes of developing the aims and choosing partners for the programme should be interconnected.

c. The management of all participating institutions supports the goals and objectives of the programme. The programme is fully recognised by all participating institutions.

d. Academic and administrative aspects of the programme are adequately staffed and funded. A sustainable funding strategy is in place.

e. Mechanisms for co-operation, including degree of institutionalisation, role of each partner, financial management, communication system etc, are spelled out and understood by all parties.

f. Responsibilities are clearly defined and shared amongst participating institutions. Lead roles and responsibilities are identified.

g. Information about the programme is easily accessible to students and others.

h. Arrangements for reaching out to and receiving guest students and scholars are in place, e.g. in terms of accommodation, mentor schemes, language courses, activities aiming at social integration, and assistance with visas and social insurance.

i. The infrastructure, e.g. library and other information sources, premises and equipment, meets the needs of the programme.

j. A language policy is in place.
2.2 Level and content
Criteria under this heading could include:

a. The programme has established its own reference points, possibly utilising the Dublin descriptors (below), to ensure that students achieve the competence level required for a Masters degree.


Qualifications that signify completion of the second cycle are awarded to students who:

 have demonstrated knowledge and understanding that is founded upon and extends and/or enhances that typically associated with Bachelors level, and that provides a basis or opportunity for originality in developing and/or applying ideas, often within a research context;
 can apply their knowledge and understanding, and problem solving abilities in new or unfamiliar environments within broader (or multidisciplinary) contexts related to their field of study;
 have the ability to integrate knowledge and handle complexity, and formulate judgements with incomplete or limited information, but that include reflecting on social and ethical responsibilities linked to the application of their knowledge and judgements;
 can communicate their conclusions, and the knowledge and rationale underpinning these, to expert and non-expert audiences clearly and unambiguously;
 have the learning skills to allow them to continue to study in a manner that may be largely self-directed or autonomous.


b. The programme encourages the further integrated development of generic competencies such as:
 Ability to work in an interdisciplinary team
 Appreciation of diversity and multiculturality
 Knowledge of the field of study
 Knowledge of the profession
 Capacity for analysis and synthesis
 Capacity for applying knowledge in practice
 Capacity for generating new ideas (creativity)
 Capacity to adapt to new situations
 Capacity to learn
 Critical and self-critical abilities
 Decision-making skills
 Computing skills
 Ethical commitment
 Interpersonal skills
 Knowledge of a second language
 Research skills
 Other

c. The programme ensures the development of subject specific competencies. [NB. Subject specific criteria will be specified for each programme. The Stockholm launching conference provides an opportunity to discuss such criteria, and participants are encouraged to bring ideas and proposals to the conference.]

d. The programme, through its joint delivery, provides an added value as compared to similar programmes delivered at national level.

e. Teacher qualifications are sufficient and appropriate to the aims of the programme.

f. Opportunities for staff development are provided.

g. The programme is linked to research activities and/or recognised professional standards.

h. The learning environment, including teaching and learning methods and assessment methods, favours the aims of the programme. Assessment methods are common to all parts of the programme or, at a minimum, agreed by all partner institutions.

2.3 Quality assurance
Criteria under this heading could include:

a. The programme formulates and implements a joint quality assurance strategy/ies. Strategies may consider e.g. changes in student demand, external expectations, developments in teaching and learning, and new research areas.

b. Quality assurance practices involve students, staff and other stakeholders from all participating institutions.

c. The programme evaluates whether its aims are met and standards upheld.

d. Quality assurance includes coordination of assessment across the whole programme to ensure that all of its expected competences/learning outcomes are achieved.

e. The programme develops mechanisms for follow-up and continuous improvement.


3. The self-evaluation process
This section presents practical information and advice on the self-evaluation exercise.
3.1 Aim of the exercise
It is important that the self-evaluation exercise fulfils the following aims:
 to provide a framework to stimulate internal discussions within and among the participating institutions;
 to include ample time and scope for reflection on emerging matters;
 to provide opportunities to comment on and assist continuous improvement in the quality of the programmes.
3.2 Self-evaluation group
For the completion of each self-evaluation report, a group should be set up consisting of representatives from institutions participating in the programme.

The group should appoint a chairperson, responsible for co-ordinating the work and for being the contact point between the self-evaluation group and the external body responsible for the overall evaluation. The group should include at least one student involved in the programme.

The self-evaluation group (including the chairperson) should be officially designated by the Board/Council/Committee responsible for the programme.

It is desirable that a draft of the self-evaluation report be circulated for comment among staff and students involved in the programme at each institution before it is submitted to the external body.

Each self-evaluation group is responsible for the preparation of their report.

3.3 Self-evaluation report
The self-evaluation report should provide accurate documentation to be used by the panel of experts in their preparations, site visit, evaluation and report writing. It should also provide clear evidence of the reflective discussions that were an integral part of the self-evaluation exercise.

The self-evaluation report should be not more than 25 pages in length, structured as far as possible in accordance with the headings of section 4 below.

The report should be submitted electronically and in 6 paper copies to the contact person of the external body responsible for the evaluation of the programme.


4. Themes for self-evaluation
This section sets out the areas to be covered in the self-evaluation exercise and self-evaluation report.

With the exception of the factual information requested in section 4.1, the report should express evaluation and analysis rather than sheer description. Identifying strengths, weaknesses, opportunities and threats will be an important aspect of this. The bulleted questions under each heading may serve as a starting point to your self-evaluation. They do not however constitute a compulsory or complete list of issues.

4.1 Facts and figures
4.1.1 Baseline information
Name of programme
Name of co-ordinating institution
Names of other participating institutions
Length of programme (academic years + ECTS)
Website address of programme
4.1.2 Students
Please copy the table and provide the following information for each institution admitting students to the programme.

Institution X
No. of applicants No. of admissions No. of degrees awarded No. of students admitted from each institution
W M W M W M X Y Z … …
2002
2003
2004


Institution Y
etc.
4.1.3 Teaching resources 2004/2005
Please copy the table and provide the following information for each institution contributing teaching resources to the programme. [NB. This table will be discussed at the Stockholm launching conference. Participants are encouraged to consider its appropriateness, and possible amendments.]




Institution X
Teacher (names not required) Title Contribution to the programme (full-time equivalent) Gender Academic degree Academic field/discipline Professional qualification (if any) Research activity (member of research team)
1
2
3
etc.

Institution Y
etc.

4.2 Organisation and management
4.2.1 Aims of the programme
Please analyse and evaluate the aims of the programme, e.g. by addressing:
• Why was the programme established?
• What are the overall aims of the programme?
• How were the aims determined?
• Other
4.2.2 Choosing partners for the programme
Please analyse and evaluate the basis on which partner institutions were selected e.g. by addressing:
• How/through what process did partner institutions come together to set up the programme?
• What motivated the various partner institutions to join?
• Other
4.2.3 Mechanisms for co-operation
Please analyse and evaluate the mechanisms of co-operation within the network, e.g. by addressing:
• What are the respective roles of the programme co-ordinating team and the different institutions within the network? Are the roles clear and satisfactory?
• To what extent have the mechanisms for co-operation been formalised and agreed?
• How are financial matters managed within the network? Is the system satisfactory?
• What are the methods of communication?
• How often and under what circumstances does the programme co-ordinating team meet to discuss the running of the programme?
• Do staff of the participating institutions meet to discuss their respective contributions to the programme?
• Other
4.2.4 Support and recognition
Please analyse and evaluate the level of support and recognition that the programme receives, e.g. by addressing:
• What is the legal status of the programme and the award?
• What are the respective roles of the different institutions in the recognition process?
• Have you encountered any obstacles with regard to recognition? If so, what?
• In what way is the programme supported, e.g. by management of the individual institutions?
• Other
4.2.5 Staffing and funding
Please analyse and evaluate the staffing and funding of the programme, e.g. by addressing:
• Is there a joint funding strategy for the programme? If so, please describe this.
• On what basis are joint resources distributed?
• What academic and administrative staff resources are attached to the programme? Please comment on teaching resource figures provided at 4.1.3.
• What funding requirements does the running of a joint programme have compared to a programme provided by one institution only?
• Generally speaking, are the resources available to the programme sufficient?
• Other
4.2.6 Infrastructure
Please analyse and evaluate the infrastructure available to the programme at each of the partner institutions, e.g. in terms of IT equipment, library and information services, and premises at the various partner institutions.
4.2.7 Recruitment and information
Please analyse and evaluate the student recruitment system, e.g. by addressing:
• How are prospective students informed about the programme by the various partner institutions?
• Other
4.2.8 Reception of students and scholars
Please analyse and evaluate the arrangements for receiving guest students and scholars, e.g. by addressing:
• How are students introduced to the programme and to the courses offered at each institution?
• What kind of assistance do guest students and scholars get at the various institutions? Is the support system satisfactory?
• Does the cultural heterogeneity of the student group require special attention? If so, in what way? How may the cultural heterogeneity benefit the programme?
• Other
4.2.9 Language policy
Please analyse and evaluate the language policy of the programme, e.g. by addressing:
• What are the language requirements for completing the programme?
• What special provisions are made regarding language?
• Does language pose any particular problems? If so, how are they solved?
• Other
4.3 Level and content
4.3.1 Programme structure
Please analyse and evaluate the structure of the programme, e.g. by addressing:
• To what extent does the programme have a compulsory ‘core’?
• What are the optional elements?
• What is the progression within the programme?
• Other
Please enclose a programme description, e.g. in the form of a chart, outlining the different parts of the programme including compulsory and optional elements.
4.3.2 Academic standard
Please analyse and evaluate the academic standard of the programme, e.g. by addressing:
• How is compatibility of academic standards throughout the programme achieved?
• Does the interdisciplinary nature of the programme present particular challenges?
• What academic added value is achieved through the joint delivery of the programme?
• What are the student admission requirements? What is the academic level of students at entry?
• To what extent does the profile of the academic staff match the aims of the programme?
• What are the opportunities for staff development?
• In what way is the programme linked to research and professional activities at the participating institutions?
• Other
Please enclose reading lists for the various parts of the programme.
4.3.3. Teaching and learning methods
Please analyse and evaluate the teaching and learning methods used in the programme, e.g. by addressing:
• What teaching and learning methods are used within the programme (e.g. lectures, small group teaching, seminars, course work, thesis preparation/supervision, laboratory work, trainee placements etc.)? What methods are the most commonly used? What are their respective pros and cons?
• What assessment methods are used within the programme (e.g. written examination, laboratory experiment write-up, essay, oral examination, project report)? What methods are most commonly used? What are their respective pros and cons?
• Is there a common approach to teaching and learning methods within the programme or do these vary from one institution to another, and/or between different stages in the programme? If so, how?
• In what way does the learning environment favour academic achievement?
• To what extent do the teaching and learning methods contribute to critical and independent thinking?
• Other
4.3.4 Competences/learning outcomes
Please analyse and evaluate the expected competences/intended learning outcomes of the programme, e.g. by addressing:
• What are the expected competences/intended learning outcomes of the programme? Which of these are subject-related and generic, respectively?
• How were the expected competences/intended learning outcomes developed?
• Are the expected competences/intended learning outcomes published and made available to students and staff?
• To what extent are the expected competences/intended learning outcomes designed to reflect the aims of the programme?
• Are the “Dublin descriptors” and/or the competencies developed within the TUNING project relevant to the programme?
• To what extent have the expected competences/intended learning outcomes been attained? How do you know?
• Other

4.4 Quality assurance
4.4.1 Quality assurance strategies
Please analyse and evaluate your strategic work on quality assurance, e.g. by addressing:
• What is the overall philosophy and strategy for quality assurance at programme level?
• Do separate strategies exist for quality assurance at programme and thematic/course level?
• Other
Please enclose strategic document/s, if available.
4.4.2 Participation in quality assurance
Please analyse and evaluate the level of participation in quality assurance, e.g. by addressing:
• Where does the responsibility for quality assurance at programme level lie?
• What parties are otherwise involved (e.g. at thematic/course level)?
• What is the level of participation by students, staff and/or other stakeholders?
• Other
4.4.3 Quality assurance practices
Please analyse and evaluate your quality assurance practices, e.g. by addressing:
• How is quality assurance organised at programme level and thematic/course level, respectively?
• How is the programme evaluated (e.g. with regard to aims and objectives, student competences/learning outcomes, teacher competence and staff development, teaching and learning methods, relevance in relation to labour market requirements)?
• What use is made of input from e.g. reports from accrediting or other external bodies, staff and student feedback, feedback from former students and their employers, response from professional organisations, labour market representatives or discipline associations, student progress information, external examiners’ reports?
• Other
4.4.4 Follow-up and improvement
Please analyse and evaluate the mechanisms for follow-up and continuous improvement, e.g. by addressing:
• Where does responsibility for follow-up lie?
• How are shortcomings identified and remedied?
• Other
Please provide example/s, if any, of how the quality assurance practices have resulted in changes to the programme to the benefit of staff, students and/or other stakeholders.
4.4.5 The self-evaluation exercise
If you wish, please comment on your experience from this self-evaluation exercise, e.g. by addressing:
• What is the usefulness of the guidelines?
• What alterations do you suggest, if any?
• Other
Your feedback is most valuable in the process of developing a model for the evaluation of joint degrees.

4.5 Summary
Please provide a summary of the main strengths of the programme and the main challenges it is facing, as you see it.


Checklist
Please submit the following documentation to the contact person of the external body responsible for the evaluation of the programme:

 Self-evaluation report ( 25 pages)
 Quality assurance strategy/ies (if available)
 Programme description (e.g. chart) outlining the different parts of the programme including compulsory and optional elements
 Reading lists for the various parts of the programme

The documentation should be submitted electronically as well as in 6 paper copies.