Own Evaluation

= Objectives =

Main goal
The main goal of this project is to create a self-evaluation software for course members, taken the example of the course UNIK4700. Through the focus on a life service, given in autumn 2013 to students of UNIK and UiO, an immediate feedback from users can be taken into consideration for the evaluation of the course.

Background and objectives
UNIK has identified eUNIK as a focus area for the further development of UNIK. Being a development

Different software tools have been evaluated during the last years, including Moodle, It's learning and Fronter. A preliminary analysis of these tools is documented in Courses and Knowledge and the Moodle Manual, pointing out that the main usage of these tools is to administrate deliveries from students to their teachers, thus allowing a documentation of the provision process. Though some of the tools support multiple-choice and the check of numbers, none of them supports the evaluation of differences between numbers, answers including a list of keywords, spelling mistakes and logic relations.

Our draft analysis of the Semantic MediaWiki (SMW) framework has verified that SMW is a suitable tool for self-evaluation, and developments will therefor focus on the implementation of these features. Our expectation is that 80% of the teachers work in evaluation can be performed through automated processes using the wiki.

In order to structure the work, the following objectives have been identified
 * establish a framework for self-evaluation, including the identification of potential bottlenecks
 * convert the questions "what have you learned today?" from each UNIK4700 lecture into quantifiable questions
 * establish a development plan for self-evaluation based on multiple choice, numbers, keywords and meaning
 * document the progress of the development in an electronic form, explaining both achievements and identified shortcomings of the approach.

The objectives are translated into activities, each of them leading to a measurable outcome. The analysis of the outcome thus allows to terminate/reduce of extend the project. = Technical work = The work is subdivided in activities, each of them having a dedicated outcome

Activity 1: Framework for self-evaluation
This activity will establish the framework for self-evaluation. The framework is seen as a list of wiki modules and a structure of pages on the wiki. The modules, being candidates for self-evaluation, will be briefly discussed to identify their potential of being used for this project. The framework will contain a high-level description of how activities 2-5 can be implemented. The structured representation in wiki pages will enhance the understanding of the implementation logic. The framework will also contain an identification of potential bottlenecks, being related to "quantifiable answers", relations between structures, wiki shortcuts and others.

The task will be executed by Skaperhuset, with commenting support from UNIK.

Activity 2: Quantifiable Questions
The first task in activity 2 is the provision of examples of questions with quantifiable answers. This task will be performed by Skaperhuset and UNIK in common. UNIK will be responsible for providing the questions for the Lectures of UNIK4700, according to the examples developed in Activity 1.

The student will answer 1 or more questions online, then submit the answers. The system will then provide not only right or wrong, but also indications on help and further reading.

Skaperhuset will provide UNIK with the structure on how to provide extended answers to submissions from students.

Activity 3: Development plan
Based on Activity 1 and an analysis of existing wiki developments, this activity will provide the development plan for self-evaluation based on multiple choice, numbers, keywords and meaning. The plan will contain an estimation of time/duration of the tasks, as well as details on the technical implementation.

Activity 1 & 3 are seen as a proof-of-concept of the envisaged project, and are the basis of adjustment for the implementation plan.

Activity 4: Development of multiple choice module
The following activities represent incremental modules for the self-evaluation of users. They are listed in chronological order based on the expected increase in the complexity.

Activity 4 handles multiple choices, where the students are met by a list of potential answers. This activity, though being straight forward to implement, needs the framework of activity 1 and the help-functionality in case of wrong answers.

Activity 5: Development of numbers module
The number-based evaluation shall not only use the "equal" functionality as known from other tools, but should also provide answers based on comparison of submissions with the correct answers. The creator of questions will add the correct number, and three indicative ranges to the answers. Each submission, falling into one of the ranges, will provide an own answer to the student.

Example:
 * The correct answer is "3.4", the ranges are set to "0.1", "1", and "3".
 * If a student submits "3.4", then the system will provide the answer of range 1, e.g. "you are within 0.1 of the result"
 * If a student submits "1.0", then the system answer might be "some of your assumptions are wrong, please check them"
 * If a student submits "0.1", then the system answer might be "you are completely off track, please check the background by reading ...."

This part is already seen as a major advance as compared to current tools.

Activity 6: Development of keywords-based self-evaluation
Keyword-based self-evaluation is seen as a major improvement as compared to current tools. While current tools only check the consistency of one keyword, which needs to be provided with the correct spelling, the semantic tool is expected to deliver answers according to logic relations.

Questions like provide at least three keywords describing radio communication between two antennas will allow submissions like:
 * "propagation characteristics, obstacles, weather," or
 * " interference, distance, frequency"
 * It will also provide links to features being omitted, e.g. an answer rain will provide an explanation of rain is part of a meteorological condition.

Further examples will be established in Activity 1, thus being more specific on the expected outcome of this activity.

Activity 7: Development of meaning evaluation
This activity is seen as an ultimate goal, which might be outside of the scope of the current development. It should provide the student with a feedback on the "meaning provided in a sentence". Though the correct analysis would require language processing, we foresee a reduced functionality based on keyword identification. As the development of this module is not clear yet, further details will be provided in the course of the project.

Activity 8: Documentation
Documentation is seen as part of and will happen in an electronic form, explaining both achievements and identified shortcomings of the approach.

= Reporting and Milestones =

Reporting
Reporting is through regular meetings once every 2nd week, and documented on UNIK's wiki. UNIK will provide the structure for this reporting, and maintain a list of action items addressing the further developments.

Milestones
The following milestones have been identified. Each milestone will represent a point of decision on continuation of the project
 * M0 at month T0: project start (envisaged start 1. September 2013)
 * M1 at month T2: Framework established, Activity 1 finished (100%), Activity 2 (40%): questions from lessons 1-5 converted, Activity 3 (100%), Activity 8 (10%) .The milestone is expected to be achieved after 2 months
 * M2 at month T3: Multiple choice-based evaluation established. Activity 1...3 (100%), Activity 4 completed 100%. Documentation 20% completed
 * M3 at month T4: Numbers-based evaluation established. Activity 1...4 (100%), Activity 5 completed 100%. Documentation 40% completed.
 * M4 at month T6: Keyword based self-evaluation established. Activity 1..5 (100%) Activity 6 completed 100%. Documentation 80% completed.
 * M5 at month T7: Adjustments identified through user feedback taken into consideration. Documentation (Activity 8) 100% completed. Updated proposal for Activity 7 and list of future actions. This milestone is added to commonly identify the way ahead.

= Contractual issues =

Partners
The envisaged project is a collaboration between UNIK and Skaperhuset. The main contact people from Skaperhuset are Monika Eknes for contractual and administrative issues and Arne Dybdahl for technical issues. UNIK is represented by Hellfrid Newman for administrative issues and Kaja Elisabeth Mosserud for student matters and Josef Noll for technical issues.

Delivery
The software shall be delivered as part of the UNIK Semantic MediaWiki infrastructure. UNIK is free to provide the software to the research community and further develop it on a non-commercial basis. Skaperhuset is entitled to use the software for their own developments.

Payments
UNIK will pay an amount of 1100 NOK/hour, but limited to 50 hours/months for the development. Each milestone represents a decision point, allowing for adjustments of the technical aspects of the contract.

Duration
The contract will have an initial duration of 2 months, and is subject to an automatic extension to reach the next milestone. Each partner can cancel the contract by a one month notice.

= Open Action Items =

=Meetings =