Evaluation of eLearning for Effective Practice

☺

On 8^th Sept. 2010, we had another teleconference session with Bronwyn. Bronwyn had invited Rachel Byars to talk about the evaluation plan she had conducted last year. Rachel Byars has accomplished a needs evaluation of the suitability of eLearning options in teaching hotel management. Unfortunately I finished work at 6:00 and it took more than 45 minutes to get home. By the time I arrived home and joined the session, Rachel had already talked about many aspects.

Although I had been doing an effectiveness evaluation, I realised that it would be interesting to see how she had approached, proposed, planned and accomplished her evaluation.

During the session, Rachel recommended a textbook to read and use for this purpose.

Again, Bronwyn kindly recorded the whole session and can be accessed at, well you should know where ....where is it Bronwyn?

☺☻

Louise and I met again at 4:00 pm at the LSC to discuss our assignment. It seemed that Louise had completed many aspects of her assignment and just needed to put certain paragraphs and items under the right headings and titles. She later realized that she needed to edit her work again and cut and paste as appropriate. She also thought that she needed to include some visual demonstrations in her work.  I needed to do some administrative work from 4:30 until 6:00 pm as we do not have an administrator after hours. We had a lovely administrator whose working visa had expired and he just could not stay any longer. We had advertised for the position and I am afraid that it will take some time to get a new after hours administrator. The centre was very quiet and we had some useful discussions in terms of the marking criteria and the structure of our assignment. We also managed to retrieve a sampler and go through some contents. The good luck did not last long and things did not go well when a student suddenly turned up and asked to make two appointments to see an  advisor.  Another student turned up and waited for the first one to finish. There were just me, Louise and another advisor. The first student asked many unnecessary questions about certain things and it took me a long time to sort things out and still could not manage to organise for a day and time for an  advisor to see her. That is because firstly, I was not quite familiar with the administrative work I was doing (which was not my job anyway) and secondly I knew that things would take a long time and I just wanted to do my assignment with Louise.  Poor Louise was waiting for me to finish. The second student left already and it seemed that the first student seemed not be paying any attention to that. Finally I was able to make an appointment for her but it seemed that she was very demanding and asked for another study group appointment. I thought it would be better to ask another learning advisor to make the second appointment for her. I asked an advisor to do that and the advisor kindly agreed but it was too late and Louise thought she needed to go home. It was 6:00 already and the centre closes at 6:00 anyway. Louise said that she would come back tomorrow.

On 27th August 2010, Louise and I met again at the LSC where I am based. We discussed the requirements of our assignment once again and started to put ideas together. I realized that I could not start as I needed to publish my eMIT lessons first. I had received a number of positive feedback from students who were already enrolled in my online lessons.

Louise started working on her part and I believe that she has put some useful thoughts together brilliantly. I asked her to explain certain aspects to me as I was really interested in her project from the very first moment she decided to develop it.

I noticed that there was a middle-aged student using the services at the LSC and was using the computer next to the computer Louise was using and she was repeatedly asking Louise unnecessary questions. It seemed that Louise was really annoyed and just wanted to concentrate on her assignment. Louise is very polite and I knew that she did not want to tell her that she was busy and was not interested in having a chat with her. It was 3:00 pm and I was expecting a group of students and indeed they arrived a little bit early and I needed to leave Louise.

I did not want to tell the student that we were meant to be focusing on our assignment. It seemed the student did not know what we were doing.

Before I left Louise, I apologized and asked Louise to carry on and told her that I was coming back soon.

My session with the students was for an hour and a half. After I came back Louise was still answering the students' concerns. It seemed that Louise did not want to embarrass the student and did not want to tell her off. Louise is a very nice person and I respect her a lot.

I offered Louise a cup of Coffee and it seemed that she needed one. Later I asked the student if she wanted one too. She thought that I was going to charge her for that and still wanted one. She asked me how much I wanted for the coffee. I told her that it was on the house. Louise realized that it was time she had left and suggested to meet up again next week.

Louise and I had arranged to meet at the Technology Learning Centre at L4 J block at MIT to join the teleconference organized by Bronwyn on Wed., 25th August 2010.

We arrived a bit early and Oriel gave us proper directions on how to set up the necessary tools for the teleconference. Oriel also gave us some useful tips in connection with certain e-learning applications. As a matter of fact we were too early and we waited for more than an hour for the others to join the conference on time. Therefore, we had plenty of time to discuss our assignment. We knew very little about the requirements of the assignment. We retrieved the instructions and realized that we needed further guidance and instructions from Bronwyn. Time was passing very quickly. We did not realize that our microphones were on and people could listen to our conversation. Indeed, Bronwyn had joined and tried to talk to us but we had not put the headphones on and we could not hear her.

It was 6:30 already and Veronique joined the teleconference. She was the first to join after Bronwyn left the session temporarily. Later Jon, Liz, Katie, Kevin, Alex joined the session.

Bronwyn started to archive the session and kindly asked us to type our concerns and questions as regards to the assignment. I was not quite certain about what to ask. Louise and I had agreed on the type of evaluation we were going to do, effectiveness evaluation. The students asked a number of useful questions. Bronwyn wanted to know more about what we were doing. Louise talked briefly about her previous project, preconception development, an eMIT course she had designed last year for her BN1 students. I mentioned that I was going to evaluate my “special topic by negotiation” for my graduate certificate in e-learning. My topic is about a few online lessons about cellular basis of life utilizing the blackboard on eMIT. The materials included in these online lessons examine a general overview of a “typical living cell” in terms of structure and function. Students will have a better understanding of the various parts of the cell and how the various organelles interact with each other to maintain best possible living circumstances of the cell and to provide optimal requirements of the various activities the cell performs. 

Later I mentioned that I wanted my students to remember the cellular basis of life permanently and Bronwyn thought I was joking. As a matter of fact I was not. I tried to explain that cellular structures and related topics can be really confusing and can be really hard to comprehend especially by those BN students who have no or limited science background. I also mentioned that some BN students had dropped out already. I thought that developing an approach to these lessons to enhance students learning would largely assist them to develop confidence and keep being positive. The principal idea of the lessons is to do a step by step activity I named it “the smiley face approach”.

Later Bronwyn asked us again to type our main question on the screen specified for this purpose. I asked how well students will be able to remember the online lessons. Bronwyn had certain concerns and asked me to elaborate on my question.

Later Louise elaborated more about my concerns (as she had seen my eMIT website and she thought that it was great) while I was trying to display some screenshots of my lessons.

I retrieved an image and the smiley face I retrieved seemed really ugly, but it was not my face.

Louise and I have developed some ideas and worked out a number of strategies as regards to the "Essay Writing" module we had chosen. We have included a number of significant screenshots to demonstrate the sequential nature of the essay writing process. Our responses are found following the screenshots.

How useful is the module for your work situation? Explain why it is or isn't, and how you would use it?

The module is very useful for essay writing for beginning students in the Bachelor of Nursing programme. Although some students are school leavers, many mature students have not written a formal essay before or not for many years. Within this group of students are a wide range of abilities and learning styles which the OIL module addresses effectively. The helpful guide takes the writer through all steps of the process from the instruction words that direct the writer to what is required of them to the final summary and conclusion of the process.

Each activity required in the process is identified for example, instruction, content and relationship words directing the writer as to how to identify the ‘how what and why’ of writing the essay.
As we teach and nurture students in essay writing at the beginning of the Bachelor of Nursing programme, the OIL module is extremely helpful and completely user friendly. The module reinforces the steps in the essay writing process and gives quite clear descriptions and prompts when the responder has identified the correct response to questions e.g. “Identify the instruction words in the topic. Click on them and if you are correct they will highlight”. The word then changes colour to identify a correct answer. There is immediate re-enforcement of correct learning.

Comment on the look and feel of the module and how 'fit for purpose', or not it is.

The module is entirely user friendly – colourful, well signposted, each signpost has navigational Properties to expound further the instruction that it describes. It fits the purpose extremely well of introducing the essay writing process to a new writer clearly.

What sort of changes would you make if customising the module?

The module comprises quite extensive reading and students notoriously don’t want to read and review lengthy instructions and materials. A suggestion would be to shorten the amount of information included and reduce the number of pages required to be read.

Why is evaluation important and how do you define it?

Evaluation requires deciding on the purpose of the evaluation and can involve the following questions and responding appropriately:

Activities in the evaluation process require one to evaluate the design of the module which must include:
1. clear instructions as to what to do and how to navigate and use the course tools and materials e.g. a site map and online support.
2. meaningful and engaging activities that will require interaction such as games.
3. minimise the amount of text information that the student must work through to achieve their learning goals.

The design in this instance requires the learner to work alone and therefore must be explicit to avoid poor understanding and an inability to achieve learning goal. The essay writing OIL module meets all of these requirements.
Which two E-learning Guidelines suit your context?

We have chosen the learner centred and teaching relationships.

We do believe that constructive learning relationships with learners is an important component of effective learner centred learning. I remember from my classes that teachers who constructed a positive learning relationship in a welcoming learning environment induced positive learning and a desire to learn more about the topic. Conversely, the opposite is applied when the teacher who was not learner friendly. I can simply formulate the following:
You love the teacher; you love their topic/subject.

What I mean by “love and positive learning relationship in a welcoming learning environment” is the following:1. Teachers accepting students for who they are

2. Teachers making lessons fun

3. Teachers having a sense of humor and friendly attitudes

4. Teachers having a thorough grounding in their topic

5. Teachers being able to teach effectively rather than talking needlessly

6. Teachers understanding learner needs and requirements

7. Teachers encouraging students to answer questions

8. Teachers being humble and honest

9. Teachers being able to guide and advise

10. Teachers being able to communicate with learners effectively

11. Teachers being able to include all students in meaningful learning activities

12. Teachers being learning counselors

13. Teachers being able to enhance equality and eliminate discrimination in the class

14. Teachers being able to enhance confidence

15. Teachers being able to play a role model

16. Teachers being able to be there for students

I had a friend and he truly believed that "Good teachers are God's greatest messengers".

Tell us how the “evaluate this” exercise went from your perspective?

We believe that the exercise (particularly from the taster) gave us insight into the evaluation process and into the depth of thought required involved.
It was interesting to identify the questions to be asked and to recognize the difference between “formative and summative” evaluation processes.

Louise and I met at 3:10 and I was really glad she made it. I needed to do the taster activity today. We worked through a Taster and have reached a number of solutions as regards the key issues we identified earlier this morning. I was also able to complete the other event booked on my calendar.

Career Development

A previously face to face course has been re-developed to cater for off-campus, distance students, working independently and completing assessments (e.g., preparation of a Curriculum Vitae (CV)). One of concerns noted during development of the online content was that the students might scroll too quickly through one of the important resources, the virtual business tour, so they can get their CV done quickly. This means they might not engage fully in the real learning aspects of the programme, and end up with an incomplete or poor quality CV. The educational developer is worried about changing a programme which used to rely on a face to face approach with students being stepped through the programme by a Tutor in class. There will be tutors facilitating the online course, but they are new to online teaching. The developer is not sure how the students will engage with the tutors, each other and with the online materials. The tutors are worried that they won’t know when the students are struggling. The developer is not sure how to evaluate the changes to the design of the materials and how to find out the level of support required for the students.

Questions:

1.      What are the key issues?

2.      How can the problem be solved? What strategies can be used?

3.      How can the evaluation process assist in finding a solution? What type of evaluation is appropriate in this situation?

Key Issues:

1.      A concern is that students might scroll too quickly

2.      Redeveloped course has not been effectively designed to take in an e-learning context

3.      The course design may not have sufficiently engaging activities to enthuse the learners to navigate appropriately through the necessary steps for learning to take place

4.      New tutors may not be competently familiar with on-line teaching tools and skills and therefore are unable to assist the student meaningfully

5.      The tutors may not be able to facilitate an online community

6.       The most important issue is the lack of knowledge and skill in how to evaluate an on-line course, its design of materials and how to investigate the level of support required for the students

Solutions to the problems:

1.       The new design must include:           

(a)    clear instructions as to what to do and how to navigate and use the course tools and materials e.g. a site map and online support

(b)   meaningful  and engaging activities that will require interaction such and games

(c)    minimise the amount of text information that the student must work through to achieve their learning goals

2.       Engage highly skilled tutors in e-learning to be involved in the delivery and support of the redeveloped course

3.       Regular interactions and feedback by the highly skilled tutors during the delivery of the course

4.       The availability of more than one tutor so that the student can have access to help quickly

5.       The developer requires to do a course in evaluation of e-learning and ask specific questions such as:

How can the evaluation process assist in finding a solution? What type of evaluation is appropriate in this situation?

The developer needs to know more about:

I believe that important forms of evaluation should include both formative and summative. In the formative category one needs to explore how useable the site is to the students and to the tutor.  As Well it is important to find out the most relevant information for the students and what that should include.  This will cut down on verbosity and assist in both the useability and the relevance of the site.   The developer also needs to know how easy the site is for students and tutors to access for otherwise both groups will not use it.  

Summative evaluation will be used to establish effectiveness once the redevelopment has been concluded.  The effectiveness of the site will establish that the intention of the developers, that is the learning outcomes have been met.    Summative evaluation will also address how readily the site is maintained and is a measure of the efficiency and the effectiveness of the course in total.   That is

If students can access it and navigate it successfully, find it user friendly and meeting their learning goals the developer knows that the site is well positioned. 

It is important also for the students and the tutors to develop skills and a toolkit for using e-learning sites.  The impact of the site, the learning incurred, the tools and skills developed will all be useful to transfer to new elearning situations.  

Yesterday Louise and I arranged for a meeting to discuss the requirements for the taster and case scenario.  We retrieved the last email from Bronwyn, and she realized that she needed to join the Google Group and apply for subscription. She was able to join the group and then she applied for subscription and received a message telling her that she needs approval from the administrator. Later she created a blog utilizing blogger and named it lrummel and can be accessed at www.lrummel.blogspot.com .  We also accessed wikieducator and went through the assessment criteria and the case scenarios. We understood that we needed to choose a case study and required to find the key issues and discuss the possible solutions. I kindly asked Louise to read through the materials provided by Bronwyn and I in turn view and study them and suggested that we should meet again. We decided to identify key issues and reach solutions individually and discuss them thoroughly and reach final solutions collectively. We arranged for another meeting at the LSC at MIt after 3: 00 pm.  I needed to leave work early at 3:30 as I had another course to attend at 4: 00. After I finished my course, I went home and when I retrieved my calendar, I realized that I had been already booked for another event at 3:00. I believe I need to contact Louise (or the student) to reschedule for another date and time. I really want to go through the case studies with Louise and I don’t want to postpone this important meeting with Louise as I realize that she is a very busy person and she has lots of things to do.