verso.co.nz

The purpose of evaluation

A key question for anyone managing or facilitating an online community is how to make it sustainable. Sustainable communities need to maintain (and grow) an active and engaged membership. Structured online events or activities can play a very important role in engaging community members and ensuring their regular and active involvement. However, these events must be effective – badly planned and/or facilitated events can turn members off and lead to the failure of the community.

The facilitator’s overall impression of whether an event is effective or not is useful. But without a more rigorous evaluation, we can miss underlying issues which have the potential to damage members’ ongoing engagement in the community. So some kind of evaluation process is vital to its ongoing success and sustainability.

This article focuses on evaluating community events: not to check that they meet a minimum standard, but as a way of engaging in a process of ‘continuous improvement’.

The evaluation process

Many educational and management approaches employ cyclical models involving reflection or evaluation – eg Kolb’s experiential learning cycle and the action research cycle. The PDSA Cycle shown here represents the stages of Deming’s approach to quality improvement in business, but can be adapted to provide a model useful in relation to online community events and activities: PLAN: plan the community event DO: facilitate the community event STUDY: evaluate the event ACT: feed the evaluation results back into further community development

Carrying out the evaluation

Most of the evaluation will take place as part of the STUDY phase. But during the event (the DO phase) the facilitator should keep notes on what’s going well and what isn’t. If it’s not too big a group, it can also be really helpful to keep notes on the level of engagement of each member – not as a form of assessment, but as data that may be useful later.

During the STUDY phase:

Decide how you will gather feedback

Without its members, the community does not exist. So feedback from participants in an event is an essential component of evaluation:

• Online tools such as surveymonkey or polldaddy are quick to complete, so community members are usually happy to complete a survey. But make the survey brief, and tell them how long it’ll take.
• Individual interviews: you may get much more meaningful information about the event’s effectiveness if you personally approach participants. This may be a phone call or skype, or using an asynchronous method such as email. Ideally, a neutral 3rd party will gather the data, since participants may be unwilling to open up to the facilitator. You may be able to set up a reciprocal arrangement with another facilitator to gather data from each other’s participants.
• Collaborative feedback: you could set up a wiki that participants can use to record their feedback. Or you could set up a forum or a synchronous discussion space that participants use to discuss the event. Ideally this would be anonymous, so you may appoint one member to gather the raw data and provide you with a summary. Again, you may be able to arrange with another facilitator to manage this and gather feedback.

Develop evaluation questions and tools

• Communicate clearly what you are evaluating – point the participants back to the activity if possible so they can reflect on it. Be specific – make sure the questions clearly identify aspects of the activity that you want feedback on.
• Focus on how well the activity met their needs, not just on how much they enjoyed the process. If possible, ask for feedback at the higher levels of Kirkpatrick’s model – eg has the activity had a positive result, has it made a difference, have they been able to apply what they learned during the activity?
• Focus separately on the design of the activity and its facilitation: they are distinct, and the effectiveness of each is essential.
• Include questions that are open and qualitative so you can find out why things happened the way they did.

Gather feedback from community members.

• Tell participants how you will make use of the feedback: if you focus on improvement and they believe you are sincerely interested in making things better next time, they are more likely to engage in the evaluation process.
• Gather feedback not just from those that took part in the event, but also from those who chose not to. Asking those who didn’t take part why they didn’t can tell you a lot about the design of the event and the way it was communicated to community members!

Reflect and evaluate
Feedback from participants is just one form of information on which to base the evaluation. Your own reflection is another essential component.

• You may want to reflect on the activity before you gather data from participants – that way your own thoughts won’t be overly influenced by feedback. But once you have gathered the data, that’s a chance to reflect on and learn from the feedback from participants.
• In your reflection, avoid placing blame on participants. Assume you can do better and learn from mistakes. If people go wrong, ask yourself how could I have communicated more clearly?
• Use your own recollections and notes from during the activity to triangulate – compare them with what participants have told you about the activity, what happened and why.
• Focus on improvement – even where you think the activity was effective, try to identify specific things you could do to make it better next time.
• Share your evaluation with the participants – it’s a community, right? Sharing the evaluation means it becomes part of the larger community collaboration and conversation, and can enhance member commitment. But… if there are comments or conclusions that you feel are private or could offend members, leave them out of the published version.

Image: A Midnight Modern Conversation by William Hogarth.

Bibliography

Davies, C (nd). Kolb Learning Cycle Tutorial. Downloaded 30 May 2010 from http://www.ldu.leeds.ac.uk/ldu/sddu_multimedia/kolb/static_version.php

Unknown author, Business Performance Ltd (nd). Why Measure Training Effectiveness? Downloaded 30 May 2010 from http://www.businessperform.com/workplace-training/evaluating_training_effectiven.html

Unknown author, Carpenter group (nd). The Deming Cycle. Downloaded 30 May 2010 from http://www.quality-improvement-matters.com/deming-cycle.html

Unknown author, Warwick University (2008). Action Research. Downloaded 30 May 2010 from http://www2.warwick.ac.uk/services/ldc/resource/evaluation/tools/action/

In my work with educators who are new to planning and facilitating online learning community events, I often see activities which disappoint participants because of simple problems with the technology. Some community members are positive and accepting, others are not so forgiving and lose interest if the facilitator doesn’t seem well prepared.

Make sure the technology will work

In my experience, most of the technical problems that new facilitators encounter are avoidable. Many can be avoided if you:

• Check and check again that the technology works before a synchronous event. First, before you publish the activity details so you know that what you have planned is feasible. And again just before the event – with enough time to repair or work around any fault that occurs.
• Before the event, tell participants clearly what they need to have and do to be ready. eg will they need a headset with microphone? And advise them to check beforehand it’s all working. Establish a fallback position – eg if their webcam doesn’t work, can they just use audio?
• Do a test drive to make sure that participants can access the event – but be aware that what you can see and do with a ‘teacher’ account is not always the same as what ‘student’ or guest account can. Log in using a dummy student or guest account and check that what you planned is possible.

Have a contingency plan

No matter how well you are prepared, technology can still cause problems, especially with synchronous activities. So you need to be prepared:

• Have a plan B on what you will do if the technology goes wrong. How will you facilitate the event if the chosen technology fails?
• Have a plan B for participants who can’t take part. If it’s an asynchronous forum they can access it later, but if it’s a synchronous activity you should create an archive for those who missed the synchronous event. eg in Wimba, click on the archive button as soon as the discussion gets underway. In a Skype chat, you can save the transcript as an HTML file and upload it to a web page.

Effective online facilitators:

• Avoid many technology problems by making sure it works beforehand
• Have a plan on how to continue when unavoidable problems do arise

Image source: Wikimedia Foundation

Learning and teaching online, especially when using learning community approaches, places an emphasis on writing. This may involve writing in discussion forums, wikis, blogs, or more specialised tools such as the Moodle database. I’ve written elsewhere about the importance of read-write activities for learning.

But writing online can be a nightmare for teachers and students who are familiar with the web as an information resource but are new to creating content online. There are a number of pitfalls: what is a relatively simple task when using desktop software can be very problematic when using an online editor such as those found in blogs, wikis or LMS software. But most problems can be avoided by using a few simple techniques:

1. Never paste from a Word document. You can end up with garbage text like that shown in the illustration, or other formatting problems that are not easy to fix. If your text is in Word, copy and paste it into a text editor first (such as Notepad in Windows or TextEdit in OSX), then copy and paste that into the online editor. All the formatting will be lost, but you’ll have a clean copy without problematic formatting.
2. Never press the ‘Submit’ button without a backup. While you were writing, the LMS may have logged you out, or your network connection or session may have been lost. You can spend a long time composing a thoughtful reflective response to a forum posting, or composing an informative web page, then lose your work when you submit. A simple strategy is to select and copy all the text before you click submit. That way, if your work is not submitted, you can paste into another editor and try again. And if what you have written is really worth keeping, consider saving it as a text file to your hard disk – just in case!
3. Don’t worry about formatting – at least at first. Most of the WYSIWYG editors built into LMS or CMS software are idiosyncratic at best, so get your thoughts onto the screen and leave putting in the bulleted lists etc to last. If you’re used to writing using desktop software, lower your expectations – your formatting options are much diminished using most online editors. And if you’re reluctant to give up a lot of text formatting, you’ll also find it helpful to know how to use a few formatting tags – HTML or wiki markup – for those occasions when the editor gets confused and you just need to make a few changes to the source code.

Teachers and learners can get really frustrated and turned off the use of LMS and other online software if they lose work or can’t format it they way they want. We need to predict that these sorts of problems will occur and inform teachers and learners how to avoid them beforehand.

How to find tweets in a specific language?

That’s an issue for many Twitter users, including language learners and native speakers of other languages. Because of the dominance of English on the web, it’s easy to find English tweets. But finding tweets in other languages is not so straightforward.

One solution could be to tag each tweet with a language code. Using IANA’s existing language codes seems an ideal solution for compatibility and ease of recognition. This coding system is used widely on the web and in its basic form uses a two-letter code for each language. For example, the code for English is en, the code for Maori is mi.

It would be possible to use a tag prefix symbol in front of each code so that we could search for tweets in that language. But we need to use a different tag prefix than is currently used for tweet topics. Ideally, we could use just a single non-alpha character: where # is used for topic tags, we could use something like the percent sign. So, a tweet in modern Greek might be tagged with %el.

It might also be useful to flag tweets written in a non-standard language character set. For example, because of the limitations of some Twitter clients, we might want an additional symbol to tag a tweet that it is in Greek but which is transliterated into an English character set. Eg %el!

Since there is a lack of documentation on which specific characters are distinguished by Twitter’s search function, any use of a new tag prefix to denote language will require some trial-and-error testing to ensure it works effectively. If the polyglot community of Twitter users could agree on such a coding system, it would make it much easier to find relevant posts in languages other than English.

Image: Brueghel’s Tower of Babel

In an earlier post I discussed how Apple’s software development efforts seem very focused on consumption of media.

I’m interested in learning which incorporates producing information (not just consuming it) and which makes effective use of Web 2.0 tools to publish, not just to read. Given the iPad currently appears to have pretty much the same features as an oversized iPod Touch, the software limitations are likely to parallel those of the iPod. These include:

• The only multitasking available seems to be that music can be played in the background while you use other apps. So moving content from one app to another is clumsy. Given the size of the iPod, this is not such a big deal. But if I purchased the much bigger iPad, I’d expect it to be more suitable for productive work such as editing web-based content.
• Many web-based systems use WYSIWYG editors for creating and editing content. These are not available using the current iPod OS, so editing is restricted to plain text – unless you can use markup. This affects all kinds of web-based systems used in education: Moodle, PBWorks, Blackboard, Mediawiki, etc. In a wiki you can use wiki markup to get around this, otherwise you’ll need to use HTML. Either way, this will be seen as a big step backward by many educators and learners!

There are many apps which allow the user to access content as consumer but few apps which allow authoring. One that I really like is the WordPress blogging app which allows me to create and edit posts and pages and manage comments. Like WordPress, it’s simple, straightforward and effective. But notice from the screenshot above (on an iPod Touch) that the editor shows only source code (HTML). Now I work in that mode most of the time anyway, but I know many of the teachers I work with would see the loss of the WYSIWYG editor as a return to the dark ages!

Since the iPad is not yet available, my comments are merely predictions based on the current technology. I hope I’m wrong, but I suspect the first iPads will not solve these problems. My advice to teachers: if you are using Web 2.0 tools or an LMS such as Moodle, you may not find a shiny new iPad is a suitable platform for creating and editing content. Unless of course you are prepared to learn some markup!