Distance education and e-learning

Student-Curated Video Collection: An Activity

AEG/Telefunken television from 1937. This was newfangled back when I started screening videos for this course. Eckhard Etzold, CC BY-SA 2.0

AEG/Telefunken television from 1937. This was newfangled back when I started screening videos for this course. Eckhard Etzold, CC BY-SA 2.0

I’ve been working on revisions to a distance-education physical geology course, and attempting to make it more interactive by offering videos. Have you ever tried to source relevant and accurate videos for multiple topics across multiple course modules? It involves going through hours and hours of videos, and rarely finding one that is directly on point or without problematic inaccuracies. My search technique has evolved to skipping anything longer than 5 minutes that doesn’t come with a transcript or clear description, and then screening the video at 1.5x speed.

So what to do about getting reliable videos without spending most of your adult life in the attempt… well, one school of thought would say let the students do it. I experimented with this kind of activity a few years back, but didn’t have an opportunity to deploy it full-scale. Here are the instructions I provided, with annotations. If you try it, let me know how it goes!

Curating Videos for Historical Geology

In this assignment you will assemble a collection of videos and complementary resources for historical geology students. You will work from the TED Ed* Lessons Worth Sharing video collection, Awesome Nature. This collection can be found at http://ed.ted.com/series/awesome-nature.

*I chose TED Ed because the videos are short. The student who did this moved on to TED Talks, which are  longer. I’d advise limiting the length of videos if you don’t want to spend hours watching videos in order to grade the results. If I were doing this today, I’d also recommend the fabulous video collection at MinuteEarth.

Your work will form the basis of a collection of resources to be made available to future students in Geology 109. If you wish, you will be acknowledged as the curator of the resources when they are posted, although I reserve the right to make any modifications that might be necessary to optimize the effectiveness of the collection.

Rationale

In the Independent Studies version of Geology 109, students do not have access to video lectures. Sometimes the textbook is unclear or written in too technical a fashion for students new to the topic to immediately understand what is being said. Videos designed by someone with a different perspective on the topic can be very helpful for reinforcing concepts, or clarifying points of confusion.

The problem is that not all videos are created equal. Some have factual errors, or even seek to mislead viewers. Some could benefit from clarifications. The task of looking for and vetting videos requires an understanding of the objectives a video should satisfy, and an assessment of how well the video accomplishes those goals. It also requires that viewers understand why they are watching the video and what they should get out of it. When an instructor looks for videos, he or she has an idea of what students find difficult, but it is really the students themselves who can most accurately identify where they need help, and what helps the most.

Your task

  1. Identify a video that satisfies one or more of the learning objectives for Geology 109. Provide the name of the video, and the link.
  2. Write an overview of the video. This should not simply restate the title of the video, but should summarize its contents in three or four sentences.
  3. List the learning objectives from the Geology 109 Course Guide that the video covers, and indicate which chapter they are from.
  4. Identify three key questions that the video answers. The questions should not be a restatement of the learning objectives, and should make it clear to other students why they would find the video useful. The questions will take the following form:
    1. Have you ever wondered …?
    2. Would you like to know how [something works or happens/ happened]?
    3. Have you ever been confused by …?
  5. Identify five terms that are technical in nature, and that are key to understanding the topic of the video. Define those terms in simple language, using your own words.
  6. Identify three “loose ends,” and explain the loose ends so that others watching the video will not be confused by them. The “loose ends” could be:
    1. Points that could be expanded upon
    2. Points that might leave some confusion in the minds of students watching the video
    3. Factual errors (hopefully there won’t be any of those)
    4. Points that are inconsistent with something in the course materials (e.g., competing hypotheses, more recent information, etc.)
  7. Write ten multiple choice questions so students can test their knowledge after watching the video. Supply the correct answers. The questions should cover key points. A good set of multiple choice questions will have the following characteristics:
    1. Four answer options (a through d)
    2. Little to no use of answer options like “all of the above” or “none of the above.”
    3. It should not be obvious to someone with no prior knowledge of the topic which is the correct answer. (Over-simplified questions are not helpful when trying to understand a topic.)
    4. Questions should be relevant to the topic of the video and to the learning objectives.
    5. After doing the questions, it should be clear to students what key points they have not understood.

Deliverables

You will write up each video following the layout supplied at the end of this document. This layout is designed to be compatible with the Blackboard system. The specific software you use to create the write-up is not important, nor is the font. (Blackboard has some formatting limitations, and formatting must be done within the Blackboard text editor, so this is something I will have to do afterward.)

Grading

Each write-up is worth up to 10 points. Those points will be calculated as follows:

  • Is the video relevant to Geology 109, and is the relevance clearly explained? (2.5 points)
  • Are all of the elements in points 1 through 7 above provided (e.g., the learning objectives, multiple choice questions, etc., are present)? (2.5 points)
  • Is the write-up scientifically accurate (e.g., definitions are correct, multiple choice answers are correct, etc.)? (5 points)

You may curate as many videos as you like*, however the maximum possible score for the assignment portion of the class will be 100%.

*This assignment was designed for a specific student. You may wish to rethink the “as many as you like” policy, or turn it into a group project to reduce the workload.

Format for submission

Square brackets mean text that you will insert. Text in italics are my notes, and don’t need to be included in your write-up.

[Video title]

[url]

 

Summary

[Three to four sentence summary of the video topic]

 

Why watch this video?

  • Have you ever wondered […]?
  • Would you like to know how [something works or happens/ happened]?
  • Have you ever been confused by […]?

 

This video addresses the following learning objectives for Geology 109:

  • [Learning objective], Chapter [chapter number]
  • [Learning objective], Chapter [chapter number]
  • [as many additional points as necessary]

 

Some key terms used in this video are:

[term 1]: [definition]

[term 2]: [definition]

[term 3]: [definition]

[term 4]: [definition]

[term 5]: [definition]

 

Special notes

  • [Loose end 1, explanation]
  • [Loose end 2, explanation]
  • [Loose end 3, explanation]

 

Note: these could take the form of, “In the video, [topic] is mentioned, but [concept] isn’t explained. Here is what it means,” or “The video says [this] about [topic], but in the textbook it says [that].   The difference is [reason].”

 

Self-test

[Questions 1 through 10]

 

[Solutions (e.g., 1a, 2b, 3d, …)]

 

Deadline

All write-ups must be submitted on or before Monday, March 30th 2015.

 

Categories: Assessment, Distance education and e-learning, Learning strategies, Learning technologies, Teaching strategies, Uncategorized | Tags: , , , , | Leave a comment

On Leaving the Circus

Circus_poster,_1890

Circus poster, 1890. (Library of Congress)

My last day of work at Athabasca University was Friday, June 3. That weekend was the most relaxing I’d had in… ever. Even finishing my PhD didn’t come close to the sensation of lightness. For the two weeks prior, it was almost impossible to concentrate because I felt like a kid about to start summer holidays.

If I had stuck around until September, it would have been 8 years. Unlike most schools, Athabasca University does not break up its year into terms or quarters. It has a continuous enrollment model, where new students start the course every month. Students can submit their assignments at any time during their contract period. The result is that there is no way to predict workload. There is also no way to predict income, because it depends partly on monthly enrollment, as well as the number of papers graded. For me this meant that my life was in a perpetual holding pattern to accommodate the irregular income and schedule. Eight years is a long time to stay in an unproductive holding pattern.

The message I got upon being hired by Athabasca was that my position was intended to be a small auxiliary source of income, kind of like babysitting is for a teenager. Nevertheless, I would be expected to offer the highest possible standard of customer service. If I could coin a phrase, it would be “work like you’re full-time.”

But I managed this. I also had the opportunity to revise my courses, which made me feel like I was actually in a teaching position. However, Athabasca’s financial concerns soon came to the forefront. There was the email requesting that employees take unpaid vacation to reduce the burden on payroll. AU entertained the notion of laying off all tutors for a day to save money. Large-scale layoffs happened. There was the move to digital textbooks, where savings realized from the lower cost of digital textbooks were not passed on to students as a decrease in fees.

And then there was the call centre. This news came in the form of a sternly worded email from the acting president that a) tutor costs were unsustainable, and b) this problem would be solved by getting rid of the system of tutors and replacing it with a “one stop shop” for all inquiries.

The claim was that students contacted their (unhelpful, unprofessional) tutors infrequently, and when they did it was mostly with administrative questions that tutors were not equipped to handle. Therefore, there should be a call centre that students would contact first. Knowledgeable and professional call centre operators (in sharp contrast to tutors) would then connect the student with appropriate resources. If it were deemed necessary, a highly qualified Academic Expert (former tutor) would be contacted and informed that the student had a question. The Academic Expert would then contact the student within 2 business days.

They argued that the centre could be open for longer hours, and on weekends, whereas there were limited office hours during which students could contact their tutors. First of all, these office hours were limited because AU was not willing to hire tutors full-time- so it’s hardly fair to blame the tutors for that. But second, and far more importantly, who uses the phone anymore? If I got more than 30 phone calls in the time I worked there, I’d be surprised. But I did get emails at all hours of the day and night, 7 days a week. The vast majority of those questions were about the course material.

The call centre would save money because the Academic Experts would be paid only for specific activities, rather than the “block pay” determined by the number of students assigned to a tutor. Getting paid would require filling out time sheets to document those activities. You can see a list of what counts here, in the appropriately named Outsider newsletter of CUPE 3911 (the tutors’ union).

And that’s were my self-respect threshold came into view. The notion of having to subdivide my job into tiny bits and pieces, and keep meticulous track of them in order to get paid, seemed incredibly burdensome. That’s not why I teach. Add to that rumblings about Academic Experts having their time sheets rejected, and the suggestion that I could expect a 40% decrease in my income, and it just didn’t seem worth it anymore.

My initial plan was to wait until my courses were moved to the call centre. I would see how things went, and then resign if the situation got as bad as I thought it would. But I got tired of waiting for the axe to fall. I got tired of there being an axe. I started to feel like a chump for staying there.

I might have been able to tolerate the problems if I had felt valued, but I didn’t. Even before the call centre, I had the sense that AU felt its tutors needed to be scolded into doing a good job. On an employee pulse survey, someone commented that tutors as a whole lacked professional development. This is in spite of the fact that no-one had bothered to ask what kind of professional development tutors had done. There also seemed to be a pro call-centre PR strategy to denigrate the abilities and work of tutors, as a means of emphasizing that AU was making the tough choices and seeking solutions.

So I felt about as valued as a piece of chewing gum stuck to the bottom of a shoe. I didn’t realize how deeply that feeling went until I received a 5-year service pin in the mail. I was surprised and confused because I honestly didn’t think tutors counted as employees for purposes of service recognition.

I’ve never quit a job before. I expected that quitting this one would happen when I was angry and bitter, but instead I was completely blissed out. If this post sounds like an angry rant, it isn’t. It is more of an exorcise- an exercise in exorcising those demons so I can leave them behind and begin whatever healing is required. Being chewing gum is hard on a person’s psyche.

My resignation letter was one sentence saying only what date my resignation would be effective. I didn’t say why I was quitting, and no-one asked. I was a little surprised that they didn’t ask for a hand-off overview of how the courses were going. They should have. Maybe I could have offered that information, but there would have been a lot to say. I’ve tried to communicate issues and solutions before, only to be disregarded, and I didn’t have it in me to try again. And anyway- not my monkeys, not my circus.

I don’t have a new job lined up, per se, but I do have a project that I’ve been meaning to start. I will get to use my research skills, and learn new things. I will have an opportunity to progress rather than being trapped in a holding pattern. I won’t have to read messages from administrators about how I’m not worth what they’re paying me. I won’t have to be afraid of decisions others are making about my future. I won’t need the approval of people who are less qualified than I am to make decisions. I’m not leaving higher ed just yet, but I am branching out and trying to make my own opportunities. What comes after remains to be seen.

Categories: Challenges, Distance education and e-learning, Uncategorized | Tags: , , , , | 12 Comments

Sadly, This Was Necessary: Meet Dr. Panchuk

In a course announcement today:

Hi everyone,

I would like to bring a change of procedure to your attention. In the past students have addressed me by my first name, but from now on I will follow standard procedure, which is for you to address me by my academic title, Dr. Panchuk.

I have never had to make this requirement in the past, however I’ve noticed that an increasing number of students are displaying disrespectful and argumentative behaviour. The vast majority do not behave this way, but I’m hoping it will serve as a reminder in those few cases of what is appropriate conduct in an academic context.

You should not take this to mean that you aren’t permitted to raise questions about your work. These are always welcome, and I’m happy to assist you. What it does mean is that you must use an appropriate tone. This is no different from what should happen in any of your other courses.

On a related matter, be aware that the appropriate level of discourse is what would occur in a business setting. What this means is that the shortcuts you might use when sending casual text messages to a friend are not appropriate. Writing “hey cn u hlp me w/ this qustn” won’t do, and for those of you unfamiliar, is not a good approach in general if you’re trying to make a good impression. Some people who receive such a message will take it as a sign of disrespect on your part.

I imagine some of you will be horrified to receive this notice, because this isn’t how you behave or communicate, and I’m sorry to have to send it out. Nevertheless, here we are. If it’s any consolation, appropriate behaviour and communication are noticed, and do set you apart. You can read the comments section of any news story to see what I mean.

I will be sending a second announcement outlining the appropriate way to submit your assignments.

Regards,

Dr. Panchuk

Categories: Challenges, Classroom management, Distance education and e-learning, Teaching strategies | Tags: , , | 1 Comment

Online Courses and The Problem That No-One Is Talking About

There are two kinds of online courses: those which are taught, and those which are facilitated. The distinction does not apply to the task of interacting with students. I’ve been both “teacher” and “facilitator,” and it’s exactly the same job from that perspective. The difference is one of autonomy, and it is a big difference.

The Gwenna Moss Centre is about to run another offering of their Introduction to Teaching Online course. Although I am a co-facilitator for this course, I would describe it as a course which is taught rather than facilitated. My co-co-facilitator and I discuss the course as it is running, and make adjustments on the fly when necessary. We take note of what worked and what didn’t, look at participants’ evaluations, and then modify the course as necessary for the next offering. Not only do we have the autonomy to make the necessary changes, it is expected that we will.

In Intro to Teaching Online, we assume that the participants will also be able to teach their online courses- that they will make pedagogical and logistical choices to respond to their students’ needs, and to make the course run as smoothly as possible. Also, that they will have the ability to revise as necessary and try new things. That’s how you teach an online course.

When you facilitate an online course, while you might take on the task of assisting students and grading their papers, what you can do beyond that is tightly restricted by a delivery model over which you have very little control. How little control will vary, but most likely it will be difficult or impossible to make substantive changes to what is taught, or how it is taught. Even if you designed the course in the first place, that “you” and facilitator you are completely different people as far as control over the course goes, and designer you lost any input as soon as the design contract was up.

If you are lucky enough to be able to request changes, the process is rather like having completed a painting, then being told you aren’t allowed to touch it anymore. If you want something to change, you must fill out a form describing in detail where the paint should go and how to move the brush. Someone more qualified than you will make the change. They might send a note back to you saying that they plan to improve your painting of a cow by adding spots. You must then explain at length that it is in fact a dog, and should not have spots. When the painting is finally modified, the dog is the wrong shade of brown. You decide it is best to not request modifications to your paintings in future.

Why does this matter? I don’t care how good you are- you never get a course exactly right the first time. If there aren’t any outright problems, then it soon becomes apparent where improvements can be made. Facilitator you gets to see the problems or areas for improvement, but must be content with grading papers and answering questions. If facilitator you is like facilitator me, this will drive you nuts. If facilitator you is subject to the same kinds of course evaluations as someone who can teach their course, and make it the best it can be, then this is not only unfair, but professionally dangerous.

While course quality is affected by this- especially if no-one sees a need to consult with facilitator you about how the course is going, or there are no mechanisms for facilitator you to communicate issues and be taken seriously- there is a bigger problem: the very integrity of the course.

At one time distance education was mostly intended to serve those who could not go to a brick-and-mortar institution for one reason or another. Maybe they had a family or a full-time job and couldn’t leave to go to school. Maybe they just couldn’t afford to move. Now things are different. While I don’t have any hard numbers, from what I can tell, at least 70% of my students are already taking classes at a brick-and-mortar school. They take an online class because they can fit it into their schedule better than one on campus, or it isn’t offered on campus at a time they need it, or they’re trying to get ahead/ complete their degrees over the summer.

What this means for the big picture is that students are far more likely to communicate with each other about the course than in the past. It might be two students who take the course together, or it could be someone who took it previously sharing information with someone currently enrolled. In the case that is causing me problems right now, a substantial number of students from one department at one school take the online course to fill a requirement. This is a facilitated course, so perhaps you can guess where this is going.

The students talk to each other. Some of it might be innocent enough, but some of it involves passing on assignments that I’ve graded to the next group of students who take the course. The course has not been updated substantively in some time, so the same assignments and exams still apply.

The problem has become ridiculous of late, with students submitting near-perfect assignments, all exactly alike plus or minus a few careless errors, and within record time. They get things right that no-one ever gets right. Clearly they are working together, but they are also referring to older assignments. I know this for certain for a few reasons: First, the correct answer will frequently appear after incomplete or even nonsensical work. They submit solutions with the answer that would have resulted if a typo, long since removed, was still in the question. They also plagiarize my comments from old assignments, sometimes reproducing them verbatim.

This course has a must-pass stipulation on the final exam. Normally that would be some comfort, because students who haven’t learned anything on the assignments would fail the exams. I’ve seen students with 95%, 99%, and 100% on assignments unable to break 20% on the final. (The exam isn’t that hard.) But over the past few months it has become apparent that the content of the exam has been shared. If not an actual copy, then a very good description of what it contains is in circulation. Exam grades have gone up, and students are regularly answering questions correctly which were rarely answered correctly in the past.

Ideally, if so many students who know each other are taking the course, the assignments should change frequently. In our hyper-connected world, it is almost certain that this kind of communication between students will happen. I even know of a homework-sharing website that has some of the solutions posted. The problem is that in order to change this, someone has to keep on top of the course full-time, and have the autonomy to make the necessary changes. The main consideration should not be the logistics of altering course materials. There’s no excuse for that when the relevant materials are or can be delivered online, and everyone and their dog knows how to upload a file to an LMS.

Nevertheless, the issue is that facilitators cannot be empowered in this way without disrupting the underlying structure of course delivery. Even more problematic is a culture amongst those who do run things- those who are not subject-matter experts but who handle the day-to-day operations- which views facilitators as incompetent, and unable to handle this responsibility. Not long ago I was handed an in-house guide to designing distance education courses. It warned readers at the outset that most faculty would be uncooperative and not understand how a distance education course should run. I felt ill, the way you would feel if you overheard your co-workers complaining about how useless you were. As I recycle that book I will contemplate with irony the damage this attitude has caused to distance education, and wonder if maybe I should take a chance and start the dog-washing business I’ve been thinking about.

There are many reasons to disempower facilitators, not the least of which is the cost savings from having them as casual workers instead of full-time ones. So here’s where I’m going to get in trouble for this post (if I haven’t already): if your concern is the bottom line, what happens when the ease with which students can cheat in your course makes other schools, employers, professional certification organizations, etc., decide that credit for your course is no longer meaningful? Even if cheating is less of a risk, what if word gets around that the course is hopelessly outdated or has problems? You don’t get enrollment, that’s what. And the people who communicate this aren’t going to be disgruntled facilitators. I’m the least of your worries. You need to worry about the students themselves who joke openly about cheating, and how little can be done about it, or who are discovered to lack skills or to have learning that is outdated.

There is a fundamental disconnect between what schools view as the appropriate way to structure a distance education program, and what actually works on the ground, when you’re expecting learning to happen. One involves online teaching and the other does not. There is a cultural gulf between those who have the power to do something about it, and those who can only look on in frustration. There are a lot of dogs to wash, but with most of them you have to spell out B-A-T-H rather than say the word, or they run off. A waterproof apron is useful, but not foolproof. You’ll need lots of towels.

Categories: Assessment, Challenges, Distance education and e-learning, Learning technologies, The business of education | Tags: , , , , , , , , , | Leave a comment

Crowd-sourcing distance education (or, Why Athabasca University’s problems are just getting started)

mortar quoteLast week there was another missive from Athabasca University interim president, Peter MacKinnon. The post appears to be damage control after a Metro article by Jeremey Nolais, Fears arise that Alberta’s Athabasca University will be lost as tough budget looms.  The post says that while Athabasca is facing “financial challenges that are acute,” and “a decline in the rate of enrollment growth,” the rumors in the media that it will be merged with another school or shut down are untrue.

What I found interesting about the article were the comments. There are only 6 of them at this point, and three of the comments were complaints about insufficient interaction with tutors. They weren’t complaints about the call centre model, where students contact the call center and a call center employee determines whether an Academic Expert should be informed that the student needs assistance (contact ensuing within 2 business days). Instead they were complaints by students who had tutors but felt that they weren’t hearing from the tutors enough. As one student says, “… I did not pay to be completely ignored and paid to be TAUGHT.” [original emphasis]. There were complaints about the quality of education, and the blame for that laid at the feet of tutors as well.

Given the controversy surrounding the call center, and the seemingly obvious thing to comment about- that those who were unhappy with insufficient contact from their tutors could rest assured that they would soon have no contact with a tutor whatsoever- it is surprising that no comments of that nature were posted. After some experimentation, I determined that those points of view were being moderated out. The upshot is that readers will come away with the conclusion that what is wrong with Athabasca is its tutors, which is very convenient for the administration at present. However, there is also a very real risk of discouraging students who might otherwise register for courses that are still running under the tutor model. If someone at AU thought that was a risk worth taking… well, make of that what you will.

I’m not saying that all tutors do the job that students want them to- some tutors may not do the best job they can for any number of reasons, and some students may have bought into the misconception that they have a full-time teacher assigned to them.

But Athabasca’s problem isn’t tutors not doing what students want them to. Its problem is a structure designed in the days when distance education required sending students boxes of paper in the mail. This is a problem because the system that can most efficiently manage hard-copy course materials is one in which teachers cannot have the autonomy to alter their course materials at will to respond to students’ needs. Think of the nightmare that keeping track of document versions would be! There would be no control on the workflow (and therefore on costs) if instructors could alter materials whenever they found a better way to teach. In order for the school to function, teaching has to take a back seat to operations.

kids these days

Kids these days

On the surface, it would seem that Athabasca has moved past this, with an online learning management system, online exams, and digital textbooks. At its core, however, Athabasca is still structured so that it is necessary to inhibit its teachers in order for it to operate as efficiently as possible. The reason I think that Athabasca’s problems are just beginning is that the costly and harmful structure they are fighting to protect is rapidly becoming unnecessary for distance education.

Think of it this way- these days most universities run courses through online learning management systems. Using these systems, instructors can post documents, set up quizzes and exams, post video links, post videos and podcasts of lectures, host class discussions, and more. It is entirely doable with a very minimal outlay for me to broadcast my in-class lectures live online, and have students answer questions in real time through Poll Everywhere while watching that lecture. We could discuss their answers as a group, and I could adjust my lecture on the fly to address issues that they were having. Students could even submit questions through Twitter during the lecture.

With courses set up this way, no-one sends students a box of paper course-materials. Students download and print what they want, access the rest online, and purchase textbooks directly from vendors in the format they want. Students can take a course and engage fully with a community of learners and an instructor without being in the classroom, whether that course is designated as distance education or not.

In contrast, Athabasca is structurally incapable of empowering its front-line teaching staff to act in their students’ best interest. It has people to grade papers and answer questions, but it doesn’t afford those people the mantle of teachers, the salary so they can be committed to students full-time, or the autonomy to fix issues with courses as they arise. Consider this: I don’t have access to the course materials that I wrote.  If I want to fix a typo, there is a separate group of people who handle that sort of thing, and I have to make a request to get it done. I had to hunt around to find out who those people were. If I had the same control over my Athabasca courses as I do with some other courses I teach, I would just take the three seconds to fix the typo myself and not tie up IT people, and who knows who else. I would likely be updating the course regularly to improve it, which means that a separate expenditure on a Subject Matter Expert (who is also me) to revise the course every so many years would be unnecessary.

From a business perspective, it might have been safe at one time to compromise on teaching if you were the only game in town that could mail out those boxes of paper. But what happens when mailing out boxes of paper becomes irrelevant to serving students at a distance? What happens when the competition is no longer other distance education schools or programs- when it becomes hundreds or even thousands of individual creative, energetic, and innovative instructors at traditional brick-and-mortar schools who choose to build and manage their own online courses? What happens when the additional cost of running those courses is trivial, because the resources are already there as part of how on-site students are served? Well, what happens is that the competition is essentially crowd-sourced, and can do a better job with lower costs and happier teachers.

I don’t know what will become of Athabasca. As long as it offers programs that no-one else does, there will be a demand for its product, and perhaps it will begin to focus on that segment of the market instead of a broad swath of undergraduate courses. But if it does offer programs that no-one else does, that will have more to do with no-one else choosing to offer those programs, rather than being unable to do so in a cost-effective manner. Athabasca will not change the way it does business because it is firmly committed to the notion that as long as the school is run as a business, the rest will take care of itself.  The call centre model- where by design, the first person students talk to will never be the person teaching them- is evidence of that. There is an entrenched culture which holds front-line teachers in such low regard as to view answer databases and non-teaching call-center employees as a better alternative.  This exists because at some level, Athabasca views itself as an organization for delivering courses rather than for teaching students.

Categories: Distance education and e-learning, Learning technologies, The business of education | Tags: , , , , , , , , | Leave a comment

Plagiarism-proof assignments: The Up-Goer Five Challenge

up_goer_fiveOk, so there’s probably no such thing as a plagiarism-proof assignment, but I think I’ve got a reasonable approximation thereof.

It originated with my frustration with the perpetual struggle to have students in my distance education classes answer questions in their own words. My students are using their textbooks to answer questions, and many seem to feel that a textbook is the exception to the rule when it comes to plagiarism. Some simply don’t understand that they’re doing anything wrong. From experience, I can tell you that many people who are not my students also see it that way, and complaining about it is a great way to be branded as unreasonable. The problem, as I’ve documented before, is that students who copy from their textbook also tend to fail the class. After last term, I’ve decided that it’s in my best interest to consume alcohol before grading assignments. I’m not allowed to ignore plagiarism, but what I don’t see

Absent blissful ignorance, the only way to deal with plagiarism (without causing myself a variety of problems) is to change the assignments so that plagiarism isn’t possible. Now, if you’ve attempted to do this, you know it isn’t easy. A search online will give you tips like having students put themselves in the position of a person experiencing a historical event, and explaining their perspective on the matter. That’s something students (most likely) can’t copy from the internet. But suggestions like that are not especially helpful when the topic is how volcanoes work. (Although now that I think about it, “Imagine you are an olivine crystal in a magma chamber…”)

The solution came from my online source of comfort, xkcd. Randall Munroe, the creator of the webcomic, set himself the challenge of labeling a diagram of NASA’s Saturn 5 rocket (Up Goer Five) with only the 1000 most commonly used words in the English language. Soon after, members of the geoscience community took up the challenge of explaining their fields of research in the 1000 most commonly used words. Here are two examples from a blog post by hydrogeologist Anne Jefferson. Anne writes:

” So I decided to see if I could explain urban hydrology and why I study it using only the words in the list. Here’s what I came up with:

I study how water moves in cities and other places. Water is under the ground and on top of it, and when we build things we change where it can go and how fast it gets there. This can lead to problems like wet and broken roads and houses. Our roads, houses, and animals, can also add bad things to the water. My job is to figure out what we have done to the water and how to help make it better. I also help people learn how to care about water and land. This might seem like a sad job, because often the water is very bad and we are not going to make things perfect, but I like knowing that I’m helping make things better.

Science, teach, observe, measure, buildings, and any synonym for waste/feces were among the words I had to write my way around. If I hadn’t had access to “water”, I might have given up in despair.

But my challenge was nothing compared to that faced by Chris, as he explained paleomagnetism without the word magnet:

I study what rocks tell us about how the ground moves and changes over many, many (more than a hundred times a hundred times a hundred) years. I can do this because little bits hidden inside a rock can remember where they were when they formed, and can give us their memories if we ask them in the right way. From these memories we can tell how far and how fast the rocks have moved, and if they have been turned around, in the time since they were made. It is important to know the stories of the past that rocks tell, because it is only by understanding that story that we really understand the place where we live, how to find the things that we need to live there, and how it might change in the years to come. We also need to know these things so we can find the places where the ground can move or shake very fast, which can be very bad for us and our homes.”

Is that brilliant, or what?! To make it even better, Theo Sanderson developed a text editor to check whether only those words have been used. This is what happened when I typed part of the introduction to the chapter on volcanoes:

Up-Goer Five text editor

Yes, fortunately it has the word “rock.”

I decided to test-drive this with my class. I gave them the option of answering their assignment questions in this way. It’s difficult, so they got bonus points for doing it. A handful attempted it, and that was probably the most fun I’ve ever had grading assignments. If you’d like to give this kind of assignment a shot, there are a few things to keep in mind:

  • Students (and colleagues) may be skeptical. Explain that the exercise requires a solid knowledge of the subject matter (in contrast to paraphrasing the textbook) and is a very effective way for students to diagnose whether they know what they think they know. In my books, that gives it a high score in the learning per unit time category.
  • The text editor has some work-arounds, like putting single quotes around a word, or adding “Mr or “Mrs” in front of a word (e.g., Mr Magma). Head those off at the pass, or you’ll get “But you didn’t say we couldn’t!”
  • You may wish to allow certain words for the assignment or for specific questions, depending on your goals. For example, if I were less diabolical, I might consider allowing the use of “lava.” The other reason for not allowing “lava” is that I want to be sure they know what it means. In contrast, I probably wouldn’t make them struggle with “North America.”
  • Make it clear that simple language does not mean simple answers. I found that students tended to give imprecise answers that didn’t address important details. I don’t think they were trying to cut corners- they just didn’t think it was necessary. If I were to do this again I would give them a rubric with examples of what is and isn’t adequate.
  • Recommend that they write out the key points of their answers in normal language first, and in a separate document, and then attempt to translate it.
  • Suggest that they use analogies or comparisons if they are stuck. For example, Randall Munroe refers to hydrogen as “the kind of air that once burned a big sky bag.”
  • Make the assignment shorter than you might otherwise, and focus on key objectives. Doing an assignment this way is a lot of work, and time consuming.
  • And finally, (as with all assignments) try it yourself first.

In that spirit:

I like to make stories with numbers to learn what happens when things go into the air that make air hot. Very old rocks from deep under water say things that help make number stories. The number stories are not perfect but they still tell us important ideas about how our home works. Some day the number stories about how old air got hot might come true again, but maybe if people know the old number stories, they will stop hurting the air. If they don’t stop hurting the air, it will be sad for us because our home will change in bad ways.

Categories: Assessment, Challenges, Distance education and e-learning, Learning strategies, Learning technologies, Teaching strategies | Tags: , , , , , | Leave a comment

Customer relationship management (CRM) as a paradigm in distance education

I’ve been trying to gain more insight into the changes that are coming to Athabasca University with the new contact centre model that I discussed in an earlier post. After finding AU’s report, Evaluating the relative efficiencies and effectiveness of the contact centre and tutor models of learner support at Athabasca University, I think I have a better idea of what’s going on.  AU is adopting a customer relationship management system, similar to those that are used to run the call centres of large businesses.  These systems have been adapted for use in higher education over the past decade.

The report outlines AU’s system as follows:

Under the Contact Centre model, undergraduate student advisors, available six days a week, field initial queries via fax, telephone, or e-mail, and act as the first point of contact for accessing other advising services. Using flexible, shared, and secure contact databases, contact centre advisors handle issues for which they have established answers and refer course-related inquiries to the appropriate academic experts. … “Frequently-asked question” databases are also available to students and advisors to answer some academic queries. If applicable, students are referred to faculty and part-time academic experts for academic assistance.

The report refers to a keynote address by James Taylor of the University of Southern Queensland at the 2001 ICDE World Conference on Open Leaning and Distance Education. In the address, Taylor describes the USQ’s e-University project which seeks to automate the delivery of information to students. Taylor explains that tutors’ responses to students’ questions are added to a database. Subsequent students’ queries are first run through the database to see if the question has already been answered. If so, the student is provided with that information. At the time of Taylor’s keynote address, tutors were involved in vetting the automated responses, but Taylor anticipated that this would soon become unnecessary. It is only when the answer does not exist in the database that a tutor is required to interact with a student, and as he put it,

As the intelligent object databases become more comprehensive, enabling personalized, immediate responsiveness to an increasing number of student queries, the institutional variable costs for the provision of effective student support will tend towards zero.

By this he means that regardless of enrollment numbers, the costs will remain the same because students will access the database rather than needing attention from tutors. The irony is, the more dedication and care that tutors put into answering questions, the more they hasten their own obsolescence.

The AU report emphasizes this outcome:

Most importantly, individually-tailored services can be provided to an increasing number of learners with the same economic resources by using knowledge management software to reduce the need for direct, human interaction in the teaching and learning process.

The AU report takes issue with the idea that student-teacher interactions are necessary. They point out that, according to the equivalency theorem of Anderson (2003), only one of the following types of interaction is required: student-teacher, student-student, or student-content. As long as one of these is done well, the others can be eliminated entirely with no negative consequences for the student. I wonder at the wisdom of leaving a group of students to their own devices, sans teacher or content, as a method of education, but perhaps I’m missing some nuance of the scenario.

My question as I read through the report was, “Does this work?” The report was written to present the results of a survey of students who had taken part in the initial roll-out of the contact centre model at AU. The data presented are a list of attributes of the contact centre and tutor models, and students’ ratings of the importance of those attributes.  I’ve summarized the results below.  Keep in mind that although the report says the survey asked about the importance of each of these attributes, some of the attributes read as though students were asked to rate the outcome of their interaction with the model in question.  I’ve kept the original wording so you can decide for yourself what the survey items mean.

Study results

Even if it were clear whether the survey were evaluating the perceived importance of or the perceived satisfaction with various attributes, it still wouldn’t answer my questions.

One thing I would like to know is how students felt about having an advisor with no specific knowledge of the course material answering their academic (i.e., course-matter specific) questions by referring to a database. The survey reports (item 2) that 76% of the 300 students sampled rated “Importance of talking directly with an academic expert for academic support” as “Important” or “Very Important.” I wonder if that number would have been even higher if it had said “communicating directly” rather than “talking directly.” Very few of my students “talk” to me because the vast majority communicate by email.  (The prevalence of email could also have affected the outcome of item 3.) More importantly, this item doesn’t answer whether or not students were happy with the amount of direct communication they had with an academic expert under one model or the other.

The report does not address how beneficial students felt either model to be in terms of their learning outcomes, and it does not provide any metrics such as differences in grades or retention. The closest it comes is addressing satisfaction with response times for academic assistance using each model (item 8). Read literally, the results are students’ rating of how important satisfaction is in this regard (i.e., how important it is that response times be satisfactory), but it is possible that students were actually asked how satisfied they were with response times.  Regardless, response time is not the same thing as help with learning.

Because this report did not tell me what I wanted to know, I spent the better part of a day searching for studies of similar systems, and the outcomes for learners.  Much to my chagrin, the only relevant thing I found was a paper by Coates et al. (2005) stating that there were no generalizable studies addressing this issue. The paper was very interesting nonetheless, and foreshadowed the present developments:

While ‘academic-free’ teaching may seem only a very distant prospect, major online delivery ventures already have business plans based on the employment of a limited number of academic staff who create content with the support of larger numbers of less expensive student support staff.

It also echoed my main concern: “What are the consequences of students increasingly seeking learning assistance from technology support staff rather than from teachers?”

The AU report concludes that there was no material difference in students’ satisfaction with response times between the two models, and that “[m]eans of first contact seems [sic] to be more effective under the Contact Centre model.” (If the last statement is based on item 4, it would appear the opposite is true.) Because there was a savings of over $60 for each student with the Contact Centre model,

Taken together, these results suggest that satisfactory educational experiences can be delivered under either model. Given this equivalency of outcomes, it is recommended that relative costs should primarily determine how student support is provided at Athabasca University.

After reading this report, my thoughts were (almost simultaneously) that response times are a dubious measure of how satisfactory an educational experience is, and that this point is likely moot for decision-making purposes at this stage. But maybe distance education is like the garment industry: at one time, it was a foregone conclusion that your clothes would be made to fit you. Now, most people aren’t particularly troubled by having to pick a ready-made garment from a clothing rack. It’s still a shirt, right?  Why should getting answers from a database/ advisor instead of from a teacher be any different?

In case you are wondering, yes, I do find the idea of purging humans from teaching to be disturbing. Aside from losing the human interactions and creative challenges that make teaching a meaningful undertaking, there are serious flaws in a system where students’ interactions with a course cannot be observed by someone who will ultimately be responsible for redesigning that course. In my present roles as tutor and course author, I have a very good idea of what is working and what isn’t, because of conversations I’ve had with my students. Sure, there are issues that commonly arise, and one could infer from the number of questions on a particular topic that something isn’t working. But if it were that easy to fix, I would have intuited the problems with my approach to begin with and avoided the issue altogether. It is only by communicating back and forth with students and asking specific questions that I can figure out exactly what’s going on. There is a diagnostic element in my relationship with students, and it is a crucial element for assisting students on a one-to-one basis, and for improving the course in general. The contact centre model is about removing the human element as far as possible.  (Wow!  At one time that statement would have been entirely facetious!)  If I were to participate in this system, I would have exactly one try to figure out what was going on, before my canned response would be distributed to all future students with a similar question. This model will result in lost access to valuable data, and these are data that can’t be recovered by a standard end-of-course evaluation by students.

There are some broader issues that I didn’t see addressed in this report, or in any of the promotional materials I read about customer relationship management software use in higher education, or in reports by administrative branches of different schools about the benefits of implementing these systems. What about confidentiality, for example? How are queries dealt with that could identify a particular student? What happens if students share personal information in their question? Does all of this become available for other students to access?

What about intellectual property rights? If students attach a file containing their own work when they query the system (which they can do), could their work be kept as an example when the question and response are added to the database? Who would use their work, and how?

What happens if there is an error in a canned response? What if I say “increase” when I meant “decrease,” or “north” when I meant “south?” If the advisor who is deciding on which response the student should receive doesn’t have the background to know the difference, will the error ever be caught? Or will it rear its ugly head in perpetuity?

What happens if we learn something new that changes everything? Will this system trap the old understanding forever, like an insect in amber? That is perhaps the most chilling outcome of handing off the job of teaching to an entity unable to think critically about the information it dispenses.

 

See what the students think: Instructional Model Survey

Categories: Distance education and e-learning, The business of education | Tags: , , , , , , | 2 Comments

The self-respect threshold

It turns out that Microsoft Excel can be more efficient than soul searching. A few days ago I received a general email sent to Athabasca University staff by the interim president, Peter MacKinnon. It was a progress report on the new Student Success Centre model for changing how students access services (including tutors like me) at AU. This is not a call centre. Everyone will tell you that, and it is quite a touchy subject. It may be the place you have to call (or email) to be directed to the service you need, including your tutor, and there might be tracking numbers involved (I’m still sketchy on the details), and your call (or email) might be logged, and then referred to the appropriate person or department… but it is not a call centre because telephones will not be the primary communication technology. So there.

I have a few problems with this. First of all, calling it the Student Success Centre sounds like a cynical branding exercise, even if it isn’t. Second, as far as I can tell (again, details are sketchy), my students will no longer be able to contact me directly. They will contact the [not a call] centre where someone will decide if it is really me that they need to talk to, or if someone else will do. If it is determined that the student does, indeed, need to contact me, I will be notified by the system to contact that student. The system will follow these transactions and apparently generate some statistics so appropriate oversight can be exercised and I can be presented with numbers to motivate my performance.

So why is this change happening? Because it will cost less. Cutting costs is done with fanatical zeal these days, and like fanatical zeal, it often does not involve consideration of the big picture. Some years hence, people will look at our gutted institutions and say, “Oops. I guess we needed that after all.” It seems that those responsible for the financial upkeep of public institutions like universities have forgotten the reasons for creating those institutions in the first place.

I am getting to Excel and soul searching. As you may have guessed, this [not a call] centre will cost less mostly because tutors will be paid less. At present, AU tutors are paid in two ways. One is a flat rate based on the number of students a tutor is assigned. This is called block pay. The second way is based on the number of assignments and exams that a tutor has graded. There are also allowances for computer use, phone, and internet expenses.  The block pay is meant to cover the time I spend communicating with students, and the related work. If I were to have a month where I graded no assignments or exams, then my wages would consist of the block pay, plus allowances. Under the new system, block pay will be eliminated. I’ve read that tutors will be paid for each interaction with a student, but to my knowledge, AU has not officially commented on exactly how this will work… likely an indication that they don’t expect it to win over any tutors.

Update (13 June 2016): Here is a list of items that Academic Experts will be paid to do.

Ok, almost to Excel and soul searching. The email from Peter MacKinnon raised my ire because it reflected many of the attitudes toward tutors that I’ve heard expressed elsewhere. I’m hesitant to post an internal email, but if you read through the comments on this post about the [not a call] centre by Tony Bates, you’ll get the idea. I would draw your attention in particular to the comments of Professor Rory McGreal of Athabasca University (who is apparently not averse to the term “call centre”):

In the call centre, they will reach a professional immediately. This professional, unlike the tutor, will have training in the most common questions, queries, concerns that student have regarding administration, schedules, programme requirements, etc.

This quote is helpful because now you won’t mistake a tutor for a professional.

Let’s try another:

The call centre model is especially designed to provide students with the response they need as soon as possible. The previous tutor model allowed for a reasonable call back time of 48hrs. This is no longer acceptable. Students demand the response they need when they need it.

The 48 hour response time exists because tutors are not paid enough to have tutoring as their only employment. It allows for flexibility so that it is possible to manage both jobs. I’m not sure if this means the 48 hours will be changed to “immediately, dammit!” or if it is meant to imply that being told your tutor will contact you counts as a response. Either way, this strikes me as being extremely out of touch with what the reality is for tutors… and it makes students sound like brats.

I crunched some numbers to see what a worst-case scenario might look like, such as a very slow month for grading. You might think this scenario would translate to a month of free time, but it doesn’t. There are a number of activities I engage in to assist my students, that aren’t represented in the pay scheme. Also, I have to keep an eye on email and make sure I respond within the required time frame (either 48 hours or immediately, dammit, I’m not sure which). Finally, I have to keep my schedule sufficiently open and not go too far from home so that I can take care of any tasks that might arise. This last point in particular amounts to a very definite opportunity cost.

The result of these calculations was laughable. I could make more money dog-walking (I like dogs… that wouldn’t be too bad), or sewing sock monkeys, or selling pressed flowers on Etsy. So, while I might otherwise have done some soul searching about my place in an organization where the people calling the shots clearly view me with contempt, Excel made it pretty easy to see the point at which it just wouldn’t be worth it to stay. We’ll call that the self-respect threshold, and I’ll be keeping a very close eye on it.

 

For an analysis of the rationale behind the call centre model: Customer relationship management (CRM) as a paradigm in distance education

Categories: Distance education and e-learning, The business of education | Tags: , , , , , | 2 Comments

When good grades are bad information

Assignment grades versus exam gradesThis week I set out to test a hypothesis. In one of my distance education courses, I regularly get final exam scores that could pass for pant sizes. I have a few reasons to suspect that the exam itself is not to blame. First, it consists of multiple-choice questions that tend toward definitions, and general queries about “what,” rather than “why” or “how.” Second, the exam questions come directly from the learning objectives, so there are no surprises. Third, if the students did nothing but study their assignments thoroughly, they would have enough knowledge to score well above the long-term class average. My hypothesis is that students do poorly because the class is easy to put on the back burner. When the exam comes around, they find themselves cramming a term’s worth of learning into a few days.

Part of the reason the class is easy to ignore is that the assignments can be accomplished with a perfunctory browsing of the textbook. In my defense, there isn’t much I can do about fixing the assignments.  Someone above my pay grade would have to start the machinery of course designers, contracts, and printing services. In defense of the course author, I’m not entirely sure how to fix the assignments. If a student were so inclined (and some have been), the assignments could be effective learning tools.

Another problem is that students tend to paraphrase the right part of the textbook.  Even if I suspect that they don’t understand what they’ve written, I have few clues about what to remedy.  The final result is that students earn high grades on their assignments. If they place any weight at all on those numbers, I fear they seriously overestimate their learning, and seriously underestimate the amount of work they need to put into the class.

So, back to testing my hypothesis: I decided to compare students’ averages on assignments with their final exam scores. I reasoned that a systematic relationship would indicate that assignment scores reflected learning, and therefore the exam was just too difficult. (Because all of the questions came undisguised from the learning objectives, I eliminated the possibility that a lack of relationship would mean the exam didn’t actually test on the course material.)

I also went one step further, and compared the results from this course (let’s call it the paraphrasing course) with another where assignments required problem-solving, and would presumably be more effective as learning tools (let’s call that the problem-solving course).

My first impression is that the paraphrasing course results look like a shotgun blast, and the problem-solving course results look more systematic. An unsophisticated application of Excel’s line fitting suggests that 67% of the data for the problem-solving course can be explained if assignment grades reflect knowledge gained, while only 27% of the data from the paraphrasing course can be explained that way.

I’m hesitant to call the hypothesis confirmed yet, because the results don’t really pass the thumb test. In the thumb test you cover various data with your thumb to see if your first impression holds. For example, if you cover the lowest exam score in the paraphrasing course with your thumb, the distribution could look a little more systematic, albeit with a high standard deviation. If you cover the two lowest exam scores in the problem-solving course, the distribution looks a little less so. There is probably a statistically sound version of the thumb test (something that measures how much the fit depends on any particular point or set of points, and gives low scores if the fit is quite sensitive) but googling “thumb test” hasn’t turned it up yet.

From looking at the results, I’ve decided that I would consider a course to be wildly successful if the grades on a reasonably set exam were systematically higher than the grades on reasonably set assignments— it would mean that the students learned something from the errors they made on their assignments, and were able to build on that knowledge.

 

Categories: Assessment, Distance education and e-learning | Tags: , , , , , | 2 Comments

Blog at WordPress.com.