Muddy Lessons

Note: this resource reflects a traditional approach to scholarly inquiry. It does not incorporate evolving ideas about graphical representations, but instead assumes the thesis as product. Still, it may contribute to your understanding of the research process. The name "Muddy Lessons" comes from a feedback technique that encourages students to submit their "muddy points" (pressing questions) about course content. This resource evolved from these questions.

Most research studies involve a lit review. You need to determine what is already known about a topic before you launch into the investigation, and you may or may not want to duplicate a study that's already been done. The process of doing the lit review teaches you many things about the topic and alerts you to many issues you need to know about. Lit reviews should justify your study, pointing out holes and problems that your study can help fill. A lit review should also show the "brick wall." Visualize a brick wall, with each brick being a study, and similar topics are grouped together. In lots of places, there are bricks missing in the wall, or whole new sections to be built. (Sometimes you might actually be reinforcing one of the bricks by replicating a study, or even replacing a brick that was installed wrong.) The lit review paints a picture of the bricks around, touching, and near to your hole so that we get an idea where it fits. So this is why lit reviews are necessary.

The best time to do a lit review: there are varying opinions on this. One position is that you need to do it right before or while you are forming your topic focus. This is the most traditional. Another position is that you should start the lit review before you do the study, and then revisit the literature as your study unfolds: to focus your topic, to learn about issues that come up in your study, and finally to compare your results against what is already known. Now: watch out, I'm going to confuse you with this final position, common in qualitative circles. Some qualitative researchers believe that you should not do one until the very end. They believe that you should go into a study with no preconceived notions, but in the end compare what you've learned to what was known before. (Not many students choose this option - it's hard to get a committee to go for this, almost like a shortcut to getting your proposal approved. I don't recommend it for Ed.S. projects.)

Levels of lit reviews: In the old days, research studies and dissertations were supposed to include "exhaustive" lit reviews - where you literally explored every single study ever done on your topic. I don't believe this is possible any more, if it ever was. You don't want to miss an important study that was done related to you topic. I've already explained about justifying your study and framing your study (above). A final option is: no lit review. Sometimes, you see this in published articles. For most of my studies, and for you, the lit review should justify and frame, and that is sufficient. Next episode: things to include in a lit review.

Next episode: things to include in a lit review

One point to make before we begin: plan to tell the story of your search in the beginning of your lit review section. You are familiar from this from I-searching. Explaining your lit review methodology (not to be confused with research methods and procedures) helps your readers feel confident that you did a thorough job, given your chosen focus. One class member asked: "What are all the different lit review methodologies?" Actually, these are just a set of techniques and strategies - nothing elaborate like you have learned from your Methodology Recipes. We will address technique in a later Episode.

What kinds of things should you include in a lit review? There is a definite hierarchy here. If there are research studies that have been done related to your topic, then you should fully plan to include them. Often, lit reviews on heavily researched areas consist mainly of research studies. (Remember how to recognize a research study from the work you did on the Crits?) The second level of material to consider are conceptual or theoretical articles or books. For example, there are many books out there on motivation theory, and several different theories of how to motivate learners. You should consider at least referencing the major relevant theorists. The third level of material to include are prescriptive pieces and models - these provide a method or approach for accomplishing a goal. A good example is Julie's I-Search book - it cites a good bit of research, but the focus of the book is to teach how to apply the I-Search model. The information literacy standards are another example of a prescription. The final category of items I would consider including are anecdotal pieces. These are often called "how I did it well" pieces (or jokingly, "how I done it good.") They can be valuable as practical guides, but don't really hold up as solid research. For example, someone who tried an email project in his classroom may well write an interesting article about the experience. It may or may not be helpful to you. Still, such an article is quite different from another teacher who planned a research project around email, obtained approval for the study, conducted the study, and used disciplined data collection to evaluate and report the results. I hope you can see the difference.

I believe that the skills you need to do a good lit review - information access, evaluation, analysis, and synthesis - are the same good old information literacy skills that you need to model for your students. Advanced ability to do all of these things may be the one competency that sets you apart from the other educators in your building.

Add one more component - application - and you have the power to solve problems. Many of you took EDIT 6300 (Administration of Media Programs) from me, and I would like to remind you about the Collaborative Problem Presentations that you did in that class. The whole point of that assignment was for you to go out and find a real media center problem,
research solutions for it, and teach the rest of the class about it. We can't possibly prepare you for even the current problems you will face in your media center, because that would take more than a single M.Ed. or Ed.S. degree program. Not to mention - many new problems will invent themselves for you in the years to come. Research begins with surveying the literature to find out what wisdom already exists on a topic, and
that's what lit reviews are all about.

I hope you won't be like me. As a first year teacher, fresh out of college, I faced challenges with discipline. I blamed my challenges on my education and tried to pretend to my fellow teachers that everything was fine. I almost quit many times in my first year and a half. What
would I do differently now?

1. Understand that new professionals always face challenges; they never know it all on the first day.

2. Ask questions: it's OK (and wise!) to ask your colleagues. They know how to solve many, many problems and will be happy to share their solutions.

This is where good lit review skills come in handy. What happened to me? Eventually, I developed enough strategies through trial and error, asking questions, and reading to be able to deal (mainly prevent) most of the discipline problems that came up. So this is how
lit reviewing should help a new media specialist.

This is the area where lit reviews often fall short. In the original planning of the topic, the researcher can easily fail to consider all the different areas that would need to be addressed in the lit review, causing the final product to come up short one or more topics. Initially, you need to think with a very wide focus. This differs markedly from trying to focus your research question. Do some preliminary hunting around - and then you might find that you need to tighten your focus. Here are two visualization techniques that help me when I start working on a lit review:

_The Funnel._ At the beginning of your lit review, think about the topics that frame your study very broadly. Then, in a few paragraphs, show us each level of context until you get to the specific area of your topic. For example, let's say someone is interested in the topic of levels of technology integration for teachers who have had the Intech experience (yes, this is an example from our class). The funnel might start very broadly at the top with a mention of literature on professional development (vast!). Next, the researcher might mention some literature on technology integration (also vast!). Finally, the researcher might talk about studies that relate specific types of tech integration to how well that plays out in the classroom. In the top of the funnel, you would describe things only in the broadest terms - the job is to show context. As you come down further in the funnel, you describe studies in more detail. I find the funnel to be a very useful image for organizing a lit review. It also helps you to identify the topics you need.

One more visualization tool that might help is the Venn diagram. Here, you take your research question or purpose statement and highlight every concept or area that seems important. List any others that might be important as well. Then, try to draw these topics in overlapping circles. In the example above, visualize a circle with "professional development" and another circle with technology integration." Where they overlap (like Venn circles usually do), might be the area of research studies relating levels of tech integration with specific types of professional development.

This was hard to explain. Whether these tricks above help you or not, the moral of today's episode is: 1. start out by thinking of all possible areas of your lit review. 2. Do some preliminary searching and reading to see how large each area might be. 3. Narrow or broaden accordingly, remembering that you should at least expect to provide context for the large areas. 4. After you've done all of this, a next good step is to draw a concept map OR to write an outline. 5. Check with your advisor to see if there might be other areas that are missing.

Of course, the place to start looking for any educational topic are the GALILEO databases ERIC and Education Abstracts. ERIC is by far the largest of these, but tends to be months behind in cataloging new articles. A librarian at any academic library could give you lots of fine points in using ERIC like a power user. Make sure you understand the difference between EJ and ED citations - give your energy to EJ citations as much as you can. I'll leave it up to you to find out why! Education Abstracts provides a much smaller range of journals, but tends to be more up to date. Another important database for school librarians is Library Literature. Abstracting in this database is skimpy, but if your topic has anything to do with libraries, you must check it. If your study has any psychology in it, make sure to check PsycInfo. (Many educational research studies do - for example, motivation.)

Even if you can't access some of the great articles you find in these databases full text (your mileage will vary depending upon the type of GALILEO login you use), compiling lists of citations at home will save you lots of time. Take the time to search carefully and select stringently from your results - you can make many elimination decisions in this way. Then, organize your list by call number, and plan a field trip to a good academic library. Unfortunately, the best library around for education - UGA Main - has closed the L call numbers that we normally need. However, even with this considerable handicap, you will be able to get a lot of work done before visiting. (Avoid football Saturdays!) We are hoping that UGA Main will reopen soon!

Favorite journals: the only research-based school library journal is School Library Media Research - which can be found totally online and free linked from www.ala.org/aasl. If you are a school librarian, don't forget to check this one! This is not to say you won't find research articles about school libraries in other journals. Library Literature will help you find these articles. For educational technology, there are more choices: Educational Technology Research & Development is one.

I'm sure you all have favorite tricks. Jo and Chad are experienced database searchers and literature compilers. One of my favorite tricks is to find ONE article that hits the nail on the head. Then, I use a) the list of references to go further (performing searches on some of those writers listed), and b) use those exact search terms/keywords for further searches. Another important strategy: keep track of all of your searching activities - including lists and search terms. Otherwise, you will repeat yourself. This process will also help you tell the story of your search - recommended in the beginning of the lit review section or chapter.

This is where many people have trouble with their lit reviews. It's easy to read and read and read and take lots of notes for months .... and then not have a clue about what to do with them. I am the middle of doing this right now, so I know what I'm talking about!

It's important to read attentively, using your I-Search strategies of highlighting and annotating as you go. I underline in pencil everything that I think is relevant to my study. If I have thoughts, I write them in the margin. I keep my photocopies of articles forever and ever, and I've learned that this pays off. (I file them in large filing cabinets by author - all indexed through the special bibliographic database Endnote).

When I've reached that point where I feel like I have read enough, or after 20-25 resources or so, I sit down and think about the categories of information I have read. I go back through my material and write little code words beside each underlined segment. The code word means: this section is about _____. For example, right now I'm writing an article about the information literacy shortcomings of college undergraduates. I'm dealing with what professors *expect* and the way students *are* - I call this the *problem.* So, I write these code words to the side of relevant passages.

The next step is to organize codes. Inspiration is great for this; sticky notes work well (write one idea per sticky, and then you can move them around). If you're linear like me, the Word outline feature is great. Just plop your codes in and group like ones together, and then put in some kind of sequential order. This process will teach you much about your topic.

Finally for today, it's important not to lose track of citations and quotes in this critical organization phase.

Next episode (not sure when this will happen!): how to gather similar bits into your outline (tracking), beads-on-a-string vs. thematic organization, and perhaps writing; using cut-and-paste to help arrange
(perhaps).

Once you have coded your material by topic area, it’s time to sort chunks. Old-fashioned notecards are one way to do this – which I don’t recommend! Use the thought as an organizer, one idea per card, each card coded to the reference – but it breaks my heart to see people writing out their literature on notecards these days. Transfer that process to your word processor, at least. If you like to type out or copy notes, make a file of these. When you’re done, you can print all of your notes out. Then, take real scissors and cut your notes apart. Then, sort them into piles according to the topical code. Then, arrange the notes in a reasonable sequence. Then, you can start writing. This sounds like a lot of work – it is. But, the thought processes that carry you through them will take you a long way down the road to writing your lit review. A quicker method might be that you make notecards that reference the ideas only, and then sort those. As a fictional example, your notecard could say: “Smith, 2003, p. 12: summer reading participation linked to higher test scores.” Then you might give that card a code of “achievement” because you’ve decided this is a category of material you’d like to address. Your stack of cards would allow you to track back and forth between the articles and your index system without having to copy the notes. I use special software to help me accomplish the task - if you're hoping to be a career writer or academic, ask me about this.

Two styles of organizing: avoid the tendency to write “beads-on-a-string” lit reviews. (This term was invented by Dr. Jude Priessle in our College, one of my professors.) This is where you summarize article after article, one after the other. The reader does not benefit much when you do this, because you have not done the hard work of synthesizing the ideas. Instead, organize your writing according to themes or ideas – the codes you have assigned to all of your notes. In other words, if you’re talking about achievement in relation to reading programs, you would cite many authors in one paragraph, relaying what each of them has to say all in one place. You add value by organizing and then summing up what you’ve learned.

Finally, a word about writing: my advice is, don’t be a perfectionist in the first draft. No one but you should see your first draft. To strive for beautiful prose as you struggle to get your thoughts out is a certain kiss of death for your motivation. This is not to say that the draft you hand to your advisor shouldn’t be clean – it should. It takes too much time to read a draft that is full of errors – don’t do this to your advisor. The draft you first share with your advisor need not be perfect, but it really should be as grammatically clean as you can make it. But don’t worry about grammar as you initially get your thoughts out. I use all kinds of motivation strategies to make myself write. The most important one is that I try to do it first thing in the morning, immediately upon arising, coffee in hand, since I'm very much a morning person. That way, I don’t have a chance to build up procrastination and to get interrupted. I say to myself: I will write for 2 hours and then be done with it for the day. It’s amazing what you can get done in 2 hours when there are no interruptions. As you end a writing session, make sure you identify the very next step – it will help you come back to the project tomorrow if the first little bit is easy.

I feel like we covered the Lit Review topic pretty thoroughly. You may still have questions - feel free to ask them.

Here's a good question from the Muddy Points pile: "If Dr. Wisenbaker is retiring, what is another option for data analysis?"

1. If you're doing a qualitative analysis, all three of us school library media professors can pretty much give you most of the advice you will need. We have LOTS of ideas and opinions on this. I wanted to make sure that you don't equate “data analysis” with statistics, because analysis covers the gamut of qualitative and quantitative.

2. There are several IT faculty who can get you started with stats analysis as well - and talented doctoral students. Dr. Tallman is particularly skilled with surveys. Any of us can help you with
descriptive statistics - all you need is Excel for that. Many, many student research studies need go no further that descriptive stats. This is nothing more than summarizing the numbers - totals, averages,
percentages, etc. Most of us can even handle a lot more than that in terms of stats, or ask one key question of an expert and then go on from there.

If you are in that small group of students who will actually need expert statistical advice, be assured that there are resources on campus to help you. Also, some school systems have statistical experts employed centrally (to help with all that test data), and such an expert might enjoy helping you out - it’s worth a look and a conversation. Here on campus, there are statistical resource centers in at least two places: the College of Ed., and in the Statistics department. Just because Dr. Wisenbaker is retiring doesn’t mean that our College stat center will disappear - we hope! I’ve heard no rumors to that effect. And, if it DID close, we have an ENTIRE building devoted to Statistics very close to here. (Imagine!) I have been over there myself with data in hand and received very competent advice. There might be a nominal fee involved.

"How do I go about posting a survey on a listserv and do I need IRB approval for that?"

I don't have any personal experience in this area, but I think we're all interested in it. I can think of 3 avenues.
1. First of all, don't forget about good old email and listservs. This is a low tech way of doing it that has lots of advantages over the paper kind - no transcribing, just cut and paste.

2. This might be a question to explore with OIT in the College. They are using online survey tools (not WebCT) to gather a variety of information, like the Course Evaluations. At the very least, you could
approach them and learn more. I don't think this is something you are going to want to try to MAKE from scratch all by your lonesome. You will definitely need server-side support unless you know a lot about programming. I know a couple of expert names in our department as well, and will be happy to share with you if you are really interested in this.

3. Look at SurveyMonkey.com. I understand this is easy to use and inexpensive. There may be other similar sites.

4. I thought of one more. WebCT can do surveys, simple ones at least. If your participant pool consists of UGA students (who have myIDs), this might be a possibility in collaboration with a professor.

As for the second part of the question: IRB approval needed? Oh yes, yes, yes. You may be able to get an implied consent approval. Two important issues: dispersion of the survey and security/privacy. You can't just send your survey to LM_NET and expect IRB to like this. They will have to approve your "recruitment" procedures even if it's an email invitation. Also, lots of listservs won't allow this kind of thing. Second: how will you ensure that the people who login to a web-based survey are the RIGHT participants? Well, you can't completely, but there are steps that can be taken. And, how will you protect their data? If you're like me, you're very cautious about typing in your personal information all over the web, because it's difficult to control or know how that information will be used. You don't want to be on the receiving end of non-private data.

It's never as easy as it sounds, is it? With all of those cautions, I urge those of you who're interested to get out there and explore. And then you can teach the rest of us.

"For my survey of students' parent permission letter, can it be of the type where they only have to return it, if they don't want their child to participate?"

I will admit to being very fuzzy about this question. "Implied consent" has been a principle I've seen used - wherein if the person fills out the survey, then consent is implied and you don't have to go to the
extra step of signing the consent form. My guess is that if you are surveying minors, parents will always have to give consent.

I made a quick visit to the IRB Guidelines http://www.ovpr.uga.edu/hso/guidelines.html), but didn't find a definitive answer in a reasonable amount of time. You are welcome to study these guidelines yourself - we did look at them earlier in the semester. Once you know what research you're going to do, they make a lot more sense.

When I'm not clear on an IRB question, I always recommend that the student simply call the Human Subjects office and pose the question. If there's any chance that you can escape doing part of the IRB, then it's worth a try! Before you call, make sure you're clear on what your procedures will be, or the time will be wasted.

As you probably have noticed by now, qualitative researchers take a different world view on a number of things. For one thing, many of them believe that it's very difficult to ever really "prove" or "validate" anything at all. This seems particularly true when you're talking about any research dealing with human beings! Think about all of the medical studies you know about, and how difficult it is to know something for sure about all human bodies. How much more difficult it is to "validate" something about human thinking and learning!

And so, you will hear qualitative researchers talk about "trustworthiness" instead of "validation." They have a number of methods for making a study as trustworthy as possible - so that the reader can feel reasonably assured that the researcher is telling the story truthfully and completely, and so that the reader can make a judgment about whether or not the study relates to personal situations.

Some of these procedures are:
*Triangulation - using 2 or more types of data to support the same finding (like: if a participant claimed something in an interview, did it hold up under observation)? This is the main trustworthiness technique used by qualitative researchers.
*Audit or Paper trail: if you reach a conclusion, can it be traced back to data? Are all the records collected in the study pointing to this conclusion?
*Member check: After an interview, the transcript is shown to the interviewee, who has a chance to repudiate any errors it may contain.

These are just a few. Any good qualitative "how-to" text will list these and more.

"How do you pilot a survey or questionnaire if your target group are the only participants that you have?"

Another good question. If you have a small population to begin with, you don't want to mess up your potential data set by trying out a version of your survey on one or more participants in that population. It's possible that having a preview of the survey might alter their eventual real answers in some way.

One reminder: the difference between populations and samples. Oftentimes in applied projects, you survey the entire population -- all the people that are eligible to be a member of a particular group, often a school staff, or a grade of children in a school. They constitute a population in some cases because they inclusively contain all the people that share a certain set of characteristics, such as being a member of THIS school, with THIS intervention going on in THIS special way. It all depends upon your research question. Sampling would be if your population is too large to survey in total - you would use some procedure to represent that population through selection. Also remember that if you invite an entire population to participate, you will seldom get them all to enlist - that's voluntary participation, which is different from random selection. If these fine distinctions seem to apply to you, reread the Leedy section about this.

So let's go back to the original question and say that your population is tiny to begin with. I would suggest that you find some kind of parallel group to try your survey on. Let's say you're surveying all
the media specialists in your county, and because that county is unique in certain respects relevant to your study, they are their own tiny population. I would contact 2 or 3 media specialists in a DIFFERENT county that has at least some similarity to your county, and ask them to try out your survey. Of course, you would not use their data except to help you improve your survey, and you should tell them that you are piloting your survey. Your peers in the cohort might be similar enough
to your real participants to serve as pilot participants.

"As a Media Specialist, would I need to go through the IRB process to conduct an informal research project to collect data to support my changing of the AR program at my school?"

If you need to conduct an evaluation of some aspect of your local school or media program, AND that data will be used only to assess and/or improve the program, this is considered normal research that is a part of your job. You will not be sharing that data beyond the local setting - if this isn't the case, you should consult locally to see if there are research guidelines to be cleared. Also, you would need to use your common sense about how you're treating your participants - like, how will teachers' opinions be used? You wouldn't want to bring risk to any participant in any way. So, your knowledge of ethical research will help you in this respect.

Once you have no more affiliation with UGA as an active student, you need not worry about IRB - only your local employer.

"In simple form, what are the main steps of a research project
(beginning to end) - in bullet form?"

I can provide the theoretical linear model for steps in a research project, which I will do below. First though, I'd like to remind you of Chutes 'n Ladders, the discussion we had briefly on Sept 27 (http://it.coe.uga.edu/~mfitzger/7320/agenda.html#sept27). That's where we said that it's tempting to approach a research project as Step 1 - Step 2 - Step 3, but in reality, there are lots of loops involved. Instead of an orderly list of bullets, your pattern may look like a wavy snake, a maze, or the game Chutes 'n Ladders (where you get to a certain point and then have to go back several steps and do something over). That said, here are the theoretical bullets, in order:

1. Know the task ahead of you. What is the assignment (applied project, dissertation, evaluation, etc.)? Find out as much as possible about the expectations. Plan a tentative timeline, and plan your project as far as possible. Choose graduation semester, and mark on your calendar any
deadlines that are associated with the project.
1.5. Who are you going to work with? Who are your collaborators? Who is your advisor? Answer all of these questions.

2. Brainstorm topic list.
3. Read/browse about the most promising (interesting) items on your brainstormed topic list.
4. Choose a topic.
5. Read much more about this topic, to find out what is already known. Use good lit review methodology from the beginning, so that nothing gets lost. Track references from the very beginning!
6. Brainstorm possible questions.
7. Choose one research question. Refine the question so that it expresses your topic as closely as possible.
8. Generate: "The purpose of this study is ..."
9. Go ahead and write the Background and Problem if you can.
10. Brainstorm a list of possible methodologies. Learn about the most promising one(s).
11. Choose a methodology, and outline your procedures. (Circling back to #4 or #7 is common here.)
12. Invent instruments.
13. Write IRB application. Also, find out what local (system) approvals may be necessary and plan those steps to coincide with UGA's IRB process.
14. Obtain approvals: your advisor and/or committee, your system/administration, IRB. This is a good time to revisit your timeline. Don't forget about early submission deadlines - usually weeks
before the end of the semester you want to graduate.
14.5. While you're waiting for approvals, write your lit review and methodology sections. Adjust if approval process requires changes. Show to your advisor; edit as suggested.
15. Collect data as planned.
16. Manage data - organize it, label it, track it, etc.
17. Analyze data.
18. Write Results section. Show to your advisor; edit as suggested.
19. What does it all mean? Write Discussion section, with Conclusions, and then Recommendations, and Suggestions for Further Research. Show whole report to your advisor; edit as suggested, as many times as necessary. (1 week is not enough for this!)
20. Defend project, if required - or present to the proper audience.
21. Edit. Finalize. Celebrate! Think about how to share - at a conference? As a journal article?
22. If appropriate, follow through with sharing to a larger audience.