The Evaluation Framework

Here, we outline, in more detail each of the major categories and subcategories of factors that teachers might consider as they evaluate iPad apps for use in their classrooms on a given day. We invite you to scroll through all seven areas below or, if you prefer, please click on a link in the table of contents to jump to a particular area of interest. As well, immediately below each area you will find a request for your input. Thank you in advance for taking a minute to share your thoughts and suggestions!

Technology logistics considerations
General pedagogical-technological considerations
Curricular considerations
Differentiation of curriculum and instruction considerations
Engagement and motivation considerations
Effectiveness considerations
Cultural and ideological considerations

Technology logistics considerations

  • Cost – How much does this app cost? Are there hidden costs that are not at first apparent? (e.g., required payment to add more voices or characters to a storytelling app)
  • Compatibility – Is this app compatible with our existing hardware and software? Does this app work with the app-management software/service (e.g., AirWatch) my school uses?
  • Set Up – Does the app require any “setting up” of any kind–e.g., creating accounts for individual students?
  • Bandwidth – Once installed, can the app be used “offline” (without wireless connection to the Internet)? If the app works only with a wireless connection, does it work with the amount of bandwidth available in the classroom?
  • Maintenance – Will I need to install updates to this app?
  • Safety – Does the app have the potential to expose users to advertising or to take them to external websites?

General pedagogical-technological considerations

  • Teacher training – What amount of initial (self-)orientation and (self-)training will I need to do to get “up to speed” with this app and comfortable using it?
  • Student training – What amount of initial explanation and guided practice will be necessary before my students can start using this app’s basic features? (e.g., Are explanations, instructions, and FAQs available within the app, such that students can consult them on their own, as necessary? Are explanations and instructions accessibly written for my students? Can a student click to hear instructions spoken aloud?)
  • Joint Use – Can this app be used by more than a single student at once?
  • Universal Accessibility – Does this app respect principles of UDL (e.g., accessibility for students with visual impairments)?
  • Time Flexibility – Can this app be used flexibly for shorter or longer segments of time, or does it require a specific time commitment (e.g., to complete a segment of activity)?
  • Progress Monitoring – Is there an efficient way for me to see the work a student has done so far with this app?
  • Sharing – Is there a way for students’ work with this app to be shared with classmates or to a larger Web community via social media tools?
  • Availability from home – Can students easily use this app on a home device?
  • One Time (Consumable) vs. Repeated Interactions? Is this an app that kids will interact with several times or is it something that is designed for single use?



Curricular considerations

  • Connection to existing curriculum – Does the app connect in helpful ways with my existing curriculum? (e.g., does a Rapunzel story app fit with a unit on fairy tales?) Does the app enhance or amplify existing elements of my curriculum?
  • Types/Aspects of Comprehension – What specific aspects of reading comprehension does this app target and develop? (e.g., making inferences, connecting new information to background knowledge, noticing and thinking about discrepancies, identifying main ideas, text structures)
  • Other reading-related skills – Beyond the aspects of reading comprehension that appear to be specifically targeted, are there other reading-related skills that the app engages and develops (e.g., picture drawing)?
  • Genre-specific reading strategies – Does this app support the development of reading strategies for a particular genre of text (e.g., strategies for reading expository informational texts)?
  • Discipline-specific ways of reading & knowing –– Does this app support the development of reading strategies, mindsets and ways of knowing that are unique to a particular discipline? (e.g., science, history, math)



Differentiation of curriculum and instruction considerations

  • Differentiation for Reading Levels – Does this app have features that will allow me to differentiate instruction for students reading at different levels? (e.g., does the app have a text-to-speech feature, a dictionary feature)
  • Built-in Scaffolding – What kinds of scaffolds does the app provide to students? How useful are these particular scaffolds likely to be to readers of different ability levels? (e.g., the option of a fill-in-the-blanks template for a storytelling app; MC questions that periodically allow the reader to assess his/her comprehension of what he/she has just read)
  • Scaffolding for ELL students – Does the app provide scaffolding geared to ELL students? (e.g., a bilingual glossary)
  • Level of Challenge – What level of challenge does the app impose (re. vocabulary, sentence structure, etc.)?
  • Adjustability of challenge – To what extent can the app’s level of challenge be adjusted by the teacher or the student using the app?
  • Media and modalities – Does the app provide reading and meaning-making cues in different modalities or media?
  • Goal setting – How versatile and open-ended is the app vis-a-vis goal setting? Does the app allow different students to work toward different self-chosen or assigned learning goals? (e.g., if the app contains quizzes, can these quizzes be skipped, so that a student can focus on something else, such as practicing oral reading fluency?)



Engagement and motivation considerations

  • Engagement and motivation – Does the app contain features designed to increase students’ engagement or motivation? (e.g., a scenario is presented in which the user is enlisted to help solve a mystery)
  • Rewards – Are there rewards or prizes of any kind built into the app’s design? (e.g., animations that come to life when a correct answer is given)
  • Extrinsic Reward – If the app provides a prize or reward, is this prize or reward something extrinsic to the the reading-learning activity and likely to promote extrinsic motivation (e.g., motivation stemming from a desire to accumulate more points or gold coins than other readers)?
  • Intrinsic Reward – If the app provides a prize or reward, is this prize or reward something intrinsic to the the reading-learning activity and likely to promote intrinsic motivation based on enjoyment derived from the activity itself?
  • Congruence of Content and Reward – Is there alignment between rewards/prizes and the app’s content or are the rewards/prizes incongruent?
  • Distracting Reward – Will rewards/prizes likely be distracting or disruptive to the user’s learning experience or to his/her classmates’ learning?
  • Affiliation & Interaction – To what extent is social or networked interaction embedded in the app as a means of motivating participatory activity in a group? Does the app lend itself to cooperative learning?
  • Authenticity – Does the app connect with purposes and contexts for reading comprehension that are authentic for K-12 students? (e.g., identifying topic sentences in an uninteresting text) How authentic are the tasks or activities the app provides?
  • Interest – Does the app feature “high-interest” materials or activities that match up well with the interests of my students?
  • Choice – Does the app engage and motivate students by giving them some degree of choice re. texts, reading tasks, or other factors?



Effectiveness considerations

  • Background knowledge – What type(s) and amount of background knowledge does the app presume or require for it to be used meaningfully and successfully by my students?
  • Cognitive Load – Is the app mindful of the amount of cognitive load my students can handle and of the kinds of things that, for them, will increase/decrease cognitive load? (e.g., having to click back and forth between a passage and the MC questions about that passage on a separate page in the app vs. having passage and MC questions visible side-by-side on the same page.)
  • Memory Processes — Does this app require children to act on/manipulate what they read and understand in ways that support schema construction and encoding to long-term memory?
  • Self-regulation — Does the app actively cultivate students’ capacity for independent decision-making, self-monitoring of performance, self-correction, stopping to think and evaluate the relevance of information, a mindset of thoughtfulness vs. impulsivity? (e.g., how many decisions will my students have to make re. where to start every time they open this app? does the app provide feedback re. the progress a student is making through the app?)
  • Multiple Sensory Channels — Do the app’s interactive elements support or amplify the app’s reading comprehension focus? Do the multi-modal elements permit children to process auditorily, visually, kinesthetically to construct meaning?
  • Cognitive Flexibility — Does the app support the development of a problem-solving mindset or engage students in activities that require simultaneous processing of multiple ideas?
  • Scaffolding – To what extent does this app scaffold students’ understanding of the texts? (Hard vs. Soft; universal vs. targeted; technical vs. teacher-provided) (Saye and Brush, 2002)
  • Pacing – Does the app allow the user to read and/or work at his/her own pace, or is the pacing of the app prescribed? If the latter, is this pacing appropriately fast or slow for my students?
  • Sequencing — Are sequential tasks within the app ordered in ways that scaffold success & understanding?  (e.g., easier tasks followed by more difficult tasks)
  • Text features — To what extent do text features position the text within the child’s zone of proximal development? (e.g., word choice, headings, lists of important ideas etc.)
  • Feedback — To what extent does the feedback that students receive support and/or extend what the child can then do independently?



Cultural and ideological considerations

  • Portrayal of reading – How does this app portray the activity of reading? (e.g., is reading portrayed as a game, as a pleasurable aesthetic experience, as a puzzle to be solved, as the mastery of one or more particular skills?)
  • Ideological messages – Does this app attach any ideological messages to reading that could be helpful or unhelpful to my students’ development as readers? (e.g., Does the app “feminize” reading by showing only female characters who read or who are interested in books? Does the app convey that readers are superior to non-readers?)
  • Race and ethnicity – Will the content of the app appeal equally to all my students, or will its content make it much more interesting or understandable to some students? Are there significant cultural biases or insensitivities built into the app that I need to be aware of?
Advertisements

2 thoughts on “The Evaluation Framework

  1. David Reinking

    Nice list. Could you provide another level of information where you given an example of an app that illustrates a positive and/or negative aspect of each item in the list? That might lead you to streamline the list a bit. I’m not sure teachers, for example, are as attuned (nor reasonably need to be) to several of the items under effectiveness (e.g., cognitive load).

    Reply
    1. Michelle Schira Hagerman Post author

      Hi David,
      Thanks so much for your comment. If you check under the Examples tab in the top navigation, you will see a few examples of apps that demonstrate a range of the variables outlined in the list. We agree that teachers don’t necessarily need to be attuned to all of the items listed under effectiveness at any given time — however, we see value in the list as an awareness heightening exercise too. Ultimately, we think that teachers will know what’s important for them and for their students on any given day — but as we developed this, we also thought that the experience of reviewing the criteria might also bring to the fore additional issues for teachers to contemplate that may (or may not) have an impact on reading comprehension development, specifically. Given the complex interactions of content, pedagogy and technology, we expect that teachers will have different objectives for different students at different points in the day, week, month, year etc. — so we included lots of options for teachers to consider as they choose an app to support reading comprehension.
      Again, thank you so much for checking out our Reading App Map!
      Best,
      Michelle, Paul and Doug

      Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s