Author: Christian Schmieder

Categories
Extension Fellowships Impact Software Technology Working Differently

Building Evaluation Capacity Through Data Jams, Part 3: Readying Extension for the Systematic Analysis of Large Qualitative Datasets

In this third blog post on the University of Wisconsin-Extension Data Jam Initiative, I will focus on four institutional outcomes of this Evaluation Capacity Building Framework.

Screenshot from the University of Wisconsin-Extension Civil Rights Legacy Datasets in MAXQDA.
Screenshot from the University of Wisconsin-Extension Civil Rights Legacy Datasets in MAXQDA.

INSTITUTIONAL OUTCOME 1: Continuous use of Institutionally Collected Data

The Data Jam Initiative provides colleagues with the tools, skills, support and community they need to engage in the analysis of large, often fragmented and hard-to-analyze textual datasets. We are currently conducting a longitudinal study measuring the initiative’s impact on analytic self-confidence and proficiency. At this early stage we observe heightened empowerment in Extension professionals, and we see a steep increase of evaluation, research and internal development projects that utilize the data from our central data collection system.

INSTITUTIONAL OUTCOME 2: Improvement of Institutional Data Quality

An essential element of the Data Jam Initiative is to communicate to colleagues and leadership how data are being used. Institutionally, this validates colleagues’ efforts regarding reporting, and it supports leadership in adjusting data collection foci based on ongoing, interdisciplinary data analysis. This, in turn, helps keeping institutional research, evaluation and communication efforts in alignment with ongoing data collection and storage.

INSTITUTIONAL OUTCOME 3: Building Interdisciplinary Capacity to Quickly Respond to Emerging Analytic Needs

All-Program area Evaluator Data Jam at the University of Wisconsin-Extension, March 2017.
All-Program area Evaluator Data Jam at the University of Wisconsin-Extension, March 2017.

Over time we create a baseline of shared techniques for analysis, and distributed proficiency in utilizing Qualitative Data Analysis software. Consequently, colleagues can tap into shared analytic frameworks when they collaborate on projects. On a larger scale, the institution can quickly and flexibly pull together analysis teams from across the state, knowing that a number of colleagues already share fundamental analytic and technical skills, even if they have never directly worked together. This allows an institution to respond quickly and efficiently to time-sensitive inquiries, and  to analyze more data more quickly, while bringing more perspectives into the process through work in larger ad-hoc analysis teams.

INSTITUTIONAL OUTCOME 4: Retaining Analytic Work through Legacy Datasets

Qualitative Data Analysis Software is designed to allow for detailed procedural documentation during analysis. This allows us to retain the analytic work of our colleagues, and to merge it into a single file. For example, we created a “Civil Rights Legacy Dataset” – a Qualitative Data Analysis Software file that contains all programming narratives containing information on expanding access to underserved or nontraditional audiences, currently from 2014 to 2016. This surmounts to approximately 1000 records, or 4000 pages of textual data. The file is available to anyone in the institution interested in learning about best practices, barriers and programmatic gaps regarding our work with nontraditional and underserved audiences.

The analyses that currently conducted on this dataset by various teams are being merged back into the “Legacy File”. Future analysts can view the work benches of prior analysts and projects, thus allowing them to use prior insights and processes as stepping stones. This enables the institution to conduct meta-analyses, maintain analytic continuity, and to more easily and reliably distribute analytic tasks over time or across multiple analysts. You can find more information on the use of Legacy Datasets in Extension in an upcoming book chapter, published in Silver & Woolf’s textbook on utilizing Qualitative Data Analysis Software.)

Beyond Qualitative Data: A Pathway for Building Digital Learning and Adaptation Skills

The outcomes above are immediate institutional effects the Data Jam Initiative was designed for. But maybe more importantly, we’re creating a base line of proficiency in negotiating between a technical tool and a workflow. Our tools change. Our methodological approaches differ from project to project. Each new project, and each new digital tool requires that we engage in this negotiation process. Every time, we need to figure out how we can best use a tool to facilitate our workflows; this skill is a fundamental asset in institutional professional development, and it transcends the topical area of evaluation.

This means that the Data Jam initiative, as an approach focused on mentorship and making by imbuing a technical tool with concrete, relevant processes, is not limited to qualitative data – it can be a framework for many contexts in which Extension professionals use software to do or build things: Be it visualization tools, digital design and web design, app development, statistics and quantitative research, or big data tools.

The development of the Data Jam Initiative Tool Kit has been supported by an eXtension Fellowship. To access the curriculum, examples, videos and training materials, please visit the UW-Extension Data Jam website: http://fyi.uwex.edu/datajams/

Categories
Extension Fellowships Impact Software Technology Working Differently

Building Evaluation Capacity Through Data Jams, Part 2: Software as a Teaching Tool for Data Analysis

 In this second blog post on the University of Wisconsin-Extension Data Jam Initiative, I will focus on the role of software in Data Jams,  and on the skills that colleagues are building in this Evaluation Capacity Building Framework.
Screenshot from the Qualitative Data Analysis Software MAXQDA.
Screenshot from the Qualitative Data Analysis Software MAXQDA.

 

What is Qualitative Data Analysis Software?

The technical backbone of the Data Jam Initiative is Qualitative Data Analysis Software – often abbreviated as QDAS, or CAQDAS (Computer Assisted Qualitative Data Analysis Software). This type of research and evaluation software is designed to support the analysis of large amounts of textual information. It allows for efficient data management and the distributed analysis of large datasets in large teams. While Qualitative Data Analysis Software (such as MAXQDA, NVIVO, Atlas.TI or Dedoose) cannot do qualitative analysis by itself, modern packages typically do offer a wide array of options for coding, documentation, teamwork, qualitative data visualization and mapping.

Focusing on Analytic Collaboration, not on where to Click

Data Jam at the University of Wisconsin-Extension, August 2016
Data Jam at the University of Wisconsin-Extension, August 2016.

In a Data Jam, groups of colleagues analyze data together while using the same analytic software tool, and similar analytic techniques. This creates a common experience of bringing a tool (the software) and a process (the analytic techniques) together. We’re not teaching how to click through menus; we’re not teaching theoretical workflows. We analyze, we make things – with a real tool, a real question, real data and concrete techniques. These concrete analytic techniques emphasize writing and documentation throughout the process, and they focus on utilizing analysis groups to drive the analysis. In a Data Jam, colleagues practice how to stay focused on their research question, and how to work as an analysis group to produce a write-up at the end of the day.

Qualitative Data Analysis Software empowers colleagues to quickly explore our large datasets, and to dive into the data in an engaging way – as such, this software is a powerful tool to illustrate and practice methodological workflows and techniques. We’re not only building individual capacity – we are building a community of practice around data analysis in our institution. I will focus on this aspect in the third blog post, but I will briefly describe outcomes on the individual level here.

Individual Capacity Building & Improved Perception of Institutional Data Collection

On the individual level, we are seeing two outcomes in our ongoing evaluation of the initiative: Firstly, we build analytic capacity and evaluation capacity. Colleagues learn how to analyze textual data using state-of-the-art analytic tools, and they learn how to integrate these tools into their evaluation and research work flows. View the 3-minute video below to view some impressions and learning outcomes from a 4-day Data Jam for Extension research teams.

https://youtu.be/IOWhots-qdc

Secondly, colleagues gain a better understanding regarding how (and that!) the data that they enter in the central data collection system are being used. Our evaluations show that colleagues leave our Data Jams with an increased understanding as to why we collect data as an institution, and as to why it is important to enter quality data. Experiencing the role of the analyst seems to have a positive effect on colleagues’ perceptions of our central data collection effort, and leaves them excited to communicate how the data are being used to their colleagues.

Not every colleague will use the software or engage in research in the future; our goal is not to make everyone an analyst. But we establish a basic level of data literacy across the institution – i.e. a common understanding of the procedures, products, pitfalls and potentials of qualitative data analysis. This type of data literacy is a crucial core skill as we are undergoing the Data Revolution.

The development of the Data Jam Initiative Tool Kit has been supported by an eXtension Fellowship. To access the curriculum, examples, videos and training materials, please visit the UW-Extension Data Jam website: http://fyi.uwex.edu/datajams/

Categories
Extension Fellowships Impact Software Technology Working Differently

Building Evaluation Capacity Through Data Jams, Part 1: A Response to the Data Challenge

Collecting large amounts of textual data is easier than ever – but analyzing those growing amounts of data remains a challenge. The University of Wisconsin – Extension responds to this challenge with the “Data Jam Initiative”, an Evaluation Capacity Building model that focuses on the collaborative, making-centered use of Qualitative Data Analysis Software.  In this first of three blog posts I will provide a brief overview over the Initiative, the tools we’re using, and the products we’re making in Data Jams.

Data Jam at the University of Wisconsin Extension, August 2016
Data Jam at the University of Wisconsin Extension, August 2016

Extension’s Data Challenge

Extensions collect large amounts of textual data, for example in the form of programming narratives, impact statements, faculty activity reports and research reports, and they continue to develop digital systems to collect and store these data. Collecting large amounts of textual data is easier than ever. Analyzing those growing amounts of data remains a challenge. Extensions and other complex organizations are expected to use data when they develop their programs and services; they are also expected to ground their communications and reports to stakeholders in rigorous data analysis.

Collaborative, Software-Supported Analysis as a Response

Data Jam at the University of Wisconsin - Extension, August 2016
Data Jam at the University of Wisconsin – Extension, August 2016

The University of Wisconsin-Extension responds to this expectation with the Data Jam Initiative, an Evaluation Capacity Framework that utilizes Qualitative Data Analysis Software. In monthly full-day Data Jams and multi-day analysis sessions, colleagues meet to explore and analyze data together. Data Jams are inspired by the concept of Game Jams. In Game Jams, game developers meet for a short amount of time in order to produce quick prototypes of games.

Asking Real Questions, Analyzing Real Data

The most important feature of Data Jams is that we work with data and questions that are relevant to our colleagues; in fact, most topics in Data Jams are brought up by specialists and educators from across the state.  By collaboratively analyzing programming narratives and impact statements from our central data collection system, we start answering questions like:

  • How are equity principles enacted in our Community Food Systems-related work?
  • How do our colleagues state-wide frame their work around ‘poverty’?
  • How does state-wide programming in Agriculture and Natural Resources address Quality Assurance?
  • How are youth applying what they’ve learned in terms of life skills in our state-wide 4-H and Youth Development programming?
  • How does FoodWIse (our state-wide nutrition education program) partner with other organizations, both internally and externally?
Data Jam products are shared with colleagues across the institution.
Data Jam products are shared with colleagues across the institution via our internal Data Jam blog.

Using Qualitative Data Analysis Software, Data Jammers produce concrete write-ups, models, initial theories and visualizations; these products are subsequently shared with colleagues, partners and relevant stakeholders.

Building Institutional Capacity to Analyze Large Datasets

Data Jam at the University of Wisconsin-Extension, February 2017
Data Jam at the University of Wisconsin-Extension, February 2017

Through the Data Jam Initiative, we build institution-wide capacity in effectively analyzing large amounts of textual data. We connect teams, researchers, evaluators and educators to develop commonly shared organizational concepts and analytic skills. These shared skills and concepts in turn enable us to distribute the analysis of large data sets across content and evaluation experts within our institution. The overall goal of the initiative is to enable our institution to systematically utilize large textual datasets.

Since early 2016, we use the Data Jam model in monthly one-day Data Jams across Wisconsin, in regular internal consulting and retreat sessions for project and program area teams, and in graduate methods education on the UW-Madison campus. We have hosted external Data Jams on the University of Washington Pullman campus and at the United Nation’s Office of Internal Oversight Services (OIOS).

The development of the Data Jam Initiative Tool Kit has been supported by an eXtension Fellowship. To access the curriculum, examples, videos and training materials, please visit the UW-Extension Data Jam website: http://fyi.uwex.edu/datajams/