Today on Teaching & Learning Champions, we’re joined by Ron Johns, Associate Dean, Assessment and Evaluation. We’re talking about institutional assessments and data around Guided Pathways.

Thanks for listening to TLC @ ACC!

Resources

Academic Outcomes Assessment website

Episode Transcript

[ Music ]

[Matthew Evins:] Welcome to another episode of Teaching and Learning Champions. I’m Matt Evins, Director of Instructional Technology and Digital Resources in the Teaching and Learning Excellence Division at ACC. Today I’m joined by Ron Johns, Associate Dean of Assessment and Evaluation as we talk about institutional assessments and data around guided pathways. Ron, thanks for joining me today.

[Ronald Johns:] Thank you, Matt.

[Matthew Evins:] Let’s get started for people who are not very familiar with you or the area that you’re in. Tell us a little bit about assessment and evaluation in institutional assessments. What are the goals and missions of your area for the college?

[Ronald Johns:] OK. So how much time do we have here?

[laughs]

[Ronald Johns:] No, seriously, what I see as the role of my area is one that is supportive of faculty, and their role is of course in helping our students to succeed. That’s the most fundamental level. The fact is obviously, we the faculty lay out and design a curriculum helping our students learn that curriculum, and helping achieve the learning outcomes that we set before them. And of course in doing so, and of course getting an education that helps them with their success in life. The trouble is that over the past few decades as there’s been increasing demands on pretty much all aspects of society, for documentation about what people are doing, and how things are justified, so on and so forth, there’s been an increasing scrutiny on what we [inaudible] at colleges and higher education in terms of helping our students succeed. And because of that, there’s an increasing demand for reporting, and matters of that nature, to various bodies. There’s the Southern Association of Colleges and Schools of course being one, but many other regional creditors and disciplinary creditors as well. So what I see as the primary mission or role of my area is to help faculty document the good work that they’re doing in a way that we can help produce some of these reports, and answer questions that those outside entities might have. But just as importantly, serve as a medium for self-reflection. To think about what is it that we are actually doing in the classroom. Is it what we — you know, are we achieving the results that we want? Are students learning what we want them to? And if the answer to those is no, then what can we do to perhaps improve that? I mean it’s pretty much a given I think that every faculty member here at ACC wants their students to succeed. You know we care about our students. We love teaching. I mean, that’s why we’re here. And it’s — but sometimes it’s good to just talk to our colleagues and our friends and sort of see if embody has a really good exercise that you know, helps students to learn a particular topic. You know, that’s something that should be shared. And while a lot of departments did have faculty meetings where they did have a lot of these sorts of discussions, discovered that you know, there were departments where that really was not the case. And I think as part of this process of review, this discipline assessment cycle as we call it, there’s just been a lot more self-reflection, a lot more discussion of these sorts of issues that I think has been very productive, and actually I think has helped move the needle in a lot of areas, on a lot of different topics. So I think that in itself has been helpful for our students, and helpful for us as faculty and employees here at the college. To really get a better focus on what we do and to show that work in a way that other people can see it.

[Matthew Evins:] So how do you — do you typically work directly with faculty? Or do you work more at the departmental or divisional level, or AOS level?

[Ronald Johns:] I work at whatever level I need to work at, which generally means a lot of — I spend a lot to time where I meet with faculty one-on-one. Oftentimes with a department chair, but other times with a whole department. And I’m happy to work with basically you know, anybody or anyone who wants to help. The — I mean there are many times where I end up working with other college administrators on things, like the master syllabus for example. But for the most part I work with faculty, department chairs, and deans probably more than others. But really I work with everybody across the college.

[Matthew Evins:] Well, it sounds like you provide a great resource for anybody looking to not only establish but enhance a lot of the evaluation metrics that they’re collecting for their courses, so.

[Ronald Johns:] Exactly. Yes, I mean, I’ve had to learn — I mean my background is in geology, so I had this sort of mid-career change to educational psychology and assessment, but — which has been interesting. But I mean I have found it very illuminating. I mean there’s so much good work going on here at ACC that you know, unless you really delve into each department you don’t see it. And I think one of the — I mean, I find it very rewarding just to see what other people are doing, and how they’re helping their students. And just you know, where I can, I submit you know, maybe help suggest you know, tweaking something here or there to try to focus more clearly on what it is that you know, they’re trying to measure. And get more actionable results. But really I — I really see my job mostly as just helping people to do something that they’re already doing, sometimes quite well.

[Matthew Evins:] So what types of data do you typically collect and analyze?

[Ronald Johns:] For the most part — well as part of the discipline assessment cycle, and for those who want to delve more deeply, the academic outcomes assessment website has a whole host of resources on the process, and various other helpful resources. Basically the data that we collect primarily is the extent to which students are achieving the learning outcomes that the faculty define for the disciplines, the programs that we offer. The assessment is focused at the program level. I mean we all do course assessments as well, within our courses. But for the most part, we’re focusing on the broader learning outcomes we expect of graduates of a program, whether that be like welding or philosophy, or whatever it happens to be. But things that we would expect graduates at the associate level at least to have achieved by the time they get that award. And so it’s mostly in those particular learning outcomes, and students’ achievement of them that we collect data that we look at.

[Matthew Evins:] What types of decisions regarding students’ success are influenced by the data collected and analyzed by your area?

[Ronald Johns:] I think for the most part, the decisions are the ones made by the department themselves. They — again I collect this data and document this so that we have a repository so we can you know, submit reports as needed. But primarily the — again the best benefit I think is the discussions dispersed among faculty about how they’re teaching the students, how they’re measuring their students’ success and their learning, and what can be done to improve that? It’s — so the decisions are generally, I think they’re the most influenced by this information, are the ones that are made by the department collectively, by the faculty looking at the data and saying, now our students are not getting this particular learning outcome, you know? Whatever it happens to be, as well as we would like them to. What can we do about it? You know, in my own field of geology, you know we teach — for example, topographic map reading. And that’s a subject that students often struggle with. And so you know, we talked about with each other a different exercises we have and exchange ideas and thing like that. So I think it’s a — it serves as a catalyst that’ll ferment discussions and departments you know, following actions then to help implement changes to help their students succeed.

[Matthew Evins:] Sure. Well, given that the theme this year for the college and you know, we typically pick up a lot of themes throughout the year, but one that’s been sort of continuous has been the guided pathways, which is the theme of this podcast. So how has the college’s adoption of guided pathways impacted the types of data, or even the results that you’re seeing that are collected by the departments, or by your area?

[Ronald Johns:] Well, one of the biggest themes, the guided pathways, is a focus on equity, which is also a focus of our strategic plan as a college, and so many of our other efforts here. The one that is obviously of great concern at the moment across the nation, as it rightly should be, the data that’s generally collected about student learning outcomes, for the most part it’s looked at in aggregate. And for many programs, especially those with relatively few number of majors, there’s really no other way you can look at it with any statistical validity. But you know, one of the important things that we need to do as an institution is look at this aggregate of the data, and see if there’s differences in performance between different demographic groups, different you know — by gender, or ethnicity, or health status, or any other parameter. And then try to figure out what we can do to try to provide support as needed. And try to help students who come to us with — you know, some situations and backgrounds that have been shall we say, less supportive than other students have had. And so I think it’s — moving forward, we’re trying to get more of a focus on that in the assessment process itself. There’s really been a key annoying — well a — there’s been a sea change, I would say in the assessment community in the past couple of years, regarding how equity figures into the whole assessment process. And the realization that some of the ways that we conduct assessments are just — it benefits certain groups more than others. People of certain backgrounds versus others. You know to use an extreme example if you’re asking you know, a physics question you know, about velocity and force or something, in the context of a yacht regatta, that’s not something a lot of people have any sort of cultural familiarity with. Whereas, you know, you could use a similar context to just say you know car, train, [inaudible] or something instead. So a lot of times it’s things that are relatively small that otherwise have an outsized impact on students’ performance. And it’s — with regards to assessment, it’s really just thinking about it, trying to recognize any sort of implicit biases, I mean implicit biases that frankly we all have in one way or another. And try to teach the subject in a more culturally responsive way, and likewise then assess the material in a more culturally responsive way as well. For example, portfolios are an excellent means of assessing student performance in a way that tends to be culturally responsive, because with a portfolio, students can produce work in a way that is most meaningful to them, in a way that is most — makes the most sense to them. But still be able to demonstrate their achievement of the learning outcomes, depending again on the discipline. You know, whether it’s art or [inaudible] for example, or things like that. I mean obviously, different disciplines, there are particular assessments that are most appropriate, but there should be at least a recognition that where possible, alternatives may exist, and that the way that we assess students is something that we have to seriously consider when dealing with these issues of equity.

[Matthew Evins:] OK. Great. Are there any data around guided pathways that you have collected so far? And I realize guided pathways is a newer adoption for the college. So that the answer to this question may be no, but is there any data that would be interesting or helpful for faulty to know in terms of the impact that faculty specifically are having on guided pathways?

[Ronald Johns:] In my area, not so much just yet. One of the things we’re trying to do — because again, to get at this [inaudible] aggregated data, you need large data sets. And at the level we’ve been operating for the most part, you know where a lot of programs have just a couple dozen majors at a time, in some cases. The — it’s hard to get large enough data sets to [inaudible] the data in any meaningful way. So one of the things we’re trying this year, and we’ll see how well it works, is that we’re sort of implementing this [inaudible] this year, is to collect student artifacts from across the college, and then have trained reviewers look at those and evaluate them. And in doing so — and doing this in the context of outside the class, this is called a faculty collaborative, because it’s totally faculty led and directed. The idea being that in doing so, we can have a sufficiently large data pool of student work that has been evaluated using a consistent metric, that we can then take that data, and then start breaking it out and look at it in smaller groups to see if there’s differences, specifically with regard to general education, like critical thinking and communication, things like that. And to date, we’ve not really done that. Because again, to date the processes that we’ve been using have been involving smaller groups. So it’ll be interesting to see what data comes out of that. And then of course, based on that information we as a college, kind of collectively, decide what the most appropriate courses of action would be to help provide support as it might be indicated. So at this point, that’s sort of a question that’s still very much up in the air.

[Matthew Evins:] Sure. Well as we sort of wind down, the last guided pathways-related question is, are there other projects or initiatives around your team’s responsibilities that would be beneficial for faculty and staff to know as it relates to guided pathways? Either things that you’re working on now, or into the future? I know you just talked about this new thing that you’re piloting now with data. But is there anything else coming up that you would want other people to know?

[Ronald Johns:] [inaudible] faculty collaborative I think is kind of the big thing at the moment. And again it’s — I want people to be aware of that, because it’s really going to require participation from faculty across the college who teach core curriculum courses. Those are the courses from which we’re going to be gathering the student artifacts. So we need them to go and identify an assignment, and basically every faculty member who teaches a core curriculum course, if they could identify just a single assignment even — or multiple would be ideal — even just a single assignment in their course each semester that would demonstrate for example, critical thinking or communication, written communication say, or teamwork. You know. Again, depending which Gen Ed competency we’re focused on that particular year. If they could just go ahead and tag it in Blackboard, which — I mean the actual process of tagging it takes literally two minutes, maybe. Then it’s set for the semester, and they would really need to do nothing else. That would just help us tremendously. And with one of the many things that of course has happened because of all the events this semester, is that there has been a tremendous shift to doing things on Blackboard, as [inaudible] or Blackboard person noted, we’ve moved about 5 years in two months with adoption of Blackboard and its various tools. So I think folks are a lot more familiar with Blackboard suddenly than they were 6 months ago. And I think that because you had used Blackboard for assignments now, there should actually be a whole lot more of that student work that’s been uploaded and in the system. So I think this — gathering this information is really just a matter of faculty’s spending as I say, literally a couple of minutes, going in and tagging artifacts. That would help move us along tremendously in terms of gathering the data and then you know, being able to interpret it and analyze it.

[Matthew Evins:] And is there information someplace, on your website perhaps, that would help faculty go through that process of tagging that information?

[Ronald Johns:] Yes. I’m glad you asked that. Yes again, the Academic Outcomes website

[laughs]. If you go to the search bar and just type in academic outcomes assessment, that’ll take you to the home page for my department. And from there, there’s a whole list of other links and so on and so forth, including a big page on equity in assessments, and how to participate in the faculty collaborative. Again, I think it’s heartening to see this increased focus on equity across higher ed, and indeed across society more broadly. And you know, we’re doing what we can to help address those issues here as well. So I think this is helping. I mean something as simple as tagging those artifacts can help us move the needle on a lot of those issues.

[Matthew Evins:] [inaudible] Well, thank you very much, Ron. Before I let you go, just one last question. Anything giving you Riverbat Pride this week?

[Ronald Johns:] Well, it’s been a troubled week obviously, across the country here. It’s — I mean I am very proud — what gives me Riverbat Pride is the knowledge that we work in — I mean, Austin in general, but Austin Community College in particular is one of the most important agents of change in our community. It helps to change people’s lives. And it helps to make a huge difference in the central Texas community. And I mean, like last week when they had the bank — getting — the food bank was distributing at the riverside. Those sorts of activities and initiatives I think, are just — they make me very glad to be a part of Austin Community College.

[Matthew Evins:] Yes, I wholeheartedly agree with that. I second that. Ron, thank you — thank you very much for joining me today. I really appreciate it. Lot of good information for all of our listeners. So thank you very much.

[Ronald Johns:] Right. Well thank you, Matt.

[Matthew Evins:] Well, that wraps up another episode of Teaching and Learning Champions. Don’t forget that you can view vlog posts for each episode on the TLED Website. I also encourage you to subscribe to the ACC District Podcasts on any of your preferred podcast apps, or listen to individual episodes on the TLED website. Thank you for tuning in, and we’ll chat next time on TLC at ACC.

[ Music ]