When I was in college I started out at a community college then transferred to the University of Washington as a Junior majoring in Psychology. When I was at orientation, my academic advisor asked if I was interested in being in the honors program. That sounded fancy, so obviously I said “Yes.” At UW you do not have to find a mentor before entering the program, instead during the first quarter you take a class where many faculty come and present about their work and you can contact them to try to join their lab. While I was watching these presentations two labs stood out to me: 1) A social psychology lab working on understanding factors which keep women from joining and persisting in STEM fields, and 2) A quantitative psychology lab focusing on latent class analysis. Now you probably know that I am now a quantitative psychologist, so you might assume that I picked #2. But in fact I didn’t. Looking back now I realize that a major factor for me was that when I interviewed for the social psych lab, I was very attracted to the built in community. There were many other undergrads working in the lab, grad students, postdocs, etc. I didn’t have much of a community at UW having just transferred, so this appealed to me greatly. The quantitative lab, though more aligned with my interests, didn’t have that community. It was just a faculty and a grad student (no other undergrads). Ultimately, I chose the social lab (and wandered back to quant later). But in starting my own lab as a faculty member at UCLA, I wanted to make sure I never lost someone for the reasons that drove me away from quantitative psychology: community. For this, and many other reasons, I prioritized integrating undergraduate research assistants into my lab from my first year as a faculty mentor.
Many quantitative psychology labs do not have undergraduate researchers, or at least have very few. This is often because we don’t usually engage in our own data collection (the primary role of undergraduate researchers in other psychology labs) and the perception that what we do is too specialized for undergraduates to contribute. However, I’ve found this second point to be untrue, and given time and training undergraduate students have become excellent contributors to the research in my lab. My first year I hired two undergraduate researchers, my second year we had 4, my third year we had 5. This year we have 6 + I’ve hired an alumni part time as my lab manager. I think this is probably the upper end of what I can handle without some additional support (e.g., postdoc and advanced graduate students).
I was recently tagged in a twitter post asking how quantitative methods folks work with undergrads in our labs. I spent a 30-minute drive just talking at my phone, and below is a summary of those musings. I hope this is helpful to those of you looking for this kind of advice. I’m also happy to elaborate more on these things I mention. Please note that I by no means claim to be an excellent mentor, I’m learning just like we all are. Many of these things I’ve learned by trial and error, and I’m sure I will be learning more and changing my mind about these things as the years go on.
What kinds of projects have worked well with undergrads?
Systematic Literature Reviews
Project where you are interested in extracting systematic information from published literature seem to work really well for undergraduates. For example, projects where you’re trying to ask the question “What is common practice?” You have undergrads going into the literature. For example, you can identify set of papers that use a specific method. My lab focuses on mediation analysis. We can give the undergrads sets of papers that use mediation analysis and have them go in and find information about how that mediation was done: what kind of confidence interval did they use (percentile bootstrap CI or bias-corrected bootstrap CI)? Do they report the individual coefficients? Pull numeric information like sample sizes. With moderated mediation we’ve had students pulling information about where the mediation is moderated (which paths of the mediation are moderated).
Projects which used student coders:
Moderated Mediation Database: https://www.jlfossum.com/home/moderated-mediation-article-database
We had multiple undergraduate coders look at papers using moderated mediation and they pulled out information about the types of models used, estimation procedures, etc.
Census of Registered Report Journals: https://online.ucpress.edu/collabra/article/7/1/24404/117300/Opening-the-Door-to-Registered-Reports-Census-of
This project had two undergraduate coders who examined common practices for journals publishing registered reports. What are the elements of the journal submission policies?
Within-subjects mediation power: https://psyarxiv.com/gqryz/ (Appendix E)
In this paper I did a simulation study looking at within-subject mediation analysis. A reviewer questioned whether a specific correlation used as a simulation parameter was representative of common applications. An undergrad in my lab pulled statistical information from published examples to calculate an implied correlation and we examined whether the values used in the simulation corresponded to what was observed in published examples (they do).
Noise in the Process (https://psyarxiv.com/ck2r5/)
In this paper, undergraduate coders pulled information from published examples of mediation analysis in marketing and consumer behavior so that we could examine whether the confidence intervals were systematically “too close” to zero to be believable. RAs pulled numeric confidence interval information (values), level of the CI, type of CI, and other information.
Testing Software/R-packages/Shiny Apps/Macros
I have an undergrad work on testing for software. I taught her how to try different things with the macro. I give her sets of tests to work on, then meet with her weekly to go over the results of the test. I usually work on fixing any bugs that she has found, send her back a new version, rinse & repeat.
Typically, the tests are in two groups: 1) re-running tests that failed last time. Typically, I try to fix the bugs, and then the RA goes back and runs the tests again to make sure everything is fixed. 2) New functionality I’m adding. So typically, I’ll give her some parameters for testing new things that I’ve added to the tool. Initially, I just had her running tests and looking for egregious error messages, and over time I’ve also taught her how to test the actual numbers that are in the output and she checks those as well. Often, we do an initial run through looking for big errors and then if all goes well she’ll run a subset of those tests to check the numeric output.
Looking for example datasets
In quantitative methods papers we usually have an applied example. Typically, the way that I do that is to find a recent published example that could have used the method that I’m describing. I will give my research assistants certain parameters for papers to look for, then set them free in a specific direction. They identify specific papers, and then we meet and go over what they found. I typically have them look for more options than I need so that even if they’re a little off base, there will still be one or two good ones. Based on their set, if there are papers that I agree are good options. Then I have a template email that I use to contact authors to ask for their data. The RAs send this email from our lab email address to try to get access to the data. Usually I take over corresponding with the authors from there, but depending on how advanced the student is I may have them communicate with the authors to coordinate getting the data.
Students can also try out the applied examples. Often times you get an example dataset and there are a lot of different combinations of variables that they could use. Essentially, I teach them how to do the analysis using one example, then ask them to try a bunch of different combinations of variables. I try to tell them what I’m looking for in the example (e.g., I want a significant conditional indirect effect in one condition, but not the other, and a significant index of moderated mediation). So essentially by teaching them to do one analysis, they can go and do all the different permutations until you find something that works (this is essentially p-hacking for the purposes of pedagogical examples).
Other Kinds of Tasks
This quarter our lab is actually doing data collection for a registered report that we needed additional participants for. Honestly, other behavioral labs probably have much better systems than I have for doing this so I won’t elaborate on that.
Survey design. We do some data collection in the lab so I’ve had students work on setting up and testing online surveys.
IRBs. When you have a student who is already involved in a project, they can usually make a good first pass at an IRB (given some example documents). And I find it’s much easier to edit what they do than generate it from scratch. They also seem to really like this task.
Independent Projects
These projects are a little more involved, where a student is working on something on their own. This could be a complex data analysis or a Monte Carlo simulation or creating a function to do a certain kind of analysis. This is probably a more typical mentorship model where you’re meeting with them on a weekly basis, checking in on what they’ve done, identifying tasks for the next week, rinse, repeat.
I actually prefer to have students work on a topic that I’m not invested in happening quickly. It seems unreasonable to put time pressure on the undergrads. They are quite variable in how quickly they pick up these things. I want to give them space to learn and understand at their own speed. The projects I’ve had students work on are more fun wish list type topics, rather than things that are central to my research agenda. They are projects that are fun and interesting, things I like thinking about.
Some independent research topics:
Charlotte: How LASSO regression handles multicategorical predictors? (https://psyarxiv.com/wc45u/)
Zach: Is numeracy related to response styles (extreme and midpoint responding)? (https://youtu.be/iMNZYzkqh2c)
Kat: Can we use propensity score models to select participants during recruitment?
Valerie: Can we use random subsets of items in a scale to reduce carryover effects in within-subjects designs? (https://www.youtube.com/watch?v=TQqlhVADP90&feature=youtu.be)
What has worked well?
Redundancy is a good thing. It can be really beneficial, especially with the coding projects, to have students working on the same thing so they can be checked against each other. It also relieves some of the stress that students feel about having to “get it right” when there are mechanisms built in to check their work.
Weekly meetings. Meeting with students every other week means slow progress. I have tried meeting with students every other week, and often this is what I do with independent projects or things that I actually WANT to move slowly. The students have a pretty predictable pattern where they do the work they need to right before the deadline, and so if they don’t have a regular meeting they won’t do the work. Or if the deadline is every other week, they’ll only do work every other week. In general though, I have found that 30 minute meetings once a week will keep things moving much more quickly than 1 hour meetings every other week, so just depending on the rate of progress you’re looking for you might consider balancing length of time with frequency.
Small, accomplishable, concrete tasks. I’ve found that giving undergrads unstructured tasks is just not very productive. Often times this means that you have to sit down to really outline what it is that the student is going to do before meeting with them. They can really struggle when part of their task is figuring out what their task is. This may be a particularly important point for someone who has only mentored graduate students, where it’s much more normative to ask the student to structure their own work. With the undergrads more structure, concrete tasks, boxes they can check. This is really really helpful.
Digital Tools. I’ve used a Shared Drive on Google from the very beginning and I really like it for working on things in the lab. It makes it easy to transfer projects and documents to multiple people. Students can work in Google Sheets at the same time. Everyone has access to all the different files and folders. More recently, we’ve picked up using Slack. I’ve found this most helpful for the data collection teams we have in the lab, but in general I am really liking it. We have channels for different projects, which makes it really easy for students working on the same project to get in touch with each other, and it makes project-based communication really easy. We also have a lab server which we use for computationally intense projects (e.g., running simulation) and for shared software (e.g., RA testing with SPSS so I don’t buy personal licenses for them). It’s easy to remote in and we can have multiple people logged in at the same time.
Repeatable Tasks. This is something that’s really nice about the coding projects is often times we’re coding 100 or so papers. We’ll do 10 a week or 15 a week. Every week is the same. They’re doing the same thing over and over again. Ultimately, the training on a specific task is what takes a long time. With the coding stuff that we do, we usually do one meeting where we go over some examples that I’ve already pre-coded. Then we’ll give them some things to code that I’ve also coded, and then everyone codes everything that week. Then we come together, we talk about the disagreements. Sometimes we do that again, until the students are feeling pretty confident. I still build in the redundancy where we have multiple students coding multiple papers. Then we have this weekly meeting where we go over all of the disagreements. Often times if a task isn’t particularly repeatable, it’s not worth the time to train the RA to do it. So I try to focus on things that we’re going to need to do over and over.
Lab meeting. I have a weekly lab meeting with all the undergrads and graduate students. It’s pretty low-key. We start by going around the room and everyone gives “lab updates and life updates.” This can help with building some community within the lab, and also gives students exposure to the other projects going on in the lab that they might not be involved with. I do announcements about progress on projects (e.g., presentations at conferences, preprints, submissions, reviews, etc.). Every quarter we use the first meeting to decide what we’ll talk about at each meeting. I try to use this to give my grad students presentation experience, get feedback on paper drafts, give the undergrads opportunities to practice presentations. We also do some professional development: grad school applications, what is the review process of manuscripts like. We’ve also brought back to lab alumni to talk about their experiences working in data science. This year we’ll have some alumni come back to talk about starting PhD programs. In general, it’s a very low maintenance activity, but it’s really fun and it’s great to get everyone in a room together and build some comradery.
Purposefully Choose Projects. I’ve found myself using my time with the undergrads to do things like force myself to read a certain literature, where we all read the same papers. I’m doing this right now where we’re reading examples of papers that report about interactions, and myself, a graduate student, and an undergrad get together and discuss two papers a week. Recently a new paper came out about interactions (in AMPPS), so I actually paused the other work we were doing to read this paper as a group and discuss it. So, it actually forced me to read a paper that I wanted to read but may not have set aside time for otherwise. In some senses I use the time with the undergrads for me to prioritize things I might not otherwise prioritize. And it’s a good opportunity for them to read the extant literature on the topic that they’re exploring.
Developing Expertise. It's pretty impressive how much the RAs can learn when they’re regularly engaging with the same stuff. The student I have working on MEMORE now, Nickie, is phenomenal. She knows much more about this tool than any of my graduate student or my graduate mentor. If there is anyone else in the world who knows a lot about MEMORE, it’s her. She knows all of the inner workings, which is so cool! It’s fun to have someone who knows a lot about it and is curious about it. It’s fun for me too to have someone to talk to about these coding things. Often times when she finds an error, once I figure it out and fix it, I will explain to what was going on and how I fixed it. She’ll ask questions. So it’s a fun way for her to learn about the inner workings of this tool. And developing this expertise is what’s allowed her to start her independent project working on an extension of the models that MEMORE fits.
What has not worked well?
Constant Task Switching and Lack of Consistency. The students need to understand the bigger picture of what they’re doing. If what they’re doing is too rote or menial then it’s hard to keep motivation high. I use lab meeting to report back to them progress that is being made on papers they’ve contributed to. The first time I supervised an undergrad when I was in graduate school, I him in but I hadn’t really thought too hard about what I was going to have him do. Every week he was doing things that were slightly different, and he was never really seeing the fruits of his labor. He wasn’t really getting closure on the tasks that he helped with. After a semester he decided that he didn’t want to work in the lab anymore, which I think was totally fair. This was one of the points at which I made the decision that anyone who I work with who wants to work on an independent project can. One thing that’s nice is that the work in the lab is kind of variable. So, if there is a kind of a default of “oh if Amanda doesn’t give me anything this week, I’ll work on my independent project.” Then they still feel like they have a through-line and something they are contributing to. The independent projects can be completely removed from the other tasks that they do or it could be tightly tied in. For example, a student could be tasked with finding and writing up the example for a paper that they’re also doing the coding for.
Little structure or instruction. Early on when we were doing the first coding project, I didn’t tell the students how many journals to do in a week, because I didn’t feel like I knew exactly how much time each journal would take and I didn’t want them staying up all night everyday of the week to get this done. But I found then different students did different amounts, which was hard. But what I ended up doing was that I picked a number, and then I asked them at the meeting “How much time did you spend on each journal?” “How much time total?” And then from there I was able to adjust the amount to align with the number of hours they were supposed to be doing for credit.
No Documentation. Very early on I wasn’t really keeping codebooks. Then I listened to this episode of Within-and-Between. They talked about keeping codebooks. They said “Make a rule, write it down, follow it.” Doing this has really improved the process that I used, because in these meetings we would be resolving disagreements and sometimes we would forget which side of the fence we landed on, and then spend a bunch of time backtracking to figure out what we did last time. Now I keep a codebook that documents these decisions, and I make it accessible to the students. So, if later a student is coding and they run into something and think “Oh I remember we ran into this before, but I don’t remember what we decided.” Then they can refer to the codebook and look at what we decided, then make a decision consistent with our previous decisions.
Working in Isolation. For me I have a macro called MEMORE that is available for SPSS and SAS. For a while I had a student who was doing testing on this, and it didn’t work super well. Then I changed the way that I was doing this for the second RA I had and it’s been going much better. One of the things that I found was that I originally had an RA working on these tests, but I wasn’t actively working on the tool. I had recently done a release and I wasn’t really engaging with the code. So even when the RA found bugs, I didn’t fix them. Which was totally on me! I found myself thinking that once we found a certain number of errors that I would work on it, but we never really got past that threshold. I think this also led the RA to be not particularly engaged with the task because they weren’t seeing change, they were just running tons and tons of tests. I think this speaks to how there needs to be a regular exchange between what the RA is working on and what you are doing. Where it’s not super helpful to have an RA working on something that you aren’t also setting aside some time to work on.
So now what I’m doing is I’m working on the next version of the tool, and I have time set aside to work on the tool (MEMORE Monday). And every Monday when I’m done with development, I make a new testing version and I send it to my RA (actually I put it on Google Drive, I should probably use Github but here we are). Then between Monday and Thursday she works on tests that I have asked her to do. On Thursday, we meet and talk through the errors that she found, and following Monday I work on fixing errors we found and adding new functionality. So, we’re constantly iterating, and we both have specific time during the week that we work on this tool.
Grad Students Mentoring Undergrads
Recently I’ve been having some of the graduate students in my lab take on some of the mentorship roles in my lab. I think it’s a good idea to start this maybe part way through second year, but also depends on the grad student. This helps them gain some mentorship experience, and I gave them a lot of the advice that I’m giving here. This also potentially expands the size of the lab, because I’m not always able to meet with each undergrad every week, but I want them to be having regular meetings as part of our lab’s model.
An example of this is with my graduate student Jessica. For part of her first-year project, she was doing this big coding project where we wanted to explore what kinds of sample sizes researchers were using for moderated mediation analyses. Also which moderated mediation models were really popular? What we ended up doing is putting together a coding team. Jessica started out as just one of the coders, so she was doing the same thing the undergrads were doing each week: coding the articles, attending the meetings, etc. This helped her get familiar with the process. When we had the group meeting, I would lead that and be the executive decision maker. Often Jessica and I would talk about some of the behind the scenes stuff during our individual meetings, and we would debrief and check in so that she understood what I was doing in the background. Then the second quarter that we worked on the project, I attended a couple meetings, but she led them. Then I completely stepped away and Jessica stepped into that leadership role. This helped her see what the process was like before taking over, and it also helped us establish some norms before she started as a leader. This also made her feel more comfortable having done the coding before. One of the things that I find quite difficult about mentorship is giving someone instructions on how to do something I’ve never done, so finding ways to give the potential mentors experience doing the thing before they lead the thing seems really helpful. We added some new coders the second quarter, and Jessica was able to train them. So, working off an “I do, We do, You do” model seemed to work pretty well for transitioning the graduate students into mentorship roles. Now Jessica supervises an undergrad to continue this database even after we’ve completed the study.