When Application and Methodological Innovation Collide: A Companion Blog to "Using Propensity Score Matching for Sequential Recruitment in Multigroup Cohort Studies"

Quantitative methodologists often say that their work is inspired through working directly with researchers who apply quantitative methods in the pursuit of their own research questions. Indeed, I’ve found that to be the case in my own work: By acting as a statistician for research projects, I see first-hand the needs of researchers in the social behavioral sciences, then these ideas go back to my lab, driving the next innovations that we focus on. However, in my experience it has been somewhat rare for the innovation and the applied work to occur hand-in-hand. Often researchers do not want to apply some new-fangled method to their own research, and if there isn’t something to cite to rationalize why this approach is a good idea, they are not open to using it. This paper is, I think, a rare example of a social-behavioral scientist taking a chance on an ambitious methodologist, and in the end something really interesting has come of it. The approach was originally designed as a tailor-made solution to a specific problem that my colleague Dr. Jennifer Sumner approached me about. But when a call for a special issue in Journal of Behavioral Medicine asked for methodological research, we thought it might be a good fit. You can read the preprint of the paper here: https://osf.io/preprints/osf/wvt8z_v2

In 2020, my colleague Dr. Jennifer Sumner, newly moved to UCLA, reached out to me about a study she was going to run and was funded through a NIH R01. The study aims to identify underlying mechanisms of the impact of PTSD on cardiovascular risk. Initially, I think, Dr. Sumner reached out to me because of my expertise in mediation, and the longitudinal nature of the study would necessitate some complex analytical models. But in speaking to Dr. Sumner there was also a more immediate and pressing problem: how are we going to recruit people with PTSD and people who experienced trauma but did not have PTSD (trauma-exposed controls), while keeping the groups relatively balanced on other characteristics known to relate to likelihood of developing PTSD following trauma exposure and cardiovascular health (e.g., age, gender). Dr. Sumner had been working with a statistician at her prior institution, and the initial plan was to test imbalance across important characteristics on a regular basis, then regulate recruitment accordingly. I had two concerns about using this approach: 1) It keeps the study right at the edge of imbalance, rather than aiming for perfect balance, and 2) doesn’t account for interactions across multiple variables. I thought we could do better, but that I needed some time to think through what was possible and do some additional reading. I was hoping I would find something in the literature to help me, but I was surprised to find it very difficult to identify methodological literature focused on study recruitment. The topic was almost non-existent.

My initial reaction was to do something with propensity scores, as this is a clean way to summarize across a large multivariate space. But propensity score matching would be incredibly restrictive, given that you would have to be finding the matches at the same time. From there the idea developed to include hold-out groups that would potentially be matches for future recruits, and I spent time thinking through how to do this without generating a lot of waste (e.g., participants who are ultimately not matched and must be excluded). I put together a proposal for what we might do, and found myself thinking “This is so crazy it just might work!” I was ecstatic when Dr. Sumner agreed to the idea!

Source: https://larspsyll.wordpress.com/2022/09/01/propensity-score-analysis-some-critical-remarks/

At the time, an eager undergraduate student had recently joined my lab and was looking for an independent project: Kathleen (Kat) Lamarque-Navarette. Kat wrote much of the initial code that we used for running this recruitment process with Dr. Sumner’s study, and conducted some tests of the method’s performance as part of her honors thesis. We had things pretty ready to start when COVID hit. Kathleen’s project shifted more hypothetical working with simulated data, as the COVID pandemic paused recruitment for this and many other studies around the globe. As is now an overtold story, we had no idea that things would last as long as they did. Kat transitioned from honor’s student to lab manager and eventually on to bigger and better things. When data collection resumed, the next lab manager took over the weekly task of running the matching algorithm and communicating back to Dr. Sumner’s team.

Kathleen Lamarque-Navarette graduated from UCLA in June 2021. Her honors thesis contributed greatly to the development of this paper!

In the summer of 2024, we finally collected the last participants we needed for the study, and as we were closing out the recruitment process we began evaluating the quality of the matches that we had. I was honestly, very shocked by how well this process had worked, how well-balanced the groups were across many different predictors. It was around this time we learned there was a call for papers at the Journal of Behavioral Medicine that was looking for methodological work. I had started talking to other colleagues about what we had done, and multiple expressed excitement about the idea. I truly had no idea that this kind of design was so common, and that this issue of recruitment was not just a niche thing for one study but something many other researchers dealt with. So, we put together this paper, describing the method, and generalizing the R code to something easy to use for others. I never expected this to be a paper in itself, but I’m very pleased at the possibility that someone might find this and it can help make the very onerous process of recruitment a little easier.

The preprint of our paper is available here: https://osf.io/preprints/osf/wvt8z_v2

Publication Timeline
Action	Journal	Date
Submitted Abstract for Special Issue Call	Journal of Behavioral Medicine	9/13/24
Invited for Special Issue Submission	Journal of Behavioral Medicine	9/20/24
Submitted full manuscript	Journal of Behavioral Medicine	02/24/25
Revise and Resubmit Decision	Journal of Behavioral Medicine	6/23/25
Submitted revision	Journal of Behavioral Medicine	8/12/25
Conditionally Accepted Pending Minor Revision	Journal of Behavioral Medicine	11/21/25
Submitted revision	Journal of Behavioral Medicine	12/12/25
Accept Decision	Journal of Behavioral Medicine	12/15/25

Journals Submitted to: 1

Months Under Review: 7