A Methodology for Discovering how to Adaptively Personalize to User Subgroups using Experimental Comparisons Joseph Jay Williams ([email protected], HarvardX, Harvard), Neil Heffernan (Worcester Polytechnic Institute). Paper is at www.josephjaywillams.com/mooclet

AdapComp Framework: Experimentally Compare Micro-Designs of Alternative Versions to discover rules for Adaptive Personalization

Abstract We explain and provide examples of a methodology for discovering how to adapt and personalize technology by using randomized experiments to compare alternative micro-designs for technology.

Maximize Target Variable: Response Rate from HarvardX students

The AdapComp Formalism [11] captures an equivalence between using technology to experiment with and personalize modular components of a technology. This provides a simple software design pattern for modular technology components that will allow the use of A/B experiments to discover how to adaptively personalize.

3 Versions of Subject Lines

Any modular technology component expressed in terms of the AdapComp Formalism can be optimized using a general-purpose algorithm that automatically trades off exploration (A/B Experimentation) and exploitation (adaptive personalization to help users).

AdapComp

Goals • •

Are there broadly applicable ways of using A/B experiments to discover rules for adaptively personalizing in real-time? How can live-deployed software be designed to reduce barriers to real-time adaptive personalization?

Contributions • •

Scalable Algorithm for using A/B Experimental Comparisons of different technology components to discover how to adaptively tailor these to user subgroups. Simple Software Design pattern that provides a unified approach to modifying, experimenting with, and adaptively personalizing technology components.

Experimental Variables/Versions 3 Subject Lines x 3 Intro Messages x 3 Response Formats = 27 Versions User Variables: Age Group = [18-22, 23-26, 27-35, >36] Number of Days Active = [0, 1, >2]

Algorithm transitions from A/B Experimentation to Adaptive Personalization: Modify Probability of Versions

ADAPCOMP Formalism User Variable Store

Delivery Rules

TARGET Variable EXPERIMENTAL Variables USER Characteristics

IF [CONDITION] = [A] then show V1 IF [AGE] = [18-22] then show V2 IF [ ] = [ ] then show [ ]

AdapComp Versions

V1

V2

VN

Proportion Students Responding to Emails

Ongoing AdapComp Experimentation & Adaptive Personalization