Heads you’re in; tails you’re out: How RCTs have evolved in DWP Jane Hall jane.hall@dwp.gsi.gov.uk Background  Increasing use of RCTs  Emphasis on evidence-based policy  Limited UK experience in social policy arena  Practical lessons not theoretical debate Overview       Chronology of RCTs in DWP Site selection and preparation Identifying the eligible population Dealing with resistance Performing the random assignment Monitoring take-up Chronology of RCTs        Restart Various New Deals Employment Zones JRRP ERA JSA Intervention Pilots ND50+ Mandatory IAP Site Selection and Preparation  Need commitment from the top  All parties need to buy-in  Set-up is resource intensive  Personal visits  Pilot the approach Identifying and recruiting the eligible population       Can they be easily identified Self-selection Suitability of the population Monitor P & C Group characteristics Selling techniques Sample sizes: Sub-group analysis Dealing with resistance  Busting the myth  Significant investment in training at all levels  Aides and FAQs  Scripts Performing the Random Allocation     Needs to be sophisticated Not open to sabotage/gaming Block allocation: Maintain P:C ratio Different techniques – NINO – Call Centre – On-line algorithm – Random numbers Monitoring take-up  Keep track of P & C Group  Ensure only P Group receive the treatment  Monitor key characteristics of P & C Group  Be prepared to redesign the random allocation Expect the Unexpected  Results may not be what you anticipate  A fair allocation of resources?  Participation rates can be disappointing Job Retention and Rehabilitation Pilot Lessons learnt in running an RCT James Holland Structure Background to JRRP Results of the trial Hypotheses Conclusions: Importance of complementary methodologies Design  - way trial  To test the effectiveness of a person centred case management approach and increased range of treatments in helping people retain work – – – – Health care focused Workplace focused Combined health care and/or workplace focused Control group  Four service providers in six parts of the country  Participants were people off work sick and unlikely to return to work without help Routes through the trial Contact Centre Project Marketing Approach Contact Centre Decline Explanation Ineligible Eligibility Screened out Screening Screened in: Randomisation Health 25% Work 25% Combined 25% Providers make contact Written Consent Assessment and Intervention Control 25% Return to work Out of work Surveys Evaluation Design  Impact and process evaluation  Cost benefit Analysis  Components of the evaluation – – – – – – Survey of those screened out and the control group Outcome survey Panel study Focus studies Database of contacts and treatments Costs exercise Impact Measures  Primary impact measure – 13 week return to work  Secondary impact measures – Health – Household income  Costs and benefits  Operation of JRRP as a RCT Results 13 Week Return to Work Intervention group % Health 43.5 Workplace 45.1 Combined 44.4 All interventions 44.4 Control 44.7 Results Week Return to Work Intervention group % Health 55.7 Workplace 56.4 Combined 56.5 All interventions 56.2 Control 53.0 Results Week Return to Work Intervention group % Health 61.5 Workplace 61.4 Combined 62.1 All interventions 61.7 Control 59.3 Results  Positive impact among those off work because of an injury  Negative impact among those off work because of a mental health condition  Unaffected return to work rates for those with other health conditions  Positive impact on health, particularly mild depression  A RCT can work in a voluntary labour market setting Hypotheses for findings The interventions were too weak The interventions were delivered in an unhelpful way Too many external barriers Withdrawal rate too high The self-selecting participants were the ‘wrong group’ Missing evidence  Evidence of problems mostly drawn from qual research  This generates hypotheses/explanations but does not allow for quantification  Biggest gap is (quant) understanding of behaviour of control group  In retrospect, needed data on self-motivation and better understanding of participant/provider interaction Thoughts on how to it better  Set out possible scenarios at start  Early qual research on behaviours of participants and control group  Early impact estimates so that later research can be adapted Conclusions  RCTs are the gold standard programme evaluation  But a number of problems, practically and methodologically  Need supplementing with a good quality process evaluation Contact Details James Holland Disability and Work Division, DWP James.holland1@dwp.gsi.gov.uk 0114 209 8280 Reports available via www.dwp.gsi.gov.uk/asd/asd5 ... policy arena  Practical lessons not theoretical debate Overview       Chronology of RCTs in DWP Site selection and preparation Identifying the eligible population Dealing with resistance Performing