Making sense of the impact of Sense.info: roadmap towards appropriate evidence

Sense.info is a crucial element of the national e-health strategy and the website serves as the entrance point for youngsters in the Netherlands to prevention (and care if needed) regarding the whole spectrum of sexual and reproductive health. The website has been online since 2009, and the number of visits grew from 643,540 in the first year to 3,036,215 in 2019. Sense.info has also been included in the Dutch recognition system for health promotion interventions as ‘theoretically sound.’ The ambition of the developers of Sense.info, which was stimulated by the evaluation committee of the recognition system, is to gain insight into its effectiveness and to aim for a higher level of recognition. CHALLENGE The combination of being available for a long time and being accessible to everybody (making it hard to randomize people), and having a wide range of aims (making selection of a primary outcome inappropriate) leads to the conclusion that the Randomized Controlled Trial (RCT) as a method to study effectiveness does not fit this situation. At the same time, there is the ambition to gain insight into the potential effectiveness of Sense.info and the impact it has. This challenge is the point of departure of the proposed research. This fits like a glove to the current call focusing on research methods that have the potential to provide appropriate evidence for e-health in prevention. EVALUATION ROADMAP We propose an evaluation roadmap that combines various research methods to gain insight into the effectiveness of Sense.info. Sense.info should be seen as a ‘demonstration case’ for this evaluation roadmap. After this project, we have proof of principle for an evaluation roadmap that can be used across domains (i.e., beyond sexual health) and media (e.g., other websites, mobile applications). This is highly relevant, as many (non-profit) organizations within the field of prevention and beyond face a similar challenge. Many e-health products are available (and already used!), but there is limited insight into effectiveness. PROPOSED STUDIES To have proof of principle for this evaluation roadmap, we conduct two studies: the first study focusing on evaluating existing elements of Sense.info; the second study on adding new elements. Steps within the evaluation roadmap, which are used in these studies, are as follows: - STEP 1: The first step is to conduct a theoretical analysis of elements on a website and how they would contribute to effectiveness. In the case of Sense.info, preparatory work has been conducted when writing the application for acknowledgement within the Dutch recognition system for health promotion interventions. - STEP 2: The second step concerns gaining insight into use of the elements that are specified in Step 1. Use is crucial to achieve real-world impact, which is a function of both reach (e.g., to what extent does the target group use certain elements on a website?; Step 2) and efficacy (e.g., to what extent do these elements show potential under ideal controlled circumstances?; Step 3). Preparatory work has been conducted within the context of an IBM Corporate Social Responsibility grant and a consultancy services project. - STEP 3: The comprehensive picture resulting from Step 2 is the starting point of Step 3. It provides insight into where there is room for improvement - both with regard to elements itself as well as pathways towards other important elements. Two research methods are combined in Step 3: think aloud followed by in-depth interviewing. A heterogeneous sample of users is included in this step, also reflecting more vulnerable segments of the target group. - STEP 4: The second step results in more insight into _what_ elements to improve and the third step works towards _how_ to improve these elements. This optimization of elements is completed in Step 4 by combining the results of Step 3 with the use of design principles. More specifically, the focus is on interface design, navigation design, and information design, which are all part of a framework for user-centered design. COLLABORATION Rik Crutzen (Maastricht University; main applicant) has a strong track record in the field of e-health. In his work, he paid a lot of attention to the development of innovative methodology and he has ample experience in evaluating e-health interventions. He has previously cooperated with Soa Aids Nederland (more specifically, Hanneke Roosjen; co-applicant) regarding the evaluation of user statistics from Sense.info. The idea for the proposed project resulted from a joint ambition of both parties involved. Having the developer and owner of Sense.info on board as co-applicant is crucial for future implementation.