Eva Leslie 1*, Anthea Magarey 1, Timothy Olds 2, Julie Ratcliffe 3, Michelle Jones 4, Lynne Cobiac 1, 5
* Corresponding author: email@example.com
1 Nutrition and Dietetics, School of Health Sciences, Flinders University, Adelaide, Australia
2 Health and Use of Time Group, University of South Australia, Adelaide, Australia
3 Flinders Clinical Effectiveness, School of Medicine, Flinders University, Adelaide, Australia
4 Public Health and Clinical Systems, Department for Health and Ageing, Adelaide, Australia
5 Preventative Health National Research Flagship, CSIRO, Adelaide, Australia
Advances in Pediatric Research 2015, 2:23
Article type: Original Research
The electronic version of this article is the complete one and can be found online at: http://www.apr-journal.com/archives/494
Submission date: 4 March 2015
Acceptance date: 29 May 2015
Publication date: 23 September 2015
© 2015 Leslie et al.
This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
The complete article is available as a PDF file.
Background: The Obesity Prevention and Lifestyle (OPAL) intervention program targets families and communities to improve children’s eating and physical activity patterns. We outline the quantitative evaluation design and recruitment results for baseline data collection.
Methods: A longitudinal quasi-experimental design, with baseline data collection and five-year follow-up. Participants targeted are children, parents, and school principals/directors from primary, secondary/R-12 schools, pre-schools, childcare and out-of-school-hours-care (OSHC) centers in 20 selected communities across South Australia (SA), and one in the Northern Territory (NT). A total of 277 (262 SA, 15 NT) schools participated; 4860 9-11 year olds and 1164 12-16 year olds completed a questionnaire. Anthropometric measures were taken from 5531 students; 6552 parents, 276 pre/school/childcare directors, 139 OSHC directors and 237 principals completed questionnaires. Data include measurements of child participants’ weight/height/waist circumference; paper-based/online surveys of informants in early childhood, primary/secondary school and community settings; and secondary growth check data for 4-5 year old children. Serial cross-sectional analyses will compare intervention to matched comparison communities.
Results: Overall school response rate was 50%. Student response rates were 20-22% and 11-13% (questionnaires and measurements respectively); 14-21% of parents, 49-55% of directors, and 26-44% of principals completed and returned questionnaires. Changes to child weight status; eating practices; sleep, physical activity/sedentary behaviors; physical environments; community capacity; and economic evaluation (Quality Adjusted Life year gain) will examine program effectiveness.
Conclusions: As the most significant program of its kind in Australia, OPAL will contribute to obesity prevention efforts on an international scale.
With about one in four Australian children being overweight or obese, childhood overweight is a leading public health concern . Management of this problem requires multi-sectoral interventions that act along the continuum of care from primary prevention to tertiary management . Obesity prevention programs that create a supportive community and family environment, and that reach the range of settings in which children spend their time, are likely to be more sustainable and effective than single setting interventions . As a response, the South Australian government committed to the Obesity Prevention and Lifestyle (OPAL) intervention program, which is a complex, system-wide, community-based childhood obesity prevention program targeting 20 lower socio-economic status (SES) communities in South Australia, and one in the Northern Territory. It is modeled on EPODE (Ensemble, Prévenons l’Obésité des Enfants), a successful intervention originating in France [4, 5]. The overall aim of OPAL is to improve eating and physical activity patterns of South Australian children, through families and communities in selected regions, thereby increasing the proportion of 0-18 year olds within the healthy weight range.
The methodology for OPAL utilizes a mixed theory approach combining community development, social marketing, and ecological systems based on the four EPODE pillars of strong political support, independent scientific expertise, social marketing approach, and partnerships . This approach recognizes the impact of the family, school and wider community, as well as the social and built environment, on the development of children’s lifestyle behaviors . A quality evaluation is paramount to understanding the effects of any program, with three key aspects being specifically identified for community-based obesity prevention programs: the inclusion of comparison groups, objectively measured height and weight, and process and contextual information . Thus, a comprehensive and rigorous evaluation framework (including qualitative and quantitative methods) was developed for the OPAL program, including a financial commitment of 10-15% of the overall program budget for evaluation (MJ, unpublished results).
The aim of this paper is to describe the quantitative component of the OPAL evaluation. The evaluation objective is to determine the effectiveness of the five-year OPAL program underpinned by ecological systems theory, community development, and social marketing approaches, and to assess changes that have occurred within children, families, organizations, communities and environments. The primary outcome of interest is the change in the percentage of children in the healthy weight category, as defined by the International Obesity Task Force (IOTF) and World Health Organization (WHO) cut-points [8-11] after five years of implementation, in the intervention sites, relative to the matched comparison sites. The secondary outcomes (impact measures) are the changes in behaviors, attitudes, policies and environments related to healthy eating and physical activity.
The primary hypothesis is that prevalence of healthy weight in 0-18 year olds will increase in OPAL intervention communities compared with those in communities not receiving the intervention, resulting in improvements in health-related quality of life. A number of secondary hypotheses focus on intermediate outcomes (healthy eating and physical activity behaviors and environments). For example, in intervention communities, more children will have healthy eating and physical activity behaviors; home, school and other organization environments will be healthier; and levels of community capacity to support healthy eating and physical activity will be greater than in comparison communities.
The OPAL intervention program is based on the French EPODE methodology , which comprises political commitment, a scientific base, social marketing, and partnerships. The OPAL program consists of the delivery of annual social marketing theme messages and goal-related interventions to increase healthy eating, reduce the consumption of energy-dense, nutrient-poor food and drink, and increase physical activity and reduce sedentariness. The OPAL program is funded by three tiers of government: local, state and federal. The OPAL program is informed by a combination of three theoretical approaches to social change, as represented by the OPAL Program Logic Model (Fig. 1). The social marketing theory of behavior change supports behavior change in the individual and family. Community development theory of action and change engages at the community organizational/settings level. Finally ecological systems theory recognizes that action and change target multiple entry points, such that a program of this scope and complexity requires change at multiple levels across society; individual, family, organizational, community, environment, and political.
The quantitative component of the evaluation described here is a quasi-experimental design with initially grouped-matched comparison communities, and then 1:1 matches once all communities have been selected. Repeat cross-sectional analysis is undertaken to examine changes in the intervention communities compared with those communities not receiving the intervention over time. The majority of data are collected at two points (Years 1 and 5). The key outcomes to be evaluated are: 1) changes in the percentage of children within the healthy weight range and in related quality of life, and 2) changes in behaviors of children, parents, organizations, and communities, and in environments likely to impact on weight status.
The evaluation targets three broad sectors: early childhood settings, schools, and community and political stakeholders; and three age cohorts: early childhood (4-5 years); primary school (9-11 years); and secondary school (14-16 years). Survey data are obtained from practitioners (directors of long day care/pre-schools) and parents of 4-5 year olds; children aged 9-11 years and their parents, primary school principals, and primary school directors of out-of-school-hours care centers; youth aged 14-16 years, and secondary school principals. Children aged 9-11 years and youth aged 14-16 years are objectively measured for height, weight and waist circumference, while routinely collected height and weight data for 4-5-year-olds are obtained from the relevant authority. A summary of data collection components is included in Table 1.
Communities are the primary evaluation unit. The OPAL program has a staggered intake of metropolitan, rural and remote communities to reach a total of 20 South Australian communities by 2012, plus one Northern Territory community.
Recruitment of communities for the OPAL intervention occurred over four phases, with six communities commencing in 2009, four in 2010, six in 2011 (including the Northern Territory), and five communities in 2012. Each community will be supported for five years to implement the program. Thus, the combined intervention program will run for a period of eight years. The quantitative evaluation is planned to mirror this program as closely as possible. A list of OPAL communities is provided in Table 2.
Principles for the selection of intervention communities included selection of geographically contiguous suburbs with higher populations of children, higher populations of Aboriginal people, higher levels of disadvantage, and higher levels of childhood overweight and obesity. Community selections were also based on their local council’s readiness, including articulated commitment to health and well-being, and financial commitment to the OPAL program.
Thus, OPAL communities are defined as those communities with contracted political buy-in from local governments. Each community has two members of staff employed through the state health department (SA Health) located within local government and acting in a defined, bounded region (part or whole local government area; LGA).
These communities were matched as closely as possible with comparison communities for maternal education, geographical location (metropolitan versus rural), Index of Relative Social Disadvantage (IRSD; a measure of socio-economic status based on a basket of income and education-related measures), and population of 0-18 year olds. Matches avoided having intervention and comparison communities from within the same LGA. In 2009, OPAL communities were selected based on need and capacity. Between 2010 and 2012, OPAL communities were selected through an expression of interest process; hence no communities are randomly selected.
The sample size calculations are based on the primary outcome variable being the percentage of children falling into the healthy weight range (currently about 70%). As there are few prior EPODE-style interventions that are well documented, it is difficult to estimate the expected rate of change.
Therefore, a range of child overweight and obesity prevalence figures, including an EPODE-style intervention study , historical Australian data  and Queensland’s Towards Q2: Tomorrow’s Queensland  were considered. It was initially estimated that a sample size of 1500 children per phase (four phases) for each of the age groups (9-11 and 14-16 years) for both intervention and the comparison communities would be required to detect a meaningful change in the percentage of children within the healthy weight range (estimated at 2.5%). Based on these calculations (95% power) the aim was to have a total of 24,000 baseline and 24,000 fifth-year evaluation weight status measures on SA children aged 9-11 and 14-16 years at the completion of the study. However, due to a limited number of communities that could be matched, this was found to be an ambitious target as changes were required to the OPAL community selection process. The original plan to target communities of 20-30,000 was revised to include a larger number of smaller rural and remote South Australian communities (populations under 10,000). Regardless of these changes, a total sample of 10,800 children is sufficient to detect a ±2.5% change in the prevalence of children in the healthy weight range (power of 80%). For the 4-5 year age group, a sample of approximately 300 children per larger community was used, based on sample sizes gained from the ‘eat well be active’ community programs, a similar South Australian program that also accessed 4-5 year old weight status data . This would suggest that 12,000 measures would be accessed at baseline, and 12,000 at follow-up (intervention and comparison communities combined), with an estimated coverage of 70% of the 4-5 year olds across the State.
The OPAL Program Logic Model identifies key informants and key indicators (Fig. 1), with the primary outcome measure being the percentage of children in the healthy weight category defined by IOTF and WHO cut-points [8-11] and health-related quality of life. Because of the complexity involved in collecting information on the intended and unintended effects of OPAL on changes to behavior, environments and social norms, several sources of data are utilized for each cohort (Early Childhood, Primary School and Secondary School). Additionally, cross-sectional 4-5 year old growth data obtained from Child and Family Health Services (CaFHS) are analyzed at baseline, and at Years 3 and 5. Based on previous experiences from community-based obesity programs in schools , and in response to ethics requirements, a detailed consent process (described below) was developed to anticipate identified problems in recruitment and response rates.
Prior to sending information to directors of long day care, pre-schools and out-of-school-hours care (OSHC) providers, as well as school principals in intervention and comparison communities, a letter is sent to all principals with a joint introductory letter from the Ministers of Health and Education, and meetings with all regional directors conducted by evaluation staff. Directors and principals are sent a comprehensive information pack containing an invitation to participate in the evaluation via an information letter and brochure, checklist and participation form. A follow-up phone call to principals obtains verbal consent for the center/school to participate in the evaluation. Hard copy parent/guardian information packs are then sent to the school in envelopes for distribution to parents/guardians. Directors/principals are asked to assist in coordinating this process, including issue of reminders and collection of consent forms. Directors/principals are also asked to complete the center survey about their center/school’s healthy eating and physical activity policies, infrastructure and environment. Parents/guardians of 4-5 year olds and 9-11 year olds are asked to complete a paper or online parent/guardian survey about their child’s eating and activity patterns, and their home environment. The parents/guardians of 9-11 year olds and 14-16 year olds are asked to provide consent for their child to complete a paper or online student survey, and for their child’s height, weight and waist measurements to be taken. All children are also required to sign a section of the parent/guardian consent form requesting their assent to complete the student survey and/or be weighed and measured. Children in primary and secondary schools complete either paper or online surveys during normal class times. A summary of the survey recruitment process is provided in Fig. 2.
The height, weight, and waist circumference of each consenting child (9-11 years; 14-16 years) is measured in private by a trained female evaluation team staff member, unless the parent/guardian and/or child requests a male to do so. Each child is requested to remove their shoes and any heavy outer garments prior to measuring. Due to body image concerns, waist measurements are taken outside the shirt or blouse using a flexible tape, using standard procedures for anthropometric assessment . All measurements are obscured from the child’s view and recorded by the measurer without being discussed with the child, in line with the Body Image Guidelines developed and endorsed by the OPAL Scientific Advisory Committee. All results are kept confidential and only raw data recorded at the time of measurement; assessment of weight status is performed away from the school. The measuring team is trained by experts in body image, cultural sensitivities, mandatory reporting and anthropometry, as well as in how to respond appropriately and respectfully, without providing measurements, to anyone who requests the information. Training includes reliability and validity (inter and intra-tester error) and monitoring of techniques and interpersonal interaction. Weight status categories are determined from BMI (weight [kg]/height [m]2), using the IOTF age and sex-specific reference standards [8-11].
Questions relating to overweight and obesity, their known determinants and possible environmental correlates have been drawn where possible from existing instruments with either proven validity or reliability, or from those used previously in international or state surveys, in order to provide comparability with our evaluation results. These instruments are designed to explore actual behaviors, knowledge, and the social and physical environmental correlates and determinants of eating and activity behaviors, based on Bronfenbrenner’s model of behavioral determinants . Thus, impact measures relating to the OPAL program activities include changes in healthy eating practices (e.g. fruit and vegetable consumption) and intake of energy-dense food and drinks; changes in sleep, physical activity and sedentary practices; changes in health status and quality of life; changes in physical environments that can impact on healthy eating and physical activity; capacity and capability development in schools and communities; changes in skills, knowledge, behaviors and attitudes of stakeholders, organizations and communities that can impact on healthy eating and physical activity opportunities, environments and policies. All questionnaires are delivered in web or paper-based formats, dependent on the preference of, and facilities available in, the school. A summary of all data (survey domains and variables) is shown in Table 3.
Nutrition and eating behaviors are assessed through a series of questions in surveys that are addressed to the parents/guardians of 4-5 and 9-11 year old children and their parents/guardians; and 14-16 year old youth. There are also nutrition knowledge questions directed at out-of-school-hours carers and principals of schools. Generally, questions directed to children/youth relating to individual eating behavior refer to what was consumed ‘yesterday’ (as opposed to ‘usual’ eating patterns). Key items to be reported include the mean number of servings of fruit and vegetables consumed by children, and proportion of children meeting Australian fruit and vegetable guidelines for their age group, on the day prior to the survey being completed.
Physical activity and sedentary behaviors are assessed using overall levels of physical activity and screen time (television, videogames and computer use). Key items to be reported include the proportion of children participating in at least 60 minutes of moderate physical activity every day of the week, and the proportion meeting guidelines for physical activity and recommendations for sedentary behavior. Questions that relate to aspects of the eating and physical activity environment are directed mainly to parents/guardians (family and home environment), principals (school and built environment), and community stakeholders (community and built environment).
Key items include attitudes and knowledge around physical activity and fruit and vegetable guidelines, physical environments (such as items of play equipment in the home, or playing fields at school), and policy factors. Where possible the questions included are aligned with the planned goal areas and possible theme areas that were part of the program implementation.
A series of survey questions using the Child Health Utility 9D (CHU9D) instrument , specifically designed for application in economic evaluation to assess the cost effectiveness of childhood interventions are used to measure health-related quality of life in Years 1 and 5. The CHU9D instrument has undergone psychometric testing in both general school and clinical pediatric populations, and has demonstrated good practicality and validity [27, 28]. An Australian-specific adolescent general population tariff of utility values for the CHU9D [24, 26] is applied to estimate the total Quality Adjusted Life Year (QALY) gain for the intervention and comparison communities.
Community capacity building is a key component of the work occurring in OPAL communities. To gain insight into community capacity building over time, a modified version of the Public Health Agency Canada Community Capacity Building Tool , designed for participatory planning and reflecting on community capacity building journeys, is completed by OPAL stakeholders in each community setting at two time periods. This takes the form of a facilitated group discussion, structured using nine community capacity building features. This provides information on the contextual dynamics occurring within a local community and provides a quantitative assessment of community capacity over time; all nine features are scored numerically and then summed to obtain a total score. This is supplemented with qualitative information gained at time period 1 (baseline), which is the focus of reflection when the tool is reapplied at time period 2 (follow-up).
Data entry, handling and statistical analysis
The web-based survey responses are exported from a dedicated online platform and collated using SPSS, where data are checked for realistic values, duplicate and incomplete surveys, project allocated school and student IDs, and sample information. At this point the anthropometric measurement data for the students is added to the data files. Only de-identified SPSS output is delivered to researchers conducting the evaluation. Contact data is removed and kept in separate Excel files by SA Health to allow for potential data linkage between sampled parents and children, and other administrative datasets at a later stage. All data are analyzed using STATA version 11.0  to allow for the clustered sampling design. As a two-stage sampling design is employed, with children sampled from within schools, which in turn are sampled from each community, the probability of each school selection is adjusted according to a probability proportionate to size approach. The appropriate ‘svy’ STATA survey commands that account for the survey design are employed when analyzing all data. Evaluation of the OPAL outcome includes a descriptive analysis of dichotomous, categorical and continuous variables as repeated cross-sectional surveys. Appropriate descriptive statistical analysis, according to the form of data, is performed at baseline and endpoints of both the OPAL intervention and comparison communities. Percentage of 9-11 and 14-16 year old children in the healthy weight range is reported as cross-sectional data for baseline and follow-up, and the changes across the five years are also reported and analyzed for significance. Cross-sectional CaFHS 4-year old weight and height data is used to determine the percentage of 4-5 year old children in the healthy weight range at baseline, mid-point, and follow-up. Changes across the five years are reported and analyzed for significance. Comprehensive data management plans have been developed for the handling of all evaluation data, including strict protocols for data de-identification, privacy, data security, transfer and storage.
Endpoint data reporting includes the findings of the triangulation of the quantitative data, with any qualitative data, the effectiveness of the intervention through analyzed differences in the prevalence rates in outcome measures, analysis of any predictors of change through multilevel modeling or logistic regression (dichotomous variables), and the equivalent analysis of covariance for continuous variables. Data are analyzed by phase and by specific community where sufficient data are available, but not by school or school type (private/public). Mean costs and effectiveness to determine the differential economic impact between the OPAL intervention and comparison communities at five years will be compared, and incremental cost effectiveness ratios presented in terms of the cost per unit increase in the prevalence of healthy weight in children aged 4-5 years, 9-11 years and 14-16 years of age and the cost per QALY .
Consent and ethics
This study was conducted according to the guidelines laid down in the Declaration of Helsinki and all procedures involving human subjects were approved by the relevant human research ethics committees from the Flinders University Social and Behavioural Research Ethics Committee, SA Health Human Research Ethics Committee, Aboriginal Health Human Research Ethics Committee, Human Research Ethics Committee of NT Department of Health and Menzies School of Health Research and by the Department for Education and Child Development and SA Catholic Education.
All adults (parents/guardians, principals, center directors, key stakeholders) participating in the evaluation provide written informed consent. Parents/guardians provide written informed consent for their child’s participation, and the child also provides additional verbal consent prior to data collection. To acknowledge the contributions of the participants a series of small thank you gifts are offered. Ethics approval to access 4-5 year growth data (CaFHS data) was obtained from the Women’s and Children’s Health Network Human Research Ethics Committee. Annual progress reports will be provided to all ethics committees.
Consent and response rates
Of the 553 schools available in intervention and comparison communities, a total of 277 agreed to participate (50% acceptance rate). This comprised 262 schools in SA and 15 schools in the NT. Table 4 shows consent and completion rates for the student questionnaires and anthropometry, and the parent, principal, and director’s questionnaires. Consent forms returned from parents include forms where the parent did not consent to the student completing the questionnaire or measurement. The consent data for student anthropometric measures (9-11 years and 14-16 years) are identical to the student survey consent form data, as the number of consent forms returned where measurement ‘was refused, or not returned because the student was absent on the day, were not recorded. Table 4 combines data from participating schools. There were slight differences between age ranges in equivalent grades in SA and NT schools (NT grades include slightly older children than their counterparts in SA). As a broader age group was recruited in the NT (grades 4-6 in primary schools and 7-10 in secondary schools) the age bracket shown in the table has been expanded from 14-16 years (representing grades 9-10 in SA) to 12-16 years, to include the additional children captured in the NT. Table 5 shows the difference in overall response rate to questionnaires by students in participating schools by intervention and comparison communities. Overall response rates for participating schools were significantly higher in intervention communities (IC=60%, CC=43%; p less than 0.0001), with a marked differential response rate for all settings combined between intervention and comparison communities in the NT sample (75%; 33% respectively).
Program logic and duplicate checks were carried out on all survey data, with data checks for correct school/unique student ID, values, range and missing data. For anthropometric data (not reported) height, weight and waist circumference were determined as the mean of two measures, or the median where three measures were taken (i.e. if the difference between the first and second measurement exceeded the threshold, necessitating a third measurement; 10.5% of cases).
OPAL is the most significant program of its kind in South Australia, resourced by three levels of government over an extended period (up to eight years). The opportunity, and indeed the responsibility, of all governments is to evaluate such a large investment in public health. Obesity prevention programs are often complex, targeting individuals and communities simultaneously, and aiming to change lifestyle behaviors through changes to knowledge, social and built environments, and policy. The evaluation of the effectiveness of such programs requires multiple approaches, using quantitative and qualitative methods . The knowledge gained through OPAL’s quantitative evaluation will be critical to better understanding community-based obesity prevention programs, and will contribute to international understanding.
The planned quantitative evaluation of OPAL described here includes two of the three minimum requisites proposed by Swinburn and colleagues , with the inclusion of objective height and weight measurements in both intervention and comparison groups, and an analysis of outcomes by demographic variables. The third element, a description of the key intervention strategies and their intensity, is a focus of the qualitative evaluation to be described elsewhere. An additional component described in this paper is the impact evaluation of the overarching strategies of increased healthy eating, increased physical activity, and decreased sedentary activity, thus providing a more comprehensive and rigorous evaluation than has previously been undertaken in EPODE communities. The longitudinal quantitative component of the evaluation design for OPAL described in this paper is a key part of the overall evaluation design, which consists of several elements, including short, medium and long-term outcomes, economic, process, context, network, and global evaluations.
Despite the quality evaluation plan for assessing the effectiveness of OPAL, which has been devised and guided by experts in the field, and a strong partnership between the governments funding the evaluation and the research team commissioned for this work, there have been several challenges in the recruitment of participants for the quantitative evaluation to date. This is to be expected in long-term community-based interventions, which are subject to social, economic, political and even demographic changes that cannot be controlled. Notably, the original power calculations were based on the assumption that OPAL communities had a population of 20-30,000. Changes to the selection process of communities to be more inclusive of smaller rural and remote communities have resulted in a much smaller than anticipated population being available for sampling; subsequently smaller numbers of schools and children were included in the sampling frame. Other challenges include schools and principals being burdened with the demands of other research and activities generated through federal school programs (for example, “Building the Education Revolution”, a program of investment in school infrastructure in response to the global financial crisis), the introduction of a national curriculum framework, competing demands on academic curricula, and the amount of administration required in the student/parent recruitment process. This appeared to be more of a factor for secondary than primary schools (response rate 44% compared to 26%), with higher response rates observed for preschool, childcare center and OSHC directors (50%, 49%, 55% respectively).
Another key difficulty for the quantitative evaluation has been obtaining active consent from parents; we received a generally poor response to requests distributed via the student. The low non-consent rate suggests possible disinterest and/or unwillingness for parents to have their child participate; more probably, it suggests that the requests failed in many cases to even reach the parent. Additionally, there were issues around lead time required to successfully recruit schools to the evaluation, the timing of visits depending on school term dates and timetabling classes for administration of the student web-based surveys to fit with availability of computers and classrooms. The differential response rate between intervention and comparison communities observed (particularly in the NT) points to poorer engagement in comparison communities, with the fieldwork team reporting feedback that the parents in those communities did not value participation as they could see no direct benefit. Despite calls for opt-out consent to be used in the evaluation and monitoring of childhood obesity , this has not been possible in the current Australian ethics climate, or within the requirements of the National Health and Medical Research Council.
It should be noted that the planned OPAL quantitative evaluation, as outlined in this paper, has undergone considerable changes since it began. Because of some initial delays resulting from a complex tendering and contracting process, time taken to finalize the evaluation protocol and obtain relevant ethics permissions, the evaluation for Phases 1 and 2 did not commence until late October 2011 (planned for 2010). Early in the data collection phase it became apparent that some of the recruitment material was too complex, thus it was modified and made simpler, following ethics approval. Minor modifications were also made to the survey tool to ensure better quality data and ease of completion. Owing to perceived sensitivities around the terms “overweight” and “obesity” the material was amended to refer only to healthy weight. In addition, the scope and depth of the evaluation was reduced between baseline data collection for Phases 3 and 4 as a result of budget cuts from the funding body, as well as advice received from the OPAL Scientific Advisory Committee regarding the response rate and sample. This subsequently resulted in the removal of the 14-16 year old cohort (youth surveys and measures), all principal surveys, parent surveys for 4-5 year olds, and director surveys of early childhood centers from the sampling frame. Thus, the main outcome data (weight status) for follow-up will be for the 4-5 year and 9-11 year cohorts.
Despite the changes described to the original intended protocol the quantitative evaluation of OPAL remains an exemplar for future evaluations of obesity prevention programs, as it provides a more rigorous and comprehensive multi-disciplinary approach than have previous evaluations. The original EPODE program, on which OPAL is modeled, limited evaluation to assessment of weight status only , while OPAL will contribute other valuable survey data, and to knowledge of the feasibility of a more comprehensive evaluation underpinned by a rigorous evaluation framework. Previously, the current intervention landscape in Australia has not reported such a comprehensive approach to identifying effective approaches for obesity prevention . Several community-based programs included both weight and behavior outcomes but were limited in their target group [34, 35]. To date, the most comprehensively evaluated ram has been the ‘eat well be active’ Community Programs intervention in South Australia, but the full range of weight status, behaviors and relevant environments were only assessed for 9-11 year olds . While the OPAL evaluation continues to collect information on the home and community environments from the child and parent, trimming of the evaluation scope means there will now be gaps in understanding the school environment, especially from the perspective of those working in the school or organization (directors, principals). Indeed, it has been recognized that comprehensive evaluations of community-based interventions for obesity prevention are by nature challenging, both financially and technically . The united commitments made by governments in the case of the OPAL program, and the quantitative evaluation component, shows strong leadership in partnering with researchers to meet these challenges as best as possible. In future, a system of universal school-based surveillance would greatly facilitate the evaluation of community-level interventions aimed at children.
As the South Australian government health department owns the intellectual property for data collected for the OPAL evaluation, de-identified data will be made available to other researchers through application, in the hope that this valuable resource will be used more widely and further research into obesity prevention. Additional OPAL datasets are being collected, including qualitative interviews with stakeholders, and comprehensive program and process evaluation data in the Single Platform, which is an online data repository program planning and evaluation tool. As the data are re-identifiable, it is also hoped that a cohort dataset may be established with linkage of the 9-11 year old weight status data with their 4-5 year old weight status data.
Overall the OPAL evaluation has potential to provide more rigorous data, with higher quality and consistency, than has been possible in previous EPODE programs (where often it is the program worker who weighs and measures the children). In Australia there is no comprehensive school nurse system as in other European countries. The OPAL evaluation has been developed in parallel with the OPAL program, and has been responsive to programmatic changes. This is in line with a developmental evaluation approach. While every effort has been made to be responsive to program change, there have, however, been some limitations. For example, OPAL themes are selected and implemented on an annual basis, and while the evaluation has attempted to predict the behavior targets for this thematic approach, it is not possible to include specific questions for every theme in the questionnaires. However, it is worth noting that complementary Computer Assisted Telephone Interview surveys have been conducted annually to address this in the broader evaluation.
Early intervention in childhood at the population level is likely to have maximal impact, but the benefits are unlikely to be seen in the short term. This suggests the need for bi-partisan support for both interventions and their evaluations, and longer term follow-up that allows these effects to be detected. The evaluation of the OPAL program has provided a unique opportunity to determine the effectiveness of community-based childhood obesity prevention interventions for informing public health policy and practice.
Members of the South Australian Department of Health also contributed to the study design and to the preparation of the manuscript.
The authors wish to thank the members of the Flinders University OPAL Evaluation Project research team for their contributions to this paper, the field research team from Colmar Brunton for collecting the data, and the OPAL Scientific Advisory Committee for providing advice and guidance to the OPAL evaluation.
The International Obesity Task Force (IOTF) as described in this paper has been subsumed into the current World Obesity organisation [See http://www.worldobesity.org/what-we-do/]. Details of the history of the work conducted by IOTF for the WHO can be accessed at:
1. Olds T, Tomkinson G, Ferrar KE, Maher CA. Trends in the prevalence of childhood overweight and obesity in Australia between 1985 and 2008. Int J Obes (Lond). 2010;34:57-66.
2. World Health Organization. Obesity – preventing and managing the global epidemic. Geneva: WHO; 1998.
3. Magarey AM, Pettman TL, Wilson A, Mastersson N. Changes in primary school children’s behaviour, knowledge, attitudes and environments related to nutrition and physical activity. ISRN Obes. 2013;19:752081.
4. Borys JM, Le Bodo Y, Jebb SA, Seidell JC, Summerbell C, Richard D, et al. EPODE approach for childhood obesity prevention: methods, progress and international development. Obes Rev. 2012;13:299-315.
5. Romon M, Lommez A, Tafflet M, Basdevant A, Oppert JM, Bresson JL, et al. Downward trends in the prevalence of childhood overweight in the setting of 12-year school-and community-based programmes. Pub Health Nutr. 2008;12:1735-42.
6. Davison K, Campbell K. Opportunities to prevent obesity in children within families: an ecological approach. Crawford D, Jeffery RW, editors. In: Obesity prevention and public health. Oxford: Oxford University Press; 2006.
7. Swinburn B, Bell C, King L, Magarey A, O’Brien K, Waters E, et al. Obesity prevention programs demand high quality evaluations. Aust N Z J Public Health. 2007;31:305-7.
8. Cole TJ, Bellizzi MC, Flegal KM, Dietz WH. Establishing a standard definition for child overweight and obesity worldwide: international survey. BMJ. 2000;320:1240-3.
9. Cole TJ, Flegal KM, Nicholls D, Jackson AA. Body mass index cut‐offs to define thinness in children and adolescents: international survey. BMJ. 2007;335:194-202.
10. World Health Organization. Physical status: the use and interpretation of anthropometry. WHO Technical Report Series 854. Geneva: WHO; 1995.
11. World Health Organization. Obesity: preventing and managing the global epidemic. WHO Technical Report Series 894. Geneva: WHO; 2000.
12. Van Koperen TM, Jebb SA, Summerbell CD, Visscher TL, Romon M, Borys JM, et al. Characterizing the EPODE logic model: unraveling the past and informing the future. Obes Rev. 2013;14:162-170.
13. State of Queensland (Department of the Premier and Cabinet). Towards Q2: Tomorrow’s Queensland. Brisbane: Queensland Government; 2008.
14. Wilson AM, Magarey AM, Dolman J, Jones M, Mastersson N. The challenges of quantitative evaluation of a multi-setting, multi-strategy community-based childhood obesity prevention programme: lessons learnt from the eat well be active Community Programs in South Australia. Public Health Nutr. 2010;13:1262-1270.
15. Marfell-Jones M, Olds T, Stewart AD. International standards for anthropometric assessment. Potchesfstroom, South Africa: International Society for the Advancement of Kinanthropometry; 2006.
16. Bronfenbrenner U. Toward an experimental ecology of human development. Am Psychol. 1977;32:513-31.
17. Maclellan-Wright M, Anderson D, Barber S, Smith N, Cantin B, Felix R, et al. The development of measures of community capacity for community-based funding programs in Canada. Health Promot Int. 2007;22(4):299-306.
18. Morley M, Scully M, Niven P, Baur LA, Crawford D, Flood V, et al. Prevalence and socio-demographic distribution of eating, physical activity and sedentary behaviours among Australian adolescents. Health Promot J Austr. 2012;23:213-218.
19. Department of Health and Ageing. 2007 Australian National Children’s Nutrition and Physical Activity Survey Main Findings. Canberra: Department of Health and Ageing; 2008.
20. de Silva-Sanigorski A, Bolton K, Haby M, Kremer P, Gibbs L, Waters E, et al. Scaling up community-based obesity prevention in Australia: background and evaluation design of the Health Promoting Communities: Being Active Eating Well initiative. BMC Public Health. 2010;10:65.
21. Booth M, Denney-Wilson E, Okley AD, Hardy LL. Methods of the NSW Schools Physical Activity and Nutrition Survey (SPANS). J Sci Med Sport. 2005;8:284-293.
22. Wareham NJ, Jakes RW, Rennie KL, Mitchell J, Hennings S, Day NE. Validity and repeatability of the EPIC-Norfolk Physical Activity Questionnaire. Int J Epidemiol. 2002;31:168-74.
23. HBSC.org [Internet]. Health behavior in school-aged children. World Health Organization collaborative cross-national survey. Survey methods [accessed October 2014]. Available from:
24. Ratcliffe J, Couzner L, Flynn T, Sawyer M, Stevens K, Brazier J, et al. Valuing Child Health Utility 9D health states with a young adolescent sample: a feasibility study to compare best-worst scaling discrete-choice experiment, standard gamble and time trade-off methods. Appl Health Econ Health Policy. 2011;9:15-27.
25. Katzmarzyk PT, Barreira TV, Broyles ST, Champagne CM, Chaput JP, Fogelholm M, et al. The International Study of Childhood Obesity, Lifestyle and the Environment (ISCOLE): design and methods. BMC Public Health. 2013;13:900.
26. Ratcliffe J, Flynn T, Terlich F, Stevens K, Brazier J, Sawyer M. Developing adolescent specific health state values for economic evaluation: an application of profile case best worst scaling to the Child Health Utility-9D. Pharmacoeconomics. 2012;30,:713-27.
27. Stevens K. Developing a descriptive system for a new preference-based measure of health related quality of life for children. Qual Life Res. 2009;18:1105-1113.
28. Stevens K, Ratcliffe J. Measuring and valuing health benefits for economic evaluation in adolescence: an assessment of the practicality and validity of the Child Health Utility 9D in the Australian adolescent population. Value Health. 2012;15:1092-9.
29. StataCorp. Stata Statistical Software: Release 11. College Station, Texas, USA: StataCorp LP; 2009.
30. Glick H, Briggs A, Polsky D. Quantifying stochastic uncertainty and presenting results of cost effectiveness analyses. Expert Rev Pharmacoecon Outcomes Res. 2001;1:25-36.
31. Lobstein T, Baur L, Uauy R, IASO International Obesity Task Force. Obesity in children and young people: a crisis in public health. Obes Rev. 2004;5(Suppl 1):4-85.
32. Lacy KE, Kremer PJ, de Silva-Sanigorski AM, Allender S, Leslie E, Jones L, et al. The appropriateness of opt-out consent for monitoring childhood obesity in Australia. Pediatr Obes. 2012;7:e62-7.
33. Nichols MS, Reynolds RC, Waters E, Gill T, King L, Swinburn BA, et al. Community-based efforts to prevent obesity: Australia-wide survey of projects. Health Promot J Austr. 2013;24:111-117.
34. Sanigorski AM, Bell AC, Kremer PJ, Cuttler R, Swinburn BA. Reducing unhealthy weight gain in children through community capacity building: results of a quasi-experimental intervention program, Be Active Eat Well. Int J Obes (Lond). 2008;32:1060-1067.
35. de-Silva-Sanigorski AM, Bell AC, Kremer P, Nicols M, Crellin M, Smith M, et al. Reducing obesity in early childhood: results from Romp & Chomp, an Australian community-wide intervention program. Am J Clin Nutr. 2010;91:831-840.
36. Magarey AM, Daniels LA, Golley RK, Perry RA. Promoting healthy eating for children and families: practical approaches of three Australian intervention trials for prevention and management of overweight. Berhardt LV, editor. In: Advances in medicine and biology, Volume 31. New York: Nova Science Publishers; 2012. pp. 49-81.
37. Haby MM, Doherty R, Welch N, Mason V. Community-based interventions for obesity prevention: lessons learned by Australian policy-makers. BMC Res Notes. 2012;5:20.