The Children's Trust aims to partner with providers to offer the highest quality services to improve the lives of children and families in our community, and also must be accountable to the public by demonstrating effective program results. It is critical for programs to include a strong evaluation plan for capturing successes in program planning, implementation and outcomes. An evaluation can encompass a wide variety of activities and services. Providers should be familiar with the tools available via SAMIS reports, trainings, and measurement/best practice toolkits provided by The Children's Trust and others. The Trust shares these resources to build evaluation capacity and improve programming; providers are encouraged to take advantage of these regardless of whether they utilize external evaluation services.
Funding requests for evaluation services are subject to approval by The Children's Trust evaluation staff. All Trust-funded providers are expected to develop internal program evaluation capacity to fulfill basic contractual data collection and reporting requirements and ensure continuous program quality improvement. The cost of evaluating the effectiveness of a program may include the purchase of measurement tools, as well as staff time for data collection, entry, management, analysis and reporting. Such costs will vary depending on the:
- primary purpose of the contract,
- program design and budget,
- sophistication of the evaluation methods, and
- specification of the final products.
The Children's Trust separately funds external evaluation of some of its major initiatives, and in these cases will not fund external evaluation services that would be duplicative of such efforts. In general, evaluation costs for a contract with a primary purpose of direct service delivery will not exceed 5% of the contract and may be considerably less. Contracting with an external evaluator is neither required nor prohibited, unless the use of the evaluator is a condition of the contract award. However, if external evaluation services will be sought, the following conditions must be met:
- primary contracted provider must have a clear and well-justified understanding of what is expected from the external evaluation, including the rationale for why such external services are more cost effective than using internal staff resources for this function;
- should not be duplicative of the standard tools and resources available through The Children's Trust SAMIS reports and calculations;
- a clear subcontract agreement must be specified (see below);
- evaluator should have experience with and work within a collaborative, participatory framework and actively involve program staff in the evaluation process to build the capacity of providers to use data for program improvement; and
- external evaluation resources should be used for functions that require specialized expertise rather than basic/routine tasks that internal staff can be trained to perform (ex., data collection of simple measures, data entry).
Contracts should be expected with all sub-contractors. The body of the contract should outline the scope of work that is agreed to by the agency and the evaluator. The Children's Trust research and evaluation staff must approve any evaluation consultant's scope of work/detailed evaluation plan prior to subcontract execution. Generally, The Trust will not accept simply a letter of agreement with an external evaluator. A more comprehensive contract or memorandum of understanding (MOU) must be established that details the work being contracted, including the following:
Fee Schedules - The costs of evaluating the effectiveness of the program should be included in the proposed budget and justification, and may include purchase of instruments as well as staff/evaluator time for the tasks detailed above. Providers are encouraged to perform some evaluation-related activities by internal staff to avoid higher evaluation costs (e.g., data entry and management).
Each item/category within the scope of work should be accompanied by the fee for that work. If a problem arises and an adjustment to the scope of work is required, knowing the costs associated with every phase/component of the evaluation will enable the agency to be better informed and to assess the modifications to the contract.
Deliverables - Every phase/component of the evaluation should include contracted deliverables. Deliverables for the planning/development phase may include: program curricula, tests, assessments, databases. Most of an evaluator's time in a project will come during the beginning and end of an evaluation. However, if the evaluator is doing ongoing training or technical assistance with program staff, then a deliverable may include an interim status report of their time/effort with the staff/program. If the evaluator is going to either assist the program staff or take the lead on reporting requirements to The Trust, then it is imperative to include contractually required analyses to inform key questions and evaluation reports.
Deliverable Due Dates - Just as each component or phase of the evaluation should include contracted deliverables, every deliverable should include a target due date. This is a critical part of the contract. Establishing deadlines will help the agency and evaluator to discuss each party's expectations including specific dates for:
- data submission to the evaluator for analysis,
- drafts of the evaluation report to the contracting agency,
- ongoing reviews and trending of data and discussion of uses for program and performance improvement between evaluator and program staff,
- analysis of outcomes for reporting to The Trust, if applicable, and
- the final report to the contracting agency.
The Children's Trust Policy on the Use of External Evaluators:
Considerations for Evaluation Sub-contractor Agreements