Why is evaluation important?
It helps ensure the program is working properly. Eviction RTC evaluations provide data on what’s working, as well as what needs to change. Providers, administrators, and coalition members can also learn whether (and where) tenants are being reached, whether the funding being allocated is sufficient, the outcomes of the cases when tenants have representation, and whether critical changes need to be made or conversations had with specific stakeholders (ex: courts).
It helps build support for sustained and expanded funding: Eviction RTC programs often have to go through an annual budget process to be funded (absent where the RTC has a dedicated funding source). An evaluation can provide powerful data about the effectiveness of the program that can boost arguments for continued and increased funding.
It provides key information for the national eviction RTC movement. Evaluations can help other jurisdictions understand how eviction RTC policies work and whether such a policy would work for them. It can also provide data that can bolster the national RTC movement by showing these programs are effective. We track and collect evaluations of the existing eviction RTC programs, and rely on this information for our public awareness materials as well as when talking to media about the national scope of the movement.
What do evaluations look like?
An evaluation may be conducted by a legal services provider, a program coordinator, or an independent evaluator. It may be part of the annual report required by an eviction RTC ordinance or law. For example, New York City’s law requires the Office of Civil Justice (the administering entity) to publish public annual reports and hold public hearings about the reports. Cleveland’s law requires organizations to report quarterly data to their Lead Partner organization, and separate from this, the United Way of Greater Cleveland (the administering entity) has voluntarily engaged Stout as an independent evaluator.
What are some best practices for evaluation?
Prepare from the very beginning.
Planning the evaluation processes should be part of the eviction RTC coalition-building process and campaign. Once eviction RTC is enacted and legal services programs begin serving clients, the evaluation process needs to already be in place so that programs can begin collecting accurate data right away. Effective evaluations require extensive conversations and decisionmaking to:
Determine your goals and the measurement metrics;
Determine who will be conducting the evaluation;
Assess necessary changes to existing data-gathering systems and practices;
Build out collaborative practices and systems among legal services providers;
Build and maintain trust between administrators, tenant communities, and legal services providers to help create honest, effective focus groups and interviews of eviction RTC clients served.
Work closely with tenant leaders, tenant organizers, and community organizers.
Tenant leaders, tenant organizers, and community organizers should be considered vital players not just for enactment, but for implementation as well. Consider forming tenant focus groups that can advise on how well the program is going. Tenant groups may also be willing to do court watch in order to report back on how systems players (legal services providers, landlords, and courts) are treating RTC clients.
Work closely with legal services providers.
Evaluating RTC will require legal services providers to collect and provide data regarding specific metrics: types of cases covered, clients represented (numbers and demographics if possible), neighborhoods served, types of services provided, length of service, client goals, and outcomes achieved. Legal services providers have data collections systems and practices, but they may not be collecting information that can be shared out or the information you’ll need.
Gather information from a variety of sources.
An evaluation may incorporate data gathered from several sources: legal aid providers designated to provide services under eviction RTC, courts, eviction RTC coordinators or administrators, tenants, landlords, and community organizations designated to conduct eviction RTC outreach.
Determine who to use as the evaluator
Using a third-party / independent evaluator: Several existing eviction RTC programs have used outside evaluators to evaluate their eviction RTC programs. Cleveland and Connecticut have used Stout, LLC as an independent evaluator. Philadelphia has used the Reinvestment Fund. Using a third-party, neutral, and independent evaluator with experience can ensure a thorough evaluation that will be perceived to be providing an unbiased analysis, but it usually costs more than having the report done by the government or by the legal services providers.
Using a city or state administrator as evaluator: In some jurisdictions, the administrator or coordinator of the program is responsible for evaluating the program annually. Think carefully about whether the department coordinating or administering the program should be the evaluator. Some legal aid programs might be concerned if the administering agency or coordinating agency evaluating the program is staffed by people who might not have any experience in legal services provision.
Having legal services do reporting: In some jurisdictions, particularly those where the law does not require annual public reporting, legal services programs will provide a report to the government. Such reports are useful but may be perceived to be biased, and the legal service programs may not have the in-depth expertise of data analysis that would come with an outside evaluator.
Gather qualitative (interview) data from tenants who have had representation.
Tenants who have been represented by the eviction RTC program can provide insight into many aspects of implementation that providers and coordinators may not be aware of, including:
whether outreach and intake systems are effective and inclusive;
whether there are on-the-ground issues with how attorneys are handling cases or approaching clients;
whether there are issues with court processes, procedures, or practices;
whether clients have enough time to discuss cases with attorneys; and
whether or not the tenant would recommend the program through word of mouth (which can be a key outreach method).
For one program, interviews with tenants revealed that courts and landlord attorneys continued discouraging tenants from seeking out counsel despite eviction RTC, tenants would interact with numerous providers and have difficulty figuring out who they spoke with, some tenants were unable to identify the level of services they received, and tenants did not have enough time to communicate with their lawyers. This kind of data is critical in making adjustments to service delivery so as to carry out the goals of the program.
Evaluations should be public.
Publicly accessible reports, and public hearings, should be required in the ordinance or law. Of the existing RTC ordinances, many require an annual report to be made public as well as a public hearing to be held. Even where not required, public evaluations can help ensure funding and service transparency, keep providers and coordinators accountable, allow key stakeholders (particularly tenant leaders and organizers) to assess and provide input into the program’s impact and value, and highlight the successes of and delays in program implementation.