• Determine goals and objectives for the evaluation
• Decide about issues of timing and expense
• Select who will conduct the evaluation
• Identify the audience(s) for the evaluation, i.e. who will read or use the evaluation findings

Type of project
For what type of project would you like to undertake an evaluation of participation? First of all you should think about the objectives and content of the project (e.g. create better stations, bring lines to life) and its implication for participation. Is it a huge or small project with regard to budget, relevance or image? How complex is the project and is it a short-term project or long-dated? How many stakeholders will be involved? Will there be new stakeholders that are not jet familiar with you and your organization? How should citizens be engaged? What are the fundholding organisation(s) requirements? Finally, how does the type of project work on the participatory approach and evaluation process?

The evaluator
Make yourself clear what are your tasks in the evaluation process and what´s the responsibility of your organisation in the regional rail system. Be aware that evaluation can be a complex process and that there are pitfalls. And maybe your evaluation reveals some critical aspects and problems you have to confront the stakeholders with. Finally you have to ensure that you have the personal skills and also resources to conduct an evaluation. If (stakeholder participation) evaluation is not a mandatory task, try to convince people that this is a worthwhile step to a better understanding of stakeholder participation and project outcomes.

Even if you are convinced of the benefits of evaluating, you potentially do not have the resources to conduct it. Furthermore, it can be quite complex to evaluate a whole programme or huge project (including the related participation processes), if you do not have the experience. This constellation is maybe appropriate to using an external evaluation expert that designs and conducts your evaluation. Maybe a funding organisation also requests that you work with experienced evaluation specialists. In this case, formulating clear objectives for the evaluation process is the key to success.

Participatory approach
Obviously, the evaluation of stakeholder participation is only reasonable if a participatory approach is implemented in the project strategy (see The Citizens’ Rail Flower Model). Regarding a typical project cycle you can in principle engage with stakeholders at every single stake. The participatory approach as a methodology covers the complete project and illustrates which stakeholders are why and at what stage engaged. How they are engaged can be best described which different participation forms. In literature and practice a nearly unlimited number of different participation forms or methods can be found which can´t be comprehensively tabulated and explained. In the list below you will find a selection of common participation forms. You can also find much more information about it on the internet (example 1, example 2 with toolbox).

We tried to arrange them after different participation levels. This allows you to merge a participation form with a participation level and a project stage. In addition, you can classify the strategies and tools with respect to adequacy based on various criteria, like visibility, intensity of engagement, notification, responsiveness, degree of communication, resourcefulness, user-friendliness, supporting one´s own initiative, accessibility or compliance¹⁷.

Some of the participation forms are useful for multiple purposes. E.g. you can use a workshop predominantly to answer questions to the public. But you can also design it as a think tank to capitalize citizens´ creativity. You can be creative by yourself and modify existing forms so that they meet your needs for the project (see for example section How and when can stakeholders be involved in your railway project?).

Participation forms
Source: Citizens’ Rail project team (based on Hillsborough County Metropolitan Planning Organization (2014) and Participation & Sustainable Development in Europe)

Read more about: Engagement with stakeholders for evaluation

• Determine focus of the evaluation in light of overall programme design and operation
• Develop appropriate research questions and measurable performance indicators based on
• programme goals and objectives
• Determine the appropriate evaluation design strategy
• Determine how to collect data based on needs and availability

Evaluation questions regarding participation
You should ask yourself if you will focus on evaluating the (intrinsic) mobilisation and participation of stakeholders (in particular citizens) or more on the effects and outcomes of participation with regards to the project content. Evaluation questions often also cover more complex and qualitative aspects, e.g. if a project met the overall needs or what worked well, what didn´t. “Overall, evaluation questions should lead to further action such as project improvement, project mainstreaming, or project redesign”¹⁸ of the current or a future project. Of course, this argumentation can be transferred to participation processes, as well.

Evaluation questions regarding participation
Source: Citizens’ Rail project team based on National Centre for Sustainability NCS & Swinburne University (2011), p. 3 (after Davidson/ Wehipeihana 2010) and DeGEval – Gesellschaft für Evaluation 2008, p.15

Evaluation types
Depending on your evaluation focus and the evaluation questions that you rise, you have to decide which type of evaluation you would like to conduct. Read the section “How to evaluate?” for more details about suitable evaluation types.

• Take steps necessary to collect high-quality data
• Conduct data entry or otherwise store data for analysis

Methods, indicators and data
Evaluation of stakeholder participation requires specific tools that heavily depend on the forms of participation that you have chosen in your project. Note that the evaluation itself may have a ‘participatory’ effect itself, e.g. by informing not yet reached stakeholder groups (see Section How and when can stakeholders be involved in your railway project?). In this sense, tools can either be understood as ‘participation’ as well as ‘evaluation’ instruments, e.g. feedback cards can engage citzizens’ during a workshop, but also can be used for workshop evaluation. The following table gives you a selective overview about methods and indicators that you can use for the evaluation of different participation tools and related goals or effects.

Participation tool, indicator and effect (examples)
Source: Citizens’ Rail project team

• Conduct analysis of data and interpret results in a way that is appropriate for the overall evaluation design (see Evaluation questions regarding participation table)

“Data analysis and interpretation can range from simple descriptive methods to highly complex statistical methods. The choice of analyses depends on the goals for the evaluation, the overall evaluation design, the type(s) of data collected, the interest of the evaluation audiences, and the timeline for completion of the evaluation.”¹⁶

• Decide what results need to be communicated
• Determine best methods for communicating results
• Prepare results in appropriate format
• Disseminate results

After completion of data analysis and interpretation it is decisive to present the result to the audience.

Possible results of from a meta-perspective could be:

• revealing of positive/ negative aspects of participation approach, e.g. satisfaction
• quality of participation approach
• suitability of participation approach regarding strategy/ goals
• amount of (un-) engaged stakeholders
• type of (un-) engaged stakeholders

Possible consequences, derived from result interpretation could be:

• adaption of participation form(s)
• change in intensity of participation (more/ less)
• adaption of resources (more/ less)
• change of stakeholder composition (e.g. including those not yet involved)
• adaption of evaluation approach (e.g. stronger process orientation)

What results need to be communicated depends also on the audience. First make clear who will be the recipient of you information. Content and style will differ, e.g. if you compare a presentation made for a funding organization to one made for citizens. Based on that issue you should decide for a method how to communicate the results. Maybe it is a classic presentation. But it you can also create a poster or write a report.

You should give the opportunity to the audience to provide feedback about the evaluation results and how they have been communicated.

Ideally conclusions of the evaluation results have to be drawn. You as project manager are responsible to discuss with your stakeholders what the consequences of evaluation are and if you rate participation successful. Maybe your evaluation yields lower satisfaction of participants, then you have to change your participation approach substantially. If you conduct a process evaluation, these conclusions have to be considered and implemented in the next project and evaluation steps. If you do a summative evaluation, then these lessons learned should be the point of departure for the next project or programme. Finally, think about adequate ways to record the comments and consequences, e.g. as written protocol or as comments complemented in your evaluation report.

¹ Outputs (e.g. ‘tangible’ factors ranging from plans and reports to new or improved lines and stations etc.) and Outcomes (e.g. ‘intangible’ factors such as improved relationship or development of trust between participants; learning effects) are criteria for the evaluation of participation processes. Outcomes are hard to measure and therefore the examples in the table include mainly outputs (based on ¹⁹).

¹ Outputs (e.g. ‘tangible’ factors ranging from plans and reports to new or improved lines and stations etc.) and Outcomes (e.g. ‘intangible’ factors such as improved relationship or development of trust between participants; learning effects) are criteria for the evaluation of participation processes. Outcomes are hard to measure and therefore the examples in the table include mainly outputs (based on ¹⁹).

Please note: Community involvement and engagement were an overall aim of the Citizens’ Rail project. This sections shows examples, where engagement itself was at the heart of the activities by using railway as an opportunity for peoples’ participation. Examples for participation in line and station projects are given above.

¹ Outputs (e.g. ‘tangible’ factors ranging from plans and reports to new or improved lines and stations etc.) and Outcomes (e.g. ‘intangible’ factors such as improved relationship or development of trust between participants; learning effects) are criteria for the evaluation of participation processes. Outcomes are hard to measure and therefore the examples in the table include mainly outputs (based on ¹⁹).