Evaluation of stakeholder participation


Why do it? – Purpose and Benefits
As you have already learned from the section Railway projects, participation and evaluation – How do they relate to each other?, participation processes are an essential part of modern railway projects and therefore evaluation of stakeholder participation should be seen as an obligatory approach within in the project design. Evaluation can be recognised as “a feedback mechanism for continuous improvement of your effort”12 to engage with stakeholders. Practically, evaluation intensifies the communication with your stakeholders, especially if you design and maybe conduct the evaluation together with them (so called participatory evaluation).

Finally, you will start a discussion about results and effects deriving from the evaluation and change strategies, goals and processes. A stakeholder evaluation can also have an impact on capacity-building. The insights you gain out of an evaluation process – the lessons learned – can strengthen the knowledge base and competences of individuals, organisations and systems. Therefore, the effectiveness and efficiency to define and achieve outcomes can grow in the future.

 

What is evaluation of stakeholder participation?
With regard to the diversity of evaluation subjects (see section “Understand the Background of Evaluation”) evaluation of stakeholder participation can be seen as a special evaluation type. In our understanding, evaluation of stakeholder participation particularly means the evaluation of stakeholder involvement or participation processes rather than its results. It is based on criteria derived from scientific theory or analysis of practical participation cases1. However it is possible to evaluate both processes and results of stakeholder participation by a combination of several methods and indicators. You should also pay attention to the ambiguity of the expression stakeholder and therefore different evaluation objectives and procedures.Some key principles in monitoring and evaluation of participation that you should keep in mind are:

Qualitative as well as quantitative Both dimension of participation must be included in the evaluation in order for the outcome to be fully understood.
Dynamic as opposed to static The evaluation of participation demands that the entire process over a period of time be evaluated and not merely a snapshot. Conventional ex post facto evaluation, therefore, will not be adequate.
Central importance of monitoring The evaluation of a process of participation is impossible without relevant and continual monitoring. Indeed monitoring is the key to the whole exercise and the only means by which the qualitative descriptions can be obtained to explain the process which has occurred.
Participatory evaluation In the entire evaluation process, the people involved in the project have a part to play; the people themselves will also have a voice.

Key principles in monitoring and evaluation of participation
Source: Government Social Research (GSR) Unit (2007), p.3 [15]

Read more


 

How to do it? – How can you apply evaluation of stakeholder participation for regional railway development?

An evaluation process has to be structured very well. Likewise, the evaluation of stakeholder participation has to consider the basic rules, standards and steps of evaluation that are described in the section “Understand the Background of Evaluation”. In this section, we present some specific methods and indicators that you can use for your evaluation of stakeholder participation. Of course, it is not a complete list and the choices depend on the levels and aims of participation (see section How and when can stakeholders be involved in your railway project?) which are relevant for your railway project. In addition, you also have to take care of some general aspects, which we present as well.

  • For what type of project would you like to undertake an evaluation of participation?
  • What are the objectives and content of the project (e.g. create better stations, bring lines to life)?
  • Is it a huge or small project with regard to budget, relevance or image?
  • How complex is the project and is it a short-term project or long-dated?
  • How should stakeholders and especially citizens be engaged?
  • How many stakeholders will be involved?
  • Will there be new stakeholders that are not yet familiar with you and your organization?
  • How should citizens be engaged?
  • What are the fundholding organisation(s) requirements?
  • Finally, how does the type of project work on the participatory approach and evaluation process?

In the remainder of this section we give you an overview about the workflow of participation evaluation (based on 16).

PRE-DESIGN, PLANNING AND PREPARATION

  • Determine goals and objectives for the evaluation
  • Decide about issues of timing and expense
  • Select who will conduct the evaluation
  • Identify the audience(s) for the evaluation, i.e. who will read or use the evaluation findings

Type of project
For what type of project would you like to undertake an evaluation of participation? First of all you should think about the objectives and content of the project (e.g. create better stations, bring lines to life) and its implication for participation. Is it a huge or small project with regard to budget, relevance or image? How complex is the project and is it a short-term project or long-dated? How many stakeholders will be involved? Will there be new stakeholders that are not jet familiar with you and your organization? How should citizens be engaged? What are the fundholding organisation(s) requirements? Finally, how does the type of project work on the participatory approach and evaluation process?

The evaluator
Make yourself clear what are your tasks in the evaluation process and what´s the responsibility of your organisation in the regional rail system. Be aware that evaluation can be a complex process and that there are pitfalls. And maybe your evaluation reveals some critical aspects and problems you have to confront the stakeholders with. Finally you have to ensure that you have the personal skills and also resources to conduct an evaluation. If (stakeholder participation) evaluation is not a mandatory task, try to convince people that this is a worthwhile step to a better understanding of stakeholder participation and project outcomes.

Even if you are convinced of the benefits of evaluating, you potentially do not have the resources to conduct it. Furthermore, it can be quite complex to evaluate a whole programme or huge project (including the related participation processes), if you do not have the experience. This constellation is maybe appropriate to using an external evaluation expert that designs and conducts your evaluation. Maybe a funding organisation also requests that you work with experienced evaluation specialists. In this case, formulating clear objectives for the evaluation process is the key to success.

Participatory approach
Obviously, the evaluation of stakeholder participation is only reasonable if a participatory approach is implemented in the project strategy (see The Citizens’ Rail Flower Model). Regarding a typical project cycle you can in principle engage with stakeholders at every single stake. The participatory approach as a methodology covers the complete project and illustrates which stakeholders are why and at what stage engaged. How they are engaged can be best described which different participation forms. In literature and practice a nearly unlimited number of different participation forms or methods can be found which can´t be comprehensively tabulated and explained. In the list below you will find a selection of common participation forms. You can also find much more information about it on the internet (example 1, example 2 with toolbox).

We tried to arrange them after different participation levels. This allows you to merge a participation form with a participation level and a project stage. In addition, you can classify the strategies and tools with respect to adequacy based on various criteria, like visibility, intensity of engagement, notification, responsiveness, degree of communication, resourcefulness, user-friendliness, supporting one´s own initiative, accessibility or compliance17.

Some of the participation forms are useful for multiple purposes. E.g. you can use a workshop predominantly to answer questions to the public. But you can also design it as a think tank to capitalize citizens´ creativity. You can be creative by yourself and modify existing forms so that they meet your needs for the project (see for example section How and when can stakeholders be involved in your railway project?).

Participation form Participation method
Public information Letter, posters, notices, signs Community news
Leaflet, brochure Banners
Newsletter Maps and plans
Broadcasting Celebrity media
Access to planning documents How to engage citizens´ guide
Information hotline Advertisement
Media kits, information packages
Internet & ICT Webpage Transportation visioning tools
Social Media Internet clips (e.g. on YouTube)
Web based forum Mobile exhibits (portable, stationary or interactive display of project background and current information)
App Mobile devices (messaging, email blasts, twitter)
E-Participation Online communities (e.g. on Facebook)
Online collaboration tools
Surveying Questionnaire surveys printed Comment cards (e.g. at the end of an event)
Questionnaire surveys online Key person interviews
Information events Information centre Business briefing
Exhibition Contests and games
Information session and briefing Field review
Public meetings Open house
Topical events Participatory Budgeting
Engagement Community visits Advisory committees
Focus groups Charrettes
Workshops Discussion
Citizen juries Technical working party
Open database (participants, stakeholders, agencies information) ‘Home games‘ (at home’ workshops for neighborhood or other stakeholder groups)
Stakeholder conference Scenario workshop
Transport visioning event Walkshop (mobile wokshop)
Planning for Real Open space event
Rail user groups
Empowerment and practical work friends of station groups Community outreach ambassadors
Volunteers station adopter schemes
Projects with schools, local business … about gardening, station improvement …

Participation forms
Source: Citizens’ Rail project team (based on Hillsborough County Metropolitan Planning Organization (2014) and Participation & Sustainable Development in Europe)

Read more about: Engagement with stakeholders for evaluation

EVALUATION DESIGN

  • Determine focus of the evaluation in light of overall programme design and operation
  • Develop appropriate research questions and measurable performance indicators based on
  • programme goals and objectives
  • Determine the appropriate evaluation design strategy
  • Determine how to collect data based on needs and availability

Evaluation questions regarding participation
You should ask yourself if you will focus on evaluating the (intrinsic) mobilisation and participation of stakeholders (in particular citizens) or more on the effects and outcomes of participation with regards to the project content. Evaluation questions often also cover more complex and qualitative aspects, e.g. if a project met the overall needs or what worked well, what didn´t. “Overall, evaluation questions should lead to further action such as project improvement, project mainstreaming, or project redesign”18 of the current or a future project. Of course, this argumentation can be transferred to participation processes, as well.


Evaluation questions regarding participation
Source: Citizens’ Rail project team based on National Centre for Sustainability NCS & Swinburne University (2011), p. 3 (after Davidson/ Wehipeihana 2010) and DeGEval – Gesellschaft für Evaluation 2008, p.15

Evaluation types
Depending on your evaluation focus and the evaluation questions that you rise, you have to decide which type of evaluation you would like to conduct. Read the section “How to evaluate?” for more details about suitable evaluation types.

EVALUATION IMPLEMENTATION

  • Take steps necessary to collect high-quality data
  • Conduct data entry or otherwise store data for analysis

Methods, indicators and data
Evaluation of stakeholder participation requires specific tools that heavily depend on the forms of participation that you have chosen in your project. Note that the evaluation itself may have a ‘participatory’ effect itself, e.g. by informing not yet reached stakeholder groups (see Section How and when can stakeholders be involved in your railway project?). In this sense, tools can either be understood as ‘participation’ as well as ‘evaluation’ instruments, e.g. feedback cards can engage citzizens’ during a workshop, but also can be used for workshop evaluation. The following table gives you a selective overview about methods and indicators that you can use for the evaluation of different participation tools and related goals or effects.

Tool Goals, Effects Indicator Analysis Method
Leaflets, brochures
  • Dissemination of information
  • Awareness raising
  • Level of knowledge
  • No. of printed documents
  • No. of distributed documents
  • Media monitoring & analysis
  • Visits to any specific webpages promoted
Webpage
  • Dissemination of information
  • Awareness raising
  • Level of knowledge
  • No. of webpage visits/visitors (web counter)
  • No., content and quality of feedback (comment field)
  • (online) media monitoring & analysis
  • Big data analysis/ web-statistics etc.
Comment/ Feedback cards
  • Immediate ‘evaluation’ of events
  • Suggestions for further improvements
  • Feeling of value and importance
  • No., content and quality of comments
  • Quantitative and qualitative methods of empirical social research (statistical analysis, transcription and interpretation)
Questionnaire
  • Systematic information about a specific topic
  • Reach not yet connected citizens
  • Feeling of value and importance
  • No. of participants/completed questionnaires
  • Content and quality of responses, response behavior
  • Quantitative and qualitative methods of empirical social research (statistical analysis, transcription and interpretation)
Workshop
  • Greater awareness and understanding of the issues
  • Diversity of information and alternatives considered
  • New insights for planner
  • Improved public image of inviting organisation
  • No. of held workshops
  • No. of workshop participants
  • No., content and quality of statements/interventions
  • Process review
Community Forums
  • Increase possibility to influence decision making
  • New contacts/ given access to new networks
  • Balanced opinion-forming process
  • More confidence and willingness to get involved in future
  • Capacity building
  • Esteem
  • No. of events
  • No. of participants
  • No., content and quality of statements/interventions
  • No. content and quality of new ideas and proposals
  • Satisfaction of participants
  • critical discourse analysis
  • Participatory evaluation/ monitoring/ Reflexive Monitoring in Action
  • Social network analysis
Volunteers
  • Specific creative improvement measures
  • Creative new ideas
  • High identification
  • Increased social capital on community level
  • No. of actions
  • No. of involved volunteers
  • Quantity (e.g. working hours) of voluntary service
  • Participatory evaluation/ monitoring/ Reflexive Monitoring in Action

Participation tool, indicator and effect (examples)
Source: Citizens’ Rail project team


DATA ANALYSIS AND INTERPRETATION

“Data analysis and interpretation can range from simple descriptive methods to highly complex statistical methods. The choice of analyses depends on the goals for the evaluation, the overall evaluation design, the type(s) of data collected, the interest of the evaluation audiences, and the timeline for completion of the evaluation.”16

WRITING AND DISTRIBUTING RESULTS
  • Decide what results need to be communicated
  • Determine best methods for communicating results
  • Prepare results in appropriate format
  • Disseminate results

After completion of data analysis and interpretation it is decisive to present the result to the audience.

Possible results of from a meta-perspective could be:

  • revealing of positive/ negative aspects of participation approach, e.g. satisfaction
  • quality of participation approach
  • suitability of participation approach regarding strategy/ goals
  • amount of (un-) engaged stakeholders
  • type of (un-) engaged stakeholders

Possible consequences, derived from result interpretation could be:

  • adaption of participation form(s)
  • change in intensity of participation (more/ less)
  • adaption of resources (more/ less)
  • change of stakeholder composition (e.g. including those not yet involved)
  • adaption of evaluation approach (e.g. stronger process orientation)

What results need to be communicated depends also on the audience. First make clear who will be the recipient of you information. Content and style will differ, e.g. if you compare a presentation made for a funding organization to one made for citizens. Based on that issue you should decide for a method how to communicate the results. Maybe it is a classic presentation. But it you can also create a poster or write a report.

You should give the opportunity to the audience to provide feedback about the evaluation results and how they have been communicated.

Ideally conclusions of the evaluation results have to be drawn. You as project manager are responsible to discuss with your stakeholders what the consequences of evaluation are and if you rate participation successful. Maybe your evaluation yields lower satisfaction of participants, then you have to change your participation approach substantially. If you conduct a process evaluation, these conclusions have to be considered and implemented in the next project and evaluation steps. If you do a summative evaluation, then these lessons learned should be the point of departure for the next project or programme. Finally, think about adequate ways to record the comments and consequences, e.g. as written protocol or as comments complemented in your evaluation report.



 

Examples – Results from Citizens’ Rail projects
This section gives you an overview about a possible structure of analysis which is adapted from academic literature19. For inspiration, it presents selected results from the Citizens’ Rail project in different partner regions in a schematic way by delivering information about the following items:

  • Project
  • Region
  • Objectives
  • Participatory Design
  • Stakeholders involved
  • Outputs
  • Outcomes

Just click on the relevant “+” button to read more.

STATIONS

Project Region Objectives Participatory Design Stakeholders involved Outputs / Outcomes1
Loire rural stations Pays de
La Loire
exploring opportunities for new uses of seven (rural) station buildings which are disused or vacant (e.g. after closure of ticket offices) to safeguard the heritage by transforming them into lively, thriving community hubs (1) technical and ‘social’ station assessment
(2) presentation of findings to local decision makers
(3) audit of local stakeholders
(4) brainstorming / Workshop sessions with a wide range of local stakeholders
(5) evaluation of local needs and opportunities by analysing results of audits, workshops etc.
– councils
– elected officials (e.g. mayors)
– local residents
– local NGOs
– business figures
– consultants (external)
– new ways of discussion
– emergence of creative collective ideas
– installation of bicycle parking at several stations
Read more
Burnley Manchester Road Lancashire – replace old by a new station building (modular design)
– provide (community) services
– provide an attractive and iconic gateway to the town
creation of a community-focused building (e.g. room for Community Rail Lancashire); Community Rail Partnership officers provided input to the design for the specification and layout of the community room Community Rail Partnership officers – more passengers
– increased customer satisfaction
Read research findings / Read more
Community forums Aachen and Parkstad Limburg – getting station plans that are broader, more imaginative and of higher quality
– allowing stakeholders to participate on equal terms to help define problems, goals and plans
community forums or workshops, student masterclasses and surveys as well as local project groups and other forms such as Dear Hunting or youth programmes were used for early consultation right from the start of the planning process of station improvements or new stations over all cases:
– local residents
– local business
– city administration
– University students;
– University Staff
– regional authorities
– transport authorities
examples:
– decision for station location
– revealing needs and priorities
– suggestions for adaptions and changes
– (Station) Action Plans

Findings of student survey

More information

1 Outputs (e.g. ‘tangible’ factors ranging from plans and reports to new or improved lines and stations etc.) and Outcomes (e.g. ‘intangible’ factors such as improved relationship or development of trust between participants; learning effects) are criteria for the evaluation of participation processes. Outcomes are hard to measure and therefore the examples in the table include mainly outputs (based on 19).

LINES

Project Region Objectives Participatory Design Stakeholders involved Outputs/
Beach train / La Roche-sur-Yon line (FR) Pays de
La Loire
– revitalising the line between La Roche-sur-Yon and Saumur
– improve infrequent services and low passenger numbers
community engagement as part of a four pillared strategy consisting of:
– additional services
– marketing, campaigns
– community engagement
– station improvementscommunity engagement to assess how the line can best serve local residents, visitors and the area’s economy
– school authority
– members of line committee
– regional authority
– train operator
– infrastructure owner
– trade unions
– 5,300 extra journeys in 2013
– 90% satisfaction rating from passengers
– 70,000 flyers distributed
– 1 in 3 people found out via word of mouth
– extended service season in 2014 and 2015
– template for ‘castle train’Read more on website and in toolkit
Riviera Line (UK) Devon/Torbay – bring under-used station buildings to life
– encourage more people to take the train
– achieve the same success as other rejuvenated “branch lines” in Devon and Cornwall (115% growth in passenger numbers between 2003 and 2013)
broad range of approaches alongside the line:
– volunteering opportunities,
– passenger feedback
– Community Forum
– Information campaigns
– School excursions
e.g.
– Devon & Cornwall Rail Partnership
– local rail company (First Great Western)
– volunteers
– more frequent services
– station improvements (e.g. gardening projects, new shelters)Read more on website and in toolkit

1 Outputs (e.g. ‘tangible’ factors ranging from plans and reports to new or improved lines and stations etc.) and Outcomes (e.g. ‘intangible’ factors such as improved relationship or development of trust between participants; learning effects) are criteria for the evaluation of participation processes. Outcomes are hard to measure and therefore the examples in the table include mainly outputs (based on 19).

COMMUNITY ENGAGEMENT

Please note: Community involvement and engagement were an overall aim of the Citizens’ Rail project. This sections shows examples, where engagement itself was at the heart of the activities by using railway as an opportunity for peoples’ participation. Examples for participation in line and station projects are given above.

Project Region Objectives Participatory Design Stakeholders involved Outputs/
Community ambassadors Lancashire – undertake social outreach to introduce local people to the railway
– reaching out to communities to understand, and then overcome, barriers to local rail travel
Northern Rail’s three community ambassadors worked to engage hard-to-reach groups, increase off-peak travel, raise the profile of train services and improve the relationship between the community and the railways. – Northern Rail (NR)
– Community Ambassadors (NR)
– schools, colleges
– Black and Minority Ethnic (BME) group
– socially excluded groups
Read about academic research into their impact
School involvement Devon/Torbay – introduce school children to rail travel
– engage the next generation of rail passengers and acclimatise them to using rail
School excursions as school train trips on the Riviera Line

(Read an example for an international school exchange)

– school children (primary school)
– teachers
– parents
– train operating company
– Community Rail Partnership
– local authorities
More than 650 pupils from 13 schools have already participated
Volunteering opportunities Devon – improving facilities
– create more attractive, more welcoming stations
– helping people feeling ownership of their station
– initiating “Friends of Station” group (5-10 people should take part in gardening day)
– recruiting (new) members
– organize gardening eventsExamples:
Friends of Dawlish Station
Friends of Teignmouth Station
– volunteers (mostly citizens)
– supporters (e.g. horticulturalist, staff from local supermarket and from Network Rail’s ‘orange army’)
– responsibles (e.g. Station Manager, Line Officer)
green makeover/ revamps of stations
new planting at stations
brighten up the station platforms and entrance
International Masterclasses with Students Aachen, Parkstad Limburg, Lancashire collection of visions and ideas in pre-planning or early planning phase respectively Masterclasses:
Aachen (DE)/ Heerlen (NL) 2013 (focus: visions for station improvements)
Preston (UK) 2015 (focus: Marketing rails travel)Exemplary activities:
– site visits to stations
– workshops in international, multi-disciplinary teams
– thought-provoking presentations by academics/ professionals
– result presentation and promotion
– students from different disciplines (e.g. geography, tourism, design and civil engineering) and countries
– university staff and professionals (as speakers, workshop coordinators)
– authorities (local and regional)
– railways authorities/ organisations
Examples for implemented ideas:

2013:
Creative Station designs

2015:
Students’ innovative Wi-Fi idea becomes a reality

1 Outputs (e.g. ‘tangible’ factors ranging from plans and reports to new or improved lines and stations etc.) and Outcomes (e.g. ‘intangible’ factors such as improved relationship or development of trust between participants; learning effects) are criteria for the evaluation of participation processes. Outcomes are hard to measure and therefore the examples in the table include mainly outputs (based on 19).

More from the toolkit

Secured By miniOrange