A Guidebook for Managing Telecentre Networks/Monitoring, Evaluation and Learning for Telecentre networks
Monitoring, Evaluation and Learning for Telecentre networksEdit
This chapter presents some of the basic elements for conducting an evaluation process of telecentre networks. Evaluation is not a fixed recipe, but should be tailored to each case; that is, the same design cannot be applied to all telecentre networks. But there are certain guidelines that can be observed for specific types of evaluations, such as those for development projects, political campaigns or, in our case, telecentre networks. For this reason, in this chapter we focus on one particular case, with an explanation of the main steps to be followed.
It is essential for a given evaluation to set out its specific purpose, questions, categories, variables and indicators, as well as tailor its methodology. This implies tailoring the data gathering techniques, analysis and dissemination of the results to the purpose and object of the evaluation exercise. An evaluation is basically a type of research that aims to provide inputs for decision-making, and must therefore meet the same rigorous requirements as an investigative process.
Describing the Evaluation ProcessEdit
Figure 8.1 outlines the evaluation process. The circles indicate the main threads, the arrows represent the flow between them, and the boxes show the main products of each process.
Figure 8.1 represents an example of an approach to the evaluation process for a telecentre network. Some of the steps and questions can be applicable to the reader’s own telecentre networks. The intention is to provide a useful reference as a guide for the elaboration of other evaluative processes that are adaptable to the needs of the specific network.
It is important to note the indicators presented within the evaluative framework. Due to the influence that the logical framework has had in the evaluation processes and the design of development programs, it has been assumed that indicators are “objectively verifiable values”. Clearly, what we propose does not follow this traditional belief and bets on what is called “indicative indicators”. These can be described as a series of statements that the person who designs the evaluation creates from his/her knowledge on the subject, to determine how a variable in the evaluated program or project is being addressed. These indicative indicators do not seek a rating or a measurement, but rather an assessment. It is also recommended that they are created in collaboration with the people involved in the project, in this case, with the telecentre network.
Demarcation I: Defining the entity to be evaluatedEdit
First, it is necessary to determine the boundaries for an evaluation by defining the entity to be evaluated, and its major guiding questions. In this case, we could define the ‘object’ as follows: a development-oriented national telecentre network.
This implies two things: that the objective is to evaluate a telecentre network, and thus the process does not focus on assessing the functioning of a single telecentre but rather the operation of telecentres as a network, including their support organizations. Moreover, this is an evaluation process; that is, it is assumed that the network is functioning, or that it exists and is active at the time that the evaluation is being developed.
Demarcation II: Formulating evaluation questionsEdit
The questions asked help to set the boundaries of an evaluation. It is not possible to assess all areas of a given evaluation object. These questions help us to understand which aspects will be prioritized in the evaluation. An evaluation question is not just any question: it is analytical, investigative and cannot be only a descriptive one. An example of a main evaluation or research question could be formulated as:
To what extent does the networking of telecentres improve the opportunities for the digital inclusion of people who have fewer opportunities to access information and communication technologies?
It is explicit in the question’s formulation that what we want to evaluate is the influence that networking is having on social transformations through digital inclusion processes.
When conducting an evaluation, various areas of analysis may be chosen depending on the case, and they may be linked from the evaluative question(s). Make sure that the questions are evaluative and not descriptive; that is, that they are analytic. To put it more simply, the questions cannot be answered with a quick answer such as “yes” or “no”.
Design I: The evaluative frameworkEdit
In order to carry out any monitoring or evaluation process, we must operationalize the key evaluation question(s). For this we create what is called the ‘evaluative framework’. This will serve as a guide to operationalize the question and to determine how to evaluate the social phenomenon. The evaluative framework depends on what will be assessed (object), the time frame covered for activities (starting, on process, recently completed, completed some time ago), the type of evaluation (that is, if it is done by someone of the same network or someone outside the network), if it is participatory or not, among others.
Following a rigorous evaluation process, once the object and main evaluation question is defined, the latter is broken down into secondary questions, categories of analysis, variables and indicators (if applicable). Once the evaluation framework is designed, the methodology needs to be defined, including the evaluation approach and the related tools and techniques with which this will be investigated. One such evaluative scheme is detailed below, with the steps are illustrated in the table below:
Design II: The line of intentionEdit
When designing an evaluative framework, as the one used as example in Box 8.1, we are making a series of assumptions based on the knowledge of what the evaluation object is supposed to do or achieve. This set of assumptions, once validated with the appropriate stakeholders, becomes the so-called ‘program theory’ or ‘line of intention of the program’. In this case, the program is a telecentre network, and some of the assumptions composing its line of intention can be stated as follows:
- Network implies a strengthening of each of its members through mutual aid, knowledge sharing and the development of joint projects;
- Networking increases and strengthens the amount of support that members of the network, and the network itself require;
- There is greater potential for impact on technology and digital inclusion policies, which positions telecentres as an option for people with fewer opportunities, because networking clarifies the role of these social actors;
- The telecentre network is aimed at meeting the demands and needs of a population with fewer possibilities of accessing development opportunities such as education, health, and income generation, among others;
- The population is indeed targeted and served by the network, and that the services offered are aimed to meet their needs, visions, demands; and in so doing it is adapted to the context of this;
- A true ICT integration in these populations will involve new communications processes, new information resources and therefore, new ways of developing one’s own knowledge.
All these aforementioned processes will result in a greater recognition of ICTs as tools of opportunity and that a telecentre network can have an impact that produces a transformation in this population. All these assumptions that make up the line of intention are subsequently monitored and evaluated to properly identify them. The line of intention is usually not previously elaborated. Thus, before initiating an evaluation process, it is advisable that this line of intention is elaborated in conjunction with the stakeholders. This will be the basis of the evaluation.
Figure 8.2 presents the line of intention in this telecentre network. It outlines the way in which the network is supposed to help transform reality. It allows clarification of what we are trying to do when creating a telecentre network. As the reader can see, it is directly linked to the evaluative framework presented in Box 8.1 above.
Once the evaluation purpose, framework and line of intention are defined, the fieldwork for the evaluation of the telecentre network can be started. This involves using the sources described in the evaluative framework (Box 8.1) to retrieve the information and data necessary to make the assessment. This includes developing the right tools and methods to approach these sources.
Surveys, interviews, life stories or observation are instruments that require the development of particular tools and methods. For the purposes of this paper, these instruments will not be described in detail here, but is important to note that they must be designed before starting to capture information. It is also important to point out that each variable has its own associated techniques and instruments for data and information collection.
Assessment – responding to questionsEdit
How the data will be analyzed must also be determined in advance, for which quantitative, qualitative or participative  are necessary by integrating the involved stakeholders into discussions and analysis. The data analysis must respond to the indicators, variables, questions and categories previously designed.
It is important to think of evaluation as a research process that allows making an informed value judgment. As such, it is meant to provide guidance with the decision-making on, for example, (i) the correction or continuation of network activities, (ii) the most appropriate use of financial support, (iii) the integration of new support to the network (e.g. local government or private enterprise) or (iv) changing the direction/the organizational transformation of the network.
A very important phase of the evaluation process is the presentation of results. This must be adapted to the language, media and various populations that have participated. It is the duty of the evaluator to present the results to each of the populations who have been consulted. In the case of telecentre networks, it is therefore important to report back to telecentre users, supporting organizations, and public sector, aside from telecentre managers and staff.
If the process presented here is not properly followed, a number of common mistakes can be committed, such as:
- Using the evaluation to control and punish, instead of to learn and improve;
- Understanding the evaluation as a measurement, rather than an assessment;
- Considering the evaluation as a “creation of indicators”;
- Considering that to “evaluate” means to “apply a survey”;
- Considering that to “evaluate” means to “collect successful stories”.
- There would be other basic evaluation questions, such as those related to the support or benefits for the individual telecentres, etc.
- This will transform the resources of these populations to establish relationships with various actors at national and international levels, which involve finding new opportunities related to income generation and use of leisure time.
- participatory actions are highly recommendable in the case of a telecentre network