Last modified on 22 February 2011, at 02:29

A Guidebook for Managing Telecentre Networks/Monitoring, Evaluation and Learning for Telecentre networks

Monitoring, Evaluation and Learning for Telecentre networksEdit

Kemly Camacho

This chapter presents some of the basic elements for conducting an evaluation process of telecentre networks. Evaluation is not a fixed recipe, but should be tailored to each case; that is, the same design cannot be applied to all telecentre networks. But there are certain guidelines that can be observed for specific types of evaluations, such as those for development projects, political campaigns or, in our case, telecentre networks. For this reason, in this chapter we focus on one particular case, with an explanation of the main steps to be followed.

It is essential for a given evaluation to set out its specific purpose, questions, categories, variables and indicators, as well as tailor its methodology. This implies tailoring the data gathering techniques, analysis and dissemination of the results to the purpose and object of the evaluation exercise. An evaluation is basically a type of research that aims to provide inputs for decision-making, and must therefore meet the same rigorous requirements as an investigative process.

Describing the Evaluation ProcessEdit

Figure 8.1 outlines the evaluation process. The circles indicate the main threads, the arrows represent the flow between them, and the boxes show the main products of each process.

Figure 8.1 A Description of the Evaluation Process


Figure 8.1 represents an example of an approach to the evaluation process for a telecentre network. Some of the steps and questions can be applicable to the reader’s own telecentre networks. The intention is to provide a useful reference as a guide for the elaboration of other evaluative processes that are adaptable to the needs of the specific network.

It is important to note the indicators presented within the evaluative framework. Due to the influence that the logical framework has had in the evaluation processes and the design of development programs, it has been assumed that indicators are “objectively verifiable values”. Clearly, what we propose does not follow this traditional belief and bets on what is called “indicative indicators”. These can be described as a series of statements that the person who designs the evaluation creates from his/her knowledge on the subject, to determine how a variable in the evaluated program or project is being addressed. These indicative indicators do not seek a rating or a measurement, but rather an assessment. It is also recommended that they are created in collaboration with the people involved in the project, in this case, with the telecentre network.

Demarcation I: Defining the entity to be evaluatedEdit

First, it is necessary to determine the boundaries for an evaluation by defining the entity to be evaluated, and its major guiding questions. In this case, we could define the ‘object’ as follows: a development-oriented national telecentre network.

This implies two things: that the objective is to evaluate a telecentre network, and thus the process does not focus on assessing the functioning of a single telecentre but rather the operation of telecentres as a network, including their support organizations. Moreover, this is an evaluation process; that is, it is assumed that the network is functioning, or that it exists and is active at the time that the evaluation is being developed.

Demarcation II: Formulating evaluation questionsEdit

The questions asked help to set the boundaries of an evaluation. It is not possible to assess all areas of a given evaluation object. These questions help us to understand which aspects will be prioritized in the evaluation. An evaluation question is not just any question: it is analytical, investigative and cannot be only a descriptive one. An example of a main evaluation or research question could be formulated as [1]:

To what extent does the networking of telecentres improve the opportunities for the digital inclusion of people who have fewer opportunities to access information and communication technologies?

It is explicit in the question’s formulation that what we want to evaluate is the influence that networking is having on social transformations through digital inclusion processes.

When conducting an evaluation, various areas of analysis may be chosen depending on the case, and they may be linked from the evaluative question(s). Make sure that the questions are evaluative and not descriptive; that is, that they are analytic. To put it more simply, the questions cannot be answered with a quick answer such as “yes” or “no”.

Design I: The evaluative frameworkEdit

In order to carry out any monitoring or evaluation process, we must operationalize the key evaluation question(s). For this we create what is called the ‘evaluative framework’. This will serve as a guide to operationalize the question and to determine how to evaluate the social phenomenon. The evaluative framework depends on what will be assessed (object), the time frame covered for activities (starting, on process, recently completed, completed some time ago), the type of evaluation (that is, if it is done by someone of the same network or someone outside the network), if it is participatory or not, among others.

Following a rigorous evaluation process, once the object and main evaluation question is defined, the latter is broken down into secondary questions, categories of analysis, variables and indicators (if applicable). Once the evaluation framework is designed, the methodology needs to be defined, including the evaluation approach and the related tools and techniques with which this will be investigated. One such evaluative scheme is detailed below, with the steps are illustrated in the table below:

Box 8.1 Example of an evaluative framework for a network of telecentres involved in development processes
Major analysis category: Networking
Secondary question #1: To what extent has networking improved the performance of individual telecentres?
Variables for the analysis of the question Indicative indicators Possible sources
1.Mutual support a.When one telecentre faces a problem, other members of the network provide support;

b.When a telecentre faces a problem it is sometimes supported by a few members of the network (but not always); c.When a telecentre faces a problem it is not supported by any of the other members of the network.

Surveys

Interviews Telecentre stories

2.Knowledge sharing (KS) a.The network is constantly sharing knowledge to help strengthen the network and its members;

b.The network runs a knowledge sharing process from time to time but doesn’t have permanent KS mechanisms; c.The network has not set up knowledge sharing processes among its members.

Surveys

Interviews Telecentre stories Review of existing knowledge sharing opportunities

3.Joint projects a.The network develops joint projects involving some of its members, e.g. for capacity building processes, annual assemblies; service provision, etc.

b.The network develops joint activities but doesn’t have medium or long-term projects; c.The network members do not develop joint projects.

Surveys

Interviews Telecentre stories Review of existing joint projects

4.Self sustainability for telecentres that are members of the network a.The network favours telecentre sustainability strategies for its members;

b.The network provides some elements for the sustainability of member telecentres but is not essential; c.The network does not provide elements for the sustainability of its members.

Surveys

Interviews Telecentre stories Review of sustainability strategies

Secondary question #2: To what extent has networking enabled the integration of other types of support for the telecentres?
Variables for the analysis of the question Indicative indicators Possible sources
1.Strengthening the support of other actors and stakeholders that are already involved a.Networking enables telecentres to integrate with public, private, and civil society organizations that can help to consolidate the network;

b.Networking identifies key actors from different sectors that have not yet integrated into the collaborative work; c.Networking does not yet have an impact on the integration of telecentres and other actors.

Interviews with actors from multiple sectors

Surveys of telecentres Interviews with telecentres Telecentre stories

2.Integration of new actors and stakeholders a.Thanks to the network, diverse social actors have been identified and are involved in supporting telecentres;

b.Through the network, people have begun to establish contacts with potential supporters, but they are not yet involved; c.The network has not managed to connect new actors to support telecentres.

Interviews with actors from multiple sectors

Surveys of telecentres Interviews with telecentres Telecentre stories

Secondary question # 3: To what extent has networking led to the positioning of telecentres at the national level?
Variables for the analysis of the question Indicative indicators Possible sources
1.Visibility of the telecentre network nationwide a.Networking has allowed for the telecentres to become key actors at national level;

b.Networking has contributed to the visibility of telecentres; c.Networking has not yet contributed to the positioning of the telecentres nationwide.

Reviews about the definition of digital inclusion policies

Interviews with people at various political levels Surveys of telecentres Interviews with telecentres Telecentre stories

2.Advocacy capacity of the network in technology public policies a.The telecentre network advocates for, and is consulted on public policies for technology and digital inclusion in the country;

b.The telecentre network delivers an opinion in regard to technology policies and digital inclusion in the country; c.The telecentre network has not yet prioritized advocacy on public policies of technology and digital inclusion.

Reviews about the definition of digital inclusion policies

Interviews with people at various political levels Surveys of telecentres Interviews with telecentres Telecentre stories

3.Clarity of the role of telecentres nationwide a.The network is recognized as a key means for the digital inclusion of populations with little access to technology opportunities;

b.The network is recognized at the national level but there is no clarity about its importance; c.The network is still not recognized as a key actor.

Reviews about the definition of digital inclusion policies

Interviews with people at various political levels Surveys of telecentres Interviews with telecentres Telecentre stories

Secondary question #4: To what extent has networking encouraged organizational strengthening at the national level?
Variables for the analysis of the question Indicative indicators Possible sources
1.Permanence of the network a.The network faces several challenges and has been consolidated over time;

b.The network is working properly but it still has to be consolidated in order to face complex challenges; c. The network is not strong and may be severely threatened if challenges arise.

Surveys of network members

Interviews with network members Documentation of the network

2.Organization level of the network a.The network has an organizational structure (formal or informal) which is apparent to all members;

b.The network has an organizational structure (formal or informal) that is not so apparent to all its members; c.The network does not have an organizational structure.

Surveys of network members

Interviews with network members Documentation of the network

3.Sustainability of the network a.The network is self-sustainable;

b.The network faces sustainability challenges but is advancing positively towards sustainability; c.The network is finding it very difficult to become sustainable.

Surveys of network members

Interviews with network members Documentation of the network

Box 8.1 Example of an evaluative framework for a network of telecentres involved in development processes
Major analysis category: digital inclusion
Secondary question #1: To what extent does the telecentre network support the digital inclusion of populations who have fewer opportunities?
Variables for the analysis of the question Indicative indicators Possible sources
1.Populations are served by the network a.The network is mostly serving people with fewer opportunities to access technology (for example, people who are elderly, handicapped, indigenous, housewives, farmers, etc.);

b.The network is serving people with less access opportunities and other populations; c.The network is mostly serving people that have more access to technology conditions (young people, population with a higher educational level for example).

Surveys of the participant population

Interviews with the participant population Focus groups Observation of the network members’ spaces

2.Actions are tailored to people with lower levels of access to technology a.Actions of the network members are oriented towards digital inclusion (i.e. people and communities digitally excluded);

b.Actions of the network members are oriented towards digital inclusion for any populations; c.Actions of the network members are oriented to the same services available in other similar venues.

Surveys of the final population

Interviews with the final population Focus groups Observation of the network members’ spaces

3.New relationships forged by the telecentre network between individuals who are socially excluded a.The populations served by the network are able to develop and build new relationships;

b.The populations served by the network make contact with new people and spaces. but do not give continuity to the relationships; c.The populations served by the network do not establish any new relationships.

Surveys of the final population

Interviews with the final population Focus groups Observation of the network members’ spaces

4.New income generation opportunities for excluded populations a.The populations served by the network are able to develop new income generation opportunities;

b.The populations served by the network become aware of new ways for income generation; c.The populations served by the network do not identify new income generation opportunities.

Surveys of the final population

Interviews with the final population Focus groups Observation of the network members’ spaces

5.New recreational opportunities for excluded populations a.People served by the network count on new entertainment tools;

b.People served by the network become aware of new entertainment tools; c.People served by the network are not able to identify new entertainment spaces.

Surveys of the final population

Interviews with the final population Focus groups Observation of the network members’ spaces

Secondary question #2: To what extent does the telecentre network encourage the development of populations who have fewer opportunities?
Variables for the analysis of the question Indicative indicators Possible sources
1.Link between technology use and community needs a.The uses of the technology promoted by the network of telecentres are in line with the needs, visions, and problems of people with lower levels of access to technology opportunities;

b.Some of the uses of technology are in line to the needs, visions, and problems of people with less access to technology opportunities; c.The uses of technology do not meet the needs, visions, and problems of people with less access to technology opportunities.

Surveys of final population

Interviews with final population Focus groups Observation of the network members’ spaces

2.Transformation of communication processes of the region served by the network a.Populations served by the network have modified and improved their communication processes at internal and external community levels;

b.Populations served by the network have identified new resources for their communication processes; c.Populations served by the network have not yet identified the potential of ICTs for their communication processes

Surveys of final population

Interviews with final population Focus groups Observation of the network members’ spaces

3.Transformation of information processes of the region served by the network a.Populations served by the network have transformed their information processes and resources;

b.Populations served by the network can locate new information resources; c.Populations served by the network have not yet transformed their information processes and resources.

Surveys of final population

Interviews with final population Focus groups Observation of the network members’ spaces

4.Transformation of knowledge processes of the region served by the network a.Populations served by the network have modified their knowledge building processes;

b.Populations served by the network have modified some aspects of their knowledge building processes; c.Populations served by the network keep the same knowledge building processes.

Surveys of final population

Interviews with final population Focus groups Observation of the network members’ spaces

Design II: The line of intentionEdit

When designing an evaluative framework, as the one used as example in Box 8.1, we are making a series of assumptions based on the knowledge of what the evaluation object is supposed to do or achieve. This set of assumptions, once validated with the appropriate stakeholders, becomes the so-called ‘program theory’ or ‘line of intention of the program’. In this case, the program is a telecentre network, and some of the assumptions composing its line of intention can be stated as follows:

  • Network implies a strengthening of each of its members through mutual aid, knowledge sharing and the development of joint projects;
  • Networking increases and strengthens the amount of support that members of the network, and the network itself require;
  • There is greater potential for impact on technology and digital inclusion policies, which positions telecentres as an option for people with fewer opportunities, because networking clarifies the role of these social actors;
  • The telecentre network is aimed at meeting the demands and needs of a population with fewer possibilities of accessing development opportunities such as education, health, and income generation, among others;
  • The population is indeed targeted and served by the network, and that the services offered are aimed to meet their needs, visions, demands; and in so doing it is adapted to the context of this [2];
  • A true ICT integration in these populations will involve new communications processes, new information resources and therefore, new ways of developing one’s own knowledge.

All these aforementioned processes will result in a greater recognition of ICTs as tools of opportunity and that a telecentre network can have an impact that produces a transformation in this population. All these assumptions that make up the line of intention are subsequently monitored and evaluated to properly identify them. The line of intention is usually not previously elaborated. Thus, before initiating an evaluation process, it is advisable that this line of intention is elaborated in conjunction with the stakeholders. This will be the basis of the evaluation.

Figure 8.2 presents the line of intention in this telecentre network. It outlines the way in which the network is supposed to help transform reality. It allows clarification of what we are trying to do when creating a telecentre network. As the reader can see, it is directly linked to the evaluative framework presented in Box 8.1 above.

Figure 8.2 Line of Intention or Program Theory of the Telecentre Network

ResearchEdit

Once the evaluation purpose, framework and line of intention are defined, the fieldwork for the evaluation of the telecentre network can be started. This involves using the sources described in the evaluative framework (Box 8.1) to retrieve the information and data necessary to make the assessment. This includes developing the right tools and methods to approach these sources.

Surveys, interviews, life stories or observation are instruments that require the development of particular tools and methods. For the purposes of this paper, these instruments will not be described in detail here, but is important to note that they must be designed before starting to capture information. It is also important to point out that each variable has its own associated techniques and instruments for data and information collection.

Assessment – responding to questionsEdit

How the data will be analyzed must also be determined in advance, for which quantitative, qualitative or participative [3] are necessary by integrating the involved stakeholders into discussions and analysis. The data analysis must respond to the indicators, variables, questions and categories previously designed.

It is important to think of evaluation as a research process that allows making an informed value judgment. As such, it is meant to provide guidance with the decision-making on, for example, (i) the correction or continuation of network activities, (ii) the most appropriate use of financial support, (iii) the integration of new support to the network (e.g. local government or private enterprise) or (iv) changing the direction/the organizational transformation of the network.

Reporting backEdit

A very important phase of the evaluation process is the presentation of results. This must be adapted to the language, media and various populations that have participated. It is the duty of the evaluator to present the results to each of the populations who have been consulted. In the case of telecentre networks, it is therefore important to report back to telecentre users, supporting organizations, and public sector, aside from telecentre managers and staff.

RisksEdit

If the process presented here is not properly followed, a number of common mistakes can be committed, such as:

  • Using the evaluation to control and punish, instead of to learn and improve;
  • Understanding the evaluation as a measurement, rather than an assessment;
  • Considering the evaluation as a “creation of indicators”;
  • Considering that to “evaluate” means to “apply a survey”;
  • Considering that to “evaluate” means to “collect successful stories”.

ReferencesEdit

  1. There would be other basic evaluation questions, such as those related to the support or benefits for the individual telecentres, etc.
  2. This will transform the resources of these populations to establish relationships with various actors at national and international levels, which involve finding new opportunities related to income generation and use of leisure time.
  3. participatory actions are highly recommendable in the case of a telecentre network