Methodology to assess the auditability of implementing the sustainable development goals

2021. 05. 10. 11:00

The General Assembly of the UN specified 17 overall Sustainable Development Goals (SDG). The fact that without measuring the success and auditing the programmes only a small part of them will be implemented is also true to these goals. Therefore, the Congress of the International Organization of Supreme Audit Institutions (INTOSAI) specified the audit of the SDGs’ implementation as a strategic target. With a series of analyses the State Audit Office of Hungary (SAO) assesses the target areas by the audit of which it could offer its best contribution to facilitating sustainable development. This document presents the methodology of these analyses.

SAO performs the audit of responsible management of public funds, state and municipality assets with general mandate. Consequently SAO can facilitate the achievement of sustainable development goals best with audits evaluating the extent of the contribution of programmes financed from public funds implemented in the target area of a given SDG to the implementation of the SDG. The conduction of performance audits assessing the success of the programmes from this point of view has three fundamental conditions:
  • the goals of the audited programmes are connected to the given SDG sub goal,
  • indicators used in the course of the programme match one or several indicators assigned to the SDG sub goal by the United Nations (UN);
  • reliable data are available to calculate the values of the indicator.
The starting point of the analyses is that the implementation of the goals specified by the UN is not an obligation for the member states. Their political commitment is for the achievement of goals relevant for them. In the course of specifying these, they must take into account that the conditions of sustainable development - for example access to clean water, large reduction of greenhouse gas emission - must also be met globally. National level audit makes sense in the case of SDG sub goals, which the given member states deems important. In Hungary it is not the task of SAO to decide this. Based on the manifestations of the Parliament and the government, upon studying strategic documents SAO can establish if national goals similar in content to a given SDG have been specified.  Therefore, the analyses of the SAO revealing the auditability of the given sustainable development goals first examine this question.

The achievement of a given goal can be verified if it is measurable. Measurability has an outstanding importance also in the process of implementing the goal as the measuring indicates the extent of approaching the goal and provides feedback on the necessity of interventions. Measuring is performed with the help of quantifiable indicators featuring the given phenomenon well. The elaborators of the UN document assigned minimum one indicator to every SDG sub goal, which shows the extent of achieving the targeted sub goal in reliable manner. The indicators must meet several scientifically established requirements but even the best indicators are not capable of presenting a comprehensive phenomenon in its complexity. Therefore the UN did not make the use of indicators applied by itself mandatory but it gave an opportunity to the member states to measure their own progress in achieving sustainable development goals with so-called surrogate indicators better expressing their own endeavours.

SAO elaborated a methodology to find out if the national strategic and programme level goals and the indicators assigned to them can match the sub goals and related indicators of the SDGs. Harmony or the lack of it can be established by filling in the matrix presented by Table 1.

Table 1 Matrix assessing the auditability of the national implementation of one given SDG

Source: Own edition based on Table No. 1 of page 8 of ÁSZ (SAO), 2020

1 I21 means first indicator for the second goal

2See column 4 of Table 2 for an explanation of the letters

The sub goals of the given SDG are named on the first line of the table. The first task of the analytic work is to reveal if there are matching national goals or (programme level) goals specified for the programmes implemented from public funds - including EU support - in the target area of the SDG. The specification of the goals generally differs at the three levels, therefore content match or similarity must be examined in the course of the analysis. Black colour in the table indicates if national or programme level goal matching the given SDG sub goal was not identified. An SDG sub goal is considered not relevant if there is no matching national goal. In such case performance audit can’t be implemented.

The next step of the analysis is to establish if progress in reaching the goals is measured with indicators matching at the three levels. The challenge here is that the indicators of the three levels are often not identical. We deem match if one of the following three conditions is met:
  • the two indicators are identical, which means they fully match;
  •  the two indicators are similar, which means that
    • one can be generated from the other with further data added (for example one indicator is an absolute quantity and the other is a specific quantity and there are available, reliable data for the denominator of the latter); or
    • the two indicators have nearly identical contents (for example one contains data per head the other data per household);
  • sub indicator: one indicator only covers part of the extent of the other indicator and their direction is the same, which means that if one indicator grows it will influence the value of the other indicator positively, too.
If an indicator of a given level does not have a matching indicator at another level, the related square is also shown in black. As the UN assigned only one or two indicators to almost every sub goal it can easily happen that there are national or programme level indicators with no matching UN indicators. Despite it the use of these indicators can be practical in the course of the audit if a more complex picture of the success of the audited programme can be obtained by their use.  

The last line of the table summarises if the possibility of audit exists based on the match of the indicators. The methodology deems the situation auditable if the audit of the programme implemented in the target area can provide quantitative evaluation of the contribution of the programme to reaching the national goal and the relevant SDG sub goal. This is indicated by letter ‘Y’ (yes, possible). Further options are also included in the table depending on what (matching) indicators belong to the given relevant SDG sub goal. They are summarised by Table 2.

Table 2 Classes of auditability of one target area based on the availability of matching indicators

Source: Own edition

In the next step of the analysis the availability of data necessary for the calculation of the indicators must be checked. The analysis covered the existence of the basic conditions of auditability after answer to this question was given. It is not practical to initiate performance audit for the lack of these as it could not provide quantitative evaluation on the success of the implementation. In addition to the existence of basic (technical) conditions several other considerations must also be considered before the launch of an audit but they are not included in the topic of this methodology.

Dr. Gyula Pulay supervisory manager