Vision for national data and analytics platform
NITI Aayog has released a vision document for the National Data and Analytics Platform (NDAP). The platform’s aim is to democratize access to publicly available government data. It will host the latest datasets from various government websites, present them coherently, and provide tools for analytics and visualization. NDAP will follow a user-centric approach and will enable data access in a simple and intuitive portal tailored to the needs of a variety of stakeholders.
NDAP will spearhead the standardization of formats in which data is presented across sectors and will cater to a wide audience of policymakers, researchers, innovators, data scientists, journalists and citizens.
‘India has made tremendous progress in generating and using data for policymaking and programme implementation. Efforts are needed to further strengthen the data ecosystem. NDAP, with its mission to be a one-stop and user-friendly data platform, will go a long way in transforming India’s data ecosystem.’
NITI Aayog has championed the use of data in all spheres of governance. It has taken the lead in providing the nation with a simple and compelling platform where various datasets can be accessed together. There has been a longstanding gap in the data ecosystem, which has been sought to be addressed.
All datasets are not published in a user-centric manner that is analyzable and visually presented. Also, there are a multitude of datasets out there, which are not easily accessible. The NDAP proposes a simple, interactive, visual, and robust platform that will host various Central and State government datasets.’
An inter-ministerial committee will oversee the progress of the development of platform and a group of leading experts have been inducted into an advisory group to provide technical guidance for the development of the platform. The success of the endeavour is dependent on the cooperation and support of various stakeholders.
The development of NDAP will take place over a period of one year. The first version of the platform is expected to be launched in 2021. The process will follow a user-centric approach, and will incorporate feedback received from various users and stakeholders throughout the course of its development.
Overview
India has rich publicly available data. Government departments have digitised various processes and reporting requirements, which in turn are feeding into management information systems and interactive dashboards. This data is being used to track and visualise progress and make iterative improvements on the ground.
However, the data landscape can improve. All public government data should be easily accessible to all stakeholders in a user-friendly manner. Further, data across sectors should be coherent to support analysis and dissemination. In addition, there should be easy access to the most recent data, published reliably.
Objectives
User-centric: Data access is user-friendly and engaging, tailored to the needs of policymakers, bureaucrats, researchers, innovators, data scientists, journalists and citizens alike.
Coherent
Multiple data sets are presented using a standardized schema, by using common geographical and temporal identifiers.
Up-to-date
Standard Operating Procedures are set to ensure data is updated regularly.
Data Sources
- Websites of over 50 Ministries and Departments of the Central Government and data.gov.in
- Websites of all Departments and Ministries of the State Government, not exceeding 250
Challenges
Data is not published in a user-centric manner. The current data formats are often not conducive for research and innovation. Many departments maintain public dashboards with visualisations, and options to download data in analysable formats. However, some datasets are only available in PDF, webpage or as an image, making it difficult for further analysis. If this issue is resolved, researchers and data scientists will significantly save time and resources in cleaning and preparing government datasets for analysis
Data ecosystem is incoherent due to different standards. Ministries and Departments do not use a shared standard for common indicators. Attributes like region and time period defined differently. This makes it difficult for datasets to speak to each other and present a coherent picture. If we can solve this, many use-cases can emerge. For example, a District Magistrate can access data across all departments for their district, easily, in one platform.
Approach
We intend to draw inspiration from the best platforms around the world. Significant progress in improving user experience while accessing data generated by the public, as can be seen in platforms such as ‘datausa.io’ and ‘data.gov.sg’. The main utility of such platforms is to provide a seamless, interactive experience in accessing data across multiple socio-economic indicators in one place. Also, they provide datasets in a machine-readable format.
We also seek to build on the success of existing Indian data platforms. Over the years, notable improvements have taken place in the data publishing ecosystem. For example, data.gov.in provides ready access to data from 165 departments. DISHA, by the Ministry of Rural Development, provides access to data from 42 schemes across 20 ministries. Various states also have a ‘Chief Minister’s Dashboard’. Initiatives like these provide a rich source of data for NDAP.
NDAP will take a user-centric approach to providing access to data. The platform will be powered by a user friendly search engine, backed by seamless navigation, with a world-class user interface. Data will be provided in a machine-readable format with customisable analytics. Dynamic visualisations will be generated based on user search terms. The platform will be designed and optimised to the needs of policymakers, bureaucrats, researchers, innovators, data scientists, journalists, and citizens.
NDAP will provide access to data from multiple sectors in one place. Data will be sourced from different Central and State Ministries and Departments. This will be presented using a common schema, with clear definitions and narratives to help the user navigate the differences between similar data from different sources.
NDAP will be a reliable platform for up-to-date data. Standard Operating Procedures (SOPs) will be developed to keep data updated. There will be regular tracking of compliance to these SOPs. Further, APIs, web-crawling and other methods will be used to keep the data fully up-to-date.
Success of NDAP relies on effective collaboration. NDAP goes beyond just developing a platform and will require extensive consultation. Inputs will be needed from data producers in the government to identify data sources and develop data processing techniques for the datasets to be coherent. Regular feedback from users -- policymakers, bureaucrats, researchers, innovators, data scientists, journalists, and citizens -- will also assist in ensuring the platform stays relevant.
NDAP has the potential to transform data-driven discourse, research, innovation, and decision-making in India. There are a large number of benefits for up-to-date and reliable data. NDAP’s vision is to democratise access to public government data and provide a worldclass user experience. We strongly believe it will be an important tool for the nation’s progress
Governance Structure
High Powered Steering Committee
A high powered inter-ministerial steering committee under the chairmanship of Vice Chairman, NITI Aayog will be set up to provide direction, oversee progress, guide on data sources, and address various inter-ministerial issues on collating data.
A Technical Advisory Group consisting of sector and technology experts will be established to provide expert consultation. TAG will provide guidance on the development of the platform, management of data, and aligning the platform for user-needs.
Established at NITI Aayog to coordinate with different stakeholders and manage various facets of NDAP.
A technology vendor will be engaged for development and operation of NDAP.
Current members of the Technical Advisory Group include:
- Mudit Kapoor, Professor, Indian Statistical Institute
- Ajit Pai, OSD, Vice Chairman, NITI Aayog
- Asha Subramanian, Founder & CEO Semantic Web India
- Monojit Choudhury, Principle Researcher, Microsoft
- Lalitesh Katragadda, Founder, Indihood
- Yamini Aiyer, President and Chief Executive of CPR
- Sam Asher, Co-Founder, Development Data Lab
Additional members would be added as the platform matures. Omidyar Network India has selected IDInsight to provide support to NITI Aayog for this initiative.
Way Forward
The first version of National Data and Analytics Platform is proposed to be released in 2021.
NDAP is a pan-India initiative by NITI Aayog. Its success requires extensive support and cooperation of various stakeholders such as Central Ministries, State Governments and guidance from experts to realise the vision and meet the timelines.
(Source: Niti Ayog/PIB)