Data Engineering Collaboration Platform Qorus Banking Innovation Awards 2023

The data engineering collaboration platform is One stop shop for various needs within marketing analytics team, that enables teams to work together and collaborate on data analytics projects. This platform undertaken initiatives to • Improve data integration capabilities • Optimize performance and scalability • Enhance data governance and security features. We have successfully implemented advanced integration capabilities, improved platform performance, and scalability, and introduced Helpdesk, self-password reset application, Dashboards to check close to real time status on data availability, server’s usage and modelOps. 1. Major services of the collaboration platform:  Data engineering helpdesk: It Is part of our collaboration platform designed to support and streamline the process of data engineering tasks. It provides a centralized system where data engineers can receive, manage, and resolve support requests related to all the projects within our department. Total there are 1513 tickets have been logged into this platform and Data engineering team worked on resolving 95 % of the tickets with effort of 4428 hours. 1. The data engineering helpdesk aim is to improve efficiency and effectiveness in troubleshooting and resolving the data engineering issues. 2. By centralized support requests and documentation, it reduces the time and effort spent in finding solutions to common problems. 3. Facilitates the communication between team members, ensuring faster resolution of the issues. 4. The reporting and analytics feature help identify areas for improvement and make informed decisions to optimize the process. Below screenshot shows the Data engineering Helpdesk with its dashboards.  Dashboards: There are multiple dashboards developed using PowerBI and embedded into our collaboration platform. The dashboards created to monitor server usage and data availability is a visual representation of the key metrics and information related to the performance and health of servers and data availability in real-time. Below screen shot shows the list of dashboards on our platform. The “Server usage matrix” dashboard typically presents information such as CPU and memory utilization, disk space usage, and other relevant server performance metrics. This allows administrators to monitor resource consumption and identify potential bottlenecks or issues that may affect server performance and take the proactive decisions to minimize the impacts. This helps all the data scientist teams to avoid potential risk of failures in the projects hosted on these servers by proactively monitoring the usage of the RAM, CPU, and storage in a close to real time manner. This dashboard helped to optimize for mitigating the risks of failures by 30 %, otherwise team need to proactively act upon all the issues and address them quickly to avoid the business impacts. In addition to server usage, the “Data availability” dashboard provides insights into data availability. It displays information about the status of data sources, data pipelines, and data processing jobs. This helps ensure that data is flowing smoothly through the pipelines, and any interruptions or errors can be quickly identified and addressed. This reduces the load of queries on the marketing DataLake by 20%, earlier all the users need to query the database to check the data availability status, now it is available in the form of an interactive dashboard. Overall, the server usage and data availability dashboard provide a centralized and visualized view of server performance and data availability, enabling administrators to proactively monitor and manage the infrastructure to ensure optimal performance and availability.  Self-help resources and training: These services and initiatives enhances the user experience of the collaboration platform to next level by offering many of the services easily accessible at one centralized platform. Below screenshot shows the self-help resources and trainings services created in our platform. Overall, these services helped to optimize the effort of all the team members by 15%. • “Self-password reset” application helps users to reset Jupyter passwords on their own without depending on admin team. It is available for 24/7 and helping team to work seamlessly without waiting for days to just reset the password. The Jupyter password reset requests have been significantly reduced from more than 200 requests per month to only less than 5. • “Onboarding process” integrated to our collaboration platform contains the complete set of instructions starting from requesting access to the required tools and technologies till steps required for the administration and development activities. This enables new resources to onboard to the work sooner without spending more trainings on all the processes. Having this comprehensive set of the documents, time taken to onboard new resource has been reduced from more than 30 days to less than 15 days. • “Data dictionary” is very essential and needed to have proper knowledge and usage of contents. Data Dictionary provides all information about names that are used in system models. Data Dictionary also provides information about entities, relationships, and attributes that are present in the system model. • “Documentation Library” provides a secure place to store files where team members can find them easily, work on them together, and access them from any device at any time. A proper teamwise folder hierarchy has been maintained along with version controlling of all the documents.