PROJECT RESULTS

Demos and Videos

DEMOS

PER Use Case DEMO

This demo showcases the complete data flow within the Urban Digital Twin (UDT). The process starts with data generated by the simulator, which is ingested into an InfluxDB instance. A Producer Service queries this data in windows and publishes it to an Apache Kafka broker. The data is then consumed by two independent, asynchronous services, which model it based on a custom-designed ontology and insert it into a Virtuoso triplestore database. When a new state is required, the State Generator Service queries the RDF data from the triplestore, generates the current state, and stores it in SkyStore. This state file is then distributed to relevant components like the Simulator and Reinforcement Learning module for further processing.

TASKA Use Case C DEMO

This video show how to operate the workflow orchestration of TASKA-C MVP on an example dataset. The purpose of this video is to demonstrate how a researcher can set up and trigger the workflow in a “Interactive” mode (without COMPSs). We show how the data (on S3 storage) are converted into a scientific radio image. The complementary video will demonstrate the “Automatic” mode of the same workflow with the help of COMPSs.

SkyStore configuration explained

This demo shows a low-level description of how SkyStore is configured to use two different S3 stores: AWS eu-west-1 and eu-central-1. This is explained by identifying the various Kubernetes resources associated with the SkyStore deployment and their respective contents

 

Converting local model inference to remote inference on KServe using custon predictor

Cross-Premise Inference Demo

This demo simulates the model transfer and inference in the PER use-case. It starts with KServe being deployed on OVH K8s cluster. It uses SkyStore as a S3 model repository. In the PER use case, the model trained in the BSC HPC environment is stored in nearby S3 storage while models served in the edge cluster in Venice use a different S3 cluster that is closer to Venice. This is simulated by storing the model in SkyStore S3-proxy connected to one S3 store using AWS CLI and then loading it directly to KServe from a different SkyStore S3-proxy connected to a separate S3 store.

Runcompss K8s

In this video, we explore the configuration of COMPSs, highlighting the specific parameters required to run it efficiently. The focus here is to demonstrate a larger workflow compared to the TASKA Use Case C demo, specifically a matrix multiplication task. This use case involves a much larger number of tasks and utilizes more computing nodes, showcasing the enhanced parallelism that COMPSs provides. The graphical representation of tasks in the Paraver trace helps us to visualize the execution across multiple nodes, highlighting the parallel tasks and their execution times.

 

Executing the TASKA C workflow using COMPSs and Lithops Software

In this video, we showcase how we implement a workflow using COMPSs for the TASKA use case, specifically focusing on TASKA Use Case C. This use case involves both an interactive part, managed manually with a Jupyter notebook, and an automatic version, which we focus on here. In the automatic version, the scientist has predefined the steps for rebinning, calibration, and imaging of the datasets, and we use COMPSs to parallelize the execution of these steps efficiently.

Additionally, each task runs a Lithops process, which helps enhance parallelism even further. An important point to note is the use of COMPSs decorators in the code, making it straightforward for developers to adapt and optimize their code for distributed execution by simply adding some decorators.

Finally, by leveraging Paraver, we are able to visualize task dependencies, execution times, and the resources where tasks were executed. This allows for deep insight into the workflow performance and highlights the effectiveness of parallelism in our solution.

Ansible DEMO for Kubernetes

This demo shows how the Kubernetes cluster creation, management and deletion is automatized with the Ansible tool, which ensures idempotency. It ensures that playbooks achieve the same result regardless of how many times they are applied. This allows for all partners of the project to create their Kubernetes clusters in the same way.

SkyStore on Kubernetes

This demo is based on the original SkyStore demonstrator from D4.2 but using K8s onboarding. In this case, both clients’ S3 operations seem to operate on a single (virtual) common S3 storage, even though they are connected to different S3 stores through their accompanying S3-proxies.

Cybersecurity Docker Registry Security Scans

The video demonstrates how Binare’s security toolkit developed specifically for EXTRACT automatically scans and identifies vulnerabilities and CVEs within docker images that are pushed to EXTRACT code and repository bases, thus making sure that everything is scanned before deployment and as such the remediation as pro-active and can be monitored and followed-up

 

VIDEOS

Transforming extreme data from radiotelescopes

EXTRACT project technology helps filter raw data from NenuFar radio telescopes in Nançay, France by a factor of 100 to allow meaningful data to be populated into high-quality datasets. *The NenuFAR telescope clips used in the videos were provided by Observatoire de Paris, CNRS PSL & Oak Productions, 2017.

A Personalized Evacuation Routing system

The EXTRACT project is harnessing the power of data and testing it in its Personalized Evacaution Route use case. It is creating trustworthy extreme data mining solutions that use cloud, edge and high-performance computing technologies together with AI to analyse huge amounts of data to aid human-centric decision-making.*The NenuFAR telescope clips used in the videos were provided by Observatoire de Paris, CNRS PSL & Oak Productions, 2017.

Citizen Engagement in EXTRACT Personalized Evacuation Route case

The challenges faced by people in an emergency situation in Venice, Italy is simulated by retired policemen and women who are familiar with the specific obstacles faced by the city in crisis due to its winding streets, canal system and high visitor population. The EXTRACT project is drawing on the knowledge, skills and passion of these volunteers to help raise awareness of the potential benefits EXTRACT solutions can offer in this regard. More information.

Navigating Personalized Evacuation Solutions for Venice

Venice’s involvement in the EXTRACT project demonstrates the real-world value of the project’s technological advancements for public administrations and local citizens. To raise awareness of these challenges and the EU-funded EXTRACT project’s commitment to addressing these challenges, the Venice consortium partner has produced a video showcasing the unique case of Venice and the project’s objectives.

Unveiling the DataNexus Cluster: Seven EU-projects address extreme data challenges

The video introduces the 7 EU-funded projects funded under the the Horizon Europe call HORIZON-CL4-2022-DATA-01-05 working to address the challenges posed by extreme data generated by Internet of Things (IoT), industrial, business, administration, environmental, scientific, societal sources. They pursue innovative approaches that integrate cutting-edge technologies, including artificial intelligence (AI), high performance computing (HPC) and Edge/Fog/Cloud Computing.

Enrico Gavagnin, Venice city Councillor, at the Smart City Expo Wolrd Congress

In this short interview, recorded at the Smart City Expo Wolrd Congress, Enrico Gavagnin, Venice city Councillor, explains briefly the main challenge of the PER Use Case and how it aims to enhance crisis response in Venice.

Big Data & AI for enhanced urban decision-making: Venice & Barcelona

Session held at the Orange Agora of the Smart City Expo World Congress, where the EXTRACT and ASCENDER projects highlighted the advances in their technology, led by researcher Eduardo Quiñones, accompanied by Enrico Gavagnin and Marc Montlleó, who detailed the use cases of the cities of Venice and Barcelona, ​​respectively.