HORIZON EUROPE RIA PROJECT - CULTURATI_Deliverable D5.4 Evaluation of AI Algorithms

Deliverable D5.4 provides a technical validation of the CULTURATI project's AI ecosystem, specifically assessing the performance and scalability of the Path Recommendation, Question Generation, and Audio Transcription subsystems at Month 36. By utilizing quantitative metrics like Cosine Similarity and Word Error Rates (WER) alongside qualitative benchmarking, the report confirms that the current open-source architecture is a robust, privacy-preserving, and cost-efficient solution- The evaluation highlights that the Path Recommendation engine, powered by the hkunlp/instructor-large model, achieves high semantic sensitivity and functional parity with proprietary models when Z-Score normalization is applied. Similarly, the Question Generation module (utilizing Leaf and Questgen) is validated for its superior factual accuracy and reliability, ensuring educational content remains deterministic and relevant. Finally, the Audio Transcription pipeline demonstrates production-grade maturity with error rates consistently below 4%, providing a high-fidelity data foundation for the entire platform. Overall, the report concludes that the integrated AI algorithms are stable, effective, and ready for deployment within cultural heritage sites.