High-Performance Computing for Artificial Intelligence 4010-HPA-OG

Artificial Intelligence (AI) has become a widely used tool in scientific work. Not only does AI enable advanced data analysis and visualization but it can also replace traditional numerical simulations or speed them up by making accurate predictions of interactions in a system. Additionally, AI allows for quick searching of scientific publication databases and generating article summaries. It is also being used more frequently for the full writing of scientific papers. However, the development of more sophisticated AI models requires the processing of huge amounts of data and the costly, time-consuming process of training AI models. This is especially evident with large language models (LLMs), where the training process can take weeks on thousands of the most powerful GPUs. Working with such complex models requires the use of large computing power: computing clusters or a cloud, which is the domain of High-Performance Computing (HPC).

HPC is a field of computer science dedicated to solving the most demanding computational problems in science. Top well-known applications include: (i) predicting the properties of semiconductors using quantum mechanical calculations with Density Functional Theory (DFT), (ii) drug design through molecular dynamics simulations, and (iii) numerical weather forecasting. Such calculations require hundreds or even thousands of CPUs. The key element of HPC is parallel computing. It allows calculations to be sped up by using many CPUs or GPUs. It also helps reduce the memory needed for each single computing core. For many years, parallel computing has been essential for scientific calculations, and now it is a crucial technique for AI. This is why the techniques, methodologies, and algorithms developed for HPC tasks are now also being applied to the training of AI models.

This lecture will focus on three main aspects:
1. What HPC is, how to work with computing clusters, and how parallel computing can be used in scientific research?
2. How to speed up Python code execution, with particular focus on mathematical operations, and how to parallelize these operations or use computational accelerators (like GPUs).
3. How to improve the efficiency of AI model training and apply parallel programming techniques to AI model training.

The classes are dedicated to students interested in artificial intelligence and high-performance computing. Technical knowledge is not required, but a basic understanding of computer science and artificial intelligence topics will be appreciated. During the classes, participants will be introduced to the basic terminology necessary to understand the lecture's subject matter.

The session will be conducted in a workshop format with live demonstrations and optional exercises to follow.

Prerequisites (description)

Basic knowledge of machine learning and computer science will be useful.

Course coordinators

Maciej Marchwiany

Type of course

general courses

Mode

Classroom

Learning outcomes

W1 - Knows the basic computer science and machine learning terminology and understands its sources and applications in related scientific disciplines - K_W02
W2 - Understands the role of High-Performance Computing in Artificial Intelligence - K_W03
W3 - Has basic knowledge of the place of time series analysis and anomaly detection in the system of sciences and of its subject-related and methodological connections to other scientific disciplines - K_W04
W4 - Knows the HPC tools used in AI and understands the advantages and disadvantages of individual research methods - K_W05
W5 - Knows what a programming language is and understands how it is constructed - K_W07

U1 - Is able to use basic terms in HPC and AI - K_U11
U2 - Is able to prepare a short scientific report - K_U09
U3 - Is able to use telecommunications tools for effective collaboration and is able to plan and implement numerical simulations for a selected application field of time series analysis and anomaly detection, analyze their results, and draw conclusions - K_U13
U4 - Knows the principles of health and safety of working with a computer and is able to perform basic economic analysis of the calculations carried out - K_U14
U5 - Can write simple queue scripts for ML model training - K_U11
U6 - Can evaluate the effectiveness of competing machine learning systems - K_U10

K1 - Independently plans systematically, and timely implements research goals, making difficult decisions when necessary - K_K08, K_K09
K2 - Is able to participate in substantive discussions, is able to work out compromises, and define a common position - K_K03, K_K05
K3 - Is able to behave politely during a discussion without offending other participants - K_K10

Assessment criteria

Learning outcomes K1-K3 are assessed continuously during classes.
Learning outcomes U1-U6 are assessed continuously during classes.
Learning outcomes W1-W5 are assessed in a test, where students also receive their final grade.

Single choice test.

Practical placement

Not applicable.

Bibliography

Materials and courses available online.

Term 2024L:

Materials and courses available online.

Notes

Term 2024L:

The classes take place at the ICM headquarters at Pawińskiego 5a, IBB PAN building, 5th floor, block D.

Additional information

Information on level of this course, year of study and semester when the course unit is delivered, types and amount of class hours - can be found in course structure diagrams of apropriate study programmes. This course is related to the following study programmes:

Additional information (registration calendar, class conductors, localization and schedules of classes), might be available in the USOSweb system:

Description of 4010-HPA-OG in USOSweb