6 ECTS credits
156 h study time

Offer 1 with catalog number 4004940ENR for all students in the 1st semester at a (E) Master - advanced level.

Semester
1st semester
Enrollment based on exam contract
Impossible
Grading method
Grading (scale from 0 to 20)
Can retake in second session
Yes
Enrollment Requirements
Registration for Parallel Systems is allowed if one has already successfully accomplished Computer Systems (001688) or Teleprocessing (000422).
Taught in
English
Faculty
Faculty of Engineering
Department
Electronics and Informatics
Educational team
Jan LEMEIRE (course titular)
Jan G. CORNELIS
Activities and contact hours
24 contact hours Lecture
36 contact hours Seminar, Exercises or Practicals
Course Content

The course 'Parallel Systems' consists of a theoretical and a practical part.
The theoretical part consists of 10 lectures, of which the first and the last are the most important since they discuss the course’s philosophy.
The practical part consists of 6 practical sessions and a programming project. During the practical sessions, the constructs for multithreading and message-passing in java are explored. The programming project happens in groups of 2 to 3 persons.

The design of simple, generic and performant parallel programs relies on deep insight in all aspects of modern hardware and software technology.
The course therefore offers a wider scope than just dealing with the 'parallel issues'.

1. Overview of Parallel Systems
Parallel systems constitute of processors that are connected in some way. They are more present than ever. Parallel processing is the only answer to the ever-increasing demand for more computational power.
Nowadays, the big giants in hardware and software, like Intel and Microsoft, are increasingly aware of it and have pounced onto the market.
This chapter presents an overview and explains why parallel resources are not for free.

2. Shared-memory paradigm: Multithreading
Multiprocessors are becoming the standard in PCs. They allow for multithreaded parallel programs, which are based on the shared-memory paradigm. In this chapter all aspects are discussed that should be taken into consideration when creating multi-threaded applications. Besides an overview we will focus on the Java constructs.

3. Message-passing paradigm: MPI
Processors that do not have access to each other's memory communicate by sending messages to each other. This is called the message-passing paradigm, of which MPI is the leading standard. It is discussed how MPI should be used and MPI is compared to more special parallel languages.

4. GPU computing

Accelerators such as GPUs offer a tremendous amount of computational power. But such power comes at the expense of the necessary reimplementation of your code and optimization of the code towards the underlying architecture.

5. Performance Analysis of Parallel Processing
The main goal of parallel processing is shortening the runtime of programs. This chapter presents the performance analysis or parallel programs.

6. Dense Matrix Algorithms
Parallelization of matrix algorithms can happen in different ways. The, sometimes subtle, differences give us the opportunity to dig deeper in the various issues involved with parallel processing.

7. Discrete Optimization Problems
More advanced issues in parallel processing can be discussed with the parallelization of discrete optimization problems.

9. Conclusions
Wrap-up of the course. The discussion of the introduction are reconsidered.

Finally, we list the main differences of this course with the course 'Multicore Programming' taught by Tom Van Cutsem.
Parallel Systems taught by Jan Lemeire   
(1) Focuses on low-level approaches to parallel computing.
(2) Discusses the well-established techniques for message-passing and multithreading which have proven their value. Standard thread creation and synchronization is treated in-depth, showing the threats of this technique to the student.
(3) Concentrates on scientific computing, such as algorithms allowing data-parallelism (‘regular concurrency’), although all applications can be parallelized with the techniques. Focus on dedicated parallel algorithms.
(4) Discusses the current trend of using GPUs for high-performance computing.
 
Multicore Programming taught by Jennifer Sartor
(1) Focuses on high-level approaches to concurrent programming.
(2) Discusses cutting-edge, innovative, scientific approaches. Considers alternative approaches to multithreading, since standard multithreading is not scalable and not robust.
(3) Focuses on concurrent applications with irregular concurrency, distributed systems, influence of concurrency on software architecture. Focus on parallelizing existing software.   
(4) Discusses important parallel software patterns such as Google’s mapreduce.

 

Course material
Digital course material (Required) : Slides of the lectures, Jan Lemeire, http://parallel.vub.ac.be
Digital course material (Required) : Practical information and documentation: http://parallel.vub.ac.be -> education, Jan Lemeire, http://parallel.vub.ac.be
Handbook (Recommended) : Introduction to Parallel Programming, A. Grama et al., 2de, Pearson, 9780201648652, 2003
Handbook (Recommended) : Principles of Parallel Programming, Calvin Lin and Larry Snyder, Pearson, 9780321487902, 2008
Additional info

This course is taught in English, but the exam can be done in Dutch.
Additional questions can be directed to jan.lemeire@vub.ac.be
Slides of the lectures, practical information and documentation: http://parallel.vub.ac.be -> education

'Introduction to Parallel Programming'
A. Grama et al.
ISBN 0-201-64865-2

'Principles of Parallel Programming'
Calvin Lin and Larry Snyder
ISBN 0321487907

Learning Outcomes

Algemene competenties

The student will understand the architecture of the various modern parallel systems. He will be able to conceive, implement and analyze parallel and distributed algorithms. He will understand why the utilization of parallel resources is not for free, that parallel processing out-of-the-box does not exist. The student will be able to utilize modern parallel technology autonomously, in the design of efficient parallel solutions for all kinds of sequential algorithms.

He will be able to apply his understanding of low-level processes and hardware characteristics to devise high-performant, scalable solutions. Since the quality of parallel programs is also determined by their genericity, usability and cost he will have learned to employ advanced principles of software engineering to design flexible and reusable parallel software.
The student will have learned to look for twofold-generic parallel solutions. On the one hand, the solution should be appropriate to a wide variety of parallel systems, such as a dedicated cluster, a GRID environment, a multiprocessor or a Graphical Processing Unit (GPU). On the other hand, it should be easy-to-use and efficient for a broad class of similar algorithms. The student will understand the trade-off between genericity and performance. This integration of high-level software issues and low-level hardware characteristics will have sharpened his general IT-skills.

By the practical project, the student will have demonstrated that he can make good judgments about complex situations and communicate his conclusions. Specific or complex parallel solutions are possible, but these are difficult to maintain and less generic. Only simple, clever solutions are feasible. The student will be able to participate to discussions about exploiting parallelism and the proper use of modern technology.

Grading

The final grade is composed based on the following categories:
Oral Exam determines 60% of the final mark.
PRAC Practical Assignment determines 40% of the final mark.

Within the Oral Exam category, the following assignments need to be completed:

  • oral exam with a relative weight of 1 which comprises 60% of the final mark.

    Note: oral exam for the theoretical part (60%).

Within the PRAC Practical Assignment category, the following assignments need to be completed:

  • evaluation project with a relative weight of 1 which comprises 40% of the final mark.

    Note: Evaluation of the project for the practical part (40%).

Additional info regarding evaluation

The end result is based on 3 parts:

A. An oral exam for the theoretical part (34%).

B. A written examination of the exercises of the practical part (33%).

C. Evaluation of the project for the practical part (33%).

Academic context

This offer is part of the following study plans:
Master of Applied Sciences and Engineering: Applied Computer Science: Standaard traject (only offered in Dutch)
Master in Applied Sciences and Engineering: Applied Computer Science: Standaard traject
Master of Applied Sciences and Engineering: Computer Science: Profile Artificial Intelligence
Master of Applied Sciences and Engineering: Computer Science: Profile Multimedia
Master of Applied Sciences and Engineering: Computer Science: Profile Software Languages and Software Engineering