Introduction to Parallel Processing

Instructor : Prof. Amanda Bienz Email :
Classroom : Online (Zoom) Class Time : M/W/F 11:00-11:50
Office : Online (Zoom) Office Hours : T 1:00-2:00, Th 4:00-5:00

Welcome to CS 442/542. This website contains information about the course. Course description and policies can also be found in the syllabus. Lectures and slides will be posted alongside the schedule.

COVID 19 Information

Due to the COVID-19 pandemic, UNM and the School of Engineering will require students, staff, and faculty to follow all health guidelines of the New Mexico Department of Health. Specifically, students, faculty and staff will adhere to social distancing guidelines, will wear masks in all buildings on campus (students are expected to provide their own masks), will maintain a distance of at least six feet from others, and will wash their hands frequently when on campus. To ensure social distancing, classes and laboratories that have enrollments larger than a third of the capacity of the scheduled classroom will be offered in a Face-to-Face + Remote Scheduled format. This will require that students attend class on campus only on certain days, watching the class live via the internet on other days. Your instructor will assign you to specific days to attend on campus – if you have a special need regarding timing, contact your instructor for an accommodation. If you have an underlying health condition that will preclude you from coming to campus, take care of a family member with an underlying condition, or simply do not feel comfortable coming to campus, you can attend remotely; please contact your instructor to make arrangements for this. It is possible that sometime during this semester the pandemic may worsen; if this happens it may be necessary to close campus and transition all classes to a remote delivery.

Course Information

Course Format

This course will be offered in a Remote schedule format. All assignments and exams will be online. The course will be taught in a flipped classroom format, with lectures available online prior to each class. We will discuss the lectures, go over questions, and step through examples during the class period, with live streaming and recordings of each class available online. Class attendance is not required and you can attend the course aynchronously if you would like.

Course Description

CS 442/542 is a senior and graduate level course in parallel programming, covering shared memory, distributed memory, and heterogeneous systems. The course focuses on understanding parallel concepts, reformulating algorithms for parallelism, and modeling and optimizing the performance of parallel applications. At the end of the semester, you should be able to model the performance of a parallel application, determine performance bottlenecks, and reduce bottlenecks found in the models. This course covers many topics in parallelism, including:

  • Concept of parallelism in programming
  • Parallel performance models
  • Shared memory parallelism
    • OpenMP
    • Non-uniform memory access (NUMA) nodes
    • Caching, prefetching, reading and writing in shared memory
    • Shared memory performance models
  • Distributed memory parallelism
    • Message-Passing Interface (MPI)
    • Distributed system topologies
    • Performance and scalability measures
    • Collectives
    • Point-to-point communication
    • Dense and sparse linear algebra
    • Distributed system performance models
    • Reliability and fault tolerance
    • Parallel I/O
  • Heterogeneous architectures
    • State-of-the-art CPU + GPU systems
    • CUDA overview (brief)
    • OpenMP for accelerators
    • Heterogeneous performance models
  • Recent research in parallel computing


Grades are given based on the standard 10-point scale (90-100 A, 80-89.9 B, etc.). There will be no curve for the course. The grades are broken down as follows:

Component Percentage of Grade
In-class questions 5%
Piazza discussion 5%
2 midterm exams 10%
Homework assignments 40%
Final project 40%

Course Materials

There is no textbook required for this course. However, the following books are helpful resources.

  • Introduction to High Performance Computing for Scientists and Engineers by Georg Hager and Gerhard Wellein. ISBN 9781439811924
  • Introduction to Parallel Computing (2nd Edition) by Ananth Grama, George Karypis, Vipin Kumar, and Anshul Gupta. ISBN 9780201648652
  • Designing and Building Parallel Programs by Ian Foster

There are other textbooks that are useful for background topics, if you want more information on them.

  • "Numerical Linear Algebra" by Llyod N. Trefethen and David Bau, III
  • "Introduction to High Performance Scientific Computing" by Victor Eijkhout


As a senior and graduate level class, it is expected that each student has basic programming experience, ideally in C, C++, or Fortran as OpenMP and MPI work with these languagues. Experience with Python or matlab will also be useful for analyzing parallel application output. You will be allowed to do homework assignments in the language of your choice, but all solutions will be posted in C++ or Python. Finally, as all parallel programs will be compiled and run on UNIX systems, command line experience is useful. Instructions will be given for how to compile and run a simple program on these systems.

Students are expected to have varying backgrounds in the topics of computer architecture, Parallel programming, and numerical methods. Prior to taking this course, each student should have experience with at least one of the following topics.

  • Computer architecture (CS341)
  • Big data computing (CS567)
  • Scientific computing (CS471/MATH471)

Class Schedule

Date Content Due
08/17/20 Course Info, Intro To Parallelism Office Hour Poll
08/19/20 Problems with Serial Computers / Intro to Caches
08/21/20 Caching and Prefetching
08/24/20 Vectorization Homework 1
08/26/20 Introduction to Performance Modeling
08/28/20 Introduction to Shared Memory Systems
08/31/20 More on Shared Memory, Threads
09/02/20 Introduction to OpenMP
09/04/20 More OpenMP Examples on False Sharing, Shared vs Private
09/07/20 Labor Day - University Holiday
09/09/20 OpenMP False Sharing
09/11/20 OpenMP Performance Homework 2
09/14/20 Sparse Matrices
09/16/20 OpenMP NUMA Nodes, Performance
09/18/20 OpenMP Benchmarking Tools
09/25/20 Homework 3
09/28/20 Exam 1 (anytime this week)
09/30/20 Exam 1 (anytime this week)
10/02/20 Exam 1 (anytime this week)
10/07/20 Wednesday Break Day - University Holiday
10/16/20 Homework 4
10/23/20 Project Proposals
11/02/20 Homework 5
11/09/20 Exam 2 (anytime this week)
11/11/20 Exam 2 (anytime this week)
11/13/20 Exam 2 (anytime this week)
11/20/20 Project Status Update
11/25/20 Tentative Short Homework 6
11/27/20 Thanksgiving Break - University Holiday
12/09/20 Exam Period 10am - 12pm Project Presentations
12/11/20 Last Day of Exams Project Summary Due

Course Policies

Attendance Policy

All classes can be attended online. Attendance (remote or online) is not required, but you will be responsible for material covered during each class. There are micro-assessments due at the end of each class. These will be posted online along with the lecture days before each class and can be completed at any time before the end of the class. Missed micro-assessments cannot be made up, but the lowest two scores will be dropped. The purpose of these micro-assessments is to show how much of the covered material is understood and to determine if further time should be spent on a topic. In total, they only count for 5% of the final grade for the course.

Incomplete and Late Assignments

Late assignments are accepted but penalized unless a valid excuse is communicated before the deadline. If the deadline is missed due to unforeseeable circumstances, we will work something out. Barring a valid excuse, an assignment will be accepted for a 25% reduction once grading has begun. If solutions have been posted when the assignment is turned in, the assignment will instead be accepted for a 50% reduction. Even if solutions have been posted, you still should not copy these, but only use them to help you complete your assignment.

Regrading Requests

Requests for regrading of assignments or exams should be made within two weeks of the date from which the assignment is returned. Assignments will not be regraded after that point. Regrading requests can be made via email.

Academic Integrity

Each student is expected to maintain the highest standards of honesty and integrity in academic and professional matters. The University reserves the right to take disciplinary action, up to and including dismissal, against any student who is found guilty of academic dishonesty or otherwise fails to meet the standards. Any student judged to have engaged in academic dishonesty in course work may receive a reduced or failing grade for the work in question and/or for the course.Academic dishonesty includes, but is not limited to, dishonesty in quizzes, tests, or assignments; claiming credit for work not done or done by others; hindering the academic work of other students; misrepresenting academic or professional qualifications within or without the University; and nondisclosure or misrepresentation in filling out applications or other University records.

Accomodations for Disabilities

In accordance with University Policy 2310 and the Americans with Disabilities Act (ADA), academic accommodations may be made for any student who notifies the instructor of the need for an accommodation. It is imperative that you take the initiative to bring such needs to the instructor’s attention, as he/she are not legally permitted to inquire. Students who may require assistance in emergency evacuations should contact the instructor as to the most appropriate procedures to follow. Contact Accessibility Resource Center at 277-3506 for additional information.

If you need accommodation regarding the course format or course requirements, please contact me as soon as possible. We will work with the Disability Services Office to develop a plan for accommodation.

Sexual Harassment and Title XI

In an effort to meet obligations under Title IX, UNM faculty, Teaching Assistants, and Graduate Assistants are considered “responsible employees” by the Department of Education \\ (see pg 15\\ This designation requires that any report of gender discrimination which includes sexual harassment, sexual misconduct and sexual violence made to a faculty member, TA, or GA must be reported to the Title IX Coordinator at the Office of Equal Opportunity ( For more information on the campus policy regarding sexual misconduct, see: