CPS3236 C ONCURRENCY , HPC AND D ISTRIBUTED C OMPUTING T HE C ONCURRENCY L ANDSCAPE Keith Bugeja / Alessio Magro / Kevin Vella Department of Computer Science Faculty of ICT, University of Malta The Concurrency Landscape C ONCURRENT S OFTWARE Sequential programs have a single control sequence. Concurrent programs contain multiple control sequences. • Two or more processes work together to solve a problem Processes may need to synchronise and/or communicate several times in order to reach a solution. C ONCURRENCY L ANDSCAPE Concurrent programming landscape is motivated by three broad overlapping classes of applications: • Multithreaded • Distributed • Parallel M ULTITHREADING Multithreaded software is written in such a way that there are several threads of control within a single application program. The number of threads is independent of the number of available processors: • More threads than processors: thread scheduler multiplexes processors among threads • More processors than threads: ef fi ciency takes a hit M ULTITHREADING Multithreading is used in: • Window systems and interactive applications to improve responsiveness • The UI thread can block while waiting for user input • One or more worker threads can continue computing in the background M ULTITHREADING Multithreading is used in: • Time-sharing and multiprocessor operating systems (OS) • Multiple applications running concurrently and largely independently, each in its own OS process • Each OS process has its own private address space and at least one thread for computing • I/O and network operations proceedings at their own pace without impacting overall system performance D ISTRIBUTED S YSTEMS The second broad class of applications is distributed computing • Software components execute at distributed locations • Coordinate their operation through a computer network Distributed software components usually execute concurrently with respect to each other. • Individual components can also exhibit internal concurrency through multithreading. The de fi ning characteristic of distributed computing is: computing and storage at distributed locations D ISTRIBUTED S YSTEMS Distributed systems on different scales: • Local • An of fi ce LAN with a fi le server and multiple clients • Global • The World Wide Web • P2P fi le sharing systems • Cloud services D ISTRIBUTED S YSTEMS Distributed system wins include: • Of fl oading intensive processing to remote servers • Accessing remote data transparently • Increasing reliability through replication at multiple locations P ARALLEL C OMPUTING The fi nal broad class of applications is Parallel computing. • Geared towards solving a problem in less time than a sequential program can. To maintain high-ef fi ciency, execution should be compute-bound rather than I/O-bound There is a limit to the amount of parallelism that can be exploited in solutions to a particular problem: • This limit can sometimes be overcome by increasing the size of the problem to be solved, typically by increasing the size of the data set. P ARALLEL C OMPUTING Parallel computing is a high-performance computing (HPC) technology. Parallel computers are built speci fi cally to run parallel programs ef fi ciently using: • large numbers of processors (1000+) with large memories, and • a high-speed, low-latency interconnecting network between processors that can scale effectively. P ARALLEL C OMPUTING Parallel computing applications include: • Scienti fi c computing, usually with large data sets • Climate modelling • Drug discovery • Protein sequence analysis • High-energy physics (simulation and experimental data analysis) • Computer-generated imagery (CGI) and visual effects (VFX) • Combinatorial optimisation problems • Financial risk modeling Concurrency Software Patterns U SEFUL P ROGRAMMING C ONSTRUCTS Fork-join executes the contained statements concurrently, each in its own thread: The fork-join statement itself completes only when all the constituent threads have completed their execution: • This is a barrier synchronisation U SEFUL P ROGRAMMING C ONSTRUCTS The parallel for loop is a replicated fork-join, parameterised over the loop counter: C ONCURRENCY S OFTWARE P ATTERNS Well-known software patterns can be reused in concurrent solutions for a broad range of problems: • Iterative parallelism • Recursive parallelism • Pipelining (producer-consumer) • Client-server • Peer-to-peer (interacting peers) Iterative Parallelism C ONCURRENCY S OFTWARE P ATTERNS : I TERATIVE P ARALLELISM Iterative parallelism features: • concurrent threads each computing a subset of the problem in an iterative fashion • possibly a subsequent phase where the results are combined An iterative sequential program is one that uses iterative constructs like for or while loops to examine data and compute results. • Loops iterating over an array can sometimes be parallelised simply by assigning threads to each iterate over a different part of the array. This can only be done if the threads do not interfere with each other’s part of the data while executing concurrently. C ONCURRENCY S OFTWARE P ATTERNS : I TERATIVE P ARALLELISM Any two threads can only safely execute concurrently if they are independent • Threads are independent if they do not write to any data that is either read or written by another concurrent thread.