The main objective of this project was to develop a parallel framework to help execute a coupled Discrete Element Method (DEM) and Population Balance Model (PBM) on supercomputers, and as a result reduce the time to perform a simulation. The ability to reduce the simulation time of a complex process model aids in building a control model for the granulation process, commonly found in the pharmaceutical industry to help reduce the waste in terms of raw materials as well as improve product quality in a timely manner.

Multi-physics models (e.g. DEM + PBM) are necessary to better understand the dynamics of the system at various scales i.e. micro, meso and macro scale. PBM provides meso-scale information while DEM provides particle scale information. The combination of these two methods helps describe the physical process with more accuracy. The calculations involved as a result of the number of particles involved in the DEM process as well as in the PBM simulation make the coupled process computationally intensive. Thus, there is a need to run these simulations in parallel on supercomputers. This was achieved by using RADICAL-Pilot, a python-based framework that supports the distributed execution of heterogeneous tasks on Supercomputers, Cloud and Grid. The work flow for this work can be seen in Figure 1.

This project can majorly be divided into 3 different studies. First the development of parallel DEM simulations using the open-source software LIGGGHTS®. Second was the development of hybrid parallel model for the PBM simulation that used a combination of 2 parallel programming interfaces namely Message Parsing Interface (MPI) and Open Multi-processing (OMP). The third component to the project being the development of an autonomous framework to couple these DEM and PBM simulations and execute them on a supercomputer without the need of manual data transfer and execution.

When DEM and PBM are coupled, the steady state DEM data is passed to the PBM. DEM studies performed took 54 hours based on 1 core for 10 seconds of simulation. The PBM was run on 1 core, which took approximately 40 minutes to simulate 100 seconds of the process. In this study, DEM studies ran in parallel using 256 cores, which was about 11 times faster than the single core studies (Figure 2). The PBM was simulated for a total of 100 simulation seconds and was about 14 times faster when using 16 MPI cores (Figure 3). The coupled simulation demonstrated the same accuracy as the serial simulation but was completed at an order of magnitude faster.

The total time taken for the simulation using the RADICAL-Pilot (RP) was‎ determined from the time the DEM was first executed until the PBM execution‎ was completed. The time of the individual DEM and PBM simulation was added to obtain‎ the total time required for a single one-way coupling simulation. ‎It was observed that the times‎ taken by RP are slightly higher than the summation of times of the‎ individual simulations. The difference is due to DEM simulations restarts.  Communication time required for RP to realize that the‎ DEM simulation has completed and to link the data from the DEM to the PBM‎ before it was executed was negligible compared to time of individual component of each simulation. This overhead in time is lower than the‎ amount of time the job would spend in the job scheduler queue. ‎Thus, when a large set of such coupled simulations need to be run for process validation, and prediction, the entire job could upload ‎ as a single job using RADICAL-pilot which in turn reduces scheduler wait times on individual ‎jobs and therefore results in significantly faster simulation times.

Last Modified: 07/25/2018
Modified by: Marianthi Ierapetritou