Five years of Prospero

We started with the basics: How has Prospero transformed your research?

For Dr Rui Martiniano, Lecturer in Human Evolutionary Genetics, the answer was scale. His group works with some of the largest genomic datasets on Earth. “Prospero has enabled my group to process large-scale genetic datasets by running high-memory jobs, parallelising tasks across many CPUs, and storing huge genomic databases,” he explains.

For my work, which involves searching for a small number of very old, informative sequences in extremely large datasets, Prospero is like finding a needle in a haystack… Only much faster.

In astrophysics, Professor Rob Crain describes a similar leap. His group studies the formation and evolution of galaxies, star clusters, and their gaseous environments. Previously, they relied on external collaborations for computing power. “The advent of Prospero enabled us to bring the development work entirely in-house at LJMU,” he says.

This in turn has enabled us to be much more ambitious, and has led to the creation of several new state-of-the-art simulation suites that simply wouldn’t exist without Prospero.

For early-career researchers in the Astrophysics Research Institute, Prospero has been just as vital. PhD candidate Sian Phillips uses it to reduce galaxy data from the ALMA observatory work. Her colleague Katherine Ormerod highlights the shift in scale: “I use Prospero for large pipeline work, changing the execution time from days to around four hours.” Both noted that prior to the advent of Prospero, projects of this size would have been impossible to achieve.

Dr Renske Smit, Reader in Observational Astrophysics, stresses another practical dimension: secure, large-scale data storage.

Prospero allows storing and backing-up in line with data requirements from funding councils in a way that I could not facilitate for my group otherwise.

Beyond astrophysics and genetics, Prospero has also transformed engineering research. Dr David Hitchmough, Research Fellow in Marine Engineering, relies on computational fluid dynamics (CFD) simulations: “Because of the high computational demands of CFD, the use of Prospero has enabled us to carry out more and larger simulations than we could previously. Prospero was an invaluable tool in the completion of both my and Dr. Andrew Spiteri’s PhDs.”

Dr Andrew Burgess describes Prospero as a “game-changer” for modelling powder flow during 3D metal printing. “Compared to prior in-house engineering clusters at LJMU, it has allowed me to dramatically scale up the amount of simulations I could run, and develop closer fidelity that resulted in real-world impact for Additive Manufacturing.” Taken together, these reflections underline Prospero’s defining impact: it hasn’t just made research faster or more efficient. It has made entirely new kinds of research possible.

When asked for one standout project that wouldn’t have happened without Prospero, researchers offered an array of landmark examples.

For Rui Martiniano, the highlight was the detection of ancient megafauna DNA in cave sediments. Using databases of ~1.6 billion sequences from 450,000 species, his team identified genetic traces of cave hyenas and mammoths that offer new insights into Ice Age population histories.

In astrophysics, Rob Crain points to the E-MOSAICS suite of simulations, the first cosmological models to capture the formation and evolution of star clusters within their host galaxies. “This project has led to over 25 research papers, including some cited several hundred times,” he notes. The simulations even predicted that the Milky Way underwent a violent merger billions of years ago, a forecast since confirmed by telescope observations.

The Astrophysics Research Institute’s Dr Joris Witstok recalls analysing a galaxy only 330 million years after the Big Bang - one of the earliest known - which showed hydrogen emission piercing through the fog of the young universe. “This surprise finding has challenged researchers to explain how this light could have pierced the thick fog of neutral hydrogen,” he says. The result was published in Nature.

Meanwhile, in engineering, Prospero has powered urgent real-world innovation. Dr Hitchmough highlights the Retrofit55 project, which aims for a 55% reduction in greenhouse gas emissions from shipping by 2030. “The ability to simulate complex air lubrication phenomena has helped clarify our understanding of this decarbonisation technology,” he explains.

And for Andrew Burgess, the standout project was deeply personal: his entire PhD. “The volume of simulations that could be run, and the quantity of particles I could include, would not have been viable without Prospero,” he says.

Looking ahead, our contributors agree: the importance of supercomputing is only set to grow.

“In my field, bigger computers enable more ambitious calculations that afford clearer insight into the physics of the cosmos,” says Rob Crain. He also highlights the role of artificial intelligence: “Like many fields in the physical sciences, machine learning and artificial intelligence will have an exciting role to play in driving astrophysics, so the addition of GPU-accelerated nodes into Prospero has been an exciting development.”

For observational astrophysics, the shift is substantial, allowing for teams to apply far more accurate methods to their work. With forthcoming projects like ESA’s Euclid telescope producing nearly full-sky datasets, the capabilities for neural networks trained and deployment on HPC systems will be essential. “Put simply, supercomputing has and continues to change and improve the analysis of galaxy properties,” notes the team.

In engineering, both Hitchmough and Burgess emphasise growing demands. CFD simulations will become ever more complex, and additive manufacturing will move toward full end-to-end digital models of printing cycles. As Burgess puts it: “Advanced supercomputing has the potential to encompass more or even all of the process, aiding in the development of end-to-end simulations for metal 3D printing cycles.”

And in genetics, Martiniano predicts that supercomputing will simply become unavoidable: “As genomic datasets expand, supercomputing will be essential for handling their scale and complexity, and I expect that anyone routinely analysing data will eventually rely on it.”

Despite its complexity, Prospero’s community emphasises that HPC is not as intimidating as it seems.

Martiniano advises newcomers not to be put off by scripting:

Don’t be deterred by learning to write SLURM scripts, it’s quick to pick up, and it makes workflows faster, more reproducible, and more scalable.

Chris Rowe, a PhD researcher in astrophysics, puts it more practically:

Cache everything! If work can be broken up into chunks with intermediate outputs then do so. Even if you don’t use them, it means you won’t have to start over if you make an error.

The Astrophysics team stresses collaboration:

Ask for help from more experienced users as you get started so that you learn how to build efficient scripts and make the best use of the facility from the beginning.

For Hitchmough, the message is one of opportunity:

Supercomputing isn’t as daunting as it initially seems; the possible uses of Prospero and applications for the resources are varied, and many areas stand to benefit from utilising Prospero.

And Burgess offers a reminder of courtesy:

Do your research. More cores and nodes does not necessarily always mean you speed and scale up your job. Optimisation is a courtesy as much as a necessity.

Finally, we asked contributors to sum up Prospero in one sentence or metaphor. The answers were as varied as the fields they represent.

• “Like finding a needle in a haystack, only much faster.” — Martiniano
• “As a user, Prospero has been completely transformative for my work… exhausting to help build, but hugely gratifying.” — Crain
• “A cosmos cruncher!” — Smit
• “Challenging at first, but the rewards have been considerable.” — Hitchmough
• “Climbing a hill to see the view — writing software for HPC can be tricky, but it’s worth the effort.” — Rowe
• “Prospero made it possible.” — Burgess

Together, these voices paint a picture of a supercomputer that is more than hardware. Prospero has become a partner in discovery, training, and a cornerstone of LJMU’s research future.