HPC Software Engineering: "Do as I say, not as I do"

By Erik Lindahl

Stockholm University

Published on

Abstract

Most scientific software projects started as a quick hack on somebody's laptop, but before you know it there might be dozens of future students and postdocs involved. However, unlike 30 years ago, today few of us are only one supercomputer from a single vendor providing a single compiler. Many modern scientific codes use multiple languages and scipt languages, they might use accelerated hardware-specific code (or even GPU support), they can rely on a whole set of external libraries, and with multiple people involved it rapidly gets almost impossible to track versions and bugs without professional software engineering tools. In fact, the latter is frequently true even for the smallest projects: How many times have you forgotten what directory contained the "fixed" version of your code, have you ever had to correct the same bug twice, or have you ever been hit by a bug in a compiler? In this talk I will not focus on science, but describe the software engineering aspect of a large open source project, including modern tools for version control, archiving developer discussions and bug reports, accepting public patch submissions, code review and quality assurance, and not least fully automated continuous integration testing. You might not need all of these for your first project, but they can potentially save you a whole lot of time and grief even when starting out in HPC!

Bio

Erik Lindahl completed undergraduate studies in engineering physics at Lund University, after which he received a PhD in theoretical biophysics at the KTH Royal Institute of Technology in Stockholm in 2001. He has performed research at Groningen University, Stanford University and the Pasteur Institute after which he assumed a position as assistant and later associate professor at Stockholm University in 2004. Since 2010 he holds dual appointments as professor of theoretical biophysics at KTH and professor of computational structural biology at Stockholm University. Since 2011 he is a member of the Swedish Young Academy.

Apart from research on membrane proteins and Gromacs development, I simply cannot help hacking assembly code on new architectures :-)

Source

Cite this work

Researchers should cite this work as follows:

  • Erik Lindahl (2013), "HPC Software Engineering: "Do as I say, not as I do"," http://nanohub.org/resources/19415.

    BibTex | EndNote

Time

Location

Eisner-Lubin Room 401, New York University, New York, NY

Submitter

NanoBio Node

University of Illinois at Urbana-Champaign

Tags

HPC Software Engineering: \"Do as I say, not as I do\"
by: Erik Lindahl
  • HPC Software Engineering 1. HPC Software Engineering "Do a… 0
    00:00/00:00
  • Molecular Dynamics 2. Molecular Dynamics 66.436363636363637
    00:00/00:00
  • GROMACS 1995-2013 3. GROMACS 1995-2013 85.238095238095241
    00:00/00:00
  • Open Source & Free Software Development Models 4. Open Source & Free Software De… 157.14285714285717
    00:00/00:00
  • GROMACS 5. GROMACS 326.09523809523813
    00:00/00:00
  • What does OSS do for us? 6. What does OSS do for us? 433.04761904761904
    00:00/00:00
  • The Community 7. The Community 580.19047619047626
    00:00/00:00
  • Tutorials: www.gormacs.org 8. Tutorials: www.gormacs.org 664.47619047619048
    00:00/00:00
  • Contributions - Academia 9. Contributions - Academia 717.61904761904759
    00:00/00:00
  • Contributions - Industry 10. Contributions - Industry 773.23809523809518
    00:00/00:00
  • Funding Development 11. Funding Development 813.71428571428567
    00:00/00:00
  • License Considerations 12. License Considerations 917.5238095238094
    00:00/00:00
  • Development Challenges 13. Development Challenges 1062.3809523809523
    00:00/00:00
  • Political Incidents from an OSS Perspective 14. Political Incidents from an OS… 1252.8571428571429
    00:00/00:00
  • Software Engineering 15. Software Engineering 1369.8095238095239
    00:00/00:00
  • Our Mistakes 16. Our Mistakes 1384.952380952381
    00:00/00:00
  • The picture until early 2011 17. The picture until early 2011 1503.6190476190475
    00:00/00:00
  • CVS Branches 18. CVS Branches 1601.1428571428571
    00:00/00:00
  • The CVS/SVN limitation 19. The CVS/SVN limitation 1650.1904761904761
    00:00/00:00
  • What is a build chain? 20. What is a build chain? 1663.6190476190477
    00:00/00:00
  • GNU autotools 21. GNU autotools 1668.717842605156
    00:00/00:00
  • Configuration 22. Configuration 1744.2315789473685
    00:00/00:00
  • Some Examples of old Gromacs autotools files 23. Some Examples of old Gromacs a… 1747.5157894736842
    00:00/00:00
  • Problems 24. Problems 1753.3263157894737
    00:00/00:00
  • New Strategy from 2011 25. New Strategy from 2011 1794.6947368421054
    00:00/00:00
  • Better Source Control: GIT 26. Better Source Control: GIT 1806.6947368421054
    00:00/00:00
  • All Branches are equal 27. All Branches are equal 1866.2526315789476
    00:00/00:00
  • GIT 28. GIT 1897.1368421052634
    00:00/00:00
  • Redmine Issue tracking 29. Redmine Issue tracking 1923.2210526315789
    00:00/00:00
  • Redmine example 30. Redmine example 1959.6248303934872
    00:00/00:00
  • Gerrit Code Review 31. Gerrit Code Review 2108.4761904761908
    00:00/00:00
  • Example 32. Example 2138.5714285714284
    00:00/00:00
  • Example 33. Example 2323.8095238095239
    00:00/00:00
  • Jenkins Continuous Integration 34. Jenkins Continuous Integration 2372.4436713941664
    00:00/00:00
  • Gerrit also allows anybody to comment directly in diff-style webpages 35. Gerrit also allows anybody to … 2404.3503772003351
    00:00/00:00
  • CMake: Cross-platform replacement for Autoconf, Automake, Libtool 36. CMake: Cross-platform replacem… 2409.3250587051325
    00:00/00:00
  • What would we do if we had to start over today? 37. What would we do if we had to … 2476.7686293672123
    00:00/00:00
  • Languages? 38. Languages? 2523.5733972438588
    00:00/00:00
  • New level of network-centric programming w. Copernicus 39. New level of network-centric p… 2710.4334288928403
    00:00/00:00
  • Our current approach 40. Our current approach 2731.7969489742241
    00:00/00:00
  • Modularization 41. Modularization 2783.0856542301626
    00:00/00:00
  • Code Documentation 42. Code Documentation 2922.7068548613847
    00:00/00:00
  • So, has it helped? 43. So, has it helped? 2941.7099934683215
    00:00/00:00
  • Acknowledgments 44. Acknowledgments 2981.9787791707454
    00:00/00:00