PyConFr 2014

Lyon, October 25-28

PyConFR 2014

samedi 11:30:00–12:00:00

Why SCons is not slow

Dirk Bächle

Audience level:
Intermediate

Description

This talk presents the results of analysing and profiling the runtime of SCons, a build system that is rumored to be slow. We found that this isn't true, and can tell you which trap we fell into...so that you possibly don't with your own Python application.

Abstract

SCons (www.scons.org) is a build system, entirely written in Python. It offers cross-platform support for languages like C/C++, D and Fortran, and uses content-based change detection (MD5) over the traditional timestamp approach. Despite all these obvious advantages over the classic "make", many people reject SCons as a build system because it is said to get very slow in large projects (10000+ files). That's why the SCons team stepped up and closely investigated some of the benchmarks on the internet that helped to spread the bad rumors. The results we found by very intensive and large-array profilings, boil down to the following point: It's actually not SCons that is slow, but the underlying process spawning mechanism (fork) that's causing the problems. The major slowdown of the running time can affect any other application (Python, or even a plain C program) that spawns single processes sequentially, while the memory usage of the parent process grows.

Against this unwanted effect, we developed a simple wrapper which redirects the standard calls of the subprocess module to posix_spawn() directly. It's going to be integrated to our source base in the upcoming release (Posix systems only), and will reduce the runtime for clean builds by a significant amount.

With this presentation we'd like to make our results and findings publicly available to a broader audience. We'd also like to get in touch with users and projects that experienced similar problems and discuss our and their solutions with them.

Voir le support de présentation

Sponsors