ChatGPT解决这个技术问题 Extra ChatGPT

Why shouldn't I use PyPy over CPython if PyPy is 6.3 times faster?

I've been hearing a lot about the PyPy project. They claim it is 6.3 times faster than the CPython interpreter on their site.

Whenever we talk about dynamic languages like Python, speed is one of the top issues. To solve this, they say PyPy is 6.3 times faster.

The second issue is parallelism, the infamous Global Interpreter Lock (GIL). For this, PyPy says it can give GIL-less Python.

If PyPy can solve these great challenges, what are its weaknesses that are preventing wider adoption? That is to say, what's preventing someone like me, a typical Python developer, from switching to PyPy right now?

Purged comments because most were things that should either be fleshed out in answers (and in some cases are), or shouldn't be said at all. Also edited to address a couple of the concerns raised regarding the subjectivity of this question. Please try to answer using facts, and back up assertions with sources if possible!
I've been using Pypy a lot. It tends to work very well. However, while Pypy is quite a bit faster for many CPU-heavy workloads, it's actually slower for the I/O-heavy workloads I've thrown at it. For example, I wrote a deduplicating backup program called backshift. For an initial backup, which does lots of file chunking, pypy is great. But for subsequent backups which are mostly just updating timestamps, CPython is faster.

V
Veedrac

NOTE: PyPy is more mature and better supported now than it was in 2013, when this question was asked. Avoid drawing conclusions from out-of-date information.

PyPy, as others have been quick to mention, has tenuous support for C extensions. It has support, but typically at slower-than-Python speeds and it's iffy at best. Hence a lot of modules simply require CPython. PyPy doesn't support numpy. Some extensions are still not supported (Pandas, SciPy, etc.), take a look at the list of supported packages before making the change. Note that many packages marked unsupported on the list are now supported. Python 3 support is experimental at the moment. has just reached stable! As of 20th June 2014, PyPy3 2.3.1 - Fulcrum is out! PyPy sometimes isn't actually faster for "scripts", which a lot of people use Python for. These are the short-running programs that do something simple and small. Because PyPy is a JIT compiler its main advantages come from long run times and simple types (such as numbers). PyPy's pre-JIT speeds can be bad compared to CPython. Inertia. Moving to PyPy often requires retooling, which for some people and organizations is simply too much work.

Those are the main reasons that affect me, I'd say.


Nice that you mention retooling. My web host, for example, has a choice between Python 2.4 and 2.5; and a "major producer of entertainment software" near me is using 2.6 with no plans to upgrade soon. Sometimes it can be a major, costly effort to even discover the cost of a conversion.
PyPy being "as fast as C" is more about generic C than highly optimized multithreaded cache-aware C libraries used for numerics. For numerics, Python is just used to ferry around pointers to big arrays. So PyPy being "as fast as C" means "your pointers+metadata get moved around as fast as C". Not a big deal. Then why bother with Python at all? Go look at the function signatures in cblas and lapacke.
@cjordan1: I don't get what you're saying. The high level numpy constructs are extremely expressive (np.sum(M[1:2*n**2:2, :2*n**2] * M[:2*n**2:2, :2*n**2].conjugate(), axis=1)?) in Python and that makes Python very suitable for the scientific community. Additionally, doing the non-intensive parts in Python and shelling out to C for the smaller intensive loops is a common and usable strategy.
@Veedrac That's what I meant. As in "Go look at the function signatures in cblas and lapacke" because they're so long and difficult to use that you'll instantly understand why we use Python to ferry around the pointers and metadata.
@tommy.carstensen This isn't really a good place to go in depth, but I'll try. 1. This was a lot more true when I wrote it than it is now. 2. "Scripts" are oft IO-heavy. PyPy's IO is still often slower than CPython's - it used to be significantly slower. 3. PyPy used to be slower than CPython at handling strings - now it's often better and rarely worse. 4. Many "scripts" are just glue code - making the interpreter faster won't improve overall runtimes in that case. 5. PyPy's warmup times used to be larger - short running scripts rarely managed to produce a lot of hot code.
s
spookylukey

That site does not claim PyPy is 6.3 times faster than CPython. To quote:

The geometric average of all benchmarks is 0.16 or 6.3 times faster than CPython

This is a very different statement to the blanket statement you made, and when you understand the difference, you'll understand at least one set of reasons why you can't just say "use PyPy". It might sound like I'm nit-picking, but understanding why these two statements are totally different is vital.

To break that down:

The statement they make only applies to the benchmarks they've used. It says absolutely nothing about your program (unless your program is exactly the same as one of their benchmarks).

The statement is about an average of a group of benchmarks. There is no claim that running PyPy will give a 6.3 times improvement even for the programs they have tested.

There is no claim that PyPy will even run all the programs that CPython runs at all, let alone faster.


Of course there is no claim that PyPy will run all Python code faster. But if you take all pure Python application I can bet that significant majority of them will run much faster (>3x times) on PyPy then on CPython.
Neither of your first two bullet points make sense. How can you say that benchmarks say "absolutely nothing about your program". It's pretty obvious that benchmarks aren't a perfect indicator of all real applications, but they can definitely be useful as an indicator. Also I don't understand what you find misleading about them reporting the average of a group of benchmarks. They state pretty clearly it's an average. If a programmer doesn't understand what an average is then they have much more serious concerns than language performance.
@SeanGeoffreyPietz - I wasn't claiming PyPy's site was in any way misleading - they have presented their results accurately. But the original question misquoted them, and was demonstrating that the author didn't understand the importance of the word 'average'. Many of the individual benchmarks are not 6.3 times faster. And if you use a different type of average you get a different value, so "6.3 x faster" is not an adequate summary of "geometric average is 6.3 x faster". "Group A is Z times faster than group B" is too vague to be meaningful.
-1: @spookylukey You seem to suggest that the benchmark suite is biased without providing evidence to support the claim. Criticism should always be backed up with evidence!
@EvgeniSergeev - no, I'm implying that all benchmarks are biased! Not necessarily deliberately, of course. The space of possible useful programs is infinite and incredibly varied, and a set of benchmarks only ever measures the performance on those benchmarks. Asking "how much faster is PyPy than CPython?" is like asking "how much faster if Fred than Joe?", which is what the OP seems to want to know.
T
Tritium21

Because pypy is not 100% compatible, takes 8 gigs of ram to compile, is a moving target, and highly experimental, where cpython is stable, the default target for module builders for 2 decades (including c extensions that don't work on pypy), and already widely deployed.

Pypy will likely never be the reference implementation, but it is a good tool to have.


According to pypy.org/download.html, PyPy needs 4 GB of RAM to compile (on a 64-bit system), not 8. And there's an option on that page to do it under 3 GB if needed.
@knite 1: that's new as of 2015, the documentation has historically read 8 GB. 2: in practice in 2015 you still need at least 8, with 6-7 free.
The memory requirement to compile is not so relevant if you use a build or distribution. As to "moving target, and highly experimental", can you give a couple of examples of stuff that breaks? Again, if people are using release builds rather than nightly builds or source, don't they have a reasonable expectation of functionality?
@smci This is an ancient question based on ancient data, with ancient answers. Consider this question and every answer to be historical for the state of pypy 4 years ago.
@Tritium21: I'm only interested in the current answer. What is it? You might like to edit your answer to say "As of 2013, comparing pypy vs version 2.x of Python was..." Also if the "6.3x geometric-average" claim in the question is out-of-date (as of 4/2017 they claim 7.5x, but even then depends on the benchmarks...), then that needs editing too (version numbers, latest data, etc.) I think the benchmark suite is not very relevant, hardly anyone would run raytracing in a scripting language on a CPU these days. I did find pybenchmarks.org
B
BrenBarn

The second question is easier to answer: you basically can use PyPy as a drop-in replacement if all your code is pure Python. However, many widely used libraries (including some of the standard library) are written in C and compiled as Python extensions. Some of these can be made to work with PyPy, some can't. PyPy provides the same "forward-facing" tool as Python --- that is, it is Python --- but its innards are different, so tools that interface with those innards won't work.

As for the first question, I imagine it is sort of a Catch-22 with the first: PyPy has been evolving rapidly in an effort to improve speed and enhance interoperability with other code. This has made it more experimental than official.

I think it's possible that if PyPy gets into a stable state, it may start getting more widely used. I also think it would be great for Python to move away from its C underpinnings. But it won't happen for a while. PyPy hasn't yet reached the critical mass where it is almost useful enough on its own to do everything you'd want, which would motivate people to fill in the gaps.


I dont think C is a language that is going anywhere any time soon (I would be willing to say, it will not disappear in our lifetime). until there is another language that will run anywhere, we will have C. (note, the JVM is written in C. Even java, the language that "runs everywhere" needs C for its everywhereness.) Otherwise I agree with this post on most of its points.
@Tritium21: Yeah, I'm just editorializing there. I'm fine with C existing, but I think Python's dependence on C is hugely detrimental, and PyPy is a great example of why: now we have the chance to get faster Python, but we're tripped up by years of relying on C. It'd be much better for Python to stand on its own two feet. It's even okay if Python itself is written in C, but the problem is the existence of an extension mechanism that encourages people to extend Python in ways that depend on C.
double edge sword on that - part of what made python so popular is its ability to extend other applications and be extended by other applications. If you take that away, I don't think we would be talking about python.
@BrenBarn It is utter folly to claim that Python's dependence on C is detrimental. Without the C-API of Python, most of the really powerful libraries and great interop that Python gained in its formative teenage years (late 90s), including the entire numeric/scientific ecosystem and GUI interfaces, would not have been possible. Look around to get some perspective on the whole universe of usages of Python, before making such blanket statements.
@PeterWang All those libraries can be written in Python, however they wouldn't be as fast as they are. What BrenBarn is saying is that now we have a chance to make python fast enough so that those libs can be written in python but we are refusing to take that chance, because taking it means losing the ability to use the C libraries. I believe that's what he meant by detrimental, not that the existence of C libraries is a bad thing but that the only way to make fast libraries is using C.
E
Eric Urban

I did a small benchmark on this topic. While many of the other posters have made good points about compatibility, my experience has been that PyPy isn't that much faster for just moving around bits. For many uses of Python, it really only exists to translate bits between two or more services. For example, not many web applications are performing CPU intensive analysis of datasets. Instead, they take some bytes from a client, store them in some sort of database, and later return them to other clients. Sometimes the format of the data is changed.

The BDFL and the CPython developers are a remarkably intelligent group of people and have a managed to help CPython perform excellent in such a scenario. Here's a shameless blog plug: http://www.hydrogen18.com/blog/unpickling-buffers.html . I'm using Stackless, which is derived from CPython and retains the full C module interface. I didn't find any advantage to using PyPy in that case.


PyPy has many, carefully run benchmarks (unlike CPython unfortunately, which doesn't really have a user-facing benchmark suite at the moment). Of course for network traffic PyPy can't magically make anything faster.
Julian, it's worth noting that the PyPy folks have been focusing a lot of effort on improving the runtimes of that particular benchmark suite for years now. To some degree it seems that they are "overfitting" their optimizations to this set of benchmarks and, in my experience, aside from purely numerical computations (which are better off in Fortran or C99 anyway), I've never gotten PyPy to be more than ~2X faster than CPython.
@AlexRubinsteyn But the view of those working on PyPy has always generally been that if you find a case where PyPy is slower than CPython, and you can turn it into a reasonable benchmark, it has a good chance of being added to the suite.
I checked your blog. In your results, the plain-python pair of (pickle, StringIO) shows that pypy is ~6.8x faster over cpython. I think this is a useful result. In your conclusion, you point out (correctly) that pypy code (which is plain python!) is slower than C code (cPickle, cStringIO), not cpython code.
@gsnedders I have offered a benchmark based on rinohtype on multiple occasions. They have not yet added it to the suite.
佚名

Q: If PyPy can solve these great challenges (speed, memory consumption, parallelism) in comparison to CPython, what are its weaknesses that are preventing wider adoption?

A: First, there is little evidence that the PyPy team can solve the speed problem in general. Long-term evidence is showing that PyPy runs certain Python codes slower than CPython and this drawback seems to be rooted very deeply in PyPy.

Secondly, the current version of PyPy consumes much more memory than CPython in a rather large set of cases. So PyPy didn't solve the memory consumption problem yet.

Whether PyPy solves the mentioned great challenges and will in general be faster, less memory hungry, and more friendly to parallelism than CPython is an open question that cannot be solved in the short term. Some people are betting that PyPy will never be able to offer a general solution enabling it to dominate CPython 2.7 and 3.3 in all cases.

If PyPy succeeds to be better than CPython in general, which is questionable, the main weakness affecting its wider adoption will be its compatibility with CPython. There also exist issues such as the fact that CPython runs on a wider range of CPUs and OSes, but these issues are much less important compared to PyPy's performance and CPython-compatibility goals.

Q: Why can't I do drop in replacement of CPython with PyPy now?

A: PyPy isn't 100% compatible with CPython because it isn't simulating CPython under the hood. Some programs may still depend on CPython's unique features that are absent in PyPy such as C bindings, C implementations of Python object&methods, or the incremental nature of CPython's garbage collector.


This answer does not cite any benchmarks or provide references.
p
pts

CPython has reference counting and garbage collection, PyPy has garbage collection only.

So objects tend to be deleted earlier and __del__ is called in a more predictable way in CPython. Some software relies on this behavior, thus they are not ready for migrating to PyPy.

Some other software works with both, but uses less memory with CPython, because unused objects are freed earlier. (I don't have any measurements to indicate how significant this is and what other implementation details affect the memory use.)


It should be stressed that relying on __del__ being called early or at all is wrong even in CPython. As you put it, it usually works and some people take that to mean it's guaranteed. If anything that references the object is caught up in a reference cycle (which is rather easy - did you know that inspecting the current exception in a certain non-contrived way creates a reference cycle?) finalization is delayed indefinitely, until the next cycle GC (which may be never). If the object is itself part of a reference cycle, __del__ will not be called at all (prior to Python 3.4).
Overhead per object is higher in CPython, which matters a LOT once you start creating lots of objects. I believe PyPy does the equivalent of slots by default, for one thing.
S
Stephan Eggermont

For a lot of projects, there is actually 0% difference between the different pythons in terms of speed. That is those that are dominated by engineering time and where all pythons have the same amount of library support.


If your project is that simple, then obviously it doesn't matter, but the same could be said of any implementation of any language: if all you do is aggregate other libraries' functions via relatively performant ABIs, then it's all irrelevant.
It doesn't have anything to do with simple. In engineering time the feedback loop is important. Sometimes much more important than run time.
Well, you're speaking very vaguely (engineering time with no reference to what's being engineered, what the constraints are, etc.; feedback loop with no reference to what is being fed back to whom, etc.), so I'm going to bow out of this conversation rather than trade cryptic references.
Nothing vague here. Take a look at the OODA loop, or PDCA.
@user Well, any run once project that takes a month to write, and a minute to run, will have a overall 0.0% speed up (1month+1min vs 1month) from using PyPy, even if PyPy were a thousand times faster. Stephan wasn't claiming that all projects would have a 0% speed up.
Y
Yishen Chen

To make this simple: PyPy provides the speed that's lacked by CPython but sacrifices its compatibility. Most people, however, choose Python for its flexibility and its "battery-included" feature (high compatibility), not for its speed (it's still preferred though).


"battery-included" means large standard library, AFAIK
l
lifolofi

I've found examples, where PyPy is slower than Python. But: Only on Windows.

C:\Users\User>python -m timeit -n10 -s"from sympy import isprime" "isprime(2**521-1);isprime(2**1279-1)"
10 loops, best of 3: 294 msec per loop

C:\Users\User>pypy -m timeit -n10 -s"from sympy import isprime" "isprime(2**521-1);isprime(2**1279-1)"
10 loops, best of 3: 1.33 sec per loop

So, if you think of PyPy, forget Windows. On Linux, you can achieve awesome accelerations. Example (list all primes between 1 and 1,000,000):

from sympy import sieve
primes = list(sieve.primerange(1, 10**6))

This runs 10(!) times faster on PyPy than on Python. But not on windows. There it is only 3x as fast.


Interesting! Some more comparisons and numbers would have been great.
q
qwr

PyPy has had Python 3 support for a while, but according to this HackerNoon post by Anthony Shaw from April 2nd, 2018, PyPy3 is still several times slower than PyPy (Python 2).

For many scientific calculations, particularly matrix calculations, numpy is a better choice (see FAQ: Should I install numpy or numpypy?).

Pypy does not support gmpy2. You can instead make use of gmpy_cffi though I haven't tested its speed and the project had one release in 2014.

For Project Euler problems, I make frequent use of PyPy, and for simple numerical calculations often from __future__ import division is sufficient for my purposes, but Python 3 support is still being worked on as of 2018, with your best bet being on 64-bit Linux. Windows PyPy3.5 v6.0, the latest as of December 2018, is in beta.


M
Martin Thoma

Supported Python Versions

To cite the Zen of Python:

Readability counts.

For example, Python 3.8 introduced fstring =.

There might be other features in Python 3.8+ which are more important to you. PyPy does not support Python 3.8+ at the moment.

Shameless self-advertisement: Killer Features by Python version - if you want to know more things you miss by using older Python versions


But is PyPy faster than CPython for the same Python versions - I can understand using python3.7 and 3.8 and getting more benefits, but if I can use PyPy on the side for some project, to bypass GIL and have faster parallel processing in case of CPU oriented processes
As of today 9/OCT/21, PyPy support or is compitable with python3.7 and now the team is working toward supporting python3.8. Ref pypy.org/posts/2021/04/…
@GhassanMaslamani Thank you! I've updated it :-)
@aspiring PyPy has a GIL.