By Zed A. Shaw

I got to the end of my post about epoll vs. poll with regards to Mongre2 and was too damn tired to tell you how to repeat my tests. A cornerstone of science is that other people need to be able to repeat your experiments and report their own results. Another key piece of science is that there's always a probability that you are wrong.

I also consider much of my posting about Mongrel2 a side goal of educating people about building servers and the research involved. Whether I'm right or wrong I'm at least getting people to think and question what they've believed all these years, and teaching them something about my work. Hopefully that inspires many other people to do new things and take risks that might make them turn out wrong.

Today I hope people take my fully disclosed test gear and try to prove me wrong. Yes, I want you to prove me wrong so that I don't infect anyone with some weird ass idea that's actually going to hold people back for decades. I am going to make sure right now that everyone understands I am in no way advocating that I am right. I am merely showing an interesting observation which leads me to an idea I might try.

This is important, because I am offering up raw stats and data, my full R environment, the full C code to the tests, my shell script that runs it all, and letting you have at it. You could have a chance to be the guy that put me in my place, and I want you do to it. I want you to prove me wrong because then I won't feel like I've been living the "epoll lie" for 8+ years.

So let's begin with getting you my stuff.

First The Full Code

You can download the entire environment, graphs, data, C code, scripts and everything I had at:

• http://mongrel2.org/static/epollio.tar.gz

In there you will also find my full R environment so you can scroll back through my commands, look at the data I had, and see if you can spot any mistakes I made.

The pipetest.c File

If you look at pipetest.c you'll see the test suite I ran to do the test. You can also look at pipetestorig.c which is the original test that I used at first. All I did with pipetest.c is remove any AIO code, so you can diff it to see if I did it right.

Remember that I DID NOT WRITE THIS PIPETEST.C FILE AT ALL. This test has been used for about 8 years as the gold standard test for comparing poll to epoll. If you do a google search for pipetest.c you'll find it mentioned quite frequently.

This means that since I did not write this test, and since it is used by everyone to do the standard poll/epoll comparison, if it is wrong then the entire premise we're operating under is false. My tests assume that this test is accurate simply because everyone else is using it, but this could be entirely false and the whole thing is suspect.

If you find that it's written wrong, then we've got some bigger problems than just what I've got here, and all of what I've done is probably false.

Running A Test Run

The run.sh file has the last run I did which was a final repeat of the 10k run, after doing 1k, 10k, and some random runs. You should be able to edit it to adjust your parameters.

It will take a long time, but the key parameter is the actives list, which determines how many of the pipes are made active on each generation. This gives you your active/total list.

Looking At The Data

The end result of a run.sh is that you get a run.tab. It's easy to load this up into R, which I'll demonstrate in a bit, but you can use anything you like. It might need some reworking to fit into your particular tool you like to use. I have backups of my raw data in run1000.tab and run10k.tab so you can confirm that I did it right.

Remember, I assume that I have done this wrong so if you find an error that's awesome.

Another thing to keep in mind is this doesn't test any kind of raw performance. It is simply comparing the ratio of epoll/poll performance to active/total FDs. If you want to look at actual performance then go ahead and play with the data. It's quite interesting.

The Graphs

There are two .png files that are the original graphs I posted, and they should be exactly as-is. It might be good for you to try to recreate those graphs from my existing data using R just to make sure you're using R right, and hell, to make sure I'm using R right.

Using R To Make The Graphs

I won't get into a whole course on R, but here's the little bit of code you need to make a graph once you get your run.tab created:

pve <- read.table("run10k.tab", header=F)

names(pve) <- c("mode", "run", "total", "active", "buffer", "tps")

poll <- pve[pve$mode == 'poll',]
epoll <- pve[pve$mode == 'epoll',]

aratio <- epoll$active / 10000

tpsrat <- epoll$tps / poll$tps

plot(tpsrat ~ aratio, ylab="epoll X faster than poll",
xlab="active/total fds", main="poll,epoll by ratio of active sockets
(10k total)")

abline(h=1.0, col='red')

When you run R, you'll actually be able to use your arrow keys (like with readline) to scroll back and see all the things I did. If you spot an error let me know.

Making Your Own Graph

If you produce results, whether they are the same or different then you can share your graph with others. The idea is to not just share that you found results that contradict mine, but to share either way. That way observations from multiple replications can be compared and you can see how you compare to others. If everyone else confirms my findings but you, then you are probably doing something wrong. If everyone confirms your findings and they contradict mine, then I'm wrong.

To make your own graph in R it's pretty easy:

png("mygraph_10k.png")

plot(tpsrat ~ aratio, ylab="epoll X faster than poll",
xlab="active/total fds", main="poll,epoll by ratio of active sockets (10k total)")

abline(h=1.0, col='red')

dev.off()

After that you should have a file "mygraph_10k.png" and you can shoot it at me on twitter @zedshaw.

A Final Statement

Some people have seriously misconstrued my blog post to mean that I've made some huge discovery that will upset the world of event based IO systems. Hardly. I'm not advocating anyone do anything, and I assume that I'm wrong. Until there's lots of people who have tested the results and confirmed or found counter evidence I'm going to assume it's all bullshit.

What this test does show me however is that using both poll and epoll or kequeue probably wouldn't harm performance. I really didn't make that idea clear in the last post, but if you think about it that's the important take away. Instead of going all or nothing with insane wrappers and callbacks just to avoid poll, I'm wondering if a more viable option is that you use poll and epoll together in "some way".

The hand wavy bullshit part is the "some way". I'm thinking superpoll might be fun to try. Another is to just put the epoll FD into poll and then keep the number of FDs in poll tight and active. Apparently kqueue also has a poll FD so this could be a decent alternative.

So keep this in mind, nobody's saying use poll. Nobody's saying use epoll. What I'm saying is there's a metric ATR (active/total ratio) that seems to indicate when to use either, and the remaining question is how well does that metric work in practice. My idea is then to try combining the two, since this at least shows that you won't lose performance by doing that.