Find the answer to your Linux question:
Results 1 to 6 of 6
Hello, Let's say that we have a system with debian 5, that we would like to upgrade it, in order to run it on new hardware. What would be the ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Just Joined!
    Join Date
    Jul 2011
    Posts
    5

    kernel upgrade vs dist-upgrade


    Hello,

    Let's say that we have a system with debian 5, that we would like to upgrade it, in order to run it on new hardware.
    What would be the most preferable solution? 3.6.x kernel upgrade (vanilla compile from kernel.org) or dist-upgrade?
    We tried the former solution (because it would be much faster) and worked fine. The system was performing ok, until we did a pgbench where the results were disappointing (much slower than the previous kernel, almost half!).

    By googling it, we saw a lot of posts saying that due to a scheduler/spinlock change (with kernels >=3.5.3), pgsql would have a 20% performance hit. We tried other kernels too (3.2, 3.4, 3.5.2) but they were doing exactly the same problem.
    So, it looks like this "bug" was not the problem in our case.
    The whole situation rose a thought whether upgrading just the kernel may not be enough after all and that the best thing to do is a userland upgrade as well.

    What do you think about that?

    Thanks

  2. #2
    Linux Newbie
    Join Date
    Apr 2005
    Location
    Clinton Township, MI
    Posts
    104
    It may not be a distribution version that is responsible for performance related issues, but I would say that you ought to seriously consider using a different Debian version anyway. Debian Squeeze, 6.0, has already had its sixth update to 6.0.6, and Debian Squeeze is hardly state of the art, so Debian Lenny, 5.0, is even older. If it were me, I'd be looking at Debian Squeeze 6.0.6 if I was running a stable production system, and Debian Wheezy, 7.0, which is at Beta 3, if I were in development or testing, preparing for a new system. Debian Wheezy is already pretty solid, and while not quite ready yet for "Stable" status as Debian defines it, this release will be the stable one, probably no later than early 2013. Debian 5.0 is truly technology that is 3-4 years old, is associated with Lenny --> OldStable software archives, so to me, that is too out of date, since even released versions of Debian are on the stable, aging end of the Linux software cycle.

    I doubt that any of this has any serious impact on performance concerns; I suppose it could, though, if your system requires any technology that was not in widespread use at the time that Lenny (5.0) was released. I think your kernel related issue is, more than likely, due to size and feature creep in the kernel, but may also be partially due to the recent kernel regressions. If that is important to you, I'd get more familiar with what's going on at kernel.org, track what's going on, and maintain your own kernel. Personally, I think that's spending time where time could be better spent elsewhere. Spend your money on faster systems and utilize your expensive expertise and apply it to solving business problems, unless you're a kernel developer, in which case, spend your time doing performance studies and apply them to kernel code improvements.
    Brian Masinick
    masinick AT yahoo DOT com

  3. #3
    Just Joined!
    Join Date
    Jul 2011
    Posts
    5
    Thank you for your reply.
    We had tried wheezy b3 too, but the results were not good. We got almost 25-30% performance drop.
    But the interesting question is not why we have a performance drop from debian 5 (2.6.26) to debian 7 (3.2.0).
    It's why we get so much performance hit (almost 3 times) when we compile any of the vanilla kernels, that makes me wondering.
    The same thing happens even on wheezy b3 (we tried compiling several latest kernel trees, even 2.6), and the bencmark is again very poor:
    pgbench with lenny bundled kernel (2.6.26-2-686): 157 tps
    pgbench with wheezy bundled kernel (3.2.0-3-686-pae): 113 tps
    pgbench with wheezy 3.0.latest vanilla kernel: 43 tps
    pgbench with wheezy 3.6.latest vanilla kernel: 43 tps
    pgbench with wheezy 2.6.latest vanilla kernel: 43 tps
    It looks like pgsql breaks completely as soon as we use any other kernel than the debian's.
    (Other than pgsql, the system is performing ok.)

    This was my original question, what could be the cause of this problem?
    (At the moment, we are trying to run some tests by compiling pgsql with each compiled kernel to see if there will be any change.)

  4. #4
    Linux Newbie
    Join Date
    Apr 2005
    Location
    Clinton Township, MI
    Posts
    104
    I do not know enough about pgsql, other than the high level description that it is an advanced database consisting of both object oriented and relational database methodologies. Clearly some combination of CPU and I/O workload are what get impacted by the different kernels. I have to wonder if it is filesystem performance that is being adversely affected. Here is a question: If the kernel and the system are being updated, are you also updating the pgsql software to keep up with the changes? Taking a quick look at the documentation, there have been a number of things changed in recent editions of pgsql that suggest some significant performance improvements; at least that is what their documentation claims. Could these changes possibly take advantage of system kernel changes, rather than being adversely impacted by the performance overhead introduced by other kernel features? There are usually trade offs to be made, so if you have not looked into the features of pgsql that have changed between Version 9.2, as opposed to versions dating back to Version 7.4, I'd suggest doing that as well, as opposed to considering only this one particular benchmark, which I know nothing about.

    Again, your choice on what to do, but I'd make sure you are getting a complete picture of what's going on. From where I sit, it seems to me that your application software could very well be three or four years out of date: the system, the kernel, and by implication, the back end software as well. If that is a correct assessment, I suggest examining ALL of these areas, not just focusing on the kernel alone.
    Brian Masinick
    masinick AT yahoo DOT com

  5. #5
    Just Joined!
    Join Date
    Jul 2011
    Posts
    5
    First of all, pgbench db needs to be recreated each time you boot with another kernel (which is very suspicious), otherwise the results are always very low!! So, my results on my previous post were misleading. As soon as I figured this out, pgbench with kernel 3.6.6 on wheezy was ok (around 110).
    Unfortunately, it didn't do the same on debian 5 (still around 60).
    This means that the problem seems to be related with some libs of the debian 5, where with new kernels it doesn't perform well.
    Doing an strace at pgbench I noticed that it's using rt_sigaction system call whereas on debian wheezy it doesn't.
    Perhaps that's the cause of the problem (we'll see if we can upgrade just those libs -identified by ldd for example-).
    Btw, newer versions of pgsql (9.x) were performing pretty much the same (not good).
    But, we can always disable pgsql's fsync (or similar settings) that improve performance...

  6. #6
    Linux Newbie
    Join Date
    Apr 2005
    Location
    Clinton Township, MI
    Posts
    104
    My overwhelming thought here is to get all of the software to the most currently supported version; Debian is never cutting edge, except in "Debian Sid" form, and even there, it is fairly well tested before packaged (but not 100% validated with other Debian packages, and that's why they call it Sid, named after the unstable kid from the Toy Story movie. The 3.6.6 kernel addresses a few regressions that were recently discovered and corrected; I am not sure to what extent, if any, that previous kernels have been patched with back ported fixes of the issues, but if it were me, I'd want the best available fix.

    Same with the application software; I'd want 3-4 years of improvements, unless it's just bloatware, but in the case of both kernel and the DB software, there have been major changes. I'd work with the latest of both, then try to figure out any performance issues you find there. But to not run the latest software, you potentially put your systems at risk for any security or functional issues that have been fixed in the latest versions, but possibly not in earlier versions. That's more important than a few machine cycles. You spend more people time micro-managing performance than you gain by keeping your hardware and software up to date.

    Just my opinion, of course, but I really think you're spending too much time on this and not enough on getting and keeping both your hardware and software up to date.
    Brian Masinick
    masinick AT yahoo DOT com

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •