Hi! Please consider following me on twitter: @hanekomu.

2008年11月25日

HTTP::Engine Conference #1 and Shibuya.pm #10

This week sees two conferences in Tokyo — they both will only last for a few hours. The first is HTTP::Engine conference #1, or HEcon1 for short. Only about 20 to 30 attendees are expected, so it's more like a tech meet with a common theme. The talks will of course be in Japanese, but you will probably get the gist from the slides. They will be streamed live over ustream and will probably be uploaded to Nico Nico Douga.

The second conference is the Shibuya.pm #10 meeting, and it is rather larger in scale. It too will only last for a few hours, but more than 100 attendees are expected, and there will be remote venues in Osaka/Kyoto and in Fukuoka. These talks will be streamed as well.

Both conferences will start in the evening in Tokyo, so they will be nice for me to watch around lunchtime at work.

Tags: , , .

Write a comment | Bookmark and Share

posted at: 20:16 | path: /misc | permalink | 0 comments | 0 trackbacks

2008年11月04日

Persuading the Cloud to run your benchmarks

I've had quite a bit of feedback about the accessor generator benchmarks; mostly requests to add this or that module to the benchmarks. Adding the code, rerunning the benchmarks and updating the blog entries took quite a bit of time, so I thought I could automate the process.

I've made the accessor generator benchmarks into a CPAN distribution called App::Benchmark::Accessors. That way you can run them yourself and the prerequisite modules will install automatically as well (if you use the CPAN shell).

At first I thought about simply having the benchmark programs in the bin/ directory, but then Ricardo Signes (rjbs) had a really cool idea: Run the benchmarks as part of the test suite and report them in a way that doesn't interfere with the TAP output.

That way the CPAN testers will run your benchmarks for you and you will get nice results for many different platforms and different perl versions. For example, take a look at the benchmark reports for App::Benchmark::Accessors.

The foundation for running benchmarks in test suites is in App::Benchmark, so you can use that to easily write your own cloudy benchmarks. However, be nice to the CPAN testers and don't run benchmarks that will take forever or use all of their machines' resources.

Tags: .

Write a comment | Bookmark and Share

posted at: 14:12 | path: /dev | permalink | 0 comments | 0 trackbacks

2008年11月01日

Dissecting the Moose Part 4 - Benchmarking Accessor Generators - Accessors Only

Update: Now uses Mojo 0.8.

Update 2: Now includes accessors and Class::MethodMaker.

Update 3: Now includes Object::Tiny, Spiffy, Class::Spiffy and direct hash access.

The previous benchmark tested mainly object construction; this benchmark performs a million get/set-operations on the same object.

As before, the benchmark was run on a 2.16 GHz MacBook with 2 GB RAM, Mac OS X 10.5.5, perl 5.8.8 (not the most recent one, I know):

Here is the program:

#!/usr/bin/env perl

use warnings;
use strict;
use Benchmark qw(cmpthese timethese :hireswallclock);


package WithMoose;
use Moose;
has myattr => ( is => 'rw' );


package WithMooseImmutable;
use Moose;
has myattr => ( is => 'rw' );
__PACKAGE__->meta->make_immutable;


package WithMouse;
use Mouse;
has myattr => ( is => 'rw' );


package WithClassAccessor;
use base qw(Class::Accessor);
__PACKAGE__->mk_accessors(qw/myattr/);


package WithClassAccessorFast;
use base qw(Class::Accessor::Fast);
__PACKAGE__->mk_accessors(qw/myattr/);


package WithClassAccessorFastXS;
use base qw(Class::Accessor::Fast::XS);
__PACKAGE__->mk_accessors(qw/myattr/);


package WithClassAccessorComplex;
use base qw(Class::Accessor::Complex);
__PACKAGE__->mk_new->mk_scalar_accessors(qw/myattr/);


package WithClassAccessorConstructor;
use base qw(Class::Accessor::Constructor Class::Accessor::Complex);
__PACKAGE__->mk_constructor->mk_scalar_accessors(qw/myattr/);


package WithMojo;
use base qw(Mojo::Base);
__PACKAGE__->attr('myattr');


package WithClassMethodMaker;
use Class::MethodMaker
    [ scalar => [ qw/myattr/ ],
      new    => [ qw/-hash new/ ],
    ];


package WithAccessors;
use accessors qw(myattr);

sub new { bless {}, shift }


package WithObjectTiny;
use Object::Tiny qw/myattr/;
sub set_myattr { $_[0]->{myattr} = $_[1] }


package WithSpiffy;
use Spiffy -base;
field 'myattr';


package WithClassSpiffy;
use Class::Spiffy -base;
field 'myattr';


package main;


my $moose                      = WithMoose->new;
my $moose_immutable            = WithMooseImmutable->new;
my $mouse                      = WithMouse->new;
my $class_accessor             = WithClassAccessor->new;
my $class_accessor_fast        = WithClassAccessorFast->new;
my $class_accessor_fast_xs     = WithClassAccessorFastXS->new;
my $class_accessor_complex     = WithClassAccessorComplex->new;
my $class_accessor_constructor = WithClassAccessorConstructor->new;
my $mojo                       = WithMojo->new;
my $class_methodmaker          = WithClassMethodMaker->new;
my $accessors                  = WithAccessors->new;
my $object_tiny                = WithObjectTiny->new;
my $spiffy                     = WithSpiffy->new;
my $class_spiffy               = WithClassSpiffy->new;
my $direct_hash                = {};

cmpthese(timethese(1_000_000,{
    moose => sub {
        $moose->myattr(27);
        my $x = $moose->myattr;
    },
    moose_immutable => sub {
        $moose_immutable->myattr(27);
        my $x = $moose_immutable->myattr;
    },
    mouse => sub {
        $mouse->myattr(27);
        my $x = $mouse->myattr;
    },
    class_accessor => sub {
        $class_accessor->myattr(27);
        my $x = $class_accessor->myattr;
    },
    class_accessor_fast => sub {
        $class_accessor_fast->myattr(27);
        my $x = $class_accessor_fast->myattr;
    },
    class_accessor_fast_xs => sub {
        $class_accessor_fast_xs->myattr(27);
        my $x = $class_accessor_fast_xs->myattr;
    },
    class_accessor_complex => sub {
        $class_accessor_complex->myattr(27);
        my $x = $class_accessor_complex->myattr;
    },
    class_accessor_constructor => sub {
        $class_accessor_constructor->myattr(27);
        my $x = $class_accessor_constructor->myattr;
    },
    mojo => sub {
        $mojo->myattr(27);
        my $x = $mojo->myattr;
    },
    class_methodmaker => sub {
        $class_methodmaker->myattr(27);
        my $x = $class_methodmaker->myattr;
    },
    accessors => sub {
        $accessors->myattr(27);
        my $x = $accessors->myattr;
    },
    object_tiny => sub {
        $object_tiny->set_myattr(27);
        my $x = $object_tiny->myattr;
    },
    spiffy => sub {
        $spiffy->myattr(27);
        my $x = $spiffy->myattr;
    },
    class_spiffy => sub {
        $class_spiffy->myattr(27);
        my $x = $class_spiffy->myattr;
    },
    direct_hash => sub {
        $direct_hash->{myattr} = 27;
        my $x = $direct_hash->{myattr};
    },
}));

To be fair, I've manually added a setter method to the Object::Tiny test class since Object::Tiny accessors only read, don't write. Here are the results:

Benchmark: timing 1000000 iterations of accessors, class_accessor, class_accessor_complex, class_accessor_constructor, class_accessor_fast, class_accessor_fast_xs, class_methodmaker, class_spiffy, direct_hash, mojo, moose, moose_immutable, mouse, object_tiny, spiffy...
 accessors: 1.9521 wallclock secs ( 1.94 usr +  0.01 sys =  1.95 CPU) @ 512820.51/s (n=1000000)
class_accessor: 4.21379 wallclock secs ( 4.18 usr +  0.02 sys =  4.20 CPU) @ 238095.24/s (n=1000000)
class_accessor_complex: 2.01732 wallclock secs ( 2.01 usr +  0.00 sys =  2.01 CPU) @ 497512.44/s (n=1000000)
class_accessor_constructor: 2.04444 wallclock secs ( 2.02 usr +  0.01 sys =  2.03 CPU) @ 492610.84/s (n=1000000)
class_accessor_fast: 2.04194 wallclock secs ( 2.01 usr +  0.01 sys =  2.02 CPU) @ 495049.50/s (n=1000000)
class_accessor_fast_xs: 0.744577 wallclock secs ( 0.74 usr +  0.01 sys =  0.75 CPU) @ 1333333.33/s (n=1000000)
class_methodmaker: 1.89544 wallclock secs ( 1.87 usr +  0.01 sys =  1.88 CPU) @ 531914.89/s (n=1000000)
class_spiffy: 2.05287 wallclock secs ( 2.05 usr +  0.01 sys =  2.06 CPU) @ 485436.89/s (n=1000000)
direct_hash: 0.405416 wallclock secs ( 0.40 usr +  0.00 sys =  0.40 CPU) @ 2500000.00/s (n=1000000)
      mojo: 1.71519 wallclock secs ( 1.70 usr + -0.01 sys =  1.69 CPU) @ 591715.98/s (n=1000000)
     moose: 1.89365 wallclock secs ( 1.87 usr +  0.01 sys =  1.88 CPU) @ 531914.89/s (n=1000000)
moose_immutable: 2.026 wallclock secs ( 2.00 usr +  0.02 sys =  2.02 CPU) @ 495049.50/s (n=1000000)
     mouse: 2.60332 wallclock secs ( 2.49 usr +  0.01 sys =  2.50 CPU) @ 400000.00/s (n=1000000)
object_tiny: 1.56868 wallclock secs ( 1.55 usr +  0.01 sys =  1.56 CPU) @ 641025.64/s (n=1000000)
    spiffy: 2.23854 wallclock secs ( 2.21 usr +  0.01 sys =  2.22 CPU) @ 450450.45/s (n=1000000)
                                Rate class_accessor mouse spiffy class_spiffy class_accessor_constructor class_accessor_fast moose_immutable class_accessor_complex accessors class_methodmaker moose mojo object_tiny class_accessor_fast_xs direct_hash
class_accessor              238095/s             --  -40%   -47%         -51%                       -52%                -52%            -52%                   -52%      -54%              -55%  -55% -60%        -63%                   -82%        -90%
mouse                       400000/s            68%    --   -11%         -18%                       -19%                -19%            -19%                   -20%      -22%              -25%  -25% -32%        -38%                   -70%        -84%
spiffy                      450450/s            89%   13%     --          -7%                        -9%                 -9%             -9%                    -9%      -12%              -15%  -15% -24%        -30%                   -66%        -82%
class_spiffy                485437/s           104%   21%     8%           --                        -1%                 -2%             -2%                    -2%       -5%               -9%   -9% -18%        -24%                   -64%        -81%
class_accessor_constructor  492611/s           107%   23%     9%           1%                         --                 -0%             -0%                    -1%       -4%               -7%   -7% -17%        -23%                   -63%        -80%
class_accessor_fast         495050/s           108%   24%    10%           2%                         0%                  --             -0%                    -0%       -3%               -7%   -7% -16%        -23%                   -63%        -80%
moose_immutable             495050/s           108%   24%    10%           2%                         0%                  0%              --                    -0%       -3%               -7%   -7% -16%        -23%                   -63%        -80%
class_accessor_complex      497512/s           109%   24%    10%           2%                         1%                  0%              0%                     --       -3%               -6%   -6% -16%        -22%                   -63%        -80%
accessors                   512821/s           115%   28%    14%           6%                         4%                  4%              4%                     3%        --               -4%   -4% -13%        -20%                   -62%        -79%
class_methodmaker           531915/s           123%   33%    18%          10%                         8%                  7%              7%                     7%        4%                --   -0% -10%        -17%                   -60%        -79%
moose                       531915/s           123%   33%    18%          10%                         8%                  7%              7%                     7%        4%                0%    -- -10%        -17%                   -60%        -79%
mojo                        591716/s           149%   48%    31%          22%                        20%                 20%             20%                    19%       15%               11%   11%   --         -8%                   -56%        -76%
object_tiny                 641026/s           169%   60%    42%          32%                        30%                 29%             29%                    29%       25%               21%   21%   8%          --                   -52%        -74%
class_accessor_fast_xs     1333333/s           460%  233%   196%         175%                       171%                169%            169%                   168%      160%              151%  151% 125%        108%                     --        -47%
direct_hash                2500000/s           950%  525%   455%         415%                       407%                405%            405%                   402%      387%              370%  370% 322%        290%                    87%          --

Some points of note:

Tags: .

Write a comment | Bookmark and Share

posted at: 12:58 | path: /dev | permalink | 0 comments | 0 trackbacks

Dissecting the Moose Part 3 - Benchmarking Accessor Generators

Update: Now also includes Mojo.

Update 2: Now uses Mojo 0.8.

Update 3: Now includes Class::MethodMaker.

Update 4: Now includes Object::Tiny, Spiffy, Class::Spiffy and direct hash access.

I wrote a small benchmarking program to compare the speeds of various accessor generators. They don't do much, just create an object while setting an accessor, then reading that accessor again, all for 100,000 times.

Here is the program:

#!/usr/bin/env perl

use warnings;
use strict;
use Benchmark qw(cmpthese timethese :hireswallclock);


package WithMoose;
use Moose;
has myattr => ( is => 'rw' );


package WithMooseImmutable;
use Moose;
has myattr => ( is => 'rw' );
__PACKAGE__->meta->make_immutable;


package WithMouse;
use Mouse;
has myattr => ( is => 'rw' );


package WithClassAccessor;
use base qw(Class::Accessor);
__PACKAGE__->mk_accessors(qw/myattr/);


package WithClassAccessorFast;
use base qw(Class::Accessor::Fast);
__PACKAGE__->mk_accessors(qw/myattr/);


package WithClassAccessorFastXS;
use base qw(Class::Accessor::Fast::XS);
__PACKAGE__->mk_accessors(qw/myattr/);


package WithClassAccessorComplex;
use base qw(Class::Accessor::Complex);
__PACKAGE__->mk_new->mk_scalar_accessors(qw/myattr/);


package WithClassAccessorConstructor;
use base qw(Class::Accessor::Constructor Class::Accessor::Complex);
__PACKAGE__->mk_constructor->mk_scalar_accessors(qw/myattr/);


package WithMojo;
use base qw(Mojo::Base);
__PACKAGE__->attr('myattr');


package WithClassMethodMaker;
use Class::MethodMaker
    [ scalar => [ qw/myattr/ ],
      new    => [ qw/-hash new/ ],
    ];


package WithObjectTiny;
use Object::Tiny qw/myattr/;


package WithSpiffy;
use Spiffy -base;
field 'myattr';


package WithClassSpiffy;
use Class::Spiffy -base;
field 'myattr';


package main;


cmpthese(timethese(100_000,{
    moose => sub {
        my $obj = WithMoose->new(myattr => 27);
        my $x = $obj->myattr;
    },
    moose_immutable => sub {
        my $obj = WithMooseImmutable->new(myattr => 27);
        my $x = $obj->myattr;
    },
    mouse => sub {
        my $obj = WithMouse->new(myattr => 27);
        my $x = $obj->myattr;
    },
    class_accessor => sub {
        my $obj = WithClassAccessor->new({ myattr => 27 });
        my $x = $obj->myattr;
    },
    class_accessor_fast => sub {
        my $obj = WithClassAccessorFast->new({ myattr => 27 });
        my $x = $obj->myattr;
    },
    class_accessor_fast_xs => sub {
        my $obj = WithClassAccessorFastXS->new({ myattr => 27 });
        my $x = $obj->myattr;
    },
    class_accessor_complex => sub {
        my $obj = WithClassAccessorComplex->new(myattr => 27);
        my $x = $obj->myattr;
    },
    class_accessor_constructor => sub {
        my $obj = WithClassAccessorConstructor->new(myattr => 27);
        my $x = $obj->myattr;
    },
    mojo => sub {
        my $obj = WithMojo->new(myattr => 27);
        my $x = $obj->myattr;
    },
    class_methodmaker => sub {
        my $obj = WithClassMethodMaker->new(myattr => 27);
        my $x = $obj->myattr;
    },
    object_tiny => sub {
        my $obj = WithObjectTiny->new(myattr => 27);
        my $x = $obj->myattr;
    },
    spiffy => sub {
        my $obj = WithSpiffy->new(myattr => 27);
        my $x = $obj->myattr;
    },
    class_spiffy => sub {
        my $obj = WithClassSpiffy->new(myattr => 27);
        my $x = $obj->myattr;
    },
    direct_hash => sub {
        my $h = {};
        $h->{myattr} = 27;
        my $x = $h->{myattr};
    },
}));

And here are the results, on a 2.16 GHz MacBook with 2 GB RAM, Mac OS X 10.5.5, perl 5.8.8 (yes, it's not the newest one):

Benchmark: timing 100000 iterations of class_accessor, class_accessor_complex, class_accessor_constructor, class_accessor_fast, class_accessor_fast_xs, class_methodmaker, class_spiffy, direct_hash, mojo, moose, moose_immutable, mouse, object_tiny, spiffy...
class_accessor: 0.680644 wallclock secs ( 0.67 usr +  0.00 sys =  0.67 CPU) @ 149253.73/s (n=100000)
class_accessor_complex: 0.890903 wallclock secs ( 0.88 usr +  0.01 sys =  0.89 CPU) @ 112359.55/s (n=100000)
class_accessor_constructor: 1.99036 wallclock secs ( 1.96 usr +  0.01 sys =  1.97 CPU) @ 50761.42/s (n=100000)
class_accessor_fast: 0.561965 wallclock secs ( 0.54 usr +  0.00 sys =  0.54 CPU) @ 185185.19/s (n=100000)
class_accessor_fast_xs: 0.4521 wallclock secs ( 0.44 usr +  0.00 sys =  0.44 CPU) @ 227272.73/s (n=100000)
class_methodmaker: 1.51813 wallclock secs ( 1.48 usr +  0.01 sys =  1.49 CPU) @ 67114.09/s (n=100000)
class_spiffy: 0.620832 wallclock secs ( 0.60 usr +  0.01 sys =  0.61 CPU) @ 163934.43/s (n=100000)
direct_hash: 0.116149 wallclock secs ( 0.12 usr +  0.00 sys =  0.12 CPU) @ 833333.33/s (n=100000)
            (warning: too few iterations for a reliable count)
      mojo: 1.00943 wallclock secs ( 0.95 usr +  0.01 sys =  0.96 CPU) @ 104166.67/s (n=100000)
     moose: 11.4151 wallclock secs (10.87 usr +  0.09 sys = 10.96 CPU) @ 9124.09/s (n=100000)
moose_immutable: 0.899679 wallclock secs ( 0.87 usr +  0.01 sys =  0.88 CPU) @ 113636.36/s (n=100000)
     mouse: 5.28056 wallclock secs ( 5.05 usr +  0.05 sys =  5.10 CPU) @ 19607.84/s (n=100000)
object_tiny: 0.382038 wallclock secs ( 0.31 usr +  0.01 sys =  0.32 CPU) @ 312500.00/s (n=100000)
            (warning: too few iterations for a reliable count)
    spiffy: 0.620199 wallclock secs ( 0.60 usr +  0.01 sys =  0.61 CPU) @ 163934.43/s (n=100000)
                               Rate moose mouse class_accessor_constructor class_methodmaker mojo class_accessor_complex moose_immutable class_accessor class_spiffy spiffy class_accessor_fast class_accessor_fast_xs object_tiny direct_hash
moose                        9124/s    --  -53%                       -82%              -86% -91%                   -92%            -92%           -94%         -94%   -94%                -95%                   -96%        -97%        -99%
mouse                       19608/s  115%    --                       -61%              -71% -81%                   -83%            -83%           -87%         -88%   -88%                -89%                   -91%        -94%        -98%
class_accessor_constructor  50761/s  456%  159%                         --              -24% -51%                   -55%            -55%           -66%         -69%   -69%                -73%                   -78%        -84%        -94%
class_methodmaker           67114/s  636%  242%                        32%                -- -36%                   -40%            -41%           -55%         -59%   -59%                -64%                   -70%        -79%        -92%
mojo                       104167/s 1042%  431%                       105%               55%   --                    -7%             -8%           -30%         -36%   -36%                -44%                   -54%        -67%        -87%
class_accessor_complex     112360/s 1131%  473%                       121%               67%   8%                     --             -1%           -25%         -31%   -31%                -39%                   -51%        -64%        -87%
moose_immutable            113636/s 1145%  480%                       124%               69%   9%                     1%              --           -24%         -31%   -31%                -39%                   -50%        -64%        -86%
class_accessor             149254/s 1536%  661%                       194%              122%  43%                    33%             31%             --          -9%    -9%                -19%                   -34%        -52%        -82%
class_spiffy               163934/s 1697%  736%                       223%              144%  57%                    46%             44%            10%           --    -0%                -11%                   -28%        -48%        -80%
spiffy                     163934/s 1697%  736%                       223%              144%  57%                    46%             44%            10%           0%     --                -11%                   -28%        -48%        -80%
class_accessor_fast        185185/s 1930%  844%                       265%              176%  78%                    65%             63%            24%          13%    13%                  --                   -19%        -41%        -78%
class_accessor_fast_xs     227273/s 2391% 1059%                       348%              239% 118%                   102%            100%            52%          39%    39%                 23%                     --        -27%        -73%
object_tiny                312500/s 3325% 1494%                       516%              366% 200%                   178%            175%           109%          91%    91%                 69%                    38%          --        -62%
direct_hash                833333/s 9033% 4150%                      1542%             1142% 700%                   642%            633%           458%         408%   408%                350%                   267%        167%          --

Some points of note:

Tags: .

Write a comment | Bookmark and Share

posted at: 10:54 | path: /dev | permalink | 0 comments | 0 trackbacks