Hi! Please consider following me on twitter: @hanekomu.

2010年05月28日

bash brace expansion

Last time we looked at bash parameter expansion. Now I would like to talk about bash brace expansion. This is mainly a blog about Perl, but Unix tools like shells and editors are part of a Perl programmer's life as well, so from time to time I'll write about those tools too.

Brace expansion in bash is a way of generating strings. The most general form is a list of strings within braces, separated by commas. There can be other strings before and/or after the braces; they will be included in all variations. For example:

$ echo {foo,baz,baz}
foo baz baz

$ echo prefix-{foo,bar,baz}-suffix
prefix-foo-suffix prefix-bar-suffix prefix-baz-suffix

$ for i in {foo,bar,baz}; do echo $i; done
foo
bar
baz

Brace expansion can be turned on and off using bash's B option. +B turns brace expansion off, -B turns it back on.

$ set +B

$ echo {foo,baz,baz}
{foo,baz,baz}

$ set -B

$ echo {foo,baz,baz}
foo baz baz

Here are some easy examples that show what is possible:

$ echo foo{,,,}
foo foo foo foo

$ echo {1..3}{1..3}{1..3}
111 112 113 121 122 123 131 132 133 211 212 213 221 222
223 231 232 233 311 312 313 321 322 323 331 332 333

$ echo {foo,bar,baz}{1..3}
foo1 foo2 foo3 bar1 bar2 bar3 baz1 baz2 baz3

$ echo {foo{1..3},bar{4..6},baz{7..9}}
foo1 foo2 foo3 bar4 bar5 bar6 baz7 baz8 baz9

$ cp /some/very/long/path/to/my.conf{,.bak}
cp /some/very/long/path/to/my.conf /some/very/long/path/to/my.conf.bak

$ mkdir -p /Foo-Bar/{bin,lib,eg,t}
mkdir -p /Foo-Bar/bin /Foo-Bar/lib /Foo-Bar/eg /Foo-Bar/t

$ mv foo{,-}bar
mv foobar foo-bar

You can also specify ranges using the {from..to} syntax, and ranges with steps using the {from..to..step} syntax. These ranges can be numeric or alphabetic.

$ echo {1..5}
1 2 3 4 5

$ echo {5..1}
5 4 3 2 1

$ echo {3..-3}
3 2 1 0 -1 -2 -3

$ echo {1..10..2}
1 3 5 7 9

$ echo {a..e}
a b c d e

$ echo {e..a}
e d c b a

$ echo {a..z..2}
a c e g i k m o q s u w y

One typical use for brace expansion is to tell wget or curl to download several files whose URLs conform to a certain pattern:

$ wget http://example.com/path/to/{014..017}.{html,png}
wget http://example.com/path/to/014.html http://example.com/path/to/014.png
http://example.com/path/to/015.html http://example.com/path/to/015.png
http://example.com/path/to/016.html http://example.com/path/to/016.png
http://example.com/path/to/017.html http://example.com/path/to/017.png

(bash version 4 retains leading zeros; earlier bash versions omitted them.)

I also use brace expansion in my ~/.bashrc to inspect certain directories to find out whether to they contain bin/, sbin/ or man/ subdirectories that should be added to $PATH and $MANPATH:

for d in {/usr,/opt,~}{,/{local,share,local/share,perl,perl/5.*.*}}
do
    test -d "$d/bin" && PATH="$d/bin:$PATH"
    test -d "$d/sbin" && PATH="$d/sbin:$PATH"
    test -d "$d/man" && MANPATH="$d/man:$MANPATH"
done

This effectively iterates over all of these directories:

/usr
/usr/local
/usr/share
/usr/local/share
/usr/perl
/usr/perl/5.*.*
/opt
/opt/local
/opt/share
/opt/local/share
/opt/perl
/opt/perl/5.*.*
~
~/local
~/share
~/local/share
~/perl
~/perl/5.*.*

Do you have interesting examples of using the bash brace expansion mechanism?

Write a comment | Bookmark and Share

posted at: 10:12 | path: /dev | permalink | 1 comment | 0 trackbacks

2010年05月27日

Aspect-Oriented Programming, Reloaded

Back in 2001, I wrote Aspect.pm, which brought Aspect-Oriented Programming (AOP) to Perl. It was a neat toy, but to be honest, for me that's all it ever was. I never used it even in tests, much less in production.

Now that Adam Kennedy++ has taken over maintenance of the Aspect distribution, he has rewritten its core, introduced new pointcuts and join point types and wrote a blog post in which he details the future plans for AOP in Perl.

Write a comment | Bookmark and Share

posted at: 12:02 | path: /dev | permalink | 0 comments | 0 trackbacks

2010年05月25日

bash parameter expansion

The bash shell, especially in version 4, has useful parameter expansion features. I've only just started to really use them, so I wanted to make a note of the ones that seem most interesting.

$ foo="hello world goodbye world"
$ echo "${foo/world/perl}"
hello perl goodbye world
$ echo "${foo//world/perl}"
hello perl goodbye perl
$ echo "${foo^}"
Hello world goodbye world
$ echo "${foo^^}"
HELLO WORLD GOODBYE WORLD

bar="HELLO"
$ echo "${bar,}"
hELLO
$ echo "${bar,,}"
hello

$ path="/path/to/the/script"
$ echo "dirname  = ${path%/*}"
dirname  = /path/to/the
$ echo "basename = ${path##*/}"
basename = script

As you can see, ${foo/bar/baz} takes the contents of the variable foo, substitutes the first occurrence of bar with baz and returns the result. The contents of foo are unaltered. If you want to replace all occurrences, not just the first, use ${foo//bar/baz}.

Starting with bash version 4, ${foo^} uppercases the first character, while ${foo^^} uppercases the whole string. Likewise, {$foo,} lowercases the first character and {$foo,,} lowercases the whole string.

${foo#pattern} tries to match the pattern from the start of the string and deletes the matching part of the string. ${foo##pattern} is the greedy version. In the example above, to get the directory path, we greedily delete everything from the start of the string to the final slash: ${path##*/}.

There is also ${foo%pattern}, which also deletes, but starts looking from the end of the string. ${foo%%pattern} is the greedy version. In the example above, to get the filename, we delete everything from the final slash — that is, we start looking from the end until we find a slash: ${path%/*}.

Using the above techniques you might not need to invoke the external dirname and basename programs, as the shell built-in features are powerful enough.

There is a lot more to parameter expansion. See man bash for more details.

Write a comment | Bookmark and Share

posted at: 23:31 | path: /dev | permalink | 1 comment | 0 trackbacks

2010年04月14日

PAUSE Web Service API and other Perl QA Hackathon results

In this blog post I'd like to talk about what I got done during the Perl QA Hackathon.

On Saturday I wrote a command for Dist::Zilla that uses Devel::Cover to generate code coverage metrics for a distribution: Dist::Zilla::App::Command::cover. Also I added plugin support to App::perlzonji (the more knowledgeable perldoc) so you can provide your own information sources. Ovid got interested in DB::Pluggable and wrote DB::Pluggable::Dumper; I'll try to refactor part of his work back into the main module.

On Sunday I talked with Andreas König about a Web Service API for PAUSE. I want to be able to edit maintainership permissions, add module registration metadata and delete files without having to use the PAUSE web site. Andreas agreed that this would be a good thing to have. In the first phase, to see how this all works out, we'll provide read-only access to the PAUSE data.

Writing a PSGI/Plack application was the obvious choice, and during dinner on Sunday evening miyagawa explained to me how Tatsumaki works. On Monday rafl, whose computer had unfortunately stopped working on Sunday, helped me with using non-blocking features of Tatsumaki.

Basically Tatsumaki is a framework for writing PSGI applications. Of course you can write raw PSGI applications, but Tatsumaki offers non-blocking support, path routing and takes care of the request and response objects for you. Writing a proof-of-concept PAUSE Web Service only took a few dozen lines. The API so far only consists of a few minor GET URLs and is not live yet. Have a look at PAUSE-Web on GitHub.

I also wrote a PAUSE client library to talk to the API, see PAUSE-Client on GitHub. This was really easy thanks to WebService::Simple.

In future posts I will talk more about the PAUSE Web Service API and how it is implemented.

Write a comment | Bookmark and Share

posted at: 10:26 | path: /dev | permalink | 0 comments | 0 trackbacks

2010年03月18日

Boolean normalization using the logical negation operator

The perlsyn manpage says the following about truth and falsehood:

The number 0, the strings '0' and '', the empty list "()", and "undef" are all false in a boolean context. All other values are true. Negation of a true value by "!" or "not" returns a special false value. When evaluated as a string it is treated as '', but as a number, it is treated as 0.

Sometimes you want to normalize a value so that it is 1 if it is a true value and 0 if it is a false value. This is sometimes called boolean normalization.

Usually you will see something like this:

my $b = $value ? 1 : 0;

There is an easier way. You could make use of the logical negation operator, !. When it is applied to a true value, it returns a dual value that is the number zero in numerical context and the empty string in string context. When the logical negation operator is applied to a false value, it returns a dual value that is the number 1 in numerical context and the string "1" in string context. Let's see this in detail:

$ perl -MDevel::Peek -le'Dump !42'
SV = PVNV(0x802000) at 0x12fef8
  REFCNT = 2147483647
  FLAGS = (IOK,NOK,POK,READONLY,pIOK,pNOK,pPOK)
  IV = 0
  NV = 0
  PV = 0x200160 ""\0
  CUR = 0
  LEN = 4

$ perl -MDevel::Peek -le'Dump !0'
SV = PVNV(0x802014) at 0x12ff08
  REFCNT = 2147483646
  FLAGS = (IOK,NOK,POK,READONLY,pIOK,pNOK,pPOK)
  IV = 1
  NV = 1
  PV = 0x200170 "1"\0
  CUR = 1
  LEN = 4

By using the logical negation operator twice you can therefore simplify the above statement to:

my $b = !!$value;

And if you wanted the opposite boolean normalization, it's even easier:

my $b = $value ? 0 : 1;

becomes:

my $b = !$value;

Write a comment | Bookmark and Share

posted at: 08:22 | path: /dev | permalink | 11 comments | 0 trackbacks

2009年12月13日

Plack::Middleware::Debug

I wrote and released a new middleware for Plack: Plack::Middleware::Debug. It was on miyagawa's wishlist for Plack and is based heavily on the Django Debug Toolbar. miyagawa also helped to get this middleware in proper shape to play nicely with the rest of Plack.

Here is an example of how to use it. Put this in your app.psgi:

use Plack::Builder;
use File::Basename;

my $app = sub { ... };

builder {
    enable 'Debug', panels =>
        [ qw(Environment Response Timer Memory DBITrace) ];
    $app;
};

The debug middleware offers a configurable set of panels that displays information about the current request and response. The information is generated only for responses with a status of 200 (OK) and a Content-Type that contains text/html and is embedded in the HTML that is sent back to the browser.

If you pass a list of panel base names to the enable() call, only those panels will be enabled. If you don't pass an argument, the default list of panels — Environment, Response, Timer and Memory — will be enabled.

As of version 0.01, the following panels are available:

The following two screenshots demonstrate the DBITrace and Environment panels.

Plack::Middleware::Debug::DBITrace

Plack::Middleware::Debug::Environment

You could use the power of the debug toolbar for your own framework or web application. It is quite easy to write custom panels. Just subclass Plack::Middleware::Debug::Base, then use your panel in the enable() call.

If you are interested, join us on #plack on irc.perl.org. The development repository is on GitHub.

Write a comment | Bookmark and Share

posted at: 12:04 | path: /dev | permalink | 0 comments | 1 trackback

2009年10月20日

HTML stack trace from the Perl debugger

Tatsuhiko Miyagawa released Devel::StackTrace::AsHTML and blogged about it.

I thought this would make a neat Perl debugger command, so I wrote DB::Pluggable::StackTraceAsHTML. It is a plugin to DB::Pluggable. It adds the Th command to the debugger, which displays a stack trace in HTML format, with lexical variables. It then opens the page in the default browser.

Here is an example of how to use it:

$ perl -d test.pl

Loading DB routines from perl5db.pl version 1.3
Editor support available.

Enter h or `h h' for help, or `man perldebug' for more help.

main::(test.pl:14): my $n = 12;
  DB<1> r                                                                                                   main::fib(test.pl:12):      return fib($i - 1) + fib($i - 2);
  DB<1> Th

The result would look something like:

To enable the plugin, just add it to your ~/.perldb, like so:

use DB::Pluggable;
use YAML;

$DB::PluginHandler = DB::Pluggable->new(config => Load <<EOYAML);
global:
  log:
    level: error

plugins:
  - module: BreakOnTestNumber
  - module: StackTraceAsHTML
EOYAML

$DB::PluginHandler->run;

By the way, to be minimally invasive to the existing Perl debugger, the command is defined using the debugger's aliasing mechanism. Normally you define an alias as a regular expression that will change the command the user enters to a known command, but here we circumvent that and call our command handler directly. The following method is from DB::Pluggable::Plugin:

sub make_command {
    my ($self, $cmd_name, $code) = @_;
    no strict 'refs';
    my $sub_name = "DB::cmd_$cmd_name";
    *{$sub_name} = $code;
    $DB::alias{$cmd_name} = "/./; &$sub_name;";
}

To define a new foo command in a plugin, you then use:

package DB::Pluggable::StackTraceAsHTML;
use strict;
use warnings;
use base qw(DB::Pluggable::Plugin);

sub register {
    my ($self, $context) = @_;
    $self->make_command(
        foo => sub {
            # ...
        }
    );
}

Write a comment | Bookmark and Share

posted at: 22:09 | path: /dev | permalink | 0 comments | 0 trackbacks

2009年10月05日

Repeatedly installing Task::* distributions

First there were Bundle::* distributions to install several dependencies at once without the actual bundle distribution doing anything, but this required magic in the toolchain. A newer concept is that of a Task::* distribution, as described in Task.

Task modules are just normal Perl modules; they don't require toolchain magic. When the dependencies listed in Makefile.PL have been installed, the actual Task module, for example Task::Catalyst, will be installed as well.

This is a problem because if the task's dependencies are being updated, the task module needs to be updated as well. For example, DBD::SQLite is a dependency of Task::Catalyst. If there is a new version of DBD::SQLite, just running install Task::Catalyst in the CPAN shell won't have any effect, because the CPAN shell will see that the latest version of Task::Catalyst is already installed. Of course you can force with force install Task::Catalyst, but even if that would work, it would be ugly because I don't want to install dependencies that fail their tests.

One solution is to prevent the task module itself to be installed. That way, every time you rerun install Task::Foo in the CPAN shell, it will see that the task module isn't installed and try to install it again, with all the desired dependency checking.

How do we prevent the module from being installed? Put these lines in your Makefile.PL — I always use Module::Install, but I think this should work for all MakeMaker-based files:

exit 0 if $ENV{AUTOMATED_TESTING};
sub MY::install { "install ::\n" }

First, we don't need to waste the time of CPAN testers by having them install all the dependencies for a task module that doesn't really do anything anyway. Second, and more importantly, we override the install :: target that is generated for the Makefile to effectively be a no-op.

I have demonstrated this technique in Task::BeLike::hanekomu, a somewhat pointless distribution that serves mainly as proof-of-concept, and as a way to install my favourite modules onto a new perl installation. If you don't mind the dependencies, then go ahead and repeatedly run install Task::BeLike::hanekomu in the CPAN shell; the task module itself should never be installed and it should recheck the dependencies every time.

I think this would be a good thing to put in existing task distributions such as Task::Kensho and Task::Catalyst.

Write a comment | Bookmark and Share

posted at: 12:27 | path: /dev | permalink | 6 comments | 1 trackback

2009年09月14日

Easy metadata for CPAN modules

Earlier I wrote about how to enhance a CPAN distribution by including metadata about that distribution's place in its infrastructure — where to find its homepage, bug tracker and repository.

There are now Module::Install plugins on CPAN that make this task easier. Tatsuhiko Miyagawa wrote Module::Install::Repository and I wrote Module::Install::Homepage and Module::Install::Bugtracker. So now your Makefile.PL can simply include these lines:

auto_set_homepage;
auto_set_bugtracker;
auto_set_repository;

To set the repository link, your distribution's version control is examined. The distribution's homepage is set to its page on http://search.cpan.org, and the distribution's bugtracker is set to its RT queue on http://rt.cpan.org.

Write a comment | Bookmark and Share

posted at: 10:13 | path: /dev | permalink | 0 comments | 0 trackbacks

2009年08月15日

In defense of polymorphism

Laurent Dami blogged Object-oriented accessors considered (sometimes) harmful and Yuval Kogman responded to it.

I agree that accessing hash elements directly can be tempting, but for me the killer argument in favor of accessor methods is polymorphism. Simply put, if you access hash elements directly, you're effectively making it completely impossible for someone else to subclass your class, or for your own class to subclass another class later. That's all.

And I agree strongly with one of Laurent's criticisms: With accessors, you never know where things really happen, making them difficult to debug. To use Laurent's example:

if ($c->request->body->length < $self->min_body) {

You have to step through a lot of methods to get to the right-hand side. It's the same with any expression.

What I would really like in the Perl debugger (but ENOTUITS) is the possibility to inspect the expression in a precedence-based tree-view like this:

1: <
2:    $c
3:        ->request
4:            ->body
5:                ->length
6:    $self
7:        ->min_body

and then being able to set a one-time breakpoint for line 7, that is, when entering the min_body() method.

Write a comment | Bookmark and Share

posted at: 13:29 | path: /dev | permalink | 1 comment | 0 trackbacks