Learning How to

   _____      _                 _   ____           _ 
  | ____|_  _| |_ ___ _ __   __| | |  _ \ ___ _ __| |
  |  _| \ \/ / __/ _ \ '_ \ / _` | | |_) / _ \ '__| |
  | |___ >  <| ||  __/ | | | (_| | |  __/  __/ |  | |
  |_____/_/\_\\__\___|_| |_|\__,_| |_|   \___|_|  |_|


Examples of code

Accompanying code to this tutorial can be gotten from examples.tar.gz.


1.0 Learning XS

Perhaps you've found yourself in a situation where you have some beautiful C code that does a task quite well. But you didn't want to program in C for the problem at hand, you wanted to code in Perl. Maybe you're writing a prototypical system in Perl and you're converting some of the slower components to C. You need a way to leverage a C library from Perl. As you'd expect, Perl comes to your rescue.

If you're like me, you like to get things done quickly and without fuss. I fussed over reading the XS tutorials and the Cookbooks. I took notes so that I could share my thoughts and hopefully save someone else the trauma for what can be a fairly straightforward process.

1.1 Introduction

The whole business is getting proper Makefiles, tests, modules, and bootstraps to show up and do their job. It's this unspoken learning curve (or speed bump) that really ticked me off the first time I tried to tackle XS without wanting to understand all the rest. I wanted to write xs, dammit, not learn these new modules and utilities.

As an aside, writing xs isn't as painful as one might think. The utility h2xs will definitely get you off to a very basic beginning, and the rest is an exercise in translating your header files in a way that XS (and Perl) can understand. I understand that SWIG will do this work for you, but if SWIG fails, you're back here. SWIG is no excuse to not want to learn all this great stuff (but it may be a convincing argument to get work done quickly).

In addition, perlxstut teaches you how to make a buildable module, not how to make a working module. So prior to testing to see if your xs is any good, you must perl Makefile.PL, make, make test, and (if you care about cleanliness) remove all the generated Makefiles and shared libraries (found in blib/).

1.2 Be familiar with h2xs

Implicit in Perl's motto: ``There's more than one way to do it!'' is that there are many passably workable ways, a few good ways, and even fewer great solutions. Making Perl modules without using h2xs is one of the passably working ways (and something I've been guilty of for many years now). h2xs has grown from being a nice way of building Perl extensions to being a great aid in helping you build testable and distributable software.

1.3 Be familiar with ExtUtils::MakeMaker

It turns out that knowing how ExtUtils::MakeMaker works is absolutely critical to generating consistently good Makefiles to put the c compiler through its paces and generate autoloadable shared libraries for Perl to slurp up. I guess, in retrospect, this is one of those less than blindingly bright moments of insight and inspiration, (eg. DUH!). Having once studied (and now blisfully forgotten, thanks to make and modern c compilers) the work involved in shared libraries.

So while you don't need to write Makefiles directly, you must write Makefile.PL instead. Consider it part of the cost of living.

1.4 Be familiar with Test::Harness (optional)

If your module has any depth and complexity (i.e. does something useful), then it greatly behooves you to spend some time with Test::Harness. ExtUtils::MakeMaker will build a test dependency rule that intelligently incorporates all the tests you might have from Test::Harness. Otherwise, ExtUtils::MakeMaker will use any file named test.pl as the command for the test dependency.

1.5 Be ready to learn Perl internals

Once you've tackled all these somewhat daunting tasks, you're ready to... learn some more! Next stop: Typemapping. The good news is that of all the Perl internals you could be faced with, most people just need to convert C datatypes to Perl datatypes. This need not be overwhelming. Thankfully, XS understands a lot of common datatypes, including most of C's single-word declarations and character pointers.

1.5.1 Simple datatypes (scalars) are free

Things like int, double, and char * are all known to XS. Things like struct *, int *, and double * must be explained to Perl in terms it can understand. Typically your choices will be blessed objects (O_OBJECT) or unblessed references (T_OBJECT). Additionally, pointers to structs must have creation (safemalloc) methods and destruction (safefree) methods to help Perl Do the Right Thing (tm). The good news is that once you've worked through it once, you can simply copy-and-paste your code and modify the class name to avoid collision of any other instances you may need.

I took the unhappy-but-it-works approach of making pointers to data into Perl objects, and defined an XS API for them (usually just get and set). While there may be a better way, this worked out great for me.

If you're extending Perl for functionality, my advice is to keep your datatypes simple. If you want bleeding-edge speed and *must* use Perl (instead of throwing your prototype away and re-building the whole thing in C), your needs are beyond the scope of this document. Sorry. I'll try to help you when I have a similar need to yours. The Cookbooks by Dean Roehrich will definitely point you in the right path, however.

1.5.2 C structs are gnarly

Converting C structs to Opaque Perl Scalars is nasty, unhappy business. But it's quite do-able. One might think that converting C structs to a hash value would be more appropriate (and I hope it is) but we'll tackle that later.

What is an Opaque Perl Scalar? Well, in C, to make a struct member assignment, you do:

        my_struct.my_member = x;

The Perl equivalent would be:

        $my_struct->set_my_member( x );

And that's exactly it. An Opaque Perl Scalar has *no way* of accessing its 'private' data members except through the API you provide. And it's up to you, the XS programmer, to make that API. (In short, the C struct is completely black-box encapsulated from Perl, hence 'Opaque'). Since in C, no two members of a struct can share the same name, a hash translation is natural (and, as of now beyond my scope).

For a good example, see ./examples/Name/Name.xs

Figure 1.0 A simple C struct and a summary of its XS API
        typedef struct {
                char *first;
                char *last;
        } Name;

        Name *new( CLASS )
        void DESTROY( self )
        void set_first( self, string )
        void set_last( self, string )
        char *first( self )
        char *last( self )

Note: Remember while writing XS, the return type must be on a line by itself. The formatting above was concatenated for illustration purposes.


        Name *
        new( CLASS )

However, if you've got some good C code, you shouldn't have to manipulate the C struct. The C code should. All you want is for Perl to be able to access that code (and to store the C struct in a Perl scalar).

1.5.3 Arrays are somewhere in-between

Arrays have a different code declaration (PPCODE) and must be explicitly EXTEND'ed. Each value to be returned must be PUSHs'ed onto the return stack. See perldoc perlguts /Argument Stack for more details.

2.0 Extending Perl

2.1 Learn how to build Perl Modules

Prior to needing to use XS, when I wanted a Perl Module, I usually started with a .pm file and began writing code. While this is functional, it is spartan. CPAN and the Perl programming community at large will benefit greatly once you've accepted the established way of writing perl modules. This is described in detail on CPAN.

The best way to start you off is to learn how to use the h2xs utility. When writing pure Perl modules, use h2xs -OXn MyModule. This will start you with a proper path setup. For extending Perl, you'll want to use h2xs -On MyModule and copy your source files into the generated path setup. Following the venerated perlxstut, I'm recommending that you first create your path setup and copy your source to the right place, and then use h2xs.

Figure 2.0: The Proper path setup

When you are done, you will have a module called My_module.pm that you can call from your Perl code. I typically use 'src' instead of 'My_source'.


So when I'm done, I'll be able to say, use Sicl; in my Perl programs and have all this wonderful C code run happily in Perl.

Note: If sicl.h had other #include preprocessor directives, they would also be recursively included (as they are in C). As it is, sicl.h is the only include file required to leverage the power of HP's Standard Instrument Control Library.

Yeah, that's right. I'm trying to get Perl to talk to Spectrum Analyzers :)>.

Warning: h2xs and ExtUtils::MakeMaker seem to have slightly different ideas on what a proper path setup should look like. Since ExtUtils::MakeMaker does a *lot* more work than h2xs, I tend to agree with the former. As a result, however, you'll always have to edit the

        #include <./My_module/My_source/My_source.h>

line in your .xs file to

        #include "./My_source/My_source.h"

However, your mileage may vary. A good reason of doing it the h2xs way is if you intend to keep all the revisions of your module in sibling paths (I tend to move them to a release path).

Figure 2.1: What h2xs thinks is good. YMMV
                        (where the h2xs-generated stuff goes)

2.2 Build your Makefile.PL for your C source code

Next you must build the Makefile.PL to control the build of your source code. ExtUtils::MakeMaker is flexible enough to support all kinds of make configurations, so consult

        perldoc ExtUtils::MakeMaker

for more details on how to solve your Makefile.PL situation.

If your C code has a perfectly good Makefile that comes with it, you should use that Makefile. See Figure 2.3 below.

Figure 2.2: An example Makefile.PL
        use ExtUtils::MakeMaker;
        $Verbose = 1; 

                NAME    => 'My_module::src', 
                SKIP    => [qw( all static static_lib dynamic dynamic_lib )],
                clean   => { FILES => 'src$(LIB_EXT)' }

        sub MY::top_targets {
        all :: static

        static :: src$(LIB_EXT)

        src$(LIB_EXT) : $(O_FILES)
                $(AR) cr src$(LIB_EXT) $(O_FILES)
                $(RANLIB) src$(LIB_EXT)

Figure 2.3: An example Makefile.PL when your C source has a good make
        use ExtUtils::MakeMaker;
        # See lib/ExtUtils/MakeMaker.pm for details of how to influence
        # the contents of the Makefile that is written.
            'NAME'      => 'MyModule',
            'VERSION_FROM' => 'MyModule.pm', # finds $VERSION
            'MYEXTLIB'  => 'MySource/MySource.o'

        sub MY::postamble {
        $(MYEXTLIB):    MySource/Makefile
                cd MySource && make MySource.o

A trend that I've noticed is to more-or-less use the default Makefile that MakeMaker will create for you, with the sole exception that you have to overload the MY::top_targets() method to get the resulting Makefile to do the right thing.

Figure 2.4: After writing your Makefile.PL for your source

2.3 Run h2xs to build an XS framework

Change your path to your module development path and run h2xs to build the XS framework.

        cd ~/dev/perl/modules
        h2xs -On Sicl ./Sicl/src/sicl.h

Where Sicl is the module name you'd like and the last argument is the C header file what incorporates the functions you wish to call from Perl. -O means you really want to ``overwrite'' ./Sicl (even though all you're doing is adding files to it, not really overwriting it).

Figure 2.5: After invoking h2xs ~/dev/perl/modules/ Sicl/ Changes MANIFEST Makefile.PL Sicl.pm Sicl.xs test.pl src/ Makefile.PL sicl.a sicl.h

2.4 Touching-up the XS work

And that, ladies and gentle-camels, was the easy part. You're probably going to want to take a look at Sicl.xs (or whatever your module happens to be named) and hope that you'll be impressed. I guarantee that you won't be. While h2xs does import #define pre- processor mandates, it doesn't do jack or squat about importing C prototypes. That job is entirely up to you. (SWIG users rejoice; SWIG *does* try its best to import all the C prototypes).

Fortunately, for the most part, all you'll need to do is copy the function prototype into something that xsubpp can read. With these declarations, xsubpp will be able to bridge the gap between C and Perl.

Figure 2.6: A few C prototypes and their corresponding xsubpp prototypes
        void hello( void );




        int sum( int a, int b );
        double square_root( double );


        sum( a, b )
                int a
                int b

        square_root( whatever_variable_you_want )
                double  whatever_variable_you_want

2.4.1 Updating the MANIFEST

The first (mindless) task is to update the MANIFEST to reflect the contents of src/. The module's Makefile.PL uses the MANIFEST to determine if all the required contents for building the module are there (lacking the contents doesn't necessarily cause Makefile.PL to throw a fit, however; it just hiccups and keeps going, deferring to 'make' to find any missing dependencies).

2.4.2 Updating the Makefile.PL

It also turns out that h2xs makes a boiler-plate Makefile.PL that isn't so bright. You'll need to reflect the contents of ./src in the module's Makefile.PL so that the C code is compiled and incorporated into the module. To do this, you'll need to add the 'MYEXTLIB' keyword to the attributes list of WriteMakefile().

You'll also need to create a postamble for WriteMakefile() to call. Remember when I suggested that if your beautiful C code comes with a Makefile that you should use it? This is where you use it. The name of the game is to get make to spew out a C library (.a file); if you're lucky, there's a blatant way in the Makefile. If you're not lucky, try your hand at 'ar'.

3.0 Building Your Module

Like a good CPAN module, your module is easy to build. In your module's root path, simply type:

        perl Makefile.PL
        make test
        make install

And the magic happens. There are quite a few diagnostics that go on between these four steps, and you'll want to pay attention to them. In fact, most of my development time is spent building the module, watching for the bug warnings, and fixing them.

3.1 Cleanliness leads to Godliness

You'll quickly come to appreciate ExtUtils::MakeMaker's 'make clean' rule. It removes all the temporary files (except Makefile.old) that were created in the installation process. It is *not* MANIFEST- dependent, so any files you may have (experiments, docs, or musings) will survive a 'make clean'. It's also a fast an easy way to start over again, in case you've touched a large number of files and want 'make' to explicitly re-build everything from scratch.

3.2 Voila! You are done!

After you've built your module, tested it, and installed it, you're ready to distribute your masterpiece! Do one last make clean and rm *.old and rm *~ (for you vi users). Make a tarball out of your Perl module and share it with the world. Or at least within your organization, if your oppressive PHB won't let you release under the Artistic License.

Thanks for reading, and happy hacking!

- m


Mike Wong, mike_w3@pacbell.net

guest writer for TomaCorp (We're not a corporation)

``I'm not an employee''