GSoC 2017 Reports: Add SUBPACKAGES support to pkgsrc, part 1

In this blog post series I will discuss about SUBPACKAGES work done during Google Summer of Code 2017.

In this first part I'll briefly introduce what are SUBPACKAGES, why and when can be useful and finally we'll give a quick look to a trivial pkgsrc package that uses them. At the end we'll also dive a bit on parts of the pkgsrc infrastructure that needed to be adjusted for implementing that.

Introduction

SUBPACKAGES (on some package systems they are known as multi-packages, but this term for pkgsrc is already used by packages that can be built against several versions (e.g. Python, PHP, Ruby packages)) consist in generating multiple binary packages from a single pkgsrc package. For example, from a pkgsrc package - local/frobnitzem - we will see how to generate three separate binary packages: frobnitzem-foo, frobnitzem-bar and frobnitzem-baz.

This can be useful to separate several components of binary packages (and avoid to run the extract and configure phase two times!), for debugpkgs (so that all *.debug files containing debug symbols are contained in a separate -debugpkg package that can be installed only when it is needed), etc..

A simple SUBPACKAGES package: frobnitzem!

To understand how SUBPACKAGES works and can be useful let's start to see an example of it in practice: frobnitzem.

frobnitzem is a trivial package that just install three scripts in ${PREFIX}/bin, let's see it:

% cd pkgsrc/local/frobnitzem
% tree
.
|-- DESCR
|-- Makefile
|-- PLIST
`-- files
    `-- frobnitzem
        |-- frobnitzem-bar
        |-- frobnitzem-baz
        `-- frobnitzem-foo

2 directories, 6 files
% find . -type f | xargs tail -n +1
==> ./Makefile <==
# $NetBSD$

DISTNAME=	frobnitzem-0
CATEGORIES=	local
MASTER_SITES=	# empty
DISTFILES=	# empty

MAINTAINER=	leot%NetBSD.org@localhost
HOMEPAGE=	http://netbsd.org/~leot/gsoc2017-diary/
COMMENT=	Simple subpackages example
LICENSE=	public-domain

FILESDIR=	${.CURDIR}/../../local/frobnitzem/files

WRKSRC=		${WRKDIR}/frobnitzem

NO_BUILD=	yes

do-extract:
	${CP} -r ${FILESDIR}/frobnitzem ${WRKDIR}

do-install:
	${INSTALL_SCRIPT_DIR} ${DESTDIR}${PREFIX}/bin
	${INSTALL_SCRIPT} ${WRKSRC}/frobnitzem-foo ${DESTDIR}${PREFIX}/bin
	${INSTALL_SCRIPT} ${WRKSRC}/frobnitzem-bar ${DESTDIR}${PREFIX}/bin
	${INSTALL_SCRIPT} ${WRKSRC}/frobnitzem-baz ${DESTDIR}${PREFIX}/bin

.include "../../mk/bsd.pkg.mk"

==> ./files/frobnitzem/frobnitzem-bar <==
#!/bin/sh

echo "bar"

==> ./files/frobnitzem/frobnitzem-baz <==
#!/bin/sh

echo "baz"

==> ./files/frobnitzem/frobnitzem-foo <==
#!/bin/sh

echo "foo"

==> ./PLIST <==
@comment $NetBSD$
bin/frobnitzem-bar
bin/frobnitzem-baz
bin/frobnitzem-foo

==> ./DESCR <==
frobnitzem, collection of foo, bar, baz scripts.

(Or, a bit more seriously this is just a very simple package to
test subpackages support!)

Nothing fancy, just three simple scripts, frobnitzem-{bar,baz,foo} that will respectively print to the standard output bar, baz and foo. Let's build and install the frobnitzem package:

% make install
===> Installing dependencies for frobnitzem-0                       
===> Overriding tools for frobnitzem-0                              
===> Extracting for frobnitzem-0  
[...]
===> Installing for frobnitzem-0  
[...]
===> Installing binary package of frobnitzem-0                      
[...]

And now let's try scripts installed as part of the frobnitzem package:

% foreach w (bar baz foo)
... frobnitzem-$w
... end
bar
baz
foo

Okay, as we expected. Despite frobnitzem-{foo,bar,baz} don't do anything particularly useful we can split the frobnitzem-0 package in three separate subpackages: frobnitzem-foo-0, frobnitzem-bar-0 and frobnitzem-baz-0 (they provides different functionalities and can also coexist if they're in separated binary packages).

To do that we need to slighty adjust Makefile, split the PLIST in PLIST.{foo,bar,baz} (one for each separate subpackage), split the DESCR in DESCR.{foo,bar,baz}. So, at the end in local/frobnitzem we'll have:

% tree
.
|-- DESCR.bar
|-- DESCR.baz
|-- DESCR.foo
|-- Makefile
|-- PLIST.bar
|-- PLIST.baz
|-- PLIST.foo
`-- files
    `-- frobnitzem
        |-- frobnitzem-bar
        |-- frobnitzem-baz
        `-- frobnitzem-foo

2 directories, 10 files

Splitting DESCR and PLIST

DESCR and PLIST splits are straightforward. We just provide a separate DESCR.<spkg> for each subpackage, e.g. for the foo subpackage:

% cat DESCR.foo
frobnitzem, collection of foo, bar, baz scripts.

(Or, a bit more seriously this is just a very simple package to
test subpackages support!)

This package provide the foo functionalities.

Similarly, regarding PLISTs, we just provide a separate PLIST.<spkg> for each subpackage, e.g. for the foo subpackage:

% cat PLIST.foo
@comment $NetBSD$
bin/frobnitzem-foo

Makefile changes

In Makefile we'll need to list all SUBPACKAGES hence we'll add the following line as first paragraph:

SUBPACKAGES=	foo bar baz

We'll then need to define a PKGNAME.<spkg> for each subpackages:

PKGNAME.foo=    frobnitzem-foo-0
PKGNAME.bar=    frobnitzem-bar-0
PKGNAME.baz=    frobnitzem-baz-0

...and similarly COMMENT variable should be defined for each subpackage via COMMENT.<spkg>:

COMMENT.foo=    Simple subpackages example (foo)
COMMENT.bar=    Simple subpackages example (bar)
COMMENT.baz=    Simple subpackages example (baz)

To recap here how we have adjusted Makefile, all the other lines rest unchanged:

% sed '/LICENSE/q' < Makefile
# $NetBSD$

SUBPACKAGES=    foo bar baz

DISTNAME=       frobnitzem-0
PKGNAME.foo=    frobnitzem-foo-0
PKGNAME.bar=    frobnitzem-bar-0
PKGNAME.baz=    frobnitzem-baz-0
MASTER_SITES=   # empty
DISTFILES=      # empty
CATEGORIES=     local

MAINTAINER=     leot%NetBSD.org@localhost
HOMEPAGE=       http://netbsd.org/~leot/gsoc2017-diary/
COMMENT.foo=    Simple subpackages example (foo)
COMMENT.bar=    Simple subpackages example (bar)
COMMENT.baz=    Simple subpackages example (baz)
LICENSE=        public-domain

Finally we can install it^Wthem! The usual make install will generate three binary packages (frobnitzem-foo-0.tgz, frobnitzem-bar-0.tgz, frobnitzem-baz-0.tgz) and install all of them:

% make install
===> Installing dependencies for frobnitzem-0
[...]
===> Overriding tools for frobnitzem-0
===> Extracting for frobnitzem-0
[...]
===> Installing for frobnitzem-0
[...]
=> Creating binary package /home/leot/repos/netbsd-github/pkgsrc/packages/All/frobnitzem-foo-0.tgz
=> Creating binary package /home/leot/repos/netbsd-github/pkgsrc/packages/All/frobnitzem-bar-0.tgz
=> Creating binary package /home/leot/repos/netbsd-github/pkgsrc/packages/All/frobnitzem-baz-0.tgz
[...]
===> Installing binary package of frobnitzem-foo-0
===> Installing binary package of frobnitzem-bar-0
===> Installing binary package of frobnitzem-baz-0
[...]

Now we can try them and use pkg_info(1) to get some information about them:

% frobnitzem-foo
foo
% pkg_info -Fe /usr/pkg/bin/frobnitzem-foo
frobnitzem-foo-0
% pkg_info frobnitzem-baz
Information for frobnitzem-baz-0:

Comment:
Simple subpackages example (baz)

Description:
frobnitzem, collection of foo, bar, baz scripts.

(Or, a bit more seriously this is just a very simple package to
test subpackages support!)

This package provide the baz functionalities.

Homepage:
http://netbsd.org/~leot/gsoc2017-diary/
% pkg_info -L frobnitzem-bar
Information for frobnitzem-bar-0:

Files:
/usr/pkg/bin/frobnitzem-bar

So we can see that make install actually installed three different binary packages.

To deinstall all SUBPACKAGES we can run make deinstall in the local/frobnitzem directory (that will remove all subpackages) or we can just manually invoke pkg_delete(1).

An high-level look at how SUBPACKAGES support is implemented

Most of the changes needed are in mk/pkgformat/pkg/ hierarchy (previously known as mk/flavour and then renamed and generalized to other package formats during Anton Panev's Google Summer of Code 2011).

The code in mk/pkgformat/${PKG_FORMAT}/ handle the interaction of pkgsrc with the particular ${PKG_FORMAT}, e.g. for pkg populate meta-data files used by pkg_create(1), install/delete packages via pkg_add(1), and pkg_delete(1), etc.

For more information mk/pkgformat/README is a good introduction to pkgformat hierarchy.

Most of the changes done respect the following template:

.if !empty(SUBPACKAGES)
.  for _spkg_ in ${SUBPACKAGES}
[... code that handles SUBPACKAGES case ...]
.  endfor
.else   # !SUBPACKAGES
[... existing (and usually completely unmodified) code ...]
.endif  # SUBPACKAGES

In particular, in mk/pkgformat/pkg/ targets were adjusted to create/install/deinstall/etc. all subpackages.

Apart mk/pkgformat other changes were needed in mk/install/install.mk in order to adjust the install phase for SUBPACKAGES.

Regarding PLIST.<spkg> handling mk/plist/plist.mk needed some adjustments to honor each PLIST per-subpackage.

mk/bsd.pkg.mk needed to be adjusted too in order to honor several per-subpackage variables (the *.<spkg> ones) and per-subpackage DESCR.<spkg>.

Conclusion

In this first part of this blog post series we have seen what are SUBPACKAGES, when and why they can be useful.

We have then seen a practical example of them taking a very trivial package and learned how to "subpackage-ify" it.

Then we have described - from an high-level perspective - the changes needed to the pkgsrc infrastructure for the SUBPACKAGES features that we have used. If you are more interested in them please give a look to the pkgsrc debugpkg branch that contains all work done described in this blog post.

In the next part we will see how to handle *DEPENDS and buildlink3 inclusion for subpackages.

I would like to thanks Google for organizing Google Summer of Code, the entire The NetBSD Foundation and in particular my mentors Taylor R. Campbell, William J. Coldwell and Thomas Klausner for providing precious guidance during these three months. A special thank you also to Jörg Sonnenberger who provided very useful suggestions. Thank you!

[0 comments]