<!-- $NetBSD: pkginstall.xml,v 1.24 2023/05/15 21:14:43 gutteridge Exp $ -->

<chapter id="pkginstall"> <?dbhtml filename="pkginstall.html"?>
<title>The pkginstall framework</title>

<para>This chapter describes the framework known as
<literal>pkginstall</literal>, whose key features are:</para>

<itemizedlist>

  <listitem>
    <para>Generic installation and manipulation of directories and files
    outside the pkgsrc-handled tree, <varname>LOCALBASE</varname>.</para>
  </listitem>

  <listitem>
    <para>Automatic handling of configuration files during installation,
    provided that packages are correctly designed.</para>
  </listitem>

  <listitem>
    <para>Generation and installation of system startup scripts.</para>
  </listitem>

  <listitem>
    <para>Registration of system users and groups.</para>
  </listitem>

  <listitem>
    <para>Registration of system shells.</para>
  </listitem>

  <listitem>
    <para>Automatic updating of fonts databases.</para>
  </listitem>

</itemizedlist>

<para>The following sections inspect each of the above points in detail.</para>

<para>You may be thinking that many of the things described here could be
easily done with simple code in the package's post-installation target
(<literal>post-install</literal>).  <emphasis>This is incorrect</emphasis>,
as the code in them is only executed when building from source.  Machines
using binary packages could not benefit from it at all (as the code itself
could be unavailable).  Therefore, the only way to achieve any of the items
described above is by means of the installation scripts, which are
automatically generated by pkginstall.</para>

<!-- ================================================================== -->

<sect1 id="files-and-dirs-outside-prefix">
<title>Files and directories outside the installation prefix</title>

<para>As you already know, the <filename>PLIST</filename> file holds a list
of files and directories that belong to a package.  The names used in it
are relative to the installation prefix (<filename>${PREFIX}</filename>),
which means that it cannot register files outside this directory (absolute
path names are not allowed).  Despite this restriction, some packages need
to install files outside this location; e.g., under
<filename>${VARBASE}</filename> or
<filename>${PKG_SYSCONFDIR}</filename>.  The only way to achieve this
is to create such files during installation time by using
installation scripts.</para>

<para>The generic installation scripts are shell scripts that can
contain arbitrary code.  The list of scripts to execute is taken from
the <varname>INSTALL_FILE</varname> variable, which defaults to
<filename>INSTALL</filename>.  A similar variable exists for package
removal (<varname>DEINSTALL_FILE</varname>, whose default is
<filename>DEINSTALL</filename>).  These scripts can run arbitrary
commands, so they have the potential to create and manage files
anywhere in the file system.</para>

<para>Using these general installation files is not recommended, but
may be needed in some special cases.  One reason for avoiding them is
that the user has to trust the packager that there is no unwanted or
simply erroneous code included in the installation script. Also,
previously there were many similar scripts for the same functionality,
and fixing a common error involved finding and changing all of
them.</para>

<para>The pkginstall framework offers another, standardized way.  It
provides generic scripts to abstract the manipulation of such files
and directories based on variables set in the package's
<filename>Makefile</filename>.  The rest of this section describes
these variables.</para>

<!-- ================================================================== -->

<sect2 id="dirs-outside-prefix">
<title>Directory manipulation</title>

<para>The following variables can be set to request the creation of
directories anywhere in the file system:</para>

<itemizedlist>

  <listitem>
    <para><varname>MAKE_DIRS</varname> and <varname>OWN_DIRS</varname>
    contain a list of directories that should be created and should attempt
    to be destroyed by the installation scripts.  The difference between
    the two is that the latter prompts the administrator to remove any
    directories that may be left after deinstallation (because they were
    not empty), while the former does not.  Example:</para>

<programlisting>
MAKE_DIRS+=             ${VARBASE}/foo/private
</programlisting>
  </listitem>

  <listitem>
    <para><varname>MAKE_DIRS_PERMS</varname> and
    <varname>OWN_DIRS_PERMS</varname> contain a list of tuples describing
    which directories should be created and should attempt to be destroyed
    by the installation scripts.  Each tuple holds the following values,
    separated by spaces: the directory name, its owner, its group and its
    numerical mode.  For example:</para>

<programlisting>
MAKE_DIRS_PERMS+=       ${VARBASE}/foo/private \
                        ${REAL_ROOT_USER} ${REAL_ROOT_GROUP} 0700
</programlisting>

    <para>The difference between the two is exactly the same as their
    non-<varname>PERMS</varname> counterparts.</para>
  </listitem>

</itemizedlist>

</sect2>

<!-- ================================================================== -->

<sect2 id="files-outside-prefix">
<title>File manipulation</title>

<para>Creating non-empty files outside the installation prefix is tricky
because the <filename>PLIST</filename> forces all files to be inside it.
To overcome this problem, the only solution is to extract the file in a
known place (i.e., inside the installation prefix) and copy it to the
appropriate location during installation (done by the installation scripts
generated by pkginstall).  We will call the former the <emphasis>reference
file</emphasis> in the following paragraphs, which describe the variables
that can be used to automatically and consistently handle files outside the
installation prefix:</para>

<itemizedlist>

  <listitem>
    <para><varname>REQD_FILES</varname> is a list of pairs of reference
    and target files.  At installation time, the reference file is copied
    to the target if and only if the latter does not exist.  Upon
    deinstallation, the reference file is removed provided that it was
    not modified by the installation.</para>
  </listitem>
    
  <listitem>
    <para><varname>REQD_FILES_PERMS</varname> contains tuples describing
    reference files and targets, including owner, group, and numeric
    permissions that should be set. For example:</para>

<programlisting>
REQD_FILES_PERMS+=      ${PREFIX}/share/somefile ${VARBASE}/somefile \
                        ${REAL_ROOT_USER} ${REAL_ROOT_GROUP} 0700
</programlisting>

  </listitem>

  <listitem>
    <para><varname>CONF_FILES</varname> and
    <varname>CONF_FILES_PERMS</varname> have the same syntax as
    <varname>REQD_FILES</varname> and
    <varname>REQD_FILES_PERMS</varname> respectively.  The difference
    is that these variables are specifically intended for handling
    configuration files, for which additional conventions and
    constraints apply.  See <xref linkend="conf-files" /> for further
    discussion.  Note in particular that while handling of
    configuration files can be disabled by the user (see <xref
    linkend="conf-files-disable" />), this setting does not affect
    <varname>REQD_FILES</varname> and
    <varname>REQD_FILES_PERMS</varname>.</para>
  </listitem>

<!--
  I can find no evidence for this behavior. If anything, it's the
  other way around. - dholland 20230511

    <para>The difference between the two variable sets is that the
    <varname>REQD</varname> versions prompt the administrator to
    remove any files that may be left after deinstallation (because
    they were not empty), while the former does not.</para>
-->
</itemizedlist>

  <para>To install an empty file, one can use these macros and
  <filename>/dev/null</filename> as the reference file.</para>

</sect2>

</sect1>

<!-- ================================================================== -->

<sect1 id="conf-files">
<title>Configuration files</title>

<para>There are two principles that govern the handling of
configuration files in pkgsrc: first, the user's configuration must
not be lost or overwritten by upgrades or reinstallations; and second,
the default configuration should always be available for reference.
To that end, pkgsrc has a framework specifically for handling
configuration files.  In general, configuration files are installed
into <filename>${PREFIX}/share/examples</filename>, and copied from
there to the pertinent <filename>etc</filename> directory as a
separate step and only as needed.</para>

<para>To make this work, there are three things that need to happen,
any of which may require patching.  First, the package must be told to
read its configuration from the correct place at runtime; this place
is <filename>${PKG_SYSCONFDIR}</filename> (or possibly a subdirectory
of it) which is often but not always
<filename>${PREFIX}/etc</filename>.  Second, the package must be
taught to install its default configuration files in
<filename>${PREFIX}/share/examples/${PKGBASE}</filename>; this is the
point at which patching often becomes required because it is not the
same place as the runtime location.  Third, the configuration files
need to be declared in the package makefile; this is the easy part.
</para>

<para>With those elements in place, the right things will happen: the
package install phase and resulting binary package will not touch
<filename>${PKG_SYSCONFDIR}</filename>, and the default configuration
will be copied into place by the pkgsrc framework only if not already
present.  Similarly, upon deinstall the configuration will not be
removed if it has been modified.  (This means that in the case of
reinstallation and upgrades, unmodified configuration files are
updated but modifications are never discarded.)</para>

<para>The following sections describe how to make these things happen
and document other relevant knobs available in the pkgsrc
infrastructure.</para>

<!-- ================================================================== -->

<sect2 id="conf-files-sysconfdir">
<title>How <varname>PKG_SYSCONFDIR</varname> is set</title>

<para>As said before, the <varname>PKG_SYSCONFDIR</varname> variable
specifies where configuration files shall be installed.  Its contents are
set based upon the following variables:</para>

<itemizedlist>

  <listitem>
    <para><varname>PKG_SYSCONFBASE</varname>: The configuration's root
    directory.  Defaults to <filename>${PREFIX}/etc</filename> although it may
    be overridden by the user to point to their preferred location (e.g.,
    <filename>/etc</filename>, <filename>/etc/pkg</filename>, etc.).
    Packages must not use it directly.</para>
  </listitem>

  <listitem>
    <para><varname>PKG_SYSCONFSUBDIR</varname>: A subdirectory of
    <varname>PKG_SYSCONFBASE</varname> under which the configuration files
    for the package being built shall be installed.  The definition of this
    variable only makes sense in the package's
    <filename>Makefile</filename> (i.e., it is not user-customizable).</para>

    <para>As an example, consider the Apache package,
    <filename role="pkg">www/apache24</filename>, which places its
    configuration files under the
    <filename>httpd/</filename> subdirectory of
    <varname>PKG_SYSCONFBASE</varname>.  This should be set in the package
    Makefile.</para>
  </listitem>

  <listitem>
    <para><varname>PKG_SYSCONFVAR</varname>: Specifies the name of the
    variable that holds this package's configuration directory (if
    different from <varname>PKG_SYSCONFBASE</varname>).  It defaults to
    <varname>PKGBASE</varname>'s value, and is always prefixed with
    <literal>PKG_SYSCONFDIR</literal>.</para>
  </listitem>

  <listitem>
    <para><varname>PKG_SYSCONFDIR.${PKG_SYSCONFVAR}</varname>: Holds the
    directory where the configuration files for the package identified by
    <varname>PKG_SYSCONFVAR</varname>'s shall be placed.</para>
  </listitem>

</itemizedlist>

<para>Based on the above variables, pkginstall determines the value of
<varname>PKG_SYSCONFDIR</varname>, which is the <emphasis>only</emphasis>
variable that may be used within a package to refer to its configuration
directory.  The algorithm used to set its value is basically the
following:</para>

<orderedlist>

  <listitem>
    <para>If <varname>PKG_SYSCONFDIR.${PKG_SYSCONFVAR}</varname> is set,
    its value is used.</para>
  </listitem>

  <listitem>
    <para>If the previous variable is not defined but
    <varname>PKG_SYSCONFSUBDIR</varname> is set in the package's
    <filename>Makefile</filename>, the resulting value is
    <filename>${PKG_SYSCONFBASE}/${PKG_SYSCONFSUBDIR}</filename>.</para>
  </listitem>

  <listitem>
    <para>Otherwise, it is set to
    <filename>${PKG_SYSCONFBASE}</filename>.</para>
  </listitem>

</orderedlist>

<para>It is worth mentioning that <filename>${PKG_SYSCONFDIR}</filename> is
automatically added to <filename>OWN_DIRS</filename>.  This causes it
to be automatically created if needed.  See <xref
linkend="dirs-outside-prefix" /> for further details.  This does not apply to
subdirectories of <filename>${PKG_SYSCONFDIR}</filename>; they must be manually
created with <varname>OWN_DIRS</varname> or
<varname>MAKE_DIRS</varname>.</para>

</sect2>

<!-- ================================================================== -->

<sect2 id="conf-files-configure">
<title>Telling the software where configuration files are</title>

<para>Given that pkgsrc (and users!) expect configuration files to be in a
known place, you need to teach each package where to install its files.  In
some cases you will have to patch the package Makefiles to achieve it.  If
you are lucky, though, it may be as easy as passing an extra flag to the
configuration script.  This is the case for packages using GNU
autoconf:</para>

<programlisting>
CONFIGURE_ARGS+= --sysconfdir=${PKG_SYSCONFDIR}
</programlisting>

<para>Note that this specifies where the package has to <emphasis>look
for</emphasis> its configuration files, not where they will be installed.
Fortunately, there is a different way to specify the latter, as seen
in the next section, although the combination is rather confusing at
first glance.</para>

</sect2>

<!-- ================================================================== -->

<sect2 id="conf-files-patching">
<title>Patching installation</title>

<para>As discussed above, <emphasis role="strong">packages themselves must not
touch the contents of <filename>${PKG_SYSCONFDIR}</filename>
directly</emphasis>.  Bad news is that many software installation scripts
will, out of the box, mess with the contents of that directory.  So what is
the correct procedure to fix this issue?</para>

<para>You must teach the package (usually by manually patching it) to
install any configuration files under the examples hierarchy,
<filename>share/examples/${PKGBASE}/</filename>.  This way, the
<filename>PLIST</filename> registers them and the administrator always
has the original copies available.</para>

<para>It turns out that for packages using GNU autoconf it is possible
to create the desired effect by setting <varname>sysconfdir</varname>
on the make command line at install time.  Consider this example taken
from <filename role="pkg">mail/mutt</filename>:</para>

<programlisting>
EGDIR=                  ${PREFIX}/share/examples/mutt
INSTALL_MAKE_FLAGS=     ${MAKE_FLAGS} sysconfdir=${EGDIR}
</programlisting>

<para>Note that the <varname>EGDIR</varname> variable, though commonly
used for this purpose, is local to that package and has no meaning
outside it.</para>

</sect2>

<!-- ================================================================== -->

<sect2 id="conf-files-declare">
<title>Declaring configuration files</title>

<para>Once the required configuration files are in place (i.e., under the
examples hierarchy), the pkginstall framework can use them as reference copies
during the package installation to update what is in
<filename>${PKG_SYSCONFDIR}</filename>.  To achieve this, the variables
<varname>CONF_FILES</varname> and <varname>CONF_FILES_PERMS</varname> are
used.  Check out <xref linkend="files-outside-prefix" /> for further
information about their syntax and their purpose.  Here is an example,
taken from the <filename role="pkg">mail/mutt</filename> package:</para>

<programlisting>
EGDIR=                  ${PREFIX}/share/examples/mutt
CONF_FILES=             ${EGDIR}/Muttrc ${PKG_SYSCONFDIR}/Muttrc
</programlisting>

<para>Note that (as in the previous section's example) the
<varname>EGDIR</varname> variable is specific to the package and has
no meaning outside it.</para>

<para>For reference, the complete example from Mutt is as follows:</para>

<programlisting>
CONFIGURE_ARGS+=        --sysconfdir=${PKG_SYSCONFDIR}
EGDIR=                  ${PREFIX}/share/examples/mutt
CONF_FILES=             ${EGDIR}/Muttrc ${PKG_SYSCONFDIR}/Muttrc
INSTALLATION_DIRS+=     ${EGDIR}
INSTALL_MAKE_FLAGS=     ${MAKE_FLAGS} sysconfdir=${EGDIR}
</programlisting>

</sect2>

<!-- ================================================================== -->

<sect2 id="conf-files-disable">
<title>Disabling handling of configuration files</title>

<!-- XXX This doesn't really belong here -->

<para>The automatic copying of config files can be toggled by setting the
environment variable <varname>PKG_CONFIG</varname> prior to package
installation.</para>

</sect2>

</sect1>

<!-- ================================================================== -->

<sect1 id="rcd-scripts">
<title>System startup scripts</title>

<para>System startup scripts are special files because they must be
installed in a place known by the underlying OS, usually outside the
installation prefix.  Therefore, the same rules described in <xref
linkend="files-and-dirs-outside-prefix" /> apply, and the same solutions
can be used.  However, pkginstall provides a special mechanism to handle
these files.</para>

<para>In order to provide system startup scripts, the package has
to:</para>

<orderedlist>

  <listitem>
    <para>Store the script inside <filename>${FILESDIR}</filename>, with
    the <literal>.sh</literal> suffix appended.  Considering the
    <filename role="pkg">print/cups-base</filename> package as an example, it
    has a <filename>cupsd.sh</filename> in its files directory.</para>
  </listitem>

  <listitem>
    <para>Tell pkginstall to handle it, appending the name of the script,
    without its extension, to the <varname>RCD_SCRIPTS</varname> variable.
    Continuing the previous example:</para>

<programlisting>
RCD_SCRIPTS+=   cupsd
</programlisting>
  </listitem>

</orderedlist>

<para>Once this is done, pkginstall will do the following steps for each
script in an automated fashion:</para>

<!-- XXX We should probably have a chapter describing how rc.d scripts are
     written. -->

<orderedlist>

  <listitem>
    <para>Process the file found in the files directory applying all the
    substitutions described in the <filename>FILES_SUBST</filename>
    variable.</para>
  </listitem>

  <listitem>
    <para>Copy the script from the files directory to the examples
    hierarchy, <filename>${PREFIX}/share/examples/rc.d/</filename>.  Note
    that this reference file must be explicitly registered in the
    <filename>PLIST</filename>.</para>
  </listitem>

  <listitem>
    <para>Add code to the installation scripts to copy the startup script
    from the examples hierarchy into the system-wide startup scripts
    directory.</para>
  </listitem>

</orderedlist>

<!-- ================================================================== -->

<sect2 id="rcd-scripts-disable">
<title>Disabling handling of system startup scripts</title>

<!-- XXX This doesn't really belong here -->

<para>The automatic copying of config files can be toggled by setting the
environment variable <varname>PKG_RCD_SCRIPTS</varname> prior to package
installation.  Note that the scripts will be always copied inside the
examples hierarchy, <filename>${PREFIX}/share/examples/rc.d/</filename>, no
matter what the value of this variable is.</para>

</sect2>

</sect1>

<!-- ================================================================== -->

<sect1 id="users-and-groups">
<title>System users and groups</title>

<para>If a package needs to create special users and/or groups during
installation, it can do so by using the pkginstall framework.</para>

<para>Users can be created by adding entries to the
<varname>PKG_USERS</varname> variable.  Each entry has the following
syntax:</para>

<programlisting>
user:group
</programlisting>

<para>Further specification of user details may be done by setting
per-user variables.
<varname>PKG_UID.<replaceable>user</replaceable></varname> is the
numeric UID for the user.
<varname>PKG_GECOS.<replaceable>user</replaceable></varname> is the
user's description or comment.
<varname>PKG_HOME.<replaceable>user</replaceable></varname> is the
user's home directory, and defaults to
<filename>/nonexistent</filename> if not specified.
<varname>PKG_SHELL.<replaceable>user</replaceable></varname> is the
user's shell, and defaults to <filename>/sbin/nologin</filename> if
not specified.</para>

<para>Similarly, groups can be created by adding entries to the
<varname>PKG_GROUPS</varname> variable, whose syntax is:</para>

<programlisting>
group
</programlisting>

<para>The numeric GID of the group may be set by defining
<varname>PKG_GID.<replaceable>group</replaceable></varname>.</para>

<para>If a package needs to create the users and groups at an earlier
stage, then it can set <varname>USERGROUP_PHASE</varname> to either
<literal>configure</literal>,<literal>build</literal>, or
<literal>pre-install</literal> to indicate the phase before which the
users and groups are created. In this case, the numeric UIDs and GIDs
of the created users and groups are automatically hardcoded into the
final installation scripts.</para>

</sect1>

<!-- ================================================================== -->

<sect1 id="shells">
<title>System shells</title>

<para>Packages that install system shells should register them in the shell
database, <filename>/etc/shells</filename>, to make things easier to the
administrator.  This must be done from the installation scripts to keep
binary packages working on any system.  pkginstall provides an easy way to
accomplish this task.</para>

<para>When a package provides a shell interpreter, it has to set the
<varname>PKG_SHELL</varname> variable to its absolute file name.  This will
add some hooks to the installation scripts to handle it.  Consider the
following example, taken from <filename role="pkg">shells/zsh</filename>:</para>

<programlisting>
PKG_SHELL=      ${PREFIX}/bin/zsh
</programlisting>

<!-- ================================================================== -->

<sect2 id="shells-disable">
<title>Disabling shell registration</title>

<!-- XXX This doesn't really belong here -->

<para>The automatic registration of shell interpreters can be disabled by
the administrator by setting the <filename>PKG_REGISTER_SHELLS</filename>
environment variable to <literal>NO</literal>.</para>

</sect2>

</sect1>

<!-- ================================================================== -->

<sect1 id="fonts">
<title>Fonts</title>

<para>Packages that install X11 fonts should update the database files
that index the fonts within each fonts directory.  This can easily be
accomplished within the pkginstall framework.</para>

<para>When a package installs X11 fonts, it must list the directories in
which fonts are installed in the
<varname>FONTS_DIRS.<replaceable>type</replaceable></varname> variables,
where <replaceable>type</replaceable> can be one of <quote>ttf</quote>,
<quote>type1</quote> or <quote>x11</quote>.  This will add hooks to the
installation scripts to run the appropriate commands to update the fonts
database files within each of those directories.  For convenience, if the
directory path is relative, it is taken to be relative to the package's
installation prefix.  Consider the following example, taken from <filename
role="pkg">fonts/dbz-ttf</filename>:</para>

<programlisting>
FONTS_DIRS.ttf= ${PREFIX}/share/fonts/X11/TTF
</programlisting>

<!-- ================================================================== -->

<sect2 id="fonts-disable">
<title>Disabling automatic update of the fonts databases</title>

<!-- XXX This doesn't really belong here -->

<para>The automatic update of fonts databases can be disabled by
the administrator by setting the <filename>PKG_UPDATE_FONTS_DB</filename>
environment variable to <literal>NO</literal>.</para>

</sect2>

</sect1>

</chapter>
