This chapter gives a detailed description on how a package is built. Building a package is separated into different phases (for example fetch, build, install), all of which are described in the following sections. Each phase is split into so-called stages, which take the name of the containing phase, prefixed by one of pre-, do- or post-. (Examples are pre-configure, post-build.) Most of the actual work is done in the do-* stages. Never override the regular targets (like fetch), if you have to, override the do-* ones instead. The basic steps for building a program are always the same. First the program's source (distfile) must be brought to the local system and then extracted. After any pkgsrc-specific patches to compile properly are applied, the software can be configured, then built (usually by compiling), and finally the generated binaries, etc. can be put into place on the system. To get more details about what is happening at each step, you can set the PKG_VERBOSE variable, or the PATCH_DEBUG variable if you are just interested in more details about the patch step. Before outlining the process performed by the &os; package system in the next section, here's a brief discussion on where programs are installed, and which variables influence this. The automatic variable PREFIX indicates where all files of the final program shall be installed. It is usually set to LOCALBASE (/usr/pkg), or CROSSBASE for pkgs in the cross category. The value of PREFIX needs to be put into the various places in the program's source where paths to these files are encoded. See and for more details. When choosing which of these variables to use, follow the following rules: PREFIX always points to the location where the current pkg will be installed. When referring to a pkg's own installation path, use ${PREFIX}. LOCALBASE is where all pkgs are installed. If you need to construct a -I or -L argument to the compiler to find includes and libraries installed by another pkg, use ${LOCALBASE}. The name LOCALBASE stems from FreeBSD, which installed all packages in /usr/local. As pkgsrc leaves /usr/local for the system administrator, this variable is a misnomer. X11BASE is where the actual X11 distribution (from xsrc, etc.) is installed. When looking for standard X11 includes (not those installed by a package), use ${X11BASE}. X11-based packages using imake must set USE_IMAKE to be installed correctly under LOCALBASE. Within ${PREFIX}, packages should install files according to &man.hier.7;, with the exception that manual pages go into ${PREFIX}/man, not ${PREFIX}/share/man. When building a package, various directories are used to store source files, temporary files, pkgsrc-internal files, and so on. These directories are explained here. Some of the directory variables contain relative pathnames. There are two common base directories for these relative directories: PKGSRCDIR/PKGPATH is used for directories that are pkgsrc-specific. WRKSRC is used for directories inside the package itself. PKGSRCDIR This is an absolute pathname that points to the pkgsrc root directory. Generally, you don't need it. PKGDIR This is an absolute pathname that points to the current package. PKGPATH This is a pathname relative to PKGSRCDIR that points to the current package. It is defined after including bsd.prefs.mk and can be used in makefile fragments that are used by several packages to distinguish between these packages. Other variables that would serve the same purpose are PKGBASE and PKGNAME, but these are only defined after including bsd.pkg.mk, which is too late. In &mk.conf;, the pkgsrc user can use PKGPATH to tweak variables like MAKE_JOBS and CFLAGS. WRKDIR This is an absolute pathname pointing to the directory where all work takes place. The distfiles are extracted to this directory. It also contains temporary directories and log files used by the various pkgsrc frameworks, like buildlink or the wrappers. WRKSRC This is an absolute pathname pointing to the directory where the distfiles are extracted. It is usually a direct subdirectory of WRKDIR, and often it's the only directory entry that isn't hidden. This variable may be changed by a package Makefile. The CREATE_WRKDIR_SYMLINK definition takes either the value yes or no and defaults to no. It indicates whether a symbolic link to the WRKDIR is to be created in the pkgsrc entry's directory. If users would like to have their pkgsrc trees behave in a read-only manner, then the value of CREATE_WRKDIR_SYMLINK should be set to no. You can run a particular phase by typing make phase, where phase is the name of the phase. This will automatically run all phases that are required for this phase. The default phase is build, that is, when you run make without parameters in a package directory, the package will be built, but not installed. The first step in building a package is to fetch the distribution files (distfiles) from the sites that are providing them. This is the task of the fetch phase. In simple cases, MASTER_SITES defines all URLs from where the distfile, whose name is derived from the DISTNAME variable, is fetched. The more complicated cases are described below. The variable DISTFILES specifies the list of distfiles that have to be fetched. Its value defaults to ${DEFAULT_DISTFILES} and its value is ${DISTNAME}${EXTRACT_SUFX}, so that most packages don't need to define it at all. EXTRACT_SUFX is .tar.gz by default, but can be changed freely. Note that if your package requires additional distfiles to the default one, you cannot just append the additional filenames using the += operator, but you have write for example: DISTFILES= ${DEFAULT_DISTFILES} additional-files.tar.gz Each distfile is fetched from a list of sites, usually MASTER_SITES. If the package has multiple DISTFILES or multiple PATCHFILES from different sites, you can set SITES.distfile to the list of URLs where the file distfile (including the suffix) can be found. DISTFILES= ${DISTNAME}${EXTRACT_SUFX} DISTFILES+= foo-file.tar.gz SITES.foo-file.tar.gz= \ https://www.somewhere.com/somehow/ \ https://www.somewhereelse.com/mirror/somehow/ When actually fetching the distfiles, each item from MASTER_SITES or SITES.* gets the name of each distfile appended to it, without an intermediate slash. Therefore, all site values have to end with a slash or other separator character. This allows for example to set MASTER_SITES to a URL of a CGI script that gets the name of the distfile as a parameter. In this case, the definition would look like: MASTER_SITES= https://www.example.com/download.cgi?file= The exception to this rule are URLs starting with a dash. In that case the URL is taken as is, fetched and the result stored under the name of the distfile. You can use this style for the case when the download URL style does not match the above common case. For example, if permanent download URL is a redirector to the real download URL, or the download file name is offered by an HTTP Content-Disposition header. In the following example, foo-1.0.0.tar.gz will be created instead of the default v1.0.0.tar.gz. DISTNAME= foo-1.0.0 MASTER_SITES= -https://www.example.com/archive/v1.0.0.tar.gz There are some predefined values for MASTER_SITES, which can be used in packages. The names of the variables should speak for themselves. @master_sites@ Some explanations for the less self-explaining ones: MASTER_SITE_BACKUP contains backup sites for packages that are maintained in . MASTER_SITE_LOCAL contains local package source distributions that are maintained in . If you choose one of these predefined sites, you may want to specify a subdirectory of that site. Since these macros may expand to more than one actual site, you must use the following construct to specify a subdirectory: MASTER_SITES= ${MASTER_SITE_GNU:=subdirectory/name/} MASTER_SITES= ${MASTER_SITE_SOURCEFORGE:=project_name/} Note the trailing slash after the subdirectory name. The fetch phase makes sure that all the distfiles exist in a local directory (DISTDIR, which can be set by the pkgsrc user). If the files do not exist, they are fetched using commands of the form ${FETCH_CMD} ${FETCH_BEFORE_ARGS} ${site}${file} ${FETCH_AFTER_ARGS} where ${site} varies through several possibilities in turn: first, MASTER_SITE_OVERRIDE is tried, then the sites specified in either SITES.file if defined, else MASTER_SITES or PATCH_SITES, as applies, then finally the value of MASTER_SITE_BACKUP. The order of all except the first and the last can be optionally sorted by the user, via setting either MASTER_SORT_RANDOM, and MASTER_SORT_AWK or MASTER_SORT_REGEX. The specific command and arguments used depend on the FETCH_USING parameter. The example above is for FETCH_USING=custom. The distfiles mirror run by the NetBSD Foundation uses the mirror-distfiles target to mirror the distfiles, if they are freely distributable. Packages setting NO_SRC_ON_FTP (usually to ${RESTRICTED}) will not have their distfiles mirrored. After the distfile(s) are fetched, their checksum is generated and compared with the checksums stored in the distinfo file. If the checksums don't match, the build is aborted. This is to ensure the same distfile is used for building, and that the distfile wasn't changed, e.g. by some malign force, deliberately changed distfiles on the master distribution site or network lossage. When the distfiles are present on the local system, they need to be extracted, as they usually come in the form of some compressed archive format. By default, all DISTFILES are extracted. If you only need some of them, you can set the EXTRACT_ONLY variable to the list of those files. Extracting the files is usually done by a little program, mk/extract/extract, which already knows how to extract various archive formats, so most likely you will not need to change anything here. But if you need, the following variables may help you: EXTRACT_OPTS_{BIN,LHA,PAX,RAR,TAR,ZIP,ZOO} Use these variables to override the default options for an extract command, which are defined in mk/extract/extract. EXTRACT_USING This variable can be set to bsdtar, gtar, nbtar (which is the default value), pax, or an absolute pathname pointing to the command with which tar archives should be extracted. It is preferred to choose bsdtar over gtar if NetBSD's pax-as-tar is not good enough. If the extract program doesn't serve your needs, you can also override the EXTRACT_CMD variable, which holds the command used for extracting the files. This command is executed in the ${WRKSRC} directory. During execution of this command, the shell variable extract_file holds the absolute pathname of the file that is going to be extracted. And if that still does not suffice, you can override the do-extract target in the package Makefile. After extraction, all the patches named by the PATCHFILES, those present in the patches subdirectory of the package as well as in $LOCALPATCHES/$PKGPATH (e.g. /usr/local/patches/graphics/png) are applied. Patchfiles ending in .Z or .gz are uncompressed before they are applied, files ending in .orig or .rej are ignored. Any special options to &man.patch.1; can be handed in PATCH_DIST_ARGS. See for more details. By default &man.patch.1; is given special arguments to make it fail if the expected text from the patch context is not found in the patched file. If that happens, fix the patch file by comparing it with the actual text in the file to be patched. This is covered in . This phase creates wrapper programs for the compilers and linkers. The following variables can be used to tweak the wrappers. ECHO_WRAPPER_MSG The command used to print progress messages. Does nothing by default. Set to ${ECHO} to see the progress messages. WRAPPER_DEBUG This variable can be set to yes (default) or no, depending on whether you want additional information in the wrapper log file. WRAPPER_UPDATE_CACHE This variable can be set to yes or no, depending on whether the wrapper should use its cache, which will improve the speed. The default value is yes, but is forced to no if the platform does not support it. WRAPPER_REORDER_CMDS A list of reordering commands. A reordering command has the form reorder:l:lib1:lib2. It ensures that that -llib1 occurs before -llib2. Most pieces of software need information on the header files, system calls, and library routines which are available on the platform they run on. The process of determining this information is known as configuration, and is usually automated. In most cases, a script is supplied with the distfiles, and its invocation results in generation of header files, Makefiles, etc. If the package contains a configure script, this can be invoked by setting HAS_CONFIGURE to yes. If the configure script is a GNU autoconf script, you should set GNU_CONFIGURE to yes instead. In the do-configure stage, a rough equivalent of the following command is run. See mk/configure/configure.mk, target do-configure-script for the exact definition. .for dir in ${CONFIGURE_DIRS} cd ${WRKSRC} && cd ${dir} \ && env ${CONFIGURE_ENV} \ ${CONFIG_SHELL} ${CONFIGURE_SCRIPT} ${CONFIGURE_ARGS} .endfor CONFIGURE_DIRS (default: .) is a list of pathnames relative to WRKSRC. In each of these directories, the configure script is run with the environment CONFIGURE_ENV and arguments CONFIGURE_ARGS. The variables CONFIGURE_ENV, CONFIGURE_SCRIPT (default: ./configure) and CONFIGURE_ARGS may all be changed by the package. If the program uses the Perl way of configuration (mainly Perl modules, but not only), i.e. a file called Makefile.PL, it should include ../../lang/perl5/module.mk. To set any parameter for Makefile.PL use the MAKE_PARAMS variable (e.g., MAKE_PARAMS+=foo=bar If the program uses an Imakefile for configuration, the appropriate steps can be invoked by setting USE_IMAKE to yes. If you only need xmkmf, add it to USE_TOOLS. You can add variables to xmkmf's environment by adding them to the SCRIPTS_ENV variable. If the program uses cmake for configuration, the appropriate steps can be invoked by including ../../devel/cmake/build.mk. You can add variables to cmake's environment by adding them to the CONFIGURE_ENV variable and arguments to cmake by adding them to the CMAKE_ARGS variable. If you want to add arguments only for particular stages, you can use the CMAKE_CONFIGURE_ARGS, CMAKE_BUILD_ARGS, and CMAKE_INSTALL_ARGS variables. You can set the CONFIGURE_DIRS variable to the directories in which CMake should be run, relative to WRKSRC. This defaults to to .. If there is no configure step at all, set NO_CONFIGURE to yes. For building a package, a rough equivalent of the following code is executed; see mk/build/build.mk, target do-build for the exact definition. .for dir in ${BUILD_DIRS} cd ${WRKSRC} && cd ${dir} \ && env ${MAKE_ENV} \ ${MAKE_PROGRAM} ${MAKE_FLAGS} ${BUILD_MAKE_FLAGS} \ -f ${MAKE_FILE} \ ${BUILD_TARGET} .endfor BUILD_DIRS (default: .) is a list of pathnames relative to WRKSRC. In each of these directories, MAKE_PROGRAM is run with the environment MAKE_ENV and arguments BUILD_MAKE_FLAGS. The variables MAKE_ENV, BUILD_MAKE_FLAGS, MAKE_FILE and BUILD_TARGET may all be changed by the package. The default value of MAKE_PROGRAM is gmake if USE_TOOLS contains gmake, make otherwise. The default value of MAKE_FILE is Makefile, and BUILD_TARGET defaults to all. If there is no build step at all, set NO_BUILD to yes. [TODO] Once the build stage has completed, the final step is to install the software in public directories, so users can access the programs and files. In the install phase, a rough equivalent of the following code is executed; see mk/install/install.mk, target do-install for the exact definition. Additionally, before and after this code, several consistency checks are run against the files-to-be-installed, see mk/check/*.mk for details. .for dir in ${INSTALL_DIRS} cd ${WRKSRC} && cd ${dir} \ && env ${INSTALL_ENV} ${MAKE_ENV} \ ${MAKE_PROGRAM} ${MAKE_FLAGS} ${INSTALL_MAKE_FLAGS} \ -f ${MAKE_FILE} ${INSTALL_TARGET} .endfor The variable's meanings are analogous to the ones in the build phase. INSTALL_DIRS defaults to BUILD_DIRS. INSTALL_TARGET is install by default, plus install.man if USE_IMAKE is defined and NO_INSTALL_MANPAGES is not defined. In the install phase, the following variables are useful. They are all variations of the &man.install.1; command that have the owner, group and permissions preset. INSTALL is the plain install command. The specialized variants, together with their intended use, are: INSTALL_PROGRAM_DIR directories that contain binaries INSTALL_SCRIPT_DIR directories that contain scripts INSTALL_LIB_DIR directories that contain shared and static libraries INSTALL_DATA_DIR directories that contain data files INSTALL_MAN_DIR directories that contain man pages INSTALL_GAME_DIR directories that contain data files for games INSTALL_PROGRAM binaries that can be stripped from debugging symbols INSTALL_SCRIPT binaries that cannot be stripped INSTALL_GAME game binaries INSTALL_LIB shared and static libraries INSTALL_DATA data files INSTALL_GAME_DATA data files for games INSTALL_MAN man pages Some other variables are: INSTALL_UNSTRIPPED If set to yes, do not run &man.strip.1; when installing binaries. Any debugging sections and symbols present in binaries will be preserved. INSTALLATION_DIRS A list of directories relative to PREFIX that are created by pkgsrc at the beginning of the install phase. The package is supposed to create all needed directories itself before installing files to it and list all other directories here. In the rare cases that a package shouldn't install anything, set NO_INSTALL to yes. This is mostly relevant for packages in the regress category. Once the install stage has completed, a binary package of the installed files can be built. These binary packages can be used for quick installation without previous compilation, e.g. by the make bin-install or by using pkg_add. By default, the binary packages are created in ${PACKAGES}/All and symlinks are created in ${PACKAGES}/category, one for each category in the CATEGORIES variable. PACKAGES defaults to pkgsrc/packages. Once you're finished with a package, you can clean the work directory by running make clean. If you want to clean the work directories of all dependencies too, use make clean-depends. pre/post-* For any of the main targets described in the previous section (configure, build, install, etc.), two auxiliary targets exist with pre- and post- used as a prefix for the main target's name. These targets are invoked before and after the main target is called, allowing extra configuration or installation steps be performed from a package's Makefile, for example, which a program's configure script or install target omitted. About 5% of the pkgsrc packages define their custom post-extract target, another 5% define pre-configure, and 10% define post-install. The other pre/post-* targets are defined even less often. do-* Should one of the main targets do the wrong thing, and should there be no variable to fix this, you can redefine it with the do-* target. (Note that redefining the target itself instead of the do-* target is a bad idea, as the pre-* and post-* targets won't be called anymore, etc.) About 15% of the pkgsrc packages override the default do-install, the other do-* targets are overridden even less often. reinstall If you did a make install and you noticed some file was not installed properly, you can repeat the installation with this target, which will ignore the already installed flag. This is the default value of DEPENDS_TARGET except in the case of make update and make package, where the defaults are package and update, respectively. deinstall This target does a &man.pkg.delete.1; in the current directory, effectively de-installing the package. The following variables can be used to tune the behaviour: PKG_VERBOSE Add a "-v" to the &man.pkg.delete.1; command. DEINSTALLDEPENDS Remove all packages that require (depend on) the given package. This can be used to remove any packages that may have been pulled in by a given package, e.g. if make deinstall DEINSTALLDEPENDS=1 is done in pkgsrc/x11/kde, this is likely to remove whole KDE. Works by adding -R to the &man.pkg.delete.1; command line. bin-install Install a binary package from local disk and via FTP from a list of sites (see the BINPKG_SITES variable), and do a make package if no binary package is available anywhere. The arguments given to pkg_add can be set via BIN_INSTALL_FLAGS e.g., to do verbose operation, etc. install-clean This target removes the state files for the "install" and later phases so that the "install" target may be re-invoked. This can be used after editing the PLIST to install the package without rebuilding it. build-clean This target removes the state files for the "build" and later phases so that the "build" target may be re-invoked. update This target causes the current package to be updated to the latest version. The package and all depending packages first get de-installed, then current versions of the corresponding packages get compiled and installed. This is similar to manually noting which packages are currently installed, then performing a series of make deinstall and make install (or whatever UPDATE_TARGET is set to) for these packages. You can use the update target to resume package updating in case a previous make update was interrupted for some reason. However, in this case, make sure you don't call make clean or otherwise remove the list of dependent packages in WRKDIR. Otherwise, you lose the ability to automatically update the current package along with the dependent packages you have installed. Resuming an interrupted make update will only work as long as the package tree remains unchanged. If the source code for one of the packages to be updated has been changed, resuming make update will most certainly fail! The following variables can be used either on the command line or in &mk.conf; to alter the behaviour of make update: UPDATE_TARGET Install target to recursively use for the updated package and the dependent packages. Defaults to DEPENDS_TARGET if set, install otherwise for make update. Other good targets are package or bin-install. Do not set this to update or you will get stuck in an endless loop! NOCLEAN Don't clean up after updating. Useful if you want to leave the work sources of the updated packages around for inspection or other purposes. Be sure you eventually clean up the source tree (see the clean-update target below) or you may run into troubles with old source code still lying around on your next make or make update. REINSTALL Deinstall each package before installing (making DEPENDS_TARGET). This may be necessary if the clean-update target (see below) was called after interrupting a running make update. DEPENDS_TARGET Allows you to disable recursion and hardcode the target for packages. The default is update for the update target, facilitating a recursive update of prerequisite packages. Only set DEPENDS_TARGET if you want to disable recursive updates. Use UPDATE_TARGET instead to just set a specific target for each package to be installed during make update (see above). clean-update Clean the source tree for all packages that would get updated if make update was called from the current directory. This target should not be used if the current package (or any of its depending packages) have already been de-installed (e.g., after calling make update) or you may lose some packages you intended to update. As a rule of thumb: only use this target before the first time you run make update and only if you have a dirty package tree (e.g., if you used NOCLEAN). If you are unsure about whether your tree is clean, you can either perform a make clean at the top of the tree, or use the following sequence of commands from the directory of the package you want to update (before running make update for the first time, otherwise you lose all the packages you wanted to update!): &rprompt; make clean-update &rprompt; make clean CLEANDEPENDS=YES &rprompt; make update The following variables can be used either on the command line or in &mk.conf; to alter the behaviour of make clean-update: CLEAR_DIRLIST After make clean, do not reconstruct the list of directories to update for this package. Only use this if make update successfully installed all packages you wanted to update. Normally, this is done automatically on make update, but may have been suppressed by the NOCLEAN variable (see above). replace Update the installation of the current package. This differs from update in that it does not replace dependent packages. You will need to install pkgtools/pkg_tarup for this target to work. Be careful when using this target! There are no guarantees that dependent packages will still work, in particular they will most certainly break if you make replace a library package whose shared library major version changed between your installed version and the new one. For this reason, this target is not officially supported and only recommended for advanced users. info This target invokes &man.pkg.info.1; for the current package. You can use this to check which version of a package is installed. index This is a top-level command, i.e. it should be used in the pkgsrc directory. It creates a database of all packages in the local pkgsrc tree, including dependencies, comment, maintainer, and some other useful information. Individual entries are created by running make describe in the packages' directories. This index file is saved as pkgsrc/INDEX. It can be displayed in verbose format by running make print-index. You can search in it with make search key=something. You can extract a list of all packages that depend on a particular one by running make show-deps PKG=somepackage. Running this command takes a very long time, some hours even on fast machines! readme This target generates a index.html file, which can be viewed using a browser such as www/firefox or www/links. The generated files contain references to any packages which are in the PACKAGES directory on the local host. The generated files can be made to refer to URLs based on FTP_PKG_URL_HOST and FTP_PKG_URL_DIR. For example, if I wanted to generate index.html files which pointed to binary packages on the local machine, in the directory /usr/packages, set FTP_PKG_URL_HOST=file://localhost and FTP_PKG_URL_DIR=/usr/packages. The ${PACKAGES} directory and its subdirectories will be searched for all the binary packages. The target can be run at the toplevel or in category directories, in which case it descends recursively. readme-all This is a top-level command, run it in pkgsrc. Use this target to create a file README-all.html which contains a list of all packages currently available in the &os; Packages Collection, together with the category they belong to and a short description. This file is compiled from the pkgsrc/*/index.html files, so be sure to run this after a make readme. cdrom-readme This is very much the same as the readme target (see above), but is to be used when generating a pkgsrc tree to be written to a CD-ROM. This target also produces index.html files, and can be made to refer to URLs based on CDROM_PKG_URL_HOST and CDROM_PKG_URL_DIR. show-distfiles This target shows which distfiles and patchfiles are needed to build the package (ALLFILES, which contains all DISTFILES and PATCHFILES, but not patches/*). show-downlevel This target shows nothing if the package is not installed. If a version of this package is installed, but is not the version provided in this version of pkgsrc, then a warning message is displayed. This target can be used to show which of your installed packages are downlevel, and so the old versions can be deleted, and the current ones added. show-pkgsrc-dir This target shows the directory in the pkgsrc hierarchy from which the package can be built and installed. This may not be the same directory as the one from which the package was installed. This target is intended to be used by people who may wish to upgrade many packages on a single host, and can be invoked from the top-level pkgsrc Makefile by using the show-host-specific-pkgs target. show-installed-depends This target shows which installed packages match the current package's DEPENDS. Useful if out of date dependencies are causing build problems. print-build-depends-list This target shows the list of packages that the current package depends on for building. print-run-depends-list This target shows the list of packages that the current package depends on for running. check-shlibs After a package is installed, check all its binaries and (on ELF platforms) shared libraries to see if they find the shared libs they need. Run by default if PKG_DEVELOPER is set in &mk.conf;. print-PLIST After a make install from a new or upgraded pkg, this prints out an attempt to generate a new PLIST from a find -newer work/.extract_done. An attempt is made to care for shared libs etc., but it is strongly recommended to review the result before putting it into PLIST. On upgrades, it's useful to diff the output of this command against an already existing PLIST file. If the package installs files via &man.tar.1; or other methods that don't update file access times, be sure to add these files manually to your PLIST, as the find -newer command used by this target won't catch them! See for more information on this target.