GSoC 2018 Reports: Configuration files versioning in pkgsrc, part 2: remote repositories (git and CVS)

Prepared by Keivan Motavalli as part of GSoC 2018.

In Configuration files versioning in pkgsrc, Part 1 basic pkgsrc support for tracking package configuration changes was introduced, as were the features that attempt to automatically merge installed configuration files with new defaults, all shown using RCS as the backing Version Control System and passing options by setting environment variables.

Some of them have changed (conf tracking now needs to be explicitly enabled), support is now also in pkg_install.conf and aside from RCS, pkgsrc now supports CVS, Git, mercurial and SVN, both locally and connecting to remote resources.

Furthermore, pkgsrc is now able to deploy configuration from packages being installed from a remote, site-specific vcs repository.

User modified files are always tracked even if automerge functionality is not enabled, and a new tool, pkgconftrack(1), exists to manually store user changes made outside of package upgrade time.

Version Control software is executed as the same user running pkg_add or make install, unless the user is "root". In this case, a separate, unprivileged user, pkgvcsconf, gets created with its own home directory and a working login shell (but no password). The home directory is not strictly necessary, it exists to facilitate migrations betweens repositories and vcs changes; it also serves to store keys used to access remote repositories.

Files, objects and other data in the working directory are only accessible by UID 0 and pkgvcsconf, but be aware, if you want to login to remote repositories via ssh, keys will now need to be placed at pkgvcsconf own $HOME/.ssh/ directory, and ssh-agent, if you need to use it, also should have its socket, as defined by SSH_AUTH_SOCK, accessible by pkgvcsconf:pkgvcsconf.

New pkgsrc options

NOVCS, which was used to disable configuration tracking via Version Control Systems, doesn't exist anymore as an option or environment variable. In its place, VCSTRACK_CONF now needs to be set truthfully in order to enable configuration files version tracking. This can be done temporarily via environment variables before using pkgsrc or installing binary packages, such as by exporting

export VCSTRACK_CONF=yes

on the tty, or permanently in pkg_install.conf located under the PREFIX in use.

After bootstrapping a pkgsrc branch that supports the new features, check new options by calling, if you are using /usr/pkg as the PREFIX:

man -M /usr/pkg/man/ pkg_install.conf

Remember to edit /usr/pkg/etc/pkg_install.conf in order to set VCSTRACK_CONF=yes. Also try out VCSAUTOMERGE=yes to attempt merging installed configuration files with new defaults, as described in the first blog post. Installed configuration files are always backed up before an attempt is made to merge changes, and if conflicts are reported no changes get automatically installed in place of the existing configuration files, so that the user can manually review them.

Both VCSTRACK_CONF and VCSAUTOMERGE variables, if set on the environment, take precedence over values defined in pkg_install.conf in order to allow for an easy override at runtime. All other vcs-related options defined in pkg_install.conf replace those defined as environment variables.

These are:

VCS, sets the backend used to track configuration files. If unset, rcs gets called. pkgsrc now also supports Git, SVN, HG and CVS. More on that will get introduced later in the post.

VCSDIR, the path used as a working directory by pkginstall scripts run at package installation by pkg_add and looked up by pkgconftrack(1). Files are copied there before being tracked in a VCS or merged, this directory also stores a local configuration repository or objects and metadata associated with remote repositories, according to the solution in use. By default, /var/confrepo is assumed as the default working directory and initialized.

REMOTEVCS, an URI describing how to access a remote repository, according to the format required by the chosen VCS. Remote repositories are disabled if this variable is unset, or if it is set to "no".

VCSCONFPULL, set it to "yes" to have pkgscr attempt to deploy configuration files from a remote repository upon package installation, instead of using package provided defaults. It now works with git, svn and mercurial.

ROLE, the role defined for the system, used when deploying configuration from a remote, site repository for packages being installed.

More details on system roles and configuration deployment are down in this post.

rcs vs network-capable systems

RCS, as stated, is the default backend because of its simplicity and widespread presence. When packages are being compiled from source, system rcs is used if present, otherwise it gets installed automatically by pkgsrc as a TOOL when the first package gets built, being an mk/pkginstall/bsd.pkginstall.mk tool dependency.

Please note that binaries defined as pkgsrc TOOLs will get their path, as on the build system, hardcoded in binary packages install scripts. This shouldn't be a problem for NetBSD binary package users since RCS is part of the base system installation and paths will match. Users who choose binary packages on other platforms and got pkgtools/pkg_install (e.g., pkg_add) by running pkgsrc bootstrap/bootstrap script should also be ok, since packages get built as part of the bootstrap procedure, making bsd.pkginstall.mk tool dependency on RCS tick. Also note that even when using other systems, automerge functionality depends on merge, part of Revision Control System.

Other software, such as git, svn, cvs and hg is not installed or called in pkginstall scripts as TOOL dependencies: they get searched in the PATH defined on the environment, and must be manually installed by the user via pkgsrc itself or other means before setting VCS= to the desired system.

Each will get briefly introduced, with an example setting up and using remote repositories.

pkgsrc will try to import existing and new files into the chosen VCS when it gets switched, and a previously undefined REMOTEVCS URI may be added to an existing repository, already existing files may get imported and pushed to the repository. It all depends on the workflow of the chosen system, but users SHOULD NOT expect pkgsrc to handle changes and the addition of remote repositories for them. At best there will be inconsistencies and the loss of revisions tracked in the old solution: the handling of repository switches is out of scope for this project.

Users who decide to switch to a different Version Control System should manually convert their local repository and start clean. Users who want to switch from local to using a remote repository should also, when setting it up, take care in populating it with local data and then start over, preferably by removing the path specified by VCSDIR or by setting it to a new path. pkgsrc will try its best to ensure new package installations don't fail because of changes made without taking the aforementioned steps, but no consistency with information stored in the old system should then be expected.

Some words on REMOTEVCS

REMOTEVCS, if defined and different from "no", points to the URI of a remote repository in a way understandable by the chosen version control system. It may include the access method, the user name, the remote FQDN or IP address, and the path to the repository. if a password gets set via userName:password@hostname, care should be taken to only set REMOTEVCS as an environment variable, leaving out login information from pkg_install.conf.

The preferred way to access remote repositories is to use ssh with asymmetrical keys. These should be placed under $HOME/.ssh/ for the user pkgsrc runs under, or under pkgvcsconf home directory if you are installing packages as root. If keys are password protected or present in non-standard search paths, ssh-agent should be started and the appropriate key unlocked via ssh-add before package installations or upgrades are run.

If running ssh-agent, SSH_AUTH_SOCK and SSH_AGENT_PID variables must be appropriately exported on the environment the package installations will take place in; CVS_RSH should be set to the path of the ssh executable and exported on the environment when using CVS to access a remote repository over ssh.

Git and remotes

git, pretty straightforwardly, will store objects, metadata and configuration at VCSDIR/.git, while VCSDIR is defined as the working-tree files get checked out to.

The configuration under VCSDIR/.git includes the definition of the remote repository, which the +VERSIONING script will set once in the PREPARE phase; the script can also switch the remote to a new one, when using git. Data consistency should be ensured by actions taken beforehand by the user.

Using git instead of rcs is simply done by setting VCS=git in pkg_install.conf

.

pkgsrc will inizialize the repository.

Please do this before installing packages that come with configuration files, then remove the old VCSDIR (e.g., by running rm -fr /var/confrepo) and let pkgsrc take care of the rest, or set VCSDIR to a new path.

If you wish to migrate an existing repository to git, seek documentation for scripts such as rcs-fast-export and git fast-import, but initialize it under a new working directory. Running git on top of an older, different repository without migrating data first will allow you to store new files, but automerge may break in strange ways when it tries to extract the first revision of a configuration file from the repository, if it is only tracked in the previous version control system.

Now, some steps needed to setup a remote git repository are also common to cvs, svn and mercurial: I will run a server at 192.168.100.112 with ssh access enabled for the user vcs. An ssh key has been generated under /root/.ssh and added to vcs@192.168.100.112 authorized_keys list for the pkgsrc host to automatically login as vcs on the remote instance.

In order to create a remote repository, login as vcs on the server and run

$ pwd                                                                                                         
/usr/home/vcs                                                                                                 
$ mkdir gitconftrack.git                                                                                      
$ cd gitconftrack.git/                                                                                        
$ git --bare init                                                                                             
Initialized empty Git repository in /usr/home/vcs/gitconftrack.git/                                           
$ cat /usr/home/vcs/.ssh/authorized_keys                                                                      
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABA[...]
$ /etc/rc.d/sshd status                                                                                       
sshd is running as pid 1343.                                                                                  

now, on the client, set the URI as ssh://vcs@192.168.100.112/usr/home/vcs/gitconftrack.git (full path here! hg and svn use paths relative to the user's home directory

pkghost# echo "VCSTRACK_CONF=yes" >> /usr/pkg/etc/pkg_install.conf
pkghost# echo "VCS=git" >> /usr/pkg/etc/pkg_install.conf
pkghost# echo "VCSDIR=/var/gitandremote" >> /usr/pkg/etc/pkg_install.conf
pkghost# echo "VCSAUTOMERGE=yes" >> /usr/pkg/etc/pkg_install.conf
pkghost# echo "REMOTEVCS=ssh://vcs@192.168.100.112/usr/home/vcs/gitconftrack.git
pkghost# cat /usr/pkg/etc/pkg_install.conf 
VCSTRACK_CONF=yes
VCS=git
VCSDIR=/var/gitandremote
VCSAUTOMERGE=yes
REMOTEVCS=ssh://vcs@192.168.100.112/usr/home/vcs/gitconftrack.git
pkghost# 

and try to install spamd as with the examples in the first post (it comes with a configuration file and is written in shell scripts that don't need to be compiled or many dependencies installed when testing things out)

pkghost# cd pkgsrc/mail/spamd
pkghost# make
=> Bootstrap dependency digest>=20010302: found digest-20160304
===> Skipping vulnerability checks.
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.
=> Fetching spamd-20060330.tar.gz

[...]

pkghost# make install
pkghost# make install
=> Bootstrap dependency digest>=20010302: found digest-20160304
===> Skipping vulnerability checks.
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.
===> Installing binary package of spamd-20060330nb2
fatal: Not a git repository: '/var/gitandremote/.git'
Initialized empty Git repository in /var/gitandremote/.git/
fatal: Couldn't find remote ref master
fatal: The remote end hung up unexpectedly
REGISTER /var/gitandremote/defaults//usr/pkg/etc/spamd.conf
spamd-20060330nb2: copying /usr/pkg/share/examples/spamd/spamd.conf to /usr/pkg/etc/spamd.conf
[master (root-commit) f84f8ee] pkgsrc: add spamd-20060330nb2
 1 file changed, 86 insertions(+)
 create mode 100644 defaults/usr/pkg/etc/spamd.conf
Conf commit: pkgsrc: add spamd-20060330nb2
Counting objects: 7, done.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (7/7), 1.34 KiB | 688.00 KiB/s, done.
Total 7 (delta 0), reused 0 (delta 0)
To ssh://192.168.100.112/usr/home/vcs/gitconftrack.git
 * [new branch]      master -> master
===========================================================================
The following files should be created for spamd-20060330nb2:

        /etc/rc.d/pfspamd (m=0755)
            [/usr/pkg/share/examples/rc.d/pfspamd]

===========================================================================
===========================================================================
$NetBSD: MESSAGE,v 1.1.1.1 2005/06/28 12:43:57 peter Exp $

Don't forget to add the spamd ports to /etc/services:

spamd           8025/tcp                # spamd(8)
spamd-cfg       8026/tcp                # spamd(8) configuration

===========================================================================
pkghost# 

Let's see what's at the VCSDIR now...

pkghost# cd /var/gitandremote/                                                                                
pkghost# ls                                                                                                   
.git       automerged defaults   user
pkghost# find defaults/ -type f -print                                                                        
defaults/usr/pkg/etc/spamd.conf
pkghost# git log                                                                                              
commit f84f8ee278ec53f55d652e5c6bf52554216cfd49 (HEAD -> master, origin/master)                               
Author: kmotavalli                                                                       
Date:   Fri Aug 3 15:00:26 2018 +0000                                                                         
                                                                                                              
    pkgsrc: add spamd-20060330nb2                                                                             
pkghost# git remote -v                                                                                        
origin  ssh://vcs@192.168.100.112/usr/home/vcs/gitconftrack.git (fetch)                                       
origin  ssh://vcs@192.168.100.112/usr/home/vcs/gitconftrack.git (push)                                        

on the server:

$ hostname                                                                                                    
vers                                                                                                          
$ pwd                                                                                                         
/usr/home/vcs/gitconftrack.git                                                                                
$ git log                                                                                                     
commit f84f8ee278ec53f55d652e5c6bf52554216cfd49 (HEAD -> master)                                              
Author: kmotavalli                                                                       
Date:   Fri Aug 3 15:00:26 2018 +0000                                                                         
                                                                                                              
    pkgsrc: add spamd-20060330nb2                                                                             
$

All fine, let's simulate automerge:

pkghost# vi /usr/pkg/etc/spamd.conf 

[...]

# Whitelists are done like this, and must be added to "all" after each                                        
# blacklist from which you want the addresses in the whitelist removed.                                       
#                                                                                                             
#whitelist:\
#       :white:\                                                                                              
#       :method=file:\                                                                                        
#       :file=/var/mail/whitelist.txt:

whitelist:\ 
        :white:\                                                                                              
        :method=file:\                                                                                        
        :file=/var/mail/whitelist.txt:
:x
/usr/pkg/etc/spamd.conf: 86 lines, 2767 characters
pkghost# cd $HOME/pkgsrc/mail/spamd; make replace
=> Bootstrap dependency digest>=20010302: found digest-20160304
===> Skipping vulnerability checks.
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.
===> Replacing for spamd-20060330nb2
===> Updating using binary package of spamd-20060330nb2
/usr/bin/env  /usr/pkg/sbin/pkg_add -K /var/db/pkg -U -D /root/pkgsrc/mail/spamd/work/.packages/spamd-20060330nb2.tgz
===========================================================================
The following users are no longer being used by spamd-20060330nb2,
and they can be removed if no other software is using them:

        _spamd

===========================================================================
===========================================================================
The following groups are no longer being used by spamd-20060330nb2,
and they can be removed if no other software is using them:

        _spamd

===========================================================================
===========================================================================
The following files are no longer being used by spamd-20060330nb2,
and they can be removed if no other packages are using them:

        /usr/pkg/etc/spamd.conf

===========================================================================
Everything up-to-date
From ssh://192.168.100.112/usr/home/vcs/gitconftrack
 * branch            master     -> FETCH_HEAD
Already up to date.
REGISTER /var/gitandremote/defaults//usr/pkg/etc/spamd.conf
spamd-20060330nb2: /usr/pkg/etc/spamd.conf already exists
spamd-20060330nb2: attempting to merge /usr/pkg/etc/spamd.conf with new defaults!
Saving the currently user-installed revision to /var/gitandremote/user//usr/pkg/etc/spamd.conf
[master eacdef9] pkgsrc: backup user conf before attempting merge for spamd-20060330nb2
 1 file changed, 86 insertions(+)
 create mode 100644 user/usr/pkg/etc/spamd.conf
Conf commit: pkgsrc: backup user conf before attempting merge for spamd-20060330nb2
Counting objects: 7, done.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (7/7), 1.44 KiB | 1.44 MiB/s, done.
Total 7 (delta 0), reused 0 (delta 0)
To ssh://192.168.100.112/usr/home/vcs/gitconftrack.git
   f84f8ee..eacdef9  master -> master
Merged with no conflicts. installing it to /usr/pkg/etc/spamd.conf!
Revert from the last revision of /var/gitandremote/user//usr/pkg/etc/spamd.conf if needed
[master 4f1d514] pkgsrc: add spamd-20060330nb2
 1 file changed, 86 insertions(+)
 create mode 100644 automerged/usr/pkg/etc/spamd.conf
Conf commit: pkgsrc: add spamd-20060330nb2
Counting objects: 2, done.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 280 bytes | 280.00 KiB/s, done.
Total 2 (delta 0), reused 0 (delta 0)
To ssh://192.168.100.112/usr/home/vcs/gitconftrack.git
   eacdef9..4f1d514  master -> master
===========================================================================
The following files should be created for spamd-20060330nb2:

        /etc/rc.d/pfspamd (m=0755)
            [/usr/pkg/share/examples/rc.d/pfspamd]

===========================================================================
===========================================================================
$NetBSD: MESSAGE,v 1.1.1.1 2005/06/28 12:43:57 peter Exp $

Don't forget to add the spamd ports to /etc/services:

spamd           8025/tcp                # spamd(8)
spamd-cfg       8026/tcp                # spamd(8) configuration

===========================================================================

A new file backing up the installed configuration, as edited by the user, has been created in the working directory at /var/gitandremote/user/usr/pkg/etc/spamd.conf and pushed to the remote repository.

You can check it out and rever from it in case something breaks:

pkghost# cd /var/gitandremote/user/                                                                           
pkghost# ls                                                                                                   
usr                                                                                                           
pkghost# git checkout usr/pkg/etc/spamd.conf
pkghost# tail /var/gitandremote/user/usr/pkg/etc/spamd.conf 
#       :method=exec:\                                                                                        
#       :file=/usr/local/bin/relaydb -4lw:                                                                    
                                                                                                              
# Whitelists are done like this, and must be added to "all" after each                                        
# blacklist from which you want the addresses in the whitelist removed.                                       
#                                                                                                             
whitelist:\                                                                                                   
        :white:\                                                                                              
        :method=file:\                                                                                        
        :file=/var/mail/whitelist.txt:                                                                        
pkghost# 

running git log on the server, we can see two new commits: one pushed when backing up the installed configuration file, one after the automerge succeded and the package got installed.

$ hostname                                                                                                    
vers                                                                                                          
$ pwd                                                                                                         
/usr/home/vcs/gitconftrack.git                                                                                
$ git log                                                                                                     
commit 4f1d514159860906154eeaaa24a10ec572a4d062 (HEAD -> master)                                              
Author: kmotavalli                                                                       
Date:   Fri Aug 3 15:12:45 2018 +0000                                                                         
                                                                                                              
    pkgsrc: add spamd-20060330nb2                                                                             
                                                                                                              
commit eacdef9f52b49e3ce6f544008dc7c75dc89cbd67                                                               
Author: kmotavalli                                                                       
Date:   Fri Aug 3 15:12:43 2018 +0000                                                                         
                                                                                                              
    pkgsrc: backup user conf before attempting merge for spamd-20060330nb2                                    
                                                                                                              
commit f84f8ee278ec53f55d652e5c6bf52554216cfd49                                                               
Author: kmotavalli                                                                       
Date:   Fri Aug 3 15:00:26 2018 +0000                                                                         
                                                                                                              
    pkgsrc: add spamd-20060330nb2                                                                             
$ 

Furthermore, I'll simulate the addition of a new comment to the package provided configuration file, then the change of a variable, which should generate an automerge conflict that needs manual user review. (Note: you don't need to change configuration files in pkgsrc package working directory! This is only done to simulate a change coming with a package upgrade!

pkghost# cd $HOME/pkgsrc/mail/spamd; make clean                                                               
===> Cleaning for spamd-20060330nb2
pkghost# make fetch                                                                                           
pkghost# sed -i 's/PKGREVISION=.*2/PKGREVISION=3/g' Makefile
pkghost# make extract                                                                                         
=> Bootstrap dependency digest>=20010302: found digest-20160304                                               
===> Skipping vulnerability checks.                                                                           
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.                                                       
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.                      
=> Checksum SHA1 OK for spamd-20060330.tar.gz                                                                 
=> Checksum RMD160 OK for spamd-20060330.tar.gz                                                               
=> Checksum SHA512 OK for spamd-20060330.tar.gz                                                               
===> Installing dependencies for spamd-20060330nb3                                                            
=> Build dependency cwrappers>=20150314: found cwrappers-20180325                                             
===> Overriding tools for spamd-20060330nb3                                                                   
===> Extracting for spamd-20060330nb3                                                                         
pkghost# head work/spamd-20060330/etc/spamd.conf 
# spamd config file, read by spamd-setup(8) for spamd(8)
#
# See spamd.conf(5)
#
# Configures whitelists and blacklists for spamd
#
# Strings follow getcap(3) convention escapes, other than you
# can have a bare colon (:) inside a quoted string and it
# will deal with it. See spamd-setup(8) for more details.

pkghost# sed -i 's/method=http/method=https/g' work/spamd-20060330/etc/spamd.conf

                                                             .                                                
pkghost# make update                                                                                          
===> Skipping vulnerability checks.                                                                           
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.                                                       
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.                      
===> Deinstalling for spamd-20060330nb4                                                                       
Running /usr/pkg/sbin/pkg_delete -K /var/db/pkg -r spamd-20060330nb3                                          
===========================================================================                                   
The following users are no longer being used by spamd-20060330nb3,                                            
and they can be removed if no other software is using them:                                                   
                                                                                                              
        _spamd                                                                                                
                                                                                                              
===========================================================================                                   
===========================================================================                                   
The following groups are no longer being used by spamd-20060330nb3,                                           
and they can be removed if no other software is using them:                                                   
                                                                                                              
        _spamd                                                                                                
                                                                                                              
===========================================================================                                   
===========================================================================                                   
The following files are no longer being used by spamd-20060330nb3,                                            
and they can be removed if no other packages are using them:                                                  
                                                                                                              
        /usr/pkg/etc/spamd.conf                                                                               
                                                                                                              
===========================================================================                                   
=> Bootstrap dependency digest>=20010302: found digest-20160304                                               
===> Skipping vulnerability checks.                                                                           
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.                                                       
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.                      
===> Patching for spamd-20060330nb4                                                                           
=> Applying pkgsrc patches for spamd-20060330nb4                                                              
=> Fixing configuration paths.                                                                                
===> Creating toolchain wrappers for spamd-20060330nb4                                                        
===> Building for spamd-20060330nb4

[...]

=> Creating /root/pkgsrc/mail/spamd/work/.rc.d/pfspamd                                                        
===> Installing for spamd-20060330nb4                                                                         
=> Generating pre-install file lists                                                                          
=> Creating installation directories                                                                          
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/man/spamd.conf.5  /ro5
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd/spamd.8  /root/8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd-setup/spamd-set8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamdb/spamdb.8  /roo8
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamlogd/spamlogd.8  8
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd-setup/spamd-c
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamd/spamd  /rootc
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamdb/spamdb  /ron
/usr/bin/install -c -s -o root -g wheel -m 755 /root/pkgsrc/mail/spamd/work/spamd-20060330/spamlogd/spamlogd c
/usr/bin/install -c -o root -g wheel -m 644 /root/pkgsrc/mail/spamd/work/spamd-20060330/etc/spamd.conf /root/d
=> Automatic manual page handling                                                                             
=> Generating post-install file lists                                                                         
=> Checking file-check results for spamd-20060330nb4                                                          
=> Creating binary package /root/pkgsrc/mail/spamd/work/.packages/spamd-20060330nb4.tgz                       
===> Building binary package for spamd-20060330nb4                                                            
=> Creating binary package /root/pkgsrc/packages/All/spamd-20060330nb4.tgz                                    
===> Installing binary package of spamd-20060330nb4                                                           
Everything up-to-date                                                                                         
From ssh://192.168.100.112/usr/home/vcs/gitconftrack                                                          
 * branch            master     -> FETCH_HEAD                                                                 
Already up to date.                                                                                           
REGISTER /var/gitretest/defaults//usr/pkg/etc/spamd.conf                                                      
spamd-20060330nb4: /usr/pkg/etc/spamd.conf already exists                                                     
spamd-20060330nb4: attempting to merge /usr/pkg/etc/spamd.conf with new defaults!                             
Saving the currently installed revision to /var/gitretest/automerged//usr/pkg/etc/spamd.conf                  
[master 8970767] pkgsrc: backup preexisting conf before attempting merge for spamd-20060330nb4                
 1 file changed, 17 insertions(+), 17 deletions(-)                                                            
Conf commit: pkgsrc: backup preexisting conf before attempting merge for spamd-20060330nb4                    
Counting objects: 7, done.                                                                                    
Compressing objects: 100% (3/3), done.                                                                        
Writing objects: 100% (7/7), 599 bytes | 599.00 KiB/s, done.                                                  
Total 7 (delta 1), reused 0 (delta 0)                                                                         
To ssh://192.168.100.112/usr/home/vcs/gitconftrack.git                                                        
   8dd7368..8970767  master -> master                                                                         
Merged with no conflict. installing it to /usr/pkg/etc/spamd.conf!                                            
--- /usr/pkg/etc/spamd.conf     2018-08-04 06:13:39.017744154 +0000                                           
+++ /var/gitretest/defaults//usr/pkg/etc/spamd.conf.automerge   2018-08-04 06:15:09.799758773 +0000           
@@ -15,7 +15,7 @@                                                                                             
 # may be applied to each blacklist.                                                                          
 #                                                                                                            
 # As of November 2004, a place to search for black lists is                                                  
-#     http://spamlinks.net/filter-bl.htm                                                                     
+#     https://spamlinks.net/filter-bl.htm                                                                    
 #                                                                                                            
 # Some of the URLs below point to www.openbsd.org locations.  Those                                          
 # files are likely to be mirrored to other OpenBSD www mirrors located                                       
@@ -25,45 +25,45 @@                                                                                           
 all:\                                                                                                        
        :spews1:china:korea:                                                                                  
                                                                                                              
-# Mirrored from http://spfilter.openrbl.org/data/sbl/SBL.cidr.bz2                                            
+# Mirrored from https://spfilter.openrbl.org/data/sbl/SBL.cidr.bz2                                           
 spamhaus:\                                                                                                   
        :black:\                                                                                              
        :msg="SPAM. Your address %A is in the Spamhaus Block List\n\                                          
-       See http://www.spamhaus.org/sbl and\                                                                  
-       http://www.abuse.net/sbl.phtml?IP=%A for more details":\                                              
-       :method=http:\                                                                                        
+       See https://www.spamhaus.org/sbl and\                                                                 
+       https://www.abuse.net/sbl.phtml?IP=%A for more details":\                                             
+       :method=https:\                                                                                       
        :file=www.openbsd.org/spamd/SBL.cidr.gz:                                                              
                                                                                                              
-# Mirrored from http://www.spews.org/spews_list_level1.txt                                                   
+# Mirrored from https://www.spews.org/spews_list_level1.txt                                                  
 spews1:\                                                                                                     
        :black:\                                                                                              
        :msg="SPAM. Your address %A is in the spews level 1 database\n\                                       
-       See http://www.spews.org/ask.cgi?x=%A for more details":\                                             
-       :method=http:\                                                                                        
+       See https://www.spews.org/ask.cgi?x=%A for more details":\                                            
+       :method=https:\                                                                                       
        :file=www.openbsd.org/spamd/spews_list_level1.txt.gz:                                                 
                                                                                                              
-# Mirrored from http://www.spews.org/spews_list_level2.txt                                                   
+# Mirrored from https://www.spews.org/spews_list_level2.txt                                                  
 spews2:\                                                                                                     
        :black:\                                                                                              
        :msg="SPAM. Your address %A is in the spews level 2 database\n\                                       
-       See http://www.spews.org/ask.cgi?x=%A for more details":\                                             
-       :method=http:\                                                                                        
+       See https://www.spews.org/ask.cgi?x=%A for more details":\                                            
+       :method=https:\                                                                                       
        :file=www.openbsd.org/spamd/spews_list_level2.txt.gz:                                                 
                                                                                                              
-# Mirrored from http://www.okean.com/chinacidr.txt                                                           
+# Mirrored from https://www.okean.com/chinacidr.txt                                                          
 china:\                                                                                                      
        :black:\                                                                                              
        :msg="SPAM. Your address %A appears to be from China\n\                                               
-       See http://www.okean.com/asianspamblocks.html for more details":\                                     
-       :method=http:\                                                                                        
+       See https://www.okean.com/asianspamblocks.html for more details":\                                    
+       :method=https:\                                                                                       
        :file=www.openbsd.org/spamd/chinacidr.txt.gz:                                                         
                                                                                                              
-# Mirrored from http://www.okean.com/koreacidr.txt                                                           
+# Mirrored from https://www.okean.com/koreacidr.txt                                                          
 korea:\                                                                                                      
        :black:\                                                                                              
        :msg="SPAM. Your address %A appears to be from Korea\n\                                               
-       See http://www.okean.com/asianspamblocks.html for more details":\                                     
-       :method=http:\                                                                                        
+       See https://www.okean.com/asianspamblocks.html for more details":\                                    
+       :method=https:\                                                                                       
        :file=www.openbsd.org/spamd/koreacidr.txt.gz:                                                         
                                                                                                              
 #relaydb-black:\                                                                                             
Revert from the penultimate revision of /var/gitretest/automerged//usr/pkg/etc/spamd.conf if needed           
[master 1eef6d8] pkgsrc: add spamd-20060330nb4                                                                
 1 file changed, 17 insertions(+), 17 deletions(-)                                                            
Conf commit: pkgsrc: add spamd-20060330nb4                                                                    
Counting objects: 7, done.                                                                                    
Compressing objects: 100% (3/3), done.                                                                        
Writing objects: 100% (7/7), 563 bytes | 563.00 KiB/s, done.                                                  
Total 7 (delta 1), reused 0 (delta 0)                                                                         
To ssh://192.168.100.112/usr/home/vcs/gitconftrack.git                                                        
   8970767..1eef6d8  master -> master                                                                         
===========================================================================                                   
The following files should be created for spamd-20060330nb4:                                                  
                                                                                                              
        /etc/rc.d/pfspamd (m=0755)                                                                            
            [/usr/pkg/share/examples/rc.d/pfspamd]                                                            
                                                                                                              
===========================================================================                                   
===========================================================================                                   
$NetBSD: MESSAGE,v 1.1.1.1 2005/06/28 12:43:57 peter Exp $                                                    
                                                                                                              
Don't forget to add the spamd ports to /etc/services:                                                         
                                                                                                              
spamd           8025/tcp                # spamd(8)                                                            
spamd-cfg       8026/tcp                # spamd(8) configuration                                              
                                                                                                              
===========================================================================                                   
===> Cleaning for spamd-20060330nb4                                                                           
[...]

pkghost# cat /var/gitandremote/automergedfiles 
/usr/pkg/etc/spamd.conf
pkghost# 

Should conflicts arise, these will be highlited as per rcs 3-way merge syntax in the file /var/gitandremote/defaults/usr/pkg/etc/spamd.conf.automerge and no action taken.

In that case, manually review the conflict opening spamd.conf.automerge in your favorite text editor, then manually copy it in place of the existing installed configuration file for spamd at /usr/pkg/etc/spamd.conf

CVS and remotes

CVS is used by having a CVSROOT directory at VCSDIR/CVSROOT, with VCSDIR as the working directory. The same precautions treated illustrating Git usage also apply when chosing to use CVS (and svn, mercurial). To use CVS, simply set VCS=cvs in PREFIX/etc/pkg_install.conf and change the VCSDIR to a new path.

pkgsrc will try to import existing files in CVS if the VCSDIR is not empty, but past files revision will not get migrated from whatever version control system was in use, nor will files present in the old repository but not checked out in the working directory. This will likely make automerge malfunction, so it's up to the user to migrate preeexisting repositories to CVS or to chose a new, empty path as the VCS working directory, if automerge has never been used up to this change (VCSAUTOMERGE not set to yes or VCSDIR/automergedfiles not existing/empty).

See CVS documentation for a description of how to form remote URIs (that will get passed to cvs with the -d flag)

In this example, I will use :ext:vcs@192.168.100.112:/usr/home/vcs/cvsconftrack as the REMOTEVCS uri

pkghost# cat /usr/pkg/etc/pkg_install.conf 
VCSTRACK_CONF=yes
VCS=cvs
VCSDIR=/var/cvsandremote
VCSAUTOMERGE=yes
REMOTEVCS=:ext:vcs@192.168.100.112:/usr/home/vcs/cvsconftrack
pkghost# echo "export CVS_RSH=ssh" >> $HOME/.profile
pkghost# export CVS_RSH=ssh                                                                                    
pkghost# 

and initialize a remote cvs repository on the server at 192.168.100.112:

$ hostname                                                                                                    
vers                                                                                                          
$ pwd                                                                                                         
/usr/home/vcs                                                                                                 
$ cvs -d/usr/home/vcs/cvsconftrack init                                                                       

manually migrate existing data from other systems to the new repository if needed, then try installing some package:

pkghost# pwd
/root/pkgsrc/mail/spamd
pkghost# make install
=> Bootstrap dependency digest>=20010302: found digest-20160304
===> Skipping vulnerability checks.
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.
===> Installing binary package of spamd-20060330nb5
cvs rlog: cannot find module `defaults' - ignored
cvs checkout: Updating automerged
cvs checkout: Updating defaults
cvs checkout: Updating user
cvs update: Updating automerged
cvs update: Updating defaults
cvs update: Updating user
? usr/pkg
Directory /usr/home/vcs/cvsconftrack/defaults/usr added to the repository
? pkg/etc
Directory /usr/home/vcs/cvsconftrack/defaults/usr/pkg added to the repository
? etc/spamd.conf
Directory /usr/home/vcs/cvsconftrack/defaults/usr/pkg/etc added to the repository
cvs add: scheduling file `spamd.conf' for addition
cvs add: use 'cvs commit' to add this file permanently
REGISTER /var/cvsandremote/defaults//usr/pkg/etc/spamd.conf
spamd-20060330nb5: copying /usr/pkg/share/examples/spamd/spamd.conf to /usr/pkg/etc/spamd.conf
RCS file: /usr/home/vcs/cvsconftrack/defaults/usr/pkg/etc/spamd.conf,v
done
Checking in defaults/usr/pkg/etc/spamd.conf;
/usr/home/vcs/cvsconftrack/defaults/usr/pkg/etc/spamd.conf,v  <--  spamd.conf
initial revision: 1.1
done
Conf commit: pkgsrc: add spamd-20060330nb5
===========================================================================
The following files should be created for spamd-20060330nb5:

        /etc/rc.d/pfspamd (m=0755)
            [/usr/pkg/share/examples/rc.d/pfspamd]

===========================================================================
===========================================================================
$NetBSD: MESSAGE,v 1.1.1.1 2005/06/28 12:43:57 peter Exp $

Don't forget to add the spamd ports to /etc/services:

spamd           8025/tcp                # spamd(8)
spamd-cfg       8026/tcp                # spamd(8) configuration

===========================================================================
pkghost# 

On a successive run, the remote repository will get used without files being imported and checked out:

pkghost# cd /root/pkgsrc/pkgtools/pkgin/
pkghost# make install
=> Bootstrap dependency digest>=20010302: found digest-20160304
===> Skipping vulnerability checks.
WARNING: No /var/db/pkg/pkg-vulnerabilities file found.
WARNING: To fix run: `/usr/pkg/sbin/pkg_admin -K /var/db/pkg fetch-pkg-vulnerabilities'.
===> Installing for pkgin-0.9.4nb8
=> Generating pre-install file lists
=> Creating installation directories
[...]
=> Automatic manual page handling
=> Generating post-install file lists
=> Checking file-check results for pkgin-0.9.4nb8
=> Creating binary package /root/pkgsrc/pkgtools/pkgin/work/.packages/pkgin-0.9.4nb8.tgz
===> Building binary package for pkgin-0.9.4nb8
=> Creating binary package /root/pkgsrc/packages/All/pkgin-0.9.4nb8.tgz
===> Installing binary package of pkgin-0.9.4nb8
cvs rlog: Logging defaults
cvs rlog: Logging defaults/usr
cvs rlog: Logging defaults/usr/pkg
cvs rlog: Logging defaults/usr/pkg/etc
cvs update: Updating automerged
cvs update: Updating defaults
cvs update: Updating defaults/usr
cvs update: Updating defaults/usr/pkg
cvs update: Updating defaults/usr/pkg/etc
cvs update: Updating user
cvs [add aborted]: there is a version in usr already
cvs [add aborted]: there is a version in pkg already
cvs [add aborted]: there is a version in etc already
? pkgin/repositories.conf
Directory /usr/home/vcs/cvsconftrack/defaults/usr/pkg/etc/pkgin added to the repository
cvs add: scheduling file `repositories.conf' for addition
cvs add: use 'cvs commit' to add this file permanently
REGISTER /var/cvsandremote/defaults//usr/pkg/etc/pkgin/repositories.conf
pkgin-0.9.4nb8: copying /usr/pkg/share/examples/pkgin/repositories.conf.example to /usr/pkg/etc/pkgin/repositories.conf
RCS file: /usr/home/vcs/cvsconftrack/defaults/usr/pkg/etc/pkgin/repositories.conf,v
done
Checking in defaults/usr/pkg/etc/pkgin/repositories.conf;
/usr/home/vcs/cvsconftrack/defaults/usr/pkg/etc/pkgin/repositories.conf,v  <--  repositories.conf
initial revision: 1.1
done
Conf commit: pkgsrc: add pkgin-0.9.4nb8
===========================================================================
$NetBSD: MESSAGE,v 1.3 2010/06/10 08:05:00 is Exp $

First steps before using pkgin.

. Modify /usr/pkg/etc/pkgin/repositories.conf to suit your platform
. Initialize the database :

        # pkgin update

===========================================================================
pkghost# 

Yes, I know, it's a bit too chatty. Once the script gets used (and tested) some more cvs should be invoked in a more quiet way.

See the next post for a walkthrough in using SVN and Mercurial!

[0 comments]