There are many reasons why you want to compile PostgreSQL yourself. E.g. customized block size, customized set of features, however, the most important one for me is the ability to do multi-homing.
By multi-homing I mean, that you can run as many different PostgreSQL versions as you wish.
e.g.
../pgproduct/pg-13.0
../pgproduct/pg-12.4
../pgproduct/pg-12.3
../pgproduct/pg-11.6
and so on.
The beauty of having multi-homing is that you don’t need to upgrade all your instances at once. You can do it one by one, which in turn makes it easier to organize downtimes for the affected applications.
Lets say, you have 10 PostgreSQL instances running, and each is responsible for a different application, Then you don’t need to upgrade all at once from 12.3 to 12.4. You can do it one by one.
So lets get started.
Before we compile the PostgreSQL software, we have to make sure that a bunch of packages are installed. Not all of them, which are listed below are necessary, but these are the one’s I usually install.
[root@ocm199 /]# dnf install kernel-doc.noarch gcc-c++ iotop nfs-utils libXrender xorg-x11-utils.x86_64 xorg-x11-xauth.x86_64 \ > dos2unix libXScrnSaver kernel-devel.x86_64 fuse-libs.x86_64 m4 nscd binutils \ > glibc.i686 glibc-devel.x86_64 glibc-devel.i686 glibc-devel.x86_64 libaio.i686 libaio.x86_64 \ > libaio-devel.i686 libaio-devel.x86_64 libX11.i686 libX11.x86_64 libXau.i686 libXau.x86_64 libXi.i686 libXi.x86_64 \ > libXtst.i686 libXtst.x86_64 libgcc.i686 libgcc.x86_64 libstdc++.i686 libstdc++.x86_64 libstdc++-devel.i686 \ > libstdc++-devel.x86_64 libxcb.i686 libxcb.x86_64 \ > smartmontools sysstat sg3_utils gcc make bison flex zlib-devel openssh-clients bzip2 \ > net-tools ksh unzip systemd-devel.x86_64 readline-devel.x86_64 openssl-libs.x86_64 openssl-devel.x86_64 \ > pam-devel.x86_64 libxml2-devel.x86_64 libxslt-devel.x86_64 openldap-devel.x86_64 \ > tcl tcl-devel.x86_64 rsync libcurl-devel.x86_64 \ > platform-python-devel.i686 platform-python-devel.x86_64 \ > perl-DBI.x86_64 perl-DBD-Pg.x86_64 perl-Time-HiRes.x86_64 perl-TermReadKey.x86_64 perl-Digest-MD5.x86_64 \ > perl-ExtUtils-Embed.noarch perl-ExtUtils-MakeMaker.noarch \ > python3-dateutil.noarch python3-psycopg2.x86_64 python3-argcomplete.noarch python3-argh.noarch \ > llvm.x86_64 llvm-devel.x86_64 llvm-libs.x86_64 clang Last metadata expiration check: 1:31:36 ago on Mon 26 Oct 2020 07:23:07 AM EDT. Package kernel-doc-4.18.0-193.19.1.el8_2.noarch is already installed. Package gcc-c++-8.3.1-5.el8.0.2.x86_64 is already installed. Package iotop-0.6-16.el8.noarch is already installed. Package nfs-utils-1:2.3.3-31.el8.x86_64 is already installed. Package libXrender-0.9.10-7.el8.x86_64 is already installed. Package xorg-x11-utils-7.5-28.el8.x86_64 is already installed. Package xorg-x11-xauth-1:1.0.9-12.el8.x86_64 is already installed. Package dos2unix-7.4.0-3.el8.x86_64 is already installed. Package libXScrnSaver-1.2.3-1.el8.x86_64 is already installed. Package kernel-devel-4.18.0-147.8.1.el8_1.x86_64 is already installed. Package kernel-devel-4.18.0-193.14.2.el8_2.x86_64 is already installed. Package kernel-devel-4.18.0-193.19.1.el8_2.x86_64 is already installed. Package fuse-libs-2.9.7-12.el8.x86_64 is already installed. Package m4-1.4.18-7.el8.x86_64 is already installed. Package nscd-2.28-101.el8.x86_64 is already installed. Package binutils-2.30-73.el8.x86_64 is already installed. Package glibc-2.28-101.el8.i686 is already installed. Package glibc-devel-2.28-101.el8.x86_64 is already installed. Package glibc-devel-2.28-101.el8.i686 is already installed. Package libaio-0.3.112-1.el8.i686 is already installed. Package libaio-0.3.112-1.el8.x86_64 is already installed. Package libaio-devel-0.3.112-1.el8.i686 is already installed. Package libaio-devel-0.3.112-1.el8.x86_64 is already installed. Package libX11-1.6.8-3.el8.i686 is already installed. Package libX11-1.6.8-3.el8.x86_64 is already installed. Package libXau-1.0.8-13.el8.i686 is already installed. Package libXau-1.0.8-13.el8.x86_64 is already installed. Package libXi-1.7.9-7.el8.i686 is already installed. Package libXi-1.7.9-7.el8.x86_64 is already installed. Package libXtst-1.2.3-7.el8.i686 is already installed. Package libXtst-1.2.3-7.el8.x86_64 is already installed. Package libgcc-8.3.1-5.el8.0.2.i686 is already installed. Package libgcc-8.3.1-5.el8.0.2.x86_64 is already installed. Package libstdc++-8.3.1-5.el8.0.2.i686 is already installed. Package libstdc++-8.3.1-5.el8.0.2.x86_64 is already installed. Package libstdc++-devel-8.3.1-5.el8.0.2.i686 is already installed. Package libstdc++-devel-8.3.1-5.el8.0.2.x86_64 is already installed. Package libxcb-1.13.1-1.el8.i686 is already installed. Package libxcb-1.13.1-1.el8.x86_64 is already installed. Package smartmontools-1:6.6-3.el8.x86_64 is already installed. Package sysstat-11.7.3-2.el8.x86_64 is already installed. Package sg3_utils-1.44-5.el8.x86_64 is already installed. Package gcc-8.3.1-5.el8.0.2.x86_64 is already installed. Package make-1:4.2.1-10.el8.x86_64 is already installed. Package bison-3.0.4-10.el8.x86_64 is already installed. Package flex-2.6.1-9.el8.x86_64 is already installed. Package zlib-devel-1.2.11-16.el8_2.x86_64 is already installed. Package openssh-clients-8.0p1-4.el8_1.x86_64 is already installed. Package bzip2-1.0.6-26.el8.x86_64 is already installed. Package net-tools-2.0-0.51.20160912git.el8.x86_64 is already installed. Package ksh-20120801-254.el8.x86_64 is already installed. Package unzip-6.0-43.el8.x86_64 is already installed. Package systemd-devel-239-31.el8_2.2.x86_64 is already installed. Package readline-devel-7.0-10.el8.x86_64 is already installed. Package openssl-libs-1:1.1.1c-15.el8.x86_64 is already installed. Package openssl-devel-1:1.1.1c-15.el8.x86_64 is already installed. Package pam-devel-1.3.1-8.el8.x86_64 is already installed. Package libxml2-devel-2.9.7-7.el8.x86_64 is already installed. Package libxslt-devel-1.1.32-4.el8.x86_64 is already installed. Package openldap-devel-2.4.46-11.el8_1.x86_64 is already installed. Package tcl-1:8.6.8-2.el8.x86_64 is already installed. Package tcl-devel-1:8.6.8-2.el8.x86_64 is already installed. Package rsync-3.1.3-7.el8.x86_64 is already installed. Package libcurl-devel-7.61.1-12.el8.x86_64 is already installed. Package platform-python-devel-3.6.8-23.el8.i686 is already installed. Package platform-python-devel-3.6.8-23.el8.x86_64 is already installed. Package perl-DBI-1.641-3.module_el8.1.0+199+8f0a6bbd.x86_64 is already installed. Package perl-DBD-Pg-3.7.4-4.module_el8.1.0+268+81255909.x86_64 is already installed. Package perl-Time-HiRes-1.9758-1.el8.x86_64 is already installed. Package perl-TermReadKey-2.37-7.el8.x86_64 is already installed. Package perl-Digest-MD5-2.55-396.el8.x86_64 is already installed. Package perl-ExtUtils-Embed-1.34-416.el8.noarch is already installed. Package perl-ExtUtils-MakeMaker-1:7.34-1.el8.noarch is already installed. Package python3-dateutil-1:2.6.1-6.el8.noarch is already installed. Package python3-psycopg2-2.7.5-7.el8.x86_64 is already installed. Package python3-argcomplete-1.9.3-6.el8.noarch is already installed. Package python3-argh-0.26.1-8.el8.noarch is already installed. Package llvm-9.0.1-5.module_el8.2.0+461+2e15bd5f.x86_64 is already installed. Package llvm-devel-9.0.1-5.module_el8.2.0+461+2e15bd5f.x86_64 is already installed. Package llvm-libs-9.0.1-5.module_el8.2.0+461+2e15bd5f.x86_64 is already installed. Package clang-9.0.1-2.module_el8.2.0+309+0c7b6b03.x86_64 is already installed. Dependencies resolved. Nothing to do. Complete! [root@ocm199 /]#
After the packages are installed, I would recommend to create a logging directory for the 13.0 compilation logs, so that you can check through all of them for any errors or warnings. But again, this is an optional step.
[postgres@ocm199 ~]$ mkdir -p ~/tmp/PG_13.0/ [postgres@ocm199 ~]$
Now we can create the staging directory for PostgreSQL 13.0. This is the place where we put the source code in.
[postgres@ocm199 ~]$ mkdir -p /u01/app/postgres/pgproduct/stage/13/13.0 [postgres@ocm199 ~]$
Download the following files from https://www.postgresql.org/ftp/source/v13.0/
and copy them to the staging directory.
postgresql-13.0.tar.gz
postgresql-13.0.tar.gz.md5
Before we continue, check the md5 checksum of the tar.gz file. They have to be the same. If they are not, stop here! Don’t continue.
[postgres@ocm199 ~]$ cd /u01/app/postgres/pgproduct/stage/13/13.0 [postgres@ocm199 13.0]$ diff <(md5sum postgresql-13.0.tar.gz) <(cat postgresql-13.0.tar.gz.md5) [postgres@ocm199 13.0]$
The next step is to create the new PostgreSQL home destination directory, and to unpack the new PostgreSQL tar.gz file.
[postgres@ocm199 ~]$ mkdir -p /u01/app/postgres/pgproduct/pg-13.0 [postgres@ocm199 ~]$ [postgres@ocm199 ~]$ cd /u01/app/postgres/pgproduct/stage/13/13.0 [postgres@ocm199 13.0]$ tar -xzvf postgresql-13.0.tar.gz ... ... postgresql-13.0/doc/MISSING_FEATURES postgresql-13.0/HISTORY postgresql-13.0/Makefile postgresql-13.0/README postgresql-13.0/COPYRIGHT postgresql-13.0/GNUmakefile.in postgresql-13.0/.gitattributes postgresql-13.0/aclocal.m4 postgresql-13.0/configure.in postgresql-13.0/INSTALL [postgres@ocm199 13.0]$
Finally we can start with the compilation of the new software. There are 3 environment variables which I set before running the ./configure command. PGHOME which is self explainatory, and SEGSIZE + BLOCKSIZE which are explained below.
- SEGSIZE: Set the segment size, in gigabytes. Large tables are divided into multiple operating-system files, each of size equal to the segment size. This avoids problems with file size limits that exist on many platforms. The default segment size, 1 gigabyte, is safe on all supported platforms. If your operating system has “largefile” support (which most do, nowadays), you can use a larger segment size. This can be helpful to reduce the number of file descriptors consumed when working with very large tables. But be careful not to select a value larger than is supported by your platform and the file systems you intend to use. Other tools you might wish to use, such as tar, could also set limits on the usable file size. It is recommended, though not absolutely required, that this value be a power of 2. Note that changing this value requires an initdb
- BLOCKSIZE: Set the block size, in kilobytes. This is the unit of storage and I/O within tables. The default, 8 kilobytes, is suitable for most situations; but other values may be useful in special cases. The value must be a power of 2 between 1 and 32 (kilobytes). Note that changing this value requires an initdb.
[postgres@ocm199 ~]$ cd /u01/app/postgres/pgproduct/stage/13/13.0/postgresql-13.0
[postgres@ocm199 postgresql-13.0]$ export PGHOME=/u01/app/postgres/pgproduct/pg-13.0
[postgres@ocm199 postgresql-13.0]$ export SEGSIZE=4
[postgres@ocm199 postgresql-13.0]$ export BLOCKSIZE=8
[postgres@ocm199 postgresql-13.0]$ export Date=`date +"%Y%m%d%H%M%S"`
[postgres@ocm199 postgresql-13.0]$ ./configure --prefix=${PGHOME} \
> --exec-prefix=${PGHOME} --bindir=${PGHOME}/bin \
> --libdir=${PGHOME}/lib --sysconfdir=${PGHOME}/etc \
> --includedir=${PGHOME}/include \
> --datarootdir=${PGHOME}/share \
> --datadir=${PGHOME}/share \
> --with-pgport=55000 \
> --with-perl --with-python \
> --with-tcl --with-openssl \
> --with-pam --with-ldap \
> --with-libxml --with-libxslt \
> --with-systemd \
> --with-segsize=${SEGSIZE} \
> --with-blocksize=${BLOCKSIZE} \
> --with-llvm LLVM_CONFIG='/usr/bin/llvm-config-64' \
> --with-extra-version=" - Pt(78)DB PostgreSQL 13.0 JIT Build" | tee -a ~/tmp/PG_13.0/PG13.0.with_JIT.configure.log.${Date}
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking which template to use... linux
checking whether NLS is wanted... no
checking for default port number... 55000
checking for block size... 8kB
checking for segment size... 4GB
checking for WAL block size... 8kB
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
...
...
config.status: linking src/backend/port/posix_sema.c to src/backend/port/pg_sema.c
config.status: linking src/backend/port/sysv_shmem.c to src/backend/port/pg_shmem.c
config.status: linking src/include/port/linux.h to src/include/pg_config_os.h
config.status: linking src/makefiles/Makefile.linux to src/Makefile.port
[postgres@ocm199 postgresql-13.0]$
After the configure command finished, check the configure log file ~/tmp/PG_13.0/PG13.0.with_JIT.configure.log.${Date} which we created beforehand for any errors. A more detailed log file is the config.log which should be checked as well.
The config.log file is created in the following directory in case you have any issues /u01/app/postgres/pgproduct/stage/13/13.0/postgresql-13.0
I also recommend to backup the config.log to our ~/tmp/PG_13.0/ directory.
[postgres@ocm199 postgresql-13.0]$ mv config.log ~/tmp/PG_13.0/PG13.0.run_configure.log.${Date}
[postgres@ocm199 postgresql-13.0]$
Now its time to start the make command. The option -j specifies the number of jobs (commands) to run simultaneously.
[postgres@ocm199 postgresql-13.0]$ make -j 4 world | tee -a ~/tmp/PG_13.0/PG13.0.make.world.log.${Date}
make -C ./src/backend generated-headers
make[1]: Entering directory '/u01/app/postgres/pgproduct/stage/13/13.0/postgresql-13.0/src/backend'
make -C catalog distprep generated-header-symlinks
make -C utils distprep generated-header-symlinks
prereqdir=`cd 'parser/' >/dev/null && pwd` && \
cd '../../src/include/parser/' && rm -f gram.h && \
ln -s "$prereqdir/gram.h" .
prereqdir=`cd 'storage/lmgr/' >/dev/null && pwd` && \
cd '../../src/include/storage/' && rm -f lwlocknames.h && \
ln -s "$prereqdir/lwlocknames.h" .
make[2]: Entering directory '/u01/app/postgres/pgproduct/stage/13/13.0/postgresql-13.0/src/backend/utils'
make[2]: Nothing to be done for 'distprep'.
...
...
...
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -O2 -fPIC -shared -o pgcrypto.so openssl.o pgp-mpi-openssl.o crypt-blowfish.o crypt-des.o crypt-gensalt.o crypt-md5.o mbuf.o pgcrypto.o pgp-armor.o pgp-cfb.o pgp-compress.o pgp-decrypt.o pgp-encrypt.o pgp-info.o pgp-mpi.o pgp-pgsql.o pgp-pubdec.o pgp-pubenc.o pgp-pubkey.o pgp-s2k.o pgp.o px-crypt.o px-hmac.o px.o -L../../src/port -L../../src/common -L/usr/lib64 -Wl,--as-needed -Wl,-rpath,'/u01/app/postgres/pgproduct/pg-13.0/lib',--enable-new-dtags -lcrypto -lz
make[2]: Leaving directory '/u01/app/postgres/pgproduct/stage/13/13.0/postgresql-13.0/contrib/pgcrypto'
make[1]: Leaving directory '/u01/app/postgres/pgproduct/stage/13/13.0/postgresql-13.0/contrib'
PostgreSQL, contrib, and documentation successfully made. Ready to install.
[postgres@ocm199 postgresql-13.0]$
It is very important to for the following last sentense. In case you don’t see it, stop here, and check for any errors in the log files.
-> check for “PostgreSQL, contrib, and documentation successfully made. Ready to install.”
Last but not least we run the make install-world command.
[postgres@ocm199 postgresql-13.0]$ make install-world | tee -a ~/tmp/PG_13.0/PG13.0.make.install-world.log.${Date}
make -C ./src/backend generated-headers
make[1]: Entering directory '/u01/app/postgres/pgproduct/stage/13/13.0/postgresql-13.0/src/backend'
make -C catalog distprep generated-header-symlinks
make[2]: Entering directory '/u01/app/postgres/pgproduct/stage/13/13.0/postgresql-13.0/src/backend/catalog'
make[2]: Nothing to be done for 'distprep'.
make[2]: Nothing to be done for 'generated-header-symlinks'.
make[2]: Leaving directory '/u01/app/postgres/pgproduct/stage/13/13.0/postgresql-13.0/src/backend/catalog'
make -C utils distprep generated-header-symlinks
make[2]: Entering directory '/u01/app/postgres/pgproduct/stage/13/13.0/postgresql-13.0/src/backend/utils'
make[2]: Nothing to be done for 'distprep'.
make[2]: Nothing to be done for 'generated-header-symlinks'.
make[2]: Leaving directory '/u01/app/postgres/pgproduct/stage/13/13.0/postgresql-13.0/src/backend/utils'
make[1]: Leaving directory '/u01/app/postgres/pgproduct/stage/13/13.0/postgresql-13.0/src/backend'
make -C doc install
make[1]: Entering directory '/u01/app/postgres/pgproduct/stage/13/13.0/postgresql-13.0/doc'
make -C src install
make[2]: Entering directory '/u01/app/postgres/pgproduct/stage/13/13.0/postgresql-13.0/doc/src'
make -C sgml install
make[3]: Entering directory '/u01/app/postgres/pgproduct/stage/13/13.0/postgresql-13.0/doc/src/sgml'
/usr/bin/mkdir -p '/u01/app/postgres/pgproduct/pg-13.0/share/doc/'/html '/u01/app/postgres/pgproduct/pg-13.0/share/man'/man1 '/u01/app/postgres/pgproduct/pg-13.0/share/man'/man3 '/u01/app/postgres/pgproduct/pg-13.0/share/man'/man7
cp -R `for f in ./html; do test -r $f && echo $f && break; done` '/u01/app/postgres/pgproduct/pg-13.0/share/doc/'
cp -R `for f in ./man1; do test -r $f && echo $f && break; done` `for f in ./man3; do test -r $f && echo $f && break; done` `for f in ./man7; do test -r $f && echo $f && break; done` '/u01/app/postgres/pgproduct/pg-13.0/share/man'
make[3]: Leaving directory '/u01/app/postgres/pgproduct/stage/13/13.0/postgresql-13.0/doc/src/sgml'
make[2]: Leaving directory '/u01/app/postgres/pgproduct/stage/13/13.0/postgresql-13.0/doc/src'
make[1]: Leaving directory '/u01/app/postgres/pgproduct/stage/13/13.0/postgresql-13.0/doc'
make -C src install
make[1]: Entering directory '/u01/app/postgres/pgproduct/stage/13/13.0/postgresql-13.0/src'
make -C common install
...
...
...
/usr/bin/install -c -m 644 ./ltree_plpythonu.control ./ltree_plpython2u.control ./ltree_plpython3u.control '/u01/app/postgres/pgproduct/pg-13.0/share/extension/'
/usr/bin/install -c -m 644 ./ltree_plpythonu--1.0.sql ./ltree_plpython2u--1.0.sql ./ltree_plpython3u--1.0.sql '/u01/app/postgres/pgproduct/pg-13.0/share/extension/'
/usr/bin/mkdir -p '/u01/app/postgres/pgproduct/pg-13.0/lib/bitcode/ltree_plpython3'
/usr/bin/mkdir -p '/u01/app/postgres/pgproduct/pg-13.0/lib/bitcode'/ltree_plpython3/
/usr/bin/install -c -m 644 ltree_plpython.bc '/u01/app/postgres/pgproduct/pg-13.0/lib/bitcode'/ltree_plpython3/./
cd '/u01/app/postgres/pgproduct/pg-13.0/lib/bitcode' && /usr/bin/llvm-lto -thinlto -thinlto-action=thinlink -o ltree_plpython3.index.bc ltree_plpython3/ltree_plpython.bc
make[2]: Leaving directory '/u01/app/postgres/pgproduct/stage/13/13.0/postgresql-13.0/contrib/ltree_plpython'
make[1]: Leaving directory '/u01/app/postgres/pgproduct/stage/13/13.0/postgresql-13.0/contrib'
PostgreSQL, contrib, and documentation installation complete.
[postgres@ocm199 postgresql-13.0]$
Again, the last sentense should be the following. If it is not the case, check your config and make log files.
— check for “PostgreSQL, contrib, and documentation installation complete.”
If everything went fine, you should end up with a pg-13.0 directory at a size of 72MB and a postgres binary of approximately 8MB.
[postgres@ocm199 pgproduct]$ du -hs pg-13.0/ 72M pg-13.0/ [postgres@ocm199 pgproduct]$ [postgres@ocm199 pgproduct]$ cd pg-13.0/bin/ [postgres@ocm199 bin]$ ls -l postgres -rwxr-xr-x. 1 postgres postgres 8326120 Oct 26 08:50 postgres
Now you are ready to go to fire up your first PostgreSQL instance. 🙂
Conclusion
Compiling PostgreSQL yourself is not too complicated, and it gives you a lot of advantages e.g. the ability to do multi-homing.
Cheers
William