Phoronix Test Suite is a Free and Open Source automated benchmarking program developed by Phoronix Media, which also publishes the Phoronix website. Linux enthusiasts most likely have seen the output of the PTS software on Phoronix in their benchmarking results. In this article we provide an overview of the installation of Phoronix Test Suite, the setting up an account on openbenchmark.org, and the process of running a few basic benchmark tests.
According to the Phoronix Test Suite website, PTS
is the most comprehensive testing and benchmarking platform available that provides an extensible framework for which new tests can be easily added. The software is designed to effectively carry out both qualitative and quantitative benchmarks in a clean, reproducible, and easy-to-use manner. The Phoronix Test Suite can be used for simply comparing your computer's performance with your friends and colleagues or can be used within your organization for internal quality assurance purposes, hardware validation, and continuous integration / performance management.
Among the strengths of Phoronix Test Suite are its ease of use, the large selection of tests available, the ability to automatically process results and present them graphically. The software also integrates the OpenBenchmarking.org infrastructure, where users can upload results and optionally share these results with other users, access previously uploaded results and the results of other users for comparison of results.
Also available is Phoromatic, a tool "that allows the automatic scheduling of tests, remote installation of new tests, and the management of multiple test systems all through an intuitive, easy-to-use web interface." I do not cover it in this article as its scope is
Installation and Configuration Overview
Phoronix Test Suite is available packaged by distributions. Arch and derivatives have the latest stable version and the latest development version in the AUR; openSUSE Tumbleweed has the latest stable version; others distributions have outdated versions as follows from least outdated to most outdated: Fedora, Ubuntu, Debian.
For those not on Arch based distributions or openSUSE Tumbleweed, PTS is available from the PTS download page as a universal .tar.gz tarball for Unix-like systems which includes an installer script. A .deb package is also available.
System-wide installation is not necessary, the .tar.gz can be extracted anywhere and the software can be run from the extracted directory in a terminal. I did this for my initial experimentation with PTS from a data directory on the second -- mechanical -- hard drive on the Acer VN7-592G-70EN laptop. This only gave access to the data partition and other native Linux partitions on the secondary hard drive. It could not access the NTFS partition on the secondary hard drive or the root file system on the primary SSD.
I suppose I could have extracted the .tar.gz in the root partition and run the program with root privileges. Instead, I later chose to delete the user directory and start again with the provided install.sh script to install on the system.
Running PTS in this mode created a hidden file in my user directory where configuration files, test results, and other resources are stored.
Installation to the system from the downloaded .tar.gz can be performed by running the included script with elevated privileges.
The system-wide installation also creates the configuration directory at /home/user/.phoronix-test-suite/
Configuration of PTS is performed by editing the main configuration file at ~/.phoronix-test-suite/user-config.xml. Among the configuration options that are available through this file are some OpenBenchmarking.org integration items, the default browser used to view results, whether system logs should be saved, and whether test logs should be saved. The options I changed were those that would allow me to easily share downloaded tests and saved test results among the different GNU/Linux installations on this computer. I modified the location of the download cache -- which can be shared among different PTS clients in various ways, including on external USB media -- and the location of saved test results, to a directory on another partition accessible to all GNU/Linux installations on this computer.
Other options useful to those who might use PTS to publish benchmarking results are at ~/.phoronix-test-suite/graph-config.json.
Unfortunately the documentation doesn't mention that configuration of PTS, when using it with elevated privileges doesn't access files in ~/.phoronix-test-suite but the analogue for the root user /var/lib/phoronix-test-suite. Other resources for running PTS as root are also accessed from this location and results are stored to this location as well and not ~/.phoronix-test-suite.
In addition to the configuration files above a multitude of environment variables are available for more advanced configuration of PTS. For the purposes of this introductory article, I did not pursue these options.
One of the first tests I ran was executed with
phoronix-test-suite benchmark stress-ng.
The above illustrates the command structure of PTS and the operation for performing basic tests.
Commands are in the form of
phoronix-test-suite command [object-of-command] [object-of-command] ....
In this case stress-ng is a test profile. If a particular test profile is not installed -- none were installed by default, when using the benchmark action, PTS downloads it from the OpenBenchmarking.org repositories and installs it before using it to perform the tests.
Of the few tests I've performed, each has numerous options that the user is prompted to specify. If all options are selected tests could take many hours. I ran the Flexible IO Tester profile to determine the impact of Plasma's Baloo using:
phoronix-test-suite benchmark fio
and chose options to minimize the time to completion of the test. I ran this test twice, once with Baloo running, and once after killing Baloo's processes and saving the results to the same name so that the two tests would be automatically merged. It should be noted that the results of these and other tests mentioned in this article are spurious, not because of anything to do with Phoronix Test Suite, but because I performed the tests while using the computer during the tests, as these testing activities were only to get accustomed to using PTS.
The documentation provided by Phoronix Test Suite is essential in order to use the software effectively. I'm sure my cursory perusal is not enough to get the most out of PTS. The PDF version of the user guide is available at http://phoronix-test-suite.com/documentation/phoronix-test-suite.pdf. An HTML version is also included in the tarball and copied to /usr/share/doc/phoronix-test-suite/. Some older documents that cover more advanced topics, such as writing custom modules, are also copied to this location.
The most problematic part of my experience with Phoronix Test Suite has been the OpenBenchmarking.org component. Although not necessary to use PTS, OpenBenchmarking.org is useful for storing test result data, sharing test profiles and test suites, and sharing results. The problems involved registering for an account at OpenBenchmarking.org and logging in to the account. First, although the site supports encrypted connections, it is not required. So if you type "openbenchmarking.org" into a browser, as I did at first, it provides an insecure connection. If you then click "Register," it again connects insecurely instead of converting the connection for the registration. Most sites do this automatically. However, I later learned that if you specifically request the secure site, by including the "https://" clicking the "register" link will open a secure registration page.
Second, in order to successfully log in top the site, the "Auto Log-In / Remember Me" check-box under the username and password entry fields must be activated. Otherwise, it simply opens the main page without logging the user into the account.
More important, although not necessary, as Phoronix Media provides enough, was that I missed resources for inexperienced users to better select test profiles that they might want to use. The only way for a user to select a test is by the name of a test then reading the brief introduction to the test on its page. For example the stress-ng page describes the test as "Stress-NG is a Linux stress tool developed by Colin King of Canonical."
There is a command however,
that will list recommended tests (See how easy it is to use PTS?) organized by category. Still there are many choices on this list, and it may be difficult to choose the most appropriate test considering the tests themselves may take a long time to complete.
To actually know which tests to use may require looking at the page for each of the listed tests and clicking the link of each individual test, if it is part of a suite, or clicking the link of the latest version of the test, if it is not part of a suite and going through the displayed components that actually comprise the test.
When clicking one of these links, the opened page will provide listings of the files that control the download and installation of the test profile, as well as the files that define the test and the results collection process. The motivated user can then research the test using the information contained in these files.
One such piece of information is a link to the test profile developer's site. For example, the specviewperf test profile's downloads.xml file indicates a domain of spec.org and the stress-ng test profile's test-definition.xml file indicates the project's URL as http://kernel.ubuntu.com/~cking/stress-ng/. Following this link provides very detailed information on the test and some caveats.
I find Phoronix Test Suite to be extremely beneficial to GNU/Linux distribution review publishing sites, such as this one, to any GNU/Linux enthusiast who wants to understand system performance, and of course to enterprise users. Making this software freely available to anyone interested in benchmarking and better understanding their systems is a great service. I also believe PTS provides what Phoronix Media promises in its description of the software, quoted at the top of this article.