Introduction to Phoronix Test Suite

Feb. 18, 2018, 6 p.m.

Last Updated: July 28, 2021, 9 p.m.

Phoronix Test Suite is a Free and Open Source automated benchmarking program developed by Phoronix Media, which also publishes the Phoronix website. Linux enthusiasts most likely have seen the output of the PTS software on Phoronix in their benchmarking results. In this article, we provide an overview of the installation of Phoronix Test Suite, the configuration of an account on openbenchmark.org -- where results can be shared, and the process of running a few basic benchmark tests.

Introduction

According to the Phoronix Test Suite website, PTS

is the most comprehensive testing and benchmarking platform available that provides an extensible framework for which new tests can be easily added. The software is designed to effectively carry out both qualitative and quantitative benchmarks in a clean, reproducible, and easy-to-use manner. The Phoronix Test Suite can be used for simply comparing your computer's performance with your friends and colleagues or can be used within your organization for internal quality assurance purposes, hardware validation, and continuous integration / performance management.

Among the strengths of Phoronix Test Suite are its ease of use, the large selection of tests available, the ability to automatically process results and, present them graphically. The software also integrates the OpenBenchmarking.org infrastructure, where users can review information on available tests and test suites, upload results to share these results with other users, and access previously uploaded results and the results of other users for comparison of results.

Also available is Phoromatic, a tool "that allows the automatic scheduling of tests, remote installation of new tests, and the management of multiple test systems all through an intuitive, easy-to-use web interface." I do not cover it here as it is beyonf the scope of this article.

Installation and Configuration Overview

Download

Phoronix Test Suite as a native package in distributions' repositories to be installed using their system package managers. Arch and derivatives have the latest stable version and the latest development version in the AUR; openSUSE Tumbleweed has the latest stable version; others distributions have outdated versions as follows from least outdated to most outdated: Fedora, Ubuntu, Debian.

For those not on Arch based distributions or openSUSE Tumbleweed, PTS is available from the PTS download page as a universal .tar.gz tarball for Unix-like systems which includes an installer script. A .deb package is also available.

The Phoronix Test Suite Download Page
PTS is available from the PTS download page as a universal .tar.gz tarball for Unix-like systems which includes an installer script. But there is no script for uninstalling.

Installation

System-wide installation is not necessary; the .tar.gz compressed archive can be extracted anywhere and the software can be run from the extracted directory in a terminal. I did this for my initial experimentation with PTS from a data directory on the second -- mechanical -- hard drive on the Acer VN7-592G-70EN laptop. This only gave access to the data partition and other native Linux partitions on the secondary hard drive. It could not access the NTFS partition on the secondary hard drive or the root file system on the primary SSD.

Running PTS Locally from the Extracted Tarball
System-wide installation is not necessary, the .tar.gz can be extracted anywhere and the software can be run from the extracted directory in a terminal.

Running PTS in this mode created a hidden file in my user directory where configuration files, test results, and other resources are stored.

I suppose I could have extracted the .tar.gz in the root partition and run the program with root privileges. Instead, I later chose to delete the user directory and start again with the provided install-sh script to install PTS system-wide.

  • The Contents of the Extracted Compressed Archive
  • The Contents of the Per-User Resources Directory
  • The Contents of the System-Wide Resources Directory (right pane of Dolphin) and Some Resource Subdirectories Reconfigured to a Different Location (left pane of Dolphin)
Phoronix Test Suite Directories
Everything PTS needs to run locally without installation is included in the extracted compressed archive directory. When PTS is run for the first time, whether locally or from a system-wide installation, a hidden directory -- /home/username/.phoronix-test-suite is created, containing the configuration of PTS, including installed tests and a directory for saving results. When run with elevated privileges using sudo (in order for tests to be able to write to / PTS creates and accesses /var/lib/phoronix-test-suite/.

Installation to the system from the downloaded .tar.gz can be performed by running the included script -- install-sh -- with elevated privileges.

  • Running the PTS install-sh Script.
  • The Output of phoronix-test-suite show-system-info
System-Wide Installation of Phoronix Test Suite
The output of the installer script in the Konsole terminal emulator shows that system-wide installation distributes the files and directories included in the .tar.gz archive in appropriate locations on the root file system. KATE is shown displaying a portion of the installer script.

After running the installer script phoronix-test-suite show-system-info can be used to test successful installation.

When PTS is run for the first time, whether locally or from a system-wide installation, a hidden directory -- /home/username/.phoronix-test-suite is created, containing the configuration of PTS, including installed tests and a directory for saving results. When run with elevated privileges using sudo (in order for tests to be able to write to / PTS creates and accesses /var/lib/phoronix-test-suite/.

Configuration

Configuration of PTS for per-user execution is performed by editing the main configuration file at ~/.phoronix-test-suite/user-config.xml. Among the configuration options that are available through this file are some OpenBenchmarking.org integration items, the default browser used to view results, whether system logs should be saved, and whether test logs should be saved.

Other options useful to those who might use PTS to publish benchmarking results are at ~/.phoronix-test-suite/graph-config.json.

Phoronix Test Suite Configuration
The browser window shows a page of the the full Phoronix Test Suite Documentation as a PDF document listing file locations. The first two items in the list are the files users can use to configure the basic operation of PTS.

Unfortunately the Phoronix Test Suite documentation does not mention that configuration of PTS, when using it with elevated privileges doesn't access files in ~/.phoronix-test-suite but the analogue for the root user /var/lib/phoronix-test-suite. Complicating things is that the configuration for privileged execution of PTS is not in the analogous resource directory, but in the typical location for system-wide configuration, /etc at /etc/phoronix-test-suite.xml. (One might expect a similarly named configuration file to /var/lib/phoronix-test-suite/phoronix-test-suite.xml). The same configuration items mentioned above for per-user configuration in ~/.phoronix-test-suite/user-config.xml can be specified in /etc/phoronix-test-suite.xml for execution with elevated permissions.

As mentioned above, resources for running PTS as root are located in /var/lib/phoronix-test-suite/phoronix-test-suite.xml, such as subdirectories for installed tests and subdirectory for storing test results. To change some of these directories, the configuration file /etc/phoronix-test-suite.xml can be modified. The options I changed were those that would allow me to easily share downloaded tests and saved test results among the different GNU/Linux installations on this computer. I modified the location of the download cache -- which can be shared among different PTS clients in various ways, including on external USB media -- and the location of saved test results, to a directory on another partition accessible to all GNU/Linux installations on this computer.

In addition to the configuration files above a multitude of environment variables are available for more advanced configuration of PTS. For the purposes of this introductory article, I did not pursue these options.

Initial Tests

One of the first tests I ran was executed with

phoronix-test-suite benchmark stress-ng

The above illustrates the command structure of PTS and the operation for performing basic tests. Commands are in the form of

phoronix-test-suite command [object-of-command] [object-of-command] ...

In this case stress-ng is a test profile, or simply a "test". If a particular test profile is not installed -- none were installed by default -- when using the benchmark command, PTS downloads it from the OpenBenchmarking.org repositories and installs it before using it to perform the test(s).

  • Dolphin Showing Stored Test Results and Running a Test in Konsole
    Directories are created in the configured location for storing test results. When running a test PTS prompts users whether to save results, the name for saving the test results, and other identifiers.
  • First Test Result
    PTS offers to open the results in a browser window.
  • Uploading Results to OpenBenchmarking.org, Merging Results, and Uploading Merged Results
  • Merged Results Opened In Browser Window
The First Tests
Running a test, viewing results, and merging results, and uploading to OpenBenchmarking.org.

Tests can have numerous options that the user is prompted to specify, in basic executions of PTS. In more advanced use, options can be specified before hand in configuration such that the user is not prompeted. If all options are selected, tests could take many hours.

To get aclimated to PTS, I ran the Flexible IO Tester profile in order to determine the impact of Plasma's Baloo using:

phoronix-test-suite benchmark fio

then when prompted, choosing options to minimize the time to completion of the test. I ran this test twice, once with Baloo running, and once after killing Baloo's processes and saving the results to the same name so that the two tests would be automatically merged. It should be noted that the results of these and other tests mentioned in this article are spurious, not because of anything to do with Phoronix Test Suite, but because I performed the tests while using the computer during the tests, as these testing activities were only to get accustomed to using PTS.

  • Running the Flexible IO Tester
    When running the benchmark action the user is prompted to enter the desired options. In the Konsole window, the first prompt for an option value is shown.
  • Running the Flexible IO Tester
    The Flexible IO test in progress in the right view of a split Konsole window. If after saving a test, subsequent tests of the same test profile with the same name under which to saver results are performed, comparison results PTS displays comparison results after each run.
  • The Flexible IO Tester Results Opened in a Browser
    Colors used in the results graphs can be changed by modifying the graph-config.json file.
Running the Flexible IO Tester and Viewing the Results

PTS Provides a System Overview with the Test Results
The hardware components, kernel version, compiler options, IO scheduler, and other relevant information regarding the system are provided.

PTS Provides a Tabular Summary of the Test Results
In the presentation of multiple tests saved to the same result name the "winner" is highlighted. The spurious nature of the results is evident from this summary.
PTS Provides Results of Individual Tests Graphically
Results of tests using the same test profile and the same name under which to save results are merged and displayed together in results graphs.

Documentation

The documentation provided by Phoronix Test Suite is essential in order to use the software effectively. I'm sure my cursory perusal is not enough to get the most out of PTS. The PDF version of the user guide is available at http://phoronix-test-suite.com/documentation/phoronix-test-suite.pdf. An HTML version is also included in the tarball and copied to /usr/share/doc/phoronix-test-suite/. Some older documents that cover more advanced topics, such as writing custom modules, are also copied to this location.

Problems

The most problematic part of my experience with Phoronix Test Suite has been the OpenBenchmarking.org component. Although not necessary to use PTS, OpenBenchmarking.org is useful for storing test result data, sharing test profiles and test suites, and sharing results. The problems involved registering for an account at OpenBenchmarking.org and logging in to the account. First, although the site supports encrypted connections, it is not required. So if you type "openbenchmarking.org" into a browser, as I did at first, it provides an insecure connection. If you then click "Register," it again connects insecurely instead of converting the connection for the registration. Most sites do this automatically. However, I later learned that if you specifically request the secure site, by including the "https://" clicking the "register" link will open a secure registration page.

Second, in order to successfully log in top the site, the "Auto Log-In / Remember Me" check-box under the username and password entry fields must be activated. Otherwise, it simply opens the main page without logging the user into the account.

More important, although not necessary, as Phoronix Media provides enough, was that I missed resources for inexperienced users to better select test profiles that they might want to use. The only way for a user to select a test is by the name of a test then reading the brief introduction to the test on its page. For example the stress-ng page describes the test as "Stress-NG is a Linux stress tool developed by Colin King of Canonical."

The OpenBenchmarking.org Workstation Test Suite Page
This page shows the basic command to run this test and the tests included in this test suite, but not much else. But clicking the links for the individual tests does open a page that shows the scripts and configuration files that make the test.

There is a command however,

phoronix-test-suite list-recommended-tests

that will list recommended tests organized by category. Still, there are many choices on this list, and it may be difficult to choose the most appropriate test considering the tests themselves may take a long time to complete.

To actually know which tests to use may require looking at the page for each of the listed tests and clicking the link of each individual test, if it is part of a suite, or clicking the link of the latest version of the test, if it is not part of a suite and going through the displayed components that actually comprise the test.

The OpenBenchmarking.org specviewperf Test Profile Page
This page shows the files that control the download and installation of the test profile, as well as the files that define the test and the results collection process.

When clicking one of these links, the opened page will provide listings of the files that control the download and installation of the test profile, as well as the files that define the test and the results collection process. The motivated user can then research the test using the information contained in these files.

One such piece of information is a link to the test profile developer's site. For example, the specviewperf test profile's downloads.xml file indicates a domain of spec.org and the stress-ng test profile's test-definition.xml file indicates the project's URL as http://kernel.ubuntu.com/~cking/stress-ng/. Following this link provides very detailed information on the test and some caveats.

Conclusion

Phoronix Test Suite to be extremely beneficial to GNU/Linux distribution review publishing sites, such as this one, to any GNU/Linux enthusiast who wants to understand system performance, and of course to its intended target, enterprise customers. Making this software freely available to anyone interested in benchmarking and better understanding their systems is a great service. I also believe PTS provides what Phoronix Media promises in its description of the software.