Instroduction to Phoronix Test Suite

Feb. 16, 2018, 6 p.m.

by ORDINATECHNIC

Phoronix Test Suite is a Free and Open Source automated benchmarking program developed by Phoronix Media, which also publishes the Phoronix website. Linux enthusiasts most likely have seen the output of the PTS software on Phoronix in their benchmarking results. In this article we provide an overview of the installation of Phoronix Test Suite, the setting up an account on openbenchmark.org, and the process of running a few basic benchmark tests.

Introduction

According to the Phoronix Test Suite website, PTS

is the most comprehensive testing and benchmarking platform available that provides an extensible framework for which new tests can be easily added. The software is designed to effectively carry out both qualitative and quantitative benchmarks in a clean, reproducible, and easy-to-use manner. The Phoronix Test Suite can be used for simply comparing your computer's performance with your friends and colleagues or can be used within your organization for internal quality assurance purposes, hardware validation, and continuous integration / performance management.

Among the strengths of Phoronix Test Suite are its ease of use, the large selection of tests available, the ability to automatically process results and present them graphically. The software also integrates the OpenBenchmarking.org infrastructure, where users can upload results and optionally share these results with other users, access previously uploaded results and the results of other users for comparison of results.

Also available is Phoromatic, a tool "that allows the automatic scheduling of tests, remote installation of new tests, and the management of multiple test systems all through an intuitive, easy-to-use web interface." I do not cover it in this article as its scope is

Installation and Configuration Overview

Download

Phoronix Test Suite is available packaged by distributions. Arch and derivatives have the latest stable version and the latest development version in the AUR; openSUSE Tumbleweed has the latest stable version; others distributions have outdated versions as follows from least outdated to most outdated: Fedora, Ubuntu, Debian.

For those not on Arch based distributions or openSUSE Tumbleweed, PTS is available from the PTS download page as a universal .tar.gz tarball for Unix-like systems which includes an installer script. A .deb package is also available.

The Phoronix Test Suite Download Page
PTS is available from the PTS download page as a universal .tar.gz tarball for Unix-like systems which includes an installer script. But there is no script for uninstalling.

Installation

System-wide installation is not necessary, the .tar.gz can be extracted anywhere and the software can be run from the extracted directory in a terminal. I did this for my initial experimentation with PTS from a data directory on the second -- mechanical -- hard drive on the Acer VN7-592G-70EN laptop. This only gave access to the data partition and other native Linux partitions on the secondary hard drive. It could not access the NTFS partition on the secondary hard drive or the root file system on the primary SSD.

Running PTS Locally from the Extracted Tarball
System-wide installation is not necessary, the .tar.gz can be extracted anywhere and the software can be run from the extracted directory in a terminal.

I suppose I could have extracted the .tar.gz in the root partition and run the program with root privileges. Instead, I later chose to delete the user directory and start again with the provided install.sh script to install on the system.

Running PTS in this mode created a hidden file in my user directory where configuration files, test results, and other resources are stored.

  • The Phoronix Test Suite Extracted Tarball Contents
    Everything PTS needs to run locally is included in this directory.
  • The Phoronix Test Suite User Directory Contents
    When run for the first time, whether locally or from a system-wide installation, a hidden directory -- /home/user/.phoronix-test-suite is created, containing the configuration of PTS, including installed tests and a directory for saving results.
  • The Phoronix Test Suite Root User Directory Contents
    When run with elevated privileges using sudo (in order for tests to be able to write to / PTS creates and accesses /var/lib/phoronix-test-suite/.
Phoronix Test Suite Files
The contents of the tarball and the user directory when PTS is used uninstalled by running from the extracted directory.

Installation to the system from the downloaded .tar.gz can be performed by running the included script with elevated privileges.

  • Running the PTS install.sh script.
    The output of the installer script in the Konsole terminal emulator shows that system-wide installation distributes the files and directories included in the .tar.gz archive in appropriate locations on the root file system. KATE is shown displaying a portion of the installer script.
  • The output of phoronix-test-suite show-system-info
    The first command I ran after running the installer to test successful installation.
System-Wide Installation of Phoronix Test Suite
The provided installer script distributes the files and directories included in the .tar.gz archive in appropriate locations on the root file system.
The system-wide installation also creates the configuration directory at /home/user/.phoronix-test-suite/

Configuration

Configuration of PTS is performed by editing the main configuration file at ~/.phoronix-test-suite/user-config.xml. Among the configuration options that are available through this file are some OpenBenchmarking.org integration items, the default browser used to view results, whether system logs should be saved, and whether test logs should be saved. The options I changed were those that would allow me to easily share downloaded tests and saved test results among the different GNU/Linux installations on this computer. I modified the location of the download cache -- which can be shared among different PTS clients in various ways, including on external USB media -- and the location of saved test results, to a directory on another partition accessible to all GNU/Linux installations on this computer.

Other options useful to those who might use PTS to publish benchmarking results are at ~/.phoronix-test-suite/graph-config.json.

Phoronix Test Suite Configuration
The browser window shows a page of the the full Phoronix Test Suite Documentation as a PDF document listing file locations. The first two items in the list are the files users can use to configure the basic operation of PTS.

Unfortunately the documentation doesn't mention that configuration of PTS, when using it with elevated privileges doesn't access files in ~/.phoronix-test-suite but the analogue for the root user /var/lib/phoronix-test-suite. Other resources for running PTS as root are also accessed from this location and results are stored to this location as well and not ~/.phoronix-test-suite.

In addition to the configuration files above a multitude of environment variables are available for more advanced configuration of PTS. For the purposes of this introductory article, I did not pursue these options.

Initial Tests

One of the first tests I ran was executed with

phoronix-test-suite benchmark stress-ng.
The above illustrates the command structure of PTS and the operation for performing basic tests. Commands are in the form of
phoronix-test-suite command [object-of-command] [object-of-command] ....
In this case stress-ng is a test profile. If a particular test profile is not installed -- none were installed by default, when using the benchmark action, PTS downloads it from the OpenBenchmarking.org repositories and installs it before using it to perform the tests.
  • Dolphin Showing Stored Test Results and Running a Test in Konsole
    Directories are created in the configured location for storing test results. When running a test PTS prompts users whether to save results, the name for saving the test results, and other identifiers.
  • First Test Result
    PTS offers to open the results in a browser window.
  • Uploading Results to OpenBenchmarking.org, Merging Results, and Uploading Merged Results
  • Merged Results Opened In Browser Window
The First Tests
Running a test, viewing results, and merging results, and uploading to OpenBenchmarking.org.

Of the few tests I've performed, each has numerous options that the user is prompted to specify. If all options are selected tests could take many hours. I ran the Flexible IO Tester profile to determine the impact of Plasma's Baloo using:

phoronix-test-suite benchmark fio
and chose options to minimize the time to completion of the test. I ran this test twice, once with Baloo running, and once after killing Baloo's processes and saving the results to the same name so that the two tests would be automatically merged. It should be noted that the results of these and other tests mentioned in this article are spurious, not because of anything to do with Phoronix Test Suite, but because I performed the tests while using the computer during the tests, as these testing activities were only to get accustomed to using PTS.
  • Running the Flexible IO Tester
    When running the benchmark action the user is prompted to enter the desired options. In the Konsole window, the first prompt for an option value is shown.
  • Running the Flexible IO Tester
    The Flexible IO test in progress in the right view of a split Konsole window. If after saving a test, subsequent tests of the same test profile with the same name under which to saver results are performed, comparison results PTS displays comparison results after each run.
  • The Flexible IO Tester Results Opened in a Browser
    Colors used in the results graphs can be changed by modifying the graph-config.jsonfile.
Running the Flexible IO Tester and Viewing the Results

PTS Provides a Tabular Summary of the Test Results
In the presentation of multiple tests saved to the same result name the "winner" is highlighted. The spurious nature of the results is evident from this summary.
PTS Provides Results of Individual Tests Graphically
Results of tests using the same test profile and the same name under which to save results are merged and displayed together in results graphs.

Documentation

The documentation provided by Phoronix Test Suite is essential in order to use the software effectively. I'm sure my cursory perusal is not enough to get the most out of PTS. The PDF version of the user guide is available at http://phoronix-test-suite.com/documentation/phoronix-test-suite.pdf. An HTML version is also included in the tarball and copied to /usr/share/doc/phoronix-test-suite/. Some older documents that cover more advanced topics, such as writing custom modules, are also copied to this location.

Problems

The most problematic part of my experience with Phoronix Test Suite has been the OpenBenchmarking.org component. Although not necessary to use PTS, OpenBenchmarking.org is useful for storing test result data, sharing test profiles and test suites, and sharing results. The problems involved registering for an account at OpenBenchmarking.org and logging in to the account. First, although the site supports encrypted connections, it is not required. So if you type "openbenchmarking.org" into a browser, as I did at first, it provides an insecure connection. If you then click "Register," it again connects insecurely instead of converting the connection for the registration. Most sites do this automatically. However, I later learned that if you specifically request the secure site, by including the "https://" clicking the "register" link will open a secure registration page.

Second, in order to successfully log in top the site, the "Auto Log-In / Remember Me" check-box under the username and password entry fields must be activated. Otherwise, it simply opens the main page without logging the user into the account.

More important, although not necessary, as Phoronix Media provides enough, was that I missed resources for inexperienced users to better select test profiles that they might want to use. The only way for a user to select a test is by the name of a test then reading the brief introduction to the test on its page. For example the stress-ng page describes the test as "Stress-NG is a Linux stress tool developed by Colin King of Canonical."

The OpenBenchmarking.org Workstation Test Suite Page
This page shows the basic command to run this test and the tests included in this test suite, but not much else. But clicking the links for the individual tests does open a page that shows the scripts and configuration files that make the test.
There is a command however,
phoronix-test-suite list-recommended-tests
that will list recommended tests (See how easy it is to use PTS?) organized by category. Still there are many choices on this list, and it may be difficult to choose the most appropriate test considering the tests themselves may take a long time to complete.

To actually know which tests to use may require looking at the page for each of the listed tests and clicking the link of each individual test, if it is part of a suite, or clicking the link of the latest version of the test, if it is not part of a suite and going through the displayed components that actually comprise the test.
The OpenBenchmarking.org specviewperf Test Profile Page
This page shows the files that control the download and installation of the test profile, as well as the files that define the test and the results collection process.
When clicking one of these links, the opened page will provide listings of the files that control the download and installation of the test profile, as well as the files that define the test and the results collection process. The motivated user can then research the test using the information contained in these files.

One such piece of information is a link to the test profile developer's site. For example, the specviewperf test profile's downloads.xml file indicates a domain of spec.org and the stress-ng test profile's test-definition.xml file indicates the project's URL as http://kernel.ubuntu.com/~cking/stress-ng/. Following this link provides very detailed information on the test and some caveats.

Conclusion

I find Phoronix Test Suite to be extremely beneficial to GNU/Linux distribution review publishing sites, such as this one, to any GNU/Linux enthusiast who wants to understand system performance, and of course to enterprise users. Making this software freely available to anyone interested in benchmarking and better understanding their systems is a great service. I also believe PTS provides what Phoronix Media promises in its description of the software, quoted at the top of this article.

Notes:

  1. [1]

blog comments powered by Disqus