PyNVMe3 User Guide
Last Modified: August 25, 2023
Copyright © 2021-2023 GENG YUN Technology Pte. Ltd.
All Rights Reserved.
- Change Log
- PAM Setup
|0.9||cranechu||2021-02-06||updated for PyNVMe3 R21.2|
|1.0||cranechu||2021-02-25||added an email for tech support|
|1.1||cranechu||2021-05-25||changed recommended platform|
|keep reports in folder results|
|1GB hugepage setup|
|updated for PyNVMe3 R21.5|
|1.2||cranechu||2021-08-03||add benchmark test suite|
|Quarch PAM setup|
|1.3||cranechu||2021-11-25||PAM is required by TCG and benchmark tests|
|1.4||cranechu||2022-05-26||support NVMe v2.0|
|updated for PyNVMe3 R22.5|
|1.5||cranechu||2022-11-28||1GB-hugepage is a required configuration|
|updated for PyNVMe3 R22.11|
|2.0||cranechu||2022-12-28||re-distribute as an online document|
|2.1||cranechu||2023-08-25||PyNVMe3 R23.8 supports Ubuntu 20.04 and 22.04|
PyNVMe3 is a test collection for NVMe SSD implemented in Python3 scripts, powered by an in-house NVMe test-dedicated driver. The driver takes full control of the Device Under Test (DUT), from registers to memory, from data buffer to commands. It provides comprehensive and powerful functions in easy-to-use Python API. PyNVMe3 builds a professional and well maintained collection of NVMe test scripts based on this driver.
|Image 1 PyNVMe3 Block Diagram|
PyNVMe3 refers to the NVMe Specification v1.4 and v2.0, but it is not limited by the NVMe Specification.
The platform running PyNVMe3 has to meet these minimum requirements:
- CPU: Intel® Core™ i5
- Memory: 8GB
- OS: Ubuntu 20.04.x and 22.04.x
- Secure boot: disabled
- Storage: enable AHCI/NVMe mode
|Image 2 BIOS Setup|
We recommend Dell Precision 3560 Mobile Workstation as the test system of PyNVMe3. It has dual M.2 slots. SSD-1 slot is used for a SATA SSD where OS installed, and SSD-2 (PCIe Gen4) slot is for NVMe DUT.
|Image 3 Recommended System|
In Ubuntu, open a Terminal to install PyNVMe3 in command lines. Internet access is required when installing PyNVMe3.
- install pip3
sudo apt install -y python3-pip
- uninstall PyNVMe3 if it has been installed before
sudo pip3 uninstall PyNVMe3
sudo rm -rf /usr/local/PyNVMe3
- install PyNVMe3 package with pip3
sudo pip3 install PyNVMe3-23.8.tar.gz
Now, PyNVMe3 is installed under folder /usr/local/PyNVMe3.
1GB-hugepage is required when scripts allocate huge buffers (e.g. HMB). Furthermore, we need to disable IOMMU on AMD platforms. We can make the configurations in following steps:
- modify the line of GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash default_hugepagesz=2M hugepagesz=1G hugepages=1 amd_iommu=off"
- sudo update-grub
- add one line in file /etc/fstab
none /mnt/huge hugetlbfs pagesize=1G,size=1G 0 0
- sudo reboot
Quarch Power Analysis Module (PAM) is a 3-rd party power module used to control PCIe power (on/off) and monitor device power consumption. It is required by TCG tests, as well as power-related benchmark tests in PyNVMe3. PAM consists of 2 units: a control unit in a white box, and the m.2 fixture in 2280 form factor.
|Image 4 Front-view of the control unit|
|Image 5 Back-view of the control unit|
|Image 6 M.2 Fixture|
There 2 units are connected to the System Under Test (SUT) in the following steps:
- insert the m.2 fixture to the SUT, for example, SSD-2 (PCIe Gen4) slot of the recommended system;
- plug the DUT into the m.2 fixture;
- connect the control unit and the m.2 fixture with the (Quarch provided) type-C to type-C USB cable;
Image 7 Connection among SUT, DUT, fixture, and control unit
- connect the control unit to the SUT with the (Quarch provided) type-B to type-A USB cable;
- connect (Quarch provided) power adaptor to the control unit, and power on;
- power on the SUT.
Image 8 Connection between SUT and controller unit
In step 4, as an alternative, when the scripts neither require PAM to control the power on/off, nor collect power data by the scripts themselves, we can connect the control unit to another computer where Quarch Power Studio (QPS) collects and presents the power data.
In Ubuntu, open a Terminal to run PyNVMe3 in command lines. Internet access is not required to run PyNVMe3 tests.
- change directory to PyNVMe3 folder.
- switch to root user.
- run the conformance tests with NVMe DUT.
- or, run the specified tests with NVMe DUT in following ways.
make test TESTS=scripts/test_folder
make test TESTS=scripts/test_folder/test_file.py
make test TESTS=scripts/test_folder/test_file.py::test_function
For more information, please refer to pytest documents.
- or when multiple NVMe DUTs are attached in the SUT, run the tests with a specified NVMe DUT by its BDF address.
make test pciaddr=BB:DD.F
NOTICE: The test may remove or overwrite the data in your NVMe DUT. Before running any PyNVMe3 test, please backup your data and delete all partitions in your DUT.
When PyNVMe3 test is running, the test-dedicated driver takes over control of NVMe DUT from Linux Kernel’s inbox driver. So, the Kernel, as well as the Kernel-based utilities (e.g. fio, nvme-cli), cannot detect and manipulate the NVMe DUT during PyNVMe3 tests.
PyNVMe3 provides conformance test suite to test DUT against specifications. Quarch PAM is required by some test scripts (e.g. TCG tests). If PAM is not available, these tests are skipped.
PyNVMe3 provides benchmark test suite to test DUT in real workloads. We can compare test results of different DUTs. Quarch PAM is required by some of the benchmark tests (e.g. power_cycle.py). Users can run benchmark test as below:
make test TESTS=./scripts/benchmark/power_cycle.py
After the test is completed, we can find the test report (.xlsx) and log (.log) files in folder PyNVMe3/results. Benchmark tests could generate more files and pictures in this folder.
|Image 9 Test result files|
There are some possible test results of each test item:
- SKIPPED: test skipped.
- FAILED: test failed.
- PASSED: test passed, but may have some warnings.
For more information on the test log and report, please refer to pytest.
When the test is terminated abnormally, users can find more debug information from the log files in the PyNVMe3/results folder. Users can also add a prefix ‘_’ to the function name of the FAILED tests to bypass them temporarily. Please provide us the test log file when we need to investigate issues together.