1 of 8

DTS RPC Proposal

Owen Hilyard

Some of you may be wondering what motivated me to propose such a big change to DTS. The primary reason I want to change this is because right now the new developer experience is not great. When you want to cause DPDK to do something in DTS, you are forced to use string formatting instead of passing parameters. You need to know the commands built into testpmd, as well as all of the possible forms the output from that command could take. Adding a new command requires digging through testpmd, which is almost evenly split between parsing or lexing code and code that interacts with DPDK. I don’t think that interacting with this provides a good developer experience for someone adding features to DPDK and then testing them in DTS or someone who is just writing DTS test suites for existing features. The amount of work to expose new functionality in testpmd seems disproportionate, and I think that using RPC would change that. It provides a more machine-oriented interface automated testing, while still allowing human use, as I’ll discuss later.

2 of 8

Current Approach

Currently, DTS uses Testpmd to conduct all testing of DPDK. Testpmd is, per it’s documentation, an example application that is designed to showcase a more complex project using DPDK. It provides a command-line REPL and exposes most of DPDK. This sequence diagram showcases what the process for a DTS test case to call a function in DPDK on the Device Under Test (DUT). It starts by creating a string that contains the testpmd command, usually via string formatting for any non-trivial command, and passing that to the testpmd REPL via a pre-established ssh connection. Testpmd then runs the command through it’s parser. After the command is parsed, it is passed a wrapper function inside of testpmd which handles actually running the command. This function will convert integer identifiers into the handles needed by DPDK function calls, and validate that the state of testpmd is correct for running this command. Then, it will call into DPDK and receive the return value of that function call. Testpmd may then either print nothing, print some “Ok” message, print out a representation of some object, or print an error message. This output is then brought back to the test case via ssh as text. The test case must then parse this text to determine whether the result of the command was as expected. Due to the wide variety of possible output, this is difficult to do under some circumstances without inspecting all of the code in testpmd that is reachable by that wrapper function, as well as handling all PMD errors. This is not always done properly, in part due to the amount of effort required to handle all error cases properly. Most of the data ends up being stringly typed, and this results in additional possibilities for human error.

3 of 8

Proposed Approach

Given that I just said that the current approach isn’t working, here’s a high level overview of what I think should be done. I have a few possible implementation methods later on, so please try to focus on whether this makes architectural sense. The way that this works is that instead of running testpmd, instead an RPC server would be exposed on the DUT at a known port. To call a DPDK function, a test suite would first call a function in the DTS framework that would wrap the RPC call, which would handle things like adding server information and processing network errors. This RPC call would be sent to the RPC server on the DUT. The RPC server would then call into a wrapper function similar to the one used by testpmd, which would gather handles and check state. This wrapper would then call into DPDK, and would handle converting it’s return value or other possible errors into a result type. This result type would look similar to value based error handling in functional languages, and would allow sending errors or an object back to the tester. This result structure would be serialized and sent back to DTS, where the test case would have the ability to easily check for the expected result. This process should provide some level of type checking to the RPC calls, since many RPC frameworks have typed methods to allow dynamically typed languages like Python to interact with statically typed languages like C safely.

4 of 8

Implementation

5 of 8

Python Standard Library XMLRPC

Benefits

Only requires python on the DUT at runtime
Would integrate very well with DTS’s current python code
Little to no work to add a new function to DTS once exposed via bindings
May be possible to automate RPC server generation from DPDK source

Costs

Python bindings to a DPDK wrapper would be required
Possibly not as fast as other approaches
GIL handling could be difficult

From the perspective of DTS, the most obvious solution is to use the XML RPC module in python’s standard library. It would easily integrate with the existing python codebase, and make handling exceptions easy. I think that the biggest benefit would be that there wouldn’t need to be any new dependencies for DTS, since python already needs to be present on the DUT. Once the RPC server has the ability to call a DPDK function, it should be trivial to call it on the client. In addition, I think that it may be possible to automate the generation of all of the DUT-side python code via C to python binding generators and the using reflection to collect them to be exposed on the server. This would be a large win for long-term maintenance cost. The biggest cost is the requirement for generating python bindings to the wrapper functions. I don’t think it’s going to be reasonable to try to generate them for all of DPDK, but generating them for headers full of functions that mostly take primitives should be easier and better separated from DPDK proper. This could be done by adding custom targets into the meson build process and using a binding generator to create python modules from the header file. The other issue is the Global Interpreter Lock in python, which could make some blocking functions cause issues. These functions may need to be started in another thread and the data transferred back once the function is over so that timeouts or other testing reliability mechanisms still work.

6 of 8

gRPC with C++/Rust RPC server and Python Client

Benefits

RPC server has zero overhead interactions with C
Statically typed interface
C++ wouldn’t need bindings

Costs

Apache License
gRPC does not support C, so another language is required
Rust might need bindings (meson has experimental support for this)
Everyone would need to learn the gRPC DSL (protobuf), which is not very hard

The next option would be to avoid using python on the side of the DUT, and instead use a systems language. This presents the issue of keeping both implementations in scope and having a standard way to communicate. I think that gRPC is probably the best way to handle this, since it has very good official support for python and C++, and Rust has a well-supported gRPC implementation called Tonic. Doing this would prevent full automation of the binding generation once the wrapper functions were written, but would mean zero-overhead calls into C and static typing. C++ also would not required any addition bindings to DPDK, and would be able to simply include the DPDK header files. As for downsides, gRPC is Apache licensed, which is an important consideration, since it is generating code. gRPC does not support C directly, and project maintainers have said that C++ is close enough for most use cases. I think that it is close enough in this case as well, and provided that a “C with classes” approach to C++ is taken it should not introduce too much cognitive overhead. Rust would need to have bindings generated for it, although there is already some work in meson towards this with the “Unstable Rust module”. It is also possible that if complex types are avoided in the wrapper function signatures then no bindings would be necessary. We could either push meson for stabilization, contribute enough to stabilize the module ourselves, or use it in it’s unstable state and fall back to the underlying program, rust-bindgen, if the unstable state causes issues. The final issue is that everyone who wants to add new wrapper functions to DTS would need to learn the gRPC DSL. This is not something that is very difficult, and the complete guide to the language is only about 7500 words including examples.

This approach might make the most sense from the perspective of being able to compile everything together on the DUT and not need to worry about python files.

7 of 8

Client and Server in C/C++/Rust, exposing a Python module

Benefits

RPC server has zero overhead interactions with C
May not need any external tools
Interactions are either cross-language or across the network, not both
Rust can easily create a python module via PyO3
Use of any RPC library/framework

Costs

Rust might bindings to DPDK wrapper
Compilation on DUT and Tester

This approach frees us from having to use an RPC library that can handle cross-language calls. It also still preserves the zero overhead interactions with C. Depending on what library or framework is used, additional build steps or tools might not be necessary. This also avoids interactions that are both cross-language and going over the network, which may help reduce cognitive overhead when adding new functionality. We still have to generate python bindings, since I don’t think that a full rewrite of DTS is feasible, but if we use Rust there is existing tooling to make that easy to do. Finally, this enables the use of any RPC library or framework, not just ones that are usable from python. This may allow the use of a library or framework that makes working from C feasible. The costs are that Rust may need to have bindings for the DPDK wrapper, similar to the last possible implementation, and that this would require compilation on both the DUT and the Tester. I think that we could structure the build so that building the client does not require compiling all of DPDK, which would make this much less of a cost, but it would mean that DTS would be unable to handle errors in meson. It should be possible for running DTS to automatically try to build the client, since it will be in the same git repository and python allows evaluating import statements after some code has been run.

8 of 8

What about Testpmd?

With this plan, testpmd would not go away. Instead, there are two paths forward that I can see. First, have testpmd call the wrapper functions and print things based off of the return value of the wrapper. This would mean that there might be some more indirection in testpmd, but it would mean that both the machine-oriented and the human-oriented interfaces would share most of their code. The other option would be to have testpmd communicate over RPC, using the same interface as DTS. This would introduce some overhead, but might open up the ability to use testpmd to remotely control a server and would mean everything goes into the wrappers through the same interface. In both cases, wrapper functions currently used by testpmd could be re-used for the wrapper functions for the RPC server, quickly bringing the RPC server up to the same level of functionality as testpmd with only minor refactoring to add the new information to the return value.