Testing Errors in Rust

Currently besides all other activities, I am developing a habit of programming following Test Driven Development (TDD) methodology. This is a perfect time because I continue to explore Rust, a new programming language to me. Moreover, this language encourages you to cultivate this best practice by providing great documentation and well-thought ecosystem.

In our programs, we often face with exceptional situations (e.g., lack of space when you try to write a file, or absence of a resource), and we need to handle them. If you follow the TDD approach, you need to ensure that these exceptional situations are also properly covered in your tests. Id est, you have to develop tests that reproduce these exceptional situations and make sure that your code detect and handle them correctly. In this post, I want to discuss how to test exceptional situations in Rust.

Error Handling in Rust

In Rust, the fact that expected exceptional situation may occur in a function is expressed using a special enum Result<T, E> in the standard library. A function that may arise an exceptional situation is expected to return either Ok(T) in case of success (T is the type of returned value) or Err(E) in case of error (E is the type of error). The calling code is expected to process these cases correspondingly.

In real life, most often we face with input-output errors. In Rust, these errors are represented by a structure std::io::Error from the standard library. Often times, you can also make use of this structure to represent your own errors so that you do not need to create a separate type for them. If you want to know more about the std::io::Error type and how to use it to wrap your custom errors, I recommend reading Alexey Kladov’s post where he dissects this standard library type.

Example

For the sake of simplicity, let’s consider the following simple example. We have requirements to develop a library that reads a txt file from a disk using the provided path and parses it into an unsigned integer number. If the provided path does not point to a file or the file does not have the .txt extension, the library has to report an error.

Let’s start from creating a library project:

$ cargo new --lib testing_errors
$ cd testing_errors

According to our requirements, the library should expose one public method, e.g., extract_integer(), that should borrow path to a file and return positive integer value if the path points to a txt file with correct data. Logically, this public method performs two operations, namely it reads a text file and parses the content. Therefore, we will create two private methods, namely read_file() and get_number(), that correspond to these operations.

So as the purpose of the article is to show how to test exceptional situations, we will violate the order enforced by the TDD methodology and start from writing the functions code, not the tests. So, let’s open the lib.rs file and put there the code of our public and private interfaces. However, at first, we need to specify all the imports that we will use in our library:

use std::{
    error, fs,
    io::{self, ErrorKind},
    path::Path,
};

Private Interface

The private interface constitute the functions that are accessible only within the library and cannot be called from external crates. These functions are defined without the pub modifier.

Private Functions

Let’s start with developing our read_file function:

fn read_file(file_path: &Path) -> Result<String, io::Error> {
    if !file_path.is_file() {
        let not_a_file_error = io::Error::new(
            ErrorKind::InvalidInput,
            format!("Not a file: {}", file_path.display()),
        );
        return Err(not_a_file_error);
    }

    if file_path.extension().unwrap() != "txt" {
        let incorrect_ext_error = io::Error::new(
            ErrorKind::InvalidInput,
            "The file should have txt extension!",
        );
        return Err(incorrect_ext_error);
    }

    let str = fs::read_to_string(file_path)?;
    Ok(str)
}

This function begins with several guards. Firstly, we check that file_path points to a file. If the provided path does not point to a file, we create a std::io::Error error of the ErrorKind::InvalidInput kind (not_a_file_error). As an internal custom error, we use a string "Not a file: <provided_file_path>". Then, we return the created Err-wrapped error from the function.

Secondly, we check that the extension of the file is txt. If it is not, similarly to the previous case, we create a std::io::Error error using the string "The file should have txt extension!" as an internal custom error. Note, akin to previous case we use the same kind for the error, namely ErrorKind::InvalidInput.

pub struct Error {
  repr: Repr,
}

enum Repr {
  Os(i32),
  Simple(ErrorKind),
  Custom(Box<Custom>),
}

struct Custom {
  kind: ErrorKind,
  error: Box<dyn error::Error + Send + Sync>,
}

#[derive(Clone, Copy)]
#[non_exhaustive]
pub enum ErrorKind {
  NotFound,
  PermissionDenied,
  Interrupted,
  ...
  Other,
}

impl Error {
  pub fn kind(&self) -> ErrorKind {
    match &self.repr {
      Repr::Os(code) => sys::decode_error_kind(*code),
      Repr::Custom(c) => c.kind,
      Repr::Simple(kind) => *kind,
    }
  }
}

If these checks pass, we read the content of the file to a string using the fs::read_to_string() method and return this wrapped string if there are no issues (note the ? sign after the method). Note that in case of an error we return wrapped io::Error type.

The function get_number() is simpler:

fn get_number(str: &str) -> Result<u32, std::num::ParseIntError> {
    let result = str.trim().parse::<u32>()?;
    Ok(result)
}

As you can see, this function trims the leading and trailing whitespaces and tries to parse a string slice into a u32 number. If the operation succeeds, the function returns the extracted integer wrapped into Result’s Ok enum variant. If the parsing fails the function automatically returns the std::num::ParseIntError error (so as we try to extract an integer) wrapped into Result’s Err enum variant.

Unit Testing Private Functions

Testing read_file() Function

Now, let’s test our private interface. We will put our tests into the same lib.rs file into a separate module tests:

#[cfg(test)]
mod tests {
    // we will write our tests here
}

Let’s start developing tests for our read_file() function. At first, we will test the normal flow of the application. Obviously, the normal flow of this function should return a string containing the content of the file given that the constraints are met (path points to a correct txt file). So, the simplest unit test may look in the following way (I have created a directory in the project root named test_data and put there a text file correct.txt that contains number 5 followed by a new line character):

#[test]
fn read_file_should_return_correct_str() {
    let actual_str = read_file(Path::new("test_data/correct.txt")).unwrap();
    let expected_str = "5\n".to_string();
    assert_eq!(actual_str, expected_str);
}

You can ensure that this test passes running the command:

$ cargo test tests::read_file_should_return_correct_str
    Finished test [unoptimized + debuginfo] target(s) in 0.00s
     Running target/debug/deps/testing_errors-4f8c6a2b04a527ed

running 1 test
test tests::read_file_should_return_correct_str ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 9 filtered out

So as the main purpose of this post is to show how to test errors, in what follows we will concentrate on the tests that cover exceptional situations. For instance, let’s consider the case when the provided path points to a directory but not a file. The easiest way to test such a situation is to write the following test:

#[test]
fn read_file_should_return_err_if_not_file() {
    assert!(read_file(Path::new("test_data/")).is_err());
}

Here, we assert if the function read_file returns an error when the passed path points to a directory. The method is_err() returns true if the read_file call finishes with an error, and false otherwise. Alternatively, to test the same logic we can use #[should_panic] test annotation and the unwrap method that unwraps returned value in case of success and panics if read_file returns an error:

#[test]
#[should_panic]
fn read_file_should_panic_if_not_file() {
    read_file(Path::new("test_data/")).unwrap();
}

$ cargo test --package testing_errors --lib --all-features -- tests::read_file_should_return_err_if_not_file --exact --nocapture
    Finished test [unoptimized + debuginfo] target(s) in 0.00s
     Running target/debug/deps/testing_errors-4f8c6a2b04a527ed

running 1 test
test tests::read_file_should_return_err_if_not_file ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 10 filtered out

$ cargo test --package testing_errors --lib --all-features -- tests::read_file_should_panic_if_not_file --exact --nocapture
    Finished test [unoptimized + debuginfo] target(s) in 0.00s
     Running target/debug/deps/testing_errors-4f8c6a2b04a527ed

running 1 test
thread 'tests::read_file_should_panic_if_not_file' panicked at 'called `Result::unwrap()` on an `Err` value: Custom { kind: InvalidInput, error: "Not a file: test_data/" }', src/lib.rs:83:44
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
test tests::read_file_should_panic_if_not_file ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 10 filtered out

Both these unit tests check if the function under the test generates an error. However, both of them conceal what was the cause of the error. For instance, both these tests will pass if the path does not point to a file or points to a file with a wrong extension. Even in case something happen during file reading, these tests will pass. Therefore, it is unclear what part of the code we test with them, i.e., they lack for specificity and may lead to false positives or false negatives. For instance, let’s consider the following test case:

#[test]
fn read_file_fail_but_should_pass_if_points_to_txt_file() {
    assert!(read_file(Path::new("test_data/perm_denied.txt")).is_err());
}

According to our requirements, if we provide a correct path to a .txt file, our function under the test should successfully read and return the content of the file, therefore, this test should pass. Unfortunately, it generates an error (false negative) because the application does not have enough permissions to read the file.

Thus, such way of testing exceptional cases may camouflage the real cause of the error. Luckily, the io::Error structure has the kind() method that shows why the error has happened. Let’s make use it in our test:

#[test]
fn read_file_should_return_error_if_perm_denied() {
    let actual_error_kind = read_file(Path::new("test_data/perm_denied.txt"))
        .unwrap_err()
        .kind();
    let expected_error_kind = ErrorKind::PermissionDenied;
    assert_eq!(actual_error_kind, expected_error_kind);
}

Unfortunately, this test is not perfect also. It may happen that several different IO errors have the same ErrorKind. For instance, in our read_file() we create two different instances of the std::io::Error errors, namely not_a_file_error and incorrect_ext_error, both of which have the same error kind. Thus, your test cases need somehow distinguish these two cases. After reading Alexey Kladov’s article and spotting the into_inner() method, you could think that you may write something like this:

#[test]
fn read_file_should_return_not_a_file_str_error_if_not_file() {
    let file_path = Path::new("test_data/");
    let actual_inner_error = read_file(file_path).unwrap_err().into_inner().unwrap();
    let expected_inner_error = &format!("Not a file: {}", file_path.display());
    assert_eq!(actual_inner_error, expected_inner_error);
}

Unfortunately, the Rust compiler complains about this code because we cannot directly compare Box<dyn std::error::Error + Send + Sync> and &String:

$ cargo test tests::read_file_should_return_not_a_file_str_error_if_not_file
   Compiling testing_errors v0.1.0 (/home/yury/PROJECTS/TESTS/RUST/BLOG/testing_errors)
error[E0369]: binary operation `==` cannot be applied to type `Box<dyn std::error::Error + Send + Sync>`
   --> src/lib.rs:116:9
    |
116 |         assert_eq!(actual_inner_error, expected_inner_error);
    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |         |
    |         Box<dyn std::error::Error + Send + Sync>
    |         &String
    |
    = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)

error: aborting due to previous error

For more information about this error, try `rustc --explain E0369`.
error: could not compile `testing_errors`

To learn more, run the command again with --verbose.

What? It seems that we can’t test these cases, can we? Nope, we can use the following hack: let’s make use the fact that errors and io::Error in particular, implement Display and Debug traits. This means that we can compare string representations of the errors. Simply applying this to our previous method we can develop the following test:

#[test]
fn read_file_should_return_not_a_file_str_error_if_not_file_alt() {
    let file_path = Path::new("test_data/");
    let actual_inner_error_disp = format!("{}", read_file(file_path).unwrap_err().into_inner().unwrap());
    let expected_inner_error_disp = format!("Not a file: {}", file_path.display());
    assert_eq!(actual_inner_error_disp, expected_inner_error_disp);
}

However, this code is quite verbose and hard to understand. Can we make it clearer? Apparently, yes, if we use the expected attribute of the should_panic test annotation, we can write the following test:

#[test]
#[should_panic(expected = "The file should have txt extension!")]
fn read_file_should_panic_if_not_txt_file() {
    read_file(Path::new("test_data/incorrect.ttt")).unwrap();
}

In the expected attribute, we write the debug string representation of the error that we expect to obtain during a run. Note that the string there is static, therefore, if you dynamically generate error string you have to use only static part of your string. For instance, if the path does not point to a file, our custom error string contains the provided path that is a runtime value and, therefore, cannot be known at compile time. Hence, the test for this case may look in the following way (I omit the dynamic part and leave only static part):

#[test]
#[should_panic(expected = "Not a file:")]
fn read_file_should_panic_expected_if_not_file() {
    read_file(Path::new("test_data/")).unwrap();
}

Testing get_number Function

Let’s continue our explorations of error testing in Rust by checking the get_number() function. As before, we add a couple of normal flow tests:

#[test]
fn get_number_should_return_integer_when_correct_string() {
    let expected_val: u32 = 5;
    let actual_val = get_number(&expected_val.to_string()).unwrap();
    assert_eq!(expected_val, actual_val);
}

#[test]
fn get_number_should_parse_correctly_when_string_with_spaces() {
    let expected_val: u32 = 5;
    let actual_val = get_number(&format!(" {}   ", expected_val)).unwrap();
    assert_eq!(expected_val, actual_val);
}

They are quite simple, and you can easily understand what is going on there. Also, similarly to the example in the previous section, it is quite easy to write tests that check if an error happened in general:

#[test]
#[should_panic]
fn get_number_should_panic_when_string_cannot_be_parsed() {
    get_number("j").unwrap();
}

#[test]
fn get_number_returns_error_when_string_cannot_be_parsed() {
    assert!(get_number("j").is_err());
}

However, if you would try to develop tests for specific kinds of errors akin to the examples in the previous section, you will find out that you cannot do this. For instance, the following code does not compile:

#[test]
fn get_number_returns_error_when_string_is_incorrect() {
    let actual_error_kind = get_number("j").unwrap_err().kind();
    let expected_error_kind = IntErrorKind::InvalidDigit;
    assert_eq!(expected_error_kind, actual_error_kind);
}

In order to understand why this test does not compile, we need to pass through the source code of the std::num::ParseIntError structure.

#[unstable(
    feature = "int_error_matching",
    reason = "it can be useful to match errors when making error messages \
              for integer parsing",
    issue = "22639"
)]
#[derive(Debug, Clone, PartialEq, Eq)]
#[non_exhaustive]
pub enum IntErrorKind {
    Empty,
    InvalidDigit,
    PosOverflow,
    NegOverflow,
    Zero,
}

#[derive(Debug, Clone, PartialEq, Eq)]
#[stable(feature = "rust1", since = "1.0.0")]
pub struct ParseIntError {
    pub(super) kind: IntErrorKind,
}

impl ParseIntError {
    /// Outputs the detailed cause of parsing an integer failing.
    #[unstable(
        feature = "int_error_matching",
        reason = "it can be useful to match errors when making error messages \
              for integer parsing",
        issue = "22639"
    )]
    pub fn kind(&self) -> &IntErrorKind {
        &self.kind
    }
    #[unstable(
        feature = "int_error_internals",
        reason = "available through Error trait and this method should \
                  not be exposed publicly",
        issue = "none"
    )]
    #[doc(hidden)]
    pub fn __description(&self) -> &str {
        match self.kind {
            IntErrorKind::Empty => "cannot parse integer from empty string",
            IntErrorKind::InvalidDigit => "invalid digit found in string",
            IntErrorKind::PosOverflow => "number too large to fit in target type",
            IntErrorKind::NegOverflow => "number too small to fit in target type",
            IntErrorKind::Zero => "number would be zero for non-zero type",
        }
    }
}

#[stable(feature = "rust1", since = "1.0.0")]
impl fmt::Display for ParseIntError {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        self.__description().fmt(f)
    }
}

You can see that in the current version of the Rust compiler (rustc 1.49.0), IntErrorKind and kind() methods are marked as unstable features. Therefore, you cannot use these things in your stable code. Therefore, if you want to distinguish different error kinds in your code, you need to match against error string representation. For instance, for our get_number() function we can use the following code:

match get_number(&str_data) {
    Ok(data) => ...,
    Err(e) => {
        let e_repr = e.to_string();
        if e_repr == "cannot parse integer from empty string" {
            ...
        }
        else if e_repr == "invalid digit found in string" {
            ...
        }
        else {
            ...
        }
    } 
}

// or using match guards
match get_number(&str_data) {
    Ok(data) => ...,
    Err(e) if e.to_string() == "cannot parse integer from empty string" => {
        ...
    }, 
    Err(e) if e.to_string() == "invalid digit found in string" => {
        ...
    }, 
}

Similarly, to test such exceptional situation we can add, e.g., the following test to ensure that we cannot parse a letter into an integer:

#[test]
fn get_number_should_return_error_when_string_if_incorrect_number() {
    match get_number("j") {
        Err(e) if e.to_string() == "invalid digit found in string" => return (),
        Err(_) => panic!("Returned incorrect Err!"),
        Ok(_) => panic!("Returned an Ok variant!"),
    }
}

There are several issues with this approach. First, so as we compare against strings we cannot be sure that all error cases are covered (Rust cannot enforce coverage of all possible strings). Second, if there is a small change in an error string, we will need to correct our code correspondingly. This may be a tough and error prone procedure because Rust compiler does not help us. However, once the IntErrorKind feature is stabilized, we would be able to write better code and tests for this error type.

Interestingly, at first I assumed that it is possible to exploit the expected attribute to test these errors similarly as we did this before. My assumption was that the following test should pass:

#[test]
#[should_panic(expected = "invalid digit found in string")]
fn get_number_returns_error_when_string_is_incorrect() {
    get_number("j").unwrap();
}

Unfortunately, this is not true. In order to understand what causes the problem we need to check the sources of the Result’s unwrap() method:

pub fn unwrap(self) -> T {
    match self {
        Ok(t) => t,
        Err(e) => unwrap_failed("called `Result::unwrap()` on an `Err` value", &e),
    }
}

// where

fn unwrap_failed(msg: &str, error: &dyn fmt::Debug) -> ! {
    panic!("{}: {:?}", msg, error)
}

As you can see, the unwrap() method produces a panic message using the Debug trait formatting. For the std::num::Error structure, this trait produces a debug string that contains the value of the structure: ParseIntError { kind: InvalidDigit }. If you still want to use the expected attribute, you can develop the following tests:

#[test]
#[should_panic(expected = "invalid digit found in string")]
fn get_number_returns_error_when_string_is_incorrect() {
    panic!(get_number("j").unwrap_err().to_string());
}

#[test]
#[should_panic(expected = "kind: InvalidDigit")]
fn get_number_returns_error_when_string_is_incorrect_alt() {
    get_number("j").unwrap();
}

With these examples I wanted to show that currently, there is no standard way of testing errors in Rust. Most probably, you would need to consider particular situations case-by-case. Hopefully, after several language edition iterations we would have a unified way to test errors, at least in the standard library.

Public Interface

Our library is very simple, it has only one public method: extract_integer(). Its functionality is very simple: at first, it calls the read_file() function, then, parses the obtained string with the help of the get_number() function and returns the result. For instance, the code of this function may look like the following:

pub fn extract_integer(file_path: &Path) -> Result<u32, E> {
    let str_data = read_file(file_path)?;
    let data = get_number(&str_data)?;
    Ok(data)
}

However, it is not clear what this function should return, i.e., what we should put instead of E in the Result<u32, E>. The problem is that the read_file() function in case of an error returns the std::io::Error type, while get_number() returns std::num::ParseIntError; and these types cannot be converted to each other. Therefore, we need to find a super type that both of them can be converted to. Luckily, the standard library has the std::error::Error trait that both std::io::Error and std::num::ParseIntError implement. Therefore, we can use a trait object and resolve at runtime a method of which structure should be called. For the sake of simplicity, let’s create an alias for this type and use it for an error type variant:

type BoxError = Box<dyn error::Error>;

pub fn extract_integer(file_path: &Path) -> Result<u32, BoxError> {
    let str_data = read_file(file_path)?;
    let data = get_number(&str_data)?;
    Ok(data)
}

Testing Public Interface Function

Of course, there is a question if we should test our public interface function. Indeed, it is very simple and consists of the private functions we have already tested. However, in real world functions will not be that simple, and you would need to develop code to test them too. So, we will also write some tests for our public interface function.

Our public interface function returns a boxed error if something fails. So, the question is how to test faulty situations in this case. Luckily, due to the implementation of the Deref trait for Box and dynamic dispatching we can use almost all the tricks we have considered for our private interface functions. For instance, we can easily write such tests:

#[test]
#[should_panic(expected = "Not a file:")]
fn extract_integer_should_panic_if_not_file() {
    extract_integer(Path::new("test_data/")).unwrap();
}

#[test]
#[should_panic(expected = "The file should have txt extension!")]
fn extract_integer_should_panic_if_not_txt_file() {
    extract_integer(Path::new("test_data/incorrect.ttt")).unwrap();
}

#[test]
fn extract_integer_should_return_err_if_incorrect_number() {
    assert!(extract_integer(Path::new("test_data/incorrect_number.txt")).is_err());
}

However, there is one particular case that stands out. We can use the is() method defined in the std::error::Error trait that can be exploited to find out what particular type of error our public interface function has generated. The tests, which use this method, work out to be quite descriptive and easy to understand to me. For instance, we can add the following test to check that the error returned by our extract_integer() function is equal to std::num::ParseIntError if the file contains incorrect number:

#[test]
fn extract_integer_should_return_parseinterror_if_incorrect_number() {
    assert!(extract_integer(Path::new("test_data/incorrect_number.txt"))
        .unwrap_err()
        .is::<std::num::ParseIntError>());
}

Therefore, if your function coerce errors to std::error::Error, you can make use of this method to find out what particular error type has caused this.

Conclusion

Rust is an iceberg. I planned to write a very short note how to use should_panic’s expected attribute to test errors, but during my explorations I have found the “under-the-surface” part, and it took me lots of hours to understand these things. I am not sure that I have dissected every piece of “ice”, but at least this topic now seems clearer to me.