Sberbank Online iOS testing

Theory of testing is usually differs from practice. Today we want to talk about our practical experience in testing application's code which is used by millions of iOS users and about the difficult path that our team finished in order to achieve stable code.

Let’s imagine: the developers successfully convinced themselves and the business of the need to cover the code with tests. Over time, in the project were created more than a dozen thousand unit- and more than a thousand UI-tests. Such a large test base araised several problems, and we successfully found the solution for them.

In the first part of the article, we will get acquainted with the difficulties that arise while working with clean (non-integration) unit-tests, in the second part we will consider UI-tests closely.

In an ideal world, with unchanged source code, unit tests should always show the same result regardless of the number and sequence of it’s starts and constantly falling tests should never pass through the Continuous Integration(CI) server barrier.

In reality we may encounter the fact that the same unit-test will show either a positive or a negative result — “blinking” test. The reason for this behavior is a bad test implementation. Moreover, such test can pass CI with successful status, and later it will begin to fall on other people's Pull Request (PR). In such situation, there is a desire to disable this test or play trigger build-roulette and run the CI again. However, this approach is anti-productive, it undermines the credibility of tests and loads CI with meaningless work.

This issue was highlighted yesteryear at Apple's WWDC international conference:

• This session discusses about parallel testing, individual target coverage analysis and test run process.
• Here Apple narrates about testing network requests, mocking, testing notifications and the performance of tests.

Unit tests

To struggle blinking tests we use the following sequence of actions:



0. Evaluate test’s code quality according to basic criteria: isolation, mocks’ correctness, etc. We follow the rule: with a blinking test, we change the test’s code, and never the code that we test.

If this does not help, proceed as follows:

1. Fix and reproduce the conditions under which the test falls;
2. Find the reason why it falls;
3. Change the test’s code;
4. Go to the first step and check whether the cause of the fall has been eliminated.

Reproduce the fall

The simplest and most obvious option to reproduce the blinking fail is to run a problem test on the same version of iOS and on the same device, and usually in this case the test is successful! We have a thought: “Everything works for me locally, I will restart the build on CI”. But in fact, the problem has not been solved, and the test continues to fall on the build of another developer.

Therefore, at the next step, you need to run locally all the unit tests of the application to identify the potential affect of one test on another. But even after this check, your test result may be positive, while the problem remains undetected.

So, if test run was successful and the expected fail was not caught, you can repeat the run many times.

To do this you need to run a loop in terminal with xcodebuild:

#! /bin/sh
x=0
while [ $x -le 100 ];
    do xcodebuild -configuration Debug -scheme "TargetScheme" -workspace App.wcworkspace -sdk iphonesimulator -destination "platfrom=iOS Simulator, OS=11.3, name=iPhone 7" test >> "report.txt";
    x=$(( $x +1 ));
done 

This should be enough to reproduce the fall and move on to the next step — identifying the cause of the recorded fall.

Test’s fall reasons and possible solutions

Let’s dive deeper into the main causes of unit-tests blinking, which you may encounter, tools to identify problems, and possible solutions.

There are three main groups of reasons for blinking tests:

Bad test isolation

By isolation we mean a special case of encapsulation: a mechanism that allows restricting access of some program components to others.

Isolation of the environment has an important role for the purity of the test, nothing should affect the tested entities. Particular attention should be paid to tests that are aimed at checking the code and use global state entities, such as: global variables, Keychain, Network, CoreData, Singleton, NSUserDefaults and so on.

Imagine that while creating a test environment, a global state is set, which is implicitly used in another test code. In such case, the test may start to “blink” — due to the fact that, depending on the sequence of tests, two situations can arise — when the global state is set and when not set.

Oftenly, the described dependencies are implicit, so you may accidentally forget to set / reset such global states.

To make the dependencies clearly visible, you can use the principle of Dependency Injection (DI): pass the dependency through the parameters of the constructor, or a property of an object. This will make it easy to substitute mock dependencies instead of a real object.

Asynchronous calls

All unit tests are performed synchronously. The difficulty of testing asynchrony arises due to the call of the test method that “freezes” awaiting of the unit-test’s scope completion. The result will be a stable fail.

//act
	[self.testService loadImageFromUrl:@"www.google.ru" handler:^(UIImage * _Nullable image, NSError * _Nullable error) {
		//assert
		OCMVerify([cacheMock imageAtPath:OCMOCK_ANY]);
		OCMVerify([cacheMock dateOfFileAtPath:OCMOCK_ANY]);
		OCMVerify([imageMock new]);
		[imageMock stopMocking];
	}];
	
	[self waitInterval:0.2];

There are several approaches to test such a case:

1. Run NSRunLoop
2. waitForExpectationsWithTimeout

Both options require to specify an argument with a timeout. However, it cannot be guaranteed that the selected interval is sufficient. Locally, your test will pass, but on a heavily loaded CI there may not be enough power and it will fall — thus “blink” will appear.

Let’s imagine that we have some kind of data processing service. We want to verify that after receiving a response from the server, it transfers this data for further processing. To send requests via the network, the service uses the client to work with it. This case can be written asynchronously using a mock server to guarantee stable network responses.

@interface Service : NSObject

@property (nonatomic, strong) id<APIClient> apiClient;

@end

@protocol APIClient <NSObject>

- (void)getDataWithCompletion:(void (^)(id responseJSONData))completion;

@end

- (void)testRequestAsync
{
  // arrange
    __auto_type service = [Service new];
    service.apiClient = [APIClient new];

    XCTestExpectation *expectation = [self expectationWithDescription:@"Request"];

    // act
    id receivedData = nil;
    [self.service receiveDataWithCompletion:^(id responseJSONData) {
        receivedData = responseJSONData;
        [expectation fulfill];
    }];

    [self waitForExpectationsWithTimeout:10 handler:^(NSError * _Nullable error) {
        expect(receivedData).notTo.beNil();
        expect(error).to.beNil();
    }];
}

But the synchronous version of the test will be more stable and will allow you to get rid of working with timeouts. For this we need a synchronous APIClient mock.

@interface APIClientMock : NSObject <APIClient>
@end

@implementation

- (void)getDataWithCompletion:(void (^)(id responseJSONData))completion
{
  __auto_type fakeData = @{ @"key" : @"value" };
  if (completion != nil)
  {
    completion(fakeData);
  }
}

@end

Then the test will look simpler and it will work stabler.

- (void)testRequestSync
{
  // arrange
    __auto_type service = [Service new];
    service.apiClient = [APIClientMock new];

    // act
    id receivedData = nil;
    [self.service receiveDataWithCompletion:^(id responseJSONData) {
        receivedData = responseJSONData;
    }];

    expect(receivedData).notTo.beNil();
    expect(error).to.beNil();
}

Asynchronous operation can be isolated by encapsulating to a separate entity, which can be tested independently. Other part of the logic should be tested synchronously. Using approach you will avoid most of the pitfalls brought by asynchrony.

As an option, in the case of updating the UI layer from the background thread, you can check whether it’s the main thread and what will happen if we make a call from the test:

func performUIUpdate(using closure: @escaping () -> Void) {
    // If we are already on the main thread, execute the closure directly
    if Thread.isMainThread {
        closure()
    } else {
        DispatchQueue.main.async(execute: closure)
    }
}

For a detailed explanation, see the article by D. Sandell.

Testing code beyond your control

Often we forget about the following things:

• the implementation of the methods may depend on the localization of the application,
• there are private methods in the SDK that can be called by framework classes,
• the implementation of the methods may depend on the version of the SDK.

These cases bring some uncertainty in the process of writing and running tests. To avoid negative consequences, you need to run tests on all locales, on all versions of iOS supported by your application. Separately, it should be noted that there is no need to test code whose implementation is hidden from you.

Finally, we want to complete this part about automated testing of the Sberbank Online iOS application, dedicated to unit testing.

The article was written with @regno — Anton Vlasov, head of iOS development.