Quick Specs with Automated Testing Instruments

by Martin Führlinger, Backend Engineer

Within the backend group we normally attempt to automate issues. Subsequently we’ve got tons of exams to confirm correctness of our code in our gems and companies. Automated exams are executed a lot quicker and with a lot increased protection than any tester can do manually in an analogous time. Over time lots of performance has been added, and because of this, lots of exams have been added. This led to our take a look at suites turning into slower over time. For instance, we’ve got a service the place round 5000 exams take about Eight minutes. One other service takes about 15 minutes for round 3000 exams. So why is service A so quick and repair B so sluggish? 

On this weblog submit I’ll present some dangerous examples of the way to write a take a look at, and the way to enhance automated testing instruments to make exams quicker. The Runtastic backend group normally makes use of `rspec` together with `factory_bot` and jruby

The Check File Instance

The next code exhibits a small a part of an actual instance of a take a look at file we had in our take a look at suite. It creates some customers and tries to seek out them with the UserSearch use case.

describe  Customers::UseCase::UserSearch::ByAnyEmail do
 describe "#run!" do
   topic { Customers::UseCase::UserSearch::ByAnyEmail.new(search_criteria).run! }
   let(:current_user_id) { nil }
   let(:search_criteria) { double(question: question, measurement: measurement, quantity: quantity, current_user_id: current_user_id) }
   let(:default_photo_url) { "#{Rails.configuration.companies.runtastic_web.public_route}/belongings/consumer/default_avatar_male.jpg" }
   def expected_search_result_for(consumer)
     UserSearchResultWrapper.new(consumer.attributes.merge("avatar_url" => default_photo_url))
   finish
   shared_examples "discover customers by electronic mail" do
     it { count on(topic.class).to eq UserSearchResult }
     it { count on(topic.customers).to return_searched_users expected_users }
     it { count on(topic.more_data_available).to eq more_data? }
   finish
   let!(:s_m_user)           { FactoryBot.create :consumer, electronic mail: "[email protected]" }
   let(:runner_gmail_user)  { FactoryBot.create :consumer, google_email: "[email protected]" }
   let!(:su_12_user)         { FactoryBot.create :consumer, electronic mail: "[email protected]" }
   let(:su_12_google_user)   { FactoryBot.create :consumer, google_email: "[email protected]" }
   let!(:user_same_mail) do
     FactoryBot.create :consumer, electronic mail: "[email protected]", google_email: "[email protected]”
   finish
   let!(:combined_user) do
     FactoryBot.create :consumer, electronic mail: "[email protected]", google_email: "[email protected]"
   finish
   let!(:johnny_gmail_user)  { FactoryBot.create :consumer, google_email: "[email protected]" }
   let!(:jane_user)          { FactoryBot.create :consumer, electronic mail: "[email protected] mail.at" }
   let!(:zorro)              { FactoryBot.create :consumer, electronic mail: "[email protected]" }
   earlier than do
     FactoryBot.create(:consumer, google_email: "[email protected] mail.at").faucet do |u|
       u.update_attribute(:deleted_at, 1.day.in the past)
     finish
     runner_gmail_user
     su_12_google_user
   finish
   context "the question is '123'" do
     it_behaves_like "discover customers by electronic mail" do
       let(:measurement)     { 4 }
       let(:quantity)   { 1 }
       let(:question) { [123] }
       let(:expected_users) { [] }
       let(:more_data?) { false }
     finish
   finish
   context "the question comprises invalid emails" do
     it_behaves_like "discover customers by electronic mail" do
       let(:question) do
         ["[email protected]", "su+12gmx.at", "", "'", "[email protected]"]
       finish
       let(:measurement)   { 50 }
       let(:quantity) { 1 }
       let(:expected_users) do
         [
           expected_search_result_for(s_m_user),
           expected_search_result_for(johnny_gmail_user)
         ]
       finish
       let(:more_data?) { false }
     finish
   finish
 finish
finish

So let’s analyze the take a look at: It has a topic, which signifies what to check. On this case: Run the use case and return the consequence. It defines a shared instance which comprises the precise exams. These shared examples assist as a result of the exams are grouped collectively and they are often reused. This manner it’s doable to only arrange the exams with completely different parameters and name the instance by utilizing it_behaves_like. The take a look at above comprises some consumer objects created with let and a earlier than block, which is known as earlier than every take a look at. The it-block comprises two contexts to explain the setup and calls the shared instance as soon as per context. So mainly this take a look at runs 6 exams (Three exams within the shared_example, known as twice). Operating them regionally on my laptop computer outcomes on this:

Customers::UseCase::UserSearch::ByAnyEmail
 #run!
   the question is '123'
     behaves like discover customers by electronic mail
       ought to return searched customers
       ought to eq false
       ought to eq UserSearchResult
   the question comprises invalid emails
     behaves like discover customers by electronic mail
       ought to eq UserSearchResult
       ought to eq false
       ought to return searched customers #<UserSearchResultWrapper:0x7e9d3832 @avatar_url="http://localhost.runtastic.com:3002/belongings/consumer/def....jpg", @country_id=nil, @gender="M", @id=51, @last_name="Doe-51", @first_name="John", @guid="ab
c51"> and #<UserSearchResultWrapper:0x33c8a528 @avatar_url="http://localhost.runtastic.com:3002/belongings/consumer/def....jpg", @country_id=nil, @gender="M", @id=55, @last_name="Doe-55", @first_name="John", @guid="abc55">
Completed in 34.78 seconds (information took 20.66 seconds to load)
6 examples, zero failures

So about 35 seconds for six exams.

Let vs Let! 

As you possibly can see, we’re utilizing let! and let. The distinction between these two strategies is, that let! all the time executes, and let solely executes if the reference is used. Within the above instance:

let!(:s_m_user)
let(:runner_gmail_user)

“s_m_user” is created all the time, “runner_gmail_user” is created provided that used. So the above let! usages are creating 7 customers for the exams.

Earlier than Block

The earlier than block can be executed each time earlier than the take a look at. If nothing is handed to the earlier than methodology, it defaults to :every. The above earlier than block creates a consumer, and references 2 different customers, which then instantly are created, too. 

So we’re creating 10 customers for every take a look at. 

rspec-it-chains

As each it is a single take a look at, the shared instance comprises Three single exams. Each take a look at will get a clear state, so the customers are created once more for every take a look at. Having a number of it blocks one after one other, referring to the identical topic, by some means seems like a sequence.

Check setup

What do the exams truly do? The primary one passes a web page measurement of Four with a question “123” to the search use case and expects, as no consumer has 123 within the electronic mail attribute, no customers to be discovered.

context "the question is '123'" do
     it_behaves_like "discover customers by electronic mail" do
       let(:measurement)     { 4 }
       let(:quantity)   { 1 }
       let(:question) { [123] }
       let(:expected_users) { [] }
       let(:more_data?) { false }
     finish
   Finish

So we’re creating Three instances (3 it blocks) 10 customers however count on no consumer to be discovered.

The second context passes a few of the emails, and a few invalid ones into the search, and count on 2 customers to be discovered. 

context "the question comprises invalid emails" do
    it_behaves_like "discover customers by electronic mail" do
      let(:question) do
        ["[email protected]", "su+12gmx.at", "", "'", "[email protected]"]
      finish
      let(:measurement)   { 50 }
      let(:quantity) { 1 }
      let(:expected_users) do
        [
          wrap(s_m_user),
          wrap(johnny_gmail_user)
        ]
      finish
      let(:more_data?) { false }
    finish
  finish

So we’re creating Three instances 10 customers to have the ability to discover 2 of them in a single take a look at and get the correct flag in one other take a look at.

Having a better take a look at the shared_example:

it { count on(topic.class).to eq UserSearchResult }
 it { count on(topic.customers).to return_searched_users expected_users }
 it { count on(topic.more_data_available).to eq more_data? }

you possibly can see that the primary one just isn’t even anticipating something user-related to be returned. It simply expects the use-case to return a particular class. The second truly exams if the consequence comprises the customers we need to discover. The third it block checks if the more_data_available flag is about correctly.

General, we’ve got 6 exams, needing 35 seconds to run, creating 10 customers for every take a look at (60 customers totally) and calling the topic 6 instances, and we mainly solely anticipate finding 2 customers as soon as.

Clearly, this may be improved.

Enchancment

Initially, let’s eliminate the it chain, mix it inside one it block.

shared_examples "discover customers by electronic mail" do
  it "returns consumer information" do
    count on(topic.class).to eq UserSearchResult
    count on(topic.customers).to return_searched_users expected_users
    count on(topic.more_data_available).to eq more_data?
  finish
finish

Combining it blocks is smart in the event that they normally take a look at an analogous factor (as above). For instance, doing a request and anticipating some response physique and standing 200 doesn’t should be two separate exams. Combining two it blocks which take a look at one thing completely different, nevertheless, doesn’t make sense, comparable to exams for the response code of a request and if that request saved the info accurately within the database.

This leads to the exams ending inside ~ 15 seconds, solely 2 examples.

The subsequent step is to not create the customers if they don’t seem to be wanted. Subsequently let’s change to let as a substitute of let!. Additionally take away the earlier than block as it’s, and solely create some correct quantity of customers mandatory for the take a look at. The exams seem like this in finish:

describe  Customers::UseCase::UserSearch::ByAnyEmail do
 describe "#run!" do
   topic { Customers::UseCase::UserSearch::ByAnyEmail.new(search_criteria).run! }
   let(:current_user_id) { nil }
   let(:search_criteria) { double(question: question, measurement: measurement, quantity: quantity, current_user_id: current_user_id) }
   let(:default_photo_url) { "#{Rails.configuration.companies.runtastic_web.public_route}/belongings/consumer/default_avatar_male.jpg" }
   def expected_search_result_for(consumer)
     UserSearchResultWrapper.new(consumer.attributes.merge("avatar_url" => default_photo_url))
   finish
   shared_examples "discover customers by electronic mail" do
     it "return consumer information" do
       count on(topic.class).to eq UserSearchResult
       count on(topic.customers).to return_searched_users expected_users
       count on(topic.more_data_available).to eq more_data?
     finish
   finish
   let(:s_m_user)           { FactoryBot.create :consumer, electronic mail: "[email protected]" }
   let(:runner_gmail_user) { FactoryBot.create :consumer, google_email: "[email protected]" }
   let(:su_12_user)         { FactoryBot.create :consumer, electronic mail: "[email protected]" }
   let(:su_12_google_user)  { FactoryBot.create :consumer, google_email: "[email protected]" }
   let(:user_same_mail) do
     FactoryBot.create :consumer, electronic mail: "[email protected]", google_email: "[email protected]"
   finish
   let(:combined_user) do
     FactoryBot.create :consumer, electronic mail: "[email protected]", google_email: "[email protected]"
   finish
   let(:johnny_gmail_user)  { FactoryBot.create :consumer, google_email: "[email protected]" }
   let(:jane_user)          { FactoryBot.create :consumer, electronic mail: "[email protected] mail.at", fb_proxied_email: "[email protected]" }
   let(:zorro)              { FactoryBot.create :consumer, electronic mail: "[email protected]" }
   let(:deleted_user) do
     FactoryBot.create(:consumer, google_email: "[email protected] mail.at").faucet do |u|
       u.update_attribute(:deleted_at, 1.day.in the past)
     finish
   finish
   context "the question is '123'" do
     earlier than do
       s_m_user
     finish
     it_behaves_like "discover customers by electronic mail" do
       let(:measurement)     { 4 }
       let(:quantity)   { 1 }
       let(:question) { [123] }
       let(:expected_users) { [] }
       let(:more_data?) { false }
     finish
   finish
   context "the question comprises invalid emails" do
     earlier than do
       s_m_user
       su_12_user
       johnny_gmail_user
     finish
     it_behaves_like "discover customers by electronic mail" do
       let(:question) do
         ["[email protected]", "su+12gmx.at", "", "'", "[email protected]"]
       finish
       let(:measurement)   { 50 }
       let(:quantity) { 1 }
       let(:expected_users) do
         [
           expected_search_result_for(s_m_user),
           expected_search_result_for(johnny_gmail_user)
         ]
       finish
       let(:more_data?) { false }
     finish
   finish
 finish
finish

And end in 

Customers::UseCase::UserSearch::ByAnyEmail
 #run!
   the question is '123'
     behaves like discover customers by electronic mail
       return consumer information
   the question comprises invalid emails
     behaves like discover customers by electronic mail
       return consumer information                                                                                                                                                                                                                                        
Completed in 8.16 seconds (information took 22.34 seconds to load)
2 examples, zero failures

As you possibly can see, I do create customers, even when I don’t count on them to be within the consequence, to show the correctness of the use case. However I don’t create 10 per take a look at, just one and three. A number of the above customers will not be created (or used) in any respect now, however as the unique take a look at file comprises extra exams, which ultimately want them once more for different contexts, I saved them within the instance too.

So now we solely create Four customers, as a substitute of 60. By simply adapting the code a bit, we’ve got the identical take a look at protection with solely 2 exams as a substitute of 6, and solely needing Eight as a substitute of 35 seconds, which is 77% much less time.

FactoryBot: create vs construct vs attribute_for

As you possibly can see above, we’re utilizing FactoryBot closely to create objects throughout the exams.

let(:consumer) { FactoryBot.create(:consumer) }

This creates a brand new consumer object as quickly as `consumer` is referenced within the exams. The disadvantage of this line is that it actually creates the consumer within the database, which is fairly typically not mandatory. The higher method, if relevant, could be to solely construct the thing with out storing it:

let(:consumer) { FactoryBot.construct(:consumer) }

Clearly this doesn’t work in case you want the thing within the database, as for the take a look at instance above, however that extremely is dependent upon the take a look at. One other much less identified characteristic of FactoryBot is to create solely the attributes for an object, represented as hash.

let(:user_attrs) { FactoryBot.attributes_for(:consumer) }

This could create a hash containing the attributes for a consumer. It doesn’t even create a Consumer object, which is even quicker than construct. 

A doable easy take a look at could be:

describe Consumer do
  50.instances do  
    topic { FactoryBot.create(:consumer) }
    it { count on(topic.has_first_login_dialog_completed).to eq(false) }
  finish
finish

Because the has_first_login_dialog_completed methodology solely wants some attributes set on a consumer, regardless of whether it is saved in a database, a construct could be a lot quicker than a create, working the take a look at 100 instances to additionally use the impact of the just-in-time compiler of the used jruby interpreter. This manner the true distinction between create and construct is extra seen. So switching from .create to .construct saves about 45% of the execution time.

Completed in 1 minute 1.61 seconds (information took 23.Four seconds to load)
100 examples, zero failures
Completed in 34.87 seconds (information took 21.69 seconds to load)
100 examples, zero failures

Abstract

So easy enhancements within the exams can result in a pleasant efficiency increase working them.

  • Keep away from it-chains if the exams correlate to one another
  • Keep away from let! in favor of let, and create the objects inside earlier than blocks when mandatory
  • Keep away from earlier than blocks creating lots of stuff which is probably not mandatory for all exams
  • Use FactoryBot.construct as a substitute of .create if relevant.

Regulate your test-suite and don’t hesitate to take away duplicate exams, perhaps already out of date exams. As (in our case) the exams are working earlier than each merge and on each commit, attempt to maintain your take a look at suite quick. 

Loading

***