In many of our gems, we have a need for Unit fixtures (parsing specifically). So, to make this easier, we have created some tools to help.
Method 1: using bin/create_fixture
If the required page does not require any specific Javascript functionality, meaning the page returned is the one we are processing, then we can use the bin/create_fixture that has been added to most gems in the bin folder. This tool uses CURL which much be present on the host machine to download the HTML file, and store it in the fixture/webmock folder.
To use it, issue the command bin/create_fixture with the argments:
wherehttps://audienti.com/ is the URL you want to retrieve, and audienti.mock is in the name of the file stored in the fixture/webmock folder.
Method 2: Using Splash Server
In cases where the page has Javascript that renders, and this rendering is what causes the final page to be delivered, you will need to use the Splash servers to retrieve the raw HTML and generate the mock file yourself.
To do this, login to the web UI for Splash. You will need the username and password. From there, you place the URL you want to retrieve in the URL box, modify the script to the one that is listed below by cutting and pasting, and then hit the retrieve button.
function main(splash, args)
splash.set_user_agent('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/603.3.8 (KHTML, like Gecko)')
assert(splash:go(args.url))
assert(splash:wait(0.5))
return {
html = splash:html(),
png = splash:png(),
har = splash:har(),
}
end
Once you do this, you'll have the option to download the HTML. Do this, paste the HTML page into a file in the appropriate fixture folder, with the correct name. Then, IF the fixture requires stubbing to a request, add to the top of this page the following code snippet to set the HTML retriever part of the fixture.
This will setup the mock just like the bin/create_fixture does.
File format
If you look at this file, it will have two primary "parts". The first part is the HTTP response codes, and headers that are returned. After this part, you will have the HTML returned. The beginning of a mock looks like this:
HTTP/1.1 200 OK
Date: Thu, 30 Nov 2017 16:11:56 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 104578
Connection: keep-alive
Server: nginx
Vary: Accept-Encoding
set-cookie: aboutme_anon_id=24cf4d94-50a4-4e40-a79d-bd93e66f1c94; Max-Age=31536000; Path=/; Expires=Fri, 30 Nov 2018 16:11:56 GMT; HttpOnly
set-cookie: pumpkinhead=63ed5bc54050134a024301efd189324b830f3c91xOFDg8wYtxu-qkfJ9tA9xaUxtwDL0GC0; Path=/
X-Frame-Options: DENY
ETag: W/"19882-1yZrFy6vZVQGPX9efbV6Dwn9fvo"
<!doctype html><html><head><title>Chris Westfall - Houston, Texas, President and CEO, Westfall and Associates LLC, Author & Speaker, Texas Christian University, Southern Methodist University | about.me</title><meta name="google-signin-client_id" content="828605907544-2lag7tdq7k3ebk4pvjiu45vdpggiu2k0.apps.googleusercontent.com"/><meta name="HandheldFriendly" content="True"/><meta name="MobileOptimized" content="320"/><meta name="viewport" content="width=device-width, minimum-scale=1"/><meta http-equiv="cleartype" content="on"/><meta name="msapplication-TileColor" content="#329c85"/><meta name="msapplication-TileImage" content="https://cdn.about.me/images/icons/favicon/favicon-144-MS.png"/><meta http-equiv="content-type" content="text/html; charset=UTF-8"/><meta name="robots" content="index"/><meta name="description" content="I am a Keynote Speaker, writer, and Entrepreneur in Houston, Texas. Watch my videos."/><meta name="keywords" content="keynote speaker, marketing, travel, arts, entrepreneurship"/><meta name="author" content="Chris Westfall"/><meta property="og:site_name" content="about.me"/><meta property="og:locale" content="en_US"/><meta property="og:url" content="https://about.me/chriswestfall"/><meta property="og:title" content="Chris Westfall on about.me"/><meta property="og:description" content="I am a Keynote Speaker, writer, and Entrepreneur in Houston, Texas. Watch my videos."/><meta property="og:image" con
Stubbing/mocking in RSpec
From there, you will need to stub/mock in RSpec. The primary way this happens, is via the stub with mock method that is defined in the support folder. This method takes a matcher as a regular expression, a filename (from the fixture folder as the root), and a method to match which defaults to :any, and will return this document and response code of this file.
Using the stubber is easy, you simply do the following:
context '#get' do
before do
stub_with_mock(/audienti/, 'webmock/audienti_home.mock')
stub_with_mock(/twitter/, 'page_parsing/twitter_iruka223.mock')
end
# YOUR TESTS HERE
end
