The Undocumented Web: scraping, private APIs, proxies and “alternative solutions”

Syntax - Tasty Web Development Treats

• Episode: The Undocumented Web: scraping, private APIs, proxies and “alternative solutions”
• Website: Syntax - Tasty Web Development Treats
• Feed URL: https://feed.syntax.fm/rss
• Duration: 01:04:49
• Published: 2018-07-25 15:00

What is the undocumented web? Scott and Wes dive into it, discussing APIs, faking, scraping, automation, proxies as well as tips and tricks for best practices.

Kyle Prinsloo’s Freelancing & Beyond — Sponsor

Kyle Prinsloo teaches you everything you need to know about freelancing, including how to quit your job, earn a side-income and start taking control of your life. Check it out at https://studywebdevelopment.com/freelancing and use the coupon code “syntax” at checkout to get 25% off.

Freshbooks - Sponsor

Get a 30 day free trial of Freshbooks at freshbooks.com/syntax and put SYNTAX in the “How did you hear about us?” section.

Show Notes

3:05 - What Is the undocumented web?

• When traditional methods don’t give you what you want
• Wild West

6:10 - Undocumented APIS

• Instagram’s private API
• Use dev tools’ XHR Request
• React Dev Tools will also have the data
• Ep 008 - Wes Bos Origin Story
• Realtor.ca - MLS Mapping
• Simon Willison’s Zeit Day Talk about XML files
• Charles Proxy App - also good for debugging requests

17:34 - Faking

• Faking User Agents
  • Sending headers from requests
  • Free Blackberry wifi on plane

23:52 - Scraping

• What is craping?
• Cheerio - jQuery like node package

27:30 - Automation

• Cron Jobs
• Filling out forms instantly
• Headless browsers
  • Puppeteer
  • PhantomJS

33:30 - Proxies

• Need a fresh IP address
• Need a different country
• Side note: Canadians buying stuff in USA M4S1P1 === 41100
• API throttling

39:00 - Tips and Tricks

• Multiple API keys
• diskDB - Text files as databases
• Search Github for API keys
• Inspect Element and Delete Overlays
• Delete Cookiesokay

44:45 - Stories

• Photo Voting
• MLS Listing
• Top Entrepreneur voting cheating
• Vanity Plate Checker
• My Hockey Team Sucks
• Vehicle auction bidding app
• youtube-dl

××× SIIIIICK ××× PIIIICKS ×××

• Scott: Octopath Traveler & OST
• Wes 1: AvE Youtube Channel
• Wes 2: Mustie1 Youtube Channel

Shameless Plugs

• Scott’s Level Up Tuts Pro
• Wes’ Courses

Tweet us your tasty treats!

• Scott’s Instagram
• LevelUpTutorials Instagram
• Wes’ Instagram
• Wes’ Twitter
• Wes’ Facebook
• Scott’s Twitter
• Make sure to include @SyntaxFM in your tweets