Optimized Optimized Web Crawling

Linear Digressions

• Episode: Optimized Optimized Web Crawling
• Website: Linear Digressions
• Feed URL: http://feeds.feedburner.com/udacity-linear-digressions?format=xml
• Duration: 00:19:42
• Published: 2018-11-04 22:38

Last week’s episode, about methods for optimized web crawling logic, left off on a bit of a cliffhanger: the data scientists had found a solution to the problem, but it wasn’t something that the engineers (who own the search codebase, remember) liked very much. It was black-boxy, hard to parallelize, and introduced a lot of complexity to their code. This episode takes a second crack, where we formulate the problem a little differently and end up with a different, arguably more elegant solution. Relevant links: http://www.unofficialgoogledatascience.com/2018/07/by-bill-richoux-critical-decisions-are.html http://www.csc.kth.se/utbildning/kth/kurser/DD3364/Lectures/KKT.pdf