10. Gzip Compression: Empower Gzip compression with your server to reduce the measurement of data files transferred amongst the server and the consumer’s browser. txt file is then parsed and will instruct the robotic regarding which web pages will not be being crawled. As being a internet search engine crawler https://www.youtube.com/watch?v=CWAXJPBjGjE