Discuss other engines here. AllTheWeb, AltaVista, Excite, Lycos, etc...
10 posts • Page 1 of 1
- Posts: 40
- Joined: Wed Jan 30, 2019 10:05 pm
Robots.txt max size is 500KB. As declared on Google's Robots.txt Specifications page, “A maximum file size could also be enforced per crawler. Content that is after the maximum file size is also ignored. Google presently enforces a size limit of 500 kilobytes (KB).
- Posts: 5
- Joined: Fri Apr 05, 2019 5:01 am
Users browsing this forum: No registered users and 2 guests