Discuss other engines here. AllTheWeb, AltaVista, Excite, Lycos, etc...
11 posts • Page 1 of 1
- Posts: 40
- Joined: Wed Jan 30, 2019 10:05 pm
Robots.txt max size is 500KB. As declared on Google's Robots.txt Specifications page, “A maximum file size could also be enforced per crawler. Content that is after the maximum file size is also ignored. Google presently enforces a size limit of 500 kilobytes (KB).
- Posts: 5
- Joined: Fri Apr 05, 2019 5:01 am
- Posts: 73
- Joined: Mon Apr 19, 2021 9:29 pm
Content which is after the maximum file size is ignored. Google currently enforces a size limit of 500 kibibytes (KiB). To reduce the size of the robots. txt file, consolidate directives that would result in an oversized robots.
Users browsing this forum: No registered users and 4 guests