Other things to try:

* Search www.gradx.net:

Below you will find everything on gradx.net referencing the keyword: robots.txt



Bursaries |  Choose a career |  Mock Interview |  Ask the Experts |  Professional organisations  |  Recruiters |  Travel Campus Career Centres |  List of SETAs