That’s a perfect example of the use of the robot.txt file and I appreciate that now, do I not only know the purpose of it, I know how and when to use it.
I can also choose not to use it and if I do enable the setting, I’ll still need to add some configurations to get it working properly.