???????????? ????? ??????????, ??? ??????? ??? ?????????? ?? ?? ????????? ??? ???? ?????????????? ?? ???? ??????????.Create robots.txt file to authenticate and cease crawler from crawling something in addition to your own home web site.????????????? ??? ???? ???? ????????? ??????????? ?????????? ?????. ?????? ? ????????? ??? ???????????? ?? ???????