Forgot your password?
typodupeerror
The Internet

Bots Now Account For 61% of Net Traffic 124

Posted by samzenpus
from the that-which-has-no-life dept.
codeusirae writes "A study by Incapsula suggests 61.5% of all website traffic is now generated by bots. The security firm said that was a 21% rise on last year's figure of 51%. From the article: 'Some of these automated software tools are malicious - stealing data or posting ads for scams in comment sections. But the firm said the biggest growth in traffic was for 'good' bots. These are tools used by search engines to crawl websites in order to index their content, by analytics companies to provide feedback about how a site is performing, and by others to carry out other specific tasks - such as helping the Internet Archive preserve content before it is deleted.'"
This discussion has been archived. No new comments can be posted.

Bots Now Account For 61% of Net Traffic

Comments Filter:
  • Youtube? (Score:5, Interesting)

    by Anonymous Coward on Thursday December 12, 2013 @09:23PM (#45676439)

    Didn't we just get studies that said youtube and netflix were 50% of the net's traffic?

    http://mashable.com/2013/11/12/internet-traffic-downstream/

    Was this just a ruse? Is this study wrong? Is there some sort of overlap?

  • by ls671 (1122017) on Thursday December 12, 2013 @11:29PM (#45677077) Homepage

    Well not on my sites.

    Ok, they still hit me but this is minimal traffic since I do not reply.

    1) Have iptables log and automatically bar offenders not on whitelisted countries.
    2) Use mod_security and do the same for web traffic.
    3) Bar the rest manually to avoid barring myself or my customers... (about 20-40 a day)

    It has become a pain but what else could you do?

    Numbers of IPs currently barred (use ipsets !!!!):
    $ grep -c . /etc/rc.d/badiptobar
    4667

    Block user agents:
    SecRule REQUEST_HEADERS:User-Agent \
    "@pm AhrefsBot Ezooms Aboundex 360Spider Mail.RU_Bot crawler.sistrix.net \
      SemrushBot SurveyBot Netseer panscient.com ADmantX ZumBot BLEXBot UnisterBot \
      seoprofiler EasouSpider" \
    "id:'12050',\
    phase:1,nolog,deny"

    SecRule REQUEST_HEADERS:User-Agent \
    "@pmFromFile /etc/httpd/extra/sec-blacklist-barip-user-agent" \
    "id:'12051',\
    phase:1,nolog,deny,exec:/usr/local/bin/modsecwritebadiptobartofile"

    Bar them automatically if not from whitelisted countries and if on any blacklist:
    SecRule GEO:COUNTRY_CODE \
    "@pm CA FR BE US CH GB AU IL NO NZ" \
    "id:'10501', \
    phase:1,nolog,pass,skipAfter:END_RBL"

    SecRule IP:PREVIOUS_RBL_CHECK "@eq 1" "phase:1,id:'11000',t:none,pass,nolog,\
    skipAfter:END_RBL_LOOKUP"

    SecRule REMOTE_ADDR "@rbl sbl-xbl.spamhaus.org" "id:'11010', \
    phase:1,nolog,deny,msg:\
    'IP address that has abusable vulnerabilities: sbl-xbl.spamhaus.org:\
      %{request_headers.user-agent}',\
      setvar:ip.spammer=1,expirevar:ip.spammer=7200,setvar:ip.previous_rbl_check=1,\
      expirevar:ip.previous_rbl_check=7200,exec:/usr/local/bin/modsecwritebadiptobartofile"

    SecRule REMOTE_ADDR "@rbl bl.blocklist.de" "id:'11011', \
    phase:1,nolog,deny,msg:\
    'IP address that has abusable vulnerabilities: bl.blocklist.de:\
      %{request_headers.user-agent}'\
      setvar:ip.spammer=1,expirevar:ip.spammer=7200,setvar:ip.previous_rbl_check=1,\
      expirevar:ip.previous_rbl_check=7200,exec:/usr/local/bin/modsecwritebadiptobartofile"

    etc. etc. etc. etc. etc.

    Have iptables log and bar offenders if not on whitelisted country

    # cat baripifex
    #!/bin/sh

    IP=${1}
    COUNTRY=`su tester -c "/usr/local/bin/geoiplookup ${IP}"`
    ###echo $COUNTRY
    ###echo $RBLCHECK

    WHITE_LISTED_COUNTRY=false

    for WHITE_COUNTRY in CA FR BE US CH GB AU IL NO NZ IP
    do
    WHITE_LISTED_COUNTRY=${WHITE_LISTED_COUNTRY}`echo -n $COUNTRY | grep -i $WHITE_COUNTRY`
    done

    if [ "$WHITE_LISTED_COUNTRY" = "false" ]
    then /home/ls/pub/mybin/baripnoout $IP $COUNTRY baripifex
    echo -n barred
    else
    echo -n noaction
    fi

    etc. etc. etc. etc. etc.

"Text processing has made it possible to right-justify any idea, even one which cannot be justified on any other grounds." -- J. Finnegan, USC.

Working...