Although I mentioned thechive.com I spend most of my time on the net looking for information, not entertainment. These ads really hinder my search speed!
So here is a quick way you can block all the ads. Not only will your surfing be faster but you will also save some bandwidth.
First off I would like to thank the fine folks at http://winhelp2002.mvps.org/ for doing all the leg work and collecting all the data necessary for this to work.
Now this is simple.
First, let's make a copy of your current hosts file. Some people still use this!
NOTE: Everything done below should be done as root, or with sudo access.
Here we will make a copy of your /etc/hosts file and save it in your home directory in a hidden file called .etchosts , but you can save it anywhere you like. Just be sure to change the script to make the location of your file.
cp /etc/hosts ~/.etchosts
Now we will make the shell script.run:
vi /root/update_hosts.sh
Fill the file with the following:
#!/bin/bash
cd /tmp
wget http://winhelp2002.mvps.org/hosts.txt
rm /etc/hosts
mv hosts.txt /etc/hosts
cat ~/.etchosts >> /etc/hosts
Now we have to make sure the script is executable:
chmod +x update_hosts.sh
So just a little explanation of what the script does, it's simple.1) cd /tmp (Changes to a temporary working directory)
2) wget http://winhelp2002.mvps.org/hosts.txt (Gets the hosts.txt file from mvps.org and saves it.
3) rm /etc/hosts (Deletes the current /etc/hosts file)
4) mv hosts.txt /etc/hosts (Moves the new downloaded file to the /etc/hosts file)
5) cat ~/.etchosts >> /etc/hosts (Moves your old host entries back into the new file)
Now all you have to do is run the script and your advertisements will disappear from your browser.
You can use a cron job to update the hosts file automatically every night. The good folks at mvps.org update the file fairly regularly, not everyday, but a couple times a month.
Let's add it to roots cron.
run:
crontab -e
Then enter the following in the file.
59 23 * * * /root/update_hosts.sh
Remember, if you need to add something to your hosts file you now want to add it to ~/.etchosts and then run the script to update /etc/hosts.
REFERENCE: http://winhelp2002.mvps.org/hosts.htm
Right up to running crontab, none of that should be done as root. You can do the earlier bit as a normal user. I'd recommend nano or jed rather than vi, much easier to learn.
ReplyDeletebut basically, you don't have anything useful to add to the mans script, do you?
DeleteEncouraging users to eschew vi for lame editors like nano or jed (jed?) is a critical mistake. Failing to learn ex/vi may someday leave you stranded with an unsolvable problem if your system becomes hosed and editing a file is the only solution, but that system only has ex/vi installed. vi is the preeminent editor on ALL unix systems. But then, you really had nothing USEFUL to add to the mans script, did you?
DeleteI don't think I've ever seen one that has not nano as default. Whenever that's the case/there's no nano, then, well, one who does not know "vi" may have to use some other PC and learn the basics/make some cheatsheet in a piece of paper and then go back to the troubled machine.
DeleteIt's perhaps not the ideal scenario, but it's not like encouraging people to walk on African jungles without knowing how to shoot a rifle. Most of the time you can go by perfectly fine without ever learning vi, somewhat like you don't need to be a mechanic to have a car, and even to know how to do some basic maintenance.
You can do just about anything as root, as long as you're not doing anything stupid. I'd argue that's perhaps more risky to have a "sudo" than do everything (specially withoug password) than to use just "su". But I don't know for sure, I may be missing something. To me it seems that Ubuntu-style omnipotent sudo is more about convenience than safety. With some distros' original sudo config you wont even be able to do barely anything with just sudo by default, su is required, at very least to define what each user can sudo.
DeleteIf you haven't used Linux or Unix lont enough to have learned vi, then you really have no business administrating such a machine.
Delete^ well said Akulkis.
DeletePersonal user preference and choice don't matter, huh?
DeleteI thought that was the whole point of GNU/Linux.
But what do I know, I use Nano.
^Akulkis
DeleteThat's just ignorant. The whole reason why alternatives like nano were developed in the first place, and added to the repos *everywhere*, was because it served the needs and wants of far more people than just those developers. Nano is a mature program that does what the users want, and what vi cannot.
The fact it's not core is either bigotry, or just simple numbers. In places where nano isn't available, you can get it.
For the record, I hate vi with a passion that cannot be described in the human language. Been around for decades and still haven't learned it, and administrate such machines daily. My lack of vi skills is wholly intended, as I availed myself of the repos and installed nano. Just as it was designed to do.
People that enjoy learning 32 trillion commands to enter into what *appears* to be an editor isn't evidence of superiority, but sadomasochism. Must be a fallout from the regex community :)
Perhaps we nano users are just the new Luddites, but I don't need to remember 32 trillion commands just to type normally. vi starts out and you can't just type into naturally right away? Insta-fail, and about the weirdest thing I've seen. Most of the time I don't complain, because there were others like me and we just use nano instead of saying that all vi users are idiots that have no business administrating Linux machines.
Lastly, for the record, it's a bad idea from a security perspective to update it nightly as suggested, in ROOT no less! It's not HTTPS, and it doesn't have a signature you could awk yourself and verify. Doing such a thing nightly just makes DNS poisoining your network and machine much easier.
If they want to do that, theire daily updates need to be signed.
sudo su
ReplyDeletecat /etc/hosts >/etc/hosts.orig
cat >/usr/local/bin/update_hosts.sh </etc/hosts
cat /etc/hosts.orig >>/etc/hosts
EOF
chmod +x /usr/local/bin/update_hosts.sh
crontab -e
59 23 * * * /usr/local/bin/update_hosts.sh
exit
Just a note of thanks for creating such clear, well written, helpful information. I've successfully implemented your instructions on my Centos 6.2 distro. Excellent results!
ReplyDeleteSorry, how does this vi thing work.
ReplyDeleteHaving trouble here
vi is a command line text editor. You can also use gedit or any text editor you like. If you want to learn vi, read this:
ReplyDeletehttp://acms.ucsd.edu/info/vi_tutorial.html
KWrite is a popular text Editor, on KDE, and many Systems.
ReplyDeleteYou could also use ABP on firefox or the same on Chromium.
ReplyDeleteRegards.
You shouldn't remove your own/default /etc/hosts
ReplyDeleteIt should suffice to:
wget http://winhelp2002.mvps.org/hosts.txt
cat hotst.txt >> /etc/hosts
rm hosts.txt
">>" will only add to the extant file
Then you edit your hosts file and remove the redundant lines, such as
127.0.0.1 localhost
I don't know if it matters so much that there are those redundancies though.
By not deleting your own hosts you won't lose some lines that may be setting the domain and whatnot. You might have lines somewhat like:
127.0.1.1 debian-joejoejoe.WORKGROUP debian-joejoejoe
Whatever it does, I'm not an expert on that.
Those would have been erased in the other method, and things like samba or network folders might not work I guess.
But if you're going to do that anyway, you only need to
rm /etc/hosts/
wget -O /etc/hosts
no need to move and rename everything all around the place.
If you want to make a script that does that periodically, you may want to consider to have a "hosts.header" (you name it) file with your current/default hosts file, and then have a script (called by cron) that will do something along the lines of
wget -O /tmp/newhosts (or /dev/shm, wherever works)
cat /etc/hosts.header > /etc/hosts (note the single, not double, ">", it erases the old file, so no "rm" is needed)
tail -n +26 /tmp/newhosts >> /etc/hosts (now it's a double, ">>", as it's only adding new stuff to the end of the file)
"tail -n +26" will trim cut out the 26 first lines of the downloaded hosts file, wich will be added to your own custom "header" via ">>".
Always backup the relevant files before testing new things.
Hey buddy, you can remove redundant lines by running
Deleteawk '!x[$0]++' /tmp/newhosts /etc/hosts
I also recommend not only to use 1 hosts provider, but more and remove all redundant lines.
Well, I also don't recommend to cut "X" lines, cause it may vary from distro to distro...
If you wanna make it run periodically, just add on boot or make a .service
After following the manual, I see a ^M after every entry in /etc/hosts, like,
ReplyDelete127.0.0.1 ads.doubleclick.com^M
Just wanted to know if its normal. Its working perfectly though.
@Anonymous - usually this occurs when the file was created on Windows. You can remove the ^M by using dos2unix.
ReplyDeletehttp://linux.about.com/od/commands/l/blcmdl1_dos2uni.htm
Thank you for the instructions. Worked well on my Linuxmint 16.
ReplyDeleteThis is my solution, based on what I've seen in this thread. These instruction work in Ubuntu from a command prompt.
ReplyDelete1) Create a copy of your original hosts file to use as a header:
sudo cp /etc/hosts /etc/hosts.header
2) Create the script to update the hosts file:
nano update_hosts.sh
3) Enter the following text in nano:
#!/bin/bash
#Change to a temprary working directory
cd /tmp
#Download the current hosts.txt file
wget -q http://winhelp2002.mvps.org/hosts.txt
#Overwrite the hosts file with the hosts header
cat /etc/hosts.header > /etc/hosts
#Append the hosts.txt data to the hosts file
tail -n +26 /tmp/hosts.txt >> /etc/hosts
#Delete hosts.txt
rm /tmp/hosts.txt
#Exit the script
exit
4) Save the file with CTRL-X in nano
5) Change ownership of the file to root and make the file executable:
sudo chown root:root /path/to/update_hosts.sh
sudo chmod 700 /path/to/update_hosts.sh
6) Ad an exception for update_hosts.sh to sudoers
sudo visudo
7 Below the line that reads '%sudo ALL=(ALL:ALL) ALL', enter the following:
user ALL=(ALL) NOPASSWD: /path/to/update_hosts.sh
*Replace 'user' with your username
8) You should now be able to execute the script without a sudo password by entering the following:
sudo ./path/to/update_hosts.sh
I decided to write a little script to update the hosts file with new changes and preserve any custom hosts entries:
ReplyDelete#!/bin/bash
if [ "$1" == "--update" ] || [ "$1" == "-u" ] || [ "$1" == "-U" ]
then
echo ""
echo "Updating hosts file with changes from /etc/hosts.orig"
echo ""
echo "Backing up original hosts file"
cp /etc/hosts /etc/hosts.bak
Done="False"
while [ "$Done" == "False" ]
do
echo ""
echo "#Don't edit this file directly instead edit /etc/hosts.orig and run hostsupdate --update"
echo ""
cat /etc/hosts.orig
cat /etc/hosts.txt
Done="True"
done > /tmp/finalHosts.txt
echo "Deleting hosts"
rm -rf /etc/hosts
echo "Hosts deleted"
echo ""
echo "Moving new hosts file"
mv /tmp/finalHosts.txt /etc/hosts
echo "Done!"
echo ""
echo "Your hosts file has been successfully updated"
echo ""
echo "If things go wrong please restore your original hosts file from /etc/hosts.bak"
echo ""
exit
fi
wget http://winhelp2002.mvps.org/hosts.txt -P /tmp -N
CurrentHosts=$(/usr/bin/md5sum --tag /etc/hosts.txt)
CurrentHosts="${CurrentHosts##* }"
echo "Current Hosts File:"
echo "$CurrentHosts"
echo ""
echo "New Hosts File"
NewHosts=$(/usr/bin/md5sum --tag /tmp/hosts.txt)
NewHosts="${NewHosts##* }"
echo "$NewHosts"
if [ "$CurrentHosts" != "$NewHosts" ]
then
echo ""
echo "DIFFERENT"
rm -rf /etc/hosts.txt
cp /tmp/hosts.txt /etc/hosts.txt
echo "Backing up original hosts file"
cp /etc/hosts /etc/hosts.bak
echo ""
echo "Hosts file backed up at /etc/hosts.bak"
Done="False"
while [ "$Done" == "False" ]
do
echo ""
echo "#Don't edit this file directly instead edit /etc/hosts.orig and run hostsupdate --update"
echo ""
cat /etc/hosts.orig
cat /tmp/hosts.txt
Done="True"
done > /tmp/finalHosts.txt
echo "Deleting hosts"
rm -rf /etc/hosts
echo "Hosts deleted"
echo ""
echo "Moving new hosts file"
mv /tmp/finalHosts.txt /etc/hosts
echo "Done!"
echo ""
echo "Your hosts file has been successfully updated"
echo ""
echo "If things go wrong please restore your original hosts file from /etc/hosts.bak"
echo ""
else
echo ""
echo "SAME"
rm -rf /tmp/hosts.txt
echo ""
fi
It basically stores the latest adblocking hosts file at /etc/hosts.txt and downloads the latest file to /tmp. The md5sum is checked and if they are the same nothing happens. If they aren't the same then any hosts entries in /etc/hosts.orig are merged with the latest adblocking hosts files allowing for custom entries to be stored in /etc/hosts.orig and be saved everytime there is an update to the adblocking hosts file.
Wow no verification of certificate and that winhelp2002.mvps.org resolves to the correct server?
ReplyDeletewget http://winhelp2002.mvps.org/hosts.txt
should be
wget https://winhelp2002.mvps.org/hosts.txt
but guess the site doesn't have it on SSL
An that to dl the file that overwrites your DNS server...
Don't be surprised if your bank site starts having certificate errors.
The above comments makes half sense. You are correct, it would be much safer with an SSL connection, but it is not offered. Other than that I am not sure what your talking about. There is no overwriting a DNS server, this is just adding static entries to your hosts file. Also it has no effect on banks or certificates.
DeleteAnother Anonymous here:
DeleteYou clearly don't understand what the person above is talking about. It is trivial to redirect a user to a bogus banking site. This is also not just adding static entries as you put it but replacing the original hosts file. It is also potentially dangerous and a violation of any reasonable security policy to download and execute scripts from random sources as root.
I agree again... We are not replacing the original hosts file, but appending to it. Either way you are correct, there is a chance that someone could feed a nasty file into mvps.org site and redirect your traffic. I also agree, that is is potentially dangerous to execute scripts downloaded from random sources as root. I would suggest people read any scripts, understand what each line is doing and use it only if they feel comfortable.
DeleteUpdate to - Sean DavisSeptember 14, 2013 at 4:23 PM post
ReplyDeleteSean im new to Linux and I made a change see below:
FROM - sudo ./path/to/update_hosts.sh
TO - sudo /path/to/update_hosts.sh
I have just removed the full stop after sudo and it works, such a great feeling.
Russ
"I notice that a lot of sites load rather slowly mostly because your waiting on content from outside the specific domain."
ReplyDeleteTypo: needs to be you're, instead of your.
The hijacking can be easily netered by forcefully rewriting the IP at the beginning of each line to 0.0.0.0
ReplyDeleteEven better: refuse accepting the script if it contains anything else than 0.0.0.0
A potential hijacker trying to subvert a legitimate domain will only trigger a warning and prevent the update.
How about:
ReplyDelete#!/bin/bash
cd /tmp
wget http://winhelp2002.mvps.org/hosts.txt
cat /tmp/hosts.txt >> /etc/hosts
AnonymousJune 16, 2014 at 4:12 PM
ReplyDelete"Wow no verification of certificate and that winhelp2002.mvps.org resolves to the correct server?"
He makes an excellent point.
Therefore this script needs to and can do more. It needs to read the first two characters of each line and accept only the following:
"# "
"#*"
"0."
"" (blank line)
"::" (In this case, it then needs to read the whole line and only accept "::1 localhost #[IPv6]")
"12" (In this case, it then needs to read the whole line and only accept "127.0.0.1 localhost")
ONLY if this validation passes should the file contents by inserted into your hosts file.
There are many ways to skin a cat, but I will leave it for someone else to write the MOST EFFICIENT method of doing this, considering there are over 15,000 lines to process!
BTW, I only use this hosts file on my RT-AC66U router, and then it blocks ads for all devices on the network, including Windows 7, but doesn't work for a WinXP test machine. I can't see anything different in IPCONFIG /ALL.
Any suggestions?
Are you sure your XP machine is using your router as it's DNS server, and not something else?
DeleteI am no programmer but for verifying that there are only 0. host adresses in the file, wouldn't it be easier to only check that there are no lines beginning by a digit between 1 and 9?
ReplyDeleteSomething like this would work to test the integrity of the downloaded file:
ReplyDelete# Download file
wget -q -O /tmp/hosts.new $remote_hosts_file
# Remove commented lines
egrep -v '^#' /tmp/hosts.new > /tmp/hosts.tmp
# Remove lines before hosts entries
sed -n -i '/0\.0\.0\.0 /,$p' /tmp/hosts.tmp
# Perform malice check to see if anything points to anything other than 0.0.0.0
if [[ $(awk '{print $1}' /tmp/hosts.tmp | uniq) != '0.0.0.0' ]]; then
printf "File has been tampered with! Exiting!\n"
exit 1
fi
Entire script I use at: https://gist.github.com/cptskyhawk/94dbc83aefdab631b37d
If you use the original verbatim script at the top, the "invisible" file, .etchosts, contains the text of the original or previous /etc/hosts file. This .etchosts file gets concatenated to the hosts file, effectively doubling the size of the /etc/hosts file every time the script is run successfully. This last line will use up disk storage very quickly and eventually die when it can no longer write the hosts file. This line should be removed from the script:
ReplyDeletecat ~/.etchosts >> /etc/hosts
remove the above line.
Hey, nice post... But I'd like to improve your script, if you don't mind...
ReplyDeleteAlso, you should prevent detecing ad block with pinging the ads server, so it would be pretty to run a local server in 0.0.0.0:8080 (if this doesn't present any security risk).
For those that like simplicity here is one line that can be added in root cron:
ReplyDelete@montly wget -O- http://winhelp2002.mvps.org/hosts.txt > /etc/host
Something that worked for me pretty well. Basically I took some of teh ideas here and threw them all together in a script.
ReplyDeleteTo make this script work, it is important that the beginn of the "Ad Block Section" is marked with "### START ADBLOCK ###" and your hosts file begins with "# /etc/hosts". These two lines are the marker for your regular hosts content. everything below "### START ADBLOCK ###" is to be replaced with every update.
So basically your hosts file should look something like this:
# /etc/hosts: Local Host Database
# IPv4 and IPv6 localhost aliases
127.0.0.1 localhost
::1 localhost
### START ADBLOCK ###
And here is the script I came up with. It does nothing else than just taking teh part between the 2 markers, copies that into another file and merges the updates into the new file and writes all back replacing the hosts file.
#!/bin/bash
cd /root
cat /etc/hosts | sed -n "/# \/etc\/hosts/,/### START ADBLOCK ###/p" > etc_hosts
wget http://winhelp2002.mvps.org/hosts.txt
# Fix line endings
sed -i -e 's/\r//g' hosts.txt
# Remove commented lines
egrep -v '^#' hosts.txt > hosts.tmp
# Remove lines before hosts entries
sed -n -i '/0\.0\.0\.0 /,$p' hosts.tmp
# Perform malice check to see if anything points to anything other than 0.0.0.0
if [[ $(awk '{print $1}' hosts.tmp | uniq) != '0.0.0.0' ]]; then
printf "File has been tampered with! Exiting!\n"
exit 1
fi
cp etc_hosts hosts_without_adblock
cat hosts.tmp >> etc_hosts
cat etc_hosts > /etc/hosts
rm hosts.txt
rm hosts.tmp
@Anonymous October 28, 2015 at 9:23 AM
ReplyDeleteThat's not correct.
The .etchosts file contains your personal/manual entries you want in the /etc/hosts file. If you use the method/script by the OP, you should not edit the /etc/hosts file anymore but make all personal changes in the future in .etchosts. As every time your run the update script /etc/hosts will be created new:
1. you need the line you want to remove from the script (cat ~/.etchosts >> /etc/hosts) to get your personal/manual entries into /etc/hosts!
2. the size of the hosts file will always be: file size hosts.txt + file size .etchosts. It won't grow if there are now new entries in either of them, resp. if their size is not growing.
What distro are you using? Fedora does not recognize the ~/.etchosts file.
DeleteWhen does Fedora not recognize ~/.etchosts? What command(s) are you running? And from where (path)?
ReplyDeleteIf you ONLY have created the script update_hosts.sh and run it, no distro will recognize ~/.etchosts as it does not exist.
You need to create it via
cp /etc/hosts ~/.etchosts
This will create a copy of your current /etc/hosts in YOUR home directory. So, if YOU run the script it should find/use it :)
Hope that helps :)
PS: If that command line (cp /etc/hosts ~/.etchosts) were part of the script, THEN Anonymous would be correct assuming the file gets bigger and bigger. But it's only a one time action to create .etchosts as a copy of the ORIGINAL hosts file.
When does Fedora not recognize ~/.etchosts? What command(s) are you running? And from where (path)?
ReplyDeleteIf you ONLY have created the script update_hosts.sh and run it, no distro will recognize ~/.etchosts as it does not exist.
You need to create it via
cp /etc/hosts ~/.etchosts
This will create a copy of your current /etc/hosts in YOUR home directory. So, if YOU run the script it should find/use it :)
Hope that helps :)
PS: If that command line (cp /etc/hosts ~/.etchosts) were part of the script, THEN Anonymous would be correct assuming the file gets bigger and bigger. But it's only a one time action to create .etchosts as a copy of the ORIGINAL hosts file.
The reason people are having trouble with ~/.etchosts is because cron is run as root, and for the root user, ~ is /root, not /home/yourusername.
ReplyDeleteAlso, automatically appending files from your home directory to your hosts file is a big security concern, but I'm guessing most people don't care.
I'm using this in Gentoo Linux. I put my original hosts file at:
ReplyDelete/etc/conf.d/hosts_local
I had to merge dos2unix. I wanted the local info listed first. This is my current cron.monthly script:
#!/bin/bash
cd /tmp
mv /etc/hosts hosts.old
cp /etc/conf.d/hosts_local /etc/hosts
wget http://winhelp2002.mvps.org/hosts.txt
dos2unix -- hosts.txt
cat hosts.txt >> /etc/hosts