Testing network connectivity

Posted by javifs on Tue 26 Apr 2005 at 14:20

Tags:

There are many situations where you install (or make changes to) a system and you need to troubleshoot what on earth has broken when you no longer have proper network connectivity.

If you are in this situation you typically look at your network interfaces, test DNS resolution, ping a few systems and see where the problem lies. This usually implies having some knowledge of Linux networking and having the concepts clear. Even though documentation isout there to help the newbie and sysadmin alike (like the HOWTO available at Ubuntu's Forum) and there are even books out there (like Network Troubleshooting Tools, which goes beyond what an average joe user needs) the tasks that need to be done are tedious and repetitive.

So, how about doing this in a script?

#!/bin/sh
# Network testing script v 1.0
# (c) 2005 Javier Fernandez-Sanguino
#
# This script will test your system's network configuration using basic
# tests and providing both information (INFO messages), warnings (WARN)
# and possible errors (ERR messages) by checking:
# - Interface status
# - Availability of configured routers, including the default route
# - Proper host resolution, including DNS checks
# - Proper network connectivity (the remote host can be configured, see
#   below)
# 
# The script does not need special privileges to run as it does not 
# do any system change. It also will not fix the errors by itself.
#
#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation; either version 2 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program; if not, write to the Free Software
#   Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
#  
# You can also find a copy of the GNU General Public License at
# http://www.gnu.org/licenses/licenses.html#TOCLGPL
#
# TODO
# - Works only on Linux, can this be generalised for other UNIX systems
#   (probably not unless rewritten in C)
# - Does not check for errors properly, use -e and test intensively
#   so that expected errors are trapped
#   (specially for tools that are not available, like netcat)
# - If the tools are localised to languages != english the script might 
#   break 
# - Ask 'host' maintainer to implement error codes as done with 
#   dlint
# - Should be able to check if DNS server is in the same network, if 
#   it doesn't answer to pings, check ARP in that case.
# - DHCP checks?
# - Other internal services tests? (LDAP if using pam...)
# - Generate summary of errors in the end (pretty report?)
# - Check if packets are being dropped by local firewall? (use dmesg
#   and look for our tests)
# - Support wireless interfaces? (use iwconfig)
# - Check other TODOs inline in the code


# BEGIN configuration
# Configure to your needs, these values will be used when
# checking DNS and Internet connectivity
# DNS name to resolve
CHECK_HOST=www.debian.org
CHECK_IP_ADRESS=194.109.137.218
# Web server to check for
CHECK_WEB_HOST=www.debian.org
CHECK_WEB_PORT=80
export CHECK_HOST CHECK_IP_ADRESS CHECK_WEB_HOST CHECK_WEB_PORT
# END configuration


# Extract the interface of our default route

defaultif=`netstat -nr |grep ^0.0.0.0 | awk '{print $8}' | head -1`
defaultroutes=`netstat -nr |grep ^0.0.0.0 | wc -l`
if [ -z "$defaultif" ] ; then
	defaultif=none
	echo "WARN: This system does not have a default route"
elif [ "$defaultroutes" -gt 1 ] ; then
	echo "WARN: This system has more than one default route"
else 
	echo "INFO: This system has exactly one default route"
fi



# Check loopback
check_local () {
# Is there a loopback interface? 
	if [ -n "`ip link show lo`" ] ; then
# OK, can we ping localhost
		if  ! check_host localhost 1; then
# Check 127.0.0.1  instead (not everybody uses this IP address however,
# although its the one commonly used)
			if  ! check_host 127.0.0.1 1; then
				echo "ERR: Cannot ping localhost (127.0.0.1), loopback is broken in this system"
			else
				echo "ERR: Localhost is not answering but 127.0.0.1, check /etc/hosts and verify localhost points to 127.0.0.1"
			fi
		else
		 echo "INFO: Loopback interface is working properly"
		fi
			
	else
		echo "ERR: There is no loopback interface in this system"
		status=1
	fi
	status=0
	return $status
}

# Check network interfaces
check_if () {
	ifname=$1
	status=0
	[ -z "$ifname" ] && return 1
# Find IP addresses for $ifname
	inetaddr=`ip addr show $ifname | grep inet | awk '{print $2}'`
	if [ -z "$inetaddr" ] ; then
		echo "WARN: The $ifname interface does not have an IP address assigned"
		status=1
	else
# TODO: WARN if more than 2 IP addresses?
		echo $inetaddr | while read ipaddr; do
			echo "INFO: The $ifname interface has IP address $ipaddr  assigned"
		done
	fi

# Lookup TX and RX statistics
# TODO: This is done using ifconfig but could use /proc/net/dev for
# more readibility or, better, 'netstat -i'
	txpkts=`ifconfig $ifname | awk '/RX packets/ { print $2 }' |sed 's/.*://'`
	rxpkts=`ifconfig $ifname | awk '/RX packets/ { print $2 }' |sed 's/.*://'`
	txerrors=`ifconfig $ifname | awk '/TX packets/ { print $3 }' |sed 's/.*://'`
	rxerrors=`ifconfig $ifname | awk '/RX packets/ { print $3 }' |sed 's/.*://'`
# TODO: Check also frames and collisions, to detect faulty cables
# or network devices (cheap hubs)
	if [ "$txpkts" -eq 0 ] && [ "$rxpkts" -eq 0 ] ; then
		echo "ERR: The $ifname interface has not tx or rx any packets. Link down?"
		status=1
	elif  [ "$txpkts" -eq 0 ]; then
		echo "WARN: The $ifname interface has not transmitted any packets."
	elif [ "$rxpkts" -eq 0 ] ; then
		echo "WARN: The $ifname interface has not received any packets."
	else
		echo "INFO: The $ifname interface has tx and rx  packets."
	fi
# TODO: It should be best if there was a comparison with tx/rx packets.
# a few errors are not uncommon if the card has been running for a long
# time. It would be better if a relative comparison was done (i.e.
# less than 1% ok, more than 20% warning, over 80% major issue, etc.)
	if [ "$txerrors" -ne 0 ]; then
		echo "WARN: The $ifname interface has tx errors."
	fi
	if [ "$rxerrors" -ne 0 ]; then
                echo "WARN: The $ifname interface has rx errors."
	fi
	return $status
}

check_netif () {
	status=0
	ip link show | egrep '^[[:digit:]]' |
	while read ifnumber ifname status extra; do
		ifname=`echo $ifname |sed -e 's/:$//'`
		if [ -z "`echo $status | grep UP\>`" ] ; then
			if  [ "$ifname" = "$defaultif" ] ; then
				echo "ERR: The $ifname interface that is associated with your defualt route is down!"
				status=1
			elif  [ "$ifname" = "lo"  ] ; then
				echo "ERR: Your lo inteface is down, this might cause issues with local applications (but not necessarily with network connectivity)"
			else
				echo "WARN: The $ifname interface is down"
			fi
		else
		# Check network routes associated with this interface
			echo "INFO: The $ifname interface is up"
			check_if $ifname
			check_netroute $ifname
		fi
	done
	return $status
}

check_netroute () {
	ifname=$1
	[ -z "$ifname" ] && return 1
	netstat -nr  | grep "${ifname}$" |
	while read network gw netmask flags mss window irtt iface; do
	# For each gw that is not the default one, ping it
		if [ "$gw" != "0.0.0.0" ] ; then
			if ! check_router $gw  ; then
				echo "ERR: The default route is not available since the default router is unreachable"
			fi
		fi
	done
}

check_router () {
# Checks if a router is up
	router=$1
	[ -z "$router" ] && return 1
	status=0
# First ping the router, if it does not answer then check arp tables and
# see if we have an arp. We use 5 packets since it is in our local network.
	ping -q -c 5 "$router" >/dev/null 2>&1 
	if [ "$?" -ne 0 ]; then
		echo "WARN: Router $router does not answer to ICMP pings"
# Router does not answer, check arp
		routerarp=`arp -n | grep "^$router" | grep -v incomplete`
		if [ -z "$routerarp" ] ; then
			echo "ERR: We cannot retrieve a MAC address for router $router"
			status=1
		fi
	fi
	if [ "$status" -eq 0 ] ; then
		echo "INFO: The router $router is reachable"
	fi
	return $status
}

check_host () {
# Check if a host is reachable
# TODO: 
# - if the host is in our local network (no route needs to be used) then
#   check ARP availability
# - if the host is not on our local network then check if we have a route
#   for it
	host=$1
	[ -z "$host" ] && return 1
# Use 10 packets as we expect this to be outside of our network
	COUNT=10
	[ -n "$2" ] && COUNT=$2
	status=0
	ping -q -c $COUNT "$host" >/dev/null 2>&1 
	if [ "$?" -ne 0 ]; then
		echo "WARN: Host $host does not answer to ICMP pings"
		status=1
	else
		echo "INFO: Host $host answers to ICMP pings"
	fi
	return $status
}

check_dns () {
# Check the nameservers defined in /etc/resolv.conf
	status=1
	nsfound=0
	nsok=0
	tempfile=`mktemp tmptestnet.XXXXXX` || { echo "ERR: Cannot create temporary file! Aborting! " >&2 ; exit 1; }
	trap " [ -f \"$tempfile\" ] && /bin/rm -f -- \"$tempfile\"" 0 1 2 3 13 15
	cat /etc/resolv.conf |grep nameserver | 
	awk '/nameserver/ { for (i=2;i<=NF;i++) {  print $i ; } }'  >$tempfile
	for nameserver in `cat $tempfile`;  do
		nsfound=$(( $nsfound + 1 ))
		echo "INFO: This system is configured to use nameserver $nameserver"
		check_host $nameserver 5
		if check_ns $nameserver ; then
			nsok=$(( $nsok +1 ))
		else
			status=$?
		fi
	done
	#Could also do:
	#nsfound=`wc -l $tempfile | awk '{print $1}'`
	/bin/rm -f -- "$tempfile"
	trap  0 1 2 3 13 15
	if [ "$nsfound" -eq 0 ] ; then
		echo "ERR: The system does not have any nameserver configured"	
	else
		if [ "$status" -ne 0 ] ; then
			if [ "$nsfound" -eq 1 ] ; then
				echo -e "ERR: There is one nameserver configured for this system but it does not work properly"
			else
				echo "ERR: There are $nsfound nameservers configured for this system and none of them works properly"
			fi
		else
			if [ "$nsfound" -eq 1 ] ; then
				echo "INFO: The nameserver configured for this system works properly"
			else
				echo "INFO: There are $nsfound nameservers is configured for this system and $nsok are working properly"
			fi
		fi
	fi
	return $status
}

check_ns () {
# Check the nameserver using host
# TODO: use nslookup?
#	nslookup $CHECK_HOST -$nameserver 
	nameserver=$1
	[ -z "$nameserver" ] && return 1
	status=1
	CHECK_RESULT="$CHECK_HOST .* $CHECK_IP_ADDRESS"
# Using dnscheck:
	dnscheck=`host -t A $CHECK_HOST $nameserver 2>&1 | tail -1`
	if [ -n "`echo $dnscheck |grep NXDOMAIN`" ] ; then
		echo "ERR: Dns server $nameserver does not resolv properly"
	elif [ -n "`echo $dnscheck | grep \"timed out\"`" ] ; then
		echo "ERR: Dns server $nameserver is not available"
	elif [ -z "`echo $dnscheck | egrep \"$CHECK_RESULT\"`" ] ; then
		echo "WARN: Dns server $nameserver did not return the expected result for $CHECK_HOST"
	else
		echo "INFO: Dns server $nameserver resolved correctly $CHECK_HOST"
		status=0
	fi

# Using dlint
#	dlint $CHECK_HOST @$nameserver >/dev/null 2>&1
#	if [ $? -eq 2 ] ; then
#		echo "ERR: Dns server $nameserver does not resolv properly"
#	elif [ $? -ne 0 ]; then
#		echo "ERR: Unexpected error when testing $nameserver"
#	else
#		echo "INFO: Dns server $nameserver resolved correctly $CHECK_HOST"
#		status=0
#	fi

	return $status
}

check_conn () {
# Checks network connectivity
	if ! check_host $CHECK_WEB_HOST >/dev/null ; then
		echo "WARN: System does not seem to reach Internet host $CHECK_WEB_HOST through ICMP"
	else
		echo "INFO: System can reach Internet host $CHECK_WEB_HOST"
	fi
# Check web access, using nc
	echo -e "HEAD / HTTP/1.0\n\n" |nc $CHECK_WEB_HOST $CHECK_WEB_PORT >/dev/null 2>&1
	if [ $? -ne 0 ] ; then
		echo "WARN: Cannot access web server at Internet host $CHECK_WEB_HOST"
	else
		echo "INFO: System can access web server at Internet host $CHECK_WEB_HOST"
	fi
}

# TODO: checks could be conditioned, i.e. if there is no proper
# interface setup don't bother with DNS and don't do some Inet checks
# if DNS is not setup properly
check_local
check_netif
check_dns
check_conn

exit 0

You might find that you need to install a couple of extra packages to use this script as it uses several commands:

If you've read through the script then you'll see it just does some basic checks and tells you whether your system is properly configured (INFO messages), there are suspicious problems (WARN messages) or there are issues that prevent network connectivity (ERR messages). Of course, fixing this is up to you, but the information might get you started on where you need to look up first.

I would like to see this packaged on a properly UI for newbie users, since I have been unable to find one of those. So if somebody wants to recode this into C (or point us to available tools that already do this) feel free to mail me.

 

 


Posted by Anonymous (196.200.xx.xx) on Thu 5 May 2005 at 14:39
It's already nice to have it as a package for non newbie users. As you mentioned, it is always a repetitive and tedious task to find out what exactly is wrong with a network. Thanks for the script! :)

[ Parent | Reply to this comment ]

Posted by Anonymous (195.190.xx.xx) on Fri 6 May 2005 at 16:30
Thank you for the script. It works also as test for a simple Netfilter script that I have on one of my machines. The Netfilter script is one of the Rusty Russels "unreliable" scripts. It seems to work just fine too. See below.

#!/bin/bash
iptables --policy INPUT DROP
iptables --policy OUTPUT DROP
iptables --policy FORWARD DROP
iptables -N stop
iptables -A stop -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A stop -m state --state NEW -i ! ppp0 -j ACCEPT
iptables -A stop -p icmp -m state --state INVALID -j DROP
iptables -A stop -j DROP
iptables -A INPUT -j stop
iptables -A OUTPUT -j stop
iptables -A FORWARD -j stop
iptables -vL

[ Parent | Reply to this comment ]

Posted by Anonymous (203.122.xx.xx) on Thu 12 May 2005 at 09:30
Cool. I prefer lots of orthogonal tools myself, but yours is a goodie for the newbie. One thing you may want to add is an upnotifier - a simple way to let you know when the connection is back for flakier ISPs. I know loads of people who cobble up something like this:

#!/bin/bash
#notifies audibly when host/ip is reachable again

aumix -w100 -v79
say testing sound levl 2>&1 > /dev/null

HOST="216.109.112.135" ; say "[traIIng jAhu d0t k0m]"

until [ 1 -eq 0 ]; do

     until [ "`fping $HOST | grep alive`" ]; do
       sleep 5
     done

  sleep 1
  echo "$HOST up"
  say up 2>&1 > /dev/null
done

In the above I check my connection to yahoo is alive using the iputils-ping debian package. I am overly fond of the say text-to-speech tool (from rsynth) so I have that too.

[ Parent | Reply to this comment ]

Posted by Anonymous (203.122.xx.xx) on Thu 12 May 2005 at 09:33
Whoops. I mean using the fping debian package.

[ Parent | Reply to this comment ]

Sign In

Username:

Password:

[Register|Advanced]

 

Flattr

 

Current Poll

Which init system are you using in Debian?






( 1029 votes ~ 6 comments )