Systems Engineering and RDBMS

Archive for the ‘Networking’ Category

FTP’ing data from one machine to another without disk storage

Posted by decipherinfosys on July 16, 2009

Recently at one of our client sites, we were troubleshooting a slowness issue in the application, which occurs only on weekends. System is working normally during weekdays. Different components of systems were checked and it all came out clean. So we decided to perform basic ftp test on good day vs. bad day. If time taken is more on bad day compared to the timing on good day then we know for sure that there is problem somewhere either on the machines involved and/or in the network between them. This is not the true test to identify the slowness as application might be communicating on the port other than the ftp port but it will give us some idea in general.

Now to perform ftp from one machine to another and vice versa we need a physical file of some size and hence need the disk storage as well. But IBM support person working with us suggested a very good approach to ftp data without involving a file and as result without involving any disk storage. Following is the basic syntax of the put command once we establish the ftp session with remote server.

put “|dd if=/dev/zero bs=32k count=1000” /dev/null

In above syntax, we are using dd command to transfer the 1000 blocks of data with each data block of 32k and we are using /dev/zero as input and /dev/null as output. Using this syntax we can transfer large file without involving disk. We can put it even in a shell script and then schedule the shell script as cron job to run at regular interval. Following is the shell script wrapped around the command.

/usr/bin/ftp -n <IP address of machine> <<END
verbose on
user <userID> <Password>
bin
put “|dd if=/dev/zero bs=32k count=10000” /dev/null
bye
END

Copy above lines into the file and save it as a .sh file. Please make sure to change the values in the <> brackets with correct IP address of the machine you want to perform ftp to. Also change the user line (line# 3) with appropriate user credentials. You will save this file and execute it on the machine from which you want to perform ftp.

Obviously this is not the route we took immediately after facing the issue. We resorted to ftp option after exhaustive checking of application and database and not finding anything suspicious. We performed above test on AIX servers. To know more about dd command, please refer to AIX manual pages (man dd) on any of the AIX servers in your environment.

Resources:

  • IBM Article – here.

Posted in Networking, Technology | Leave a Comment »

Why does the IP address and subnet mask matter?

Posted by decipherinfosys on June 19, 2009

We had an interesting issue that relates to networking and SQL heartbeat with one of our clients that we would like to share with our readers.

One of our clients has their servers distributed across 3 data centers in US. Let us call them A, B and C. They have a VPN tunnel connecting all their sites together using Cisco ASA firewalls. Their IT staff access the entire infrastructure through one data center (A) which has remote access VPN enabled in the Cisco ASA. They recently reported an issue of not being able to access the servers in SQL cluster across remote access VPN located in data center (B). However, those servers are accessible from data center A, C and within B itself and others servers in data center B were accessible through remote access VPN.

As their remote VPN connection terminates at Cisco ASA at data center (A), various teams were involved to find out the cause. The following were checked to identify the cause of this issue:

1. It was ensured that the remote access VPN subnet (10.1.1.x) is added to the crypto-map on the site to site VPN configuration.
2. We also made sure all the servers are connected to the same switch and residing on same VLAN.
3. There was no specific access list (ACL) or firewall or IPS or F5 configuration that was blocking traffic to the database servers from remote access VPN subnet (10.1.1.x).
4. We ensured all the servers have the same IP default gateway configured.

During packet tracing, it is found that the traffic reaches to the ASA at data center A and also reaches data center B. It is found that the traffic was not going back from the servers to the remote access VPN subnet. We realized that there would be something wrong on the SQL cluster servers and started looking in depth on its network configuration. We identified that SQL server’s heart beat NIC was configured with the IP address of 10.0.0.x with a subnet mask of 255.0.0.0 (/8) allowing to have 16777214 hosts where only 2 IP addresses are needed for heart beat. So, all the incoming traffic from remote access VPN was forwarded to the heart beat NIC on the SQL servers and not going back to the remote access VPN ASA.

Having a subnet mask of 255.255.255.252 on SQL servers heart beat network would have allowed it to have only two IP addresses that are needed for heartbeat on the SQL cluster. As it is a production SQL network, we did not want to change the heart beat network’s IP address or subnet mask. As an alternative workaround, we used persistent route in Windows 2003 to configure the remote access VPN traffic to reach the correct NIC and gateway. A helpful article on windows 2003 routing can be found here. Once we added the persistent route, the remote access VPN users were able to access the SQL servers.

Lesson’s learned:

1. Make sure you aware of all the network addresses and subnets involved in all the locations.
2. Assign a subnet mask for the required number of host addresses.

Posted in Networking, Protocols, SQL Server, Technology | Leave a Comment »

Finding Active Network Ports Using ‘netstat.exe’

Posted by decipherinfosys on April 15, 2007

The netstat.exe command is a simple yet powerful command line tool that can help you with basic networking issues, such as what ports are active on your personal computer, all the way up to assisting with complex tasks like troubleshooting network connectivity problems within a distributed computing environment.

As many of us move away from command line tools in favor of GUI-based network monitoring tools, it is always nice to review the features of a ‘back to basics’ command.

Netstat has approximately 10 switches associated with it, and typing netstat ? provides a brief summary of each. 

Running netstat without any switches displays all ports open on the machine at the time the command is run.

If you need more information, like a list of all ports that are listening on a machine, simply add the -a switch. The output of a netstat -acommand is displayed in the the following screen-shot:

netstat_a

If you prefer to see the Local Addresses in numerical IP address form, you can add the -n switch to the command, which changes the output to this:

netstat_an

In Windows 2003, netstat has gained the -oswitch. This switch is extremely helpful because it identifies which process identifier (PID), or program, is listening on a given port. So netstat -ano will provide the following information:

netstat_ano

As you can see from the above examples, netstat.exe is a basic command that can provide some very useful information.

Posted in Networking, Windows | Leave a Comment »

Network Port Configurations for MSDTC

Posted by decipherinfosys on March 8, 2007

What is MSDTC?

MSDTC is the Microsoft Distributed Transaction Coordinator, which is a transaction manager program that permits client applications to include several different sources of data into one transaction. MSDTC then coordinates committing the transaction across all of the servers that are listed in the transaction. MSDTC runs on all Windows operating systems, and is also installed by a variety of Microsoft applications, including Personal Web Server and SQL Server.

What Network Ports are Used?

MSDTC uses a number of TCP network ports for sending and receiving messages. This fact must be considered when MSDTC is running in a network environment where the servers involved in the transactions. Say you are running a multi-tier application, and each tier is separated by a router or firewall for security purposes. An example would be an application server in one tier communicating with a SQL Server database You will need to know what port numbers need to be opened for MSDTC transaction information to be able to pass through successfully.

For sending out transaction messages, MSDTC always uses the same TCP port – 135. Dealing with the response message is a little more tricky. MSDTC response messages return on a dynamically assigned port anywhere in a range from 1024 – 5000.

Configuring the MSDTC Respone Port Range

As most of you can probably guess, network administrators are not very fond of opening a wide range of ports all at once. So in order for MSDTC communications to still work and keep the network administrator happy at the same time, you will need to reduce the port range used by the response messages. This change is configured in the registry of the servers involved in the MSDTC communications. You will need to add a couple of keys to the registry.

Note: Please make sure to always take a backup of the registry prior to making any changes!

The location of the change is:

HKEY_LOCAL_MACHINE\Software\Microsoft\Rpc\Internet

The following entries need to be added:

Ports : REG_MULTI-SZ : 1024-1054

PortsInternetAvailable : REG_SZ : Y

UseInternetPorts : REG_SZ : Y

(There is a space both before and after each colon)

In this particular example we are limiting the responses to a port range of only 1024-1054. The exact range of ports to use is basically up to each individual organization.

Testing MSDTC Communications

So how do you know your changes were successful? Microsoft provides a handy little tool called DTCPing.exe that you can use to test MSDTC communications between servers. DTCPing can be downloaded from:

http://www.microsoft.com/downloads/details.aspx?FamilyID=5e325025-4dcd-4658-a549-1d549ac17644&displaylang=en

This file is a self-extracting zip file.  Confusingly, the zip file and actual executable file are both named “DTCPing.exe”, so you need to make sure to extract to a separate directory, otherwise you will receive a ‘Cannot create output file’ error message.

Once extracted, simply launch DTCPing.exe. The tool must be up and running on both the sending and receving servers, otherwise the test will fail. The initial screen of the tool will look like this:

DTC Ping

From here just type in either the NetBIOS name or IP Address of the remote node and click Ping. Test messages will appear in the DTCPing windows of both the sending and responding servers, and a summary of the test will be presented at the end. The test scenario will also be written to a log file that can be found in the same directory as the DTCPing.exe file.

Posted in Networking | Leave a Comment »

Troubleshoot Your Network with NETDIAG.EXE

Posted by decipherinfosys on February 17, 2007

What is Netdiag.exe?

Netdiag.exe is a Windows 2000 and 2003 Server command line tool that can be used to effectively test the network connectivity of a computer, and provides valuable insight to the overall health of your network.  Netdiag can help you solve any number of network issues including:

  • Checking Virtual Private Networks (VPN) network tunnels
  • Domain Name Service (DNS) or Windows Internet Naming Service (WINS) name resolution problems
  • Active directory replication
  • Verifying the binding of a server’s network cards
  • Problems with Internet Protocol Security (IPSEC)
  • Winsock corruption
  • Verifying the ability of domain controllers to use Lightweight Directory Access Protocol (LDAP)

Installing Netdiag.exe

Netdiag is included as part of the Support Tools on the Windows Server CD. Once the Support Tools have been installed you can simply run ‘netdiag.exe’ from a command line. 

Using Netdiage.exe

Properly using netdiag involves a number of command line switches that need to be entered in a certain order. Not all of the switches are required, but the correct full syntax if you were to use them all is as follows:

netdiag [/q] [/v] [/l] [/debug] [/d:domain_name] [/fix] [/dcaccountenum] [/test:test_name] [/skip:test_name]

Below are the definitions of the various parameters:

/q: Specifies quite output and only displays errors

/v:Runs Netdiag in verbose mode, which dispays each action as it is being performed

/l:Sends the output of the Netdiag results to a Netdiag.log file

/debug:Runs Netdiag in debug mode

/d:domain_name: Used to locate domain controllers in a specified domain

/fix:This parameter detects and correct issues with DNS. It verifies that all DNS entries contained on a server are correct, and updates any invalid entries.

/dcaccountenum: Enumerates the computer accounts of the domain controller

/test:test_name:This parameter can be used to specify form a long list of netdiag tests that you can run. test_name can be any of the following values:

                  Autonet: Automatic Private IP Addressing (APIPA) address test
                  Bindings: Bindings test
                  Browser: Redir and Browser test
                  DcList: Domain controller list test
                  DefGw: Default gateway test
                  DNS: Domain Name Service (DNS) test
                  DsGetDc: Domain controller discovery test
                  IpConfig: IP address configuration test
                  IpLoopBk: IP address loopback ping test
                  IPSec: Internet Protocol security (IPSec) security test
                  IPX: Internetwork Packet Exchange (IPX) test
                  Kerberos: Kerberos Test
                  Ldap: Lightweight Directory Access Protocol (LDAP) test
                  Member: Domain membership test
                  Modem: Modem diagnostics test
                  NbtNm: NetBIOS over TCP/IP (NetBT) name test
                  Ndis: Netcard queries test
                  NetBTTransports: NetBT transports test
                  Netstat: Netstat information test
                  NetWare: NetWare test
                  Route: Routing table test
                  Trust: Trust relationship test
                  WAN: Wide Area Network (WAN) configuration test
                  WINS: Windows Internet Naming Services (WINS) service test
                  Winsock: Winsock test

You can specifiy multiple tests  by using multiple instances of the /test:test_namecommand, each separated with a space. So, for example, if you wanted to run three tests: DNS, IPSec, and WINS, a typical Netdiag command line would look like this:

netdiag /v /dcaccountenum /test:DNS /test:IPSec /test:WINS

 – /skip:test_name:Allows you specify one or more of the above tests that you want to skip during a particular Netdiag session. As with /test:test_name, you can specify multiple tests to skip by using multiple instances of the /skip:test_name command, each separated with a space.

Even in today’s point-and-click world, there are still a huge number of effective and powerful command line tools available for troubleshooting and monitoring. Netdiag is just one of many, but it is most certainly useful when examining your Windows Server infrastructure.

Posted in Networking | Leave a Comment »