Systems Engineering and RDBMS

Archive for the ‘Linux’ Category

Learning Hadoop

Posted by decipherinfosys on March 20, 2012

In a recently concluded project, we had the opportunity to work on Hadoop.  There was a learning curve since none of us had worked in Hadoop before.  Here are some URLs to help you get started with your learning process in this regard:

Basics of Hadoop:

The article on gigaom or the series of articles on cloudera’s site will get your started:

http://gigaom.com/cloud/what-it-really-means-when-someone-says-hadoop/

http://www.cloudera.com/what-is-hadoop/

Sign up with Cloudera and you will have access to a lot of very good learning material on Hadoop, example:

http://www.cloudera.com/resource/introduction-to-apache-mapreduce-and-hdfs/  is a good starter’s video on MapReduce and HDFS.

or this one: http://www.cloudera.com/resource/apache-hadoop-ecosystem/ for understanding the Hadoop ecosystem.

And this whitepaper from Gartner on Hadoop and MapReduce for Big Data Analytics:

http://info.cloudera.com/GartnerReportHadoopJanuary2011.html

If you like to have text available for your learning purposes rather than video tutorials, here is a good chapter on HDFS: http://www.aosabook.org/en/hdfs.html

Setting up Hadoop cluster:

And once you are ready to jump in, there are some excellent tutorials by Michael G. Noll to guide you:

To set up your first Hadoop node: http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/

And then multiple node cluster: http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/

And here are some additional good tutorial references: http://www.delicious.com/jhofman/tutorials+hadoop

Microsoft and BigData

Recently, MSFT also announced their support for Apache Hadoop.  You can read more on MSFT’s big data solution from here:

http://www.microsoft.com/sqlserver/en/us/solutions-technologies/business-intelligence/big-data-solution.aspx

and the work done by HortonWorks for extending Apache Hadoop to Windows:

http://hortonworks.com/blog/extending-apache-hadoop-to-millions-of-new-microsoft-users/

Posted in Big Data, Linux, SQL Server, Unix, Windows | 1 Comment »

Creating an empty file in Unix and Windows

Posted by decipherinfosys on February 15, 2009

Recently at a client site, we came across a situation where we had to create empty files with a specific size on Windows.  We had to do this for R&D purposes in order to be able to mimic ASM in Windows environment without actually having different disks.

Windows has a utility called fsutil.exe. This is a command line utility. Mainly it is used for maintaining file and disk properties and is an advance level utility. But one of its fine usages is to create the empty file of a specific size. Following is the command to create an empty file.

C:\>fsutil file createnew c:\ttt.txt 2048
File c:\ttt.txt is created

C:\>dir ttt.txt
Volume in drive C is C-Drive
Volume Serial Number is 2482-1E9C

Directory of C:\

02/12/2009 05:10 PM 2,048 ttt.txt

Similarly in Unix, there is a touch command. The touch command updates the access and modification times of each file specified by the parameter or command line argument. If time value is not specified then touch command will use current time. If we specify a file that does not exist, it will create that file with current date time unless –c flag is specified. When –c flag is specified it will not create the file, if it does not exist. Following command shows that.

$ touch -c ttt.txt
$ ls -l ttt.txt
ls: 0653-341 The file ttt.txt does not exist.

Now let us issue same command without using –c flag.

$ touch ttt.txt
$ ls -l ttt.txt
-rw-r–r– 1 b01234 uga 0 Feb 13 17:19 ttt.txt

Posted in Linux, Unix, Windows | 1 Comment »

Sharing Files between Windows and Linux/Unix machines

Posted by decipherinfosys on June 10, 2007

It can get quite frustrating at times when you want to access files on your linux/unix machines which host your Oracle or DB2 LUW databases. MSFT offers a simple solution via the program suite called Services for Unix (SFU). Samba also has a suite of tools for UNIX systems for sharing files over the network with MSFT Windows. Here are a couple of resources to help you further to sort out file sharing issues:

http://samba.org –> Samba’s web-site and a lot of very good information and tools.

http://support.microsoft.com/kb/304040 –> Configuring File Sharing in Windows XP

Windows Services for Unix download – here and the homepage for SFU information – here.

Posted in Linux, Unix, Windows | Leave a Comment »

Configuring RedHat Linux AS 3.0 virtual machine using VMWare

Posted by decipherinfosys on June 7, 2007

Here are the steps to configure a RedHat AS 3.0 virtual machine using VMWare:

1)    Start the Host Configuration in the VMWARE console.

2)    Right click on the Virtual Machine you wish to configure and choose “Virtual Machine Settings”.

3)    In the screen that pops up, choose the NIC (usually NIC1) you wish to configure.

4)    On the right hand side of the screen change the “Adapter Type” to vmxnet.

5)    Once this is done you will need to start the Virtual Machine.

6)    Once the Virtual Machine is booted, log in as root.

7)    Once logged in you will need to open a terminal. (Command prompt).

8)    Type the following using the same case:
unset DISPLAY

9)    Execute vmware-config-tools.pl  (when prompted, choose “no” when asked if you wish to change your guest x resolution).

10)    Execute the following command:
/etc/init.d/network start

11)    Click on the start button in the bottom left (a red hat).  Choose “System Settings” then choose Network.

12)    A screen will pop up.  Delete any failed/non-working devices.

13)    Click New, then Ethernet Connection, then Forward.

14)    Choose “vmxnet” and click Forward, then Forward again, then Apply.

15)    Save the Changes when prompted.

Posted in Linux, VMWare | Leave a Comment »

Configuring RedHat Linux for Oracle 10gR2 client installation

Posted by decipherinfosys on June 4, 2007

Here is a blog post detailing how to configure an environment on a RedHat Linux OS for the installation of a 10gR2 Oracle Client. It assumes some familiarity with both RHEL and Oracle.

These are the steps:

1) Create the Oracle user and necessary groups:

a. Log into the machine as root.

i. Either telnet as a lesser user and su to root, or use putty or some other ssh connection.

b. Before we can create the user, we must first create the groups that we are going to assign the user to. We need two groups for the installation: oinstall and dba. Although it is not required for the installation, we try to use the same group and userid’s across platforms. For the Oracle user we want to use 501, oinstall – 300, dba – 500. It is very possible these userid’s are not available (ie used by another group). If this cannot be changed, then you can pick another id, or have the system do it for you. The purpose of this is to make file sharing simpler when mounting volumes for other machines. The –g option indicates you’re a going to manually specify the group id. Below are the commands to create the necessary groups.

i. groupadd -g 300 oinstall
ii. groupadd –g 500 dba

c. Create the Oracle user (You can also use a GUI tool for this, if available). For the user, we want the id to be 501. Again another can be used. The common id convention is for convenience. Below is the command to create the oracle user.

i. useradd -u 501 -g oinstall -G dba -d /home/oracle -s /bin/ksh oracle

1. –u sets the userid to 500
2. –g specified the primary group to be oinstall
3. –G specifies the dba as an additional group oracle will belong to. This can be a comma delimited list of groups.
4. –d specifies /home/oracle to be the user’s system home directory. It will be created automatically.
5. –s specifies the default shell for the oracle user to be /bin/ksh. This is the korn shell.

ii. This will create the user oracle home, but the group for the oracle home will be oinstall. This should be changed as it has nothing to do with the installation. Go to the /home directory and execute the following:
chgrp -R dba oracle
The /home/oracle directory is a directory where DBA’s can keep scripts and such.
iii. The password for the oracle user needs to be set. Use the passwd command to do this:
passwd oracle
You will be prompted to enter the password and then confirm.

iv. In korn shell, the user’s environment is initialize via settings in the “.profile” file. The two attachments below can be used to initialize the environments. The ‘.’ in front of the file name indicates that the file is hidden. It is not a typo.

1. “.profile” This will be the initializing file. For oracle, our convention is to keep this file fairly simple. Since Oracle software can have multiple “Oracle” (not system) homes, we create additional “.” environment files that allow DBA’s to easily change the environment for specific homes. This will become clearer during the oracle client installation instruction.
export PATH=.:$PATH:/sbin

# set up the terminal:
stty istrip
stty erase “^?” kill “^U” intr “^C” eof “^D”

TERM=xterm
export TERM
# Set up the shell environment:
set -u
trap “echo ‘logout'” 0

# Set up the shell variables:
EDITOR=vi
export EDITOR

set -o vi

#. ./.oracle92RACenv
. ./.oracle10gRACenv

2. “.” environment file can be call what ever is appropriate. In this case, we will call it “.oracle10gR2ClientEnv”. This will be more detailed in the environmental variables. It will contain the settings for a specific “Oracle” Home. There are two important values in this file, that will need to be changed. That is the ORACLE_HOME value and the ORACLE_BASE value. Where ever the ORACLE_HOME is for this installation, that full path will need to be specified. Same for the ORACLE_BASE. These will be defined in the following steps. You will need to revisit this file to ensure it is configured properly.

PATH=/usr/bin:/etc:/usr/sbin:/usr/ucb:${HOME}/bin:/usr/bin/X11:/sbin:.;export PATH
umask 022
set -o vi
alias ll=’ls -ltr’

ORACLE_BASE=/u01/app/oracle;export ORACLE_BASE
ORACLE_HOME=${ORACLE_BASE}/product/Ora102;export ORACLE_HOME

NLS_LANG=AMERICAN_AMERICA.AL32UTF8;export NLS_LANG

ORA_NLS33=${ORACLE_HOME}/ocommon/nls/admin/data;export ORA_NLS33

PATH=${ORACLE_HOME}/bin:/usr/ccs/bin:/usr/local/bin:/usr/vacpp/bin:/opt/IBMJava2-131/bin:/opt/IBMJava2-131/jre/bin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin; export PATH
LIBPATH=${ORACLE_HOME}/lib;export LIBPATH
CLASSPATH=${ORACLE_HOME}/JRE/lib:${ORACLE_HOME}/jlib:${ORACLE_HOME}/rdbms/jlib:${ORACLE_HOME}/network/jlib;export CLASSPATH
LD_LIBRARY_PATH=${ORACLE_HOME}/lib:${ORACLE_HOME}/oracm/lib;export LD_LIBRARY_PATH

TMPDIR=/u01/tmp;export TMPDIR

MACHINE_NAME=`hostname -s`;export MACHINE_NAME
PS1=’10g_${ORACLE_SID}_${MACHINE_NAME} ${PWD}=> ‘;export PS1

2) Create the installation directories for the oracle binaries. For this example /u01 will be used as the base volume.

a. As root create an “app” directory off of /u01.
b. Change the owner and group of the “app” directory as follows:
chown oracle:dba app
c. As oracle, Create the following path: “/u01/app/oracle/product/Ora102” This is the value needed for the ORACLE_HOME parameter in the environment file.
At this point, the group associated with the files and directories above “app” should be oinstall. This will be done by default as you create the directories.
d. In the environment file, set the ORACLE_BASE value to “/u01/app/oracle”.
set the ORACLE_HOME value to “${ORACLE_BASE}/ product/Ora102”.
e. Once these are set correctly, re-initialize by executing “. ./.profile”.
f. At the command prompt, confirm the settings by using the following example: (note* the bolded values should be correct)

10gR2Client_RHBLD01 /home/oracle=> . ./.profile
10gR2Client_RHBLD01 /home/oracle=>
10gR2Client_RHBLD01 /home/oracle=> echo $ORACLE_HOME
/u01/app/oracle/product/Ora102
10gR2Client_RHBLD01 /home/oracle=> echo $ORACLE_BASE
/u01/app/oracle
10gR2Client_RHBLD01 /home/oracle=> cd $ORACLE_HOME
10gR2Client_RHBLD01 /u01/app/oracle/product/Ora102=> cd $ORACLE_BASE
10gR2Client_RHBLD01 /u01/app/oracle=>

g. Create the temporary directory that is specified by TMPDIR in the “.” environment file. Do not forget to do this. The client installation may continue all the way through with out error, but when you attempt to execute sqlplus and connect to a database, you will get errors.
Change the owner ship of the temporary directory using: “chown oracle:dba tmp”
Validate by: cd $TMPDIR. This should take you to the temporary directory specified.

3) Pre-Installation requirements:

a. Kernel version: 2.6.9-5.EL

10gR2Client_RHBLD01 /u01/Installation_Media/client=> uname -a
Linux RHBLD01 2.6.9-5.ELsmp #1 SMP Wed Jan 5 19:30:39 EST 2005 i686 i686 i386 GNU/Linux

b. Required packages(version must be these or higher):

binutils-2.15.92.0.2-13.EL4
compat-db-4.1.25-9
compat-libstdc++-296-2.96-132.7.2
control-center-2.8.0-12
gcc-3.4.3-22.1.EL4
gcc-c++-3.4.3-22.1.EL44
glibc-2.3.4-2.9
glibc-common-2.3.4-2.9
gnome-libs-1.4.1.2.90-44.1
libstdc++-3.4.3-22.1
libstdc++-devel-3.4.3-22.1
make-3.80-5
pdksh-5.2.14-30
sysstat-5.0.5-1
xscreensaver-4.18-5.rhel4.2
setarch-1.6-1

If not installed, then install. If not the appropriate version, then upgrade the component.
The packages can be checked for their version by executing the following:

rpm –q <package_name>

example(installed):
10gR2Client_RHBLD01 /u01/Installation_Media/client=> rpm -q binutils
binutils-2.15.92.0.2-10.EL4

example(not installed):
10gR2Client_RHBLD01 /u01/Installation_Media/client=> rpm -q xscreensaveer
package xscreensaveer is not installed

There are several ways to update/install the packages:

1) manually update/install the packages using rpm. (becomes painful with dependencies)
2) use up2date to get the latest version of the packages. Up2date handles dependencies for you by installing or upgrading dependent packages.
3) Use the up2data gui if available.

4) In order to perform the installation, you will need to have an xserver (Exceed) running on your local machine or you will have to go directly to the console, which will need a gui interface. I will give instruction for the remote connection using an xserver.

a. telnet to the machine as oracle.
b. Go to the directory where you have unpacked the installation software.
c. Execute “export DISPLAY=<your ipaddress>:0.0” This is case sensitive.
d. Start the xserver on your machine in passive mode.
e. Execute ./runInstaller
f. The gui installation should appear on your machine for the Oracle client.

5) You are now ready to begin the client installation process. Please refer to the installing Oracle 10gR2 client installation document for more details. You can also refer our previous blog post for it.

Posted in Linux, Oracle | Leave a Comment »

Chosing between Windows and Linux

Posted by decipherinfosys on March 25, 2007

Do you use Windows or Linux at your workplace? If you are looking to make a decision, this article at computer world might help you make the right decision.  It is pretty detailed and does a good job on pointing out the pros and cons.

Posted in Linux, Windows | Leave a Comment »

Some useful Unix commands

Posted by decipherinfosys on February 23, 2007

Here is a cheat-sheet for the Unix commands to find swap, RAM and OS version on different Unix and Linux versions:
To Find Swap, RAM, and OS Version

OS     SWAP             RAM                     OS VERSION
———————————————————————————-
AIX     /usr/sbin/lsps -a    /usr/sbin/lsattr -HE -l sys0 -a realmem    oslevel

HP     swapinfo -q        dmesg | grep -i mem            uname -a

Tru64     swapon -s        vmstat -P                /usr/sbin/sizer -v

Solaris     swap -s        /usr/sbin/prtconf | grep -i memory    uname -r

Linux     free             free                    uname -a

Posted in Linux, Unix | Leave a Comment »

Detecting CPU Bottleneck

Posted by decipherinfosys on February 22, 2007

When do you categorize a server to be running into a CPU bottleneck?  And how do you go about collecting that information on Windows and other Unix systems?  The yard stick that is very commonly used is that if an individual user process is consuming greater than 10% of the CPU then there is a need to investigate further.  Also, if the cumulative CPU usage is consistently more than 80%, then this also indicates a CPU bottleneck. If there are no individual processes using a lot of CPU, then you are trying to put too much work on the server, and should upgrade to either faster processors or add more processing power. Here is what you can do to automatically capture this information on different OS:

HP/UX or AIX or Linux or Solaris:

“sar –o <filename> 60 10 >/dev/null 2>&1 &” will sample the CPU activity every minute for 10 minutes and save the output in a file.  If you are running into issues, reduce the time-frame for the data collection.

If user+nice+system > 80% for an extended period of time, then you have a CPU bottleneck.

AIX specific:

“vmstat 10 500” will display system CPU usage every 10 secs. If id=0 or us+sy>80 for an extended period of time, then this indicates that the system is CPU-bound.

Solaris specific:

“top” will show you the top processes by CPU activity

Note: You may have to install SUNWaccu and SUNWaccr packages if sar is not installed.

Windows:

a) Use the task manager and select the processes tab.
b) Then click either the “CPU” column title to sort the processes

A better approach though is to use perfmon (Performance Monitor) and schedule it to collect the data:

c) Type perfmon from the command line (Start/Run and then perfmon and enter)
d) Then click on the “+” symbol and add “%processor time” for each of the processors you have on your system.  You can save this as a blg file and then do further analysis on it later on.

Posted in Linux, Unix, Windows | Leave a Comment »

Installing FTP on Red Hat Linux AS 3.0

Posted by decipherinfosys on January 29, 2007

The RPM files referenced in the document can be found in the following location on the installation media: the sub-directories containing the RPMS have the following Format: “CDn/RedHatRPMS”

To set up an FTP server, make sure the vsftpd package is installed by issuing the following command as root:
rpm -q vsftpd

You should see:
vsftpd-1.2.0-4

If not installed, then use the following directions.

1)    The vsftp package can be obtained on your distribution media, or through your system appropriate channel of the Red Hat Network

vsftpd-1.2.0-4.i386.rpm  (CD1/RedHat/RPMS)

2)    If the package is not installed, insert the installation CD-ROM containing the appropriate package, change to the /mnt/cdrom/RedHat/RPMS/ directory, and, as root, type the following command:

rpm -i vsftpd-1.2.0-4.i386.rpm

3)    Check for installation:

rpm -qa | grep vsftp               (alternatively rpm -i vsftp)

You should see:
vsftpd-1.2.0-4

4)    Once installed, start the vsftpd service by issuing the following command:

service vsftpd start

5)    To make the service available for the next boot sequence chkconfig the service on by issuing the following command:

chkconfig vsftpd on

Additional information can be found in the vsftpd man page:
man vsftpd

Posted in Linux | Leave a Comment »

Installing telnet on Red-Hat Linux AS 3.0

Posted by decipherinfosys on January 29, 2007

The RPM files referenced in this blog post can be found in the following location on the installation media.: the sub-directories containing the RPMS have the following Format: “CDn/RedHatRPMS”

1) Both telnet-server-0.17-26 and telnet-0.17-26 need to be installed

Check this by executing: rpm -qa | grep telnet

2) If missing, then you will need to get the required packages from the installation media, or a compatible source, and install them.
telnet-0.17-26.i386.rpm (CD2/RedHat/RPMS)
telnet-server-0.17-26.i386.rpm (CD3/RedHat/RPMS)

3) Once the installation packages are on the machine (*.rpm files), you can install them in the following manner. (It is common to
place these rpm files in the /tmp or /temp directory).

rpm -i <rpm file>

4) Execute the following command to ensure the packages are installed:

rpm -qa | grep telnet

You should see the following: telnet-server-0.17-26
telnet-0.17-26

5) Make sure the installation was done properly:

chkconfig –list telnet

You should see:
telnet off

6) You now will need to change the telnet configuration to on

chkconfig telnet on

7) Check the configuration:

chkconfig –list telnet

You should see:
telnet on

8 To start the service, you will need to restart xinetd:

service xinetd restart

You should see:

Stopping xinetd: [ OK ]
Starting xinetd:  [ OK ]

Posted in Linux | 1 Comment »