text:
For those who weren't aware.....starting with AIX 5.3 TL7, quorum changes are allowed online without having to varyoff and varyon the volume group. This also means no reboot is required when changing quorum for rootvg. Of course this also applies to AIX 6.1.
Chris's AIX Blog
Online quorum change
Using the AIX splitvg command.
Just the other day, I needed to use the AIX splitvg command in order to copy some data from one system to another.
I thought I'd share the experience here.
The splitvg command can split a single mirror copy of a fully mirrored volume group into a separate "snapshot" volume group.
From the man page:
The original volume group VGname will stop using the disks that are now part of the snapshot volume group SnapVGname. Both volume groups will keep track of the writes within the volume group so that when the snapshot volume group is rejoined with the original volume group consistent data is maintained across the rejoined mirrors copies. Notes:
1 To split a volume group, all logical volumes in the volume group must have the target mirror copy and the mirror must exist on a disk or set of disks. Only the target mirror copy must exist on the target disk or disks.
2 The splitvg command will fail if any of the disks to be split are not active within the original volume group.
3 In the unlikely event of a system crash or loss of quorum while running this command, the joinvg command must be run to rejoin the disks back to the original volume group.
4 There is no concurrent or enhanced concurrent mode support for creating snapshot volume groups.
5 New logical volumes and file system mount points will be created in the snapshot volume group.
6 The splitvg command is not supported for the rootvg.
7 The splitvg command is not supported for a volume group that has an active paging space.
8 When the splitvg command targets a concurrent-capable volume group which is varied on in non-concurrent mode, the new volume group that is created will not be varied on when the splitvg command completes. The new volume group must be varied on manually.
So, looking at point 4, above, if you are using enhanced concurrent volume groups (for example with PowerHA), then you will not be able to use the splitvg command. This is disappointing as this would have been very handy in some of the large PowerHA systems I have worked with…..perhaps this will be supported in the future?
Anyway, back to my example. I had wanted to break-off one of the mirrors of a mirrored volume group and then assign the "split" volume group to another host to copy some data off.
The volume group datavg contained two disks, hdisk0 and hdisk3, as shown in the lspv output below.
# lspv
There were only two logical volumes (loglvl00, jfs2 log and fslv00, data) and a single file system (/data) in this volume group.
# lsvg -l datavg
The volume group datavg, was mirrored across hdisk0 and hdisk3, as shown in the lsvg output below.
# lsvg -p datavg
The logical volumes for the /data file system and the JFS2 log were both mirrored, as shown in the lslv/lspv output.
# lslv fslv00
# lslv loglv00
# lspv -l hdisk0
# lspv -l hdisk3
Using the splitvg command I was able to break-off one of the disks from the mirrored pair. This created a new volume group on hdisk3 called vg00.
# splitvg -c1 datavg
The new volume group contains a new logical volume (pre-fixed with fs i.e. fsfslv00) and a file system (pre-fixed with /fs i.e. /fs/data). I can mount this file system and access the data in the file system and create and/or modify files (as shown below).
# mount /fs/data
# cd /fs/data
# touch 3
At this point I was able to export the volume group and import it on another system. I had to re-map the Virtual SCSI disk on my VIOS first.
# cd
# varyoffvg vg00
# rmdev –dl hdisk3
; Re-map the disk at the VIOS layer to the other AIX LPAR.
; Configure the disk on the other LPAR (with cfgmgr) and import the volume group (with importvg).
Once I was finished with vg00 on the other LPAR, I re-mapped the disk to the original AIX LPAR, configured the disk and then re-joined the disk to the datavg volume group, with the joinvg command.
; Export the VG and remove the disk on the other AIX LPAR.
; Re-map the disk at the VIOS layer to the original AIX LPAR.
# cfgmgr
# lsdev –Cc | grep hdisk3
hdisk3 Available Virtual SCSI Disk Drive
# joinvg datavg
The volume group is now fully mirrored again. However the partitions are stale and need to be sync'ed. I manually re-sync the volume group with the syncvg command.
# lsvg -l datavg
# syncvg –v datavg
# lsvg -l datavg
More information can be found here: http://www-01.ibm.com/support/docview.wss?uid=isg3T1010934
nohup -p on AIX.
The nohup command in AIX has the ability to "nohup" an existing process and protect if from hangup signals. Using the -p flag, an existing, running process can be "nohup'ed" and told to ignore SIGHUPs.
"Why is this useful?" I hear you ask.
Well, I'll often start a "long running" process on a system at work, like a backup for instance. I'll sometimes forget to use the nohup command, so that my long running job will ignore any SIGHUPs and continue running, say for example if my ssh session to the server dies. The nohup command also generates a nohup.out file to capture any messages (errors or otherwise) generated by the process e.g.
$ nohup ./processname &
$ ls –l nohup.out
-rw------- 1 cgibson staff 0 May 26 19:42 nohup.out
In older releases of AIX, if I forgot to use nohup, I had two choices. I could either stop the process and start it again with nohup (often not desirable or possible) or I could leave my ssh session logged in on my work PC and hope that it didn't power down or crash or that my ssh connection did not drop over night.
Fortunately with more recent releases of AIX, this is not a problem anymore. If I forget to "nohup" a process, I can use the -p flag to inform the already running process that it should ignore all hangup (SIGHUP) signals. This will allow the process to continue running if my ssh session (terminal) dies and will also capture any output from the process into the nohup.out file, so I don't lose any important output from the process.
Here's an example. I'm at work, it's 5pm and I've started a "long running" process named mycmd. This was started via an ssh (Putty) session from my PC.
$ ./mycmd
; Partial output from topas showing mycmd running.
Name PID CPU%
mycmd 823432 98.8
; proctree output displaying PIDs associcated with mycmd and the associated ssh session (tty).
$ proctree 823432
520394 -ksh
585738 -ksh
823432 /home/cgibson/mycmd
I've walked out the door and I'm now on my way home for the evening.
Oh no! I've forgotten to start the process with nohup! I don't want to restart my process but I want to protect it from SIGHUPs, while it executes after hours.
No problem, I'll use nohup to modify the running process, so I can rest assured it will continue to run, even if my ssh connection drops over night. From home I start a new ssh session on the server and run nohup against the running process:
$ nohup -p 823432
At work the next morning, I discover that my original Putty session has indeed dropped! But, my process is still happily running! :)
; Partial output from topas showing mycmd still running.
Name PID CPU%
mycmd 823432 99.3
$ proctree 823432
520394 -ksh
585738 -ksh
823432 /home/cgibson/mycmd
You might find this useful if you ever forget to "nohup" a process and don't want to restart it again.
catman -w on AIX.
Have you ever encountered this error message on AIX before?
# man -k loopmount Of course
you have! If you've ever installed AIX from scratch then you have undoubtedly
seen this message. And of
course, you dutifully executed the 'catman –w' command to rectify the situation. Didn't you? I recently came
across this minor issue. Running catman produced the following error message: # catman -w catman: 0703-102 /usr/lbin/mkwhatis: The error
number is 1. The first
place to check was the MANPATH environment variable (which didn't look right). # echo $MANPATH :/opt/csm/man So I unset
it and ran catman again. This time,
with 'great success'. # unset MANPATH # catman -w Now I could search
for keywords against the /usr/share/man/whatis database that contain the string "loopmount". # man -k loopmount loopmount(1) - Associate an image file to a loopback
device. Another
interesting issue I've also come across is when catman runs with apparent success but I still can't find
keyword entries when I run 'man –k'. # catman -w # man -k loopmount loopmount: Cannot find matching
entry. I found
that manually setting MANPATH and re-running catman rectifies
this situation. # export MANPATH=/usr/share/man # catman -w # man -k lsvg loopmount(1) - Associate an image file to a loopback
device. Of course
the whatis command can also perform a similar function. Only difference is
that you need to know the command beforehand. # whatis
loopmount loopmount(1) - Associate an image file to a loopback
device. The catman command: http://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=/com.ibm.aix.cmds/doc/aixcmds1/catman.htm The whatis command: http://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=/com.ibm.aix.cmds/doc/aixcmds6/whatis.htm Locating a command by keyword: http://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=/com.ibm.aix.baseadmn/doc/baseadmndita/HT_locating_command_keyword.htm The /usr/share directory: http://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=/com.ibm.aix.baseadmn/doc/baseadmndita/usr_share_dir.htm
Of course you have! If you've ever installed AIX from scratch then you have undoubtedly seen this message.
And of course, you dutifully executed the 'catman –w' command to rectify the situation. Didn't you?
I recently came across this minor issue. Running catman produced the following error message:
# catman -w
catman: 0703-102 /usr/lbin/mkwhatis: The error number is 1.
The first place to check was the MANPATH environment variable (which didn't look right).
# echo $MANPATH
So I unset it and ran catman again. This time, with 'great success'.
# unset MANPATH
# catman -w
Now I could search for keywords against the /usr/share/man/whatis database that contain the string "loopmount".
# man -k loopmount
loopmount(1) - Associate an image file to a loopback device.
Another interesting issue I've also come across is when catman runs with apparent success but I still can't find keyword entries when I run 'man –k'.
# catman -w
# man -k loopmount
loopmount: Cannot find matching entry.
I found that manually setting MANPATH and re-running catman rectifies this situation.
# export MANPATH=/usr/share/man
# catman -w
# man -k lsvg
loopmount(1) - Associate an image file to a loopback device.
Of course the whatis command can also perform a similar function. Only difference is that you need to know the command beforehand.
# whatis loopmount
loopmount(1) - Associate an image file to a loopback device.
The catman command:
The whatis command:
Locating a command by keyword:
The /usr/share directory:
saposcol and shared memory on AIX.
cggibbo 270000TMUJ Tags:  aix ipcs ipcrm memory gibson chris shared saposcol 1 Comment 26,898 Views
If you are a system administrator that is responsible for managing AIX systems that run SAP, then you've probably had an experience similar to the following?
OK, so one day my SAP Basis administrator contacts me and says "I can't start saposcol......can you please reboot the system?" I quickly reply, "Was there an error message when trying to restart saposcol?". He replies, "No". Again I return very quickly, "OK, have you checked to see if there are any shared memory segments left for saposcol?". Just as quick, he replies "How do I do that?"
Together we try starting saposcol and what we find is that it thinks it's already running (as shown below, PID 327924). But there is no such process!
$ saposcol -l
09:37:24 22.01.2009 LOG: Effective User Id is root
* This is Saposcol Version COLL 20.95 700 - AIX v11.15 5L-64 bit 080317
* Usage: saposcol -l: Start OS Collector
* saposcol -k: Stop OS Collector
* saposcol -d: OS Collector Dialog Mode
* saposcol -s: OS Collector Status
* The OS Collector (PID 327924) is already running .....
$ ps -fp 327924
So we try to stop saposcol. This of course fails, as there is no PID 327924 running!
$ saposcol -k
Setting Stop Flag :
09:19:29 22.01.2009 LOG: ==== Stop Flag was set by saposcol (kill_collector()).
09:19:29 22.01.2009 LOG: ==== The collection process will stop as soon as possible
can't kill process 327924.
kill: No such process
ERROR:No reaction from collecting process 327924.
Please kill collecting process.
My conclusion is that there must be a shared memory segment still allocated for saposcol. There were many other SAP processes still running happily, so there were several shared memory segments to sift through. So, what shared memory ID does saposcol use?
Now, according to the following website, shared memory key 4dbe is used by saposcol on AIX.
So I run ipcs to check for the existence of 4dbe. And I find an entry for this key.
There are several process id's 'attached' to this segment. However, only one of them actually exists (PID 2293794).
# ipcs -ma | grep 4dbe
m 2097156 0x00004dbe --rw-rw-rw- root sapsys root sapsys 0 1870188 467164 2293794 9:39:43 9:39:43 8:12:43
# ps -fp 1870188
# ps -fp 467164
# ps -fp 2293794
sapadm 2293794 1 0 19:26:41 - 1:07 sapccm4x -DCCMS pf=/sapmnt/SAP/profile/SAP_DVEBMGS00_aix01
I ask the SAP admin to stop this process, which he does. Now I remove the shared memory segment and there is no evidence of 4dbe in the ipcs output.
# ipcrm -m 2097156
# ipcs -ma | grep 4dbe
We were then able to start saposcol again with success. The process is running and the shared memory segment 4dbe has returned.
# ipcs -ma | grep 4dbe
m 3145732 0x00004dbe --rw-rw-rw- root sapsys root sapsys 1 1870188 2293818 2207892 10:14:25 10:14:25 10:12:47
# ps -ef | grep oscol
sapadm 2064616 1 0 10:12:53 - 0:00 saposcol -l
# /usr/sap/SAP/SYS/exe/run/saposcol -s
Collector Versions :
running : COLL 20.95 700 - AIX v11.15 5L-64 bit 080317
dialog : COLL 20.95 700 - AIX v11.15 5L-64 bit 080317
Shared Memory : attached
Number of records : 17640
Active Flag : active (01)
Operating System : AIX aix01 3 5 00C01C704C00
Collector PID : 2064616 (001F80E8)
Collector : running
Start time coll. : Thu Jan 22 10:12:53 2009
Current Time : Thu Jan 22 10:17:46 2009
Last write access : Thu Jan 22 10:17:38 2009
Last Read Access : Thu Jan 22 10:16:16 2009
Collection Interval : 10 sec (next delay).
Collection Interval : 10 sec (last ).
Status : free
Collect Details : required
Refresh : required
Header Extention Structure
Number of x-header Records : 1
Number of Communication Records : 60
Number of free Com. Records : 60
Resulting offset to 1.data rec. : 61
Trace level : 2
Collector in IDLE - mode ? : NO
become idle after 300 sec without read access.
Length of Idle Interval : 60 sec
Length of norm.Interval : 10 sec
Problem solved and no reboot required.....this is AIX after all! :)
AIX 7 Open Beta is live - it's time to test!
OK, it's official. The AIX 7 Open Beta is now available for testing!
An overview of the program is available here:
Essentially this program is designed to give IBM customers, ISVs and IBM BPs the opportunity to gain early experience with the latest release of AIX prior to general availability. This is a great time to join forces and help IBM mould the next generation of the AIX OS.
I got involved in the AIX 6 Open Beta back in 2007. It was a worthwhile experience. The time I spent learning new features like WPARs and RBAC, put me in a good position when it came time to actually implement these outside of my lab environment. It was also a good opportunity to provide feedback to the IBM AIX development community. Several AIX developers monitored the comments/questions in the Beta Forum and provided advice (and sometimes fixes) for known (and unknown!) issues with the beta release. It also provided the developers with plenty of real world feedback that they could take back to the labs, long before the product was officially released. This certainly helped fix bugs and improve certain enhancements before customers starting using the OS in their computing environments.
The AIX 7 beta code is available for download from here:
The user forum, where you can provide feedback and ask questions, is here:
I recommend you download the documentation and start by reading the Getting Started guide. The documentation is available here:
The Getting Started guide provides useful information that you will need to know before attempting to install the OS. For example, the beta code will run on any IBM System p, eServer pSeries or POWER system that is based on PPC970, POWER4, POWER5, POWER6 or POWER7 processors.
The guide also describes what new functionality has been included in this release of the beta. If this program is anything like the AIX 6 beta, there may be more than one release of the code, with further enhancements available in each release. The new function in this release includes:
A. AIX 5.2 Workload Partitions for AIX 7.
B. Removal of WPAR local storage device restrictions.
C. Domain support in Role Based Access Control (RBAC).
D. Etherchannel enhancements across stacked switches in IEEE 802.3ad mode.
E. NIM thin server support for IPv6 and NFSv4.
I'm interested in testing A, B and D. Here's why.
A. AIX 5.2 Workload Partitions for AIX 7.
AIX 5.2 Workload Partitions for AIX 7 provides the capability to create a WPAR running AIX 5.2 TL10 SP8. This allows a migration path for an AIX 5.2 system running on old hardware to move to POWER7. All that is required is to create a mksysb image of the AIX 5.2 system and then provide this image when creating the WPAR. The WPAR must be created on a system running AIX 7 on POWER7 hardware. This is a very interesting feature, one that I am eager to test.
B. Removal of WPAR local storage device restrictions.
AIX 7 will allow for exporting a virtual or physical fibre channel adapter to a WPAR. The WPAR will essentially own the physical adapter and its child devices. This will allow for SAN storage devices to be directly assigned to the WPAR's FC adapter(s). This means it will not be necessary to provision the storage in the Global environment first and then export it to the WPAR. This is also interesting as we may now be able to assign SAN disk to a WPAR for both rootvg and data volume groups. Maybe even FC tape devices within a WPAR will work?
D. Etherchannel enhancements in 802.3ad mode.
There are some enhancements to AIX 7 EtherChannel support for 802.3AD mode. The enhancement makes sure that the link is LACP ready before sending data packets. If I'm interpreting this correctly, this will ensure that the aggregated link is configured appropriately. If it's not, it will provide an error in the AIX errpt stating that the link is not configured correctly. This can help avoid situations where the AIX EtherChannel is configured but the Network Switch is not. At present there is very little an AIX administrator can do as the link will appear to be functioning even if the Switch end has not been configured for an aggregated link.
And the most important point in the Getting Started guide has to be, how to install the AIX beta code! An ISO image of the code is provided for download. The installation steps are straightforward as the image can be installed via a DVD device. Using a media repository on a VIO server could be one way to accomplish this task. Unfortunately, there is no mention of NIM install support yet. Here are the basic steps from the guide:
Installing the AIX Open Beta Driver
The AIX 7 Open Beta driver is delivered by restoring a system backup (mksysb) of the code downloaded via DVD ISO image from the AIX Open Beta web-site.
Once you have downloaded and created the AIX 7 Open Beta media (as described above) follow the following steps to install the 'mksysb'.
1. Put the DVD of the AIX Open Beta in the DVD drive. A series of screens/menus will be displayed. Follow the instruction on the screens and make the following selections:
• Type 1 and press Enter to have English during install.
• Type 1 to continue with the install.
• Type 1 to Start Install Now with Default Setting.
2. The system will start installing the AIX 7.0 BETA.
3. Upon completion of the install, the system will reboot. You can then login as "root", no password is required.
Next I recommend that you take a look at the Release Notes. It provides a few bits of information that may come in handy when planning for the install, such as:
· The Open Beta code is being delivered via an "mksysb" install image. Migration installation is not supported with the open beta driver.
· The open beta driver does not support IBM Systems Director Agent.
· When installing in a disk smaller than 15.36 GB, the following warning is displayed: A disk of size 15360 was specified in the bosinst.data file, but there is not a disk of at least that size on the system. You can safely ignore this warning.
· The image is known to install without issues on an 8 GB disk.
· oslevel output shows V7BETA.
By the way, if you are unable to find a spare system or LPAR on which to install the beta, perhaps you can consider using the IBM Virtual Loaner Program (VLP). They are planning to support LPARs running the AIX 7 Open Beta starting from July 17th. I use the VLP all the time and found it be a fantastic way to try new things without the need for, or expense of, my own IBM POWER system. There are some drawbacks, such as not having access to your own dedicated hardware, HMC, VIO server, NIM master, but still it's great if you just want to test something on an AIX system.
I'll report back once I've got my AIX 7 Open Beta system up and running!
rmsock on AIX.
I like lsof.
Whenever I'm building a new AIX system I always make sure to install it. I really like the fact that I can quickly list processes that are connected to TCP and UDP ports on my system. For example, to check for the current SSH connections on my system I can run lsof and check port 22 (SSH). Immediately I have a good idea of the existing SSH sessions/connections. I can also check to see if the SSH server (sshd daemon) is running and listening (LISTEN) on my AIX partition.
# lsof -i tcp:22
Value of I :77 np:0
sshd 340040 root 3u IPv4 0xf100020002e4a3b0 0t277394 TCP aix01:ssh->172.29.131.16:49948 (ESTABLISHED)
sshd 385184 u0008904 3u IPv4 0xf100020002e4a3b0 0t277394 TCP aix01:ssh->172.29.131.16:49948 (ESTABLISHED)
sshd 409808 root 3u IPv4 0xf100020001c48bb0 0t0 TCP *:ssh (LISTEN)
But sometimes I work on systems that don't have lsof installed. It may not be practical or appropriate for me to install it either. So I have to find another tool (or tools) that will do something similar.
Of course, I could use netstat to check that a server daemon was listening on a particular TCP port and view any established connections. But this doesn't give me the associated process id's.
$ netstat -a | grep -i ssh
tcp4 0 0 *.ssh *.* LISTEN
tcp4 0 48 aix01.ssh 172.29.131.16.50284 ESTABLISHED
Fortunately, the rmsock command can provide that information. So if I wanted to find the process id for the sshd daemon that is listening on my system I'd do the following. First I need to find the socket id using netstat*.
# netstat -@aA | grep -i ssh | grep LIST | grep Global
Global f1000700049303b0 tcp4 0 0 *.ssh *.* LISTEN
Then I can use rmsock to discover the process id associated with the sockect. In this case it's PID 282700.
$ rmsock f1000200003e9bb0 tcpcb
The socket 0x3e9808 is being held by proccess 282700 (sshd).
Unlike what its name implies, rmsock does not remove the socket, if it is being used by a process. It just reports the process holding the socket. Note that the second argument of rmsock is the protocol. It's tcpcb in this example to indicate that the protocol is TCP. The results of the command are also logged to /var/adm/ras/rmsock.log.
# tail /var/adm/ras/rmsock.log
socket 0xf100020001c45008 held by process 434420 (writesrv) can't be removed.
socket 0xf100020000663008 held by process 418040 (java) can't be removed.
socket 0xf1000200012ad008 held by process 418040 (java) can't be removed.
socket 0xf100020000dec008 held by process 163840 (inetd) can't be removed.
socket 0xf100020000deb008 held by process 163840 (inetd) can't be removed.
socket 0xf10002000016f808 held by process 192554 (snmpdv3ne) can't be removed.
socket 0xf100020001c51808 held by process 442596 (dtlogin) can't be removed.
socket 0xf1000200012a4008 held by process 418040 (java) can't be removed.
socket 0xf100020000666008 held by process 315640 (java) can't be removed.
socket 0xf100020000deb808 held by process 163840 (inetd) can't be removed.
The following links have more information on the rmsock command:
*Note: In my example I specified the @ symbol with the netstat command. I also grep'ed for the string Global. You may have to do the same if you have WPARs running on your system. In my case I have two active WPARs who both have their own sshd process. My Global environment also has an sshd process. So in total there are three sshd daemons that I can view from the Global environment. By specifiying the @ symbol with netstat, I can quickly determine which process belongs to the Global environment and those that exist within each WPAR.
# netstat -aA | grep -i ssh | grep LISTEN
f1000700049303b0 tcp4 0 0 *.ssh *.* LISTEN
f100070000198bb0 tcp4 0 0 *.ssh *.* LISTEN
f100070000194bb0 tcp4 0 0 *.ssh *.* LISTEN
# netstat -@aA | grep -i ssh | grep LISTEN
Global f1000700049303b0 tcp4 0 0 *.ssh *.* LISTEN
wpar1 f100070000198bb0 tcp4 0 0 *.ssh *.* LISTEN
wpar2 f100070000194bb0 tcp4 0 0 *.ssh *.* LISTEN
A 'first look' at creating an AIX 5.2 WPAR on AIX 7.
I created my first 'versioned' AIX 5.2 WPAR on AIX 7 today. To get me started, I followed the steps in the following guide from the AIX 7 Open Beta On-line Information Centre:
In order to create the WPAR, I needed an AIX 5.2 mksysb file to supply to the mkwpar command.
Fortunately, I just happened to have an old AIX 5.2 mksysb image in my archives!
I then executed the following command to build the WPAR:
# mkwpar -n wpar1 -C -B /home/cgibson/AIX5202_64bit-mksysb
The flags to the command are:
-n wparname
Specifies the name for the workload partition to be created. You must specify a name, either using the -n flag or in a specification file using the -f flag, unless the -p name or both –w and -o flags are used.
-B wparbackupdevice
Specifies a device containing a workload partition backup image. This image is used to populate the workload partition file systems. The wparBackupDevice parameter is a workload partition image that is created with the savewpar, mkcd, or mkdvd command. The -B flag is used by the restwpar command as part of the process of creating a workload partition from a backup image.
Creates a versioned workload partition. This option is valid only when additional versioned workload partition software has been installed.
I was then able to start my new AIX 5.2 WPAR successfully!
# startwpar -v wpar1
Starting workload partition wpar1.
Mounting all workload partition file systems.
Mounting /wpars/wpar1
Mounting /wpars/wpar1/home
Mounting /wpars/wpar1/mksysb
Mounting /wpars/wpar1/nre/opt
Mounting /wpars/wpar1/nre/sbin
Mounting /wpars/wpar1/nre/usr
Mounting /wpars/wpar1/opt
Mounting /wpars/wpar1/proc
Mounting /wpars/wpar1/tmp
Mounting /wpars/wpar1/usr
Mounting /wpars/wpar1/usr/local
Mounting /wpars/wpar1/var
Mounting /wpars/wpar1/var/log
Mounting /wpars/wpar1/var/tsm/log
Loading workload partition.
Exporting workload partition devices.
Exporting workload partition kernel extensions.
Starting workload partition subsystem cor_wpar1.
0513-059 The cor_wpar1 Subsystem has been started. Subsystem PID is 8388822.
Verifying workload partition startup.
Return Status = SUCCESS.
The WPAR was now in an active state and the associated file systems were mounted (as shown from the Global environment).
# lswpar
Name State Type Hostname Directory RootVG WPAR
wpar1 A S wpar1 /wpars/wpar1 no
# mount | grep wpar
/dev/lv00 /wpars/wpar1 jfs Jul 26 20:13 rw,log=/dev/loglv00
/dev/lv01 /wpars/wpar1/home jfs Jul 26 20:13 rw,log=/dev/loglv00
/dev/lv02 /wpars/wpar1/mksysb jfs Jul 26 20:13 rw,log=/dev/loglv00
/opt /wpars/wpar1/nre/opt namefs Jul 26 20:13 ro
/sbin /wpars/wpar1/nre/sbin namefs Jul 26 20:13 ro
/usr /wpars/wpar1/nre/usr namefs Jul 26 20:13 ro
/dev/lv03 /wpars/wpar1/opt jfs Jul 26 20:13 rw,log=/dev/loglv00
/proc /wpars/wpar1/proc namefs Jul 26 20:13 rw
/dev/lv04 /wpars/wpar1/tmp jfs Jul 26 20:13 rw,log=/dev/loglv00
/dev/lv05 /wpars/wpar1/usr jfs Jul 26 20:13 rw,log=/dev/loglv00
/dev/lv06 /wpars/wpar1/usr/local jfs Jul 26 20:13 rw,log=/dev/loglv00
/dev/lv07 /wpars/wpar1/var jfs Jul 26 20:13 rw,log=/dev/loglv00
/dev/lv08 /wpars/wpar1/var/log jfs Jul 26 20:13 rw,log=/dev/loglv00
/dev/lv09 /wpars/wpar1/var/tsm/log jfs Jul 26 20:13 rw,log=/dev/loglv00
I was curious what the WPAR environment was going to look like, so I used clogin to access it and run a few commands.
From the Global environment I confirmed I was indeed on an AIX 7 system.
# uname -W
# oslevel
From within the WPAR, I confirmed that I was indeed running AIX 5.2! Wow!
# clogin wpar1
wpar1 : / # oslevel
And I could see all 8 logical CPUs (4 hardware threads per POWER7 processor i.e. SMT-4).
wpar1 : / # sar -P ALL 1 5
AIX wpar1 2 5 00F602734C00 07/26/10
wpar1 configuration: @lcpu=8 @mem=4096MB @ent=0.50
20:22:20 cpu %usr %sys %wio %idle physc %entc
20:22:21 0 7 78 1 14 0.01 0.0
1 1 70 0 29 0.01 0.0
2 0 1 0 99 0.00 0.0
3 0 0 0 100 0.01 0.0
4 0 35 0 65 0.00 0.0
7 0 28 0 72 0.00 0.0
- 0 3 0 97 0.03 0.0
I noticed an interesting device in the lscfg output.
wpar1 : / # lscfg
The following resources are installed on the machine.
+/- = Added or deleted from Resource List.
* = Diagnostic support not available.
Model Architecture: chrp
Model Implementation: Multiple Processor, PCI bus
+ sys0 System Object
* wio0 WPAR I/O Subsystem
Also noticed some new and interesting mount points, for example /nre/opt.
wpar1 : / # df
Filesystem 512-blocks Free %Used Iused %Iused Mounted on
Global 131072 99928 24% 1424 5% /
Global 131072 126704 4% 70 1% /home
Global 1048576 1015560 4% 17 1% /mksysb
Global 786432 428904 46% 7331 14% /nre/opt
Global 458752 88400 81% 10020 47% /nre/sbin
Global 4980736 24872 100% 53698 87% /nre/usr
Global 131072 63800 52% 1640 11% /opt
Global - - - - - /proc
Global 131072 125080 5% 52 1% /tmp
Global 1572864 165744 90% 23183 12% /usr
Global 524288 494464 6% 154 1% /usr/local
Global 131072 111512 15% 493 4% /var
Global 262144 253744 4% 28 1% /var/log
Global 131072 126832 4% 20 1% /var/tsm/log
I did have one minor problem when I first tried to start my WPAR, but that issue was quickly resolved by the AIX developers on the AIX 7 Open Beta Forum.
Such a fast response to my query demonstrates IBMs commitment to the Open Beta program.
I was notified this morning that I've been selected to work on an upcoming AIX Redbook in Austin, Texas. I'm really looking forward to it.
AIX Differences Guide, PW-0001-R01: http://www.redbooks.ibm.com/residents.nsf/50da6a28780ffa688525701b004a4f21/9032e219ecda947c852576f900691af5?OpenDocument
Follow me on Twitter: http://twitter.com/cgibbo
AIX or Linux on POWER?
IBM made some announcements today relating to their latest POWER7 server offerings. The new line of systems includes new entry level systems and the highly anticipated high-end system, the POWER7 795! They also officially outlined some of the new features available in AIX 7.1. You can review the details here. I've discussed some of these new features here and here. The official AIX 7.1 announcement details are available here.
The announcement got me thinking about my recent customer engagements and why some have chosen to deploy AIX into their IBM POWER environments, while others are considering a Linux on POWER solution.
I've found that it usually comes down to a skills decision more than anything else. Most customers are happy to either continue working with AIX (if they are existing AIX users) or migrate from another UNIX OS to AIX. I've seen very few customers actually migrate to Linux on POWER, but I've worked with several that have seriously considered it. Those that have chosen to deploy Linux are doing so purely because they have in-house Linux skills. They are concerned that migrating to AIX may be too big a jump for their technical staff. I find this thinking interesting, as most of the customers I've dealt with who run other UNIX OS's like Tru64, Solaris or HP-UX are more than happy to migrate to AIX. They believe the move is relatively minor and doesn't require massive re-training of their UNIX admins. I tend to agree.
For me, AIX is my preferred "Enterprise class" UNIX Operating System. Notice I'm prefacing this with the words Enterprise class. Don't get me wrong, I have worked with Linux systems in both small and large customer environments. It is a great OS. But I've found that it really only fits into environments that have a relatively small number of users and where significant downtime can be tolerated for things like operating system maintenance. This doesn't fit the Enterprise class of UNIX server OS's that I'm thinking of here. When I contemplate the word Enterprise, I think of servers and operating systems that can respond to business demands in terms of performance, reliability, stability and availability. An Enterprise UNIX can provide all of these things without compromise. Linux can offer performance and reliability (in my opinion). However, from what I've seen, it lacks features & functions in the areas of stability and availability. AIX on the other hand ticks all the boxes. Again, this is just my opinion based on my experiences with both AIX and Linux in the Enterprise landscape. Others will no doubt have their own experiences that may or may not match my own.
So when I'm designing an Enterprise UNIX server environment for a customer, I always start with an AIX on POWER base. If the customer wants Linux, sure I can look at that too, but I strongly recommend AIX as my preferred choice for large systems. Most of my customers are running relatively large SAP/Oracle systems. AIX on POWER is a great combination for large Enterprise systems. If you need to deploy large database systems that must service tens of thousands of users (like a big SAP system), then I believe AIX is the perfect OS on which to provide a platform for these large scale systems.
AIX is a very mature and powerful UNIX OS. It has been a major player in the UNIX server market for over 20 years (as shown below). Some people are just not aware of how mature, robust and stable the AIX OS has become over the years. There are many impressive aspects of the OS in the areas of performance, scalability, reliability, management and administration.
Just looking at some of the administration capabilities built into AIX are enough for me to always recommend AIX over Linux (or any other UNIX OS), when it comes to large Enterprise servers.
For example, the System Management Interface Tool (SMIT) can make the UNIX admins life a lot simpler. This is an interactive tool that is part of the AIX OS. Almost all tasks that an AIX administrator may need to perform can be executed using this tool. It is a text-based tool (there is also an X interface but I recommend sticking with the text-based menus). Everything it does, it does through standard AIX commands and Korn shell functions. This feature is especially useful when you need to automate a repetitive task; you can have SMIT create the proper command-line sequence, and you can then use those commands in your own script. My compatriot, Anthony English, has a nice intro to SMIT on his AIX blog.
The AIX Logical Volume Manager (LVM) is built into the OS, for free. AIX LVM helps UNIX system administrators manage their storage in a very flexible manner. LVM allows logical volumes to span multiple physical volumes. Data on logical volumes appears to be contiguous to the user, but might not be contiguous on the physical volume. This allows file systems, paging space, and other logical volumes to be resized or relocated, span multiple physical volumes, and have their contents replicated for greater flexibility and availability. It provides capabilities for mirroring data across disks, migrating data across disks & storage subsystems, expand/shrink filesystems and more.......all of which can be performed dynamically.....no downtime required. The concept, implementation and interface to the AIX LVM is one of a kind. All of its features support the 'continuous availability' philosophy.
One of the biggest reasons that I love AIX over Linux is the mksysb. It's built into the OS and allows you to create a bootable image of your AIX system. This image can be used to restore a broken AIX system or for cloning other systems. The cloning feature is truly amazing. You can take an image created on a low-end system and deploy it on any POWER system, all the way up to the high-end POWER boxes. This simplifies the installation and cloning processing when you need to install and manage many AIX LPARs. By using an SOE mksysb image you can deploy consistent AIX images across your Enterprise POWER server environment.
This brings me to another wonderful feature of AIX, the Network Installation Manager (NIM). NIM is powerful network installation tool (comparable to Linux Kickstart). Using NIM you can backup/restore, update and upgrade one or more AIX systems either individually or simultaneously. This can all be achieved over a network connection, removing the need for handling physical installation media forever.
Another fine example of AIXs superior OS management tools is multibos. This tool allows an AIX administrator to create and maintain two separate, bootable instances of the AIX OS within the same root volume group (rootvg). This second instance of rootvg is known as a standby Base Operating System (BOS) and is an extremely handy tool for performing AIX TL and Service Pack (SP) updates. Multibos lets you install, update and customize a standby instance of the AIX OS without impacting the running and active production instance of the AIX OS. This is valuable in environments with tight maintenance windows.
When it comes to upgrading the OS to a new release of AIX, the nimadm utility can assist the administrator greatly in this task. The nimadm utility offers several advantages. For example, a system administrator can use nimadm to create a copy of a NIM client's rootvg and migrate the disk to a newer version or release of AIX. All of this can be done without disruption to the client (there is no outage required to perform the migration). After the migration is finished, the only downtime required will be a single scheduled reboot of the system.
AIX 6.1 introduced new capability that most UNIX operating systems are still working on. Concurrent Updates of the AIX Kernel.....without a reboot! IBM is always working hard at making AIX an OS that can provide continuous availability, even if it needs to be patched. AIX now has the ability to update certain kernel components and kernel extensions in place, without needing a system reboot. In addition, concurrent updates can be removed from the system without needing a reboot. Can you do this with other UNIX OSs?
And that's just some of the features that make AIX the only UNIX OS that I recommend for Enterprise systems. There are many more tools and features that I couldn't live without (like alt_disk_install, savevg, installp, WPARs, etc, the list goes on). If you are new to AIX and you are considering what your next UNIX OS should be, then I recommend you take a very close look at AIX.
And finally, the support provided by IBM, is first class. Whenever I've needed assistance with an AIX issue or query, I have always received timely, professional and useful advice. On the rare occasions where I've uncovered a new bug, IBM AIX support have always been quick to provide me with an interim fix to resolve or workaround a problem. That's the sort of support you'd expect for an Enterprise UNIX OS, isn't it? What's the support like from your current UNIX (or Linux) OS vendor?
Linux is still a viable UNIX operating system. However, I think it's more suited to certain workloads like small to medium size mail, web and other utility servers and services. AIX, however, would be my platform of choice for my 10TB Oracle database running SAP ERP, not just for performance reasons, but primarily because of the system administration features of AIX that allow me to support and manage the system without impacting my customers or enforcing reboots/outages whenever I need to change something on the system.
All IBM need to do is create their own Linux distribution (Blue Linux perhaps?) that has all the features of AIX built in and then I'm sold! But why would they? We already have AIX.
Follow me on Twitter: http://twitter.com/cgibbo
Renaming devices in AIX 7.1
I remember reading this interesting document about some challenges renaming AIX devices.
This poor guy had to jump through some hoops to rename some disk and network devices in a clustered AIX (Oracle RAC) environment.
This comment stayed with me:
"The easy way to get network adapters and disk names to match between systems would be to run something like
# chdev –l old_name –n new_name
Except, this command option does NOT exist."
Since then I've always been on the "look out" for an easy way to rename devices in AIX.
In particular in large clustered environments where it is sometimes very important to have the same disk and network device names in sync across all nodes in a cluster. And besides, it's a lot easier to verify a cluster configuration if the hdisk names are all the same. Matching PVIDs works but it requires a lot more effort! For example, knowing that hdisk123 is the same device on all nodes makes life easier than scanning lspv output for a PVID like 00f6048868b4gead. Of course you can script things to make this easier but it would be great if you didn't need to do this and that there was a way to rename devices as needed, without resorting to unsupported methods.
Well, this is no longer an issue for AIX.
Starting with AIX 7.1, you can now easily rename devices. A new command called rendev was introduced to allow AIX administrators to rename devices as required.
From the man page:
The rendev command enables devices to be renamed. The device to be renamed, is specified with the -l flag, and the new desired name is specified with the -n flag.
The new desired name must not exceed 15 characters in length. If the name has already been used or is present in the /dev directory, the operation fails. If the name formed by appending the new name after the character r is already used as a device name, or appears in the /dev directory, the operation fails.
If the device is in the Available state, the rendev command must unconfigure the device before renaming it. This is similar to the operation performed by the rmdev -l Name command. If the unconfigure operation fails, the renaming will also fail. If the unconfigure succeeds, the rendev command will configure the device, after renaming it, to restore it to the Available state. The -u flag may be used to prevent the device from being configured again after it is renamed.
Some devices may have special requirements on their names in order for other devices or applications to use them. Using the rendev command to rename such a device may result in the device being unusable. Note: To protect the configuration database, the rendev command cannot be interrupted once it has started. Trying to stop this command before completion, could result in a corrupted database.
This functionality was required to help simplify cluster device configuration with the new Cluster Aware AIX (CAA) feature in AIX 7.1.
I'll be blogging about CAA in a future post.
Here are some examples of using the rendev command on AIX 7.1 (GA) system. In the first example I will rename hdisk3 to hdisk300. Note: hdisk3 is not in use (busy).
If the disk had been allocated to a volume group, I would have needed to varyoff the volume group first.
# lspv
hdisk0 00f61ab2f73e46e2 rootvg active
hdisk1 00f61ab20bf28ac6 None
hdisk2 00f61ab2202f7c0b None
hdisk4 00f61ab20b97190d None
hdisk3 00f61ab2202f93ab None
# rendev -l hdisk3 -n hdisk300
# lspv
hdisk0 00f61ab2f73e46e2 rootvg active
hdisk1 00f61ab20bf28ac6 None
hdisk2 00f61ab2202f7c0b None
hdisk4 00f61ab20b97190d None
hdisk300 00f61ab2202f93ab None
Next, I'll rename a virtual SCSI adapter. I renamed vscsi0 to vscsi2. Note: I placed the adapter, vscsi0, in a Defined state before renaming the device.
# rmdev -Rl vscsi0
# lsdev -Cc adapter
ent0 Available Virtual I/O Ethernet Adapter (l-lan)
ent1 Available Virtual I/O Ethernet Adapter (l-lan)
vsa0 Available LPAR Virtual Serial Adapter
vscsi0 Defined Virtual SCSI Client Adapter
vscsi1 Available Virtual SCSI Client Adapter
# rendev -l vscsi0 -n vscsi2
# lsdev -Cc adapter
ent0 Available Virtual I/O Ethernet Adapter (l-lan)
ent1 Available Virtual I/O Ethernet Adapter (l-lan)
vsa0 Available LPAR Virtual Serial Adapter
vscsi1 Available Virtual SCSI Client Adapter
vscsi2 Defined Virtual SCSI Client Adapter
Now I'll rename a network adapter from ent0 to ent10. I bring down the interface before changing the device name
# lsdev -Cc adapter
ent0 Available Virtual I/O Ethernet Adapter (l-lan)
ent1 Available Virtual I/O Ethernet Adapter (l-lan)
vsa0 Available LPAR Virtual Serial Adapter
vscsi1 Available Virtual SCSI Client Adapter
vscsi2 Defined Virtual SCSI Client Adapter
# ifconfig en0
en0: flags=1e080863,480<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),CHAIN>
inet 10.1.20.19 netmask 0xffff0000 broadcast 10.153.255.255
tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1
# ifconfig en0 down detach
# rendev -l ent0 -n ent10
# lsdev -Cc adapter
ent1 Available Virtual I/O Ethernet Adapter (l-lan)
ent10 Available Virtual I/O Ethernet Adapter (l-lan)
vsa0 Available LPAR Virtual Serial Adapter
vscsi1 Available Virtual SCSI Client Adapter
vscsi2 Defined Virtual SCSI Client Adapter
# rendev -l en0 -n en10
# chdev -l en10 -a state=up
en10 changed
# mkdev -l inet0
inet0 Available
# ifconfig en10
en10: flags=1e080863,480<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),CHAIN>
inet 10.1.20.19 netmask 0xffff0000 broadcast 10.153.255.255
tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1
If you want to be creative you can rename devices to anything you like (as long as it's not more than 15 characters). For example I'll rename vscsi2 to myvscsiadapter.
# rendev -l vscsi2 -n myvscsiadapter
# lsdev -Cc adapter
ent1 Available Virtual I/O Ethernet Adapter (l-lan)
myadapter Available Virtual I/O Ethernet Adapter (l-lan)
myvscsiadapter Defined Virtual SCSI Client Adapter
vsa0 Available LPAR Virtual Serial Adapter
vscsi1 Available Virtual SCSI Client Adapter
And in the last example I'll demonstrate changing virtual SCSI adapter device names on a live system.
This is single disk system (hdisk0), with two vscsi adapters.
# lspv
hdisk0 00f6048868b4deee rootvg active
# lspath
Enabled hdisk0 vscsi0
Enabled hdisk0 vscsi1
# lsdev -Cc adapter
ent0 Available Virtual I/O Ethernet Adapter (l-lan)
ent1 Available Virtual I/O Ethernet Adapter (l-lan)
vsa0 Available LPAR Virtual Serial Adapter
vscsi0 Available Virtual SCSI Client Adapter
vscsi1 Available Virtual SCSI Client Adapter
We ensure the adapter is in a Defined state before renaming it. This will fail otherwise.
# rmdev -Rl vscsi1
vscsi1 Defined
# lsdev -Cc adapter | grep vscsi
vscsi0 Available Virtual SCSI Client Adapter
vscsi1 Defined Virtual SCSI Client Adapter
Now we rename the adapter vscsi1 to vscsi3.
# rendev -l vscsi1 -n vscsi3
# lsdev -Cc adapter | grep vscsi
vscsi0 Available Virtual SCSI Client Adapter
vscsi3 Defined Virtual SCSI Client Adapter
That was easy enough. Now I need to bring the adapter and path online with cfgmgr. The lspath output displays an additional path to vscsi3.
# lspath
Enabled hdisk0 vscsi0
Defined hdisk0 vscsi1
# cfgmgr
Method error (/etc/methods/cfgscsidisk -l hdisk0 ):
0514-082 The requested function could only be performed for some
of the specified paths.
# lspath
Enabled hdisk0 vscsi0
Defined hdisk0 vscsi1
Enabled hdisk0 vscsi3
Now I need to remove the old path to vscsi1. The path to vscsi3 is now Enabled. The adapter, vscsi3, is in an Available state. All is good.
# rmpath -l hdisk0 -p vscsi1 -d
path Deleted
# lspath
Enabled hdisk0 vscsi0
Enabled hdisk0 vscsi3
# lsdev -Cc adapter | grep vscsi
vscsi0 Available Virtual SCSI Client Adapter
vscsi3 Available Virtual SCSI Client Adapter
The same steps need to be repeated for the vscsi0 adapter. This is renamed to vscsi2.
# rmdev -Rl vscsi0
vscsi0 Defined
# lsdev -Cc adapter | grep vscsi
vscsi0 Defined Virtual SCSI Client Adapter
vscsi3 Available Virtual SCSI Client Adapter
# rendev -l vscsi0 -n vscsi2
# lsdev -Cc adapter | grep vscsi
vscsi2 Defined Virtual SCSI Client Adapter
vscsi3 Available Virtual SCSI Client Adapter
# lspath
Defined hdisk0 vscsi0
Enabled hdisk0 vscsi3
# cfgmgr
Method error (/etc/methods/cfgscsidisk -l hdisk0 ):
0514-082 The requested function could only be performed for some
of the specified paths.
# lspath
Defined hdisk0 vscsi0
Enabled hdisk0 vscsi2
Enabled hdisk0 vscsi3
# rmpath -l hdisk0 -p vscsi0 -d
path Deleted
# cfgmgr
# lspath
Enabled hdisk0 vscsi2
Enabled hdisk0 vscsi3
That's it. Both adapters have been renamed while the system was in use. No downtime required.
# lsdev -Cc adapter | grep vscsi
vscsi2 Available Virtual SCSI Client Adapter
vscsi3 Available Virtual SCSI Client Adapter
# lspath
Enabled hdisk0 vscsi2
Enabled hdisk0 vscsi3
Please refer to the AIX 7.1 command reference for more information on this new command:
And by the way, here's a draft of what I've been working on for the last 5 weeks:
IBM AIX Version 7.1 Differences Guide:
Follow me on Twitter:
Cluster Aware AIX
cggibbo 270000TMUJ Tags:  caa_private0 event caa aware ahafs cluster mkcluster aix 7.1 cldisk1 lscluster 11 Comments 75,584 Views
Using Cluster Aware AIX with AIX 7.1 you can create a cluster of AIX nodes. This interconnected cluster of nodes can immediately provide several capabilities, such as:
· Clusterwide event management.
· Communication and storage events:
o Node UP and node DOWN
o Network adapter UP and DOWN
o Network address change
o Point-of-contact UP and DOWN
o Disk UP and DOWN
o Predefined and user-defined events
· Clusterwide storage naming service
· Clusterwide command distribution
· Clusterwide communication making use of networking and storage connectivity
All of the capabilities of CAA are built into the AIX Version 7.1 operating system. CAA is essentially a set of tools and services embedded into AIX that help to manage a cluster of AIX nodes and assist in running cluster software on AIX. Cluster products, such as PowerHA and RSCT from IBM, can utilise CAA to simplify the configuration, management and monitoring of an AIX cluster.
CAA does not form a cluster by itself. It is a tool set. There is no notion of quorum. (If 20 nodes of a 21 node cluster are down, CAA still runs on the remaining node). CAA does not eject nodes from a cluster. CAA provides tools to fence a node but never fences a node and will continue to run on a fenced node.
As you can see in the following diagram, cluster products, like PowerHA, will integrate with CAA to help form and manage highly available clusters.
So you will still need some form of cluster product, either from IBM or another vendor, in order to build a cluster that provides high availability capabilities like node failover/takeover.
What follows are some of the most important "snippets" from the CAA documentation (IMHO), available in the AIX 7.1 Information Centre (see the link at the bottom of the page).
Just like any cluster, each node that is added to a cluster by using CAA must have common storage devices available, for example via SAN storage device. These storage devices are used for the cluster repository disk and for any clustered shared data disks.
The Storage Naming Service provides a global device view across all the nodes in the cluster. The Storage Naming Service also provides a single global device name for a disk from any node in the cluster. The global device name, for example, cldisk1, refers to the same physical disk from any node in the cluster.
The cluster repository disk is used as the central repository for the cluster configuration data. The cluster repository disk must be accessible from all nodes in the cluster and is a minimum of 10 GB in size. Given the importance of the cluster configuration data, the cluster repository disk should be backed up by a redundant and highly available storage configuration. Even though the cluster repository disk is visible as a disk device to AIX, it should be treated as a special device for the cluster. The use of LVM commands is not supported when used on a cluster repository disk. The AIX LVM commands are designed as single node administrative commands, and are not applicable in a clustered configuration.
The cluster repository disk is renamed to a private device name (caa_private0). A raw section of the disk and a section of the disk that contains a special volume group and special logical volumes are used during cluster operations.
A multicast address is used for cluster communications between the nodes in the cluster.
These are configured automatically during the creation of the cluster. These nodes support cluster monitoring of events and cluster configuration attributes.
Scalable reliable multicasting is implemented in the cluster with a special gossip protocol over the multicast address. The gossip protocol determines the node configuration and then transmits the gossip packets over all available networking and storage communication interfaces (either Fibre Channel and/or SAS adapters). If no storage communication interfaces are configured, only the traditional networking interfaces are used.
When you first configure CAA, the following actions are performed:
• The cluster is created using the mkcluster command.
• The cluster configuration is written to the raw section of the cluster repository disk.
• Primary and secondary database nodes are selected from the list of candidate nodes in the mkcluster command.
• Special volume groups and logical volumes are created on the cluster repository disk.
• Cluster file systems are created on the special volume group.
• The cluster repository database is created on both primary and secondary nodes.
• The cluster repository database is started.
• Cluster services are made available to other functions in the operating system, such as Reliable Scalable Cluster Technology (RSCT) and PowerHA.
• Storage framework register lists are created on the cluster repository disk.
• A global device namespace is created and interaction with LVM starts for handling associated volume group events.
• A clusterwide multicast address is established.
• The node discovers all of the available communication interfaces.
• The cluster interface monitoring starts.
• The cluster interacts with Autonomic Health Advisory File System (AHAFS) for clusterwide event distribution.
• The cluster exports cluster messaging and cluster socket services to other functions in the operating system, such as Reliable Scalable Cluster Technology (RSCT) and PowerHA.
In the following example I created a two node cluster using CAA tools. First of all, I had to use the mkcluster command to define the cluster nodes, shared storage and repository disk. The node names are 7502lp01 and 7502lp02. The shared data storage disks are hdisk2, hdisk3 and hdisk4. The repository disk, used to house the cluster configuration data, is hdisk1.
# mkcluster -n mycluster -r hdisk1 -d hdisk2,hdisk3,hdisk4 -m 7502lp01,7502lp02
mkcluster: Cluster shared disks are automatically renamed to names such as
cldisk1, [cldisk2, ...] on all cluster nodes. However, this cannot
take place while a disk is busy or on a node which is down or not
reachable. If any disks cannot be renamed now, they will be renamed
later by the clconfd daemon, when the node is available and the disks
are not busy.
You'll notice that the mkcluster command informed me that the cluster shared disks are automatically renamed to cluster disk names like cldisk1. After I'd run the command I noticed something quite interesting and impressive. Prior to configuring the cluster the shared disks on both nodes had names like hdiskX. Afterwards the shared disks had been renamed across both nodes. Now these disks had the same name!
This is going to simplify cluster configuration and management. No more will I need to remove and recreate disks in order to resolve disk naming inconsistencies in a cluster. The lspv output (shown below) from both nodes shows that I have three shared disks (cldisk1, cldisk2 and cldisk3). These would be used for shared data. The disk named caa_private0 is my cluster repository disk. This is used to store and share the cluster configuration data.
# lspv | sort
caa_private0 00f61ab20b97190d caavg_private active
cldisk1 00f61ab20bf28ac6 None
cldisk2 none None
cldisk3 none None
hdisk0 00f61ab2f73e46e2 rootvg active
# lspv | sort
caa_private0 00f61ab20b97190d caavg_private active
cldisk1 00f61ab20bf28ac6 None
cldisk2 none None
cldisk3 none None
hdisk0 00f61ab2895e4cbe rootvg active
Cluster Aware AIX tells you what nodes are in the cluster plus information on those nodes, including state. A special "gossip" protocol is used over the multicast address to determine node information and implement scalable reliable multicast. No traditional heartbeat mechanism is employed. Gossip packets travel over all interfaces, including storage. Immediately after running mkcluster, I was able to query the status of the nodes in my cluster without any further configuration!
# lscluster -m
Calling node query for all nodes
Node query number of nodes examined: 2
Node name: 7502lp01
Cluster shorthand id for node: 1
uuid for node: 3cd9cb00-bf55-11df-b015-6e8dd0af6304
State of node: UP
Smoothed rtt to node: 7
Mean Deviation in network rtt to node: 3
Number of zones this node is a member in: 0
Number of clusters node is a member in: 1
mycluster local 267ce7fc-bf55-11df-a3b9-6e8dd877b814
Number of points_of_contact for node: 1
Point-of-contact interface & contact state
en0 UP
Node name: 7502lp02
Cluster shorthand id for node: 2
uuid for node: d1a46164-bf46-11df-94b3-6e8dd877b814
State of node: UP NODE_LOCAL
Smoothed rtt to node: 0
Mean Deviation in network rtt to node: 0
Number of zones this node is a member in: 0
Number of clusters node is a member in: 1
mycluster local 267ce7fc-bf55-11df-a3b9-6e8dd877b814
Number of points_of_contact for node: 0
Point-of-contact interface & contact state
CAA tells you what interfaces have been discovered on a node plus information on those interfaces, including state.
# lscluster –i –n
Node 7502lp01
Node uuid = 110b2422-7efc-11df-aed7-1612a0003002
Number of interfaces discovered = 1
Interface number 1 en0
ifnet type = 6 ndd type = 7
Mac address length = 6
Mac address = 16.12.a0.0.30.2
Smoothed rrt across interface = 7
Mean Deviation in network rrt across interface = 3
Probe interval for interface = 100 ms
ifnet flags for interface = 0x1e080863
ndd flags for interface = 0x21081b
Interface state UP
Number of regular addresses configured on interface = 1
IPV4 ADDRESS: 22.214.171.124 broadcast 188.8.131.52 netmask 255.255.255.224
Number of cluster multicast addresses configured on interface = 1
IPV4 MULTICAST ADDRESS: 184.108.40.206 broadcast 0.0.0.0 netmask 0.0.0.0
CAA tells you what disks are in the cluster plus information on those disks, including the state and type.
# lscluster –d
Storage Interface Query
Cluster Name: mycluster
Cluster uuid: a1bde416-83b9-11df-b438-1612a0004002
Number of nodes reporting = 2
Number of nodes expected = 2
Node oscar-dev8.austin.ibm.com
Node uuid = 9807465b-3194-f1e5-8a04-1f044bc82593
Number of disk discovered = 2
state : UP
uDid : 533E3E213600A0B8000475C200000E7114BC825930F1818 FAStT03IB
type : CLUSDISK
uUid : 9807465b-3194-f1e5-8a04-1f044bc82593
state : UP
uDid :
uUid : 600a0b80-0047-5d0a-0000-e6094bc82690
type : REPDISK
You can run cluster wide commands on all nodes immediately after configuring your cluster using the clcmd command. No further configuration required.
# clcmd ps -ef
NODE 7502lp01
root 1 0 0 Aug 25 - 0:00 /etc/init
root 655526 1 0 Aug 25 - 0:18 [cimserve]
root 1310840 1 0 Aug 25 - 0:00 /usr/ccs/bin/shlap64
root 2490482 1 0 Aug 25 - 0:42 /usr/sbin/srcmstr
root 3145874 3801262 0 11:17:13 - 0:00 telnetd -a
root 3276994 2490482 0 Aug 25 - 0:10 /usr/sbin/snmpd
NODE 7502lp02
root 1 0 1 Aug 28 - 0:00 /etc/init
root 1441934 1 0 Aug 28 - 0:39 /usr/sbin/syncd 60
root 1638496 1 0 Aug 28 - 0:00 /usr/ccs/bin/shlap64
root 2228252 2556054 0 Aug 28 - 0:00 /usr/sbin/hostmibd
I believe that PowerHA SystemMirror 7.1 will be the first cluster product from IBM to provide high availability on AIX using the new Cluster Aware AIX features. An IBM Redbook Residency recently covered the integration of CAA and PowerHA. I'm eagerly awaiting the first draft of this book so that I can learn how these two components will "play" together.
Refer to the AIX 7.1 Information Centre for more information on Cluster Aware AIX:
Refer to the AIX 7.1 Information Centre for more information on AHAFS and the new AIX Event Infrastructure:
A journey into the world of VLAN tagging and Virtual I/O!
cggibbo 270000TMUJ Tags:  powervm lemal chris aix vlan vio ibm bernard gibson 6 Comments 15,890 Views
I just had to share this entertaining presentation with everyone.
Or the PDF. It's not as interesting or fun, as the animations don't work but you get the idea. I'm sharing the PDF because it appears that you can open this file on Windows fine (in read-only mode) but on a Mac it prompts for a password without an option for opening in read-only. Shame.
It was written by Bernard Lemal. He works for IBM GTS Belgium in AIX support.
It's a humorous look at the life of an IP packet and its journey in the world of VLANs, VLAN tagging and Virtual I/O.
Plenty of interesting and informative commentary along the way.
The part of the IP packet is portrayed by Bernard himself. Take a look at the presentation to see what I mean!
I wish I'd seen a presentation like this when I was first introduced to the concept of VLAN tagging, it might have answered several questions early on.
The next generation Virtual I/O Server!
Interested in learning more about the next generation of Virtual I/O Server? Yes? Then take a look at this presentation.
The following diagram gives you a high-level overview of the classic VIOS compared with the NextGen VIOS. Should be enough to spark your interest!
Happy reading!
Is my POWER7 system AME capable? Is AME supported with SAP in production?
cggibbo 270000TMUJ Tags:  ame gibson power7 sap lparstat hmc memory chris amepat expansion active aix lssyscfg 1 Comment 22,157 Views
A colleague of mine contacted me during the week to ask how one could determine if a POWER7 system was capable of Active Memory Expansion. I thought I'd share my response with everyone who follows my blog.
You can check if the system is capable of providing Active Memory
Expansion (AME), from the HMC command line using the lssyscfg command, as shown here:
Once you've concluded that your POWER7 system is
AME capable, the next step is to check if AME is enabled or disabled for
an LPAR. This is easily done from the AIX command line with the lparstat and/or amepat commands, as shown in the following example:
$ lparstat
-i| grep Exp
;Running lparstat and amepat on a LPAR with AME enabled.
$ lparstat -i | grep -i exp
Memory Mode : Dedicated-Expanded
Target Memory Expansion Factor : 1.00
Target Memory Expansion Size : 6144 MB
$ amepat | grep -i exp
WARNING: Running in no modeling mode.
Active Memory Expansion : Enabled
Target Expanded Memory Size : 6.00 GB
Target Memory
Expansion factor : 1.00
By the
way, did you know that AME is now supported in SAP production environments running
on AIX and POWER7? Both SAP application servers, as well as database servers
running DB2 LUW are supported. Unfortunately there's no support for LPARs with
Oracle RDBMS at this stage. Thanks to George Manousos from IBM Australia for
providing me with this update.
Please refer to the following SAP note.
The SAP note also states:
Specific tests have been successfully executed with SAP ERP 6.0 (ABAP)
workload. The results of those test can be found in the following whitepaper:
As AME is not recommended with large page support, AME will disable AIX
64KB pages by default.
AME monitoring capability in CCMS
Monitoring capabilities are available with saposcol v12.46 for more details see SAP note 710975.
It looks as though SAP have been quick to support AME on AIX/POWER7. This is a great benefit to SAP customers running with PowerVM on the IBM POWER platform. The following screenshot shows the SAP CCMS with AME statistics being reported.
PowerVM Integration into SAP CCMS Monitoring
expanded to include Virtualized Memory Metrics
For those who are not familiar with what AME actually is and how it can benefit you, I suggest you take a look at the IBM AME Wiki site first:
And watch Nigel Griffiths AME movie:
AIX 7.1 – DSM dgetmacs.
I received an email this week from a colleague that worked with me on the NIM Redbook back in 2006. He was experiencing an issue with DSM and NIM. He was attempting to use the dgetmacs command to obtain the MAC address of the network adapters on an LPAR. The command was failing to return the right information.
This was the output from his attempt with dgetmacs:
# dgetmacs -n lpar1
Using an adapter type of "ent".
Could not dsh to node lpar1.
Attempting to use openfirmware method to collect MAC addresses.
Acquiring adapter information from Open Firmware for node lpar1.
dgetmacs: 2760-185 Issuing the command "/opt/ibm/sysmgt/dsm/dsmbin/lpar_netboot -M -n -t ent -J hmc1 -j hmc -F /etc/ibm/sysmgt/dsm/config/my_password_file –A 4 10582FP 8202-E4B" gave a return code of 1. The routine will continue.
No adapters found for node lpar1.
Could not get mac address for this aix node "lpar1"
I experienced this very issue during the writing of the AIX 7.1 Differences Guide Redbook. And given that I was in Austin, sitting in the same building as the AIX development team, I was able to speak with the developers directly about the issue. At that time they provided me with the following workaround.
First they asked me to check the size of the /usr/lib/nls/msg/en_US/IBMhsc.netboot.cat message catalog file.
# ls –l /usr/lib/nls/msg/en_US/IBMhsc.netboot.cat
-rw-r--r-- 1 bin bin 3905 Aug 08 09:54
They were surprised to find that the file appeared to be "too small". They promptly sent me the catalog file from one of their development AIX 7.1 systems. I replaced the file as follows:
# cd /usr/lib/nls/msg/en_US/
# ls -ltr IBMhsc*
-rw-r--r-- 1 bin bin 3905 Aug 08 09:54 IBMhsc.netboot.cat
# cp -p IBMhsc.netboot.cat IBMhsc.netboot.cat.old
# cp /tmp/lpar1/IBMhsc.netboot.cat.new IBMhsc.netboot.cat
# ls -ltr IBMhsc*
-rw-r--r-- 1 bin bin 3905 Aug 08 09:54 IBMhsc.netboot.cat.old
-rw-r--r-- 1 bin bin 26374 Dec 23 11:24 IBMhsc.netboot.cat
This fixed the problem for me during the residency.
So I asked my friend to do the same (after I sent him the message catalog file). He ran the dgetmacs command again and this time it returned the MAC address for all the network adapters in his LPAR. Success!
# dgetmacs -n lpar1
Using an adapter type of "ent".
Could not dsh to node lpar1.
Attempting to use openfirmware method to collect MAC addresses.
Acquiring adapter information from Open Firmware for node lpar1.
# Node::adapter_type::interface_name::MAC_address::location::media_speed::adapter_duplex::UNUSED::install_gateway::ping_status::machine_type::netaddr::subnet_mask
This then allowed him to network boot his LPAR and install AIX 7.1 via NIM.
# nim -o bos_inst -a bosinst_data=noprompt -a source=rte -a installp_flags=agX -a accept_licenses=yes -a spot=spotaix7tl0sp2 -a lpp_source=aix7tl0sp2 lpar1
dnetboot Status: Invoking /opt/ibm/sysmgt/dsm/dsmbin/lpar_netboot lpar1
dnetboot Status: Was successful network booting node lpar1.
I thought this might have been resolved in the latest service pack for AIX 7.1. However, both of us have the latest service pack installed (i.e. TL0 SP2 for AIX 7.1).
# oslevel -s
If anyone encounters this problem, feel free to contact me (firstname.lastname@example.com).
For more information, please refer to section 5.2.7 Using DSM and NIM, in the AIX 7.1 Differences Guide Redbook:


text:
Home >> Electronics >> Camera & Photo >> Accessories >> Lens Adapters & Converters >> Camera Black Metal 58mm Macro Lens Reverse Adapter Ring
Customers who's bought this item also viewed these..
- Household Camera Lens Filter Step Down...
- $3.85
- Free Shipping
|Product Name
|Camera Reverse Adapter Ring
|Ring Diameter
|Total Size
|64 x 10mm/ 2.5" x 0.39" (D*T)
|Package Content
|1 x Reverse Adapter for Canon EOS
- This in Non-OEM product. Accessory Only!
- With this adapter, you can experiment by taking close pictures of various small items such as stamps, coins and insects ect.
- Simply screw the camera adapter on the bayonet of your camera's body, and then screw lens reversely on it.
Other Choice (in stock)
- 28mm-37mm 28mm to 37mm Black Step up Ring Adapter for Camera
- $3.44
- Free Shipping
- Camera Repairing 49mm-55mm Metal Step Up Filter Ring...
- $6.14
- Free Shipping
- 52-46mm 52mm to 46mm Step Down Ring Filter Adapter for...
- $4.81
- Free Shipping
- 27mm-37mm 27mm to 37mm Black Step Up Ring Adapter for Camera
- $5.55
- Free Shipping
- 37mm-34mm 37mm to 34mm Black Step Down Ring Adapter for...
- $4.20
- Free Shipping
- 55mm-82mm 55mm to 82mm Black Step Up Ring Adapter for Camera
- $5.77
- Free Shipping

subset: commoncrawl


text:
First, though, let's talk about how to make the pulled pork itself.
Oven roasted or barbecue pulled pork recipe
There are several ways people make pulled pork, including in the oven or on the barbecue pit. Pulled pork made either of these two ways will give you delicious results, but there is something about barbecue pulled pork that makes it special. However, barbecue pulled pork takes 90 minutes per pound of pork. That means a decent size butt will take 12 hours before it becomes BBQ pulled pork.
Oven roasting the pork butt takes only about an hour per pound before you have delicious pulled pork and more people feel comfortable running their oven for long periods of time rather than their grills. The following recipe gives cook times for both recipes and also uses minimal spices so that the pulled pork may be used on several dishes.
Pulled Pork
1, 6 or 7 pound pork butt
2 tablespoons each salt, black pepper, garlic powder, paprika
2 tablespoons olive oil
2 to 4 cups chicken broth
1. Preheat oven to 300 degrees F.
2. Make shallow incisions in the pork shoulder and coat liberally with the spices.
3. Grease an oven-safe cooking dish with olive oil, then put the pork butt in the baking dish. Pour in about a cup of the broth and put the pork shoulder in the oven.
4. Roast until the pork has reached about 170 degrees in the oven. This will take about 6 to 7 hours, but check the temperature periodically to prevent over-cooking and to add more broth, which will steam the meat.
5. When the pork is done, shred the meat with two forks.
Barbecue (grilled) pulled pork recipe variation
1. Make shallow incisions in the pork shoulder and coat liberally with the spices.
2. Start a fire in your grill. Heap all of the coals in your grill onto one side and put a bowl with the broth on the other side. Oil the slats above the bowl with the olive oil and put the pork on them.
3. Cook for 90 minutes per pound, or until the pork has reached an internal temperature of 170 degrees.
4. Shred with forks.
Now that you have your oven roasted, or barbecue pulled pork, how are you going to fix it?
Meal 1: Pulled Pork Sandwiches with Coleslaw
This is by far the simplest meal, but you spent all day yesterday cooking pulled pork, so you deserve a break. Buy coleslaw, BBQ sauce, and buns. Heat the pork with the BBQ sauce, top with coleslaw, and serve as sandwiches.
Meal 2: Pulled Pork Tacos
This will be a quick meal as well. Reheat the pork with a little olive oil and some enchilada sauce, if you have any. Serve with refried beans, warmed corn tortillas, rice, white cheese, and green salsa.
Meal 3: Pork Fried Rice and Pork Egg Rolls
You should be about at the end of your pulled pork, but you should have enough for this fried rice with homemade egg rolls. To make the fried rice, first start with leftover white rice (or make new if you don't have leftovers). Next, put two tablespoons of oil in a skillet or wok and cook the rice with soy sauce, pulled pork, sliced green onions, and peas.
Here is the recipe for egg rolls:
2 tablespoons olive oil
Salt and pepper
1/2 head of cabbage
1 carrot shredded
2 tablespoons soy sauce
1 tablespoon ground ginger
8-ounces pulled pork
1 egg
Egg roll wraps
Oil for frying
2. In a skillet or wok over medium high heat, add the oil and let it get hot. Then add the cabbage, carrots, and a pinch of salt and pepper, and cook until the cabbage is wilted. Add the soy sauce and ginger and continue to cook.
3. Finally, mix in the pulled pork and let it get hot. Set the mixture aside to cool.
4. Beat the egg with water to make an egg wash.
5. Lay out the egg roll wraps and brush all sides with egg wash. Add a heaping tablespoon of egg roll mixture near one side, fold two of the edges over the filling and roll like a burrito. Repeat with the other wraps.
6. Fry the egg rolls in 350 degree F oil until they are golden brown.


text:
Edit Article
wikiHow to Sneak an MP3 Player in School
A simple way to listen to music during lectures or other boring matters.
1Take Beanie or skully and pull the band that folds up,down.
2Cut 2 small holes where your ear would be.
3Fold flap back up, over the ear holes
4Wrap iPod/mp3 around band of hat.
5Put head phones in ear.
6You can also put on a long sleeve shirt and put the ear phones through the sleeves so you can put in your hand and then just act like you are bored in class and lean against your hand. Simple as that. Good luck.
7And, don't get caught.
Community Q&A
Ask a Question
200 characters leftSubmit
- Hats need to be allowed first.
- be sure the shirt is long enough to cover the wires.
- be sure to not allow the other headphone to fall out of the bottom of your shirt.
- keep mp3 in pocket so no one can see it.
- Beanies with brims will NOT work.
- Make sure that the beanie has the flap that folds up or this won't work.
- Don't put the earphone in your ear the teachers are more likely to see it and someone may rat you out.
Article Info
Categories: Portable Media Players
Thanks to all authors for creating a page that has been read 5,123 times.
Is this article up to date?


text:
Forgot your password? Enter the email address you signed up with and we will send you a link to create a new password.
Your password reset has been submitted to the address provided.
Standard character mark
Medical services, namely, providing the evaluation, diagnosis and treatment of back and neck pain; physical therapy and rehabilitation services; physical and occupational therapy, namely, therapy to relieve neck and back pain
Business in Abilene, TX


text:
I enjoy painting landscape, still life and portraits and embrace the use of all mediums. My interest lies in responding to the energy of my chosen subject. I like to have colour and movement in my paintings, linked with atmosphere and a bit of excitement. I work intuitively and instinctively, making reference to my many sketchbooks and photographs I have taken. Major influences have been Rose Hilton, Frances Bacon, Patrick Heron and Frank Auerbach.
Please take some time to look at my work.
Please take some time to look at my work.
All text and images are the sole copyright of Barbara Davis and are not to be reproduced without written permission of the artist


text:
New Jersey agriculture is distinguished by a large number of fruit, vegetable, greenhouse, and nursery farms operated by family farmers with the assistance of a significant migrant and minority labor force. A study conducted in 1992 by Rutgers University, for the U. S. Farmers Home Administration, reveals that during peak labor demand, the number of agricultural workers employed on 5,943 New Jersey commercial farms exceeds forty thousand (40,604). Among them, approximately 16,500, or 40%, are migrant workers, with the remaining 60% comprised of day haul, seasonal workers from the local area, and full or part- time labor. The state is also characterized by a significant agricultural service industry, including ornamentals, turf, and landscape maintenance. Many traditional farms in the state produce corn, soybeans, and others field crops (over 10% of total agricultural sales). Horses account for another 14% of the agricultural cash receipts. A sizable and diverse fishing and marine service industry contributes over 9% to the state agriculture.
Agriculture ranks among the most hazardous industries. The National Institute for Occupational Safety and Health (NIOSH) provides that farmers are at high risk for fatal and nonfatal injuries, work-related lung diseases, noise-induced hearing loss, skin diseases, and certain cancers associated with chemical use and prolonged sun exposure. Further, farmers are subject to the stresses of financial uncertainty and losses, intensified time pressure, natural disasters such as drought, intergenerational conflicts, and health and safety concerns. Farming is one of the few industries in which the families (who often share the work and live on the premises) are also at risk for injuries, illness, and death.
Additionally, the diversity of the New Jersey agricultural industry leads to unique occupational safety and health problems. These include musculoskeletal injuries sustained through overexertion and poor job and task ergonomics, accidents during moving farm equipment on heavy traveled highways, as well as exposure from applying chemicals (including routine use of fertilizers and pesticides) in small confined fields and environments.
Awareness of farm hazards and minimizing associated risks is key to reducing injuries in the agricultural industry of New Jersey. In our state, the New Jersey Agricultural Experiment Station Rutgers Cooperative Extension (RCE) serves the farm community with outreach training, publications and media, and web resources in safety and worker protection. Some of these are provided in English as well as Spanish since English is a second language for such a high percentage of New Jersey's agricultural workers.
IMPORTANT: The NJDEP conducts periodic inspections for compliance with the WPS. The NJDEP WPS Inspection Checklist provides the key elements that inspectors check for compliance. You may contact NJDEP's Nancy Santiago at 609-984-6914, your County Extension Agent, or the the Rutgers Cooperative Research and Extension Pest Management Office at 732-932-9802 for assistance.
(Raymond J. Samulis, Burlington County Agricultural Agent)
This would be suitable material for a CORE training segment for NJDEP approved trainers (see guide). The Facilitator's Guide to "An American Farm Tale. Chronic Organophosphate Exposure and Treatment: The Rea Farm Case Study" is designed to reinforce and enhance concepts introduced in the companion video, through facilitator-guided discussion. The Facilitator Guide was created to provide you with background information about the Reas, pesticide poisoning, methods of preventing pesticide poisoning, information farmers should tell their doctors about the pesticides they use, and resources for more information.
Your County Rutgers Cooperative Extension Agricultural & Resource Management Agent
Search This Site:


text:
Complaint Review: American Homes 4 Rent
- American Homes 4 Rent
1624 Westgate Circle, Suite 125
Brentwood, Tennessee
- Phone:
- Web: http://www.americanhomes4rent.com/n...
- Category: Property Management
American Homes 4 Rent Beware of loopholes in AH4R "Buyer's Contract" Brentwood Tennessee
American Homes 4 Rent Buyer's Contract is a joke.
American Homes 4 Rent makes fair offers on homes; cash deals with 21 days to close and $5,000 in earnest money. They send realtors out and make offers on homes site unseen. If the house passes all inspections, they will send a manager out to view the home and decide if they actually want to go through with the sale.
American Homes 4 Rent Buyer's Contract leaves them the option to back out of the contract for no reason whatsoever. They walk away with their earnest money.
If you sign a "contract" with American Homes 4 Rent, do not jump through hoops and extend yourself financially (sign a lease or put earnest money on another home) because if they bail out, you will be left holding the bag.
Does your business have a bad reputation?Corporate Advocacy Program™Fix it the right way.
Set the record straight:Arbitration Program
This report was posted on Ripoff Report on 05/06/2013 06:21 AM and is a permanent record located here: http://www.ripoffreport.com/reports/american-homes-4-rent/brentwood-tennessee-37027/american-homes-4-rent-beware-of-loopholes-in-ah4r-buyers-contract-brentwood-tennesse-1048693. The posting time indicated is Arizona local time. Arizona does not observe daylight savings so the post time may be Mountain or Pacific depending on the time of year. Ripoff Report has an exclusive license to this report. It may not be copied without the written permission of Ripoff Report. READ: Foreign websites steal our content
If you would like to see more Rip-off Reports on this company/individual, search here:
ZipBooks Accounting Software
Advertisers above have met our
strict standards for business conduct.


text:
The leader of Lebanon's Shiite militant group Hezbollah has accused the U.S. Embassy in Lebanon of spying for Israel.
Sheik Hassan Nasrallah spoke Friday in a televised speech. He said his group has information that the U.S. Embassy is "a spy and recruitment center for Israel."
He says the group has discovered several cases in which the CIA recruited Lebanese agents to gather information about Hezbollah for Israel.
Hezbollah and Israel are sworn enemies.


text:
" Brand: Baby Einstein / Type: Finishing Touches "
When it comes to keeping your child entertained when you are in the car alone it is not easy so when I saw this in my local Home Bargains store priced at £7.99 I thought it would be a worthwhile investment as I had also spotted it in Mothercare for the higher price of £22.99.
*~*About Baby Einstein*~*
Over 10 years ago Julie Clark a mother decided to create products that offered interactive experiences for a mother and child to discover the world together, Baby Einstein was created and offer a range of products to encourage parents to interact with their children.
*~*About the Move and Go Mirror*~*
The move and go mirror is a cloud shape with a caterpillar on the main body of the product. The caterpillar is made up of 5 different colours red, blue, purple, yellow and green. The caterpillar body is mounted onto a padded service which is good fo younger babies as they reach out with their fists and ideal as your child gets older and kicks it when it is being used with a forward facing seat. There are 3 languages to allow baby to learn colours in english, spanish and french and the lights flash on and off to keep baby entertained.
The mirror is attached to the top of this to enable baby to view themselves and for the driver to be able to keep an eye on baby whilst in a rear facing seat.The mirror weighs 862g and measures 36.6 x 34.3 x 6.4cm. It also comes with a remote control that parents can use to control the mirrors four play modes.
There is a volume control complete with mute and there is a music only mode which is ideal for soothing baby to sleep.
The mirror is suitable from birth and lasts up to approximately 36 months and can be used on rear and forward facing car seats so it offers longevity in terms of a worthwhile toy investment.
*~*My Opinion*~*
We started to use this mirror within the first month of having Alex when he was starting to recognise sounds and he followed the colours as they flashed on and off and certainly kept him occupied before he dropped off to sleep. The remote control was a bit bulky and not all that reactive especially when you use it as advertised attached to your sun visor and trying to control via the simple buttons to deactivate or change the volume.
The mirror is made of robust plastic so is not all that clear and can leave images a bit distorted and not ideal for its purpose which is to be able to keep and eye on your baby whilst driving.
The product can only be sponge cleaned and despite thinking when I bought it that you could opearte by pressing the various different coloured parts of the caterpillar this is not so, I found this caused some confusion for Alex as he reached to press and when nothing happened he got a little distressed not ideal if you are driving but fortunately with the remote control you can rectify the situation quite quickly.
I had heard quite positive things about Baby Einstein from a friend which is why I snapped this up at the price it was but really do not think it was worth the high RRP that it is sold at by other high end stores.
The product is operated by batteries and to replace them you simply need to unscrew the panel at the rear and replace.
It is not one of the best products I have bought for Alex and I got swayed by the Disney Baby Einstein logo and wish I had not been so impulsive in purchasing this item and looked for other items to purchase. I really don't like the different languages it offers especially as it is being used from such an early age and wishes it offered more learning words in english to help with baby's development. I wish also that the baby could tap the colours or that the product asked the child where the red circle was and encourage them to tap the colour so it lights up to help them learn more.


text:
I tip my hat and put aside my seething hatred of Steely Dan and all the Hifi Audiophile Jackasses who think it is the pinnacle of audio and music perfection,
RIP Mr. Nichols. and don't worry, no Steely Dan will be playing in heaven.
it is surely reserved for hell.


text:
Research Topics
Reza Yaesoubi
Detail Information
- Identifying dynamic tuberculosis case-finding policies for HIV/TB coepidemicsReza Yaesoubi
Department of Epidemiology and the Center for Communicable Disease Dynamics, Harvard School of Public Health, Boston, MA 02115, USA
Proc Natl Acad Sci U S A 110:9457-62. 2013..We also find that the use of a diagnostic tool with better sensitivity for detecting smear-negative cases (e.g., Xpert MTB/RIF) further improves the incremental benefit of these dynamic case-finding policies...
- A game-theoretic framework for estimating a health purchaser's willingness-to-pay for health and for expansionReza Yaesoubi
Harvard Medical School, 641 Huntington Ave, Boston, MA 02115, USA
Health Care Manag Sci 13:358-77. 2010..Applying the proposed framework to colorectal cancer screening tests, we estimate the WTP for health and the WTP for expansion of colorectal cancer screening tests for the 2005 US population...
- Dynamic health policies for controlling the spread of emerging infections: influenza as an exampleReza Yaesoubi
Center for Communicable Disease Dynamics, Harvard School of Public Health, Boston, Massachusetts, United States of America
PLoS ONE 6:e24043. 2011....
- Payment contracts in a preventive health care system: a perspective from operations managementReza Yaesoubi
Edward P Fitts Department of Industrial and Systems Engineering, North Carolina State University, Raleigh, NC 27695 7906, USA
J Health Econ 30:1188-96. 2011....


text:
Public Invited to Segments of Upcoming National Maritime History Conference on Cal Maritime Campus May 14-17
Expand Messages
- For those interested in maritime history, these two evening programs look most interesting. The first is a private screening of "The Men Who Sailed the Liberty Ships" 1830 Thursday, May 14. The second is a one man play "The Life and Times of Harry Bridges" 1830 Friday, May 15.Also at the California Maritime Academy 0830 Saturday, May 2, is the commissioning of the graduating seniors in NROTC and related programs. Afterwards is an opportunity to meet them and their families then attend the presentation of maritime awards. I will be there and encourage interested Navy Leaguers to attend as well. You can depart after the awards ceremony and skip the lunch scheduled for students, parents and relatives and the crowded afternoon commencement.Anchors Aweigh,Phelps Hobart, PresidentPacific Merchant Marine Council, NLUS_________________________________----- Original Message -----From: Webster, DougSent: Monday, April 27, 2009 2:26 PMSubject: Public Invited to Segments of Upcoming National Maritime History Conference on Cal Maritime Campus May 14-17
The National Maritime History Conference will be coming to Cal Maritime May 14-17 and the public is invited to attend two free special events during the program. Details on the conference and the public programs can be found in a news release now posted on the campus website at:
Doug Webster
Director of Public Relations
The California Maritime Academy
200 Maritime Academy Drive
Vallejo , CA 94590-8181
707-328-1887 Cell
Your message has been successfully submitted and would be delivered to recipients shortly.


text:
Given calls for united front
Shay Given has urged supporters to get behind the team and manager Alex McLeish when Aston Villa make the trip to Blackburn on Saturday.
Last Updated: 02/03/12 9:57pm
Shay Given has urged supporters to get behind the team and manager Alex McLeish when Aston Villa make the trip to Blackburn on Saturday.
McLeish has struggled to win over some of the fans since his summer arrival from local rivals Birmingham and came in for criticism during last weekend's clash at Wigan.
Given feels it is vital to show a united front for the rest of the season as Villa strive to get their campaign back on track and stay clear of relegation trouble.
"It's very important that we stick together as a group, not just the players, or the players and the staff, but the fans, the players, the staff and everyone together," said the goalkeeper.
"That's what's best for Aston Villa and it's the most important thing on a match day that the fans do stick together.
"They might not agree with a decision or even a managerial decision, but the manager is doing it for the benefit of the team and for the players and the team out there.
"At that moment, it's what he thinks is best for Aston Villa and people should understand that. It's not about personalities or individuals, it's about the club as a whole."
Given added: "I've been in this kind of position before where teams are struggling and it doesn't help when people might be booing or whatever.
"While I would say I'm an older, more experienced player, you can't speak for every player and some of the lads are still young and still learning their way in the Premier League.
"It's at times like that, when the team is struggling a little bit, that they need support and they need everyone behind them.
"I'm not saying that the fans have been on our backs at all, in fact, they have been superb. But the more encouragement they can give us over the 90 minutes the better.
"They all have opinions and they're totally entitled to them but if they keep them for during the week and, during the game give us as much support as they can, it really helps."
Given has a lot of respect for McLeish and the way he is trying to revive Villa's fortunes.
He said: "The manager is here morning, noon and night trying to get the best out of the players, as individuals and as a team collectively.
"I know fans don't see that, but he's extremely passionate about what he does as a manager. He leaves no stone unturned in trying to be successful."


text:
PENGGUNAAN EM4 DALAM PAKAN BUATAN UNTUK MENINGKATKAN KEEFISIENAN PAKAN DAN PERTUMBUHAN IKAN NILA GIFT (Oreochromis sp.)
The fish growth is influenced by the quality of food consumed. EM4 with difference doses (0, 5, 10, and 15 ml/kg diet) was carried out to investigate feed efficiency and the growth of nila gift. An experimental method based on completely randomized design in four replicates was used for 35 days. nila gifts with 12 – 18 g in body weight were used. Fishes were fed twice daily as much as 5% of their body weight and unfed one was taken an hour after feeding. The result showed that the differences of EM4 on diet did not affect feed efficiency and the growth of nila gift. However, EM4 suplementation on diet could increase its protein content.
Full Text:PDF
- There are currently no refbacks.


text:
Hotels in Ancona, Italy
Search & Compare Ancona Hotels
Collect 10 nights, get 1 free night
And look out for Secret Prices on select hotels
Best hotels in Ancona
Ancona overviewYou can take in the local culture of Ancona at Teatro delle Muse and Diocesan Museum. A stop at other sights like Piazza del Plebiscito and Loggia dei Mercanti will create memories. If you're looking for a new favorite getaway, look no further than Ancona.
Where to stay in AnconaWhether you're planning to stay for a night or for the week, the area around Ancona has accommodations to fit every need. Search for hotels in Ancona with Hotels.com by checking our online map. Our maps display the areas and neighborhoods around all Ancona hotels so you can see how close you are from landmarks and attractions, and then refine your search within the larger area. The best Ancona hotel deals are here with our lowest price guarantee.
How to get to AnconaFly in to Ancona (AOI-Falconara), 7.8 mi (12.6 km) from downtown Ancona.
Train stations include:
- • Ancona Marittima Station
- • Ancona Station
What to see around AnconaMemorable for its landmarks including:
- • Piazza del Plebiscito
- • Loggia dei Mercanti
- • Chiesa di Santa Maria della Piazza
- • Vanvitelli's Lazzaretto
- • Cathedral of San Ciriaco
- • Teatro delle Muse
- • Diocesan Museum
- • Arches of Trajan
When to visit AnconaIf you're still deciding when to visit, here are the seasonal norms to help you plan:
Seasonal temperature averages
- • January-March: 60°F day, 39°F night
- • April-June: 82°F day, 50°F night
- • July-September: 85°F day, 63°F night
- • October-December: 73°F day, 42°F night
- • January-March: 0.28 inches
- • April-June: 0.39 inches
- • July-September: 0.35 inches
- • October-December: 0.35 inches