text:
|Attention: DocWiki has reached EOL and will be decommissioned January 25, 2019.
Cisco MDS SanOS Troubleshooting Guide -- Troublelshooting Tools and Methodology
From DocWiki
Troubleshooting Tools and Methodology
This appendix describes the troubleshooting tools and methodology available for the Cisco MDS 9000 Family multilayer directors and fabric switches. It includes the following sections:
- Using Cisco MDS 9000 Family Tools
- Using Cisco Network Management Products
- Using Other Troubleshooting Products
- Using Host Diagnostic Tools
Using Cisco MDS 9000 Family Tools
If the server does not see its storage and you cannot use the information available on the host side to determine the root cause of the problem, you can obtain additional information from a different viewpoint using the troubleshooting tools provided with the Cisco MDS 9000 Family switches. This section introduces these tools and describes the kinds of problems for which you can use each tool. It includes the following topics:
- Command-Line Interface Troubleshooting Commands
- CLI Debug
- FC Ping and FC Traceroute
- Monitoring Processes and CPUs
- Using On-Board Failure Logging
- Fabric Manager Tools
- Fibre Channel Name Service
- SNMP and RMON Support
- Using RADIUS
- Using Syslog
- Using Fibre Channel SPAN
Command-Line Interface Troubleshooting Commands
The command-line interface (CLI) lets you configure and monitor a Cisco MDS 9000 Family switch using a local console or remotely using a Telnet or SSH session. The CLI provides a command structure similar to Cisco IOS� software, with context-sensitive help, show commands, multi-user support, and roles-based access control.
CLI Debug
The Cisco MDS 9000 Family switches support an extensive debugging feature set for actively troubleshooting a storage network. Using the CLI, you can enable debugging modes for each switch feature and view a real-time updated activity log of the control protocol exchanges. Each log entry is time-stamped and listed in chronological order. Access to the debug feature can be limited through the CLI roles mechanism and can be partitioned on a per-role basis. While debug commands show realtime information, the show commands can be used to list historical information as well as realtime.
|You can log debug messages to a special log file, which is more secure and easier to process than sending the debug output to the console.
By using the '?' option, you can see the options that are available for any switch feature, such as FSPF. A log entry is created for each entered command in addition to the actual debug output. The debug output shows a time-stamped account of activity occurring between the local switch and other adjacent switches.
You can use the debug facility to keep track of events, internal messages, and protocol errors. However, you should be careful with using the debug utility in a production environment, because some options may prevent access to the switch by generating too many messages to the console or if very CPU-intensive may seriously affect switch performance.
The following is an example of the output from the debug flogi event command:
switch# '''debug flogi event interface fc1/1 Dec 10 23:40:26 flogi: current state [FLOGI_ST_FLOGI_RECEIVED] Dec 10 23:40:26 flogi: current event [FLOGI_EV_VALID_FLOGI] Dec 10 23:40:26 flogi: next state [FLOGI_ST_GET_FCID] Dec 10 23:40:26 flogi: fu_fsm_execute: ([1]21:00:00:e0:8b:08:96:22) Dec 10 23:40:26 flogi: current state [FLOGI_ST_GET_FCID] Dec 10 23:40:26 flogi: current event [FLOGI_EV_VALID_FCID] Dec 10 23:40:26 flogi: next state [FLOGI_ST_PERFORM_CONFIG] Dec 10 23:40:26 flogi: fu_fsm_execute: ([1]21:00:00:e0:8b:08:96:22) Dec 10 23:40:26 flogi: current state [FLOGI_ST_PERFORM_CONFIG] Dec 10 23:40:26 flogi: current event [FLOGI_EV_CONFIG_DONE_PENDING] Dec 10 23:40:26 flogi: next state [FLOGI_ST_PERFORM_CONFIG] Dec 10 23:40:26 flogi: fu_fsm_execute: ([1]21:00:00:e0:8b:08:96:22) Dec 10 23:40:26 flogi: current state [FLOGI_ST_PERFORM_CONFIG] Dec 10 23:40:26 flogi: current event [FLOGI_EV_RIB_RESPOSE] Dec 10 23:40:26 flogi: next state [FLOGI_ST_PERFORM_CONFIG]
The following is a summary of some of the common debug commands available Cisco SAN-OS:
Debug command
Enables AAA debugging.
Enables all debugging.
Enables BIOS daemon debugging.
Enables bootvar debugging.
Enables debugging for Call Home.
Enables CDP debugging.
Enables Cisco Fabric Services debugging.
Enables CIM server debugging.
Enables core daemon debugging.
Enables device alias debugging.
Enables delta statistics debugging.
Enables port debugging.
Enables exception log debugging.
Enables Fibre Channel tunnel debugging.
Enables FC2 debugging.
Enables FC2D debugging.
Enables Fibre Channel congestion debugging.
Enables fcdomain debugging.
Enables fcfwd debugging.
Enables Fibre Channel name server debugging.
Enables Fabric Configuration Server debugging.
Enables FDMI debugging.
Enables fabric login debugging.
Enables feature manager debugging.
Enables FSPF debugging.
Enables hardware, kernel loadable module parameter debugging.
Enables idehsd manager debugging.
Enables ilc-helper debugging.
Enables IP ACL debugging.
Enables IP configuration debugging.
Enables IPFC debugging.
Enables kernel loadable module parameter debugging.
Enables license debugging.
Directs the debug command output to a logfile.
Enables module manager debugging.
Enables NTP debugging.
Enables platform manager debugging.
Enables port debugging.
Enables PortChannel debug.
Enables QOS Manager debugging.
Enables RADIUS debugging.
Enables RIB debugging.
Enables RLIR debugging.
Enables RSCN debugging.
Enables scsi target daemon debugging.
Enables security and accounting debugging.
Enables SNMP debugging.
Enables SPAN debugging.
Enables SVC debugging.
Enables System debugging.
Enables TL Port debugging.
Enables virtual network interface debugging.
Enables VRRP debugging.
Enables VSAN manager debugging.
Enables WWN manager debugging.
Enables zone server debugging.
FC Ping and FC Traceroute
|Use the Fibre Channel ping and Fibre Channel traceroute features to troubleshoot problems with connectivity and path choices. Do not use them to identify or resolve performance issues.
Ping and traceroute are two of the most useful tools for troubleshooting TCP/IP networking problems. The ping utility generates a series of echo packets to a destination across a TCP/IP internetwork. When the echo packets arrive at the destination, they are rerouted and sent back to the source. Using ping, you can verify connectivity to a particular destination across an IP routed network.
The traceroute utility operates in a similar fashion, but can also determine the specific path that a frame takes to its destination on a hop-by-hop basis.
These tools have been migrated to Fibre Channel for use with the Cisco MDS 9000 Family switches and are called FC ping and FC traceroute. You can access FC ping and FC traceroute from the CLI or from Fabric Manager.
This section contains the following topics:
- Using FC Ping]
- Using FC Traceroute]
Using FC Ping
The FC ping tool:
- Checks end-to-end connectivity.
- Uses an pWWN or FCID.
FC ping allows you to ping a Fibre Channel N port or end device. (See Example B-1.) By specifying the FCID or Fibre Channel address, you can send a series of frames to a target N port. Once these frames reach the target device's N port, they are looped back to the source and a time-stamp is taken. FC ping helps you to verify the connectivity to an end N port. FC ping uses the PRLI Extended Link Service, and verifies the presence of a Fibre Channel entity in case of positive or negative answers.
The FC Ping feature verifies reachability of a node by checking its end-to-end connectivity.
- Choose Tools > Ping to access FC ping using Fabric Manager.
- Invoke the FC ping feature using the CLI by providing the FC ID or the destination port WWN information in the following ways:
switch# fcping pwwn 20:00:00:2e:c4:91:d4:54 switch# fcping fcid 0x123abc
Example B-1 FC Ping Command
switch# fcping fcid 0xef02c9 vsan 1 28 bytes from 0xef02c9 time = 1408 usec 28 bytes from 0xef02c9 time = 379 usec 28 bytes from 0xef02c9 time = 347 usec 28 bytes from 0xef02c9 time = 361 usec 28 bytes from 0xef02c9 time = 363 usec 5 frames sent, 5 frames received, 0 timeouts Round-trip min/avg/max = 347/571/1408 usec
Using FC Traceroute
Use the FC Trace feature to:
- Trace the route followed by data traffic.
- Compute inter-switch (hop-to-hop) latency.
FC traceroute identifies the path taken on a hop-by-hop basis and includes a timestamp at each hop in both directions. (See Example B-2.) FC ping and FC traceroute are useful tools to check for network connectivity problems or verify the path taken toward a specific destination. You can use FC traceroute to test the connectivity of TE ports along the path between the generating switch and the switch closest to the destination.
Choose Tools > Traceroute on Fabric Manager or use the fctrace CLI command to access this feature.
Use FC Trace by providing the FC ID, the N port, or the NL port WWN of the destination. The frames are routed normally as long as they are forwarded through TE ports. After the frame reaches the edge of the fabric (the F port or FL port connected to the end node with the given port WWN or the FC ID), the frame is looped back (swapping the source ID and the destination ID) to the originator.
If the destination cannot be reached, the path discovery starts, which traces the path up to the point of failure.
The FC Trace feature works only on TE Ports. Make sure that only TE ports exist in the path to the destination. If there is an E Port in the path:
- The FC Trace frame will be dropped by that switch.
- The FC Trace will time out in the originator.
- Path discovery will not start.
|FC traceroute will only work across EISL links.
Example B-2 fctraceroute Command
switch# fctrace fcid 0xef0000 vsan 1 Route present for : 0xef0000 20:00:00:05:30:00:59:de(0xfffcee) Latency: 0 msec 20:00:00:05:30:00:58:1e(0xfffc6c) Timestamp Invalid. 20:00:00:05:30:00:59:1e(0xfffcef) Latency: 0 msec 20:00:00:05:30:00:59:1e(0xfffcef) Latency: 174860 msec 20:00:00:05:30:00:58:1e(0xfffc6c)
|The values rendered by the FC traceroute process do not reflect the actual latency across the switches. The actual trace value interpretation is shown in the example below.
switch# show fcns database vsan 600 VSAN 600 -------------------------------------------------------------------------- FCID TYPE PWWN (VENDOR) FC4-TYPEFEATURE -------------------------------------------------------------------------- 0xeb01e8 NL 210000203767f7a2 (Seagate) scsi-fcptarget 0xec00e4 NL 210000203767f48a (Seagate) scsi-fcp 0xec00e8 NL 210000203767f507 (Seagate) scsi-fcp Total number of entries = 3 switch# fctrace fcid 0xeb01e8 vsan 600 Route present for 0xeb01e8 2000000530007ade(0xfffcee) ---> MDS originating the trace Latency 0 msec 2000000c30575ec0(0xfffced) --->first hop MDS towards destination FCID Latency 30820 msec 2000000c306c2440(0xfffceb) --> MDS which connects directly to the traced FCID (0xeb01e8) 2000000c306c2440(0xfffceb) -->idem, but looped around Latency 0 msec 2000000c30575ec0(0xfffced) --> first hop MDS on the return path from traced FCID to originor switch#
Monitoring Processes and CPUs
There are features in both CLI and Device Manager for monitoring switch processes and CPU status and utilization.
This section contains the following topics:
- Viewing Running Processes on Device Manager
- Using the show processes CLI Command
- Viewing CPU Time In Device Manager
- Using the show processes cpu CLI Command
- Using the show system resource CLI Command
Viewing Running Processes on Device Manager
Choose Admin > Running Processes on Device Manager to view information about the processes currently running on a switch. The Running Processes dialog box (See Figure B-1.)
The dialog display includes:
- Process ID
- The name associated with this process
- The sum of all dynamically allocated memory that this process has received from the system; this includes memory that may have been returned to the system
- The amount of CPU time the process has used, in microseconds
Figure B-1 Running Processes Dialog Box
Using the show processes CLI Command
Use the show processes command to identify the processes that are running and the status of each process. (See Example B-3.) The command output includes:
- process ID.
- State = process state.
- PC = current program counter in hex format.
- Start_cnt = how many times a process has been started (or restarted).
- TTY = terminal that controls the process. A "-" usually means a daemon not running on any particular TTY.
- Process = name of the process.
Process states are:
- D = uninterruptible sleep (usually I/O).
- R = runnable (on run queue).
- S = sleeping.
- T = traced or stopped.
- Z = defunct ("zombie") process.
- NR = not-running.
- ER = should be running but currently not-running.
|The ER state typically designates a process that has been restarted too many times, causing the system to classify it as faulty and disable it.
Example B-3 show processes Command
switch# show processes ? cpu Show processes CPU Info log Show information about process logs memory Show processes Memory Info switch# show processes PID State PC Start_cnt TTY Process ----- ----- -------- ----------- ---- ------------- . . . 457 S 2abaa76f 1 - portmap 1218 S 2acbac24 1 - licmgr 1249 S 2ade633e 1 - xbar_client 1250 S 2aca833e 1 - wwn 1251 S 2aebbc24 1 - vsan 1253 S 2ade433e 1 - ttyd 1254 S 2ac51ef4 1 - sysinfo 1255 S 2af7333e 1 - span
Viewing CPU Time In Device Manager
The Running Processes dialog display can be sorted based on any column header. To sort on CPU utilization, click the CPU column header. An arrow in the column header indicates the order of CPU utilization. Click the column header to toggle between ascending or descending order.
Example B-4 CPU Time Column Header
Using the show processes cpu CLI Command
Use the show processes cpu command to display CPU utilization. The command output includes:
- Runtime(ms) = CPU time the process has used, expressed in milliseconds.
- Invoked = number of times the process has been invoked.
- uSecs = microseconds of CPU time in average for each process invocation.
- 1Sec = CPU utilization in percentage for the last one second.
Example B-5 show processes cpu Command
switch# show processes cpu PID Runtime(ms) Invoked uSecs 1Sec Process ----- ----------- -------- ----- ----- ----------- 1016 7 2 3714 0.0 tftpd 1017 20627 2921172 7 0.0 syslogd 1218 299 11710 25 0.0 licmgr 1219 25 38 676 0.0 fs-daemon 1220 1558 6985 223 0.0 feature-mgr 1221 263 11772 22 0.0 fcfwd 1223 512 8996 56 0.0 capability 1237 313 29072 10 0.0 syslogd 1249 912 18815 48 0.0 xbar_client 1250 1481 6214 238 0.0 wwn 1251 1460 68079 21 0.0 vsan 1253 457 29220 15 0.0 ttyd 1254 138 6309 21 0.0 sysinfo
Using the show system resource CLI Command
Use the show system resources command to display system-related CPU and memory statistics. The output includes the following:
- Load is defined as number of running processes. The average reflects the system load over the past 1, 5, and 15 minutes.
- Processes displays the number of processes in the system, and how many are actually running when the command is issued.
- CPU states shows the CPU usage percentage in user mode, kernel mode, and idle time in the last one second.
- Memory usage provides the total memory, used memory, free memory, memory used for buffers, and memory used for cache in KB. Buffers and cache are also included in the used memory statistics.
Example B-6 show system resources Command
switch# show system resources Load average: 1 minute: 0.00 5 minutes: 0.00 15 minutes: 0.00 Processes : 152 total, 3 running CPU states : 0.0% user, 0.0% kernel, 100.0% idle Memory usage: 960080K total, 412900K used, 547180K free 2340K buffers, 292380K cache
Using On-Board Failure Logging
The Generation 2 Fibre Channel switching modules provide the facility to log failure data to persistent storage, which can be retrieved and displayed for analysis. This on-board failure logging (OBFL) feature stores failure and environmental information in nonvolatile memory on the module. The information will help in post-mortem analysis of failed cards.
The data stored by the OBFL facility includes the following:
- Time of initial power-on
- Slot number of the card in the chassis
- Initial temperature of the card
- Firmware, BIOS, FPGA, and ASIC versions
- Serial number of the card
- Stack trace for crashes
- CPU hog information
- Memory leak information
- Software error messages
- Hardware exception logs
- Environmental history
- OBFL specific history information
- ASIC interrupt and error statistics history
- ASIC register dumps
Configuring OBFL for the Switch
To configure OBFL for all the modules on the switch, follow these steps
Step 1
switch# config terminal
Enters configuration mode.
Step 2
switch(config)# hw-module logging onboard
Enables all OBFL features.
switch(config)# hw-module logging onboard cpu-hog
Enables the OBFL CPU hog events.
switch(config)# hw-module logging onboard environmental-history
Enables the OBFL environmental history.
switch(config)# hw-module logging onboard error-stats
Enables the OBFL error statistics.
switch(config)# hw-module logging onboard interrupt-stats
Enables the OBFL interrupt statistics.
switch(config)# hw-module logging onboard mem-leak
Enables the OBFL memory leak events.
switch(config)# hw-module logging onboard miscellaneous-error
Enables the OBFL miscellaneous information.
switch(config)# hw-module logging onboard obfl-log
Enables the boot uptime, device version, and OBFL history.
switch(config)# no hw-module logging onboard
Disables all OBFL features.
Use the show logging onboard status command to display the configuration status of OBFL.
switch# show logging onboard status Switch OBFL Log: Enabled Module: 6 OBFL Log: Enabled error-stats Enabled exception-log Enabled miscellaneous-error Enabled obfl-log (boot-uptime/device-version/obfl-history) Enabled system-health Enabled stack-trace Enabled
Configuring OBFL for a Module
To configure OBFL for specific modules on the switch, follow these steps
Step 1
switch# config terminal
Enters configuration mode.
Step 2
switch(config)# hw-module logging onboard module 1
Enables all OBFL features on a module.
switch(config)# hw-module logging onboard module 1 cpu-hog
Enables the OBFL CPU hog events on a module.
switch(config)# hw-module logging onboard module 1 environmental-history
Enables the OBFL environmental history on a module.
switch(config)# hw-module logging onboard module 1 error-stats
Enables the OBFL error statistics on a module.
switch(config)# hw-module logging onboard module 1 interrupt-stats
Enables the OBFL interrupt statistics on a module.
switch(config)# hw-module logging onboard module 1 mem-leak
Enables the OBFL memory leak events on a module.
switch(config)# hw-module logging onboard module 1 miscellaneous-error
Enables the OBFL miscellaneous information on a module.
switch(config)# hw-module logging onboard module 1 obfl-log
Enables the boot uptime, device version, and OBFL history on a module.
switch(config)# no hw-module logging onboard module 1
Disables all OBFL features on a module.
Use the show logging onboard status command to display the configuration status of OBFL.
switch# show logging onboard status Switch OBFL Log: Enabled Module: 6 OBFL Log: Enabled error-stats Enabled exception-log Enabled miscellaneous-error Enabled obfl-log (boot-uptime/device-version/obfl-history) Enabled system-health Enabled stack-trace Enabled
Displaying OBFL Logs
To display OBFL information stored in CompactFlash on a module, use the following commands:
show logging onboard boot-uptime
Displays the boot and uptime information.
show logging onboard cpu-hog
Displays information for CPU hog events.
show logging onboard device-version
Displays device version information.
show logging onboard endtime
Displays OBFL logs to an end time.
show logging onboard environmental-history
Displays environmental history.
show logging onboard error-stats
Displays error statistics.
show logging onboard exception-log
Displays exception log information.
show logging onboard interrupt-stats
Displays interrupt statistics.
show logging onboard mem-leak
Displays memory leak information.
show logging onboard miscellaneous-error
Displays miscellaneous error information.
show logging onboard module slot
Displays OBFL information for a specific module.
show logging onboard obfl-history
Displays history information.
show logging onboard register-log
Displays register log information.
show logging onboard stack-trace
Displays kernel stack trace information.
show logging onboard starttime
Displays OBFL logs from a specified start time.
show logging onboard system-health
Displays system health information.
Fabric Manager Tools
Fabric Manager provides fabric-wide management capabilities including discovery, multiple switch configuration, network monitoring, and troubleshooting. It provides the troubleshooting features described in the following topics:
- Fabric Manager and Device Manager
- Analyzing Switch Device Health
- Analyzing End-to-End Connectivity
- Analyzing Switch Fabric Configuration
- Analyzing the Results of Merging Zones
- Alerts and Alarms
- Device Manager: RMON Threshold Manager
|For detailed information about using Cisco Fabric Manager, refer to the Cisco MDS 9000 Family Fabric Manager Configuration Guide.
Fabric Manager and Device Manager
Fabric Manager provides a map of the discovered fabric and includes tables that display statistical information about the switches in the fabric. You can also select troubleshooting tools from the Fabric Manager Tools menu.
|When you click on a zone or VSAN in Fabric Manager, the members of the zone or VSAN are highlighted on the Fabric Manager Map pane.
Device Manager provides a graphic display of a specific switch and shows the status of each port on the switch. From Device Manager, you can drill down to get detailed statistics about a specific switch or port.
Figure B-2 shows the Device Manager Summary View window.
Figure B-2 Cisco Device Manager Summary View
The Summary View window lets you analyze switch performance issues, diagnose problems, and change parameters to resolve problems or inconsistencies. This view shows aggregated statistics for the active Supervisor Module and all switch ports. Information is presented in tabular or graphical formats, with bar, line, area, and pie chart options. You can also use the Summary View to capture the current state of information for export to a file or output to a printer.
Analyzing Switch Device Health
Choose the Switch Health option from the Fabric Manager Tools menu to determine the status of the components of a specific switch.
Figure B-3 Switch Health Analysis Window
The Switch Health Analysis window displays any problems affecting the selected switches.
Analyzing End-to-End Connectivity
Select Tools > End to End Connectivity option from Fabric Manager to determine connectivity and routes among devices with the switch fabric. The connectivity tool checks to see that every pair of end devices in an active zone can talk to each other, using a Ping test and by determining if they are in the same VSAN. This option uses versions of the ping and traceroute commands modified for Fibre Channel networks.
The End to End Connectivity Analysis window displays the selected end points with the switch to which each is attached, and the source and target ports used to connect it.
The output shows all the requests which have failed. The possible descriptions are:
- Ignoring empty zone—No requests are issued for this zone.
- Ignoring zone with single member—No requests are issued for this zone.
- Source/Target are unknown—No nameserver entries exist for the ports or we have not discovered the port during discovery.
- Both devices are on the same switch—The devices are not redundantly connected.
- No paths exist.
- Only one unique path exists.
- VSAN does not have an active zone set.
- Average time... micro secs—The latency value was more than the threshold supplied.
Analyzing Switch Fabric Configuration
Select the Fabric Configuration option from the Fabric Manager Tools menu to analyze the configuration of a switch by comparing the current configuration to a specific switch or to a policy file. You can save a switch configuration to a file and then compare all switches against the configuration in the file.
Figure B-4 Fabric Configuration Analysis Window
You use a policy file to define the rules to be applied when running the Fabric Checker. When you create a policy file, the system saves the rules selected for the selected switch.
Analyzing the Results of Merging Zones
Cisco Fabric Manager provides a very useful tool for troubleshooting problems that occur when merging zones configured on different switches.
Select the Zone Merge option on the Fabric Manager Tools menu to determine if two connected switches have compatible zone configurations.
Figure B-5 Zone Merge Analysis Window
The Zone Merge Analysis window displays any inconsistencies between the zone configuration of the two selected switches.
You can use the following options on the Fabric Manager Tools menu to verify connectivity to a selected object or to open other management tools:
- Traceroute—Verify connectivity between two end devices that are currently selected on the Map pane.
- Device Manager— Launch Device Manager for the switch selected on the Map pane.
- Command Line Interface—Open a Telnet or SSH session for the switch selected on the Map pane.
Alerts and Alarms
You can configure and monitor SNMP, RMON, Syslog, and Call Home alarms and notifications using the different options on the Device Manager Events menu. SNMP provides a set of preconfigured traps and informs that are automatically generated and sent to the destinations (trap receivers) that you identify. The RMON Threshold Manager lets you configure thresholds for specific events that trigger log entries or notifications. You can use either Fabric Manager or Device Manager to identify Syslog servers that will record different events or to configure Call Home, which can alert you through e-mail messages or paging when specific events occur.
Device Manager: RMON Threshold Manager
Use the options on the Device Manager Events menu to configure and monitor Simple Network Management Protocol (SNMP), Remote Monitor (RMON), Syslog, and Call Home alarms and notifications. SNMP provides a set of preconfigured traps and informs that are automatically generated and sent to the destinations (trap receivers) chosen by the user.
Use the RMON Threshold Manager to configure event thresholds that will trigger log entries or notifications. Use either Fabric Manager or Device Manager to:
- Identify Syslog servers that will record events.
- Configure Call Home, which can issue alerts via e-mail messages or paging when specific events occur.
The RMON groups that have been adapted for use with Fibre Channel include the AlarmGroup and EventGroup. The AlarmGroup provides services to set alarms. Alarms can be set on one or multiple parameters within a device. For example, an RMON alarm can be set for a specific level of CPU utilization or crossbar utilization on a switch. The EventGroup allows configuration of events (actions to be taken) based on an alarm condition. Supported event types include logging, SNMP traps, and log-and-trap.
Figure B-6 RMON Threshold Manager
Fibre Channel Name Service
The Fibre Channel name service is a distributed service in which all connected devices participate. As new SCSI target devices attach to the fabric, they register themselves with the name service, which is then distributed among all participating fabric switches. This information can then be used to help determine the identity and topology of nodes connected to the fabric.
SCSI Target Discovery
The SCSI Target Discovery feature provides added insight into connected SCSI targets. This feature allows the switch to briefly log into connected SCSI target devices and issue a series of SCSI inquiry commands to help discover additional information. The additional information that is queried includes logical unit number (LUN) details including the number of LUNs, the LUN IDs, and the sizes of the LUNs.
This information is then compiled and made available to through CLI commands, through the Cisco Fabric Manager, and also via an embedded SNMP MIB which allows the information to be easily retrieved by an upstream management application. Using the SCSI Target Discovery feature, you can have a much more detailed view of the fabric and its connected SCSI devices.
The following is an example of output from the discover scsi-target command:
switch# discover scsi-target local remote discovery started switch# show scsi-target lun vsan 1 - ST318203FC from SEAGATE (Rev 0004) FCID is 0xef02b5 in VSAN 1, PWWN is 21:00:00:20:37:46:78:97 ----------------------------------------------------------------------------- LUN Capacity Status Serial Number Device-Id (MB) ----------------------------------------------------------------------------- 0x0 18210 Online LRA2510000007027 C:1 A:0 T:3 20:00:00:20:37:46:78:97 - ST318203FC from SEAGATE (Rev 0004) FCID is 0xef02b6 in VSAN 1, PWWN is 21:00:00:20:37:5b:cf:b9 ----------------------------------------------------------------------------- LUN Capacity Status Serial Number Device-Id (MB) ----------------------------------------------------------------------------- 0x0 18210 Online LR94873000007029 C:1 A:0 T:3 20:00:00:20:37:5b:cf:b9 - ST318203FC from SEAGATE (Rev 0004) FCID is 0xef02b9 in VSAN 1, PWWN is 21:00:00:20:37:18:6f:90 ----------------------------------------------------------------------------- LUN Capacity Status Serial Number Device-Id (MB) ----------------------------------------------------------------------------- 0x0 18210 Online LR18591800001004 C:1 A:0 T:3 20:00:00:20:37:18:6f:90
For more information about SCSI target discovery, refer to the Cisco MDS 9000 Family Configuration Guide.
|This tool can be effective to find out the number of LUNs exported by a storage subsystem, but it may be ineffective when LUN Zoning/LUN Security tools are used.
SNMP and RMON Support
The Cisco MDS 9000 Family switches provide extensive SNMPv1, v2, and v3support, including Management Information Bases (MIBs) and notifications (traps and informs).
The applications provided by Cisco that use SNMP include Fabric Manager and CiscoWorks RME. Also, the SNMP standard allows any third-party applications that support the different MIBs to manage and monitor Cisco MDS 9000 Family switches.
SNMPv3 provides extended security. Each switch can be selectively enabled or disabled for SNMP service. In addition, each switch can be configured with a method of handling SNMPv1 and v2 requests.
|During initial configuration of your switch, the system prompts you to define SNMP v1 or V2 community strings and to create a SNMP v3 username and password.
Cisco MDS 9000 Family switches support over 50 different MIBs, which can be divided into the following six categories:
- IETF Standards-based Entity MIBs (for example, RFC273óENTITY-MIB)ó
- These MIBs are used to report information on the physical devices themselves in terms of physical attributes etc.
- Cisco-Proprietary Entity MIBs (for example, CISCO-ENTITY-FRU-CONTROL-MIB)ó
- These MIBs are used to report additional physical device information about Cisco-only devices such as their configuration.
- IETF IP Transport-oriented MIBs (for example, RFC2013óUDP-MIB)ó
- These MIBs are used to report transport-oriented statistics on such protocols as IP, TCP, and UDP. These transports are used in the management of the Cisco MDS 9000 Family through the OOB Ethernet interface on the Supervisor module.
- Cisco-Proprietary Storage and Storage Network MIBs (for example, NAME-SERVER-MIB)
- These MIBs were written by Cisco to help expose information that is discovered within a fabric to management applications not connected to the fabric itself. In addition to exposing configuration details for features like zoning and Virtual SANs (VSANs) via MIBs, discovered information from sources like the FC-GS-3 Name Server can be pulled via a MIB. Additionally, MIBs are provided to configure/enable features within the Cisco MDS 9000 Family. There are over 20 new MIBs provided by Cisco for this information and configuration capability.
- IETF IP Storage Working Group MIBs (for example, ISCSI-MIB)
- While many of these MIBs are still work-in-progress, Cisco is helping to draft such MIBs for protocols such as iSCSI and Fibre Channel-over-IP (FCIP) to be standardized within the IETF.
- Miscellaneous MIBs (for example, SNMP-FRAMEWORK-MIB)
- There are several other MIBs provided in the Cisco MDS 9000 Family switches for tasks such as defining the SNMP framework or creating SNMP partitioned views.
- These MIBs are used to report transport-oriented statistics on such protocols as IP, TCP, and UDP. These transports are used in the management of the Cisco MDS 9000 Family through the OOB Ethernet interface on the Supervisor module.
You can use SNMPv3 to assign different SNMP capabilities to specific roles.
Cisco MDS 9000 Family switches also support Remote Monitoring (RMON) for Fibre Channel. RMON provides a standard method to monitor the basic operations of network protocols providing connectivity between SNMP management stations and monitoring agents. RMON also provides a powerful alarm and event mechanism for setting thresholds and sending notifications based on changes in network behavior.
The RMON groups that have been adapted for use with Fibre Channel include the AlarmGroup and the EventGroup. The AlarmGroup provides services to set alarms. Alarms can be set on one or multiple parameters within a device. For example, you can set an RMON alarm for a specific level of CPU utilization or crossbar utilization on a switch. The EventGroup lets you configure events that are actions to be taken based on an alarm condition. The types of events that are supported include logging, SNMP traps, and log-and-trap.
|To configure events within an RMON group, use the Events > Threshold Manager option from Device Manager. See the "Device Manager: RMON Threshold Manager" section.
Using RADIUS
RADIUS is fully supported for the Cisco MDS 9000 Family switches through the Fabric Manager and the CLI. RADIUS is a protocol used for the exchange of attributes or credentials between a head-end RADIUS server and a client device. These attributes relate to three classes of services:
- Authentication
- Authorization
- Accounting
Authentication refers to the authentication of users for access to a specific device. You can use RADIUS to manage user accounts for access to Cisco MDS 9000 Family switches. When you try to log into a switch, the switch validates you with information from a central RADIUS server.
Authorization refers to the scope of access that you have once you have been authenticated. Assigned roles for users can be stored in a RADIUS server along with a list of actual devices that the user should have access to. Once the user has been authenticated, then switch can then refer to the RADIUS server to determine the extent of access the user will have within the switch network.
Accounting refers to the log information that is kept for each management session in a switch. This information may be used to generate reports for troubleshooting purposes and user accountability. Accounting can be implemented locally or remotely (using RADIUS).
The following is an example of an accounting log entries.
switch# show accounting log Sun Dec 15 04:02:27 2002:start:/dev/pts/0_1039924947:admin Sun Dec 15 04:02:28 2002:stop:/dev/pts/0_1039924947:admin:vsh exited normally Sun Dec 15 04:02:33 2002:start:/dev/pts/0_1039924953:admin Sun Dec 15 04:02:34 2002:stop:/dev/pts/0_1039924953:admin:vsh exited normally Sun Dec 15 05:02:08 2002:start:snmp_1039928528_172.22.95.167:public Sun Dec 15 05:02:08 2002:update:snmp_1039928528_172.22.95.167:public:Switchname
|The accounting log only shows the beginning and ending (start and stop) for each session.
Using Syslog
The system message logging software saves messages in a log file or directs the messages to other devices. This feature provides the following capabilities:
- Logging information for monitoring and troubleshooting.
- Selection of the types of logging information to be captured.
- Selection of the destination of the captured logging information.
Syslog lets you store a chronological log of system messages locally or sent to a central Syslog server. Syslog messages can also be sent to the console for immediate use. These messages can vary in detail depending on the configuration that you choose.
Syslog messages are categorized into 7 severity levels from debug to critical events. You can limit the severity levels that are reported for specific services within the switch. For example, you may wish only to report debug events for the FSPF service but record all severity level events for the Zoning service.
A unique feature within the Cisco MDS 9000 Family switches is the ability to send RADIUS accounting records to the Syslog service. The advantage of this feature is that you can consolidate both types of messages for easier correlation. For example, when you log into a switch and change an FSPF parameter, Syslog and RADIUS provide complimentary information that will help you formulate a complete picture of the event.
Log messages are not saved across system reboots. However, a maximum of 100 log messages with a severity level of critical and below (levels 0, 1, and 2) are saved in NVRAM. You can view this log at any time with the show logging nvram command.
Logging Levels
The MDS supports the following logging levels:
- 0-emergency
- 1-alert
- 2-critical
- 3-error
- 4-warning
- 5-notification
- 6-informational
- 7-debugging
By default, the switch logs normal but significant system messages to a log file and sends these messages to the system console. Users can specify which system messages should be saved based on the type of facility and the severity level. Messages are time-stamped to enhance real-time debugging and management.
Enabling Logging for Telnet or SSH
System logging messages are sent to the console based on the default or configured logging facility and severity values.
Users can disable logging to the console or enable logging to a given Telnet or SSH session.
- To disable console logging, use the no logging console command in CONFIG mode.
- To enable logging for telnet or SSH, use the terminal monitor command in EXEC mode.
The no logging console command shown in Example B-7]:
- Disables console logging
- Enabled by default
Example B-7 no logging console Command
switch(config)# no logging console
The terminal monitor command shown in Example B-8:
- Enables logging for telnet or SSH
Disabled by default
Example B-8 terminal monitor Command
switch# terminal monitor
Using Fibre Channel SPAN
You can use the Switched Port Analyzer (SPAN) utility to perform detailed troubleshooting or to take a sample of traffic from a particular application host for proactive monitoring and analysis. This utility is most helpful when you have a Fibre Channel protocol analyzer available and you are monitoring user traffic between two FC IDs.
When you have a problem in your storage network that you cannot solve by fixing the device configuration, you typically need to take a look at the protocol level. You can use debug commands to look at the control traffic between an end node and a switch. However, when you need to focus on all the traffic originating from or destined to a particular end node such as a host or a disk, you can use a protocol analyzer to capture protocol traces.
To use a protocol analyzer, you must insert the analyzer in-line with the device under analysis, which disrupts input and output (I/O) to and from the device. This problem is worse when the point of analysis is on an Inter-Switch Link (ISL) link between two switches. In this case, the disruption may be significant depending on what devices are downstream from the severed ISL link.
In Ethernet networks, this problem can be solved using the SPAN utility, which is provided with the Cisco Catalyst Family of Ethernet switches. SPAN has also been implemented with the Cisco MDS 9000 Family switches for use in Fibre Channel networks. SPAN lets you take a copy of all traffic and direct it to another port within the switch. The process is non-disruptive to any connected devices and is facilitated in hardware, which prevents any unnecessary CPU load. Using Fibre Channel SPAN, you can connect a Fibre Channel analyzer, such as a Finisar analyzer, to an unused port on the switch and then SPAN a copy of the traffic from a port under analysis to the analyzer in a non-disruptive fashion.
SPAN allows you to create up to 16 independent SPAN sessions within the switch. Each session can have up to four unique sources and one destination port. In addition, you can apply a filter to capture only the traffic received or the traffic transmitted. With Fibre Channel SPAN, you can even capture traffic from a particular Virtual SAN (VSAN).
To start the SPAN utility use the CLI command span session session_num, where session_num identifies a specific SPAN session. When you enter this command, the system displays a submenu, which lets you configure the destination interface and the source VSAN or interfaces.
switch2# config terminal switch2(config)# span session 1 <<=== Create a span session switch2(config-span)# source interface fc1/8 <<=== Specify the port to be spanned switch2(config-span)# destination interface fc1/3 <<==== Specify the span destination port switch2(config-span)# end switch2# show span session 1 Session 1 (active) Destination is fc1/1 No session filters configured Ingress (rx) sources are fc1/8, Egress (tx) sources are fc1/8,
For more information about configuring SPAN, refer to the Cisco MDS 9000 Family Configuration Guide.
Using Cisco Network Management Products
This section describes network management tools that are available from Cisco and are useful for troubleshooting problems with Cisco MDS 9000 Family switches and connected devices and includes the following topics:
- Cisco MDS 9000 Family Port Analyzer Adapter
- Cisco Fabric Analyzer
- IP Network Simulator
Cisco MDS 9000 Family Port Analyzer Adapter
The Cisco MDS 9000 Family Port Analyzer Adapter is a stand-alone adapter card that converts Fibre Channel frames to Ethernet frames by encapsulating each Fibre Channel frame into an Ethernet frame. This product is meant to be used for analyzing SPAN traffic from a Fibre channel port on a Cisco MDS 9000 Family switch.
The Cisco MDS 9000 Family Port Analyzer Adapter provides two physical interfaces:
- A Fiber Channel interface that connects to the SPAN port of a Cisco MDS 9000 Family switch
- A 100/1000 Mb/s Ethernet port that forwards the encapsulated Fibre Channel traffic with a broadcast destination MAC Address
|The Cisco MDS 9000 Family Port Analyzer Adapter does not support half-duplex mode and for this reason, it will not work when connected to a hub.
The Cisco MDS 9000 Family Port Analyzer Adapter provides the following features:
- Encapsulates Fibre Channel frames into Ethernet frames.
- Sustains 32 maximum size Fibre Channel frames burst (in 100 Mbps mode).
- Line rate at 1Gbps (for Fibre Channel frames larger than 91 bytes).
- 64 KB of onboard frame buffer.
- Configurable option for Truncating Fibre Channel frames to 256 bytes (for greater burst).
- Configurable option for Deep Truncating Fibre Channel frames to 64 bytes (best frames burst).
- Configurable option for Ethernet Truncating Fibre Channel frames to 1496 bytes (maximum size E-net frames).
- Configurable option for No Truncate Mode (sends jumbo frames on E-net side).
- Packet Counter (Indicates number of previous packet drops).
- SOF/EOF type information embedded.
- 100/1000 Mb/s Ethernet interface (option on board).
- Auto Configuration on power up.
- Fibre Channel and Ethernet Link up indicator LEDs.
- Checks Fibre Channel frame CRC.
When used in conjunction with the open source protocol analyzer, Ethereal (http//www.ethereal.com), the Cisco MDS 9000 Family Port Analyzer Adapter provides a cost-effective and powerful troubleshooting tool. It allows any PC with a Ethernet card to provide the functionality of a flexible Fibre Channel analyzer. For more information on using the Cisco MDS 9000 Family Port Analyzer Adapter see the Cisco MDS 9000 Family Port Analyzer Adapter Installation and Configuration Guide.
Cisco Fabric Analyzer
The ultimate tool for troubleshooting network protocol problems is the protocol analyzer. Protocol analyzers promiscuously capture network traffic and completely decode the captured frames down to the protocol level. Using a protocol analyzer, you can conduct a detailed analysis by taking a sample of a storage network transaction and by mapping the transaction on a frame-by-frame basis, complete with timestamps. This kind of information lets you pinpoint a problem with a high degree of accuracy and arrive at a solution more quickly. However, dedicated protocol analyzers are expensive and they must be placed locally at the point of analysis within the network.
With the Cisco Fabric Analyzer, Cisco has brought Fibre Channel protocol analysis within a storage network to a new level of capability. Using Cisco Fabric Analyzer, you can capture Fibre Channel control traffic from a switch and decode it without having to disrupt any connectivity, and without having to be present locally at the point of analysis.
The Cisco Fabric Analyzer consists of three main components:
- An agent embedded in the Cisco MDS 9000 Family switches. This agent can be selectively enabled to promiscuously capture designated control traffic.
- A text-based interface to the control and decoded output of the analyzer.
- GUI-based client application that you can install on any workstation to provide a full-function interface to the decoded data.
The text-based interface to the Cisco Fabric Analyzer is a CLI-based program for controlling the analyzer and providing output of the decoded results. Using the CLI, you can remotely access an Cisco MDS 9000 Family switch, using Telnet or a secure method such as Secure Shell (SSH). You can then capture and decode Fibre Channel control traffic, which offers a convenient method for conducting detailed, remote troubleshooting. In addition, because this tool is CLI-based, you can use roles-based policies to limit access to this tool as required.
The GUI-based implementation (Ethereal) can be installed on any Windows or Linux workstation. This application provides an easier-to-use interface that is more easily customizable. The GUI interface lets you easily sort, filter, crop, and save traces to your local workstation.
The Ethereal application allows remote access to Fibre Channel control traffic and does not require a Fibre Channel connection on the remote workstation.
The Cisco Fabric Analyzer lets you capture and decode Fibre Channel traffic remotely over Ethernet. It captures Fibre Channel traffic, encapsulates it in TCP/IP, and transports it over an Ethernet network to the remote client. The remote client then deencapsulates and fully decodes the Fibre Channel frames. This capability provides flexibility for troubleshooting problems in remote locations.
The Cisco Fabric Analyzer captures and analyzes control traffic coming to the Supervisor Card. This tool is much more effective than the debug facility for packet trace and traffic analysis, because it is not very CPU intensive and it provides a graphic interface for easy analysis and decoding of the captured traffic.
switch# config terminal switch(config)# fcanalyzer local brief Capturing on eth2 0.000000 ff.ff.fd -> ff.ff.fd SW_ILS 1 0x59b7 0xffff 0x7 -> 0xf HLO 0.000089 ff.ff.fd -> ff.ff.fd FC 1 0x59b7 0x59c9 0xff -> 0x0 Link Ctl, ACK1 1.991615 ff.ff.fd -> ff.ff.fd SW_ILS 1 0x59ca 0xffff 0xff -> 0x0 HLO 1.992024 ff.ff.fd -> ff.ff.fd FC 1 0x59ca 0x59b8 0x7 -> 0xf Link Ctl, ACK1 fcanalyer example of fully decoded frame. switch2(config)# fcanalyzer local Capturing on eth2 Frame 1 (96 bytes on wire, 96 bytes captured) Arrival Time Jan 13, 2003 135038.787671000 Time delta from previous packet 0.000000000 seconds Time relative to first packet 0.000000000 seconds Frame Number 1 Packet Length 96 bytes Capture Length 96 bytes Ethernet II, Src 00000000000a, Dst 00000000ee00 Destination 00000000ee00 (00000000ee00) Source 00000000000a (00000000000a) Type Vegas FC Frame Transport (0xfcfc) MDS Header(SOFf/EOFn) MDS Header Packet Len 66 .... 0000 0001 11.. = Dst Index 0x0007 .... ..00 1111 1111 = Src Index 0x00ff .... 0000 0000 0001 = VSAN 1 MDS Trailer EOF EOFn (3) Fibre Channel R_CTL 0x02 Dest Addr ff.fc.7e CS_CTL 0x00 Src Addr ff.fc.7f Type SW_ILS (0x22) F_CTL 0x290000 (Exchange Originator, Seq Initiator, Exchg First, Seq Last, CS_CTL, Transfer Seq Initiative, Last Data Frame - No Info, ABTS - Abort/MS, ) SEQ_ID 0x11 DF_CTL 0x00 SEQ_CNT 0 OX_ID 0x5a06 RX_ID 0x0000 Parameter 0x00000000 SW_ILS Cmd Code SW_RSCN (0x1b) 0010 .... = Event Type Port is offline (2) .... 0000 = Address Format Port Addr Format (0) Affected Port ID 7f.00.01 Detection Function Fabric Detected (0x00000001) Num Entries 1 Device Entry 0 Port State 0x20 Port Id 7f.00.01 Port WWN 1000000530005f1f (000530) Node WWN 1000000530005f1f (000530)
However, the Cisco Fabric Analyzer is not the right tool for troubleshooting end-to-end problems because it cannot access any traffic between the server and storage subsystems. That traffic is switched locally on the linecards, and does not reach the Supervisor card. In order to debug issues related to the communication between server and storage subsystems, you need to use Fibre Channel SPAN with an external protocol analyzer.
There are two ways you can start the Cisco Fabric Analyzer from the CLI.
- fcanalyzer local-—Launches the text-based version on the analyzer directly on the console screen or on a file local to the system.
- fcanalyzer remote ip address—-Activates the remote capture agent on the switch, where ip address is the address of the management station running Ethereal.
For more information about using the Cisco Fabric Analyzer, refer to the Cisco MDS 9000 Family Configuration Guide.
IP Network Simulator
Network simulators let you simulate various kinds of IP data network conditions. A simulator allows you to troubleshoot IP network problems, and can also help you understand the potential impact of additional traffic or specific network conditions to your existing network configuration.
Network simulator is a generic tool that provides simulation features for all Ethernet traffic, and handles full duplex Gigabit Ethernet traffic at full line rate. It is supported on the 8-port IP Storage Services (IPS-8) module and on the 4-port IP Storage Services (IPS-4) module only. It requires the SAN Extension Tuner, which is available in either the SAN extension over IP package for IPS-8 modules (SAN_EXTN_OVER_IP) or the SAN extension over IP package for IPS-4 modules (SAN_EXTN_OVER_IP_IPS4).
The network simulator tool can simulate the following network functions:
- Network delays (maximum network delays of 150 ms)
- Limiting maximum throughput to the given bandwidth
- Finite queue size
- Dropping packets
- Reordering packets
For more information about using the IP Network Simulator, refer to the Cisco MDS 9000 Family CLI Configuration Guide.
Using Other Troubleshooting Products
This section describes products from other vendors that you might find useful when troubleshooting problems with your storage network and connected devices. It includes the following topics:
- Fibre Channel Testers
- Fibre Channel Protocol Analyzers
Fibre Channel Testers
Fibre Channel testers are generally used to troubleshoot low-level protocol functions (such as Link Initialization). Usually these devices operate at 1- or 2-Gbps and provide the capability to create customized low-level Fibre Channel primitive sequences.
Fibre Channel testers are primarily used to ensure physical connectivity and low-level protocol compatibility, such as with different operative modes like Point-to-Point or Loop mode.
Fibre Channel testers and more generalized optical testers may used to spot broken cables, speed mismatch, link initialization problems and transmission errors. These devices sometimes incorporate higher-level protocol analysis tools and may bundled with generic protocol analyzers.
Fibre Channel Protocol Analyzers
An external protocol analyzer (for example from Finisar), is capable of capturing and decoding link level issues and the fibre channel ordered sets which comprise the fibre channel frame. The Cisco MDS 9000 Family Port Analyzer Adapter, does not capture and decode at the ordered set level.
A Fibre Channel protocol analyzer captures transmitted information from the physical layer of the Fibre Channel network. Because these devices are physically located on the network instead of at a software re-assembly layer like most Ethernet analyzers, Fibre Channel protocol analyzers can monitor data from the 8b/10b level all the way to the embedded upper-layer protocols.
Fibre Channel network devices (HBAs, switches, and storage subsystems) are not able to monitor many SAN behavior patterns. Also, management tools that gather data from these devices are not necessarily aware of problems occurring at the Fibre Channel physical, framing, or SCSI upper layers for a number of reasons.
Fibre Channel devices are specialized for handling and distributing incoming and outgoing data streams. When devices are under maximum loads, which is when problems often occur, the device resources available for error reporting are typically at a minimum and are frequently inadequate for accurate error tracking. Also, Fibre Channel host bus adapters (HBAs) do not provide the ability to capture raw network data.
For these reasons, a protocol analyzer may be more important in troubleshooting a storage network than in a typical Ethernet network. There are a number of common SAN problems that occur in deployed systems and test environments that are visible only with a Fibre Channel analyzer. These include the following:
- Credit starvation
- Missing, malformed, or non-standard-compliant frames or primitives
- Protocol errors
Using Host Diagnostic Tools
Most host systems provide utilities or other tools that you can use for troubleshooting the connection to the allocated storage. For example, on a Windows system, you can use the Diskmon or Disk Management tool to verify accessibility of the storage and to perform some basic monitoring and administrative tasks on the visible volumes.
Alternatively, you can use Iometer, an I/O subsystem measurement and characterization tool, to generate a simulated load and measure performance. Iometer is a public domain software utility for Windows, originally written by Intel, that provides correlation functionality to assist with performance analysis.
Iometer measures the end-to-end performance of a SAN without cache hits. This can be an important measurement because if write or read requests go to the cache on the controller (a cache hit) rather than to the disk subsystems, performance metrics will be artificially high. You can obtain Iometer from SourceForge.net at the following URL:
Iometer is not the only I/O generator you can use to simulate traffic through the SAN fabric. Other popular I/O generators and benchmark tools used for SAN testing include Iozone and Postmark. Iozone is a file system benchmark tool that generates and measures a variety of file operations. It has been ported to many systems and is useful for performing a broad range of file system tests and analysis.
Postmark was designed to create a large pool of continually changing files, which simulates the transaction rates of a large Internet mail server.
PostMark generates an initial pool of random text files in a configurable range of sizes. Creation of the pool produces statistics on continuous small file creation performance. Once the pool is created, PostMark generates a specified number of transactions, each of which consists of a pair of smaller transactions:
- Create file or Delete file
- Read file or Append file
Benchmarking tools offer a variety of capabilities and you should select the one that provides the best I/O characteristics of your application environment.
Utilities provided by the Sun Solaris operating system let you determine if the remote storage has been recognized and exported to you in form of a raw device or mounted file system, and to issue some basic queries and tests to the storage. You can measure performance and generate loads using the iostat utility, the perfmeter GUI utility, the dd utility, or a third-party utility like Extreme SCSI.
Every UNIX version provides similar utilities, but this guide only provides examples for Solaris. Refer to the documentation for your specific operating system for details.


text:
League News
We are social too, ..Follow us
Best Web design Award is a creative web design award website that rewards the most beautiful, creative, user-friendly, innovative and well designed flash and CSS websites around the world. We aim to be a source for web design inspiration for web designers. We award the Best Websites both flash sites and css sites..

subset: commoncrawl


text:
This action will take you to use the latest version of the MSN Games Download Games Manager. This version does not support MSN Games Discount Program or All Access subscriptions created before March 2018.
If you downloaded games at MSN since March 2018, select "Continue" below, if not, click "Dismiss".
Find out what happened to your grandfather!
Your review will be published within 24 to 48 hours.


text:
A young girl on a long and lonely voyage, a beautiful stone house by the sea, and a book of fiercely guarded recipes. These are just some of the elements which compose the fascinating history of Mayrig.
Tucked away at the far end of Beirut's Gemayzeh quarter, Mayrig is an Armenian restaurant that adapts the popular classics of Armenian cuisine, while generously borrowing from Lebanese and other Mediterranean influences, in order to feast guests on a dazzling array of flavors.
Our "haven of dining perfection", as food critics have hailed it, uses spicy ingredients and dewy-fresh harvests from the entire region, delicately hand-crafted creations, and a charming setting that still evokes Beirut's glory days.
Somehow, what you get is even more sublime than the sum of its parts…not surprising for a restaurant which still believes in serving food, and welcomes, very warm.
We are now recruiting.
Interested candidates can send their CV's to the following e-mail address: email@example.com
Amazing Buffet for your parties from our catering corner.
Contact us on:04-521777


text:
Monday, Feb. 25, was a short legislative day but I had several meetings to prepare for floor work during the week.
Monday, Feb. 25, was a short legislative day but I had several meetings to prepare for floor work during the week.
Probably one of the real secrets of legislation is how many hours of brain storming, research, and revising are required before we are ready to proceed with a bill. Even now, legislation is being prepared for next year!
Sometimes, even a good idea is ready before it's time. Many times, reps rush into a bill presentation without thinking it through thoroughly. Probably the worst thing that can happen is for a bill to become law and then we find out that it has unintended consequences. Sometimes very important legislation is overturned years later by the courts simply because a phrase or even just one word is out of place or incorrect. We owe it to our constituents to try our very best to present good legislation that is well thought out and properly prepared.
Monday evening the Farm Bureau had their annual Legislative Dinner. We heard from the director of agriculture who gave an update on the consequences of the drought and what might be in store for us this upcoming growing season. Although no one can predict the weather with certainty, we should be prepared for a continued dry spell. Of course there will be exceptions, but overall the next year looks to be much drier than normal.
On Tuesday, Feb. 26, we took up debate on HB34, which is known as the School Construction Act. This would allow school districts around the state the option of adding on or building new structures without having to pay prevailing wages. Many school districts in rural areas are prevented from doing much needed repair because they simply cannot afford the prevailing wage. The option would be up to each individual school board. we passed the bill out of the House and now the Senate must review and debate the merits before they vote.
Wednesday, Feb. 27, was a long day but we did pass some meaningful legislation. Bill Reiboldt was working on passage of a "Right to Farm" bill. It doesn't seem like we need to assert a right to farm or raise livestock, but with outside special interests always looking for a way to press their agendas, we have to ascertain that we in Missouri have the right to raise livestock and farm in time honored traditions.
We also passed HB320. This bill deals with Workman's Comp Reform. As is often the case, there were some loopholes in existing law that allowed trial lawyers to take advantage of situations and bring lawsuits against employees and employers. Some of these litigations were so costly that businesses were ruined and hundreds of people were put out of work. This would be a tragic situation under any circumstances, but when it happens simply because there are loopholes that allow frivolous suits to turn into multi- million dollar settlements that the lawyers get 50 percent of, we need to rethink the laws.
I've gotten lots of calls and messages concerning the Medicaid Expansion. Frankly, it's way too early to make a decision, but look at it this way; the federal government says that if we raise the eligibility of Medicaid to 138 percent of the poverty rate and add 300,000 people to the list of recipients, they will pay for 100 percent of the cost. The proposal is that after three years, we pay 10 percent and they pay 90. The governor has been going all over the state saying that if the Government doesn't keep it's word, we will remove those folks from the expanded coverage.
First off, neither the governor or the president are going to be around to deal with 300,000 or 400,000 angry people when you take it away from them.
Secondly, if you really believe that the government is going to keep it's promise, find a Native American and ask them what "As long as the wind blows and the grass grows" means.We need to take our time and watch what other states do before we bargain away the resources that are paying for our schools and Senior Services.
Finally, I heard a bill presented in committee that addresses the potential shortages of primary care in rural areas. The idea was that with computers and face to face online services, a doctor could have a clinic in one town and a remote location in another town staffed by a physicians' assistant. If the assistant was having trouble diagnosing an ailment, they could link the patient and doctor in a face to face and the doctor could ask the patient questions and direct the Physician's Assistant in tests and diagnosis. It really is an intriguing idea and sure would help solve the doctor shortage in rural areas.
More next week, until then I am and remain in your service.
Bill Lant represents the people of Southwest Missouri in the Mo. House of Representatives. Contact him locally at 437-8223 or at his Jefferson City office at (573) 751-9801 or email him at email@example.com.


text:
The County Commission met at 6 p.m. Monday evening for their first meeting of the month. Several items were on the agenda.
The County Commission met at 6 p.m. Monday evening for their first meeting of the month. Several items were on the agenda.
Commissioners approved a $1,000 contribution to the Dodge City Night event which is held each year in Topeka. A local resident requested to temporarily set up a disk-golf course at the Ford County Lake for a tournament that will be held Feb. 23. All money raised at the tournament will be donated to charity. Sheriff Dean Bush requested funds for a K9 replacement, and additional funds to travel out of state and pick up the dog, and for some training. It will be a patrol/drug dog.
"We had a K9 unit several years ago," Sharon Seibel said in a phone interview on Tuesday, "and the commission approved the request to reactivate the program."
The commission also accepted the resignation of Sheriff Bush from the Area Mental Health Board, and appointed Erica Deandra-Sotero as his replacement.
In other business, the commission:
-approved an application from the health department to apply to the Kansas Department of Health and Environment for several grants from the state. The total amount of the grant is $167,465.
-approved bids from A.V. Energy for fuel for the counties vehicles.
-approved a change order for $36,225 for work that is being done at the landfill.
The next County Commission meeting will be held Feb. 18.
Follow Julia on twitter @julia_dcglobe


text:
Dear ______________,
It is with pleasure that I write this letter of recommendation for Ms. Peterson and her cat, Buddy. I wish all my residents were as well behaved as Buddy. When Ms. Peterson and Buddy vacated their apartment unit, it was in better condition than when they moved into it.
During their five-year residency at Brookside Apartments, Ms. Peterson and Buddy have been excellent members of our rental community, and I would be very happy to discuss why they should be welcomed in yours.
John Doe
Property Manager Brookside Apartments
Copyright © 2004 The Humane Society of the United States All rights reserved.


text:
25 Listings:
2 Sold:
Transworld Business Advisors is the world leader in the marketing and sales of businesses, franchises and commercial real estate. Whether you represent an acquisition-minded corporation, or are personally interested in owning your own company, Transworld offers the professional services that successfully bring buyers and sellers together. From busi
J. Darrow Graham
Transworld Business Advisors Grapevine
Bosque County, TX
Liquor Store. Inside sales $38,000 per month. American owned.
Convenience store with gas, diesel, deli, car wash, and coin operated laundry. One year left on fuel contract. American owned. Inside sales $50,000, Outside sales 25,000 gallons per month.
Thank you for signing up for the BizBuySell bulletin.
Sign up for the BizBuySell Bulletin and get advice on:
You can opt out of the BizBuySell Bulletin at any time.
Business listed by:
Unknown Broker
Unknown Broker Company
Full name is required.
Get free alerts via email when new businesses are listed matching your Bosque County Retail search criteria.
Reveal hidden alerts and unlock advanced features on search results and detail pages to help you find the perfect business.
Premium Members also get valuable insight into the buying process via our
video series, access to our advanced
business search agent, and more!


text:
How would that sound?
That is precisely the score I heard as I was beginning to wake up from a vivid dream. I know it's just a dream but who knows? I was waking up from a dream where King 5's Paul Silvi was talking about how the Seahawks crushed the Falcons 53-17. Perhaps it's little more than the wishes and hopes in my subconscious manifesting itself, but what if it was something more?
Who knows? Would be a kick in the pants if it comes to fruition tomorrow.


text:
A few months ago there was a detailed article in the Butler Eagle, along with a drawing, of what improvements the county Parks and Recreation Department was proposing for Alameda Park.
The new pool area looked like some resort.
Recently there was another article about improvements at the Alameda pool carrying a cost of $900,000. Yes, there's going to be a grant applied for the sum of $400,000, but where do people think the other $500,000 will come from?
That money will come from my pocket and from the pockets of other county residents.
What amuses me about the whole project is that Commissioner Jim Eckstein sits at all the commissioners meetings nit-picking nickels and dimes over how much hotels are costing for county employees to attend different conferences. He thinks the employees should be staying in cheaper hotels.
Then he had the nerve to say he thought the Alameda work was a great idea and "would benefit people in all municipalities."
That's $900,000 or, with the grant, $500,000, to benefit whom? I would imagine the answer is the residents of Butler Township and the city of Butler.
Evans City and other small towns work hard to keep their parks going. Oh, the county throws them out a little "treat" every once in awhile to say the county helped them, but the treat isn't really close to $900,000.
The commissioners are going to spend this outrageous amount of money for a pool that's open three and a half months out of the year. With 8,500 members?
That unbelievable.
We've cut spending in so many more important areas, but we have this amount of money to spend on a pool to benefit 8,500 people.
I'm asking the county commissioners to rethink this project.And, if Eckstein is so strong in his feelings to do his "due diligence," I would hope he would change his mind on how wonderful he thinks this project would be.


text:
Want to create great watches for your fans? Get started for free!
Learn more >
All Modify Watches are Made-to-Order. That means we print and assemble your watches here in our office in California as soon as you place your order.
Proud member of SFMade.
We want you to love your watch. Sooooo, if it stops working, we'll fix it. And if your battery dies in a few years, we'll fix that too. You're in the ModiFamily now :)
Sold Out


text:
Swimming lessons in Watertown CT are significant for a child's growth and development. In some instances, children learn to swim before they can walk, this is especially true if their parents are avid swimmers. However, there are children in Connecticut whose parents have never undertaken the joy of learning to swim, but it's never too late. With time and a bit of effort, even adults can learn to swim and can then pass the love of swimming along to their children. As an adult looking to learn to swim you can sign up for classes at the local Watertown YMCA or YWCA or even a local Watertown high school that has a swimming pool. There are instructors in Watertown CT that can teach individuals of all ages and all skill levels. If you're not in the market for taking a group swimming class and want to learn in the privacy of your own swimming pool in your own backyard, you can hire an instructor to come out and teach you there. You will want to either observe the instructor or meet with him or her prior to the class to make certain there is a good rapport between the two of you as it will make learning to swim much easier if you're taking lessons from someone you feel comfortable with.
A core of successful cultivation of swimming skills at an early age is parents' involvement in swimming lessons. Most swimming instructors in Watertown agree that for the best results kids under the age of four must be accompanied by adults when they learn to swim. Watertown Connecticut abounds in various swimming pools, so there is no problem in finding the one which you can attend together with your child. Parental involvement in swimming lessons is beneficial for many reasons. They include: fine motor development, child and parent relationship, safe and secure learning environment, parental education and maximum time for practice.
Fine Motor Development
From numerous academic studies and our instructors' education background, we know that until the age of 4, children don't possess the fine motor skills necessary to coordinate "conventional swimming". For better results kids need one-on-one instruction and manipulation. Such instructions and manipulations are best provided by parents. Generally, the children who have their parents involved in the swimming lessons will coordinate better become skillful swimmers long before those do not have such opportunity. As a beginning swimmer you will start your lessons in the shallow end of the pool which should take away any fear of water you may have. Your instructor will teach you how to breathe in the water, how to move through the water and a respect of the water. In your swimming lesson you should be prepared for a session in which you will learn to become comfortable with putting your face in the water, holding your breathe and then going back under again.
Safe and Secure Learning Environment
One of the greatest causes of accidental deaths among preschool children is drowning. Most of these deaths occur in backyard swimming pools to the children who just begin to learn to swim (Connecticut is not an exception in this sad statistics). Hence it is of particular importance that parents are involved in learning their children to swim, since this way kids learn a cautious approach to the water and get awareness that it is prohibited to swim unless being accompanied by a parent. Adhering to this principle decreases the likelihood for kids getting into hazardous situations. It should be kept in mind that it is impossible for one adult to adequately supervise numerous children under the age of 4. Hence the early swim lessons which don't keep one-on-one ratio are potentially dangerous.
As the lessons progress, you will learn to bob underwater, do front and back floats and eventually move up to more advanced swim strokes. Before you learn swim strokes the instructor will lead the class into deeper water to teach you how to move around and maintain your balance in the water, along with this will likely come a lesson on treading water. When you're learning to float on your stomach the instructor will either have the students hang onto the side of the swimming pool or use a kickboard to help them stay afloat.
Child and Parent Relationship
Swimming is a perfect way for children and parents both to learn to swim and have fun together. For many children it becomes a treat, as they are allocated one-on-one time to play with their parents. Here a magical bond can be established between children and parents as they learn together in the children swimming lessons. When you're first learning to swim it's essential that you practice several times a week. Try to block out time when you can devote yourself to 30-minute sessions as a way to both build endurance and practice kicking and swim strokes. In no time you'll learn to enjoy yourself in the water so you can swim and play with your children in the comfort of your own pool. Research has shown that during the young ages of swimming, it has helped them in the development in their psycho- motor abilities, strength and others. It also created a learning environment where the infant or toddlers can start to master water safety at a young age. These lessons will give them a solid foundation in swimming and an excellent head start in their further swimming lessons. Parents can also take this opportunity to bond with their children during this period.
Adult Swimming Lessons in Connecticut
There are two main reasons why adults who do not know how to swim were not able to do so when they were young. The first is a lack of opportunity and the second more common reason is an irrational fear of the water. They continue to avoid learning it in their adult years mainly because of embarrassment at not being able to do something many children appear to find easy. These are not really strong reasons to remain unable to swim. It is an accepted fact that as we grow older, our bodies no longer develop. After a certain point, it gradually deteriorates. This is seen in the more stooped posture of the elderly, becoming winded earlier when exercising or engaging in strenuous activity, and discomfort or even pain in ones joints when moving from one place to another. Some people make this an excuse not to learn to swim, or worse, to no longer exercise as regularly as before. Swimming can actually be engaged in despite these age related changes.
The older one gets, the more attractive this form of exercise becomes because it does not stress the joints to the same degree that dry land exercises do. Joint pain is thus either absent or very minimal after exercise. Also, increased blood circulation to both the muscles and the joints definitely aid in maintaining the joint within the acceptable range of mobility for a particular age group. Nearly everyone in the health care or fitness profession will agree that the best exercise for those in their fifties or older is swimming. Another hurdle to an adult's decision to learn to swim is the amount of time needed to learn it. They assume that because they had difficulty learning when they were young, they will encounter the same level of difficulty – for those who are pessimistic by nature, even more difficulty. It is actually easier to teach adults to swim. There are those who can even learn to do so, from scratch, in thirty minutes. This is because adults are more logical and are better at overcoming their apprehensions through logic. A child, especially a toddler, because their reasoning has not developed yet is prone to react emotional to the stresses involved, resulting in fear or even phobia.
Adults go through the same process as children when learning to swim. The first step is becoming comfortable in the water. The next step is adjusting to the head being submerged and the technique used when breathing while swimming. This is then followed by floating, propulsion, and confidence in deep water. Each of these steps mentioned can be discussed rationally with the adult then attempted in a logical and systematic manner. Any difficulty that arises is objectively assessed and solved making the objective, learning to swim, an inevitable result after some effort.
More Swimming Lessons in Connecticut:


text:
Obituary: Kenneth E. Fox
Kenneth E. Fox, 76, of Leawood, Kansas, passed away peacefully Thursday, November 15, 2012, at Freedom Pointe Assisted Living in Overland Park, Kansas.
Funeral services will be 1:00 p.m. Tuesday, November 20, 2012, at the Penwell-Gabel Funeral Home, Olathe Chapel. Burial will be 11:00 a.m. Wednesday at Ozawkie Cemetery, Ozawkie, KS.
Ken Fox was born September 11, 1936, in Hamilton, Ohio to Charles and Frances Fox. He was a longtime banker in Kansas and Missouri, serving as CEO or senior executive of banks in Kansas City, Moberly, St. Louis, Sedalia, and Topeka. After retirement he served as a banking consultant, most recently to Community National Bank in Chanute, Kansas.
Ken was preceded in death by his parents and two brothers. He is survived by his wife of 55 years, Belva Fox; by his sons, Kenton, Kevin, and Darrin Fox; and by his granddaughters, Yvonne Fox and Deborah Wilson.
To leave a special message for the family please visit www.PenwellGabelOlathe.com.


text:
Print / Digital Subscription »
Rumors, spy photos, speculation and guesses. It's all over. The 2015 Ski-Doo Summits have finally been released to the public.
The long and yellow mountain sleds have been causing quite a commotion. It seems that a 174-inch long track was just enough of a fantasy that it was brushed off as a rumor by most. And then the leaked photos started popping up and the conversation turned from what if to how would it work. We'll get to that in a moment.
First off, here's everything that's new on the 2015 Summit X T3 models:
We've had the chance to spend a few days on the new T3 174 and T3 163 Summits.
The higher ride height from the RAS 2 and dropped skid location improves this sled's ability to float through snow and cling to a sidehill. The RAS 2 front end is noticeable, and improves the sled's handling both on rough terrain going into the hills and on sidehills and climbs.
The 3-inch lugs on both tracks make a huge difference. The sled may have lower track speed, but that added half inch of traction propels the sled forward better and makes it feel like the sled has more power. The sled can climb out of its own trench and holes like it's toying with you to get stuck. Short of a thick crusty mid layer or a big tree, there's very little that will stop a sled with a 3-inch track, let alone a 174. It's a game-changer for Ski-Doo.
The T3 174 may look like a big, long massive sled that will be nothing but a handful on the hill. Not so. Its huge footprint on deep snow allows it to creep forward at very low speeds. The idea behind it is not to climb straight up (though it does that well) but to allow the backcountry explorer to slow things down and pick better lines with less chance of getting buried on a sidehill cut. We didn't need to carry a lot of momentum to hold steep sidehill lines like we did on the 154 Summit X.
The 174 T3 doesn't necessarily feel any longer than the 163. It's surprisingly well-balanced and nimble in the trees. Actually, it's better in the trees than, say a 154, because it will hold a level plane better while traversing a sidehill. Especially on hills with trenches, old hard tracks, rollers and fingers. In firmer snow and on quick 90-degree turns, you may feel the track driving the sled a bit, but the DS 3 skis are a great match for the added traction.
Where the 174 T3 is all about getting around where others can't in chest-deep snow, the 163 T3 is more of the everyman's sled. It's everything that is superb about the current Summit X 163—just better. The 3-inch track lets this sled do things the 163 2.5-incher just cannot do. It claws at the snow for traction and refuses to just flat out get buried. Even when you think you're stuck, just hammer the throttle and shake the sled side to side. It will climb out of its own hole.
Like what you read?
Want to know when we have important news, updates or interviews?
Join our newsletter today!
Send to your friends!


text:
TIONESTA, Forest County — Scott A. Black of Butler was planning to marry his longtime girlfriend. She was sending out wedding invitations in late December.
But something went wrong. They broke up. She took another boyfriend. Black grew sullen. And jealous. And angry.
He started sending threatening text messages to his former fiance and her new beau, authorities said. He drove by her remote home in Forest County. She conveyed her concern to friends and co-workers.
On Friday night, Black showed up her mobile home. There was trouble. But he left. His ex-girlfriend and her boyfriend called state police.
Troopers arrived and searched for Black. When they couldn't find him, they left.
Later, authorities said, he returned. There was a clash between Black, his ex and his perceived rival. Her teenage daughter was there and ran for help. It would come too late.
Police said Black fatally stabbed Marcelle I. Edwards, 42, and Donald L. Shay, 43, of Leechburg, Armstrong County, who had just moved in with Edwards.
Police in Forest County hours later arrested Black but not before troopers in Butler joined in the manhunt Saturday morning, briefly surrounding his mother's house on Virginia Avenue, where he recently lived.
District Judge George Gregory in Forest County shortly after noon arraigned Black, who turned 38 on Friday, on two counts each of homicide and aggravated assault. He remains in the Warren County Jail without bail.
Black's preliminary hearing is 9:30 a.m. May 7. But Gregory said the hearing would likely be rescheduled to allow the defendant's attorney more preparation time.
During the arraignment, Black told Gregory he planned to apply for a public defender.


text:
Mesa Verde National Park could see more than $300,000 cut from the rest of its 2012-13 fiscal year budget if Congress doesn't reach a deficit reduction compromise by March 1.
According to a memo sent in late January by National Park Service Director Jonathan Jarvis, all 398 national parks, monuments and other historic sites should brace for an across-the-board budget cut of 5 percent. In Mesa Verde's case, that equals $321,000 between now and Sept. 30, the end of the fiscal year.
The cuts are proportional to size of the budget and park. Yellowstone National Park, for example, is pegged for a $1.75 million reduction.
The memo was first obtained by The Coalition of National Park Service Retirees, an advocacy group of 800-plus former NPS employees.
In the memo, Jarvis said permanent hires should be delayed until the budget impasse is resolved, and while temporary seasonal hiring plans can move forward, offers should not be made yet.
In a typical year, Mesa Verde has 60 year-round and 120 seasonal employees (from roughly Memorial Day to Labor Day) on staff, according to spokeswoman Betty Lieurance.
Despite the memo instructing individual parks to detail how projected cutbacks will affect their programs, officials have been vague about possible impacts on Mesa Verde specifically.
Jeffrey G. Olson, NPS acting chief spokesman in Washington, D.C., said he had no park-specific information to offer, but in a prepared statement described some of the potential ramifications for national parks as a whole.
"Visitors would see reduced hours of operation for visitor centers, a shorter season, and possibly closing of camping, hiking and other recreational areas when there is insufficient staff to ensure the protection of visitors, employees, and historic, cultural and natural resources," he wrote.
Olson added that smaller budgets would limit parks' ability to "sustain a full complement of seasonal employees needed for interpretive programs, maintenance, law enforcement and other visitor services as we are preparing for the busy summer season."
Congress will return from a one-week recess on Feb. 25. That leaves just four days to approve an alternative to the automatic cuts, totalling $85 billion this year across all agencies.
The possible dire situation stems from protracted political gridlock over government spending.
In August 2011, Congress raised the federal debt ceiling to allow the U.S. government to continue borrowing money. But in exchange it established a bipartisan group of 12 senators and representatives - "the super committee" - to decide on $1.2 trillion in spending cuts spread out over a decade.
The incentive for them to cooperate was a blanket slashing of most federal agencies, defense and non-defense - called the "sequester" - if they failed.
They failed, setting up the "fiscal cliff" showdown on New Year's Day when the sequester was scheduled to take effect. As part of a last-minute deal that also raised income tax rates on high-earners and ended the payroll tax cut, Congress instead punted the deadline two months into the future: March 1.


text:
Winter jacket from the Repsol-Honda Replica collection. This is the jacket...
Sleeveless winter jacket from the Repsol-Honda Replica collection. This is...
Sign up to get the latest on sales, new releases and more
|Gruppo Pritelli
Gp Racing Apparel is the official merchandise manufacturer for the most important Motorbike riders and Motorcycle manufacturers of MotoGP, Superbike and MXGP.
Jorge Lorenzo, Marc Marquez, Marco Simoncelli, Andrea Iannone, Andrea Dovizioso, Nicky Hayden, Aleix Espargaro, Tom Sykes, Alex Marquez, Takaaki Nakagami, Ducati Corse, Kawasaki and Yamaha have chosen Gruppo Pritelli.
GP Racing Apparel was founded in 2007 by the passion for the world of motorsport. Over the years, the company evolved to offer its customers a huge variety of clothing and accessories dedicated to the world of Motorbike riders. A rich catalog, including: T-shirts, Sweatshirts, Hoodies, Caps, Polo shirts, Jackets, Keyrings, Flags, Stickers and many other accessories for the riders' fans. Each article is distinguished by original design, rich and distinctive, able to adapt to fans of all ages.✖
Haga sus compras con el sistema de pago más popular en el mundo.
Compre con cualquier tarjeta de crédito a través del circuito bancario Setefi.


text:
If your father or husband is a DIYer like my husband, then he probably has a lot of tool boxes. But, does he have a great tool bag? Veto Pro Pac tool bags are the best quality for the money, and they can replace many of those tool boxes your man has.
Veto Pro Pac Tool Bags are tough, rugged, heavy, waterproof, and hold a lot of tools so it can be used on any job. You can help get your man organized.
My husband and I got to review the Veto Pro Pac Closed Top LC and (Open Top) OT- XL bags. Henry loves them both. He can't stop talking about how well made they are.
The Veto Pro Pac OT-XL tool bag below, is just as great. It's heavy duty for a pro to carry to the job site or a DIYer to use forever and not have to replace. It has 40 tiered and four zippered pockets. Cost is $179.95. A smaller version of this bag is available.
Henry shows how large the zipper is so you can see it's heavy duty. It works easily, too.
The LC caters to the more job-specific crowd. With 57 pockets capable of handling 75 hand tools. It's constructed from bomb-proof pvc-impregnated 1800 denier and bottomed off with a 3mm thick polypropylene bottom. An outside-mounted tape measure clip is a favorite feature of the bag. And are you ready? These bags have ergonomic handles and a five-year warranty.
The OT-XL has a capacity to hold over 75 hand tools. Top-in loading and unloading allows for grab and go. They're made of pvc-impregnated 1800 denier waterproof nylon bodies and 3mm thick polypropylene bases
Buy it here
Subscribe to Veto Pro Pac newsletter by email, after clicking, it's in the top right corner, then let me know what email you used here so it can be verified with the sponsor. (3 entries)
You also must leave your email or have it available in your profile to be contacted, or you will be disqualified and another name will be drawn. I have to be able to reach you.
1. Follow my blog on GFC and leave name used- 2 entries
2. Subscribe to my blog RSS - 1 entry
3. Follow me on Twitter http://twitter.com/blp3 and leave your Twitter name - 1 entry
4. Follow me on Facebook http://www.facebook.com/barbaralplatt and leave me your FB name (2 entries)
5. Follow me on Network Blog and leave your name - right sidebar (2 entries)
6. Enter any of my other giveaways and tell me which ones - 1 entry each
7. Add my button to your blog and leave me a link - right sidebar ( 3 entries)
8. Subscribe to my blog, leave email - 3 entries
9. Tweet about this contest, leaving a comment with the link - 1 entry each
10. Like Veto Pro Pac on Facebook leave your name here - 2 entries
11. Make a Veto Pro Pac purchase (leave info about it here) - 15 entries
12. Subscribe to Veto Pro Pac You Tube leave a message there and leave your name here that you signed in with - 2 entries
13. Vote for my Beat and tell me which one – 1 entry each one daily
14. Blog about this contest and leave me a link - 10 entries
Post pictures of your current tool bag/box on the Veto Pro Pac FB page Contest thread and say Barbara's Beat sent you, then write a brief description of how you think a Veto bag will make life easier for you. All that enter will win a Veto t-shirt. The best entry might win a Veto tool bag of their choice. (Veto has a history of awarding more than one winner!). If you happen to own a Veto Pro Pac tool bag, then post pictures of your Veto tool bag and write a brief description of how your Veto has made your life easier or why you like the bag.
or Post a video on the CONTEST thread and say Barbara's Beat sent you, with your existing bag and explain the shortcomings and why you think a Veto bag will make life easier for you.
Don't forget Veto Pro Pac is offering a 5 year No Downtime warranty on all defects, which means for example, the annual cost of the LC bag is about $26. You can't beat that math.
Full disclosure: We received Veto Pro Pac Tool Bags to do this review.


text:
After a bone-dry summer and fall, the storm system that passed through southern Colorado last weekend was cause for celebration and relief among area ski resorts, which are gearing up for opening day.
First on the scene, as usual, is Wolf Creek Ski Area. The recent front, dubbed Winter Storm Brutus, dumped about 20 inches of powder on the slopes, and four of five lifts will be operational by Wednesday. The lone exception is the Raven Lift, which plans to start running around Thanksgiving. Rosanne Pitcher, vice president of marketing and sales, expected the fresh snow to settle to a base depth of 18 inches.
Nov. 14 is more than one month later than opening day last year. Early-season snowfall last year allowed Wolf Creek to open for business Oct. 8, 2011, and led to a record number of 227,000 visitors, although the snowy weather fizzled from February onward. Despite the later start in 2012, Pitcher is optimistic.
Were excited to open as much of the mountain as we are, about 500 acres of beginner, intermediate and some expert terrain. We dont tell people to stay only on groomed trails, so we advise skiers to use caution and watch out for early-season obstacles, she said.
Lift tickets will be sold at a discounted rate until more of the mountain opens.
Wolf Creek skiers and snowboarders will find two new hilltop respites from the cold this year. The Continental, a café at the apex of Treasure Lift, will dispense hot drinks and freshly baked pastries, and was designed for sustainability.
The structure was originally going to be just restrooms, but we decided to make a food venue out of it. (The building) is made of Wolf Creek timber and features water-free composting restrooms, Pitcher said.
The Ravens Nest, a pizzeria and bar atop Raven Lift, was partly built last year and had a soft open, but will debut in full this season.
Skiers looking to take advantage of Purgatory at Durango Mountain Resorts opening day will have to shake off their tryptophan-induced stupor Nov. 23, the day after Thanksgiving. As of Monday night, snow cannons were supplementing the foot of powder that fell during the weekend.
Several new developments are underfoot for the 2012-13 season.
DMRs Ski School is partnering with manufacturer Rossignol to incorporate the latest rocker designs into lessons to help expedite the learning curve for new skiers. The shape mimics water skis and is supposed to help beginners stay afloat atop soft powder and maneuver with greater ease.
Continuing to upgrade mountain conditions is another primary objective at Purgatory.
Building on last years high angle grooming initiative, a new pick point for the state-of-the-art winch cat has been installed on Lower Hades (run), spokeswoman Kim Oyler said in a statement. During the off-season, the mountain operations crew has been removing brush and small conifers from the trails, which will create a better early season slope experience.
In keeping with the emphasis on quality grub, DMR restaurants are now offering more choice for visitors with dietary restrictions, including vegetarian and gluten-free options. The resort also is making an effort to feature locally sourced products on the menu, Oyler said.
Purgatory is, again, selling a two-day punch pass for $89, valid from Jan. 7, 2013, to Feb. 15 (a full-price one-day adult lift ticket runs $75). The pass can be used on separate days by one individual or shared on the same day by two people. A new addition is the Local 50/50 Card, which skiers can pick up at the DMR office in Bodo Industrial Park. It allows skiers to purchase $50 lift tickets throughout the season, and is ideal for those who plan to ski three to five times, Oyler said.
Local Benefit Days, featuring $40 lift tickets, of which a portion is donated to charity, are Dec. 19, Jan. 16, Feb 13 and March 6.
DMR is extending an olive branch to senior citizens indignant about season pass price hikes that go into effect this year. The new Senior Days program allows those 65 and older to purchase a $59 package, on select days, that includes a lift ticket, valet parking and a lunch special. Those with season passes can get the deal for $19.
For the more ruggedly minded, Silverton Mountain is slated to open Dec. 1, but manager Aaron Brill said that date is flexible, and it could be moved up if conditions allow.
Telluride Ski Resort is targeting Thanksgiving Day, Nov. 22.
As for the long-term winter outlook, forecasters are ambivalent.
Temperatures are expected to be warmer than average, but precipitation is tough to predict, said meteorologist Chris Cuoco with the National Weather Service in Grand Junction.
Were in an El Niño pattern, which in this part of the country doesnt give a good signal or indication of rain and snow frequency, he said. So far, it has shaped up to be on the drier side.


text:
- New:
- 770-674-6077
- Used:
- 770-674-6077
- Service:
- 770-674-6077
- Parts:
- 770-674-6077
- Contact Us Now:
- 770-674-6077
AutoNation Ford Marietta
AutoNation Ford Marietta Videos
Check out our extensive collection of videos. Just click any video to learn more!Select Another Make:
AutoNation Ford Marietta Ford Certified PreOwned 15 Video
Ford Certified PreOwned 15 Video
**Although every reasonable effort has been made to ensure the accuracy of the information contained on this site, absolute accuracy cannot be guaranteed. This site, and all information and materials appearing on it, are presented to the user "as is" without warranty of any kind, either express or implied. All vehicles are subject to prior sale. Price does not include applicable tax, title, license, processing and/or documentation fees, and destination charges. Not responsible for typographical or data errors. See your local dealer for actual price of vehicles displayed.