text:
SNAP To It: Dell Proposes Way Of Seeing Returns On Social Media Investments
Are you seeing a return on your investment in social media? When the question about whether such returns exist was put to the audience at the recent Sentiment Analytics Symposium by Shree Dandekar, Dell Software's chief strategist and senior director of BI and analytics, only a few hands went up. But Dandekar explained that it's more possible to realize returns than many people may believe.
Dell's Social Net Advocacy Pulse, or SNAP, tool and program is designed to help drive those returns. "Social ROI is not a myth but a reality," Dandekar said. "It starts with a social media strategy and text analytics is a crucial underpinning of that journey," which takes a company from social media listening and monitoring, to capturing and aggregating data from that, to engaging on and deriving insights from social media, to bringing that information into context with enterprise data for better lead- and opportunities-tracking. Dell is an in-house user of SNAP, bringing in 25,000 to 30,000 conversations a day for dell, he noted, and it runs a Social Media Command Center for facilitating listening to those conversations. (It also helps customers implement their own Command Centers.)

subset: commoncrawl


text:
- System Configuration
- Access to Blacklight
- File Spaces
- Transferring Files
- Improving Your File Transfer Performance
- Creating Programs
- Running Jobs
- Monitoring and Killing Jobs
- Debugging
- Improving Performance
- Software Packages
- The Module Command
- Blacklight and XSEDE
- Stay Informed
- Publications
- Reporting a Problem
System Configuration
Blacklight is an SGI UV 1000cc-NUMA shared-memory system comprising 256 blades. Each blade holds 2 Intel Xeon X7560 (Nehalem) eight-core processors, for a total of 4096 cores across the whole machine. Each core has a clock rate of 2.27 GHz, supports two hardware threads and can perform 9 Gflops. The sixteen cores on each blade share 128 Gbytes of local memory. Users can run shared memory jobs of up to 16 Tbytes.
Blacklight runs an enhanced version of the SuSE Linux operating system.
Blacklight also has frontend processors. You login to one of these frontend processors, not to blacklight's compute processors. You will usually prepare your job files and compile your programs on one of these front ends, but you should not run your programs on the frontends because they have limited resources. You should instead run your programs using blacklight's batch system. Thus, frontend computations will be killed if they consume too much processor computational time or processor memory. A message will be sent to your display indicating how you violated these resource limits if you run one of these offending computations. How to run a job on blacklight's compute processors with the qsub command is discussed below.
The Intel C, C++ and Fortran compilers and the Gnu Fortran, C and C++ compilers are installed on blacklight, as are the facilities to enable you to run threaded, MPI and hybrid threaded and MPI programs. OpenMP programs are a common type of threaded programs that are supported on blacklight. UPC and Java are available on the machine and are briefly documented below. In the future, more programming models which exploit the unique architectural features of blacklight will be available. When those models are available their use will be described in this document.
Also available are an array of performance tools, debuggers and libraries. A complete list of packages installed on blacklight is available online.
Access to Blacklight
Should you get a blacklight account?
XSEDE provides a set of tools and services to help you to decide if blacklight or any other XSEDE supercomputer is appropriate for your work. XSEDE provides a catalog of available XSEDE resources and their features. XSEDE also provides a list of available science gateways. Gateways are specialized interfaces that can make your work easier. Finally, XSEDE provides a list of Campus Champions. A Campus Champion is a local contact you can use to discuss whether a supercomputer would be useful for your work.
Once you decide you would like to use an XSEDE resource, such as blacklight, you must apply for a grant through the XSEDE User Portal. The first step in using the Portal is to apply for a Portal account. Then you can determine if you are eligible to become a Principal Investigator for an XSEDE grant. Normally your first XSEDE proposal will be for a Startup grant, but you can also apply directly for a Research grant. If you are applying for a Startup grant do not spend a lot of time estimating your SU usage. One of the purposes of a Startup grant is to make such estimates. A complete description of the allocation policies for applying for grants for using XSEDE resources is available online.
Getting an account on blacklight
PSC's blacklight compute resource allocations are managed by the XSEDE program, which is supported by the National Science Foundation. For more information on applying for an allocation on blacklight see the instructions at
For security reasons each user should apply for his or her own PSC account.
Managing your blacklight account
A second issue that can derail your project is running out of SUs. A Service Unit (SU) is defined as the use of one core for one hour. When you receive a grant, you are awarded an amount of computing time on a resource defined in terms of SUs. Each grant also has an expiration date. To prevent the situation of running out of SUs from occurring, so that you can continue to compute without interruption, you should frequently monitor your SU balance with the blacklight xbanner command. If you are running out of SUs you have several options. If your grant does not expire for at least three months, you can submit a Renewal proposal for the next XRAC meeting and then ask for up to 25% of your SUs in your new award as an advance. If your grant expires in less than three months you can request a Supplement to your current grant. A third option is to transfer SUs from another machine on your grant to blacklight.
These actions of submitting a Renewal request, asking for an advance, asking for a Supplement or transferring SUs must all be done by the grant's PI through the XSEDE POPS system. This system must be accessed through your User Portal account. Detailed descriptions for these operations with your SUs are available online.
Another event that can interrupt your computing is the expiration of your grant. Even though you have distinct PSC and XSEDE Portal userids they are linked as far as expiration is concerned. When one expires the other will expire. To insure that you are not caught unawares by a grant expiration you should continually monitor your expiration dates with the blacklight xbanner command. The XRAC committee meets every three months. Thus, when your grant approaches three months from expiration the PI for the grant should submit a Renewal request to XRAC. If you have an expiring Startup grant the PI should almost always submit a Research proposal to XRAC. Renewals of Startup grants are rarely given. The data you gathered from using your Startup grant can be used as justification in your Research proposal.
If your grant is going to expire and you estimate that you will have SUs left when it does, you can respond differently. For example, you obtained a Startup grant, but did not use it very much. In this situation, instead of submitting a proposal for a new grant, you can apply for an Extension of up to six months for your existing grant.
These actions of submitting a Renewal request, submitting a Research proposal or asking for a grant Extension must all be done by the grant's PI through the XSEDE POPS system. This system must be accessed through your User Portal account. Detailed descriptions of the policies for these operations are available online.
A fourth impediment to the progress of your project is the need to add a user to a grant. The PI for the grant must add all users to a grant by using the Add User form, which must be accessed through the PI's User Portal account. Each user to be added must have already created, by using the User Portal, a User Portal userid. The policies for adding a user are described online
Running Gaussian on blacklight
Just getting an account on blacklight is not sufficient to give you access to Gaussian if you want to use Gaussian. You must fill out our online PSC Gaussian User Agreement to get access to Gaussian at PSC.
Connecting to blacklight
There are three methods you can use to connect to blacklight.
You can connect to blacklight by using XSEDE's single-signon process, either through the XSEDE Portal or by downloading the necessary software components to your local machine. There is also a single-signon hub you can use if you want a command-line interface, but still want to have single-signon access to your XSEDE resources. These methods are especially useful if you have allocations on more than one XSEDE resource. To use these methods you must have an XSEDE grant. To connect through the Portal you use your XSEDE Portal userid and password.
You can also connect to blacklight by using ssh to connect to blacklight.psc.xsede.org. When you are prompted for a password by ssh enter your PSC Kerberos password. Your PSC Kerberos password is not the same as your XSEDE Portal password nor is your PSC userid necessarily the same as your XSEDE Portal userid. You will need to set your PSC Kerberos password before the first time you connect to blacklight using ssh. This method can be used with any type of blacklight grant.
Finally, you can connect to blacklight using a public-private key pair. The procedure to do this is available online. You will need to set your PSC Kerberos password before you can use this procedure. This method can be used with any type of blacklight grant.
Changing your PSC Kerberos password
Your PSC Kerberos password is not the same as your XSEDE Portal password. Resetting one password does not change the value of the other password. You use your PSC password to connect to blacklight and other PSC machines with ssh. You use your XSEDE password when you connect using the XSEDE Portal.
There are two ways to change or reset your PSC Kerberos password:
- Use the web-based PSC password change utility
- Use the kpasswd command to change your PSC Kerberos password. Do not use the passwd command.
You have the same password on all PSC production platforms. When you change your password, whether you do it via the online utility or via the kpasswd command on one PSC system, you change it on all PSC systems.
PSC Kerberos passwords must be at least 8 characters in length. They must also contain characters from at least 3 of the character classes:
- lower-case letters
- upper-case letters
- digits
- special characters, excluding ' and "
Finally, they must not be the same as any of your previous passwords.
You must change your blacklight password within 30 days of the date on your initial password form or your password will be disabled. We will also disable your password if you do not change it at least once a year. We will send you an email notice warning you that your password is about to be disabled in the latter case. See the PSC password policies for more information.
If you have a password issue and communicate with PSC about this problem through remarks do not include your password in an email messasge.
Changing your login shell
You can use the chsh command to change your login shell. When doing so, specify a shell from the /usr/psc/shells directory.
Accounting on blacklight
One core-hour on blacklight is one SU. Because resources are allocated by blades and not by cores--jobs do not share blades--your SU charges will always be based on core usage that is a multiple of sixteen. A one blade job that runs for one hour costs 16 SUs.
There are several tools you can use to track your account usage data. User accounting data is available with the xbanner command. Account information including the initial SU allocation for a grant, the number of unused SUs remaining for a grant and the date of the last job that charged to a grant are displayed.
Accounting information is available at the Web-based PSC Grant Management System. You will need your PSC Kerberos password to access this system. This system provides more detailed information than xbanner, although some of the information is only available to PIs, co-PIS and individuals designated by the PI to see accounting data. The system has extensive internal documentation.
You can use the xdusage command on blacklight to see your blacklight accounting data. You can also use the -r option to xdusage to check your accounting data on other XSEDE resources. A man page is available for xdusage as well as online documentation.
Finally, the Portal login dashboard displays your accounting data for all your XSEDE resources.
File Spaces
File systems
File systems are file storage spaces directly connected to a system. There are currently three such areas available to you on blacklight.
This is your home directory. Your $HOME directory has a 5-Gbyte quota. $HOME is visible to all of blacklight's compute and frontend processors. $HOME is backed up daily, although it is still a good idea to store your important $HOME files to the Data Supercell. The Data Supercell, PSC's file archival system, is discussed below.
You can check your home directory space usage using the command
du -sh
Once you cd to your home directory you can issue this command.
This is blacklight's scratch area to be used as a working space for your running jobs. $SCRATCH is visible to all of blacklight's compute and frontend processors. $SCRATCH is a parallel file system. The current capacity of $SCRATCH is 291 Tbytes.
$SCRATCH is not a permanent storage space. Files can only remain on $SCRATCH for up to 21 days and then we will delete them. In addition, we will delete $SCRATCH files if we need to free up space to keep jobs running. Finally, $SCRATCH is not backed up. For these three reasons, you should store copies of your $SCRATCH files to your local site or to the Data Supercell as soon as you can after you create them. The Data Supercell, PSC's file archival system, is discussed below. For information on improving your $SCRATCH IO performance see the section on IO optimization below.
You should use $SCRATCH as your scratch area. You should not use /tmp for this purpose. You should never need to write data to /tmp.
You can check your scratch directory space usage by using the command
du -sh
Once you cd to $SCRATCH you can issue this command.
XWFS is a wide area file system mounted on blacklight. It is also mounted on all major XSEDE computational machines to facilitate seamless file sharing between XSEDE systems.
You apply for an XWFS allocation through the XSEDE Portal. When you apply for an XSEDE allocation you specify a space quota, a list of users and a project name. The project name determines the name of your XWFS project directory under the directory /xwfs/projects, which is mounted on all major XSEDE computational platforms. Your XWFS quota is a group quota. The group includes all the users you listed in your XWFS allocation. Eventually the PI will be able to see the group's usage through the XSEDE Portal. For now you can use local Unix commands like du -sh.
You can use Unix commands, such as ls or cp, to manipulate your XWFS files on XSEDE systems. Any files you locally create and store into the XWFS file system will be available on other XSEDE machines. The XWFS file system is only mounted on frontend nodes, not on compute nodes, for performance reasons. Any file transfer commands you can use with your other files can be used with XWFS files.
File repositories
File repositories are discrete file storage spaces. You will not, for example, open a file that resides in a file repository nor will you run a program on a file repository. You will not login to a file repository. You currently have one file repository available to you on blacklight: the Data Supercell, PSC's file archival system.
- The Data Supercell
The Data Supercell is a complex disk-based archival system.
Transferring Files
A variety of transfer methods are available to copy files to and from blacklight's file systems. These methods are discussed below.
Since your blacklight home directory has a limited amount of space, you will not be able to transfer much data into your home directory. Exceeding your home directory quota will prevent you from writing more data to your home directory, and will adversely impact other operations you might want to perform. Instead, you should transfer your files to the Data Supercell. Then, when you need your files on blacklight you can transfer them to your blacklight scratch area. Your scratch area is not a permanent storage space. Thus, you must manage the transfer of your files between $SCRATCH and the Data Supercell so that files you want to retain are not deleted by the 21-day wiper. Nor are your scratch files backed up, which is an additional motivation for you to move your files to the Data Supercell or to your local storage in a timely manner.
You can also copy files directly into your scratch area, although you must be aware of the 21-day wiper and of the fact that scratch files are not backed up when storing files in this area. The discussion of file transfer methods below will describe how to copy files directly into $SCRATCH. You should consult the Data Supercell user document for information on copying files into the Data Supercell.
Our recommend method for file transfer into blacklight is Globus Online, both between your local file system and blacklight and between blacklight and the Data Supercell. To be able to use Globus Online you must obtain proper credentials, either from XSEDE or InCommon. If you cannot use Globus Online, but do have access to Globus client software, our next recommended method is to globus-url-copy. Otherwise, you should use either the tar methods discussed below or scp. You should the former methods if you are transferring a lot of small files.
Globus Online
Globus Online is our recommended method of transferring data to and from the Data Supercell. To use Globus Online you must first create a Globus Online userid and password at the Globus Online Web site. Once you have logged in to Globus Online you can initiate your file transfer. For each file transfer you must select two Globus Online endpoints, to which you must then authenticate. The endpoint to use for blacklight is xsede#pscdata. If you are an XSEDE user you can use your XSEDE Portal userid and password to authenticate to the Globus Online endpoint xsede#pscdata. When connecting to the xsede#pscdata endpoint on Globus Online you may be redirected to the XSEDE OAuth page to enter your XSEDE Portal username and password for authentication. After authentication, you will automatically be returned to the Globus Online site to initiate your transfers.
If you are affiliated with an InCommon institution you can use your userid and password for that institution to authenticate to endpoint psc#dsc-cilogon if you have previosuly registered with PSC as an InCommon user. To register with PSC as an InCommon user you must follow the steps below. You only need to follow these steps once.
- Point your browser to https://cilogon.org/, select your institution from the 'Select and Identity Provider' list, and click on the 'Log On' button. This should take you automatically to a login page for your institution.
- Enter your username and password for your institution on the login page, and click on the 'Login' button. You should then get redirected back to the CILogon Service webpage.
- In a box near the top of the CILogon Service webpage there should appear a field called "Certificate Subject" with a string like
/DC=org/DC=cilogon/C=US/O=My Institution/CN=My Name A1234Copy this string to a clipboard or text file. You'll need to paste it into a field on another webpage shortly.
- Log Off from the CILogon Service webpage.
- Point your browser to
https://dirs.psc.edu/cgi-bin/teragrid/userpage/list.pland login with your PSC userid and password. This site lists the certificate subjects (DNs) that we have in our PSC database for your PSC account. You will be adding the CILogon certificate subject (DN) to this list.
- Click on the 'Add DN' link at the top left. This should get you to the "Adding DN" page, where you paste into the DN: field the certificate subject that you copied in step 3 above. Make sure there are no extra spaces before or after the pasted string. Click on 'Create' to add your new CILogon DN (certificate subject) to the PSC database.
- Click on the 'List DNs' link at the top left to confirm that your new DN was added.
If you do not enter a path for the endpoints xsede#pscdata or psc#dsc-cilogon your destination will be your Data Supercell home directory. Thus, you must explicitly enter the path to your blacklight scratch directory to make it the target of your file transfer. We do not recommend transferring large files into your blacklight home directory, because your home directory space is limited. To use Globus Online to transfer files into your home directory you must enter the path to your home directory.
You can also use Globus Online to transfer files between your blacklight scratch space and the Data Supercell. In fact, Globus Online is our recommended method for performing this type of transfer. To perform this transfer you would use xsede#pscdata for both endpoints, since xsede#pscdata can also be used to point to blacklight. To make your second endpoint point to your brashear files you must enter as your Globus Online path for this endpoint the complete path to your brashear directory. Otherwise, Globus Online will use your Data Supercell home directory as your path in the transfer.
For example, suppose you want to transfer brashear file largematrix.dat to your Data Supercell home directory. You would enter xsede#pscdata for both endpoints. For the path for the endpoint you are using to point to blacklight you must enter your path as /brashear/joeuser/largematrix.dat. For the other path, the path pointing to the Data Supercell, you can enter just largematrix.dat, since you are storing the file in your home directory on the Data Supercell. Then you can initiate the transfer by clicking the appropriate arrow button.
The tar command, in conjunction with the ssh command, can also be used to transfer files to blacklight. This approach is especially useful if you are transferring a lot of small files.
There are two methods of using tar. In both methods you use tar as part of your transfer command. The first method enables you to have a tarball on blacklight without having to create one on your source system. If you create a tarball on your source machine before you transfer it to blacklight you double your storage usage on your source system. The first method of using tar in your transfer command avoids this doubling of space usage. A sample command is
tar cf - sourcedir/ | ssh blacklight.psc.xsede.org "cat > /brashear/joeuser/bldir.tar"
For 'sourcedir' you substitute the name of your source directory and for 'joeuser' you substitute the name of your PSC userid. The result of this command will be the tarball bldir.tar on blacklight without the need to first create it on your source system. You can run the process in reverse when you copy files from blacklight back to your source system.
In the second method the result is not a tarball, but the same set of individual files you have on your source system. A sample command is
tar -cf sourcedir/ | ssh blacklight.psc.xsede.org tar xf - -C /brashear/joeuser
For 'sourcedir' you substitute the name of your source directory and for 'joeuser' you substitute your PSC userid. The result will be the same as the equivalent scp command, but the transfer will be faster. You should use the first method of using tar if you have a lot of individual files, because your performance will deteriorate if you transfer a lot of individual files.
If you are not concerned about your local disk storage or the performance of scp or want to use a simpler method, you can use tar on your local system and then use scp to transfer your file to the Data Supercell and from there to your blacklight scratch area.
A sample set of commands on your local machine would be
tar cf sourcedir.tar sourcedir
For 'joeuser' you substitute your PSC userid. You can compress your tarball before you transfer it to speed up your transfer times. Then you could login to blacklight and issue the commands
cd $SCRATCH
tar xf /arc/users/joeuser/sourcedir.tar
Again for 'joeuser' you substitute your userid. This will unroll your tar file in your scratch directory. The scp command is discussed below.
Sftp and scp
You can use the SSH file transfer clients sftp and scp to transfer files between your local systems and blacklight filesystems. When using sftp or scp to transfer files to and from blacklight you do not connect directly to blacklight.You transfer files using a PSC high-speed data conduit named data.psc.xsede.org. You transfer files to and from blacklight via data.psc.xsede.org. If you are not connected to the data conduit from an XSEDE host you must use the name data.psc.edu for the data conduit. If you have a graphical sftp or scp client application on your local system, you can use it to connect and authenticate to data.psc.xsede.org or data.psc.edu and transfer files accordingly. Use your PSC userid and password for authentication.
If you need to (re)set your PSC password, you can do so via the kpasswd command on any PSC production system, or using the http://apr.psc.edu/ Web form.
You can use the command-line sftp client to transfer files to and from blacklight interactively. When using sftp from the command line, you first connect and authenticate to data.psc.xsede.org, and then issue commands at the
sftp> prompt to transfer and manage files:
joeuser is your PSC userid. The first time you connect to data.psc.xsede.org using sftp or scp, you may be prompted to accept the server's host key. Enter
yes to accept the host key:
The authenticity of host 'data.psc.xsede.org (22.214.171.124)' can't be established. RSA key fingerprint is d5:77:f2:d9:07:f6:32:b6:c3:eb:0d:d1:29:ed:9b:80. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'data.psc.xsede.org' (RSA) to the list of known hosts.
You will then be prompted to enter your PSC password:
Connected to data.psc.xsede.org.
At the
sftp> prompt, you can then enter sftp commands (e.g.,
get, etc.) to manage and transfer your files to/from blacklight. Enter a question mark (
?) for a list of available sftp commands.
Your default directory when using data.psc.xsede.org is your Data Supercell home directory. To access your blacklight scratch directory you must change your directory to be your blacklight scratch directory or use your entire scratch directory path in your sftp command. Your scratch directory path will be similar to /brashear/joeuser. To access your blacklight home directory you must change your directory to be your blacklight home directory or use your entire home directory path in your sftp command. Your home directory path will be similar to /usr/users/0/joeuser.
Examples (where
joeuser is your PSC userid, and entered commands appear in
- Where am I (on the data.psc.xsede.org server)?
sftp> pwd Remote working directory: /arc/users/joeuser
- Change directories to my blacklight
$SCRATCHdirectory :
sftp> cd /brashear/joeuser Remote working directory: /brashear/joeuser
- Where am I on my local system?
sftp> lpwd Local working directory: /Users/JoeUser/Documents
- Change directories on my local system to
sftp> lcd /usr/local/projects/example/data
- Make a new directory called "
newdata" under my blacklight
$SCRATCHdirectory :
sftp> mkdir /brashear/joeuser/newdata
- Copy a file (
file1.dat) from my current local directory to my blacklight
$SCRATCH/newdatasubdirectory :
sftp> put file1.dat /brashear/joeuser/newdata/file1.dat Uploading file1.dat to /brashear/joeuser/newdata/file1.dat file1.dat 100% 1016KB 1.0MB/s 00:00
- Copy a file from my blacklight
$HOMEdirectory (
/usr/users/0/joeuser/file1) to
/usr/local/projects/example/data/newfile1on my local system :
sftp> get /usr/users/0/joeuser/file1 /usr/local/projects/example/data/newfile1 Fetching /usr/users/0/joeuser/file1 to /usr/local/projects/example/data/newfile1 /usr/users/0/joeuser/file1 100% 31 0.0KB/s 00:00
- Exit from this sftp session :
sftp> exit
sftp> bye
For scripted transfers, or transfers that you want to execute directly from your command-line shell, you can use the SSH scp client:
Examples (where
joeuser is your PSC userid, and entered commands appear in
bold, and you enter your PSC password when prompted):
- Copy my local file (
/usr/local/projects/example/data/file1.dat) to my blacklight
$SCRATCHdirectory :
- Copy the contents of my blacklight
$SCRATCH/newdatadirectory to
/usr/local/projectson my local system (creating
/user/local/projects/newdataand copying all the files there):
XSEDE users may use GridFTP to transfer files to and from blacklight filesystems.
To use the command-line globus-url-copy client on a blacklight login node, first ensure that you have a current user proxy certificate for authentication with enough time on it to complete your transfer, e.g.:
joeuser@tg-login1:~> grid-proxy-info subject : /C=US/O=National Center for Supercomputing Applications/CN=Joe User issuer : /C=US/O=National Center for Supercomputing Applications/OU=Certificate Authorities/CN=MyProxy identity : /C=US/O=National Center for Supercomputing Applications/CN=Joe User type : end entity credential strength : 2048 bits path : /tmp/x509up_u99999 timeleft : 11:58:33
If the timeleft is not sufficient, or you get an "
ERROR: Couldn't find a valid proxy" message, then use myproxy-logon (or if you have your own long term user certificate, grid-proxy-init) to obtain a new user proxy certificate, e.g.:
joeuser@tg-login1:~> myproxy-logon -l joexsedeuser -t 24 Enter MyProxy pass phrase: A credential has been received for user joexsedeuser in /tmp/x509up_u99999.
joexsedeuser is your XSEDE User Portal login name,
-t 24 requests a 24-hour certificate, and the
MyProxy pass phrase entered is your XSEDE User Portal password.
You can then use globus-url-copy to transfer files to/from blacklight filesystems using the GridFTP server address gsiftp://gridftp.psc.xsede.org. This transfer will go through the PSC high-speed data conduit data.psc.xsede.org.
Note: The
gsiftp:// URLs are absolute paths to files. This means that when referring to a file or directory in your blacklight
$SCRATCH directory, you must use
gsiftp://gridftp.psc.xsede.org/brashear/joeuser/, where
joeuser is your blacklight userid. Likewise, for your blacklight
$HOME directory, you must use
- List the files in my blacklight
$SCRATCHdirectory :
joeuser@tg-login1:~> globus-url-copy -list gsiftp://gridftp.psc.xsede.org/brashear/joeuser gsiftp://gridftp.psc.xsede.org/brashear/joeuser/ file1.dat file2.dat file3.dat newdata/ olddata/
- Transfer a file (
testfile) from my scratch space on TACC lonestar to my PSC blacklight
$SCRATCH/newdatadirectory :
joeuser@tg-login1:~> globus-url-copy -stripe -tcp-bs 32M \ gsiftp://gridftp.lonestar.tacc.xsede.org/scratch/99999/tg987654/testfile \ gsiftp://gridftp.psc.xsede.org/brashear/joeuser/newdata/
-tcp-bs 32Mare used to improve transfer performance, and
/scratch/99999/tg987654is your scratch directory on lonestar at TACC.
Gsisftp and gsiscp
If you have a current user proxy certificate you can also use gsisftp or gsiscp to tranfer files to and from blacklight. The method of obtaining a valid user proxy certificate is described above in the discussion of the globus-url-copy command. The default directory for both gsisftp and gsiscp is your Data Supercell home directory. Thus, to transfer files to and from blacklight you must specify the full path to your blacklight directory.
A sample gsiscp session is
gsisftp data.psc.xsede.org sftp>get /brashear/joeuser/file1.dat localfile1.dat Fetching /brashear/joeuser/file1.dat to /usr/local/projects/localfile1.dat /brashear/joeuser/file1.dat 100% 31 0.0KB/s 00:00 sftp>bye
A sample gsiscp command is
gsiscp data.psc.xsede.org:/brashear/joeuser/file1.dat localfile1.dat file1.dat 100% 1016KB 101.0 MB/s 00:00
To transfer files between the Data Supercell and blacklight's file systems you can also use PSC's far program. Far is available on all PSC production platforms. In addition to file transfers, the far program can be used for file and directory management, such as getting a list of your files on the Data Supercell. See the far documentation for more information. Globus Online is our recommended method for transferring data between blacklight and the Data Supercell. The far utility is being maintained for backward compatiblity with older scripts. At some time in the future it will not be supported.
We recommend that if you do use far you execute your far commands outside of your batch job scripts so that your jobs do not tie-up compute processors and you do not expend your computing allocation transferring files.
Improving Your File Transfer Performance
File transfer performance between your local systems and Blacklight filesystems can be significantly improved by ensuring that your local systems' networking parameters are optimized. Guidance is available at PSC's Enabling High Performance Data Transfers webpage.
For improved performance when using SSH (sftp or scp), we recommend using an SSH package that includes PSC's High Performance Networking (HPN) patches, e.g., GSI-OpenSSH. For instructions to build OpenSSH with PSC's HPN patches, consult the PSC High Performance SSH/SCP - HPN-SSH webpage.
Creating Programs
The Intel C, C++ and Fortran compilers and the GNU Fortran, C and C++ compilers are installed on blacklight and they can be used to create OpenMP, pthreads, MPI, hybrid and serial programs. The commands you should use to create each of these types of programs are shown in the table below.
|Intel Fortran
|ifort -openmp myopenmp.f90
|ifort -pthread mypthread.f90
|ifort mympi.f90 -lmpi
|ifort -openmp myhybrid.f90 -lmpi
|ifort myserial.f90
|Intel C
|icc -openmp myopenmp.c
|icc -pthread mypthread.c
|icc mympi.c -lmpi
|icc -openmp myhybrid.c -lmpi
|icc myserial.c
|Intel C++
|icpc -openmp myopenmp.cc
|icpc -pthread mypthread.cc
|icpc mympi.cc -lmpi -lmpi++
|icpc -openmp myhyrid.cc -lmpi -lmpi++
|icpc myserial.cc
|GNU Fortran
|gfortran -fopenmp myopenmp.f90
|gfortran -pthread mypthread.f90
|gfortran mympi.f90 -lmpi
|gfortran -fopenmp myhybrid.f90 -lmpi
|gfortran myserial.f90
|gcc -fopenmp myopenmp.c
|gcc -pthread mypthread.c
|gcc mympi.c -lmpi
|gcc -fopenmp myhybrid.c -lmpi
|gcc myserial.c
|g++ -fopenmp myopenmp.cc
|g++ -pthread mypthread.cc
|g++ mympi.cc -lmpi -lmpi++
|g++ -fopenmp myhybrid.cc -lmpi -lmpi++
|g++ myserial.cc
Man pages are available for ifort, icc and icpc and for gfortran, gcc and g++.
The UPC compiler is installed on blacklight. Online instructions are available for its use.
Two versions of Java are available on blacklight: an IBM version and a Sun version. You should load the module for the version you want to use. If you issue the command
module load java
you will load the Sun version. Once you load a Java module the Java compiler and interpreter will then be available to use. A java man page is also available.
Running Jobs
Queue structure
Torque, an open source version of the Portable Batch System (PBS), controls all access to blacklight's compute processors, for both batch and interactive jobs. Currently blacklight has two queues: the batch queue and the debug queue. Interactive jobs can run in the debug queue and the batch queue and the method for doing so is discussed below.
Batch queue jobs can ask for a maximum of 168 hours of walltime.
The maximum walltime for jobs in the debug queue is 30 minutes. You must request 16 cores. The debug queue is not to be used for short production runs. You can only have one job in the debug queue at at time.
You determine how much memory your job will be allocated through the value of your core request.
Scheduling policies
The batch queue is basically a FIFO queue. However, there are mechanisms in place to prevent a single user from dominating the batch queue and to prevent idle time on the machine. The result is some deviation from a strictly FIFO scheme.
Sample batch jobs
To run a batch job on blacklight you submit a batch script to the scheduler. A job script consists of PBS directives, comments and executable commands. The last line of your batch script must end with a newline.
A sample job script to run an OpenMP program is
#!/bin/csh #PBS -l ncpus=16 #ncpus must be a multiple of 16 #PBS -l walltime=5:00 #PBS -j oe #PBS -q batch set echo #move to my $SCRATCH directory cd $SCRATCH #copy executable to $SCRATCH cp $HOME/myopenmp . #run my executable setenv OMP_NUM_THREADS 16 omplace -nt $OMP_NUM_THREADS ./myopenmp
The first line in the script cannot be a PBS directive. Any PBS directive in the first line is ignored. Here, the first line identifies which shell should be used for your batch job.
The four #PBS lines are PBS directives.
- #PBS -l ncpus=16
This directive determines the number of cores allocated to your job. For performance reasons the actual allocation of resources is done by blades, with each blade containing sixteen cores. Jobs do not share blades. You must specify a value for ncpus that is a multiple of sixteen. Within your batch script the environment variable PBS_NCPUS is set to the value of ncpus.
Each blade has 128 Gbytes of physical memory. If your job exceeds the amount of physical memory available to it--a job with a ncpus value of 64 will run on 4 blades and thus have 512 Gbytes of memory available to it--it will be killed by the system with a message similar to
PBS: Job killed: cpuset memory_pressure 10562 reached/exceeded limit 1 (numa memused is 134200964 kb)
written to its stderr. A cpuset is the set of blades--cores and associated memory--assigned to your job. Memory pressure is a metric that indicates whether the processes on a blade are attempting to free up in-use memory on the blade to satisfy additional memory requests. Since this use of memory would result in significantly lower performance, a job that attempts to do this is killed by the system. For more information about cpusets and memory pressure see the man page man 4 cpuset.
Blacklight is a large-memory machine. Thus, the number of cores you select will often be determined by the amount of memory your program needs. Below is a table that gives the amount of memory available for representative numbers of requested cores.
# Cores Memory
16 128 64 512 256 2048 512 4096 1024 8192 1424 13952
These values are only examples. You can request any number of cores that is a multiple of sixteen.
- #PBS -l walltime=5:00
The second directive requests 5 minutes of walltime. Specify the time in the format HH:MM:SS. At most two digits can be used for minutes and seconds. Do not use leading zeroes in your walltime specification.
- #PBS -j oe
The next directive combines your .o and .e output into one file, in this case your .o file. This makes your job easier to debug.
Your stdout and stderr files are each limited to 20 Mbytes if they are not redirected to a file. If your job exceeds either of these limits it will be killed by the system. If you have a program that you think will exceed either of these limits you should redirect either your stdout or stderr output or both to a $SCRATCH file. Moreover, unless you redirect your stdout and stderr output you cannot see it until your job ends.
- #PBS -q batch
The final PBS directive requests that your job be run in the batch queue.
The remaining lines in the script are comments and command lines.
- set echo
This command causes your batch output to display each command next to its corresponding output. This makes your job easier to debug. If you are using the Bourne shell or one of its descendants use
set -xinstead.
- Comment lines
The other lines in the sample script that begin with '#' are comment lines. The '#' for comments and PBS directives must be in column one of your scripts.
- setenv OMP_NUM_THREADS 16
This command sets the number of threads your OpenMP program will use. In the example, your program will have 1 thread per core, since you have requested 16 cores. You can also set the value of OMP_NUM_THREADS so that you have more than 1 thread per core. However, we have found that most threaded programs do not scale well above 32 threads, no matter how many cores you have requested. For most threaded programs 16 threads is an appropriate number to request. However, this number is application-dependent. You should test your application to see what the best number of threads is for your program.
- omplace -nt $OMP_NUM_THREADS ./myopenmp
This command runs your executable. The omplace command insures that your threads do not migrate across your cores.
A sample job to run a pthreads program is
#!/bin/csh #PBS -l ncpus=16 #ncpus must be a multiple of 16 #PBS -l walltime=5:00 #PBS -j oe #PBS -q batch set echo #move to my $SCRATCH directory cd $SCRATCH #copy executable to $SCRATCH cp $HOME/mypthread . #run my executable setenv OMP_NUM_THREADS 16 omplace -nt $OMP_NUM_THREADS ./mypthread
This script is identical to the OpenMP script, except for the name of the executable. The information about thread counts is also the same as for OpenMP programs.
A sample job to run a Java program is
#!/bin/csh #PBS -l ncpus=16 #ncpus must be a multiple of 16 #PBS -l walltime=5:00 #PBS -j oe #PBS -q batch set echo source /usr/share/modules/init/csh #move to my $SCRATCH directory cd $SCRATCH #copy executable to $SCRATCH cp $HOME/MyJavaApp.class . module load java #run my executable java -XX:ParallelGCThreads=16 MyJavaApp
This script is similar to the OpenMP script. The source command is needed because your job is using the module command. In place of the omplace command you use the java interpreter to run your program.
If you are using the IBM version of Java the option to request 16 threads would be
java -Xgcthreads16 MyJavaApp
The Java system call Runtime.getRuntime().availableProcessors() will always return 4096. To get the correct vallue for your number of cores you should instead call System.getenv("PBS_NCPUS") and multiply the returned value by two.
A sample job to run an MPI program is
#!/bin/csh #PBS -l ncpus=16 #ncpus must be a multiple of 16 #PBS -l walltime=5:00 #PBS -j oe #PBS -q batch set echo #move to my $SCRATCH directory cd $SCRATCH #copy executable to $SCRATCH cp $HOME/mympi . #run my executable mpirun -np $PBS_NCPUS ./mympi
You must use mpirun to run your MPI executable. In addition, MPI programs will only run on blacklight's compute nodes. Thus, they must be run using qsub.
A sample job to run a UPC program is
#!/bin/csh #PBS -l ncpus=16 #ncpus must be a multiple of 16 #PBS -l walltime=5:00 #PBS -j oe #PBS -q batch set echo source /usr/share/modules/init/csh #move to my $SCRATCH directory cd $SCRATCH #copy executable to $SCRATCH cp $HOME/myupc . #load the UPC module module load sgi-upc #run my executable mpirun -np $PBS_NCPUS ./myupc
This script is identical to the MPI script above--other than the loading of the sgi-upc module and the associated source command--because SGI UPC uses MPI in its implementation. Thus, the above discussions of the mpirun command and threads apply to UPC jobs.
Information on compiling UPC programs is available online. If you load the sgi-upc module a man page is also available.
A sample job to run a hybrid OpenMP and MPI program is
#!/bin/csh #PBS -l ncpus=256 #ncpus must be a multiple of 16 #PBS -l walltime=5:00 #PBS -j oe #PBS -q batch set echo #move to my $SCRATCH directory cd $SCRATCH #copy executable to $SCRATCH cp $HOME/myhybrid . #run my executable setenv OMP_NUM_THREADS 16 mpirun -np 16 omplace -nt 16 ./myhybrid
The omplace command insures that each of your OpenMP threads do not migrate across your cores. You must use mpirun to run your hybrid executable or it will run on a frontend and degrade overall system performance.
A sample job to run a serial program is
#!/bin/csh #PBS -l ncpus=32 #ncpus must be a multiple of 16 #PBS -l walltime=5:00 #PBS -j oe #PBS -q batch set echo #move to my $SCRATCH directory cd $SCRATCH #copy executable to $SCRATCH cp $HOME/myserial . #run my executable ./myserial
To run a serial program you just give the name of your program as a statement in your batch script. Since you are running a serial program hyperthreading is not an issue. You use your ncpus paramater to ask for the number of cores you need so that you have enough memory in the blades allocated to your job for your program to run. Your serial program will have access to all the memory in all the blades allocated to your job. In the above example your program will have access to 256 Gbytes of memory.
Qsub command
After you create your batch script you submit it to PBS with the qsub command.
qsub myscript.job
Your batch output--your .o and .e files--is returned to the directory from which you issued the qsub command after your job finishes.
You can also specify PBS directives as command-line options. Thus, you could omit the PBS directives from the first sample script above and submit the script with the command
qsub -l ncpus=16 -l walltime=5:00 -j oe -q batch myscript.job
Command-line directives override directives in your scripts.
Flexible walltime requests
Two other qsub options are available for specifying your job's walltime request.
-l walltime_min=HH:MM:SS -l walltime_max=HH:MM:SS
You can use these two options instead of "-l walltime" to make your walltime request flexible or malleable. A flexible walltime request can improve your job's turnaround in several circumstances.
For example, to accommodate large jobs, the system actively drains blades to create dynamic reservations. The blades being drained for these reservations create backfill up to the reservation start time that may be used by other jobs. Using flexible walltime limits increases the opportunity for your job to run on backfill blades.
As an example, if your job requests 64 cores and a range of walltime between 2 and 4 hours and a 64-core slot is available for 3 hours, your job could run in this slot with a walltime request of 3 hours. If your job had asked for a fixed walltime request of 4 hours it would not be started.
Another situation in which specifying a flexible walltime could improve your turnaround is the period leading up to a full drain for system maintenance. The system will not start a job that will not finish before the system maintenance time begins. A job with a flexible walltime could start if the flexible walltime range overlaps the period when the maintenance time starts. A job with a fixed walltime that would not finish until after the maintenance period begins would not be started.
If the system starts one of your jobs with a flexible walltime request, the system selects a walltime within the two specified limits. This walltime will not change during your job's execution. You can determine the actual walltime your job was assigned by examining the Resource_List.walltime field of the output of the qstat -f command. The command
qstat -f $PBS_JOBID
will give this output for the current job. You can capture this output to find the value of the Resource_List.walltime field.
You may need to provide this value to your program so that your program can make appropriate decisions about writing checkpoint files. In the above example, you would tell your program that it is running for 3 hours and thus should begin writing checkpoint files sufficiently in advance of the 3-hour limit so that the file writing is completed when the limit is reached. The functions mpi_wtime and omp_get_wtime can be used to track how long your program has been running so that it writes checkpoint files to make sure you save results from your program's processing.
You may also want to save time at the end of your job to allow your job to transfer files after your program ends but before your job ends. You can use the timeout command to specify in seconds how long you want your program to run. Once your job determines what its actual walltime is you can, after subtracting the amount of time you want for file transfer at the end of your job, use this value in a timeout command. For example, assume your job is assigned a walltime of 1 hour and you want your program to stop 10 minutes before your job ends to allow your job to have adequate time for file transfer. To accomplish this you could use a command like the following
timeout --timeout=$PROGRAM_TIME -- mpirun -np 32 ./mympi
The example assumes that your script has retrieved your job's walltime, converted it to seconds--values given to timeout must be in seconds--subtracted 600 from it and assigned the value of 3000 to the variable PROGRAM_TIME. You will probably also want to provide this value to your program. Your program can then use this value to appropriately write out checkpoint files. When your program ends your job will have time to perform necessary file transfers before your job ends.
How to improve your turnaround
We have several suggestions for how to improve your job turnaround. Firstly, you should try to be as accurate as possible in estimating the walltime request for your job. Asking for more time than your job will actually need will almost certainly result in poorer turnaround for your job. Thus, unreflectively asking for the maximum walltime you can ask for a job will almost always result in poorer turnaround.
Our second recommendation is that you always use flexible walltime requests if possible. This is especially helpful if your minimum walltime in your pair of walltime values is less than 8 hours.
Finally, due to system limitations, we must limit the number of concurrent 16-core jobs on blacklight. Since the number of queued 16-core jobs usually is above this limit, if you are running 16-core jobs, it is to your advantage to pack multiple 16-core executions into a single job. How to pack jobs is discussed in the section below on "Packing jobs."
Interactive access
A form of interactive access is available on blacklight by using the -I option to qsub. For example, the command
qsub -I -l ncpus=16 -l walltime=5:00
requests interactive access to 16 cores for 5 minutes in the. Your qsub -I request will wait until it can be satisfied. If you want to cancel your request you should type ^C.
When you get your shell prompt back your interactive job is ready to start. At this point any commands you enter will be run as if you had entered them in a batch script. Stdin, stdout, and stderr are connected to your terminal. To run an MPI or hybrid program you must use the mpirun command just as you would in a batch script.
When you finish your interactive session type ^D. When you use qsub -I you are charged for the entire time you hold your processors whether you are computing or not. Thus, as soon as you are done executing commands you should type ^D.
X-11 connections in interactive use
In order to use any X-11 tool, you must also include -X on the qsub command line:
qsub -X -I -l ncpus=16 -l walltime=5:00
This assumes that the DISPLAY variable is set. Two ways in which DISPLAY is automatically set for you are:
- Connecting to blacklight with ssh -X blacklight.psc.teragrid.org
- Enabling X-11 tunneling in your Windows ssh tool
Fluent and TAU are among the packages which require X-11 connections.
Other qsub options
Besides those options mentioned above, there are several other options to qsub that may be useful. See man qsub for a complete list.
- -m a|b|e|n
- Defines the conditions under which a mail message will be sent about a job. If "a", mail is sent when the job is aborted by the system. If "b", mail is sent when the job begins execution. If "e", mail is sent when the job ends. If "n",no mail is sent. This is the default.
- -M userlist
- Specifies the users to receive mail about the job. Userlist is a comma-separated list of email addresses. If omitted, it defaults to the user submitting the job. You should specify your full Internet email address when using the -M option.
- -v variable_list
- This option exports those environment variables named in the variable_list to the environment of your batch job. The -V option, which exports all your environment variables, has been disabled on blacklight.
- -r y|n
- Indicates whether or not a job should be automatically restarted if it fails due to a system problem. The default is to not restart the job. Note that a job which fails because of a problem in the job itself will not be restarted.
- -W group_list=charge_id
- Indicates to which charge_id you want a job to be charged. If you only have one grant on blacklight you do not need to use this option; otherwise, you should charge each job to the appropriate grant.
You can see your valid charge_ids by typing
groupsat the blacklight prompt. Typical output will look like
sy2be6n ec3l53p eb3267p jb3l60q
Your default charge_id is the first group in the list; in this example "sy2be6n". If you do not specify
-W group_listfor your job, this is the grant that will be charged.
- -W depend=dependency:jobid
- Specifies how the execution of this job depends on the status of other jobs. Some values for dependencyare:
after this job can be scheduled after job jobid begins execution. afterok this job can be scheduled after job jobid finishes successfully. afternotok this job can be scheduled after job jobid finishes unsucessfully. afterany this job can be scheduled after job jobid finishes in any state. before this job must begin execution before job jobid can be scheduled. beforeok this job must finish successfully before job jobid begins beforenotok this job must finish unsuccessfully before job jobid begins beforeany this job must finish in any state before job jobid begins
Specifying "before" dependencies requires that job jobid be submitted with -W depend=on:count. See the man page for details on this and other dependencies.
Packing jobs
Running many small jobs places a great burden on the scheduler and is probably inconvenient for you. An alternative is to pack many executions into a single job, which you then submit to PBS with a single qsub command. The basic method to use to pack jobs is to run each program execution in the background and place a wait command after all your executions. A sample job to pack serial executions is
#!/bin/csh #PBS -l ncpus=128 #ncpus must be a multiple of 16 #PBS -l walltime=5:00 #PBS -j oe #PBS -q batch set echo #move to my $SCRATCH directory cd $SCRATCH #copy executables and input files to $SCRATCH cp $HOME/myserial* . cp $HOME/serial* . #run my executables dplace -c 0 ./myserial1 > serial1.out < serial1.dat &
dplace -c 32 ./myserial2 > serial2.out < serial2.dat &
dplace -c 64 ./myserial3 > serial3.out < serial3.dat &
dplace -c 96 ./myserial4 > serial4.out < serial4.dat &
Each serial execution will run on 2 blades. The dplace command insures that each execution will run on its own set of 2 blades. The executions will run concurrently.
To pack a job with executables that use threads such as OpenMP executables you should replace the dplace command with the omplace command. A sample job to pack OpenMP executables is
#!/bin/csh #PBS -l ncpus=128 #ncpus must be a multiple of 16 #PBS -l walltime=5:00 #PBS -j oe #PBS -q batch set echo #move to my $SCRATCH directory cd $SCRATCH #copy executables and input files to $SCRATCH cp $HOME/myopen* . #run my executables omplace -nt 32 -c 0 ./myopenmp1 > openmp1.out < openmp1.dat &
omplace -nt 32 -c 32 ./myopenmp2 > openmp2.out < openmp2.dat &
omplace -nt 32 -c 64 ./myopenmp3 > openmp3.out < openmp3.dat &
omplace -nt -32 -c 96 ./myopenmp4 > openmp4.out < openmp4.dat &
A sample job to pack MPI executables is
#!/bin/csh #PBS -l ncpus=64 #ncpus must be a multiple of 16 #PBS -l walltime=5:00 #PBS -j oe #PBS -q batch set echo #move to my $SCRATCH directory cd $SCRATCH #copy executables and input files to $SCRATCH cp $HOME/mympi* . #run my executable mpirun -np 16 dplace -c 0-15 ./mympi > mpi1.out < mpi1.dat &
mpirun -np 16 dplace -c 16-31 ./mympi > mpi2.out < mpi2.dat &
mpirun -np 16 dplace -c 42-47 ./mympi > mpi3.out < mpi3.dat &
mpirun -np 16 dplace -c 48-63 ./mympi > mpi4.out < mpi4.dat &
Packing jobs is especially useful to do if you are running 16-core jobs. Due to system limitations, we must limit the number of concurrent 16-core jobs on blacklight. Since the number of queued 16-core jobs usually exceeds this limit, if you are running 16-core jobs, it is to your advantage to pack multiple 16-core executions into a single job.
Monitoring and Killing Jobs
The qstat -a command displays the status of the queues. It shows running and queued jobs. For each job it shows the amount of walltime and the number of cores and processors requested. To get the actual number of cores a job is using you must divide the displayed value by two. For running jobs it shows the amount of walltime the job has already used
The commands qstat -s, qstat -f and pbsnodes -a can be used to give status information about the system and your jobs. The comments that these commands provide can be used to determine why your jobs have not started. The qstat -f command take a jobid as an argument.
The qdel command is used to kill queued and running jobs. An example is the command
qdel 54
Debugging strategy
Your first few runs should be on a small version of your problem. Your first run should not be for your largest problem size. It is easier to solve code problems if you are using fewer processors. This strategy should be followed even if you are porting a working code from another system.
Do not run a debugging run on any of blacklight's front ends. You should always run a blacklight program with qsub.
Compiler options
Several compiler options can be useful to you when you are debugging your program. If you use the -g option to the Intel or GNU compilers, the error messages you receive when your program fails will probably be more informative. For example, you will probably be given the line number of the source code statement that caused the failure. Once you have a production version of your code you should not use the -g option or your program will run slower.
The -check bounds option to the ifort compiler will cause your program to tell you if it exceeds an array bounds while running.
Variables on blacklight are not automatically initialized. This can cause your program to fail if it relies on variables being initialized. The -check uninit and -ftrupuv options to the ifort compiler will catch certain cases of uninitialized variables, as will the -Wall and -O options to the GNU compilers.
There are more options to the Intel and GNU compilers that may assist you in your debugging. For more information see the appropriate man pages.
Core files
The key to making core files on blacklight is to allow them to be written by increasing the maximum file size allowable for core files. The default size is 0 bytes. If you are using sh-type shells you do this by issuing the command
ulimit -c unlimited
For csh-type shells you issue the command
limit coredumpsize unlimited
Core files are created in directory ~/tmp. For more information about core files issue the command
man 5 core
Little endian versus big endian
The data bytes in a binary floating point number or a binary integer can be stored in a different order on different machines. Blacklight is a little endian machine, which means that the low-order byte of a number is stored in the memory location with the lowest address for that number while the high-order byte is stored in the highest address for that number. The data bytes are stored in the reverse order on a big endian machine.
If your machine has Tcl installed you can tell whether the machine is little endian or big endian by issuing the command
echo 'puts $tcl_platform(byteOrder)' | tclsh
You can read a big endian file on blacklight if you are using the Intel ifort compiler. Before you run your program issue the command
setenv FORT_CONVERTn big_endian
for each Fortran unit number from which you are reading a big endian file. For 'n' substitute the appropriate unit number.
Improving Performance
Calculating Mflops
You can calculate your code's Mflops rate using the TAU utility. The TAU examples show how to determine timing data and floating point operation counts for your program, from which you can calculate your Mflops rate.
Cache performance
Cache performance can have a significant impact on the performance of your program. Each blacklight core has three levels of cache. The primary data and instruction caches are 32 Kbytes each. The L2 cache is 256 Kbytes. The L3 cache, which is shared by the 8 processors on a core, is 24 Mbytes. When hyper-threading is enabled the two threads on a core share the L1 and L2 caches.
Turbo Boost
Blacklight's Nehalem processors have a feature referred to as Turbo Boost. Under certain workload conditions its processor cores can automatically and dynamically run faster than their base clockrate of 2.27 GHz. Although the activation of the Turbo Boost feature is application dependent, we have found that it is most often activiated when only a few cores per processor are being used, because its activation depends on the processor's power consumption and temperature.
Collecting timing data
Collecting timing data is essential for measuring and improving program performance. We recommend five approaches for collecting timing data. The ja and /usr/bin/time utilities can be used to collect data at the program level. They report results to the hundredths of seconds. The TAU utility and the omp_get_wtime and MPI_Wtime functions can be used to collect timing data at a finer grain. The default precision for TAU is microseconds, but the -linuxtimers or -papi option can be used to obtain nanosecond precision. The precision for omp_get_wtime is microseconds, while the precision for MPI_Wtime is nanoseconds.
The ja command turns on job accounting for your job. This allows you to obtain information on the elapsed time and memory and IO usage of your program, plus other data. You should place the ja command in your batch script after your PBS specifications and before your executable commands.
You must pair your initial ja command with another ja command at the end of your job. We recommend you use the command ja -chlst for this second command. The option -t to this second ja command turns off job accounting and writes your accounting data to stdout. The other options to the second ja command determine what output you will receive from ja. We recommend the -chls options because we think they will provide detailed but useful information about your job's processes. However, you can look at the man page for ja to see what reporting options you want to use.
There is no overhead to using ja. We strongly recommend you use ja when you want to understand the resource usage of your jobs. You can use this information when you submit future jobs. The output from ja can also be used for debugging and performance improvement purposes.
Memory files
Blacklight's operating system creates a file system out of its blade memory. Thus, your program can perform IO to blade memory rather than to disk. Memory IO is several orders of magnitude faster than disk IO. However, each blacklight job can only perform memory IO to the blades associated with that job. A job cannot write to the memory of blades assigned to other jobs.
The environment variable $SCRATCH_RAMDISK is set to point to the memory associated with each job. Unlike $SCRATCH, this variable is given a new value for each job. Otherwise, this variable can be treated like $SCRATCH. From within your job, you can cd to it, you can copy files to and from it, and you can use it to open files.
Memory IO is faster than disk IO, but it does have disadvantages. Each job's memory filespace is cleared whenever the job terminates, whether normally or abnormally. Thus, if you are using memory IO you must copy your memory files back from $SCRATCH_RAMDISK before your job ends or the files are lost. If your job terminates abnormally your files will be lost. Moreover, memory IO is limited in size relative to disk IO. Each job can only use the memory associated with that job. Furthermore, memory IO is limited to the memory available after memory is allocated for your program. Moreover, the largest size a single memory file can be is 256 Gbytes. Therefore, the use of memory files is best suited to IO-intensive jobs that perform IO to lots of small files.
IO optimization
File striping
If your program reads or writes large files you should use $SCRATCH. Your $HOME space is limited. In addition, the $SCRATCH file space is implemented using the Lustre parallel file system. A program that uses $SCRATCH can perform parallel IO and thus can significantly improve its performance. File striping can be used to tune your parallel IO performance and is particularly effective for files that are 1 Gbyte or larger.
A Lustre file system is created from an underlying set of file systems called Object Storage Targets (OSTs). Your program can read from and write to multiple OSTs concurrently. This is how you can use Lustre as a parallel file system.
A striped file is one that is spread across multiple OSTs. Thus, striping a file is one way for you to be able to use multiple OSTs concurrently. However, striping is not suitable for all files. Whether it is appropriate for a file depends on the IO structure of your program.
For example, if each of your cores writes to its own file you should not stripe these files. If each file is placed on its own OST then as each core writes to its own file you will achieve a concurrent use of the OSTs because of the IO structure of your program. File striping in this case could actually lead to an IO performance degradation because of the contention between the cores as they perform IO to the pieces of their files spread across the OSTs.
An application ideally suited to file striping would be one in which there is a large volume of IO but a single core performs all the IO. In this situation you will need to use striping to be able to use multiple OSTs concurrently.
However, there are other disadvantages besides possible IO contention to striping and these must be considered when making your striping decisions. Many interactive file commands such as ls -l or unlink will take longer for striped files.
You use the lfs setstripe command to set the striping parameters for a file. You have to set the striping parameters for a file before you create it.
The format of the lfs setstripe command is
lfs setstripe filename -c stripe-count
A value of -1 for the stripe count means the file should be spread across all the available OSTs.
For example, the command
lfs setstripe bigfile.out -c -1
sets the stripe count for bigfile.out to be all available OSTs.
The command
lfs setstripe manyfiles.out -c 1
has a stripe count of 1. Each file will be placed on its own OST. This is suitable for the situation where each core writes its own file and you do not want to stripe these files.
You can also specify a directory instead of a filename in the lfs setstripe command. The result will be that each file created in that directory will have the indicated striping. You can override this striping by issuing an lfs setstripe command for individual files within that directory.
The kind of striping that is best for your files is very application dependent. Your application will probably fall between the two extreme cases discussed above. You will therefore need to experiment with several approaches to see which is best for your application. A value of -1 for stripe count will probably give you the best performance if you are going to use file striping, but you should try several values. The maximum value you can give for stripe count on blacklight is currently 8.
Third-party software
Third-party routines can often perform better than routines you code yourself. You should investigate whether there is a third-party routine available to replace any of the routines you have written yourself.
For examples, we recommend the FFTW libraryfor FFTs. For linear algebra routines we recommend the MKL library.
Performance monitoring tools
Performance improvement assistance: the Memory Advantage Program
Blacklight is a very large hardware-coherent shared memory machine. Blacklight is thus suitable for a range of memory-intensive computations that cannot readily be deployed on a distributed-memory machine.
PSC has established the Memory Advantage Program (MAP) to enable users to take advantage of blacklight's unique capabilities. MAP includes consulting assistance from PSC, special queue handling if necessary and service unit discounts.
Software Packages
The Module Command
Before you can run many software packages, you must define paths and other environment variables. To use a different version of a package, these definitions often have to be modified. The module command makes this process easier. For use of the module command, including its use in batch jobs, see the module documentation.
Blacklight and XSEDE
Blacklight is on XSEDE. Thus, you have access to XSEDE methods of connecting to blacklight and to XSEDE methods of transferring files between your local machine and blacklight and arc. For information on using XSEDE see the general online documentation for XSEDE.
Stay Informed
As a user of blacklight, it is imperative that you stay informed of changes to the machine's environment. Refer to this document frequently.
You will also periodically receive email from PSC with information about blacklight. In order to insure that you receive this email, you should make sure your email forwarding is set properly by following the instructions for setting your email forwarding.
PSC requests that a copy of any publication (preprint or reprint) resulting from research done on blacklight be sent to the PSC Allocations Coordinator. In addition, if your research was funded by the NSF you should log your publications at the XSEDE Portal. We also request that you include an acknowledgement of PSC in your publication.
Reporting a Problem
You have several options for reporting problems on blacklight.
- You can call the User Services Hotline at 412-268-6350 from 9:00 a.m. until 5:00 p.m., Eastern time, Monday through Friday.


text:
2014 COMBAT Portent Senior League Baseball Bat: PORSL112
With Free Shipping
Both Ways
Product Features
What do you think about the 2014 COMBAT Portent Senior League Baseball Bat: PORSL112 ?
|or Cancel
19 Product Reviews
Questions and Answers
Got a question about the 2014 COMBAT Portent Senior League Baseball Bat: PORSL112? Ask our team of experts, and they will respond within 24 hours.Ask A Question
Is this a youth big barrel bat?
Yes, the 2014 COMBAT Portent Senior League Baseball Bat (PORSL112) will be a Youth Big Barrel Bat. Senior League and Youth Big Barrel are the same thing. Sorry for the inconvenience and confusion!
Is there any harm in using a 2014 bat in the 2015 season? Also, are there any other drop 12 bats available in the 2 5/8 barrel? Either 2014 or 2015 models.
There is no harm in using a 2014 model for the 2015 season. The 2014 COMBAT Portent Senior League Baseball Bat (PORSL112) features the USSSA BPF 1.15 stamp on the taper of the bat. Another -12 drop with a 2 5/8" is the 2015 Louisville Slugger Catalyst Senior League Baseball (SLCT152).
Does this bat come in a 2 3/4" size?
Yes, the 2014 COMBAT Portent XL Senior League Baseball Bat (PORSL210) is the 2 3/4" model.
Product Description
New for 2014! For those players that started with a small barrel COMBAT, you will not be disappointed in the all new B4 youth big barrel bat. For those of you that have never experienced the COMBAT difference, here is what you can expect: a swing weight that is right on point, a sweet spot that's almost not even fair, and zero sting/vibration. And that's just the tip of the iceberg. It all starts with COMBAT's Localized Stiffening Technology that allows their engineers to lower barrel wall stiffness on both sides of the normal sweet spot, resulting in an enlarged Double Sweetspot area where absolute maximum performance can be achieved! The 100% composite, one-piece Portent utilizes COMBAT's Blended Fiber Technology that blends several different fibers in varying proportions at carefully controlled angles to create a composite material that results in both resilient durability and increased trampoline effect. Separation Barrier Technology also allows this bat to be HOT out of the wrapper! The B4 Portent draws much of its durability from its Seamless Construction; meaning that there are not cuts or splices - just continuous fiber all the way around. Feel, Durability, and Sweet Spot. That's what makes COMBAT the preferred composite bat. COMBAT: For Players, By Players! The B4 Portent features the new USSSA 1.15 BPF Stamp. It is backed by a Full Twelve (12) Month Manufacturer's Warranty. Free Shipping!
About COMBAT
Holy Cow, that's great service. I look forward to using the bat in my over thirty league game this coming sunday. Great job.
Related Products
Shop With Confidence
You just checked out the 2014 COMBAT Portent Senior League Baseball Bat: PORSL112. What would you like to do next?Order Now or Browse More Bats


text:
Black Meets White
mouse over photo to zoom click for larger image
One-Shoulder Dot Dress
Price Reflects Savings
So flirty in a gentle breeze—a soft layer of chiffon skims over silky charmeuse with the same print. Includes a removable, skinny grosgrain belt. Shell and lining: 100% Polyester. Machine wash or dry clean. Imported.
- Hourglass fit.
- Pleats at the shoulder and waist. Boned bodice. Hidden side zip with hook-and-eye closure.
- Fully lined in silky charmeuse.
- Length: 38.5".
We're so sorry: this item sold out sooner than expected. For an equally stylish substitute, please call our Style Experts at 1.877.948.2525. Or click on Live Help below for immediate assistance.


text:
Aloha Surf Clinics With Nancy Emerson
Join us on Facebook
Follow us on Twitter
Reservation Info
Phone:(808) 294-5544
Fax:(808) 356-0700
Other:+61 413-380-933
Nancy Emerson, Surfing Legend and Pioneer surfing professional is the originator of "LEARN TO SURF IN ONE LESSON". NANCY EMERSON has created the best surfing experience world wide on Maui & Oahu. Aloha Surf Clinics lessons cover the total surfing experience. Nancy has a holistic approach she enjoys to use to help any one wanting to learn to surf, body board, or body surfing enjoy and understand how to read the ocean.
Along with her team of accredited instructors, Nancy will teach you the basics of form and technique, as well as water safety, ocean awareness and wave conditions in the one simple surfing lesson that has set past champions on the path to success. Maui & Oahu are the best spot on the earth to learn to surf. Come enjoy a Maui surfing clinic or lesson or request a surf guide for the more experienced surfer around the most beautiful islands Maui & Oahu. Your surfing experience with Nancy Emerson and instructors at the school will be EASY, FUN and SAFE!
Hours of Operation
Cards Accepted
Reservation: Yes


text:
LOS ANGELES (AP) -- Kobe Bryant and Pau Gasol are returning to the lineup for the Los Angeles Lakers.
Coach Mike D'Antoni says both will start against the Washington Wizards on Friday night at Staples Center.
Bryant has missed two games with a severely sprained left ankle. Gasol has been out since Feb. 5 with a right foot injury. They'll join Dwight Howard, Steve Nash and Metta World Peace in the lineup. When the big five have all started this season, the Lakers are 0-5.
The Lakers bring a seven-game home winning streak into the game against John Wall and the Wizards.
Copyright 2013 The Associated Press. All rights reserved. This material may not be published, broadcast, rewritten or redistributed.


text:
Sarah Marquart, CFCA project specialist, talks about Maria, a sponsored aging woman in Ecuador. She received dentures from the dental program funded by CFCA. Now that her smile has been restored, she is truly a happier woman.
Feb 3 2009
yo trabajo para cfca actulamente y queremos realizar una caminat a en nuestro subproyecto como puedo bajar estos videos y mostrarselos alas familias apadirnadas para utilizarlo como elemento de motivacion para ellos.
As a longtime sponsor of a child in the Guayquil project, I cannot tell you how happy I am to learn that CFCA is helping with medical and dental care. My sponsored child mentioned in her last letter that she might be going for her first checkup ever (she is 14 yrs old!), but I did not realize it was through CFCA and I was a little concerned. Now I am not only releaved, I am thankful! God Bless you and your project coordinators!


text:
Libraries are for Making...March 8 -14, 2015
Nancy Klein has not received any gifts yet
Give Nancy Klein a Gift
Join Teen Tech Week
Welcome toTeen Tech Week
Sign Upor Sign In
Connect teens to technology at your library this March with YALSA! #TTW15
By joining this site, you're eligible for these benefits & resources:
Started by Carla Avitabile in Sample Title. Last reply by Carla Avitabile on Friday.
1 Reply
Many of us in the public library sector are always looking for ways to partner with public schools. Teen Tech Week is a perfect time to make those connections. We all have a similar mission, and that…Continue
Started by YALSA in Sample Title. Last reply by Carla Avitabile on Friday.
2 Replies
Please share your ideas on program ideas for the 2015 theme.Continue
Started by YALSA in Uncategorized. Last reply by Janae Pitts-Murdock Dec 16.
32 Replies
Do you work at a public library? Use this space to find and share ideas for TTW programs that are hosted through public libraries!Continue
Started by Jaina Lewis in Sample Title Nov 25.
0 Replies
A new way of looking at the word FAIL is to define it as "first attempt…Continue
Added by Albany Public Library
0 Comments
Added by Gail Zachariah
0 Comments
Get Badge
© 2014 Created by YALSA.
Powered by
Badges |
Report an Issue |
Terms of Service
Please check your browser settings or contact your system administrator.


text:
Other Ingredients:
Proprietary non-GMO protein blend (rice protein, pea protein and soy [isolated soy protein and fermented soy]), fructose, Dutch cocoa, tri-calcium phosphate, potassium citrate, guar gum, magnesium oxide, psyllium, natural vanilla flavor, oat bran, microcrystalline cellulose, spirulina, vitamin C, vitamin E acetate, choline bitartrate, inositol, apple pectin, bee pollen, niacinamide, vitamin A palmitate, zinc oxide, manganese sulfate, ferrous fumarate, calcium pantothenate, lecithin, lemon bioflavonoids, papaya, bromelain, chlorophyll, pyridoxine HCl, riboflavin, thiamine HCl, vitamin B12, vitamin D, folic acid, biotin, potassium iodide, chromium chloride, sodium selenite, sodium molybdate.
Food Sensitivity:
Vegetarian, non-GMO and yeast free.
Keep tightly closed in a cool, dry place.Keep out of reach of children.Not to be used as the sole source of dietary calories.


text:
Women's adidas Supernova Racer BraItem #AIL1692
Now:$35.95 - $39.99
VIP:$32.36 - $35.99 Get This Price
Loading product details ...
Reviews (5)
Need Help?
Product Description:
Women's ADIDAS® SUPERNOVA™ RACER BRA :: You deserve a sports bra that's as unique and diverse as you are. The criss-cross racer back and deep V front of the adidas® supernova™ Racer Bra make this a great choice for any run wardrobe. You'll feel warm without getting too hot thanks to perfectly placed ventilation plus sweat-wicking material. Light, cool and incredibly comfortable!
Features & Benefits:
Best For:
REVIEW SNAPSHOT®by PowerReviews
Best Uses
Reviewed by 5 customers
Sort by
Best Uses
Comments about Womens adidas Supernova Racer Sports Bra:
this bra is not stretchy like most sports bras. it is a little difficult to get on but feels great and provides good compression. I was looking for support to prevent movement and this does its job.
Best Uses
Comments about Womens adidas Supernova Racer Sports Bra:
Love the fit. It's supportive without squishing me flat, and it's very comfortable.
Best Uses
Comments about Womens adidas Supernova Racer Sports Bra:
This is a great running bra, it's got support, a slight extra pad (very thin, but covers), and it looks great and doesn't chafe. I'd buy another one in a heartbeat.
Best Uses
Comments about Womens adidas Supernova Racer Sports Bra:
I use this bra mostly to run but, also to weight train and on the eliptical. It is extremely flattering. The cups are a little too big for a medium but, the band is perfect. Sad they don't make it in Pink and black anymore.
Comments about Womens adidas Supernova Racer Sports Bra:
Was looking for a good running bra. This did not fit the bill. The sizing ran differently from other manufacturers and had to return.
Your Help Is Here
Shop with confidence knowing your team of Fit Experts is waiting to answer all your questions. You can always find your perfect product and your perfect fit. Your Road Runner Sports Fit Experts will make sure you get exactly what you want and have tons of fun in the process! Give us a call, chat, or email!
Call 800-743-3206 to
Send us an email and count on a Fit Expert to reply to you as quickly as possible
You May Also Like
More Related Products
Women's adidas® SUPERNOVA™ SEQUENCE SHORT :: You're free to run at your own pace with the lightweight women's adidas® Supernova™ ...
Women's ADIDAS® ENERGY FULL SUPPORT BRA :: If you're a D or a DD, you don't want to miss out ...
Women's MOVING COMFORT® JUSTRIGHT RACER BRA :: Keep your cool during even the hottest of sweat sessions with the most ...


text:
Processing Time - Orders typically ship in 2-5 business days. Transit via UPS Ground can take up to 6 business days. Braided and custom-made products may take 2 to 3 weeks for delivery.
Free Shipping to Contiguous United States - We do not ship to Alaska, Hawaii or the Caribbean Islands at this time.
Oversized items - Orders that ship freight may take up to 2 weeks from shipment date to be delivered. You will be contacted by the trucking company to schedule a delivery appointment.
Shipping to Canada - Orders shipping to Canada will be subject to a 15% shipping surcharge fee. The shipping surcharge does not cover duties, customs, taxes or brokerage fees which may be charged to you by the shipping courier at time of delivery. At this time we are not shipping to Yukon and the Northwest Territories in Canada.
|Pin It
Jaipur Rugs Provenance - Wool PW01 African Violet Rug
Item #: 229PW01J-P Add to Wishlist
Description: Everything old is new again. Tapping into one of today's hottest trends, the Provenance series of collection takes the concept of patchwork rugs to beautiful new heights. High quality traditional designs are given new, transitional life through a unique process that begins by cutting apart various existing designs. Color is removed from the fibers and then overdyed in vivid, fashion-forward shades. Pieces from various rugs are then artfully reassembled, creating one-of-a-kind works of art for the floor.
- Collection: Provenance - Wool
- Brand: Jaipur Rugs
- Material: 100% Handspun Wool
- Weave: Hand Knotted
- Styles:
- Origin: India
- Available in the following sizes: 2' x 3' and 8' x 10'
- Clean & Care: Vacuum regularly as new wool rugs can shed yarn fibers for up to three months. Avoid direct and continuous exposure to sunlight. Do not pull loose ends, clip them with scissors to remove. Clean liquid spills immediately by blotting with cloth or sponge. For hard to remove stains, professional rug cleaning is recommended.
Price: $2,016
Jaipur Rugs
"Beauty of whatever kind, in its supreme development, invariably excites the sensitive soul to tears. -- Edgar Allan Poe
Memorable, beautiful, comfortable and distinctive . . . at Jaipur Rugs we believe a rug should provide all this and more.
Our rugs are created using ancient techniques of master tribal weavers from the mountains of Northern India. Through this rich tradition, we produce strikingly beautiful hand-knotted, hand-tufted and flat woven designs for our discerning clientele.
Jaipur rugs range from contemporary and transitional to lush traditionals with rich colors and stunning appearance. Best of all, Jaipur Rugs brings this excellence, beauty, and distinction --affordably-- into contemporary American homes
Fill out the information below.


text:
Book your room at Executive Hotel Le Soleil
Executive Suite, 1 Bedroom (Two Twin Beds)Max Occupancy: 4 guests (up to 3 children, 3 infants)
Executive Suite, 1 Bedroom (One Queen Bed)Max Occupancy: 4 guests (up to 3 children, 3 infants)
Executive Suite, 1 Bedroom (One King Bed)Max Occupancy: 4 guests (up to 3 children, 2 infants)
Suite, 1 Bedroom, Corner (One Queen Bed)Max Occupancy: 4 guests (up to 3 children, 3 infants)
Suite, 1 Bedroom, Corner (One King Bed)Max Occupancy: 4 guests (up to 3 children, 1 infant)
Senior Penthouse, 1 King BedMax Occupancy: 2 guests
Downtown Vancouver hotel near Canada Place
With a stay at Executive Hotel Le Soleil, you'll be centrally located in Vancouver, steps from Bill Reid Gallery of Northwest Coast Art and minutes from Christ Church Cathedral.
This 4-star hotel is close to Granville Island Public Market and Stanley Park.
Make yourself at home in one of the 119 air-conditioned rooms featuring minibars and flat-screen televisions. Your pillowtop bed comes with down comforters and Egyptian cotton sheets, and all rooms are furnished with sofa beds. Complimentary wired and wireless Internet access keeps you connected, and digital programming provides entertainment. Bathrooms have designer toiletries and complimentary toiletries.
Make use of convenient amenities such as wireless Internet access (surcharge), babysitting/childcare, and ski storage. Additional features include wedding services, a fireplace in the lobby, and discounted use of a nearby fitness facility.
Grab a bite to eat at the hotel's restaurant, which features a bar, or stay in and take advantage of room service (during limited hours). Quench your thirst with your favorite drink at a bar/lounge. Breakfast is available for a fee.
Featured amenities include high-speed (wired) Internet access (surcharge), a business center, and limo/town car service.Information missing or incorrect? Tell us!
At your hotel
At your hotel
This modern all-suite downtown Vancouver hotel is 1 block from Pacific Centre Mall, 2 blocks from upscale shopping along Robson Street, and 3 blocks from the Vancouver Convention and Exhibition Centre, and cruise ship terminal at Canada Place.
Guests at Executive Hotel Le Soleil may use the fitness and recreation facilities at a health and wellness center located next door to the hotel (surcharge). Business and secretarial support services are available, including faxing, photocopying, and printing. Guests can check their e-mail on the hotel computer at no charge. Other amenities include a fireplace in the lobby, wedding services, and banquet facilities.
This hotel is smoke free (fines apply for violations).
- Complimentary newspapers in lobby
- Coffee shop or café
- Breakfast available (surcharge)
- Bar/lounge
- Restaurant
- Limo or Town Car service available
- Multilingual staff
- Safe-deposit box at front desk
- 24-hour front desk
- Porter/bellhop
- Express check-out
- Total number of rooms - 119
- Babysitting or childcare
- Dry cleaning/laundry service
- Ski storage
- Valet parking (surcharge)
- Elevator/lift
- Fireplace in lobby
- Wireless Internet access - surcharge
- Wired (high-speed) Internet access - surcharge
- Computer station
- One meeting room
- Laundry facilities
- Business center
- Number of floors - 16
- Luggage storage
- Smoke-free property
- Use of nearby fitness center (discount)
- Wedding services
Stay connected
Available in all rooms: Free Wi-Fi , Free wired high-speed Internet
Available in some public areas: Wireless Internet access - surcharge , Wired (high-speed) Internet access - surcharge
Park your car
Valet parking (surcharge)
In your room
- Free Wi-Fi
- Free wired high-speed Internet
- Down comforter
- Egyptian-cotton sheets
- Pillowtop mattress
- Pay movies
- Premium bedding
- Hypo-allergenic bedding available
- Room service (limited hours)
- Separate sitting area
- Climate control
- Complimentary weekday newspaper
- Air conditioning
- Phone
- Minibar
- Coffee/tea maker
- Daily housekeeping
- Designer toiletries
- Bathrobes
- Complimentary toiletries
- Hair dryer
- Slippers
- Iron/ironing board
- Desk
- Sofa bed
- Cribs/infant beds available
- In-room safe (laptop compatible)
- Digital TV service
- Flat-panel TV
Eat and drink
The Copper Chimney Restaurant and Bar - Offering contemporary Indian inspired West Coast cuisine for breakfast, lunch, and dinner. The executive chef and his talented team will prepare dishes combining the freshest of seasonal ingredients with exotic spices. The open-styled kitchen with its copper chimney and oven takes center stage, complimented by high vaulted ceilings, spiral staircase and balcony dining. Private dining tables, an exquisite cocktail bar and in-house DJ spun tunes on Fridays and Saturdays, create a lively night scene in the heart of Vancouver.
Guests are welcome to use, for a surcharge, a 2,787 square meter (30,000 square foot) fitness and aquatic facility located next to the Executive Hotel Le Soleil. Full of natural light it features open spaces with state-of-the-art equipment, a 25-metre ozone-purified swimming pool, steam room and ozone-purified whirlpool, excellent group classes and certified personal trainers.
The hotel is 4 blocks from the beginning of the Vancouver seawall, a 12-mile paved pathway on the waterfront suitable for running, in-line skating, cycling, and strolling.
The recreational activities listed below are available either on site or near the hotel; fees may apply.
- Golfing nearby
- Downhill skiing nearby
- Bicycle rentals nearby
- Hiking/biking trails nearby
To make requests for specific accessibility needs, complete the "Special or Accessibility Requests" section of "Trip Preferences" when you book your hotel.
- Accessible path of travel
- Accessible bathroom
- Accessible parking
- In-room accessibility
- Accessibility equipment for the deaf
- Braille or raised signage
The fine print for Executive Hotel Le Soleil
When you arrive
Check-in time starts at 4 PM
Special check-in instructions:
Guests booking the penthouse must pay a CAD500 deposit at check-in. Deposit is subject to change.
When you leave
Check-out time is noon
How you can pay
Your pets
Pets allowed
You need to know
Extra-person charges may apply and vary depending on hotel policy.
Government-issued photo identification and a credit card are required at check-in for incidental charges.
Special requests are subject to availability upon check-in and may incur additional charges. Special requests cannot be guaranteed.
No rollaway/extra beds available
- Valet parking fee: CAD 29 per night (in/out privileges)
- Pet fee: CAD 25 per room, per night (maximum CAD 200.00 per stay)
We should mention
Children 18 years old and younger stay free when occupying the parent or guardian's room, using existing bedding. Only registered guests are allowed in the guestrooms.


text:
1950s sportswear tee
by LEVI'S VINTAGE CLOTHING
Levi's Vintage Clothing is a meticulously crafted collection dedicated to capturing the spirit of American workwear. The collection is a comprehensive assortment of authentic Levi's reproductions pulling from archives dating back the 1870's. Each garment faithfully reproduces the fits, fabrics and characteristics associated with a specific period in time. Back To Product View All From This Designer
Men's Size Chart
Product Details
Soft, slub jersey t-shirt with a vintage hand and single front pocket.
• rib crewneck
• special treatment given to dying process to create natural nuances of shade
• 100% cotton
• imported


text:
- NVIDIA Tegra 3 1.2GHz Quad Core CPU
- 1GB Memory
- 32GB Internal Storage
- 7" 1280x800 Back-lit IPS Display w/ 1.2MP Front-Facing Camera
- WiFi N
- 4325mAh Battery
- Android 4.1 (Jelly Bean)
Review and Hands-On courtesy of Engadget
We've heard your feedback and are continuing to build a better Slickdeals. Click here to check out the updated site.
Frontpage Deal
BuyDig via Ebay has 32GB Asus Nexus 7 7" Android 4.1 Tablet (ASUS-1B32) for $230 with free shipping. Thanks yuugotserved
Review and Hands-On courtesy of Engadget
Original Post
yuugotserved ( ) Nov. 17, 2012 at 8:01am
BuyDig via Ebay
32GB ASUS Nexus 7 7" Tablet [ebay.com]
$230 with free shipping
Wiki Community Board
This can and should be used by you to post updated deal information.
Be sure to read this because it may contain answers to your questions!
Use Discover with your PayPal 5% and sign up for the weekend promotion of 2%
Use the referral below. Tou'll also get $4.6 ebay cash out of this!
Good luck!
Out of Stock!!!
View Forum Thread
84 Comments
RagingWookiee ( ) Nov. 17, 2012 at 8:02am
Sultanofdeals ( ) Nov. 17, 2012 at 8:04am
Get 1 percent back with that e b a t e thing
Erechtheus ( ) Nov. 17, 2012 at 8:09am
Beware that instead of the 32 GB I bought, I got a 16 GB model when they did this last week and BuyDig hasn't been wonderful about correcting their error as of yet.
ETA: I actually thought through one of the ways to resolve my issue the CSR I worked with proposed and bought on this deal, selecting overnight shipping, and will take them up on their offer to pay for both the overnight upgrade and the shipping on the return. This is handy after all.
davidgrote ( ) Nov. 17, 2012 at 8:12am
thanks. repped and bought. free shipping plus no tax and no hassle with pricematching makes this pretty easy.
spacemanvt ( ) Nov. 17, 2012 at 8:13am
hmmm wondering if this will drop more in the near future.
JaguarSavages ( ) Nov. 17, 2012 at 8:17am
In for 1, thanks
arunrathna ( ) Nov. 17, 2012 at 8:19am
thank you. ordered one.
blackbeard2 ( ) Nov. 17, 2012 at 8:24am
binaryvisions ( ) Nov. 17, 2012 at 8:26am
Popular Deals (By Votes)
Use Coupon Code: HOLIDAZE
at American Eagle Outfitters


text:
Moderator: Mod Squad
Warbear wrote:Gravity only applies to Persian royalty.
Lily wrote:Warbear wrote:Gravity only applies to Persian royalty.
then that means...I AM ROYALTY.
icha_icha_paradise wrote:Ha! Gravity is only a theory! You can't prove it!
BeeAre wrote:new discussion
do retarded people have retarded souls
Return to Central Compton Botanical Gardens
Users browsing this forum: Mir@k, Senel, Yahoo [Bot] and 4 guests


text:
Matilda - The Musical tickets
September 27, 2014 at 8:00 pm Shubert Theatre - NY :: New York, New York
Currently we do not have any Matilda - The Musical tickets available.
Sometimes an event may have been announced, but tickets may not yet be available to anyone.
Check back soon for updates.


text:
I would like to see special offers for my region
This is the first time you log in with your social account. In order to verify your person, please login with your Lufthansa credentials. This procedure is only needed once.
Not registered yet? New registration
Further conditions apply.Kindly be advised that the fares may differ slightly due to currency fluctuations and/or government tax regulations.
Minimum stay: 4 Days
Maximum stay: 12 Months
Rebooking before departure: EUR 120
Cancellation before/after departure: EUR 120
Stopovers: 1 free and 1 @ EUR 75


text:
required Required
We're sorry. There are no Papa John's stores within 50 miles of the address you entered.
Click this link to enter a city and state.
Click this link to enter a city and province.
Click this link to enter a ZIP code.
Click this link to enter a postal code.
You are currently on the full site.
Go to the Tablet Experience.
Go to the Mobile Site.
Go to the WAP site.
Online Ordering Customer Service 877-547-PAPA (7272). Consumer Services Team, Attention: Privacy Officer, P.O. Box 99900, Louisville, Kentucky 40269-0900, firstname.lastname@example.com.
Offer and Prices will vary in Alaska, Hawaii and Canada. Offer good for a limited time at participating Papa John's restaurants. No double toppings or extra cheese. Not valid with any other coupons or discounts. Limited delivery area. Delivery fee may apply. Customer responsible for all applicable taxes. Papa John's pizzas are hand-made to order in your local restaurant; your pizza may vary in appearance from the pizza represented here.
No purchase necessary. MSG and Data rates may apply. Visit www.papajohns.com/text for terms and conditions. Text HELP for help. Six (6) message per month. Text CANCEL to cancel.
* Up to 5 toppings or specialty.
© 2014 Papa John's International, Inc. All Rights Reserved.
06-184.108.40.206 126.96.36.199
Invalid credentials. Please enter a valid email address and password.


text:
Featured Health Policy Brief
What is the Link Between Having Health Insurance and Enjoying Better Health and Finances?
The Oregon Health Insurance Experiment showed that having health insurance profoundly affects an individual's health and well-being. In the study, people with health insurance reported better health, both mental and physical, and better access to care than those without insurance.