User-agent: *
Disallow: /pwdata
Disallow: /pwdata/scrapedxml
Disallow: /pwdata/cmpages
Disallow: /user

User-agent: Slurp
Crawl-delay: 5

User-agent: *
Disallow: /pwdata
Disallow: /pwdata/scrapedxml
Disallow: /pwdata/cmpages
Disallow: /user

User-agent: Slurp
Disallow: /pwdata
Disallow: /pwdata/scrapedxml
Disallow: /pwdata/cmpages
Disallow: /user
Crawl-delay: 1


Copy Disallows into the Slurp specific bit.
Copy Disallows into the Slurp specific bit.
