Memory efficient structure for membership checking without false positive
For web crawler scale why not use a distributed database like Apache Cassandra? Lookups on indexes are efficient and no false positives.