Which Big Data technology stack is most suitable for processing tweets, extracting/expanding URLs and pushing (only) new links into 3rd party system?
I'd suggest Apache Kafka as message store and any stream processing solution of your choice like Apache Camel or Twitter Storm