A Web Scraping Methodology for Bypassing Twitter API Restrictions
A. Hernandez-Suarez G. Sanchez-Perez K. Toscano-Medina V. Martinez-Hernandez V. Sanchez H. Perez-Meana




RESEARCH ARTICLE






^1   Instituto Politecnico Nacional, Graduate School ESIME Culhuacan; hmperezm@ipn.mx

^2   University of Warwick, Department of Computer Science,  CV4 7AL, UK ; v.f.sanchez-silva@warwick.ac.uk

Instituto Politecnico Nacional, Graduate School ESIME Culhuacan, San Francisco Culhuacan, CTM V, 04430 CDMX, Mexico; hmperezm@ipn.mx.


Retrieving information from social networks is the first and primordial step many data analysis fields such as Natural Language Processing, Sentiment Analysis and Machine Learning. Important data science tasks relay on historical data gathering for further predictive results. Most of the recent works use Twitter API, a public platform for collecting public streams of information, which allows querying chronological tweets for no more than three weeks old. In this paper, we present a new methodology for collecting historical tweets within any date range using web scraping techniques bypassing for Twitter API restrictions.




A Web Scraping Methodology for Bypassing Twitter API Restrictions
     A. Hernandez-Suarez ^1 ,G. Sanchez-Perez^1, K. Toscano-Medina^1, V. Martinez-Hernandez^1, V. Sanchez^2 and H. Perez-Meana ^1
    December 30, 2023
=================================================================================================================================







§ INTRODUCTION

Gathering proper information for training and testing data science algorithms is a primordial task that must be accomplished in order to obtain useful results. Many fields related to Natural Language Processing, Sentiment Analysis and Machine Learning use Online Social Network platforms to retrieve user information and transform it into machine-readable inputs, which are used by various algorithms to obtain predictive outputs like flu spreading detection <cit.>, forecasting future marketing outcomes <cit.> and predicting political elections <cit.>. Twitter is becoming the preferred  social network for data collection, in this network users can post short messages, also referred to as tweets, in a real-time manner; which has facilitated topic clustering research like rumour spreading analysis <cit.>, human mobility sensing <cit.>, spam & botnet detection <cit.> and disaster response <cit.>. The embedded information in a tweet may include images, geographic locations, url references and videos. Because of its usability and widespread, Twitter engines may dispatch approximately 1-billion of user-generated content per month, which is re-distributed over several countries around the world. 

Data researchers have started using Twitter for scientific approaches due to the facility for querying and collecting large volumes of data in a short time. As presented in early works like <cit.>, collecting tweets can be done by scraping streams of public available data by using Twitter "Gardenhouse" API <cit.>, which is an endpoint designed for reading and writing tweets. 
After registering an application, the platform then returns a set of tokens granting  access to the streaming API. Well known works <cit.> <cit.> have used keywords for querying the Twitter API endpoint and retrieving formatted objects containing tweets with additional fields <cit.>; e.g., geographic coordinates, chronological information , retweet-metadata, favorite counts and language information. Although tweets are easily collected from the previously mentioned API, a big limitation occurs when stream rates exceed the number of retrieved tweets given n number of queries. Twitter addresses this issue by limiting the flows of data to 15-minute interval by user application. In addition, another limitation is the fact that  
historical tweets are only available for a maximum range of three weeks by registering an enterprise API <cit.> or by applying to a full-search access <cit.> where potential clients are subjected to social and financial evaluations.   
Unfortunately, little is known about how to overcome this problem <cit.>. Some sites like Gnip<cit.> and Sifter<cit.> offer a full archive of old tweets by registering and paying for volumes of tweets, sometimes at expensive prices.  In this paper, we propose a web scraping methodology for crawling <cit.> and parsing tweets bypassing Twitter API restrictions taking advantage of public search endpoints , such that,  given a query with optional parameters and set of HTTP headers we can request an advanced search going deeper in collecting data.




§ A WEB SCRAPING SOLUTION


Web scraping techniques are used to extract information from websites in an automatic way <cit.> by parsing hypertext tags and retrieving plain text information embedded onto them. We propose a new methodology for scraping Twitter Search endpoint and customizing queries fields in order to extend searching capabilities. At a glance, web scrapping seems an easy task but analyzing HTTP requests and responses <cit.> can be really complex. By using Scrapy, an open source and collaborative framework for extracting data from websites <cit.> written in Python, we enhance the power of scraping engines to obtain an unlimited volume of tweets bypassing date ranges limitations. Our proposal is graphically depicted in Figure <ref>.

  

The work flow is described next. Given any valid set of HTTP-HEADERS, an array of words (also known as querying  terms) , an array of dates (date range) and optional parameters, a query request is sent to the Twitter Search endpoint, composing then an URL(<ref>)


    https://twitter.com/search?f=tweets&vertical=default&q=
    words+array of dates+
    parameters


A rendered HTTP response is then processed by a web spider and the HTML payload is redirected to a download layer. Finally,  unprocessed data containing tweets is fed into the  Scrapy engine in order to strip hypertext tags by  objects known as tag-selectors; each tweet is treated as an independent structure composed of plain text, date of creation and geographic information (if exists). When scraped, responses are retrieved with a maximum-position  class attribute from an <div> HTML tag; this information is a pagination identifier from the last appended tweets. The reason for having this attribute is because if we want to scrap deeper tweets, the  maximum-position information must be fed into a second search endpoint, as shown in URL (<ref>)


    https://twitter.com/i/search/timeline?f=tweets&vertical=default&q=words
    +
    array of dates+
    parameters&src=typd  
    &min_position=maximum-position


Approximately, 20 tweets can be collected in a single request, but those sent to the second search endpoint can scrap k rounds of  asynchronous responses containing blocks of tweets. Unlike the first search endpoint, the second one is designed for users scrolling the main Twitter search feed and appending older tweets retrieved in JSON (JavaScript Object Notation) format, which are  internally processed by Twitter scripts. Collected tweets by Scrapy crawlers are processed through  a pipeline configured to store each one as an item in a relational or non-relational database client. The arguments for an advanced query based in <cit.> are listed in Table <ref>, the code for running the scraper with parameters on command line is depicted in Listing <ref>:


	
	[language=bash,caption=Bash version,label=bash,xleftmargin=-1.0cm]
	#!/bin/bash
	scrapy crawl twitter -a [K-ROUNDS ] -a [ALL-WORDS] 
	-a [DATE-RANGE] -a [PARAMETERS]
	
	


Our proposed methodology is described in Algorithm <ref>

§ DEPLOYING AND DEAMONIZING THE SCRAPER


In most applications, it is important to consider  usability <cit.>; because of the nature of Python applications, scripts are run via  command line  with required arguments and the binary code is then launched to perform programmed operations.  In this work, we build a Graphical User Interface (GUI) for testing our scraping approach. Specifically we develop a web service with Django<cit.>, a framework for developing web applications using Python as base language,  
which includes a model-view-controller design for quickly project escalation. Scrapy spiders cannot start instances for web crawling ; this is  because each task is triggered as an independent process and due to Django security restrictions (regarding  operating system privileges) all non-root processes are denied. To solve this issue, we bind the Django GUI to scrapyd<cit.>, a service that damenonize crawling tasks. Similarly to unix-like cron (job scheduler) tasks, Scrapy spiders can be scheduled to crawl targeted websites by calling scrapyd background instances using curl commands as shown in Listing <ref>.:

	[language=bash,caption=bash command for scheduling scrapyd job,label=bash2,xleftmargin=-1.0cm]
	#!/bin/bash
	curl http://localhost:6800/schedule.json -d project=twitter
	_spider -d spider=twitter -d [PARAMETERS]
	

Django view layer contains handlers for HTTP requests and responses from user activity performed on the GUI  template. When a request is received, a signal is sent to launch the scraper, thus  binding a scheduled job deployed by scrapyd to crawl  the previously mentioned Twitter search endpoints.  The work flow for daemonizing Scrapy is depicted in Figure <ref>.

  

Scrapyd daemon contains also an interface for monitoring stacked jobs launched from Django view layer. Figure <ref> depicts the scheduled jobs interface. 


  

Example of values for building a single query to scrap tweets are shown in Table <ref>, a log describing crawled urls and debugging information from 
Twitter Search Endpoints is depicted in Figure <ref> 

Responses from the Scrapy engine are retrieved as instances from a Scrapy class namedItems,  but they can be transformed into comma-separated values or plain text files. An example of plaint text tweets scraped from the Twitter search endpoints based on the query in Table <ref> are depicted in  Fig. <ref>.




§ EVALUATING THE SCRAPER PERFORMANCE


Evaluating scraping performance is useful to contrast Twitter Garden house API (Stream & Search) and our proposed methodology (Twitter Scrapy). The set of metrics for comparing both approaches are : the total amount of time for retrieving blocks of tweets, the volume of tweets retrieved for a queryqand the
 maximum number of historical tweets given a range of dates. 
The number of retrieved tweets by querying the stream without any filters is plot in  Fig <ref>a;  Fig <ref>b plots the number of retrieved tweets by filtering original statuses (no retweets and mentions) and finally <ref>c plots the number of retrieved tweets for a span of one week.

 

In Table <ref>. is compared the results of the proposed metrics of Twitter API and Twitter  Scrapy  methodology 



§ CONCLUSIONS


 Gathering information from Online Social Networks is a primordial step in many data science fields allowing researchers to work with different and more detailed datasets. Although an important proportion of the scientific community uses the Twitter streaming API for collecting data, a limitation occurs when queries exceed rating intervals and time ranges. 
In this article, we presented a new methodology for querying Twitter Search Endpoints bypassing their "gardenhouse" API restrictions, in order to retrieve large volumes of data generated over longer periods of time with faster results. 
We also showed that by using Python technologies such as Scrapy, Django and customized daemons, it is possible to develop and escalate a web interface for launching, controlling and retrieving information from web crawlers. Our development can be tested in an Amazon EC2 Web Services cloud environment. The application can be accessed publicly at <http://ec2-13-58-43-111.us-east-2.compute.amazonaws.com:8001/twitter_scrapper/> or by an ip address <http://13.58.43.111:8001/twitter_scrapper/>.




§ CONFLICT OF INTEREST DISCLOSURE 


The author(s) declare(s) that there is no conflict of interest regarding the publication of this paper

999

lampos
V. Lampos and N.Cristianini. Tracking the flu pandemic by monitoring the social web.  In Proceedings of the Second International Workshop on Cognitive Information Processing , Naregno, Elba Island, Italy, 14-16 June 2010; pp. 411-416.

asur
S. Asur and B. A. Huberman. Predicting the future with social media. In  Web Intelligence and Intelligent Agent Technology, Proceedings of the IEEE-WIC-ACM International Conference on  Web Intelligence, Toronto, Canada, 31 August - 3 September, 2010; Volume 1; pp. 492-499.

tuma
A. Tumasjan, T. O. Sprenger, P. G. Sandner and I. M. Welpe. Predicting elections with twitter: What 140 characters reveal about political sentiment. Proceedings of the Fourth International AAAI Conference on Weblogs and Social Media, Washington D.C. USA, 23-26 May 2010; Volume 10, pp. 178-185.

liakata
A. Zubiaga, M. Liakata, R. Procter, G. W. S. Hoi, and P Tolmie. Analysing how people orient to and spread rumours in social media by looking at conversational threads.  PloS one2016,  11, e0150989, doi:10.1371/journal.pone.0150989.

cuenca
J. Cuenca-Jara, F. Terroso-Saenz, M. Valdes-Vela and A. F. Skarmeta. Fuzzy Modelling for Human Dynamics Based on Online Social Networks.  Sensors2017,   11(9),1949, doi:10.3390/s17091949.

haustein
S. Haustein, T. D. Bowman, K. Holmberg, A. Tsou, C. R. Sugimoto and  V. Larivière. Tweets as impact indicators: Examining the implications of automated “bot” accounts on Twitter.  Journal of the Association for Information Science and Technology2016,  67(1), 232-238, doi:10.1002/asi.23456.

nandi

Z. Ashktorab, C. Brown, M. Nandi and A. Culotta. Tweedr: Mining twitter to inform disaster response. Proceedings of the Eleventh International Conference on Information Systems for Crisis Response and Management, PA, USA, 18-21 May 2014; pp. 354-358.


connor
B. O'Connor, R. Balasubramanyan, B. R. Routledge and N. A. Smith. From tweets to polls: Linking text sentiment to public opinion time series. Proceedings of the Fourth International AAAI Conference on Weblogs and Social Media, Washington D.C. USA, 23-26 May 2010; Volume 11, Number 122-129, pp. 1-2.


twitter
Twitter Streaming APIs. Available online: <https://dev.twitter.com/streaming/overview> (accessed on 4-09-2017)

musta
E. Mustafaraj and P. Metaxas. From obscurity to prominence in minutes: Political speech and real-time search. Proceedings of the Web Science Conference
: Extending the Frontiers of Society On-Line, Raleigh, North Carolina, USA, 26-27 April 2010.

conover
J. Ratkiewicz, M. Conover, M. R. Meiss, B. Gonçalves, A. Flammini and F. Menczer. Detecting and Tracking Political Abuse in Social Media. Proceedings of the Fifth International AAAI Conference on Weblogs and Social Media, Barcelona, Spain, 17-21 July 2011; Volume 11, pp. 297-304.

twitterd
Twitter API Overview. Available online: <https://dev.twitter.com/overview/api/tweets> (accessed on 4-09-2017)

search-api
30-day Search API.  Available online: <https://developer.twitter.com/en/docs/tweets/search/overview/30-day-search>full-api

Full-Archive Search API.  Available online: <https://developer.twitter.com/en/docs/tweets/search/overview/full-archive-search>mckenna
McKenna, B., Myers, M. D. and Newman, M. Social media in qualitative research: Challenges and recommendations.  Information and Organization2017, 27(2), 87-99.

gnip
Gnip. Available online: <https://gnip.com/> (accessed on 4-09-2017)

sifter
Sifter. Available online:  <https://sifter.texifter.com/> (accessed on 4-09-2017)

khalil
Khalil, S. and Fakir, M.  RCrawler: An R package for parallel web crawling and scraping.  SoftwareX 2017,  6, 98-106.

suganya
R. Suganya Devi, D. Manjula, and R. K. Siddharth. An Efficient Approach for Web Indexing of Big Data through Hyperlinks in Web Crawling.   The Scientific World Journal2015,   2015, 9, doi:10.1155/2015/739286.


singh
Singh, A. K. and Goyal, N. MalCrawler: A Crawler for Seeking and Crawling Malicious Websites.  Distributed Computing and Internet Technology2017, Springer International Publishing,  pp. 210-223.


scrapy
Scrapy. Available online: <https://scrapy.org/> (accessed on 4-09-2017)

jones
Jones, A. S., Horsburgh, J. S., Jackson-Smith, D., Ramírez, M., Flint, C. G. and  Caraballo, J. A web-based, interactive visualization tool for social environmental survey data.  Environmental Modelling & Software2016, 84, 412-426, doi:10.1016/j.envsoft.2016.07.013.

django
Django. Available online: < https://www.djangoproject.com/> (accessed on 4-09-2017)

scrapyd
Scrapyd. Available online: <https://scrapyd.readthedocs.io/en/stable/> (accessed on 4-09-2017)

twitters

Twitter Search Home. Available online: <https://twitter.com/search-home/> (accessed on 4-09-2017)