Coronavirus (COVID-19) Geo-tagged Tweets Dataset

Thanks, sadiksha!

My pleasure, Hanaa!

Thanks, Curran! I am glad that you found the dataset useful.

Hey Kim. I think you meant using twarc (https://github.com/DocNow/twarc). That was just a pseudo-code which I had mentioned in the abstract (I've now replaced it with an excerpt of the real code to avoid confusion). 

It does not matter how you are getting your JSON archived. Just make sure to add the following "if clause" in whatever way you're trying to pull the tweets. The "if clause" below will only be TRUE if the tweet contains an exact pin location.

data = json.loads(data)

if data["coordinates"]:

       longitude, latitude = data["coordinates"]["coordinates"]

Now you can store the longitude and latitude values as per your convenience. I hope this helps!

Hello Charu. Twitter's data sharing policy does not allow anyone to share tweet information other than tweet ID and/or user ID. The list of IDs should be hydrated to re-create a full fresh tweet dataset. For this purpose, you can use applications such as DocNow's Hydrator or QCRI's Tweets Downloader.

Hello Abhay. Yes, I have heard from a couple of people about getting the tweet IDs fixed on their machines. That is why I am also uploading the JSON for those experiencing this issue.

Can you confirm if the IDs are fixed even when opened using some text editors (Notepad or Sublime)? I think you're opening the CSV files with MS Excel. I've seen multiple posts regarding Excel, at Stack Exchange, truncating the digits after 15.

Thanks for getting back.

If you use the DocNow's hydrator app you can straightway import the downloaded CSV file for the hydrating purpose (while removing the sentiment column). However, QCRI's Tweets Downloader requires a TXT file (with a single tweet ID per line). So you'll have to play around the CSV file, to some extent, for the task to be done.

I have been reached by a very handful of people having an issue similar to this. Most of them were opening the CSV files with MS Excel to remove the sentiment column. The problem was not even there when the downloaded CSV was imported as a pandas data frame and the sentiment column was dropped, and the final data frame was exported as a CSV file ready to be hydrated.

Roger-that.

[updated on August 7, 2020] Hello Gayathri. You'll have to hydrate the tweet IDs provided in this dataset to get your work done. I'd suggest you use twarc for this purpose. I am guessing you'll only need the tweet and geo-location for your work.

#import libraries

from twarc import Twarc

import sqlite3

#create a database

connection = sqlite3.connect('database.db')

c = connection.cursor()

#creating a table

def table():

     try:

          c.execute("CREATE TABLE IF NOT EXISTS geo_map(tweet TEXT, longitude REAL, latitude REAL)")

          connection.commit()

     except Exception as e:

          print(str(e))

table()

#Initializing Twitter API keys

consumer_key=""

consumer_secret=""

access_token=""

access_token_secret=""

t = Twarc(consumer_key, consumer_secret, access_token, access_token_secret)

#hydrating the tweet IDs

for tweet in t.hydrate(open('ready_july5_july6.csv')):

     text = tweet["full_text"]

     longitude, latitude = tweet["coordinates"]["coordinates"]

     c.execute("INSERT INTO geo_map (tweet, longitude, latitude) VALUES (?, ?, ?)", (text, longitude, latitude))

     connection.commit()

Now you can simply make a connection to the above database-table to read its contents and plot the tweets using libraries such as Plotly. I hope this helps. Good luck!

Hello Mohit. Filtering geo-tagged tweets from the rest is quite straightforward if you use twarc for hydrating the tweet IDs. You'll have to add a condition to the "coordinates" Twitter object. 

for tweet in t.hydrate(open('/path/to/tweet/file.csv')):

     if tweet["coordinates"]:

          #now you can extract whichever information you want

          longitude, latitude = tweet["coordinates"]["coordinates"] #for getting geo-coordinates

You can go-through the code snippet replied to the comment thread just above this one for getting a headstart with storing the extracted information to a database.

Hello Lucas. The JSON files were initially present in this dataset and were lately removed as they seemed redundant. The JSON files also included the same content that the CSV files had.

There are more than 140k tweet IDs in the dataset together.

Yes, there are 140k get-tagged tweets in this dataset. These are the tweets that have "point" location information. If you are okay with having a boundary location instead, you'll have to hydrate the tweets in this dataset (https://ieee-dataport.org/open-access/coronavirus-covid-19-tweets-dataset) and consider conditioning the ["place"] twitter object. The Coronavirus (COVID-19) Tweets Dataset has more than 310 million tweets, and I guess you'll be able to come up with a few million of tweets with the boundary condition enabled.

No. This is a global dataset.

Glad to be of help.

Hello Danqing. Twitter has rate limits for its APIs. Both the hydrator app and twarc handle the rate limits and pull the JSON accordingly. If you're searching for some way to get the hydration process to expedite, I'd recommend involving some other person who has access to the Twitter Devs, and you can ask him/her to hydrate a portion of the IDs.

Hello Trupti. Just to give you a headstart: If I were you, I would play around the location-specific Twitter Objects at three different levels. First, I would check if the tweet is geo-tagged (if it contains an exact location). Secondly, if the tweet is not geo-tagged, chances are that it might have a region or a country boundary box defined. Third, if none of the criteria satisfy, I would simply try to extract location information from the user's profile.

Here's an example of using twarc as a python library for this purpose.

from twarc import Twarc

consumer_key=""

consumer_secret=""

access_token=""

access_token_secret=""

t = Twarc(consumer_key, consumer_secret, access_token, access_token_secret)

for tweet in t.hydrate(open('tweet_ids.txt')):

    if tweet["coordinates"]:

        loc = tweet[‘‘place"]["country"] #place based on the "point" location

        '''check the value in "loc" if it is from a country of your interest'''

        '''however do check if tweet["place"] is of NoneType. In that condition get the long, lat from tweet["coordinates"]["coordinates"] and convert it to human readable format.

    elif tweet["place"]:

        loc = tweet[‘‘place"]["country"] #bounding box region

        '''check the value in "loc" if it is from a country of your interest'''

    else:

        loc_profile = tweet["user"]["location"] #location from profile

        '''check the value in "loc_profile" if it is from a country of your interest'''

However, this dataset contains the geo-tagged tweets IDs. I'd suggest you to use the Coronavirus (COVID-19) Tweets Dataset, that contains more than 386 million tweet IDs. Applying these geo specific conditions on that dataset would help you extract more tweets for your work. I hope this helps. 

Thanks, Antony. It's streaming API. Retweets have NULL geo and place objects. Therefore, retweets won't be making their way to this dataset. However, Quote tweets are included as they can have their own geo and place objects.

Hello Molu. The TextBlob library has been used to compute the sentiment scores.

Please refer to my previous comments.

Hello Yang. Glad to know that you found this dataset useful.
(i) To curate this dataset, the real-time Twitter stream is filtered by tracking 90+ COVID-19 specific keywords (view the attached keywords.tsv file). All the tweets received from the stream make their way to the primary dataset. The primary dataset can be considered a comprehensive collection for all kinds of analyses (sentiment, geo, fact check, trend, etc.).

(ii) Please refer to my previous comments: https://ieee-dataport.org/open-access/coronavirus-covid-19-geo-tagged-tw... and https://ieee-dataport.org/open-access/coronavirus-covid-19-geo-tagged-tw...