PT7 Web, an Annotated Portuguese Language Corpus

Citation Author(s):
Jairson
Rodrigues
Federal University of Pernambuco
Germano
Vasconcelos
Federal University of Pernambuco
Paulo
Maciel
Federal University of Pernambuco
Submitted by:
Jairson Rodrigues
Last updated:
Sun, 12/06/2020 - 16:48
DOI:
10.21227/fhrm-n966
Data Format:
License:
0
0 ratings - Please login to submit your rating.

Abstract 

PT7 Web is an annotated Portuguese language Corpus built from samples collected from Sep 2018 to Mar 2020 from seven Portuguese-speaking countries: Angola, Brazil, Portugal, Cape Verde, Guinea-Bissau, Macao e Mozambique. The records were filtered from Common Crawl — a public domain petabyte-scale dataset of webpages in many languages, mixed together in temporal snapshots of the web, monthly available [1]. The Brazilian pages were labeled as the positive class and the others as the negative class (non-Brazillian Portuguese). The dataset totalized 249.74 GB of raw HTML text related to 16,346,693 unique web pages. The data was preprocessed to produce high dimensionality (2 to the power of 18 = 262,144 features) vectors of word distribution as input for the training and test phases. A demo of the use of this data may be checked in a two-level fractional design to investigate cluster performance on Spark [2].

Dataset Files

LOGIN TO ACCESS DATASET FILES
Open Access dataset files are accessible to all logged in  users. Don't have a login?  Create a free IEEE account.  IEEE Membership is not required.