Datasets
Open Access
PT7 Web, an Annotated Portuguese Language Corpus
- Citation Author(s):
- Submitted by:
- Jairson Rodrigues
- Last updated:
- Sun, 12/06/2020 - 16:48
- DOI:
- 10.21227/fhrm-n966
- Data Format:
- License:
- Categories:
- Keywords:
Abstract
PT7 Web is an annotated Portuguese language Corpus built from samples collected from Sep 2018 to Mar 2020 from seven Portuguese-speaking countries: Angola, Brazil, Portugal, Cape Verde, Guinea-Bissau, Macao e Mozambique. The records were filtered from Common Crawl — a public domain petabyte-scale dataset of webpages in many languages, mixed together in temporal snapshots of the web, monthly available [1]. The Brazilian pages were labeled as the positive class and the others as the negative class (non-Brazillian Portuguese). The dataset totalized 249.74 GB of raw HTML text related to 16,346,693 unique web pages. The data was preprocessed to produce high dimensionality (2 to the power of 18 = 262,144 features) vectors of word distribution as input for the training and test phases. A demo of the use of this data may be checked in a two-level fractional design to investigate cluster performance on Spark [2].
Dataset Files
- unzip_instructions.txt (1015 bytes)
- pt7-corpus.tbz2 (132.40 GB)
Open Access dataset files are accessible to all logged in users. Don't have a login? Create a free IEEE account. IEEE Membership is not required.