Magi Practical Web Article Corpus

Magi Practical Web Article Corpus (MPWAC) contains 10 million Chinese web articles consisting of more than 10 billion words. Articles have been shuffled to mitigate the negative influence on mini-batch based training. Each article has been refined such that only the main body of the text is left, without advertisements or other noises. This corpus is extracted in May 2019, by magi.com crawlers located inside Mainland China. The merged file should be processed with Gzip stream. It is encoded in UTF-8, and has the format of '0xFE [URL] 0xFF [TEXT] ...'.