General advice to find raw text corpus in your language, look up:

• NLP groups in universities of your country. What have they done?
• NLP projects in the national/royal library. Have they compiled any corpus?
• NLP projects in prominent book/news publishing houses.
• Transcripts of speeches and/or proceedings from parliament/bureaucracy of your country or public administration documents

Specific multilanguage sources:

• Leipzig Corpora Newspaper texts or texts randomly collected from the web, foreign sentences removed
  • 10,000 sentences up to 1 million sentences: download norwegian
• Open subtitles, in total 62 languages, 22G tokens: here, with parser for downloading and extracting tokens into one corups:
  • github.com/…/opensubtitles-parser
  • github.com/…/opensubtitles
• TenTen Corpus Family Corpus for 40 langauges, targets +10 billion words per language, text from web, 2.47 billion words, in bokmål. Source is from a “linguistically valuable corpus” curated from web content.
• TalkOfEurope Project from 2016, that covers all plenary debates held in the European Parliament (EP) between 1999-07 – 2014-01. Access to data here (2.4GB compressed).
• Tatoeba collection of sentences and translations. Their default sentence.csv dataset consists of 8M sentences (500MB), in various languages. Used by Facebook research in their LASER paper, their processed data in github here.
• Common Crawl web crawl data, from +40 languages supported, released monthly. ~ 250 TB of data stored on AWS in WARC format, from billions of web pages. Note: might need substantial cleaning for pre-training BERT. See examples to get started. (Disclaimer: I have not worked with this data set myself).

• OSCAR Open Super-Large Crawl Data, for many languages, see table.

  Table 1: Example of some of the data from OSCAR that is of interest to me

  Language	Words original	Size original	Words deduplicated	Size deduplicated
  Norwegian	1,344,326,388	8.0G	804,894,377	4.7G
  Norwegian Nynorsk	14,764,980	85M	9,435,139	54M
  Swedish	7,155,994,312	44G	4,106,120,608	25G
  Finnish	3,196,666,419	27G	1,597,855,468	13G
  Danish	2,637,463,889	16G	1,620,091,317	9.5G
  Esperanto	48,486,161	299M	37,324,446	228M
  English	418,187,793,408	2.3T	215,841,256,971	1.2T
  French	46,896,036,417	282G	23,206,776,649	138G

• Project Gutenberg: based on 3000 books in English, but worth investigating if there’s something similar for other languages as well?
• Wikipedia-dumps, plus wiki-extractor.py. Note see warning about using wikipedia data further down.
• WikiText the source which multilanguage BERT is trained on. 100M tokens, 110 times larger than Penn Treebank (PTB) (Or is this English only?)
• Homemade BookCorpus Are there non-English books in this data set of 200k books in raw txt?

Other

• conceptnet-numberbatch The best pre-computed word embeddings you can use, word embeddings in 78 different langauges.

• NoReC: The Norwegian Review Corpus 15M tokens, 35.2k full text reviews scored 1-6, from newspapers 2003–2017, created by the SANT project. The data is stored in CoNNL-U format. This can be read from python using following options:
  • pyconll — Fetches review body but not head(?), where the review score is.
  • conllu — Smaller than pyconll. Problem converting to raw text?
  • norec — They provide their own data reading library, I found this to be the simplest to get both text and review score out
• talk-of-norway Norwegian Parliament Corpus 1998–2016, with both metadata, and raw text from speeches.
• noTenTen: Corpus of the Norwegian Web, 2.47 billion words (bokmål), 44 million (nynorsk).
• bokelskere.no, has book review dataset (local goodreads.com equivalent), about 180k reviews, however, only 20% have set a score.
• www.legelisten.no site for doctor, psychiatrist, and dentist reviews. I’ve scraped this manually (47000 reviews, from 4400 unique doctors, as of 2020-09) data not provided, method described further down.
• Språkbanken from Nasjonalbiblioteket, Looks to have many resources, need to lookinto it.
• Aftenposten archive with more than 150 year worth of articles. Requires account on aftenposten, and perhaps/likely not in “raw” format, perhaps one can buy a data-dump from them?
• Norwegian NLP Resources github readme with collection of NLP resources for Norwegian NLP. Great resource.
• Wikipedia.no Norwegian wikipedia, (maybe use wikiextractor).
• Other resources, e.g. Research groups at University in Oslo (UiO):
  • Language Technology Group (LTG), NLP research group at Oslo University.
  • Sentiment Analysis for Norwegian Text (SANT). Research group behind the NoReC data set,
  • Evaluating semantic vectors for Norwegian (Master thesis)

A word of caution when training on wikipedia text, illustrated by example below:

The two biggest non-english languages on wikipedia is Cebuano (language from Philippines) & Swedish. How come? Because the contributor Sverker Johansson (Swedish with wife from Philippines) has written a bot for automatic article generation in these languages, by constructing sentences augmented with data from data bases, e.g.

The average giraffe is [height] meters tall and weighs [mass] kg.

i.e. data not good for training NLP.

Google maps collects a lot of reviews for all kinds of establishments, like restaurants, hotels, parks, landmarks, continents, and weirder things.

There seems to be many options for web scraping, but so far I have not found any way of doing it for free. The available options:

Google Maps Places API

Google provides their own API that is supposedly very straightforward to use, and well documented, it’s a pay-as-you-go pricing model.“

Google Maps Scraper

github repo that extracts reviews from a file urls.txt containing coordinates. I.e. the problem is now to compile the csv-file with locations. Useage:

python scraper.py --N 50 -i url_input_file.txt

generates a csv file containing last 50 reviews of places present in urls.txt

Google Maps Crawler

From company botsol offer free trial before you buy.

Octoparse

Third party scraping “made easy” (no coding needed). Offers 14 days free trial. Tutorial here for scraping data in google maps, and template here.

Apify

module can scrape map at different zoom levels: country, state, city. But must run on their platform? Can also scrape instagram, facebook, etc. There’s a free tier “developer” package.

Google Map Extractor

From Ahmadsoftware. Made to find business leads, extracts rating (but not review text?), only windows?

To get software reviews on apps, there is a readily available python tool for scraping reviews from Google play appstore.

pip install google_play_scraper

There’s a chapter in the free online book Get SH*T Done with PyTorch here, code is also mirrored on colab.

There are a number of ways to do web scraping using python.

Often web scraped reviews/pages can be “polluted” by foreign language, i.e. Norwegian reviews typically contain also Danish, Swedish and English (Or worse: the language has two different ways of writing: bokmål and nynorks).

For example, the bokelskere-set has the following distribution:

This can be filtered out using e.g. polyglot. For Norwegian, and evaluate the below function once, to download the needed resources:

from polyglot.downloader import downloader
from polyglot.text import Text
from polyglot.detect import Detector

def download_polyglot():
    "Helper function with examples for downloading the resources we need"

    print(downloader.download_dir)
    print(downloader.default_download_dir())

    # show supported modules for language
    print(downloader.supported_tasks(lang="en"))

    # check which languages are supported in sentiment analysis
    print(downloader.supported_languages_table(task="sentiment2"))

    # show downloaded packages from polyglot
    print(downloader.list(show_packages=False))

    # download:
    downloader.download("embeddings2.en")

    # Download Norwegian, bokmaal & nynorsk; & English
    downloader.download("sentiment2.no", quiet=False)
    downloader.download("sentiment2.nn", quiet=False)
    downloader.download("sentiment2.en", quiet=False)

    for model in ['sentiment2', 'embeddings2', 'ner2']:
        for lang in ['no', 'nn', 'en']:
            downloader.download(f"{model}.{lang}", quiet=False)

Then we can apply the following filter on a pandas data frame:

def filter_norwegian(df):
    """Return masks for norwegian: bokmaal, and nynorsk

    Parameters
    ----------
    df: PandasSeries
        Column of review text

    Returns
    -------
    no: PandasSeries
        Masking column for bokmaal, Boolean
    nn: PandasSeries
        Masking column for nynorsk, Boolean
    """

    # Filter out non-norwegian
    # df['lang'] = df.text.apply(lambda x: Detector(x).language.code)

    l = []
    for t in df.values:
        try:
            l.append(Detector(t, quiet=True).language.code)
        except Exception as e:
            l.append(np.nan)
            print(e)
            print(t)

    lang = pd.DataFrame(l)
    print(lang.value_counts())

    nn = lang.apply(lambda x: x == 'nn')  # nynorsk
    no = lang.apply(lambda x: x == 'no')  # bokmaal
    return no, nn

Summary of summary:

• BPE counts the frequency of each word in the training corpus. It then begins from the list of all characters, and will learn merge rules to form a new token from two symbols in the vocabulary until it has learned a vocabulary of the desired size (this is a hyperparameter to pick).
• Byte-level BPE by operating on byte level, no <UNK> tokens are needed, and more compact/efficient than BPE.
• WordPiece the difference to BPE is that it doesn’t choose the pair that is the most frequent but the one that will maximize the likelihood on the corpus once merged. It’s subtly different from what BPE does in the sense that it evaluates what it “loses” by merging two symbols and makes sure it’s worth it.
• Unigram Instead of merging to form words, it starts with words and splits. not used directly, only a part of SentencePiece
• SentencePiece deal with not all languages separate words with space. Used in ALBERT and XLNet.

• github.com/huggingface/tokenizers (in Rust, bindings in Python)
  • “You can either use a pre-trained tokenizer or train your own tokenizer”
  • Blog-post: Hugging Face Introduces Tokenizers
  • Supports: BPETokenizer, ByteLevelBPETokenizer, SentencePieceBPETokenizer, BertWordPieceTokenizer
• github.com/glample/fastBPE
• github.com/VKCOM/YouTokenToMe
• github.com/google/sentencepiece

The huggingface tutorial starts simple with the smallest possible code sample for BERT for sentiment analysis by using pre-trained English model, without fine-tuning:

from transformers import pipeline

nlp = pipeline("sentiment-analysis")

for s in ["I despise you", "I've taken a liking to you"]:
    result = nlp(s)[0]
    print(f"label: {result['label']}, with score: {round(result['score'], 4)}")

It’s worth also looking into the following HuggingFace resources:

• tokenizer summary (Excellent. Covers Byte-Pair, WordPiece, Unigram)
• quick tour (minimal example)
• training/fine-tune + links in article
• pre-processing

The best hands-on tutorial using huggingface is Chris McCormick’s tutorial, also available as video. It’s very similar in structure, and excellent to do after, PyTorch From First Principles.

It follows the PyTorch way, by implementing optimizer, scheduler, data loader, i.e. quite verbose but clear, compared to the more recent new way of relying on torch.Train(). He makes use of BertTokenizer and BertForSequenceClassification.

The dataset is The Corpus of Linguistic Acceptability (CoLA). It’s a set of sentences labeled as grammatically correct or incorrect, i.e. binary single sentence classification.

Also worth looking into his tutorial on BERT word embeddings, compare to “normal” embedding, using just a single auto-encoder layer, but in BERT we have 12+1 (“+1” due to input embedding layer), so how should a word be represented? Different layers encode different kind of information.

There is a library, bert-as-service, that already extracts word embedding from BERT in “smart way”:

…uses BERT as a sentence encoder and hosts it as a service via ZeroMQ, allowing you to map sentences into fixed-length representations in just two lines of code.

Spoiler: average embedding for each word piece (sub word token) that makes up the word, typically in the second to last layer. Also, note: words are not uniquely represented, i.e. “river bank” and “money bank”.

The next logical BERT-tutorial in the order is from the free online book Getting things done with PyTorch, specifically the chapter “Sentiment Analysis with BERT and Transformers by Hugging Face using PyTorch and Python”.

This is interesting, since he doesn’t use the BertForSequenceClassification model (a normal Bert + linear layer on top), like McCormick & co, but rather he uses the bare bones language model found in BertModel and then wraps it manually in a PyTorch model class, where he extracts the parameters from the last pooling-layer of BertModel, and connect that to a fully connected layer on top.

Code sample:

from torch import nn
from transformers import BertModel

class SentimentClassifier(nn.Module):
    """Note that we are returning the raw output of the last layer since
    that is required for the cross-entropy loss function in PyTorch to
    work.

    Will work like any other PyTorch model
    """

    def __init__(self, n_classes):
        super().__init__()
        self.bert = BertModel.from_pretrained(PRE_TRAINED_MODEL_NAME)
        # dropout for regularization:
        self.drop = nn.Dropout(p=0.3)
        # fully connected for output
        self.out = nn.Linear(self.bert.config.hidden_size, n_classes)

    def forward(self, input_ids, attention_mask):
        _, pooled_output = self.bert(
            input_ids=input_ids,
            attention_mask=attention_mask)
        output = self.drop(pooled_output)
        return self.out(output)


# Can now use it like any other PyTorch model
model = SentimentClassifier(len(class_names))
model = model.to(device)

New PyTorch method that abstracts away a lot of the boilerplate code for training loop. All functionality is still available through arguments.

See section Multilanguage BERT for tutorial that uses it.

Using the Auto tool of PyTorch it automatically selects the correct class for the model and tokenizer we want to use, by simply specifying the model_name.

from transformers import AutoTokenizer, AutoModel

On using the Auto class, for easier model experimentation Automatic Model Loading using Hugging Face

Also covered by in the quicktour.

In regard to multilanguage BERT’s performance on small languages, BotXO, said the following in an interview:

The multilingual model performs poorly for languages such as the Nordic languages like Danish or Norwegian because of underrepresentation in the training data. For example, only 1% of the total amount of data constitutes the Danish text. To illustrate, the BERT model has a vocabulary of 120,000 words, which leaves room for about 1,200 Danish words. Here comes BotXO’s model, which has a vocabulary of 32,000 Danish words.

• Google Research has a multilanguage BERT is trained on 104 languages.
• Hugging face has transformer models BERT, XLM, and XLM RoBERTa for multilanguage use, with the following checkpoints:
  • bert-base-multilingual-uncased 102 languages
  • bert-base-multilingual-cased 104 languages
  • xlm-roberta-base 100 languages, outperforms mBERT
  • xlm-roberta-large 100 languages, outperforms mBERT

(I do not yet have results when running these models)

From facebook (blog, github) built on PyTorch, to do zero-shot transfer of NLP models from one language, such as English, to scores of others.

LASER sets a new state of the art on zero-shot cross-lingual natural language inference accuracy for 13 of the 14 languages in the XNLI corpus.

• Trained on 93 languages, all using the same BiLSTM encoder
• (…but tokenizer is language specific)
• Is trained on sentence-paris from different languages, to embed all languages jointly in single shared space.

https://engineering.fb.com/ai-research/laser-multilingual-sentence-embeddings/

On of the tasks of interest might be Application to cross-lingual document classification where they train a classifier on English, then apply to several other languages. However, most other applications for the model seems to be measuring distance in embedded space between same sentence in different languages, or similar.

From Google (arXiv, blog), in TF2 available on tfhub. The blog post stats that accuracy decreases slowly with more languages added.

Idea is to leverage small corpus languages by multi language embedding:

… cross-lingual sentence embeddings for 109 languages. The model is trained on 17 billion monolingual sentences and 6 billion bilingual sentence pairs …resulting in a model that is effective even on low-resource languages for which there is no data available during training … interestingly, LaBSE even performs relatively well for many of the 30+ Tatoeba languages for which it has no training data (see below)

It can do various translations / sorting of translation matches.

(I do not have results on running this on any tasks)

BotXO.ai is a danish company (that develops chat bots?), that have trained their own Nordic BERT models (interview) using google TPUs, and common crawl data:

• Norwegian BERT Trained on 4.5 GB text.
• Danish BERT Trained on 9.5 GB text.
• Swedish BERT Trained on 25 GB text.
• models on their github github.com/botxo/nordic-bert

Next, to load these models, I’ve explored using the native TensorFlow-way (equals pain), and then PyTorch. The pre-trained model checkpoints contain the following files:

bert_config.json
bert_model.ckpt.data-00000-of-00001
bert_model.ckpt.index
bert_model.ckpt.meta
vocab.txt

• The Swedish royal library has trained multiple models (currently 5), both BERT and ALBERT; available on their github, described in paper.
• GPT-2 generated Karin Boye (swedish) poems

Question is how much data is needed to train a transformer model? GPT-2 (82M parameters) is trained on wikiText-103 (100M tokens, 200k vocab).

There are examples of fine-tuning on 900k tokens & 10k vocabulary, or 100M tokens and 200k vocabulary. Generally, typical values for VOC_SIZE are somewhere in between 32000 and 128000, and 100M lines is sufficient for reasonable sized BERT-base (according to blog).

SpaCy writes in a blog post, (about interfacing with huggingface), that “Strubell (2019) calculates that pretraining the BERT base model produces carbon emissions roughly equal to a transatlantic flight.”

Tensorflow-models, can be converted to PyTorch, if that is preferable. Using TF, to pre-training from scratch, google writes:

our recommended recipe is to pre-train a BERT-Base on a single preemptible Cloud TPU v2, which takes about 2 weeks at a cost of about $500 USD (based on the pricing in October 2018). You will have to scale down the batch size when only training on a single Cloud TPU, compared to what was used in the paper. It is recommended to use the largest batch size that fits into TPU memory.

Also of interest:

• TPU-pricing
• Training on TPUv2 took ~54h for this tutorial, but google will terminate running colab notebooks every ~8h, so need pre-paid instance.

General:

• Input data file needs to be as shard, since it stores all at once in memory
• The input is a plain text file, with one sentence per line. Documents are delimited by empty lines. The output is a set of tf.train.Examples serialized into 0TFRecord= file format.

Resources and tutorials to keep in mind:

• TF: Text Classification with Movie Reviews
• Blog: Pre-training BERT from scratch with cloud TPU (code)

• github: SentencePiece

  SentencePiece supports two segmentation algorithms, Byte-Pair-Encoding (BPE) and unigram language model. … SentencePiece allows us to make a purely end-to-end system that does not depend on language-specific pre/postprocessing. … Fast and lightweight.

If we want to train a new model from scratch, e.g. a new language, there’s a huggingface tutorial for Esperanto that is useful. The model size is comparable to DistilBERT (84M parameters = 6 layers x 768 hidden, 12 attention heads) trained on 3GB dataset (oscar + Leipzig). The tutorial is mirrored in a colab version (git), as that makes use of the new PyTorch Trainer() method, rather than as a running the training as scrip from the command line with long string of arguments.

In spite of the model being relatively small, on my Nvidia GPU (GeForce GTX 1050 Ti Mobile) it would take 40h to train a single Epcoh of 1M Esperanto sequences, in batches of 8 (reduce from default 64, to fit on 4GB GPU RAM).

The following from the tutorial caused me some head scratching:

“We choose to train a byte-level Byte-pair encoding tokenizer (the same as GPT-2), with the same special tokens as RoBERTa. Let’s arbitrarily pick its size to be 52,000. We recommend training a byte-level BPE (rather than let’s say, a WordPiece tokenizer like BERT) because it will start building its vocabulary from an alphabet of single bytes, so all words will be decomposable into tokens (no more <unk> tokens!).”

Stuff I’m currently reading:

• NVIDIA NeMo “Make sure you have nemo and nemo_nlp installed before starting this tutorial.” NeMo is a framework-agnostic toolkit for building AI applications powered by Neural Modules. Current support is for PyTorch (minimum v.1.4) framework.
• “Multi Class Text Classification With Deep Learning Using BERT Natural Language Processing, NLP, Hugging Face” (probably copy-paste of McCormick’s blog) https://towardsdatascience.com/multi-class-text-classification-with-deep-learning-using-bert-b59ca2f5c613 git - notebook

There is more then just BERT to the huggingface library of pretrained transformer-models. Here’s a short summary of (the summary of) the models I found interesting in my quest for text classification.

Note that the only difference between autoregressive models (e.g. GPT) and autoencoding models (e.g. BERT) is in the way the model is pretrained.

• BERT
  • Models: bert-base-uncased
• ALBERT (A lite BERT …)
  • Smaller embedding layer, than hidden layer (E << H)
  • Group layers that share weights (save RAM)
  • Models: albert-base-v1
• RoBERTa > BERT - A Robustly Optimized BERT Pretraining Approach
  • Better optimized hyperparameters compared to BERT
  • Same implementation as BertModel with a tiny embeddings tweak as well as a setup for RoBERTa pretrained models
  • RoBERTa shares architecture with BERT, but uses a byte-level BPE as a tokenizer (same as GPT-2) and uses a different pre-training scheme.
  • RoBERTa doesn’t have token_type_ids,
  • Models: distilroberta-base
• DistilBERT Destilled BERT. A faster lighter cheaper version of BERT.
  • Same implementation, derived from a pre-trained BERT
  • Significantly fewer parameters.
  • Models: distilbert-base-uncased, distilbert-base-multilingual-cased
• mBERT Multilingual BERT, see above
• XLM Cross-lingual Language Model Pretraining (arXiv)
  • On top of positional embeddings, the model has language embeddings
  • Three different checkpoints (for three different types of training)
    1. Causal language modeling (CLM)
    2. Masked language modeling (MLM) (like RoBERTa)
    3. Combined MLM & Translation language modeling (TLM)
  • Models: xlm-mlm-100-1280 trained on 100 langauges
• XLM-RoBERTa
  • Uses RoBERTa tricks on the XLM approach, sans TLM

  • Doesn’t use the language embeddings, so it’s capable of detecting the input language by itself.

    Our model, dubbed XLM-R, significantly outperforms multilingual BERT (mBERT) on a variety of cross-lingual benchmarks“… This implementation is the same as RoBERTa

  • Models: xlm-roberta-base
• XLNet Generalized Autoregressive Pretraining for Language Understanding
  • Huggingface documentation
  • Has no sequence length limit
  • Models: XLNetForSequenceClassification, XLNetForMultipleChoice, XLNetForTokenClassification

There are many models that go beyond BERT, and out perform it on bench marks. Some of interest might be:

• XLNet (Google), Generalized Autoregressive Pretraining for Language Understanding
  • uses “transformer-XL” (better at handling long complicated sentences)
  • does the word-masking in training differently to BERT (shuffled)
  • Autoregressive (like GPT-1/2), i.e. good at generating new text, rather than autoencoder, which are good at reconstructing learned text, as BERT
  • Is implemented in Huggingface, doc
• ERNIE (Baidu), Enhanced Representation through Knowledge Integration, record breaking general language model, by also including knowledge about the world.
• ERNIE 2.0 “Experimental results demonstrate that ERNIE 2.0 outperforms BERT and XLNet on 16 tasks
• Language-Agnostic BERT Sentence Embedding (google) as covered previously above .