Text Analysis¶

Text Analysis¶

We're going to bring in some tools from the pure language side such as nltk, the natural language toolkit. This let's you easily bring in some public domain works, particularly those from Project Gutenberg..ipynb_checkpoints/lecture_4-checkpoint.ipynb: "def generate_sequences(vec, l):\n",

Let's see what books we get from this particular archive. Some good stuff here. Sense and Sensibility!

It's kind of wonky, but you can read each of these through a file-like object. Let's look at an early paragraph from Moby Dick.

Now let's look at how long each book is (in terms of characters) and display it as a DataFrame.

Before using this for anything serious, we're going to want to eliminate any unusual punctuation and merge any whitespace together. Then we're left with something kind of standardized but still readable.

From here we use the sklearn tools for text vectorization. The first is a raw count of words, meaning we'll get back a sparse matrix with one row for each book and one column for each word in our vocabulary.

Note that here we call fit_transform, meaning the fit part determines our vocabulary from the corpus and the transform part actually does the numerical conversion. You may sometimes want to do these separately or to use a pre-existing vocabular instead. There are options for this. The other slightly more advanced alternative is to down-weight each word by its usage frequency in the overall corpus, thus reducing focus on common words such as "the" and "is". This returns a sparse float matrix instead.

Text Similarity¶

We should always try to deal with $\ell^2$ normalized vectors. In the case of general vectors, the cosine similarity is $$ cos(\theta(x,y)) = \frac{x \cdot y}{||x|| \cdot ||y||} $$ But when our vectors are $\ell^2$ normalized, we have $||x|| = ||y|| = 1$, so the cosine similarity is simply the dot product $x \cdot y$. Also note that in this case, the Euclidean distance between two vectors is $$ d(x,y) = (x-y) \cdot (x-y) = ||x|| + ||y|| - 2 x \cdot y = 2 ( 1 - x \cdot y ) $$ which is a monotone and linear function of the cosine similarity, meaning they are essentially equivalent. Let's check that all of our document vectors are properly normalized.

One nice thing about vector representations is that our corpus is simply a matrix, meaning we can do this like find the similarity between all pairs of documents with a simply matrix product. Here we can see some clustering within Austen and Shakespeare.

We also may want to compute word-level statistics. Here we print our a list of the most common words in the corpus. This might seem a bit odd at first, but if you note that the King James Version (KVJ) bible is far longer than any other book, it makes a bit more sense. We could downweight this a bit by replacing the sum below with a mean.

Very common words like the and and account for a very large fraction of total words. We can visualize this by looking at the cumulative share of words accounted by each word in order of overall frequency. This is actually just the Lorenz curve that is used to calculate a Gini coefficient. So we can also calculate a Gini coefficient over words by integrating this. Here we find a value of $86\%$.

Now let's make a general function that calculates the Gini coefficient for a single book and apply this to each book. Here we can see that our boy Melville is a real outlier again! Moby Dick has by far the lowest Gini value, meanining it has a more equal distribution over word usage.

Text Visualization¶

One common method of visualizing word frequency vectors is to project them down onto a much lower dimensional space, in this case two dimensions. The most widely used such method is called TSNE and is included in sklearn.

Now we can plot these in a plane, and it's evident that you see clustering of a similar nature to that seen in the similarity matrix.

Embeddings¶

One issue with using work frequency vectors, however we weight them, is that we won't really be handling synonyms well. Embeddings are just mappings from words or tokens into dense numerical representations, rather than the sparse frequency vectors we've seen so far. These embeddings will (hopefully) map similar words and concepts into similar vectors, meaning we can understand document similarity by looking at the their vector similarity.

Usually you'll be using pre-trained embeddings that have proven accuracy at various tasks, but first let's look at the basic technical construct in torch. That would be nn.Embedding. This requires two arguments: the size of your vocabulary and the size of the resulting vector. Once created, the embedding is a function (actually it's a class that implements __call__) that maps from integer indices into vectors. These indices are just the index of a given word in the vocabulary, so to actually get the embedding for a given word, you have to map from word to index (tokenize) then pass that index to the embedding.

The above code creates the embedding then looks at the vectors for the first three indices. Of course, we haven't trained or initialized this embedding in any way, so the numbers are just random. We don't have an associated vocabulary, so this is all pretty abstract. Additionally, we generally don't want to find the embedding for just one word, we want it for an entire string or set of strings.

For this we need to move on to sentence level embeddings. These are typically based on the transformer models that power most large language models today, though they are much smaller in terms of parameter count that LLMs. We'll go over the inner workings of transforms in the next lecture, for now we just need their output. To access these, we'll go through the sentence_transformers library from HuggingFace. There are tons of different models to choose from, but I'll use one that's particularly high performance for its size.

If you just want to go straight from text to embeddings, use mod.encode. I'm going to do the tokenization (mod.tokenizer) and embedding (mod.forward) steps separately, because we need to break these into chunks for it to work. First we tokenize into a matrix of word ids.

Then we embed these ids into vectors. Note the torch.no_grad! This is important, without it your memory usage will explode because torch will be tracking gradients.

If we wanted to something like semantic search over this corpus, it would be fine to keep the embeddings chunked, but we may also want to look at document level statistics, so lets reaggregate them up to the document level.

Now we can again look at the similarity matrix. Here we see that the clustering is a bit more distinct, and Chesterton's unique style becomes more evident, as does the KJV.

Finally, let's compute TSNE on these vectors as well. These are much smaller (D=384) than our word vectors (D=41757), but it's still too much to visualize for mere humans.

Here we end up seeing similar clustering patterns, though the groups are a bit more distinct.

In general, Huggingface is your friend! They are the go to place for machine learning models of almost any sort. For instance, you can read about the TaylorAI/bge-micro-v2 model at https://huggingface.co/TaylorAI/bge-micro-v2 (kind of like GitHub in the URL pattern). There's a leaderboard for embedding models at https://huggingface.co/spaces/mteb/leaderboard. And I have a blog post going over some of the speed and accuracy considerations at http://doughanley.com/blogs/?post=embed.

Classification¶

Now let's unpack these books a little bit and look at sentence level embeddings again. To do this, instead of breaking up the documents arbitrarily as above, we'll do it at the sentence level and simply truncate longer sentences. Chunking documents is actually pretty difficult sometimes due to inconsistencies in formatting. We're just going to split by periods and throw out short sentneces.

Now we split each book into sentences and concatenate these into one big list of sentences. Additionally, we construct an array indicating which book each sentence belongs to.

The task of embeddings these sentences is considerably easier when we can just use encode, which sorts out tokenization and batching for us, and even displays a fancy progress bar.

As before, we'll project these down to two dimensions using TSNE. Note that the runtime here is considerably larger now since we're doing it over roughly 45K vectors of dimensionality 384.

I'm just showing some Austen, some Shakespeare, and Moby Dick here for clarity, but you can see how authors tend to cluster together and there is considerable variation in how densely packed the various books are.