Use Python to save Earning Call transcript from Motley Fool

This is for those who wants to get Earning Call(EC) transcript plain text data from Motley fool.

Table of Contents:

• Observe HTML structure (Motley Fool)
• Use BS4 and re to find nodes where we can get target data
• Parse data and save to txt

Photo by Patrick Tomasso on Unsplash

Observe HTML Structure

URL - https://www.fool.com/author/20032/?page=1

We want to get the URL embedded in highlighted texts

By inspecting html code, we can see the link is within class “list-content”

Start writing code

#import required tools
!pip install bs4
from bs4 import BeautifulSoup
import requests
import os
import time
import re

Create a function for getting url. Use time.sleep() to avoid requesting too frequently. For each url, run the parsing script we will do later.

Next, we need to know which nodes transcript within. the HTML tells us that the transcript is text of tag <p> under Class ‘article-content’

Based on the result, create function for transcript. And we can use the second paragraph to get year / quarter / stock quote for creating txt file. Note that we have to test RE rule for different pattern of texts. In this function, we also create folder for each stocks and save transcripts to them.

Last, run the file_gen function and create log file

Output

Code