{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Intro\n", "\n", "\n", "In this notebook we will apply REGEX & BeautifulSoup to find useful financial information in 10-Ks. In particular, we will extract text from Items 1A, 7, and 7A of 10-K." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# STEP 1 : Import Libraries\n", "\n", "Note that, we will need parser for BeautifulSoup, there are many parsers, we will be using 'lxml' which can be pre-installed as follows & it help BeatifulSoup read HTML, XML documents:\n", "\n", "!pip install lxml" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "# Import requests to retrive Web Urls example HTML. TXT \n", "import requests\n", "\n", "# Import BeautifulSoup\n", "from bs4 import BeautifulSoup\n", "\n", "# import re module for REGEXes\n", "import re\n", "\n", "# import pandas\n", "import pandas as pd" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# STEP 2 : Get Apple's [AAPL] 2018 10-K \n", "\n", "Though we are using AAPL as example 10-K here, the pipeline being built is generic & can be used for other companies 10-K\n", " \n", "[SEC Website URL for 10-K (TEXT version)](https://www.sec.gov/Archives/edgar/data/320193/000032019318000145/0000320193-18-000145.txt)\n", "\n", "[SEC Website URL for 10-K (HTML version)](https://www.sec.gov/Archives/edgar/data/320193/000032019318000145/a10-k20189292018.htm)\n", "It will be good to view/study along in html format to see how the below code would apply.\n", "\n", "All the documents can be easily ssearched via CIK or company details via [SEC's search tool](https://www.sec.gov/cgi-bin/browse-edgar?CIK=0000320193&owner=exclude&action=getcompany&Find=Search)" ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [], "source": [ "# Get the HTML data from the 2018 10-K from Apple\n", "r = requests.get('https://www.sec.gov/Archives/edgar/data/320193/000032019318000145/0000320193-18-000145.txt')\n", "raw_10k = r.text" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If we print the `raw_10k` string we will see that it has many sections. In the code below, we print part of the `raw_10k` string:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0000320193-18-000145.txt : 20181105\n", "0000320193-18-000145.hdr.sgml : 20181105\n", "20181105080140\n", "ACCESSION NUMBER:\t\t0000320193-18-000145\n", "CONFORMED SUBMISSION TYPE:\t10-K\n", "PUBLIC DOCUMENT COUNT:\t\t88\n", "CONFORMED PERIOD OF REPORT:\t20180929\n", "FILED AS OF DATE:\t\t20181105\n", "DATE AS OF CHANGE:\t\t20181105\n", "\n", "FILER:\n", "\n", "\tCOMPANY DATA:\t\n", "\t\tCOMPANY CONFORMED NAME:\t\t\tAPPLE INC\n", "\t\tCENTRAL INDEX KEY:\t\t\t0000320193\n", "\t\tSTANDARD INDUSTRIAL CLASSIFICATION:\tELECTRONIC COMPUTERS [3571]\n", "\t\tIRS NUMBER:\t\t\t\t942404110\n", "\t\tSTATE OF INCORPORATION:\t\t\tCA\n", "\t\tFISCAL YEAR END:\t\t\t0930\n", "\n", "\tFILING VALUES:\n", "\t\tFORM TYPE:\t\t10-K\n", "\t\tSEC ACT:\t\t1934 Act\n", "\t\tSEC FILE NUMBER:\t001-36743\n", "\t\tFILM NUMBER:\t\t181158788\n", "\n", "\tBUSINESS ADDRESS:\t\n", "\t\tSTREET 1:\t\tONE APPLE PARK WAY\n", "\t\tCITY:\t\t\tCUPERTINO\n", "\t\tSTATE:\t\t\tCA\n", "\t\tZIP:\t\t\t95014\n", "\t\tBUSINESS PHONE:\t\t(408) 996-1010\n", "\n", "\tMAIL ADDRESS:\t\n", "\t\tSTREET 1:\t\tONE APPLE PARK WAY\n", "\t\tCITY:\t\t\tCUPERTINO\n", "\t\tSTATE:\t\t\tCA\n", "\t\tZIP:\t\t\t95014\n", "\n", "\tFORMER COMPANY:\t\n", "\t\tFORMER CONFORMED NAME:\tAPPLE COMPUTER INC\n", "\t\tDATE OF NAME CHANGE:\t19970808\n", "\n", "\n", "10-K\n", "1\n", "a10-k20189292018.htm\n", "10-K\n", "\n", "\n", "\n", "\t\n", "\t\t\n", "\t\t\\n\\t\\t\\n\\t\\tDocument\\n\\t\\n\\t\\n
STEP 3 : Apply REGEXes to find Item 1A, 7, and 7A under 10-K Section \n", "\n", "The items in this `document` can be found in four different patterns. For example Item 1A can be found in either of the following patterns:\n", "\n", "1. `>Item 1A`\n", "\n", "2. `>Item 1A` \n", "\n", "3. `>Item 1A`\n", "\n", "4. `ITEM 1A` \n", "\n", "In the code below we will write a single regular expression that can match all four patterns for Items 1A, 7, and 7A. Then use the `.finditer()` method to match the regex to `document['10-K']`.\n", "\n", "Note that Item 1B & Item 8 are added to find out end of section Item 1A & Item 7A subsequently." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "<_sre.SRE_Match object; span=(38318, 38327), match='>Item 1A.'>\n", "<_sre.SRE_Match object; span=(39347, 39356), match='>Item 1B.'>\n", "<_sre.SRE_Match object; span=(46148, 46156), match='>Item 7.'>\n", "<_sre.SRE_Match object; span=(47281, 47290), match='>Item 7A.'>\n", "<_sre.SRE_Match object; span=(48357, 48365), match='>Item 8.'>\n", "<_sre.SRE_Match object; span=(119131, 119140), match='>Item 1A.'>\n", "<_sre.SRE_Match object; span=(197023, 197032), match='>Item 1B.'>\n", "<_sre.SRE_Match object; span=(333318, 333326), match='>Item 7.'>\n", "<_sre.SRE_Match object; span=(729984, 729993), match='>Item 7A.'>\n", "<_sre.SRE_Match object; span=(741774, 741782), match='>Item 8.'>\n" ] } ], "source": [ "# Write the regex\n", "regex = re.compile(r'(>Item(\\s| | )(1A|1B|7A|7|8)\\.{0,1})|(ITEM\\s(1A|1B|7A|7|8))')\n", "\n", "# Use finditer to math the regex\n", "matches = regex.finditer(document['10-K'])\n", "\n", "# Write a for loop to print the matches\n", "for match in matches:\n", " print(match)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Notice that each item is matched twice. This is because each item appears first in the index and then in the corresponding section. We will now have to remove the matches that correspond to the index. We will do this using Pandas in the next section." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In the code below we will create a pandas dataframe with the following column names: `'item','start','end'`. In the `item` column save the `match.group()` in lower case letters, in the ` start` column save the `match.start()`, and in the `end` column save the ``match.end()`. " ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
itemstartend
0>item 1a.3831838327
1>item 1b.3934739356
2>item 7.4614846156
3>item 7a.4728147290
4>item 8.4835748365
\n", "
" ], "text/plain": [ " item start end\n", "0 >item 1a. 38318 38327\n", "1 >item 1b. 39347 39356\n", "2 >item 7. 46148 46156\n", "3 >item 7a. 47281 47290\n", "4 >item 8. 48357 48365" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Matches\n", "matches = regex.finditer(document['10-K'])\n", "\n", "# Create the dataframe\n", "test_df = pd.DataFrame([(x.group(), x.start(), x.end()) for x in matches])\n", "\n", "test_df.columns = ['item', 'start', 'end']\n", "test_df['item'] = test_df.item.str.lower()\n", "\n", "# Display the dataframe\n", "test_df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Eliminate Unnecessary Characters\n", "\n", "As we can see, our dataframe, in particular the `item` column, contains some unnecessary characters such as `>` and periods `.`. In some cases, we will also get unicode characters such as ` ` and ` `. In the code below, we will use the Pandas dataframe method `.replace()` with the keyword `regex=True` to replace all whitespaces, the above mentioned unicode characters, the `>` character, and the periods from our dataframe. We want to do this because we want to use the `item` column as our dataframe index later on." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
itemstartend
0item1a3831838327
1item1b3934739356
2item74614846156
3item7a4728147290
4item84835748365
\n", "
" ], "text/plain": [ " item start end\n", "0 item1a 38318 38327\n", "1 item1b 39347 39356\n", "2 item7 46148 46156\n", "3 item7a 47281 47290\n", "4 item8 48357 48365" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Get rid of unnesesary charcters from the dataframe\n", "test_df.replace(' ',' ',regex=True,inplace=True)\n", "test_df.replace(' ',' ',regex=True,inplace=True)\n", "test_df.replace(' ','',regex=True,inplace=True)\n", "test_df.replace('\\.','',regex=True,inplace=True)\n", "test_df.replace('>','',regex=True,inplace=True)\n", "\n", "# display the dataframe\n", "test_df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Remove Duplicates\n", "\n", "Now that we have removed all unnecessary characters form our dataframe, we can go ahead and remove the Item matches that correspond to the index. In the code below we will use the Pandas dataframe `.drop_duplicates()` method to only keep the last Item matches in the dataframe and drop the rest. Just as precaution ensure that the `start` column is sorted in ascending order before dropping the duplicates." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
itemstartend
5item1a119131119140
6item1b197023197032
7item7333318333326
8item7a729984729993
9item8741774741782
\n", "
" ], "text/plain": [ " item start end\n", "5 item1a 119131 119140\n", "6 item1b 197023 197032\n", "7 item7 333318 333326\n", "8 item7a 729984 729993\n", "9 item8 741774 741782" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Drop duplicates\n", "pos_dat = test_df.sort_values('start', ascending=True).drop_duplicates(subset=['item'], keep='last')\n", "\n", "# Display the dataframe\n", "pos_dat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Set Item to Index\n", "\n", "In the code below use the Pandas dataframe `.set_index()` method to set the `item` column as the index of our dataframe." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
startend
item
item1a119131119140
item1b197023197032
item7333318333326
item7a729984729993
item8741774741782
\n", "
" ], "text/plain": [ " start end\n", "item \n", "item1a 119131 119140\n", "item1b 197023 197032\n", "item7 333318 333326\n", "item7a 729984 729993\n", "item8 741774 741782" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Set item as the dataframe index\n", "pos_dat.set_index('item', inplace=True)\n", "\n", "# display the dataframe\n", "pos_dat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " Get The Financial Information From Each Item \n", "\n", "The above dataframe contains the starting and end index of each match for Items 1A, 7, and 7A. In the code below, we will save all the text from the starting index of `item1a` till the starting index of `item1b` into a variable called `item_1a_raw`. Similarly, save all the text from the starting index of `item7` till the starting index of `item7a` into a variable called `item_7_raw`. Finally, save all the text from the starting index of `item7a` till the starting of `item8` into a variable called `item_7a_raw`. We can accomplish all of this by making the correct slices of `document['10-K']`." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "# Get Item 1a\n", "item_1a_raw = document['10-K'][pos_dat['start'].loc['item1a']:pos_dat['start'].loc['item1b']]\n", "\n", "# Get Item 7\n", "item_7_raw = document['10-K'][pos_dat['start'].loc['item7']:pos_dat['start'].loc['item7a']]\n", "\n", "# Get Item 7a\n", "item_7a_raw = document['10-K'][pos_dat['start'].loc['item7a']:pos_dat['start'].loc['item8']]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that we have each item saved into a separate variable we can view them separately. For illustration purposes we will display Item 1a, but the other items will look similar." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'>Item 1A.
Risk Factors
The following discussion of risk factors contains forward-looking statements. These risk factors may be important to understanding other statements in this Form 10-K. The following information should be read in conjunction with Part II, Item 7, “Management’s Discussion and Analysis of Financial Condition and Results of Operations” and the consolidated financial statements and related notes in Part II, Item 8, “Financial Statements and Supplementary Data” of this Form 10-K.
STEP 4 : Apply BeautifulSoup to refine the content" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [], "source": [ "### First convert the raw text we have to exrtacted to BeautifulSoup object \n", "item_1a_content = BeautifulSoup(item_1a_raw, 'lxml')" ] }, { "cell_type": "code", "execution_count": 36, "metadata": { "scrolled": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", " \n", "

\n", " >Item 1A.\n", "

\n", " \n", "
\n", " \n", " Risk Factors\n", " \n", "
\n", " \n", "
\n", " \n", " The following discussion of risk factors contains forward-looking statements. These risk factors may be important to understanding other statements in this Form 10-K. The following information should be read in conjunction with Part II, Item 7, “Management’s Discussion and Analysis of Financial Condition and Results of Operations” and the consolidated financial statements and related notes in Part II, Item 8, “Financial Statements and Supplementary Data” of this Form 10-K.\n", " \n", "
\n", "
Item 1A.\n", "\n", "Risk Factors\n", "\n", "The following discussion of risk factors contains forward-looking statements. These risk factors may be important to understanding other statements in this Form 10-K. The following information should be read in conjunction with Part II, Item 7, “Management’s Discussion and Analysis of Financial Condition and Results of Operations” and the consolidated financial statements and related notes in Part II, Item 8, “Financial Statements and Supplementary Data” of this Form 10-K.\n", "\n", "The business, financial condition and operating results of the Company can be affected by a number of factors, whether currently known or unknown, including but not limited to those described below, any one or more of which could, directly or indirectly, cause the Company’s actual financial condition and operating results to vary materially from past, or from anticipated future, financial condition and operating results. Any of these factors, in whole or in part, could materially and adversely affect the Company’s business, financial condition, operating results and stock price.\n", "\n", "Because of the following factors, as well as other factors affecting the Company’s financial condition and operating results, past financial performance should not be considered to be a reliable indicator of future performance, and investors should not use historical trends to anticipate results or trends in future periods.\n", "\n", "Global and regional economic conditions could materially adversely affect the Comp\n" ] } ], "source": [ "### Our goal is though to remove html tags and see the content\n", "### Method get_text() is what we need, \\n\\n is optional, I just added this to read text \n", "### more cleanly, it's basically new line character between sections. \n", "print(item_1a_content.get_text(\"\\n\\n\")[0:1500])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Summary..." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As we have seen that simply applying REGEX & BeautifulSoup combination we can form a very powerful combination extracting/scarpping content from any web-content very easily. Having said this, not all 10-Ks are well crafted HTML, TEXT formats, example older 10-Ks, hence there may be adjustments needed to adopt to the circumstances." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.8" } }, "nbformat": 4, "nbformat_minor": 2 }