Web Scraping with Langchain and html2text

Introduction

In this blog post, you will be guided on how to perform a simple web scrapping using the available open source python packages. We are going to make use of the langchain and html2text.

Hands on

First, we need to make sure to install the langchain and html2text packages.

!pip install -q langchain playwright beautifulsoup4 html2text

Here's the code snippet for accomplishing the web scrapping. The following code is utilizing the langchain's AsyncHtmlLoader and the Html2TextTransformer from html2text package …

beginners blog install introduction langchain open source pip programming python scraping simple tutorial web web scraping will