MySQL and MariaDB are relational database engines that are very popular systems used with Python. They use a relational model with tables to describe relationships between data. MySQL and MariaDB follow a client-server relationship. In other words, the database exists on a server. To make changes to the database, you send a request to the server. The request contains various SQL statements or instructions selecting, updating, or deleting data.…

SQLAlchemy is an object-relational mapping tool, or ORM, that allows developers to work with databases in a more Pythonic way. Tools like SQLAlchemy let you work with Python or even a different language instead of SQL to query and manipulate the data in your databases. Python has many ORM libraries you can use with SQLAlchemy being one of the most popular ones for relational databases. SQLAlchemy also works with many web frameworks including flask and databases, such as SQLite, MySQL, and Postgres.…

In order to access and use a database in Python, you first need to decide which type of database to use and get the associated Python module. These modules might have different syntax for how to interact with them. Because of this Python offers a common specification for these systems to follow so that programmers can manage their own databases. This is known as the Python Database API.…

The Python Scrapy framework has a concept known as Item Loaders. These Item Loaders are used to load data into Scrapy Items once they have been defined. During this process, we can apply input processors and output processors which clean up the extracted data in various ways. With an ItemLoader class and a few small but useful functions, you can strip unwanted characters, clean up whitespace characters, or otherwise modify the data being collected however you see fit.…

An Item in Scrapy is a logical grouping of extracted data points from a website that represents a real-world thing. You do not have to make use of Scrapy Items right away, as we saw in earlier Scrapy tutorials. You can simply yield page elements as they are extracted and do with the data as you wish. Items provide the ability to better structure the data you scrape, as well as massaging the data with Item Loaders rather than directly in the default Spider parse() method.…

Following links during data extraction using Python Scrapy is pretty straightforward. The first thing we need to do is find the navigation links on the page. Many times this is a link containing the text ‘Next’, but it may not always be. Then we need to construct either an XPath or CSS selector query to get the value contained in the href attribute of the anchor element we need.…

To create a project in Scrapy, you first want to make sure you have a good introduction to the framework. This will ensure that Scrapy is installed and ready to go. Once you are ready to go, we’ll look at how to create a new Python Scrapy project and what to do once it is created. The process is similar for all Scrapy projects, and this is a good exercise to practice web scraping using Scrapy.…