[python]Download file and use tqdm for progress bar.

Download file with requests To download the file from web with requests module is the easiest, you just need to turn on stream while using get method. Then download the file chunk by chunk by using the iter_content method. Here is the example code on how to download file with requests. import requests from pathlib … Continue reading [python]Download file and use tqdm for progress bar.

[python]Download geckodriver for windows

I am writing a python script that uses requests module to download the latest release of geckodriver.exe for windows 64-bit, I added a few functions to make the download slightly more dynamic by identifying the OS type and the system architecture, and after downloaded unzip the package into the defined path I specified. The current … Continue reading [python]Download geckodriver for windows

[selenium] Solution to HTML5 drag and drop with python

Selenium is a great web app automation tool which have several variants, I am using python to implement selenium on this https://the-internet.herokuapp.com/drag_and_drop, Selenium has an ActionChain that can do drag and drop, but this is not working in HTML5, the solution in another language (perhaps is ruby?) can be found here. I have converted the … Continue reading [selenium] Solution to HTML5 drag and drop with python

[python] Adding extension to geckodriver with selenium

Webdriver I am using firefox hence I am downloading geckodriver to work with selenium. To set up the webdriver in the code do this: from selenium import webdriver driver_path = r"E:\webdriver\firefox\geckodriver.exe" driver = webdriver.Firefox(executable_path=driver_path) Install extension To install an extension to the webdriver for each instance do this: buster = r"E:\webdriver\firefox\buster_captcha_solver_for_humans-1.0.1-an+fx.xpi" driver.install_addon(buster, temporary=True) On every … Continue reading [python] Adding extension to geckodriver with selenium

[python]SQLAlchemy usage example

This is an aggregation on how to use SQLAlchemy to create a database, table and insert data into the created table. Refer to this tutorial on how to use SQLAlchemy. from sqlalchemy_utils import database_exists, create_database from sqlalchemy import create_engine, Column, Integer, String from sqlalchemy.ext.declarative import declarative_base from sqlalchemy.orm import sessionmaker url = "mysql+pymysql://username:password@database_address/database_name" # Create … Continue reading [python]SQLAlchemy usage example

[python]VARCHAR requires a length on dialect mysql with sqlalchemy

I remember I did not need to define the varchar length, however I encountered this error: sqlalchemy.exc.CompileError: (in table 'test_table', column 'name'): VARCHAR requires a length on dialect mysql So I would need to define the length of varchar with the String object from SqlAlchemy. meta = MetaData() """ Define the table. The below is … Continue reading [python]VARCHAR requires a length on dialect mysql with sqlalchemy

[python]Create database if not exists with sqlalchemy

This is an example of using SQLAlchemy module to create database if it does not exist otherwise connect to the requested database. from sqlalchemy import create_engine from sqlalchemy_utils import database_exists, create_database from getpass import getpass """ database url is dialect+driver://username:password@db_address/db_name To connect to mysql/mariadb, pymysql module is required to install. The purpose of using SqlAlchemy … Continue reading [python]Create database if not exists with sqlalchemy