Rotating Proxy Servers in Python
Recently, I have used ProxyMesh proxy server for a project. ProxyMesh offers 15 proxy servers, each denoting a specific location (such as us-dc) with 10 IP addresses rotating twice per day.
However, the free trial allows you to have one proxy server at a time. This means that if you are working with a server or CDN that strictly throttles IPs, you have to change the proxy server manually to keep rotating between the 15 proxy servers.
Fortunately, ProxyMesh provides an API that can be used to add and delete the proxy servers in user dashboard. Once a proxy server is assigned to the user, it can be fetched and used. All we need is a script that rotates between the proxy servers, deleting and adding them as required.
Based on this, I have written a script that would rotate through ProxyMesh proxy servers using their API.
An account with ProxyMesh, either free trial or paid. Set the user name and password in rotatingproxy.py
self.user = "" self.password = ""
Setting the Proxy Server
from rotatingproxy import RotatingProxy rproxy = RotatingProxy()
The proxy server can either be set randomly or selected from an available list of proxy servers.
The active proxy server is saved in a text file which can be accessed as required.
rproxy.set_proxy(israndom="r") # select a random proxy server rproxy.set_proxy(proxy_num=1) # select proxy server with index=1 from the list of proxy servers.
Accessing the Proxy Server
def get_proxy_from_file(): # fetches proxy from proxy.txt with open("proxy.txt", "r") as f: return loads(f.read()) proxy = get_proxy_from_file()
The proxy can now be used with requests:
import requests response = requests.get("url-to-fetch", proxies=proxy)