Managing proxies for scaled data scraping and other automation operations will eventually require something like ProxyNest. ProxyNest is a proxy managment API that is well-suited for mid-scale and will soon be made for large ones.
- FastAPI
- Uvicorn
- MongoDB
- add/update/delete proxies and fetch by country tag or codes.
- reset all proxies if anything goes wrong.
- reserve proxies for instances.
- refresh the reservation on request.
- clear the instance reservation.
- periodic check for proxy uptime and latency.
- periodic check for country code of the IP.
- cleanup for instances that expired.
the default username is admin and password is password :p
Make sure you have docker installed and a mongoDB hosted. let's start by building the docker image, get into the directory using any terminal and run the docker build command.
docker build -t proxynest .
then we'll start it and pass the right variables to your mongoDB
docker run -d -p 8000:8000 --name proxynest \
-e DB_URL='your_db_url' \
-e DB_NAME='your_db_name' \
-e DB_USER='your_db_user' \
-e DB_PASSWORD='your_db_password' \
-e PORT=8000 \ proxynest
you'll have it running in no time serving requests !
requires Python 3.10, click here.
1 - Modify your environment variables.for linux change setenv.sh then run this command.
bash setenv.sh
for windows change setenv.bat then just double click the bat file.
2 - install the required packages using
pip install -r requirements.txt
3 - run the API.py
python API.py
feel free to edit this part of the code in APY.py to change uvicorn launch settings.
if __name__ == "__main__":
uvicorn.run("API:app", host="0.0.0.0", port=8042)
Including the overall routes in here, you can read indepth about the documentation when you read the webserver, or the wiki once its setup.
| Route url | Usage |
|---|---|
| /Docs | Full routes documentation |
| /add_proxies | Add proxies. |
| /proxies | Get proxies |
| /assign_proxy | Assign proxies to an instance |
| /update_proxy/{proxy_id} | update proxy |
| /delete_proxy/{proxy_id} | delete proxy |
| /refresh_proxy_usage/{proxy_id} | refresh proxy |
| /clear_instance_proxies/{instance_id} | clearn instance proxies |
| /clear_instance_from_specific_proxy/{proxy_id}/{instance_id} | clear instance from specific proxy |
| /reset_all_proxies | reset all proxies |
| /reset-password/ | reset auth password |
- fix tag system with proxy reservation as its unused there for now.
- write an actual documentation of the project.
- build a docker image automatically and publish using Github actions.
- perfecounter usage for response time instead of what is being currently used.
- Integrated rotating proxy creation.
- Better authentication.
- Considerably better task queuing instead of using asyncio task queuing for very large scale operations.
- Custom host to measure latency.
- Multi user support.
- Write more documentation
- Stress test with k6
- more profiling
This project includes GeoLite2 data created by MaxMind, available from https://www.maxmind.com.