1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299
|
"""
The tool to check the availability or syntax of domain, IP or URL.
::
██████╗ ██╗ ██╗███████╗██╗ ██╗███╗ ██╗ ██████╗███████╗██████╗ ██╗ ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║ ██║████╗ ██║██╔════╝██╔════╝██╔══██╗██║ ██╔════╝
██████╔╝ ╚████╔╝ █████╗ ██║ ██║██╔██╗ ██║██║ █████╗ ██████╔╝██║ █████╗
██╔═══╝ ╚██╔╝ ██╔══╝ ██║ ██║██║╚██╗██║██║ ██╔══╝ ██╔══██╗██║ ██╔══╝
██║ ██║ ██║ ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝ ╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝
Welcome to PyFunceble!
PyFunceble is the little sister of Funceble
(https://github.com/funilrys/funceble) which was archived on 13th March
2018. In March 2018, because Funceble was starting to become a huge unmanageable
script, I - Nissar Chababy aka `@funilrys`_ - decided to make it a Python tool
for the purpose of extending my Python knowledge. It was meant for my own use case.
Back then, my problem was that I didn't want to download a huge hosts file
knowing that most of the entries do not exist anymore. That's how Py-Funceble
started.
My objective - now - through this tool is to provide a tool and a Python API
which helps the world test the availability of domains, IPs and URL through
the gathering and interpretation of information from existing tools or
protocols like WHOIS records, DNS lookup, or even HTTP status codes.
The base of this tool was my idea.
But as with many Open Source (related) projects, communities or
individuals, we evolve with the people we meet, exchange with or just discuss
with privately. PyFunceble was and is still not an exception to that.
My main idea was to check the availability of domains in hosts files.
But 3 years later, PyFunceble is now capable of a lot including:
- The testing of domains, IPs, and URLs.
- The checking of the syntax or reputation of a domain, IPs, and URLs.
- The decoding of AdBlock filters, RPZ records, or plain files before a test
from the CLI.
PyFunceble evolved and will probably continue to evolve with the time
and the people using it.
In June 2020, The PyFunceble-dev PyPI package - which gets everything as
soon as possible compared to the PyFunceble (stable) package - reached 1 million
total downloads. I never noticed it until I was reached by someone informing me
of it. But, I was shocked.
I never thought that something I built from A to Z in my free time will ever
reach that point.
I was thankful to that nice person for informing me of it. But at the same time
concerned about PyFunceble and how it will evolve. That's why I started the
development of PyFunceble 4.0.0. My idea as I was refactoring it was to provide
a better Python API and implementation of my core ideas along with a better
incorporation and extension capability.
Indeed, in the last few years, I was so much obsessed with the CLI that I
really never wrote each component individually. They were all dependent - if
not part of - the CLI. With 4.0.0, you can now import one of the components
of PyFunceble and start straight away. No real need to play with the
configuration unless you want something very specific.
That's how I see the future of PyFunceble.
As of today, PyFunceble is running actively - if not daily - within several
servers, laptops, PCs, and Raspberry Pis. It is even used - thanks to our
auto continue dataset and component - with CI engines like GitHub Action,
Travis CI, and GitLab CI.
PyFunceble is my tool. But it is indirectly also become yours.
Therefore, I invite you to let me know how you use PyFunceble or simply open a
discussion - or join an existing one - about anything you do with PyFunceble.
But also anything that you - would - like - or dislike - in PyFunceble.
Happy testing with PyFunceble!
Author:
Nissar Chababy, @funilrys, contactTATAfunilrysTODTODcom
Special thanks:
https://pyfunceble.github.io/#/special-thanks
Contributors:
https://pyfunceble.github.io/#/contributors
Project link:
https://github.com/funilrys/PyFunceble
Project documentation:
https://docs.pyfunceble.com
Project homepage:
https://pyfunceble.github.io/
License:
::
Copyright 2017, 2018, 2019, 2020, 2022, 2023, 2024 Nissar Chababy
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
https://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
"""
import os
import platform
import re
from typing import List
import setuptools
def is_win_platform():
"""
Checks if the current platform is Windows.
"""
WIN_PLATFORMS = ["windows", "cygwin", "cygwin_nt-10.0"]
return platform.system().lower() in WIN_PLATFORMS
def get_requirements(*, mode="standard"):
"""
This function extract all requirements from requirements.txt.
"""
mode2files = {
"standard": ["requirements.txt"],
"dev": ["requirements.dev.txt"],
"docs": ["requirements.docs.txt"],
"test": ["requirements.test.txt"],
"psql": ["requirements.txt"],
}
if is_win_platform():
for known_mode, files in mode2files.items():
new_files = set()
for file in files:
win_file = file.replace(".txt", ".win.txt")
if os.path.isfile(win_file):
new_files.add(win_file)
else:
new_files.add(file)
mode2files[known_mode] = list(new_files)
mode2files["full"] = [y for x in mode2files.values() for y in x]
result = set()
for file in mode2files[mode]:
with open(file, "r", encoding="utf-8") as file_stream:
for line in file_stream:
line = line.strip()
if not line or line.startswith("#"):
continue
if "#" in line:
line = line[: line.find("#")].strip()
if not line:
continue
result.add(line)
if mode == "psql":
result.add("psycopg2")
return sorted(list(result))
def get_version():
"""
This function will extract the version from PyFunceble/__init__.py
"""
to_match = re.compile(r'PROJECT_VERSION.*=\s+"(.*)"')
try:
extracted = to_match.findall(
open("PyFunceble/storage.py", encoding="utf-8").read()
)[0]
except FileNotFoundError: # pragma: no cover
extracted = to_match.findall(
open("../PyFunceble/storage.py", encoding="utf-8").read()
)[0]
return extracted[: extracted.rfind(".")]
def get_long_description(): # pragma: no cover
"""
This function return the long description.
"""
return open("README.md", encoding="utf-8").read()
def get_console_scripts(): # pragma: no cover
"""
Provides the console scripts based on the environment.
"""
result = [
"PyFunceble=PyFunceble.cli.entry_points.pyfunceble.cli:tool",
"pyfunceble=PyFunceble.cli.entry_points.pyfunceble.cli:tool",
"clean-pyfunceble=PyFunceble.cli.entry_points.clean:cleaner",
]
dev_console_vars: List[str] = [
"PYFUNCEBLE_INSTALL_DEVTOOLS",
"PYFUNCEBLE_DEVTOOLS",
]
helper_console_vars = [
"PYFUNCEBLE_INSTALL_HELPERS",
"PYFUNCEBLE_HELPTOOLS",
]
if any(x in os.environ for x in dev_console_vars):
result.append(
"production-pyfunceble=PyFunceble.cli.entry_points.production:producer"
)
if any(x in os.environ for x in helper_console_vars + dev_console_vars):
result.extend(
[
"public-suffix-pyfunceble=PyFunceble.cli.entry_points.public_suffix:generator",
"iana-pyfunceble=PyFunceble.cli.entry_points.iana:generator",
]
)
return result
if __name__ == "__main__":
setuptools.setup(
name="PyFunceble-dev",
version=get_version(),
python_requires=">=3.8, <4",
install_requires=get_requirements(mode="standard"),
extras_require={
"docs": get_requirements(mode="docs"),
"dev": get_requirements(mode="dev"),
"test": get_requirements(mode="test"),
"psql": get_requirements(mode="psql"),
"full": get_requirements(mode="full"),
},
description="The tool to check the availability or syntax of domain, IP or URL.",
long_description=get_long_description(),
long_description_content_type="text/markdown",
author="funilrys",
author_email="contact@funilrys.com",
license="Apache 2.0",
url="https://github.com/funilrys/PyFunceble",
project_urls={
"Documentation": "https://docs.pyfunceble.com",
"Funding": "https://github.com/sponsors/funilrys",
"Source": "https://github.com/funilrys/PyFunceble/tree/dev",
"Tracker": "https://github.com/funilrys/PyFunceble/issues",
},
platforms=["any"],
packages=setuptools.find_packages(
exclude=("*.tests", "*.tests.*", "tests.*", "tests")
),
include_package_data=True,
keywords=[
"PyFunceble",
"syntax-checker",
"reputation-checker",
"availability-checker",
],
classifiers=[
"Environment :: Console",
"Topic :: Internet",
"Development Status :: 4 - Beta",
"Intended Audience :: Developers",
"Programming Language :: Python",
"Programming Language :: Python :: 3",
"License :: OSI Approved",
],
test_suite="setup._test_suite",
entry_points={"console_scripts": get_console_scripts()},
)
|