GitHub漏洞库批量下载与更新脚本

工具介绍

本项目是一款漏洞库知识库的批量下载更新脚本，主要收集了GitHub上的一些优质漏洞库知识库，并提供一键下载及更新功能，便于在离线情况下的访问检索，适用于Windows平台。

目前收录

[adysec/POC]https://github.com/adysec/POC
[vulhub/vulhub]https://github.com/vulhub/vulhub
[Threekiii/Awesome-POC]https://github.com/Threekiii/Awesome-POC
[Mr-xn/Penetration_Testing_POC]https://github.com/Mr-xn/Penetration_Testing_POC
[ax1sX/SecurityList]https://github.com/ax1sX/SecurityList
[XiaomingX/data-cve-poc]https://github.com/XiaomingX/data-cve-poc
[trickest/cve]https://github.com/trickest/cve
[adysec/nuclei_poc]https://github.com/adysec/nuclei_poc
[coffeehb/Some-PoC-oR-ExP]https://github.com/coffeehb/Some-PoC-oR-ExP
[ycdxsb/PocOrExp_in_Github]https://github.com/ycdxsb/PocOrExp_in_Github
[zhzyker/exphub]https://github.com/zhzyker/exphub

使用说明

usage: run.py [-h] [--proxy PROXY] [--mode {ssh,git}] [--urlpath URLPATH] {init,upgrade}  Bulk pull and update vulnerability databases  positional arguments:   {init,upgrade}        需要进行的操作，'init'或'upgrade'         The operation that needs to be done, init or upgrade  options:   -h, --help            show this help message and exit   --proxy PROXY, -P PROXY                         代理地址，默认为空                         Proxy address, default is empty   --mode {ssh,git}, -m {ssh,git}                         拉取模式，'ssh'或'git'，默认为'git'                         Pull mode, 'ssh' or 'git', defaults to 'git'   --urlpath URLPATH, -p URLPATH                         url文件路径，默认为'./urls.txt'                         URL file path, which defaults to './urls.txt'

首次使用该项目，请先执行以下命令拉取项目进行初始化：

python run.py init

此后更新项目只需执行以下命令即可：

python run.py upgrade

为避免网络不稳定或项目过大导致的无法正常下载，建议使用’ssh’的模式拉取项目的，使用’ssh’模式拉取项目请先完成’ssh’配置。

[GitHub通过ssh方法下载详细配置过程]https://blog.csdn.net/boybs/article/details/124222148

项目预置默认库全部拉取下来预计需要15+GB左右的空间。

run.py

import subprocess import threading import platform import argparse  def check_platform():     if platform.system() == "Windows":         return True     else:         print(f"Sorry, {platform.system()} platform is not supported at the moment...")         return False  def threaded_function(command, thread_name):     print(f"Thread {thread_name} start...")     subprocess.Popen(f'start cmd.exe /K "{command}"', shell=True)     print(f"Thread {thread_name} end...")  def add_suffix(s) -> str:     if s[-4:] != ".git":         s += ".git"             return s  def ssh_mode(s):     s = add_suffix(s)     s = s.replace("https://github.com/", "git@github.com:")     return s  def git_mode(s) -> str:     s = add_suffix(s)     s = s.replace("git@github.com:", "https://github.com/")     return s  def get_repo_name(url) -> str:     last_part = url.split('/')[-1]          if last_part[-4:] == ".git":         last_part = last_part[:-4]     return last_part  def import_urls(path, mode) -> list:     urls = []      with open(path, "r") as f:         data = f.readlines()         for url in data:             url = url.strip()             if mode == "ssh":                 url = ssh_mode(url)             elif mode == "git":                 url = git_mode(url)             else:                 print(f"\033[31m[!] Error Mode: {mode} !\033[0m")                 return []                          urls.append(url)      return urls  def run(commands):     threads = []     for i, cmd in enumerate(commands):         thread = threading.Thread(target=threaded_function, args=(cmd, f'Thread-{i+1}'))         threads.append(thread)         thread.start()      for thread in threads:         thread.join()   if not check_platform():     exit(1)  if __name__ == "__main__":     parser = argparse.ArgumentParser(description="Bulk pull and update vulnerability databases")     parser.add_argument('operate', choices=['init', 'upgrade'], type=str, help="The operation that needs to be done, init or upgrade")     parser.add_argument('--proxy', '-P', type=str, default="", help="Proxy address, default is empty")     parser.add_argument('--mode', '-m', type=str, choices=['ssh', 'git'], default="git", help="Pull mode, 'ssh' or 'git', defaults to 'git'")     parser.add_argument('--urlpath', '-p', type=str, default="./urls.txt", help="URL file path, which defaults to './urls.txt'")      args = parser.parse_args()      commands = []      urls = import_urls(args.urlpath, args.mode)     for url in urls:         command = "git config --global core.autocrlf input && "          if args.operate == "init":             command += f"git {'' if args.proxy == '' else f'-c http.proxy={args.proxy}'} clone {url} {get_repo_name(url)}"         elif args.operate == "upgrade":             command += f"cd {get_repo_name(url)} && git {'' if args.proxy == '' else f'-c http.proxy={args.proxy}'} pull"                  commands.append(command)      run(commands)

urls.txt

https://github.com/adysec/POC https://github.com/vulhub/vulhub https://github.com/Threekiii/Awesome-POC https://github.com/Mr-xn/Penetration_Testing_POC https://github.com/ax1sX/SecurityList https://github.com/XiaomingX/data-cve-poc https://github.com/trickest/cve https://github.com/adysec/nuclei_poc https://github.com/coffeehb/Some-PoC-oR-ExP https://github.com/ybdt/exp-hub https://github.com/ycdxsb/PocOrExp_in_Github https://github.com/zhzyker/exphub