¶ Repo Scraper
https://github.com/zeinlol/repo-scraper
¶ About
Repo Scraper is a tool designed to analyze projects for possible password leaks or other sensitive data exposure. It scans files and repositories to detect potentially dangerous elements such as passwords, IP addresses, and other confidential information. The tool also supports Git repositories by analyzing changes between commits.
¶ Main Commands
The tool provides the following commands for operation:
- repo-scraper — The main command that launches the analysis of a folder or repository.
- check_dir — Performs checks on the files in a specified directory and its subdirectories.
- check_repo — Performs a scan in a Git repository, analyzing changes between commits and applying regular expressions to detect sensitive data.
¶ How it work
-
check_dir (Directory Check)
This command traverses all files in the specified folder and its subdirectories. It applies regular expressions to search for passwords, IP addresses, and other sensitive information. To optimize the process and avoid scanning irrelevant files, several filters are applied: -
File size: Files larger than 1 MB are ignored, but a warning is displayed.
File extension: Files with unsupported extensions are ignored, and a warning is displayed. (See the "Notes" section for details on why extensions are used instead of MIME types.) -
Base64 data: If a file contains Base64-encoded data, it is removed before applying regular expressions to avoid unnecessary processing.
check_repo (Git Repository Check)
The check_repo command works similarly to check_dir but is focused on analyzing changes in the Git repository's commit history. Instead of checking each commit entirely (which would be slow), the tool uses git diff to analyze only the changes between commits:
- It checks the first commit in the repository’s history using check_dir.
- Then, it iterates through subsequent commits, using git diff to process only the differences between consecutive commits.
Filters similar to those in check_dir are applied:
- Number of lines and number of characters: Limits are set on the number of lines and characters in the analyzed changes.
- File extension: Files with unsupported extensions are ignored, and a warning is displayed.
- Base64 data: Base64-encoded data is removed before applying regular expressions.
¶ Findings Examples
¶ License
Repo Scraper is distributed under the MIT License https://github.com/zeinlol/repo-scraper/blob/master/LICENSE