gg/README.md

# Grokkin' Grep (`gg`)

GG is a minimal, fast, and **portable** grep-like utility that finds files
containing a literal substring. `gg` prints the paths of every file that contains
the search term — one path per line — and is designed to be competitive with
The Silver Searcher and ripgrep on large source trees while depending on nothing
but POSIX and the C standard library.

```
$ gg "public domain" ~/proj
/home/user/proj/foo/license.txt
/home/user/proj/bar/COPYING
...
```

## About

GG answers one question well: *which files contain this exact text?*
It is intentionally narrow:

- **Literal, case-sensitive substring search.** No regex, no flags, no
  surprises. The term you pass is matched byte-for-byte.
- **File-finding, not line-finding.** Output is a list of matching file paths,
  not the matching lines. (Pipe to `grep -n` if you want line numbers.)
- **Binary-aware.** Files containing a NUL byte are treated as binary and
  skipped. A literal match that occurs *before* the first NUL in an
  otherwise-text file is still reported.
- **Recursive by default.** Given a directory it walks the whole tree.
- **Git-aware by default.** When walking a tree, `gg` honours every `.gitignore`
  it encounters and always skips `.git` directories — just like `git` and `rg`.
  Set `GG_GIT_ALL=1` to search everything instead (equivalent to `rg
  --no-ignore`).
- **Self-contained.** No third-party libraries, no shelling out to other
  programs. Just `libc` + `libpthread`.

`gg` was built to be a clean, readable reference implementation that is also
fast: on a 16 GB / ~209 000-file tree it completes the equivalent of
`rg --no-ignore --hidden -l` in roughly **0.4 s**, about **1.5× faster than
`rg`** on the same machine and query, using only portable POSIX interfaces. (With `GG_GIT_ALL` unset, the equivalent is `rg --hidden -l`.)

## Installation

### From source

You need a C compiler (`gcc` or `clang`) and `make`. No other dependencies.

```sh
git clone <repo-url> gg && cd gg
make
# optional, system-wide:
sudo make install PREFIX=/usr/local
```

This produces a single executable named `gg`.

### Build options

- **Portable build (no x86-specific flags):**

  ```sh
  make MARCH=""
  ```

  `gg` uses POSIX `memmem()` for matching; glibc (and other libcs) already
  accelerate that with SIMD on whatever CPU you run on. The default `-mavx2`
  flag is therefore optional and can be dropped for maximum portability.

- **Cross-compiling for ARM / other targets:**

  ```sh
  aarch64-linux-gnu-gcc -O2 -std=c11 -Wall -Wextra \
    -D_POSIX_C_SOURCE=200809L -D_DEFAULT_SOURCE -o gg gg.c -lpthread
  ```

  `gg` contains no SIMD intrinsics and no platform-specific code, so it builds
  and runs unchanged on ARM64 and other POSIX platforms.

### Uninstall

```sh
sudo make uninstall PREFIX=/usr/local
```

## Usage

```
gg SEARCH_TERM [FILE_OR_DIRECTORY]
```

- `SEARCH_TERM` — required. A case-sensitive literal substring to find. Must be
  non-empty.
- `FILE_OR_DIRECTORY` — optional. Defaults to the current working directory.
  May be a single regular file or a directory (which is walked recursively).

### Examples

```sh
# Find every file under the current directory containing "TODO"
gg TODO

# Search a specific tree for a license header
gg "public domain" ~/proj

# Check a single file
gg "def main" src/main.py
```

### Exit codes

| Code | Meaning |
|------|---------|
| `0`  | At least one file matched. |
| `1`  | No files matched. |
| `2`  | Usage or input error (missing/empty term, bad path, etc.). |

### Behaviour notes

- **Symlinks are not followed.** Symbolic links to files or directories are
  skipped, which prevents infinite loops on symlink cycles.
- **Hidden files and directories are searched.** `gg` does not skip hidden
  paths. It *does* consult `.gitignore` files and skips `.git` directories by
  default; set `GG_GIT_ALL=1` to disable that and search everything (including
  ignored and `.git` paths).
- **Binary files are skipped.** A file containing a NUL byte is never reported,
  even if it also contains the search term (except when the match occurs before
  the first NUL, in which case it is reported — see FAQ).

## How it works

`gg` is a small, readable program. The full algorithm, in high-level terms:

1. **Parse arguments.** Read the search term (must be non-empty) and the optional
   target (a file or directory, defaulting to the current directory).

2. **Parallel walk + scan (the core idea).** Rather than walking the whole tree
   first and *then* scanning, `gg` overlaps the two. A shared, mutex-protected
   **directory work-queue** holds directories still to be processed. `N` worker
   threads (one per online CPU) each:

   a. **Pop a directory** from the queue.
   b. **Open it** with `opendir`/`readdir` and iterate its entries.
   c. For each entry, use `d_type` from `readdir` to classify it cheaply:
      - `DT_REG` (regular file) → scan it immediately (see step 3).
      - `DT_DIR` (sub-directory) → **push it onto the work-queue** for another
        worker to pick up.
      - `DT_LNK` (symlink) → skip it (no following, no cycles).
      - `DT_UNKNOWN` → fall back to `lstat` to decide file vs. directory, then
        act accordingly.
   d. **Free the directory** and loop back to pop the next one.

   Because directories are handed out one at a time and sub-directories are
   re-enqueued, the metadata walk and the data scan proceed concurrently across
   all cores, and no single deep directory becomes a bottleneck for any one
   thread.

3. **Scan a file (early binary detection + streaming match).** For each regular
   file the worker:

   a. Opens it with `open(O_RDONLY | O_CLOEXEC)`.
   b. Reads it in **fixed-size chunks** (32 KB) into a small per-thread buffer.
      Chunks intentionally **overlap by `len(term) - 1` bytes**, so a match that
      straddles a chunk boundary is never missed.
   c. On each chunk, scans for a NUL byte with `memchr`:
      - If a NUL is found, the file is **binary**: `gg` stops reading
        immediately (no need to scan the rest) and reports nothing for it.
      - Otherwise it searches the text portion of the chunk for the term with
        `memmem`. If found, the file is a match and scanning stops.
   d. If the term is not in this chunk and no NUL was seen, it carries the
      overlap into the next read and continues.

   Stopping at the first NUL is what makes `gg` fast on trees full of binary
   artifacts (archives, object files, images): those files are abandoned after a
   single tiny read instead of being fully scanned.

4. **Report matches.** The first time any worker finds a match, it records a
   global "found" flag and prints the file path. Reporting is guarded by a mutex
   so output stays coherent; because matching files are comparatively rare, this
   lock is almost never contended.

5. **Finish.** When the work-queue is empty and all workers have exited, `gg`
   returns `0` if anything matched, otherwise `1`.

In short: **per-directory work units + inline file scanning + stop-at-NUL binary
skipping + POSIX `memmem` matching**, all parallelised across the online CPUs.
No `mmap`, no regex engine, no external processes — just portable syscalls and
library calls.

## FAQ

### Why is it called Grokkin' Grep?

A playful name for a tool that "groks" your files to find text. `gg` is also
nicely short to type.

### Is the search case-sensitive?

Yes. `gg` matches the term exactly as given. There is no case-insensitive mode.

### Does it support regular expressions?

No. `gg` is a literal substring finder. For regex, use `grep`, `rg`, or `ag`.

### Why does it print file paths instead of matching lines?

`gg` is a *file finder* — it answers "which files contain this?". Pipe its output
through `xargs grep -n` (or `rg`) if you need the lines and line numbers.

### How is a "binary file" decided, and why skip it?

A file is considered binary if it contains a NUL (`\0`) byte. This matches
`ripgrep`'s default heuristic. Such files are skipped because text search over
them is usually meaningless. If your term appears *before* the first NUL in an
otherwise-readable file, `gg` still reports it (it only abandons the file once it
reaches the NUL).

### Does it follow symlinks?

No. Symbolic links to files or directories are skipped, which makes `gg` safe
against symlink cycles and avoids double-counting.

### Does it respect `.gitignore`?

Yes, by default. `gg` reads every `.gitignore` found while walking and skips
matching files and directories, and it always skips `.git` directories — the
same rules `git` itself uses (including `!` negation, `**` globs, and
trailing-`/` directory-only patterns). Hidden files and directories are still
searched; only `.gitignore`-matched paths and `.git` are excluded.

To search everything — ignored files, `.git`, the lot — set the `GG_GIT_ALL`
environment variable (to any value) before running `gg`:

```sh
GG_GIT_ALL=1 gg "needle" .
```

With `GG_GIT_ALL` set, `gg`'s results match `rg --no-ignore --hidden`.

### Why not `mmap` the files?

`mmap` was prototyped and rejected: across hundreds of thousands of files the
`munmap` syscall storm dominated, and `read()` into a small per-thread buffer
scaled better. `read()` also avoids faulting the entire file into memory.

### Is it really portable / dependency-free?

Yes. The only external links are `libc` and `libpthread`. There are no SIMD
intrinsics in the source; matching uses `memmem()`, which your libc already
accelerates with SIMD on the host CPU. It builds and runs on x86-64 and ARM64
alike.

### How fast is it?

On a warm cache over a 16 GB / ~209 000-file tree, `gg "public domain" ~/proj`
completes in roughly **0.4 s** — about **1.5× faster** than the equivalent
`rg --no-ignore --hidden -l` on the same hardware, using only portable POSIX
interfaces.

## Credits

Created by Luxferre in 2026, released into the public domain with no warranties.