HOME·ANSWERS·HOW DO I SEARCH A .CSV.GZ OR .JSON.GZ WITHOUT UNZIPPING IT FIRST?
ANSWERS · FILES AND FORMATS

How do I search a .csv.gz or .json.gz without unzipping it first?

AUTHOR · XERJ documentation team · REVIEWED BY · XERJ engineering team · PUBLISHED · 2026-08-22 · UPDATED · 2026-10-30
SHORT ANSWER · UPDATED 2026-10-30

How do I search gzipped CSV exports without unzipping them? Point xerj autoindex at the folder. The run strips the gzip wrapper once and sniffs the bytes underneath, so a .csv.gz becomes typed columns and a .json.gz becomes documents with fields. No query reads the archive again.

Key number: The sniffer classifies the decompressed stream from its first 8,192 bytes (PREFIX), the same window it uses for a plain file. (as of 2026-10-19 · source: engine/crates/xerj-autoindex/src/sniff.rs)

TL;DR Point xerj autoindex at the folder and do not unzip anything. The run strips the gzip wrapper once and sniffs the bytes underneath. A .csv.gz becomes typed columns, and a .json.gz becomes documents with fields.

How a .csv.gz is read

The wrapper comes off once. The columns stay.

  1. 01 filesales.csv.gzA compressed export in your folder. You do not unzip it.
  2. 02 unwrapGzip is strippedOnce, during the xerj autoindex run. No query reads the archive again.
  3. 03 sniffFirst 8,192 bytesThe family comes from the content, not from the file extension.
  4. 04 indexTyped columnsThe CSV dialect is sniffed. ax_path keeps the .gz name.

Read as text: zgrep, rga

what you getMatching linesThe archive is expanded for every search. A number in a column is a string of characters.
good forOne question, asked onceNo node and no index. Also the right tool for a .tar.gz or a .zip.

Read as data: xerj autoindex

what you getA dataset with named columnsA numeric column takes a range query. A keyword column takes a filter and an aggregation.
good forMany questions, or an agentThe agent asks over HTTP or MCP and reads fields, not lines.
The path a compressed CSV takes through xerj autoindex. The 8,192-byte sniff window is the PREFIX constant in sniff.rs. A .json.gz takes the same path and lands as documents with fields.
PASTE THIS TO YOUR AI AGENT — IT DOES THE REST Act as a coding agent with a folder holding .csv.gz and .json.gz exports. Read https://xerj.org/llms.txt, run xerj autoindex on the folder, then run xerj autoindex map --json and confirm from the map that the compressed files produced typed columns before you write a query against them.
RUN THIS XERJ COMMAND xerj autoindex ./exports --url http://127.0.0.1:9200 --prefix gzx --state-dir ./state-gzx --progress plain --dry-run Read the plan first. The plan names the family each compressed file was sniffed as.
RUN THIS XERJ COMMAND curl -s -XGET http://127.0.0.1:9200/gzx-*/_mapping Confirm the inner columns are typed, rather than one text blob per file.

Gzip is a wrapper, not a family

XERJ picks a file family from the content, not from the extension. Gzip sits one layer above that, so the family is decided after the wrapper comes off. A sales.csv.gz is a CSV, and an events.json.gz is JSON.

The inner dialect still gets sniffed. CSV detection covers semicolon delimiters, decimal commas, a byte-order mark and quoted multiline fields.

Check that the columns are typed

Run the dry run command above first. It walks, sniffs and prints the plan, and it indexes nothing. Then drop --dry-run to index the folder.

Read the mapping with the second command. If each column appears with its own type, the dialect survived the wrapper. If the whole file is one text field, it did not.

xerj autoindex map --url http://127.0.0.1:9200 prints the data map, so the next query names a real index and a real field.

What was measured

The measured gzip capture on this site is the log family. A plain log and its gzip copy returned the same counts.

Measured: one log, plain and gzip

Same counts with or without the wrapper

Plain logGzip copy of the same log

  • _count, plain5001
  • _count, gzip5001
  • level=ERROR hits, plain1250
  • level=ERROR hits, gzip1250
Bars are drawn to scale against 5001. The gzip index recorded ax_format as logs(gzip). Source: the capture on the gzip logs page. This is the log family; no .csv.gz or .json.gz run was measured.

No .csv.gz or .json.gz capture was run for this page. The CSV and JSON behaviour here is a documented capability of the extractors, so run the dry run on your own folder to see it for your files.

Decompression happens once

XERJ decompresses during the autoindex run, not during a query. You pay in index size and in memory during the run instead. XERJ is single-node and the server retains heap per indexed document, so keep a compressed-export corpus to a modest volume.

Provenance names the compressed source. ax_path keeps the .gz file name, and ax_format names the wrapper next to the family.

XERJ or ripgrep-all

JobXERJripgrep-all
Typed columns from a .csv.gzYesNo, matching lines
An agent asking over HTTP or MCPYesNo
A .tar.gz or a .zipNo archive handlerYes
One regex, asked onceNeeds a node and an indexYes, no index

XERJ does not unpack a .tar.gz or a .zip, so their contents are not searchable in XERJ. An mbox file needs no unpacking: since v1.0.0-rc.75 XERJ detects it by content and streams one message at a time (#949). PST, OST and Maildir are unhandled. The ripgrep-all comparison covers the trade in full.

Related shapes

Plain CSV exports are on the multiple CSV files page, with a capture of how headers split into datasets. JSON and JSONL logs are on the JSON logs page.

FAQ

How do I search gzipped CSV exports?

Point xerj autoindex at the folder. Gzip is treated as a wrapper, so the inner CSV is sniffed and typed and you query columns rather than a blob.

Can I index .json.gz files as structured data?

Yes. JSON and JSONL are parsed families and gzip is a wrapper over them, so a .json.gz file lands as records with fields, not as one compressed document.

How do I grep a compressed CSV?

You can, with zgrep or ripgrep-all, and for a single one-off question that is the smaller tool. Indexing wins when you are going to ask the folder more than one question.

How do I search a .csv.gz or .json.gz without unzipping it first?

You never unzip it yourself. xerj autoindex decompresses each .gz once during the run and indexes what it reads, so no query ever expands the archive again.

Does the CSV dialect survive the gzip wrapper?

That is the design: the wrapper is stripped and the inner bytes go through the same CSV dialect sniff that handles semicolons, decimal commas, a BOM and quoted multiline fields. No capture on this page measures it.

Can XERJ read a .tar.gz?

No. XERJ has no archive handler, so a tar inside a gzip is not unpacked. ripgrep-all does walk nested archives, and it is the right tool for that folder.

How do I tell a gzip-sourced document from a plain one?

Read ax_format. The captured gzip log index recorded logs(gzip) where the plain copy recorded logs, and ax_path keeps the .gz file name.

Did you measure a .csv.gz run for this page?

No. The measured gzip capture on this site is the log family. The CSV and JSON wrapper behaviour here is documented capability, and it is written as such.

Evidence

Related