Skip to content

Commit 448a8ab

Browse files
authored
fix: resolve all v0.20.0 binary bugs (#433-#485) and prepare v0.21.0 (#486)
* fix: resolve all v0.20.0 binary bugs (#433-#485) and prepare v0.21.0 Resolve the v0.20.0 hardening bugs tracked under the roadmap (#148), grouped by theme, each with Go regression tests and shellspec binary-level coverage. Breaking changes: - Reject explicit transaction control, VACUUM/VACUUM INTO, and ATTACH/DETACH with a clear sqly error: each statement runs in its own transaction on a single in-memory connection, so they cannot work across statements, and ATTACH would write external SQLite files outside the import/save model (#441, #442, #443, #444, #457, #458, #463). - Reject a non-interactive --save/--save-dir run whose SQL changes schema or runs a maintenance statement (ALTER/DROP/REINDEX/ANALYZE, CREATE/DROP of a table/view/index/trigger, CTAS), since write-back can only persist DML on imported tables (#433-#438, #469-#484). Bug fixes: - Report neutral success for a no-rowset DDL/PRAGMA/maintenance statement instead of a misleading affected-row count (#439). - Run setter and no-row command PRAGMAs on the exec path (#440, #485). - Allow a batch/--sql-file script that imports its own input with .import and then modifies it under save flags (#456). - Accept schema-qualified names in .schema/.describe/.header/.dump (#445-#448). - List session views and TEMP tables in .tables; print CREATE VIEW for a view and the stored DDL for a constrained TEMP table (#449, #450, #451, #464). - Import empty compressed JSON/JSONL as zero-row tables (#452, #453). - Strip all compression suffixes before the ACH/Fedwire output guard (#459, #460). - Accept /dev/stdin, /dev/stdout, /dev/stderr, and /proc/<pid|self>/fd/* input paths (#461, #462). - Validate LTSV labels on output and reject duplicate labels on output and import (#465, #466, #467). - Keep a multiline CREATE TRIGGER body as one statement in batch/--sql-file parsing (#468). * fix: import pseudo-files end-to-end and adopt upstream LTSV root fix Address review follow-ups on the v0.20.0 hardening work: - #461/#462: a pseudo-file input (/dev/stdin, /dev/stdout, /dev/stderr, /proc/<pid|self>/fd/*) now imports end-to-end, not just passing path validation. An extensionless pseudo-file is staged as CSV; use --stdin FORMAT for another format. - #467: bump filesql to v0.14.0, which rejects a duplicate label within an LTSV record on import (upstream root fix via fileparser v0.5.2), and remove the temporary sqly-side duplicate-label check. The regression tests now exercise filesql's rejection. - Fix a doc comment that named the wrong test function (CodeRabbit).
1 parent fbd64f5 commit 448a8ab

38 files changed

Lines changed: 1785 additions & 134 deletions

CHANGELOG.md

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,26 @@
11
# CHANGELOG
22

3+
## [v0.21.0](https://github.com/nao1215/sqly/compare/v0.20.0...v0.21.0) (2026-06-01)
4+
5+
### Breaking Changes
6+
* Unsupported Statements Rejected Clearly: Explicit transaction control (`BEGIN`/`COMMIT`/`ROLLBACK`/`SAVEPOINT`/`RELEASE`), `VACUUM`/`VACUUM INTO`, and `ATTACH`/`DETACH DATABASE` are now rejected with a clear sqly error. sqly runs each statement in its own transaction on a single in-memory connection, so these cannot work across statements, and ATTACH would let a session read or write external SQLite files outside the import/save model (#441, #442, #443, #444, #457, #458, #463).
7+
* Write-Back Rejects Schema-Only Runs: A non-interactive `--save`/`--save-dir` run now fails up front when the SQL changes schema or runs a maintenance statement (ALTER, DROP, REINDEX, ANALYZE, CREATE/DROP of a table/view/index/trigger, including `CREATE TABLE AS SELECT`), since write-back can only persist `INSERT`/`UPDATE`/`DELETE` on imported tables. Previously such a run exited 0 and reported success while leaving the source unchanged (#433, #434, #435, #436, #437, #438, #469, #470, #471, #472, #473, #474, #475, #476, #477, #478, #479, #480, #481, #482, #483, #484).
8+
9+
### Bug Fixes
10+
* Neutral Result Message For Non-DML: A DDL, PRAGMA, or maintenance statement now reports `statement executed successfully` instead of a misleading `affected is N row(s)` count (#439).
11+
* PRAGMA On The Exec Path: A setter PRAGMA (`PRAGMA user_version = 1`) and a no-row command PRAGMA (`PRAGMA incremental_vacuum`) now run successfully instead of failing with a "no records" error (#440, #485).
12+
* Batch .import Under Save Flags: A batch or `--sql-file` script that imports its own input with `.import` and then modifies it is now allowed under `--save`/`--save-dir`; write-back is validated after the import runs (#456).
13+
* Schema-Qualified Helper Commands: `.schema`, `.describe`, `.header`, and `.dump` accept schema-qualified names such as `main.user` (#445, #446, #447, #448).
14+
* TEMP Tables And Views In Helper Commands: `.tables` lists session-created views and TEMP tables (#449, #450); `.schema` prints the real `CREATE VIEW` for a view (#451) and reads the stored definition for a constrained TEMP table instead of a lossy reconstruction (#464).
15+
* Empty Compressed JSON And JSONL: An empty compressed JSON array (`.json.gz`) and an empty compressed JSONL file now import as a zero-row table, matching the uncompressed inputs (#452, #453).
16+
* Output Destination Safety: `--output` and `.dump` strip every trailing compression suffix before checking for an input-only ACH/Fedwire extension, so a path like `out.ach.gz.zst` is rejected instead of receiving CSV bytes (#459, #460).
17+
* Pseudo-File Inputs: `/dev/stdin`, `/dev/stdout`, `/dev/stderr`, and the Linux `/proc/<pid|self>/fd/*` aliases pass input-path validation and import end-to-end. An extensionless pseudo-file is staged as CSV (use `--stdin FORMAT` for another format), matching the already-allowed `/dev/fd/*` (#461, #462).
18+
* LTSV Label Validation: LTSV output rejects a column name that is not a valid LTSV label (for example `foo:bar`) or that duplicates another, and LTSV import rejects a row that repeats a label, so LTSV stays round-trippable instead of silently losing values (#465, #466, #467).
19+
* Multiline CREATE TRIGGER: Batch and `--sql-file` parsing keeps a `CREATE TRIGGER ... BEGIN ... END` body as one statement instead of splitting it at the inner semicolons (#468).
20+
21+
### Dependencies
22+
* filesql: 0.13.0 → 0.14.0, which rejects a duplicate label within an LTSV record on import (the upstream root fix for #467, replacing the temporary sqly-side check) and pulls in fileparser 0.5.2.
23+
324
## [v0.20.0](https://github.com/nao1215/sqly/compare/v0.19.0...v0.20.0) (2026-06-01)
425

526
### Bug Fixes

domain/model/export.go

Lines changed: 18 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -287,23 +287,32 @@ func BuildOutputPath(path string, format ExportFormat, comp Compression) string
287287

288288
// IsInputOnlyExtension reports whether a destination path targets an input-only
289289
// format that sqly can read but not write: ACH (.ach) and Fedwire (.fed), which
290-
// require multi-record coordination the export path cannot produce. The
291-
// compression suffix is stripped first, so .ach.gz and .fed.gz are detected too.
292-
// It lets --output and .dump reject these destinations instead of silently
293-
// writing CSV bytes to a misleading path. Ref #421, #422.
290+
// require multi-record coordination the export path cannot produce. All trailing
291+
// compression suffixes are stripped first, so a path that hides the extension
292+
// behind several codecs (".ach.gz.zst", ".fed.gz.zst") is detected too. It lets
293+
// --output and .dump reject these destinations instead of silently writing CSV
294+
// bytes to a misleading path. Ref #421, #422, #459, #460.
294295
func IsInputOnlyExtension(path string) bool {
295-
base := path
296-
if _, ok := CompressionFromExtension(filepath.Ext(path)); ok {
297-
base = strings.TrimSuffix(path, filepath.Ext(path))
298-
}
299-
switch strings.ToLower(filepath.Ext(base)) {
296+
switch strings.ToLower(filepath.Ext(stripCompressionSuffixes(path))) {
300297
case ".ach", ".fed":
301298
return true
302299
default:
303300
return false
304301
}
305302
}
306303

304+
// stripCompressionSuffixes removes every trailing compression extension from a
305+
// path (e.g. "out.ach.gz.zst" -> "out.ach"), so a check on the remaining base
306+
// extension is not fooled by stacked codecs. Ref #459, #460.
307+
func stripCompressionSuffixes(path string) string {
308+
for {
309+
if _, ok := CompressionFromExtension(filepath.Ext(path)); !ok {
310+
return path
311+
}
312+
path = strings.TrimSuffix(path, filepath.Ext(path))
313+
}
314+
}
315+
307316
// ExportFormatFromPrintMode converts a PrintMode to an ExportFormat.
308317
// PrintModeTable falls back to ExportCSV since table format is display-only.
309318
func ExportFormatFromPrintMode(m PrintMode) ExportFormat {

domain/model/export_test.go

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -379,8 +379,14 @@ func TestIsInputOnlyExtension(t *testing.T) {
379379
{"out.ACH", true},
380380
{"out.ach.gz", true},
381381
{"out.fed.zst", true},
382+
// Multiple stacked compression suffixes must still be seen through. Ref
383+
// #459, #460.
384+
{"out.ach.gz.zst", true},
385+
{"out.fed.gz.zst", true},
386+
{"out.ach.gz.gz.gz", true},
382387
{"out.csv", false},
383388
{"out.csv.gz", false},
389+
{"out.csv.gz.zst", false},
384390
{"out.parquet", false},
385391
{"out", false},
386392
}

domain/model/table.go

Lines changed: 49 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -413,10 +413,13 @@ func (t *Table) writeDelimited(out io.Writer, comma rune) error {
413413
// up front instead of emitting output that no longer round-trips as LTSV. Ref
414414
// #382, #383.
415415
func (t *Table) printLTSV(out io.Writer) error {
416+
if err := EnsureLTSVHeaderWritable(t.Header()); err != nil {
417+
return err
418+
}
416419
for _, v := range t.Records() {
417420
r := make(Record, 0, len(v))
418421
for i, data := range v {
419-
if err := ensureLTSVRepresentable(t.Header()[i], data); err != nil {
422+
if err := ensureLTSVValueRepresentable(t.Header()[i], data); err != nil {
420423
return err
421424
}
422425
r = append(r, t.Header()[i]+":"+data)
@@ -426,13 +429,52 @@ func (t *Table) printLTSV(out io.Writer) error {
426429
return nil
427430
}
428431

429-
// ensureLTSVRepresentable reports an error when a label or value contains a byte
430-
// LTSV cannot represent (tab or newline), so the caller rejects it before writing
431-
// output that cannot be re-imported as LTSV. Ref #382, #383.
432-
func ensureLTSVRepresentable(label, value string) error {
433-
if strings.ContainsAny(label, "\t\n\r") {
434-
return fmt.Errorf("ltsv: column name %q contains a tab or newline, which LTSV cannot represent", label)
432+
// isValidLTSVLabel reports whether label matches the LTSV label grammar
433+
// [0-9A-Za-z_.-]+ (https://ltsv.org). A label outside this set — empty, or
434+
// containing ':', a space, a tab, or any other character — cannot be written as a
435+
// distinct "label:value" field that re-imports to the same column, so the LTSV
436+
// writers reject it. Ref #465.
437+
func isValidLTSVLabel(label string) bool {
438+
if label == "" {
439+
return false
435440
}
441+
for _, r := range label {
442+
switch {
443+
case r >= '0' && r <= '9',
444+
r >= 'a' && r <= 'z',
445+
r >= 'A' && r <= 'Z',
446+
r == '_', r == '.', r == '-':
447+
default:
448+
return false
449+
}
450+
}
451+
return true
452+
}
453+
454+
// EnsureLTSVHeaderWritable validates a header for LTSV output: every column name
455+
// must be a valid LTSV label, and the names must be unique. LTSV encodes each
456+
// column as a "label:value" field with no escaping, so an invalid label (e.g.
457+
// "foo:bar") is ambiguous on re-import and a duplicate label silently keeps only
458+
// the last value. Rejecting both up front keeps LTSV output round-trippable.
459+
// Ref #465, #466.
460+
func EnsureLTSVHeaderWritable(header Header) error {
461+
seen := make(map[string]struct{}, len(header))
462+
for _, label := range header {
463+
if !isValidLTSVLabel(label) {
464+
return fmt.Errorf("ltsv: column name %q is not a valid LTSV label (allowed: letters, digits, '_', '.', '-')", label)
465+
}
466+
if _, ok := seen[label]; ok {
467+
return fmt.Errorf("ltsv: duplicate column name %q; LTSV labels must be unique or earlier values are lost on re-import", label)
468+
}
469+
seen[label] = struct{}{}
470+
}
471+
return nil
472+
}
473+
474+
// ensureLTSVValueRepresentable reports an error when a value contains a byte LTSV
475+
// cannot represent (tab or newline), so the caller rejects it before writing
476+
// output that cannot be re-imported as LTSV. Ref #382, #383.
477+
func ensureLTSVValueRepresentable(label, value string) error {
436478
if strings.ContainsAny(value, "\t\n\r") {
437479
return fmt.Errorf("ltsv: value for column %q contains a tab or newline, which LTSV cannot represent; use csv/tsv/json for such values", label)
438480
}

domain/model/table_test.go

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -870,3 +870,80 @@ func TestTablePrintEscaping(t *testing.T) {
870870
}
871871
})
872872
}
873+
874+
// TestEnsureLTSVHeaderWritable verifies that LTSV output rejects column names that
875+
// are not valid LTSV labels and rejects duplicate labels, so LTSV output stays
876+
// valid and round-trippable. Ref #465, #466.
877+
func TestEnsureLTSVHeaderWritable(t *testing.T) {
878+
t.Parallel()
879+
880+
valid := []Header{
881+
{"x"},
882+
{"user_name", "identifier"},
883+
{"a.b", "c-d", "e_f"},
884+
{"Col1", "Col2"},
885+
}
886+
for _, h := range valid {
887+
if err := EnsureLTSVHeaderWritable(h); err != nil {
888+
t.Errorf("EnsureLTSVHeaderWritable(%v) = %v, want nil", h, err)
889+
}
890+
}
891+
892+
invalid := []struct {
893+
name string
894+
header Header
895+
}{
896+
{"colon in label", Header{"foo:bar"}},
897+
{"space in label", Header{"foo bar"}},
898+
{"tab in label", Header{"foo\tbar"}},
899+
{"newline in label", Header{"foo\nbar"}},
900+
{"empty label", Header{""}},
901+
{"duplicate labels", Header{"x", "x"}},
902+
{"duplicate among valid", Header{"a", "b", "a"}},
903+
}
904+
for _, tt := range invalid {
905+
t.Run(tt.name, func(t *testing.T) {
906+
t.Parallel()
907+
if err := EnsureLTSVHeaderWritable(tt.header); err == nil {
908+
t.Errorf("EnsureLTSVHeaderWritable(%v) = nil, want an error", tt.header)
909+
}
910+
})
911+
}
912+
}
913+
914+
// TestTablePrintLTSV_RejectsInvalidLabels verifies that printing a table as LTSV
915+
// fails for an invalid or duplicate label rather than emitting ambiguous output.
916+
// Ref #465, #466.
917+
func TestTablePrintLTSV_RejectsInvalidLabels(t *testing.T) {
918+
t.Parallel()
919+
920+
tests := []struct {
921+
name string
922+
header Header
923+
}{
924+
{"label with colon", Header{"foo:bar"}},
925+
{"duplicate labels", Header{"x", "x"}},
926+
}
927+
for _, tt := range tests {
928+
t.Run(tt.name, func(t *testing.T) {
929+
t.Parallel()
930+
tbl := NewTable("t", tt.header, []Record{make(Record, len(tt.header))})
931+
var buf bytes.Buffer
932+
if err := tbl.Print(&buf, PrintModeLTSV); err == nil {
933+
t.Errorf("Print(LTSV) for header %v = nil error, want rejection; output=%q", tt.header, buf.String())
934+
}
935+
})
936+
}
937+
938+
t.Run("valid labels still print", func(t *testing.T) {
939+
t.Parallel()
940+
tbl := NewTable("t", Header{"a", "b"}, []Record{{"1", "2"}})
941+
var buf bytes.Buffer
942+
if err := tbl.Print(&buf, PrintModeLTSV); err != nil {
943+
t.Fatalf("Print(LTSV) for a valid header returned error: %v", err)
944+
}
945+
if got := buf.String(); got != "a:1\tb:2\n" {
946+
t.Errorf("Print(LTSV) = %q, want %q", got, "a:1\tb:2\n")
947+
}
948+
})
949+
}

domain/repository/sqlite.go

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,10 +3,17 @@ package repository
33

44
import (
55
"context"
6+
"errors"
67

78
"github.com/nao1215/sqly/domain/model"
89
)
910

11+
// ErrNoRows is returned by SQLite3Repository.Query when a statement produced no
12+
// result columns (for example a setter or command PRAGMA routed through the query
13+
// path). It is part of the Query contract so callers in any layer can detect the
14+
// no-rowset case with errors.Is without depending on the infrastructure layer.
15+
var ErrNoRows = errors.New("execute query, however return no records")
16+
1017
//go:generate mockgen -typed -source=$GOFILE -destination=../../infrastructure/mock/$GOFILE -package mock
1118

1219
// SQLite3Repository is a repository that handles SQLite3.
@@ -15,6 +22,9 @@ type SQLite3Repository interface {
1522
CreateTable(ctx context.Context, t *model.Table) error
1623
// TablesName return all table name.
1724
TablesName(ctx context.Context) ([]*model.Table, error)
25+
// SchemaObjects returns every queryable table and view in the session,
26+
// including TEMP tables and views, for enumeration by .tables.
27+
SchemaObjects(ctx context.Context) ([]*model.Table, error)
1828
// Insert set records in DB
1929
Insert(ctx context.Context, t *model.Table) error
2030
// List get records in the specified table

go.mod

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ require (
99
github.com/google/go-cmp v0.7.0
1010
github.com/google/wire v0.7.0
1111
github.com/mattn/go-colorable v0.1.15
12-
github.com/nao1215/filesql v0.13.0
12+
github.com/nao1215/filesql v0.14.0
1313
github.com/nao1215/prompt v0.0.5
1414
github.com/olekukonko/tablewriter v1.1.4
1515
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2
@@ -46,12 +46,12 @@ require (
4646
github.com/mattn/go-tty v0.0.8 // indirect
4747
github.com/minio/asm2plan9s v0.0.0-20200509001527-cdd76441f9d8 // indirect
4848
github.com/minio/c2goasm v0.0.0-20190812172519-36a3d3bbc4f3 // indirect
49-
github.com/moov-io/ach v1.60.1 // indirect
50-
github.com/moov-io/base v0.61.1 // indirect
49+
github.com/moov-io/ach v1.61.0 // indirect
50+
github.com/moov-io/base v0.61.2 // indirect
5151
github.com/moov-io/iso3166 v0.4.0 // indirect
5252
github.com/moov-io/iso4217 v0.3.2 // indirect
5353
github.com/moov-io/wire v0.15.7 // indirect
54-
github.com/nao1215/fileparser v0.5.1 // indirect
54+
github.com/nao1215/fileparser v0.5.2 // indirect
5555
github.com/ncruces/go-strftime v1.0.0 // indirect
5656
github.com/olekukonko/cat v0.0.0-20250911104152-50322a0618f6 // indirect
5757
github.com/olekukonko/errors v1.3.0 // indirect
@@ -71,13 +71,13 @@ require (
7171
golang.org/x/mod v0.35.0 // indirect
7272
golang.org/x/net v0.53.0 // indirect
7373
golang.org/x/sync v0.20.0 // indirect
74-
golang.org/x/sys v0.43.0 // indirect
74+
golang.org/x/sys v0.44.0 // indirect
7575
golang.org/x/telemetry v0.0.0-20260428171046-76f71b9afea0 // indirect
76-
golang.org/x/text v0.36.0 // indirect
76+
golang.org/x/text v0.37.0 // indirect
7777
golang.org/x/tools v0.44.0 // indirect
7878
golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da // indirect
79-
google.golang.org/genproto/googleapis/rpc v0.0.0-20260427160629-7cedc36a6bc4 // indirect
80-
google.golang.org/grpc v1.80.0 // indirect
79+
google.golang.org/genproto/googleapis/rpc v0.0.0-20260504160031-60b97b32f348 // indirect
80+
google.golang.org/grpc v1.81.0 // indirect
8181
google.golang.org/protobuf v1.36.11 // indirect
8282
gopkg.in/yaml.v3 v3.0.1 // indirect
8383
modernc.org/libc v1.72.3 // indirect

0 commit comments

Comments
 (0)