Compare commits

..

10 Commits

Author SHA1 Message Date
Cursor Agent 80189ba449 fix(unstable): correct Parser.Range for non-suffix highlights
Parser.Range used len(data)-len(highlight), which only matches suffix
slices. Single-byte highlights like b[0:1] are subslices of the
remaining buffer, so the wrong offset pointed at the end of the
document (issue #1047). Use pointer-based subslice offset like
wrapDecodeError.

Add regression tests for unstable.Parser and toml.Unmarshal error
positions and human-readable output.

Co-authored-by: Thomas Pelletier <thomas@pelletier.dev>
2026-04-12 12:26:03 +00:00
Thomas Pelletier f36a3ece9e Reduce marshal and unmarshal overhead (#1044)
* Reduce marshal and unmarshal overhead

Targeted optimizations to reduce performance overhead introduced by
recent feature additions and the unsafe removal.

Unmarshal:
- parseKeyval: access the node directly in the builder's slice to set
  Raw, bypassing NodeAt which triggers a GC write barrier for the
  nodes-pointer on every key-value expression.
- Iterator.Next: cache the *nodes slice dereference in a local variable
  to avoid repeated pointer-to-slice indirection in the hot loop.

Marshal:
- Guard shouldOmitZero calls with an inlineable options.omitzero check.
  shouldOmitZero has inlining cost 1145 (budget 80), so avoiding the
  function call when omitzero is not set removes per-field overhead.
- Inline the isNil check in encodeMap. isNil has inlining cost 93
  (budget 80), so expanding it at the single hot call site avoids
  per-map-entry function call overhead.

Update README benchmarks.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 11:08:39 +00:00
Thomas Pelletier 77f3862df4 Fix benchmark script replacing internal package imports (#1042)
* Fix benchmark script replacing internal package imports

The sed command in bench() was replacing all occurrences of the go-toml
module path, including sub-package imports like internal/assert. This
caused the BurntSushi/toml benchmark to fail because it tried to import
github.com/BurntSushi/toml/internal/assert which doesn't exist.

Fix by anchoring the sed pattern to only match the import path when
followed by a closing quote, preserving internal package imports.

Also add a guard in the benchstathtml Python script to give a clear
error instead of an IndexError when no benchmark results are available.

https://claude.ai/code/session_016JGASo49PeFSfCaDxvrGFE

* Update benchmark results in README

https://claude.ai/code/session_016JGASo49PeFSfCaDxvrGFE

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-03-23 22:00:18 -04:00
Thomas Pelletier 16b1ef5508 Fix parser error pointing to wrong line when last line has no trailing newline (#1041)
When parsing a key without '=' at EOF (e.g., "a = 1\nb = 2\nc"), the
error highlight was an empty slice, causing subsliceOffset to return 0
and the error to point at line 1 instead of line 3. Pass the consumed
key bytes as the highlight instead of the empty remainder.

Fixes #1032

https://claude.ai/code/session_01UWv8pyc8P1ktAPfHpveixj

Co-authored-by: Claude <noreply@anthropic.com>
2026-03-23 21:34:12 -04:00
dependabot[bot] e14bde7c1d build(deps): bump docker/login-action from 3 to 4 (#1039)
Bumps [docker/login-action](https://github.com/docker/login-action) from 3 to 4.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/v3...v4)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-version: '4'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-23 21:04:21 -04:00
dependabot[bot] 4b1ff01eb3 build(deps): bump docker/setup-buildx-action from 3 to 4 (#1040)
Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 3 to 4.
- [Release notes](https://github.com/docker/setup-buildx-action/releases)
- [Commits](https://github.com/docker/setup-buildx-action/compare/v3...v4)

---
updated-dependencies:
- dependency-name: docker/setup-buildx-action
  dependency-version: '4'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-23 21:04:11 -04:00
Thomas Pelletier 048a25f0f2 Go 1.26 (#1030)
* ci(release): drop unsupported windows/arm targets
2026-03-03 01:29:35 -05:00
dependabot[bot] b3575580f9 build(deps): bump goreleaser/goreleaser-action from 6 to 7 (#1035) 2026-03-03 00:47:47 -05:00
dependabot[bot] a0be52f4c1 build(deps): bump actions/upload-artifact from 6 to 7 (#1036) 2026-03-03 00:47:35 -05:00
Thomas Pelletier 316bfc66a4 Support Unmarshaler interface for tables and array tables (#1027)
Fixes #873

Extend the unstable.Unmarshaler interface support to work with tables
and array tables, not just single values.

When a type implementing unstable.Unmarshaler is the target of a table
(e.g., [table] or [[array]]), the UnmarshalTOML method receives a
synthetic InlineTable node containing all the key-value pairs belonging
to that table.

Key changes:
- Add handleKeyValuesUnmarshaler to collect and process table content
- Add copyExpressionNodes to deep-copy AST nodes for synthetic tables
- Add helper functions in unstable/ast.go for node manipulation
- Update documentation for EnableUnmarshalerInterface
- Add comprehensive tests for table and array table unmarshaling

* Implement bytes-based Unmarshaler interface for tables and arrays (#873)

This change brings back support for the unstable.Unmarshaler interface
for tables and array tables, addressing issue #873.

Key changes:
- Changed UnmarshalTOML signature from (*Node) to ([]byte) to provide
  raw TOML bytes instead of AST nodes
- Added RawMessage type (similar to json.RawMessage) for capturing raw
  TOML bytes for later processing
- Updated handleKeyValuesUnmarshaler to reconstruct key-value lines
  from the parsed keys and raw value bytes
- Added support for slice types implementing Unmarshaler (e.g., RawMessage)
- Removed unused AST helper functions from unstable/ast.go

The bytes-based interface allows users to:
- Get raw TOML bytes for custom parsing
- Delay TOML decoding using RawMessage
- Implement custom unmarshaling logic for complex types

Tests added for:
- Table unmarshaler with various scenarios
- Array table unmarshaler
- Split tables (same parent defined in multiple places)
- RawMessage usage
- Nested tables and mixed regular fields

* Fix lint issues and improve test coverage for Unmarshaler interface

- Apply De Morgan's law in keyNeedsQuoting to satisfy staticcheck QF1001
- Remove unused splitTableUnmarshaler type from test
- Fix unused parameter lint warning in errorUnmarshaler873
- Add test for quoted keys that need special handling
- Add test for error propagation from UnmarshalTOML
- Update customTable873 parser to handle quoted keys properly

Coverage improved:
- handleKeyValuesUnmarshaler: 80.0% -> 93.3%
- keyNeedsQuoting: 66.7% -> 83.3%
- Overall main package: 97.2% -> 97.5%

* Add test for dotted keys to improve coverage

Add TestIssue873_DottedKeys to test dotted key handling (e.g., sub.key = value)
in the Unmarshaler interface. This improves coverage for handleKeyValuesUnmarshaler
from 93.3% to 96.7%.

* Add double pointer test to achieve 100% coverage for handleKeyValues

Add TestIssue873_DoublePointerUnmarshaler to test pointer-to-pointer
to Unmarshaler types. This covers the pointer dereferencing loop in
handleKeyValues, bringing its coverage from 88% to 100%.

Total coverage: 97.4%

* Add Example tests and fix raw value extraction for boolean types

Add two godoc Example tests:
- ExampleDecoder_EnableUnmarshalerInterface_dynamicConfig: shows dynamic
  unmarshaling based on a type field
- ExampleDecoder_EnableUnmarshalerInterface_rawMessage: demonstrates
  RawMessage usage for deferred parsing

Fix handleKeyValuesUnmarshaler to handle values where Raw.Length == 0
(like boolean types) by using value.Data as fallback.

* Preserve original formatting in Unmarshaler by using raw byte ranges

Instead of reconstructing key-value lines from parsed components, now
uses the original raw bytes from the document. This preserves:
- Whitespace around '=' (e.g., "key   =   value")
- String quoting style (basic vs literal)
- Number formats (hex, octal, binary)
- Inline table formatting

Changes:
- Add Raw range tracking to KeyValue expressions in parseKeyval
- Update handleKeyValuesUnmarshaler to use expr.Raw directly
- Remove keyNeedsQuoting helper (no longer needed)
- Add TestIssue873_FormattingPreservation test
- Update expected output in ExampleParser_comments

* Prevent test matrix from canceling on first failure

Add fail-fast: false to the test workflow strategy so that all
OS/Go version combinations continue running even if one fails.
This provides better visibility into which specific combinations
have issues.

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-02-11 09:57:23 -05:00
23 changed files with 1295 additions and 2050 deletions
+1 -1
View File
@@ -19,7 +19,7 @@ jobs:
dry-run: false
language: go
- name: Upload Crash
uses: actions/upload-artifact@v6
uses: actions/upload-artifact@v7
if: failure() && steps.build.outcome == 'success'
with:
name: artifacts
+1 -1
View File
@@ -15,6 +15,6 @@ jobs:
- name: Setup go
uses: actions/setup-go@v6
with:
go-version: "1.25"
go-version: "1.26"
- name: Run tests with coverage
run: ./ci.sh coverage -d "${GITHUB_BASE_REF-HEAD}"
+2 -2
View File
@@ -20,7 +20,7 @@ jobs:
fetch-depth: 0
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
uses: docker/setup-buildx-action@v4
- name: Run Go versions compatibility test
run: |
@@ -28,7 +28,7 @@ jobs:
./test-go-versions.sh --output ./test-results $VERSIONS
- name: Upload test results
uses: actions/upload-artifact@v6
uses: actions/upload-artifact@v7
with:
name: go-versions-test-results
path: |
+1 -1
View File
@@ -15,7 +15,7 @@ jobs:
- name: Setup go
uses: actions/setup-go@v6
with:
go-version: "1.24"
go-version: "1.26"
- name: Run golangci-lint
uses: golangci/golangci-lint-action@v9
with:
+3 -3
View File
@@ -22,15 +22,15 @@ jobs:
- name: Set up Go
uses: actions/setup-go@v6
with:
go-version: "1.25"
go-version: "1.26"
- name: Login to GitHub Container Registry
uses: docker/login-action@v3
uses: docker/login-action@v4
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Run GoReleaser
uses: goreleaser/goreleaser-action@v6
uses: goreleaser/goreleaser-action@v7
with:
distribution: goreleaser
version: '~> v2'
+2 -1
View File
@@ -10,9 +10,10 @@ on:
jobs:
build:
strategy:
fail-fast: false
matrix:
os: [ 'ubuntu-latest', 'windows-latest', 'macos-latest', 'macos-14' ]
go: [ '1.24', '1.25' ]
go: [ '1.25', '1.26' ]
runs-on: ${{ matrix.os }}
name: ${{ matrix.go }}/${{ matrix.os }}
steps:
-3
View File
@@ -22,7 +22,6 @@ builds:
- linux_riscv64
- windows_amd64
- windows_arm64
- windows_arm
- darwin_amd64
- darwin_arm64
- id: tomljson
@@ -42,7 +41,6 @@ builds:
- linux_riscv64
- windows_amd64
- windows_arm64
- windows_arm
- darwin_amd64
- darwin_arm64
- id: jsontoml
@@ -62,7 +60,6 @@ builds:
- linux_arm
- windows_amd64
- windows_arm64
- windows_arm
- darwin_amd64
- darwin_arm64
universal_binaries:
+29 -29
View File
@@ -2,7 +2,7 @@
Go library for the [TOML](https://toml.io/en/) format.
This library supports [TOML v1.1.0](https://toml.io/en/v1.1.0).
This library supports [TOML v1.0.0](https://toml.io/en/v1.0.0).
[🐞 Bug Reports](https://github.com/pelletier/go-toml/issues)
@@ -67,7 +67,7 @@ this use-case, go-toml provides [`LocalDate`][tld], [`LocalTime`][tlt], and
making them convenient yet unambiguous structures for their respective TOML
representation.
[ldt]: https://toml.io/en/v1.1.0#local-date-time
[ldt]: https://toml.io/en/v1.0.0#local-date-time
[tld]: https://pkg.go.dev/github.com/pelletier/go-toml/v2#LocalDate
[tlt]: https://pkg.go.dev/github.com/pelletier/go-toml/v2#LocalTime
[tldt]: https://pkg.go.dev/github.com/pelletier/go-toml/v2#LocalDateTime
@@ -235,17 +235,17 @@ the AST level. See https://pkg.go.dev/github.com/pelletier/go-toml/v2/unstable.
Execution time speedup compared to other Go TOML libraries:
<table>
<thead>
<tr><th>Benchmark</th><th>go-toml v1</th><th>BurntSushi/toml</th></tr>
</thead>
<tbody>
<tr><td>Marshal/HugoFrontMatter-2</td><td>1.9x</td><td>2.2x</td></tr>
<tr><td>Marshal/ReferenceFile/map-2</td><td>1.7x</td><td>2.1x</td></tr>
<tr><td>Marshal/ReferenceFile/struct-2</td><td>2.2x</td><td>3.0x</td></tr>
<tr><td>Unmarshal/HugoFrontMatter-2</td><td>2.9x</td><td>2.7x</td></tr>
<tr><td>Unmarshal/ReferenceFile/map-2</td><td>2.6x</td><td>2.7x</td></tr>
<tr><td>Unmarshal/ReferenceFile/struct-2</td><td>4.6x</td><td>5.1x</td></tr>
</tbody>
<thead>
<tr><th>Benchmark</th><th>go-toml v1</th><th>BurntSushi/toml</th></tr>
</thead>
<tbody>
<tr><td>Marshal/HugoFrontMatter-2</td><td>2.1x</td><td>2.0x</td></tr>
<tr><td>Marshal/ReferenceFile/map-2</td><td>2.0x</td><td>2.0x</td></tr>
<tr><td>Marshal/ReferenceFile/struct-2</td><td>2.3x</td><td>2.5x</td></tr>
<tr><td>Unmarshal/HugoFrontMatter-2</td><td>3.3x</td><td>2.8x</td></tr>
<tr><td>Unmarshal/ReferenceFile/map-2</td><td>2.9x</td><td>3.0x</td></tr>
<tr><td>Unmarshal/ReferenceFile/struct-2</td><td>4.8x</td><td>5.0x</td></tr>
</tbody>
</table>
<details><summary>See more</summary>
<p>The table above has the results of the most common use-cases. The table below
@@ -253,22 +253,22 @@ contains the results of all benchmarks, including unrealistic ones. It is
provided for completeness.</p>
<table>
<thead>
<tr><th>Benchmark</th><th>go-toml v1</th><th>BurntSushi/toml</th></tr>
</thead>
<tbody>
<tr><td>Marshal/SimpleDocument/map-2</td><td>1.8x</td><td>2.7x</td></tr>
<tr><td>Marshal/SimpleDocument/struct-2</td><td>2.7x</td><td>3.8x</td></tr>
<tr><td>Unmarshal/SimpleDocument/map-2</td><td>3.8x</td><td>3.0x</td></tr>
<tr><td>Unmarshal/SimpleDocument/struct-2</td><td>5.6x</td><td>4.1x</td></tr>
<tr><td>UnmarshalDataset/example-2</td><td>3.0x</td><td>3.2x</td></tr>
<tr><td>UnmarshalDataset/code-2</td><td>2.3x</td><td>2.9x</td></tr>
<tr><td>UnmarshalDataset/twitter-2</td><td>2.6x</td><td>2.7x</td></tr>
<tr><td>UnmarshalDataset/citm_catalog-2</td><td>2.2x</td><td>2.3x</td></tr>
<tr><td>UnmarshalDataset/canada-2</td><td>1.8x</td><td>1.5x</td></tr>
<tr><td>UnmarshalDataset/config-2</td><td>4.1x</td><td>2.9x</td></tr>
<tr><td>geomean</td><td>2.7x</td><td>2.8x</td></tr>
</tbody>
<thead>
<tr><th>Benchmark</th><th>go-toml v1</th><th>BurntSushi/toml</th></tr>
</thead>
<tbody>
<tr><td>Marshal/SimpleDocument/map-2</td><td>2.0x</td><td>2.9x</td></tr>
<tr><td>Marshal/SimpleDocument/struct-2</td><td>2.5x</td><td>3.6x</td></tr>
<tr><td>Unmarshal/SimpleDocument/map-2</td><td>4.2x</td><td>3.4x</td></tr>
<tr><td>Unmarshal/SimpleDocument/struct-2</td><td>5.9x</td><td>4.4x</td></tr>
<tr><td>UnmarshalDataset/example-2</td><td>3.2x</td><td>2.9x</td></tr>
<tr><td>UnmarshalDataset/code-2</td><td>2.4x</td><td>2.8x</td></tr>
<tr><td>UnmarshalDataset/twitter-2</td><td>2.7x</td><td>2.5x</td></tr>
<tr><td>UnmarshalDataset/citm_catalog-2</td><td>2.3x</td><td>2.3x</td></tr>
<tr><td>UnmarshalDataset/canada-2</td><td>1.9x</td><td>1.5x</td></tr>
<tr><td>UnmarshalDataset/config-2</td><td>5.4x</td><td>3.0x</td></tr>
<tr><td>geomean</td><td>2.9x</td><td>2.8x</td></tr>
</tbody>
</table>
<p>This table can be generated with <code>./ci.sh benchmark -a -html</code>.</p>
</details>
+6 -1
View File
@@ -147,7 +147,7 @@ bench() {
pushd "$dir"
if [ "${replace}" != "" ]; then
find ./benchmark/ -iname '*.go' -exec sed -i -E "s|github.com/pelletier/go-toml/v2|${replace}|g" {} \;
find ./benchmark/ -iname '*.go' -exec sed -i -E "s|github.com/pelletier/go-toml/v2\"|${replace}\"|g" {} \;
go get "${replace}"
fi
@@ -195,6 +195,11 @@ for line in reversed(lines[2:]):
"%.1fx" % (float(line[3])/v2), # v1
"%.1fx" % (float(line[7])/v2), # bs
])
if not results:
print("No benchmark results to display.", file=sys.stderr)
sys.exit(1)
# move geomean to the end
results.append(results[0])
del results[0]
+18 -23
View File
@@ -162,7 +162,7 @@ func parseLocalDateTime(b []byte) (LocalDateTime, []byte, error) {
const localDateTimeByteMinLen = 11
if len(b) < localDateTimeByteMinLen {
return dt, nil, unstable.NewParserError(b, "local datetimes are expected to have the format YYYY-MM-DDTHH:MM[:SS[.NNNNNNNNN]]")
return dt, nil, unstable.NewParserError(b, "local datetimes are expected to have the format YYYY-MM-DDTHH:MM:SS[.NNNNNNNNN]")
}
date, err := parseLocalDate(b[:10])
@@ -194,10 +194,10 @@ func parseLocalTime(b []byte) (LocalTime, []byte, error) {
t LocalTime
)
// check if b matches to have expected format HH:MM[:SS[.NNNNNN]]
const localTimeByteMinLen = 5
if len(b) < localTimeByteMinLen {
return t, nil, unstable.NewParserError(b, "times are expected to have the format HH:MM[:SS[.NNNNNN]]")
// check if b matches to have expected format HH:MM:SS[.NNNNNN]
const localTimeByteLen = 8
if len(b) < localTimeByteLen {
return t, nil, unstable.NewParserError(b, "times are expected to have the format HH:MM:SS[.NNNNNN]")
}
var err error
@@ -221,26 +221,21 @@ func parseLocalTime(b []byte) (LocalTime, []byte, error) {
if t.Minute > 59 {
return t, nil, unstable.NewParserError(b[3:5], "minutes cannot be greater 59")
}
b = b[5:]
if len(b) >= 1 && b[0] == ':' {
if len(b) < 3 {
return t, nil, unstable.NewParserError(b, "incomplete seconds")
}
t.Second, err = parseDecimalDigits(b[1:3])
if err != nil {
return t, nil, err
}
if t.Second > 59 {
return t, nil, unstable.NewParserError(b[1:3], "seconds cannot be greater than 59")
}
b = b[3:]
if b[5] != ':' {
return t, nil, unstable.NewParserError(b[5:6], "expecting colon between minutes and seconds")
}
t.Second, err = parseDecimalDigits(b[6:8])
if err != nil {
return t, nil, err
}
if t.Second > 59 {
return t, nil, unstable.NewParserError(b[6:8], "seconds cannot be greater than 59")
}
b = b[8:]
if len(b) >= 1 && b[0] == '.' {
frac := 0
precision := 0
+38
View File
@@ -202,6 +202,38 @@ func TestDecodeError_Accessors(t *testing.T) {
assert.Equal(t, "bar", e.String())
}
func TestDecodeError_InvalidKeyStartAfterComment(t *testing.T) {
// Regression for https://github.com/pelletier/go-toml/issues/1047: the "="
// that starts an invalid keyval must be reported on line 2, column 1, with
// the human-readable context pointing at that byte (not the document end).
doc := "# comment\n= \"value\""
var v map[string]any
err := Unmarshal([]byte(doc), &v)
if err == nil {
t.Fatal("expected an error")
}
var derr *DecodeError
if !errors.As(err, &derr) {
t.Fatalf("expected *DecodeError, got %T", err)
}
row, col := derr.Position()
if row != 2 || col != 1 {
t.Errorf("Position(): got row %d col %d, want row 2 col 1", row, col)
}
human := derr.String()
if !strings.Contains(human, `2| = "value"`) {
t.Errorf("human output should show the error line; got:\n%s", human)
}
// Caret line uses line-number column width padding; only the "| ~" part is stable here.
if !strings.Contains(human, "| ~ invalid character at start of key") {
t.Errorf("human output should underline '=' and include the parser message; got:\n%s", human)
}
}
func TestDecodeError_DuplicateContent(t *testing.T) {
// This test verifies that when the same content appears multiple times
// in the document, the error correctly points to the actual location
@@ -259,6 +291,12 @@ func TestDecodeError_Position(t *testing.T) {
expectedRow: 3,
minCol: 5,
},
{
name: "missing equals on last line without trailing newline",
doc: "a = 1\nb = 2\nc",
expectedRow: 3,
minCol: 1,
},
}
for _, e := range examples {
-7
View File
@@ -67,13 +67,6 @@ func TestLocalTime_UnmarshalMarshalText(t *testing.T) {
assert.Error(t, err)
}
func TestLocalTime_UnmarshalText_WithoutSeconds(t *testing.T) {
d := toml.LocalTime{}
err := d.UnmarshalText([]byte("14:15"))
assert.NoError(t, err)
assert.Equal(t, toml.LocalTime{14, 15, 0, 0, 0}, d)
}
func TestLocalTime_RoundTrip(t *testing.T) {
var d struct{ A toml.LocalTime }
err := toml.Unmarshal([]byte("a=20:12:01.500"), &d)
+12 -9
View File
@@ -704,15 +704,18 @@ func (enc *Encoder) encodeMap(b []byte, ctx encoderCtx, v reflect.Value) ([]byte
for iter.Next() {
v := iter.Value()
if isNil(v) {
// For nil pointers, convert to zero value of the element type.
// This allows round-trip marshaling of maps with nil pointer values.
// For nil interfaces and nil maps, skip since we can't derive a type.
if v.Kind() == reflect.Ptr {
// Handle nil values: convert nil pointers to zero value,
// skip nil interfaces and nil maps.
switch v.Kind() {
case reflect.Ptr:
if v.IsNil() {
v = reflect.Zero(v.Type().Elem())
} else {
}
case reflect.Interface, reflect.Map:
if v.IsNil() {
continue
}
default:
}
k, err := enc.keyToString(iter.Key())
@@ -936,7 +939,7 @@ func (enc *Encoder) encodeTable(b []byte, ctx encoderCtx, t table) ([]byte, erro
if shouldOmitEmpty(kv.Options, kv.Value) {
continue
}
if shouldOmitZero(kv.Options, kv.Value) {
if kv.Options.omitzero && shouldOmitZero(kv.Options, kv.Value) {
continue
}
hasNonEmptyKV = true
@@ -958,7 +961,7 @@ func (enc *Encoder) encodeTable(b []byte, ctx encoderCtx, t table) ([]byte, erro
if shouldOmitEmpty(table.Options, table.Value) {
continue
}
if shouldOmitZero(table.Options, table.Value) {
if table.Options.omitzero && shouldOmitZero(table.Options, table.Value) {
continue
}
if first {
@@ -995,7 +998,7 @@ func (enc *Encoder) encodeTableInline(b []byte, ctx encoderCtx, t table) ([]byte
if shouldOmitEmpty(kv.Options, kv.Value) {
continue
}
if shouldOmitZero(kv.Options, kv.Value) {
if kv.Options.omitzero && shouldOmitZero(kv.Options, kv.Value) {
continue
}
+5 -4
View File
@@ -9,7 +9,7 @@ YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color
# Go versions to test (1.11 through 1.25)
# Go versions to test (1.11 through 1.26)
GO_VERSIONS=(
"1.11"
"1.12"
@@ -26,6 +26,7 @@ GO_VERSIONS=(
"1.23"
"1.24"
"1.25"
"1.26"
)
# Default values
@@ -64,7 +65,7 @@ EXAMPLES:
$0 # Test all Go versions in parallel
$0 --sequential # Test all Go versions sequentially
$0 1.21 1.22 1.23 # Test specific versions
$0 --verbose --output ./results 1.24 1.25 # Verbose output to custom directory
$0 --verbose --output ./results 1.25 1.26 # Verbose output to custom directory
EXIT CODES:
0 Recent Go versions pass (good compatibility)
@@ -136,8 +137,8 @@ fi
# Validate Go versions
for version in "${GO_VERSIONS[@]}"; do
if ! [[ "$version" =~ ^1\.(1[1-9]|2[0-5])$ ]]; then
log_error "Invalid Go version: $version. Supported versions: 1.11-1.25"
if ! [[ "$version" =~ ^1\.(1[1-9]|2[0-6])$ ]]; then
log_error "Invalid Go version: $version. Supported versions: 1.11-1.26"
exit 1
fi
done
+2 -2
View File
@@ -1,5 +1,5 @@
//go:generate go run github.com/toml-lang/toml-test/v2/cmd/toml-test@v2.1.0 copy -toml 1.1 ./tests
//go:generate go run ./cmd/tomltestgen/main.go -r v2.1.0 -o toml_testgen_test.go
//go:generate go run github.com/toml-lang/toml-test/cmd/toml-test@v1.6.0 -copy ./tests
//go:generate go run ./cmd/tomltestgen/main.go -r v1.6.0 -o toml_testgen_test.go
package toml_test
+615 -1234
View File
File diff suppressed because it is too large Load Diff
+85 -15
View File
@@ -56,13 +56,18 @@ func (d *Decoder) DisallowUnknownFields() *Decoder {
// EnableUnmarshalerInterface allows to enable unmarshaler interface.
//
// With this feature enabled, types implementing the unstable/Unmarshaler
// With this feature enabled, types implementing the unstable.Unmarshaler
// interface can be decoded from any structure of the document. It allows types
// that don't have a straightforward TOML representation to provide their own
// decoding logic.
//
// Currently, types can only decode from a single value. Tables and array tables
// are not supported.
// The UnmarshalTOML method receives raw TOML bytes:
// - For single values: the raw value bytes (e.g., `"hello"` for a string)
// - For tables: all key-value lines belonging to that table
// - For inline tables/arrays: the raw bytes of the inline structure
//
// The unstable.RawMessage type can be used to capture raw TOML bytes for
// later processing, similar to json.RawMessage.
//
// *Unstable:* This method does not follow the compatibility guarantees of
// semver. It can be changed or removed without a new major version being
@@ -599,18 +604,28 @@ func (d *decoder) handleArrayTablePart(key unstable.Iterator, v reflect.Value) (
// cannot handle it.
func (d *decoder) handleTable(key unstable.Iterator, v reflect.Value) (reflect.Value, error) {
if v.Kind() == reflect.Slice {
if v.Len() == 0 {
return reflect.Value{}, unstable.NewParserError(key.Node().Data, "cannot store a table in a slice")
// For non-empty slices, work with the last element
if v.Len() > 0 {
elem := v.Index(v.Len() - 1)
x, err := d.handleTable(key, elem)
if err != nil {
return reflect.Value{}, err
}
if x.IsValid() {
elem.Set(x)
}
return reflect.Value{}, nil
}
elem := v.Index(v.Len() - 1)
x, err := d.handleTable(key, elem)
if err != nil {
return reflect.Value{}, err
// Empty slice - check if it implements Unmarshaler (e.g., RawMessage)
// and we're at the end of the key path
if d.unmarshalerInterface && !key.Next() {
if v.CanAddr() && v.Addr().CanInterface() {
if outi, ok := v.Addr().Interface().(unstable.Unmarshaler); ok {
return d.handleKeyValuesUnmarshaler(outi)
}
}
}
if x.IsValid() {
elem.Set(x)
}
return reflect.Value{}, nil
return reflect.Value{}, unstable.NewParserError(key.Node().Data, "cannot store a table in a slice")
}
if key.Next() {
// Still scoping the key
@@ -624,6 +639,24 @@ func (d *decoder) handleTable(key unstable.Iterator, v reflect.Value) (reflect.V
// Handle root expressions until the end of the document or the next
// non-key-value.
func (d *decoder) handleKeyValues(v reflect.Value) (reflect.Value, error) {
// Check if target implements Unmarshaler before processing key-values.
// This allows types to handle entire tables themselves.
if d.unmarshalerInterface {
vv := v
for vv.Kind() == reflect.Ptr {
if vv.IsNil() {
vv.Set(reflect.New(vv.Type().Elem()))
}
vv = vv.Elem()
}
if vv.CanAddr() && vv.Addr().CanInterface() {
if outi, ok := vv.Addr().Interface().(unstable.Unmarshaler); ok {
// Collect all key-value expressions for this table
return d.handleKeyValuesUnmarshaler(outi)
}
}
}
var rv reflect.Value
for d.nextExpr() {
expr := d.expr()
@@ -653,6 +686,41 @@ func (d *decoder) handleKeyValues(v reflect.Value) (reflect.Value, error) {
return rv, nil
}
// handleKeyValuesUnmarshaler collects all key-value expressions for a table
// and passes them to the Unmarshaler as raw TOML bytes.
func (d *decoder) handleKeyValuesUnmarshaler(u unstable.Unmarshaler) (reflect.Value, error) {
// Collect raw bytes from all key-value expressions for this table.
// We use the Raw field on each KeyValue expression to preserve the
// original formatting (whitespace, quoting style, etc.) from the document.
var buf []byte
for d.nextExpr() {
expr := d.expr()
if expr.Kind != unstable.KeyValue {
d.stashExpr()
break
}
_, err := d.seen.CheckExpression(expr)
if err != nil {
return reflect.Value{}, err
}
// Use the raw bytes from the original document to preserve formatting
if expr.Raw.Length > 0 {
raw := d.p.Raw(expr.Raw)
buf = append(buf, raw...)
}
buf = append(buf, '\n')
}
if err := u.UnmarshalTOML(buf); err != nil {
return reflect.Value{}, err
}
return reflect.Value{}, nil
}
type (
handlerFn func(key unstable.Iterator, v reflect.Value) (reflect.Value, error)
valueMakerFn func() reflect.Value
@@ -697,7 +765,8 @@ func (d *decoder) handleValue(value *unstable.Node, v reflect.Value) error {
if d.unmarshalerInterface {
if v.CanAddr() && v.Addr().CanInterface() {
if outi, ok := v.Addr().Interface().(unstable.Unmarshaler); ok {
return outi.UnmarshalTOML(value)
// Pass raw bytes from the original document
return outi.UnmarshalTOML(d.p.Raw(value.Raw))
}
}
}
@@ -1201,7 +1270,8 @@ func (d *decoder) handleKeyValuePart(key unstable.Iterator, value *unstable.Node
if d.unmarshalerInterface {
if v.CanAddr() && v.Addr().CanInterface() {
if outi, ok := v.Addr().Interface().(unstable.Unmarshaler); ok {
return reflect.Value{}, outi.UnmarshalTOML(value)
// Pass raw bytes from the original document
return reflect.Value{}, outi.UnmarshalTOML(d.p.Raw(value.Raw))
}
}
}
+357 -446
View File
@@ -96,6 +96,132 @@ func ExampleUnmarshal() {
// tags: [go toml]
}
// pluginConfig demonstrates how to implement dynamic unmarshaling
// based on a "type" field. This pattern is useful for plugin systems
// or polymorphic configuration.
type pluginConfig struct {
Type string
Config any
}
func (p *pluginConfig) UnmarshalTOML(data []byte) error {
// First, decode just the type field
var typeOnly struct {
Type string `toml:"type"`
}
if err := toml.Unmarshal(data, &typeOnly); err != nil {
return err
}
p.Type = typeOnly.Type
// Now decode the config based on the type
switch typeOnly.Type {
case "database":
var cfg struct {
Type string `toml:"type"`
Host string `toml:"host"`
Port int `toml:"port"`
}
if err := toml.Unmarshal(data, &cfg); err != nil {
return err
}
p.Config = map[string]any{"host": cfg.Host, "port": cfg.Port}
case "cache":
var cfg struct {
Type string `toml:"type"`
TTL int `toml:"ttl"`
}
if err := toml.Unmarshal(data, &cfg); err != nil {
return err
}
p.Config = map[string]any{"ttl": cfg.TTL}
}
return nil
}
// This example demonstrates dynamic unmarshaling based on a discriminator
// field. The pluginConfig type uses UnmarshalTOML to first read the "type"
// field, then decode the rest of the configuration based on that type.
// This pattern is useful for plugin systems or configuration that varies
// by type.
func ExampleDecoder_EnableUnmarshalerInterface_dynamicConfig() {
doc := `
[[plugins]]
type = "database"
host = "localhost"
port = 5432
[[plugins]]
type = "cache"
ttl = 300
`
type Config struct {
Plugins []pluginConfig `toml:"plugins"`
}
var cfg Config
err := toml.NewDecoder(strings.NewReader(doc)).
EnableUnmarshalerInterface().
Decode(&cfg)
if err != nil {
panic(err)
}
for _, p := range cfg.Plugins {
fmt.Printf("type=%s config=%v\n", p.Type, p.Config)
}
// Output:
// type=database config=map[host:localhost port:5432]
// type=cache config=map[ttl:300]
}
// This example demonstrates using RawMessage to capture raw TOML bytes
// for later processing. RawMessage is similar to json.RawMessage - it
// delays decoding so you can inspect the raw content or decode it
// differently based on context.
func ExampleDecoder_EnableUnmarshalerInterface_rawMessage() {
doc := `
[plugin]
name = "example"
version = "1.0"
enabled = true
`
type Config struct {
Plugin unstable.RawMessage `toml:"plugin"`
}
var cfg Config
err := toml.NewDecoder(strings.NewReader(doc)).
EnableUnmarshalerInterface().
Decode(&cfg)
if err != nil {
panic(err)
}
// cfg.Plugin contains the raw TOML bytes
fmt.Printf("Raw TOML captured:\n%s", cfg.Plugin)
// You can later decode it into a specific type
var plugin struct {
Name string `toml:"name"`
Version string `toml:"version"`
Enabled bool `toml:"enabled"`
}
if err := toml.Unmarshal(cfg.Plugin, &plugin); err != nil {
panic(err)
}
fmt.Printf("Decoded: name=%s version=%s enabled=%v\n",
plugin.Name, plugin.Version, plugin.Enabled)
// Output:
// Raw TOML captured:
// name = "example"
// version = "1.0"
// enabled = true
// Decoded: name=example version=1.0 enabled=true
}
type badReader struct{}
func (r *badReader) Read([]byte) (int, error) {
@@ -600,96 +726,6 @@ foo = "bar"`,
}
},
},
{
desc: "local-time without seconds",
input: `a = 14:15`,
gen: func() test {
var v map[string]interface{}
return test{
target: &v,
expected: &map[string]interface{}{
"a": toml.LocalTime{Hour: 14, Minute: 15},
},
}
},
},
{
desc: "local-datetime without seconds using T",
input: `a = 2010-02-03T14:15`,
gen: func() test {
var v map[string]interface{}
return test{
target: &v,
expected: &map[string]interface{}{
"a": toml.LocalDateTime{
LocalDate: toml.LocalDate{2010, 2, 3},
LocalTime: toml.LocalTime{Hour: 14, Minute: 15},
},
},
}
},
},
{
desc: "local-datetime without seconds using space",
input: `a = 2010-02-03 14:15`,
gen: func() test {
var v map[string]interface{}
return test{
target: &v,
expected: &map[string]interface{}{
"a": toml.LocalDateTime{
LocalDate: toml.LocalDate{2010, 2, 3},
LocalTime: toml.LocalTime{Hour: 14, Minute: 15},
},
},
}
},
},
{
desc: "datetime without seconds with Z",
input: `a = 2010-02-03T14:15Z`,
gen: func() test {
var v map[string]time.Time
return test{
target: &v,
expected: &map[string]time.Time{
"a": time.Date(2010, 2, 3, 14, 15, 0, 0, time.UTC),
},
}
},
},
{
desc: "datetime without seconds with offset",
input: `a = 2010-02-03T14:15+05:00`,
gen: func() test {
var v map[string]time.Time
return test{
target: &v,
expected: &map[string]time.Time{
"a": time.Date(2010, 2, 3, 14, 15, 0, 0, time.FixedZone("", 5*3600)),
},
}
},
},
{
desc: "local-time with seconds and fractional regression",
input: `a = 14:15:30.123`,
gen: func() test {
var v map[string]interface{}
return test{
target: &v,
expected: &map[string]interface{}{
"a": toml.LocalTime{Hour: 14, Minute: 15, Second: 30, Nanosecond: 123000000, Precision: 3},
},
}
},
},
{
desc: "local-time missing digit",
input: `a = 12:08:0`,
@@ -849,104 +885,6 @@ huey = 'dewey'
}
},
},
{
desc: "basic string escape character",
input: `A = "\e"`,
gen: func() test {
type doc struct {
A string
}
return test{
target: &doc{},
expected: &doc{A: "\x1B"},
}
},
},
{
desc: "multiline basic string escape character",
input: `A = """\e"""`,
gen: func() test {
type doc struct {
A string
}
return test{
target: &doc{},
expected: &doc{A: "\x1B"},
}
},
},
{
desc: "escape character combined with bracket",
input: `A = "\e["`,
gen: func() test {
type doc struct {
A string
}
return test{
target: &doc{},
expected: &doc{A: "\x1B["},
}
},
},
{
desc: "basic string hex escape lowercase letter",
input: `A = "\x61"`,
gen: func() test {
type doc struct {
A string
}
return test{
target: &doc{},
expected: &doc{A: "a"},
}
},
},
{
desc: "basic string hex escape null byte",
input: `A = "\x00"`,
gen: func() test {
type doc struct {
A string
}
return test{
target: &doc{},
expected: &doc{A: "\x00"},
}
},
},
{
desc: "basic string hex escape max value",
input: `A = "\xFF"`,
gen: func() test {
type doc struct {
A string
}
return test{
target: &doc{},
expected: &doc{A: "\u00FF"},
}
},
},
{
desc: "multiline basic string hex escape",
input: `A = """\x61"""`,
gen: func() test {
type doc struct {
A string
}
return test{
target: &doc{},
expected: &doc{A: "a"},
}
},
},
{
desc: "spaces around dotted keys",
input: "a . b = 1",
@@ -1094,87 +1032,6 @@ B = "data"`,
}
},
},
{
desc: "multiline inline table",
input: "Name = {\n First = \"hello\",\n Last = \"world\"\n}",
gen: func() test {
type name struct {
First string
Last string
}
type doc struct {
Name name
}
return test{
target: &doc{},
expected: &doc{Name: name{
First: "hello",
Last: "world",
}},
}
},
},
{
desc: "inline table with trailing comma",
input: `Name = {First = "hello", Last = "world",}`,
gen: func() test {
type name struct {
First string
Last string
}
type doc struct {
Name name
}
return test{
target: &doc{},
expected: &doc{Name: name{
First: "hello",
Last: "world",
}},
}
},
},
{
desc: "multiline inline table with trailing comma and comments",
input: "Name = {\n # first name\n First = \"hello\",\n # last name\n Last = \"world\",\n}",
gen: func() test {
type name struct {
First string
Last string
}
type doc struct {
Name name
}
return test{
target: &doc{},
expected: &doc{Name: name{
First: "hello",
Last: "world",
}},
}
},
},
{
desc: "nested multiline inline tables",
input: "A = {\n B = {\n C = 1,\n },\n}",
gen: func() test {
var v map[string]interface{}
return test{
target: &v,
expected: &map[string]interface{}{
"A": map[string]interface{}{
"B": map[string]interface{}{
"C": int64(1),
},
},
},
}
},
},
{
desc: "inline table inside array",
input: `Names = [{First = "hello", Last = "world"}, {First = "ab", Last = "cd"}]`,
@@ -3414,7 +3271,7 @@ world'`,
{
desc: "bad char between minutes and seconds",
data: `a = 2021-03-30 21:312:0`,
msg: `extra characters at the end of a local date time`,
msg: `expecting colon between minutes and seconds`,
},
{
desc: "invalid hour value",
@@ -3529,18 +3386,6 @@ world'`,
desc: `invalid escape char basic multiline string`,
data: `A = """\z"""`,
},
{
desc: `invalid hex escape non-hex character in basic string`,
data: `A = "\xGG"`,
},
{
desc: `incomplete hex escape in basic string`,
data: `A = "\x6"`,
},
{
desc: `invalid hex escape non-hex character in multiline basic string`,
data: `A = """\xGG"""`,
},
{
desc: `invalid inf`,
data: `A = ick`,
@@ -3727,30 +3572,6 @@ world'`,
desc: `backspace in comment`,
data: "# this is a test\ba=1",
},
{
desc: `inline table comma at start`,
data: `a = { , b = 1 }`,
},
{
desc: `inline table missing separator`,
data: `a = { b = 1 c = 2 }`,
},
{
desc: `inline table double comma across newline`,
data: "a = { b = 1,\n, c = 2 }",
},
{
desc: `incomplete inline table`,
data: "a = { b = 1,\n",
},
{
desc: `incomplete hex escape in multiline basic string`,
data: `A = """\x6"""`,
},
{
desc: `invalid escape char in basic string`,
data: `A = "\z"`,
},
}
for _, e := range examples {
@@ -4205,8 +4026,8 @@ type CustomUnmarshalerKey struct {
A int64
}
func (k *CustomUnmarshalerKey) UnmarshalTOML(value *unstable.Node) error {
item, err := strconv.ParseInt(string(value.Data), 10, 64)
func (k *CustomUnmarshalerKey) UnmarshalTOML(data []byte) error {
item, err := strconv.ParseInt(string(data), 10, 64)
if err != nil {
return fmt.Errorf("error converting to int64, %w", err)
}
@@ -4294,7 +4115,7 @@ foo = "bar"`,
type doc994 struct{}
func (d *doc994) UnmarshalTOML(*unstable.Node) error {
func (d *doc994) UnmarshalTOML([]byte) error {
return errors.New("expected-error")
}
@@ -4317,8 +4138,8 @@ type doc994ok struct {
S string
}
func (d *doc994ok) UnmarshalTOML(value *unstable.Node) error {
d.S = string(value.Data) + " from unmarshaler"
func (d *doc994ok) UnmarshalTOML(data []byte) error {
d.S = string(data) + " from unmarshaler"
return nil
}
@@ -4331,7 +4152,8 @@ func TestIssue994_OK(t *testing.T) {
Decode(&d)
assert.NoError(t, err)
assert.Equal(t, "bar from unmarshaler", d.S)
// With bytes-based interface, raw TOML bytes are passed including quotes
assert.Equal(t, "\"bar\" from unmarshaler", d.S)
}
func TestIssue995(t *testing.T) {
@@ -4691,175 +4513,264 @@ func TestIssue1028(t *testing.T) {
})
}
// customFieldUnmarshaler implements unstable.Unmarshaler and captures all
// key-value pairs directed to it, including unknown fields.
type customFieldUnmarshaler struct {
// Tests for issue #873 - Bring back toml.Unmarshaler for tables and arrays
type customTable873 struct {
Keys []string
Values map[string]string
}
func (c *customFieldUnmarshaler) UnmarshalTOML(value *unstable.Node) error {
c.Values = map[string]string{
"kind": value.Kind.String(),
"data": string(value.Data),
func (c *customTable873) UnmarshalTOML(data []byte) error {
c.Keys = []string{}
c.Values = make(map[string]string)
// Parse the raw TOML bytes into a map to extract keys in order
// For this test, we use a simple line-by-line parser to preserve order
lines := bytes.Split(data, []byte{'\n'})
for _, line := range lines {
line = bytes.TrimSpace(line)
if len(line) == 0 {
continue
}
// Skip table headers
if line[0] == '[' {
continue
}
// Parse key = value
eqIdx := bytes.Index(line, []byte{'='})
if eqIdx < 0 {
continue
}
key := string(bytes.TrimSpace(line[:eqIdx]))
// Remove quotes from quoted keys
if len(key) >= 2 && key[0] == '"' && key[len(key)-1] == '"' {
key = key[1 : len(key)-1]
}
valueBytes := bytes.TrimSpace(line[eqIdx+1:])
// Remove quotes from string values
if len(valueBytes) >= 2 && valueBytes[0] == '"' && valueBytes[len(valueBytes)-1] == '"' {
valueBytes = valueBytes[1 : len(valueBytes)-1]
}
c.Keys = append(c.Keys, key)
c.Values[key] = string(valueBytes)
}
return nil
}
func TestUnmarshalerInterface_StructFieldFallback(t *testing.T) {
// When EnableUnmarshalerInterface is active and a struct field is not found,
// the decoder should fall back to the Unmarshaler interface on the struct.
// Test for split tables - when the same parent table is defined in multiple places
// This is a key requirement for issue #873: if type A implements Unmarshaler,
// and [a.b] and [a.d] are defined with another table [x] in between,
// A should receive content for both b and d, but not x.
func TestIssue873_SplitTables(t *testing.T) {
// For this test, we expect each sub-table to be handled separately
// The parent doesn't receive the sub-tables directly - each sub-table
// (b and d) gets its own call to handleKeyValues
type Config struct {
Name string `toml:"name"`
A struct {
B customTable873 `toml:"b"`
D customTable873 `toml:"d"`
} `toml:"a"`
X customTable873 `toml:"x"`
}
t.Run("unknown field with unmarshaler", func(t *testing.T) {
doc := `name = "hello"
unknown = "world"`
var cfg Config
decoder := toml.NewDecoder(bytes.NewReader([]byte(doc)))
decoder.EnableUnmarshalerInterface()
err := decoder.Decode(&cfg)
assert.NoError(t, err)
assert.Equal(t, "hello", cfg.Name)
})
}
doc := `
[a.b]
C = "1"
func TestUnmarshalerInterface_Value(t *testing.T) {
// Test that EnableUnmarshalerInterface delegates value decoding
// to the UnmarshalTOML method.
type Config struct {
Field customFieldUnmarshaler `toml:"field"`
}
[x]
Y = "100"
[a.d]
E = "2"
`
doc := `field = "test-value"`
var cfg Config
decoder := toml.NewDecoder(bytes.NewReader([]byte(doc)))
decoder.EnableUnmarshalerInterface()
err := decoder.Decode(&cfg)
err := toml.NewDecoder(bytes.NewReader([]byte(doc))).
EnableUnmarshalerInterface().
Decode(&cfg)
assert.NoError(t, err)
assert.Equal(t, "test-value", cfg.Field.Values["data"])
// Each sub-table should have received its own key-values
assert.Equal(t, []string{"C"}, cfg.A.B.Keys)
assert.Equal(t, "1", cfg.A.B.Values["C"])
assert.Equal(t, []string{"E"}, cfg.A.D.Keys)
assert.Equal(t, "2", cfg.A.D.Values["E"])
assert.Equal(t, []string{"Y"}, cfg.X.Keys)
assert.Equal(t, "100", cfg.X.Values["Y"])
}
func TestTypeMismatchString_StructFieldContext(t *testing.T) {
// Exercise the typeMismatchString code path that includes struct field info
// in the error message.
type Inner struct {
Value int `toml:"value"`
}
// Test using RawMessage to capture raw TOML bytes
func TestIssue873_RawMessage(t *testing.T) {
type Config struct {
Inner Inner `toml:"inner"`
Plugin unstable.RawMessage `toml:"plugin"`
}
doc := `inner = "not-a-table"`
doc := `
[plugin]
name = "example"
version = "1.0"
`
var cfg Config
err := toml.Unmarshal([]byte(doc), &cfg)
err := toml.NewDecoder(bytes.NewReader([]byte(doc))).
EnableUnmarshalerInterface().
Decode(&cfg)
assert.NoError(t, err)
// RawMessage should contain the raw key-value bytes
expected := "name = \"example\"\nversion = \"1.0\"\n"
assert.Equal(t, expected, string(cfg.Plugin))
}
// Test keys that need quoting (contain special characters)
func TestIssue873_QuotedKeys(t *testing.T) {
type Config struct {
Section customTable873 `toml:"section"`
}
doc := `
[section]
"key with spaces" = "value1"
"key.with.dots" = "value2"
`
var cfg Config
err := toml.NewDecoder(bytes.NewReader([]byte(doc))).
EnableUnmarshalerInterface().
Decode(&cfg)
assert.NoError(t, err)
assert.Equal(t, 2, len(cfg.Section.Keys))
assert.Equal(t, "value1", cfg.Section.Values["key with spaces"])
assert.Equal(t, "value2", cfg.Section.Values["key.with.dots"])
}
// errorUnmarshaler873 is used to test error propagation from UnmarshalTOML
type errorUnmarshaler873 struct{}
func (e *errorUnmarshaler873) UnmarshalTOML([]byte) error {
return errors.New("intentional error")
}
// Test error propagation from UnmarshalTOML
func TestIssue873_UnmarshalerError(t *testing.T) {
doc := `
[section]
key = "value"
`
type Config struct {
Section errorUnmarshaler873 `toml:"section"`
}
var cfg Config
err := toml.NewDecoder(bytes.NewReader([]byte(doc))).
EnableUnmarshalerInterface().
Decode(&cfg)
assert.Error(t, err)
assert.True(t, strings.Contains(err.Error(), "intentional error"))
}
func TestUnmarshalInlineTable_IncompatibleType(t *testing.T) {
// Exercise the default branch of unmarshalInlineTable when the target
// is not a map, struct, or interface.
type doc struct {
A int `toml:"a"`
// Test dotted keys in a table (e.g., a.b = value)
func TestIssue873_DottedKeys(t *testing.T) {
type Config struct {
Section customTable873 `toml:"section"`
}
var v doc
err := toml.Unmarshal([]byte(`a = {b = 1}`), &v)
assert.Error(t, err)
}
func TestTypeMismatchString_NoStructContext(t *testing.T) {
// Exercise the typeMismatchString code path without struct field context (line 186).
// Decoding a string into a bare int triggers this path.
var v map[string]int
err := toml.Unmarshal([]byte(`a = "hello"`), &v)
assert.Error(t, err)
}
doc := `
[section]
sub.key = "value1"
another.nested.key = "value2"
`
var cfg Config
err := toml.NewDecoder(bytes.NewReader([]byte(doc))).
EnableUnmarshalerInterface().
Decode(&cfg)
func TestMultilineInlineTable_EmptyWithNewlines(t *testing.T) {
doc := "a = {\n\n}"
var v map[string]interface{}
err := toml.Unmarshal([]byte(doc), &v)
assert.NoError(t, err)
inner := v["a"]
if inner == nil {
t.Fatal("expected key 'a' to be present")
}
m, ok := inner.(map[string]interface{})
if !ok {
t.Fatalf("expected map, got %T", inner)
}
if len(m) != 0 {
t.Fatalf("expected empty map, got %v", m)
}
assert.Equal(t, 2, len(cfg.Section.Keys))
// The dotted keys should be preserved in the raw output
assert.Equal(t, "value1", cfg.Section.Values["sub.key"])
assert.Equal(t, "value2", cfg.Section.Values["another.nested.key"])
}
func TestMultilineInlineTable_CommentsOnly(t *testing.T) {
doc := "a = {\n # just a comment\n}"
var v map[string]interface{}
err := toml.Unmarshal([]byte(doc), &v)
// Test pointer to pointer to Unmarshaler (covers pointer dereferencing loop)
func TestIssue873_DoublePointerUnmarshaler(t *testing.T) {
type Config struct {
Section **customTable873 `toml:"section"`
}
doc := `
[section]
key = "value"
`
var cfg Config
err := toml.NewDecoder(bytes.NewReader([]byte(doc))).
EnableUnmarshalerInterface().
Decode(&cfg)
assert.NoError(t, err)
inner := v["a"]
if inner == nil {
t.Fatal("expected key 'a' to be present")
}
m, ok := inner.(map[string]interface{})
if !ok {
t.Fatalf("expected map, got %T", inner)
}
if len(m) != 0 {
t.Fatalf("expected empty map, got %v", m)
}
assert.True(t, cfg.Section != nil)
assert.True(t, *cfg.Section != nil)
assert.Equal(t, []string{"key"}, (*cfg.Section).Keys)
assert.Equal(t, "value", (*cfg.Section).Values["key"])
}
func TestMultilineInlineTable_CommentAfterComma(t *testing.T) {
// Exercises comment handling after comma in inline table (parser lines 518-524).
doc := "a = { b = 1, # comment\nc = 2 }"
var v map[string]interface{}
err := toml.Unmarshal([]byte(doc), &v)
assert.NoError(t, err)
m, ok := v["a"].(map[string]interface{})
if !ok {
t.Fatal("expected a map")
}
if m["b"] != int64(1) {
t.Fatalf("expected b=1, got %v", m["b"])
}
if m["c"] != int64(2) {
t.Fatalf("expected c=2, got %v", m["c"])
}
// formattingCapture captures the raw TOML bytes to verify formatting preservation
type formattingCapture struct {
RawBytes string
}
func TestMultilineInlineTable_CommentAfterValue(t *testing.T) {
// Exercises comment handling after keyval in inline table (parser lines 542-548).
doc := "a = { b = 1 # comment\n, c = 2 }"
var v map[string]interface{}
err := toml.Unmarshal([]byte(doc), &v)
assert.NoError(t, err)
m, ok := v["a"].(map[string]interface{})
if !ok {
t.Fatal("expected a map")
}
if m["b"] != int64(1) {
t.Fatalf("expected b=1, got %v", m["b"])
}
if m["c"] != int64(2) {
t.Fatalf("expected c=2, got %v", m["c"])
}
func (f *formattingCapture) UnmarshalTOML(data []byte) error {
f.RawBytes = string(data)
return nil
}
func TestMultilineInlineTable_LeadingComma(t *testing.T) {
doc := "a = { b = 1\n, c = 2 }"
var v map[string]interface{}
err := toml.Unmarshal([]byte(doc), &v)
func TestIssue873_FormattingPreservation(t *testing.T) {
type Config struct {
Section *formattingCapture `toml:"section"`
}
// Test that various formatting styles are preserved:
// - Extra spaces around '='
// - Literal strings (single quotes)
// - Hex numbers
// - Inline tables
doc := `[section]
key1 = "value with spaces"
key2 = 'literal string'
hex_val = 0xDEADBEEF
inline = { a = 1, b = 2 }
`
var cfg Config
err := toml.NewDecoder(bytes.NewReader([]byte(doc))).
EnableUnmarshalerInterface().
Decode(&cfg)
assert.NoError(t, err)
m, ok := v["a"].(map[string]interface{})
if !ok {
t.Fatal("expected a map")
}
if m["b"] != int64(1) {
t.Fatalf("expected b=1, got %v", m["b"])
}
if m["c"] != int64(2) {
t.Fatalf("expected c=2, got %v", m["c"])
}
assert.True(t, cfg.Section != nil)
// The raw bytes should preserve original formatting
raw := cfg.Section.RawBytes
// Check that extra spaces around '=' are preserved
assert.True(t, strings.Contains(raw, "key1 = \"value with spaces\""),
"Expected spacing to be preserved, got: %s", raw)
// Check that literal string style is preserved
assert.True(t, strings.Contains(raw, "key2 = 'literal string'"),
"Expected literal string to be preserved, got: %s", raw)
// Check that hex format is preserved
assert.True(t, strings.Contains(raw, "hex_val = 0xDEADBEEF"),
"Expected hex format to be preserved, got: %s", raw)
// Check that inline table is preserved
assert.True(t, strings.Contains(raw, "inline = { a = 1, b = 2 }"),
"Expected inline table to be preserved, got: %s", raw)
}
+7 -3
View File
@@ -28,12 +28,16 @@ func (c *Iterator) Next() bool {
if c.nodes == nil {
return false
}
nodes := *c.nodes
if !c.started {
c.started = true
} else if c.idx >= 0 {
c.idx = (*c.nodes)[c.idx].next
} else {
idx := c.idx
if idx >= 0 && int(idx) < len(nodes) {
c.idx = nodes[idx].next
}
}
return c.idx >= 0 && int(c.idx) < len(*c.nodes)
return c.idx >= 0 && int(c.idx) < len(nodes)
}
// IsLast returns true if the current node of the iterator is the last
+41 -70
View File
@@ -3,6 +3,7 @@ package unstable
import (
"bytes"
"fmt"
"reflect"
"unicode"
"github.com/pelletier/go-toml/v2/internal/characters"
@@ -69,8 +70,8 @@ func (p *Parser) Data() []byte {
// panics.
func (p *Parser) Range(b []byte) Range {
return Range{
Offset: uint32(p.subsliceOffset(b)), //nolint:gosec // TOML documents are small
Length: uint32(len(b)), //nolint:gosec // TOML documents are small
Offset: uint32(subsliceOffset(p.data, b)), //nolint:gosec // TOML documents are small
Length: uint32(len(b)), //nolint:gosec // TOML documents are small
}
}
@@ -82,11 +83,21 @@ func (p *Parser) rangeOfToken(token, rest []byte) Range {
return Range{Offset: uint32(offset), Length: uint32(len(token))} //nolint:gosec // TOML documents are small
}
// subsliceOffset returns the byte offset of subslice b within p.data.
// b must be a suffix (tail) of p.data.
func (p *Parser) subsliceOffset(b []byte) int {
// b is a suffix of p.data, so its offset is len(p.data) - len(b)
return len(p.data) - len(b)
// subsliceOffset returns the byte offset of subslice b within data.
// b must share the same backing array as data (any subslice of data).
func subsliceOffset(data, b []byte) int {
if len(b) == 0 {
return 0
}
dataPtr := reflect.ValueOf(data).Pointer()
bPtr := reflect.ValueOf(b).Pointer()
offset := int(bPtr - dataPtr)
if offset < 0 || offset > len(data) {
panic("subslice is not within data")
}
return offset
}
// Raw returns the slice corresponding to the bytes in the given range.
@@ -328,6 +339,9 @@ func (p *Parser) parseStdTable(b []byte) (reference, []byte, error) {
func (p *Parser) parseKeyval(b []byte) (reference, []byte, error) {
// keyval = key keyval-sep val
// Track the start position for Raw range
startB := b
ref := p.builder.Push(Node{
Kind: KeyValue,
})
@@ -342,7 +356,7 @@ func (p *Parser) parseKeyval(b []byte) (reference, []byte, error) {
b = p.parseWhitespace(b)
if len(b) == 0 {
return invalidReference, nil, NewParserError(b, "expected = after a key, but the document ends there")
return invalidReference, nil, NewParserError(startB[:len(startB)-len(b)], "expected = after a key, but the document ends there")
}
b, err = expect('=', b)
@@ -360,6 +374,11 @@ func (p *Parser) parseKeyval(b []byte) (reference, []byte, error) {
p.builder.Chain(valRef, key)
p.builder.AttachChild(ref, valRef)
// Set Raw to span the entire key-value expression.
// Access the node directly in the slice to avoid the write barrier
// that NodeAt's nodes-pointer setup would trigger.
p.builder.tree.nodes[ref].Raw = p.rangeOfToken(startB[:len(startB)-len(b)], b)
return ref, b, err
}
@@ -460,14 +479,12 @@ func (p *Parser) parseLiteralString(b []byte) ([]byte, []byte, []byte, error) {
return v, v[1 : len(v)-1], rest, nil
}
//nolint:funlen,cyclop,dupl
func (p *Parser) parseInlineTable(b []byte) (reference, []byte, error) {
// inline-table = inline-table-open [ inline-table-keyvals ] inline-table-close
// inline-table-open = %x7B ws ; {
// inline-table-close = ws %x7D ; }
// inline-table-sep = ws %x2C ws ; , Comma
// inline-table-keyvals = keyval [ inline-table-sep inline-table-keyvals ]
tableStart := b
parent := p.builder.Push(Node{
Kind: InlineTable,
Raw: p.rangeOfToken(b[:1], b[1:]),
@@ -475,77 +492,45 @@ func (p *Parser) parseInlineTable(b []byte) (reference, []byte, error) {
first := true
lastChild := invalidReference
addChild := func(ref reference) {
if lastChild == invalidReference {
p.builder.AttachChild(parent, ref)
} else {
p.builder.Chain(lastChild, ref)
}
lastChild = ref
}
var child reference
b = b[1:]
var err error
for len(b) > 0 {
var cref reference
cref, b, err = p.parseOptionalWhitespaceCommentNewline(b)
if err != nil {
return parent, nil, err
}
if cref != invalidReference {
addChild(cref)
}
previousB := b
b = p.parseWhitespace(b)
if len(b) == 0 {
return parent, nil, NewParserError(tableStart[:1], "inline table is incomplete")
return parent, nil, NewParserError(previousB[:1], "inline table is incomplete")
}
if b[0] == '}' {
break
}
if b[0] == ',' {
if first {
return parent, nil, NewParserError(b[0:1], "inline table cannot start with comma")
}
b = b[1:]
cref, b, err = p.parseOptionalWhitespaceCommentNewline(b)
if !first {
b, err = expect(',', b)
if err != nil {
return parent, nil, err
}
if cref != invalidReference {
addChild(cref)
}
} else if !first {
return parent, nil, NewParserError(b[0:1], "inline table entries must be separated by commas")
}
// trailing comma: if '}' follows, stop
if len(b) > 0 && b[0] == '}' {
break
b = p.parseWhitespace(b)
}
var kv reference
kv, b, err = p.parseKeyval(b)
if err != nil {
return parent, nil, err
}
addChild(kv)
cref, b, err = p.parseOptionalWhitespaceCommentNewline(b)
if err != nil {
return parent, nil, err
}
if cref != invalidReference {
addChild(cref)
if first {
p.builder.AttachChild(parent, kv)
} else {
p.builder.Chain(child, kv)
}
child = kv
first = false
}
@@ -555,7 +540,7 @@ func (p *Parser) parseInlineTable(b []byte) (reference, []byte, error) {
return parent, rest, err
}
//nolint:funlen,cyclop,dupl
//nolint:funlen,cyclop
func (p *Parser) parseValArray(b []byte) (reference, []byte, error) {
// array = array-open [ array-values ] ws-comment-newline array-close
// array-open = %x5B ; [
@@ -819,13 +804,6 @@ func (p *Parser) parseMultilineBasicString(b []byte) ([]byte, []byte, []byte, er
builder.WriteByte('\t')
case 'e':
builder.WriteByte(0x1B)
case 'x':
x, err := hexToRune(atmost(token[i+1:], 2), 2)
if err != nil {
return nil, nil, nil, err
}
builder.WriteRune(x)
i += 2
case 'u':
x, err := hexToRune(atmost(token[i+1:], 4), 4)
if err != nil {
@@ -985,13 +963,6 @@ func (p *Parser) parseBasicString(b []byte) ([]byte, []byte, []byte, error) {
builder.WriteByte('\t')
case 'e':
builder.WriteByte(0x1B)
case 'x':
x, err := hexToRune(token[i+1:len(token)-1], 2)
if err != nil {
return nil, nil, nil, err
}
builder.WriteRune(x)
i += 2
case 'u':
x, err := hexToRune(token[i+1:len(token)-1], 4)
if err != nil {
+36
View File
@@ -0,0 +1,36 @@
package unstable
import (
"errors"
"testing"
)
// Regression test for https://github.com/pelletier/go-toml/issues/1047:
// Parser.Range must use the real slice offset, not len(data)-len(slice).
func TestParser_Range_HighlightAfterComment(t *testing.T) {
input := []byte("# comment\n= \"value\"")
var p Parser
p.Reset(input)
for p.NextExpression() {
}
err := p.Error()
if err == nil {
t.Fatal("expected an error")
}
var perr *ParserError
if !errors.As(err, &perr) {
t.Fatalf("expected *ParserError, got %T", err)
}
r := p.Range(perr.Highlight)
shape := p.Shape(r)
if r.Offset != 10 {
t.Errorf("Range offset: got %d, want 10", r.Offset)
}
if shape.Start.Line != 2 || shape.Start.Column != 1 {
t.Errorf("position: got %d:%d, want 2:1", shape.Start.Line, shape.Start.Column)
}
}
+6 -192
View File
@@ -331,154 +331,6 @@ func TestParser_AST(t *testing.T) {
},
},
},
{
desc: "multiline inline table",
input: "name = {\n first = \"Tom\",\n last = \"Preston-Werner\"\n}",
ast: astNode{
Kind: KeyValue,
Children: []astNode{
{
Kind: InlineTable,
Children: []astNode{
{
Kind: KeyValue,
Children: []astNode{
{Kind: String, Data: []byte(`Tom`)},
{Kind: Key, Data: []byte(`first`)},
},
},
{
Kind: KeyValue,
Children: []astNode{
{Kind: String, Data: []byte(`Preston-Werner`)},
{Kind: Key, Data: []byte(`last`)},
},
},
},
},
{
Kind: Key,
Data: []byte(`name`),
},
},
},
},
{
desc: "inline table with trailing comma",
input: `name = { first = "Tom", last = "Preston-Werner", }`,
ast: astNode{
Kind: KeyValue,
Children: []astNode{
{
Kind: InlineTable,
Children: []astNode{
{
Kind: KeyValue,
Children: []astNode{
{Kind: String, Data: []byte(`Tom`)},
{Kind: Key, Data: []byte(`first`)},
},
},
{
Kind: KeyValue,
Children: []astNode{
{Kind: String, Data: []byte(`Preston-Werner`)},
{Kind: Key, Data: []byte(`last`)},
},
},
},
},
{
Kind: Key,
Data: []byte(`name`),
},
},
},
},
{
desc: "empty inline table with newline",
input: "name = {\n}",
ast: astNode{
Kind: KeyValue,
Children: []astNode{
{
Kind: InlineTable,
Children: nil,
},
{
Kind: Key,
Data: []byte(`name`),
},
},
},
},
{
desc: "inline table with leading comma",
input: "name = { first = \"Tom\"\n, last = \"Werner\" }",
ast: astNode{
Kind: KeyValue,
Children: []astNode{
{
Kind: InlineTable,
Children: []astNode{
{
Kind: KeyValue,
Children: []astNode{
{Kind: String, Data: []byte(`Tom`)},
{Kind: Key, Data: []byte(`first`)},
},
},
{
Kind: KeyValue,
Children: []astNode{
{Kind: String, Data: []byte(`Werner`)},
{Kind: Key, Data: []byte(`last`)},
},
},
},
},
{
Kind: Key,
Data: []byte(`name`),
},
},
},
},
{
desc: "inline table with leading trailing comma",
input: "name = { first = \"Tom\"\n, }",
ast: astNode{
Kind: KeyValue,
Children: []astNode{
{
Kind: InlineTable,
Children: []astNode{
{
Kind: KeyValue,
Children: []astNode{
{Kind: String, Data: []byte(`Tom`)},
{Kind: Key, Data: []byte(`first`)},
},
},
},
},
{
Kind: Key,
Data: []byte(`name`),
},
},
},
},
{
desc: "inline table comma at start is error",
input: "name = { , first = \"Tom\" }",
err: true,
},
{
desc: "inline table double comma across newline is error",
input: "name = { first = \"Tom\",\n, last = \"Werner\" }",
err: true,
},
}
for _, e := range examples {
@@ -498,44 +350,6 @@ func TestParser_AST(t *testing.T) {
}
}
func TestParseInlineTable_CommentsWithKeepComments(t *testing.T) {
// Exercise comment reference handling inside parseInlineTable when
// KeepComments is true. This covers the addChild(cref) branches
// at the start of the loop, after comma, and after keyval.
examples := []struct {
desc string
input string
}{
{
desc: "comment at start of inline table",
input: "a = {\n# comment\nb = 1\n}",
},
{
desc: "comment after comma",
input: "a = {b = 1,\n# comment\nc = 2\n}",
},
{
desc: "comment after keyval",
input: "a = {b = 1 # comment\n, c = 2}",
},
{
desc: "comment only in inline table",
input: "a = {\n# just a comment\n}",
},
}
for _, e := range examples {
e := e
t.Run(e.desc, func(t *testing.T) {
p := Parser{KeepComments: true}
p.Reset([]byte(e.input))
p.NextExpression()
err := p.Error()
assert.NoError(t, err)
})
}
}
func BenchmarkParseBasicStringWithUnicode(b *testing.B) {
p := &Parser{}
b.Run("4", func(b *testing.B) {
@@ -725,7 +539,7 @@ key5 = [ # Next to start of inline array.
// ---
// 6:1->6:22 (105->126) | Comment [# Above simple value.]
// ---
// 1:1->1:1 (0->0) | KeyValue []
// 7:1->7:14 (127->140) | KeyValue []
// 7:7->7:14 (133->140) | String [value]
// 7:1->7:4 (127->130) | Key [key]
// 7:15->7:38 (141->164) | Comment [# Next to simple value.]
@@ -738,12 +552,12 @@ key5 = [ # Next to start of inline array.
// ---
// 14:1->14:22 (252->273) | Comment [# Above inline table.]
// ---
// 1:1->1:1 (0->0) | KeyValue []
// 15:1->15:50 (274->323) | KeyValue []
// 15:8->15:9 (281->282) | InlineTable []
// 1:1->1:1 (0->0) | KeyValue []
// 15:10->15:23 (283->296) | KeyValue []
// 15:18->15:23 (291->296) | String [Tom]
// 15:10->15:15 (283->288) | Key [first]
// 1:1->1:1 (0->0) | KeyValue []
// 15:25->15:48 (298->321) | KeyValue []
// 15:32->15:48 (305->321) | String [Preston-Werner]
// 15:25->15:29 (298->302) | Key [last]
// 15:1->15:5 (274->278) | Key [name]
@@ -753,7 +567,7 @@ key5 = [ # Next to start of inline array.
// ---
// 18:1->18:15 (371->385) | Comment [# Above array.]
// ---
// 1:1->1:1 (0->0) | KeyValue []
// 19:1->19:20 (386->405) | KeyValue []
// 1:1->1:1 (0->0) | Array []
// 19:11->19:12 (396->397) | Integer [1]
// 19:14->19:15 (399->400) | Integer [2]
@@ -765,7 +579,7 @@ key5 = [ # Next to start of inline array.
// ---
// 22:1->22:26 (448->473) | Comment [# Above multi-line array.]
// ---
// 1:1->1:1 (0->0) | KeyValue []
// 23:1->31:2 (474->694) | KeyValue []
// 1:1->1:1 (0->0) | Array []
// 23:10->23:42 (483->515) | Comment [# Next to start of inline array.]
// 24:3->24:38 (518->553) | Comment [# Second line before array content.]
+28 -3
View File
@@ -1,7 +1,32 @@
package unstable
// The Unmarshaler interface may be implemented by types to customize their
// behavior when being unmarshaled from a TOML document.
// Unmarshaler is implemented by types that can unmarshal a TOML
// description of themselves. The input is a valid TOML document
// containing the relevant portion of the parsed document.
//
// For tables (including split tables defined in multiple places),
// the data contains the raw key-value bytes from the original document
// with adjusted table headers to be relative to the unmarshaling target.
type Unmarshaler interface {
UnmarshalTOML(value *Node) error
UnmarshalTOML(data []byte) error
}
// RawMessage is a raw encoded TOML value. It implements Unmarshaler
// and can be used to delay TOML decoding or capture raw content.
//
// Example usage:
//
// type Config struct {
// Plugin RawMessage `toml:"plugin"`
// }
//
// var cfg Config
// toml.NewDecoder(r).EnableUnmarshalerInterface().Decode(&cfg)
// // cfg.Plugin now contains the raw TOML bytes for [plugin]
type RawMessage []byte
// UnmarshalTOML implements Unmarshaler.
func (m *RawMessage) UnmarshalTOML(data []byte) error {
*m = append((*m)[0:0], data...)
return nil
}