Compare commits

...

29 Commits

Author SHA1 Message Date
Cursor Agent 89f970069c Remove Parser.Range and subsliceOffset
Range() existed to recover byte offsets from Highlight subslices.
Now that ParserError carries an explicit Offset field, Range() is
unnecessary. Remove it along with the private subsliceOffset helper
in ast.go.

Tests now use perr.Offset directly and construct Range literals
for Shape() calls.

Co-authored-by: Thomas Pelletier <thomas@pelletier.dev>
2026-04-12 19:17:12 +00:00
Cursor Agent a646ffd9fa Make error position tracking explicit with Offset field on ParserError
Thread byte offset information through all error creation sites,
eliminating the need for SubsliceOffset to recover position from
pointer comparison.

Changes:
- Add Offset field to ParserError struct
- Add offset parameter to NewParserError
- Add Parser.offsetOf helper for suffix-length arithmetic
- Thread base offset through scanner functions (scanComment,
  scanBasicString, scanMultilineBasicString, scanLiteralString,
  scanMultilineLiteralString, scanWindowsNewline)
- Thread base offset through standalone functions (expect, hexToRune)
- Thread base offset through all decode functions (parseInteger,
  parseFloat, parseLocalDate, parseLocalTime, parseLocalDateTime,
  parseDateTime, checkAndRemoveUnderscores*)
- Update all unmarshaler call sites to pass value.Raw.Offset
- Update localtime.go UnmarshalText methods with base=0
- Update strict.go to populate Offset from key ranges
- Change wrapDecodeError to read de.Offset directly
- Change Utf8TomlValidAlreadyEscaped to return int index (-1 if valid)
  instead of a byte subslice
- Unexport SubsliceOffset (now only used internally by Range())

This makes error positions self-describing: each ParserError carries its
own byte offset, so callers no longer need the original document slice
and address arithmetic to determine where an error occurred.

Co-authored-by: Thomas Pelletier <thomas@pelletier.dev>
2026-04-12 19:08:55 +00:00
Cursor Agent d75117e61f Consolidate subslice offset into a single SubsliceOffset function
Remove the private subsliceOffset methods from both parser.go and
errors.go. Replace them with a single exported SubsliceOffset function
in ast.go (next to the Range type it serves).

SubsliceOffset finds the byte offset by comparing element addresses:
&data[i] == &subslice[0]. This is well-defined Go pointer comparison
on elements of the same backing array.

This fixes the v2.3.0 regression (#1047) where the parser's
subsliceOffset used len(data) - len(b), which only works for suffix
slices, not arbitrary subslices like error highlights. It also removes
the reflect-based implementation from errors.go.

Fixes #1047

Co-authored-by: Thomas Pelletier <thomas@pelletier.dev>
2026-04-12 18:33:22 +00:00
Cursor Agent 19174a4293 Remove cap tricks, use address comparison for subslice offset
Replace cap(parent) - cap(subslice) with a straightforward scan
that compares element addresses: &data[i] == &subslice[0]. This is
well-defined Go pointer comparison on elements of the same backing
array, with no dependency on capacity semantics, reflect, or unsafe.

The scan is O(n) but only runs on error paths, and TOML documents
are small per the project's design constraints.

Also remove the Offset field from ParserError and the setErrOffset
machinery — the offset is computed at the point of consumption
(wrapDecodeError, Parser.Range) rather than cached on the error.

Co-authored-by: Thomas Pelletier <thomas@pelletier.dev>
2026-04-12 18:17:55 +00:00
Cursor Agent 96ac48eb74 Remove optional offset and fallback, guarantee offset by construction
ParserError.Offset is now a plain exported int field, always set:
- The parser sets it via setErrOffset() when capturing parse errors
- strict.go sets it from the key's Raw range at construction
- wrapDecodeError computes it inline from cap(document) - cap(highlight)

This eliminates:
- The SetOffset/Offset() accessor methods and offsetValid flag
- The subsliceOffset fallback function in errors.go
- Any conditional logic around whether the offset is present

The offset is guaranteed by construction at every path that creates
or consumes a ParserError.

Co-authored-by: Thomas Pelletier <thomas@pelletier.dev>
2026-04-12 17:25:32 +00:00
Cursor Agent f7136d052b Replace reflect-based subslice offset with cap arithmetic
Use cap(parent) - cap(subslice) to compute byte offsets between slices
that share a backing array. This is safe pure Go: subslicing preserves
the backing array and adjusts the capacity accordingly, so the
difference in capacities equals the byte offset.

This removes the reflect import from both errors.go and
unstable/parser.go, eliminating the last reflect-based pointer
arithmetic used for error position tracking.

Co-authored-by: Thomas Pelletier <thomas@pelletier.dev>
2026-04-12 17:08:58 +00:00
Cursor Agent 154d80392f Cache error offset in ParserError for safer position tracking
Instead of requiring downstream consumers to re-derive the byte offset
from pointer arithmetic on the Highlight slice, compute and cache the
offset inside the parser at error-capture time via setErrOffset().

This is safer because:
- The parser is the one place where the backing-array guarantee is known
  to hold (Highlight is always a subslice of the parse buffer)
- Downstream consumers (wrapDecodeError) can use the cached offset
  directly, avoiding the need for pointer comparison
- Errors created outside the parser (strict.go) set the offset from
  existing Raw ranges, which are already correct by construction

Add ParserError.SetOffset/Offset methods for setting and retrieving the
cached offset. Update wrapDecodeError to prefer the cached offset when
available, falling back to subsliceOffset for backward compatibility.

Co-authored-by: Thomas Pelletier <thomas@pelletier.dev>
2026-04-12 13:00:38 +00:00
Cursor Agent d528d3c6b4 Fix Parser.Range returning wrong offset for error highlights
The subsliceOffset method incorrectly computed offset as
len(p.data) - len(b), which only works when b is a suffix (tail) of
p.data. However, error highlights (ParserError.Highlight) are arbitrary
subslices from the middle of the input (e.g., b[0:1] from parseSimpleKey),
so their len has no relationship to their position.

This was a regression introduced in commit 3aaf147 (Remove unsafe package
usage) which replaced danger.SubsliceOffset (pointer arithmetic) with the
incorrect len-based approach.

Fix by using reflect.ValueOf().Pointer() to compute the actual byte
offset between slice data pointers, matching the approach already used
in errors.go:subsliceOffset.

Fixes #1047

Co-authored-by: Thomas Pelletier <thomas@pelletier.dev>
2026-04-12 12:40:14 +00:00
Thomas Pelletier f36a3ece9e Reduce marshal and unmarshal overhead (#1044)
* Reduce marshal and unmarshal overhead

Targeted optimizations to reduce performance overhead introduced by
recent feature additions and the unsafe removal.

Unmarshal:
- parseKeyval: access the node directly in the builder's slice to set
  Raw, bypassing NodeAt which triggers a GC write barrier for the
  nodes-pointer on every key-value expression.
- Iterator.Next: cache the *nodes slice dereference in a local variable
  to avoid repeated pointer-to-slice indirection in the hot loop.

Marshal:
- Guard shouldOmitZero calls with an inlineable options.omitzero check.
  shouldOmitZero has inlining cost 1145 (budget 80), so avoiding the
  function call when omitzero is not set removes per-field overhead.
- Inline the isNil check in encodeMap. isNil has inlining cost 93
  (budget 80), so expanding it at the single hot call site avoids
  per-map-entry function call overhead.

Update README benchmarks.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 11:08:39 +00:00
Thomas Pelletier 77f3862df4 Fix benchmark script replacing internal package imports (#1042)
* Fix benchmark script replacing internal package imports

The sed command in bench() was replacing all occurrences of the go-toml
module path, including sub-package imports like internal/assert. This
caused the BurntSushi/toml benchmark to fail because it tried to import
github.com/BurntSushi/toml/internal/assert which doesn't exist.

Fix by anchoring the sed pattern to only match the import path when
followed by a closing quote, preserving internal package imports.

Also add a guard in the benchstathtml Python script to give a clear
error instead of an IndexError when no benchmark results are available.

https://claude.ai/code/session_016JGASo49PeFSfCaDxvrGFE

* Update benchmark results in README

https://claude.ai/code/session_016JGASo49PeFSfCaDxvrGFE

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-03-23 22:00:18 -04:00
Thomas Pelletier 16b1ef5508 Fix parser error pointing to wrong line when last line has no trailing newline (#1041)
When parsing a key without '=' at EOF (e.g., "a = 1\nb = 2\nc"), the
error highlight was an empty slice, causing subsliceOffset to return 0
and the error to point at line 1 instead of line 3. Pass the consumed
key bytes as the highlight instead of the empty remainder.

Fixes #1032

https://claude.ai/code/session_01UWv8pyc8P1ktAPfHpveixj

Co-authored-by: Claude <noreply@anthropic.com>
2026-03-23 21:34:12 -04:00
dependabot[bot] e14bde7c1d build(deps): bump docker/login-action from 3 to 4 (#1039)
Bumps [docker/login-action](https://github.com/docker/login-action) from 3 to 4.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/v3...v4)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-version: '4'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-23 21:04:21 -04:00
dependabot[bot] 4b1ff01eb3 build(deps): bump docker/setup-buildx-action from 3 to 4 (#1040)
Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 3 to 4.
- [Release notes](https://github.com/docker/setup-buildx-action/releases)
- [Commits](https://github.com/docker/setup-buildx-action/compare/v3...v4)

---
updated-dependencies:
- dependency-name: docker/setup-buildx-action
  dependency-version: '4'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-23 21:04:11 -04:00
Thomas Pelletier 048a25f0f2 Go 1.26 (#1030)
* ci(release): drop unsupported windows/arm targets
2026-03-03 01:29:35 -05:00
dependabot[bot] b3575580f9 build(deps): bump goreleaser/goreleaser-action from 6 to 7 (#1035) 2026-03-03 00:47:47 -05:00
dependabot[bot] a0be52f4c1 build(deps): bump actions/upload-artifact from 6 to 7 (#1036) 2026-03-03 00:47:35 -05:00
Thomas Pelletier 316bfc66a4 Support Unmarshaler interface for tables and array tables (#1027)
Fixes #873

Extend the unstable.Unmarshaler interface support to work with tables
and array tables, not just single values.

When a type implementing unstable.Unmarshaler is the target of a table
(e.g., [table] or [[array]]), the UnmarshalTOML method receives a
synthetic InlineTable node containing all the key-value pairs belonging
to that table.

Key changes:
- Add handleKeyValuesUnmarshaler to collect and process table content
- Add copyExpressionNodes to deep-copy AST nodes for synthetic tables
- Add helper functions in unstable/ast.go for node manipulation
- Update documentation for EnableUnmarshalerInterface
- Add comprehensive tests for table and array table unmarshaling

* Implement bytes-based Unmarshaler interface for tables and arrays (#873)

This change brings back support for the unstable.Unmarshaler interface
for tables and array tables, addressing issue #873.

Key changes:
- Changed UnmarshalTOML signature from (*Node) to ([]byte) to provide
  raw TOML bytes instead of AST nodes
- Added RawMessage type (similar to json.RawMessage) for capturing raw
  TOML bytes for later processing
- Updated handleKeyValuesUnmarshaler to reconstruct key-value lines
  from the parsed keys and raw value bytes
- Added support for slice types implementing Unmarshaler (e.g., RawMessage)
- Removed unused AST helper functions from unstable/ast.go

The bytes-based interface allows users to:
- Get raw TOML bytes for custom parsing
- Delay TOML decoding using RawMessage
- Implement custom unmarshaling logic for complex types

Tests added for:
- Table unmarshaler with various scenarios
- Array table unmarshaler
- Split tables (same parent defined in multiple places)
- RawMessage usage
- Nested tables and mixed regular fields

* Fix lint issues and improve test coverage for Unmarshaler interface

- Apply De Morgan's law in keyNeedsQuoting to satisfy staticcheck QF1001
- Remove unused splitTableUnmarshaler type from test
- Fix unused parameter lint warning in errorUnmarshaler873
- Add test for quoted keys that need special handling
- Add test for error propagation from UnmarshalTOML
- Update customTable873 parser to handle quoted keys properly

Coverage improved:
- handleKeyValuesUnmarshaler: 80.0% -> 93.3%
- keyNeedsQuoting: 66.7% -> 83.3%
- Overall main package: 97.2% -> 97.5%

* Add test for dotted keys to improve coverage

Add TestIssue873_DottedKeys to test dotted key handling (e.g., sub.key = value)
in the Unmarshaler interface. This improves coverage for handleKeyValuesUnmarshaler
from 93.3% to 96.7%.

* Add double pointer test to achieve 100% coverage for handleKeyValues

Add TestIssue873_DoublePointerUnmarshaler to test pointer-to-pointer
to Unmarshaler types. This covers the pointer dereferencing loop in
handleKeyValues, bringing its coverage from 88% to 100%.

Total coverage: 97.4%

* Add Example tests and fix raw value extraction for boolean types

Add two godoc Example tests:
- ExampleDecoder_EnableUnmarshalerInterface_dynamicConfig: shows dynamic
  unmarshaling based on a type field
- ExampleDecoder_EnableUnmarshalerInterface_rawMessage: demonstrates
  RawMessage usage for deferred parsing

Fix handleKeyValuesUnmarshaler to handle values where Raw.Length == 0
(like boolean types) by using value.Data as fallback.

* Preserve original formatting in Unmarshaler by using raw byte ranges

Instead of reconstructing key-value lines from parsed components, now
uses the original raw bytes from the document. This preserves:
- Whitespace around '=' (e.g., "key   =   value")
- String quoting style (basic vs literal)
- Number formats (hex, octal, binary)
- Inline table formatting

Changes:
- Add Raw range tracking to KeyValue expressions in parseKeyval
- Update handleKeyValuesUnmarshaler to use expr.Raw directly
- Remove keyNeedsQuoting helper (no longer needed)
- Add TestIssue873_FormattingPreservation test
- Update expected output in ExampleParser_comments

* Prevent test matrix from canceling on first failure

Add fail-fast: false to the test workflow strategy so that all
OS/Go version combinations continue running even if one fails.
This provides better visibility into which specific combinations
have issues.

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-02-11 09:57:23 -05:00
Thomas Pelletier 2edc61f171 Fix panic when unmarshaling datetime values to incompatible types (#1028) (#1029)
Return a type mismatch error instead of panicking when datetime values
(DateTime, LocalDate, LocalTime, LocalDateTime) are unmarshaled into
incompatible Go types. This makes the decoder safer for processing
untrusted TOML input.

https://claude.ai/code/session_011jwvtDS5M2KncLrqJpgMr5

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-23 22:04:40 -05:00
Thomas Pelletier 4a1b05ca08 UnmarshalText fallbacks to struct unmarshaling for tables and arrays (#1026)
When a type implements encoding.TextUnmarshaler, the unmarshaler now
skips calling UnmarshalText for Array and InlineTable TOML values.
This allows types to support both:
- Simple string values via UnmarshalText
- Structured table values via field-by-field unmarshaling

Previously, UnmarshalText was called unconditionally, which prevented
proper struct unmarshaling when the TOML value was a table or array
of tables.

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-09 13:46:38 -05:00
Thomas Pelletier 003aa0993b Fix nil pointer map values not being marshaled (#1025)
When marshaling a map with nil pointer values, the keys were being
silently dropped, breaking round-trip fidelity. For example:

    map[string]*struct{}{"foo": nil}

Would produce an empty TOML document instead of "[foo]".

This change converts nil pointer values in maps to their zero values
(consistent with how nil pointers in slices are handled), allowing the
keys to be preserved as empty tables.

Nil interface values (map[string]any{"foo": nil}) are still skipped
since there's no type information to derive a zero value.

Fixes #975

Also, pin golangci-lint version to v2.8.0 in CI and document in AGENTS.md

- Explicitly set golangci-lint version in lint.yml to ensure consistent
  behavior across CI runs
- Update AGENTS.md with instructions to use the same linter version locally

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-09 11:08:31 -05:00
dependabot[bot] 84d730b6c4 build(deps): bump golangci/golangci-lint-action from 8 to 9 (#1022) 2026-01-05 21:23:56 -05:00
dependabot[bot] 97bd897177 build(deps): bump actions/setup-go from 5 to 6 (#1023) 2026-01-05 21:23:35 -05:00
dependabot[bot] 7924b1816f build(deps): bump actions/checkout from 5 to 6 (#1024) 2026-01-05 21:23:15 -05:00
Thomas Pelletier 2a07b6d9db Update to Go 1.25 (#1018)
Update CI workflows to test against Go 1.24 and 1.25, and use Go 1.25 for
coverage and release builds.

## Benchstat Report: Go 1.24 vs Go 1.25

Benchmark comparison between Go 1.24.7 and Go 1.25.1 (10 runs each):

### Execution Time (sec/op)

| Benchmark | Go 1.24 | Go 1.25 | Delta |
|-----------|---------|---------|-------|
| UnmarshalDataset/config | 26.25ms | 26.00ms | ~ (p=0.280) |
| UnmarshalDataset/canada | 88.71ms | 84.94ms | **-4.26%**  |
| UnmarshalDataset/citm_catalog | 33.71ms | 34.06ms | ~ (p=0.684) |
| UnmarshalDataset/twitter | 17.19ms | 17.33ms | ~ (p=0.971) |
| UnmarshalDataset/code | 107.4ms | 108.1ms | ~ (p=0.393) |
| UnmarshalDataset/example | 237.9µs | 251.3µs | +5.64% |
| Unmarshal/SimpleDocument/struct | 872.3ns | 848.9ns | ~ (p=0.165) |
| Unmarshal/SimpleDocument/map | 1.191µs | 1.278µs | +7.31% |
| Unmarshal/ReferenceFile/struct | 57.14µs | 57.95µs | ~ (p=0.089) |
| Unmarshal/ReferenceFile/map | 87.89µs | 92.88µs | +5.69% |
| Unmarshal/HugoFrontMatter | 16.06µs | 15.95µs | ~ (p=0.529) |
| Marshal/SimpleDocument/struct | 536.5ns | 563.5ns | +5.03% |
| Marshal/SimpleDocument/map | 651.0ns | 675.1ns | +3.72% |
| Marshal/ReferenceFile/struct | 44.63µs | 50.84µs | +13.91% |
| Marshal/ReferenceFile/map | 51.58µs | 57.06µs | +10.61% |
| Marshal/HugoFrontMatter | 10.04µs | 10.57µs | +5.27% |
| **geomean** | 140.6µs | 145.1µs | +3.18% |

### Summary

- Notable improvement: UnmarshalDataset/canada shows a 4.26% speedup
- Memory allocation and allocation counts remain identical
- Some marshal operations show slight slowdowns (likely Go runtime changes)

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-04 13:59:18 -05:00
Thomas Pelletier 692b98560b Support custom IsZero() methods with omitzero tag (#1020)
The omitzero tag now respects custom IsZero() methods on types,
similar to how encoding/json handles this. Previously, only
reflect.Value.IsZero() was used, which ignores user-defined
implementations.

Fixes #1003

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-04 13:58:47 -05:00
Thomas Pelletier 99cd40b175 Reject leap seconds to prevent year overflow (#1019)
Go's time.Date() normalizes leap seconds (second=60) by adding 1 minute.
When parsing the maximum valid TOML date 9999-12-31 23:59:60z, this causes
the year to overflow to 10000, which exceeds the valid TOML year range
(0000-9999) and breaks round-trip serialization.

The fix rejects leap seconds (second > 59) during parsing. This is
consistent with the resolution of issue #913 which determined that
emitting an error is less surprising than silently normalizing leap
seconds.

Fixes #1015

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-04 13:40:19 -05:00
Thomas Pelletier 3aaf147e3e Remove unsafe package usage (#1021)
Removes all unsafe operations from go-toml, making the codebase
fully safe Go code. The internal/danger package that contained
unsafe operations has been deleted.

Changes:
- Replace pointer-based node navigation with index-based navigation
- Node.next and Node.child now store absolute indices into the
  backing nodes slice instead of relative offsets
- Add nodes pointer to Node and Iterator for safe navigation
- Replace danger.TypeID with reflect.Type for cache keys
- Delete internal/danger package entirely

Performance overhead is under 10% compared to the unsafe version,
which is acceptable for the safety and maintainability benefits.

[Cursor][claude-sonnet-4-20250514]
2026-01-04 13:16:47 -05:00
Nathan Baulch a675c6b3e2 Upgrade to golangci-lint v2 (#1008) 2026-01-04 09:54:29 -05:00
Thomas Pelletier 9702fae9b8 Add AGENTS.md for AI agent contribution guidelines (#1017)
This file provides a concise summary of the contribution guidelines
from CONTRIBUTING.md, specifically tailored for AI agents working on
the codebase. It covers testing requirements, backward compatibility,
performance considerations, and code style expectations.

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-03 21:41:34 -05:00
62 changed files with 2367 additions and 1305 deletions
+1 -1
View File
@@ -19,7 +19,7 @@ jobs:
dry-run: false
language: go
- name: Upload Crash
uses: actions/upload-artifact@v6
uses: actions/upload-artifact@v7
if: failure() && steps.build.outcome == 'success'
with:
name: artifacts
+1 -1
View File
@@ -15,6 +15,6 @@ jobs:
- name: Setup go
uses: actions/setup-go@v6
with:
go-version: "1.24"
go-version: "1.26"
- name: Run tests with coverage
run: ./ci.sh coverage -d "${GITHUB_BASE_REF-HEAD}"
+2 -2
View File
@@ -20,7 +20,7 @@ jobs:
fetch-depth: 0
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
uses: docker/setup-buildx-action@v4
- name: Run Go versions compatibility test
run: |
@@ -28,7 +28,7 @@ jobs:
./test-go-versions.sh --output ./test-results $VERSIONS
- name: Upload test results
uses: actions/upload-artifact@v6
uses: actions/upload-artifact@v7
with:
name: go-versions-test-results
path: |
+22
View File
@@ -0,0 +1,22 @@
name: lint
on:
pull_request:
branches:
- v2
jobs:
golangci:
name: lint
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Setup go
uses: actions/setup-go@v6
with:
go-version: "1.26"
- name: Run golangci-lint
uses: golangci/golangci-lint-action@v9
with:
version: v2.8.0
+3 -3
View File
@@ -22,15 +22,15 @@ jobs:
- name: Set up Go
uses: actions/setup-go@v6
with:
go-version: "1.24"
go-version: "1.26"
- name: Login to GitHub Container Registry
uses: docker/login-action@v3
uses: docker/login-action@v4
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Run GoReleaser
uses: goreleaser/goreleaser-action@v6
uses: goreleaser/goreleaser-action@v7
with:
distribution: goreleaser
version: '~> v2'
+2 -1
View File
@@ -10,9 +10,10 @@ on:
jobs:
build:
strategy:
fail-fast: false
matrix:
os: [ 'ubuntu-latest', 'windows-latest', 'macos-latest', 'macos-14' ]
go: [ '1.23', '1.24' ]
go: [ '1.25', '1.26' ]
runs-on: ${{ matrix.os }}
name: ${{ matrix.go }}/${{ matrix.os }}
steps:
+33 -41
View File
@@ -1,84 +1,76 @@
[service]
golangci-lint-version = "1.39.0"
[linters-settings.wsl]
allow-assign-and-anything = true
[linters-settings.exhaustive]
default-signifies-exhaustive = true
version = "2"
[linters]
disable-all = true
default = "none"
enable = [
"asciicheck",
"bodyclose",
"cyclop",
"deadcode",
"depguard",
"dogsled",
"dupl",
"durationcheck",
"errcheck",
"errorlint",
"exhaustive",
# "exhaustivestruct",
"exportloopref",
"forbidigo",
# "forcetypeassert",
"funlen",
"gci",
# "gochecknoglobals",
"gochecknoinits",
"gocognit",
"goconst",
"gocritic",
"gocyclo",
"godot",
"godox",
# "goerr113",
"gofmt",
"gofumpt",
"godoclint",
"goheader",
"goimports",
"golint",
"gomnd",
# "gomoddirectives",
"gomodguard",
"goprintffuncname",
"gosec",
"gosimple",
"govet",
# "ifshort",
"importas",
"ineffassign",
"lll",
"makezero",
"mirror",
"misspell",
"nakedret",
"nestif",
"nilerr",
# "nlreturn",
"noctx",
"nolintlint",
#"paralleltest",
"perfsprint",
"prealloc",
"predeclared",
"revive",
"rowserrcheck",
"sqlclosecheck",
"staticcheck",
"structcheck",
"stylecheck",
# "testpackage",
"thelper",
"tparallel",
"typecheck",
"unconvert",
"unparam",
"unused",
"varcheck",
"usetesting",
"wastedassign",
"whitespace",
# "wrapcheck",
# "wsl"
]
[linters.settings.exhaustive]
default-signifies-exhaustive = true
[linters.settings.lll]
line-length = 150
[[linters.exclusions.rules]]
path = ".test.go"
linters = ["goconst", "gosec"]
[[linters.exclusions.rules]]
path = "main.go"
linters = ["forbidigo"]
[[linters.exclusions.rules]]
path = "internal"
linters = ["revive"]
text = "(exported|indent-error-flow): "
[formatters]
enable = [
"gci",
"gofmt",
"gofumpt",
"goimports",
]
-3
View File
@@ -22,7 +22,6 @@ builds:
- linux_riscv64
- windows_amd64
- windows_arm64
- windows_arm
- darwin_amd64
- darwin_arm64
- id: tomljson
@@ -42,7 +41,6 @@ builds:
- linux_riscv64
- windows_amd64
- windows_arm64
- windows_arm
- darwin_amd64
- darwin_arm64
- id: jsontoml
@@ -62,7 +60,6 @@ builds:
- linux_arm
- windows_amd64
- windows_arm64
- windows_arm
- darwin_amd64
- darwin_arm64
universal_binaries:
+64
View File
@@ -0,0 +1,64 @@
# Agent Guidelines for go-toml
This file provides guidelines for AI agents contributing to go-toml. All agents must follow these rules derived from [CONTRIBUTING.md](./CONTRIBUTING.md).
## Project Overview
go-toml is a TOML library for Go. The goal is to provide an easy-to-use and efficient TOML implementation that gets the job done without getting in the way.
## Code Change Rules
### Backward Compatibility
- **No backward-incompatible changes** unless explicitly discussed and approved
- Avoid breaking people's programs unless absolutely necessary
### Testing Requirements
- **All bug fixes must include regression tests**
- **All new code must be tested**
- Run tests before submitting: `go test -race ./...`
- Test coverage must not decrease. Check with:
```bash
go test -covermode=atomic -coverprofile=coverage.out
go tool cover -func=coverage.out
```
- All lines of code touched by changes should be covered by tests
### Performance Requirements
- go-toml aims to stay efficient; avoid performance regressions
- Run benchmarks to verify: `go test ./... -bench=. -count=10`
- Compare results using [benchstat](https://pkg.go.dev/golang.org/x/perf/cmd/benchstat)
### Documentation
- New features or feature extensions must include documentation
- Documentation lives in [README.md](./README.md) and throughout source code
### Code Style
- Follow existing code format and structure
- Code must pass `go fmt`
- Code must pass linting with the same golangci-lint version as CI (see version in `.github/workflows/lint.yml`):
```bash
# Install specific version (check lint.yml for current version)
curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/HEAD/install.sh | sh -s -- -b $(go env GOPATH)/bin <version>
# Run linter
golangci-lint run ./...
```
### Commit Messages
- Commit messages must explain **why** the change is needed
- Keep messages clear and informative even if details are in the PR description
## Pull Request Checklist
Before submitting:
1. Tests pass (`go test -race ./...`)
2. No backward-incompatible changes (unless discussed)
3. Relevant documentation added/updated
4. No performance regression (verify with benchmarks)
5. Title is clear and understandable for changelog
+27 -27
View File
@@ -235,17 +235,17 @@ the AST level. See https://pkg.go.dev/github.com/pelletier/go-toml/v2/unstable.
Execution time speedup compared to other Go TOML libraries:
<table>
<thead>
<tr><th>Benchmark</th><th>go-toml v1</th><th>BurntSushi/toml</th></tr>
</thead>
<tbody>
<tr><td>Marshal/HugoFrontMatter-2</td><td>1.9x</td><td>2.2x</td></tr>
<tr><td>Marshal/ReferenceFile/map-2</td><td>1.7x</td><td>2.1x</td></tr>
<tr><td>Marshal/ReferenceFile/struct-2</td><td>2.2x</td><td>3.0x</td></tr>
<tr><td>Unmarshal/HugoFrontMatter-2</td><td>2.9x</td><td>2.7x</td></tr>
<tr><td>Unmarshal/ReferenceFile/map-2</td><td>2.6x</td><td>2.7x</td></tr>
<tr><td>Unmarshal/ReferenceFile/struct-2</td><td>4.6x</td><td>5.1x</td></tr>
</tbody>
<thead>
<tr><th>Benchmark</th><th>go-toml v1</th><th>BurntSushi/toml</th></tr>
</thead>
<tbody>
<tr><td>Marshal/HugoFrontMatter-2</td><td>2.1x</td><td>2.0x</td></tr>
<tr><td>Marshal/ReferenceFile/map-2</td><td>2.0x</td><td>2.0x</td></tr>
<tr><td>Marshal/ReferenceFile/struct-2</td><td>2.3x</td><td>2.5x</td></tr>
<tr><td>Unmarshal/HugoFrontMatter-2</td><td>3.3x</td><td>2.8x</td></tr>
<tr><td>Unmarshal/ReferenceFile/map-2</td><td>2.9x</td><td>3.0x</td></tr>
<tr><td>Unmarshal/ReferenceFile/struct-2</td><td>4.8x</td><td>5.0x</td></tr>
</tbody>
</table>
<details><summary>See more</summary>
<p>The table above has the results of the most common use-cases. The table below
@@ -253,22 +253,22 @@ contains the results of all benchmarks, including unrealistic ones. It is
provided for completeness.</p>
<table>
<thead>
<tr><th>Benchmark</th><th>go-toml v1</th><th>BurntSushi/toml</th></tr>
</thead>
<tbody>
<tr><td>Marshal/SimpleDocument/map-2</td><td>1.8x</td><td>2.7x</td></tr>
<tr><td>Marshal/SimpleDocument/struct-2</td><td>2.7x</td><td>3.8x</td></tr>
<tr><td>Unmarshal/SimpleDocument/map-2</td><td>3.8x</td><td>3.0x</td></tr>
<tr><td>Unmarshal/SimpleDocument/struct-2</td><td>5.6x</td><td>4.1x</td></tr>
<tr><td>UnmarshalDataset/example-2</td><td>3.0x</td><td>3.2x</td></tr>
<tr><td>UnmarshalDataset/code-2</td><td>2.3x</td><td>2.9x</td></tr>
<tr><td>UnmarshalDataset/twitter-2</td><td>2.6x</td><td>2.7x</td></tr>
<tr><td>UnmarshalDataset/citm_catalog-2</td><td>2.2x</td><td>2.3x</td></tr>
<tr><td>UnmarshalDataset/canada-2</td><td>1.8x</td><td>1.5x</td></tr>
<tr><td>UnmarshalDataset/config-2</td><td>4.1x</td><td>2.9x</td></tr>
<tr><td>geomean</td><td>2.7x</td><td>2.8x</td></tr>
</tbody>
<thead>
<tr><th>Benchmark</th><th>go-toml v1</th><th>BurntSushi/toml</th></tr>
</thead>
<tbody>
<tr><td>Marshal/SimpleDocument/map-2</td><td>2.0x</td><td>2.9x</td></tr>
<tr><td>Marshal/SimpleDocument/struct-2</td><td>2.5x</td><td>3.6x</td></tr>
<tr><td>Unmarshal/SimpleDocument/map-2</td><td>4.2x</td><td>3.4x</td></tr>
<tr><td>Unmarshal/SimpleDocument/struct-2</td><td>5.9x</td><td>4.4x</td></tr>
<tr><td>UnmarshalDataset/example-2</td><td>3.2x</td><td>2.9x</td></tr>
<tr><td>UnmarshalDataset/code-2</td><td>2.4x</td><td>2.8x</td></tr>
<tr><td>UnmarshalDataset/twitter-2</td><td>2.7x</td><td>2.5x</td></tr>
<tr><td>UnmarshalDataset/citm_catalog-2</td><td>2.3x</td><td>2.3x</td></tr>
<tr><td>UnmarshalDataset/canada-2</td><td>1.9x</td><td>1.5x</td></tr>
<tr><td>UnmarshalDataset/config-2</td><td>5.4x</td><td>3.0x</td></tr>
<tr><td>geomean</td><td>2.9x</td><td>2.8x</td></tr>
</tbody>
</table>
<p>This table can be generated with <code>./ci.sh benchmark -a -html</code>.</p>
</details>
+4 -4
View File
@@ -12,7 +12,7 @@ import (
"github.com/pelletier/go-toml/v2/internal/assert"
)
var bench_inputs = []struct {
var benchInputs = []struct {
name string
jsonLen int
}{
@@ -30,7 +30,7 @@ var bench_inputs = []struct {
}
func TestUnmarshalDatasetCode(t *testing.T) {
for _, tc := range bench_inputs {
for _, tc := range benchInputs {
t.Run(tc.name, func(t *testing.T) {
buf := fixture(t, tc.name)
@@ -45,7 +45,7 @@ func TestUnmarshalDatasetCode(t *testing.T) {
}
func BenchmarkUnmarshalDataset(b *testing.B) {
for _, tc := range bench_inputs {
for _, tc := range benchInputs {
b.Run(tc.name, func(b *testing.B) {
buf := fixture(b, tc.name)
b.SetBytes(int64(len(buf)))
@@ -69,7 +69,7 @@ func fixture(tb testing.TB, path string) []byte {
tb.Skip("benchmark fixture not found:", file)
}
assert.NoError(tb, err)
defer f.Close()
defer func() { _ = f.Close() }()
gz, err := gzip.NewReader(f)
assert.NoError(tb, err)
+16 -16
View File
@@ -18,7 +18,7 @@ func TestUnmarshalSimple(t *testing.T) {
err := toml.Unmarshal(doc, &d)
if err != nil {
panic(err)
t.Error(err)
}
}
@@ -38,7 +38,7 @@ func BenchmarkUnmarshal(b *testing.B) {
err := toml.Unmarshal(doc, &d)
if err != nil {
panic(err)
b.Error(err)
}
}
})
@@ -52,7 +52,7 @@ func BenchmarkUnmarshal(b *testing.B) {
d := map[string]interface{}{}
err := toml.Unmarshal(doc, &d)
if err != nil {
panic(err)
b.Error(err)
}
}
})
@@ -72,7 +72,7 @@ func BenchmarkUnmarshal(b *testing.B) {
d := benchmarkDoc{}
err := toml.Unmarshal(bytes, &d)
if err != nil {
panic(err)
b.Error(err)
}
}
})
@@ -85,7 +85,7 @@ func BenchmarkUnmarshal(b *testing.B) {
d := map[string]interface{}{}
err := toml.Unmarshal(bytes, &d)
if err != nil {
panic(err)
b.Error(err)
}
}
})
@@ -99,7 +99,7 @@ func BenchmarkUnmarshal(b *testing.B) {
d := map[string]interface{}{}
err := toml.Unmarshal(hugoFrontMatterbytes, &d)
if err != nil {
panic(err)
b.Error(err)
}
}
})
@@ -123,7 +123,7 @@ func BenchmarkMarshal(b *testing.B) {
err := toml.Unmarshal(doc, &d)
if err != nil {
panic(err)
b.Error(err)
}
b.ReportAllocs()
@@ -134,7 +134,7 @@ func BenchmarkMarshal(b *testing.B) {
for i := 0; i < b.N; i++ {
out, err = marshal(d)
if err != nil {
panic(err)
b.Error(err)
}
}
@@ -145,7 +145,7 @@ func BenchmarkMarshal(b *testing.B) {
d := map[string]interface{}{}
err := toml.Unmarshal(doc, &d)
if err != nil {
panic(err)
b.Error(err)
}
b.ReportAllocs()
@@ -156,7 +156,7 @@ func BenchmarkMarshal(b *testing.B) {
for i := 0; i < b.N; i++ {
out, err = marshal(d)
if err != nil {
panic(err)
b.Error(err)
}
}
@@ -174,7 +174,7 @@ func BenchmarkMarshal(b *testing.B) {
d := benchmarkDoc{}
err := toml.Unmarshal(bytes, &d)
if err != nil {
panic(err)
b.Error(err)
}
b.ReportAllocs()
b.ResetTimer()
@@ -184,7 +184,7 @@ func BenchmarkMarshal(b *testing.B) {
for i := 0; i < b.N; i++ {
out, err = marshal(d)
if err != nil {
panic(err)
b.Error(err)
}
}
@@ -195,7 +195,7 @@ func BenchmarkMarshal(b *testing.B) {
d := map[string]interface{}{}
err := toml.Unmarshal(bytes, &d)
if err != nil {
panic(err)
b.Error(err)
}
b.ReportAllocs()
@@ -205,7 +205,7 @@ func BenchmarkMarshal(b *testing.B) {
for i := 0; i < b.N; i++ {
out, err = marshal(d)
if err != nil {
panic(err)
b.Error(err)
}
}
@@ -217,7 +217,7 @@ func BenchmarkMarshal(b *testing.B) {
d := map[string]interface{}{}
err := toml.Unmarshal(hugoFrontMatterbytes, &d)
if err != nil {
panic(err)
b.Error(err)
}
b.ReportAllocs()
@@ -228,7 +228,7 @@ func BenchmarkMarshal(b *testing.B) {
for i := 0; i < b.N; i++ {
out, err = marshal(d)
if err != nil {
panic(err)
b.Error(err)
}
}
+6 -1
View File
@@ -147,7 +147,7 @@ bench() {
pushd "$dir"
if [ "${replace}" != "" ]; then
find ./benchmark/ -iname '*.go' -exec sed -i -E "s|github.com/pelletier/go-toml/v2|${replace}|g" {} \;
find ./benchmark/ -iname '*.go' -exec sed -i -E "s|github.com/pelletier/go-toml/v2\"|${replace}\"|g" {} \;
go get "${replace}"
fi
@@ -195,6 +195,11 @@ for line in reversed(lines[2:]):
"%.1fx" % (float(line[3])/v2), # v1
"%.1fx" % (float(line[7])/v2), # bs
])
if not results:
print("No benchmark results to display.", file=sys.stderr)
sys.exit(1)
# move geomean to the end
results.append(results[0])
del results[0]
+1
View File
@@ -1,3 +1,4 @@
// Package gotoml-test-decoder is a minimal decoder program used to compare this library with other TOML implementations.
package main
import (
@@ -1,3 +1,4 @@
// Package gotoml-test-encoder is a minimal encoder program used to compare this library with other TOML implementations.
package main
import (
@@ -24,7 +25,7 @@ func main() {
}
func usage() {
log.Printf("Usage: %s < toml-file\n", path.Base(os.Args[0]))
log.Printf("Usage: %s < json-file\n", path.Base(os.Args[0]))
flag.PrintDefaults()
os.Exit(1)
}
+4 -4
View File
@@ -34,10 +34,10 @@ Reading from a file:
jsontoml file.json > file.toml
`
var useJsonNumber bool
var useJSONNumber bool
func main() {
flag.BoolVar(&useJsonNumber, "use-json-number", false, "unmarshal numbers into `json.Number` type instead of as `float64`")
flag.BoolVar(&useJSONNumber, "use-json-number", false, "unmarshal numbers into `json.Number` type instead of as `float64`")
p := cli.Program{
Usage: usage,
@@ -52,9 +52,9 @@ func convert(r io.Reader, w io.Writer) error {
d := json.NewDecoder(r)
e := toml.NewEncoder(w)
if useJsonNumber {
if useJSONNumber {
d.UseNumber()
e.SetMarshalJsonNumbers(true)
e.SetMarshalJSONNumbers(true)
}
err := d.Decode(&v)
+3 -3
View File
@@ -14,7 +14,7 @@ func TestConvert(t *testing.T) {
input string
expected string
errors bool
useJsonNumber bool
useJSONNumber bool
}{
{
name: "valid json",
@@ -30,7 +30,7 @@ a = 42.0
},
{
name: "use json number",
useJsonNumber: true,
useJSONNumber: true,
input: `
{
"mytoml": {
@@ -50,7 +50,7 @@ a = 42
for _, e := range examples {
b := new(bytes.Buffer)
useJsonNumber = e.useJsonNumber
useJSONNumber = e.useJSONNumber
err := convert(strings.NewReader(e.input), b)
if e.errors {
assert.Error(t, err)
+2 -2
View File
@@ -2,7 +2,7 @@ package main
import (
"bytes"
"fmt"
"errors"
"io"
"strings"
"testing"
@@ -56,5 +56,5 @@ a = 42`),
type badReader struct{}
func (r *badReader) Read([]byte) (int, error) {
return 0, fmt.Errorf("reader failed on purpose")
return 0, errors.New("reader failed on purpose")
}
+21 -18
View File
@@ -18,6 +18,7 @@ import (
"strings"
"text/template"
"time"
"unicode"
)
type invalid struct {
@@ -28,7 +29,7 @@ type invalid struct {
type valid struct {
Name string
Input string
JsonRef string
JSONRef string
}
type testsCollection struct {
@@ -39,12 +40,11 @@ type testsCollection struct {
Count int
}
const srcTemplate = "// Generated by tomltestgen for toml-test ref {{.Ref}} on {{.Timestamp}}\n" +
const srcTemplate = "// Code generated by tomltestgen for toml-test ref {{.Ref}} on {{.Timestamp}}. DO NOT EDIT.\n" +
"package toml_test\n" +
" import (\n" +
" \"testing\"\n" +
")\n" +
"{{range .Invalid}}\n" +
"func TestTOMLTest_Invalid_{{.Name}}(t *testing.T) {\n" +
" input := {{.Input|gostr}}\n" +
@@ -55,28 +55,31 @@ const srcTemplate = "// Generated by tomltestgen for toml-test ref {{.Ref}} on {
"{{range .Valid}}\n" +
"func TestTOMLTest_Valid_{{.Name}}(t *testing.T) {\n" +
" input := {{.Input|gostr}}\n" +
" jsonRef := {{.JsonRef|gostr}}\n" +
" jsonRef := {{.JSONRef|gostr}}\n" +
" testgenValid(t, input, jsonRef)\n" +
"}\n" +
"{{end}}\n"
func kebabToCamel(kebab string) string {
camel := ""
var buf strings.Builder
nextUpper := true
for _, c := range kebab {
if nextUpper {
camel += strings.ToUpper(string(c))
buf.WriteRune(unicode.ToUpper(c))
nextUpper = false
} else if c == '-' {
nextUpper = true
} else if c == '/' {
nextUpper = true
camel += "_"
} else {
camel += string(c)
switch c {
case '-':
nextUpper = true
case '/':
nextUpper = true
buf.WriteByte('_')
default:
buf.WriteRune(c)
}
}
}
return camel
return buf.String()
}
func templateGoStr(input string) string {
@@ -110,7 +113,7 @@ func main() {
log.Printf("> [%s] %s\n", "invalid", name)
tomlContent, err := os.ReadFile(f)
tomlContent, err := os.ReadFile(f) // #nosec G304
if err != nil {
fmt.Printf("failed to read test file: %s\n", err)
os.Exit(1)
@@ -131,14 +134,14 @@ func main() {
log.Printf("> [%s] %s\n", "valid", name)
tomlContent, err := os.ReadFile(f)
tomlContent, err := os.ReadFile(f) // #nosec G304
if err != nil {
fmt.Printf("failed reading test file: %s\n", err)
os.Exit(1)
}
filename = strings.TrimSuffix(f, ".toml")
jsonContent, err := os.ReadFile(filename + ".json")
jsonContent, err := os.ReadFile(filename + ".json") // #nosec G304
if err != nil {
fmt.Printf("failed reading validation json: %s\n", err)
os.Exit(1)
@@ -147,7 +150,7 @@ func main() {
collection.Valid = append(collection.Valid, valid{
Name: name,
Input: string(tomlContent),
JsonRef: string(jsonContent),
JSONRef: string(jsonContent),
})
collection.Count++
}
@@ -173,7 +176,7 @@ func main() {
return
}
err = os.WriteFile(*out, outputBytes, 0o644)
err = os.WriteFile(*out, outputBytes, 0o600)
if err != nil {
panic(err)
}
+79 -95
View File
@@ -9,64 +9,60 @@ import (
"github.com/pelletier/go-toml/v2/unstable"
)
func parseInteger(b []byte) (int64, error) {
func parseInteger(b []byte, base int) (int64, error) {
if len(b) > 2 && b[0] == '0' {
switch b[1] {
case 'x':
return parseIntHex(b)
return parseIntHex(b, base)
case 'b':
return parseIntBin(b)
return parseIntBin(b, base)
case 'o':
return parseIntOct(b)
return parseIntOct(b, base)
default:
panic(fmt.Errorf("invalid base '%c', should have been checked by scanIntOrFloat", b[1]))
}
}
return parseIntDec(b)
return parseIntDec(b, base)
}
func parseLocalDate(b []byte) (LocalDate, error) {
// full-date = date-fullyear "-" date-month "-" date-mday
// date-fullyear = 4DIGIT
// date-month = 2DIGIT ; 01-12
// date-mday = 2DIGIT ; 01-28, 01-29, 01-30, 01-31 based on month/year
func parseLocalDate(b []byte, base int) (LocalDate, error) {
var date LocalDate
if len(b) != 10 || b[4] != '-' || b[7] != '-' {
return date, unstable.NewParserError(b, "dates are expected to have the format YYYY-MM-DD")
return date, unstable.NewParserError(b, base, "dates are expected to have the format YYYY-MM-DD")
}
var err error
date.Year, err = parseDecimalDigits(b[0:4])
date.Year, err = parseDecimalDigits(b[0:4], base)
if err != nil {
return LocalDate{}, err
}
date.Month, err = parseDecimalDigits(b[5:7])
date.Month, err = parseDecimalDigits(b[5:7], base+5)
if err != nil {
return LocalDate{}, err
}
date.Day, err = parseDecimalDigits(b[8:10])
date.Day, err = parseDecimalDigits(b[8:10], base+8)
if err != nil {
return LocalDate{}, err
}
if !isValidDate(date.Year, date.Month, date.Day) {
return LocalDate{}, unstable.NewParserError(b, "impossible date")
return LocalDate{}, unstable.NewParserError(b, base, "impossible date")
}
return date, nil
}
func parseDecimalDigits(b []byte) (int, error) {
func parseDecimalDigits(b []byte, base int) (int, error) {
v := 0
for i, c := range b {
if c < '0' || c > '9' {
return 0, unstable.NewParserError(b[i:i+1], "expected digit (0-9)")
return 0, unstable.NewParserError(b[i:i+1], base+i, "expected digit (0-9)")
}
v *= 10
v += int(c - '0')
@@ -75,21 +71,18 @@ func parseDecimalDigits(b []byte) (int, error) {
return v, nil
}
func parseDateTime(b []byte) (time.Time, error) {
// offset-date-time = full-date time-delim full-time
// full-time = partial-time time-offset
// time-offset = "Z" / time-numoffset
// time-numoffset = ( "+" / "-" ) time-hour ":" time-minute
dt, b, err := parseLocalDateTime(b)
func parseDateTime(b []byte, base int) (time.Time, error) {
origLen := len(b)
dt, b, err := parseLocalDateTime(b, base)
if err != nil {
return time.Time{}, err
}
tzBase := base + origLen - len(b)
var zone *time.Location
if len(b) == 0 {
// parser should have checked that when assigning the date time node
panic("date time should have a timezone")
}
@@ -99,7 +92,7 @@ func parseDateTime(b []byte) (time.Time, error) {
} else {
const dateTimeByteLen = 6
if len(b) != dateTimeByteLen {
return time.Time{}, unstable.NewParserError(b, "invalid date-time timezone")
return time.Time{}, unstable.NewParserError(b, tzBase, "invalid date-time timezone")
}
var direction int
switch b[0] {
@@ -108,27 +101,27 @@ func parseDateTime(b []byte) (time.Time, error) {
case '+':
direction = +1
default:
return time.Time{}, unstable.NewParserError(b[:1], "invalid timezone offset character")
return time.Time{}, unstable.NewParserError(b[:1], tzBase, "invalid timezone offset character")
}
if b[3] != ':' {
return time.Time{}, unstable.NewParserError(b[3:4], "expected a : separator")
return time.Time{}, unstable.NewParserError(b[3:4], tzBase+3, "expected a : separator")
}
hours, err := parseDecimalDigits(b[1:3])
hours, err := parseDecimalDigits(b[1:3], tzBase+1)
if err != nil {
return time.Time{}, err
}
if hours > 23 {
return time.Time{}, unstable.NewParserError(b[:1], "invalid timezone offset hours")
return time.Time{}, unstable.NewParserError(b[:1], tzBase, "invalid timezone offset hours")
}
minutes, err := parseDecimalDigits(b[4:6])
minutes, err := parseDecimalDigits(b[4:6], tzBase+4)
if err != nil {
return time.Time{}, err
}
if minutes > 59 {
return time.Time{}, unstable.NewParserError(b[:1], "invalid timezone offset minutes")
return time.Time{}, unstable.NewParserError(b[:1], tzBase, "invalid timezone offset minutes")
}
seconds := direction * (hours*3600 + minutes*60)
@@ -141,7 +134,7 @@ func parseDateTime(b []byte) (time.Time, error) {
}
if len(b) > 0 {
return time.Time{}, unstable.NewParserError(b, "extra bytes at the end of the timezone")
return time.Time{}, unstable.NewParserError(b, tzBase, "extra bytes at the end of the timezone")
}
t := time.Date(
@@ -157,15 +150,15 @@ func parseDateTime(b []byte) (time.Time, error) {
return t, nil
}
func parseLocalDateTime(b []byte) (LocalDateTime, []byte, error) {
func parseLocalDateTime(b []byte, base int) (LocalDateTime, []byte, error) {
var dt LocalDateTime
const localDateTimeByteMinLen = 11
if len(b) < localDateTimeByteMinLen {
return dt, nil, unstable.NewParserError(b, "local datetimes are expected to have the format YYYY-MM-DDTHH:MM:SS[.NNNNNNNNN]")
return dt, nil, unstable.NewParserError(b, base, "local datetimes are expected to have the format YYYY-MM-DDTHH:MM:SS[.NNNNNNNNN]")
}
date, err := parseLocalDate(b[:10])
date, err := parseLocalDate(b[:10], base)
if err != nil {
return dt, nil, err
}
@@ -173,10 +166,10 @@ func parseLocalDateTime(b []byte) (LocalDateTime, []byte, error) {
sep := b[10]
if sep != 'T' && sep != ' ' && sep != 't' {
return dt, nil, unstable.NewParserError(b[10:11], "datetime separator is expected to be T or a space")
return dt, nil, unstable.NewParserError(b[10:11], base+10, "datetime separator is expected to be T or a space")
}
t, rest, err := parseLocalTime(b[11:])
t, rest, err := parseLocalTime(b[11:], base+11)
if err != nil {
return dt, nil, err
}
@@ -188,53 +181,53 @@ func parseLocalDateTime(b []byte) (LocalDateTime, []byte, error) {
// parseLocalTime is a bit different because it also returns the remaining
// []byte that is didn't need. This is to allow parseDateTime to parse those
// remaining bytes as a timezone.
func parseLocalTime(b []byte) (LocalTime, []byte, error) {
func parseLocalTime(b []byte, base int) (LocalTime, []byte, error) {
var (
nspow = [10]int{0, 1e8, 1e7, 1e6, 1e5, 1e4, 1e3, 1e2, 1e1, 1e0}
t LocalTime
)
// check if b matches to have expected format HH:MM:SS[.NNNNNN]
const localTimeByteLen = 8
if len(b) < localTimeByteLen {
return t, nil, unstable.NewParserError(b, "times are expected to have the format HH:MM:SS[.NNNNNN]")
return t, nil, unstable.NewParserError(b, base, "times are expected to have the format HH:MM:SS[.NNNNNN]")
}
var err error
t.Hour, err = parseDecimalDigits(b[0:2])
t.Hour, err = parseDecimalDigits(b[0:2], base)
if err != nil {
return t, nil, err
}
if t.Hour > 23 {
return t, nil, unstable.NewParserError(b[0:2], "hour cannot be greater 23")
return t, nil, unstable.NewParserError(b[0:2], base, "hour cannot be greater 23")
}
if b[2] != ':' {
return t, nil, unstable.NewParserError(b[2:3], "expecting colon between hours and minutes")
return t, nil, unstable.NewParserError(b[2:3], base+2, "expecting colon between hours and minutes")
}
t.Minute, err = parseDecimalDigits(b[3:5])
t.Minute, err = parseDecimalDigits(b[3:5], base+3)
if err != nil {
return t, nil, err
}
if t.Minute > 59 {
return t, nil, unstable.NewParserError(b[3:5], "minutes cannot be greater 59")
return t, nil, unstable.NewParserError(b[3:5], base+3, "minutes cannot be greater 59")
}
if b[5] != ':' {
return t, nil, unstable.NewParserError(b[5:6], "expecting colon between minutes and seconds")
return t, nil, unstable.NewParserError(b[5:6], base+5, "expecting colon between minutes and seconds")
}
t.Second, err = parseDecimalDigits(b[6:8])
t.Second, err = parseDecimalDigits(b[6:8], base+6)
if err != nil {
return t, nil, err
}
if t.Second > 60 {
return t, nil, unstable.NewParserError(b[6:8], "seconds cannot be greater 60")
if t.Second > 59 {
return t, nil, unstable.NewParserError(b[6:8], base+6, "seconds cannot be greater than 59")
}
b = b[8:]
base += 8
if len(b) >= 1 && b[0] == '.' {
frac := 0
@@ -244,7 +237,7 @@ func parseLocalTime(b []byte) (LocalTime, []byte, error) {
for i, c := range b[1:] {
if !isDigit(c) {
if i == 0 {
return t, nil, unstable.NewParserError(b[0:1], "need at least one digit after fraction point")
return t, nil, unstable.NewParserError(b[0:1], base, "need at least one digit after fraction point")
}
break
}
@@ -252,13 +245,6 @@ func parseLocalTime(b []byte) (LocalTime, []byte, error) {
const maxFracPrecision = 9
if i >= maxFracPrecision {
// go-toml allows decoding fractional seconds
// beyond the supported precision of 9
// digits. It truncates the fractional component
// to the supported precision and ignores the
// remaining digits.
//
// https://github.com/pelletier/go-toml/discussions/707
continue
}
@@ -268,7 +254,7 @@ func parseLocalTime(b []byte) (LocalTime, []byte, error) {
}
if precision == 0 {
return t, nil, unstable.NewParserError(b[:1], "nanoseconds need at least one digit")
return t, nil, unstable.NewParserError(b[:1], base, "nanoseconds need at least one digit")
}
t.Nanosecond = frac * nspow[precision]
@@ -279,36 +265,35 @@ func parseLocalTime(b []byte) (LocalTime, []byte, error) {
return t, b, nil
}
//nolint:cyclop
func parseFloat(b []byte) (float64, error) {
func parseFloat(b []byte, base int) (float64, error) {
if len(b) == 4 && (b[0] == '+' || b[0] == '-') && b[1] == 'n' && b[2] == 'a' && b[3] == 'n' {
return math.NaN(), nil
}
cleaned, err := checkAndRemoveUnderscoresFloats(b)
cleaned, err := checkAndRemoveUnderscoresFloats(b, base)
if err != nil {
return 0, err
}
if cleaned[0] == '.' {
return 0, unstable.NewParserError(b, "float cannot start with a dot")
return 0, unstable.NewParserError(b, base, "float cannot start with a dot")
}
if cleaned[len(cleaned)-1] == '.' {
return 0, unstable.NewParserError(b, "float cannot end with a dot")
return 0, unstable.NewParserError(b, base, "float cannot end with a dot")
}
dotAlreadySeen := false
for i, c := range cleaned {
if c == '.' {
if dotAlreadySeen {
return 0, unstable.NewParserError(b[i:i+1], "float can have at most one decimal point")
return 0, unstable.NewParserError(b[i:i+1], base+i, "float can have at most one decimal point")
}
if !isDigit(cleaned[i-1]) {
return 0, unstable.NewParserError(b[i-1:i+1], "float decimal point must be preceded by a digit")
return 0, unstable.NewParserError(b[i-1:i+1], base+i-1, "float decimal point must be preceded by a digit")
}
if !isDigit(cleaned[i+1]) {
return 0, unstable.NewParserError(b[i:i+2], "float decimal point must be followed by a digit")
return 0, unstable.NewParserError(b[i:i+2], base+i, "float decimal point must be followed by a digit")
}
dotAlreadySeen = true
}
@@ -319,54 +304,54 @@ func parseFloat(b []byte) (float64, error) {
start = 1
}
if cleaned[start] == '0' && len(cleaned) > start+1 && isDigit(cleaned[start+1]) {
return 0, unstable.NewParserError(b, "float integer part cannot have leading zeroes")
return 0, unstable.NewParserError(b, base, "float integer part cannot have leading zeroes")
}
f, err := strconv.ParseFloat(string(cleaned), 64)
if err != nil {
return 0, unstable.NewParserError(b, "unable to parse float: %w", err)
return 0, unstable.NewParserError(b, base, "unable to parse float: %w", err)
}
return f, nil
}
func parseIntHex(b []byte) (int64, error) {
cleaned, err := checkAndRemoveUnderscoresIntegers(b[2:])
func parseIntHex(b []byte, base int) (int64, error) {
cleaned, err := checkAndRemoveUnderscoresIntegers(b[2:], base+2)
if err != nil {
return 0, err
}
i, err := strconv.ParseInt(string(cleaned), 16, 64)
if err != nil {
return 0, unstable.NewParserError(b, "couldn't parse hexadecimal number: %w", err)
return 0, unstable.NewParserError(b, base, "couldn't parse hexadecimal number: %w", err)
}
return i, nil
}
func parseIntOct(b []byte) (int64, error) {
cleaned, err := checkAndRemoveUnderscoresIntegers(b[2:])
func parseIntOct(b []byte, base int) (int64, error) {
cleaned, err := checkAndRemoveUnderscoresIntegers(b[2:], base+2)
if err != nil {
return 0, err
}
i, err := strconv.ParseInt(string(cleaned), 8, 64)
if err != nil {
return 0, unstable.NewParserError(b, "couldn't parse octal number: %w", err)
return 0, unstable.NewParserError(b, base, "couldn't parse octal number: %w", err)
}
return i, nil
}
func parseIntBin(b []byte) (int64, error) {
cleaned, err := checkAndRemoveUnderscoresIntegers(b[2:])
func parseIntBin(b []byte, base int) (int64, error) {
cleaned, err := checkAndRemoveUnderscoresIntegers(b[2:], base+2)
if err != nil {
return 0, err
}
i, err := strconv.ParseInt(string(cleaned), 2, 64)
if err != nil {
return 0, unstable.NewParserError(b, "couldn't parse binary number: %w", err)
return 0, unstable.NewParserError(b, base, "couldn't parse binary number: %w", err)
}
return i, nil
@@ -376,8 +361,8 @@ func isSign(b byte) bool {
return b == '+' || b == '-'
}
func parseIntDec(b []byte) (int64, error) {
cleaned, err := checkAndRemoveUnderscoresIntegers(b)
func parseIntDec(b []byte, base int) (int64, error) {
cleaned, err := checkAndRemoveUnderscoresIntegers(b, base)
if err != nil {
return 0, err
}
@@ -389,18 +374,18 @@ func parseIntDec(b []byte) (int64, error) {
}
if len(cleaned) > startIdx+1 && cleaned[startIdx] == '0' {
return 0, unstable.NewParserError(b, "leading zero not allowed on decimal number")
return 0, unstable.NewParserError(b, base, "leading zero not allowed on decimal number")
}
i, err := strconv.ParseInt(string(cleaned), 10, 64)
if err != nil {
return 0, unstable.NewParserError(b, "couldn't parse decimal number: %w", err)
return 0, unstable.NewParserError(b, base, "couldn't parse decimal number: %w", err)
}
return i, nil
}
func checkAndRemoveUnderscoresIntegers(b []byte) ([]byte, error) {
func checkAndRemoveUnderscoresIntegers(b []byte, base int) ([]byte, error) {
start := 0
if b[start] == '+' || b[start] == '-' {
start++
@@ -411,11 +396,11 @@ func checkAndRemoveUnderscoresIntegers(b []byte) ([]byte, error) {
}
if b[start] == '_' {
return nil, unstable.NewParserError(b[start:start+1], "number cannot start with underscore")
return nil, unstable.NewParserError(b[start:start+1], base+start, "number cannot start with underscore")
}
if b[len(b)-1] == '_' {
return nil, unstable.NewParserError(b[len(b)-1:], "number cannot end with underscore")
return nil, unstable.NewParserError(b[len(b)-1:], base+len(b)-1, "number cannot end with underscore")
}
// fast path
@@ -437,7 +422,7 @@ func checkAndRemoveUnderscoresIntegers(b []byte) ([]byte, error) {
c := b[i]
if c == '_' {
if !before {
return nil, unstable.NewParserError(b[i-1:i+1], "number must have at least one digit between underscores")
return nil, unstable.NewParserError(b[i-1:i+1], base+i-1, "number must have at least one digit between underscores")
}
before = false
} else {
@@ -449,13 +434,13 @@ func checkAndRemoveUnderscoresIntegers(b []byte) ([]byte, error) {
return cleaned, nil
}
func checkAndRemoveUnderscoresFloats(b []byte) ([]byte, error) {
func checkAndRemoveUnderscoresFloats(b []byte, base int) ([]byte, error) {
if b[0] == '_' {
return nil, unstable.NewParserError(b[0:1], "number cannot start with underscore")
return nil, unstable.NewParserError(b[0:1], base, "number cannot start with underscore")
}
if b[len(b)-1] == '_' {
return nil, unstable.NewParserError(b[len(b)-1:], "number cannot end with underscore")
return nil, unstable.NewParserError(b[len(b)-1:], base+len(b)-1, "number cannot end with underscore")
}
// fast path
@@ -478,27 +463,26 @@ func checkAndRemoveUnderscoresFloats(b []byte) ([]byte, error) {
switch c {
case '_':
if !before {
return nil, unstable.NewParserError(b[i-1:i+1], "number must have at least one digit between underscores")
return nil, unstable.NewParserError(b[i-1:i+1], base+i-1, "number must have at least one digit between underscores")
}
if i < len(b)-1 && (b[i+1] == 'e' || b[i+1] == 'E') {
return nil, unstable.NewParserError(b[i+1:i+2], "cannot have underscore before exponent")
return nil, unstable.NewParserError(b[i+1:i+2], base+i+1, "cannot have underscore before exponent")
}
before = false
case '+', '-':
// signed exponents
cleaned = append(cleaned, c)
before = false
case 'e', 'E':
if i < len(b)-1 && b[i+1] == '_' {
return nil, unstable.NewParserError(b[i+1:i+2], "cannot have underscore after exponent")
return nil, unstable.NewParserError(b[i+1:i+2], base+i+1, "cannot have underscore after exponent")
}
cleaned = append(cleaned, c)
case '.':
if i < len(b)-1 && b[i+1] == '_' {
return nil, unstable.NewParserError(b[i+1:i+2], "cannot have underscore after decimal point")
return nil, unstable.NewParserError(b[i+1:i+2], base+i+1, "cannot have underscore after decimal point")
}
if i > 0 && b[i-1] == '_' {
return nil, unstable.NewParserError(b[i-1:i], "cannot have underscore before decimal point")
return nil, unstable.NewParserError(b[i-1:i], base+i-1, "cannot have underscore before decimal point")
}
cleaned = append(cleaned, c)
default:
+5 -5
View File
@@ -5,7 +5,6 @@ import (
"strconv"
"strings"
"github.com/pelletier/go-toml/v2/internal/danger"
"github.com/pelletier/go-toml/v2/unstable"
)
@@ -58,13 +57,14 @@ func (s *StrictMissingError) String() string {
//
// Implements errors.Join() interface.
func (s *StrictMissingError) Unwrap() []error {
var errs []error
errs := make([]error, len(s.Errors))
for i := range s.Errors {
errs = append(errs, &s.Errors[i])
errs[i] = &s.Errors[i]
}
return errs
}
// Key represents a TOML key as a sequence of key parts.
type Key []string
// Error returns the error message contained in the DecodeError.
@@ -99,7 +99,7 @@ func (e *DecodeError) Key() Key {
//
//nolint:funlen
func wrapDecodeError(document []byte, de *unstable.ParserError) *DecodeError {
offset := danger.SubsliceOffset(document, de.Highlight)
offset := de.Offset
errMessage := de.Error()
errLine, errColumn := positionAtEnd(document[:offset])
@@ -259,5 +259,5 @@ func positionAtEnd(b []byte) (row int, column int) {
}
}
return
return row, column
}
+182 -6
View File
@@ -13,7 +13,6 @@ import (
//nolint:funlen
func TestDecodeError(t *testing.T) {
examples := []struct {
desc string
doc [3]string
@@ -161,18 +160,18 @@ line 5`,
for _, e := range examples {
e := e
t.Run(e.desc, func(t *testing.T) {
b := bytes.Buffer{}
b.Write([]byte(e.doc[0]))
b.WriteString(e.doc[0])
start := b.Len()
b.Write([]byte(e.doc[1]))
b.WriteString(e.doc[1])
end := b.Len()
b.Write([]byte(e.doc[2]))
b.WriteString(e.doc[2])
doc := b.Bytes()
hl := doc[start:end]
err := wrapDecodeError(doc, &unstable.ParserError{
Highlight: hl,
Offset: start,
Message: e.msg,
})
@@ -189,7 +188,6 @@ line 5`,
}
func TestDecodeError_Accessors(t *testing.T) {
e := DecodeError{
message: "foo",
line: 1,
@@ -205,6 +203,184 @@ func TestDecodeError_Accessors(t *testing.T) {
assert.Equal(t, "bar", e.String())
}
func TestDecodeError_DuplicateContent(t *testing.T) {
// This test verifies that when the same content appears multiple times
// in the document, the error correctly points to the actual location
// of the error, not the first occurrence of the content.
//
// The document has "1__2" on line 1 and "3__4" on line 2.
// Both have "__" which is invalid, but we want to ensure errors
// on line 2 report line 2, not line 1.
doc := `a = 1
b = 3__4`
var v map[string]int
err := Unmarshal([]byte(doc), &v)
var derr *DecodeError
if !errors.As(err, &derr) {
t.Fatal("error not in expected format")
}
row, col := derr.Position()
// The error should be on line 2 where "3__4" is
if row != 2 {
t.Errorf("expected error on row 2, got row %d", row)
}
// Column should point to the "__" part (after "3")
if col < 5 {
t.Errorf("expected error at column >= 5, got column %d", col)
}
}
func TestDecodeError_Position(t *testing.T) {
// Test that error positions are correctly reported for various error locations
examples := []struct {
name string
doc string
expectedRow int
minCol int
}{
{
name: "error on first line",
doc: `a = 1__2`,
expectedRow: 1,
minCol: 5,
},
{
name: "error on second line",
doc: "a = 1\nb = 2__3",
expectedRow: 2,
minCol: 5,
},
{
name: "error on third line",
doc: "a = 1\nb = 2\nc = 3__4",
expectedRow: 3,
minCol: 5,
},
{
name: "missing equals on last line without trailing newline",
doc: "a = 1\nb = 2\nc",
expectedRow: 3,
minCol: 1,
},
}
for _, e := range examples {
t.Run(e.name, func(t *testing.T) {
var v map[string]int
err := Unmarshal([]byte(e.doc), &v)
var derr *DecodeError
if !errors.As(err, &derr) {
t.Fatal("error not in expected format")
}
row, col := derr.Position()
assert.Equal(t, e.expectedRow, row)
if col < e.minCol {
t.Errorf("expected column >= %d, got %d", e.minCol, col)
}
})
}
}
func TestDecodeError_PositionAfterComments(t *testing.T) {
examples := []struct {
name string
doc string
expectedRow int
expectedCol int
errContains string
}{
{
name: "invalid key after comment",
doc: "# comment\n= \"value\"",
expectedRow: 2,
expectedCol: 1,
errContains: "invalid character at start of key",
},
{
name: "invalid key after multiple comments",
doc: "# line 1\n# line 2\n= \"value\"",
expectedRow: 3,
expectedCol: 1,
errContains: "invalid character at start of key",
},
{
name: "invalid key after valid assignment and comment",
doc: "a = 1\n# comment\n= \"value\"",
expectedRow: 3,
expectedCol: 1,
errContains: "invalid character at start of key",
},
{
name: "invalid key on first line",
doc: "= \"value\"",
expectedRow: 1,
expectedCol: 1,
errContains: "invalid character at start of key",
},
{
name: "invalid key with leading whitespace",
doc: "# comment\n = \"value\"",
expectedRow: 2,
expectedCol: 3,
errContains: "invalid character at start of key",
},
}
for _, e := range examples {
t.Run(e.name, func(t *testing.T) {
var v map[string]interface{}
err := Unmarshal([]byte(e.doc), &v)
if err == nil {
t.Fatal("expected an error")
}
var derr *DecodeError
if !errors.As(err, &derr) {
t.Fatalf("expected DecodeError, got %T: %v", err, err)
}
row, col := derr.Position()
if row != e.expectedRow {
t.Errorf("row: got %d, want %d (error: %s)", row, e.expectedRow, derr.String())
}
if col != e.expectedCol {
t.Errorf("col: got %d, want %d (error: %s)", col, e.expectedCol, derr.String())
}
if !strings.Contains(derr.Error(), e.errContains) {
t.Errorf("error %q does not contain %q", derr.Error(), e.errContains)
}
})
}
}
func TestDecodeError_HumanStringAfterComments(t *testing.T) {
doc := "# comment\n= \"value\""
var v map[string]interface{}
err := Unmarshal([]byte(doc), &v)
if err == nil {
t.Fatal("expected an error")
}
var derr *DecodeError
if !errors.As(err, &derr) {
t.Fatalf("expected DecodeError, got %T: %v", err, err)
}
human := derr.String()
if !strings.Contains(human, "= \"value\"") {
t.Errorf("human-readable error should show the offending line, got:\n%s", human)
}
if !strings.Contains(human, "2|") {
t.Errorf("human-readable error should reference line 2, got:\n%s", human)
}
}
func TestStrictErrorUnwrap(t *testing.T) {
fo := bytes.NewBufferString(`
Missing = 1
+1 -1
View File
@@ -12,7 +12,7 @@ import (
func FuzzUnmarshal(f *testing.F) {
file, err := os.ReadFile("benchmark/benchmark.toml")
if err != nil {
panic(err)
f.Error(err)
}
f.Add(file)
+31 -25
View File
@@ -1,3 +1,4 @@
// Package assert provides assertion functions for unit testing.
package assert
import (
@@ -9,66 +10,67 @@ import (
)
// True asserts that an expression is true.
func True(t testing.TB, ok bool, msgAndArgs ...any) {
func True(tb testing.TB, ok bool, msgAndArgs ...any) {
tb.Helper()
if ok {
return
}
t.Helper()
t.Fatal(formatMsgAndArgs("Expected expression to be true", msgAndArgs...))
tb.Fatal(formatMsgAndArgs("Expected expression to be true", msgAndArgs...))
}
// False asserts that an expression is false.
func False(t testing.TB, ok bool, msgAndArgs ...any) {
func False(tb testing.TB, ok bool, msgAndArgs ...any) {
tb.Helper()
if !ok {
return
}
t.Helper()
t.Fatal(formatMsgAndArgs("Expected expression to be false", msgAndArgs...))
tb.Fatal(formatMsgAndArgs("Expected expression to be false", msgAndArgs...))
}
// Equal asserts that "expected" and "actual" are equal.
func Equal[T any](t testing.TB, expected, actual T, msgAndArgs ...any) {
func Equal[T any](tb testing.TB, expected, actual T, msgAndArgs ...any) {
tb.Helper()
if objectsAreEqual(expected, actual) {
return
}
t.Helper()
msg := formatMsgAndArgs("Expected values to be equal:", msgAndArgs...)
t.Fatalf("%s\n%s", msg, diff(expected, actual))
tb.Fatalf("%s\n%s", msg, diff(expected, actual))
}
// Error asserts that an error is not nil.
func Error(t testing.TB, err error, msgAndArgs ...any) {
func Error(tb testing.TB, err error, msgAndArgs ...any) {
tb.Helper()
if err != nil {
return
}
t.Helper()
t.Fatal(formatMsgAndArgs("Expected an error", msgAndArgs...))
tb.Fatal(formatMsgAndArgs("Expected an error", msgAndArgs...))
}
// NoError asserts that an error is nil.
func NoError(t testing.TB, err error, msgAndArgs ...any) {
func NoError(tb testing.TB, err error, msgAndArgs ...any) {
tb.Helper()
if err == nil {
return
}
t.Helper()
msg := formatMsgAndArgs("Unexpected error:", msgAndArgs...)
t.Fatalf("%s\n%+v", msg, err)
tb.Fatalf("%s\n%+v", msg, err)
}
// Panics asserts that the given function panics.
func Panics(t testing.TB, fn func(), msgAndArgs ...any) {
t.Helper()
func Panics(tb testing.TB, fn func(), msgAndArgs ...any) {
tb.Helper()
defer func() {
if recover() == nil {
msg := formatMsgAndArgs("Expected function to panic", msgAndArgs...)
t.Fatal(msg)
tb.Fatal(msg)
}
}()
fn()
}
// Zero asserts that a value is its zero value.
func Zero[T any](t testing.TB, value T, msgAndArgs ...any) {
func Zero[T any](tb testing.TB, value T, msgAndArgs ...any) {
tb.Helper()
var zero T
if objectsAreEqual(value, zero) {
return
@@ -77,22 +79,26 @@ func Zero[T any](t testing.TB, value T, msgAndArgs ...any) {
if (val.Kind() == reflect.Slice || val.Kind() == reflect.Map || val.Kind() == reflect.Array) && val.Len() == 0 {
return
}
t.Helper()
msg := formatMsgAndArgs("Expected zero value but got:", msgAndArgs...)
t.Fatalf("%s\n%v", msg, value)
tb.Fatalf("%s\n%v", msg, value)
}
func NotZero[T any](t testing.TB, value T, msgAndArgs ...any) {
func NotZero[T any](tb testing.TB, value T, msgAndArgs ...any) {
tb.Helper()
var zero T
if !objectsAreEqual(value, zero) {
val := reflect.ValueOf(value)
if !((val.Kind() == reflect.Slice || val.Kind() == reflect.Map || val.Kind() == reflect.Array) && val.Len() == 0) {
switch val.Kind() {
case reflect.Slice, reflect.Map, reflect.Array:
if val.Len() > 0 {
return
}
default:
return
}
}
t.Helper()
msg := formatMsgAndArgs("Unexpected zero value:", msgAndArgs...)
t.Fatalf("%s\n%v", msg, value)
tb.Fatalf("%s\n%v", msg, value)
}
func formatMsgAndArgs(msg string, args ...any) string {
+99 -66
View File
@@ -1,6 +1,7 @@
package assert
import (
"errors"
"fmt"
"testing"
)
@@ -12,135 +13,167 @@ type Data struct {
func TestBadMessage(t *testing.T) {
invalidMessage := func() { True(t, false, 1234) }
assertOk(t, "Non-fmt message value", func(t testing.TB) {
Panics(t, invalidMessage)
assertOk(t, "Non-fmt message value", func(tb testing.TB) {
tb.Helper()
Panics(tb, invalidMessage)
})
assertFail(t, "Non-fmt message value", func(t testing.TB) {
True(t, false, "example %s", "message")
assertFail(t, "Non-fmt message value", func(tb testing.TB) {
tb.Helper()
True(tb, false, "example %s", "message")
})
}
func TestTrue(t *testing.T) {
assertOk(t, "Succeed", func(t testing.TB) {
True(t, 1 > 0)
assertOk(t, "Succeed", func(tb testing.TB) {
tb.Helper()
True(tb, 1 > 0)
})
assertFail(t, "Fail", func(t testing.TB) {
True(t, 1 < 0)
assertFail(t, "Fail", func(tb testing.TB) {
tb.Helper()
True(tb, 1 < 0)
})
}
func TestFalse(t *testing.T) {
assertOk(t, "Succeed", func(t testing.TB) {
False(t, 1 < 0)
assertOk(t, "Succeed", func(tb testing.TB) {
tb.Helper()
False(tb, 1 < 0)
})
assertFail(t, "Fail", func(t testing.TB) {
False(t, 1 > 0)
assertFail(t, "Fail", func(tb testing.TB) {
tb.Helper()
False(tb, 1 > 0)
})
}
func TestEqual(t *testing.T) {
assertOk(t, "Nil", func(t testing.TB) {
Equal(t, interface{}(nil), interface{}(nil))
assertOk(t, "Nil", func(tb testing.TB) {
tb.Helper()
Equal(tb, interface{}(nil), interface{}(nil))
})
assertOk(t, "Identical structs", func(t testing.TB) {
Equal(t, Data{"expected", 1234}, Data{"expected", 1234})
assertOk(t, "Identical structs", func(tb testing.TB) {
tb.Helper()
Equal(tb, Data{"expected", 1234}, Data{"expected", 1234})
})
assertFail(t, "Different structs", func(t testing.TB) {
Equal(t, Data{"expected", 1234}, Data{"actual", 1234})
assertFail(t, "Different structs", func(tb testing.TB) {
tb.Helper()
Equal(tb, Data{"expected", 1234}, Data{"actual", 1234})
})
assertOk(t, "Identical numbers", func(t testing.TB) {
Equal(t, 1234, 1234)
assertOk(t, "Identical numbers", func(tb testing.TB) {
tb.Helper()
Equal(tb, 1234, 1234)
})
assertFail(t, "Identical numbers", func(t testing.TB) {
Equal(t, 1234, 1324)
assertFail(t, "Identical numbers", func(tb testing.TB) {
tb.Helper()
Equal(tb, 1234, 1324)
})
assertOk(t, "Zero-length byte arrays", func(t testing.TB) {
Equal(t, []byte(nil), []byte(""))
assertOk(t, "Zero-length byte arrays", func(tb testing.TB) {
tb.Helper()
Equal(tb, []byte(nil), []byte(""))
})
assertOk(t, "Identical byte arrays", func(t testing.TB) {
Equal(t, []byte{1, 2, 3, 4}, []byte{1, 2, 3, 4})
assertOk(t, "Identical byte arrays", func(tb testing.TB) {
tb.Helper()
Equal(tb, []byte{1, 2, 3, 4}, []byte{1, 2, 3, 4})
})
assertFail(t, "Different byte arrays", func(t testing.TB) {
Equal(t, []byte{1, 2, 3, 4}, []byte{1, 3, 2, 4})
assertFail(t, "Different byte arrays", func(tb testing.TB) {
tb.Helper()
Equal(tb, []byte{1, 2, 3, 4}, []byte{1, 3, 2, 4})
})
assertOk(t, "Identical strings", func(t testing.TB) {
Equal(t, "example", "example")
assertOk(t, "Identical strings", func(tb testing.TB) {
tb.Helper()
Equal(tb, "example", "example")
})
assertFail(t, "Identical strings", func(t testing.TB) {
Equal(t, "example", "elpmaxe")
assertFail(t, "Identical strings", func(tb testing.TB) {
tb.Helper()
Equal(tb, "example", "elpmaxe")
})
}
func TestError(t *testing.T) {
assertOk(t, "Error", func(t testing.TB) {
Error(t, fmt.Errorf("example"))
assertOk(t, "Error", func(tb testing.TB) {
tb.Helper()
Error(tb, errors.New("example"))
})
assertFail(t, "Nil", func(t testing.TB) {
Error(t, nil)
assertFail(t, "Nil", func(tb testing.TB) {
tb.Helper()
Error(tb, nil)
})
}
func TestNoError(t *testing.T) {
assertFail(t, "Error", func(t testing.TB) {
NoError(t, fmt.Errorf("example"))
assertFail(t, "Error", func(tb testing.TB) {
tb.Helper()
NoError(tb, errors.New("example"))
})
assertOk(t, "Nil", func(t testing.TB) {
NoError(t, nil)
assertOk(t, "Nil", func(tb testing.TB) {
tb.Helper()
NoError(tb, nil)
})
}
func TestPanics(t *testing.T) {
willPanic := func() { panic("example") }
wontPanic := func() {}
assertOk(t, "Will panic", func(t testing.TB) {
Panics(t, willPanic)
assertOk(t, "Will panic", func(tb testing.TB) {
tb.Helper()
Panics(tb, willPanic)
})
assertFail(t, "Won't panic", func(t testing.TB) {
Panics(t, wontPanic)
assertFail(t, "Won't panic", func(tb testing.TB) {
tb.Helper()
Panics(tb, wontPanic)
})
}
func TestZero(t *testing.T) {
assertOk(t, "Empty struct", func(t testing.TB) {
Zero(t, Data{})
assertOk(t, "Empty struct", func(tb testing.TB) {
tb.Helper()
Zero(tb, Data{})
})
assertFail(t, "Non-empty struct", func(t testing.TB) {
Zero(t, Data{Label: "example"})
assertFail(t, "Non-empty struct", func(tb testing.TB) {
tb.Helper()
Zero(tb, Data{Label: "example"})
})
assertOk(t, "Nil slice", func(t testing.TB) {
assertOk(t, "Nil slice", func(tb testing.TB) {
tb.Helper()
var slice []int
Zero(t, slice)
Zero(tb, slice)
})
assertFail(t, "Non-empty slice", func(t testing.TB) {
assertFail(t, "Non-empty slice", func(tb testing.TB) {
tb.Helper()
slice := []int{1, 2, 3, 4}
Zero(t, slice)
Zero(tb, slice)
})
assertOk(t, "Zero-length slice", func(t testing.TB) {
assertOk(t, "Zero-length slice", func(tb testing.TB) {
tb.Helper()
slice := []int{}
Zero(t, slice)
Zero(tb, slice)
})
}
func TestNotZero(t *testing.T) {
assertFail(t, "Empty struct", func(t testing.TB) {
assertFail(t, "Empty struct", func(tb testing.TB) {
tb.Helper()
zero := Data{}
NotZero(t, zero)
NotZero(tb, zero)
})
assertOk(t, "Non-empty struct", func(t testing.TB) {
assertOk(t, "Non-empty struct", func(tb testing.TB) {
tb.Helper()
notZero := Data{Label: "example"}
NotZero(t, notZero)
NotZero(tb, notZero)
})
assertFail(t, "Nil slice", func(t testing.TB) {
assertFail(t, "Nil slice", func(tb testing.TB) {
tb.Helper()
var slice []int
NotZero(t, slice)
NotZero(tb, slice)
})
assertFail(t, "Zero-length slice", func(t testing.TB) {
assertFail(t, "Zero-length slice", func(tb testing.TB) {
tb.Helper()
slice := []int{}
NotZero(t, slice)
NotZero(tb, slice)
})
assertOk(t, "Non-empty slice", func(t testing.TB) {
assertOk(t, "Non-empty slice", func(tb testing.TB) {
tb.Helper()
slice := []int{1, 2, 3, 4}
NotZero(t, slice)
NotZero(tb, slice)
})
}
@@ -157,7 +190,7 @@ func (t *testCase) Fatalf(message string, args ...interface{}) {
t.failed = fmt.Sprintf(message, args...)
}
func assertFail(t *testing.T, name string, fn func(t testing.TB)) {
func assertFail(t *testing.T, name string, fn func(testing.TB)) {
t.Helper()
t.Run(name, func(t *testing.T) {
t.Helper()
@@ -171,7 +204,7 @@ func assertFail(t *testing.T, name string, fn func(t testing.TB)) {
})
}
func assertOk(t *testing.T, name string, fn func(t testing.TB)) {
func assertOk(t *testing.T, name string, fn func(testing.TB)) {
t.Helper()
t.Run(name, func(t *testing.T) {
t.Helper()
+3 -3
View File
@@ -1,6 +1,6 @@
package characters
var invalidAsciiTable = [256]bool{
var invalidASCIITable = [256]bool{
0x00: true,
0x01: true,
0x02: true,
@@ -37,6 +37,6 @@ var invalidAsciiTable = [256]bool{
0x7F: true,
}
func InvalidAscii(b byte) bool {
return invalidAsciiTable[b]
func InvalidASCII(b byte) bool {
return invalidASCIITable[b]
}
+25 -53
View File
@@ -1,20 +1,12 @@
// Package characters provides functions for working with string encodings.
package characters
import (
"unicode/utf8"
)
type utf8Err struct {
Index int
Size int
}
func (u utf8Err) Zero() bool {
return u.Size == 0
}
// Verified that a given string is only made of valid UTF-8 characters allowed
// by the TOML spec:
// Utf8TomlValidAlreadyEscaped verifies that a given string is only made of
// valid UTF-8 characters allowed by the TOML spec:
//
// Any Unicode character may be used except those that must be escaped:
// quotation mark, backslash, and the control characters other than tab (U+0000
@@ -23,8 +15,8 @@ func (u utf8Err) Zero() bool {
// It is a copy of the Go 1.17 utf8.Valid implementation, tweaked to exit early
// when a character is not allowed.
//
// The returned utf8Err is Zero() if the string is valid, or contains the byte
// index and size of the invalid character.
// The returned slice is empty if the string is valid, or contains the bytes
// of the invalid character.
//
// quotation mark => already checked
// backslash => already checked
@@ -32,85 +24,65 @@ func (u utf8Err) Zero() bool {
// 0x9 => tab, ok
// 0xA - 0x1F => invalid
// 0x7F => invalid
func Utf8TomlValidAlreadyEscaped(p []byte) (err utf8Err) {
func Utf8TomlValidAlreadyEscaped(p []byte) int {
consumed := 0
// Fast path. Check for and skip 8 bytes of ASCII characters per iteration.
offset := 0
for len(p) >= 8 {
// Combining two 32 bit loads allows the same code to be used
// for 32 and 64 bit platforms.
// The compiler can generate a 32bit load for first32 and second32
// on many platforms. See test/codegen/memcombine.go.
first32 := uint32(p[0]) | uint32(p[1])<<8 | uint32(p[2])<<16 | uint32(p[3])<<24
second32 := uint32(p[4]) | uint32(p[5])<<8 | uint32(p[6])<<16 | uint32(p[7])<<24
if (first32|second32)&0x80808080 != 0 {
// Found a non ASCII byte (>= RuneSelf).
break
}
for i, b := range p[:8] {
if InvalidAscii(b) {
err.Index = offset + i
err.Size = 1
return
if InvalidASCII(b) {
return consumed + i
}
}
p = p[8:]
offset += 8
consumed += 8
}
n := len(p)
for i := 0; i < n; {
pi := p[i]
if pi < utf8.RuneSelf {
if InvalidAscii(pi) {
err.Index = offset + i
err.Size = 1
return
if InvalidASCII(pi) {
return consumed + i
}
i++
continue
}
x := first[pi]
if x == xx {
// Illegal starter byte.
err.Index = offset + i
err.Size = 1
return
return consumed + i
}
size := int(x & 7)
if i+size > n {
// Short or invalid.
err.Index = offset + i
err.Size = n - i
return
return consumed + i
}
accept := acceptRanges[x>>4]
if c := p[i+1]; c < accept.lo || accept.hi < c {
err.Index = offset + i
err.Size = 2
return
} else if size == 2 {
return consumed + i
} else if size == 2 { //revive:disable:empty-block
} else if c := p[i+2]; c < locb || hicb < c {
err.Index = offset + i
err.Size = 3
return
} else if size == 3 {
return consumed + i
} else if size == 3 { //revive:disable:empty-block
} else if c := p[i+3]; c < locb || hicb < c {
err.Index = offset + i
err.Size = 4
return
return consumed + i
}
i += size
}
return
return -1
}
// Return the size of the next rune if valid, 0 otherwise.
// Utf8ValidNext returns the size of the next rune if valid, 0 otherwise.
func Utf8ValidNext(p []byte) int {
c := p[0]
if c < utf8.RuneSelf {
if InvalidAscii(c) {
if InvalidASCII(c) {
return 0
}
return 1
@@ -129,10 +101,10 @@ func Utf8ValidNext(p []byte) int {
accept := acceptRanges[x>>4]
if c := p[1]; c < accept.lo || accept.hi < c {
return 0
} else if size == 2 {
} else if size == 2 { //nolint:revive
} else if c := p[2]; c < locb || hicb < c {
return 0
} else if size == 3 {
} else if size == 3 { //nolint:revive
} else if c := p[3]; c < locb || hicb < c {
return 0
}
+8 -8
View File
@@ -1,3 +1,4 @@
// Package cli provides common functions for command-line programs.
package cli
import (
@@ -27,17 +28,16 @@ func (p *Program) Execute() {
os.Exit(p.main(flag.Args(), os.Stdin, os.Stdout, os.Stderr))
}
func (p *Program) main(files []string, input io.Reader, output, error io.Writer) int {
func (p *Program) main(files []string, input io.Reader, output, stderr io.Writer) int {
err := p.run(files, input, output)
if err != nil {
var derr *toml.DecodeError
if errors.As(err, &derr) {
fmt.Fprintln(error, derr.String())
_, _ = fmt.Fprintln(stderr, derr.String())
row, col := derr.Position()
fmt.Fprintln(error, "error occurred at row", row, "column", col)
_, _ = fmt.Fprintln(stderr, "error occurred at row", row, "column", col)
} else {
fmt.Fprintln(error, err.Error())
_, _ = fmt.Fprintln(stderr, err.Error())
}
return -1
@@ -54,7 +54,7 @@ func (p *Program) run(files []string, input io.Reader, output io.Writer) error {
if err != nil {
return err
}
defer f.Close()
defer func() { _ = f.Close() }()
input = f
}
return p.Fn(input, output)
@@ -71,7 +71,7 @@ func (p *Program) runAllFilesInPlace(files []string) error {
}
func (p *Program) runFileInPlace(path string) error {
in, err := os.ReadFile(path)
in, err := os.ReadFile(path) // #nosec G304
if err != nil {
return err
}
@@ -83,5 +83,5 @@ func (p *Program) runFileInPlace(path string) error {
return err
}
return os.WriteFile(path, out.Bytes(), 0600)
return os.WriteFile(path, out.Bytes(), 0o600)
}
+16 -20
View File
@@ -2,7 +2,7 @@ package cli
import (
"bytes"
"fmt"
"errors"
"io"
"os"
"path"
@@ -23,7 +23,7 @@ func TestProcessMainStdin(t *testing.T) {
stderr := new(bytes.Buffer)
input := strings.NewReader("this is the input")
exit := processMain([]string{}, input, stdout, stderr, func(r io.Reader, w io.Writer) error {
exit := processMain([]string{}, input, stdout, stderr, func(io.Reader, io.Writer) error {
return nil
})
@@ -37,8 +37,8 @@ func TestProcessMainStdinErr(t *testing.T) {
stderr := new(bytes.Buffer)
input := strings.NewReader("this is the input")
exit := processMain([]string{}, input, stdout, stderr, func(r io.Reader, w io.Writer) error {
return fmt.Errorf("something bad")
exit := processMain([]string{}, input, stdout, stderr, func(io.Reader, io.Writer) error {
return errors.New("something bad")
})
assert.Equal(t, -1, exit)
@@ -51,7 +51,7 @@ func TestProcessMainStdinDecodeErr(t *testing.T) {
stderr := new(bytes.Buffer)
input := strings.NewReader("this is the input")
exit := processMain([]string{}, input, stdout, stderr, func(r io.Reader, w io.Writer) error {
exit := processMain([]string{}, input, stdout, stderr, func(io.Reader, io.Writer) error {
var v interface{}
return toml.Unmarshal([]byte(`qwe = 001`), &v)
})
@@ -62,16 +62,16 @@ func TestProcessMainStdinDecodeErr(t *testing.T) {
}
func TestProcessMainFileExists(t *testing.T) {
tmpfile, err := os.CreateTemp("", "example")
tmpfile, err := os.CreateTemp(t.TempDir(), "example")
assert.NoError(t, err)
defer os.Remove(tmpfile.Name())
_, err = tmpfile.Write([]byte(`some data`))
_, err = tmpfile.WriteString(`some data`)
assert.NoError(t, err)
assert.NoError(t, tmpfile.Close())
stdout := new(bytes.Buffer)
stderr := new(bytes.Buffer)
exit := processMain([]string{tmpfile.Name()}, nil, stdout, stderr, func(r io.Reader, w io.Writer) error {
exit := processMain([]string{tmpfile.Name()}, nil, stdout, stderr, func(io.Reader, io.Writer) error {
return nil
})
@@ -84,7 +84,7 @@ func TestProcessMainFileDoesNotExist(t *testing.T) {
stdout := new(bytes.Buffer)
stderr := new(bytes.Buffer)
exit := processMain([]string{"/lets/hope/this/does/not/exist"}, nil, stdout, stderr, func(r io.Reader, w io.Writer) error {
exit := processMain([]string{"/lets/hope/this/does/not/exist"}, nil, stdout, stderr, func(io.Reader, io.Writer) error {
return nil
})
@@ -94,16 +94,14 @@ func TestProcessMainFileDoesNotExist(t *testing.T) {
}
func TestProcessMainFilesInPlace(t *testing.T) {
dir, err := os.MkdirTemp("", "")
assert.NoError(t, err)
defer os.RemoveAll(dir)
dir := t.TempDir()
path1 := path.Join(dir, "file1")
path2 := path.Join(dir, "file2")
err = os.WriteFile(path1, []byte("content 1"), 0600)
err := os.WriteFile(path1, []byte("content 1"), 0o600)
assert.NoError(t, err)
err = os.WriteFile(path2, []byte("content 2"), 0600)
err = os.WriteFile(path2, []byte("content 2"), 0o600)
assert.NoError(t, err)
p := Program{
@@ -136,17 +134,15 @@ func TestProcessMainFilesInPlaceErrRead(t *testing.T) {
}
func TestProcessMainFilesInPlaceFailFn(t *testing.T) {
dir, err := os.MkdirTemp("", "")
assert.NoError(t, err)
defer os.RemoveAll(dir)
dir := t.TempDir()
path1 := path.Join(dir, "file1")
err = os.WriteFile(path1, []byte("content 1"), 0600)
err := os.WriteFile(path1, []byte("content 1"), 0o600)
assert.NoError(t, err)
p := Program{
Fn: func(io.Reader, io.Writer) error { return fmt.Errorf("oh no") },
Fn: func(io.Reader, io.Writer) error { return errors.New("oh no") },
Inplace: true,
}
-65
View File
@@ -1,65 +0,0 @@
package danger
import (
"fmt"
"reflect"
"unsafe"
)
const maxInt = uintptr(int(^uint(0) >> 1))
func SubsliceOffset(data []byte, subslice []byte) int {
datap := (*reflect.SliceHeader)(unsafe.Pointer(&data))
hlp := (*reflect.SliceHeader)(unsafe.Pointer(&subslice))
if hlp.Data < datap.Data {
panic(fmt.Errorf("subslice address (%d) is before data address (%d)", hlp.Data, datap.Data))
}
offset := hlp.Data - datap.Data
if offset > maxInt {
panic(fmt.Errorf("slice offset larger than int (%d)", offset))
}
intoffset := int(offset)
if intoffset > datap.Len {
panic(fmt.Errorf("slice offset (%d) is farther than data length (%d)", intoffset, datap.Len))
}
if intoffset+hlp.Len > datap.Len {
panic(fmt.Errorf("slice ends (%d+%d) is farther than data length (%d)", intoffset, hlp.Len, datap.Len))
}
return intoffset
}
func BytesRange(start []byte, end []byte) []byte {
if start == nil || end == nil {
panic("cannot call BytesRange with nil")
}
startp := (*reflect.SliceHeader)(unsafe.Pointer(&start))
endp := (*reflect.SliceHeader)(unsafe.Pointer(&end))
if startp.Data > endp.Data {
panic(fmt.Errorf("start pointer address (%d) is after end pointer address (%d)", startp.Data, endp.Data))
}
l := startp.Len
endLen := int(endp.Data-startp.Data) + endp.Len
if endLen > l {
l = endLen
}
if l > startp.Cap {
panic(fmt.Errorf("range length is larger than capacity"))
}
return start[:l]
}
func Stride(ptr unsafe.Pointer, size uintptr, offset int) unsafe.Pointer {
// TODO: replace with unsafe.Add when Go 1.17 is released
// https://github.com/golang/go/issues/40481
return unsafe.Pointer(uintptr(ptr) + uintptr(int(size)*offset))
}
-176
View File
@@ -1,176 +0,0 @@
package danger_test
import (
"testing"
"unsafe"
"github.com/pelletier/go-toml/v2/internal/assert"
"github.com/pelletier/go-toml/v2/internal/danger"
)
func TestSubsliceOffsetValid(t *testing.T) {
examples := []struct {
desc string
test func() ([]byte, []byte)
offset int
}{
{
desc: "simple",
test: func() ([]byte, []byte) {
data := []byte("hello")
return data, data[1:]
},
offset: 1,
},
}
for _, e := range examples {
t.Run(e.desc, func(t *testing.T) {
d, s := e.test()
offset := danger.SubsliceOffset(d, s)
assert.Equal(t, e.offset, offset)
})
}
}
func TestSubsliceOffsetInvalid(t *testing.T) {
examples := []struct {
desc string
test func() ([]byte, []byte)
}{
{
desc: "unrelated arrays",
test: func() ([]byte, []byte) {
return []byte("one"), []byte("two")
},
},
{
desc: "slice starts before data",
test: func() ([]byte, []byte) {
full := []byte("hello world")
return full[5:], full[1:]
},
},
{
desc: "slice starts after data",
test: func() ([]byte, []byte) {
full := []byte("hello world")
return full[:3], full[5:]
},
},
{
desc: "slice ends after data",
test: func() ([]byte, []byte) {
full := []byte("hello world")
return full[:5], full[3:8]
},
},
}
for _, e := range examples {
t.Run(e.desc, func(t *testing.T) {
d, s := e.test()
assert.Panics(t, func() {
danger.SubsliceOffset(d, s)
})
})
}
}
func TestStride(t *testing.T) {
a := []byte{1, 2, 3, 4}
x := &a[1]
n := (*byte)(danger.Stride(unsafe.Pointer(x), unsafe.Sizeof(byte(0)), 1))
assert.Equal(t, &a[2], n)
n = (*byte)(danger.Stride(unsafe.Pointer(x), unsafe.Sizeof(byte(0)), -1))
assert.Equal(t, &a[0], n)
}
func TestBytesRange(t *testing.T) {
type fn = func() ([]byte, []byte)
examples := []struct {
desc string
test fn
expected []byte
}{
{
desc: "simple",
test: func() ([]byte, []byte) {
full := []byte("hello world")
return full[1:3], full[6:8]
},
expected: []byte("ello wo"),
},
{
desc: "full",
test: func() ([]byte, []byte) {
full := []byte("hello world")
return full[0:1], full[len(full)-1:]
},
expected: []byte("hello world"),
},
{
desc: "end before start",
test: func() ([]byte, []byte) {
full := []byte("hello world")
return full[len(full)-1:], full[0:1]
},
},
{
desc: "nils",
test: func() ([]byte, []byte) {
return nil, nil
},
},
{
desc: "nils start",
test: func() ([]byte, []byte) {
return nil, []byte("foo")
},
},
{
desc: "nils end",
test: func() ([]byte, []byte) {
return []byte("foo"), nil
},
},
{
desc: "start is end",
test: func() ([]byte, []byte) {
full := []byte("hello world")
return full[1:3], full[1:3]
},
expected: []byte("el"),
},
{
desc: "end contained in start",
test: func() ([]byte, []byte) {
full := []byte("hello world")
return full[1:7], full[2:4]
},
expected: []byte("ello w"),
},
{
desc: "different backing arrays",
test: func() ([]byte, []byte) {
one := []byte("hello world")
two := []byte("hello world")
return one, two
},
},
}
for _, e := range examples {
t.Run(e.desc, func(t *testing.T) {
start, end := e.test()
if e.expected == nil {
assert.Panics(t, func() {
danger.BytesRange(start, end)
})
} else {
res := danger.BytesRange(start, end)
assert.Equal(t, e.expected, res)
}
})
}
}
-23
View File
@@ -1,23 +0,0 @@
package danger
import (
"reflect"
"unsafe"
)
// typeID is used as key in encoder and decoder caches to enable using
// the optimize runtime.mapaccess2_fast64 function instead of the more
// expensive lookup if we were to use reflect.Type as map key.
//
// typeID holds the pointer to the reflect.Type value, which is unique
// in the program.
//
// https://github.com/segmentio/encoding/blob/master/json/codec.go#L59-L61
type TypeID unsafe.Pointer
func MakeTypeID(t reflect.Type) TypeID {
// reflect.Type has the fields:
// typ unsafe.Pointer
// ptr unsafe.Pointer
return TypeID((*[2]unsafe.Pointer)(unsafe.Pointer(&t))[1])
}
@@ -1,4 +1,4 @@
package imported_tests
package imported_tests //revive:disable:var-naming
// Those tests have been imported from v1, but adjust to match the new
// defaults of v2.
@@ -21,12 +21,12 @@ func TestDocMarshal(t *testing.T) {
Subdocs testDocSubs `toml:"subdoc"`
Basics testDocBasics `toml:"basic"`
SubDocList []testSubDoc `toml:"subdoclist"`
err int `toml:"shouldntBeHere"`
err int `toml:"shouldntBeHere"` //nolint:unused
unexported int `toml:"shouldntBeHere"`
Unexported2 int `toml:"-"`
}
var docData = testDoc{
docData := testDoc{
Title: "TOML Marshal Testing",
unexported: 0,
Unexported2: 0,
@@ -128,8 +128,7 @@ String2 = 'Two'
String2 = 'Three'
`
assert.Equal(t, string(expected), string(result))
assert.Equal(t, expected, string(result))
}
func TestEmptyMarshal(t *testing.T) {
@@ -164,7 +163,7 @@ stringlist = []
[map]
`
assert.Equal(t, string(expected), string(result))
assert.Equal(t, expected, string(result))
}
type textMarshaler struct {
@@ -1,4 +1,4 @@
package imported_tests
package imported_tests //revive:disable:var-naming
// Those tests were imported directly from go-toml v1
// https://raw.githubusercontent.com/pelletier/go-toml/a2e52561804c6cd9392ebf0048ca64fe4af67a43/marshal_test.go
@@ -149,9 +149,6 @@ type quotedKeyMarshalTestStruct struct {
SubList []basicMarshalTestSubStruct `toml:"W.sublist-𝟘"`
}
// TODO: Remove nolint once var is used by a test
//
//nolint:deadcode,unused,varcheck
var quotedKeyMarshalTestData = quotedKeyMarshalTestStruct{
String: "Hello",
Float: 3.5,
@@ -161,7 +158,7 @@ var quotedKeyMarshalTestData = quotedKeyMarshalTestStruct{
// TODO: Remove nolint once var is used by a test
//
//nolint:deadcode,unused,varcheck
//nolint:unused
var quotedKeyMarshalTestToml = []byte(`"Yfloat-𝟘" = 3.5
"Z.string-àéù" = "Hello"
@@ -183,11 +180,12 @@ type testDoc struct {
Subdocs testDocSubs `toml:"subdoc"`
Basics testDocBasics `toml:"basic"`
SubDocList []testSubDoc `toml:"subdoclist"`
err int `toml:"shouldntBeHere"` // nolint:structcheck,unused
err int `toml:"shouldntBeHere"` //nolint:unused
unexported int `toml:"shouldntBeHere"`
Unexported2 int `toml:"-"`
}
//nolint:unused
type testMapDoc struct {
Title string `toml:"title"`
BasicMap map[string]string `toml:"basic_map"`
@@ -274,7 +272,7 @@ var docData = testDoc{
// TODO: Remove nolint once var is used by a test
//
//nolint:deadcode,unused,varcheck
//nolint:unused
var mapTestDoc = testMapDoc{
Title: "TOML Marshal Testing",
BasicMap: map[string]string{
@@ -543,31 +541,35 @@ func TestNestedUnmarshal(t *testing.T) {
assert.Equal(t, nestedTestData, result)
}
//nolint:unused
type customMarshalerParent struct {
Self customMarshaler `toml:"me"`
Friends []customMarshaler `toml:"friends"`
}
//nolint:unused
type customMarshaler struct {
FirstName string
LastName string
}
//nolint:unused
func (c customMarshaler) MarshalTOML() ([]byte, error) {
fullName := fmt.Sprintf("%s %s", c.FirstName, c.LastName)
return []byte(fullName), nil
}
//nolint:unused
var customMarshalerData = customMarshaler{FirstName: "Sally", LastName: "Fields"}
// TODO: Remove nolint once var is used by a test
//
//nolint:deadcode,unused,varcheck
//nolint:unused
var customMarshalerToml = []byte(`Sally Fields`)
// TODO: Remove nolint once var is used by a test
//
//nolint:deadcode,unused,varcheck
//nolint:unused
var nestedCustomMarshalerData = customMarshalerParent{
Self: customMarshaler{FirstName: "Maiku", LastName: "Suteda"},
Friends: []customMarshaler{customMarshalerData},
@@ -575,7 +577,7 @@ var nestedCustomMarshalerData = customMarshalerParent{
// TODO: Remove nolint once var is used by a test
//
//nolint:deadcode,unused,varcheck
//nolint:unused
var nestedCustomMarshalerToml = []byte(`friends = ["Sally Fields"]
me = "Maiku Suteda"
`)
@@ -590,7 +592,7 @@ func (x *IntOrString) MarshalTOML() ([]byte, error) {
s := *(*string)(x)
_, err := strconv.Atoi(s)
if err != nil {
return []byte(fmt.Sprintf(`"%s"`, s)), nil
return []byte(fmt.Sprintf(`"%s"`, s)), nil //nolint:nilerr
}
return []byte(s), nil
}
@@ -662,7 +664,7 @@ func (m *textPointerMarshaler) MarshalText() ([]byte, error) {
// TODO: Remove nolint once var is used by a test
//
//nolint:deadcode,unused,varcheck
//nolint:unused
var commentTestToml = []byte(`
# it's a comment on type
[postgres]
@@ -687,6 +689,7 @@ var commentTestToml = []byte(`
My = "Baar"
`)
//nolint:unused
type mapsTestStruct struct {
Simple map[string]string
Paths map[string]string
@@ -700,7 +703,7 @@ type mapsTestStruct struct {
// TODO: Remove nolint once var is used by a test
//
//nolint:deadcode,unused,varcheck
//nolint:unused
var mapsTestData = mapsTestStruct{
Simple: map[string]string{
"one plus one": "two",
@@ -724,7 +727,7 @@ var mapsTestData = mapsTestStruct{
// TODO: Remove nolint once var is used by a test
//
//nolint:deadcode,unused,varcheck
//nolint:unused
var mapsTestToml = []byte(`
[Other]
"testing" = 3.9999
@@ -747,7 +750,7 @@ var mapsTestToml = []byte(`
// TODO: Remove nolint once type is used by a test
//
//nolint:deadcode,unused
//nolint:unused
type structArrayNoTag struct {
A struct {
B []int64
@@ -757,7 +760,7 @@ type structArrayNoTag struct {
// TODO: Remove nolint once var is used by a test
//
//nolint:deadcode,unused,varcheck
//nolint:unused
var customTagTestToml = []byte(`
[postgres]
password = "bvalue"
@@ -772,7 +775,7 @@ var customTagTestToml = []byte(`
// TODO: Remove nolint once var is used by a test
//
//nolint:deadcode,unused,varcheck
//nolint:unused
var customCommentTagTestToml = []byte(`
# db connection
[postgres]
@@ -786,7 +789,7 @@ var customCommentTagTestToml = []byte(`
// TODO: Remove nolint once var is used by a test
//
//nolint:deadcode,unused,varcheck
//nolint:unused
var customCommentedTagTestToml = []byte(`
[postgres]
# password = "bvalue"
@@ -841,7 +844,7 @@ func TestUnmarshalTabInStringAndQuotedKey(t *testing.T) {
// TODO: Remove nolint once var is used by a test
//
//nolint:deadcode,unused,varcheck
//nolint:unused
var customMultilineTagTestToml = []byte(`int_slice = [
1,
2,
@@ -851,7 +854,7 @@ var customMultilineTagTestToml = []byte(`int_slice = [
// TODO: Remove nolint once var is used by a test
//
//nolint:deadcode,unused,varcheck
//nolint:unused
var testDocBasicToml = []byte(`
[document]
bool_val = true
@@ -862,16 +865,12 @@ var testDocBasicToml = []byte(`
uint_val = 5001
`)
// TODO: Remove nolint once type is used by a test
//
//nolint:deadcode
//nolint:unused
type testDocCustomTag struct {
Doc testDocBasicsCustomTag `file:"document"`
}
// TODO: Remove nolint once type is used by a test
//
//nolint:deadcode
//nolint:unused
type testDocBasicsCustomTag struct {
Bool bool `file:"bool_val"`
Date time.Time `file:"date_val"`
@@ -882,9 +881,7 @@ type testDocBasicsCustomTag struct {
unexported int `file:"shouldntBeHere"`
}
// TODO: Remove nolint once var is used by a test
//
//nolint:deadcode,varcheck
//nolint:unused
var testDocCustomTagData = testDocCustomTag{
Doc: testDocBasicsCustomTag{
Bool: true,
@@ -987,13 +984,13 @@ func TestUnmarshalInvalidPointerKind(t *testing.T) {
// TODO: Remove nolint once var is used by a test
//
//nolint:deadcode,unused
//nolint:unused
type testDuration struct {
Nanosec time.Duration `toml:"nanosec"`
Microsec1 time.Duration `toml:"microsec1"`
Microsec2 *time.Duration `toml:"microsec2"`
Millisec time.Duration `toml:"millisec"`
Sec time.Duration `toml:"sec"`
Sec time.Duration `toml:"sec"` //nolint:staticcheck
Min time.Duration `toml:"min"`
Hour time.Duration `toml:"hour"`
Mixed time.Duration `toml:"mixed"`
@@ -1002,7 +999,7 @@ type testDuration struct {
// TODO: Remove nolint once var is used by a test
//
//nolint:deadcode,unused,varcheck
//nolint:unused
var testDurationToml = []byte(`
nanosec = "1ns"
microsec1 = "1us"
@@ -1017,7 +1014,7 @@ a_string = "15s"
// TODO: Remove nolint once var is used by a test
//
//nolint:deadcode,unused,varcheck
//nolint:unused
var testDurationToml2 = []byte(`a_string = "15s"
hour = "1h0m0s"
microsec1 = "1µs"
@@ -1031,15 +1028,14 @@ sec = "1s"
// TODO: Remove nolint once type is used by a test
//
//nolint:deadcode,unused
//nolint:unused
type testBadDuration struct {
Val time.Duration `toml:"val"`
}
// TODO: add back camelCase test
var testCamelCaseKeyToml = []byte(`fooBar = 10`) //nolint:unused
var testCamelCaseKeyToml = []byte(`fooBar = 10`)
//nolint:unused
func TestUnmarshalCamelCaseKey(t *testing.T) {
t.Skipf("don't know if it is a good idea to automatically convert like that yet")
var x struct {
@@ -1058,7 +1054,7 @@ func TestUnmarshalCamelCaseKey(t *testing.T) {
func TestUnmarshalNegativeUint(t *testing.T) {
t.Skipf("not sure if we this should always error")
type check struct{ U uint } // nolint:unused
type check struct{ U uint }
err := toml.Unmarshal([]byte("U = -1"), &check{})
assert.Error(t, err)
}
@@ -1535,7 +1531,7 @@ func TestUnmarshalLocalDateTime(t *testing.T) {
}
for i, example := range examples {
doc := fmt.Sprintf(`date = %s`, example.in)
doc := "date = " + example.in
t.Run(fmt.Sprintf("ToLocalDateTime_%d_%s", i, example.name), func(t *testing.T) {
type dateStruct struct {
@@ -1621,7 +1617,7 @@ func TestUnmarshalLocalTime(t *testing.T) {
}
for i, example := range examples {
doc := fmt.Sprintf(`Time = %s`, example.in)
doc := "Time = " + example.in
t.Run(fmt.Sprintf("ToLocalTime_%d_%s", i, example.name), func(t *testing.T) {
type dateStruct struct {
@@ -1906,19 +1902,12 @@ func TestUnmarshalMixedTypeSlice(t *testing.T) {
ArrayField []interface{}
}
//doc := []byte(`ArrayField = [3.14,100,true,"hello world",{Field = "inner1"},[{Field = "inner2"},{Field = "inner3"}]]
//`)
doc := []byte(`ArrayField = [{Field = "inner1"},[{Field = "inner2"},{Field = "inner3"}]]
`)
actual := TestStruct{}
expected := TestStruct{
ArrayField: []interface{}{
//3.14,
//int64(100),
//true,
//"hello world",
map[string]interface{}{
"Field": "inner1",
},
@@ -2004,9 +1993,10 @@ func TestDecoderStrict(t *testing.T) {
"Expected a *toml.StrictMissingError, got: %v", reflect.TypeOf(err),
)
se := err.(*toml.StrictMissingError)
var se *toml.StrictMissingError
assert.True(t, errors.As(err, &se))
keys := []toml.Key{}
keys := make([]toml.Key, 0, len(se.Errors))
for _, e := range se.Errors {
keys = append(keys, e.Key())
@@ -2026,6 +2016,7 @@ func TestDecoderStrict(t *testing.T) {
var m map[string]interface{}
err = decoder(input).Decode(&m)
assert.NoError(t, err)
}
func TestDecoderStrictValid(t *testing.T) {
@@ -2062,19 +2053,6 @@ func (d *docUnmarshalTOML) UnmarshalTOML(i interface{}) error {
return nil
}
func TestDecoderStrictCustomUnmarshal(t *testing.T) {
t.Skip()
//input := `key = "ok"`
//var doc docUnmarshalTOML
//err := NewDecoder(bytes.NewReader([]byte(input))).Strict(true).Decode(&doc)
//if err != nil {
// t.Fatal("unexpected error:", err)
//}
//if doc.Decoded.Key != "ok" {
// t.Errorf("Bad unmarshal: expected ok, got %v", doc.Decoded.Key)
//}
}
type parent struct {
Doc docUnmarshalTOML
DocPointer *docUnmarshalTOML
@@ -2278,7 +2256,7 @@ type Custom struct {
v string
}
func (c *Custom) UnmarshalTOML(v interface{}) error {
func (c *Custom) UnmarshalTOML(interface{}) error {
c.v = "called"
return nil
}
@@ -2303,14 +2281,14 @@ type durationString struct {
time.Duration
}
func (d *durationString) UnmarshalTOML(v interface{}) error {
func (d *durationString) UnmarshalTOML(interface{}) error {
d.Duration = 10 * time.Second
return nil
}
type config437Error struct{}
func (e *config437Error) UnmarshalTOML(v interface{}) error {
func (e *config437Error) UnmarshalTOML(interface{}) error {
return errors.New("expected")
}
+8 -7
View File
@@ -3,17 +3,18 @@ package testsuite
import (
"fmt"
"math"
"strconv"
"time"
"github.com/pelletier/go-toml/v2"
)
// addTag adds JSON tags to a data structure as expected by toml-test.
func addTag(key string, tomlData interface{}) interface{} {
func addTag(tomlData interface{}) interface{} {
// Switch on the data type.
switch orig := tomlData.(type) {
default:
//return map[string]interface{}{}
// return map[string]interface{}{}
panic(fmt.Sprintf("Unknown type: %T", tomlData))
// A table: we don't need to add any tags, just recurse for every table
@@ -21,7 +22,7 @@ func addTag(key string, tomlData interface{}) interface{} {
case map[string]interface{}:
typed := make(map[string]interface{}, len(orig))
for k, v := range orig {
typed[k] = addTag(k, v)
typed[k] = addTag(v)
}
return typed
@@ -30,13 +31,13 @@ func addTag(key string, tomlData interface{}) interface{} {
case []map[string]interface{}:
typed := make([]map[string]interface{}, len(orig))
for i, v := range orig {
typed[i] = addTag("", v).(map[string]interface{})
typed[i] = addTag(v).(map[string]interface{})
}
return typed
case []interface{}:
typed := make([]interface{}, len(orig))
for i, v := range orig {
typed[i] = addTag("", v)
typed[i] = addTag(v)
}
return typed
@@ -52,11 +53,11 @@ func addTag(key string, tomlData interface{}) interface{} {
// Tag primitive values: bool, string, int, and float64.
case bool:
return tag("bool", fmt.Sprintf("%v", orig))
return tag("bool", strconv.FormatBool(orig))
case string:
return tag("string", orig)
case int64:
return tag("integer", fmt.Sprintf("%d", orig))
return tag("integer", strconv.FormatInt(orig, 10))
case float64:
// Special case for nan since NaN == NaN is false.
if math.IsNaN(orig) {
+10 -10
View File
@@ -9,6 +9,7 @@ import (
)
func CmpJSON(t *testing.T, key string, want, have interface{}) {
t.Helper()
switch w := want.(type) {
case map[string]interface{}:
cmpJSONMaps(t, key, w, have)
@@ -22,6 +23,7 @@ func CmpJSON(t *testing.T, key string, want, have interface{}) {
}
func cmpJSONMaps(t *testing.T, key string, want map[string]interface{}, have interface{}) {
t.Helper()
haveMap, ok := have.(map[string]interface{})
if !ok {
mismatch(t, key, "table", want, haveMap)
@@ -61,6 +63,7 @@ func cmpJSONMaps(t *testing.T, key string, want map[string]interface{}, have int
}
func cmpJSONArrays(t *testing.T, key string, want, have interface{}) {
t.Helper()
wantSlice, ok := want.([]interface{})
if !ok {
panic(fmt.Sprintf("'value' should be a JSON array when 'type=array', but it is a %T", want))
@@ -83,6 +86,7 @@ func cmpJSONArrays(t *testing.T, key string, want, have interface{}) {
}
func cmpJSONValues(t *testing.T, key string, want, have map[string]interface{}) {
t.Helper()
wantType, ok := want["type"].(string)
if !ok {
panic(fmt.Sprintf("'type' should be a string, but it is a %T", want["type"]))
@@ -126,6 +130,7 @@ func cmpJSONValues(t *testing.T, key string, want, have map[string]interface{})
}
func cmpAsStrings(t *testing.T, key string, want, have string) {
t.Helper()
if want != have {
t.Fatalf("Values for key '%s' don't match:\n"+
" Expected: %s\n"+
@@ -135,6 +140,7 @@ func cmpAsStrings(t *testing.T, key string, want, have string) {
}
func cmpFloats(t *testing.T, key string, want, have string) {
t.Helper()
// Special case for NaN, since NaN != NaN.
if strings.HasSuffix(want, "nan") || strings.HasSuffix(have, "nan") {
if want != have {
@@ -177,6 +183,7 @@ var layouts = map[string]string{
}
func cmpAsDatetimes(t *testing.T, key string, kind, want, have string) {
t.Helper()
layout, ok := layouts[kind]
if !ok {
panic("should never happen")
@@ -200,15 +207,6 @@ func cmpAsDatetimes(t *testing.T, key string, kind, want, have string) {
}
}
func cmpAsDatetimesLocal(t *testing.T, key string, want, have string) {
if datetimeRepl.Replace(want) != datetimeRepl.Replace(have) {
t.Fatalf("Values for key '%s' don't match:\n"+
" Expected: %v\n"+
" Your encoder: %v",
key, want, have)
}
}
func kjoin(old, key string) string {
if len(old) == 0 {
return key
@@ -230,6 +228,7 @@ func isValue(m map[string]interface{}) bool {
}
func mismatch(t *testing.T, key string, wantType string, want, have interface{}) {
t.Helper()
t.Fatalf("Key '%s' is not an %s but %[4]T:\n"+
" Expected: %#[3]v\n"+
" Your encoder: %#[4]v",
@@ -237,8 +236,9 @@ func mismatch(t *testing.T, key string, wantType string, want, have interface{})
}
func valMismatch(t *testing.T, key string, wantType, haveType string, want, have interface{}) {
t.Helper()
t.Fatalf("Key '%s' is not an %s but %s:\n"+
" Expected: %#[3]v\n"+
" Your encoder: %#[4]v",
key, wantType, want, have)
key, wantType, haveType, want, have)
}
-69
View File
@@ -1,69 +0,0 @@
package testsuite
import (
"bytes"
"encoding/json"
"fmt"
"github.com/pelletier/go-toml/v2"
)
type parser struct{}
func (p parser) Decode(input string) (output string, outputIsError bool, retErr error) {
defer func() {
if r := recover(); r != nil {
switch rr := r.(type) {
case error:
retErr = rr
default:
retErr = fmt.Errorf("%s", rr)
}
}
}()
var v interface{}
if err := toml.Unmarshal([]byte(input), &v); err != nil {
return err.Error(), true, nil
}
j, err := json.MarshalIndent(addTag("", v), "", " ")
if err != nil {
return "", false, retErr
}
return string(j), false, retErr
}
func (p parser) Encode(input string) (output string, outputIsError bool, retErr error) {
defer func() {
if r := recover(); r != nil {
switch rr := r.(type) {
case error:
retErr = rr
default:
retErr = fmt.Errorf("%s", rr)
}
}
}()
var tmp interface{}
err := json.Unmarshal([]byte(input), &tmp)
if err != nil {
return "", false, err
}
rm, err := rmTag(tmp)
if err != nil {
return err.Error(), true, retErr
}
buf := new(bytes.Buffer)
err = toml.NewEncoder(buf).Encode(rm)
if err != nil {
return err.Error(), true, retErr
}
return buf.String(), false, retErr
}
+4 -17
View File
@@ -9,7 +9,7 @@ import (
)
// Remove JSON tags to a data structure as returned by toml-test.
func rmTag(typedJson interface{}) (interface{}, error) {
func rmTag(typedJSON interface{}) (interface{}, error) {
// Check if key is in the table m.
in := func(key string, m map[string]interface{}) bool {
_, ok := m[key]
@@ -17,8 +17,7 @@ func rmTag(typedJson interface{}) (interface{}, error) {
}
// Switch on the data type.
switch v := typedJson.(type) {
switch v := typedJSON.(type) {
// Object: this can either be a TOML table or a primitive with tags.
case map[string]interface{}:
// This value represents a primitive: remove the tags and return just
@@ -56,7 +55,7 @@ func rmTag(typedJson interface{}) (interface{}, error) {
}
// The top level must be an object or array.
return nil, fmt.Errorf("unrecognized JSON format '%T'", typedJson)
return nil, fmt.Errorf("unrecognized JSON format '%T'", typedJSON)
}
// Return a primitive: read the "type" and convert the "value" to that.
@@ -79,7 +78,7 @@ func untag(typed map[string]interface{}) (interface{}, error) {
}
return f, nil
//toml.LocalDate{Year:2020, Month:12, Day:12}
// toml.LocalDate{Year:2020, Month:12, Day:12}
case "datetime":
return time.Parse("2006-01-02T15:04:05.999999999Z07:00", v)
case "datetime-local":
@@ -115,15 +114,3 @@ func untag(typed map[string]interface{}) (interface{}, error) {
return nil, fmt.Errorf("untag: unrecognized tag type %q", t)
}
func parseTime(v, format string, local bool) (t time.Time, err error) {
if local {
t, err = time.ParseInLocation(format, v, time.Local)
} else {
t, err = time.Parse(format, v)
}
if err != nil {
return time.Time{}, fmt.Errorf("Could not parse %q as a datetime: %w", v, err)
}
return t, nil
}
+4 -4
View File
@@ -27,7 +27,7 @@ func Unmarshal(data []byte, v interface{}) error {
// ValueToTaggedJSON takes a data structure and returns the tagged JSON
// representation.
func ValueToTaggedJSON(doc interface{}) ([]byte, error) {
return json.MarshalIndent(addTag("", doc), "", " ")
return json.MarshalIndent(addTag(doc), "", " ")
}
// DecodeStdin is a helper function for the toml-test binary interface. TOML input
@@ -37,13 +37,13 @@ func DecodeStdin() error {
var decoded map[string]interface{}
if err := toml.NewDecoder(os.Stdin).Decode(&decoded); err != nil {
return fmt.Errorf("Error decoding TOML: %s", err)
return fmt.Errorf("error decoding TOML: %w", err)
}
j := json.NewEncoder(os.Stdout)
j.SetIndent("", " ")
if err := j.Encode(addTag("", decoded)); err != nil {
return fmt.Errorf("Error encoding JSON: %s", err)
if err := j.Encode(addTag(decoded)); err != nil {
return fmt.Errorf("error encoding JSON: %w", err)
}
return nil
+1 -1
View File
@@ -36,7 +36,7 @@ func (t *KeyTracker) Pop(node *unstable.Node) {
}
}
// Key returns the current key
// Key returns the current key.
func (t *KeyTracker) Key() []string {
k := make([]string, len(t.k))
copy(k, t.k)
+7 -6
View File
@@ -288,11 +288,12 @@ func (s *SeenTracker) checkKeyValue(node *unstable.Node) (bool, error) {
idx = s.create(parentIdx, k, tableKind, false, true)
} else {
entry := s.entries[idx]
if it.IsLast() {
switch {
case it.IsLast():
return false, fmt.Errorf("toml: key %s is already defined", string(k))
} else if entry.kind != tableKind {
case entry.kind != tableKind:
return false, fmt.Errorf("toml: expected %s to be a table, not a %s", string(k), entry.kind)
} else if entry.explicit {
case entry.explicit:
return false, fmt.Errorf("toml: cannot redefine table %s that has already been explicitly defined", string(k))
}
}
@@ -309,16 +310,16 @@ func (s *SeenTracker) checkKeyValue(node *unstable.Node) (bool, error) {
return s.checkInlineTable(value)
case unstable.Array:
return s.checkArray(value)
default:
return false, nil
}
return false, nil
}
func (s *SeenTracker) checkArray(node *unstable.Node) (first bool, err error) {
it := node.Children()
for it.Next() {
n := it.Node()
switch n.Kind {
switch n.Kind { //nolint:exhaustive
case unstable.InlineTable:
first, err = s.checkInlineTable(n)
if err != nil {
+4 -3
View File
@@ -1,8 +1,8 @@
package tracker
import (
"reflect"
"testing"
"unsafe"
"github.com/pelletier/go-toml/v2/internal/assert"
)
@@ -12,9 +12,10 @@ func TestEntrySize(t *testing.T) {
// performance of unmarshaling documents. Should only be increased with care
// and a very good reason.
maxExpectedEntrySize := 48
entrySize := int(reflect.TypeOf(entry{}).Size())
assert.True(t,
int(unsafe.Sizeof(entry{})) <= maxExpectedEntrySize,
entrySize <= maxExpectedEntrySize,
"Expected entry to be less than or equal to %d, got: %d",
maxExpectedEntrySize, int(unsafe.Sizeof(entry{})),
maxExpectedEntrySize, entrySize,
)
}
+1
View File
@@ -1 +1,2 @@
// Package tracker provides functions for keeping track of AST nodes.
package tracker
+6 -6
View File
@@ -32,7 +32,7 @@ func (d LocalDate) MarshalText() ([]byte, error) {
// UnmarshalText parses b using RFC 3339 to fill d.
func (d *LocalDate) UnmarshalText(b []byte) error {
res, err := parseLocalDate(b)
res, err := parseLocalDate(b, 0)
if err != nil {
return err
}
@@ -45,7 +45,7 @@ func (d *LocalDate) UnmarshalText(b []byte) error {
type LocalTime struct {
Hour int // Hour of the day: [0; 24[
Minute int // Minute of the hour: [0; 60[
Second int // Second of the minute: [0; 60[
Second int // Second of the minute: [0; 59]
Nanosecond int // Nanoseconds within the second: [0, 1000000000[
Precision int // Number of digits to display for Nanosecond.
}
@@ -75,9 +75,9 @@ func (d LocalTime) MarshalText() ([]byte, error) {
// UnmarshalText parses b using RFC 3339 to fill d.
func (d *LocalTime) UnmarshalText(b []byte) error {
res, left, err := parseLocalTime(b)
res, left, err := parseLocalTime(b, 0)
if err == nil && len(left) != 0 {
err = unstable.NewParserError(left, "extra characters")
err = unstable.NewParserError(left, len(b)-len(left), "extra characters")
}
if err != nil {
return err
@@ -109,9 +109,9 @@ func (d LocalDateTime) MarshalText() ([]byte, error) {
// UnmarshalText parses b using RFC 3339 to fill d.
func (d *LocalDateTime) UnmarshalText(data []byte) error {
res, left, err := parseLocalDateTime(data)
res, left, err := parseLocalDateTime(data, 0)
if err == nil && len(left) != 0 {
err = unstable.NewParserError(left, "extra characters")
err = unstable.NewParserError(left, len(data)-len(left), "extra characters")
}
if err != nil {
return err
+84 -49
View File
@@ -4,6 +4,7 @@ import (
"bytes"
"encoding"
"encoding/json"
"errors"
"fmt"
"io"
"math"
@@ -42,7 +43,7 @@ type Encoder struct {
arraysMultiline bool
indentSymbol string
indentTables bool
marshalJsonNumbers bool
marshalJSONNumbers bool
}
// NewEncoder returns a new Encoder that writes to w.
@@ -89,14 +90,14 @@ func (enc *Encoder) SetIndentTables(indent bool) *Encoder {
return enc
}
// SetMarshalJsonNumbers forces the encoder to serialize `json.Number` as a
// SetMarshalJSONNumbers forces the encoder to serialize `json.Number` as a
// float or integer instead of relying on TextMarshaler to emit a string.
//
// *Unstable:* This method does not follow the compatibility guarantees of
// semver. It can be changed or removed without a new major version being
// issued.
func (enc *Encoder) SetMarshalJsonNumbers(indent bool) *Encoder {
enc.marshalJsonNumbers = indent
func (enc *Encoder) SetMarshalJSONNumbers(indent bool) *Encoder {
enc.marshalJSONNumbers = indent
return enc
}
@@ -179,7 +180,7 @@ func (enc *Encoder) Encode(v interface{}) error {
ctx.inline = enc.tablesInline
if v == nil {
return fmt.Errorf("toml: cannot encode a nil interface")
return errors.New("toml: cannot encode a nil interface")
}
b, err := enc.encode(b, ctx, reflect.ValueOf(v))
@@ -269,16 +270,15 @@ func (enc *Encoder) encode(b []byte, ctx encoderCtx, v reflect.Value) ([]byte, e
case LocalDateTime:
return append(b, x.String()...), nil
case json.Number:
if enc.marshalJsonNumbers {
if enc.marshalJSONNumbers {
if x == "" { /// Useful zero value.
return append(b, "0"...), nil
} else if v, err := x.Int64(); err == nil {
return enc.encode(b, ctx, reflect.ValueOf(v))
} else if f, err := x.Float64(); err == nil {
return enc.encode(b, ctx, reflect.ValueOf(f))
} else {
return nil, fmt.Errorf("toml: unable to convert %q to int64 or float64", x)
}
return nil, fmt.Errorf("toml: unable to convert %q to int64 or float64", x)
}
}
@@ -312,7 +312,7 @@ func (enc *Encoder) encode(b []byte, ctx encoderCtx, v reflect.Value) ([]byte, e
return enc.encodeSlice(b, ctx, v)
case reflect.Interface:
if v.IsNil() {
return nil, fmt.Errorf("toml: encoding a nil interface is not supported")
return nil, errors.New("toml: encoding a nil interface is not supported")
}
return enc.encode(b, ctx, v.Elem())
@@ -329,28 +329,30 @@ func (enc *Encoder) encode(b []byte, ctx encoderCtx, v reflect.Value) ([]byte, e
case reflect.Float32:
f := v.Float()
if math.IsNaN(f) {
switch {
case math.IsNaN(f):
b = append(b, "nan"...)
} else if f > math.MaxFloat32 {
case f > math.MaxFloat32:
b = append(b, "inf"...)
} else if f < -math.MaxFloat32 {
case f < -math.MaxFloat32:
b = append(b, "-inf"...)
} else if math.Trunc(f) == f {
case math.Trunc(f) == f:
b = strconv.AppendFloat(b, f, 'f', 1, 32)
} else {
default:
b = strconv.AppendFloat(b, f, 'f', -1, 32)
}
case reflect.Float64:
f := v.Float()
if math.IsNaN(f) {
switch {
case math.IsNaN(f):
b = append(b, "nan"...)
} else if f > math.MaxFloat64 {
case f > math.MaxFloat64:
b = append(b, "inf"...)
} else if f < -math.MaxFloat64 {
case f < -math.MaxFloat64:
b = append(b, "-inf"...)
} else if math.Trunc(f) == f {
case math.Trunc(f) == f:
b = strconv.AppendFloat(b, f, 'f', 1, 64)
} else {
default:
b = strconv.AppendFloat(b, f, 'f', -1, 64)
}
case reflect.Bool:
@@ -388,7 +390,28 @@ func shouldOmitEmpty(options valueOptions, v reflect.Value) bool {
}
func shouldOmitZero(options valueOptions, v reflect.Value) bool {
return options.omitzero && v.IsZero()
if !options.omitzero {
return false
}
// Check if the type implements isZeroer interface (has a custom IsZero method).
if v.Type().Implements(isZeroerType) {
return v.Interface().(isZeroer).IsZero()
}
// Check if pointer type implements isZeroer.
if reflect.PointerTo(v.Type()).Implements(isZeroerType) {
if v.CanAddr() {
return v.Addr().Interface().(isZeroer).IsZero()
}
// Create a temporary addressable copy to call the pointer receiver method.
pv := reflect.New(v.Type())
pv.Elem().Set(v)
return pv.Interface().(isZeroer).IsZero()
}
// Fall back to reflect's IsZero for types without custom IsZero method.
return v.IsZero()
}
func (enc *Encoder) encodeKv(b []byte, ctx encoderCtx, options valueOptions, v reflect.Value) ([]byte, error) {
@@ -441,8 +464,9 @@ func isEmptyValue(v reflect.Value) bool {
return v.Float() == 0
case reflect.Interface, reflect.Ptr:
return v.IsNil()
default:
return false
}
return false
}
func isEmptyStruct(v reflect.Value) bool {
@@ -486,7 +510,7 @@ func (enc *Encoder) encodeString(b []byte, v string, options valueOptions) []byt
func needsQuoting(v string) bool {
// TODO: vectorize
for _, b := range []byte(v) {
if b == '\'' || b == '\r' || b == '\n' || characters.InvalidAscii(b) {
if b == '\'' || b == '\r' || b == '\n' || characters.InvalidASCII(b) {
return true
}
}
@@ -580,9 +604,9 @@ func (enc *Encoder) encodeUnquotedKey(b []byte, v string) []byte {
return append(b, v...)
}
func (enc *Encoder) encodeTableHeader(ctx encoderCtx, b []byte) ([]byte, error) {
func (enc *Encoder) encodeTableHeader(ctx encoderCtx, b []byte) []byte {
if len(ctx.parentKey) == 0 {
return b, nil
return b
}
b = enc.encodeComment(ctx.indent, ctx.options.comment, b)
@@ -602,10 +626,9 @@ func (enc *Encoder) encodeTableHeader(ctx encoderCtx, b []byte) ([]byte, error)
b = append(b, "]\n"...)
return b, nil
return b
}
//nolint:cyclop
func (enc *Encoder) encodeKey(b []byte, k string) []byte {
needsQuotation := false
cannotUseLiteral := false
@@ -642,30 +665,33 @@ func (enc *Encoder) encodeKey(b []byte, k string) []byte {
func (enc *Encoder) keyToString(k reflect.Value) (string, error) {
keyType := k.Type()
switch {
case keyType.Kind() == reflect.String:
return k.String(), nil
case keyType.Implements(textMarshalerType):
if keyType.Implements(textMarshalerType) {
keyB, err := k.Interface().(encoding.TextMarshaler).MarshalText()
if err != nil {
return "", fmt.Errorf("toml: error marshalling key %v from text: %w", k, err)
}
return string(keyB), nil
}
case keyType.Kind() == reflect.Int || keyType.Kind() == reflect.Int8 || keyType.Kind() == reflect.Int16 || keyType.Kind() == reflect.Int32 || keyType.Kind() == reflect.Int64:
switch keyType.Kind() {
case reflect.String:
return k.String(), nil
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
return strconv.FormatInt(k.Int(), 10), nil
case keyType.Kind() == reflect.Uint || keyType.Kind() == reflect.Uint8 || keyType.Kind() == reflect.Uint16 || keyType.Kind() == reflect.Uint32 || keyType.Kind() == reflect.Uint64:
case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:
return strconv.FormatUint(k.Uint(), 10), nil
case keyType.Kind() == reflect.Float32:
case reflect.Float32:
return strconv.FormatFloat(k.Float(), 'f', -1, 32), nil
case keyType.Kind() == reflect.Float64:
case reflect.Float64:
return strconv.FormatFloat(k.Float(), 'f', -1, 64), nil
default:
return "", fmt.Errorf("toml: type %s is not supported as a map key", keyType.Kind())
}
return "", fmt.Errorf("toml: type %s is not supported as a map key", keyType.Kind())
}
func (enc *Encoder) encodeMap(b []byte, ctx encoderCtx, v reflect.Value) ([]byte, error) {
@@ -678,8 +704,18 @@ func (enc *Encoder) encodeMap(b []byte, ctx encoderCtx, v reflect.Value) ([]byte
for iter.Next() {
v := iter.Value()
if isNil(v) {
continue
// Handle nil values: convert nil pointers to zero value,
// skip nil interfaces and nil maps.
switch v.Kind() {
case reflect.Ptr:
if v.IsNil() {
v = reflect.Zero(v.Type().Elem())
}
case reflect.Interface, reflect.Map:
if v.IsNil() {
continue
}
default:
}
k, err := enc.keyToString(iter.Key())
@@ -769,9 +805,8 @@ func walkStruct(ctx encoderCtx, t *table, v reflect.Value) {
walkStruct(ctx, t, f.Elem())
}
continue
} else {
k = fieldType.Name
}
k = fieldType.Name
}
if isNil(f) {
@@ -891,10 +926,7 @@ func (enc *Encoder) encodeTable(b []byte, ctx encoderCtx, t table) ([]byte, erro
}
if !ctx.skipTableHeader {
b, err = enc.encodeTableHeader(ctx, b)
if err != nil {
return nil, err
}
b = enc.encodeTableHeader(ctx, b)
if enc.indentTables && len(ctx.parentKey) > 0 {
ctx.indent++
@@ -907,7 +939,7 @@ func (enc *Encoder) encodeTable(b []byte, ctx encoderCtx, t table) ([]byte, erro
if shouldOmitEmpty(kv.Options, kv.Value) {
continue
}
if shouldOmitZero(kv.Options, kv.Value) {
if kv.Options.omitzero && shouldOmitZero(kv.Options, kv.Value) {
continue
}
hasNonEmptyKV = true
@@ -929,7 +961,7 @@ func (enc *Encoder) encodeTable(b []byte, ctx encoderCtx, t table) ([]byte, erro
if shouldOmitEmpty(table.Options, table.Value) {
continue
}
if shouldOmitZero(table.Options, table.Value) {
if table.Options.omitzero && shouldOmitZero(table.Options, table.Value) {
continue
}
if first {
@@ -966,7 +998,7 @@ func (enc *Encoder) encodeTableInline(b []byte, ctx encoderCtx, t table) ([]byte
if shouldOmitEmpty(kv.Options, kv.Value) {
continue
}
if shouldOmitZero(kv.Options, kv.Value) {
if kv.Options.omitzero && shouldOmitZero(kv.Options, kv.Value) {
continue
}
@@ -997,11 +1029,14 @@ func willConvertToTable(ctx encoderCtx, v reflect.Value) bool {
if !v.IsValid() {
return false
}
if v.Type() == timeType || v.Type().Implements(textMarshalerType) || (v.Kind() != reflect.Ptr && v.CanAddr() && reflect.PointerTo(v.Type()).Implements(textMarshalerType)) {
t := v.Type()
if t == timeType || t.Implements(textMarshalerType) {
return false
}
if v.Kind() != reflect.Ptr && v.CanAddr() && reflect.PointerTo(t).Implements(textMarshalerType) {
return false
}
t := v.Type()
switch t.Kind() {
case reflect.Map, reflect.Struct:
return !ctx.inline
+341 -14
View File
@@ -3,6 +3,7 @@ package toml_test
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"math"
"math/big"
@@ -28,7 +29,7 @@ func (k marshalTextKey) MarshalText() ([]byte, error) {
type marshalBadTextKey struct{}
func (k marshalBadTextKey) MarshalText() ([]byte, error) {
return nil, fmt.Errorf("error")
return nil, errors.New("error")
}
func toFloat(x interface{}) float64 {
@@ -44,6 +45,7 @@ func toFloat(x interface{}) float64 {
}
func inDelta(t *testing.T, expected, actual interface{}, delta float64) {
t.Helper()
dt := toFloat(expected) - toFloat(actual)
assert.True(t,
dt < -delta && dt < delta,
@@ -617,12 +619,36 @@ hello = 'world'
expected: ``,
},
{
desc: "nil value in map is ignored",
desc: "nil interface value in map is ignored",
v: map[string]interface{}{
"A": nil,
},
expected: ``,
},
{
desc: "nil pointer to struct in map produces empty table",
v: map[string]*struct{}{
"A": nil,
},
expected: `[A]
`,
},
{
desc: "nil pointer to int in map produces zero value",
v: map[string]*int{
"A": nil,
},
expected: `A = 0
`,
},
{
desc: "nil pointer to string in map produces empty string",
v: map[string]*string{
"A": nil,
},
expected: `A = ''
`,
},
{
desc: "new line in table key",
v: map[string]interface{}{
@@ -942,7 +968,6 @@ nan = nan
assert.Equal(t, expected, string(actual))
}
//nolint:funlen
func TestMarshalIndentTables(t *testing.T) {
examples := []struct {
desc string
@@ -1011,7 +1036,7 @@ type customTextMarshaler struct {
func (c *customTextMarshaler) MarshalText() ([]byte, error) {
if c.value == 1 {
return nil, fmt.Errorf("cannot represent 1 because this is a silly test")
return nil, errors.New("cannot represent 1 because this is a silly test")
}
return []byte(fmt.Sprintf("::%d", c.value)), nil
}
@@ -1051,7 +1076,7 @@ func TestMarshalTextMarshaler(t *testing.T) {
type brokenWriter struct{}
func (b *brokenWriter) Write([]byte) (int, error) {
return 0, fmt.Errorf("dead")
return 0, errors.New("dead")
}
func TestEncodeToBrokenWriter(t *testing.T) {
@@ -1074,10 +1099,10 @@ func TestEncoderSetIndentSymbol(t *testing.T) {
assert.Equal(t, expected, w.String())
}
func TestEncoderSetMarshalJsonNumbers(t *testing.T) {
func TestEncoderSetMarshalJSONNumbers(t *testing.T) {
var w strings.Builder
enc := toml.NewEncoder(&w)
enc.SetMarshalJsonNumbers(true)
enc.SetMarshalJSONNumbers(true)
err := enc.Encode(map[string]interface{}{
"A": json.Number("1.1"),
"B": json.Number("42e-3"),
@@ -1194,11 +1219,291 @@ IP = '192.168.178.35'
assert.Equal(t, expected, string(b))
}
// customZeroType has a custom IsZero method that returns true
// when Value is less than 10.
type customZeroType struct {
Value int
}
func (c customZeroType) IsZero() bool {
return c.Value < 10
}
// customZeroPointerType has a custom IsZero method on the pointer receiver.
type customZeroPointerType struct {
Value int
}
func (c *customZeroPointerType) IsZero() bool {
return c.Value < 10
}
func TestEncoderOmitzeroCustomIsZero(t *testing.T) {
type doc struct {
Custom customZeroType `toml:",omitzero"`
Normal int `toml:",omitzero"`
}
// Custom.Value = 5, which is < 10, so custom IsZero returns true
d := doc{
Custom: customZeroType{Value: 5},
Normal: 0,
}
b, err := toml.Marshal(d)
assert.NoError(t, err)
// Both fields should be omitted: Custom because custom IsZero returns true,
// Normal because its reflect zero value is true.
expected := ``
assert.Equal(t, expected, string(b))
}
func TestEncoderOmitzeroCustomIsZeroNotZero(t *testing.T) {
type doc struct {
Custom customZeroType `toml:",omitzero"`
Normal int `toml:",omitzero"`
}
// Custom.Value = 15, which is >= 10, so custom IsZero returns false
d := doc{
Custom: customZeroType{Value: 15},
Normal: 42,
}
b, err := toml.Marshal(d)
assert.NoError(t, err)
// Both fields should be present
expected := `Normal = 42
[Custom]
Value = 15
`
assert.Equal(t, expected, string(b))
}
func TestEncoderOmitzeroCustomIsZeroPointerReceiver(t *testing.T) {
type doc struct {
Custom customZeroPointerType `toml:",omitzero"`
}
// Custom.Value = 5, which is < 10, so custom IsZero returns true
d := doc{
Custom: customZeroPointerType{Value: 5},
}
b, err := toml.Marshal(d)
assert.NoError(t, err)
// Field should be omitted because custom IsZero returns true
expected := ``
assert.Equal(t, expected, string(b))
}
func TestEncoderOmitzeroCustomIsZeroPointerReceiverNotZero(t *testing.T) {
type doc struct {
Custom customZeroPointerType `toml:",omitzero"`
}
// Custom.Value = 15, which is >= 10, so custom IsZero returns false
d := doc{
Custom: customZeroPointerType{Value: 15},
}
b, err := toml.Marshal(d)
assert.NoError(t, err)
// Field should be present
expected := `[Custom]
Value = 15
`
assert.Equal(t, expected, string(b))
}
// TestEncoderOmitzeroCustomIsZeroPointerReceiverAddressable tests the v.CanAddr() path
// by marshaling a pointer to a struct, which makes fields addressable.
func TestEncoderOmitzeroCustomIsZeroPointerReceiverAddressable(t *testing.T) {
type doc struct {
Custom customZeroPointerType `toml:",omitzero"`
}
// Custom.Value = 5, which is < 10, so custom IsZero returns true
d := &doc{
Custom: customZeroPointerType{Value: 5},
}
b, err := toml.Marshal(d)
assert.NoError(t, err)
// Field should be omitted because custom IsZero returns true
expected := ``
assert.Equal(t, expected, string(b))
}
// TestEncoderOmitzeroCustomIsZeroPointerReceiverAddressableNotZero tests the v.CanAddr() path
// when custom IsZero returns false.
func TestEncoderOmitzeroCustomIsZeroPointerReceiverAddressableNotZero(t *testing.T) {
type doc struct {
Custom customZeroPointerType `toml:",omitzero"`
}
// Custom.Value = 15, which is >= 10, so custom IsZero returns false
d := &doc{
Custom: customZeroPointerType{Value: 15},
}
b, err := toml.Marshal(d)
assert.NoError(t, err)
// Field should be present
expected := `[Custom]
Value = 15
`
assert.Equal(t, expected, string(b))
}
// TestEncoderOmitzeroCustomIsZeroInlineTable tests omitzero with inline tables.
func TestEncoderOmitzeroCustomIsZeroInlineTable(t *testing.T) {
type doc struct {
Custom customZeroType `toml:",omitzero,inline"`
}
// Custom.Value = 5, which is < 10, so custom IsZero returns true
d := doc{
Custom: customZeroType{Value: 5},
}
b, err := toml.Marshal(d)
assert.NoError(t, err)
// Field should be omitted
expected := ``
assert.Equal(t, expected, string(b))
}
// TestEncoderOmitzeroCustomIsZeroInlineTableNotZero tests omitzero with inline tables when not zero.
func TestEncoderOmitzeroCustomIsZeroInlineTableNotZero(t *testing.T) {
type doc struct {
Custom customZeroType `toml:",omitzero,inline"`
}
// Custom.Value = 15, which is >= 10, so custom IsZero returns false
d := doc{
Custom: customZeroType{Value: 15},
}
b, err := toml.Marshal(d)
assert.NoError(t, err)
// Field should be present as inline table
expected := `Custom = {Value = 15}
`
assert.Equal(t, expected, string(b))
}
// TestEncoderOmitzeroCustomIsZeroMixedTypes tests omitzero with a mix of custom and regular types.
func TestEncoderOmitzeroCustomIsZeroMixedTypes(t *testing.T) {
type doc struct {
Custom customZeroType `toml:",omitzero"`
Regular int `toml:",omitzero"`
NoOmit customZeroType `toml:""`
Pointer *int `toml:",omitzero"`
}
d := doc{
Custom: customZeroType{Value: 5}, // IsZero returns true
Regular: 0, // zero value
NoOmit: customZeroType{Value: 5}, // not omitted (no omitzero tag)
Pointer: nil, // nil pointer
}
b, err := toml.Marshal(d)
assert.NoError(t, err)
// Custom is omitted (custom IsZero true), Regular is omitted (zero value),
// NoOmit is present (no omitzero tag), Pointer is omitted (nil)
expected := `[NoOmit]
Value = 5
`
assert.Equal(t, expected, string(b))
}
// TestEncoderOmitzeroCustomIsZeroSlice tests omitzero with slices containing custom types.
func TestEncoderOmitzeroCustomIsZeroSlice(t *testing.T) {
type doc struct {
Items []customZeroType `toml:",omitzero"`
}
// Nil slice should be omitted (IsZero returns true for nil slices)
d := doc{
Items: nil,
}
b, err := toml.Marshal(d)
assert.NoError(t, err)
expected := ``
assert.Equal(t, expected, string(b))
// Empty but non-nil slice is NOT zero, so it's included
d2 := doc{
Items: []customZeroType{},
}
b2, err := toml.Marshal(d2)
assert.NoError(t, err)
expected2 := `Items = []
`
assert.Equal(t, expected2, string(b2))
}
// TestEncoderOmitzeroCustomIsZeroNestedStruct tests omitzero with nested structs.
func TestEncoderOmitzeroCustomIsZeroNestedStruct(t *testing.T) {
type inner struct {
Custom customZeroType `toml:",omitzero"`
Value int `toml:",omitzero"`
}
type doc struct {
Inner inner `toml:",omitzero"`
}
// Inner struct has all zero fields, but the struct itself is not zero
// (reflect.Value.IsZero checks if all fields are zero)
d := doc{
Inner: inner{
Custom: customZeroType{Value: 5}, // custom IsZero returns true
Value: 0, // zero value
},
}
b, err := toml.Marshal(d)
assert.NoError(t, err)
// Inner is present but its fields are omitted
expected := `[Inner]
`
assert.Equal(t, expected, string(b))
}
func TestEncoderTagFieldName(t *testing.T) {
type doc struct {
String string `toml:"hello"`
OkSym string `toml:"#"`
Bad string `toml:"\"`
Bad string `toml:"\"` //nolint:govet
}
d := doc{String: "world"}
@@ -1762,14 +2067,14 @@ func ExampleMarshal() {
func ExampleMarshal_commented() {
type Common struct {
Listen string `toml:"listen" comment:"general listener"`
PprofListen string `toml:"pprof-listen" comment:"listener to serve /debug/pprof requests. '-pprof' argument overrides it"`
MaxMetricsPerTarget int `toml:"max-metrics-per-target" comment:"limit numbers of queried metrics per target in /render requests, 0 or negative = unlimited"`
PprofListen string `toml:"pprof-listen" comment:"listener to serve /debug/pprof requests. '-pprof' argument overrides it"` //nolint:lll
MaxMetricsPerTarget int `toml:"max-metrics-per-target" comment:"limit numbers of queried metrics per target in /render requests, 0 or negative = unlimited"` //nolint:lll
MemoryReturnInterval time.Duration `toml:"memory-return-interval" comment:"daemon will return the freed memory to the OS when it>0"`
}
type Costs struct {
Cost *int `toml:"cost" comment:"default cost (for wildcarded equivalence or matched with regex, or if no value cost set)"`
ValuesCost map[string]int `toml:"values-cost" comment:"cost with some value (for equivalence without wildcards) (additional tuning, usually not needed)"`
ValuesCost map[string]int `toml:"values-cost" comment:"cost with some value (for equivalence without wildcards) (additional tuning, usually not needed)"` //nolint:lll
}
type ClickHouse struct {
@@ -1784,7 +2089,7 @@ func ExampleMarshal_commented() {
DateTreeTableVersion int `toml:"date-tree-table-version,commented"`
TreeTimeout time.Duration `toml:"tree-timeout,commented"`
TagTable string `toml:"tag-table,commented"`
ExtraPrefix string `toml:"extra-prefix" comment:"add extra prefix (directory in graphite) for all metrics, w/o trailing dot"`
ExtraPrefix string `toml:"extra-prefix" comment:"add extra prefix (directory in graphite) for all metrics, w/o trailing dot"` //nolint:lll
ConnectTimeout time.Duration `toml:"connect-timeout" comment:"TCP connection timeout"`
DataTableLegacy string `toml:"data-table,commented"`
RollupConfLegacy string `toml:"rollup-conf,commented"`
@@ -1887,12 +2192,12 @@ func TestReadmeComments(t *testing.T) {
type Config struct {
Host string `toml:"host" comment:"Host IP to connect to."`
Port int `toml:"port" comment:"Port of the remote server."`
Tls TLS `toml:"TLS,commented" comment:"Encryption parameters (optional)"`
TLS TLS `toml:"TLS,commented" comment:"Encryption parameters (optional)"`
}
example := Config{
Host: "127.0.0.1",
Port: 4242,
Tls: TLS{
TLS: TLS{
Cipher: "AEAD-AES128-GCM-SHA256",
Version: "TLS 1.3",
},
@@ -1912,3 +2217,25 @@ port = 4242
`
assert.Equal(t, expected, string(out))
}
// TestMarshalIssue975 tests that nil pointer values in maps are marshaled as
// empty tables, allowing round-trip marshaling to work correctly.
// See https://github.com/pelletier/go-toml/issues/975
func TestMarshalIssue975(t *testing.T) {
// Test case from the issue: map[string]*struct{}
oldMap := map[string]*struct{}{
"foo": nil,
}
doc, err := toml.Marshal(&oldMap)
assert.NoError(t, err)
assert.Equal(t, "[foo]\n", string(doc))
var newMap map[string]*struct{}
err = toml.Unmarshal(doc, &newMap)
assert.NoError(t, err)
// Verify the key is preserved after round-trip
_, exists := newMap["foo"]
assert.True(t, exists, "key 'foo' should exist after round-trip")
}
+2
View File
@@ -1,3 +1,4 @@
// Package ossfuzz provides a fuzzing target for OSS-Fuzz.
package ossfuzz
import (
@@ -8,6 +9,7 @@ import (
"github.com/pelletier/go-toml/v2"
)
// FuzzToml is the fuzzing target.
func FuzzToml(data []byte) int {
if len(data) >= 2048 {
return 0
+17 -8
View File
@@ -1,7 +1,6 @@
package toml
import (
"github.com/pelletier/go-toml/v2/internal/danger"
"github.com/pelletier/go-toml/v2/internal/tracker"
"github.com/pelletier/go-toml/v2/unstable"
)
@@ -13,6 +12,9 @@ type strict struct {
key tracker.KeyTracker
missing []unstable.ParserError
// Reference to the document for computing key ranges.
doc []byte
}
func (s *strict) EnterTable(node *unstable.Node) {
@@ -52,10 +54,12 @@ func (s *strict) MissingTable(node *unstable.Node) {
return
}
loc, offset := s.keyLocation(node)
s.missing = append(s.missing, unstable.ParserError{
Highlight: keyLocation(node),
Highlight: loc,
Message: "missing table",
Key: s.key.Key(),
Offset: offset,
})
}
@@ -64,10 +68,12 @@ func (s *strict) MissingField(node *unstable.Node) {
return
}
loc, offset := s.keyLocation(node)
s.missing = append(s.missing, unstable.ParserError{
Highlight: keyLocation(node),
Highlight: loc,
Message: "missing field",
Key: s.key.Key(),
Offset: offset,
})
}
@@ -88,7 +94,7 @@ func (s *strict) Error(doc []byte) error {
return err
}
func keyLocation(node *unstable.Node) []byte {
func (s *strict) keyLocation(node *unstable.Node) ([]byte, int) {
k := node.Key()
hasOne := k.Next()
@@ -96,12 +102,15 @@ func keyLocation(node *unstable.Node) []byte {
panic("should not be called with empty key")
}
start := k.Node().Data
end := k.Node().Data
firstRaw := k.Node().Raw
lastRaw := firstRaw
for k.Next() {
end = k.Node().Data
lastRaw = k.Node().Raw
}
return danger.BytesRange(start, end)
start := firstRaw.Offset
end := lastRaw.Offset + lastRaw.Length
return s.doc[start:end], int(start)
}
+5 -4
View File
@@ -9,7 +9,7 @@ YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color
# Go versions to test (1.11 through 1.25)
# Go versions to test (1.11 through 1.26)
GO_VERSIONS=(
"1.11"
"1.12"
@@ -26,6 +26,7 @@ GO_VERSIONS=(
"1.23"
"1.24"
"1.25"
"1.26"
)
# Default values
@@ -64,7 +65,7 @@ EXAMPLES:
$0 # Test all Go versions in parallel
$0 --sequential # Test all Go versions sequentially
$0 1.21 1.22 1.23 # Test specific versions
$0 --verbose --output ./results 1.24 1.25 # Verbose output to custom directory
$0 --verbose --output ./results 1.25 1.26 # Verbose output to custom directory
EXIT CODES:
0 Recent Go versions pass (good compatibility)
@@ -136,8 +137,8 @@ fi
# Validate Go versions
for version in "${GO_VERSIONS[@]}"; do
if ! [[ "$version" =~ ^1\.(1[1-9]|2[0-5])$ ]]; then
log_error "Invalid Go version: $version. Supported versions: 1.11-1.25"
if ! [[ "$version" =~ ^1\.(1[1-9]|2[0-6])$ ]]; then
log_error "Invalid Go version: $version. Supported versions: 1.11-1.26"
exit 1
fi
done
+3 -2
View File
@@ -1,11 +1,11 @@
//go:generate go run github.com/toml-lang/toml-test/cmd/toml-test@v1.6.0 -copy ./tests
//go:generate go run ./cmd/tomltestgen/main.go -r v1.6.0 -o toml_testgen_test.go
// This is a support file for toml_testgen_test.go
package toml_test
import (
"encoding/json"
"errors"
"testing"
"github.com/pelletier/go-toml/v2"
@@ -39,7 +39,8 @@ func testgenValid(t *testing.T, input string, jsonRef string) {
err := testsuite.Unmarshal([]byte(input), &doc)
if err != nil {
if de, ok := err.(*toml.DecodeError); ok {
de := &toml.DecodeError{}
if errors.As(err, &de) {
t.Logf("%s\n%s", err, de)
}
t.Fatalf("failed parsing toml: %s", err)
+1 -1
View File
@@ -1,4 +1,4 @@
// Generated by tomltestgen for toml-test ref v1.6.0 on 2025-10-22T16:33:06+11:00
// Code generated by tomltestgen for toml-test ref v1.6.0 on 2025-10-22T16:33:06+11:00. DO NOT EDIT.
package toml_test
import (
+15 -6
View File
@@ -6,9 +6,18 @@ import (
"time"
)
var timeType = reflect.TypeOf((*time.Time)(nil)).Elem()
var textMarshalerType = reflect.TypeOf((*encoding.TextMarshaler)(nil)).Elem()
var textUnmarshalerType = reflect.TypeOf((*encoding.TextUnmarshaler)(nil)).Elem()
var mapStringInterfaceType = reflect.TypeOf(map[string]interface{}(nil))
var sliceInterfaceType = reflect.TypeOf([]interface{}(nil))
var stringType = reflect.TypeOf("")
// isZeroer is used to check if a type has a custom IsZero method.
// This allows custom types to define their own zero-value semantics.
type isZeroer interface {
IsZero() bool
}
var (
timeType = reflect.TypeOf((*time.Time)(nil)).Elem()
textMarshalerType = reflect.TypeOf((*encoding.TextMarshaler)(nil)).Elem()
textUnmarshalerType = reflect.TypeOf((*encoding.TextUnmarshaler)(nil)).Elem()
isZeroerType = reflect.TypeOf((*isZeroer)(nil)).Elem()
mapStringInterfaceType = reflect.TypeOf(map[string]interface{}(nil))
sliceInterfaceType = reflect.TypeOf([]interface{}(nil))
stringType = reflect.TypeOf("")
)
+153 -64
View File
@@ -12,7 +12,6 @@ import (
"sync/atomic"
"time"
"github.com/pelletier/go-toml/v2/internal/danger"
"github.com/pelletier/go-toml/v2/internal/tracker"
"github.com/pelletier/go-toml/v2/unstable"
)
@@ -57,13 +56,18 @@ func (d *Decoder) DisallowUnknownFields() *Decoder {
// EnableUnmarshalerInterface allows to enable unmarshaler interface.
//
// With this feature enabled, types implementing the unstable/Unmarshaler
// With this feature enabled, types implementing the unstable.Unmarshaler
// interface can be decoded from any structure of the document. It allows types
// that don't have a straightforward TOML representation to provide their own
// decoding logic.
//
// Currently, types can only decode from a single value. Tables and array tables
// are not supported.
// The UnmarshalTOML method receives raw TOML bytes:
// - For single values: the raw value bytes (e.g., `"hello"` for a string)
// - For tables: all key-value lines belonging to that table
// - For inline tables/arrays: the raw bytes of the inline structure
//
// The unstable.RawMessage type can be used to capture raw TOML bytes for
// later processing, similar to json.RawMessage.
//
// *Unstable:* This method does not follow the compatibility guarantees of
// semver. It can be changed or removed without a new major version being
@@ -123,6 +127,7 @@ func (d *Decoder) Decode(v interface{}) error {
dec := decoder{
strict: strict{
Enabled: d.strict,
doc: b,
},
unmarshalerInterface: d.unmarshalerInterface,
}
@@ -226,7 +231,7 @@ func (d *decoder) FromParser(v interface{}) error {
}
if r.IsNil() {
return fmt.Errorf("toml: decoding pointer target cannot be nil")
return errors.New("toml: decoding pointer target cannot be nil")
}
r = r.Elem()
@@ -273,7 +278,7 @@ func (d *decoder) handleRootExpression(expr *unstable.Node, v reflect.Value) err
var err error
var first bool // used for to clear array tables on first use
if !(d.skipUntilTable && expr.Kind == unstable.KeyValue) {
if !d.skipUntilTable || expr.Kind != unstable.KeyValue {
first, err = d.seen.CheckExpression(expr)
if err != nil {
return err
@@ -378,7 +383,7 @@ func (d *decoder) handleArrayTableCollectionLast(key unstable.Iterator, v reflec
case reflect.Array:
idx := d.arrayIndex(true, v)
if idx >= v.Len() {
return v, fmt.Errorf("%s at position %d", d.typeMismatchError("array table", v.Type()), idx)
return v, fmt.Errorf("%w at position %d", d.typeMismatchError("array table", v.Type()), idx)
}
elem := v.Index(idx)
_, err := d.handleArrayTable(key, elem)
@@ -453,14 +458,14 @@ func (d *decoder) handleArrayTableCollection(key unstable.Iterator, v reflect.Va
case reflect.Array:
idx := d.arrayIndex(false, v)
if idx >= v.Len() {
return v, fmt.Errorf("%s at position %d", d.typeMismatchError("array table", v.Type()), idx)
return v, fmt.Errorf("%w at position %d", d.typeMismatchError("array table", v.Type()), idx)
}
elem := v.Index(idx)
_, err := d.handleArrayTable(key, elem)
return v, err
default:
return d.handleArrayTable(key, v)
}
return d.handleArrayTable(key, v)
}
func (d *decoder) handleKeyPart(key unstable.Iterator, v reflect.Value, nextFn handlerFn, makeFn valueMakerFn) (reflect.Value, error) {
@@ -494,7 +499,8 @@ func (d *decoder) handleKeyPart(key unstable.Iterator, v reflect.Value, nextFn h
mv := v.MapIndex(mk)
set := false
if !mv.IsValid() {
switch {
case !mv.IsValid():
// If there is no value in the map, create a new one according to
// the map type. If the element type is interface, create either a
// map[string]interface{} or a []interface{} depending on whether
@@ -507,13 +513,13 @@ func (d *decoder) handleKeyPart(key unstable.Iterator, v reflect.Value, nextFn h
mv = reflect.New(t).Elem()
}
set = true
} else if mv.Kind() == reflect.Interface {
case mv.Kind() == reflect.Interface:
mv = mv.Elem()
if !mv.IsValid() {
mv = makeFn()
}
set = true
} else if !mv.CanAddr() {
case !mv.CanAddr():
vt := v.Type()
t := vt.Elem()
oldmv := mv
@@ -598,18 +604,28 @@ func (d *decoder) handleArrayTablePart(key unstable.Iterator, v reflect.Value) (
// cannot handle it.
func (d *decoder) handleTable(key unstable.Iterator, v reflect.Value) (reflect.Value, error) {
if v.Kind() == reflect.Slice {
if v.Len() == 0 {
return reflect.Value{}, unstable.NewParserError(key.Node().Data, "cannot store a table in a slice")
// For non-empty slices, work with the last element
if v.Len() > 0 {
elem := v.Index(v.Len() - 1)
x, err := d.handleTable(key, elem)
if err != nil {
return reflect.Value{}, err
}
if x.IsValid() {
elem.Set(x)
}
return reflect.Value{}, nil
}
elem := v.Index(v.Len() - 1)
x, err := d.handleTable(key, elem)
if err != nil {
return reflect.Value{}, err
// Empty slice - check if it implements Unmarshaler (e.g., RawMessage)
// and we're at the end of the key path
if d.unmarshalerInterface && !key.Next() {
if v.CanAddr() && v.Addr().CanInterface() {
if outi, ok := v.Addr().Interface().(unstable.Unmarshaler); ok {
return d.handleKeyValuesUnmarshaler(outi)
}
}
}
if x.IsValid() {
elem.Set(x)
}
return reflect.Value{}, nil
return reflect.Value{}, unstable.NewParserError(key.Node().Data, int(key.Node().Raw.Offset), "cannot store a table in a slice")
}
if key.Next() {
// Still scoping the key
@@ -623,6 +639,24 @@ func (d *decoder) handleTable(key unstable.Iterator, v reflect.Value) (reflect.V
// Handle root expressions until the end of the document or the next
// non-key-value.
func (d *decoder) handleKeyValues(v reflect.Value) (reflect.Value, error) {
// Check if target implements Unmarshaler before processing key-values.
// This allows types to handle entire tables themselves.
if d.unmarshalerInterface {
vv := v
for vv.Kind() == reflect.Ptr {
if vv.IsNil() {
vv.Set(reflect.New(vv.Type().Elem()))
}
vv = vv.Elem()
}
if vv.CanAddr() && vv.Addr().CanInterface() {
if outi, ok := vv.Addr().Interface().(unstable.Unmarshaler); ok {
// Collect all key-value expressions for this table
return d.handleKeyValuesUnmarshaler(outi)
}
}
}
var rv reflect.Value
for d.nextExpr() {
expr := d.expr()
@@ -652,6 +686,41 @@ func (d *decoder) handleKeyValues(v reflect.Value) (reflect.Value, error) {
return rv, nil
}
// handleKeyValuesUnmarshaler collects all key-value expressions for a table
// and passes them to the Unmarshaler as raw TOML bytes.
func (d *decoder) handleKeyValuesUnmarshaler(u unstable.Unmarshaler) (reflect.Value, error) {
// Collect raw bytes from all key-value expressions for this table.
// We use the Raw field on each KeyValue expression to preserve the
// original formatting (whitespace, quoting style, etc.) from the document.
var buf []byte
for d.nextExpr() {
expr := d.expr()
if expr.Kind != unstable.KeyValue {
d.stashExpr()
break
}
_, err := d.seen.CheckExpression(expr)
if err != nil {
return reflect.Value{}, err
}
// Use the raw bytes from the original document to preserve formatting
if expr.Raw.Length > 0 {
raw := d.p.Raw(expr.Raw)
buf = append(buf, raw...)
}
buf = append(buf, '\n')
}
if err := u.UnmarshalTOML(buf); err != nil {
return reflect.Value{}, err
}
return reflect.Value{}, nil
}
type (
handlerFn func(key unstable.Iterator, v reflect.Value) (reflect.Value, error)
valueMakerFn func() reflect.Value
@@ -679,7 +748,7 @@ func (d *decoder) tryTextUnmarshaler(node *unstable.Node, v reflect.Value) (bool
if v.CanAddr() && v.Addr().Type().Implements(textUnmarshalerType) {
err := v.Addr().Interface().(encoding.TextUnmarshaler).UnmarshalText(node.Data)
if err != nil {
return false, unstable.NewParserError(d.p.Raw(node.Raw), "%w", err)
return false, unstable.NewParserError(d.p.Raw(node.Raw), int(node.Raw.Offset), "%w", err)
}
return true, nil
@@ -696,14 +765,21 @@ func (d *decoder) handleValue(value *unstable.Node, v reflect.Value) error {
if d.unmarshalerInterface {
if v.CanAddr() && v.Addr().CanInterface() {
if outi, ok := v.Addr().Interface().(unstable.Unmarshaler); ok {
return outi.UnmarshalTOML(value)
// Pass raw bytes from the original document
return outi.UnmarshalTOML(d.p.Raw(value.Raw))
}
}
}
ok, err := d.tryTextUnmarshaler(value, v)
if ok || err != nil {
return err
// Only try TextUnmarshaler for scalar types. For Array and InlineTable,
// fall through to struct/map unmarshaling to allow flexible unmarshaling
// where a type can implement UnmarshalText for string values but still
// be populated field-by-field from a table. See issue #974.
if value.Kind != unstable.Array && value.Kind != unstable.InlineTable {
ok, err := d.tryTextUnmarshaler(value, v)
if ok || err != nil {
return err
}
}
switch value.Kind {
@@ -820,7 +896,7 @@ func (d *decoder) unmarshalInlineTable(itable *unstable.Node, v reflect.Value) e
}
return d.unmarshalInlineTable(itable, elem)
default:
return unstable.NewParserError(d.p.Raw(itable.Raw), "cannot store inline table in Go type %s", v.Kind())
return unstable.NewParserError(d.p.Raw(itable.Raw), int(itable.Raw.Offset), "cannot store inline table in Go type %s", v.Kind())
}
it := itable.Children()
@@ -840,65 +916,70 @@ func (d *decoder) unmarshalInlineTable(itable *unstable.Node, v reflect.Value) e
}
func (d *decoder) unmarshalDateTime(value *unstable.Node, v reflect.Value) error {
dt, err := parseDateTime(value.Data)
dt, err := parseDateTime(value.Data, int(value.Raw.Offset))
if err != nil {
return err
}
if v.Kind() != reflect.Interface && v.Type() != timeType {
return unstable.NewParserError(d.p.Raw(value.Raw), int(value.Raw.Offset), "%s", d.typeMismatchString("datetime", v.Type()))
}
v.Set(reflect.ValueOf(dt))
return nil
}
func (d *decoder) unmarshalLocalDate(value *unstable.Node, v reflect.Value) error {
ld, err := parseLocalDate(value.Data)
ld, err := parseLocalDate(value.Data, int(value.Raw.Offset))
if err != nil {
return err
}
if v.Kind() != reflect.Interface && v.Type() != timeType {
return unstable.NewParserError(d.p.Raw(value.Raw), int(value.Raw.Offset), "%s", d.typeMismatchString("local date", v.Type()))
}
if v.Type() == timeType {
cast := ld.AsTime(time.Local)
v.Set(reflect.ValueOf(cast))
v.Set(reflect.ValueOf(ld.AsTime(time.Local)))
return nil
}
v.Set(reflect.ValueOf(ld))
return nil
}
func (d *decoder) unmarshalLocalTime(value *unstable.Node, v reflect.Value) error {
lt, rest, err := parseLocalTime(value.Data)
lt, rest, err := parseLocalTime(value.Data, int(value.Raw.Offset))
if err != nil {
return err
}
if len(rest) > 0 {
return unstable.NewParserError(rest, "extra characters at the end of a local time")
return unstable.NewParserError(rest, int(value.Raw.Offset)+len(value.Data)-len(rest), "extra characters at the end of a local time")
}
if v.Kind() != reflect.Interface {
return unstable.NewParserError(d.p.Raw(value.Raw), int(value.Raw.Offset), "%s", d.typeMismatchString("local time", v.Type()))
}
v.Set(reflect.ValueOf(lt))
return nil
}
func (d *decoder) unmarshalLocalDateTime(value *unstable.Node, v reflect.Value) error {
ldt, rest, err := parseLocalDateTime(value.Data)
ldt, rest, err := parseLocalDateTime(value.Data, int(value.Raw.Offset))
if err != nil {
return err
}
if len(rest) > 0 {
return unstable.NewParserError(rest, "extra characters at the end of a local date time")
return unstable.NewParserError(rest, int(value.Raw.Offset)+len(value.Data)-len(rest), "extra characters at the end of a local date time")
}
if v.Kind() != reflect.Interface && v.Type() != timeType {
return unstable.NewParserError(d.p.Raw(value.Raw), int(value.Raw.Offset), "%s", d.typeMismatchString("local datetime", v.Type()))
}
if v.Type() == timeType {
cast := ldt.AsTime(time.Local)
v.Set(reflect.ValueOf(cast))
v.Set(reflect.ValueOf(ldt.AsTime(time.Local)))
return nil
}
v.Set(reflect.ValueOf(ldt))
return nil
}
@@ -911,14 +992,14 @@ func (d *decoder) unmarshalBool(value *unstable.Node, v reflect.Value) error {
case reflect.Interface:
v.Set(reflect.ValueOf(b))
default:
return unstable.NewParserError(value.Data, "cannot assign boolean to a %t", b)
return unstable.NewParserError(value.Data, int(value.Raw.Offset), "cannot assign boolean to a %t", b)
}
return nil
}
func (d *decoder) unmarshalFloat(value *unstable.Node, v reflect.Value) error {
f, err := parseFloat(value.Data)
f, err := parseFloat(value.Data, int(value.Raw.Offset))
if err != nil {
return err
}
@@ -928,13 +1009,13 @@ func (d *decoder) unmarshalFloat(value *unstable.Node, v reflect.Value) error {
v.SetFloat(f)
case reflect.Float32:
if f > math.MaxFloat32 {
return unstable.NewParserError(value.Data, "number %f does not fit in a float32", f)
return unstable.NewParserError(value.Data, int(value.Raw.Offset), "number %f does not fit in a float32", f)
}
v.SetFloat(f)
case reflect.Interface:
v.Set(reflect.ValueOf(f))
default:
return unstable.NewParserError(value.Data, "float cannot be assigned to %s", v.Kind())
return unstable.NewParserError(value.Data, int(value.Raw.Offset), "float cannot be assigned to %s", v.Kind())
}
return nil
@@ -953,8 +1034,9 @@ const (
// compile time, so it is computed during initialization.
var maxUint int64 = math.MaxInt64
func init() {
func init() { //nolint:gochecknoinits
m := uint64(^uint(0))
// #nosec G115
if m < uint64(maxUint) {
maxUint = int64(m)
}
@@ -966,7 +1048,7 @@ func (d *decoder) unmarshalInteger(value *unstable.Node, v reflect.Value) error
return d.unmarshalFloat(value, v)
}
i, err := parseInteger(value.Data)
i, err := parseInteger(value.Data, int(value.Raw.Offset))
if err != nil {
return err
}
@@ -1034,7 +1116,7 @@ func (d *decoder) unmarshalInteger(value *unstable.Node, v reflect.Value) error
case reflect.Interface:
r = reflect.ValueOf(i)
default:
return unstable.NewParserError(d.p.Raw(value.Raw), "%s", d.typeMismatchString("integer", v.Type()))
return unstable.NewParserError(d.p.Raw(value.Raw), int(value.Raw.Offset), "%s", d.typeMismatchString("integer", v.Type()))
}
if !r.Type().AssignableTo(v.Type()) {
@@ -1053,7 +1135,7 @@ func (d *decoder) unmarshalString(value *unstable.Node, v reflect.Value) error {
case reflect.Interface:
v.Set(reflect.ValueOf(string(value.Data)))
default:
return unstable.NewParserError(d.p.Raw(value.Raw), "%s", d.typeMismatchString("string", v.Type()))
return unstable.NewParserError(d.p.Raw(value.Raw), int(value.Raw.Offset), "%s", d.typeMismatchString("string", v.Type()))
}
return nil
@@ -1104,35 +1186,39 @@ func (d *decoder) keyFromData(keyType reflect.Type, data []byte) (reflect.Value,
return reflect.Value{}, fmt.Errorf("toml: error unmarshalling key type %s from text: %w", stringType, err)
}
return mk.Elem(), nil
}
case keyType.Kind() == reflect.Int || keyType.Kind() == reflect.Int8 || keyType.Kind() == reflect.Int16 || keyType.Kind() == reflect.Int32 || keyType.Kind() == reflect.Int64:
switch keyType.Kind() {
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
key, err := strconv.ParseInt(string(data), 10, 64)
if err != nil {
return reflect.Value{}, fmt.Errorf("toml: error parsing key of type %s from integer: %w", stringType, err)
}
return reflect.ValueOf(key).Convert(keyType), nil
case keyType.Kind() == reflect.Uint || keyType.Kind() == reflect.Uint8 || keyType.Kind() == reflect.Uint16 || keyType.Kind() == reflect.Uint32 || keyType.Kind() == reflect.Uint64:
case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:
key, err := strconv.ParseUint(string(data), 10, 64)
if err != nil {
return reflect.Value{}, fmt.Errorf("toml: error parsing key of type %s from unsigned integer: %w", stringType, err)
}
return reflect.ValueOf(key).Convert(keyType), nil
case keyType.Kind() == reflect.Float32:
case reflect.Float32:
key, err := strconv.ParseFloat(string(data), 32)
if err != nil {
return reflect.Value{}, fmt.Errorf("toml: error parsing key of type %s from float: %w", stringType, err)
}
return reflect.ValueOf(float32(key)), nil
case keyType.Kind() == reflect.Float64:
case reflect.Float64:
key, err := strconv.ParseFloat(string(data), 64)
if err != nil {
return reflect.Value{}, fmt.Errorf("toml: error parsing key of type %s from float: %w", stringType, err)
}
return reflect.ValueOf(float64(key)), nil
default:
return reflect.Value{}, fmt.Errorf("toml: cannot convert map key of type %s to expected type %s", stringType, keyType)
}
return reflect.Value{}, fmt.Errorf("toml: cannot convert map key of type %s to expected type %s", stringType, keyType)
}
func (d *decoder) handleKeyValuePart(key unstable.Iterator, value *unstable.Node, v reflect.Value) (reflect.Value, error) {
@@ -1184,7 +1270,8 @@ func (d *decoder) handleKeyValuePart(key unstable.Iterator, value *unstable.Node
if d.unmarshalerInterface {
if v.CanAddr() && v.Addr().CanInterface() {
if outi, ok := v.Addr().Interface().(unstable.Unmarshaler); ok {
return reflect.Value{}, outi.UnmarshalTOML(value)
// Pass raw bytes from the original document
return reflect.Value{}, outi.UnmarshalTOML(d.p.Raw(value.Raw))
}
}
}
@@ -1294,13 +1381,13 @@ func fieldByIndex(v reflect.Value, path []int) reflect.Value {
type fieldPathsMap = map[string][]int
var globalFieldPathsCache atomic.Value // map[danger.TypeID]fieldPathsMap
var globalFieldPathsCache atomic.Value // map[reflect.Type]fieldPathsMap
func structFieldPath(v reflect.Value, name string) ([]int, bool) {
t := v.Type()
cache, _ := globalFieldPathsCache.Load().(map[danger.TypeID]fieldPathsMap)
fieldPaths, ok := cache[danger.MakeTypeID(t)]
cache, _ := globalFieldPathsCache.Load().(map[reflect.Type]fieldPathsMap)
fieldPaths, ok := cache[t]
if !ok {
fieldPaths = map[string][]int{}
@@ -1311,8 +1398,8 @@ func structFieldPath(v reflect.Value, name string) ([]int, bool) {
fieldPaths[strings.ToLower(name)] = path
})
newCache := make(map[danger.TypeID]fieldPathsMap, len(cache)+1)
newCache[danger.MakeTypeID(t)] = fieldPaths
newCache := make(map[reflect.Type]fieldPathsMap, len(cache)+1)
newCache[t] = fieldPaths
for k, v := range cache {
newCache[k] = v
}
@@ -1336,7 +1423,9 @@ func forEachField(t reflect.Type, path []int, do func(name string, path []int))
continue
}
fieldPath := append(path, i)
fieldPath := make([]int, 0, len(path)+1)
fieldPath = append(fieldPath, path...)
fieldPath = append(fieldPath, i)
fieldPath = fieldPath[:len(fieldPath):len(fieldPath)]
name := f.Tag.Get("toml")
+582 -50
View File
@@ -33,8 +33,8 @@ func (k *unmarshalTextKey) UnmarshalText(text []byte) error {
type unmarshalBadTextKey struct{}
func (k *unmarshalBadTextKey) UnmarshalText(text []byte) error {
return fmt.Errorf("error")
func (k *unmarshalBadTextKey) UnmarshalText([]byte) error {
return errors.New("error")
}
func ExampleDecoder_DisallowUnknownFields() {
@@ -96,10 +96,136 @@ func ExampleUnmarshal() {
// tags: [go toml]
}
// pluginConfig demonstrates how to implement dynamic unmarshaling
// based on a "type" field. This pattern is useful for plugin systems
// or polymorphic configuration.
type pluginConfig struct {
Type string
Config any
}
func (p *pluginConfig) UnmarshalTOML(data []byte) error {
// First, decode just the type field
var typeOnly struct {
Type string `toml:"type"`
}
if err := toml.Unmarshal(data, &typeOnly); err != nil {
return err
}
p.Type = typeOnly.Type
// Now decode the config based on the type
switch typeOnly.Type {
case "database":
var cfg struct {
Type string `toml:"type"`
Host string `toml:"host"`
Port int `toml:"port"`
}
if err := toml.Unmarshal(data, &cfg); err != nil {
return err
}
p.Config = map[string]any{"host": cfg.Host, "port": cfg.Port}
case "cache":
var cfg struct {
Type string `toml:"type"`
TTL int `toml:"ttl"`
}
if err := toml.Unmarshal(data, &cfg); err != nil {
return err
}
p.Config = map[string]any{"ttl": cfg.TTL}
}
return nil
}
// This example demonstrates dynamic unmarshaling based on a discriminator
// field. The pluginConfig type uses UnmarshalTOML to first read the "type"
// field, then decode the rest of the configuration based on that type.
// This pattern is useful for plugin systems or configuration that varies
// by type.
func ExampleDecoder_EnableUnmarshalerInterface_dynamicConfig() {
doc := `
[[plugins]]
type = "database"
host = "localhost"
port = 5432
[[plugins]]
type = "cache"
ttl = 300
`
type Config struct {
Plugins []pluginConfig `toml:"plugins"`
}
var cfg Config
err := toml.NewDecoder(strings.NewReader(doc)).
EnableUnmarshalerInterface().
Decode(&cfg)
if err != nil {
panic(err)
}
for _, p := range cfg.Plugins {
fmt.Printf("type=%s config=%v\n", p.Type, p.Config)
}
// Output:
// type=database config=map[host:localhost port:5432]
// type=cache config=map[ttl:300]
}
// This example demonstrates using RawMessage to capture raw TOML bytes
// for later processing. RawMessage is similar to json.RawMessage - it
// delays decoding so you can inspect the raw content or decode it
// differently based on context.
func ExampleDecoder_EnableUnmarshalerInterface_rawMessage() {
doc := `
[plugin]
name = "example"
version = "1.0"
enabled = true
`
type Config struct {
Plugin unstable.RawMessage `toml:"plugin"`
}
var cfg Config
err := toml.NewDecoder(strings.NewReader(doc)).
EnableUnmarshalerInterface().
Decode(&cfg)
if err != nil {
panic(err)
}
// cfg.Plugin contains the raw TOML bytes
fmt.Printf("Raw TOML captured:\n%s", cfg.Plugin)
// You can later decode it into a specific type
var plugin struct {
Name string `toml:"name"`
Version string `toml:"version"`
Enabled bool `toml:"enabled"`
}
if err := toml.Unmarshal(cfg.Plugin, &plugin); err != nil {
panic(err)
}
fmt.Printf("Decoded: name=%s version=%s enabled=%v\n",
plugin.Name, plugin.Version, plugin.Enabled)
// Output:
// Raw TOML captured:
// name = "example"
// version = "1.0"
// enabled = true
// Decoded: name=example version=1.0 enabled=true
}
type badReader struct{}
func (r *badReader) Read([]byte) (int, error) {
return 0, fmt.Errorf("testing error")
return 0, errors.New("testing error")
}
func TestDecodeReaderError(t *testing.T) {
@@ -111,7 +237,6 @@ func TestDecodeReaderError(t *testing.T) {
assert.Error(t, err)
}
// nolint:funlen
func TestUnmarshal_Integers(t *testing.T) {
examples := []struct {
desc string
@@ -195,7 +320,6 @@ func TestUnmarshal_Integers(t *testing.T) {
}
}
//nolint:funlen
func TestUnmarshal_Floats(t *testing.T) {
examples := []struct {
desc string
@@ -333,7 +457,6 @@ func TestUnmarshal_Floats(t *testing.T) {
}
}
//nolint:funlen
func TestUnmarshal(t *testing.T) {
type test struct {
target interface{}
@@ -410,6 +533,7 @@ foo = "bar"`,
target: &doc{},
expected: &doc{{A: "a", B: "1"}: "foo"},
assert: func(t *testing.T, test test) {
t.Helper()
// Despite the documentation:
// Pointer variable equality is determined based on the equality of the
// referenced values (as opposed to the memory addresses).
@@ -1346,7 +1470,7 @@ B = "data"`,
input: `foo = "bar"`,
gen: func() test {
type doc struct {
foo string
foo string //nolint:unused
}
return test{
target: &doc{},
@@ -1939,9 +2063,6 @@ B = "data"`,
return test{
target: &map[int]string{},
expected: &map[int]string{1: "a"},
assert: func(t *testing.T, test test) {
assert.Equal(t, test.expected, test.target)
},
}
},
},
@@ -1952,9 +2073,6 @@ B = "data"`,
return test{
target: &map[int8]string{},
expected: &map[int8]string{1: "a"},
assert: func(t *testing.T, test test) {
assert.Equal(t, test.expected, test.target)
},
}
},
},
@@ -1965,9 +2083,6 @@ B = "data"`,
return test{
target: &map[int64]string{},
expected: &map[int64]string{1: "a"},
assert: func(t *testing.T, test test) {
assert.Equal(t, test.expected, test.target)
},
}
},
},
@@ -1978,9 +2093,6 @@ B = "data"`,
return test{
target: &map[uint]string{},
expected: &map[uint]string{1: "a"},
assert: func(t *testing.T, test test) {
assert.Equal(t, test.expected, test.target)
},
}
},
},
@@ -1991,9 +2103,6 @@ B = "data"`,
return test{
target: &map[uint8]string{},
expected: &map[uint8]string{1: "a"},
assert: func(t *testing.T, test test) {
assert.Equal(t, test.expected, test.target)
},
}
},
},
@@ -2004,9 +2113,6 @@ B = "data"`,
return test{
target: &map[uint64]string{},
expected: &map[uint64]string{1: "a"},
assert: func(t *testing.T, test test) {
assert.Equal(t, test.expected, test.target)
},
}
},
},
@@ -2027,9 +2133,6 @@ B = "data"`,
return test{
target: &map[float64]string{},
expected: &map[float64]string{1.01: "a"},
assert: func(t *testing.T, test test) {
assert.Equal(t, test.expected, test.target)
},
}
},
},
@@ -2050,9 +2153,6 @@ B = "data"`,
return test{
target: &map[float32]string{},
expected: &map[float32]string{1.01: "a"},
assert: func(t *testing.T, test test) {
assert.Equal(t, test.expected, test.target)
},
}
},
},
@@ -2204,7 +2304,8 @@ port = "bad"
err := toml.NewDecoder(file).Decode(&cfg)
assert.Error(t, err)
x := err.(*toml.DecodeError)
var x *toml.DecodeError
assert.True(t, errors.As(err, &x))
assert.Equal(t, "toml: cannot decode TOML string into struct field toml_test.Server.Port of type int", x.Error())
expected := `1| [server]
2| path = "/my/path"
@@ -2235,7 +2336,8 @@ port = 50
err := toml.NewDecoder(file).Decode(&cfg)
assert.Error(t, err)
x := err.(*toml.DecodeError)
var x *toml.DecodeError
assert.True(t, errors.As(err, &x))
assert.Equal(t, "toml: cannot decode TOML integer into struct field toml_test.Server.Path of type string", x.Error())
expected := `1| [server]
2| path = 100
@@ -2488,7 +2590,7 @@ func TestIssue508(t *testing.T) {
t1 := text{}
err := toml.Unmarshal(b, &t1)
assert.NoError(t, err)
assert.Equal(t, "This is a title", t1.head.Title)
assert.Equal(t, "This is a title", t1.Title)
}
func TestIssue507(t *testing.T) {
@@ -2500,7 +2602,7 @@ func TestIssue507(t *testing.T) {
type uuid [16]byte
func (u *uuid) UnmarshalText(text []byte) (err error) {
func (u *uuid) UnmarshalText([]byte) (err error) {
// Note: the original reported issue had a more complex implementation
// of this function. But the important part is to verify that a
// non-struct type implementing UnmarshalText works with the unmarshal
@@ -2543,7 +2645,7 @@ xz_hash = "1a48f723fea1f17d786ce6eadd9d00914d38062d28fd9c455ed3c3801905b388"
`)
type target struct {
XZ_URL string
XZ_URL string //revive:disable:var-naming
}
type pkg struct {
@@ -2794,7 +2896,7 @@ func TestIssue772(t *testing.T) {
config := Config{}
err := toml.Unmarshal(defaultConfigFile, &config)
assert.NoError(t, err)
assert.Equal(t, "reach-masterdev-", config.FileHandling.FilePattern)
assert.Equal(t, "reach-masterdev-", config.FilePattern)
}
func TestIssue774(t *testing.T) {
@@ -2954,7 +3056,7 @@ blah.a = "def"`)
assert.NoError(t, err)
assert.Equal(t, "abc", cfg.Fizz)
assert.Equal(t, "def", cfg.blah.A)
assert.Equal(t, "def", cfg.A)
assert.Equal(t, "def", cfg.A)
}
@@ -3184,7 +3286,17 @@ world'`,
{
desc: "invalid seconds value",
data: `a=1979-05-27T12:45:99`,
msg: `seconds cannot be greater 60`,
msg: `seconds cannot be greater than 59`,
},
{
desc: "leap second not supported",
data: `a=1979-05-27T12:45:60`,
msg: `seconds cannot be greater than 59`,
},
{
desc: "leap second with max date causes overflow",
data: `s=9999-12-31 23:59:60z`,
msg: `seconds cannot be greater than 59`,
},
{
desc: `binary with invalid digit`,
@@ -3484,7 +3596,7 @@ world'`,
func TestOmitEmpty(t *testing.T) {
type inner struct {
private string
private string //nolint:unused
Skip string `toml:"-"`
V string
}
@@ -3600,7 +3712,6 @@ func TestASCIIControlCharacters(t *testing.T) {
}
}
//nolint:funlen
func TestLocalDateTime(t *testing.T) {
examples := []struct {
desc string
@@ -3915,10 +4026,10 @@ type CustomUnmarshalerKey struct {
A int64
}
func (k *CustomUnmarshalerKey) UnmarshalTOML(value *unstable.Node) error {
item, err := strconv.ParseInt(string(value.Data), 10, 64)
func (k *CustomUnmarshalerKey) UnmarshalTOML(data []byte) error {
item, err := strconv.ParseInt(string(data), 10, 64)
if err != nil {
return fmt.Errorf("error converting to int64, %v", err)
return fmt.Errorf("error converting to int64, %w", err)
}
k.A = item
return nil
@@ -4004,7 +4115,7 @@ foo = "bar"`,
type doc994 struct{}
func (d *doc994) UnmarshalTOML(value *unstable.Node) error {
func (d *doc994) UnmarshalTOML([]byte) error {
return errors.New("expected-error")
}
@@ -4027,8 +4138,8 @@ type doc994ok struct {
S string
}
func (d *doc994ok) UnmarshalTOML(value *unstable.Node) error {
d.S = string(value.Data) + " from unmarshaler"
func (d *doc994ok) UnmarshalTOML(data []byte) error {
d.S = string(data) + " from unmarshaler"
return nil
}
@@ -4041,7 +4152,8 @@ func TestIssue994_OK(t *testing.T) {
Decode(&d)
assert.NoError(t, err)
assert.Equal(t, "bar from unmarshaler", d.S)
// With bytes-based interface, raw TOML bytes are passed including quotes
assert.Equal(t, "\"bar\" from unmarshaler", d.S)
}
func TestIssue995(t *testing.T) {
@@ -4237,8 +4349,428 @@ func TestIssue995_SliceNonEmpty_UsesLastElement(t *testing.T) {
}
assert.Equal(t, 2, len(r.Rules[0].Allowlists))
// Values presence check
got := []string{r.Rules[0].Allowlists[0].Description, r.Rules[0].Allowlists[1].Description}
if !(got[0] == "a" && got[1] == "b") && !(got[0] == "b" && got[1] == "a") {
got := [...]string{r.Rules[0].Allowlists[0].Description, r.Rules[0].Allowlists[1].Description}
if got != [2]string{"a", "b"} && got != [2]string{"b", "a"} {
t.Fatalf("unexpected values in allowlists: %v", got)
}
}
// fooConfig974 is a struct that implements UnmarshalText for simple string
// parsing, but can also be populated field-by-field from a TOML table.
type fooConfig974 struct {
Name string `toml:"name"`
Value string `toml:"value"`
}
func (f *fooConfig974) UnmarshalText(text []byte) error {
s := string(text)
f.Name = s
f.Value = s
return nil
}
type config974 struct {
Foo []fooConfig974 `toml:"foo"`
}
func TestIssue974_UnmarshalTextFallbackToStructForInlineTable(t *testing.T) {
// When the TOML value is an inline table, the unmarshaler should skip
// UnmarshalText and populate the struct fields directly.
doc := `foo = [{name = "a", value = "a"}, {name = "b", value = "b"}]`
var cfg config974
err := toml.Unmarshal([]byte(doc), &cfg)
assert.NoError(t, err)
assert.Equal(t, config974{
Foo: []fooConfig974{
{Name: "a", Value: "a"},
{Name: "b", Value: "b"},
},
}, cfg)
}
func TestIssue974_UnmarshalTextStillWorksForStrings(t *testing.T) {
// When the TOML value is a string, UnmarshalText should still be used.
doc := `foo = ["a", "b"]`
var cfg config974
err := toml.Unmarshal([]byte(doc), &cfg)
assert.NoError(t, err)
assert.Equal(t, config974{
Foo: []fooConfig974{
{Name: "a", Value: "a"},
{Name: "b", Value: "b"},
},
}, cfg)
}
// singleFooConfig974 tests the inline table case for a single value (not array)
type singleConfig974 struct {
Foo fooConfig974 `toml:"foo"`
}
func TestIssue974_SingleInlineTable(t *testing.T) {
// A single inline table should also skip UnmarshalText
doc := `foo = {name = "hello", value = "world"}`
var cfg singleConfig974
err := toml.Unmarshal([]byte(doc), &cfg)
assert.NoError(t, err)
assert.Equal(t, singleConfig974{
Foo: fooConfig974{Name: "hello", Value: "world"},
}, cfg)
}
func TestIssue974_SingleString(t *testing.T) {
// A single string should use UnmarshalText
doc := `foo = "hello"`
var cfg singleConfig974
err := toml.Unmarshal([]byte(doc), &cfg)
assert.NoError(t, err)
assert.Equal(t, singleConfig974{
Foo: fooConfig974{Name: "hello", Value: "hello"},
}, cfg)
}
func TestIssue974_TableSyntax(t *testing.T) {
// Regular table syntax should also work (uses struct unmarshaling)
doc := `
[foo]
name = "hello"
value = "world"
`
var cfg singleConfig974
err := toml.Unmarshal([]byte(doc), &cfg)
assert.NoError(t, err)
assert.Equal(t, singleConfig974{
Foo: fooConfig974{Name: "hello", Value: "world"},
}, cfg)
}
func TestIssue974_ArrayTableSyntax(t *testing.T) {
// Array of tables syntax should also work
doc := `
[[foo]]
name = "a"
value = "a"
[[foo]]
name = "b"
value = "b"
`
var cfg config974
err := toml.Unmarshal([]byte(doc), &cfg)
assert.NoError(t, err)
assert.Equal(t, config974{
Foo: []fooConfig974{
{Name: "a", Value: "a"},
{Name: "b", Value: "b"},
},
}, cfg)
}
func TestIssue1028(t *testing.T) {
// Datetime values assigned to incompatible types should return an error,
// not panic.
type Item struct {
Name string `toml:"name"`
}
type Config struct {
Items map[string]Item `toml:"items"`
}
// Error: "cannot decode TOML datetime into struct field Config.Items of type map[string]Item"
t.Run("OffsetDateTime", func(t *testing.T) {
var c Config
err := toml.Unmarshal([]byte(`items = 2023-01-01T10:20:30Z`), &c)
assert.Error(t, err)
})
// Error: "cannot decode TOML local datetime into struct field Config.Items of type map[string]Item"
t.Run("LocalDateTime", func(t *testing.T) {
var c Config
err := toml.Unmarshal([]byte(`items = 2023-01-01T10:20:30`), &c)
assert.Error(t, err)
})
// Error: "cannot decode TOML local date into struct field Config.Items of type map[string]Item"
t.Run("LocalDate", func(t *testing.T) {
var c Config
err := toml.Unmarshal([]byte(`items = 2023-01-01`), &c)
assert.Error(t, err)
})
// Error: "cannot decode TOML local time into struct field Config.Items of type map[string]Item"
t.Run("LocalTime", func(t *testing.T) {
var c Config
err := toml.Unmarshal([]byte(`items = 10:20:30`), &c)
assert.Error(t, err)
})
}
// Tests for issue #873 - Bring back toml.Unmarshaler for tables and arrays
type customTable873 struct {
Keys []string
Values map[string]string
}
func (c *customTable873) UnmarshalTOML(data []byte) error {
c.Keys = []string{}
c.Values = make(map[string]string)
// Parse the raw TOML bytes into a map to extract keys in order
// For this test, we use a simple line-by-line parser to preserve order
lines := bytes.Split(data, []byte{'\n'})
for _, line := range lines {
line = bytes.TrimSpace(line)
if len(line) == 0 {
continue
}
// Skip table headers
if line[0] == '[' {
continue
}
// Parse key = value
eqIdx := bytes.Index(line, []byte{'='})
if eqIdx < 0 {
continue
}
key := string(bytes.TrimSpace(line[:eqIdx]))
// Remove quotes from quoted keys
if len(key) >= 2 && key[0] == '"' && key[len(key)-1] == '"' {
key = key[1 : len(key)-1]
}
valueBytes := bytes.TrimSpace(line[eqIdx+1:])
// Remove quotes from string values
if len(valueBytes) >= 2 && valueBytes[0] == '"' && valueBytes[len(valueBytes)-1] == '"' {
valueBytes = valueBytes[1 : len(valueBytes)-1]
}
c.Keys = append(c.Keys, key)
c.Values[key] = string(valueBytes)
}
return nil
}
// Test for split tables - when the same parent table is defined in multiple places
// This is a key requirement for issue #873: if type A implements Unmarshaler,
// and [a.b] and [a.d] are defined with another table [x] in between,
// A should receive content for both b and d, but not x.
func TestIssue873_SplitTables(t *testing.T) {
// For this test, we expect each sub-table to be handled separately
// The parent doesn't receive the sub-tables directly - each sub-table
// (b and d) gets its own call to handleKeyValues
type Config struct {
A struct {
B customTable873 `toml:"b"`
D customTable873 `toml:"d"`
} `toml:"a"`
X customTable873 `toml:"x"`
}
doc := `
[a.b]
C = "1"
[x]
Y = "100"
[a.d]
E = "2"
`
var cfg Config
err := toml.NewDecoder(bytes.NewReader([]byte(doc))).
EnableUnmarshalerInterface().
Decode(&cfg)
assert.NoError(t, err)
// Each sub-table should have received its own key-values
assert.Equal(t, []string{"C"}, cfg.A.B.Keys)
assert.Equal(t, "1", cfg.A.B.Values["C"])
assert.Equal(t, []string{"E"}, cfg.A.D.Keys)
assert.Equal(t, "2", cfg.A.D.Values["E"])
assert.Equal(t, []string{"Y"}, cfg.X.Keys)
assert.Equal(t, "100", cfg.X.Values["Y"])
}
// Test using RawMessage to capture raw TOML bytes
func TestIssue873_RawMessage(t *testing.T) {
type Config struct {
Plugin unstable.RawMessage `toml:"plugin"`
}
doc := `
[plugin]
name = "example"
version = "1.0"
`
var cfg Config
err := toml.NewDecoder(bytes.NewReader([]byte(doc))).
EnableUnmarshalerInterface().
Decode(&cfg)
assert.NoError(t, err)
// RawMessage should contain the raw key-value bytes
expected := "name = \"example\"\nversion = \"1.0\"\n"
assert.Equal(t, expected, string(cfg.Plugin))
}
// Test keys that need quoting (contain special characters)
func TestIssue873_QuotedKeys(t *testing.T) {
type Config struct {
Section customTable873 `toml:"section"`
}
doc := `
[section]
"key with spaces" = "value1"
"key.with.dots" = "value2"
`
var cfg Config
err := toml.NewDecoder(bytes.NewReader([]byte(doc))).
EnableUnmarshalerInterface().
Decode(&cfg)
assert.NoError(t, err)
assert.Equal(t, 2, len(cfg.Section.Keys))
assert.Equal(t, "value1", cfg.Section.Values["key with spaces"])
assert.Equal(t, "value2", cfg.Section.Values["key.with.dots"])
}
// errorUnmarshaler873 is used to test error propagation from UnmarshalTOML
type errorUnmarshaler873 struct{}
func (e *errorUnmarshaler873) UnmarshalTOML([]byte) error {
return errors.New("intentional error")
}
// Test error propagation from UnmarshalTOML
func TestIssue873_UnmarshalerError(t *testing.T) {
doc := `
[section]
key = "value"
`
type Config struct {
Section errorUnmarshaler873 `toml:"section"`
}
var cfg Config
err := toml.NewDecoder(bytes.NewReader([]byte(doc))).
EnableUnmarshalerInterface().
Decode(&cfg)
assert.Error(t, err)
assert.True(t, strings.Contains(err.Error(), "intentional error"))
}
// Test dotted keys in a table (e.g., a.b = value)
func TestIssue873_DottedKeys(t *testing.T) {
type Config struct {
Section customTable873 `toml:"section"`
}
doc := `
[section]
sub.key = "value1"
another.nested.key = "value2"
`
var cfg Config
err := toml.NewDecoder(bytes.NewReader([]byte(doc))).
EnableUnmarshalerInterface().
Decode(&cfg)
assert.NoError(t, err)
assert.Equal(t, 2, len(cfg.Section.Keys))
// The dotted keys should be preserved in the raw output
assert.Equal(t, "value1", cfg.Section.Values["sub.key"])
assert.Equal(t, "value2", cfg.Section.Values["another.nested.key"])
}
// Test pointer to pointer to Unmarshaler (covers pointer dereferencing loop)
func TestIssue873_DoublePointerUnmarshaler(t *testing.T) {
type Config struct {
Section **customTable873 `toml:"section"`
}
doc := `
[section]
key = "value"
`
var cfg Config
err := toml.NewDecoder(bytes.NewReader([]byte(doc))).
EnableUnmarshalerInterface().
Decode(&cfg)
assert.NoError(t, err)
assert.True(t, cfg.Section != nil)
assert.True(t, *cfg.Section != nil)
assert.Equal(t, []string{"key"}, (*cfg.Section).Keys)
assert.Equal(t, "value", (*cfg.Section).Values["key"])
}
// formattingCapture captures the raw TOML bytes to verify formatting preservation
type formattingCapture struct {
RawBytes string
}
func (f *formattingCapture) UnmarshalTOML(data []byte) error {
f.RawBytes = string(data)
return nil
}
func TestIssue873_FormattingPreservation(t *testing.T) {
type Config struct {
Section *formattingCapture `toml:"section"`
}
// Test that various formatting styles are preserved:
// - Extra spaces around '='
// - Literal strings (single quotes)
// - Hex numbers
// - Inline tables
doc := `[section]
key1 = "value with spaces"
key2 = 'literal string'
hex_val = 0xDEADBEEF
inline = { a = 1, b = 2 }
`
var cfg Config
err := toml.NewDecoder(bytes.NewReader([]byte(doc))).
EnableUnmarshalerInterface().
Decode(&cfg)
assert.NoError(t, err)
assert.True(t, cfg.Section != nil)
// The raw bytes should preserve original formatting
raw := cfg.Section.RawBytes
// Check that extra spaces around '=' are preserved
assert.True(t, strings.Contains(raw, "key1 = \"value with spaces\""),
"Expected spacing to be preserved, got: %s", raw)
// Check that literal string style is preserved
assert.True(t, strings.Contains(raw, "key2 = 'literal string'"),
"Expected literal string to be preserved, got: %s", raw)
// Check that hex format is preserved
assert.True(t, strings.Contains(raw, "hex_val = 0xDEADBEEF"),
"Expected hex format to be preserved, got: %s", raw)
// Check that inline table is preserved
assert.True(t, strings.Contains(raw, "inline = { a = 1, b = 2 }"),
"Expected inline table to be preserved, got: %s", raw)
}
+41 -28
View File
@@ -1,10 +1,8 @@
package unstable
import (
"errors"
"fmt"
"unsafe"
"github.com/pelletier/go-toml/v2/internal/danger"
)
// Iterator over a sequence of nodes.
@@ -19,30 +17,43 @@ import (
// // do something with n
// }
type Iterator struct {
nodes *[]Node
idx int32
started bool
node *Node
}
// Next moves the iterator forward and returns true if points to a
// node, false otherwise.
func (c *Iterator) Next() bool {
if c.nodes == nil {
return false
}
nodes := *c.nodes
if !c.started {
c.started = true
} else if c.node.Valid() {
c.node = c.node.Next()
} else {
idx := c.idx
if idx >= 0 && int(idx) < len(nodes) {
c.idx = nodes[idx].next
}
}
return c.node.Valid()
return c.idx >= 0 && int(c.idx) < len(nodes)
}
// IsLast returns true if the current node of the iterator is the last
// one. Subsequent calls to Next() will return false.
func (c *Iterator) IsLast() bool {
return c.node.next == 0
return c.nodes == nil || c.idx < 0 || (*c.nodes)[c.idx].next < 0
}
// Node returns a pointer to the node pointed at by the iterator.
func (c *Iterator) Node() *Node {
return c.node
if c.nodes == nil || c.idx < 0 {
return nil
}
n := &(*c.nodes)[c.idx]
n.nodes = c.nodes
return n
}
// Node in a TOML expression AST.
@@ -65,11 +76,12 @@ type Node struct {
Raw Range // Raw bytes from the input.
Data []byte // Node value (either allocated or referencing the input).
// References to other nodes, as offsets in the backing array
// from this node. References can go backward, so those can be
// negative.
next int // 0 if last element
child int // 0 if no child
// Absolute indices into the backing nodes slice. -1 means none.
next int32
child int32
// Reference to the backing nodes slice for navigation.
nodes *[]Node
}
// Range of bytes in the document.
@@ -80,24 +92,24 @@ type Range struct {
// Next returns a pointer to the next node, or nil if there is no next node.
func (n *Node) Next() *Node {
if n.next == 0 {
if n.next < 0 {
return nil
}
ptr := unsafe.Pointer(n)
size := unsafe.Sizeof(Node{})
return (*Node)(danger.Stride(ptr, size, n.next))
next := &(*n.nodes)[n.next]
next.nodes = n.nodes
return next
}
// Child returns a pointer to the first child node of this node. Other children
// can be accessed calling Next on the first child. Returns nil if this Node
// has no child.
func (n *Node) Child() *Node {
if n.child == 0 {
if n.child < 0 {
return nil
}
ptr := unsafe.Pointer(n)
size := unsafe.Sizeof(Node{})
return (*Node)(danger.Stride(ptr, size, n.child))
child := &(*n.nodes)[n.child]
child.nodes = n.nodes
return child
}
// Valid returns true if the node's kind is set (not to Invalid).
@@ -111,13 +123,14 @@ func (n *Node) Valid() bool {
func (n *Node) Key() Iterator {
switch n.Kind {
case KeyValue:
value := n.Child()
if !value.Valid() {
panic(fmt.Errorf("KeyValue should have at least two children"))
child := n.child
if child < 0 {
panic(errors.New("KeyValue should have at least two children"))
}
return Iterator{node: value.Next()}
valueNode := &(*n.nodes)[child]
return Iterator{nodes: n.nodes, idx: valueNode.next}
case Table, ArrayTable:
return Iterator{node: n.Child()}
return Iterator{nodes: n.nodes, idx: n.child}
default:
panic(fmt.Errorf("Key() is not supported on a %s", n.Kind))
}
@@ -132,5 +145,5 @@ func (n *Node) Value() *Node {
// Children returns an iterator over a node's children.
func (n *Node) Children() Iterator {
return Iterator{node: n.Child()}
return Iterator{nodes: n.nodes, idx: n.child}
}
+16 -14
View File
@@ -5,12 +5,14 @@ import (
"testing"
)
var valid10Ascii = []byte("1234567890")
var valid10Utf8 = []byte("日本語a")
var valid1kUtf8 = bytes.Repeat([]byte("0123456789日本語日本語日本語日abcdefghijklmnopqrstuvwx"), 16)
var valid1MUtf8 = bytes.Repeat(valid1kUtf8, 1024)
var valid1kAscii = bytes.Repeat([]byte("012345678998jhjklasDJKLAAdjdfjsdklfjdslkabcdefghijklmnopqrstuvwx"), 16)
var valid1MAscii = bytes.Repeat(valid1kAscii, 1024)
var (
valid10ASCII = []byte("1234567890")
valid10Utf8 = []byte("日本語a")
valid1kUtf8 = bytes.Repeat([]byte("0123456789日本語日本語日本語日abcdefghijklmnopqrstuvwx"), 16)
valid1MUtf8 = bytes.Repeat(valid1kUtf8, 1024)
valid1kASCII = bytes.Repeat([]byte("012345678998jhjklasDJKLAAdjdfjsdklfjdslkabcdefghijklmnopqrstuvwx"), 16)
valid1MASCII = bytes.Repeat(valid1kASCII, 1024)
)
func BenchmarkScanComments(b *testing.B) {
wrap := func(x []byte) []byte {
@@ -18,9 +20,9 @@ func BenchmarkScanComments(b *testing.B) {
}
inputs := map[string][]byte{
"10Valid": wrap(valid10Ascii),
"1kValid": wrap(valid1kAscii),
"1MValid": wrap(valid1MAscii),
"10Valid": wrap(valid10ASCII),
"1kValid": wrap(valid1kASCII),
"1MValid": wrap(valid1MASCII),
"10ValidUtf8": wrap(valid10Utf8),
"1kValidUtf8": wrap(valid1kUtf8),
"1MValidUtf8": wrap(valid1MUtf8),
@@ -33,7 +35,7 @@ func BenchmarkScanComments(b *testing.B) {
b.ResetTimer()
for i := 0; i < b.N; i++ {
scanComment(input)
_, _, _ = scanComment(input, 0)
}
})
}
@@ -45,9 +47,9 @@ func BenchmarkParseLiteralStringValid(b *testing.B) {
}
inputs := map[string][]byte{
"10Valid": wrap(valid10Ascii),
"1kValid": wrap(valid1kAscii),
"1MValid": wrap(valid1MAscii),
"10Valid": wrap(valid10ASCII),
"1kValid": wrap(valid1kASCII),
"1MValid": wrap(valid1MASCII),
"10ValidUtf8": wrap(valid10Utf8),
"1kValidUtf8": wrap(valid1kUtf8),
"1MValidUtf8": wrap(valid1MUtf8),
@@ -63,7 +65,7 @@ func BenchmarkParseLiteralStringValid(b *testing.B) {
for i := 0; i < b.N; i++ {
_, _, _, err := p.parseLiteralString(input)
if err != nil {
panic(err)
b.Error(err)
}
}
})
+10 -17
View File
@@ -7,15 +7,6 @@ type root struct {
nodes []Node
}
// Iterator over the top level nodes.
func (r *root) Iterator() Iterator {
it := Iterator{}
if len(r.nodes) > 0 {
it.node = &r.nodes[0]
}
return it
}
func (r *root) at(idx reference) *Node {
return &r.nodes[idx]
}
@@ -33,12 +24,10 @@ type builder struct {
lastIdx int
}
func (b *builder) Tree() *root {
return &b.tree
}
func (b *builder) NodeAt(ref reference) *Node {
return b.tree.at(ref)
n := b.tree.at(ref)
n.nodes = &b.tree.nodes
return n
}
func (b *builder) Reset() {
@@ -48,24 +37,28 @@ func (b *builder) Reset() {
func (b *builder) Push(n Node) reference {
b.lastIdx = len(b.tree.nodes)
n.next = -1
n.child = -1
b.tree.nodes = append(b.tree.nodes, n)
return reference(b.lastIdx)
}
func (b *builder) PushAndChain(n Node) reference {
newIdx := len(b.tree.nodes)
n.next = -1
n.child = -1
b.tree.nodes = append(b.tree.nodes, n)
if b.lastIdx >= 0 {
b.tree.nodes[b.lastIdx].next = newIdx - b.lastIdx
b.tree.nodes[b.lastIdx].next = int32(newIdx) //nolint:gosec // TOML ASTs are small
}
b.lastIdx = newIdx
return reference(b.lastIdx)
}
func (b *builder) AttachChild(parent reference, child reference) {
b.tree.nodes[parent].child = int(child) - int(parent)
b.tree.nodes[parent].child = int32(child) //nolint:gosec // TOML ASTs are small
}
func (b *builder) Chain(from reference, to reference) {
b.tree.nodes[from].next = int(to) - int(from)
b.tree.nodes[from].next = int32(to) //nolint:gosec // TOML ASTs are small
}
+16 -4
View File
@@ -6,28 +6,40 @@ import "fmt"
type Kind int
const (
// Meta
// Invalid represents an invalid meta node.
Invalid Kind = iota
// Comment represents a comment meta node.
Comment
// Key represents a key meta node.
Key
// Top level structures
// Table represents a top-level table.
Table
// ArrayTable represents a top-level array table.
ArrayTable
// KeyValue represents a top-level key value.
KeyValue
// Containers values
// Array represents an array container value.
Array
// InlineTable represents an inline table container value.
InlineTable
// Values
// String represents a string value.
String
// Bool represents a boolean value.
Bool
// Float represents a floating point value.
Float
// Integer represents an integer value.
Integer
// LocalDate represents a a local date value.
LocalDate
// LocalTime represents a local time value.
LocalTime
// LocalDateTime represents a local date/time value.
LocalDateTime
// DateTime represents a data/time value.
DateTime
)
+103 -93
View File
@@ -6,7 +6,6 @@ import (
"unicode"
"github.com/pelletier/go-toml/v2/internal/characters"
"github.com/pelletier/go-toml/v2/internal/danger"
)
// ParserError describes an error relative to the content of the document.
@@ -17,6 +16,7 @@ type ParserError struct {
Highlight []byte
Message string
Key []string // optional
Offset int
}
// Error is the implementation of the error interface.
@@ -28,9 +28,10 @@ func (e *ParserError) Error() string {
//
// Warning: Highlight needs to be a subslice of Parser.data, so only slices
// returned by Parser.Raw are valid candidates.
func NewParserError(highlight []byte, format string, args ...interface{}) error {
func NewParserError(highlight []byte, offset int, format string, args ...interface{}) error {
return &ParserError{
Highlight: highlight,
Offset: offset,
Message: fmt.Errorf(format, args...).Error(),
}
}
@@ -65,14 +66,16 @@ func (p *Parser) Data() []byte {
return p.data
}
// Range returns a range description that corresponds to a given slice of the
// input. If the argument is not a subslice of the parser input, this function
// panics.
func (p *Parser) Range(b []byte) Range {
return Range{
Offset: uint32(danger.SubsliceOffset(p.data, b)),
Length: uint32(len(b)),
}
func (p *Parser) offsetOf(b []byte) int {
return len(p.data) - len(b)
}
// rangeOfToken computes the Range of a token given the remaining bytes after the token.
// This is used when the token was extracted from the beginning of some position,
// and 'rest' is what remains after the token.
func (p *Parser) rangeOfToken(token, rest []byte) Range {
offset := len(p.data) - len(token) - len(rest)
return Range{Offset: uint32(offset), Length: uint32(len(token))} //nolint:gosec // TOML documents are small
}
// Raw returns the slice corresponding to the bytes in the given range.
@@ -158,9 +161,17 @@ type Shape struct {
End Position
}
func (p *Parser) position(b []byte) Position {
offset := danger.SubsliceOffset(p.data, b)
// Shape returns the shape of the given range in the input. Will
// panic if the range is not a subslice of the input.
func (p *Parser) Shape(r Range) Shape {
return Shape{
Start: p.positionAt(int(r.Offset)),
End: p.positionAt(int(r.Offset + r.Length)),
}
}
// positionAt returns the position at the given byte offset in the document.
func (p *Parser) positionAt(offset int) Position {
lead := p.data[:offset]
return Position{
@@ -170,36 +181,26 @@ func (p *Parser) position(b []byte) Position {
}
}
// Shape returns the shape of the given range in the input. Will
// panic if the range is not a subslice of the input.
func (p *Parser) Shape(r Range) Shape {
raw := p.Raw(r)
return Shape{
Start: p.position(raw),
End: p.position(raw[r.Length:]),
}
}
func (p *Parser) parseNewline(b []byte) ([]byte, error) {
if b[0] == '\n' {
return b[1:], nil
}
if b[0] == '\r' {
_, rest, err := scanWindowsNewline(b)
_, rest, err := scanWindowsNewline(b, p.offsetOf(b))
return rest, err
}
return nil, NewParserError(b[0:1], "expected newline but got %#U", b[0])
return nil, NewParserError(b[0:1], p.offsetOf(b), "expected newline but got %#U", b[0])
}
func (p *Parser) parseComment(b []byte) (reference, []byte, error) {
ref := invalidReference
data, rest, err := scanComment(b)
data, rest, err := scanComment(b, p.offsetOf(b))
if p.KeepComments && err == nil {
ref = p.builder.Push(Node{
Kind: Comment,
Raw: p.Range(data),
Raw: p.rangeOfToken(data, rest),
Data: data,
})
}
@@ -279,12 +280,12 @@ func (p *Parser) parseArrayTable(b []byte) (reference, []byte, error) {
p.builder.AttachChild(ref, k)
b = p.parseWhitespace(b)
b, err = expect(']', b)
b, err = expect(']', b, p.offsetOf(b))
if err != nil {
return ref, nil, err
}
b, err = expect(']', b)
b, err = expect(']', b, p.offsetOf(b))
return ref, b, err
}
@@ -309,13 +310,16 @@ func (p *Parser) parseStdTable(b []byte) (reference, []byte, error) {
b = p.parseWhitespace(b)
b, err = expect(']', b)
b, err = expect(']', b, p.offsetOf(b))
return ref, b, err
}
func (p *Parser) parseKeyval(b []byte) (reference, []byte, error) {
// keyval = key keyval-sep val
// Track the start position for Raw range
startB := b
ref := p.builder.Push(Node{
Kind: KeyValue,
})
@@ -330,10 +334,10 @@ func (p *Parser) parseKeyval(b []byte) (reference, []byte, error) {
b = p.parseWhitespace(b)
if len(b) == 0 {
return invalidReference, nil, NewParserError(b, "expected = after a key, but the document ends there")
return invalidReference, nil, NewParserError(startB[:len(startB)-len(b)], p.offsetOf(startB), "expected = after a key, but the document ends there")
}
b, err = expect('=', b)
b, err = expect('=', b, p.offsetOf(b))
if err != nil {
return invalidReference, nil, err
}
@@ -348,6 +352,11 @@ func (p *Parser) parseKeyval(b []byte) (reference, []byte, error) {
p.builder.Chain(valRef, key)
p.builder.AttachChild(ref, valRef)
// Set Raw to span the entire key-value expression.
// Access the node directly in the slice to avoid the write barrier
// that NodeAt's nodes-pointer setup would trigger.
p.builder.tree.nodes[ref].Raw = p.rangeOfToken(startB[:len(startB)-len(b)], b)
return ref, b, err
}
@@ -357,7 +366,7 @@ func (p *Parser) parseVal(b []byte) (reference, []byte, error) {
ref := invalidReference
if len(b) == 0 {
return ref, nil, NewParserError(b, "expected value, not eof")
return ref, nil, NewParserError(b, p.offsetOf(b), "expected value, not eof")
}
var err error
@@ -376,7 +385,7 @@ func (p *Parser) parseVal(b []byte) (reference, []byte, error) {
if err == nil {
ref = p.builder.Push(Node{
Kind: String,
Raw: p.Range(raw),
Raw: p.rangeOfToken(raw, b),
Data: v,
})
}
@@ -394,7 +403,7 @@ func (p *Parser) parseVal(b []byte) (reference, []byte, error) {
if err == nil {
ref = p.builder.Push(Node{
Kind: String,
Raw: p.Range(raw),
Raw: p.rangeOfToken(raw, b),
Data: v,
})
}
@@ -402,7 +411,7 @@ func (p *Parser) parseVal(b []byte) (reference, []byte, error) {
return ref, b, err
case 't':
if !scanFollowsTrue(b) {
return ref, nil, NewParserError(atmost(b, 4), "expected 'true'")
return ref, nil, NewParserError(atmost(b, 4), p.offsetOf(b), "expected 'true'")
}
ref = p.builder.Push(Node{
@@ -413,7 +422,7 @@ func (p *Parser) parseVal(b []byte) (reference, []byte, error) {
return ref, b[4:], nil
case 'f':
if !scanFollowsFalse(b) {
return ref, nil, NewParserError(atmost(b, 5), "expected 'false'")
return ref, nil, NewParserError(atmost(b, 5), p.offsetOf(b), "expected 'false'")
}
ref = p.builder.Push(Node{
@@ -440,7 +449,7 @@ func atmost(b []byte, n int) []byte {
}
func (p *Parser) parseLiteralString(b []byte) ([]byte, []byte, []byte, error) {
v, rest, err := scanLiteralString(b)
v, rest, err := scanLiteralString(b, p.offsetOf(b))
if err != nil {
return nil, nil, nil, err
}
@@ -456,7 +465,7 @@ func (p *Parser) parseInlineTable(b []byte) (reference, []byte, error) {
// inline-table-keyvals = keyval [ inline-table-sep inline-table-keyvals ]
parent := p.builder.Push(Node{
Kind: InlineTable,
Raw: p.Range(b[:1]),
Raw: p.rangeOfToken(b[:1], b[1:]),
})
first := true
@@ -472,7 +481,7 @@ func (p *Parser) parseInlineTable(b []byte) (reference, []byte, error) {
b = p.parseWhitespace(b)
if len(b) == 0 {
return parent, nil, NewParserError(previousB[:1], "inline table is incomplete")
return parent, nil, NewParserError(previousB[:1], p.offsetOf(previousB), "inline table is incomplete")
}
if b[0] == '}' {
@@ -480,7 +489,7 @@ func (p *Parser) parseInlineTable(b []byte) (reference, []byte, error) {
}
if !first {
b, err = expect(',', b)
b, err = expect(',', b, p.offsetOf(b))
if err != nil {
return parent, nil, err
}
@@ -504,7 +513,7 @@ func (p *Parser) parseInlineTable(b []byte) (reference, []byte, error) {
first = false
}
rest, err := expect('}', b)
rest, err := expect('}', b, p.offsetOf(b))
return parent, rest, err
}
@@ -542,7 +551,7 @@ func (p *Parser) parseValArray(b []byte) (reference, []byte, error) {
var err error
for len(b) > 0 {
cref := invalidReference
var cref reference
cref, b, err = p.parseOptionalWhitespaceCommentNewline(b)
if err != nil {
return parent, nil, err
@@ -553,7 +562,7 @@ func (p *Parser) parseValArray(b []byte) (reference, []byte, error) {
}
if len(b) == 0 {
return parent, nil, NewParserError(arrayStart[:1], "array is incomplete")
return parent, nil, NewParserError(arrayStart[:1], p.offsetOf(arrayStart), "array is incomplete")
}
if b[0] == ']' {
@@ -562,7 +571,7 @@ func (p *Parser) parseValArray(b []byte) (reference, []byte, error) {
if b[0] == ',' {
if first {
return parent, nil, NewParserError(b[0:1], "array cannot start with comma")
return parent, nil, NewParserError(b[0:1], p.offsetOf(b), "array cannot start with comma")
}
b = b[1:]
@@ -574,7 +583,7 @@ func (p *Parser) parseValArray(b []byte) (reference, []byte, error) {
addChild(cref)
}
} else if !first {
return parent, nil, NewParserError(b[0:1], "array elements must be separated by commas")
return parent, nil, NewParserError(b[0:1], p.offsetOf(b), "array elements must be separated by commas")
}
// TOML allows trailing commas in arrays.
@@ -601,7 +610,7 @@ func (p *Parser) parseValArray(b []byte) (reference, []byte, error) {
first = false
}
rest, err := expect(']', b)
rest, err := expect(']', b, p.offsetOf(b))
return parent, rest, err
}
@@ -611,12 +620,13 @@ func (p *Parser) parseOptionalWhitespaceCommentNewline(b []byte) (reference, []b
latestCommentRef := invalidReference
addComment := func(ref reference) {
if rootCommentRef == invalidReference {
switch {
case rootCommentRef == invalidReference:
rootCommentRef = ref
} else if latestCommentRef == invalidReference {
case latestCommentRef == invalidReference:
p.builder.AttachChild(rootCommentRef, ref)
latestCommentRef = ref
} else {
default:
p.builder.Chain(latestCommentRef, ref)
latestCommentRef = ref
}
@@ -655,7 +665,7 @@ func (p *Parser) parseOptionalWhitespaceCommentNewline(b []byte) (reference, []b
}
func (p *Parser) parseMultilineLiteralString(b []byte) ([]byte, []byte, []byte, error) {
token, rest, err := scanMultilineLiteralString(b)
token, rest, err := scanMultilineLiteralString(b, p.offsetOf(b))
if err != nil {
return nil, nil, nil, err
}
@@ -684,7 +694,7 @@ func (p *Parser) parseMultilineBasicString(b []byte) ([]byte, []byte, []byte, er
// mlb-quotes = 1*2quotation-mark
// mlb-unescaped = wschar / %x21 / %x23-5B / %x5D-7E / non-ascii
// mlb-escaped-nl = escape ws newline *( wschar / newline )
token, escaped, rest, err := scanMultilineBasicString(b)
token, escaped, rest, err := scanMultilineBasicString(b, p.offsetOf(b))
if err != nil {
return nil, nil, nil, err
}
@@ -701,14 +711,15 @@ func (p *Parser) parseMultilineBasicString(b []byte) ([]byte, []byte, []byte, er
// fast path
startIdx := i
endIdx := len(token) - len(`"""`)
tokenBase := p.offsetOf(token)
if !escaped {
str := token[startIdx:endIdx]
verr := characters.Utf8TomlValidAlreadyEscaped(str)
if verr.Zero() {
invalidIdx := characters.Utf8TomlValidAlreadyEscaped(str)
if invalidIdx < 0 {
return token, str, rest, nil
}
return nil, nil, nil, NewParserError(str[verr.Index:verr.Index+verr.Size], "invalid UTF-8")
return nil, nil, nil, NewParserError(str[invalidIdx:invalidIdx+1], tokenBase+startIdx+invalidIdx, "invalid UTF-8")
}
var builder bytes.Buffer
@@ -744,7 +755,7 @@ func (p *Parser) parseMultilineBasicString(b []byte) ([]byte, []byte, []byte, er
i += j
for ; i < len(token)-3; i++ {
c := token[i]
if !(c == '\n' || c == '\r' || c == ' ' || c == '\t') {
if c != '\n' && c != '\r' && c != ' ' && c != '\t' {
i--
break
}
@@ -773,14 +784,14 @@ func (p *Parser) parseMultilineBasicString(b []byte) ([]byte, []byte, []byte, er
case 'e':
builder.WriteByte(0x1B)
case 'u':
x, err := hexToRune(atmost(token[i+1:], 4), 4)
x, err := hexToRune(atmost(token[i+1:], 4), tokenBase+i+1, 4)
if err != nil {
return nil, nil, nil, err
}
builder.WriteRune(x)
i += 4
case 'U':
x, err := hexToRune(atmost(token[i+1:], 8), 8)
x, err := hexToRune(atmost(token[i+1:], 8), tokenBase+i+1, 8)
if err != nil {
return nil, nil, nil, err
}
@@ -788,13 +799,13 @@ func (p *Parser) parseMultilineBasicString(b []byte) ([]byte, []byte, []byte, er
builder.WriteRune(x)
i += 8
default:
return nil, nil, nil, NewParserError(token[i:i+1], "invalid escaped character %#U", c)
return nil, nil, nil, NewParserError(token[i:i+1], tokenBase+i, "invalid escaped character %#U", c)
}
i++
} else {
size := characters.Utf8ValidNext(token[i:])
if size == 0 {
return nil, nil, nil, NewParserError(token[i:i+1], "invalid character %#U", c)
return nil, nil, nil, NewParserError(token[i:i+1], tokenBase+i, "invalid character %#U", c)
}
builder.Write(token[i : i+size])
i += size
@@ -820,7 +831,7 @@ func (p *Parser) parseKey(b []byte) (reference, []byte, error) {
ref := p.builder.Push(Node{
Kind: Key,
Raw: p.Range(raw),
Raw: p.rangeOfToken(raw, b),
Data: key,
})
@@ -836,7 +847,7 @@ func (p *Parser) parseKey(b []byte) (reference, []byte, error) {
p.builder.PushAndChain(Node{
Kind: Key,
Raw: p.Range(raw),
Raw: p.rangeOfToken(raw, b),
Data: key,
})
} else {
@@ -849,12 +860,9 @@ func (p *Parser) parseKey(b []byte) (reference, []byte, error) {
func (p *Parser) parseSimpleKey(b []byte) (raw, key, rest []byte, err error) {
if len(b) == 0 {
return nil, nil, nil, NewParserError(b, "expected key but found none")
return nil, nil, nil, NewParserError(b, p.offsetOf(b), "expected key but found none")
}
// simple-key = quoted-key / unquoted-key
// unquoted-key = 1*( ALPHA / DIGIT / %x2D / %x5F ) ; A-Z / a-z / 0-9 / - / _
// quoted-key = basic-string / literal-string
switch {
case b[0] == '\'':
return p.parseLiteralString(b)
@@ -864,7 +872,7 @@ func (p *Parser) parseSimpleKey(b []byte) (raw, key, rest []byte, err error) {
key, rest = scanUnquotedKey(b)
return key, key, rest, nil
default:
return nil, nil, nil, NewParserError(b[0:1], "invalid character at start of key: %c", b[0])
return nil, nil, nil, NewParserError(b[0:1], p.offsetOf(b), "invalid character at start of key: %c", b[0])
}
}
@@ -884,7 +892,7 @@ func (p *Parser) parseBasicString(b []byte) ([]byte, []byte, []byte, error) {
// escape-seq-char =/ %x74 ; t tab U+0009
// escape-seq-char =/ %x75 4HEXDIG ; uXXXX U+XXXX
// escape-seq-char =/ %x55 8HEXDIG ; UXXXXXXXX U+XXXXXXXX
token, escaped, rest, err := scanBasicString(b)
token, escaped, rest, err := scanBasicString(b, p.offsetOf(b))
if err != nil {
return nil, nil, nil, err
}
@@ -895,13 +903,15 @@ func (p *Parser) parseBasicString(b []byte) ([]byte, []byte, []byte, error) {
// Fast path. If there is no escape sequence, the string should just be
// an UTF-8 encoded string, which is the same as Go. In that case,
// validate the string and return a direct reference to the buffer.
tokenBase := p.offsetOf(token)
if !escaped {
str := token[startIdx:endIdx]
verr := characters.Utf8TomlValidAlreadyEscaped(str)
if verr.Zero() {
invalidIdx := characters.Utf8TomlValidAlreadyEscaped(str)
if invalidIdx < 0 {
return token, str, rest, nil
}
return nil, nil, nil, NewParserError(str[verr.Index:verr.Index+verr.Size], "invalid UTF-8")
return nil, nil, nil, NewParserError(str[invalidIdx:invalidIdx+1], tokenBase+startIdx+invalidIdx, "invalid UTF-8")
}
i := startIdx
@@ -932,7 +942,7 @@ func (p *Parser) parseBasicString(b []byte) ([]byte, []byte, []byte, error) {
case 'e':
builder.WriteByte(0x1B)
case 'u':
x, err := hexToRune(token[i+1:len(token)-1], 4)
x, err := hexToRune(token[i+1:len(token)-1], tokenBase+i+1, 4)
if err != nil {
return nil, nil, nil, err
}
@@ -940,7 +950,7 @@ func (p *Parser) parseBasicString(b []byte) ([]byte, []byte, []byte, error) {
builder.WriteRune(x)
i += 4
case 'U':
x, err := hexToRune(token[i+1:len(token)-1], 8)
x, err := hexToRune(token[i+1:len(token)-1], tokenBase+i+1, 8)
if err != nil {
return nil, nil, nil, err
}
@@ -948,13 +958,13 @@ func (p *Parser) parseBasicString(b []byte) ([]byte, []byte, []byte, error) {
builder.WriteRune(x)
i += 8
default:
return nil, nil, nil, NewParserError(token[i:i+1], "invalid escaped character %#U", c)
return nil, nil, nil, NewParserError(token[i:i+1], tokenBase+i, "invalid escaped character %#U", c)
}
i++
} else {
size := characters.Utf8ValidNext(token[i:])
if size == 0 {
return nil, nil, nil, NewParserError(token[i:i+1], "invalid character %#U", c)
return nil, nil, nil, NewParserError(token[i:i+1], tokenBase+i, "invalid character %#U", c)
}
builder.Write(token[i : i+size])
i += size
@@ -964,15 +974,15 @@ func (p *Parser) parseBasicString(b []byte) ([]byte, []byte, []byte, error) {
return token, builder.Bytes(), rest, nil
}
func hexToRune(b []byte, length int) (rune, error) {
func hexToRune(b []byte, base int, length int) (rune, error) {
if len(b) < length {
return -1, NewParserError(b, "unicode point needs %d character, not %d", length, len(b))
return -1, NewParserError(b, base, "unicode point needs %d character, not %d", length, len(b))
}
b = b[:length]
var r uint32
for i, c := range b {
d := uint32(0)
var d uint32
switch {
case '0' <= c && c <= '9':
d = uint32(c - '0')
@@ -981,13 +991,13 @@ func hexToRune(b []byte, length int) (rune, error) {
case 'A' <= c && c <= 'F':
d = uint32(c - 'A' + 10)
default:
return -1, NewParserError(b[i:i+1], "non-hex character")
return -1, NewParserError(b[i:i+1], base+i, "non-hex character")
}
r = r*16 + d
}
if r > unicode.MaxRune || 0xD800 <= r && r < 0xE000 {
return -1, NewParserError(b, "escape sequence is invalid Unicode code point")
return -1, NewParserError(b, base, "escape sequence is invalid Unicode code point")
}
return rune(r), nil
@@ -1007,23 +1017,23 @@ func (p *Parser) parseIntOrFloatOrDateTime(b []byte) (reference, []byte, error)
switch b[0] {
case 'i':
if !scanFollowsInf(b) {
return invalidReference, nil, NewParserError(atmost(b, 3), "expected 'inf'")
return invalidReference, nil, NewParserError(atmost(b, 3), p.offsetOf(b), "expected 'inf'")
}
return p.builder.Push(Node{
Kind: Float,
Data: b[:3],
Raw: p.Range(b[:3]),
Raw: p.rangeOfToken(b[:3], b[3:]),
}), b[3:], nil
case 'n':
if !scanFollowsNan(b) {
return invalidReference, nil, NewParserError(atmost(b, 3), "expected 'nan'")
return invalidReference, nil, NewParserError(atmost(b, 3), p.offsetOf(b), "expected 'nan'")
}
return p.builder.Push(Node{
Kind: Float,
Data: b[:3],
Raw: p.Range(b[:3]),
Raw: p.rangeOfToken(b[:3], b[3:]),
}), b[3:], nil
case '+', '-':
return p.scanIntOrFloat(b)
@@ -1148,7 +1158,7 @@ func (p *Parser) scanIntOrFloat(b []byte) (reference, []byte, error) {
return p.builder.Push(Node{
Kind: Integer,
Data: b[:i],
Raw: p.Range(b[:i]),
Raw: p.rangeOfToken(b[:i], b[i:]),
}), b[i:], nil
}
@@ -1172,11 +1182,11 @@ func (p *Parser) scanIntOrFloat(b []byte) (reference, []byte, error) {
return p.builder.Push(Node{
Kind: Float,
Data: b[:i+3],
Raw: p.Range(b[:i+3]),
Raw: p.rangeOfToken(b[:i+3], b[i+3:]),
}), b[i+3:], nil
}
return invalidReference, nil, NewParserError(b[i:i+1], "unexpected character 'i' while scanning for a number")
return invalidReference, nil, NewParserError(b[i:i+1], p.offsetOf(b)+i, "unexpected character 'i' while scanning for a number")
}
if c == 'n' {
@@ -1184,18 +1194,18 @@ func (p *Parser) scanIntOrFloat(b []byte) (reference, []byte, error) {
return p.builder.Push(Node{
Kind: Float,
Data: b[:i+3],
Raw: p.Range(b[:i+3]),
Raw: p.rangeOfToken(b[:i+3], b[i+3:]),
}), b[i+3:], nil
}
return invalidReference, nil, NewParserError(b[i:i+1], "unexpected character 'n' while scanning for a number")
return invalidReference, nil, NewParserError(b[i:i+1], p.offsetOf(b)+i, "unexpected character 'n' while scanning for a number")
}
break
}
if i == 0 {
return invalidReference, b, NewParserError(b, "incomplete number")
return invalidReference, b, NewParserError(b, p.offsetOf(b), "incomplete number")
}
kind := Integer
@@ -1207,7 +1217,7 @@ func (p *Parser) scanIntOrFloat(b []byte) (reference, []byte, error) {
return p.builder.Push(Node{
Kind: kind,
Data: b[:i],
Raw: p.Range(b[:i]),
Raw: p.rangeOfToken(b[:i], b[i:]),
}), b[i:], nil
}
@@ -1232,13 +1242,13 @@ func isValidBinaryRune(r byte) bool {
return r == '0' || r == '1' || r == '_'
}
func expect(x byte, b []byte) ([]byte, error) {
func expect(x byte, b []byte, base int) ([]byte, error) {
if len(b) == 0 {
return nil, NewParserError(b, "expected character %c but the document ended here", x)
return nil, NewParserError(b, base, "expected character %c but the document ended here", x)
}
if b[0] != x {
return nil, NewParserError(b[0:1], "expected character %c", x)
return nil, NewParserError(b[0:1], base, "expected character %c", x)
}
return b[1:], nil
+168 -9
View File
@@ -1,6 +1,7 @@
package unstable
import (
"errors"
"fmt"
"strconv"
"strings"
@@ -358,7 +359,7 @@ func BenchmarkParseBasicStringWithUnicode(b *testing.B) {
b.SetBytes(int64(len(input)))
for i := 0; i < b.N; i++ {
p.parseBasicString(input)
_, _, _, _ = p.parseBasicString(input)
}
})
b.Run("8", func(b *testing.B) {
@@ -367,7 +368,7 @@ func BenchmarkParseBasicStringWithUnicode(b *testing.B) {
b.SetBytes(int64(len(input)))
for i := 0; i < b.N; i++ {
p.parseBasicString(input)
_, _, _, _ = p.parseBasicString(input)
}
})
}
@@ -383,7 +384,7 @@ func BenchmarkParseBasicStringsEasy(b *testing.B) {
b.SetBytes(int64(len(input)))
for i := 0; i < b.N; i++ {
p.parseBasicString(input)
_, _, _, _ = p.parseBasicString(input)
}
})
}
@@ -539,7 +540,7 @@ key5 = [ # Next to start of inline array.
// ---
// 6:1->6:22 (105->126) | Comment [# Above simple value.]
// ---
// 1:1->1:1 (0->0) | KeyValue []
// 7:1->7:14 (127->140) | KeyValue []
// 7:7->7:14 (133->140) | String [value]
// 7:1->7:4 (127->130) | Key [key]
// 7:15->7:38 (141->164) | Comment [# Next to simple value.]
@@ -552,12 +553,12 @@ key5 = [ # Next to start of inline array.
// ---
// 14:1->14:22 (252->273) | Comment [# Above inline table.]
// ---
// 1:1->1:1 (0->0) | KeyValue []
// 15:1->15:50 (274->323) | KeyValue []
// 15:8->15:9 (281->282) | InlineTable []
// 1:1->1:1 (0->0) | KeyValue []
// 15:10->15:23 (283->296) | KeyValue []
// 15:18->15:23 (291->296) | String [Tom]
// 15:10->15:15 (283->288) | Key [first]
// 1:1->1:1 (0->0) | KeyValue []
// 15:25->15:48 (298->321) | KeyValue []
// 15:32->15:48 (305->321) | String [Preston-Werner]
// 15:25->15:29 (298->302) | Key [last]
// 15:1->15:5 (274->278) | Key [name]
@@ -567,7 +568,7 @@ key5 = [ # Next to start of inline array.
// ---
// 18:1->18:15 (371->385) | Comment [# Above array.]
// ---
// 1:1->1:1 (0->0) | KeyValue []
// 19:1->19:20 (386->405) | KeyValue []
// 1:1->1:1 (0->0) | Array []
// 19:11->19:12 (396->397) | Integer [1]
// 19:14->19:15 (399->400) | Integer [2]
@@ -579,7 +580,7 @@ key5 = [ # Next to start of inline array.
// ---
// 22:1->22:26 (448->473) | Comment [# Above multi-line array.]
// ---
// 1:1->1:1 (0->0) | KeyValue []
// 23:1->31:2 (474->694) | KeyValue []
// 1:1->1:1 (0->0) | Array []
// 23:10->23:42 (483->515) | Comment [# Next to start of inline array.]
// 24:3->24:38 (518->553) | Comment [# Second line before array content.]
@@ -605,6 +606,164 @@ key5 = [ # Next to start of inline array.
// 36:1->36:21 (804->824) | Comment [# After array table.]
}
func TestIterator_IsLast(t *testing.T) {
// Test IsLast on an iterator with multiple elements using public Parser API
doc := `array = [1, 2, 3]`
p := Parser{}
p.Reset([]byte(doc))
p.NextExpression()
e := p.Expression()
arr := e.Value() // The array node
it := arr.Children()
count := 0
lastCount := 0
for it.Next() {
count++
if it.IsLast() {
lastCount++
}
}
assert.Equal(t, 3, count)
assert.Equal(t, 1, lastCount)
}
func TestNodeChaining(t *testing.T) {
// Test that sibling nodes are correctly chained via Next()
// This exercises the internal PushAndChain functionality through public APIs
doc := `a.b.c = 1`
p := Parser{}
p.Reset([]byte(doc))
p.NextExpression()
e := p.Expression()
// KeyValue has children: value, then key parts (a, b, c)
keyIt := e.Key()
// Collect all key parts by following the iterator
var keys []string
for keyIt.Next() {
keys = append(keys, string(keyIt.Node().Data))
}
assert.Equal(t, []string{"a", "b", "c"}, keys)
}
func TestMultipleExpressions(t *testing.T) {
// Test parsing multiple top-level expressions
// This exercises root iteration through public APIs
doc := `
key1 = "value1"
key2 = "value2"
key3 = "value3"
`
p := Parser{}
p.Reset([]byte(doc))
var keys []string
for p.NextExpression() {
e := p.Expression()
keyIt := e.Key()
keyIt.Next()
keys = append(keys, string(keyIt.Node().Data))
}
assert.NoError(t, p.Error())
assert.Equal(t, []string{"key1", "key2", "key3"}, keys)
}
func TestErrorOffsetAfterComment(t *testing.T) {
input := []byte("# comment\n= \"value\"")
p := Parser{}
p.Reset(input)
for p.NextExpression() {
}
err := p.Error()
if err == nil {
t.Fatal("expected an error")
}
var perr *ParserError
if !errors.As(err, &perr) {
t.Fatalf("expected ParserError, got %T", err)
}
if perr.Offset != 10 {
t.Errorf("offset: got %d, want 10", perr.Offset)
}
shape := p.Shape(Range{Offset: uint32(perr.Offset), Length: uint32(len(perr.Highlight))})
if shape.Start.Line != 2 || shape.Start.Column != 1 {
t.Errorf("position: got %d:%d, want 2:1", shape.Start.Line, shape.Start.Column)
}
}
func TestErrorHighlightPositions(t *testing.T) {
examples := []struct {
desc string
input string
wantLine int
wantColumn int
}{
{
desc: "invalid key start after comment",
input: "# comment\n= \"value\"",
wantLine: 2,
wantColumn: 1,
},
{
desc: "invalid key start on first line",
input: "= \"value\"",
wantLine: 1,
wantColumn: 1,
},
{
desc: "invalid key after multiple comments",
input: "# comment 1\n# comment 2\n= \"value\"",
wantLine: 3,
wantColumn: 1,
},
{
desc: "invalid key after valid key-value",
input: "a = 1\n= \"value\"",
wantLine: 2,
wantColumn: 1,
},
{
desc: "invalid key after whitespace on line",
input: "a = 1\n = \"value\"",
wantLine: 2,
wantColumn: 3,
},
}
for _, e := range examples {
t.Run(e.desc, func(t *testing.T) {
p := Parser{}
p.Reset([]byte(e.input))
for p.NextExpression() {
}
err := p.Error()
if err == nil {
t.Fatal("expected an error")
}
var perr *ParserError
if !errors.As(err, &perr) {
t.Fatalf("expected ParserError, got %T", err)
}
shape := p.Shape(Range{Offset: uint32(perr.Offset), Length: uint32(len(perr.Highlight))})
if shape.Start.Line != e.wantLine {
t.Errorf("line: got %d, want %d", shape.Start.Line, e.wantLine)
}
if shape.Start.Column != e.wantColumn {
t.Errorf("column: got %d, want %d", shape.Start.Column, e.wantColumn)
}
})
}
}
func ExampleParser() {
doc := `
hello = "world"
+27 -72
View File
@@ -47,48 +47,31 @@ func isUnquotedKeyChar(r byte) bool {
return (r >= 'A' && r <= 'Z') || (r >= 'a' && r <= 'z') || (r >= '0' && r <= '9') || r == '-' || r == '_'
}
func scanLiteralString(b []byte) ([]byte, []byte, error) {
// literal-string = apostrophe *literal-char apostrophe
// apostrophe = %x27 ; ' apostrophe
// literal-char = %x09 / %x20-26 / %x28-7E / non-ascii
func scanLiteralString(b []byte, base int) ([]byte, []byte, error) {
for i := 1; i < len(b); {
switch b[i] {
case '\'':
return b[:i+1], b[i+1:], nil
case '\n', '\r':
return nil, nil, NewParserError(b[i:i+1], "literal strings cannot have new lines")
return nil, nil, NewParserError(b[i:i+1], base+i, "literal strings cannot have new lines")
}
size := characters.Utf8ValidNext(b[i:])
if size == 0 {
return nil, nil, NewParserError(b[i:i+1], "invalid character")
return nil, nil, NewParserError(b[i:i+1], base+i, "invalid character")
}
i += size
}
return nil, nil, NewParserError(b[len(b):], "unterminated literal string")
return nil, nil, NewParserError(b[len(b):], base+len(b), "unterminated literal string")
}
func scanMultilineLiteralString(b []byte) ([]byte, []byte, error) {
// ml-literal-string = ml-literal-string-delim [ newline ] ml-literal-body
// ml-literal-string-delim
// ml-literal-string-delim = 3apostrophe
// ml-literal-body = *mll-content *( mll-quotes 1*mll-content ) [ mll-quotes ]
//
// mll-content = mll-char / newline
// mll-char = %x09 / %x20-26 / %x28-7E / non-ascii
// mll-quotes = 1*2apostrophe
func scanMultilineLiteralString(b []byte, base int) ([]byte, []byte, error) {
for i := 3; i < len(b); {
switch b[i] {
case '\'':
if scanFollowsMultilineLiteralStringDelimiter(b[i:]) {
i += 3
// At that point we found 3 apostrophe, and i is the
// index of the byte after the third one. The scanner
// needs to be eager, because there can be an extra 2
// apostrophe that can be accepted at the end of the
// string.
if i >= len(b) || b[i] != '\'' {
return b[:i], b[i:], nil
}
@@ -100,39 +83,39 @@ func scanMultilineLiteralString(b []byte) ([]byte, []byte, error) {
i++
if i < len(b) && b[i] == '\'' {
return nil, nil, NewParserError(b[i-3:i+1], "''' not allowed in multiline literal string")
return nil, nil, NewParserError(b[i-3:i+1], base+i-3, "''' not allowed in multiline literal string")
}
return b[:i], b[i:], nil
}
case '\r':
if len(b) < i+2 {
return nil, nil, NewParserError(b[len(b):], `need a \n after \r`)
return nil, nil, NewParserError(b[len(b):], base+len(b), `need a \n after \r`)
}
if b[i+1] != '\n' {
return nil, nil, NewParserError(b[i:i+2], `need a \n after \r`)
return nil, nil, NewParserError(b[i:i+2], base+i, `need a \n after \r`)
}
i += 2 // skip the \n
i += 2
continue
}
size := characters.Utf8ValidNext(b[i:])
if size == 0 {
return nil, nil, NewParserError(b[i:i+1], "invalid character")
return nil, nil, NewParserError(b[i:i+1], base+i, "invalid character")
}
i += size
}
return nil, nil, NewParserError(b[len(b):], `multiline literal string not terminated by '''`)
return nil, nil, NewParserError(b[len(b):], base+len(b), `multiline literal string not terminated by '''`)
}
func scanWindowsNewline(b []byte) ([]byte, []byte, error) {
func scanWindowsNewline(b []byte, base int) ([]byte, []byte, error) {
const lenCRLF = 2
if len(b) < lenCRLF {
return nil, nil, NewParserError(b, "windows new line expected")
return nil, nil, NewParserError(b, base, "windows new line expected")
}
if b[1] != '\n' {
return nil, nil, NewParserError(b, `windows new line should be \r\n`)
return nil, nil, NewParserError(b, base, `windows new line should be \r\n`)
}
return b[:lenCRLF], b[lenCRLF:], nil
@@ -151,13 +134,7 @@ func scanWhitespace(b []byte) ([]byte, []byte) {
return b, b[len(b):]
}
func scanComment(b []byte) ([]byte, []byte, error) {
// comment-start-symbol = %x23 ; #
// non-ascii = %x80-D7FF / %xE000-10FFFF
// non-eol = %x09 / %x20-7F / non-ascii
//
// comment = comment-start-symbol *non-eol
func scanComment(b []byte, base int) ([]byte, []byte, error) {
for i := 1; i < len(b); {
if b[i] == '\n' {
return b[:i], b[i:], nil
@@ -166,11 +143,11 @@ func scanComment(b []byte) ([]byte, []byte, error) {
if i+1 < len(b) && b[i+1] == '\n' {
return b[:i+1], b[i+1:], nil
}
return nil, nil, NewParserError(b[i:i+1], "invalid character in comment")
return nil, nil, NewParserError(b[i:i+1], base+i, "invalid character in comment")
}
size := characters.Utf8ValidNext(b[i:])
if size == 0 {
return nil, nil, NewParserError(b[i:i+1], "invalid character in comment")
return nil, nil, NewParserError(b[i:i+1], base+i, "invalid character in comment")
}
i += size
@@ -179,12 +156,7 @@ func scanComment(b []byte) ([]byte, []byte, error) {
return b, b[len(b):], nil
}
func scanBasicString(b []byte) ([]byte, bool, []byte, error) {
// basic-string = quotation-mark *basic-char quotation-mark
// quotation-mark = %x22 ; "
// basic-char = basic-unescaped / escaped
// basic-unescaped = wschar / %x21 / %x23-5B / %x5D-7E / non-ascii
// escaped = escape escape-seq-char
func scanBasicString(b []byte, base int) ([]byte, bool, []byte, error) {
escaped := false
i := 1
@@ -193,31 +165,20 @@ func scanBasicString(b []byte) ([]byte, bool, []byte, error) {
case '"':
return b[:i+1], escaped, b[i+1:], nil
case '\n', '\r':
return nil, escaped, nil, NewParserError(b[i:i+1], "basic strings cannot have new lines")
return nil, escaped, nil, NewParserError(b[i:i+1], base+i, "basic strings cannot have new lines")
case '\\':
if len(b) < i+2 {
return nil, escaped, nil, NewParserError(b[i:i+1], "need a character after \\")
return nil, escaped, nil, NewParserError(b[i:i+1], base+i, "need a character after \\")
}
escaped = true
i++ // skip the next character
}
}
return nil, escaped, nil, NewParserError(b[len(b):], `basic string not terminated by "`)
return nil, escaped, nil, NewParserError(b[len(b):], base+len(b), `basic string not terminated by "`)
}
func scanMultilineBasicString(b []byte) ([]byte, bool, []byte, error) {
// ml-basic-string = ml-basic-string-delim [ newline ] ml-basic-body
// ml-basic-string-delim
// ml-basic-string-delim = 3quotation-mark
// ml-basic-body = *mlb-content *( mlb-quotes 1*mlb-content ) [ mlb-quotes ]
//
// mlb-content = mlb-char / newline / mlb-escaped-nl
// mlb-char = mlb-unescaped / escaped
// mlb-quotes = 1*2quotation-mark
// mlb-unescaped = wschar / %x21 / %x23-5B / %x5D-7E / non-ascii
// mlb-escaped-nl = escape ws newline *( wschar / newline )
func scanMultilineBasicString(b []byte, base int) ([]byte, bool, []byte, error) {
escaped := false
i := 3
@@ -227,12 +188,6 @@ func scanMultilineBasicString(b []byte) ([]byte, bool, []byte, error) {
if scanFollowsMultilineBasicStringDelimiter(b[i:]) {
i += 3
// At that point we found 3 apostrophe, and i is the
// index of the byte after the third one. The scanner
// needs to be eager, because there can be an extra 2
// apostrophe that can be accepted at the end of the
// string.
if i >= len(b) || b[i] != '"' {
return b[:i], escaped, b[i:], nil
}
@@ -244,27 +199,27 @@ func scanMultilineBasicString(b []byte) ([]byte, bool, []byte, error) {
i++
if i < len(b) && b[i] == '"' {
return nil, escaped, nil, NewParserError(b[i-3:i+1], `""" not allowed in multiline basic string`)
return nil, escaped, nil, NewParserError(b[i-3:i+1], base+i-3, `""" not allowed in multiline basic string`)
}
return b[:i], escaped, b[i:], nil
}
case '\\':
if len(b) < i+2 {
return nil, escaped, nil, NewParserError(b[len(b):], "need a character after \\")
return nil, escaped, nil, NewParserError(b[len(b):], base+len(b), "need a character after \\")
}
escaped = true
i++ // skip the next character
case '\r':
if len(b) < i+2 {
return nil, escaped, nil, NewParserError(b[len(b):], `need a \n after \r`)
return nil, escaped, nil, NewParserError(b[len(b):], base+len(b), `need a \n after \r`)
}
if b[i+1] != '\n' {
return nil, escaped, nil, NewParserError(b[i:i+2], `need a \n after \r`)
return nil, escaped, nil, NewParserError(b[i:i+2], base+i, `need a \n after \r`)
}
i++ // skip the \n
}
}
return nil, escaped, nil, NewParserError(b[len(b):], `multiline basic string not terminated by """`)
return nil, escaped, nil, NewParserError(b[len(b):], base+len(b), `multiline basic string not terminated by """`)
}
+28 -3
View File
@@ -1,7 +1,32 @@
package unstable
// The Unmarshaler interface may be implemented by types to customize their
// behavior when being unmarshaled from a TOML document.
// Unmarshaler is implemented by types that can unmarshal a TOML
// description of themselves. The input is a valid TOML document
// containing the relevant portion of the parsed document.
//
// For tables (including split tables defined in multiple places),
// the data contains the raw key-value bytes from the original document
// with adjusted table headers to be relative to the unmarshaling target.
type Unmarshaler interface {
UnmarshalTOML(value *Node) error
UnmarshalTOML(data []byte) error
}
// RawMessage is a raw encoded TOML value. It implements Unmarshaler
// and can be used to delay TOML decoding or capture raw content.
//
// Example usage:
//
// type Config struct {
// Plugin RawMessage `toml:"plugin"`
// }
//
// var cfg Config
// toml.NewDecoder(r).EnableUnmarshalerInterface().Decode(&cfg)
// // cfg.Plugin now contains the raw TOML bytes for [plugin]
type RawMessage []byte
// UnmarshalTOML implements Unmarshaler.
func (m *RawMessage) UnmarshalTOML(data []byte) error {
*m = append((*m)[0:0], data...)
return nil
}