Compare commits

...

163 Commits

Author SHA1 Message Date
Claude fa98989475 fix: resolve lint issues and improve test coverage for TOML v1.1.0
- Fix dupl lint: add nolint:dupl to parseInlineTable and parseValArray
  which have intentionally similar loop structures
- Fix gci lint: correct alignment in multiline basic string test entries
- Add unmarshaler tests for new TOML v1.1.0 features exercising public
  APIs: multiline inline tables with comments, trailing commas, leading
  commas, error cases (comma at start, missing separator, double comma,
  incomplete table), escape sequences, and type mismatch errors
- Add parser test for inline table comment handling with KeepComments
- Coverage increases from 97.37% to 97.44% vs v2 base

https://claude.ai/code/session_01RdiWykFQdmwkQ2nbLwJwwP
2026-03-29 17:37:29 +00:00
João Fernandes 189bf9820b test: parsing inline table 2026-03-03 14:03:57 +00:00
João Fernandes 8455b10edd test: regen toml-test tests targetting TOML v1.1.0 2026-02-11 11:49:40 +00:00
João Fernandes 6de639d0ae docs: update README 2026-02-11 11:15:44 +00:00
João Fernandes 517ceb4eb8 feat: allow newlines and trailing commas in inline tables
TOML v1.1.0 relaxes inline table syntax to allow newlines, comments,
and trailing commas, matching the existing behavior of arrays.
2026-02-11 11:15:33 +00:00
João Fernandes dd7970eb93 feat: make seconds optional in datetime and time values
TOML v1.1.0 allows times to be specified as HH:MM without the seconds
component (previously HH:MM:SS was required). This applies to local
times, local datetimes, and offset datetimes.
2026-02-11 11:15:16 +00:00
João Fernandes 3405e8a1d9 test: \e escape character 2026-02-11 11:14:59 +00:00
João Fernandes 5794be6251 feat: add \xHH escape sequence support in basic strings
TOML v1.1.0 introduces the \xHH escape notation for basic strings,
allowing two-digit hex escapes for Unicode code points U+0000 to
U+00FF.

We keep emitting \u00XX for backwards compatibility.
2026-02-11 11:14:23 +00:00
Thomas Pelletier 2edc61f171 Fix panic when unmarshaling datetime values to incompatible types (#1028) (#1029)
Return a type mismatch error instead of panicking when datetime values
(DateTime, LocalDate, LocalTime, LocalDateTime) are unmarshaled into
incompatible Go types. This makes the decoder safer for processing
untrusted TOML input.

https://claude.ai/code/session_011jwvtDS5M2KncLrqJpgMr5

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-23 22:04:40 -05:00
Thomas Pelletier 4a1b05ca08 UnmarshalText fallbacks to struct unmarshaling for tables and arrays (#1026)
When a type implements encoding.TextUnmarshaler, the unmarshaler now
skips calling UnmarshalText for Array and InlineTable TOML values.
This allows types to support both:
- Simple string values via UnmarshalText
- Structured table values via field-by-field unmarshaling

Previously, UnmarshalText was called unconditionally, which prevented
proper struct unmarshaling when the TOML value was a table or array
of tables.

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-09 13:46:38 -05:00
Thomas Pelletier 003aa0993b Fix nil pointer map values not being marshaled (#1025)
When marshaling a map with nil pointer values, the keys were being
silently dropped, breaking round-trip fidelity. For example:

    map[string]*struct{}{"foo": nil}

Would produce an empty TOML document instead of "[foo]".

This change converts nil pointer values in maps to their zero values
(consistent with how nil pointers in slices are handled), allowing the
keys to be preserved as empty tables.

Nil interface values (map[string]any{"foo": nil}) are still skipped
since there's no type information to derive a zero value.

Fixes #975

Also, pin golangci-lint version to v2.8.0 in CI and document in AGENTS.md

- Explicitly set golangci-lint version in lint.yml to ensure consistent
  behavior across CI runs
- Update AGENTS.md with instructions to use the same linter version locally

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-09 11:08:31 -05:00
dependabot[bot] 84d730b6c4 build(deps): bump golangci/golangci-lint-action from 8 to 9 (#1022) 2026-01-05 21:23:56 -05:00
dependabot[bot] 97bd897177 build(deps): bump actions/setup-go from 5 to 6 (#1023) 2026-01-05 21:23:35 -05:00
dependabot[bot] 7924b1816f build(deps): bump actions/checkout from 5 to 6 (#1024) 2026-01-05 21:23:15 -05:00
Thomas Pelletier 2a07b6d9db Update to Go 1.25 (#1018)
Update CI workflows to test against Go 1.24 and 1.25, and use Go 1.25 for
coverage and release builds.

## Benchstat Report: Go 1.24 vs Go 1.25

Benchmark comparison between Go 1.24.7 and Go 1.25.1 (10 runs each):

### Execution Time (sec/op)

| Benchmark | Go 1.24 | Go 1.25 | Delta |
|-----------|---------|---------|-------|
| UnmarshalDataset/config | 26.25ms | 26.00ms | ~ (p=0.280) |
| UnmarshalDataset/canada | 88.71ms | 84.94ms | **-4.26%**  |
| UnmarshalDataset/citm_catalog | 33.71ms | 34.06ms | ~ (p=0.684) |
| UnmarshalDataset/twitter | 17.19ms | 17.33ms | ~ (p=0.971) |
| UnmarshalDataset/code | 107.4ms | 108.1ms | ~ (p=0.393) |
| UnmarshalDataset/example | 237.9µs | 251.3µs | +5.64% |
| Unmarshal/SimpleDocument/struct | 872.3ns | 848.9ns | ~ (p=0.165) |
| Unmarshal/SimpleDocument/map | 1.191µs | 1.278µs | +7.31% |
| Unmarshal/ReferenceFile/struct | 57.14µs | 57.95µs | ~ (p=0.089) |
| Unmarshal/ReferenceFile/map | 87.89µs | 92.88µs | +5.69% |
| Unmarshal/HugoFrontMatter | 16.06µs | 15.95µs | ~ (p=0.529) |
| Marshal/SimpleDocument/struct | 536.5ns | 563.5ns | +5.03% |
| Marshal/SimpleDocument/map | 651.0ns | 675.1ns | +3.72% |
| Marshal/ReferenceFile/struct | 44.63µs | 50.84µs | +13.91% |
| Marshal/ReferenceFile/map | 51.58µs | 57.06µs | +10.61% |
| Marshal/HugoFrontMatter | 10.04µs | 10.57µs | +5.27% |
| **geomean** | 140.6µs | 145.1µs | +3.18% |

### Summary

- Notable improvement: UnmarshalDataset/canada shows a 4.26% speedup
- Memory allocation and allocation counts remain identical
- Some marshal operations show slight slowdowns (likely Go runtime changes)

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-04 13:59:18 -05:00
Thomas Pelletier 692b98560b Support custom IsZero() methods with omitzero tag (#1020)
The omitzero tag now respects custom IsZero() methods on types,
similar to how encoding/json handles this. Previously, only
reflect.Value.IsZero() was used, which ignores user-defined
implementations.

Fixes #1003

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-04 13:58:47 -05:00
Thomas Pelletier 99cd40b175 Reject leap seconds to prevent year overflow (#1019)
Go's time.Date() normalizes leap seconds (second=60) by adding 1 minute.
When parsing the maximum valid TOML date 9999-12-31 23:59:60z, this causes
the year to overflow to 10000, which exceeds the valid TOML year range
(0000-9999) and breaks round-trip serialization.

The fix rejects leap seconds (second > 59) during parsing. This is
consistent with the resolution of issue #913 which determined that
emitting an error is less surprising than silently normalizing leap
seconds.

Fixes #1015

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-04 13:40:19 -05:00
Thomas Pelletier 3aaf147e3e Remove unsafe package usage (#1021)
Removes all unsafe operations from go-toml, making the codebase
fully safe Go code. The internal/danger package that contained
unsafe operations has been deleted.

Changes:
- Replace pointer-based node navigation with index-based navigation
- Node.next and Node.child now store absolute indices into the
  backing nodes slice instead of relative offsets
- Add nodes pointer to Node and Iterator for safe navigation
- Replace danger.TypeID with reflect.Type for cache keys
- Delete internal/danger package entirely

Performance overhead is under 10% compared to the unsafe version,
which is acceptable for the safety and maintainability benefits.

[Cursor][claude-sonnet-4-20250514]
2026-01-04 13:16:47 -05:00
Nathan Baulch a675c6b3e2 Upgrade to golangci-lint v2 (#1008) 2026-01-04 09:54:29 -05:00
Thomas Pelletier 9702fae9b8 Add AGENTS.md for AI agent contribution guidelines (#1017)
This file provides a concise summary of the contribution guidelines
from CONTRIBUTING.md, specifically tailored for AI agents working on
the codebase. It covers testing requirements, backward compatibility,
performance considerations, and code style expectations.

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-03 21:41:34 -05:00
Alexander Hecke 3cf1eb2312 improve Unmarshaling documentation (#1016) 2026-01-03 21:12:35 -05:00
Nathan Baulch 2af3554f90 Update toml-test to v1.6.0 (#1007) 2026-01-03 20:45:06 -05:00
dependabot[bot] 180c6ba2ba build(deps): bump actions/setup-go from 5 to 6 (#1002)
Bumps [actions/setup-go](https://github.com/actions/setup-go) from 5 to 6.
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](https://github.com/actions/setup-go/compare/v5...v6)

---
updated-dependencies:
- dependency-name: actions/setup-go
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-01-03 20:43:53 -05:00
dependabot[bot] dafc4173ef build(deps): bump github/codeql-action from 3 to 4 (#1006)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 3 to 4.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/v3...v4)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-version: '4'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-01-03 20:43:43 -05:00
dependabot[bot] f1a83be671 build(deps): bump actions/upload-artifact from 4 to 6 (#1011)
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 4 to 6.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v4...v6)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-01-03 20:43:33 -05:00
dependabot[bot] 5aeb70b3f0 build(deps): bump actions/checkout from 5 to 6 (#1010)
Bumps [actions/checkout](https://github.com/actions/checkout) from 5 to 6.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v5...v6)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-01-03 20:43:20 -05:00
W. Michael Petullo 8384a5683c Use constant format strings with Printf-like functions (#1013)
Recent versions of Go object to the use of non-constant variables a
format strings. This commit fixes errors like this:

cli.go:26:47: non-constant format string in call to fmt.Fprintf

Signed-off-by: W. Michael Petullo <mike@flyn.org>
2026-01-03 20:42:58 -05:00
Étienne BERSAC 4369957cb4 Unwrap strict errors (#1012) 2025-12-21 16:20:24 +01:00
dependabot[bot] a0e8464967 build(deps): bump actions/checkout from 4 to 5 (#1001)
Bumps [actions/checkout](https://github.com/actions/checkout) from 4 to 5.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-08-26 09:53:08 +02:00
Nathan Baulch c57d0d559f Add omitzero tag support (#998) 2025-08-25 08:06:48 +02:00
Thomas Pelletier 644602b845 Script to test all versions of go (#1000) 2025-08-24 12:40:29 +02:00
Nathan Baulch 36df8eef6e General cleanup (#999) 2025-08-24 12:18:46 +02:00
Thomas Pelletier 18a2148713 Handle array table into an empty slice (#997)
Fix #995
2025-08-21 12:05:41 +02:00
Thomas Pelletier bc9958322f Add missing UnmarshalTOML call (#996)
Fixes #994.
2025-08-21 10:39:23 +02:00
Dustin Spicuzza 6d56ac8027 marshal: don't escape quotes unnecessarily (#991)
Only 3 consecutive quotation marks need to be quoted. We choose to quote
all quotation marks in a sequence if there are 3 or more consecutive
present.

Fixes #990

---------

Co-authored-by: Thomas Pelletier <thomas@pelletier.dev>
2025-08-21 08:19:16 +02:00
dependabot[bot] 098464b61b build(deps): bump actions/checkout from 4 to 5 (#993)
Bumps [actions/checkout](https://github.com/actions/checkout) from 4 to 5.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-08-21 08:10:55 +02:00
Oleksandr Redko 85e2448ce5 refactor: Simplify t.Fatalf (#984) 2025-05-10 15:14:34 -04:00
Thomas Pelletier ee07c9203b Update to go 1.24 (#982) 2025-04-07 07:11:38 -04:00
Alex Mikitik 014204cfb7 Replace stretchr/testify with an internal test suite (#981)
As recommended, an `internal/assert` package was added with a reduced set of assertions. All tests were then refactored to use the internal assertions. When more complex assertions were used, they have been rewritten using logic and the simplified assertions.

Fancy formatting for failures was omitted. The `internal/assert/assertions.diff` function could be overwritten for better formatting. That is where diff libraries are used in other test suites.

Refs: #872

Co-authored-by: Alex Mikitik <alex.mikitik@oracle.com>
2025-04-07 06:36:37 -04:00
Oleksandr Redko 923b2ab478 Fix typos in comments and tests (#972) 2024-11-16 11:30:13 -05:00
Thomas Pelletier af236b689f Fix goreleaser deprecated attribute name (#964)
https://goreleaser.com/deprecations/#snapshotname_template
2024-08-23 13:56:48 -04:00
Thomas Pelletier b730b2be5d Bump testing to go 1.23 (#961) 2024-08-17 16:26:05 -04:00
vito a437caafe5 Fix reflect.Pointer backward compatibility (#956) 2024-08-17 16:07:56 -04:00
guoguangwu be6c57be30 Fix readme typo(#951) 2024-08-17 15:56:40 -04:00
Daniel Weiße d55304782e Allow int, uint, and floats as map keys (#958)
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2024-08-17 15:44:21 -04:00
Daniel Weiße 0977c05dd5 Update goreleaser action to v6 and set goreleaser binary to v2 (#959)
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2024-08-17 15:40:55 -04:00
Daniel Martí bccd6e48f4 allocate unstable.Parser as part of decoder (#953)
This way, calls to Unmarshal or Decoder.Decode allocate once
at the start rather than twice.

                                │    old     │               new                │
                                │ allocs/op  │ allocs/op   vs base              │
    Unmarshal/HugoFrontMatter-8   141.0 ± 0%   140.0 ± 0%  -0.71% (p=0.002 n=6)
2024-05-24 14:49:06 -04:00
Daniel Martí 9b890cf9c5 go.mod: bump minimum and language to 1.21 (#949)
* go.mod: bump minimum and language to 1.21

CI only tests Go 1.21 and 1.22, and older versions of Go are no longer
getting any bug or security fixes, so advertise that we only support
Go 1.21 or later via go.mod.

While here, ensure the module is tidy and resolve deprecation warnings,
and remove now-unnecessary Go version build tags.

* replace sort.Slice with slices.SortFunc

The latter is more efficient, and allocates less, since sort.Slice
needs to go through sort.Interface which causes allocations.

    goos: linux
    goarch: amd64
    pkg: github.com/pelletier/go-toml/v2/benchmark
    cpu: AMD Ryzen 7 PRO 5850U with Radeon Graphics
                              │     old     │                new                 │
                              │   sec/op    │   sec/op     vs base               │
    Marshal/HugoFrontMatter-8   7.612µ ± 1%   6.730µ ± 1%  -11.59% (p=0.002 n=6)

                              │     old      │                 new                 │
                              │     B/s      │     B/s       vs base               │
    Marshal/HugoFrontMatter-8   65.52Mi ± 1%   74.11Mi ± 1%  +13.11% (p=0.002 n=6)

                              │     old      │                new                 │
                              │     B/op     │     B/op      vs base              │
    Marshal/HugoFrontMatter-8   5.672Ki ± 0%   5.266Ki ± 0%  -7.16% (p=0.002 n=6)

                              │    old     │                new                │
                              │ allocs/op  │ allocs/op   vs base               │
    Marshal/HugoFrontMatter-8   85.00 ± 0%   73.00 ± 0%  -14.12% (p=0.002 n=6)
2024-05-24 10:58:39 -04:00
大可 a3d5a0bb53 fix: sync pool race condition (#947) 2024-04-29 06:02:54 -04:00
Daniel Weiße d00d2cca6e Fix indentation of custom type arrays (#944)
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2024-04-12 10:42:12 -04:00
dependabot[bot] 86608d7fca build(deps): bump github/codeql-action from 2 to 3 (#919)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2 to 3.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/v2...v3)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-03-19 13:24:46 -04:00
dependabot[bot] 4a1877957a build(deps): bump actions/setup-go from 4 to 5 (#916)
Bumps [actions/setup-go](https://github.com/actions/setup-go) from 4 to 5.
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](https://github.com/actions/setup-go/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/setup-go
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-03-19 13:24:37 -04:00
dependabot[bot] 3021d6d033 build(deps): bump actions/upload-artifact from 3 to 4 (#920)
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 3 to 4.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-03-19 13:24:24 -04:00
Thomas Pelletier 32788f26f8 Update release instructions (#941) 2024-03-19 12:47:39 -04:00
rszyma 8ed6d131eb Decode: unstable/Unmarshal interface (#940)
Co-authored-by: Pavlos Karakalidis <pkarakal@pkarakal.com>
Co-authored-by: Thomas Pelletier <thomas@pelletier.codes>
2024-03-19 12:33:12 -04:00
dependabot[bot] 7dad87762a build(deps): bump github.com/stretchr/testify from 1.8.4 to 1.9.0 (#936)
Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.8.4 to 1.9.0.
- [Release notes](https://github.com/stretchr/testify/releases)
- [Commits](https://github.com/stretchr/testify/compare/v1.8.4...v1.9.0)

---
updated-dependencies:
- dependency-name: github.com/stretchr/testify
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-03-12 09:06:45 -04:00
Thomas Pelletier 2b69615b5d Go 1.22 support (#935) 2024-02-27 15:30:13 -05:00
Thomas Pelletier 06fb30bf2e Decode: fix reuse of slice for array tables (#934)
When decoding into a non-empty slice, it needs to be emptied so that only the
tables contained in the document are present in the resulting value.

Arrays are not impacted because their unmarshal offset is tracked separately.

Fixes #931
2024-02-27 15:28:49 -05:00
Thomas Pelletier 2e087bdf5f Run tests on macos/M1 (#929)
https://github.blog/changelog/2024-01-30-github-actions-introducing-the-new-m1-macos-runner-available-to-open-source/
2024-01-30 19:15:50 -05:00
Thomas Pelletier caeb9f9631 Fix marshaler typos (#927) 2024-01-30 19:01:55 -05:00
Rdbo e7223fb40e fix: odd indentation in README (#928) 2024-01-30 19:01:43 -05:00
Jakub Wilk 05bedf36d8 Fix README typo (#925) 2024-01-25 15:21:33 -08:00
Daniel Graña f5486d590f Support encoding json.Number type (#923)
Co-authored-by: Thomas Pelletier <thomas@pelletier.codes>
2024-01-25 15:21:02 -08:00
Daniel Graña 2ca21fb7b4 Support encoding of pointers to embedded structs (#924) 2024-01-23 13:06:33 -05:00
Thomas Pelletier 34765b4a9e Fix unmarshaling of nested non-exported struct (#917)
Fixes #915
2023-12-11 14:17:49 -05:00
Moritz Poldrack 358c8d2c23 Use toml-test to generate tests (#911)
Fixes: #909
2023-10-26 12:05:02 -06:00
Martin Tournoij fd8d0bf4d9 Add cmd/gotoml-test-encoder (#907) 2023-10-23 14:40:44 -06:00
Thomas Pelletier a76e18e8c5 Fix benchmark script (#905) 2023-10-02 13:49:01 -04:00
dependabot[bot] dff0c128d0 build(deps): bump docker/login-action from 2 to 3 (#901)
Bumps [docker/login-action](https://github.com/docker/login-action) from 2 to 3.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/v2...v3)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-17 18:11:24 -04:00
Thomas Pelletier 3573ce3770 Update SECURITY.md
Remove placeholder.
2023-09-04 09:43:32 -04:00
dependabot[bot] ae933f2e2a build(deps): bump actions/checkout from 3 to 4 (#896)
Bumps [actions/checkout](https://github.com/actions/checkout) from 3 to 4.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-04 09:42:37 -04:00
Thomas Pelletier 3175efb395 encode: fix commented table with comment (#894)
And updated README.
2023-08-29 08:46:34 -04:00
Thomas Pelletier 9dd7f1af78 encoder: add back the commented option (#893) 2023-08-28 14:29:26 -04:00
Mikhail f. Shiryaev 4a5c27c299 decode: fix wrong indention for tables' comments (#892) 2023-08-28 13:23:11 -04:00
Thomas Pelletier 76cc96f6d8 Update LICENSE 2023-08-28 11:22:32 -04:00
Thomas Pelletier 4835627845 Decode: improve errors on integers and strings (#891) 2023-08-28 11:17:48 -04:00
Thomas Pelletier cef80b96a4 parser: add raw to integers (#890) 2023-08-28 11:02:16 -04:00
Thomas Pelletier 4040373cfd Encode: fix ignored indent of array tables (#889)
Fixes #888
2023-08-28 09:52:11 -04:00
Haiyang Wang bb026cae89 Decode: fix panic when parsing '0' as a float (#887)
Fixes #886
2023-08-22 18:07:39 +02:00
michalbiesek f7d9b9ba53 Build riscv64 binaries for linux (#884)
Signed-off-by: Michal Biesek <michalbiesek@gmail.com>
2023-08-20 19:48:24 +02:00
michalbiesek fac33d6fa8 Add support for Go 1.21 (#885)
* Add support for `Go 1.21`

Signed-off-by: Michal Biesek <michalbiesek@gmail.com>

* add go1.21 guard to fuzz_test.go

* ci: only build last two go versions

* fix workflow yaml syntax error

---------

Signed-off-by: Michal Biesek <michalbiesek@gmail.com>
Co-authored-by: Thomas Pelletier <thomas@pelletier.codes>
2023-08-20 19:47:04 +02:00
David Barroso e183db7e69 Decode: assigned empty struct to empty defined sections (#879)
Co-authored-by: Thomas Pelletier <thomas@pelletier.codes>
2023-07-12 10:53:17 -04:00
dependabot[bot] 60e4af8cca build(deps): bump github.com/stretchr/testify from 1.8.3 to 1.8.4 (#877)
Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.8.3 to 1.8.4.
- [Release notes](https://github.com/stretchr/testify/releases)
- [Commits](https://github.com/stretchr/testify/compare/v1.8.3...v1.8.4)

---
updated-dependencies:
- dependency-name: github.com/stretchr/testify
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-05-30 09:12:07 -04:00
MrJetBOX 8bb1e08dc7 Encode: fix support for arrays (#876) 2023-05-29 09:41:33 -04:00
Thomas Pelletier 7b980e792b Use PtrTo to not require Go 1.18 (#874) 2023-05-23 18:22:22 -04:00
dependabot[bot] 44c1513ccd build(deps): bump github.com/stretchr/testify from 1.8.2 to 1.8.3 (#871)
Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.8.2 to 1.8.3.
- [Release notes](https://github.com/stretchr/testify/releases)
- [Commits](https://github.com/stretchr/testify/compare/v1.8.2...v1.8.3)

---
updated-dependencies:
- dependency-name: github.com/stretchr/testify
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-05-19 10:46:40 -04:00
Thomas Pelletier fcf9d37d0c Comments support in unstable/Parser (#860) 2023-05-19 10:44:02 -04:00
Thomas Pelletier 986afffb7c Decode: fix decode into unsettable structs (#868)
Fixes #866
2023-05-16 09:29:50 -04:00
Thomas Pelletier 8c2c9cc986 Add example on how to use TextUnmarshaler (#867)
Ref #865
2023-05-16 08:17:21 -04:00
manunio 55ca4e35e4 fuzz: improve target perf (#864)
This attempts to improve issue types like timeout, slow_unit
and speed, as seen in the following oss-fuzz performance report:
https://oss-fuzz.com/performance-report/libFuzzer_go-toml_fuzz_toml/libfuzzer_asan_go-toml/2023-05-11
2023-05-12 16:21:17 +02:00
Gordon d34104d493 Support text Un/Marshaller for map keys (#863) 2023-05-09 17:56:57 +02:00
manunio 2aa08368fa fuzz: move fuzz_target from oss-fuzz (#861) 2023-04-28 21:38:13 -04:00
dependabot[bot] 654811fbc3 build(deps): bump actions/setup-go from 3 to 4 (#859)
Bumps [actions/setup-go](https://github.com/actions/setup-go) from 3 to 4.
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](https://github.com/actions/setup-go/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/setup-go
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-21 13:35:40 -04:00
dependabot[bot] 5c05d4d863 build(deps): bump github.com/stretchr/testify from 1.8.1 to 1.8.2 (#852)
Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.8.1 to 1.8.2.
- [Release notes](https://github.com/stretchr/testify/releases)
- [Commits](https://github.com/stretchr/testify/compare/v1.8.1...v1.8.2)

---
updated-dependencies:
- dependency-name: github.com/stretchr/testify
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-20 13:57:54 -04:00
Thomas Pelletier 643c251c4b Fix panic when unmarshaling into a map twice (#854)
Fixes #851
2023-02-28 17:34:24 +01:00
Thomas Pelletier 8a416daa69 Fix error report of type mismatch on inline tables (#853)
Parser did not track the location of the faulty inline table in the
document, and unmarshaler tried to the use the non-raw data field of the
AST node, both resulting into a panic when generating the parser error.

Fixes #850
2023-02-28 17:06:49 +01:00
Andreas Deininger fcd9179b7d Fixes typos (#849) 2023-02-13 03:57:48 -08:00
Marty 9f5726004e Allow integers to be unmarshaled into floats (#841)
Co-authored-by: Marty <martin@windscribe.com>
2023-02-09 12:02:25 -05:00
Thomas Pelletier c4a2eef8a4 Fix go 1.20 version in github actions (#848)
YAML interprets 1.20 as 1.2 without explicitely being a string.
2023-02-09 12:00:14 -05:00
Thomas Pelletier b58c20aa49 Upgrade go 1.20 (#847)
Fixes #842
2023-02-08 18:59:58 -05:00
Cuong Manh Le 090cccf4ba Fix inline table first key value whitespace (#837)
Co-authored-by: Cuong Manh Le <cuong@windscribe.com>
2023-02-01 12:00:09 +01:00
DavidKorczynski 58a592bbf8 ci: add CIFuzz integration (#831)
Signed-off-by: David Korczynski <david@adalogics.com>
2022-11-21 18:51:48 -05:00
dependabot[bot] 94bd3ddcd6 build(deps): bump actions/setup-go from 2 to 3 (#820)
Bumps [actions/setup-go](https://github.com/actions/setup-go) from 2 to 3.
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](https://github.com/actions/setup-go/compare/v2...v3)

---
updated-dependencies:
- dependency-name: actions/setup-go
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-11-09 16:14:19 -05:00
Thomas Pelletier e195b58fd0 Expose parser API as unstable (#827) 2022-11-09 16:12:39 -05:00
dependabot[bot] c83d001c6d build(deps): bump github.com/stretchr/testify from 1.8.0 to 1.8.1 (#825)
Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.8.0 to 1.8.1.
- [Release notes](https://github.com/stretchr/testify/releases)
- [Commits](https://github.com/stretchr/testify/compare/v1.8.0...v1.8.1)

---
updated-dependencies:
- dependency-name: github.com/stretchr/testify
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-30 13:53:41 -04:00
Johanan Idicula b9e3b9c370 refactor: Use typeMismatchError rather than raw string error (#826)
Uses the existing method to DRY up the error message generation, and decorates
with position index where needed. No behaviour is changed, but it allows for
further changes to make error messaging more specific.

Related to: pelletier/go-toml#806
2022-10-30 13:44:16 -04:00
Olivier Mengué d26887310c Reduce init time allocation when declaring types used for reflect (#821)
In declaration of types used for reflect, use
reflect.TypeOf((*T)).Elem() instead of reflect.TypeOf(T{}) to avoid
init-time allocations.

See related stdlib issue: https://github.com/golang/go/issues/55973
2022-10-07 14:28:37 +02:00
Thomas Pelletier 942841787a Fix reflect.Pointer backward compatibility (#813)
Though we don't officially support older versions of Go, this is an easy fix to
unblock people.

Fixes #812
2022-08-26 09:15:03 -04:00
Thomas Pelletier 28f1efc7d3 Decode: don't break on non-struct embed field (#810) 2022-08-22 18:39:11 -04:00
Piotr Buliński 7d69e4a728 Add missing '+build' comment to fuzz_test.go (#809) 2022-08-22 14:05:37 -04:00
Thomas Pelletier e46d245c09 Decode: don't crash on embedded nil pointers (#808)
Also has the perks of reducing the overhead of FindByIndex:

```
name                                old time/op    new time/op    delta
UnmarshalDataset/config-32            17.0ms ± 1%    17.0ms ± 1%    ~     (p=1.000 n=5+5)
UnmarshalDataset/canada-32            71.6ms ± 1%    71.4ms ± 1%    ~     (p=1.000 n=5+5)
UnmarshalDataset/citm_catalog-32      24.2ms ± 3%    23.5ms ± 2%  -3.03%  (p=0.032 n=5+5)
UnmarshalDataset/twitter-32           9.37ms ± 1%    9.09ms ± 2%  -2.97%  (p=0.032 n=5+5)
UnmarshalDataset/code-32              75.4ms ± 2%    74.9ms ± 0%    ~     (p=0.222 n=5+5)
UnmarshalDataset/example-32            147µs ±10%     136µs ± 1%  -7.14%  (p=0.008 n=5+5)
Unmarshal/SimpleDocument/struct-32     512ns ± 2%     500ns ± 0%  -2.35%  (p=0.008 n=5+5)
Unmarshal/SimpleDocument/map-32        721ns ± 2%     702ns ± 1%  -2.68%  (p=0.008 n=5+5)
Unmarshal/ReferenceFile/struct-32     40.1µs ± 0%    39.6µs ± 0%  -1.30%  (p=0.008 n=5+5)
Unmarshal/ReferenceFile/map-32        62.3µs ± 1%    60.6µs ± 0%  -2.83%  (p=0.008 n=5+5)
Unmarshal/HugoFrontMatter-32          10.8µs ± 1%    10.5µs ± 1%  -2.86%  (p=0.008 n=5+5)

name                                old speed      new speed      delta
UnmarshalDataset/config-32          61.8MB/s ± 1%  61.8MB/s ± 1%    ~     (p=1.000 n=5+5)
UnmarshalDataset/canada-32          30.8MB/s ± 1%  30.8MB/s ± 1%    ~     (p=1.000 n=5+5)
UnmarshalDataset/citm_catalog-32    23.0MB/s ± 3%  23.8MB/s ± 2%  +3.09%  (p=0.032 n=5+5)
UnmarshalDataset/twitter-32         47.2MB/s ± 1%  48.6MB/s ± 2%  +3.09%  (p=0.032 n=5+5)
UnmarshalDataset/code-32            35.6MB/s ± 2%  35.9MB/s ± 0%    ~     (p=0.222 n=5+5)
UnmarshalDataset/example-32         55.3MB/s ±10%  59.4MB/s ± 1%  +7.36%  (p=0.008 n=5+5)
Unmarshal/SimpleDocument/struct-32  21.5MB/s ± 2%  22.0MB/s ± 0%  +2.41%  (p=0.008 n=5+5)
Unmarshal/SimpleDocument/map-32     15.2MB/s ± 2%  15.7MB/s ± 1%  +2.74%  (p=0.008 n=5+5)
Unmarshal/ReferenceFile/struct-32    131MB/s ± 0%   132MB/s ± 0%  +1.31%  (p=0.008 n=5+5)
Unmarshal/ReferenceFile/map-32      84.1MB/s ± 1%  86.6MB/s ± 0%  +2.91%  (p=0.008 n=5+5)
Unmarshal/HugoFrontMatter-32        50.6MB/s ± 1%  52.1MB/s ± 1%  +2.93%  (p=0.008 n=5+5)

name                                old alloc/op   new alloc/op   delta
UnmarshalDataset/config-32            5.86MB ± 0%    5.86MB ± 0%    ~     (p=0.579 n=5+5)
UnmarshalDataset/canada-32            83.0MB ± 0%    83.0MB ± 0%    ~     (p=0.651 n=5+5)
UnmarshalDataset/citm_catalog-32      34.7MB ± 0%    34.7MB ± 0%    ~     (p=0.548 n=5+5)
UnmarshalDataset/twitter-32           12.7MB ± 0%    12.7MB ± 0%    ~     (p=1.000 n=5+5)
UnmarshalDataset/code-32              22.2MB ± 0%    22.2MB ± 0%    ~     (p=0.841 n=5+5)
UnmarshalDataset/example-32            186kB ± 0%     186kB ± 0%    ~     (p=0.111 n=5+5)
Unmarshal/SimpleDocument/struct-32      805B ± 0%      805B ± 0%    ~     (all equal)
Unmarshal/SimpleDocument/map-32       1.13kB ± 0%    1.13kB ± 0%    ~     (all equal)
Unmarshal/ReferenceFile/struct-32     20.9kB ± 0%    20.9kB ± 0%    ~     (p=0.643 n=5+5)
Unmarshal/ReferenceFile/map-32        38.3kB ± 0%    38.3kB ± 0%    ~     (p=0.397 n=5+5)
Unmarshal/HugoFrontMatter-32          7.44kB ± 0%    7.44kB ± 0%    ~     (all equal)

name                                old allocs/op  new allocs/op  delta
UnmarshalDataset/config-32              227k ± 0%      227k ± 0%    ~     (p=1.000 n=5+5)
UnmarshalDataset/canada-32              782k ± 0%      782k ± 0%    ~     (all equal)
UnmarshalDataset/citm_catalog-32        192k ± 0%      192k ± 0%    ~     (p=0.968 n=4+5)
UnmarshalDataset/twitter-32            56.9k ± 0%     56.9k ± 0%    ~     (p=0.429 n=4+5)
UnmarshalDataset/code-32               1.05M ± 0%     1.05M ± 0%    ~     (p=0.556 n=4+5)
UnmarshalDataset/example-32            1.36k ± 0%     1.36k ± 0%    ~     (all equal)
Unmarshal/SimpleDocument/struct-32      9.00 ± 0%      9.00 ± 0%    ~     (all equal)
Unmarshal/SimpleDocument/map-32         13.0 ± 0%      13.0 ± 0%    ~     (all equal)
Unmarshal/ReferenceFile/struct-32        183 ± 0%       183 ± 0%    ~     (all equal)
Unmarshal/ReferenceFile/map-32           642 ± 0%       642 ± 0%    ~     (all equal)
Unmarshal/HugoFrontMatter-32             141 ± 0%       141 ± 0%    ~     (all equal)
```

Fixes #807
2022-08-20 21:24:03 -04:00
Thomas Pelletier 7baa23f493 Decode: error on array table mismatched type (#804)
Prevent the decoder from continuing if it encounters a type it cannot decode an
array table into.

Fixes #799
2022-08-15 16:38:07 -04:00
Thomas Pelletier 2d8433b69e Encode: don't inherit omitempty (#803)
Fixes #786.
2022-08-15 11:29:46 -04:00
Thomas Pelletier 67bc5422f3 Go 1.19 (#802) 2022-08-15 10:56:33 -04:00
Thomas Pelletier fb6d1d6c2b Marshal: define and fix newlines behavior when using omitempty (#798)
Ref #786
2022-07-24 15:40:20 -04:00
dependabot[bot] d017a6dc89 build(deps): bump github.com/stretchr/testify from 1.7.5 to 1.8.0 (#795) 2022-06-29 09:51:28 -04:00
dependabot[bot] d6d3196163 build(deps): bump github.com/stretchr/testify from 1.7.4 to 1.7.5 (#794) 2022-06-24 12:49:56 -04:00
dependabot[bot] 41718a6db3 build(deps): bump github.com/stretchr/testify from 1.7.2 to 1.7.4 (#793)
Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.7.2 to 1.7.4.
- [Release notes](https://github.com/stretchr/testify/releases)
- [Commits](https://github.com/stretchr/testify/compare/v1.7.2...v1.7.4)

---
updated-dependencies:
- dependency-name: github.com/stretchr/testify
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-06-21 08:32:13 -04:00
Thomas Pelletier 216628222f Build arm + arm64 binaries for linux and windows (#790)
* Build arm + arm64 binaries for linux and windows

* Type MaxInt64 to avoid overflow on 32 bits arch

On a 32 bits arch, math.MaxIn64 is interpreted as an int, and therefore
overflows. This causes compilation to build on those platforms.
2022-06-08 18:05:42 -04:00
dependabot[bot] 322e0b15d2 build(deps): bump github.com/stretchr/testify from 1.7.1 to 1.7.2 (#788)
Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.7.1 to 1.7.2.
- [Release notes](https://github.com/stretchr/testify/releases)
- [Commits](https://github.com/stretchr/testify/compare/v1.7.1...v1.7.2)

---
updated-dependencies:
- dependency-name: github.com/stretchr/testify
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-06-08 17:44:59 -04:00
Thomas Pelletier 85bfc0ed51 Encode: add bound check for uint64 > math.Int64 (#785)
As brought up on #782, there is an asymetry between numbers go-toml can encode
and decode. Specifically, unsigned numbers larger than `math.Int64` could be
encoded but not decoded.

We considered two options: allow decoding of those numbers, or prevent those
numbers to be decoded.

Ultimately we decided to narrow the range of numbers to be encoded. The TOML
specification only requires parsers to support at least the 64 bits integer
range. Allowing larger numbers would create non-standard TOML documents, which
may not be readable (at best) by other implementations. It is a safer default to
generate documents valid by default. People who wish to deal with larger numbers
can provide a custom type implementing `encoding.TextMarshaler`.

Refs #781
2022-05-31 22:10:41 -04:00
dependabot[bot] 295a720dfb build(deps): bump goreleaser/goreleaser-action from 2 to 3 (#783)
Bumps [goreleaser/goreleaser-action](https://github.com/goreleaser/goreleaser-action) from 2 to 3.
- [Release notes](https://github.com/goreleaser/goreleaser-action/releases)
- [Commits](https://github.com/goreleaser/goreleaser-action/compare/v2...v3)

---
updated-dependencies:
- dependency-name: goreleaser/goreleaser-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-05-23 15:05:00 +02:00
Thomas Pelletier 0a422e3dbd Decoder: check max uint on 32 bit platforms (#778)
Fixes #777
2022-05-10 15:43:26 +02:00
Thomas Pelletier 627dade0c7 Encode: support comment on array tables (#776)
Fixes #774
2022-05-10 15:17:36 +02:00
Thomas Pelletier b2e0231cc9 Encode: fix multiline comment (#775)
Fixes #768
2022-05-10 14:53:26 +02:00
dependabot[bot] ba95863cd3 build(deps): bump docker/login-action from 1 to 2 (#771)
Bumps [docker/login-action](https://github.com/docker/login-action) from 1 to 2.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/v1...v2)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-05-09 18:59:02 +02:00
Erik Hartwig db679df765 Typo in README.md fix (#770)
Change "document was not prevent in the target structure" to "document was not present in the target structure".
2022-05-09 18:46:46 +02:00
Thomas Pelletier c5ca2c682b Fix embedded struct with explicit field name (#773)
Fixes #772
2022-05-09 18:45:02 +02:00
Thomas Pelletier ed80712cb4 Copy version policy from v1 2022-04-28 11:55:39 -04:00
dependabot[bot] b24772942d build(deps): bump github/codeql-action from 1 to 2 (#764)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 1 to 2.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/v1...v2)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-28 08:24:14 -04:00
dependabot[bot] 9501a05ed7 build(deps): bump actions/checkout from 2 to 3 (#765)
Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 3.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v2...v3)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-28 08:23:56 -04:00
Thomas Pelletier 171a592663 Prepare repository for v2.0.0 (#762) 2022-04-27 21:29:16 -04:00
Thomas Pelletier 5aaf5ef13b Fix internal entry size test (#761)
The test meant to assert that the size of entry does not grow beyond
what it is today. 48 is the current size in bytes on x64. On 32-bit
platform that value should be less. As written the test was doing the
opposite comparison.

Fixes #760
2022-04-24 18:50:54 -04:00
Thomas Pelletier adacebd8c7 Update benchmarks in README (#756) 2022-04-10 22:24:19 -04:00
Thomas Pelletier 8bbb673431 Fuzzing setup and fixes (#755)
* encode: fix localdate formatting
* encode: fix empty key marshaling
* encode: fix invalid quotation of time.Time
* encode: ensure control chars are escaped
* decode: always use UTC for zero tz
* encode: check for invalid characters in keys
* encode: always construct map for empty array tables
* fuzz: add go 1.18 fuzz test
* encode: handle NaNs
* encode: allow new lines in quoted keys
* encode: never emit table inside array
* encode: don't capitalize inf
2022-04-10 21:37:12 -04:00
Thomas Pelletier 2377ac4bc0 encode: fix embedded interfaces (#753)
Resolve marshaling regression when handling an embedded interface in a
struct.

Fixes #752
2022-04-08 09:25:54 -04:00
Thomas Pelletier f5cc8c49eb decoder: remove mention of UnmarshalText in errors (#751)
Fixes #737
2022-04-07 21:58:19 -04:00
Thomas Pelletier 89d7b412d8 decode: allow subtables to be defined later (#750)
Fixes #739
2022-04-07 21:49:16 -04:00
Thomas Pelletier 88a8aecdd4 tools: display error context when it exists (#749)
For example when failing to decode toml, display the context around the
error and the location of the problem.
2022-04-07 20:33:09 -04:00
Thomas Pelletier 9804fc57e0 decoder: support \e escape sequence (#748) 2022-04-07 20:18:30 -04:00
Thomas Pelletier 068279f13b encode: respect stdlib rules for embedded structs (#747) 2022-04-07 19:51:09 -04:00
Thomas Pelletier b9edbeb611 Update testify dependency (#746) 2022-03-30 15:10:57 -04:00
Thomas Pelletier a97c9317d4 Go 1.18 (#745) 2022-03-23 10:26:12 -04:00
Gregory Oschwald 3229a0abfb Decode: convert table key to correct type (#741)
Fixes #740.
2022-03-02 09:24:01 -05:00
Thomas Pelletier 3f5d8a6b06 Mention removal of go-toml/query (#736)
Fixes #722
2022-01-07 18:58:33 -05:00
Cameron Moore 146f70ea8a Decode: use cleaned byte slice throughout parseFloat (#735)
Fixes #734
2022-01-06 14:34:27 -05:00
Thomas Pelletier e83cf535f5 Decoder: rename SetStrict to DisallowUnknownFields (#731) 2022-01-02 14:32:34 -05:00
Thomas Pelletier c3ba3ef97a readme: add docker image 2022-01-01 09:53:14 -05:00
Thomas Pelletier 7ee3c8ff25 build: login to github repository 2022-01-01 09:24:52 -05:00
Thomas Pelletier 1e85aa6d78 build: add contents permissions 2021-12-31 20:47:50 -05:00
Thomas Pelletier 46fa3225e2 build: allows the token to write packages 2021-12-31 20:25:11 -05:00
Thomas Pelletier 4d51831dab build: replace grc.io with ghcr.io 2021-12-31 20:10:36 -05:00
Thomas Pelletier 5a1a96cb2d build: pass the github token to goreleaser 2021-12-31 20:03:18 -05:00
Thomas Pelletier ea9040ae83 build: change workflow reference to v2 2021-12-31 19:57:41 -05:00
Thomas Pelletier 2373685f1e Docker + goreleaser (#729) 2021-12-31 19:55:31 -05:00
Thomas Pelletier f1391952d4 Update and move testsuite to internal package (#730)
* Regenerate test suite

* Move test suite to /internal
2021-12-31 18:52:26 -05:00
Thomas Pelletier 4a73a200ed Clean up tools godoc 2021-12-31 15:56:40 -05:00
Thomas Pelletier 4807229e94 tomll: port to v2 (#727)
Fixes #721
2021-12-31 15:40:18 -05:00
Thomas Pelletier d8ddc00c61 jsontoml: port to v2 (#726)
Fixes #719
2021-12-31 14:40:20 -05:00
Thomas Pelletier 82f8dad811 tomljson: port to v2 (#725) 2021-12-31 13:25:53 -05:00
Thomas Pelletier 75db1016e8 Remove extra words from CONTRIBUTING 2021-12-29 10:26:22 -05:00
Thomas Pelletier de6d715bd2 Update CONTRIBUTING.md 2021-12-29 10:25:28 -05:00
Thomas Pelletier 3ab2fc2b87 Update release proces documentation 2021-12-29 10:24:01 -05:00
92 changed files with 11219 additions and 2974 deletions
+1
View File
@@ -1,3 +1,4 @@
* text=auto * text=auto
benchmark/benchmark.toml text eol=lf benchmark/benchmark.toml text eol=lf
testdata/** text eol=lf
+26
View File
@@ -0,0 +1,26 @@
name: CIFuzz
on: [pull_request]
jobs:
Fuzzing:
runs-on: ubuntu-latest
steps:
- name: Build Fuzzers
id: build
uses: google/oss-fuzz/infra/cifuzz/actions/build_fuzzers@master
with:
oss-fuzz-project-name: 'go-toml'
dry-run: false
language: go
- name: Run Fuzzers
uses: google/oss-fuzz/infra/cifuzz/actions/run_fuzzers@master
with:
oss-fuzz-project-name: 'go-toml'
fuzz-seconds: 300
dry-run: false
language: go
- name: Upload Crash
uses: actions/upload-artifact@v6
if: failure() && steps.build.outcome == 'success'
with:
name: artifacts
path: ./out/artifacts
+5 -5
View File
@@ -35,11 +35,11 @@ jobs:
steps: steps:
- name: Checkout repository - name: Checkout repository
uses: actions/checkout@v2 uses: actions/checkout@v6
# Initializes the CodeQL tools for scanning. # Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL - name: Initialize CodeQL
uses: github/codeql-action/init@v1 uses: github/codeql-action/init@v4
with: with:
languages: ${{ matrix.language }} languages: ${{ matrix.language }}
# If you wish to specify custom queries, you can do so here or in a config file. # If you wish to specify custom queries, you can do so here or in a config file.
@@ -47,10 +47,10 @@ jobs:
# Prefix the list here with "+" to use these queries and those in the config file. # Prefix the list here with "+" to use these queries and those in the config file.
# queries: ./path/to/local/query, your-org/your-repo/queries@main # queries: ./path/to/local/query, your-org/your-repo/queries@main
# Autobuild attempts to build any compiled languages (C/C++, C#, or Java). # Autobuild attempts to build any compiled languages (C/C++, C#, or Java).
# If this step fails, then you should remove it and run the build manually (see below) # If this step fails, then you should remove it and run the build manually (see below)
- name: Autobuild - name: Autobuild
uses: github/codeql-action/autobuild@v1 uses: github/codeql-action/autobuild@v4
# ️ Command-line programs to run using the OS shell. # ️ Command-line programs to run using the OS shell.
# 📚 https://git.io/JvXDl # 📚 https://git.io/JvXDl
@@ -64,4 +64,4 @@ jobs:
# make release # make release
- name: Perform CodeQL Analysis - name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v1 uses: github/codeql-action/analyze@v4
+3 -3
View File
@@ -9,12 +9,12 @@ jobs:
runs-on: "ubuntu-latest" runs-on: "ubuntu-latest"
name: report name: report
steps: steps:
- uses: actions/checkout@master - uses: actions/checkout@v6
with: with:
fetch-depth: 0 fetch-depth: 0
- name: Setup go - name: Setup go
uses: actions/setup-go@master uses: actions/setup-go@v6
with: with:
go-version: 1.16 go-version: "1.25"
- name: Run tests with coverage - name: Run tests with coverage
run: ./ci.sh coverage -d "${GITHUB_BASE_REF-HEAD}" run: ./ci.sh coverage -d "${GITHUB_BASE_REF-HEAD}"
+36
View File
@@ -0,0 +1,36 @@
name: Go Versions Compatibility Test
on:
workflow_dispatch:
inputs:
go_versions:
description: 'Go versions to test (space-separated, e.g., "1.21 1.22 1.23")'
required: false
default: ''
type: string
jobs:
test:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Run Go versions compatibility test
run: |
VERSIONS="${{ github.event.inputs.go_versions }}"
./test-go-versions.sh --output ./test-results $VERSIONS
- name: Upload test results
uses: actions/upload-artifact@v6
with:
name: go-versions-test-results
path: |
test-results/
retention-days: 30
+22
View File
@@ -0,0 +1,22 @@
name: lint
on:
pull_request:
branches:
- v2
jobs:
golangci:
name: lint
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Setup go
uses: actions/setup-go@v6
with:
go-version: "1.24"
- name: Run golangci-lint
uses: golangci/golangci-lint-action@v9
with:
version: v2.8.0
+39
View File
@@ -0,0 +1,39 @@
name: release
on:
push:
tags:
- "v2.*"
workflow_call:
inputs:
args:
description: "Extra arguments to pass goreleaser"
default: ""
required: false
type: string
jobs:
release:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Set up Go
uses: actions/setup-go@v6
with:
go-version: "1.25"
- name: Login to GitHub Container Registry
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Run GoReleaser
uses: goreleaser/goreleaser-action@v6
with:
distribution: goreleaser
version: '~> v2'
args: release ${{ inputs.args }} --clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+11 -4
View File
@@ -11,15 +11,22 @@ jobs:
build: build:
strategy: strategy:
matrix: matrix:
os: [ 'ubuntu-latest', 'windows-latest', 'macos-latest'] os: [ 'ubuntu-latest', 'windows-latest', 'macos-latest', 'macos-14' ]
go: [ '1.16', '1.17' ] go: [ '1.24', '1.25' ]
runs-on: ${{ matrix.os }} runs-on: ${{ matrix.os }}
name: ${{ matrix.go }}/${{ matrix.os }} name: ${{ matrix.go }}/${{ matrix.os }}
steps: steps:
- uses: actions/checkout@master - uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Setup go ${{ matrix.go }} - name: Setup go ${{ matrix.go }}
uses: actions/setup-go@master uses: actions/setup-go@v6
with: with:
go-version: ${{ matrix.go }} go-version: ${{ matrix.go }}
- name: Run unit tests - name: Run unit tests
run: go test -race ./... run: go test -race ./...
release-check:
if: ${{ github.ref != 'refs/heads/v2' }}
uses: ./.github/workflows/release.yml
with:
args: --snapshot
+3
View File
@@ -3,3 +3,6 @@ fuzz/
cmd/tomll/tomll cmd/tomll/tomll
cmd/tomljson/tomljson cmd/tomljson/tomljson
cmd/tomltestgen/tomltestgen cmd/tomltestgen/tomltestgen
dist
tests/
test-results
+33 -41
View File
@@ -1,84 +1,76 @@
[service] version = "2"
golangci-lint-version = "1.39.0"
[linters-settings.wsl]
allow-assign-and-anything = true
[linters-settings.exhaustive]
default-signifies-exhaustive = true
[linters] [linters]
disable-all = true default = "none"
enable = [ enable = [
"asciicheck", "asciicheck",
"bodyclose", "bodyclose",
"cyclop",
"deadcode",
"depguard",
"dogsled", "dogsled",
"dupl", "dupl",
"durationcheck", "durationcheck",
"errcheck", "errcheck",
"errorlint", "errorlint",
"exhaustive", "exhaustive",
# "exhaustivestruct",
"exportloopref",
"forbidigo", "forbidigo",
# "forcetypeassert",
"funlen",
"gci",
# "gochecknoglobals",
"gochecknoinits", "gochecknoinits",
"gocognit",
"goconst", "goconst",
"gocritic", "gocritic",
"gocyclo", "godoclint",
"godot",
"godox",
# "goerr113",
"gofmt",
"gofumpt",
"goheader", "goheader",
"goimports",
"golint",
"gomnd",
# "gomoddirectives",
"gomodguard", "gomodguard",
"goprintffuncname", "goprintffuncname",
"gosec", "gosec",
"gosimple",
"govet", "govet",
# "ifshort",
"importas", "importas",
"ineffassign", "ineffassign",
"lll", "lll",
"makezero", "makezero",
"mirror",
"misspell", "misspell",
"nakedret", "nakedret",
"nestif",
"nilerr", "nilerr",
# "nlreturn",
"noctx", "noctx",
"nolintlint", "nolintlint",
#"paralleltest", "perfsprint",
"prealloc", "prealloc",
"predeclared", "predeclared",
"revive", "revive",
"rowserrcheck", "rowserrcheck",
"sqlclosecheck", "sqlclosecheck",
"staticcheck", "staticcheck",
"structcheck",
"stylecheck",
# "testpackage",
"thelper", "thelper",
"tparallel", "tparallel",
"typecheck",
"unconvert", "unconvert",
"unparam", "unparam",
"unused", "unused",
"varcheck", "usetesting",
"wastedassign", "wastedassign",
"whitespace", "whitespace",
# "wrapcheck", ]
# "wsl"
[linters.settings.exhaustive]
default-signifies-exhaustive = true
[linters.settings.lll]
line-length = 150
[[linters.exclusions.rules]]
path = ".test.go"
linters = ["goconst", "gosec"]
[[linters.exclusions.rules]]
path = "main.go"
linters = ["forbidigo"]
[[linters.exclusions.rules]]
path = "internal"
linters = ["revive"]
text = "(exported|indent-error-flow): "
[formatters]
enable = [
"gci",
"gofmt",
"gofumpt",
"goimports",
] ]
+127
View File
@@ -0,0 +1,127 @@
version: 2
before:
hooks:
- go mod tidy
- go fmt ./...
- go test ./...
builds:
- id: tomll
main: ./cmd/tomll
binary: tomll
env:
- CGO_ENABLED=0
flags:
- -trimpath
ldflags:
- -X main.version={{.Version}} -X main.commit={{.Commit}} -X main.date={{.CommitDate}}
mod_timestamp: '{{ .CommitTimestamp }}'
targets:
- linux_amd64
- linux_arm64
- linux_arm
- linux_riscv64
- windows_amd64
- windows_arm64
- windows_arm
- darwin_amd64
- darwin_arm64
- id: tomljson
main: ./cmd/tomljson
binary: tomljson
env:
- CGO_ENABLED=0
flags:
- -trimpath
ldflags:
- -X main.version={{.Version}} -X main.commit={{.Commit}} -X main.date={{.CommitDate}}
mod_timestamp: '{{ .CommitTimestamp }}'
targets:
- linux_amd64
- linux_arm64
- linux_arm
- linux_riscv64
- windows_amd64
- windows_arm64
- windows_arm
- darwin_amd64
- darwin_arm64
- id: jsontoml
main: ./cmd/jsontoml
binary: jsontoml
env:
- CGO_ENABLED=0
flags:
- -trimpath
ldflags:
- -X main.version={{.Version}} -X main.commit={{.Commit}} -X main.date={{.CommitDate}}
mod_timestamp: '{{ .CommitTimestamp }}'
targets:
- linux_amd64
- linux_arm64
- linux_riscv64
- linux_arm
- windows_amd64
- windows_arm64
- windows_arm
- darwin_amd64
- darwin_arm64
universal_binaries:
- id: tomll
replace: true
name_template: tomll
- id: tomljson
replace: true
name_template: tomljson
- id: jsontoml
replace: true
name_template: jsontoml
archives:
- id: jsontoml
format: tar.xz
builds:
- jsontoml
files:
- none*
name_template: "{{ .Binary }}_{{.Version}}_{{ .Os }}_{{ .Arch }}"
- id: tomljson
format: tar.xz
builds:
- tomljson
files:
- none*
name_template: "{{ .Binary }}_{{.Version}}_{{ .Os }}_{{ .Arch }}"
- id: tomll
format: tar.xz
builds:
- tomll
files:
- none*
name_template: "{{ .Binary }}_{{.Version}}_{{ .Os }}_{{ .Arch }}"
dockers:
- id: tools
goos: linux
goarch: amd64
ids:
- jsontoml
- tomljson
- tomll
image_templates:
- "ghcr.io/pelletier/go-toml:latest"
- "ghcr.io/pelletier/go-toml:{{ .Tag }}"
- "ghcr.io/pelletier/go-toml:v{{ .Major }}"
skip_push: false
checksum:
name_template: 'sha256sums.txt'
snapshot:
version_template: "{{ incpatch .Version }}-next"
release:
github:
owner: pelletier
name: go-toml
draft: true
prerelease: auto
mode: replace
changelog:
use: github-native
announce:
skip: true
+64
View File
@@ -0,0 +1,64 @@
# Agent Guidelines for go-toml
This file provides guidelines for AI agents contributing to go-toml. All agents must follow these rules derived from [CONTRIBUTING.md](./CONTRIBUTING.md).
## Project Overview
go-toml is a TOML library for Go. The goal is to provide an easy-to-use and efficient TOML implementation that gets the job done without getting in the way.
## Code Change Rules
### Backward Compatibility
- **No backward-incompatible changes** unless explicitly discussed and approved
- Avoid breaking people's programs unless absolutely necessary
### Testing Requirements
- **All bug fixes must include regression tests**
- **All new code must be tested**
- Run tests before submitting: `go test -race ./...`
- Test coverage must not decrease. Check with:
```bash
go test -covermode=atomic -coverprofile=coverage.out
go tool cover -func=coverage.out
```
- All lines of code touched by changes should be covered by tests
### Performance Requirements
- go-toml aims to stay efficient; avoid performance regressions
- Run benchmarks to verify: `go test ./... -bench=. -count=10`
- Compare results using [benchstat](https://pkg.go.dev/golang.org/x/perf/cmd/benchstat)
### Documentation
- New features or feature extensions must include documentation
- Documentation lives in [README.md](./README.md) and throughout source code
### Code Style
- Follow existing code format and structure
- Code must pass `go fmt`
- Code must pass linting with the same golangci-lint version as CI (see version in `.github/workflows/lint.yml`):
```bash
# Install specific version (check lint.yml for current version)
curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/HEAD/install.sh | sh -s -- -b $(go env GOPATH)/bin <version>
# Run linter
golangci-lint run ./...
```
### Commit Messages
- Commit messages must explain **why** the change is needed
- Keep messages clear and informative even if details are in the PR description
## Pull Request Checklist
Before submitting:
1. Tests pass (`go test -race ./...`)
2. No backward-incompatible changes (unless discussed)
3. Relevant documentation added/updated
4. No performance regression (verify with benchmarks)
5. Title is clear and understandable for changelog
+64 -11
View File
@@ -33,7 +33,7 @@ The documentation is present in the [README][readme] and thorough the source
code. On release, it gets updated on [pkg.go.dev][pkg.go.dev]. To make a change code. On release, it gets updated on [pkg.go.dev][pkg.go.dev]. To make a change
to the documentation, create a pull request with your proposed changes. For to the documentation, create a pull request with your proposed changes. For
simple changes like that, the easiest way to go is probably the "Fork this simple changes like that, the easiest way to go is probably the "Fork this
project and edit the file" button on Github, displayed at the top right of the project and edit the file" button on GitHub, displayed at the top right of the
file. Unless it's a trivial change (for example a typo), provide a little bit of file. Unless it's a trivial change (for example a typo), provide a little bit of
context in your pull request description or commit message. context in your pull request description or commit message.
@@ -92,6 +92,48 @@ However, given GitHub's new policy to _not_ run Actions on pull requests until a
maintainer clicks on button, it is highly recommended that you run them locally maintainer clicks on button, it is highly recommended that you run them locally
as you make changes. as you make changes.
### Test across Go versions
The repository includes tooling to test go-toml across multiple Go versions
(1.11 through 1.25) both locally and in GitHub Actions.
#### Local testing with Docker
Prerequisites: Docker installed and running, Bash shell, `rsync` command.
```bash
# Test all Go versions in parallel (default)
./test-go-versions.sh
# Test specific versions
./test-go-versions.sh 1.21 1.22 1.23
# Test sequentially (slower but uses less resources)
./test-go-versions.sh --sequential
# Verbose output with custom results directory
./test-go-versions.sh --verbose --output ./my-results 1.24 1.25
# Show all options
./test-go-versions.sh --help
```
The script creates Docker containers for each Go version and runs the full test
suite. Results are saved to a `test-results/` directory with individual logs and
a comprehensive summary report.
The script only exits with a non-zero status code if either of the two most
recent Go versions fail.
#### GitHub Actions testing (maintainers)
1. Go to the **Actions** tab in the GitHub repository
2. Select **"Go Versions Compatibility Test"** from the workflow list
3. Click **"Run workflow"**
4. Optionally customize:
- **Go versions**: Space-separated list (e.g., `1.21 1.22 1.23`)
- **Execution mode**: Parallel (faster) or sequential (more stable)
### Check coverage ### Check coverage
We use `go tool cover` to compute test coverage. Most code editors have a way to We use `go tool cover` to compute test coverage. Most code editors have a way to
@@ -111,7 +153,7 @@ code lowers the coverage.
Go-toml aims to stay efficient. We rely on a set of scenarios executed with Go's Go-toml aims to stay efficient. We rely on a set of scenarios executed with Go's
builtin benchmark systems. Because of their noisy nature, containers provided by builtin benchmark systems. Because of their noisy nature, containers provided by
Github Actions cannot be reliably used for benchmarking. As a result, you are GitHub Actions cannot be reliably used for benchmarking. As a result, you are
responsible for checking that your changes do not incur a performance penalty. responsible for checking that your changes do not incur a performance penalty.
You can run their following to execute benchmarks: You can run their following to execute benchmarks:
@@ -155,6 +197,8 @@ Checklist:
- Does not introduce backward-incompatible changes (unless discussed). - Does not introduce backward-incompatible changes (unless discussed).
- Has relevant doc changes. - Has relevant doc changes.
- Benchstat does not show performance regression. - Benchstat does not show performance regression.
- Pull request is [labeled appropriately][pr-labels].
- Title will be understandable in the changelog.
1. Merge using "squash and merge". 1. Merge using "squash and merge".
2. Make sure to edit the commit message to keep all the useful information 2. Make sure to edit the commit message to keep all the useful information
@@ -163,13 +207,22 @@ Checklist:
### New release ### New release
1. Go to [releases][releases]. Click on "X commits to master since this 1. Decide on the next version number. Use semver. Review commits since last
release". version to assess.
2. Make note of all the changes. Look for backward incompatible changes, 2. Tag release. For example:
new features, and bug fixes. ```
3. Pick the new version using the above and semver. git checkout v2
4. Create a [new release][new-release]. git pull
5. Follow the same format as [1.1.0][release-110]. git tag v2.2.0
git push --tags
```
3. CI automatically builds a draft GitHub release. Review it and edit as
necessary. Look for "Other changes". That would indicate a pull request not
labeled properly. Tweak labels and pull request titles until changelog looks
good for users.
4. Check "create discussion" box, in the "Releases" category.
5. If new version is an alpha or beta only, check pre-release box.
[issues-tracker]: https://github.com/pelletier/go-toml/issues [issues-tracker]: https://github.com/pelletier/go-toml/issues
[bug-report]: https://github.com/pelletier/go-toml/issues/new?template=bug_report.md [bug-report]: https://github.com/pelletier/go-toml/issues/new?template=bug_report.md
@@ -177,6 +230,6 @@ Checklist:
[readme]: ./README.md [readme]: ./README.md
[fork]: https://help.github.com/articles/fork-a-repo [fork]: https://help.github.com/articles/fork-a-repo
[pull-request]: https://help.github.com/en/articles/creating-a-pull-request [pull-request]: https://help.github.com/en/articles/creating-a-pull-request
[releases]: https://github.com/pelletier/go-toml/releases
[new-release]: https://github.com/pelletier/go-toml/releases/new [new-release]: https://github.com/pelletier/go-toml/releases/new
[release-110]: https://github.com/pelletier/go-toml/releases/tag/v1.1.0 [gh]: https://github.com/cli/cli
[pr-labels]: https://github.com/pelletier/go-toml/blob/v2/.github/release.yml
+5
View File
@@ -0,0 +1,5 @@
FROM scratch
ENV PATH "$PATH:/bin"
COPY tomll /bin/tomll
COPY tomljson /bin/tomljson
COPY jsontoml /bin/jsontoml
+2 -1
View File
@@ -1,6 +1,7 @@
The MIT License (MIT) The MIT License (MIT)
Copyright (c) 2013 - 2021 Thomas Pelletier, Eric Anderton go-toml v2
Copyright (c) 2021 - 2023 Thomas Pelletier
Permission is hereby granted, free of charge, to any person obtaining a copy Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal of this software and associated documentation files (the "Software"), to deal
+209 -105
View File
@@ -2,18 +2,7 @@
Go library for the [TOML](https://toml.io/en/) format. Go library for the [TOML](https://toml.io/en/) format.
This library supports [TOML v1.0.0](https://toml.io/en/v1.0.0). This library supports [TOML v1.1.0](https://toml.io/en/v1.1.0).
## Development status
This is the upcoming major version of go-toml. It is currently in active
development. As of release v2.0.0-beta.1, the library has reached feature parity
with v1, and fixes a lot known bugs and performance issues along the way.
If you do not need the advanced document editing features of v1, you are
encouraged to try out this version.
[👉 Roadmap for v2](https://github.com/pelletier/go-toml/discussions/506)
[🐞 Bug Reports](https://github.com/pelletier/go-toml/issues) [🐞 Bug Reports](https://github.com/pelletier/go-toml/issues)
@@ -21,7 +10,8 @@ encouraged to try out this version.
## Documentation ## Documentation
Full API, examples, and implementation notes are available in the Go documentation. Full API, examples, and implementation notes are available in the Go
documentation.
[![Go Reference](https://pkg.go.dev/badge/github.com/pelletier/go-toml/v2.svg)](https://pkg.go.dev/github.com/pelletier/go-toml/v2) [![Go Reference](https://pkg.go.dev/badge/github.com/pelletier/go-toml/v2.svg)](https://pkg.go.dev/github.com/pelletier/go-toml/v2)
@@ -48,23 +38,22 @@ operations should not be shockingly slow. See [benchmarks](#benchmarks).
### Strict mode ### Strict mode
`Decoder` can be set to "strict mode", which makes it error when some parts of `Decoder` can be set to "strict mode", which makes it error when some parts of
the TOML document was not prevent in the target structure. This is a great way the TOML document was not present in the target structure. This is a great way
to check for typos. [See example in the documentation][strict]. to check for typos. [See example in the documentation][strict].
[strict]: https://pkg.go.dev/github.com/pelletier/go-toml/v2#example-Decoder.SetStrict [strict]: https://pkg.go.dev/github.com/pelletier/go-toml/v2#example-Decoder.DisallowUnknownFields
### Contextualized errors ### Contextualized errors
When most decoding errors occur, go-toml returns [`DecodeError`][decode-err]), When most decoding errors occur, go-toml returns [`DecodeError`][decode-err],
which contains a human readable contextualized version of the error. For which contains a human readable contextualized version of the error. For
example: example:
``` ```
2| key1 = "value1" 1| [server]
3| key2 = "missing2" 2| path = 100
| ~~~~ missing field | ~~~ cannot decode TOML integer into struct field toml_test.Server.Path of type string
4| key3 = "missing3" 3| port = 50
5| key4 = "value4"
``` ```
[decode-err]: https://pkg.go.dev/github.com/pelletier/go-toml/v2#DecodeError [decode-err]: https://pkg.go.dev/github.com/pelletier/go-toml/v2#DecodeError
@@ -78,27 +67,51 @@ this use-case, go-toml provides [`LocalDate`][tld], [`LocalTime`][tlt], and
making them convenient yet unambiguous structures for their respective TOML making them convenient yet unambiguous structures for their respective TOML
representation. representation.
[ldt]: https://toml.io/en/v1.0.0#local-date-time [ldt]: https://toml.io/en/v1.1.0#local-date-time
[tld]: https://pkg.go.dev/github.com/pelletier/go-toml/v2#LocalDate [tld]: https://pkg.go.dev/github.com/pelletier/go-toml/v2#LocalDate
[tlt]: https://pkg.go.dev/github.com/pelletier/go-toml/v2#LocalTime [tlt]: https://pkg.go.dev/github.com/pelletier/go-toml/v2#LocalTime
[tldt]: https://pkg.go.dev/github.com/pelletier/go-toml/v2#LocalDateTime [tldt]: https://pkg.go.dev/github.com/pelletier/go-toml/v2#LocalDateTime
### Commented config
Since TOML is often used for configuration files, go-toml can emit documents
annotated with [comments and commented-out values][comments-example]. For
example, it can generate the following file:
```toml
# Host IP to connect to.
host = '127.0.0.1'
# Port of the remote server.
port = 4242
# Encryption parameters (optional)
# [TLS]
# cipher = 'AEAD-AES128-GCM-SHA256'
# version = 'TLS 1.3'
```
[comments-example]: https://pkg.go.dev/github.com/pelletier/go-toml/v2#example-Marshal-Commented
## Getting started ## Getting started
Given the following struct, let's see how to read it and write it as TOML: Given the following struct, let's see how to read it and write it as TOML:
```go ```go
type MyConfig struct { type MyConfig struct {
Version int Version int
Name string Name string
Tags []string Tags []string
} }
``` ```
### Unmarshaling ### Unmarshaling
[`Unmarshal`][unmarshal] reads a TOML document and fills a Go structure with its [`Unmarshal`][unmarshal] reads a TOML document and fills a Go structure with its
content. For example: content.
Note that the struct variable names are _capitalized_, while the variables in the toml document are _lowercase_.
For example:
```go ```go
doc := ` doc := `
@@ -110,7 +123,7 @@ tags = ["go", "toml"]
var cfg MyConfig var cfg MyConfig
err := toml.Unmarshal([]byte(doc), &cfg) err := toml.Unmarshal([]byte(doc), &cfg)
if err != nil { if err != nil {
panic(err) panic(err)
} }
fmt.Println("version:", cfg.Version) fmt.Println("version:", cfg.Version)
fmt.Println("name:", cfg.Name) fmt.Println("name:", cfg.Name)
@@ -124,6 +137,62 @@ fmt.Println("tags:", cfg.Tags)
[unmarshal]: https://pkg.go.dev/github.com/pelletier/go-toml/v2#Unmarshal [unmarshal]: https://pkg.go.dev/github.com/pelletier/go-toml/v2#Unmarshal
Here is an example using tables with some simple nesting:
```go
doc := `
age = 45
fruits = ["apple", "pear"]
# these are very important!
[my-variables]
first = 1
second = 0.2
third = "abc"
# this is not so important.
[my-variables.b]
bfirst = 123
`
var Document struct {
Age int
Fruits []string
Myvariables struct {
First int
Second float64
Third string
B struct {
Bfirst int
}
} `toml:"my-variables"`
}
err := toml.Unmarshal([]byte(doc), &Document)
if err != nil {
panic(err)
}
fmt.Println("age:", Document.Age)
fmt.Println("fruits:", Document.Fruits)
fmt.Println("my-variables.first:", Document.Myvariables.First)
fmt.Println("my-variables.second:", Document.Myvariables.Second)
fmt.Println("my-variables.third:", Document.Myvariables.Third)
fmt.Println("my-variables.B.Bfirst:", Document.Myvariables.B.Bfirst)
// Output:
// age: 45
// fruits: [apple pear]
// my-variables.first: 1
// my-variables.second: 0.2
// my-variables.third: abc
// my-variables.B.Bfirst: 123
```
### Marshaling ### Marshaling
[`Marshal`][marshal] is the opposite of Unmarshal: it represents a Go structure [`Marshal`][marshal] is the opposite of Unmarshal: it represents a Go structure
@@ -131,14 +200,14 @@ as a TOML document:
```go ```go
cfg := MyConfig{ cfg := MyConfig{
Version: 2, Version: 2,
Name: "go-toml", Name: "go-toml",
Tags: []string{"go", "toml"}, Tags: []string{"go", "toml"},
} }
b, err := toml.Marshal(cfg) b, err := toml.Marshal(cfg)
if err != nil { if err != nil {
panic(err) panic(err)
} }
fmt.Println(string(b)) fmt.Println(string(b))
@@ -150,22 +219,33 @@ fmt.Println(string(b))
[marshal]: https://pkg.go.dev/github.com/pelletier/go-toml/v2#Marshal [marshal]: https://pkg.go.dev/github.com/pelletier/go-toml/v2#Marshal
## Unstable API
This API does not yet follow the backward compatibility guarantees of this
library. They provide early access to features that may have rough edges or an
API subject to change.
### Parser
Parser is the unstable API that allows iterative parsing of a TOML document at
the AST level. See https://pkg.go.dev/github.com/pelletier/go-toml/v2/unstable.
## Benchmarks ## Benchmarks
Execution time speedup compared to other Go TOML libraries: Execution time speedup compared to other Go TOML libraries:
<table> <table>
<thead> <thead>
<tr><th>Benchmark</th><th>go-toml v1</th><th>BurntSushi/toml</th></tr> <tr><th>Benchmark</th><th>go-toml v1</th><th>BurntSushi/toml</th></tr>
</thead> </thead>
<tbody> <tbody>
<tr><td>Marshal/HugoFrontMatter-2</td><td>1.9x</td><td>1.9x</td></tr> <tr><td>Marshal/HugoFrontMatter-2</td><td>1.9x</td><td>2.2x</td></tr>
<tr><td>Marshal/ReferenceFile/map-2</td><td>1.7x</td><td>1.9x</td></tr> <tr><td>Marshal/ReferenceFile/map-2</td><td>1.7x</td><td>2.1x</td></tr>
<tr><td>Marshal/ReferenceFile/struct-2</td><td>2.4x</td><td>2.6x</td></tr> <tr><td>Marshal/ReferenceFile/struct-2</td><td>2.2x</td><td>3.0x</td></tr>
<tr><td>Unmarshal/HugoFrontMatter-2</td><td>2.9x</td><td>2.5x</td></tr> <tr><td>Unmarshal/HugoFrontMatter-2</td><td>2.9x</td><td>2.7x</td></tr>
<tr><td>Unmarshal/ReferenceFile/map-2</td><td>2.7x</td><td>2.6x</td></tr> <tr><td>Unmarshal/ReferenceFile/map-2</td><td>2.6x</td><td>2.7x</td></tr>
<tr><td>Unmarshal/ReferenceFile/struct-2</td><td>4.8x</td><td>5.1x</td></tr> <tr><td>Unmarshal/ReferenceFile/struct-2</td><td>4.6x</td><td>5.1x</td></tr>
</tbody> </tbody>
</table> </table>
<details><summary>See more</summary> <details><summary>See more</summary>
<p>The table above has the results of the most common use-cases. The table below <p>The table above has the results of the most common use-cases. The table below
@@ -173,22 +253,22 @@ contains the results of all benchmarks, including unrealistic ones. It is
provided for completeness.</p> provided for completeness.</p>
<table> <table>
<thead> <thead>
<tr><th>Benchmark</th><th>go-toml v1</th><th>BurntSushi/toml</th></tr> <tr><th>Benchmark</th><th>go-toml v1</th><th>BurntSushi/toml</th></tr>
</thead> </thead>
<tbody> <tbody>
<tr><td>Marshal/SimpleDocument/map-2</td><td>1.7x</td><td>2.1x</td></tr> <tr><td>Marshal/SimpleDocument/map-2</td><td>1.8x</td><td>2.7x</td></tr>
<tr><td>Marshal/SimpleDocument/struct-2</td><td>2.5x</td><td>2.8x</td></tr> <tr><td>Marshal/SimpleDocument/struct-2</td><td>2.7x</td><td>3.8x</td></tr>
<tr><td>Unmarshal/SimpleDocument/map-2</td><td>4.1x</td><td>3.1x</td></tr> <tr><td>Unmarshal/SimpleDocument/map-2</td><td>3.8x</td><td>3.0x</td></tr>
<tr><td>Unmarshal/SimpleDocument/struct-2</td><td>6.4x</td><td>4.3x</td></tr> <tr><td>Unmarshal/SimpleDocument/struct-2</td><td>5.6x</td><td>4.1x</td></tr>
<tr><td>UnmarshalDataset/example-2</td><td>3.4x</td><td>3.2x</td></tr> <tr><td>UnmarshalDataset/example-2</td><td>3.0x</td><td>3.2x</td></tr>
<tr><td>UnmarshalDataset/code-2</td><td>2.2x</td><td>2.5x</td></tr> <tr><td>UnmarshalDataset/code-2</td><td>2.3x</td><td>2.9x</td></tr>
<tr><td>UnmarshalDataset/twitter-2</td><td>2.8x</td><td>2.7x</td></tr> <tr><td>UnmarshalDataset/twitter-2</td><td>2.6x</td><td>2.7x</td></tr>
<tr><td>UnmarshalDataset/citm_catalog-2</td><td>2.2x</td><td>2.0x</td></tr> <tr><td>UnmarshalDataset/citm_catalog-2</td><td>2.2x</td><td>2.3x</td></tr>
<tr><td>UnmarshalDataset/canada-2</td><td>1.8x</td><td>1.4x</td></tr> <tr><td>UnmarshalDataset/canada-2</td><td>1.8x</td><td>1.5x</td></tr>
<tr><td>UnmarshalDataset/config-2</td><td>4.4x</td><td>2.9x</td></tr> <tr><td>UnmarshalDataset/config-2</td><td>4.1x</td><td>2.9x</td></tr>
<tr><td>[Geo mean]</td><td>2.8x</td><td>2.6x</td></tr> <tr><td>geomean</td><td>2.7x</td><td>2.8x</td></tr>
</tbody> </tbody>
</table> </table>
<p>This table can be generated with <code>./ci.sh benchmark -a -html</code>.</p> <p>This table can be generated with <code>./ci.sh benchmark -a -html</code>.</p>
</details> </details>
@@ -207,6 +287,44 @@ In case of trouble: [Go Modules FAQ][mod-faq].
[mod-faq]: https://github.com/golang/go/wiki/Modules#why-does-installing-a-tool-via-go-get-fail-with-error-cannot-find-main-module [mod-faq]: https://github.com/golang/go/wiki/Modules#why-does-installing-a-tool-via-go-get-fail-with-error-cannot-find-main-module
## Tools
Go-toml provides three handy command line tools:
* `tomljson`: Reads a TOML file and outputs its JSON representation.
```
$ go install github.com/pelletier/go-toml/v2/cmd/tomljson@latest
$ tomljson --help
```
* `jsontoml`: Reads a JSON file and outputs a TOML representation.
```
$ go install github.com/pelletier/go-toml/v2/cmd/jsontoml@latest
$ jsontoml --help
```
* `tomll`: Lints and reformats a TOML file.
```
$ go install github.com/pelletier/go-toml/v2/cmd/tomll@latest
$ tomll --help
```
### Docker image
Those tools are also available as a [Docker image][docker]. For example, to use
`tomljson`:
```
docker run -i ghcr.io/pelletier/go-toml:v2 tomljson < example.toml
```
Multiple versions are available on [ghcr.io][docker].
[docker]: https://github.com/pelletier/go-toml/pkgs/container/go-toml
## Migrating from v1 ## Migrating from v1
This section describes the differences between v1 and v2, with some pointers on This section describes the differences between v1 and v2, with some pointers on
@@ -235,16 +353,16 @@ element in the interface to decode the object. For example:
```go ```go
type inner struct { type inner struct {
B interface{} B interface{}
} }
type doc struct { type doc struct {
A interface{} A interface{}
} }
d := doc{ d := doc{
A: inner{ A: inner{
B: "Before", B: "Before",
}, },
} }
data := ` data := `
@@ -283,7 +401,7 @@ contained in the doc is superior to the capacity of the array. For example:
```go ```go
type doc struct { type doc struct {
A [2]string A [2]string
} }
d := doc{} d := doc{}
err := toml.Unmarshal([]byte(`A = ["one", "two", "many"]`), &d) err := toml.Unmarshal([]byte(`A = ["one", "two", "many"]`), &d)
@@ -458,27 +576,20 @@ is not necessary anymore.
V1 used to provide multiple struct tags: `comment`, `commented`, `multiline`, V1 used to provide multiple struct tags: `comment`, `commented`, `multiline`,
`toml`, and `omitempty`. To behave more like the standard library, v2 has merged `toml`, and `omitempty`. To behave more like the standard library, v2 has merged
`toml`, `multiline`, and `omitempty`. For example: `toml`, `multiline`, `commented`, and `omitempty`. For example:
```go ```go
type doc struct { type doc struct {
// v1 // v1
F string `toml:"field" multiline:"true" omitempty:"true"` F string `toml:"field" multiline:"true" omitempty:"true" commented:"true"`
// v2 // v2
F string `toml:"field,multiline,omitempty"` F string `toml:"field,multiline,omitempty,commented"`
} }
``` ```
Has a result, the `Encoder.SetTag*` methods have been removed, as there is just Has a result, the `Encoder.SetTag*` methods have been removed, as there is just
one tag now. one tag now.
#### `commented` tag has been removed
There is no replacement for the `commented` tag. This feature would be better
suited in a proper document model for go-toml v2, which has been [cut from
scope][nodoc] at the moment.
#### `Encoder.ArraysWithOneElementPerLine` has been renamed #### `Encoder.ArraysWithOneElementPerLine` has been renamed
The new name is `Encoder.SetArraysMultiline`. The behavior should be the same. The new name is `Encoder.SetArraysMultiline`. The behavior should be the same.
@@ -488,45 +599,38 @@ The new name is `Encoder.SetArraysMultiline`. The behavior should be the same.
The new name is `Encoder.SetIndentSymbol`. The behavior should be the same. The new name is `Encoder.SetIndentSymbol`. The behavior should be the same.
#### Embedded structs are tables #### Embedded structs behave like stdlib
V1 defaults to merging embedded struct fields into the embedding struct. This V1 defaults to merging embedded struct fields into the embedding struct. This
behavior was unexpected because it does not follow the standard library. To behavior was unexpected because it does not follow the standard library. To
avoid breaking backward compatibility, the `Encoder.PromoteAnonymous` method was avoid breaking backward compatibility, the `Encoder.PromoteAnonymous` method was
added to make the encoder behave correctly. Given backward compatibility is not added to make the encoder behave correctly. Given backward compatibility is not
a problem anymore, v2 does the right thing by default. There is no way to revert a problem anymore, v2 does the right thing by default: it follows the behavior
to the old behavior, and `Encoder.PromoteAnonymous` has been removed. of `encoding/json`. `Encoder.PromoteAnonymous` has been removed.
```go
type Embedded struct {
Value string `toml:"value"`
}
type Doc struct {
Embedded
}
d := Doc{}
fmt.Println("v1:")
b, err := v1.Marshal(d)
fmt.Println(string(b))
fmt.Println("v2:")
b, err = v2.Marshal(d)
fmt.Println(string(b))
// Output:
// v1:
// value = ""
//
// v2:
// [Embedded]
// value = ''
```
[nodoc]: https://github.com/pelletier/go-toml/discussions/506#discussioncomment-1526038 [nodoc]: https://github.com/pelletier/go-toml/discussions/506#discussioncomment-1526038
### `query`
go-toml v1 provided the [`go-toml/query`][query] package. It allowed to run
JSONPath-style queries on TOML files. This feature is not available in v2. For a
replacement, check out [dasel][dasel].
This package has been removed because it was essentially not supported anymore
(last commit May 2020), increased the complexity of the code base, and more
complete solutions exist out there.
[query]: https://github.com/pelletier/go-toml/tree/f99d6bbca119636aeafcf351ee52b3d202782627/query
[dasel]: https://github.com/TomWright/dasel
## Versioning
Expect for parts explicitly marked otherwise, go-toml follows [Semantic
Versioning](https://semver.org). The supported version of
[TOML](https://github.com/toml-lang/toml) is indicated at the beginning of this
document. The last two major versions of Go are supported (see [Go Release
Policy](https://golang.org/doc/devel/release.html#policy)).
## License ## License
The MIT License (MIT). Read [LICENSE](LICENSE). The MIT License (MIT). Read [LICENSE](LICENSE).
-3
View File
@@ -2,9 +2,6 @@
## Supported Versions ## Supported Versions
Use this section to tell people about which versions of your project are
currently being supported with security updates.
| Version | Supported | | Version | Supported |
| ---------- | ------------------ | | ---------- | ------------------ |
| Latest 2.x | :white_check_mark: | | Latest 2.x | :white_check_mark: |
+14 -14
View File
@@ -3,16 +3,16 @@ package benchmark_test
import ( import (
"compress/gzip" "compress/gzip"
"encoding/json" "encoding/json"
"io/ioutil" "io"
"os" "os"
"path/filepath" "path/filepath"
"testing" "testing"
"github.com/pelletier/go-toml/v2" "github.com/pelletier/go-toml/v2"
"github.com/stretchr/testify/require" "github.com/pelletier/go-toml/v2/internal/assert"
) )
var bench_inputs = []struct { var benchInputs = []struct {
name string name string
jsonLen int jsonLen int
}{ }{
@@ -30,22 +30,22 @@ var bench_inputs = []struct {
} }
func TestUnmarshalDatasetCode(t *testing.T) { func TestUnmarshalDatasetCode(t *testing.T) {
for _, tc := range bench_inputs { for _, tc := range benchInputs {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
buf := fixture(t, tc.name) buf := fixture(t, tc.name)
var v interface{} var v interface{}
require.NoError(t, toml.Unmarshal(buf, &v)) assert.NoError(t, toml.Unmarshal(buf, &v))
b, err := json.Marshal(v) b, err := json.Marshal(v)
require.NoError(t, err) assert.NoError(t, err)
require.Equal(t, len(b), tc.jsonLen) assert.Equal(t, len(b), tc.jsonLen)
}) })
} }
} }
func BenchmarkUnmarshalDataset(b *testing.B) { func BenchmarkUnmarshalDataset(b *testing.B) {
for _, tc := range bench_inputs { for _, tc := range benchInputs {
b.Run(tc.name, func(b *testing.B) { b.Run(tc.name, func(b *testing.B) {
buf := fixture(b, tc.name) buf := fixture(b, tc.name)
b.SetBytes(int64(len(buf))) b.SetBytes(int64(len(buf)))
@@ -53,7 +53,7 @@ func BenchmarkUnmarshalDataset(b *testing.B) {
b.ResetTimer() b.ResetTimer()
for i := 0; i < b.N; i++ { for i := 0; i < b.N; i++ {
var v interface{} var v interface{}
require.NoError(b, toml.Unmarshal(buf, &v)) assert.NoError(b, toml.Unmarshal(buf, &v))
} }
}) })
} }
@@ -68,13 +68,13 @@ func fixture(tb testing.TB, path string) []byte {
if os.IsNotExist(err) { if os.IsNotExist(err) {
tb.Skip("benchmark fixture not found:", file) tb.Skip("benchmark fixture not found:", file)
} }
require.NoError(tb, err) assert.NoError(tb, err)
defer f.Close() defer func() { _ = f.Close() }()
gz, err := gzip.NewReader(f) gz, err := gzip.NewReader(f)
require.NoError(tb, err) assert.NoError(tb, err)
buf, err := ioutil.ReadAll(gz) buf, err := io.ReadAll(gz)
require.NoError(tb, err) assert.NoError(tb, err)
return buf return buf
} }
+24 -24
View File
@@ -2,12 +2,12 @@ package benchmark_test
import ( import (
"bytes" "bytes"
"io/ioutil" "os"
"testing" "testing"
"time" "time"
"github.com/pelletier/go-toml/v2" "github.com/pelletier/go-toml/v2"
"github.com/stretchr/testify/require" "github.com/pelletier/go-toml/v2/internal/assert"
) )
func TestUnmarshalSimple(t *testing.T) { func TestUnmarshalSimple(t *testing.T) {
@@ -18,7 +18,7 @@ func TestUnmarshalSimple(t *testing.T) {
err := toml.Unmarshal(doc, &d) err := toml.Unmarshal(doc, &d)
if err != nil { if err != nil {
panic(err) t.Error(err)
} }
} }
@@ -38,7 +38,7 @@ func BenchmarkUnmarshal(b *testing.B) {
err := toml.Unmarshal(doc, &d) err := toml.Unmarshal(doc, &d)
if err != nil { if err != nil {
panic(err) b.Error(err)
} }
} }
}) })
@@ -52,14 +52,14 @@ func BenchmarkUnmarshal(b *testing.B) {
d := map[string]interface{}{} d := map[string]interface{}{}
err := toml.Unmarshal(doc, &d) err := toml.Unmarshal(doc, &d)
if err != nil { if err != nil {
panic(err) b.Error(err)
} }
} }
}) })
}) })
b.Run("ReferenceFile", func(b *testing.B) { b.Run("ReferenceFile", func(b *testing.B) {
bytes, err := ioutil.ReadFile("benchmark.toml") bytes, err := os.ReadFile("benchmark.toml")
if err != nil { if err != nil {
b.Fatal(err) b.Fatal(err)
} }
@@ -72,7 +72,7 @@ func BenchmarkUnmarshal(b *testing.B) {
d := benchmarkDoc{} d := benchmarkDoc{}
err := toml.Unmarshal(bytes, &d) err := toml.Unmarshal(bytes, &d)
if err != nil { if err != nil {
panic(err) b.Error(err)
} }
} }
}) })
@@ -85,7 +85,7 @@ func BenchmarkUnmarshal(b *testing.B) {
d := map[string]interface{}{} d := map[string]interface{}{}
err := toml.Unmarshal(bytes, &d) err := toml.Unmarshal(bytes, &d)
if err != nil { if err != nil {
panic(err) b.Error(err)
} }
} }
}) })
@@ -99,7 +99,7 @@ func BenchmarkUnmarshal(b *testing.B) {
d := map[string]interface{}{} d := map[string]interface{}{}
err := toml.Unmarshal(hugoFrontMatterbytes, &d) err := toml.Unmarshal(hugoFrontMatterbytes, &d)
if err != nil { if err != nil {
panic(err) b.Error(err)
} }
} }
}) })
@@ -123,7 +123,7 @@ func BenchmarkMarshal(b *testing.B) {
err := toml.Unmarshal(doc, &d) err := toml.Unmarshal(doc, &d)
if err != nil { if err != nil {
panic(err) b.Error(err)
} }
b.ReportAllocs() b.ReportAllocs()
@@ -134,7 +134,7 @@ func BenchmarkMarshal(b *testing.B) {
for i := 0; i < b.N; i++ { for i := 0; i < b.N; i++ {
out, err = marshal(d) out, err = marshal(d)
if err != nil { if err != nil {
panic(err) b.Error(err)
} }
} }
@@ -145,7 +145,7 @@ func BenchmarkMarshal(b *testing.B) {
d := map[string]interface{}{} d := map[string]interface{}{}
err := toml.Unmarshal(doc, &d) err := toml.Unmarshal(doc, &d)
if err != nil { if err != nil {
panic(err) b.Error(err)
} }
b.ReportAllocs() b.ReportAllocs()
@@ -156,7 +156,7 @@ func BenchmarkMarshal(b *testing.B) {
for i := 0; i < b.N; i++ { for i := 0; i < b.N; i++ {
out, err = marshal(d) out, err = marshal(d)
if err != nil { if err != nil {
panic(err) b.Error(err)
} }
} }
@@ -165,7 +165,7 @@ func BenchmarkMarshal(b *testing.B) {
}) })
b.Run("ReferenceFile", func(b *testing.B) { b.Run("ReferenceFile", func(b *testing.B) {
bytes, err := ioutil.ReadFile("benchmark.toml") bytes, err := os.ReadFile("benchmark.toml")
if err != nil { if err != nil {
b.Fatal(err) b.Fatal(err)
} }
@@ -174,7 +174,7 @@ func BenchmarkMarshal(b *testing.B) {
d := benchmarkDoc{} d := benchmarkDoc{}
err := toml.Unmarshal(bytes, &d) err := toml.Unmarshal(bytes, &d)
if err != nil { if err != nil {
panic(err) b.Error(err)
} }
b.ReportAllocs() b.ReportAllocs()
b.ResetTimer() b.ResetTimer()
@@ -184,7 +184,7 @@ func BenchmarkMarshal(b *testing.B) {
for i := 0; i < b.N; i++ { for i := 0; i < b.N; i++ {
out, err = marshal(d) out, err = marshal(d)
if err != nil { if err != nil {
panic(err) b.Error(err)
} }
} }
@@ -195,7 +195,7 @@ func BenchmarkMarshal(b *testing.B) {
d := map[string]interface{}{} d := map[string]interface{}{}
err := toml.Unmarshal(bytes, &d) err := toml.Unmarshal(bytes, &d)
if err != nil { if err != nil {
panic(err) b.Error(err)
} }
b.ReportAllocs() b.ReportAllocs()
@@ -205,7 +205,7 @@ func BenchmarkMarshal(b *testing.B) {
for i := 0; i < b.N; i++ { for i := 0; i < b.N; i++ {
out, err = marshal(d) out, err = marshal(d)
if err != nil { if err != nil {
panic(err) b.Error(err)
} }
} }
@@ -217,7 +217,7 @@ func BenchmarkMarshal(b *testing.B) {
d := map[string]interface{}{} d := map[string]interface{}{}
err := toml.Unmarshal(hugoFrontMatterbytes, &d) err := toml.Unmarshal(hugoFrontMatterbytes, &d)
if err != nil { if err != nil {
panic(err) b.Error(err)
} }
b.ReportAllocs() b.ReportAllocs()
@@ -228,7 +228,7 @@ func BenchmarkMarshal(b *testing.B) {
for i := 0; i < b.N; i++ { for i := 0; i < b.N; i++ {
out, err = marshal(d) out, err = marshal(d)
if err != nil { if err != nil {
panic(err) b.Error(err)
} }
} }
@@ -344,11 +344,11 @@ type benchmarkDoc struct {
} }
func TestUnmarshalReferenceFile(t *testing.T) { func TestUnmarshalReferenceFile(t *testing.T) {
bytes, err := ioutil.ReadFile("benchmark.toml") bytes, err := os.ReadFile("benchmark.toml")
require.NoError(t, err) assert.NoError(t, err)
d := benchmarkDoc{} d := benchmarkDoc{}
err = toml.Unmarshal(bytes, &d) err = toml.Unmarshal(bytes, &d)
require.NoError(t, err) assert.NoError(t, err)
expected := benchmarkDoc{ expected := benchmarkDoc{
Table: struct { Table: struct {
@@ -627,7 +627,7 @@ trimmed in raw strings.
}, },
} }
require.Equal(t, expected, d) assert.Equal(t, expected, d)
} }
var hugoFrontMatterbytes = []byte(` var hugoFrontMatterbytes = []byte(`
+26 -13
View File
@@ -76,8 +76,10 @@ cover() {
fi fi
pushd "$dir" pushd "$dir"
go test -covermode=atomic -coverprofile=coverage.out ./... go test -covermode=atomic -coverpkg=./... -coverprofile=coverage.out.tmp ./...
grep -Ev '(fuzz|testsuite|tomltestgen|gotoml-test-decoder|gotoml-test-encoder)' coverage.out.tmp > coverage.out
go tool cover -func=coverage.out go tool cover -func=coverage.out
echo "Coverage profile for ${branch}: ${dir}/coverage.out" >&2
popd popd
if [ "${branch}" != "HEAD" ]; then if [ "${branch}" != "HEAD" ]; then
@@ -103,16 +105,23 @@ coverage() {
echo "" echo ""
target_pct="$(cat ${target_out} |sed -E 's/.*total.*\t([0-9.]+)%/\1/;t;d')" target_pct="$(tail -n2 ${target_out} | head -n1 | sed -E 's/.*total.*\t([0-9.]+)%.*/\1/')"
head_pct="$(cat ${head_out} |sed -E 's/.*total.*\t([0-9.]+)%/\1/;t;d')" head_pct="$(tail -n2 ${head_out} | head -n1 | sed -E 's/.*total.*\t([0-9.]+)%/\1/')"
echo "Results: ${target} ${target_pct}% HEAD ${head_pct}%" echo "Results: ${target} ${target_pct}% HEAD ${head_pct}%"
delta_pct=$(echo "$head_pct - $target_pct" | bc -l) delta_pct=$(echo "$head_pct - $target_pct" | bc -l)
echo "Delta: ${delta_pct}" echo "Delta: ${delta_pct}"
if [[ $delta_pct = \-* ]]; then if [[ $delta_pct = \-* ]]; then
echo "Regression!"; echo "Regression!";
return 1
target_diff="${output_dir}/target.diff.txt"
head_diff="${output_dir}/head.diff.txt"
cat "${target_out}" | grep -E '^github.com/pelletier/go-toml' | tr -s "\t " | cut -f 2,3 | sort > "${target_diff}"
cat "${head_out}" | grep -E '^github.com/pelletier/go-toml' | tr -s "\t " | cut -f 2,3 | sort > "${head_diff}"
diff --side-by-side --suppress-common-lines "${target_diff}" "${head_diff}"
return 1
fi fi
return 0 return 0
;; ;;
@@ -143,7 +152,7 @@ bench() {
fi fi
export GOMAXPROCS=2 export GOMAXPROCS=2
nice -n -19 taskset --cpu-list 0,1 go test '-bench=^Benchmark(Un)?[mM]arshal' -count=5 -run=Nothing ./... | tee "${out}" go test '-bench=^Benchmark(Un)?[mM]arshal' -count=10 -run=Nothing ./... | tee "${out}"
popd popd
if [ "${branch}" != "HEAD" ]; then if [ "${branch}" != "HEAD" ]; then
@@ -152,10 +161,12 @@ bench() {
} }
fmktemp() { fmktemp() {
if mktemp --version|grep GNU >/dev/null; then if mktemp --version &> /dev/null; then
mktemp --suffix=-$1; # GNU
mktemp --suffix=-$1
else else
mktemp -t $1; # BSD
mktemp -t $1
fi fi
} }
@@ -175,12 +186,14 @@ with open(sys.argv[1]) as f:
lines.append(line.split(',')) lines.append(line.split(','))
results = [] results = []
for line in reversed(lines[1:]): for line in reversed(lines[2:]):
if len(line) < 8 or line[0] == "":
continue
v2 = float(line[1]) v2 = float(line[1])
results.append([ results.append([
line[0].replace("-32", ""), line[0].replace("-32", ""),
"%.1fx" % (float(line[3])/v2), # v1 "%.1fx" % (float(line[3])/v2), # v1
"%.1fx" % (float(line[5])/v2), # bs "%.1fx" % (float(line[7])/v2), # bs
]) ])
# move geomean to the end # move geomean to the end
results.append(results[0]) results.append(results[0])
@@ -251,10 +264,10 @@ benchmark() {
if [ "$1" = "-html" ]; then if [ "$1" = "-html" ]; then
tmpcsv=`fmktemp csv` tmpcsv=`fmktemp csv`
benchstat -csv -geomean go-toml-v2.txt go-toml-v1.txt bs-toml.txt > $tmpcsv benchstat -format csv go-toml-v2.txt go-toml-v1.txt bs-toml.txt > $tmpcsv
benchstathtml $tmpcsv benchstathtml $tmpcsv
else else
benchstat -geomean go-toml-v2.txt go-toml-v1.txt bs-toml.txt benchstat go-toml-v2.txt go-toml-v1.txt bs-toml.txt
fi fi
rm -f go-toml-v2.txt go-toml-v1.txt bs-toml.txt rm -f go-toml-v2.txt go-toml-v1.txt bs-toml.txt
+2 -1
View File
@@ -1,3 +1,4 @@
// Package gotoml-test-decoder is a minimal decoder program used to compare this library with other TOML implementations.
package main package main
import ( import (
@@ -6,7 +7,7 @@ import (
"os" "os"
"path" "path"
"github.com/pelletier/go-toml/v2/testsuite" "github.com/pelletier/go-toml/v2/internal/testsuite"
) )
func main() { func main() {
+31
View File
@@ -0,0 +1,31 @@
// Package gotoml-test-encoder is a minimal encoder program used to compare this library with other TOML implementations.
package main
import (
"flag"
"log"
"os"
"path"
"github.com/pelletier/go-toml/v2/internal/testsuite"
)
func main() {
log.SetFlags(0)
flag.Usage = usage
flag.Parse()
if flag.NArg() != 0 {
flag.Usage()
}
err := testsuite.EncodeStdin()
if err != nil {
log.Fatal(err)
}
}
func usage() {
log.Printf("Usage: %s < json-file\n", path.Base(os.Args[0]))
flag.PrintDefaults()
os.Exit(1)
}
+66
View File
@@ -0,0 +1,66 @@
// Package jsontoml is a program that converts JSON to TOML.
//
// # Usage
//
// Reading from stdin:
//
// cat file.json | jsontoml > file.toml
//
// Reading from a file:
//
// jsontoml file.json > file.toml
//
// # Installation
//
// Using Go:
//
// go install github.com/pelletier/go-toml/v2/cmd/jsontoml@latest
package main
import (
"encoding/json"
"flag"
"io"
"github.com/pelletier/go-toml/v2"
"github.com/pelletier/go-toml/v2/internal/cli"
)
const usage = `jsontoml can be used in two ways:
Reading from stdin:
cat file.json | jsontoml > file.toml
Reading from a file:
jsontoml file.json > file.toml
`
var useJSONNumber bool
func main() {
flag.BoolVar(&useJSONNumber, "use-json-number", false, "unmarshal numbers into `json.Number` type instead of as `float64`")
p := cli.Program{
Usage: usage,
Fn: convert,
}
p.Execute()
}
func convert(r io.Reader, w io.Writer) error {
var v interface{}
d := json.NewDecoder(r)
e := toml.NewEncoder(w)
if useJSONNumber {
d.UseNumber()
e.SetMarshalJSONNumbers(true)
}
err := d.Decode(&v)
if err != nil {
return err
}
return e.Encode(v)
}
+62
View File
@@ -0,0 +1,62 @@
package main
import (
"bytes"
"strings"
"testing"
"github.com/pelletier/go-toml/v2/internal/assert"
)
func TestConvert(t *testing.T) {
examples := []struct {
name string
input string
expected string
errors bool
useJSONNumber bool
}{
{
name: "valid json",
input: `
{
"mytoml": {
"a": 42
}
}`,
expected: `[mytoml]
a = 42.0
`,
},
{
name: "use json number",
useJSONNumber: true,
input: `
{
"mytoml": {
"a": 42
}
}`,
expected: `[mytoml]
a = 42
`,
},
{
name: "invalid json",
input: `{ foo`,
errors: true,
},
}
for _, e := range examples {
b := new(bytes.Buffer)
useJSONNumber = e.useJSONNumber
err := convert(strings.NewReader(e.input), b)
if e.errors {
assert.Error(t, err)
} else {
assert.NoError(t, err)
assert.Equal(t, e.expected, b.String())
}
}
}
+63
View File
@@ -0,0 +1,63 @@
// Package tomljson is a program that converts TOML to JSON.
//
// # Usage
//
// Reading from stdin:
//
// cat file.toml | tomljson > file.json
//
// Reading from a file:
//
// tomljson file.toml > file.json
//
// # Installation
//
// Using Go:
//
// go install github.com/pelletier/go-toml/v2/cmd/tomljson@latest
package main
import (
"encoding/json"
"errors"
"fmt"
"io"
"github.com/pelletier/go-toml/v2"
"github.com/pelletier/go-toml/v2/internal/cli"
)
const usage = `tomljson can be used in two ways:
Reading from stdin:
cat file.toml | tomljson > file.json
Reading from a file:
tomljson file.toml > file.json
`
func main() {
p := cli.Program{
Usage: usage,
Fn: convert,
}
p.Execute()
}
func convert(r io.Reader, w io.Writer) error {
var v interface{}
d := toml.NewDecoder(r)
err := d.Decode(&v)
if err != nil {
var derr *toml.DecodeError
if errors.As(err, &derr) {
row, col := derr.Position()
return fmt.Errorf("%s\nerror occurred at row %d column %d", derr.String(), row, col)
}
return err
}
e := json.NewEncoder(w)
e.SetIndent("", " ")
return e.Encode(v)
}
+60
View File
@@ -0,0 +1,60 @@
package main
import (
"bytes"
"errors"
"io"
"strings"
"testing"
"github.com/pelletier/go-toml/v2/internal/assert"
)
func TestConvert(t *testing.T) {
examples := []struct {
name string
input io.Reader
expected string
errors bool
}{
{
name: "valid toml",
input: strings.NewReader(`
[mytoml]
a = 42`),
expected: `{
"mytoml": {
"a": 42
}
}
`,
},
{
name: "invalid toml",
input: strings.NewReader(`bad = []]`),
errors: true,
},
{
name: "bad reader",
input: &badReader{},
errors: true,
},
}
for _, e := range examples {
b := new(bytes.Buffer)
err := convert(e.input, b)
if e.errors {
assert.Error(t, err)
} else {
assert.NoError(t, err)
assert.Equal(t, e.expected, b.String())
}
}
}
type badReader struct{}
func (r *badReader) Read([]byte) (int, error) {
return 0, errors.New("reader failed on purpose")
}
+58
View File
@@ -0,0 +1,58 @@
// Package tomll is a linter program for TOML.
//
// # Usage
//
// Reading from stdin, writing to stdout:
//
// cat file.toml | tomll
//
// Reading and updating a list of files in place:
//
// tomll a.toml b.toml c.toml
//
// # Installation
//
// Using Go:
//
// go install github.com/pelletier/go-toml/v2/cmd/tomll@latest
package main
import (
"io"
"github.com/pelletier/go-toml/v2"
"github.com/pelletier/go-toml/v2/internal/cli"
)
const usage = `tomll can be used in two ways:
Reading from stdin, writing to stdout:
cat file.toml | tomll > file.toml
Reading and updating a list of files in place:
tomll a.toml b.toml c.toml
When given a list of files, tomll will modify all files in place without asking.
`
func main() {
p := cli.Program{
Usage: usage,
Fn: convert,
Inplace: true,
}
p.Execute()
}
func convert(r io.Reader, w io.Writer) error {
var v interface{}
d := toml.NewDecoder(r)
err := d.Decode(&v)
if err != nil {
return err
}
e := toml.NewEncoder(w)
return e.Encode(v)
}
+44
View File
@@ -0,0 +1,44 @@
package main
import (
"bytes"
"strings"
"testing"
"github.com/pelletier/go-toml/v2/internal/assert"
)
func TestConvert(t *testing.T) {
examples := []struct {
name string
input string
expected string
errors bool
}{
{
name: "valid toml",
input: `
mytoml.a = 42.0
`,
expected: `[mytoml]
a = 42.0
`,
},
{
name: "invalid toml",
input: `[what`,
errors: true,
},
}
for _, e := range examples {
b := new(bytes.Buffer)
err := convert(strings.NewReader(e.input), b)
if e.errors {
assert.Error(t, err)
} else {
assert.NoError(t, err)
assert.Equal(t, e.expected, b.String())
}
}
}
+62 -102
View File
@@ -3,25 +3,22 @@
// //
// Within the go-toml package, run `go generate`. Otherwise, use: // Within the go-toml package, run `go generate`. Otherwise, use:
// //
// go run github.com/pelletier/go-toml/cmd/tomltestgen -o toml_testgen_test.go // go run github.com/pelletier/go-toml/cmd/tomltestgen -o toml_testgen_test.go
package main package main
import ( import (
"archive/zip"
"bytes" "bytes"
"flag" "flag"
"fmt" "fmt"
"go/format" "go/format"
"io"
"io/ioutil"
"log" "log"
"net/http"
"os" "os"
"regexp" "path/filepath"
"strconv" "strconv"
"strings" "strings"
"text/template" "text/template"
"time" "time"
"unicode"
) )
type invalid struct { type invalid struct {
@@ -32,7 +29,7 @@ type invalid struct {
type valid struct { type valid struct {
Name string Name string
Input string Input string
JsonRef string JSONRef string
} }
type testsCollection struct { type testsCollection struct {
@@ -43,12 +40,11 @@ type testsCollection struct {
Count int Count int
} }
const srcTemplate = "// Generated by tomltestgen for toml-test ref {{.Ref}} on {{.Timestamp}}\n" + const srcTemplate = "// Code generated by tomltestgen for toml-test ref {{.Ref}} on {{.Timestamp}}. DO NOT EDIT.\n" +
"package toml_test\n" + "package toml_test\n" +
" import (\n" + " import (\n" +
" \"testing\"\n" + " \"testing\"\n" +
")\n" + ")\n" +
"{{range .Invalid}}\n" + "{{range .Invalid}}\n" +
"func TestTOMLTest_Invalid_{{.Name}}(t *testing.T) {\n" + "func TestTOMLTest_Invalid_{{.Name}}(t *testing.T) {\n" +
" input := {{.Input|gostr}}\n" + " input := {{.Input|gostr}}\n" +
@@ -59,65 +55,31 @@ const srcTemplate = "// Generated by tomltestgen for toml-test ref {{.Ref}} on {
"{{range .Valid}}\n" + "{{range .Valid}}\n" +
"func TestTOMLTest_Valid_{{.Name}}(t *testing.T) {\n" + "func TestTOMLTest_Valid_{{.Name}}(t *testing.T) {\n" +
" input := {{.Input|gostr}}\n" + " input := {{.Input|gostr}}\n" +
" jsonRef := {{.JsonRef|gostr}}\n" + " jsonRef := {{.JSONRef|gostr}}\n" +
" testgenValid(t, input, jsonRef)\n" + " testgenValid(t, input, jsonRef)\n" +
"}\n" + "}\n" +
"{{end}}\n" "{{end}}\n"
func downloadTmpFile(url string) string {
log.Println("starting to download file from", url)
resp, err := http.Get(url)
if err != nil {
panic(err)
}
defer resp.Body.Close()
tmpfile, err := ioutil.TempFile("", "toml-test-*.zip")
if err != nil {
panic(err)
}
defer tmpfile.Close()
copiedLen, err := io.Copy(tmpfile, resp.Body)
if err != nil {
panic(err)
}
if resp.ContentLength > 0 && copiedLen != resp.ContentLength {
panic(fmt.Errorf("copied %d bytes, request body had %d", copiedLen, resp.ContentLength))
}
return tmpfile.Name()
}
func kebabToCamel(kebab string) string { func kebabToCamel(kebab string) string {
camel := "" var buf strings.Builder
nextUpper := true nextUpper := true
for _, c := range kebab { for _, c := range kebab {
if nextUpper { if nextUpper {
camel += strings.ToUpper(string(c)) buf.WriteRune(unicode.ToUpper(c))
nextUpper = false nextUpper = false
} else if c == '-' {
nextUpper = true
} else if c == '/' {
nextUpper = true
camel += "_"
} else { } else {
camel += string(c) switch c {
case '-':
nextUpper = true
case '/':
nextUpper = true
buf.WriteByte('_')
default:
buf.WriteRune(c)
}
} }
} }
return camel return buf.String()
}
func readFileFromZip(f *zip.File) string {
reader, err := f.Open()
if err != nil {
panic(err)
}
defer reader.Close()
bytes, err := ioutil.ReadAll(reader)
if err != nil {
panic(err)
}
return string(bytes)
} }
func templateGoStr(input string) string { func templateGoStr(input string) string {
@@ -138,61 +100,59 @@ func main() {
flag.Usage = usage flag.Usage = usage
flag.Parse() flag.Parse()
url := "https://codeload.github.com/BurntSushi/toml-test/zip/" + *ref
resultFile := downloadTmpFile(url)
defer os.Remove(resultFile)
log.Println("file written to", resultFile)
zipReader, err := zip.OpenReader(resultFile)
if err != nil {
panic(err)
}
defer zipReader.Close()
collection := testsCollection{ collection := testsCollection{
Ref: *ref, Ref: *ref,
Timestamp: time.Now().Format(time.RFC3339), Timestamp: time.Now().Format(time.RFC3339),
} }
zipFilesMap := map[string]*zip.File{} dirContent, _ := filepath.Glob("tests/invalid/**/*.toml")
for _, f := range dirContent {
filename := strings.TrimPrefix(f, "tests/valid/")
name := kebabToCamel(strings.TrimSuffix(filename, ".toml"))
name = strings.ReplaceAll(name, ".", "_")
for _, f := range zipReader.File { log.Printf("> [%s] %s\n", "invalid", name)
zipFilesMap[f.Name] = f
tomlContent, err := os.ReadFile(f) // #nosec G304
if err != nil {
fmt.Printf("failed to read test file: %s\n", err)
os.Exit(1)
}
collection.Invalid = append(collection.Invalid, invalid{
Name: name,
Input: string(tomlContent),
})
collection.Count++
} }
testFileRegexp := regexp.MustCompile(`([^/]+/tests/(valid|invalid)/(.+))\.(toml)`) dirContent, _ = filepath.Glob("tests/valid/**/*.toml")
for _, f := range zipReader.File { for _, f := range dirContent {
groups := testFileRegexp.FindStringSubmatch(f.Name) filename := strings.TrimPrefix(f, "tests/valid/")
if len(groups) > 0 { name := kebabToCamel(strings.TrimSuffix(filename, ".toml"))
name := kebabToCamel(groups[3]) name = strings.ReplaceAll(name, ".", "_")
testType := groups[2]
log.Printf("> [%s] %s\n", testType, name) log.Printf("> [%s] %s\n", "valid", name)
tomlContent := readFileFromZip(f) tomlContent, err := os.ReadFile(f) // #nosec G304
if err != nil {
switch testType { fmt.Printf("failed reading test file: %s\n", err)
case "invalid": os.Exit(1)
collection.Invalid = append(collection.Invalid, invalid{
Name: name,
Input: tomlContent,
})
collection.Count++
case "valid":
baseFilePath := groups[1]
jsonFilePath := baseFilePath + ".json"
jsonContent := readFileFromZip(zipFilesMap[jsonFilePath])
collection.Valid = append(collection.Valid, valid{
Name: name,
Input: tomlContent,
JsonRef: jsonContent,
})
collection.Count++
default:
panic(fmt.Sprintf("unknown test type: %s", testType))
}
} }
filename = strings.TrimSuffix(f, ".toml")
jsonContent, err := os.ReadFile(filename + ".json") // #nosec G304
if err != nil {
fmt.Printf("failed reading validation json: %s\n", err)
os.Exit(1)
}
collection.Valid = append(collection.Valid, valid{
Name: name,
Input: string(tomlContent),
JSONRef: string(jsonContent),
})
collection.Count++
} }
log.Printf("Collected %d tests from toml-test\n", collection.Count) log.Printf("Collected %d tests from toml-test\n", collection.Count)
@@ -202,7 +162,7 @@ func main() {
} }
t := template.Must(template.New("src").Funcs(funcMap).Parse(srcTemplate)) t := template.Must(template.New("src").Funcs(funcMap).Parse(srcTemplate))
buf := new(bytes.Buffer) buf := new(bytes.Buffer)
err = t.Execute(buf, collection) err := t.Execute(buf, collection)
if err != nil { if err != nil {
panic(err) panic(err)
} }
@@ -216,7 +176,7 @@ func main() {
return return
} }
err = os.WriteFile(*out, outputBytes, 0644) err = os.WriteFile(*out, outputBytes, 0o600)
if err != nil { if err != nil {
panic(err) panic(err)
} }
+71 -57
View File
@@ -5,6 +5,8 @@ import (
"math" "math"
"strconv" "strconv"
"time" "time"
"github.com/pelletier/go-toml/v2/unstable"
) )
func parseInteger(b []byte) (int64, error) { func parseInteger(b []byte) (int64, error) {
@@ -32,7 +34,7 @@ func parseLocalDate(b []byte) (LocalDate, error) {
var date LocalDate var date LocalDate
if len(b) != 10 || b[4] != '-' || b[7] != '-' { if len(b) != 10 || b[4] != '-' || b[7] != '-' {
return date, newDecodeError(b, "dates are expected to have the format YYYY-MM-DD") return date, unstable.NewParserError(b, "dates are expected to have the format YYYY-MM-DD")
} }
var err error var err error
@@ -53,7 +55,7 @@ func parseLocalDate(b []byte) (LocalDate, error) {
} }
if !isValidDate(date.Year, date.Month, date.Day) { if !isValidDate(date.Year, date.Month, date.Day) {
return LocalDate{}, newDecodeError(b, "impossible date") return LocalDate{}, unstable.NewParserError(b, "impossible date")
} }
return date, nil return date, nil
@@ -64,7 +66,7 @@ func parseDecimalDigits(b []byte) (int, error) {
for i, c := range b { for i, c := range b {
if c < '0' || c > '9' { if c < '0' || c > '9' {
return 0, newDecodeError(b[i:i+1], "expected digit (0-9)") return 0, unstable.NewParserError(b[i:i+1], "expected digit (0-9)")
} }
v *= 10 v *= 10
v += int(c - '0') v += int(c - '0')
@@ -97,7 +99,7 @@ func parseDateTime(b []byte) (time.Time, error) {
} else { } else {
const dateTimeByteLen = 6 const dateTimeByteLen = 6
if len(b) != dateTimeByteLen { if len(b) != dateTimeByteLen {
return time.Time{}, newDecodeError(b, "invalid date-time timezone") return time.Time{}, unstable.NewParserError(b, "invalid date-time timezone")
} }
var direction int var direction int
switch b[0] { switch b[0] {
@@ -106,11 +108,11 @@ func parseDateTime(b []byte) (time.Time, error) {
case '+': case '+':
direction = +1 direction = +1
default: default:
return time.Time{}, newDecodeError(b[:1], "invalid timezone offset character") return time.Time{}, unstable.NewParserError(b[:1], "invalid timezone offset character")
} }
if b[3] != ':' { if b[3] != ':' {
return time.Time{}, newDecodeError(b[3:4], "expected a : separator") return time.Time{}, unstable.NewParserError(b[3:4], "expected a : separator")
} }
hours, err := parseDecimalDigits(b[1:3]) hours, err := parseDecimalDigits(b[1:3])
@@ -118,7 +120,7 @@ func parseDateTime(b []byte) (time.Time, error) {
return time.Time{}, err return time.Time{}, err
} }
if hours > 23 { if hours > 23 {
return time.Time{}, newDecodeError(b[:1], "invalid timezone offset hours") return time.Time{}, unstable.NewParserError(b[:1], "invalid timezone offset hours")
} }
minutes, err := parseDecimalDigits(b[4:6]) minutes, err := parseDecimalDigits(b[4:6])
@@ -126,16 +128,20 @@ func parseDateTime(b []byte) (time.Time, error) {
return time.Time{}, err return time.Time{}, err
} }
if minutes > 59 { if minutes > 59 {
return time.Time{}, newDecodeError(b[:1], "invalid timezone offset minutes") return time.Time{}, unstable.NewParserError(b[:1], "invalid timezone offset minutes")
} }
seconds := direction * (hours*3600 + minutes*60) seconds := direction * (hours*3600 + minutes*60)
zone = time.FixedZone("", seconds) if seconds == 0 {
zone = time.UTC
} else {
zone = time.FixedZone("", seconds)
}
b = b[dateTimeByteLen:] b = b[dateTimeByteLen:]
} }
if len(b) > 0 { if len(b) > 0 {
return time.Time{}, newDecodeError(b, "extra bytes at the end of the timezone") return time.Time{}, unstable.NewParserError(b, "extra bytes at the end of the timezone")
} }
t := time.Date( t := time.Date(
@@ -156,7 +162,7 @@ func parseLocalDateTime(b []byte) (LocalDateTime, []byte, error) {
const localDateTimeByteMinLen = 11 const localDateTimeByteMinLen = 11
if len(b) < localDateTimeByteMinLen { if len(b) < localDateTimeByteMinLen {
return dt, nil, newDecodeError(b, "local datetimes are expected to have the format YYYY-MM-DDTHH:MM:SS[.NNNNNNNNN]") return dt, nil, unstable.NewParserError(b, "local datetimes are expected to have the format YYYY-MM-DDTHH:MM[:SS[.NNNNNNNNN]]")
} }
date, err := parseLocalDate(b[:10]) date, err := parseLocalDate(b[:10])
@@ -167,7 +173,7 @@ func parseLocalDateTime(b []byte) (LocalDateTime, []byte, error) {
sep := b[10] sep := b[10]
if sep != 'T' && sep != ' ' && sep != 't' { if sep != 'T' && sep != ' ' && sep != 't' {
return dt, nil, newDecodeError(b[10:11], "datetime separator is expected to be T or a space") return dt, nil, unstable.NewParserError(b[10:11], "datetime separator is expected to be T or a space")
} }
t, rest, err := parseLocalTime(b[11:]) t, rest, err := parseLocalTime(b[11:])
@@ -188,10 +194,10 @@ func parseLocalTime(b []byte) (LocalTime, []byte, error) {
t LocalTime t LocalTime
) )
// check if b matches to have expected format HH:MM:SS[.NNNNNN] // check if b matches to have expected format HH:MM[:SS[.NNNNNN]]
const localTimeByteLen = 8 const localTimeByteMinLen = 5
if len(b) < localTimeByteLen { if len(b) < localTimeByteMinLen {
return t, nil, newDecodeError(b, "times are expected to have the format HH:MM:SS[.NNNNNN]") return t, nil, unstable.NewParserError(b, "times are expected to have the format HH:MM[:SS[.NNNNNN]]")
} }
var err error var err error
@@ -202,10 +208,10 @@ func parseLocalTime(b []byte) (LocalTime, []byte, error) {
} }
if t.Hour > 23 { if t.Hour > 23 {
return t, nil, newDecodeError(b[0:2], "hour cannot be greater 23") return t, nil, unstable.NewParserError(b[0:2], "hour cannot be greater 23")
} }
if b[2] != ':' { if b[2] != ':' {
return t, nil, newDecodeError(b[2:3], "expecting colon between hours and minutes") return t, nil, unstable.NewParserError(b[2:3], "expecting colon between hours and minutes")
} }
t.Minute, err = parseDecimalDigits(b[3:5]) t.Minute, err = parseDecimalDigits(b[3:5])
@@ -213,22 +219,27 @@ func parseLocalTime(b []byte) (LocalTime, []byte, error) {
return t, nil, err return t, nil, err
} }
if t.Minute > 59 { if t.Minute > 59 {
return t, nil, newDecodeError(b[3:5], "minutes cannot be greater 59") return t, nil, unstable.NewParserError(b[3:5], "minutes cannot be greater 59")
}
if b[5] != ':' {
return t, nil, newDecodeError(b[5:6], "expecting colon between minutes and seconds")
} }
t.Second, err = parseDecimalDigits(b[6:8]) b = b[5:]
if err != nil {
return t, nil, err
}
if t.Second > 60 { if len(b) >= 1 && b[0] == ':' {
return t, nil, newDecodeError(b[6:8], "seconds cannot be greater 60") if len(b) < 3 {
} return t, nil, unstable.NewParserError(b, "incomplete seconds")
}
b = b[8:] t.Second, err = parseDecimalDigits(b[1:3])
if err != nil {
return t, nil, err
}
if t.Second > 59 {
return t, nil, unstable.NewParserError(b[1:3], "seconds cannot be greater than 59")
}
b = b[3:]
}
if len(b) >= 1 && b[0] == '.' { if len(b) >= 1 && b[0] == '.' {
frac := 0 frac := 0
@@ -238,7 +249,7 @@ func parseLocalTime(b []byte) (LocalTime, []byte, error) {
for i, c := range b[1:] { for i, c := range b[1:] {
if !isDigit(c) { if !isDigit(c) {
if i == 0 { if i == 0 {
return t, nil, newDecodeError(b[0:1], "need at least one digit after fraction point") return t, nil, unstable.NewParserError(b[0:1], "need at least one digit after fraction point")
} }
break break
} }
@@ -262,7 +273,7 @@ func parseLocalTime(b []byte) (LocalTime, []byte, error) {
} }
if precision == 0 { if precision == 0 {
return t, nil, newDecodeError(b[:1], "nanoseconds need at least one digit") return t, nil, unstable.NewParserError(b[:1], "nanoseconds need at least one digit")
} }
t.Nanosecond = frac * nspow[precision] t.Nanosecond = frac * nspow[precision]
@@ -273,7 +284,6 @@ func parseLocalTime(b []byte) (LocalTime, []byte, error) {
return t, b, nil return t, b, nil
} }
//nolint:cyclop
func parseFloat(b []byte) (float64, error) { func parseFloat(b []byte) (float64, error) {
if len(b) == 4 && (b[0] == '+' || b[0] == '-') && b[1] == 'n' && b[2] == 'a' && b[3] == 'n' { if len(b) == 4 && (b[0] == '+' || b[0] == '-') && b[1] == 'n' && b[2] == 'a' && b[3] == 'n' {
return math.NaN(), nil return math.NaN(), nil
@@ -285,40 +295,40 @@ func parseFloat(b []byte) (float64, error) {
} }
if cleaned[0] == '.' { if cleaned[0] == '.' {
return 0, newDecodeError(b, "float cannot start with a dot") return 0, unstable.NewParserError(b, "float cannot start with a dot")
} }
if cleaned[len(cleaned)-1] == '.' { if cleaned[len(cleaned)-1] == '.' {
return 0, newDecodeError(b, "float cannot end with a dot") return 0, unstable.NewParserError(b, "float cannot end with a dot")
} }
dotAlreadySeen := false dotAlreadySeen := false
for i, c := range cleaned { for i, c := range cleaned {
if c == '.' { if c == '.' {
if dotAlreadySeen { if dotAlreadySeen {
return 0, newDecodeError(b[i:i+1], "float can have at most one decimal point") return 0, unstable.NewParserError(b[i:i+1], "float can have at most one decimal point")
} }
if !isDigit(cleaned[i-1]) { if !isDigit(cleaned[i-1]) {
return 0, newDecodeError(b[i-1:i+1], "float decimal point must be preceded by a digit") return 0, unstable.NewParserError(b[i-1:i+1], "float decimal point must be preceded by a digit")
} }
if !isDigit(cleaned[i+1]) { if !isDigit(cleaned[i+1]) {
return 0, newDecodeError(b[i:i+2], "float decimal point must be followed by a digit") return 0, unstable.NewParserError(b[i:i+2], "float decimal point must be followed by a digit")
} }
dotAlreadySeen = true dotAlreadySeen = true
} }
} }
start := 0 start := 0
if b[0] == '+' || b[0] == '-' { if cleaned[0] == '+' || cleaned[0] == '-' {
start = 1 start = 1
} }
if b[start] == '0' && isDigit(b[start+1]) { if cleaned[start] == '0' && len(cleaned) > start+1 && isDigit(cleaned[start+1]) {
return 0, newDecodeError(b, "float integer part cannot have leading zeroes") return 0, unstable.NewParserError(b, "float integer part cannot have leading zeroes")
} }
f, err := strconv.ParseFloat(string(cleaned), 64) f, err := strconv.ParseFloat(string(cleaned), 64)
if err != nil { if err != nil {
return 0, newDecodeError(b, "unable to parse float: %w", err) return 0, unstable.NewParserError(b, "unable to parse float: %w", err)
} }
return f, nil return f, nil
@@ -332,7 +342,7 @@ func parseIntHex(b []byte) (int64, error) {
i, err := strconv.ParseInt(string(cleaned), 16, 64) i, err := strconv.ParseInt(string(cleaned), 16, 64)
if err != nil { if err != nil {
return 0, newDecodeError(b, "couldn't parse hexadecimal number: %w", err) return 0, unstable.NewParserError(b, "couldn't parse hexadecimal number: %w", err)
} }
return i, nil return i, nil
@@ -346,7 +356,7 @@ func parseIntOct(b []byte) (int64, error) {
i, err := strconv.ParseInt(string(cleaned), 8, 64) i, err := strconv.ParseInt(string(cleaned), 8, 64)
if err != nil { if err != nil {
return 0, newDecodeError(b, "couldn't parse octal number: %w", err) return 0, unstable.NewParserError(b, "couldn't parse octal number: %w", err)
} }
return i, nil return i, nil
@@ -360,7 +370,7 @@ func parseIntBin(b []byte) (int64, error) {
i, err := strconv.ParseInt(string(cleaned), 2, 64) i, err := strconv.ParseInt(string(cleaned), 2, 64)
if err != nil { if err != nil {
return 0, newDecodeError(b, "couldn't parse binary number: %w", err) return 0, unstable.NewParserError(b, "couldn't parse binary number: %w", err)
} }
return i, nil return i, nil
@@ -383,12 +393,12 @@ func parseIntDec(b []byte) (int64, error) {
} }
if len(cleaned) > startIdx+1 && cleaned[startIdx] == '0' { if len(cleaned) > startIdx+1 && cleaned[startIdx] == '0' {
return 0, newDecodeError(b, "leading zero not allowed on decimal number") return 0, unstable.NewParserError(b, "leading zero not allowed on decimal number")
} }
i, err := strconv.ParseInt(string(cleaned), 10, 64) i, err := strconv.ParseInt(string(cleaned), 10, 64)
if err != nil { if err != nil {
return 0, newDecodeError(b, "couldn't parse decimal number: %w", err) return 0, unstable.NewParserError(b, "couldn't parse decimal number: %w", err)
} }
return i, nil return i, nil
@@ -405,11 +415,11 @@ func checkAndRemoveUnderscoresIntegers(b []byte) ([]byte, error) {
} }
if b[start] == '_' { if b[start] == '_' {
return nil, newDecodeError(b[start:start+1], "number cannot start with underscore") return nil, unstable.NewParserError(b[start:start+1], "number cannot start with underscore")
} }
if b[len(b)-1] == '_' { if b[len(b)-1] == '_' {
return nil, newDecodeError(b[len(b)-1:], "number cannot end with underscore") return nil, unstable.NewParserError(b[len(b)-1:], "number cannot end with underscore")
} }
// fast path // fast path
@@ -431,7 +441,7 @@ func checkAndRemoveUnderscoresIntegers(b []byte) ([]byte, error) {
c := b[i] c := b[i]
if c == '_' { if c == '_' {
if !before { if !before {
return nil, newDecodeError(b[i-1:i+1], "number must have at least one digit between underscores") return nil, unstable.NewParserError(b[i-1:i+1], "number must have at least one digit between underscores")
} }
before = false before = false
} else { } else {
@@ -445,11 +455,11 @@ func checkAndRemoveUnderscoresIntegers(b []byte) ([]byte, error) {
func checkAndRemoveUnderscoresFloats(b []byte) ([]byte, error) { func checkAndRemoveUnderscoresFloats(b []byte) ([]byte, error) {
if b[0] == '_' { if b[0] == '_' {
return nil, newDecodeError(b[0:1], "number cannot start with underscore") return nil, unstable.NewParserError(b[0:1], "number cannot start with underscore")
} }
if b[len(b)-1] == '_' { if b[len(b)-1] == '_' {
return nil, newDecodeError(b[len(b)-1:], "number cannot end with underscore") return nil, unstable.NewParserError(b[len(b)-1:], "number cannot end with underscore")
} }
// fast path // fast path
@@ -472,10 +482,10 @@ func checkAndRemoveUnderscoresFloats(b []byte) ([]byte, error) {
switch c { switch c {
case '_': case '_':
if !before { if !before {
return nil, newDecodeError(b[i-1:i+1], "number must have at least one digit between underscores") return nil, unstable.NewParserError(b[i-1:i+1], "number must have at least one digit between underscores")
} }
if i < len(b)-1 && (b[i+1] == 'e' || b[i+1] == 'E') { if i < len(b)-1 && (b[i+1] == 'e' || b[i+1] == 'E') {
return nil, newDecodeError(b[i+1:i+2], "cannot have underscore before exponent") return nil, unstable.NewParserError(b[i+1:i+2], "cannot have underscore before exponent")
} }
before = false before = false
case '+', '-': case '+', '-':
@@ -484,15 +494,15 @@ func checkAndRemoveUnderscoresFloats(b []byte) ([]byte, error) {
before = false before = false
case 'e', 'E': case 'e', 'E':
if i < len(b)-1 && b[i+1] == '_' { if i < len(b)-1 && b[i+1] == '_' {
return nil, newDecodeError(b[i+1:i+2], "cannot have underscore after exponent") return nil, unstable.NewParserError(b[i+1:i+2], "cannot have underscore after exponent")
} }
cleaned = append(cleaned, c) cleaned = append(cleaned, c)
case '.': case '.':
if i < len(b)-1 && b[i+1] == '_' { if i < len(b)-1 && b[i+1] == '_' {
return nil, newDecodeError(b[i+1:i+2], "cannot have underscore after decimal point") return nil, unstable.NewParserError(b[i+1:i+2], "cannot have underscore after decimal point")
} }
if i > 0 && b[i-1] == '_' { if i > 0 && b[i-1] == '_' {
return nil, newDecodeError(b[i-1:i], "cannot have underscore before decimal point") return nil, unstable.NewParserError(b[i-1:i], "cannot have underscore before decimal point")
} }
cleaned = append(cleaned, c) cleaned = append(cleaned, c)
default: default:
@@ -538,3 +548,7 @@ func daysIn(m int, year int) int {
func isLeap(year int) bool { func isLeap(year int) bool {
return year%4 == 0 && (year%100 != 0 || year%400 == 0) return year%4 == 0 && (year%100 != 0 || year%400 == 0)
} }
func isDigit(r byte) bool {
return r >= '0' && r <= '9'
}
+42 -28
View File
@@ -2,10 +2,11 @@ package toml
import ( import (
"fmt" "fmt"
"reflect"
"strconv" "strconv"
"strings" "strings"
"github.com/pelletier/go-toml/v2/internal/danger" "github.com/pelletier/go-toml/v2/unstable"
) )
// DecodeError represents an error encountered during the parsing or decoding // DecodeError represents an error encountered during the parsing or decoding
@@ -27,7 +28,7 @@ type DecodeError struct {
// corresponding field in the target value. It contains all the missing fields // corresponding field in the target value. It contains all the missing fields
// in Errors. // in Errors.
// //
// Emitted by Decoder when SetStrict(true) was called. // Emitted by Decoder when DisallowUnknownFields() was called.
type StrictMissingError struct { type StrictMissingError struct {
// One error per field that could not be found. // One error per field that could not be found.
Errors []DecodeError Errors []DecodeError
@@ -53,27 +54,20 @@ func (s *StrictMissingError) String() string {
return buf.String() return buf.String()
} }
type Key []string // Unwrap returns wrapped decode errors
//
// internal version of DecodeError that is used as the base to create a // Implements errors.Join() interface.
// DecodeError with full context. func (s *StrictMissingError) Unwrap() []error {
type decodeError struct { errs := make([]error, len(s.Errors))
highlight []byte for i := range s.Errors {
message string errs[i] = &s.Errors[i]
key Key // optional
}
func (de *decodeError) Error() string {
return de.message
}
func newDecodeError(highlight []byte, format string, args ...interface{}) error {
return &decodeError{
highlight: highlight,
message: fmt.Errorf(format, args...).Error(),
} }
return errs
} }
// Key represents a TOML key as a sequence of key parts.
type Key []string
// Error returns the error message contained in the DecodeError. // Error returns the error message contained in the DecodeError.
func (e *DecodeError) Error() string { func (e *DecodeError) Error() string {
return "toml: " + e.message return "toml: " + e.message
@@ -96,20 +90,21 @@ func (e *DecodeError) Key() Key {
return e.key return e.key
} }
// decodeErrorFromHighlight creates a DecodeError referencing a highlighted // wrapDecodeError creates a DecodeError referencing a highlighted
// range of bytes from document. // range of bytes from document.
// //
// highlight needs to be a sub-slice of document, or this function panics. // highlight needs to be a sub-slice of document, or this function panics.
// //
// The function copies all bytes used in DecodeError, so that document and // The function copies all bytes used in DecodeError, so that document and
// highlight can be freely deallocated. // highlight can be freely deallocated.
//
//nolint:funlen //nolint:funlen
func wrapDecodeError(document []byte, de *decodeError) *DecodeError { func wrapDecodeError(document []byte, de *unstable.ParserError) *DecodeError {
offset := danger.SubsliceOffset(document, de.highlight) offset := subsliceOffset(document, de.Highlight)
errMessage := de.Error() errMessage := de.Error()
errLine, errColumn := positionAtEnd(document[:offset]) errLine, errColumn := positionAtEnd(document[:offset])
before, after := linesOfContext(document, de.highlight, offset, 3) before, after := linesOfContext(document, de.Highlight, offset, 3)
var buf strings.Builder var buf strings.Builder
@@ -139,7 +134,7 @@ func wrapDecodeError(document []byte, de *decodeError) *DecodeError {
buf.Write(before[0]) buf.Write(before[0])
} }
buf.Write(de.highlight) buf.Write(de.Highlight)
if len(after) > 0 { if len(after) > 0 {
buf.Write(after[0]) buf.Write(after[0])
@@ -157,7 +152,7 @@ func wrapDecodeError(document []byte, de *decodeError) *DecodeError {
buf.WriteString(strings.Repeat(" ", len(before[0]))) buf.WriteString(strings.Repeat(" ", len(before[0])))
} }
buf.WriteString(strings.Repeat("~", len(de.highlight))) buf.WriteString(strings.Repeat("~", len(de.Highlight)))
if len(errMessage) > 0 { if len(errMessage) > 0 {
buf.WriteString(" ") buf.WriteString(" ")
@@ -182,7 +177,7 @@ func wrapDecodeError(document []byte, de *decodeError) *DecodeError {
message: errMessage, message: errMessage,
line: errLine, line: errLine,
column: errColumn, column: errColumn,
key: de.key, key: de.Key,
human: buf.String(), human: buf.String(),
} }
} }
@@ -265,5 +260,24 @@ func positionAtEnd(b []byte) (row int, column int) {
} }
} }
return return row, column
}
// subsliceOffset returns the byte offset of subslice within data.
// subslice must share the same backing array as data.
func subsliceOffset(data []byte, subslice []byte) int {
if len(subslice) == 0 {
return 0
}
// Use reflect to get the data pointers of both slices.
// This is safe because we're only reading the pointer values for comparison.
dataPtr := reflect.ValueOf(data).Pointer()
subPtr := reflect.ValueOf(subslice).Pointer()
offset := int(subPtr - dataPtr)
if offset < 0 || offset > len(data) {
panic("subslice is not within data")
}
return offset
} }
+101 -10
View File
@@ -7,12 +7,12 @@ import (
"strings" "strings"
"testing" "testing"
"github.com/stretchr/testify/assert" "github.com/pelletier/go-toml/v2/internal/assert"
"github.com/pelletier/go-toml/v2/unstable"
) )
//nolint:funlen //nolint:funlen
func TestDecodeError(t *testing.T) { func TestDecodeError(t *testing.T) {
examples := []struct { examples := []struct {
desc string desc string
doc [3]string doc [3]string
@@ -160,19 +160,18 @@ line 5`,
for _, e := range examples { for _, e := range examples {
e := e e := e
t.Run(e.desc, func(t *testing.T) { t.Run(e.desc, func(t *testing.T) {
b := bytes.Buffer{} b := bytes.Buffer{}
b.Write([]byte(e.doc[0])) b.WriteString(e.doc[0])
start := b.Len() start := b.Len()
b.Write([]byte(e.doc[1])) b.WriteString(e.doc[1])
end := b.Len() end := b.Len()
b.Write([]byte(e.doc[2])) b.WriteString(e.doc[2])
doc := b.Bytes() doc := b.Bytes()
hl := doc[start:end] hl := doc[start:end]
err := wrapDecodeError(doc, &decodeError{ err := wrapDecodeError(doc, &unstable.ParserError{
highlight: hl, Highlight: hl,
message: e.msg, Message: e.msg,
}) })
var derr *DecodeError var derr *DecodeError
@@ -188,7 +187,6 @@ line 5`,
} }
func TestDecodeError_Accessors(t *testing.T) { func TestDecodeError_Accessors(t *testing.T) {
e := DecodeError{ e := DecodeError{
message: "foo", message: "foo",
line: 1, line: 1,
@@ -204,6 +202,99 @@ func TestDecodeError_Accessors(t *testing.T) {
assert.Equal(t, "bar", e.String()) assert.Equal(t, "bar", e.String())
} }
func TestDecodeError_DuplicateContent(t *testing.T) {
// This test verifies that when the same content appears multiple times
// in the document, the error correctly points to the actual location
// of the error, not the first occurrence of the content.
//
// The document has "1__2" on line 1 and "3__4" on line 2.
// Both have "__" which is invalid, but we want to ensure errors
// on line 2 report line 2, not line 1.
doc := `a = 1
b = 3__4`
var v map[string]int
err := Unmarshal([]byte(doc), &v)
var derr *DecodeError
if !errors.As(err, &derr) {
t.Fatal("error not in expected format")
}
row, col := derr.Position()
// The error should be on line 2 where "3__4" is
if row != 2 {
t.Errorf("expected error on row 2, got row %d", row)
}
// Column should point to the "__" part (after "3")
if col < 5 {
t.Errorf("expected error at column >= 5, got column %d", col)
}
}
func TestDecodeError_Position(t *testing.T) {
// Test that error positions are correctly reported for various error locations
examples := []struct {
name string
doc string
expectedRow int
minCol int
}{
{
name: "error on first line",
doc: `a = 1__2`,
expectedRow: 1,
minCol: 5,
},
{
name: "error on second line",
doc: "a = 1\nb = 2__3",
expectedRow: 2,
minCol: 5,
},
{
name: "error on third line",
doc: "a = 1\nb = 2\nc = 3__4",
expectedRow: 3,
minCol: 5,
},
}
for _, e := range examples {
t.Run(e.name, func(t *testing.T) {
var v map[string]int
err := Unmarshal([]byte(e.doc), &v)
var derr *DecodeError
if !errors.As(err, &derr) {
t.Fatal("error not in expected format")
}
row, col := derr.Position()
assert.Equal(t, e.expectedRow, row)
if col < e.minCol {
t.Errorf("expected column >= %d, got %d", e.minCol, col)
}
})
}
}
func TestStrictErrorUnwrap(t *testing.T) {
fo := bytes.NewBufferString(`
Missing = 1
OtherMissing = 1
`)
var out struct{}
err := NewDecoder(fo).DisallowUnknownFields().Decode(&out)
assert.Error(t, err)
strictErr := &StrictMissingError{}
assert.True(t, errors.As(err, &strictErr))
assert.Equal(t, 2, len(strictErr.Unwrap()))
}
func ExampleDecodeError() { func ExampleDecodeError() {
doc := `name = 123__456` doc := `name = 123__456`
+37
View File
@@ -0,0 +1,37 @@
package toml_test
import (
"fmt"
"log"
"strconv"
"github.com/pelletier/go-toml/v2"
)
type customInt int
func (i *customInt) UnmarshalText(b []byte) error {
x, err := strconv.ParseInt(string(b), 10, 32)
if err != nil {
return err
}
*i = customInt(x * 100)
return nil
}
type doc struct {
Value customInt
}
func ExampleUnmarshal_textUnmarshal() {
var x doc
data := []byte(`value = "42"`)
err := toml.Unmarshal(data, &x)
if err != nil {
log.Fatal(err)
}
fmt.Println(x)
// Output:
// {4200}
}
+21 -14
View File
@@ -4,21 +4,28 @@ import (
"testing" "testing"
"github.com/pelletier/go-toml/v2" "github.com/pelletier/go-toml/v2"
"github.com/stretchr/testify/require" "github.com/pelletier/go-toml/v2/internal/assert"
) )
func TestFastSimple(t *testing.T) { func TestFastSimpleInt(t *testing.T) {
m := map[string]int64{} m := map[string]int64{}
err := toml.Unmarshal([]byte(`a = 42`), &m) err := toml.Unmarshal([]byte(`a = 42`), &m)
require.NoError(t, err) assert.NoError(t, err)
require.Equal(t, map[string]int64{"a": 42}, m) assert.Equal(t, map[string]int64{"a": 42}, m)
}
func TestFastSimpleFloat(t *testing.T) {
m := map[string]float64{}
err := toml.Unmarshal([]byte("a = 42\nb = 1.1\nc = 12341234123412341234123412341234"), &m)
assert.NoError(t, err)
assert.Equal(t, map[string]float64{"a": 42, "b": 1.1, "c": 1.2341234123412342e+31}, m)
} }
func TestFastSimpleString(t *testing.T) { func TestFastSimpleString(t *testing.T) {
m := map[string]string{} m := map[string]string{}
err := toml.Unmarshal([]byte(`a = "hello"`), &m) err := toml.Unmarshal([]byte(`a = "hello"`), &m)
require.NoError(t, err) assert.NoError(t, err)
require.Equal(t, map[string]string{"a": "hello"}, m) assert.Equal(t, map[string]string{"a": "hello"}, m)
} }
func TestFastSimpleInterface(t *testing.T) { func TestFastSimpleInterface(t *testing.T) {
@@ -26,8 +33,8 @@ func TestFastSimpleInterface(t *testing.T) {
err := toml.Unmarshal([]byte(` err := toml.Unmarshal([]byte(`
a = "hello" a = "hello"
b = 42`), &m) b = 42`), &m)
require.NoError(t, err) assert.NoError(t, err)
require.Equal(t, map[string]interface{}{ assert.Equal(t, map[string]interface{}{
"a": "hello", "a": "hello",
"b": int64(42), "b": int64(42),
}, m) }, m)
@@ -39,8 +46,8 @@ func TestFastMultipartKeyInterface(t *testing.T) {
a.interim = "test" a.interim = "test"
a.b.c = "hello" a.b.c = "hello"
b = 42`), &m) b = 42`), &m)
require.NoError(t, err) assert.NoError(t, err)
require.Equal(t, map[string]interface{}{ assert.Equal(t, map[string]interface{}{
"a": map[string]interface{}{ "a": map[string]interface{}{
"interim": "test", "interim": "test",
"b": map[string]interface{}{ "b": map[string]interface{}{
@@ -59,8 +66,8 @@ func TestFastExistingMap(t *testing.T) {
ints.one = 1 ints.one = 1
ints.two = 2 ints.two = 2
strings.yo = "hello"`), &m) strings.yo = "hello"`), &m)
require.NoError(t, err) assert.NoError(t, err)
require.Equal(t, map[string]interface{}{ assert.Equal(t, map[string]interface{}{
"ints": map[string]interface{}{ "ints": map[string]interface{}{
"one": int64(1), "one": int64(1),
"two": int64(2), "two": int64(2),
@@ -83,9 +90,9 @@ func TestFastArrayTable(t *testing.T) {
m := map[string]interface{}{} m := map[string]interface{}{}
err := toml.Unmarshal(b, &m) err := toml.Unmarshal(b, &m)
require.NoError(t, err) assert.NoError(t, err)
require.Equal(t, map[string]interface{}{ assert.Equal(t, map[string]interface{}{
"root": map[string]interface{}{ "root": map[string]interface{}{
"nested": []interface{}{ "nested": []interface{}{
map[string]interface{}{ map[string]interface{}{
+53
View File
@@ -0,0 +1,53 @@
package toml_test
import (
"os"
"strings"
"testing"
"github.com/pelletier/go-toml/v2"
"github.com/pelletier/go-toml/v2/internal/assert"
)
func FuzzUnmarshal(f *testing.F) {
file, err := os.ReadFile("benchmark/benchmark.toml")
if err != nil {
f.Error(err)
}
f.Add(file)
f.Fuzz(func(t *testing.T, b []byte) {
if strings.Contains(string(b), "nan") {
// Current limitation of testify.
// https://github.com/stretchr/testify/issues/624
t.Skip("can't compare NaNs")
}
t.Log("INITIAL DOCUMENT ===========================")
t.Log(string(b))
var v interface{}
err := toml.Unmarshal(b, &v)
if err != nil {
return
}
t.Log("DECODED VALUE ===========================")
t.Logf("%#+v", v)
encoded, err := toml.Marshal(v)
if err != nil {
t.Fatalf("cannot marshal unmarshaled document: %s", err)
}
t.Log("ENCODED DOCUMENT ===========================")
t.Log(string(encoded))
var v2 interface{}
err = toml.Unmarshal(encoded, &v2)
if err != nil {
t.Fatalf("failed round trip: %s", err)
}
assert.Equal(t, v, v2)
})
}
+1 -4
View File
@@ -1,6 +1,3 @@
module github.com/pelletier/go-toml/v2 module github.com/pelletier/go-toml/v2
go 1.16 go 1.21.0
// latest (v1.7.0) doesn't have the fix for time.Time
require github.com/stretchr/testify v1.7.1-0.20210427113832-6241f9ab9942
-11
View File
@@ -1,11 +0,0 @@
github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.7.1-0.20210427113832-6241f9ab9942 h1:t0lM6y/M5IiUZyvbBTcngso8SZEZICH7is9B6g/obVU=
github.com/stretchr/testify v1.7.1-0.20210427113832-6241f9ab9942/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c h1:dUUwHk2QECo/6vqA44rthZ8ie2QXMNeKRTHCNY2nXvo=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+141
View File
@@ -0,0 +1,141 @@
// Package assert provides assertion functions for unit testing.
package assert
import (
"bytes"
"fmt"
"reflect"
"strings"
"testing"
)
// True asserts that an expression is true.
func True(tb testing.TB, ok bool, msgAndArgs ...any) {
tb.Helper()
if ok {
return
}
tb.Fatal(formatMsgAndArgs("Expected expression to be true", msgAndArgs...))
}
// False asserts that an expression is false.
func False(tb testing.TB, ok bool, msgAndArgs ...any) {
tb.Helper()
if !ok {
return
}
tb.Fatal(formatMsgAndArgs("Expected expression to be false", msgAndArgs...))
}
// Equal asserts that "expected" and "actual" are equal.
func Equal[T any](tb testing.TB, expected, actual T, msgAndArgs ...any) {
tb.Helper()
if objectsAreEqual(expected, actual) {
return
}
msg := formatMsgAndArgs("Expected values to be equal:", msgAndArgs...)
tb.Fatalf("%s\n%s", msg, diff(expected, actual))
}
// Error asserts that an error is not nil.
func Error(tb testing.TB, err error, msgAndArgs ...any) {
tb.Helper()
if err != nil {
return
}
tb.Fatal(formatMsgAndArgs("Expected an error", msgAndArgs...))
}
// NoError asserts that an error is nil.
func NoError(tb testing.TB, err error, msgAndArgs ...any) {
tb.Helper()
if err == nil {
return
}
msg := formatMsgAndArgs("Unexpected error:", msgAndArgs...)
tb.Fatalf("%s\n%+v", msg, err)
}
// Panics asserts that the given function panics.
func Panics(tb testing.TB, fn func(), msgAndArgs ...any) {
tb.Helper()
defer func() {
if recover() == nil {
msg := formatMsgAndArgs("Expected function to panic", msgAndArgs...)
tb.Fatal(msg)
}
}()
fn()
}
// Zero asserts that a value is its zero value.
func Zero[T any](tb testing.TB, value T, msgAndArgs ...any) {
tb.Helper()
var zero T
if objectsAreEqual(value, zero) {
return
}
val := reflect.ValueOf(value)
if (val.Kind() == reflect.Slice || val.Kind() == reflect.Map || val.Kind() == reflect.Array) && val.Len() == 0 {
return
}
msg := formatMsgAndArgs("Expected zero value but got:", msgAndArgs...)
tb.Fatalf("%s\n%v", msg, value)
}
func NotZero[T any](tb testing.TB, value T, msgAndArgs ...any) {
tb.Helper()
var zero T
if !objectsAreEqual(value, zero) {
val := reflect.ValueOf(value)
switch val.Kind() {
case reflect.Slice, reflect.Map, reflect.Array:
if val.Len() > 0 {
return
}
default:
return
}
}
msg := formatMsgAndArgs("Unexpected zero value:", msgAndArgs...)
tb.Fatalf("%s\n%v", msg, value)
}
func formatMsgAndArgs(msg string, args ...any) string {
if len(args) == 0 {
return msg
}
format, ok := args[0].(string)
if !ok {
panic("message argument must be a fmt string")
}
return fmt.Sprintf(format, args[1:]...)
}
func diff(expected, actual any) string {
lines := []string{
"expected:",
fmt.Sprintf("%v", expected),
"actual:",
fmt.Sprintf("%v", actual),
}
return strings.Join(lines, "\n")
}
func objectsAreEqual(expected, actual any) bool {
if expected == nil || actual == nil {
return expected == actual
}
if exp, eok := expected.([]byte); eok {
if act, aok := actual.([]byte); aok {
return bytes.Equal(exp, act)
}
}
if exp, eok := expected.(string); eok {
if act, aok := actual.(string); aok {
return exp == act
}
}
return reflect.DeepEqual(expected, actual)
}
+217
View File
@@ -0,0 +1,217 @@
package assert
import (
"errors"
"fmt"
"testing"
)
type Data struct {
Label string
Value int64
}
func TestBadMessage(t *testing.T) {
invalidMessage := func() { True(t, false, 1234) }
assertOk(t, "Non-fmt message value", func(tb testing.TB) {
tb.Helper()
Panics(tb, invalidMessage)
})
assertFail(t, "Non-fmt message value", func(tb testing.TB) {
tb.Helper()
True(tb, false, "example %s", "message")
})
}
func TestTrue(t *testing.T) {
assertOk(t, "Succeed", func(tb testing.TB) {
tb.Helper()
True(tb, 1 > 0)
})
assertFail(t, "Fail", func(tb testing.TB) {
tb.Helper()
True(tb, 1 < 0)
})
}
func TestFalse(t *testing.T) {
assertOk(t, "Succeed", func(tb testing.TB) {
tb.Helper()
False(tb, 1 < 0)
})
assertFail(t, "Fail", func(tb testing.TB) {
tb.Helper()
False(tb, 1 > 0)
})
}
func TestEqual(t *testing.T) {
assertOk(t, "Nil", func(tb testing.TB) {
tb.Helper()
Equal(tb, interface{}(nil), interface{}(nil))
})
assertOk(t, "Identical structs", func(tb testing.TB) {
tb.Helper()
Equal(tb, Data{"expected", 1234}, Data{"expected", 1234})
})
assertFail(t, "Different structs", func(tb testing.TB) {
tb.Helper()
Equal(tb, Data{"expected", 1234}, Data{"actual", 1234})
})
assertOk(t, "Identical numbers", func(tb testing.TB) {
tb.Helper()
Equal(tb, 1234, 1234)
})
assertFail(t, "Identical numbers", func(tb testing.TB) {
tb.Helper()
Equal(tb, 1234, 1324)
})
assertOk(t, "Zero-length byte arrays", func(tb testing.TB) {
tb.Helper()
Equal(tb, []byte(nil), []byte(""))
})
assertOk(t, "Identical byte arrays", func(tb testing.TB) {
tb.Helper()
Equal(tb, []byte{1, 2, 3, 4}, []byte{1, 2, 3, 4})
})
assertFail(t, "Different byte arrays", func(tb testing.TB) {
tb.Helper()
Equal(tb, []byte{1, 2, 3, 4}, []byte{1, 3, 2, 4})
})
assertOk(t, "Identical strings", func(tb testing.TB) {
tb.Helper()
Equal(tb, "example", "example")
})
assertFail(t, "Identical strings", func(tb testing.TB) {
tb.Helper()
Equal(tb, "example", "elpmaxe")
})
}
func TestError(t *testing.T) {
assertOk(t, "Error", func(tb testing.TB) {
tb.Helper()
Error(tb, errors.New("example"))
})
assertFail(t, "Nil", func(tb testing.TB) {
tb.Helper()
Error(tb, nil)
})
}
func TestNoError(t *testing.T) {
assertFail(t, "Error", func(tb testing.TB) {
tb.Helper()
NoError(tb, errors.New("example"))
})
assertOk(t, "Nil", func(tb testing.TB) {
tb.Helper()
NoError(tb, nil)
})
}
func TestPanics(t *testing.T) {
willPanic := func() { panic("example") }
wontPanic := func() {}
assertOk(t, "Will panic", func(tb testing.TB) {
tb.Helper()
Panics(tb, willPanic)
})
assertFail(t, "Won't panic", func(tb testing.TB) {
tb.Helper()
Panics(tb, wontPanic)
})
}
func TestZero(t *testing.T) {
assertOk(t, "Empty struct", func(tb testing.TB) {
tb.Helper()
Zero(tb, Data{})
})
assertFail(t, "Non-empty struct", func(tb testing.TB) {
tb.Helper()
Zero(tb, Data{Label: "example"})
})
assertOk(t, "Nil slice", func(tb testing.TB) {
tb.Helper()
var slice []int
Zero(tb, slice)
})
assertFail(t, "Non-empty slice", func(tb testing.TB) {
tb.Helper()
slice := []int{1, 2, 3, 4}
Zero(tb, slice)
})
assertOk(t, "Zero-length slice", func(tb testing.TB) {
tb.Helper()
slice := []int{}
Zero(tb, slice)
})
}
func TestNotZero(t *testing.T) {
assertFail(t, "Empty struct", func(tb testing.TB) {
tb.Helper()
zero := Data{}
NotZero(tb, zero)
})
assertOk(t, "Non-empty struct", func(tb testing.TB) {
tb.Helper()
notZero := Data{Label: "example"}
NotZero(tb, notZero)
})
assertFail(t, "Nil slice", func(tb testing.TB) {
tb.Helper()
var slice []int
NotZero(tb, slice)
})
assertFail(t, "Zero-length slice", func(tb testing.TB) {
tb.Helper()
slice := []int{}
NotZero(tb, slice)
})
assertOk(t, "Non-empty slice", func(tb testing.TB) {
tb.Helper()
slice := []int{1, 2, 3, 4}
NotZero(tb, slice)
})
}
type testCase struct {
*testing.T
failed string
}
func (t *testCase) Fatal(args ...interface{}) {
t.failed = fmt.Sprint(args...)
}
func (t *testCase) Fatalf(message string, args ...interface{}) {
t.failed = fmt.Sprintf(message, args...)
}
func assertFail(t *testing.T, name string, fn func(testing.TB)) {
t.Helper()
t.Run(name, func(t *testing.T) {
t.Helper()
test := &testCase{T: t}
fn(test)
if test.failed == "" {
t.Fatal("Test expected to fail but did not")
} else {
t.Log(test.failed)
}
})
}
func assertOk(t *testing.T, name string, fn func(testing.TB)) {
t.Helper()
t.Run(name, func(t *testing.T) {
t.Helper()
test := &testCase{T: t}
fn(test)
if test.failed != "" {
t.Fatal("Test expected to succeed but did not:\n", test.failed)
}
})
}
-144
View File
@@ -1,144 +0,0 @@
package ast
import (
"fmt"
"unsafe"
"github.com/pelletier/go-toml/v2/internal/danger"
)
// Iterator starts uninitialized, you need to call Next() first.
//
// For example:
//
// it := n.Children()
// for it.Next() {
// it.Node()
// }
type Iterator struct {
started bool
node *Node
}
// Next moves the iterator forward and returns true if points to a
// node, false otherwise.
func (c *Iterator) Next() bool {
if !c.started {
c.started = true
} else if c.node.Valid() {
c.node = c.node.Next()
}
return c.node.Valid()
}
// IsLast returns true if the current node of the iterator is the last
// one. Subsequent call to Next() will return false.
func (c *Iterator) IsLast() bool {
return c.node.next == 0
}
// Node returns a copy of the node pointed at by the iterator.
func (c *Iterator) Node() *Node {
return c.node
}
// Root contains a full AST.
//
// It is immutable once constructed with Builder.
type Root struct {
nodes []Node
}
// Iterator over the top level nodes.
func (r *Root) Iterator() Iterator {
it := Iterator{}
if len(r.nodes) > 0 {
it.node = &r.nodes[0]
}
return it
}
func (r *Root) at(idx Reference) *Node {
return &r.nodes[idx]
}
// Arrays have one child per element in the array. InlineTables have
// one child per key-value pair in the table. KeyValues have at least
// two children. The first one is the value. The rest make a
// potentially dotted key. Table and Array table have one child per
// element of the key they represent (same as KeyValue, but without
// the last node being the value).
type Node struct {
Kind Kind
Raw Range // Raw bytes from the input.
Data []byte // Node value (either allocated or referencing the input).
// References to other nodes, as offsets in the backing array
// from this node. References can go backward, so those can be
// negative.
next int // 0 if last element
child int // 0 if no child
}
type Range struct {
Offset uint32
Length uint32
}
// Next returns a copy of the next node, or an invalid Node if there
// is no next node.
func (n *Node) Next() *Node {
if n.next == 0 {
return nil
}
ptr := unsafe.Pointer(n)
size := unsafe.Sizeof(Node{})
return (*Node)(danger.Stride(ptr, size, n.next))
}
// Child returns a copy of the first child node of this node. Other
// children can be accessed calling Next on the first child. Returns
// an invalid Node if there is none.
func (n *Node) Child() *Node {
if n.child == 0 {
return nil
}
ptr := unsafe.Pointer(n)
size := unsafe.Sizeof(Node{})
return (*Node)(danger.Stride(ptr, size, n.child))
}
// Valid returns true if the node's kind is set (not to Invalid).
func (n *Node) Valid() bool {
return n != nil
}
// Key returns the child nodes making the Key on a supported
// node. Panics otherwise. They are guaranteed to be all be of the
// Kind Key. A simple key would return just one element.
func (n *Node) Key() Iterator {
switch n.Kind {
case KeyValue:
value := n.Child()
if !value.Valid() {
panic(fmt.Errorf("KeyValue should have at least two children"))
}
return Iterator{node: value.Next()}
case Table, ArrayTable:
return Iterator{node: n.Child()}
default:
panic(fmt.Errorf("Key() is not supported on a %s", n.Kind))
}
}
// Value returns a pointer to the value node of a KeyValue.
// Guaranteed to be non-nil. Panics if not called on a KeyValue node,
// or if the Children are malformed.
func (n *Node) Value() *Node {
return n.Child()
}
// Children returns an iterator over a node's children.
func (n *Node) Children() Iterator {
return Iterator{node: n.Child()}
}
-51
View File
@@ -1,51 +0,0 @@
package ast
type Reference int
const InvalidReference Reference = -1
func (r Reference) Valid() bool {
return r != InvalidReference
}
type Builder struct {
tree Root
lastIdx int
}
func (b *Builder) Tree() *Root {
return &b.tree
}
func (b *Builder) NodeAt(ref Reference) *Node {
return b.tree.at(ref)
}
func (b *Builder) Reset() {
b.tree.nodes = b.tree.nodes[:0]
b.lastIdx = 0
}
func (b *Builder) Push(n Node) Reference {
b.lastIdx = len(b.tree.nodes)
b.tree.nodes = append(b.tree.nodes, n)
return Reference(b.lastIdx)
}
func (b *Builder) PushAndChain(n Node) Reference {
newIdx := len(b.tree.nodes)
b.tree.nodes = append(b.tree.nodes, n)
if b.lastIdx >= 0 {
b.tree.nodes[b.lastIdx].next = newIdx - b.lastIdx
}
b.lastIdx = newIdx
return Reference(b.lastIdx)
}
func (b *Builder) AttachChild(parent Reference, child Reference) {
b.tree.nodes[parent].child = int(child) - int(parent)
}
func (b *Builder) Chain(from Reference, to Reference) {
b.tree.nodes[from].next = int(to) - int(from)
}
+42
View File
@@ -0,0 +1,42 @@
package characters
var invalidASCIITable = [256]bool{
0x00: true,
0x01: true,
0x02: true,
0x03: true,
0x04: true,
0x05: true,
0x06: true,
0x07: true,
0x08: true,
// 0x09 TAB
// 0x0A LF
0x0B: true,
0x0C: true,
// 0x0D CR
0x0E: true,
0x0F: true,
0x10: true,
0x11: true,
0x12: true,
0x13: true,
0x14: true,
0x15: true,
0x16: true,
0x17: true,
0x18: true,
0x19: true,
0x1A: true,
0x1B: true,
0x1C: true,
0x1D: true,
0x1E: true,
0x1F: true,
// 0x20 - 0x7E Printable ASCII characters
0x7F: true,
}
func InvalidASCII(b byte) bool {
return invalidASCIITable[b]
}
+24 -89
View File
@@ -1,20 +1,12 @@
package toml // Package characters provides functions for working with string encodings.
package characters
import ( import (
"unicode/utf8" "unicode/utf8"
) )
type utf8Err struct { // Utf8TomlValidAlreadyEscaped verifies that a given string is only made of
Index int // valid UTF-8 characters allowed by the TOML spec:
Size int
}
func (u utf8Err) Zero() bool {
return u.Size == 0
}
// Verified that a given string is only made of valid UTF-8 characters allowed
// by the TOML spec:
// //
// Any Unicode character may be used except those that must be escaped: // Any Unicode character may be used except those that must be escaped:
// quotation mark, backslash, and the control characters other than tab (U+0000 // quotation mark, backslash, and the control characters other than tab (U+0000
@@ -23,8 +15,8 @@ func (u utf8Err) Zero() bool {
// It is a copy of the Go 1.17 utf8.Valid implementation, tweaked to exit early // It is a copy of the Go 1.17 utf8.Valid implementation, tweaked to exit early
// when a character is not allowed. // when a character is not allowed.
// //
// The returned utf8Err is Zero() if the string is valid, or contains the byte // The returned slice is empty if the string is valid, or contains the bytes
// index and size of the invalid character. // of the invalid character.
// //
// quotation mark => already checked // quotation mark => already checked
// backslash => already checked // backslash => already checked
@@ -32,9 +24,8 @@ func (u utf8Err) Zero() bool {
// 0x9 => tab, ok // 0x9 => tab, ok
// 0xA - 0x1F => invalid // 0xA - 0x1F => invalid
// 0x7F => invalid // 0x7F => invalid
func utf8TomlValidAlreadyEscaped(p []byte) (err utf8Err) { func Utf8TomlValidAlreadyEscaped(p []byte) []byte {
// Fast path. Check for and skip 8 bytes of ASCII characters per iteration. // Fast path. Check for and skip 8 bytes of ASCII characters per iteration.
offset := 0
for len(p) >= 8 { for len(p) >= 8 {
// Combining two 32 bit loads allows the same code to be used // Combining two 32 bit loads allows the same code to be used
// for 32 and 64 bit platforms. // for 32 and 64 bit platforms.
@@ -48,24 +39,19 @@ func utf8TomlValidAlreadyEscaped(p []byte) (err utf8Err) {
} }
for i, b := range p[:8] { for i, b := range p[:8] {
if invalidAscii(b) { if InvalidASCII(b) {
err.Index = offset + i return p[i : i+1]
err.Size = 1
return
} }
} }
p = p[8:] p = p[8:]
offset += 8
} }
n := len(p) n := len(p)
for i := 0; i < n; { for i := 0; i < n; {
pi := p[i] pi := p[i]
if pi < utf8.RuneSelf { if pi < utf8.RuneSelf {
if invalidAscii(pi) { if InvalidASCII(pi) {
err.Index = offset + i return p[i : i+1]
err.Size = 1
return
} }
i++ i++
continue continue
@@ -73,44 +59,34 @@ func utf8TomlValidAlreadyEscaped(p []byte) (err utf8Err) {
x := first[pi] x := first[pi]
if x == xx { if x == xx {
// Illegal starter byte. // Illegal starter byte.
err.Index = offset + i return p[i : i+1]
err.Size = 1
return
} }
size := int(x & 7) size := int(x & 7)
if i+size > n { if i+size > n {
// Short or invalid. // Short or invalid.
err.Index = offset + i return p[i:n]
err.Size = n - i
return
} }
accept := acceptRanges[x>>4] accept := acceptRanges[x>>4]
if c := p[i+1]; c < accept.lo || accept.hi < c { if c := p[i+1]; c < accept.lo || accept.hi < c {
err.Index = offset + i return p[i : i+2]
err.Size = 2 } else if size == 2 { //revive:disable:empty-block
return
} else if size == 2 {
} else if c := p[i+2]; c < locb || hicb < c { } else if c := p[i+2]; c < locb || hicb < c {
err.Index = offset + i return p[i : i+3]
err.Size = 3 } else if size == 3 { //revive:disable:empty-block
return
} else if size == 3 {
} else if c := p[i+3]; c < locb || hicb < c { } else if c := p[i+3]; c < locb || hicb < c {
err.Index = offset + i return p[i : i+4]
err.Size = 4
return
} }
i += size i += size
} }
return return nil
} }
// Return the size of the next rune if valid, 0 otherwise. // Utf8ValidNext returns the size of the next rune if valid, 0 otherwise.
func utf8ValidNext(p []byte) int { func Utf8ValidNext(p []byte) int {
c := p[0] c := p[0]
if c < utf8.RuneSelf { if c < utf8.RuneSelf {
if invalidAscii(c) { if InvalidASCII(c) {
return 0 return 0
} }
return 1 return 1
@@ -129,10 +105,10 @@ func utf8ValidNext(p []byte) int {
accept := acceptRanges[x>>4] accept := acceptRanges[x>>4]
if c := p[1]; c < accept.lo || accept.hi < c { if c := p[1]; c < accept.lo || accept.hi < c {
return 0 return 0
} else if size == 2 { } else if size == 2 { //nolint:revive
} else if c := p[2]; c < locb || hicb < c { } else if c := p[2]; c < locb || hicb < c {
return 0 return 0
} else if size == 3 { } else if size == 3 { //nolint:revive
} else if c := p[3]; c < locb || hicb < c { } else if c := p[3]; c < locb || hicb < c {
return 0 return 0
} }
@@ -140,47 +116,6 @@ func utf8ValidNext(p []byte) int {
return size return size
} }
var invalidAsciiTable = [256]bool{
0x00: true,
0x01: true,
0x02: true,
0x03: true,
0x04: true,
0x05: true,
0x06: true,
0x07: true,
0x08: true,
// 0x09 TAB
// 0x0A LF
0x0B: true,
0x0C: true,
// 0x0D CR
0x0E: true,
0x0F: true,
0x10: true,
0x11: true,
0x12: true,
0x13: true,
0x14: true,
0x15: true,
0x16: true,
0x17: true,
0x18: true,
0x19: true,
0x1A: true,
0x1B: true,
0x1C: true,
0x1D: true,
0x1E: true,
0x1F: true,
// 0x20 - 0x7E Printable ASCII characters
0x7F: true,
}
func invalidAscii(b byte) bool {
return invalidAsciiTable[b]
}
// acceptRange gives the range of valid values for the second byte in a UTF-8 // acceptRange gives the range of valid values for the second byte in a UTF-8
// sequence. // sequence.
type acceptRange struct { type acceptRange struct {
+87
View File
@@ -0,0 +1,87 @@
// Package cli provides common functions for command-line programs.
package cli
import (
"bytes"
"errors"
"flag"
"fmt"
"io"
"os"
"github.com/pelletier/go-toml/v2"
)
type ConvertFn func(r io.Reader, w io.Writer) error
type Program struct {
Usage string
Fn ConvertFn
// Inplace allows the command to take more than one file as argument and
// perform conversion in place on each provided file.
Inplace bool
}
func (p *Program) Execute() {
flag.Usage = func() { fmt.Fprint(os.Stderr, p.Usage) }
flag.Parse()
os.Exit(p.main(flag.Args(), os.Stdin, os.Stdout, os.Stderr))
}
func (p *Program) main(files []string, input io.Reader, output, stderr io.Writer) int {
err := p.run(files, input, output)
if err != nil {
var derr *toml.DecodeError
if errors.As(err, &derr) {
_, _ = fmt.Fprintln(stderr, derr.String())
row, col := derr.Position()
_, _ = fmt.Fprintln(stderr, "error occurred at row", row, "column", col)
} else {
_, _ = fmt.Fprintln(stderr, err.Error())
}
return -1
}
return 0
}
func (p *Program) run(files []string, input io.Reader, output io.Writer) error {
if len(files) > 0 {
if p.Inplace {
return p.runAllFilesInPlace(files)
}
f, err := os.Open(files[0])
if err != nil {
return err
}
defer func() { _ = f.Close() }()
input = f
}
return p.Fn(input, output)
}
func (p *Program) runAllFilesInPlace(files []string) error {
for _, path := range files {
err := p.runFileInPlace(path)
if err != nil {
return err
}
}
return nil
}
func (p *Program) runFileInPlace(path string) error {
in, err := os.ReadFile(path) // #nosec G304
if err != nil {
return err
}
out := new(bytes.Buffer)
err = p.Fn(bytes.NewReader(in), out)
if err != nil {
return err
}
return os.WriteFile(path, out.Bytes(), 0o600)
}
+166
View File
@@ -0,0 +1,166 @@
package cli
import (
"bytes"
"errors"
"io"
"os"
"path"
"strings"
"testing"
"github.com/pelletier/go-toml/v2"
"github.com/pelletier/go-toml/v2/internal/assert"
)
func processMain(args []string, input io.Reader, stdout, stderr io.Writer, f ConvertFn) int {
p := Program{Fn: f}
return p.main(args, input, stdout, stderr)
}
func TestProcessMainStdin(t *testing.T) {
stdout := new(bytes.Buffer)
stderr := new(bytes.Buffer)
input := strings.NewReader("this is the input")
exit := processMain([]string{}, input, stdout, stderr, func(io.Reader, io.Writer) error {
return nil
})
assert.Equal(t, 0, exit)
assert.Zero(t, stdout.String())
assert.Zero(t, stderr.String())
}
func TestProcessMainStdinErr(t *testing.T) {
stdout := new(bytes.Buffer)
stderr := new(bytes.Buffer)
input := strings.NewReader("this is the input")
exit := processMain([]string{}, input, stdout, stderr, func(io.Reader, io.Writer) error {
return errors.New("something bad")
})
assert.Equal(t, -1, exit)
assert.Zero(t, stdout.String())
assert.NotZero(t, stderr.String())
}
func TestProcessMainStdinDecodeErr(t *testing.T) {
stdout := new(bytes.Buffer)
stderr := new(bytes.Buffer)
input := strings.NewReader("this is the input")
exit := processMain([]string{}, input, stdout, stderr, func(io.Reader, io.Writer) error {
var v interface{}
return toml.Unmarshal([]byte(`qwe = 001`), &v)
})
assert.Equal(t, -1, exit)
assert.Zero(t, stdout.String())
assert.True(t, strings.Contains(stderr.String(), "error occurred at"))
}
func TestProcessMainFileExists(t *testing.T) {
tmpfile, err := os.CreateTemp(t.TempDir(), "example")
assert.NoError(t, err)
_, err = tmpfile.WriteString(`some data`)
assert.NoError(t, err)
assert.NoError(t, tmpfile.Close())
stdout := new(bytes.Buffer)
stderr := new(bytes.Buffer)
exit := processMain([]string{tmpfile.Name()}, nil, stdout, stderr, func(io.Reader, io.Writer) error {
return nil
})
assert.Equal(t, 0, exit)
assert.Zero(t, stdout.String())
assert.Zero(t, stderr.String())
}
func TestProcessMainFileDoesNotExist(t *testing.T) {
stdout := new(bytes.Buffer)
stderr := new(bytes.Buffer)
exit := processMain([]string{"/lets/hope/this/does/not/exist"}, nil, stdout, stderr, func(io.Reader, io.Writer) error {
return nil
})
assert.Equal(t, -1, exit)
assert.Zero(t, stdout.String())
assert.NotZero(t, stderr.String())
}
func TestProcessMainFilesInPlace(t *testing.T) {
dir := t.TempDir()
path1 := path.Join(dir, "file1")
path2 := path.Join(dir, "file2")
err := os.WriteFile(path1, []byte("content 1"), 0o600)
assert.NoError(t, err)
err = os.WriteFile(path2, []byte("content 2"), 0o600)
assert.NoError(t, err)
p := Program{
Fn: dummyFileFn,
Inplace: true,
}
exit := p.main([]string{path1, path2}, os.Stdin, os.Stdout, os.Stderr)
assert.Equal(t, 0, exit)
v1, err := os.ReadFile(path1)
assert.NoError(t, err)
assert.Equal(t, "1", string(v1))
v2, err := os.ReadFile(path2)
assert.NoError(t, err)
assert.Equal(t, "2", string(v2))
}
func TestProcessMainFilesInPlaceErrRead(t *testing.T) {
p := Program{
Fn: dummyFileFn,
Inplace: true,
}
exit := p.main([]string{"/this/path/is/invalid"}, os.Stdin, os.Stdout, os.Stderr)
assert.Equal(t, -1, exit)
}
func TestProcessMainFilesInPlaceFailFn(t *testing.T) {
dir := t.TempDir()
path1 := path.Join(dir, "file1")
err := os.WriteFile(path1, []byte("content 1"), 0o600)
assert.NoError(t, err)
p := Program{
Fn: func(io.Reader, io.Writer) error { return errors.New("oh no") },
Inplace: true,
}
exit := p.main([]string{path1}, os.Stdin, os.Stdout, os.Stderr)
assert.Equal(t, -1, exit)
v1, err := os.ReadFile(path1)
assert.NoError(t, err)
assert.Equal(t, "content 1", string(v1))
}
func dummyFileFn(r io.Reader, w io.Writer) error {
b, err := io.ReadAll(r)
if err != nil {
return err
}
v := strings.SplitN(string(b), " ", 2)[1]
_, err = w.Write([]byte(v))
return err
}
-65
View File
@@ -1,65 +0,0 @@
package danger
import (
"fmt"
"reflect"
"unsafe"
)
const maxInt = uintptr(int(^uint(0) >> 1))
func SubsliceOffset(data []byte, subslice []byte) int {
datap := (*reflect.SliceHeader)(unsafe.Pointer(&data))
hlp := (*reflect.SliceHeader)(unsafe.Pointer(&subslice))
if hlp.Data < datap.Data {
panic(fmt.Errorf("subslice address (%d) is before data address (%d)", hlp.Data, datap.Data))
}
offset := hlp.Data - datap.Data
if offset > maxInt {
panic(fmt.Errorf("slice offset larger than int (%d)", offset))
}
intoffset := int(offset)
if intoffset > datap.Len {
panic(fmt.Errorf("slice offset (%d) is farther than data length (%d)", intoffset, datap.Len))
}
if intoffset+hlp.Len > datap.Len {
panic(fmt.Errorf("slice ends (%d+%d) is farther than data length (%d)", intoffset, hlp.Len, datap.Len))
}
return intoffset
}
func BytesRange(start []byte, end []byte) []byte {
if start == nil || end == nil {
panic("cannot call BytesRange with nil")
}
startp := (*reflect.SliceHeader)(unsafe.Pointer(&start))
endp := (*reflect.SliceHeader)(unsafe.Pointer(&end))
if startp.Data > endp.Data {
panic(fmt.Errorf("start pointer address (%d) is after end pointer address (%d)", startp.Data, endp.Data))
}
l := startp.Len
endLen := int(endp.Data-startp.Data) + endp.Len
if endLen > l {
l = endLen
}
if l > startp.Cap {
panic(fmt.Errorf("range length is larger than capacity"))
}
return start[:l]
}
func Stride(ptr unsafe.Pointer, size uintptr, offset int) unsafe.Pointer {
// TODO: replace with unsafe.Add when Go 1.17 is released
// https://github.com/golang/go/issues/40481
return unsafe.Pointer(uintptr(ptr) + uintptr(int(size)*offset))
}
-178
View File
@@ -1,178 +0,0 @@
package danger_test
import (
"testing"
"unsafe"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/pelletier/go-toml/v2/internal/danger"
)
func TestSubsliceOffsetValid(t *testing.T) {
examples := []struct {
desc string
test func() ([]byte, []byte)
offset int
}{
{
desc: "simple",
test: func() ([]byte, []byte) {
data := []byte("hello")
return data, data[1:]
},
offset: 1,
},
}
for _, e := range examples {
t.Run(e.desc, func(t *testing.T) {
d, s := e.test()
offset := danger.SubsliceOffset(d, s)
assert.Equal(t, e.offset, offset)
})
}
}
func TestSubsliceOffsetInvalid(t *testing.T) {
examples := []struct {
desc string
test func() ([]byte, []byte)
}{
{
desc: "unrelated arrays",
test: func() ([]byte, []byte) {
return []byte("one"), []byte("two")
},
},
{
desc: "slice starts before data",
test: func() ([]byte, []byte) {
full := []byte("hello world")
return full[5:], full[1:]
},
},
{
desc: "slice starts after data",
test: func() ([]byte, []byte) {
full := []byte("hello world")
return full[:3], full[5:]
},
},
{
desc: "slice ends after data",
test: func() ([]byte, []byte) {
full := []byte("hello world")
return full[:5], full[3:8]
},
},
}
for _, e := range examples {
t.Run(e.desc, func(t *testing.T) {
d, s := e.test()
require.Panics(t, func() {
danger.SubsliceOffset(d, s)
})
})
}
}
func TestStride(t *testing.T) {
a := []byte{1, 2, 3, 4}
x := &a[1]
n := (*byte)(danger.Stride(unsafe.Pointer(x), unsafe.Sizeof(byte(0)), 1))
require.Equal(t, &a[2], n)
n = (*byte)(danger.Stride(unsafe.Pointer(x), unsafe.Sizeof(byte(0)), -1))
require.Equal(t, &a[0], n)
}
func TestBytesRange(t *testing.T) {
type fn = func() ([]byte, []byte)
examples := []struct {
desc string
test fn
expected []byte
}{
{
desc: "simple",
test: func() ([]byte, []byte) {
full := []byte("hello world")
return full[1:3], full[6:8]
},
expected: []byte("ello wo"),
},
{
desc: "full",
test: func() ([]byte, []byte) {
full := []byte("hello world")
return full[0:1], full[len(full)-1:]
},
expected: []byte("hello world"),
},
{
desc: "end before start",
test: func() ([]byte, []byte) {
full := []byte("hello world")
return full[len(full)-1:], full[0:1]
},
},
{
desc: "nils",
test: func() ([]byte, []byte) {
return nil, nil
},
},
{
desc: "nils start",
test: func() ([]byte, []byte) {
return nil, []byte("foo")
},
},
{
desc: "nils end",
test: func() ([]byte, []byte) {
return []byte("foo"), nil
},
},
{
desc: "start is end",
test: func() ([]byte, []byte) {
full := []byte("hello world")
return full[1:3], full[1:3]
},
expected: []byte("el"),
},
{
desc: "end contained in start",
test: func() ([]byte, []byte) {
full := []byte("hello world")
return full[1:7], full[2:4]
},
expected: []byte("ello w"),
},
{
desc: "different backing arrays",
test: func() ([]byte, []byte) {
one := []byte("hello world")
two := []byte("hello world")
return one, two
},
},
}
for _, e := range examples {
t.Run(e.desc, func(t *testing.T) {
start, end := e.test()
if e.expected == nil {
require.Panics(t, func() {
danger.BytesRange(start, end)
})
} else {
res := danger.BytesRange(start, end)
require.Equal(t, e.expected, res)
}
})
}
}
-23
View File
@@ -1,23 +0,0 @@
package danger
import (
"reflect"
"unsafe"
)
// typeID is used as key in encoder and decoder caches to enable using
// the optimize runtime.mapaccess2_fast64 function instead of the more
// expensive lookup if we were to use reflect.Type as map key.
//
// typeID holds the pointer to the reflect.Type value, which is unique
// in the program.
//
// https://github.com/segmentio/encoding/blob/master/json/codec.go#L59-L61
type TypeID unsafe.Pointer
func MakeTypeID(t reflect.Type) TypeID {
// reflect.Type has the fields:
// typ unsafe.Pointer
// ptr unsafe.Pointer
return TypeID((*[2]unsafe.Pointer)(unsafe.Pointer(&t))[1])
}
@@ -1,4 +1,4 @@
package imported_tests package imported_tests //revive:disable:var-naming
// Those tests have been imported from v1, but adjust to match the new // Those tests have been imported from v1, but adjust to match the new
// defaults of v2. // defaults of v2.
@@ -9,7 +9,7 @@ import (
"time" "time"
"github.com/pelletier/go-toml/v2" "github.com/pelletier/go-toml/v2"
"github.com/stretchr/testify/require" "github.com/pelletier/go-toml/v2/internal/assert"
) )
func TestDocMarshal(t *testing.T) { func TestDocMarshal(t *testing.T) {
@@ -21,12 +21,12 @@ func TestDocMarshal(t *testing.T) {
Subdocs testDocSubs `toml:"subdoc"` Subdocs testDocSubs `toml:"subdoc"`
Basics testDocBasics `toml:"basic"` Basics testDocBasics `toml:"basic"`
SubDocList []testSubDoc `toml:"subdoclist"` SubDocList []testSubDoc `toml:"subdoclist"`
err int `toml:"shouldntBeHere"` err int `toml:"shouldntBeHere"` //nolint:unused
unexported int `toml:"shouldntBeHere"` unexported int `toml:"shouldntBeHere"`
Unexported2 int `toml:"-"` Unexported2 int `toml:"-"`
} }
var docData = testDoc{ docData := testDoc{
Title: "TOML Marshal Testing", Title: "TOML Marshal Testing",
unexported: 0, unexported: 0,
Unexported2: 0, Unexported2: 0,
@@ -67,6 +67,7 @@ func TestDocMarshal(t *testing.T) {
} }
marshalTestToml := `title = 'TOML Marshal Testing' marshalTestToml := `title = 'TOML Marshal Testing'
[basic_lists] [basic_lists]
floats = [12.3, 45.6, 78.9] floats = [12.3, 45.6, 78.9]
bools = [true, false, true] bools = [true, false, true]
@@ -89,7 +90,6 @@ name = 'Second'
[subdoc.first] [subdoc.first]
name = 'First' name = 'First'
[basic] [basic]
uint = 5001 uint = 5001
bool = true bool = true
@@ -101,34 +101,34 @@ date = 1979-05-27T07:32:00Z
[[subdoclist]] [[subdoclist]]
name = 'List.First' name = 'List.First'
[[subdoclist]] [[subdoclist]]
name = 'List.Second' name = 'List.Second'
` `
result, err := toml.Marshal(docData) result, err := toml.Marshal(docData)
require.NoError(t, err) assert.NoError(t, err)
require.Equal(t, marshalTestToml, string(result)) assert.Equal(t, marshalTestToml, string(result))
} }
func TestBasicMarshalQuotedKey(t *testing.T) { func TestBasicMarshalQuotedKey(t *testing.T) {
result, err := toml.Marshal(quotedKeyMarshalTestData) result, err := toml.Marshal(quotedKeyMarshalTestData)
require.NoError(t, err) assert.NoError(t, err)
expected := `'Z.string-àéù' = 'Hello' expected := `'Z.string-àéù' = 'Hello'
'Yfloat-𝟘' = 3.5 'Yfloat-𝟘' = 3.5
['Xsubdoc-àéù'] ['Xsubdoc-àéù']
String2 = 'One' String2 = 'One'
[['W.sublist-𝟘']] [['W.sublist-𝟘']]
String2 = 'Two' String2 = 'Two'
[['W.sublist-𝟘']] [['W.sublist-𝟘']]
String2 = 'Three' String2 = 'Three'
` `
require.Equal(t, string(expected), string(result)) assert.Equal(t, expected, string(result))
} }
func TestEmptyMarshal(t *testing.T) { func TestEmptyMarshal(t *testing.T) {
@@ -152,18 +152,18 @@ func TestEmptyMarshal(t *testing.T) {
Map: map[string]string{}, Map: map[string]string{},
} }
result, err := toml.Marshal(doc) result, err := toml.Marshal(doc)
require.NoError(t, err) assert.NoError(t, err)
expected := `title = 'Placeholder' expected := `title = 'Placeholder'
bool = false bool = false
int = 0 int = 0
string = '' string = ''
stringlist = [] stringlist = []
[map]
[map]
` `
require.Equal(t, string(expected), string(result)) assert.Equal(t, expected, string(result))
} }
type textMarshaler struct { type textMarshaler struct {
@@ -186,13 +186,13 @@ func TestTextMarshaler(t *testing.T) {
t.Run("at root", func(t *testing.T) { t.Run("at root", func(t *testing.T) {
_, err := toml.Marshal(m) _, err := toml.Marshal(m)
// in v2 we do not allow TextMarshaler at root // in v2 we do not allow TextMarshaler at root
require.Error(t, err) assert.Error(t, err)
}) })
t.Run("leaf", func(t *testing.T) { t.Run("leaf", func(t *testing.T) {
res, err := toml.Marshal(wrap{m}) res, err := toml.Marshal(wrap{m})
require.NoError(t, err) assert.NoError(t, err)
require.Equal(t, "TM = 'Sally Fields'\n", string(res)) assert.Equal(t, "TM = 'Sally Fields'\n", string(res))
}) })
} }
@@ -1,4 +1,4 @@
package imported_tests package imported_tests //revive:disable:var-naming
// Those tests were imported directly from go-toml v1 // Those tests were imported directly from go-toml v1
// https://raw.githubusercontent.com/pelletier/go-toml/a2e52561804c6cd9392ebf0048ca64fe4af67a43/marshal_test.go // https://raw.githubusercontent.com/pelletier/go-toml/a2e52561804c6cd9392ebf0048ca64fe4af67a43/marshal_test.go
@@ -16,8 +16,7 @@ import (
"time" "time"
"github.com/pelletier/go-toml/v2" "github.com/pelletier/go-toml/v2"
"github.com/stretchr/testify/assert" "github.com/pelletier/go-toml/v2/internal/assert"
"github.com/stretchr/testify/require"
) )
type basicMarshalTestStruct struct { type basicMarshalTestStruct struct {
@@ -123,7 +122,7 @@ func TestInterface(t *testing.T) {
var config Conf var config Conf
config.Inter = &NestedStruct{} config.Inter = &NestedStruct{}
err := toml.Unmarshal(doc, &config) err := toml.Unmarshal(doc, &config)
require.NoError(t, err) assert.NoError(t, err)
expected := Conf{ expected := Conf{
Name: "rui", Name: "rui",
Age: 18, Age: 18,
@@ -139,8 +138,8 @@ func TestInterface(t *testing.T) {
func TestBasicUnmarshal(t *testing.T) { func TestBasicUnmarshal(t *testing.T) {
result := basicMarshalTestStruct{} result := basicMarshalTestStruct{}
err := toml.Unmarshal(basicTestToml, &result) err := toml.Unmarshal(basicTestToml, &result)
require.NoError(t, err) assert.NoError(t, err)
require.Equal(t, basicTestData, result) assert.Equal(t, basicTestData, result)
} }
type quotedKeyMarshalTestStruct struct { type quotedKeyMarshalTestStruct struct {
@@ -150,8 +149,6 @@ type quotedKeyMarshalTestStruct struct {
SubList []basicMarshalTestSubStruct `toml:"W.sublist-𝟘"` SubList []basicMarshalTestSubStruct `toml:"W.sublist-𝟘"`
} }
// TODO: Remove nolint once var is used by a test
//nolint:deadcode,unused,varcheck
var quotedKeyMarshalTestData = quotedKeyMarshalTestStruct{ var quotedKeyMarshalTestData = quotedKeyMarshalTestStruct{
String: "Hello", String: "Hello",
Float: 3.5, Float: 3.5,
@@ -160,7 +157,8 @@ var quotedKeyMarshalTestData = quotedKeyMarshalTestStruct{
} }
// TODO: Remove nolint once var is used by a test // TODO: Remove nolint once var is used by a test
//nolint:deadcode,unused,varcheck //
//nolint:unused
var quotedKeyMarshalTestToml = []byte(`"Yfloat-𝟘" = 3.5 var quotedKeyMarshalTestToml = []byte(`"Yfloat-𝟘" = 3.5
"Z.string-àéù" = "Hello" "Z.string-àéù" = "Hello"
@@ -182,11 +180,12 @@ type testDoc struct {
Subdocs testDocSubs `toml:"subdoc"` Subdocs testDocSubs `toml:"subdoc"`
Basics testDocBasics `toml:"basic"` Basics testDocBasics `toml:"basic"`
SubDocList []testSubDoc `toml:"subdoclist"` SubDocList []testSubDoc `toml:"subdoclist"`
err int `toml:"shouldntBeHere"` // nolint:structcheck,unused err int `toml:"shouldntBeHere"` //nolint:unused
unexported int `toml:"shouldntBeHere"` unexported int `toml:"shouldntBeHere"`
Unexported2 int `toml:"-"` Unexported2 int `toml:"-"`
} }
//nolint:unused
type testMapDoc struct { type testMapDoc struct {
Title string `toml:"title"` Title string `toml:"title"`
BasicMap map[string]string `toml:"basic_map"` BasicMap map[string]string `toml:"basic_map"`
@@ -272,7 +271,8 @@ var docData = testDoc{
} }
// TODO: Remove nolint once var is used by a test // TODO: Remove nolint once var is used by a test
//nolint:deadcode,unused,varcheck //
//nolint:unused
var mapTestDoc = testMapDoc{ var mapTestDoc = testMapDoc{
Title: "TOML Marshal Testing", Title: "TOML Marshal Testing",
BasicMap: map[string]string{ BasicMap: map[string]string{
@@ -297,7 +297,7 @@ func TestDocUnmarshal(t *testing.T) {
result := testDoc{} result := testDoc{}
err := toml.Unmarshal(marshalTestToml, &result) err := toml.Unmarshal(marshalTestToml, &result)
expected := docData expected := docData
require.NoError(t, err) assert.NoError(t, err)
assert.Equal(t, expected, result) assert.Equal(t, expected, result)
} }
@@ -337,7 +337,7 @@ shouldntBeHere = 2
func TestUnexportedUnmarshal(t *testing.T) { func TestUnexportedUnmarshal(t *testing.T) {
result := unexportedMarshalTestStruct{} result := unexportedMarshalTestStruct{}
err := toml.Unmarshal(unexportedTestToml, &result) err := toml.Unmarshal(unexportedTestToml, &result)
require.NoError(t, err) assert.NoError(t, err)
assert.Equal(t, unexportedTestData, result) assert.Equal(t, unexportedTestData, result)
} }
@@ -453,7 +453,7 @@ func TestEmptytomlUnmarshal(t *testing.T) {
result := emptyMarshalTestStruct{} result := emptyMarshalTestStruct{}
err := toml.Unmarshal(emptyTestToml, &result) err := toml.Unmarshal(emptyTestToml, &result)
require.NoError(t, err) assert.NoError(t, err)
assert.Equal(t, emptyTestData, result) assert.Equal(t, emptyTestData, result)
} }
@@ -501,7 +501,7 @@ Str = "Hello"
func TestPointerUnmarshal(t *testing.T) { func TestPointerUnmarshal(t *testing.T) {
result := pointerMarshalTestStruct{} result := pointerMarshalTestStruct{}
err := toml.Unmarshal(pointerTestToml, &result) err := toml.Unmarshal(pointerTestToml, &result)
require.NoError(t, err) assert.NoError(t, err)
assert.Equal(t, pointerTestData, result) assert.Equal(t, pointerTestData, result)
} }
@@ -537,40 +537,47 @@ StringPtr = [["Three", "Four"]]
func TestNestedUnmarshal(t *testing.T) { func TestNestedUnmarshal(t *testing.T) {
result := nestedMarshalTestStruct{} result := nestedMarshalTestStruct{}
err := toml.Unmarshal(nestedTestToml, &result) err := toml.Unmarshal(nestedTestToml, &result)
require.NoError(t, err) assert.NoError(t, err)
assert.Equal(t, nestedTestData, result) assert.Equal(t, nestedTestData, result)
} }
//nolint:unused
type customMarshalerParent struct { type customMarshalerParent struct {
Self customMarshaler `toml:"me"` Self customMarshaler `toml:"me"`
Friends []customMarshaler `toml:"friends"` Friends []customMarshaler `toml:"friends"`
} }
//nolint:unused
type customMarshaler struct { type customMarshaler struct {
FirstName string FirstName string
LastName string LastName string
} }
//nolint:unused
func (c customMarshaler) MarshalTOML() ([]byte, error) { func (c customMarshaler) MarshalTOML() ([]byte, error) {
fullName := fmt.Sprintf("%s %s", c.FirstName, c.LastName) fullName := fmt.Sprintf("%s %s", c.FirstName, c.LastName)
return []byte(fullName), nil return []byte(fullName), nil
} }
//nolint:unused
var customMarshalerData = customMarshaler{FirstName: "Sally", LastName: "Fields"} var customMarshalerData = customMarshaler{FirstName: "Sally", LastName: "Fields"}
// TODO: Remove nolint once var is used by a test // TODO: Remove nolint once var is used by a test
//nolint:deadcode,unused,varcheck //
//nolint:unused
var customMarshalerToml = []byte(`Sally Fields`) var customMarshalerToml = []byte(`Sally Fields`)
// TODO: Remove nolint once var is used by a test // TODO: Remove nolint once var is used by a test
//nolint:deadcode,unused,varcheck //
//nolint:unused
var nestedCustomMarshalerData = customMarshalerParent{ var nestedCustomMarshalerData = customMarshalerParent{
Self: customMarshaler{FirstName: "Maiku", LastName: "Suteda"}, Self: customMarshaler{FirstName: "Maiku", LastName: "Suteda"},
Friends: []customMarshaler{customMarshalerData}, Friends: []customMarshaler{customMarshalerData},
} }
// TODO: Remove nolint once var is used by a test // TODO: Remove nolint once var is used by a test
//nolint:deadcode,unused,varcheck //
//nolint:unused
var nestedCustomMarshalerToml = []byte(`friends = ["Sally Fields"] var nestedCustomMarshalerToml = []byte(`friends = ["Sally Fields"]
me = "Maiku Suteda" me = "Maiku Suteda"
`) `)
@@ -585,7 +592,7 @@ func (x *IntOrString) MarshalTOML() ([]byte, error) {
s := *(*string)(x) s := *(*string)(x)
_, err := strconv.Atoi(s) _, err := strconv.Atoi(s)
if err != nil { if err != nil {
return []byte(fmt.Sprintf(`"%s"`, s)), nil return []byte(fmt.Sprintf(`"%s"`, s)), nil //nolint:nilerr
} }
return []byte(s), nil return []byte(s), nil
} }
@@ -611,6 +618,7 @@ func TestUnmarshalTextMarshaler(t *testing.T) {
} }
// TODO: Remove nolint once type and methods are used by a test // TODO: Remove nolint once type and methods are used by a test
//
//nolint:unused //nolint:unused
type precedentMarshaler struct { type precedentMarshaler struct {
FirstName string FirstName string
@@ -629,6 +637,7 @@ func (m precedentMarshaler) MarshalTOML() ([]byte, error) {
} }
// TODO: Remove nolint once type and method are used by a test // TODO: Remove nolint once type and method are used by a test
//
//nolint:unused //nolint:unused
type customPointerMarshaler struct { type customPointerMarshaler struct {
FirstName string FirstName string
@@ -641,6 +650,7 @@ func (m *customPointerMarshaler) MarshalTOML() ([]byte, error) {
} }
// TODO: Remove nolint once type and method are used by a test // TODO: Remove nolint once type and method are used by a test
//
//nolint:unused //nolint:unused
type textPointerMarshaler struct { type textPointerMarshaler struct {
FirstName string FirstName string
@@ -653,7 +663,8 @@ func (m *textPointerMarshaler) MarshalText() ([]byte, error) {
} }
// TODO: Remove nolint once var is used by a test // TODO: Remove nolint once var is used by a test
//nolint:deadcode,unused,varcheck //
//nolint:unused
var commentTestToml = []byte(` var commentTestToml = []byte(`
# it's a comment on type # it's a comment on type
[postgres] [postgres]
@@ -678,6 +689,7 @@ var commentTestToml = []byte(`
My = "Baar" My = "Baar"
`) `)
//nolint:unused
type mapsTestStruct struct { type mapsTestStruct struct {
Simple map[string]string Simple map[string]string
Paths map[string]string Paths map[string]string
@@ -690,7 +702,8 @@ type mapsTestStruct struct {
} }
// TODO: Remove nolint once var is used by a test // TODO: Remove nolint once var is used by a test
//nolint:deadcode,unused,varcheck //
//nolint:unused
var mapsTestData = mapsTestStruct{ var mapsTestData = mapsTestStruct{
Simple: map[string]string{ Simple: map[string]string{
"one plus one": "two", "one plus one": "two",
@@ -713,7 +726,8 @@ var mapsTestData = mapsTestStruct{
} }
// TODO: Remove nolint once var is used by a test // TODO: Remove nolint once var is used by a test
//nolint:deadcode,unused,varcheck //
//nolint:unused
var mapsTestToml = []byte(` var mapsTestToml = []byte(`
[Other] [Other]
"testing" = 3.9999 "testing" = 3.9999
@@ -735,7 +749,8 @@ var mapsTestToml = []byte(`
`) `)
// TODO: Remove nolint once type is used by a test // TODO: Remove nolint once type is used by a test
//nolint:deadcode,unused //
//nolint:unused
type structArrayNoTag struct { type structArrayNoTag struct {
A struct { A struct {
B []int64 B []int64
@@ -744,7 +759,8 @@ type structArrayNoTag struct {
} }
// TODO: Remove nolint once var is used by a test // TODO: Remove nolint once var is used by a test
//nolint:deadcode,unused,varcheck //
//nolint:unused
var customTagTestToml = []byte(` var customTagTestToml = []byte(`
[postgres] [postgres]
password = "bvalue" password = "bvalue"
@@ -758,7 +774,8 @@ var customTagTestToml = []byte(`
`) `)
// TODO: Remove nolint once var is used by a test // TODO: Remove nolint once var is used by a test
//nolint:deadcode,unused,varcheck //
//nolint:unused
var customCommentTagTestToml = []byte(` var customCommentTagTestToml = []byte(`
# db connection # db connection
[postgres] [postgres]
@@ -771,7 +788,8 @@ var customCommentTagTestToml = []byte(`
`) `)
// TODO: Remove nolint once var is used by a test // TODO: Remove nolint once var is used by a test
//nolint:deadcode,unused,varcheck //
//nolint:unused
var customCommentedTagTestToml = []byte(` var customCommentedTagTestToml = []byte(`
[postgres] [postgres]
# password = "bvalue" # password = "bvalue"
@@ -818,14 +836,15 @@ func TestUnmarshalTabInStringAndQuotedKey(t *testing.T) {
t.Run(test.desc, func(t *testing.T) { t.Run(test.desc, func(t *testing.T) {
result := Test{} result := Test{}
err := toml.Unmarshal(test.input, &result) err := toml.Unmarshal(test.input, &result)
require.NoError(t, err) assert.NoError(t, err)
assert.Equal(t, test.expected, result) assert.Equal(t, test.expected, result)
}) })
} }
} }
// TODO: Remove nolint once var is used by a test // TODO: Remove nolint once var is used by a test
//nolint:deadcode,unused,varcheck //
//nolint:unused
var customMultilineTagTestToml = []byte(`int_slice = [ var customMultilineTagTestToml = []byte(`int_slice = [
1, 1,
2, 2,
@@ -834,7 +853,8 @@ var customMultilineTagTestToml = []byte(`int_slice = [
`) `)
// TODO: Remove nolint once var is used by a test // TODO: Remove nolint once var is used by a test
//nolint:deadcode,unused,varcheck //
//nolint:unused
var testDocBasicToml = []byte(` var testDocBasicToml = []byte(`
[document] [document]
bool_val = true bool_val = true
@@ -845,14 +865,12 @@ var testDocBasicToml = []byte(`
uint_val = 5001 uint_val = 5001
`) `)
// TODO: Remove nolint once type is used by a test //nolint:unused
//nolint:deadcode
type testDocCustomTag struct { type testDocCustomTag struct {
Doc testDocBasicsCustomTag `file:"document"` Doc testDocBasicsCustomTag `file:"document"`
} }
// TODO: Remove nolint once type is used by a test //nolint:unused
//nolint:deadcode
type testDocBasicsCustomTag struct { type testDocBasicsCustomTag struct {
Bool bool `file:"bool_val"` Bool bool `file:"bool_val"`
Date time.Time `file:"date_val"` Date time.Time `file:"date_val"`
@@ -863,8 +881,7 @@ type testDocBasicsCustomTag struct {
unexported int `file:"shouldntBeHere"` unexported int `file:"shouldntBeHere"`
} }
// TODO: Remove nolint once var is used by a test //nolint:unused
//nolint:deadcode,varcheck
var testDocCustomTagData = testDocCustomTag{ var testDocCustomTagData = testDocCustomTag{
Doc: testDocBasicsCustomTag{ Doc: testDocBasicsCustomTag{
Bool: true, Bool: true,
@@ -927,6 +944,29 @@ func TestUnmarshalMapWithTypedKey(t *testing.T) {
} }
} }
func TestUnmarshalTypeTableHeader(t *testing.T) {
testToml := []byte(`
[test]
a = 1
`)
type header string
var result map[header]map[string]int
err := toml.Unmarshal(testToml, &result)
if err != nil {
t.Errorf("Received unexpected error: %s", err)
return
}
expected := map[header]map[string]int{
"test": {"a": 1},
}
if !reflect.DeepEqual(result, expected) {
t.Errorf("Bad unmarshal: expected %v, got %v", expected, result)
}
}
func TestUnmarshalNonPointer(t *testing.T) { func TestUnmarshalNonPointer(t *testing.T) {
a := 1 a := 1
err := toml.Unmarshal([]byte{}, a) err := toml.Unmarshal([]byte{}, a)
@@ -943,13 +983,14 @@ func TestUnmarshalInvalidPointerKind(t *testing.T) {
} }
// TODO: Remove nolint once var is used by a test // TODO: Remove nolint once var is used by a test
//nolint:deadcode,unused //
//nolint:unused
type testDuration struct { type testDuration struct {
Nanosec time.Duration `toml:"nanosec"` Nanosec time.Duration `toml:"nanosec"`
Microsec1 time.Duration `toml:"microsec1"` Microsec1 time.Duration `toml:"microsec1"`
Microsec2 *time.Duration `toml:"microsec2"` Microsec2 *time.Duration `toml:"microsec2"`
Millisec time.Duration `toml:"millisec"` Millisec time.Duration `toml:"millisec"`
Sec time.Duration `toml:"sec"` Sec time.Duration `toml:"sec"` //nolint:staticcheck
Min time.Duration `toml:"min"` Min time.Duration `toml:"min"`
Hour time.Duration `toml:"hour"` Hour time.Duration `toml:"hour"`
Mixed time.Duration `toml:"mixed"` Mixed time.Duration `toml:"mixed"`
@@ -957,7 +998,8 @@ type testDuration struct {
} }
// TODO: Remove nolint once var is used by a test // TODO: Remove nolint once var is used by a test
//nolint:deadcode,unused,varcheck //
//nolint:unused
var testDurationToml = []byte(` var testDurationToml = []byte(`
nanosec = "1ns" nanosec = "1ns"
microsec1 = "1us" microsec1 = "1us"
@@ -971,7 +1013,8 @@ a_string = "15s"
`) `)
// TODO: Remove nolint once var is used by a test // TODO: Remove nolint once var is used by a test
//nolint:deadcode,unused,varcheck //
//nolint:unused
var testDurationToml2 = []byte(`a_string = "15s" var testDurationToml2 = []byte(`a_string = "15s"
hour = "1h0m0s" hour = "1h0m0s"
microsec1 = "1µs" microsec1 = "1µs"
@@ -984,15 +1027,15 @@ sec = "1s"
`) `)
// TODO: Remove nolint once type is used by a test // TODO: Remove nolint once type is used by a test
//nolint:deadcode,unused //
//nolint:unused
type testBadDuration struct { type testBadDuration struct {
Val time.Duration `toml:"val"` Val time.Duration `toml:"val"`
} }
// TODO: add back camelCase test // TODO: add back camelCase test
var testCamelCaseKeyToml = []byte(`fooBar = 10`) //nolint:unused var testCamelCaseKeyToml = []byte(`fooBar = 10`)
//nolint:unused
func TestUnmarshalCamelCaseKey(t *testing.T) { func TestUnmarshalCamelCaseKey(t *testing.T) {
t.Skipf("don't know if it is a good idea to automatically convert like that yet") t.Skipf("don't know if it is a good idea to automatically convert like that yet")
var x struct { var x struct {
@@ -1011,7 +1054,7 @@ func TestUnmarshalCamelCaseKey(t *testing.T) {
func TestUnmarshalNegativeUint(t *testing.T) { func TestUnmarshalNegativeUint(t *testing.T) {
t.Skipf("not sure if we this should always error") t.Skipf("not sure if we this should always error")
type check struct{ U uint } // nolint:unused type check struct{ U uint }
err := toml.Unmarshal([]byte("U = -1"), &check{}) err := toml.Unmarshal([]byte("U = -1"), &check{})
assert.Error(t, err) assert.Error(t, err)
} }
@@ -1037,16 +1080,12 @@ func TestUnmarshalCheckConversionFloatInt(t *testing.T) {
desc: "int", desc: "int",
input: `I = 1e300`, input: `I = 1e300`,
}, },
{
desc: "float",
input: `F = 9223372036854775806`,
},
} }
for _, test := range testCases { for _, test := range testCases {
t.Run(test.desc, func(t *testing.T) { t.Run(test.desc, func(t *testing.T) {
err := toml.Unmarshal([]byte(test.input), &conversionCheck{}) err := toml.Unmarshal([]byte(test.input), &conversionCheck{})
require.Error(t, err) assert.Error(t, err)
}) })
} }
} }
@@ -1081,7 +1120,7 @@ func TestUnmarshalOverflow(t *testing.T) {
for _, test := range testCases { for _, test := range testCases {
t.Run(test.desc, func(t *testing.T) { t.Run(test.desc, func(t *testing.T) {
err := toml.Unmarshal([]byte(test.input), &overflow{}) err := toml.Unmarshal([]byte(test.input), &overflow{})
require.Error(t, err) assert.Error(t, err)
}) })
} }
} }
@@ -1492,7 +1531,7 @@ func TestUnmarshalLocalDateTime(t *testing.T) {
} }
for i, example := range examples { for i, example := range examples {
doc := fmt.Sprintf(`date = %s`, example.in) doc := "date = " + example.in
t.Run(fmt.Sprintf("ToLocalDateTime_%d_%s", i, example.name), func(t *testing.T) { t.Run(fmt.Sprintf("ToLocalDateTime_%d_%s", i, example.name), func(t *testing.T) {
type dateStruct struct { type dateStruct struct {
@@ -1578,7 +1617,7 @@ func TestUnmarshalLocalTime(t *testing.T) {
} }
for i, example := range examples { for i, example := range examples {
doc := fmt.Sprintf(`Time = %s`, example.in) doc := "Time = " + example.in
t.Run(fmt.Sprintf("ToLocalTime_%d_%s", i, example.name), func(t *testing.T) { t.Run(fmt.Sprintf("ToLocalTime_%d_%s", i, example.name), func(t *testing.T) {
type dateStruct struct { type dateStruct struct {
@@ -1701,7 +1740,7 @@ Age = 23
} }
actual := OuterStruct{} actual := OuterStruct{}
err := toml.Unmarshal(doc, &actual) err := toml.Unmarshal(doc, &actual)
require.NoError(t, err) assert.NoError(t, err)
assert.Equal(t, expected, actual) assert.Equal(t, expected, actual)
} }
@@ -1786,7 +1825,7 @@ InnerField = "After4"
} }
err := toml.Unmarshal(doc, &actual) err := toml.Unmarshal(doc, &actual)
require.NoError(t, err) assert.NoError(t, err)
assert.Equal(t, expected, actual) assert.Equal(t, expected, actual)
} }
@@ -1835,7 +1874,7 @@ type arrayTooSmallStruct struct {
func TestUnmarshalSlice(t *testing.T) { func TestUnmarshalSlice(t *testing.T) {
var actual sliceStruct var actual sliceStruct
err := toml.Unmarshal(sliceTomlDemo, &actual) err := toml.Unmarshal(sliceTomlDemo, &actual)
require.NoError(t, err) assert.NoError(t, err)
expected := sliceStruct{ expected := sliceStruct{
Slice: []string{"Howdy", "Hey There"}, Slice: []string{"Howdy", "Hey There"},
SlicePtr: &[]string{"Howdy", "Hey There"}, SlicePtr: &[]string{"Howdy", "Hey There"},
@@ -1863,19 +1902,12 @@ func TestUnmarshalMixedTypeSlice(t *testing.T) {
ArrayField []interface{} ArrayField []interface{}
} }
//doc := []byte(`ArrayField = [3.14,100,true,"hello world",{Field = "inner1"},[{Field = "inner2"},{Field = "inner3"}]]
//`)
doc := []byte(`ArrayField = [{Field = "inner1"},[{Field = "inner2"},{Field = "inner3"}]] doc := []byte(`ArrayField = [{Field = "inner1"},[{Field = "inner2"},{Field = "inner3"}]]
`) `)
actual := TestStruct{} actual := TestStruct{}
expected := TestStruct{ expected := TestStruct{
ArrayField: []interface{}{ ArrayField: []interface{}{
//3.14,
//int64(100),
//true,
//"hello world",
map[string]interface{}{ map[string]interface{}{
"Field": "inner1", "Field": "inner1",
}, },
@@ -1886,7 +1918,7 @@ func TestUnmarshalMixedTypeSlice(t *testing.T) {
}, },
} }
err := toml.Unmarshal(doc, &actual) err := toml.Unmarshal(doc, &actual)
require.NoError(t, err) assert.NoError(t, err)
assert.Equal(t, expected, actual) assert.Equal(t, expected, actual)
} }
@@ -1895,7 +1927,7 @@ func TestUnmarshalArray(t *testing.T) {
var actual arrayStruct var actual arrayStruct
err = toml.Unmarshal(sliceTomlDemo, &actual) err = toml.Unmarshal(sliceTomlDemo, &actual)
require.NoError(t, err) assert.NoError(t, err)
expected := arrayStruct{ expected := arrayStruct{
Slice: [4]string{"Howdy", "Hey There"}, Slice: [4]string{"Howdy", "Hey There"},
@@ -1925,7 +1957,7 @@ func decoder(doc string) *toml.Decoder {
func strictDecoder(doc string) *toml.Decoder { func strictDecoder(doc string) *toml.Decoder {
d := decoder(doc) d := decoder(doc)
d.SetStrict(true) d.DisallowUnknownFields()
return d return d
} }
@@ -1954,11 +1986,17 @@ func TestDecoderStrict(t *testing.T) {
} }
err := strictDecoder(input).Decode(&doc) err := strictDecoder(input).Decode(&doc)
require.Error(t, err) assert.Error(t, err)
require.IsType(t, &toml.StrictMissingError{}, err)
se := err.(*toml.StrictMissingError)
keys := []toml.Key{} assert.Equal(t,
reflect.TypeOf(err), reflect.TypeOf(&toml.StrictMissingError{}),
"Expected a *toml.StrictMissingError, got: %v", reflect.TypeOf(err),
)
var se *toml.StrictMissingError
assert.True(t, errors.As(err, &se))
keys := make([]toml.Key, 0, len(se.Errors))
for _, e := range se.Errors { for _, e := range se.Errors {
keys = append(keys, e.Key()) keys = append(keys, e.Key())
@@ -1971,13 +2009,14 @@ func TestDecoderStrict(t *testing.T) {
{"undecoded", "array"}, {"undecoded", "array"},
} }
require.Equal(t, expectedKeys, keys) assert.Equal(t, expectedKeys, keys)
err = decoder(input).Decode(&doc) err = decoder(input).Decode(&doc)
require.NoError(t, err) assert.NoError(t, err)
var m map[string]interface{} var m map[string]interface{}
err = decoder(input).Decode(&m) err = decoder(input).Decode(&m)
assert.NoError(t, err)
} }
func TestDecoderStrictValid(t *testing.T) { func TestDecoderStrictValid(t *testing.T) {
@@ -1992,7 +2031,7 @@ func TestDecoderStrictValid(t *testing.T) {
} }
err := strictDecoder(input).Decode(&doc) err := strictDecoder(input).Decode(&doc)
require.NoError(t, err) assert.NoError(t, err)
} }
type docUnmarshalTOML struct { type docUnmarshalTOML struct {
@@ -2014,19 +2053,6 @@ func (d *docUnmarshalTOML) UnmarshalTOML(i interface{}) error {
return nil return nil
} }
func TestDecoderStrictCustomUnmarshal(t *testing.T) {
t.Skip()
//input := `key = "ok"`
//var doc docUnmarshalTOML
//err := NewDecoder(bytes.NewReader([]byte(input))).Strict(true).Decode(&doc)
//if err != nil {
// t.Fatal("unexpected error:", err)
//}
//if doc.Decoded.Key != "ok" {
// t.Errorf("Bad unmarshal: expected ok, got %v", doc.Decoded.Key)
//}
}
type parent struct { type parent struct {
Doc docUnmarshalTOML Doc docUnmarshalTOML
DocPointer *docUnmarshalTOML DocPointer *docUnmarshalTOML
@@ -2043,7 +2069,7 @@ func TestCustomUnmarshal(t *testing.T) {
var d parent var d parent
err := toml.Unmarshal([]byte(input), &d) err := toml.Unmarshal([]byte(input), &d)
require.NoError(t, err) assert.NoError(t, err)
assert.Equal(t, "ok1", d.Doc.Decoded.Key) assert.Equal(t, "ok1", d.Doc.Decoded.Key)
assert.Equal(t, "ok2", d.DocPointer.Decoded.Key) assert.Equal(t, "ok2", d.DocPointer.Decoded.Key)
} }
@@ -2109,7 +2135,7 @@ Int = 21
Float = 2.0 Float = 2.0
` `
err := toml.Unmarshal([]byte(input), &doc) err := toml.Unmarshal([]byte(input), &doc)
require.NoError(t, err) assert.NoError(t, err)
assert.Equal(t, 12, doc.UnixTime.Value) assert.Equal(t, 12, doc.UnixTime.Value)
assert.Equal(t, 42, doc.Version.Value) assert.Equal(t, 42, doc.Version.Value)
assert.Equal(t, 1, doc.Bool.Value) assert.Equal(t, 1, doc.Bool.Value)
@@ -2179,7 +2205,10 @@ func TestUnmarshalEmptyInterface(t *testing.T) {
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
require.IsType(t, map[string]interface{}{}, v) assert.Equal(t,
reflect.TypeOf(map[string]interface{}{}), reflect.TypeOf(v),
"Expected map[string]interface{}{} type, got: %v", reflect.TypeOf(v),
)
x := v.(map[string]interface{}) x := v.(map[string]interface{})
assert.Equal(t, "pelletier", x["User"]) assert.Equal(t, "pelletier", x["User"])
@@ -2227,12 +2256,12 @@ type Custom struct {
v string v string
} }
func (c *Custom) UnmarshalTOML(v interface{}) error { func (c *Custom) UnmarshalTOML(interface{}) error {
c.v = "called" c.v = "called"
return nil return nil
} }
func TestGithubIssue431(t *testing.T) { func TestGitHubIssue431(t *testing.T) {
doc := `key = "value"` doc := `key = "value"`
var c Config var c Config
if err := toml.Unmarshal([]byte(doc), &c); err != nil { if err := toml.Unmarshal([]byte(doc), &c); err != nil {
@@ -2252,14 +2281,14 @@ type durationString struct {
time.Duration time.Duration
} }
func (d *durationString) UnmarshalTOML(v interface{}) error { func (d *durationString) UnmarshalTOML(interface{}) error {
d.Duration = 10 * time.Second d.Duration = 10 * time.Second
return nil return nil
} }
type config437Error struct{} type config437Error struct{}
func (e *config437Error) UnmarshalTOML(v interface{}) error { func (e *config437Error) UnmarshalTOML(interface{}) error {
return errors.New("expected") return errors.New("expected")
} }
@@ -2270,7 +2299,7 @@ type config437 struct {
} `toml:"HTTP"` } `toml:"HTTP"`
} }
func TestGithubIssue437(t *testing.T) { func TestGitHubIssue437(t *testing.T) {
t.Skipf("unmarshalTOML not implemented") t.Skipf("unmarshalTOML not implemented")
src := ` src := `
[HTTP] [HTTP]
@@ -3,17 +3,18 @@ package testsuite
import ( import (
"fmt" "fmt"
"math" "math"
"strconv"
"time" "time"
"github.com/pelletier/go-toml/v2" "github.com/pelletier/go-toml/v2"
) )
// addTag adds JSON tags to a data structure as expected by toml-test. // addTag adds JSON tags to a data structure as expected by toml-test.
func addTag(key string, tomlData interface{}) interface{} { func addTag(tomlData interface{}) interface{} {
// Switch on the data type. // Switch on the data type.
switch orig := tomlData.(type) { switch orig := tomlData.(type) {
default: default:
//return map[string]interface{}{} // return map[string]interface{}{}
panic(fmt.Sprintf("Unknown type: %T", tomlData)) panic(fmt.Sprintf("Unknown type: %T", tomlData))
// A table: we don't need to add any tags, just recurse for every table // A table: we don't need to add any tags, just recurse for every table
@@ -21,7 +22,7 @@ func addTag(key string, tomlData interface{}) interface{} {
case map[string]interface{}: case map[string]interface{}:
typed := make(map[string]interface{}, len(orig)) typed := make(map[string]interface{}, len(orig))
for k, v := range orig { for k, v := range orig {
typed[k] = addTag(k, v) typed[k] = addTag(v)
} }
return typed return typed
@@ -30,13 +31,13 @@ func addTag(key string, tomlData interface{}) interface{} {
case []map[string]interface{}: case []map[string]interface{}:
typed := make([]map[string]interface{}, len(orig)) typed := make([]map[string]interface{}, len(orig))
for i, v := range orig { for i, v := range orig {
typed[i] = addTag("", v).(map[string]interface{}) typed[i] = addTag(v).(map[string]interface{})
} }
return typed return typed
case []interface{}: case []interface{}:
typed := make([]interface{}, len(orig)) typed := make([]interface{}, len(orig))
for i, v := range orig { for i, v := range orig {
typed[i] = addTag("", v) typed[i] = addTag(v)
} }
return typed return typed
@@ -52,11 +53,11 @@ func addTag(key string, tomlData interface{}) interface{} {
// Tag primitive values: bool, string, int, and float64. // Tag primitive values: bool, string, int, and float64.
case bool: case bool:
return tag("bool", fmt.Sprintf("%v", orig)) return tag("bool", strconv.FormatBool(orig))
case string: case string:
return tag("string", orig) return tag("string", orig)
case int64: case int64:
return tag("integer", fmt.Sprintf("%d", orig)) return tag("integer", strconv.FormatInt(orig, 10))
case float64: case float64:
// Special case for nan since NaN == NaN is false. // Special case for nan since NaN == NaN is false.
if math.IsNaN(orig) { if math.IsNaN(orig) {
@@ -9,6 +9,7 @@ import (
) )
func CmpJSON(t *testing.T, key string, want, have interface{}) { func CmpJSON(t *testing.T, key string, want, have interface{}) {
t.Helper()
switch w := want.(type) { switch w := want.(type) {
case map[string]interface{}: case map[string]interface{}:
cmpJSONMaps(t, key, w, have) cmpJSONMaps(t, key, w, have)
@@ -22,6 +23,7 @@ func CmpJSON(t *testing.T, key string, want, have interface{}) {
} }
func cmpJSONMaps(t *testing.T, key string, want map[string]interface{}, have interface{}) { func cmpJSONMaps(t *testing.T, key string, want map[string]interface{}, have interface{}) {
t.Helper()
haveMap, ok := have.(map[string]interface{}) haveMap, ok := have.(map[string]interface{})
if !ok { if !ok {
mismatch(t, key, "table", want, haveMap) mismatch(t, key, "table", want, haveMap)
@@ -61,6 +63,7 @@ func cmpJSONMaps(t *testing.T, key string, want map[string]interface{}, have int
} }
func cmpJSONArrays(t *testing.T, key string, want, have interface{}) { func cmpJSONArrays(t *testing.T, key string, want, have interface{}) {
t.Helper()
wantSlice, ok := want.([]interface{}) wantSlice, ok := want.([]interface{})
if !ok { if !ok {
panic(fmt.Sprintf("'value' should be a JSON array when 'type=array', but it is a %T", want)) panic(fmt.Sprintf("'value' should be a JSON array when 'type=array', but it is a %T", want))
@@ -83,6 +86,7 @@ func cmpJSONArrays(t *testing.T, key string, want, have interface{}) {
} }
func cmpJSONValues(t *testing.T, key string, want, have map[string]interface{}) { func cmpJSONValues(t *testing.T, key string, want, have map[string]interface{}) {
t.Helper()
wantType, ok := want["type"].(string) wantType, ok := want["type"].(string)
if !ok { if !ok {
panic(fmt.Sprintf("'type' should be a string, but it is a %T", want["type"])) panic(fmt.Sprintf("'type' should be a string, but it is a %T", want["type"]))
@@ -126,6 +130,7 @@ func cmpJSONValues(t *testing.T, key string, want, have map[string]interface{})
} }
func cmpAsStrings(t *testing.T, key string, want, have string) { func cmpAsStrings(t *testing.T, key string, want, have string) {
t.Helper()
if want != have { if want != have {
t.Fatalf("Values for key '%s' don't match:\n"+ t.Fatalf("Values for key '%s' don't match:\n"+
" Expected: %s\n"+ " Expected: %s\n"+
@@ -135,6 +140,7 @@ func cmpAsStrings(t *testing.T, key string, want, have string) {
} }
func cmpFloats(t *testing.T, key string, want, have string) { func cmpFloats(t *testing.T, key string, want, have string) {
t.Helper()
// Special case for NaN, since NaN != NaN. // Special case for NaN, since NaN != NaN.
if strings.HasSuffix(want, "nan") || strings.HasSuffix(have, "nan") { if strings.HasSuffix(want, "nan") || strings.HasSuffix(have, "nan") {
if want != have { if want != have {
@@ -177,6 +183,7 @@ var layouts = map[string]string{
} }
func cmpAsDatetimes(t *testing.T, key string, kind, want, have string) { func cmpAsDatetimes(t *testing.T, key string, kind, want, have string) {
t.Helper()
layout, ok := layouts[kind] layout, ok := layouts[kind]
if !ok { if !ok {
panic("should never happen") panic("should never happen")
@@ -200,15 +207,6 @@ func cmpAsDatetimes(t *testing.T, key string, kind, want, have string) {
} }
} }
func cmpAsDatetimesLocal(t *testing.T, key string, want, have string) {
if datetimeRepl.Replace(want) != datetimeRepl.Replace(have) {
t.Fatalf("Values for key '%s' don't match:\n"+
" Expected: %v\n"+
" Your encoder: %v",
key, want, have)
}
}
func kjoin(old, key string) string { func kjoin(old, key string) string {
if len(old) == 0 { if len(old) == 0 {
return key return key
@@ -230,6 +228,7 @@ func isValue(m map[string]interface{}) bool {
} }
func mismatch(t *testing.T, key string, wantType string, want, have interface{}) { func mismatch(t *testing.T, key string, wantType string, want, have interface{}) {
t.Helper()
t.Fatalf("Key '%s' is not an %s but %[4]T:\n"+ t.Fatalf("Key '%s' is not an %s but %[4]T:\n"+
" Expected: %#[3]v\n"+ " Expected: %#[3]v\n"+
" Your encoder: %#[4]v", " Your encoder: %#[4]v",
@@ -237,8 +236,9 @@ func mismatch(t *testing.T, key string, wantType string, want, have interface{})
} }
func valMismatch(t *testing.T, key string, wantType, haveType string, want, have interface{}) { func valMismatch(t *testing.T, key string, wantType, haveType string, want, have interface{}) {
t.Helper()
t.Fatalf("Key '%s' is not an %s but %s:\n"+ t.Fatalf("Key '%s' is not an %s but %s:\n"+
" Expected: %#[3]v\n"+ " Expected: %#[3]v\n"+
" Your encoder: %#[4]v", " Your encoder: %#[4]v",
key, wantType, want, have) key, wantType, haveType, want, have)
} }
+27 -21
View File
@@ -4,10 +4,12 @@ import (
"fmt" "fmt"
"strconv" "strconv"
"time" "time"
"github.com/pelletier/go-toml/v2"
) )
// Remove JSON tags to a data structure as returned by toml-test. // Remove JSON tags to a data structure as returned by toml-test.
func rmTag(typedJson interface{}) (interface{}, error) { func rmTag(typedJSON interface{}) (interface{}, error) {
// Check if key is in the table m. // Check if key is in the table m.
in := func(key string, m map[string]interface{}) bool { in := func(key string, m map[string]interface{}) bool {
_, ok := m[key] _, ok := m[key]
@@ -15,8 +17,7 @@ func rmTag(typedJson interface{}) (interface{}, error) {
} }
// Switch on the data type. // Switch on the data type.
switch v := typedJson.(type) { switch v := typedJSON.(type) {
// Object: this can either be a TOML table or a primitive with tags. // Object: this can either be a TOML table or a primitive with tags.
case map[string]interface{}: case map[string]interface{}:
// This value represents a primitive: remove the tags and return just // This value represents a primitive: remove the tags and return just
@@ -40,7 +41,7 @@ func rmTag(typedJson interface{}) (interface{}, error) {
} }
return m, nil return m, nil
// Array: remove tags from all itenm. // Array: remove tags from all items.
case []interface{}: case []interface{}:
a := make([]interface{}, len(v)) a := make([]interface{}, len(v))
for i := range v { for i := range v {
@@ -54,7 +55,7 @@ func rmTag(typedJson interface{}) (interface{}, error) {
} }
// The top level must be an object or array. // The top level must be an object or array.
return nil, fmt.Errorf("unrecognized JSON format '%T'", typedJson) return nil, fmt.Errorf("unrecognized JSON format '%T'", typedJSON)
} }
// Return a primitive: read the "type" and convert the "value" to that. // Return a primitive: read the "type" and convert the "value" to that.
@@ -76,14 +77,31 @@ func untag(typed map[string]interface{}) (interface{}, error) {
return nil, fmt.Errorf("untag: %w", err) return nil, fmt.Errorf("untag: %w", err)
} }
return f, nil return f, nil
// toml.LocalDate{Year:2020, Month:12, Day:12}
case "datetime": case "datetime":
return parseTime(v, "2006-01-02T15:04:05.999999999Z07:00", false) return time.Parse("2006-01-02T15:04:05.999999999Z07:00", v)
case "datetime-local": case "datetime-local":
return parseTime(v, "2006-01-02T15:04:05.999999999", true) var t toml.LocalDateTime
err := t.UnmarshalText([]byte(v))
if err != nil {
return nil, fmt.Errorf("untag: %w", err)
}
return t, nil
case "date-local": case "date-local":
return parseTime(v, "2006-01-02", true) var t toml.LocalDate
err := t.UnmarshalText([]byte(v))
if err != nil {
return nil, fmt.Errorf("untag: %w", err)
}
return t, nil
case "time-local": case "time-local":
return parseTime(v, "15:04:05.999999999", true) var t toml.LocalTime
err := t.UnmarshalText([]byte(v))
if err != nil {
return nil, fmt.Errorf("untag: %w", err)
}
return t, nil
case "bool": case "bool":
switch v { switch v {
case "true": case "true":
@@ -96,15 +114,3 @@ func untag(typed map[string]interface{}) (interface{}, error) {
return nil, fmt.Errorf("untag: unrecognized tag type %q", t) return nil, fmt.Errorf("untag: unrecognized tag type %q", t)
} }
func parseTime(v, format string, local bool) (t time.Time, err error) {
if local {
t, err = time.ParseInLocation(format, v, time.Local)
} else {
t, err = time.Parse(format, v)
}
if err != nil {
return time.Time{}, fmt.Errorf("Could not parse %q as a datetime: %w", v, err)
}
return t, nil
}
@@ -10,7 +10,7 @@ import (
"github.com/pelletier/go-toml/v2" "github.com/pelletier/go-toml/v2"
) )
// Marshal is a helpfer function for calling toml.Marshal // Marshal is a helper function for calling toml.Marshal
// //
// Only needed to avoid package import loops. // Only needed to avoid package import loops.
func Marshal(v interface{}) ([]byte, error) { func Marshal(v interface{}) ([]byte, error) {
@@ -27,7 +27,7 @@ func Unmarshal(data []byte, v interface{}) error {
// ValueToTaggedJSON takes a data structure and returns the tagged JSON // ValueToTaggedJSON takes a data structure and returns the tagged JSON
// representation. // representation.
func ValueToTaggedJSON(doc interface{}) ([]byte, error) { func ValueToTaggedJSON(doc interface{}) ([]byte, error) {
return json.MarshalIndent(addTag("", doc), "", " ") return json.MarshalIndent(addTag(doc), "", " ")
} }
// DecodeStdin is a helper function for the toml-test binary interface. TOML input // DecodeStdin is a helper function for the toml-test binary interface. TOML input
@@ -37,14 +37,32 @@ func DecodeStdin() error {
var decoded map[string]interface{} var decoded map[string]interface{}
if err := toml.NewDecoder(os.Stdin).Decode(&decoded); err != nil { if err := toml.NewDecoder(os.Stdin).Decode(&decoded); err != nil {
return fmt.Errorf("Error decoding TOML: %s", err) return fmt.Errorf("error decoding TOML: %w", err)
} }
j := json.NewEncoder(os.Stdout) j := json.NewEncoder(os.Stdout)
j.SetIndent("", " ") j.SetIndent("", " ")
if err := j.Encode(addTag("", decoded)); err != nil { if err := j.Encode(addTag(decoded)); err != nil {
return fmt.Errorf("Error encoding JSON: %s", err) return fmt.Errorf("error encoding JSON: %w", err)
} }
return nil return nil
} }
// EncodeStdin is a helper function for the toml-test binary interface. Tagged
// JSON is read from STDIN and a resulting TOML representation is written to
// STDOUT.
func EncodeStdin() error {
var j interface{}
err := json.NewDecoder(os.Stdin).Decode(&j)
if err != nil {
return err
}
rm, err := rmTag(j)
if err != nil {
return fmt.Errorf("removing tags: %w", err)
}
return toml.NewEncoder(os.Stdout).Encode(rm)
}
+6 -8
View File
@@ -1,8 +1,6 @@
package tracker package tracker
import ( import "github.com/pelletier/go-toml/v2/unstable"
"github.com/pelletier/go-toml/v2/internal/ast"
)
// KeyTracker is a tracker that keeps track of the current Key as the AST is // KeyTracker is a tracker that keeps track of the current Key as the AST is
// walked. // walked.
@@ -11,19 +9,19 @@ type KeyTracker struct {
} }
// UpdateTable sets the state of the tracker with the AST table node. // UpdateTable sets the state of the tracker with the AST table node.
func (t *KeyTracker) UpdateTable(node *ast.Node) { func (t *KeyTracker) UpdateTable(node *unstable.Node) {
t.reset() t.reset()
t.Push(node) t.Push(node)
} }
// UpdateArrayTable sets the state of the tracker with the AST array table node. // UpdateArrayTable sets the state of the tracker with the AST array table node.
func (t *KeyTracker) UpdateArrayTable(node *ast.Node) { func (t *KeyTracker) UpdateArrayTable(node *unstable.Node) {
t.reset() t.reset()
t.Push(node) t.Push(node)
} }
// Push the given key on the stack. // Push the given key on the stack.
func (t *KeyTracker) Push(node *ast.Node) { func (t *KeyTracker) Push(node *unstable.Node) {
it := node.Key() it := node.Key()
for it.Next() { for it.Next() {
t.k = append(t.k, string(it.Node().Data)) t.k = append(t.k, string(it.Node().Data))
@@ -31,14 +29,14 @@ func (t *KeyTracker) Push(node *ast.Node) {
} }
// Pop key from stack. // Pop key from stack.
func (t *KeyTracker) Pop(node *ast.Node) { func (t *KeyTracker) Pop(node *unstable.Node) {
it := node.Key() it := node.Key()
for it.Next() { for it.Next() {
t.k = t.k[:len(t.k)-1] t.k = t.k[:len(t.k)-1]
} }
} }
// Key returns the current key // Key returns the current key.
func (t *KeyTracker) Key() []string { func (t *KeyTracker) Key() []string {
k := make([]string, len(t.k)) k := make([]string, len(t.k))
copy(k, t.k) copy(k, t.k)
+63 -55
View File
@@ -5,7 +5,7 @@ import (
"fmt" "fmt"
"sync" "sync"
"github.com/pelletier/go-toml/v2/internal/ast" "github.com/pelletier/go-toml/v2/unstable"
) )
type keyKind uint8 type keyKind uint8
@@ -57,7 +57,11 @@ type SeenTracker struct {
currentIdx int currentIdx int
} }
var pool sync.Pool var pool = sync.Pool{
New: func() interface{} {
return &SeenTracker{}
},
}
func (s *SeenTracker) reset() { func (s *SeenTracker) reset() {
// Always contains a root element at index 0. // Always contains a root element at index 0.
@@ -79,6 +83,7 @@ type entry struct {
name []byte name []byte
kind keyKind kind keyKind
explicit bool explicit bool
kv bool
} }
// Find the index of the child of parentIdx with key k. Returns -1 if // Find the index of the child of parentIdx with key k. Returns -1 if
@@ -111,7 +116,7 @@ func (s *SeenTracker) clear(idx int) {
s.entries[idx].child = -1 s.entries[idx].child = -1
} }
func (s *SeenTracker) create(parentIdx int, name []byte, kind keyKind, explicit bool) int { func (s *SeenTracker) create(parentIdx int, name []byte, kind keyKind, explicit bool, kv bool) int {
e := entry{ e := entry{
child: -1, child: -1,
next: s.entries[parentIdx].child, next: s.entries[parentIdx].child,
@@ -119,6 +124,7 @@ func (s *SeenTracker) create(parentIdx int, name []byte, kind keyKind, explicit
name: name, name: name,
kind: kind, kind: kind,
explicit: explicit, explicit: explicit,
kv: kv,
} }
var idx int var idx int
if s.entries[0].next >= 0 { if s.entries[0].next >= 0 {
@@ -137,31 +143,35 @@ func (s *SeenTracker) create(parentIdx int, name []byte, kind keyKind, explicit
func (s *SeenTracker) setExplicitFlag(parentIdx int) { func (s *SeenTracker) setExplicitFlag(parentIdx int) {
for i := s.entries[parentIdx].child; i >= 0; i = s.entries[i].next { for i := s.entries[parentIdx].child; i >= 0; i = s.entries[i].next {
s.entries[i].explicit = true if s.entries[i].kv {
s.entries[i].explicit = true
s.entries[i].kv = false
}
s.setExplicitFlag(i) s.setExplicitFlag(i)
} }
} }
// CheckExpression takes a top-level node and checks that it does not contain // CheckExpression takes a top-level node and checks that it does not contain
// keys that have been seen in previous calls, and validates that types are // keys that have been seen in previous calls, and validates that types are
// consistent. // consistent. It returns true if it is the first time this node's key is seen.
func (s *SeenTracker) CheckExpression(node *ast.Node) error { // Useful to clear array tables on first use.
func (s *SeenTracker) CheckExpression(node *unstable.Node) (bool, error) {
if s.entries == nil { if s.entries == nil {
s.reset() s.reset()
} }
switch node.Kind { switch node.Kind {
case ast.KeyValue: case unstable.KeyValue:
return s.checkKeyValue(node) return s.checkKeyValue(node)
case ast.Table: case unstable.Table:
return s.checkTable(node) return s.checkTable(node)
case ast.ArrayTable: case unstable.ArrayTable:
return s.checkArrayTable(node) return s.checkArrayTable(node)
default: default:
panic(fmt.Errorf("this should not be a top level node type: %s", node.Kind)) panic(fmt.Errorf("this should not be a top level node type: %s", node.Kind))
} }
} }
func (s *SeenTracker) checkTable(node *ast.Node) error { func (s *SeenTracker) checkTable(node *unstable.Node) (bool, error) {
if s.currentIdx >= 0 { if s.currentIdx >= 0 {
s.setExplicitFlag(s.currentIdx) s.setExplicitFlag(s.currentIdx)
} }
@@ -183,11 +193,11 @@ func (s *SeenTracker) checkTable(node *ast.Node) error {
idx := s.find(parentIdx, k) idx := s.find(parentIdx, k)
if idx < 0 { if idx < 0 {
idx = s.create(parentIdx, k, tableKind, false) idx = s.create(parentIdx, k, tableKind, false, false)
} else { } else {
entry := s.entries[idx] entry := s.entries[idx]
if entry.kind == valueKind { if entry.kind == valueKind {
return fmt.Errorf("toml: expected %s to be a table, not a %s", string(k), entry.kind) return false, fmt.Errorf("toml: expected %s to be a table, not a %s", string(k), entry.kind)
} }
} }
parentIdx = idx parentIdx = idx
@@ -196,25 +206,27 @@ func (s *SeenTracker) checkTable(node *ast.Node) error {
k := it.Node().Data k := it.Node().Data
idx := s.find(parentIdx, k) idx := s.find(parentIdx, k)
first := false
if idx >= 0 { if idx >= 0 {
kind := s.entries[idx].kind kind := s.entries[idx].kind
if kind != tableKind { if kind != tableKind {
return fmt.Errorf("toml: key %s should be a table, not a %s", string(k), kind) return false, fmt.Errorf("toml: key %s should be a table, not a %s", string(k), kind)
} }
if s.entries[idx].explicit { if s.entries[idx].explicit {
return fmt.Errorf("toml: table %s already exists", string(k)) return false, fmt.Errorf("toml: table %s already exists", string(k))
} }
s.entries[idx].explicit = true s.entries[idx].explicit = true
} else { } else {
idx = s.create(parentIdx, k, tableKind, true) idx = s.create(parentIdx, k, tableKind, true, false)
first = true
} }
s.currentIdx = idx s.currentIdx = idx
return nil return first, nil
} }
func (s *SeenTracker) checkArrayTable(node *ast.Node) error { func (s *SeenTracker) checkArrayTable(node *unstable.Node) (bool, error) {
if s.currentIdx >= 0 { if s.currentIdx >= 0 {
s.setExplicitFlag(s.currentIdx) s.setExplicitFlag(s.currentIdx)
} }
@@ -233,11 +245,11 @@ func (s *SeenTracker) checkArrayTable(node *ast.Node) error {
idx := s.find(parentIdx, k) idx := s.find(parentIdx, k)
if idx < 0 { if idx < 0 {
idx = s.create(parentIdx, k, tableKind, false) idx = s.create(parentIdx, k, tableKind, false, false)
} else { } else {
entry := s.entries[idx] entry := s.entries[idx]
if entry.kind == valueKind { if entry.kind == valueKind {
return fmt.Errorf("toml: expected %s to be a table, not a %s", string(k), entry.kind) return false, fmt.Errorf("toml: expected %s to be a table, not a %s", string(k), entry.kind)
} }
} }
@@ -247,22 +259,23 @@ func (s *SeenTracker) checkArrayTable(node *ast.Node) error {
k := it.Node().Data k := it.Node().Data
idx := s.find(parentIdx, k) idx := s.find(parentIdx, k)
if idx >= 0 { firstTime := idx < 0
if firstTime {
idx = s.create(parentIdx, k, arrayTableKind, true, false)
} else {
kind := s.entries[idx].kind kind := s.entries[idx].kind
if kind != arrayTableKind { if kind != arrayTableKind {
return fmt.Errorf("toml: key %s already exists as a %s, but should be an array table", kind, string(k)) return false, fmt.Errorf("toml: key %s already exists as a %s, but should be an array table", kind, string(k))
} }
s.clear(idx) s.clear(idx)
} else {
idx = s.create(parentIdx, k, arrayTableKind, true)
} }
s.currentIdx = idx s.currentIdx = idx
return nil return firstTime, nil
} }
func (s *SeenTracker) checkKeyValue(node *ast.Node) error { func (s *SeenTracker) checkKeyValue(node *unstable.Node) (bool, error) {
parentIdx := s.currentIdx parentIdx := s.currentIdx
it := node.Key() it := node.Key()
@@ -272,15 +285,16 @@ func (s *SeenTracker) checkKeyValue(node *ast.Node) error {
idx := s.find(parentIdx, k) idx := s.find(parentIdx, k)
if idx < 0 { if idx < 0 {
idx = s.create(parentIdx, k, tableKind, false) idx = s.create(parentIdx, k, tableKind, false, true)
} else { } else {
entry := s.entries[idx] entry := s.entries[idx]
if it.IsLast() { switch {
return fmt.Errorf("toml: key %s is already defined", string(k)) case it.IsLast():
} else if entry.kind != tableKind { return false, fmt.Errorf("toml: key %s is already defined", string(k))
return fmt.Errorf("toml: expected %s to be a table, not a %s", string(k), entry.kind) case entry.kind != tableKind:
} else if entry.explicit { return false, fmt.Errorf("toml: expected %s to be a table, not a %s", string(k), entry.kind)
return fmt.Errorf("toml: cannot redefine table %s that has already been explicitly defined", string(k)) case entry.explicit:
return false, fmt.Errorf("toml: cannot redefine table %s that has already been explicitly defined", string(k))
} }
} }
@@ -292,51 +306,45 @@ func (s *SeenTracker) checkKeyValue(node *ast.Node) error {
value := node.Value() value := node.Value()
switch value.Kind { switch value.Kind {
case ast.InlineTable: case unstable.InlineTable:
return s.checkInlineTable(value) return s.checkInlineTable(value)
case ast.Array: case unstable.Array:
return s.checkArray(value) return s.checkArray(value)
default:
return false, nil
} }
return nil
} }
func (s *SeenTracker) checkArray(node *ast.Node) error { func (s *SeenTracker) checkArray(node *unstable.Node) (first bool, err error) {
it := node.Children() it := node.Children()
for it.Next() { for it.Next() {
n := it.Node() n := it.Node()
switch n.Kind { switch n.Kind { //nolint:exhaustive
case ast.InlineTable: case unstable.InlineTable:
err := s.checkInlineTable(n) first, err = s.checkInlineTable(n)
if err != nil { if err != nil {
return err return false, err
} }
case ast.Array: case unstable.Array:
err := s.checkArray(n) first, err = s.checkArray(n)
if err != nil { if err != nil {
return err return false, err
} }
} }
} }
return nil return first, nil
} }
func (s *SeenTracker) checkInlineTable(node *ast.Node) error { func (s *SeenTracker) checkInlineTable(node *unstable.Node) (first bool, err error) {
if pool.New == nil {
pool.New = func() interface{} {
return &SeenTracker{}
}
}
s = pool.Get().(*SeenTracker) s = pool.Get().(*SeenTracker)
s.reset() s.reset()
it := node.Children() it := node.Children()
for it.Next() { for it.Next() {
n := it.Node() n := it.Node()
err := s.checkKeyValue(n) first, err = s.checkKeyValue(n)
if err != nil { if err != nil {
return err return false, err
} }
} }
@@ -347,5 +355,5 @@ func (s *SeenTracker) checkInlineTable(node *ast.Node) error {
// redefinition of its keys: check* functions cannot walk into // redefinition of its keys: check* functions cannot walk into
// a value. // a value.
pool.Put(s) pool.Put(s)
return nil return first, nil
} }
+21
View File
@@ -0,0 +1,21 @@
package tracker
import (
"reflect"
"testing"
"github.com/pelletier/go-toml/v2/internal/assert"
)
func TestEntrySize(t *testing.T) {
// Validate no regression on the size of entry{}. This is a critical bit for
// performance of unmarshaling documents. Should only be increased with care
// and a very good reason.
maxExpectedEntrySize := 48
entrySize := int(reflect.TypeOf(entry{}).Size())
assert.True(t,
entrySize <= maxExpectedEntrySize,
"Expected entry to be less than or equal to %d, got: %d",
maxExpectedEntrySize, entrySize,
)
}
+1
View File
@@ -1 +1,2 @@
// Package tracker provides functions for keeping track of AST nodes.
package tracker package tracker
+5 -3
View File
@@ -4,6 +4,8 @@ import (
"fmt" "fmt"
"strings" "strings"
"time" "time"
"github.com/pelletier/go-toml/v2/unstable"
) )
// LocalDate represents a calendar day in no specific timezone. // LocalDate represents a calendar day in no specific timezone.
@@ -43,7 +45,7 @@ func (d *LocalDate) UnmarshalText(b []byte) error {
type LocalTime struct { type LocalTime struct {
Hour int // Hour of the day: [0; 24[ Hour int // Hour of the day: [0; 24[
Minute int // Minute of the hour: [0; 60[ Minute int // Minute of the hour: [0; 60[
Second int // Second of the minute: [0; 60[ Second int // Second of the minute: [0; 59]
Nanosecond int // Nanoseconds within the second: [0, 1000000000[ Nanosecond int // Nanoseconds within the second: [0, 1000000000[
Precision int // Number of digits to display for Nanosecond. Precision int // Number of digits to display for Nanosecond.
} }
@@ -75,7 +77,7 @@ func (d LocalTime) MarshalText() ([]byte, error) {
func (d *LocalTime) UnmarshalText(b []byte) error { func (d *LocalTime) UnmarshalText(b []byte) error {
res, left, err := parseLocalTime(b) res, left, err := parseLocalTime(b)
if err == nil && len(left) != 0 { if err == nil && len(left) != 0 {
err = newDecodeError(left, "extra characters") err = unstable.NewParserError(left, "extra characters")
} }
if err != nil { if err != nil {
return err return err
@@ -109,7 +111,7 @@ func (d LocalDateTime) MarshalText() ([]byte, error) {
func (d *LocalDateTime) UnmarshalText(data []byte) error { func (d *LocalDateTime) UnmarshalText(data []byte) error {
res, left, err := parseLocalDateTime(data) res, left, err := parseLocalDateTime(data)
if err == nil && len(left) != 0 { if err == nil && len(left) != 0 {
err = newDecodeError(left, "extra characters") err = unstable.NewParserError(left, "extra characters")
} }
if err != nil { if err != nil {
return err return err
+35 -28
View File
@@ -5,73 +5,80 @@ import (
"time" "time"
"github.com/pelletier/go-toml/v2" "github.com/pelletier/go-toml/v2"
"github.com/stretchr/testify/require" "github.com/pelletier/go-toml/v2/internal/assert"
) )
func TestLocalDate_AsTime(t *testing.T) { func TestLocalDate_AsTime(t *testing.T) {
d := toml.LocalDate{2021, 6, 8} d := toml.LocalDate{2021, 6, 8}
cast := d.AsTime(time.UTC) cast := d.AsTime(time.UTC)
require.Equal(t, time.Date(2021, time.June, 8, 0, 0, 0, 0, time.UTC), cast) assert.Equal(t, time.Date(2021, time.June, 8, 0, 0, 0, 0, time.UTC), cast)
} }
func TestLocalDate_String(t *testing.T) { func TestLocalDate_String(t *testing.T) {
d := toml.LocalDate{2021, 6, 8} d := toml.LocalDate{2021, 6, 8}
require.Equal(t, "2021-06-08", d.String()) assert.Equal(t, "2021-06-08", d.String())
} }
func TestLocalDate_MarshalText(t *testing.T) { func TestLocalDate_MarshalText(t *testing.T) {
d := toml.LocalDate{2021, 6, 8} d := toml.LocalDate{2021, 6, 8}
b, err := d.MarshalText() b, err := d.MarshalText()
require.NoError(t, err) assert.NoError(t, err)
require.Equal(t, []byte("2021-06-08"), b) assert.Equal(t, []byte("2021-06-08"), b)
} }
func TestLocalDate_UnmarshalMarshalText(t *testing.T) { func TestLocalDate_UnmarshalMarshalText(t *testing.T) {
d := toml.LocalDate{} d := toml.LocalDate{}
err := d.UnmarshalText([]byte("2021-06-08")) err := d.UnmarshalText([]byte("2021-06-08"))
require.NoError(t, err) assert.NoError(t, err)
require.Equal(t, toml.LocalDate{2021, 6, 8}, d) assert.Equal(t, toml.LocalDate{2021, 6, 8}, d)
err = d.UnmarshalText([]byte("what")) err = d.UnmarshalText([]byte("what"))
require.Error(t, err) assert.Error(t, err)
} }
func TestLocalTime_String(t *testing.T) { func TestLocalTime_String(t *testing.T) {
d := toml.LocalTime{20, 12, 1, 2, 9} d := toml.LocalTime{20, 12, 1, 2, 9}
require.Equal(t, "20:12:01.000000002", d.String()) assert.Equal(t, "20:12:01.000000002", d.String())
d = toml.LocalTime{20, 12, 1, 0, 0} d = toml.LocalTime{20, 12, 1, 0, 0}
require.Equal(t, "20:12:01", d.String()) assert.Equal(t, "20:12:01", d.String())
d = toml.LocalTime{20, 12, 1, 0, 9} d = toml.LocalTime{20, 12, 1, 0, 9}
require.Equal(t, "20:12:01.000000000", d.String()) assert.Equal(t, "20:12:01.000000000", d.String())
d = toml.LocalTime{20, 12, 1, 100, 0} d = toml.LocalTime{20, 12, 1, 100, 0}
require.Equal(t, "20:12:01.0000001", d.String()) assert.Equal(t, "20:12:01.0000001", d.String())
} }
func TestLocalTime_MarshalText(t *testing.T) { func TestLocalTime_MarshalText(t *testing.T) {
d := toml.LocalTime{20, 12, 1, 2, 9} d := toml.LocalTime{20, 12, 1, 2, 9}
b, err := d.MarshalText() b, err := d.MarshalText()
require.NoError(t, err) assert.NoError(t, err)
require.Equal(t, []byte("20:12:01.000000002"), b) assert.Equal(t, []byte("20:12:01.000000002"), b)
} }
func TestLocalTime_UnmarshalMarshalText(t *testing.T) { func TestLocalTime_UnmarshalMarshalText(t *testing.T) {
d := toml.LocalTime{} d := toml.LocalTime{}
err := d.UnmarshalText([]byte("20:12:01.000000002")) err := d.UnmarshalText([]byte("20:12:01.000000002"))
require.NoError(t, err) assert.NoError(t, err)
require.Equal(t, toml.LocalTime{20, 12, 1, 2, 9}, d) assert.Equal(t, toml.LocalTime{20, 12, 1, 2, 9}, d)
err = d.UnmarshalText([]byte("what")) err = d.UnmarshalText([]byte("what"))
require.Error(t, err) assert.Error(t, err)
err = d.UnmarshalText([]byte("20:12:01.000000002 bad")) err = d.UnmarshalText([]byte("20:12:01.000000002 bad"))
require.Error(t, err) assert.Error(t, err)
}
func TestLocalTime_UnmarshalText_WithoutSeconds(t *testing.T) {
d := toml.LocalTime{}
err := d.UnmarshalText([]byte("14:15"))
assert.NoError(t, err)
assert.Equal(t, toml.LocalTime{14, 15, 0, 0, 0}, d)
} }
func TestLocalTime_RoundTrip(t *testing.T) { func TestLocalTime_RoundTrip(t *testing.T) {
var d struct{ A toml.LocalTime } var d struct{ A toml.LocalTime }
err := toml.Unmarshal([]byte("a=20:12:01.500"), &d) err := toml.Unmarshal([]byte("a=20:12:01.500"), &d)
require.NoError(t, err) assert.NoError(t, err)
require.Equal(t, "20:12:01.500", d.A.String()) assert.Equal(t, "20:12:01.500", d.A.String())
} }
func TestLocalDateTime_AsTime(t *testing.T) { func TestLocalDateTime_AsTime(t *testing.T) {
@@ -80,7 +87,7 @@ func TestLocalDateTime_AsTime(t *testing.T) {
toml.LocalTime{20, 12, 1, 2, 9}, toml.LocalTime{20, 12, 1, 2, 9},
} }
cast := d.AsTime(time.UTC) cast := d.AsTime(time.UTC)
require.Equal(t, time.Date(2021, time.June, 8, 20, 12, 1, 2, time.UTC), cast) assert.Equal(t, time.Date(2021, time.June, 8, 20, 12, 1, 2, time.UTC), cast)
} }
func TestLocalDateTime_String(t *testing.T) { func TestLocalDateTime_String(t *testing.T) {
@@ -88,7 +95,7 @@ func TestLocalDateTime_String(t *testing.T) {
toml.LocalDate{2021, 6, 8}, toml.LocalDate{2021, 6, 8},
toml.LocalTime{20, 12, 1, 2, 9}, toml.LocalTime{20, 12, 1, 2, 9},
} }
require.Equal(t, "2021-06-08T20:12:01.000000002", d.String()) assert.Equal(t, "2021-06-08T20:12:01.000000002", d.String())
} }
func TestLocalDateTime_MarshalText(t *testing.T) { func TestLocalDateTime_MarshalText(t *testing.T) {
@@ -97,22 +104,22 @@ func TestLocalDateTime_MarshalText(t *testing.T) {
toml.LocalTime{20, 12, 1, 2, 9}, toml.LocalTime{20, 12, 1, 2, 9},
} }
b, err := d.MarshalText() b, err := d.MarshalText()
require.NoError(t, err) assert.NoError(t, err)
require.Equal(t, []byte("2021-06-08T20:12:01.000000002"), b) assert.Equal(t, []byte("2021-06-08T20:12:01.000000002"), b)
} }
func TestLocalDateTime_UnmarshalMarshalText(t *testing.T) { func TestLocalDateTime_UnmarshalMarshalText(t *testing.T) {
d := toml.LocalDateTime{} d := toml.LocalDateTime{}
err := d.UnmarshalText([]byte("2021-06-08 20:12:01.000000002")) err := d.UnmarshalText([]byte("2021-06-08 20:12:01.000000002"))
require.NoError(t, err) assert.NoError(t, err)
require.Equal(t, toml.LocalDateTime{ assert.Equal(t, toml.LocalDateTime{
toml.LocalDate{2021, 6, 8}, toml.LocalDate{2021, 6, 8},
toml.LocalTime{20, 12, 1, 2, 9}, toml.LocalTime{20, 12, 1, 2, 9},
}, d) }, d)
err = d.UnmarshalText([]byte("what")) err = d.UnmarshalText([]byte("what"))
require.Error(t, err) assert.Error(t, err)
err = d.UnmarshalText([]byte("2021-06-08 20:12:01.000000002 bad")) err = d.UnmarshalText([]byte("2021-06-08 20:12:01.000000002 bad"))
require.Error(t, err) assert.Error(t, err)
} }
+397 -108
View File
@@ -3,15 +3,19 @@ package toml
import ( import (
"bytes" "bytes"
"encoding" "encoding"
"encoding/json"
"errors"
"fmt" "fmt"
"io" "io"
"math" "math"
"reflect" "reflect"
"sort" "slices"
"strconv" "strconv"
"strings" "strings"
"time" "time"
"unicode" "unicode"
"github.com/pelletier/go-toml/v2/internal/characters"
) )
// Marshal serializes a Go value as a TOML document. // Marshal serializes a Go value as a TOML document.
@@ -35,10 +39,11 @@ type Encoder struct {
w io.Writer w io.Writer
// global settings // global settings
tablesInline bool tablesInline bool
arraysMultiline bool arraysMultiline bool
indentSymbol string indentSymbol string
indentTables bool indentTables bool
marshalJSONNumbers bool
} }
// NewEncoder returns a new Encoder that writes to w. // NewEncoder returns a new Encoder that writes to w.
@@ -54,7 +59,7 @@ func NewEncoder(w io.Writer) *Encoder {
// This behavior can be controlled on an individual struct field basis with the // This behavior can be controlled on an individual struct field basis with the
// inline tag: // inline tag:
// //
// MyField `inline:"true"` // MyField `toml:",inline"`
func (enc *Encoder) SetTablesInline(inline bool) *Encoder { func (enc *Encoder) SetTablesInline(inline bool) *Encoder {
enc.tablesInline = inline enc.tablesInline = inline
return enc return enc
@@ -65,7 +70,7 @@ func (enc *Encoder) SetTablesInline(inline bool) *Encoder {
// //
// This behavior can be controlled on an individual struct field basis with the multiline tag: // This behavior can be controlled on an individual struct field basis with the multiline tag:
// //
// MyField `multiline:"true"` // MyField `multiline:"true"`
func (enc *Encoder) SetArraysMultiline(multiline bool) *Encoder { func (enc *Encoder) SetArraysMultiline(multiline bool) *Encoder {
enc.arraysMultiline = multiline enc.arraysMultiline = multiline
return enc return enc
@@ -85,11 +90,22 @@ func (enc *Encoder) SetIndentTables(indent bool) *Encoder {
return enc return enc
} }
// SetMarshalJSONNumbers forces the encoder to serialize `json.Number` as a
// float or integer instead of relying on TextMarshaler to emit a string.
//
// *Unstable:* This method does not follow the compatibility guarantees of
// semver. It can be changed or removed without a new major version being
// issued.
func (enc *Encoder) SetMarshalJSONNumbers(indent bool) *Encoder {
enc.marshalJSONNumbers = indent
return enc
}
// Encode writes a TOML representation of v to the stream. // Encode writes a TOML representation of v to the stream.
// //
// If v cannot be represented to TOML it returns an error. // If v cannot be represented to TOML it returns an error.
// //
// Encoding rules // # Encoding rules
// //
// A top level slice containing only maps or structs is encoded as [[table // A top level slice containing only maps or structs is encoded as [[table
// array]]. // array]].
@@ -107,10 +123,30 @@ func (enc *Encoder) SetIndentTables(indent bool) *Encoder {
// a newline character or a single quote. In that case they are emitted as // a newline character or a single quote. In that case they are emitted as
// quoted strings. // quoted strings.
// //
// Unsigned integers larger than math.MaxInt64 cannot be encoded. Doing so
// results in an error. This rule exists because the TOML specification only
// requires parsers to support at least the 64 bits integer range. Allowing
// larger numbers would create non-standard TOML documents, which may not be
// readable (at best) by other implementations. To encode such numbers, a
// solution is a custom type that implements encoding.TextMarshaler.
//
// When encoding structs, fields are encoded in order of definition, with their // When encoding structs, fields are encoded in order of definition, with their
// exact name. // exact name.
// //
// Struct tags // Tables and array tables are separated by empty lines. However, consecutive
// subtables definitions are not. For example:
//
// [top1]
//
// [top2]
// [top2.child1]
//
// [[array]]
//
// [[array]]
// [array.child2]
//
// # Struct tags
// //
// The encoding of each public struct field can be customized by the format // The encoding of each public struct field can be customized by the format
// string in the "toml" key of the struct field's tag. This follows // string in the "toml" key of the struct field's tag. This follows
@@ -126,9 +162,15 @@ func (enc *Encoder) SetIndentTables(indent bool) *Encoder {
// //
// The "omitempty" option prevents empty values or groups from being emitted. // The "omitempty" option prevents empty values or groups from being emitted.
// //
// The "omitzero" option prevents zero values or groups from being emitted.
//
// The "commented" option prefixes the value and all its children with a comment
// symbol.
//
// In addition to the "toml" tag struct tag, a "comment" tag can be used to emit // In addition to the "toml" tag struct tag, a "comment" tag can be used to emit
// a TOML comment before the value being annotated. Comments are ignored inside // a TOML comment before the value being annotated. Comments are ignored inside
// inline tables. // inline tables. For array tables, the comment is only present before the first
// element of the array.
func (enc *Encoder) Encode(v interface{}) error { func (enc *Encoder) Encode(v interface{}) error {
var ( var (
b []byte b []byte
@@ -138,7 +180,7 @@ func (enc *Encoder) Encode(v interface{}) error {
ctx.inline = enc.tablesInline ctx.inline = enc.tablesInline
if v == nil { if v == nil {
return fmt.Errorf("toml: cannot encode a nil interface") return errors.New("toml: cannot encode a nil interface")
} }
b, err := enc.encode(b, ctx, reflect.ValueOf(v)) b, err := enc.encode(b, ctx, reflect.ValueOf(v))
@@ -157,6 +199,8 @@ func (enc *Encoder) Encode(v interface{}) error {
type valueOptions struct { type valueOptions struct {
multiline bool multiline bool
omitempty bool omitempty bool
omitzero bool
commented bool
comment string comment string
} }
@@ -182,6 +226,9 @@ type encoderCtx struct {
// Indentation level // Indentation level
indent int indent int
// Prefix the current value with a comment.
commented bool
// Options coming from struct tags // Options coming from struct tags
options valueOptions options valueOptions
} }
@@ -208,15 +255,35 @@ func (ctx *encoderCtx) isRoot() bool {
} }
func (enc *Encoder) encode(b []byte, ctx encoderCtx, v reflect.Value) ([]byte, error) { func (enc *Encoder) encode(b []byte, ctx encoderCtx, v reflect.Value) ([]byte, error) {
if !v.IsZero() { i := v.Interface()
i, ok := v.Interface().(time.Time)
if ok { switch x := i.(type) {
return i.AppendFormat(b, time.RFC3339), nil case time.Time:
if x.Nanosecond() > 0 {
return x.AppendFormat(b, time.RFC3339Nano), nil
}
return x.AppendFormat(b, time.RFC3339), nil
case LocalTime:
return append(b, x.String()...), nil
case LocalDate:
return append(b, x.String()...), nil
case LocalDateTime:
return append(b, x.String()...), nil
case json.Number:
if enc.marshalJSONNumbers {
if x == "" { /// Useful zero value.
return append(b, "0"...), nil
} else if v, err := x.Int64(); err == nil {
return enc.encode(b, ctx, reflect.ValueOf(v))
} else if f, err := x.Float64(); err == nil {
return enc.encode(b, ctx, reflect.ValueOf(f))
}
return nil, fmt.Errorf("toml: unable to convert %q to int64 or float64", x)
} }
} }
hasTextMarshaler := v.Type().Implements(textMarshalerType) hasTextMarshaler := v.Type().Implements(textMarshalerType)
if hasTextMarshaler || (v.CanAddr() && reflect.PtrTo(v.Type()).Implements(textMarshalerType)) { if hasTextMarshaler || (v.CanAddr() && reflect.PointerTo(v.Type()).Implements(textMarshalerType)) {
if !hasTextMarshaler { if !hasTextMarshaler {
v = v.Addr() v = v.Addr()
} }
@@ -241,11 +308,11 @@ func (enc *Encoder) encode(b []byte, ctx encoderCtx, v reflect.Value) ([]byte, e
return enc.encodeMap(b, ctx, v) return enc.encodeMap(b, ctx, v)
case reflect.Struct: case reflect.Struct:
return enc.encodeStruct(b, ctx, v) return enc.encodeStruct(b, ctx, v)
case reflect.Slice: case reflect.Slice, reflect.Array:
return enc.encodeSlice(b, ctx, v) return enc.encodeSlice(b, ctx, v)
case reflect.Interface: case reflect.Interface:
if v.IsNil() { if v.IsNil() {
return nil, fmt.Errorf("toml: encoding a nil interface is not supported") return nil, errors.New("toml: encoding a nil interface is not supported")
} }
return enc.encode(b, ctx, v.Elem()) return enc.encode(b, ctx, v.Elem())
@@ -260,16 +327,33 @@ func (enc *Encoder) encode(b []byte, ctx encoderCtx, v reflect.Value) ([]byte, e
case reflect.String: case reflect.String:
b = enc.encodeString(b, v.String(), ctx.options) b = enc.encodeString(b, v.String(), ctx.options)
case reflect.Float32: case reflect.Float32:
if math.Trunc(v.Float()) == v.Float() { f := v.Float()
b = strconv.AppendFloat(b, v.Float(), 'f', 1, 32)
} else { switch {
b = strconv.AppendFloat(b, v.Float(), 'f', -1, 32) case math.IsNaN(f):
b = append(b, "nan"...)
case f > math.MaxFloat32:
b = append(b, "inf"...)
case f < -math.MaxFloat32:
b = append(b, "-inf"...)
case math.Trunc(f) == f:
b = strconv.AppendFloat(b, f, 'f', 1, 32)
default:
b = strconv.AppendFloat(b, f, 'f', -1, 32)
} }
case reflect.Float64: case reflect.Float64:
if math.Trunc(v.Float()) == v.Float() { f := v.Float()
b = strconv.AppendFloat(b, v.Float(), 'f', 1, 64) switch {
} else { case math.IsNaN(f):
b = strconv.AppendFloat(b, v.Float(), 'f', -1, 64) b = append(b, "nan"...)
case f > math.MaxFloat64:
b = append(b, "inf"...)
case f < -math.MaxFloat64:
b = append(b, "-inf"...)
case math.Trunc(f) == f:
b = strconv.AppendFloat(b, f, 'f', 1, 64)
default:
b = strconv.AppendFloat(b, f, 'f', -1, 64)
} }
case reflect.Bool: case reflect.Bool:
if v.Bool() { if v.Bool() {
@@ -278,7 +362,11 @@ func (enc *Encoder) encode(b []byte, ctx encoderCtx, v reflect.Value) ([]byte, e
b = append(b, "false"...) b = append(b, "false"...)
} }
case reflect.Uint64, reflect.Uint32, reflect.Uint16, reflect.Uint8, reflect.Uint: case reflect.Uint64, reflect.Uint32, reflect.Uint16, reflect.Uint8, reflect.Uint:
b = strconv.AppendUint(b, v.Uint(), 10) x := v.Uint()
if x > uint64(math.MaxInt64) {
return nil, fmt.Errorf("toml: not encoding uint (%d) greater than max int64 (%d)", x, int64(math.MaxInt64))
}
b = strconv.AppendUint(b, x, 10)
case reflect.Int64, reflect.Int32, reflect.Int16, reflect.Int8, reflect.Int: case reflect.Int64, reflect.Int32, reflect.Int16, reflect.Int8, reflect.Int:
b = strconv.AppendInt(b, v.Int(), 10) b = strconv.AppendInt(b, v.Int(), 10)
default: default:
@@ -297,28 +385,45 @@ func isNil(v reflect.Value) bool {
} }
} }
func shouldOmitEmpty(options valueOptions, v reflect.Value) bool {
return options.omitempty && isEmptyValue(v)
}
func shouldOmitZero(options valueOptions, v reflect.Value) bool {
if !options.omitzero {
return false
}
// Check if the type implements isZeroer interface (has a custom IsZero method).
if v.Type().Implements(isZeroerType) {
return v.Interface().(isZeroer).IsZero()
}
// Check if pointer type implements isZeroer.
if reflect.PointerTo(v.Type()).Implements(isZeroerType) {
if v.CanAddr() {
return v.Addr().Interface().(isZeroer).IsZero()
}
// Create a temporary addressable copy to call the pointer receiver method.
pv := reflect.New(v.Type())
pv.Elem().Set(v)
return pv.Interface().(isZeroer).IsZero()
}
// Fall back to reflect's IsZero for types without custom IsZero method.
return v.IsZero()
}
func (enc *Encoder) encodeKv(b []byte, ctx encoderCtx, options valueOptions, v reflect.Value) ([]byte, error) { func (enc *Encoder) encodeKv(b []byte, ctx encoderCtx, options valueOptions, v reflect.Value) ([]byte, error) {
var err error var err error
if !ctx.hasKey {
panic("caller of encodeKv should have set the key in the context")
}
if (ctx.options.omitempty || options.omitempty) && isEmptyValue(v) {
return b, nil
}
if !ctx.inline { if !ctx.inline {
b = enc.encodeComment(ctx.indent, options.comment, b) b = enc.encodeComment(ctx.indent, options.comment, b)
b = enc.commented(ctx.commented, b)
b = enc.indent(ctx.indent, b)
} }
b = enc.indent(ctx.indent, b) b = enc.encodeKey(b, ctx.key)
b, err = enc.encodeKey(b, ctx.key)
if err != nil {
return nil, err
}
b = append(b, " = "...) b = append(b, " = "...)
// create a copy of the context because the value of a KV shouldn't // create a copy of the context because the value of a KV shouldn't
@@ -336,8 +441,17 @@ func (enc *Encoder) encodeKv(b []byte, ctx encoderCtx, options valueOptions, v r
return b, nil return b, nil
} }
func (enc *Encoder) commented(commented bool, b []byte) []byte {
if commented {
return append(b, "# "...)
}
return b
}
func isEmptyValue(v reflect.Value) bool { func isEmptyValue(v reflect.Value) bool {
switch v.Kind() { switch v.Kind() {
case reflect.Struct:
return isEmptyStruct(v)
case reflect.Array, reflect.Map, reflect.Slice, reflect.String: case reflect.Array, reflect.Map, reflect.Slice, reflect.String:
return v.Len() == 0 return v.Len() == 0
case reflect.Bool: case reflect.Bool:
@@ -350,8 +464,37 @@ func isEmptyValue(v reflect.Value) bool {
return v.Float() == 0 return v.Float() == 0
case reflect.Interface, reflect.Ptr: case reflect.Interface, reflect.Ptr:
return v.IsNil() return v.IsNil()
default:
return false
} }
return false }
func isEmptyStruct(v reflect.Value) bool {
// TODO: merge with walkStruct and cache.
typ := v.Type()
for i := 0; i < typ.NumField(); i++ {
fieldType := typ.Field(i)
// only consider exported fields
if fieldType.PkgPath != "" {
continue
}
tag := fieldType.Tag.Get("toml")
// special field name to skip field
if tag == "-" {
continue
}
f := v.Field(i)
if !isEmptyValue(f) {
return false
}
}
return true
} }
const literalQuote = '\'' const literalQuote = '\''
@@ -365,7 +508,13 @@ func (enc *Encoder) encodeString(b []byte, v string, options valueOptions) []byt
} }
func needsQuoting(v string) bool { func needsQuoting(v string) bool {
return strings.ContainsAny(v, "'\b\f\n\r\t") // TODO: vectorize
for _, b := range []byte(v) {
if b == '\'' || b == '\r' || b == '\n' || characters.InvalidASCII(b) {
return true
}
}
return false
} }
// caller should have checked that the string does not contain new lines or ' . // caller should have checked that the string does not contain new lines or ' .
@@ -377,7 +526,6 @@ func (enc *Encoder) encodeLiteralString(b []byte, v string) []byte {
return b return b
} }
//nolint:cyclop
func (enc *Encoder) encodeQuotedString(multiline bool, b []byte, v string) []byte { func (enc *Encoder) encodeQuotedString(multiline bool, b []byte, v string) []byte {
stringQuote := `"` stringQuote := `"`
@@ -400,12 +548,26 @@ func (enc *Encoder) encodeQuotedString(multiline bool, b []byte, v string) []byt
del = 0x7f del = 0x7f
) )
for _, r := range []byte(v) { bv := []byte(v)
for i := 0; i < len(bv); i++ {
r := bv[i]
switch r { switch r {
case '\\': case '\\':
b = append(b, `\\`...) b = append(b, `\\`...)
case '"': case '"':
b = append(b, `\"`...) if multiline {
// Quotation marks do not need to be quoted in multiline strings unless
// it contains 3 consecutive. If 3+ quotes appear, quote all of them
// because it's visually better
if i+2 > len(bv) || bv[i+1] != '"' || bv[i+2] != '"' {
b = append(b, r)
} else {
b = append(b, `\"\"\"`...)
i += 2
}
} else {
b = append(b, `\"`...)
}
case '\b': case '\b':
b = append(b, `\b`...) b = append(b, `\b`...)
case '\f': case '\f':
@@ -437,57 +599,49 @@ func (enc *Encoder) encodeQuotedString(multiline bool, b []byte, v string) []byt
return b return b
} }
// called should have checked that the string is in A-Z / a-z / 0-9 / - / _ . // caller should have checked that the string is in A-Z / a-z / 0-9 / - / _ .
func (enc *Encoder) encodeUnquotedKey(b []byte, v string) []byte { func (enc *Encoder) encodeUnquotedKey(b []byte, v string) []byte {
return append(b, v...) return append(b, v...)
} }
func (enc *Encoder) encodeTableHeader(ctx encoderCtx, b []byte) ([]byte, error) { func (enc *Encoder) encodeTableHeader(ctx encoderCtx, b []byte) []byte {
if len(ctx.parentKey) == 0 { if len(ctx.parentKey) == 0 {
return b, nil return b
} }
b = enc.encodeComment(ctx.indent, ctx.options.comment, b) b = enc.encodeComment(ctx.indent, ctx.options.comment, b)
b = enc.commented(ctx.commented, b)
b = enc.indent(ctx.indent, b) b = enc.indent(ctx.indent, b)
b = append(b, '[') b = append(b, '[')
var err error b = enc.encodeKey(b, ctx.parentKey[0])
b, err = enc.encodeKey(b, ctx.parentKey[0])
if err != nil {
return nil, err
}
for _, k := range ctx.parentKey[1:] { for _, k := range ctx.parentKey[1:] {
b = append(b, '.') b = append(b, '.')
b = enc.encodeKey(b, k)
b, err = enc.encodeKey(b, k)
if err != nil {
return nil, err
}
} }
b = append(b, "]\n"...) b = append(b, "]\n"...)
return b, nil return b
} }
//nolint:cyclop func (enc *Encoder) encodeKey(b []byte, k string) []byte {
func (enc *Encoder) encodeKey(b []byte, k string) ([]byte, error) {
needsQuotation := false needsQuotation := false
cannotUseLiteral := false cannotUseLiteral := false
if len(k) == 0 {
return append(b, "''"...)
}
for _, c := range k { for _, c := range k {
if (c >= 'A' && c <= 'Z') || (c >= 'a' && c <= 'z') || (c >= '0' && c <= '9') || c == '-' || c == '_' { if (c >= 'A' && c <= 'Z') || (c >= 'a' && c <= 'z') || (c >= '0' && c <= '9') || c == '-' || c == '_' {
continue continue
} }
if c == '\n' {
return nil, fmt.Errorf("toml: new line characters in keys are not supported")
}
if c == literalQuote { if c == literalQuote {
cannotUseLiteral = true cannotUseLiteral = true
} }
@@ -495,21 +649,52 @@ func (enc *Encoder) encodeKey(b []byte, k string) ([]byte, error) {
needsQuotation = true needsQuotation = true
} }
if needsQuotation && needsQuoting(k) {
cannotUseLiteral = true
}
switch { switch {
case cannotUseLiteral: case cannotUseLiteral:
return enc.encodeQuotedString(false, b, k), nil return enc.encodeQuotedString(false, b, k)
case needsQuotation: case needsQuotation:
return enc.encodeLiteralString(b, k), nil return enc.encodeLiteralString(b, k)
default: default:
return enc.encodeUnquotedKey(b, k), nil return enc.encodeUnquotedKey(b, k)
}
}
func (enc *Encoder) keyToString(k reflect.Value) (string, error) {
keyType := k.Type()
if keyType.Implements(textMarshalerType) {
keyB, err := k.Interface().(encoding.TextMarshaler).MarshalText()
if err != nil {
return "", fmt.Errorf("toml: error marshalling key %v from text: %w", k, err)
}
return string(keyB), nil
}
switch keyType.Kind() {
case reflect.String:
return k.String(), nil
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
return strconv.FormatInt(k.Int(), 10), nil
case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:
return strconv.FormatUint(k.Uint(), 10), nil
case reflect.Float32:
return strconv.FormatFloat(k.Float(), 'f', -1, 32), nil
case reflect.Float64:
return strconv.FormatFloat(k.Float(), 'f', -1, 64), nil
default:
return "", fmt.Errorf("toml: type %s is not supported as a map key", keyType.Kind())
} }
} }
func (enc *Encoder) encodeMap(b []byte, ctx encoderCtx, v reflect.Value) ([]byte, error) { func (enc *Encoder) encodeMap(b []byte, ctx encoderCtx, v reflect.Value) ([]byte, error) {
if v.Type().Key().Kind() != reflect.String {
return nil, fmt.Errorf("toml: type %s is not supported as a map key", v.Type().Key().Kind())
}
var ( var (
t table t table
emptyValueOptions valueOptions emptyValueOptions valueOptions
@@ -517,11 +702,22 @@ func (enc *Encoder) encodeMap(b []byte, ctx encoderCtx, v reflect.Value) ([]byte
iter := v.MapRange() iter := v.MapRange()
for iter.Next() { for iter.Next() {
k := iter.Key().String()
v := iter.Value() v := iter.Value()
if isNil(v) { if isNil(v) {
continue // For nil pointers, convert to zero value of the element type.
// This allows round-trip marshaling of maps with nil pointer values.
// For nil interfaces and nil maps, skip since we can't derive a type.
if v.Kind() == reflect.Ptr {
v = reflect.Zero(v.Type().Elem())
} else {
continue
}
}
k, err := enc.keyToString(iter.Key())
if err != nil {
return nil, err
} }
if willConvertToTableOrArrayTable(ctx, v) { if willConvertToTableOrArrayTable(ctx, v) {
@@ -538,8 +734,8 @@ func (enc *Encoder) encodeMap(b []byte, ctx encoderCtx, v reflect.Value) ([]byte
} }
func sortEntriesByKey(e []entry) { func sortEntriesByKey(e []entry) {
sort.Slice(e, func(i, j int) bool { slices.SortFunc(e, func(a, b entry) int {
return e[i].Key < e[j].Key return strings.Compare(a.Key, b.Key)
}) })
} }
@@ -555,16 +751,25 @@ type table struct {
} }
func (t *table) pushKV(k string, v reflect.Value, options valueOptions) { func (t *table) pushKV(k string, v reflect.Value, options valueOptions) {
for _, e := range t.kvs {
if e.Key == k {
return
}
}
t.kvs = append(t.kvs, entry{Key: k, Value: v, Options: options}) t.kvs = append(t.kvs, entry{Key: k, Value: v, Options: options})
} }
func (t *table) pushTable(k string, v reflect.Value, options valueOptions) { func (t *table) pushTable(k string, v reflect.Value, options valueOptions) {
for _, e := range t.tables {
if e.Key == k {
return
}
}
t.tables = append(t.tables, entry{Key: k, Value: v, Options: options}) t.tables = append(t.tables, entry{Key: k, Value: v, Options: options})
} }
func (enc *Encoder) encodeStruct(b []byte, ctx encoderCtx, v reflect.Value) ([]byte, error) { func walkStruct(ctx encoderCtx, t *table, v reflect.Value) {
var t table
// TODO: cache this // TODO: cache this
typ := v.Type() typ := v.Type()
for i := 0; i < typ.NumField(); i++ { for i := 0; i < typ.NumField(); i++ {
@@ -575,8 +780,6 @@ func (enc *Encoder) encodeStruct(b []byte, ctx encoderCtx, v reflect.Value) ([]b
continue continue
} }
k := fieldType.Name
tag := fieldType.Tag.Get("toml") tag := fieldType.Tag.Get("toml")
// special field name to skip field // special field name to skip field
@@ -584,13 +787,25 @@ func (enc *Encoder) encodeStruct(b []byte, ctx encoderCtx, v reflect.Value) ([]b
continue continue
} }
name, opts := parseTag(tag) k, opts := parseTag(tag)
if isValidName(name) { if !isValidName(k) {
k = name k = ""
} }
f := v.Field(i) f := v.Field(i)
if k == "" {
if fieldType.Anonymous {
if fieldType.Type.Kind() == reflect.Struct {
walkStruct(ctx, t, f)
} else if fieldType.Type.Kind() == reflect.Ptr && !f.IsNil() && f.Elem().Kind() == reflect.Struct {
walkStruct(ctx, t, f.Elem())
}
continue
}
k = fieldType.Name
}
if isNil(f) { if isNil(f) {
continue continue
} }
@@ -598,6 +813,8 @@ func (enc *Encoder) encodeStruct(b []byte, ctx encoderCtx, v reflect.Value) ([]b
options := valueOptions{ options := valueOptions{
multiline: opts.multiline, multiline: opts.multiline,
omitempty: opts.omitempty, omitempty: opts.omitempty,
omitzero: opts.omitzero,
commented: opts.commented,
comment: fieldType.Tag.Get("comment"), comment: fieldType.Tag.Get("comment"),
} }
@@ -607,15 +824,30 @@ func (enc *Encoder) encodeStruct(b []byte, ctx encoderCtx, v reflect.Value) ([]b
t.pushTable(k, f, options) t.pushTable(k, f, options)
} }
} }
}
func (enc *Encoder) encodeStruct(b []byte, ctx encoderCtx, v reflect.Value) ([]byte, error) {
var t table
walkStruct(ctx, &t, v)
return enc.encodeTable(b, ctx, t) return enc.encodeTable(b, ctx, t)
} }
func (enc *Encoder) encodeComment(indent int, comment string, b []byte) []byte { func (enc *Encoder) encodeComment(indent int, comment string, b []byte) []byte {
if comment != "" { for len(comment) > 0 {
var line string
idx := strings.IndexByte(comment, '\n')
if idx >= 0 {
line = comment[:idx]
comment = comment[idx+1:]
} else {
line = comment
comment = ""
}
b = enc.indent(indent, b) b = enc.indent(indent, b)
b = append(b, "# "...) b = append(b, "# "...)
b = append(b, comment...) b = append(b, line...)
b = append(b, '\n') b = append(b, '\n')
} }
return b return b
@@ -642,6 +874,8 @@ type tagOptions struct {
multiline bool multiline bool
inline bool inline bool
omitempty bool omitempty bool
omitzero bool
commented bool
} }
func parseTag(tag string) (string, tagOptions) { func parseTag(tag string) (string, tagOptions) {
@@ -653,7 +887,7 @@ func parseTag(tag string) (string, tagOptions) {
} }
raw := tag[idx+1:] raw := tag[idx+1:]
tag = string(tag[:idx]) tag = tag[:idx]
for raw != "" { for raw != "" {
var o string var o string
i := strings.Index(raw, ",") i := strings.Index(raw, ",")
@@ -669,6 +903,10 @@ func parseTag(tag string) (string, tagOptions) {
opts.inline = true opts.inline = true
case "omitempty": case "omitempty":
opts.omitempty = true opts.omitempty = true
case "omitzero":
opts.omitzero = true
case "commented":
opts.commented = true
} }
} }
@@ -685,10 +923,7 @@ func (enc *Encoder) encodeTable(b []byte, ctx encoderCtx, t table) ([]byte, erro
} }
if !ctx.skipTableHeader { if !ctx.skipTableHeader {
b, err = enc.encodeTableHeader(ctx, b) b = enc.encodeTableHeader(ctx, b)
if err != nil {
return nil, err
}
if enc.indentTables && len(ctx.parentKey) > 0 { if enc.indentTables && len(ctx.parentKey) > 0 {
ctx.indent++ ctx.indent++
@@ -696,10 +931,21 @@ func (enc *Encoder) encodeTable(b []byte, ctx encoderCtx, t table) ([]byte, erro
} }
ctx.skipTableHeader = false ctx.skipTableHeader = false
hasNonEmptyKV := false
for _, kv := range t.kvs { for _, kv := range t.kvs {
ctx.setKey(kv.Key) if shouldOmitEmpty(kv.Options, kv.Value) {
continue
}
if shouldOmitZero(kv.Options, kv.Value) {
continue
}
hasNonEmptyKV = true
b, err = enc.encodeKv(b, ctx, kv.Options, kv.Value) ctx.setKey(kv.Key)
ctx2 := ctx
ctx2.commented = kv.Options.commented || ctx2.commented
b, err = enc.encodeKv(b, ctx2, kv.Options, kv.Value)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -707,17 +953,33 @@ func (enc *Encoder) encodeTable(b []byte, ctx encoderCtx, t table) ([]byte, erro
b = append(b, '\n') b = append(b, '\n')
} }
first := true
for _, table := range t.tables { for _, table := range t.tables {
if shouldOmitEmpty(table.Options, table.Value) {
continue
}
if shouldOmitZero(table.Options, table.Value) {
continue
}
if first {
first = false
if hasNonEmptyKV {
b = append(b, '\n')
}
} else {
b = append(b, "\n"...)
}
ctx.setKey(table.Key) ctx.setKey(table.Key)
ctx.options = table.Options ctx.options = table.Options
ctx2 := ctx
ctx2.commented = ctx2.commented || ctx.options.commented
b, err = enc.encode(b, ctx, table.Value) b, err = enc.encode(b, ctx2, table.Value)
if err != nil { if err != nil {
return nil, err return nil, err
} }
b = append(b, '\n')
} }
return b, nil return b, nil
@@ -730,6 +992,13 @@ func (enc *Encoder) encodeTableInline(b []byte, ctx encoderCtx, t table) ([]byte
first := true first := true
for _, kv := range t.kvs { for _, kv := range t.kvs {
if shouldOmitEmpty(kv.Options, kv.Value) {
continue
}
if shouldOmitZero(kv.Options, kv.Value) {
continue
}
if first { if first {
first = false first = false
} else { } else {
@@ -745,7 +1014,7 @@ func (enc *Encoder) encodeTableInline(b []byte, ctx encoderCtx, t table) ([]byte
} }
if len(t.tables) > 0 { if len(t.tables) > 0 {
panic("inline table cannot contain nested tables, online key-values") panic("inline table cannot contain nested tables, only key-values")
} }
b = append(b, "}"...) b = append(b, "}"...)
@@ -757,11 +1026,14 @@ func willConvertToTable(ctx encoderCtx, v reflect.Value) bool {
if !v.IsValid() { if !v.IsValid() {
return false return false
} }
if v.Type() == timeType || v.Type().Implements(textMarshalerType) || (v.Kind() != reflect.Ptr && v.CanAddr() && reflect.PtrTo(v.Type()).Implements(textMarshalerType)) { t := v.Type()
if t == timeType || t.Implements(textMarshalerType) {
return false
}
if v.Kind() != reflect.Ptr && v.CanAddr() && reflect.PointerTo(t).Implements(textMarshalerType) {
return false return false
} }
t := v.Type()
switch t.Kind() { switch t.Kind() {
case reflect.Map, reflect.Struct: case reflect.Map, reflect.Struct:
return !ctx.inline return !ctx.inline
@@ -779,13 +1051,16 @@ func willConvertToTable(ctx encoderCtx, v reflect.Value) bool {
} }
func willConvertToTableOrArrayTable(ctx encoderCtx, v reflect.Value) bool { func willConvertToTableOrArrayTable(ctx encoderCtx, v reflect.Value) bool {
if ctx.insideKv {
return false
}
t := v.Type() t := v.Type()
if t.Kind() == reflect.Interface { if t.Kind() == reflect.Interface {
return willConvertToTableOrArrayTable(ctx, v.Elem()) return willConvertToTableOrArrayTable(ctx, v.Elem())
} }
if t.Kind() == reflect.Slice { if t.Kind() == reflect.Slice || t.Kind() == reflect.Array {
if v.Len() == 0 { if v.Len() == 0 {
// An empty slice should be a kv = []. // An empty slice should be a kv = [].
return false return false
@@ -824,8 +1099,14 @@ func (enc *Encoder) encodeSlice(b []byte, ctx encoderCtx, v reflect.Value) ([]by
func (enc *Encoder) encodeSliceAsArrayTable(b []byte, ctx encoderCtx, v reflect.Value) ([]byte, error) { func (enc *Encoder) encodeSliceAsArrayTable(b []byte, ctx encoderCtx, v reflect.Value) ([]byte, error) {
ctx.shiftKey() ctx.shiftKey()
var err error
scratch := make([]byte, 0, 64) scratch := make([]byte, 0, 64)
scratch = enc.commented(ctx.commented, scratch)
if enc.indentTables {
scratch = enc.indent(ctx.indent, scratch)
}
scratch = append(scratch, "[["...) scratch = append(scratch, "[["...)
for i, k := range ctx.parentKey { for i, k := range ctx.parentKey {
@@ -833,18 +1114,26 @@ func (enc *Encoder) encodeSliceAsArrayTable(b []byte, ctx encoderCtx, v reflect.
scratch = append(scratch, '.') scratch = append(scratch, '.')
} }
scratch, err = enc.encodeKey(scratch, k) scratch = enc.encodeKey(scratch, k)
if err != nil {
return nil, err
}
} }
scratch = append(scratch, "]]\n"...) scratch = append(scratch, "]]\n"...)
ctx.skipTableHeader = true ctx.skipTableHeader = true
b = enc.encodeComment(ctx.indent, ctx.options.comment, b)
if enc.indentTables {
ctx.indent++
}
for i := 0; i < v.Len(); i++ { for i := 0; i < v.Len(); i++ {
if i != 0 {
b = append(b, "\n"...)
}
b = append(b, scratch...) b = append(b, scratch...)
var err error
b, err = enc.encode(b, ctx, v.Index(i)) b, err = enc.encode(b, ctx, v.Index(i))
if err != nil { if err != nil {
return nil, err return nil, err
+1396 -128
View File
File diff suppressed because it is too large Load Diff
+44
View File
@@ -0,0 +1,44 @@
// Package ossfuzz provides a fuzzing target for OSS-Fuzz.
package ossfuzz
import (
"fmt"
"reflect"
"strings"
"github.com/pelletier/go-toml/v2"
)
// FuzzToml is the fuzzing target.
func FuzzToml(data []byte) int {
if len(data) >= 2048 {
return 0
}
if strings.Contains(string(data), "nan") {
return 0
}
var v interface{}
err := toml.Unmarshal(data, &v)
if err != nil {
return 0
}
encoded, err := toml.Marshal(v)
if err != nil {
panic(fmt.Sprintf("failed to marshal unmarshaled document: %s", err))
}
var v2 interface{}
err = toml.Unmarshal(encoded, &v2)
if err != nil {
panic(fmt.Sprintf("failed round trip: %s", err))
}
if !reflect.DeepEqual(v, v2) {
panic(fmt.Sprintf("not equal: %#+v %#+v", v, v2))
}
return 1
}
-450
View File
@@ -1,450 +0,0 @@
package toml
import (
"strconv"
"strings"
"testing"
"github.com/pelletier/go-toml/v2/internal/ast"
"github.com/stretchr/testify/require"
)
//nolint:funlen
func TestParser_AST_Numbers(t *testing.T) {
examples := []struct {
desc string
input string
kind ast.Kind
err bool
}{
{
desc: "integer just digits",
input: `1234`,
kind: ast.Integer,
},
{
desc: "integer zero",
input: `0`,
kind: ast.Integer,
},
{
desc: "integer sign",
input: `+99`,
kind: ast.Integer,
},
{
desc: "integer hex uppercase",
input: `0xDEADBEEF`,
kind: ast.Integer,
},
{
desc: "integer hex lowercase",
input: `0xdead_beef`,
kind: ast.Integer,
},
{
desc: "integer octal",
input: `0o01234567`,
kind: ast.Integer,
},
{
desc: "integer binary",
input: `0b11010110`,
kind: ast.Integer,
},
{
desc: "float zero",
input: `0.0`,
kind: ast.Float,
},
{
desc: "float positive zero",
input: `+0.0`,
kind: ast.Float,
},
{
desc: "float negative zero",
input: `-0.0`,
kind: ast.Float,
},
{
desc: "float pi",
input: `3.1415`,
kind: ast.Float,
},
{
desc: "float negative",
input: `-0.01`,
kind: ast.Float,
},
{
desc: "float signed exponent",
input: `5e+22`,
kind: ast.Float,
},
{
desc: "float exponent lowercase",
input: `1e06`,
kind: ast.Float,
},
{
desc: "float exponent uppercase",
input: `-2E-2`,
kind: ast.Float,
},
{
desc: "float fractional with exponent",
input: `6.626e-34`,
kind: ast.Float,
},
{
desc: "float underscores",
input: `224_617.445_991_228`,
kind: ast.Float,
},
{
desc: "inf",
input: `inf`,
kind: ast.Float,
},
{
desc: "inf negative",
input: `-inf`,
kind: ast.Float,
},
{
desc: "inf positive",
input: `+inf`,
kind: ast.Float,
},
{
desc: "nan",
input: `nan`,
kind: ast.Float,
},
{
desc: "nan negative",
input: `-nan`,
kind: ast.Float,
},
{
desc: "nan positive",
input: `+nan`,
kind: ast.Float,
},
}
for _, e := range examples {
e := e
t.Run(e.desc, func(t *testing.T) {
p := parser{}
p.Reset([]byte(`A = ` + e.input))
p.NextExpression()
err := p.Error()
if e.err {
require.Error(t, err)
} else {
require.NoError(t, err)
expected := astNode{
Kind: ast.KeyValue,
Children: []astNode{
{Kind: e.kind, Data: []byte(e.input)},
{Kind: ast.Key, Data: []byte(`A`)},
},
}
compareNode(t, expected, p.Expression())
}
})
}
}
type (
astNode struct {
Kind ast.Kind
Data []byte
Children []astNode
}
)
func compareNode(t *testing.T, e astNode, n *ast.Node) {
t.Helper()
require.Equal(t, e.Kind, n.Kind)
require.Equal(t, e.Data, n.Data)
compareIterator(t, e.Children, n.Children())
}
func compareIterator(t *testing.T, expected []astNode, actual ast.Iterator) {
t.Helper()
idx := 0
for actual.Next() {
n := actual.Node()
if idx >= len(expected) {
t.Fatal("extra child in actual tree")
}
e := expected[idx]
compareNode(t, e, n)
idx++
}
if idx < len(expected) {
t.Fatal("missing children in actual", "idx =", idx, "expected =", len(expected))
}
}
//nolint:funlen
func TestParser_AST(t *testing.T) {
examples := []struct {
desc string
input string
ast astNode
err bool
}{
{
desc: "simple string assignment",
input: `A = "hello"`,
ast: astNode{
Kind: ast.KeyValue,
Children: []astNode{
{
Kind: ast.String,
Data: []byte(`hello`),
},
{
Kind: ast.Key,
Data: []byte(`A`),
},
},
},
},
{
desc: "simple bool assignment",
input: `A = true`,
ast: astNode{
Kind: ast.KeyValue,
Children: []astNode{
{
Kind: ast.Bool,
Data: []byte(`true`),
},
{
Kind: ast.Key,
Data: []byte(`A`),
},
},
},
},
{
desc: "array of strings",
input: `A = ["hello", ["world", "again"]]`,
ast: astNode{
Kind: ast.KeyValue,
Children: []astNode{
{
Kind: ast.Array,
Children: []astNode{
{
Kind: ast.String,
Data: []byte(`hello`),
},
{
Kind: ast.Array,
Children: []astNode{
{
Kind: ast.String,
Data: []byte(`world`),
},
{
Kind: ast.String,
Data: []byte(`again`),
},
},
},
},
},
{
Kind: ast.Key,
Data: []byte(`A`),
},
},
},
},
{
desc: "array of arrays of strings",
input: `A = ["hello", "world"]`,
ast: astNode{
Kind: ast.KeyValue,
Children: []astNode{
{
Kind: ast.Array,
Children: []astNode{
{
Kind: ast.String,
Data: []byte(`hello`),
},
{
Kind: ast.String,
Data: []byte(`world`),
},
},
},
{
Kind: ast.Key,
Data: []byte(`A`),
},
},
},
},
{
desc: "inline table",
input: `name = { first = "Tom", last = "Preston-Werner" }`,
ast: astNode{
Kind: ast.KeyValue,
Children: []astNode{
{
Kind: ast.InlineTable,
Children: []astNode{
{
Kind: ast.KeyValue,
Children: []astNode{
{Kind: ast.String, Data: []byte(`Tom`)},
{Kind: ast.Key, Data: []byte(`first`)},
},
},
{
Kind: ast.KeyValue,
Children: []astNode{
{Kind: ast.String, Data: []byte(`Preston-Werner`)},
{Kind: ast.Key, Data: []byte(`last`)},
},
},
},
},
{
Kind: ast.Key,
Data: []byte(`name`),
},
},
},
},
}
for _, e := range examples {
e := e
t.Run(e.desc, func(t *testing.T) {
p := parser{}
p.Reset([]byte(e.input))
p.NextExpression()
err := p.Error()
if e.err {
require.Error(t, err)
} else {
require.NoError(t, err)
compareNode(t, e.ast, p.Expression())
}
})
}
}
func BenchmarkParseBasicStringWithUnicode(b *testing.B) {
p := &parser{}
b.Run("4", func(b *testing.B) {
input := []byte(`"\u1234\u5678\u9ABC\u1234\u5678\u9ABC"`)
b.ReportAllocs()
b.SetBytes(int64(len(input)))
for i := 0; i < b.N; i++ {
p.parseBasicString(input)
}
})
b.Run("8", func(b *testing.B) {
input := []byte(`"\u12345678\u9ABCDEF0\u12345678\u9ABCDEF0"`)
b.ReportAllocs()
b.SetBytes(int64(len(input)))
for i := 0; i < b.N; i++ {
p.parseBasicString(input)
}
})
}
func BenchmarkParseBasicStringsEasy(b *testing.B) {
p := &parser{}
for _, size := range []int{1, 4, 8, 16, 21} {
b.Run(strconv.Itoa(size), func(b *testing.B) {
input := []byte(`"` + strings.Repeat("A", size) + `"`)
b.ReportAllocs()
b.SetBytes(int64(len(input)))
for i := 0; i < b.N; i++ {
p.parseBasicString(input)
}
})
}
}
func TestParser_AST_DateTimes(t *testing.T) {
examples := []struct {
desc string
input string
kind ast.Kind
err bool
}{
{
desc: "offset-date-time with delim 'T' and UTC offset",
input: `2021-07-21T12:08:05Z`,
kind: ast.DateTime,
},
{
desc: "offset-date-time with space delim and +8hours offset",
input: `2021-07-21 12:08:05+08:00`,
kind: ast.DateTime,
},
{
desc: "local-date-time with nano second",
input: `2021-07-21T12:08:05.666666666`,
kind: ast.LocalDateTime,
},
{
desc: "local-date-time",
input: `2021-07-21T12:08:05`,
kind: ast.LocalDateTime,
},
{
desc: "local-date",
input: `2021-07-21`,
kind: ast.LocalDate,
},
}
for _, e := range examples {
e := e
t.Run(e.desc, func(t *testing.T) {
p := parser{}
p.Reset([]byte(`A = ` + e.input))
p.NextExpression()
err := p.Error()
if e.err {
require.Error(t, err)
} else {
require.NoError(t, err)
expected := astNode{
Kind: ast.KeyValue,
Children: []astNode{
{Kind: e.kind, Data: []byte(e.input)},
{Kind: ast.Key, Data: []byte(`A`)},
},
}
compareNode(t, expected, p.Expression())
}
})
}
}
+29 -22
View File
@@ -1,9 +1,8 @@
package toml package toml
import ( import (
"github.com/pelletier/go-toml/v2/internal/ast"
"github.com/pelletier/go-toml/v2/internal/danger"
"github.com/pelletier/go-toml/v2/internal/tracker" "github.com/pelletier/go-toml/v2/internal/tracker"
"github.com/pelletier/go-toml/v2/unstable"
) )
type strict struct { type strict struct {
@@ -12,10 +11,13 @@ type strict struct {
// Tracks the current key being processed. // Tracks the current key being processed.
key tracker.KeyTracker key tracker.KeyTracker
missing []decodeError missing []unstable.ParserError
// Reference to the document for computing key ranges.
doc []byte
} }
func (s *strict) EnterTable(node *ast.Node) { func (s *strict) EnterTable(node *unstable.Node) {
if !s.Enabled { if !s.Enabled {
return return
} }
@@ -23,7 +25,7 @@ func (s *strict) EnterTable(node *ast.Node) {
s.key.UpdateTable(node) s.key.UpdateTable(node)
} }
func (s *strict) EnterArrayTable(node *ast.Node) { func (s *strict) EnterArrayTable(node *unstable.Node) {
if !s.Enabled { if !s.Enabled {
return return
} }
@@ -31,7 +33,7 @@ func (s *strict) EnterArrayTable(node *ast.Node) {
s.key.UpdateArrayTable(node) s.key.UpdateArrayTable(node)
} }
func (s *strict) EnterKeyValue(node *ast.Node) { func (s *strict) EnterKeyValue(node *unstable.Node) {
if !s.Enabled { if !s.Enabled {
return return
} }
@@ -39,7 +41,7 @@ func (s *strict) EnterKeyValue(node *ast.Node) {
s.key.Push(node) s.key.Push(node)
} }
func (s *strict) ExitKeyValue(node *ast.Node) { func (s *strict) ExitKeyValue(node *unstable.Node) {
if !s.Enabled { if !s.Enabled {
return return
} }
@@ -47,27 +49,27 @@ func (s *strict) ExitKeyValue(node *ast.Node) {
s.key.Pop(node) s.key.Pop(node)
} }
func (s *strict) MissingTable(node *ast.Node) { func (s *strict) MissingTable(node *unstable.Node) {
if !s.Enabled { if !s.Enabled {
return return
} }
s.missing = append(s.missing, decodeError{ s.missing = append(s.missing, unstable.ParserError{
highlight: keyLocation(node), Highlight: s.keyLocation(node),
message: "missing table", Message: "missing table",
key: s.key.Key(), Key: s.key.Key(),
}) })
} }
func (s *strict) MissingField(node *ast.Node) { func (s *strict) MissingField(node *unstable.Node) {
if !s.Enabled { if !s.Enabled {
return return
} }
s.missing = append(s.missing, decodeError{ s.missing = append(s.missing, unstable.ParserError{
highlight: keyLocation(node), Highlight: s.keyLocation(node),
message: "missing field", Message: "missing field",
key: s.key.Key(), Key: s.key.Key(),
}) })
} }
@@ -88,7 +90,7 @@ func (s *strict) Error(doc []byte) error {
return err return err
} }
func keyLocation(node *ast.Node) []byte { func (s *strict) keyLocation(node *unstable.Node) []byte {
k := node.Key() k := node.Key()
hasOne := k.Next() hasOne := k.Next()
@@ -96,12 +98,17 @@ func keyLocation(node *ast.Node) []byte {
panic("should not be called with empty key") panic("should not be called with empty key")
} }
start := k.Node().Data // Get the range from the first key to the last key.
end := k.Node().Data firstRaw := k.Node().Raw
lastRaw := firstRaw
for k.Next() { for k.Next() {
end = k.Node().Data lastRaw = k.Node().Raw
} }
return danger.BytesRange(start, end) // Compute the slice from the document using the ranges.
start := firstRaw.Offset
end := lastRaw.Offset + lastRaw.Length
return s.doc[start:end]
} }
+596
View File
@@ -0,0 +1,596 @@
#!/usr/bin/env bash
set -uo pipefail
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color
# Go versions to test (1.11 through 1.25)
GO_VERSIONS=(
"1.11"
"1.12"
"1.13"
"1.14"
"1.15"
"1.16"
"1.17"
"1.18"
"1.19"
"1.20"
"1.21"
"1.22"
"1.23"
"1.24"
"1.25"
)
# Default values
PARALLEL=true
VERBOSE=false
OUTPUT_DIR="test-results"
DOCKER_TIMEOUT="10m"
usage() {
cat << EOF
Usage: $0 [OPTIONS] [GO_VERSIONS...]
Test go-toml across multiple Go versions using Docker containers.
The script reports the lowest continuous supported Go version (where all subsequent
versions pass) and only exits with non-zero status if either of the two most recent
Go versions fail, indicating immediate attention is needed.
Note: For Go versions < 1.21, the script automatically updates go.mod to match the
target version, but older versions may still fail due to missing standard library
features (e.g., the 'slices' package introduced in Go 1.21).
OPTIONS:
-h, --help Show this help message
-s, --sequential Run tests sequentially instead of in parallel
-v, --verbose Enable verbose output
-o, --output DIR Output directory for test results (default: test-results)
-t, --timeout TIME Docker timeout for each test (default: 10m)
--list List available Go versions and exit
ARGUMENTS:
GO_VERSIONS Specific Go versions to test (default: all supported versions)
Examples: 1.21 1.22 1.23
EXAMPLES:
$0 # Test all Go versions in parallel
$0 --sequential # Test all Go versions sequentially
$0 1.21 1.22 1.23 # Test specific versions
$0 --verbose --output ./results 1.24 1.25 # Verbose output to custom directory
EXIT CODES:
0 Recent Go versions pass (good compatibility)
1 Recent Go versions fail (needs attention) or script error
EOF
}
log() {
echo -e "${BLUE}[$(date +'%H:%M:%S')]${NC} $*" >&2
}
log_success() {
echo -e "${GREEN}[$(date +'%H:%M:%S')] ✓${NC} $*" >&2
}
log_error() {
echo -e "${RED}[$(date +'%H:%M:%S')] ✗${NC} $*" >&2
}
log_warning() {
echo -e "${YELLOW}[$(date +'%H:%M:%S')] ⚠${NC} $*" >&2
}
# Parse command line arguments
while [[ $# -gt 0 ]]; do
case $1 in
-h|--help)
usage
exit 0
;;
-s|--sequential)
PARALLEL=false
shift
;;
-v|--verbose)
VERBOSE=true
shift
;;
-o|--output)
OUTPUT_DIR="$2"
shift 2
;;
-t|--timeout)
DOCKER_TIMEOUT="$2"
shift 2
;;
--list)
echo "Available Go versions:"
printf '%s\n' "${GO_VERSIONS[@]}"
exit 0
;;
-*)
echo "Unknown option: $1" >&2
usage
exit 1
;;
*)
# Remaining arguments are Go versions
break
;;
esac
done
# If specific versions provided, use those instead of defaults
if [[ $# -gt 0 ]]; then
GO_VERSIONS=("$@")
fi
# Validate Go versions
for version in "${GO_VERSIONS[@]}"; do
if ! [[ "$version" =~ ^1\.(1[1-9]|2[0-5])$ ]]; then
log_error "Invalid Go version: $version. Supported versions: 1.11-1.25"
exit 1
fi
done
# Check if Docker is available
if ! command -v docker &> /dev/null; then
log_error "Docker is required but not installed or not in PATH"
exit 1
fi
# Check if Docker daemon is running
if ! docker info &> /dev/null; then
log_error "Docker daemon is not running"
exit 1
fi
# Create output directory
mkdir -p "$OUTPUT_DIR"
# Function to test a single Go version
test_go_version() {
local go_version="$1"
local container_name="go-toml-test-${go_version}"
local result_file="${OUTPUT_DIR}/go-${go_version}.txt"
local dockerfile_content
log "Testing Go $go_version..."
# Create a temporary Dockerfile for this version
# For Go versions < 1.21, we need to update go.mod to match the Go version
local needs_go_mod_update=false
if [[ $(echo "$go_version 1.21" | tr ' ' '\n' | sort -V | head -n1) == "$go_version" && "$go_version" != "1.21" ]]; then
needs_go_mod_update=true
fi
dockerfile_content="FROM golang:${go_version}-alpine
# Install git (required for go mod)
RUN apk add --no-cache git
# Set working directory
WORKDIR /app
# Copy source code
COPY . ."
# Add go.mod update step for older Go versions
if [[ "$needs_go_mod_update" == true ]]; then
dockerfile_content="$dockerfile_content
# Update go.mod to match Go version (required for Go < 1.21)
RUN if [ -f go.mod ]; then sed -i 's/^go [0-9]\\+\\.[0-9]\\+\\(\\.[0-9]\\+\\)\\?/go $go_version/' go.mod; fi
# Note: Go versions < 1.21 may fail due to missing standard library packages (e.g., slices)
# This is expected for projects that use Go 1.21+ features"
fi
dockerfile_content="$dockerfile_content
# Run tests
CMD [\"sh\", \"-c\", \"go version && echo '--- Running go test ./... ---' && go test ./...\"]"
# Create temporary directory for this test
local temp_dir
temp_dir=$(mktemp -d)
# Copy source to temp directory (excluding test results and git)
rsync -a --exclude="$OUTPUT_DIR" --exclude=".git" --exclude="*.test" . "$temp_dir/"
# Create Dockerfile in temp directory
echo "$dockerfile_content" > "$temp_dir/Dockerfile"
# Build and run container
local exit_code=0
local output
if $VERBOSE; then
log "Building Docker image for Go $go_version..."
fi
# Capture both stdout and stderr, and the exit code
if output=$(cd "$temp_dir" && timeout "$DOCKER_TIMEOUT" docker build -t "$container_name" . 2>&1 && \
timeout "$DOCKER_TIMEOUT" docker run --rm "$container_name" 2>&1); then
log_success "Go $go_version: PASSED"
echo "PASSED" > "${result_file}.status"
else
exit_code=$?
log_error "Go $go_version: FAILED (exit code: $exit_code)"
echo "FAILED" > "${result_file}.status"
fi
# Save full output
echo "$output" > "$result_file"
# Clean up
docker rmi "$container_name" &> /dev/null || true
rm -rf "$temp_dir"
if $VERBOSE; then
echo "--- Go $go_version output ---"
echo "$output"
echo "--- End Go $go_version output ---"
fi
return $exit_code
}
# Function to run tests in parallel
run_parallel() {
local pids=()
local failed_versions=()
log "Starting parallel tests for ${#GO_VERSIONS[@]} Go versions..."
# Start all tests in background
for version in "${GO_VERSIONS[@]}"; do
test_go_version "$version" &
pids+=($!)
done
# Wait for all tests to complete
for i in "${!pids[@]}"; do
local pid=${pids[$i]}
local version=${GO_VERSIONS[$i]}
if ! wait $pid; then
failed_versions+=("$version")
fi
done
return ${#failed_versions[@]}
}
# Function to run tests sequentially
run_sequential() {
local failed_versions=()
log "Starting sequential tests for ${#GO_VERSIONS[@]} Go versions..."
for version in "${GO_VERSIONS[@]}"; do
if ! test_go_version "$version"; then
failed_versions+=("$version")
fi
done
return ${#failed_versions[@]}
}
# Main execution
main() {
local start_time
start_time=$(date +%s)
log "Starting Go version compatibility tests..."
log "Testing versions: ${GO_VERSIONS[*]}"
log "Output directory: $OUTPUT_DIR"
log "Parallel execution: $PARALLEL"
local failed_count
if $PARALLEL; then
run_parallel
failed_count=$?
else
run_sequential
failed_count=$?
fi
local end_time
end_time=$(date +%s)
local duration=$((end_time - start_time))
# Collect results for display
local passed_versions=()
local failed_versions=()
local unknown_versions=()
local passed_count=0
for version in "${GO_VERSIONS[@]}"; do
local status_file="${OUTPUT_DIR}/go-${version}.txt.status"
if [[ -f "$status_file" ]]; then
local status
status=$(cat "$status_file")
if [[ "$status" == "PASSED" ]]; then
passed_versions+=("$version")
((passed_count++))
else
failed_versions+=("$version")
fi
else
unknown_versions+=("$version")
fi
done
# Generate summary report
local summary_file="${OUTPUT_DIR}/summary.txt"
{
echo "Go Version Compatibility Test Summary"
echo "====================================="
echo "Date: $(date)"
echo "Duration: ${duration}s"
echo "Parallel: $PARALLEL"
echo ""
echo "Results:"
for version in "${GO_VERSIONS[@]}"; do
local status_file="${OUTPUT_DIR}/go-${version}.txt.status"
if [[ -f "$status_file" ]]; then
local status
status=$(cat "$status_file")
if [[ "$status" == "PASSED" ]]; then
echo " Go $version: ✓ PASSED"
else
echo " Go $version: ✗ FAILED"
fi
else
echo " Go $version: ? UNKNOWN (no status file)"
fi
done
echo ""
echo "Summary: $passed_count/${#GO_VERSIONS[@]} versions passed"
if [[ $failed_count -gt 0 ]]; then
echo ""
echo "Failed versions details:"
for version in "${failed_versions[@]}"; do
echo ""
echo "--- Go $version (FAILED) ---"
local result_file="${OUTPUT_DIR}/go-${version}.txt"
if [[ -f "$result_file" ]]; then
tail -n 30 "$result_file"
fi
done
fi
} > "$summary_file"
# Find lowest continuous supported version and check recent versions
local lowest_continuous_version=""
local recent_versions_failed=false
# Sort versions to ensure proper order
local sorted_versions=()
for version in "${GO_VERSIONS[@]}"; do
sorted_versions+=("$version")
done
# Sort versions numerically (1.11, 1.12, ..., 1.25)
IFS=$'\n' sorted_versions=($(sort -V <<< "${sorted_versions[*]}"))
# Find lowest continuous supported version (all versions from this point onwards pass)
for version in "${sorted_versions[@]}"; do
local status_file="${OUTPUT_DIR}/go-${version}.txt.status"
local all_subsequent_pass=true
# Check if this version and all subsequent versions pass
local found_current=false
for check_version in "${sorted_versions[@]}"; do
if [[ "$check_version" == "$version" ]]; then
found_current=true
fi
if [[ "$found_current" == true ]]; then
local check_status_file="${OUTPUT_DIR}/go-${check_version}.txt.status"
if [[ -f "$check_status_file" ]]; then
local status
status=$(cat "$check_status_file")
if [[ "$status" != "PASSED" ]]; then
all_subsequent_pass=false
break
fi
else
all_subsequent_pass=false
break
fi
fi
done
if [[ "$all_subsequent_pass" == true ]]; then
lowest_continuous_version="$version"
break
fi
done
# Check if the two most recent versions failed
local num_versions=${#sorted_versions[@]}
if [[ $num_versions -ge 2 ]]; then
local second_recent="${sorted_versions[$((num_versions-2))]}"
local most_recent="${sorted_versions[$((num_versions-1))]}"
local second_recent_status_file="${OUTPUT_DIR}/go-${second_recent}.txt.status"
local most_recent_status_file="${OUTPUT_DIR}/go-${most_recent}.txt.status"
local second_recent_failed=false
local most_recent_failed=false
if [[ -f "$second_recent_status_file" ]]; then
local status
status=$(cat "$second_recent_status_file")
if [[ "$status" != "PASSED" ]]; then
second_recent_failed=true
fi
else
second_recent_failed=true
fi
if [[ -f "$most_recent_status_file" ]]; then
local status
status=$(cat "$most_recent_status_file")
if [[ "$status" != "PASSED" ]]; then
most_recent_failed=true
fi
else
most_recent_failed=true
fi
if [[ "$second_recent_failed" == true || "$most_recent_failed" == true ]]; then
recent_versions_failed=true
fi
elif [[ $num_versions -eq 1 ]]; then
# Only one version tested, check if it's the most recent and failed
local only_version="${sorted_versions[0]}"
local only_status_file="${OUTPUT_DIR}/go-${only_version}.txt.status"
if [[ -f "$only_status_file" ]]; then
local status
status=$(cat "$only_status_file")
if [[ "$status" != "PASSED" ]]; then
recent_versions_failed=true
fi
else
recent_versions_failed=true
fi
fi
# Display summary
echo ""
log "Test completed in ${duration}s"
log "Summary report: $summary_file"
echo ""
echo "========================================"
echo " FINAL RESULTS"
echo "========================================"
echo ""
# Display passed versions
if [[ ${#passed_versions[@]} -gt 0 ]]; then
log_success "PASSED (${#passed_versions[@]}/${#GO_VERSIONS[@]}):"
# Sort passed versions for display
local sorted_passed=()
for version in "${sorted_versions[@]}"; do
for passed_version in "${passed_versions[@]}"; do
if [[ "$version" == "$passed_version" ]]; then
sorted_passed+=("$version")
break
fi
done
done
for version in "${sorted_passed[@]}"; do
echo -e " ${GREEN}${NC} Go $version"
done
echo ""
fi
# Display failed versions
if [[ ${#failed_versions[@]} -gt 0 ]]; then
log_error "FAILED (${#failed_versions[@]}/${#GO_VERSIONS[@]}):"
# Sort failed versions for display
local sorted_failed=()
for version in "${sorted_versions[@]}"; do
for failed_version in "${failed_versions[@]}"; do
if [[ "$version" == "$failed_version" ]]; then
sorted_failed+=("$version")
break
fi
done
done
for version in "${sorted_failed[@]}"; do
echo -e " ${RED}${NC} Go $version"
done
echo ""
# Show failure details
echo "========================================"
echo " FAILURE DETAILS"
echo "========================================"
echo ""
for version in "${sorted_failed[@]}"; do
echo -e "${RED}--- Go $version FAILURE LOGS (last 30 lines) ---${NC}"
local result_file="${OUTPUT_DIR}/go-${version}.txt"
if [[ -f "$result_file" ]]; then
tail -n 30 "$result_file" | sed 's/^/ /'
else
echo " No log file found: $result_file"
fi
echo ""
done
fi
# Display unknown versions
if [[ ${#unknown_versions[@]} -gt 0 ]]; then
log_warning "UNKNOWN (${#unknown_versions[@]}/${#GO_VERSIONS[@]}):"
for version in "${unknown_versions[@]}"; do
echo -e " ${YELLOW}?${NC} Go $version (no status file)"
done
echo ""
fi
echo "========================================"
echo " COMPATIBILITY SUMMARY"
echo "========================================"
echo ""
if [[ -n "$lowest_continuous_version" ]]; then
log_success "Lowest continuous supported version: Go $lowest_continuous_version"
echo " (All versions from Go $lowest_continuous_version onwards pass)"
else
log_error "No continuous version support found"
echo " (No version has all subsequent versions passing)"
fi
echo ""
echo "========================================"
echo "Full detailed logs available in: $OUTPUT_DIR"
echo "========================================"
# Determine exit code based on recent versions
if [[ "$recent_versions_failed" == true ]]; then
log_error "OVERALL RESULT: Recent Go versions failed - this needs attention!"
if [[ -n "$lowest_continuous_version" ]]; then
echo "Note: Continuous support starts from Go $lowest_continuous_version"
fi
exit 1
else
log_success "OVERALL RESULT: Recent Go versions pass - compatibility looks good!"
if [[ -n "$lowest_continuous_version" ]]; then
echo "Continuous support starts from Go $lowest_continuous_version"
fi
exit 0
fi
}
# Trap to clean up on exit
cleanup() {
# Kill any remaining background processes
jobs -p | xargs -r kill 2>/dev/null || true
# Clean up any remaining Docker containers
docker ps -q --filter "name=go-toml-test-" | xargs -r docker stop 2>/dev/null || true
docker images -q --filter "reference=go-toml-test-*" | xargs -r docker rmi 2>/dev/null || true
}
trap cleanup EXIT
# Run main function
main
@@ -0,0 +1,2 @@
go test fuzz v1
[]byte("0=0000-01-01 00:00:00")
@@ -0,0 +1,2 @@
go test fuzz v1
[]byte("\"\\n\"=\"\"")
@@ -0,0 +1,2 @@
go test fuzz v1
[]byte("''=0")
@@ -0,0 +1,2 @@
go test fuzz v1
[]byte("0=0000-01-01")
@@ -0,0 +1,2 @@
go test fuzz v1
[]byte("0=\"\"\"\\U00000000\"\"\"")
@@ -0,0 +1,2 @@
go test fuzz v1
[]byte("0=[[{}]]")
@@ -0,0 +1,2 @@
go test fuzz v1
[]byte("\"\\b\"=\"\"")
@@ -0,0 +1,2 @@
go test fuzz v1
[]byte("0=inf")
@@ -0,0 +1,2 @@
go test fuzz v1
[]byte("0=0000-01-01 00:00:00+00:00")
@@ -0,0 +1,2 @@
go test fuzz v1
[]byte("0=[{}]")
@@ -0,0 +1,2 @@
go test fuzz v1
[]byte("0=nan")
-69
View File
@@ -1,69 +0,0 @@
package testsuite
import (
"bytes"
"encoding/json"
"fmt"
"github.com/pelletier/go-toml/v2"
)
type parser struct{}
func (p parser) Decode(input string) (output string, outputIsError bool, retErr error) {
defer func() {
if r := recover(); r != nil {
switch rr := r.(type) {
case error:
retErr = rr
default:
retErr = fmt.Errorf("%s", rr)
}
}
}()
var v interface{}
if err := toml.Unmarshal([]byte(input), &v); err != nil {
return err.Error(), true, nil
}
j, err := json.MarshalIndent(addTag("", v), "", " ")
if err != nil {
return "", false, retErr
}
return string(j), false, retErr
}
func (p parser) Encode(input string) (output string, outputIsError bool, retErr error) {
defer func() {
if r := recover(); r != nil {
switch rr := r.(type) {
case error:
retErr = rr
default:
retErr = fmt.Errorf("%s", rr)
}
}
}()
var tmp interface{}
err := json.Unmarshal([]byte(input), &tmp)
if err != nil {
return "", false, err
}
rm, err := rmTag(tmp)
if err != nil {
return err.Error(), true, retErr
}
buf := new(bytes.Buffer)
err = toml.NewEncoder(buf).Encode(rm)
if err != nil {
return err.Error(), true, retErr
}
return buf.String(), false, retErr
}
+11 -9
View File
@@ -1,15 +1,16 @@
//go:generate go run ./cmd/tomltestgen/main.go -o toml_testgen_test.go //go:generate go run github.com/toml-lang/toml-test/v2/cmd/toml-test@v2.1.0 copy -toml 1.1 ./tests
//go:generate go run ./cmd/tomltestgen/main.go -r v2.1.0 -o toml_testgen_test.go
// This is a support file for toml_testgen_test.go
package toml_test package toml_test
import ( import (
"encoding/json" "encoding/json"
"errors"
"testing" "testing"
"github.com/pelletier/go-toml/v2" "github.com/pelletier/go-toml/v2"
"github.com/pelletier/go-toml/v2/testsuite" "github.com/pelletier/go-toml/v2/internal/assert"
"github.com/stretchr/testify/require" "github.com/pelletier/go-toml/v2/internal/testsuite"
) )
func testgenInvalid(t *testing.T, input string) { func testgenInvalid(t *testing.T, input string) {
@@ -38,21 +39,22 @@ func testgenValid(t *testing.T, input string, jsonRef string) {
err := testsuite.Unmarshal([]byte(input), &doc) err := testsuite.Unmarshal([]byte(input), &doc)
if err != nil { if err != nil {
if de, ok := err.(*toml.DecodeError); ok { de := &toml.DecodeError{}
if errors.As(err, &de) {
t.Logf("%s\n%s", err, de) t.Logf("%s\n%s", err, de)
} }
t.Fatalf("failed parsing toml: %s", err) t.Fatalf("failed parsing toml: %s", err)
} }
j, err := testsuite.ValueToTaggedJSON(doc) j, err := testsuite.ValueToTaggedJSON(doc)
require.NoError(t, err) assert.NoError(t, err)
var ref interface{} var ref interface{}
err = json.Unmarshal([]byte(jsonRef), &ref) err = json.Unmarshal([]byte(jsonRef), &ref)
require.NoError(t, err) assert.NoError(t, err)
var actual interface{} var actual interface{}
err = json.Unmarshal([]byte(j), &actual) err = json.Unmarshal(j, &actual)
require.NoError(t, err) assert.NoError(t, err)
testsuite.CmpJSON(t, "", ref, actual) testsuite.CmpJSON(t, "", ref, actual)
} }
+2453 -450
View File
File diff suppressed because it is too large Load Diff
+15 -6
View File
@@ -6,9 +6,18 @@ import (
"time" "time"
) )
var timeType = reflect.TypeOf(time.Time{}) // isZeroer is used to check if a type has a custom IsZero method.
var textMarshalerType = reflect.TypeOf(new(encoding.TextMarshaler)).Elem() // This allows custom types to define their own zero-value semantics.
var textUnmarshalerType = reflect.TypeOf(new(encoding.TextUnmarshaler)).Elem() type isZeroer interface {
var mapStringInterfaceType = reflect.TypeOf(map[string]interface{}{}) IsZero() bool
var sliceInterfaceType = reflect.TypeOf([]interface{}{}) }
var stringType = reflect.TypeOf("")
var (
timeType = reflect.TypeOf((*time.Time)(nil)).Elem()
textMarshalerType = reflect.TypeOf((*encoding.TextMarshaler)(nil)).Elem()
textUnmarshalerType = reflect.TypeOf((*encoding.TextUnmarshaler)(nil)).Elem()
isZeroerType = reflect.TypeOf((*isZeroer)(nil)).Elem()
mapStringInterfaceType = reflect.TypeOf(map[string]interface{}(nil))
sliceInterfaceType = reflect.TypeOf([]interface{}(nil))
stringType = reflect.TypeOf("")
)
+363 -161
View File
File diff suppressed because it is too large Load Diff
+1842 -118
View File
File diff suppressed because it is too large Load Diff
+145
View File
@@ -0,0 +1,145 @@
package unstable
import (
"errors"
"fmt"
)
// Iterator over a sequence of nodes.
//
// Starts uninitialized, you need to call Next() first.
//
// For example:
//
// it := n.Children()
// for it.Next() {
// n := it.Node()
// // do something with n
// }
type Iterator struct {
nodes *[]Node
idx int32
started bool
}
// Next moves the iterator forward and returns true if points to a
// node, false otherwise.
func (c *Iterator) Next() bool {
if c.nodes == nil {
return false
}
if !c.started {
c.started = true
} else if c.idx >= 0 {
c.idx = (*c.nodes)[c.idx].next
}
return c.idx >= 0 && int(c.idx) < len(*c.nodes)
}
// IsLast returns true if the current node of the iterator is the last
// one. Subsequent calls to Next() will return false.
func (c *Iterator) IsLast() bool {
return c.nodes == nil || c.idx < 0 || (*c.nodes)[c.idx].next < 0
}
// Node returns a pointer to the node pointed at by the iterator.
func (c *Iterator) Node() *Node {
if c.nodes == nil || c.idx < 0 {
return nil
}
n := &(*c.nodes)[c.idx]
n.nodes = c.nodes
return n
}
// Node in a TOML expression AST.
//
// Depending on Kind, its sequence of children should be interpreted
// differently.
//
// - Array have one child per element in the array.
// - InlineTable have one child per key-value in the table (each of kind
// InlineTable).
// - KeyValue have at least two children. The first one is the value. The rest
// make a potentially dotted key.
// - Table and ArrayTable's children represent a dotted key (same as
// KeyValue, but without the first node being the value).
//
// When relevant, Raw describes the range of bytes this node is referring to in
// the input document. Use Parser.Raw() to retrieve the actual bytes.
type Node struct {
Kind Kind
Raw Range // Raw bytes from the input.
Data []byte // Node value (either allocated or referencing the input).
// Absolute indices into the backing nodes slice. -1 means none.
next int32
child int32
// Reference to the backing nodes slice for navigation.
nodes *[]Node
}
// Range of bytes in the document.
type Range struct {
Offset uint32
Length uint32
}
// Next returns a pointer to the next node, or nil if there is no next node.
func (n *Node) Next() *Node {
if n.next < 0 {
return nil
}
next := &(*n.nodes)[n.next]
next.nodes = n.nodes
return next
}
// Child returns a pointer to the first child node of this node. Other children
// can be accessed calling Next on the first child. Returns nil if this Node
// has no child.
func (n *Node) Child() *Node {
if n.child < 0 {
return nil
}
child := &(*n.nodes)[n.child]
child.nodes = n.nodes
return child
}
// Valid returns true if the node's kind is set (not to Invalid).
func (n *Node) Valid() bool {
return n != nil
}
// Key returns the children nodes making the Key on a supported node. Panics
// otherwise. They are guaranteed to be all be of the Kind Key. A simple key
// would return just one element.
func (n *Node) Key() Iterator {
switch n.Kind {
case KeyValue:
child := n.child
if child < 0 {
panic(errors.New("KeyValue should have at least two children"))
}
valueNode := &(*n.nodes)[child]
return Iterator{nodes: n.nodes, idx: valueNode.next}
case Table, ArrayTable:
return Iterator{nodes: n.nodes, idx: n.child}
default:
panic(fmt.Errorf("Key() is not supported on a %s", n.Kind))
}
}
// Value returns a pointer to the value node of a KeyValue.
// Guaranteed to be non-nil. Panics if not called on a KeyValue node,
// or if the Children are malformed.
func (n *Node) Value() *Node {
return n.Child()
}
// Children returns an iterator over a node's children.
func (n *Node) Children() Iterator {
return Iterator{nodes: n.nodes, idx: n.child}
}
@@ -1,16 +1,18 @@
package toml package unstable
import ( import (
"bytes" "bytes"
"testing" "testing"
) )
var valid10Ascii = []byte("1234567890") var (
var valid10Utf8 = []byte("日本語a") valid10ASCII = []byte("1234567890")
var valid1kUtf8 = bytes.Repeat([]byte("0123456789日本語日本語日本語日abcdefghijklmnopqrstuvwx"), 16) valid10Utf8 = []byte("日本語a")
var valid1MUtf8 = bytes.Repeat(valid1kUtf8, 1024) valid1kUtf8 = bytes.Repeat([]byte("0123456789日本語日本語日本語日abcdefghijklmnopqrstuvwx"), 16)
var valid1kAscii = bytes.Repeat([]byte("012345678998jhjklasDJKLAAdjdfjsdklfjdslkabcdefghijklmnopqrstuvwx"), 16) valid1MUtf8 = bytes.Repeat(valid1kUtf8, 1024)
var valid1MAscii = bytes.Repeat(valid1kAscii, 1024) valid1kASCII = bytes.Repeat([]byte("012345678998jhjklasDJKLAAdjdfjsdklfjdslkabcdefghijklmnopqrstuvwx"), 16)
valid1MASCII = bytes.Repeat(valid1kASCII, 1024)
)
func BenchmarkScanComments(b *testing.B) { func BenchmarkScanComments(b *testing.B) {
wrap := func(x []byte) []byte { wrap := func(x []byte) []byte {
@@ -18,9 +20,9 @@ func BenchmarkScanComments(b *testing.B) {
} }
inputs := map[string][]byte{ inputs := map[string][]byte{
"10Valid": wrap(valid10Ascii), "10Valid": wrap(valid10ASCII),
"1kValid": wrap(valid1kAscii), "1kValid": wrap(valid1kASCII),
"1MValid": wrap(valid1MAscii), "1MValid": wrap(valid1MASCII),
"10ValidUtf8": wrap(valid10Utf8), "10ValidUtf8": wrap(valid10Utf8),
"1kValidUtf8": wrap(valid1kUtf8), "1kValidUtf8": wrap(valid1kUtf8),
"1MValidUtf8": wrap(valid1MUtf8), "1MValidUtf8": wrap(valid1MUtf8),
@@ -33,7 +35,7 @@ func BenchmarkScanComments(b *testing.B) {
b.ResetTimer() b.ResetTimer()
for i := 0; i < b.N; i++ { for i := 0; i < b.N; i++ {
scanComment(input) _, _, _ = scanComment(input)
} }
}) })
} }
@@ -45,9 +47,9 @@ func BenchmarkParseLiteralStringValid(b *testing.B) {
} }
inputs := map[string][]byte{ inputs := map[string][]byte{
"10Valid": wrap(valid10Ascii), "10Valid": wrap(valid10ASCII),
"1kValid": wrap(valid1kAscii), "1kValid": wrap(valid1kASCII),
"1MValid": wrap(valid1MAscii), "1MValid": wrap(valid1MASCII),
"10ValidUtf8": wrap(valid10Utf8), "10ValidUtf8": wrap(valid10Utf8),
"1kValidUtf8": wrap(valid1kUtf8), "1kValidUtf8": wrap(valid1kUtf8),
"1MValidUtf8": wrap(valid1MUtf8), "1MValidUtf8": wrap(valid1MUtf8),
@@ -55,7 +57,7 @@ func BenchmarkParseLiteralStringValid(b *testing.B) {
for name, input := range inputs { for name, input := range inputs {
b.Run(name, func(b *testing.B) { b.Run(name, func(b *testing.B) {
p := parser{} p := Parser{}
b.SetBytes(int64(len(input))) b.SetBytes(int64(len(input)))
b.ReportAllocs() b.ReportAllocs()
b.ResetTimer() b.ResetTimer()
@@ -63,7 +65,7 @@ func BenchmarkParseLiteralStringValid(b *testing.B) {
for i := 0; i < b.N; i++ { for i := 0; i < b.N; i++ {
_, _, _, err := p.parseLiteralString(input) _, _, _, err := p.parseLiteralString(input)
if err != nil { if err != nil {
panic(err) b.Error(err)
} }
} }
}) })
+64
View File
@@ -0,0 +1,64 @@
package unstable
// root contains a full AST.
//
// It is immutable once constructed with Builder.
type root struct {
nodes []Node
}
func (r *root) at(idx reference) *Node {
return &r.nodes[idx]
}
type reference int
const invalidReference reference = -1
func (r reference) Valid() bool {
return r != invalidReference
}
type builder struct {
tree root
lastIdx int
}
func (b *builder) NodeAt(ref reference) *Node {
n := b.tree.at(ref)
n.nodes = &b.tree.nodes
return n
}
func (b *builder) Reset() {
b.tree.nodes = b.tree.nodes[:0]
b.lastIdx = 0
}
func (b *builder) Push(n Node) reference {
b.lastIdx = len(b.tree.nodes)
n.next = -1
n.child = -1
b.tree.nodes = append(b.tree.nodes, n)
return reference(b.lastIdx)
}
func (b *builder) PushAndChain(n Node) reference {
newIdx := len(b.tree.nodes)
n.next = -1
n.child = -1
b.tree.nodes = append(b.tree.nodes, n)
if b.lastIdx >= 0 {
b.tree.nodes[b.lastIdx].next = int32(newIdx) //nolint:gosec // TOML ASTs are small
}
b.lastIdx = newIdx
return reference(b.lastIdx)
}
func (b *builder) AttachChild(parent reference, child reference) {
b.tree.nodes[parent].child = int32(child) //nolint:gosec // TOML ASTs are small
}
func (b *builder) Chain(from reference, to reference) {
b.tree.nodes[from].next = int32(to) //nolint:gosec // TOML ASTs are small
}
+3
View File
@@ -0,0 +1,3 @@
// Package unstable provides APIs that do not meet the backward compatibility
// guarantees yet.
package unstable
+19 -5
View File
@@ -1,35 +1,49 @@
package ast package unstable
import "fmt" import "fmt"
// Kind represents the type of TOML structure contained in a given Node.
type Kind int type Kind int
const ( const (
// meta // Invalid represents an invalid meta node.
Invalid Kind = iota Invalid Kind = iota
// Comment represents a comment meta node.
Comment Comment
// Key represents a key meta node.
Key Key
// top level structures // Table represents a top-level table.
Table Table
// ArrayTable represents a top-level array table.
ArrayTable ArrayTable
// KeyValue represents a top-level key value.
KeyValue KeyValue
// containers values // Array represents an array container value.
Array Array
// InlineTable represents an inline table container value.
InlineTable InlineTable
// values // String represents a string value.
String String
// Bool represents a boolean value.
Bool Bool
// Float represents a floating point value.
Float Float
// Integer represents an integer value.
Integer Integer
// LocalDate represents a a local date value.
LocalDate LocalDate
// LocalTime represents a local time value.
LocalTime LocalTime
// LocalDateTime represents a local date/time value.
LocalDateTime LocalDateTime
// DateTime represents a data/time value.
DateTime DateTime
) )
// String implementation of fmt.Stringer.
func (k Kind) String() string { func (k Kind) String() string {
switch k { switch k {
case Invalid: case Invalid:
File diff suppressed because it is too large Load Diff
+883
View File
@@ -0,0 +1,883 @@
package unstable
import (
"fmt"
"strconv"
"strings"
"testing"
"github.com/pelletier/go-toml/v2/internal/assert"
)
func TestParser_AST_Numbers(t *testing.T) {
examples := []struct {
desc string
input string
kind Kind
err bool
}{
{
desc: "integer just digits",
input: `1234`,
kind: Integer,
},
{
desc: "integer zero",
input: `0`,
kind: Integer,
},
{
desc: "integer sign",
input: `+99`,
kind: Integer,
},
{
desc: "integer hex uppercase",
input: `0xDEADBEEF`,
kind: Integer,
},
{
desc: "integer hex lowercase",
input: `0xdead_beef`,
kind: Integer,
},
{
desc: "integer octal",
input: `0o01234567`,
kind: Integer,
},
{
desc: "integer binary",
input: `0b11010110`,
kind: Integer,
},
{
desc: "float zero",
input: `0.0`,
kind: Float,
},
{
desc: "float positive zero",
input: `+0.0`,
kind: Float,
},
{
desc: "float negative zero",
input: `-0.0`,
kind: Float,
},
{
desc: "float pi",
input: `3.1415`,
kind: Float,
},
{
desc: "float negative",
input: `-0.01`,
kind: Float,
},
{
desc: "float signed exponent",
input: `5e+22`,
kind: Float,
},
{
desc: "float exponent lowercase",
input: `1e06`,
kind: Float,
},
{
desc: "float exponent uppercase",
input: `-2E-2`,
kind: Float,
},
{
desc: "float fractional with exponent",
input: `6.626e-34`,
kind: Float,
},
{
desc: "float underscores",
input: `224_617.445_991_228`,
kind: Float,
},
{
desc: "inf",
input: `inf`,
kind: Float,
},
{
desc: "inf negative",
input: `-inf`,
kind: Float,
},
{
desc: "inf positive",
input: `+inf`,
kind: Float,
},
{
desc: "nan",
input: `nan`,
kind: Float,
},
{
desc: "nan negative",
input: `-nan`,
kind: Float,
},
{
desc: "nan positive",
input: `+nan`,
kind: Float,
},
}
for _, e := range examples {
e := e
t.Run(e.desc, func(t *testing.T) {
p := Parser{}
p.Reset([]byte(`A = ` + e.input))
p.NextExpression()
err := p.Error()
if e.err {
assert.Error(t, err)
} else {
assert.NoError(t, err)
expected := astNode{
Kind: KeyValue,
Children: []astNode{
{Kind: e.kind, Data: []byte(e.input)},
{Kind: Key, Data: []byte(`A`)},
},
}
compareNode(t, expected, p.Expression())
}
})
}
}
type (
astNode struct {
Kind Kind
Data []byte
Children []astNode
}
)
func compareNode(t *testing.T, e astNode, n *Node) {
t.Helper()
assert.Equal(t, e.Kind, n.Kind)
assert.Equal(t, e.Data, n.Data)
compareIterator(t, e.Children, n.Children())
}
func compareIterator(t *testing.T, expected []astNode, actual Iterator) {
t.Helper()
idx := 0
for actual.Next() {
n := actual.Node()
if idx >= len(expected) {
t.Fatal("extra child in actual tree")
}
e := expected[idx]
compareNode(t, e, n)
idx++
}
if idx < len(expected) {
t.Fatal("missing children in actual", "idx =", idx, "expected =", len(expected))
}
}
//nolint:funlen
func TestParser_AST(t *testing.T) {
examples := []struct {
desc string
input string
ast astNode
err bool
}{
{
desc: "simple string assignment",
input: `A = "hello"`,
ast: astNode{
Kind: KeyValue,
Children: []astNode{
{
Kind: String,
Data: []byte(`hello`),
},
{
Kind: Key,
Data: []byte(`A`),
},
},
},
},
{
desc: "simple bool assignment",
input: `A = true`,
ast: astNode{
Kind: KeyValue,
Children: []astNode{
{
Kind: Bool,
Data: []byte(`true`),
},
{
Kind: Key,
Data: []byte(`A`),
},
},
},
},
{
desc: "array of strings",
input: `A = ["hello", ["world", "again"]]`,
ast: astNode{
Kind: KeyValue,
Children: []astNode{
{
Kind: Array,
Children: []astNode{
{
Kind: String,
Data: []byte(`hello`),
},
{
Kind: Array,
Children: []astNode{
{
Kind: String,
Data: []byte(`world`),
},
{
Kind: String,
Data: []byte(`again`),
},
},
},
},
},
{
Kind: Key,
Data: []byte(`A`),
},
},
},
},
{
desc: "array of arrays of strings",
input: `A = ["hello", "world"]`,
ast: astNode{
Kind: KeyValue,
Children: []astNode{
{
Kind: Array,
Children: []astNode{
{
Kind: String,
Data: []byte(`hello`),
},
{
Kind: String,
Data: []byte(`world`),
},
},
},
{
Kind: Key,
Data: []byte(`A`),
},
},
},
},
{
desc: "inline table",
input: `name = { first = "Tom", last = "Preston-Werner" }`,
ast: astNode{
Kind: KeyValue,
Children: []astNode{
{
Kind: InlineTable,
Children: []astNode{
{
Kind: KeyValue,
Children: []astNode{
{Kind: String, Data: []byte(`Tom`)},
{Kind: Key, Data: []byte(`first`)},
},
},
{
Kind: KeyValue,
Children: []astNode{
{Kind: String, Data: []byte(`Preston-Werner`)},
{Kind: Key, Data: []byte(`last`)},
},
},
},
},
{
Kind: Key,
Data: []byte(`name`),
},
},
},
},
{
desc: "multiline inline table",
input: "name = {\n first = \"Tom\",\n last = \"Preston-Werner\"\n}",
ast: astNode{
Kind: KeyValue,
Children: []astNode{
{
Kind: InlineTable,
Children: []astNode{
{
Kind: KeyValue,
Children: []astNode{
{Kind: String, Data: []byte(`Tom`)},
{Kind: Key, Data: []byte(`first`)},
},
},
{
Kind: KeyValue,
Children: []astNode{
{Kind: String, Data: []byte(`Preston-Werner`)},
{Kind: Key, Data: []byte(`last`)},
},
},
},
},
{
Kind: Key,
Data: []byte(`name`),
},
},
},
},
{
desc: "inline table with trailing comma",
input: `name = { first = "Tom", last = "Preston-Werner", }`,
ast: astNode{
Kind: KeyValue,
Children: []astNode{
{
Kind: InlineTable,
Children: []astNode{
{
Kind: KeyValue,
Children: []astNode{
{Kind: String, Data: []byte(`Tom`)},
{Kind: Key, Data: []byte(`first`)},
},
},
{
Kind: KeyValue,
Children: []astNode{
{Kind: String, Data: []byte(`Preston-Werner`)},
{Kind: Key, Data: []byte(`last`)},
},
},
},
},
{
Kind: Key,
Data: []byte(`name`),
},
},
},
},
{
desc: "empty inline table with newline",
input: "name = {\n}",
ast: astNode{
Kind: KeyValue,
Children: []astNode{
{
Kind: InlineTable,
Children: nil,
},
{
Kind: Key,
Data: []byte(`name`),
},
},
},
},
{
desc: "inline table with leading comma",
input: "name = { first = \"Tom\"\n, last = \"Werner\" }",
ast: astNode{
Kind: KeyValue,
Children: []astNode{
{
Kind: InlineTable,
Children: []astNode{
{
Kind: KeyValue,
Children: []astNode{
{Kind: String, Data: []byte(`Tom`)},
{Kind: Key, Data: []byte(`first`)},
},
},
{
Kind: KeyValue,
Children: []astNode{
{Kind: String, Data: []byte(`Werner`)},
{Kind: Key, Data: []byte(`last`)},
},
},
},
},
{
Kind: Key,
Data: []byte(`name`),
},
},
},
},
{
desc: "inline table with leading trailing comma",
input: "name = { first = \"Tom\"\n, }",
ast: astNode{
Kind: KeyValue,
Children: []astNode{
{
Kind: InlineTable,
Children: []astNode{
{
Kind: KeyValue,
Children: []astNode{
{Kind: String, Data: []byte(`Tom`)},
{Kind: Key, Data: []byte(`first`)},
},
},
},
},
{
Kind: Key,
Data: []byte(`name`),
},
},
},
},
{
desc: "inline table comma at start is error",
input: "name = { , first = \"Tom\" }",
err: true,
},
{
desc: "inline table double comma across newline is error",
input: "name = { first = \"Tom\",\n, last = \"Werner\" }",
err: true,
},
}
for _, e := range examples {
e := e
t.Run(e.desc, func(t *testing.T) {
p := Parser{}
p.Reset([]byte(e.input))
p.NextExpression()
err := p.Error()
if e.err {
assert.Error(t, err)
} else {
assert.NoError(t, err)
compareNode(t, e.ast, p.Expression())
}
})
}
}
func TestParseInlineTable_CommentsWithKeepComments(t *testing.T) {
// Exercise comment reference handling inside parseInlineTable when
// KeepComments is true. This covers the addChild(cref) branches
// at the start of the loop, after comma, and after keyval.
examples := []struct {
desc string
input string
}{
{
desc: "comment at start of inline table",
input: "a = {\n# comment\nb = 1\n}",
},
{
desc: "comment after comma",
input: "a = {b = 1,\n# comment\nc = 2\n}",
},
{
desc: "comment after keyval",
input: "a = {b = 1 # comment\n, c = 2}",
},
{
desc: "comment only in inline table",
input: "a = {\n# just a comment\n}",
},
}
for _, e := range examples {
e := e
t.Run(e.desc, func(t *testing.T) {
p := Parser{KeepComments: true}
p.Reset([]byte(e.input))
p.NextExpression()
err := p.Error()
assert.NoError(t, err)
})
}
}
func BenchmarkParseBasicStringWithUnicode(b *testing.B) {
p := &Parser{}
b.Run("4", func(b *testing.B) {
input := []byte(`"\u1234\u5678\u9ABC\u1234\u5678\u9ABC"`)
b.ReportAllocs()
b.SetBytes(int64(len(input)))
for i := 0; i < b.N; i++ {
_, _, _, _ = p.parseBasicString(input)
}
})
b.Run("8", func(b *testing.B) {
input := []byte(`"\u12345678\u9ABCDEF0\u12345678\u9ABCDEF0"`)
b.ReportAllocs()
b.SetBytes(int64(len(input)))
for i := 0; i < b.N; i++ {
_, _, _, _ = p.parseBasicString(input)
}
})
}
func BenchmarkParseBasicStringsEasy(b *testing.B) {
p := &Parser{}
for _, size := range []int{1, 4, 8, 16, 21} {
b.Run(strconv.Itoa(size), func(b *testing.B) {
input := []byte(`"` + strings.Repeat("A", size) + `"`)
b.ReportAllocs()
b.SetBytes(int64(len(input)))
for i := 0; i < b.N; i++ {
_, _, _, _ = p.parseBasicString(input)
}
})
}
}
func TestParser_AST_DateTimes(t *testing.T) {
examples := []struct {
desc string
input string
kind Kind
err bool
}{
{
desc: "offset-date-time with delim 'T' and UTC offset",
input: `2021-07-21T12:08:05Z`,
kind: DateTime,
},
{
desc: "offset-date-time with space delim and +8hours offset",
input: `2021-07-21 12:08:05+08:00`,
kind: DateTime,
},
{
desc: "local-date-time with nano second",
input: `2021-07-21T12:08:05.666666666`,
kind: LocalDateTime,
},
{
desc: "local-date-time",
input: `2021-07-21T12:08:05`,
kind: LocalDateTime,
},
{
desc: "local-date",
input: `2021-07-21`,
kind: LocalDate,
},
}
for _, e := range examples {
e := e
t.Run(e.desc, func(t *testing.T) {
p := Parser{}
p.Reset([]byte(`A = ` + e.input))
p.NextExpression()
err := p.Error()
if e.err {
assert.Error(t, err)
} else {
assert.NoError(t, err)
expected := astNode{
Kind: KeyValue,
Children: []astNode{
{Kind: e.kind, Data: []byte(e.input)},
{Kind: Key, Data: []byte(`A`)},
},
}
compareNode(t, expected, p.Expression())
}
})
}
}
// This example demonstrates how to parse a TOML document and preserving
// comments. Comments are stored in the AST as Comment nodes. This example
// displays the structure of the full AST generated by the parser using the
// following structure:
//
// 1. Each root-level expression is separated by three dashes.
// 2. Bytes associated to a node are displayed in square brackets.
// 3. Siblings have the same indentation.
// 4. Children of a node are indented one level.
func ExampleParser_comments() {
doc := `# Top of the document comment.
# Optional, any amount of lines.
# Above table.
[table] # Next to table.
# Above simple value.
key = "value" # Next to simple value.
# Below simple value.
# Some comment alone.
# Multiple comments, on multiple lines.
# Above inline table.
name = { first = "Tom", last = "Preston-Werner" } # Next to inline table.
# Below inline table.
# Above array.
array = [ 1, 2, 3 ] # Next to one-line array.
# Below array.
# Above multi-line array.
key5 = [ # Next to start of inline array.
# Second line before array content.
1, # Next to first element.
# After first element.
# Before second element.
2,
3, # Next to last element
# After last element.
] # Next to end of array.
# Below multi-line array.
# Before array table.
[[products]] # Next to array table.
# After array table.
`
var printGeneric func(*Parser, int, *Node)
printGeneric = func(p *Parser, indent int, e *Node) {
if e == nil {
return
}
s := p.Shape(e.Raw)
x := fmt.Sprintf("%d:%d->%d:%d (%d->%d)", s.Start.Line, s.Start.Column, s.End.Line, s.End.Column, s.Start.Offset, s.End.Offset)
fmt.Printf("%-25s | %s%s [%s]\n", x, strings.Repeat(" ", indent), e.Kind, e.Data)
printGeneric(p, indent+1, e.Child())
printGeneric(p, indent, e.Next())
}
printTree := func(p *Parser) {
for p.NextExpression() {
e := p.Expression()
fmt.Println("---")
printGeneric(p, 0, e)
}
if err := p.Error(); err != nil {
panic(err)
}
}
p := &Parser{
KeepComments: true,
}
p.Reset([]byte(doc))
printTree(p)
// Output:
// ---
// 1:1->1:31 (0->30) | Comment [# Top of the document comment.]
// ---
// 2:1->2:33 (31->63) | Comment [# Optional, any amount of lines.]
// ---
// 4:1->4:15 (65->79) | Comment [# Above table.]
// ---
// 1:1->1:1 (0->0) | Table []
// 5:2->5:7 (81->86) | Key [table]
// 5:9->5:25 (88->104) | Comment [# Next to table.]
// ---
// 6:1->6:22 (105->126) | Comment [# Above simple value.]
// ---
// 1:1->1:1 (0->0) | KeyValue []
// 7:7->7:14 (133->140) | String [value]
// 7:1->7:4 (127->130) | Key [key]
// 7:15->7:38 (141->164) | Comment [# Next to simple value.]
// ---
// 8:1->8:22 (165->186) | Comment [# Below simple value.]
// ---
// 10:1->10:22 (188->209) | Comment [# Some comment alone.]
// ---
// 12:1->12:40 (211->250) | Comment [# Multiple comments, on multiple lines.]
// ---
// 14:1->14:22 (252->273) | Comment [# Above inline table.]
// ---
// 1:1->1:1 (0->0) | KeyValue []
// 15:8->15:9 (281->282) | InlineTable []
// 1:1->1:1 (0->0) | KeyValue []
// 15:18->15:23 (291->296) | String [Tom]
// 15:10->15:15 (283->288) | Key [first]
// 1:1->1:1 (0->0) | KeyValue []
// 15:32->15:48 (305->321) | String [Preston-Werner]
// 15:25->15:29 (298->302) | Key [last]
// 15:1->15:5 (274->278) | Key [name]
// 15:51->15:74 (324->347) | Comment [# Next to inline table.]
// ---
// 16:1->16:22 (348->369) | Comment [# Below inline table.]
// ---
// 18:1->18:15 (371->385) | Comment [# Above array.]
// ---
// 1:1->1:1 (0->0) | KeyValue []
// 1:1->1:1 (0->0) | Array []
// 19:11->19:12 (396->397) | Integer [1]
// 19:14->19:15 (399->400) | Integer [2]
// 19:17->19:18 (402->403) | Integer [3]
// 19:1->19:6 (386->391) | Key [array]
// 19:21->19:46 (406->431) | Comment [# Next to one-line array.]
// ---
// 20:1->20:15 (432->446) | Comment [# Below array.]
// ---
// 22:1->22:26 (448->473) | Comment [# Above multi-line array.]
// ---
// 1:1->1:1 (0->0) | KeyValue []
// 1:1->1:1 (0->0) | Array []
// 23:10->23:42 (483->515) | Comment [# Next to start of inline array.]
// 24:3->24:38 (518->553) | Comment [# Second line before array content.]
// 25:3->25:4 (556->557) | Integer [1]
// 25:6->25:30 (559->583) | Comment [# Next to first element.]
// 26:3->26:25 (586->608) | Comment [# After first element.]
// 27:3->27:27 (611->635) | Comment [# Before second element.]
// 28:3->28:4 (638->639) | Integer [2]
// 29:3->29:4 (643->644) | Integer [3]
// 29:6->29:28 (646->668) | Comment [# Next to last element]
// 30:3->30:24 (671->692) | Comment [# After last element.]
// 23:1->23:5 (474->478) | Key [key5]
// 31:3->31:26 (695->718) | Comment [# Next to end of array.]
// ---
// 32:1->32:26 (719->744) | Comment [# Below multi-line array.]
// ---
// 34:1->34:22 (746->767) | Comment [# Before array table.]
// ---
// 1:1->1:1 (0->0) | ArrayTable []
// 35:3->35:11 (770->778) | Key [products]
// 35:14->35:36 (781->803) | Comment [# Next to array table.]
// ---
// 36:1->36:21 (804->824) | Comment [# After array table.]
}
func TestIterator_IsLast(t *testing.T) {
// Test IsLast on an iterator with multiple elements using public Parser API
doc := `array = [1, 2, 3]`
p := Parser{}
p.Reset([]byte(doc))
p.NextExpression()
e := p.Expression()
arr := e.Value() // The array node
it := arr.Children()
count := 0
lastCount := 0
for it.Next() {
count++
if it.IsLast() {
lastCount++
}
}
assert.Equal(t, 3, count)
assert.Equal(t, 1, lastCount)
}
func TestNodeChaining(t *testing.T) {
// Test that sibling nodes are correctly chained via Next()
// This exercises the internal PushAndChain functionality through public APIs
doc := `a.b.c = 1`
p := Parser{}
p.Reset([]byte(doc))
p.NextExpression()
e := p.Expression()
// KeyValue has children: value, then key parts (a, b, c)
keyIt := e.Key()
// Collect all key parts by following the iterator
var keys []string
for keyIt.Next() {
keys = append(keys, string(keyIt.Node().Data))
}
assert.Equal(t, []string{"a", "b", "c"}, keys)
}
func TestMultipleExpressions(t *testing.T) {
// Test parsing multiple top-level expressions
// This exercises root iteration through public APIs
doc := `
key1 = "value1"
key2 = "value2"
key3 = "value3"
`
p := Parser{}
p.Reset([]byte(doc))
var keys []string
for p.NextExpression() {
e := p.Expression()
keyIt := e.Key()
keyIt.Next()
keys = append(keys, string(keyIt.Node().Data))
}
assert.NoError(t, p.Error())
assert.Equal(t, []string{"key1", "key2", "key3"}, keys)
}
func ExampleParser() {
doc := `
hello = "world"
value = 42
`
p := Parser{}
p.Reset([]byte(doc))
for p.NextExpression() {
e := p.Expression()
fmt.Printf("Expression: %s\n", e.Kind)
value := e.Value()
it := e.Key()
k := it.Node() // shortcut: we know there is no dotted key in the example
fmt.Printf("%s -> (%s) %s\n", k.Data, value.Kind, value.Data)
}
// Output:
// Expression: KeyValue
// hello -> (String) world
// Expression: KeyValue
// value -> (Integer) 42
}
+26 -25
View File
@@ -1,4 +1,6 @@
package toml package unstable
import "github.com/pelletier/go-toml/v2/internal/characters"
func scanFollows(b []byte, pattern string) bool { func scanFollows(b []byte, pattern string) bool {
n := len(pattern) n := len(pattern)
@@ -54,16 +56,16 @@ func scanLiteralString(b []byte) ([]byte, []byte, error) {
case '\'': case '\'':
return b[:i+1], b[i+1:], nil return b[:i+1], b[i+1:], nil
case '\n', '\r': case '\n', '\r':
return nil, nil, newDecodeError(b[i:i+1], "literal strings cannot have new lines") return nil, nil, NewParserError(b[i:i+1], "literal strings cannot have new lines")
} }
size := utf8ValidNext(b[i:]) size := characters.Utf8ValidNext(b[i:])
if size == 0 { if size == 0 {
return nil, nil, newDecodeError(b[i:i+1], "invalid character") return nil, nil, NewParserError(b[i:i+1], "invalid character")
} }
i += size i += size
} }
return nil, nil, newDecodeError(b[len(b):], "unterminated literal string") return nil, nil, NewParserError(b[len(b):], "unterminated literal string")
} }
func scanMultilineLiteralString(b []byte) ([]byte, []byte, error) { func scanMultilineLiteralString(b []byte) ([]byte, []byte, error) {
@@ -98,39 +100,39 @@ func scanMultilineLiteralString(b []byte) ([]byte, []byte, error) {
i++ i++
if i < len(b) && b[i] == '\'' { if i < len(b) && b[i] == '\'' {
return nil, nil, newDecodeError(b[i-3:i+1], "''' not allowed in multiline literal string") return nil, nil, NewParserError(b[i-3:i+1], "''' not allowed in multiline literal string")
} }
return b[:i], b[i:], nil return b[:i], b[i:], nil
} }
case '\r': case '\r':
if len(b) < i+2 { if len(b) < i+2 {
return nil, nil, newDecodeError(b[len(b):], `need a \n after \r`) return nil, nil, NewParserError(b[len(b):], `need a \n after \r`)
} }
if b[i+1] != '\n' { if b[i+1] != '\n' {
return nil, nil, newDecodeError(b[i:i+2], `need a \n after \r`) return nil, nil, NewParserError(b[i:i+2], `need a \n after \r`)
} }
i += 2 // skip the \n i += 2 // skip the \n
continue continue
} }
size := utf8ValidNext(b[i:]) size := characters.Utf8ValidNext(b[i:])
if size == 0 { if size == 0 {
return nil, nil, newDecodeError(b[i:i+1], "invalid character") return nil, nil, NewParserError(b[i:i+1], "invalid character")
} }
i += size i += size
} }
return nil, nil, newDecodeError(b[len(b):], `multiline literal string not terminated by '''`) return nil, nil, NewParserError(b[len(b):], `multiline literal string not terminated by '''`)
} }
func scanWindowsNewline(b []byte) ([]byte, []byte, error) { func scanWindowsNewline(b []byte) ([]byte, []byte, error) {
const lenCRLF = 2 const lenCRLF = 2
if len(b) < lenCRLF { if len(b) < lenCRLF {
return nil, nil, newDecodeError(b, "windows new line expected") return nil, nil, NewParserError(b, "windows new line expected")
} }
if b[1] != '\n' { if b[1] != '\n' {
return nil, nil, newDecodeError(b, `windows new line should be \r\n`) return nil, nil, NewParserError(b, `windows new line should be \r\n`)
} }
return b[:lenCRLF], b[lenCRLF:], nil return b[:lenCRLF], b[lenCRLF:], nil
@@ -149,7 +151,6 @@ func scanWhitespace(b []byte) ([]byte, []byte) {
return b, b[len(b):] return b, b[len(b):]
} }
//nolint:unparam
func scanComment(b []byte) ([]byte, []byte, error) { func scanComment(b []byte) ([]byte, []byte, error) {
// comment-start-symbol = %x23 ; # // comment-start-symbol = %x23 ; #
// non-ascii = %x80-D7FF / %xE000-10FFFF // non-ascii = %x80-D7FF / %xE000-10FFFF
@@ -165,11 +166,11 @@ func scanComment(b []byte) ([]byte, []byte, error) {
if i+1 < len(b) && b[i+1] == '\n' { if i+1 < len(b) && b[i+1] == '\n' {
return b[:i+1], b[i+1:], nil return b[:i+1], b[i+1:], nil
} }
return nil, nil, newDecodeError(b[i:i+1], "invalid character in comment") return nil, nil, NewParserError(b[i:i+1], "invalid character in comment")
} }
size := utf8ValidNext(b[i:]) size := characters.Utf8ValidNext(b[i:])
if size == 0 { if size == 0 {
return nil, nil, newDecodeError(b[i:i+1], "invalid character in comment") return nil, nil, NewParserError(b[i:i+1], "invalid character in comment")
} }
i += size i += size
@@ -192,17 +193,17 @@ func scanBasicString(b []byte) ([]byte, bool, []byte, error) {
case '"': case '"':
return b[:i+1], escaped, b[i+1:], nil return b[:i+1], escaped, b[i+1:], nil
case '\n', '\r': case '\n', '\r':
return nil, escaped, nil, newDecodeError(b[i:i+1], "basic strings cannot have new lines") return nil, escaped, nil, NewParserError(b[i:i+1], "basic strings cannot have new lines")
case '\\': case '\\':
if len(b) < i+2 { if len(b) < i+2 {
return nil, escaped, nil, newDecodeError(b[i:i+1], "need a character after \\") return nil, escaped, nil, NewParserError(b[i:i+1], "need a character after \\")
} }
escaped = true escaped = true
i++ // skip the next character i++ // skip the next character
} }
} }
return nil, escaped, nil, newDecodeError(b[len(b):], `basic string not terminated by "`) return nil, escaped, nil, NewParserError(b[len(b):], `basic string not terminated by "`)
} }
func scanMultilineBasicString(b []byte) ([]byte, bool, []byte, error) { func scanMultilineBasicString(b []byte) ([]byte, bool, []byte, error) {
@@ -243,27 +244,27 @@ func scanMultilineBasicString(b []byte) ([]byte, bool, []byte, error) {
i++ i++
if i < len(b) && b[i] == '"' { if i < len(b) && b[i] == '"' {
return nil, escaped, nil, newDecodeError(b[i-3:i+1], `""" not allowed in multiline basic string`) return nil, escaped, nil, NewParserError(b[i-3:i+1], `""" not allowed in multiline basic string`)
} }
return b[:i], escaped, b[i:], nil return b[:i], escaped, b[i:], nil
} }
case '\\': case '\\':
if len(b) < i+2 { if len(b) < i+2 {
return nil, escaped, nil, newDecodeError(b[len(b):], "need a character after \\") return nil, escaped, nil, NewParserError(b[len(b):], "need a character after \\")
} }
escaped = true escaped = true
i++ // skip the next character i++ // skip the next character
case '\r': case '\r':
if len(b) < i+2 { if len(b) < i+2 {
return nil, escaped, nil, newDecodeError(b[len(b):], `need a \n after \r`) return nil, escaped, nil, NewParserError(b[len(b):], `need a \n after \r`)
} }
if b[i+1] != '\n' { if b[i+1] != '\n' {
return nil, escaped, nil, newDecodeError(b[i:i+2], `need a \n after \r`) return nil, escaped, nil, NewParserError(b[i:i+2], `need a \n after \r`)
} }
i++ // skip the \n i++ // skip the \n
} }
} }
return nil, escaped, nil, newDecodeError(b[len(b):], `multiline basic string not terminated by """`) return nil, escaped, nil, NewParserError(b[len(b):], `multiline basic string not terminated by """`)
} }
+7
View File
@@ -0,0 +1,7 @@
package unstable
// The Unmarshaler interface may be implemented by types to customize their
// behavior when being unmarshaled from a TOML document.
type Unmarshaler interface {
UnmarshalTOML(value *Node) error
}