Compare commits

...

53 Commits

Author SHA1 Message Date
Daniel Weiße d00d2cca6e Fix indentation of custom type arrays (#944)
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2024-04-12 10:42:12 -04:00
dependabot[bot] 86608d7fca build(deps): bump github/codeql-action from 2 to 3 (#919)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2 to 3.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/v2...v3)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-03-19 13:24:46 -04:00
dependabot[bot] 4a1877957a build(deps): bump actions/setup-go from 4 to 5 (#916)
Bumps [actions/setup-go](https://github.com/actions/setup-go) from 4 to 5.
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](https://github.com/actions/setup-go/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/setup-go
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-03-19 13:24:37 -04:00
dependabot[bot] 3021d6d033 build(deps): bump actions/upload-artifact from 3 to 4 (#920)
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 3 to 4.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-03-19 13:24:24 -04:00
Thomas Pelletier 32788f26f8 Update release instructions (#941) 2024-03-19 12:47:39 -04:00
rszyma 8ed6d131eb Decode: unstable/Unmarshal interface (#940)
Co-authored-by: Pavlos Karakalidis <pkarakal@pkarakal.com>
Co-authored-by: Thomas Pelletier <thomas@pelletier.codes>
2024-03-19 12:33:12 -04:00
dependabot[bot] 7dad87762a build(deps): bump github.com/stretchr/testify from 1.8.4 to 1.9.0 (#936)
Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.8.4 to 1.9.0.
- [Release notes](https://github.com/stretchr/testify/releases)
- [Commits](https://github.com/stretchr/testify/compare/v1.8.4...v1.9.0)

---
updated-dependencies:
- dependency-name: github.com/stretchr/testify
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-03-12 09:06:45 -04:00
Thomas Pelletier 2b69615b5d Go 1.22 support (#935) 2024-02-27 15:30:13 -05:00
Thomas Pelletier 06fb30bf2e Decode: fix reuse of slice for array tables (#934)
When decoding into a non-empty slice, it needs to be emptied so that only the
tables contained in the document are present in the resulting value.

Arrays are not impacted because their unmarshal offset is tracked separately.

Fixes #931
2024-02-27 15:28:49 -05:00
Thomas Pelletier 2e087bdf5f Run tests on macos/M1 (#929)
https://github.blog/changelog/2024-01-30-github-actions-introducing-the-new-m1-macos-runner-available-to-open-source/
2024-01-30 19:15:50 -05:00
Thomas Pelletier caeb9f9631 Fix marshaler typos (#927) 2024-01-30 19:01:55 -05:00
Rdbo e7223fb40e fix: odd indentation in README (#928) 2024-01-30 19:01:43 -05:00
Jakub Wilk 05bedf36d8 Fix README typo (#925) 2024-01-25 15:21:33 -08:00
Daniel Graña f5486d590f Support encoding json.Number type (#923)
Co-authored-by: Thomas Pelletier <thomas@pelletier.codes>
2024-01-25 15:21:02 -08:00
Daniel Graña 2ca21fb7b4 Support encoding of pointers to embedded structs (#924) 2024-01-23 13:06:33 -05:00
Thomas Pelletier 34765b4a9e Fix unmarshaling of nested non-exported struct (#917)
Fixes #915
2023-12-11 14:17:49 -05:00
Moritz Poldrack 358c8d2c23 Use toml-test to generate tests (#911)
Fixes: #909
2023-10-26 12:05:02 -06:00
Martin Tournoij fd8d0bf4d9 Add cmd/gotoml-test-encoder (#907) 2023-10-23 14:40:44 -06:00
Thomas Pelletier a76e18e8c5 Fix benchmark script (#905) 2023-10-02 13:49:01 -04:00
dependabot[bot] dff0c128d0 build(deps): bump docker/login-action from 2 to 3 (#901)
Bumps [docker/login-action](https://github.com/docker/login-action) from 2 to 3.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/v2...v3)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-17 18:11:24 -04:00
Thomas Pelletier 3573ce3770 Update SECURITY.md
Remove placeholder.
2023-09-04 09:43:32 -04:00
dependabot[bot] ae933f2e2a build(deps): bump actions/checkout from 3 to 4 (#896)
Bumps [actions/checkout](https://github.com/actions/checkout) from 3 to 4.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-04 09:42:37 -04:00
Thomas Pelletier 3175efb395 encode: fix commented table with comment (#894)
And updated README.
2023-08-29 08:46:34 -04:00
Thomas Pelletier 9dd7f1af78 encoder: add back the commented option (#893) 2023-08-28 14:29:26 -04:00
Mikhail f. Shiryaev 4a5c27c299 decode: fix wrong indention for tables' comments (#892) 2023-08-28 13:23:11 -04:00
Thomas Pelletier 76cc96f6d8 Update LICENSE 2023-08-28 11:22:32 -04:00
Thomas Pelletier 4835627845 Decode: improve errors on integers and strings (#891) 2023-08-28 11:17:48 -04:00
Thomas Pelletier cef80b96a4 parser: add raw to integers (#890) 2023-08-28 11:02:16 -04:00
Thomas Pelletier 4040373cfd Encode: fix ignored indent of array tables (#889)
Fixes #888
2023-08-28 09:52:11 -04:00
Haiyang Wang bb026cae89 Decode: fix panic when parsing '0' as a float (#887)
Fixes #886
2023-08-22 18:07:39 +02:00
michalbiesek f7d9b9ba53 Build riscv64 binaries for linux (#884)
Signed-off-by: Michal Biesek <michalbiesek@gmail.com>
2023-08-20 19:48:24 +02:00
michalbiesek fac33d6fa8 Add support for Go 1.21 (#885)
* Add support for `Go 1.21`

Signed-off-by: Michal Biesek <michalbiesek@gmail.com>

* add go1.21 guard to fuzz_test.go

* ci: only build last two go versions

* fix workflow yaml syntax error

---------

Signed-off-by: Michal Biesek <michalbiesek@gmail.com>
Co-authored-by: Thomas Pelletier <thomas@pelletier.codes>
2023-08-20 19:47:04 +02:00
David Barroso e183db7e69 Decode: assigned empty struct to empty defined sections (#879)
Co-authored-by: Thomas Pelletier <thomas@pelletier.codes>
2023-07-12 10:53:17 -04:00
dependabot[bot] 60e4af8cca build(deps): bump github.com/stretchr/testify from 1.8.3 to 1.8.4 (#877)
Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.8.3 to 1.8.4.
- [Release notes](https://github.com/stretchr/testify/releases)
- [Commits](https://github.com/stretchr/testify/compare/v1.8.3...v1.8.4)

---
updated-dependencies:
- dependency-name: github.com/stretchr/testify
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-05-30 09:12:07 -04:00
MrJetBOX 8bb1e08dc7 Encode: fix support for arrays (#876) 2023-05-29 09:41:33 -04:00
Thomas Pelletier 7b980e792b Use PtrTo to not require Go 1.18 (#874) 2023-05-23 18:22:22 -04:00
dependabot[bot] 44c1513ccd build(deps): bump github.com/stretchr/testify from 1.8.2 to 1.8.3 (#871)
Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.8.2 to 1.8.3.
- [Release notes](https://github.com/stretchr/testify/releases)
- [Commits](https://github.com/stretchr/testify/compare/v1.8.2...v1.8.3)

---
updated-dependencies:
- dependency-name: github.com/stretchr/testify
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-05-19 10:46:40 -04:00
Thomas Pelletier fcf9d37d0c Comments support in unstable/Parser (#860) 2023-05-19 10:44:02 -04:00
Thomas Pelletier 986afffb7c Decode: fix decode into unsettable structs (#868)
Fixes #866
2023-05-16 09:29:50 -04:00
Thomas Pelletier 8c2c9cc986 Add example on how to use TextUnmarshaler (#867)
Ref #865
2023-05-16 08:17:21 -04:00
manunio 55ca4e35e4 fuzz: improve target perf (#864)
This attempts to improve issue types like timeout, slow_unit
and speed, as seen in the following oss-fuzz performance report:
https://oss-fuzz.com/performance-report/libFuzzer_go-toml_fuzz_toml/libfuzzer_asan_go-toml/2023-05-11
2023-05-12 16:21:17 +02:00
Gordon d34104d493 Support text Un/Marshaller for map keys (#863) 2023-05-09 17:56:57 +02:00
manunio 2aa08368fa fuzz: move fuzz_target from oss-fuzz (#861) 2023-04-28 21:38:13 -04:00
dependabot[bot] 654811fbc3 build(deps): bump actions/setup-go from 3 to 4 (#859)
Bumps [actions/setup-go](https://github.com/actions/setup-go) from 3 to 4.
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](https://github.com/actions/setup-go/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/setup-go
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-21 13:35:40 -04:00
dependabot[bot] 5c05d4d863 build(deps): bump github.com/stretchr/testify from 1.8.1 to 1.8.2 (#852)
Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.8.1 to 1.8.2.
- [Release notes](https://github.com/stretchr/testify/releases)
- [Commits](https://github.com/stretchr/testify/compare/v1.8.1...v1.8.2)

---
updated-dependencies:
- dependency-name: github.com/stretchr/testify
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-20 13:57:54 -04:00
Thomas Pelletier 643c251c4b Fix panic when unmarshaling into a map twice (#854)
Fixes #851
2023-02-28 17:34:24 +01:00
Thomas Pelletier 8a416daa69 Fix error report of type mismatch on inline tables (#853)
Parser did not track the location of the faulty inline table in the
document, and unmarshaler tried to the use the non-raw data field of the
AST node, both resulting into a panic when generating the parser error.

Fixes #850
2023-02-28 17:06:49 +01:00
Andreas Deininger fcd9179b7d Fixes typos (#849) 2023-02-13 03:57:48 -08:00
Marty 9f5726004e Allow integers to be unmarshaled into floats (#841)
Co-authored-by: Marty <martin@windscribe.com>
2023-02-09 12:02:25 -05:00
Thomas Pelletier c4a2eef8a4 Fix go 1.20 version in github actions (#848)
YAML interprets 1.20 as 1.2 without explicitely being a string.
2023-02-09 12:00:14 -05:00
Thomas Pelletier b58c20aa49 Upgrade go 1.20 (#847)
Fixes #842
2023-02-08 18:59:58 -05:00
Cuong Manh Le 090cccf4ba Fix inline table first key value whitespace (#837)
Co-authored-by: Cuong Manh Le <cuong@windscribe.com>
2023-02-01 12:00:09 +01:00
DavidKorczynski 58a592bbf8 ci: add CIFuzz integration (#831)
Signed-off-by: David Korczynski <david@adalogics.com>
2022-11-21 18:51:48 -05:00
39 changed files with 3368 additions and 680 deletions
+26
View File
@@ -0,0 +1,26 @@
name: CIFuzz
on: [pull_request]
jobs:
Fuzzing:
runs-on: ubuntu-latest
steps:
- name: Build Fuzzers
id: build
uses: google/oss-fuzz/infra/cifuzz/actions/build_fuzzers@master
with:
oss-fuzz-project-name: 'go-toml'
dry-run: false
language: go
- name: Run Fuzzers
uses: google/oss-fuzz/infra/cifuzz/actions/run_fuzzers@master
with:
oss-fuzz-project-name: 'go-toml'
fuzz-seconds: 300
dry-run: false
language: go
- name: Upload Crash
uses: actions/upload-artifact@v4
if: failure() && steps.build.outcome == 'success'
with:
name: artifacts
path: ./out/artifacts
+4 -4
View File
@@ -35,11 +35,11 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@v3
uses: actions/checkout@v4
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v2
uses: github/codeql-action/init@v3
with:
languages: ${{ matrix.language }}
# If you wish to specify custom queries, you can do so here or in a config file.
@@ -50,7 +50,7 @@ jobs:
# Autobuild attempts to build any compiled languages (C/C++, C#, or Java).
# If this step fails, then you should remove it and run the build manually (see below)
- name: Autobuild
uses: github/codeql-action/autobuild@v2
uses: github/codeql-action/autobuild@v3
# ️ Command-line programs to run using the OS shell.
# 📚 https://git.io/JvXDl
@@ -64,4 +64,4 @@ jobs:
# make release
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v2
uses: github/codeql-action/analyze@v3
+3 -3
View File
@@ -9,12 +9,12 @@ jobs:
runs-on: "ubuntu-latest"
name: report
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Setup go
uses: actions/setup-go@v3
uses: actions/setup-go@v5
with:
go-version: 1.19
go-version: "1.22"
- name: Run tests with coverage
run: ./ci.sh coverage -d "${GITHUB_BASE_REF-HEAD}"
+4 -4
View File
@@ -16,15 +16,15 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Set up Go
uses: actions/setup-go@v3
uses: actions/setup-go@v5
with:
go-version: 1.19
go-version: "1.22"
- name: Login to GitHub Container Registry
uses: docker/login-action@v2
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
+4 -4
View File
@@ -11,16 +11,16 @@ jobs:
build:
strategy:
matrix:
os: [ 'ubuntu-latest', 'windows-latest', 'macos-latest']
go: [ '1.18', '1.19' ]
os: [ 'ubuntu-latest', 'windows-latest', 'macos-latest', 'macos-14' ]
go: [ '1.21', '1.22' ]
runs-on: ${{ matrix.os }}
name: ${{ matrix.go }}/${{ matrix.os }}
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Setup go ${{ matrix.go }}
uses: actions/setup-go@v3
uses: actions/setup-go@v5
with:
go-version: ${{ matrix.go }}
- name: Run unit tests
+2 -1
View File
@@ -3,4 +3,5 @@ fuzz/
cmd/tomll/tomll
cmd/tomljson/tomljson
cmd/tomltestgen/tomltestgen
dist
dist
tests/
+3
View File
@@ -18,6 +18,7 @@ builds:
- linux_amd64
- linux_arm64
- linux_arm
- linux_riscv64
- windows_amd64
- windows_arm64
- windows_arm
@@ -37,6 +38,7 @@ builds:
- linux_amd64
- linux_arm64
- linux_arm
- linux_riscv64
- windows_amd64
- windows_arm64
- windows_arm
@@ -55,6 +57,7 @@ builds:
targets:
- linux_amd64
- linux_arm64
- linux_riscv64
- linux_arm
- windows_amd64
- windows_arm64
+14 -17
View File
@@ -165,25 +165,22 @@ Checklist:
### New release
1. Decide on the next version number. Use semver.
2. Generate release notes using [`gh`][gh]. Example:
1. Decide on the next version number. Use semver. Review commits since last
version to assess.
2. Tag release. For example:
```
$ gh api -X POST \
-F tag_name='v2.0.0-beta.5' \
-F target_commitish='v2' \
-F previous_tag_name='v2.0.0-beta.4' \
--jq '.body' \
repos/pelletier/go-toml/releases/generate-notes
git checkout v2
git pull
git tag v2.2.0
git push --tags
```
3. Look for "Other changes". That would indicate a pull request not labeled
properly. Tweak labels and pull request titles until changelog looks good for
users.
4. [Draft new release][new-release].
5. Fill tag and target with the same value used to generate the changelog.
6. Set title to the new tag value.
7. Paste the generated changelog.
8. Check "create discussion", in the "Releases" category.
9. Check pre-release if new version is an alpha or beta.
3. CI automatically builds a draft Github release. Review it and edit as
necessary. Look for "Other changes". That would indicate a pull request not
labeled properly. Tweak labels and pull request titles until changelog looks
good for users.
4. Check "create discussion" box, in the "Releases" category.
5. If new version is an alpha or beta only, check pre-release box.
[issues-tracker]: https://github.com/pelletier/go-toml/issues
[bug-report]: https://github.com/pelletier/go-toml/issues/new?template=bug_report.md
+2 -1
View File
@@ -1,6 +1,7 @@
The MIT License (MIT)
Copyright (c) 2013 - 2022 Thomas Pelletier, Eric Anderton
go-toml v2
Copyright (c) 2021 - 2023 Thomas Pelletier
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
+87 -74
View File
@@ -45,16 +45,15 @@ to check for typos. [See example in the documentation][strict].
### Contextualized errors
When most decoding errors occur, go-toml returns [`DecodeError`][decode-err]),
When most decoding errors occur, go-toml returns [`DecodeError`][decode-err],
which contains a human readable contextualized version of the error. For
example:
```
2| key1 = "value1"
3| key2 = "missing2"
| ~~~~ missing field
4| key3 = "missing3"
5| key4 = "value4"
1| [server]
2| path = 100
| ~~~ cannot decode TOML integer into struct field toml_test.Server.Path of type string
3| port = 50
```
[decode-err]: https://pkg.go.dev/github.com/pelletier/go-toml/v2#DecodeError
@@ -73,15 +72,35 @@ representation.
[tlt]: https://pkg.go.dev/github.com/pelletier/go-toml/v2#LocalTime
[tldt]: https://pkg.go.dev/github.com/pelletier/go-toml/v2#LocalDateTime
### Commented config
Since TOML is often used for configuration files, go-toml can emit documents
annotated with [comments and commented-out values][comments-example]. For
example, it can generate the following file:
```toml
# Host IP to connect to.
host = '127.0.0.1'
# Port of the remote server.
port = 4242
# Encryption parameters (optional)
# [TLS]
# cipher = 'AEAD-AES128-GCM-SHA256'
# version = 'TLS 1.3'
```
[comments-example]: https://pkg.go.dev/github.com/pelletier/go-toml/v2#example-Marshal-Commented
## Getting started
Given the following struct, let's see how to read it and write it as TOML:
```go
type MyConfig struct {
Version int
Name string
Tags []string
Version int
Name string
Tags []string
}
```
@@ -100,7 +119,7 @@ tags = ["go", "toml"]
var cfg MyConfig
err := toml.Unmarshal([]byte(doc), &cfg)
if err != nil {
panic(err)
panic(err)
}
fmt.Println("version:", cfg.Version)
fmt.Println("name:", cfg.Name)
@@ -121,14 +140,14 @@ as a TOML document:
```go
cfg := MyConfig{
Version: 2,
Name: "go-toml",
Tags: []string{"go", "toml"},
Version: 2,
Name: "go-toml",
Tags: []string{"go", "toml"},
}
b, err := toml.Marshal(cfg)
if err != nil {
panic(err)
panic(err)
}
fmt.Println(string(b))
@@ -156,17 +175,17 @@ the AST level. See https://pkg.go.dev/github.com/pelletier/go-toml/v2/unstable.
Execution time speedup compared to other Go TOML libraries:
<table>
<thead>
<tr><th>Benchmark</th><th>go-toml v1</th><th>BurntSushi/toml</th></tr>
</thead>
<tbody>
<tr><td>Marshal/HugoFrontMatter-2</td><td>1.9x</td><td>1.9x</td></tr>
<tr><td>Marshal/ReferenceFile/map-2</td><td>1.7x</td><td>1.8x</td></tr>
<tr><td>Marshal/ReferenceFile/struct-2</td><td>2.2x</td><td>2.5x</td></tr>
<tr><td>Unmarshal/HugoFrontMatter-2</td><td>2.9x</td><td>2.9x</td></tr>
<tr><td>Unmarshal/ReferenceFile/map-2</td><td>2.6x</td><td>2.9x</td></tr>
<tr><td>Unmarshal/ReferenceFile/struct-2</td><td>4.4x</td><td>5.3x</td></tr>
</tbody>
<thead>
<tr><th>Benchmark</th><th>go-toml v1</th><th>BurntSushi/toml</th></tr>
</thead>
<tbody>
<tr><td>Marshal/HugoFrontMatter-2</td><td>1.9x</td><td>2.2x</td></tr>
<tr><td>Marshal/ReferenceFile/map-2</td><td>1.7x</td><td>2.1x</td></tr>
<tr><td>Marshal/ReferenceFile/struct-2</td><td>2.2x</td><td>3.0x</td></tr>
<tr><td>Unmarshal/HugoFrontMatter-2</td><td>2.9x</td><td>2.7x</td></tr>
<tr><td>Unmarshal/ReferenceFile/map-2</td><td>2.6x</td><td>2.7x</td></tr>
<tr><td>Unmarshal/ReferenceFile/struct-2</td><td>4.6x</td><td>5.1x</td></tr>
</tbody>
</table>
<details><summary>See more</summary>
<p>The table above has the results of the most common use-cases. The table below
@@ -174,22 +193,22 @@ contains the results of all benchmarks, including unrealistic ones. It is
provided for completeness.</p>
<table>
<thead>
<tr><th>Benchmark</th><th>go-toml v1</th><th>BurntSushi/toml</th></tr>
</thead>
<tbody>
<tr><td>Marshal/SimpleDocument/map-2</td><td>1.8x</td><td>2.9x</td></tr>
<tr><td>Marshal/SimpleDocument/struct-2</td><td>2.7x</td><td>4.2x</td></tr>
<tr><td>Unmarshal/SimpleDocument/map-2</td><td>4.5x</td><td>3.1x</td></tr>
<tr><td>Unmarshal/SimpleDocument/struct-2</td><td>6.2x</td><td>3.9x</td></tr>
<tr><td>UnmarshalDataset/example-2</td><td>3.1x</td><td>3.5x</td></tr>
<tr><td>UnmarshalDataset/code-2</td><td>2.3x</td><td>3.1x</td></tr>
<tr><td>UnmarshalDataset/twitter-2</td><td>2.5x</td><td>2.6x</td></tr>
<tr><td>UnmarshalDataset/citm_catalog-2</td><td>2.1x</td><td>2.2x</td></tr>
<tr><td>UnmarshalDataset/canada-2</td><td>1.6x</td><td>1.3x</td></tr>
<tr><td>UnmarshalDataset/config-2</td><td>4.3x</td><td>3.2x</td></tr>
<tr><td>[Geo mean]</td><td>2.7x</td><td>2.8x</td></tr>
</tbody>
<thead>
<tr><th>Benchmark</th><th>go-toml v1</th><th>BurntSushi/toml</th></tr>
</thead>
<tbody>
<tr><td>Marshal/SimpleDocument/map-2</td><td>1.8x</td><td>2.7x</td></tr>
<tr><td>Marshal/SimpleDocument/struct-2</td><td>2.7x</td><td>3.8x</td></tr>
<tr><td>Unmarshal/SimpleDocument/map-2</td><td>3.8x</td><td>3.0x</td></tr>
<tr><td>Unmarshal/SimpleDocument/struct-2</td><td>5.6x</td><td>4.1x</td></tr>
<tr><td>UnmarshalDataset/example-2</td><td>3.0x</td><td>3.2x</td></tr>
<tr><td>UnmarshalDataset/code-2</td><td>2.3x</td><td>2.9x</td></tr>
<tr><td>UnmarshalDataset/twitter-2</td><td>2.6x</td><td>2.7x</td></tr>
<tr><td>UnmarshalDataset/citm_catalog-2</td><td>2.2x</td><td>2.3x</td></tr>
<tr><td>UnmarshalDataset/canada-2</td><td>1.8x</td><td>1.5x</td></tr>
<tr><td>UnmarshalDataset/config-2</td><td>4.1x</td><td>2.9x</td></tr>
<tr><td>geomean</td><td>2.7x</td><td>2.8x</td></tr>
</tbody>
</table>
<p>This table can be generated with <code>./ci.sh benchmark -a -html</code>.</p>
</details>
@@ -214,24 +233,24 @@ Go-toml provides three handy command line tools:
* `tomljson`: Reads a TOML file and outputs its JSON representation.
```
$ go install github.com/pelletier/go-toml/v2/cmd/tomljson@latest
$ tomljson --help
```
```
$ go install github.com/pelletier/go-toml/v2/cmd/tomljson@latest
$ tomljson --help
```
* `jsontoml`: Reads a JSON file and outputs a TOML representation.
```
$ go install github.com/pelletier/go-toml/v2/cmd/jsontoml@latest
$ jsontoml --help
```
```
$ go install github.com/pelletier/go-toml/v2/cmd/jsontoml@latest
$ jsontoml --help
```
* `tomll`: Lints and reformats a TOML file.
```
$ go install github.com/pelletier/go-toml/v2/cmd/tomll@latest
$ tomll --help
```
```
$ go install github.com/pelletier/go-toml/v2/cmd/tomll@latest
$ tomll --help
```
### Docker image
@@ -242,7 +261,7 @@ Those tools are also available as a [Docker image][docker]. For example, to use
docker run -i ghcr.io/pelletier/go-toml:v2 tomljson < example.toml
```
Multiple versions are availble on [ghcr.io][docker].
Multiple versions are available on [ghcr.io][docker].
[docker]: https://github.com/pelletier/go-toml/pkgs/container/go-toml
@@ -274,16 +293,16 @@ element in the interface to decode the object. For example:
```go
type inner struct {
B interface{}
B interface{}
}
type doc struct {
A interface{}
A interface{}
}
d := doc{
A: inner{
B: "Before",
},
A: inner{
B: "Before",
},
}
data := `
@@ -322,7 +341,7 @@ contained in the doc is superior to the capacity of the array. For example:
```go
type doc struct {
A [2]string
A [2]string
}
d := doc{}
err := toml.Unmarshal([]byte(`A = ["one", "two", "many"]`), &d)
@@ -497,27 +516,20 @@ is not necessary anymore.
V1 used to provide multiple struct tags: `comment`, `commented`, `multiline`,
`toml`, and `omitempty`. To behave more like the standard library, v2 has merged
`toml`, `multiline`, and `omitempty`. For example:
`toml`, `multiline`, `commented`, and `omitempty`. For example:
```go
type doc struct {
// v1
F string `toml:"field" multiline:"true" omitempty:"true"`
F string `toml:"field" multiline:"true" omitempty:"true" commented:"true"`
// v2
F string `toml:"field,multiline,omitempty"`
F string `toml:"field,multiline,omitempty,commented"`
}
```
Has a result, the `Encoder.SetTag*` methods have been removed, as there is just
one tag now.
#### `commented` tag has been removed
There is no replacement for the `commented` tag. This feature would be better
suited in a proper document model for go-toml v2, which has been [cut from
scope][nodoc] at the moment.
#### `Encoder.ArraysWithOneElementPerLine` has been renamed
The new name is `Encoder.SetArraysMultiline`. The behavior should be the same.
@@ -553,10 +565,11 @@ complete solutions exist out there.
## Versioning
Go-toml follows [Semantic Versioning](http://semver.org/). The supported version
of [TOML](https://github.com/toml-lang/toml) is indicated at the beginning of
this document. The last two major versions of Go are supported
(see [Go Release Policy](https://golang.org/doc/devel/release.html#policy)).
Expect for parts explicitely marked otherwise, go-toml follows [Semantic
Versioning](https://semver.org). The supported version of
[TOML](https://github.com/toml-lang/toml) is indicated at the beginning of this
document. The last two major versions of Go are supported (see [Go Release
Policy](https://golang.org/doc/devel/release.html#policy)).
## License
-3
View File
@@ -2,9 +2,6 @@
## Supported Versions
Use this section to tell people about which versions of your project are
currently being supported with security updates.
| Version | Supported |
| ---------- | ------------------ |
| Latest 2.x | :white_check_mark: |
+14 -9
View File
@@ -77,8 +77,9 @@ cover() {
pushd "$dir"
go test -covermode=atomic -coverpkg=./... -coverprofile=coverage.out.tmp ./...
cat coverage.out.tmp | grep -v testsuite | grep -v tomltestgen | grep -v gotoml-test-decoder > coverage.out
grep -Ev '(fuzz|testsuite|tomltestgen|gotoml-test-decoder|gotoml-test-encoder)' coverage.out.tmp > coverage.out
go tool cover -func=coverage.out
echo "Coverage profile for ${branch}: ${dir}/coverage.out" >&2
popd
if [ "${branch}" != "HEAD" ]; then
@@ -151,7 +152,7 @@ bench() {
fi
export GOMAXPROCS=2
nice -n -19 taskset --cpu-list 0,1 go test '-bench=^Benchmark(Un)?[mM]arshal' -count=5 -run=Nothing ./... | tee "${out}"
go test '-bench=^Benchmark(Un)?[mM]arshal' -count=10 -run=Nothing ./... | tee "${out}"
popd
if [ "${branch}" != "HEAD" ]; then
@@ -160,10 +161,12 @@ bench() {
}
fmktemp() {
if mktemp --version|grep GNU >/dev/null; then
mktemp --suffix=-$1;
if mktemp --version &> /dev/null; then
# GNU
mktemp --suffix=-$1
else
mktemp -t $1;
# BSD
mktemp -t $1
fi
}
@@ -183,12 +186,14 @@ with open(sys.argv[1]) as f:
lines.append(line.split(','))
results = []
for line in reversed(lines[1:]):
for line in reversed(lines[2:]):
if len(line) < 8 or line[0] == "":
continue
v2 = float(line[1])
results.append([
line[0].replace("-32", ""),
"%.1fx" % (float(line[3])/v2), # v1
"%.1fx" % (float(line[5])/v2), # bs
"%.1fx" % (float(line[7])/v2), # bs
])
# move geomean to the end
results.append(results[0])
@@ -259,10 +264,10 @@ benchmark() {
if [ "$1" = "-html" ]; then
tmpcsv=`fmktemp csv`
benchstat -csv -geomean go-toml-v2.txt go-toml-v1.txt bs-toml.txt > $tmpcsv
benchstat -format csv go-toml-v2.txt go-toml-v1.txt bs-toml.txt > $tmpcsv
benchstathtml $tmpcsv
else
benchstat -geomean go-toml-v2.txt go-toml-v1.txt bs-toml.txt
benchstat go-toml-v2.txt go-toml-v1.txt bs-toml.txt
fi
rm -f go-toml-v2.txt go-toml-v1.txt bs-toml.txt
@@ -0,0 +1,30 @@
package main
import (
"flag"
"log"
"os"
"path"
"github.com/pelletier/go-toml/v2/internal/testsuite"
)
func main() {
log.SetFlags(0)
flag.Usage = usage
flag.Parse()
if flag.NArg() != 0 {
flag.Usage()
}
err := testsuite.EncodeStdin()
if err != nil {
log.Fatal(err)
}
}
func usage() {
log.Printf("Usage: %s < toml-file\n", path.Base(os.Args[0]))
flag.PrintDefaults()
os.Exit(1)
}
+12 -1
View File
@@ -19,6 +19,7 @@ package main
import (
"encoding/json"
"flag"
"io"
"github.com/pelletier/go-toml/v2"
@@ -33,7 +34,11 @@ Reading from a file:
jsontoml file.json > file.toml
`
var useJsonNumber bool
func main() {
flag.BoolVar(&useJsonNumber, "use-json-number", false, "unmarshal numbers into `json.Number` type instead of as `float64`")
p := cli.Program{
Usage: usage,
Fn: convert,
@@ -45,11 +50,17 @@ func convert(r io.Reader, w io.Writer) error {
var v interface{}
d := json.NewDecoder(r)
e := toml.NewEncoder(w)
if useJsonNumber {
d.UseNumber()
e.SetMarshalJsonNumbers(true)
}
err := d.Decode(&v)
if err != nil {
return err
}
e := toml.NewEncoder(w)
return e.Encode(v)
}
+19 -4
View File
@@ -11,10 +11,11 @@ import (
func TestConvert(t *testing.T) {
examples := []struct {
name string
input string
expected string
errors bool
name string
input string
expected string
errors bool
useJsonNumber bool
}{
{
name: "valid json",
@@ -26,6 +27,19 @@ func TestConvert(t *testing.T) {
}`,
expected: `[mytoml]
a = 42.0
`,
},
{
name: "use json number",
useJsonNumber: true,
input: `
{
"mytoml": {
"a": 42
}
}`,
expected: `[mytoml]
a = 42
`,
},
{
@@ -37,6 +51,7 @@ a = 42.0
for _, e := range examples {
b := new(bytes.Buffer)
useJsonNumber = e.useJsonNumber
err := convert(strings.NewReader(e.input), b)
if e.errors {
require.Error(t, err)
+43 -88
View File
@@ -7,17 +7,13 @@
package main
import (
"archive/zip"
"bytes"
"flag"
"fmt"
"go/format"
"io"
"io/ioutil"
"log"
"net/http"
"os"
"regexp"
"path/filepath"
"strconv"
"strings"
"text/template"
@@ -64,30 +60,6 @@ const srcTemplate = "// Generated by tomltestgen for toml-test ref {{.Ref}} on {
"}\n" +
"{{end}}\n"
func downloadTmpFile(url string) string {
log.Println("starting to download file from", url)
resp, err := http.Get(url)
if err != nil {
panic(err)
}
defer resp.Body.Close()
tmpfile, err := ioutil.TempFile("", "toml-test-*.zip")
if err != nil {
panic(err)
}
defer tmpfile.Close()
copiedLen, err := io.Copy(tmpfile, resp.Body)
if err != nil {
panic(err)
}
if resp.ContentLength > 0 && copiedLen != resp.ContentLength {
panic(fmt.Errorf("copied %d bytes, request body had %d", copiedLen, resp.ContentLength))
}
return tmpfile.Name()
}
func kebabToCamel(kebab string) string {
camel := ""
nextUpper := true
@@ -107,19 +79,6 @@ func kebabToCamel(kebab string) string {
return camel
}
func readFileFromZip(f *zip.File) string {
reader, err := f.Open()
if err != nil {
panic(err)
}
defer reader.Close()
bytes, err := ioutil.ReadAll(reader)
if err != nil {
panic(err)
}
return string(bytes)
}
func templateGoStr(input string) string {
return strconv.Quote(input)
}
@@ -138,61 +97,57 @@ func main() {
flag.Usage = usage
flag.Parse()
url := "https://codeload.github.com/BurntSushi/toml-test/zip/" + *ref
resultFile := downloadTmpFile(url)
defer os.Remove(resultFile)
log.Println("file written to", resultFile)
zipReader, err := zip.OpenReader(resultFile)
if err != nil {
panic(err)
}
defer zipReader.Close()
collection := testsCollection{
Ref: *ref,
Timestamp: time.Now().Format(time.RFC3339),
}
zipFilesMap := map[string]*zip.File{}
dirContent, _ := filepath.Glob("tests/invalid/**/*.toml")
for _, f := range dirContent {
filename := strings.TrimPrefix(f, "tests/valid/")
name := kebabToCamel(strings.TrimSuffix(filename, ".toml"))
for _, f := range zipReader.File {
zipFilesMap[f.Name] = f
log.Printf("> [%s] %s\n", "invalid", name)
tomlContent, err := os.ReadFile(f)
if err != nil {
fmt.Printf("failed to read test file: %s\n", err)
os.Exit(1)
}
collection.Invalid = append(collection.Invalid, invalid{
Name: name,
Input: string(tomlContent),
})
collection.Count++
}
testFileRegexp := regexp.MustCompile(`([^/]+/tests/(valid|invalid)/(.+))\.(toml)`)
for _, f := range zipReader.File {
groups := testFileRegexp.FindStringSubmatch(f.Name)
if len(groups) > 0 {
name := kebabToCamel(groups[3])
testType := groups[2]
dirContent, _ = filepath.Glob("tests/valid/**/*.toml")
for _, f := range dirContent {
filename := strings.TrimPrefix(f, "tests/valid/")
name := kebabToCamel(strings.TrimSuffix(filename, ".toml"))
log.Printf("> [%s] %s\n", testType, name)
log.Printf("> [%s] %s\n", "valid", name)
tomlContent := readFileFromZip(f)
switch testType {
case "invalid":
collection.Invalid = append(collection.Invalid, invalid{
Name: name,
Input: tomlContent,
})
collection.Count++
case "valid":
baseFilePath := groups[1]
jsonFilePath := baseFilePath + ".json"
jsonContent := readFileFromZip(zipFilesMap[jsonFilePath])
collection.Valid = append(collection.Valid, valid{
Name: name,
Input: tomlContent,
JsonRef: jsonContent,
})
collection.Count++
default:
panic(fmt.Sprintf("unknown test type: %s", testType))
}
tomlContent, err := os.ReadFile(f)
if err != nil {
fmt.Printf("failed reading test file: %s\n", err)
os.Exit(1)
}
filename = strings.TrimSuffix(f, ".toml")
jsonContent, err := os.ReadFile(filename + ".json")
if err != nil {
fmt.Printf("failed reading validation json: %s\n", err)
os.Exit(1)
}
collection.Valid = append(collection.Valid, valid{
Name: name,
Input: string(tomlContent),
JsonRef: string(jsonContent),
})
collection.Count++
}
log.Printf("Collected %d tests from toml-test\n", collection.Count)
@@ -202,7 +157,7 @@ func main() {
}
t := template.Must(template.New("src").Funcs(funcMap).Parse(srcTemplate))
buf := new(bytes.Buffer)
err = t.Execute(buf, collection)
err := t.Execute(buf, collection)
if err != nil {
panic(err)
}
@@ -216,7 +171,7 @@ func main() {
return
}
err = os.WriteFile(*out, outputBytes, 0644)
err = os.WriteFile(*out, outputBytes, 0o644)
if err != nil {
panic(err)
}
+1 -1
View File
@@ -318,7 +318,7 @@ func parseFloat(b []byte) (float64, error) {
if cleaned[0] == '+' || cleaned[0] == '-' {
start = 1
}
if cleaned[start] == '0' && isDigit(cleaned[start+1]) {
if cleaned[start] == '0' && len(cleaned) > start+1 && isDigit(cleaned[start+1]) {
return 0, unstable.NewParserError(b, "float integer part cannot have leading zeroes")
}
+37
View File
@@ -0,0 +1,37 @@
package toml_test
import (
"fmt"
"log"
"strconv"
"github.com/pelletier/go-toml/v2"
)
type customInt int
func (i *customInt) UnmarshalText(b []byte) error {
x, err := strconv.ParseInt(string(b), 10, 32)
if err != nil {
return err
}
*i = customInt(x * 100)
return nil
}
type doc struct {
Value customInt
}
func ExampleUnmarshal_textUnmarshal() {
var x doc
data := []byte(`value = "42"`)
err := toml.Unmarshal(data, &x)
if err != nil {
log.Fatal(err)
}
fmt.Println(x)
// Output:
// {4200}
}
+8 -1
View File
@@ -7,13 +7,20 @@ import (
"github.com/stretchr/testify/require"
)
func TestFastSimple(t *testing.T) {
func TestFastSimpleInt(t *testing.T) {
m := map[string]int64{}
err := toml.Unmarshal([]byte(`a = 42`), &m)
require.NoError(t, err)
require.Equal(t, map[string]int64{"a": 42}, m)
}
func TestFastSimpleFloat(t *testing.T) {
m := map[string]float64{}
err := toml.Unmarshal([]byte("a = 42\nb = 1.1\nc = 12341234123412341234123412341234"), &m)
require.NoError(t, err)
require.Equal(t, map[string]float64{"a": 42, "b": 1.1, "c": 1.2341234123412342e+31}, m)
}
func TestFastSimpleString(t *testing.T) {
m := map[string]string{}
err := toml.Unmarshal([]byte(`a = "hello"`), &m)
+2 -2
View File
@@ -1,5 +1,5 @@
//go:build go1.18 || go1.19
// +build go1.18 go1.19
//go:build go1.18 || go1.19 || go1.20 || go1.21 || go1.22
// +build go1.18 go1.19 go1.20 go1.21 go1.22
package toml_test
+1 -1
View File
@@ -2,4 +2,4 @@ module github.com/pelletier/go-toml/v2
go 1.16
require github.com/stretchr/testify v1.8.1
require github.com/stretchr/testify v1.9.0
+4 -2
View File
@@ -6,10 +6,12 @@ github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZN
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk=
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg=
github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+1 -1
View File
@@ -18,7 +18,7 @@ type Program struct {
Usage string
Fn ConvertFn
// Inplace allows the command to take more than one file as argument and
// perform convertion in place on each provided file.
// perform conversion in place on each provided file.
Inplace bool
}
@@ -1085,10 +1085,6 @@ func TestUnmarshalCheckConversionFloatInt(t *testing.T) {
desc: "int",
input: `I = 1e300`,
},
{
desc: "float",
input: `F = 9223372036854775806`,
},
}
for _, test := range testCases {
+23 -4
View File
@@ -4,6 +4,8 @@ import (
"fmt"
"strconv"
"time"
"github.com/pelletier/go-toml/v2"
)
// Remove JSON tags to a data structure as returned by toml-test.
@@ -76,14 +78,31 @@ func untag(typed map[string]interface{}) (interface{}, error) {
return nil, fmt.Errorf("untag: %w", err)
}
return f, nil
//toml.LocalDate{Year:2020, Month:12, Day:12}
case "datetime":
return parseTime(v, "2006-01-02T15:04:05.999999999Z07:00", false)
return time.Parse("2006-01-02T15:04:05.999999999Z07:00", v)
case "datetime-local":
return parseTime(v, "2006-01-02T15:04:05.999999999", true)
var t toml.LocalDateTime
err := t.UnmarshalText([]byte(v))
if err != nil {
return nil, fmt.Errorf("untag: %w", err)
}
return t, nil
case "date-local":
return parseTime(v, "2006-01-02", true)
var t toml.LocalDate
err := t.UnmarshalText([]byte(v))
if err != nil {
return nil, fmt.Errorf("untag: %w", err)
}
return t, nil
case "time-local":
return parseTime(v, "15:04:05.999999999", true)
var t toml.LocalTime
err := t.UnmarshalText([]byte(v))
if err != nil {
return nil, fmt.Errorf("untag: %w", err)
}
return t, nil
case "bool":
switch v {
case "true":
+18
View File
@@ -48,3 +48,21 @@ func DecodeStdin() error {
return nil
}
// EncodeStdin is a helper function for the toml-test binary interface. Tagged
// JSON is read from STDIN and a resulting TOML representation is written to
// STDOUT.
func EncodeStdin() error {
var j interface{}
err := json.NewDecoder(os.Stdin).Decode(&j)
if err != nil {
return err
}
rm, err := rmTag(j)
if err != nil {
return fmt.Errorf("removing tags: %w", err)
}
return toml.NewEncoder(os.Stdout).Encode(rm)
}
+33 -29
View File
@@ -149,8 +149,9 @@ func (s *SeenTracker) setExplicitFlag(parentIdx int) {
// CheckExpression takes a top-level node and checks that it does not contain
// keys that have been seen in previous calls, and validates that types are
// consistent.
func (s *SeenTracker) CheckExpression(node *unstable.Node) error {
// consistent. It returns true if it is the first time this node's key is seen.
// Useful to clear array tables on first use.
func (s *SeenTracker) CheckExpression(node *unstable.Node) (bool, error) {
if s.entries == nil {
s.reset()
}
@@ -166,7 +167,7 @@ func (s *SeenTracker) CheckExpression(node *unstable.Node) error {
}
}
func (s *SeenTracker) checkTable(node *unstable.Node) error {
func (s *SeenTracker) checkTable(node *unstable.Node) (bool, error) {
if s.currentIdx >= 0 {
s.setExplicitFlag(s.currentIdx)
}
@@ -192,7 +193,7 @@ func (s *SeenTracker) checkTable(node *unstable.Node) error {
} else {
entry := s.entries[idx]
if entry.kind == valueKind {
return fmt.Errorf("toml: expected %s to be a table, not a %s", string(k), entry.kind)
return false, fmt.Errorf("toml: expected %s to be a table, not a %s", string(k), entry.kind)
}
}
parentIdx = idx
@@ -201,25 +202,27 @@ func (s *SeenTracker) checkTable(node *unstable.Node) error {
k := it.Node().Data
idx := s.find(parentIdx, k)
first := false
if idx >= 0 {
kind := s.entries[idx].kind
if kind != tableKind {
return fmt.Errorf("toml: key %s should be a table, not a %s", string(k), kind)
return false, fmt.Errorf("toml: key %s should be a table, not a %s", string(k), kind)
}
if s.entries[idx].explicit {
return fmt.Errorf("toml: table %s already exists", string(k))
return false, fmt.Errorf("toml: table %s already exists", string(k))
}
s.entries[idx].explicit = true
} else {
idx = s.create(parentIdx, k, tableKind, true, false)
first = true
}
s.currentIdx = idx
return nil
return first, nil
}
func (s *SeenTracker) checkArrayTable(node *unstable.Node) error {
func (s *SeenTracker) checkArrayTable(node *unstable.Node) (bool, error) {
if s.currentIdx >= 0 {
s.setExplicitFlag(s.currentIdx)
}
@@ -242,7 +245,7 @@ func (s *SeenTracker) checkArrayTable(node *unstable.Node) error {
} else {
entry := s.entries[idx]
if entry.kind == valueKind {
return fmt.Errorf("toml: expected %s to be a table, not a %s", string(k), entry.kind)
return false, fmt.Errorf("toml: expected %s to be a table, not a %s", string(k), entry.kind)
}
}
@@ -252,22 +255,23 @@ func (s *SeenTracker) checkArrayTable(node *unstable.Node) error {
k := it.Node().Data
idx := s.find(parentIdx, k)
if idx >= 0 {
firstTime := idx < 0
if firstTime {
idx = s.create(parentIdx, k, arrayTableKind, true, false)
} else {
kind := s.entries[idx].kind
if kind != arrayTableKind {
return fmt.Errorf("toml: key %s already exists as a %s, but should be an array table", kind, string(k))
return false, fmt.Errorf("toml: key %s already exists as a %s, but should be an array table", kind, string(k))
}
s.clear(idx)
} else {
idx = s.create(parentIdx, k, arrayTableKind, true, false)
}
s.currentIdx = idx
return nil
return firstTime, nil
}
func (s *SeenTracker) checkKeyValue(node *unstable.Node) error {
func (s *SeenTracker) checkKeyValue(node *unstable.Node) (bool, error) {
parentIdx := s.currentIdx
it := node.Key()
@@ -281,11 +285,11 @@ func (s *SeenTracker) checkKeyValue(node *unstable.Node) error {
} else {
entry := s.entries[idx]
if it.IsLast() {
return fmt.Errorf("toml: key %s is already defined", string(k))
return false, fmt.Errorf("toml: key %s is already defined", string(k))
} else if entry.kind != tableKind {
return fmt.Errorf("toml: expected %s to be a table, not a %s", string(k), entry.kind)
return false, fmt.Errorf("toml: expected %s to be a table, not a %s", string(k), entry.kind)
} else if entry.explicit {
return fmt.Errorf("toml: cannot redefine table %s that has already been explicitly defined", string(k))
return false, fmt.Errorf("toml: cannot redefine table %s that has already been explicitly defined", string(k))
}
}
@@ -303,30 +307,30 @@ func (s *SeenTracker) checkKeyValue(node *unstable.Node) error {
return s.checkArray(value)
}
return nil
return false, nil
}
func (s *SeenTracker) checkArray(node *unstable.Node) error {
func (s *SeenTracker) checkArray(node *unstable.Node) (first bool, err error) {
it := node.Children()
for it.Next() {
n := it.Node()
switch n.Kind {
case unstable.InlineTable:
err := s.checkInlineTable(n)
first, err = s.checkInlineTable(n)
if err != nil {
return err
return false, err
}
case unstable.Array:
err := s.checkArray(n)
first, err = s.checkArray(n)
if err != nil {
return err
return false, err
}
}
}
return nil
return first, nil
}
func (s *SeenTracker) checkInlineTable(node *unstable.Node) error {
func (s *SeenTracker) checkInlineTable(node *unstable.Node) (first bool, err error) {
if pool.New == nil {
pool.New = func() interface{} {
return &SeenTracker{}
@@ -339,9 +343,9 @@ func (s *SeenTracker) checkInlineTable(node *unstable.Node) error {
it := node.Children()
for it.Next() {
n := it.Node()
err := s.checkKeyValue(n)
first, err = s.checkKeyValue(n)
if err != nil {
return err
return false, err
}
}
@@ -352,5 +356,5 @@ func (s *SeenTracker) checkInlineTable(node *unstable.Node) error {
// redefinition of its keys: check* functions cannot walk into
// a value.
pool.Put(s)
return nil
return first, nil
}
+93 -14
View File
@@ -3,6 +3,7 @@ package toml
import (
"bytes"
"encoding"
"encoding/json"
"fmt"
"io"
"math"
@@ -37,10 +38,11 @@ type Encoder struct {
w io.Writer
// global settings
tablesInline bool
arraysMultiline bool
indentSymbol string
indentTables bool
tablesInline bool
arraysMultiline bool
indentSymbol string
indentTables bool
marshalJsonNumbers bool
}
// NewEncoder returns a new Encoder that writes to w.
@@ -87,6 +89,17 @@ func (enc *Encoder) SetIndentTables(indent bool) *Encoder {
return enc
}
// SetMarshalJsonNumbers forces the encoder to serialize `json.Number` as a
// float or integer instead of relying on TextMarshaler to emit a string.
//
// *Unstable:* This method does not follow the compatibility guarantees of
// semver. It can be changed or removed without a new major version being
// issued.
func (enc *Encoder) SetMarshalJsonNumbers(indent bool) *Encoder {
enc.marshalJsonNumbers = indent
return enc
}
// Encode writes a TOML representation of v to the stream.
//
// If v cannot be represented to TOML it returns an error.
@@ -148,6 +161,9 @@ func (enc *Encoder) SetIndentTables(indent bool) *Encoder {
//
// The "omitempty" option prevents empty values or groups from being emitted.
//
// The "commented" option prefixes the value and all its children with a comment
// symbol.
//
// In addition to the "toml" tag struct tag, a "comment" tag can be used to emit
// a TOML comment before the value being annotated. Comments are ignored inside
// inline tables. For array tables, the comment is only present before the first
@@ -180,6 +196,7 @@ func (enc *Encoder) Encode(v interface{}) error {
type valueOptions struct {
multiline bool
omitempty bool
commented bool
comment string
}
@@ -205,6 +222,9 @@ type encoderCtx struct {
// Indentation level
indent int
// Prefix the current value with a comment.
commented bool
// Options coming from struct tags
options valueOptions
}
@@ -245,6 +265,18 @@ func (enc *Encoder) encode(b []byte, ctx encoderCtx, v reflect.Value) ([]byte, e
return append(b, x.String()...), nil
case LocalDateTime:
return append(b, x.String()...), nil
case json.Number:
if enc.marshalJsonNumbers {
if x == "" { /// Useful zero value.
return append(b, "0"...), nil
} else if v, err := x.Int64(); err == nil {
return enc.encode(b, ctx, reflect.ValueOf(v))
} else if f, err := x.Float64(); err == nil {
return enc.encode(b, ctx, reflect.ValueOf(f))
} else {
return nil, fmt.Errorf("toml: unable to convert %q to int64 or float64", x)
}
}
}
hasTextMarshaler := v.Type().Implements(textMarshalerType)
@@ -273,7 +305,7 @@ func (enc *Encoder) encode(b []byte, ctx encoderCtx, v reflect.Value) ([]byte, e
return enc.encodeMap(b, ctx, v)
case reflect.Struct:
return enc.encodeStruct(b, ctx, v)
case reflect.Slice:
case reflect.Slice, reflect.Array:
return enc.encodeSlice(b, ctx, v)
case reflect.Interface:
if v.IsNil() {
@@ -357,9 +389,10 @@ func (enc *Encoder) encodeKv(b []byte, ctx encoderCtx, options valueOptions, v r
if !ctx.inline {
b = enc.encodeComment(ctx.indent, options.comment, b)
b = enc.commented(ctx.commented, b)
b = enc.indent(ctx.indent, b)
}
b = enc.indent(ctx.indent, b)
b = enc.encodeKey(b, ctx.key)
b = append(b, " = "...)
@@ -378,6 +411,13 @@ func (enc *Encoder) encodeKv(b []byte, ctx encoderCtx, options valueOptions, v r
return b, nil
}
func (enc *Encoder) commented(commented bool, b []byte) []byte {
if commented {
return append(b, "# "...)
}
return b
}
func isEmptyValue(v reflect.Value) bool {
switch v.Kind() {
case reflect.Struct:
@@ -526,6 +566,8 @@ func (enc *Encoder) encodeTableHeader(ctx encoderCtx, b []byte) ([]byte, error)
b = enc.encodeComment(ctx.indent, ctx.options.comment, b)
b = enc.commented(ctx.commented, b)
b = enc.indent(ctx.indent, b)
b = append(b, '[')
@@ -577,11 +619,23 @@ func (enc *Encoder) encodeKey(b []byte, k string) []byte {
}
}
func (enc *Encoder) encodeMap(b []byte, ctx encoderCtx, v reflect.Value) ([]byte, error) {
if v.Type().Key().Kind() != reflect.String {
return nil, fmt.Errorf("toml: type %s is not supported as a map key", v.Type().Key().Kind())
}
func (enc *Encoder) keyToString(k reflect.Value) (string, error) {
keyType := k.Type()
switch {
case keyType.Kind() == reflect.String:
return k.String(), nil
case keyType.Implements(textMarshalerType):
keyB, err := k.Interface().(encoding.TextMarshaler).MarshalText()
if err != nil {
return "", fmt.Errorf("toml: error marshalling key %v from text: %w", k, err)
}
return string(keyB), nil
}
return "", fmt.Errorf("toml: type %s is not supported as a map key", keyType.Kind())
}
func (enc *Encoder) encodeMap(b []byte, ctx encoderCtx, v reflect.Value) ([]byte, error) {
var (
t table
emptyValueOptions valueOptions
@@ -589,13 +643,17 @@ func (enc *Encoder) encodeMap(b []byte, ctx encoderCtx, v reflect.Value) ([]byte
iter := v.MapRange()
for iter.Next() {
k := iter.Key().String()
v := iter.Value()
if isNil(v) {
continue
}
k, err := enc.keyToString(iter.Key())
if err != nil {
return nil, err
}
if willConvertToTableOrArrayTable(ctx, v) {
t.pushTable(k, v, emptyValueOptions)
} else {
@@ -674,6 +732,8 @@ func walkStruct(ctx encoderCtx, t *table, v reflect.Value) {
if fieldType.Anonymous {
if fieldType.Type.Kind() == reflect.Struct {
walkStruct(ctx, t, f)
} else if fieldType.Type.Kind() == reflect.Pointer && !f.IsNil() && f.Elem().Kind() == reflect.Struct {
walkStruct(ctx, t, f.Elem())
}
continue
} else {
@@ -688,6 +748,7 @@ func walkStruct(ctx encoderCtx, t *table, v reflect.Value) {
options := valueOptions{
multiline: opts.multiline,
omitempty: opts.omitempty,
commented: opts.commented,
comment: fieldType.Tag.Get("comment"),
}
@@ -747,6 +808,7 @@ type tagOptions struct {
multiline bool
inline bool
omitempty bool
commented bool
}
func parseTag(tag string) (string, tagOptions) {
@@ -774,6 +836,8 @@ func parseTag(tag string) (string, tagOptions) {
opts.inline = true
case "omitempty":
opts.omitempty = true
case "commented":
opts.commented = true
}
}
@@ -809,8 +873,10 @@ func (enc *Encoder) encodeTable(b []byte, ctx encoderCtx, t table) ([]byte, erro
hasNonEmptyKV = true
ctx.setKey(kv.Key)
ctx2 := ctx
ctx2.commented = kv.Options.commented || ctx2.commented
b, err = enc.encodeKv(b, ctx, kv.Options, kv.Value)
b, err = enc.encodeKv(b, ctx2, kv.Options, kv.Value)
if err != nil {
return nil, err
}
@@ -835,8 +901,10 @@ func (enc *Encoder) encodeTable(b []byte, ctx encoderCtx, t table) ([]byte, erro
ctx.setKey(table.Key)
ctx.options = table.Options
ctx2 := ctx
ctx2.commented = ctx2.commented || ctx.options.commented
b, err = enc.encode(b, ctx, table.Value)
b, err = enc.encode(b, ctx2, table.Value)
if err != nil {
return nil, err
}
@@ -914,7 +982,7 @@ func willConvertToTableOrArrayTable(ctx encoderCtx, v reflect.Value) bool {
return willConvertToTableOrArrayTable(ctx, v.Elem())
}
if t.Kind() == reflect.Slice {
if t.Kind() == reflect.Slice || t.Kind() == reflect.Array {
if v.Len() == 0 {
// An empty slice should be a kv = [].
return false
@@ -954,6 +1022,13 @@ func (enc *Encoder) encodeSliceAsArrayTable(b []byte, ctx encoderCtx, v reflect.
ctx.shiftKey()
scratch := make([]byte, 0, 64)
scratch = enc.commented(ctx.commented, scratch)
if enc.indentTables {
scratch = enc.indent(ctx.indent, scratch)
}
scratch = append(scratch, "[["...)
for i, k := range ctx.parentKey {
@@ -969,6 +1044,10 @@ func (enc *Encoder) encodeSliceAsArrayTable(b []byte, ctx encoderCtx, v reflect.
b = enc.encodeComment(ctx.indent, ctx.options.comment, b)
if enc.indentTables {
ctx.indent++
}
for i := 0; i < v.Len(); i++ {
if i != 0 {
b = append(b, "\n"...)
+505 -1
View File
@@ -15,6 +15,21 @@ import (
"github.com/stretchr/testify/require"
)
type marshalTextKey struct {
A string
B string
}
func (k marshalTextKey) MarshalText() ([]byte, error) {
return []byte(k.A + "-" + k.B), nil
}
type marshalBadTextKey struct{}
func (k marshalBadTextKey) MarshalText() ([]byte, error) {
return nil, fmt.Errorf("error")
}
func TestMarshal(t *testing.T) {
someInt := 42
@@ -97,6 +112,53 @@ also = 'that'
a = 'test'
`,
},
{
desc: `map with text key`,
v: map[marshalTextKey]string{
{A: "a", B: "1"}: "value 1",
{A: "a", B: "2"}: "value 2",
{A: "b", B: "1"}: "value 3",
},
expected: `a-1 = 'value 1'
a-2 = 'value 2'
b-1 = 'value 3'
`,
},
{
desc: `table with text key`,
v: map[marshalTextKey]map[string]string{
{A: "a", B: "1"}: {"value": "foo"},
},
expected: `[a-1]
value = 'foo'
`,
},
{
desc: `map with ptr text key`,
v: map[*marshalTextKey]string{
{A: "a", B: "1"}: "value 1",
{A: "a", B: "2"}: "value 2",
{A: "b", B: "1"}: "value 3",
},
expected: `a-1 = 'value 1'
a-2 = 'value 2'
b-1 = 'value 3'
`,
},
{
desc: `map with bad text key`,
v: map[marshalBadTextKey]string{
{}: "value 1",
},
err: true,
},
{
desc: `map with bad ptr text key`,
v: map[*marshalBadTextKey]string{
{}: "value 1",
},
err: true,
},
{
desc: "simple string array",
v: map[string][]string{
@@ -144,6 +206,45 @@ a = 'test'
[[top]]
'map2.1' = 'v2.1'
`,
},
{
desc: "fixed size string array",
v: map[string][3]string{
"array": {"one", "two", "three"},
},
expected: `array = ['one', 'two', 'three']
`,
},
{
desc: "fixed size nested string arrays",
v: map[string][2][2]string{
"array": {{"one", "two"}, {"three"}},
},
expected: `array = [['one', 'two'], ['three', '']]
`,
},
{
desc: "mixed strings and fixed size nested string arrays",
v: map[string][]interface{}{
"array": {"a string", [2]string{"one", "two"}, "last"},
},
expected: `array = ['a string', ['one', 'two'], 'last']
`,
},
{
desc: "fixed size array of maps",
v: map[string][2]map[string]string{
"ftop": {
{"map1.1": "v1.1"},
{"map2.1": "v2.1"},
},
},
expected: `[[ftop]]
'map1.1' = 'v1.1'
[[ftop]]
'map2.1' = 'v2.1'
`,
},
{
@@ -487,9 +588,14 @@ foo = 42
},
{
desc: "invalid map key",
v: map[int]interface{}{},
v: map[int]interface{}{1: "a"},
err: true,
},
{
desc: "invalid map key but empty",
v: map[int]interface{}{},
expected: "",
},
{
desc: "unhandled type",
v: struct {
@@ -842,6 +948,29 @@ func TestEncoderSetIndentSymbol(t *testing.T) {
assert.Equal(t, expected, w.String())
}
func TestEncoderSetMarshalJsonNumbers(t *testing.T) {
var w strings.Builder
enc := toml.NewEncoder(&w)
enc.SetMarshalJsonNumbers(true)
err := enc.Encode(map[string]interface{}{
"A": json.Number("1.1"),
"B": json.Number("42e-3"),
"C": json.Number("42"),
"D": json.Number("0"),
"E": json.Number("0.0"),
"F": json.Number(""),
})
require.NoError(t, err)
expected := `A = 1.1
B = 0.042
C = 42
D = 0
E = 0.0
F = 0
`
assert.Equal(t, expected, w.String())
}
func TestEncoderOmitempty(t *testing.T) {
type doc struct {
String string `toml:",omitempty,multiline"`
@@ -1095,6 +1224,45 @@ randomize = true
require.Equal(t, expected, buf.String())
}
func TestMarhsalIssue888(t *testing.T) {
type Thing struct {
FieldA string `comment:"my field A"`
FieldB string `comment:"my field B"`
}
type Cfg struct {
Custom []Thing `comment:"custom config"`
}
buf := new(bytes.Buffer)
config := Cfg{
Custom: []Thing{
{FieldA: "field a 1", FieldB: "field b 1"},
{FieldA: "field a 2", FieldB: "field b 2"},
},
}
encoder := toml.NewEncoder(buf).SetIndentTables(true)
encoder.Encode(config)
expected := `# custom config
[[Custom]]
# my field A
FieldA = 'field a 1'
# my field B
FieldB = 'field b 1'
[[Custom]]
# my field A
FieldA = 'field a 2'
# my field B
FieldB = 'field b 2'
`
require.Equal(t, expected, buf.String())
}
func TestMarshalNestedAnonymousStructs(t *testing.T) {
type Embedded struct {
Value string `toml:"value" json:"value"`
@@ -1159,6 +1327,38 @@ value = ''
require.Equal(t, expected, string(result))
}
func TestMarshalNestedAnonymousStructs_PointerEmbedded(t *testing.T) {
type Embedded struct {
Value string `toml:"value" json:"value"`
Omitted string `toml:"omitted,omitempty"`
Ptr *string `toml:"ptr"`
}
type Named struct {
Value string `toml:"value" json:"value"`
}
type Doc struct {
*Embedded
*Named `toml:"named" json:"named"`
Anonymous struct {
*Embedded
Value *string `toml:"value" json:"value"`
} `toml:"anonymous,omitempty" json:"anonymous,omitempty"`
}
doc := &Doc{
Embedded: &Embedded{Value: "foo"},
}
expected := `value = 'foo'
`
result, err := toml.Marshal(doc)
require.NoError(t, err)
require.Equal(t, expected, string(result))
}
func TestLocalTime(t *testing.T) {
v := map[string]toml.LocalTime{
"a": {
@@ -1190,6 +1390,152 @@ func TestMarshalUint64Overflow(t *testing.T) {
require.Error(t, err)
}
func TestIndentWithInlineTable(t *testing.T) {
x := map[string][]map[string]string{
"one": {
{"0": "0"},
{"1": "1"},
},
}
expected := `one = [
{0 = '0'},
{1 = '1'}
]
`
var buf bytes.Buffer
enc := toml.NewEncoder(&buf)
enc.SetIndentTables(true)
enc.SetTablesInline(true)
enc.SetArraysMultiline(true)
require.NoError(t, enc.Encode(x))
assert.Equal(t, expected, buf.String())
}
type C3 struct {
Value int `toml:",commented"`
Values []int `toml:",commented"`
}
type C2 struct {
Int int64
String string
ArrayInts []int
Structs []C3
}
type C1 struct {
Int int64 `toml:",commented"`
String string `toml:",commented"`
ArrayInts []int `toml:",commented"`
Structs []C3 `toml:",commented"`
}
type Commented struct {
Int int64 `toml:",commented"`
String string `toml:",commented"`
C1 C1
C2 C2 `toml:",commented"` // same as C1, but commented at top level
}
func TestMarshalCommented(t *testing.T) {
c := Commented{
Int: 42,
String: "root",
C1: C1{
Int: 11,
String: "C1",
ArrayInts: []int{1, 2, 3},
Structs: []C3{
{Value: 100},
{Values: []int{4, 5, 6}},
},
},
C2: C2{
Int: 22,
String: "C2",
ArrayInts: []int{1, 2, 3},
Structs: []C3{
{Value: 100},
{Values: []int{4, 5, 6}},
},
},
}
out, err := toml.Marshal(c)
require.NoError(t, err)
expected := `# Int = 42
# String = 'root'
[C1]
# Int = 11
# String = 'C1'
# ArrayInts = [1, 2, 3]
# [[C1.Structs]]
# Value = 100
# Values = []
# [[C1.Structs]]
# Value = 0
# Values = [4, 5, 6]
# [C2]
# Int = 22
# String = 'C2'
# ArrayInts = [1, 2, 3]
# [[C2.Structs]]
# Value = 100
# Values = []
# [[C2.Structs]]
# Value = 0
# Values = [4, 5, 6]
`
require.Equal(t, expected, string(out))
}
func TestMarshalIndentedCustomTypeArray(t *testing.T) {
c := struct {
Nested struct {
NestedArray []struct {
Value int
}
}
}{
Nested: struct {
NestedArray []struct {
Value int
}
}{
NestedArray: []struct {
Value int
}{
{Value: 1},
{Value: 2},
},
},
}
expected := `[Nested]
[[Nested.NestedArray]]
Value = 1
[[Nested.NestedArray]]
Value = 2
`
var buf bytes.Buffer
enc := toml.NewEncoder(&buf)
enc.SetIndentTables(true)
require.NoError(t, enc.Encode(c))
require.Equal(t, expected, buf.String())
}
func ExampleMarshal() {
type MyConfig struct {
Version int
@@ -1214,3 +1560,161 @@ func ExampleMarshal() {
// Name = 'go-toml'
// Tags = ['go', 'toml']
}
// Example that uses the 'commented' field tag option to generate an example
// configuration file that has commented out sections (example from
// go-graphite/graphite-clickhouse).
func ExampleMarshal_commented() {
type Common struct {
Listen string `toml:"listen" comment:"general listener"`
PprofListen string `toml:"pprof-listen" comment:"listener to serve /debug/pprof requests. '-pprof' argument overrides it"`
MaxMetricsPerTarget int `toml:"max-metrics-per-target" comment:"limit numbers of queried metrics per target in /render requests, 0 or negative = unlimited"`
MemoryReturnInterval time.Duration `toml:"memory-return-interval" comment:"daemon will return the freed memory to the OS when it>0"`
}
type Costs struct {
Cost *int `toml:"cost" comment:"default cost (for wildcarded equalence or matched with regex, or if no value cost set)"`
ValuesCost map[string]int `toml:"values-cost" comment:"cost with some value (for equalence without wildcards) (additional tuning, usually not needed)"`
}
type ClickHouse struct {
URL string `toml:"url" comment:"default url, see https://clickhouse.tech/docs/en/interfaces/http. Can be overwritten with query-params"`
RenderMaxQueries int `toml:"render-max-queries" comment:"Max queries to render queiries"`
RenderConcurrentQueries int `toml:"render-concurrent-queries" comment:"Concurrent queries to render queiries"`
TaggedCosts map[string]*Costs `toml:"tagged-costs,commented"`
TreeTable string `toml:"tree-table,commented"`
ReverseTreeTable string `toml:"reverse-tree-table,commented"`
DateTreeTable string `toml:"date-tree-table,commented"`
DateTreeTableVersion int `toml:"date-tree-table-version,commented"`
TreeTimeout time.Duration `toml:"tree-timeout,commented"`
TagTable string `toml:"tag-table,commented"`
ExtraPrefix string `toml:"extra-prefix" comment:"add extra prefix (directory in graphite) for all metrics, w/o trailing dot"`
ConnectTimeout time.Duration `toml:"connect-timeout" comment:"TCP connection timeout"`
DataTableLegacy string `toml:"data-table,commented"`
RollupConfLegacy string `toml:"rollup-conf,commented"`
MaxDataPoints int `toml:"max-data-points" comment:"max points per metric when internal-aggregation=true"`
InternalAggregation bool `toml:"internal-aggregation" comment:"ClickHouse-side aggregation, see doc/aggregation.md"`
}
type Tags struct {
Rules string `toml:"rules"`
Date string `toml:"date"`
ExtraWhere string `toml:"extra-where"`
InputFile string `toml:"input-file"`
OutputFile string `toml:"output-file"`
}
type Config struct {
Common Common `toml:"common"`
ClickHouse ClickHouse `toml:"clickhouse"`
Tags Tags `toml:"tags,commented"`
}
cfg := &Config{
Common: Common{
Listen: ":9090",
PprofListen: "",
MaxMetricsPerTarget: 15000, // This is arbitrary value to protect CH from overload
MemoryReturnInterval: 0,
},
ClickHouse: ClickHouse{
URL: "http://localhost:8123?cancel_http_readonly_queries_on_client_close=1",
ExtraPrefix: "",
ConnectTimeout: time.Second,
DataTableLegacy: "",
RollupConfLegacy: "auto",
MaxDataPoints: 1048576,
InternalAggregation: true,
},
Tags: Tags{},
}
out, err := toml.Marshal(cfg)
if err != nil {
panic(err)
}
err = toml.Unmarshal(out, &cfg)
if err != nil {
panic(err)
}
fmt.Println(string(out))
// Output:
// [common]
// # general listener
// listen = ':9090'
// # listener to serve /debug/pprof requests. '-pprof' argument overrides it
// pprof-listen = ''
// # limit numbers of queried metrics per target in /render requests, 0 or negative = unlimited
// max-metrics-per-target = 15000
// # daemon will return the freed memory to the OS when it>0
// memory-return-interval = 0
//
// [clickhouse]
// # default url, see https://clickhouse.tech/docs/en/interfaces/http. Can be overwritten with query-params
// url = 'http://localhost:8123?cancel_http_readonly_queries_on_client_close=1'
// # Max queries to render queiries
// render-max-queries = 0
// # Concurrent queries to render queiries
// render-concurrent-queries = 0
// # tree-table = ''
// # reverse-tree-table = ''
// # date-tree-table = ''
// # date-tree-table-version = 0
// # tree-timeout = 0
// # tag-table = ''
// # add extra prefix (directory in graphite) for all metrics, w/o trailing dot
// extra-prefix = ''
// # TCP connection timeout
// connect-timeout = 1000000000
// # data-table = ''
// # rollup-conf = 'auto'
// # max points per metric when internal-aggregation=true
// max-data-points = 1048576
// # ClickHouse-side aggregation, see doc/aggregation.md
// internal-aggregation = true
//
// # [tags]
// # rules = ''
// # date = ''
// # extra-where = ''
// # input-file = ''
// # output-file = ''
}
func TestReadmeComments(t *testing.T) {
type TLS struct {
Cipher string `toml:"cipher"`
Version string `toml:"version"`
}
type Config struct {
Host string `toml:"host" comment:"Host IP to connect to."`
Port int `toml:"port" comment:"Port of the remote server."`
Tls TLS `toml:"TLS,commented" comment:"Encryption parameters (optional)"`
}
example := Config{
Host: "127.0.0.1",
Port: 4242,
Tls: TLS{
Cipher: "AEAD-AES128-GCM-SHA256",
Version: "TLS 1.3",
},
}
out, err := toml.Marshal(example)
require.NoError(t, err)
expected := `# Host IP to connect to.
host = '127.0.0.1'
# Port of the remote server.
port = 4242
# Encryption parameters (optional)
# [TLS]
# cipher = 'AEAD-AES128-GCM-SHA256'
# version = 'TLS 1.3'
`
require.Equal(t, expected, string(out))
}
+45
View File
@@ -0,0 +1,45 @@
//go:build go1.18 || go1.19 || go1.20 || go1.21 || go1.22
// +build go1.18 go1.19 go1.20 go1.21 go1.22
package ossfuzz
import (
"fmt"
"reflect"
"strings"
"github.com/pelletier/go-toml/v2"
)
func FuzzToml(data []byte) int {
if len(data) >= 2048 {
return 0
}
if strings.Contains(string(data), "nan") {
return 0
}
var v interface{}
err := toml.Unmarshal(data, &v)
if err != nil {
return 0
}
encoded, err := toml.Marshal(v)
if err != nil {
panic(fmt.Sprintf("failed to marshal unmarshaled document: %s", err))
}
var v2 interface{}
err = toml.Unmarshal(encoded, &v2)
if err != nil {
panic(fmt.Sprintf("failed round trip: %s", err))
}
if !reflect.DeepEqual(v, v2) {
panic(fmt.Sprintf("not equal: %#+v %#+v", v, v2))
}
return 1
}
+1
View File
@@ -1,3 +1,4 @@
//go:generate go run github.com/toml-lang/toml-test/cmd/toml-test@master -copy ./tests
//go:generate go run ./cmd/tomltestgen/main.go -o toml_testgen_test.go
// This is a support file for toml_testgen_test.go
+1364 -351
View File
File diff suppressed because it is too large Load Diff
+113 -29
View File
@@ -35,6 +35,9 @@ type Decoder struct {
// global settings
strict bool
// toggles unmarshaler interface
unmarshalerInterface bool
}
// NewDecoder creates a new Decoder that will read from r.
@@ -54,13 +57,31 @@ func (d *Decoder) DisallowUnknownFields() *Decoder {
return d
}
// EnableUnmarshalerInterface allows to enable unmarshaler interface.
//
// With this feature enabled, types implementing the unstable/Unmarshaler
// interface can be decoded from any structure of the document. It allows types
// that don't have a straightfoward TOML representation to provide their own
// decoding logic.
//
// Currently, types can only decode from a single value. Tables and array tables
// are not supported.
//
// *Unstable:* This method does not follow the compatibility guarantees of
// semver. It can be changed or removed without a new major version being
// issued.
func (d *Decoder) EnableUnmarshalerInterface() *Decoder {
d.unmarshalerInterface = true
return d
}
// Decode the whole content of r into v.
//
// By default, values in the document that don't exist in the target Go value
// are ignored. See Decoder.DisallowUnknownFields() to change this behavior.
//
// When a TOML local date, time, or date-time is decoded into a time.Time, its
// value is represented in time.Local timezone. Otherwise the approriate Local*
// value is represented in time.Local timezone. Otherwise the appropriate Local*
// structure is used. For time values, precision up to the nanosecond is
// supported by truncating extra digits.
//
@@ -108,6 +129,7 @@ func (d *Decoder) Decode(v interface{}) error {
strict: strict{
Enabled: d.strict,
},
unmarshalerInterface: d.unmarshalerInterface,
}
return dec.FromParser(v)
@@ -127,6 +149,10 @@ type decoder struct {
// need to be skipped.
skipUntilTable bool
// Flag indicating that the current array/slice table should be cleared because
// it is the first encounter of an array table.
clearArrayTable bool
// Tracks position in Go arrays.
// This is used when decoding [[array tables]] into Go arrays. Given array
// tables are separate TOML expression, we need to keep track of where we
@@ -139,6 +165,9 @@ type decoder struct {
// Strict mode
strict strict
// Flag that enables/disables unmarshaler interface.
unmarshalerInterface bool
// Current context for the error.
errorContext *errorContext
}
@@ -149,12 +178,16 @@ type errorContext struct {
}
func (d *decoder) typeMismatchError(toml string, target reflect.Type) error {
return fmt.Errorf("toml: %s", d.typeMismatchString(toml, target))
}
func (d *decoder) typeMismatchString(toml string, target reflect.Type) string {
if d.errorContext != nil && d.errorContext.Struct != nil {
ctx := d.errorContext
f := ctx.Struct.FieldByIndex(ctx.Field)
return fmt.Errorf("toml: cannot decode TOML %s into struct field %s.%s of type %s", toml, ctx.Struct, f.Name, f.Type)
return fmt.Sprintf("cannot decode TOML %s into struct field %s.%s of type %s", toml, ctx.Struct, f.Name, f.Type)
}
return fmt.Errorf("toml: cannot decode TOML %s into a Go value of type %s", toml, target)
return fmt.Sprintf("cannot decode TOML %s into a Go value of type %s", toml, target)
}
func (d *decoder) expr() *unstable.Node {
@@ -242,9 +275,10 @@ Rules for the unmarshal code:
func (d *decoder) handleRootExpression(expr *unstable.Node, v reflect.Value) error {
var x reflect.Value
var err error
var first bool // used for to clear array tables on first use
if !(d.skipUntilTable && expr.Kind == unstable.KeyValue) {
err = d.seen.CheckExpression(expr)
first, err = d.seen.CheckExpression(expr)
if err != nil {
return err
}
@@ -263,6 +297,7 @@ func (d *decoder) handleRootExpression(expr *unstable.Node, v reflect.Value) err
case unstable.ArrayTable:
d.skipUntilTable = false
d.strict.EnterArrayTable(expr)
d.clearArrayTable = first
x, err = d.handleArrayTable(expr.Key(), v)
default:
panic(fmt.Errorf("parser should not permit expression of kind %s at document root", expr.Kind))
@@ -303,6 +338,10 @@ func (d *decoder) handleArrayTableCollectionLast(key unstable.Iterator, v reflec
reflect.Copy(nelem, elem)
elem = nelem
}
if d.clearArrayTable && elem.Len() > 0 {
elem.SetLen(0)
d.clearArrayTable = false
}
}
return d.handleArrayTableCollectionLast(key, elem)
case reflect.Ptr:
@@ -321,6 +360,10 @@ func (d *decoder) handleArrayTableCollectionLast(key unstable.Iterator, v reflec
return v, nil
case reflect.Slice:
if d.clearArrayTable && v.Len() > 0 {
v.SetLen(0)
d.clearArrayTable = false
}
elemType := v.Type().Elem()
var elem reflect.Value
if elemType.Kind() == reflect.Interface {
@@ -417,7 +460,10 @@ func (d *decoder) handleKeyPart(key unstable.Iterator, v reflect.Value, nextFn h
vt := v.Type()
// Create the key for the map element. Convert to key type.
mk := reflect.ValueOf(string(key.Node().Data)).Convert(vt.Key())
mk, err := d.keyFromData(vt.Key(), key.Node().Data)
if err != nil {
return reflect.Value{}, err
}
// If the map does not exist, create it.
if v.IsNil() {
@@ -569,7 +615,7 @@ func (d *decoder) handleKeyValues(v reflect.Value) (reflect.Value, error) {
break
}
err := d.seen.CheckExpression(expr)
_, err := d.seen.CheckExpression(expr)
if err != nil {
return reflect.Value{}, err
}
@@ -627,6 +673,14 @@ func (d *decoder) handleValue(value *unstable.Node, v reflect.Value) error {
v = initAndDereferencePointer(v)
}
if d.unmarshalerInterface {
if v.CanAddr() && v.Addr().CanInterface() {
if outi, ok := v.Addr().Interface().(unstable.Unmarshaler); ok {
return outi.UnmarshalTOML(value)
}
}
}
ok, err := d.tryTextUnmarshaler(value, v)
if ok || err != nil {
return err
@@ -746,7 +800,7 @@ func (d *decoder) unmarshalInlineTable(itable *unstable.Node, v reflect.Value) e
}
return d.unmarshalInlineTable(itable, elem)
default:
return unstable.NewParserError(itable.Data, "cannot store inline table in Go type %s", v.Kind())
return unstable.NewParserError(d.p.Raw(itable.Raw), "cannot store inline table in Go type %s", v.Kind())
}
it := itable.Children()
@@ -887,6 +941,11 @@ func init() {
}
func (d *decoder) unmarshalInteger(value *unstable.Node, v reflect.Value) error {
kind := v.Kind()
if kind == reflect.Float32 || kind == reflect.Float64 {
return d.unmarshalFloat(value, v)
}
i, err := parseInteger(value.Data)
if err != nil {
return err
@@ -894,7 +953,7 @@ func (d *decoder) unmarshalInteger(value *unstable.Node, v reflect.Value) error
var r reflect.Value
switch v.Kind() {
switch kind {
case reflect.Int64:
v.SetInt(i)
return nil
@@ -955,7 +1014,7 @@ func (d *decoder) unmarshalInteger(value *unstable.Node, v reflect.Value) error
case reflect.Interface:
r = reflect.ValueOf(i)
default:
return d.typeMismatchError("integer", v.Type())
return unstable.NewParserError(d.p.Raw(value.Raw), d.typeMismatchString("integer", v.Type()))
}
if !r.Type().AssignableTo(v.Type()) {
@@ -974,7 +1033,7 @@ func (d *decoder) unmarshalString(value *unstable.Node, v reflect.Value) error {
case reflect.Interface:
v.Set(reflect.ValueOf(string(value.Data)))
default:
return unstable.NewParserError(d.p.Raw(value.Raw), "cannot store TOML string into a Go %s", v.Kind())
return unstable.NewParserError(d.p.Raw(value.Raw), d.typeMismatchString("string", v.Type()))
}
return nil
@@ -1004,6 +1063,31 @@ func (d *decoder) handleKeyValueInner(key unstable.Iterator, value *unstable.Nod
return reflect.Value{}, d.handleValue(value, v)
}
func (d *decoder) keyFromData(keyType reflect.Type, data []byte) (reflect.Value, error) {
switch {
case stringType.AssignableTo(keyType):
return reflect.ValueOf(string(data)), nil
case stringType.ConvertibleTo(keyType):
return reflect.ValueOf(string(data)).Convert(keyType), nil
case keyType.Implements(textUnmarshalerType):
mk := reflect.New(keyType.Elem())
if err := mk.Interface().(encoding.TextUnmarshaler).UnmarshalText(data); err != nil {
return reflect.Value{}, fmt.Errorf("toml: error unmarshalling key type %s from text: %w", stringType, err)
}
return mk, nil
case reflect.PtrTo(keyType).Implements(textUnmarshalerType):
mk := reflect.New(keyType)
if err := mk.Interface().(encoding.TextUnmarshaler).UnmarshalText(data); err != nil {
return reflect.Value{}, fmt.Errorf("toml: error unmarshalling key type %s from text: %w", stringType, err)
}
return mk.Elem(), nil
}
return reflect.Value{}, fmt.Errorf("toml: cannot convert map key of type %s to expected type %s", stringType, keyType)
}
func (d *decoder) handleKeyValuePart(key unstable.Iterator, value *unstable.Node, v reflect.Value) (reflect.Value, error) {
// contains the replacement for v
var rv reflect.Value
@@ -1014,16 +1098,9 @@ func (d *decoder) handleKeyValuePart(key unstable.Iterator, value *unstable.Node
case reflect.Map:
vt := v.Type()
mk := reflect.ValueOf(string(key.Node().Data))
mkt := stringType
keyType := vt.Key()
if !mkt.AssignableTo(keyType) {
if !mkt.ConvertibleTo(keyType) {
return reflect.Value{}, fmt.Errorf("toml: cannot convert map key of type %s to expected type %s", mkt, keyType)
}
mk = mk.Convert(keyType)
mk, err := d.keyFromData(vt.Key(), key.Node().Data)
if err != nil {
return reflect.Value{}, err
}
// If the map does not exist, create it.
@@ -1034,15 +1111,9 @@ func (d *decoder) handleKeyValuePart(key unstable.Iterator, value *unstable.Node
mv := v.MapIndex(mk)
set := false
if !mv.IsValid() {
if !mv.IsValid() || key.IsLast() {
set = true
mv = reflect.New(v.Type().Elem()).Elem()
} else {
if key.IsLast() {
var x interface{}
mv = reflect.ValueOf(&x).Elem()
set = true
}
}
nv, err := d.handleKeyValueInner(key, value, mv)
@@ -1072,6 +1143,19 @@ func (d *decoder) handleKeyValuePart(key unstable.Iterator, value *unstable.Node
d.errorContext.Field = path
f := fieldByIndex(v, path)
if !f.CanAddr() {
// If the field is not addressable, need to take a slower path and
// make a copy of the struct itself to a new location.
nvp := reflect.New(v.Type())
nvp.Elem().Set(v)
v = nvp.Elem()
_, err := d.handleKeyValuePart(key, value, v)
if err != nil {
return reflect.Value{}, err
}
return nvp.Elem(), nil
}
x, err := d.handleKeyValueInner(key, value, f)
if err != nil {
return reflect.Value{}, err
@@ -1137,10 +1221,10 @@ func initAndDereferencePointer(v reflect.Value) reflect.Value {
// Same as reflect.Value.FieldByIndex, but creates pointers if needed.
func fieldByIndex(v reflect.Value, path []int) reflect.Value {
for i, x := range path {
for _, x := range path {
v = v.Field(x)
if i < len(path)-1 && v.Kind() == reflect.Ptr {
if v.Kind() == reflect.Ptr {
if v.IsNil() {
v.Set(reflect.New(v.Type().Elem()))
}
+568 -4
View File
@@ -12,10 +12,32 @@ import (
"time"
"github.com/pelletier/go-toml/v2"
"github.com/pelletier/go-toml/v2/unstable"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
type unmarshalTextKey struct {
A string
B string
}
func (k *unmarshalTextKey) UnmarshalText(text []byte) error {
parts := strings.Split(string(text), "-")
if len(parts) != 2 {
return fmt.Errorf("invalid text key: %s", text)
}
k.A = parts[0]
k.B = parts[1]
return nil
}
type unmarshalBadTextKey struct{}
func (k *unmarshalBadTextKey) UnmarshalText(text []byte) error {
return fmt.Errorf("error")
}
func ExampleDecoder_DisallowUnknownFields() {
type S struct {
Key1 string
@@ -69,7 +91,6 @@ func ExampleUnmarshal() {
fmt.Println("version:", cfg.Version)
fmt.Println("name:", cfg.Name)
fmt.Println("tags:", cfg.Tags)
// Output:
// version: 2
// name: go-toml
@@ -220,6 +241,11 @@ func TestUnmarshal_Floats(t *testing.T) {
input: `0E0`,
expected: 0.0,
},
{
desc: "float zero without decimals",
input: `0`,
expected: 0.0,
},
{
desc: "float fractional with exponent",
input: `6.626e-34`,
@@ -315,6 +341,7 @@ func TestUnmarshal(t *testing.T) {
target interface{}
expected interface{}
err bool
assert func(t *testing.T, test test)
}
examples := []struct {
skip bool
@@ -350,6 +377,96 @@ func TestUnmarshal(t *testing.T) {
}
},
},
{
desc: "kv text key",
input: `a-1 = "foo"`,
gen: func() test {
type doc = map[unmarshalTextKey]string
return test{
target: &doc{},
expected: &doc{{A: "a", B: "1"}: "foo"},
}
},
},
{
desc: "table text key",
input: `["a-1"]
foo = "bar"`,
gen: func() test {
type doc = map[unmarshalTextKey]map[string]string
return test{
target: &doc{},
expected: &doc{{A: "a", B: "1"}: map[string]string{"foo": "bar"}},
}
},
},
{
desc: "kv ptr text key",
input: `a-1 = "foo"`,
gen: func() test {
type doc = map[*unmarshalTextKey]string
return test{
target: &doc{},
expected: &doc{{A: "a", B: "1"}: "foo"},
assert: func(t *testing.T, test test) {
// Despite the documentation:
// Pointer variable equality is determined based on the equality of the
// referenced values (as opposed to the memory addresses).
// assert.Equal does not work properly with maps with pointer keys
// https://github.com/stretchr/testify/issues/1143
expected := make(map[unmarshalTextKey]string)
for k, v := range *(test.expected.(*doc)) {
expected[*k] = v
}
got := make(map[unmarshalTextKey]string)
for k, v := range *(test.target.(*doc)) {
got[*k] = v
}
assert.Equal(t, expected, got)
},
}
},
},
{
desc: "kv bad text key",
input: `a-1 = "foo"`,
gen: func() test {
type doc = map[unmarshalBadTextKey]string
return test{
target: &doc{},
err: true,
}
},
},
{
desc: "kv bad ptr text key",
input: `a-1 = "foo"`,
gen: func() test {
type doc = map[*unmarshalBadTextKey]string
return test{
target: &doc{},
err: true,
}
},
},
{
desc: "table bad text key",
input: `["a-1"]
foo = "bar"`,
gen: func() test {
type doc = map[unmarshalBadTextKey]map[string]string
return test{
target: &doc{},
err: true,
}
},
},
{
desc: "time.time with negative zone",
input: `a = 1979-05-27T00:32:00-07:00 `, // space intentional
@@ -1166,6 +1283,64 @@ B = "data"`,
}
},
},
{
desc: "array table into maps with pointer on last key",
input: `[[foo]]
bar = "hello"`,
gen: func() test {
type doc struct {
Foo **[]interface{}
}
x := &[]interface{}{
map[string]interface{}{
"bar": "hello",
},
}
return test{
target: &doc{},
expected: &doc{
Foo: &x,
},
}
},
},
{
desc: "array table into maps with pointer on intermediate key",
input: `[[foo.foo2]]
bar = "hello"`,
gen: func() test {
type doc struct {
Foo **map[string]interface{}
}
x := &map[string]interface{}{
"foo2": []interface{}{
map[string]interface{}{
"bar": "hello",
},
},
}
return test{
target: &doc{},
expected: &doc{
Foo: &x,
},
}
},
},
{
desc: "array table into maps with pointer on last key with invalid leaf type",
input: `[[foo]]
bar = "hello"`,
gen: func() test {
type doc struct {
Foo **[]map[string]int
}
return test{
target: &doc{},
err: true,
}
},
},
{
desc: "unexported struct fields are ignored",
input: `foo = "bar"`,
@@ -1521,6 +1696,16 @@ B = "data"`,
}
},
},
{
desc: "empty map into map with invalid key type",
input: ``,
gen: func() test {
return test{
target: &map[int]string{},
expected: &map[int]string{},
}
},
},
{
desc: "into map with convertible key type",
input: `A = "hello"`,
@@ -1777,7 +1962,11 @@ B = "data"`,
require.Error(t, err)
} else {
require.NoError(t, err)
assert.Equal(t, test.expected, test.target)
if test.assert != nil {
test.assert(t, test)
} else {
assert.Equal(t, test.expected, test.target)
}
}
})
}
@@ -1860,6 +2049,68 @@ func TestUnmarshalErrors(t *testing.T) {
require.Equal(t, "toml: cannot decode TOML integer into struct field toml_test.mystruct.Bar of type string", err.Error())
}
func TestUnmarshalStringInvalidStructField(t *testing.T) {
type Server struct {
Path string
Port int
}
type Cfg struct {
Server Server
}
var cfg Cfg
data := `[server]
path = "/my/path"
port = "bad"
`
file := strings.NewReader(data)
err := toml.NewDecoder(file).Decode(&cfg)
require.Error(t, err)
x := err.(*toml.DecodeError)
require.Equal(t, "toml: cannot decode TOML string into struct field toml_test.Server.Port of type int", x.Error())
expected := `1| [server]
2| path = "/my/path"
3| port = "bad"
| ~~~~~ cannot decode TOML string into struct field toml_test.Server.Port of type int`
require.Equal(t, expected, x.String())
}
func TestUnmarshalIntegerInvalidStructField(t *testing.T) {
type Server struct {
Path string
Port int
}
type Cfg struct {
Server Server
}
var cfg Cfg
data := `[server]
path = 100
port = 50
`
file := strings.NewReader(data)
err := toml.NewDecoder(file).Decode(&cfg)
require.Error(t, err)
x := err.(*toml.DecodeError)
require.Equal(t, "toml: cannot decode TOML integer into struct field toml_test.Server.Path of type string", x.Error())
expected := `1| [server]
2| path = 100
| ~~~ cannot decode TOML integer into struct field toml_test.Server.Path of type string
3| port = 50`
require.Equal(t, expected, x.String())
}
func TestUnmarshalInvalidTarget(t *testing.T) {
x := "foo"
err := toml.Unmarshal([]byte{}, x)
@@ -2466,6 +2717,183 @@ func TestIssue807(t *testing.T) {
require.Equal(t, "foo", m.Name)
}
func TestIssue850(t *testing.T) {
data := make(map[string]string)
err := toml.Unmarshal([]byte("foo = {}"), &data)
require.Error(t, err)
}
func TestIssue851(t *testing.T) {
type Target struct {
Params map[string]string `toml:"params"`
}
content := "params = {a=\"1\",b=\"2\"}"
var target Target
err := toml.Unmarshal([]byte(content), &target)
require.NoError(t, err)
require.Equal(t, map[string]string{"a": "1", "b": "2"}, target.Params)
err = toml.Unmarshal([]byte(content), &target)
require.NoError(t, err)
require.Equal(t, map[string]string{"a": "1", "b": "2"}, target.Params)
}
func TestIssue866(t *testing.T) {
type Pipeline struct {
Mapping map[string]struct {
Req [][]string `toml:"req"`
Res [][]string `toml:"res"`
} `toml:"mapping"`
}
type Pipelines struct {
PipelineMapping map[string]*Pipeline `toml:"pipelines"`
}
var badToml = `
[pipelines.register]
mapping.inst.req = [
["param1", "value1"],
]
mapping.inst.res = [
["param2", "value2"],
]
`
pipelines := new(Pipelines)
if err := toml.NewDecoder(bytes.NewBufferString(badToml)).DisallowUnknownFields().Decode(pipelines); err != nil {
t.Fatal(err)
}
if pipelines.PipelineMapping["register"].Mapping["inst"].Req[0][0] != "param1" {
t.Fatal("unmarshal failed with mismatch value")
}
var goodTooToml = `
[pipelines.register]
mapping.inst.req = [
["param1", "value1"],
]
`
pipelines = new(Pipelines)
if err := toml.NewDecoder(bytes.NewBufferString(goodTooToml)).DisallowUnknownFields().Decode(pipelines); err != nil {
t.Fatal(err)
}
if pipelines.PipelineMapping["register"].Mapping["inst"].Req[0][0] != "param1" {
t.Fatal("unmarshal failed with mismatch value")
}
var goodToml = `
[pipelines.register.mapping.inst]
req = [
["param1", "value1"],
]
res = [
["param2", "value2"],
]
`
pipelines = new(Pipelines)
if err := toml.NewDecoder(bytes.NewBufferString(goodToml)).DisallowUnknownFields().Decode(pipelines); err != nil {
t.Fatal(err)
}
if pipelines.PipelineMapping["register"].Mapping["inst"].Req[0][0] != "param1" {
t.Fatal("unmarshal failed with mismatch value")
}
}
func TestIssue915(t *testing.T) {
type blah struct {
A string `toml:"a"`
}
type config struct {
Fizz string `toml:"fizz"`
blah `toml:"blah"`
}
b := []byte(`
fizz = "abc"
blah.a = "def"`)
var cfg config
err := toml.Unmarshal(b, &cfg)
require.NoError(t, err)
require.Equal(t, "abc", cfg.Fizz)
require.Equal(t, "def", cfg.blah.A)
require.Equal(t, "def", cfg.A)
}
func TestIssue931(t *testing.T) {
type item struct {
Name string
}
type items struct {
Slice []item
}
its := items{[]item{{"a"}, {"b"}}}
b := []byte(`
[[Slice]]
Name = 'c'
[[Slice]]
Name = 'd'
`)
toml.Unmarshal(b, &its)
require.Equal(t, items{[]item{{"c"}, {"d"}}}, its)
}
func TestIssue931Interface(t *testing.T) {
type items struct {
Slice interface{}
}
type item = map[string]interface{}
its := items{[]interface{}{item{"Name": "a"}, item{"Name": "b"}}}
b := []byte(`
[[Slice]]
Name = 'c'
[[Slice]]
Name = 'd'
`)
toml.Unmarshal(b, &its)
require.Equal(t, items{[]interface{}{item{"Name": "c"}, item{"Name": "d"}}}, its)
}
func TestIssue931SliceInterface(t *testing.T) {
type items struct {
Slice []interface{}
}
type item = map[string]interface{}
its := items{
[]interface{}{
item{"Name": "a"},
item{"Name": "b"},
},
}
b := []byte(`
[[Slice]]
Name = 'c'
[[Slice]]
Name = 'd'
`)
toml.Unmarshal(b, &its)
require.Equal(t, items{[]interface{}{item{"Name": "c"}, item{"Name": "d"}}}, its)
}
func TestUnmarshalDecodeErrors(t *testing.T) {
examples := []struct {
desc string
@@ -2742,7 +3170,7 @@ world'`,
data: "a = \"aaaa\xE2\x80\x00\"",
},
{
desc: "invalid 4rd byte of 4-byte utf8 character in string with no escape sequence",
desc: "invalid 4th byte of 4-byte utf8 character in string with no escape sequence",
data: "a = \"aaaa\xF2\x81\x81\x00\"",
},
{
@@ -2758,7 +3186,7 @@ world'`,
data: "a = 'aaaa\xE2\x80\x00'",
},
{
desc: "invalid 4rd byte of 4-byte utf8 character in literal string",
desc: "invalid 4th byte of 4-byte utf8 character in literal string",
data: "a = 'aaaa\xF2\x81\x81\x00'",
},
{
@@ -3301,3 +3729,139 @@ func TestUnmarshalEmbedNonString(t *testing.T) {
require.NoError(t, err)
require.Nil(t, d.Foo)
}
func TestUnmarshal_Nil(t *testing.T) {
type Foo struct {
Foo *Foo `toml:"foo,omitempty"`
Bar *Foo `toml:"bar,omitempty"`
}
examples := []struct {
desc string
input string
expected string
err bool
}{
{
desc: "empty",
input: ``,
expected: ``,
},
{
desc: "simplest",
input: `
[foo]
[foo.foo]
`,
expected: "[foo]\n[foo.foo]\n",
},
}
for _, ex := range examples {
e := ex
t.Run(e.desc, func(t *testing.T) {
foo := Foo{}
err := toml.Unmarshal([]byte(e.input), &foo)
if e.err {
require.Error(t, err)
} else {
require.NoError(t, err)
j, err := toml.Marshal(foo)
require.NoError(t, err)
assert.Equal(t, e.expected, string(j))
}
})
}
}
type CustomUnmarshalerKey struct {
A int64
}
func (k *CustomUnmarshalerKey) UnmarshalTOML(value *unstable.Node) error {
item, err := strconv.ParseInt(string(value.Data), 10, 64)
if err != nil {
return fmt.Errorf("error converting to int64, %v", err)
}
k.A = item
return nil
}
func TestUnmarshal_CustomUnmarshaler(t *testing.T) {
type MyConfig struct {
Unmarshalers []CustomUnmarshalerKey `toml:"unmarshalers"`
Foo *string `toml:"foo,omitempty"`
}
examples := []struct {
desc string
disableUnmarshalerInterface bool
input string
expected MyConfig
err bool
}{
{
desc: "empty",
input: ``,
expected: MyConfig{Unmarshalers: []CustomUnmarshalerKey{}, Foo: nil},
},
{
desc: "simple",
input: `unmarshalers = [1,2,3]`,
expected: MyConfig{
Unmarshalers: []CustomUnmarshalerKey{
{A: 1},
{A: 2},
{A: 3},
},
Foo: nil,
},
},
{
desc: "unmarshal string and custom unmarshaler",
input: `unmarshalers = [1,2,3]
foo = "bar"`,
expected: MyConfig{
Unmarshalers: []CustomUnmarshalerKey{
{A: 1},
{A: 2},
{A: 3},
},
Foo: func(v string) *string {
return &v
}("bar"),
},
},
{
desc: "simple example, but unmarshaler interface disabled",
disableUnmarshalerInterface: true,
input: `unmarshalers = [1,2,3]`,
err: true,
},
}
for _, ex := range examples {
e := ex
t.Run(e.desc, func(t *testing.T) {
foo := MyConfig{}
decoder := toml.NewDecoder(bytes.NewReader([]byte(e.input)))
if !ex.disableUnmarshalerInterface {
decoder.EnableUnmarshalerInterface()
}
err := decoder.Decode(&foo)
if e.err {
require.Error(t, err)
} else {
require.NoError(t, err)
require.Equal(t, len(foo.Unmarshalers), len(e.expected.Unmarshalers))
for i := 0; i < len(foo.Unmarshalers); i++ {
require.Equal(t, foo.Unmarshalers[i], e.expected.Unmarshalers[i])
}
require.Equal(t, foo.Foo, e.expected.Foo)
}
})
}
}
+1 -1
View File
@@ -58,7 +58,7 @@ func (c *Iterator) Node() *Node {
// - Table and ArrayTable's children represent a dotted key (same as
// KeyValue, but without the first node being the value).
//
// When relevant, Raw describes the range of bytes this node is refering to in
// When relevant, Raw describes the range of bytes this node is referring to in
// the input document. Use Parser.Raw() to retrieve the actual bytes.
type Node struct {
Kind Kind
+119 -21
View File
@@ -49,8 +49,6 @@ func NewParserError(highlight []byte, format string, args ...interface{}) error
// For performance reasons, go-toml doesn't make a copy of the input bytes to
// the parser. Make sure to copy all the bytes you need to outlive the slice
// given to the parser.
//
// The parser doesn't provide nodes for comments yet, nor for whitespace.
type Parser struct {
data []byte
builder builder
@@ -58,6 +56,8 @@ type Parser struct {
left []byte
err error
first bool
KeepComments bool
}
// Data returns the slice provided to the last call to Reset.
@@ -132,16 +132,54 @@ func (p *Parser) NextExpression() bool {
}
// Expression returns a pointer to the node representing the last successfully
// parsed expresion.
// parsed expression.
func (p *Parser) Expression() *Node {
return p.builder.NodeAt(p.ref)
}
// Error returns any error that has occured during parsing.
// Error returns any error that has occurred during parsing.
func (p *Parser) Error() error {
return p.err
}
// Position describes a position in the input.
type Position struct {
// Number of bytes from the beginning of the input.
Offset int
// Line number, starting at 1.
Line int
// Column number, starting at 1.
Column int
}
// Shape describes the position of a range in the input.
type Shape struct {
Start Position
End Position
}
func (p *Parser) position(b []byte) Position {
offset := danger.SubsliceOffset(p.data, b)
lead := p.data[:offset]
return Position{
Offset: offset,
Line: bytes.Count(lead, []byte{'\n'}) + 1,
Column: len(lead) - bytes.LastIndex(lead, []byte{'\n'}),
}
}
// Shape returns the shape of the given range in the input. Will
// panic if the range is not a subslice of the input.
func (p *Parser) Shape(r Range) Shape {
raw := p.Raw(r)
return Shape{
Start: p.position(raw),
End: p.position(raw[r.Length:]),
}
}
func (p *Parser) parseNewline(b []byte) ([]byte, error) {
if b[0] == '\n' {
return b[1:], nil
@@ -155,6 +193,19 @@ func (p *Parser) parseNewline(b []byte) ([]byte, error) {
return nil, NewParserError(b[0:1], "expected newline but got %#U", b[0])
}
func (p *Parser) parseComment(b []byte) (reference, []byte, error) {
ref := invalidReference
data, rest, err := scanComment(b)
if p.KeepComments && err == nil {
ref = p.builder.Push(Node{
Kind: Comment,
Raw: p.Range(data),
Data: data,
})
}
return ref, rest, err
}
func (p *Parser) parseExpression(b []byte) (reference, []byte, error) {
// expression = ws [ comment ]
// expression =/ ws keyval ws [ comment ]
@@ -168,7 +219,7 @@ func (p *Parser) parseExpression(b []byte) (reference, []byte, error) {
}
if b[0] == '#' {
_, rest, err := scanComment(b)
ref, rest, err := p.parseComment(b)
return ref, rest, err
}
@@ -190,7 +241,10 @@ func (p *Parser) parseExpression(b []byte) (reference, []byte, error) {
b = p.parseWhitespace(b)
if len(b) > 0 && b[0] == '#' {
_, rest, err := scanComment(b)
cref, rest, err := p.parseComment(b)
if cref != invalidReference {
p.builder.Chain(ref, cref)
}
return ref, rest, err
}
@@ -402,6 +456,7 @@ func (p *Parser) parseInlineTable(b []byte) (reference, []byte, error) {
// inline-table-keyvals = keyval [ inline-table-sep inline-table-keyvals ]
parent := p.builder.Push(Node{
Kind: InlineTable,
Raw: p.Range(b[:1]),
})
first := true
@@ -470,17 +525,33 @@ func (p *Parser) parseValArray(b []byte) (reference, []byte, error) {
Kind: Array,
})
// First indicates whether the parser is looking for the first element
// (non-comment) of the array.
first := true
var lastChild reference
lastChild := invalidReference
addChild := func(valueRef reference) {
if lastChild == invalidReference {
p.builder.AttachChild(parent, valueRef)
} else {
p.builder.Chain(lastChild, valueRef)
}
lastChild = valueRef
}
var err error
for len(b) > 0 {
b, err = p.parseOptionalWhitespaceCommentNewline(b)
cref := invalidReference
cref, b, err = p.parseOptionalWhitespaceCommentNewline(b)
if err != nil {
return parent, nil, err
}
if cref != invalidReference {
addChild(cref)
}
if len(b) == 0 {
return parent, nil, NewParserError(arrayStart[:1], "array is incomplete")
}
@@ -495,10 +566,13 @@ func (p *Parser) parseValArray(b []byte) (reference, []byte, error) {
}
b = b[1:]
b, err = p.parseOptionalWhitespaceCommentNewline(b)
cref, b, err = p.parseOptionalWhitespaceCommentNewline(b)
if err != nil {
return parent, nil, err
}
if cref != invalidReference {
addChild(cref)
}
} else if !first {
return parent, nil, NewParserError(b[0:1], "array elements must be separated by commas")
}
@@ -514,17 +588,16 @@ func (p *Parser) parseValArray(b []byte) (reference, []byte, error) {
return parent, nil, err
}
if first {
p.builder.AttachChild(parent, valueRef)
} else {
p.builder.Chain(lastChild, valueRef)
}
lastChild = valueRef
addChild(valueRef)
b, err = p.parseOptionalWhitespaceCommentNewline(b)
cref, b, err = p.parseOptionalWhitespaceCommentNewline(b)
if err != nil {
return parent, nil, err
}
if cref != invalidReference {
addChild(cref)
}
first = false
}
@@ -533,15 +606,34 @@ func (p *Parser) parseValArray(b []byte) (reference, []byte, error) {
return parent, rest, err
}
func (p *Parser) parseOptionalWhitespaceCommentNewline(b []byte) ([]byte, error) {
func (p *Parser) parseOptionalWhitespaceCommentNewline(b []byte) (reference, []byte, error) {
rootCommentRef := invalidReference
latestCommentRef := invalidReference
addComment := func(ref reference) {
if rootCommentRef == invalidReference {
rootCommentRef = ref
} else if latestCommentRef == invalidReference {
p.builder.AttachChild(rootCommentRef, ref)
latestCommentRef = ref
} else {
p.builder.Chain(latestCommentRef, ref)
latestCommentRef = ref
}
}
for len(b) > 0 {
var err error
b = p.parseWhitespace(b)
if len(b) > 0 && b[0] == '#' {
_, b, err = scanComment(b)
var ref reference
ref, b, err = p.parseComment(b)
if err != nil {
return nil, err
return invalidReference, nil, err
}
if ref != invalidReference {
addComment(ref)
}
}
@@ -552,14 +644,14 @@ func (p *Parser) parseOptionalWhitespaceCommentNewline(b []byte) ([]byte, error)
if b[0] == '\n' || b[0] == '\r' {
b, err = p.parseNewline(b)
if err != nil {
return nil, err
return invalidReference, nil, err
}
} else {
break
}
}
return b, nil
return rootCommentRef, b, nil
}
func (p *Parser) parseMultilineLiteralString(b []byte) ([]byte, []byte, []byte, error) {
@@ -921,6 +1013,7 @@ func (p *Parser) parseIntOrFloatOrDateTime(b []byte) (reference, []byte, error)
return p.builder.Push(Node{
Kind: Float,
Data: b[:3],
Raw: p.Range(b[:3]),
}), b[3:], nil
case 'n':
if !scanFollowsNan(b) {
@@ -930,6 +1023,7 @@ func (p *Parser) parseIntOrFloatOrDateTime(b []byte) (reference, []byte, error)
return p.builder.Push(Node{
Kind: Float,
Data: b[:3],
Raw: p.Range(b[:3]),
}), b[3:], nil
case '+', '-':
return p.scanIntOrFloat(b)
@@ -1054,6 +1148,7 @@ func (p *Parser) scanIntOrFloat(b []byte) (reference, []byte, error) {
return p.builder.Push(Node{
Kind: Integer,
Data: b[:i],
Raw: p.Range(b[:i]),
}), b[i:], nil
}
@@ -1077,6 +1172,7 @@ func (p *Parser) scanIntOrFloat(b []byte) (reference, []byte, error) {
return p.builder.Push(Node{
Kind: Float,
Data: b[:i+3],
Raw: p.Range(b[:i+3]),
}), b[i+3:], nil
}
@@ -1088,6 +1184,7 @@ func (p *Parser) scanIntOrFloat(b []byte) (reference, []byte, error) {
return p.builder.Push(Node{
Kind: Float,
Data: b[:i+3],
Raw: p.Range(b[:i+3]),
}), b[i+3:], nil
}
@@ -1110,6 +1207,7 @@ func (p *Parser) scanIntOrFloat(b []byte) (reference, []byte, error) {
return p.builder.Push(Node{
Kind: kind,
Data: b[:i],
Raw: p.Range(b[:i]),
}), b[i:], nil
}
+157
View File
@@ -448,6 +448,163 @@ func TestParser_AST_DateTimes(t *testing.T) {
}
}
// This example demonstrates how to parse a TOML document and preserving
// comments. Comments are stored in the AST as Comment nodes. This example
// displays the structure of the full AST generated by the parser using the
// following structure:
//
// 1. Each root-level expression is separated by three dashes.
// 2. Bytes associated to a node are displayed in square brackets.
// 3. Siblings have the same indentation.
// 4. Children of a node are indented one level.
func ExampleParser_comments() {
doc := `# Top of the document comment.
# Optional, any amount of lines.
# Above table.
[table] # Next to table.
# Above simple value.
key = "value" # Next to simple value.
# Below simple value.
# Some comment alone.
# Multiple comments, on multiple lines.
# Above inline table.
name = { first = "Tom", last = "Preston-Werner" } # Next to inline table.
# Below inline table.
# Above array.
array = [ 1, 2, 3 ] # Next to one-line array.
# Below array.
# Above multi-line array.
key5 = [ # Next to start of inline array.
# Second line before array content.
1, # Next to first element.
# After first element.
# Before second element.
2,
3, # Next to last element
# After last element.
] # Next to end of array.
# Below multi-line array.
# Before array table.
[[products]] # Next to array table.
# After array table.
`
var printGeneric func(*Parser, int, *Node)
printGeneric = func(p *Parser, indent int, e *Node) {
if e == nil {
return
}
s := p.Shape(e.Raw)
x := fmt.Sprintf("%d:%d->%d:%d (%d->%d)", s.Start.Line, s.Start.Column, s.End.Line, s.End.Column, s.Start.Offset, s.End.Offset)
fmt.Printf("%-25s | %s%s [%s]\n", x, strings.Repeat(" ", indent), e.Kind, e.Data)
printGeneric(p, indent+1, e.Child())
printGeneric(p, indent, e.Next())
}
printTree := func(p *Parser) {
for p.NextExpression() {
e := p.Expression()
fmt.Println("---")
printGeneric(p, 0, e)
}
if err := p.Error(); err != nil {
panic(err)
}
}
p := &Parser{
KeepComments: true,
}
p.Reset([]byte(doc))
printTree(p)
// Output:
// ---
// 1:1->1:31 (0->30) | Comment [# Top of the document comment.]
// ---
// 2:1->2:33 (31->63) | Comment [# Optional, any amount of lines.]
// ---
// 4:1->4:15 (65->79) | Comment [# Above table.]
// ---
// 1:1->1:1 (0->0) | Table []
// 5:2->5:7 (81->86) | Key [table]
// 5:9->5:25 (88->104) | Comment [# Next to table.]
// ---
// 6:1->6:22 (105->126) | Comment [# Above simple value.]
// ---
// 1:1->1:1 (0->0) | KeyValue []
// 7:7->7:14 (133->140) | String [value]
// 7:1->7:4 (127->130) | Key [key]
// 7:15->7:38 (141->164) | Comment [# Next to simple value.]
// ---
// 8:1->8:22 (165->186) | Comment [# Below simple value.]
// ---
// 10:1->10:22 (188->209) | Comment [# Some comment alone.]
// ---
// 12:1->12:40 (211->250) | Comment [# Multiple comments, on multiple lines.]
// ---
// 14:1->14:22 (252->273) | Comment [# Above inline table.]
// ---
// 1:1->1:1 (0->0) | KeyValue []
// 15:8->15:9 (281->282) | InlineTable []
// 1:1->1:1 (0->0) | KeyValue []
// 15:18->15:23 (291->296) | String [Tom]
// 15:10->15:15 (283->288) | Key [first]
// 1:1->1:1 (0->0) | KeyValue []
// 15:32->15:48 (305->321) | String [Preston-Werner]
// 15:25->15:29 (298->302) | Key [last]
// 15:1->15:5 (274->278) | Key [name]
// 15:51->15:74 (324->347) | Comment [# Next to inline table.]
// ---
// 16:1->16:22 (348->369) | Comment [# Below inline table.]
// ---
// 18:1->18:15 (371->385) | Comment [# Above array.]
// ---
// 1:1->1:1 (0->0) | KeyValue []
// 1:1->1:1 (0->0) | Array []
// 19:11->19:12 (396->397) | Integer [1]
// 19:14->19:15 (399->400) | Integer [2]
// 19:17->19:18 (402->403) | Integer [3]
// 19:1->19:6 (386->391) | Key [array]
// 19:21->19:46 (406->431) | Comment [# Next to one-line array.]
// ---
// 20:1->20:15 (432->446) | Comment [# Below array.]
// ---
// 22:1->22:26 (448->473) | Comment [# Above multi-line array.]
// ---
// 1:1->1:1 (0->0) | KeyValue []
// 1:1->1:1 (0->0) | Array []
// 23:10->23:42 (483->515) | Comment [# Next to start of inline array.]
// 24:3->24:38 (518->553) | Comment [# Second line before array content.]
// 25:3->25:4 (556->557) | Integer [1]
// 25:6->25:30 (559->583) | Comment [# Next to first element.]
// 26:3->26:25 (586->608) | Comment [# After first element.]
// 27:3->27:27 (611->635) | Comment [# Before second element.]
// 28:3->28:4 (638->639) | Integer [2]
// 29:3->29:4 (643->644) | Integer [3]
// 29:6->29:28 (646->668) | Comment [# Next to last element]
// 30:3->30:24 (671->692) | Comment [# After last element.]
// 23:1->23:5 (474->478) | Key [key5]
// 31:3->31:26 (695->718) | Comment [# Next to end of array.]
// ---
// 32:1->32:26 (719->744) | Comment [# Below multi-line array.]
// ---
// 34:1->34:22 (746->767) | Comment [# Before array table.]
// ---
// 1:1->1:1 (0->0) | ArrayTable []
// 35:3->35:11 (770->778) | Key [products]
// 35:14->35:36 (781->803) | Comment [# Next to array table.]
// ---
// 36:1->36:21 (804->824) | Comment [# After array table.]
}
func ExampleParser() {
doc := `
hello = "world"
-1
View File
@@ -151,7 +151,6 @@ func scanWhitespace(b []byte) ([]byte, []byte) {
return b, b[len(b):]
}
//nolint:unparam
func scanComment(b []byte) ([]byte, []byte, error) {
// comment-start-symbol = %x23 ; #
// non-ascii = %x80-D7FF / %xE000-10FFFF
+7
View File
@@ -0,0 +1,7 @@
package unstable
// The Unmarshaler interface may be implemented by types to customize their
// behavior when being unmarshaled from a TOML document.
type Unmarshaler interface {
UnmarshalTOML(value *Node) error
}