Compare commits

..

22 Commits

Author SHA1 Message Date
Thomas Pelletier 5dc006fb52 Update readme to point to v2 2022-04-27 21:51:02 -04:00
Cameron Moore fed1464066 Fix invalid type assertion in LoadBytes (#733)
Fixes #732
2022-01-05 09:17:32 -05:00
Thomas Pelletier 1baee4630f README: don't mention document editing in v2 2021-12-31 13:31:17 -05:00
Thomas Pelletier 352072d51a Create SECURITY.md 2021-10-17 20:56:41 -04:00
Wendell Sun c42c3365f3 Support unmarshaling up to uint64 (#608) 2021-09-27 19:11:21 -04:00
Bruno FERNANDO b8ba995eaa tomll: add multiLineArray flag to linter (#578) 2021-09-05 21:38:35 -04:00
Thomas Pelletier 8e44986c28 Configure issues templates
Ask people to go to /discussions for anything that is not a bug report.
2021-06-14 10:56:19 -04:00
Thomas Pelletier 837c1d09ee Clarify license and comply with Apache 2.0 (#554) 2021-06-08 13:57:30 -04:00
Thomas Pelletier 8410c965c2 Suggest using go-toml v2 2021-06-03 22:04:06 -04:00
Mikhail f. Shiryaev d083470585 Add Encoder.CompactComments to omit extra new line (#541) 2021-05-12 09:22:40 -04:00
Mikhail f. Shiryaev c893dbf25c Fix empty trees line counting (#539)
Refs #450
2021-05-11 08:50:05 -04:00
dependabot-preview[bot] 2a1df71375 Upgrade to GitHub-native Dependabot (#531)
Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2021-04-29 19:33:19 -04:00
Ikko Ashimine a2f5197638 Fix typo in README.md (#513)
availble -> available
2021-04-16 08:03:59 -04:00
Thomas Pelletier bb65137dc4 Move v2 warning into a comment 2021-04-10 19:54:52 -04:00
Thomas Pelletier 99782c87cf Add v2 warning in bug report template 2021-04-10 19:50:55 -04:00
Thomas Pelletier ce6fbd7bc0 Support literal multiline marshal (#485)
Use struct tag `multiline:"true" literal:"true"` to enable it.
2021-03-25 20:57:38 -04:00
Sapphire Becker b59c12a70d Remove go-spew dependency (#483) 2021-03-10 20:18:32 -05:00
Thomas Pelletier 6a307ac0d0 Update CI for Go 1.16 (#482)
Fixes #479
2021-03-04 09:58:18 -05:00
Thomas Pelletier a2e5256180 CI should also run on master
Otherwise codecov diff is incorrect.
2021-02-06 08:03:04 -05:00
Thomas Pelletier 5163266f16 Create codeql-analysis.yml (#473) 2021-02-05 08:33:55 -05:00
Vincent Serpoul b4f0a950bf Value string representation public function (#469)
Fixes #468
2021-02-03 08:48:53 -05:00
Thomas Pelletier ef48fb2be1 Expose MarshalOrder (#470)
It is needed as argument to some already public function.

Fixes #459
Ref #469
2021-02-02 12:12:10 -05:00
145 changed files with 18416 additions and 24675 deletions
-4
View File
@@ -1,4 +0,0 @@
* text=auto
benchmark/benchmark.toml text eol=lf
testdata/** text eol=lf
+11 -2
View File
@@ -1,9 +1,18 @@
---
name: Bug report
about: Create a report to help us improve
---
<!--
‼️ Main development focus is on the upcoming go-toml v2 ⚠️
As a result, v1.x bugs will likely not see a fix on a v1.x version.
However, reporting the bug is the best way to ensure that it will be fixed in v2.
See https://github.com/pelletier/go-toml/discussions/506.
-->
**Describe the bug**
A clear and concise description of what the bug is.
@@ -14,7 +23,7 @@ Steps to reproduce the behavior. Including TOML files.
A clear and concise description of what you expected to happen, if other than "should work".
**Versions**
- go-toml: version (git sha)
- go-toml: version (or git sha)
- go: version
- operating system: e.g. macOS, Windows, Linux
+5
View File
@@ -0,0 +1,5 @@
blank_issues_enabled: false
contact_links:
- name: Questions and discussions
url: https://github.com/pelletier/go-toml/discussions
about: Please ask and answer questions here.
-19
View File
@@ -1,19 +0,0 @@
<!--
Thank you for your pull request!
Please read the Code changes section of the CONTRIBUTING.md file,
and make sure you have followed the instructions.
https://github.com/pelletier/go-toml/blob/v2/CONTRIBUTING.md#code-changes
-->
Explanation of what this pull request does.
More detailed description of the decisions being made and the reasons why (if
the patch is non-trivial).
---
Paste `benchstat` results here
+2 -11
View File
@@ -1,17 +1,8 @@
version: 2
updates:
- package-ecosystem: gomod
directory: /
schedule:
interval: daily
open-pull-requests-limit: 10
- package-ecosystem: github-actions
directory: /
schedule:
interval: daily
open-pull-requests-limit: 10
- package-ecosystem: docker
directory: /
directory: "/"
schedule:
interval: daily
time: "13:00"
open-pull-requests-limit: 10
-21
View File
@@ -1,21 +0,0 @@
changelog:
exclude:
labels:
- build
- testing
categories:
- title: What's new
labels:
- feature
- title: Performance
labels:
- performance
- title: Fixed bugs
labels:
- bug
- title: Documentation
labels:
- doc
- title: Other changes
labels:
- "*"
-26
View File
@@ -1,26 +0,0 @@
name: CIFuzz
on: [pull_request]
jobs:
Fuzzing:
runs-on: ubuntu-latest
steps:
- name: Build Fuzzers
id: build
uses: google/oss-fuzz/infra/cifuzz/actions/build_fuzzers@master
with:
oss-fuzz-project-name: 'go-toml'
dry-run: false
language: go
- name: Run Fuzzers
uses: google/oss-fuzz/infra/cifuzz/actions/run_fuzzers@master
with:
oss-fuzz-project-name: 'go-toml'
fuzz-seconds: 300
dry-run: false
language: go
- name: Upload Crash
uses: actions/upload-artifact@v7
if: failure() && steps.build.outcome == 'success'
with:
name: artifacts
path: ./out/artifacts
+6 -6
View File
@@ -13,7 +13,7 @@ name: "CodeQL"
on:
push:
branches: [ master, v2 ]
branches: [ master ]
pull_request:
# The branches below must be a subset of the branches above
branches: [ master ]
@@ -35,11 +35,11 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@v6
uses: actions/checkout@v2
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v4
uses: github/codeql-action/init@v1
with:
languages: ${{ matrix.language }}
# If you wish to specify custom queries, you can do so here or in a config file.
@@ -47,10 +47,10 @@ jobs:
# Prefix the list here with "+" to use these queries and those in the config file.
# queries: ./path/to/local/query, your-org/your-repo/queries@main
# Autobuild attempts to build any compiled languages (C/C++, C#, or Java).
# Autobuild attempts to build any compiled languages (C/C++, C#, or Java).
# If this step fails, then you should remove it and run the build manually (see below)
- name: Autobuild
uses: github/codeql-action/autobuild@v4
uses: github/codeql-action/autobuild@v1
# ️ Command-line programs to run using the OS shell.
# 📚 https://git.io/JvXDl
@@ -64,4 +64,4 @@ jobs:
# make release
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v4
uses: github/codeql-action/analyze@v1
-20
View File
@@ -1,20 +0,0 @@
name: coverage
on:
pull_request:
branches:
- v2
jobs:
report:
runs-on: "ubuntu-latest"
name: report
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Setup go
uses: actions/setup-go@v6
with:
go-version: "1.26"
- name: Run tests with coverage
run: ./ci.sh coverage -d "${GITHUB_BASE_REF-HEAD}"
-36
View File
@@ -1,36 +0,0 @@
name: Go Versions Compatibility Test
on:
workflow_dispatch:
inputs:
go_versions:
description: 'Go versions to test (space-separated, e.g., "1.21 1.22 1.23")'
required: false
default: ''
type: string
jobs:
test:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
- name: Run Go versions compatibility test
run: |
VERSIONS="${{ github.event.inputs.go_versions }}"
./test-go-versions.sh --output ./test-results $VERSIONS
- name: Upload test results
uses: actions/upload-artifact@v7
with:
name: go-versions-test-results
path: |
test-results/
retention-days: 30
-22
View File
@@ -1,22 +0,0 @@
name: lint
on:
pull_request:
branches:
- v2
jobs:
golangci:
name: lint
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Setup go
uses: actions/setup-go@v6
with:
go-version: "1.26"
- name: Run golangci-lint
uses: golangci/golangci-lint-action@v9
with:
version: v2.8.0
-39
View File
@@ -1,39 +0,0 @@
name: release
on:
push:
tags:
- "v2.*"
workflow_call:
inputs:
args:
description: "Extra arguments to pass goreleaser"
default: ""
required: false
type: string
jobs:
release:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Set up Go
uses: actions/setup-go@v6
with:
go-version: "1.26"
- name: Login to GitHub Container Registry
uses: docker/login-action@v4
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Run GoReleaser
uses: goreleaser/goreleaser-action@v7
with:
distribution: goreleaser
version: '~> v2'
args: release ${{ inputs.args }} --clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-33
View File
@@ -1,33 +0,0 @@
name: test
on:
push:
branches:
- v2
pull_request:
branches:
- v2
jobs:
build:
strategy:
fail-fast: false
matrix:
os: [ 'ubuntu-latest', 'windows-latest', 'macos-latest', 'macos-14' ]
go: [ '1.25', '1.26' ]
runs-on: ${{ matrix.os }}
name: ${{ matrix.go }}/${{ matrix.os }}
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Setup go ${{ matrix.go }}
uses: actions/setup-go@v6
with:
go-version: ${{ matrix.go }}
- name: Run unit tests
run: go test -race ./...
release-check:
if: ${{ github.ref != 'refs/heads/v2' }}
uses: ./.github/workflows/release.yml
with:
args: --snapshot
-3
View File
@@ -3,6 +3,3 @@ fuzz/
cmd/tomll/tomll
cmd/tomljson/tomljson
cmd/tomltestgen/tomltestgen
dist
tests/
test-results
-76
View File
@@ -1,76 +0,0 @@
version = "2"
[linters]
default = "none"
enable = [
"asciicheck",
"bodyclose",
"dogsled",
"dupl",
"durationcheck",
"errcheck",
"errorlint",
"exhaustive",
"forbidigo",
"gochecknoinits",
"goconst",
"gocritic",
"godoclint",
"goheader",
"gomodguard",
"goprintffuncname",
"gosec",
"govet",
"importas",
"ineffassign",
"lll",
"makezero",
"mirror",
"misspell",
"nakedret",
"nilerr",
"noctx",
"nolintlint",
"perfsprint",
"prealloc",
"predeclared",
"revive",
"rowserrcheck",
"sqlclosecheck",
"staticcheck",
"thelper",
"tparallel",
"unconvert",
"unparam",
"unused",
"usetesting",
"wastedassign",
"whitespace",
]
[linters.settings.exhaustive]
default-signifies-exhaustive = true
[linters.settings.lll]
line-length = 150
[[linters.exclusions.rules]]
path = ".test.go"
linters = ["goconst", "gosec"]
[[linters.exclusions.rules]]
path = "main.go"
linters = ["forbidigo"]
[[linters.exclusions.rules]]
path = "internal"
linters = ["revive"]
text = "(exported|indent-error-flow): "
[formatters]
enable = [
"gci",
"gofmt",
"gofumpt",
"goimports",
]
-124
View File
@@ -1,124 +0,0 @@
version: 2
before:
hooks:
- go mod tidy
- go fmt ./...
- go test ./...
builds:
- id: tomll
main: ./cmd/tomll
binary: tomll
env:
- CGO_ENABLED=0
flags:
- -trimpath
ldflags:
- -X main.version={{.Version}} -X main.commit={{.Commit}} -X main.date={{.CommitDate}}
mod_timestamp: '{{ .CommitTimestamp }}'
targets:
- linux_amd64
- linux_arm64
- linux_arm
- linux_riscv64
- windows_amd64
- windows_arm64
- darwin_amd64
- darwin_arm64
- id: tomljson
main: ./cmd/tomljson
binary: tomljson
env:
- CGO_ENABLED=0
flags:
- -trimpath
ldflags:
- -X main.version={{.Version}} -X main.commit={{.Commit}} -X main.date={{.CommitDate}}
mod_timestamp: '{{ .CommitTimestamp }}'
targets:
- linux_amd64
- linux_arm64
- linux_arm
- linux_riscv64
- windows_amd64
- windows_arm64
- darwin_amd64
- darwin_arm64
- id: jsontoml
main: ./cmd/jsontoml
binary: jsontoml
env:
- CGO_ENABLED=0
flags:
- -trimpath
ldflags:
- -X main.version={{.Version}} -X main.commit={{.Commit}} -X main.date={{.CommitDate}}
mod_timestamp: '{{ .CommitTimestamp }}'
targets:
- linux_amd64
- linux_arm64
- linux_riscv64
- linux_arm
- windows_amd64
- windows_arm64
- darwin_amd64
- darwin_arm64
universal_binaries:
- id: tomll
replace: true
name_template: tomll
- id: tomljson
replace: true
name_template: tomljson
- id: jsontoml
replace: true
name_template: jsontoml
archives:
- id: jsontoml
format: tar.xz
builds:
- jsontoml
files:
- none*
name_template: "{{ .Binary }}_{{.Version}}_{{ .Os }}_{{ .Arch }}"
- id: tomljson
format: tar.xz
builds:
- tomljson
files:
- none*
name_template: "{{ .Binary }}_{{.Version}}_{{ .Os }}_{{ .Arch }}"
- id: tomll
format: tar.xz
builds:
- tomll
files:
- none*
name_template: "{{ .Binary }}_{{.Version}}_{{ .Os }}_{{ .Arch }}"
dockers:
- id: tools
goos: linux
goarch: amd64
ids:
- jsontoml
- tomljson
- tomll
image_templates:
- "ghcr.io/pelletier/go-toml:latest"
- "ghcr.io/pelletier/go-toml:{{ .Tag }}"
- "ghcr.io/pelletier/go-toml:v{{ .Major }}"
skip_push: false
checksum:
name_template: 'sha256sums.txt'
snapshot:
version_template: "{{ incpatch .Version }}-next"
release:
github:
owner: pelletier
name: go-toml
draft: true
prerelease: auto
mode: replace
changelog:
use: github-native
announce:
skip: true
-64
View File
@@ -1,64 +0,0 @@
# Agent Guidelines for go-toml
This file provides guidelines for AI agents contributing to go-toml. All agents must follow these rules derived from [CONTRIBUTING.md](./CONTRIBUTING.md).
## Project Overview
go-toml is a TOML library for Go. The goal is to provide an easy-to-use and efficient TOML implementation that gets the job done without getting in the way.
## Code Change Rules
### Backward Compatibility
- **No backward-incompatible changes** unless explicitly discussed and approved
- Avoid breaking people's programs unless absolutely necessary
### Testing Requirements
- **All bug fixes must include regression tests**
- **All new code must be tested**
- Run tests before submitting: `go test -race ./...`
- Test coverage must not decrease. Check with:
```bash
go test -covermode=atomic -coverprofile=coverage.out
go tool cover -func=coverage.out
```
- All lines of code touched by changes should be covered by tests
### Performance Requirements
- go-toml aims to stay efficient; avoid performance regressions
- Run benchmarks to verify: `go test ./... -bench=. -count=10`
- Compare results using [benchstat](https://pkg.go.dev/golang.org/x/perf/cmd/benchstat)
### Documentation
- New features or feature extensions must include documentation
- Documentation lives in [README.md](./README.md) and throughout source code
### Code Style
- Follow existing code format and structure
- Code must pass `go fmt`
- Code must pass linting with the same golangci-lint version as CI (see version in `.github/workflows/lint.yml`):
```bash
# Install specific version (check lint.yml for current version)
curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/HEAD/install.sh | sh -s -- -b $(go env GOPATH)/bin <version>
# Run linter
golangci-lint run ./...
```
### Commit Messages
- Commit messages must explain **why** the change is needed
- Keep messages clear and informative even if details are in the PR description
## Pull Request Checklist
Before submitting:
1. Tests pass (`go test -race ./...`)
2. No backward-incompatible changes (unless discussed)
3. Relevant documentation added/updated
4. No performance regression (verify with benchmarks)
5. Title is clear and understandable for changelog
+72 -175
View File
@@ -1,74 +1,74 @@
# Contributing
## Contributing
Thank you for your interest in go-toml! We appreciate you considering
contributing to go-toml!
The main goal is the project is to provide an easy-to-use and efficient TOML
implementation for Go that gets the job done and gets out of your way dealing
with TOML is probably not the central piece of your project.
The main goal is the project is to provide an easy-to-use TOML
implementation for Go that gets the job done and gets out of your way
dealing with TOML is probably not the central piece of your project.
As the single maintainer of go-toml, time is scarce. All help, big or small, is
more than welcomed!
As the single maintainer of go-toml, time is scarce. All help, big or
small, is more than welcomed!
## Ask questions
### Ask questions
Any question you may have, somebody else might have it too. Always feel free to
ask them on the [discussion board][discussions]. We will try to answer them as
clearly and quickly as possible, time permitting.
Any question you may have, somebody else might have it too. Always feel
free to ask them on the [issues tracker][issues-tracker]. We will try to
answer them as clearly and quickly as possible, time permitting.
Asking questions also helps us identify areas where the documentation needs
improvement, or new features that weren't envisioned before. Sometimes, a
seemingly innocent question leads to the fix of a bug. Don't hesitate and ask
away!
seemingly innocent question leads to the fix of a bug. Don't hesitate and
ask away!
[discussions]: https://github.com/pelletier/go-toml/discussions
### Improve the documentation
## Improve the documentation
The best way to share your knowledge and experience with go-toml is to
improve the documentation. Fix a typo, clarify an interface, add an
example, anything goes!
The best way to share your knowledge and experience with go-toml is to improve
the documentation. Fix a typo, clarify an interface, add an example, anything
goes!
The documentation is present in the [README][readme] and thorough the
source code. On release, it gets updated on [pkg.go.dev][pkg.go.dev]. To make a
change to the documentation, create a pull request with your proposed
changes. For simple changes like that, the easiest way to go is probably
the "Fork this project and edit the file" button on Github, displayed at
the top right of the file. Unless it's a trivial change (for example a
typo), provide a little bit of context in your pull request description or
commit message.
The documentation is present in the [README][readme] and thorough the source
code. On release, it gets updated on [pkg.go.dev][pkg.go.dev]. To make a change
to the documentation, create a pull request with your proposed changes. For
simple changes like that, the easiest way to go is probably the "Fork this
project and edit the file" button on GitHub, displayed at the top right of the
file. Unless it's a trivial change (for example a typo), provide a little bit of
context in your pull request description or commit message.
### Report a bug
## Report a bug
Found a bug! Sorry to hear that :(. Help us and other track them down and
fix by reporting it. [File a new bug report][bug-report] on the [issues
tracker][issues-tracker]. The template should provide enough guidance on
what to include. When in doubt: add more details! By reducing ambiguity and
providing more information, it decreases back and forth and saves everyone
time.
Found a bug! Sorry to hear that :(. Help us and other track them down and fix by
reporting it. [File a new bug report][bug-report] on the [issues
tracker][issues-tracker]. The template should provide enough guidance on what to
include. When in doubt: add more details! By reducing ambiguity and providing
more information, it decreases back and forth and saves everyone time.
## Code changes
### Code changes
Want to contribute a patch? Very happy to hear that!
First, some high-level rules:
- A short proposal with some POC code is better than a lengthy piece of text
with no code. Code speaks louder than words. That being said, bigger changes
should probably start with a [discussion][discussions].
- No backward-incompatible patch will be accepted unless discussed. Sometimes
it's hard, but we try not to break people's programs unless we absolutely have
* A short proposal with some POC code is better than a lengthy piece of
text with no code. Code speaks louder than words.
* No backward-incompatible patch will be accepted unless discussed.
Sometimes it's hard, and Go's lack of versioning by default does not
help, but we try not to break people's programs unless we absolutely have
to.
- If you are writing a new feature or extending an existing one, make sure to
write some documentation.
- Bug fixes need to be accompanied with regression tests.
- New code needs to be tested.
- Your commit messages need to explain why the change is needed, even if already
included in the PR description.
* If you are writing a new feature or extending an existing one, make sure
to write some documentation.
* Bug fixes need to be accompanied with regression tests.
* New code needs to be tested.
* Your commit messages need to explain why the change is needed, even if
already included in the PR description.
It does sound like a lot, but those best practices are here to save time overall
and continuously improve the quality of the project, which is something everyone
benefits from.
It does sound like a lot, but those best practices are here to save time
overall and continuously improve the quality of the project, which is
something everyone benefits from.
### Get started
#### Get started
The fairly standard code contribution process looks like that:
@@ -76,153 +76,50 @@ The fairly standard code contribution process looks like that:
2. Make your changes, commit on any branch you like.
3. [Open up a pull request][pull-request]
4. Review, potential ask for changes.
5. Merge.
5. Merge. You're in!
Feel free to ask for help! You can create draft pull requests to gather
some early feedback!
### Run the tests
#### Run the tests
You can run tests for go-toml using Go's test tool: `go test -race ./...`.
You can run tests for go-toml using Go's test tool: `go test ./...`.
When creating a pull requests, all tests will be ran on Linux on a few Go
versions (Travis CI), and on Windows using the latest Go version
(AppVeyor).
During the pull request process, all tests will be ran on Linux, Windows, and
MacOS on the last two versions of Go.
#### Style
However, given GitHub's new policy to _not_ run Actions on pull requests until a
maintainer clicks on button, it is highly recommended that you run them locally
as you make changes.
### Test across Go versions
The repository includes tooling to test go-toml across multiple Go versions
(1.11 through 1.25) both locally and in GitHub Actions.
#### Local testing with Docker
Prerequisites: Docker installed and running, Bash shell, `rsync` command.
```bash
# Test all Go versions in parallel (default)
./test-go-versions.sh
# Test specific versions
./test-go-versions.sh 1.21 1.22 1.23
# Test sequentially (slower but uses less resources)
./test-go-versions.sh --sequential
# Verbose output with custom results directory
./test-go-versions.sh --verbose --output ./my-results 1.24 1.25
# Show all options
./test-go-versions.sh --help
```
The script creates Docker containers for each Go version and runs the full test
suite. Results are saved to a `test-results/` directory with individual logs and
a comprehensive summary report.
The script only exits with a non-zero status code if either of the two most
recent Go versions fail.
#### GitHub Actions testing (maintainers)
1. Go to the **Actions** tab in the GitHub repository
2. Select **"Go Versions Compatibility Test"** from the workflow list
3. Click **"Run workflow"**
4. Optionally customize:
- **Go versions**: Space-separated list (e.g., `1.21 1.22 1.23`)
- **Execution mode**: Parallel (faster) or sequential (more stable)
### Check coverage
We use `go tool cover` to compute test coverage. Most code editors have a way to
run and display code coverage, but at the end of the day, we do this:
```
go test -covermode=atomic -coverprofile=coverage.out
go tool cover -func=coverage.out
```
and verify that the overall percentage of tested code does not go down. This is
a requirement. As a rule of thumb, all lines of code touched by your changes
should be covered. On Unix you can use `./ci.sh coverage -d v2` to check if your
code lowers the coverage.
### Verify performance
Go-toml aims to stay efficient. We rely on a set of scenarios executed with Go's
builtin benchmark systems. Because of their noisy nature, containers provided by
GitHub Actions cannot be reliably used for benchmarking. As a result, you are
responsible for checking that your changes do not incur a performance penalty.
You can run their following to execute benchmarks:
```
go test ./... -bench=. -count=10
```
Benchmark results should be compared against each other with
[benchstat][benchstat]. Typical flow looks like this:
1. On the `v2` branch, run `go test ./... -bench=. -count 10` and save output to
a file (for example `old.txt`).
2. Make some code changes.
3. Run `go test ....` again, and save the output to an other file (for example
`new.txt`).
4. Run `benchstat old.txt new.txt` to check that time/op does not go up in any
test.
On Unix you can use `./ci.sh benchmark -d v2` to verify how your code impacts
performance.
It is highly encouraged to add the benchstat results to your pull request
description. Pull requests that lower performance will receive more scrutiny.
[benchstat]: https://pkg.go.dev/golang.org/x/perf/cmd/benchstat
### Style
Try to look around and follow the same format and structure as the rest of the
code. We enforce using `go fmt` on the whole code base.
Try to look around and follow the same format and structure as the rest of
the code. We enforce using `go fmt` on the whole code base.
---
## Maintainers-only
### Maintainers-only
### Merge pull request
#### Merge pull request
Checklist:
- Passing CI.
- Does not introduce backward-incompatible changes (unless discussed).
- Has relevant doc changes.
- Benchstat does not show performance regression.
- Pull request is [labeled appropriately][pr-labels].
- Title will be understandable in the changelog.
* Passing CI.
* Does not introduce backward-incompatible changes (unless discussed).
* Has relevant doc changes.
* Has relevant unit tests.
1. Merge using "squash and merge".
2. Make sure to edit the commit message to keep all the useful information
nice and clean.
3. Make sure the commit title is clear and contains the PR number (#123).
### New release
1. Decide on the next version number. Use semver. Review commits since last
version to assess.
2. Tag release. For example:
```
git checkout v2
git pull
git tag v2.2.0
git push --tags
```
3. CI automatically builds a draft GitHub release. Review it and edit as
necessary. Look for "Other changes". That would indicate a pull request not
labeled properly. Tweak labels and pull request titles until changelog looks
good for users.
4. Check "create discussion" box, in the "Releases" category.
5. If new version is an alpha or beta only, check pre-release box.
#### New release
1. Go to [releases][releases]. Click on "X commits to master since this
release".
2. Make note of all the changes. Look for backward incompatible changes,
new features, and bug fixes.
3. Pick the new version using the above and semver.
4. Create a [new release][new-release].
5. Follow the same format as [1.1.0][release-110].
[issues-tracker]: https://github.com/pelletier/go-toml/issues
[bug-report]: https://github.com/pelletier/go-toml/issues/new?template=bug_report.md
@@ -230,6 +127,6 @@ Checklist:
[readme]: ./README.md
[fork]: https://help.github.com/articles/fork-a-repo
[pull-request]: https://help.github.com/en/articles/creating-a-pull-request
[releases]: https://github.com/pelletier/go-toml/releases
[new-release]: https://github.com/pelletier/go-toml/releases/new
[gh]: https://github.com/cli/cli
[pr-labels]: https://github.com/pelletier/go-toml/blob/v2/.github/release.yml
[release-110]: https://github.com/pelletier/go-toml/releases/tag/v1.1.0
+10 -4
View File
@@ -1,5 +1,11 @@
FROM golang:1.12-alpine3.9 as builder
WORKDIR /go/src/github.com/pelletier/go-toml
COPY . .
ENV CGO_ENABLED=0
ENV GOOS=linux
RUN go install ./...
FROM scratch
ENV PATH "$PATH:/bin"
COPY tomll /bin/tomll
COPY tomljson /bin/tomljson
COPY jsontoml /bin/jsontoml
COPY --from=builder /go/bin/tomll /usr/bin/tomll
COPY --from=builder /go/bin/tomljson /usr/bin/tomljson
COPY --from=builder /go/bin/jsontoml /usr/bin/jsontoml
+227 -2
View File
@@ -1,7 +1,16 @@
The bulk of github.com/pelletier/go-toml is distributed under the MIT license
(see below), with the exception of localtime.go and localtime.test.go.
Those two files have been copied over from Google's civil library at revision
ed46f5086358513cf8c25f8e3f022cb838a49d66, and are distributed under the Apache
2.0 license (see below).
github.com/pelletier/go-toml:
The MIT License (MIT)
go-toml v2
Copyright (c) 2021 - 2023 Thomas Pelletier
Copyright (c) 2013 - 2021 Thomas Pelletier, Eric Anderton
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
@@ -20,3 +29,219 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
localtime.go, localtime_test.go:
Originals:
https://raw.githubusercontent.com/googleapis/google-cloud-go/ed46f5086358513cf8c25f8e3f022cb838a49d66/civil/civil.go
https://raw.githubusercontent.com/googleapis/google-cloud-go/ed46f5086358513cf8c25f8e3f022cb838a49d66/civil/civil_test.go
Changes:
* Renamed files from civil* to localtime*.
* Package changed from civil to toml.
* 'Local' prefix added to all structs.
License:
https://raw.githubusercontent.com/googleapis/google-cloud-go/ed46f5086358513cf8c25f8e3f022cb838a49d66/LICENSE
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
+29
View File
@@ -0,0 +1,29 @@
export CGO_ENABLED=0
go := go
go.goos ?= $(shell echo `go version`|cut -f4 -d ' '|cut -d '/' -f1)
go.goarch ?= $(shell echo `go version`|cut -f4 -d ' '|cut -d '/' -f2)
out.tools := tomll tomljson jsontoml
out.dist := $(out.tools:=_$(go.goos)_$(go.goarch).tar.xz)
sources := $(wildcard **/*.go)
.PHONY:
tools: $(out.tools)
$(out.tools): $(sources)
GOOS=$(go.goos) GOARCH=$(go.goarch) $(go) build ./cmd/$@
.PHONY:
dist: $(out.dist)
$(out.dist):%_$(go.goos)_$(go.goarch).tar.xz: %
if [ "$(go.goos)" = "windows" ]; then \
tar -cJf $@ $^.exe; \
else \
tar -cJf $@ $^; \
fi
.PHONY:
clean:
rm -rf $(out.tools) $(out.dist)
+5
View File
@@ -0,0 +1,5 @@
**Issue:** add link to pelletier/go-toml issue here
Explanation of what this pull request does.
More detailed description of the decisions being made and the reasons why (if the patch is non-trivial).
+129 -576
View File
@@ -1,636 +1,189 @@
# go-toml v2
# go-toml
Go library for the [TOML](https://toml.io/en/) format.
Go library for the [TOML](https://toml.io/) format.
This library supports [TOML v1.0.0](https://toml.io/en/v1.0.0).
[🐞 Bug Reports](https://github.com/pelletier/go-toml/issues)
⚠️ This readme is for go-toml v1. As for 2022-04-27,
[go-toml v2](https://github.com/pelletier/go-toml/tree/v2) has been released.
[💬 Anything else](https://github.com/pelletier/go-toml/discussions)
The new version contains tons of bug fixes, is much faster, and more
importantly maintained. You are strongly encouraged to use it instead of v1!
## Documentation
[👉 go-toml v2](https://github.com/pelletier/go-toml/tree/v2)
Full API, examples, and implementation notes are available in the Go
documentation.
v1 will not receive any updates.
[![Go Reference](https://pkg.go.dev/badge/github.com/pelletier/go-toml/v2.svg)](https://pkg.go.dev/github.com/pelletier/go-toml/v2)
---
This library supports TOML version
[v1.0.0-rc.3](https://toml.io/en/v1.0.0-rc.3)
[![Go Reference](https://pkg.go.dev/badge/github.com/pelletier/go-toml.svg)](https://pkg.go.dev/github.com/pelletier/go-toml)
[![license](https://img.shields.io/github/license/pelletier/go-toml.svg)](https://github.com/pelletier/go-toml/blob/master/LICENSE)
[![Build Status](https://dev.azure.com/pelletierthomas/go-toml-ci/_apis/build/status/pelletier.go-toml?branchName=master)](https://dev.azure.com/pelletierthomas/go-toml-ci/_build/latest?definitionId=1&branchName=master)
[![codecov](https://codecov.io/gh/pelletier/go-toml/branch/master/graph/badge.svg)](https://codecov.io/gh/pelletier/go-toml)
[![Go Report Card](https://goreportcard.com/badge/github.com/pelletier/go-toml)](https://goreportcard.com/report/github.com/pelletier/go-toml)
[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fpelletier%2Fgo-toml.svg?type=shield)](https://app.fossa.io/projects/git%2Bgithub.com%2Fpelletier%2Fgo-toml?ref=badge_shield)
## Development status
**️ Consider go-toml v2!**
The next version of go-toml is in [active development][v2-dev], and
[nearing completion][v2-map].
Though technically in beta, v2 is already more tested, [fixes bugs][v1-bugs],
and [much faster][v2-bench]. If you only need reading and writing TOML documents
(majority of cases), those features are implemented and the API unlikely to
change.
The remaining features will be added shortly. While pull-requests are welcome on
v1, no active development is expected on it. When v2.0.0 is released, v1 will be
deprecated.
👉 [go-toml v2][v2]
[v2]: https://github.com/pelletier/go-toml/tree/v2
[v2-map]: https://github.com/pelletier/go-toml/discussions/506
[v2-dev]: https://github.com/pelletier/go-toml/tree/v2
[v1-bugs]: https://github.com/pelletier/go-toml/issues?q=is%3Aissue+is%3Aopen+label%3Av2-fixed
[v2-bench]: https://github.com/pelletier/go-toml/tree/v2#benchmarks
## Features
Go-toml provides the following features for using data parsed from TOML documents:
* Load TOML documents from files and string data
* Easily navigate TOML structure using Tree
* Marshaling and unmarshaling to and from data structures
* Line & column position data for all parsed elements
* [Query support similar to JSON-Path](query/)
* Syntax errors contain line and column numbers
## Import
```go
import "github.com/pelletier/go-toml/v2"
import "github.com/pelletier/go-toml"
```
See [Modules](#Modules).
## Usage example
## Features
### Stdlib behavior
As much as possible, this library is designed to behave similarly as the
standard library's `encoding/json`.
### Performance
While go-toml favors usability, it is written with performance in mind. Most
operations should not be shockingly slow. See [benchmarks](#benchmarks).
### Strict mode
`Decoder` can be set to "strict mode", which makes it error when some parts of
the TOML document was not present in the target structure. This is a great way
to check for typos. [See example in the documentation][strict].
[strict]: https://pkg.go.dev/github.com/pelletier/go-toml/v2#example-Decoder.DisallowUnknownFields
### Contextualized errors
When most decoding errors occur, go-toml returns [`DecodeError`][decode-err],
which contains a human readable contextualized version of the error. For
example:
```
1| [server]
2| path = 100
| ~~~ cannot decode TOML integer into struct field toml_test.Server.Path of type string
3| port = 50
```
[decode-err]: https://pkg.go.dev/github.com/pelletier/go-toml/v2#DecodeError
### Local date and time support
TOML supports native [local date/times][ldt]. It allows to represent a given
date, time, or date-time without relation to a timezone or offset. To support
this use-case, go-toml provides [`LocalDate`][tld], [`LocalTime`][tlt], and
[`LocalDateTime`][tldt]. Those types can be transformed to and from `time.Time`,
making them convenient yet unambiguous structures for their respective TOML
representation.
[ldt]: https://toml.io/en/v1.0.0#local-date-time
[tld]: https://pkg.go.dev/github.com/pelletier/go-toml/v2#LocalDate
[tlt]: https://pkg.go.dev/github.com/pelletier/go-toml/v2#LocalTime
[tldt]: https://pkg.go.dev/github.com/pelletier/go-toml/v2#LocalDateTime
### Commented config
Since TOML is often used for configuration files, go-toml can emit documents
annotated with [comments and commented-out values][comments-example]. For
example, it can generate the following file:
```toml
# Host IP to connect to.
host = '127.0.0.1'
# Port of the remote server.
port = 4242
# Encryption parameters (optional)
# [TLS]
# cipher = 'AEAD-AES128-GCM-SHA256'
# version = 'TLS 1.3'
```
[comments-example]: https://pkg.go.dev/github.com/pelletier/go-toml/v2#example-Marshal-Commented
## Getting started
Given the following struct, let's see how to read it and write it as TOML:
Read a TOML document:
```go
type MyConfig struct {
Version int
Name string
Tags []string
}
config, _ := toml.Load(`
[postgres]
user = "pelletier"
password = "mypassword"`)
// retrieve data directly
user := config.Get("postgres.user").(string)
// or using an intermediate object
postgresConfig := config.Get("postgres").(*toml.Tree)
password := postgresConfig.Get("password").(string)
```
### Unmarshaling
[`Unmarshal`][unmarshal] reads a TOML document and fills a Go structure with its
content.
Note that the struct variable names are _capitalized_, while the variables in the toml document are _lowercase_.
For example:
Or use Unmarshal:
```go
doc := `
version = 2
name = "go-toml"
tags = ["go", "toml"]
`
var cfg MyConfig
err := toml.Unmarshal([]byte(doc), &cfg)
if err != nil {
panic(err)
type Postgres struct {
User string
Password string
}
type Config struct {
Postgres Postgres
}
fmt.Println("version:", cfg.Version)
fmt.Println("name:", cfg.Name)
fmt.Println("tags:", cfg.Tags)
// Output:
// version: 2
// name: go-toml
// tags: [go toml]
doc := []byte(`
[Postgres]
User = "pelletier"
Password = "mypassword"`)
config := Config{}
toml.Unmarshal(doc, &config)
fmt.Println("user=", config.Postgres.User)
```
[unmarshal]: https://pkg.go.dev/github.com/pelletier/go-toml/v2#Unmarshal
Here is an example using tables with some simple nesting:
Or use a query:
```go
doc := `
age = 45
fruits = ["apple", "pear"]
# these are very important!
[my-variables]
first = 1
second = 0.2
third = "abc"
# this is not so important.
[my-variables.b]
bfirst = 123
`
var Document struct {
Age int
Fruits []string
Myvariables struct {
First int
Second float64
Third string
B struct {
Bfirst int
}
} `toml:"my-variables"`
// use a query to gather elements without walking the tree
q, _ := query.Compile("$..[user,password]")
results := q.Execute(config)
for ii, item := range results.Values() {
fmt.Printf("Query result %d: %v\n", ii, item)
}
err := toml.Unmarshal([]byte(doc), &Document)
if err != nil {
panic(err)
}
fmt.Println("age:", Document.Age)
fmt.Println("fruits:", Document.Fruits)
fmt.Println("my-variables.first:", Document.Myvariables.First)
fmt.Println("my-variables.second:", Document.Myvariables.Second)
fmt.Println("my-variables.third:", Document.Myvariables.Third)
fmt.Println("my-variables.B.Bfirst:", Document.Myvariables.B.Bfirst)
// Output:
// age: 45
// fruits: [apple pear]
// my-variables.first: 1
// my-variables.second: 0.2
// my-variables.third: abc
// my-variables.B.Bfirst: 123
```
## Documentation
### Marshaling
[`Marshal`][marshal] is the opposite of Unmarshal: it represents a Go structure
as a TOML document:
```go
cfg := MyConfig{
Version: 2,
Name: "go-toml",
Tags: []string{"go", "toml"},
}
b, err := toml.Marshal(cfg)
if err != nil {
panic(err)
}
fmt.Println(string(b))
// Output:
// Version = 2
// Name = 'go-toml'
// Tags = ['go', 'toml']
```
[marshal]: https://pkg.go.dev/github.com/pelletier/go-toml/v2#Marshal
## Unstable API
This API does not yet follow the backward compatibility guarantees of this
library. They provide early access to features that may have rough edges or an
API subject to change.
### Parser
Parser is the unstable API that allows iterative parsing of a TOML document at
the AST level. See https://pkg.go.dev/github.com/pelletier/go-toml/v2/unstable.
## Benchmarks
Execution time speedup compared to other Go TOML libraries:
<table>
<thead>
<tr><th>Benchmark</th><th>go-toml v1</th><th>BurntSushi/toml</th></tr>
</thead>
<tbody>
<tr><td>Marshal/HugoFrontMatter-2</td><td>2.1x</td><td>2.0x</td></tr>
<tr><td>Marshal/ReferenceFile/map-2</td><td>2.0x</td><td>2.0x</td></tr>
<tr><td>Marshal/ReferenceFile/struct-2</td><td>2.3x</td><td>2.5x</td></tr>
<tr><td>Unmarshal/HugoFrontMatter-2</td><td>3.3x</td><td>2.8x</td></tr>
<tr><td>Unmarshal/ReferenceFile/map-2</td><td>2.9x</td><td>3.0x</td></tr>
<tr><td>Unmarshal/ReferenceFile/struct-2</td><td>4.8x</td><td>5.0x</td></tr>
</tbody>
</table>
<details><summary>See more</summary>
<p>The table above has the results of the most common use-cases. The table below
contains the results of all benchmarks, including unrealistic ones. It is
provided for completeness.</p>
<table>
<thead>
<tr><th>Benchmark</th><th>go-toml v1</th><th>BurntSushi/toml</th></tr>
</thead>
<tbody>
<tr><td>Marshal/SimpleDocument/map-2</td><td>2.0x</td><td>2.9x</td></tr>
<tr><td>Marshal/SimpleDocument/struct-2</td><td>2.5x</td><td>3.6x</td></tr>
<tr><td>Unmarshal/SimpleDocument/map-2</td><td>4.2x</td><td>3.4x</td></tr>
<tr><td>Unmarshal/SimpleDocument/struct-2</td><td>5.9x</td><td>4.4x</td></tr>
<tr><td>UnmarshalDataset/example-2</td><td>3.2x</td><td>2.9x</td></tr>
<tr><td>UnmarshalDataset/code-2</td><td>2.4x</td><td>2.8x</td></tr>
<tr><td>UnmarshalDataset/twitter-2</td><td>2.7x</td><td>2.5x</td></tr>
<tr><td>UnmarshalDataset/citm_catalog-2</td><td>2.3x</td><td>2.3x</td></tr>
<tr><td>UnmarshalDataset/canada-2</td><td>1.9x</td><td>1.5x</td></tr>
<tr><td>UnmarshalDataset/config-2</td><td>5.4x</td><td>3.0x</td></tr>
<tr><td>geomean</td><td>2.9x</td><td>2.8x</td></tr>
</tbody>
</table>
<p>This table can be generated with <code>./ci.sh benchmark -a -html</code>.</p>
</details>
## Modules
go-toml uses Go's standard modules system.
Installation instructions:
- Go ≥ 1.16: Nothing to do. Use the import in your code. The `go` command deals
with it automatically.
- Go ≥ 1.13: `GO111MODULE=on go get github.com/pelletier/go-toml/v2`.
In case of trouble: [Go Modules FAQ][mod-faq].
[mod-faq]: https://github.com/golang/go/wiki/Modules#why-does-installing-a-tool-via-go-get-fail-with-error-cannot-find-main-module
The documentation and additional examples are available at
[pkg.go.dev](https://pkg.go.dev/github.com/pelletier/go-toml).
## Tools
Go-toml provides three handy command line tools:
* `tomljson`: Reads a TOML file and outputs its JSON representation.
* `tomll`: Reads TOML files and lints them.
```
$ go install github.com/pelletier/go-toml/v2/cmd/tomljson@latest
$ tomljson --help
```
```
go install github.com/pelletier/go-toml/cmd/tomll
tomll --help
```
* `tomljson`: Reads a TOML file and outputs its JSON representation.
```
go install github.com/pelletier/go-toml/cmd/tomljson
tomljson --help
```
* `jsontoml`: Reads a JSON file and outputs a TOML representation.
```
$ go install github.com/pelletier/go-toml/v2/cmd/jsontoml@latest
$ jsontoml --help
```
* `tomll`: Lints and reformats a TOML file.
```
$ go install github.com/pelletier/go-toml/v2/cmd/tomll@latest
$ tomll --help
```
```
go install github.com/pelletier/go-toml/cmd/jsontoml
jsontoml --help
```
### Docker image
Those tools are also available as a [Docker image][docker]. For example, to use
`tomljson`:
Those tools are also available as a Docker image from
[dockerhub](https://hub.docker.com/r/pelletier/go-toml). For example, to
use `tomljson`:
```
docker run -i ghcr.io/pelletier/go-toml:v2 tomljson < example.toml
docker run -v $PWD:/workdir pelletier/go-toml tomljson /workdir/example.toml
```
Multiple versions are available on [ghcr.io][docker].
Only master (`latest`) and tagged versions are published to dockerhub. You
can build your own image as usual:
[docker]: https://github.com/pelletier/go-toml/pkgs/container/go-toml
## Migrating from v1
This section describes the differences between v1 and v2, with some pointers on
how to get the original behavior when possible.
### Decoding / Unmarshal
#### Automatic field name guessing
When unmarshaling to a struct, if a key in the TOML document does not exactly
match the name of a struct field or any of the `toml`-tagged field, v1 tries
multiple variations of the key ([code][v1-keys]).
V2 instead does a case-insensitive matching, like `encoding/json`.
This could impact you if you are relying on casing to differentiate two fields,
and one of them is a not using the `toml` struct tag. The recommended solution
is to be specific about tag names for those fields using the `toml` struct tag.
[v1-keys]: https://github.com/pelletier/go-toml/blob/a2e52561804c6cd9392ebf0048ca64fe4af67a43/marshal.go#L775-L781
#### Ignore preexisting value in interface
When decoding into a non-nil `interface{}`, go-toml v1 uses the type of the
element in the interface to decode the object. For example:
```go
type inner struct {
B interface{}
}
type doc struct {
A interface{}
}
d := doc{
A: inner{
B: "Before",
},
}
data := `
[A]
B = "After"
`
toml.Unmarshal([]byte(data), &d)
fmt.Printf("toml v1: %#v\n", d)
// toml v1: main.doc{A:main.inner{B:"After"}}
```
docker build -t go-toml .
```
In this case, field `A` is of type `interface{}`, containing a `inner` struct.
V1 sees that type and uses it when decoding the object.
## Contribute
When decoding an object into an `interface{}`, V2 instead disregards whatever
value the `interface{}` may contain and replaces it with a
`map[string]interface{}`. With the same data structure as above, here is what
the result looks like:
Feel free to report bugs and patches using GitHub's pull requests system on
[pelletier/go-toml](https://github.com/pelletier/go-toml). Any feedback would be
much appreciated!
```go
toml.Unmarshal([]byte(data), &d)
fmt.Printf("toml v2: %#v\n", d)
### Run tests
// toml v2: main.doc{A:map[string]interface {}{"B":"After"}}
```
`go test ./...`
This is to match `encoding/json`'s behavior. There is no way to make the v2
decoder behave like v1.
### Fuzzing
#### Values out of array bounds ignored
When decoding into an array, v1 returns an error when the number of elements
contained in the doc is superior to the capacity of the array. For example:
```go
type doc struct {
A [2]string
}
d := doc{}
err := toml.Unmarshal([]byte(`A = ["one", "two", "many"]`), &d)
fmt.Println(err)
// (1, 1): unmarshal: TOML array length (3) exceeds destination array length (2)
```
In the same situation, v2 ignores the last value:
```go
err := toml.Unmarshal([]byte(`A = ["one", "two", "many"]`), &d)
fmt.Println("err:", err, "d:", d)
// err: <nil> d: {[one two]}
```
This is to match `encoding/json`'s behavior. There is no way to make the v2
decoder behave like v1.
#### Support for `toml.Unmarshaler` has been dropped
This method was not widely used, poorly defined, and added a lot of complexity.
A similar effect can be achieved by implementing the `encoding.TextUnmarshaler`
interface and use strings.
#### Support for `default` struct tag has been dropped
This feature adds complexity and a poorly defined API for an effect that can be
accomplished outside of the library.
It does not seem like other format parsers in Go support that feature (the
project referenced in the original ticket #202 has not been updated since 2017).
Given that go-toml v2 should not touch values not in the document, the same
effect can be achieved by pre-filling the struct with defaults (libraries like
[go-defaults][go-defaults] can help). Also, string representation is not well
defined for all types: it creates issues like #278.
The recommended replacement is pre-filling the struct before unmarshaling.
[go-defaults]: https://github.com/mcuadros/go-defaults
#### `toml.Tree` replacement
This structure was the initial attempt at providing a document model for
go-toml. It allows manipulating the structure of any document, encoding and
decoding from their TOML representation. While a more robust feature was
initially planned in go-toml v2, this has been ultimately [removed from
scope][nodoc] of this library, with no plan to add it back at the moment. The
closest equivalent at the moment would be to unmarshal into an `interface{}` and
use type assertions and/or reflection to manipulate the arbitrary
structure. However this would fall short of providing all of the TOML features
such as adding comments and be specific about whitespace.
#### `toml.Position` are not retrievable anymore
The API for retrieving the position (line, column) of a specific TOML element do
not exist anymore. This was done to minimize the amount of concepts introduced
by the library (query path), and avoid the performance hit related to storing
positions in the absence of a document model, for a feature that seemed to have
little use. Errors however have gained more detailed position
information. Position retrieval seems better fitted for a document model, which
has been [removed from the scope][nodoc] of go-toml v2 at the moment.
### Encoding / Marshal
#### Default struct fields order
V1 emits struct fields order alphabetically by default. V2 struct fields are
emitted in order they are defined. For example:
```go
type S struct {
B string
A string
}
data := S{
B: "B",
A: "A",
}
b, _ := tomlv1.Marshal(data)
fmt.Println("v1:\n" + string(b))
b, _ = tomlv2.Marshal(data)
fmt.Println("v2:\n" + string(b))
// Output:
// v1:
// A = "A"
// B = "B"
// v2:
// B = 'B'
// A = 'A'
```
There is no way to make v2 encoder behave like v1. A workaround could be to
manually sort the fields alphabetically in the struct definition, or generate
struct types using `reflect.StructOf`.
#### No indentation by default
V1 automatically indents content of tables by default. V2 does not. However the
same behavior can be obtained using [`Encoder.SetIndentTables`][sit]. For example:
```go
data := map[string]interface{}{
"table": map[string]string{
"key": "value",
},
}
b, _ := tomlv1.Marshal(data)
fmt.Println("v1:\n" + string(b))
b, _ = tomlv2.Marshal(data)
fmt.Println("v2:\n" + string(b))
buf := bytes.Buffer{}
enc := tomlv2.NewEncoder(&buf)
enc.SetIndentTables(true)
enc.Encode(data)
fmt.Println("v2 Encoder:\n" + string(buf.Bytes()))
// Output:
// v1:
//
// [table]
// key = "value"
//
// v2:
// [table]
// key = 'value'
//
//
// v2 Encoder:
// [table]
// key = 'value'
```
[sit]: https://pkg.go.dev/github.com/pelletier/go-toml/v2#Encoder.SetIndentTables
#### Keys and strings are single quoted
V1 always uses double quotes (`"`) around strings and keys that cannot be
represented bare (unquoted). V2 uses single quotes instead by default (`'`),
unless a character cannot be represented, then falls back to double quotes. As a
result of this change, `Encoder.QuoteMapKeys` has been removed, as it is not
useful anymore.
There is no way to make v2 encoder behave like v1.
#### `TextMarshaler` emits as a string, not TOML
Types that implement [`encoding.TextMarshaler`][tm] can emit arbitrary TOML in
v1. The encoder would append the result to the output directly. In v2 the result
is wrapped in a string. As a result, this interface cannot be implemented by the
root object.
There is no way to make v2 encoder behave like v1.
[tm]: https://golang.org/pkg/encoding/#TextMarshaler
#### `Encoder.CompactComments` has been removed
Emitting compact comments is now the default behavior of go-toml. This option
is not necessary anymore.
#### Struct tags have been merged
V1 used to provide multiple struct tags: `comment`, `commented`, `multiline`,
`toml`, and `omitempty`. To behave more like the standard library, v2 has merged
`toml`, `multiline`, `commented`, and `omitempty`. For example:
```go
type doc struct {
// v1
F string `toml:"field" multiline:"true" omitempty:"true" commented:"true"`
// v2
F string `toml:"field,multiline,omitempty,commented"`
}
```
Has a result, the `Encoder.SetTag*` methods have been removed, as there is just
one tag now.
#### `Encoder.ArraysWithOneElementPerLine` has been renamed
The new name is `Encoder.SetArraysMultiline`. The behavior should be the same.
#### `Encoder.Indentation` has been renamed
The new name is `Encoder.SetIndentSymbol`. The behavior should be the same.
#### Embedded structs behave like stdlib
V1 defaults to merging embedded struct fields into the embedding struct. This
behavior was unexpected because it does not follow the standard library. To
avoid breaking backward compatibility, the `Encoder.PromoteAnonymous` method was
added to make the encoder behave correctly. Given backward compatibility is not
a problem anymore, v2 does the right thing by default: it follows the behavior
of `encoding/json`. `Encoder.PromoteAnonymous` has been removed.
[nodoc]: https://github.com/pelletier/go-toml/discussions/506#discussioncomment-1526038
### `query`
go-toml v1 provided the [`go-toml/query`][query] package. It allowed to run
JSONPath-style queries on TOML files. This feature is not available in v2. For a
replacement, check out [dasel][dasel].
This package has been removed because it was essentially not supported anymore
(last commit May 2020), increased the complexity of the code base, and more
complete solutions exist out there.
[query]: https://github.com/pelletier/go-toml/tree/f99d6bbca119636aeafcf351ee52b3d202782627/query
[dasel]: https://github.com/TomWright/dasel
The script `./fuzz.sh` is available to
run [go-fuzz](https://github.com/dvyukov/go-fuzz) on go-toml.
## Versioning
Expect for parts explicitly marked otherwise, go-toml follows [Semantic
Versioning](https://semver.org). The supported version of
[TOML](https://github.com/toml-lang/toml) is indicated at the beginning of this
document. The last two major versions of Go are supported (see [Go Release
Policy](https://golang.org/doc/devel/release.html#policy)).
Go-toml follows [Semantic Versioning](http://semver.org/). The supported version
of [TOML](https://github.com/toml-lang/toml) is indicated at the beginning of
this document. The last two major versions of Go are supported
(see [Go Release Policy](https://golang.org/doc/devel/release.html#policy)).
## License
The MIT License (MIT). Read [LICENSE](LICENSE).
The MIT License (MIT) + Apache 2.0. Read [LICENSE](LICENSE).
+3
View File
@@ -2,6 +2,9 @@
## Supported Versions
Use this section to tell people about which versions of your project are
currently being supported with security updates.
| Version | Supported |
| ---------- | ------------------ |
| Latest 2.x | :white_check_mark: |
+188
View File
@@ -0,0 +1,188 @@
trigger:
- master
stages:
- stage: run_checks
displayName: "Check"
dependsOn: []
jobs:
- job: fmt
displayName: "fmt"
pool:
vmImage: ubuntu-latest
steps:
- task: GoTool@0
displayName: "Install Go 1.16"
inputs:
version: "1.16"
- task: Go@0
displayName: "go fmt ./..."
inputs:
command: 'custom'
customCommand: 'fmt'
arguments: './...'
- job: coverage
displayName: "coverage"
pool:
vmImage: ubuntu-latest
steps:
- task: GoTool@0
displayName: "Install Go 1.16"
inputs:
version: "1.16"
- task: Go@0
displayName: "Generate coverage"
inputs:
command: 'test'
arguments: "-race -coverprofile=coverage.txt -covermode=atomic"
- task: Bash@3
inputs:
targetType: 'inline'
script: 'bash <(curl -s https://codecov.io/bash) -t ${CODECOV_TOKEN}'
env:
CODECOV_TOKEN: $(CODECOV_TOKEN)
- job: benchmark
displayName: "benchmark"
pool:
vmImage: ubuntu-latest
steps:
- task: GoTool@0
displayName: "Install Go 1.16"
inputs:
version: "1.16"
- script: echo "##vso[task.setvariable variable=PATH]${PATH}:/home/vsts/go/bin/"
- task: Bash@3
inputs:
filePath: './benchmark.sh'
arguments: "master $(Build.Repository.Uri)"
- job: go_unit_tests
displayName: "unit tests"
strategy:
matrix:
linux 1.16:
goVersion: '1.16'
imageName: 'ubuntu-latest'
mac 1.16:
goVersion: '1.16'
imageName: 'macOS-latest'
windows 1.16:
goVersion: '1.16'
imageName: 'windows-latest'
linux 1.15:
goVersion: '1.15'
imageName: 'ubuntu-latest'
mac 1.15:
goVersion: '1.15'
imageName: 'macOS-latest'
windows 1.15:
goVersion: '1.15'
imageName: 'windows-latest'
pool:
vmImage: $(imageName)
steps:
- task: GoTool@0
displayName: "Install Go $(goVersion)"
inputs:
version: $(goVersion)
- task: Go@0
displayName: "go test ./..."
inputs:
command: 'test'
arguments: './...'
- stage: build_binaries
displayName: "Build binaries"
dependsOn: run_checks
jobs:
- job: build_binary
displayName: "Build binary"
strategy:
matrix:
linux_amd64:
GOOS: linux
GOARCH: amd64
darwin_amd64:
GOOS: darwin
GOARCH: amd64
windows_amd64:
GOOS: windows
GOARCH: amd64
pool:
vmImage: ubuntu-latest
steps:
- task: GoTool@0
displayName: "Install Go"
inputs:
version: 1.16
- task: Bash@3
inputs:
targetType: inline
script: "make dist"
env:
go.goos: $(GOOS)
go.goarch: $(GOARCH)
- task: CopyFiles@2
inputs:
sourceFolder: '$(Build.SourcesDirectory)'
contents: '*.tar.xz'
TargetFolder: '$(Build.ArtifactStagingDirectory)'
- task: PublishBuildArtifacts@1
inputs:
pathtoPublish: '$(Build.ArtifactStagingDirectory)'
artifactName: binaries
- stage: build_binaries_manifest
displayName: "Build binaries manifest"
dependsOn: build_binaries
jobs:
- job: build_manifest
displayName: "Build binaries manifest"
steps:
- task: DownloadBuildArtifacts@0
inputs:
buildType: 'current'
downloadType: 'single'
artifactName: 'binaries'
downloadPath: '$(Build.SourcesDirectory)'
- task: Bash@3
inputs:
targetType: inline
script: "cd binaries && sha256sum --binary *.tar.xz | tee $(Build.ArtifactStagingDirectory)/sha256sums.txt"
- task: PublishBuildArtifacts@1
inputs:
pathtoPublish: '$(Build.ArtifactStagingDirectory)'
artifactName: manifest
- stage: build_docker_image
displayName: "Build Docker image"
dependsOn: run_checks
jobs:
- job: build
displayName: "Build"
pool:
vmImage: ubuntu-latest
steps:
- task: Docker@2
inputs:
command: 'build'
Dockerfile: 'Dockerfile'
buildContext: '.'
addPipelineData: false
- stage: publish_docker_image
displayName: "Publish Docker image"
dependsOn: build_docker_image
condition: and(succeeded(), eq(variables['Build.SourceBranchName'], 'master'))
jobs:
- job: publish
displayName: "Publish"
pool:
vmImage: ubuntu-latest
steps:
- task: Docker@2
inputs:
containerRegistry: 'DockerHub'
repository: 'pelletier/go-toml'
command: 'buildAndPush'
Dockerfile: 'Dockerfile'
buildContext: '.'
tags: 'latest'
Executable
+35
View File
@@ -0,0 +1,35 @@
#!/bin/bash
set -ex
reference_ref=${1:-master}
reference_git=${2:-.}
if ! `hash benchstat 2>/dev/null`; then
echo "Installing benchstat"
go get golang.org/x/perf/cmd/benchstat
fi
tempdir=`mktemp -d /tmp/go-toml-benchmark-XXXXXX`
ref_tempdir="${tempdir}/ref"
ref_benchmark="${ref_tempdir}/benchmark-`echo -n ${reference_ref}|tr -s '/' '-'`.txt"
local_benchmark="`pwd`/benchmark-local.txt"
echo "=== ${reference_ref} (${ref_tempdir})"
git clone ${reference_git} ${ref_tempdir} >/dev/null 2>/dev/null
pushd ${ref_tempdir} >/dev/null
git checkout ${reference_ref} >/dev/null 2>/dev/null
go test -bench=. -benchmem | tee ${ref_benchmark}
cd benchmark
go test -bench=. -benchmem | tee -a ${ref_benchmark}
popd >/dev/null
echo ""
echo "=== local"
go test -bench=. -benchmem | tee ${local_benchmark}
cd benchmark
go test -bench=. -benchmem | tee -a ${local_benchmark}
echo ""
echo "=== diff"
benchstat -delta-test=none ${ref_benchmark} ${local_benchmark}
-80
View File
@@ -1,80 +0,0 @@
package benchmark_test
import (
"compress/gzip"
"encoding/json"
"io"
"os"
"path/filepath"
"testing"
"github.com/pelletier/go-toml/v2"
"github.com/pelletier/go-toml/v2/internal/assert"
)
var benchInputs = []struct {
name string
jsonLen int
}{
// from https://gist.githubusercontent.com/feeeper/2197d6d734729625a037af1df14cf2aa/raw/2f22b120e476d897179be3c1e2483d18067aa7df/config.toml
{"config", 806507},
// converted from https://github.com/miloyip/nativejson-benchmark
{"canada", 2090234},
{"citm_catalog", 479897},
{"twitter", 428778},
{"code", 1940472},
// converted from https://raw.githubusercontent.com/mailru/easyjson/master/benchmark/example.json
{"example", 7779},
}
func TestUnmarshalDatasetCode(t *testing.T) {
for _, tc := range benchInputs {
t.Run(tc.name, func(t *testing.T) {
buf := fixture(t, tc.name)
var v interface{}
assert.NoError(t, toml.Unmarshal(buf, &v))
b, err := json.Marshal(v)
assert.NoError(t, err)
assert.Equal(t, len(b), tc.jsonLen)
})
}
}
func BenchmarkUnmarshalDataset(b *testing.B) {
for _, tc := range benchInputs {
b.Run(tc.name, func(b *testing.B) {
buf := fixture(b, tc.name)
b.SetBytes(int64(len(buf)))
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
var v interface{}
assert.NoError(b, toml.Unmarshal(buf, &v))
}
})
}
}
// fixture returns the uncompressed contents of path.
func fixture(tb testing.TB, path string) []byte {
tb.Helper()
file := path + ".toml.gz"
f, err := os.Open(filepath.Join("testdata", file))
if os.IsNotExist(err) {
tb.Skip("benchmark fixture not found:", file)
}
assert.NoError(tb, err)
defer func() { _ = f.Close() }()
gz, err := gzip.NewReader(f)
assert.NoError(tb, err)
buf, err := io.ReadAll(gz)
assert.NoError(tb, err)
return buf
}
+164
View File
@@ -0,0 +1,164 @@
{
"array": {
"key1": [
1,
2,
3
],
"key2": [
"red",
"yellow",
"green"
],
"key3": [
[
1,
2
],
[
3,
4,
5
]
],
"key4": [
[
1,
2
],
[
"a",
"b",
"c"
]
],
"key5": [
1,
2,
3
],
"key6": [
1,
2
]
},
"boolean": {
"False": false,
"True": true
},
"datetime": {
"key1": "1979-05-27T07:32:00Z",
"key2": "1979-05-27T00:32:00-07:00",
"key3": "1979-05-27T00:32:00.999999-07:00"
},
"float": {
"both": {
"key": 6.626e-34
},
"exponent": {
"key1": 5e+22,
"key2": 1000000,
"key3": -0.02
},
"fractional": {
"key1": 1,
"key2": 3.1415,
"key3": -0.01
},
"underscores": {
"key1": 9224617.445991227,
"key2": 1e+100
}
},
"fruit": [{
"name": "apple",
"physical": {
"color": "red",
"shape": "round"
},
"variety": [{
"name": "red delicious"
},
{
"name": "granny smith"
}
]
},
{
"name": "banana",
"variety": [{
"name": "plantain"
}]
}
],
"integer": {
"key1": 99,
"key2": 42,
"key3": 0,
"key4": -17,
"underscores": {
"key1": 1000,
"key2": 5349221,
"key3": 12345
}
},
"products": [{
"name": "Hammer",
"sku": 738594937
},
{},
{
"color": "gray",
"name": "Nail",
"sku": 284758393
}
],
"string": {
"basic": {
"basic": "I'm a string. \"You can quote me\". Name\tJosé\nLocation\tSF."
},
"literal": {
"multiline": {
"lines": "The first newline is\ntrimmed in raw strings.\n All other whitespace\n is preserved.\n",
"regex2": "I [dw]on't need \\d{2} apples"
},
"quoted": "Tom \"Dubs\" Preston-Werner",
"regex": "\u003c\\i\\c*\\s*\u003e",
"winpath": "C:\\Users\\nodejs\\templates",
"winpath2": "\\\\ServerX\\admin$\\system32\\"
},
"multiline": {
"continued": {
"key1": "The quick brown fox jumps over the lazy dog.",
"key2": "The quick brown fox jumps over the lazy dog.",
"key3": "The quick brown fox jumps over the lazy dog."
},
"key1": "One\nTwo",
"key2": "One\nTwo",
"key3": "One\nTwo"
}
},
"table": {
"inline": {
"name": {
"first": "Tom",
"last": "Preston-Werner"
},
"point": {
"x": 1,
"y": 2
}
},
"key": "value",
"subtable": {
"key": "another value"
}
},
"x": {
"y": {
"z": {
"w": {}
}
}
}
}
+1 -1
View File
@@ -186,7 +186,7 @@ key3 = 1979-05-27T00:32:00.999999-07:00
key1 = [ 1, 2, 3 ]
key2 = [ "red", "yellow", "green" ]
key3 = [ [ 1, 2 ], [3, 4, 5] ]
key4 = [ [ 1, 2 ], ["a", "b", "c"] ] # this is ok
#key4 = [ [ 1, 2 ], ["a", "b", "c"] ] # this is ok
# Arrays can also be multiline. So in addition to ignoring whitespace, arrays
# also ignore newlines between the brackets. Terminating commas are ok before
+121
View File
@@ -0,0 +1,121 @@
---
array:
key1:
- 1
- 2
- 3
key2:
- red
- yellow
- green
key3:
- - 1
- 2
- - 3
- 4
- 5
key4:
- - 1
- 2
- - a
- b
- c
key5:
- 1
- 2
- 3
key6:
- 1
- 2
boolean:
'False': false
'True': true
datetime:
key1: '1979-05-27T07:32:00Z'
key2: '1979-05-27T00:32:00-07:00'
key3: '1979-05-27T00:32:00.999999-07:00'
float:
both:
key: 6.626e-34
exponent:
key1: 5.0e+22
key2: 1000000
key3: -0.02
fractional:
key1: 1
key2: 3.1415
key3: -0.01
underscores:
key1: 9224617.445991227
key2: 1.0e+100
fruit:
- name: apple
physical:
color: red
shape: round
variety:
- name: red delicious
- name: granny smith
- name: banana
variety:
- name: plantain
integer:
key1: 99
key2: 42
key3: 0
key4: -17
underscores:
key1: 1000
key2: 5349221
key3: 12345
products:
- name: Hammer
sku: 738594937
- {}
- color: gray
name: Nail
sku: 284758393
string:
basic:
basic: "I'm a string. \"You can quote me\". Name\tJosé\nLocation\tSF."
literal:
multiline:
lines: |
The first newline is
trimmed in raw strings.
All other whitespace
is preserved.
regex2: I [dw]on't need \d{2} apples
quoted: Tom "Dubs" Preston-Werner
regex: "<\\i\\c*\\s*>"
winpath: C:\Users\nodejs\templates
winpath2: "\\\\ServerX\\admin$\\system32\\"
multiline:
continued:
key1: The quick brown fox jumps over the lazy dog.
key2: The quick brown fox jumps over the lazy dog.
key3: The quick brown fox jumps over the lazy dog.
key1: |-
One
Two
key2: |-
One
Two
key3: |-
One
Two
table:
inline:
name:
first: Tom
last: Preston-Werner
point:
x: 1
y: 2
key: value
subtable:
key: another value
x:
y:
z:
w: {}
+81 -541
View File
@@ -1,241 +1,17 @@
package benchmark_test
package benchmark
import (
"bytes"
"os"
"encoding/json"
"io/ioutil"
"testing"
"time"
"github.com/pelletier/go-toml/v2"
"github.com/pelletier/go-toml/v2/internal/assert"
burntsushi "github.com/BurntSushi/toml"
"github.com/pelletier/go-toml"
"gopkg.in/yaml.v2"
)
func TestUnmarshalSimple(t *testing.T) {
doc := []byte(`A = "hello"`)
d := struct {
A string
}{}
err := toml.Unmarshal(doc, &d)
if err != nil {
t.Error(err)
}
}
func BenchmarkUnmarshal(b *testing.B) {
b.Run("SimpleDocument", func(b *testing.B) {
doc := []byte(`A = "hello"`)
b.Run("struct", func(b *testing.B) {
b.SetBytes(int64(len(doc)))
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
d := struct {
A string
}{}
err := toml.Unmarshal(doc, &d)
if err != nil {
b.Error(err)
}
}
})
b.Run("map", func(b *testing.B) {
b.SetBytes(int64(len(doc)))
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
d := map[string]interface{}{}
err := toml.Unmarshal(doc, &d)
if err != nil {
b.Error(err)
}
}
})
})
b.Run("ReferenceFile", func(b *testing.B) {
bytes, err := os.ReadFile("benchmark.toml")
if err != nil {
b.Fatal(err)
}
b.Run("struct", func(b *testing.B) {
b.SetBytes(int64(len(bytes)))
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
d := benchmarkDoc{}
err := toml.Unmarshal(bytes, &d)
if err != nil {
b.Error(err)
}
}
})
b.Run("map", func(b *testing.B) {
b.SetBytes(int64(len(bytes)))
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
d := map[string]interface{}{}
err := toml.Unmarshal(bytes, &d)
if err != nil {
b.Error(err)
}
}
})
})
b.Run("HugoFrontMatter", func(b *testing.B) {
b.SetBytes(int64(len(hugoFrontMatterbytes)))
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
d := map[string]interface{}{}
err := toml.Unmarshal(hugoFrontMatterbytes, &d)
if err != nil {
b.Error(err)
}
}
})
}
func marshal(v interface{}) ([]byte, error) {
var b bytes.Buffer
enc := toml.NewEncoder(&b)
err := enc.Encode(v)
return b.Bytes(), err
}
func BenchmarkMarshal(b *testing.B) {
b.Run("SimpleDocument", func(b *testing.B) {
doc := []byte(`A = "hello"`)
b.Run("struct", func(b *testing.B) {
d := struct {
A string
}{}
err := toml.Unmarshal(doc, &d)
if err != nil {
b.Error(err)
}
b.ReportAllocs()
b.ResetTimer()
var out []byte
for i := 0; i < b.N; i++ {
out, err = marshal(d)
if err != nil {
b.Error(err)
}
}
b.SetBytes(int64(len(out)))
})
b.Run("map", func(b *testing.B) {
d := map[string]interface{}{}
err := toml.Unmarshal(doc, &d)
if err != nil {
b.Error(err)
}
b.ReportAllocs()
b.ResetTimer()
var out []byte
for i := 0; i < b.N; i++ {
out, err = marshal(d)
if err != nil {
b.Error(err)
}
}
b.SetBytes(int64(len(out)))
})
})
b.Run("ReferenceFile", func(b *testing.B) {
bytes, err := os.ReadFile("benchmark.toml")
if err != nil {
b.Fatal(err)
}
b.Run("struct", func(b *testing.B) {
d := benchmarkDoc{}
err := toml.Unmarshal(bytes, &d)
if err != nil {
b.Error(err)
}
b.ReportAllocs()
b.ResetTimer()
var out []byte
for i := 0; i < b.N; i++ {
out, err = marshal(d)
if err != nil {
b.Error(err)
}
}
b.SetBytes(int64(len(out)))
})
b.Run("map", func(b *testing.B) {
d := map[string]interface{}{}
err := toml.Unmarshal(bytes, &d)
if err != nil {
b.Error(err)
}
b.ReportAllocs()
b.ResetTimer()
var out []byte
for i := 0; i < b.N; i++ {
out, err = marshal(d)
if err != nil {
b.Error(err)
}
}
b.SetBytes(int64(len(out)))
})
})
b.Run("HugoFrontMatter", func(b *testing.B) {
d := map[string]interface{}{}
err := toml.Unmarshal(hugoFrontMatterbytes, &d)
if err != nil {
b.Error(err)
}
b.ReportAllocs()
b.ResetTimer()
var out []byte
for i := 0; i < b.N; i++ {
out, err = marshal(d)
if err != nil {
b.Error(err)
}
}
b.SetBytes(int64(len(out)))
})
}
type benchmarkDoc struct {
Table struct {
Key string
@@ -249,7 +25,7 @@ type benchmarkDoc struct {
}
Point struct {
X int64
Y int64
U int64
}
}
}
@@ -322,7 +98,6 @@ type benchmarkDoc struct {
Key2 []string
Key3 [][]int64
// TODO: Key4 not supported by go-toml's Unmarshal
Key4 []interface{}
Key5 []int64
Key6 []int64
}
@@ -334,321 +109,86 @@ type benchmarkDoc struct {
Fruit []struct {
Name string
Physical struct {
Color string
Shape string
}
Variety []struct {
Name string
Color string
Shape string
Variety []struct {
Name string
}
}
}
}
func TestUnmarshalReferenceFile(t *testing.T) {
bytes, err := os.ReadFile("benchmark.toml")
assert.NoError(t, err)
d := benchmarkDoc{}
err = toml.Unmarshal(bytes, &d)
assert.NoError(t, err)
expected := benchmarkDoc{
Table: struct {
Key string
Subtable struct{ Key string }
Inline struct {
Name struct {
First string
Last string
}
Point struct {
X int64
Y int64
}
}
}{
Key: "value",
Subtable: struct{ Key string }{
Key: "another value",
},
// note: x.y.z.w is purposefully missing
Inline: struct {
Name struct {
First string
Last string
}
Point struct {
X int64
Y int64
}
}{
Name: struct {
First string
Last string
}{
First: "Tom",
Last: "Preston-Werner",
},
Point: struct {
X int64
Y int64
}{
X: 1,
Y: 2,
},
},
},
String: struct {
Basic struct{ Basic string }
Multiline struct {
Key1 string
Key2 string
Key3 string
Continued struct {
Key1 string
Key2 string
Key3 string
}
}
Literal struct {
Winpath string
Winpath2 string
Quoted string
Regex string
Multiline struct {
Regex2 string
Lines string
}
}
}{
Basic: struct{ Basic string }{
Basic: "I'm a string. \"You can quote me\". Name\tJos\u00E9\nLocation\tSF.",
},
Multiline: struct {
Key1 string
Key2 string
Key3 string
Continued struct {
Key1 string
Key2 string
Key3 string
}
}{
Key1: "One\nTwo",
Key2: "One\nTwo",
Key3: "One\nTwo",
Continued: struct {
Key1 string
Key2 string
Key3 string
}{
Key1: `The quick brown fox jumps over the lazy dog.`,
Key2: `The quick brown fox jumps over the lazy dog.`,
Key3: `The quick brown fox jumps over the lazy dog.`,
},
},
Literal: struct {
Winpath string
Winpath2 string
Quoted string
Regex string
Multiline struct {
Regex2 string
Lines string
}
}{
Winpath: `C:\Users\nodejs\templates`,
Winpath2: `\\ServerX\admin$\system32\`,
Quoted: `Tom "Dubs" Preston-Werner`,
Regex: `<\i\c*\s*>`,
Multiline: struct {
Regex2 string
Lines string
}{
Regex2: `I [dw]on't need \d{2} apples`,
Lines: `The first newline is
trimmed in raw strings.
All other whitespace
is preserved.
`,
},
},
},
Integer: struct {
Key1 int64
Key2 int64
Key3 int64
Key4 int64
Underscores struct {
Key1 int64
Key2 int64
Key3 int64
}
}{
Key1: 99,
Key2: 42,
Key3: 0,
Key4: -17,
Underscores: struct {
Key1 int64
Key2 int64
Key3 int64
}{
Key1: 1000,
Key2: 5349221,
Key3: 12345,
},
},
Float: struct {
Fractional struct {
Key1 float64
Key2 float64
Key3 float64
}
Exponent struct {
Key1 float64
Key2 float64
Key3 float64
}
Both struct{ Key float64 }
Underscores struct {
Key1 float64
Key2 float64
}
}{
Fractional: struct {
Key1 float64
Key2 float64
Key3 float64
}{
Key1: 1.0,
Key2: 3.1415,
Key3: -0.01,
},
Exponent: struct {
Key1 float64
Key2 float64
Key3 float64
}{
Key1: 5e+22,
Key2: 1e6,
Key3: -2e-2,
},
Both: struct{ Key float64 }{
Key: 6.626e-34,
},
Underscores: struct {
Key1 float64
Key2 float64
}{
Key1: 9224617.445991228313,
Key2: 1e100,
},
},
Boolean: struct {
True bool
False bool
}{
True: true,
False: false,
},
Datetime: struct {
Key1 time.Time
Key2 time.Time
Key3 time.Time
}{
Key1: time.Date(1979, 5, 27, 7, 32, 0, 0, time.UTC),
Key2: time.Date(1979, 5, 27, 0, 32, 0, 0, time.FixedZone("", -7*3600)),
Key3: time.Date(1979, 5, 27, 0, 32, 0, 999999000, time.FixedZone("", -7*3600)),
},
Array: struct {
Key1 []int64
Key2 []string
Key3 [][]int64
Key4 []interface{}
Key5 []int64
Key6 []int64
}{
Key1: []int64{1, 2, 3},
Key2: []string{"red", "yellow", "green"},
Key3: [][]int64{{1, 2}, {3, 4, 5}},
Key4: []interface{}{
[]interface{}{int64(1), int64(2)},
[]interface{}{"a", "b", "c"},
},
Key5: []int64{1, 2, 3},
Key6: []int64{1, 2},
},
Products: []struct {
Name string
Sku int64
Color string
}{
{
Name: "Hammer",
Sku: 738594937,
},
{},
{
Name: "Nail",
Sku: 284758393,
Color: "gray",
},
},
Fruit: []struct {
Name string
Physical struct {
Color string
Shape string
}
Variety []struct{ Name string }
}{
{
Name: "apple",
Physical: struct {
Color string
Shape string
}{
Color: "red",
Shape: "round",
},
Variety: []struct{ Name string }{
{Name: "red delicious"},
{Name: "granny smith"},
},
},
{
Name: "banana",
Variety: []struct{ Name string }{
{Name: "plantain"},
},
},
},
func BenchmarkParseToml(b *testing.B) {
fileBytes, err := ioutil.ReadFile("benchmark.toml")
if err != nil {
b.Fatal(err)
}
b.ResetTimer()
for i := 0; i < b.N; i++ {
_, err := toml.LoadReader(bytes.NewReader(fileBytes))
if err != nil {
b.Fatal(err)
}
}
assert.Equal(t, expected, d)
}
var hugoFrontMatterbytes = []byte(`
categories = ["Development", "VIM"]
date = "2012-04-06"
description = "spf13-vim is a cross platform distribution of vim plugins and resources for Vim."
slug = "spf13-vim-3-0-release-and-new-website"
tags = [".vimrc", "plugins", "spf13-vim", "vim"]
title = "spf13-vim 3.0 release and new website"
include_toc = true
show_comments = false
func BenchmarkUnmarshalToml(b *testing.B) {
bytes, err := ioutil.ReadFile("benchmark.toml")
if err != nil {
b.Fatal(err)
}
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
target := benchmarkDoc{}
err := toml.Unmarshal(bytes, &target)
if err != nil {
b.Fatal(err)
}
}
}
[[cascade]]
background = "yosemite.jpg"
[cascade._target]
kind = "page"
lang = "en"
path = "/blog/**"
func BenchmarkUnmarshalBurntSushiToml(b *testing.B) {
bytes, err := ioutil.ReadFile("benchmark.toml")
if err != nil {
b.Fatal(err)
}
b.ResetTimer()
for i := 0; i < b.N; i++ {
target := benchmarkDoc{}
err := burntsushi.Unmarshal(bytes, &target)
if err != nil {
b.Fatal(err)
}
}
}
[[cascade]]
background = "goldenbridge.jpg"
[cascade._target]
kind = "section"
`)
func BenchmarkUnmarshalJson(b *testing.B) {
bytes, err := ioutil.ReadFile("benchmark.json")
if err != nil {
b.Fatal(err)
}
b.ResetTimer()
for i := 0; i < b.N; i++ {
target := benchmarkDoc{}
err := json.Unmarshal(bytes, &target)
if err != nil {
b.Fatal(err)
}
}
}
func BenchmarkUnmarshalYaml(b *testing.B) {
bytes, err := ioutil.ReadFile("benchmark.yml")
if err != nil {
b.Fatal(err)
}
b.ResetTimer()
for i := 0; i < b.N; i++ {
target := benchmarkDoc{}
err := yaml.Unmarshal(bytes, &target)
if err != nil {
b.Fatal(err)
}
}
}
+11
View File
@@ -0,0 +1,11 @@
module github.com/pelletier/go-toml/benchmark
go 1.12
require (
github.com/BurntSushi/toml v0.3.1
github.com/pelletier/go-toml v0.0.0
gopkg.in/yaml.v2 v2.3.0
)
replace github.com/pelletier/go-toml => ../
+6
View File
@@ -0,0 +1,6 @@
github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v2 v2.3.0 h1:clyUAQHOM3G0M3f5vQj7LuJrETvjVot3Z5el9nffUtU=
gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
Binary file not shown.
Binary file not shown.
BIN
View File
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
-289
View File
@@ -1,289 +0,0 @@
#!/usr/bin/env bash
stderr() {
echo "$@" 1>&2
}
usage() {
b=$(basename "$0")
echo $b: ERROR: "$@" 1>&2
cat 1>&2 <<EOF
DESCRIPTION
$(basename "$0") is the script to run continuous integration commands for
go-toml on unix.
Requires Go and Git to be available in the PATH. Expects to be ran from the
root of go-toml's Git repository.
USAGE
$b COMMAND [OPTIONS...]
COMMANDS
benchmark [OPTIONS...] [BRANCH]
Run benchmarks.
ARGUMENTS
BRANCH Optional. Defines which Git branch to use when running
benchmarks.
OPTIONS
-d Compare benchmarks of HEAD with BRANCH using benchstats. In
this form the BRANCH argument is required.
-a Compare benchmarks of HEAD against go-toml v1 and
BurntSushi/toml.
-html When used with -a, emits the output as HTML, ready to be
embedded in the README.
coverage [OPTIONS...] [BRANCH]
Generates code coverage.
ARGUMENTS
BRANCH Optional. Defines which Git branch to use when reporting
coverage. Defaults to HEAD.
OPTIONS
-d Compare coverage of HEAD with the one of BRANCH. In this form,
the BRANCH argument is required. Exit code is non-zero when
coverage percentage decreased.
EOF
exit 1
}
cover() {
branch="${1}"
dir="$(mktemp -d)"
stderr "Executing coverage for ${branch} at ${dir}"
if [ "${branch}" = "HEAD" ]; then
cp -r . "${dir}/"
else
git worktree add "$dir" "$branch"
fi
pushd "$dir"
go test -covermode=atomic -coverpkg=./... -coverprofile=coverage.out.tmp ./...
grep -Ev '(fuzz|testsuite|tomltestgen|gotoml-test-decoder|gotoml-test-encoder)' coverage.out.tmp > coverage.out
go tool cover -func=coverage.out
echo "Coverage profile for ${branch}: ${dir}/coverage.out" >&2
popd
if [ "${branch}" != "HEAD" ]; then
git worktree remove --force "$dir"
fi
}
coverage() {
case "$1" in
-d)
shift
target="${1?Need to provide a target branch argument}"
output_dir="$(mktemp -d)"
target_out="${output_dir}/target.txt"
head_out="${output_dir}/head.txt"
cover "${target}" > "${target_out}"
cover "HEAD" > "${head_out}"
cat "${target_out}"
cat "${head_out}"
echo ""
target_pct="$(tail -n2 ${target_out} | head -n1 | sed -E 's/.*total.*\t([0-9.]+)%.*/\1/')"
head_pct="$(tail -n2 ${head_out} | head -n1 | sed -E 's/.*total.*\t([0-9.]+)%/\1/')"
echo "Results: ${target} ${target_pct}% HEAD ${head_pct}%"
delta_pct=$(echo "$head_pct - $target_pct" | bc -l)
echo "Delta: ${delta_pct}"
if [[ $delta_pct = \-* ]]; then
echo "Regression!";
target_diff="${output_dir}/target.diff.txt"
head_diff="${output_dir}/head.diff.txt"
cat "${target_out}" | grep -E '^github.com/pelletier/go-toml' | tr -s "\t " | cut -f 2,3 | sort > "${target_diff}"
cat "${head_out}" | grep -E '^github.com/pelletier/go-toml' | tr -s "\t " | cut -f 2,3 | sort > "${head_diff}"
diff --side-by-side --suppress-common-lines "${target_diff}" "${head_diff}"
return 1
fi
return 0
;;
esac
cover "${1-HEAD}"
}
bench() {
branch="${1}"
out="${2}"
replace="${3}"
dir="$(mktemp -d)"
stderr "Executing benchmark for ${branch} at ${dir}"
if [ "${branch}" = "HEAD" ]; then
cp -r . "${dir}/"
else
git worktree add "$dir" "$branch"
fi
pushd "$dir"
if [ "${replace}" != "" ]; then
find ./benchmark/ -iname '*.go' -exec sed -i -E "s|github.com/pelletier/go-toml/v2\"|${replace}\"|g" {} \;
go get "${replace}"
fi
export GOMAXPROCS=2
go test '-bench=^Benchmark(Un)?[mM]arshal' -count=10 -run=Nothing ./... | tee "${out}"
popd
if [ "${branch}" != "HEAD" ]; then
git worktree remove --force "$dir"
fi
}
fmktemp() {
if mktemp --version &> /dev/null; then
# GNU
mktemp --suffix=-$1
else
# BSD
mktemp -t $1
fi
}
benchstathtml() {
python3 - $1 <<'EOF'
import sys
lines = []
stop = False
with open(sys.argv[1]) as f:
for line in f.readlines():
line = line.strip()
if line == "":
stop = True
if not stop:
lines.append(line.split(','))
results = []
for line in reversed(lines[2:]):
if len(line) < 8 or line[0] == "":
continue
v2 = float(line[1])
results.append([
line[0].replace("-32", ""),
"%.1fx" % (float(line[3])/v2), # v1
"%.1fx" % (float(line[7])/v2), # bs
])
if not results:
print("No benchmark results to display.", file=sys.stderr)
sys.exit(1)
# move geomean to the end
results.append(results[0])
del results[0]
def printtable(data):
print("""
<table>
<thead>
<tr><th>Benchmark</th><th>go-toml v1</th><th>BurntSushi/toml</th></tr>
</thead>
<tbody>""")
for r in data:
print(" <tr><td>{}</td><td>{}</td><td>{}</td></tr>".format(*r))
print(""" </tbody>
</table>""")
def match(x):
return "ReferenceFile" in x[0] or "HugoFrontMatter" in x[0]
above = [x for x in results if match(x)]
below = [x for x in results if not match(x)]
printtable(above)
print("<details><summary>See more</summary>")
print("""<p>The table above has the results of the most common use-cases. The table below
contains the results of all benchmarks, including unrealistic ones. It is
provided for completeness.</p>""")
printtable(below)
print('<p>This table can be generated with <code>./ci.sh benchmark -a -html</code>.</p>')
print("</details>")
EOF
}
benchmark() {
case "$1" in
-d)
shift
target="${1?Need to provide a target branch argument}"
old=`fmktemp ${target}`
bench "${target}" "${old}"
new=`fmktemp HEAD`
bench HEAD "${new}"
benchstat "${old}" "${new}"
return 0
;;
-a)
shift
v2stats=`fmktemp go-toml-v2`
bench HEAD "${v2stats}" "github.com/pelletier/go-toml/v2"
v1stats=`fmktemp go-toml-v1`
bench HEAD "${v1stats}" "github.com/pelletier/go-toml"
bsstats=`fmktemp bs-toml`
bench HEAD "${bsstats}" "github.com/BurntSushi/toml"
cp "${v2stats}" go-toml-v2.txt
cp "${v1stats}" go-toml-v1.txt
cp "${bsstats}" bs-toml.txt
if [ "$1" = "-html" ]; then
tmpcsv=`fmktemp csv`
benchstat -format csv go-toml-v2.txt go-toml-v1.txt bs-toml.txt > $tmpcsv
benchstathtml $tmpcsv
else
benchstat go-toml-v2.txt go-toml-v1.txt bs-toml.txt
fi
rm -f go-toml-v2.txt go-toml-v1.txt bs-toml.txt
return $?
esac
bench "${1-HEAD}" `mktemp`
}
case "$1" in
coverage) shift; coverage $@;;
benchmark) shift; benchmark $@;;
*) usage "bad argument $1";;
esac
-31
View File
@@ -1,31 +0,0 @@
// Package gotoml-test-decoder is a minimal decoder program used to compare this library with other TOML implementations.
package main
import (
"flag"
"log"
"os"
"path"
"github.com/pelletier/go-toml/v2/internal/testsuite"
)
func main() {
log.SetFlags(0)
flag.Usage = usage
flag.Parse()
if flag.NArg() != 0 {
flag.Usage()
}
err := testsuite.DecodeStdin()
if err != nil {
log.Fatal(err)
}
}
func usage() {
log.Printf("Usage: %s < toml-file\n", path.Base(os.Args[0]))
flag.PrintDefaults()
os.Exit(1)
}
-31
View File
@@ -1,31 +0,0 @@
// Package gotoml-test-encoder is a minimal encoder program used to compare this library with other TOML implementations.
package main
import (
"flag"
"log"
"os"
"path"
"github.com/pelletier/go-toml/v2/internal/testsuite"
)
func main() {
log.SetFlags(0)
flag.Usage = usage
flag.Parse()
if flag.NArg() != 0 {
flag.Usage()
}
err := testsuite.EncodeStdin()
if err != nil {
log.Fatal(err)
}
}
func usage() {
log.Printf("Usage: %s < json-file\n", path.Base(os.Args[0]))
flag.PrintDefaults()
os.Exit(1)
}
+62 -46
View File
@@ -1,66 +1,82 @@
// Package jsontoml is a program that converts JSON to TOML.
// Jsontoml reads JSON and converts to TOML.
//
// # Usage
//
// Reading from stdin:
//
// cat file.json | jsontoml > file.toml
//
// Reading from a file:
//
// jsontoml file.json > file.toml
//
// # Installation
//
// Using Go:
//
// go install github.com/pelletier/go-toml/v2/cmd/jsontoml@latest
// Usage:
// cat file.toml | jsontoml > file.json
// jsontoml file1.toml > file.json
package main
import (
"encoding/json"
"flag"
"fmt"
"io"
"io/ioutil"
"os"
"github.com/pelletier/go-toml/v2"
"github.com/pelletier/go-toml/v2/internal/cli"
"github.com/pelletier/go-toml"
)
const usage = `jsontoml can be used in two ways:
Reading from stdin:
cat file.json | jsontoml > file.toml
Reading from a file:
jsontoml file.json > file.toml
`
var useJSONNumber bool
func main() {
flag.BoolVar(&useJSONNumber, "use-json-number", false, "unmarshal numbers into `json.Number` type instead of as `float64`")
p := cli.Program{
Usage: usage,
Fn: convert,
flag.Usage = func() {
fmt.Fprintln(os.Stderr, "jsontoml can be used in two ways:")
fmt.Fprintln(os.Stderr, "Writing to STDIN and reading from STDOUT:")
fmt.Fprintln(os.Stderr, "")
fmt.Fprintln(os.Stderr, "")
fmt.Fprintln(os.Stderr, "Reading from a file name:")
fmt.Fprintln(os.Stderr, " tomljson file.toml")
}
p.Execute()
flag.Parse()
os.Exit(processMain(flag.Args(), os.Stdin, os.Stdout, os.Stderr))
}
func convert(r io.Reader, w io.Writer) error {
var v interface{}
func processMain(files []string, defaultInput io.Reader, output io.Writer, errorOutput io.Writer) int {
// read from stdin and print to stdout
inputReader := defaultInput
d := json.NewDecoder(r)
e := toml.NewEncoder(w)
if useJSONNumber {
d.UseNumber()
e.SetMarshalJSONNumbers(true)
if len(files) > 0 {
file, err := os.Open(files[0])
if err != nil {
printError(err, errorOutput)
return -1
}
inputReader = file
defer file.Close()
}
err := d.Decode(&v)
s, err := reader(inputReader)
if err != nil {
return err
printError(err, errorOutput)
return -1
}
io.WriteString(output, s)
return 0
}
func printError(err error, output io.Writer) {
io.WriteString(output, err.Error()+"\n")
}
func reader(r io.Reader) (string, error) {
jsonMap := make(map[string]interface{})
jsonBytes, err := ioutil.ReadAll(r)
if err != nil {
return "", err
}
err = json.Unmarshal(jsonBytes, &jsonMap)
if err != nil {
return "", err
}
return e.Encode(v)
tree, err := toml.TreeFromMap(jsonMap)
if err != nil {
return "", err
}
return mapToTOML(tree)
}
func mapToTOML(t *toml.Tree) (string, error) {
tomlBytes, err := t.ToTomlString()
if err != nil {
return "", err
}
return string(tomlBytes[:]), nil
}
+82 -52
View File
@@ -2,61 +2,91 @@ package main
import (
"bytes"
"io/ioutil"
"os"
"runtime"
"strings"
"testing"
"github.com/pelletier/go-toml/v2/internal/assert"
)
func TestConvert(t *testing.T) {
examples := []struct {
name string
input string
expected string
errors bool
useJSONNumber bool
}{
{
name: "valid json",
input: `
{
"mytoml": {
"a": 42
}
}`,
expected: `[mytoml]
a = 42.0
`,
},
{
name: "use json number",
useJSONNumber: true,
input: `
{
"mytoml": {
"a": 42
}
}`,
expected: `[mytoml]
a = 42
`,
},
{
name: "invalid json",
input: `{ foo`,
errors: true,
},
}
for _, e := range examples {
b := new(bytes.Buffer)
useJSONNumber = e.useJSONNumber
err := convert(strings.NewReader(e.input), b)
if e.errors {
assert.Error(t, err)
} else {
assert.NoError(t, err)
assert.Equal(t, e.expected, b.String())
}
func expectBufferEquality(t *testing.T, name string, buffer *bytes.Buffer, expected string) {
output := buffer.String()
if output != expected {
t.Errorf("incorrect %s: \n%sexpected %s: \n%s", name, output, name, expected)
t.Log([]rune(output))
t.Log([]rune(expected))
}
}
func expectProcessMainResults(t *testing.T, input string, args []string, exitCode int, expectedOutput string, expectedError string) {
inputReader := strings.NewReader(input)
outputBuffer := new(bytes.Buffer)
errorBuffer := new(bytes.Buffer)
returnCode := processMain(args, inputReader, outputBuffer, errorBuffer)
expectBufferEquality(t, "output", outputBuffer, expectedOutput)
expectBufferEquality(t, "error", errorBuffer, expectedError)
if returnCode != exitCode {
t.Error("incorrect return code:", returnCode, "expected", exitCode)
}
}
func TestProcessMainReadFromStdin(t *testing.T) {
expectedOutput := `
[mytoml]
a = 42.0
`
input := `{
"mytoml": {
"a": 42
}
}
`
expectedError := ``
expectedExitCode := 0
expectProcessMainResults(t, input, []string{}, expectedExitCode, expectedOutput, expectedError)
}
func TestProcessMainReadFromFile(t *testing.T) {
input := `{
"mytoml": {
"a": 42
}
}
`
tmpfile, err := ioutil.TempFile("", "example.json")
if err != nil {
t.Fatal(err)
}
if _, err := tmpfile.Write([]byte(input)); err != nil {
t.Fatal(err)
}
defer os.Remove(tmpfile.Name())
expectedOutput := `
[mytoml]
a = 42.0
`
expectedError := ``
expectedExitCode := 0
expectProcessMainResults(t, ``, []string{tmpfile.Name()}, expectedExitCode, expectedOutput, expectedError)
}
func TestProcessMainReadFromMissingFile(t *testing.T) {
var expectedError string
if runtime.GOOS == "windows" {
expectedError = `open /this/file/does/not/exist: The system cannot find the path specified.
`
} else {
expectedError = `open /this/file/does/not/exist: no such file or directory
`
}
expectProcessMainResults(t, ``, []string{"/this/file/does/not/exist"}, -1, ``, expectedError)
}
+53 -45
View File
@@ -1,63 +1,71 @@
// Package tomljson is a program that converts TOML to JSON.
// Tomljson reads TOML and converts to JSON.
//
// # Usage
//
// Reading from stdin:
//
// cat file.toml | tomljson > file.json
//
// Reading from a file:
//
// tomljson file.toml > file.json
//
// # Installation
//
// Using Go:
//
// go install github.com/pelletier/go-toml/v2/cmd/tomljson@latest
// Usage:
// cat file.toml | tomljson > file.json
// tomljson file1.toml > file.json
package main
import (
"encoding/json"
"errors"
"flag"
"fmt"
"io"
"os"
"github.com/pelletier/go-toml/v2"
"github.com/pelletier/go-toml/v2/internal/cli"
"github.com/pelletier/go-toml"
)
const usage = `tomljson can be used in two ways:
Reading from stdin:
cat file.toml | tomljson > file.json
Reading from a file:
tomljson file.toml > file.json
`
func main() {
p := cli.Program{
Usage: usage,
Fn: convert,
flag.Usage = func() {
fmt.Fprintln(os.Stderr, "tomljson can be used in two ways:")
fmt.Fprintln(os.Stderr, "Writing to STDIN and reading from STDOUT:")
fmt.Fprintln(os.Stderr, " cat file.toml | tomljson > file.json")
fmt.Fprintln(os.Stderr, "")
fmt.Fprintln(os.Stderr, "Reading from a file name:")
fmt.Fprintln(os.Stderr, " tomljson file.toml")
}
p.Execute()
flag.Parse()
os.Exit(processMain(flag.Args(), os.Stdin, os.Stdout, os.Stderr))
}
func convert(r io.Reader, w io.Writer) error {
var v interface{}
func processMain(files []string, defaultInput io.Reader, output io.Writer, errorOutput io.Writer) int {
// read from stdin and print to stdout
inputReader := defaultInput
d := toml.NewDecoder(r)
err := d.Decode(&v)
if err != nil {
var derr *toml.DecodeError
if errors.As(err, &derr) {
row, col := derr.Position()
return fmt.Errorf("%s\nerror occurred at row %d column %d", derr.String(), row, col)
if len(files) > 0 {
var err error
inputReader, err = os.Open(files[0])
if err != nil {
printError(err, errorOutput)
return -1
}
return err
}
e := json.NewEncoder(w)
e.SetIndent("", " ")
return e.Encode(v)
s, err := reader(inputReader)
if err != nil {
printError(err, errorOutput)
return -1
}
io.WriteString(output, s+"\n")
return 0
}
func printError(err error, output io.Writer) {
io.WriteString(output, err.Error()+"\n")
}
func reader(r io.Reader) (string, error) {
tree, err := toml.LoadReader(r)
if err != nil {
return "", err
}
return mapToJSON(tree)
}
func mapToJSON(tree *toml.Tree) (string, error) {
treeMap := tree.ToMap()
bytes, err := json.MarshalIndent(treeMap, "", " ")
if err != nil {
return "", err
}
return string(bytes[:]), nil
}
+73 -43
View File
@@ -2,59 +2,89 @@ package main
import (
"bytes"
"errors"
"io"
"io/ioutil"
"os"
"runtime"
"strings"
"testing"
"github.com/pelletier/go-toml/v2/internal/assert"
)
func TestConvert(t *testing.T) {
examples := []struct {
name string
input io.Reader
expected string
errors bool
}{
{
name: "valid toml",
input: strings.NewReader(`
[mytoml]
a = 42`),
expected: `{
func expectBufferEquality(t *testing.T, name string, buffer *bytes.Buffer, expected string) {
output := buffer.String()
if output != expected {
t.Errorf("incorrect %s:\n%s\n\nexpected %s:\n%s", name, output, name, expected)
t.Log([]rune(output))
t.Log([]rune(expected))
}
}
func expectProcessMainResults(t *testing.T, input string, args []string, exitCode int, expectedOutput string, expectedError string) {
inputReader := strings.NewReader(input)
outputBuffer := new(bytes.Buffer)
errorBuffer := new(bytes.Buffer)
returnCode := processMain(args, inputReader, outputBuffer, errorBuffer)
expectBufferEquality(t, "output", outputBuffer, expectedOutput)
expectBufferEquality(t, "error", errorBuffer, expectedError)
if returnCode != exitCode {
t.Error("incorrect return code:", returnCode, "expected", exitCode)
}
}
func TestProcessMainReadFromStdin(t *testing.T) {
input := `
[mytoml]
a = 42`
expectedOutput := `{
"mytoml": {
"a": 42
}
}
`,
},
{
name: "invalid toml",
input: strings.NewReader(`bad = []]`),
errors: true,
},
{
name: "bad reader",
input: &badReader{},
errors: true,
},
}
`
expectedError := ``
expectedExitCode := 0
for _, e := range examples {
b := new(bytes.Buffer)
err := convert(e.input, b)
if e.errors {
assert.Error(t, err)
} else {
assert.NoError(t, err)
assert.Equal(t, e.expected, b.String())
}
}
expectProcessMainResults(t, input, []string{}, expectedExitCode, expectedOutput, expectedError)
}
type badReader struct{}
func TestProcessMainReadFromFile(t *testing.T) {
input := `
[mytoml]
a = 42`
func (r *badReader) Read([]byte) (int, error) {
return 0, errors.New("reader failed on purpose")
tmpfile, err := ioutil.TempFile("", "example.toml")
if err != nil {
t.Fatal(err)
}
if _, err := tmpfile.Write([]byte(input)); err != nil {
t.Fatal(err)
}
defer os.Remove(tmpfile.Name())
expectedOutput := `{
"mytoml": {
"a": 42
}
}
`
expectedError := ``
expectedExitCode := 0
expectProcessMainResults(t, ``, []string{tmpfile.Name()}, expectedExitCode, expectedOutput, expectedError)
}
func TestProcessMainReadFromMissingFile(t *testing.T) {
var expectedError string
if runtime.GOOS == "windows" {
expectedError = `open /this/file/does/not/exist: The system cannot find the path specified.
`
} else {
expectedError = `open /this/file/does/not/exist: no such file or directory
`
}
expectProcessMainResults(t, ``, []string{"/this/file/does/not/exist"}, -1, ``, expectedError)
}
+67 -42
View File
@@ -1,58 +1,83 @@
// Package tomll is a linter program for TOML.
// Tomll is a linter for TOML
//
// # Usage
//
// Reading from stdin, writing to stdout:
//
// cat file.toml | tomll
//
// Reading and updating a list of files in place:
//
// tomll a.toml b.toml c.toml
//
// # Installation
//
// Using Go:
//
// go install github.com/pelletier/go-toml/v2/cmd/tomll@latest
// Usage:
// cat file.toml | tomll > file_linted.toml
// tomll file1.toml file2.toml # lint the two files in place
package main
import (
"bytes"
"flag"
"fmt"
"io"
"io/ioutil"
"os"
"github.com/pelletier/go-toml/v2"
"github.com/pelletier/go-toml/v2/internal/cli"
"github.com/pelletier/go-toml"
)
const usage = `tomll can be used in two ways:
Reading from stdin, writing to stdout:
cat file.toml | tomll > file.toml
Reading and updating a list of files in place:
tomll a.toml b.toml c.toml
When given a list of files, tomll will modify all files in place without asking.
`
func main() {
p := cli.Program{
Usage: usage,
Fn: convert,
Inplace: true,
multiLineArray := flag.Bool("multiLineArray", false, "sets up the linter to encode arrays with more than one element on multiple lines instead of one.")
flag.Usage = func() {
fmt.Fprintln(os.Stderr, "tomll can be used in two ways:")
fmt.Fprintln(os.Stderr, "Writing to STDIN and reading from STDOUT:")
fmt.Fprintln(os.Stderr, " cat file.toml | tomll > file.toml")
fmt.Fprintln(os.Stderr, "")
fmt.Fprintln(os.Stderr, "Reading and updating a list of files:")
fmt.Fprintln(os.Stderr, " tomll a.toml b.toml c.toml")
fmt.Fprintln(os.Stderr, "")
fmt.Fprintln(os.Stderr, "When given a list of files, tomll will modify all files in place without asking.")
fmt.Fprintln(os.Stderr, "When given a list of files, tomll will modify all files in place without asking.")
fmt.Fprintln(os.Stderr, "")
fmt.Fprintln(os.Stderr, "Flags:")
fmt.Fprintln(os.Stderr, "-multiLineArray sets up the linter to encode arrays with more than one element on multiple lines instead of one.")
}
flag.Parse()
// read from stdin and print to stdout
if flag.NArg() == 0 {
s, err := lintReader(os.Stdin, *multiLineArray)
if err != nil {
io.WriteString(os.Stderr, err.Error())
os.Exit(-1)
}
io.WriteString(os.Stdout, s)
} else {
// otherwise modify a list of files
for _, filename := range flag.Args() {
s, err := lintFile(filename, *multiLineArray)
if err != nil {
io.WriteString(os.Stderr, err.Error())
os.Exit(-1)
}
ioutil.WriteFile(filename, []byte(s), 0644)
}
}
p.Execute()
}
func convert(r io.Reader, w io.Writer) error {
var v interface{}
d := toml.NewDecoder(r)
err := d.Decode(&v)
func lintFile(filename string, multiLineArray bool) (string, error) {
tree, err := toml.LoadFile(filename)
if err != nil {
return err
return "", err
}
e := toml.NewEncoder(w)
return e.Encode(v)
buf := new(bytes.Buffer)
if err := toml.NewEncoder(buf).ArraysWithOneElementPerLine(multiLineArray).Encode(tree); err != nil {
panic(err)
}
return buf.String(), nil
}
func lintReader(r io.Reader, multiLineArray bool) (string, error) {
tree, err := toml.LoadReader(r)
if err != nil {
return "", err
}
buf := new(bytes.Buffer)
if err := toml.NewEncoder(buf).ArraysWithOneElementPerLine(multiLineArray).Encode(tree); err != nil {
panic(err)
}
return buf.String(), nil
}
-44
View File
@@ -1,44 +0,0 @@
package main
import (
"bytes"
"strings"
"testing"
"github.com/pelletier/go-toml/v2/internal/assert"
)
func TestConvert(t *testing.T) {
examples := []struct {
name string
input string
expected string
errors bool
}{
{
name: "valid toml",
input: `
mytoml.a = 42.0
`,
expected: `[mytoml]
a = 42.0
`,
},
{
name: "invalid toml",
input: `[what`,
errors: true,
},
}
for _, e := range examples {
b := new(bytes.Buffer)
err := convert(strings.NewReader(e.input), b)
if e.errors {
assert.Error(t, err)
} else {
assert.NoError(t, err)
assert.Equal(t, e.expected, b.String())
}
}
}
+116 -80
View File
@@ -1,24 +1,26 @@
// tomltestgen retrieves a given version of the language-agnostic TOML test suite in
// https://github.com/BurntSushi/toml-test and generates go-toml unit tests.
// Tomltestgen is a program that retrieves a given version of
// https://github.com/BurntSushi/toml-test and generates go code for go-toml's unit tests
// based on the test files.
//
// Within the go-toml package, run `go generate`. Otherwise, use:
//
// go run github.com/pelletier/go-toml/cmd/tomltestgen -o toml_testgen_test.go
// Usage: go run github.com/pelletier/go-toml/cmd/tomltestgen > toml_testgen_test.go
package main
import (
"archive/zip"
"bytes"
"flag"
"fmt"
"go/format"
"io"
"io/ioutil"
"log"
"net/http"
"os"
"path/filepath"
"regexp"
"strconv"
"strings"
"text/template"
"time"
"unicode"
)
type invalid struct {
@@ -29,7 +31,7 @@ type invalid struct {
type valid struct {
Name string
Input string
JSONRef string
JsonRef string
}
type testsCollection struct {
@@ -40,55 +42,96 @@ type testsCollection struct {
Count int
}
const srcTemplate = "// Code generated by tomltestgen for toml-test ref {{.Ref}} on {{.Timestamp}}. DO NOT EDIT.\n" +
"package toml_test\n" +
const srcTemplate = "// Generated by tomltestgen for toml-test ref {{.Ref}} on {{.Timestamp}}\n" +
"package toml\n" +
" import (\n" +
" \"testing\"\n" +
")\n" +
"{{range .Invalid}}\n" +
"func TestTOMLTest_Invalid_{{.Name}}(t *testing.T) {\n" +
"func TestInvalid{{.Name}}(t *testing.T) {\n" +
" input := {{.Input|gostr}}\n" +
" testgenInvalid(t, input)\n" +
"}\n" +
"{{end}}\n" +
"\n" +
"{{range .Valid}}\n" +
"func TestTOMLTest_Valid_{{.Name}}(t *testing.T) {\n" +
"func TestValid{{.Name}}(t *testing.T) {\n" +
" input := {{.Input|gostr}}\n" +
" jsonRef := {{.JSONRef|gostr}}\n" +
" jsonRef := {{.JsonRef|gostr}}\n" +
" testgenValid(t, input, jsonRef)\n" +
"}\n" +
"{{end}}\n"
func downloadTmpFile(url string) string {
log.Println("starting to download file from", url)
resp, err := http.Get(url)
if err != nil {
panic(err)
}
defer resp.Body.Close()
tmpfile, err := ioutil.TempFile("", "toml-test-*.zip")
if err != nil {
panic(err)
}
defer tmpfile.Close()
copiedLen, err := io.Copy(tmpfile, resp.Body)
if err != nil {
panic(err)
}
if resp.ContentLength > 0 && copiedLen != resp.ContentLength {
panic(fmt.Errorf("copied %d bytes, request body had %d", copiedLen, resp.ContentLength))
}
return tmpfile.Name()
}
func kebabToCamel(kebab string) string {
var buf strings.Builder
camel := ""
nextUpper := true
for _, c := range kebab {
if nextUpper {
buf.WriteRune(unicode.ToUpper(c))
camel += strings.ToUpper(string(c))
nextUpper = false
} else if c == '-' {
nextUpper = true
} else {
switch c {
case '-':
nextUpper = true
case '/':
nextUpper = true
buf.WriteByte('_')
default:
buf.WriteRune(c)
}
camel += string(c)
}
}
return buf.String()
return camel
}
func readFileFromZip(f *zip.File) string {
reader, err := f.Open()
if err != nil {
panic(err)
}
defer reader.Close()
bytes, err := ioutil.ReadAll(reader)
if err != nil {
panic(err)
}
return string(bytes)
}
func templateGoStr(input string) string {
return strconv.Quote(input)
if len(input) > 0 && input[len(input)-1] == '\n' {
input = input[0 : len(input)-1]
}
if strings.Contains(input, "`") {
lines := strings.Split(input, "\n")
for idx, line := range lines {
lines[idx] = strconv.Quote(line + "\n")
}
return strings.Join(lines, " + \n")
}
return "`" + input + "`"
}
var (
ref = flag.String("r", "master", "git reference")
out = flag.String("o", "", "output file")
)
func usage() {
@@ -100,59 +143,61 @@ func main() {
flag.Usage = usage
flag.Parse()
url := "https://codeload.github.com/BurntSushi/toml-test/zip/" + *ref
resultFile := downloadTmpFile(url)
defer os.Remove(resultFile)
log.Println("file written to", resultFile)
zipReader, err := zip.OpenReader(resultFile)
if err != nil {
panic(err)
}
defer zipReader.Close()
collection := testsCollection{
Ref: *ref,
Timestamp: time.Now().Format(time.RFC3339),
}
dirContent, _ := filepath.Glob("tests/invalid/**/*.toml")
for _, f := range dirContent {
filename := strings.TrimPrefix(f, "tests/valid/")
name := kebabToCamel(strings.TrimSuffix(filename, ".toml"))
name = strings.ReplaceAll(name, ".", "_")
zipFilesMap := map[string]*zip.File{}
log.Printf("> [%s] %s\n", "invalid", name)
tomlContent, err := os.ReadFile(f) // #nosec G304
if err != nil {
fmt.Printf("failed to read test file: %s\n", err)
os.Exit(1)
}
collection.Invalid = append(collection.Invalid, invalid{
Name: name,
Input: string(tomlContent),
})
collection.Count++
for _, f := range zipReader.File {
zipFilesMap[f.Name] = f
}
dirContent, _ = filepath.Glob("tests/valid/**/*.toml")
for _, f := range dirContent {
filename := strings.TrimPrefix(f, "tests/valid/")
name := kebabToCamel(strings.TrimSuffix(filename, ".toml"))
name = strings.ReplaceAll(name, ".", "_")
testFileRegexp := regexp.MustCompile(`([^/]+/tests/(valid|invalid)/(.+))\.(toml)`)
for _, f := range zipReader.File {
groups := testFileRegexp.FindStringSubmatch(f.Name)
if len(groups) > 0 {
name := kebabToCamel(groups[3])
testType := groups[2]
log.Printf("> [%s] %s\n", "valid", name)
log.Printf("> [%s] %s\n", testType, name)
tomlContent, err := os.ReadFile(f) // #nosec G304
if err != nil {
fmt.Printf("failed reading test file: %s\n", err)
os.Exit(1)
tomlContent := readFileFromZip(f)
switch testType {
case "invalid":
collection.Invalid = append(collection.Invalid, invalid{
Name: name,
Input: tomlContent,
})
collection.Count++
case "valid":
baseFilePath := groups[1]
jsonFilePath := baseFilePath + ".json"
jsonContent := readFileFromZip(zipFilesMap[jsonFilePath])
collection.Valid = append(collection.Valid, valid{
Name: name,
Input: tomlContent,
JsonRef: jsonContent,
})
collection.Count++
default:
panic(fmt.Sprintf("unknown test type: %s", testType))
}
}
filename = strings.TrimSuffix(f, ".toml")
jsonContent, err := os.ReadFile(filename + ".json") // #nosec G304
if err != nil {
fmt.Printf("failed reading validation json: %s\n", err)
os.Exit(1)
}
collection.Valid = append(collection.Valid, valid{
Name: name,
Input: string(tomlContent),
JSONRef: string(jsonContent),
})
collection.Count++
}
log.Printf("Collected %d tests from toml-test\n", collection.Count)
@@ -162,7 +207,7 @@ func main() {
}
t := template.Must(template.New("src").Funcs(funcMap).Parse(srcTemplate))
buf := new(bytes.Buffer)
err := t.Execute(buf, collection)
err = t.Execute(buf, collection)
if err != nil {
panic(err)
}
@@ -170,14 +215,5 @@ func main() {
if err != nil {
panic(err)
}
if *out == "" {
fmt.Println(string(outputBytes))
return
}
err = os.WriteFile(*out, outputBytes, 0o600)
if err != nil {
panic(err)
}
fmt.Println(string(outputBytes))
}
-549
View File
@@ -1,549 +0,0 @@
package toml
import (
"fmt"
"math"
"strconv"
"time"
"github.com/pelletier/go-toml/v2/unstable"
)
func parseInteger(b []byte) (int64, error) {
if len(b) > 2 && b[0] == '0' {
switch b[1] {
case 'x':
return parseIntHex(b)
case 'b':
return parseIntBin(b)
case 'o':
return parseIntOct(b)
default:
panic(fmt.Errorf("invalid base '%c', should have been checked by scanIntOrFloat", b[1]))
}
}
return parseIntDec(b)
}
func parseLocalDate(b []byte) (LocalDate, error) {
// full-date = date-fullyear "-" date-month "-" date-mday
// date-fullyear = 4DIGIT
// date-month = 2DIGIT ; 01-12
// date-mday = 2DIGIT ; 01-28, 01-29, 01-30, 01-31 based on month/year
var date LocalDate
if len(b) != 10 || b[4] != '-' || b[7] != '-' {
return date, unstable.NewParserError(b, "dates are expected to have the format YYYY-MM-DD")
}
var err error
date.Year, err = parseDecimalDigits(b[0:4])
if err != nil {
return LocalDate{}, err
}
date.Month, err = parseDecimalDigits(b[5:7])
if err != nil {
return LocalDate{}, err
}
date.Day, err = parseDecimalDigits(b[8:10])
if err != nil {
return LocalDate{}, err
}
if !isValidDate(date.Year, date.Month, date.Day) {
return LocalDate{}, unstable.NewParserError(b, "impossible date")
}
return date, nil
}
func parseDecimalDigits(b []byte) (int, error) {
v := 0
for i, c := range b {
if c < '0' || c > '9' {
return 0, unstable.NewParserError(b[i:i+1], "expected digit (0-9)")
}
v *= 10
v += int(c - '0')
}
return v, nil
}
func parseDateTime(b []byte) (time.Time, error) {
// offset-date-time = full-date time-delim full-time
// full-time = partial-time time-offset
// time-offset = "Z" / time-numoffset
// time-numoffset = ( "+" / "-" ) time-hour ":" time-minute
dt, b, err := parseLocalDateTime(b)
if err != nil {
return time.Time{}, err
}
var zone *time.Location
if len(b) == 0 {
// parser should have checked that when assigning the date time node
panic("date time should have a timezone")
}
if b[0] == 'Z' || b[0] == 'z' {
b = b[1:]
zone = time.UTC
} else {
const dateTimeByteLen = 6
if len(b) != dateTimeByteLen {
return time.Time{}, unstable.NewParserError(b, "invalid date-time timezone")
}
var direction int
switch b[0] {
case '-':
direction = -1
case '+':
direction = +1
default:
return time.Time{}, unstable.NewParserError(b[:1], "invalid timezone offset character")
}
if b[3] != ':' {
return time.Time{}, unstable.NewParserError(b[3:4], "expected a : separator")
}
hours, err := parseDecimalDigits(b[1:3])
if err != nil {
return time.Time{}, err
}
if hours > 23 {
return time.Time{}, unstable.NewParserError(b[:1], "invalid timezone offset hours")
}
minutes, err := parseDecimalDigits(b[4:6])
if err != nil {
return time.Time{}, err
}
if minutes > 59 {
return time.Time{}, unstable.NewParserError(b[:1], "invalid timezone offset minutes")
}
seconds := direction * (hours*3600 + minutes*60)
if seconds == 0 {
zone = time.UTC
} else {
zone = time.FixedZone("", seconds)
}
b = b[dateTimeByteLen:]
}
if len(b) > 0 {
return time.Time{}, unstable.NewParserError(b, "extra bytes at the end of the timezone")
}
t := time.Date(
dt.Year,
time.Month(dt.Month),
dt.Day,
dt.Hour,
dt.Minute,
dt.Second,
dt.Nanosecond,
zone)
return t, nil
}
func parseLocalDateTime(b []byte) (LocalDateTime, []byte, error) {
var dt LocalDateTime
const localDateTimeByteMinLen = 11
if len(b) < localDateTimeByteMinLen {
return dt, nil, unstable.NewParserError(b, "local datetimes are expected to have the format YYYY-MM-DDTHH:MM:SS[.NNNNNNNNN]")
}
date, err := parseLocalDate(b[:10])
if err != nil {
return dt, nil, err
}
dt.LocalDate = date
sep := b[10]
if sep != 'T' && sep != ' ' && sep != 't' {
return dt, nil, unstable.NewParserError(b[10:11], "datetime separator is expected to be T or a space")
}
t, rest, err := parseLocalTime(b[11:])
if err != nil {
return dt, nil, err
}
dt.LocalTime = t
return dt, rest, nil
}
// parseLocalTime is a bit different because it also returns the remaining
// []byte that is didn't need. This is to allow parseDateTime to parse those
// remaining bytes as a timezone.
func parseLocalTime(b []byte) (LocalTime, []byte, error) {
var (
nspow = [10]int{0, 1e8, 1e7, 1e6, 1e5, 1e4, 1e3, 1e2, 1e1, 1e0}
t LocalTime
)
// check if b matches to have expected format HH:MM:SS[.NNNNNN]
const localTimeByteLen = 8
if len(b) < localTimeByteLen {
return t, nil, unstable.NewParserError(b, "times are expected to have the format HH:MM:SS[.NNNNNN]")
}
var err error
t.Hour, err = parseDecimalDigits(b[0:2])
if err != nil {
return t, nil, err
}
if t.Hour > 23 {
return t, nil, unstable.NewParserError(b[0:2], "hour cannot be greater 23")
}
if b[2] != ':' {
return t, nil, unstable.NewParserError(b[2:3], "expecting colon between hours and minutes")
}
t.Minute, err = parseDecimalDigits(b[3:5])
if err != nil {
return t, nil, err
}
if t.Minute > 59 {
return t, nil, unstable.NewParserError(b[3:5], "minutes cannot be greater 59")
}
if b[5] != ':' {
return t, nil, unstable.NewParserError(b[5:6], "expecting colon between minutes and seconds")
}
t.Second, err = parseDecimalDigits(b[6:8])
if err != nil {
return t, nil, err
}
if t.Second > 59 {
return t, nil, unstable.NewParserError(b[6:8], "seconds cannot be greater than 59")
}
b = b[8:]
if len(b) >= 1 && b[0] == '.' {
frac := 0
precision := 0
digits := 0
for i, c := range b[1:] {
if !isDigit(c) {
if i == 0 {
return t, nil, unstable.NewParserError(b[0:1], "need at least one digit after fraction point")
}
break
}
digits++
const maxFracPrecision = 9
if i >= maxFracPrecision {
// go-toml allows decoding fractional seconds
// beyond the supported precision of 9
// digits. It truncates the fractional component
// to the supported precision and ignores the
// remaining digits.
//
// https://github.com/pelletier/go-toml/discussions/707
continue
}
frac *= 10
frac += int(c - '0')
precision++
}
if precision == 0 {
return t, nil, unstable.NewParserError(b[:1], "nanoseconds need at least one digit")
}
t.Nanosecond = frac * nspow[precision]
t.Precision = precision
return t, b[1+digits:], nil
}
return t, b, nil
}
func parseFloat(b []byte) (float64, error) {
if len(b) == 4 && (b[0] == '+' || b[0] == '-') && b[1] == 'n' && b[2] == 'a' && b[3] == 'n' {
return math.NaN(), nil
}
cleaned, err := checkAndRemoveUnderscoresFloats(b)
if err != nil {
return 0, err
}
if cleaned[0] == '.' {
return 0, unstable.NewParserError(b, "float cannot start with a dot")
}
if cleaned[len(cleaned)-1] == '.' {
return 0, unstable.NewParserError(b, "float cannot end with a dot")
}
dotAlreadySeen := false
for i, c := range cleaned {
if c == '.' {
if dotAlreadySeen {
return 0, unstable.NewParserError(b[i:i+1], "float can have at most one decimal point")
}
if !isDigit(cleaned[i-1]) {
return 0, unstable.NewParserError(b[i-1:i+1], "float decimal point must be preceded by a digit")
}
if !isDigit(cleaned[i+1]) {
return 0, unstable.NewParserError(b[i:i+2], "float decimal point must be followed by a digit")
}
dotAlreadySeen = true
}
}
start := 0
if cleaned[0] == '+' || cleaned[0] == '-' {
start = 1
}
if cleaned[start] == '0' && len(cleaned) > start+1 && isDigit(cleaned[start+1]) {
return 0, unstable.NewParserError(b, "float integer part cannot have leading zeroes")
}
f, err := strconv.ParseFloat(string(cleaned), 64)
if err != nil {
return 0, unstable.NewParserError(b, "unable to parse float: %w", err)
}
return f, nil
}
func parseIntHex(b []byte) (int64, error) {
cleaned, err := checkAndRemoveUnderscoresIntegers(b[2:])
if err != nil {
return 0, err
}
i, err := strconv.ParseInt(string(cleaned), 16, 64)
if err != nil {
return 0, unstable.NewParserError(b, "couldn't parse hexadecimal number: %w", err)
}
return i, nil
}
func parseIntOct(b []byte) (int64, error) {
cleaned, err := checkAndRemoveUnderscoresIntegers(b[2:])
if err != nil {
return 0, err
}
i, err := strconv.ParseInt(string(cleaned), 8, 64)
if err != nil {
return 0, unstable.NewParserError(b, "couldn't parse octal number: %w", err)
}
return i, nil
}
func parseIntBin(b []byte) (int64, error) {
cleaned, err := checkAndRemoveUnderscoresIntegers(b[2:])
if err != nil {
return 0, err
}
i, err := strconv.ParseInt(string(cleaned), 2, 64)
if err != nil {
return 0, unstable.NewParserError(b, "couldn't parse binary number: %w", err)
}
return i, nil
}
func isSign(b byte) bool {
return b == '+' || b == '-'
}
func parseIntDec(b []byte) (int64, error) {
cleaned, err := checkAndRemoveUnderscoresIntegers(b)
if err != nil {
return 0, err
}
startIdx := 0
if isSign(cleaned[0]) {
startIdx++
}
if len(cleaned) > startIdx+1 && cleaned[startIdx] == '0' {
return 0, unstable.NewParserError(b, "leading zero not allowed on decimal number")
}
i, err := strconv.ParseInt(string(cleaned), 10, 64)
if err != nil {
return 0, unstable.NewParserError(b, "couldn't parse decimal number: %w", err)
}
return i, nil
}
func checkAndRemoveUnderscoresIntegers(b []byte) ([]byte, error) {
start := 0
if b[start] == '+' || b[start] == '-' {
start++
}
if len(b) == start {
return b, nil
}
if b[start] == '_' {
return nil, unstable.NewParserError(b[start:start+1], "number cannot start with underscore")
}
if b[len(b)-1] == '_' {
return nil, unstable.NewParserError(b[len(b)-1:], "number cannot end with underscore")
}
// fast path
i := 0
for ; i < len(b); i++ {
if b[i] == '_' {
break
}
}
if i == len(b) {
return b, nil
}
before := false
cleaned := make([]byte, i, len(b))
copy(cleaned, b)
for i++; i < len(b); i++ {
c := b[i]
if c == '_' {
if !before {
return nil, unstable.NewParserError(b[i-1:i+1], "number must have at least one digit between underscores")
}
before = false
} else {
before = true
cleaned = append(cleaned, c)
}
}
return cleaned, nil
}
func checkAndRemoveUnderscoresFloats(b []byte) ([]byte, error) {
if b[0] == '_' {
return nil, unstable.NewParserError(b[0:1], "number cannot start with underscore")
}
if b[len(b)-1] == '_' {
return nil, unstable.NewParserError(b[len(b)-1:], "number cannot end with underscore")
}
// fast path
i := 0
for ; i < len(b); i++ {
if b[i] == '_' {
break
}
}
if i == len(b) {
return b, nil
}
before := false
cleaned := make([]byte, 0, len(b))
for i := 0; i < len(b); i++ {
c := b[i]
switch c {
case '_':
if !before {
return nil, unstable.NewParserError(b[i-1:i+1], "number must have at least one digit between underscores")
}
if i < len(b)-1 && (b[i+1] == 'e' || b[i+1] == 'E') {
return nil, unstable.NewParserError(b[i+1:i+2], "cannot have underscore before exponent")
}
before = false
case '+', '-':
// signed exponents
cleaned = append(cleaned, c)
before = false
case 'e', 'E':
if i < len(b)-1 && b[i+1] == '_' {
return nil, unstable.NewParserError(b[i+1:i+2], "cannot have underscore after exponent")
}
cleaned = append(cleaned, c)
case '.':
if i < len(b)-1 && b[i+1] == '_' {
return nil, unstable.NewParserError(b[i+1:i+2], "cannot have underscore after decimal point")
}
if i > 0 && b[i-1] == '_' {
return nil, unstable.NewParserError(b[i-1:i], "cannot have underscore before decimal point")
}
cleaned = append(cleaned, c)
default:
before = true
cleaned = append(cleaned, c)
}
}
return cleaned, nil
}
// isValidDate checks if a provided date is a date that exists.
func isValidDate(year int, month int, day int) bool {
return month > 0 && month < 13 && day > 0 && day <= daysIn(month, year)
}
// daysBefore[m] counts the number of days in a non-leap year
// before month m begins. There is an entry for m=12, counting
// the number of days before January of next year (365).
var daysBefore = [...]int32{
0,
31,
31 + 28,
31 + 28 + 31,
31 + 28 + 31 + 30,
31 + 28 + 31 + 30 + 31,
31 + 28 + 31 + 30 + 31 + 30,
31 + 28 + 31 + 30 + 31 + 30 + 31,
31 + 28 + 31 + 30 + 31 + 30 + 31 + 31,
31 + 28 + 31 + 30 + 31 + 30 + 31 + 31 + 30,
31 + 28 + 31 + 30 + 31 + 30 + 31 + 31 + 30 + 31,
31 + 28 + 31 + 30 + 31 + 30 + 31 + 31 + 30 + 31 + 30,
31 + 28 + 31 + 30 + 31 + 30 + 31 + 31 + 30 + 31 + 30 + 31,
}
func daysIn(m int, year int) int {
if m == 2 && isLeap(year) {
return 29
}
return int(daysBefore[m] - daysBefore[m-1])
}
func isLeap(year int) bool {
return year%4 == 0 && (year%100 != 0 || year%400 == 0)
}
func isDigit(r byte) bool {
return r >= '0' && r <= '9'
}
+22 -1
View File
@@ -1,2 +1,23 @@
// Package toml is a library to read and write TOML documents.
// Package toml is a TOML parser and manipulation library.
//
// This version supports the specification as described in
// https://github.com/toml-lang/toml/blob/master/versions/en/toml-v0.5.0.md
//
// Marshaling
//
// Go-toml can marshal and unmarshal TOML documents from and to data
// structures.
//
// TOML document as a tree
//
// Go-toml can operate on a TOML document as a tree. Use one of the Load*
// functions to parse TOML data and obtain a Tree instance, then one of its
// methods to manipulate the tree.
//
// JSONPath-like queries
//
// The package github.com/pelletier/go-toml/query implements a system
// similar to JSONPath to quickly retrieve elements of a TOML document using a
// single expression. See the package documentation for more information.
//
package toml
+170
View File
@@ -0,0 +1,170 @@
// code examples for godoc
package toml_test
import (
"fmt"
"log"
"os"
toml "github.com/pelletier/go-toml"
)
func Example_tree() {
config, err := toml.LoadFile("config.toml")
if err != nil {
fmt.Println("Error ", err.Error())
} else {
// retrieve data directly
directUser := config.Get("postgres.user").(string)
directPassword := config.Get("postgres.password").(string)
fmt.Println("User is", directUser, " and password is", directPassword)
// or using an intermediate object
configTree := config.Get("postgres").(*toml.Tree)
user := configTree.Get("user").(string)
password := configTree.Get("password").(string)
fmt.Println("User is", user, " and password is", password)
// show where elements are in the file
fmt.Printf("User position: %v\n", configTree.GetPosition("user"))
fmt.Printf("Password position: %v\n", configTree.GetPosition("password"))
}
}
func Example_unmarshal() {
type Employer struct {
Name string
Phone string
}
type Person struct {
Name string
Age int64
Employer Employer
}
document := []byte(`
name = "John"
age = 30
[employer]
name = "Company Inc."
phone = "+1 234 567 89012"
`)
person := Person{}
toml.Unmarshal(document, &person)
fmt.Println(person.Name, "is", person.Age, "and works at", person.Employer.Name)
// Output:
// John is 30 and works at Company Inc.
}
func ExampleMarshal() {
type Postgres struct {
User string `toml:"user"`
Password string `toml:"password"`
Database string `toml:"db" commented:"true" comment:"not used anymore"`
}
type Config struct {
Postgres Postgres `toml:"postgres" comment:"Postgres configuration"`
}
config := Config{Postgres{User: "pelletier", Password: "mypassword", Database: "old_database"}}
b, err := toml.Marshal(config)
if err != nil {
log.Fatal(err)
}
fmt.Println(string(b))
// Output:
// # Postgres configuration
// [postgres]
//
// # not used anymore
// # db = "old_database"
// password = "mypassword"
// user = "pelletier"
}
func ExampleUnmarshal() {
type Postgres struct {
User string
Password string
}
type Config struct {
Postgres Postgres
}
doc := []byte(`
[postgres]
user = "pelletier"
password = "mypassword"`)
config := Config{}
toml.Unmarshal(doc, &config)
fmt.Println("user=", config.Postgres.User)
// Output:
// user= pelletier
}
func ExampleEncoder_anonymous() {
type Credentials struct {
User string `toml:"user"`
Password string `toml:"password"`
}
type Protocol struct {
Name string `toml:"name"`
}
type Config struct {
Version int `toml:"version"`
Credentials
Protocol `toml:"Protocol"`
}
config := Config{
Version: 2,
Credentials: Credentials{
User: "pelletier",
Password: "mypassword",
},
Protocol: Protocol{
Name: "tcp",
},
}
fmt.Println("Default:")
fmt.Println("---------------")
def := toml.NewEncoder(os.Stdout)
if err := def.Encode(config); err != nil {
log.Fatal(err)
}
fmt.Println("---------------")
fmt.Println("With promotion:")
fmt.Println("---------------")
prom := toml.NewEncoder(os.Stdout).PromoteAnonymous(true)
if err := prom.Encode(config); err != nil {
log.Fatal(err)
}
// Output:
// Default:
// ---------------
// password = "mypassword"
// user = "pelletier"
// version = 2
//
// [Protocol]
// name = "tcp"
// ---------------
// With promotion:
// ---------------
// version = 2
//
// [Credentials]
// password = "mypassword"
// user = "pelletier"
//
// [Protocol]
// name = "tcp"
}
-283
View File
@@ -1,283 +0,0 @@
package toml
import (
"fmt"
"reflect"
"strconv"
"strings"
"github.com/pelletier/go-toml/v2/unstable"
)
// DecodeError represents an error encountered during the parsing or decoding
// of a TOML document.
//
// In addition to the error message, it contains the position in the document
// where it happened, as well as a human-readable representation that shows
// where the error occurred in the document.
type DecodeError struct {
message string
line int
column int
key Key
human string
}
// StrictMissingError occurs in a TOML document that does not have a
// corresponding field in the target value. It contains all the missing fields
// in Errors.
//
// Emitted by Decoder when DisallowUnknownFields() was called.
type StrictMissingError struct {
// One error per field that could not be found.
Errors []DecodeError
}
// Error returns the canonical string for this error.
func (s *StrictMissingError) Error() string {
return "strict mode: fields in the document are missing in the target struct"
}
// String returns a human readable description of all errors.
func (s *StrictMissingError) String() string {
var buf strings.Builder
for i, e := range s.Errors {
if i > 0 {
buf.WriteString("\n---\n")
}
buf.WriteString(e.String())
}
return buf.String()
}
// Unwrap returns wrapped decode errors
//
// Implements errors.Join() interface.
func (s *StrictMissingError) Unwrap() []error {
errs := make([]error, len(s.Errors))
for i := range s.Errors {
errs[i] = &s.Errors[i]
}
return errs
}
// Key represents a TOML key as a sequence of key parts.
type Key []string
// Error returns the error message contained in the DecodeError.
func (e *DecodeError) Error() string {
return "toml: " + e.message
}
// String returns the human-readable contextualized error. This string is multi-line.
func (e *DecodeError) String() string {
return e.human
}
// Position returns the (line, column) pair indicating where the error
// occurred in the document. Positions are 1-indexed.
func (e *DecodeError) Position() (row int, column int) {
return e.line, e.column
}
// Key that was being processed when the error occurred. The key is present only
// if this DecodeError is part of a StrictMissingError.
func (e *DecodeError) Key() Key {
return e.key
}
// wrapDecodeError creates a DecodeError referencing a highlighted
// range of bytes from document.
//
// highlight needs to be a sub-slice of document, or this function panics.
//
// The function copies all bytes used in DecodeError, so that document and
// highlight can be freely deallocated.
//
//nolint:funlen
func wrapDecodeError(document []byte, de *unstable.ParserError) *DecodeError {
offset := subsliceOffset(document, de.Highlight)
errMessage := de.Error()
errLine, errColumn := positionAtEnd(document[:offset])
before, after := linesOfContext(document, de.Highlight, offset, 3)
var buf strings.Builder
maxLine := errLine + len(after) - 1
lineColumnWidth := len(strconv.Itoa(maxLine))
// Write the lines of context strictly before the error.
for i := len(before) - 1; i > 0; i-- {
line := errLine - i
buf.WriteString(formatLineNumber(line, lineColumnWidth))
buf.WriteString("|")
if len(before[i]) > 0 {
buf.WriteString(" ")
buf.Write(before[i])
}
buf.WriteRune('\n')
}
// Write the document line that contains the error.
buf.WriteString(formatLineNumber(errLine, lineColumnWidth))
buf.WriteString("| ")
if len(before) > 0 {
buf.Write(before[0])
}
buf.Write(de.Highlight)
if len(after) > 0 {
buf.Write(after[0])
}
buf.WriteRune('\n')
// Write the line with the error message itself (so it does not have a line
// number).
buf.WriteString(strings.Repeat(" ", lineColumnWidth))
buf.WriteString("| ")
if len(before) > 0 {
buf.WriteString(strings.Repeat(" ", len(before[0])))
}
buf.WriteString(strings.Repeat("~", len(de.Highlight)))
if len(errMessage) > 0 {
buf.WriteString(" ")
buf.WriteString(errMessage)
}
// Write the lines of context strictly after the error.
for i := 1; i < len(after); i++ {
buf.WriteRune('\n')
line := errLine + i
buf.WriteString(formatLineNumber(line, lineColumnWidth))
buf.WriteString("|")
if len(after[i]) > 0 {
buf.WriteString(" ")
buf.Write(after[i])
}
}
return &DecodeError{
message: errMessage,
line: errLine,
column: errColumn,
key: de.Key,
human: buf.String(),
}
}
func formatLineNumber(line int, width int) string {
format := "%" + strconv.Itoa(width) + "d"
return fmt.Sprintf(format, line)
}
func linesOfContext(document []byte, highlight []byte, offset int, linesAround int) ([][]byte, [][]byte) {
return beforeLines(document, offset, linesAround), afterLines(document, highlight, offset, linesAround)
}
func beforeLines(document []byte, offset int, linesAround int) [][]byte {
var beforeLines [][]byte
// Walk the document backward from the highlight to find previous lines
// of context.
rest := document[:offset]
backward:
for o := len(rest) - 1; o >= 0 && len(beforeLines) <= linesAround && len(rest) > 0; {
switch {
case rest[o] == '\n':
// handle individual lines
beforeLines = append(beforeLines, rest[o+1:])
rest = rest[:o]
o = len(rest) - 1
case o == 0:
// add the first line only if it's non-empty
beforeLines = append(beforeLines, rest)
break backward
default:
o--
}
}
return beforeLines
}
func afterLines(document []byte, highlight []byte, offset int, linesAround int) [][]byte {
var afterLines [][]byte
// Walk the document forward from the highlight to find the following
// lines of context.
rest := document[offset+len(highlight):]
forward:
for o := 0; o < len(rest) && len(afterLines) <= linesAround; {
switch {
case rest[o] == '\n':
// handle individual lines
afterLines = append(afterLines, rest[:o])
rest = rest[o+1:]
o = 0
case o == len(rest)-1:
// add last line only if it's non-empty
afterLines = append(afterLines, rest)
break forward
default:
o++
}
}
return afterLines
}
func positionAtEnd(b []byte) (row int, column int) {
row = 1
column = 1
for _, c := range b {
if c == '\n' {
row++
column = 1
} else {
column++
}
}
return row, column
}
// subsliceOffset returns the byte offset of subslice within data.
// subslice must share the same backing array as data.
func subsliceOffset(data []byte, subslice []byte) int {
if len(subslice) == 0 {
return 0
}
// Use reflect to get the data pointers of both slices.
// This is safe because we're only reading the pointer values for comparison.
dataPtr := reflect.ValueOf(data).Pointer()
subPtr := reflect.ValueOf(subslice).Pointer()
offset := int(subPtr - dataPtr)
if offset < 0 || offset > len(data) {
panic("subslice is not within data")
}
return offset
}
-390
View File
@@ -1,390 +0,0 @@
package toml
import (
"bytes"
"errors"
"fmt"
"strings"
"testing"
"github.com/pelletier/go-toml/v2/internal/assert"
"github.com/pelletier/go-toml/v2/unstable"
)
//nolint:funlen
func TestDecodeError(t *testing.T) {
examples := []struct {
desc string
doc [3]string
msg string
expected string
}{
{
desc: "no context",
doc: [3]string{"", "morning", ""},
msg: "this is wrong",
expected: `
1| morning
| ~~~~~~~ this is wrong`,
},
{
desc: "one line",
doc: [3]string{"good ", "morning", " everyone"},
msg: "this is wrong",
expected: `
1| good morning everyone
| ~~~~~~~ this is wrong`,
},
{
desc: "exactly 3 lines",
doc: [3]string{`line1
line2
line3
before `, "highlighted", ` after
post line 1
post line 2
post line 3`},
msg: "this is wrong",
expected: `
1| line1
2| line2
3| line3
4| before highlighted after
| ~~~~~~~~~~~ this is wrong
5| post line 1
6| post line 2
7| post line 3`,
},
{
desc: "more than 3 lines",
doc: [3]string{`should not be seen1
should not be seen2
line1
line2
line3
before `, "highlighted", ` after
post line 1
post line 2
post line 3
should not be seen3
should not be seen4`},
msg: "this is wrong",
expected: `
3| line1
4| line2
5| line3
6| before highlighted after
| ~~~~~~~~~~~ this is wrong
7| post line 1
8| post line 2
9| post line 3`,
},
{
desc: "more than 10 total lines",
doc: [3]string{`should not be seen 0
should not be seen1
should not be seen2
should not be seen3
line1
line2
line3
before `, "highlighted", ` after
post line 1
post line 2
post line 3
should not be seen3
should not be seen4`},
msg: "this is wrong",
expected: `
5| line1
6| line2
7| line3
8| before highlighted after
| ~~~~~~~~~~~ this is wrong
9| post line 1
10| post line 2
11| post line 3`,
},
{
desc: "last line of more than 10",
doc: [3]string{`should not be seen
should not be seen
should not be seen
should not be seen
should not be seen
should not be seen
should not be seen
line1
line2
line3
before `, "highlighted", ``},
msg: "this is wrong",
expected: `
8| line1
9| line2
10| line3
11| before highlighted
| ~~~~~~~~~~~ this is wrong
`,
},
{
desc: "handle empty lines in the before/after blocks",
doc: [3]string{
`line1
line 2
before `, "highlighted", ` after
line 3
line 4
line 5`,
},
expected: `1| line1
2|
3| line 2
4| before highlighted after
| ~~~~~~~~~~~
5| line 3
6|
7| line 4`,
},
{
desc: "handle remainder of the error line when there is only one line",
doc: [3]string{`P=`, `[`, `#`},
msg: "array is incomplete",
expected: `1| P=[#
| ~ array is incomplete`,
},
}
for _, e := range examples {
e := e
t.Run(e.desc, func(t *testing.T) {
b := bytes.Buffer{}
b.WriteString(e.doc[0])
start := b.Len()
b.WriteString(e.doc[1])
end := b.Len()
b.WriteString(e.doc[2])
doc := b.Bytes()
hl := doc[start:end]
err := wrapDecodeError(doc, &unstable.ParserError{
Highlight: hl,
Message: e.msg,
})
var derr *DecodeError
if !errors.As(err, &derr) {
t.Errorf("error not in expected format")
return
}
assert.Equal(t, strings.Trim(e.expected, "\n"), derr.String())
})
}
}
func TestDecodeError_Accessors(t *testing.T) {
e := DecodeError{
message: "foo",
line: 1,
column: 2,
key: []string{"one", "two"},
human: "bar",
}
assert.Equal(t, "toml: foo", e.Error())
r, c := e.Position()
assert.Equal(t, 1, r)
assert.Equal(t, 2, c)
assert.Equal(t, Key{"one", "two"}, e.Key())
assert.Equal(t, "bar", e.String())
}
func TestDecodeError_DuplicateContent(t *testing.T) {
// This test verifies that when the same content appears multiple times
// in the document, the error correctly points to the actual location
// of the error, not the first occurrence of the content.
//
// The document has "1__2" on line 1 and "3__4" on line 2.
// Both have "__" which is invalid, but we want to ensure errors
// on line 2 report line 2, not line 1.
doc := `a = 1
b = 3__4`
var v map[string]int
err := Unmarshal([]byte(doc), &v)
var derr *DecodeError
if !errors.As(err, &derr) {
t.Fatal("error not in expected format")
}
row, col := derr.Position()
// The error should be on line 2 where "3__4" is
if row != 2 {
t.Errorf("expected error on row 2, got row %d", row)
}
// Column should point to the "__" part (after "3")
if col < 5 {
t.Errorf("expected error at column >= 5, got column %d", col)
}
}
func TestDecodeError_Position(t *testing.T) {
// Test that error positions are correctly reported for various error locations
examples := []struct {
name string
doc string
expectedRow int
minCol int
}{
{
name: "error on first line",
doc: `a = 1__2`,
expectedRow: 1,
minCol: 5,
},
{
name: "error on second line",
doc: "a = 1\nb = 2__3",
expectedRow: 2,
minCol: 5,
},
{
name: "error on third line",
doc: "a = 1\nb = 2\nc = 3__4",
expectedRow: 3,
minCol: 5,
},
{
name: "missing equals on last line without trailing newline",
doc: "a = 1\nb = 2\nc",
expectedRow: 3,
minCol: 1,
},
}
for _, e := range examples {
t.Run(e.name, func(t *testing.T) {
var v map[string]int
err := Unmarshal([]byte(e.doc), &v)
var derr *DecodeError
if !errors.As(err, &derr) {
t.Fatal("error not in expected format")
}
row, col := derr.Position()
assert.Equal(t, e.expectedRow, row)
if col < e.minCol {
t.Errorf("expected column >= %d, got %d", e.minCol, col)
}
})
}
}
func TestStrictErrorUnwrap(t *testing.T) {
fo := bytes.NewBufferString(`
Missing = 1
OtherMissing = 1
`)
var out struct{}
err := NewDecoder(fo).DisallowUnknownFields().Decode(&out)
assert.Error(t, err)
strictErr := &StrictMissingError{}
assert.True(t, errors.As(err, &strictErr))
assert.Equal(t, 2, len(strictErr.Unwrap()))
}
func TestDecodeError_PositionAfterComment(t *testing.T) {
// Regression test for https://github.com/pelletier/go-toml/issues/1047
// Error positions must be correct when the error occurs after comments or
// other content that was already scanned past.
examples := []struct {
desc string
doc string
expectedRow int
expectedCol int
expectedStr string
}{
{
desc: "invalid key after comment",
doc: "# comment\n= \"value\"",
expectedRow: 2,
expectedCol: 1,
expectedStr: "1| # comment\n2| = \"value\"\n | ~ invalid character at start of key: =",
},
{
desc: "invalid key after two comments",
doc: "# one\n# two\n= \"value\"",
expectedRow: 3,
expectedCol: 1,
expectedStr: "1| # one\n2| # two\n3| = \"value\"\n | ~ invalid character at start of key: =",
},
{
desc: "invalid key after key-value pair",
doc: "a = 1\n= 2",
expectedRow: 2,
expectedCol: 1,
expectedStr: "1| a = 1\n2| = 2\n | ~ invalid character at start of key: =",
},
{
desc: "invalid key after blank line",
doc: "a = 1\n\n= 2",
expectedRow: 3,
expectedCol: 1,
expectedStr: "1| a = 1\n2|\n3| = 2\n | ~ invalid character at start of key: =",
},
}
for _, e := range examples {
t.Run(e.desc, func(t *testing.T) {
var v interface{}
err := Unmarshal([]byte(e.doc), &v)
if err == nil {
t.Fatal("expected an error")
}
var derr *DecodeError
if !errors.As(err, &derr) {
t.Fatalf("error not a *DecodeError: %T: %v", err, err)
}
row, col := derr.Position()
if row != e.expectedRow {
t.Errorf("row: got %d, want %d (error: %s)", row, e.expectedRow, derr.String())
}
if col != e.expectedCol {
t.Errorf("col: got %d, want %d (error: %s)", col, e.expectedCol, derr.String())
}
assert.Equal(t, e.expectedStr, derr.String())
})
}
}
func ExampleDecodeError() {
doc := `name = 123__456`
s := map[string]interface{}{}
err := Unmarshal([]byte(doc), &s)
fmt.Println(err)
var derr *DecodeError
if errors.As(err, &derr) {
fmt.Println(derr.String())
row, col := derr.Position()
fmt.Println("error occurred at row", row, "column", col)
}
// Output:
// toml: number must have at least one digit between underscores
// 1| name = 123__456
// | ~~ number must have at least one digit between underscores
// error occurred at row 1 column 11
}
+30
View File
@@ -0,0 +1,30 @@
# This is a TOML document. Boom.
title = "TOML Example"
[owner]
name = "Tom Preston-Werner"
organization = "GitHub"
bio = "GitHub Cofounder & CEO\nLikes tater tots and beer."
dob = 1979-05-27T07:32:00Z # First class dates? Why not?
[database]
server = "192.168.1.1"
ports = [ 8001, 8001, 8002 ]
connection_max = 5000
enabled = true
[servers]
# You can indent as you please. Tabs or spaces. TOML don't care.
[servers.alpha]
ip = "10.0.0.1"
dc = "eqdc10"
[servers.beta]
ip = "10.0.0.2"
dc = "eqdc10"
[clients]
data = [ ["gamma", "delta"], [1, 2] ] # just an update to make sure parsers support it
score = 4e-08 # to make sure leading zeroes in exponent parts of floats are supported
+30
View File
@@ -0,0 +1,30 @@
# This is a TOML document. Boom.
title = "TOML Example"
[owner]
name = "Tom Preston-Werner"
organization = "GitHub"
bio = "GitHub Cofounder & CEO\nLikes tater tots and beer."
dob = 1979-05-27T07:32:00Z # First class dates? Why not?
[database]
server = "192.168.1.1"
ports = [ 8001, 8001, 8002 ]
connection_max = 5000
enabled = true
[servers]
# You can indent as you please. Tabs or spaces. TOML don't care.
[servers.alpha]
ip = "10.0.0.1"
dc = "eqdc10"
[servers.beta]
ip = "10.0.0.2"
dc = "eqdc10"
[clients]
data = [ ["gamma", "delta"], [1, 2] ] # just an update to make sure parsers support it
score = 4e-08 # to make sure leading zeroes in exponent parts of floats are supported
-37
View File
@@ -1,37 +0,0 @@
package toml_test
import (
"fmt"
"log"
"strconv"
"github.com/pelletier/go-toml/v2"
)
type customInt int
func (i *customInt) UnmarshalText(b []byte) error {
x, err := strconv.ParseInt(string(b), 10, 32)
if err != nil {
return err
}
*i = customInt(x * 100)
return nil
}
type doc struct {
Value customInt
}
func ExampleUnmarshal_textUnmarshal() {
var x doc
data := []byte(`value = "42"`)
err := toml.Unmarshal(data, &x)
if err != nil {
log.Fatal(err)
}
fmt.Println(x)
// Output:
// {4200}
}
-107
View File
@@ -1,107 +0,0 @@
package toml_test
import (
"testing"
"github.com/pelletier/go-toml/v2"
"github.com/pelletier/go-toml/v2/internal/assert"
)
func TestFastSimpleInt(t *testing.T) {
m := map[string]int64{}
err := toml.Unmarshal([]byte(`a = 42`), &m)
assert.NoError(t, err)
assert.Equal(t, map[string]int64{"a": 42}, m)
}
func TestFastSimpleFloat(t *testing.T) {
m := map[string]float64{}
err := toml.Unmarshal([]byte("a = 42\nb = 1.1\nc = 12341234123412341234123412341234"), &m)
assert.NoError(t, err)
assert.Equal(t, map[string]float64{"a": 42, "b": 1.1, "c": 1.2341234123412342e+31}, m)
}
func TestFastSimpleString(t *testing.T) {
m := map[string]string{}
err := toml.Unmarshal([]byte(`a = "hello"`), &m)
assert.NoError(t, err)
assert.Equal(t, map[string]string{"a": "hello"}, m)
}
func TestFastSimpleInterface(t *testing.T) {
m := map[string]interface{}{}
err := toml.Unmarshal([]byte(`
a = "hello"
b = 42`), &m)
assert.NoError(t, err)
assert.Equal(t, map[string]interface{}{
"a": "hello",
"b": int64(42),
}, m)
}
func TestFastMultipartKeyInterface(t *testing.T) {
m := map[string]interface{}{}
err := toml.Unmarshal([]byte(`
a.interim = "test"
a.b.c = "hello"
b = 42`), &m)
assert.NoError(t, err)
assert.Equal(t, map[string]interface{}{
"a": map[string]interface{}{
"interim": "test",
"b": map[string]interface{}{
"c": "hello",
},
},
"b": int64(42),
}, m)
}
func TestFastExistingMap(t *testing.T) {
m := map[string]interface{}{
"ints": map[string]int{},
}
err := toml.Unmarshal([]byte(`
ints.one = 1
ints.two = 2
strings.yo = "hello"`), &m)
assert.NoError(t, err)
assert.Equal(t, map[string]interface{}{
"ints": map[string]interface{}{
"one": int64(1),
"two": int64(2),
},
"strings": map[string]interface{}{
"yo": "hello",
},
}, m)
}
func TestFastArrayTable(t *testing.T) {
b := []byte(`
[root]
[[root.nested]]
name = 'Bob'
[[root.nested]]
name = 'Alice'
`)
m := map[string]interface{}{}
err := toml.Unmarshal(b, &m)
assert.NoError(t, err)
assert.Equal(t, map[string]interface{}{
"root": map[string]interface{}{
"nested": []interface{}{
map[string]interface{}{
"name": "Bob",
},
map[string]interface{}{
"name": "Alice",
},
},
},
}, m)
}
+31
View File
@@ -0,0 +1,31 @@
// +build gofuzz
package toml
func Fuzz(data []byte) int {
tree, err := LoadBytes(data)
if err != nil {
if tree != nil {
panic("tree must be nil if there is an error")
}
return 0
}
str, err := tree.ToTomlString()
if err != nil {
if str != "" {
panic(`str must be "" if there is an error`)
}
panic(err)
}
tree, err = Load(str)
if err != nil {
if tree != nil {
panic("tree must be nil if there is an error")
}
return 0
}
return 1
}
Executable
+15
View File
@@ -0,0 +1,15 @@
#! /bin/sh
set -eu
go get github.com/dvyukov/go-fuzz/go-fuzz
go get github.com/dvyukov/go-fuzz/go-fuzz-build
if [ ! -e toml-fuzz.zip ]; then
go-fuzz-build github.com/pelletier/go-toml
fi
rm -fr fuzz
mkdir -p fuzz/corpus
cp *.toml fuzz/corpus
go-fuzz -bin=toml-fuzz.zip -workdir=fuzz
-53
View File
@@ -1,53 +0,0 @@
package toml_test
import (
"os"
"strings"
"testing"
"github.com/pelletier/go-toml/v2"
"github.com/pelletier/go-toml/v2/internal/assert"
)
func FuzzUnmarshal(f *testing.F) {
file, err := os.ReadFile("benchmark/benchmark.toml")
if err != nil {
f.Error(err)
}
f.Add(file)
f.Fuzz(func(t *testing.T, b []byte) {
if strings.Contains(string(b), "nan") {
// Current limitation of testify.
// https://github.com/stretchr/testify/issues/624
t.Skip("can't compare NaNs")
}
t.Log("INITIAL DOCUMENT ===========================")
t.Log(string(b))
var v interface{}
err := toml.Unmarshal(b, &v)
if err != nil {
return
}
t.Log("DECODED VALUE ===========================")
t.Logf("%#+v", v)
encoded, err := toml.Marshal(v)
if err != nil {
t.Fatalf("cannot marshal unmarshaled document: %s", err)
}
t.Log("ENCODED DOCUMENT ===========================")
t.Log(string(encoded))
var v2 interface{}
err = toml.Unmarshal(encoded, &v2)
if err != nil {
t.Fatalf("failed round trip: %s", err)
}
assert.Equal(t, v, v2)
})
}
+2 -2
View File
@@ -1,3 +1,3 @@
module github.com/pelletier/go-toml/v2
module github.com/pelletier/go-toml
go 1.21.0
go 1.12
View File
-141
View File
@@ -1,141 +0,0 @@
// Package assert provides assertion functions for unit testing.
package assert
import (
"bytes"
"fmt"
"reflect"
"strings"
"testing"
)
// True asserts that an expression is true.
func True(tb testing.TB, ok bool, msgAndArgs ...any) {
tb.Helper()
if ok {
return
}
tb.Fatal(formatMsgAndArgs("Expected expression to be true", msgAndArgs...))
}
// False asserts that an expression is false.
func False(tb testing.TB, ok bool, msgAndArgs ...any) {
tb.Helper()
if !ok {
return
}
tb.Fatal(formatMsgAndArgs("Expected expression to be false", msgAndArgs...))
}
// Equal asserts that "expected" and "actual" are equal.
func Equal[T any](tb testing.TB, expected, actual T, msgAndArgs ...any) {
tb.Helper()
if objectsAreEqual(expected, actual) {
return
}
msg := formatMsgAndArgs("Expected values to be equal:", msgAndArgs...)
tb.Fatalf("%s\n%s", msg, diff(expected, actual))
}
// Error asserts that an error is not nil.
func Error(tb testing.TB, err error, msgAndArgs ...any) {
tb.Helper()
if err != nil {
return
}
tb.Fatal(formatMsgAndArgs("Expected an error", msgAndArgs...))
}
// NoError asserts that an error is nil.
func NoError(tb testing.TB, err error, msgAndArgs ...any) {
tb.Helper()
if err == nil {
return
}
msg := formatMsgAndArgs("Unexpected error:", msgAndArgs...)
tb.Fatalf("%s\n%+v", msg, err)
}
// Panics asserts that the given function panics.
func Panics(tb testing.TB, fn func(), msgAndArgs ...any) {
tb.Helper()
defer func() {
if recover() == nil {
msg := formatMsgAndArgs("Expected function to panic", msgAndArgs...)
tb.Fatal(msg)
}
}()
fn()
}
// Zero asserts that a value is its zero value.
func Zero[T any](tb testing.TB, value T, msgAndArgs ...any) {
tb.Helper()
var zero T
if objectsAreEqual(value, zero) {
return
}
val := reflect.ValueOf(value)
if (val.Kind() == reflect.Slice || val.Kind() == reflect.Map || val.Kind() == reflect.Array) && val.Len() == 0 {
return
}
msg := formatMsgAndArgs("Expected zero value but got:", msgAndArgs...)
tb.Fatalf("%s\n%v", msg, value)
}
func NotZero[T any](tb testing.TB, value T, msgAndArgs ...any) {
tb.Helper()
var zero T
if !objectsAreEqual(value, zero) {
val := reflect.ValueOf(value)
switch val.Kind() {
case reflect.Slice, reflect.Map, reflect.Array:
if val.Len() > 0 {
return
}
default:
return
}
}
msg := formatMsgAndArgs("Unexpected zero value:", msgAndArgs...)
tb.Fatalf("%s\n%v", msg, value)
}
func formatMsgAndArgs(msg string, args ...any) string {
if len(args) == 0 {
return msg
}
format, ok := args[0].(string)
if !ok {
panic("message argument must be a fmt string")
}
return fmt.Sprintf(format, args[1:]...)
}
func diff(expected, actual any) string {
lines := []string{
"expected:",
fmt.Sprintf("%v", expected),
"actual:",
fmt.Sprintf("%v", actual),
}
return strings.Join(lines, "\n")
}
func objectsAreEqual(expected, actual any) bool {
if expected == nil || actual == nil {
return expected == actual
}
if exp, eok := expected.([]byte); eok {
if act, aok := actual.([]byte); aok {
return bytes.Equal(exp, act)
}
}
if exp, eok := expected.(string); eok {
if act, aok := actual.(string); aok {
return exp == act
}
}
return reflect.DeepEqual(expected, actual)
}
-217
View File
@@ -1,217 +0,0 @@
package assert
import (
"errors"
"fmt"
"testing"
)
type Data struct {
Label string
Value int64
}
func TestBadMessage(t *testing.T) {
invalidMessage := func() { True(t, false, 1234) }
assertOk(t, "Non-fmt message value", func(tb testing.TB) {
tb.Helper()
Panics(tb, invalidMessage)
})
assertFail(t, "Non-fmt message value", func(tb testing.TB) {
tb.Helper()
True(tb, false, "example %s", "message")
})
}
func TestTrue(t *testing.T) {
assertOk(t, "Succeed", func(tb testing.TB) {
tb.Helper()
True(tb, 1 > 0)
})
assertFail(t, "Fail", func(tb testing.TB) {
tb.Helper()
True(tb, 1 < 0)
})
}
func TestFalse(t *testing.T) {
assertOk(t, "Succeed", func(tb testing.TB) {
tb.Helper()
False(tb, 1 < 0)
})
assertFail(t, "Fail", func(tb testing.TB) {
tb.Helper()
False(tb, 1 > 0)
})
}
func TestEqual(t *testing.T) {
assertOk(t, "Nil", func(tb testing.TB) {
tb.Helper()
Equal(tb, interface{}(nil), interface{}(nil))
})
assertOk(t, "Identical structs", func(tb testing.TB) {
tb.Helper()
Equal(tb, Data{"expected", 1234}, Data{"expected", 1234})
})
assertFail(t, "Different structs", func(tb testing.TB) {
tb.Helper()
Equal(tb, Data{"expected", 1234}, Data{"actual", 1234})
})
assertOk(t, "Identical numbers", func(tb testing.TB) {
tb.Helper()
Equal(tb, 1234, 1234)
})
assertFail(t, "Identical numbers", func(tb testing.TB) {
tb.Helper()
Equal(tb, 1234, 1324)
})
assertOk(t, "Zero-length byte arrays", func(tb testing.TB) {
tb.Helper()
Equal(tb, []byte(nil), []byte(""))
})
assertOk(t, "Identical byte arrays", func(tb testing.TB) {
tb.Helper()
Equal(tb, []byte{1, 2, 3, 4}, []byte{1, 2, 3, 4})
})
assertFail(t, "Different byte arrays", func(tb testing.TB) {
tb.Helper()
Equal(tb, []byte{1, 2, 3, 4}, []byte{1, 3, 2, 4})
})
assertOk(t, "Identical strings", func(tb testing.TB) {
tb.Helper()
Equal(tb, "example", "example")
})
assertFail(t, "Identical strings", func(tb testing.TB) {
tb.Helper()
Equal(tb, "example", "elpmaxe")
})
}
func TestError(t *testing.T) {
assertOk(t, "Error", func(tb testing.TB) {
tb.Helper()
Error(tb, errors.New("example"))
})
assertFail(t, "Nil", func(tb testing.TB) {
tb.Helper()
Error(tb, nil)
})
}
func TestNoError(t *testing.T) {
assertFail(t, "Error", func(tb testing.TB) {
tb.Helper()
NoError(tb, errors.New("example"))
})
assertOk(t, "Nil", func(tb testing.TB) {
tb.Helper()
NoError(tb, nil)
})
}
func TestPanics(t *testing.T) {
willPanic := func() { panic("example") }
wontPanic := func() {}
assertOk(t, "Will panic", func(tb testing.TB) {
tb.Helper()
Panics(tb, willPanic)
})
assertFail(t, "Won't panic", func(tb testing.TB) {
tb.Helper()
Panics(tb, wontPanic)
})
}
func TestZero(t *testing.T) {
assertOk(t, "Empty struct", func(tb testing.TB) {
tb.Helper()
Zero(tb, Data{})
})
assertFail(t, "Non-empty struct", func(tb testing.TB) {
tb.Helper()
Zero(tb, Data{Label: "example"})
})
assertOk(t, "Nil slice", func(tb testing.TB) {
tb.Helper()
var slice []int
Zero(tb, slice)
})
assertFail(t, "Non-empty slice", func(tb testing.TB) {
tb.Helper()
slice := []int{1, 2, 3, 4}
Zero(tb, slice)
})
assertOk(t, "Zero-length slice", func(tb testing.TB) {
tb.Helper()
slice := []int{}
Zero(tb, slice)
})
}
func TestNotZero(t *testing.T) {
assertFail(t, "Empty struct", func(tb testing.TB) {
tb.Helper()
zero := Data{}
NotZero(tb, zero)
})
assertOk(t, "Non-empty struct", func(tb testing.TB) {
tb.Helper()
notZero := Data{Label: "example"}
NotZero(tb, notZero)
})
assertFail(t, "Nil slice", func(tb testing.TB) {
tb.Helper()
var slice []int
NotZero(tb, slice)
})
assertFail(t, "Zero-length slice", func(tb testing.TB) {
tb.Helper()
slice := []int{}
NotZero(tb, slice)
})
assertOk(t, "Non-empty slice", func(tb testing.TB) {
tb.Helper()
slice := []int{1, 2, 3, 4}
NotZero(tb, slice)
})
}
type testCase struct {
*testing.T
failed string
}
func (t *testCase) Fatal(args ...interface{}) {
t.failed = fmt.Sprint(args...)
}
func (t *testCase) Fatalf(message string, args ...interface{}) {
t.failed = fmt.Sprintf(message, args...)
}
func assertFail(t *testing.T, name string, fn func(testing.TB)) {
t.Helper()
t.Run(name, func(t *testing.T) {
t.Helper()
test := &testCase{T: t}
fn(test)
if test.failed == "" {
t.Fatal("Test expected to fail but did not")
} else {
t.Log(test.failed)
}
})
}
func assertOk(t *testing.T, name string, fn func(testing.TB)) {
t.Helper()
t.Run(name, func(t *testing.T) {
t.Helper()
test := &testCase{T: t}
fn(test)
if test.failed != "" {
t.Fatal("Test expected to succeed but did not:\n", test.failed)
}
})
}
-42
View File
@@ -1,42 +0,0 @@
package characters
var invalidASCIITable = [256]bool{
0x00: true,
0x01: true,
0x02: true,
0x03: true,
0x04: true,
0x05: true,
0x06: true,
0x07: true,
0x08: true,
// 0x09 TAB
// 0x0A LF
0x0B: true,
0x0C: true,
// 0x0D CR
0x0E: true,
0x0F: true,
0x10: true,
0x11: true,
0x12: true,
0x13: true,
0x14: true,
0x15: true,
0x16: true,
0x17: true,
0x18: true,
0x19: true,
0x1A: true,
0x1B: true,
0x1C: true,
0x1D: true,
0x1E: true,
0x1F: true,
// 0x20 - 0x7E Printable ASCII characters
0x7F: true,
}
func InvalidASCII(b byte) bool {
return invalidASCIITable[b]
}
-175
View File
@@ -1,175 +0,0 @@
// Package characters provides functions for working with string encodings.
package characters
import (
"unicode/utf8"
)
// Utf8TomlValidAlreadyEscaped verifies that a given string is only made of
// valid UTF-8 characters allowed by the TOML spec:
//
// Any Unicode character may be used except those that must be escaped:
// quotation mark, backslash, and the control characters other than tab (U+0000
// to U+0008, U+000A to U+001F, U+007F).
//
// It is a copy of the Go 1.17 utf8.Valid implementation, tweaked to exit early
// when a character is not allowed.
//
// The returned slice is empty if the string is valid, or contains the bytes
// of the invalid character.
//
// quotation mark => already checked
// backslash => already checked
// 0-0x8 => invalid
// 0x9 => tab, ok
// 0xA - 0x1F => invalid
// 0x7F => invalid
func Utf8TomlValidAlreadyEscaped(p []byte) []byte {
// Fast path. Check for and skip 8 bytes of ASCII characters per iteration.
for len(p) >= 8 {
// Combining two 32 bit loads allows the same code to be used
// for 32 and 64 bit platforms.
// The compiler can generate a 32bit load for first32 and second32
// on many platforms. See test/codegen/memcombine.go.
first32 := uint32(p[0]) | uint32(p[1])<<8 | uint32(p[2])<<16 | uint32(p[3])<<24
second32 := uint32(p[4]) | uint32(p[5])<<8 | uint32(p[6])<<16 | uint32(p[7])<<24
if (first32|second32)&0x80808080 != 0 {
// Found a non ASCII byte (>= RuneSelf).
break
}
for i, b := range p[:8] {
if InvalidASCII(b) {
return p[i : i+1]
}
}
p = p[8:]
}
n := len(p)
for i := 0; i < n; {
pi := p[i]
if pi < utf8.RuneSelf {
if InvalidASCII(pi) {
return p[i : i+1]
}
i++
continue
}
x := first[pi]
if x == xx {
// Illegal starter byte.
return p[i : i+1]
}
size := int(x & 7)
if i+size > n {
// Short or invalid.
return p[i:n]
}
accept := acceptRanges[x>>4]
if c := p[i+1]; c < accept.lo || accept.hi < c {
return p[i : i+2]
} else if size == 2 { //revive:disable:empty-block
} else if c := p[i+2]; c < locb || hicb < c {
return p[i : i+3]
} else if size == 3 { //revive:disable:empty-block
} else if c := p[i+3]; c < locb || hicb < c {
return p[i : i+4]
}
i += size
}
return nil
}
// Utf8ValidNext returns the size of the next rune if valid, 0 otherwise.
func Utf8ValidNext(p []byte) int {
c := p[0]
if c < utf8.RuneSelf {
if InvalidASCII(c) {
return 0
}
return 1
}
x := first[c]
if x == xx {
// Illegal starter byte.
return 0
}
size := int(x & 7)
if size > len(p) {
// Short or invalid.
return 0
}
accept := acceptRanges[x>>4]
if c := p[1]; c < accept.lo || accept.hi < c {
return 0
} else if size == 2 { //nolint:revive
} else if c := p[2]; c < locb || hicb < c {
return 0
} else if size == 3 { //nolint:revive
} else if c := p[3]; c < locb || hicb < c {
return 0
}
return size
}
// acceptRange gives the range of valid values for the second byte in a UTF-8
// sequence.
type acceptRange struct {
lo uint8 // lowest value for second byte.
hi uint8 // highest value for second byte.
}
// acceptRanges has size 16 to avoid bounds checks in the code that uses it.
var acceptRanges = [16]acceptRange{
0: {locb, hicb},
1: {0xA0, hicb},
2: {locb, 0x9F},
3: {0x90, hicb},
4: {locb, 0x8F},
}
// first is information about the first byte in a UTF-8 sequence.
var first = [256]uint8{
// 1 2 3 4 5 6 7 8 9 A B C D E F
as, as, as, as, as, as, as, as, as, as, as, as, as, as, as, as, // 0x00-0x0F
as, as, as, as, as, as, as, as, as, as, as, as, as, as, as, as, // 0x10-0x1F
as, as, as, as, as, as, as, as, as, as, as, as, as, as, as, as, // 0x20-0x2F
as, as, as, as, as, as, as, as, as, as, as, as, as, as, as, as, // 0x30-0x3F
as, as, as, as, as, as, as, as, as, as, as, as, as, as, as, as, // 0x40-0x4F
as, as, as, as, as, as, as, as, as, as, as, as, as, as, as, as, // 0x50-0x5F
as, as, as, as, as, as, as, as, as, as, as, as, as, as, as, as, // 0x60-0x6F
as, as, as, as, as, as, as, as, as, as, as, as, as, as, as, as, // 0x70-0x7F
// 1 2 3 4 5 6 7 8 9 A B C D E F
xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, // 0x80-0x8F
xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, // 0x90-0x9F
xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, // 0xA0-0xAF
xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, // 0xB0-0xBF
xx, xx, s1, s1, s1, s1, s1, s1, s1, s1, s1, s1, s1, s1, s1, s1, // 0xC0-0xCF
s1, s1, s1, s1, s1, s1, s1, s1, s1, s1, s1, s1, s1, s1, s1, s1, // 0xD0-0xDF
s2, s3, s3, s3, s3, s3, s3, s3, s3, s3, s3, s3, s3, s4, s3, s3, // 0xE0-0xEF
s5, s6, s6, s6, s7, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, xx, // 0xF0-0xFF
}
const (
// The default lowest and highest continuation byte.
locb = 0b10000000
hicb = 0b10111111
// These names of these constants are chosen to give nice alignment in the
// table below. The first nibble is an index into acceptRanges or F for
// special one-byte cases. The second nibble is the Rune length or the
// Status for the special one-byte case.
xx = 0xF1 // invalid: size 1
as = 0xF0 // ASCII: size 1
s1 = 0x02 // accept 0, size 2
s2 = 0x13 // accept 1, size 3
s3 = 0x03 // accept 0, size 3
s4 = 0x23 // accept 2, size 3
s5 = 0x34 // accept 3, size 4
s6 = 0x04 // accept 0, size 4
s7 = 0x44 // accept 4, size 4
)
-87
View File
@@ -1,87 +0,0 @@
// Package cli provides common functions for command-line programs.
package cli
import (
"bytes"
"errors"
"flag"
"fmt"
"io"
"os"
"github.com/pelletier/go-toml/v2"
)
type ConvertFn func(r io.Reader, w io.Writer) error
type Program struct {
Usage string
Fn ConvertFn
// Inplace allows the command to take more than one file as argument and
// perform conversion in place on each provided file.
Inplace bool
}
func (p *Program) Execute() {
flag.Usage = func() { fmt.Fprint(os.Stderr, p.Usage) }
flag.Parse()
os.Exit(p.main(flag.Args(), os.Stdin, os.Stdout, os.Stderr))
}
func (p *Program) main(files []string, input io.Reader, output, stderr io.Writer) int {
err := p.run(files, input, output)
if err != nil {
var derr *toml.DecodeError
if errors.As(err, &derr) {
_, _ = fmt.Fprintln(stderr, derr.String())
row, col := derr.Position()
_, _ = fmt.Fprintln(stderr, "error occurred at row", row, "column", col)
} else {
_, _ = fmt.Fprintln(stderr, err.Error())
}
return -1
}
return 0
}
func (p *Program) run(files []string, input io.Reader, output io.Writer) error {
if len(files) > 0 {
if p.Inplace {
return p.runAllFilesInPlace(files)
}
f, err := os.Open(files[0])
if err != nil {
return err
}
defer func() { _ = f.Close() }()
input = f
}
return p.Fn(input, output)
}
func (p *Program) runAllFilesInPlace(files []string) error {
for _, path := range files {
err := p.runFileInPlace(path)
if err != nil {
return err
}
}
return nil
}
func (p *Program) runFileInPlace(path string) error {
in, err := os.ReadFile(path) // #nosec G304
if err != nil {
return err
}
out := new(bytes.Buffer)
err = p.Fn(bytes.NewReader(in), out)
if err != nil {
return err
}
return os.WriteFile(path, out.Bytes(), 0o600)
}
-166
View File
@@ -1,166 +0,0 @@
package cli
import (
"bytes"
"errors"
"io"
"os"
"path"
"strings"
"testing"
"github.com/pelletier/go-toml/v2"
"github.com/pelletier/go-toml/v2/internal/assert"
)
func processMain(args []string, input io.Reader, stdout, stderr io.Writer, f ConvertFn) int {
p := Program{Fn: f}
return p.main(args, input, stdout, stderr)
}
func TestProcessMainStdin(t *testing.T) {
stdout := new(bytes.Buffer)
stderr := new(bytes.Buffer)
input := strings.NewReader("this is the input")
exit := processMain([]string{}, input, stdout, stderr, func(io.Reader, io.Writer) error {
return nil
})
assert.Equal(t, 0, exit)
assert.Zero(t, stdout.String())
assert.Zero(t, stderr.String())
}
func TestProcessMainStdinErr(t *testing.T) {
stdout := new(bytes.Buffer)
stderr := new(bytes.Buffer)
input := strings.NewReader("this is the input")
exit := processMain([]string{}, input, stdout, stderr, func(io.Reader, io.Writer) error {
return errors.New("something bad")
})
assert.Equal(t, -1, exit)
assert.Zero(t, stdout.String())
assert.NotZero(t, stderr.String())
}
func TestProcessMainStdinDecodeErr(t *testing.T) {
stdout := new(bytes.Buffer)
stderr := new(bytes.Buffer)
input := strings.NewReader("this is the input")
exit := processMain([]string{}, input, stdout, stderr, func(io.Reader, io.Writer) error {
var v interface{}
return toml.Unmarshal([]byte(`qwe = 001`), &v)
})
assert.Equal(t, -1, exit)
assert.Zero(t, stdout.String())
assert.True(t, strings.Contains(stderr.String(), "error occurred at"))
}
func TestProcessMainFileExists(t *testing.T) {
tmpfile, err := os.CreateTemp(t.TempDir(), "example")
assert.NoError(t, err)
_, err = tmpfile.WriteString(`some data`)
assert.NoError(t, err)
assert.NoError(t, tmpfile.Close())
stdout := new(bytes.Buffer)
stderr := new(bytes.Buffer)
exit := processMain([]string{tmpfile.Name()}, nil, stdout, stderr, func(io.Reader, io.Writer) error {
return nil
})
assert.Equal(t, 0, exit)
assert.Zero(t, stdout.String())
assert.Zero(t, stderr.String())
}
func TestProcessMainFileDoesNotExist(t *testing.T) {
stdout := new(bytes.Buffer)
stderr := new(bytes.Buffer)
exit := processMain([]string{"/lets/hope/this/does/not/exist"}, nil, stdout, stderr, func(io.Reader, io.Writer) error {
return nil
})
assert.Equal(t, -1, exit)
assert.Zero(t, stdout.String())
assert.NotZero(t, stderr.String())
}
func TestProcessMainFilesInPlace(t *testing.T) {
dir := t.TempDir()
path1 := path.Join(dir, "file1")
path2 := path.Join(dir, "file2")
err := os.WriteFile(path1, []byte("content 1"), 0o600)
assert.NoError(t, err)
err = os.WriteFile(path2, []byte("content 2"), 0o600)
assert.NoError(t, err)
p := Program{
Fn: dummyFileFn,
Inplace: true,
}
exit := p.main([]string{path1, path2}, os.Stdin, os.Stdout, os.Stderr)
assert.Equal(t, 0, exit)
v1, err := os.ReadFile(path1)
assert.NoError(t, err)
assert.Equal(t, "1", string(v1))
v2, err := os.ReadFile(path2)
assert.NoError(t, err)
assert.Equal(t, "2", string(v2))
}
func TestProcessMainFilesInPlaceErrRead(t *testing.T) {
p := Program{
Fn: dummyFileFn,
Inplace: true,
}
exit := p.main([]string{"/this/path/is/invalid"}, os.Stdin, os.Stdout, os.Stderr)
assert.Equal(t, -1, exit)
}
func TestProcessMainFilesInPlaceFailFn(t *testing.T) {
dir := t.TempDir()
path1 := path.Join(dir, "file1")
err := os.WriteFile(path1, []byte("content 1"), 0o600)
assert.NoError(t, err)
p := Program{
Fn: func(io.Reader, io.Writer) error { return errors.New("oh no") },
Inplace: true,
}
exit := p.main([]string{path1}, os.Stdin, os.Stdout, os.Stderr)
assert.Equal(t, -1, exit)
v1, err := os.ReadFile(path1)
assert.NoError(t, err)
assert.Equal(t, "content 1", string(v1))
}
func dummyFileFn(r io.Reader, w io.Writer) error {
b, err := io.ReadAll(r)
if err != nil {
return err
}
v := strings.SplitN(string(b), " ", 2)[1]
_, err = w.Write([]byte(v))
return err
}
@@ -1,198 +0,0 @@
package imported_tests //revive:disable:var-naming
// Those tests have been imported from v1, but adjust to match the new
// defaults of v2.
import (
"fmt"
"testing"
"time"
"github.com/pelletier/go-toml/v2"
"github.com/pelletier/go-toml/v2/internal/assert"
)
func TestDocMarshal(t *testing.T) {
type testDoc struct {
Title string `toml:"title"`
BasicLists testDocBasicLists `toml:"basic_lists"`
SubDocPtrs []*testSubDoc `toml:"subdocptrs"`
BasicMap map[string]string `toml:"basic_map"`
Subdocs testDocSubs `toml:"subdoc"`
Basics testDocBasics `toml:"basic"`
SubDocList []testSubDoc `toml:"subdoclist"`
err int `toml:"shouldntBeHere"` //nolint:unused
unexported int `toml:"shouldntBeHere"`
Unexported2 int `toml:"-"`
}
docData := testDoc{
Title: "TOML Marshal Testing",
unexported: 0,
Unexported2: 0,
Basics: testDocBasics{
Bool: true,
Date: time.Date(1979, 5, 27, 7, 32, 0, 0, time.UTC),
Float32: 123.4,
Float64: 123.456782132399,
Int: 5000,
Uint: 5001,
String: &biteMe,
unexported: 0,
},
BasicLists: testDocBasicLists{
Floats: []*float32{&float1, &float2, &float3},
Bools: []bool{true, false, true},
Dates: []time.Time{
time.Date(1979, 5, 27, 7, 32, 0, 0, time.UTC),
time.Date(1980, 5, 27, 7, 32, 0, 0, time.UTC),
},
Ints: []int{8001, 8001, 8002},
Strings: []string{"One", "Two", "Three"},
UInts: []uint{5002, 5003},
},
BasicMap: map[string]string{
"one": "one",
"two": "two",
},
Subdocs: testDocSubs{
First: testSubDoc{"First", 0},
Second: &subdoc,
},
SubDocList: []testSubDoc{
{"List.First", 0},
{"List.Second", 0},
},
SubDocPtrs: []*testSubDoc{&subdoc},
}
marshalTestToml := `title = 'TOML Marshal Testing'
[basic_lists]
floats = [12.3, 45.6, 78.9]
bools = [true, false, true]
dates = [1979-05-27T07:32:00Z, 1980-05-27T07:32:00Z]
ints = [8001, 8001, 8002]
uints = [5002, 5003]
strings = ['One', 'Two', 'Three']
[[subdocptrs]]
name = 'Second'
[basic_map]
one = 'one'
two = 'two'
[subdoc]
[subdoc.second]
name = 'Second'
[subdoc.first]
name = 'First'
[basic]
uint = 5001
bool = true
float = 123.4
float64 = 123.456782132399
int = 5000
string = 'Bite me'
date = 1979-05-27T07:32:00Z
[[subdoclist]]
name = 'List.First'
[[subdoclist]]
name = 'List.Second'
`
result, err := toml.Marshal(docData)
assert.NoError(t, err)
assert.Equal(t, marshalTestToml, string(result))
}
func TestBasicMarshalQuotedKey(t *testing.T) {
result, err := toml.Marshal(quotedKeyMarshalTestData)
assert.NoError(t, err)
expected := `'Z.string-àéù' = 'Hello'
'Yfloat-𝟘' = 3.5
['Xsubdoc-àéù']
String2 = 'One'
[['W.sublist-𝟘']]
String2 = 'Two'
[['W.sublist-𝟘']]
String2 = 'Three'
`
assert.Equal(t, expected, string(result))
}
func TestEmptyMarshal(t *testing.T) {
type emptyMarshalTestStruct struct {
Title string `toml:"title"`
Bool bool `toml:"bool"`
Int int `toml:"int"`
String string `toml:"string"`
StringList []string `toml:"stringlist"`
Ptr *basicMarshalTestStruct `toml:"ptr"`
Map map[string]string `toml:"map"`
}
doc := emptyMarshalTestStruct{
Title: "Placeholder",
Bool: false,
Int: 0,
String: "",
StringList: []string{},
Ptr: nil,
Map: map[string]string{},
}
result, err := toml.Marshal(doc)
assert.NoError(t, err)
expected := `title = 'Placeholder'
bool = false
int = 0
string = ''
stringlist = []
[map]
`
assert.Equal(t, expected, string(result))
}
type textMarshaler struct {
FirstName string
LastName string
}
func (m textMarshaler) MarshalText() ([]byte, error) {
fullName := fmt.Sprintf("%s %s", m.FirstName, m.LastName)
return []byte(fullName), nil
}
func TestTextMarshaler(t *testing.T) {
type wrap struct {
TM textMarshaler
}
m := textMarshaler{FirstName: "Sally", LastName: "Fields"}
t.Run("at root", func(t *testing.T) {
_, err := toml.Marshal(m)
// in v2 we do not allow TextMarshaler at root
assert.Error(t, err)
})
t.Run("leaf", func(t *testing.T) {
res, err := toml.Marshal(wrap{m})
assert.NoError(t, err)
assert.Equal(t, "TM = 'Sally Fields'\n", string(res))
})
}
File diff suppressed because it is too large Load Diff
-75
View File
@@ -1,75 +0,0 @@
package testsuite
import (
"fmt"
"math"
"strconv"
"time"
"github.com/pelletier/go-toml/v2"
)
// addTag adds JSON tags to a data structure as expected by toml-test.
func addTag(tomlData interface{}) interface{} {
// Switch on the data type.
switch orig := tomlData.(type) {
default:
// return map[string]interface{}{}
panic(fmt.Sprintf("Unknown type: %T", tomlData))
// A table: we don't need to add any tags, just recurse for every table
// entry.
case map[string]interface{}:
typed := make(map[string]interface{}, len(orig))
for k, v := range orig {
typed[k] = addTag(v)
}
return typed
// An array: we don't need to add any tags, just recurse for every table
// entry.
case []map[string]interface{}:
typed := make([]map[string]interface{}, len(orig))
for i, v := range orig {
typed[i] = addTag(v).(map[string]interface{})
}
return typed
case []interface{}:
typed := make([]interface{}, len(orig))
for i, v := range orig {
typed[i] = addTag(v)
}
return typed
// Datetime: tag as datetime.
case toml.LocalTime:
return tag("time-local", orig.String())
case toml.LocalDate:
return tag("date-local", orig.String())
case toml.LocalDateTime:
return tag("datetime-local", orig.String())
case time.Time:
return tag("datetime", orig.Format("2006-01-02T15:04:05.999999999Z07:00"))
// Tag primitive values: bool, string, int, and float64.
case bool:
return tag("bool", strconv.FormatBool(orig))
case string:
return tag("string", orig)
case int64:
return tag("integer", strconv.FormatInt(orig, 10))
case float64:
// Special case for nan since NaN == NaN is false.
if math.IsNaN(orig) {
return tag("float", "nan")
}
return tag("float", fmt.Sprintf("%v", orig))
}
}
func tag(typeName string, data interface{}) map[string]interface{} {
return map[string]interface{}{
"type": typeName,
"value": data,
}
}
-244
View File
@@ -1,244 +0,0 @@
package testsuite
import (
"fmt"
"strconv"
"strings"
"testing"
"time"
)
func CmpJSON(t *testing.T, key string, want, have interface{}) {
t.Helper()
switch w := want.(type) {
case map[string]interface{}:
cmpJSONMaps(t, key, w, have)
case []interface{}:
cmpJSONArrays(t, key, w, have)
default:
t.Errorf(
"Key '%s' in expected output should be a map or a list of maps, but it's a %T",
key, want)
}
}
func cmpJSONMaps(t *testing.T, key string, want map[string]interface{}, have interface{}) {
t.Helper()
haveMap, ok := have.(map[string]interface{})
if !ok {
mismatch(t, key, "table", want, haveMap)
return
}
// Check to make sure both or neither are values.
if isValue(want) && !isValue(haveMap) {
t.Fatalf("Key '%s' is supposed to be a value, but the parser reports it as a table", key)
}
if !isValue(want) && isValue(haveMap) {
t.Fatalf("Key '%s' is supposed to be a table, but the parser reports it as a value", key)
}
if isValue(want) && isValue(haveMap) {
cmpJSONValues(t, key, want, haveMap)
return
}
// Check that the keys of each map are equivalent.
for k := range want {
if _, ok := haveMap[k]; !ok {
bunk := kjoin(key, k)
t.Fatalf("Could not find key '%s' in parser output.", bunk)
}
}
for k := range haveMap {
if _, ok := want[k]; !ok {
bunk := kjoin(key, k)
t.Fatalf("Could not find key '%s' in expected output.", bunk)
}
}
// Okay, now make sure that each value is equivalent.
for k := range want {
CmpJSON(t, kjoin(key, k), want[k], haveMap[k])
}
}
func cmpJSONArrays(t *testing.T, key string, want, have interface{}) {
t.Helper()
wantSlice, ok := want.([]interface{})
if !ok {
panic(fmt.Sprintf("'value' should be a JSON array when 'type=array', but it is a %T", want))
}
haveSlice, ok := have.([]interface{})
if !ok {
t.Fatalf("Malformed output from your encoder: 'value' is not a JSON array: %T", have)
}
if len(wantSlice) != len(haveSlice) {
t.Fatalf("Array lengths differ for key '%s':\n"+
" Expected: %d\n"+
" Your encoder: %d",
key, len(wantSlice), len(haveSlice))
}
for i := 0; i < len(wantSlice); i++ {
CmpJSON(t, key, wantSlice[i], haveSlice[i])
}
}
func cmpJSONValues(t *testing.T, key string, want, have map[string]interface{}) {
t.Helper()
wantType, ok := want["type"].(string)
if !ok {
panic(fmt.Sprintf("'type' should be a string, but it is a %T", want["type"]))
}
haveType, ok := have["type"].(string)
if !ok {
t.Fatalf("Malformed output from your encoder: 'type' is not a string: %T", have["type"])
}
if wantType != haveType {
valMismatch(t, key, wantType, haveType, want, have)
}
// If this is an array, then we've got to do some work to check equality.
if wantType == "array" {
cmpJSONArrays(t, key, want, have)
return
}
// Atomic values are always strings
wantVal, ok := want["value"].(string)
if !ok {
panic(fmt.Sprintf("'value' %v should be a string, but it is a %[1]T", want["value"]))
}
haveVal, ok := have["value"].(string)
if !ok {
panic(fmt.Sprintf("Malformed output from your encoder: %T is not a string", have["value"]))
}
// Excepting floats and datetimes, other values can be compared as strings.
switch wantType {
case "float":
cmpFloats(t, key, wantVal, haveVal)
case "datetime", "datetime-local", "date-local", "time-local":
cmpAsDatetimes(t, key, wantType, wantVal, haveVal)
default:
cmpAsStrings(t, key, wantVal, haveVal)
}
}
func cmpAsStrings(t *testing.T, key string, want, have string) {
t.Helper()
if want != have {
t.Fatalf("Values for key '%s' don't match:\n"+
" Expected: %s\n"+
" Your encoder: %s",
key, want, have)
}
}
func cmpFloats(t *testing.T, key string, want, have string) {
t.Helper()
// Special case for NaN, since NaN != NaN.
if strings.HasSuffix(want, "nan") || strings.HasSuffix(have, "nan") {
if want != have {
t.Fatalf("Values for key '%s' don't match:\n"+
" Expected: %v\n"+
" Your encoder: %v",
key, want, have)
}
return
}
wantF, err := strconv.ParseFloat(want, 64)
if err != nil {
panic(fmt.Sprintf("Could not read '%s' as a float value for key '%s'", want, key))
}
haveF, err := strconv.ParseFloat(have, 64)
if err != nil {
panic(fmt.Sprintf("Malformed output from your encoder: key '%s' is not a float: '%s'", key, have))
}
if wantF != haveF {
t.Fatalf("Values for key '%s' don't match:\n"+
" Expected: %v\n"+
" Your encoder: %v",
key, wantF, haveF)
}
}
var datetimeRepl = strings.NewReplacer(
" ", "T",
"t", "T",
"z", "Z")
var layouts = map[string]string{
"datetime": time.RFC3339Nano,
"datetime-local": "2006-01-02T15:04:05.999999999",
"date-local": "2006-01-02",
"time-local": "15:04:05",
}
func cmpAsDatetimes(t *testing.T, key string, kind, want, have string) {
t.Helper()
layout, ok := layouts[kind]
if !ok {
panic("should never happen")
}
wantT, err := time.Parse(layout, datetimeRepl.Replace(want))
if err != nil {
panic(fmt.Sprintf("Could not read '%s' as a datetime value for key '%s'", want, key))
}
haveT, err := time.Parse(layout, datetimeRepl.Replace(want))
if err != nil {
t.Fatalf("Malformed output from your encoder: key '%s' is not a datetime: '%s'", key, have)
return
}
if !wantT.Equal(haveT) {
t.Fatalf("Values for key '%s' don't match:\n"+
" Expected: %v\n"+
" Your encoder: %v",
key, wantT, haveT)
}
}
func kjoin(old, key string) string {
if len(old) == 0 {
return key
}
return old + "." + key
}
func isValue(m map[string]interface{}) bool {
if len(m) != 2 {
return false
}
if _, ok := m["type"]; !ok {
return false
}
if _, ok := m["value"]; !ok {
return false
}
return true
}
func mismatch(t *testing.T, key string, wantType string, want, have interface{}) {
t.Helper()
t.Fatalf("Key '%s' is not an %s but %[4]T:\n"+
" Expected: %#[3]v\n"+
" Your encoder: %#[4]v",
key, wantType, want, have)
}
func valMismatch(t *testing.T, key string, wantType, haveType string, want, have interface{}) {
t.Helper()
t.Fatalf("Key '%s' is not an %s but %s:\n"+
" Expected: %#[3]v\n"+
" Your encoder: %#[4]v",
key, wantType, haveType, want, have)
}
-116
View File
@@ -1,116 +0,0 @@
package testsuite
import (
"fmt"
"strconv"
"time"
"github.com/pelletier/go-toml/v2"
)
// Remove JSON tags to a data structure as returned by toml-test.
func rmTag(typedJSON interface{}) (interface{}, error) {
// Check if key is in the table m.
in := func(key string, m map[string]interface{}) bool {
_, ok := m[key]
return ok
}
// Switch on the data type.
switch v := typedJSON.(type) {
// Object: this can either be a TOML table or a primitive with tags.
case map[string]interface{}:
// This value represents a primitive: remove the tags and return just
// the primitive value.
if len(v) == 2 && in("type", v) && in("value", v) {
ut, err := untag(v)
if err != nil {
return ut, fmt.Errorf("tag.Remove: %w", err)
}
return ut, nil
}
// Table: remove tags on all children.
m := make(map[string]interface{}, len(v))
for k, v2 := range v {
var err error
m[k], err = rmTag(v2)
if err != nil {
return nil, err
}
}
return m, nil
// Array: remove tags from all items.
case []interface{}:
a := make([]interface{}, len(v))
for i := range v {
var err error
a[i], err = rmTag(v[i])
if err != nil {
return nil, err
}
}
return a, nil
}
// The top level must be an object or array.
return nil, fmt.Errorf("unrecognized JSON format '%T'", typedJSON)
}
// Return a primitive: read the "type" and convert the "value" to that.
func untag(typed map[string]interface{}) (interface{}, error) {
t := typed["type"].(string)
v := typed["value"].(string)
switch t {
case "string":
return v, nil
case "integer":
n, err := strconv.ParseInt(v, 10, 64)
if err != nil {
return nil, fmt.Errorf("untag: %w", err)
}
return n, nil
case "float":
f, err := strconv.ParseFloat(v, 64)
if err != nil {
return nil, fmt.Errorf("untag: %w", err)
}
return f, nil
// toml.LocalDate{Year:2020, Month:12, Day:12}
case "datetime":
return time.Parse("2006-01-02T15:04:05.999999999Z07:00", v)
case "datetime-local":
var t toml.LocalDateTime
err := t.UnmarshalText([]byte(v))
if err != nil {
return nil, fmt.Errorf("untag: %w", err)
}
return t, nil
case "date-local":
var t toml.LocalDate
err := t.UnmarshalText([]byte(v))
if err != nil {
return nil, fmt.Errorf("untag: %w", err)
}
return t, nil
case "time-local":
var t toml.LocalTime
err := t.UnmarshalText([]byte(v))
if err != nil {
return nil, fmt.Errorf("untag: %w", err)
}
return t, nil
case "bool":
switch v {
case "true":
return true, nil
case "false":
return false, nil
}
return nil, fmt.Errorf("untag: could not parse %q as a boolean", v)
}
return nil, fmt.Errorf("untag: unrecognized tag type %q", t)
}
-68
View File
@@ -1,68 +0,0 @@
// Package testsuite provides helper functions for interoperating with the
// language-agnostic TOML test suite at github.com/BurntSushi/toml-test.
package testsuite
import (
"encoding/json"
"fmt"
"os"
"github.com/pelletier/go-toml/v2"
)
// Marshal is a helper function for calling toml.Marshal
//
// Only needed to avoid package import loops.
func Marshal(v interface{}) ([]byte, error) {
return toml.Marshal(v)
}
// Unmarshal is a helper function for calling toml.Unmarshal.
//
// Only needed to avoid package import loops.
func Unmarshal(data []byte, v interface{}) error {
return toml.Unmarshal(data, v)
}
// ValueToTaggedJSON takes a data structure and returns the tagged JSON
// representation.
func ValueToTaggedJSON(doc interface{}) ([]byte, error) {
return json.MarshalIndent(addTag(doc), "", " ")
}
// DecodeStdin is a helper function for the toml-test binary interface. TOML input
// is read from STDIN and a resulting tagged JSON representation is written to
// STDOUT.
func DecodeStdin() error {
var decoded map[string]interface{}
if err := toml.NewDecoder(os.Stdin).Decode(&decoded); err != nil {
return fmt.Errorf("error decoding TOML: %w", err)
}
j := json.NewEncoder(os.Stdout)
j.SetIndent("", " ")
if err := j.Encode(addTag(decoded)); err != nil {
return fmt.Errorf("error encoding JSON: %w", err)
}
return nil
}
// EncodeStdin is a helper function for the toml-test binary interface. Tagged
// JSON is read from STDIN and a resulting TOML representation is written to
// STDOUT.
func EncodeStdin() error {
var j interface{}
err := json.NewDecoder(os.Stdin).Decode(&j)
if err != nil {
return err
}
rm, err := rmTag(j)
if err != nil {
return fmt.Errorf("removing tags: %w", err)
}
return toml.NewEncoder(os.Stdout).Encode(rm)
}
-48
View File
@@ -1,48 +0,0 @@
package tracker
import "github.com/pelletier/go-toml/v2/unstable"
// KeyTracker is a tracker that keeps track of the current Key as the AST is
// walked.
type KeyTracker struct {
k []string
}
// UpdateTable sets the state of the tracker with the AST table node.
func (t *KeyTracker) UpdateTable(node *unstable.Node) {
t.reset()
t.Push(node)
}
// UpdateArrayTable sets the state of the tracker with the AST array table node.
func (t *KeyTracker) UpdateArrayTable(node *unstable.Node) {
t.reset()
t.Push(node)
}
// Push the given key on the stack.
func (t *KeyTracker) Push(node *unstable.Node) {
it := node.Key()
for it.Next() {
t.k = append(t.k, string(it.Node().Data))
}
}
// Pop key from stack.
func (t *KeyTracker) Pop(node *unstable.Node) {
it := node.Key()
for it.Next() {
t.k = t.k[:len(t.k)-1]
}
}
// Key returns the current key.
func (t *KeyTracker) Key() []string {
k := make([]string, len(t.k))
copy(k, t.k)
return k
}
func (t *KeyTracker) reset() {
t.k = t.k[:0]
}
-359
View File
@@ -1,359 +0,0 @@
package tracker
import (
"bytes"
"fmt"
"sync"
"github.com/pelletier/go-toml/v2/unstable"
)
type keyKind uint8
const (
invalidKind keyKind = iota
valueKind
tableKind
arrayTableKind
)
func (k keyKind) String() string {
switch k {
case invalidKind:
return "invalid"
case valueKind:
return "value"
case tableKind:
return "table"
case arrayTableKind:
return "array table"
}
panic("missing keyKind string mapping")
}
// SeenTracker tracks which keys have been seen with which TOML type to flag
// duplicates and mismatches according to the spec.
//
// Each node in the visited tree is represented by an entry. Each entry has an
// identifier, which is provided by a counter. Entries are stored in the array
// entries. As new nodes are discovered (referenced for the first time in the
// TOML document), entries are created and appended to the array. An entry
// points to its parent using its id.
//
// To find whether a given key (sequence of []byte) has already been visited,
// the entries are linearly searched, looking for one with the right name and
// parent id.
//
// Given that all keys appear in the document after their parent, it is
// guaranteed that all descendants of a node are stored after the node, this
// speeds up the search process.
//
// When encountering [[array tables]], the descendants of that node are removed
// to allow that branch of the tree to be "rediscovered". To maintain the
// invariant above, the deletion process needs to keep the order of entries.
// This results in more copies in that case.
type SeenTracker struct {
entries []entry
currentIdx int
}
var pool = sync.Pool{
New: func() interface{} {
return &SeenTracker{}
},
}
func (s *SeenTracker) reset() {
// Always contains a root element at index 0.
s.currentIdx = 0
if len(s.entries) == 0 {
s.entries = make([]entry, 1, 2)
} else {
s.entries = s.entries[:1]
}
s.entries[0].child = -1
s.entries[0].next = -1
}
type entry struct {
// Use -1 to indicate no child or no sibling.
child int
next int
name []byte
kind keyKind
explicit bool
kv bool
}
// Find the index of the child of parentIdx with key k. Returns -1 if
// it does not exist.
func (s *SeenTracker) find(parentIdx int, k []byte) int {
for i := s.entries[parentIdx].child; i >= 0; i = s.entries[i].next {
if bytes.Equal(s.entries[i].name, k) {
return i
}
}
return -1
}
// Remove all descendants of node at position idx.
func (s *SeenTracker) clear(idx int) {
if idx >= len(s.entries) {
return
}
for i := s.entries[idx].child; i >= 0; {
next := s.entries[i].next
n := s.entries[0].next
s.entries[0].next = i
s.entries[i].next = n
s.entries[i].name = nil
s.clear(i)
i = next
}
s.entries[idx].child = -1
}
func (s *SeenTracker) create(parentIdx int, name []byte, kind keyKind, explicit bool, kv bool) int {
e := entry{
child: -1,
next: s.entries[parentIdx].child,
name: name,
kind: kind,
explicit: explicit,
kv: kv,
}
var idx int
if s.entries[0].next >= 0 {
idx = s.entries[0].next
s.entries[0].next = s.entries[idx].next
s.entries[idx] = e
} else {
idx = len(s.entries)
s.entries = append(s.entries, e)
}
s.entries[parentIdx].child = idx
return idx
}
func (s *SeenTracker) setExplicitFlag(parentIdx int) {
for i := s.entries[parentIdx].child; i >= 0; i = s.entries[i].next {
if s.entries[i].kv {
s.entries[i].explicit = true
s.entries[i].kv = false
}
s.setExplicitFlag(i)
}
}
// CheckExpression takes a top-level node and checks that it does not contain
// keys that have been seen in previous calls, and validates that types are
// consistent. It returns true if it is the first time this node's key is seen.
// Useful to clear array tables on first use.
func (s *SeenTracker) CheckExpression(node *unstable.Node) (bool, error) {
if s.entries == nil {
s.reset()
}
switch node.Kind {
case unstable.KeyValue:
return s.checkKeyValue(node)
case unstable.Table:
return s.checkTable(node)
case unstable.ArrayTable:
return s.checkArrayTable(node)
default:
panic(fmt.Errorf("this should not be a top level node type: %s", node.Kind))
}
}
func (s *SeenTracker) checkTable(node *unstable.Node) (bool, error) {
if s.currentIdx >= 0 {
s.setExplicitFlag(s.currentIdx)
}
it := node.Key()
parentIdx := 0
// This code is duplicated in checkArrayTable. This is because factoring
// it in a function requires to copy the iterator, or allocate it to the
// heap, which is not cheap.
for it.Next() {
if it.IsLast() {
break
}
k := it.Node().Data
idx := s.find(parentIdx, k)
if idx < 0 {
idx = s.create(parentIdx, k, tableKind, false, false)
} else {
entry := s.entries[idx]
if entry.kind == valueKind {
return false, fmt.Errorf("toml: expected %s to be a table, not a %s", string(k), entry.kind)
}
}
parentIdx = idx
}
k := it.Node().Data
idx := s.find(parentIdx, k)
first := false
if idx >= 0 {
kind := s.entries[idx].kind
if kind != tableKind {
return false, fmt.Errorf("toml: key %s should be a table, not a %s", string(k), kind)
}
if s.entries[idx].explicit {
return false, fmt.Errorf("toml: table %s already exists", string(k))
}
s.entries[idx].explicit = true
} else {
idx = s.create(parentIdx, k, tableKind, true, false)
first = true
}
s.currentIdx = idx
return first, nil
}
func (s *SeenTracker) checkArrayTable(node *unstable.Node) (bool, error) {
if s.currentIdx >= 0 {
s.setExplicitFlag(s.currentIdx)
}
it := node.Key()
parentIdx := 0
for it.Next() {
if it.IsLast() {
break
}
k := it.Node().Data
idx := s.find(parentIdx, k)
if idx < 0 {
idx = s.create(parentIdx, k, tableKind, false, false)
} else {
entry := s.entries[idx]
if entry.kind == valueKind {
return false, fmt.Errorf("toml: expected %s to be a table, not a %s", string(k), entry.kind)
}
}
parentIdx = idx
}
k := it.Node().Data
idx := s.find(parentIdx, k)
firstTime := idx < 0
if firstTime {
idx = s.create(parentIdx, k, arrayTableKind, true, false)
} else {
kind := s.entries[idx].kind
if kind != arrayTableKind {
return false, fmt.Errorf("toml: key %s already exists as a %s, but should be an array table", kind, string(k))
}
s.clear(idx)
}
s.currentIdx = idx
return firstTime, nil
}
func (s *SeenTracker) checkKeyValue(node *unstable.Node) (bool, error) {
parentIdx := s.currentIdx
it := node.Key()
for it.Next() {
k := it.Node().Data
idx := s.find(parentIdx, k)
if idx < 0 {
idx = s.create(parentIdx, k, tableKind, false, true)
} else {
entry := s.entries[idx]
switch {
case it.IsLast():
return false, fmt.Errorf("toml: key %s is already defined", string(k))
case entry.kind != tableKind:
return false, fmt.Errorf("toml: expected %s to be a table, not a %s", string(k), entry.kind)
case entry.explicit:
return false, fmt.Errorf("toml: cannot redefine table %s that has already been explicitly defined", string(k))
}
}
parentIdx = idx
}
s.entries[parentIdx].kind = valueKind
value := node.Value()
switch value.Kind {
case unstable.InlineTable:
return s.checkInlineTable(value)
case unstable.Array:
return s.checkArray(value)
default:
return false, nil
}
}
func (s *SeenTracker) checkArray(node *unstable.Node) (first bool, err error) {
it := node.Children()
for it.Next() {
n := it.Node()
switch n.Kind { //nolint:exhaustive
case unstable.InlineTable:
first, err = s.checkInlineTable(n)
if err != nil {
return false, err
}
case unstable.Array:
first, err = s.checkArray(n)
if err != nil {
return false, err
}
}
}
return first, nil
}
func (s *SeenTracker) checkInlineTable(node *unstable.Node) (first bool, err error) {
s = pool.Get().(*SeenTracker)
s.reset()
it := node.Children()
for it.Next() {
n := it.Node()
first, err = s.checkKeyValue(n)
if err != nil {
return false, err
}
}
// As inline tables are self-contained, the tracker does not
// need to retain the details of what they contain. The
// keyValue element that creates the inline table is kept to
// mark the presence of the inline table and prevent
// redefinition of its keys: check* functions cannot walk into
// a value.
pool.Put(s)
return first, nil
}
-21
View File
@@ -1,21 +0,0 @@
package tracker
import (
"reflect"
"testing"
"github.com/pelletier/go-toml/v2/internal/assert"
)
func TestEntrySize(t *testing.T) {
// Validate no regression on the size of entry{}. This is a critical bit for
// performance of unmarshaling documents. Should only be increased with care
// and a very good reason.
maxExpectedEntrySize := 48
entrySize := int(reflect.TypeOf(entry{}).Size())
assert.True(t,
entrySize <= maxExpectedEntrySize,
"Expected entry to be less than or equal to %d, got: %d",
maxExpectedEntrySize, entrySize,
)
}
-2
View File
@@ -1,2 +0,0 @@
// Package tracker provides functions for keeping track of AST nodes.
package tracker
+112
View File
@@ -0,0 +1,112 @@
// Parsing keys handling both bare and quoted keys.
package toml
import (
"errors"
"fmt"
)
// Convert the bare key group string to an array.
// The input supports double quotation and single quotation,
// but escape sequences are not supported. Lexers must unescape them beforehand.
func parseKey(key string) ([]string, error) {
runes := []rune(key)
var groups []string
if len(key) == 0 {
return nil, errors.New("empty key")
}
idx := 0
for idx < len(runes) {
for ; idx < len(runes) && isSpace(runes[idx]); idx++ {
// skip leading whitespace
}
if idx >= len(runes) {
break
}
r := runes[idx]
if isValidBareChar(r) {
// parse bare key
startIdx := idx
endIdx := -1
idx++
for idx < len(runes) {
r = runes[idx]
if isValidBareChar(r) {
idx++
} else if r == '.' {
endIdx = idx
break
} else if isSpace(r) {
endIdx = idx
for ; idx < len(runes) && isSpace(runes[idx]); idx++ {
// skip trailing whitespace
}
if idx < len(runes) && runes[idx] != '.' {
return nil, fmt.Errorf("invalid key character after whitespace: %c", runes[idx])
}
break
} else {
return nil, fmt.Errorf("invalid bare key character: %c", r)
}
}
if endIdx == -1 {
endIdx = idx
}
groups = append(groups, string(runes[startIdx:endIdx]))
} else if r == '\'' {
// parse single quoted key
idx++
startIdx := idx
for {
if idx >= len(runes) {
return nil, fmt.Errorf("unclosed single-quoted key")
}
r = runes[idx]
if r == '\'' {
groups = append(groups, string(runes[startIdx:idx]))
idx++
break
}
idx++
}
} else if r == '"' {
// parse double quoted key
idx++
startIdx := idx
for {
if idx >= len(runes) {
return nil, fmt.Errorf("unclosed double-quoted key")
}
r = runes[idx]
if r == '"' {
groups = append(groups, string(runes[startIdx:idx]))
idx++
break
}
idx++
}
} else if r == '.' {
idx++
if idx >= len(runes) {
return nil, fmt.Errorf("unexpected end of key")
}
r = runes[idx]
if !isValidBareChar(r) && r != '\'' && r != '"' && r != ' ' {
return nil, fmt.Errorf("expecting key part after dot")
}
} else {
return nil, fmt.Errorf("invalid key character: %c", r)
}
}
if len(groups) == 0 {
return nil, fmt.Errorf("empty key")
}
return groups, nil
}
func isValidBareChar(r rune) bool {
return isAlphanumeric(r) || r == '-' || isDigit(r)
}
+79
View File
@@ -0,0 +1,79 @@
package toml
import (
"fmt"
"testing"
)
func testResult(t *testing.T, key string, expected []string) {
parsed, err := parseKey(key)
t.Logf("key=%s expected=%s parsed=%s", key, expected, parsed)
if err != nil {
t.Fatal("Unexpected error:", err)
}
if len(expected) != len(parsed) {
t.Fatal("Expected length", len(expected), "but", len(parsed), "parsed")
}
for index, expectedKey := range expected {
if expectedKey != parsed[index] {
t.Fatal("Expected", expectedKey, "at index", index, "but found", parsed[index])
}
}
}
func testError(t *testing.T, key string, expectedError string) {
res, err := parseKey(key)
if err == nil {
t.Fatalf("Expected error, but successfully parsed key %s", res)
}
if fmt.Sprintf("%s", err) != expectedError {
t.Fatalf("Expected error \"%s\", but got \"%s\".", expectedError, err)
}
}
func TestBareKeyBasic(t *testing.T) {
testResult(t, "test", []string{"test"})
}
func TestBareKeyDotted(t *testing.T) {
testResult(t, "this.is.a.key", []string{"this", "is", "a", "key"})
}
func TestDottedKeyBasic(t *testing.T) {
testResult(t, "\"a.dotted.key\"", []string{"a.dotted.key"})
}
func TestBaseKeyPound(t *testing.T) {
testError(t, "hello#world", "invalid bare key character: #")
}
func TestUnclosedSingleQuotedKey(t *testing.T) {
testError(t, "'", "unclosed single-quoted key")
}
func TestUnclosedDoubleQuotedKey(t *testing.T) {
testError(t, "\"", "unclosed double-quoted key")
}
func TestInvalidStartKeyCharacter(t *testing.T) {
testError(t, "/", "invalid key character: /")
}
func TestInvalidSpaceInKey(t *testing.T) {
testError(t, "invalid key", "invalid key character after whitespace: k")
}
func TestQuotedKeys(t *testing.T) {
testResult(t, `hello."foo".bar`, []string{"hello", "foo", "bar"})
testResult(t, `"hello!"`, []string{"hello!"})
testResult(t, `foo."ba.r".baz`, []string{"foo", "ba.r", "baz"})
// escape sequences must not be converted
testResult(t, `"hello\tworld"`, []string{`hello\tworld`})
}
func TestEmptyKey(t *testing.T) {
testError(t, ``, "empty key")
testError(t, ` `, "empty key")
testResult(t, `""`, []string{""})
}
+1031
View File
File diff suppressed because it is too large Load Diff
+1247
View File
File diff suppressed because it is too large Load Diff
+248 -83
View File
@@ -1,122 +1,287 @@
// Implementation of TOML's local date/time.
//
// Copied over from Google's civil to avoid pulling all the Google dependencies.
// Originals:
// https://raw.githubusercontent.com/googleapis/google-cloud-go/ed46f5086358513cf8c25f8e3f022cb838a49d66/civil/civil.go
// Changes:
// * Renamed files from civil* to localtime*.
// * Package changed from civil to toml.
// * 'Local' prefix added to all structs.
//
// Copyright 2016 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package civil implements types for civil time, a time-zone-independent
// representation of time that follows the rules of the proleptic
// Gregorian calendar with exactly 24-hour days, 60-minute hours, and 60-second
// minutes.
//
// Because they lack location information, these types do not represent unique
// moments or intervals of time. Use time.Time for that purpose.
package toml
import (
"fmt"
"strings"
"time"
"github.com/pelletier/go-toml/v2/unstable"
)
// LocalDate represents a calendar day in no specific timezone.
// A LocalDate represents a date (year, month, day).
//
// This type does not include location information, and therefore does not
// describe a unique 24-hour timespan.
type LocalDate struct {
Year int
Month int
Day int
Year int // Year (e.g., 2014).
Month time.Month // Month of the year (January = 1, ...).
Day int // Day of the month, starting at 1.
}
// AsTime converts d into a specific time instance at midnight in zone.
func (d LocalDate) AsTime(zone *time.Location) time.Time {
return time.Date(d.Year, time.Month(d.Month), d.Day, 0, 0, 0, 0, zone)
// LocalDateOf returns the LocalDate in which a time occurs in that time's location.
func LocalDateOf(t time.Time) LocalDate {
var d LocalDate
d.Year, d.Month, d.Day = t.Date()
return d
}
// String returns RFC 3339 representation of d.
// ParseLocalDate parses a string in RFC3339 full-date format and returns the date value it represents.
func ParseLocalDate(s string) (LocalDate, error) {
t, err := time.Parse("2006-01-02", s)
if err != nil {
return LocalDate{}, err
}
return LocalDateOf(t), nil
}
// String returns the date in RFC3339 full-date format.
func (d LocalDate) String() string {
return fmt.Sprintf("%04d-%02d-%02d", d.Year, d.Month, d.Day)
}
// MarshalText returns RFC 3339 representation of d.
// IsValid reports whether the date is valid.
func (d LocalDate) IsValid() bool {
return LocalDateOf(d.In(time.UTC)) == d
}
// In returns the time corresponding to time 00:00:00 of the date in the location.
//
// In is always consistent with time.LocalDate, even when time.LocalDate returns a time
// on a different day. For example, if loc is America/Indiana/Vincennes, then both
// time.LocalDate(1955, time.May, 1, 0, 0, 0, 0, loc)
// and
// civil.LocalDate{Year: 1955, Month: time.May, Day: 1}.In(loc)
// return 23:00:00 on April 30, 1955.
//
// In panics if loc is nil.
func (d LocalDate) In(loc *time.Location) time.Time {
return time.Date(d.Year, d.Month, d.Day, 0, 0, 0, 0, loc)
}
// AddDays returns the date that is n days in the future.
// n can also be negative to go into the past.
func (d LocalDate) AddDays(n int) LocalDate {
return LocalDateOf(d.In(time.UTC).AddDate(0, 0, n))
}
// DaysSince returns the signed number of days between the date and s, not including the end day.
// This is the inverse operation to AddDays.
func (d LocalDate) DaysSince(s LocalDate) (days int) {
// We convert to Unix time so we do not have to worry about leap seconds:
// Unix time increases by exactly 86400 seconds per day.
deltaUnix := d.In(time.UTC).Unix() - s.In(time.UTC).Unix()
return int(deltaUnix / 86400)
}
// Before reports whether d1 occurs before d2.
func (d1 LocalDate) Before(d2 LocalDate) bool {
if d1.Year != d2.Year {
return d1.Year < d2.Year
}
if d1.Month != d2.Month {
return d1.Month < d2.Month
}
return d1.Day < d2.Day
}
// After reports whether d1 occurs after d2.
func (d1 LocalDate) After(d2 LocalDate) bool {
return d2.Before(d1)
}
// MarshalText implements the encoding.TextMarshaler interface.
// The output is the result of d.String().
func (d LocalDate) MarshalText() ([]byte, error) {
return []byte(d.String()), nil
}
// UnmarshalText parses b using RFC 3339 to fill d.
func (d *LocalDate) UnmarshalText(b []byte) error {
res, err := parseLocalDate(b)
if err != nil {
return err
}
*d = res
return nil
// UnmarshalText implements the encoding.TextUnmarshaler interface.
// The date is expected to be a string in a format accepted by ParseLocalDate.
func (d *LocalDate) UnmarshalText(data []byte) error {
var err error
*d, err = ParseLocalDate(string(data))
return err
}
// LocalTime represents a time of day of no specific day in no specific
// timezone.
// A LocalTime represents a time with nanosecond precision.
//
// This type does not include location information, and therefore does not
// describe a unique moment in time.
//
// This type exists to represent the TIME type in storage-based APIs like BigQuery.
// Most operations on Times are unlikely to be meaningful. Prefer the LocalDateTime type.
type LocalTime struct {
Hour int // Hour of the day: [0; 24[
Minute int // Minute of the hour: [0; 60[
Second int // Second of the minute: [0; 59]
Nanosecond int // Nanoseconds within the second: [0, 1000000000[
Precision int // Number of digits to display for Nanosecond.
Hour int // The hour of the day in 24-hour format; range [0-23]
Minute int // The minute of the hour; range [0-59]
Second int // The second of the minute; range [0-59]
Nanosecond int // The nanosecond of the second; range [0-999999999]
}
// String returns RFC 3339 representation of d.
// If d.Nanosecond and d.Precision are zero, the time won't have a nanosecond
// component. If d.Nanosecond > 0 but d.Precision = 0, then the minimum number
// of digits for nanoseconds is provided.
func (d LocalTime) String() string {
s := fmt.Sprintf("%02d:%02d:%02d", d.Hour, d.Minute, d.Second)
if d.Precision > 0 {
s += fmt.Sprintf(".%09d", d.Nanosecond)[:d.Precision+1]
} else if d.Nanosecond > 0 {
// Nanoseconds are specified, but precision is not provided. Use the
// minimum.
s += strings.Trim(fmt.Sprintf(".%09d", d.Nanosecond), "0")
}
return s
// LocalTimeOf returns the LocalTime representing the time of day in which a time occurs
// in that time's location. It ignores the date.
func LocalTimeOf(t time.Time) LocalTime {
var tm LocalTime
tm.Hour, tm.Minute, tm.Second = t.Clock()
tm.Nanosecond = t.Nanosecond()
return tm
}
// MarshalText returns RFC 3339 representation of d.
func (d LocalTime) MarshalText() ([]byte, error) {
return []byte(d.String()), nil
}
// UnmarshalText parses b using RFC 3339 to fill d.
func (d *LocalTime) UnmarshalText(b []byte) error {
res, left, err := parseLocalTime(b)
if err == nil && len(left) != 0 {
err = unstable.NewParserError(left, "extra characters")
}
// ParseLocalTime parses a string and returns the time value it represents.
// ParseLocalTime accepts an extended form of the RFC3339 partial-time format. After
// the HH:MM:SS part of the string, an optional fractional part may appear,
// consisting of a decimal point followed by one to nine decimal digits.
// (RFC3339 admits only one digit after the decimal point).
func ParseLocalTime(s string) (LocalTime, error) {
t, err := time.Parse("15:04:05.999999999", s)
if err != nil {
return err
return LocalTime{}, err
}
*d = res
return nil
return LocalTimeOf(t), nil
}
// LocalDateTime represents a time of a specific day in no specific timezone.
// String returns the date in the format described in ParseLocalTime. If Nanoseconds
// is zero, no fractional part will be generated. Otherwise, the result will
// end with a fractional part consisting of a decimal point and nine digits.
func (t LocalTime) String() string {
s := fmt.Sprintf("%02d:%02d:%02d", t.Hour, t.Minute, t.Second)
if t.Nanosecond == 0 {
return s
}
return s + fmt.Sprintf(".%09d", t.Nanosecond)
}
// IsValid reports whether the time is valid.
func (t LocalTime) IsValid() bool {
// Construct a non-zero time.
tm := time.Date(2, 2, 2, t.Hour, t.Minute, t.Second, t.Nanosecond, time.UTC)
return LocalTimeOf(tm) == t
}
// MarshalText implements the encoding.TextMarshaler interface.
// The output is the result of t.String().
func (t LocalTime) MarshalText() ([]byte, error) {
return []byte(t.String()), nil
}
// UnmarshalText implements the encoding.TextUnmarshaler interface.
// The time is expected to be a string in a format accepted by ParseLocalTime.
func (t *LocalTime) UnmarshalText(data []byte) error {
var err error
*t, err = ParseLocalTime(string(data))
return err
}
// A LocalDateTime represents a date and time.
//
// This type does not include location information, and therefore does not
// describe a unique moment in time.
type LocalDateTime struct {
LocalDate
LocalTime
Date LocalDate
Time LocalTime
}
// AsTime converts d into a specific time instance in zone.
func (d LocalDateTime) AsTime(zone *time.Location) time.Time {
return time.Date(d.Year, time.Month(d.Month), d.Day, d.Hour, d.Minute, d.Second, d.Nanosecond, zone)
}
// Note: We deliberately do not embed LocalDate into LocalDateTime, to avoid promoting AddDays and Sub.
// String returns RFC 3339 representation of d.
func (d LocalDateTime) String() string {
return d.LocalDate.String() + "T" + d.LocalTime.String()
}
// MarshalText returns RFC 3339 representation of d.
func (d LocalDateTime) MarshalText() ([]byte, error) {
return []byte(d.String()), nil
}
// UnmarshalText parses b using RFC 3339 to fill d.
func (d *LocalDateTime) UnmarshalText(data []byte) error {
res, left, err := parseLocalDateTime(data)
if err == nil && len(left) != 0 {
err = unstable.NewParserError(left, "extra characters")
// LocalDateTimeOf returns the LocalDateTime in which a time occurs in that time's location.
func LocalDateTimeOf(t time.Time) LocalDateTime {
return LocalDateTime{
Date: LocalDateOf(t),
Time: LocalTimeOf(t),
}
}
// ParseLocalDateTime parses a string and returns the LocalDateTime it represents.
// ParseLocalDateTime accepts a variant of the RFC3339 date-time format that omits
// the time offset but includes an optional fractional time, as described in
// ParseLocalTime. Informally, the accepted format is
// YYYY-MM-DDTHH:MM:SS[.FFFFFFFFF]
// where the 'T' may be a lower-case 't'.
func ParseLocalDateTime(s string) (LocalDateTime, error) {
t, err := time.Parse("2006-01-02T15:04:05.999999999", s)
if err != nil {
return err
t, err = time.Parse("2006-01-02t15:04:05.999999999", s)
if err != nil {
return LocalDateTime{}, err
}
}
*d = res
return nil
return LocalDateTimeOf(t), nil
}
// String returns the date in the format described in ParseLocalDate.
func (dt LocalDateTime) String() string {
return dt.Date.String() + "T" + dt.Time.String()
}
// IsValid reports whether the datetime is valid.
func (dt LocalDateTime) IsValid() bool {
return dt.Date.IsValid() && dt.Time.IsValid()
}
// In returns the time corresponding to the LocalDateTime in the given location.
//
// If the time is missing or ambigous at the location, In returns the same
// result as time.LocalDate. For example, if loc is America/Indiana/Vincennes, then
// both
// time.LocalDate(1955, time.May, 1, 0, 30, 0, 0, loc)
// and
// civil.LocalDateTime{
// civil.LocalDate{Year: 1955, Month: time.May, Day: 1}},
// civil.LocalTime{Minute: 30}}.In(loc)
// return 23:30:00 on April 30, 1955.
//
// In panics if loc is nil.
func (dt LocalDateTime) In(loc *time.Location) time.Time {
return time.Date(dt.Date.Year, dt.Date.Month, dt.Date.Day, dt.Time.Hour, dt.Time.Minute, dt.Time.Second, dt.Time.Nanosecond, loc)
}
// Before reports whether dt1 occurs before dt2.
func (dt1 LocalDateTime) Before(dt2 LocalDateTime) bool {
return dt1.In(time.UTC).Before(dt2.In(time.UTC))
}
// After reports whether dt1 occurs after dt2.
func (dt1 LocalDateTime) After(dt2 LocalDateTime) bool {
return dt2.Before(dt1)
}
// MarshalText implements the encoding.TextMarshaler interface.
// The output is the result of dt.String().
func (dt LocalDateTime) MarshalText() ([]byte, error) {
return []byte(dt.String()), nil
}
// UnmarshalText implements the encoding.TextUnmarshaler interface.
// The datetime is expected to be a string in a format accepted by ParseLocalDateTime
func (dt *LocalDateTime) UnmarshalText(data []byte) error {
var err error
*dt, err = ParseLocalDateTime(string(data))
return err
}
+438 -100
View File
@@ -1,118 +1,456 @@
package toml_test
// Implementation of TOML's local date/time.
//
// Copied over from Google's civil to avoid pulling all the Google dependencies.
// Originals:
// https://raw.githubusercontent.com/googleapis/google-cloud-go/ed46f5086358513cf8c25f8e3f022cb838a49d66/civil/civil_test.go
// Changes:
// * Renamed files from civil* to localtime*.
// * Package changed from civil to toml.
// * 'Local' prefix added to all structs.
//
// Copyright 2016 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package toml
import (
"encoding/json"
"reflect"
"testing"
"time"
"github.com/pelletier/go-toml/v2"
"github.com/pelletier/go-toml/v2/internal/assert"
)
func TestLocalDate_AsTime(t *testing.T) {
d := toml.LocalDate{2021, 6, 8}
cast := d.AsTime(time.UTC)
assert.Equal(t, time.Date(2021, time.June, 8, 0, 0, 0, 0, time.UTC), cast)
func cmpEqual(x, y interface{}) bool {
return reflect.DeepEqual(x, y)
}
func TestLocalDate_String(t *testing.T) {
d := toml.LocalDate{2021, 6, 8}
assert.Equal(t, "2021-06-08", d.String())
}
func TestLocalDate_MarshalText(t *testing.T) {
d := toml.LocalDate{2021, 6, 8}
b, err := d.MarshalText()
assert.NoError(t, err)
assert.Equal(t, []byte("2021-06-08"), b)
}
func TestLocalDate_UnmarshalMarshalText(t *testing.T) {
d := toml.LocalDate{}
err := d.UnmarshalText([]byte("2021-06-08"))
assert.NoError(t, err)
assert.Equal(t, toml.LocalDate{2021, 6, 8}, d)
err = d.UnmarshalText([]byte("what"))
assert.Error(t, err)
}
func TestLocalTime_String(t *testing.T) {
d := toml.LocalTime{20, 12, 1, 2, 9}
assert.Equal(t, "20:12:01.000000002", d.String())
d = toml.LocalTime{20, 12, 1, 0, 0}
assert.Equal(t, "20:12:01", d.String())
d = toml.LocalTime{20, 12, 1, 0, 9}
assert.Equal(t, "20:12:01.000000000", d.String())
d = toml.LocalTime{20, 12, 1, 100, 0}
assert.Equal(t, "20:12:01.0000001", d.String())
}
func TestLocalTime_MarshalText(t *testing.T) {
d := toml.LocalTime{20, 12, 1, 2, 9}
b, err := d.MarshalText()
assert.NoError(t, err)
assert.Equal(t, []byte("20:12:01.000000002"), b)
}
func TestLocalTime_UnmarshalMarshalText(t *testing.T) {
d := toml.LocalTime{}
err := d.UnmarshalText([]byte("20:12:01.000000002"))
assert.NoError(t, err)
assert.Equal(t, toml.LocalTime{20, 12, 1, 2, 9}, d)
err = d.UnmarshalText([]byte("what"))
assert.Error(t, err)
err = d.UnmarshalText([]byte("20:12:01.000000002 bad"))
assert.Error(t, err)
}
func TestLocalTime_RoundTrip(t *testing.T) {
var d struct{ A toml.LocalTime }
err := toml.Unmarshal([]byte("a=20:12:01.500"), &d)
assert.NoError(t, err)
assert.Equal(t, "20:12:01.500", d.A.String())
}
func TestLocalDateTime_AsTime(t *testing.T) {
d := toml.LocalDateTime{
toml.LocalDate{2021, 6, 8},
toml.LocalTime{20, 12, 1, 2, 9},
func TestDates(t *testing.T) {
for _, test := range []struct {
date LocalDate
loc *time.Location
wantStr string
wantTime time.Time
}{
{
date: LocalDate{2014, 7, 29},
loc: time.Local,
wantStr: "2014-07-29",
wantTime: time.Date(2014, time.July, 29, 0, 0, 0, 0, time.Local),
},
{
date: LocalDateOf(time.Date(2014, 8, 20, 15, 8, 43, 1, time.Local)),
loc: time.UTC,
wantStr: "2014-08-20",
wantTime: time.Date(2014, 8, 20, 0, 0, 0, 0, time.UTC),
},
{
date: LocalDateOf(time.Date(999, time.January, 26, 0, 0, 0, 0, time.Local)),
loc: time.UTC,
wantStr: "0999-01-26",
wantTime: time.Date(999, 1, 26, 0, 0, 0, 0, time.UTC),
},
} {
if got := test.date.String(); got != test.wantStr {
t.Errorf("%#v.String() = %q, want %q", test.date, got, test.wantStr)
}
if got := test.date.In(test.loc); !got.Equal(test.wantTime) {
t.Errorf("%#v.In(%v) = %v, want %v", test.date, test.loc, got, test.wantTime)
}
}
cast := d.AsTime(time.UTC)
assert.Equal(t, time.Date(2021, time.June, 8, 20, 12, 1, 2, time.UTC), cast)
}
func TestLocalDateTime_String(t *testing.T) {
d := toml.LocalDateTime{
toml.LocalDate{2021, 6, 8},
toml.LocalTime{20, 12, 1, 2, 9},
func TestDateIsValid(t *testing.T) {
for _, test := range []struct {
date LocalDate
want bool
}{
{LocalDate{2014, 7, 29}, true},
{LocalDate{2000, 2, 29}, true},
{LocalDate{10000, 12, 31}, true},
{LocalDate{1, 1, 1}, true},
{LocalDate{0, 1, 1}, true}, // year zero is OK
{LocalDate{-1, 1, 1}, true}, // negative year is OK
{LocalDate{1, 0, 1}, false},
{LocalDate{1, 1, 0}, false},
{LocalDate{2016, 1, 32}, false},
{LocalDate{2016, 13, 1}, false},
{LocalDate{1, -1, 1}, false},
{LocalDate{1, 1, -1}, false},
} {
got := test.date.IsValid()
if got != test.want {
t.Errorf("%#v: got %t, want %t", test.date, got, test.want)
}
}
assert.Equal(t, "2021-06-08T20:12:01.000000002", d.String())
}
func TestLocalDateTime_MarshalText(t *testing.T) {
d := toml.LocalDateTime{
toml.LocalDate{2021, 6, 8},
toml.LocalTime{20, 12, 1, 2, 9},
func TestParseDate(t *testing.T) {
for _, test := range []struct {
str string
want LocalDate // if empty, expect an error
}{
{"2016-01-02", LocalDate{2016, 1, 2}},
{"2016-12-31", LocalDate{2016, 12, 31}},
{"0003-02-04", LocalDate{3, 2, 4}},
{"999-01-26", LocalDate{}},
{"", LocalDate{}},
{"2016-01-02x", LocalDate{}},
} {
got, err := ParseLocalDate(test.str)
if got != test.want {
t.Errorf("ParseLocalDate(%q) = %+v, want %+v", test.str, got, test.want)
}
if err != nil && test.want != (LocalDate{}) {
t.Errorf("Unexpected error %v from ParseLocalDate(%q)", err, test.str)
}
}
b, err := d.MarshalText()
assert.NoError(t, err)
assert.Equal(t, []byte("2021-06-08T20:12:01.000000002"), b)
}
func TestLocalDateTime_UnmarshalMarshalText(t *testing.T) {
d := toml.LocalDateTime{}
err := d.UnmarshalText([]byte("2021-06-08 20:12:01.000000002"))
assert.NoError(t, err)
assert.Equal(t, toml.LocalDateTime{
toml.LocalDate{2021, 6, 8},
toml.LocalTime{20, 12, 1, 2, 9},
}, d)
err = d.UnmarshalText([]byte("what"))
assert.Error(t, err)
err = d.UnmarshalText([]byte("2021-06-08 20:12:01.000000002 bad"))
assert.Error(t, err)
func TestDateArithmetic(t *testing.T) {
for _, test := range []struct {
desc string
start LocalDate
end LocalDate
days int
}{
{
desc: "zero days noop",
start: LocalDate{2014, 5, 9},
end: LocalDate{2014, 5, 9},
days: 0,
},
{
desc: "crossing a year boundary",
start: LocalDate{2014, 12, 31},
end: LocalDate{2015, 1, 1},
days: 1,
},
{
desc: "negative number of days",
start: LocalDate{2015, 1, 1},
end: LocalDate{2014, 12, 31},
days: -1,
},
{
desc: "full leap year",
start: LocalDate{2004, 1, 1},
end: LocalDate{2005, 1, 1},
days: 366,
},
{
desc: "full non-leap year",
start: LocalDate{2001, 1, 1},
end: LocalDate{2002, 1, 1},
days: 365,
},
{
desc: "crossing a leap second",
start: LocalDate{1972, 6, 30},
end: LocalDate{1972, 7, 1},
days: 1,
},
{
desc: "dates before the unix epoch",
start: LocalDate{101, 1, 1},
end: LocalDate{102, 1, 1},
days: 365,
},
} {
if got := test.start.AddDays(test.days); got != test.end {
t.Errorf("[%s] %#v.AddDays(%v) = %#v, want %#v", test.desc, test.start, test.days, got, test.end)
}
if got := test.end.DaysSince(test.start); got != test.days {
t.Errorf("[%s] %#v.Sub(%#v) = %v, want %v", test.desc, test.end, test.start, got, test.days)
}
}
}
func TestDateBefore(t *testing.T) {
for _, test := range []struct {
d1, d2 LocalDate
want bool
}{
{LocalDate{2016, 12, 31}, LocalDate{2017, 1, 1}, true},
{LocalDate{2016, 1, 1}, LocalDate{2016, 1, 1}, false},
{LocalDate{2016, 12, 30}, LocalDate{2016, 12, 31}, true},
{LocalDate{2016, 1, 30}, LocalDate{2016, 12, 31}, true},
} {
if got := test.d1.Before(test.d2); got != test.want {
t.Errorf("%v.Before(%v): got %t, want %t", test.d1, test.d2, got, test.want)
}
}
}
func TestDateAfter(t *testing.T) {
for _, test := range []struct {
d1, d2 LocalDate
want bool
}{
{LocalDate{2016, 12, 31}, LocalDate{2017, 1, 1}, false},
{LocalDate{2016, 1, 1}, LocalDate{2016, 1, 1}, false},
{LocalDate{2016, 12, 30}, LocalDate{2016, 12, 31}, false},
} {
if got := test.d1.After(test.d2); got != test.want {
t.Errorf("%v.After(%v): got %t, want %t", test.d1, test.d2, got, test.want)
}
}
}
func TestTimeToString(t *testing.T) {
for _, test := range []struct {
str string
time LocalTime
roundTrip bool // ParseLocalTime(str).String() == str?
}{
{"13:26:33", LocalTime{13, 26, 33, 0}, true},
{"01:02:03.000023456", LocalTime{1, 2, 3, 23456}, true},
{"00:00:00.000000001", LocalTime{0, 0, 0, 1}, true},
{"13:26:03.1", LocalTime{13, 26, 3, 100000000}, false},
{"13:26:33.0000003", LocalTime{13, 26, 33, 300}, false},
} {
gotTime, err := ParseLocalTime(test.str)
if err != nil {
t.Errorf("ParseLocalTime(%q): got error: %v", test.str, err)
continue
}
if gotTime != test.time {
t.Errorf("ParseLocalTime(%q) = %+v, want %+v", test.str, gotTime, test.time)
}
if test.roundTrip {
gotStr := test.time.String()
if gotStr != test.str {
t.Errorf("%#v.String() = %q, want %q", test.time, gotStr, test.str)
}
}
}
}
func TestTimeOf(t *testing.T) {
for _, test := range []struct {
time time.Time
want LocalTime
}{
{time.Date(2014, 8, 20, 15, 8, 43, 1, time.Local), LocalTime{15, 8, 43, 1}},
{time.Date(1, 1, 1, 0, 0, 0, 0, time.UTC), LocalTime{0, 0, 0, 0}},
} {
if got := LocalTimeOf(test.time); got != test.want {
t.Errorf("LocalTimeOf(%v) = %+v, want %+v", test.time, got, test.want)
}
}
}
func TestTimeIsValid(t *testing.T) {
for _, test := range []struct {
time LocalTime
want bool
}{
{LocalTime{0, 0, 0, 0}, true},
{LocalTime{23, 0, 0, 0}, true},
{LocalTime{23, 59, 59, 999999999}, true},
{LocalTime{24, 59, 59, 999999999}, false},
{LocalTime{23, 60, 59, 999999999}, false},
{LocalTime{23, 59, 60, 999999999}, false},
{LocalTime{23, 59, 59, 1000000000}, false},
{LocalTime{-1, 0, 0, 0}, false},
{LocalTime{0, -1, 0, 0}, false},
{LocalTime{0, 0, -1, 0}, false},
{LocalTime{0, 0, 0, -1}, false},
} {
got := test.time.IsValid()
if got != test.want {
t.Errorf("%#v: got %t, want %t", test.time, got, test.want)
}
}
}
func TestDateTimeToString(t *testing.T) {
for _, test := range []struct {
str string
dateTime LocalDateTime
roundTrip bool // ParseLocalDateTime(str).String() == str?
}{
{"2016-03-22T13:26:33", LocalDateTime{LocalDate{2016, 03, 22}, LocalTime{13, 26, 33, 0}}, true},
{"2016-03-22T13:26:33.000000600", LocalDateTime{LocalDate{2016, 03, 22}, LocalTime{13, 26, 33, 600}}, true},
{"2016-03-22t13:26:33", LocalDateTime{LocalDate{2016, 03, 22}, LocalTime{13, 26, 33, 0}}, false},
} {
gotDateTime, err := ParseLocalDateTime(test.str)
if err != nil {
t.Errorf("ParseLocalDateTime(%q): got error: %v", test.str, err)
continue
}
if gotDateTime != test.dateTime {
t.Errorf("ParseLocalDateTime(%q) = %+v, want %+v", test.str, gotDateTime, test.dateTime)
}
if test.roundTrip {
gotStr := test.dateTime.String()
if gotStr != test.str {
t.Errorf("%#v.String() = %q, want %q", test.dateTime, gotStr, test.str)
}
}
}
}
func TestParseDateTimeErrors(t *testing.T) {
for _, str := range []string{
"",
"2016-03-22", // just a date
"13:26:33", // just a time
"2016-03-22 13:26:33", // wrong separating character
"2016-03-22T13:26:33x", // extra at end
} {
if _, err := ParseLocalDateTime(str); err == nil {
t.Errorf("ParseLocalDateTime(%q) succeeded, want error", str)
}
}
}
func TestDateTimeOf(t *testing.T) {
for _, test := range []struct {
time time.Time
want LocalDateTime
}{
{time.Date(2014, 8, 20, 15, 8, 43, 1, time.Local),
LocalDateTime{LocalDate{2014, 8, 20}, LocalTime{15, 8, 43, 1}}},
{time.Date(1, 1, 1, 0, 0, 0, 0, time.UTC),
LocalDateTime{LocalDate{1, 1, 1}, LocalTime{0, 0, 0, 0}}},
} {
if got := LocalDateTimeOf(test.time); got != test.want {
t.Errorf("LocalDateTimeOf(%v) = %+v, want %+v", test.time, got, test.want)
}
}
}
func TestDateTimeIsValid(t *testing.T) {
// No need to be exhaustive here; it's just LocalDate.IsValid && LocalTime.IsValid.
for _, test := range []struct {
dt LocalDateTime
want bool
}{
{LocalDateTime{LocalDate{2016, 3, 20}, LocalTime{0, 0, 0, 0}}, true},
{LocalDateTime{LocalDate{2016, -3, 20}, LocalTime{0, 0, 0, 0}}, false},
{LocalDateTime{LocalDate{2016, 3, 20}, LocalTime{24, 0, 0, 0}}, false},
} {
got := test.dt.IsValid()
if got != test.want {
t.Errorf("%#v: got %t, want %t", test.dt, got, test.want)
}
}
}
func TestDateTimeIn(t *testing.T) {
dt := LocalDateTime{LocalDate{2016, 1, 2}, LocalTime{3, 4, 5, 6}}
got := dt.In(time.UTC)
want := time.Date(2016, 1, 2, 3, 4, 5, 6, time.UTC)
if !got.Equal(want) {
t.Errorf("got %v, want %v", got, want)
}
}
func TestDateTimeBefore(t *testing.T) {
d1 := LocalDate{2016, 12, 31}
d2 := LocalDate{2017, 1, 1}
t1 := LocalTime{5, 6, 7, 8}
t2 := LocalTime{5, 6, 7, 9}
for _, test := range []struct {
dt1, dt2 LocalDateTime
want bool
}{
{LocalDateTime{d1, t1}, LocalDateTime{d2, t1}, true},
{LocalDateTime{d1, t1}, LocalDateTime{d1, t2}, true},
{LocalDateTime{d2, t1}, LocalDateTime{d1, t1}, false},
{LocalDateTime{d2, t1}, LocalDateTime{d2, t1}, false},
} {
if got := test.dt1.Before(test.dt2); got != test.want {
t.Errorf("%v.Before(%v): got %t, want %t", test.dt1, test.dt2, got, test.want)
}
}
}
func TestDateTimeAfter(t *testing.T) {
d1 := LocalDate{2016, 12, 31}
d2 := LocalDate{2017, 1, 1}
t1 := LocalTime{5, 6, 7, 8}
t2 := LocalTime{5, 6, 7, 9}
for _, test := range []struct {
dt1, dt2 LocalDateTime
want bool
}{
{LocalDateTime{d1, t1}, LocalDateTime{d2, t1}, false},
{LocalDateTime{d1, t1}, LocalDateTime{d1, t2}, false},
{LocalDateTime{d2, t1}, LocalDateTime{d1, t1}, true},
{LocalDateTime{d2, t1}, LocalDateTime{d2, t1}, false},
} {
if got := test.dt1.After(test.dt2); got != test.want {
t.Errorf("%v.After(%v): got %t, want %t", test.dt1, test.dt2, got, test.want)
}
}
}
func TestMarshalJSON(t *testing.T) {
for _, test := range []struct {
value interface{}
want string
}{
{LocalDate{1987, 4, 15}, `"1987-04-15"`},
{LocalTime{18, 54, 2, 0}, `"18:54:02"`},
{LocalDateTime{LocalDate{1987, 4, 15}, LocalTime{18, 54, 2, 0}}, `"1987-04-15T18:54:02"`},
} {
bgot, err := json.Marshal(test.value)
if err != nil {
t.Fatal(err)
}
if got := string(bgot); got != test.want {
t.Errorf("%#v: got %s, want %s", test.value, got, test.want)
}
}
}
func TestUnmarshalJSON(t *testing.T) {
var d LocalDate
var tm LocalTime
var dt LocalDateTime
for _, test := range []struct {
data string
ptr interface{}
want interface{}
}{
{`"1987-04-15"`, &d, &LocalDate{1987, 4, 15}},
{`"1987-04-\u0031\u0035"`, &d, &LocalDate{1987, 4, 15}},
{`"18:54:02"`, &tm, &LocalTime{18, 54, 2, 0}},
{`"1987-04-15T18:54:02"`, &dt, &LocalDateTime{LocalDate{1987, 4, 15}, LocalTime{18, 54, 2, 0}}},
} {
if err := json.Unmarshal([]byte(test.data), test.ptr); err != nil {
t.Fatalf("%s: %v", test.data, err)
}
if !cmpEqual(test.ptr, test.want) {
t.Errorf("%s: got %#v, want %#v", test.data, test.ptr, test.want)
}
}
for _, bad := range []string{"", `""`, `"bad"`, `"1987-04-15x"`,
`19870415`, // a JSON number
`11987-04-15x`, // not a JSON string
} {
if json.Unmarshal([]byte(bad), &d) == nil {
t.Errorf("%q, LocalDate: got nil, want error", bad)
}
if json.Unmarshal([]byte(bad), &tm) == nil {
t.Errorf("%q, LocalTime: got nil, want error", bad)
}
if json.Unmarshal([]byte(bad), &dt) == nil {
t.Errorf("%q, LocalDateTime: got nil, want error", bad)
}
}
}
+1308
View File
File diff suppressed because it is too large Load Diff
+39
View File
@@ -0,0 +1,39 @@
title = "TOML Marshal Testing"
[basic_lists]
floats = [12.3,45.6,78.9]
bools = [true,false,true]
dates = [1979-05-27T07:32:00Z,1980-05-27T07:32:00Z]
ints = [8001,8001,8002]
uints = [5002,5003]
strings = ["One","Two","Three"]
[[subdocptrs]]
name = "Second"
[basic_map]
one = "one"
two = "two"
[subdoc]
[subdoc.second]
name = "Second"
[subdoc.first]
name = "First"
[basic]
uint = 5001
bool = true
float = 123.4
float64 = 123.456782132399
int = 5000
string = "Bite me"
date = 1979-05-27T07:32:00Z
[[subdoclist]]
name = "List.First"
[[subdoclist]]
name = "List.Second"
+4131
View File
File diff suppressed because it is too large Load Diff
+39
View File
@@ -0,0 +1,39 @@
title = "TOML Marshal Testing"
[basic]
bool = true
date = 1979-05-27T07:32:00Z
float = 123.4
float64 = 123.456782132399
int = 5000
string = "Bite me"
uint = 5001
[basic_lists]
bools = [true,false,true]
dates = [1979-05-27T07:32:00Z,1980-05-27T07:32:00Z]
floats = [12.3,45.6,78.9]
ints = [8001,8001,8002]
strings = ["One","Two","Three"]
uints = [5002,5003]
[basic_map]
one = "one"
two = "two"
[subdoc]
[subdoc.first]
name = "First"
[subdoc.second]
name = "Second"
[[subdoclist]]
name = "List.First"
[[subdoclist]]
name = "List.Second"
[[subdocptrs]]
name = "Second"
-1202
View File
File diff suppressed because it is too large Load Diff
-2241
View File
File diff suppressed because it is too large Load Diff
-44
View File
@@ -1,44 +0,0 @@
// Package ossfuzz provides a fuzzing target for OSS-Fuzz.
package ossfuzz
import (
"fmt"
"reflect"
"strings"
"github.com/pelletier/go-toml/v2"
)
// FuzzToml is the fuzzing target.
func FuzzToml(data []byte) int {
if len(data) >= 2048 {
return 0
}
if strings.Contains(string(data), "nan") {
return 0
}
var v interface{}
err := toml.Unmarshal(data, &v)
if err != nil {
return 0
}
encoded, err := toml.Marshal(v)
if err != nil {
panic(fmt.Sprintf("failed to marshal unmarshaled document: %s", err))
}
var v2 interface{}
err = toml.Unmarshal(encoded, &v2)
if err != nil {
panic(fmt.Sprintf("failed round trip: %s", err))
}
if !reflect.DeepEqual(v, v2) {
panic(fmt.Sprintf("not equal: %#+v %#+v", v, v2))
}
return 1
}
+507
View File
@@ -0,0 +1,507 @@
// TOML Parser.
package toml
import (
"errors"
"fmt"
"math"
"reflect"
"strconv"
"strings"
"time"
)
type tomlParser struct {
flowIdx int
flow []token
tree *Tree
currentTable []string
seenTableKeys []string
}
type tomlParserStateFn func() tomlParserStateFn
// Formats and panics an error message based on a token
func (p *tomlParser) raiseError(tok *token, msg string, args ...interface{}) {
panic(tok.Position.String() + ": " + fmt.Sprintf(msg, args...))
}
func (p *tomlParser) run() {
for state := p.parseStart; state != nil; {
state = state()
}
}
func (p *tomlParser) peek() *token {
if p.flowIdx >= len(p.flow) {
return nil
}
return &p.flow[p.flowIdx]
}
func (p *tomlParser) assume(typ tokenType) {
tok := p.getToken()
if tok == nil {
p.raiseError(tok, "was expecting token %s, but token stream is empty", tok)
}
if tok.typ != typ {
p.raiseError(tok, "was expecting token %s, but got %s instead", typ, tok)
}
}
func (p *tomlParser) getToken() *token {
tok := p.peek()
if tok == nil {
return nil
}
p.flowIdx++
return tok
}
func (p *tomlParser) parseStart() tomlParserStateFn {
tok := p.peek()
// end of stream, parsing is finished
if tok == nil {
return nil
}
switch tok.typ {
case tokenDoubleLeftBracket:
return p.parseGroupArray
case tokenLeftBracket:
return p.parseGroup
case tokenKey:
return p.parseAssign
case tokenEOF:
return nil
case tokenError:
p.raiseError(tok, "parsing error: %s", tok.String())
default:
p.raiseError(tok, "unexpected token %s", tok.typ)
}
return nil
}
func (p *tomlParser) parseGroupArray() tomlParserStateFn {
startToken := p.getToken() // discard the [[
key := p.getToken()
if key.typ != tokenKeyGroupArray {
p.raiseError(key, "unexpected token %s, was expecting a table array key", key)
}
// get or create table array element at the indicated part in the path
keys, err := parseKey(key.val)
if err != nil {
p.raiseError(key, "invalid table array key: %s", err)
}
p.tree.createSubTree(keys[:len(keys)-1], startToken.Position) // create parent entries
destTree := p.tree.GetPath(keys)
var array []*Tree
if destTree == nil {
array = make([]*Tree, 0)
} else if target, ok := destTree.([]*Tree); ok && target != nil {
array = destTree.([]*Tree)
} else {
p.raiseError(key, "key %s is already assigned and not of type table array", key)
}
p.currentTable = keys
// add a new tree to the end of the table array
newTree := newTree()
newTree.position = startToken.Position
array = append(array, newTree)
p.tree.SetPath(p.currentTable, array)
// remove all keys that were children of this table array
prefix := key.val + "."
found := false
for ii := 0; ii < len(p.seenTableKeys); {
tableKey := p.seenTableKeys[ii]
if strings.HasPrefix(tableKey, prefix) {
p.seenTableKeys = append(p.seenTableKeys[:ii], p.seenTableKeys[ii+1:]...)
} else {
found = (tableKey == key.val)
ii++
}
}
// keep this key name from use by other kinds of assignments
if !found {
p.seenTableKeys = append(p.seenTableKeys, key.val)
}
// move to next parser state
p.assume(tokenDoubleRightBracket)
return p.parseStart
}
func (p *tomlParser) parseGroup() tomlParserStateFn {
startToken := p.getToken() // discard the [
key := p.getToken()
if key.typ != tokenKeyGroup {
p.raiseError(key, "unexpected token %s, was expecting a table key", key)
}
for _, item := range p.seenTableKeys {
if item == key.val {
p.raiseError(key, "duplicated tables")
}
}
p.seenTableKeys = append(p.seenTableKeys, key.val)
keys, err := parseKey(key.val)
if err != nil {
p.raiseError(key, "invalid table array key: %s", err)
}
if err := p.tree.createSubTree(keys, startToken.Position); err != nil {
p.raiseError(key, "%s", err)
}
destTree := p.tree.GetPath(keys)
if target, ok := destTree.(*Tree); ok && target != nil && target.inline {
p.raiseError(key, "could not re-define exist inline table or its sub-table : %s",
strings.Join(keys, "."))
}
p.assume(tokenRightBracket)
p.currentTable = keys
return p.parseStart
}
func (p *tomlParser) parseAssign() tomlParserStateFn {
key := p.getToken()
p.assume(tokenEqual)
parsedKey, err := parseKey(key.val)
if err != nil {
p.raiseError(key, "invalid key: %s", err.Error())
}
value := p.parseRvalue()
var tableKey []string
if len(p.currentTable) > 0 {
tableKey = p.currentTable
} else {
tableKey = []string{}
}
prefixKey := parsedKey[0 : len(parsedKey)-1]
tableKey = append(tableKey, prefixKey...)
// find the table to assign, looking out for arrays of tables
var targetNode *Tree
switch node := p.tree.GetPath(tableKey).(type) {
case []*Tree:
targetNode = node[len(node)-1]
case *Tree:
targetNode = node
case nil:
// create intermediate
if err := p.tree.createSubTree(tableKey, key.Position); err != nil {
p.raiseError(key, "could not create intermediate group: %s", err)
}
targetNode = p.tree.GetPath(tableKey).(*Tree)
default:
p.raiseError(key, "Unknown table type for path: %s",
strings.Join(tableKey, "."))
}
if targetNode.inline {
p.raiseError(key, "could not add key or sub-table to exist inline table or its sub-table : %s",
strings.Join(tableKey, "."))
}
// assign value to the found table
keyVal := parsedKey[len(parsedKey)-1]
localKey := []string{keyVal}
finalKey := append(tableKey, keyVal)
if targetNode.GetPath(localKey) != nil {
p.raiseError(key, "The following key was defined twice: %s",
strings.Join(finalKey, "."))
}
var toInsert interface{}
switch value.(type) {
case *Tree, []*Tree:
toInsert = value
default:
toInsert = &tomlValue{value: value, position: key.Position}
}
targetNode.values[keyVal] = toInsert
return p.parseStart
}
var errInvalidUnderscore = errors.New("invalid use of _ in number")
func numberContainsInvalidUnderscore(value string) error {
// For large numbers, you may use underscores between digits to enhance
// readability. Each underscore must be surrounded by at least one digit on
// each side.
hasBefore := false
for idx, r := range value {
if r == '_' {
if !hasBefore || idx+1 >= len(value) {
// can't end with an underscore
return errInvalidUnderscore
}
}
hasBefore = isDigit(r)
}
return nil
}
var errInvalidUnderscoreHex = errors.New("invalid use of _ in hex number")
func hexNumberContainsInvalidUnderscore(value string) error {
hasBefore := false
for idx, r := range value {
if r == '_' {
if !hasBefore || idx+1 >= len(value) {
// can't end with an underscore
return errInvalidUnderscoreHex
}
}
hasBefore = isHexDigit(r)
}
return nil
}
func cleanupNumberToken(value string) string {
cleanedVal := strings.Replace(value, "_", "", -1)
return cleanedVal
}
func (p *tomlParser) parseRvalue() interface{} {
tok := p.getToken()
if tok == nil || tok.typ == tokenEOF {
p.raiseError(tok, "expecting a value")
}
switch tok.typ {
case tokenString:
return tok.val
case tokenTrue:
return true
case tokenFalse:
return false
case tokenInf:
if tok.val[0] == '-' {
return math.Inf(-1)
}
return math.Inf(1)
case tokenNan:
return math.NaN()
case tokenInteger:
cleanedVal := cleanupNumberToken(tok.val)
base := 10
s := cleanedVal
checkInvalidUnderscore := numberContainsInvalidUnderscore
if len(cleanedVal) >= 3 && cleanedVal[0] == '0' {
switch cleanedVal[1] {
case 'x':
checkInvalidUnderscore = hexNumberContainsInvalidUnderscore
base = 16
case 'o':
base = 8
case 'b':
base = 2
default:
panic("invalid base") // the lexer should catch this first
}
s = cleanedVal[2:]
}
err := checkInvalidUnderscore(tok.val)
if err != nil {
p.raiseError(tok, "%s", err)
}
var val interface{}
val, err = strconv.ParseInt(s, base, 64)
if err == nil {
return val
}
if s[0] != '-' {
if val, err = strconv.ParseUint(s, base, 64); err == nil {
return val
}
}
p.raiseError(tok, "%s", err)
case tokenFloat:
err := numberContainsInvalidUnderscore(tok.val)
if err != nil {
p.raiseError(tok, "%s", err)
}
cleanedVal := cleanupNumberToken(tok.val)
val, err := strconv.ParseFloat(cleanedVal, 64)
if err != nil {
p.raiseError(tok, "%s", err)
}
return val
case tokenLocalTime:
val, err := ParseLocalTime(tok.val)
if err != nil {
p.raiseError(tok, "%s", err)
}
return val
case tokenLocalDate:
// a local date may be followed by:
// * nothing: this is a local date
// * a local time: this is a local date-time
next := p.peek()
if next == nil || next.typ != tokenLocalTime {
val, err := ParseLocalDate(tok.val)
if err != nil {
p.raiseError(tok, "%s", err)
}
return val
}
localDate := tok
localTime := p.getToken()
next = p.peek()
if next == nil || next.typ != tokenTimeOffset {
v := localDate.val + "T" + localTime.val
val, err := ParseLocalDateTime(v)
if err != nil {
p.raiseError(tok, "%s", err)
}
return val
}
offset := p.getToken()
layout := time.RFC3339Nano
v := localDate.val + "T" + localTime.val + offset.val
val, err := time.ParseInLocation(layout, v, time.UTC)
if err != nil {
p.raiseError(tok, "%s", err)
}
return val
case tokenLeftBracket:
return p.parseArray()
case tokenLeftCurlyBrace:
return p.parseInlineTable()
case tokenEqual:
p.raiseError(tok, "cannot have multiple equals for the same key")
case tokenError:
p.raiseError(tok, "%s", tok)
default:
panic(fmt.Errorf("unhandled token: %v", tok))
}
return nil
}
func tokenIsComma(t *token) bool {
return t != nil && t.typ == tokenComma
}
func (p *tomlParser) parseInlineTable() *Tree {
tree := newTree()
var previous *token
Loop:
for {
follow := p.peek()
if follow == nil || follow.typ == tokenEOF {
p.raiseError(follow, "unterminated inline table")
}
switch follow.typ {
case tokenRightCurlyBrace:
p.getToken()
break Loop
case tokenKey, tokenInteger, tokenString:
if !tokenIsComma(previous) && previous != nil {
p.raiseError(follow, "comma expected between fields in inline table")
}
key := p.getToken()
p.assume(tokenEqual)
parsedKey, err := parseKey(key.val)
if err != nil {
p.raiseError(key, "invalid key: %s", err)
}
value := p.parseRvalue()
tree.SetPath(parsedKey, value)
case tokenComma:
if tokenIsComma(previous) {
p.raiseError(follow, "need field between two commas in inline table")
}
p.getToken()
default:
p.raiseError(follow, "unexpected token type in inline table: %s", follow.String())
}
previous = follow
}
if tokenIsComma(previous) {
p.raiseError(previous, "trailing comma at the end of inline table")
}
tree.inline = true
return tree
}
func (p *tomlParser) parseArray() interface{} {
var array []interface{}
arrayType := reflect.TypeOf(newTree())
for {
follow := p.peek()
if follow == nil || follow.typ == tokenEOF {
p.raiseError(follow, "unterminated array")
}
if follow.typ == tokenRightBracket {
p.getToken()
break
}
val := p.parseRvalue()
if reflect.TypeOf(val) != arrayType {
arrayType = nil
}
array = append(array, val)
follow = p.peek()
if follow == nil || follow.typ == tokenEOF {
p.raiseError(follow, "unterminated array")
}
if follow.typ != tokenRightBracket && follow.typ != tokenComma {
p.raiseError(follow, "missing comma")
}
if follow.typ == tokenComma {
p.getToken()
}
}
// if the array is a mixed-type array or its length is 0,
// don't convert it to a table array
if len(array) <= 0 {
arrayType = nil
}
// An array of Trees is actually an array of inline
// tables, which is a shorthand for a table array. If the
// array was not converted from []interface{} to []*Tree,
// the two notations would not be equivalent.
if arrayType == reflect.TypeOf(newTree()) {
tomlArray := make([]*Tree, len(array))
for i, v := range array {
tomlArray[i] = v.(*Tree)
}
return tomlArray
}
return array
}
func parseToml(flow []token) *Tree {
result := newTree()
result.position = Position{1, 1}
parser := &tomlParser{
flowIdx: 0,
flow: flow,
tree: result,
currentTable: make([]string, 0),
seenTableKeys: make([]string, 0),
}
parser.run()
return result
}
+1171
View File
File diff suppressed because it is too large Load Diff
+29
View File
@@ -0,0 +1,29 @@
// Position support for go-toml
package toml
import (
"fmt"
)
// Position of a document element within a TOML document.
//
// Line and Col are both 1-indexed positions for the element's line number and
// column number, respectively. Values of zero or less will cause Invalid(),
// to return true.
type Position struct {
Line int // line within the document
Col int // column within the line
}
// String representation of the position.
// Displays 1-indexed line and column numbers.
func (p Position) String() string {
return fmt.Sprintf("(%d, %d)", p.Line, p.Col)
}
// Invalid returns whether or not the position is valid (i.e. with negative or
// null values)
func (p Position) Invalid() bool {
return p.Line <= 0 || p.Col <= 0
}
+29
View File
@@ -0,0 +1,29 @@
// Testing support for go-toml
package toml
import (
"testing"
)
func TestPositionString(t *testing.T) {
p := Position{123, 456}
expected := "(123, 456)"
value := p.String()
if value != expected {
t.Errorf("Expected %v, got %v instead", expected, value)
}
}
func TestInvalid(t *testing.T) {
for i, v := range []Position{
{0, 1234},
{1234, 0},
{0, 0},
} {
if !v.Invalid() {
t.Errorf("Position at %v is valid: %v", i, v)
}
}
}
+201
View File
@@ -0,0 +1,201 @@
# Query package
## Overview
Package query performs JSONPath-like queries on a TOML document.
The query path implementation is based loosely on the JSONPath specification:
http://goessner.net/articles/JsonPath/.
The idea behind a query path is to allow quick access to any element, or set
of elements within TOML document, with a single expression.
```go
result, err := query.CompileAndExecute("$.foo.bar.baz", tree)
```
This is roughly equivalent to:
```go
next := tree.Get("foo")
if next != nil {
next = next.Get("bar")
if next != nil {
next = next.Get("baz")
}
}
result := next
```
err is nil if any parsing exception occurs.
If no node in the tree matches the query, result will simply contain an empty list of
items.
As illustrated above, the query path is much more efficient, especially since
the structure of the TOML file can vary. Rather than making assumptions about
a document's structure, a query allows the programmer to make structured
requests into the document, and get zero or more values as a result.
## Query syntax
The syntax of a query begins with a root token, followed by any number
sub-expressions:
```
$
Root of the TOML tree. This must always come first.
.name
Selects child of this node, where 'name' is a TOML key
name.
['name']
Selects child of this node, where 'name' is a string
containing a TOML key name.
[index]
Selcts child array element at 'index'.
..expr
Recursively selects all children, filtered by an a union,
index, or slice expression.
..*
Recursive selection of all nodes at this point in the
tree.
.*
Selects all children of the current node.
[expr,expr]
Union operator - a logical 'or' grouping of two or more
sub-expressions: index, key name, or filter.
[start:end:step]
Slice operator - selects array elements from start to
end-1, at the given step. All three arguments are
optional.
[?(filter)]
Named filter expression - the function 'filter' is
used to filter children at this node.
```
## Query Indexes And Slices
Index expressions perform no bounds checking, and will contribute no
values to the result set if the provided index or index range is invalid.
Negative indexes represent values from the end of the array, counting backwards.
```go
// select the last index of the array named 'foo'
query.CompileAndExecute("$.foo[-1]", tree)
```
Slice expressions are supported, by using ':' to separate a start/end index pair.
```go
// select up to the first five elements in the array
query.CompileAndExecute("$.foo[0:5]", tree)
```
Slice expressions also allow negative indexes for the start and stop
arguments.
```go
// select all array elements except the last one.
query.CompileAndExecute("$.foo[0:-1]", tree)
```
Slice expressions may have an optional stride/step parameter:
```go
// select every other element
query.CompileAndExecute("$.foo[0::2]", tree)
```
Slice start and end parameters are also optional:
```go
// these are all equivalent and select all the values in the array
query.CompileAndExecute("$.foo[:]", tree)
query.CompileAndExecute("$.foo[::]", tree)
query.CompileAndExecute("$.foo[::1]", tree)
query.CompileAndExecute("$.foo[0:]", tree)
query.CompileAndExecute("$.foo[0::]", tree)
query.CompileAndExecute("$.foo[0::1]", tree)
```
## Query Filters
Query filters are used within a Union [,] or single Filter [] expression.
A filter only allows nodes that qualify through to the next expression,
and/or into the result set.
```go
// returns children of foo that are permitted by the 'bar' filter.
query.CompileAndExecute("$.foo[?(bar)]", tree)
```
There are several filters provided with the library:
```
tree
Allows nodes of type Tree.
int
Allows nodes of type int64.
float
Allows nodes of type float64.
string
Allows nodes of type string.
time
Allows nodes of type time.Time.
bool
Allows nodes of type bool.
```
## Query Results
An executed query returns a Result object. This contains the nodes
in the TOML tree that qualify the query expression. Position information
is also available for each value in the set.
```go
// display the results of a query
results := query.CompileAndExecute("$.foo.bar.baz", tree)
for idx, value := results.Values() {
fmt.Println("%v: %v", results.Positions()[idx], value)
}
```
## Compiled Queries
Queries may be executed directly on a Tree object, or compiled ahead
of time and executed discretely. The former is more convenient, but has the
penalty of having to recompile the query expression each time.
```go
// basic query
results := query.CompileAndExecute("$.foo.bar.baz", tree)
// compiled query
query, err := toml.Compile("$.foo.bar.baz")
results := query.Execute(tree)
// run the compiled query again on a different tree
moreResults := query.Execute(anotherTree)
```
## User Defined Query Filters
Filter expressions may also be user defined by using the SetFilter()
function on the Query object. The function must return true/false, which
signifies if the passed node is kept or discarded, respectively.
```go
// create a query that references a user-defined filter
query, _ := query.Compile("$[?(bazOnly)]")
// define the filter, and assign it to the query
query.SetFilter("bazOnly", func(node interface{}) bool{
if tree, ok := node.(*Tree); ok {
return tree.Has("baz")
}
return false // reject all other node types
})
// run the query
query.Execute(tree)
```
+173
View File
@@ -0,0 +1,173 @@
// Package query performs JSONPath-like queries on a TOML document.
//
// The query path implementation is based loosely on the JSONPath specification:
// http://goessner.net/articles/JsonPath/.
//
// The idea behind a query path is to allow quick access to any element, or set
// of elements within TOML document, with a single expression.
//
// result, err := query.CompileAndExecute("$.foo.bar.baz", tree)
//
// This is roughly equivalent to:
//
// next := tree.Get("foo")
// if next != nil {
// next = next.Get("bar")
// if next != nil {
// next = next.Get("baz")
// }
// }
// result := next
//
// err is nil if any parsing exception occurs.
//
// If no node in the tree matches the query, result will simply contain an empty list of
// items.
//
// As illustrated above, the query path is much more efficient, especially since
// the structure of the TOML file can vary. Rather than making assumptions about
// a document's structure, a query allows the programmer to make structured
// requests into the document, and get zero or more values as a result.
//
// Query syntax
//
// The syntax of a query begins with a root token, followed by any number
// sub-expressions:
//
// $
// Root of the TOML tree. This must always come first.
// .name
// Selects child of this node, where 'name' is a TOML key
// name.
// ['name']
// Selects child of this node, where 'name' is a string
// containing a TOML key name.
// [index]
// Selcts child array element at 'index'.
// ..expr
// Recursively selects all children, filtered by an a union,
// index, or slice expression.
// ..*
// Recursive selection of all nodes at this point in the
// tree.
// .*
// Selects all children of the current node.
// [expr,expr]
// Union operator - a logical 'or' grouping of two or more
// sub-expressions: index, key name, or filter.
// [start:end:step]
// Slice operator - selects array elements from start to
// end-1, at the given step. All three arguments are
// optional.
// [?(filter)]
// Named filter expression - the function 'filter' is
// used to filter children at this node.
//
// Query Indexes And Slices
//
// Index expressions perform no bounds checking, and will contribute no
// values to the result set if the provided index or index range is invalid.
// Negative indexes represent values from the end of the array, counting backwards.
//
// // select the last index of the array named 'foo'
// query.CompileAndExecute("$.foo[-1]", tree)
//
// Slice expressions are supported, by using ':' to separate a start/end index pair.
//
// // select up to the first five elements in the array
// query.CompileAndExecute("$.foo[0:5]", tree)
//
// Slice expressions also allow negative indexes for the start and stop
// arguments.
//
// // select all array elements except the last one.
// query.CompileAndExecute("$.foo[0:-1]", tree)
//
// Slice expressions may have an optional stride/step parameter:
//
// // select every other element
// query.CompileAndExecute("$.foo[0::2]", tree)
//
// Slice start and end parameters are also optional:
//
// // these are all equivalent and select all the values in the array
// query.CompileAndExecute("$.foo[:]", tree)
// query.CompileAndExecute("$.foo[::]", tree)
// query.CompileAndExecute("$.foo[::1]", tree)
// query.CompileAndExecute("$.foo[0:]", tree)
// query.CompileAndExecute("$.foo[0::]", tree)
// query.CompileAndExecute("$.foo[0::1]", tree)
//
// Query Filters
//
// Query filters are used within a Union [,] or single Filter [] expression.
// A filter only allows nodes that qualify through to the next expression,
// and/or into the result set.
//
// // returns children of foo that are permitted by the 'bar' filter.
// query.CompileAndExecute("$.foo[?(bar)]", tree)
//
// There are several filters provided with the library:
//
// tree
// Allows nodes of type Tree.
// int
// Allows nodes of type int64.
// float
// Allows nodes of type float64.
// string
// Allows nodes of type string.
// time
// Allows nodes of type time.Time.
// bool
// Allows nodes of type bool.
//
// Query Results
//
// An executed query returns a Result object. This contains the nodes
// in the TOML tree that qualify the query expression. Position information
// is also available for each value in the set.
//
// // display the results of a query
// results := query.CompileAndExecute("$.foo.bar.baz", tree)
// for idx, value := results.Values() {
// fmt.Println("%v: %v", results.Positions()[idx], value)
// }
//
// Compiled Queries
//
// Queries may be executed directly on a Tree object, or compiled ahead
// of time and executed discretely. The former is more convenient, but has the
// penalty of having to recompile the query expression each time.
//
// // basic query
// results := query.CompileAndExecute("$.foo.bar.baz", tree)
//
// // compiled query
// query, err := toml.Compile("$.foo.bar.baz")
// results := query.Execute(tree)
//
// // run the compiled query again on a different tree
// moreResults := query.Execute(anotherTree)
//
// User Defined Query Filters
//
// Filter expressions may also be user defined by using the SetFilter()
// function on the Query object. The function must return true/false, which
// signifies if the passed node is kept or discarded, respectively.
//
// // create a query that references a user-defined filter
// query, _ := query.Compile("$[?(bazOnly)]")
//
// // define the filter, and assign it to the query
// query.SetFilter("bazOnly", func(node interface{}) bool{
// if tree, ok := node.(*Tree); ok {
// return tree.Has("baz")
// }
// return false // reject all other node types
// })
//
// // run the query
// query.Execute(tree)
//
package query
+357
View File
@@ -0,0 +1,357 @@
// TOML JSONPath lexer.
//
// Written using the principles developed by Rob Pike in
// http://www.youtube.com/watch?v=HxaD_trXwRE
package query
import (
"fmt"
"github.com/pelletier/go-toml"
"strconv"
"strings"
"unicode/utf8"
)
// Lexer state function
type queryLexStateFn func() queryLexStateFn
// Lexer definition
type queryLexer struct {
input string
start int
pos int
width int
tokens chan token
depth int
line int
col int
stringTerm string
}
func (l *queryLexer) run() {
for state := l.lexVoid; state != nil; {
state = state()
}
close(l.tokens)
}
func (l *queryLexer) nextStart() {
// iterate by runes (utf8 characters)
// search for newlines and advance line/col counts
for i := l.start; i < l.pos; {
r, width := utf8.DecodeRuneInString(l.input[i:])
if r == '\n' {
l.line++
l.col = 1
} else {
l.col++
}
i += width
}
// advance start position to next token
l.start = l.pos
}
func (l *queryLexer) emit(t tokenType) {
l.tokens <- token{
Position: toml.Position{Line: l.line, Col: l.col},
typ: t,
val: l.input[l.start:l.pos],
}
l.nextStart()
}
func (l *queryLexer) emitWithValue(t tokenType, value string) {
l.tokens <- token{
Position: toml.Position{Line: l.line, Col: l.col},
typ: t,
val: value,
}
l.nextStart()
}
func (l *queryLexer) next() rune {
if l.pos >= len(l.input) {
l.width = 0
return eof
}
var r rune
r, l.width = utf8.DecodeRuneInString(l.input[l.pos:])
l.pos += l.width
return r
}
func (l *queryLexer) ignore() {
l.nextStart()
}
func (l *queryLexer) backup() {
l.pos -= l.width
}
func (l *queryLexer) errorf(format string, args ...interface{}) queryLexStateFn {
l.tokens <- token{
Position: toml.Position{Line: l.line, Col: l.col},
typ: tokenError,
val: fmt.Sprintf(format, args...),
}
return nil
}
func (l *queryLexer) peek() rune {
r := l.next()
l.backup()
return r
}
func (l *queryLexer) accept(valid string) bool {
if strings.ContainsRune(valid, l.next()) {
return true
}
l.backup()
return false
}
func (l *queryLexer) follow(next string) bool {
return strings.HasPrefix(l.input[l.pos:], next)
}
func (l *queryLexer) lexVoid() queryLexStateFn {
for {
next := l.peek()
switch next {
case '$':
l.pos++
l.emit(tokenDollar)
continue
case '.':
if l.follow("..") {
l.pos += 2
l.emit(tokenDotDot)
} else {
l.pos++
l.emit(tokenDot)
}
continue
case '[':
l.pos++
l.emit(tokenLeftBracket)
continue
case ']':
l.pos++
l.emit(tokenRightBracket)
continue
case ',':
l.pos++
l.emit(tokenComma)
continue
case '*':
l.pos++
l.emit(tokenStar)
continue
case '(':
l.pos++
l.emit(tokenLeftParen)
continue
case ')':
l.pos++
l.emit(tokenRightParen)
continue
case '?':
l.pos++
l.emit(tokenQuestion)
continue
case ':':
l.pos++
l.emit(tokenColon)
continue
case '\'':
l.ignore()
l.stringTerm = string(next)
return l.lexString
case '"':
l.ignore()
l.stringTerm = string(next)
return l.lexString
}
if isSpace(next) {
l.next()
l.ignore()
continue
}
if isAlphanumeric(next) {
return l.lexKey
}
if next == '+' || next == '-' || isDigit(next) {
return l.lexNumber
}
if l.next() == eof {
break
}
return l.errorf("unexpected char: '%v'", next)
}
l.emit(tokenEOF)
return nil
}
func (l *queryLexer) lexKey() queryLexStateFn {
for {
next := l.peek()
if !isAlphanumeric(next) {
l.emit(tokenKey)
return l.lexVoid
}
if l.next() == eof {
break
}
}
l.emit(tokenEOF)
return nil
}
func (l *queryLexer) lexString() queryLexStateFn {
l.pos++
l.ignore()
growingString := ""
for {
if l.follow(l.stringTerm) {
l.emitWithValue(tokenString, growingString)
l.pos++
l.ignore()
return l.lexVoid
}
if l.follow("\\\"") {
l.pos++
growingString += "\""
} else if l.follow("\\'") {
l.pos++
growingString += "'"
} else if l.follow("\\n") {
l.pos++
growingString += "\n"
} else if l.follow("\\b") {
l.pos++
growingString += "\b"
} else if l.follow("\\f") {
l.pos++
growingString += "\f"
} else if l.follow("\\/") {
l.pos++
growingString += "/"
} else if l.follow("\\t") {
l.pos++
growingString += "\t"
} else if l.follow("\\r") {
l.pos++
growingString += "\r"
} else if l.follow("\\\\") {
l.pos++
growingString += "\\"
} else if l.follow("\\u") {
l.pos += 2
code := ""
for i := 0; i < 4; i++ {
c := l.peek()
l.pos++
if !isHexDigit(c) {
return l.errorf("unfinished unicode escape")
}
code = code + string(c)
}
l.pos--
intcode, err := strconv.ParseInt(code, 16, 32)
if err != nil {
return l.errorf("invalid unicode escape: \\u" + code)
}
growingString += string(rune(intcode))
} else if l.follow("\\U") {
l.pos += 2
code := ""
for i := 0; i < 8; i++ {
c := l.peek()
l.pos++
if !isHexDigit(c) {
return l.errorf("unfinished unicode escape")
}
code = code + string(c)
}
l.pos--
intcode, err := strconv.ParseInt(code, 16, 32)
if err != nil {
return l.errorf("invalid unicode escape: \\u" + code)
}
growingString += string(rune(intcode))
} else if l.follow("\\") {
l.pos++
return l.errorf("invalid escape sequence: \\" + string(l.peek()))
} else {
growingString += string(l.peek())
}
if l.next() == eof {
break
}
}
return l.errorf("unclosed string")
}
func (l *queryLexer) lexNumber() queryLexStateFn {
l.ignore()
if !l.accept("+") {
l.accept("-")
}
pointSeen := false
digitSeen := false
for {
next := l.next()
if next == '.' {
if pointSeen {
return l.errorf("cannot have two dots in one float")
}
if !isDigit(l.peek()) {
return l.errorf("float cannot end with a dot")
}
pointSeen = true
} else if isDigit(next) {
digitSeen = true
} else {
l.backup()
break
}
if pointSeen && !digitSeen {
return l.errorf("cannot start float with a dot")
}
}
if !digitSeen {
return l.errorf("no digit in that number")
}
if pointSeen {
l.emit(tokenFloat)
} else {
l.emit(tokenInteger)
}
return l.lexVoid
}
// Entry point
func lexQuery(input string) chan token {
l := &queryLexer{
input: input,
tokens: make(chan token),
line: 1,
col: 1,
}
go l.run()
return l.tokens
}
+179
View File
@@ -0,0 +1,179 @@
package query
import (
"github.com/pelletier/go-toml"
"testing"
)
func testQLFlow(t *testing.T, input string, expectedFlow []token) {
ch := lexQuery(input)
for idx, expected := range expectedFlow {
token := <-ch
if token != expected {
t.Log("While testing #", idx, ":", input)
t.Log("compared (got)", token, "to (expected)", expected)
t.Log("\tvalue:", token.val, "<->", expected.val)
t.Log("\tvalue as bytes:", []byte(token.val), "<->", []byte(expected.val))
t.Log("\ttype:", token.typ.String(), "<->", expected.typ.String())
t.Log("\tline:", token.Line, "<->", expected.Line)
t.Log("\tcolumn:", token.Col, "<->", expected.Col)
t.Log("compared", token, "to", expected)
t.FailNow()
}
}
tok, ok := <-ch
if ok {
t.Log("channel is not closed!")
t.Log(len(ch)+1, "tokens remaining:")
t.Log("token ->", tok)
for token := range ch {
t.Log("token ->", token)
}
t.FailNow()
}
}
func TestLexSpecialChars(t *testing.T) {
testQLFlow(t, " .$[]..()?*", []token{
{toml.Position{1, 2}, tokenDot, "."},
{toml.Position{1, 3}, tokenDollar, "$"},
{toml.Position{1, 4}, tokenLeftBracket, "["},
{toml.Position{1, 5}, tokenRightBracket, "]"},
{toml.Position{1, 6}, tokenDotDot, ".."},
{toml.Position{1, 8}, tokenLeftParen, "("},
{toml.Position{1, 9}, tokenRightParen, ")"},
{toml.Position{1, 10}, tokenQuestion, "?"},
{toml.Position{1, 11}, tokenStar, "*"},
{toml.Position{1, 12}, tokenEOF, ""},
})
}
func TestLexString(t *testing.T) {
testQLFlow(t, "'foo\n'", []token{
{toml.Position{1, 2}, tokenString, "foo\n"},
{toml.Position{2, 2}, tokenEOF, ""},
})
}
func TestLexDoubleString(t *testing.T) {
testQLFlow(t, `"bar"`, []token{
{toml.Position{1, 2}, tokenString, "bar"},
{toml.Position{1, 6}, tokenEOF, ""},
})
}
func TestLexStringEscapes(t *testing.T) {
testQLFlow(t, `"foo \" \' \b \f \/ \t \r \\ \u03A9 \U00012345 \n bar"`, []token{
{toml.Position{1, 2}, tokenString, "foo \" ' \b \f / \t \r \\ \u03A9 \U00012345 \n bar"},
{toml.Position{1, 55}, tokenEOF, ""},
})
}
func TestLexStringUnfinishedUnicode4(t *testing.T) {
testQLFlow(t, `"\u000"`, []token{
{toml.Position{1, 2}, tokenError, "unfinished unicode escape"},
})
}
func TestLexStringUnfinishedUnicode8(t *testing.T) {
testQLFlow(t, `"\U0000"`, []token{
{toml.Position{1, 2}, tokenError, "unfinished unicode escape"},
})
}
func TestLexStringInvalidEscape(t *testing.T) {
testQLFlow(t, `"\x"`, []token{
{toml.Position{1, 2}, tokenError, "invalid escape sequence: \\x"},
})
}
func TestLexStringUnfinished(t *testing.T) {
testQLFlow(t, `"bar`, []token{
{toml.Position{1, 2}, tokenError, "unclosed string"},
})
}
func TestLexKey(t *testing.T) {
testQLFlow(t, "foo", []token{
{toml.Position{1, 1}, tokenKey, "foo"},
{toml.Position{1, 4}, tokenEOF, ""},
})
}
func TestLexRecurse(t *testing.T) {
testQLFlow(t, "$..*", []token{
{toml.Position{1, 1}, tokenDollar, "$"},
{toml.Position{1, 2}, tokenDotDot, ".."},
{toml.Position{1, 4}, tokenStar, "*"},
{toml.Position{1, 5}, tokenEOF, ""},
})
}
func TestLexBracketKey(t *testing.T) {
testQLFlow(t, "$[foo]", []token{
{toml.Position{1, 1}, tokenDollar, "$"},
{toml.Position{1, 2}, tokenLeftBracket, "["},
{toml.Position{1, 3}, tokenKey, "foo"},
{toml.Position{1, 6}, tokenRightBracket, "]"},
{toml.Position{1, 7}, tokenEOF, ""},
})
}
func TestLexSpace(t *testing.T) {
testQLFlow(t, "foo bar baz", []token{
{toml.Position{1, 1}, tokenKey, "foo"},
{toml.Position{1, 5}, tokenKey, "bar"},
{toml.Position{1, 9}, tokenKey, "baz"},
{toml.Position{1, 12}, tokenEOF, ""},
})
}
func TestLexInteger(t *testing.T) {
testQLFlow(t, "100 +200 -300", []token{
{toml.Position{1, 1}, tokenInteger, "100"},
{toml.Position{1, 5}, tokenInteger, "+200"},
{toml.Position{1, 10}, tokenInteger, "-300"},
{toml.Position{1, 14}, tokenEOF, ""},
})
}
func TestLexFloat(t *testing.T) {
testQLFlow(t, "100.0 +200.0 -300.0", []token{
{toml.Position{1, 1}, tokenFloat, "100.0"},
{toml.Position{1, 7}, tokenFloat, "+200.0"},
{toml.Position{1, 14}, tokenFloat, "-300.0"},
{toml.Position{1, 20}, tokenEOF, ""},
})
}
func TestLexFloatWithMultipleDots(t *testing.T) {
testQLFlow(t, "4.2.", []token{
{toml.Position{1, 1}, tokenError, "cannot have two dots in one float"},
})
}
func TestLexFloatLeadingDot(t *testing.T) {
testQLFlow(t, "+.1", []token{
{toml.Position{1, 1}, tokenError, "cannot start float with a dot"},
})
}
func TestLexFloatWithTrailingDot(t *testing.T) {
testQLFlow(t, "42.", []token{
{toml.Position{1, 1}, tokenError, "float cannot end with a dot"},
})
}
func TestLexNumberWithoutDigit(t *testing.T) {
testQLFlow(t, "+", []token{
{toml.Position{1, 1}, tokenError, "no digit in that number"},
})
}
func TestLexUnknown(t *testing.T) {
testQLFlow(t, "^", []token{
{toml.Position{1, 1}, tokenError, "unexpected char: '94'"},
})
}

Some files were not shown because too many files have changed in this diff Show More