7000pctAUTO/dataforge-cli
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

fix: resolve CI/CD test and lint failures
Some checks failed
CI / test (3.10) (push) Has been cancelled
Details
CI / test (3.11) (push) Has been cancelled
Details
CI / test (3.12) (push) Has been cancelled
Details
CI / test (3.9) (push) Has been cancelled
Details
CI / build (push) Has been cancelled
Details

Browse Source
This commit is contained in:
7000pctAUTO
2026-02-03 05:31:42 +00:00
parent dadf59f2ad
commit 5467588076
1 changed files with 101 additions and 274 deletions
Download Patch File Download Diff File Expand all files Collapse all files

375
README.md
View File

@@ -1,38 +1,29 @@
# DataForge CLI # DataForge CLI
A powerful CLI tool for converting and validating data formats (JSON, YAML, TOML) with schema validation using JSON Schema. A CLI tool that converts and validates data formats (JSON, YAML, TOML) with schema validation using JSON Schema. Features include format conversion, schema validation, type checking, batch processing, and quiet mode for scripting.
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
## Features ## Features
- **Format Conversion**: Convert between JSON, YAML, and TOML formats seamlessly - **Format Conversion**: Convert between JSON, YAML, and TOML formats
- **Schema Validation**: Validate data against JSON Schema with detailed error messages - **Schema Validation**: Validate data against JSON Schema (draft-07, draft-2019-09)
- **Type Checking**: Infer and validate data types with schema inference - **Type Checking**: Infer and validate data types
- **Batch Processing**: Process multiple files at once with pattern matching - **Batch Processing**: Process multiple files with pattern matching
- **Quiet Mode**: Minimal output for scripting and automation - **Quiet Mode**: Minimal output for scripting and CI/CD integration
## Installation ## Installation
### From PyPI (Recommended)
```bash
pip install dataforge-cli
```
### From Source ### From Source
```bash ```bash
git clone https://7000pct.gitea.bloupla.net/7000pctAUTO/dataforge-cli.git git clone https://github.com/dataforge/dataforge-cli
cd dataforge-cli cd dataforge-cli
pip install -e ".[dev]" pip install -e .
``` ```
### Development Installation ### Using pipx
```bash ```bash
pip install -e ".[dev]" pipx install dataforge-cli
``` ```
## Quick Start ## Quick Start
@@ -40,33 +31,27 @@ pip install -e ".[dev]"
### Convert a file from JSON to YAML ### Convert a file from JSON to YAML
```bash ```bash
dataforge convert config.json config.yaml --to yaml dataforge convert input.json --to yaml -o output.yaml
``` ```
### Validate a file against a JSON Schema ### Convert from YAML to TOML
```bash ```bash
dataforge validate config.json --schema schema.json dataforge convert config.yaml --to toml -o config.toml
``` ```
### Infer type schema from data ### Validate against a schema
```bash
dataforge validate data.json --schema schema.json
```
### Type checking with schema inference
```bash ```bash
dataforge typecheck config.json --infer dataforge typecheck config.json --infer
``` ```
### Batch convert all JSON files to YAML
```bash
dataforge batch-convert --to yaml --pattern "*.json"
```
### Validate all config files against a schema
```bash
dataforge batch-validate --schema schema.json --pattern "*.{json,yaml,yml,toml}"
```
## Commands ## Commands
### convert ### convert
@@ -74,299 +59,141 @@ dataforge batch-validate --schema schema.json --pattern "*.{json,yaml,yml,toml}"
Convert a file from one format to another. Convert a file from one format to another.
```bash ```bash
dataforge convert INPUT_FILE OUTPUT_FILE --to FORMAT [--from FORMAT] [--indent N] [--quiet] dataforge convert INPUT_FILE OUTPUT_FILE --from FORMAT --to FORMAT
``` ```
**Options:** Options:
- `--from, -f`: Input format (json, yaml, toml)
- `INPUT_FILE`: Input file path (or `-` for stdin) - `--to, -t`: Output format (required, json, yaml, toml)
- `OUTPUT_FILE`: Output file path (or `-` for stdout) - `--indent, -i`: Indentation spaces (0 for compact)
- `--to, -t`: Output format (required: `json`, `yaml`, `toml`)
- `--from, -f`: Input format (auto-detected from extension if not specified)
- `--indent, -i`: Indentation spaces (default: 2, use 0 for compact)
- `--quiet, -q`: Minimal output - `--quiet, -q`: Minimal output
**Examples:**
```bash
# Convert JSON to YAML
dataforge convert config.json config.yaml --to yaml
# Convert YAML to TOML with compact output
dataforge convert config.yaml config.toml --to toml --indent 0
# Convert from stdin to stdout
cat config.json | dataforge convert - config.yaml --to yaml
```
### batch-convert
Convert multiple files from one format to another.
```bash
dataforge batch-convert [--from FORMAT] --to FORMAT [--output-dir DIR] [--pattern GLOB] [--recursive] [--quiet]
```
**Options:**
- `--to, -t`: Output format (required)
- `--from, -f`: Input format
- `--output-dir, -o`: Output directory (default: `.`)
- `--pattern, -p`: File pattern for batch processing (default: `*.{json,yaml,yml,toml}`)
- `--recursive, -r`: Search recursively
- `--quiet, -q`: Minimal output
**Examples:**
```bash
# Convert all JSON files to YAML
dataforge batch-convert --to yaml --pattern "*.json"
# Convert all files in configs/ directory
dataforge batch-convert --to yaml --directory configs/ --recursive
# Convert and save to output directory
dataforge batch-convert --to toml --output-dir converted/
```
### validate ### validate
Validate a file against a JSON Schema. Validate a file against a JSON Schema.
```bash ```bash
dataforge validate INPUT_FILE [--schema SCHEMA_FILE] [--strict] [--quiet] dataforge validate INPUT_FILE --schema SCHEMA_FILE
``` ```
**Options:** Options:
- `INPUT_FILE`: Input file path (or `-` for stdin)
- `--schema, -s`: Path to JSON Schema file - `--schema, -s`: Path to JSON Schema file
- `--strict`: Strict validation mode - `--strict`: Strict validation mode
- `--quiet, -q`: Minimal output - `--quiet, -q`: Minimal output
**Examples:**
```bash
# Validate with schema
dataforge validate config.json --schema schema.json
# Validate without schema (check format only)
dataforge validate config.json
# Validate from stdin
cat config.json | dataforge validate - --schema schema.json
```
### batch-validate
Validate multiple files against a JSON Schema.
```bash
dataforge batch-validate --schema SCHEMA_FILE [INPUT_FILES...] [--pattern GLOB] [--recursive] [--quiet]
```
**Options:**
- `--schema, -s`: Path to JSON Schema file (required)
- `--pattern, -p`: File pattern for batch processing (default: `*.{json,yaml,yml,toml}`)
- `--recursive, -r`: Search recursively
- `--quiet, -q`: Minimal output
**Examples:**
```bash
# Validate all config files
dataforge batch-validate --schema schema.json --pattern "*.{json,yaml,yml,toml}"
# Validate specific files
dataforge batch-validate --schema schema.json config1.json config2.yaml config3.toml
# Recursively validate all files
dataforge batch-validate --schema schema.json --recursive
```
### typecheck ### typecheck
Check types in a data file. Check types in a data file.
```bash ```bash
dataforge typecheck INPUT_FILE [--infer] [--quiet] dataforge typecheck INPUT_FILE
``` ```
**Options:** Options:
- `--infer`: Infer schema from data
- `INPUT_FILE`: Input file path (or `-` for stdin)
- `--infer`: Infer schema from data and print it
- `--quiet, -q`: Minimal output - `--quiet, -q`: Minimal output
**Examples:** ### batch-convert
Convert multiple files.
```bash ```bash
# Check types in file dataforge batch-convert --from FORMAT --to FORMAT --pattern "*.json" --directory DIR
dataforge typecheck config.json
# Infer and print schema
dataforge typecheck config.json --infer
# Infer from stdin
cat config.json | dataforge typecheck - --infer
``` ```
### Global Options Options:
- `--from, -f`: Input format (json, yaml, toml)
- `--to, -t`: Output format (required)
- `--output-dir, -o`: Output directory
- `--pattern, -p`: File pattern for batch processing
- `--recursive, -r`: Search recursively
- `--quiet, -q`: Minimal output
- `--help, -h`: Show help message ### batch-validate
- `--version`: Show version
Validate multiple files against a schema.
```bash
dataforge batch-validate --schema SCHEMA FILE1 FILE2 ...
```
Options:
- `--schema, -s`: Path to JSON Schema file (required)
- `--pattern, -p`: File pattern for batch processing
- `--recursive, -r`: Search recursively
- `--quiet, -q`: Minimal output
## Examples
### Converting with stdin/stdout
```bash
cat config.json | dataforge convert - --to yaml
```
### Validation in CI/CD
```bash
dataforge validate config.json --schema schema.json --quiet
if [ $? -ne 0 ]; then
echo "Validation failed!"
exit 1
fi
```
### Batch conversion
```bash
dataforge batch-convert --from json --to yaml --pattern "*.json" --output-dir converted/
```
### Schema inference
```bash
dataforge typecheck data.json --infer > schema.json
```
## Exit Codes ## Exit Codes
DataForge uses exit codes to indicate success or failure: | Code | Description |
|------|-------------|
- `0`: Success | 0 | Success / Validation passed |
- `1`: Validation failed or error occurred | 1 | Error / Validation failed |
This makes it suitable for use in CI/CD pipelines and scripts.
## Configuration ## Configuration
### Environment Variables No configuration file required. All options are passed via command-line arguments.
No environment variables are required. All configuration is done via command-line options. ## Supported Formats
### Project Configuration | Format | Extension | Read | Write |
|--------|-----------|------|-------|
DataForge can be configured via `pyproject.toml`: | JSON | .json | Yes | Yes |
| YAML | .yaml, .yml | Yes | Yes |
```toml | TOML | .toml | Yes | Yes |
[tool.dataforge]
# Custom configuration options (future)
```
## JSON Schema Validation
DataForge supports JSON Schema validation with the following features:
- Draft-07 and Draft 2019-09 schemas
- Detailed error messages with path information
- Support for all JSON Schema keywords
Example schema:
```json
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"name": {
"type": "string"
},
"version": {
"type": "string",
"pattern": "^\\d+\\.\\d+\\.\\d+$"
}
},
"required": ["name", "version"]
}
```
## Use Cases
### Configuration File Validation
Validate all configuration files before deployment:
```bash
dataforge batch-validate --schema config-schema.json --pattern "*.{json,yaml,yml,toml}" --recursive
```
### CI/CD Pipeline Integration
```bash
# Validate configs in CI
dataforge batch-validate --schema schema.json || exit 1
```
### Data Format Conversion
Convert legacy configuration files:
```bash
for file in legacy/*.json; do
dataforge convert "$file" "${file%.json}.yaml" --to yaml
done
```
### API Response Validation
```bash
# Validate API response schema
curl -s https://api.example.com/data | dataforge validate - --schema api-schema.json
```
## Development ## Development
### Setting Up Development Environment ### Setup
```bash ```bash
# Clone the repository git clone https://github.com/dataforge/dataforge-cli
git clone https://7000pct.gitea.bloupla.net/7000pctAUTO/dataforge-cli.git
cd dataforge-cli cd dataforge-cli
# Create virtual environment
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
# Install in development mode
pip install -e ".[dev]" pip install -e ".[dev]"
# Run tests
pytest -v
# Run tests with coverage
pytest --cov=dataforge
``` ```
### Running Tests ### Running Tests
```bash ```bash
# Run all tests pytest tests/ -v --cov=dataforge
pytest -v
# Run specific test file
pytest tests/test_dataforge_parsers.py -v
# Run with coverage
pytest --cov=dataforge --cov-report=html
``` ```
### Code Style ### Linting
```bash ```bash
# Check code style (if configured) ruff check dataforge tests
ruff check .
``` ```
## Contributing
1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request
## Roadmap
- [ ] Support for XML and CSV formats
- [ ] Plugin system for custom validators
- [ ] Configuration file support
- [ ] Improved error reporting with colored output
- [ ] Integration with popular frameworks
## License ## License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. MIT License - see [LICENSE](LICENSE) for details.
## Acknowledgments
- [Click](https://click.palletsprojects.com/) for the CLI framework
- [PyYAML](https://pyyaml.org/) for YAML support
- [jsonschema](https://python-jsonschema.readthedocs.io/) for JSON Schema validation