yeyu-nvidiaandClaude Opus 4.6 289a239ca5 fix: use data_dir for directory paths in ShardedDataset (#1301) ...

## Summary
- `datasets`' `resolve_pattern` only matches entries with
`type=="file"`, so passing a bare directory path as `data_files` to
`load_dataset` results in `FileNotFoundError` even when the directory
exists on disk
- Detect directory paths in `ShardedDataset._load_dataset()` and pass
them via `data_dir` instead of `data_files`

## Reproduction
```python
from datasets import load_dataset
# This fails with FileNotFoundError:
load_dataset("json", data_files="/path/to/data_directory")
# This works:
load_dataset("json", data_dir="/path/to/data_directory")
```

## Test plan
- [ ] Verify existing EAGLE3/DFlash training pipelines that pass
directory paths work
- [ ] Verify file path and glob patterns still work (falls through to
`data_files`)
- [ ] Verify `data_files=None` (no data_files arg) still works

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

## Bug Fixes

* Fixed an issue with dataset loading that prevented proper handling of
directory-based data sources. Directories are now correctly detected and
processed during dataset initialization.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Signed-off-by: Ye Yu <yeyu@nvidia.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

2026-04-20 19:10:02 +00:00

..

deploy

Fix LLM deploy test failure by defaulting expert parallelism to 1 (#1273)

2026-04-16 16:18:38 -07:00

onnx

Add ResNet50 support for torch_onnx quantization workflow (#1263)

2026-04-18 14:30:45 +00:00

recipe

use typed quantize config instead of a raw dict (#1249)

2026-04-13 18:24:22 -07:00

torch

fix: use data_dir for directory paths in ShardedDataset (#1301)

2026-04-20 19:10:02 +00:00

__init__.py

Add experimental support for transformers>=5.0 + min torch 2.8 (#975)

2026-04-09 09:59:37 +05:30