Xi Yan 8b45d147df [/datasetio] drop columns not specified by dataset schema for huggingface provider (#611) ... # What does this PR do? **Why** - huggingface datasets could have extra unused columns, some of these columns (e.g. images) is unable to be casted as JSON over http requests for datasetio. - it is also inefficient to create a new dataset that's a subset of columns **Solution** - drop columns not specified by dataset schema ## Test Plan Tested with script: https://gist.github.com/yanxi0830/23be5725e0d82d79e24cc5dd1d21b571 ## Sources Please link relevant resources if necessary. ## Before submitting - [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case). - [ ] Ran pre-commit to handle lint / formatting issues. - [ ] Read the [contributor guideline](https://github.com/meta-llama/llama-stack/blob/main/CONTRIBUTING.md), Pull Request section? - [ ] Updated relevant documentation. - [ ] Wrote necessary unit or integration tests.		2024-12-12 10:23:09 -08:00
..
huggingface	[/datasetio] drop columns not specified by dataset schema for huggingface provider (#611)	2024-12-12 10:23:09 -08:00
__init__.py	add missing __init__	2024-11-25 17:23:27 -08:00