136 commits

Author	SHA1	Message	Date
Xi Yan	52fe165db8	address comments	2024-10-25 11:08:40 -07:00
Xi Yan	6b0baa6d53	scoring_function -> scoring_fn rename, scorer -> scoring_fn rename	2024-10-24 21:06:38 -07:00
Xi Yan	cdfd584a8f	Merge branch 'main' into evals_6	2024-10-24 17:29:22 -07:00
Xi Yan	2372de70f8	error checking	2024-10-24 16:16:34 -07:00
Xi Yan	f6340a47d1	inclusion->subsetof	2024-10-24 16:13:49 -07:00
Xi Yan	29e48cc5c1	chatcompletion & completion input type validation	2024-10-24 16:11:25 -07:00
Dinesh Yeduguru	3e1c3fdb3f	completion() for tgi (#295)	2024-10-24 16:02:41 -07:00
Xi Yan	e468e23249	Merge branch 'main' into evals_6	2024-10-24 14:59:41 -07:00
Ashwin Bharambe	7b1a45ee0f	Fix score threshold in faiss	2024-10-24 14:57:10 -07:00
Dalton Flanagan	1721d91c95	Update iOS inference instructions for new quantization	2024-10-24 14:57:10 -07:00
Xi Yan	cb84034567	[Evals API][3/n] scoring_functions / scoring meta-reference implementations (#296) ... * wip * dataset validation * test_scoring * cleanup * clean up test * comments * error checking * dataset client * test client: * datasetio client * clean up * basic scoring function works * scorer wip * equality scorer * score batch impl * score batch * update scoring test * refactor * validate scorer input * address comments * add all rows scores to ScoringResult * bugfix * scoring function def rename	2024-10-24 14:52:30 -07:00
Xi Yan	d4887fc746	address comments	2024-10-24 14:49:02 -07:00
Xi Yan	ba0186f2c8	refactor	2024-10-24 14:00:41 -07:00
Xi Yan	3db1b3fbcd	rebase name	2024-10-24 13:53:41 -07:00
Xi Yan	97ca72288c	Merge branch 'evals_5' into evals_6	2024-10-24 13:53:00 -07:00
Xi Yan	6053b8dd34	scoring function def rename	2024-10-24 13:51:11 -07:00
Xi Yan	689990b48b	Merge branch 'evals_5' into evals_6	2024-10-24 13:06:11 -07:00
Xi Yan	42bac85e1f	bugfix	2024-10-24 12:16:28 -07:00
Ashwin Bharambe	205bcfdd4e	Fix score threshold in faiss	2024-10-24 12:11:58 -07:00
Xi Yan	24dce9cb7a	minor typing	2024-10-24 12:08:57 -07:00
Xi Yan	32a496ab0f	Merge branch 'evals_5' into evals_6	2024-10-24 12:01:41 -07:00
Xi Yan	a3a8f32541	add all rows scores to ScoringResult	2024-10-24 11:53:15 -07:00
Dalton Flanagan	8eceebec98	Update iOS inference instructions for new quantization	2024-10-24 14:47:27 -04:00
Xi Yan	737fcb795f	evals with generation	2024-10-24 11:30:13 -07:00
Xi Yan	071dba8871	Merge branch 'main' into evals_5	2024-10-24 09:18:15 -07:00
Ashwin Bharambe	7afe51c84d	New quantized models (#301)	2024-10-24 08:38:56 -07:00
Xi Yan	afa0c2b146	address comments	2024-10-23 22:17:38 -07:00
Ashwin Bharambe	05a8d47b98	Add a meta-reference-quantized-gpu distribution	2024-10-23 21:45:50 -07:00
Xi Yan	59c93548bc	validate scorer input	2024-10-23 17:43:41 -07:00
Xi Yan	0ee82571a8	refactor	2024-10-23 17:30:10 -07:00
Xi Yan	7c803cef86	update scoring test	2024-10-23 17:22:48 -07:00
Xi Yan	3c6555c408	score batch	2024-10-23 16:38:00 -07:00
Xi Yan	eb572faf6f	score batch impl	2024-10-23 16:19:25 -07:00
Xi Yan	4b1d7da030	equality scorer	2024-10-23 16:07:17 -07:00
Xi Yan	caf253e08f	Merge branch 'main' into evals_5	2024-10-23 15:33:00 -07:00
Xi Yan	35981a1a3b	scorer wip	2024-10-23 15:02:54 -07:00
Xi Yan	70c08e694d	basic scoring function works	2024-10-23 14:42:28 -07:00
Xi Yan	5930a92dc7	datasetio client	2024-10-23 14:04:51 -07:00
Xi Yan	bb43369521	dataset client	2024-10-23 13:53:58 -07:00
Xi Yan	c5db025320	error checking	2024-10-23 13:17:47 -07:00
Xi Yan	d8bbce6f7c	comments	2024-10-23 13:16:08 -07:00
Xi Yan	5e1323b5bf	clean up test	2024-10-23 13:08:42 -07:00
Xi Yan	555f6e1531	cleanup	2024-10-23 13:07:15 -07:00
Xi Yan	92e32f80ad	test_scoring	2024-10-23 13:01:49 -07:00
Xi Yan	7c280e18fb	dataset validation	2024-10-23 12:08:39 -07:00
Dinesh Yeduguru	21f2e9adf5	dont set num_predict for all providers (#294)	2024-10-23 11:44:04 -07:00
Ashwin Bharambe	ffb561070d	Support structured output for Together (#289)	2024-10-22 22:36:38 -07:00
Xi Yan	aefa84e70a	wip	2024-10-22 20:00:43 -07:00
Sarthak Deshpande	2e5e46d896	Added tests for persistence (#274)	2024-10-22 19:41:46 -07:00
Xi Yan	821810657f	[Evals API][2/n] datasets / datasetio meta-reference implementation (#288) ... * skeleton dataset / datasetio * dataset datasetio * config * address comments * delete dataset_utils * address comments * naming fix	2024-10-22 16:12:16 -07:00