287 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
powe97	779b979b9b	Fix credit count parsing	2024-03-07 11:03:59 -06:00
powe97	0f3652d8cc	Lint	2024-03-06 16:16:36 -06:00
powe97	7c87221256	Fix typo	2024-03-06 16:16:23 -06:00
powe97	69d8946f37	Improve credit count parsing	2024-03-06 16:13:13 -06:00
powe97	9c374bf130	Fix timeout issue (again)	2024-03-06 13:20:09 -06:00
powe97	10360ff57c	Wait for table	2024-03-06 12:48:48 -06:00
powe97	3b239f9cef	Merge branch 'main' of https://github.com/quatalog/quatalog	2024-03-06 12:41:13 -06:00
powe97	de89a56808	Add more debug printing	2024-03-06 12:41:08 -06:00
powe97	45cfba68ac	Sort output JSON	2024-03-06 12:10:23 -06:00
powe97	aec272e28e	Run once a day	2024-03-06 02:43:41 -06:00
powe97	c21dec05ad	Re-enable fail-fast	2024-03-06 02:36:33 -06:00
powe97	81ba2fdc80	Make failing actually fail the program	2024-03-06 02:35:18 -06:00
powe97	92c3327b1a	Fix debug prints	2024-03-06 02:09:42 -06:00
powe97	6a1395c054	Fix workflow not being able to commit and remove tee	2024-03-06 02:05:49 -06:00
powe97	912b07f6f3	Add retrying first page	2024-03-06 01:18:49 -06:00
powe97	8b15438a98	Actually use the retry version of the function...	2024-03-06 01:03:23 -06:00
powe97	0007bde18a	Recombine JSONs	2024-03-06 00:43:26 -06:00
powe97	c98b928125	Add retrying	2024-03-05 22:54:42 -05:00
powe97	a0b9081f8f	--headless	2024-03-05 21:14:32 -05:00
powe97	4f69c1d8a0	Re-get the page to try circumvent timeout	2024-03-05 21:14:00 -05:00
powe97	56c9268398	Disable fail-fast	2024-03-05 20:49:08 -05:00
powe97	02b383b90b	Extend timeout	2024-03-05 20:47:41 -05:00
powe97	95e8238786	Merge branch 'main' of https://github.com/quatalog/quatalog	2024-03-05 19:10:16 -05:00
powe97	fc72fda5de	Remove jump debug print	2024-03-05 19:10:10 -05:00
powe97	e45318404d	Update transfer.yml	2024-03-05 19:06:49 -05:00
powe97	10715c89e3	Update transfer.yml	2024-03-05 19:05:41 -05:00
powe97	52fdab6ce6	Make everything stderr print	2024-03-05 19:03:54 -05:00
powe97	42dbf3c19a	Update transfer.yml	2024-03-05 18:46:02 -05:00
powe97	985f40c4e7	Set up matrix jobs	2024-03-05 18:42:05 -05:00
powe97	cb24d84b46	Merge branch 'main' of https://github.com/quatalog/quatalog	2024-03-05 18:38:17 -05:00
powe97	ce2f22b23b	Merge branch 'main' of https://github.com/quatalog/quatalog	2024-03-05 18:38:12 -05:00
powe97	c8eadc06ee	Merge branch 'main' of https://github.com/quatalog/quatalog	2024-03-05 18:34:02 -05:00
powe97	6ad6f85708	Redesign scraper to not be unbearably slow	2024-03-05 18:33:54 -05:00
powe97	acdd08168f	Update transfer.yml	2024-03-05 18:27:51 -05:00
powe97	976b553b14	Reduce wait time	2024-03-04 17:03:13 -05:00
powe97	faf303ec27	Add termination	2024-03-03 23:53:25 -05:00
powe97	0b97a431e1	Clarify steps	2024-03-03 18:56:33 -05:00
powe97	9eb5e195dc	Use dd instead of rsync	2024-03-03 17:40:25 -05:00
powe97	57d9f5dc4c	See previous commit	2024-03-02 19:08:06 -05:00
powe97	58e509d1bb	Clean up and potentially fix rsync	2024-03-02 18:35:44 -05:00
powe97	c3e0d558d7	List data folder before copying (rsync seems to not work every time?)	2024-03-02 16:17:11 -05:00
powe97	ae286917c1	Formatting	2024-03-02 02:26:58 -05:00
powe97	80f9ed1d95	Merge branch 'main' of https://github.com/quatalog/quatalog	2024-03-02 02:25:26 -05:00
powe97	bc07c559bc	Uh-oh	2024-03-02 02:22:30 -05:00
powe97	d5f58d8576	Create json dir if it doesn't already exist	2024-03-02 01:44:28 -05:00
powe97	c508798f20	Create courses dir if it doesn't already exist	2024-03-02 01:41:43 -05:00
powe97	30f4f49cdb	Fix bug where only 1 page is scraped per school and refactor	2024-03-01 20:32:00 -05:00
powe97	baa74b8ee6	Fix issue where only 1 page per school would get scraped properly	2024-03-01 18:17:53 -05:00
powe97	5ea6816c90	Fix capitalization next to smart apostrophes (really?)	2024-03-01 17:21:45 -05:00
powe97	6b5356c84f	Fix typo leading to bad capitalization	2024-03-01 15:01:20 -05:00