278 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
powe97	aec272e28e	Run once a day	2024-03-06 02:43:41 -06:00
powe97	c21dec05ad	Re-enable fail-fast	2024-03-06 02:36:33 -06:00
powe97	81ba2fdc80	Make failing actually fail the program	2024-03-06 02:35:18 -06:00
powe97	92c3327b1a	Fix debug prints	2024-03-06 02:09:42 -06:00
powe97	6a1395c054	Fix workflow not being able to commit and remove tee	2024-03-06 02:05:49 -06:00
powe97	912b07f6f3	Add retrying first page	2024-03-06 01:18:49 -06:00
powe97	8b15438a98	Actually use the retry version of the function...	2024-03-06 01:03:23 -06:00
powe97	0007bde18a	Recombine JSONs	2024-03-06 00:43:26 -06:00
powe97	c98b928125	Add retrying	2024-03-05 22:54:42 -05:00
powe97	a0b9081f8f	--headless	2024-03-05 21:14:32 -05:00
powe97	4f69c1d8a0	Re-get the page to try circumvent timeout	2024-03-05 21:14:00 -05:00
powe97	56c9268398	Disable fail-fast	2024-03-05 20:49:08 -05:00
powe97	02b383b90b	Extend timeout	2024-03-05 20:47:41 -05:00
powe97	95e8238786	Merge branch 'main' of https://github.com/quatalog/quatalog	2024-03-05 19:10:16 -05:00
powe97	fc72fda5de	Remove jump debug print	2024-03-05 19:10:10 -05:00
powe97	e45318404d	Update transfer.yml	2024-03-05 19:06:49 -05:00
powe97	10715c89e3	Update transfer.yml	2024-03-05 19:05:41 -05:00
powe97	52fdab6ce6	Make everything stderr print	2024-03-05 19:03:54 -05:00
powe97	42dbf3c19a	Update transfer.yml	2024-03-05 18:46:02 -05:00
powe97	985f40c4e7	Set up matrix jobs	2024-03-05 18:42:05 -05:00
powe97	cb24d84b46	Merge branch 'main' of https://github.com/quatalog/quatalog	2024-03-05 18:38:17 -05:00
powe97	ce2f22b23b	Merge branch 'main' of https://github.com/quatalog/quatalog	2024-03-05 18:38:12 -05:00
powe97	c8eadc06ee	Merge branch 'main' of https://github.com/quatalog/quatalog	2024-03-05 18:34:02 -05:00
powe97	6ad6f85708	Redesign scraper to not be unbearably slow	2024-03-05 18:33:54 -05:00
powe97	acdd08168f	Update transfer.yml	2024-03-05 18:27:51 -05:00
powe97	976b553b14	Reduce wait time	2024-03-04 17:03:13 -05:00
powe97	faf303ec27	Add termination	2024-03-03 23:53:25 -05:00
powe97	0b97a431e1	Clarify steps	2024-03-03 18:56:33 -05:00
powe97	9eb5e195dc	Use dd instead of rsync	2024-03-03 17:40:25 -05:00
powe97	57d9f5dc4c	See previous commit	2024-03-02 19:08:06 -05:00
powe97	58e509d1bb	Clean up and potentially fix rsync	2024-03-02 18:35:44 -05:00
powe97	c3e0d558d7	List data folder before copying (rsync seems to not work every time?)	2024-03-02 16:17:11 -05:00
powe97	ae286917c1	Formatting	2024-03-02 02:26:58 -05:00
powe97	80f9ed1d95	Merge branch 'main' of https://github.com/quatalog/quatalog	2024-03-02 02:25:26 -05:00
powe97	bc07c559bc	Uh-oh	2024-03-02 02:22:30 -05:00
powe97	d5f58d8576	Create json dir if it doesn't already exist	2024-03-02 01:44:28 -05:00
powe97	c508798f20	Create courses dir if it doesn't already exist	2024-03-02 01:41:43 -05:00
powe97	30f4f49cdb	Fix bug where only 1 page is scraped per school and refactor	2024-03-01 20:32:00 -05:00
powe97	baa74b8ee6	Fix issue where only 1 page per school would get scraped properly	2024-03-01 18:17:53 -05:00
powe97	5ea6816c90	Fix capitalization next to smart apostrophes (really?)	2024-03-01 17:21:45 -05:00
powe97	6b5356c84f	Fix typo leading to bad capitalization	2024-03-01 15:01:20 -05:00
powe97	001825d3dc	Merge branch 'main' of https://github.com/quatalog/quatalog	2024-03-01 13:32:09 -05:00
powe97	3b608fad41	Fix Roman numerals issue	2024-03-01 13:32:02 -05:00
powe97	5fe4ee9f13	Change manually run workflow to have timeout of 2 mins	2024-03-01 13:12:23 -05:00
powe97	997d3c16a8	2hr timeout → 45m	2024-03-01 12:48:20 -05:00
powe97	d03be03aeb	Move debug print to be more accurate	2024-03-01 01:50:01 -05:00
powe97	019b777228	Make transfer scraper run continuously (at least as much as Github allows)	2024-03-01 01:45:06 -05:00
powe97	1a4542e20e	Fix crashing without timeout arg and re-add --headless	2024-03-01 00:29:34 -05:00
powe97	b0acd0e745	Dammit python	2024-02-29 22:31:09 -05:00
powe97	53891400ea	Every 15 minutes	2024-02-29 22:29:51 -05:00