phantop/quatalog-scraper
1
0
Fork 0
mirror of https://github.com/quatalog/quatalog.git synced 2024-11-18 04:52:51 +00:00
Code Issues Packages Projects Releases Wiki Activity
251 commits 1 branch 0 tags 6.4 MiB
Commit graph

251 commits

This branch
This branch
All branches
Author SHA1 Message Date
powe97 0b97a431e1
Clarify steps 2024-03-03 18:56:33 -05:00
powe97 9eb5e195dc
Use dd instead of rsync 2024-03-03 17:40:25 -05:00
powe97 57d9f5dc4c
See previous commit 2024-03-02 19:08:06 -05:00
powe97 58e509d1bb
Clean up and potentially fix rsync 2024-03-02 18:35:44 -05:00
powe97 c3e0d558d7
List data folder before copying (rsync seems to not work every time?) 2024-03-02 16:17:11 -05:00
powe97 ae286917c1
Formatting 2024-03-02 02:26:58 -05:00
powe97 80f9ed1d95
Merge branch 'main' of https://github.com/quatalog/quatalog 2024-03-02 02:25:26 -05:00
powe97 bc07c559bc
Uh-oh 2024-03-02 02:22:30 -05:00
powe97 d5f58d8576
Create json dir if it doesn't already exist 2024-03-02 01:44:28 -05:00
powe97 c508798f20
Create courses dir if it doesn't already exist 2024-03-02 01:41:43 -05:00
powe97 30f4f49cdb
Fix bug where only 1 page is scraped per school and refactor 2024-03-01 20:32:00 -05:00
powe97 baa74b8ee6
Fix issue where only 1 page per school would get scraped properly 2024-03-01 18:17:53 -05:00
powe97 5ea6816c90
Fix capitalization next to smart apostrophes (really?) 2024-03-01 17:21:45 -05:00
powe97 6b5356c84f
Fix typo leading to bad capitalization 2024-03-01 15:01:20 -05:00
powe97 001825d3dc
Merge branch 'main' of https://github.com/quatalog/quatalog 2024-03-01 13:32:09 -05:00
powe97 3b608fad41
Fix Roman numerals issue 2024-03-01 13:32:02 -05:00
powe97 5fe4ee9f13
Change manually run workflow to have timeout of 2 mins 2024-03-01 13:12:23 -05:00
powe97 997d3c16a8
2hr timeout → 45m 2024-03-01 12:48:20 -05:00
powe97 d03be03aeb
Move debug print to be more accurate 2024-03-01 01:50:01 -05:00
powe97 019b777228
Make transfer scraper run continuously (at least as much as Github allows) 2024-03-01 01:45:06 -05:00
powe97 1a4542e20e
Fix crashing without timeout arg and re-add --headless 2024-03-01 00:29:34 -05:00
powe97 b0acd0e745
Dammit python 2024-02-29 22:31:09 -05:00
powe97 53891400ea
Every 15 minutes 2024-02-29 22:29:51 -05:00
powe97 c6e28d399a
Make timeout field have default value 2024-02-29 22:28:00 -05:00
powe97 682b1679b4
Run every 15 mins 2024-02-29 22:25:22 -05:00
powe97 aa4af079f8
Merge branch 'main' of https://github.com/quatalog/quatalog 2024-02-29 22:13:54 -05:00
powe97 cf2abf7193
Fix partial updates when KeyboardInterrupt happens mid-institution 2024-02-29 22:13:44 -05:00
powe97 55e34c9dd4
Bump versions of actions 2024-02-29 22:06:23 -05:00
powe97 efad1e9103
Bump versions for actions 2024-02-29 22:01:40 -05:00
powe97 cf953b2f02
Merge branch 'main' of https://github.com/quatalog/quatalog 2024-02-29 21:45:29 -05:00
powe97 44067261c3
Don't put whole repo in artifact 2024-02-29 21:45:22 -05:00
powe97 d268233d8b
Update transfer.yml 2024-02-29 21:38:17 -05:00
powe97 8a3e8a84d8
See previous commit 2024-02-29 21:25:53 -05:00
powe97 fd2da56aee
Make checkout data repo actually check the data repo out 2024-02-29 21:23:29 -05:00
powe97 12d844ca28
Fix global var fuckery 2024-02-29 21:21:39 -05:00
powe97 4916feeb19
Add debug timeout to workflow 2024-02-29 21:16:07 -05:00
powe97 b304e9f8d2
Fix scraper 2024-02-29 21:02:38 -05:00
powe97 f216c45748
Add if __name__ == "__main__" and fix workflow 2024-02-29 20:49:45 -05:00
powe97 15b09123ee
Set up workflow for transfer scraper 2024-02-29 20:40:15 -05:00
powe97 382f9080e5
Add transfer workflow 2024-02-29 17:18:50 -05:00
powe97 9585cc37e4
Add transfer scraper 2024-02-29 16:51:07 -05:00
powe97 db033af56e
src -> courseinfo_scraper 2024-02-29 16:43:38 -05:00
powe97 03fd5b8494
Disable autocompile 2024-02-01 23:28:41 -05:00
Quatalog Compiler e5ebe91a5c Recompile scraper : Fri Feb 2 04:28:20 UTC 2024 2024-02-02 04:28:20 +00:00
powe97 78e39968b2
Delete src/terms_offered.json 2024-02-01 23:27:55 -05:00
powe97 1134a8722d
Delete src/terms_list.json 2024-02-01 23:27:48 -05:00
powe97 e698598693
Delete src/prerequisites.json 2024-02-01 23:27:40 -05:00
powe97 a4bb5bdac2
Delete src/catalog.json 2024-02-01 23:27:27 -05:00
Quatalog Compiler 986649bade Recompile scraper : Wed Aug 9 01:38:18 UTC 2023 2023-08-09 01:38:18 +00:00
powe97 b81968e43a
Change to an ordered set so that the courses_list doesn't change order every rescrape 2023-08-08 21:37:19 -04:00