phantop/quatalog-scraper
1
0
Fork 0
mirror of https://github.com/quatalog/quatalog.git synced 2025-08-30 02:14:56 +00:00
Code Issues Packages Projects Releases Wiki Activity
325 commits 2 branches 0 tags 6.9 MiB
Commit graph

51 commits

Author SHA1 Message Date
powe97 bdc6b2bcbc
Short circuit on IP ban 2024-03-16 21:16:14 -04:00
powe97 5e9e464ad0
Add CSV generator 2024-03-16 02:17:31 -04:00
powe97 ba3e5c77d1
Lint 2024-03-16 01:17:49 -04:00
powe97 c3be28e520
Add JSON converter from by-institution to by-course 2024-03-16 01:14:58 -04:00
powe97 061f9b14e5
Sort keys 2024-03-16 01:07:16 -04:00
powe97 1fa7ab61af
Rename transfer scraper 2024-03-16 00:32:59 -04:00
powe97 517952f977
Re-add catalog name scraping 2024-03-13 23:52:51 -04:00
powe97 f1a47dca48
stderr 2024-03-13 22:19:56 -04:00
powe97 af25410c5d
Log IP 2024-03-13 22:18:13 -04:00
powe97 b017436be9
Fix handling for courses that come in as multiple courses 2024-03-13 15:48:53 -04:00
powe97 4c0517f6c4
Print title 2024-03-07 16:14:54 -06:00
powe97 779b979b9b
Fix credit count parsing 2024-03-07 11:03:59 -06:00
powe97 0f3652d8cc
Lint 2024-03-06 16:16:36 -06:00
powe97 7c87221256
Fix typo 2024-03-06 16:16:23 -06:00
powe97 69d8946f37
Improve credit count parsing 2024-03-06 16:13:13 -06:00
powe97 9c374bf130
Fix timeout issue (again) 2024-03-06 13:20:09 -06:00
powe97 10360ff57c
Wait for table 2024-03-06 12:48:48 -06:00
powe97 de89a56808
Add more debug printing 2024-03-06 12:41:08 -06:00
powe97 81ba2fdc80
Make failing actually fail the program 2024-03-06 02:35:18 -06:00
powe97 92c3327b1a
Fix debug prints 2024-03-06 02:09:42 -06:00
powe97 912b07f6f3
Add retrying first page 2024-03-06 01:18:49 -06:00
powe97 8b15438a98
Actually use the retry version of the function... 2024-03-06 01:03:23 -06:00
powe97 c98b928125
Add retrying 2024-03-05 22:54:42 -05:00
powe97 a0b9081f8f
--headless 2024-03-05 21:14:32 -05:00
powe97 4f69c1d8a0
Re-get the page to try circumvent timeout 2024-03-05 21:14:00 -05:00
powe97 02b383b90b
Extend timeout 2024-03-05 20:47:41 -05:00
powe97 fc72fda5de
Remove jump debug print 2024-03-05 19:10:10 -05:00
powe97 52fdab6ce6
Make everything stderr print 2024-03-05 19:03:54 -05:00
powe97 ce2f22b23b
Merge branch 'main' of https://github.com/quatalog/quatalog 2024-03-05 18:38:12 -05:00
powe97 6ad6f85708
Redesign scraper to not be unbearably slow 2024-03-05 18:33:54 -05:00
powe97 976b553b14
Reduce wait time 2024-03-04 17:03:13 -05:00
powe97 faf303ec27
Add termination 2024-03-03 23:53:25 -05:00
powe97 ae286917c1
Formatting 2024-03-02 02:26:58 -05:00
powe97 80f9ed1d95
Merge branch 'main' of https://github.com/quatalog/quatalog 2024-03-02 02:25:26 -05:00
powe97 bc07c559bc
Uh-oh 2024-03-02 02:22:30 -05:00
powe97 30f4f49cdb
Fix bug where only 1 page is scraped per school and refactor 2024-03-01 20:32:00 -05:00
powe97 baa74b8ee6
Fix issue where only 1 page per school would get scraped properly 2024-03-01 18:17:53 -05:00
powe97 5ea6816c90
Fix capitalization next to smart apostrophes (really?) 2024-03-01 17:21:45 -05:00
powe97 6b5356c84f
Fix typo leading to bad capitalization 2024-03-01 15:01:20 -05:00
powe97 3b608fad41
Fix Roman numerals issue 2024-03-01 13:32:02 -05:00
powe97 d03be03aeb
Move debug print to be more accurate 2024-03-01 01:50:01 -05:00
powe97 1a4542e20e
Fix crashing without timeout arg and re-add --headless 2024-03-01 00:29:34 -05:00
powe97 b0acd0e745
Dammit python 2024-02-29 22:31:09 -05:00
powe97 c6e28d399a
Make timeout field have default value 2024-02-29 22:28:00 -05:00
powe97 cf2abf7193
Fix partial updates when KeyboardInterrupt happens mid-institution 2024-02-29 22:13:44 -05:00
powe97 8a3e8a84d8
See previous commit 2024-02-29 21:25:53 -05:00
powe97 12d844ca28
Fix global var fuckery 2024-02-29 21:21:39 -05:00
powe97 4916feeb19
Add debug timeout to workflow 2024-02-29 21:16:07 -05:00
powe97 b304e9f8d2
Fix scraper 2024-02-29 21:02:38 -05:00
powe97 f216c45748
Add if __name__ == "__main__" and fix workflow 2024-02-29 20:49:45 -05:00