phantop/quatalog-scraper
1
0
Fork 0
mirror of https://github.com/quatalog/quatalog.git synced 2024-11-16 03:52:55 +00:00
Code Issues Packages Projects Releases Wiki Activity
299 commits 1 branch 0 tags 6.4 MiB
Commit graph

299 commits

This branch
This branch
All branches
Author SHA1 Message Date
powe97 de7360f1c7
Convert JSON to by-course format 2024-03-16 01:28:33 -04:00
powe97 ba3e5c77d1
Lint 2024-03-16 01:17:49 -04:00
powe97 c3be28e520
Add JSON converter from by-institution to by-course 2024-03-16 01:14:58 -04:00
powe97 061f9b14e5
Sort keys 2024-03-16 01:07:16 -04:00
powe97 1fa7ab61af
Rename transfer scraper 2024-03-16 00:32:59 -04:00
Quatalog Compiler bd930f05e9 Recompile scraper : Sat Mar 16 03:47:31 UTC 2024 2024-03-16 03:47:31 +00:00
powe97 77423ec069
Show music lessons properly 2024-03-15 23:46:28 -04:00
powe97 517952f977
Re-add catalog name scraping 2024-03-13 23:52:51 -04:00
powe97 f1a47dca48
stderr 2024-03-13 22:19:56 -04:00
powe97 af25410c5d
Log IP 2024-03-13 22:18:13 -04:00
powe97 b017436be9
Fix handling for courses that come in as multiple courses 2024-03-13 15:48:53 -04:00
powe97 4c0517f6c4
Print title 2024-03-07 16:14:54 -06:00
powe97 779b979b9b
Fix credit count parsing 2024-03-07 11:03:59 -06:00
powe97 0f3652d8cc
Lint 2024-03-06 16:16:36 -06:00
powe97 7c87221256
Fix typo 2024-03-06 16:16:23 -06:00
powe97 69d8946f37
Improve credit count parsing 2024-03-06 16:13:13 -06:00
powe97 9c374bf130
Fix timeout issue (again) 2024-03-06 13:20:09 -06:00
powe97 10360ff57c
Wait for table 2024-03-06 12:48:48 -06:00
powe97 3b239f9cef
Merge branch 'main' of https://github.com/quatalog/quatalog 2024-03-06 12:41:13 -06:00
powe97 de89a56808
Add more debug printing 2024-03-06 12:41:08 -06:00
powe97 45cfba68ac
Sort output JSON 2024-03-06 12:10:23 -06:00
powe97 aec272e28e
Run once a day 2024-03-06 02:43:41 -06:00
powe97 c21dec05ad
Re-enable fail-fast 2024-03-06 02:36:33 -06:00
powe97 81ba2fdc80
Make failing actually fail the program 2024-03-06 02:35:18 -06:00
powe97 92c3327b1a
Fix debug prints 2024-03-06 02:09:42 -06:00
powe97 6a1395c054
Fix workflow not being able to commit and remove tee 2024-03-06 02:05:49 -06:00
powe97 912b07f6f3
Add retrying first page 2024-03-06 01:18:49 -06:00
powe97 8b15438a98
Actually use the retry version of the function... 2024-03-06 01:03:23 -06:00
powe97 0007bde18a
Recombine JSONs 2024-03-06 00:43:26 -06:00
powe97 c98b928125
Add retrying 2024-03-05 22:54:42 -05:00
powe97 a0b9081f8f
--headless 2024-03-05 21:14:32 -05:00
powe97 4f69c1d8a0
Re-get the page to try circumvent timeout 2024-03-05 21:14:00 -05:00
powe97 56c9268398
Disable fail-fast 2024-03-05 20:49:08 -05:00
powe97 02b383b90b
Extend timeout 2024-03-05 20:47:41 -05:00
powe97 95e8238786
Merge branch 'main' of https://github.com/quatalog/quatalog 2024-03-05 19:10:16 -05:00
powe97 fc72fda5de
Remove jump debug print 2024-03-05 19:10:10 -05:00
powe97 e45318404d
Update transfer.yml 2024-03-05 19:06:49 -05:00
powe97 10715c89e3
Update transfer.yml 2024-03-05 19:05:41 -05:00
powe97 52fdab6ce6
Make everything stderr print 2024-03-05 19:03:54 -05:00
powe97 42dbf3c19a
Update transfer.yml 2024-03-05 18:46:02 -05:00
powe97 985f40c4e7
Set up matrix jobs 2024-03-05 18:42:05 -05:00
powe97 cb24d84b46
Merge branch 'main' of https://github.com/quatalog/quatalog 2024-03-05 18:38:17 -05:00
powe97 ce2f22b23b
Merge branch 'main' of https://github.com/quatalog/quatalog 2024-03-05 18:38:12 -05:00
powe97 c8eadc06ee
Merge branch 'main' of https://github.com/quatalog/quatalog 2024-03-05 18:34:02 -05:00
powe97 6ad6f85708
Redesign scraper to not be unbearably slow 2024-03-05 18:33:54 -05:00
powe97 acdd08168f
Update transfer.yml 2024-03-05 18:27:51 -05:00
powe97 976b553b14
Reduce wait time 2024-03-04 17:03:13 -05:00
powe97 faf303ec27
Add termination 2024-03-03 23:53:25 -05:00
powe97 0b97a431e1
Clarify steps 2024-03-03 18:56:33 -05:00
powe97 9eb5e195dc
Use dd instead of rsync 2024-03-03 17:40:25 -05:00