phantop/quatalog-scraper
1
0
Fork 0
mirror of https://github.com/quatalog/quatalog.git synced 2024-11-16 03:52:55 +00:00
Code Issues Packages Projects Releases Wiki Activity
314 commits 1 branch 0 tags 6.4 MiB
Commit graph

314 commits

This branch
This branch
All branches
Author SHA1 Message Date
powe97 3b16ecc9bb
Update transfer.yml 2024-04-08 14:23:51 -04:00
powe97 0109006d12
Nevermind 2024-03-17 20:07:48 -04:00
powe97 cae1b09981
Add zero padding 2024-03-17 20:05:07 -04:00
powe97 6de711b23e
Update transfer.yml 2024-03-17 10:38:39 -04:00
powe97 99ba8873bd
Run twice a day 2024-03-16 21:25:42 -04:00
powe97 eb77e84535
Fix artifact upload 2024-03-16 21:23:23 -04:00
powe97 bdc6b2bcbc
Short circuit on IP ban 2024-03-16 21:16:14 -04:00
powe97 ecfb176c46
Create scrape_page.yml 2024-03-16 20:46:25 -04:00
powe97 ea09d33ac2
Convert to use reusable workflow 2024-03-16 20:45:57 -04:00
powe97 f7c5c2461a
Update transfer.yml 2024-03-16 13:07:22 -04:00
powe97 130dca8b89
Oops 2024-03-16 12:55:53 -04:00
powe97 1403b42a5c
Oops 2024-03-16 12:43:11 -04:00
powe97 463a2aac03
Create generate-csvs.yml 2024-03-16 12:40:14 -04:00
powe97 d0f2f969d2
Run CSV generator 2024-03-16 12:19:01 -04:00
powe97 5e9e464ad0
Add CSV generator 2024-03-16 02:17:31 -04:00
powe97 de7360f1c7
Convert JSON to by-course format 2024-03-16 01:28:33 -04:00
powe97 ba3e5c77d1
Lint 2024-03-16 01:17:49 -04:00
powe97 c3be28e520
Add JSON converter from by-institution to by-course 2024-03-16 01:14:58 -04:00
powe97 061f9b14e5
Sort keys 2024-03-16 01:07:16 -04:00
powe97 1fa7ab61af
Rename transfer scraper 2024-03-16 00:32:59 -04:00
Quatalog Compiler bd930f05e9 Recompile scraper : Sat Mar 16 03:47:31 UTC 2024 2024-03-16 03:47:31 +00:00
powe97 77423ec069
Show music lessons properly 2024-03-15 23:46:28 -04:00
powe97 517952f977
Re-add catalog name scraping 2024-03-13 23:52:51 -04:00
powe97 f1a47dca48
stderr 2024-03-13 22:19:56 -04:00
powe97 af25410c5d
Log IP 2024-03-13 22:18:13 -04:00
powe97 b017436be9
Fix handling for courses that come in as multiple courses 2024-03-13 15:48:53 -04:00
powe97 4c0517f6c4
Print title 2024-03-07 16:14:54 -06:00
powe97 779b979b9b
Fix credit count parsing 2024-03-07 11:03:59 -06:00
powe97 0f3652d8cc
Lint 2024-03-06 16:16:36 -06:00
powe97 7c87221256
Fix typo 2024-03-06 16:16:23 -06:00
powe97 69d8946f37
Improve credit count parsing 2024-03-06 16:13:13 -06:00
powe97 9c374bf130
Fix timeout issue (again) 2024-03-06 13:20:09 -06:00
powe97 10360ff57c
Wait for table 2024-03-06 12:48:48 -06:00
powe97 3b239f9cef
Merge branch 'main' of https://github.com/quatalog/quatalog 2024-03-06 12:41:13 -06:00
powe97 de89a56808
Add more debug printing 2024-03-06 12:41:08 -06:00
powe97 45cfba68ac
Sort output JSON 2024-03-06 12:10:23 -06:00
powe97 aec272e28e
Run once a day 2024-03-06 02:43:41 -06:00
powe97 c21dec05ad
Re-enable fail-fast 2024-03-06 02:36:33 -06:00
powe97 81ba2fdc80
Make failing actually fail the program 2024-03-06 02:35:18 -06:00
powe97 92c3327b1a
Fix debug prints 2024-03-06 02:09:42 -06:00
powe97 6a1395c054
Fix workflow not being able to commit and remove tee 2024-03-06 02:05:49 -06:00
powe97 912b07f6f3
Add retrying first page 2024-03-06 01:18:49 -06:00
powe97 8b15438a98
Actually use the retry version of the function... 2024-03-06 01:03:23 -06:00
powe97 0007bde18a
Recombine JSONs 2024-03-06 00:43:26 -06:00
powe97 c98b928125
Add retrying 2024-03-05 22:54:42 -05:00
powe97 a0b9081f8f
--headless 2024-03-05 21:14:32 -05:00
powe97 4f69c1d8a0
Re-get the page to try circumvent timeout 2024-03-05 21:14:00 -05:00
powe97 56c9268398
Disable fail-fast 2024-03-05 20:49:08 -05:00
powe97 02b383b90b
Extend timeout 2024-03-05 20:47:41 -05:00
powe97 95e8238786
Merge branch 'main' of https://github.com/quatalog/quatalog 2024-03-05 19:10:16 -05:00