powe97
|
6a1395c054
|
Fix workflow not being able to commit and remove tee
|
2024-03-06 02:05:49 -06:00 |
|
powe97
|
912b07f6f3
|
Add retrying first page
|
2024-03-06 01:18:49 -06:00 |
|
powe97
|
8b15438a98
|
Actually use the retry version of the function...
|
2024-03-06 01:03:23 -06:00 |
|
powe97
|
0007bde18a
|
Recombine JSONs
|
2024-03-06 00:43:26 -06:00 |
|
powe97
|
c98b928125
|
Add retrying
|
2024-03-05 22:54:42 -05:00 |
|
powe97
|
a0b9081f8f
|
--headless
|
2024-03-05 21:14:32 -05:00 |
|
powe97
|
4f69c1d8a0
|
Re-get the page to try circumvent timeout
|
2024-03-05 21:14:00 -05:00 |
|
powe97
|
56c9268398
|
Disable fail-fast
|
2024-03-05 20:49:08 -05:00 |
|
powe97
|
02b383b90b
|
Extend timeout
|
2024-03-05 20:47:41 -05:00 |
|
powe97
|
95e8238786
|
Merge branch 'main' of https://github.com/quatalog/quatalog
|
2024-03-05 19:10:16 -05:00 |
|
powe97
|
fc72fda5de
|
Remove jump debug print
|
2024-03-05 19:10:10 -05:00 |
|
powe97
|
e45318404d
|
Update transfer.yml
|
2024-03-05 19:06:49 -05:00 |
|
powe97
|
10715c89e3
|
Update transfer.yml
|
2024-03-05 19:05:41 -05:00 |
|
powe97
|
52fdab6ce6
|
Make everything stderr print
|
2024-03-05 19:03:54 -05:00 |
|
powe97
|
42dbf3c19a
|
Update transfer.yml
|
2024-03-05 18:46:02 -05:00 |
|
powe97
|
985f40c4e7
|
Set up matrix jobs
|
2024-03-05 18:42:05 -05:00 |
|
powe97
|
cb24d84b46
|
Merge branch 'main' of https://github.com/quatalog/quatalog
|
2024-03-05 18:38:17 -05:00 |
|
powe97
|
ce2f22b23b
|
Merge branch 'main' of https://github.com/quatalog/quatalog
|
2024-03-05 18:38:12 -05:00 |
|
powe97
|
c8eadc06ee
|
Merge branch 'main' of https://github.com/quatalog/quatalog
|
2024-03-05 18:34:02 -05:00 |
|
powe97
|
6ad6f85708
|
Redesign scraper to not be unbearably slow
|
2024-03-05 18:33:54 -05:00 |
|
powe97
|
acdd08168f
|
Update transfer.yml
|
2024-03-05 18:27:51 -05:00 |
|
powe97
|
976b553b14
|
Reduce wait time
|
2024-03-04 17:03:13 -05:00 |
|
powe97
|
faf303ec27
|
Add termination
|
2024-03-03 23:53:25 -05:00 |
|
powe97
|
0b97a431e1
|
Clarify steps
|
2024-03-03 18:56:33 -05:00 |
|
powe97
|
9eb5e195dc
|
Use dd instead of rsync
|
2024-03-03 17:40:25 -05:00 |
|
powe97
|
57d9f5dc4c
|
See previous commit
|
2024-03-02 19:08:06 -05:00 |
|
powe97
|
58e509d1bb
|
Clean up and potentially fix rsync
|
2024-03-02 18:35:44 -05:00 |
|
powe97
|
c3e0d558d7
|
List data folder before copying (rsync seems to not work every time?)
|
2024-03-02 16:17:11 -05:00 |
|
powe97
|
ae286917c1
|
Formatting
|
2024-03-02 02:26:58 -05:00 |
|
powe97
|
80f9ed1d95
|
Merge branch 'main' of https://github.com/quatalog/quatalog
|
2024-03-02 02:25:26 -05:00 |
|
powe97
|
bc07c559bc
|
Uh-oh
|
2024-03-02 02:22:30 -05:00 |
|
powe97
|
d5f58d8576
|
Create json dir if it doesn't already exist
|
2024-03-02 01:44:28 -05:00 |
|
powe97
|
c508798f20
|
Create courses dir if it doesn't already exist
|
2024-03-02 01:41:43 -05:00 |
|
powe97
|
30f4f49cdb
|
Fix bug where only 1 page is scraped per school and refactor
|
2024-03-01 20:32:00 -05:00 |
|
powe97
|
baa74b8ee6
|
Fix issue where only 1 page per school would get scraped properly
|
2024-03-01 18:17:53 -05:00 |
|
powe97
|
5ea6816c90
|
Fix capitalization next to smart apostrophes (really?)
|
2024-03-01 17:21:45 -05:00 |
|
powe97
|
6b5356c84f
|
Fix typo leading to bad capitalization
|
2024-03-01 15:01:20 -05:00 |
|
powe97
|
001825d3dc
|
Merge branch 'main' of https://github.com/quatalog/quatalog
|
2024-03-01 13:32:09 -05:00 |
|
powe97
|
3b608fad41
|
Fix Roman numerals issue
|
2024-03-01 13:32:02 -05:00 |
|
powe97
|
5fe4ee9f13
|
Change manually run workflow to have timeout of 2 mins
|
2024-03-01 13:12:23 -05:00 |
|
powe97
|
997d3c16a8
|
2hr timeout → 45m
|
2024-03-01 12:48:20 -05:00 |
|
powe97
|
d03be03aeb
|
Move debug print to be more accurate
|
2024-03-01 01:50:01 -05:00 |
|
powe97
|
019b777228
|
Make transfer scraper run continuously (at least as much as Github allows)
|
2024-03-01 01:45:06 -05:00 |
|
powe97
|
1a4542e20e
|
Fix crashing without timeout arg and re-add --headless
|
2024-03-01 00:29:34 -05:00 |
|
powe97
|
b0acd0e745
|
Dammit python
|
2024-02-29 22:31:09 -05:00 |
|
powe97
|
53891400ea
|
Every 15 minutes
|
2024-02-29 22:29:51 -05:00 |
|
powe97
|
c6e28d399a
|
Make timeout field have default value
|
2024-02-29 22:28:00 -05:00 |
|
powe97
|
682b1679b4
|
Run every 15 mins
|
2024-02-29 22:25:22 -05:00 |
|
powe97
|
aa4af079f8
|
Merge branch 'main' of https://github.com/quatalog/quatalog
|
2024-02-29 22:13:54 -05:00 |
|
powe97
|
cf2abf7193
|
Fix partial updates when KeyboardInterrupt happens mid-institution
|
2024-02-29 22:13:44 -05:00 |