phantop
/
quatalog-scraper
mirror of https://github.com/quatalog/quatalog.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Compare commits

base: phantop:30f4f49cdb3ddfdeba9cda56a1a25f462aca85f5
Branches Tags
phantop:main
...
compare: phantop:ae286917c163fdaebd504379df0166ffd218d5dc
Branches Tags
phantop:main

5 Commits
30f4f49cdb ... ae286917c1

Author SHA1 Message Date
powe97 ae286917c1
Formatting 2024-03-02 02:26:58 -05:00
powe97 80f9ed1d95
Merge branch 'main' of https://github.com/quatalog/quatalog 2024-03-02 02:25:26 -05:00
powe97 bc07c559bc
Uh-oh 2024-03-02 02:22:30 -05:00
powe97 d5f58d8576
Create json dir if it doesn't already exist 2024-03-02 01:44:28 -05:00
powe97 c508798f20
Create courses dir if it doesn't already exist 2024-03-02 01:41:43 -05:00
2 changed files with 3 additions and 0 deletions
Show Stats Expand all files Collapse all files

1
.github/workflows/scraper.yml vendored
View File

@ -110,6 +110,7 @@ jobs:
- name: Merge data
run: |
mkdir -p quatalog-site/courses quatalog-site/json
rsync -avz courses/ quatalog-site/courses/
rsync -avz searchable_catalog.json quatalog-site/json/searchable_catalog.json
rsync -avz courses_list.json quatalog-site/json/courses_list.json

2
transfer_scraper/main.py
View File

@ -169,6 +169,8 @@ def main():
"https://tes.collegesource.com/publicview/TES_publicview01.aspx?rid=f080a477-bff8-46df-a5b2-25e9affdd4ed&aid=27b576bb-cd07-4e57-84d0-37475fde70ce"
)
print(f'Title is {driver.find_element(By.TAG_NAME, "title").get_attribute("innerText").strip()}', file=sys.stderr)
num_pages = int(
driver.find_element("id", "lblInstWithEQPaginationInfo").text.split()[-1]
)