All purpose scraping

Playing with the Schoology API and Zoom API!

Scrapes https://finance.ucsd.edu/Home/ListFunded and maybe Instagram

• oauth-3leg: Schoology API 3-legged OAuth
• explorer: Schoology API
• irrelevant-scripts-maybe: Schoology API
• scraper: Schoology API
• archiving: Archiving Schoology
• zoom: Zoom API
• hdh: Scraping UCSD HDH's dining hall menus
• contacts: Scraping Google Contacts, based on prior work
• lib: Utils for explore/
• explore/finance: Scraping UCSD A.S. Finance
• explore/police: Scraping UCSD crime logs

Usage

The code in this repo was largely written just to explore and play around with what's available, so there's little documentation available. Sorry about that!

If it gives you any consolation, I'm also shooting my future self in the foot shrouding all this in mystery.

In general, each script has a comment at the top of the file with the command required to run the script.

• If it starts with deno, you'll need Deno 1.x installed.

• If it starts with node or tsx, you'll Node installed. Hopefully the latest version still works.

  Then, run the following command to install the dependencies.

  $ npm install

  For tsx, you'll also need to install tsx globally.

  $ npm install -g tsx

Schoology

Put your API Credentials in a api-creds.json file like this:

{
  "key": "87a6b8e78c0d897897a9f7e99a6d7c9",
  "secret": "0a08c87b75e43da26589d008f76bc"
}

You can then play with get-things-from-schoology.js directly:

$ node
Welcome to Node.js v14.4.0.
Type ".help" for more information.
> const askSgy = require('./get-things-from-schoology.js')
undefined

// These probably won't work for you unless you change 2017219 to your user ID.

askSgy('/user/2017219/updates') // GET
  .then(console.log)

askSgy('/user/2017219/updates/2230965068/comments', {
  // POST
  comment: 'comment text',
  uid: '2017219'
}).then(console.log)

If you get an error like the following, I have no idea why. Maybe Schoology is dumb, or the Node library I'm using for OAuth is dumb.

Duplicate timestamp/nonce combination, possible replay attack. Request rejected.

Explorer

node explorer/app.js

Zoom

Create an OAuth app for Zoom, then fill out the necessary information for the Install button in the Activation tab. Copy Client ID and Secret, and also the redirect URL you put, and paste them into a new file at zoom/credentials.json. See zoom/credentials.example.json for a template.

When you click on the Install button in the Activation tab, it'll send you to the specified redirect URL with a URL parameter code in the URL. Copy its value, then run

node zoom/start.js <authorization code here>

It should output Done..

oauth-3leg

Three-legged OAuth with Schoology.

In Node

# Does not work
node oauth-3leg/app.js

# Works
node oauth-3leg/app2.js

http://localhost:3000/

In Python, some venv stuff

# Activate venv
source venv/Scripts/activate

# Save
pip freeze > requirements.txt

# Load
pip install -r requirements.txt

Then run server

export FLASK_APP=oauth-3leg/hello.py
export FLASK_ENV=development
flask run

http://127.0.0.1:5000/

Scraper

Gets all the course materials from Schoology. You'll need to populate .env first:

HOST=pausd.schoology.com
UID=
CSRF_KEY=
CSRF_TOKEN=
SESS_ID=

On Schoology, you can get the first four by running the following in the console:

;`HOST=${window.location.hostname}
UID=${Drupal.settings.s_common.user.uid}
CSRF_KEY=${Drupal.settings.s_common.csrf_key}
CSRF_TOKEN=${Drupal.settings.s_common.csrf_token}
SESS_ID=`

To get SESS_ID, you need to look for the SESS cookie in the Application tab of devtools.

# In scraper/:

# Scrape Schoology -> ./private/ (Uses Deno)
deno run --allow-read=./ --allow-write=./private/ --allow-env --allow-net index.ts

# Web server to view ./private/ (Uses Node)
node app.js

Irrelevant side note

Comparing grades.json, which gets randomly jambled:

git show <commit>:grades.json > ../../private/grades-old.json
git show <commit>:grades.json > ../../private/grades-new.json
deno run --allow-read --allow-write irrelevant-scripts-maybe/sort-grades.ts private/grades-old.json
deno run --allow-read --allow-write irrelevant-scripts-maybe/sort-grades.ts private/grades-new.json
diff private/grades-old.json private/grades-new.json --color=always -c | sed -e 's/\t/ /g'