Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Setup Script #31

Merged
merged 6 commits into from
Jan 30, 2024
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,10 @@ about them, just click on the script name.

This program _requires_ the `webregautoin` helper program.

## Documentation
Basic documentation is provided in the [wiki](https://github.com/ewang2002/webreg_scraper/wiki).
## Setup
To run this project, feel free to explore the individual scripts or crates above; setup guides for each are provided.

If you want to get an Ubuntu environment ready with all the necessary files needed to run this project, you can run the setup script in the [`setup`](https://github.com/ewang2002/webreg_scraper/tree/master/setup) folder. More information will be provided there.

## License
Everything in this repository is licensed under the MIT license.
15 changes: 15 additions & 0 deletions setup/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Setup Environment
If you have a new _Ubuntu_ environment, you can use the `setup.sh` script to install all necessary files and dependencies needed to run this project. Other Linux distributions have not been tested.
ewang2002 marked this conversation as resolved.
Show resolved Hide resolved

I recommend creating a [DigitalOcean droplet](https://www.digitalocean.com/); the cheapest plan will suffice, and students with the GitHub student pack are eligible for [$200 in DigitalOcean credits for 1 year](https://education.github.com/pack/offers).

To start, copy both `setup.sh` and `nginx.conf` to the directory where you want all necessary project files to be stored at (e.g., your home directory, `~`). Then, run `sudo setup.sh`. This script will
ewang2002 marked this conversation as resolved.
Show resolved Hide resolved
- Set the timezone to Pacific Time
- Install all dependencies needed for puppeteer to work
- Install `nvm` and the LTS version of `node.js`
ewang2002 marked this conversation as resolved.
Show resolved Hide resolved
- Install `pm2` and `typescript` globally
- Download the latest version of the WebReg scraper with the `authmanager` executable
- Clone the repository, and extract and compile the `notifier` and `webregautoin` scripts.
- Install `nginx` and replace the default configuration file with the one provided here.

In other words, this setup script will get your environment ready to run the scraper and the login script, and make the scraper's API available to anyone through `nginx`. You do not need to install anything else to make this script work.
28 changes: 28 additions & 0 deletions setup/nginx.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
user www-data;
worker_processes auto;
pid /run/nginx.pid;
error_log /var/log/nginx/error.log;
include /etc/nginx/modules-enabled/*.conf;

events {
worker_connections 768;
}

http {
sendfile on;
tcp_nopush on;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
access_log /var/log/nginx/access.log;
gzip on;
server {
listen 80;

location /ucsd/ {
proxy_pass http://localhost:3000/;
}
}
}
68 changes: 68 additions & 0 deletions setup/setup.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
#!/bin/bash

if [ "$(id -u)" -ne 0 ]; then
echo "Please run w/ sudo." >&2;
exit 1;
fi

apt-get update
ewang2002 marked this conversation as resolved.
Show resolved Hide resolved

# Set the timezone to Pacific time
timedatectl set-timezone America/Los_Angeles

# Install puppeteer dependencies
# See https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md#running-puppeteer-on-wsl-windows-subsystem-for-linux
apt install -y libgtk-3-dev libnotify-dev libnss3 libxss1 libasound2

# Install jq for JSON parsing (we need this when getting the latest release tag for the scraper)
apt install -y jq

# Install nvm and the LTS version of node.js
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.5/install.sh
source ~/.bash_profile
nvm install --lts
source ~/.bash_profile

# Install pm2 and typescript
npm install -g pm2
npm install -g typescript

# Get latest scraper tag
tag=$(curl --silent "https://api.github.com/repos/ewang2002/webreg_scraper/releases" | jq -r "first | .tag_name")

# Setup webreg_scraper
mkdir scraper
cd scraper
wget https://github.com/ewang2002/webreg_scraper/releases/download/$tag/authmanager-x86_64-unknown-linux-gnu.tar.gz
tar -xvzf authmanager-x86_64-unknown-linux-gnu.tar.gz
wget https://github.com/ewang2002/webreg_scraper/releases/download/$tag/webreg-x86_64-unknown-linux-gnu-auth.tar.gz
tar -xvzf webreg-x86_64-unknown-linux-gnu-auth.tar.gz
rm *.tar.gz
cd ..

# Setup the login & notifier scripts
git clone https://github.com/ewang2002/webreg_scraper
mv webreg_scraper/scripts/notifierbot notifier
mv webreg_scraper/scripts/webregautoin login
rm -rf webreg_scraper

# Setup the notifier
cd notifier
npm i
npm run compile
cd ..

# Setup the login script
cd login
npm i
npm run compile
cd ..

# Install nginx
sudo apt install -y nginx
rm -f /etc/nginx/nginx.conf
mv nginx.conf /etc/nginx

# All done.
echo "All done. Make sure to configure the notifier bot, login script, and the scraper."
exit 0
Loading