All Projects → kippnorcal → google_classroom

kippnorcal / google_classroom

Licence: GPL-3.0 license
Google Classroom Data Pipeline

Programming Languages

python
139335 projects - #7 most used programming language
Dockerfile
14818 projects
shell
77523 projects

Projects that are alternatives of or similar to google classroom

Data Science Complete Tutorial
For extensive instructor led learning
Stars: ✭ 1,027 (+5941.18%)
Mutual labels:  pipeline, pandas
Pdpipe
Easy pipelines for pandas DataFrames.
Stars: ✭ 590 (+3370.59%)
Mutual labels:  pipeline, pandas
skippa
SciKIt-learn Pipeline in PAndas
Stars: ✭ 33 (+94.12%)
Mutual labels:  pipeline, pandas
Pipeline Live
Pipeline Extension for Live Trading
Stars: ✭ 154 (+805.88%)
Mutual labels:  pipeline, pandas
mydataharbor
🇨🇳 MyDataHarbor是一个致力于解决任意数据源到任意数据源的分布式、高扩展性、高性能、事务级的数据同步中间件。帮助用户可靠、快速、稳定的对海量数据进行准实时增量同步或者定时全量同步,主要定位是为实时交易系统服务,亦可用于大数据的数据同步(ETL领域)。
Stars: ✭ 28 (+64.71%)
Mutual labels:  pipeline
degiro-trading-tracker
Simplified tracking of your investments
Stars: ✭ 16 (-5.88%)
Mutual labels:  pandas
howtheydevops
A curated collection of publicly available resources on how companies around the world practice DevOps
Stars: ✭ 318 (+1770.59%)
Mutual labels:  pipeline
flow-platform-x
Continuous Integration Platform
Stars: ✭ 21 (+23.53%)
Mutual labels:  pipeline
Python-for-data-analysis
No description or website provided.
Stars: ✭ 18 (+5.88%)
Mutual labels:  pandas
five-minute-midas
Predicting Profitable Day Trading Positions using Decision Tree Classifiers. scikit-learn | Flask | SQLite3 | pandas | MLflow | Heroku | Streamlit
Stars: ✭ 41 (+141.18%)
Mutual labels:  pandas
biojupies
Automated generation of tailored bioinformatics Jupyter Notebooks via a user interface.
Stars: ✭ 96 (+464.71%)
Mutual labels:  pipeline
gw2raidar
A log parsing website for Guild Wars 2 combat logs
Stars: ✭ 19 (+11.76%)
Mutual labels:  pandas
ga-fetcher
Fetch Google Analytics data with Google APIs in Node.js 🚠
Stars: ✭ 14 (-17.65%)
Mutual labels:  google-oauth2
jupyter-django
Using Jupyter Notebook with Django: a presentation
Stars: ✭ 42 (+147.06%)
Mutual labels:  pandas
pantab
Read/Write pandas DataFrames with Tableau Hyper Extracts
Stars: ✭ 64 (+276.47%)
Mutual labels:  pandas
classification
Catalyst.Classification
Stars: ✭ 35 (+105.88%)
Mutual labels:  pipeline
ydata-quality
Data Quality assessment with one line of code
Stars: ✭ 311 (+1729.41%)
Mutual labels:  pandas
get phylomarkers
A pipeline to select optimal markers for microbial phylogenomics and species tree estimation using coalescent and concatenation approaches
Stars: ✭ 34 (+100%)
Mutual labels:  pipeline
golang-docker-example
An example of how to run a Golang project in Docker in a Buildkite pipeline
Stars: ✭ 18 (+5.88%)
Mutual labels:  pipeline
EF-Migrations-Script-Generator-Task
No description or website provided.
Stars: ✭ 20 (+17.65%)
Mutual labels:  pipeline

googleclassroom

Dependencies:

Getting Started

Setup Environment

  1. Clone this repo
git clone https://github.com/kipp-bayarea/google_classroom.git
  1. Install dependencies
  • Docker can be installed from docker.com.
  • Docker-Compose can also be installed via the website, or via pip.
  1. Create .env file with project secrets

The environment file should fit the following template:

# Basic Configuration Info
ACCOUNT_EMAIL=Email of admin account that will be used to pull data.
STUDENT_ORG_UNIT=Name of the Google Admin organizational unit for students (optional — filters student reports to that organization)
SCHOOL_YEAR_START=YYYY-MM-DD

# Database variables
DB_TYPE=The type of database you are using. Current options: mssql, postgres, sqlite
DB_SERVER=
DB=
DB_USER=
DB_PWD=
DB_SCHEMA=

# (Optional) Data Pulls To Enable. Set to "YES" to include that pull.
# These can be left out in favor of command line arguments.
PULL_ALL=
PULL_USAGE=
PULL_COURSES=
PULL_TOPICS=
PULL_COURSEWORK=
PULL_STUDENTS=
PULL_TEACHERS=
PULL_GUARDIANS=
PULL_SUBMISSIONS=
PULL_GUARDIAN_INVITES=
PULL_ALIASES=
PULL_INVITATIONS=
PULL_ANNOUNCEMENTS=
PULL_MEET=

# (Optional) Syncing from a file. Set to "YES" to sync files (see instructions below).
SYNC=

# (Optional) Debug parameters. Set to "YES" to include debug logs or files.
DEBUG=
DEBUGFILE=

# (Optional) Batch parameters. Can be configured and changed to optimize performance.
# *_BATCH_SIZE is the number of dates or courses to batch at a time. MAX: 1000
# Lower batch sizes are useful for high volume or slow endpoints to avoid timeouts.
ORG_UNIT_BATCH_SIZE=
USAGE_BATCH_SIZE=
COURSES_BATCH_SIZE=
TOPICS_BATCH_SIZE=
COURSEWORK_BATCH_SIZE=
STUDENTS_BATCH_SIZE=
TEACHERS_BATCH_SIZE=
GUARDIANS_BATCH_SIZE=
SUBMISSIONS_BATCH_SIZE=
GUARDIAN_INVITES_BATCH_SIZE=
ALIASES_BATCH_SIZE=
INVITATIONS_BATCH_SIZE=
ANNOUNCEMENTS_BATCH_SIZE=
MEET_BATCH_SIZE=
PAGE_SIZE=The number of items to page at once.

# Email notification variables
# Set DISABLE_MAILER to "YES" if you do not want email notifications to be sent.
DISABLE_MAILER=
SENDER_EMAIL=
SENDER_PWD=
RECIPIENT_EMAIL=
# If using a standard Gmail account you can set these to smtp.gmail.com on port 465
EMAIL_SERVER=
EMAIL_PORT=
  1. Enable APIs in Developer Console
  • Navigate to the API library in the developer console.
  • Search for Google Classroom, and Enable it.
  • Search for Admin SDK, and Enable it.
  1. Create a service account.
  • In the Google Developer Console (console.developers.google.com), go to Credentials.
  • Click on "Create Credentials -> Service Account"
  • Create a name for your service account.
  • Select the "Owner" role for the service account.
  • Create a key, saving the result file as service.json in the /google_classroom directory.
  • Check the box for "Enable G Suite Domain-Wide Delegation"
  • Click "Done".
  1. Add scopes for the service account.
  • In the Google Admin Console (admin.google.com), go to Security.
  • Click on "Advanced Settings -> Manage API client access"
  • For the client name, use the Unique ID of the service account.
  • In the API Scopes, add the following scopes and click "Authorize".
https://www.googleapis.com/auth/admin.directory.orgunit,
https://www.googleapis.com/auth/admin.reports.usage.readonly,
https://www.googleapis.com/auth/classroom.announcements,
https://www.googleapis.com/auth/classroom.courses,
https://www.googleapis.com/auth/classroom.coursework.students,
https://www.googleapis.com/auth/classroom.guardianlinks.students,
https://www.googleapis.com/auth/classroom.profile.emails,
https://www.googleapis.com/auth/classroom.rosters,
https://www.googleapis.com/auth/classroom.student-submissions.students.readonly,
https://www.googleapis.com/auth/classroom.topics,
https://www.googleapis.com/auth/admin.reports.audit.readonly

Syncing data to Google Classroom

NOTE: This functionality is currently in development and does not yet work.

In addition to pulling data from Google Classroom, this script can write data back. The sync logic takes the provided file as "source of truth" and matches that to Google Classroom. To enable this logic:

  1. Add the file courses.csv to the folder /google_classroom/sync_files matching the format of courses_sample.csv.

  2. Set SYNC=YES in your .env file or pass the --sync arg via the command line.

Running the job

Locally

Install the dependencies.

pipenv install --skip-lock

Run the job.

pipenv run python main.py

Using Docker Compose:

docker-compose up --build

You can also use the command line instead of environment variables to specify which endpoints to run. For a list of all command line arguments, run docker-compose run app --help.

docker-compose build
docker-compose run app --teachers --students --guardians

Running tests

./run_tests

Yearly maintenance

  1. Confirm the org unit ID (used to get Student Usage) in the .env.
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].