Data is nutrients for the soul

Provides simple datasets for data visualization, statistical analysis and modelling
Suitable for those starting out in data science and of course all who find the datasets useful
Data visualizations can be found here
Tutorials/ exercises can be found here

List of datasets along with descriptions

Dataset: akcdogs.csv
Description: Cleaned data on dog breeds scraped from akc.org (as at 17 Jan 2018)
Variables: Breed , Trait1, Trait2, Trait3, Energy level, Size, Rank, Good with Children, Good with other Dogs, Shedding, Grooming, Trainability, Height, Weight, Life expectancy, Barking level, Group
Mode of data collection: Web scraping
Source: American Kennel Club

Dataset: arrivals2018.csv
Description: Top 20 cities based on 2017 arrivals and 2018 estimates
Variables: rank , city, country, arrivals_2017 (actual arrival count for 2017), arrivals_2018 (estimated arrival count for 2018)
Mode of data collection: Web scraping
Source: Most visited: World's top cities for tourism

Dataset: bookdepo.csv
Description: Raw data on bestsellers scraped from bookdepository.com (as at 11 Jan 2018)
Variables: (blank) (row index number) , name (book title), material (book material), author (author), rank (bestsellers rank), maincat (main category), subcat (sub category), rating (rating by readers), ratingcount (number of readers who gave ratings), saleprice (discounted price in S$), listprice (original price in S$), numofpages (number of pages), datepub (date published), isbn13 (ISBN13 number)
Mode of data collection: Web scraping
Source: Book Depository

Dataset: bookdepobest.csv
Description: Cleaned data on bestsellers scraped from bookdepository.com (as at 11 Jan 2018)
Variables: SN, name, rank, maincat, subcat, rating, saleprice, listprice, datepub, isbn13, GoodreadsRateCount, BookMaterial, Author(s), PageCount
Mode of data collection: Web scraping
Source: Book Depository

Dataset: Class1.csv
Description: Hypothetical dataset consisting score results of 100 students for three tests
Variables: id, gender, test1, test2, test3
Mode of data collection: N.A.
Source: N.A.

Dataset: Class2.csv
Description: Hypothetical dataset consisting score results of 100 students for four tests
Variables: id, gender, test1, test2, test3, test4
Mode of data collection: N.A.
Source: N.A.

Dataset: covid19_sg.csv
Description: Covid-19 case time series data in Singapore (more details here)// Data is contributed to https://github.com/neherlab/covid19_scenarios
Variables: Date, Daily_Confirmed_, False_Positives_Found, Cumulative_Confirmed, Daily_Discharged, Passed_but_not_due_to_COVID, Cumulative_Discharged, Discharged_to_Isolation, Still_Hospitalised, Daily_Deaths, Cumulative_Deaths, Tested_positive_demise, Daily_Imported, Daily_Local_transmission, Local_cases_residing_in_dorms_MOH_report, Local_cases_not_residing_in_doms_MOH_report, Intensive_Care_Unit_(ICU), General_Wards_MOH_report, In_Isolation_MOH_report, Total_Completed_Isolation_MOH_report, Total_Hospital_Discharged_MOH_report
Mode of data collection: Manual
Source: Ministry of Health

Dataset: DisneySongs25.csv
Description: Top 25 Disney Songs (as at 4 Apr 2016)
Variables: Rank , Song Title, Movie, Year, Lyrics
Mode of data collection: Manual
Source: Top 25 Disney Songs - IGN, Metrolyrics, Disneyclips.com

Dataset: emojis.csv
Description: names and descriptions of emojis
Variables: id (number to identify unique emoji in dataset), index (running number in dataset), name (name of emoji), desc (description/ alternative names)
Mode of data collection: Web scraping
Source: Emoji Cheat Sheet

Dataset: FreqWordsObama.csv
Description: Frequently mentioned words in Barack Obama's tweets between 2007 and 2017 (as at 12 Dec 2017)
Variables: Year (year of tweet), Word (frequently mentioned word), Count (number of tweets containing word), Year Volume (volume of tweet in the year), Percentage (percentage of tweets containing word)
Mode of data collection: Twitter web scraping and text mining
Source: Barack Obama's Twitter account

Dataset: GovSG.csv
Description: Addresses with GIS location and contact information of Ministries and Statutory Boards in Singapore
Variables: Organisation, Type (Ministry/ Statutory Board), Zipcode, Latitude, Longitude, Website, Tel, Fax, Email, Enquiry/ Feedback Form (url), Parent Ministry (Statutory Boards under respective Ministries)
Mode of data collection: Web scraping, Manual, Tableau-generated latitude/longitude based on Zipcode
Source: Singapore Government Directory, The Public Service | [email protected]

Dataset: gov-sg-terms-translations.tsv
Descriptions: Official English-Mandarin translations of Singapore Government Terms.
Variables: english, mandarin
Mode of data collection: Webscraping
Source: Government Terms Translated

Dataset: mrtfaretime.csv
Description: Travel time and fare information between train (MRT/LRT) stations in Singapore (as at Oct 2018)
Variables: Station_start (Boarding station), Station_end (Alighting station), Time (Travel time in mins), Adult (Adult fare), Senior (Fare for Seniors and Persons with Disabilities), Standard (Fare for Standard Ticket), Student (Student fare), WTCS (Fare under Workfare Transport Concession Scheme), REF_STNSTART, Latitude_Start, Longitude_Start, REF_STNEND, Latitude_End, Longitude_End
Mode of data collection: Web scraping
Source: TransitLink Electronic Guide

Dataset: mrtsg.csv
Description: Latitude and longitude of train (MRT/LRT) stations in Singapore (as at Dec 2020)
Variables: OBJECTID (id) , STN_NAME (station name), STN_NO (station number), X (X coord in SVY21 format), Y (Y coord in SVY21 format), Latitude, Longitude, COLOR (color of train line)
Mode of data collection: Public dataset, Coordinate conversion web scraping
Source: LTA DataMall, OneMap Singapore

Dataset: passport.csv
Description: Top 10 Passports (in the 2017 Global Passport Power Rank) and their visa requirements to other countries
Variables: Top 10 Country (name of country with passport in Top 10), Country (visiting country), Type of Visa (visa required)
Mode of data collection: Manual
Source: Passport Index 2017

Dataset: pokemon.csv
Description: Pokemon and their attack and defense statistics
Variables: HP, Attack, Defense, Sp Atk, Sp Def, Speed, Total, HP Percentile, Attack Percentile, Defense Percentile, Sp Atk Percentile, Sp Def Percentile, Speed Percentile, Total Percentile
Mode of data collection: Manual, Excel function for percentile ranking
Source: Pokemon database

Dataset: populationsg.csv
Description: Count of Residents By Age Group & Type Of Dwelling, Annual from 2000 to 2019
Variables: agegroup, dwelling, value, year
Mode of data collection: API
Source: Singapore Department of Statistics

Dataset: primaryschoolsg.csv
Description: Locations of Primary schools in Singapore and their popularity
Variables: Name, Type, GenderMix, Area, Zone, PostalCode, Latitude, Longitude, PlacestakenuptillPhase2B
Mode of data collection: Public data, Web scraping, Tableau-generated latitude/longitude based on Postal code
Source: Wikipedia, School Information Service - MOE, Salary.sg

Dataset: secsch_cleaned.csv
Description: Locations of Secondary schools in Singapore and their 2017 PSLE cut-off scores
Variables: row (row number), SCHNAME (name of school in upper case), zipcode, area, zone, type, latitude, longitude, School (name of school for PSLE cut-off scores as matching key), Rank2017, IB, IP, SAP, Girls, Boys, Co-ed, O-level track (PSLE cut-off score for O-level track), PSLE2017 Cut Off, Gender Mix
Mode of data collection: Web scraping, Tableau-generated latitude/longitude based on zipcode, Manual
Source: School Information Service - MOE, Salary.sg

Dataset: seedly_sal.csv
Description: Annual base salaries ($'000) for different roles in various industries in Singapore (as at Jan 2020)
Variables: Industry, Role, Minimum, Median, Maximum
Mode of data collection: Web scraping
Source: "The Ultimate Salary Guide For Singaporeans" on Seedly

Dataset: tfresults02.csv
Description: Results of 100m and 200m national track-and-field finals for "A" division boys and girls between 2002 and 2016
Variables: Year, Event, Division, Gender, School, Name, Position, Timing (in s)
Mode of data collection: Manual
Source: Singapore Athletics LIVE Results

Dataset: unesco_sites.csv
Description: UNESCO World Heritage List (as at 30 Oct 2019)
Variables: country, site, type (category of site)
Mode of data collection: Webscraping
Source: UNESCO World Heritage Centre - World Heritage List, more data can be downloaded here.

Dataset: usunirankings.csv
Description: Average U.S. News Rankings for 123 Universities: 2013-2020
Variables: school, rank_2013, rank_2014, rank_2015, rank_2016, rank_2017, rank_2018, rank_2019, rank_2020, avg_rank, chg (change between 2013 and 2020)
Mode of data collection: Webscraping
Source: Public University Honors

Dataset: wastestats.csv
Description: 15 years of data from 2003 to 2017 on waste management and recycling statistics (Waste type is inconsistently named across years. eg. C&D should refer to construction debris, and some categories are capitalized in some years but not others)
Variables: waste_type, waste_disposed_of_tonne, total_waste_recycled_tonne, total_waste_general_tonne, recycling_rate, year
Mode of data collection: Manual (Consolidated data in CSV format from PDF and website)
Source: National Environment Agency

List of notebooks along with descriptions

Notebook: AKCDogs.ipynb
Description: Python code for scraping akc.org dog information

Notebook: bookdepository.ipynb
Description: Python code for scraping bookdepository.com bestsellers information

Notebook: emojis.ipynb
Description: Python code for scraping Emoji Cheat Sheet information

Notebook: imdb.ipynb
Description: Python code for scraping imdb.com most popular movies information

Notebook: covid_spread.ipynb
Description: Python code for understanding the pace of increase in number of COVID-19 cases for each country using data provided by the Johns Hopkins University Center for Systems Science and Engineering (JHU CSSE)

Notebook: Creating Datasets in Python.ipynb
Description: Python code for importing practice datasets from R to Python and creating hypothetical datasets

Notebook: Creating Datasets in R.ipynb
Description: R code for creating hypothetical datasets

Notebook: Data Cleaning in Python.ipynb
Description: Python code for performing various data cleaning tasks

Notebook: Data Cleaning in R.ipynb
Description: R code for performing various data cleaning tasks

Notebook: engineeringjobs.ipynb
Description: Python code for scraping information relating to engineering job openings in Singapore on a government portal as at 31 Jul 2020 using selenium and BeautifulSoup

Notebook: Gov.sg Translations.ipynb
Description: Python code for scraping Official English-Mandarin translations of Singapore Government Terms from Government Terms Translated

Notebook: Json.ipynb
Description: Python code for working with Json

Notebook: linecharts.ipynb
Description: Python code for plotting line charts using matplotlib and seaborn involving major/ minor ticks, markers and data labels for the last point

Notebook: lta_tweets.ipynb
Description: Python code for working with text and datetime data from LTATrafficNews tweets (between 12-Dec-2019 and 29-Dec-2019)

Notebook: mohcovid.ipynb
Description: Python code for scraping updates on COVID-19 local situation (Singapore) on Ministry of Health's website since January and an interactive visualization created using Plotly (download/run the notebook locally to view it) with some static charts created using Seaborn

Notebook: Monte_Carlo_COVID.ipynb
Description: Python code for Monte Carlo simulations on number of COVID-19 cases in Singapore using 20 days of historical data

Notebook: pytesseract_onlineimage.ipynb
Description: Python code for extracting text data from an image hosted online

Notebook: QOTD.ipynb
Description: Python code for scraping quotes off "Inhale, Exhale and Repeat After Me! 150 Best Quotes About Life" by Parade.com

Notebook: seleniummrt.ipynb
Description: Python code for scraping time and fare information between train stations in Singapore from TransitLink Electronic Guide

Notebook: SingStat API.ipynb
Description: Python code for calling data from SingStat Table Builder/ Singapore Department of Statistics using the API function

Notebook: Statistical tests.ipynb
Description: R code for performing various types of two-sample tests and correlation checks

Notebook: StatutoryBoardSG.ipynb
Description: Python code for scraping addresses/ contact information of statutory boards in Singapore from Singapore Government Directory and automating download of organisation logo images

Notebook: Text Frequency Analysis.ipynb
Description: R code for getting frequency distribution of words in a chunk of text

Notebook: textgen-covid_resilience.ipynb
Description: Python code for text-generating neural network trained using text from Singapore's Budget 2020 - Resilience Budget/ Supplementary Budget Statement with textgenrnn

Notebook: unesco.ipynb
Description: Python code for scraping UNESCO World Heritage countries and sites from UNESCO World Heritage Centre - World Heritage List

Notebook: WiDS.ipynb
Description: R code for predicting gender of survey respondents as part of the WiDS 2018 Datathlon

Notebook: youtube_data_analysis-need_key.ipynb
Description: Python code for getting YouTube data for analysis based on a list of search terms using YouTube Data API v3