Deep Learning for Computer Vision related project for nostalgic Poké weebs
We recreated a Pokédex clone, which recognizes pictures of Pokémon from the first generation, using a Convolutional Neural Network. It's built on Raspberry Pi4 with LCD display, PiCamera, speaker and some other components attached.
The case is made of recycled cardboard.
Italiano · Docs · View Report · Presentation · Request Feature|Report Bug
Table of Contents
Demo
See our demo on YouTube!
Demo video
Features
- Main menu with functionalities and
Aboutpanel. - Pokémon-image acquisition from app.
- Pokémon prediction of captured pic, with an excellent accuracy, using a Convolutional Neural Network.
- Different objects recognition, such as cards, plushies, figures.
- Pokémon info (name, id, types, description and stats) and the previous and following evolution stages of Pokémon recognized.
- Audio description of the predicted Pokémon.
- Play Pokémon cry.
Settingsview where users can edit parameters, such as language (English and Italian), fullscreen, volume.- Debug mode to check the values of the prediction.
- Easter egg, try and find it!
Usage
To use the application follow these steps: TO-DO
Prerequisites
TO-DO
Installation
TO-DO
Tools
Resources
- Pokémon data:
- Camera calibration with OpenCV
- Tensorflow-lite inference
- Tensorflow-lite conversion
- Tensorflow-lite post-training quantization
Roadmap
- Dataset
- find a decent dataset for the neural network
- fix (cut pictures) and extend it
- Classifier
- CNN with 3 conv layers and 2 FC layers
- data augmentation (random flip, rotation, contrast,
brightness) - try dropout
- try batch norm
- loss and accuracy graphs
- test real life pics
- improve old CNN
- Application
- Pokémon repository
- find .json file and load it into a dictionary
- check and fix it
- create Pokémon class
- video input
- create a separate class
- make a function that can take a frame from the picamera (to test)
- display the image inside a canvas
- GUI structure
- create a main menu
- create an about panel
- create a main app view divided in 2 (left-side for video input, right-side for Pokémon details)
- create a settings panel
- button to get the current frame
- labels and entry for the Pokémon details (stats with dynamic bars and different colors)
- add buttons to scroll between multiple evolutions (example: Eevee has different evolutions)
- change the "type(s)" entry (from text to image)
- button to play cry
- collect cry audio files
- description voice reading
- collect description audio files with a bot
- different language update
- debug mode
- Pokémon repository
- Raspberry Setup
- buy components
- LDC display
- PiCamera
- power supply (powerbank)
- speaker
- push buttons
- type-C elbow adapter
- A/D converter (ADS1115)
- integrate components
- LCD display
- PiCamera
- power supply
- speaker
- buttons
- analog joystick
- prepare OS (disable password, enable interfaces, ...)
- buy components
- App Deployment
- prepare environment (install python3 and required packages)
- clone the repo
- test the app
- Case Prototype
- project
- cardboard cutout
- painting
- Report
- setup a LaTeX document
- draft a possible subdivision into chapters
- write the report
- Presentation
- Demo Video
- Extra & Future Developments
- use a more comples neural network with the new dataset
- use new form of data augmentation
- add an amplifier to speaker
- insert one or more white LEDs near the camera lens
- add settings option to enable/disable the video freezing after taking a picture
- finish the 3D model and print it
- extend the Pokédex to the following Pokémon generations
- porting of the application to mobile systems (Android, iOS)
License
Distributed under the GPLv3 License. See LICENSE for more information.