Granblue Automation For Android using MediaProjection, AccessibilityService, and OpenCV

Discord here: https://discord.gg/5Yv4kqjAbm

Python version here: https://github.com/steve1316/granblue-automation-pyautogui

This Android application written in Kotlin and UI in React Native is designed for educational research purposes on studying the Android architecture and using system services such as MediaProjection and AccessibilityService in conjunction with OpenCV to automate certain workflows via image template matching.

Screenshots are taken via the MediaProjection and OpenCV performs image processing. Gestures are performed by AccessibilityService in order to perform various actions such as tapping, swiping or scrolling the screen.

Table of Contents

• Features
• Requirements
• Instructions
• Technologies used

Disclaimer

By downloading this application, you consent to your account potentially getting flagged for excessive usage and banned by KMR. I hold no responsibility for how much you use this application for. I trust you have the self-control necessary to use this application in reasonable bursts of time and to give yourself breaks in between.

Features

• Customize what skills to use during each turn in a user-created plan. Users can select which plan to use when starting the bot.
• GUI to keep track of bot logs, adjust settings, and select what combat script to use.
• Farm user-defined amounts of specified materials from the supported Farming Modes.
• Support for the following game modes:
  • Quest
  • Special
  • Coop
  • Raid
  • Event
  • Event (Token Drawboxes)
  • Rise of the Beasts
  • Guild Wars
  • Dread Barrage
  • Proving Grounds
  • Xeno Clash
  • Arcarum
• Alert for when anti-bot CAPTCHA pops up.
• Discord integration for informing you of status updates like loot drops via private DMs.

Requirements

1. Android Device or Emulator (Nougat 7.0+)
   • APK available to download is for devices that support arm64-v8a and armeabi-v7a ABI. For devices that use a different ABI, follow the instructions below on how to build your own APK.
   • (Experimental) Tablets supported with minimum 1600 pixel width like the Galaxy Tab S7. If oriented portrait, Chrome needs to have Desktop Mode turned off and situated on the left half of the tablet. If landscape, Chrome needs to have Desktop Mode turned on and situated on the left half of the tablet.
   • Tested emulator was Bluestacks 5. Make sure to have the device be in Portrait Mode BEFORE starting the bot as emulators do not have a way to tell the bot that it rotated.
     • In addition, use SkyLeap instead of Google Chrome. Room codes are not able to be properly pasted into Google Chrome while Raid farming on emulators.
2. Granblue Fantasy account
3. Twitter Developer account (optional, needed to farm Raids)

Instructions

1. Download the .apk file from the Releases section on the right and install it on your Android device. If you want to build the .apk yourself, do the following:
   1. Download and extract the project repository.
   2. Go to https://opencv.org/releases/ and download OpenCV 4.5.1 (make sure to download the Android version of OpenCV) and extract it.
   3. Create a new folder inside the root of the android folder named opencv and copy the extracted files in /OpenCV-android-sdk/sdk/ from Step 2 into it.
   4. Open the android folder in Android Studio and you can now build and run on your Android Device or build your own .apk file.
   5. You can set universalApk to true in the build.gradle for the application to build a one-for-all .apk file or adjust the include to customize which ABI(s) to build the .apk file for.
2. Once the application has started, navigate to the Settings page to fill out the necessary sections. Additional settings may be found in the Extra Settings page.
3. Once settings have been filled out, a snackbar will pop up indicating when the bot is ready to start.
4. Now you can tap on the Start button back in the Home page. If this is the first time, it will ask you to give the application Overlay permission and then to enable its Accessibility Service.
5. Once both are enabled, tapping on the Start button again will display an overlay button that you can move around the screen.
6. Navigate yourself to Granblue Fantasy's main page and press the overlay button to start. It is recommended to move the overlay button to the bottom left corner (Google Chrome) or anywhere at the bottom of the screen (SkyLeap) such that it does not cover the Back button to avoid interruption.
7. Whenever you want to stop while the bot is running, press the overlay button again to terminate the bot.
8. You can get rid of the overlay button by tapping the Stop button back in the Home page or tapping the Stop button on the application's notification.

If you want to farm Raids, follow the instructions below:

1. Make sure to run the application at least once. Check the application's files folder located inside internal storage's Android/data/com.steve1316.granblue_automation_android/ directory.
2. Locate the file named settings.json file. For ease of use, send it to your computer.
3. Now follow the instructions in this wiki page to obtain your own consumer keys and tokens to connect to the Twitter API.
4. Once you have them, copy them into the respective fields inside settings.json using your preferred text editor. After that, move the file back to where it belonged on your Android device.

Instructions to set up Discord integration

• Visit this wiki page for instructions on setting up Discord integration.

Technologies Used

1. MediaProjection - Used to obtain full screenshots
2. AccessibilityService - Used to dispatch gestures like tapping and scrolling
3. OpenCV Android 4.5.1 - Used to template match
4. Google Firebase ML Kit - Used for text detection
5. Hoplite - Used to read settings.json for customized user settings
6. Twitter4j - Used to connect to Twitter API to fetch room codes
7. AppUpdater 2.7 - For automatically checking and notifying the user for new app updates
8. React Native 0.64.3 - Used to display the UI and manage bot settings