awesome-dronecraft

I originally created this as a short to-do list of study topics for becoming a software engineer, I am creating this list to share my 3 year list of topics that I studied on the side during the curriculum of a Computer Engineering and Informatics degree. This is a list of short and medium length study topics to obtain knowledge regarding autonomous rotorcraft. The items listed here will give you enough knowledge to be able to understand how they work, their limitations and effort required to make them fly. Happy studying!

Format of this is hugely inspired by jwasham/coding-interview-university

Contributions are welcome, so please open a PR if you can improve this list or something is missing!

What is it?

This is my big study plan for going from simple programmer dude to software engineer who can understand how autonomous drones work.

There are references and video of various popular open source flight controller firmware and guidance projects in order to give the reader a sense of realism and real-sense and real-world value of operations, as well as act as get-to-know-the-market info.

This is meant for software engineers or people who already know how to code and also got basic knowledge of computer science topics including math (probablility, calculus, linear algebra). If you have many years of experience this can be easier to read through, but this is not guaranteed.

Disclaimer: I am not employed nor have I got any work experience on commerial or research rotorcraft.

Table of Contents

• What is it?

• Why use it?

• How to use it

• Don't feel you aren't smart enough

• You can't pick only one language

• Book List

• Before you Get Started

• What you Won't See Covered

• Prerequisite Knowledge

• Optional Courses

• Simulation and Control

• Control Theory

• Sensors and State Estimation

• Simultaneous Localization and Mapping

• Path Planning

• Mechatronics

• Existing Drone Software

• Existing Drone Hardware

• The Flight Controller

• Building a Racing FPV Quadcopter

• Building Fully Autonomous Rotorcraft

• Once You've Finished

• Other Resources

• Drone Usage in Industry

• Other Kinds of Vehicles

• More Advanced Topics

• Other Interesting

Why use it?

When I started this project, I didn't know how a drone could see, how the math behind the control algorithm worked, what the limitations are or anything like that. I started skimming through any resources I could after being inspired by the newly released Skydio r1 and it's CEO's numerous guest lectures at youtube. I slowly started building enough sensors to be able to grasp every sub-component of such flying systems. I have not learned everything yet, but the list is big enough to share and will always be evolving with more stuff added.

It's a long plan. It may take you months. If you are familiar with a lot of this already it will take you a lot less time.

How to use it

Everything below is an outline, and you should tackle the items in order from top to bottom.

I'm using Github's special markdown flavor, including tasks lists to check progress.

Create a new branch so you can check items like this, just put an x in the brackets: [x]

Fork a branch and follow the commands below

git checkout -b progress

git remote add zarkopafilis https://github.com/zarkopafilis/awesome-dronecraft

git fetch --all

Mark all boxes with X after you completed your changes

git add .

git commit -m "Marked x"

git rebase zarkopafilis/master

git push --force

More about Github-flavored markdown

Don't feel you aren't smart enough

• Successful software engineers are smart, but many have an insecurity that they aren't smart enough.
• The myth of the Genius Programmer
• It's Dangerous to Go Alone: Battling the Invisible Monsters in Tech
• Believe you can change
• Think you're not smart enough to work at Google? Well, think again

You can't pick only one language

You can't use only 1 language you are comfortable in to do the coding of the drones. You'd have to rely on many tools. Here are some for example.

• C++
• C
• Python
• Matlab (Yes, I know)

You may also need to learn a bit of other tools in the proccess. This is NOT something that you should feel worried about. With studying consistency there is nothing you can't achieve!

Book List

• Quan, Quan - Introduction to Multicopter Design and Control        A very good all-around book. It provides a more than quick-overview on all the different topics regarding quadcopters, from frame design, to electronics and control, diving deep enough on technical and practical design practises and considerations. It provides intuition behind every subject matter and does not containt ambiguities. The mathematics may seem too hard to deal with for some people, but that's not a thing to worry about. A person without mechanical or electrical engineering background can easily tackle this with some calculus and basic geometry knowledge.

Before you Get Started

This list grew over many months, and yes, it kind of got out of hand.

Here are some mistakes I made so you'll have a better experience.

You can't take it all in at once

Take breaks, write down possible things you need to re-review and watch them later or in a different way. This is a lot to take in and breaks are mandatory.

Some concepts can take much longer to understand fully. So if you don't understand it immediately do not worry. I suggest re-studying some of the parts that seemed hard to you after doing one pass on this study list.

Focus

There are a lot of distractions that can take up valuable time. Focus and concentration are hard.

What you won't see covered

• Don't know yet

Prerequisite Knowledge

Learn to code. This is only required for you to understand how the different algorithms and techniques that will be presented are actually implemented. You need this knowledge to be able to understand the source code of the popular data structure implementations, open source flight controllers and more. Basic data structure and algorithmic complexity should be included.

• [ ] C, C++, Python, Anything
  • Information available on the internet is widely available and you can find lot's of stuff by googling.
  • I could recommend to start with Python and work your way down to C and C++.
  • You can learn to code in parallel with this study plan, but things will be a lot harder and will take more time.
  • [ ] A complete computer science study plan to become a software engineer

The next part is needed to be able to understand the math behind the stuff we are going to use. It's not at all harder compared to other fields of study. Hang on and in the end, you'll be surprised by how easy it is to understand everything. This is not the only path to learn these, it's just what I would take. (Yes, I like university lectures and whitepapers.)

• [ ] Basic linear algebra, calculus, probablility and elementary physics
  • [ ] Single Variable Calculus - MIT OCW
  • [ ] Multivariable Calculus - MIT OCW
  • [ ] Gilbert Strang's Linear Algebra - MIT OCW there is really no better way to learn this in the world
  • [ ] Introduction to Probability and Statistics- MIT OCW
  • [ ] Digital Signal Proccessing - Rensselaer Polytechnic Institute

Courses

Optional

• These can get expensive, but here is a list of the popular ones. Most are cheap or free.

• Flying Cars Nanodegree

• Aerial Robotics

• Make an Open Source Drone

• Drone Programming Primer

• AI For Robotics

• Computer Vision

• Land Surveying with Drones

• 2D & 3D Drone Modeling and Mapping With Agisoft Metashape

• EDX Drone Courses: Applications and Agriculture

Simulation and Control

• This is the basis of rotorcraft that is going to get covered first. These few resources will make you understand what a drone needs to fly bad, good, with the help of extra autonomy engines or with the help of a pilot.

• After this part, the corks and screws of each subsystem is going to be investigated thoroughly.

• [ ] Matlab Tech Talks - Understanding Control Systems

• [ ] Matlab Tech Talks - Drone Simulation and Control

• [ ] Introduction to 6-DOF Simulation of Air Vehicles (pdf)

• [ ] Quadcopter Dynamics, Simulation and Control (paper)

• [ ] PID, LQR and LQR-PID on a quadcopter platform (paper)

• [ ] Betaflight: PID Tuning Guide

• [ ] Pixhawk: PID Auto Tune

At this point, you might wonder: This is only for four rotors. Don't worry, the extra ones are only used to have resilience. In the future this is going to be populated with more types of rotorcraft like submarines, VTOL drones and wings.

Control Theory

• Now that you have got a rough understanding of how you are going to make things fly, it's time to take a better look into the mathematical concepts behind the control of systems.

• [ ] Matlab Tech Talks - State Space

• [ ] Matlab Tech Talks - Control Systems in practise

• [ ] Matlab Tech Talks - Understanding PID control

Sensors and State Estimation

• In order to get the control theory to work, we have to obtain a best-effort state estimate by observing our rotorcraft system.

• [ ] Kalman Filters

  • [ ] Matlab Tech Talks - Understanding Kalman Filters
  • [ ] (or) Michel van Biezen - Kalman Filters

• [ ] Particle Filters

• [ ] (optional) Michel van Biezen - How GPS works

• [ ] Map Projections

  • [ ] Map Projections Explained
  • [ ] Equirectangular Projection (site)

• [ ] Inertia Measurement Units

  • [ ] How IMUs Work
  • [ ] How to implement an IMU
  • [ ] Soft mounting and vibrations

• [ ] Cameras and Optical Sensors

  • [ ] How does a camera work
  • [ ] Rolling and Global Shutter
  • [ ] Optical Flow
    • [ ] Computerphile - Optical Flow
    • [ ] Computerphile - Optical Flow Solutions
  • [ ] Depth Perception
    • How Kinect works in 2 minutes
    • Stereoscopic 3D basics
    • How does a LiDAR work
    • The correspondence problem]
  • [ ] PX4: Gimbal Control (site)
  • [ ] MAVLink: Gimbal Protocol (site)

• [ ] Ultrasonic Distance Sensor

• [ ] Corrections and Calibration

  • [ ] What is Sensor Calibration and Why is it Important?
  • [ ] Low Pass Filter (site)
  • [ ] Sampling Rates for Analog Sensors (site)
  • [ ] Signal Reconstruction

• [ ] Communication Protocols

  • [ ] Understanding the I2C Bus (pdf)
  • [ ] How I2C Works
  • [ ] UART (pdf)
  • [ ] How UART Works
  • [ ] SPI for Beginners
  • [ ] Analog to Digital Converters
  • [ ] SBUS and IBUS
  • [ ] Understanding Pulse Width Modulation
  • [ ] MAVLink
  • [ ] XBee full walkthrough

SLAM

• The sensor subsystem is complete. We can now procceed to construct a map of the environment as well as figure out where our UAV is located in the world.

• [ ] Quadtrees and Octrees (site)

• [ ] Octomap: An Efficient Probabilistic 3D Mapping Framework Based on Octrees (site)

• [ ] EKF SLAM - Cyrill Stachniss

• [ ] LiDAR Data Visualisation

• [ ] Autonomous Driving: Localization and Deep Learning Powered Mapping

Path Planning

• After proper localization, we are ready to path or mission plan, to make the UAV complete a predefined course of trajectory along with actions

• [ ] Robotic Path Planning: RTT, RTT* (article)

• [ ] 3DVFH+: Real-Time Three-Dimensional Obstacle Avoidance Using an Octomap (paper)

• [ ] QGroundControl: Mission Planning (site)

• [ ] ArduPilot: ZigZag Mode (site)

• [ ] ArduPilot: Automatic Launch (site)

Mechatronics

• [ ] How brushless motors work and how to control them
• [ ] Advantages and Disadvantages of Brushed and Brushless Motors
• [ ] How to choose the right brushless motor for your drone
• [ ] Slow retracting landing gear for drones
• [ ] Drone with Dual Robot Arms

Existing Drone Software

• For your own interest. Take a look into the source and issue tracker, maybe join the weekly dev calls.

• Autopilots:

• [ ] PX4

• [ ] Ardupilot

• [ ] BetaFlight

• [ ] Cleanflight

• [ ] INAV

• SDKs:

• [ ] MAVSDK

• [ ] Dronekit Dev Tools

• [ ] Parrot SDKs

• [ ] DJI SDKs

• [ ] Skydio Skills

• Ground Control Stations:

• [ ] QGroundControl

• Simulators:

• [ ] Gazebo

• [ ] Airsim

• Middleware:

• [ ] Robot Operating System

• Protocols:

• [ ] UAVCAN

• [ ] MAVLink

• [ ] Betaflight:Open ESC Firmware

• [ ] BLHeli

• Design Tools:

• [ ] Flight Evaluation - Rule and Constraint based airframe design and evaluation

Existing Drone Hardware

• For your own interest as well.

• [ ] Pixhawk Controllers

• [ ] Betaflight F4

• [ ] Kakute F7

• [ ] Kiss FC

• [ ] Zubax UAVCAN Propulsion Kit

• [ ] NEO-M8N GPS

• [ ] Skydio

• [ ] Intel Aero RTF

• [ ] QAV 250 RTF

• [ ] Intel Realsense

The Flight Controller

• [ ] Betaflight: Software Architecture Diagrams
• [ ] Betaflight: Unified Board Targets
• [ ] Pixhawk: Architectural Overview
• [ ] ArduPilot: Flight Modes
• [ ] Pixhawk: Flight Review

Building a Racing FPV Quadcopter

• In order to do that, you have to find a set of required hardware that is compatible, burn and configure your desired flight controller firmware.

• [ ] Minimum Hardware

  • [ ] Frame. For racing/fully manual control, typically carbon fiber, but you can even 3d print one or carve one out of wood. Take 2 dimensions in mind: flight controller stack size and propeller size. Most common propellers would be about 5".
  • [ ] Flight Controller. You know enough to choose one. I'd use betaflight.
  • [ ] ESCs. You can go with a 4 x 1 ESC or All in one board combo. Each has got advantages and disadvantages, search about it. But first, check your battery. It should be able to handle the burst current.
  • [ ] PDB. You need a power distribution board. These are relatively small and uncomplicated. You can find PDB+ESC+FC all in one combos that include battery step down and all that. But first, check your battery.
  • [ ] Battery. LiPo. At the time of this writing you can go from 4s to 6s. Price depends on capacity, discharge rating and cell count.
  • [ ] Brushless Motors. You know, to make the thing move.
  • [ ] RC Transceiver. Should be compatible with your RC Controller and it's protocol should be compatible with your FC board. Typically 2.4GHz.
  • [ ] RC Controller. Flysky, Taranis, there are too many to list.
  • After configuring frameware you should be able to fly this, but there are several stuff missing.

• [ ] Optional Hardware

  • [ ] Camera. Not the kind you are used to. This one sends out (analog) PAL/VTSC video and should support OSD
  • [ ] VTX. Stands for video transmitter. This is used to broadcast the video to typically 5.8GHz. Research how much mW you need.
  • [ ] FPV Goggles. To view the video. Should bear compatible antenna with the VTX. Diversity modules which dynamically choose the best signal source from an omnidirectional and a directional antenna are better.
  • [ ] Aftermarket Antennas. For beter reception.
  • [ ] Beeper. Beeps if you lose it.
  • [ ] Battery Balance Charger. You can't use discharged batteries.
  • [ ] Battery Balance Charger Power Supply. If your charger does not come with one. (Most IMAX B6s do not, for example)
  • [ ] 3D Printed GoPro Session Case
  • [ ] Insulation Tape
  • [ ] Zipties
  • [ ] LEDs for the bottom of the craft
  • [ ] Velcro Battery Straps

• [ ] Software and Further Configuration

  • [ ] Blackbox Logging. Important information regarding your flight. Can help you out for post-flight analysis.
  • [ ] RC Channel Mapping
  • [ ] Make sure that ESCs spin in the correct directions. You can change this in the ESC software or flight controller software.
  • [ ] Pair RC Transceiver with your RC Controller.
  • [ ] PID and Rates Tuning
  • [ ] Keep a configuration backup along with the exact firmware number so you don't forget.

Experienced racing pilots recommend that you stard by flying 'acro'/'freestyle' mode straight from the start. Do not do that with real drone hardware, the probability of crashing within 2 seconds is almost 100%. Take your time on a simulator first. There are many ones that you can find on the most popular gaming platforms like Steam. For example: Liftoff, DRL Sim, etc...

Flying 'angle'/'aided' mode is much easier and the same to flying your typical DJI/Parrot/etc drone around. You can fly this 'line of sight' even if you dont have goggles.

Building Fully Autonomous Rotorcraft

• Building Fully Autonomous Rotorcraft is much harder and more resource intensive than building a computer-aided one. You need to do better calibration because there is no pilot directly flying it-only supervising it.

• Basic Hardware. Basic in the sense that you need at least those for this to fly autonomously. You could maybe leave the GPS out of that, but you need at least optical flow for exact stabilization.

  • [ ] Flight Controller. Prefer a pre-tuned pre-built one that's resilient and robust enough. The Pixhawk 4 is a very good option, with dual IMUs, a ton of sensors, standarized connectors and a ton of connectivity options.
  • [ ] GPS
  • [ ] and/or Bottom Mounted Camera(s). Example of a drone that has got cameras on the bottom but no GPS, is the one from the matlab tech talks.
  • [ ] Frame
  • [ ] Propellers
  • [ ] Brushless Motors
  • [ ] Power Distribution Board
  • [ ] LiPo Battery
  • [ ] Connectivity
    • [ ] Some way to send mission control data and receive information
    • [ ] RC Controller to pilot-override the trajectory

• [ ] Optional Hardware

  • Basic Obstacle Avoidance
    • [ ] Front Mounted Camera (or RGB-D or long range ultrasonic for slow moving drones)
    • [ ] Basic Co-Processor if the flight controller firmware can't handle obstacle avoidance on it's own
  • Auto Land and Hand Land
    • [ ] Bottom mounted ultrasonic sensor
    • [ ] and/or bottom mounted camera(s) -- two cameras needed for depth in order to be more reliable than optical flow
  • Payload Carry
    • [ ] Some kind of magnet or servo to drop payloads
  • Advanced Camera Ops
    • [ ] Rotating Gimbal. In order not to couple the field of view along with the movement of the drone.
    • [ ] RGB-D Camera (for better SLAM)
    • [ ] LiDAR (SLAM too)
  • [ ] Retracting Landing Gear
  • Extra Connectivity
    • [ ] WiFi
    • [ ] LoRa
    • [ ] E-3G-4G-5G
    • [ ] Bluetooth

• Useful Co-Proccessor Compute Boards

  • Raspberry Pi

  • ODroid

  • nVidia Jetson

  • Coral USB ML Accelerator

  • Intel Neural Compute USB Stick

Usually, you start of from the 'job' requirements ex. Mapping, Crop Spraying, Inspection and then decide what hardware is capable of doing this job on a cost-benefit scenario. Configure everything and you should be ready to go.

Once You've Finished

Congratulations!

Keep learning.

You're never really done. But still, good job :).

Other Resources

• [ ] Digital Night Vision Insights

• [ ] How to Make an Infared Night Vision Camera From a Regular Digital Camera

• [ ] The Reconfigurable Aerial Robotic Chain: Modeling and Control

• [ ] Skydio: Adam Bry Guest Lecture CS287

• [ ] Autonomous Exploration inside Building Corridors

• [ ] Autonomous Self-deployment of Communication Breadcrumbs

• [ ] Army's Next Generation Silent Nano Drone

Drone Usage in Industry

Other Kinds of Vehicles

• Wings
• VTOL
• Underwater

More Advanced Topics

• [ ] UAV Swarms

• [ ] Automatic Resupply and Mission Resum

• [ ] Deep Learning

  • [ ] Andrew Ng: Deep Learning Specialization
  • [ ] Fast AI Courses

Now that you have made it so far, here are the last resources I am able to point you to. From this point on you should have a good understanding of how drones work and how to have a digital perception of the environment. The last parts are exactly a few missing pieces and how to pull everything together:

• [ ] Efficient and Accurate Tracking Objects Through Video
  • [ ] Kernelized Correlation Filters. This is the efficient approach #1. A very simple training step with a very simple detection and on-line update step.
  • [ ] End-to-end representation learning for Correlation Filter based tracking. This is the step #2, sacrificing some computational power and complexity in order to get a more robust and transformation-occlusion invariant tracking
• [ ] Lightweight Deep Neural Network Computations for Mobile and Embedded Platforms
  • [ ] MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. There is also a V2 version of MobileNets.
  • [ ] Intel OpenVINO. Optimizing deep learning models for the edge. nVidia and other companies provide their own toolkits as well, but they all work on a similar pruning-and-reducing neurons while maintaining a high enough performance metrics.
• [ ] Towards a Robust Aerial Cinematography Platform: Localizing and Tracking Moving Targets in Unstructured Environments. AKA the Skydio paper. The authors of this paper have published a lot of work on the subject and will probably keep doing them. I suggest that you look the up on google scholar.

Other Interesting

• Lex Fridman AI Podcasts
• 2 Minute Papers

Lastly, I hope that enjoyed reading and learning from this simple text file as much as I enjoyed researching and writing about it.