Hacking my android phone to drive my car
This blog post is an overview journey of me and friends making an affordable and safe self driving car system.
Will divide this blog into 3 parts: hardware, software and results.
This is an old story which I think would be good to share with the community.
The Hardware
What really do you need to make an ADAS system that keeps you in lane and adjusts speed of the car based on lead vehicles ?
- Sensors: Accelerometer, Gyroscope, Compass, GPS
- Compute units: A good CPU, some good RAM, a GPU for running your neural networks.
- User Interface: A display, a touch screen.
Wait, does this sounds familiar ? .. This is a smartphone ?!
We searched around and found project openpilot. It does exactly what I wanted to achieve.
But the problem I encountered was that openpilot only runs on 2 models of android phones, the oneplus 3t and the leeco pro 3 that too only on snapdragon based chips. And the newer versions of openpilot only run on a c3 which is a proprietary chip from comma. I dived down deeper into the openpilot codebase and realized why its difficult to port to other phones. Its specifically because openpilot uses custom lower level code to communicate with camera and some kernel hacks that allow booting NEOS/agnos on these devices.
But my question was, why there exists no app similar to openpilot that can be just be downloaded and worked out of the box ?
The software
We decided to modify openpilot codebase to make it able to run on technically ANY android phone, let it be snapdragon, exynos, google tensor, mediatek, etc!
Now, the first hurdle is, a lot of the openpilot codebase is in c++ and python. But android dosen’t run c++ or python executables. How do you account for this ?
Running a linux distribution on phone!
The answer is chroot/proot. chroot/proot allows us to emulate any linux distro such as ubuntu, arch linux, etc as long as they are the same architecture of the host machine.
Android runs bionic rather than glibc. Bionic is an implementation of the standard C library, developed by Google for its Android operating system. It differs from the GNU C Library (glibc) (which runs on standard linux distros) in being designed for devices with less memory and processor power than a typical Linux system. Nor does android include all of the GNU libraries you’d find on a typical Linux distribution. It also doesn’t include an X server like Xorg, so you can’t run standard graphical Linux applications.
This means that all the libraries that are compiled for standard linux cannot be used with android unless you cross compile them with the android NDK to use bionic. This already seems to be to much work and there is no guarantee at all that all the libraries that openpilot uses can be compiled using bionic for the arm architecture. The problem is not the arm architecture, it is widely supported and almost all the libraries that are available on standard x86 linux distro are available for arm too. The main problem is the bionic. Almost none of the libraries come pre compilied with bionic.
So instead of cross compiling the libraries for android, flowpilot bypasses this by using chroot or proot. think of chroot as a kind of a ‘naive virtual machine’. With this you can emulate running a standard linux distribution.
I use termux for getting access to terminal on android.
I named this project “flowpilot”, a modified version of openpilot that can run on any android and linux machine.
Generalized Android Hardware Access, the flowpilot APK
Now you can compile parts of openpilot fully in the chroot. But wait, what about display/ui ? How would you access android cameras, gpu, other hardware ? This cannot be done via chroot environment because although the devices like gpu, cameras are available at /dev, we dont have access to the their drivers which are proprietary.
For this I had to build a separate APK. This apk will handle running the gui, neural networks on gpu/dsp, accessing sensors and stuff since vendors mostly do a good job of adhering with hardware access APIs mandated by the Android OSP.
The app and the chroot environment communicate with each other over sockets.
We particularly love the openpilot UI and due to lack of my creativity… I shamelessly copied the openpilot UI style and re-wrote it in libgdx. I used libgdx because it’s cross-platform and wanted to maintain a unified code base for android and linux desktop PCs.
Testing & Results
This is a redmi note 9 pro driving a suzuki alto car!!
The car is a research vehicle of a local university where they fitted custom motors for controlling steering. The phone is communicating via a wireless CAN bus interface.
Full footage will be available on youtube later.
What happened next ?
We’ve been developing flowpilot on simulators as we don’t have permanent access to the car shown in the video. We added panda support and made flowpilot fully compatible with openpilot.
The project is open sourced and available on github here.
It has been out there for quite some time. Many people from the openpilot community came to test this out.
Imagine going to a local shop, buying some cables, downloading an android app and getting access to a world class autopilot system, openpilot
Safety concerns
Although, it came out as a cool project, use it at your own risk.
Here is an excerpt from openpilot safety model.
- The driver must always be capable to immediately retake manual control of the vehicle, by stepping on the brake pedal or by pressing the cancel button.
- The vehicle must not alter its trajectory too quickly for the driver to safely react. This means that while the system is engaged, the actuators are constrained to operate within reasonable limits.
A “safety layer” serves as a good enough way to ensure safety. A safety layer filters the inputs and the outputs of the safety layer are guaranteed to be within safe limits.
There are multiple safety layers between the phone and the actuators, this includes the panda which runs all the safety critical safety code written in MISRA C. And the highest level of safety is the Steering motor’s ECU itself. ECU limits the input commands. Its torque and ‘torque-rate’ limited.
All the ADAS code is undeterministic. But the safety layer will filter these inputs for the system to be ‘deterministic’.
So DO NOT bypass these safety layers unless you know what you are doing and have proper professionals backing you up.
More In-Action
How you can help.
Join our discord community and drop a star on github!
Ending Notes
Although the system works on most phones, we saw that if there are stripped down kernels optimized for running this, it would improve the overall experience. We now have ‘phone-brands’ channels on our discord where people can exploit the limits of android phones.
~ Congratulations on making it to the end of the blog post! I Hope you liked it. Have a nice day!
