Android’s defense-in-depth strategy applies not only to the Android OS running on the Application Processor (AP) but also the firmware that runs on devices. We particularly prioritize hardening the cellular baseband given its unique combination of running in an elevated privilege and parsing untrusted inputs that are remotely delivered into the device.

This post covers how to use two high-value sanitizers which can prevent specific classes of vulnerabilities found within the baseband. They are architecture agnostic, suitable for bare-metal deployment, and should be enabled in existing C/C++ code bases to mitigate unknown vulnerabilities. Beyond security, addressing the issues uncovered by these sanitizers improves code health and overall stability, reducing resources spent addressing bugs in the future.

An increasingly popular attack surface

As we outlined previously, security research focused on the baseband has highlighted a consistent lack of exploit mitigations in firmware. Baseband Remote Code Execution (RCE) exploits have their own categorization in well-known third-party marketplaces with a relatively low payout. This suggests baseband bugs may potentially be abundant and/or not too complex to find and exploit, and their prominent inclusion in the marketplace demonstrates that they are useful.

Baseband security and exploitation has been a recurring theme in security conferences for the last decade. Researchers have also made a dent in this area in well-known exploitation contests. Most recently, this area has become prominent enough that it is common to find practical baseband exploitation trainings in top security conferences.

Acknowledging this trend, combined with the severity and apparent abundance of these vulnerabilities, last year we introduced updates to the severity guidelines of Android’s Vulnerability Rewards Program (VRP). For example, we consider vulnerabilities allowing Remote Code Execution (RCE) in the cellular baseband to be of CRITICAL severity.

Mitigating Vulnerability Root Causes with Sanitizers

Common classes of vulnerabilities can be mitigated through the use of sanitizers provided by Clang-based toolchains. These sanitizers insert runtime checks against common classes of vulnerabilities. GCC-based toolchains may also provide some level of support for these flags as well, but will not be considered further in this post. We encourage you to check your toolchain’s documentation.

Two sanitizers included in Undefined Behavior Sanitizer (UBSan) will be our focus – Integer Overflow Sanitizer (IntSan) and BoundsSanitizer (BoundSan). These have been widely deployed in Android userspace for years following a data-driven approach. These two are well suited for bare-metal environments such as the baseband since they do not require support from the OS or specific architecture features, and so are generally supported for all Clang targets.

Integer Overflow Sanitizer (IntSan)

IntSan causes signed and unsigned integer overflows to abort execution unless the overflow is made explicit. While unsigned integer overflows are technically defined behavior, it can often lead to unintentional behavior and vulnerabilities – especially when they’re used to index into arrays.

As both intentional and unintentional overflows are likely present in most code bases, IntSan may require refactoring and annotating the code base to prevent intentional or benign overflows from trapping (which we consider a false positive for our purposes). Overflows which need to be addressed can be uncovered via testing (see the Deploying Sanitizers section)

BoundsSanitizer (BoundSan)

BoundSan inserts instrumentation to perform bounds checks around some array accesses. These checks are only added if the compiler cannot prove at compile time that the access will be safe and if the size of the array will be known at runtime, so that it can be checked against. Note that this will not cover all array accesses as the size of the array may not be known at runtime, such as function arguments which are arrays.

As long as the code is correctly written C/C++, BoundSan should produce no false positives. Any violations discovered when first enabling BoundSan is at least a bug, if not a vulnerability. Resolving even those which aren’t exploitable can greatly improve stability and code quality.

Modernize your toolchains

Adopting modern mitigations also means adopting (and maintaining) modern toolchains. The benefits of this go beyond utilizing sanitizers however. Maintaining an old toolchain is not free and entails hidden opportunity costs. Toolchains contain bugs which are addressed in subsequent releases. Newer toolchains bring new performance optimizations, valuable in the highly constrained bare-metal environment that basebands operate in. Security issues can even exist in the generated code of out-of-date compilers.

Maintaining a modern up-to-date toolchain for the baseband entails some costs in terms of maintenance, especially at first if the toolchain is particularly old, but over time the benefits, as outlined above, outweigh the costs.

Where to apply sanitizers

Both BoundSan and IntSan have a measurable performance overhead. Although we were able to significantly reduce this overhead in the past (for example to less than 1% in media codecs), even very small increases in CPU load can have a substantial impact in some environments.

Enabling sanitizers over the entire codebase provides the most benefit, but enabling them in security-critical attack surfaces can serve as a first step in an incremental deployment. For example:

  • Functions parsing messages delivered over the air in 2G, 3G, 4G, and 5G (especially functions handling pre-authentication messages that can be injected with a false/malicious base station)
  • Libraries encoding/decoding complex formats (e.g. ASN.1, XML, DNS, etc…)
  • IMS, TCP and IP stacks
  • Messaging functions (SMS, MMS)

In the particular case of 2G, the best strategy is to disable the stack altogether by supporting Android’s “2G toggle”. However, 2G is still a necessary mobile access technology in certain parts of the world and some users might need to have this legacy protocol enabled.

Deploying Sanitizers

Having a clear plan for deployment of sanitizers saves a lot of time and effort. We think of the deployment process as having three stages:

  • Detecting (and fixing) violations
  • Measuring and reducing overhead
  • Soaking in pre-production

We also introduce two modes in which sanitizers should be run: diagnostics mode and trapping mode. These will be discussed in the following sections, but briefly: diagnostics mode recovers from violations and provides valuable debug information, while trapping mode actively mitigates vulnerabilities by trapping execution on violations.

Detecting (and Fixing) Violations

To successfully ship these sanitizers, any benign integer overflows must be made explicit and accidental out-of-bounds accesses must be addressed. These will have to be uncovered through testing. The higher the code coverage your tests provide, the more issues you can uncover at this stage and the easier deployment will be later on.

To diagnose violations uncovered in testing, sanitizers can emit calls to runtime handlers with debug information such as the file, line number, and values leading to the violation. Sanitizers can optionally continue execution after a violation has occurred, allowing multiple violations to be discovered in a single test run. We refer to using the sanitizers in this way as running them in “diagnostics mode”. Diagnostics mode is not intended for production as it provides no security benefits and adds high overhead.

Diagnostics mode for the sanitizers can be set using the following flags:

-fsanitize=signed-integer-overflow,unsigned-integer-overflow,bounds -fsanitize-recover=all

Since Clang does not provide a UBSan runtime for bare-metal targets, a runtime will need to be defined and provided at link time:

// integer overflow handlers
__ubsan_handle_add_overflow(OverflowData *data, ValueHandle lhs, ValueHandle rhs)
__ubsan_handle_sub_overflow(OverflowData *data, ValueHandle lhs, ValueHandle rhs)
__ubsan_handle_mul_overflow(OverflowData *data, ValueHandle lhs, ValueHandle rhs)
__ubsan_handle_divrem_overflow(OverflowData *data, ValueHandle lhs, ValueHandle rhs)
__ubsan_handle_negate_overflow(OverflowData *data, ValueHandle old_val)
// boundsan handler
__ubsan_handle_out_of_bounds_overflow(OverflowData *data, ValueHandle old_val)

As an example, see the default Clang implementation; the Linux Kernels implementation may also be illustrative.

With the runtime defined, enable the sanitizer over the entire baseband codebase and run all available tests to uncover and address any violations. Vulnerabilities should be patched, and overflows should either be refactored or made explicit through the use of checked arithmetic builtins which do not trigger sanitizer violations. Certain functions which are expected to have intentional overflows (such as cryptographic functions) can be preemptively excluded from sanitization (see next section).

Aside from uncovering security vulnerabilities, this stage is highly effective at uncovering code quality and stability bugs that could result in instability on user devices.

Once violations have been addressed and tests are no longer uncovering new violations, the next stage can begin.

Measuring and Reducing Overhead

Once shallow violations have been addressed, benchmarks can be run and the overhead from the sanitizers (performance, code size, memory footprint) can be measured.

Measuring overhead must be done using production flags – namely “trapping mode”, where violations cause execution to abort. The diagnostics runtime used in the first stage carries significant overhead and is not indicative of the actual performance sanitizer overhead.

Trapping mode can be enabled using the following flags:

-fsanitize=signed-integer-overflow,unsigned-integer-overflow,bounds -fsanitize-trap=all

Most of the overhead is likely due to a small handful of “hot functions”, for example those with tight long-running loops. Fine-grained per-function performance metrics (similar to what Simpleperf provides for Android), allows comparing metrics before and after sanitizers and provides the easiest means to identify hot functions. These functions can either be refactored or, after manual inspection to verify that they are safe, have sanitization disabled.

Sanitizers can be disabled either inline in the source or through the use of ignorelists and the -fsanitize-ignorelist flag.

The physical layer code, with its extremely tight performance margins and lower chance of exploitable vulnerabilities, may be a good candidate to disable sanitization wholesale if initial performance seems prohibitive.

Soaking in Pre-production

With overhead minimized and shallow bugs resolved, the final stage is enabling the sanitizers in trapping mode to mitigate vulnerabilities.

We strongly recommend a long period of internal soak in pre-production with test populations to uncover any remaining violations not discovered in testing. The more thorough the test coverage and length of the soak period, the less risk there will be from undiscovered violations.

As above, the configuration for trapping mode is as follows:

-fsanitize=signed-integer-overflow,unsigned-integer-overflow,bounds -fsanitize-trap=all

Having infrastructure in place to collect bug reports which result from any undiscovered violations can help minimize the risk they present.

Transitioning to Memory Safe Languages

The benefits from deploying sanitizers in your existing code base are tangible, however ultimately they address only the lowest hanging fruit and will not result in a code base free of vulnerabilities. Other classes of memory safety vulnerabilities remain unaddressed by these sanitizers. A longer term solution is to begin transitioning today to memory-safe languages such as Rust.

Rust is ready for bare-metal environments such as the baseband, and we are already using it in other bare-metal components in Android. There is no need to rewrite everything in Rust, as Rust provides a strong C FFI support and easily interfaces with existing C codebases. Just writing new code in Rust can rapidly reduce the number of memory safety vulnerabilities. Rewrites should be limited/prioritized only for the most critical components, such as complex parsers handling untrusted data.

The Android team has developed a Rust training meant to help experienced developers quickly ramp up Rust fundamentals. An entire day for bare-metal Rust is included, and the course has been translated to a number of different languages.

While the Rust compiler may not explicitly support your bare-metal target, because it is a front-end for LLVM, any target supported by LLVM can be supported in Rust through custom target definitions.

Raising the Bar

As the high-level operating system becomes a more difficult target for attackers to successfully exploit, we expect that lower level components such as the baseband will attract more attention. By using modern toolchains and deploying exploit mitigation technologies, the bar for attacking the baseband can be raised as well. If you have any questions, let us know – we’re here to help!

The App Defense Alliance (ADA), an industry-leading collaboration launched by Google in 2019 dedicated to ensuring the safety of the app ecosystem, is taking a major step forward. We are proud to announce that the App Defense Alliance is moving under the umbrella of the Linux Foundation, with Meta, Microsoft, and Google as founding steering members.

This strategic migration represents a pivotal moment in the Alliance’s journey, signifying a shared commitment by the members to strengthen app security and related standards across ecosystems. This evolution of the App Defense Alliance will enable us to foster more collaborative implementation of industry standards for app security.

Uniting for App Security

The digital landscape is continually evolving, and so are the threats to user security. With the ever-increasing complexity of mobile apps and the growing importance of data protection, now is the perfect time for this transition. The Linux Foundation is renowned for its dedication to fostering open-source projects that drive innovation, security, and sustainability. By combining forces with additional members under the Linux Foundation, we can adapt and respond more effectively to emerging challenges.

The commitment of the newly structured App Defense Alliance’s founding steering members – Meta, Microsoft, and Google – is pivotal in making this transition a reality. With a member community spanning an additional 16 General and Contributor Members, the Alliance will support industry-wide adoption of app security best practices and guidelines, as well as countermeasures against emerging security risks.

Continuing the Malware Mitigation Program

The App Defense Alliance was formed with the mission of reducing the risk of app-based malware and better protecting Android users. Malware defense remains an important focus for Google and Android, and we will continue to partner closely with the Malware Mitigation Program members – ESET, Lookout, McAfee, Trend Micro, Zimperium – on direct signal sharing. The migration of ADA under the Linux Foundation will enable broader threat intelligence sharing across leading ecosystem partners and researchers.

Looking Ahead and Connecting With the ADA

We invite you to stay connected with the newly structured App Defense Alliance under the Linux foundation umbrella. Join the conversation to help make apps more secure. Together with the steering committee, alliance partners, and the broader ecosystem, we look forward to building more secure and trustworthy app ecosystems.

Since 2018, Google has partnered with ARM and collaborated with many ecosystem partners (SoCs vendors, mobile phone OEMs, etc.) to develop Memory Tagging Extension (MTE) technology. We are now happy to share the growing adoption in the ecosystem. MTE is now available on some OEM devices (as noted in a recent blog post by Project Zero) with Android 14 as a developer option, enabling developers to use MTE to discover memory safety issues in their application easily.

The security landscape is changing dynamically, new attacks are becoming more complex and costly to mitigate. It’s becoming increasingly important to detect and prevent security vulnerabilities early in the software development cycle and also have the capability to mitigate the security attacks at the first moment of exploitation in production.

The biggest contributor to security vulnerabilities are memory safety related defects and Google has invested in a set of technologies to help mitigate memory safety risks. These include but are not limited to:

MTE is a hardware based capability that can detect unknown memory safety vulnerabilities in testing and/or mitigate them in production. It works by tagging the pointers and memory regions and comparing the tags to identify mismatches (details). In addition to the security benefits, MTE can also help ensure integrity because memory safety bugs remain one of the major contributors to silent data corruption that not only impact customer trust, but also cause lost productivity for software developers.

At the moment, MTE is supported on some of the latest chipsets:

  • Focusing on security for Android devices, the MediaTek Dimensity 9300 integrates support for MTE via ARM’s latest v9 architecture (which is what Cortex-X4 and Cortex-A720 processors are based on). This feature can be switched on and off in the bootloader by users and developers instead of having it always on or always off.
  • Tensor G3 integrates support for MTE only within the developer mode toggle. Feature can be activated by developers.

For both chipsets, this feature can be switched on and off by developers, making it easier to find memory-related bugs during development and after deployment. MTE can help users stay safe while also improving time to market for OEMs.

Application developers will be the first to leverage this feature as a way to improve their application security and reliability in the software development lifecycle. MTE can effectively help them to discover hard-to-detect memory safety vulnerabilities (buffer overflows, user-after-free, etc.) with clear & actionable stack trace information in integration testing or pre-production environments. Another benefit of MTE is that the engineering cost of memory-safety testing is drastically reduced because heap bug detection (which is majority of all memory safety bugs) does not require any source or binary changes to leverage MTE, i.e. advanced memory-safety can be achieved with just a simple environment or configuration change.

We believe that MTE will play a very important role in detecting and preventing memory safety vulnerabilities and provide a promising path towards improving software security.

Notes


  1. ASAN = Address Sanitizer; HWASAN = HW based ASAN;GWP-ASAN = sampling based ASAN 

Last year we wrote about how moving native code in Android from C++ to Rust has resulted in fewer security vulnerabilities. Most of the components we mentioned then were system services in userspace (running under Linux), but these are not the only components typically written in memory-unsafe languages. Many security-critical components of an Android system run in a “bare-metal” environment, outside of the Linux kernel, and these are historically written in C. As part of our efforts to harden firmware on Android devices, we are increasingly using Rust in these bare-metal environments too.

To that end, we have rewritten the Android Virtualization Framework’s protected VM (pVM) firmware in Rust to provide a memory safe foundation for the pVM root of trust. This firmware performs a similar function to a bootloader, and was initially built on top of U-Boot, a widely used open source bootloader. However, U-Boot was not designed with security in a hostile environment in mind, and there have been numerous security vulnerabilities found in it due to out of bounds memory access, integer underflow and memory corruption. Its VirtIO drivers in particular had a number of missing or problematic bounds checks. We fixed the specific issues we found in U-Boot, but by leveraging Rust we can avoid these sorts of memory-safety vulnerabilities in future. The new Rust pVM firmware was released in Android 14.

As part of this effort, we contributed back to the Rust community by using and contributing to existing crates where possible, and publishing a number of new crates as well. For example, for VirtIO in pVM firmware we’ve spent time fixing bugs and soundness issues in the existing virtio-drivers crate, as well as adding new functionality, and are now helping maintain this crate. We’ve published crates for making PSCI and other Arm SMCCC calls, and for managing page tables. These are just a start; we plan to release more Rust crates to support bare-metal programming on a range of platforms. These crates are also being used outside of Android, such as in Project Oak and the bare-metal section of our Comprehensive Rust course.

Training engineers

Many engineers have been positively surprised by how productive and pleasant Rust is to work with, providing nice high-level features even in low-level environments. The engineers working on these projects come from a range of backgrounds. Our comprehensive Rust course has helped experienced and novice programmers quickly come up to speed. Anecdotally the Rust type system (including the borrow checker and lifetimes) helps avoid making mistakes that are easily made in C or C++, such as leaking pointers to stack-allocated values out of scope.

One of our bare-metal Rust course attendees had this to say:

"types can be built that bring in all of Rust's niceties and safeties and 
yet still compile down to extremely efficient code like writes
of constants to memory-mapped IO."

97% of attendees that completed a survey agreed the course was worth their time.

Advantages and challenges

Device drivers are often written in an object-oriented fashion for flexibility, even in C. Rust traits, which can be seen as a form of compile-time polymorphism, provide a useful high-level abstraction for this. In many cases this can be resolved entirely at compile time, with no runtime overhead of dynamic dispatch via vtables or structs of function pointers.

There have been some challenges. Safe Rust’s type system is designed with an implicit assumption that the only memory the program needs to care about is allocated by the program (be it on the stack, the heap, or statically), and only used by the program. Bare-metal programs often have to deal with MMIO and shared memory, which break this assumption. This tends to require a lot of unsafe code and raw pointers, with limited tools for encapsulation. There is some disagreement in the Rust community about the soundness of references to MMIO space, and the facilities for working with raw pointers in stable Rust are currently somewhat limited. The stabilisation of offset_of, slice_ptr_get, slice_ptr_len, and other nightly features will improve this, but it is still challenging to encapsulate cleanly. Better syntax for accessing struct fields and array indices via raw pointers without creating references would also be helpful.

The concurrency introduced by interrupt and exception handlers can also be awkward, as they often need to access shared mutable state but can’t rely on being able to take locks. Better abstractions for critical sections will help somewhat, but there are some exceptions that can’t practically be disabled, such as page faults used to implement copy-on-write or other on-demand page mapping strategies.

Another issue we’ve had is that some unsafe operations, such as manipulating the page table, can’t be encapsulated cleanly as they have safety implications for the whole program. Usually in Rust we are able to encapsulate unsafe operations (operations which may cause undefined behaviour in some circumstances, because they have contracts which the compiler can’t check) in safe wrappers where we ensure the necessary preconditions so that it is not possible for any caller to cause undefined behaviour. However, mapping or unmapping pages in one part of the program can make other parts of the program invalid, so we haven’t found a way to provide a fully general safe interface to this. It should be noted that the same concerns apply to a program written in C, where the programmer always has to reason about the safety of the whole program.

Some people adopting Rust for bare-metal use cases have raised concerns about binary size. We have seen this in some cases; for example our Rust pVM firmware binary is around 460 kB compared to 220 kB for the earlier C version. However, this is not a fair comparison as we also added more functionality which allowed us to remove other components from the boot chain, so the overall size of all VM boot chain components was comparable. We also weren’t particularly optimizing for binary size in this case; speed and correctness were more important. In cases where binary size is critical, compiling with size optimization, being careful about dependencies, and avoiding Rust’s string formatting machinery in release builds usually allows comparable results to C.

Architectural support is another concern. Rust is generally well supported on the Arm and RISC-V cores that we see most often, but support for more esoteric architectures (for example, the Qualcomm Hexagon DSP included in many Qualcomm SoCs used in Android phones) can be lacking compared to C.

The future of bare-metal Rust

Overall, despite these challenges and limitations, we’ve still found Rust to be a significant improvement over C (or C++), both in terms of safety and productivity, in all the bare-metal use cases where we’ve tried it so far. We plan to use it wherever practical.

As well as the work in the Android Virtualization Framework, the team working on Trusty (the open-source Trusted Execution Environment used on Pixel phones, among others) have been hard at work adding support for Trusted Applications written in Rust. For example, the reference KeyMint Trusted Application implementation is now in Rust. And there’s more to come in future Android devices, as we continue to use Rust to improve security of the devices you trust.

Android 14 is the third major Android release with Rust support. We are already seeing a number of benefits:

These positive early results provided an enticing motivation to increase the speed and scope of Rust adoption. We hoped to accomplish this by investing heavily in training to expand from the early adopters.

Scaling up from Early Adopters

Early adopters are often willing to accept more risk to try out a new technology. They know there will be some inconveniences and a steep learning curve but are willing to learn, often on their own time.

Scaling up Rust adoption required moving beyond early adopters. For that we need to ensure a baseline level of comfort and productivity within a set period of time. An important part of our strategy for accomplishing this was training. Unfortunately, the type of training we wanted to provide simply didn’t exist. We made the decision to write and implement our own Rust training.

Training Engineers

Our goals for the training were to:

  • Quickly ramp up engineers: It is hard to take people away from their regular work for a long period of time, so we aimed to provide a solid foundation for using Rust in days, not weeks. We could not make anybody a Rust expert in so little time, but we could give people the tools and foundation needed to be productive while they continued to grow. The goal is to enable people to use Rust to be productive members of their teams. The time constraints meant we couldn’t teach people programming from scratch; we also decided not to teach macros or unsafe Rust in detail.
  • Make it engaging (and fun!): We wanted people to see a lot of Rust while also getting hands-on experience. Given the scope and time constraints mentioned above, the training was necessarily information-dense. This called for an interactive setting where people could quickly ask questions to the instructor. Research shows that retention improves when people can quickly verify assumptions and practice new concepts.
  • Make it relevant for Android: The Android-specific tooling for Rust was already documented, but we wanted to show engineers how to use it via worked examples. We also wanted to document emerging standards, such as using thiserror and anyhow crates for error handling. Finally, because Rust is a new language in the Android Platform (AOSP), we needed to show how to interoperate with existing languages such as Java and C++.

With those three goals as a starting point, we looked at the existing material and available tools.

Existing Material

Documentation is a key value of the Rust community and there are many great resources available for learning Rust. First, there is the freely available Rust Book, which covers almost all of the language. Second, the standard library is extensively documented.

Because we knew our target audience, we could make stronger assumptions than most material found online. We created the course for engineers with at least 2–3 years of coding experience in either C, C++, or Java. This allowed us to move quickly when explaining concepts familiar to our audience, such as “control flow”, “stack vs heap”, and “methods”. People with other backgrounds can learn Rust from the many other resources freely available online.

Technology

For free-form documentation, mdBook has become the de facto standard in the Rust community. It is used for official documentation such as the Rust Book and Rust Reference.

A particularly interesting feature is the ability to embed executable snippets of Rust code. This is key to making the training engaging since the code can be edited live and executed directly in the slides:

In addition to being a familiar community standard, mdBook offers the following important features:

  • Maintainability: mdbook test compiles and executes every code snippet in the course. This allowed us to evolve the class over time while ensuring that we always showed valid code to the participants.
  • Extensibility: mdBook has a plugin system which allowed us to extend the tool as needed. We relied on this feature for translations and ASCII art diagrams.

These features made it easy for us to choose mdBook. While mdBook is not designed for presentations, the output looked OK on a projector when we limited the vertical size of each page.

Supporting Translations

Android has developers and OEM partners in many countries. It is critical that they can adapt existing Rust code in AOSP to fit their needs. To support translations, we developed mdbook-i18n-helpers. Support for multilingual documentation has been a community wish since 2015 and we are glad to see the plugins being adopted by several other projects to produce maintainable multilingual documentation for everybody.

Comprehensive Rust

With the technology and format nailed down, we started writing the course. We roughly followed the outline from the Rust Book since it covered most of what we need to cover. This gave us a three day course which we called Rust Fundamentals. We designed it to run for three days for five hours a day and encompass Rust syntax, semantics, and important concepts such as traits, generics, and error handling.

We then extended Rust Fundamentals with three deep dives:

  • Rust in Android: a half-day course on using Rust for AOSP development. It includes interoperability with C, C++, and Java.
  • Bare-metal Rust: a full-day class on using Rust for bare-metal development. Android devices ship significant amounts of firmware. These components are often foundational in nature (for example, the bootloader, which establishes the trust for the rest of the system), thus they must be secure.
  • Concurrency in Rust: a full-day class on concurrency in Rust. We cover both multithreading with blocking synchronization primitives (such as mutexes) and async/await concurrency (cooperative multitasking using futures).

A large set of in-house and community translators have helped translate the course into several languages. The full translations were Brazilian Portuguese and Korean. We are working on Simplified Chinese and Traditional Chinese translations as well.

Course Reception

We started teaching the class in late 2022. In 2023, we hired a vendor, Immunant, to teach the majority of classes for Android engineers. This was important for scalability and for quality: dedicated instructors soon discovered where the course participants struggled and could adapt the delivery. In addition, over 30 Googlers have taught the course worldwide.

More than 500 Google engineers have taken the class. Feedback has been very positive: 96% of participants agreed it was worth their time. People consistently told us that they loved the interactive style, highlighting how it helped to be able to ask clarifying questions at any time. Instructors noted that people gave the course their undivided attention once they realized it was live. Live-coding demands a lot from the instructor, but it is worth it due to the high engagement it achieves.

Most importantly, people exited this course and were able to be immediately productive with Rust in their day jobs. When participants were asked three months later, they confirmed that they were able to write and review Rust code. This matched the results from the much larger survey we made in 2022.

Looking Forward

We have been teaching Rust classes at Google for a year now. There are a few things that we want to improve: better topic ordering, more exercises, and more speaker notes. We would also like to extend the course with more deep dives. Pull requests are very welcome!

The full course is available for free at https://google.github.io/comprehensive-rust/. We are thrilled to see people starting to use Comprehensive Rust for classes around the world. We hope it can be a useful resource for the Rust community and that it will help both small and large teams get started on their Rust journey!

Thanks!

We are grateful to the 190+ contributors from all over the world who created more than 1,000 pull requests and issues on GitHub. Their bug reports, fixes, and feedback improved the course in countless ways. This includes the 50+ people who worked hard on writing and maintaining the many translations.

Special thanks to Andrew Walbran for writing Bare-metal Rust and to Razieh Behjati, Dustin Mitchell, and Alexandre Senges for writing Concurrency in Rust.

We also owe a great deal of thanks to the many volunteer instructors at Google who have been spending their time teaching classes around the globe. Your feedback has helped shape the course.

Finally, thanks to Jeffrey Vander Stoep, Ivan Lozano, Matthew Maurer, Dmytro Hrybenko, and Lars Bergstrom for providing feedback on this post.

Fuzzing is an effective technique for finding software vulnerabilities. Over the past few years Android has been focused on improving the effectiveness, scope, and convenience of fuzzing across the organization. This effort has directly resulted in improved test coverage, fewer security/stability bugs, and higher code quality. Our implementation of continuous fuzzing allows software teams to find new bugs/vulnerabilities, and prevent regressions automatically without having to manually initiate fuzzing runs themselves. This post recounts a brief history of fuzzing on Android, shares how Google performs fuzzing at scale, and documents our experience, challenges, and success in building an infrastructure for automating fuzzing across Android. If you’re interested in contributing to fuzzing on Android, we’ve included instructions on how to get started, and information on how Android’s VRP rewards fuzzing contributions that find vulnerabilities.

A Brief History of Android Fuzzing

Fuzzing has been around for many years, and Android was among the early large software projects to automate fuzzing and prioritize it similarly to unit testing as part of the broader goal to make Android the most secure and stable operating system. In 2019 Android kicked off the fuzzing project, with the goal to help institutionalize fuzzing by making it seamless and part of code submission. The Android fuzzing project resulted in an infrastructure consisting of Pixel phones and Google cloud based virtual devices that enabled scalable fuzzing capabilities across the entire Android ecosystem. This project has since grown to become the official internal fuzzing infrastructure for Android and performs thousands of fuzzing hours per day across hundreds of fuzzers.

Under the Hood: How Is Android Fuzzed

Step 1: Define and find all the fuzzers in Android repo

The first step is to integrate fuzzing into the Android build system (Soong) to enable build fuzzer binaries. While developers are busy adding features to their codebase, they can include a fuzzer to fuzz their code and submit the fuzzer alongside the code they have developed. Android Fuzzing uses a build rule called cc_fuzz (see example below). cc_fuzz (we also support rust_fuzz and java_fuzz) defines a Soong module with source file(s) and dependencies that can be built into a binary.

cc_fuzz {
  name: "fuzzer_foo",

  srcs: [
    "fuzzer_foo.cpp",
  ],

  static_libs: [
    "libfoo",
  ],

  host_supported: true,
}

A packaging rule in Soong finds all of these cc_fuzz definitions and builds them automatically. The actual fuzzer structure itself is very simple and consists of one main method (LLVMTestOneInput):

#include <stddef.h>
#include <stdint.h>

extern "C" int LLVMFuzzerTestOneInput(
               const uint8_t *data,
               size_t size) {

  // Here you invoke the code to be fuzzed. 
  return 0;
}

This fuzzer gets automatically built into a binary and along with its static/dynamic dependencies (as specified in the Android build file) are packaged into a zip file which gets added to the main zip containing all fuzzers as shown in the example below.

Step 2: Ingest all fuzzers into Android builds

Once the fuzzers are found in the Android repository and they are built into binaries, the next step is to upload them to the cloud storage in preparation to run them on our backend. This process is run multiple times daily. The Android fuzzing infrastructure uses an open source continuous fuzzing framework (Clusterfuzz) to run fuzzers continuously on Android devices and emulators. In order to run the fuzzers on clusterfuzz, the fuzzers zip files are renamed after the build and the latest build gets to run (see diagram below):

The fuzzer zip file contains the fuzzer binary, corresponding dictionary as well as a subfolder containing its dependencies and the git revision numbers (sourcemap) corresponding to the build. Sourcemaps are used to enhance stack traces and produce crash reports.

Step 3: Run fuzzers continuously and find bugs

Running fuzzers continuously is done through scheduled jobs where each job is associated with a set of physical devices or emulators. A job is also backed by a queue that represents the fuzzing tasks that need to be run. These tasks are a combination of running a fuzzer, reproducing a crash found in an earlier fuzzing run, or minimizing the corpus, among other tasks.

Each fuzzer is run for multiple hours, or until they find a crash. After the run, Android fuzzing takes all of the interesting input discovered during the run and adds it to the fuzzer corpus. This corpus is then shared across fuzzer runs and grows over time. The fuzzer is then prioritized in subsequent runs according to the growth of new coverage and crashes found (if any). This ensures we provide the most effective fuzzers more time to run and find interesting crashes.

Step 4: Generate fuzzers line coverage

What good is a fuzzer if it’s not fuzzing the code you care about? To improve the quality of the fuzzer and to monitor the overall progress of Android fuzzing, two types of coverage metrics are calculated and available to Android developers. The first metric is for edge coverage which refers to edges in the Control Flow Graph (CFG). By instrumenting the fuzzer and the code being fuzzed, the fuzzing engine can track small snippets of code that get triggered every time execution flow reaches them. That way, fuzzing engines know exactly how many (and how many times) each of these instrumentation points got hit on every run so they can aggregate them and calculate the coverage.

INFO: Seed: 2859304549
INFO: Loaded 1 modules   (773 inline 8-bit counters): 773 [0x5610921000, 0x5610921305),
INFO: Loaded 1 PC tables (773 PCs): 773 [0x5610921308,0x5610924358),
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: A corpus is not provided, starting from an empty corpus
#2      INITED cov: 2 ft: 2 corp: 1/1b lim: 4 exec/s: 0 rss: 24Mb
#413    NEW    cov: 3 ft: 3 corp: 2/9b lim: 8 exec/s: 0 rss: 24Mb L: 8/8 MS: 1 InsertRepeatedBytes-
#3829   NEW    cov: 4 ft: 4 corp: 3/17b lim: 38 exec/s: 0 rss: 24Mb L: 8/8 MS: 1 ChangeBinInt-
...

Line coverage inserts instrumentation points specifying lines in the source code. Line coverage is very useful for developers as they can pinpoint areas in the code that are not covered and update their fuzzers accordingly to hit those areas in future fuzzing runs.

Drilling into any of the folders can show the stats per file:

Further clicking on one of the files shows the lines that were touched and lines that never got coverage. In the example below, the first line has been fuzzed ~5 million times, but the fuzzer never makes it into lines 3 and 4, indicating a gap in the coverage for this fuzzer.

We have dashboards internally that measure our fuzzing coverage across our entire codebase. In order to generate these coverage dashboards yourself, you follow these steps.

Another measurement of the quality of the fuzzers is how many fuzzing iterations can be done in one second. It has a direct relationship with the computation power and the complexity of the fuzz target. However, this parameter alone can not measure how good or effective the fuzzing is.

How we handle fuzzer bugs

Android fuzzing utilizes the Clusterfuzz fuzzing infrastructure to handle any found crashes and file a ticket to the Android security team. Android security makes an assessment of the crash based on the Android Severity Guidelines and then routes the vulnerability to the proper team for remediation. This entire process of finding the reproducible crash, routing to Android Security, and then assigning the issue to a team responsible can take as little as two hours, and up to a week depending on the type of crash and the severity of the vulnerability.

One example of a recent fuzzer success is (CVE 2022-20473), where an internal team wrote a 20-line fuzzer and submitted it to run on Android fuzzing infra. Within a day, the fuzzer was ingested and pushed to our fuzzing infrastructure to begin fuzzing, and shortly found a critical severity vulnerability! A patch for this CVE has been applied by the service team.

Why Android Continues to Invest in Fuzzing

Protection Against Code Regressions

The Android Open Source Project (AOSP) is a large and complex project with many contributors. As a result, there are thousands of changes made to the project every day. These changes can be anything from small bug fixes to large feature additions, and fuzzing helps to find vulnerabilities that may be inadvertently introduced and not caught during code review.

Continuous fuzzing has helped to find these vulnerabilities before they are introduced in production and exploited by attackers. One real-life example is (CVE-2023-21041), a vulnerability discovered by a fuzzer written three years ago. This vulnerability affected Android firmware and could have led to local escalation of privilege with no additional execution privileges needed. This fuzzer was running for many years with limited findings until a code regression led to the introduction of this vulnerability. This CVE has since been patched.

Protection against unsafe memory language pitfalls

Android has been a huge proponent of Rust, with Android 13 being the first Android release with the majority of new code in a memory safe language. The amount of new memory-unsafe code entering Android has decreased, but there are still millions of lines of code that remain, hence the need for fuzzing persists.

No One Code is Safe: Fuzzing code in memory-safe languages

Our work does not stop with non-memory unsafe languages, and we encourage fuzzer development in languages like Rust as well. While fuzzing won’t find common vulnerabilities that you would expect to see memory unsafe languages like C/C++, there have been numerous non-security issues discovered and remediated which contribute to the overall stability of Android.

Fuzzing Challenges

In addition to generic C/C++ binaries issues such as missing dependencies, fuzzers can have their own classes of problems:

Low executions per second: in order to fuzz efficiently, the number of mutations has to be in the order of hundreds per second otherwise the fuzzing will take a very long time to cover the code. We addressed this issue by adding a set of alerts that continuously monitor the health of the fuzzers as well as any sudden drop in coverage. Once a fuzzer is identified as underperforming, an automated email is sent to the fuzzer author with details to help them improve the fuzzer.

Fuzzing the wrong code: Like all resources, fuzzing resources are limited. We want to ensure that those resources give us the highest return, and that generally means devoting them towards fuzzing code that processes untrusted (i.e. potentially attacker controlled) inputs. This can cover any way that the phone can receive input including Bluetooth, NFC, USB, web, etc. Parsing structured input is particularly interesting since there is room for programming errors due to specs complexity. Code that generates output is not particularly interesting to fuzz. Similarly internal code that is not exposed publicly is also less of a security concern. We addressed this issue by identifying the most vulnerable code (see the following section).

What to fuzz

In order to fuzz the most important components of the Android source code, we focus on libraries that have:

  1. A history of vulnerabilities: the history should not be the distant history since context change but more focus on the last 12 months.
  2. Recent code changes: research indicates that more vulnerabilities are found in recently changed code than code that is more stable.
  3. Remote access: vulnerabilities in code that are reachable remotely can be critical.
  4. Privileged: Similarly to #3, vulnerabilities in code that runs in privileged processes can be critical.

How to submit a fuzzer to AOSP

We’re constantly writing and improving fuzzers internally to cover some of the most sensitive areas of Android, but there is always room for improvement. If you’d like to get started writing your own fuzzer for an area of AOSP, you’re welcome to do so to make Android more secure (example CL):

  1. Get Android source code
  2. Have a testing phone?
  3. Write a fuzz target (follow guidelines in ‘What to fuzz’ section)
  4. Upload your fuzzer to AOSP.

Get started by reading our documentation on Fuzzing with libFuzzer and check your fuzzer into the Android Open Source project. If your fuzzer finds a bug, you can submit it to the Android Bug Bounty Program and could be eligible for a reward!

Android is the first mobile operating system to introduce advanced cellular security mitigations for both consumers and enterprises. Android 14 introduces support for IT administrators to disable 2G support in their managed device fleet. Android 14 also introduces a feature that disables support for null-ciphered cellular connectivity.

Hardening network security on Android

The Android Security Model assumes that all networks are hostile to keep users safe from network packet injection, tampering, or eavesdropping on user traffic. Android does not rely on link-layer encryption to address this threat model. Instead, Android establishes that all network traffic should be end-to-end encrypted (E2EE).

When a user connects to cellular networks for their communications (data, voice, or SMS), due to the distinctive nature of cellular telephony, the link layer presents unique security and privacy challenges. False Base Stations (FBS) and Stingrays exploit weaknesses in cellular telephony standards to cause harm to users. Additionally, a smartphone cannot reliably know the legitimacy of the cellular base station before attempting to connect to it. Attackers exploit this in a number of ways, ranging from traffic interception and malware sideloading, to sophisticated dragnet surveillance.

Recognizing the far reaching implications of these attack vectors, especially for at-risk users, Android has prioritized hardening cellular telephony. We are tackling well-known insecurities such as the risk presented by 2G networks, the risk presented by null ciphers, other false base station (FBS) threats, and baseband hardening with our ecosystem partners.

2G and a history of inherent security risk

The mobile ecosystem is rapidly adopting 5G, the latest wireless standard for mobile, and many carriers have started to turn down 2G service. In the United States, for example, most major carriers have shut down 2G networks. However, all existing mobile devices still have support for 2G. As a result, when available, any mobile device will connect to a 2G network. This occurs automatically when 2G is the only network available, but this can also be remotely triggered in a malicious attack, silently inducing devices to downgrade to 2G-only connectivity and thus, ignoring any non-2G network. This behavior happens regardless of whether local operators have already sunset their 2G infrastructure.

2G networks, first implemented in 1991, do not provide the same level of security as subsequent mobile generations do. Most notably, 2G networks based on the Global System for Mobile Communications (GSM) standard lack mutual authentication, which enables trivial Person-in-the-Middle attacks. Moreover, since 2010, security researchers have demonstrated trivial over-the-air interception and decryption of 2G traffic.

The obsolete security of 2G networks, combined with the ability to silently downgrade the connectivity of a device from both 5G and 4G down to 2G, is the most common use of FBSs, IMSI catchers and Stingrays.

Stingrays are obscure yet very powerful surveillance and interception tools that have been leveraged in multiple scenarios, ranging from potentially sideloading Pegasus malware into journalist phones to a sophisticated phishing scheme that allegedly impacted hundreds of thousands of users with a single FBS. This Stingray-based fraud attack, which likely downgraded device’s connections to 2G to inject SMSishing payloads, has highlighted the risks of 2G connectivity.

To address this risk, Android 12 launched a new feature that enables users to disable 2G at the modem level. Pixel 6 was the first device to adopt this feature and it is now supported by all Android devices that conform to Radio HAL 1.6+. This feature was carefully designed to ensure that users are not impacted when making emergency calls.

Mitigating 2G security risks for enterprises

The industry acknowledged the significant security and privacy benefits and impact of this feature for at-risk users, and we recognized how critical disabling 2G could also be for our Android Enterprise customers.

Enterprises that use smartphones and tablets require strong security to safeguard sensitive data and Intellectual Property. Android Enterprise provides robust management controls for connectivity safety capabilities, including the ability to disable WiFi, Bluetooth, and even data signaling over USB. Starting in Android 14, enterprise customers and government agencies managing devices using Android Enterprise will be able to restrict a device’s ability to downgrade to 2G connectivity.

The 2G security enterprise control in Android 14 enables our customers to configure mobile connectivity according to their risk model, allowing them to protect their managed devices from 2G traffic interception, Person-in-the-Middle attacks, and other 2G-based threats. IT administrators can configure this protection as necessary, always keeping the 2G radio off or ensuring employees are protected when traveling to specific high-risk locations.

These new capabilities are part of the comprehensive set of 200+ management controls that Android provides IT administrators through Android Enterprise. Android Enterprise also provides comprehensive audit logging with over 80 events including these new management controls. Audit logs are a critical part of any organization’s security and compliance strategy. They provide a detailed record of all activity on a system, which can be used to track down unauthorized access, identify security breaches, and troubleshoot system problems.

Also in Android 14

The upcoming Android release also tackles the risk of cellular null ciphers. Although all IP-based user traffic is protected and E2EE by the Android platform, cellular networks expose circuit-switched voice and SMS traffic. These two particular traffic types are strictly protected only by the cellular link layer cipher, which is fully controlled by the network without transparency to the user. In other words, the network decides whether traffic is encrypted and the user has no visibility into whether it is being encrypted.

Recent reports identified usage of null ciphers in commercial networks, which exposes user voice and SMS traffic (such as One-Time Password) to trivial over the air interception. Moreover, some commercial Stingrays provide functionality to trick devices into believing ciphering is not supported by the network, thus downgrading the connection to a null cipher and enabling traffic interception.

Android 14 introduces a user option to disable support, at the modem-level, for null-ciphered connections. Similarly to 2G controls, it’s still possible to place emergency calls over an unciphered connection. This functionality will greatly improve communication privacy for devices that adopt the latest radio hardware abstraction layer (HAL). We expect this new connectivity security feature to be available in more devices over the next few years as it is adopted by Android OEMs.

Continuing to partner to raise the industry bar for cellular security

Alongside our Android-specific work, the team is regularly involved in the development and improvement of cellular security standards. We actively participate in standards bodies such as GSMA Fraud and Security Group as well as the 3rd Generation Partnership Project (3GPP), particularly its security and privacy group (SA3). Our long-term goal is to render FBS threats obsolete.

In particular, Android security is leading a new initiative within GSMA’s Fraud and Security Group (FASG) to explore the feasibility of modern identity, trust and access control techniques that would enable radically hardening the security of telco networks.

Our efforts to harden cellular connectivity adopt Android’s defense-in-depth strategy. We regularly partner with other internal Google teams as well, including the Android Red Team and our Vulnerability Rewards Program.

Moreover, in alignment with Android’s openness in security, we actively partner with top academic groups in cellular security research. For example, in 2022 we funded via our Android Security and Privacy Research grant (ASPIRE) a project to develop a proof-of-concept to evaluate cellular connectivity hardening in smartphones. The academic team presented the outcome of that project in the last ACM Conference on Security and Privacy in Wireless and Mobile Networks.

The security journey continues

User security and privacy, which includes the safety of all user communications, is a priority on Android. With upcoming Android releases, we will continue to add more features to harden the platform against cellular security threats.

We look forward to discussing the future of telco network security with our ecosystem and industry partners and standardization bodies. We will also continue to partner with academic institutions to solve complex problems in network security. We see tremendous opportunities to curb FBS threats, and we are excited to work with the broader industry to solve them.

Special thanks to our colleagues who were instrumental in supporting our cellular network security efforts: Nataliya Stanetsky, Robert Greenwalt, Jayachandran C, Gil Cukierman, Dominik Maier, Alex Ross, Il-Sung Lee, Kevin Deus, Farzan Karimi, Xuan Xing, Wes Johnson, Thiébaud Weksteen, Pauline Anthonysamy, Liz Louis, Alex Johnston, Kholoud Mohamed, Pavel Grafov

Most modern consumer messaging platforms (including Google Messages) support end-to-end encryption, but users today are limited to communicating with contacts who use the same platform. This is why Google is strongly supportive of regulatory efforts that require interoperability for large end-to-end messaging platforms.

For interoperability to succeed in practice, however, regulations must be combined with open, industry-vetted, standards, particularly in the area of privacy, security, and end-to-end encryption. Without robust standardization, the result will be a spaghetti of ad hoc middleware that could lower security standards to cater for the lowest common denominator and raise implementation costs, particularly for smaller providers. Lack of standardization would also make advanced features such as end-to-end encrypted group messaging impossible in practice – group messages would have to be encrypted and delivered multiple times to cater for every different protocol.

With the recent publication of the IETF’s Message Layer Security (MLS) specification RFC 9420, messaging users can look forward to this reality. For the first time, MLS enables practical interoperability across services and platforms, scaling to groups of thousands of multi-device users. It is also flexible enough to allow providers to address emerging threats to user privacy and security, such as quantum computing.

By ensuring a uniformly high security and privacy bar that users can trust, MLS will unleash a huge field of new opportunities for the users and developers of interoperable messaging services that adopt it. This is why we intend to build MLS into Google Messages and support its wide deployment across the industry by open sourcing our implementation in the Android codebase.

It has been another incredible year for the Vulnerability Reward Programs (VRPs) at Google! Working with security researchers throughout 2022, we have been able to identify and fix over 2,900 security issues and continue to make our products more secure for our users around the world.

We are thrilled to see significant year-over-year growth for our VRPs, and have had yet another record-breaking year for our programs! In 2022 we awarded over $12 million in bounty rewards – with researchers donating over $230,000 to a charity of their choice.

As in past years, we are sharing our 2022 Year in Review statistics across all of our programs. We would like to give a special thank you to all of our dedicated researchers for their continued work with our programs – we look forward to more collaboration in the future!

Android and Devices

The Android VRP had an incredible record breaking year in 2022 with $4.8 million in rewards and the highest paid report in Google VRP history of $605,000!

In our continued effort to ensure the security of Google device users, we have expanded the scope of Android and Google Devices in our program and are now incentivizing vulnerability research in the latest versions of Google Nest and Fitbit! For more information on the latest program version and qualifying vulnerability reports, please visit our public rules page.

We are also excited to share that the invite-only Android Chipset Security Reward Program (ACSRP) – a private vulnerability reward program offered by Google in collaboration with manufacturers of Android chipsets – rewarded $486,000 in 2022 and received over 700 valid security reports.

We would like to give a special shoutout to some of our top researchers, whose continued hard work helps to keep Android safe and secure:

  • Submitting an impressive 200+ vulnerabilities to the Android VRP this year, Aman Pandey of Bugsmirror remains one of our program’s top researchers. Since submitting their first report in 2019, Aman has reported more than 500 vulnerabilities to the program. Their hard work helps ensure the safety of our users; a huge thank you for all of their hard work!
  • Zinuo Han of OPPO Amber Security Lab quickly rose through our program’s ranks, becoming one of our top researchers. In the last year they have identified 150 valid vulnerabilities in Android.
  • Finding yet another critical exploit chain, gzobqq submitted our highest valued exploit to date.
  • Yu-Cheng Lin (林禹成) (@AndroBugs) remains one of our top researchers submitting just under 100 reports this year.

Chrome

Chrome VRP had another unparalleled year, receiving 470 valid and unique security bug reports, resulting in a total of $4 million of VRP rewards. Of the $4M, $3.5 million was rewarded to researchers for 363 reports of security bugs in Chrome Browser and nearly $500,000 was rewarded for 110 reports of security bugs in ChromeOS.

This year, Chrome VRP re-evaluated and refactored the Chrome VRP reward amounts to increase the reward amounts for the most exploitable and harmful classes and types of security bugs, as well as added a new category for memory corruption bugs in highly privileged processes, such as the GPU and network process, to incentivize research in these critical areas. The Chrome VRP increased the fuzzer bonuses for reports from VRP-submitted fuzzers running on the Google ClusterFuzz infrastructure as part of the Chrome Fuzzing program. A new bisect bonus was introduced for bisections performed as part of the bug report submission, which helps the security team with our triage and bug reproduction.

2023 will be the year of experimentation in the Chrome VRP! Please keep a lookout for announcements of experiments and potential bonus opportunities for Chrome Browser and ChromeOS security bugs.

The entire Chrome team sincerely appreciates the contributions of all our researchers in 2022 who helped keep Chrome Browser, ChromeOS, and all the browsers and software based on Chromium secure for billions of users across the globe.

In addition to posting about our Top 0-22 Researchers in 2022, Chrome VRP would like to specifically acknowledge some specific researcher achievements made in 2022:

  • Rory McNamara, a six-year participant in Chrome VRP as a ChromeOS researcher, became the highest rewarded researcher of all time in the Chrome VRP. Most impressive is that Rory has achieved this in a total of only 40 security bug submissions, demonstrating just how impactful his findings have been – from ChromeOS persistent root command execution, resulting in a $75,000 reward back in 2018, to his many reports of root privilege escalation both with and without persistence. Rory was also kind enough to speak at the Chrome Security Summit in 2022 to share his experiences participating in the Chrome VRP over the years. Thank you, Rory!
  • SeongHwan Park (SeHwa), a participant in the Chrome VRP since mid-2021, has been an amazing contributor of ANGLE / GPU security bug reports in 2022 with 11 solid quality reports of GPU bugs earning them a spot on Chrome VRP 2022 top researchers list. Thank you, SeHwa!

Securing Open Source

Recognizing the fact that Google is one of the largest contributors and users of open source in the world, in August 2022 we launched OSS VRP to reward vulnerabilities in Google’s open source projects – covering supply chain issues of our packages, and vulnerabilities that may occur in end products using our OSS. Since then, over 100 bughunters have participated in the program and were rewarded over $110,000.

Sharing Knowledge

We’re pleased to announce that in 2022, we’ve made the learning opportunities for bug hunters available at our Bug Hunter University (BHU) more diverse and accessible. In addition to our existing collections of articles, which support improving your reports and avoiding invalid reports, we’ve made more than 20 instructional videos available. Clocking in at around 10 minutes each, these videos cover the most relevant learning topics and trends we’ve observed over the past years.

To make this happen, we teamed up with some of your favorite and best-known security researchers from around the globe, including LiveOverflow, PwnFunction, stacksmashing, InsiderPhD, PinkDraconian, and many more!

If you’re tired of reading our articles, or simply curious and looking for an alternative way to expand your bug hunting skills, these videos are for you. Check out our overview, or hop right in to the BHU YouTube playlist. Happy watching & learning!

Google Play

2022 was a year of change for the Google Play Security Reward Program. In May we onboarded both new teammates and some old friends to triage and lead GPSRP. We also sponsored NahamCon ‘22, BountyCon in Singapore, and NahamCon Europe’s online event. In 2023 we hope to continue to grow the program with new bug hunters and partner on more events focused on Android & Google Play apps.

Research Grants

In 2022 we continued our Vulnerability Research Grant program with success. We’ve awarded more than $250,000 in grants to over 170 security researchers. Last year we also piloted collaboration double VRP rewards for selected grants and are looking forward to expanding it even more in 2023.

If you are a Google VRP researcher and want to be considered for a Vulnerability Research Grant, make sure you opted in on your bughunters profile.

Looking Forward

Without our incredible security researchers we wouldn’t be here sharing this amazing news today. Thank you again for your continued hard work!

Also, in case you haven’t seen Hacking Google yet, make sure to check out the “Bug Hunters” episode, featuring some of our very own super talented bug hunters.

Thank you again for helping to make Google, the Internet, and our users more safe and secure! Follow us on @GoogleVRP for other news and updates.

Thank you to Adam Bacchus, Dirk Göhmann, Eduardo Vela, Sarah Jacobus, Amy Ressler, Martin Straka, Jan Keller, Tony Mendez, Rishika Hooda, Medha Jain

A modern Android powered smartphone is a complex hardware device: Android OS runs on a multi-core CPU – also called an Application Processor (AP). And the AP is one of many such processors of a System On Chip (SoC). Other processors on the SoC perform various specialized tasks — such as security functions, image & video processing, and most importantly cellular communications. The processor performing cellular communications is often referred to as the baseband. For the purposes of this blog, we refer to the software that runs on all these other processors as “Firmware”.

Securing the Android Platform requires going beyond the confines of the Application Processor (AP). Android’s defense-in-depth strategy also applies to the firmware running on bare-metal environments in these microcontrollers, as they are a critical part of the attack surface of a device.

A popular attack vector within the security research community

As the security of the Android Platform has been steadily improved, some security researchers have shifted their focus towards other parts of the software stack, including firmware. Over the last decade there have been numerous publications, talks, Pwn2Own contest winners, and CVEs targeting exploitation of vulnerabilities in firmware running in these secondary processors. Bugs remotely exploitable over the air (eg. WiFi and cellular baseband bugs) are of particular concern and, therefore, are popular within the security research community. These types of bugs even have their own categorization in well known 3rd party exploit marketplaces.

Regardless of whether it is remote code execution within the WiFi SoC or within the cellular baseband, a common and resonating theme has been the consistent lack of exploit mitigations in firmware. Conveniently, Android has significant experience in enabling exploit mitigations across critical attack surfaces.

Applying years worth of lessons learned in systems hardening

Over the last few years, we have successfully enabled compiler-based mitigations in Android — on the AP — which add additional layers of defense across the platform, making it harder to build reproducible exploits and to prevent certain types of bugs from becoming vulnerabilities. Building on top of these successes and lessons learned, we’re applying the same principles to hardening the security of firmware that runs outside of Android per se, directly on the bare-metal hardware.

In particular, we are working with our ecosystem partners in several areas aimed at hardening the security of firmware that interacts with Android:

Bare-metal support

Compiler-based sanitizers have no runtime requirements in trapping mode, which provides a meaningful layer of protection we want: it causes the program to abort execution when detecting undefined behavior. As a result, memory corruption vulnerabilities that would otherwise be exploitable are now stopped entirely. To aid developers in testing, troubleshooting, and generating bug reports on debug builds, both minimal and full diagnostics modes can be enabled, which require defining and linking the requisite runtime handlers.

Most Control Flow Integrity (CFI) schemes also work for bare-metal targets in trapping mode. LLVM’s1 CFI across shared libraries scheme (cross-DSO) is the exception as it requires a runtime to be defined for the target. Shadow Call Stack, an AArch64-only feature, has a runtime component which initializes the shadow stack. LLVM does not provide this runtime for any target, so bare-metal users would need to define that runtime to use it.

The challenge

Enabling exploit mitigations in firmware running on bare metal targets is no easy feat. While the AP (Application Processor) hosts a powerful operating system (Linux) with comparatively abundant CPU and memory resources, bare metal targets are often severely resource-constrained, and are tuned to run a very specific set of functions. Any perturbation in compute and/or memory consumption introduced by enabling, for example, compiler-based sanitizers, could have a significant impact in functionality, performance, and stability.

Therefore, it is critical to optimize how and where exploit mitigations are turned on. The goal is to maximize impact — harden the most exposed attack surface — while minimizing any performance/stability impact. For example, in the case of the cellular baseband, we recommend focusing on code and libraries responsible for parsing messages delivered over the air (particularly for pre-authentication protocols such as RRC and NAS, which are the most exposed attack surface), libraries encoding/decoding complex formats (for example ASN.1), and libraries implementing IMS (IP Multimedia System) functionality, or parsing SMS and/or MMS.

Fuzzing and Vulnerability Rewards Program

Enabling exploit mitigations and compiler-based sanitizers are excellent techniques to minimize the chances of unknown bugs becoming exploitable. However, it is also important to continuously look for, find, and patch bugs.

Fuzzing continues to be a highly efficient method to find impactful bugs. It’s also been proven to be effective for signaling larger design issues in code. Our team partners closely with Android teams working on fuzzing and security assessments to leverage their expertise and tools with bare metal targets.

This collaboration also allowed us to scale fuzzing activities across Google by deploying central infrastructure that allows fuzzers to run in perpetuity. This is a high-value approach known as continuous fuzzing.

In parallel, we also accept and reward external contributions via our Vulnerability Rewards Program. Along with the launch of Android 13, we updated the severity guidelines to further highlight remotely exploitable bugs in connectivity firmware. We look forward to the contributions from the security research community to help us find and patch bugs in bare metal targets.

On the horizon

In Android 12 we announced support for Rust in the Android platform, and Android 13 is the first release with a majority of new code written in a memory safe language. We see a lot of potential in also leveraging memory-safe languages for bare metal targets, particularly for high risk and exposed attack surface.

Hardening firmware running on bare metal to materially increase the level of protection – across more surfaces in Android – is one of the priorities of Android Security. Moving forward, our goal is to expand the use of these mitigation technologies for more bare metal targets, and we strongly encourage our partners to do the same. We stand ready to assist our ecosystem partners to harden bare metal firmware.

Special thanks to our colleagues who contributed to this blog post and our firmware security hardening efforts: Diana Baker, Farzan Karimi, Jeffrey Vander Stoep, Kevin Deus, Eugene Rodionov, Pirama Arumuga Nainar, Sami Tolvanen, Stephen Hines, Xuan Xing, Yomna Nasser.

Notes


  1. LLVM – is a compiler framework used by multiple programming languages