mobile libraries Rusty Days 2020 - Jan-Erik / @badboy_ About me - - PowerPoint PPT Presentation

▶
mobile libraries
SMART_READER_LITE
LIVE PREVIEW

mobile libraries Rusty Days 2020 - Jan-Erik / @badboy_ About me - - PowerPoint PPT Presentation

Sep 23, 2022 237 likes •1.02k views

Leveraging Rust to build cross-platform mobile libraries Rusty Days 2020 - Jan-Erik / @badboy_ About me Firefox Telemetry engineer at Mozilla Rust Community Team member Scuba diver Twitter: @badboy_ Blog: fnordi g.de Slides:

slide-1
SLIDE 1

Leveraging Rust

to build cross-platform

mobile libraries

Rusty Days 2020 - Jan-Erik / @badboy_
slide-2
SLIDE 2

About me

  • Firefox Telemetry engineer at Mozilla
  • Rust Community Team member
  • Scuba diver

Twitter: @badboy_ Blog: fnordig.de Slides:

fnordig.de/talks/2020/rustydays/slides.pdf

slide-3
SLIDE 3
slide-4
SLIDE 4
  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
4
slide-5
SLIDE 5
  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
5
slide-6
SLIDE 6

Firefox Telemetry

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
6
slide-7
SLIDE 7

Firefox Telemetry

A quick overview

  • 1. Performance metrics for our

products

  • 2. Packaged in pings sent at

controlled schedules

  • 3. Following our Lean Data

Practices

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
7
slide-8
SLIDE 8

Metric: Time spent running the JS GC1

1 Measurement Dashboard at https://telemetry.mozilla.org/new-pipeline/dist.html
  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
8
slide-9
SLIDE 9

Lean Data Practices

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
9
slide-10
SLIDE 10

Collecting Data Responsibly and at Scale 2

2 StarCon 2019 Talk by chutten.
  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
10
slide-11
SLIDE 11
  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
11
slide-12
SLIDE 12
  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
12
slide-13
SLIDE 13

The Glean SDK is a modern approach for a Telemetry library and is part of the Glean project. Introducing Glean — Telemetry for humans by Georg Fritzsche.

Firefox for Android

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
13
slide-14
SLIDE 14

Telemetry API in Firefox Desktop

Services.telemetry.scalarAdd( "browser_engagement.max_concurrent_tab_count", 1 );

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
14
slide-15
SLIDE 15

Scalars.yaml

browser.engagement: max_concurrent_tab_count: bug_numbers:

  • 1271304

description: > The count of maximum number of tabs open during a subsession, across all windows, including tabs in private windows and restored at startup. expires: "81" kind: uint notification_emails:

  • someone@mozilla.com

release_channel_collection: opt-out products:

  • 'firefox'

record_in_processes:

  • 'main'
  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
15
slide-16
SLIDE 16

Telemetry API in Firefox Desktop

Services.telemetry.scalarAdd( "browser_engagement.max_concurrent_tab_count", 1 );

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
16
slide-17
SLIDE 17

BrowserEngagement.max_concurrent_tab_count.add(1)

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
17
slide-18
SLIDE 18

BrowserEngagement.max_concurrent_tab_count.add(1) |---------------| |--| Category object integer |------------------------| Counter metric |---| increment function

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
18
slide-19
SLIDE 19

New Telemetry requirements

  • Declarative definitions of metrics
  • Share core implementation cross-platform
  • Ergonomic API per target language
  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
19
slide-20
SLIDE 20

https://github.com/mozilla/glean

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
20
slide-21
SLIDE 21

Glean SDK stack

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
21
slide-22
SLIDE 22

Glean Core

A Rust crate

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
22
slide-23
SLIDE 23

Glean Core - a Rust crate

#[derive(Debug)] struct Glean { data_path: PathBuf, upload_enabled: bool, data_store: Database, event_data_store: EventDatabase, core_metrics: CoreMetrics, // ... }

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
23
slide-24
SLIDE 24

Metrics implemented as Rust types

struct CounterMetric { meta: CommonMetricData, } impl CounterMetric { fn add(&self, glean: &Glean, amount: i32) { glean .storage() .record_with(&self.meta, |old_value| old_value.add(amount)) } }

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
24
slide-25
SLIDE 25

Glean FFI

the connection

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
25
slide-26
SLIDE 26

Foreign Function Interface

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
26
slide-27
SLIDE 27

Calling C functions

extern { fn c_rusty_days(days: c_int); } // .. unsafe { c_rusty_days(30); }

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
27
slide-28
SLIDE 28

Getting called from C

#[no_mangle] pub extern "C" fn hello_rusty_days(data: c_int) -> *const u8 { "Rusty Days!\0".as_ptr() }

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
28
slide-29
SLIDE 29

"Rusty Days!\0"

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
29
slide-30
SLIDE 30

CString to the rescue

let s = CString::new("Rusty days!").unwrap(); assert!(s.into_bytes_with_nul() == b"Rusty days!\0");

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
30
slide-31
SLIDE 31

cbindgen

cbindgen creates C headers for Rust libraries which expose a public C API.

#[no_mangle] pub extern "C" fn hello_rusty_days(data: c_int)

  • > *const u8 {

"Rusty Days!\0".as_ptr() }

#include <stdint.h> #include <stdlib.h> const uint8_t *hello_rusty_days(int data);

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
31
slide-32
SLIDE 32

ffi-support

Support library to simplify implementing FFI libraries*.

* as done by application-services3 & Glean

3 https://github.com/mozilla/application-services
  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
32
slide-33
SLIDE 33

ffi-support: IntoFFI

Convert Rust types into FFI-compatible types

unsafe trait IntoFfi: Sized { type Value; fn ffi_default() -> Self::Value; fn into_ffi_value(self) -> Self::Value; } unsafe impl IntoFfi for String { type Value = *mut c_char; // ... }

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
33
slide-34
SLIDE 34

ffi-support: FfiStr

A safe wrapper around a null-terminated string.

pub struct FfiStr<'a> { /* fields omitted */ } #[no_mangle] extern "C" fn hello_rusty_days(data: FfiStr) { // Use of `data` after this function returns is impossible }

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
34
slide-35
SLIDE 35

ffi-support: ConcurrentHandleMap

A locked map with handles to use across the FFI.

static COUNTER: Lazy<ConcurrentHandleMap<CounterMetric>> = Lazy::new(ConcurrentHandleMap::new); extern "C" fn glean_new_counter_metric(name: ffi_support::FfiStr) -> u64 { COUNTER.insert_with_log(|| { Ok(glean_core::metrics::CounterMetric::new(name.as_str())) }) }

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
35
slide-36
SLIDE 36

Compile Targets

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
36
slide-37
SLIDE 37

$ rustup target list aarch64-apple-ios (installed) aarch64-fuchsia aarch64-linux-android (installed) aarch64-pc-windows-msvc aarch64-unknown-linux-gnu aarch64-unknown-linux-musl aarch64-unknown-none [...] x86_64-apple-darwin (installed) x86_64-apple-ios (installed) [...] x86_64-unknown-redox

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
37
slide-38
SLIDE 38

<arch><sub>-<vendor>-<sys>-<abi>

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
38
slide-39
SLIDE 39

Glean targets

rustup target add armv7-linux-androideabi # for arm rustup target add i686-linux-android # for x86 rustup target add aarch64-linux-android # for arm64 rustup target add x86_64-linux-android # for x86_64 (& simulator) rustup target add x86_64-unknown-linux-gnu # for linux-x86-64 rustup target add x86_64-apple-darwin # for macOS rustup target add x86_64-pc-windows-gnu # for win32-x86-64-gnu rustup target add x86_64-pc-windows-msvc # for win32-x86-64-msvc rustup target add aarch64-apple-ios # iOS (actual devices) rustup target add x86_64-apple-ios # iOS simulator

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
39
slide-40
SLIDE 40

Glean Kotlin

The Kotlin implementation

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
40
slide-41
SLIDE 41

JNI is the Java Native Interface. It defines a way for the bytecode that Android compiles from managed code to interact with native code.

— from Android - JNI tips

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
41
slide-42
SLIDE 42

Hello World with JNI

Rust

#[no_mangle] extern "system" fn Java_HelloWorld_hello( env: JNIEnv, _class: JClass, input: JString, ) -> jstring { // ... }

Java

class HelloWorld { static native String hello(String input); static { System.loadLibrary("mylib"); } }

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
42
slide-43
SLIDE 43

Otavio Pace: Interop with Android, IOS and WASM in the same project

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
43
slide-44
SLIDE 44

JNA provides Java programs easy access to native shared libraries without writing anything but Java code - no JNI or native code is required.

— from github.com/java-native-access/jna

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
44
slide-45
SLIDE 45

Hello World with JNA

#[no_mangle] pub extern "C" fn hello() -> *const c_char { "Rusty Days!\0".as_ptr() }

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
45
slide-46
SLIDE 46

JNA - loading a dynamic library

internal interface LibGleanFFI : Library { companion object { internal var INSTANCE: LibGleanFFI = Native.load("glean_ffi", LibGleanFFI::class.java) } fun glean_initialize(cfg: FfiConfiguration): Byte fun glean_new_counter_metric(name: String, lifetime: Int): Long } // ... LibGleanFFI.INSTANCE.glean_initialize(cfg)

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
46
slide-47
SLIDE 47

Building a Cargo project using Gradle4

apply plugin: 'org.mozilla.rust-android-gradle.rust-android' cargo { module = "../ffi" libname = "glean_ffi" targets = ["arm", "x86"] }

4 github.com/mozilla/rust-android-gradle
  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
47
slide-48
SLIDE 48

Other Glean implementations

  • Swift - it speaks C!
  • Python - similar to Swift, using cffi
  • C# - similar to Kotlin
  • Soon:
  • C++
  • JavaScript
  • Rust
  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
48
slide-49
SLIDE 49

Challenges of developing cross- platform Rust

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
49
slide-50
SLIDE 50

Data types

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
50
slide-51
SLIDE 51

Data types: Numbers

Rust Kotlin Swift u8 Byte UInt8 i32 Int Int32 i64 Long Int64 isize IntegerType Int

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
51
slide-52
SLIDE 52

Data types: Bool in reality

1 # true 0 # false 1 bit

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
52
slide-53
SLIDE 53

Data types: Bool in Rust

0000 0001 # true 0000 0000 # false 8 bit = 1 byte

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
53
slide-54
SLIDE 54

Data types: Bool in Kotlin

0000 0000 0000 0000 0000 0000 0000 0001 # true 0000 0000 0000 0000 0000 0000 0000 0000 # false 32 bit = 4 byte

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
54
slide-55
SLIDE 55

Data types: Strings

fun glean_submit_ping_by_name( ping_name: String, reason: String? ): Byte

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
55
slide-56
SLIDE 56

Data types: Strings

// Rust extern "C" fn glean_string_get(metric_id: i64) -> *mut c_char // Kotlin fun glean_string_get(metric_id: Long): Pointer?

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
56
slide-57
SLIDE 57

Data types: String

fun Pointer.getAndConsumeRustString(): String { try { return this.getRustString() } finally { LibGleanFFI.INSTANCE.glean_str_free(this) } } fun Pointer.getRustString(): String { return this.getString(0, "utf8") }

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
57
slide-58
SLIDE 58

Data types: Plain ol' Enums

Kotlin

enum class TimeUnit { Microsecond, Millisecond, Second, Minute, }

Rust

enum TimeUnit { Microsecond, Millisecond, Second, Minute, }

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
58
slide-59
SLIDE 59

Data types: Enums with data

Rust

#[repr(u8)] pub enum FfiPingUploadTask { Upload { document_id: *mut c_char, body: ByteBuffer, }, Wait, Done, }

Tagged unions in C (through C bindgen)

enum FfiPingUploadTask_Tag { FfiPingUploadTask_Upload, FfiPingUploadTask_Wait, FfiPingUploadTask_Done, }; typedef uint8_t FfiPingUploadTask_Tag; typedef struct { FfiPingUploadTask_Tag tag; char *document_id; ByteBuffer body; } FfiPingUploadTask_Upload_Body; typedef union { FfiPingUploadTask_Tag tag; FfiPingUploadTask_Upload_Body upload; } FfiPingUploadTask;

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
59
slide-60
SLIDE 60

Data types: Enums with data

Rust

#[repr(u8)] pub enum FfiPingUploadTask { Upload { document_id: *mut c_char, body: ByteBuffer, }, Wait, Done, }

Tagged unions in Kotlin

enum class UploadTaskTag { Upload, Wait, Done } @Structure.FieldOrder("tag", "documentId", "body") internal class UploadBody( val tag: Byte, val documentId: Pointer?, var body: RustBuffer, ) : Structure() { } internal open class FfiPingUploadTask( var tag: Byte = UploadTaskTag.Done.ordinal.toByte(), var upload: UploadBody = UploadBody() ) : Union() { }

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
60
slide-61
SLIDE 61

Data types: Other rich data - JSON

Rust

extern "C" fn glean_get_json(metric_id: i64)

  • > *mut c_char {

let data = Glean.string_get(metric_id); let json = serde_json::to_string(&data); json.into_ffi_value() }

Kotlin

// Declaration: fun glean_get_json(metric_id: Long): Pointer? // Usage: val ptr = glean_get_json(handle)!! jsonRes = JSONArray(ptr.getAndConsumeRustString()) return jsonRes.toList()

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
61
slide-62
SLIDE 62

Data types: Other rich data - ProtoBuf5

5 Crossing the Rust FFI frontier with Protocol Buffers
  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
62
slide-63
SLIDE 63

Optimizer: R8

  • R8 minifies and optimizes the JVM bytecode
  • It's buggy and might "over-optimize" JNA code
  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
63
slide-64
SLIDE 64

Optimizer: R8

proguard-consumer-rules.pro:

# JNA specific rules

  • dontwarn java.awt.*
  • keep class com.sun.jna.* { *; }
  • keepclassmembers class * extends com.sun.jna.* { public *; }

# Glean specific rules

  • keep class mozilla.telemetry.** { *; }
  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
64
slide-65
SLIDE 65

Extra libs

  • build.rs exists, but everyone does it differently
  • Build and link your C dependencies statically
  • Consider precompiling them
  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
65
slide-66
SLIDE 66

What about the platform?

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
66
slide-67
SLIDE 67

Glean goes on-direction only

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
67
slide-68
SLIDE 68

Things that Kotlin does:

  • Data storage path
  • System & app information
  • HTTP/network communication
  • Time
  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
68
slide-69
SLIDE 69

The Future is Glean

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
69
slide-70
SLIDE 70

Future: Component Interface Definition

uniffi: Create boilerplate from IDL files6

interface Counter { constructor(string category, string name); void add(integer amount); }

6 github.com/mozilla/uniffi-rs
  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
70
slide-71
SLIDE 71

Future: Firefox on Glean

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
71
slide-72
SLIDE 72

Who else is using Rust to build cross-platform libraries, targetting mobile?

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
72
slide-73
SLIDE 73

Async Rust, but using the platform?

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
73
slide-74
SLIDE 74

Thanks to

the Telemetry team: Alessio, Bea, Chris, Travis, Mike and Georg. the application-services team.

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
74
slide-75
SLIDE 75

Links

  • Slides: fnordig.de/talks/2020/rustydays/slides.pdf
  • Glean SDK repository: github.com/mozilla/glean
  • Glean SDK docs: mozilla.github.io/glean
  • Mozilla Data blog: blog.mozilla.org/data
  • me on Twitter: @badboy_
  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
75
slide-76
SLIDE 76

Questions?

  • Cross-platform mobile libraries - Rusty Days 2020 - Jan-Erik / @badboy_
76