Hugo Peixoto

Building an android app with free software only

Published on May 08, 2020

Table of contents

I built a couple of personal android applications a few years ago. One of them is an overlay thing for pokemon go and the other one is an image uploader for dl.hugopeixoto.net, my personal image hosting service. I would like to be able to build them without having to accept EULAs.

Current state

I have a half-uncommited private repository for my image uploader app. I don’t remember why I didn’t publish this. Probably because I wanted to clean the repository before publishing it, classic mistake.

The app has two parts (from what I recall):

Let’s look at the tree of this directory to see what’s up:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

hugopeixoto@laptop:~/w/p/http-share-android$ tree -L 3
.
├── android.iml
├── app
│   ├── app.iml
│   ├── build
│   │   ├── generated
│   │   ├── intermediates
│   │   ├── outputs
│   │   ├── retrolambda
│   │   └── tmp
│   ├── build.gradle
│   └── src
│       └── main
├── app-signed.apk
├── build
│   ├── generated
│   │   └── mockable-android-23.jar
│   └── intermediates
│       ├── dex-cache
│       └── lint-cache
├── build.gradle
├── gradle
│   └── wrapper
│       ├── gradle-wrapper.jar
│       └── gradle-wrapper.properties
├── gradlew
├── gradlew.bat
├── local.properties
└── settings.gradle

/build/ and /app/build/, that sounds promising. I must have been in the process of messing with the build process. I’ll remove both these directories.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

hugopeixoto@laptop:~/w/p/http-share-android$ tree -I "drawable-*|mipmap-*"
.
├── android.iml
├── app
│   ├── app.iml
│   ├── build.gradle
│   └── src
│       └── main
│           ├── AndroidManifest.xml
│           ├── java
│           │   └── net
│           │       └── hugopeixoto
│           │           └── customsharing
│           │               ├── AppCompatPreferenceActivity.java
│           │               ├── RxUtils.java
│           │               ├── SettingsActivity.java
│           │               ├── UploadActivity.java
│           │               └── Uploader.java
│           └── res
│               ├── layout
│               │   ├── activity_main.xml
│               │   └── content_main.xml
│               ├── values
│               │   ├── colors.xml
│               │   ├── dimens.xml
│               │   ├── strings.xml
│               │   └── styles.xml
│               ├── values-v21
│               │   └── styles.xml
│               ├── values-w820dp
│               │   └── dimens.xml
│               └── xml
│                   └── preferences.xml
├── app-signed.apk
├── build.gradle
├── gradle
│   └── wrapper
│       ├── gradle-wrapper.jar
│       └── gradle-wrapper.properties
├── gradlew
├── gradlew.bat
├── local.properties
└── settings.gradle

15 directories, 26 files

I ignored app/src/res/{drawable,mipmap}-* because it contains a bunch of icons in different resolutions that I don’t want to think about just yet.

So, what have we got?

Looking at the source files:

1
2
3
4
5
6
7
8
9

hugopeixoto@laptop:~/w/p/http-share-android$ tree app/src/main/java/net/hugopeixoto/customsharing/
app/src/main/java/net/hugopeixoto/customsharing/
├── AppCompatPreferenceActivity.java
├── RxUtils.java
├── SettingsActivity.java
├── UploadActivity.java
└── Uploader.java

0 directories, 5 files

I see the three activity files, AppCompatPreferenceActivity, SettingsActivity and UploadActivity. The last two match the parts I mentioned at the beginning. SettingsActivity inherits from AppCompatPreferenceActivity, so it could be related to some android compatibility wrapper.

RxUtils contains a single static method, but it lets me know that I’m using some functional library. I recall playing around with lambdas, and having to use a transpiler of sorts to get it to work. Retrolambda. Since I don’t really care about compatibility with old devices for my personal apps, maybe I don’t need to use this anymore.

I should have left myself a README with instructions. I remember that whenever I had to build this, I had to run some commands. One for building, one for signing. I don’t have those written anywhere, except in my bash history. I’ll go fetch them.

Found this:

1
2
3

$ gradle assembleRelease
$ jarsigner -sigalg SHA1withRSA -digestalg SHA1 -keystore my.keystore ./app/build/outputs/apk/app-release-unsigned.apk app
$ http-share httpshare.apk ./app/build/outputs/apk/app-release-unsigned.apk

The first command builds the apk, the second one signs it, and the fourth one uploads the signed version to a web server, using http-share-cli.

I know that I wanted to build this from a docker image, instead of relying on having everything installed on the OS. I also remember that that didn’t work properly with openjdk, so I stopped there. I still have a Dockerfile somewhere.

1
2
3
4
5
6
7
8
9

FROM debian
#alpine:3.6

RUN apt update
RUN apt install -y openjdk-8-jdk unzip  wget
RUN wget https://dl.google.com/android/repository/sdk-tools-linux-3859397.zip -O android.zip && mkdir android && unzip android.zip -d android/ && rm -r android.zip
#RUN echo y | android/tools/android update sdk --no-ui -a --filter tools,platform-tools,extra-android-m2repository,extra-android-support,extra-google-google_play_services,extra-google-m2repository,android-23,build-tools-25.0.2,build-tools-23.0.1,build-tools-25.0.0
#
#ENV PATH ${PATH}:/android/tools/:/android/tools/bin/:/android/platform-tools/

This looks very much like work in progress. I’ll start over.

F-droid

I use F-droid. They’re focused on free software, and they build everything from source. It might be a good starting point.

Sidenote: I saw a talk about OpenPush in FOSDEM, I need to see if there are any news

The following page contains some interesting notes:

https://f-droid.org/en/2020/04/29/big-website-update.html

all APK files must be signed to be installable on Android

it is not possible to upgrade it in place to a new version signed with a different key without first uninstalling the original. This may present an inconvenience to users, as the process of uninstalling loses any data associated with the previous installation.

Run fdroid build to build any applications that are not already built.

There seems to be a fdroid cli that handles building

More about fdroid build

Yes please.

To build a single version of a single application, you could run the following: fdroid build org.fdroid.fdroid:16

Another option for using fdroid build is to use a metadata file that is included in the app’s source itself, rather than in a metadata/ folder with lots of other apps. The .fdroid.yml metadata file should be in the root of your source repo.

Now, how does this fdroid build work, exactly, and how do I get my app compiled using this? Let’s see if the FAQ helps.

https://f-droid.org/en/docs/FAQ_-_App_Developers/

The quickest way to get an app included is to make a merge request to fdroiddata, following these instructions.

We also support reproducible builds, so we can build a version from source and check against your official release. If they match (ignoring the signature) we can then publish your official APK with your signature used. This is a tedious task, since we have to standardize on the build parameters and tools, but it should be worth it in the long run. We also try to verify our own builds and get a lot of binary differences, see our verification server results. However, things will improve over time.

Reproducible builds is an interesting topic, but I’m probably not going to tackle this right now.

Which build system to use?

We have good support for “ant” and “gradle” based builds, while “maven” was only used for a short period of time and for dependencies. For other build systems, you might have to provide us some detailed information on how to handle that, so we can setup the app correctly or maybe even incorporate them into our server tools.

So this means that I’ll probably continue to keep using gradle.

Let’s look at the CONTRIBUTING.md, referenced above.

OK, this seems to describe how to create the metadata file.

I also found a docker image for the fdroid server:

https://gitlab.com/fdroid/docker-executable-fdroidserver

But it mentions that:

It does not include a full Android SDK, so if you want to use the full Android SDK with this image, you’ll need to mount your own local copy.

This smells of licensing issues. Let’s look for other docs that might point out to how this whole thing works.

https://f-droid.org/en/docs/Installing_the_Server_and_Repo_Tools

Proprietary, Non-Free libraries

The Android SDK is made available by Google under a proprietary license. Within that, the essential build tools, SDK platforms, support library and some other components are under the Apache license and source code is provided.

So, the Android SDK is proprietary? How can this be, if AOSP is free? Doesn’t make much sense. Let me explore this a bit.

Android SDK licensing

There’s a debian package package called android-sdk:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

hugopeixoto@hylia:/m/h/h/hugopeixoto$ apt show android-sdk
Package: android-sdk
Version: 25.0.0+12
Priority: optional
Section: metapackages
Source: android-sdk-meta
Maintainer: Android Tools Maintainers <android-tools-devel@lists.alioth.debian.org>
Installed-Size: 30.7 kB
Depends: android-sdk-build-tools, android-sdk-common (>= 25.0.0+12), android-sdk-platform-tools (>= 20), proguard-cli
Recommends: gradle, default-jdk-headless
Suggests: android-sdk-platform-23, maven, proguard-gui
Download-Size: 4824 B
APT-Sources: http://deb.debian.org/debian testing/main amd64 Packages
Description: Software development kit for Android platform
 The Android SDK includes a variety of tools that help you develop mobile
 applications for the Android platform. The tools are classified into 3 groups:
 SDK Tools, Platform-tools and Build-tools.
 .
 SDK Tools are platform independent and are required no matter which Android
 platform you are developing on. It is the base toolset of Android SDK.
 .
 This metapackage pulls the entire Android SDK.

So this in debian, in the main section, so how could this be proprietary? It could just be a downloader for the proprietary binaries, but then it would have to be in contrib, not in main. Let me check Debian’s wiki:

https://wiki.debian.org/AndroidTools

Sidenote: the wiki page points to https://guardianproject.info/2015/04/30/getting-android-tools-into-debian/, which seems to be an interesting read.

The binaries for the Android SDK downloadable from Google have a proprietary license but the source code is free software so Debian is packaging it. Not all Android SDK packages can be installed from Debian, some never will be in Debian because they are too specific to Android. Sylvain Beucler’s libre Android rebuilds and/or Google’s non-free binaries can also be used with the Debian Android SDK.

OK, so the binaries are proprietary, but the source is not. This makes some sense. So going back to the fdroid quote:

The Android SDK is made available by Google under a proprietary license. Within that, the essential build tools, SDK platforms, support library and some other components are under the Apache license and source code is provided.

When they say that the SDK is made available under a proprietary license, that means the binaries. Makes sense, since they follow up with “x, y, z are under the Apache license”. So that is just weird phrasing / lack of reading.

So this means that ideally, I’d be installing these from source, and not using Google’s binaries.

This also means that there are some packages that are installable via google’s android sdk thingy that are proprietary. This includes Google services. No surprises here, this is something I was aware of.

Let me check that Sylvain Beucler’s libre Android rebuilds about page page.

Android Studio, which includes the SDK and NDK, is supposed to be freely licensed (mainly Apache 2.0) but the official binaries aren’t.

On download, we are requested to accept conditions such as:

“3.2 You may not use this SDK to develop applications for other platforms (including non-compatible implementations of Android) or to develop another SDK.”

“4.2 You agree to use the SDK and write applications only for purposes that are permitted by (a) the License Agreement and (b) any applicable law, regulation or [etc.]”

This is all great.

I also saw some references to micro-g while searching. It’s a free implementation of some google libraries that are widely used. Let me just browse a list or something to get acquainted with it, in case I need it:

Service Core (GmsCore) is a library app, providing the functionality required to run apps that use Google Play Services or Google Maps Android API (v2).

Services Framework Proxy (GsfProxy) is a small helper utility to allow apps developed for Google Cloud to Device Messaging (C2DM) to use the compatible Google Cloud Messaging service included with GmsCore.

Unified Network Location Provider (UnifiedNlp) is a library that provides Wi-Fi- and Cell-tower-based geolocation to applications that use Google’s network location provider. It is included in GmsCore but can also run independently on most Android systems.

[some deprecated maps api replacement]

Store (Phonesky) is a frontend application providing access to the Google Play Store to download and update applications. Development is in early stage and there is no usable application yet.

OK, so they’re basically free implementations of interfaces to proprietary services. I don’t see myself needing these for now, but the UnifiedNlp might be useful in the future. I may be using Google Play Services accidentally, so this is good to know.

I hope I’m not accidentally using any proprietary libraries, but the compat libraries might fall into that category.

So, back to building an application. If debian packages android-sdk, let’s try to build a docker file using those.

Sidenote: I randomly learned that the F-Droid client started out as a fork of Aptoide.

Debian based Dockerfile for android builds

I don’t even know where to start. Let’s go back to the Debian wiki page:

https://wiki.debian.org/AndroidTools

There are many advantages to having the SDK and tools in Debian, rather than relying only on the Google distributions:

Point 1, 2, and 3 sound great. I kinda dislike having random gradlew files in my repository.

If you’re just starting out with building apps, we suggest that you first read the Introduction to build packages with Debian’s Android SDK.

The linked page also contains a walkthrough. Might come in handy.

Currently there is only the target platform of API Level 23 packaged, so only apps targeted at android-23 can be built with only Debian packages. We will add more API platform packages via backports afterwards. Only Build-Tools 24.0.0 is available, so in order to use the SDK, build scripts need to be modified. Beware that the Lint in this version of Gradle Android Plugin is still problematic, so running the :lint tasks might not work.

This is going to be interesting. Let me check what version I was using.

1
2
3
4

hugopeixoto@laptop:~/w/p/http-share-android$ grep 'android {' app/build.gradle -A2
android {
    compileSdkVersion 23
    buildToolsVersion "23.0.1"

Two 23s. That means I’ll probably have to change buildTools to 24. This is, assuming that the wiki information is still up to date. Let’s double check.

1
2
3
4
5
6
7
8
9
10
11

hugopeixoto@laptop:~/w/p/http-share-android$ apt search -q --names-only android-sdk-platform
Sorting...
Full Text Search...
android-sdk-platform-23/testing,unstable 6.0.1+r72-5 all
  Android SDK Platform for API Level 23 (6.0 Marshmallow)

android-sdk-platform-tools/testing,unstable 27.0.0+12 amd64
  Tools for interacting with an Android platform

android-sdk-platform-tools-common/testing,unstable,now 27.0.0+12 all [installed,automatic]
  Tools for interacting with an Android platform - Common files

Confirmed, 23. the -tools packages mention a version 27, though. Let me check what are the most recent versions in upstream.

https://developer.android.com/studio/releases/platforms

Android 10 (API level 29) … Android 8.1 (API level 27) … Android 6.0 (API level 23)

So this is kind of lagging behind. I have no idea what any of these mean, and from https://wiki.debian.org/AndroidTools#Android.27s_upstream_version_names, it seems to be a bit crazy to begin with. I’ll get back to it later.

Let’s start writing the Dockerfile.

First hiccup: I had to use debian:sid. This is what I have so far:

1
2
3
4
5
6
7
8
9

FROM debian:sid

RUN apt update
RUN apt -y install android-sdk android-sdk-platform-23 git libgradle-android-plugin-java

ADD . /app
WORKDIR /app

CMD bash

I’m using CMD bash because I’ll be exploring things through bash before fully automating this.

1

$ docker build . -t http-share-android && docker run -ti http-share-android

Running gradle build downloaded a bunch of POM files and then it complained about mismatched gradle versions:

Gradle version 2.10 is required. Current version is 4.4.1. If using the gradle wrapper, try editing the distributionUrl in /root/.gradle/daemon/4.4.1/gradle/wrapper/gradle-wrapper.properties to gradle-2.10-all.zip

This is because I need to adapt build.gradle to use the debian packages instead of the android ones. I’m having some trouble getting that to work. Instructions may be out of date. There is an android-sdk-helper package that patches existing gradle files, but I rather write them manually for now to understand what’s going on.

They’re telling me to change build.gradle to include:

1
2
3
4
5
6
7
8
9

buildscript {
  repositories {
    maven { url 'file:///usr/share/maven-repo' }
  }

  dependencies {
    compile 'com.android.tools.build:gradle:debian'
  }
}

But the compile line doesn’t… compile. Where is this compile thing defined?

https://stackoverflow.com/questions/34286407/gradle-what-is-the-difference-between-classpath-and-compile-dependencies

I’m going to guess that you’re referencing compile and classpath within the dependencies {} block. If that is so, those are dependency Configurations.

The compile configuration is created by the Java plugin. The classpath configuration is commonly seen in the buildSrc {} block where one needs to declare dependencies for the build.gradle, itself (for plugins, perhaps).

Checking gradle docs:

https://docs.gradle.org/current/userguide/java_library_plugin.html#sec:java_library_configurations_graph

I see a compileClasspath and compileOnly, but no compile. This may have changed in gradle. Let me check versions.

Debian has 4.4.1 while the latest gradle is… 6.x. 4.4.1 is from 2017, ouch. Let me check debian’s tracker to see if there’s anything that points to a new version being worked on:

https://tracker.debian.org/pkg/gradle

Checking all bugs associated with this package, here’s a relevant one:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=933264

[29 Jul 2019] Newer version of Gradle requires Kotlin, which is being worked on right now. See https://java-team.pages.debian.net/gsoc-kotlin-blog

So gradle is stuck on 4.4.1 because it depends on Kotlin, and Kotlin is still not packaged in Debian. And one of the reasons kotlin is not packaged, I bet, is because it requires a more recent version of gradle.

Downgrade the build from gradle 5.1 to gradle 4.4.1 which is the version available in Debian Sid.

I knew it. I wanted to eventually move the apps from kotlin to java, but I guess that’s going to have to wait.

I have finished downgrading the project to be buildable using gradle 4.4.1. The project still needed a part of gradle 4.8 that I have successfully patched into sid gradle. here is the link to the changes that I have made.

So, Debian’s gradle 4.4.1 contains some features of 4.8. The docs for 4.8 are still available on gradle.org, so let’s check that.

No compile there. Maybe they meant classpath. Let me try that, and let me change the build tools version to 27 while I’m at it.

I got some other errors:

java.lang.ClassNotFoundException: org.debian.gradle.plugin.MavenResolverHook

java.io.FileNotFoundException: /root/.android/analytics.settings (No such file or directory)

These errors didn’t seem to stop the compilation from going. It then failed with

The SDK directory ‘/home/hugopeixoto/work/contrib/android-sdk’ does not exist.

This is my old android sdk path. I probably have to change something to the debian one. Changed local.properties:

1

sdk.dir=/usr/lib/android-sdk

Still getting a ton of exceptions, but it’s moving still along… and it failed after 33 seconds:

1
2
3

* What went wrong:
A problem occurred configuring project ':app'.
> java.lang.NullPointerException (no error message)

I created an empty analytics.settings file just to get those warnings out of the way, but the NPE persists. Running it again with --stacktrace:

1
2
3
4
5
6
7
8
9
10

org.gradle.api.ProjectConfigurationException: A problem occurred configuring project ':app'.
  at org.gradle.configuration.project.LifecycleProjectEvaluator.addConfigurationFailure(LifecycleProjectEvaluator.java:94)
  ...
Caused by: java.lang.NullPointerException
  at com.android.repository.impl.meta.SchemaModuleUtil.marshal(SchemaModuleUtil.java:270)
  at com.android.repository.impl.manager.LocalRepoLoaderImpl.writePackage(LocalRepoLoaderImpl.java:285)
  at com.android.repository.impl.manager.LocalRepoLoaderImpl.parsePackages(LocalRepoLoaderImpl.java:178)
  at com.android.repository.impl.manager.LocalRepoLoaderImpl.getPackages(LocalRepoLoaderImpl.java:154)
  at com.android.repository.impl.manager.RepoManagerImpl$LoadTask.run(RepoManagerImpl.java:653)
  at com.android.repository.api.RepoManager$DummyProgressRunner.runSyncWithProgress(RepoManager.java:398)

Ah, this is a bit more helpful. Let me quickly search for the source:

https://github.com/gradle/gradle/blob/v4.4.1/subprojects/core/src/main/java/org/gradle/configuration/project/LifecycleProjectEvaluator.java

https://github.com/JetBrains/adt-tools-base/blob/master/repository/src/main/java/com/android/repository/impl/meta/SchemaModuleUtil.java

And some stack overflow links that I found along the way:

https://stackoverflow.com/questions/30643802/what-is-jaxbcontext-newinstancestring-contextpath

https://stackoverflow.com/questions/43574426/how-to-resolve-java-lang-noclassdeffounderror-javax-xml-bind-jaxbexception-in-j

OK, the source isn’t being that helpful. Let me try to delete most of the contents in app/build.gradle to see if it changes something. This is the current state:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

plugins {
    id "me.tatarka.retrolambda" version "3.2.5"
}

apply plugin: 'com.android.application'

android {
    compileSdkVersion 23
    buildToolsVersion "27.0.1"

    defaultConfig {
        applicationId "net.hugopeixoto.customsharing"
        minSdkVersion 15
        targetSdkVersion 23
        versionCode 1
        versionName "1.0"
    }

    buildTypes {
        release {
            minifyEnabled false
        }
    }

    compileOptions {
        sourceCompatibility JavaVersion.VERSION_1_8
        targetCompatibility JavaVersion.VERSION_1_8
    }
}


dependencies {
    compile fileTree(dir: 'libs', include: ['*.jar'])
    testCompile 'junit:junit:4.12'
    compile 'com.android.support:appcompat-v7:23.1.1'
    compile 'com.android.support:design:23.1.1'
    compile 'com.squareup.okhttp3:okhttp:3.1.2'
    compile 'com.google.code.gson:gson:2.2.+'
    compile 'com.google.guava:guava:12.0'
    compile 'com.android.support:support-v4:23.1.1'
    compile 'io.reactivex:rxandroid:1.2.0'
    compile 'io.reactivex:rxjava:1.1.5'
}

I can get different errors if I delete the android section. It starts complaining about the lack of buildToolsVersion, but if I just add that line, it NPEs again.

Could this be related to the java version I’m running?

Let me find a sample app. I was looking through the bugs on android-sdk-meta, and came across this:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=854483

I tested it using https://gitlab.com/cde/AndroidHelloWorld, as seamlik proposed in #debian-mobile.

Let me clone this and try to build it.

1
2
3
4
5
6
7
8

hugopeixoto@laptop:~/w/p/http-share-android$ git clone https://gitlab.com/cde/AndroidHelloWorld
hugopeixoto@laptop:~/w/p/http-share-android$ docker build . -t http-share-android && docker run -ti http-share-android
root@56951ec38e98:/app# cd AndroidHelloWorld/
root@56951ec38e98:/app/AndroidHelloWorld# gradle build
> Gradle version 2.10 is required. Current version is 4.4.1. If using the
> gradle wrapper, try editing the distributionUrl in
> /root/.gradle/daemon/4.4.1/gradle/wrapper/gradle-wrapper.properties to
> gradle-2.10-all.zip

Derp. After adapting build.gradle according to the debian instructions, I got the same NPE error. Let me try the android-sdk-helper package.

1
2

root@56951ec38e98:/app/AndroidHelloWorld# apt install android-sdk-helper
root@56951ec38e98:/app/AndroidHelloWorld# gradle build --init-script /usr/share/android-sdk-helper/init.gradle

It was unable to detect ANDROID_HOME, so I set it through sdk.dir in local.properties.

1
2

cp ../local.properties .
root@56951ec38e98:/app/AndroidHelloWorld# gradle build --init-script /usr/share/android-sdk-helper/init.gradle

Same NPE as before. So it seems that I’m not doing anything wrong when editing build.gradle. I could try to build these dependencies with some extra debug statements.

Trying to build android-platform-tools-base

After looking through the wrong package, I ran the following commands:

1
2
3
4
5
6
7
8

hugopeixoto@laptop:~/w/contrib$ apt source libgradle-android-plugin-java
hugopeixoto@laptop:~/w/contrib$ cd android-platform-tools-base-2.2.2/
hugopeixoto@laptop:~/w/c/android-platform-tools-base-2.2.2$ ack SchemaModuleUtil
repository/src/main/java/com/android/repository/impl/manager/LocalRepoLoaderImpl.java
31:import com.android.repository.impl.meta.SchemaModuleUtil;
285:            SchemaModuleUtil.marshal(factory.generateRepository(repo),
311:            repo = (Repository) SchemaModuleUtil.unmarshal(mFop.newFileInputStream(packageXml),
[...]

I got a bunch of matches, but the first one matches the stacktrace and line number, so it sounds like a good starting point. Cross-referencing that with the JetBrains link.

Can I build this? I don’t remember anything of building debian packages. Running dpkg-buildpackage -us -uc tells me I need a lot of build dependencies. There’s an apt build-dep command.

1
2
3
4
5
6

root@laptop:~# apt build-dep android-platform-tools-base
hugopeixoto@laptop:~/w/c/android-platform-tools-base-2.2.2$ dpkg-buildpackage -us -uc
[...]
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':base:common:compileJava'.
[...]
Caused by: org.gradle.api.internal.tasks.compile.CompilationFailedException: Compilation failed; see the compiler error output for details.

Great. I joined #debian-mobile to see if I can get some help while I try to fix this.

Looking through the output of dpkg-buildpackage -us -uc, I found this error:

1
2
3
4
5
6
7
8
9
10
11
12
13

[...]/common/src/main/java/com/android/xml/AndroidXPathFactory.java:40:
error: AndroidNamespaceContext is not abstract and does not override abstract method getPrefixes(String) in NamespaceContext
    private static class AndroidNamespaceContext implements NamespaceContext {
                   ^
[...]/common/src/main/java/com/android/xml/AndroidXPathFactory.java:84:
error: getPrefixes(String) in AndroidNamespaceContext cannot implement getPrefixes(String) in NamespaceContext
        public Iterator<?> getPrefixes(String namespaceURI) {
                           ^
  return type Iterator<?> is not compatible with Iterator<String>
[...]/common/src/main/java/com/android/xml/AndroidXPathFactory.java:83:
error: method does not override or implement a method from a supertype
        @Override
        ^

So it seems like the inheritance thing is mismatching on the return type. Let me swap Iterator<?> with Iterator<String>.

OK, this unblocked the compilation, but now I’m getting a ton of errors. Some examples:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

[...]/repository/src/main/java/com/android/repository/api/RepoPackage.java:28:
error: package javax.xml.bind.annotation does not exist
import javax.xml.bind.annotation.XmlTransient;
                                ^
[...]/repository/src/main/java/com/android/repository/api/RepoPackage.java:34:
error: cannot find symbol
@XmlTransient
 ^
  symbol: class XmlTransient
[...]/repository/src/main/java/com/android/repository/impl/meta/TypeDetails.java:21:
error: package javax.xml.bind does not exist
import javax.xml.bind.JAXBElement;
                     ^
[...]/repository/src/main/java/com/android/repository/impl/meta/TypeDetails.java:22:
error: package javax.xml.bind.annotation does not exist
import javax.xml.bind.annotation.XmlTransient;
                                ^
[...]/repository/src/main/java/com/android/repository/impl/meta/TypeDetails.java:35:
error: cannot find symbol
@XmlTransient
 ^
  symbol: class XmlTransient

This is ringing bells. Earlier, I saw this stackoverflow question:

https://stackoverflow.com/questions/43574426/how-to-resolve-java-lang-noclassdeffounderror-javax-xml-bind-jaxbexception-in-j

The JAXB APIs are considered to be Java EE APIs and therefore are no longer contained on the default classpath in Java SE 9. In Java 11, they are completely removed from the JDK.

The extra Java EE APIs are provided in the following modules:

So there’s java.xml.bind, which was completely removed in Java 11. This is another indicator that I should probably be compiling this with an older version of java. Let me find any other evidence that point to the usage of a specific java version.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

hugopeixoto@laptop:~/w/c/android-platform-tools-base-2.2.2$ ack -i "java 11"
hugopeixoto@laptop:~/w/c/android-platform-tools-base-2.2.2$ ack -i "java 10"
hugopeixoto@laptop:~/w/c/android-platform-tools-base-2.2.2$ ack -i "openjdk"
debian/additionalSrc/intellij-community/platform/util/src/com/intellij/Patches.java
75:   * https://bugs.openjdk.java.net/browse/JDK-8020443
89:   * See https://bugs.openjdk.java.net/browse/JDK-7179799.
95:   * See http://bugs.openjdk.java.net/browse/JDK-8007219
122:   * See <a href="https://bugs.openjdk.java.net/browse/JDK-8042123">JDK-8042123</a>
129:   * Corresponding JDK request for enhancement - <a href="https://bugs.openjdk.java.net/browse/JDK-8139151">JDK-8139151</a>.
135:   * The issue was fixed in JDK 1.8.0_60 as part of <a href="https://bugs.openjdk.java.net/browse/JDK-8064833">JDK-8064833</a>.
146:   * See https://bugs.openjdk.java.net/browse/JDK-8004103.
hugopeixoto@laptop:~/w/c/android-platform-tools-base-2.2.2$ ack -i "java 8"
[~20 matches, including:]

debian/changelog
19:    * Depends on JDK8 since the code uses Java 8 Streams and lambda.

.idea/inspectionProfiles/Java_8.xml
3:    <option name="myName" value="Java 8" />

Searching for 1_8 also yields a bunch of matches.

The stackoverflow answer also mentions this:

For Gradle or Android Studio developer: (JDK 9 and beyond)

Add the following dependencies to your build.gradle file:

1
2
3
4
5

dependencies {
    // JAX-B dependencies for JDK 9+
    implementation "jakarta.xml.bind:jakarta.xml.bind-api:2.3.2"
    implementation "org.glassfish.jaxb:jaxb-runtime:2.3.2"
}

Now I have two choices. Either try to make this work using openjdk11, or try to use openjdk8. Let’s try to downgrade to openjdk8.

1
2
3

root@laptop:~# apt purge openjdk-11-{jre,jdk-headless,jre-headless}
root@laptop:~# apt install openjdk-8-{jre,jdk,jdk-headless,jre-headless}
root@laptop:~# apt build-dep android-platform-tools-base

I’m going in circles here. build-dep is reinstalling openjdk11. I’ll use the update-alternatives command to set 8 as the default.

1
2
3
4
5
6
7
8
9
10
11

root@laptop:~# update-alternatives --config java
There are 2 choices for the alternative java (providing /usr/bin/java).

  Selection    Path                                            Priority   Status
------------------------------------------------------------
* 0            /usr/lib/jvm/java-11-openjdk-amd64/bin/java      1111      auto mode
  1            /usr/lib/jvm/java-11-openjdk-amd64/bin/java      1111      manual mode
  2            /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java   1081      manual mode

Press <enter> to keep the current choice[*], or type selection number: 2
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java to provide /usr/bin/java (java) in manual mode

Is this good enough to change something? Who knows. Trying to build again.

1

error: package javax.xml.bind.annotation does not exist

I guess it isn’t. I also tried update-alternatives --config javac, no difference.

1
2

root@laptop:~# update-alternatives --get-selections | grep 11-openjdk | wc -l
32

This doesn’t seem like a good way forward. Purging openjdk-8 and going the other route.

1
2
3
4
5

root@laptop:~# apt purge openjdk-8-{jdk,jre,jdk-headless,jre-headless}
hugopeixoto@laptop:~/w/c/android-platform-tools-base-2.2.2$ java --version
openjdk 11.0.7 2020-04-14
OpenJDK Runtime Environment (build 11.0.7+10-post-Debian-3)
OpenJDK 64-Bit Server VM (build 11.0.7+10-post-Debian-3, mixed mode, sharing)

OK, so what’s up with this implementation gradle thing? Where does it go? Do I need to install any packages? Let me see what’s in my system:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

hugopeixoto@laptop:~/w/c/android-platform-tools-base-2.2.2$ dpkg -l | grep jaxb
ii  libjaxb-api-java                        2.3.1-1                         all          Java Architecture for XML Binding API

hugopeixoto@laptop:~/w/c/android-platform-tools-base-2.2.2$ dpkg-query -L libjaxb-api-java
/.
/usr
/usr/share
/usr/share/doc
/usr/share/doc/libjaxb-api-java
/usr/share/doc/libjaxb-api-java/changelog.Debian.gz
/usr/share/doc/libjaxb-api-java/copyright
/usr/share/java
/usr/share/java/jaxb-api.jar
/usr/share/maven-repo
/usr/share/maven-repo/javax
/usr/share/maven-repo/javax/xml
/usr/share/maven-repo/javax/xml/bind
/usr/share/maven-repo/javax/xml/bind/jaxb-api
/usr/share/maven-repo/javax/xml/bind/jaxb-api/2.3.1
/usr/share/maven-repo/javax/xml/bind/jaxb-api/2.3.1/jaxb-api-2.3.1.pom
/usr/share/maven-repo/javax/xml/bind/jaxb-api/debian
/usr/share/maven-repo/javax/xml/bind/jaxb-api/debian/jaxb-api-debian.pom
/usr/share/maven-repo/javax/xml/bind/jaxb-api-parent
/usr/share/maven-repo/javax/xml/bind/jaxb-api-parent/2.3.1
/usr/share/maven-repo/javax/xml/bind/jaxb-api-parent/2.3.1/jaxb-api-parent-2.3.1.pom
/usr/share/maven-repo/javax/xml/bind/jaxb-api-parent/debian
/usr/share/maven-repo/javax/xml/bind/jaxb-api-parent/debian/jaxb-api-parent-debian.pom
/usr/share/java/jaxb-api-2.3.1.jar
/usr/share/maven-repo/javax/xml/bind/jaxb-api/2.3.1/jaxb-api-2.3.1.jar
/usr/share/maven-repo/javax/xml/bind/jaxb-api/debian/jaxb-api-debian.jar

So there’s a few javax/xml/bind/ files. Why is this not picking them up? Do I need the maven { url 'file:///usr/share/maven-repo' } thing here as well? Searching a bit more.

https://tracker.debian.org/pkg/android-platform-tools-base

https://bugs.debian.org/cgi-bin/pkgreport.cgi?repeatmerged=no&src=android-platform-tools-base

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=894284

This bug seems similar to the one I’m having.

The javax.xml.bind errors can be simply fixed by adding a dependency on jaxb-api: [adding “compile ‘javax.xml.bind:jaxb-api:debian’” to repository/build.gradle]

Cool, this seems simple.

There is another error with Java 10 caused by a change in the return type of javax.xml.namespace.NamespaceContext.getPrefixes(). It’s easily fixed with: […]

Hey, there’s the getPrefixes bug I fixed.

Unfortunately there is another issue with SignedJarBuilder: […]

Once kotlin is in Debian, then we can use newer upstream versions, which support the latest JDK.

So it seems that I’ve hit the kotlin hole again. Fixing the javax.xml.bind does hit the SignedJarBuilder issue they mention:

1
2
3
4
5
6
7
8
9
10
11
12
13

[...]/sdklib/src/main/java/com/android/sdklib/internal/build/SignedJarBuilder.java:22: error: cannot find symbol
import sun.misc.BASE64Encoder;
               ^
  symbol:   class BASE64Encoder
  location: package sun.misc
[...]/sdklib/src/main/java/com/android/sdklib/internal/build/SignedJarBuilder.java:23: error: package sun.security.pkcs is not visible
import sun.security.pkcs.ContentInfo;
                   ^
  (package sun.security.pkcs is declared in module java.base, which does not export it to the unnamed module)
[...]/sdklib/src/main/java/com/android/sdklib/internal/build/SignedJarBuilder.java:26: error: package sun.security.x509 is not visible
import sun.security.x509.AlgorithmId;
                   ^
  (package sun.security.x509 is declared in module java.base, which does not export it to the unnamed module)

I can try to replace BASE64Encoder with java.util.Base64. I should be tracking these changes in git or something. Getting the Base64 thing to compile was straightforward. Unsure what to do with the sun.security.* packages.

SignedJarBuilder is deprecated. Maybe I can just remove its implementation and side step the errors completely? This is going to be great. Removing this probably means that I will be unable to generate signed APKs, but if I can live with that for now.

1
2
3
4
5
6
7
8
9
10
11

Compiling with JDK Java compiler API.
[...]/sdk-common/src/main/java/com/android/ide/common/vectordrawable/VdPreview.java:23:
error: package com.sun.org.apache.xml.internal.serialize is not visible
import com.sun.org.apache.xml.internal.serialize.OutputFormat;
                                      ^
  (package com.sun.org.apache.xml.internal.serialize is declared in module java.xml, which does not export it)
[...]/sdk-common/src/main/java/com/android/ide/common/vectordrawable/VdPreview.java:24:
error: package com.sun.org.apache.xml.internal.serialize is not visible
import com.sun.org.apache.xml.internal.serialize.XMLSerializer;
                                      ^
  (package com.sun.org.apache.xml.internal.serialize is declared in module java.xml, which does not export it)

This is not getting me anywhere. I’m just removing code that doesn’t compile. I’m pretending to port android-platform-tools base to openjdk11, this can’t be easy or a quick way to get this done. Let’s backtrack a bit.

When I started setting up the Dockerfile, I had to use sid because libgradle-android-plugin-java was not available in testing. But it is available in another release: stretch. Let’s try using that instead.

Dockerfile using debian:stretch

It seems that android-sdk-helper was not available in stretch, so I’ll need to patch build.gradle manually.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

root@781970948c66:/app/AndroidHelloWorld# git diff
diff --git a/build.gradle b/build.gradle
index c3ec2c3..54c4f60 100644
--- a/build.gradle
+++ b/build.gradle
@@ -1,10 +1,11 @@
 buildscript {
     repositories {
         mavenCentral()
+        maven { url 'file:///usr/share/maven-repo' }
     }

     dependencies {
-        classpath 'com.android.tools.build:gradle:2.1.2'
+        classpath 'com.android.tools.build:gradle:debian'
     }
 }

@@ -12,7 +13,7 @@ apply plugin: 'com.android.application'

 android {
     compileSdkVersion 23
-    buildToolsVersion "23.0.2"
+    buildToolsVersion "24.0.0"

     lintOptions {
         abortOnError false
root@781970948c66:/app/AndroidHelloWorld# tail -n1 local.properties
sdk.dir=/usr/lib/android-sdk
root@781970948c66:/app/AndroidHelloWorld# gradle build
[...]
ECJ compiler crashed
java.lang.RuntimeException: ECJ module temporarily disabled!
[...]
[repeat ECJ exception a few more times]
[...]


BUILD SUCCESSFUL

Total time: 9.303 secs
root@781970948c66:/app/AndroidHelloWorld# find . -name "*.apk"
./build/outputs/apk/AndroidHelloWorld-release-unsigned.apk
./build/outputs/apk/AndroidHelloWorld-debug.apk

Well, at least I got something working. Kind of sad that kotlin doesn’t work and that I need to use stretch. Here’s the current Dockerfile:

1
2
3
4
5
6
7
8
9
10
11
12
13

FROM debian:stretch

RUN apt update
RUN apt -y install android-sdk android-sdk-platform-23 git libgradle-android-plugin-java
RUN apt -y install vim

ADD . /app
WORKDIR /app

RUN mkdir -p /root/.android/
RUN touch /root/.android/analytics.settings

CMD bash

Let’s go back to my app. Same thing, setting buildToolsVersion "24.0.0" and running gradle build:

1
2

> You have not accepted the license agreements of the following SDK components:
  [Android Support Repository].

Ah, here we go. I have these two dependencies:

1
2

compile 'com.android.support:appcompat-v7:23.1.1'
compile 'com.android.support:design:23.1.1'

And Android Support Library (com.android.support) is in the “Needs doing / low hanging fruit” section of Debian’s wiki page. And I seem to be using it:

1
2
3
4
5
6
7
8
9
10
11
12

hugopeixoto@laptop:~/w/p/http-share-android$ ack android\.support app/src/main/java/net/hugopeixoto/
app/src/main/java/net/hugopeixoto/customsharing/UploadActivity.java
19:import android.support.design.widget.FloatingActionButton;
20:import android.support.v4.app.NotificationCompat;
21:import android.support.v7.app.AppCompatActivity;

app/src/main/java/net/hugopeixoto/customsharing/AppCompatPreferenceActivity.java
6:import android.support.annotation.LayoutRes;
7:import android.support.annotation.Nullable;
8:import android.support.v7.app.ActionBar;
9:import android.support.v7.app.AppCompatDelegate;
10:import android.support.v7.widget.Toolbar;

I guess I could try to build this from source, or even package it. But that’s going to be for another time.

Conclusions

To build an android application, you need (maybe you don’t?) the Android SDK. Google distributes Android SDK binaries, but you need to accept an EULA to use these. The source code is free, though.

Debian builds android-sdk packages. This sidesteps the need for accepting EULAs, along with some other advantages, listed in their wiki.

Recent versions of android-sdk depend on Kotlin, and Debian does not package kotlin yet. Until this dependency is solved, the only versions that are available in Debian are in Debian stretch, since it is based openjdk8. Debian Sid, for example, is based on openjdk11, so things don’t work that well.

Even though I was able to compile a sample app in debian:stretch, I can’t build my application, as it depends on com.android.support, which isn’t packaged yet.