Michael Zent
Published: May 19th, 2021 | Dedicated to Schuyler W. Zent |
Updated: Apr 7th, 2024 |
The following arose from the wish to develop applications for mobiles on mobile without monster IDEs. Consequently without OS emulator, and even without internet connection. But no renouncements concerning the functional complexity of the applications. Should be quick and effective thanks to the mighty Command Line.
The great attention, in the current moment 2245 views from all over the globe for this rather theoretical article, confirm that the efforts haven't been in vain. Particularly regarding some feral, even desperate, discussions on the topic [1].
The software created for the here covered issue can be downloaded in the current version for free as a .deb
package [2][3], firstly for Android OS.
Finally, the author considers it necessary to assure hereby, that neither the following text, nor the ideas comprised, result even in the slightest from the usage of AI tools. Accordingly the author explicitly prohibits the use of this article, and of the author's hence resulting software, in automated systems without proper referencing. That is valid also for humans.
Android [4], an open-source operating system derived from Linux, is the dominant OS on mobile devices like smartphones, tablets and smartwatches [5]. It is also used for internet-capable TVs, car controls, IoT appliances and various other purposes. Therefore many people are steadily programming new applications, commonly abbreviated apps, for Android.
Even if a professional developer leaves his desktop computer he continues to work — everywhere. Moreover he won't desist from appreciating the advantages of the Command Line Environment. Hence arises the motivation for this APK Building project(a), aiming to provide a way to build mobile applications for Android OS straight on their mobile devices — without limitations concerning a project's complexity, and without depending on a personal computer, such at minimal hardware and software requirements as well as resource consumption. The built application should be testable directly on a real mobile device, without having to use an emulator.
This article examines primarily the structure of Application Projects and Application Packages for Android to define and implement with the gained knowledge the desired toolchain, which transforms the first to latter, and runs in a Command Line Environement.
The exact structure of an Android application project mostly depends on the used IDE like Android Studio, Eclipse, IntelliJ and the build systems it supports, as Gradle, Maven, Ant [6]. However nobody is forced to use any of these and so there is actually no fixedly defined structure. The core bricks are the same everywhere, thus forming a logical project structure, and are depicted in Fig.1. Note that the directory names are those used by convention — they could have any name. Moreover each one's content could be scattered on several folders.
Lets take a look at the components of an Android app project. First of all there are some mandatory files.
.apk
file. This is required to install or update the app on an Android device.The project also contains some directories, subdividable into three groups — resources, sources and libraries. They are all optional and are added at need.
Files which provide some kind of content or information to be utilized by the app at runtime are called resources. According to their use one distinguishes two types with an own folder for each — resources and assets(b) [7].
It's common practice to keep the contents of the app apart from the code which drives the app's behavior. That simplifies maintenance and the provision of alternative resources to support compatibility with specific device configurations, especially concerning screen properties and localization. This externalization of resources is supported by Android in two ways — the high-level Resources
management system and the lower-level AssetManager
.
Android's Resources
system is used for elements which depend on the device configuration, so mostly UI related ones. At runtime the resources appropriate for the current configuration are selected from the available alternatives, allowing to adapt to hardware variations and to respond automatically to changes e.g. in language and region, device orientation and pixel density, to name a few. The only measure one has to take is to place them into the folder res
in subdirectories, properly named according to the type of resource and, at need, the supported specific configuration(c). All resources in the res
directory are accessed through resource IDs which are automatically generated during the resource compilation process.
The AssetManager
is generally used for resources which are independent of the current device configuration, e.g. in-game data and databases. This spares the costs for Android's configuration system. There are no constraints considering naming and hierarchic structure for files in the assets
folder. They are accessed as filesystem entries instead of being assigned IDs.
Note, that though audio and video files are oftenly mentioned as examples of assets
folder content while other sources refer to the res/raw
directory being the proper place for them, neither of these recommendations is purposive. The core question is whether the usage of these files is somehow configuration-dependent — the sound file to play may change with the screen orientation, and the video should maybe match the display resolution. In that case res/raw
should be used to employ the Resources
system. Otherwise the assets
folder for the AssetManager
is the better choice.
Four languages are officially supported for Android Development — Java, Kotlin, C and C++(d). The source code written is stored into corresponding folders.
Android apps run in an instance of the Android Runtime (ART) — an JVM which is optimized for running multiple VM instances on low-memory devices. The runtime interprets an intermediate machine-independent form of compiled code, called bytecode, into machine-specific native code and performs it. Such the same bytecode can be run on devices with different processor architectures if a corresponding JVM is installed [8].
As the API framework is therefore written in Java this language logically became kind of the traditional language to program Android apps. Since 2017, with Android 8.0, also Kotlin is supported and in 2019 even was declared by Google to be the preferred development language. However Java and Kotlin code are interoperable and both are compiled into Java bytecode [9].
Further an app can be implemented partly or entirely(f) in native code, using C/C++ as low-level languages. The native module is executed outside the ART in its own native thread(g). This is relevant for applications like multimedia, networking and gaming, where execution speed, latency and energy efficieny are important factors which can be optimized by accessing the hardware on a low level, but also for developers who need a good cross-platform portability for their projects [10].
When the same functionalities and components are to be used time and again, and even in various projects, they are usually bundled into so-called libraries for simplified maintenance and reuse. Those belonging to an Android app project are, roughly said, stored into two directories — libs and lib. Remind, that once again these are logical folders, which can be the sum of several physical folders.
libs comprises Java .jar
(JAR) and/or Android .aar
(AAR) archive files mostly for the compilation phase. Though the abbreviation JAR bears the name of the Java language, these libraries are likewise used in context with Kotlin because of the aforementioned fact that source code of these interoperable languages both are compiled into Java bytecode. In turn the AAR format is specific for Android. When building multiple apps that make use of the same components, these can be bundled into Android Archives, which are ZIP archives with an .aar
extension and an anatomy similar to an Android application project. They may provide .jar
files as well as native shared libraries, source code, UI resources, assets and other things needed to build an app [11].
Any kind of Java/Kotlin and resource dependency of an Android app project is placed into libs
, except of the android.jar
file which contains the basic set of Android platform API classes. Latter is provided with any correctly established APK building toolchain and will be included into the build process per default. While first-party dependencies, i.e. those originating from the developer himself, might be put in right away, third-party libraries are usually retrieved from Maven repositories, either manually or using auxiliary tools, being able also to resolve transitive dependencies [3]. Note, that the generalizing term Artifact [12] is used for any kind of file in a Maven repository that can be addressed via corresponding coordinates [13].
Whether a JAR, stand-alone or from an AAR, is indeed to be used only in compile-time or shall be packed into the APK for linkage in runtime, is usually determined either by an accompanying Project Object Model (POM) file or configuration files specific to the used build system.
lib contains native shared .so
and/or static .a
library files. It also may incorporate subdirectories to be integrated as-is into an APK, named according the supported Application Binary Interfaces (ABIs), and containing shared libraries needed only at runtime. All libraries outside these ABI-subdirectories are thought to be used in the compilation phase only.
It is to note, however, that the compile-time libraries in lib
will primarily be first-party ones. Third-party libraries will usually be not found there, as those will be installed from remote repositories using the operatings system's package manager, which also automatically resolves dependencies recursively. After installation these libraries will be accessed via well-known system paths, configured by the setting of environment variables and usable by a correctly installed build system.
The runtime libraries to be packaged as-is are usually first-party libraries and their third-party dependencies. While the first will be provided right-away by the project's developer resp. generated during the application's build process, the latter will be retrieved from remote repositories using the system's package manager or auxiliary tools [3], and put into the ABI-subdirectories.
It is obvious that a developer and the building system employed have to take care of the purpose and placement of the project's libraries at all stages. Moreover, the libraries themselves have their dependencies, and there can be plenty of them, so it is recommended to use a suitable transitive dependency resolver [3].
An Android application is provided as an Android Package, which is simply a ZIP archive with an .apk
extension, and has a typical structure as shown in Fig.2, containing everything needed to run the app after installation [14].
It almost always embodies the following files, though only AndroidManifest.xml
is obligatory.
.dex
) file understood by the virtual machines Dalvik (used till Android 4.4) and Android Runtime (ART, since Android 5.0). It uses a bytecode format designed specially for Android to minimize the memory usage. Under circumstances an app contains additional supporting .dex
files, named according the scheme classesN.dex
. This is called multidex app configuration..xml
files in the res/values
folder — strings, style definitions and some other resource types — are stored directly into this file. For all other resources their paths are entered, relative to the .apk
file's root.Also an APK mostly includes the following directories, among which only META-INF
is mandatory.
CERT.SF
and CERT.RSA
signature files, as well as the MANIFEST.MF
manifest file.res/values
— being compressed and preserving their relative directory structure..apk
file.libs
folder can be met, comprising .jar
files for runtime linkage.Note that an APK may contain further files and folders, but here shall be focused on those elements common to all Android apps.
The route to an APK is illustrated in Fig.3 in a simplified way. At first there is a project edited by the programmer, who also defines what the target of his work is. A largely automated toolchain than processes everything and ideally yields the desired result.
As the name says a toolchain is a collection of subsequently called tool-programs, where the output of one is the input of another. Ideally the user of a toolchain doesn't need to know what goes on during this process — one just feeds it with the raw materials and gets the final product. So here the content of an Android app project has to become an APK.
The .aar
files included via the libs
folder are unzipped and their files and directories are merged with the corresponding counterparts in the project. Libraries and assets are basically copied as-is into the project, while resource- and manifest-merging require some heuristics. There are however some files, e.g. lint.jar
, which are not meant to be included into the APK building process, as they are only auxiliary during the development phase, and have otherwise to be ignored [15].
As shown in Fig.4, the file AndroidManifest.xml
is compiled into Android's binary XML format and then packaged.
The first step is to merge the resources — as they may come from different sources, both folders and .aar
files — into a single assets
resp. res
directory. As illustrated in Fig.5 the two resource types are further processed dissimilarly according to their difference in character [16].
While the assets are then packaged as-is, the resources in res
are compiled firstly. This creates the resource table file resources.arsc
and a res
folder, where the values
subfolder is missing as its content was directly entered into the resource table. Both are compressed and ready for packaging.
Additionally the Java source file R.java
is generated which assigns to each resource an ID, necessary for referring to it during runtime. Though human-readable, this file should never be edited manually as it is highly probable to crash the app because of erroneous references. There is no good reason imaginable why one should want to edit R.java
anyway. The file is then included into the Java compilation process.
When the R.java
file is generated, an ID is assigned to each resource for referring to it during the app's runtime. "The hexadecimal values in a R.java
file are the jumping-off points for Android's resource management mechanism. Android uses these numbers for quick and easy loading of the things you store in the res
branch" [17]. Fig.6 depicts the fields of a resource ID.
They are
0x7f
(application), but also 0x01
(Android framework) and 0x00
(shared resource library) are common.Note, that because of the fields type
and entry
there's always the chance of changes in the ID assignment by the resource compiler during compile time, even in consecutive runs. In the app's runtime the IDs will be static.
Further, .aar
files may include a R.txt
file. It serves two purposes — to ease debugging purposes for the AAR in question, and to cache resource information for the build of .apk
or .aar
files which depend on that AAR.
The functionality of R.txt
is basically the same as of R.java
— mapping resources to IDs —, therefore one will find a R.txt
in an .aar
file with a non-empty res
folder. In case of an empty res
folder there's either no R.txt
at all, or an empty R.txt
.
As described in [19], R.txt
is an intermediate file on the way to an APK, containing a list of resources and their IDs. It is also archived to help with debugging, as a simple .txt
file is easier to parse than the final .java
file. Actually it's not that intermediate, but also characterizable as a final file as e.g. Android Studio puts it also into the output directory next to the final .apk
file. Now, in case of an library .aar
file it is put directly into the output AAR together with other debugging and configuration files like lint.jar
.
In an .aar
file, which can be used as a dependency for other libraries as well as for APKs, R.txt
has an additional role. Except for being used for debugging it also serves as an input for the next level file build in the dependency hierarchy.
Lets take a look on the entries in R.txt
. Their structure is as follows, thereby read "res" as "resource". A line ends with a newline.
[type] [res_class] [res_name] [res_id]
This information matches the information in R.java
like this:
public static final class [res_class] {
public static final [type] [res_name] = [res_id];
//...
}
A sample R.java
from package org.chromium.ui
is shown in [19].
public final class R {
public static final class attr {
public static final int buttonAlignment = 0x7f030038;
public static final int buttonColor = 0x7f03003e;
public static final int layout = 0x7f030094;
public static final int roundedfillColor = 0x7f0300bf;
public static final int secondaryButtonText = 0x7f0300c4;
public static final int stackedMargin = 0x7f0300d4;
}
public static final class id {
public static final int apart = 0x7f080021;
public static final int dropdown_body_footer_divider = 0x7f08003d;
public static final int dropdown_body_list = 0x7f08003e;
public static final int dropdown_footer = 0x7f08003f;
}
public static final class layout {
public static final int dropdown_item = 0x7f0a0022;
public static final int dropdown_window = 0x7f0a0023;
}
}
The corresponding R.txt
will then be:
int attr buttonAlignment 0x7f030038
int attr buttonColor 0x7f03003e
int attr layout 0x7f030094
int attr roundedfillColor 0x7f0300bf
int attr secondaryButtonText 0x7f0300c4
int attr stackedMargin 0x7f0300d4
int id apart 0x7f080021
int id dropdown_body_footer_divider 0x7f08003d
int id dropdown_body_list 0x7f08003e
int id dropdown_footer 0x7f08003f
int layout dropdown_item 0x7f0a0022
int layout dropdown_window 0x7f0a0023
Now, in an APK building without AARs, the resource IDs from R.txt
will be taken over as-is to R.java
. However, this won't work with AARs because of the mentioned variable structure of resource ID's. The type
field won't be guaranteedly the same for the same resource type across several .aar
files. A layout
resource might be of type 0x0a
in one file, but be 0x03
in an other, while in the first one 0x03
stands for int
. The entry
field is an autoincremented number, which per se can only be valid without collision within the same file.
So, this way one can have the same ID, lets say 0x7f030038
, in several R.txt
files from all the dependency AARs, which will describe completely different resources. Therefore, while building an APK, or another higher-level AAR, all the R.txt
files from dependencies will be put together with the APK's own temporary R.txt
file and a new, final, R.txt
will be written, with unique IDs for the APK's resources.
As shown in Fig.7, Java and Kotlin source codes are compiled into Java bytecode (.class
) files which then are converted into ART-specific bytecode files, called Dalvik Executables (DEX, whereby Dalvik being the predecessor of ART). These are then packaged into the APK.
Additional .class
files provided by JARs are used to satisfy import dependencies and are compiled to .dex
files too. The archive android.jar
provides the Android framework APIs and as such is part of all projects per definition. Because the APIs are part of the OS there is no need to convert the content of android.jar
to .dex
files.
Native code is embedded as dynamic shared libraries(h), so-called Shared Object (.so
) files, as shown in Fig.8.
The compiler translates each human-readable C/C++ source code file into a machine-specific binary code file, called Object (.o
) file. All objects are then linked with one another, and static respectively dynamic libraries, into a single executable .so
file, whose methods are loaded during runtime at need. The Android Package Manager expects to find native libraries on filepaths inside the APK matching the pattern lib/<abi>/lib<name>.so
, where <abi>
is the name of the ABI the library was compiled for. All ABI directories — generated and pre-built ones — are merged, stored into a lib
folder and get packaged.
The results of the compilation stage are packed into a ZIP archive with .apk
extension. To make the app installable and updateable on an Android device it is required to sign it, using a private App Signing Key.
The signer adds the folder META-INF
to the APK, containing the signature files CERT.SF
and CERT.RSA
, and the manifest file MANIFEST.MF
.
Therewith Fig.9 depicts the final general toolchain architecture.
Usually a Build System is used to automate and manage the invocation of the toolchain according to the Build Configuration which defines a set of source files, dependencies and other properties of relevance, and is described in corresponding script files.
Common build systems for Android app projects are Gradle, Maven, and Ant. Though in Java development in general Maven is the most used build automation tool, and even its predecessor Ant still has out of legacy reasons a relevant user basis, Gradle has been made the default build system for Android as well as Kotlin development because of its relative simplicity, flexibility and speed..There are various comparisons between the three on the Web, and Maven has some advantages, too, e.g.considering customizability and integration in business environments, however we'll focus on the main points, why neither of the three are that suitable for us here.
Our task now is to design a lightweight replacement for Gradle and Co. using the available Termux packages, thereby minimizing the need for configuration without tieing the user to a fixed project structure.
Note, that for C/C++ compilation another build system, usually CMake, is invoked.That's a common and sleek tool, there's nothing to change here.
In general coders work on a PC, so a desktop or laptop, as their development computer, using an Integrated Development Environment (IDE) which not only implements the whole building toolchain, but also offers a variety of assisting tools. Usual are Android Studio and Eclipse. In short — there are no APKs for this software available for an installation on Android. And it's pretty obvious that a straightforward attempt of porting is foolish if done by an outsider. If at all — direct ports to mobile systems usually suffer at least from ergonomic and performance problems — there has to be an official version for Android made by the community in charge.
An alternative would be to load the basic command line tools onto the device and to chain them with a shell script [21]. Ignoring various other problems, the main obstacle is that Android not only refuses to execute programs in the user partition but also denies access to the system partition. Yet this is actually the solution as there is a work-around which will be elucidated in a moment.
There are indeed Android applications which provide an IDE allowing to code on the whole as with analogous desktop software. Ideally it should have the following features.
Such IDE applications vary in emphasis and completeness. A good, and rare, example for an extensive IDE for Android is AIDE. However, here shall not be discussed the pros and cons of various mobile IDEs. The intrinsic point is, that they all have some significant drawbacks in common.
Android's own shell has been made barely usable out of security considerations. However, one can remedy, without compromising the system's integrity, by using e.g. Termux, a terminal and Linux environment emulator for the Android OS [22]. Such one gets a Command Line Interace for mobile development, and it solves the aforementioned problems of mobile IDEs, rendering them unnecessary. One can count the following to its main advantages.
apt
or dpkg
— with plenty of various utilities. Furthermore language-specific repositories like for Node.js, Perl, Python, Ruby or TeX Live are accessible after installing the correspondent language support. If a package is missing it can be requested or contributed by oneself. Additionally the functionality of Termux itself can be extended with free add-ons.The Termux repositories contain all packages needed for assembling a fully functional APK building toolchain [23]. Note that if not otherwise mentioned the package name corresponds to the tool program's name.
As .aar
files are actually usual ZIP archives it is enough to install unzip
to extract them [24].
The Android Asset Packaging Tool aapt
covers several tasks — manifest and resource compilation as well as the final APK packaging [25]. Though aapt
has become deprecated and was replaced by a seemingly more convenient aapt2
[26][27] in 2017, the author had to make the experience, that aapt2
can be pretty moody when used stand-alone, incl. throwing hardly traceable errors. As the older aapt
works reliably, and apparently is the basis for aapt2
to date anyway, it is reasonable to stick with it until the flaws of aapt2
are adjusted.
For the compilation of Java code one has the choice between the Eclipse Compiler for Java ecj
[28] and the Java Compiler javac
[29] shipped with openjdk-17
. The first one has inter alia the benefit of being pretty compact as it dosen't require an JDK installation, and bringing the android.jar
file needed for compiling Java/Kotlin code for Android. However, other packages depend on openjdk-17
, so it will be delivered anyway. Overall there are good reasons, which shall not be discussed here, to integrate both compilers into the toolchain, set javac
as the default compiler, and give the user the possibility to choose ecj
[30][31]. To compile Kotlin code the package kotlin
provides kotlinc
[32].
The resulting .class
files can be converted into .dex
files with the help of either the dx
or the d8
tools. In this case there is no reason to support both programs, as the older dx
is less performant than d8
and was deprecated in 2021 [33][34][35]. Additionally d8
comes with the Java code shrinker r8
. As its alternative proguard
is not available in the Termux repositories, the choice between dx
without a shrinker and d8
with r8
is easy. Alas, r8
relies on prework of aapt2
and as long as one has to renounce aapt2
because of the aforementioned reasons, one has to abstain from using r8
for now, too.
The preferred tool for building the native code components of an app is cmake
[36]. Its installation comes with the complete native code toolchain based on llvm
/clang
. By setting up an additional repository also gcc
becomes available as an alternative compiler, however the overhead of additional 200MB seems unjustified considering the performance of clang
[37].
The final step of signing the created .apk
file can be achieved either by jarsigner
[38], provided by openjdk-17
, or by apksigner
[39]. Developers are encouraged to use latter as it has been introduced 2016 as a tool specialized for this task, while the first one aims for .jar
files in general.
So finally Fig.10 shows the toochain's concretised architecture.
A convenient way to chain the tools is to write a shell script which arranges all the input, then calls the programs in their right order, and finally organizes the output. Termux provides a number of shells, however it seems meaningful to select bash
as it is the most common standard shell and such opens a perspective to port the toolchain to other linuxoid systems.
Next the script, together with a default signing key, has to be packed into an installable .deb
file. This can be done with the help of dpkg
or the tool termux-create-package
.
Here the overall toolchain architecture has been sketched, further discussion and the implementation got an own article [2].
The knowledge gathered in this article about the internals of Application Projects as well as Application Packages for Android, and the corresponding building procedures, laid the path to the development of a toolchain organizing software and optional auxiliary tools. To each of them, separate pages are dedicated, where they also can be downloaded for free [2][3].
Except the initial setup of the necessary components from tools [2] to library dependencies [3] the work is thereby carried out offline. An internet connection may be needed once more when the Application Project has been modified to such an extent, that further libraries have to be added as dependencies.
In the light of this article one can convince oneself that a fully-fledged development of applications for mobile on mobile is unproblematic, despite the limitations of Android OS.
native_activity.h
interface which connects to the NativeActivity
helper Java class provided by the Android framework. Such the JNI is just hidden. The app still runs in its own JVM instance and a new thread is spawned for the native module.I thank
God and my mother for my being, my father for his equicontinuous motivation and all the coffee,
Leo, the plush beast, for generously been allowed to drool (at) him when the world was still shiny,
Mr. Linnert, Prof. Dr. Prechelt and Prof. Dr. Schiller for inspiration and example.
This reflects selected real questions and posts, esp. by the author, on the StackOverflow platform, shortened to the essence.
Q: How to manually generate R.java
?
A: Don't, for more see .
Q: How to generate R.java
with only the resource IDs needed by the package?
A: Use the code shrinker r8
, and the code inspector lint
, for more see .
Q: What's the purpose of the IDs in R.java
?
A: Those IDs are jumping-off points for Android's resource management mechanism to accelerate the loading of resources from the res
folder, for more see .
Q: How to enforce, that aapt2
generates the same resource IDs as aapt
did?
A: Not at all, for more see .
Q: What's the purpose of the lint.jar
and inspector.jar
that can be found in some AARs?
A: Code inspection, for more see .
Q: Will the app work if one deletes the lint.jar
and inspector.jar
found in dependency AARs?
A: Yes, for more see .
Q: What's the purpose of .jar
files in the libs
directory of an AAR?
A: To provide Java libraries for integration into the APK which depends on that AAR to be linked dynamically during runtime, for more see .
Apr 7th, 2024
United States (19.0%), Germany (16.6%), India (8.9%), France (5.8%), United Kingdom (3.2%), Italy (2.7%), Netherlands (2.6%), Russia (2.5%), China (2.4%), Brazil (1.9%), Canada (1.9%), Indonesia (1.9%), Sweden (1.6%), Malaysia (1.5%), Singapore (1.4%), Iran (1.3%), Turkey (1.2%), Bangladesh (1.1%), Australia (1.0%), Philippines (1.0%), Other (20.5%)
This page is intended to be viewed online and may not be printed.