TWiStErRob

Android SQLite FTS4 tokenize=unicode61 remove_diacritics=2

2023-10-21T00:00:00+00:00

Android developer wants to create a quick full text search for an international audience with case-insensitivity.

TL;DR

See the Summary section below.

TL;DR
The High of Feasibility
Testing a Proof of Concept
Compassion
Reproducer
In Depth Research
Summary
Conclusion

The High of Feasibility

Android developer searches a bit and finds SQLite, FTS4 and unicode61 as potential solution.

The FTS4 documentation reads as follows:

FTS4 is an enhancement to FTS3.
FTS3 has been available since SQLite version 3.5.0 (2007-09-04)
The enhancements for FTS4 were added with SQLite version 3.7.4 (2010-12-07).
[…]
The “unicode61” tokenizer is available beginning with SQLite version 3.7.13 (2012-06-11).

Quick check of Android SQLite versions reveals the following table:

Android API	SQLite Version	Comment
API 1	3.4
API 3	3.5	FTS3 from this point onwards.
API 8	3.6
API 11	3.7	FTS4 was added somewhere here.
API 21	3.8	FTS4 and `unicode61` definitely from this point onwards.
API 24	3.9
API 26	3.18
API 27	3.19
API 28	3.22
API 30	3.28
API 31	3.32

Let’s say that the app under development has an “old” minSdk = 21 which covers 99.34% of the market in 2023¹. This is great, this means we can use FTS4 for the app, no restrictions. Quick further check of StackOverflow: Version of SQLite used in Android? reveals the exact version we could expect from API 21 is 3.8.6. Further check of the release notes shows:

The unicode61 tokenizer is now included in FTS4 by default.

Such, great, news! Feasibility confirmed, let continue reading the docs and find the option to remove diacritics (e.g. accents, which makes the search a little bit more dynamic in ways I won’t go into detail here.):

The remove_diacritics option may be set to “0”, “1” or “2”. The default value is “1”. […]
This is technically a bug […]
If this option is set to “2”, then diacritics are correctly removed from all Latin characters.

OK, so the default behavior is bugged in some cases², let’s just set the option to 2 to ensure the best behavior for an international audience.

Testing a Proof of Concept

So, now equipped with a very simple SQL statement based on all the above documentation:

CREATE VIRTUAL TABLE Search USING FTS4 (
	tokenize=unicode61 'remove_diacritics=2',
	content
);

let’s try this out. I was using the latest emulator API 34, and it was working fine. Then I tried API 21 and got greeted with

android.database.sqlite.SQLiteException: unknown tokenizer (code 1)
	at android.database.sqlite.SQLiteConnection.nativeExecuteForChangedRowCount(Native Method)
	at android.database.sqlite.SQLiteConnection.executeForChangedRowCount(SQLiteConnection.java:734)
	at android.database.sqlite.SQLiteSession.executeForChangedRowCount(SQLiteSession.java:754)
	at android.database.sqlite.SQLiteStatement.executeUpdateDelete(SQLiteStatement.java:64)
	at android.database.sqlite.SQLiteDatabase.executeSql(SQLiteDatabase.java:1676)
	at android.database.sqlite.SQLiteDatabase.execSQL(SQLiteDatabase.java:1605)

Uhm, OK, this shouldn’t happen according to the version checks I did earlier, right? unicode61 was available much earlier, and is now included by default too. What’s going on?

Compassion

Failing to make it work, I turned to our big G friend to find some answers.

My first try to search for the error message SQLiteException: unknown tokenizer (code 1) pretty much failed, as I only found false positives which were a similar error showing me how “icu” and “unicode61” tokenizers were not found. These seemed like proper errors where the tokenizer was simply not available.

On to more direct search based on what I knew fts4 unicode61 remove_diacritics also gave me no usable results, so I went back to try an exact search "SQLiteException: unknown tokenizer (code 1)" which yielded a Bingo! result at Unknown tokenizer - TOKENIZER_UNICODE61 which was exactly the same problem. I was glad to find some info, but there were no answers, just guessing.

I also tried android "unicode61" "remove_diacritics" which lead me to find Uwe’s issue where he apparently also had a similar issue at first, but in the end he just went with remove_diacritics=0.

I wasn’t satisfied with finding only questions, without knowing why it’s not possible. Luckily the above references gave me some ideas, so I started researching deeper.

Reproducer

At this point I knew I needed more info: what is failing exactly and where? So I went on to create a small standalone project and see what’s possible.

I tested variations of FTS3/FTS4/FTS5 with tokenize=simple|porter|icu|unicode61 and all possible remove_diacritics=* values and executed this on all relevant API levels (21-34).

The results (see project for test and CI code):

FTS5 is not supported on Android at all.
Almost everything in FTS3 and FTS4 works on all versions, except the tokenize=unicode61 'remove_diacritics=2' combination I wanted, which throws up on API 21-29:
```
android.database.sqlite.SQLiteException: unknown tokenizer (code 1 SQLITE_ERROR)
```
and works from 30 onwards.

I would like to call out a few things here:

tokenize=unicode61 'remove_diacritics=0' and tokenize=unicode61 'remove_diacritics=1' works, but tokenize=unicode61 'remove_diacritics=2' doesn’t.
The same problem occurs with FTS3 and FTS4.
API level 30 and onwards works, so there must be something with that version. ³

In Depth Research

I wanted to answer the question: why does tokenize=unicode61 'remove_diacritics=2' only work on API 30 and above, if FTS4 and unicode61 already works from API 21. ⁴

Let’s start with what we already know from Android’s SQLite package:

Android API	SQLite Version	Status
API 21	3.8	fails
…	…	fails
API 28	3.22	fails
API 29	3.22	fails
API 30	3.28	works

From here I’ve reviewed all the SQLite release notes between 3.22 and 3.28 looking for FTS/tokenizer/unicode61 keywords. There were 3 release notes that contained these, but the big reveal was in 3.27.0:

Added the remove_diacritics=2 option to FTS3 and FTS5.

So this confirms why it started to work in API 30.

To double-check this finding, I looked at the FTS3 documentation before and after that release, and found that the sentence about remove_diacritics=2 was added after 7th February 2019 which is exactly the release date of SQLite 3.27.0.

Summary ⁵

Android API 30 (including SQLite 3.28) is the first one that supports remove_diacritics=2, which was added in SQLite 3.27.0.

Date	What happened?
2007-09-04	SQLite 3.5.0 was released with FTS3 (no mention in release notes, but in documentation).
2010-12-07	SQLite 3.7.4 was released with FTS4.
2012-06-11	SQLite 3.7.13 was released with the `unicode61` tokenizer.
2014-08-15	SQLite 3.8.6 was released where the `unicode61` tokenizer is included in FTS4 by default.
2014-11	Android 5 / L / API 21 was released including SQLite 3.8 .6.
2018-01-22	SQLite 3.22.0 was released.
2018-08	Android 9 / P / API 28 was released including SQLite 3.22.
2019-02-06	archive.org took a snapshot before the release.
2019-02-07	SQLite 3.27.0 was released including support for `remove_diacritics=2`
2019-03-24	archive.org took a snapshot of the FTS3 documentation including `remove_diacritics=2` mention.
2019-04-16	SQLite 3.28.0 was released.
2020-09	Android 11 / R / API 30 was released including SQLite 3.28.
2023-…	Android developers are reading the latest version of the FTS documentation.

Conclusion

When it comes to Android, we need to be more aware of versioning and history.
After 10 years I’m still not used to this, as showcased by this above story. SQLite team is wonderful at keeping a version history, archive.org is a godsend when it comes to time travel.
Read the error message, and write good error messages!
Think about how much difference it had made if the error was:
```
SQLiteException: unknown tokenizer option value 2 for remove_diacritics
```
Test assumptions if possible.
Luckily I tested on my minSdk version before releasing this time and found the problem early.

You can substitute many versions here for a few coming years. According to apilevels.com which sources data from GlobalStats. ↩
Foreshadowing: At this point something was fishy to me: using the first SQLite version where FTS4 was available had “historical reasons” already. But I dismissed feeling on account of the high of the great feasibility news. ↩
I set out to find and document the exact problem here, the end result is what you’re reading right now. ↩
In hindsight the answer is very simple, but finding it wasn’t that trivial. ↩
Research based on Android Release dates, Android embedded SQLite versions, SQLite FTS3 and FTS4 documentation and its historical snapshots on archive.org, SQLite release dates and release notes. ↩

Practical JFixture

2019-03-10T00:00:00+00:00

Tests need data, let’s see how can we create them using JFixture. In this article we’ll go through some practical usage patterns of JFixture, and compare those to some other alternatives.

Introduction

JFixture is a Java library to assist in the writing of Unit Tests, particularly when following Test Driven Development. It generates types based on the concept of ‘constrained non-determinism’, which is an implementation of the Generated Value xUnit test pattern. JFixture README.md

In practice this means, is that JFixture can create any data object with very little developer effort, here’s an example:

private val fixture = JFixture()

@Test fun test() {
    val fixtJourney: Journey = fixture();

    val result = sut.process(fixtJourney);

    assertNotNull(result);
}

Compare this to the classic approach where we would need to create each object individually, filling in the properties with dummy values, and potentially creating constants for these values. No matter how deep or complicated it gets, JFixture usually finds a way to create an instance with all the data filled in.

val journey = Journey(
    "",
    listOf(
        Leg(
            Stop("", ""),
            LocalDateTime.now(),
            TransportMode.TRAIN,
            Stop("", ""),
            LocalDataTime.now()
        ),
        Leg(
            Stop("", ""),
            LocalDateTime.now(),
            TransportMode.TRAIN,
            Stop("", ""),
            LocalDataTime.now()
        )
    )
)

Introduction
The need for JFixture
Data-oriented testing
Best practices
Pitfalls
Conclusion
References

Note: In this article I use JUnit Jupiter and Mockito + Mockito Kotlin as the testing frameworks; and Kotlin as the language, because of its conciseness. While JFixture was written for the JVM with Java in mind it works for Kotlin too due to Kotlin’s awesome interoperability. Everything you see here applies to Java as well (unless it’s about using some Kotlin language feature).

The need for JFixture

In this section, we’ll go through developing and testing a fictitious component. Starting simply, and getting more complicated, we’ll find solutions to problems that come up on the way.

From the ground up

Let’s imagine we’re writing a user interface for displaying some info based on a journey. Journey data is coming from a data source, and we transform it to display it on the UI.

Let’s start with a simple data structure:

data class Journey(
    val id: String,
    val origin: String,
    val destination: String
)

data class Model(/*...*/)

Additionally, let’s say this is the class that orchestrates the loading and displaying:

class JourneyPresenter(
    private val view: JourneyView,
    private val dataSource: DataSource<Journey>,
    private val mapper: (Journey) -> Model
) {
    fun load(journeyId: String) {
        dataSource
            .getById(journeyId)
            .map(mapper)
            .subscribe { model -> view.show(model) }
    }
}

Here are the collaborator interfaces for completeness (using RxJava’s Single):

interface JourneyView {
    fun show(model: Model)
}
interface DataSource<T> {
    fun getById(id: String): Single<T>
}

To test this, we will need to mock the collaborators and stub their inputs and return values. Mind you, we’re testing the data flow here: dataSource → mapper → view. At this point, we don’t really care what the data is, as long as it’s of the right type:

private val mockView: JourneyView = mock()
private val mockDataSource: DataSource<Journey> = mock()
private val mockMapper: (Journey) -> Model = mock()

private val sut = JourneyPresenter(mockView, mockDataSource, mockMapper)

@Test fun `loads journey and presents it to view`() {
    whenever(mockDataSource.getById(fixtJourneyId)).thenReturn(fixtJourney)
    whenever(mockMapper.invoke(fixtJourney)).thenReturn(fixtModel)

    sut.load(fixtJourneyId)

    verify(mockView).show(fixtModel)
}

You might notice, some of the declarations are missing, namely fixtJourneyId, fixtJourney and fixtModel. These are left out, as they will be the focus of the next sections.

Classic approach

Usually when we’re faced with a problem of creating an instance of a class, we use instantiation (the good old new in Java). The simplest way to fill in the missing pieces above is this:

private val fixtJourneyId = ""
private val fixtJourney = Journey("", "", "")
private val fixtModel = Model("", 0, 0)

Here, we filled in the values, so that it compiles. At the same time, we don’t really care about what the values are, so we mostly use default values such as use empty strings, false, or 0s.

More complex model

Let’s expand our very simple model to a more realistic example:

data class Journey(
    val id: String,
    val legs: List<Leg>
)

data class Leg(
    val origin: Stop,
    val departure: LocalDateTime,
    val mode: TransportMode,
    val destination: Stop,
    val arrival: LocalDateTime
)

enum class TransportMode { WALK, TRAIN, TAXI }

data class Stop(
    val id: String,
    val name: String
)

Now let’s update our classic approach:

private val fixtJourney = Journey("", emptyList())

OK… I took a shortcut here, but what will happen if we can’t assume to have an empty list there? Say, the production code needs to loop through the list and we need to check the outcome based on that. Let’s go with the assumption that we need at least 2 legs:

private val fixtJourney = Journey(
    "",
    listOf(
        Leg(
            Stop("", ""),
            LocalDateTime.now(),
            TransportMode.WALK,
            Stop("", ""),
            LocalDataTime.now()
        ),
        Leg(
            Stop("", ""),
            LocalDateTime.now(),
            TransportMode.WALK,
            Stop("", ""),
            LocalDataTime.now()
        )
    )
)

What can we observe so far? Instantiate one Leg, instantiate multiple Stops, pick an TransportMode to use; and then instantiate another leg, and so forth. It’s simply too much to write, read, and maintain; and when we scale it, there’s repetition.

Enter factories

What do we do when we see repeated code? We extract methods/classes. So we go on and refactor our code to introduce a factory method for leg creation:

private val fixtJourney = Journey("", listOf(createLeg(), createLeg()))

private fun createLeg() = Leg(
    Stop("", ""),
    LocalDateTime.now(),
    TransportMode.WALK,
    Stop("", ""),
    LocalDataTime.now()
)

Cool, it works! Using code looks clean, but is that scalable? How many factory methods should we create to cover all the model types. Is it easily maintainable? How many variations of the same factory method should we have to satisfy all the test scenarios? Okay, we could parameterise each factory method by passing in the parameters that we are interested in; we could even leverage Kotlin’s default parameters for functions. But…, what happens if it’s a complex model? How many parameters and/or overloads should we have?

Enter builders

So then, let’s move on to the next solution… builders! The perfect tool to solve all our issues in terms of maintainability, is it not? It alleviates the repetition and allows dynamism for creating objects; for example something like this would help with Journeys:

class JourneyBuilder(
    private var id: String = "",
    private val legs: MutableList<Leg> = mutableListOf()
) {
    fun build() = Journey(id, legs)

    fun setId(id: String): JourneyBuilder = this.apply { this.id = id }
    fun addLeg(leg: Leg): JourneyBuilder = this.apply { legs.add(leg) }
}

class LegBuilder(
    private var origin: Stop = Stop("", ""),
    private var departure: LocalDateTime = LocalDateTime.now(),
    private var mode: TransportMode = TransportMode.WALK,
    private var destination: Stop = Stop("", ""),
    private var arrival: LocalDateTime = LocalDateTime.now()
) {
    fun build() = Leg(origin, departure, mode, destination, arrival)

    fun setOrigin(origin: Stop): LegBuilder = this.apply { this.origin = origin }
    fun setMode(mode: TransportMode): LegBuilder = this.apply { this.mode = mode }
    // ...
}

Phew, that was a lot of code to type — and I didn’t even write out all the properties (e.g. destination), and sub-builders (e.g. Stop), nor did I use Kotlin DSL and lambdas.

Anyway, let’s use this builder to replace the factory:

private val fixtJourney = JourneyBuilder()
    .addLeg(LegBuilder().build())
    .addLeg(LegBuilder().build())
    .build()

Nice! Easy to read, we can create our own domain specific language. But I think it’s easy to see how the builder approach might get complicated really fast as the system grows and more data types, properties are added and combined together in various ways.

JFixture to the rescue

Remember JourneyBuilder we created just now? Turns out a very similar effect can be achieved by using the JFixture library:

private val fixtJourney: Journey = fixture.create(Journey::class.java)

… but what does this do? and what are the benefits? That’s what we’ll explore next.

How does JFixture create an object?

When we call fixture.create(Class), JFixture tries numerous ways to create our objects:

built-in types ( see Default Supported Types and DefaultEngineParts )
customise() (if any)
- lazyInstance
- sameInstance
- useSubType
public constructor with the least number of arguments
public factory method with the least number of arguments
package visible constructor with the least number of arguments

Pro tip: public constructor with the least number of arguments can be changed to most when a class has overloaded constructors:

fixture.customise(GreedyConstructorCustomisation(SomeType::class.java))

When it succeeds, after creation it’ll also use:

customise()
- propertyOf
- intercept
setters for mutable properties
write visible mutable fields

Notice that while this may seems like a lot to digest at first, we’re usually only using 1 or 2 of these.

Benefits: No builder required

JFixture is a builder by itself, except it adapts to the shape of your code automatically.

less code to write: We can save a lot of code that would just create our objects by using the library.
less code to maintain: Every time a new property is added all the usages of the class have to be revised and the constructor calls/builders adjusted. Most of the time these are unrelated places and the new field doesn’t affect behaviour of the tests, yet we still have to modify them. With builders the situation is much better than the classic or factory approach, but maintenance is still needed.
customisation of objects is still possible: The flexibility of the builder is that we can pick which parts to be set and how. For example a TRAIN-only journey with multiple legs; this is still easily possible.

Benefits: Data is randomised

Don’t worry, it’s the good type of random.

no conflicts from random data

Consider this simple thing:

 val journeys = setOf(
     JourneyBuilder().addLeg(LegBuilder().build()).build(),
     JourneyBuilder().addLeg(LegBuilder().build()).build()
 )
 assertThat(journeys, hasSize(2))

With the hand-built builders we would have to write randomising code ourselves to get this to pass. What’s even worse: sometimes it would pass, sometimes it wouldn’t; depending on what time it is now and how fast our computer is.

more confidence in verification

Let’s imagine we have a bug, and the production code is using the wrong leg to calculate the origin station’s name. With the current builder implementation each leg’s station names are all the same. This means that we could easily end up with a false positive verification. JFixture puts different data everywhere, so it would catch the problem: With the current builder implementation

 // same as   fixtJourney.legs[i].origin.name
 assertEquals(fixtJourney.legs[0].origin.name, model.origin)

A note on Kotlin

Writing fixture.create(Journey::class.java) over and over again gets old really fast. In Kotlin we can do better:

inline fun <reified T> JFixture.build(): T = create(T::class.java)
inline operator fun <reified T> JFixture.invoke(): T = create(T::class.java)

The first one gives us val fixtString: String = fixture.build() with inferred type arguments. The second one enables val fixtString: String = fixture() similar to mock() (given val fixture: JFixture in scope of course). It’s up to you which way you go, based on how much Kotlin magic is acceptable. I recommend writing extension methods for most of the customise() methods using KClass, or inline + reified so they feel more idiomatic.

I’ll use operator invoke in the rest of the article.

Data-oriented testing

Most of the examples in this section demonstrate features that are also listed on the JFixture cheat sheet.

Until now the focus was on the presenter, and data flow; let’s move on to the JourneyMapper’s tests to demonstrate how JFixture can help.

Controlling collections

Let’s say the model has a field which represents how many changes passengers have to make on a journey:

data class Model(
    // ...,
    val changeCount: Int
)

this is of course calculated as follows:

override fun invoke(journey: Journey) = Model(
    // ...,
    changeCount = journey.legs.size - 1
)

If we were to use the classic approach, tests for this would get unwieldy. Builders are a bit better, but testing for multiple counts in a parameterised test still requires some form of looping, or a custom factory just to create a certain amount of legs. With JFixture this is supported out of the box for any type:

@CsvSource("1, 0", "2, 1", "3, 2", "4, 3")
@ParameterizedTest
fun `changeCount is mapped correctly`(legCount: Int, expectedChanges: Int) {
    fixture.customise().repeatCount(legCount)
    val fixtJourney: Journey = fixture()

    val result = sut.invoke(fixtJourney)

    assertThat(result.changeCount, equalTo(expectedChanges))
}

repeatCount is a built-in method that changes how many elements each fixtured collection will contain; the default is 3.

Warning: This actually does not scale well, but good enough for simple tests.
More on this in “Fine-grained repetition”.

Property customisation

Let’s look at how we can calculate if the journey is train-only:

override fun invoke(journey: Journey) = Model(
    // ...,
    trainOnly = journey.legs.all { it.mode == TRAIN }
)

The tests for this involve setting all the leg modes to TRAIN. With the builders approach we could use LegBuilder.setMode, with JFixture we just say: all transport modes are TRAIN:

@Test fun `trainOnly is true for TRAIN-only journey`() {
    fixture.customise().sameInstance(TransportMode::class.java, TRAIN)
    val fixtJourney: Journey = fixture()

    val result = sut.invoke(fixtJourney)

    assertThat(result.trainOnly, equalTo(true))
}

sameInstance is a built-in method that can be used to make sure a type is always resolved to the same instance.

We need to test another scenario though: when there are no TRAIN legs. In this case each leg should have a mode of anything but TRAIN. Here’s a way to do this:

@Test fun `trainOnly is false for a TRAIN-less journey`() {
    fixture.customise().lazyInstance(TransportMode::class.java) {
        fixture.create().fromList(*Enum.valuesExcluding(TRAIN))
    }
    val fixtJourney: Journey = fixture()

    val result = sut.invoke(fixtJourney)

    assertThat(result.trainOnly, equalTo(false))
}

lazyInstance will be called each time a Leg is being created to fill its mode.
fromList will pick an item from the array.

Enum.valuesExcluding is not in a library, it’s one of the few utilities we use to make parameterised tests and fixturing nicer:

inline fun <reified E : Enum<E>>
Enum.Companion.valuesExcluding(vararg excluded: E): Array<E> =
    (enumValues<E>().toList() - excluded).toTypedArray()

You may notice that this lazyInstance + fromList + valuesExcluding combination has potential for re-use, and you’re right: it is possible to extract customisation logic via SpecimenSupplier and/or Customisation interfaces.

Mutating the immutable

I warn you, things are going to get controversial. You may have noticed that the only var I used is in the builder: we use a lot of immutable data types. These data types still need to be created, and we’re using JFixture as our builders. This is where this very useful small utility comes into play:

fun Any.setField(name: String, value: Any?) {
    this::class.java.getDeclaredField(name).apply {
        isAccessible = true
        set(this@setField, value)
    }
}

An example usage would look like this:

@Test fun `duration returns time for whole journey`() {
    val expectedMinutes: Int = fixture()
    val fixtJourney: Journey = fixture()
    fixtJourney.legs.last().setField("arrival", fixtJourney.legs.first().departure.plusMinutes(expectedMinutes))

    val result = sut.invoke(fixtJourney)

    assertThat(result.length, equalTo(Duration.of(expectedMinutes, ChronoUnit.MINUTES)))
}

You can see that it gives us the benefit of naturally mutating objects that are otherwise immutable. Additionally it really focuses on the bits that we care about and leaves everything else up to JFixture. The reflection may look counter-productive, but in the rare instance we rename properties or change types of properties the tests will fail at run-time; which is slower to detect than a compile error, but we’re still protected from false positives.

Sometimes each object has to contain self-consistent data, for example: each leg has to be a non-zero length travel. In this case intercept comes in handy combined with setField:

fixture.customise().intercept(Leg::class.java) {
    it.setField("arrival", it.departure.plusMinutes(fixture<Long>()))
}

intercept is a built-in method that lets you mutate the created object further after creation. It will be called once for each instance.

JFixture has a propertyOf customisation which we could use instead of setField, but it’s a no-go when using immutable classes. It will only work on vars, not vals.

Fine-grained property customisation

Here’s the thing about lazyInstance: it affects every single object created from the given fixture factory. This may be unwanted, and needs to be resolved. Let’s consider what happens if there are Passengers in the system:

data class Passenger(
    val preferredMode: TransportMode
)

In this case, creating a Journey and a Passenger in the same test would mean we would be always creating them with the same TransportMode when using sameInstance. To make this more explicit we can use:

fixture.customise().intercept(Leg::class.java) {
    it.setField("mode", TransportMode.TRAIN)
}
fixture.customise().intercept(Passenger::class.java) {
    it.setField("preferredMode", TransportMode.WALK)
}

Fine-grained repetition

Similarly repeatCount is a tricky one. It changes each collection’s size at once, which is probably not what we want. For example we may need to test 2 passengers over 3 legs:

data class Journey(
    // ...,
    val legs: List<Leg>,
    val passengers: List<Passenger>
)

val fixtJourney: Journey = fixture()
fixtJourney.setField("passengers", fixture.createList<Passenger>(size = 2))

createList is not a library function, it is one of the extensions we added to help us create lists in a very simple way. JFixture — being designed for Java — has some quirks in Kotlin, so hiding them in utility functions helps to write nicer code:

@Suppress("UNCHECKED_CAST") // can't have List<T>::class literal, so need to cast
inline fun <reified T : Any> JFixture.createList(size: Int = 3): List<T> =
    this.collections().createCollection(List::class.java as Class<List<T>>, T::class.java, size)

Best practices

Now that we have the tools, here are some tips to prevent hurting ourselves.

Shared JFixture

It might be tempting to create globals, but try not to. It’s not worth the hours of debugging when you accidentally end up with the wrong fixtured data because another test customised it in a counter-productive way.

val fixture = JFixture()
// or
object fixture : JFixture() { ... }

@TestMethodOrder(Random::class)
class SharedStateTest {
    @Test fun customiser() {
        fixture.customise().sameInstance(String::class.java, "")

        assertThat(fixture<String>(), emptyString())
    }

    @Test fun annoyedObserver() {
        assertThat(fixture<String>(), not(emptyString()))
    }
}

In the above example the fixture global property outlives the test’s lifecycle and leaks the "" customisation to the other test, which expects JFixture’s default behaviour. This expectation is usually implicit when writing fixtured tests.

As seen above, sharing state between a single test class’ methods is risky. We always use non-static JFixture instance set up in @BeforeEach to prevent cross-test customisations:

private lateinit var fixture: JFixture
@BeforeEach fun setUp() {
    fixture = JFixture()
}

If we share fixture like this and put customisations into the setUp method, we could end up with not fully self-contained tests. Your mileage may vary on how much shared setup is acceptable in your project. This is a similar decision to how/where mock() variables are set up and stubbed.

Note that having private val fixture = JFixture() inside the class may not be enough, because the test runner may create an instance per test class, not per test method: Google @TestInstance(PER_CLASS) for more information on when this could break down.

That being said, sharing setup is a good idea. For example, if all the tests create objects which have CharSequence fields, the following setup makes sure that is possible in each test:

@BeforeEach fun setUp() {
    fixture = JFixture()
    fixture.apply {
        customise().useSubType(CharSequence::class.java, String::class.java)
    }
}

Notice that it’s only the code being shared, not the run-time objects.

Centralized Fixture

Taking it a step further, it’s very likely we’ll end up having very common customisations. CharSequence → String is a good example; it could occur often in Android. It is recommended to create a custom type that builds on JFixture:

class MyFixture : JFixture() {
    init {
        customise().useSubType(CharSequence::class.java, String::class.java)
    }
}

It helps:

sharing customisations
sharing fixture related utility methods
(not so beneficial when we can use Kotlin extension methods, but it is in Java)
creating a better API fitting your style
(if you go with encapsulation over inheritance)

That said, we are not using this pattern, because we prefer easily discoverable and focused test setups.

Stubbing fixture behaviour

Let’s see what happens if our data classes have calculated properties, for example:

data class Journey(
    // ...
    val legs: List<Leg>,
) {
    val changeCount get() = legs.size - 1
}

This is very simple, but it can easily get complicated, in which case setting up the fixture to be exactly correct for the calculated property could be tricky. At this point you might be thinking: let’s mock the logic to isolate it, but it’s on a JFixture generated class. Mockito spies come to the rescue:

val spyJourney = spy(fixture<Journey>())
doReturn(6).whenever(spyJourney).changeCount

Beware: when using spys, we must always use the doReturn(result).when(spy).method(args) pattern, otherwise the real method gets called, which can cause problems.

Tip: even when using annotated setup, it’s possible to combine the two frameworks:

@Mock lateinit var mockView: JourneyView
@Mock lateinit var mockDataSource: DataSource<Journey>
@Mock lateinit var mockMapper: (Journey) -> Model

@Fixture lateinit var fixtJourneyId: String
@Spy @Fixture lateinit var spyJourney: Journey
@Fixture lateinit var fixtModel: Model

private lateinit var fixture: JFixture

private lateinit var sut: JourneyPresenter

@BeforeEach fun setUp() {
    fixture = JFixture()
    FixtureAnnotations.initFixtures(this, fixture)
    MockitoAnnotations.initMocks(this)

    sut = JourneyPresenter(mockView, mockDataSource, mockMapper)
}

The important thing here is to have initMocks after initFixtures.

Pitfalls

No tool is without its drawbacks — here are some interesting problems we have encountered while using JFixture.

Kotlin generics interoperability

Kotlin makes every usage of a generic out T (e.g. kotlin.collections.List<T>) automatically appear in the class files as ? extends T. JFixture doesn’t like this, because there’s no right choice for it to pick when creating test data. The workaround we use is adding @JvmSuppressWildcards on the field declarations like so:

@Fixture lateinit var fixtJourneys: List<@JvmSuppressWildcards Journey>

Customise the right JFixture

When using the annotated setup, it’s quite easy to end up using the wrong JFixture instance and wonder why none of our customisations work:

@Fixture lateinit var fixtModel: Model

private lateinit var fixture: JFixture

@BeforeEach fun setUp() {
    fixture = JFixture()
    fixture.customise()....
    FixtureAnnotations.initFixtures(this)
}

The problem here is using the wrong initFixtures overload. When we create and customise a JFixture instance, we need to call initFixtures(this, fixture); otherwise initFixtures will create a new non-customised JFixture instance.

Beware of overloads

fixture.customise().useSubType(CharSequence::class.java, String::class.java)
fixture.customise().sameInstance(CharSequence::class.java, String::class.java)

Let’s say we wanted to use useSubType, but accidentally used sameInstance. This makes a subtle difference when it’s one of the hundreds of lines of test code. Sadly, this compiles, and when JFixture tries to create an instance of CharSequence, it throws a ClassCastException: cannot cast Class<String> to CharSequence. This is because sameInstance has an overload that matches the arguments:

public interface FluentCustomisation {
    <T> FluentCustomisation sameInstance(Class<T> clazz, T instance);
    <T> FluentCustomisation sameInstance(Type type, T instance);
}

Class<*> implements Type so when the instance argument is not a subclass of T in clazz, the overload resolution finds the Type overload and T becomes Class<String>.

Flaky tests

One of the main benefits of JFixture is reproducibility through constrained non-determinism. In practice this means that if tests fail on one computer (e.g. CI) they’ll fail the same way on another (e.g. dev machine). Mind you, this failure may not be immediate, but it’s eventually reproducible. Don’t fret though, it’s not as bad as the infinite monkey theorem states. In practice, we observed that if we see a flaky test on the CI, we can simply run the test a 100 times and we’ll see some failures:

Here’s an example flaky test that will pass 90% of the time:

enum class Type { Other, Type1, Type2, Type3, Type4, Type5, Type6, Type7, Type8, Type9 }

fun Type.isValidType() = this != Other

@Test fun flaky90() {
    val fixture = JFixture()
    val fixtType: Type = fixture()

    val result = fixtType.isValidType()

    assertTrue(result)
}

This could mean that if you run the test a few times, and the code reviewer runs it, and the CI runs it; it’s still going to pass all those times, but then when you merge to the default branch, it fails. The above example is over-simplified, but even the most complex flaky tests we had of this type weren’t that hard to debug and fix. Notice that the reason for failure is assuming what the fixture will generate. If this isValidType is used in an if, that behaviour will trigger 90% of the time. To fix this test, we need to use valuesExcluding technique described in the Property Customisation section, and write a separate test for Other specifically. Bear in mind that in some cases we need to consider that the isValidType’s input should be parameterised to test all possibilities (not just a random one), but that’s a topic for another time.

Conclusion

Introducing JFixture to an existing project is a big leap, but we think it’s worth the effort and the learning curve is not that bad. There will be times when you scratch your heads, but you’ll learn something new each time, until you develop a solid stable usage pattern. I personally found it very useful to debug into JFixture when something is wrong. Give it a whirl and let me know how you find the framework.

References

Import a GIT repository to an SVN repository's subfolder preserving history

2017-12-01T00:00:00+00:00

I had an interview for which I created a GIT repo, but I wanted to import it into my personal monolyth SVN repo. After some Googling I found some examples, and this was the simplest solution. I decided to re-write from the original it in hopes that it’ll be more discoverable.

Intro

I’ll be doing everything on Windows and using spaces in paths, because that’s trickier, and to show that these things are not evil ;). Obviously, all these work on Unix systems as well, just convert the paths to use forward slashes and ignore the drive letters.

Initial state

GIT repository with a project on a branch
(simple Gradle project shown, with 3 commits that created it)

 + src
     + main
         + java
             - Test.java
 - build.gradle
 - README.md

 main: A --- B --- C

SVN repository with projects in subfolders
(I have 4 projects in 2 groups, with 6 commits that created these)

 + project group 1
      + project 1
      + project 2
 + project group 2
      + project 3
      + project 4

 svn: pg1 --- p1 --- p2 --- pg2 --- p3 --- p4

Goal

To grab the GIT repository contents into project group 2/project 5 subfolder of the SVN repository.

+ project group 1
     + project 1
     + project 2
+ project group 2
     + project 3
     + project 4
     + project 5
        + src
            + main
                + java
                    - Test.java
        - build.gradle
        - README.md

svn: pg1 --- p1 --- p2 --- pg2 --- p3 --- p4 --- A --- B --- C

Set-up

I’ll be using local repositories to do everything; meaning there’s no network requests going on. It should be easy to generalize this to remote repositories.

The GIT repository is located in: P:\temp\git repo

git init "git repo"
cd "git repo"
echo "Test repo" > README.md
git add --all && git commit --message "Init repo"
echo "apply plugin: java" > build.gradle
git add --all && git commit --message "Init project"
mkdir src\main\java
echo "public class Test { public static void main(String... args) { System.out.println(""Hello""); } }" > src\main\java\Test.java
git add --all && git commit --message "Add code"

The SVN repository is located in: P:\temp\svn repo (Note: original repository, not the checked-out files)

svnadmin create "svn repo"
svn mkdir "file://P:/temp/svn repo/project group 1" -m "add project group 1"
svn mkdir "file://P:/temp/svn repo/project group 1/project 1" -m "add project 1"
svn mkdir "file://P:/temp/svn repo/project group 1/project 2" -m "add project 2"
svn mkdir "file://P:/temp/svn repo/project group 2" -m "add project group 2"
svn mkdir "file://P:/temp/svn repo/project group 2/project 3" -m "add project 3"
svn mkdir "file://P:/temp/svn repo/project group 2/project 4" -m "add project 4"
svn checkout "file://P:/temp/svn repo" svn

Sadly I couldn’t find a way yet to clone the SVN repo via git svn with file:/// url, so in the SVN repo you’ll need to svnserve -d --listen-port 1234 -r P:\temp\svn-repo for it to be accessible on svn://localhost:1234 (edit P:\temp\svn-repo\conf\svn-serve.conf to include anon-access=write in [general] block; or otherwise configure your access rights).

Steps

Prepare folder (svn mkdir)
Check out SVN via git (git svn clone)
Link GIT (git remote add)
Link commits (git rebase)
Send to SVN (git merge --ff-only & git svn dcommit)

Create hosting folder in SVN repository

A folder needs to be created to host the project in SVN. This is required to be able to put the branch from GIT repo into a subfolder.

svn mkdir "svn://localhost:1234/project group 2/project 5" -m "prepare project 5"

at this point we’ve added an extra commit to SVN. It would be nice to skip this from the steps, but didn’t find a way yet.

root
    + ...
    + project group 2
         + project 5

svn: pg1 --- p1 --- p2 --- pg2 --- p3 --- p4 --- prep

Prepare bridge repo from SVN

We’ll clone the actual project subfolder from the SVN repo, this is why it was required to exist beforehand.

git svn clone "svn://localhost:1234/project group 2/project 5" "svn via git"

Link up the bridge repo with GIT

cd "svn via git"
git remote add bridge "P:\temp\git repo"
git fetch bridge

main: pg1 --- p1 --- p2 --- pg2 --- p3 --- p4 --- prep
bridge/main: A --- B --- C

It is really important to step into the bridge repo folder, otherwise you may get an error like this:

fatal: Not a git repository (or any of the parent directories): .git

Prepare commits to be put on top of the SVN repo

Create working copy of the GIT repo (bridge/main) and put its history (import) on top of the SVN folder’s history (main).

git checkout -b import bridge/main
git rebase -i main

main: pg1 --- p1 --- p2 --- pg2 --- p3 --- p4 --- prep
import: A --- B --- C
bridge/main: A --- B --- C

main: pg1 --- p1 --- p2 --- pg2 --- p3 --- p4 --- prep
                                                       \
import:                                                  A' --- B' --- C'
bridge/main: A --- B --- C

This is the point where you can edit the history, filter the branch, reword commits, etc.. Just rebase -i main as much as you want.

Finalize the process

git checkout main
git merge --ff-only import

main: pg1 --- p1 --- p2 --- pg2 --- p3 --- p4 --- prep --- A' --- B' --- C'
                                                       \
import:                                                  A' --- B' --- C'
bridge/main: A --- B --- C

After checking that everything is correct git log, gitk and the like push the changes to the SVN repo via git svn dcommit.

References

Color support for XML string resources in Android

2016-09-20T00:00:00+00:00

So you read somewhere that Android supports  tags to color string resources, when used with Resources#getText()? Well, yeah, except…

… there’s no uniform way across SDK platform versions to represent “this text should be red” with just a String resource.

Following is a summary and excerpt of the code handling XML font colors in Android. I tried to make it easier to digest than just raw code. The codes did not change between the listed versions unless otherwise noted. The sources come from

android.content.res.StringBlock: parses tags and attributes
com.android.internal.util.XmlUtils: parses numbers as colors
android.graphics.Color: color handling

(They’re all available in android-sdk/sources folder, but to see them in IDEA you have use the Navigate > File action instead of Navigate > Class because they’re @hide.)

API Level 10, 14—17

Attribute names

fgcolor
(via XmlUtils.convertValueToUnsignedInt -> parseLong)

Base prefixes

0 (octal), 0x (hexadecimal), # (hexadecimal)

Number formats

Bd*, B-d*
(B is the base prefix above, d* is any resonable number of digits in that base)

Color range

full ARGB

Exceptions

null: no effect (guarded in StringBlock)
empty string: crash
@system_color_res: crash NumberFormatException
named color: crash NumberFormatException

public static final int
convertValueToUnsignedInt(String value, int defaultValue)
{
    if (null == value)
        return defaultValue;

    return parseUnsignedIntAttribute(value);
}

public static final int
parseUnsignedIntAttribute(CharSequence charSeq)
{
    String  value = charSeq.toString();

    long    bits;
    int     index = 0;
    int     len = value.length();
    int     base = 10;

    if ('0' == value.charAt(index)) {
        //  Quick check for zero by itself
        if (index == (len - 1))
            return 0;

        char    c = value.charAt(index + 1);

        if ('x' == c || 'X' == c) {     //  check for hex
            index += 2;
            base = 16;
        } else {                        //  check for octal
            index++;
            base = 8;
        }
    } else if ('#' == value.charAt(index)) {
        index++;
        base = 16;
    }

    return (int) Long.parseLong(value.substring(index), base);
}

API Level 18—22

Attribute names

fgcolor, color
(via getColor -> Color.getHtmlColor -> XmlUtils.convertValueToInt -> parseInt)

Base prefixes

0 (octal), 0x (hexadecimal), # (hexadecimal)

Number formats

Bd*, -Bd*, B-d*, -B+d*, -B-d*
(B is the base prefix above, d* is any resonable number of digits in that base)

Color range

partial ARGB (alpha <= 0x7f)

Exceptions

null: no effect (guarded in StringBlock)
empty string: 0xff000000 (= BLACK)
@system_color_res: works
named color: works
(except it may be problematic on API Level 18: the explicit Locale.ROOT is missing from toLowerCase. For example, this causes color="LIME" to be interpreted as lıme in tr-TR (Turkish) locale; notice there is no dot on the i!)
invalid named color: -1 (= 0xFFFFFFFF = WHITE)
invalid format: -1 (= 0xFFFFFFFF = WHITE)
numeric color above 0x7fffffff: -1 (= 0xFFFFFFFF = WHITE)

private static CharacterStyle getColor(String color, boolean foreground) {
    int c = 0xff000000;

    if (!TextUtils.isEmpty(color)) {
        if (color.startsWith("@")) {
            Resources res = Resources.getSystem();
            String name = color.substring(1);
            int colorRes = res.getIdentifier(name, "color", "android");
            if (colorRes != 0) {
                ColorStateList colors = res.getColorStateList(colorRes);
                if (foreground) {
                    return new TextAppearanceSpan(null, 0, 0, colors, null);
                } else {
                    c = colors.getDefaultColor();
                }
            }
        } else {
            c = Color.getHtmlColor(color);
        }
    }

    if (foreground) {
        return new ForegroundColorSpan(c);
    } else {
        return new BackgroundColorSpan(c);
    }
}

public static int getHtmlColor(String color) {
    Integer i = sColorNameMap.get(color.toLowerCase(Locale.ROOT));
    if (i != null) {
        return i;
    } else {
        try {
            return XmlUtils.convertValueToInt(color, -1);
        } catch (NumberFormatException nfe) {
            return -1;
        }
    }
}

public static final int
convertValueToInt(CharSequence charSeq, int defaultValue)
{
    if (null == charSeq)
        return defaultValue;

    String nm = charSeq.toString();

    // XXX This code is copied from Integer.decode() so we don't
    // have to instantiate an Integer!

    int value;
    int sign = 1;
    int index = 0;
    int len = nm.length();
    int base = 10;

    if ('-' == nm.charAt(0)) {
        sign = -1;
        index++;
    }

    if ('0' == nm.charAt(index)) {
        //  Quick check for a zero by itself
        if (index == (len - 1))
            return 0;

        char    c = nm.charAt(index + 1);

        if ('x' == c || 'X' == c) {
            index += 2;
            base = 16;
        } else {
            index++;
            base = 8;
        }
    }
    else if ('#' == nm.charAt(index))
    {
        index++;
        base = 16;
    }

    return Integer.parseInt(nm.substring(index), base) * sign;
}

API Level 23—28

Attribute names

fgcolor, color
(via getColor -> Color.parseColor -> parseLong)

Base prefixes

# (hexadecimal)

Number formats

#xxxxxx, #xxxxxxxx
(# is literally the hashmark, x is hexadecimal digit, strictly 6 or 8 digits)

Color range

full ARGB

Exceptions

null: no effect (guarded in StringBlock)
empty string: 0xff000000 (= BLACK)
@system_color_res: works
named color: works
invalid named color: BLACK (= 0xFF000000)
invalid format: BLACK (= 0xFF000000)

private static CharacterStyle getColor(String color, boolean foreground) {
    int c = 0xff000000;

    if (!TextUtils.isEmpty(color)) {
        if (color.startsWith("@")) {
            Resources res = Resources.getSystem();
            String name = color.substring(1);
            int colorRes = res.getIdentifier(name, "color", "android");
            if (colorRes != 0) {
                ColorStateList colors = res.getColorStateList(colorRes, null);
                if (foreground) {
                    return new TextAppearanceSpan(null, 0, 0, colors, null);
                } else {
                    c = colors.getDefaultColor();
                }
            }
        } else {
            try {
                c = Color.parseColor(color);
            } catch (IllegalArgumentException e) {
                c = Color.BLACK;
            }
        }
    }

    if (foreground) {
        return new ForegroundColorSpan(c);
    } else {
        return new BackgroundColorSpan(c);
    }
}

public static int parseColor(String colorString) {
    if (colorString.charAt(0) == '#') {
        // Use a long to avoid rollovers on #ffXXXXXX
        long color = Long.parseLong(colorString.substring(1), 16);
        if (colorString.length() == 7) {
            // Set the alpha value
            color |= 0x00000000ff000000;
        } else if (colorString.length() != 9) {
            throw new IllegalArgumentException("Unknown color");
        }
        return (int)color;
    } else {
        Integer color = sColorNameMap.get(colorString.toLowerCase(Locale.ROOT));
        if (color != null) {
            return color;
        }
    }
    throw new IllegalArgumentException("Unknown color");
}

Special cases

HTC One M8 4.4.2 where named colors crash with NumberFormatException: Invalid long: "cyan" (sample size is 1, and I’m not sure if it’s a custom ROM or not)

Solution?

Forget colors! But seriously, after days of investigation I found no built-in way of doing it. For full support we have to write code that builds/fixes the ForegroundColorSpans’ alpha values or our own parser. At first I thought it’ll work with color="@res_name", but that format only supports color resources from the framework’s resources, no custom ones.

Here are some partial solutions that does’t require (much) code. Most of the hacks I tried were based on the fact that color is parsed after fgcolor in the code of StringBlock and CharSequence’s spans are created in the order that allows color to win on newer platforms. If you don’t like multiple attributes, you could also nest  tags, then the outermost wins.

: If you want to use a named color you’re in luck! Your designers won’t be happy, but you CAN actually do this. The above solutions work on all 3 variants of code: the old platforms understand fgcolor=ARGB correctly, the new platforms support named colors. Note: this will probably crash on special cases.
: If you don’t care about new platforms… (Who does that?!), then you can use this to support anything below API 23. Old platforms understand fgcolor=ARGB correctly, and the not-so-old ones support the negative hack. Note: on new platforms the color will be black, so be careful if you have a dark theme.
: If you want only alpha < 128 you’re in luck, because all platforms uniformly support fgcolor=ARGB in that range.
Html.fromHtml("text"): If you only want opaque colors and willing to write some minor code, you can use HTML (string can be stored as <![CDATA[ in an XML string resource). But this only supports RGB colors, no alpha channel. Works on all platforms.
…: Please share if you find more minimal-code workarounds, I’ll add it here!

References

Original conversation and investigation

Xalan Java Extensions on Android

2016-08-13T00:00:00+00:00

I wanted to do a simple check to see whether Xalan’s extension functions are available. This would’ve helped speeding up an XML-to-CSV transformation a bit.

<xsl:stylesheet xmlns:str="xalan://java.lang.String">
...
<xsl:variable name="quote">&quot;</xsl:variable>
<xsl:variable name="double-quote">&quot;&quot;</xsl:variable>
<xsl:choose>
    <xsl:when test="function-available('str:replaceAll')">
        <!-- Happy days, fast replacement! -->
        <xsl:value-of select="str:replaceAll(string($text), $quote, $double-quote)" />
    </xsl:when>
    <xsl:otherwise>
        <!-- Oh well, we have this nice XSLT 1.0-compatbile implementation -->
        <xsl:variable name="escaped">
            <!-- @see https://stackoverflow.com/a/10528912/253468 -->
            <xsl:call-template name="string-replace-all">
                <xsl:with-param name="text" select="$text" />
                <xsl:with-param name="replace" select="$quote" />
                <xsl:with-param name="with" select="$double-quote" />
            </xsl:call-template>
        </xsl:variable>
        <xsl:value-of select="$escaped" />
    </xsl:otherwise>
</xsl:choose>

After running it on my Android phone it just kept spewing NullPointerExceptions at me:

W/System.err: SystemId Unknown; Line #98; Column #3; Attempt to invoke virtual method ‘boolean org.apache.xalan.extensions.ExtensionsTable.functionAvailable(java.lang.String, java.lang.String)’ on a null object reference

Everything was still working, but I don’t like warnings, so I started looking for the root cause.

Xalan Version

The source code is not available in the Android SDK. This means that it’s nearly impossible to comperehend what’s going on and use the stepping facilities of the IDE. I didn’t even know where the exception is coming from, because there’s no stack trace.

So I set out to figure out which version of Xalan is running on the device, so I can get the sources. I found that calling org.apache.xml.serializer.Version.main(new String[0]) would give me the answer. It’s not possible to call this from code, not even from an Evaluate while debugging, but the problem was only that it wasn’t known to IDEA, so this worked:

Class
    .forName("org.apache.xml.serializer.Version")
    .getDeclaredMethod("main", String[].class)
    .invoke(null, new Object[] { new String[0] });

The answer was: Serializer Java 2.7.1 on both a 2.3.7 and a 5.0.0 device. So I tried to reproduce the issue on my desktop, but the function-available call worked real good when using xalan:xalan:2.7.1. The problem was only reproducible on the phone. So now, I knew the version of Xalan, and added it to my source path so the IDE can handle debugging. As usual with Android source code, there are some problems with line alignment. I still couldn’t set meaningful breakpoints, but it was possible to set method breakpoints, however slow.

Finding the Sources

So the official Xalan release is not the one I needed. A quick Google search for TransformerImpl.java revealed that it was once available as /luni/src/main/java/... in the AOSP platform/libcore repository. I tried to get the latest by simply jumping to the master branch, but it wasn’t there. Quick cloning and git log -- luni/... showed me that the whole org.apache.xalan package was moved to /apache-xml in f029395. That one still wasn’t on the master branch, so another git log revealed that it was deleted and moved to an external repository in e590b9c. At first, I was puzzled by what that means, but after a quick look at the repository listing I found platform/external/apache-xml. Finally this one contained up-to-date code for Lollipop.

Finally Debugging

Adding the above sources to the IDE source path I was able to browse and debug. I found out the full stack trace:

java.lang.NullPointerException: Attempt to invoke virtual method 'boolean org.apache.xalan.extensions.ExtensionsTable.functionAvailable(java.lang.String, java.lang.String)' on a null object reference
    at org.apache.xalan.transformer.TransformerImpl.functionAvailable(TransformerImpl.java:396)
    at org.apache.xpath.functions.FuncExtFunctionAvailable.execute(FuncExtFunctionAvailable.java:89)
    at org.apache.xpath.Expression.bool(Expression.java:186)
    at org.apache.xpath.XPath.bool(XPath.java:412)
    at org.apache.xalan.templates.ElemChoose.execute(ElemChoose.java:103)
    at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2223)
    at org.apache.xalan.templates.ElemTemplate.execute(ElemTemplate.java:389)
    at org.apache.xalan.templates.ElemCallTemplate.execute(ElemCallTemplate.java:241)
    at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2223)
    at org.apache.xalan.transformer.TransformerImpl.transformToRTF(TransformerImpl.java:1830)
    at org.apache.xalan.transformer.TransformerImpl.transformToRTF(TransformerImpl.java:1752)
    at org.apache.xalan.templates.ElemVariable.getValue(ElemVariable.java:302)
    at org.apache.xalan.templates.ElemVariable.execute(ElemVariable.java:245)
    at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2223)
    at org.apache.xalan.templates.ElemTemplate.execute(ElemTemplate.java:389)
    at org.apache.xalan.templates.ElemCallTemplate.execute(ElemCallTemplate.java:241)
    at org.apache.xalan.templates.ElemApplyTemplates.transformSelectedNodes(ElemApplyTemplates.java:370)
    at org.apache.xalan.templates.ElemApplyTemplates.execute(ElemApplyTemplates.java:175)
    at org.apache.xalan.templates.ElemApplyTemplates.transformSelectedNodes(ElemApplyTemplates.java:370)
    at org.apache.xalan.templates.ElemApplyTemplates.execute(ElemApplyTemplates.java:175)
    at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2223)
    at org.apache.xalan.transformer.TransformerImpl.applyTemplateToNode(TransformerImpl.java:2096)
    at org.apache.xalan.transformer.TransformerImpl.transformNode(TransformerImpl.java:1228)
    at org.apache.xalan.transformer.TransformerImpl.transform(TransformerImpl.java:614)
    at org.apache.xalan.transformer.TransformerImpl.transform(TransformerImpl.java:1145)
    at org.apache.xalan.transformer.TransformerImpl.transform(TransformerImpl.java:1123)

The reason it worked even with the error is because XPath#bool catches the exception and just returns false after logging.

So the question now is: Why is the ExtensionsTable null?. It is only set in one place:

void setExtensionsTable(StylesheetRoot sroot)
		throws javax.xml.transform.TransformerException {
	try {
		if (sroot.getExtensions() != null)
			m_extensionsTable = new ExtensionsTable(sroot);
	} catch (javax.xml.transform.TransformerException te) {
		te.printStackTrace();
	}

The extensions are correctly recognized, I saw earlier during debug that sroot.getExtensions() is a non-null Vector containing one element. After a debug session, it turns out that an exception was thrown, however that printStackTrace had no effect and it wasn’t visible. Don’t fret, just use printStackTrace(PrintWriter(StringWriter)) to get the trace:

javax.xml.transform.TransformerException: java.lang.ClassNotFoundException: org.apache.xalan.extensions.ExtensionHandlerJavaClass
    at org.apache.xalan.extensions.ExtensionNamespaceSupport.launch(ExtensionNamespaceSupport.java:101)
    at org.apache.xalan.extensions.ExtensionsTable.<init>(ExtensionsTable.java:66)
    at org.apache.xalan.transformer.TransformerImpl.setExtensionsTable(TransformerImpl.java:385)
    at org.apache.xalan.transformer.TransformerImpl.transformNode(TransformerImpl.java:1184)
    at org.apache.xalan.transformer.TransformerImpl.transform(TransformerImpl.java:614)
    at org.apache.xalan.transformer.TransformerImpl.transform(TransformerImpl.java:1145)
    at org.apache.xalan.transformer.TransformerImpl.transform(TransformerImpl.java:1123)
Caused by: java.lang.ClassNotFoundException: org.apache.xalan.extensions.ExtensionHandlerJavaClass
    at java.lang.Class.classForName(Native Method)
    at java.lang.BootClassLoader.findClass(ClassLoader.java:781)
    at java.lang.BootClassLoader.loadClass(ClassLoader.java:841)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:469)
    at org.apache.xalan.extensions.ObjectFactory.findProviderClass(ObjectFactory.java:100)
    at org.apache.xalan.extensions.ExtensionHandler.getClassForName(ExtensionHandler.java:65)
    at org.apache.xalan.extensions.ExtensionNamespaceSupport.launch(ExtensionNamespaceSupport.java:76)
    ... 22 more
Caused by: java.lang.NoClassDefFoundError: Class not found using the boot class loader; no stack available

Surprise, surprise, there’s a class missing. I diffed the Android Xalan and Xalan 2.7.1 sources and indeed, the whole XSLTC and most of the extension classes are not included in Android. This exception causes m_extensionsTable to be null and that’s a problem in functionAvailable. So I went ahead tried to ensure that it’s non-null:

// transformer created by factory.newTransformer(...)
TransformerImpl xalanTransformer = (TransformerImpl)transformer;
StylesheetRoot sroot = new StylesheetRoot((ErrorListener)null);
// creates an empty Vector to be returned from getExtensions()
sroot.getExtensionNamespacesManager();
// make sure the workaround works in Marsmallow too
sroot.setSecureProcessing(false);
// sets up an empty ExtensionTable, so functionAvailable works without NPE
xalanTransformer.setExtensionsTable(sroot); // default visibility

It’s worth noting that this code cannot be executed as is, because the compile classpath doesn’t have these classes. However, it’s a trivial conversion to reflective calls.

Summary

This indeed fixed the problem, however it is not a good solution, even if you accept reflective hacks. The problem being that even though the XSLT ran, and there was no error, the test="function-available" check still evaluated to false and the <xsl:template> version was used. I now think it’s pointless to do the hack to ensure an extension table, because the only place where replaceAll could have benefit, it doesn’t work. In the end I just removed the Xalan Java Extension and swallowed the XSLT 1.0-compatible workaround.

Formatting used in this blog

2016-02-16T00:00:00+00:00

I created this blog more than half a year ago and a lot of convetion I tried to introduce was lost in memory. This document will help to remember what’s possible and how it will look like.

It should also help to quickly test if formatting would break with any future updates of the site.

TL;DR

This heading is not part of the TOC. All headings can be linked via their short name: this heading.

Formatting
Code
Quotes
Alerts

Formatting

Belonging words should have   between to prevent wrapping: Papp Róbert.
Long list of alternatives should have  between them to allow wrapping: this one/that one/other thing
Use <samp> for sample output: Exit code: 1.
Use <samp> for math: 4 people × 5 days = 20 man hours.
Use <var> for something representing a number: your age times.
Use  for callout Grammar Nazis
Use  for UI elements: Press Next to proceed.
Use <abbr> to show an abbreviation: shot, but it’s not necessary if the abbr is defined in markdown
Custom colors when referencing something highlighted on an image: Save button.

Code

File extensions with <code>: .css, .js, .html
Paths and file names with <samp>: lib/code/path/file.name
Liquid-like code may need some escaping: {{tag}}
Fencing code blocks’ ticks can be output like this: <code>```</code> → ```.
Don’t be lazy with alternative/code/formatting.

Normal code block

/** Comment */
public void method();

Some extra explanation to help understand the code.

Code block with title

/** Comment */
public void method();

Code block with Liquid code

{% for tag in site.data.tags %}<li>{{ tag.name }}{% unless forloop.last %}, {% endunless %}</li>{% endfor %}

Code block for shell

me@laptop$ command args "more args"
output text from command
though output may be highlighted weirdly

Quotes

Here’s a short quotation which is in the middle of a sentence.

This is a long quotation by someone, normal markdown formatting rules apply:
strong, em, em, strong em, html bold, code, kbd, samp, ins, ~~del~~ TWiStErRob

Program output text output from file.name in library

Alerts

All alerts support markdown and their names are all lowercase, because they’re used as CSS classes, for example TODO is alert todo=. The TODO: prefix is not automatically inserted, it’s for name calling only here.

Alert:
This is like any normal markdown, even when used from non-markdown context:
strong, em, em, strong em, html bold, code, kbd, samp, ins, ~~del~~.

Warning: call out a caveat that’s easy to trigger
This is like any normal markdown, even when used from non-markdown context:
strong, em, em, strong em, html bold, code, kbd, samp, ins, ~~del~~.

Info: supplementary information, for example links to further reading or documentation.
This is like any normal markdown, even when used from non-markdown context:
strong, em, em, strong em, html bold, code, kbd, samp, ins, ~~del~~.

Tip: call out something non-trivial that could help.
This is like any normal markdown, even when used from non-markdown context:
strong, em, em, strong em, html bold, code, kbd, samp, ins, ~~del~~.

Success:
This is like any normal markdown, even when used from non-markdown context:
strong, em, em, strong em, html bold, code, kbd, samp, ins, ~~del~~.

Text:
This is like any normal markdown, even when used from non-markdown context:
strong, em, em, strong em, html bold, code, kbd, samp, ins, ~~del~~.

TODO: reminder to myself that something needs to be done here
This is like any normal markdown, even when used from non-markdown context:
strong, em, em, strong em, html bold, code, kbd, samp, ins, ~~del~~.

Terminal:
This is like any normal markdown, even when used from non-markdown context:
strong, em, em, strong em, html bold, code, kbd, samp, ins, ~~del~~.

How to do a PR?

2015-06-13T00:00:00+00:00

Many times when you ask for a feature in a GitHub issue, you’re welcomed with:

Can you create a PR please?

… even if it’s a one-character typo. Here’s how I create a PR.

What is a PR?

A Pull Request is request created on GitHub to merge code from one fork of the repository to another. Usually there’s a feature/bug branch in the fork and the PR is made against the default branch (usually called main or master) in the base repository.

GitHub setup

As easy as pressing .

Checkout

Click the HTTPS link below the clone URL box and then copy the HTTPS clone URL which can be then used to clone a repo:

me@laptop$ git clone https://github.com/me/repo-name.git .
Cloning into '.'...

Immediately after this I go back to the original repository (where I forked from) and repeat the HTTPS link copying and add the remote:

me@laptop$ git remote add -f mint https://github.com/stranger/repo-name.git # -f == immediate fetch
Updating mint
me@laptop$ git remote set-url --push mint DISALLOWED

This is useful if the original repo is active and you want to keep up to date. I call it mint (after the adjective in “mint condition”), because the normal clone operation creates an origin remote, also mint suggests to keep it clean.

me@laptop$ git remote -v # == git remote --verbose show
mint    https://github.com/stranger/repo-name.git (fetch)
mint    DISALLOWED (push)
origin  https://github.com/me/repo-name.git (fetch)
origin  https://github.com/me/repo-name.git (push)

Update remote

stranger-head-branch is usually the main branch and my-pr-branch’s name is highly correlated with what your PR is representing.

me@laptop$ git checkout stranger-head-branch
Switched to branch 'stranger-head-branch'

me@laptop$ git merge mint/stranger-head-branch
Updating abc1230..abc123f
Fast-forward
...

me@laptop$ git checkout my-pr-branch
Switched to branch 'my-pr-branch'

me@laptop$ git rebase stranger-head-branch
First, rewinding head to replay your work on top of it...
Applying: Commit message 1
Applying: Commit message 2
...

Create PR

me@laptop$ git push --all

Upon success just go to the original repo or your fork on GitHub and you should see a Your recently pushed branches: tip with a Compare & pull request button.

Review changes, fill in description and you’re good to go.

Future of a PR

If the repository owners like what you did and it adheres to rules usually defined in CONTRIBUTING.md they’ll merge your PR and you’ll see a Pull request successfully merged and closed message with a Delete branch button next to it. That helps to keep your fork’s working branches to a minimum. Once it’s merged the commits are copied to their repository so it’s safe to delete your branch.

There’s of course the possibility that something goes wrong, like the CI build fails or there’s some modification needed in the PR. In that case make some more commits to your fork, or even rebase or squash commits; then push it to the working branch and the PR will be updated automatically.

Jekyll enhancements for development

2015-06-09T00:00:00+00:00

Jekyll is nice if you like the defaults, but as soon as you start customizing you run into walls. Here are some tips to make it less painful.

Continous Development
Debugging Liquid
- Debugging Example Walkthrough
- More Debugging options
Hidden Gems
Liquid Error Messages

Continous Development

I’m sure you’ve met this line if you started jekyll serve at least once:

Regenerating: 1 file(s) changed at 2015-06-09 12:34:56 ... done in 3.14 seconds. output from watcher.rb in jekyll-watch gem

Now, if you’re like me, you immediate ask the question: What are those files?, especially when it says multiple files changed and you only changed one.

removed.each { |file| Jekyll.logger.info("Changes:", "Removed #{file.slice(site.source.length + 1, file.length)}"); }
added.each { |file| Jekyll.logger.info("Changes:", "Added #{file.slice(site.source.length + 1, file.length)}"); }
modified.each { |file| Jekyll.logger.info("Changes:", "Modified #{file.slice(site.source.length + 1, file.length)}"); }
print Jekyll.logger.message("Regenerating:", "#{n} file#{n>1?"s":""} changed at #{t.strftime("%Y-%m-%d %H:%M:%S")} ")

...
     Changes: Added _posts/2015-06-09-Jekyll-hacks-for-development
Regenerating: 1 file changed at 2015-06-09 15:18:56 ...done in 3.220606 seconds.
     Changes: Removed _posts/2015-06-09-Jekyll-hacks-for-development
     Changes: Added _posts/2015-06-09-Jekyll-hacks-for-development.md
Regenerating: 2 files changed at 2015-06-09 15:19:00 ...done in 3.286006 seconds.
     Changes: Modified _sass/_09_elements.scss
Regenerating: 1 file changed at 2015-06-09 15:20:27 ...done in 3.239607 seconds.
     Changes: Modified _posts/2015-06-09-Jekyll-hacks-for-development.md
Regenerating: 1 file changed at 2015-06-09 15:20:53 ...done in 3.246334 seconds.

The paths are relative to Auto-regeneration: enabled for '...'.

Debugging Liquid

When copy-pasting-building a site from articles found on the internet you’ll most likely find yourself wondering: Why is it not working?. There’s a built-in | inspect, but it just dumps the text without any escaping or formatting, messing up HTML pages. Luckily Jade Dominguez included a little plugin in his bootstrap which would come to the rescue in this case. The following is a more advanced version of his plugin.

This is compatible with GitHub Pages! The default mode of Jekyll is unsafe, but GitHub starts it with --safe. So plugins can be used during local development; just don’t forget to remove | debugs before commit.

# A simple way to inspect liquid template variables.
# Based on: https://github.com/plusjade/jekyll-bootstrap/blob/master/_plugins/debug.rb
# The filters below can be used anywhere liquid syntax is parsed (templates, includes, posts/pages/collections)

require 'pp'  # obj.pretty_inspect
require 'cgi' # CGI.escape_html

module Jekyll
	# Need to overwrite the inspect methods, because the original uses a strange format
	# and we're trying to output JSON. <> also conflicts with HTML code if output literally.
	class Post
		# Replace original #<Jekyll:Post @id="self.id">
		def inspect
			"{ \"type\": \"Jekyll:Post\", \"id\": #{self.id.inspect} }"
		end
	end
	class Page
		# Replace original #<Jekyll:Page @name="self.name">
		def inspect
			"{ \"type\": \"Jekyll:Page\", \"name\": #{self.name.inspect} }"
		end
	end

	module DebugFilter
		# Returns a highlighted HTML code block displaying the received object.
		# Example usages:
		# * <tt>{{ site.pages | debug }}</tt>
		# * <tt>{{ site.pages | debug: 'pages' }}</tt>
		def debug(obj, label = nil)
			pretty = obj.pretty_inspect
			pretty = pretty.gsub(/\=\>/, ': ') # approximate JSON syntax
			pretty = "#{prefix(obj, label)}\n#{pretty}" # prefix with type
			highlight = Jekyll::Tags::HighlightBlock.new('highlight', 'json', [ pretty, "{% endhighlight %}" ])
			pretty = highlight.render_pygments(pretty, true)
			pretty = highlight.add_code_tag(pretty)
			pretty = pretty.sub(/<div class="highlight">/, "<div class=\"highlight debug\" title=\"#{prefix(obj, label)}\">")
			return pretty
		end

		# Returns a non-highlighted HTML code block displaying the received object.
		# Example usages:
		# * <tt>{{ site.pages | dump_html }}</tt>
		# * <tt>{{ site.pages | dump_html: 'pages' }}</tt>
		def dump_html(obj, label = nil)
			pretty = obj.pretty_inspect
			pretty = CGI.escape_html(pretty)
			pretty = "#{prefix(obj, label)}\n#{pretty}" # prefix with type
			pretty = "<pre class=\"debug\" title=\"#{prefix(obj, label)}\">#{pretty}</pre>"
			return pretty
		end

		# Returns pretty-printed plain text displaying the received object.
		# Example usages:
		# * <tt>{{ site.pages | dump_text }}</tt>
		# * <tt>{{ site.pages | dump_text: 'pages' }}</tt>
		def dump_text(obj, label = nil)
			pretty = obj.pretty_inspect
			return "#{prefix(obj, label)}#{pretty.strip}"
		end

		# Prints pretty-printed plain text displaying the received object to the console.
		# Returns the original object, making it chainable.
		# Example usages:
		# * <tt>{% assign upperTitle = page.title | dump_console | upcase | dump_console %}</tt>
		# * <tt>{% assign upperTitle = page.title | dump_console: 'original' | upcase | dump_console: 'upcased' %}</tt>
		def dump_console(obj, label = nil)
			pretty = obj.pretty_inspect
			puts "#{prefix(obj, label)}#{pretty.strip}"
			return obj
		end

		private
		def prefix(obj, label)
			clazz = "(#{obj.class})" if obj
			label = "#{label}: " if label
			return "#{label}#{clazz}"
		end
	end # DebugFilter
end # Jekyll

Liquid::Template.register_filter(Jekyll::DebugFilter)

Debugging Example Walkthrough

Suppose there’s a tags data file in the format I suggested:

tag1:
  name: "Tag 1 Long Name"
tag2:
  name: "Tag 2 Long Name"

… and you’re trying to list all the tags on the site, like LovesTha did:

<ul>
	{% for tag in site.data.tags %}
	<li><a href='/blog/tag/{{ tag }}/'>{{ tag.name }}</a></li>
	{% endfor %}
</ul>

<ul>
	<li><a href='/blog/tag/tag1{"name"=>"Tag 1 Long Name"}/'></a></li>
	<li><a href='/blog/tag/tag2{"name"=>"Tag 2 Long Name"}/'></a></li>
</ul>

For some reason {{tag}} comes up as tag1{"name"=>"Tag 1 Long Name"} instead of the expected tag1 and {{tag.name}} is empty. Let’s augment the code to see what’s going wrong (the argument to debug and dump_* is optional):

{{ site.data.tags | debug: "data.tags" }}
<ul>
	{% for tag in site.data.tags %}
	<li><a href='/blog/tag/{{ tag }}/'>{{ tag.name | dump_text }}</a>{{ tag | dump_html: forloop.index }}</li>
	{% endfor %}
</ul>

… and here’s how it looks like on the page:

data.tags:(Hash)
{"tag1": {"name": "Tag 1 Long Name"}, "tag2": {"name": "Tag 2 Long Name"}}

nil

1: (Array)
["tag1", {"name"=>"Tag 1 Long Name"}]

nil

2: (Array)
["tag2", {"name"=>"Tag 2 Long Name"}]

From the above the following is revealed:

the iteration was correctly going through the Hash (map) as expected
{{tag.name}} is nil displayed as an empty string
at each iteration tag is an array and not the key of the hash

In Liquid a hash entry is stored as [key, value]; compare to for..in loop in JavaScript.

Based on the above it’s clear that the code must be changed to correctly read the value from the entry:

{% for tag_entry in site.data.tags %}
    {% assign tag_key = tag_entry[0] %}
    {% assign tag_data = tag_entry[1] %}
    <li><a href="/blog/tag/{{ tag_key }}/">{{ tag_data.name }}</a></li>
{% endfor %}

More Debugging options

In case you’re generating something other than HTML, dump_text and dump_console come in handy.

Different languages have different rules, make sure you escape everything accordingly.
For example XML comments cannot contain -- so it’s required to strip (| remove: '--') or collapse them (| replace: '--', '-') to make parsers happy.

<url>
	<!-- {{ link | dump_text | remove: '--' }} -->
	<loc>{{ site.url }}{{ site.baseurl }}{{ link.url | dump_console: "original" | remove: 'index.html' | dump_console: "stripped" }}</loc>
	...
</url>

<url>
	<!-- (Hash){...
		"dir"=>"/blog",
		"name"=>"index.html",
		"path"=>"blog/index.html",
		"url"=>"/blog/index.html"} -->
	<loc>http://localhost:4000/dev/blog/</loc>

     Regenerating: 1 file changed at 2015-07-29 14:16:06
original: (String)"/blog/index.html"
stripped: (String)"/blog/"
original: (String)"/blog/tags/"
stripped: (String)"/blog/tags/"
...

It’s also worth looking at Octopress Debugger which offers different features.

Hidden Gems

Filter Documentation

Not all the filters documented are available when using an older version of either Jekyll or Liquid. The most relevant documentation and list of filters can be found in your local Ruby installation:

Jekyll: lib/ruby/gems/2.2.0/gems/jekyll-2.4.0/lib/jekyll/filters.rb
Liquid lib/ruby/gems/2.2.0/gems/liquid-2.6.2/lib/liquid/standardfilters.rb

Working With Arrays

By default Liquid doesn’t have array manipulation, but our friends at Jekyll were kind enough to implement it, and we can even create new arrays with the split trick.

{% assign r = "," | split: ","    | dump_console: "new array" %}
{% assign r = r   | push: "c"     | dump_console: "insert last" %}
{% assign r = r   | push: "d"     | dump_console: "insert last" %}
{% assign r = r   | unshift: "b"  | dump_console: "insert first" %}
{% assign r = r   | unshift: "a"  | dump_console: "insert first" %}
{% assign r = r   | shift: 2      | dump_console: "remove first 2" %}
{% assign r = r   | pop: 1        | dump_console: "remove last 1" %}
{% assign r = r   | unshift: "z"  | dump_console: "insert first" %}
{% assign r = r   | push: "a"     | dump_console: "insert last" %}
{% assign r = r   | sort          | dump_console: "order by contents" %}

insert last: (Array)["c"]
insert last: (Array)["c", "d"]
insert first: (Array)["b", "c", "d"]
insert first: (Array)["a", "b", "c", "d"]
remove first 2: (Array)["c", "d"]
remove last 1: (Array)["c"]
insert first: (Array)["z", "c"]
insert last: (Array)["z", "c", "a"]
order by contents: (Array)["a", "c", "z"]

Liquid code highlight

Luckily the page’s code is first run through Liquid and then kramdown, so it’s possible to use liquid tags to generate the markdown code. Based on this it’s also possible to nest them weirdly, notice that the ``` pair is interleaving with the raw/endraw tags; I prefer this because the markdown indentation is natural, only the end of the line is a little different.

```liquid{% raw %}
{% assign var = other_var | sort | split: ' ' %}
```{% endraw %}
{: title="caption of code"}

Liquid Error Messages

Maybe https://saimonmoore.com/tumblog/200612/debugging-liquid-templates.html can help with this.

Among other modifications made to my tags display, I wanted to sort tags case insensitively and I got the following error message:

Liquid Exception: no implicit conversion from nil to integer in tags.html

Not knowing which line gave the error, I tried to fix and remove parts of the file which were dealing with numbers like 0 < post.size. After many minutes of trying I found out that the above line gives the message and the problem is that I missed the quotes on downcase: quotes are required for map as method are called via reflection.

{% assign tag_words = site_tags | split: ',' | map: downcase | sort %}

Getting a full Thread Dump on Android

2015-04-13T00:00:00+00:00

The Android operating system does a thread dump whenever it receives SIGQUIT signal. However you need an unlocked device to send that signal. Fear not, there’s a workaround to get the exact same thread dump with a little more effort on a production device with a debuggable application.

Background

There are a few sources on the Internet on how to make a thread dump in Android, but most of them don’t work or are lacking the intrinsic locking information, here’s an example:

If you have a developer / rooted device, you can ask the Dalvik VM to dump thread stacks by sending a SIGQUIT to the app process you’re interested in. How to make Java Thread Dump in Android?

If you try to send a signal on a non-rooted/non-dev device you’ll get:

me@laptop$ adb shell "ps | grep twister"
USER     PID   PPID  VSIZE  RSS     WCHAN    PC         NAME
u0_a504   10904 278   914108 23164 ffffffff 00000000 S net.twisterrob.app

me@laptop$ adb shell kill -s SIGQUIT 10904
/system/bin/sh: kill: 10904: Operation not permitted

An answer to How to stop an android application from adb without force-stop or root? suggests to use run-as:

me@laptop$ adb shell run-as net.twisterrob.app kill -3 10904
run-as: Package 'net.twisterrob.app' is unknown

while the OP states that force-stop and am kill is ruled out, we could use them here:

me@laptop$ adb shell am force-stop net.twisterrob.app
me@laptop$ adb shell am kill --user all net.twisterrob.app

…the first really kills the app, and the second does nothing, in any case there’s no thread dump.

Solution

A solution for the above issues follows, based on the fact that you have control over your own Android process from the app’s code.

Step 0: Fake a UI lock-down

To have something to validate that we have the right information I suggest to lock on a known object. So later we’ll see who/where and when locked. I put the following into my main Activity’s onCreate:

// we want to see what "this" is in a thread dump
synchronized (this) {
	try {
		wait();
	} catch (InterruptedException e) {
		e.printStackTrace();
	}
}

Step 1: Always available debugger break entry point

If you want a thread dump you probably have a deadlock in your code in which case it might not be trivial to know when and where to stop the code execution. You’ll need to start a background thread in your app so that you can stop the execution any time and be able to mess around in immediate mode too.

There’s a Pause Program action available in IntelliJ IDEA’s Debug view:

but you can’t execute any code in that mode:

Target VM is not paused by breakpoint request. Evaluation of methods is not possible in this mode

So to have a live breakable point, put the following snippet somewhere in your app:

new Thread("Debug-Breaker") {
	@Override public void run() {
		while (true) {
			try {
				Thread.sleep(1); // breakpoint here
			} catch (InterruptedException e) {
				e.printStackTrace();
			}
		}
	}
}.start();

Anywhere is ok that you know will surely execute before you need the thread dump. Even an button’s event handler will work. I choose to put mine in:

public class App extends Application {
	@Override public void onCreate() {

Don’t forget to register it in your manifest: <application android:name=".App".

Compile and run your app. If you did step 0 as I did, your app will be plain black/white and do nothing at this point, because wait() blocked the UI thread.

Step 2: Create thread dump

Go to your IDE and do Run > Attach debugger to Android process

and select your app’s process and then go to the place where you have Thread.sleep(1); and put a breakpoint on it:

The breakpoint should be hit immediately:

Now there’s a context you can execute code in, go to Run > Evaluate Expression... and enter:

android.os.Process.sendSignal(android.os.Process.myPid(), android.os.Process.SIGNAL_QUIT)

In the Android LogCat view you’ll see the following information printed:

04-13 16:36:21.971  10904-10954/net.twisterrob.app I/Process﹕ Sending signal. PID: 10904 SIG: 3
04-13 16:36:21.971  10904-10909/net.twisterrob.app I/dalvikvm﹕ threadid=3: reacting to signal 3
04-13 16:36:22.061  10904-10909/net.twisterrob.app I/dalvikvm﹕ Wrote stack traces to '/data/anr/traces.txt'

The app may be still running at this point, but it doesn’t matter any more. Feel free to detach debugger and/or stop it on the phone.

Step 3: Acquire thread dump

Now it should be a simple pull from your device like this, but hey it’s not that simple…

me@laptop$ adb pull /data/anr/traces.txt
failed to copy '/data/anr/traces.txt' to './traces.txt': Permission denied

for some reason I wasn’t able to copy the file, but I can read it… so it’s simple after all:

me@laptop$ adb shell "cat /data/anr/traces.txt" > traces.txt

Step 4: Read thread dump

Now if you open traces.txt on you machine and scroll from the bottom you should find your app:

----- pid 10904 at 2015-04-13 17:16:40 -----
Cmd line: net.twisterrob.app

... lots of stack traces of threads ...

----- end 10904 -----

Here’s the UI thread’s state, remember in Step 0 I synchronized on this, see line #6 (starting with “waiting on”):

"main" prio=5 tid=1 WAIT
    | group="main" sCount=1 dsCount=0 obj=0x418ccea0 self=0x417c7388
    | sysTid=13183 nice=-11 sched=0/0 cgrp=apps handle=1074684244
    | state=S schedstat=( 0 0 0 ) utm=7 stm=5 core=3
    at java.lang.Object.wait(Native Method)
    - waiting on <0x42aeb7a8> (a net.twisterrob.app.MainActivity)
    at java.lang.Object.wait(Object.java:364)
    at net.twisterrob.app.MainActivity.onCreate(MainActivity.java:45)
    at android.app.Activity.performCreate(Activity.java:5426)
    ...

Summary

The above steps should give you access to thread dumps for any application you develop from your real device. The full thread dumps contain locking information like the waiting on line above which helps to diagnose deadlocks and synchronization issues. If you just want to see where your app is at the current time I suggest you use the Threads view of the SDK’s DDMS monitor, or Thread.getAllStackTraces() programmatically or while debugging.

TWiStErRob

Android SQLite FTS4 tokenize=unicode61 remove_diacritics=2

TL;DR

Table of Contents

The High of Feasibility

Testing a Proof of Concept

Compassion

Reproducer

In Depth Research

Summary 5

Conclusion

Recommended reading

Practical JFixture

Introduction

Table of Contents

The need for JFixture

From the ground up

Classic approach

More complex model

Enter factories

Enter builders

JFixture to the rescue

How does JFixture create an object?

Benefits: No builder required

Benefits: Data is randomised

A note on Kotlin

Data-oriented testing

Controlling collections

Property customisation

Mutating the immutable

Fine-grained property customisation

Fine-grained repetition

Best practices

Shared JFixture

Sharing data setup

Centralized Fixture

Stubbing fixture behaviour

Pitfalls

Kotlin generics interoperability

Customise the right JFixture

Beware of overloads

Flaky tests

Conclusion

References

Import a GIT repository to an SVN repository's subfolder preserving history

Intro

Initial state

Goal

Set-up

Steps

Create hosting folder in SVN repository

Prepare bridge repo from SVN

Link up the bridge repo with GIT

Prepare commits to be put on top of the SVN repo

Finalize the process

References

Color support for XML string resources in Android

API Level 10, 14—17

API Level 18—22

API Level 23—28

Special cases

Solution?

References

Xalan Java Extensions on Android

Xalan Version

Finding the Sources

Finally Debugging

Summary

Formatting used in this blog

TL;DR

Table of Contents

Formatting

Code

Normal code block

Code block with title

Code block with Liquid code

Code block for shell

Quotes

Alerts

How to do a PR?

Summary ⁵