Math Obfuscation of Java Bytecode

Earlier today, I visited the Tigress C obfuscator website today (I haven’t for awhile) and it looks much nicer now ;). Anyways, something that particularly interested me was Tigress’s page on EncodeArithmetic. Since I, a maintainer of a Java bytecode obfuscator, am always looking for cool things to try, found the book they linked of interest. For reference $\neg$ is the bitwise NOT, $\oplus$ is XOR, $\wedge$ is AND, and $\vee$ is OR. If this is interesting to you, consider visiting Wikipedia’s page on boolean algebra. In linked book, several identities are given but here are the ones relevant to today’s blog post:

\[\begin{aligned} -x&=\begin{cases} \neg x+1\\ \neg(x-1) \end{cases}\\ x+y&=\begin{cases} x-\neg y-1\\ (x\oplus y)+2(x\wedge y)\\ (x\vee y)+(x\wedge y)\\ 2(x\vee y)-(x\oplus y) \end{cases}\\ x-y&=\begin{cases} x+\neg y+1\\ (x\oplus y)-2(\neg x\wedge y)\\ (x\wedge\neg y)-(\neg x\wedge y)\\ 2(x\wedge\neg y)-(x\oplus y) \end{cases}\\ x\oplus y&=(x\vee y)-(x\wedge y)\\ x\vee y&=(x\wedge\neg y)+y\\ x\wedge y&=(\neg x\vee y)-\neg x \end{aligned}\]

Now, the relevant opcodes for these identities are ineg, iadd, isub, ixor, ior, and iand.

Note that Java’s logical operator NOT is not present in the identities. This is because $\neg x$ is compiled to $x\oplus-1$.

Also note that these “identities” are not limited to just the ones I just ran through - there are a lot more if you choose to find them yourself. With some creativity, you can apply this not only to integers, but floats as well.

Here are some of what the above identities implemented as a transformer would do:

ineg

Before

iload 0   // x
ineg

After

iload 0   // x
iconst_m1
ixor
iconst_1
iadd

iadd

Before

iload 0   // x
iload 1   // y
iadd

After

iload 0   // x
iload 1   // y
ior
iconst_1
ishl
iload 0   // x
iload 1   // y
ixor
isub

isub

Before

iload 0   // x
iload 1   // y
isub

After

iload 0   // x
iload 1   // y
iconst_m1
ixor
iadd
iconst_1
iadd

ixor

Before

iload 0   // x
iload 1   // y
ixor

After

iload 0   // x
iload 1   // y
ior
iload 0   // x
iload 1   // y
iand
isub

ior

Before

iload 0   // x
iload 1   // y
ior

After

iload 0   // x
iload 1   // y
iconst_m1
ixor
iand
iload 1   // y
iadd

iand

Before

iload 0   // x
iload 1   // y
iand

After

iload 0   // x
iconst_m1
ixor
iload 1   // y
ior
iload 0   // x
iconst_m1
ixor
isub

Comments