VolgaCTF quals 19 – TrustedVM writeup

This challenge was solved by @R3x and @sherl0ck. This writeup is written by @R3x.

The challenge binary was slightly obfuscated. On reversing it we found that the algorithm for encryption was as such.

1. The VM loads a key from the program it’s running.
2. The input file is divided into 64 bit blocks
3. Each blocked with the key and a its rotated bitwise(in a slightly complicated operation – check the python script below)
4. At each iteration the key is also rotated bitwise (in the same operation as before) and then xored with the encrypted block.

Below is a rough python script that gives us the unencrypted message.

from textwrap import wrap

keyl = wrap('9c44150b2ee1a9e1',2)[::-1] + wrap('6e9c911af3dcdc08',2)[::-1]
keyl += wrap('f15fe2f95ee45c34',2)[::-1] + wrap('53046e70a80586f0',2)[::-1]
keyl += wrap('74f6eaab10ec319d',2)[::-1] + wrap('2c374053280f7944',2)[::-1]
keyl += wrap('274b2f9567c39a17',2)[::-1] + wrap('5d77702a1df93fd9',2)[::-1]

key = []
for i in keyl:
    key.append(int(i, 16))

f = open('data.enc', 'r')
fp = open('data.dec', 'w')

cont = f.read()

for i in xrange(0, len(cont), 64):
    fin = []
    dest = []
    final = []
    for j in range(9, 64):
        final.append(cont[i + j])
    for j in range(0, 9):
        final.append(cont[i + j])
    for j in range(64):
        dest.append(((ord(final[j]) >> 5) | (ord(final[(j + 1) & 63]) << 3)) & 0xff)
    for j in range(64):
        fin.append(dest[j] ^ key[j])
    keyi = []
    keym = []
    keyf = []
    for j in range(64):
        keym.append(((key[j]  1)) & 0xff)
    for j in range(51, 64):
        keyi.append(keym[j])
    for j in range(0, 51):
        keyi.append(keym[j])
    for j in range(64):
        keyf.append(keyi[j] ^ fin[j])
    key = keyf
    fp.write(''.join([chr(x) for x in fin]))

The result of running this script on the encrypted file gives us a png file.