This challenge was solved by @R3x and @sherl0ck. This writeup is written by @R3x.

The challenge binary was slightly obfuscated. On reversing it we found that the algorithm for encryption was as such.

1. The VM loads a key from the program it’s running.
2. The input file is divided into 64 bit blocks
3. Each blocked with the key and a its rotated bitwise(in a slightly complicated operation – check the python script below)
4. At each iteration the key is also rotated bitwise (in the same operation as before) and then xored with the encrypted block.

Below is a rough python script that gives us the unencrypted message.

```from textwrap import wrap

keyl = wrap('9c44150b2ee1a9e1',2)[::-1] + wrap('6e9c911af3dcdc08',2)[::-1]
keyl += wrap('f15fe2f95ee45c34',2)[::-1] + wrap('53046e70a80586f0',2)[::-1]
keyl += wrap('74f6eaab10ec319d',2)[::-1] + wrap('2c374053280f7944',2)[::-1]
keyl += wrap('274b2f9567c39a17',2)[::-1] + wrap('5d77702a1df93fd9',2)[::-1]

key = []
for i in keyl:
key.append(int(i, 16))

f = open('data.enc', 'r')
fp = open('data.dec', 'w')

for i in xrange(0, len(cont), 64):
fin = []
dest = []
final = []
for j in range(9, 64):
final.append(cont[i + j])
for j in range(0, 9):
final.append(cont[i + j])
for j in range(64):
dest.append(((ord(final[j]) >> 5) | (ord(final[(j + 1) & 63]) << 3)) & 0xff)
for j in range(64):
fin.append(dest[j] ^ key[j])
keyi = []
keym = []
keyf = []
for j in range(64):
keym.append(((key[j]  1)) & 0xff)
for j in range(51, 64):
keyi.append(keym[j])
for j in range(0, 51):
keyi.append(keym[j])
for j in range(64):
keyf.append(keyi[j] ^ fin[j])
key = keyf
fp.write(''.join([chr(x) for x in fin]))

```

The result of running this script on the encrypted file gives us a png file.