NMT:firmware: Difference between revisions

From Lundman Wiki
mNo edit summary
No edit summary
Line 1: Line 1:
== Network Media Tank Firmware ==
== Transcoding ==


I downloaded a random version of the firmware, which looks like:
Transcoding with llink is a relatively new features (as of llink-2.3.2) which still needs a fair bit of work. Each device will need its own settings, and tweaks.


-rw-rw-r--  1 lundman lundman  32380583 Nov  8 00:58 01-13-071101-13-POP-402.zip
As an example, Playstation 3 can not play ''.mkv'' files (as of 3.70 anyway). So we ask llink to change any file ending with ''.mkv'' into a different format. Since it is streaming (not writing the whole encoded file on disk before sending) what we send to the PS3 has to be a format capable of streaming. Like that of MPEG-TS. So we convert the file into a ''.mpg'' file, or rather, ''.mkv.mpg''.


To detect that it is a PS3 that is talking to llink, and not a player that CAN handle ''.mkv'' files, we need to make some match rules. Usually, these operate on the ''User-Agent'' header in HTTP requests, or the ''SERVER:'' tag in UPNP's ''BrowseDirectChildren''.


32Megs compressed. Inside that we have:
To define where to find the transcode binary, I have been using ''ffmpeg'', but other programs could probably be used. In fact, one day one might be able to pass it to a "transcode -for ps3" program, wouldn't that be nice.


-rw-rw-r-- 1 lundman lundman 17970150 Nov  2 11:11 01-13-071101-13-POP-402-000.bin
So, a '''TRANSCODE''' line might look something like:
-rw-rw-r-- 1 lundman lundman      723 Nov  7 23:32 README.txt
-rw-rw-r-- 1 lundman lundman 14536029 Nov  7 15:44 syb8634.nmt
-rw-rw-r-- 1 lundman lundman      108 Nov  2 22:27 usbupdate.html


Looking at the large '''01-13-071101-13-POP-402-000.bin''' file first, we notice that it has a 76 bytes header.
TRANSCODE|USERAGENT=*playstation*|ext=*.mkv|newext=.mpg|args=./ffmpeg -d -i "%s" -threads 4 -vbsf h264_mp4toannexb -f mpegts -acodec libmp3lame -ar 48000 -ab 64k -ac 2 -vcodec copy -async 2 -


It is a 3des encrypted header which contains information about the image. The first 4 bytes contains the length of the header and should match 76.
In this case, we detect a PS3 if the ''User-Agent:'' tag contains "playstation" anywhere in it (case-insensitive).
Next we have the size of the flash counted in DWORDS and next we have the number of blocks. It also includes a CRC and for each block we have an ASCII
We only change files of ''.mkv'' type.
description of the block type followed by the length.
We will produce new files of ''.mpg'' type. (MIME type is then based on ''.mpg'' and becomes ''video/mpeg''.)
The header ends with a couple of version strings. A small program parsing and decrypting the header gives the following:
llink will execute


  Flash size is 33554432? and number of blocks is 3
  '''./ffmpeg -d -i "http ://localhost:8001/path/to/video.mkv" -threads 4 -vbsf h264_mp4toannexb -f mpegts -acodec libmp3lame -ar 48000 -ab 64k -ac 2 -vcodec copy -async 2 -'''
Block 1 type: ker, length 5368832
Block 2 type: xos, length 33012
Block 3 type: app, length 13161224
Version 1: xPe0t2, Version 2: c402dPOPk14


The first block is a romfs called '''SPLASH_BOOT'''.
The important parts here is '''-i "%s"''' would specify input file, in this case llink will replace '''%s''' with the actual URL to play. Note that ffmpeg appears to be very picky with argument order. Ie, specifying '''-i "%s"''' at the end will not work.


00000000  4c 00 00 00 22 5b 91 94  6c 2f 9f 6e 37 20 a1 2e  |L..."[..l/.n7 ..|
'''The final minus "-" is important. It tells ffmpeg to send the stream to stdout. Which is why you have to use ''-f mpegts'' to set the output format.''' Note that stderr would end up in the stream too, so with ffmpeg we use '''-d''' to make it completely silent.
00000010  8f 31 3c cd 61 59 d4 c4  53 aa 66 5b e6 00 ae 58  |.1<.aY..S.f[...X|
00000020  ee 3a 1a 92 0c e6 02 b3  22 8b 29 7c 50 9f 8e d0  |.:......".)|P...|
00000030  87 8a 91 09 32 a9 df df  68 0a 86 43 3d 7c 59 93  |....2...h..C=|Y.|
00000040  ce 85 27 59 56 bd 36 bf  76 8d 6d db 2d 72 6f 6d  |..'YV.6.v.m.-rom|
00000050  31 66 73 2d 00 53 18 f0  40 a7 b4 ad 53 50 4c 41  |1fs-.S..@...SPLA|
00000060  53 48 5f 42 4f 4f 54 00  00 00 00 00 00 00 00 49  |SH_BOOT........I|


If we cut out the first 76 bytes and mount it, we get:
If you wish to see what ''User-Agent:'' headers your device send, use


  -rw-r--r-- 1 root root  881252 Jan  1  1970 10xrpc_xload_audio_ucode_SMP8634_2.7.176sybs1.7972x_GCC4_facsprod.bin
  ./llink -d -v 16
-rw-r--r-- 1 root root  326500 Jan  1  1970 11xrpc_xload_video_ucode_SMP8634_2.7.176sybs1.7972x_GCC4_facsprod.bin
  [skin] checking for matching transcoders to user-agent 'Platinum/0.5.3.0, DLNADOC/1.50'
  -rw-r--r-- 1 root root  30724 Jan  1  1970 12xrpc_xload_demux_ucode_SMP8634_2.7.176sybs1.7972x_GCC4_facsprod.bin
  [skin] matches '*latinum*' (ext '*.avi')
-rwxr-xr-x 1 root root    773 Jan  1 1970 30vsyncparam_SMP8634_2.7.176sybs1.7972x_GCC4_facsprod.zbf
  -rwxr-xr-x 1 root root  34262 Jan  1  1970 31bitmap_SMP8634_2.7.176sybs1.7972x_GCC4_facsprod.zbf
-rw-r--r-- 1 root root    6404 Jan  1  1970 32xrpc_xload_dviinit_prod.bin
-rw-r--r-- 1 root root  189524 Jan  1  1970 33xrpc_xload_irq_handler_SMP8634_2.7.176sybs1.7972x_GCC4_facsprod.bin
-rw-r--r-- 1 root root 3975492 Jan  1  1970 50xrpc_xload_vmlinux_ES4_prod.bin
-rwxr-xr-x 1 root root      64 Jan  1  1970 dvi.bin


Definitely hardware boot. Loads the various microcodes for the Sigma 8635 chip, which personally I am not interested in. Lastly we appear to have the kernel itself at about 4Megs. However, all up, the whole thing is only '''5MB''' in size. So there is more in the first file '''after the romfs'''. One of the values in the header is probably an offset. The size of the romfs is '''roughly''' 5.2MB, or 00531c00 in hex. Romfs header has '''005318f0''', plus 76 bytes at a guess.  
As far as transcoding goes, we do almost none. Video codec (h264) and audio of the ''.mkv'' file is copied without changing. We are just changing transport encapsulation from ''.mkv'' to ''mpeg-ts''. Very low CPU in this case.


But romfs are padded up to nearest 1024. So the size will be '''00531c00''', plus 76 bytes. This becomes '''00531c4c'''.


005318e0  00 00 00 00 00 00 00 40  39 20 28 88 64 76 69 2e  |.......@9 (.dvi.|
To test your own transcoding argument, I would recommend running the command line yourself, like this:
005318f0  62 69 6e 00 00 00 00 00  00 00 00 00 00 00 00 00  |bin.............|
00531900  01 00 00 00 00 00 64 00  72 00 00 00 02 00 00 00  |......d.r.......|
00531910  08 00 00 00 37 00 00 00  0c 00 00 00 89 00 00 00  |....7...........|
00531920  0f 00 00 00 04 00 00 00  33 00 00 00 30 00 00 00  |........3...0...|
00531930  3e 00 00 00 00 00 00 00  03 fe 9b ff 00 00 00 00  |>...............|
00531940  00 00 00 00 00 00 00 00  00 00 00 00 '''00 00 00 00'''  |................|
00531c50  '''05 00 00 00 c0 7c 00 00  00 00 00 00 00 00 00 00'''  |.....|..........|
00531c60  '''00 00 00 00 00 00 00 00  f4 80 00 00 04 00 ff 07'''  |................|
00531c70  '''d5 94 c5 33 40 8d 18 9e  17 41 5c e2 ad 49 9e 19'''  |...3@....A\..I..|
00531c80  '''11 6b a1 d5 21 76 76 4e  10 60 40 9e 7a 1d 01 52'''  |.k..!vvN.`@.z..R|
00531c90  '''89 5f c7 3a 98 bc 7f ef  b5 fe fd fa 7b 36 0b 32'''  |._.:........{6.2|
00531ca0  '''7f 29 ba 91 91 85 2c 77  fe 4a 14 c8 cf 91 0c 34'''  |.)....,w.J.....4|
00531cb0  '''5b 55 44 45 32 85 c7 9f  ed d0 26 d7 93 3d c7 b1'''  |[UDE2.....&..=..|
00531cc0  '''1a 7c 59 6b de db 10 c5  48 da 73 c7 6c a2 f1 0e'''  |.|Yk....H.s.l...|


It is not clear what this block contains but at offset '''00531c68''' we find the length, leading us to the next block at '''0053194c+000080f4'''.
./ffmpeg -i "http ://localhost:8001/path/to/video.mkv" -threads 4 -vbsf h264_mp4toannexb -f mpegts -acodec libmp3lame -ar 48000 -ab 64k -ac 2 -vcodec copy -async 2 - > test-encode.mpg


Let us take a look at 01-15-071218-14-POP-402-000.bin:
I removed the -d so you now get ''stderr'' output and can see any encoding errors. I keep the "-" to send to stream, and sent it to a file "> test-encode.mpg". This is to make sure you are using stream, and not doing formats that can not stream. (Like mp4).


0051EC50  05 00 00 00  C0 7C 00 00  00 00 00 00  00 00 00 00  .....|..........
Then simply test-play the file "test-encode.mpg" on your device to make sure it is ok.
0051EC60  00 00 00 00  00 00 00 00  '''F4 80 00 00'''  04 00 FF 07  ................
0051EC70  D5 94 C5 33  40 8D 18 9E  17 41 5C E2  AD 49 9E 19  ...3@....A\..I..


Next block is thus at '''0051ec50+0000f480''':
In my tests, I am using:


  00526D40  E8 D2 C8 00  31 34 00 00  50 4F 50 00  34 30 32 00  ....14..POP.402.
  ffmpeg version 0.8.4
00526D50  EE DB 2F 12  00 00 00 00  00 00 00 00  00 00 00 00  ../.............
00526D60  1F 8B 08 00  BD D3 67 47  00 03 EC 9A  7B 9C 96 53  ......gG....{..S


What we have at '''00526D60''' is a gzipped tar-image.
Below, insert any good transcoding options you might find.


00526D40  E8 D2 C8 00  31 34 00 00  50 4F 50 00  34 30 32 00  ....14..POP.402.
=== Playstation 3 ===
00526D50  EE DB 2F 12  00 00 00 00  00 00 00 00  00 00 00 00  ../.............
00526D60  1F 8B 08 00  BD D3 67 47  00 03 EC 9A  7B 9C 96 53  ......gG....{..S


Unpack with <code>dd if=01-15-071218-14-POP-402-000.bin bs=5401952 skip=1|tar xzfv -</code>
Change MKV to MPG, appears to work for Scene standard tv.
TRANSCODE|USERAGENT=*playstation*|ext=*.mkv|newext=.mpg|args=./ffmpeg -d -i "%s" -threads 4 -vbsf h264_mp4toannexb -f mpegts -acodec libmp3lame -ar 48000 -ab 64k -ac 2 -vcodec copy -async 2 -


At '''00526D50''' is the CRC of the filesystem. It can be calculated using the following code:


<pre>
=== XBox 360 ===
      for(j = 0; j < length; j++) {
  unsigned int b = buffer[j]<<8;
  for(k=0; k<8; k++) {
    if((b ^ crc) & 0x8000) {
      crc = (crc << 1) ^ 0x1021;
    } else {
      crc <<= 1;
    }
    b <<= 1;
  }
      }
</pre>


If you read the kernel image itself (to find out what filesystems it supports), you will find it has the same header:


00000000  00 00 00 00 05 00 00 00  20 a6 3c 00 00 00 00 13  |........ .<.....|
=== Apple ipad ===
00000010  02 00 00 00 03 00 00 00  04 00 00 00 44 a9 3c 00  |............D.<.|
00000020  0c 00 01 ff 97 80 fc f4  6b 69 38 93 ef 8a 6e cf  |........ki8...n.|
00000320  3c 0f e2 a6 46 4e 49 42  11 00 00 10 00 00 02 90  |<...FNIB........|
00000330  00 00 02 90 ac f6 e5 ba  00 00 03 02 f1 a5 3c 00  |..............<.|
00000340  00 00 00 00 1f 8b 08 08  26 a6 2a 47 02 03 76 6d  |........&.*G..vm|
00000350  6c 69 6e 75 78 2e 62 69  6e 00 ec 5b 0f 6c 1c 55  |linux.bin..[.l.U|
00000360  7a ff 76 76 76 bd 0e 1b  3c 76 36 61 43 02 d9 b1  |z.vvv...<v6aC...|




Indeed all files inside the romfs has this header. Could just mean they are signed. It is interesting that the filename is there is plain-text.
=== Apple ipad 2 ===


The '''syb8634.nmt''' contains a gzipped tarball, which can be extracted with


dd if=syb8634.nmt skip=1 bs=60 | tar -xzvf -
=== Apple iphone ===


=== 50xrpc_xload_vmlinux_ES4_prod.bin ===


The kernel resides in this file and to unpack we can do:
=== Google Android ===
dd if=50xrpc_xload_vmlinux_ES4_prod.bin skip=1 bs=836 |zcat >vmlinux.bin


This image also contains the initial ramdisk which can be extracted to cwd with:
Tested with ''Softmedia Player Trial'' and ''Mirage''.
dd if=vmlinux.bin skip=1 bs=4399104|zcat|cpio -id --no-absolute-filenames


=== fw_image ===
TRANSCODE|USERAGENT=*android*|ext=*.avi|newext=.mpg|args=./ffmpeg -d -i "%s" -acodec libmp3lame -ar 48000 -ab 128k -ac 2 -s 720x480 -vcodec libx264 -b 1200k -flags +loop+mv4 -cmp 256 -partitions +parti4x4+partp8x8+partb8x8 -subq 7 -trellis 1 -refs 3 -coder 0 -me_range 16 -keyint_min 25 -sc_threshold 40 -i_qfactor 0.71 -bt 1200k -maxrate 1200k -bufsize 1200k -rc_eq 'blurCplx^(1-qComp)' -qcomp 0.6 -qmin 10 -qmax 51 -qdiff 4 -level 30 -aspect 16:9 -r 30 -g 90 -async 2 -f mpegts -
Almost works, plays fine for a while, but drops to 5 fps or so at times. Appears not to recover.


The tool '''fw_image''' that is on the platform, has extern references to '''des3_decrypt_block''', so that makes me think it uses des3 on the image, it also has the CRC functions for each part, before flashing them to '''/dev/mtd*'''


The function '''des3_decrypt_block''' is found in /lib/libdes.so.1.0.0 and confirmed to be a standard 3des (at least it is compatible with the one in OpenSSL).
=== More here ===


The function prototype looks like this:
EyeCon sends
  extern void des3_decrypt_block(void *data, int size, uint8_t *key);
  User-Agent: Eyecon ControlPoint 2.0 | DMC/1.0


The key has a length of 16 bytes and includes parity bits. The data buffer serves as both input and output.
AcePlayer sends
Absolutely nothing :(


This makes me confident that I can find the decryption keys should I need to. I have no interest in doing so at this time, I only wanted to know I had the option should Syabas decide to attempt to plug the hole that lets me have root. So even if they did so now, I believe I have all the information needed to decrypt any future firmware (yes, even if they change it in future).
MLPlayer Lite
USER-AGENT: Darwin/11.0.0 UPnP/1.0 DLNADOC/1.50


PlugPlayer
USER-AGENT: iPad/4.3.2, UPnP/1.0, PlugPlayer/3.8.0


=== Other firmwares ===
YXPlayer
User-Agent: Platinum/0.5.3.0, DLNADOC/1.50


Pull out anything starting with '''gzip''' header from a binary file:
Playstation 3


  perl -e '$file="test" ; $A=`cat $file` ; while ($A =~ /\x1f\x8b\x08/g) { $pos = length $`; $pos += 0; printf "%X\n", $pos; system("dd if=$file  of=2test.$pos.gz bs=$pos skip=1 > /dev/null 2>&1 "); system("gunzip -c 2test.$pos.gz > 2test.$pos")};'
  X-AV-Client-Info: av=5.0; cn="Sony Computer Entertainment Inc."; mn="PLAYSTATION 3"; mv="1.0";
 
User-Agent: UPnP/1.0 DLNADOC/1.50
You can then run '''file test.*''' to see interesting things, in particular tarball rootfs, and kernels.

Revision as of 13:35, 5 October 2011

Transcoding

Transcoding with llink is a relatively new features (as of llink-2.3.2) which still needs a fair bit of work. Each device will need its own settings, and tweaks.

As an example, Playstation 3 can not play .mkv files (as of 3.70 anyway). So we ask llink to change any file ending with .mkv into a different format. Since it is streaming (not writing the whole encoded file on disk before sending) what we send to the PS3 has to be a format capable of streaming. Like that of MPEG-TS. So we convert the file into a .mpg file, or rather, .mkv.mpg.

To detect that it is a PS3 that is talking to llink, and not a player that CAN handle .mkv files, we need to make some match rules. Usually, these operate on the User-Agent header in HTTP requests, or the SERVER: tag in UPNP's BrowseDirectChildren.

To define where to find the transcode binary, I have been using ffmpeg, but other programs could probably be used. In fact, one day one might be able to pass it to a "transcode -for ps3" program, wouldn't that be nice.

So, a TRANSCODE line might look something like:

TRANSCODE|USERAGENT=*playstation*|ext=*.mkv|newext=.mpg|args=./ffmpeg -d -i "%s" -threads 4 -vbsf h264_mp4toannexb -f mpegts -acodec libmp3lame -ar 48000 -ab 64k -ac 2 -vcodec copy -async 2 -

In this case, we detect a PS3 if the User-Agent: tag contains "playstation" anywhere in it (case-insensitive). We only change files of .mkv type. We will produce new files of .mpg type. (MIME type is then based on .mpg and becomes video/mpeg.) llink will execute

./ffmpeg -d -i "http ://localhost:8001/path/to/video.mkv" -threads 4 -vbsf h264_mp4toannexb -f mpegts -acodec libmp3lame -ar 48000 -ab 64k -ac 2 -vcodec copy -async 2 -

The important parts here is -i "%s" would specify input file, in this case llink will replace %s with the actual URL to play. Note that ffmpeg appears to be very picky with argument order. Ie, specifying -i "%s" at the end will not work.

The final minus "-" is important. It tells ffmpeg to send the stream to stdout. Which is why you have to use -f mpegts to set the output format. Note that stderr would end up in the stream too, so with ffmpeg we use -d to make it completely silent.

If you wish to see what User-Agent: headers your device send, use

./llink -d -v 16
[skin] checking for matching transcoders to user-agent 'Platinum/0.5.3.0, DLNADOC/1.50'
[skin] matches '*latinum*' (ext '*.avi')

As far as transcoding goes, we do almost none. Video codec (h264) and audio of the .mkv file is copied without changing. We are just changing transport encapsulation from .mkv to mpeg-ts. Very low CPU in this case.


To test your own transcoding argument, I would recommend running the command line yourself, like this:

./ffmpeg -i "http ://localhost:8001/path/to/video.mkv" -threads 4 -vbsf h264_mp4toannexb -f mpegts -acodec libmp3lame -ar 48000 -ab 64k -ac 2 -vcodec copy -async 2 - > test-encode.mpg

I removed the -d so you now get stderr output and can see any encoding errors. I keep the "-" to send to stream, and sent it to a file "> test-encode.mpg". This is to make sure you are using stream, and not doing formats that can not stream. (Like mp4).

Then simply test-play the file "test-encode.mpg" on your device to make sure it is ok.

In my tests, I am using:

ffmpeg version 0.8.4

Below, insert any good transcoding options you might find.

Playstation 3

Change MKV to MPG, appears to work for Scene standard tv.

TRANSCODE|USERAGENT=*playstation*|ext=*.mkv|newext=.mpg|args=./ffmpeg -d -i "%s" -threads 4 -vbsf h264_mp4toannexb -f mpegts -acodec libmp3lame -ar 48000 -ab 64k -ac 2 -vcodec copy -async 2 -


XBox 360

Apple ipad

Apple ipad 2

Apple iphone

Google Android

Tested with Softmedia Player Trial and Mirage.

TRANSCODE|USERAGENT=*android*|ext=*.avi|newext=.mpg|args=./ffmpeg -d -i "%s" -acodec libmp3lame -ar 48000 -ab 128k -ac 2 -s 720x480 -vcodec libx264 -b 1200k -flags +loop+mv4 -cmp 256 -partitions +parti4x4+partp8x8+partb8x8 -subq 7 -trellis 1 -refs 3 -coder 0 -me_range 16 -keyint_min 25 -sc_threshold 40 -i_qfactor 0.71 -bt 1200k -maxrate 1200k -bufsize 1200k -rc_eq 'blurCplx^(1-qComp)' -qcomp 0.6 -qmin 10 -qmax 51 -qdiff 4 -level 30 -aspect 16:9 -r 30 -g 90 -async 2 -f mpegts -

Almost works, plays fine for a while, but drops to 5 fps or so at times. Appears not to recover.


More here

EyeCon sends

User-Agent: Eyecon ControlPoint 2.0 | DMC/1.0

AcePlayer sends

Absolutely nothing :(

MLPlayer Lite

USER-AGENT: Darwin/11.0.0 UPnP/1.0 DLNADOC/1.50

PlugPlayer

USER-AGENT: iPad/4.3.2, UPnP/1.0, PlugPlayer/3.8.0

YXPlayer

User-Agent: Platinum/0.5.3.0, DLNADOC/1.50

Playstation 3

X-AV-Client-Info: av=5.0; cn="Sony Computer Entertainment Inc."; mn="PLAYSTATION 3"; mv="1.0";
User-Agent: UPnP/1.0 DLNADOC/1.50