Mediatomb on NetBSD uPnP to PS3

[sudog]

Desire
I want to stream mod, med, s3m, xm, it modules, flac music files, watch YouTube videos, and see thumbnails on my Playstation 3 (PS3).
Problems

• MediaTomb is running on my NetBSD machine
• pkgsrc doesn't have an updated pkgsrc entry
• The current pkgsrc entry for ffmpegthumbnailer doesn't have a buildlink3.mk
• The pkgsrc entry is 0.11.0 which is basically about 40% broken
• 0.11.0 is the current stable version available to most, if not all, distros including pkgsrc and FreeBSD ports
• 0.12.0 must be built from the code currently in SVN, which must be checked-out manually
• CentOS 5 doesn't have a recent-enough autoconf/automake to build mediatomb from source

Preamble
I have a Playstation 3, and with recent software updates, it now has the ability to browse and play media including H.264, MP4 (via what appears to be a commercial DivX implementation,) all kinds of MPEG1/2 streams, and even some Microsoft WMV type formats from a uPnP media server.

Since the only major piece of software capable of doing this with any alacrity is Mediatomb, I dove right in. Mediatomb, after all, sits pretty in NetBSD's pkgsrc tree, which means I wouldn't have to go through huge gyrations to get it installed, and wouldn't be stuck fiddling around deleting obscurely-installed junk if I wanted to be rid of it or upgrade.

Installing 0.11.0 on NetBSD
Getting 0.11.0 (the version currently in pkgsrc as of this writing) was easy:

pchroot (pop into a waiting pkg_comp chroot'd build environment)
cd /usr/pkgsrc/multimedia/mediatomb
make package
exit

(In real environment:)

pkg_add /usr/pkgsrc/packages/All/mediatomb-0.11.0nb5.tgz

Getting it going on my PS3 was NOT easy. I eventually ended up using config.xml.old (attached) to get it playing basic media mounted by NFS on shog, my NetBSD dual-core EM64T Pentium-D 930 all-purpose machine, from my big multi-terabyte bad-ass server, khan.

This works for the most part, except for the fact that my PS3 is connected via wireless to the rest of my network. If it is ever starved of video data, it throws up error messages and the movie has to be stopped and started again. Bandwidth must be carefully, carefully monitored and manually conserved either from the transcoding process itself (which can be set for a maximum bandwidth utilisation) or by judicious use of pause, or, in worst case, stopping and restarting the movie.

Luckily the PS3 knows where it left off, so stopping and restarting is a pretty painless way to fix movie hiccoughs.

The helper script mencoder-tr (attached) was what I was using to generically transcode media from PS3-unknown formats on-the-fly to my PS3 in a clean streaming format it was natively capable of playing--specifically MPEG-2. Originally I'd been using the commented-out mencoder section, but abandoned that because for some reason video and audio were desync'ing unpredictably. So, I sacrificed external subtitles (usually the .srt files) for the stability of ffmpeg. Oh well. (And yes, I checked the source. ffmpeg currently has no concept of .srt files. Just mpeg-style icon overlay subtitles.)

Changing the uPnP MediaTomb icons
As an aside, I got irritated with the MT playbutton Mediatomb uPnP icon displayed on the browser screens of the PS3 and ended up changing it to something else after grovelling through the source for its actual location, because this turned out not to be documented anywhere. But it was easy enough to find: all that needs replacing are the three icons in the web/icons directory:

/usr/pkg/share/mediatomb/web/icons/
    mt-icon120.bmp
    mt-icon120.jpg
    mt-icon120.png

The icons must be 120x120 square, but can otherwise be anything.

So far so good: 0.11.0 works acceptably, but after fiddling for a few hours trying to stream my old Amiga mod collection using the attached "mikmod-tr" translator, and failing over and over with that obnoxiously unhelpful "Data type not supported." error message from the PS3 no matter what I did, I thought..  ah screw it. I'll try compiling mediatomb from SVN and see if I can get YouTube videos going at least. Plus I saw this URL here which suggested there was an issue with streaming PCM from Mediatomb-0.11.0 and the PS3:

ChangeLog, SVN revision 1979, prior to 0.12.0 release

Note the ChangeLog entry:
   16           added fixes to allow PCM playback on the PS3 and other devices,
   17           thanks to mru for his findings. This allows streaming transcoded
   18           OGG and FLAC media to the PS3."

So it turned out that 0.11.0 was broken with the PS3 w.r.t. streaming PCM streams.. regardless of all the references to PCM and OGG and FLAC in the sample config.xml file. Doh! Major psyche-out!
Building a new Mediatomb pkgsrc entry for SVN-derived sources
First I thought it would be easier to build and run MediaTomb on my Linux machine because that's where it was developed. Unfortunately, CentOS 5 (running on khan) doesn't have the most up-to-date autoconf and automakes, which MediaTomb absolutely requires to build. CentOS installs autoconf 2.59, but MediaTomb's configure.ac bitches and whines about a minimum autoconf 2.61 or later. Oh well. I can see Linux would be an uphill battle..  this particular problem is easy to solve, but endless fiddling around in Linux, and then tweaking a spec file to get rpmbuild to package it up properly..  my god.. what a pain.

Instead, this turned out to be way easier to do on my NetBSD machine than I thought it was going to be. All the NetBSD-specific patches were helpfully added to the code by the MediaTomb people, so grabbing the latest SVN copy with:

svn co https://svn.mediatomb.cc/svnroot/mediatomb/trunk/mediatomb mediatomb

... obviated the need to use the pkgsrc patches in the 0.11.0 MediaTomb pkgsrc directory. What I decided to do was simulate a 0.12.0 release.

mv mediatomb mediatomb-0.12.0
(The above is necessary because pkgsrc expects the tarball to extract into a directory named after the pkgsrc entry and version. It's not strictly necessary, but it does allow me to use the default pkgsrc behaviour without modifying the pkgsrc Makefile too much.)
tar -czvf mediatomb-0.12.0.tar.gz
(The above creates a distfile which we can use to trick pkgsrc into starting the build process.)
mv mediatomb-0.12.0.tar.gz /usr/pkgsrc/distfiles/

Using the attached patch, I built a fresh pkgsrc entry which can be used to build from directly. Additionally, to get the ffmpegthumbnailer correctly building into MediaTomb and being recognised during the build process I had to add the attached buildlink3.mk file to ffmpegthumbnailer, and then include it via the mediatomb-svn Makefile (note the references in the patch.)

Once all this was done, it could be forcefully built with:

pchroot
(Back in built chroot bowl..)
cd /usr/pkgsrc/multimedia/mediatomb-svn/
make clean
make NO_CHECKSUM=yes package
Updating config.xml
It turns out that the best way to get a sample config.xml is to let MediaTomb build one from scratch after it tries and fails to locate one. After that, it was a simple matter to merge the two of them together to maintain all the transcoding work I'd done. And guess what?

I can now stream ALL my Amiga mods.

But sudog, you might be thinking: what's this uade123 thing in your mikmod-tr?

Well, uade123 is a stripped-down Amiga simulator which is capable of playing obscure Amiga music module formats--hundreds of them--by emulating an Amiga CPU and music chips, running some native Amiga software on them, and virtually capturing the output in sample form. Neat huh?

Except uade is NOT really natively available in too many places. Luckily downloading it and building it are easy. You can get it from here:

UADE Downloads Page

... and make sure you get the latest version. Building it is a snap:

tar -xzvf uade-2.12.tar.bz2
(On NetBSD, you don't have to use 'j' to decompress .bz2. That's a stupid, stupid lame Linux-ism.)

It does require libao, but that's in pkgsrc so.. meh.

Building it is a pseudo-snap:

cd uade-2.12
./configure --only-uade123
gmake
gmake install
(Dumps it unceremoniously into /usr/local/. Ah well. Can't be perfect all the time.)

./configure --only-uadecore
gmake
gmake install
(Again..  same place.)

And now we have uade123.
NOTE:

This is really important. In total, this software and my scripts absolutely require all kinds of additional software to function, from bash for the scripts (sh should work though) to uade, to sox, to mikmod, to mediatomb and all the libraries it depends on. I haven't done an exhaustive step-by-step here because all the other little bits are lame and boring. But you might run into them if you try to duplicate my efforts.

Still, nothing quite beats watching YouTube in a virtual directory, seeing thumbnails for all kinds of video and avi, listening to Amiga MOD music through my PS3, finally being able to play my substantial FLAC collection, and so on.

As a bonus, and because I forgot to do it earlier, here are the *-tr scripts I use for the whole kit.

In the near future..

Coming soon! MOD and other music formats, in compressed form, handled automatically. Auto-on-the-fly extraction from .LZH, .LhA, and .LZX forms, and .ZIP too, for all those great Modarchive and UnExotica modules I have.. (all 45GB of them.)
Unsolved Issues

• Wireless Bandwidth Lameness (electrician wiring needed)
• Correct WAV file playing
• OGG file testing
• YouTube videos quitting in the middle of playing with [flv @ 0x520000]skipping flv packet: type 67, size 7302772, flags 0

[sudog]

Cool..  Apple Trailers are now available through a virtual directory/interface. I just found out they're making an AstroBoy movie. AWESOME! :-)

Other bits and pieces of discoveries:

IT IS possible to pause streaming media on the PS3

... all you have to do is press the PS button and skip to the main menu screen for a moment. The video pauses in the background.

[sudog]

A new wav translator using correct arguments in sox..  seems to work with all the samples I've tried it with so far.

I think there are some types of WAV file that sox doesn't correctly support--that use obscure encodings and usually originate in Windows. But, for those that sox can understand, this translator will work: and so far this includes every sample I've tried.

[sudog]

It looks a lot like this guy here (old-school NetBSD'er) The zhadum guy has also set up his NetBSD machine to be a mediatomb streaming server to his PS3..  but not quite to the extent that I have. So far anyway! :-) (Perhaps he's done a pile more since the article and I can get some more tips out of him.. heh heh.)

[sudog]

SID transcoding is now working

Unfortunately, the problem was that none of the sidplay/sidplay2 packages are capable of streaming a WAV out properly to stdout. They have issues with it because of a peculiar little problem with C++'s ofstream classes and the way it tries to do an implicit lseek() depending on the mode you feed it to open up a file for writing.

My method of playing SID tunes with these misbehaving programs relies on a special UNIX-y file called /dev/stdout, which when opened basically tricks your program to opening its own stdout channel on another file descriptor when your program is too stupid to do it on its own..  but since stdio streams aren't seek-able, this unfortunately means that the implicit ofstream lseek() going on was failing.

Also, another thing sid2wav was doing was trying to ftell() on the stream (once I'd fixed the ofstream open mode). Well, what does that mean on an stdio stream? Nothing. So that would fail too. But why bother? Why not just write to the file the way the user requested?

So I had to patch sid2wav which hasn't really been maintained in like..  I dunno.. like five years or more. The homepage was an old Geocities homepage and that disappeared long ago when Geocities stopped being cool and started being site-killing turkeys. The old maintainer could've just moved to Sourceforge, but he seems to have disappeared (except I managed to find a possible modern reference to him in the Fedora Project..  see below for a URL..) Too bad, that old page had some dynamite material on it.

Meh..  what with the Nelly Furtado/Timbaland "Do It" fiasco where Timbaland made her look bad by essentially stealing wholesale a SID tune from some European guy, and the truly old-school popularity of the old C64 tunes and games, it's not surprising that sidplay and sidplay2 have enjoyed popularity long after the projects actually died.

Solution
Enough jerking around. The solution to get Mediatomb streaming SID chiptunes is as follows:

Modify config.xml to include the following additional sections:

in <mappings><extension-mimetype> add the following:
        <map from="sid" to="audio/x-sid"/>
        <map from="SID" to="audio/x-sid"/>


I know audio/x-sid isn't the official mimetype for these files. According to This IANA webpage here the official mimetype is audio/prs.sid, but at this point I'm committed so I'm continuing to use audio/x-sid.

In <mimetype-contenttype> add the following entry:
        <treat mimetype="audio/x-sid" as="pcm"/>


In <transcoding enabled="yes"><mimetype-profile-mappings> add the following transcode mapping:
        <transcode mimetype="audio/x-sid" using="sid-tr"/>


And finally, in the <transcoding><profiles> section, add the following profile entry:
      <profile name="sid-tr" enabled="yes" type="external" >
        <mimetype>audio/L16</mimetype>
        <first-resource>yes</first-resource>
        <accept-url>yes</accept-url>
        <sample-frequency>44100</sample-frequency>
        <audio-channels>2</audio-channels>
        <hide-original-resource>yes</hide-original-resource>
        <agent command="/usr/local/bin/sid-tr" arguments="%in %out"/>
        <buffer size="1048576" chunk-size="4096" fill-size="1048576"/>
      </profile>


The sid-tr transcoder should be attached, along with my most-current config.xml.

Finally, you'll need to modify sid2wav to tell it not to pollute its stdout channel with superfluous crap, and then also to suppress its WAV header, because sox for some reason doesn't think that particular WAV header is accurate. I believe it has something to do with the host OS being 64-bit, but since I got it working without fiddling around with any data structures this is irrelevant to our needs. (hexedit, my favourite hex editor, suggests the RIFF/WAV header in the sid2wav output is a good few bytes longer than a standard WAV file that exists elsewhere in my WAV samples directories.)

I am attaching a sid2wav patch which adds the following functionality:

   -r output only raw sample data
   -q do not print any status to stdout

Apply the patch in mid-build post-configure but prior to an actual build and it should Just Work for you.

WARNINGS

• Note that sidplay appears to only play a certain set number of seconds into the SID file.
• Note that sidplay does NOT appear to understand RSID nor PSID v2

I'll likely be incorporating libsid2 and sidplay2 in this particular transcoder in the near future. I stuck with sidplay for the time being because it uses wayyyy less CPU, and it was simpler.

Hrm.. the original author of sidplay appears to be active in the Fedora project:

https://fedoraproject.org/wiki/User:Mschwendt

Interesting SID-related information can be found here:

sidplay2 doc list
PSID v2 NG Spec

[euxneks]

Aaand... to do it on ubuntu:

Code:

sudo apt-get install mediatomb

done.

[sudog]

Aaand... to do it on ubuntu:

Code:

sudo apt-get install mediatomb

done.

If I wanted 0.11.0 on NetBSD I could just:

cd /usr/pkgsrc/multimedia/mediatomb
make install

But you didn't read my post. I wanted the version currently in SVN, which corresponds to the upcoming 0.12.0 release. 0.11.0 is old and busted.

So a 0.11.0 default install is minus most of the transcoding..  minus the config.xml customisation, minus the working sid2wav function, minus my new streaming MIDI transcoder, minus YouTube, minus an Apple movie trailers virtual directory, minus minus minus.

[sudog]

Getting MIDI Transcoding with Timidity Working
This is pretty simple too. This transcoding stuff is getting easier the more I do it.

Here's my updated config.xml file which is the most up-to-date version I have, and includes MIDI transcoding in it too. Using timidity is mercifully simple. My mid-tr script consists of:

#!/usr/pkg/bin/bash

exec timidity --quiet=6 -L /usr/pkg/share/freepats -c timidity.cfg -Or1slx -o "$2" "$1"


The two packages I've installed are:

pkgsrc/audio/timidity
pkgsrc/audio/freepats

timidity by itself has no samples to draw on to synthesize the MIDI instruments into WAV/RAW form. To prevent all errors, I was forced to copy the freepats timidity.cfg into /usr/pkg/etc/timidity.cfg. But now it streams very nicely and I can hear MIDI files now on my PS3.

Yay!
Reminder
Remember that Mediatomb isn't smart enough to re-scan files that it freshly knows what to do with. Notice in my config.xml that I've added some MOD formats too, in addition to the MIDI and other stuff. In order to correctly refresh Mediatomb, I will sometimes have to do the following:

• Stop Mediatomb, modify the uuid in config.xml (it LOOKS like the PS3 sometimes caches results)
• Stop Mediatomb when config.xml has been changed to get it to read the new one
• Remove the enclosing directories entirely, and then re-add them via the web UI

Directory removing/re-adding is a pain because it means that one giant overarching directory entry with a refresh interval can't be used. You have to go and add ALL the subdirectories as timed-scan directories one at a time and juggle them all around. The reason this is, is because Mediatomb refuses to diverge a subdirectory from a parent directory, and it refuses to allow you to create a timed-scan subdirectory when a directory higher than that is already timed-scan.

Oh well.

[euxneks]

Hrm.. I wonder if I could use mediatomb to transcode and send video over WiFi to my iPod...

[sudog]

Hrm.. I wonder if I could use mediatomb to transcode and send video over WiFi to my iPod...

If your iPod can do uPnP, then yes. Doesn't look like it can happen natively, but this does mean there are some people who are working on a uPnP client for iPod. And then there's this one here: http://plugplayer.com/ but that one costs money and doesn't appear to be open source.

Then there's this one here: http://cgimediasuite.sourceforge.net/ but it looks like you'd have to build it yourself if you don't want to get it from the AppStore.

Plus with MediaTomb you can transcode on-the-fly to your iPod in the correct format (it accepts only two types, looks like.) But it does require enormous computational power to encode in H.26x format, so you'll probably have to stick to MP4 for on-the-fly transcoding unless you have some kind of uber-god machine faster than any of mine. (And my new Mythbox is very strong.)

[euxneks]

Hey dude, have you tried out Rygel?
http://live.gnome.org/Rygel

Looks like it might be interesting with which to fiddle.  The gnome guys are all about ease of use.

[sudog]

Hey dude, have you tried out Rygel?
http://live.gnome.org/Rygel

Looks like it might be interesting with which to fiddle.  The gnome guys are all about ease of use.

Not a big fan of Gnome right now. Kinda agree with Linus on that one.

But I have an Icecast2 server up and running nicely. And I've replaced Mediatomb (which I'm not even so sure is maintained anymore) with playstation mediaserver. Or pms as it's colloquially called. All Java, very slick, and requires zero configuration. Well..  almost zero. You have to tell it which directories to serve from.

But it's pretty awesome man.