Spiders

[tswhite70]

I got a chance to write a spider for buch.de. I think it's working correctly based on my "incredible" German language skills (aka Babefish). I did have to do a little something weird on this one. Apparently Xlobby sends 2 seperate http get request when spidering, the first does not include the %searchstring% variable, I'm guessing Xlobby uses this to verify the site is available before sending a second get request with the %searchstring% included. Buch.de returns a 404 error if you don't have a value for the search instead of the typicall HTTP 200/OK message. This causes Xlobby not to send the second request, bummer. I figured out that I could include a default "1" in the search and Buch.de would respond correctly to the first query and it doesn't seem to mess up the search in the testing I've done - of course I'm sure it will screw some search (ie searching for 10 - Bo Derek movie would probably get you 101 Dalmations instead...).

I've put the two German spiders in a single zip...
http://home.comcast.net/~twhite644/spid ... German.zip

good luck,
tsw

[CaM]

hi there all, does anyone know why the amazon music spider is not currently working?

thanks

CaM

[tswhite70]

Updated the Amazon.com music spider... (Pre-Calrad tested only)

http://home.comcast.net/~twhite644/spiders/UpdatedMusicSpiders.zip

Contents:
music - amazon.com
music - buy.com
music - cdbaby.com
music - cduniverse.com
music - elusivedisc.com
music - walmart.com

good luck,
tsw

[tswhite70]

I finally had a chance to update some of my DVD spiders...

http://home.comcast.net/~twhite644/spid ... piders.zip

good luck,
tsw

[Myth]

Is there a way to make the stills function work? It hasnt worked for quite a while (on my setup, anyway)
thanks
Myth

[tswhite70]

Myth - the answer is no, the stills download/spider is internal Xlobby code and since Steven is no longer providing support for the Pre-Calrad version of Xlobby we are out of luck.

On a happier note - I do have a Xscript script to pull stills for movies that I've been using for the past year or so. I'll be happy to upload it tonight, it pulls stills from movies.com and IMDB.com and is configurable for the # of stills you want.

I've also done a bunch of preliminary work on a Spider plugin to allow for some expanded functionality, most of the code is written I've just got to wrap it up as a plugin. I've got at least 3 things ahead of it though so it will definitely be a while.

I'll try and remember to upload the GetStills script tonight....

EDIT: I updloaded the script - see here
tsw

[slaman]

Thanks tswhite70...

I've been trying to tweak this spider to get it working for both single-line plot movies (Juno) and multi-line plot movies (American Gangster) but it doesn't seem to work!!! I get garbage characters for the multi-line plot movies...

Here's what I have... I can't combine single/multi-plot movies...


url=http://us.imdb.com/find?q=%searchstring%;tt=1
results=<a href="(?<url>/title/.*?/).*?">(?<display>.*?)</a>

//title & year
<div id="tn15title">.*?<h1>(?<title>.*?) <span>\(<a href="/Sections/Years/.*?">(?<year>.*?)</a>\)</span></h1>

//Directors
<h5>Director:</h5>(?<variable>.*?)<br/>
<a href="/name/.*?">(?<director>.*?)</a>

//Genres
<h5>Genre:</h5>(?<variable>.*?)</div>
<a href="/Sections/Genres/.*?/">(?<genre>.*?)</a>

//Rating
<h5><a href="/mpaa">MPAA</a>:</h5>.*?Rated (?<rating>.*?\.).*?</div>

//Runtime
<h5>Runtime:</h5>.*?(?<runtime>.*?)</div>

//Actors
<table class="cast">(?<variable>.*?)</tr></table>
<td class="hs"><a href="/name/.*?/">(?<actors>.*?)</a></td>

//Languages
<h5>Language:</h5>(?<variable>.*?)</div>
<a href="/Sections/Languages/.*?/">(?<language>.*?)</a>

//Full Plot
<h5>Plot Outline:</h5>.*?<a class="tn15more inline" href="(?<url>/title/tt.*?/plotsummary)" onClick=.*?more</a>
<p class="plotpar">(?<plot>.*?)<i>

//Coverart
<a name="poster" href="photogallery" title=".*?"><img border="0" alt=".*?" title=".*?" src="(?<coverart>.*?)".*?></a>

replace=coverart:m.jpg:f.jpg

[tswhite70]

slaman - I took a look at my IMDB spider and there was a problem with American Gangster, although it turned out to be a result of the actors regex and not the plot. I've updated my zip file with new IMDB and IMDB_Direct spiders and also fixed the Ratings regex for Amazon.com. I tested them on the movies you mentioned plus a couple others and they seem to working.

The answer to your question about single vs. mutiline plots is that you have to have 2 separate plot regexes, singlel-line first then the multi-line second. Xlobby will always do the single-line plot, but it will be replaced by the multi-line if it is available - see the IMDB spider in the zip.

http://home.comcast.net/~twhite644/spid ... piders.zip

good luck,
tsw

[slaman]

tswhite70 - thanks, but it didn't work... The actors regexp is still broke (get extra lines) and the "plot" variable doesn't get replaced, it gets appended to.

Perhaps it's because of Vista, because when I was running XP, this spider was fine?

[tswhite70]

I don't have a Vista system to test on, but it's possible that your .Net version (3.0 or 3.5?) changed something in the regex functions that is causing the different result. You could try testing the regex with Expresso on your Vista machine and see if you can figure it out. Sorry I couldn't help.

good luck,
tsw

[slaman]

Will test it out right now - thanks for the link - was looking for a good test application. Re-starting XLobby after each spider change to "test" was getting annoying...

[slaman]

Weird - doesn't replace the variable "plot"
tswhite, wnated to ask you, since you seem to be the guru of spiders here... how do I get the proper "fancy" characters, like french or german letters... I get weird characters when those things appear... an dI have to manually paste them in.

[homepc]

I have always used the F2 configuration to use spiders for my movies. This weekend I finally clicked on the spider button on my movieviewer overlay screen. Up pops this spider screen. (If I knew how to attach picture I would show you). At the top of the screen, I have three slections, Direct, WEB, ABC. When I click direct, nothing happens. When I click on WEB, it fires up the spider scripts and show coverart and info to choose for the movie. I looked at the event for these buttons and they are looking at "spider:spider search direct movies" for the direct, and "spider:spider search direct movies" for the WEB button. Can anyone tell me if these events are looking at the same spider scripts I use when I go directly to the F2 configuation screens and select spiders?

Thanks

[tswhite70]

slaman - As for the non-english characters there's not a way around the problem that I am aware of. It would have to be coded in Xlobby to translate those characters into the correct syntax.

homepc - Are you using the pay version of Xlobby?

Seems like I've said this before, but I'm a free-lobby user. Any spiders I publish are designed to be used with the Pre-Calrad version of XL. They may work with the current pay-version, but I don't have a way to troubleshoot any issues. Sorry I can't be of more help.

good luck,
tsw

[homepc]

tswhite70, I am a free loader! haahaa I was just curious if the spider skin that was created, used a spider that was internal to XL or used the spiders that you see when you go directly in the configuration screens. does the event "spiders:spider search web movies" use an internal spider?

thanks
Ray