miMTGCrawler

So as per my previous post, I have written a very simple program to show me the value of Magic Cards. All you do is run the program and type in the name of the card. Then it prints out the value of the card for each version it is in. In addition it also saves and formats the printed bits in a txt file, which is compatable with spreadsheet software. See images below.

Simply type in the card you are looking for and press enter.

That is then saved here.

Which you can then import...

Here!

It also takes into account cards that do not exist. But not half typed. i.e. If I typed 'myr' it would return everything with myr in the title... No really worth fixing. More the error / aid of the website it's searching.

[Small error fixed: 'Mirrodin Besieged Singles' was picked up from the ads on the side bar. Now the first category is correct. I'm not replacing the screenshots though. You get the picture. (Spot the pun)]

Code below:

#!C:/Program Files/Python/Python 2.7

import urllib

print "miMTGCrawler version 0.1"

f = open('cardList.txt', 'a')

# The below URL is the link to a card. The card name is appended, also '+' replaces ' '. Case is irrelevent.
# e.g. 'Myr Enforcer' = 'Myr+Enforcer'
# Searches the website starcitygames.com for the information.
scLink = 'http://sales.starcitygames.com/search.php?substring='

while(True):
    print "Please enter card name: "
    card = raw_input()
    if card == 'q':
        break
   
    #Builds the URL for the card.
    cardLink = scLink + card.replace(' ', '+')
    page = urllib.urlopen(cardLink).read()
   
    # Search for the category. e.g. 'Mirridon'
    categoryString = 'http://sales.starcitygames.com//category.php?cat='
   
    # Jumps to the first listing.
    pos = page.find('tooltip')
       
    pos = page.find(categoryString, pos)
    # Loops through this process until there are no more valid categories on the page.
    while (pos != -1):
        pos = page.find('>', pos) + 1
        # This will build a string for the Card and it's categories and values of the listing found.
        cardString = card + '\t' + page[pos : page.find('<', pos)] + '\t'
        pos = page.find('">$', pos)
        # Checks to see if a price was found following the category.
        if pos == -1:
            print 'There is no such card: ' + card
            break
        cardString = cardString + page[page.find('$', pos) : page.find('</td>', pos)]
        print cardString + '\n'
        f.write(cardString + '\n')
        # Look for the next category.
        pos = page.find(categoryString, pos)

      

Back With a Plan

I'm back. Fiji was awesome. Got a wee bit burnt... But no visible tan due to the copious amounts of sunblock I was applying. Damn. Also I left the sunblock at home so never found out about taking it through customs. Crystal clear water and fish everywhere. Also a lot of frogs. I found that strange. I don't have the camera with me at the moment so pictures will have to wait. Though they will be on here at some point.

As for tanks: No progress. Aside from the visuals there isn't too much to do (that is fun to code). So I'll have a chat with Foo to see where we are heading with this and will let you know.

In the mean time. I did come home to a large amount of Magic Cards at the door. I'm not expecting to use them all, large is an entire shoe box full which I got for NZD$10.50, so I going to try to make some of my money back by reselling them individually on TradeMe.

I'm not a hardcore fan so I have little to no idea of the worth of the cards, so what I am going to do is write another web crawler, like this one. Woo! Finally back to some coding. This is going to search StarCity for each card and return the value of said card for me, then dump it into a txt file; which I can directly import to a spreadsheet. Should let me know the most valuable to least.

I'm going to be using Python for this. Will post the source later on Ok? Ok.

miCrawler

Problem: At my work it was decided that we were to do mail drops. Rather than leaving the office and walking around the local area, we hit up local real estates offices etc and found all the addresses of houses for sale, our target audience. As it happened the real estate people were not helpful, there websites were. More helpful then these was the TradeMe website. As it turned out most of the real estate agents also listed the houses for sale on there anyway.
For a while it was my job to go through every listing and put them into a spreadsheet for some processing. After a day of this I thought it would be easier to write a web crawler to do it all for me.

tl;dr We need to get all the addresses off TradeMe, that are local and recent.

Solution:
Having recently done Python at university, I decided to use that. As all I wanted was a CLI program that I could implement quickly, it seemed to be a good fit. The program searches the list of listings for the listing number and then searches each listing for the address. If it has one, it is appended to a txt file. This file can then be added to a spreadsheet using imports.
Credit must go to a couple of places for this:
http://forums.devshed.com/python-programming-11/webcrawler-in-python-73318.html
http://www.daniweb.com/code/snippet216420.html
daniweb is a great source for help, although I can't stand the website. Found it tricky to navigate. Google helped with that.

Code:

#!C:/Program Files/Python/Python 2.7
import urllib, time
print "miCrawler version 0.1"
f = open('addressFile.txt', 'a')
# The below URL is the houses for sale in the manukau area, in list view, sorted by most recent, page 1
# May need to be updated occassionally.
tmManukau1 = 'http://www.trademe.co.nz/browse/property/regionlistings.aspx?mcat=0350-5748-3399-&v=List&key=100055&page='
tmManukau2 = '&sort_order=expiry_desc'
for i in range(0, 60):
    tmManukau = tmManukau1 + str(i) + tmManukau2
   
    page = urllib.urlopen(tmManukau).read()
    # Note that line 637 is the first occurance of a url for a particular listing.
    # The 9 digits following this string will represent a listing.
   
    auctionString = '/Trade-me-property/Residential-property/Houses-for-sale/auction-'
   
    pos = 1
    pos = page.find(auctionString)
    # Loops through this process until there are no more valid links on the page.
    while (pos != -1):
        # This will build a string for the URL of the listing found.
        listingURL = 'http://www.trademe.co.nz' + page[pos : pos + 77]
        listing = urllib.urlopen(listingURL).read()
        # Gets the address from the listing, the 26 skips the <td> and spaces.
        addSPos = listing.find('<td>', listing.find('Location:')) + 26
        addEPos = listing.find('</td>', addSPos)
        address = listing[addSPos : addEPos]
        try:
            int(address[0])
            address = address.replace('<br />', '\t')
            print (address + '\n')
            f.write(address + '\n')
        except ValueError:
            print (address + 'HAS NO NUMBER \n')
        finally:
            # Each address occurs twice, skip the second.
            pos = page.find(auctionString, page.find(auctionString, pos + 78) + 78)

Note:
This is very specific. It only searches Manukau and surrounding areas. Small changes on the urls will need to be made for that to change. The range in the loop determines which pages are to be read. Also I have no idea what that mcat number is and may need to be updated.
I suppose the main key here is this line:

 listing = urllib.urlopen(listingURL).read()

'listing' now contains the entire html code for the page. Performing some search functions on here for the auction numbers allows the program to find the relevent listing and from that page find the address. Simple enough.
The random hard coded numbers are there due to formatting. Easiest way I could think of to get around it. So that's that one. Never used all the 1,180 addresses. Kept the boss off my back for a while though.