Big Picture Cataloguer: An update

Archive for the ‘Computers’ Category

You are currently browsing the archives for the Computers category.

4 Feb 2012

Albumart Replacer

The Albumart Replacer is a little script I wrote to automatically find higher-resolution copies of album covers. For listening to music, I use foobar2000 and although the wonderful foo_discogs automatically adds album art to my music, sometimes the artwork on discogs.com isn’t the best.

Enter Albumart Replacer. If I’m listening to music and I notice the album art isn’t up to scratch, e.g.:

I simply run the script, it sends the image’s data to tineye.com, and if any higher-resolution copies of the same image are found, it’ll grab the best quality one:

This is all done without any user interaction. Using foo_run, it’s easy to set up a keybinding in foobar2000 so that it just takes one key press (simply pass the script a song’s %PATH% variable from foobar and it’ll do the rest.)

Source code, downloads and further details are available at the bitbucket repository.

Enjoy!

4 February, 2012 at 18:33 by aengus

Posted in Art, Code, Computers, Music | 1 Comment »

9 Jan 2012

@grammer_man who the fuck is this nigga and why u comin at me like that #Hoeassnigga

Had a spare hour last Thursday and decided to write a little twitter bot. There he is above. His name is Grammer_Man and he corrects other twitter users’ misspellings, using data scraped from these Wikipedia pages.

Responses have been pouring in already, some agitated, some confused, but most positive — which was a pleasant surprise. In any event, the minimal amount of effort in coding has paid off many times over in entertainment.

You can see who’s responding at the moment by searching for @grammer_man, and also by checking his list of favourites.

Here is the (somewhat slapdash) code that powers our fearless spelling Nazi:

grabber.py

This module grabs the spelling data from Wikipedia.

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import os
import pickle

import requests
from BeautifulSoup import BeautifulSoup

def grab(letter):
    '''
    Grabs spellings from wikipedia
    '''
    url = 'http://en.wikipedia.org/wiki/Wikipedia:Lists_of_common_misspellings/%s' % letter
    html = requests.get(url).content
    soup = BeautifulSoup(html)
    bullets = soup.findAll('li')
    retval = {}
    for bullet in bullets:
        if 'plainlinks' in repr(bullet):
            values = bullet.text.split('(')
            if len(values) == 2:
                retval[values[0]] = values[1][:-1] # shave off the ) at end
    return retval

def get_spellings():
    '''
    Returns a dictionary of {false: correct} spellings
    '''
    if not os.path.exists('words.pkl'):
        retval = {}
        for c in 'ABCDEFGHIJKLMNOPQRSTUVWXYZ':
            print 'Getting typos - %s' % c
            retval.update(grab(c))
        print 'Dumping...'
        f = open('words.pkl', 'w')
        pickle.dump(retval, f)
        f.close()
        return retval
    else:
        f = open('words.pkl', 'r')
        retval = pickle.load(f)
        f.close()
        return retval
    
if __name__ == '__main__':
    get_spellings()

#!/usr/bin/env python

# -*- coding: utf-8 -*-

import os

import pickle

import requests

from BeautifulSoup import BeautifulSoup

def grab(letter):

'''

Grabs spellings from wikipedia

'''

url = 'http://en.wikipedia.org/wiki/Wikipedia:Lists_of_common_misspellings/%s' % letter

html = requests.get(url).content

soup = BeautifulSoup(html)

bullets = soup.findAll('li')

retval = {}

for bullet in bullets:

if 'plainlinks' in repr(bullet):

values = bullet.text.split('(')

if len(values) == 2:

retval[values[0]] = values[1][:-1] # shave off the ) at end

return retval

def get_spellings():

'''

Returns a dictionary of {false: correct} spellings

'''

if not os.path.exists('words.pkl'):

retval = {}

for c in 'ABCDEFGHIJKLMNOPQRSTUVWXYZ':

print 'Getting typos - %s' % c

retval.update(grab(c))

print 'Dumping...'

f = open('words.pkl', 'w')

pickle.dump(retval, f)

f.close()

return retval

else:

f = open('words.pkl', 'r')

retval = pickle.load(f)

f.close()

return retval

if __name__ == '__main__':

get_spellings()

bot.py

The bot. Selects misspellings at random, searches for them, responds to them, while also taking breaks between tweets and longer breaks every few hours.

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import os
import random
import time
import pickle

import twitter

from grabber import get_spellings

API = twitter.Api(consumer_key='XXX',
                  consumer_secret='XXX',
                  access_token_key='XXX',
                  access_token_secret='XXX')

MESSAGES = u'''
$USERNAME sooo you might wanna spell $CORRECT the right way next time!! Not your fault bro.
#
# All messages stored in here, one per line. 
# Edited out in order to save space in this blog post.
#
'''.split('\n')

def compose_message(twitter_post, mistake, correct):
    '''
    Choose a message from MESSAGES at random, substitute fields to personalise it and 
    check if it exceeds the twitter message limit. Try this 100 times before failing.
    '''
    retries = 0
    while retries < 100:
        retries += 1
        message = MESSAGES[random.randint(0, len(MESSAGES) - 1)]
        message = message.replace('$USERNAME', '@%s' % twitter_post.user.screen_name)
        message = message.replace('$MISTAKE', '"%s"' % mistake).replace('$CORRECT', '"%s"' % correct)
        if message and len(message) < 141:
            return message
    return None

def correct_spelling(twitter_post, mistake, correct):
    '''
    Correct someone's spelling in a twitter_post
    '''
    print u'Correcting @%s for using %s...' %(twitter_post.user.screen_name,
                                            mistake)
    message = compose_message(twitter_post, mistake, correct)
    if not message:
        print u'All messages were too long... Aborting...'
        return None
    else:
        API.PostUpdate(message, in_reply_to_status_id=twitter_post.id)
        return True

def search(word):
    '''
    Search twitter for uses of a word, return one if it's been used recently.
    Otherwise return None.
    
    TODO: Add time awareness.
    '''
    print 'Searching for uses of %s...' % word
    results = API.GetSearch(word)
    if results:
        for result in results:
            if not check_if_done(result.id) and not result.user.screen_name == 'grammer_man' and word in result.text:
                return result
    return None

def check_if_done(id):
    '''
    Checks if a tweet has already been responded to
    '''
    if os.path.exists('done.pkl'):
        f = open('done.pkl', 'r')
        done = pickle.load(f)
        f.close()
        if id in done:
            return True
    return False

def update_done(id):
    '''
    Updates a list of tweets that've been replied to
    '''
    if os.path.exists('done.pkl'):
        f = open('done.pkl', 'r')
        done = pickle.load(f)
        f.close()
    else:
        done = []

    done.append(id)

    f = open('done.pkl', 'w')
    pickle.dump(done, f)
    f.close()

def main():
    '''
    Main program flow
    '''
    words = get_spellings()
    counter = 0
    while True:
        word = random.choice(words.keys())
        post = search(word)
        if counter > 100:
            rand_time = random.randint(120*60, 240*60)
            print 'Done %s tweets, sleeping for %s minutes' % (counter, rand_time/60)
            time.sleep(rand_time)
            counter = 0
        # TODO: PROPERLY PRUNE THE MISTAKES/CORRECTIONS FROM WIKIPEDIA AND REMOVE THIS:
        if not u',' in word + words[word] and not u';' in word + words[word]:
            if post:
                result = correct_spelling(post, word, words[word])
                if result:
                    counter += 1
                    print '#%s Done' % counter
                    update_done(post.id)
                    time.sleep(random.randint(300,500))


if __name__ == '__main__':
    main()

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

#!/usr/bin/env python

# -*- coding: utf-8 -*-

import os

import random

import time

import pickle

import twitter

from grabber import get_spellings

API = twitter.Api(consumer_key='XXX',

consumer_secret='XXX',

access_token_key='XXX',

access_token_secret='XXX')

MESSAGES = u'''

$USERNAME sooo you might wanna spell $CORRECT the right way next time!! Not your fault bro.

# All messages stored in here, one per line.

# Edited out in order to save space in this blog post.

'''.split('\n')

def compose_message(twitter_post, mistake, correct):

'''

Choose a message from MESSAGES at random, substitute fields to personalise it and

check if it exceeds the twitter message limit. Try this 100 times before failing.

'''

retries = 0

while retries < 100:

retries += 1

message = MESSAGES[random.randint(0, len(MESSAGES) - 1)]

message = message.replace('$USERNAME', '@%s' % twitter_post.user.screen_name)

message = message.replace('$MISTAKE', '"%s"' % mistake).replace('$CORRECT', '"%s"' % correct)

if message and len(message) < 141:

return message

return None

def correct_spelling(twitter_post, mistake, correct):

'''

Correct someone's spelling in a twitter_post

'''

print u'Correcting @%s for using %s...' %(twitter_post.user.screen_name,

mistake)

message = compose_message(twitter_post, mistake, correct)

if not message:

print u'All messages were too long... Aborting...'

return None

else:

API.PostUpdate(message, in_reply_to_status_id=twitter_post.id)

return True

def search(word):

'''

Search twitter for uses of a word, return one if it's been used recently.

Otherwise return None.

TODO: Add time awareness.

'''

print 'Searching for uses of %s...' % word

results = API.GetSearch(word)

if results:

for result in results:

if not check_if_done(result.id) and not result.user.screen_name == 'grammer_man' and word in result.text:

return result

return None

def check_if_done(id):

'''

Checks if a tweet has already been responded to

'''

if os.path.exists('done.pkl'):

f = open('done.pkl', 'r')

done = pickle.load(f)

f.close()

if id in done:

return True

return False

def update_done(id):

'''

Updates a list of tweets that've been replied to

'''

if os.path.exists('done.pkl'):

f = open('done.pkl', 'r')

done = pickle.load(f)

f.close()

else:

done = []

done.append(id)

f = open('done.pkl', 'w')

pickle.dump(done, f)

f.close()

def main():

'''

Main program flow

'''

words = get_spellings()

counter = 0

while True:

word = random.choice(words.keys())

post = search(word)

if counter > 100:

rand_time = random.randint(120*60, 240*60)

print 'Done %s tweets, sleeping for %s minutes' % (counter, rand_time/60)

time.sleep(rand_time)

counter = 0

# TODO: PROPERLY PRUNE THE MISTAKES/CORRECTIONS FROM WIKIPEDIA AND REMOVE THIS:

if not u',' in word + words[word] and not u';' in word + words[word]:

if post:

result = correct_spelling(post, word, words[word])

if result:

counter += 1

print '#%s Done' % counter

update_done(post.id)

time.sleep(random.randint(300,500))

if __name__ == '__main__':

main()

Grammer_Man uses the following libraries:

python-twitter (Be warned: no proxy support)
requests
BeautifulSoup

9 January, 2012 at 20:06 by aengus

Posted in Code, Computers, Funny, Idiots, Internet, Oddities | 14 Comments »

23 Jan 2011

bigpicture Cataloguer 0.5 Released

Mainly a bug-fix release, with problems relating to HTML in captions fixed.

If you already downloaded the “Haiti: One Year Later” photo album, you might want to delete it and run this version of the cataloguer.

Download available here.

23 January, 2011 at 20:49 by aengus

Posted in Code, Computers | 1 Comment »

9 Oct 2010

Bigpicture Cataloguer 0.4 Released

New version of The Big Picture Cataloguer available from here. Thanks for your patience; sorry it took so long.

9 October, 2010 at 11:19 by aengus

Posted in Code, Internet | No Comments »

23 Sep 2010

Critical Bug in bigpicture Cataloguer

Update: Version 0.4 now released. Please upgrade immediately.

I’m aware of and have fixed a critical bug in the Big Picture cataloguer. The cataloguer stops downloading when it reaches a recent photo album, since its title ends with a full stop.

I moved recently and unfortunately my main computer was destroyed in the process. With it went my proper development environment.

This means I’ll not be able to update the executable versions of the cataloguer for perhaps a week, but in the meantime, a new version of the script is up.

If you are encountering this bug, please check back next week for an updated version of the executables. The program will resume where it left off; you will not have lost the chance to get any galleries.

23 September, 2010 at 19:24 by aengus

Posted in Code | 3 Comments »

20 Aug 2010

The Videogame Music Preservation Foundation

A friend came across this website a few weeks ago, and I was very excited about it – an archive of plenty of video game music (mainly for DOS, which is what I grew up with), all recorded properly in order to maximise the nostalgia, and made available in ogg format.

I contacted the guy who runs it about setting up a torrent of the entire archive, and he very kindly obliged. You can get the entire collection here (~4.4GB in total). Enjoy!

20 August, 2010 at 18:24 by aengus

Posted in Art, Computer Games, Computers, Downloads, Electronic, Internet, Music | No Comments »

24 Jul 2010

Why the future doesn’t need us.

I finally managed to get around to reading Bill Joy’s article Why the future doesn’t need us the other day while waiting to board a plane. Bill Joy is a renowned computer scientist who co-founded Sun Microsystems and authored the popular UNIX text editor vi. The article is concerned with the ever increasing speed of “progress” in fields of new technology (primarily robotics, nanotechnology and genetic engineering) which Joy views with apprehension, arguing that the products of these fields will eventually render mankind obsolete and lead to our self-destruction.

There’s no point trying to quote it, so instead you can read the article here, read more about Bill Joy here, or read responses and criticism of the article here.

24 July, 2010 at 1:21 by aengus

Posted in Computer Science, Computers, Internet, Robotics, Science | No Comments »

12 Jun 2010

Rise of the Robots

A good, short blog post from the wonderful ginandtacos blog on the increasing prevalence of unmanned vehicles in war, ending with a very sobering thought:

Won’t it be great when the military can send in the tanks without having to put crews in harm’s way?

Yes and no. The fewer casualties, the better. But what becomes of our reluctance to send the military galavanting around the sordid parts of the world once American casualties are taken out of the equation? We have almost no restraint as it is. I shudder to think of how easily Presidents and legislators will make the decision to go to war when the attitude of “We can just send robots to do it!” becomes entrenched. We saw what the advancements in design of cruise missiles in the 1980s did to the Executive Branch; if someone’s acting up, just lob a dozen Tomahawks at them from a few hundred miles away. It became the easy way to intervene without actually making a commitment or putting Americans at risk. Collateral damage isn’t much of a deterrent to our political class. UAVs are another step in that direction, a step toward a future with more remotely operated and even autonomous means of doing the dirty work.

It’s great that technology allows more American soldiers to come home alive and in one piece, but if we remove the U.S. body count from the decision-making process the only restraints on waging war will be common sense, morality, and logic. Yeah, let’s start taking bets on how well that works.

12 June, 2010 at 2:58 by aengus

Posted in America, Computers, Politics, Robotics, Science, War | No Comments »

22 May 2010

Tricks with python and music

From Music Machinery:

One of my favorite hacks at last weekend’s Music Hack Day is Tristan’s Swinger. The Swinger is a bit of python code that takes any song and makes it swing. It does this be taking each beat and time-stretching the first half of each beat while time-shrinking the second half. It has quite a magical effect. Some examples:
Every Breath You Take (swing version) by TeeJay

Sweet Child O’ Mine (Swing Version) by plamere

You can find more examples in the original blog post. The results really are truly impressive. I’m looking forward to playing with Tristan Jehan‘s code, and also having a look at his PhD thesis:

Machines have the power and potential to make expressive music on their own. This thesis aims to computationally model the process of creating music using experience from listening to examples. Our unbiased signal-based solution models the life cycle of listening, composing, and performing, turning the machine into an active musician, instead of simply an instrument. We accomplish this through an analysis-synthesis technique by combined perceptual and structural modeling of the musical surface, which leads to a minimal data representation.

Fascinating stuff!

22 May, 2010 at 17:30 by aengus

Posted in Art, Code, Computer Science, Computers, Experimental, Generative Music, Music, Music theory, Oddities | No Comments »

20 May 2010

Big Picture Cataloguer: An update

In just over a week since I released the Big Picture Cataloguer, there’s been a surprising amount of interest and enthusiasm about it. Since I still haven’t gotten binary versions of the program for OS X and Linux up (I’ve no access to an OS X computer, and getting the required libraries installed on Linux has proved to be quite difficult), I’ve decided to relent and share the source code of the cataloguer under a Creative Commons license.

The script makes use of pyexiv2 – the 0.2 branch – for metadata editing, mechanize for grabbing pages and submitting error reports, the very handy unaccented_map() class (included) for unicode trickery and of course the wonderful XML parser, BeautifulSoup.

Naturally, it’s available from the Big Picture Cataloguer’s page in the Code section of this site.

Given how much The Big Picture galleries’ HTML format has subtly changed over time, and the fact I wrote this in a rush, it’s quite messy, but it does the job.

Today’s update is of version 0.3, which has an optional “quiet mode” to enable users to schedule the program to run frequently. Enjoy!

20 May, 2010 at 20:11 by aengus

Posted in Art, Code, Computers, Internet, Photography | No Comments »