Sunday, 3 May 2015

Fun Beneath the Sheets

I've been loathed to type anything here for a while, cos it means shunting down that lovely still from Desk Set. But we can't stay stuck in the past forever, now, can we? [cue obligatory Librarian hand-wringing]...

 An empty spreadsheet

My love of Excel is well documented. My disregard for the Google Suite is also quite well attested. At the centre of this Venn Diagram is Google Sheets. At first glance, it's an inferior Excel clone: inferior because there's clearly rather a lot of stuff missing, and because it works in a web-browser (or doesn't work in a web-browser, depending on the browser). But while there's a lot Excel can do that Google Sheets can't, there's also a bit that Google Sheets can do that Excel can't. Here's some of the more entertaining bits.

Query

The most obvious thing is the interaction with Google Forms. Excel is not incapable of harvesting live data but it's bloody hard work. Google Sheets has been designed to be fed this hot, juicy data. And, because of the collaborative nature of Google Sheets, it's also been designed to make it possible to query that data without screwing everything up.

=query() is something approximating magic. Excel's pivot tables excite me immensely (now I know how to use them), but that joy is but a fraction of =query(). =query() will run a search query on your data and then produce a whole new array based on what it finds. The SQL syntax it uses takes a bit of getting used to, but once you get the basics you'll find yourself building wonderful query engines. Google's =query() bible explains the basics, but doesn't even begin to express its greatness. Imagine you've got a spreadsheet pulling in data on all tweets ever, but you're only interested in the ones I've made where I use a rude word. You could set up a spreadsheet that queried this master spreadsheet and returned just the tweets by me that say "bum", "knickers" or "trump". You could then play with that to your heart's content. You could even set up an email alert to let you know whenever I tweeted filth.

And because it's Google Sheets, you can even use the query function in a HTML context: Here's a simple javascript webform I built that generates a query function, applies it to a Google Sheet and brings back the results as a webpage. This form is just looking at two columns (date and tweet) and returning the same fields for any matching data), but it could be far more elaborate, and can return any part of the data you want. In this case the form is generating a URL as follows:

https://docs.google.com/a/york.ac.uk/spreadsheets/d/
[ID OF THE SHARED SPREADSHEET]/
gviz/tq?tqx=out:html&tq=
select+D,F+where+D+starts+with+'2014-12-[DATE]'
+and+upper(F)+contains+upper('[FREETEXT]')

...so it's bringing back columns D and F of my sheet for any row where the contents of D starts with "2014-12-"... and the entered date, and an uppercase version of the contents of F contains an uppercase version of what was entered in the freetext field. 

The spreadsheet equivalent of this (having created a form whereby the date goes in A2 and the freetext goes in B2) is:

=query(Sheet1!A1:J583,"select D,F where D starts with '2014-12-"&A2&"' and upper(F) contains upper('"&B2&"')")


Join, Split, Unique and Sort

Before you go formula-blind, let's jump from =query() to something else quite fun. I had a spreadsheet containing a list of names (sometimes more than one name in the same cell, comma separated), and I wanted to collate them into something useful. In Excel, I'd have to mess about concatenating, substituting, lefting, righting, mid-ing, or even reprocessing the data (CSV-style) using the "Text to Columns" tool on the Data tab. But Google have come up with a new pair of functions: =join() and =split() which have the benefit of being something you can apply in a formula rather than a stand-alone gadget. They do exactly what you might expect, and save an awful lot of effort.

My list of names was in column A, so I cobbled together the following:

=sort(unique(proper(transpose(split(substitute(join(",",trim(A2:A)),", ",","),",")))))

First of all I'm trimming my fields to get rid of any trailing space characters.
Then I'm joining the trimmed cells using a single comma to delineate them. =join() is basically a posh concatenation.
Next I use substitute to get rid of any space characters that may have been entered after the commas in the cells that contained multiple names. I could potentially rely on trim again later, but this seemed as well as anything. It's given me a single cell full of names, each separated by a single comma.
Now I need to break this cell up to give one name per cell, so I use =split(): it will transform any comma into a cell division (much as if you were to open a CSV file in Excel).
The cells run across the row, but I want them all in a single column, so I transpose the array.
Some of the names have been entered entirely in lower case, so I use =proper() to change them all to Proper Case.
My list of names is looking good, but now I want to weed out the duplicates. Fortunately, Google have considered that too: =unique() will return only one of everything!
Finally, I pop all the names in alphabetical order by first name using another flash Google function: =sort().

The result is a neat, organised list of names. If I wanted to sort on the surname I've got problems, of course, and I've lost the capitalised second "C" in McCoy. There are things I can do to deal with these issues, but we're going to need a bigger spreadsheet. Let's save that for another day.

The Advent Calendar


At Christmas I built a spreadsheet Advent Calendar. Here's how it worked:

Each cell on the home sheet references an equivalent cell on a hidden sheet. This was not the original plan, but a flaw of Google Sheets is that it loads a snapshot view of the sheet before it updates the formulae, and unfortunately, the snapshot it took was for the end of the month, when the picture was at its most complete! This was quite a flaw, hence the dramatic rebuild.

The principle taking place on the hidden sheet is really rather simple: each cell has a character which maps to a conditional format. So "A" gives the wall colour, "2" gives the green of the grass and the trees, etc. A basic conditional formula checks the current date and displays the appropriate character: for instance, V30 (a patch of grass) has =if($B$39<17,2,""), so is green before the 17th and white thereafter (B39 contains =day(now()) to give the day's date).

There are a couple of more-complicated things going on: 

The sky has =if($A$38="N","H",3), relying on A38's =if(hour(now())>16,"N", if(hour(now()<8),"N","D")). This changes the sky colour depending on the time of day. I'd've liked to have added more shades to this, but the snowflake glyphs limited the conditional formatting options available. 

The Christmas Cards use Google's =image() function (e.g. image("http://www.avwoman.co.uk/advert/skaterx.jpg",2)). This is another place in which Google has the upper hand over Excel, as it makes a formal link between an image and its cell location. It's still a bit more limited than would be ideal (I'd've loved to have put an animated .gif in the fireplace, but that's beyond its capabilities), but it's a massive improvement on the mess you get in Excel.  

That, then, is an overview of some of the fun you can have with Google Sheets that you can't have with Excel. For a full list of Google Sheets functions, see this full list of Google Sheets functions


Sunday, 1 March 2015

Hollywood Answered the Google Question in 1957


That big Chock-A-Block wannabe there is EMMARAC, the new computer that's going to revolutionize the reference library of the Federal Broadcasting Network. The librarians looking at this great machine are about to get the pink slips from payroll that indicate their impending redundancy. The entire library has been rendered as machine code and fed into the computer, enabling it to retrieve in seconds an answer that took the woman in the black and red several weeks to ascertain.

This is a still from the 1957 film Desk Set, written by Phoebe and Henry Ephron from a 1955 play by William Marchant. It stars Katherine Hepburn as a librarian (which is kind of like looking in a mirror for me, as you can imagine) and Spencer Tracy as the efficiency expert re(/dis)organising her library and her life. A Kat Hepburn film is seldom a bad thing, and this is no exception to that rule. Indeed, it is rather lovely. It's got Katherine Hepburn and librarians: how can it fail to be rather lovely?

But let us get to the point of this post: The Google Question. It is an assumption among some people out there in the wild (sometimes even people in government) that the library and the librarian have been made redundant by Google. We, of course, know this assumption to be flawed: even ignoring the limited range of Google with respect to the Deep Web; even casting aside the slightly privileged assumption that Google and its fruits are an affordable and available presence in every home; the truth is that Google is only as smart as the person at the keyboard: to get the right answer, one sometimes has to ask the right question.

And this is precisely where EMMARAC fails too. For example, when asked a question about the King of the Watusi, EMMARAC's best effort is a film review of King Solomon's Mines. By way of comparison, Google also resorts to a film as first guess (in this case 1959's Watusi; it's not until the 9th hit that King Solomon's Mines turns up; oh, and Desk Set is the third hit). To get an informed response, one has to know a little bit about what one is typing about. In most day-to-day cases that may be easy enough. For everything else you can call upon your local Information Professional. Google is an amazing tool but getting the best out of it requires knowledge and skill.

Desk Set reaches the same conclusion about EMMARAC. This machine is not a replacement for librarians: it is a shortcut; a new index to consult when required. There will be no redundancies in this reorganization (indeed, a new business venture may, conveniently, even require some extra staff). The panic subsides and everyone lives happily ever after. Or more likely they spend the next fifty-odd years continually reproving their worth as further technological efficiencies are introduced.

Librarians are not, and have never really been, in a war with Google; rather their adversaries in this case are those who think that Google renders librarians obsolete. Here's 103 minutes of half-a-century-old gentle romantic comedy that clearly and relevantly demonstrates the shortcomings of that notion. Not only that, but they have the sort of office parties to which we should all aspire.

 
 Trailer (contains spoilers)
or from your local lending library]

Friday, 2 January 2015

A brief web-history of time-wasting.

Last year, in another bean-counting exercise, I gave you my top 40 most-visited sites of 2013. Now it's time to give 2014 the same treatment.

Before the table, the necessary caveats: This list is just the websites I visited using the Opera 12 browser on my home computer in 2014 (there's a start date of Jan 13th, so there's a couple of weeks missing from this time last year). In the last few months I've been using IE for Tweetdeck as it no-longer works properly in the old Opera. This means that the tally below is missing about a quarter's worth of twitter-derived traffic. As Opera 12 gets increasingly past it, I've also been using Firefox and (more recently) Pale Moon for certain things (not least these 'blog posts). But the vast majority of my home webbing is through Opera and here are the results (hover over the site names for context):




Total (h:m)
Daily av.
(m:s)

%


OVERALL 43d 01:19 02h 35:38 -50:22

1 = tweetdeck.twitter.com 10d 03:02 41:19 -21:38 26.5 -4.5
2 = en.wikipedia.org 5d 13:31 22:31 +10:41 14.5 +8.8
3 NE feedly.com 3d 14:30 14:32
9.3 +3.6
4 NE games.usvsth3m.com 2d 14:21 10:36
6.8
5 = www.youtube.com 2d 06:19 09:14 -00:11 5.9 +1.3
6 ▲1 www.bbc.co.uk 2d 00:02 08:10 +01:20 5.2 +1.9
7 ▼1 www.facebook.com 1d 18:17 07:11 -01:29 4.6 +0.4
8 = www.google.com 21:41 03:41 -02:27 2.4 -0.6
9 NE www.dailymotion.com 15:49 02:41
1.7
10 NE veritypodcast.wordpress.com 15:29 02:38
1.7
11 ▼7 www.blogger.com 15:04 02:34 -07:14 1.6 -3.2
12 ▲2 www.avwoman.co.uk 14:25 02:27 +01:08 1.6 +1.0
13 NE gabrielecirulli.github.io 12:58 02:12
1.4
14 ▼4 twitter.com 12:47 02:10 -00:50 1.4 -0.1
15 ▲9 lawrencemiles.blogspot.co.uk 12:17 02:05 +01:33 1.3 +1.1
16 NE thedoctorwhopodcast.com 12:11 02:04
1.3
17 NE www.radiotimes.com 09:57 01:41
1.1 +0.7
18 ▼7 localhost 09:39 01:38 -01:10 1.1 -0.3
19 ▼16 theoldreader.com 07:40 01:18 -10:31 0.8 -4.9
20 = www.sporcle.com 05:16 00:54 +00:10 0.6 +0.2
21 ▼12 maps.google.co.uk 05:00 00:51 -03:54 0.5 -1.8
22 NE www.british-history.ac.uk 04:29 00:46
0.5
23 NE vacancies.leedsmet.ac.uk 03:19 00:34
0.4
24 ▼7 jobs.york.ac.uk 03:11 00:32 -00:21 0.3 -0.1
25 NE www.fchd.info 02:52 00:29
0.3
26 NE www.chambers.co.uk 02:49 00:29
0.3
27 NE usvsth3m.com 02:34 00:26
0.3
28 ▲9 www.imdb.com 02:34 00:26 +00:10 0.3 +0.2
29 NE www.buzzfeed.com 02:22 00:24
0.3
30 NE www.librarything.com 02:20 00:24
0.3
31 NE www.dalek6388.co.uk 02:18 00:23
0.3
32 NE planningaccess.york.gov.uk 02:15 00:23
0.2
33
www.channel4.com 02:15 00:23
0.2
34 NE docs.google.com 02:11 00:22
0.2
35 ▼19 www.theguardian.com 02:00 00:20 -00:43 0.2 -0.3
36 NE tardis.wikia.com 01:55 00:20
0.2
37 ▲2 www.york.ac.uk 01:53 00:19 +00:05 0.2 +0.1
38 ▼25 www.splendidchaps.com 01:40 00:17 -01:02 0.2 -0.4
39 NE twominutetimelord.com 01:37 00:16
0.2
40 ▼11 knowyourmeme.com 01:34 00:16 -00:09 0.2 =

The highest New Entry is Feedly: it's been a torrid time for RSS ever since Google did evil by killing its reader. My original replacement, The Old Reader, slips 16 places as a consequence of a decision to limit the number of feeds served by the free package. So Feedly is my new home. I've had no problems with it. 

Another big faller is The Guardian, though only by 43" a day, so it's hardly a crisis. Worth noting, though, is the appearance of web magazines Buzzfeed and the Mirror's usvsth3m. I've kept the latter's games domain separate from the rest, as that's basically there as a consequence of the things that inspired my 2048 and Fight! simulators.

Sites involved in last year's grand mapping project have given way to sites about Doctor Who as the combined efforts of last year's Splendid Chaps and the arrival of Peter Capaldi in the title role kindled a taste for Doctor Who podcasts of various flavours (not least the amazing Verity!, new in at number 10).

There's a lack of retail outlets this year, which is something worthy of note (the first turns up at 41). Instead there's a few job application sites (yawn). YouTube has some competition from the rockier shores of DailyMotion, and I've started getting my TV listings from Radio Times rather than TVCatchup (so as to limit to Freeview channels). But that's pretty much all there is of note, except that as a trained librarian I've now got my googling time down to under four minutes a day, like the search ninja that I am.

I'll keep keeping a tally through 2015, though whether Opera can last that long remains to be seen. For now, though, that's quite enough stats.