2014 in bellybuttons

So this is New Year's Eve, and what have I done? Another year over, in which I made nine 'blog posts (eight if you ignore this one) - half the number of last year's tally. What's gone wrong? Is the 'blog dead??

The first two posts of the year were my usual navel-gazing stat-counts. I followed this up with a post about Excel functions. Then in March I had a spring-clean and the site got a new name. I claimed in that post that the next job would be to sort out my increasingly creaky website, which has sat pretty-much dormant for the last two and a half years (coincidentally the same amount of time I've been in a proper full-time job since I graduated, all those moons ago... what are the odds??). I have not lived up to this claim: I have done nothing to the site.

What did I do instead? In April I unleashed my Excel version of 2048 upon an unsuspecting world. I count this among the year's greatest achievements. In May I put up a PowerPoint slide (again, more showing off disguised as a helpful hint), then in July I did another Excel-related post, showing how to build your own spreadsheet version of usvsth3m's Fight! tool. This is what I had become.

Then, to prove I still had some connection to libraries, I had a good moan about CILIP (as is tradition). And that, ladies and mesters, was the last peep for five months.

This is the point where I should excuse my laxity by announcing that I've been writing a novel. But I've not added anything to that since March. Alas, the first half of the year was rather taken up with job applications, and (family aside) nothing destroys free time quite so much as those things. Free time and creative spirit. 

By the time that job app season was over, I was hard at work at work on various website and spreadsheet related gadgets, and the thought of carrying that stuff over into my leisure time didn't exactly appeal. So I watched a load of Doctor Who, listened to a lot of podcasts and generally vegged out for a bit. Frankly, I needed a rest.

I interrupted my resting for two Library Camps in the back end of this year (one in Leeds, and the national one in Newcastle), both of which were fun, and neither of which I wrote up (though I did do some live tweeting). Likewise, where's my entertaining spin on Sieghart? I could've had you rolling in the stacks with that one, I'm sure! But I didn't. Mainly because I felt I'd be rehearsing old posts (in much the same way that Sieghart said nothing very new). I didn't want to add to the water-treading... it would just have become a tedious tirade against lazy journalists, and given how much I've failed to write this year, my house is too well glazed for stones like that. 

The 'blog isn't dead, though. It's just resting. And while it rests, I'm still twittering away like nobody's business. 

Or am I...

Let's take a look at the evidence!

Total tweet tally for 2014: 6,000 (up from 3,787 in 2013). That's an average of 16 tweets a day (up six on last year).

5 RTs per day (up three) and 6 @s (up two). One contains the wordstem "libr*". Pulling that data out to a week, we get a table like this one:

	2012	2013	2014
Tweets/week	43	73	115
@s	21 (49%)	29 (39%)	40 (34%)
RTs	8 (18%)	17 (24%)	35 (30%)
libr*	3 (8%)	4 (6%)	9 (8%)

My proportion of @s is down slightly. My RTs are up by roughly the same margin, and my library tweeting has more than doubled in real terms. Well that's probably something at least?

Enough waffle. Let's cut to the word cloud!

  

  
When I'm not retweeting, I seem to be talking to @greebstreebling an awful lot, and the presence of @r4isstatic, @JMFrankland and @GrevilleTombs puts the brunt of that chatter at several Sundays spent watching Doctor Who on UKTV Drama. Had @ned_potter not changed his twitter handle mid-year (obviously inspired by my own rebranding), he would be up there too, proving that I'm more likely to tweet to someone than to shout across the office.

Library is down there in the bottom right, smaller than last year, and about the same size as Eurovision and Strictly. There's also a Libraries above @Girlinthe's head. The popularity of amp is actually due to my use of the ampersand, but I can see I've been getting to have the odd bit of a think now and then. This wordcloud doesn't include any tweets I've done this evening, but I doubt they'd alter things too drastically.

Peak tweeting this year was in July when I clocked in 712 (23 tweets a day), in part the result of me live-tweeting the Commonwealth Games opening ceremony (an evening I had completely forgotten about until I did the research just now), partly also #UHMLG14. My twittering low came in April, when I only managed 228 tweets (8 per day), presumably because I had better things to do.

So that's the story of my 2014... a story of better things to do, and of a greater variety of things to say when I do say them. Next year I hope to keep that up, while still having more things to offer here than tweet counts. Let's see how it goes.

I hope 2015 is very very lovely to you and that you thoroughly enjoy your time in it! See you later!

The Macro Terror

So usvsth3m came up with Fight! which lets you pitch things against other things to generate a ranking of things. One of their Fight!s seeks to rank Doctor Who stories, and I spent quite a while prodding the buttons to see how my opinion fared. Having done that for far too long I decided that the consensus emerging in the Fight! tally was a bit wrong, and I began to wonder what my own hierarchy would look like. I could’ve just scored each story individually but, being me, I turned to Excel.

Vincent & the Doctor vs Robots of Death. Hm.

To start with, I collated my list of stories, went to Fight! and did a spot of simultaneous manual marking just to get some numbers into the spreadsheet. That being done, it was time to rig up an Excel version of the Fight! system: essentially, I need to pick two stories at random (not the same one), and have a button for each story. Clicking on a button adds 1 to the winner’s WIN column and adds 1 to the loser’s LOSE column. Where to begin?

First of all, let’s get the two stories picked at random. That’s easily done. =RANDBETWEEN(1,241) will generate a random number between 1 and 241, and we can have two of these, plus a formula like =IF(J1=K1,IF(J1+1=242,1,J1+1),K1) to make sure that the two (nominally placed in J1 and K1) don’t give us the same number. We can then have the spreadsheet call the stories in question. A simple way is to make use of the row numbers by using an OFFSET formula, eg: =OFFSET(A1,J1-1,0) which will return a value from column A at the row-number specified by our random number in J1 (the OFFSET basically takes a cell reference (in this case A1) and then moves the specified number of places down and across. It’s great for mapping to a value that’s always in a particular place in the sheet).

The trouble with =RANDBETWEEN() ( and with its little sibling =RAND() ) is that the value updates every time we update the spreadsheet, and this is going to cause us trouble when it comes to adding our 1s into two different columns (the first 1 can be added, but then the random numbers regenerate and we lose our bearings). We were always going to need to use macros in this task, but it turns out we need them earlier than we thought. Time to open up Visual Basic and add a module...

At some point down the line we’re going to have two buttons in the spreadsheet, so we can click to vote for whichever of the two stories we prefer. I’m therefore going to write two subroutines: one for each button. These two subroutines will be fundamentally similar so we can build them up at the same time. 

If you want to play along at home, you’ll need to get the Developer menu displaying in Excel. It doesn’t show by default: you have to go into Excel Options and tick the “Show Developer tab in the Ribbon” box in the “Top options...” list. That being done you can go to the Developer tab, click on Visual Basic then Insert > Module to get started.

The magic portal to fun-aplenty

Let’s start with a couple of blank subroutines by typing:

Sub Button1()

End Sub

Sub Button2()

End Sub

Now let’s make it so that clicking one or the other button will generate random numbers in J1 and L1:

Sub Button1()

Range("J1") = Rnd() * 241

Range("L1") = Rnd() * 241

End Sub

Sub Button2()

Range("J1") = Rnd() * 241

Range("L1") = Rnd() * 241

End Sub

Rnd() is the Visual Basic version of =RAND(), and we’re multiplying by 241 as that’s how many stories there are this side of the 12^th/13^th Doctor (well, for the purposes of this exercise anyway).

Look at all those form elements. We're just scratching the surface here.

Go back to the spreadsheet, and from the Developer ribbon’s Controls panel click Insert and select the Button option (versions of Excel differ, of course, just to make life interesting). Make yourself a nice button and right-click to assign a macro to it. Assign Button1 to one of the buttons and Button2 to the other. Now click on the buttons and hopefully they’ll start belching random numbers into cells J1 and L1. 

We still need to ensure that we don’t get identical numbers in both cells and we can do that as we did before. We might also want to use the =ROUND function. I have =IF(ROUND(J1,0)=0,ROUND(J1,0)+1,ROUND(J1,0)) in J2 and =IF(ROUND(L1,0)=0,ROUND(L1,0)+2,IF(ROUND(L1,0)=J2,IF((ROUND(L1,0)+1)=242,240,ROUND(L1,0)+1),ROUND(L1,0))) in L2, just to keep my selections nice and tidy. You can then use the OFFSET formula to call up your stories’ titles.

That’s the easy bit done. Now we need to get the buttons to start adding figures to columns too.

I have my win tally in column B and my lose tally in column C. If the story in row 1 beats the story in row 2, the value in B1 needs to increase by 1 and the value in C2 also needs to increase by 1. With Visual Basic it’s not as difficult as it might first seem.

Sub Button1()

Cells(Range("J2"), 2).Select

Selection.Value = Selection.Value + 1

Cells(Range("L2"), 3).Select

Selection.Value = Selection.Value + 1

Range("J1") = Rnd() * 241

Range("L1") = Rnd() * 241

End Sub

Sub Button2()

Cells(Range("J2"), 3).Select

Selection.Value = Selection.Value + 1

Cells(Range("L2"), 2).Select

Selection.Value = Selection.Value + 1

Range("J1") = Rnd() * 241

Range("L1") = Rnd() * 241

End Sub

The first line here is the Visual Basic version of =OFFSET. The “Select” bit says we’re going to ‘click on’ the “Cell(s)” specified by the coordinates in the brackets: 2 = Column B; Range(“J2”) = the Row specified by the value in J2.

The second line says what to do when we get there: namely that the value of the selected cell should now equal the value of the selected cell plus 1. Quite a neat little line really.

Go back and have a few clicks to see how it works. You’ll notice that your selection ends up somewhere in column B, which is not very helpful if we have to keep scrolling back up the page, so you could go back and add something like:

Cells(1, 14).Select

...before both “End Sub”s, just so that a cell at the top of the sheet is selected. It just gets the cursor back to a sensible place.

That’s the hard work done. Now you can start merrily clicking away and building up the raw data you’ll need to create a ranking. Sticking something like =(B1-C1)/(B1+C1) in D1 will give you a normalised score from 1 (wins every time) to -1 (loses every time) which you can then set about ranking using the =RANK formula. You could then build a dynamic leaderboard. I built mine using a =VLOOKUP: =VLOOKUP(P1,E$1:F$241,2,FALSE) where column P has running place-numbers, column E has place rankings, and column F concatenates the story names from A with the hit stats from B and C. The problem with this approach is that it relies on there being no tied places in your rankings, which is frankly unlikely. There are some elaborate ways of avoiding this kind of thing, or you could do what I did and cheat (I basically beefed up the figures in D and then added a tiny arbitrary fraction (e.g. 0.0000241 for D1, 0.0000240 for D2 etc.) effectively breaking the tie by alphabetical order). Cheating can be your friend for such things as this, but for something more important you could just record yourself a nice, neat sorting macro.

Ok. So everything is lovely. You’re able to call up story titles and vote for your favourite, and get a running ranking as you go along. You can even call up various stats at the same time and conditionally format them to give you a cue as to which story is likely to be the better one. You might also use a deduping condition to highlight a contending story in your leaderboard. But there’s something wrong. Something it may take you a while to notice, but it’s something that is very very wrong...

After a few thousand clicks; a few sessions, perfecting my formulae and my hierarchy; that’s when I spotted it: That’s when I noticed that my supposedly randomly selected pair-offs were becoming disturbingly predictable. You can’t help but notice when you get “Curse of Fenric” vs “Curse of Peladon” a few goes before “Abominable Snowmen” vs “Snowmen”. My random selections were following a defined pattern.

Computers don’t really do ‘random’. It’s not in their nature. Instead they play around, doing entertaining maths with ‘seed’ numbers. The problem I was having was that the seed number wasn’t changing, and every time I made an alteration to my macros and started again I saw some familiar face-offs.

One of the ‘more random’ seeds a computer can use to make its ‘random’ numbers is the clock. Time is constantly changing, from second to second. What we can do is tell our macro to get a new seed from the clock every time it generates a random number. Sticking Randomize Timer as a new line before each of our Rnd() lines should do the trick.

Phew.

Me being me, I wasn’t happy with just having a ranking based on the ratio of wins to losses. I wanted something a little more elaborate. My current fights therefore have more at stake. Each story is allocated simple ranking points (241 for 1^st place, 240 for 2^nd place, etc), and these inform another tally of fighting points: If the winner of a battle has less ranking points than the loser, they trade these in the form of fighting points. So if the bottom-placed story beat the top-placed story, the bottom story would net itself 241 fighting points while the top story would have 1 fighting point. Fighting points therefore effectively represent a sort of scalp trophy. The ranking and fighting points are added together to give an overall ranking. This system allows stories to move much more quickly through the league table and makes the order far more accurate.  The current order still has a few wrinkles, but it’s now a pretty reasonable reflection of my tastes and opinions.

Blink is about to cement its place ahead of Caves of Androzani.

The clicking is never done...

Surely there are better ways of doing this? Alas, a full round-robin would require 28,920 clicks and I've pulled together my current ranking from but a mere 8,131. However, I may have to come up with a way to fix the random number generators to select two stories within, say, five places of each other, just to get rid of pointless battles like the one above.

There you go, then. That’s how to take the fun out of a diverting web amusement. You may rest safe in the knowledge that Victory of the Daleks sits safely plumb-last in my list. Fiat 500 Daleks are not for me.

"Will no one rid us of this turbulent membership?" Mustering techniques to avoid.

I am a member of CILIP. I am a member of CILIP because my peers at Library School were also members of CILIP. I am a member of CILIP because maybe at some point I’ll get around to doing the Chartership thing, even though I’m not still 100% convinced that the whole enterprise is not just a white elephant money-making racket for CILIP. I am a member of CILIP because I believe it is important for an area such as librarianship to have a strong body that represents the interests of libraries, librarians and library users.

Things I am not a member of CILIP for:

CILIP Update; training events (though some would be nice); opportunities to engage in democratic processes; small print scrutiny; trying to get my head around an at first glance bizarre organizational structure; reading a CILIP councillor describe dissatisfaction over a rebranding exercise as betrayal; reading an ambassadorial figure within CILIP shrugging off dwindling membership as aninevitable function of library closures and deprofessionalization; reading the same figure address the same issue of dwindling membership in terms of “quality over quantity” (I don’t want this to drift anywhere remotely close to an ‘answer-post’, let alone character assassination -- there’s something wrong-place-at-the-wrong-time about the CILIP President role and a slip of the keyboard is all too easy -- but just take a few seconds to mull the ramifications of that statement)...

There were more different types when I were a kid.

In case you’ve missed it, the fiddles are out at CILIP again, this time regarding the recent Governance Review. You may have missed that too. I’ve a distant hazy memory that I once followed a link from an email (though a search of my inbox suggests that I deleted whatever email provoked it), saw lots of boring governance stuff and then got distracted by BuzzFeed or something. It’s not really been a debate I’ve followed, partly because it’s not really been a debate. I could express in broad strokes (and reasonable reasoning) my sense of rights and wrongs, but the simple truth of the matter is that this one seems to have slipped under the radar in a way that the whole ILPUK debacle didn’t. It’s all building up to become another deck-chair shuffle of an AGM.

The first deckchair to be vacated was that of enfant terrible Tom Roper, who explained his reasoning in this blogpost. His resignation brought the detail of the Governance Review to the attention of twitter. In a way it’s a shame he didn’t go before the review’s close-date (think of the traffic!), but such is the nature of cause and effect. Council chair Martyn Wade responded to Roper’s post, and Roper’s replied here. And this weekend President Band added her thoughts. 

From my perspective, CILIP’s governance is overly elaborate: an elected President and Vice-President with so little power and influence that nobody stands anymore; an appointed Chair (to assume the Presidential office in the proposed regime) overseeing a mixed bag ofelected and appointed Councillors; a paid CEO “delivering the change programme and effective charity management”. Some simplification might not go amiss. The problem seems to be that the proposed simplification is a simplification that reduces rather than inspires member involvement: the role of President becomes more relevant but we’re not allowed to vote for who might get it anymore. The number of elected Councillors drops from nine (or potentially all) to eight, which seems another backwards step.

Voting in these things is always going to be pretty low. We don’t generally know enough about the candidates to feel too strongly. Therefore, CILIP elections should be at worst a formality and at best a way to engage any membership so inclined. So why reduce the membership’s say? That just seems stupid. Almost as stupid as ILPUK. Seriously... why would you do this?

Why?

And it’s the fact that CILIP’s powers-that-be seem to make such a mess of the easy things like sensible, positive member engagement that is so frustrating. These things should be non-issues.

So will this September's AGM be dominated by yet more belly-button stare-offs? Or will there be some motions tabled that actually engage with the very real threats that library services (in all sectors) are facing? Perhaps you have a motion you’d like to put forward. You’ll have to be quick, though. You’ve got until 5pm tomorrow (Monday 21^st July). More details on how to submit a motion can be found here. I hope some of you are more inspired than I am.

Barbara Band says some good stuff in one of her replies to a comment on her post:

“[We] are best served by having a national professional body that can have a positive strategic impact on Government policy... [via] the work it has done with the All-Party Parliamentary Group on Libraries, or the work on Information Literacy, or the policy responses it has submitted, or the frankly bloody good work that Annie Maguer does 7 days a week [weekends are important, Annie, but thanks all the same] on securing influence and credibility with policymakers... [This is] the only effective way to defend the long-term and precious principle of the right of access to library services... CILIP needs to influence Downing Street and Treasury... CILIP needs you... like we need everyone in the library and information professions.”

I agree with most of what’s being said here. This is the important stuff. Getting a clear message deep into the halls of Whitehall and Westminster. Saving libraries. 

Barbara continues: “We need a credible brand and a professional presentation.” This is at the heart of CILIP’s recent efforts regarding rebranding and governance. Maybe CILIP are right: politics has always had an obsession with image after all, especially in the last couple of decades. “Most of all,” Barbara adds, “we need people to understand that whether the battle is personal or professional, we get absolutely nowhere by fighting amongst ourselves.”

Which is the underlying sentiment of this post. It’s also why members get affronted when CILIP’s governance seems to start picking fights with them, be it by conducting lip-service surveys ahead of an unpopular namechange, or by reducing the few democratic pathways though which the membership can influence said governance. This is a particular affront at a time when some of the membership may feel that the governors in question are not doing enough on things other than name-changes and internal restructures.

It seems that I agree with Barbara (and probably the rest of the CILIP great-and-good) that we should stop fighting among ourselves and start concentrating on fighting for the future of libraries. What we probably disagree upon is the reason the in-fighting is there in the first place (and indeed the potential solutions to it). The disgruntled membership are not out to cause trouble for trouble’s sake and I doubt CILIP's leaders are too. We’re not the agents in this so it’s a bit galling when we’re told to be quiet.

The argument can perhaps be summed up as: 

CILIP MEMBERSHIP: Stop messing about!

CILIP GOVERNANCE: Stop arguing with us!

CILIP MEMBERSHIP: Stop making changes we don’t like!

CILIP GOVERNANCE: Shut up!

The elephant in the 'blog post.

I’ll be paying my subs for another year because I really really really want CILIP to be out there fighting for libraries. And I really want to be able to support them in that fight. But I’ll not be shutting up. And I will be exercising my ballot at the AGM. I voted for Tom Roper in the Council elections last year and I thank him for his efforts in representing the membership during his brief tenure. I also thank him for being a one-man publicity machine for CILIP’s internal democratic processes, as without his annual agitations the AGM and its agenda would doubtless pass me by un-noticed. Finally, I thank Barbara Band for everything she does in being an ambassador for The Cause. It’s far far more than I do and it’s very much appreciated. 

Full and frank debate is healthy and proper, and a necessary part of good governance (wherever possible taking place ex camera so we all know what’s going on). In the course of such debate we should, of course, never fail to lose sight of the fact that the real enemy is without.

Just a bit of fun...

Of all the PowerPoint slides I've ever made, I think this one may be my favourite:

It's really quite a simple one too: repeated freeform motion path animations with the smoothing turned off. Right click and save-as to download a version of it (1.7M).

2048.xlsm

Today I had a day off work, and what better way to spend it than by programming an Excel spreadsheet 'port' of the web game 2048? So that is exactly what I did!

This is a beta version of the game: I'm not entirely convinced that I've got every aspect of the gameplay quite right (my random weightings for the spawning of 2s and 4s may be a little out), and there's always the small possibility of the odd formula error somewhere along the line causing tiles to misbehave. Anything like that can be easily ironed out. The point is that it works!

[Edit - I spotted one problem that needed fixing: new tiles shouldn't spawn when all adjacents in a direction are doubled and shunted. Tut. I've added a new line of sums to monitor and block spawning in this situation, and things are working a lot better as a consequence.]

The game was designed in Excel 2007, and requires macros to be enabled. Right click here to download yourself a copy at a tidy 32K. If your browser tries to save the file as a .zip, be sure to correct it and to save it as a.xlsm file.

The gubbins that works the game can be found to the right of the sheet, explored and amended to suit your tastes. Basically, the five buttons (four cursors and a "New Game" reset) copy the current grid to a space below it (or in the case of the reset, copy a blank grid), and then the gubbins does the rest. 

Each move follows a three-stage process: shunt the tiles together to skip the blanks; double up adjacent duplicates; shunt again. Calculations are made for all four directions. This was the trickiest bit to work out, and I'm thankful that there are only four rows to work with at a time. There was a fair bit of trial and error beyond the initial idea of what should happen.

The random tile was another tough thing to implement. It is calculated by totting up the number of empty spaces, then allocating each one a number (using =COUNTIF), and picking one of the spaces at random using the =RANDBETWEEN function. I'm quite proud of this, though like I say, I'm not sure I've got the weighting right as I'm clocking in some high scores.

Scoring is achieved by totting up the difference between the shunted grid and the doubled grid, and applying a square/root to knock off any negatives.

Anyway, feel free to have a trawl of the cells and let me know if you find any glaring errors. If you'd like me to make a step-by-step work-through I'll give it a go.

Marathon (or "Hello again, World...")

Since the late 1990s I've been making things under the moniker A/V Woman, most obviously my website, which itself is old enough to make babies. So here's one.

When I first dabbled with twitter I used the name @KingConstance by way of an OED-inspired, Sarah Grand-related pun. When I started tweeting about libraries I created a new twitter account: @SaintEvelin : it's a title/name form that riffs on the @KingConstance account (the avatar nods that way too) while also being a tenuous pun on my first names. I suspect, too, that @AVWoman was being used by someone at the time, though maybe I'm making that up. Maybe I just wanted a further separation there. 

At the same time as setting up @SaintEvelin, I set up the Succentorship without Sneers 'blog: http://saintevelin.blogspot.co.uk/ (this 'blog, essentially). Both have served me very nicely, I'd like to think. 

But I've tired of late of the SaintEvelin pun, not least because I'm not called Evelin. I'm also wanting to do something more with my website (I'm not entirely sure quite what of yet, but that's something to look forward to). And I'm feeling that A/V Woman is a perfectly good alias for a technologically minded librarian to hide behind. I've been a bit quiet recently, both on here and to a certain extent on twitter too, so now seems as good a moment as any to put the swap into action. 

I made the twitter-switch at tea-time yesterday. Most twitter platforms these days make more play of the screen name than the handle, so switching handles isn't as obvious as it used to be. This has its pros and cons, but I think for the most part the avi and the screen name carry the brunt. In the interests of consistency I won't be swapping these. Not at the moment, anyway. I can cause more confusion in the long term this way!

Twitter is very good at handling a handle switch. Continuity is pretty seamless. The biggest drawback is if you cold-call me on my old moniker, but see the previous paragraph. I think I'm the last person left who tweets by text message. But I'm determined not to miss any messages. To account for this I've set up a holding account on my old handle. In a month I'll switch it to some other use, but in the meantime it's a neat way of occupying the void (it also staves off any would-be scavenger, keen to don my shedded skin to the further confusion of all involved). Later I'll still mop up the off-chance with a tweetdeck search column.

Then there's the 'blog. I could've just swapped the domain over from saintevelin.blogspot to avwoman.blogspot, but again I'm concerned about continuity. And in this case continuity is all the more important: I must've made it to someone's news feed, and I know there are one or two links to my 'blog from hither and thither. I don't want to lay waste to such connections. So I've trod a little more carefully in that regard.

So it is that this tabula rasa at avwoman.blogspot.co.uk is not as violent as it might have been. The old posts are still there at the old address, and the search box in the side panel will lead you back into its depths. More importantly, I'm posting this entry (and any that might follow) to the Succentorship without Sneers RSS and Atom feeds. So while this post is the first at the avwoman domain, and sits on a different page to my previous posts, the newsfeed remains a continuity of service. I think that's important.

All this adventurous plumbing being done, it will be time for me to weld it all to the website, which is kind of the whole point. But I'm still a lot hazy on what might go there. Until then, my apologies to you all for the mucking about. I'm as bad as those Olay/Ulay people, I know.

You may now updated your bookmarks... 

TLFKASE 

Some more Excellent tricks

I've been playing around with Excel an awful lot lately, which is always guaranteed to cheer me up. Time, then to add some more little discoveries to the pile of Excel-based bits and bobs I've put your way in the past (specifically here, here and here).

Blank skip

How many times have you been infuriated by empty rows getting in the way of good analysis? Perhaps the answer to that question is "none", in which case "clever you". Me, though; I'm always getting myself into situations where for whatever reason I find empty cells getting in my way. One example is to be found in my personal accounts, where I have daily entries, and then weekly and monthly summaries appearing down the side. These summaries are cast adrift amidst rows of emptiness, and that can have implications when it comes to wanting to do anything very involved (graphs, for instance, can be a pain). There are ways around this: I could producing running figures down the page, but that would be ugly and would defeat the idea of the summary; I could copy and paste somewhere else, skipping blanks or sorting them away. That would look better but it would have dynamic repercussions, and I want to perform calculations using live data rather than a copy and paste of a particular point in time. What I need is an array formula.

In this example I have three columns: A contains the days of the week (my x-axis, if you like), while B and C contain data entered on certain days. To abbreviate that table to just those rows that have data in column B, I've popped the following into D1: 

=INDEX($A:$C,SMALL(IF($B:$B<>"",ROW($B:$B)), 

ROWS(D$1:D1)),COLUMNS($D1:D1))

Most importantly (and this is the weird and counter-intuitive bit) you need to press control, shift and enter at the same time (rather than just enter) to make this formula stick properly. Doing this will add those curly brackets round things ({}) and make everything work as if by magic. Infuriatingly, just typing the curly brackets doesn't work. It's all about the ctrl-shift-return. 

Once you've ^⇧↵ed into D1, you can expand the formula across the rest of the cells in the usual way (and colour everything all rainbow-like, if that's how you roll). There are errors in D5-F7, because there's no data to feed them, but that can be sorted out with an =IFERROR in the right places if it bugs you. 

So what is the formula doing? Let's pull it apart:

The bones of the formula is a simple cell reference from a defined array (the first three columns):

=INDEX(array, row number, column number)

The column number bit is a simple column-counting formula that will increase as we drag across E and F. In D1, the value is 1, in E1 it is 2, and in F1 it is 3.

The row number is more involved because this is the bit that's doing the hard work. There's a conditional formula in a SMALL formula that relies on the row number. The condition is that we should return the row number only if the value of a cell in B is greater than or less than blank. The rest of the SMALL formula states that we're only interested in the kth returned value, where k corresponds to the row we're on (so if we're on row 1, which D1 is, then we're after the first value where the value of B is greater than or less than nothingness). 

Doing all that with a normal formula won't get you very far, because you'd be unable to keep count of those successfully met conditions (you could keep a running tally, but we're trying to keep our sheet clean of that sort of thing), but turning it into a magic ^⇧↵ array formula lets Excel tot up various details across a whole range of cells.

Array calculations do make Excel have to think that little bit harder, so you might want to avoid whole-column arrays like the ones I've used.

Offset Mapping

You know what's really irritating? When you've got a lot of data in Sheet2 that's reliant on the data in Sheet1, you delete a row from Sheet1 and everything on Sheet2 goes all error-y. Likewise if you sort Sheet1 in a new way and Sheet2 becomes a befuddled mess. What you need is to break the link between a cell in Sheet1 and a cell in Sheet2 while still carrying over the data. What you need is to be able to map to a particular cell, regardless of its contents. One way of doing it is to use INDEX (like in the example above), for which you must establish an array range. But perhaps easier is to use OFFSET. While INDEX pinpoints a location within an established array, OFFSET pinpoints a location relative to an origin (e.g. cell A1). The difference is subtle but slightly more flexible.

If we want to reproduce the contents of any cell in Sheet1 in a corresponding cell location on Sheet2 (without tying the two together in a relative formula), we can use:

=OFFSET(Sheet1!$A$1,ROW()-1,COLUMN()-1)

The ROW() and COLUMN() bits get the current coordinates, and these are applied with reference to cell A1 on Sheet1 (the "-1"s account for the fact that A1 takes up a cell of physical space). The offset location can be anything we want it to be, as, for that matter, can the offsets (the example above is just a convenient starting-place). 

I'm currently using this to make reading lists. I can use one sheet as a sandpit (inserting and deleting rows, and shuffling content about the place), then have a second sheet which takes a snapshot of that content and arranges it in a pretty way without me having to do anything. It's my favourite thing in Excel at the moment.

The Name Game

Ok, this one's a brag more than a formula...

I just mentioned that I'm using Excel to generate reading lists, and I thought it would be wise to come up with a set of formulae to standardise whatever I put in and turn it into a Harvard citation. On the face of it, this shouldn't be too tricky. We can all look at a citation and work out what's what and how we might move the commas and brackets about to switch from one referencing style to another. Why a program like EndNote makes the process of importing citations such hard work I'm not entirely sure, given that even allowing for the vast wealth of different formats and models, the basic codes are all variations on a theme and surely a reference manager could work out which it was looking at if it could but really be bothered to try. But that's a different quibble. What I'm wanting to do is take a pre-existing citation of indeterminate style, dismantle it into its constituent (citation-manager friendly) parts and reassemble them. Unsurprisingly, the hardest bit of that is the identification of the dismantled parts. The presence of numbers, capitals, and various forms of punctuation are a clear help in this regard, whatever style is in play. Isolating the publication date is very easily done, and things start to unravel nicely from there because there are only so many ways you can describe the elements of a citation (at least in the common forms and for the common formats, which is what I'm playing with). A title is a title and varies very little as a consequence. Even volume and issue numbers follow some sort of discernibly extractable logic. In fact, there is only one part of your general, generic, stylistically vague common-all-garden reference that poses an especially pressing problem to a poorly programmed computer and that is the author name. Having extracted every other necessary building block, I was not to let myself be beaten by a few names.

There are a number of reasons why names cause trouble. They can be awash with commas and full stops, performing a range of different and often contradictory roles: commas between surnames and forenames, commas between collaborating authors; full-stops after each initial, or all initials, or none. Which brings us then to the swirling mess of first names, surnames, middle names, initials etc. But as humans we can look at these things and work our way through them with not too much trouble, so surely a computer can too. Admittedly, part of the human advantage is a reference database of typical forenames and surnames: something which could be emulated in Excel if needs be, but it seems an extreme length to have to go to. There must be a simpler way.

Well, reader, I managed it. Having already isolated the name portion of the original citation, I then dismantled it word by word, classing a comma as a word but eliminating full stops from the equation, to give something like:

Smith , A , Jones , B and McDonald , C

 Now to work out what each cell actually is. Commas and "and"s are easy to determine. Initials were identified using something like: 

=IF(OR(LEN(A1)=1,IFERROR(CODE(MID(A1,2,1))=CODE(UPPER(MID(A1,2,1))),FALSE)),"I1,"")

...in other words, if the length of a cell is one character, or if the second character is upper-case, consider the cell to contain one or more initials.

This is a good starting point and will serve us in good stead. The use of "I1" is to indicate the first instance of initials. Later initials can be given "I2", "I3", etc. This will help when it comes to piecing things back together properly.

Initials are easy. Forenames are pretty easy too, if they immediately follow a comma. An entry that goes SURNAME COMMA FORENAME is easy to tag. SURNAME COMMA FORENAME COMMA SURNAME COMMA FORENAME... is slightly more troublesome, but here maths is our friend: if we assume some sort of internal consistency, then the first comma indicates a forename, the second a new surname, etc.

Sometimes, commas are used more sparingly, eg. SURNAME INITIAL, SURNAME INITIAL, etc., but here we have the benefit of an initial to help us through the soup. Thankfully, SURNAME FORENAME is not something that tends to appear in the occidental world, so we can probably apply the assumption that nothing like that will bother us.

From these foundations, it's quite easy to put together some formulae to draw out the names we need in the majority of cases. But some people have really awkward names. And some people aren't people at all. People with two forenames or two surnames are especially troublesome. Again, commas (and "and") are our friends here. And we can begin to string together some really messy formulae based on relative positions of our respective bits of names. There's a lot of trial and error to this, but after some messing about it's possible to get something together that works almost universally (I've run my current version across about 400 citations and it seems to be working ok). Then it's just a case of glueing everything back together again.

Here's a link to a very-much beta version. The gubbins is very messy trial-and-error stuff, and I'm sure you'll be able to break it quite easily if you try, but it might give you a place to start should you ever require something similar.

The 40 most-visited-by-me sites of 2013

How much time do I spend on the internet when I'm at home, and what am I looking at? A few years ago I decided to set myself that very question, and to answer it I stuck a clock on my browser. Unfortunately, I forgot to collate the first couple of years' stats. But I've since tweaked the counter a few times, and this year I've actually remembered to crunch the stats therein. The list consists of my 50 most-visited domains, some of which I've combined because they amount to the same site, and one of which I've removed because it is buggy (I did not spend five days watching Channel 4, and the suggestion that I did seems to have stemmed from some corrupt data left in the file from the previous year, but their website should probably be in this list somewhere). Another note before I show you the list: for the first half of the year it was clocking stats from all open tabs, and in the second half it was limited to the active tab. Also omitted is browsing from any other platform except Opera (so my brief flirtation with Netflix is not listed as it took place in Firefox owing to compatibility issues).

Here, then, are my 40 most-visited sites (caveats already noted). For more information about a site, hover over the name.

		Total (h:m)	Total (d:h)	Daily av. (m:s)	%
	OVERALL	1254:00	52d 6h	3h 26:00
1	Tweetdeck	382:57	15d 23h	62:57	31
2	Wikipedia	72:02	3d	11:50	5.7
3	Old Reader	71:54	3d	11:49	5.7
4	Blogger (my 'blogs)	59:37	2d 12h	9:48	4.8
5	YouTube	57:19	2d 9h	9:25	4.6
6	Facebook	52:41	2d 12h	8:40	4.2
7	BBC	41:36	1d 18h	6:50	3.3
8	Google	37:18	1d 33h	6:08	3
9	Google Maps	28:51	1d 12h	4:45	2.3
10	Twitter	18:15		3:00	1.5
11	(Local Host)	17:02		2:48	1.4
12	Historic Digimap	9:26		1:33	0.8
13	Splendid Chaps	8:03		1:19	0.6
14	A/V Woman	7:49		1:17	0.6
15	Alveley Historical Society	7:11		1:11	0.6
16	The Guardian	6:24		1:03	0.5
17	jobs.york.ac.uk	5:25		53s	0.4
18	TVCatchup	5:08		51s	0.4
19	Bing Maps	4:34		45s	0.4
20	Sporcle	4:25		44s	0.4
21	RightMove	4:10		41s	0.3
22	(Router)	3:38		36s	0.3
23	subjectguides.york.ac.uk	3:20		33s	0.3
24	Lawrence Miles ('blogs)	3:17		32s	0.3
25	Sheffield Planning Applications	3:08		31s	0.2
26	NRK	2:50		28s	0.2
27	Lady V London	2:48		28s	0.2
28	BBC Good Food	2:47		27s	0.2
29	Know Your Meme	2:30		25s	0.2
30	Pinterest	2:27		24s	0.2
31	Amazon	2:21		23s	0.2
32	Flickr	1:52		18s	0.1
33	Out Of This World ('blog)	1:47		18s	0.1
34	The Golden Era of GP Racing	1:43		17s	0.1
35	Tattuinardoelasaga	1:41		17s	0.1
36	thetrainline.com	1:40		16s	0.1
37	IMDb	1:39		16s	0.1
38	Gizoogle	1:36		16s	0.1
39	University of York	1:28		14s	0.1
40	Doctor Macro	1:27		14s	0.1

So there you go. The above list accounts for about 85% of my browsing (a stat that includes the otherwise omitted Channel 4 entry for necessary mathematical reasons). The colour-banding groups things together semi-thematically. What do we learn from all of this? That I like maps? We already knew that (or I did anyway). 

To get an idea of what this is telling us, we need another data-set. My other data-set is anecdotal. It is me remembering what sites I used most about a decade ago. About a decade ago I mainly used Google, Wikipedia, BBC and The Guardian. Those sites constituted a Big Four, with my own content probably filling in a fifth slot. The rest of my internet activity can be discovered from the array of bookmarks I still have in my browser from Them Days: home-made websites (largely replaced by 'blogs now, and consumed via my feed reader). Gone now are the days when I sat grazing the news at The Grauniad and the Beeb (International Version). Twitter feeds now deliver that news to me. But the difference is not as radical as if we roll back the clock another five years, when my most-used sites would've been Alta-Vista and Yahoo! We've come a long way since the Web Ring. We've gone full-circle and found ourselves back in Usenet, but with a 140 character limit.

I'm now going to clear the stat counter and hopefully come back in another 12 months to do some proper, more meaningful analysis of my browsing trends. Between those things, I need to do the pots and make some tea.