A Junior Academic’s Take on Digital Scholarship

I’d like to offer my perspective as a junior academic, who is in the process of applying for early career positions, on Martin Weller’s recent and excellent talk on Digital Scholarship. (The talk starts about 20 minutes into the recording.) Deep down I believe in the open, networked, digital scholarship that Martin describes.  The question that I am struggling with is “am I willing to put my life on the line for this belief”.  As Martin points out, tenure and promotion is based on teaching, research, and service.   Peer-reviewed journal publication, as a proxy for research merit assessment, is likely what my hiring and promotion will be based on.  For that reason, I need to start loading the publication pipeline.  Publications don’t happen overnight, and so it’s important for me to focus on this even in the absence of a firm job offer.  At this point in time I’m not willing to jeopardize my family’s well-being to change the system.  Maybe I will be lucky beyond my wildest dreams and wind up at an institution that values open, digital, and networked scholarship, in which case I will be far less concerned about maximizing my peer-reviewed journal publications.  But until that happens the risks associated with committing substantial effort to open scholarship outweigh the benefits.

That doesn’t mean that I won’t blog or participate in learning networks.  It just means that I will adopt a balanced, pragmatic approach that should maximize my chances of success in these exciting times of changing scholarship.

Copenhagen V: Codename 'FARGO'

It’s been a week since I described what I was going to try to do with Repertory Grid on Facebook. It’s been a very productive week and I am pleased with the early (alpha) version of a Facebook app that is codenamed FARGO (Formative Assessment using Repertory Grid Online).

I started with the demo app that Facebook distributes called “Run with Friends” and worked it up into an app that can elicit a Repertory Grid, show some results, and (optionally) post some of the results to a Facebook page. There is still a fair bit of work to be done before it’s ready for use but the early results are encouraging. It has also been a real boost to get feedback from other NEXT-TELL team members.

I was faced with an interesting problem from an Human-Computer Interaction perspective: how do I elicit a “real” grid, given that not everyone who will be testing the app is in the same course? What do these people have in common that could be used as elements in a Repertory Grid? It occurred to me that because we’re using Facebook as the front end I could request access to the list of users’ friends and then use a random sampling of their friends to elicit constructs. There’s a lot of interesting (psychological) research to be done on that sort of topic, but that’s not my primary focus. I am most interested in creating better visualizations of the data that comes from a repertory grid. As a proof of concept I hooked up the repertory grid output to a dynamic visualization using protovis to create a network diagram. As I expected the network diagram, in its current form, isn’t particularly useful to or usable by end users. No big surprises there!

I want to backtrack a bit change the app to elicit some feedback on some of the more traditional representations of the repertory grid, such as the (raw) matrix of elements and constructs, as well as the clustering output. The former is easy to implement. The latter may require me to hook up to Rep 5 to get the clustering done, or perhaps I can just use a python library to do so the clustering on the server end of things. The python-cluster library looks promising (I need a 100% python implementation due to restrictions put in place by Google App Engine). I could then use the dendrogram visualizations provided by protovis.

So those are the next steps: get some of the more traditional representations hooked up and get more feedback from folks who are willing to test out the app. I also need to provide a bit more context for the “lead-in”, as well as doing a better job at the end so that when one decides to share via Facebook one is redirected somewhere nicer that just the same page. And in the event that the user doesn’t want to share via Facebook I need to make the link to their visualization available.

I also need to start thinking about how to generalize this out to be about formative assessment on any topic… perhaps allowing some sort of “preconfiguration” for a particular topic. Rep 5 does this but the UI is a bit unfriendly. So I need to strive a balance of something that just works and the ability to customize it to different domains.

Copenhagen IV: Repertory Grid Implementation

I’m working on the use of Repertory Grid Technique (RGT) for formative assessment. We are interested in using Rep 5 but there seem to be delays in the ordering process and we are facing some pretty hard deadlines with the end of the elementary school year and the beginning of the summer (university) semester approaching. So we don’t really have much choice other than to implement our own RGT software.

I have been interested in Google App Engine (GAE) for some time, but up until now I had no good reason to pursue it seriously.

For the past couple of decades I have always had access to a machine running some variant of the unix operating system. That all changed recently when I moved to what is, for the most part, a Microsoft shop. Of course I have my trusty laptop running Mac OS X, which sports a lovely unix-like operating system under its flashy hood. Prototyping on my laptop is easy for me, but when it comes to deploying software for others, running production services on my laptop just doesn’t cut it.

I toyed with the idea of buying some space in the cloud but I thought it might be edifying to see if I could cope without my own (at least virtually) dedicated machine.

Over the past couple of years I have also become quite interested in python, particularly for scientific computing. Python is a good fit with a lot of the work I’m doing with topic models.

We want to try using the RGT with some summer students at Copenhagen Business School. Being good HCI folks we are naturally concerned about the user experience. We hear a lot about the hassle of multiple login systems and the like so I need to deploy the software in a way that allows me to authenticate users with whatever system they’re using. I’m lucky in that the course in which I’m planning to deploy the software requires students to use Facebook, so I can use the Facebook authentication system. Therefore, it makes sense to deploy my software as a Facebook app.

I’m not only interested in using RGT, but also in improving the representation of the results. The current “best practice” calls for either clustering using dendrograms, or using principal components analysis to create a “map” of the elements and the constructs. We know, based on some pilot work and from talking to other researchers that the current representations of RGT certainly leave room for improvement. I’m planning to create interactive visualizations for the representation of the Repertory Grids. My current plan is to use a javascript-based visualization toolkit to create the interactive representations. If I manage to do this well then we will have a system that allows learners to create and share visualizations of the conceptual understanding of a topic area. That would meet the requirements for a prototype of a communication and negotiation layer for learner models.

Overall, then, I am planning to implement Repertory Grid Technique software as a Facebook app using Google App Engine as the back end and protovis to create the representational layer.

Copenhagen III: Repertory Grid

I have been investigating the use of Repertory Grid Technique (RGT) for formative assessment. The technique is based on Kelly’s (1955) Personal Construct Psychology. The general idea is to interview someone to elicit elements (things) and then pull out the relationships between elements by eliciting constructs. There are many ways to do this, and we’ve been experimenting with some of the better established means that are part of WebGrid. Once you have a bunch of elements, you present triads of those elements (randomly chosen — or not) and ask the participant to choose one element from the triad that’s different from the other two. You use that grouping (two similar elements plus one different element) to create the poles of an axis, which you then ask the participant to label.

I’m new to this technique and so I’m just ramping up on it. I got the chance to work with a group of teachers to create a Repertory Grid of their understanding of formative assessment. (Note that this is not what I meant by using it for formative assessment in the previous paragraph. It’s just that the technique is very flexible and it was illustrative to use the topic of “formative assessment” to elicit elements and constructs from the teachers.)

We used the WebGrid software developed by Brian Gaines and Mildred Shaw, professors emeriti of the University of Calgary.

We asked them for examples of formative assessment that they currently used. They enumerated nine: questioning, classroom monitoring, observations, in-class assignments, games, little tests, homework, workbooks, and reviewing written work. We then used triad to elicit constructs and we got four of them: “involves students – teacher’s job”, “closed questions – open questions”,”skills – competencies”, “outside learning situation – takes place in learning situation”. We were pressed for time and part of the exercise was to expose them to WebGrid so we stopped eliciting constructs at that point.

Here’s what we learned: (1) RGT is a promising approach, (2) the existing WebGrid software needs to be customized and localized to make it usable, (3) the representations of the output needs to be improved. That’s good news. We can customize the WebGrid software quite easily (or so the manual tells us). That means we can focus our attention on the representation layer.

We need to figure out the best way to use RGT as a formative assessment tool with students. THe gist of what we’re going to try is an elicitation of the key concepts for a particular curricular unit (for example, rocks and minerals) from each student in a class. Each student will then go through a construct elicitation step, and we’ll present the results. Perhaps the students will share their RepGrids with the teacher and perhaps with each other. The students will then work their way through the unit and repeat the RGT exercise after the unit. Again, sharing would be great. We can then compare the before and after RepGrids.

Kelly, G. (1955). Principles of Personal Construct Psychology. New York: Norton.

Copenhagen II: On time

[This is a posting that I composed near the end of March, 2011 but never finished. I am posting it anyway, with the hope of returning to some of the ideas. Someday.]

Some of my time this past week was spent reflecting on the paper that I wrote as part of the book that we are working on as an outcome of a series of workshops on Multivocality in Research.  I presented a new method for the analysis of networks of learners, one that takes into account the notion of a Zone of Proximal Development and builds on the automated analysis of participant contributions.  The work was done on an existing data set (data reuse is another research interest) and whereas that provided a number of benefits it was frustrating because we couldn’t actually DO anything to effect change:  we could only study what had already happened.  Nobuko made some great suggestions about backing up to earlier points in the data, making predictions about what would likely happen in the following week and then seeing if the actually happened.  We’ll present these ideas at the upcoming Alpine Rendezvous in La Clusaz and see what others think.  So we want to create a time machine that will allow us to travel back in time and examine the situation at any point in history!

I was also working on firming up the details of another workshop we’re planning:  this one at CSCL2011.  I think we managed to attract some first-rate data presenters and analysts and together we’re going to try to tackle the problem of linking multiple levels of learning in networked communities.  Planning this workshop certainly has taken a lot of time but I think it is worth the effort.

I continue to act as a mentor to the new programming team that has taken over work on Knowledge Forum.  There are two sides of this task, one of which I enjoy tremendously and one that I dislike immensely.  I enjoy working with talented, energetic students who want to build on previous work.  Christian Perreault is one such student.  He’s doing some great work using MySQL representations of Knowledge Forum databases.  The other side of the work, the one which I don’t like so much, is one where I see much effort being applied in the absence of good design.  Fixing that is beyond the scope of what I can do but I continue to provide input when asked.

Copenhagen I: The Research

As many of you know I have taken up a postdoctoral fellowship at the Copenhagen Business School. Two years ago I could not have imagined being here, yet it is clear that it is a good fit. I’m very lucky to be working on the NEXT-TELL project and we have just completed our first deliverable reports across all the work packages. We are reviewing them internally before we compile them for external review.

My contributions was mainly to Work Package 4, which is concerned with Open Learner Models (OLMs). My main responsibility in the project is to design the communication and negotiation layer for the OLMs and my main contributions was to summarize the results of our Participatory Design workshops and to translate those findings into early Design Specifications. It was challenging, but tremendously rewarding! I also got to contribute to the visualization parts of other work packages.

NEXT-TELL keeps me pretty busy but there’s a bit of time left over to pursue other interests.

This week was a deadline for chapter authors to submit drafts of their work for a book that I am editing with Dan Suthers, Kris Lund, Carolyn Rosé, Nancy Law and Greg Dyke.  The book is the outcome of a series of workshops that we have co-organized around the topic of “productive multivocality in the analysis of collaborative learning”.  We are getting together one more time at the STELLARnet Alpine Rendezvous in La Clusaz, France in a couple of weeks.  After that we’re going to put together a symposium at CSCL 2011 in Hong Kong.  Dan and I are going to conduct another workshop on “Connecting Levels of Learning in Networked Communities” at CSCL 2011.

I also managed to finish up edits to the two papers that represent the output of my time in France.  ”Productive re-use of CSCL data and analytic tools to provide a new perspective on group cohesion” will be presented at CSCL2011 in Hong Kong and |Réutilisation d’un corpus pour une nouvelle analyse des réseaux sociaux grâce à l’adaptation de l’outil KSV” will be presented at EIAH2011 in Belgium.

Those are the collaborative projects from the past.  One of the advantages to being in a new department is the opportunity to take advantage of new opportunities for collaboration.  I am fortunate to be working with a very talented group of researchers at the Center for Applied ICT.  One of the projects that I’m working on with Janni Nielsen, Leif Rasmussen, and Thea Bruun de Neergard involves extending the theoretical framework in the paper they presented at the recent Participatory Innovation Conference in Sonderberg, Denmark by adding a practical component provided by my KSV software.  We hope to submit the resulting paper to the major Information Systems conference!

Another project that I’m involved in with Kim Balle, Mads Bødker, Jan Damsgaard, Nobuko Fujita, Jonas Hedman, Abid Hussain,  Nikhil Srinivasan, and Ravi Vatrapu seeks to understand the use of Social Media in e-Government.  We’re still in the exploratory phase of the work we’re planning, but it was exciting for me to finally get a chance to apply Latent Dirichilet Allocation (LDA) to a corpus built up from more than 11,000 Facebook wall posts.  I got to see how various topics waxed and waned over several months.

Finally, I was reminded that we need to start moving on a book project with the NordForsk Network on Teaching Problem Based Learning in Virtual Environments (ScandLE).

So… lots to do!

France V: The Knowledge Space Visualizer meets Learner Models

I have been working a lot on the Knowledge Space Visualizer to allow us to investigate the relationships between social and latent semantic networks. A key piece in the transition from the old KSV to the new KSV has been the enhancement of author-level analyses. Earlier versions of the KSV tended to focus on the note as the unit of analysis, which was fraught with difficulties. I have expanded the functionality of the KSV in several important ways. First, it no longer relies on Knowledge Forum for data input and storage. Instead, GraphML representations of the data are used to ensure portability. This is a major leap forward, as this has been a limitation of the KSV that has been identified in the literature (see, for example, the proceedings of the most recent JOCAIR conference). Second, the KSV can now create latent semantic representations of participants in the discourse environment. For the case that we’re studying right now, that discourse environment is WebCT (now BlackBoard). In this iteration of the research, the user model is simply the latent semantic vector space representation of all the user’s contributions to the environment. The mathematics of Latent Semantic Analysis make this possible: it’s a matter of scaling the vector representations of each piece of writing appropriately and then just adding the vectors together.

This, in turn, allows me to calculate the relatedness not only between documents, but between the participants based on the documents that they have contributed. An easy future enhancement to this approach will be to also incorporate the documents that an author has read, in addition to the documents that they have contributed. So that means that I can represent the Latent Semantic Network between participants.

It has always been possible to represent the Social Network between participants, although Christophe and I have had good discussions about the relevance of certain measures. For example, he has eloquently made the case the common network measures such as betweenness centrality are virtually meaningless when access to information is not controlled by individuals. In the case of forums (rather than email) it is certainly the case that most individuals have free access to postings (i.e. they are not dependent on other members to send them individual postings). The nature of the social network and its relationship to the intensity of communication was the focus of his 2003 paper.

We are now poised to make meaningful comparisons between the social network, based on who has read whose postings, and the latent semantic network. We can use a variety of techniques to make these comparisons. For example, we can outright compare the networks that are formed using the two techniques. Alternatively, we can examine the network that is formed using one set of relations and overlay the network structure from the other set of relations. For example, the Latent Semantic Network, looks like the following for one subset of forums from the data set (LSA-based links are shown as red lines between authors who are represented as purple triangles):

The Social Network, based on who has read whose postings, can be overlaid (black links indicate more than 50 postings read by both parties) and we can look for interesting cases where the two networks do or do not coincide:

We are going to spend some time thinking about what all this means, with the hopes of generating some publications as a result. What is clear, though, is that we seem to be on to a relatively new type of learner modelling: we are creating a representation of the learner that is the sum of what they have contributed to the discourse space. It will be interesting to see how this fits with other conceptions about learner models, and even more interesting to see how agency over this representation could be turned over to the user. How does this relate to Open Learner Models?

France IV: A Day in the Life

Here is a typical day for me in France:

1. Wake up, pull on jogging clothes and head out for a morning jog around le Jardin de Luxembourg. There are a lot of joggers out first thing in the morning! I typically do two or three laps (3 laps is 5km), depending on how late it is.
2. Stop by the local artisanal boulangerie to pick up one or two sourdough baguettes (or, if it’s a Tuesday or a Friday, perhaps a croissant and a pain au chocolate).



3. Say good morning to Nobuko and Tatiana. Coffee is usually ready, so we sit down for a typically French petit déjeuner of coffee, baguette, butter, jam, and nutella
4. Hop in the shower, get dressed, and head out the the RER station with the family. I get a kiss goodbye and I’m off from Luxembourg to Bagneux. The trip takes about 20 minutes and it’s not crowded. It’s cheap too: 1.80EUR each way. I suppose I could take a vélo, but there are a couple of tricky spots on the way thanks to construction.
5. Get out at Bagneux station and walk down the hill to École Normale Supérieur de Cachan. It’s a pleasant walk.



6. Get to work! I share an office with Christophe Reffay and Françoise Tort.
7. I often take a sandwich with me for lunch. I am informed this is not the proper way to do things in France, and that a leisurely lunch with colleagues is the preferred way. I try to join colleagues from the STEF lab for lunch a couple of times a week but time is so short here that I usually work through lunch.
8. More work! Usually I’m running out of steam, so I hit the Nespresso machine in the salle de réunion for one or two espressos.
9. Leave ENS and head up to Bagneux station. There’s a boulangerie at the RER station where I often pick up a baguette to snack on on the way home, and there’s also a Franprix in case I want to pick up some groceries.
10. Head to Luxembourg station on the RER and then a quick walk home.
11. Dinner is usually ready for me when I get home, although I try to help out a couple of times during the work week. Nobuko is an excellent cook and she works wonders with the limited facilities we have available to us.
12. We often head out for a quick after-dinner walk around the neighbourhood, then we return to our flat, bathe Tatiana, read her some books, and put her to bed. That gives us a couple of hours for adult time, tea, and cookies.

France III: Beginning the Research

It has been a week since we arrived in France. This week saw the beginning of the research project that I am working on while I am here. ENS Cachan is a wonderful campus
but unfortunately the building that houses the STEF lab is currently under renovation, so we’re working in temporary housing. Nevertheless, it’s nice to see your name on the door, and an office is an office.

My hosts, Christophe Reffay and Eric Bruillard were at a workshop at EUROCALL, the European Association for Computer Assisted Language Learning, this past week in Bordeaux so I was left to fend for myself.

My days are spent largely working with Eclipse to extend my Knowledge Space Visualizer (KSV) to handle other types of data. Here’s what my screen usually looks like:

The data from the Simuligne corpus are very rich. They consist of mail, forum, and chat data from higher education. Thanks to a strong background in dealing with XML (thanks, Bob McLean, for supporting an independent study course during my doctorate that allowed me to become familiar with XML processing), I was able to parse (using a java DOM parser) both the reduced and the full data sets from the MULCE site.

After about three days of work, I was able to get some of the first visualizations of the Simuligne data set done that show the relationship between the semantic and the social networks:

I look forward to improving these representations and working with Christophe to refine the sorts of analyses we are doing. I am hopeful that this work will yield something that we can publish in an upcoming special issue of Educational Technology and Society!

This past Friday I delivered a seminar on the work that I did for my doctorate. I was happy with how the talk went, particularly because I delivered the 90-minute talk in French (thanks to Christophe for helping with the translation of the slides).

It is wonderful to be reminded of how enjoyable research can be when one isn’t always in crisis mode.

France II: The first few days

We left Toronto for Paris on Thursday evening. Unfortunately we could not take Snug with us but we are quite confident that he will enjoy the backyard and the boys at the Reeve family residence. After a harried pack and clean, we took a cab to the airport where we arrived comfortably early for our Air France flight to CDG. We were very impressed with the service we received — all the way from the help we got checking in at the automated kiosks, to the baggage check, and even security was pretty darn good.

Once we were through security we grabbed a quick bite to eat — mostly so that Tatiana would have a full belly by the time we were airborne. Our plan worked beautifully: we just finished noshing on pasta and a burger when the flight started boarding. We were fortunate to have seats on the upper deck of the 747… and it turned out we were in the first row of seats just behind the cockpit, which meant plenty of leg room for me. Moreover, no one else was in our row so we had our own private “family row”.

We were pleasantly surprised to learn that not every airline offers service as abominably poor as Air Canada. If we ever get the chance to go to France again it will be a no-brainer: Air France all the way! The cabin crew was polite, accommodating, helpful, and exceptionally courteous. It was the little things that made it wonderful: printed menu cards explaining what was on offer for dinner, offering aperitifs (why yes, we would love some Champagne!), and generally doing everything to make the flight a good experience.

We arrived at Charles de Gaulle airport a few minutes early and we disembarked only to find massive line-ups at immigration. Again, amazing service saw various bits of line-ups re-routed to less busy ones so we made it through in about 20 minutes. Not so bad!

Our host met us just outside the baggage area and drove us to ENS-Cachan where Nobuko and Tatiana were dropped off in an office and I went to a 2-hour meeting where, among other things, I was introduced to the research group. After that meeting we had an informal reception with lots of wine, cheese, bread, and pizza. After that our host drove us to Villa Louis Pasteur, which is where we will be living for the next two months. We were pleasantly surprised to find our flat has a kitchenette — a godsend given that we’re travelling with an infant.

We dropped off our luggage, made a quick list of the essentials that we forgot to bring (e.g. soap), and headed out to a local supermarket (Ed) to pick up necessary supplies. We then returned to the Villa to try to put Tatiana to sleep. It wasn’t easy, and Nobuko certainly deserves a medal for dealing with a 13-month old who thinks she’s still on Toronto time. We eventually all got to sleep.

Of course, getting to sleep so late meant that we overslept the next day, getting up at 11:00 local time. We got cleaned up and headed out to explore le Jardin du Luxembourg. We were completely amazed! We weren’t sure what to expect, but the vast gardens, the Sénat, the family-friendly activities (sandboxes, wading pools, playgrounds, and pony rides) made us feel that this was, indeed, an excellent place for a family to stay in Paris. We wandered a bit further into the 6e arrondisement to find some wonderful sandwiches that we took back to le Jardin du Luxembourg where we ate them al fresco. After lunch we headed north on Boulevard Saint-Michel in search of Monoprix in hopes of picking up some coffee-making equipment. We found it, picked up more supplies for home, and then we headed back to the villa to rest for a while. We decided that we should head out for an afternoon excursion. Because we were still missing some supplies, and worried that most things were closed the next day (Sunday), we headed over to rue Mouffetard to check out Franprix, another supermarket. Little did we know that rue Mouffetard was a most amazing market (and a foodie’s dream). In addition to the requisite groceries, we also managed to pick up a couple of slices of tarte (one with salmon and spinach, and another one with goat cheese, mushrooms, and tomatoes) that served as a late afternoon snack (magnifique!). A couple of still-warm baguettes, a slice of tarte aux pommes and we were ready to head home (one of the baguettes didn’t make it home intact, serving as a snack for the little one).

A simple pasta dish for dinner would have capped off a great day, but then tragedy struck: the newly purchased coffee press fell out of the pantry and shattered onto the floor. After much wiping of glass shards, I was dispatched to acquire a replacement. Monoprix was out of them by the time I got there, but Starbuck’s had some, so all was well.

All of that, and it was only Saturday night!

Sunday started out with an excursion to le Jardin des Plantes. Again, we weren’t sure what to expect, so we were completely amazed by the menagerie (zoo), where Tatiana was completely in awe with all the animals. We grabbed a quick lunch there, explored a bit more until Tatiana fell asleep and then headed back to the Villa. After a quick rest, we decided to head out once again and explore parts south and west of the Villa. We wound up on rue Montparnasse, where most shops were closed but then managed to get onto rue Vavin where we found the most amazing collection of children’s clothing shops. We also discovered that it was very close to home (just across le Jardin du Luxembourg), so Nobuko and Tatiana will be returning soon!

We headed back home, made a quick dinner of chicken curry, and then Tatiana went to sleep… apparently back on a normal schedule of being back in bed by 7pm.

Tomorrow I am headed to ENS-Cachan and so Nobuko and Tatiana will be left to their own devices. I am very much looking forward to starting work on our research project. It’s nice to look forward to doing research again!