Saturday, December 31, 2011

Digital bookshelf - no really

As we prepare for our relocation, we've been digging through our substantial book collection attempting to figure out what to keep and what to shed.  Each time I have to remind myself that at this point, most of my book reading is done electronically.  The thought occurred to me that this will lead to the decline of the bookshelf, and therefore the hallowed tradition of inspecting the bookshelves of a host as a means of either distraction or a vague attempt at insight.

I wondered then if it would be possible to create a virtual bookshelf.  Not a kindle type deal with a little set of icons on a tablet, but an actual, virtual bookshelf.  A projection of some kind of a bookshelf, 3' x 6' on a wall that you could see and perhaps even interact with.  The obvious rendition is some kind of LCD screen that is the same size.  TVs are getting so big these days, I'm thinking it wouldn't be much of a stretch to use one as a virtual bookshelf.  I think this is a very narrow vision though.  I think it would be better to do something new or at least different.  I could envision libraries revolutionized with monolith type structures that are virtual bookshelves.  You could browse the shelves of the mystery novels just as you do now, pausing to flick through the pages of a random book whose title looks intriguing.  Just touch a button on the virtual book to add it to your rental queue, and on with the browsing.

Wednesday, December 28, 2011

SBT

If I can ever crack SBT and IntelliJ, I'm gonna post about it, a lot. I just can't seem to get to a point where I can get things working properly with a Scala config. Every time I try, I just get frustrated and give up. The biggest barrier into Scala for me right now is documentation. Critical features like SBT just don't have good docs. I can't seem to run Scalatest Spec tests in IntelliJ, I just end up with weird compiler errors. I can't seem to get the SBT console working in IntelliJ either, and I can't seem to generate an IntelliJ project with either IDEA plugin for SBT, I just end up with weird dependency errors or something else going on. The local Scala users group seems to be barely active. They have one event listed, in June 2012, and are asking people if they are going. It's December 2011, that's six months away, I don't even know if I'm going to be in this state in six months, let alone able to attend some event. They don't seem to have regular meet-ups from what I can gather from their page, which is not a whole lot. I've resorted now to posting questions on Stack overflow, which seems to be turning out pretty well to be honest. People answer quickly and often include lots of helpful information on questions that once I've read the answer seem like they probably made me look pretty dumb. Well, I guess if you don't ask, you'll never know, so I'll take dumb yesterday, smart today over dumb forever. I might separate actual Scala posts with code from the main blog at some point too. Seems like they are confusing each other.

Sunday, December 18, 2011

Scala, encore un fois!

How often do you have complex functionality in a declarative language work the first time out of the gate? Almost never right? Not so much with functional. The number of times a function I've written has worked out of the gate, or virtually of the gate is astounding me. Talk about getting this done faster. The only problem I'm having right now is that I'm still not that familiar with the syntax nuances, so I'm tripping up over that a bit. Really gotta sit down with the user guid and get up to speed on all the little bits and bobs that I'm missing right now. I read through the List class spec, and that was pretty helpful, but there's so much more!

Friday, December 16, 2011

Amazon EC2 - Failing the promise... again

I have a couple of instances on EC2 that I use to host my website, and a bunch of files. They were scheduled for a mandatory reboot for an upgrade. Guess what, they didn't come back up cleanly. Imagine my surprise, except, I wasn't surprised, I was anticipating this. I've worked with EC2 enough to dread a host reboot, mandatory or otherwise. It's a pain in the ass, and a crapshoot if the system will come back up or not. To add insult to injury, a reboot assigns a new IP address. If other systems in your configuration are using that system for something, which has an internal IP address, so unless you have your own BIND set up to deal with that, you're using host entries, the host, if it comes back up, is not at the same address it was before. It's great. Not. You can build a VPC of course, which I have thankfully, so the damage wasn't too bad. I figured out that the "upgrade" had caused my /etc/fstab to be "upgraded" and Linux was attempting to mount my EBS as ext3, but it was actually ext2, so it wasn't mounting. Other providers are providing increasingly compelling options for cloud hosting. The lure of EBS is starting to diminish as other providers are offering bigger disk allocations than they used to, sometimes dynamically. Not only that, but the performance of EBS has been pretty dismal, along with the instances themselves, so I'm not really at big EC2 fan at this point. Current contenders: Rackpace Gandi.net Linode Linode has a good system, but it's instances are bit pricey. Gandi.net has a nice system that more customizable, and the prices are pretty good, but I can't get a good bead on the exact terms of the contract, wether the billing is by hour, or by month, or how it exactly works. Rackspace's cloud offering is really great, BUT, lacks any kind of dynamic disk assignment last I checked. They have a good range of instances to suit all needs, far better than Amazon for small use systems. Just that darn disk allocation issue. I have loads of data acquired over the years, but my website doesn't get much traffic at all, so I don't need a big honking machine. Gandi.net claims to have the underlying disks on RAID 10, but if they are, then they must either be overloading nodes pretty hard, or they just have crappy hardware. I ran a bonnie++ benchmark, and it was downright awful. Yet again, caught between a rock and a hard place on hosting, SO annoying.

Monday, December 12, 2011

Coding, how to put things where they go.

My first rule of programming is "Put Shit Where it Goes".

For many years, I've struggled to effectively quantify this. It has just seemed "obvious" to me very often where shit goes.

I think talking with a remote team over the past couple of weeks has helped my try to think through this a little more.

Give what's to be received, take only what is given

Here's a part I was thinking of today. It's the concept of things that are provided or given, and things that are required or received. You don't get things from an object that requires them, and you don't give things to an object that provides them.

Here's an example:

MyBusinessObject o = new MyBusinessObject();
o.setThingyDao(localThingyDao);

MyOutputFormatter f = new MyOutputFormatter():
f.setThingyDao(o.getThingyDao());
When I see this, I want to reach through my screen and slap someone. I liken it to using an apple to make an Apple pie, then trying to use the apple from the the pie in a glass of cider! You can tell me that this analogy is wrong because it's an object, a thing in memory that multiple places can use, like an apple that exists in parallel universes where it is a pie in one and cider in another. I believe this counter-argument is mistaken. When your apple exists in two different contexts, the results can be unpredictable, even catastrophic when one context changes the apple, or the apple's previous existence is now removed.

If we remove MyBusinessobject from the code above, the assignment to MyOutputFormatter will then fail. My IDE will catch this, as will compilation, but it's a correction that should never have needed to be corrected, and you spend time correcting that syntax error that didn't need to be spent.

The reverse situation is even clearer. You don't give things to a provider. It's like taking an orange out of a bag and putting it on an orange tree.

FruitProvider orangeTree = new FruitProvider();
Fruitrovider appleTree = new FruitProvider();

FruitConsumer bakery = new Bakery();
FruitConsumer caterer = new Caterer();
FruitConsumer footballer = new Footballer();

bakery.setFruit(orangeTree.getFruit());

Then to the astonished onlookers:
appleTree.setFruit(bakery.getFruit());

footballer.setFruit(appleTree.getFruit());


It may seem laughable when written like this, but I see code like this from time to time. It's also an example of why strong typing can save your ass (more on that later I think). If you were writing 'good' code, this would never compile. A "Tree" should simply not have a setter for its fruit. How many of us fire and forget in the IDE though? Just hit "generate code" and not think twice about it? How often does a framework force us to create setters we don't want to? Just like this fruit tree, if we are retrieving an object from a database, the ORM may use a setter to build our object from the database. The setter ends up being public because of it, and now any fool can put data in our object where it shouldn't be permitted.

There are some solutions to this that I've seen, though few good ones. I've seen the setter marked as @Deprecated so that at least the IDE and Compiler give you a big nasty warning if you try and use it, though it's not forcibly prohibited. It does seem like it should be a pretty fundamental feature to have. And in some ways, it is. This is what constructors are often used for. But, in the case of an ORM, a very common need is to initialize data lazily. We can't half create an object all that easily, so, the object gets created, then properties get set later. Almost like an orange tree. We can't have oranges until the tree is grown and the season has come. The problem is that the ORM is acting kind of like god in this scenario, who ends up being personally responsible for putting oranges on the tree. In fact, I think I like this analogy, I'm going to call this kind of code God-Code.

The problem with God-Code, is that it means the worshipers end up rather lacking in willpower and loose the sense to control their own destiny, so any old god can come along and mess with them.

Don't do for yourself what somebody else can do much better
or: don't Let the Plumber fix your Car

This is another instance of the "Put Shit where it Goes" rule. Let's say you want to get a car, make it a blue car, with tinted windows and a really great sound system. The dealership has the car for sale, but it doesn't come off the lot with tinted windows and an upgraded sound system. You don't buy the car from the dealer, then take it to your local plumber to get the windows tinted, or have the librarian put a premium sound system in. You have it done either by the deanship, or by a general mechanic, or by a specialist in that kind of installation.

The same principle applies to helper methods. If you ever intend to put tinted windows in more than one car, don't have your plumber tint the windows. If you have a Service Bean that sends XML, but gets given an object, you don't build XML Serialization into your Service Bean. You're gonna want to serialize to XML more than this one service call, so you make another bean that is responsible for serializing XML, or better yet, use someone else's bean/library.

This is a good example of a separate concern. XML Serialization is largely a concern that is independent from serving up data over HTTP. When you want to test your HTTP server, you don't want to end up having to test XML Serialization as part of that test. Vice-versa, you don't want to test HTTP functionality from your XML Serializer. You might ask, why not? It's simple math. If you have an object t that does two things, f and g which may be combined, potentially tightly coupled, you have to write test for the following for a given test fixture x:
t.f(x)
t.g(x)
t.f(g(x))
t.g(f(x))

!t.f(x)
!t.g(x)
!t.f(g(x))
!t.g(f(x))

Don't forget to write those test cases that demonstrate the functionality fails when it's supposed to!

If we have separated concerns that are decoupled in two objects a and b, we end up writing only:
a.f(x)
!a.f(x)

b.g(x)
!b.g(x)
We've now tested our system with the same efficacy with four tests instead of eight. Now work out the situation when you have an object that does three things, or four things. It's exponentially more work to test as you get more tightly coupled concerns within an object.

In additional thinking, you can sometimes fix the car yourself.  Save all that money on labour, spent it on something fun right?  If you're not a mechanism, think again.  The mechanic has the right tools to do the job.  He has contacts for getting the materials, and this is his job, he's good at it and can get it done more quickly than you (if he's any good).  You might be able to buy an off-label turbo cheaper, which is great, until you break off some fitting when you try to install it and have to buy another.  Or until you realize you need some special set of wrenches that costs twice as much as the turbo itself.  Then you end up spending a month getting it installed instead of getting a mechanic who could have done it in an afternoon.  You end up spending four times as much doing it yourself than it would have cost to buy brand-name and have the mechanic install it.

Don't reinvent the wheel is the usual metaphor.  If somebody else has built an XML serialization library, check it out first.  Be sure it can't meet your needs before you dive into writing your own.

Save yourself a permanent headache, and put shit where it goes!

Of course, if your goal is purely fun, or you need XML serialized in a special magical kind of way, you roll your own.  If you're nice, you make it free software so we can all have a magical XML serializer.  If everyone writes their own XML serializers, the world only ends up with a whole bunch of esoteric serializers that aren't any good to anyone but the people who made them, and they are probably going to end up wishing they'd spent their time building code that did something truly unique and useful.

I'm gonna cover tangling and scattering at some point I guess too.

Tuesday, December 6, 2011

Data Payload Facet Pattern

I have no idea if this already exists, or if there's a better way, but here's a little thing I came up with yesterday.  I'm calling it the data payload facet pattern. I guess it's pretty similar to the Façade pattern, but a bit more specific.

What is this pattern?  It's a way of creating what is essentially an Object Oriented version of a SQL view.

If you have a generic payload object that has a Map<String,Object> concept, then anything you set or get from this Map is, to a point unsafe.  You can use the class argument improvement at least on getters, but it's still pretty weak.

Enter the Payload Facet Object.
 
public class DataPayload {
  properties = new HashMap<String,Object>();
  private Object payload;

  public <T> T getProperty(String name, Class<? extends T> c) {
    if (!c.containsKey(name) || c.isInstance(properties.get(name))) {
      return properties.get(name);
    }
    throw new IllegalArgumentException("The property "+name+" has type "+properties.get(name).getClass().getName()+" but you requested an object of type "+c.getName());
  }

  public DataPayload setProperty(String name, Object value) {
    properties.put(name, value); // not worrying about replacing here
    return this;
  }
}

So we have the pretty awful property object that is horribly error prone, particularly with String names.

We layer a Facet Object on top of this:

public class UserEntryPayloadFacet {
  public static final String USERNAME="Username";
  public static final String PASSWORD="Password";
  public static final String EMAIL="Email";

  private DataPayload proxied;

  public String getUsername() {
    return super.getProperty(UserEntryPayloadFacet.Username, String.class);
  }

  public UserEntryPayloadFacet setUsername(String username) {
    super.setProperty(UserEntryPayloadFacet.Username, username);
    return this;
  }

  public String getPassword() {
    return super.getProperty(UserEntryPayloadFacet.Password, String.class);
  }

  public UserEntryPayloadFacet setPassword(String password) {
    super.setProperty(UserEntryPayloadFacet.Password, password); // of course - passwords are never in plain text, right?
    return this;
  }
  
  ....
  // If you feel like it...
  public DataPayload unwrap() {
    return proxied;
  }
}

You get the general idea.  In my implementation I've defined an interface to include an unwrap() method call to get at the payload, and in my case, I'm also exposing the underlying data object properties via proxy method calls, so the facet would have a getProperty(...) call and a setProperty(...) call.  Your mileage may vary.

Sunday, December 4, 2011

Scala vs Groovy

I've been working recently on a project based in Scala and just started to use SBT for the build tool. A significant project I have, my accounting system (G'Kané), is written in Grails.

As I'm flipping back and forth right now, a few things are sticking out.  I'll probably update this as I go along, but so far:

I really like the ? operator in Groovy, getThing()?.doSomething(), only runs doSomething() if getThing() returns a non-null value.  The only way I know how to do this right now in Scala is via a match statement, possibly using the Option[] construct, which is really clunky in comparison:

getThing() match {
  case Some(p) => p.doSomething()
  case _ => null
}

On the plus side for Scala is case classes.  The same thing that causes the above travesty is also something to be rejoiced in.  I know I can pass a map to a Groovy constructor:

new Thing([name : "myThing", value: 4])

but it's completely type unsafe.  In comparison, a Scala case class has a default constructor with the properties:

new Thing("myThing", 4)

Granted, it does mean the order has to be correct, which is a bit of a pain, but you do get type sanity.

In the black-eye department for Groovy is the bewildering mix of Groovy method declarations and Java style one, and in-between.

def thing = { "something here" }

but if I want it typed, suddenly it's back to almost Java:

def String thing() { "something here" }

Having untyped methods in Scala seems pretty near impossible though, so I suppose it's not quite fair, but:

def thing : String = { "something here" }

in and of itself seems closer to the syntax that would seem like it should be idiomatic for Groovy, though it's actually Scala. This debacle leaves us wondering if we should be calling Bar.thing, or Bar.thing() in Groovy, and it might change, even though the function of the method didn't.

The Grails compilation on the fly feature, whilst nice, at least in theory, is a source of major pain for an old hacker like me who has the Save key-press burned into the brain as a function to be performed after every single change made in a file.  The poor on-the-fly compilation just gets really angry because I frequently save a file multiple times per minute, and also when it's far from working.  SBT has a similar feature, but it's not coupled to my app in progress (at least not yet).

On traits and interfaces: I'm really loving traits in Scala.  They are fun and a neat way to implement cross-cutting concerns in a way that's not quite as all-powerful as Aspects, but yet still code-visible to a mere mortal.  Interfaces in Groovy, I honestly still don't understand how to work that.  I've looked it up a few times, but it either won't stick, or just hasn't made sense to me yet.  For instance, I have a set of services which all do similar things that I would normally group in an Interface in Java.  They are responsible for creating a Mapping description object.  They have a single method columnMap() which returns a ColumnMap object.  I'd create an interface called ColumnMapper or something in Java to describe this contract.  In Groovy, I can create an abstract class, but where does it live?  In my Grails view in IntelliJ, it seems a bit unclear.  Do I make it a service that's just declared abstract?  What happens if I try to bind that abstract class?  If it were Java I'd get a compiler error, I'm guessing I'd get a runtime error in Groovy, and I'm not sure if my IDE is smart enough to figure it out and show it as an error.

Speaking of IDEs, the Scala plugin for IntelliJ 11 is improved, but still has a ways to go.  It gets confused sometimes about a project, and will only let me create scripts, which then turn into the appropriate icon once I create the code.  Bit weird.  Still, given the very weak type binding in Groovy, which leaves the poor IDE guessing more often than not, I'm currently leaning to Scala on the matter of code inspection, at least in IntelliJ.

Saturday, December 3, 2011

OMG <3 Scala

Today was a good day. I wrote a somewhat complex piece of logic for a jump planner for EVE online, and, because of the awesomeness of Scala, it worked as conceived, first time.

This is one of the many reasons I love functional programming. As you take your problem and reduce it down to it's most basic components, strip out logic, simplify to a mathematical expression, simplify that expression, suddenly errors become clear in the very writing of it.

Scala feels so much closer to that expression than Java. So much verbosity is inherently prone to error, writing error free code in Scala seems dramatically more possible than in Java.

SBT and Gradle

Liistening to a not quite cent episode of Java Posse podcast, I heard them talk about two new build tools that are in the running to succeed Maven and Apache Ivy. They are SBT and Gradle respectively.

As with the rest of the Java community who are moving on, one is Scala based and the other is Groovy based.

I'm not going to get into a Scala vs Groovy thing here, as i can't say I know either of them to a high enough level to really make a fair comparison.

I've had a poke at both of them, and so far, I'm really diggin the simplicity of SBT and it's interactive features for working with Scala, which are super nice.

I've only got a small example going with my EVE project, and it so small it's barely worth posting. Having said that, it did clue me in to working with scalatest, which is much more elegant than JUnit. I'm curious to use it also with JUnit and java code, I've heard that it can work well with both Scala sources and Java. The interactive test suite features seem like a really awesome feature, continuous integratinon built right in.

Friday, November 25, 2011

Building a Scala Restful web service

Given how little documentation I've seen out there that is anywhere near close to comprehensive, I figured I'd document some of my trials and tribulations learning Scala, and working in the Scala world.  I am adding the very large disclaimer here that as I am just learning things, I am by no means a Scala expert, and take nothing here as perfect, or even good.  Feel free to admonish my techniques in the comments if you wish, I'll try to post corrections where I can.

As many who know me know, I play EVE Online.  It's a big MMO which has a significant API and database dump available for players to work with.  I've been working on and off over the years on tools to work with the EVE universe through the data and the API.  I'm going to use this platform as my first experiment with Scala.

I have the Programming in Scala book, I bought it some time ago, and it languished on my shelf for a long time.  I finally picked it up and started working with it, but I found it severely wanting.  When I start a new language like this, I want working examples that do stuff.  I want to see how language concepts work in practice, not in examples so clinical they are more or less worthless.  To that end, I started writing code based on what I had read, and what I could google.

I've picked the O/R Broker database API and Jersey to run my Rest service.  I remember reading about a Rest server that Twitter used somewhere, but I can't seem to find it now, so I'm going with something else that's pretty well known.

O/R Broker is a bit verbose, but I like how it uses real SQL and case classes to achieve a pretty effective ORMish style system.

What follows is a simple example of serving up two services: a solar system information service, and a route service that shows a path between two solar systems through jump gates.  I'm guessing if you're reading this, you can figure out the database structure easily enough, and I'll leave acquiring the EVE database as an exercise for the reader if you really want to do it (though I'll be happy to answer questions on that if anyone cares).

Starting with the SQL and moving upward, using ORBroker you create plain text files with the SQL in them, map them with a Token object then use those to make read calls.  I'm using Maven as my build tools, so directory naming conventions follow Maven conventions for the most part, I think I started with a Scala archetype, but I don't fully recall.

First, I'm designing the SQL to return the values I want to use from the appropriate tables.  To start with, I'm going to retrieve only basic information about a solar system:

select
  a.solar_system_id,
  a.solar_system,
  a.x,
  a.y,
  a.z,
  a.security
from
  solar_system a
where
  a.solar_system = :solarSystemName


I defined the model class for our SolarSystem object.  It's created as a case class as that is what ORBroker is expecting, and it has a number of benefits including public members, but many others that I don't fully understand yet to be honest.

src/main/scala/com/plexq/eve/model/SolarSystem.scala:

case class SolarSystem (id: Option[Long],
                        name: String,
                        x: Double,
                        y: Double,
                        z: Double,
                        security: Double)

The next piece is mapping the SQL and model classes to extractor objects.  These objects are responsible for turning the results of the SQL into Scala objects.  In the first service, we have only a simple type, so we use the RowExtractor super class:

src/main/scala/com/plexq/eve/db/SolarSystemExtractor.scala:

object SolarSystemExtractor extends RowExtractor[SolarSystem] {
  def extract(row: Row) = {
    new SolarSystem(
      row.bigInt("solar_system_id"),
      row.string("solar_system").get,
      row.decimal("x").get.doubleValue(),
      row.decimal("y").get.doubleValue(),
      row.decimal("z").get.doubleValue(),
      row.decimal("security").get.doubleValue()
    )
  }
}

The row data is mapped into a constructor call for the model object, but because a SQL query can legitimately return a null for a column, the type of a row field is an Option type.  The Option type in Scala is a case class used to distinguish explicitly between null values and actual values.  It allows nulls to be type safe for one thing.  The get method on an Option object retrieves the actual object, assuming there is one (I don't know what happens when there isn't, an exception I'd guess).

Now we want to perform the database operation in the context of a Rest call, which using Jersey is pretty easy:

@Path("/solarSystem")
class SolarSystemService(@QueryParam("name") name: String) {
  @Produces(Array("text/plain"))
  @GET
  def getSystem = {
    val broker = DatabaseContainer.getBroker
    broker.readOnly() { session =>
      session.selectOne(Tokens.selectSolarSystem, "solarSystemName"->name).get match {
        case Some(p) => {
            "Solar System:" + p.name + "\n" +
            "Security:" + p.security +"\n" +
            "x:" + p.x + "\n" +
            "y:" + p.y + "\n" +
            "z:" + p.z + "\n"
        }
        case _ => "Failed to find solar system "+name
      }
    }
  }
}

I'm just sending back text for the time being so that I can easily read if the output is correct.  I've found that XML or JSON is surprisingly tricky in Scala as of yet, and the mechanisms I've tried didn't work out of the box or as designed/described.

We also need to create the object to contain our database information, modify to your local environment as usual:

object DatabaseContainer {
  val ds = new PGSimpleDataSource();
  ds.setServerName("localhost")
  ds.setUser("eve")
  ds.setPassword("xxxx")
  ds.setDatabaseName("eve")
  val builder = new BrokerBuilder(ds)
  FileSystemRegistrant(new java.io.File("src/main/sql/orbroker")).register(builder)
  builder.verify(Tokens.idSet)
  val broker = builder.build()

  def getBroker = broker
}

The final piece to the puzzle is a web.xml that initializes the Jersey servlet, and I'm using Jetty as a container here because I can't be arsed to get Glassfish sorted out, and I like Jetty:

src/main/webapp/WEB-INF/web.xml:

We can now run this web service and pull back information on a solar system!

I'm going to throw some stuff up here about getting JSON out of this whole thing, but it seems a subject that is unclear at best with my current knowledge, at least in a concise way and using Jersey.  Lift seems overly complex for what I want, so I want to do it outside of that framework.  There is some support in Jackson for Scala, but it doesn't seem to work quite right, at least the way I have it configured.

To get it going, I've changed web.xml, SolarSystem.scala and SolarSystemService.scala.  I added the POJO option to the Jersey container:

I created a Jackson mapper in my Service object and updated the output to use that instead of my String concatenation:

@Path("/solarSystem")
class SolarSystemService(@QueryParam("name") name: String) {
  @Produces(Array(MediaType.APPLICATION_JSON))
  @GET
  def getSystem = {
    val mapper = new ObjectMapper()
    mapper.registerModule(DefaultScalaModule)

    val broker = DatabaseContainer.getBroker
    broker.readOnly() { session =>
      session.selectOne(Tokens.selectSolarSystem, "solarSystemName"->name) match {
        case Some(p) => {
          mapper.writeValueAsString(p);
        }
        case _ => "Failed to find solar system "+name
      }
    }
  }
}

And for the most perplexing part, I updated the SolarSystem object to define explicit getters.  This is a bit odd because Jackson is supposed to cope with public fields, but it's not working for me.  I've read some things about version incompatibilities, so maybe that's it, but I'm going with what I have so far:

case class SolarSystem (id: Option[Long],
                        name: String,
                        x: Double,
                        y: Double,
                        z: Double,
                        security: Double) {
  def getName = name
  def getId = id
  def getSecurity = security
  def getX = x
  def getY = y
  def getZ = z
}

Now when I run the service and ask for information on Jita, I get the following:

{"id":1358,"name":"Jita","x":-1.29064861734878E17,"y":6.07553069099636E16,"z":-1.1746922706009E17,"security":0.945913116664839}

Much easier to digest.

In the next service, I want to build a route between two solar systems, kind of a travel plan.  To do this I'm going to need to retrieve a list of systems that a given system leads to, a list of destinations.  Constructing the SQL for this is a little more interesting, but not particularly challenging:

src/main/sql/orbroker/selectDestinations.sql:

select
  a.solar_system_id,
  a.solar_system,
  a.x,
  a.y,
  a.z,
  a.security,
  d.solar_system_id as destination_id,
  d.solar_system as destination_name
from
  solar_system a,
  stargate b,
  stargate c,
  solar_system d
where
  b.solar_system_id=a.solar_system_id
  and b.destination_id=c.stargate_id
  and c.solar_system_id=d.solar_system_id
  and a.solar_system = :solarSystemName

As you can see above, we're performing a join, so we need to use two of the extractor types provided by ORBroker, a RowExtractor and a JoinExtractor.  The information we are retrieving here is a one-to-many relationship between a solar system and its destinations.  A RowExtractor is responsible for the most frequent output information, the data from the join that is unique on each row and represents the child objects, which in this case is the destination solar systems.  The extractor we already have for SolarSystem is find for that.  The low frequency information, which is the parent object, is the source solar system.  The source solar system is therefore extracted using the JoinExtractor.  The JoinExtractor needs to know what field the identity for the parent record is so that it can separate objects that belong to the parent, and those that belong to the child.  The identity column is provided by overriding the 'key' property.  All rows that share this identity column are assumed to be a single parent object.  All rows within that set that have different entries are mapped as children of that parent object.

src/main/scala/com/plexq/eve/db/SolarSystemDestinationExtractor.scala:

object SolarSystemDestinationExtractor extends JoinExtractor[SolarSystemDestination] {
  val key = Set("solar_system_id")

  def extract(row: Row, join: Join) = {
    new SolarSystemDestination(
      new SolarSystem(
        row.bigInt("solar_system_id"),
        row.string("solar_system").get,
        row.decimal("x").get.doubleValue(),
        row.decimal("y").get.doubleValue(),
        row.decimal("z").get.doubleValue()
      ),
      join.extractSeq(SolarSystemExtractor, Map("solar_system_id"->"destination_id", "solar_system" -> "destination_name"))
    )
  }
}


Given the data structure and the constructors above, we can define the new model class for SolarSystemDestinations:

src/main/scala/com/plexq/eve/model/SolarSystemDestination.scala:

case class SolarSystemDestination(solarSystem : SolarSystem,
                                  destination : IndexedSeq[SolarSystem])

Now we have enough code to store the result of a SQL query that retrieves information about solar system destinations.  We need some code to read it and turn it into a route.  I'm building a simple b-tree style object here that contains a route-in-progress:

src/main/scala/com/plexq/eve/map/RouteTree.scala:

class RouteTree(solarSystem: SolarSystem) {
  var nodes : List[RouteTree] = List[RouteTree]()

  def contains(v: SolarSystem) : Boolean = (v.name == solarSystem.name) || nodes.exists {_.contains(v)}

  def leaves() : List[RouteTree] = {
    nodes.length match {
      case 0 => List(this)
      case _ => nodes.flatMap {x=>x.leaves()}
    }
  }

  def getSolarSystem : SolarSystem = solarSystem
  def setNodes(n : List[RouteTree]) : RouteTree = {
    nodes = n
    this
  }

  def path(end: SolarSystem, filter: (SolarSystem) => (Double)) : List[SolarSystem] = {
    if (contains(end)) {
      nodes.length match {
        case 0 => List(solarSystem)
        case _ => List((solarSystem, filter(solarSystem))) ::: nodes.find {_.contains(end)}.get.path(end, filter)
      }
    }
    else List()
  }

  def count : Int = (nodes.length/:nodes)(_+_.count)
}

and a builder object to construct a route:

class RouteBuilder {
  def buildRoute(broker: Broker, route: RouteTree, end: SolarSystem) : List[SolarSystem] = {
    var s = route.count

    route.leaves().foreach { x : RouteTree =>
      x.setNodes(SolarSystemDataService.getSolarSystemDestinations(broker, x.getSolarSystem.name).filterNot {
        route.contains(_)
      }.map {
        new RouteTree(_)
      }.toList)
    }

    /* Bug out if the list didn't get any bigger */
    if (route.count == s) {
      return List()
    }

    route.contains(end) match {
      case false => buildRoute(broker, route, end)
      case _ => {
        route.path(end)
      }
    }
  }
}

Now we can create our service class:

@Path("/route")
class RouteService(@QueryParam("start") start: String,
                   @QueryParam("end") end: String
                  ) {
    @Produces(Array(MediaType.APPLICATION_JSON))
    @GET
    def getRoute = {
      val mapper = new ObjectMapper()
      mapper.registerModule(DefaultScalaModule)

      val broker = DatabaseContainer.getBroker
      var error : scala.collection.mutable.ListBuffer[String] = ListBuffer()

      val routeTree = broker.readOnly() { session =>
        session.selectOne(Tokens.selectSolarSystem, "solarSystemName"->start).get match {
          case p : SolarSystem => new RouteTree(p)
          case _ => {
            error+=("Failed to find Start System "+start)
            null
          }
        }
      }

      val endSystem : SolarSystem = broker.readOnly() { session =>
        session.selectOne(Tokens.selectSolarSystem, "solarSystemName"->end).get match {
          case p : SolarSystem => p
          case _ => {
            error+=("Failed to find End System "+end)
            null
          }
        }
      }

      error.length match {
        case 0 => {
          mapper.writeValueAsString(new RouteBuilder().buildRoute(broker, routeTree, endSystem))
        }
        case _ => {
          ("[\""/:error)(_+"\",\""+_)+"]"
        }
      }
    }
}

In our service class we check to make sure the start and end systems exist, and I'm thinking there has to be a better way than this to do it, but this works.  The main difference here is the mapper class now registers the DefaultScalaModule.  This is provided by the jackson-scala dependency, and will cope with Scala classes that the default Java bindings don't, like List() objects, which is what we get in this case.  Now when we ask for the route from Jita to Amarr, we get back a nice JSON list:


[{"id":1358,"name":"Jita","x":-1.29064861734878E17,"y":6.07553069099636E16,"z":-1.1746922706009E17,"security":0.945913116664839},
{"id":1360,"name":"Perimeter","x":-1.29064861734878E17,"y":6.07553069099636E16,"z":-1.1746922706009E17,"security":0.945913116664839},
{"id":1355,"name":"Urlen","x":-1.43265233088943008E17,"y":6.4923714928938896E16,"z":-1.04178623206742E17,"security":0.953123230586721},
{"id":4028,"name":"Sirppala","x":-1.39376796022883008E17,"y":7.1476647043998E16,"z":-9.9524016578104608E16,"security":0.959995210823471},
{"id":4025,"name":"Inaro","x":-1.37550934148756E17,"y":7.8077592063385904E16,"z":-8.6193987480218304E16,"security":0.88322702239899},
{"id":4026,"name":"Kaaputenen","x":-1.3575371976577E17,"y":7.79504770996252E16,"z":-8.2362867465824608E16,"security":0.836977572149063},
{"id":4750,"name":"Niarja","x":-1.38143247136544992E17,"y":6.6032260761458E16,"z":-7.5317306241481296E16,"security":0.779168558516838},
{"id":4749,"name":"Madirmilire","x":-1.84441638429595008E17,"y":4.9352410074477104E16,"z":2.47548529253837E16,"security":0.541991606217488},
{"id":4736,"name":"Ashab","x":-1.86411855995097984E17,"y":5.1383654517254496E16,"z":2.95630167990767E16,"security":0.603228207163472},
{"id":3412,"name":"Amarr","x":-1.95782999035935008E17,"y":5.4527294158362E16,"z":5.51292598732268E16,"security":0.909459990985168}]



After this, I start to descend into madness around providing a filter mechanism to provide routes only in high-sec etc, and a service for capital jump planning, but that's another story!


I'm not a Scala expert by far, this is pretty much my first Scala app, so suggestions and comments welcome.  I hope this was useful.  I can post Maven deps if that's useful, but largely it's just Scala, Jackson, ORBroker and Jersey, I grabbed the latest versions of each using mvnrepository.com.

Thursday, November 24, 2011

Functional Programming


I've been working in the US as a programmer for over a decade, and during that time, I've worked with a huge variety of engineers and hackers alike.

If you asked most people what the difference was, I think many would have a hard time quantifying it well.  My mind goes back to my first semester at Southampton University and our functional programming course, which was taught in SML.  The simplest problem, how to reverse a list was a challenge in functional programming that many couldn't wrap their heads around at first.  The idea of calling a function recursively to perform what in a declarative language was a loop was quite foreign, even to those of us who had done quite a significant amount of programming in our youth.

Here's an example of a solution in Scala (yes, I know there's a built in call for this, it's an example okay?):

def myReverse(x : List[String]) : List[String] = (List()/:x)((x:List[String],y:String) => List(y) ::: x)


Some folks might write something like this in Java:

public List<T> reverse(List<T> input) {
  List<T> l = new ArrayList<T>();
  for (int a = 0; a<input.size(); a++) {
    l.add(input.get(input.size()-a-1));
  }
  return l;
}

Or maybe even:

public void reverse(List<T> input) {
  for (int a = 0; a<(input.size()/2); a++) {
    T x = input.get(a);
    input.set(a,input.get(input.size()-a-1));
    input.set(input.size()-a-1, x);
  }
}

modify the list in place, what a good idea eh?  Sometimes, but mostly not so much, and many can't tell you why.  (And yes, there are slightly more elegant ways in pure Java, but few use them).  As we add complexity to this problem, the code gets increasingly worse.

As we add other features that are really great if you know how to use them, simple things like list comprehensions and similar, the code starts getting more obscure looking, but better.

And there lies the problem.  As we right more "efficient" code, it requires more and more skill to manipulate.  At what point do you draw the line and decide you've gone from good code to obscure code?

I have a few simple rules to write code by, the first one is "put shit where it goes".  It's a simple expression for avoiding tangling and scattering, a concept I learned about much later.  Some mechanisms fundamentally break this rule.  One such mechanism is Aspect Oriented Programming.  It is a very very powerful tool, and in the right hands, amazing.  The problem is is that it is probably the very definition of obscure.  Without a good toolset, you'd have no idea that the code even existed or was being executed against one of your classes, particularly if it didn't show up in a stack trace because the problem was being precipitated as a knock-on effect.  It could take a long time, possible forever to track down a problem.

Functional programming is less so, and I think keeps to the "put shit where it goes" rule much better.  With things like list comprehensions and closures, you can write shorter code that goes directly where it should.  Having more collections immutable makes things more reliable as you're less likely to get a modification vs copy problem.  I'm not sure of the performance cost of doing it this way though, but I get the feeling that of all performance problems in software development today, copy versus modify-in-place is not the worst of them by far.  I think it makes code more readable and clearer.  I think it can significantly reduce development time as well, at least if you don't strive to write "perfect" code.

Thursday, November 17, 2011

Scala FTW!

I've been working a bit with Groovy and Grails lately, and I'm finding them pretty awesome.  I figured I'd take it to the next level, and dive into Scala.

Scala has more powerful functional features than Groovy, but it seems to have kind of spotty framework support.  This turns out not to be such a bad thing.  The folks using Scala seem to care about making good solutions to things, and looking at GORM in Grails, it works fine, but it's really slow, and suffers the same problems as Hibernate.  Database access in Scala isn't so locked down yet.  There a few systems out there, so I picked on that looked interesting: ORBroker and ran with it.

My goal here was really just to get something working that could read data from a database and do something with it.

As per usual, I do testing with my EVE database, so I decided to start with something that could read a solar system and list the destinations.  Not too bad once I figured out how the ORBroker mapping worked, which took a bit of doing.  The documentation is a bit sparse, but with a bit of general ORM experience and a few years under the belt of development, I got it going.

I'm gonna put some code in here, then talk a bit more about it, and some of the fun with ORBroker:

object App extends Application {
  val ds = new PGSimpleDataSource();
  ...
  val builder = new BrokerBuilder(ds)
  FileSystemRegistrant(new java.io.File("src/main/sql/orbroker")).register(builder)
  val broker = builder.build()

  val selectDestinations = Token('selectDestinations, SolarSystemDestinationExtractor)

  val jitaDestinations = broker.readOnly() { session =>
    session.selectOne(selectDestinations, "solarSystemName"->"Jita")
  }

  def lf(session: QuerySession, y: List[SolarSystem]): List[SolarSystem] = {
    if (y.isEmpty)
      List()
    else {
      session.selectOne(selectDestinations, "solarSystemName"->y.head.name) match {
        case Some(p) => lf(session, y.tail).filterNot {p.destination.toList.contains _} ::: p.destination.toList
        case None => List()
      }
    }
  }

  broker.readOnly() { session =>
    lf(session, jitaDestinations match {
      case Some(p) => p.destination.toList
      case None => List()
    }).foreach(y => println(y.name))
  }
}

The code uses a SQL query defined in selectDestinations.sql to pull back a list of destination solar systems given a solar system.  This isn't quite as easy as it might first appear because in my incarnation of the EVE database, solar systems aren't joined directly, but via a Stargate relation.  Because ORBroker doesn't do ORM, it maps queries to objects via the extractors, this is much saner than it would be with an ORM, even if it is a bit more verbose.

We grab a list of destinations from the initial query, which returns a list of SolarSystem objects.  Then we send that list through the lf closure to retrieve the destinations for those destinations.  We are getting the list of systems reachable by making two jumps from the start point - which rather typically is Jita.  The cool thing that becomes easy with a functional language is the list construction piece.

It's all very nice to implement list iteration by recursion, it's first year computer science stuff in the UK, but I still love it, and, because of list comprehensions and filters, we can go one better with only a tiny bit more code; we can filter the list and eliminate systems that have already been added, or more accurately that are going to be added.  Now we're talking something vaguely useful.

I haven't tried the ternary operator yet in Scala, not sure if it exists, but if it did, the closure would be one line long, if it wasn't for the case situation.

This is one place where I greatly prefer Groovy's solution to the problem of null results, maybe I'm missing something Scala can do here because I only just cracked the book yesterday, but so far, this is kinda annoying.  To separate null returns from valid returns, we have to disambiguate an Option[] group.  There seem to be a couple of ways to do this, but some of then aren't appropriate here, and well, this worked.  The match/case operator pairs do a match on the Option group, and allow us to do something in both null and non-null situations without an ugly if operator, and matching can do a whole lot more, but for this situation, it seems a bit overkill.  The Groovy suffix ? does as much with less fuss.

We filter the return to we only concatenate elements in the return list that don't exist in the destination list, and because we are applying it recursively, it always filters to the latest concatenation, and we magically get the right result.  List comprehensions FTW.

The interesting _ parameter kind of feels a bit like Perl's $_, so I kinda have to hate it.  Honestly though, it beats Groovy's it parameter which is just kinda lame.

The ORBroker is supposed to be able to build Token objects through the Tokens thingie, but I haven't got it to work yet.

Tuesday, November 8, 2011

Hooking up an SSD to an older Mac without tearing it apart

I'm posting about my attempts to hook up an SSD to my 2009 iMac, which doesn't have a thunderbolt port.

It does however have a firewire 800 port.  In theory, I should be able to get a pretty good transfer rate over this, but we'll see!

I got my shiny SSD in the mail a few days ago, I got a Corsair Force 120GB drive which has pretty good number on the anandtech review.  I bought an external drive chassis on Amazon that will hold two drive in one of RAID 0, RAID 1 or JBOD.  The drive enclosure product blurb claims transfer speeds up to 200MB/sec, which should exceed this SSD's write rate, but we'll see.  I haven't researched the signaling speed of firewire 800, so I don't know if that is even physically possible currently.  (Update: DUH, it's called Firewire 800, because it's 800Mbit/sec).



The first problem was the the external enclosure only had mounting rails for a 3.5" internal drive, and the SSD is 2.5".  Not a big setback, adapter kits are pretty easy to come-by and I found a pretty nifty one on Amazon that wasn't too expensive (picture to come when I get it).

Until that arrives, I have hooked up the SSD through two other methods that are bit of a hack.  I have a plethora of external hard drives that sit on a shelf next to me.  Most are Western Digital MyBooks, a couple aren't.  I got a Seagate Freeagent awhile ago as it was the first 3TB drive available, and I was desperately in need of a bigger drive for my backups.  The thing crapped out when I tried to restore and only some quality time with the Mac drive utility tool allowed me to access it and get my data off!  The good part is that the enclosure has available an adapter for firewire 800.  Over the years I've also bought a couple of Western Digital drives that have Firewire ports.  First thing to do was to pull apart the enclosure and remote the electronics that have the firewire interface.  The Wester Digital required some heavy convincing with a screw driver to come apart, but I got the guts out well enough.  The Seagate Freeagent was very easy.  The base on the model I have detaches, and I initially though I would have to pull the enclosure apart to reveal the adapter from the base to SATA, but it turns out that the base's adapter is itself just a SATA port! (so much for paying attention).  You can take the base and plug the drive in without further ado.

The Seagate Freeagent base from the side.  You can see the Firewire 800 port and the power connector

The Seagate Freeagent base from the back.  You can see the addition firewire port and a USB port.  The main connector is visible here and if you're paying attention (unlike me), you can see it's a standard SATA style connector.

This Western Digital MyBook enclosure.  The enclosure comes apart by forcing the back loose, then sliding the middle bit out.  The middle bit didn't slide out without some serious convincing with a screw driver to break loose the sides.  I'm not sure if they were bonded or not, but they did give a reassuring pop sound when they came loose!

The back of the Western Digital enclosure.  This enclosure has holes where the two Firewire 800 ports are, a hole for an eSATA connection and a USB port.  Many of these enclosures only have a USB port, and I had purchased this with the knowledge that I wanted a firewire port available.  It turned out to be a bit of a non-issue in the long-run, but it sure came in handy for this!

The Seagate Freeagent enclosure.  The enclosure comes apart from one side only, which is the side opposite the connector.  The casing pops apart, though in doing so I snapped a few of the clips on the inside.  Don't bet on being able to put either of these back together afterwards!

The bottom of the Seagate enclosure.  You can see here the hole where the drive port is exposed.  There is no adapter that lives here!  The base plugs directly into the drive with clips for the main part of the enclosure itself.

This is the SSD docked onto the Seagate base.  It's a bit precarious in this position, but it makes a good picture!  The drive docks directly onto the base, and the connection isn't great because the base is expecting a gap between the bottom of the drive and the base where the enclosure housing would normally go.


Here is the df from the drive.  If you can see, it's showing about 116GB usable and 12.6GB used (give or take).

This is the guts of the Western Digital enclosure.  You can see the SATA adapter on the left.  You can plug in the drive to this, and it almost rests on the rest of the case.  Slightly less precarious, but also less fast!!

The Seagate Freeagent base.


I did some benchmarks with no other apps running on my iMac, and the results were surprising.

The Seagate base did pretty well, clocking in around 60-70MB/sec both read and write with seeks in the 10,000 range.  Not too bad, about the same speed as a regular disk running at full speed, with much better seek performance.  The Western Digital interface was a bit disappointing, clocking in at only 30MB/sec or so.  The interface has a firewire 800 port, but I suspect it's not running at full speed.

I formatted the drive once I had it back on the Seagate base and mounting a PostgreSQL tablespace.  I've created a fairly large database on there, about 12GB, so it gives a fair amount of data to play with.

More to come...

Saturday, October 29, 2011

Facebook Graph

I've been meaning to get around to hooking up Facebook to something so I can mess with it a bit. Turns out there is a grails plugin:

http://www.grails.org/plugin/facebook-graph

This had me up and running with Facebook in about 30 minutes, maybe less. I now have myself logged in to Facebook on my app and it pulling my friends list and basic profile information.

Scary, and cool.

Tuesday, October 18, 2011

Where is the internet I grew up with?

Following a short post on g+, and a short rant on Facebook, I decided to fill out the idea a bit more.

Where is the internet we grew up with?

For those of us in our 30s, the internet was fresh and shiny in the late nineties (for some, earlier). Altavista was a search engine of choice, eBay and Amazon were in their infancy. Hotmail was just getting serious and Facebook had probably not really even been thought of. Like many others, I had a geocities page, and life seemed good. It was the new digital frontier. Rough and ready, available to be shaped by anyone who could come along and take ahold of the new emerging technologies.

Before long, eBay and Amazon become the power-houses they are today. Many other internet ideas came and went, most of them because they just weren't all that compelling, some because of bad business management.

Here we stand, looking into 2012. What is the internet today? Who's hot, and who's not?

Facebook, Twitter, Flickr and Tubmlr, today's engines of "Social Media". eBay and Amazon, what feel like two of the last vanguard of the old ways. Amazon always had a great business case: sell stuff cheap through economies of scale, great shipping and a superior customer experience. People can buy anything they want in a few clicks (1-click maybe), and have it at their door in hours. Gone are the days of 28 days for shipping. Instant gratification for just about any kind of thing you can want (more or less). eBay, once a giant auction site is pretty far down the slope of decline. They are now charging such high fees, and their site has been encumbered with so much commercial content that regular Joe auctions are buried amongst the dross. Wierdly, they feel like they are trying to be Amazon, except that Amazon is already Amazon, and does a darn good job of it. In my opinion, unless eBay wises up fast, they are going to end up holding a bag of stuff with nowhere left to go. People using eBay are complaining more and more, and shifting ways of selling things to other venues like Craig's list. At this point unfortunately, there doesn't seem to be much else in public view that presents as an alternative to eBay. It is hard to create a vacuum in such a large market, but eBay is pulling the air out of what used to be their core business model. If that vacuum gets to a critical level, a whole slew of new blood will enter the market, and eBay's dominance will shatter under the pressure their vacuum has created leaving them with what exactly? A not-quite-as-good-as-amazon for sale site?

Flickr seems to be doing okay, but it's photo presence have been significantly eroded by Facebook itself and other sites like Picassa and for serious photographers, pay sites like SmugMug who do a much much better job than Flickr with content management, presentation and semi-pro features.

For many, Facebook has become the internet. Twitter is an add on, and Tubmlr a fascination.

I don't know how things are at Twitter financially, I'm not up on that so much, but I hope things are holding up for them. It's a neat medium, but I'm not sure many people understand why. I think it might be that 140 character limit. It's a nice short tweet. Not intrusive, something you can glance at without being completely distracted from whatever else you are doing. It's the essence of tl;dr. Perfection for a world with ADD. I hope they don't cook what I believe to be their golden goose.

Facebook is in a precarious position. Sitting atop the chaos of the modern age. The hate for Facebook is pretty big, and whilst Google plus impressed some people, there hasn't been the mass exodus that many had hoped for, or predicted. Google screwed up the name situation. I feel they had good reasons, but ultimately, it shot them in the foot. I know that many of the technorati, the people who could make or break it, are the same people who are idealists, and who care about those kind of details. These are the people who still remember the way Facebook has betrayed them year over year. The name situation made them feel that Google was incompetent and didn't "get it". With them deserting in droves, Google Plus may never be more than a sideshow.

I have to say that so far, I haven't really embraced Tumblr. My daughter has a Tumblr, and so do a few other people I know. I forget if I have created one or not.

What does the next decade of the internet look like? What kinds of things will shape it? Some say the API revolution is a big deal. I'm not so sure. APIs require skill to use, and the number of students taking computer science courses at universities has been dropping off. The software industry was one place the 99% were promised a bright future, and honestly, it's still one place they might be able to get it. The API trend I worry is something being driven by the big players. The corporately dominated players. Those that can afford to do API things without compromising their core business model.

What is going on to combat this movement towards utter corporate dominance of the internet of the next decade? I think there is still room for a few garage-built systems to make a hit. People aren't taking a leaf out of Apple's book. Make new technologies, create new revenue streams, future-proof your company by always living in the future. Until they do, the bright and the young can always be a step ahead. The question is if their one-step ahead is really one-step ahead, or one step sideways.

There are some innovations in the software space that are helping with this, things like Ruby on Rails and it's software cousins of a similar ilk. These systems are making it possible for the garage bands to make their dreams happen quicker and easier. I'm concerned that some of this 'innovation' is more like a step sideways than a step forward. People are exchanging tried and tested methodologies for bleeding edge systems relying on "the cloud" to build their ideas into reality. This wouldn't be so bad, but the big promise of the cloud, on demand scalability, is an empty promise for DIY programmers. To achieve real scalability takes knowledge and experience, especially within that environment. For Joe average, this is not present.

The young and the bright can be a step ahead, or a step sideways, but with current technology, hitting a glass ceiling happens too fast. With so many internet users, it takes only one tsunami to hit the shores of your fledgling system to bring it down, where too often it will stay down with no path to recovery. What used to be a quick incremental increase in traffic is today a massive exponential rush. A site can go "viral" in 24 hours or less, and go from a few thousand users to a million people crushing the system. Without experience, it's pretty hard for Joe average to deal with that. The bar to enter the Cyclone that is today's internet has become very high indeed.

How do we solve this problem? I think for some that is the $64,000 question. How do you scale with such rapidity? Systems like MongoDB are being pushed as potential answers. They work in the cloud (more or less), and offer what seems like a good proposition. Unfortunately, they can't do 80% of what a traditional SQL database can do, and once you start growing, that 80% becomes really important. Things like reporting and data warehousing that can help you understand your website users who are now your customers. Other technologies like Map-Reduce are impressive, but most folks don't understand the real value-proposition for Map-Reduce, and worse, most Joe average programmers aren't fluent in the ways of functional programming. Heck, I know very few CompSci grads who can put together programs in a functional style after a decade in the field working declarative systems in Java and others. Lisp and SML and similar are thought on with a level of slight revulsion, a bad college memory of what seems like an arcane way to do things.

So what now? I don't yet know, and that's half the fun! It's undiscovered territory, and no-one really knows what will happen next.

Friday, October 14, 2011

iOS 5 - 24 hours later

I've had iOS 5 in my hands on my iPhone for just shy of 24 hours.

What's it like?

Well, it's nice. The biggest things I've appreciated so far are the new notification mechanism, the reminder app and the upgraded bluetooth support. It also feels like the keyboard is more responsive and I get less key misses typing on it. It almost feels like it's slightly predictive, but it could just be my imagination.

The new twitter integration seems pretty nifty, I might use that, I did on my old phone here and there.

The upgraded bluetooth support allows my car stereo to display track information and repeat status on it's LCD screen. It's a nice add that allows me to see what random track is playing now my iPhone decided to grab a huge random chunk of my library when I last synced, somewhat to my surprise, and a bit to my dismay. I probably should have checked the settings a bit more carefully!

The notification mechanism is a really good upgrade. The pull down is easy to use, and with weather right there, I think they made it excellent. Stock ticker is probably nice for some, but a bit gimmicky for most I'd imagine.

The reminder app is something I was looking forward too, and it did not disappoint. I set up a reminder to tell me to go pick up packing supplies when I had dropped my daughter off at school. It was easy to set up a location based notification, figured it out in a few moments. The one thing that I couldn't see was how to trigger a notification based on an arbitrary address. It seemed like you could only do it based on addresses in your contact list. That was a bit annoying. Had to add the local school to my list to make it work.

The news-stand app looks like it could be cool, but the two publications I looked at had a 'free' news-stand presence, but required a subscription to get any good content. I don't mind paying a few bucks up front, but an ongoing subscription for more than the print edition per month, that's asking a bit much.

Instant on camera is awesome, but my current case somehow managed to get built without a camera hole. I did have another case, and although it had a camera hole, it lacked any other good features save the colour, so I switched back. Once I get a better case, I'll probably use it often.

One big thing that still feels missing, or it's there and I haven't figured it out, is the ability to suck in a phone number or an address from Safari. If I highlight an address or phone number, It should give me the ability to create a new contact. It does give me the ability to call the number, but if it can do that, why can't I create a contact from that location?

I'm not sure, but it also felt like it was chowing down on my battery a bit much, though that could be more because I was using the wifi hotspot for a bit last night, and that always takes a pretty good toll.

I didn't notice yet, but in iOS 4, switching between messaging and phone was pretty annoying. I hope they made that a bit smoother. The number of times I want to call someone who I've just been messaging is pretty high, and I don't know why it's not easier to do so. Either that or I'm just missing it.

iPhone 4S? Well, if I was in the market for a new phone, there's no question I'd get it. But at $700 new, I think I'll have to wait for iPhone 5, or until my contract lets me get a new phone as part of the deal. I'd get it just for the upgraded camera to be honest, but for $700, that would have to be a damn fancy camera.

Overall, iOS 5 has not disappointed. Features are easy to see, intuitive, no manual required. Apple has again delivered us a piece of technology that is beautiful and functional. Features we can use, and aren't so hard to work with, we get frustrated ten seconds after pulling out the phone. When I think back to my LG touch-screen phone, the iPhone is a world apart. It costs me three times more per month, but I wouldn't switch back unless I had no other choice.

Wednesday, October 12, 2011

On Steve Jobs

I'm writing this more as a meta-post than a post, in and of itself.

I could wax lyrical about the way Jobs and Apple has impacted my life, but it would be the same as thousands of other stories, and therein lies some of what I'm interested in talking about.

A number of criticisms have been leveled at Jobs, and Apple. Some of the favorites are the walled-garden haters, the overpriced gadget shouters, the Jobs-is-an-evil-capitalist hecklers and the what did Steve Jobs ever do for me crowd, who need to go watch some Monty Python and think about things a little more carefully.

Apple's walled garden, or so it's labelled is a remarkable feat of FUD. The garden that is wide open, that you can enter, or leave at any time you please. Buy an iPhone, if you hate it, then buy something else. This walled garden that includes Netflix, and vimeo, most major TV networks, internet radio, and even free software. Your walled garden is run on an open source operating system, and an open source browser. Your walled garden can export files to Excel or Word and that can browse any website. Look at your own garden, be it Microsoft or Linux. You are more walled in than those in the Apple garden! IE, constantly lagging in standards, Linux applications that can't or won't important foreign file formats properly, Windows that has no app store, no way for you to share your idea with the world easily, Linux, that requires amazingly arcane knowledge to operate on any long-term basis successfully.

Apple's overpriced gadget shouters are apparently missing a fundamental principle of the free market: things are only worth what people will pay for them. If Apple's gadgets were truly overpriced, then very few would buy them. As it happens, Apple's phones still make up a very significant portion of the market, despite making no more than two models at any given time with one Operating System variant. Two models vs the hundreds of devices practically given away by every network provider out there. Two vs 100s. I can't imagine how anyone can label this is anything short of amazing. This is Thermopylae all over again, except that as of today, the Spartans are winning, and aren't looking like being shaken any time soon. Despite giving consumers less choice and a more expensive device with fewer carriers, Apple's product is wildly successful. This doesn't seem like the mark of an overpriced gadget. We could run the same comparisons on the rest of the Apple product line. Walk through an airport sometime and observe the number of folks carrying a MacBook of some kind. How many other 11" devices are there that come close to the MacBook air for flexibility, power and usability for a business traveler?

Steve Jobs has been both applauded and demonized by the Wall Street capitalists, and mostly for the same things. He doesn't not create product to drive profit. Every model we have today for building stuff say you should increase profits by providing the widest appeal at the lowest cost. That you don't bring out new products that undermine your own markets. That your R&D funding should shrink relative to your overall cost structure. Steve Jobs broke every single one of these rules. If Steve Jobs is a rampant capitalist, then we have to seriously reconsider our notion of capitalism as it stands today.

As for the "what did Steve Jobs ever do for us" crowd, I refer you to the Monty Python sketch from the life of Brian which had a similar theme about the Romans, and more ironically perhaps, the inane arguing amongst the factions of freedom fighters over such petty things as their names. Apple has driven personal technology innovation for a decade. I don't know precisely which things they invented themselves per se, but they did as least identify products that had the potential to be game-changers, and bring them to market more quickly and with more elegance than anyone else. The early iPods with their touch-dial interface and remarkably tiny hard drives. The iMac that has shown again and again the power of a computer you can purchase, set on your desk, plug in a power cable, and increasingly fewer others, turn on and start doing cool stuff out of the box. The bundled software, the usability, these things have still not been achieved by any other vendors. If they were easy to do, then why aren't both Windows and Ubuntu doing better? They've had almost a decade to consider the situation, and are still failing miserably. The Apple TV, which has a level of integration and simplicity that made this kind of device possible, and usable. The courage to put IPS displays in all its devices, something that is still lacking on other manufacturers, and one reason that Apple devices have been considered overpriced gadgets. Steve Jobs gave you a screen that was capable of displaying your photos and videos as they were captured, without awfully over saturating them and driving up contrast, but just as they are. I'm constantly astounded at how much I have to exaggerate features in my photographs for those will 'normal' screens.

Walk into a modern office building. Listen for awhile. Sooner or later you'll here what happens when a major meeting approaches. The synchronous chime of the iPhones all indicating to their owners that a meeting is approaching is almost eery. Many of them never having had their defaults changed provides an audible witness to the impact that Steve Jobs and Apple have on our lives today. The very point of Apple's walled garden is to get the minutiae out of your way. Let you do what you love to do, assuming it's not theming your window manager... again.

I've heard the 'Macs are for creative people, not for office or development'. In 2011 this angle is ridiculous. Out of the box, the Mac comes with development tools for several major languages, including Java, Python and Ruby. With an Xcode download, you get the full suite of compilers and tools if you really really want to build C, C++ or Objective-C applications for a fraction of the cost of Visual Studio. Apple's numbers and Pages are much easier to use than Microsoft Office, and their tablet counterparts are a demonstration of what is possible on a tablet once you rethink and realize that there is no longer a viable pointing device. The amount of time I've had to sit and troubleshoot a fellow developer's Windows environment because something just stopped working is horrible. It feels like many of them have opted to get a Mac out of embarrassment as much as anything, and didn't live to regret it.

You can't walk down the street, ride the tube or sit in an office without seeing their devices, to deny the impact that Steve Jobs and Apple have had on technology just seems like it can only be the vantage of a blind man.

I don't love Apple, I don't worship Steve Jobs, but I'm not blind either. I have four Apple devices within reach as I write this, my high-end PC laptop gathering dust in a corner somewhere, largely forgotten.

Saturday, September 3, 2011

To 4G or not to 4G

This is an easy question to answer: Not.

Why? Simple math. Right now, a Verizon plan costs $80/mo for 10GB for a 4G Mifi. Current speed tests show you can get 13Mb/s. 10GB worth of bandwidth at about 1.3MB/s goes in about 2 hours.

2 hours.

You pay $80/month for 2 hours of full speed 4G service.

10GB is about 5 HD movies give or take.

So not so much.

Friday, August 26, 2011

On User Experience and Engagement

You have a website that sells stuff. You want people to buy more stuff. How do you make that happen?

I've seen a few ideas on this, things like:

Better User Experience
Easier Transactions
Compelling Pricing
Good recommendations

Two companies have solved this problem pretty well: Amazon and their somewhat recent acquisition, Zappos.

What is their answer: Customer Focus.

And in my opinion that wraps up a general principle. Your customer has to trust you or have an emotional connection with you. If they don't they will jump ship at the first opportunity. They have no brand loyalty, because they don't trust in the brand, and that's only if you have a strong brand to start with.

I have a general principle: I'll buy from a more trusted source at up to a 15% higher price.

A key word in there is "more". If it came off the back of a lorry, I think I'll pass regardless.

This is where a company like Zappos has hit the jackpot. I trust Zappos so much, I'll pay upwards of 50% more than somewhere else. I'm a bit shy when I'm working in a domain I'm not an expert in, and I find dealing with retail staff a bit trying, so I'd rather just order it from a website that can tell me five times as much as a retail person can. The conventional downside? Returns, and the ability to try something on.

Enter Zappos. Free returns. 100% Free? Not exactly. I'm guessing the reason I pay a bit of a premium for their products is because of this. But you know what, I'm glad for it. I don't have time to get to a brick and mortar store often, and normally, they only have a few dozen items that I'm interested in. I don't want to deal with retail staff who know a fraction of what I'd like to know, and I'd like to have a chance to see what other people think about a product.

Some of the bullets above are captured in the concept of trust, but some aren't. Low prices are compelling, but convenience and trust is far more compelling. Variety is also very compelling, but finding things I want is also compelling, so recommendations are very important, and also, very good search functionality.

In my opinion, Search Functionality is something Amazon and Zappos have yet to fully master. They do a far better job than others, but ultimately, I still struggle to find what I'm looking for sometimes.

Why is that? One reason I've noticed is that there are often terms of art in clothing that I'm not familiar with. I can't pick from a search facet that uses words I don't understand. There is no link to explain what they mean most times, so I have no way to find out, short of Googling each term, and that starts to chip away at the convenience factor pretty quickly. Another reason is the choice of facets. I'm guessing they've done their research and they show the best overall facets. I know I'm not a typical user, but I am a user that spends a lot of money online, and I rarely get the facets I want. Price bands are rarely where I would draw the lines, and feature facets are often not the features I care about. I imagine I'd have a better shot at completing my impulse buy if I could find what my impulse desired more easily!!

Windows 8 - iPad Killer, PC Killer, or just another fad

Ars Technica has an interesting article about the potential market and purview of Windows 8:

http://arstechnica.com/microsoft/news/2011/08/a-sort-of-pc-how-windows-8-will-invade-tablets-and-why-it-might-work.ars?comments=1&p=21997821#comment-21997821

I can't help but feel this article is rather missing the point of a tablet, and also the limitations of technology at present.

Use Case: I want to edit a complex Excel document for a presentation.
Requirements: Highly portable device, long battery life, fully functional keyboard, pointing device, sophisticated user interface, rapid software response rate

Can a tablet meet this use case. No. And here's why:

Slow CPU (comparatively)
Limited Storage
No pointing Device
No Keyboard

Let's look at the last two, as it's reasonable to challenge those. No pointing device, and no keyboard. You can have a keyboard and pointing device connected via bluetooth. But why would you want to? Even Apple's super slim keyboard design isn't all that compact, and a mouse isn't very compact either, being rather fat. Now you need a bag to carry these accessories and this solution is looking considerably inferior to a light portable laptop.

The best solution to this use case is a device like the MacBook Air. Why?

Portable: check
Long Battery: check (relative to other laptops at least)
Pointing device: check
Full Keyboard: check
Sophisticated UI: check
Rapid software Response: check

Economically, if you're doing this, you probably don't have an entry level iPad. MacBook Air starts at $1000, and iPads top out at $850? The Air has dramatically more power, storage, screen real estate. Better keyboard, trackpad, and isn't all that much heavier.
The cost difference just isn't that big given the functional difference, and if you're a business user with this requirement, I'm guessing even a $500 difference is bordering on meaningless.

If this is really the functionality you want. Buy a laptop, not a tablet.

Add on the other use cases that this kind of user has, and the tablet isn't really a contender at all. How often would a tablet form factor be better than a laptop for this user. Hardly ever. This user doesn't want a tablet that's also a laptop, they want a laptop that's occasionally a tablet. They have these if that's what you really want, they're laptops with touch screens.

One device simply cannot rule them all. That's life. There's a reason there are 60 kinds of Spaghetti sauce at the grocery store. Different people like different things. The really sad thing to me, is that there are 60, not 20, which would satisfy the market, and leave room for different kinds of items beyond spaghetti sauce that would also appeal to a customer.