SimpleDateFormat sillyness

Ran into an amazingly dumb bug yesterday. I would say that this is clearly a bug in the behavior of SimpleDateFormat in Java. Why is it that when I give it a date that looks valid, and a format string that's not right, and in fact contains invalid numbers, it will go ahead and parse my date string producing a ridiculous result. But not so ridiculous it's obvious, or in fact, just throws an Exception, which for my money would be the desired outcome.

So this is the scenario. Parsing a date like this:

val dateStr = "2013-02-04 05:35:24.693 GMT"

with the date parsing string:

val dateFormat = new SimpleDateFormat("yyyy-MM-dd HH:MM:ss.SSS z")

If you're paying very close attention, you will see the problem here; the month component is repeated. This yields the following date in the resulting Date object: "Tue Nov 03 21:00:24 PST 2015"

This result is clearly very different than what was sent in. I see two problems here. The date string contained two references to the same field. I can see where sometimes this might be useful, but honestly, I feel like you should have to set a flag or something for this to be silently ignored. In most cases having a reference to the same part of a date within a format string seems like a warning flag at least. The second problem is that the erroneous value for month that was give is beyond the scope of a calendar year. You can't have 35 months in a year. In my opinion this should have thrown an exception. I understand that potentially in some calendar system somewhere on this earth there maybe more than 35 'months' in a year or something, but this is very unexpected behavior, way outside of what I would considered normal.

In short, if you have a date string that is being parsed and coming out the other end with a very strange unexpected and wrong result, there's a good chance the format string is off, and probably only very slightly and in a way that's hard to spot without a very very careful eye.

On folding, reducing and mapping

I haven't updated in a while, but it's time to get back into doing this blog.

Today I want to do a brief little thing on folding, reducing and mapping. In the past, I've found myself doing things like this when building a SQL query:

val k = Map("name" -> "John", "address" -> "123 Main St", "city" -> "Los Angeles")

(k.foldLeft("")((a,b) => a + " and " + b._1 + " = ?")).drop(6)

Which on the face of it doesn't seem so bad, might have done something not that disimilar using a for loop in Java.  The thing about this is that it's actually kind of stupid.  There is a more idiomatic way to do this.  When you think about it, the initial operation is actually a matter of mapping the input list to a set of string, and then joining those strings with the and clause.  In Scala, the reduce operation essentially does what join does in the new Java, or other languages, but with any list.  When we think about it this way, we can therefore do this:

val k = Map("name" -> "John", "address" -> "123 Main St", "city" -> "Los Angeles")

k.map(x = > x._1 + " = ?").reduce(_+" and "+_)

or:

k.map(x = > x._1 + " = ?").mkString(" and ")

Ultimately ending up as something like:

val query: PreparedStatement = buildStatement(k.map(x = > x._1 + " = ?").mkString(" and "))
k.zipWithIndex.foreach(x => query.setObject(x._1, x._2))

Much cleaner, and more idiomatic.  Some might say this is obvious, but, it wasn't obvious to me immediately, and so I'm gonna guess it's not to others either!

Database Record Updates with Slick in Scala (and Play)

This is a simple operation that I found absolutely zero reference to in the documentation or the tutorials or the slides.  Eventually after digging through old mailing lists, I came across the solution:

        
(for { m <- MessageQ if m.id === oldMessage.id } yield(m))
  .mutate(r=>(r.row = newMessage))(session)

This is for a simple message class and a function that takes two arguments: oldMessage and newMessage.  The frustrating thing is that this is inconsistent with the simple formula for a single column update:

MessageQ.filter(_.id === 1234L).map(_.subject)
  .update("A new subject")(session)

When you try to apply this thinking to an update, you end up at a dead end.  The mutate operator is also used for deletion too:

MessageQ.filter(_.id === 1234L)
  .mutate(_.delete())(session)

Note that you can typically leave out the session argument as it's declared implicit within the appropriate scope. I'm also switching between syntax alternates because for some reason, either my IDE or the compiler gets grumpy when I try to use the filter() style rather than the list comprehension style in certain contexts. I still have to figure that out.

I'd like to write a longer post later at some point, but this at least covers the highlights.

Collect Chains for Creation - Sublime or Stupid?

I'm working on a piece of code to deserialize Facebook messages, and I'm looking at the mess I wrote to do this, and it occurred to me that I could do it as a progressive collect chain.  Each time we know something new, we collect on it, or pass-thru if it's a false condition.

def makeNewMessage(id: Long)(json: JsValue)(facebookClient: FacebookClient)(session: scala.slick.session.Session): Option[Message] = {
  (((json \ "message").asOpt[String], (json \ "created_time").asOpt[String], (json \ "id").asOpt[String]) match {
    case ((Some(message), Some(createdTime), Some(fbId))) => {
      Some((userId: Long) => (likes: Int) => Message(
        id = id,
        subject = None,
        systemUserId = Some(userId),
        message = Some(message),
        createdTimestamp = Some(new sql.Date(facebookDateParser.parse(createdTime).getTime)),
        facebookId = Some(fbId),
        upVotes = Some(likes)
      ))
    }
  }).collect({ case f => {
    scaffoldFacebookUser(json)(facebookClient)(session) match {
      case Right(scaffoldedId: Int) => f(scaffoldedId)
    }
  }}).collect({ case f => f(((json \ "likes").asOpt[Int], (json \\ "likes").size) match {
    case (Some(likes), 0) => likes
    case (_, likes) => likes
  })})
}

The result of the first piece is a 2-ary function that may get populated with a userId and a like count.  Once we figure out if we can build a user, we populate that, then we figure out the like count.  If at any point the chain fails, it just returns None.

This all feels very imperative to me, perhaps I'm just tired, and it will come to me later. I swear I remember better ways of doing this using unapply magic, but I can't seem to figure it out, so this is where I'm going right now!

Head shaking moments - an ongoing Saga

I think I might start to keep track of examples of bad code that are right out there in the public view.  Some of these examples are the language tutorials themselves even!

Today I'm gonna single out the object example in the CoffeScript guide.  In this guide we get the Animal class example:

class Animal
  constructor: (@name) ->

  move: (meters) ->
    alert @name + " moved #{meters}m."

class Snake extends Animal
  move: ->
    alert "Slithering..."
    super 5

class Horse extends Animal
  move: ->
    alert "Galloping..."
    super 45

sam = new Snake "Sammy the Python"
tom = new Horse "Tommy the Palomino"

sam.move()
tom.move()

This code is in violation of the Call Super code smell.  We finally get half decent classes in Javascript using CoffeeScript, and this is the first example given - one that is considered by some to be an anti-pattern.  Below I'm going to feature an attempt to refactor out this smell.

Updated Code

class Animal
  constructor: (@name) ->

  move: (meters = @distance()) ->
    alert @movingMessage()
    alert @name + " moved #{meters}m."

  movingMessage: -> "Moving..."
  distance: -> 10


class Snake extends Animal
  movingMessage: -> "Slithering..."
  distance: -> 5

class Horse extends Animal
  movingMessage: -> "Galloping..."
  distance: -> 45

sam = new Snake "Sammy the Python"
tom = new Horse "Tommy the Palomino"

sam.move()
tom.move()

Delegate methods are a much better use for an inheritance contract than methods that override a super to provide essentially the same behavior, only with different parameters. I would argue that the second version is substantially clearer as each method does precisely one thing, including the move method of Animal.

Dealing with Annoying JSON

How often do you have to work with an API that supplies badly formatted output? It's fine if you're a weakly typed mushy language like Javascript that doesn't care until it has to, but for those of us who like something a little more structured, and a little more performant, it presents a challenge. I'm gonna look at a way to deal with annoying JSON like this in Scala.  The most recent one I'm running into is a field that may come back as a string, or may come back as a list.

For JSON like this, Jackson provides us with a way to cope with this. The solution doesn't seem to work well with case classes, and seems to require a good deal more annotations that it should, but it does get the job done in a none too egregious way.
Here's a sample of bad JSON:

[{
  "startDate" : "2010-01-01",
  "city" : "Las Vegas",
  "channel": "Alpha"
},{
  "startDate" : "2010-02-01",
  "city": "Tucson",
  "channel": ["Alpha","Beta"]
}]

You can see that in the first element, the field 'channel' is supplied as a string, and in the second, it's now a list.  If you set the type of your field to List[String] in Scala, it will throw an error when deserializing a plain String rather than just converting it to a single element list.  I understand why it's a good idea for deserialization to do this, but really, if you're using JSON, then schema compliance probably isn't at the top of the list of requirements.

You can deal with this using the JsonAnySetter annotation.  Unfortunately, once you use this, it seems all hell breaks loose and you must then use JsonProperty on everything and it's brother.  The method that you defined annotation by JsonAnySetter will accept two arguments that function as a key value pair.  The key and value will be typed appropriately, so the key is always a String, and the value will be whatever type deserialization found most appropriate.  In this case, it will be a String or an java.util.ArrayList.  We can disambiguate these types with a case match construct, which for this seems perfect:

@BeanInfo
class Data(@BeanProperty @JsonProperty("startDate") var startDate: String,
  @BeanProperty @JsonProperty("city") var city: String,
  @BeanProperty @JsonIgnore("channel") var channel: List[String]) {

  // No argument constructor more or less needed for Jackson
  def Data() = this("", "", Nil)

  @JsonAnySetter
  def setter(key: String, value: Any) = {
    key match {
      case "channel" => {
        value match {
          case s: String => { channel = List(s) }
          case l: java.util.ArrayList[String] => {
            channel = Range(0,l.size()).map(l.get(_)).toList
          }
          case _ => { // No-op if you're ignoring it, or exception if not
          }
        }
      }
    }
  }
}

Now when the bad JSON gets passed into deserialization it will get mapped more smartly than it was generated, and we win!

I might have a poke at it to see if I can get it working with less additional annotation crazy too.

Testing with FluentLenium and Scala

I posted some time ago about Browser based testing in Scala using Cucumber leveraging JUnit and Selenium.  That mechanism is pretty complicated, and there seems a much better way of doing it.  The FluentLenium library gives a good way to integrate browser based testing into Scala.  There are still some challenges with continuous integration that have to be solved, and I'll talk about that later.

What does a FluentLenium test case look like with this system?  Here's a simple example that opens a home page and clicks on a link:

class HomePageSpec extends FunSuite with ShouldMatchers {

  test("Visit the links page") {
    withBrowser {
      startAt(baseUrl) then click("a#linksPage") then
        assertTitle("A List of Awesome Links") then testFooter
    }
  }
}

And we can fill in a form and submit it like this:

class RegistrationSpec extends FunSuite with ShouldMatchers {
  val testUser = "ciTestUser"

  test("Creating a fake user account") {
    withBrowser {
      startAt(baseUrl) then click("a#registerMe") then
        formText("#firstName", "test") then
        formText("#lastName", "user") then
        formText("#username", testUser) then
        formText("#email", "test.user@example.com") then
        formText("#password", "123") then
        formText("#verify", "123") then
        hangAround(500) then click("#registerSubmit")
    }
  }
}

Much of what you see above isn't out of the box functionality with FluentLenium.  Scala gives us the power to create simple DSLs to provide very powerful functionality that is easy to read and easy write. People often don't like writing tests, and Scala is a language that is still somewhat obscure.  A DSL like this makes it trivial for any developer, even one who is totally unfamiliar with Scala to construct effective browser-based tests.

Now I'm going to delve into some of the specifics of how this is constructed! (example code can be found at: git://gitorious.org/technology-madness-examples/technology-madness-examples.git)

The first piece is the basic configuration for such a project.  I'm using the play project template to start with as it offers some basic helper functionality that's pretty handy.  The first thing to do is create a bare play project

play create fluentlenium-example

I personally prefer ScalaTest to the built-in test mechanism in play, and the fluentlenium dependencies are needed, so the project's Build.scala gets updated with the following changes:

val appDependencies = Seq(
    "org.scalatest" %% "scalatest" % "1.6.1" % "test",
    "org.fluentlenium" % "fluentlenium-core" % "0.6.0",
    "org.fluentlenium" % "fluentlenium-festassert" % "0.6.0"
)

val main = PlayProject(appName, appVersion, appDependencies, mainLang = JAVA).settings(
// Add your own project settings here
  testOptions in Test := Nil
)

Now for the main test constructs.  A wrapper object is constructed to allow us to chain function calls, and that object is instantiated with the function startAt():
case class BrowserTestWrapper(fl: List[TestBrowser => Unit]) extends Function1[TestBrowser, Unit] {   def apply(browser: TestBrowser) {     fl.foreach(x => x(browser))   }   def then(f: TestBrowser => Unit): BrowserTestWrapper = {     BrowserTestWrapper(fl :+ f)   } }
This object is the container if you will for a list of test predicates that will execute once the test has been constructed.  It is essentially a wrapped list of functions which we can see from the type List[TestBrowser => Unit].  Each test function doesn't have a return value because it's using the test systems built-in assertion system and therefore doesn't return anything useful.  When this object is executed as a function, it simply runs through it's contained list and executed the tests against the browser object that is passed in.

The special sauce here is the then() method.  This method takes in a new function, and builds a new BrowserTestWrapper instance with the currently list plus the new function.  Each piece of the test chain simply creates a new Wrapper object!

Now we add a few helper functions in the companion object:

object BrowserTestWrapper {
  def startAt(url: String): BrowserTestWrapper = {
    BrowserTestWrapper(List({browser => browser.goTo(url)}, hangAround(5000)))
  }

  def hangAround(t: Long)(browser: TestBrowser = null) {
    println("hanging around")
    Thread.sleep(t)
  }


  def click(selector:String, index: Int = 0)(browser:TestBrowser) {
    waitForSelector(selector, browser)
    browser.$(selector).get(index).click()
  }


  def formText(selector: String, text: String)(browser: TestBrowser) {
    waitForSelector(selector, browser)
    browser.$(selector).text(text)
  }

  def waitForSelector(selector: String, browser: TestBrowser) {
    waitFor(3000, NonZeroPredicate(selector))(browser)
  }


  def waitFor(timeout: Long, predicate: WaitPredicate): TestBrowser => Unit = { implicit browser =>
    val startTime = new Date().getTime

    while(!predicate(browser) && new Date().getTime < (startTime + timeout)) {
      hangAround(100)()
    }
  }
}

sealed trait WaitPredicate extends Function[TestBrowser, Boolean] {
}

case class NonZeroPredicate(selector: String) extends WaitPredicate {
  override def apply(browser: TestBrowser) = browser.$(selector).size() !=0
}

This gives us the basic pieces for the test chain itself.  Now we need to define the withBrowser function so that the test chain gets executed:

object WebDriverFactory {
  def withBrowser(t: BrowserTestWrapper) {
    val browser = TestBrowser(getDriver)
    try {
      t(browser)
    }
    catch {
      case e: Exception => {
        browser.takeScreenShot(System.getProperty("user.home")+"/fail-shot-"+("%tF".format(new Date())+".png"))
        throw e
      }
    }
    browser.quit()
  }

  def getDriver = {
      (getDriverFromSimpleName orElse defaultDriver orElse failDriver)(System.getProperty("driverName"))
  }

  def baseUrl = {
    Option[String](System.getProperty("baseUrl")).getOrElse("http://www.mysite.com").reverse.dropWhile(_=='/').reverse + "/"
  }

  val defaultDriver: PartialFunction[String, WebDriver] = {
    case null => internetExplorerDriver
  }

  val failDriver: PartialFunction[String, WebDriver] = { case x = > throw new RuntimeException("Unknown browser driver specified: " +  x) }

  val getDriverFromSimpleName: PartialFunction[String, WebDriver] = {
    case "Firefox" => firefoxDriver
    case "InternetExplorer" => internetExplorerDriver
  }

  def firefoxDriver = new FirefoxDriver()

  def internetExplorerDriver = new InternetExplorerDriver()
}

This gives us just about all the constructs we need to run a browser driven test.  I'll leave the implementation of assertTitle() and some of the other test functions up to the reader.

Once we have this structure, we can run browser tests from our local system, but it doesn't dovetail easily with a Continuous Integration server.  As I write this, my CI of choice doesn't have an SBT plugin, so, I have to go a different route.  Pick your poison as you may, mine is Maven, so I create a Maven pom file for the CI to execute that looks something like this:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.example</groupId>
  <artifactId>fluentlenium-tests</artifactId>
  <version>1.0.0</version>
  <inceptionYear>2012</inceptionYear>
  <packaging>war</packaging>
  <properties>
    <scala.version>2.9.1</scala.version>
  </properties>

  <repositories>
    <repository>
      <id>scala-tools.org</id>
      <name>Scala-Tools Maven2 Repository</name>
      <url>http://scala-tools.org/repo-releases</url>
    </repository>
    <repository>
      <id>typesafe</id>
      <name>typesafe-releases</name>
      <url>http://repo.typesafe.com/typesafe/repo</url>
    </repository>
    <repository>
      <id>codahale</id>
      <name>Codahale Repository</name>
      <url>http://repo.codahale.com</url>
    </repository>
  </repositories>

  <pluginRepositories>
    <pluginRepository>
      <id>scala-tools.org</id>
      <name>Scala-Tools Maven2 Repository</name>
      <url>http://scala-tools.org/repo-releases</url>
    </pluginRepository>
  </pluginRepositories>

  <dependencies>
    <dependency>
      <groupId>org.scala-lang</groupId>
      <artifactId>scala-library</artifactId>
      <version>${scala.version}</version>
    </dependency>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.4</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.scalatest</groupId>
      <artifactId>scalatest_${scala.version}</artifactId>
      <version>1.8</version>
    </dependency>

    <dependency>
      <groupId>org.fluentlenium</groupId>
      <artifactId>fluentlenium-core</artifactId>
      <version>0.7.2</version>
    </dependency>
    <dependency>
      <groupId>org.fluentlenium</groupId>
      <artifactId>fluentlenium-festassert</artifactId>
      <version>0.7.2</version>
    </dependency>
    <dependency>
      <groupId>play</groupId>
      <artifactId>play_${scala.version}</artifactId>
      <version>2.0.3</version>
    </dependency>
    <dependency>
      <groupId>play</groupId>
      <artifactId>play-test_${scala.version}</artifactId>
      <version>2.0.3</version>
    </dependency>
    <dependency>
      <groupId>org.scala-tools.testing</groupId>
      <artifactId>specs_${scala.version}</artifactId>
      <version>1.6.9</version>
    </dependency>
  </dependencies>

  <build>
    <sourceDirectory>app</sourceDirectory>
    <testSourceDirectory>test</testSourceDirectory>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <configuration>
          <source>1.6</source>
          <target>1.6</target>
        </configuration>
      </plugin>
      <plugin>
        <groupId>org.scala-tools</groupId>
        <artifactId>maven-scala-plugin</artifactId>
        <executions>
          <execution>
            <goals>
              <goal>compile</goal>
              <goal>testCompile</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <scalaVersion>${scala.version}</scalaVersion>
          <args>
            <arg>-target:jvm-1.5</arg>
          </args>
        </configuration>
      </plugin>

      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <configuration>
          <argLine>-DdriverName=Firefox</argLine>
          <includes>
            <include>**/*Spec.class</include>
          </includes>
        </configuration>
      </plugin>
    </plugins>
  </build>
  <reporting>
    <plugins>
      <plugin>
        <groupId>org.scala-tools</groupId>
        <artifactId>maven-scala-plugin</artifactId>
        <configuration>
          <scalaVersion>${scala.version}</scalaVersion>
        </configuration>
      </plugin>
    </plugins>
  </reporting>
</project>

You might notice that the above Maven configuration uses JUnit to execute out Spec tests.  This doesn't happen by default, as JUnit doesn't pick up those classes, so we have to add an annotation at the head of the class to signal JUnit to pick up the test:

@RunWith(classOf[JUnitRunner]) class HomePageSpec extends FunSuite with ShouldMatchers { ... }