Scala and Selenium

Update: I'm now using fluentlenium instead of this mechanim with the play framework.  It's much better!

I've been working with Scala for awhile, and I've just dipped my toe into Selenium.  Selenium is a web testing tool that allows you to drive a browser like Firefox, to do more detailed testing of web pages.  Coupled with Cucumber, doing BDD based testing is pretty cool.

Setting this up in Java is a fair bit of work, and I managed to get it going with a little help from the internet.  Then I wondered what it would look like in Scala.  He below lies a configuration for settings up Selenium testing using Cucumber in Scala.

You can find all the pieces for this out there, but I hope having them all in one happy place will save a few folks some time! I could have used sbt, but chances are, if you're using sbt, you can easily figure out how to convert this pom. I use maven here because it's a common denominator (perhaps the lowest, but still), you can import it easily into IntelliJ or Eclipse or Netbeans.

As per usual, I'm no Scala expert, but in my project this works though there may be more idiomatic ways of solving some of the things in here. Please please comment if you have thoughts or ideas on this. The pom file:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.example</groupId>
    <artifactId>selenium-tests-scala</artifactId>
    <version>1.0.0</version>
    <inceptionYear>2012</inceptionYear>
    <packaging>jar</packaging>
    <properties>
        <scala.version>2.9.1</scala.version>
    </properties>

    <repositories>
        <repository>
            <id>scala-tools.org</id>
            <name>Scala-Tools Maven2 Repository</name>
            <url>http://scala-tools.org/repo-releases</url>
        </repository>
        <repository>
            <id>sonatype-snapshots</id>
            <url>https://oss.sonatype.org/content/repositories/snapshots</url>
        </repository>
    </repositories>

    <pluginRepositories>
        <pluginRepository>
            <id>scala-tools.org</id>
            <name>Scala-Tools Maven2 Repository</name>
            <url>http://scala-tools.org/repo-releases</url>
        </pluginRepository>
    </pluginRepositories>

    <dependencies>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
        </dependency>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-compiler</artifactId>
            <version>${scala.version}</version>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.10</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.specs</groupId>
            <artifactId>specs</artifactId>
            <version>1.2.5</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.seleniumhq.selenium</groupId>
            <artifactId>selenium-java</artifactId>
            <version>2.17.0</version>
        </dependency>
        <dependency>
            <groupId>info.cukes</groupId>
            <artifactId>cucumber-junit</artifactId>
            <version>1.0.0.RC16</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>info.cukes</groupId>
            <artifactId>cucumber-scala</artifactId>
            <version>1.0.0.RC16</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <sourceDirectory>src/main/scala</sourceDirectory>
        <testSourceDirectory>src/test/scala</testSourceDirectory>
        <plugins>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <executions>
                    <execution>
                        <goals>
                            <goal>compile</goal>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <scalaVersion>${scala.version}</scalaVersion>
                    <args>
                        <arg>-target:jvm-1.5</arg>
                    </args>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-eclipse-plugin</artifactId>
                <configuration>
                    <downloadSources>true</downloadSources>
                    <buildcommands>
                        <buildcommand>ch.epfl.lamp.sdt.core.scalabuilder</buildcommand>
                    </buildcommands>
                    <additionalProjectnatures>
                        <projectnature>ch.epfl.lamp.sdt.core.scalanature</projectnature>
                    </additionalProjectnatures>
                    <classpathContainers>
                        <classpathContainer>org.eclipse.jdt.launching.JRE_CONTAINER</classpathContainer>
                        <classpathContainer>ch.epfl.lamp.sdt.launching.SCALA_CONTAINER</classpathContainer>
                    </classpathContainers>
                </configuration>
            </plugin>

        </plugins>
    </build>
    <reporting>
        <plugins>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <configuration>
                    <scalaVersion>${scala.version}</scalaVersion>
                </configuration>
            </plugin>
        </plugins>
    </reporting>
</project>

The features file is used to speak BDD language. This file is processed by Cucumber and matched against the step definitions file.

Feature: The home page should show useful content

Scenario: Load the home page

     Given I want to see my home page
     Then I should see valid content

This class functions essentially as a marker to JUnit to execute Cucumber to build the test instead of scanning this class for methods directly. You'd think there'd be an easier way to do this without having to have something that amounts to no more than a stub here.

package com.example.selenium.cucumber

import cucumber.junit.Cucumber
import cucumber.junit.Feature
import org.junit.runner.RunWith

/**
 * Test definition class that indicates this test is to be run using Cucumber and with the given feature definition.
 * The feature definition is normally put in src/main/resources
 */
@RunWith(classOf[Cucumber])
@Feature("Home.feature")
class HomePageTest {
}

The step definitions file is used by Cucumber to match the features text agains. There are several basic predicates in Cucumber like "When" and "Then" and "And" that you can use via the DSL. Cucumber supports many languages in this regard so you're not limited to English for this.

package com.example.selenium.cucumber

import org.openqa.selenium.WebDriver
import junit.framework.Assert.assertEquals
import junit.framework.Assert.assertTrue
import com.example.selenium.pageobject.HomePage
import com.example.selenium.WebDriverFactory
import cucumber.runtime.{EN, ScalaDsl}

/**
 * Step definitions for the Home Page behaviours
 */
class HomeStepDefinitions extends ScalaDsl with EN {
    Before() {
        driver = new FirefoxDriver()
    }

    After() {
        driver.close()
    }

    Given("^I want to see my home page$") {
        home = new HomePage(driver)
    }

    Then("^I should see valid content$") {
        var actualHeadLine: String = home.getTitle
        assertEquals(actualHeadLine, "My Awesome Home Page!")
        assertTrue("Page content should be longer than this", home.getContent.length > 4096)
    }

    private var driver: WebDriver = null
    private var home: HomePage = null
}

The HomePage object here is used to store information about the home page, and potentially actions that you can perform whilst on the HomePage which I've omitted here for simplicity. In my working system, I make most of this live in a base class that all my page classes extend. Things like the ability to screen shot are useful everywhere, as is reading the title and page content. Some generic tests that can simply match the title and content for a whole bunch of pages can be very useful for bootstrapping the test process. They might not be good test, but they're definitely better than nothing!

package com.example.selenium.pageobject

import org.openqa.selenium.By
import org.openqa.selenium.WebDriver
import org.openqa.selenium.WebElement

/**
 * A page object used to represent the home page and associated functionality.
 */
class HomePage(driver: WebDriver, baseUrl: String) {

    private var driver: WebDriver = null
    private var baseUrl: String = null

    def getDriver: WebDriver = driver

    protected def takeScreenShot(e: RuntimeException, fileName: String): Throwable = {
        FileUtils.copyFile(
            (driver.asInstanceOf[TakesScreenshot]).getScreenshotAs(OutputType.FILE),
            new File(fileName + ".png")
        )
        e
    }

    def getContent: String = getDriver.getPageSource

    def getTitle: String = getDriver.findElement(By.tagName("title")).getText
}

I hope I haven't left anything critical out that would be required to get this system bootstrapped. Obviously this set up doesn't do very much, but, it will at least get you up and running. You can probably figure out the rest from the Cucumber and Selenium documentation at this point. I might post a follow up with some more detail, but it took me a few weeks to get this posted!

Hosting

I've been looking at server hosting for a decade, and I'm a little perplexed at this point.  For many years with a managed hosting environment, we'd sign up for 12 months, then at the end of the year, we'd migrate to a new environment.  The new environment was typically the same cost as the old one, but somewhere between 1.5x and 2x times as powerful with about the same again in disk space.

Then the cloud came along.  I've been using Rackspace's cloud solution, Amazon EC2, Linode and a few others over the last few years, and I'm not seeing the capability of the server time you spend money on increasing at anywhere near the same rate as previously.  Whilst CPUs have risen in core count, and memory capacity has gone up, my cloud server still costs about the same as it did three years ago and still has the same capability as it did three years ago.  Amazon et al have had some small price decrements, but nothing close to the doubling every year we used to see.

I think this is perhaps one very big drawback for cloud computing that many people didn't bet on.  My guess is that once a cloud infrastructure is created, the same systems just sit in their racks happily chugging away.  There is only minor incentive to upgrade these systems as their usage is on-demand and they aren't being actively provisioned like traditional server hardware was.  You can no longer easily compare and contrast hosting options because it's complicated now.  The weird thing is that this situation seems to have infected regular hosting also!  I am in the process of trying to reallocate my hosting to reduce my costs, and it seems everywhere I turn it's the same story.  Looking at Serverbeach, their systems have barely shifted upwards, other than CPU model in five years.  my $100/mo still buys a dual core system with about the same RAM and disk as it did five years ago, albeit the CPU is a newer model.

For those of us developing on heavier platforms, like Grails or JEE, the memory requirements of our applications are increasing, and the memory on traditional servers and our developer machines is increasing in step, but cloud computing resources are not.  I simply cannot run a swath of Grails applications on a small EC2 instance, the memory capacity just isn't big enough.  My desktop has 8GB of RAM today, and it won't be long before it has 16GB, yet my Amazon instance is stuck at 1.7GB.  Looking at shared hosting, the story is the same.  Tomcat hosting can be had quite cheaply, if you don't mind only have 32-64MB of heap.  You couldn't even start a grails app in that space, it's recommended that PermGen is set to 192MB at least.

The story isn't universally the same I've noticed, some hosting providers have increased the disk capacity availability quite dramatically, but with regard to RAM, the story seems pretty consistent.  You just don't get more than you used to.

What does this mean for the little guy like me, trying to host applications in a cost effective way?  At this point I really don't know.  I'm starting to consider investigating co-location at this point; talk about dialing the clock back a decade.  I can throw together a headless machine for less than $400 which has 10x the capacity of a small instance, seems kinda sad.  Right now, I'm considering shifting away from rich server architectures and refocusing on a more Web 2.0 approach, which brings a small shudder, but still, I guess there's a reason.  A simple REST server doesn't need a big heavy framework, so I can build something that will fit in that measly 64MB heap.

Moore's law might apply to computing power, but apparently not to hosting companies.

Anonymous function, part 2, win. Epic win.

I think I've figured out why anonymous functions are great, and here's why:

function doSomething() {
  var x
  if (someLogic) {
     x = somethingHere
  }

  var k = doOtherStuff()

  if (k) {
    x = k.something
  }

  if (x) {
    doSomethingHere(x)
  }
}

Looking at this, it becomes very easy to forget to set x, or to screw up dealing with x. This code is all backwards. the thing we care about is x, not all the crap around it.

function doSomething() {
  ((function(x) {
    doSomethingHere(x)
  })(someLogic ? somethingHere : 
       (function(k) {k ? k.something : null})(doOtherStuff())))
}

Why is this good? It looks a bit like a cluster-fuck at first glance. I think there are two reasons why it's good. And I'm going to talk briefly about why I've discovered I LOVE the ternary operator.

With the ternary operator, we have no choice but to declare the result for both sides of the conditional. We can't just fall through if we don't care, we always have to care. This means we eliminate lazy code paths. We can't accidentally forget about a potentially important condition. And that bubbles up to the overall logic. Instead of some complex chain of if then and honestly maybe, we have a clear decision tree. With sequential if statements we essentially create a new clause which I'm going to call "maybe".

To rewrite the line

if (someLogic) {
  x = somethingHere
}

to
x maybe assigned somethingHere if someLogic happens to be true

I know that doesn't really scan in some ways, because ultimately, x will definitely be assigned is someLogic is true. But, we're not saying what happens after that, and that's when it becomes a maybe. Because there is no code path on the negative side of k, by the time we reach our doSomethingHere call, x maybe somethingHere, or it maybe k.something, or maybe it's nothing at all!

If we make this function more and more complex it becomes increasingly hard to track what order things needed to be in and create a bug because we screwed up the if statement order. If the order is important, then it should have been nested properly not created sequentially, and that's where our crazy functional syntax comes in. You can't write code sequentially in this style. As a result, you have to express the conditions clearly as a decision tree.

Anonymous functions, epic win, or total fail?

As I've been working with Scala over the past weeks and months, I've started using the anonymous function construct a fair bit.  I like the way it wraps things up, and it seems like the best way to do certain things.

The same construct is available in some other languages, most notable Javascript, so something like:

function doSomething() {
  var o = document.getElementById('thing');
  o.innerHTML = "Stuff";
  o.style.height="300px";
  o.style.backgroundColor="#ffffff;"
}

could end up as:

function doSomething() {
  (function(o) {
    o.innerHTML = "Stuff";
    o.style.height="300px";
    o.style.backgroundColor="#ffffff";
  })(document.getElementById('thing');
}

why is this good?

I'm not entirely sure. I think that when you have multiple blocks and when you have multiple arguments it could help with scoping. I think it might make things clearer potentially, though I'm not so sure. It would mean the args to the function are at the bottom not the top. That could be seen as either a good thing or a bad thing.

I could see it being helpful syntactically when processing lists perhaps?

function doSomething() {
  (function(v,c) {
    for (var t = 0; t<v.length; t+=1) {
      v[t].style.backgroundColor=c;
    }
  })(document.getElementByTagName('div'),'#ffffff');
}

Or is it really just functional programming wankery that doesn't have any place outside the confines of Scala and other similar things?

I've noticed there is a way to call an anonymous function recursively too, but I think that is definitely descending into the realm of esoteric to the point of silliness.

Digital bookshelf - no really

As we prepare for our relocation, we've been digging through our substantial book collection attempting to figure out what to keep and what to shed.  Each time I have to remind myself that at this point, most of my book reading is done electronically.  The thought occurred to me that this will lead to the decline of the bookshelf, and therefore the hallowed tradition of inspecting the bookshelves of a host as a means of either distraction or a vague attempt at insight.

I wondered then if it would be possible to create a virtual bookshelf.  Not a kindle type deal with a little set of icons on a tablet, but an actual, virtual bookshelf.  A projection of some kind of a bookshelf, 3' x 6' on a wall that you could see and perhaps even interact with.  The obvious rendition is some kind of LCD screen that is the same size.  TVs are getting so big these days, I'm thinking it wouldn't be much of a stretch to use one as a virtual bookshelf.  I think this is a very narrow vision though.  I think it would be better to do something new or at least different.  I could envision libraries revolutionized with monolith type structures that are virtual bookshelves.  You could browse the shelves of the mystery novels just as you do now, pausing to flick through the pages of a random book whose title looks intriguing.  Just touch a button on the virtual book to add it to your rental queue, and on with the browsing.

SBT

If I can ever crack SBT and IntelliJ, I'm gonna post about it, a lot. I just can't seem to get to a point where I can get things working properly with a Scala config. Every time I try, I just get frustrated and give up. The biggest barrier into Scala for me right now is documentation. Critical features like SBT just don't have good docs. I can't seem to run Scalatest Spec tests in IntelliJ, I just end up with weird compiler errors. I can't seem to get the SBT console working in IntelliJ either, and I can't seem to generate an IntelliJ project with either IDEA plugin for SBT, I just end up with weird dependency errors or something else going on. The local Scala users group seems to be barely active. They have one event listed, in June 2012, and are asking people if they are going. It's December 2011, that's six months away, I don't even know if I'm going to be in this state in six months, let alone able to attend some event. They don't seem to have regular meet-ups from what I can gather from their page, which is not a whole lot. I've resorted now to posting questions on Stack overflow, which seems to be turning out pretty well to be honest. People answer quickly and often include lots of helpful information on questions that once I've read the answer seem like they probably made me look pretty dumb. Well, I guess if you don't ask, you'll never know, so I'll take dumb yesterday, smart today over dumb forever. I might separate actual Scala posts with code from the main blog at some point too. Seems like they are confusing each other.

Scala, encore un fois!

How often do you have complex functionality in a declarative language work the first time out of the gate? Almost never right? Not so much with functional. The number of times a function I've written has worked out of the gate, or virtually of the gate is astounding me. Talk about getting this done faster. The only problem I'm having right now is that I'm still not that familiar with the syntax nuances, so I'm tripping up over that a bit. Really gotta sit down with the user guid and get up to speed on all the little bits and bobs that I'm missing right now. I read through the List class spec, and that was pretty helpful, but there's so much more!