Necessity is the mother of invention.

Jan 24 2018

Twitter Hashtag Trending Report

I created this Spark Streaming service to analyze hashtags coming out of Twitter.  Why?  Politics can be entertaining, or irritating, depending on your perception, but discussions about our culture and its people are worth having.

This tool is open source and provides a "top ten" report of Tweets matching any of a given (configurable) set of keywords.  Enjoy!

P.S. some assembly required

Mar 22 2017

Mikrotik expired certificate with OpenVPN

Did your OpenVPN / Mikrotik certificates (both CA and client certs) expire?  It turns out they were only good for one year!

The default duration for these VPN certificates, and Certificate Authority, is 365 days.  My OpenVPN VPN client quit working recently (it appears stuck in a loop retrying the connection and authentication process).  Oops.

The fix is to remove and re-create, then distribute, new Certificates for the OpenVPN Clients.  We must first remove keys and certificates for the Mikrotik CA, Server and Clients'.  If there is a simpler way, please shoot me a note below in the comments.

First, log in to your router with ssh and remove the old client certificates, server certificate and the certificate authority entries:

remove 2
remove 1
remove 0

Then, add back a new, 2-year duration Certificate Authority:

add name=CA2017 common-name=CA2017 key-usage=key-cert-sign,crl-sign days-valid=730
sign CA2017 ca-crl-host= name=CA2017

Update your OpenVPN configuration to use the new CA:

/interface ovpn-server server set certificate=CA2017

Next, create a new 2-year server certificate:

add name=server common-name=server days-valid=730
sign server ca=CA2017 name=server

Finish up on the router by creating a new 2-year client certificate for each client:

add name=client1 common-name=server days-valid=730
sign client1 ca=CA2017 name=client1

add name=client2 common-name=server days-valid=730
sign client2 ca=CA2017 name=client2

The certificates must be exported from the router and downloaded to your PC or laptop for dissemination.  Export from the command line, then visit the admin website or use sftp to retrieve the certificate files.

export-certificate export-passphrase=mysecret CA2017
export-certificate export-passphrase=mysecret client1
export-certificate export-passphrase=mysecret client2

From the Files section of the Mikrotik Web GUI you should see:


We can now move over to your client's .ovpn file and substitute new CA and client keys and certificates.  For this procedure we refer you back to the OpenVPN Client Configuration section of last year's blog post.  The .crt files can be inserted/substituted in your per-client .ovpn file as-is.  The .key file must be decrypted with your chosen export-passphrase thusly:

openssl rsa -in cert_export_client1.key -text

Hope this cures your issue and my apologies for being short-sighted on my original post.  I'm up to RouterOS version 6.41.3 on this end for this solution / workaround (updated 3/28/2018).

Jul 22 2016

Chrome for Ubuntu Xenial

Lately I've been working on a couple Ubuntu open source projects, MAAS and Juju.  While I have been making good use of the Ubuntu server distribution (16.04 LTS), I hadn't installed the desktop until today.

When I tried to add the Google Chrome web browser, I found a little work was required to add the Google PPA and get the install to work.  There are three steps required:

First, as root, create an APT source list file with a 64-bit architecture predicate:

echo 'deb [arch=amd64] stable main' > /etc/apt/sources.list.d/google.list

Second, add the Google PPA key:

wget -q -O - | apt-key add -

apt-get update

Third, install the stable chrome browser from Google's PPA:

apt-get install google-chrome-stable

Mar 04 2014

sort on two fields in mongo

Sorting in the mongo shell (JavaScript)

Sorting the results of a MongoDB query is straightforward in the mongo JavaScript shell.  One provides a JSON document with associative array listing the fields to sort on and their sort order (1 for ascending, -1 for descending).

db.myCollection.find().sort( { fieldA: 1 } )

When one wants to sort on two fields, for example, fieldA first, then fieldB second, the mongo JavaScript shell obeys the left-to-right order in the associative array without complaint.  An example of a two field sort in the mongo shell:

db.myCollection.find().sort( { fieldA: 1, fieldB: 1 } )

Sorting in python using pymongo

But what about a Python script using pymongo?  Can one use the same associative array syntax?  No.

However, one can use a List of immutable Tuples in Python to achieve the same effect.  An example of the same two field sort in a Python script using pymongo:

db.myCollection.find().sort( [ ('fieldA', 1), ('fieldB', 1) ] )

Documentation reference

For documentation on the JavaScript sort syntax:

For documentation on the Python pymongo sort syntax:

Feb 27 2014

datastax-agent tar installation

If you are like me, you sometimes want to install a product in an arbitrary location and not pollute your root filesystem (Linux, UNIX and MacOSX).

Normal package management-based installers (.deb and .rpm formats) require adding the DataStax repository to your list of authorized repositories.  See

The automated install (initiated from OpsCenter management web application) will configure the IP address for the Opscenter daemon's stomp listener for you.  If you perform a tar-based installation of DSE or OpsCenter you will need to take some additional steps to configure your datastax-agent (formerly known as "opscenter-agent").

step one - configure stomp listener endpoint

After you untar the dse-x.y.z-bin.tar.gz archive, locate the dse-x.y.z/datastax-agent/conf directory relative to the install directory.  Substitute the DSE version for x.y.z, e.g. dse-4.0.0/datastax-agent/conf.

Edit dse-x.y.z/datastax-agent/conf/address.yaml and set the IP address where you have installed OpsCenter (this could be a different node or host).  Use your own IP address in place of the one below.


step two - configure path to cassandra.yaml

This step is tricky because the DataStax documentation (URL above) is misleading.  When a tar-based install is conducted, datastax-agent cannot locate DSE / Cassandra configuration files using a relative path (as of DSE 4.0.0).  You will get the following error message:

datastax-agent startup error
ERROR [Initialization] 2014-02-27 07:57:24,306 Exception in thread "Initialization"
ERROR [StompConnection receiver] 2014-02-27 07:57:24,306 failed calling listener
clojure.lang.ExceptionInfo: throw+: {:type :bad-permissions, :message "Unable to locate the cassandra.yaml configuration file. If your configuration file is not located with the Cassandra install, please set the 'conf_location' option in the Cassandra section of the OpsCenter cluster configuration file and restart opscenterd. Checked the following directories: [\"/etc/dse/cassandra/cassandra.yaml\" \"/etc/cassandra/conf/cassandra.yaml\" \"/etc/cassandra/cassandra.yaml\" \"/Users/resources/cassandra/conf/cassandra.yaml\"]"} {:object {:type :bad-permissions, :message "Unable to locate the cassandra.yaml configuration file. If your configuration file is not located with the Cassandra install, please set the 'conf_location' option in the Cassandra section of the OpsCenter cluster configuration file and restart opscenterd. Checked the following directories: [\"/etc/dse/cassandra/cassandra.yaml\" \"/etc/cassandra/conf/cassandra.yaml\" \"/etc/cassandra/cassandra.yaml\" \"/Users/resources/cassandra/conf/cassandra.yaml\"]"}, :environment {tar-location "/Users/resources/cassandra/conf/cassandra.yaml", conf nil, checked-files ["/etc/dse/cassandra/cassandra.yaml" "/etc/cassandra/conf/cassandra.yaml" "/etc/cassandra/cassandra.yaml" "/Users/resources/cassandra/conf/cassandra.yaml"]}}
        at opsagent.util.cassandra_util$cassandra_conf_location.invoke(cassandra_util.clj:118)
        at opsagent.util.cassandra_util$get_cassandra_conf.invoke(cassandra_util.clj:130)
        at opsagent.opsagent$create_thrift_conf_vars.invoke(opsagent.clj:52)
        at opsagent.opsagent$post_interface_startup.doInvoke(opsagent.clj:95)
        at clojure.lang.RestFn.invoke(
        at opsagent.conf$handle_new_conf.invoke(conf.clj:171)
        at opsagent.messaging$message_callback$fn__5192.invoke(messaging.clj:30)
        at opsagent.messaging.proxy$java.lang.Object$StompConnection$Listener$7f16bc72.onMessage(Unknown Source)
        at org.jgroups.client.StompConnection.notifyListeners(

 The workaround for this error is to explicitly tell datastax-agent where to find cassandra.yaml.  This will be in the following path relative to your DSE installation directory:  resources/cassandra/conf/cassandra.yaml

Edit datastax-agent/conf/address.yaml again and add the following directive.  Note that this is undocumented and the official documentation is misleading at this time.  This will be fixed in an upcoming release but this workaround should help for the impatient.

Assuming you have installed DSE 4.0.0 to /apps/dse-4.0.0:

cassandra_conf: /apps/dse-4.0.0/resources/cassandra/conf/cassandra.yaml

Re-start your datastax-agent using "datastax-agent/bin/datastax-agent", run from the DSE installation directory (/apps/dse-4.0.0 in this example).

Feb 12 2014

NoSQL database acronyms

In my over-simplified view of the world, here is my summary of distributed database acronyms, in the order I remember their introduction.


What every web application has to do, repeatedly.

  • Create
  • Read
  • Update
  • Delete


  • Atomicity       (transaction context where operations all fail or all succeed)
  • Consistency  (data and relationships are valid -- think check constraint and foreign key constraint)
  • Isolation        (one client's uncommitted transaction invisible to others)
  • Durability       (committed writes survive power loss, other perils)

CAP (aka Brewer Theorem)

Choose two.

  • Consistency           (all nodes see same data at same time)
  • Availability              (every request gets a pass/fail response)
  • Partition tolerance  (loss of node)


  • Basically Available      (every request gets a pass/fail response)
  • Soft state                    (state may change without request arriving)
  • Eventual consistency  (system is async, stream-like.  it eventually becomes consistent after cessation of requests


  • helps solve the "last writer" problem due to "speed of light race condition"
  • forms consensus in network of unreliable processors (e.g. in presence of packet loss)
  • paxos is a consensus protocol, a state machine
  • proposer (asserts), acceptor (votes), learner (takes action upon consensus)

Feb 11 2014

Remove old Solaris boot snapshots

Removal or deletion of old boot images is useful from time to time. Deleting the old, unneeded entries from the boot menu is one advantage, but the recovery of disk space is the more significant benefit.

Each boot menu item, like 'OpenIndiana-1', or 'OpenIndiana' has a corresponding Boot Environment. Each of these Boot Environments is realized with a ZFS snapshot of the root filesystem. The following utilizes OpenIndiana for the example environment, but the utility works the same way in OpenSolaris and Oracle Solaris.

To remove the ZFS snapshot, we first want to remove the boot menu entry. Management of Boot Environments, as the Solaris documentation refers to them, is not too difficult -- the beadm utility is used.  

First obtain a list of existing BootEnvironments. Decide which one(s) to keep and which ones to expunge.

root@castor:/# beadm list -a
BE/Dataset/Snapshot                             Active Mountpoint Space Policy Created
   rpool/ROOT/openindiana                       -      -          13.2M static 2012-11-04 17:39
   rpool/ROOT/openindiana-1                     -      -          18.0M static 2012-11-04 20:32
   rpool/ROOT/openindiana-2                     -      -          20.7M static 2013-09-16 20:18
   rpool/ROOT/openindiana-3                     NR     /          9.20G static 2013-12-07 14:44
   rpool/ROOT/openindiana-3@2012-11-05-02:32:06 -      -          156M  static 2012-11-04 20:32
   rpool/ROOT/openindiana-3@2013-09-17-01:18:25 -      -          1.47G static 2013-09-16 20:18
   rpool/ROOT/openindiana-3@2013-12-07-20:44:51 -      -          1.13G static 2013-12-07 14:44
   rpool/ROOT/openindiana-3@install             -      -          68.5M static 2012-11-04 17:44

Note that the corresponding ZFS snapshot names are shown, indented, below each boot environment (boot menu entry) name.

Make sure the boot entry / boot environment you choose does not have "NR". "R" or "N" in the Active column of "beadm list".  Those are your bootable or soon-to-be-bootable snapshots!

In this case, pretend we want to remove the (original) 'openindiana' boot environment.  

root@castor:/# beadm destroy openindiana
Are you sure you want to destroy openindiana?
This action cannot be undone (y/[n]): y
Destroyed successfully

One could repeat this step as needed to remove other unneeded boot environments.  

For the paranoid, to verify the snapshot was truly deleted and disk space recovered, one can check that the ZFS snapshot no longer appears:

root@castor:/# zfs list -t snapshot
BE/Dataset/Snapshot                 Active Mountpoint Space Policy Created
   rpool/ROOT/openindiana-3         NR     /          6.07G static 2013-12-07 14:44
   rpool/ROOT/openindiana-3@install -      -          1.93G static 2012-11-04 17:44

For further reading, refer to Oracle's Solaris documentation on beadm:

Nov 13 2011

OpenWrt 10.03.1 dnsmasq.conf for gPXE

In the latest version of OpenWrt backfire version 10.03.1 the OpenWrt development team has added many new capabilities for configuring dnsmasq and the many options related to DHCP.

Dnsmasq in OpenWrt can be configured to support iSCSI and gPXE without resorting to creating a dnsmasq.conf file as I did in the 10.03 OpenWrt release.

The following link has the dnsmasq configuration syntax for the 10.03.1 release of OpenWrt.

This link has details on the dnsmasq daemon command line options.  It provides insight to use of the /etc/dnsmasq.conf file, if you choose to use it.

The syntax of /etc/dnsmasq.conf changed between 10.03 and 10.03.1 because dnsmasq changed versions from 2.52-2 to 2.55-6.

The 10.03 OpenWrt gPXE post can be referred to and the new dnsmasq.conf excerpt is below:

# required by old versions of gPXE (e.g. the one that comes with OracleVM)

# MAC-specific tags
dhcp-mac=set:ECS,00:14:aa:bb:cc:dd       # ECS mobo / Socket AM2 Athlon
dhcp-mac=set:SLI,00:11:bb:cc:dd:ee       # Asus A8N-SLI mobo forcedeth / Socket 939 Athlon X2

dhcp-match=set:OTHER,175            # tags the request with OTHER if the gPXE option was supplied in DHCP request

dhcp-option=175,8:1:1          # turn on the keep-san option to allow iSCSI-capable OS installation


## Last match wins!

#dhcp-boot=pxelinux.0                  # ONLY SET IF want gPXE to run script or chain-load pxelinux
dhcp-boot=tag:#OTHER,gpxe.0             # Here #OTHER means 'not OTHER': i.e. the tag is not set
#dhcp-boot=tag:ECS,sanboot-test.gpxe   # run gpxe script

Nov 07 2011

Windows Service for Jetty 7

How to install Jetty 7 as a Windows service

Earlier versions of Jetty came with a Windows Service wrapper but Jetty 7 does not.  To use Jetty in a production environment it is helpful to create a Windows service to run Jetty so that the service may run without manual intervention, like when the machine boots.  

One way to accomplish this task is to make use of the (Jakarta) Commons Daemon project.  In the example that follows I am using XWiki 3.2 that bundles Jetty 7.  XWiki 3.2 is unzipped to C:\apps\xwiki32 and Jetty 7 resides within at C:\apps\xwiki32\jetty.

Commons Daemon

First read about the commons daemon project at the following URL:

Download the native binaries for Windows from this URL:

The daemon project essentially provides two Executables for the Windows operating system:  prunmgr.exe and prunsrv.exe, collectively referred to as procrun.  In essence, procrun provides the capability to run arbitrary Java main methods as a Windows service.

Determine Java arguments when starting Jetty

In my case I wanted to run a web application called XWiki deployed to Jetty, and have Jetty run under the Windows SYSTEM account whenever the machine boots up.  I modified the start_xwiki.bat batch file to echo the Java command line before running java so I could capture all the arguments to Java.  You'll need these later when configuring procrun.

In my case, the Java command line XWiki 3.2 wanted to use when running Java was:

java -Xmx512m -XX:MaxPermSize=128m -Djetty.port=8080 -Djetty.home=jetty -DSTOP.KEY=xwiki -DSTOP.PORT=8079 -Dfile.encoding=UTF8 -Dorg.mortbay.jetty.Request.maxFormContentSize=1000000 -jar jetty/start.jar

Rename prunsrv.exe and prunmgr.exe to reflect your Jetty Service

Because Windows' Service manager (accessible from the Control Panel) identifies services by the executable filename used to start and stop the service, it is helpful to rename the two windows executables, prunsrv.exe and prunmgr.exe to reflect the fact this is a XWiki Jetty service.

cd \apps\xwiki32
copy prunsrv.exe XWikiJettyService.exe
copy prunmgr.exe XWikiJettyServiceManager.exe

Install Service

The following step configures procrun and installs the Windows Service.  The JVM arguments below correspond to the arguments we captured from the original Java command line invocation from start_xwiki.bat.

XWikiJettyService.exe  //IS//XWikiJettyServiceManager --DisplayName="XWiki Jetty Service" --Install=C:\apps\xwiki32\XWikiJettyService.exe --LogPath=C:\apps\xwiki32\jetty\logs --LogLevel=Debug --StdOutput=auto --StdError=auto --StartMode=Java --StopMode=Java --Jvm=auto --Startup=auto --JvmMx=512 --StartPath=C:\apps\xwiki32 ++JvmOptions=-XX:MaxPermSize=128m --Classpath=C:\apps\xwiki32\jetty\start.jar --StartClass=org.eclipse.jetty.start.Main --StopClass=org.eclipse.jetty.start.Main ++StopParams=--stop ++JvmOptions=-Djetty.home=C:\apps\xwiki32\jetty ++JvmOptions=-Djetty.port=8080 ++JvmOptions=-DSTOP.PORT=8079 ++JvmOptions=-DSTOP.KEY=xwiki ++JvmOptions=-Djetty.logs=C:\apps\xwiki32\jetty\logs ++JvmOptions=-Dfile.encoding=UTF8 ++JvmOptions=-Dorg.mortbay.jetty.Request.maxFormContentSize=1000000

Test procrun

The following will allow you to test your shiny new Windows Service by running it in the foreground so you can see any errors that may occur if the configuration isn't quite right.

XWikiJettyService.exe //TS/XWikiJettyServiceManager

Procrun logs, Stdout and Stderr logs for your service

These logs will be saved to the Jetty logs directory in the scenario described.  The location of the procrun daemon logs are specified by the
--LogPath=C:\apps\xwiki32\jetty\logs argument when you ran the XWikiJettyService.exe (prunsrv.exe) above.

The actual daemon stdout and stderr logs for Jetty are controlled by the -Djetty.logs=C:\apps\xwiki32\jetty\logs JVM argument.

Remove Service

cd \apps\xwiki32
XWikiJettyService.exe  //DS//XWikiJettyServiceManager

Created by Administrator on 07/09/2013
This website content is not licensed to you. All rights reserved.
XWiki Enterprise 9.11.1 - Documentation