Are you the publisher? Claim or contact us about this channel

Embed this content in your HTML


Report adult content:

click to rate:

Account: (login)

More Channels

Channel Catalog

Articles on this Page

(showing articles 1 to 50 of 50)
(showing articles 1 to 50 of 50)

    0 0

    Friday's news was full of breaking panic about an "attack" on the NHS, making it sound like someone had deliberately made an attempt to get in there and cause damage.

    It turns out that it wasn't an attack against the NHS itself, just a wide scale ransomware attack which combined click-through installation and intranet propagation by way of a vulnerability which the NSA had kept for internal use for some time.

    Laptops, Lan ports and SICP

    The NHS got decimated for a combination of issues:

    1. A massive intranet for SMB worms to run free.
    2. Clearly, lots of servers/desktops running the SMB protocol.
    3. One or more people reading an email with the original attack, bootstrapping the payload into the network.
    4. A tangible portion of the machines within some parts of the network running unpatched versions of Windows, clearly caused in part by the failure of successive governments to fund a replacement program while not paying MSFT for long-term support.
    5. Some of these systems within part of medical machines: MRI scanners, VO2 test systems, CAT scanners, whatever they use in the radiology dept —to name but some of the NHS machines I've been through in the past five years.
    The overall combination then is: a large network/set of networks with unsecured, unpatched targets were vulnerable to a drive-by attack, the kind of attack, which, unlike a nation state itself, you may stand a chance of actually defending against.

    What went wrong?

    Issue 1: The intranet. Topic for another post.

    Issue 2: SMB.

    In servers this can be justified, though it's a shame that SMB sucks as a protocol. Desktops? It's that eternal problem: these things get stuck in as "features", but which sometimes come to burn you. Every process listening on a TCP or UDP port is a potential attack point. A 'netstat -a" will list running vulnerabilities on your system; enumerating running services "COM+, Sane.d? mDNS, ..." which you should review and decide whether they could be halted. Not that you can turn mDNS off on a macbook...

    Issue 3: Email

    With many staff, email clickthrough is a function of scale and probability: someone will, eventually. Probability always wins.

    Issue 4: The unpatched XP boxes.

    This is why Jeremy Hunt is in hiding, but it's also why our last Home Secretary, tasked with defending the nation's critical infrastructure, might want to avoid answering questions. Not that she is answering questions right now.

    Finally, 5: The medical systems.

    This is a complication on the "patch everything" story because every update to a server needs to be requalified. Why? Therac-25.

    What's critical here is that the NHS was 0wned, not by some malicious nation state or dedicated hacker group: it fell victim to drive-by ransomware targeted at home users, small businesses, and anyone else with a weak INFOSEC policy This is the kind of thing that you do actually stand a chance of defending against, at least in the laptop, desktop and server.


    Defending against malicious nation state is probably near-impossible given physical access to the NHS network is trivial: phone up at 4am complaining of chest pains and you get a bed with a LAN port alongside it and told to stay there until there's a free slot in the radiology clinic.

    What about the fact that the NSA had an exploit for the SMB vulnerability and were keeping quiet on it until the Shadow Brokers stuck up online? This is a complex issue & I don't know what the right answer is.

    Whenever critical security patches go out, people try and reverse engineer them to get an attack which will work against unpatched versions of: IE, Flash, Java, etc. The problems here were:
    • the Shadow Broker upload included a functional exploit, 
    • it was over the network to enable worms, 
    • and it worked against widely deployed yet unsupported windows versions.
    The cost of developing the exploit was reduced, and the target space vast, especially in a large organisation. Which, for a worm scanning and attacking vulnerable hosts, is a perfect breeding ground.

    If someone else had found and fixed the patch, there'd still have been exploits out against it -the published code just made it easier and reduced the interval between patch and live exploit

    The fact that it ran against an old windows version is also something which would have existed -unless MSFT were notified of the issue while they were still supporting WinXP. The disincentive for the NSA to disclose that is that a widely exploitable network attack is probably the equivalent of a strategic armament, one step below anything that can cut through a VPN and the routers, so getting you inside a network in the first place.

    The issues we need to look at are
    1. How long is it defensible to hold on to an exploit like this?
    2. How to keep the exploit code secure during that period, while still using it when considered appropriate?
    Here the MSFT "tomahawk" metaphor could be pushed a bit further. The US govt may have tomahawk missiles with nuclear payloads, but the ones they use are the low-damage conventional ones. That's what got out this time.

    WMD in the Smithsonia

    One thing that MSFT have to consider is: can they really continue with the "No more WinXP support" policy? I know they don't want to do it, the policy of making customers who care paying for the ongoing support is a fine way to do it, it's just it leaves multiple vulnerabilites. People at home, organisations without the money and who think "they won't be a target", and embedded systems everywhere -like a pub I visited last year whose cash registers were running Windows XP embedded; all those ATMs out there, etc, etc.

    Windows XP systems are a de-facto part of the nation's critical infrastructure.

    Having the UK and US governments pay for patches for the NHS and everyone else could be a cost effective way of securing a portion of the national infrastructure, for the NHS and beyond.

    (Photos: me working on SICP during an unplanned five day stay and the Bristol Royal Infirmary. There's a LAN port above the bed I kept staring at; Windows XP Retail packaging, Smithsonian aerospace museum, the Mall, Washington DC)

    0 0

    Its nice to see, Sir Tim Berners-Lee as a recipient of A.M. Turing Award. More details are available on

    0 0
  • 05/17/17--08:53: Nick Kew: The Great B Minor
  • This Sunday, May 21st, we’re performing Bach’s B Minor Mass at the Guildhall, Plymouth.  This work needs no introduction, and I have no hesitation recommending it for readers who enjoy music and are within evening-out distance of Plymouth.

    Tickets are cheaper in advance than on the door, so you might want to visit your favourite regular ticket vendor or google for online sales.

    Minor curiosity: the edition we’re using was edited by Arthur Sullivan.  Yes, he of G&S, and an entirely different era and genre of music!   It’s also the Novello edition used in most performances in Britain.

    0 0

    As you may already know Apache CXF has been offering a simple but effective support for tracing CXF client and server calls with HTrace since 2015.

    What is interesting about this feature is that it was done after the DevMind attended to Apache Con NA 2015 and got inspired about integrating CXF with HTrace.

    You'll be glad to know this feature has now been enhanced to get the trace details propagated to the logs which is the least intrusive way of working with HTrace though should you need more advanced control, CXF will help, see this section for example.

    CXF has also been integrated with Brave. That should do better for CXF OSGI users. The integration work with Brave 4 is under way now.

    0 0

    Microsoft logo

    The last week I joined Microsoft Build 2017 in Seattle together with some members of the Technical Advisory Group (TAG).

    I would like to share some of the good points for developers and the Enterprise field in terms of how Microsoft is continuing to adopt Open Source and Open Standards in different ways.

    0 0

    Here’s another blog post of mine that was initially published by Computerworld UK.

    My current Fiat Punto Sport is the second Diesel car that I own, and I love those engines. Very smooth yet quite powerful acceleration, good fuel savings, a discount on state taxes thanks to low pollution, and it’s very reliable and durable. And fun to drive. How often does Grandma go “wow” when you put the throttle down in your car? That happens here, and that Grandma is not usually a car freak.

    Diesel engines used to be boring, but they have made incredible progress in the last few years – while staying true to their basic principles of simplicity, robustness and reliability.

    The recent noise about the Apache Software Foundation (ASF) moving to Git, or not, made me think that the ASF might well be the (turbocharged, like my car) Diesel engine of open source. And that might be a good thing.

    The ASF’s best practices are geared towards project sustainability, and building communities around our projects. That might not be as flashy as creating a cool new project in three days, but sometimes you need to build something durable, and you need to be able to provide your users with some reassurances that that will be the case – or that they can take over cleanly if not.

    In a similar way to a high tech Diesel engine that’s built to last and operate smoothly, I think the ASF is well suited for projects that have a long term vision. We often encourage projects that want to join the ASF via its Incubator to first create a small community and release some initial code, at some other place, before joining the Foundation. That’s one way to help those projects prove that they are doing something viable, and it’s also clearly faster to get some people together and just commit some code to one of the many available code sharing services, than following the ASF’s rules for releases, voting etc.

    A Japanese 4-cylinder 600cc gasoline-powered sports bike might be more exciting than my Punto on a closed track, but I don’t like driving those in day-to-day traffic or on long trips. Too brutal, requires way too much attention. There’s space for both that and my car’s high tech Diesel engine, and I like both styles actually, depending on the context.

    Open Source communities are not one-size-fits-all: there’s space for different types of communities, and by exposing each community’s positive aspects, instead of trying to get them to fight each other, we might just grow the collective pie and live happily ever after (there’s a not-so-hidden message to sensationalistic bloggers in that last paragraph).

    I’m very happy with the ASF being the turbocharged Diesel engine of Open Source – it does have to stay on its toes to make sure it doesn’t turn into a boring old-style Diesel, but there’s no need to rush evolution. There’s space for different styles.

    0 0

    It looks like the Russians interfered with the US elections, not just from the alleged publishing of the stolen emails, or through the alleged close links with the Trump campaign, but in the social networks, creating astroturfed campaigns and repeating the messages the country deemed important.

    Now the UK is having an election. And no doubt the bots will be out. But if the Russians can do bots: so can I.

    This then, is @dissidentbot.

    Dissident bot is a Raspbery Pi running a 350 line ruby script tasked with heckling politicans
    unrelated comments seem to work, if timely
    It offers:

    • The ability to listen to tweets from a number of sources: currently a few UK politicians
    • To respond pick a random responses from a set of replies written explicitly for each one
    • To tweet the reply after a 20-60s sleep.
    • Admin CLI over Twitter Direct Messaging
    • Live update of response sets via github.
    • Live add/remove of new targets (just follow/unfollow from the twitter UI)
    • Ability to assign a probability of replying, 0-100
    • Random response to anyone tweeting about it when that is not a reply (disabled due to issues)
    • Good PUE numbers, being powered off the USB port of the wifi base station, SSD storage and fanless naturally cooled DC. Oh, and we're generating a lot of solar right now, so zero-CO2 for half the day.
    It's the first Ruby script of more than ten lines I've ever written; interesting experience, and I've now got three chapters into a copy of the Pickaxe Book I've had sitting unloved alongside "ML for the working programmer".  It's nice to be able to develop just by saving the file & reloading it in the interpreter...not done that since I was Prolog programming. Refreshing.
    Strong and Stable my arse
    Without type checking its easy to ship code that's broken. I know, that's what tests are meant to find, but as this all depends on the live twitter APIs, it'd take effort, including maybe some split between Model and Control. Instead: broken the code into little methods I can run in the CLI.

    As usual, the real problems surface once you go live:
    1. The bot kept failing overnight; nothing in the logs. Cause: its powered by the router and DD-WRT was set to reboot every night. Fix: disable.
    2. It's "reply to any reference which isn't a reply itself" doesn't work right. I think it's partly RT related, but not fully tracked it down.
    3. Although it can do a live update of the dissident.rb script, it's not yet restarting: I need to ssh in for that.
    4. I've been testing it by tweeting things myself, so I've been having to tweet random things during testing.
    5. Had to add handling of twitter blocking from too many API calls. Again: sleep a bit before retries.
    6. It's been blocked by the conservative party. That was because they've been tweeting 2-4 times/hour, and dissidentbot originally didn't have any jitter/sleep. After 24h of replying with 5s of their tweets, it's blocked.
    The loopback code is the most annoying bug; nothing too serious though.

    The DM CLI is nice, the fact that I haven't got live restart something which interferes with the workflow.
      Dissidentbot CLI via Twitter DM
    Because the Pi is behind the firewall, I've no off-prem SSH access.

    The fact the conservatives have blocked me, that's just amusing. I'll need another account.

    One of the most amusing things is people argue with the bot. Even with "bot" in the name, a profile saying "a raspberry pi", people argue.
    Arguing with Bots and losing

    Overall the big barrier is content.  It turns out that you don't need to do anything clever about string matching to select the right tweet: random heckles seems to blend in. That's probably a metric of political debate in social media: a 350 line ruby script tweeting random phrases from a limited set is indistinguishable from humans.

    I will accept Pull Requests of new content. Also: people are free to deploy their own copies. without the self.txt file it won't reply to any random mentions, just listen to its followed accounts and reply to those with a matching file in the data dir.

    If the Russians can do it, so can we.

    0 0

    The board design went off to PCBWay via web browser and 5 days later 5 boards arrived by DHL from China. The whole process was unbelievably smooth. This was the first time I had ordered boards using the output of KiCad so I was impressed with both KiCad and PCBWay. The boards were simple, being 2 layer, but complex being large with some areas needing to carry high amps. So how did I do ?

    I made 1 mistake on the footprints. The 2 terminal connectors for the 600v ultrasound output didn’t have pads on both sides. This didn’t matter as being through hole the connectors soldered ok. Other than that PCBWay did exactly what I had instructed them to. Even the Arduino Mega footprint fitted perfectly.

    How did it perform ?

    Once populated the board initially appeared to perform well. Random frequency from 20KHz to 150KHz worked. The drive waveform from the Mostfet drivers into the Mosfet was near perfect with no high frequency ringing on the edges with levels going from 0-12v and back in much less than 1us. However I noticed some problems with the PWM control. There was none. With PWM pulses at 10% the MOSFETS would turn on for 90% of the time and drive a wildly resonant waverform through the coil. Rather like a little hammer hitting a bit pendulum and having it feedback into resonance. On further investigation the scope showed that when the Mosfet tried to switch off the inductor carried on producing a flyback voltage causing the MostFet to continue conducting till the opposing mosfet turned on. Initially I thought this was ringing, but it turned out a simple pair of 1A high frequency Schottky diodes across each winding of the primary coil returned the energy to the the 12V line eliminating the fly back. Now I had ringing, at 10MHz, but control over the power output via a digital pot. I could leave it at that, but this 10MHz would probably transmit and cause problems with other equipment on the boat.

    I think the difference between the red and blue signals is due to slightly different track lengths on each Mosfet. The shorter track not ringing nearly as much shown in the blue signal. The longer track with more capacitance ringing more and inducing a parasitic ring in the blue track. To eliminate this 2 things were done. Traditional Snubber RC networks had little or no impact. So a 100nF cap as close as possible to the Drain and Source on each Mosfet (RPF50N6) eliminated some of the high frequency, and a 100uF cap on the center tap to store the energy returned to the 12V line by flyback. This reduced the peak current.

    There is still some ringing, but now the frequency is less and it is less violent. The ripple on the 12V line is now less than 0.2v and filtered out by decoupling caps on the supply pins to the Ardiono Mega. All of these modifications have been accommodated on the underside of the board.

    The board now produces 60W per transducer between 20 and 150 KHz at 50% PWM drawing 5A from the supply. This is very loud on my desk and far louder than the Ultrasound Antifouling installed in Isador, which seems to work. I will need to implement a control program that balances power consumption against noise levels against effectiveness, but that is all software. There are sensors on board for temperature, current and voltage so it should be possible to have the code adapt to its environment.

    Board Layout mistakes

    Apart from the circuit errors, I made some mistakes in the MoSFET power connections. Rev2 of the board will have the MosFETS placed as close to the primary of the transformer with identical track lengths. Hopefully this will eliminate the ringing seen on the red trace and made both line the blue trace.

    I have 4 spare unpopulated PCBs. If I do a rev2 board, I will use PCBWay again. Their boards were perfect, all the mistakes were mine.



    0 0

    Auto bootstrapping is a handy feature when it comes to growing an Apache Cassandra cluster. There are some unknowns about how this feature works which can lead to data inconsistencies in the cluster. In this post I will go through a bit about the history of the feature, the different knobs and levers available to operate it, and resolving some of the common issues that may arise.


    Here are links to the various sections of the post to give you an idea of what I will cover.


    The bootstrap feature in Apache Cassandra controls the ability for the data in cluster to be automatically redistributed when a new node is inserted. The new node joining the cluster is defined as an empty node without system tables or data.

    When a new node joins the cluster using the auto bootstrap feature, it will perform the following operations

    • Contact the seed nodes to learn about gossip state.
    • Transition to Up and Joining state (to indicate it is joining the cluster; represented by UJ in the nodetool status).
    • Contact the seed nodes to ensure schema agreement.
    • Calculate the tokens that it will become responsible for.
    • Stream replica data associated with the tokens it is responsible for from the former owners.
    • Transition to Up and Normal state once streaming is complete (to indicate it is now part of the cluster; represented by UN in the nodetool status).

    The above operations can be seen in the logs.

    Contact the seed nodes to learn about gossip state

    INFO  [HANDSHAKE-/] 2017-05-12 16:14:45,290 - Handshaking version with /
    INFO  [GossipStage:1] 2017-05-12 16:14:45,318 - Node / is now part of the cluster
    INFO  [GossipStage:1] 2017-05-12 16:14:45,325 - Node / is now part of the cluster
    INFO  [GossipStage:1] 2017-05-12 16:14:45,326 - Node / is now part of the cluster
    INFO  [GossipStage:1] 2017-05-12 16:14:45,328 - Node / is now part of the cluster
    INFO  [SharedPool-Worker-1] 2017-05-12 16:14:45,331 - InetAddress / is now UP
    INFO  [HANDSHAKE-/] 2017-05-12 16:14:45,331 - Handshaking version with /
    INFO  [HANDSHAKE-/] 2017-05-12 16:14:45,383 - Handshaking version with /
    INFO  [HANDSHAKE-/] 2017-05-12 16:14:45,387 - Handshaking version with /
    INFO  [SharedPool-Worker-1] 2017-05-12 16:14:45,438 - InetAddress / is now UP
    INFO  [SharedPool-Worker-2] 2017-05-12 16:14:45,438 - InetAddress / is now UP
    INFO  [SharedPool-Worker-3] 2017-05-12 16:14:45,438 - InetAddress / is now UP
    INFO  [main] 2017-05-12 16:14:46,289 - Starting up server gossip

    Transition to Up and Joining state

    INFO  [main] 2017-05-12 16:14:46,396 - JOINING: waiting for ring information

    Contact the seed nodes to ensure schema agreement

    Take note of the last entry in this log snippet.

    INFO  [GossipStage:1] 2017-05-12 16:14:49,081 - Node / is now part of the cluster
    INFO  [SharedPool-Worker-1] 2017-05-12 16:14:49,082 - InetAddress / is now UP
    INFO  [GossipStage:1] 2017-05-12 16:14:49,095 - Updating topology for /
    INFO  [GossipStage:1] 2017-05-12 16:14:49,096 - Updating topology for /
    INFO  [HANDSHAKE-/] 2017-05-12 16:14:49,096 - Handshaking version with /
    INFO  [GossipStage:1] 2017-05-12 16:14:49,098 - Node / is now part of the cluster
    INFO  [SharedPool-Worker-1] 2017-05-12 16:14:49,102 - InetAddress / is now UP
    INFO  [GossipStage:1] 2017-05-12 16:14:49,103 - Updating topology for /
    INFO  [HANDSHAKE-/] 2017-05-12 16:14:49,104 - Handshaking version with /
    INFO  [GossipStage:1] 2017-05-12 16:14:49,104 - Updating topology for /
    INFO  [GossipStage:1] 2017-05-12 16:14:49,106 - Node / is now part of the cluster
    INFO  [SharedPool-Worker-1] 2017-05-12 16:14:49,111 - InetAddress / is now UP
    INFO  [GossipStage:1] 2017-05-12 16:14:49,112 - Updating topology for /
    INFO  [HANDSHAKE-/] 2017-05-12 16:14:49,195 - Handshaking version with /
    INFO  [GossipStage:1] 2017-05-12 16:14:49,236 - Updating topology for /
    INFO  [GossipStage:1] 2017-05-12 16:14:49,247 - Node / is now part of the cluster
    INFO  [SharedPool-Worker-1] 2017-05-12 16:14:49,248 - InetAddress / is now UP
    INFO  [InternalResponseStage:1] 2017-05-12 16:14:49,252 - Enqueuing flush of schema_keyspaces: 1444 (0%) on-heap, 0 (0%) off-heap
    INFO  [MemtableFlushWriter:2] 2017-05-12 16:14:49,254 - Writing Memtable-schema_keyspaces@1493033009(0.403KiB serialized bytes, 10 ops, 0%/0% of on/off-heap limit)
    INFO  [MemtableFlushWriter:2] 2017-05-12 16:14:49,256 - Completed flushing .../node5/data0/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-tmp-ka-1-Data.db (0.000KiB)for commitlog position ReplayPosition(segmentId=1494569684606, position=119856)
    INFO  [InternalResponseStage:1] 2017-05-12 16:14:49,367 - Enqueuing flush of schema_columnfamilies: 120419 (0%) on-heap, 0 (0%) off-heap
    INFO  [MemtableFlushWriter:1] 2017-05-12 16:14:49,368 - Writing Memtable-schema_columnfamilies@1679976057(31.173KiB serialized bytes, 541 ops, 0%/0% of on/off-heap limit)
    INFO  [MemtableFlushWriter:1] 2017-05-12 16:14:49,396 - Completed flushing .../node5/data0/system/schema_columnfamilies-45f5b36024bc3f83a3631034ea4fa697/system-schema_columnfamilies-tmp-ka-1-Data.db (0.000KiB)for commitlog position ReplayPosition(segmentId=1494569684606, position=119856)
    INFO  [InternalResponseStage:5] 2017-05-12 16:14:50,824 - Enqueuing flush of schema_usertypes: 160 (0%) on-heap, 0 (0%) off-heap
    INFO  [MemtableFlushWriter:2] 2017-05-12 16:14:50,824 - Writing Memtable-schema_usertypes@1946148009(0.008KiB serialized bytes, 1 ops, 0%/0% of on/off-heap limit)
    INFO  [MemtableFlushWriter:2] 2017-05-12 16:14:50,826 - Completed flushing .../node5/data0/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/system-schema_usertypes-tmp-ka-10-Data.db (0.000KiB)for commitlog position ReplayPosition(segmentId=1494569684606, position=252372)
    INFO  [main] 2017-05-12 16:14:50,404 - JOINING: schema complete, ready to bootstrap

    Calculate the tokens that it will become responsible for

    INFO  [main] 2017-05-12 16:14:50,404 - JOINING: waiting for pending range calculation
    INFO  [main] 2017-05-12 16:14:50,404 - JOINING: calculation complete, ready to bootstrap
    INFO  [main] 2017-05-12 16:14:50,405 - JOINING: getting bootstrap token

    Stream replica data associated with the tokens it is responsible for from the former owners

    Take note of the first and last entries in this log snippet.

    INFO  [main] 2017-05-12 16:15:20,440 - JOINING: Starting to bootstrap...
    INFO  [main] 2017-05-12 16:15:20,461 - [Stream #604b5690-36da-11e7-aeb6-9d89ad20c2d3] Executing streaming plan for Bootstrap
    INFO  [StreamConnectionEstablisher:1] 2017-05-12 16:15:20,462 - [Stream #604b5690-36da-11e7-aeb6-9d89ad20c2d3] Starting streaming to /
    INFO  [StreamConnectionEstablisher:2] 2017-05-12 16:15:20,462 - [Stream #604b5690-36da-11e7-aeb6-9d89ad20c2d3] Starting streaming to /
    INFO  [StreamConnectionEstablisher:3] 2017-05-12 16:15:20,462 - [Stream #604b5690-36da-11e7-aeb6-9d89ad20c2d3] Starting streaming to /
    INFO  [StreamConnectionEstablisher:1] 2017-05-12 16:15:20,478 - [Stream #604b5690-36da-11e7-aeb6-9d89ad20c2d3, ID#0] Beginning stream session with /
    INFO  [StreamConnectionEstablisher:2] 2017-05-12 16:15:20,478 - [Stream #604b5690-36da-11e7-aeb6-9d89ad20c2d3, ID#0] Beginning stream session with /
    INFO  [StreamConnectionEstablisher:3] 2017-05-12 16:15:20,478 - [Stream #604b5690-36da-11e7-aeb6-9d89ad20c2d3, ID#0] Beginning stream session with /
    INFO  [STREAM-IN-/] 2017-05-12 16:15:24,339 - [Stream #604b5690-36da-11e7-aeb6-9d89ad20c2d3 ID#0] Prepare completed. Receiving 11 files(10176549820 bytes), sending 0 files(0 bytes)
    INFO  [STREAM-IN-/] 2017-05-12 16:15:27,201 - [Stream #604b5690-36da-11e7-aeb6-9d89ad20c2d3] Session with / is complete
    INFO  [STREAM-IN-/] 2017-05-12 16:15:33,256 - [Stream #604b5690-36da-11e7-aeb6-9d89ad20c2d3] Session with / is complete
    INFO  [StreamReceiveTask:1] 2017-05-12 16:36:31,249 - [Stream #604b5690-36da-11e7-aeb6-9d89ad20c2d3] Session with / is complete
    INFO  [StreamReceiveTask:1] 2017-05-12 16:36:31,256 - [Stream #604b5690-36da-11e7-aeb6-9d89ad20c2d3] All sessions completed
    INFO  [main] 2017-05-12 16:36:31,257 - Bootstrap completed! for the tokens [1577102245397509090, -713021257351906154, 5943548853755748481, -186427637333122985, 89474807595263595, -3872409873927530770, 269282297308186556, -2090619435347582830, -7442271648674805532, 1993467991047389706, 3250292341615557960, 3680244045045170206, -6121195565829299067, 2336819841643904893, 8366041580813128754, -1539294702421999531, 5559860204752248078, 4990559483982320587, -5978802488822380342, 7738662906313460122, -8543589077123834538, 8470022885937685086, 7921538168239180973, 5167628632246463806, -8217637230111416952, 7867074371397881074, -6728907721317936873, -5403440910106158938, 417632467923200524, -5024952230859509916, -2145251677903377866, 62038536271402824]

    Transition to Up and Normal state once streaming is complete

    INFO  [main] 2017-05-12 16:36:31,348 - Node / state jump to NORMAL

    During the bootstrapping process, the new node joining the cluster has no effect on the existing data in terms of Replication Factor (RF). However, the new node will accept new writes for the token ranges acquired while existing data from the other nodes is being streamed to it. This ensures that no new writes are missed while data changes hands. In addition, it ensures that Consistency Level (CL) is respected all the time during the streaming process and even in the case of bootstrap failure. Once the bootstrapping process for the new node completes, it will begin to serve read requests (and continue to receive writes). Like the pre-existing nodes in the cluster, it too will then have an effect on the data in terms of RF and CL.

    While the bootstrapping feature can be a time saver when expanding a cluster, there are some “gotchas” that are worth noting. But before we do, we need first revisit some basics.

    Back to basics

    Cassandra uses a token system to work out which nodes will hold which partition keys for the primary replica of data. To work out where data is stored in the cluster, Cassandra will first apply a hashing function to the partition key. The generated hash is then used to calculate a token value using an algorithm; most commonly Murmur3 or RandomPartitioner.

    As seen from the log snippets, when a new node is added to the cluster it will calculate the tokens of the different data replicas that is to be responsible for. This process where tokens are calculated and acquired by the new node is often referred to as a range movement. i.e. token ranges are being moved between nodes. Once the range movement has completed, the node will by default begin the bootstrapping process where it streams data for the acquired tokens from other nodes.


    Range movements

    Whilst range movements may sound simple, the process can create implications with maintaining data consistency. A number of patches have been added over time to help maintain data consistency during range movements. A fairly well known issue was CASSANDRA-2434 where it was highlighted that range movements violated consistency for Apache Cassandra versions below 2.1.x using vnodes.

    A fix was added for the issue CASSANDRA-2434 to ensure range movements between nodes were consistent when using vnodes. Prior to this patch inconsistencies could be caused during bootstrapping as per the example Jeff Jirsa gave on the dev mailing list.

    Consider the case of a cluster containing three nodes A, B and D with a RF of 3. If node B was offline and a key ‘foo’ was written with CL of QUORUM, the value for key ‘foo’ would go to nodes A and D.

    At a later point in time node B is resurrected and added back into the cluster. Around the same time a node C is added to the cluster and begins bootstrapping.

    One of the tokens node C calculates and acquires during the bootstrap process is for key ‘foo’. Node B is the closest node with data for the newly acquired token and thus node C begins streaming from the neighbouring node B. This process violates the consistency guarantees of Cassandra. This is because the data on node C will be the same as node B, and both are missing the value for key ‘foo’.

    Thus, a query with a CL of QUORUM may query nodes B and C and return no data which is incorrect, despite there being data for ‘foo’ on node A. Node D previously had the correct data, but it stopped being a replica after C was inserted into the cluster.

    The above issue was solved in CASSANDRA-2434 by changing the default behaviour to always trying to perform a consistent range movement. That is, when node C is added (in the previous example), data is streamed from the correct replica it is replacing, node D. In this case all queries with CL of QUORUM for the key ‘foo’ would always return the correct value.

    The JVM option cassandra.consistent.rangemovement was added as part of this patch. The option allows consistent range movements during bootstrapping to be disabled should the user desire this behaviour. This fix is no silver bullet though, because it requires that the correct node be available for a consistent range moment during a bootstrap. This may not always be possible, and in such cases there are two options:

    1. Get the required node back online (preferred option).
    2. If the required node is unrecoverable, set JVM_OPTS="$JVM_OPTS -Dcassandra.consistent.rangemovement=false" in the file to perform inconsistent range movements when auto bootstrapping. Once bootstrapping is complete, a repair will need to be run using the following command on the node. This is to ensure the data it streamed is consistent with the rest of the replicas.
    nodetool repair -full

    Adding multiple nodes

    Another common cause of grief for users was bootstrapping multiple node simultaneously; captured in CASSANDRA-7069. Adding two new nodes simultaneously to a cluster could potentially be harmful, given the operations performed by a new node when joining. Waiting two minutes for the gossip state to propagate before adding a new node is possible, however as noted in CASSANDRA-9667, there is no coordination between nodes during token selection. For example consider that case if Node A was bootstrapped, then two minutes later Node B was bootstrapped. Node B could potentially pick token ranges already selected by Node A.

    The above issue was solved in CASSANDRA-7069 by changing the default behaviour such that adding a node would fail if another node was already bootstrapping in a cluster. Similar to CASSANDRA-2434, this behaviour could be disabled by setting the JVM option JVM_OPTS="$JVM_OPTS -Dcassandra.consistent.rangemovement=false" in the file on the bootstrapping node. This means that if cassandra.consistent.rangemovement=false is set to allow multiple nodes to bootstrap, the cluster runs the risk of violating consistency guarantees because of CASSANDRA-2434.

    Changes made by CASSANDRA-7069 mean that the default behaviour forces a user to add a single node at a time to expand the cluster. This is the safest way of adding nodes to expand a cluster and ensure that the correct amount of data is streamed between nodes.

    Data streaming

    To further add to the confusion there is a misconception about what the auto_bootstrap property does in relation to a node being added to the cluster. Despite its name, this property controls the data streaming step only in the bootstrap process. The boolean property is by default set to true. When set to true, the data streaming step will be performed during the bootstrap process.

    Setting auto_bootstrap to false when bootstrapping a new node exposes the cluster to huge inconsistencies. This is because all the other steps in the process are carried out but no data is streamed to the node. Hence, the node would be in the UN state without having any data for the token ranges it has been allocated! Furthermore, the new node without data will be serving reads and nodes that previously owned the tokens will no longer be serving reads. Effectively, the token ranges for that replica would be replaced with no data.

    It is worth noting that the other danger to using auto_bootstrap set to false is no IP address collision check occurs. As per CASSANDRA-10134, if a new node has auto_bootstrap set to false and has the same address as an existing down node, the new node will take over the token range of the old node. No error is thrown, only a warning messages such as the following one below is written to the logs of the other nodes in the cluster. At the time of writing this post, the fix for this issue only appears in Apache Cassandra version 3.6 and above.

    WARN  [GossipStage:1] 2017-05-19 17:35:10,994 - Changing /'s host ID from 1938db5d-5f23-46e8-921c-edde18e9c829 to c30fbbb8-07ae-412c-baea-90865856104e

    The behaviour of auto_bootstrap: false can lead to data inconsistencies in the following way. Consider the case of a cluster containing three nodes A, B and D with a RF of 3. If node B was offline and a key ‘foo’ was written with CL of QUORUM, the value for key ‘foo’ would go to nodes A and D. In this scenario Node D is the owner of the token relating to the key ‘foo’.

    At a later point in time node B is resurrected and added back into the cluster. Around the same time a node C is added to the cluster with auto_bootstrap set to false and begins the joining process.

    One of the tokens node C calculates and acquires during the bootstrap process is for key ‘foo’. Now node D is no longer the owner and hence its data for the key ‘foo’ will no longer be used during reads/writes. This process causes inconsistencies in Cassandra because both nodes B and C contain no data for key ‘foo’.

    Thus, a query with a CL of QUORUM may query nodes B and C and return no data which is incorrect, despite there being data for ‘foo’ on node A. Node D previously had data, but it stopped being a replica after C was inserted.

    This confusing behaviour is one of the reasons why if you look into the cassandra.yaml file you will notice that the auto_bootstrap configuration property is missing. Exposure of the property in the cassandra.yaml was short lived, as it was removed via CASSANDRA-2447 in version 1.0.0. As a result, the property is hidden and its default value of true means that new nodes will stream data when they join the cluster.

    Adding a replacement node

    So far we have examined various options that control the bootstrapping default behaviour when a new node is added to a cluster. Adding a new node is just one case where bootstrapping is performed, what about the case of replacing a node in the cluster if one goes down?

    Should an existing node go down and needs to be replaced, the JVM option cassandra.replace_address can be used. Note that this option is only available for Apache Cassandra versions 2.x.x and higher. This feature has been around for a while and blogged about by other users in the past.

    As the name suggests, it effectively replaces a down or dead node in the cluster with a new node. It is because of this that replace address option should only be used if the node is in a Down and Normal state (represented by DN in the nodetool status). Furthermore, there are no range movements that occur when using this feature, the new replacement node will simply inherit the old dead node’s token ranges. This is simpler than decommissioning the dead node and bootstrapping a fresh one, which would involve two range movements and two streaming phases. Yuck! To use the option, simply add JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=<IP_ADDRESS>" to the file of the new node that will be replacing the old node. Where <IP_ADDRESS> is the IP address of the node to be replaced.

    Once the node completes bootstrapping and joins the cluster, the JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=<IP_ADDRESS>" option must be removed from the file or the node will fail to start on a restart. This is a short coming of the cassandra.replace_address feature. Many operators will typically be worried about a dead node being replaced and as a result forget to update the file after the job is complete. It was for this reason that CASSANDRA-7356 was raised and resulted in a new option being added; cassandra.replace_address_first_boot. This option works once when Cassandra is first started and the replacement node inserted into the cluster. After that, the option is ignored for all subsequent restarts. It works in the same way as its predecessor; simply add JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address_first_boot=<IP_ADDRESS>" to the and the new node is ready to be inserted.

    Hang on! What about adding a replacement a seed node?

    Ok, so you need to replace a seed node. Seed nodes are just like every other node in the cluster. As per the Apache Cassandra documentation, the only difference being seed nodes are the go to node when a new node joins the cluster.

    There are a few extra steps to replace a seed node and bootstrap a new one in its place. Before adding the replacement seed node, the IP address of the seed node will need to be removed from the seed_provider list in the cassandra.yaml file and replaced with another node in the cluster. This needs to be done for all the nodes in the cluster. Naturally, a rolling restart will need to be performed for the changes to take effect. Once the change is complete the replacement node can be inserted as described in the previous section of this post.

    What to do after it completes successfully

    Once your node has successfully completed the bootstrapping process, it will transition to Up and Normal state (represented by UN in the nodetool status) to indicate it is now part of the cluster. At this point it is time to cleanup the nodes on your cluster. Yes, your nodes are dirty and need to be cleaned. “Why?” you ask, well the reason is the data that has been acquired by the newly added node still remains on the nodes that previously owned it. Whilst the nodes that previously owned the data have streamed it to the new node and relinquished the associated tokens, the data that was streamed still remains on the original nodes. This “orphaned” data is consuming valuable disk space, and in the cases large data sets; probably consuming a significant amount.

    However, before running off to the console to remove the orphaned data from the nodes, make sure it is done as a last step in a cluster expansion. If the expansion of the cluster requires only one node to be added, perform the cleanup after the node has successfully completed bootstrapping and joined the cluster. If the expansion requires three nodes to be added, perform the cleanup after all three nodes have successfully completed bootstrapping and joined the cluster. This is because the cleanup will need to be executed on all nodes in the cluster, except for the last node that was added to the cluster. The last node added to the cluster will contain only the data it needed for the tokens acquired, where as other nodes may contain data for tokens they no longer have. It is still ok to run cleanup on the last node, it will likely return immediately after it is called.

    The cleanup can be executed on each node using the following command.

    nodetool cleanup -j <COMPACTION_SLOTS>

    Where <COMPACTION_SLOTS> is the number of compaction slots to use for cleanup. By default this is 2. If set to 0 it will use use all available compaction threads.

    It is probably worth limiting the number of compaction slots used by cleanup otherwise it could potentially block compactions.

    Help! It failed

    The bootstrap process for a joining node can fail. Bootstrapping will put extra load on the network so should bootstrap fail, you could try tweaking the streaming_socket_timeout_in_ms. Set streaming_socket_timeout_in_ms in the cassandra.yaml file to 24 hours (60 * 60 * 24 * 1000 = 86,400,000ms). Having a socket timeout set is crucial for catching streams that hang and reporting them via an exception in the logs as per CASSANDRA-11286.

    If the bootstrap process fails in Cassandra version 2.1.x, the process will need to be restarted all over again. This can be done using the following steps.

    1. Stop Cassandra on the node.
    2. Delete all files and directories from the data, commitlog and save_cache directories but leave the directories there.
    3. Wait about two minutes.
    4. Start Cassandra on the node.

    If the bootstrap process fails in Cassandra 2.2.x, the process can be easily be resumed using the following command thanks to CASSANDRA-8942.

    nodetool bootstrap resume

    Testing the theory

    We have gone through a lot of theory in this post, so I thought it would be good to test some of it out to demonstrate what can happen when bootstrapping multiple nodes at the same time.


    In my test I used a three node local cluster running Apache Cassandra 2.1.14 which was created with the ccm tool. Each node was configured to use vnodes; specifically num_tokens was set to 32 in the cassandra.yaml file. The cluster was loaded with around 20 GB of data generated from the killrweather dataset. Data loading was performed in batches using cdm. Prior to starting the test the cluster looked like this.

    $ ccm node1 nodetool status
    Datacenter: datacenter1
    |/ State=Normal/Leaving/Joining/Moving
    --  Address    Load       Tokens  Owns (effective)  Host ID                               Rack
    UN  19.19 GB   32      29.1%             cfb50e13-52a4-4821-bca2-4dba6061d38a  rack1
    UN  9.55 GB    32      37.4%             5176598f-bbab-4165-8130-e33e39017f7e  rack1
    UN  19.22 GB   32      33.5%             d261faaf-628f-4b86-b60b-3825ed552aba  rack1

    It was not the most well balanced cluster, however it was good enough for testing. It should be noted that the node with IP address was set to be the only seed node in the cluster. Taking a quick peak at the keyspace configuration in using CQLSH and we can see that it was using replication_factor: 1 i.e. RF = 1.

    cqlsh> describe killrweather
    CREATE KEYSPACE killrweather WITH replication ={'class': 'SimpleStrategy', 'replication_factor': '1'}  AND durable_writes =true;

    Adding a new node

    A new node (node4) was added to the cluster.

    $ ccm node4 start

    After a minute or so node4 was in the UJ state and began the bootstrap process.

    $ ccm node1 nodetool status
    Datacenter: datacenter1
    |/ State=Normal/Leaving/Joining/Moving
    --  Address    Load       Tokens  Owns (effective)  Host ID                               Rack
    UN  19.19 GB   32      29.1%             cfb50e13-52a4-4821-bca2-4dba6061d38a  rack1
    UN  9.55 GB    32      37.4%             5176598f-bbab-4165-8130-e33e39017f7e  rack1
    UN  19.22 GB   32      33.5%             d261faaf-628f-4b86-b60b-3825ed552aba  rack1
    UJ  14.44 KB   32      ?                 ae0a26a6-fab5-4cab-a189-697818be3c95  rack1

    It was observed that node4 had started streaming data from node1 (IP address and node2 (IP address

    $ ccm node4 nodetool netstats
    Mode: JOINING
    Bootstrap f4e54a00-36d9-11e7-b18e-9d89ad20c2d3
            Receiving 9 files, 10258729018 bytes total. Already received 2 files, 459059994 bytes total
                .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-3-Data.db 452316846/452316846 bytes(100%) received from idx:0/
                .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-2-Data.db 6743148/6743148 bytes(100%) received from idx:0/
            Receiving 11 files, 10176549820 bytes total. Already received 1 files, 55948069 bytes total
                .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-1-Data.db 55948069/55948069 bytes(100%) received from idx:0/
    Read Repair Statistics:
    Attempted: 0
    Mismatch (Blocking): 0
    Mismatch (Background): 0
    Pool Name                    Active   Pending      Completed
    Commands                        n/a         0              6
    Responses                       n/a         0            471

    Adding another new node

    A few minutes later another new node (node5) was added to the cluster. To add this node to the cluster while node4 was bootstrapping the JVM option JVM_OPTS="$JVM_OPTS -Dcassandra.consistent.rangemovement=false" was added to the node’s file. The node was then started.

    $ ccm node5 start

    After about a minute node5 was in the UJ state and it too began the bootstrap process.

    $ ccm node1 nodetool status
    Datacenter: datacenter1
    |/ State=Normal/Leaving/Joining/Moving
    --  Address    Load       Tokens  Owns (effective)  Host ID                               Rack
    UN  19.19 GB   32      29.1%             cfb50e13-52a4-4821-bca2-4dba6061d38a  rack1
    UN  9.55 GB    32      37.4%             5176598f-bbab-4165-8130-e33e39017f7e  rack1
    UN  19.22 GB   32      33.5%             d261faaf-628f-4b86-b60b-3825ed552aba  rack1
    UJ  106.52 KB  32      ?                 ae0a26a6-fab5-4cab-a189-697818be3c95  rack1
    UJ  14.43 KB   32      ?                 a71ed178-f353-42ec-82c8-d2b03967753a  rack1

    It was observed that node5 had started streaming data from node2 as well; the same node that node4 was streaming data from.

    $ ccm node5 nodetool netstats
    Mode: JOINING
    Bootstrap 604b5690-36da-11e7-aeb6-9d89ad20c2d3
            Receiving 11 files, 10176549820 bytes total. Already received 1 files, 55948069 bytes total
                .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-1-Data.db 55948069/55948069 bytes(100%) received from idx:0/
    Read Repair Statistics:
    Attempted: 0
    Mismatch (Blocking): 0
    Mismatch (Background): 0
    Pool Name                    Active   Pending      Completed
    Commands                        n/a         0              8
    Responses                       n/a         0            255

    The interesting point to note when looking at the netstats was that both node4 and node5 were each streaming a Data.db file exactly 55948069 bytes from node2.

    Data streaming much

    It had appeared that both node4 and node5 were streaming the same data from node2. This continued as the bootstrapping process progressed; the size of the files being streamed from node2 were the same for both node4 and node5. Checking the netstats on node4 produced the following.

    $ ccm node4 nodetool netstats
    Bootstrap f4e54a00-36d9-11e7-b18e-9d89ad20c2d3
            Receiving 9 files, 10258729018 bytes total. Already received 6 files, 10112487796 bytes total
                .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-13-Data.db 1788940555/1788940555 bytes(100%) received from idx:0/
                .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-5-Data.db 7384377358/7384377358 bytes(100%) received from idx:0/
                .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-12-Data.db 27960312/27960312 bytes(100%) received from idx:0/
                .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-3-Data.db 452316846/452316846 bytes(100%) received from idx:0/
                .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-11-Data.db 452149577/452149577 bytes(100%) received from idx:0/
                .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-2-Data.db 6743148/6743148 bytes(100%) received from idx:0/
            Receiving 11 files, 10176549820 bytes total. Already received 10 files, 10162463079 bytes total
                .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-1-Data.db 55948069/55948069 bytes(100%) received from idx:0/
                .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-9-Data.db 55590043/55590043 bytes(100%) received from idx:0/
                .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-6-Data.db 901588743/901588743 bytes(100%) received from idx:0/
                .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-15-Data.db 14081154/14081154 bytes(100%) received from idx:0/
                .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-16-Data.db 1450179/1450179 bytes(100%) received from idx:0/
                .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-8-Data.db 901334951/901334951 bytes(100%) received from idx:0/
                .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-10-Data.db 3622476547/3622476547 bytes(100%) received from idx:0/
                .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-17-Data.db 56277615/56277615 bytes(100%) received from idx:0/
                .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-4-Data.db 3651310715/3651310715 bytes(100%) received from idx:0/
                .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-7-Data.db 902405063/902405063 bytes(100%) received from idx:0/
    Read Repair Statistics:
    Attempted: 0
    Mismatch (Blocking): 0
    Mismatch (Background): 0
    Pool Name                    Active   Pending      Completed
    Commands                        n/a         0              6
    Responses                       n/a         0           4536

    Then checking netstats on node5 produced the following.

    $ ccm node5 nodetool netstats
    Mode: JOINING
    Bootstrap 604b5690-36da-11e7-aeb6-9d89ad20c2d3
            Receiving 11 files, 10176549820 bytes total. Already received 9 files, 10106185464 bytes total
                .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-2-Data.db 3651310715/3651310715 bytes(100%) received from idx:0/
                .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-1-Data.db 55948069/55948069 bytes(100%) received from idx:0/
                .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-9-Data.db 1450179/1450179 bytes(100%) received from idx:0/
                .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-3-Data.db 901588743/901588743 bytes(100%) received from idx:0/
                .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-6-Data.db 55590043/55590043 bytes(100%) received from idx:0/
                .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-4-Data.db 902405063/902405063 bytes(100%) received from idx:0/
                .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-8-Data.db 14081154/14081154 bytes(100%) received from idx:0/
                .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-5-Data.db 901334951/901334951 bytes(100%) received from idx:0/
                .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-7-Data.db 3622476547/3622476547 bytes(100%) received from idx:0/
    Read Repair Statistics:
    Attempted: 0
    Mismatch (Blocking): 0
    Mismatch (Background): 0
    Pool Name                    Active   Pending      Completed
    Commands                        n/a         0              8
    Responses                       n/a         0           4383

    To be absolutely sure about what was being observed, I ran a command to order the netstats output by file size for both node4 and node5.

    $ for file_size in$(ccm node4 nodetool netstats  | grep '(100%)\ received' | grep '' | tr -s ' ' | cut -d' ' -f3 | cut -d'/' -f1 | sort -g); do ccm node4 nodetool netstats | grep ${file_size} | tr -s ' '; done
     .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-16-Data.db 1450179/1450179 bytes(100%) received from idx:0/
     .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-15-Data.db 14081154/14081154 bytes(100%) received from idx:0/
     .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-9-Data.db 55590043/55590043 bytes(100%) received from idx:0/
     .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-1-Data.db 55948069/55948069 bytes(100%) received from idx:0/
     .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-17-Data.db 56277615/56277615 bytes(100%) received from idx:0/
     .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-8-Data.db 901334951/901334951 bytes(100%) received from idx:0/
     .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-6-Data.db 901588743/901588743 bytes(100%) received from idx:0/
     .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-7-Data.db 902405063/902405063 bytes(100%) received from idx:0/
     .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-10-Data.db 3622476547/3622476547 bytes(100%) received from idx:0/
     .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-4-Data.db 3651310715/3651310715 bytes(100%) received from idx:0/
    $ for file_size in$(ccm node5 nodetool netstats  | grep '(100%)\ received' | grep '' | tr -s ' ' | cut -d' ' -f3 | cut -d'/' -f1 | sort -g); do ccm node5 nodetool netstats | grep ${file_size} | tr -s ' '; done
     .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-9-Data.db 1450179/1450179 bytes(100%) received from idx:0/
     .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-8-Data.db 14081154/14081154 bytes(100%) received from idx:0/
     .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-6-Data.db 55590043/55590043 bytes(100%) received from idx:0/
     .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-1-Data.db 55948069/55948069 bytes(100%) received from idx:0/
     .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-5-Data.db 901334951/901334951 bytes(100%) received from idx:0/
     .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-3-Data.db 901588743/901588743 bytes(100%) received from idx:0/
     .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-4-Data.db 902405063/902405063 bytes(100%) received from idx:0/
     .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-7-Data.db 3622476547/3622476547 bytes(100%) received from idx:0/
     .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-2-Data.db 3651310715/3651310715 bytes(100%) received from idx:0/

    With the exception of one file being streamed by node4, killrweather-raw_weather_data-tmp-ka-17-Data.db (size 56277615 bytes), node4 and node5 looked to be streaming the same data from node2. This was the first confirmation that node5 had stolen the tokens that where originally calculated by node4. Furthermore, it looked like node 4 was performing unnecessary streaming from node2. I noted down the file sizes displayed by node5’s netstats output to help track down data files on each node.

    $ ccm node5 nodetool netstats | grep '(100%)\ received' | grep '' | tr -s ' ' | cut -d' ' -f3 | cut -d'/' -f1 | sort -g > file_sizes.txt; cat file_sizes.txt

    Token and the thief

    Once both nodes had finished bootstrapping and had successfully joined the cluster it looked like this.

    $ ccm node1 nodetool status
    Datacenter: datacenter1
    |/ State=Normal/Leaving/Joining/Moving
    --  Address    Load       Tokens  Owns (effective)  Host ID                               Rack
    UN  19.19 GB   32      14.8%             cfb50e13-52a4-4821-bca2-4dba6061d38a  rack1
    UN  9.55 GB    32      22.0%             5176598f-bbab-4165-8130-e33e39017f7e  rack1
    UN  19.22 GB   32      23.6%             d261faaf-628f-4b86-b60b-3825ed552aba  rack1
    UN  19.17 GB   32      17.5%             ae0a26a6-fab5-4cab-a189-697818be3c95  rack1
    UN  9.55 GB    32      22.1%             a71ed178-f353-42ec-82c8-d2b03967753a  rack1

    Using the file sizes I captured earlier from node5 netstats, I checked the data directories of node4 and node5 to confirm both nodes contained files of those sizes.

    $ for file_size in$(cat file_sizes.txt); do ls -al .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/ | grep ${file_size}; done
    -rw-r--r--    1 anthony  staff     1450179 12 May 16:33 killrweather-raw_weather_data-ka-16-Data.db
    -rw-r--r--    1 anthony  staff    14081154 12 May 16:33 killrweather-raw_weather_data-ka-15-Data.db
    -rw-r--r--    1 anthony  staff    55590043 12 May 16:33 killrweather-raw_weather_data-ka-9-Data.db
    -rw-r--r--    1 anthony  staff    55948069 12 May 16:33 killrweather-raw_weather_data-ka-1-Data.db
    -rw-r--r--    1 anthony  staff   901334951 12 May 16:33 killrweather-raw_weather_data-ka-8-Data.db
    -rw-r--r--    1 anthony  staff   901588743 12 May 16:33 killrweather-raw_weather_data-ka-6-Data.db
    -rw-r--r--    1 anthony  staff   902405063 12 May 16:33 killrweather-raw_weather_data-ka-7-Data.db
    -rw-r--r--    1 anthony  staff  3622476547 12 May 16:33 killrweather-raw_weather_data-ka-10-Data.db
    -rw-r--r--    1 anthony  staff  3651310715 12 May 16:33 killrweather-raw_weather_data-ka-4-Data.db
    $ for file_size in$(cat file_sizes.txt); do ls -al  .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/ | grep ${file_size}; done
    -rw-r--r--    1 anthony  staff     1450179 12 May 16:36 killrweather-raw_weather_data-ka-9-Data.db
    -rw-r--r--    1 anthony  staff    14081154 12 May 16:36 killrweather-raw_weather_data-ka-8-Data.db
    -rw-r--r--    1 anthony  staff    55590043 12 May 16:36 killrweather-raw_weather_data-ka-6-Data.db
    -rw-r--r--    1 anthony  staff    55948069 12 May 16:36 killrweather-raw_weather_data-ka-1-Data.db
    -rw-r--r--    1 anthony  staff   901334951 12 May 16:36 killrweather-raw_weather_data-ka-5-Data.db
    -rw-r--r--    1 anthony  staff   901588743 12 May 16:36 killrweather-raw_weather_data-ka-3-Data.db
    -rw-r--r--    1 anthony  staff   902405063 12 May 16:36 killrweather-raw_weather_data-ka-4-Data.db
    -rw-r--r--    1 anthony  staff  3622476547 12 May 16:36 killrweather-raw_weather_data-ka-7-Data.db
    -rw-r--r--    1 anthony  staff  3651310715 12 May 16:36 killrweather-raw_weather_data-ka-2-Data.db

    So both nodes contained files of the same size. I then decided to check if the files on each node that were the same size had the same data content. This check was done by performing an MD5 check of file pairs that were the same size.

    $ BASE_DIR=...; DATA_DIR=data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d; for file_size in$(cat file_sizes.txt); do node_4_file=$(ls -al ${BASE_DIR}/node4/${DATA_DIR}/ | grep ${file_size} | tr -s ' ' | cut -d' ' -f9); node_5_file=$(ls -al ${BASE_DIR}/node5/${DATA_DIR}/ | grep ${file_size} | tr -s ' ' | cut -d' ' -f9); md5 ${BASE_DIR}/node4/${DATA_DIR}/${node_4_file}${BASE_DIR}/node5/${DATA_DIR}/${node_5_file}; echo; done
    MD5 (.../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-16-Data.db)= a9edb85f70197c7f37aa021c817de2a2
    MD5 (.../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-9-Data.db)= a9edb85f70197c7f37aa021c817de2a2
    MD5 (.../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-15-Data.db)= 975f184ae36cbab07a9c28b032532f88
    MD5 (.../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-8-Data.db)= 975f184ae36cbab07a9c28b032532f88
    MD5 (.../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-9-Data.db)= f0160cf8e7555031b6e0835951e1896a
    MD5 (.../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-6-Data.db)= f0160cf8e7555031b6e0835951e1896a
    MD5 (.../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-1-Data.db)= 7789b794bb3ef24338282d4a1a960903
    MD5 (.../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-1-Data.db)= 7789b794bb3ef24338282d4a1a960903
    MD5 (.../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-8-Data.db)= 1738695bb6b4bd237b3592e80eb785f2
    MD5 (.../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-5-Data.db)= 1738695bb6b4bd237b3592e80eb785f2
    MD5 (.../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-6-Data.db)= f7d1faa5c59a26a260038d61e4983022
    MD5 (.../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-3-Data.db)= f7d1faa5c59a26a260038d61e4983022
    MD5 (.../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-7-Data.db)= d791179432dcdbaf9a9b315178fb04c7
    MD5 (.../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-4-Data.db)= d791179432dcdbaf9a9b315178fb04c7
    MD5 (.../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-10-Data.db)= 3e6623c2f06bcd3f5caeacee1917898b
    MD5 (.../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-7-Data.db)= 3e6623c2f06bcd3f5caeacee1917898b
    MD5 (.../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-4-Data.db)= 8775f5df08882df353427753f946bf10
    MD5 (.../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-2-Data.db)= 8775f5df08882df353427753f946bf10

    Now I had absolute proof that both nodes did in fact stream the same data from node2. It did look as though that when node5 joined the cluster it had taken tokens calculated by node4. If this were the case, it would mean that the data files on node4 that are the same on node5 would no longer be needed. One way to prove that there is “orphaned” data on node4 i.e. data not associated to any of node4’s tokens, would be to run cleanup on the cluster. If there is orphaned data on node4 the cleanup would technically delete all or some of those files. Before running cleanup on the cluster, I took note of the files on node4 which were the same as the ones on node5.

    $ for file_size in$(cat file_sizes.txt); do ls -al .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/ | grep ${file_size}; | tr -s ' ' | cut -d' '  -f9; done> node4_orphaned_files.txt; cat node4_orphaned_files.txt

    I then ran a cleanup on all the nodes in the cluster.

    $ ccm node1 nodetool cleanup
    $ ccm node2 nodetool cleanup
    $ ccm node3 nodetool cleanup
    $ ccm node4 nodetool cleanup
    $ ccm node5 nodetool cleanup
    $ ccm node1 nodetool status
    Datacenter: datacenter1
    |/ State=Normal/Leaving/Joining/Moving
    --  Address    Load       Tokens  Owns (effective)  Host ID                               Rack
    UN  9.57 GB    32      14.8%             cfb50e13-52a4-4821-bca2-4dba6061d38a  rack1
    UN  138.92 KB  32      22.0%             5176598f-bbab-4165-8130-e33e39017f7e  rack1
    UN  19.22 GB   32      23.6%             d261faaf-628f-4b86-b60b-3825ed552aba  rack1
    UN  9.62 GB    32      17.5%             ae0a26a6-fab5-4cab-a189-697818be3c95  rack1
    UN  9.55 GB    32      22.1%             a71ed178-f353-42ec-82c8-d2b03967753a  rack1

    From this output it was obvious that node4 contained orphaned data. Earlier I had run a nodetool status which was just after both nodes completed bootstrapping and moved to the UN state, and prior to running cleanup. The output produced at that point showed that node4 had a Load of 19.17 GB. Now after cleanup it was showing to have a load of 9.62 GB. As a final verification, I iterated through the list of files on node4 which were the same as the ones on node5 (node4_orphaned_files.txt) and checked if they still were present on node4.

    $ for file_name in$(cat node4_orphaned_files.txt); do ls .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/${file_name}; done
    ls: .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-16-Data.db: No such file or directory
    ls: .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-15-Data.db: No such file or directory
    ls: .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-9-Data.db: No such file or directory
    ls: .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-1-Data.db: No such file or directory
    ls: .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-8-Data.db: No such file or directory
    ls: .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-6-Data.db: No such file or directory
    ls: .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-7-Data.db: No such file or directory
    ls: .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-10-Data.db: No such file or directory
    ls: .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-4-Data.db: No such file or directory

    As it can be seen the files were deleted as part of the cleanup on node4. Which means that during bootstrap node4 originally calculated tokens for that data. It then asked for a list of files that related to those tokens from node2 and began streaming them. A little while later node5 was added to the cluster while node4 was still bootstrapping. It then calculated tokens that overlapped with node4’s tokens. Node5 then asked for a list of files that related to those tokens from node2 and started streaming data for them as well. The issue here is node4 was never notified that it no longer required to stream files from node2. Hence, unnecessary resources were being consumed as a result of bootstrapping two nodes at the same time.


    Auto bootstrapping combined with vnodes is probably one of the most handy features in Cassandra. It takes the pain out of manually having to move data around ensure a continuous availability while expanding the cluster in a reliable and efficient way. There a number of knobs and levers for controlling the default behaviour of bootstrapping.

    Configuration properties

    • auto_bootstrap - controls whether data is streamed to the new node when inserted.
    • streaming_socket_timeout_in_ms - sets socket timeout for streaming operations.

    JVM options

    • cassandra.consistent.rangemovement - controls consistent range movements and multiple node bootstrapping.
    • cassandra.replace_address_first_boot=<IP_ADDRESS> - allows a down node to be replaced with a new node.

    As demonstrated by setting the JVM option cassandra.consistent.rangemovement=false the cluster runs the risk of over streaming of data and worse still, it can violate consistency. For new users to Cassandra, the safest way to add multiple nodes into a cluster is to add them one at a time. Stay tuned as I will be following up with another post on bootstrapping.

    0 0

    JOSE, the primary mechanism for securing various OAuth2/OIDC tokens, slowly but surely is becoming the main technology for securing the data in the wider contexts. JOSE, alongside COSE, will become more and more visible going forward.

    I talked about Apache CXF JOSEimplementation in this post. One of the practical aspects of this implementation is that one can apply JOSE to securing the regular HTTP payloads, with the best attempt at keeping the streaming going made by the sender side filters, with the JOSE protection of these payloads (JWS signature or JWE encryption) being able to 'stay' with the data even beyond the HTTP request-response time if needed.

    In CXF 3.1.12 I have enhanced this feature to support the signing of HTTP attachments. It depends on JWS Detached Content and Unencoded Content features which allow to integrity-protect the payload which can continue flowing to its destination in a clear form.

    Combining it with the super-flexible mechanism of processing the attachments in Apache CXF, and particularly with the newly introduced Multipart filters which let pre-process individual multipart attachment streams, helped produce the final solution.  

    Besides, as part of this effort, the optional binding of the outer HTTP headers to the secure JWS or JWE payloads has also been realized.

    Be the first in experimenting with this IMHO very cool feature, try it and provide the feedback, enjoy !

    0 0

    A one-liner to run a SSL Docker registry generating a Let’s Encrypt certificate.

    This command will create a registry proxying the Docker hub, caching the images in a registry volume.

    LetsEncrypt certificate will be auto generated and stored in the host dir as letsencrypt.json. You could also use a Docker volume to store it.

    In order for the certificate generation to work the registry needs to be accessible from the internet in port 443. After the certificate is generated that’s no longer needed.

    docker run -d -p 443:5000 --name registry \
      -v `pwd`:/etc/docker/registry/ \
      -v registry:/var/lib/registry \
      -e REGISTRY_HTTP_TLS_LETSENCRYPT_CACHEFILE=/etc/docker/registry/letsencrypt.json \
      -e \

    You can also create a config.yml in this dir and run the registry using the file instead of environment variables

    version: 0.1
          cachefile: /etc/docker/registry/letsencrypt.json

    Then run

    docker run -d -p 443:5000 --name registry \
      -v `pwd`:/etc/docker/registry/ \
      -v registry:/var/lib/registry \

    If you want to use this as a remote repository and not just for proxying, remove the proxy entry in the configuration

    0 0

    The 2016-17 Ski Season was a fun one for the Raible Family. Abbie and Jack are good enough that they can zoom down the mountain without looking back. Their preferred runs are now blacks and they're no longer intimidated by moguls. We spent most of the season skiing at Mary Jane and Winter Park, but also had some trips to Crested Butte, Steamboat, and Montana.

    Mary Jane for Trish's BirthdayFamily Ski Day at Mary Jane!

    On top of the world!

    On our way to Crested Butte (near the end of January), Stout the Syncro's front drive train started making crunching noises and we had to pull over. It was almost midnight, and we knew we couldn't fix it on the side of the road, so we called AAA and spent the night in a nearby hotel. I rode back to Denver the next day with the tow truck driver on the bumpiest ride of my life. The tow truck's tires were out of balance and it felt like things were going to fall apart at any moment. We later discovered that several things failed that night: one axle broke, a CV went out, and a wheel bearing was gone too. Trish and the kids spent the day at a hot springs in Buena Vista while I drive Trish's Tahoe back to complete the trip. While we missed a day skiing, the day we made it for was great and CB continues to be one of our favorite resorts.

    Trish and Laura on HeadwallMontanya Rum!

    James, Jenny, and Josie

    I had some awesome solo skiing adventures too: skiing in Sweden after Jfokus, skiing in Steamboat during Winter Wonder Grass, and Miller Time 2017 in Northstar, California. Miller Time is an annual event to remember a great friend, Jason Miller, who passed on way too early in October 2015.

    It was a beautiful day for skiing at Klövsjö!

    ShadowsBumps at SteamboatA beautiful day with brothers

    Miller Time 2017!

    The highlight of the season was when we traveled to Big Sky Resort in Montana. I'd never skied at Big Sky before and it's been on my bucket list for years. We stayed a week during the kids' Spring Break and experienced some of the best skiing of our lives. When I asked Jack if he had any goals for Spring Break, he said he wanted to "ski 5 double blacks". There were so many double blacks on the mountain, it was unbelievable. We had his goal accomplished by noon on Tuesday. Unfortunately, I didn't remember to use our GoPro until Wednesday. However, that was the day we skipped skiing and did zip lining, so that was a ton of fun. I left my phone and laptop at home that week and thouroughly enjoyed being disconnected.

    The video below shows our zip lining adventures, as well as some footage of our best day skiing. It was a powder day, we were on one of the first chairs, and we had fresh tracks all morning. There was an immmense amount of giggline, smiling, and hooting and hollering that morning.

    More photos of our ski season adventures can be found on Flickr.

    Stout 5.0 and Hefe 3.0

    Now that ski season is over, it's time to focus on our favorite summertime activities: VWs, horseback riding, mountain biking, and rafting. Stout 5.0 and Hefe 3.0 were released at the beginning of May and they've both been running great!

    Driving Hefe around with his rockin' stereo has been a lot of fun. You can imagine the kids giggling in the back with no seatbelts and ragtop down.

    Happy MattTrish is driving Hefe - watch out!

    #porschebusSo many windows

    Hefe won best in class (Custom Bus 49-67) at VWs on the Green last weekend. Two years in a row baby!

    1st Place! Good job Hefe!!

    More photos on Flickr ? Stout 5.0 and Hefe 3.0

    Speaking of winning, Abbie and Tucker have been tearing it up at the Colorado Horse Park. They've won 24 ribbons over the last five months!

    24 ribbons over the course of five months!

    Ski season is over, summer is almost upon us, and both our VWs are in tip-top shape. I really can't ask for anything more!

    0 0

    As a Developer Advocate at Okta, I'm expected to travel up to 25% per month to speak at conferences and meetups. This May was more like 50%! I had opportunities to contribute to a number of cool conferences in exotic cities that I was eager to accept.

    My adventure began on Monday, May 8 when I flew to Amsterdam to speak at the J-Spring conference. It was the first time the NLJUG hosted this conference in several years. I marveled at the venue and especially liked the outdoor area it offered during breaks. The walk from/to the train station was pretty nice too.

    J-Spring Outdoor AreaAmsterdam Bike Paths

    I spoke about Microservices for the Masses with Spring Boot, JHipster, and JWT. Feedback I received mentioned it was a bit too fast and I crammed too much into the 50-minute time slot. I do tend to mention everything I know about topics when I speak, so I apologize for trying to cram too much in.

    After J-Spring, I flew to London to speak at Devoxx UK. I arrived just in time to catch the speaker's dinner and had fun seeing and catching up with old friends from the conference circuit.

    View from Room 404 in LondonDevoxx UK Venue

    Thursday morning, I had an Angular workshop and did my microservices presentation in the afternoon. Friday, I had an early morning talk on Front End Development for Back End Developers. You can find all my presentations below.

    I rushed straight from my last talk on Friday to the airport to catch a flight to Boston for the weekend. In Boston, we celebrated Trish's brother's 50th birthday, Mother's Day, and had a blast with friends and family.

    Happy Mother's Day!

    The following Monday, I hopped on a plane to return to Europe with Krakow (for GeeCON) as my destination. Three flights later and I arrived in time to take a nice stroll around the city, enjoying the greenery.


    At GeeCON, I spoke about how to build a progressive web app with Ionic, Angular, and Spring Boot. Half of my talk was live coding and I almost got all my demos working. Deploying to Cloud Foundry and my phone was the final step, and due to Xcode updating, that demo failed. I wrote a tutorial about Ionic for the Okta developer blog that has everything (and more!) that I showed in my demo.

    I had to head straight to the airport after finishing my talk, this time heading for Spring I/O in Barcelona. Barcelona has always been on Trish's bucket list, so I easily talked her into joining me. At Spring I/O, I did a workshop on developing with Spring Boot and Angular, followed by my Front End Development for Back End Developers talk. There weren't that many talks on front-end development, so I felt privileged to be one of the few talking about UI development.

    I also enjoyed Deepu's talk on JHipster and Sebastien's talk on Keycloak. It was the first time I'd met these great guys in person, so that was a lot of fun.

    On Friday, Trish and I hit some of the sites in Barcelona and had a wonderful time. The weather was beautiful, the architecture was amazing, and the experience was awesome.

    Amazing Architecture in BarcelonaBarcelona

    Barcelona Fountains

    HappinessSagrada Familia

    Sagrada Familia

    More photos on Flickr → European Speaking Tour - May 2017

    Thanks to the organizers of each conference for allowing me to speak and for covering my travel expenses. My company doesn't pay for overseas conferences (yet!), but they do pay me while I'm there, so that's nice. To everyone that attended my sessions - thank you! I really appreciate the feedback and will do my best to improve future talks. If you have additional feedback, feel free to contact me.

    In the meantime, keep an eye on the Okta developer blog. I've been writing a lot of articles lately and there's more to come in the pipeline! Here's a few that've been published in the last month.

    0 0

    No pun intended

    The JDK 8u131 has backported a nice feature in JDK 9, which is the ability of the JVM to detect how much memory is available when running inside a Docker container.

    I have talked multiple times about the problems running a JVM inside a container, how it will default in most cases to a max heap of 1/4 of the host memory, not the container.

    For example in my machine

    $ docker run -m 100MB openjdk:8u121 java -XshowSettings:vm -version
    VM settings:
        Max. Heap Size (Estimated): 444.50M
        Ergonomics Machine Class: server
        Using VM: OpenJDK 64-Bit Server VM

    Wait, WAT? I set a container memory of 100MB and my JVM sets a max heap of 444M ? It is very likely that it is going to cause the Kernel to kill my JVM at some point.

    Let’s try the JDK 8u131 with the experimental option -XX:+UseCGroupMemoryLimitForHeap

    $ docker run -m 100MB openjdk:8u131 java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XshowSettings:vm -version
    VM settings:
        Max. Heap Size (Estimated): 44.50M
        Ergonomics Machine Class: server
        Using VM: OpenJDK 64-Bit Server VM

    Ok this makes more sense, the JVM was able to detect the container has only 1000MB and set the max heap to 44M.

    Let’s try in a bigger container

    $ docker run -m 1GB openjdk:8u131 java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XshowSettings:vm -version
    VM settings:
        Max. Heap Size (Estimated): 228.00M
        Ergonomics Machine Class: server
        Using VM: OpenJDK 64-Bit Server VM

    Mmm, now the container has 1GB but JVM is only using 228M as max heap. Can we optimize this even more, given that nothing else other than the JVM is running in the container? Yes we can!

    $ docker run -m 1GB openjdk:8u131 java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=1 -XshowSettings:vm -version
    VM settings:
        Max. Heap Size (Estimated): 910.50M
        Ergonomics Machine Class: server
        Using VM: OpenJDK 64-Bit Server VM

    Using -XX:MaxRAMFraction we are telling the JVM to use available memory/MaxRAMFraction as max heap. Using -XX:MaxRAMFraction=1 we are using almost all the available memory as max heap.

    0 0

    This is the lightning talk I gave this evening at ApacheCon: 

    ApacheCon is a high point of my year, every year, going back to March of 2000.

    In late 1999, Ken Coar told me I should submit a talk for ApacheCon. Astonishingly, my talks were all accepted, and I found myself in Orlando speaking in front of a few hundred people who thought I knew what I was talking about. I have since made a career out of that particular game.

    This is the 28th ApacheCon since the creation of the Apache Software Foundation. 29 if you count the event in 1998 before there was a Foundation. I don’t count it, because I missed it. I also missed the ApacheCon in Sinsheim, Germany, in 2012, for which I will never forgive my boss at the time. But I *think* I have been to more ApacheCons than anyone else. 27 of them.

    I love being on stage. With hundreds of people looking at me, hanging on my every word, believing I know what I’m talking about.

    But there’s other reasons I love ApacheCon. It’s the place I go to see some of my oldest friends – many of whom I first met at ApacheCon, including some new ones this week.

    I love ApacheCon because it shows me that I’m not alone. As C. S. Lewis said, we read to know that we’re not alone. Except that he didn’t say it. It’s actually just a quote from a movie about him.

    I love ApacheCon because I love Apache. And Hawaiian shirts. Shane’s lightning talks are another high point of my year, because they are both entertaining and informational. Except I hear he’s not giving one this year.

    I love ApacheCon because of the passion that I see in the people that attend. People that love Apache, and also love solving actual real world problems. The sessions here at ApacheCon range from the esoteric and philosophical to the deeply practical, but at the heart of each one is a desire to solve problems in the real world. To scratch your own itch, as the saying goes.

    I love ApacheCon because of our sponsors. Talking to sponsors about why they are here at ApacheCon has the effect of re-centering us. Sure, open source is about having fun and tinkering, but it’s also about solving problems for real people that rely on us. People that depend on Apache because we have a reputation for vendor-neutral, high-quality software which is sustainable because of those esoteric philosophies that we cling to even in the face of practical realities.

    I love ApacheCon because of the time I’ve put into it. I’ve worked on ApacheCon for 18 years now. I often refer to ApacheCon as my life’s work. I spend hundreds of hours on it, and so do many other people, including our amazing producers, our numerous volunteers, our tireless Infra contractors, our beloved Melissa, and our supportive board of directors. ApacheCon is my sweat and tears, literally and figuratively. It’s older than two of my kids, and the oldest kid grew up knowing that Dad loves ApacheCon. The wall in my office is covered with ApacheCon attendee badges – 27 of them. And ApacheCon has become a part of my identity.

    So as we look forward to the next ApacheCon (details coming very, very soon, I hope) we need to figure out what *you* want ApacheCon to be, and make it that, rather than doing it just because it’s what we do, and what we’ve always done. ApacheCon is about building community, more than it’s about anything else, and that’s really why I love ApacheCon. I love seeing communities come together around a common goal, and believing that I was a catalyst in making that happen.

    So, thank you so much for coming to ApacheCon, my friends. I hope you’ll come again, and I hope that you’ll come to love it as much as I do. But that might not be possible.

    0 0

    XPath 2.0atomization as a concept, as applicable to XML Schema 1.1 validation is worth looking at. I would like to ponder on this in this post, with a brief example.

    Consider the following brief XML Schema 1.1 validation example.

    XSD 1.1 document:

    <?xml version="1.0"?>
    <xs:schema xmlns:xs="">
       <xs:element name="X">
                <xs:element name="a" type="xs:integer"/>
                <xs:element name="b" type="xs:integer"/>
             <xs:assert test="a gt b"/>

    XML document that is fed to the XML Schema 1.1 validator, when validating with the above XSD document:

    <?xml version="1.0"?>

    This XML document fails validation, because "a" is less than "b". Now what is XPath 2.0 atomization, as for in this example that I want to talk about?

    Since the XML document has been validated with the mentioned XSD, while building the XPath tree to apply <assert>, the nodes of the tree are annotated (or decorated) with the XSD types as mentioned in the XSD document. Therefore, the <assert> XPath 2.0 expression "a gt b", comes with runtime availability of the corresponding XSD types on <assert> tree nodes for "a" and "b". In XPath 2.0 terms, the <assert> tree nodes for "a" and "b" have been atomized. The Schema aware XPath 2.0 in action, is same in <assert> as in the generic Schema aware XPath 2.0 processor.

    The XSD 1.1 processor that I've used for this example is located at

    0 0

    Event Report, ApacheCon North America 2017

    May 15-19, 2017

    (This is an abridged version of the report I sent to my manager.)

    Last week I attended ApacheCon North America in Miami. I am the conference chair of ApacheCon, and have been for on and off for  about 15 years. Red Hat has been a sponsor of ApacheCon almost every single time since we started doing it 17 years ago. In addition to being deeply involved in specific projects, such as Tomcat, ActiveMQ, and Mesos, we are tangentially involved in many of the other projects at the Apache Software Foundation.

    Presentations from ApacheCon may be found at (Yes, that’s the Linux Foundation’s YouTube channel – this ApacheCon was produced by the LF events team.)

    I’d like to draw specific attention to Alan Gates, at Hortonworks, who has developed a course to train people at the company in how to work with upstream projects. He did this because, as the company expanded from a small group of founders who deeply understood open source, to thousands of employees who kinda sorta got it, but not always.

    Also of great interest was the keynote by Sandra Matz about what your social media profile tells the world about you. It’s worth watching all the way to the end, as she doesn’t just talk about the reasons to be terrified of the interwebs, but also about how this kind of analysis can actually be used for good rather than evil.

    0 0

    I just noticed in my linked in profile (, that Noah Mendelsohn has endorsed me for XML.

    Here's the snapshot from my linked in profile.

    Thanks Noah :)

    0 0

    ApacheCon North America 2017 sponsor Cloudera talk about why they like to sponsor the event.

    0 0
  • 06/05/17--16:21: Nick Kew: Sultana
  • (The title derives from here).

    I was as surprised as anyone when our prime minister called a surprise election.  OK, with Libdems knocked out in 2015 and Labour tearing themselves apart, she has no opposition in most of the country: she’ll walk it, right?  But just after setting the clock ticking on brexit???   Good grief, how can we afford the time for this nonsense?  Her policy platform looked like the progeny of an unlikely match of Farage and Miliband, with a touch more of Blairite authoritarianism that either of the main parents would seem likely to favour.

    To state my own prior position, I was a strong supporter of Mrs Thatcher in my youth, but have become much-disillusioned with her successors, as browsing this blog (e.g. here) will reveal.  I had hoped that the Libdems might come to the election with a positive programme I could support despite inevitable elements of gratuitous Political Correctness and the Loony Left, but they were quick to disappoint.  Once again, I say None of the Above.

    The justification seemed dodgy from the start, raising a strawman argument about being frustrated by … well, in fact, an exceptionally supine parliament.   A couple of outright lies put my back up somewhat.  But anyway, the Chattering Classes soon came up with some ideas: she wanted a personal mandate; she needed a big majority to stand up to the loony fringe of her own party.  Really?

    I live in a very marginal constituency, so I expected to be on the receiving end of some campaigning.  The first I received on the doormat some weeks ago was a large glossy from which a mugshot of Jeremy Corbyn stared up at me.  Interesting: Labour have got into gear commendably fast?  Nope, this was Tory literature, featuring a bogeyman as its most important message.

    When the (less-glossy) actual Labour leaflet followed, the only mugshot in it was the candidate himself.  And a set of policies that read like a checklist of opposing everything the Tories are trying to do right.  Ugh.  No mention of Corbyn: is this candidate trying to dissociate himself from his own leader?

    A second Tory letter – this time in an envelope – calls for a mandate not for my candidate, nor for the Party, but for Mrs May herself.  Well, sorry, I can’t vote for that.  Even if I wanted to live in Maidenhead (her constituency), I’ll never be able to afford it, so I don’t get the chance to vote either for or against her.  But the message is becoming clearer than ever: we are to dispense with Parliament, relegate them to something more like a US-style electoral college, and crown our Supreme Leader.  This cult of personality is not entirely new: perhaps we should be glad that she’s being more open about it than in the past?  But coupled with her authoritarian leanings and secrecy over her agenda beyond the coronation, it scares me.

    No more leaflets until last Friday, when a sudden flurry brought one each from the Libdems, UKIP, and an Independent, plus three more from the Tories for a total of five from them (good grief)!  Only the Greens missing (perhaps they practice what they preach?), and sadly our Green party is solidly Loony Left.  The Independent candidate actually has an anti-party platform I could strongly support, but sadly falls down on other issues.  And neither the Libdem nor UKIP feature their respective party leaders, so maybe I was being unduly cynical about Labour doing likewise.

    But I’m getting ahead of myself.   Surprisingly, her bogeyman doesn’t seem to be doing the job of annihilating himself.  Indeed, Corbyn is looking the most statesmanlike of a dreadful bunch, and his own party have suppressed their hatred for him and moved from attacking him to ‘clarifying’ what he says.  The “strong and stable” and “coalition of chaos” slogans have come back to bite her as it becomes painfully clear she herself is more chaos than strength, and the latest image of Corbyn “naked and alone“(!!!) with all those Eurocrats sounds almost like panic.  It’s obviously nonsense: brexit negotiations will be conducted by Sir Humphrey’s civil servants regardless of who wins the election.  In the still-inconceivable event of Labour beating the tories, I expect their political master would be Sir Kier Starmer, KCB, QC, not Corbyn himself.

    So Corbyn has momentum.  How far can it take him?  Not into government, but perhaps far enough to upset the master plan.  We need a bigger rallying point than that mugshot.  What do people respond to, fast?  Not any new promises: messing with the manifesto is just more egg on the face.  It’s got to be a real threat.  Big enough to grab the headlines and the national conversation.  And preferably focus attention on matters where We Beat Them in public trust.

    Where can we find such a threat?  Given the tight timescale, we’re never going to make it with a foreign power.  But there are a fair few alienated idiots in Blighty, susceptible to being inspired by heroes like the biblical Samson.  We’re told our security forces have thwarted no fewer than five terrorist attacks in two months between the Westminster Bridge attack (March 22nd) and the Manchester one (May 25th – being more than a month into the election campaign).  That’s more than one a fortnight, so it’s unlikely to be long before a next attempt.  If one of those gets through, we have our threat and out enemy to rally against, and of course security is precisely where both the parties and their leaders individually are very clearly differentiated!

    With that in mind, it seems an extraordinarily convenient coincidence that Manchester happened when it did: surely the security theatre of raising the threat level and deploying troops on the streets would kill that momentum and distract the media from the manifesto fiasco?  Against all expectations, it didn’t!  Then we had London Bridge, and this time a firm No Nonsense message: playing directly to traditional strengths.

    Of course suggesting a connection is deep into conspiracy theory.  But for the security forces – who routinely prevent terrorist attacks – to have failed twice in such quick succession – is extremely unlikely to be purely random.  Did someone quietly send 007 on a wild goose chase – like for instance looking for Russian influence in the election – and leave Clouseau in charge back home?  No, that’s a bit far-fetched.  A botched information system update disrupting communication among anti-terrorist forces would make far more sense.  And since all the people concerned work on a need-to-know basis and only see small parts of the overall systems, no individual would actually know what was going on!

    And just to add icing to the conspiracy, what if the botch messed with third-party systems that must access the anti-terrorist information system, like an airline’s passenger information?  What unlikely account might the airline be able to give of it if they were unable to operate?  No, ignore that, it’s too far-fetched: BA is much more likely to have been hit by their own botch, perhaps with the aid of the big thunderstorms we had on the Friday night.

    0 0

    Suresh Marru talks about Apache Airvata, and his presentations at ApacheCon North America, Miami, May 2017

    0 0

    Here we talk to Sudip Chakrabarti about his upcoming ApacheCon NA Miami keynote, the  Future of IoT,  A VC perspective

    His complete keynote presentation can be found at the following link


    0 0
  • 06/09/17--13:52: Nick Kew: Schadenfreude
  • So, which is more satisfying in today’s election results?  A bloody nose for Mrs MegaloMayniac?  Or a kick up the backside for the Labour party Establishment who loathe Corbyn as much as they hate democracy, and have spent a year and a half in civil war?  All without delivering the other hypothetically-possible disaster of a Corbyn government.

    The only fly in the ointment is what the Coalition of Chaos that now looks likely may do to Northern Ireland.  The DUP will want their price, and the Tories’ desperation will surely strengthen the hand of the more extreme elements in the DUP.  Talk about setting a match to a powder keg!

    0 0

    Yes, we just interviewed Carlos a few weeks ago. But while at ApacheCon, he had a few more things to say. Here’s Carlos talking about OpenWhisk some more, filling in some of the details. So, go listen to that other one, and then come listen to this:


    0 0

    Monday, April 24th marked the third anniversary of my spinal cord injury. It seems like the injury took place so long ago now and yet it has only been three years. I now feel like I am emerging from the other side and I believe I have finally found peace with this whole ordeal.

    As I look back at all the photos and videos Janene has taken over this three year period, read through the Caring Bridge posts and my blog posts, the progress I have made is pretty amazing to me. Believe it or not, I actually have a lot of gratitude for the fact that this experience happened to me. Yes, I just said that I am thankful for the experience. I did not arrive at this place easily or lightly, so stop and consider that statement for a moment. After three years of pondering every aspect of this entire situation, I feel that I am a better person for it in many ways. This whole experience forced me to get myself in order and I'm now a better person for it.

    From the beginning of this experience three years ago, I have been lucky enough to be surrounded by people who provided me a constant stream of positive support. From the folks I worked with at the hospitals to all of my family, friends and co-workers, the positive vibes are what has inspired me to keep going.  There were also a couple of notable things that two people told me that I have hung on to that have kept me going to this day:
    • My wife Janene has always taught our girls that no matter what you're doing in life, you need to 'fake it 'til you make it'. This catchphrase helps you to feel confident and optimistic about something until you gain the necessary experience to actually feel genuinely assured that you have reached a successful point. Although she has always intended this for the benefit of our daughters, I have been able to internalize it and use it to my own benefit in my recovery. Repeating this statement in my head has taken me quite far and I continue to use it to this day. Thank you so much, Janene. I love you!
    • My friend Greg, who has had two spinal cord injuries in his life (can you believe that?!), told me something very early on in my journey, that I held in my head to help me get through the first year and beyond. He said something like, 'I know you you are not in a place where you can understand what this means yet, but you will get there in time. Just do everything you can to make it through the first year and everything will seem 1000% better. You won't be totally healed in one year, but you will feel much, much better.' Ironically, I saw Greg the week of my first year anniversary and I told him about this and he didn't even remember telling me this. I think he was quite surprised that I held on to it for so long, but it was truly a lifeline. Thank you, Greg.
    I have learned a lot in three years as this experience has taught me a lot. Most importantly is the way that you handle the experience. Most importantly, I've learned that when you are faced with a horribly painful experience (emotionally, mentally, physically) that changes your life, you can choose to one of two paths:
    1. Either, you can be angry, resistant, resentful and stuck on the fact that something was taken from you. I have met plenty of people on my journey who were here and until they change their outlook, they won't be able to move on.
    2. Or, you can acknowledge that it sucks but still feel gratitude for the positive aspects and for being able to be alive to experience it all. I haven't met any people who can say that they feel thankful for their experience with a spinal cord injury, but I have read about some. It wasn't easy for me to get to this point.
    And this certainly doesn't mean that I'm done. My recovery will continue for years.

    On that note, a singer-songwriter who I have listened to for years says this best here (Blogger won't let me embed this video correctly):

    He says it best by summarizing it this way: Pain helps us learn who we really are.

    Just recently, one of my colleagues from our Munich, Germany headquarters visited my office in Boulder. I have not seen this guy in person since before the accident so he was really shocked to see me walking and to see how well I am doing now. He said he was so surprised because the last he heard from me I was still in the wheelchair (the look on his face was priceless!). It's moments like this one that remind me of how far I've come.

    Thank you to everyone who has helped me in any way along this journey.

    0 0

    TL;DR: I had a super fun summer traveling with my family, recently moved to a new house in the country, and I've joined Stormpath as a Developer Evangelist. Wahoo!

    I've written several "life update" posts in the past, but there's been few as epic as this one. When I wrote my 2016 Goals, I listed "July in Montana" as one of them. And y'all know that "finish the bus" was #1. Since the bus is done, I wanted to show it off in my hometown 4th of July parade in Condon, Montana.

    A path to these goals became clear in mid-June, shortly after visiting London and Tallinn with my Mom. It involved a lot of driving, but I was determined to make it happen. I was so excited about my plan that I sent an email to my best friend, Owen, on June 15.

    Hey Owen!

    Hope you're doing well and enjoying the summer. I just wanted to give you a heads up that I'll be in Montana quite a bit this summer. First of all, we'll be driving up with the bus (it's finally done!) next week. We're not staying though - we have a flight out of Missoula to Hawaii on Saturday morning. We're going with Trish's family to celebrate her parent's 50th wedding anniversary.

    We return on June 3rd, just in time to drive the bus in the parade on the 4th! I'll likely drive it down to Seeley Lake to be in theirs as well.

    We're driving back to Denver on July 9 to be in a VW show on the 10th. Then I'll be driving our van back to MT starting on July 11. I plan to take it slow and work from the van on my way. My goal is to arrive by Thursday, July 14 for the Bob Marshall Music Festival.

    Trish will fly in for the weekend. My birthday is that Saturday, the 16th! I plan to camp in Seeley and enjoy the music. Then, I'll hang out another week with my folks, working remotely and playing. I'll return to Denver on July 23.

    Can't wait to drink some beer with you old buddy!



    Fast forward to today, September 26, 2016 → it's fun to reminisce that I traveled to Hawaii for a wonderful anniversary celebration, flew to Montana for two 4th of July parades, showed Hefe at the Colorado Bug-In, and lived the #vanlife on the way to The Cabin via Grand Junction, Jackson Hole, Missoula and Seeley Lake.

    You can see why it was a summer to remember. Here's some pics:

    Hefe is ready to road trip!Sunset in Hanalei BayHanalei Happy Hour

    Happy 50th Mau and Joe!

    Jack is wiped after last night's red eye. It's great to be at The Cabin!Our 2nd parade of the day! Hefe is lookin' good. Happy 4th y'all!Hefe on Front Road

    Hefe in the Swan Valley Parade

    Hefe at his finestGot Runner Up in our class. Lost to a very nice Karmann Ghia. However, they trailered theirs home and I'm driving mine!

    I like how this birthday week is progressing!It's easy to fall in love with Jackson, Wyoming on a day like today!Idaho Sunset in Teton Valley

    The siding is almost finished on The House! Dad rented a high lift and finished 80% today. This is the before photo.My parents finally finished the siding on their epic retirement house! Congrats awesome people!

    Celebrating our third anniversary at Garden of the Gods

    Jack's first day of middle school and Abbie's first day of 8th grade! Same school for the first time in three years. They're so excited to be reunited!Buses at the BreweryJack and his buddies on his 12th birthday!

    New House
    Trish and I've been looking for a new house for a while now. I wrote about how we tried to sell our home near DU last year. For the first year of our search, we were focused on finding a place near the kids' school. This spring, we shifted our focus to looking for horse property. Trish was a competitive equestrian jumper in high school and she's got Abbie into it so why not? After all, their horse hobby is much less expensive than my VW obsession. ;)

    In late July, Trish found the property of her dreams in Centennial, Colorado (just north of Parker). At the time, I was still in Montana and the kids were in Florida with their Mom. Nevertheless, it felt right and we went for it. We put an offer on the home on July 19 and today I'm writing this blog post inside our new home. We moved in last week and couldn't be happier. Of course, we wish we were fully moved in, but we realize that'll probably take another week or two. The only downside of the move is we had to sell our mountain views in Fraser. The good news is we have new mountain views, two acres of land and plenty of room for our classic VW collection.

    Home Sweet Home!


    New Gig at Stormpath
    I've been working with Stormpath as a consultant since April. I contributed a fair amount of my time to helping them develop and launch their Java SDK 1.0 release. After the first couple months, I started talking with their co-founder and CTO, Les Hazlewood, about joining Stormpath full-time. At first I was more interested in continuing my career as an independent consultant. Then I really started to think about what I liked to do for work. It came down to a few things: develop open source software, write about what I've learned as a developer and preach the gospel of good developer tools. I sent Les a "Stormpath Employment Proposal" in late July.

    After many conversations with Les and Alex (Stormpath's CEO), I flew out for interviews in early September.

    High plainsThe Rocky MountainsLanding at SFO

    More photos on Flickr → 2016: A Summer to Remember

    The interviews went great and I'm happy to say that Stormpath has offered me an tremendous opportunity to code, learn, and educate. Simply put: they gave me an offer I couldn't refuse. I quickly accepted and today is my first day as a Stormpath Developer Evangelist.

    It takes a special kind of company to entice me to work for them full-time and I'm looking forward to doing great things with Stormpath. Now more than ever, there's a good chance I'll see you on the road or at a conference soon!

    0 0

    Not long ago, XSLT 3.0 has reached the W3C Recommendation status. Below is the link to the XSLT 3.0 spec:

    XSLT 3.0 is a very advanced and useful language, as compared to XSLT 2.0. One of the main features (among others) introduced in XSLT 3.0, is that the XSLT transformation can be done in streaming mode.

    0 0

    Note: This post is about my second Dirty Kanza 200 experience on June 3, 2017.

    It’s broken into seven parts:

    Part I – Prep / Training

    Part II – Preamble

    Part III – Starting Line

    Part IV – Checkpoint One

    Part V – Checkpoint Two – coming soon

    Part VI – Checkpoint Three – coming soon

    Part VII – Finish Line – coming soon


    Running nearly the same course as last year, I knew what to expect.  The first leg is probably my favorite.  It’s wet, wild and the scenery stunning.  Sometimes it felt otherworldly.

    Other times, just fun…

    But this year I was determined to maintain a focus, preserve fuel by taking it easy on the climbs; staying down in the saddle.


    MS Photos

    In this I was successful.  That 36T granny gear worked perfectly and I was feeling good rolling into Madison somewhere after 9am.

    I signed up the Crew-for-Hire support this year, but only needed them at the first checkpoint.  They were great btw, filled my tanks, handed me cookies, cokes and whatever I needed.  Highly recommended.  There was also the SRAM team, providing us help with the bikes.  I handed mine over and it was returned, running smoothly — again.  Much needed as the water crossings dried out my drivetrain.

    One small mishap occurred when I let my bike fall over during the reload.  It broke the gopro mount on helmet.  Oh well, should probably be paying more attention to the road.

    Another change, from now on it’s Kelly & Co. as my pit crew.  He rallied the troops, after finishing his DK-lite route, drove down to Eureka, for our scheduled rendezvous at checkpoint two.


    Kelly finishing his ride Saturday.

    Here’s what it looked like rolling into Madison.


    And on into the checkpoint.

    Looking OK on time so far…

    Screen Shot 2017-06-13 at 5.07.18 PM

    Next Post: Part V – Checkpoint Two – coming soon



    0 0

    Somewhat with my recent trip to France in mind, I picked up Hilary Mantel's A Place of Greater Safety.

    In the end, I lugged the book around Alsace, but ended up reading it both before and after the trip rather than while I was there, which was actually just fine.

    Mantel is well-known to me: I adored her Wolf Hall and Bringing up the Bodies, which are more recent work by her. A Place of Greater Safety is just celebrating its 25th birthday. But it doesn't read like an early work; she was every bit the master writer even then.

    I doubt I gave A Place of Greater Safety the time and attention it deserves. At 750 pages of rather dense prose, in which you are reluctant to miss even a single word, much less a phrase or paragraph, I pushed through it as fast as I could, but still it took 2 months.

    In the end, what do I think I learned?

    • In a Revolution, almost by definition, nobody is in control of events, even, perhaps most especially, not those who THINK they are in control of events.

    • The French Revolution is, nowadays, remembered as a time and a place when wondrous progress was made in the world of ideas (The Declaration of the Rights of Man, "Liberte, Egalite, Fraternite", the Republic, feminism, secularism, etc.), but the actual story was much a story of people and what they said and did day-to-day. That is, nobody sat down one day and decreed the entire thing; events happened incrementally, in the heat of the moment, under pressures and in situations that it can be hard to envision.

    • People are, in the end, people. Nobody really set out to Have A Revolution, and even once it was underway, lots of other life went on at the same time: people got married, had children, had affairs, got sick, got divorced, moved house, got in arguments, all sorts of things. We might want to write A Clean Story Of The French Revolution Leaving Out All The Boring Humdrum Stuff, but that wouldn't be telling the story the way it really happened.

    • Mantel is a lovely writer, and it it is always a treat to read her books, even if one must thus read about awful events (not all the events she relates are awful; some are vividly inspirational, but there is plenty of misery to go around)

    0 0

    This is the second in a series of tutorials on securing Apache Storm. The first post looked at setting up a simple Storm cluster that authenticates users via Kerberos, and deploying a topology. Apache Storm also ships with a UI (and REST API) that can be used to download configuration, start/stop topologies, etc. This post looks at deploying the Storm UI using Kerberos, and accessing it via a REST client.

    1) Configure the Apache Storm UI

    The first step is to follow the previous tutorial to deploy the Apache Kerby KDC, to configure Apache Zookeeper, and to download and deploy Apache Storm (sections 1-3). Note that there is a bug in Kerby that is not yet fixed in the 1.0.0 release that you might run in to when using curl (see below), depending on whether the MIT libraries are installed or not. In additional to the principals listed in the last post, the Kerby deployment test for Storm also contains a principal for the Storm UI (HTTP/

    Now edit 'conf/storm.yaml' and add the following properties:

    • ui.filter: ""
    •  ui.filter.params:
      • "type": "kerberos"
      • "kerberos.principal": "HTTP/"
      • "kerberos.keytab": "/"
      • "": "RULE:[2:$1@$0]([jt]t@.*EXAMPLE.COM)s/.*/$MAPRED_USER/ RULE:[2:$1@$0]([nd]n@.*EXAMPLE.COM)s/.*/$HDFS_USER/DEFAULT"
    Start the UI with:
    • bin/storm ui
    2) Invoke on the Storm UI REST API

    We will invoke on the Storm UI REST API using "curl" on the command line. This can be done as follows:
    • export KRB5_CONFIG=/
    • kinit -k -t / alice
    • curl --negotiate -u : -b ~/cookiejar.txt -c ~/cookiejar.txt http://localhost:8080/api/v1/cluster/configuration
    You should see the cluster configuration in JSON format if the call is successful.

    0 0

    This is the second (and final for now) post in a short series of blog posts on securing Apache HBase. The first post looked at how to set up a standalone instance of HBase and how to authorize access to a table using Apache Ranger. In this post, we will look at how Apache Ranger can create "tag" based authorization policies for Apache HBase using Apache Atlas.

    1) Start Apache Atlas and create entities/tags for HBase

    First let's look at setting up Apache Atlas. Download the latest released version (0.8-incubating) and extract it. Build the distribution that contains an embedded HBase and Solr instance via:

    • mvn clean package -Pdist,embedded-hbase-solr -DskipTests
    The distribution will then be available in 'distro/target/apache-atlas-0.8-incubating-bin'. To launch Atlas, we need to set some variables to tell it to use the local HBase and Solr instances:
    • export MANAGE_LOCAL_HBASE=true
    • export MANAGE_LOCAL_SOLR=true
    Now let's start Apache Atlas with 'bin/'. Open a browser and go to 'http://localhost:21000/', logging on with credentials 'admin/admin'. Click on "TAGS" and create a new tag called "customer_data". Now click on "Search" and then follow the "Create new entity" link of type "hbase_table" with the following parameters:
    • Name: data
    • QualifiedName: data@cl1
    • Uri: data
    Now add the 'customer_data' tag to the entity that we have created.

    2) Use the Apache Ranger TagSync service to import tags from Atlas into Ranger

    To create tag based policies in Apache Ranger, we have to import the entity + tag we have created in Apache Atlas into Ranger via the Ranger TagSync service. After building Apache Ranger then extract the file called "target/ranger-<version>-tagsync.tar.gz". Edit '' as follows:
    • Set TAG_SOURCE_ATLAS_ENABLED to "false"
    • Set TAG_SOURCE_ATLASREST_DOWNLOAD_INTERVAL_IN_MILLIS to "60000" (just for testing purposes)
    Save '' and install the tagsync service via "sudo ./". Start the Apache Ranger admin service via "sudo ranger-admin start" and then the tagsync service via "sudo start".

    3) Create Tag-based authorization policies in Apache Ranger

    Now let's create a tag-based authorization policy in the Apache Ranger admin UI. Click on "Access Manager" and then "Tag based policies". Create a new Tag service called "HBaseTagService". Create a new policy for this service called "CustomerDataPolicy". In the "TAG" field enter a "c" and the "customer_data" tag should pop up, meaning that it was successfully synced in from Apache Atlas. Create an "Allow" condition for the user "bob" with the "Read" permission for the "HBase" component.

    We also need to do is to go back to the Resource based policies and edit "cl1_hbase" and select the tag service we have created above. Now we are ready to test the authorization policy we have created with HBase. Start the shell as "bob" and we should be able to read the table we created in the first tutorial:
    • sudo -E -u bob bin/hbase shell
    • scan 'data'

    0 0

    After years of being involved in open-source development at Apache, we’ve seen security issues pop up in Apache Commons like arbitrary remote code execution, and denial of service attacks (CVE-2016-3092 and CVE-2014-0050). While some threats are real, other are just FUD. Even when they are real, it is important to consider context. There may be problems that end users never see because the “hole” is not reachable by users’ actions.

    I started to wonder how we can short-circuit FUD and proactively deal with security issues before we hear about them in the press and in panicked mailing list posts. Is there something we can do at build time? A FindBugs or PMD on steroids? Surely someone else must be thinking about this. Can we apply the principle of shift left testing to security?

    It turns out there is a tool out there that does and it’s called, not coincidentally, The idea behind ShiftLeft is to break old habits of building a product and then, later, figuring out how to fend off attacks and plug up security holes. Today, we take for granted that unit testing, integration testing, continuous integration, and continuous delivery are common place. ShiftLeft propose to make security analysis as ubiquitous.

    Getting started

    Since ShiftLeft is free for open source projects, I decided to look what it reports for Apache Commons IO2.5, an Apache Commons Java component. While I won’t divulge right now any real security issues, I’ll show what is safe to report about Commons IO. I’ll also show the cool stuff ShiftLeft points out to write more bullet-proof software. To use ShiftLeft, you go to their web site and submit the URL to your GitHub repository. ShiftLeft has a 30 day disclosure policy so you have plenty of time to fix up your FOSS project.

    What I got is a JSON report file containing all sorts of data and what ShiftLeft calls conclusions. These are the potentially actionable items. As we’ll see, even if you find some conclusions non-actionable, these will do a great deal to raise your awareness of potential security issues for code that you’ll write tomorrow or need to maintain today.

    Let’s start with a simple conclusion and get deeper in the weeds after that.

    No memory for you!

    The following example shows what “untrusted” data is and how it can affect your application.

    I have a ShiftLeft conclusion that tells me the method IOUtils.buffer(Writer, int) does not support handling untrusted data to be passed as parameter size because it controls the size of a buffer, giving an attacker the chance to starve the system of memory:

     * Returns the given Writer if it is already a {@link BufferedWriter}, otherwise creates a BufferedWriter from the
     * given Writer.
     * @param writer the Writer to wrap or return (not null)
     * @param size the buffer size, if a new BufferedWriter is created.
     * @return the given Writer or a new {@link BufferedWriter} for the given Writer
     * @throws NullPointerException if the input parameter is null
     * @since 2.5
     public static BufferedWriter buffer(final Writer writer, int size) {
      return writer instanceof BufferedWriter ? (BufferedWriter) writer : new BufferedWriter(writer, size);

    While this example may seem trivial, ShiftLeft shows understanding of what the code does in this method: We are allowing call sites to control memory usage in an unbounded manner. ShiftLeft gives us the exact method and line number:

    While some ShiftLeft conclusions may not all be actionable, I must say that the report has made me more aware of what to potentially secure when writing new code or maintaining old an code base.

    zzfireplaceTangent: What could you do in this case? You could hard-code an upper bound of say 10 MB. You could make the bound configurable with a static non-final variable (effectively creating a global variable, an anti-pattern for sure.) You could move all the static methods to the instance side, create a memory profile class to define this bound, and then build IOUtils instances with this profile in the constructor. In addition, you could also use a different buffer class internally to make sure the buffer does not grow beyond a given size. And so on. I am not proposing these changes in the context of the Apache Commons IO 2.x line since we take great care to maintain binary compatibility within a major release across all of Apache Commons. But I can definitively see proposing changes for 3.0!

    ShiftLeft’s understanding of code is deeper than this example thanks to Enhanced Code Property Graphs so let’s look at a more complex example.

    Digging deeper

    Here is a conclusion in raw form (edited for brevity):

     "id": ",",
     "description": "The method `copyFileToDirectory` does not support handling **sensitive data** to be passed as parameter `srcFile` because it is leaked over I/O **File**.",
     "unsupportedDataType": "SENSITIVE",
     "interfaceId": "FILE/false",
     "methodId": ",",
     "codeLocationUrl": "",
     "state": "NEUTRAL",
     "externalIssueUrl": "https://todo"

    Looking at the methodId tells us to go look at FileUtils.copyFileToDirectory(File, File) where we find:

     * Copies a file to a directory preserving the file date.
     * This method copies the contents of the specified source file
     * to a file of the same name in the specified destination directory.
     * The destination directory is created if it does not exist.
     * If the destination file exists, then this method will overwrite it.
     * &amp;amp;lt;strong&amp;amp;gt;Note:&amp;amp;lt;/strong&amp;amp;gt; This method tries to preserve the file's last
     * modified date/times using {@link File#setLastModified(long)}, however
     * it is not guaranteed that the operation will succeed.
     * If the modification operation fails, no indication is provided.
     * @param srcFile an existing file to copy, must not be {@code null}
     * @param destDir the directory to place the copy in, must not be {@code null}
     * @throws NullPointerException if source or destination is null
     * @throws IOException if source or destination is invalid
     * @throws IOException if an IO error occurs during copying
     * @see #copyFile(File, File, boolean)
     public static void copyFileToDirectory(final File srcFile, final File destDir) throws IOException {
      copyFileToDirectory(srcFile, destDir, true);

    This method just delegates to another copyFileToDirectory() with an added parameter, no big deal. What is interesting is that the codeLocationUrl points to code not in this method but to a private utility method:

    FileUtils at line 1141 is in the guts of a private method called, File, boolean) which is where ShiftLeft flags an issue where the method creates a new FileInputStream. To be honest, this is a beta and I am not convinced that the line numbers are always dead on. What is important here is that ShiftLeft is working with a code graph. Therefore, when I search the JSON conclusions for this URL, I find a total of 14 conclusions that use this URL. This tells me that this code fragment creates 14 possible attack points in the component; with a careful emphasis on possible since context is important.

    Another key point is to realize that there are two key technologies at work here and that I expect both to get better as the beta progresses: First, building a code graph to give us the power to see that once a problem has been identified on a line of code, that all (I assume public) call-sites can be flagged. Second, what constitutes a problem or a conclusion in ShiftLeft’s neutral parlance will improve and be configurable, filterable and sortable.

    In this example, the conclusion description reads:

    The method `copyFileToDirectory` does not support handling **sensitive data** to be passed as parameter `srcFile` because it is leaked over I/O **File**.

    What goes through my head when I read that is: Yeah, I do not want just anybody to be able to copy any file anywhere like overwriting a password vault a la copyFileToDirectory(myFile, "/etc/shadow"). Granted, Commons IO is a library, not an application, so there is no alarm bell to ring here, but you get the idea.

    Stepping back, I think it is important to reiterate what happened here: ShiftLeft found an issue (less dramatic than a problem) on a line of code in a private methods, then, using its code graph, created conclusions (report items) for each public facing method that may eventually call this private method in its code path.

    Are you Trusted and Sensitive?

    ShiftLeft uses two terms in conclusion descriptions that I want to review. Based on the limited subset of 255 (a very computer friendly number!) conclusions I saw for all of Commons IO, we can see two types of issues: trusted and sensitive.

    sensitive-data-in-the-cloud-blog-image-1Conclusions described as dealing with sensitive data: This says: Lookout, if you have a password in this variable, it’s in plain text. Now, it’s up to me to make sure that this password does not end up in a clear text file or anywhere else that is not secure. This is where context matters, you are the SME of your code, you know how much trouble you can get yourself and your users into, ShiftLeft has no opinion, it offers ‘conclusions.’

    Conclusions referring to untrusted data: This tells me I should take precautions before doing anything with that data. Should I just execute this script? Should I need to worry about JSON Hijacking? See Why does Google prepend while(1); to their JSON responses?

    Flipping it around on ShiftLeft

    Let’s flip it around on ShiftLeft for a minute. I am now thinking about not writing sensitive data like passwords and financial reports to disk insecurely. I know we have many API in FileUtils that write strings to files. Will ShiftLeft tell me about, for example FileUtils.write(File, CharSequence, Charset)? Here is the method I should never use to write passwords or any sensitive data:

     * Writes a CharSequence to a file creating the file if it does not exist.
     * @param file the file to write
     * @param data the content to write to the file
     * @param encoding the encoding to use, {@code null} means platform default
     * @throws IOException in case of an I/O error
     * @since 2.3
     public static void write(final File file, final CharSequence data, final Charset encoding) throws IOException {
      write(file, data, encoding, false);

    This in turn calls:

     * Writes a CharSequence to a file creating the file if it does not exist.
     * @param file the file to write
     * @param data the content to write to the file
     * @param encoding the encoding to use, {@code null} means platform default
     * @param append if {@code true}, then the data will be added to the
     * end of the file rather than overwriting
     * @throws IOException in case of an I/O error
     * @since 2.3
     public static void write(final File file, final CharSequence data, final Charset encoding, final boolean append)
     throws IOException {
      final String str = data == null ? null : data.toString();
      writeStringToFile(file, str, encoding, append);

    Which calls:

     * Writes a String to a file creating the file if it does not exist.
     * @param file the file to write
     * @param data the content to write to the file
     * @param encoding the encoding to use, {@code null} means platform default
     * @param append if {@code true}, then the String will be added to the
     * end of the file rather than overwriting
     * @throws IOException in case of an I/O error
     * @since 2.3
     public static void writeStringToFile(final File file, final String data, final Charset encoding, final boolean
     append) throws IOException {
       OutputStream out = null;
       try {
         out = openOutputStream(file, append);
         IOUtils.write(data, out, encoding);
         out.close(); // don't swallow close Exception if copy completes normally
       } finally {

    Which calls IOUtils‘:

     * Writes chars from a &amp;lt;code&amp;gt;String&amp;lt;/code&amp;gt; to bytes on an
     * &amp;lt;code&amp;gt;OutputStream&amp;lt;/code&amp;gt; using the specified character encoding.
     * This method uses {@link String#getBytes(String)}.
     * @param data the &amp;lt;code&amp;gt;String&amp;lt;/code&amp;gt; to write, null ignored
     * @param output the &amp;lt;code&amp;gt;OutputStream&amp;lt;/code&amp;gt; to write to
     * @param encoding the encoding to use, null means platform default
     * @throws NullPointerException if output is null
     * @throws IOException if an I/O error occurs
     * @since 2.3
     public static void write(final String data, final OutputStream output, final Charset encoding) throws IOException {
       if (data != null) {

    Knowing what I know, I expect ShiftLeft to conclude that all these methods do not support sensitive data. Working back up the stack, I find:

    •, OutputStream, Charset)
      Nothing on that one; did I miss it due to the 255 conclusion limit?
    •, String, Charset, boolean)
      Got it:

       "id": ",java.lang.String,boolean)/file/1",
       "description": "The method `writeStringToFile` does not support handling **untrusted data** to be passed as parameter `file` because it controls access to I/O **File** in a manner that would allow an attacker to abuse it.",
       "unsupportedDataType": "ATTACKER_CONTROLLED",
       "interfaceId": "FILE/false",
       "methodId": ",java.lang.String,boolean)",
       "codeLocation": {
         "file": "org/apache/commons/io/",
         "lineNumber": 360,
         "symbol": "parent"
       "codeLocationUrl": "",
       "state": "NEUTRAL",
       "externalIssueUrl": "https://todo"
    •, CharSequence, Charset, boolean)
      Got it:

       "id": ",java.lang.CharSequence,java.nio.charset.Charset,boolean)/file/2",
       "description": "The method `write` does not support handling **sensitive data** to be passed as parameter `file` because it is leaked over I/O **File**.",
       "unsupportedDataType": "SENSITIVE",
       "interfaceId": "FILE/false",
       "methodId": ",java.lang.CharSequence,java.nio.charset.Charset,boolean)",
       "codeLocation": {
         "file": "org/apache/commons/io/",
         "lineNumber": 355,
         "symbol": "parent"
       "codeLocationUrl": "",
       "state": "NEUTRAL",
       "externalIssueUrl": "https://todo"
    •, CharSequence, Charset)
      Got it:

       "id": ",java.lang.CharSequence,java.nio.charset.Charset)/file/2",
       "description": "The method `write` does not support handling **sensitive data** to be passed as parameter `file` because it is leaked over I/O **File**.",
       "unsupportedDataType": "SENSITIVE",
       "interfaceId": "FILE/false",
       "methodId": ",java.lang.CharSequence,java.nio.charset.Charset)",
       "codeLocation": {
         "file": "org/apache/commons/io/",
         "lineNumber": 355,
         "symbol": "parent"
       "codeLocationUrl": "",
       "state": "NEUTRAL",
       "externalIssueUrl": "https://todo"

    So yeah, that all hangs nicely together, thank you code graphs!

    Finding arbitrary code attacks

    Did I mention there 255 are conclusions in the JSON report file? It takes a while to go through these. I am hoping that ShiftLeft will have their UI in place soon so I can filter and sort all this information! Now that I am about 20% through the file, I see:

    The method `createSymbolicLink` does not support handling **untrusted data** to be passed as parameter `symlink` because it could allow an attacker to run arbitrary code.


    Yikes! let’s take a deeper look and see if this is for real or a false positive.

    Here is the raw conclusion:

     "id": ",",
     "description": "The method `createSymbolicLink` does not support handling **untrusted data** to be passed as parameter `symlink` because it could allow an attacker to run arbitrary code.",
     "unsupportedDataType": "ATTACKER_CONTROLLED",
     "methodId": ",",
     "codeLocation": {
     "file": "org/apache/commons/io/",
     "lineNumber": 128,
     "symbol": "file"
     "codeLocationUrl": "",
     "state": "NEUTRAL",
     "externalIssueUrl": "https://todo"

    Our codeLocationUrl for this conclusion points us to the Java7Support class at line 128: where we find:

     * Indicates if a symlunk target exists
     * @param file The symlink file
     * @return true if the target exists
     * @throws IOException upon error
     private static boolean exists(File file)
     throws IOException {
       try {
         Object path = toPath.invoke(file);
         final Boolean result = (Boolean) exists.invoke(null, path, emptyLinkOpts);
         return result.booleanValue();
       } catch (IllegalAccessException e) {
         throw new RuntimeException(e);
       } catch (InvocationTargetException e) {
         throw (RuntimeException) e.getTargetException();

    ShiftLeft points to the line:

    Object path = toPath.invoke(file);

    The instance variable toPath is a java.lang.reflect.Method which can and does execute code as shown above. Looking narrowly at the code so far we can say that yes, this code run anything since toPath is a Method.

    However, widening our view to the field declaration we see the following in the class static initialization block:

    toPath = File.class.getMethod("toPath");

    This makes sense in the context of the class: Java7Support is used to access Java 7 features while running on pre-Java 7 platforms. Here we are setting up toPath to run one method. I would expect toPath to be a static final but it is not:

    private static Method toPath;

    Why is it not static? Well, it’s just that the way the static initialize block is written does not allow you to just add final to the declaration. The static block needs to be rewritten to allow for toPath to be final which we will leave as ‘an exercise to the reader’ 😉 as it is out of scope for an already long blog post.

    I would be curious to see how ShiftLeft responds to such a code change.

    I am not sure if this is really a problem though. The variable is private now, but not final. Yes its type (Method) is all about executing code. Under normal circumstances, this value cannot be changed outside this class. I can use reflection of course to force a new value in toPath. Does that mean that anytime I use a Method instance variable I am going to get an arbitrary code execution conclusion? Another corner-case to examine.

    What if I rewrote the static block and declared the variable final. Would ShiftLeft still reach the same conclusion? If yes, would that be because I could still use reflection to hammer any value in the field.

    Concluding on this first arbitrary code attack

    The more I explore these test results, the more I realize how tricky security is and how much context matters. I now know that the Java7Support class in Apache Commons IO 2.5 is open to an arbitrary code attack under the narrow use case of another piece of code using reflection. But if that code is allowed to use reflection, surely it could achieve its goal without going through the extra hoop of Java7Support hacking.

    Stepping back, the realization is that I should think twice about using the Method class because I could open my application up to an attack unless that Method field is properly protected.

    Looking for more arbitrary code attacks

    Now that ShiftLeft has whetted my appetite, I wonder if there are more arbitrary code attacks lurking. A quick search through the file reveals to total of five. Not surprisingly, these are all in the Java7Support class and all follow the same pattern as above: calling the invoke method of a Method object where the Methodis initialized in the static block.

    Flipping it around once more, let’s look at the Java7Support class variable declarations and see if all Method objects end up being accounted for by ShiftLeft:

     * Java7 feature detection and reflection based feature access.
     * <p/>
     * Taken from maven-shared-utils, only for private usage until we go full java7
    class Java7Support {
      private static final boolean IS_JAVA7;
      private static Method isSymbolicLink;
      private static Method delete;
      private static Method toPath;
      private static Method exists;
      private static Method toFile;
      private static Method readSymlink;
      private static Method createSymlink;

    We have seven static Method declarations which I see initialized in the static block:

    static {
     boolean isJava7x = true;
     try {
       ClassLoader cl = Thread.currentThread().getContextClassLoader();
       Class<?> files = cl.loadClass("java.nio.file.Files");
       Class<?> path = cl.loadClass("java.nio.file.Path");
       Class<?> fa = cl.loadClass("java.nio.file.attribute.FileAttribute");
       Class<?> linkOption = cl.loadClass("java.nio.file.LinkOption");
       isSymbolicLink = files.getMethod("isSymbolicLink", path);
       delete = files.getMethod("delete", path);
       readSymlink = files.getMethod("readSymbolicLink", path);
       emptyFileAttributes = Array.newInstance(fa, 0);
       createSymlink = files.getMethod("createSymbolicLink", path, path, emptyFileAttributes.getClass());
       emptyLinkOpts = Array.newInstance(linkOption, 0);
       exists = files.getMethod("exists", path, emptyLinkOpts.getClass());
       toPath = File.class.getMethod("toPath");
       toFile = path.getMethod("toFile");
       } catch (ClassNotFoundException e) {
         isJava7x = false;
       } catch (NoSuchMethodException e) {
         isJava7x = false;
       IS_JAVA7 = isJava7x;

    ShiftLeft gives me five conclusions:

    The method `isSymLink` does not support handling **untrusted data** to be passed as parameter `file` because it could allow an attacker to run arbitrary code.
    The method `createSymbolicLink` does not support handling **untrusted data** to be passed as parameter `symlink` because it could allow an attacker to run arbitrary code.
    The method `delete` does not support handling **untrusted data** to be passed as parameter `file` because it could allow an attacker to run arbitrary code.
    The method `createSymbolicLink` does not support handling **untrusted data** to be passed as parameter `target` because it could allow an attacker to run arbitrary code.
    The method `readSymbolicLink` does not support handling **untrusted data** to be passed as parameter `symlink` because it could allow an attacker to run arbitrary code.

    All of the Method declarations in this class are used by all of the methods listed above. Nice.


    fin_3_0I’d like to wrap up this exploration of ShiftLeft with a quick summary of what we found: a tool we can add to our build pipelines to find potential security issues. There are a lot of data here, and this is just for Apache Commons IO 2.5, so another lesson is that context matters. ShiftLeft currently provides ‘conclusions’ based on a code graph. We found conclusions about untrusted data (I’m not sure what’s in here so don’t go executing it), sensitive data (don’t save passwords in plain text!) and arbitrary code attacks.

    I hope revisit this story and run ShiftLeft on other Apache Commons projects soon. This sure is fun!

    Happy Coding,
    Gary Gregory

    0 0

    The brambles are flowering more than ever, hinting at the likelihood of a super-abundant blackberry season to come.  And now that the unsettled weather at the beginning of the month has given way to warm and sunny, the bees are out there enjoying them.

    I expect I shall enjoy fresh blackberries daily for a while, then a little later in the season move to the cooked desserts – pies and crumbles – and stew some for freezing.  And brew up another batch of chutney.  And if I put some through the juicer to drink, that uses vast quantities.   But I’m thinking, if I had yet more uses for them, this would be a good year to experiment.  Maybe see if they go well in gin or somesuch as a strong drink?

    0 0

    Note: This post is about my second Dirty Kanza 200 experience on June 3, 2017.

    It’s broken into seven parts:

    Part I – Prep / Training

    Part II – Preamble

    Part III – Starting Line

    Part IV – Checkpoint One

    Part V – Checkpoint Two

    Part VI – Checkpoint Three – coming soon

    Part VII – Finish Line – coming soon


    I mentioned earlier that this course was the same as last year’s, and so after finishing then, knew what to expect.  It’s about 55 miles and has the most climbing.

    Screen Shot 2017-06-16 at 7.37.52 AM

    Leg Two begins @ mile 49

    You can get an idea what I’m talking about from its elevation profile.  From mile 49, there’s 30 miles, mostly uphill.  The roads cross ranch land and are lightly maintained, if at all.  This means fun; this means pain.


    Mile 62 was still fun

    The winds were tame, the sun hidden behind a thick cloud cover keeping the temps down but it was muggy.


    Mile 76 getting tougher

    It was this time last year I hit the Wall.  But I adjusted and was confident that trouble could be avoided.


    As the going slows, the tedium grows, the mind struggles to find something to latch onto, and begins to play tricks.  Seemingly big things are downplayed.  For example, my left leg began cramping at mile 85, but I largely ignored it.  The terrain becomes treacherous, but I was unconcerned — just white noise.

    But small things become a big deal.  For example, an airplane buzzing overhead, I became obsessed with getting its picture.  It seems stupid now to pull out a phone, aim it upwards, and shoot, while navigating the most challenging terrain on the course.


    Took this one while riding…. airplane.

    Anything to take the mind off of the pain.  And that nameplate, on its back, facing me, were images of sad riders, my friends, who couldn’t manage to mount it correctly.

    Screen Shot 2017-06-16 at 9.44.40 AM

    Image on the back of nameplate of the happy and sad riders

    And the one happy rider, who got it right, I wanted to rip his face off.  Was I angry for being so damn happy whilst we were suffering?  Was he mocking us?  Did I make a mistake and hang my nameplate wrong, which doomed my fate to be unhappy like the others?

    Yeah, I know it doesn’t make any sense.

    For the most part I was doing OK, just slowed down quite a bit.  The changes to the rear cassette, and the hill training, helped tremendously.  I remained in the saddle the entire time and only walked one hill — the Bitch.  I could have rode it but my cramping left quad begged me otherwise and so I relented, just this once.


    I went into great detail last year about running out of water during the middle of the second leg.  This year I made changes, added a 2.5L Camelbak to a 1.8L bladder and two more bottles in the cages — 1.5L.  That’s about 1.5 gallons for those keeping score back home.

    In addition to water I also used (more) electrolytes, although not enough due to the cramping I experienced.

    I still ran out of water on this leg, around mile 95, about nine miles from checkpoint two!!  Fortunately, there was a husband and wife duo parked at the end of their driveway, outside their country home, with a truckload of iced water bottles beside them.  I stopped and asked if they would be so kind as to share some water with me.

    “Are you dry”, the man asked me in his Kansas twang, to which I replied that I most certainly was.

    “Take all you want”, he told me.  I downed a pint as we exchanged pleasantries, grabbed another for the road and just like that I’m good.

    I grew up in Kansas, already knew that good Samaritans run aplenty, but still am inspired by their warmth and hospitality. One of the reasons I keep coming back here is to be reminded that these people are still here.


    From here on out I have in my pit crew, from l to r, Cheri, Gregg, Janice, Kelly and Kyle.


    Kelly had just completed the DKlite, and was working his magic down in Eureka keeping the crew operating like a well-oiled machine.  Kyle was in from Seattle and Janice (Mom) from Salina.  You may recall that Gregg was my riding partner last year.  He couldn’t commit to the training but made sure he was there to lend a hand and offer encouragement along with Cheri, his partner, who also was in our pit last year.


    Kelly wearing the colors

    That way when I rolled into town, weary and tired from the road…


    Rolling into CP2

    All I had to do was hand over my bike, eat, hydrate, and try to relax a bit.  I can’t tell you how much it helped me to have them there.


    That time spent in checkpoint two renewed my spirit and provided resolve.


    Gregg made sure I tried his preferred energy drink.

    I had a rough go in that second leg (again) but was feeling better than last year.  I could eat and had plenty of gas left to finish.  The muscles in my neck were beginning to ache and I took a few aspirin, changed into dry socks, ate and drank a bit, and hit the road once again.

    Screen Shot 2017-06-16 at 8.48.38 AM

    At 58 miles, the third leg is the longest.  I was feeling fine but storm clouds were brewing and I began to wonder what it would be like to ride through one…

    Next Post: Part V – Checkpoint Three – coming soon



    0 0

    This blog entry continues the series started with the introduction to Apache CXF JOSE implementation followed recently with the post talking about the signing of HTTP attachments.

    So CXF helps with shipping JOSE filters which can protect the application data by wrapping them into JOSE JWS or JWE envelopes or verify that the data has been properly encrypted and/or signed. In these cases the application code is not even aware that the JOSE processors are involved.

    How would one approach the task of signing/verifying and/or encrypting/decrypting the data directly in the application code ? For example, what if an individual property of the bigger payload needs to be JOSE protected ?

    The most obvious approach is to use either CXF JOSE or the preferred 3rd party library to deal with the JOSE primitives in the application code. This is Option 1. It is a must option if one needs to have a closer control over the JOSE envelope creation process.

    Or you can basically do nearly nothing at all and let CXF handle it for you, this is Option 2. This is a CXF Way Option - make it as easy as possible for the users to embrace the advanced technologies fast. It is not though only about making it easy - but is also about having a more flexible and even portable JOSE-aware code.

    In this case such requirements as "sign only" or "encrypt only" or "sign and encrypt" and similarly for the "verify/decrypt" are not encoded in the code - it is managed at the time of configuring the JOSE helpers from the application contexts (by default they only sign/verify).

    Likewise, the signature and encryption algorithm and key properties are controlled externally.

    I know, it is hard to believe that it can be so easy. Try it to believe it. Enjoy !

    0 0

    Earlier this week, Bloomberg journalist Matthew Leising's in-depth article about the Ethereum smart contracts incident was published on the Bloomberg site: Ether Thief Remains Mystery Year After $55 Million Digital Heist

    It's an interesting article; it's as much about social organizations as it is about computer organizations.

    Once Van de Sande got in touch with Green in Germany, along with two or three others, the foundation was laid for what would become known as the Robin Hood group—white hat hackers who’d devise a bold good-guy plan to drain the remaining DAO. To save the DAO, they’d have to steal the remaining ether, then give it back to its rightful owners.

    And yet as they scrambled that Friday, qualms emerged within the group. “What does it even mean to hack something?” Van de Sande asks. No one knew if what they were about to do was legal. Also, wouldn’t their hack look just as bad as the theft they were trying to stop? Then there were the practical issues. “Who pushes the button?” he remembers wondering. Doing so would initiate their counterattack and alert the community. “Someone has to push the button.”

    The blockchain concepts are absolutely fascinating to me, although I became obsessed with learning about the blockchain in a rather odd way; I arrived there by studying how git compared to Perforce.

    The basic notion of the blockchain is also the under-pinning of the most important versioning software in the world, git. There are many ways in which the two algorithms are similar. The most important way in which they differ is that git is designed for use by people who desire and intend to collaborate; the blockchain is designed for use by people who don't. Danno Ferrin does a much better job of explaining this here.

    Some of the best coverage of Ethereum and the DAO, I think, comes from Bloomberg's Matt Levine, who has been writing about this topic for several years, including this excellent article from a year ago: Blockchain Company Wants to Reinvent Companies.

    As Levine has pointed out, and continues to point out, efforts like Ethereum and the DAO are full of algorithms and computers and science, but they are also inevitably inter-twined with the social interactions of the human beings that want to use these algorithms:

    Smart contracts are cool! Companies are weird bundles of contractual relationships that have become stereotyped and calcified over time, and re-imagining those relationships for a new and more technology-enabled age is a good project. But companies aren't just networks of contracts; they aren't pure agreements negotiated freely between willing participants and no one else. They are also structures that are embedded in society, with rights and responsibilities that are regulated by background rules as well as by contracts. The blockchain-y reinvention of everything in the financial world -- money, contracts, companies -- is fascinating and impressive and, viewed from a certain angle, adorable. But sometimes it could stand to learn from what has gone before. After all, the elements of finance -- money, contracts, companies -- have already been invented. Perhaps their historical development might hold some lessons for their re-inventors.

    Note that, when it comes to re-inventing, you'll see that many of the links from Levine's year-old article no longer work. Web sites get re-invented all the time, as people change their minds about what they think and what they want to say.

    With git and Perforce, recognizing that software is not just pure science, but exists to serve the goals of the human beings who use it, lends depth and nuance to the analysis and comparison. Yes, it's cool that everything is a SHA; on the other hand, have you ever looked at two indirectly-related commit SHAs and tried to understand the underlying history that led those repositories to get to those states? Less-efficient systems may be much easier for humans to use.

    Blockchain systems often suffer from similar chasms of (un-)usability, as Leising observes:

    Who, exactly, were they at war with?

    No one really knows, but there are some clues. One address the attacker used is 0xF35e2cC8E6523d683eD44870f5B7cC785051a77D. Got that? Like everything else in a blockchain, a user’s address is an anonymous string of characters. But every address leaves behind a history on the blockchain that’s open for examination. Not that it makes sense to 99.9 percent of humankind, but Green gets it.

    It's just an algorithm, it's just code, and it is completely accurate to note that there is a complete "history on the blockchain that's open for examination." (It's just as true with git.)

    But as Levine points out, the most interesting aspects of all of this are not really the technical ones, but the human ones:

    "What does it even mean to hack something" seems to be the key philosophical question of the DAO episode, and I submit to you that you probably don't want to invest your life savings in projects that raise philosophical questions like that. If your bank ever said to you "well, what does it even mean to hack something," you would worry. You know what it means! It's just that, with the DAO, the code didn't know what it means, and the whole point of the DAO was to substitute the code's judgment for yours.

    And here, then, is one of the crucial differences between using git, and using a blockchain system. With git, if somebody does something disruptive, like a merge instead of a rebase, or an unwanted force push, the community using that particular collection of repositories collaborates and cooperates to repair the damage.

    But blockchains are intentionally intended for situations where the users explicitly do NOT collaborate and cooperate. And, in Ethereum, there is a challenge, because some people viewed the hack as damage, and wanted to undo it, where others did not, leading to the situation described by Leising:

    Then something else unexpected happened. The original ethereum blockchain, the one with the DAO attack in it, kept growing. Imagine a hard fork is a branch of a tree that sprouts in a different direction at the end of the main limb. The end of that limb is supposed to wither after a hard fork, but here it continued to grow as a small group of users continued to process transactions on that version of the blockchain. Instead of dying, this became a second form of ethereum, quickly dubbed ethereum classic, complete with a digital currency that now had value. Even in the science fiction world of blockchain, this was an unprecedented turn of events.

    Computers are fascinating. Algorithms and software are fascinating.

    People are more fascinating still.

    0 0

    Note: This post is about my second Dirty Kanza 200 experience on June 3, 2017.

    It’s broken into seven parts:

    Part I – Prep / Training

    Part II – Preamble

    Part III – Starting Line

    Part IV – Checkpoint One

    Part V – Checkpoint Two

    Part VI – Checkpoint Three

    Part VII – Finish Line – coming soon

    Don’t Worry Be Happy

    My thoughts as I roll out of Eureka @ 3:30pm…

    • Thirty minutes at a checkpoint is too long, double the plan, but was overheated and feel much better now.
    • I’m enjoying myself.
    • It’s only a hundred miles back to Emporia, I could do that in my sleep.
    • What’s that a storm cloud headed our way?  It’s gonna feel good when it gets here.

    Mud & Camaraderie Mix

    That first century was a frantic pace and there’s not much time or energy for team building.  We help each other out, but it’s all business.

    The second part is when stragglers clump into semi-cohesive units.   It’s only natural and in any case, foolish to ride alone.  A group of riders will always be safer than one, assuming everyone does their job properly.  Each new set of eyes brings another brain to identify and solve problems.

    There’s Jim, who took a few years off from his securities job down in Atlanta, Georgia to help his wife with their Montessori school, and train for this race.  He and I teamed up during the first half of the third leg.  As the worst of the thunderstorms rolled over.

    Before we crossed the US hiway 54, a rider was waiting to be picked up by her support team.  Another victim of muddy roads, a derailleur twisted, bringing an early end to a long day.  We stopped, checked and offered encouragement as a car whizzed by us.

    “That’s a storm chaser!!”, someone called out, leaving me to wonder just how bad these storms were gonna get.

    Derrick, is an IT guy from St. Joseph, Missouri, riding a single-speed bike on his way to a fifth finish, and with it a Goblet commemorating 1000 miles of toil.

    We rode for a bit at the end of the third, right at dusk.  My GPS, up to now worked flawlessly had changed into the nightime display mode and I could no longer make out which lines to follow, missed a turn and heard the buzzer telling me I’d veered off course.

    I stopped and pulled out my cue sheets.  Those were tucked safely and sealed to stay nice and dry.  What, I forgot to seal, its pages wet, stuck together and useless?

    I was tired and let my mind drift.  Why didn’t I bring a headlamp on this leg?  I’d be able to read the nav screen better.  And where is everybody?  How long have I been on the wrong path?  Am I lost?

    Be calm.  Get your focus and above all think.  What about my phone, I can get to the maps from it.  It’s almost dead but there’s plenty of reserve power available.

    Just then Derrick’s dim headlight appeared in the distance.  He stopped and we quietly discussed my predicament.  For some reason his GPS device couldn’t figure that turn out either.  It was then we noticed tire tracks off to our right, turned and got back on track, both nav devices mysteriously resumed working once again.

    Jeremy is the service manager at one of the better bike shops in Topeka, Kansas.  He’s making a third attempt.  Two years ago, he broke down in what turned into a mudfest.  Last year, he completed the course, but twenty minutes past due and didn’t make the 3:00 am cutoff.

    His bike was a grinder of sorts with some fats.  It sounded like a Mack truck on the downhills, but geared like a mountain goat on the uphills.  I want one of them bikes.  Going to have to look him up at that bike shop one day.

    Last year I remembered him lying at the roadside, probably ten maybe fifteen miles outside of Emporia.

    “You alright?”, we stopped and asked.  It was an hour or more past midnight and the blackest of night.

    “Yeah man, just tired, and need to rest a bit.  You guys go on, I’m fine”, he calmly told us.

    There’s the guy from Iowa, who normally wouldn’t be at the back-of-the-pack (with us), but his derailleur snapped and he’d just converted to a single-speed as I caught up with him, and his buddy.  This was a first attempt for both.  They’d been making good until the rains hit.

    Or the four chicks, from where I do not know, who were much faster than I, but somehow kept passing me.  How I would get past them again remains a mystery.

    Also, all of the others, whose names can’t be placed, but the stories can…


    Seven miles into that third leg came the rain.  It felt good, but introduced challenges.  The roads become slippery and a rider could easily go down.  They become muddy and the bike very much wants to break down.

    Both are critical risk factors in terms of finishing.  One’s outcome much worse than the other.

    Fortunately, both problems have good solutions.  The first, slow down the descents, pick through the rocks, pools of mud and water — carefully.  If in doubt stop and walk a section, although I never had to on this day, except for that one crossing with peanut butter on the other side.

    By the way, these pictures that I’m posting are from the calmer sections.  It’s never a good idea to stop along a dangerous roadside just to take one.  That will create a hazard for the other riders, who then have to deal with you in their pathways which limits their choices for a good line.  When the going is tricky, keep it moving, if possible to do so safely.

    The second problem means frequent stops to flush the grit from the drivetrains.  When it starts grinding, it’s time to stop and flush.  Mind the grind.  Once I pulled out two centimeter chunks of rocks lodged in the derailleurs and chain guards.

    Use whatever is on hand.  River, water, bottles, puddles.  There was mud — everywhere.  In the chain, gears and brakes.  It’d get lodged in the pedals and cleats of our shoes making it impossible to click in or (worse) to click out.  I’d use rocks to remove other rocks or whatever is handy and/or expedient.  It helps to be resourceful at times like this.  That’s not a fork, it’s an extended, multi-pronged, mud and grit extraction tool.

    The good folks alongside the road were keeping us supplied with plenty of water.  It wasn’t needed for hydration, but for maintenance.  I’d ask before using it like this, to not offend them.  Pouring their bottles of water over my bike, but they understood and didn’t seem to mind.

    We got rerouted once because the water crossing decided it wanted to be a lake.  This detour added a couple of miles to a ride that was already seven over two hundred.

    The rain made for slow but I was having a good time and didn’t want the fun to end.

    Enjoy this moment.  Look over there, all the flowers growing alongside the road.  The roads were still muddy but the fields were clean and fresh, the temperatures were cool.


    wild flowers along the third leg

    Madison (once again)

    Rolled in about 930p under the cover of night.


    930p @ Madison CP3

    After all that fussing over nameplates in the previous leg and found out it was mounted incorrectly.  It partially blocked the headlight beam and had to be fixed.


    Cheri lends a hand remounting the nameplate so I can be a happy rider

    It was Cheri’s second year doing support.  Last year it was her and Kelly crewing for Gregg and I.  This year, she and Gregg came as well.  As I said earlier, the best part of this race is experiencing it with friends and family.

    I was in good spirits, but hungry, my neck ached, and my bike was in some serious need of attention.  All of this was handled with calm efficiency by Kelly & Co.

    Kyle, who’s an RN, provided medical support with pain relievers and ice packs.  They knew I liked pizza late in the race and Gregg handed some over that had just been pulled fresh from the oven, across the street, at the EZ-mart. It may not sound like much now, but gave me the needed energy boost, from something that doesn’t get squeezed out of a tube.

    As soon as Cheri finished the nameplate, Gregg got the drivetrain running smoothly once again.

    All the while, Kelly and Mom were assisting and directing.  There’s the headlamp needing to be mounted, fresh battery packs, change to the clear lens on the glasses, socks, gloves, cokes, energy drinks, refilling water tanks, electrolytes, gels and more.  There’s forty-some to go, total darkness, unmarked roads.  Possibly more mud on the remaining B roads.  Weather forecast clear and mild.

    Let’s Finish This

    “Who are you riding with?”, Gregg called out as I was leaving.  He ran alongside for a bit, urging me on.


    Gregg runs alongside as I leave CP3

    “Derrick and I are gonna team up”, I called back, which was true, that was the plan as we rolled into town.  Now I just had to find him.  Madison was practically deserted at this hour, its checkpoint regions, i.e. red, green, blue, orange, were spread out, and what color did he say he was again??

    Twenty two minutes spent refueling at checkpoint three and into the darkness again.  That last leg started @ 10 pm with 45 miles to go.  I could do that in my sleep, may need to.

    Screen Shot 2017-06-17 at 9.32.30 PM

    Next Post: Part VII – Finish Line – coming soon


    0 0

    We've had an extensive demonstration of how to enable Swagger UI for CXF endpoints returning Swagger documents for a while but the only 'problem' was that our demos only showed how to unpack a SwaggerUI module into a local folder with the help of a Maven plugin and make these unpacked resources available to browsers.
    It was not immediately obvious to the users how to activate SwaggerUI and with the news coming from a SpringBoot land that apparently it is really easy over there to do it it was time to look at making it easier for CXF users.
    So Aki, Andriy and myself talked and this is what CXF 3.1.7 users have to do:

    1. Have Swagger2Feature activated to get Swagger JSON returned
    2. Add a swagger-ui dependency  to the runtime classpath.
    3. Access Swagger UI

    For example, run a description_swagger2 demo. After starting a server go to the CXF Services page and you will see:

    Click on the link and see a familiar Swagger UI page showing your endpoint's API.

    Have you wondered what do some developers mean when they say it is a child's play to try whatever they have done ? You'll find it hard to find a better example of it after trying Swagger UI with CXF 3.1.7 :-)

    Note in CXF 3.1.8-SNAPSHOT we have already fixed it to work for Blueprint endpoints in OSGI (with the help from Łukasz Dywicki).  SwaggerUI auto-linking code has also been improved to support some older browsers better.

    Besides, CXF 3.1.8 will also offer a proper support for Swagger correctly representing multiple JAX-RS endpoints based on the fix contributed by Andriy and available in Swagger 1.5.10 or when API interface and implementations are available in separate (OSGI) bundles (Łukasz figured out how to make it work).

    Before I finish let me return to the description_swagger2 demo. Add a cxf-rt-rs-service-description dependency to pom.xml. Start the server and check the services page:

    Of course some users do and will continue working with XML-based services and WADL is the best language available around to describe such services. If you click on a WADL link you will see an XML document returned. WADLGenerator can be configured with an XSLT template reference and if you have a good template you can get UI as good as this Apache Syncope document.

    Whatever your data representation preferences are, CXF will get you supported.


    0 0

    Recent blog posts have described how to set up authorization for Apache HBase using Apache Ranger. However the posts just covered inputing and reading data using the HBase Shell. In this post, we will show how Talend Open Studio for Big Data can be used to read data stored in Apache HBase. This post is along the same lines of other recent tutorials on reading data from Kafka and HDFS.

    1) HBase setup

    Follow this tutorial on setting up Apache HBase in standalone mode, and creating a 'data' table with some sample values using the HBase Shell.

    2) Download Talend Open Studio for Big Data and create a job

    Now we will download Talend Open Studio for Big Data (6.4.0 was used for the purposes of this tutorial). Unzip the file when it is downloaded and then start the Studio using one of the platform-specific scripts. It will prompt you to download some additional dependencies and to accept the licenses. Click on "Create a new job" called "HBaseRead". In the search bar on the right-hand side, enter "hbase" and hit enter. Drag "tHBaseConnection" and "tHBaseInput" onto the palette, as well as "tLogRow".

    "tHBaseConnection" is used to set up the connection to "HBase", "tHBaseInput" uses the connection to read data from HBase, and "tLogRow" will log the data that was read so that we can see that the job ran successfully. Right-click on "tHBaseConnection" and select "Trigger/On Subjob Ok" and drag the resulting arrow to the "tHBaseInput" component. Now right click on "tHBaseInput" and select "Row/Main" and drag the arrow to "tLogRow".
    3) Configure the components

    Now let's configure the individual components. Double click on "tHBaseConnection" and select the distribution "Hortonworks" and Version "HDP V2.5.0" (from an earlier tutorial we are using HBase 1.2.6). We are not using Kerberos here so we can skip the rest of the security configuration. Now double click on "tHBaseInput". Select the "Use an existing connection" checkbox. Now hit "Edit Schema" and add two entries to map the column we created in two different column families: "c1" which matches DB "col1" of type String, and "c2" which matches DB "col1" of type String.

    Select "data" for the table name back in tHBaseInput and add a mapping for "c1" to "colfam1", and "c2" to "colfam2".

    Now we are ready to run the job. Click on the "Run" tab and then hit the "Run" button. You should see "val1" and "val2" appear in the console window.

    0 0

    Since we created our hard fork of Spotify’s great repair tool, Reaper, we’ve been committed to make it the “de facto” community tool to manage repairing Apache Cassandra clusters.
    This required Reaper to support all versions of Apache Cassandra (starting from 1.2) and some features it lacked like incremental repair.
    Another thing we really wanted to bring in was to remove the dependency on a Postgres database to store Reaper data. As Apache Cassandra users, it felt natural to store these in our favorite database.

    Reaper 0.6.1

    We are happy to announce the release of Reaper 0.6.1.

    Apache Cassandra as a backend storage for Reaper was introduced in 0.4.0, but it appeared that it was creating a high load on the cluster hosting its data.
    While the Postgres backend could rely on indexes to search efficiently for segments to process, the C* backend had to scan all segments and filter afterwards. The initial data model didn’t account for the frequency of those scans, which generated a lot of requests per seconds once you had repairs with hundreds (if not thousands) of segments.
    Then it seems, Reaper was designed to work on clusters that do not use vnodes. Computing the number of possible parallel segment repairs for a job used the number of tokens divided by the replication factor, instead of using the number of nodes divided by the replication factor.
    This lead to create a lot of overhead with threads trying and failing to repair segments because the nodes were already involved in a repair operation, each attempt generating a full scan of all segments.

    Both issues are fixed in Reaper 0.6.1 with a brand new data model which requires a single query to get all segments for a run, the use of timeuuids instead of long ids in order to avoid lightweight transactions when generating repair/segment ids and a fixed computation of the number of possible parallel repairs.

    The following graph shows the differences before and after the fix, observed on a 3 nodes cluster using 32 vnodes :

    The load on the nodes is now comparable to running Reaper with the memory backend :

    This release makes Apache Cassandra a first class citizen as a Reaper backend!

    Upcoming features with the Apache Cassandra backend

    On top of not having to administer yet another kind of database on top of Apache Cassandra to run Reaper, we can now better integrate with multi region clusters and handle security concerns related to JMX access.

    First, the Apache Cassandra backend allows us to start several instances of Reaper instead of one, bringing it fault tolerance. Instances will share the work on segments using lightweight transactions and metrics will be stored in the database. On multi region clusters, where the JMX port is closed in cross DC communications, it will give the opportunity to start one or more instances of Reaper in each region. They will coordinate together through the backend and Reaper will still be able to apply backpressure mechanisms, by monitoring the whole cluster for running repairs and pending compactions.

    Next, comes the “local mode”, for companies that apply strict security policies for the JMX port and forbid all remote access. In this specific case, a new parameter was added in the configuration yaml file to activate the local mode and you will need to start one instance of Reaper on each C* node. Each instance will then only communicate with the local node on and ignore all tokens for which this node isn’t a replica.

    Those feature are both available in a feature branch that will be merged before the next release.

    While the fault tolerant features have been tested in different scenarios and considered ready for use, the local mode still needs a little bit of work before usage on real clusters.

    Improving the frontend too

    So far, we hadn’t touched the frontend and focused on the backend.
    Now we are giving some love to the UI as well. On top of making it more usable and good looking, we are pushing some new features that will make Reaper “not just a tool for managing repairs”.

    The first significant addition is the new cluster health view on the home screen :

    One quick look at this screen will give you the nodes individual status (up/down) and the size on disk for each node, rack and datacenter of the clusters Reaper is connected to.

    Then we’ve reorganized the other screens, making forms and lists collapsible, and adding a bit of color :

    All those UI changes were just merged into master for your testing pleasure, so feel free to build, deploy and be sure to give us feedback on the reaper mailing list!

    0 0

    The Oakland Fire Department has released their official report on last December's Ghost Ship Fire: Origin and Cause Report: Incident # 2016-085231.

    The report is long, detailed, thorough, and terribly, terribly sad.

    It is vividly illustrated with many pictures, which are simultaneously fascinating and heart-breaking.

    In the end, the report accepts the limits of what is known:

    No witnesses to the incipient stage of the fire were located. Based on witness statements and analysis of fire patterns, an area of origin was identified in the northwest area of the ground floor of the warehouse. In support of this hypothesis, fire patterns and fire behavior were considered, including ventilation effects of door openings, and the fuel load, consisting of large amounts of non-traditional building materials. This analysis was challenged with alternate hypotheses of fire origins, away from, or communicating to an area remote from, the immediate area of fire origin. Several potential ignition sources were considered. No conclusive determination of the initial heat source or the first materials ignited was made. The fire classification is UNDETERMINED.

    In their Pulitzer Prize-winning coverage of the fire, At least nine dead, many missing in Oakland warehouse fire ,the East Bay Times highlighted the eccentricities of the collective's building, details which are thoroughly corroborated by the OFD's detailed report.

    Alemany had advertised on Facebook and Craigslist looking for renters seeking "immediate change and loving revolution," who enjoyed "poetics, dramatics, film, tantric kitten juggling and nude traffic directing." He described it as 10,000 square feet of vintage redwood and antique steel "styled beyond compare."

    His 1951 purple Plymouth remained parked Saturday in front of the building that burned so hot, the “Ghost Ship” letters painted across the front had all but melted away.

    "They are ex-Burning Man people and had their kids in the place -- three kids running around with no shoes," said DeL Lee, 34, who lived there for three months two years ago. "It was nuts."

    He described the place as a filthy firetrap, with frequent power outages, overloaded outlets, sparks and the smell of burning wire. A camping stove with butane tanks served as the kitchen, and a hole had been chiseled through the concrete wall to access the bathroom at the adjoining automotive repair shop next door.

    The staircase, which had two switchbacks to get to the second floor, was built of pallets, plywood and footholds -- like a ship’s gangplank -- and was like "climbing a fort" to get up and down, say people who had visited the building.

    Pianos and old couches doubled as room dividers. Pallets covered with shingles and elaborate trim formed sculptural walls. Often, Lee said, the place was filled with the sounds of sawing and hammering as Alemany continued to build.

    And the OFD report confirms that chaos:

    The front staircase was located along the east wall at the front of the structure. It was constructed of various wooden planks and wooden studs, as well as portions of wooden pallets at its top where it accessed the second floor. One of two bathrooms was elevated slightly off ground level where the lower staircase landing was located. The orientation of the staircase was such that it first led eastward where it bordered the east wall, then turned north (rearward), where it then turned slightly west at the very top of the staircase at the second floor.

    As the Mercury News observes, releasing the report is a milestone but it's not the last we'll hear of this; the next steps will involve the courts:

    Almena and the collective’s creative director, Max Harris, are charged with 36 counts of involuntary manslaughter in Alameda County Superior Court. Prosecutors charge the men knowingly created a fire trap and invited the public inside. Lawyers for the two men say their clients are being used as scapegoats and say the building owner Chor Ng should be facing criminal charges.

    Ng, Almena, PG&E, are named in a wrongful death lawsuit that is in its early stages. The City of Oakland is expected to be named in the suit as soon as this week.

    “Of course, we’d like to have them say what the cause of the fire is but we also have our experts and they are not done with their analysis. I don’t think it’s the end of the story,” said Mary Alexander, the lead attorney for the victim’s families.

    Meanwhile, as Rick Paulas writes at The Awl, the media attention has already brought many changes to the "artist collective" aspect of the Bay Area and beyond: How The Media Mishandled The Ghost Ship Fire

    “It was a tragedy, then it became a tragedy on an entirely different spectrum,” said Friday. “The original was the loss of such vibrant, wonderful people. And that was forgotten and it just became about housing ordinances, and this witch hunt of warehouses.”

    The hunt continues, not just in Oakland, but around the country. City governments — rather than focusing efforts on bringing warehouses up to code, of empathizing with tenants forced to live in unsafe conditions due to increasing rents and lower wages, hell, even pragmatically understanding the added value these fringe residents add to a city’s cultural cachet — are simply closing down venues, then moving on to close down the next. Two days after Ghost Ship, the tenants of Baltimore’s iconic Bell Foundry were evicted. A week after that, the punk venue Burnt Ramen in Richmond, CA was shut down. On April 27th, eight people living in an artist collective in San Francisco were evicted.

    The cities say they don’t want another Ghost Ship, implying they mean another massive loss of life. But the speed at which they’re performing these evictions — and the lack of solutions they’re offering to those displaced — suggests that what they really don’t want is another Ghost Ship media event: vultures descending, camera lights illuminating the dark corners of their own institutional failures.

    It's important that we not forget the Ghost Ship tragedy, but it's a hard story to (re-)read and (re-)consider, with plenty of sadness to go around.

    0 0

    Note: This post is about my second Dirty Kanza 200 experience on June 3, 2017.

    It’s broken into seven parts:

    Part I – Prep / Training

    Part II – Preamble

    Part III – Starting Line

    Part IV – Checkpoint One

    Part V – Checkpoint Two

    Part VI – Checkpoint Three

    Part VII – Finish Line


    I went looking for Derrick but couldn’t find him.  A woman, found out later his wife…

    “Are you John?”, she asked.

    I replied with my name and didn’t make the connection.  I’d forgotten the color of his support team and he got my name wrong so that made us even.

    He caught up ten miles later, by then chasing the fast chicks.  I called out as they zoomed past, wished them well.  This is how it works.  Alliances change according to the conditions and needs from one moment to the next.

    A lone rider stopped at the edge of downtown — Rick from Dewitt, Arkansas.  He was ready for takeoff.

    “You headed out, how bout we team up?”, I asked matter-of-factly.  The deal was struck and then there were two.

    Eventually, maybe twenty miles later, we picked up Jeremy, which made three.  It worked pretty well.  Not much small talk, but lots of operational chatter.  You’d thought we were out on military maneuvers.

    • “Rocks on left.”
    • “Mud — go right!”
    • “Off course, turning around.”
    • “Rough! Slowing!”

    There were specializations.  For example, Jeremy was the scout.  His bike had fat tires and so he’d bomb the downhills, call back to us what he saw, letting us know of the dangers.  Rick did most of the navigating.  I kept watch on time, distance and set the pace.

    By this time we were all suffering and made brief stops every ten miles or so.  We’d agreed that it was OK, had plenty of time, and weren’t worried.

    Caught up with Derrick six miles from home.  Apparently he couldn’t keep up with the fast chicks either, but gave it the college try, and we had a merry reunion.

    We rolled over the finish line somewhat past 2:00 am.  Here’s the video feed:


    Rick and I @ the FL

    My support team was there along with a smattering of hearty locals to cheer us and offer congratulations.

    Jeremy, Rick and I had a brief moment where we congratulated each other before Lelan handed over our Breakfast Club finishers patches and I overheard Rick in his southern drawl…

    “I don’t care if it does say breakfast club on there.”

    Next were the hugs and pictures with my pit crew and I was nearly overcome with emotion.  Felt pretty good about the finish and I don’t care if it says breakfast club on there either.


    The Pit Crew, l to r, Me, Gregg, Kelly, Janice, Cheri, Kyle


    In addition to my pit crew…

    My wife Cindy deserves most of the credit.  She bought the bike four years ago that got me all fired up again about cycling.  Lots of times when I’m out there riding I should be home working.  Throughout this she continues to support without complaint.  Thanks baby, you’re the best, I love you.

    Next, are the guys at the bike shop — Arkansas Cycle and Fitness, back home in Little Rock.  They tolerate abysmal mechanical abilities, patiently listen to requirements, and teach when necessary (often).  Time and again the necessary adjustments were made to correct the issues I was having with the bike.  They’ve encouraged and cheered, offering suggestions and help along the way.

    Finally, my cycling buddies — the Crackheads.  Truth be known they’re probably more trail runners than cyclists, but they’re incredible athletes from whom I’ve learned much.  In the summertime, when the skeeters and chiggers get too bad for Arkansas trail running, they come out and ride which makes me happy.

    Screen Shot 2017-06-20 at 12.01.19 AM

    The End

    0 0

    In George R R Martin’s books “A Song of Fire and Ice” (which you may know by the name “A Game of Thrones”), the people of Braavos,
    have a saying – “Valar Morghulis” – which means “All men must die.” As you follow the story, you quickly realize that this statement is not made in a morbid, or defeatist sense, but reflects on what we must do while alive so that the death, while inevitable, isn’t meaningless. Thus, the traditional response is “Valar Dohaeris” – all men must serve – to give meaning to their life.

    So it is with software. All software must die. And this should be viewed as a natural part of the life cycle of software development, not as a blight, or something to be embarrassed about.

    Software is about solving problems – whether that problem is calculating launch trajectories, optimizing your financial investments, or entertaining your kids. And problems evolve over time. In the short term, this leads to the evolution of the software solving them. Eventually, however, it may lead to the death of the software. It’s important what you choose to do next.

    You win, or you die

    One of the often-cited advantages of open source is that anybody can pick up a project and carry it forward, even if the original developers have given up on it. While this is, of course, true, the reality is more complicated.

    As we say at the Apache Software Foundation, “Community > Code”. Which is to say, software is more than just lines of source code in a text file. It’s a community of users, and a community of developers. It’s documentation, tutorial videos, and local meetups. It’s conferences, business deals and interpersonal relationships. And it’s real people solving real-world problems, while trying to beat deadlines and get home to their families.

    So, yes, you can pick up the source code, and you can make your changes and solve your own problems – scratch your itch, as the saying goes. But a software project, as a whole, cannot necessarily be kept on life support just because someone publishes the code publicly. One must also plan for the support of the ecosystem that grows up around any successful software project.

    Eric Raymond just recently released the source code for the 1970s
    computer game Colossal Cave Adventure on Github. This is cool, for us greybeard geeks, and also for computer historians. It remains to be seen whether the software actually becomes an active open source project, or if it has merely moved to its final resting place.

    The problem that the software solved – people want to be entertained – still exists, but that problem has greatly evolved over the years, as new and different games have emerged, and our expectations of computer games have radically changed. The software itself is still an enjoyable game, and has a huge nostalgia factor for those of us who played it on greenscreens all those years ago. But it doesn’t measure up to the alternatives that are now available.

    Software Morghulis. Not because it’s awful, but because its time has

    Winter is coming

    The words of the house of Stark in “A Song of Fire and Ice”, are “Winter is coming.” As with “Valar Morghulis,” this is about planning ahead for the inevitable, and not being caught surprised and unprepared.

    How we plan for our own death, with insurance, wills, and data backups, isn’t morbid or defeatist. Rather, it is looking out for those that will survive us. We try to ensure continuity of those things which are possible, and closure for those things which are not.

    Similarly, Planning ahead for the inevitable death of a project isn’t defeatist. Rather, it shows concern for the community. When a software project winds down, there will often be a number of people who will continue to use it. This may be because they have built a business around it. It may be because it perfectly solves their particular problem. And it may be that they simply can’t afford the time, or cost, of migrating to something else.

    How we plan for the death of the project prioritizes the needs of this community, rather than focusing merely on the fact that we, the developers, are no longer interested in working on it, and have moved on to something else.

    At Apache, we have established the Attic as a place for software projects to come to rest once the developer community has dwindled. While the project itself may reach a point where they can no longer adequately shepherd the project, the Foundation as a whole still has a responsibility to the users, companies, and customers, who rely on the software itself.

    The Apache Attic provides a place for the code, downloadable releases, documentation, and archived mailing lists, for projects that are no longer actively developed.

    In some cases, these projects are picked up and rejuvenated by a new community of developers and users. However, this is uncommon, since there’s usually a very good reason that a project has ceased operation. In many cases, it’s because a newer, better solution has been developed for the problem that the project solved. And in many cases, it’s because, with the evolution of technology, the problem is no longer important to a large enough audience.

    However, if you do rely on a particular piece of software, you can rely on it always being available there.

    The Attic does not provide ongoing bug fixes or make additional releases. Nor does it make any attempt to restart communities. It is
    merely there, like your grandmother’s attic, to provide long-term storage. And, occasionally, you’ll find something useful and reusable as you’re looking through what’s in there.

    Software Dohaeris

    The Apache Software Foundation exists to provide software for the public good. That’s our stated mission. And so we must always be looking out for that public good. One critical aspect of that is ensuring that software projects are able to provide adequate oversight, and continuing support.

    One measure of this is that there are always (at least) three members of the Project Management Committee (PMC) who can review commits, approve releases, and ensure timely security fixes. And when that’s no longer the case, we must take action, so that the community depending on the code has clear and correct expectations of what they’re downloading.

    In the end, software is a tool to accomplish a task. All software must serve. When it no longer serves, it must die.

    0 0
    0 0

    Is it possible that I am the first person to tell you about Geraldine DeRuiter's new book: All Over the Place: Adventures in Travel, True Love, and Petty Theft?

    If I am, then yay!

    For this is a simply wonderful book, and I hope everybody finds out about it.

    As anyone who has read the book (or has read her blog) knows, DeRuiter can be just screamingly funny. More than once I found myself causing a distraction on the bus, as my fellow riders wondered what was causing me to laugh out loud. Many books are described as "laugh out loud funny," but DeRuiter's book truly is.

    Much better, though, are the parts of her book that aren't the funny parts, for it is here where her writing skills truly shine.

    DeRuiter is sensitive, perceptive, honest, and caring. Best of all, however, is that she is able to write about affairs of the heart in a way that is warm and generous, never cloying or cringe-worthy.

    So yes: you'll laugh, you'll cry, you'll look at the world with fresh new eyes. What more could you want from a book?

    All Over the Place is all of that.

    0 0

    0 0

    0 0

    Apache Syncope has recently added SSO support for its REST services in the 2.0.3 release. Previously, access to the REST services of Syncope was via HTTP Basic Authentication. From the 2.0.3 release, SSO support is available using JSON Web Tokens (JWT). In this post, we will look at how this works and how it can be configured.

    1) Obtaining an SSO token from Apache Syncope

    As stated above, in the past it was necessary to supply HTTP Basic Authentication credentials when invoking on the REST API. Let's look at an example using curl. Assume we have a running Apache Syncope instance with a user "alice" with password "ecila". We can make a GET request to the user self service via:

    • curl -u alice:ecila http://localhost:8080/syncope/rest/users/self
    It may be inconvenient to supply user credentials on each request or the authentication process might not scale very well if we are authenticating the password to a backend resource. From Apache Syncope 2.0.3, we can instead get an SSO token by sending a POST request to "accessTokens/login" as follows:
    • curl -I -u alice:ecila -X POST http://localhost:8080/syncope/rest/accessTokens/login
    The response contains two headers:
    • X-Syncope-Token: A JWT token signed according to the JSON Web Signature (JWS) spec.
    • X-Syncope-Token-Expire: The expiry date of the token
    The token in question is signed using the (symmetric) "HS512" algorithm. It contains the subject "alice" and the issuer of the token ("ApacheSyncope"), as well as a random token identifier, and timestamps that indicate when the token was issued, when it expires, and when it should not be accepted before.

    The signing key and the issuer name can be changed by editing '' and specifying new values for 'jwsKey' and 'jwtIssuer'. Please note that it is critical to change the signing key from the default value! It is also possible to change the signature algorithm from the next 2.0.4 release via a custom 'securityContext.xml' (see here). The default lifetime of the token (120 minutes) can be changed via the "jwt.lifetime.minutes" configuration property for the domain.

    2) Using the SSO token to invoke on a REST service

    Now that we have an SSO token, we can use it to invoke on a REST service instead of specifying our username and password as before, e.g.:
    • curl -H "X-Syncope-Token: eyJ0e..." http://localhost:8080/syncope/rest/users/self
    The signature is first checked on the token, then the issuer is verified so that it matches what is configured, and then the expiry and not-before dates are checked. If the identifier matches that of a saved access token then authentication is successful.

    Finally, SSO tokens can be seen in the admin console under "Dashboard/Access Token", where they can be manually revoked by the admin user:

    0 0

    0 0

    Most datacenter automation tools operate on the basis of desired state. Desired state describes what should be the end state but not how to get there. To simplify a great deal, if the thing being automated is the speed of a car, the desired state may be “60mph”. How to get there (braking, accelerator, gear changes, turbo) isn’t specified. Something (an “agent”) promises to maintain that desired speed.


    The desired state and changes to the desired state are sent from the orchestrator to various agents in a datacenter. For example, the desired state may be “two apache containers running on host X”. An agent on host X will ensure that the two containers are running. If one or more containers die, then the agent on host X will start enough containers to bring the count up to two. When the orchestrator changes the desired state to “3 apache containers running on host X”, then the agent on host X will create another container to match the desired state.

    Transfer of desired state is another way to achieve idempotence (a problem described here)

    We can see that there are two sources of changes that the agent has to react to:

    1. changes to desired state sent from the orchestrator and
    2. drift in the actual state due to independent / random events.

    Let’s examine #1 in greater detail. There’s a few ways to communicate the change in desired state:

    1. Send the new desired state to the agent (a “command” pattern). This approach works most of the time, except when the size of the state is very large. For instance, consider an agent responsible for storing a million objects. Deleting a single object would involve sending the whole desired state (999999 items). Another problem is that the command may not reach the agent (“the network is not reliable”). Finally, the agent may not be able to keep up with rate of change of desired state and start to drop some commands.  To fix this issue, the system designer might be tempted to run more instances of the agent; however, this usually leads to race conditions and out-of-order execution problems.
    2. Send just the delta from the previous desired state. This is fraught with problems. This assumes that the controller knows for sure that the previous desired state was successfully communicated to the agent, and that the agent has successfully implemented the previous desired state. For example, if the first desired state was “2 running apache containers” and the delta that was sent was “+1 apache container”, then the final actual state may or may not be “3 running apache containers”. Again, network reliability is a problem here. The rate of change is an even bigger potential problem here: if the agent is unable to keep up with the rate of change, it may drop intermediate delta requests. The final actual state of the system may be quite different from the desired state, but the agent may not realize it! Idempotence in the delta commands helps in this case.
    3. Send just an indication of change (“interrupt”). The agent has to perform the additional step of fetching the desired state from the controller. The agent can compute the delta and change the actual state to match the delta. This has the advantage that the agent is able to combine the effects of multiple changes (“interrupt debounce”). By coalescing the interrupts, the agent is able to limit the rate of change. Of course the network could cause some of these interrupts to get “lost” as well. Lost interrupts can cause the actual state to diverge from the desired state for long periods of time. Finally, if the desired state is very large, the agent and the orchestrator have to coordinate to efficiently determine the change to the desired state.
    4. The agent could poll the controller for the desired state. There is no problem of lost interrupts; the next polling cycle will always fetch the latest desired state. The polling rate is critical here: if it is too fast, it risks overwhelming the orchestrator even when there are no changes to the desired state; if too slow, it will not converge the the actual state to the desired state quickly enough.

    To summarize the potential issues:

    1. The network is not reliable. Commands or interrupts can be lost or agents can restart / disconnect: there has to be some way for the agent to recover the desired state
    2. The desired state can be prohibitively large. There needs to be some way to efficiently but accurately communicate the delta to the agent.
    3. The rate of change of the desired state can strain the orchestrator, the network and the agent. To preserve the stability of the system, the agent and orchestrator need to coordinate to limit the rate of change, the polling rate and to execute the changes in the proper linear order.
    4. Only the latest desired state matters. There has to be some way for the agent to discard all the intermediate (“stale”) commands and interrupts that it has not been able to process.
    5. Delta computation (the difference between two consecutive sets of desired state) can sometimes be more efficiently performed at the orchestrator, in which case the agent is sent the delta. Loss of the delta message or reordering of execution can lead to irrecoverable problems.

    A persistent message queue can solve some of these problems. The orchestrator sends its commands or interrupts to the queue and the agent reads from the queue. The message queue buffers commands or interrupts while the agent is busy processing a desired state request.  The agent and the orchestrator are nicely decoupled: they don’t need to discover each other’s location (IP/FQDN). Message framing and transport are taken care of (no more choosing between Thrift or text or HTTP or gRPC etc).


    There are tradeoffs however:

    1. With the command pattern, if the desired state is large, then the message queue could reach its storage limits quickly. If the agent ends up discarding most commands, this can be quite inefficient.
    2. With the interrupt pattern, a message queue is not adding much value since the agent will talk directly to the orchestrator anyway.
    3. It is not trivial to operate / manage / monitor a persistent queue. Messages may need to be aggressively expired / purged, and the promise of persistence may not actually be realized. Depending on the scale of the automation, this overhead may not be worth the effort.
    4. With an “at most once” message queue, it could still lose messages. With  “at least once” semantics, the message queue could deliver multiple copies of the same message: the agent has to be able to determine if it is a duplicate. The orchestrator and agent still have to solve some of the end-to-end reliability problems.
    5. Delta computation is not solved by the message queue.

    OpenStack (using RabbitMQ) and CloudFoundry (using NATS) have adopted message queues to communicate desired state from the orchestrator to the agent.  Apache CloudStack doesn’t have any explicit message queues, although if one digs deeply, there are command-based message queues simulated in the database and in memory.

    Others solve the problem with a combination of interrupts and polling – interrupt to execute the change quickly, poll to recover from lost interrupts.

    Kubernetes is one such framework. There are no message queues, and it uses an explicit interrupt-driven mechanism to communicate desired state from the orchestrator (the “API Server”) to its agents (called “controllers”).

    Courtesy of Heptio

    (Image courtesy:

    Developers can use (but are not forced to use) a controller framework to write new controllers. An instance of a controller embeds an “Informer” whose responsibility is to watch for changes in the desired state and execute a controller function when there is a change. The Informer takes care of caching the desired state locally and computing the delta state when there are changes. The Informer leverages the “watch” mechanism in the Kubernetes API Server (an interrupt-like system that delivers a network notification when there is a change to a stored key or value). The deltas to the desired state are queued internally in the Informer’s memory. The Informer ensures the changes are executed in the correct order.

    • Desired states are versioned, so it is easier to decide to compute a delta, or to discard an interrupt.
    • The Informer can be configured to do a periodic full resync from the orchestrator (“API Server”) – this should take care of the problem of lost interrupts.
    • Apparently, there is no problem of the desired state being too large, so Kubernetes does not explicitly handle this issue.
    • It is not clear if the Informer attempts to rate-limit itself when there are excessive watches being triggered.
    • It is also not clear if at some point the Informer “fast-forwards” through its queue of changes.
    • The watches in the API Server use Etcdwatches in turn. The watch server in the API server only maintains a limited set of watches received from Etcd and discards the oldest ones.
    • Etcd itself is a distributed data store that is more complex to operate than say, an SQL database. It appears that the API server hides the Etcd server from the rest of the system, and therefore Etcd could be replaced with some other store.

    I wrote a Network Policy Controller for Kubernetes using this framework and it was the easiest integration I’ve written.

    It is clear that the Kubernetes creators put some thought into the architecture, based on their experiences at Google. The Kubernetes design should inspire other orchestrator-writers, or perhaps, should be re-used for other datacenter automation purposes. A few issues to consider:

    • The agents (“controllers”) need direct network reachability to the API Server. This may not be possible in all scenarios, needing another level of indirection
    • The API server is not strictly an orchestrator, it is better described as a choreographer. I hope to describe this difference in a later blog post, but note that the API server never explicitly carries out a step-by-step flow of operations.