Big Data Isn’t all That Big


Back in ancient history (2004) Google’s Jeff Dean and Sanjay Ghemawat presented their innovative idea for dealing with huge data sets — a novel idea called MapReduce. Jeff and Sanjay presented that a typical cluster was made of hundreds to thousands of machines with 2 CPUs and 2-4 GB RAM each. They presented that in…

If you had a computer with 1 TB of memory an 64 cores you can process most big data step processing very quickly.  AWS will rent you one of these for $7/hr, need something larger, no problem just pay a few more bucks.  It doesn’t have to take days or hours to process big data.

via  DZone Big Data Zone

Scalable Offline-Ready GraphQL Applications with AWS AppSync & React


With GraphQL, you can define the shape of your data in a schema, and GraphQL will fulfill the request with the precise data you need. When you need it. This doesn’t come without a cost. GraphQL adds a lot of complexity to your application by requiring you to define your schemas through types, mutations, and resolvers.

AppSync is a fully managed serverless GraphQL service that fast-tracks your API development. You can define a single type, and AppSync auto-generates a schema, queries, mutators, and subscriptions for you.

Just when you think you have learned one thing, bang, up springs something new and possibly better.  Got to know REST and JSON and, bang, graphQL is flexing it’s three year old muscles.  graphQL has a bit more upfront work to get it working.  Looks like AWS setted up and is providing a service.  This course from is the intro to graphQL on AWS.

 via: Egghead.IO

Moving a JIRA Cloud instance to a Server instance using AWS

JIRACloudLogo.pngWe have been using JIRA On-Demand or JIRA Cloud for a few years now.  It was an easy SaaS decision and helped us get started very quickly.  After a while, we realized that most of the cool add-ons for JIRA are only available to the JIRA Server instance.  Our web applications have an Issue-Track (Critical-Response) form  that users fill out to report a problem or ask for an enhancement.  Currently we take these forms and manually transfer them into JIRA.  The JIRA licenses are too expensive to give all our customers a JIRA user ID’s.  So we designed an integrated JIRA interface into our Issue Track page that talks direct to our JIRA-Cloud  Instance. No more copy-paste.  One of our government customers has a security rule that only allows API usage to servers with a single IP address.  Well, JIRA Cloud instances from Atlassian are actually hosted on Amazon Web Services which uses a pool of IP addresses for better performance which makes sense.  For us to get a single IP we will need to migrate our JIRA Cloud instance to a JIRA server instance.  We have decided to stay as close to the way Atlassian manages their cloud instances so we will be using AWS as well but, with a single IP address.  I will document this journey over time in this Blog, one for some reference documentation and two because I like to share.



GeekMustHave is now on YouTube

Hello and welcome to Geek Must Have YouTube channel. Everyone once and awhile have you come across someone who you know, is going to try and build something cool… that’s me.  This channel is going to be the Geek’s musing about technologies like Electronic Gadgets that blink, buzz, talk, listen and keep you awake at night. Computers of all sizes from the tiny single chip ones all the way up to big servers in the cloud.  Coding from Cobol to JavaScript and everything in between.  Databases starting with the humble flat file, tried and true relational and Big Data, noSQL, REST and JSON.  Communications with wireless, GSM, Bluetooth and amateur radio.  Lots of cheap bits, bobs and do-dads from China.  This channel plans to have mailbags, project builds, deconstructions, reviews, recommendations, tools, and tips

Check out the champion blog at HTTP://GeekMustHave.COM

This a new channel and the Geek needs your help, please click on the subscribe button,
watch the videos and click on the like button, leave comments and questions.

The Geek is busy learning and building stuff, so don’t be upset if the response isn’t immediate.

Thank you and now ….“Let’s build something…”

Install PowerShell on AWS Linux and Tools

awlogo.pngAWS Linux is based on CentOS when considering how to install Windows PowerShell.  This installation is done from the AWS Command Line using SSH into the Public-DNS key for your AWS Linux instance.  The instructions from Microsoft and Amazon are not that easy to follow for AWS Linux instances.  This post summarizes the research and testing done to implement PowerShell on a AWS Linux instance.

You will need to use ‘wget’ to download the most current version of the Powershell installable. Type the following at the AWS Linux command prompt.


The response will look something like

Resolving (
Connecting to (||:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 40391065 (39M) [application/octet-stream]
Saving to: ‘powershell-6.0.0_alpha.9-1.el7.centos.x86_64.rpm’

powershell-6.0.0_al 100%[===================>] 38.52M 26.6MB/s in 1.4s

2017-01-11 19:21:31 (26.6 MB/s) - ‘powershell-6.0.0_alpha.9-1.el7.centos.x86_64.rpm’ saved [40391065/40391065]

Next step is to install this RPM package, type the following at the AWS Linux command prompt.

sudo yum install powershell-6.0.0_alpha.9-1.el7.centos.x86_64.rpm

You will be asked if it is OK to proceed, followed by a lengthy response ending with something that looks like.

Is this ok [y/d/N]: y
Downloading packages:
libunwind-1.1-10.8.amzn1.x86_64.rpm | 72 kB 00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : libunwind-1.1-10.8.amzn1.x86_64 1/2
Installing : powershell-6.0.0_alpha.9-1.x86_64 2/2
Verifying : powershell-6.0.0_alpha.9-1.x86_64 1/2
Verifying : libunwind-1.1-10.8.amzn1.x86_64 2/2

powershell.x86_64 0:6.0.0_alpha.9-1

Dependency Installed:
libunwind.x86_64 0:1.1-10.8.amzn1


Now to test if it worked, enter the following at the command prompt.


If the install worked you will see the standard PowerShell prompt.

Copyright (C) 2016 Microsoft Corporation. All rights reserved.

PS /home/ec2-user>


Now to see what got installed run the following command at the PS prompt

Get-Module -ListAvailable

The response will be a list of the modules installed, you may need more, but this is a good start.

ModuleType Version Name ExportedCommands
---------- ------- ---- ----------------
Manifest Microsoft.PowerShell.Archive {Compress-Archive, Expand-Archive}
Manifest Microsoft.PowerShell.Host {Start-Transcript, Stop-Transcript}
Manifest Microsoft.PowerShell.Management {Add-Content, Clear-Content, Clear-ItemProp...
Manifest Microsoft.PowerShell.Security {Get-Credential, Get-ExecutionPolicy, Set-E...
Manifest Microsoft.PowerShell.Utility {Format-List, Format-Custom, Format-Table, ...
Binary PackageManagement {Find-Package, Get-Package, Get-PackageProv...
Script 3.3.9 Pester {Describe, Context, It, Should...}
Script PowerShellGet {Install-Module, Find-Module, Save-Module, ...
Script 0.0 PSDesiredStateConfiguration {IsHiddenResource, StrongConnect, Write-Met...
Script 1.2 PSReadLine {Get-PSReadlineKeyHandler, Set-PSReadlineKe...

Remember with AWS Linux it’s YUM not APT-GET.

Powershell Tools for AWS

There is a reference for installation of AWS Tools for Powershell at

There is a set of Powershell tools that should be installed.  Enter the following at the PS prompt.

Install-Package -Name AWSPowerShell.NetCore -Source -ProviderName NuGet -ExcludeVersion -Destination ~/.local/share/powershell/Modules

The will result in something that looks like

Name                           Version          Source                           Summary                 
----                           -------          ------                           -------                 
AWSPowerShell.NetCore         https://www.powershellgallery... The AWS Tools for Po...

With this set of Cmdlet’s you can now perform almost all of the AWS services management using PowerShell commands.  You will need to import the module by entering the following at the PS prompt.

import-module AWSPowerShell.NetCore

Then verify that the module actually got loaded


Which should result in something looking like, notice the new binary at the top of the list which is the PowerShell library for AWS

ModuleType Version  Name                            ExportedCommands
---------- -------  ----                            ----------------
Binary AWSPowerShell.NetCore           {Add-AASScalableTarget, Add-ACMCertificateT...
Manifest  Microsoft.PowerShell.Management {Add-Content, Clear-Content, Clear-Item, Cl...
Manifest  Microsoft.PowerShell.Security   {ConvertFrom-SecureString, ConvertTo-Secure...
Manifest  Microsoft.PowerShell.Utility    {Add-Member, Add-Type, Clear-Variable, Comp...
Binary  PackageManagement               {Find-Package, Find-PackageProvider, Get-Pa...
Script     1.2      PSReadLine                      {Get-PSReadlineKeyHandler, Get-PSReadlineOp...

Now check what the version of the AWS Tools for Powershell is by typing the following at the PS prompt.


Which should result in something that looks like the following.

AWS Tools for PowerShell Core
Copyright 2012-2017, Inc. or its affiliates. All Rights Reserved.

Amazon Web Services SDK for .NET
Core Runtime Version
Copyright 2009-2015, Inc. or its affiliates. All Rights Reserved.

Release notes:

This software includes third party software subject to the following copyrights:
- Logging from log4net, Apache License

Congratulations, now you can AWS your heart out using PowerShell scripting commands.