Open Source software pitfalls; why you should test in a very restricted VM:

TL;DR: a free open source software package with buried install routine that tries to AWS Cloud Formation a server cluster… would incur ~$22K/month.


In recent weeks, I’ve been reviewing dozens of open source applications. Some of the stuff I’ve found deeper in the details has been truly amazing.

The best one so far has been an open source map rendering application. The concept is pretty straightforward… it provides a client app and a server app. On the client, a user creates a set of points (coordinates), and the server renders a map for the client.

The client/server model should enable multiple users to utilize low powered mobile devices to gather data and let one server do the heavier processing.

Sounds good; I added it to my “evaluations” list.

My primary goal for these evaluation activities is not to install/run these apps, but rather to collect some apps that I can use as working examples for my own software development “continuing education” projects. I learn more by taking things apart.

After some preliminary reviews, fiddling around with a live online demo site, and using some packaged bitnami/docker type bundles… I git cloned the source code and started exploring.

As I followed a few threads of packages dependencies, I stumbled into a package of utilities and scripts… best described as “deploy an n-tiered high capacity solution.”

Notice that I found this in package dependencies and scripts.

Many app developers provide this kind of info in documentation. Not these folks. It’s deep in the installation/deployment code published in their GitHub repo.

So what happens when someone downloads and deploys the content of their GitHub repo. For most people, I’d expect the process to fail rather quickly.

However, if attempted on a host that is used to admin/deploy stuff on AWS… better hope someone configured AWS roles and account limits/restrictions.

If you’ve ever setup “AWS CLI” to make it easier to administer AWS services… and then ran the command line “aws configure” to set up your keys… well… you should be getting a feeling that a train wreck is leaving the station.

That client/server deployment process buried in the GitHub repo… it contains cloud formation scripts to deploy a “high capacity rendering cluster”.

After I reviewed the target configuration, and looked up the current AWS pricing, I estimated the services turned on would have a monthly cost of about $22K.


That isn’t the only open source repo I’ve found that hides a narly surprise, but it’s currently the top of my list.

A couple others, that take somewhat smaller bite, but nevertheless demonstrate why caution is necessary.

A package that is described as a howto guide walks the user through steps to set up a free Azure Functions demonstration… at the completion, the user account is provisioned with a persistent (always on) service that prices out as $1,803 per month. Unfortunately, the Microsoft Azure admin interface does nothing to warn the user or even ask permission. I only found that little gift by thoroughly reviewing all of the created services afterwards… and promptly deleting them. Had something external interrupted me, and I’d delayed that post config review, the unexpected bill would have been a real kick in the backside.

In another open source git repo, I found what appeared to be a promising little utility for scraping news sites and parsing the results into simple/clean text saved locally for offline reading. Turns out that streamlined reading material comes at a cost. The app’s install routine included a “pip install” of numpy, matplotlib, scipy, and tensorflow. Congratulations to anyone who installed this one… you just donated your machine to the authors machine learning network.


If you’re thinking , “Gee, Wally how many people actually make these kinda mistakes? Don’t detect them? And, don’t clean them up?”

Google search “wasted cloud spend”.

An article published Jan 3rd, 2019 puts the current cloud waste at $14.1 Billion.

iOS GPS Location Tracking

FYI:  Apple has responded to the GPS Location Tracking kerfuffle and announced that they’ll soon be putting out another iOS update which changes how the location services stores data.

Article re location services

Apple Q&A on location services

 

My opinion on the issue:  The GPS feature was a significant factor in our selection of these devices.  Despite the media induced confusion (which has spread to congress), the iPads/iPhones are working appropriately for a GPS device and are not using the location data in any inappropriate or unexpected ways. In fact, a portion of what the iPhone location service is doing was mandated by e911 requirements from Congress and the FCC.

Mobility and Broadband… comments from President and FCC Chairman

Feb 10thObama Administration’s goal: 98 percent of US covered by 4G broadband

Feb 8thFCC moves to redirect phone subsidies to broadband… the FCC Chairman was interviewed by NPR on this topic Thurs Feb 10th, so at least three or four other people have heard about it by now.

Most of the Obama plan is a proposal Congress would need to enact.  The FCC changes to the Universal Service Fund (USF) have been approved by the FCC Commissioners and will move forward unless Congress specifically passes a new law to block them.

It’s possible the WhiteHouse could help both of these things along by setting priorities for Government Agencies like RD, FSA, HUD, DHS, and any other that provide services and funding into rural areas.


An overview of our iPad pilot

Who:  primary pilot users are mobile field office personnel, managers, and GIS professionals.  Additionally, a group of IT personnel supporting the pilot have been issued iPads to assist the field the users and manage the evaluation of the technology.  This group includes members from the Help Desk, Architecture, Messaging, Procurement, and Security teams.

What: Weighing 1.6 lbs, the iPad 3G+64GB includes a 9.7” multi-touch LCD screen with 1024×768 resolution, a 1Ghz Apple A4 processor, and 256MB Ram.  Sensors include ambient light, 3-axis accelerometer,  a Broadcom 4750 Assisted-GPS, and a digital compass.  Wireless connectivity includes 3G HSDPA, 2G EDGE, 802.11 (a/b/g/n), and Bluetooth 2.1+EDR.  Selected accessories include the iPad Camera Connection Kit, the VGA Adapter, and the BlueTooth Keyboard.

When: The iPad pilot is an ongoing activity (additional users will be added pending contract negotiations of corporate AT&T 3G data plans).

Where:  anywhere mobile field office personnel need to be during the performance of their duties.  As part of a larger goal of enabling workforce mobility, this iPad evaluation aims to assist in a transition away from worker dependency upon a specific physically location.

Why:   Use cases selected for currently available features, accessories, applications, and services include:

  • Mobile e-mail – evaluate as a secure alternative to more costly laptops.
  • Mobile productivity applications – evaluate apps for office tasks, document management, and other workflows.
  • Portable reference documents – evaluate as an ereader reference materials.
  • Mobile GIS, including ESRI ArcGIS Mobile – evaluate as a GPS alternative to single purpose equipment such as the Garmin GPSMap 76csx or Nuvi.
  • Thin Client – *dependent upon our IT Citrix and VMware View projects, evaluate as a portable thin client and an alternative to stationary terminals.
  • Mobile consultations, training sessions, or presentations – evaluate as a tool for consulting and training with customers and workers.

iPads for GIS

I get a lot of questions from folks wondering why we’re sending iPads to remote locations.

GIS is our #1 business case for getting these iPads into the field for our users. Each device/user is to receive ESRI’s ArcGIS and MotionX GPS HD. The iPad 3G models contain an internal GPS receiver (Broadcom 4750) which matches the Garmin NUVI.  I discussed the topic of wireless data service availability in an earlier post.

The ArcGIS app can use a number of data sources, including our ESRI ArcGIS servers. Other sources include the USDA Geospatial Data Gateway.  I’ve listed other sources in various posts, and at some point will dedicated an article to describing the data sources available.

The MotionX app matches pretty much all of the Garmin GPS MAP76 features. I own a GPSMap76s and a GPSMAP 76cs (and a few other Garmins) so I can make this comparison with some experience.  The Gamin units with sensors (the “s” designation) have some useful features at high altitudes, in marine applications, and for weather guessing.

Most of the GIS field work our organization is dealing with, those extra Garmin features are just extras.  Our primary concern is recording an accurate track and creating a documentation trail of the property inspection.  The iPhone’s GPS and Camera are very good for this and the iPad’s display features help with reviewing data and completing reports in while still in the field.  The ability to turn around reports from the field is a tremendous financial value for our organization and customers.

The iPads and iPhones aren’t ready to be plugged into a tractor’s navigation array for precision farming applications (yet), but I believe you’ll like the functionality they do provide in the field now.

Mobile Data Services for Working in Rural Areas: remote GIS with iPad/iPhone

I get a lot of questions about using mobile devices in rural areas. Often they are along the lines of complaining about lack of wireless data signal and whether or not it make sense to deploy devices for users in these areas.

I’ve recently driven a lot of the rural areas in the Midwest and South regions of the United States. In many of these remote areas, gravel may be considered “an improved road”. Along the way on these drives, I field tested the GPS and wireless data capabilities of the iPhone and iPad. I also compared the GPS features of these new devices with Garmin GPSMAP76s, GPSMAP76cs, and Nuvi 255w.

In a lot of these areas where folks have anecdotally reported cell coverage as “basically non-existent”, I’ve found that you can usually locate a place to pull over and get an “Edge” connection with data speeds in the range of 14Kbps to 60Kbps. Yes, that’s a far cry from the multi-megabit services we’re coming to expect in Metropolitan/Urban communities. However, it’s not bad if you live/work in a rural area where the alternative is driving ~30 miles back to a house/office with a land line connection with service speeds ranging from 56K dialup to 256K Fractional T1.

Usually the key to getting a usable signal in these areas is to (a) slow down below 45MPH or (b) pull over to a complete stop. In urban areas, customers can often make a connection while moving 70MPH+ within 2 to 3 miles of a service provider’s tower. In a rural areas, a stationary customer can often get a lower speed connection as far out as 7 to 9 miles from the nearest tower. Continuing along the road to a higher location often helps, and GIS Professionals are usually pretty good at looking at the terrain and making educated guesses about where the signal may be a little better.

For a worker whose primary pattern of usage is large emails and very heavy web page sessions, these low speed connections may prove useless. However, for a GIS type person trying to retrieve a set of coordinates from the ESRI ArcGIS server or read the text of a simple email message, these connections could save hours of unnecessary travel.

I should also note, most of these signal/service questions come from managers and IT folks who live in more urban or metropolitan areas. I’ve found that GIS Pros, Surveyors, Engineers, and other who do field work in remote areas already understand these issues. Again, I can’t emphasize enough how much those folks doing the remote field can benefit from any wireless service which saves hours per day of driving to and from land line locations.

As for the GPS features of the iPad/iPhone vs Garmin products, the results were very positive. By using Apps and downloaded map tiles, the i devices matched almost all of the Garmin features. The geo-tagged photos and wireless data service elevated the i devices far above the Garmins for field activities like remote property inspections.

One final note on wireless data services. All regions of the country have varying levels of connectivity and service reliability. Currently, all telecommunications services in the United States have geographic gaps in coverage (this applies to wireline services as well as wireless services). As a result, anyone procuring mobile devices for a widely dispersed group of user should check the (1) service providers coverage maps, consult with target users in that area, and (3) focus on the use cases for remote field work. Metropolitan usage patterns and user expectations are significantly different than the geographically remote folks. Don’t let the metropolitan users obfuscate the benefits your organization might recieve from providing wireless data services to your field workers.

iPad for GIS usage – a first look at some capabilities

Have quite a few folks at work asking about using these in the field for GIS activities.  Here’s a summary of what I’ve been able to tell them so far.

•    ArcMap:  ESRI (the ArcMap vendor) has published an app for iOS (iPad and/or iPhone).  As soon as we get the test units, I plan to work with one of the GIS guys here at Beacon to work out how to make a connection to an ArcGIS server.   The iPad app is free, and I have it loaded on my personal iPad.  So far, I don’t believe the iPad ArcGIS client requires a license (but we’ll confirm that soon).
o    Note: the current ESRI ArcGIS doesn’t have map editing tools.  However, there is an alternate application (called GISRoam) which does enable capturing GPS Shape files and making edits on the iPad.  It’s likely that ESRI will have an update in the works which adds more features.
o    Note #2:  The GPS chip in the iPad is roughly equivalent to a Garmin Nuvi in terms of accuracy and signal tracking.  So it will be a long time before an iPad can really compete with a “Trimble ArcGIS Mobile” unit for data capture and on board GIS editing.  Personnally, I’m hoping for a Bluetooth accessory/interface that enables use of an external receiver so that the iPads would be more useful for data capture at Archeological sites and other in-the-field uses.
•    GoogleEarth:  There’s a native GoogleEarth app for the iPads.  At this time, I don’t see an option to load GoogleEarth within ESRI’s ArcGIS client.  However, the ESRI iPad app can load maps from any number of sources including Bing, OpenStreetMap, USGS, NOAA, etc.,  so it may be a matter of fiddling with options to get Google Earth maps into it ArcGIS (if that’s actually needed).
•    Take Picture:  The iPad can’t take pictures, but we’re ordering them with the camera connection kit so that they can load pictures from cards or usb camera connections.
•    Upload local tax information:  We need more details about this requirement.  What’s the source/format of the tax data.  Are we tagging tax information it to map records?  Uploading coordinate data into a tax application?  Or something else entirely?
•    Sketch or Draw and Measure GPS:  1st question here is, do we have a firm requirement for a single app that can do both (ie., Sketch and MeasureGPS)?  Or would a standalone sketch/draw application have some usefulness?
o    If these can be treated separately for now; there are several sketchpad type application available.  So far I do not know of any GPS enabled drawing apps which can capture/import a track-log file and then annotate/edit the resulting shapefiles.
o    The GISRoam application does have the ability to capture new shapefiles and edit some attributes, but my skills in this area are a bit limited.  So far, I’ve only done some preliminary tests where I’ve captured new shapefiles and made simple calculations like length and area.
o    The GISRoam application also has the ability to download a shapefile from a server, make edits, and then upload back to the server.  Currently the app uses a subscription server for this feature and I don’t know if/when it can make a direct connection to an ArcGIS server.  Will test/investigate this further when we get the initial testing units.

iOS and ArcGIS

ESRI has an ArcGIS app for the iOS platform, here are some sites with with additional GIS Maps, ArcGIS shapefiles, and geospatial data files:

http://gisroam.com/?page_id=83
http://gisroam.com/User.html

http://data.geocomm.com/
http://www.lib.ncsu.edu/gis/data.html
http://www.mapcruzin.com/download-free-arcgis-shapefiles.htm

http://mapserver.org/
http://www.gdal.org/index.html

http://www.giscafe.com/

Garmin GPS w/ weather overlay on the map display

Received a question from someone wondering if Garmin offered a product which could display a weather overlay atop the current map layer.  I knew this was available in the fixed mount marine products, but wasn’t sure about the portable products.

Well, they do build them… The cheapest one I could find… GPS Map 430SX GPS for $599 + GXM 31 antenna $264

The NUVI series is uing MSN Direct (over FM radio) which only works in some cities.  The full on satellite weather info is coming thru XM, and it appears Garmin is only putting that on the Marine and Aviation products.  Some of the discontinued StreetPilot models were XM Weather capable, but I can’t find anything to confirm if the StreetPilots would overlay weather radar on the map display.

Here’s the link to the XM Weather service:  http://www.xmwxweather.com/overview/     According to the XM website, the 2009 Accura includes XMNavWeather in it’s navigation package.  Might be interesting to go look at one just to see.  Doesn’t say if it’s full radar display; might be just a collection of dumbed down weather icons displaying on the map.

This is the first time I’ve thought to look for bluetooth XM recievers… they’re certainly not as abundant as bluetooth gps recievers.  Did find several avaition related vendors offering gadgets to connect a hardwired XM Weather reciever to a PDA via bluetooth.  These guys, anywheremap.com, offer an iPaq kit with maps, bluetooth gps reciever, and what appears to be a bluetooth XM reciever for $1,495. 

The biggest obstacle for now is probably the XM subscription costs.  The “Marine SkyWatch” subscription is $9.99/month, but it’s pretty limited.  The more comprehensive weather runs $30-$100 per month.  In comparison, the full featured XM Radio subscription tops out at $20/month.

If I understand what I’ve read so far, the XM Weather recievers and XM Radio recievers are both listening to the same spectrum.  It’s just which portion of the signal stream they’re configured to decode.  Some of the high end nav units offer both Weather and Radio.  So it’s probably just a software code.  Ug, yep, XM radio is all digital, only takes one antenna to recieve the entire signal range.  The radio signal wouldn’t tax a modern processor. Neither would the data stream for weather; so yeah, those cheap little portable xm radio recievers they sell at walmart should have more than enough processing power to decode both weather and audio in real-time.  (By the way, the hardware manufacturers can by those little XM receiver chip for about 10¢ apiece.)  

Would be interesting to find a bluetooth XM receiver supported by one of the currently available, GPS enabled, smartphone products.