Vagrant

Week of May 29th

Posted on by Mark Shay

“This is the return of the space cowboy…Interplanetary, good vibe zone”

Happy Friday!


A short time ago in a galaxy not that far, far away….” 

Actually it was only just a few weeks ago that we traveled 🚀 through time 🕰 and space💫 and into the DevOps Universe. For those who recall, we just merely scratched the surface as we dipped our toe🦶in the Lunar 🌚 mare 🌊 with HashiCorp’s Vagrant. At the time, we played around with combination of Vagrant and Oracle’s Virtualbox. This time we wanted to go full tilt into the Microservices galaxy 🌌 with the infamous “batteries🔋 included but removable approach” a.k.a. Docker.

We had explored the rings🪐 of Docker🐳 once before or I should formally say images and containers but it had been about 4 years since we last journeyed there. At time, we were lead by the charismatic and often enthusiastic Docker🐳 Captain 👨‍✈️ Nigel Poulton 

Since so much time has past we were overdue for a refresher and who better to lead us through this exploration than equally ebullient👨‍✈️ Captain Bret Fisher as he would take us through his brilliantly put together courseware Docker Mastery: with Kubernetes +Swarm from a Docker Captain

Bret’s courseware contains a chock full of DevOps awesomeness 😎 including full lessons on Docker🐳 , Docker🐳 Compose, Docker🐳 Swam, Docker🐳  Registries, and Kubernetes. There were even quizzes after most lessons to see how much you were able to absorb. Unfortunately, time would only allow for us time to explore just the Docker🐳  portion. Leaving us with Docker🐳 Compose, Docker🐳  Swam, Docker🐳 Registries, and Kubernetes for future missions. Our initial goal was to use to utilize Vagrant with Docker 🐳 (similarly to how we used Vagrant with VBox a few weeks back). However, to get into the spirt of how ubiquitous Docker🐳  is we decided we would try to install Docker🐳  on multiple platforms. All was well, until we decided to install Docker CE on Windows 10. 

Since we are primarily running on the Mac book💻 we would need to create a Windows 10 Pro Image on VirtualBox.. Sure no problem.. After several minutes we had our Windows Desktop up and running. For the finishing touches we just need to install Docker🐳 before we would head over to the next lesson. After Docker 🐳 completed it’s installation on Windows, we tried to fire🔥 up Docker 🐳…  Bam💥, Slam, Crash ☄️! Uh, oh what now?? Well, actually this was a familiar scenario we had seen a few weeks back when we tried to install Vagrant & VirtualBox under an Ubuntu VM that was already running on VirtualBox for the Mac book💻.

At the time we just chalked this up to an overly convoluted solution because it was virtualization inside of virtualization. But apparently, there is an actual technical term for this which is called “nested virtualization”. This configuration is not generally recommended but it is supported in certain scenarios. The way to make this solution fly Under VBox is to enable “Enable Nested VT -x/AMD-V” Settings -> Processor in Virtualbox.
Of course on the Mac book 💻 this is inconveniently greyed out. Not to fret, just like the hidden trick to increase the VDI disk size . Their is hidden trick to enable Nested Virtualization on VBox on the Mac book 💻

Ok, so now we enabled the Nested Virtualization setting and made some additional recommended local setting changes on the Windows VM. After rebooting the VM, we would try to install Docker 🐳 but again no luck. It’s turns out that Docker 🐳 will only work with nested virtualization with VMware Fusion and we weren’t going to go there. So after spending enough time on this we decided to cut or loses and just work with Vagrant with Docker 🐳 and Ubuntu which in reality is the best way to run docker (Under a native Linux OS)..

Now with our a Docker🐳 environment ready to go and fired 🔥 up it was time to have 👨‍✈️ Captain Fisher take us to warp drive 💨. As we charted our course, we hit a minor bug when building the popular Docker🐳 container nginx image on our environment.  We would execute the below syntax which worked fine without any issues:

docker container run –publish 8080:80 nginx:latest

However, after we killed the container with control-c and ran docker info we noticed the following error message

WARNING: No swap limit support

Although, the nginx container still ran fine, we are not big fans of those pesky little error messages.  So we resolved the issue by running the below syntax in the Docker 🐳 container:

vagrant@node-1:~$ sudo nano /etc/default/grub

GRUB_CMDLINE_LINUX=”cgroup_enable=memory swapaccount=1 

$ sudo update-grub

$ reboot -iThen From: Mac book 💻

vagrant halt

vagrant up

For good measure we filed a bug with the developer on Github. Fortunately we know the developer, so this minor bug was fixed almost immediately. So it can be used with future builds. Next, we moved on as our journey touched on all the essential topics for Docker 🐳:

  • Manage Multiple Containers
  • Management of Docker Networking
    • DNS and How Containers Find Each Other
    • Using Containers for CLI Testing
    • DNS Round Robin Test
  • Images and Their Layers
  • Image Tagging and Pushing to Docker Hub
  • Dockerfiles and Run Containers From it
  • Persistent Data: Data Volumes, Bind Mounting
  • Database Upgrades with Named Volumes
  • Running In Containers with Bind Mounts

For more details on syntax see log

“DevOps is the union of people, process, and products to enable continuous delivery of value to our end users.”

Below are some topics I am considering for my Journey next week:

  • More with Captain Brett on  Compose, Swam, Registries, and Kubernetes
  • More with with Data Pipelines
  • Google Cloud Data Fusion (ETL/ELT)
  • More on Machine Learning
  • ONTAP Cluster Fundamentals
  • Google Big Query
  • Data Visualization Tools (i.e. Looker)
  • ETL Solutions (Stitch, FiveTran) 
  • Process and Transforming data/Explore data through ML (i.e. Databricks) .
  • Getting Started with Kubernetes with an old buddy (Nigel)

Stay safe and Be well –

–MCS 

Week of May 8th

Posted on by Mark Shay

“Now it’s time to leave the capsule if you dare..”

Happy Friday!

Before we could end our voyage and return our first mate Slonik to the zookeeper, we would first need to put a bow on our Postgres journey (for now) by covering a few loose ends on advanced features. Saturday, we kicked it off with a little review on Isolation levels in Postgres (including a deep dive on Serializable Snapshot Isolation (SSI))  Then on to Third-parting monitoring for database health and streaming replication, and for the la cerise sur le gâteau… Declarative Partitioning and Sharding!

Third-Party Monitoring

We evaluated 2 solutions OpsDash and PgDash. Both were easy to set up and both gave valuable information in regards to Postgres. OpsDash provided more counters and is it can monitor system information as well as other services running on Linux where as PgDash is Postgres specific and will give you a deeper look into Postgres and Streaming Replications than just querying the native system views

Declarative Partitioning

It was fairly straight forward to implement Declarative partitioning. We reinforcement such concepts by turning to Creston’s plethora of videos on the topics as well as turning to several blog posts. See below for detailed log.

Sharding Your Data with PostgreSQL

There are third party solutions like Citus Data that seem to offer a more scalable solution but out of the box you can implement Sharding with using Declarative Partitioning set up on a Primary Server and using a Foreign Data Wrapper configured on a remote Server. Then you combine Partitioning and FDW to create Sharding. This was quite an interesting solution although I have strong doubts about how scalable this would be in production.

On Sunday, we took a much-needed respite as the weather was very agreeable in NYC to escape the quarantine… 

On Monday, with our rig now dry docked.  We would travel through different means to another dimension, a dimension not only of sight and sound but of mind. A journey into a wondrous land of imagination. Next stop, the DevOps Zone!

To begin our initiation into this realm we would start off with HashiCorp’s Vagrant.

For those who not familiar with Vagrant it is not a transient mendicant that the name would otherwise imply but a nifty open-source solution for building and maintaining lightweight and portable DEV environments.

It’s kind of similar to docker for those more familiar but it generally works with virtual machines (although can be used with containers).

At the most basic level, Vagrant uses a smaller version of VMs whereas Docker is kind of the “most minimalistic version for process and OS bifurcation by leveraging containers”.

The reason to go this route opposed to the more popular Docker was that it is generally easier to standup a DEV environment.

With that being said we wound up spending a considerable amount of time on Monday and Tuesday this week Working on this. As I ran into some issues with SSH and “Vagrant UP” process. The crux of issue was related using Vagrant/VirtualBox under an Ubuntu VM that was already running VirtualBox on a Mac. This convoluted solution didn’t seem to play nice. Go figure?

Once we decided to install Vagrant with VirtualBox natively on the Mac we were up and running were easily able to spin up and deploy VMs seamlessly.

Next, we played a little bit with Git. Getting some practicing with the work flow of editing configuration files and pushing the changes straight to the repo.

On Wednesday, we decided to begin our expropriation of a strange new worlds, to seek out new life and new civilizations and of course boldly go where maybe some have dared to go before?  That would be of course Machine Learning where the data is the oil and the algorithm is the engine. We would start off slow by just trying to grasp the jargon like training data, training a model, testing a model, Supervised learning, and Unsupervised Learning.

The best way for us to absorb this unfamiliar lingo would be to head over to Pluralsight where David Chappell offered a great introductory course on Understanding Machine Learning

“Now that she’s back in the atmosphere… With drops of Jupiter in her hair, hey, hey”

On Thursday we would go further down the rabbit hole of Machine Learning with Jerry Kurata’s Understanding Machine Learning with Python  

There we would be indoctrinated by the powerful tool of Jupyter Notebook. Now armed with this great “Bat gadet” we would reunite with some of our old heroes from the “Guardians of the Python” like “Projectile” Pandas, matplotlib “the masher” and of course numpy “ the redhead step child of Thanos”. In addition, we would also be introduced to a new and special super hero scikit-learn.

For those not familiar with this powerful library “scikit-lean” has unique and empathic powers to our friends Numpy, Pandas and SciPy. This handy py lib ultimately unlocks the key to the Machine Learning Universe through Python.

Despite all this roistering with our exemplars of Python, our voyage wasn’t  all rainbows and Unicorns.

We got introduced to all sorts of new space creatures like Bayesian and Gaussian Algos each conjuring up bêtes noires. The mere thought of Bayes theorem drudged up old memories buried deep in the bowls back in college when I was studying probability and just the mere mention of Gaussian functions jarred memories from the Black Swan (and not the ballet movie with fine actresses Natalie Portman and Mila Kunis) but the well-written and often irritating NYT Best seller by Nassim Nicholas Taleb.

Unfortunately, It didn’t get any cozier when we started our course for powerful and complex ensemble of the Random Forrest Algo. There we got bombarded by meteorites such as “Over Fitting”, “Regularization Hyper-parameters” , and “Cross Validation”, and not to mention the dreaded “Bias – variance tradeoff”. Ouch! My head hurts…

Here is the detailed log of my week’s journey

“With so many light years to go…. And things to be found (to be found)”

Below are some topics I am considering for next week’s odyssey :

  • Run Python Scripts in SQL Server Agent
  • More with Machine Learning
  • ONTAP Cluster Fundamentals
  • Google Big Query
  • Python -> Stream Data from IEX ->
  • MSSQLData Visualization Tools (i.e. Looker)
  • ETL Solutions (Stitch, FiveTran) 
  • Process and Transforming data/Explore data through ML (i.e. Databricks) 
  • Getting Started with Kubernetes with an old buddy (Nigel)

Stay safe and Be well

—MCS