Week of November 20th

“Climb🧗‍♀️ in the back with your head👤 in the Clouds☁️☁️… And you’re gone

Hi All –

Happy Name Your PC💻 Day!

Forward yesterday makes me wanna stay…”

“Welcome back, to that same old place that you laughed 😂 about”. So, after a short recess we made our splendiferous return this week. To where else? …But to no other than Google Cloud Platform a.k.a GCP☁️ , of course! 😊 So after completing our three-part Cloud Journey, we were feeling the need for a little refresher… Also, there were still had a few loose ends we needed to sew🧵 up. The wonderful folks at Google Cloud☁️ put together amazing compilation on GCP☁️ through their Google Cloud Certified Associate Cloud Engineer Path but we were feeling the need for a little more coverage on GCP CLI i.e. “gcloud”, “gsutil”, and “bq” . In addition, we had a great zest to learn a little more about some of the service offerings like GCP Development Services and APIs. Fortunately, we knew exactly who could deliver tremendous content on GCP☁️ as well as hit the sweet spot on some of the areas where we felt we were lacking a bit. That would be of course one of our favorite Canucks 🇨🇦 Mattias Andersson

For those who are not familiar with Mattias, he is one of the legendary instructors on A Cloud Guru. Mattias is especially well-known for his critically acclaimed Google Certified Associate Cloud Engineer 2020 course.

In this brilliantly produced course Mattias delivers the goods and then some! The goal of the course is to prepare those interested in preparing for Google’s Associate Cloud Engineer (ACE) Certification exam but it’s structured in a manner to efficiently to provide you with the skills to troubleshoot GCP through having a better understanding of “Data flows”. Throughout the course Mattias emphasizes the “see one, do one, teach one” technique in order to get the best ROI out of the tutorial.

So, after some warm salutations and a great overview of the ACE Exam, Mattias takes right to an introductions of all the Google Cloud product and Services. He accentuates the importance of Data Flow in fully understanding how all GCP solutions work. “Data Flow is taking data or information and it’s moving it around, processing it and remembering it.

Data flows – are the foundation of every system

  • Moving, Processing, Remembering
    • Not just Network, Compute, Storage
  • Build mental models
    • Helps you make predictions
  • Identify and think through data flows
    • Highlights potential issues
  • Requirement and options not always clear
    • Especially in the real world🌎
  • Critical skills for both real world🌎 and exam📝 questions

“Let’s get it started, in here…And the base keep runnin’ 🏃‍♂️ runnin’ 🏃‍♂️, and runnin’ 🏃‍♂️ runnin’ 🏃‍♂️, and runnin’ 🏃‍♂️ runnin’ 🏃‍♂️, and runnin’ 🏃‍♂️ runnin’ 🏃‍♂️, and runnin’ 🏃‍♂️ runnin’ 🏃‍♂️, and runnin’ 🏃‍♂️ runnin’ 🏃‍♂️, and runnin’ 🏃‍♂️ runnin’ 🏃‍♂️, and runnin’ 🏃‍♂️ runnin’ 🏃‍♂️, and…”

After walking🚶‍♀️ us through how to create a Free account it was time ⏰ to kick off 🦵 us with a little Billing and Billing Export.

“Share it fairly, but don’t take a slice of my pie 🥧”

Billing Export –to BigQuery enables you to export your daily usage and cost estimates automatically throughout the day to a BigQuery dataset.

  • Export must be set up per billing account
  • Resources should be placed into appropriate projects
  • Resources should be tagged with labels🏷
  • Billing export is not real-time
    • Delay is hours

Billing IAM – Role: Billing Account User

  • Link🔗 projects to billing accounts
  • Restrictive permissions
  • Along with the Project Creator allow a user to create new projects linked to billing

Budgets – Help with project planning and controlling costs

  • Setting a budget lets you track spend
  • Apply budget to billing account or a Project

Alerts 🔔 – notify billing administrators when spending exceeds a percentage of your budget

Google Cloud Shell 🐚 – provides with CLI access to Cloud☁️ Resources directly from your browser.

  • Command-line tool🔧 to interact GCP☁️
  • Basic Syntax
 gcloud–project=myprojid compute instances list
gcloud compute instances create myvm
gcloud services list --available
gsutil ls
gsutil mb -l northamerica-northeast1 gs://storage-lab-cli
gsutil label set bucketlables.json gs://storage-lab-cli

GCS via gsutil in Command Line

 gcloud config list
 gcloud config set project igneous-visitor-293922
 gsutil ls
 gsutil ls gs://storage-lab-console-088/
 gsutil ls gs://storage-lab-console-088/**
 gsutil mb --help
 gsutil mb -l northamerica-northeast1 gs://storage-lab-cli-088
 gsutil label get gs://storage-lab-console-088/
 gsutil label get gs://storage-lab-console-088/ > bucketlabels.json
 cat bucketlabels.json
 gsutil label get gs://storage-lab-cli-088
 gsutil label set bucketlabels.json gs://storage-lab-cli-088
 gsutil label ch -l "extralable:etravalue" gs://storage-lab-cli-088
 gsutil versioning get gs://storage-lab-cli-088
 gsutil versioning set on gs://storage-lab-cli-088
 gsutil versioning get gs://storage-lab-cli-088
 gsutil cp README-Cloudshell.txt gs://storage-lab-cli-088
 gsutil ls -a gs://storage-lab-cli-088
 gsutil rm gs://storage-lab-cli-088/README-Cloudshell.txt
 gsutil cp gs://storage-lab-console-088/** gs://storage-lab-cli-088/
 gsutil acl ch -u AllUsers:R gs://storage-lab-cli-088/shutterstock.jpg 

Create VM via gsutil in Command Line

 gcloud config get-value project
 gcloud compute instances list
 gcloud services list
 gcloud services list --enabled
 gcloud services list --help
 gcloud services list –available
 gcloud services list --available |grep compute
 gcloud services -h
 gcloud compute instances create myvm
 gcloud compute instances delete myvm 

Security🔒 Concepts

Confidentiality, Integrity, and Availability (CIA)

  • You cannot view data you shouldn’t
  • You cannot change data you shouldn’t
  • You can access data you should

Authentication, Authorization, Accounting (AIA)

  • Authentication – Who are you?
  • Authorization – What are you allowed to do?
  • Accounting – What did you do?
  • Resiliency – Keep it running 🏃‍♂️
  • Security🔒 Products
  • Security🔒 Features
  • Security🔒 Mindset
    • Includes Availability Mindset

Key🔑 Security🔒 Mindset (Principles)

  • Least privilege
  • Defense in depth
  • Fail Securely

Key🔑 Security🔒 Products/Features

  • Identity hierarchy👑 (Google Groups)
  • Resource⚙️ hierarchy👑 (Organization, Folders📂, Projects)
  • Identity and Access Management (IAM)
    • Permissions
    • Roles
    • Bindings
  • GCS ACLs
  • Billing management
  • Networking structure & restrictions
  • Audit / Activity Logs (provided by Stackdriver)
  • Billing export
    • To BigQuery
    • To file (in GCS bucket🗑)
      • Can be JSON or CSV format
  • GCS object Lifecycle Management

IAM – Resource Hierarchy👑

  • Resource⚙️
    • Something you create in GCP☁️
  • Project
    • Container for a set of related resources
  • Folder📂
    • Contains any number of Projects and Subfolders📂
  • Organization
    • Tied to G Suite or Cloud☁️ Identity domain

IAM – Permissions & Roles

Permissions – allows you to a perform a certain action

  • Each one follows the form Service.Resource.Verb
  • Usually correspond to REST API methods
    • pubsub.subcription.consume
    • pubsub.topics.publish

Roles – is a collection of permissions to use or manage GCP☁️ resources

  • Primitive Roles – Project-level and often too broad
    • Viewer is read-only
    • Editor can view and change things
    • Owner can also control access & billing
  • Predefined Roles
    • roles/bigquery.dataEditor, roles/pub.subscriber
    • Read through the list of roles for each product! Think about why each exists
  • Custom Role – Project or Org-Level collection you define of granular permissions

IAM – Members & Groups

Members – some Google-known identity

  • Each member is identifying by unique  email📧 address
  • Can be:
    • user: Specific Google account
      • G Suite, Cloud☁️ Identity, gmail, or validated email
    • serviceAccount: Service account for apps/services
    • group: Google group of users and services accounts
    • domain: whole domain managed by G Suite or Cloud☁️ Identity
    • allAuthenticatedUsers: Any Google account or service account
    • allUsers: Anyone on the internet (Public)

Groups – a collection of Google accounts and service accounts

  • Every group has a unique  email📧 address that is associated with the group
  • You never act as the group
    • But membership in a group can grant capabilities to individuals
  • Use them for everything
  • Can be used for owner when within an organization
  • Can nest groups in an organization
    • One group for each department, all those in group for all staff

IAM – Policies

Policies – binds members to roles for some scope of resources

  • Enforce who can do what to which thing(s)
  • Attached to some level in the Resource⚙️ history
    • Organization
    • Folder📂
    • Project Resource⚙️
  • Roles and Members listed in policy, but Resources identified by attachment
  • Always additive (Allow) and never subtractive (no Deny)
  • One policy per Resource⚙️
  • Max 1500-member binding per policy
 gCloud[GROUP] add-iam-policy-binding [Resource-NAME]
 --role [ROLE-ID-TO-GRANT] –member user: [USER-EMAIL]
 gCloud[GROUP] remove-iam-policy-binding [Resource-NAME]
 --role [ROLE-ID-TO-REVOKE] –member user: [USER-EMAIL] 

Billing Accounts – represents some way to pay for GCP☁️ service usuage

  • Type of Resource⚙️ that lives outside of Projects
  • Can belong to an Organization
    • Inherits Org-level IAM policies
  • Can be linked to projects
    • Not the Owner
      • No impact on project IAM
Billing Account CreatorCreate new self-service billing accountsOrg
Billing Account AdministratorManage billing accountsBilling Account
Billing Account UserLink Projects to billing accountsBilling Account
Billing Account ViewerView billing account cost information and transactionsBilling Account
Project Billing ManagerLink/unlink the project to/from a billing accountProject

Monthly Invoiced Billing – Billed monthly and pay by invoice due date

  • Pay via check or wire transfer
  • Increase project and quota limits
  • Billing administrator of org’s current billing account contacts Cloud☁️ Billing Support
    • To Determine eligibility
    • To apply to switch to monthly invoicing
  • Eligibility depends on
    • Account age
    • Typical monthly spend
    • Country


            Choose the right solution to get data to the right Resource⚙️

  • Latency reduction – Use Servers physically close to clients
  • Load Balancing – Separate from auto-scaling
  • System design – Different servers may handle different parts of the system
  • Cross-Region Load Balancing – with Global🌎 Anycast IPs
  • Cloud☁️ Load Balancer 🏋️‍♀️ – all types; internal and external
  • HTTP(S) Load Balancer 🏋️‍♀️ (With URL Map)

Unicast vs Anycast

Unicast – There is only one unique device in the world that can handle this; send it there.

Anycast – There are multiple devices that could handle this; send it to anyone – but ideally the closest.

            Load Balancing – Layer 4 vs Layer 7

  • TCP is usually called Layer 4 (L4)
  • HTTP and HTTPS work at Layer (L7)
  • Each layer is built on the one below it
    • To route based on URL paths, routing needs to understand L7
    • L4 cannot route based on the URL paths defined in L7

DNS – Name resolution (via the Domain Name System) can be the first step in routing

  • Some known issues with DNS
    • Layer 4 – Cannot route L4 based on L7s URL paths
    • Chunky – DNS queries often cached and reused for huge client sets
    • Sticky – DNS lookup “locks on” and refreshing per request has high cost
      • Extra latency because each request includes another round-trip
      • More money for additional DNS request processing
    • Not Robust – Relies on the client always doing the right thing
  • Premium tier routing with Global🌎 anycast Ips avoids these problems

Options for Data from one Resource to another

  • VPC (Global🌎) Virtual Private Cloud☁️ – Private SDN space in GCP☁️
    • Not just Resource-to-Resource – also manages the doors to outside & peers
  • Subnets (regional) – create logical spaces to contain resources
    • All Subnets can reach all others – Globally without any need for VPNs
  • Routes (Global🌎) define “next hop” for traffic🚦 based on destination IP
    • Routes are Global🌎 and apply by Instance-level Tags, not by Subnet
    • No route to the Internet gateway means no such data can flow
  • Firewall🔥 Rules (Global🌎) further filter data flow that would otherwise route
    • All FW Rules are Global🌎 and apply by Instance-level Tags or Service Acct.
    • Default Firewall🔥 Rules are restrictive inbound and permissive outbound


  • IP Address is (dotted quad) where each piece is 0-255
  • CIDR block is group of IP addresses specified in <IP>/xy notation
    • Turn IP address into 32-bit binary number
    • -> 00001010 00001010 00000000 11111110
    • /xy in CIDR notation locks highest (leftmost) bits in IP address (0-32)
    • abc.efg.hij.klm/32 is single IP address ( because all 32 bits are looked
    • abc.efg.hij.klm /24 is 24 is 256 (255.555.255.0) IP addresses because last 8 bits can vary
    • means “any IP address” because no bits are locked
  • RFC1918 defines private (i.e non-internet) address ranges you can use:
    •, and

Subnet CIDR Ranges

  • You can edit a subnet to increase its CIDR range
  • No need to recreate subnet or instances
  • New range must contain old range (i.e. old range must be subnet)

Shared VPC

  • In an Organization, you can share VPCs among multiple projects
    • Host Project: One project owns the Shared VPC
    • Service Projects: Other projects granted access to use all/part of Shared VPC
  • Lets multiple projects coexist on same local network (private IP space)
  • Let’s a centralized team manage network security🔒

“Ride, captain👨🏿‍✈️ ride upon your mystery ship⛵️


A Kubernetes ☸️ cluster is a set of nodes that run containerized applications. Containerizing applications packages an app with its dependences and some necessary services.

K8s ☸️ you know that the control plane consists of the kube-apiserver, kube-scheduler, kube-controller-manager and an etcd datastore. 

Deploy and manage clusters on-prem

Step 1: The container runtime

Step 2: Installing kubeadm

Step 3: Starting the Kubernetes cluster ☸️

Step 4: Joining a node to the Kubernetes cluster ☸️

Deploy and manage clusters on-prem in the Cloud☁️

To deploy and manage your containerized applications and other workloads on your Google Kubernetes Engine (GKE) cluster, you use the K8s ☸️ system to create K8s ☸️  controller objects. These controller objects represent the applications, daemons, and batch jobs running 🏃‍♂️ on your clusters.

            Cloud Native Application Properties

  • Use Cloud☁️ platform services.
  • Scale horizontally.
  • Scale automatically, using proactive and reactive actions.
  • Handle node and transient failures without degrading.
  • Feature non-blocking asynchronous communication in a loosely coupled architecture.

Kubernetes fits into the Cloud-native ecosystem

K8s ☸️ native technologies (tools/systems/interfaces) are those that are primarily designed and built for Kubernetes ☸️.

  • They don’t support any other container or infrastructure orchestration systems
  • K8s ☸️ accommodative technologies are those that embrace multiple orchestration mechanisms, K8s ☸️ being one of them.
  • They generally existed in pre-Kubernetes☸️ era and then added support for K8s ☸️ in their design.
  • Non-Kubernetes ☸️ technologies are Cloud☁️ native but don’t support K8s ☸️.

Deploy and manage applications on Kubernetes ☸️

K8s ☸️ deployments can be managed via Kubernetes ☸️ command line interface kubectl. Kubectl uses the Kubernetes ☸️ API to interact with the cluster. 

When creating a deployment, you will need to specify the container image for your application and the number of replicas that you need in your cluster.

  • Create Application
    • create the application we will be deploying to our cluster
  • Create a Docker🐳 container image
    • create an image that will contain the app built.
  • Create a K8s ☸️ Deployment
    • K8s ☸️ deployments are responsible for creating and managing pods
    • K8s ☸️ pod is a group of one or more containers, tied together for the purpose of administration and networking. 
    • K8s ☸️ Deployments can be created in two ways
      •  kubectl run command
      •  YAML configuration

Declarative Management of Kubernetes☸️ Objects Using Configuration Files

K8s ☸️ objects can be created, updated, and deleted by storing multiple object configuration files in a directory and using kubectl apply to recursively create and update those objects as needed.

This method retains writes made to live objects without merging the changes back into the object configuration files. kubectl diff also gives you a preview of what changes apply will make.


A DaemonSet ensures that all (or some) Nodes run a copy of a Pod. As nodes are added to the cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are garbage collected. Deleting a DaemonSet will clean up the Pods it created.

Some typical uses of a DaemonSet are:

  • Running 🏃‍♂️ a cluster storage🗄 daemon on every node
  • Running 🏃‍♂️ a logs collection daemon on every node
  • Running 🏃‍♂️ a node monitoring🎛 daemon on every node

Cloud Load Balancer 🏋️‍♀️ that GKE created

Google Kubernetes☸️ Engine (GKE) offers integrated support for two types of Cloud☁️ Load Balancing for a publicly accessible application:

When you specify type:LoadBalancer 🏋️‍♀️ in the Resource⚙️ manifest:

  1. GKE creates a Service of type LoadBalancer 🏋️‍♀️. GKE makes appropriate Google Cloud API calls to create either an external network load balancer 🏋️‍♀️ or an internal TCP/UDP load balancer 🏋️‍♀️.
  • GKE creates an internal TCP/UDP load balancer 🏋️‍♀️ when you add the Cloud.google.com/load-balancer 🏋️‍♀️-type: “Internal” annotation; otherwise, GKE creates an external network load balancer 🏋️‍♀️.

Although you can use either of these types of load balancers 🏋️‍♀️ for HTTP(S) traffic🚦, they operate in OSI layers 3/4 and are not aware of HTTP connections or individual HTTP requests and responses.

Imagine all the people👥 sharing all the world🌎

GCP Services


Compute Engine (GCE) – (Zonal) (IaaS) – Fast-booting Virtual Machines (VMs) for rent/demand

  • Pick set machine type – standard, high memory, high CPU-or custom CPU/RAM
  • Pay by the second (60 second min.) for CPUs, RAM
  • Automatically cheaper if you keep running 🏃‍♂️ it (“sustained use discount”)
  • Even cheaper for “preemptible” or long-term use commitment in a region
  • Can add GPUs and paid OSes for extra cost*
  • Live Migration: Google seamlessly moves instance across hosts, as needed

Kubernetes Engine (GKE) – (Regional (IaaS/Paas) -Managed Kubernetes ☸️ cluster for running 🏃‍♂️ Docker🐳 containers (with autoscaling)

  • Kubernetes☸️ DNS on by default for service discovery
  • NO IAM integration (unlike AWS ECS)
  • Integrates with Persistent Disk for storage
  • Pay for underlying GCE instances
    • Production cluster should have 3+ nodes*
  • No GKE management fee, no matter how many nodes in cluster

  App Engine (GAE) – (Regional (PaaS) that takes your code and runs it

  • Much more than just compute – Integrates storage, queues, NoSQL
  • Flex mode (“App Engine Flex”) can run any container & access VPC
  • Auto-Scales⚖️ based on load
    • Standard (non-Flex) mode can turn off las instance when no traffic🚦
  • Effectively pay for underlying GCE instances and other services

Cloud Functions – (Regional (FaaS), “Serverless” -Managed K8s☸️ cluster for running 🏃‍♂️ Docker🐳 containers (with autoscaling)

  • Runs code in response to an event – Node.js Python🐍, Java☕️, Go🟢
  • Pay for CPU and RAM assigned to function, per 100ms (min. 100ms)
  • Each function automatically gets an HTTP endpoint
  • Can be triggered by GCS objects, Pub/Sub messages, etc.
  • Massively Scalable⚖️ (horizontally) – Runs🏃‍♂️ many copies when needed


Persistent Disk (PD) – (Zonal) Flexible🧘‍♀️, block-based🧱 network-attached storage; boot disk for every GCE instance

  • Perf Scales⚖️ with volume size; max way below Local SSD, but still plenty fast🏃‍♂️
  • Persistent disks persist, and are replicated (zone or region) for durability
  • Can resize while in use (up to 64TB), but will need file system update with VM
  • Snapshots (and machine images🖼 ) add even more capability and flexibility
  • Not file based NAS, but can mount to multiple instances if all are read-only
  • Pay for GB/mo provisioned depending on perf.class;plus snapshot GB/mo used

  Cloud Filestore – (Zonal) Fully managed file-based storage

  • “Predictably fast🏃‍♂️ performance for your file-based workloads”
  • Accessible to GCE and GKE through your VPC, via NFSv3 protocol
  • Primary use case is application migration to Cloud☁️ (“lift and shift”)🚜
  • Fully manages file serving, but not backups
  • Pay for provisioned TBs in “Standard” (slow) or “Premium” (fast🏃‍♂️) mode
  • Minimum provisioned capacity of 1TB (Standard) or 2.5TB (Premium)

Cloud Storage (GCS) – (Regional, Multi-Regional) Infinitely Scalable⚖️, fully managed, versioned, and highly durable object storage

  • Designed for 99.999999999% (11 9’s) durability
  • Strong consistent💪 (even for overwrite PUTs and DELETEs)
  • Integrated site hosting and CDN functionality
  • Lifecycle♻️ transitions across classes: Multi-Regional, Regional, Nearline, Coldline🥶
    • Diffs in cost & availability (99.5%, 99.9%, 99%, 99%), not latency (no thaw delay)
  • All Classes have same API, so can use gsutil and gcsfuse


            Cloud SQL – (Regional, Fully managed and reliable MySQL and PostgreSQL databases

  • Supports automatic replication, backup, failover, etc.
  • Scaling is manual (both vertically and horizontally)
  • Effectively pay for underlying GCE instances and PDs
    • Plus, some baked-in service fees

Cloud Spanner – (Regional, Multi-Regional), Global🌎 horizontally Scalable⚖️, strongly consistent 💪, relational database service”

  • “From 1 to 100s or 1000s of nodes”
    • “A minimum of 3 nodes is recommended for production environments.”
  • Chooses Consistency and Partition – Tolerance (CP and CAP theorem)
  • But still high Availability: SLA has 99.999% SLO (five nines) for multi-region
    • Nothing is actually 100%, really
    • Not based on fail-over
  • Pay for provisioned node time (by region/multi-region) plus used storage-time

BigQuery (BQ)– Multi-Regional Serverless column-store data warehouse for analytics using SQL

  • Scales⚖️ internally (TB in seconds and PB in minutes)
  • Pay for GBs actually considered (scanned) during queries
    • Attempts to reuse cached results, which are free
  • Pay for data stored (GB-months)
    • Relatively inexpensive
    • Even cheaper when table not modified for 90 days (reading still fine)
  • Pay for GBs added via streaming inserts

Cloud Datastore – (Regional, Multi-Regional) Managed & autoscale⚖️ NoSQL DB with indexes, queries, and ACID trans, support

  • No joins or aggregates and must line up with indexes
  • NOT, OR, and NOT EQUALS (<>,!=) operations not natively supported
  • Automatic “built-in” indexes for simple filtering and sorting (ASC, DESC)
  • Manual “composite” indexes for more complicated, but beware them “exploding”
  • Pay for GB-months of storage🗄 used (including indexes)
  • Pay for IO operations (deletes, reads, writes) performed (i.e. no pre-provisioning)

Cloud Bigtable – (Zonal) Low latency & high throughput NoSQL DB for large operational & analytical apps

  • Supports open-source HBase API
  • Integrates with Hadoop, Dataflow, Dataproc
  • Scales⚖️ seamlessly and unlimitedly
    • Storage🗄 autoscales⚖️
    • Processing nodes must be scaled manually
  • Pay for processing node hours
  • GB-hours used for storage 🗄 (cheap HDD or fast🏃‍♂️ SSD)

Firebase Realtime DB & Cloud Firestore 🔥 – (Regional, Multi-Regional) NoSQL document📃 stores with ~real-time client updates via managed WebSockets

  • Firebase DB is single (potentially huge) JSON doc, located only in central US
  • Cloud☁️ Firestore has collection, documents📃, and contained data
  • Free tier (Spark⚡️), flat tier (Flame🔥), or usage-based pricing (Blaze)
    • Realtime Db: Pay more GB/month stored and GB downloaded
    • Firestore: Pay for operations and much less for storage🗄 and transfer

Data Transfer ↔️

Data Transfer Appliance – Rackable, high-capacity storage 🗄 server to physically ship data to GCS

  • Ingest only; not a way to avoid egress charges
  • 100 TB or 480 TB/week is faster than a saturated 6 Gbps link🔗

Storage Transfer Service – (Global) Copies objects for you, so you don’t need to set up a machine to do it

  • Destination is always GCS bucket 🗑
  • Source can S3, HTTP/HTTPS endpoint, or another GCS bucket 🗑
  • One-time or scheduled recurring transfers
  • Free to use, but you pay for its actions

External Networking

Google Domains – (Global) Google’s registrar for domain names

  • Private Whois records
  • Built-in DNS or custom nameservers
  • Support DNSSEC
  •  email📧 forwarding with automatic setup of SPF and DKIM (for built-in DNS)

Cloud DNS– (Global) Scalable⚖️, reliable, & managed authoritative Domain (DNS) service

  • 100 % uptime guarantee
  • Public and private managed zones
  • Low latency Globally
  • Supports DNSSEC
  • Manage via UI, CLI, or API
  • Pay fixed fee per managed zone to store and distribute DNS records
  • Pay for DNS lookups (i.e. usage)

Static IP Addresses – (Regional, Global🌎 Reserve static IP addresses in projects and assign them to resources

  • Regional IPs used for GCE instances & Network Load Balancers🏋️‍♀️
  • Global IPs used for Global load balancers🏋️‍♀️:
  • HTTP(S) SSL proxy, and TCP proxy
  • Pay for reserved IPs that are not in use, to discourage wasting them

Cloud Load Balancing (CLB) – (Regional, Global🌎 High-perf, Scalable ⚖️ traffic🚦 distribution integrated with autoscaling & Cloud☁️ CDN

  • SDN naturally handles spikes without any prewarming, no instances or devices
  • Regional Network Load Balancer 🏋️‍♀️: health checks, round robin, session affinity
    • Forwarding rules based on IP, protocol (e.g. TCP, UDP), and (optionally) port
  • Global load balancers 🏋️‍♀️ w/ multi-region failover for HTTP(S), SSL proxy, & TCP proxy
    • Prioritize low-latency connection to region near user, then gently fail over in bits
    • Reacts quickly (unlike DNS) to changes in users, traffic🚦, network, health, etc.
  • Pay by making ingress traffic🚦 billable (Cheaper than egress) plus hourly per rule

Cloud CDN – (Global) Low-latency content delivery based on HTTP(S) CLB integrated w/ GCE & GCS

  • Supports HTTP/2 and HTTPS, but no custom origins (GCP☁️ only)
  • Simple checkbox ✅ on HTTP(S) Load Balancer 🏋️‍♀️ config turns this on
  • On cache miss, pay origin-> POP “cache fill” egress charges (cheaper for in-region)
  • Always pay POP->clint egress charges, depending on location
  • Pay for HTTP(S) request volume
  • Pay per cache invalidation request (not per Resource⚙️ invalidated)
  • Origin costs (e.g. CLB, GCS) can be much lower because cache hits reduce load

Virtual Private Cloud (VPC) – (Regional, Global), Global IP v4 unicast Software-Defined Network (SDN) for GCP☁️ resources

  • Automatic mode is easy; custom mode gives control
  • Configure subnets (each with a private IP range), routes, firewalls🔥, VPNs, BGP, etc.
  • VPC is Global🌎 and subnets are regional (not zonal)
  • Can be shared across multiple projects in same org and peered with other VPCs
  • Can enable private (internal IP) access to some GCP☁️ services (e.g. BQ, GCS)
  • Free to configure VPC (container)
  • Pay to use certain services (e.g. VPN) and for network egress

Cloud Interconnect – (Regional, Multi-Regional) Options for connecting external networks to Google’s network

  • Private connections to VPC via Cloud VPN or Dedicated/Partner Interconnect
  • Public Google services (incl. GCP) accessible via External Peering (no SLAs)
    • Direct Peering for high volume
    • Carrier Peering via a partner for lower volume
  • Significantly lower egress fees
    • Except Cloud VPN, which remains unchanged

Internal Networking

Cloud Virtual Private Network (VPN)– (Regional) IPSEC VPN to connect to VPC via public internet for low-volume data connections

  • For persistent, static connections between gateways (i.e. not for a dynamic client)
    • Peer VPN gateway must have static (unchanging) IP
  • Encrypted 🔐 link🔗 to VPC (as opposed to Dedicated interconnect), into one subnet
  • Supports both static and dynamic routing
  • 99.9% availability SLA
  • Pay per tunnel-hour
  • Normal traffic🚦 charges apply

Dedicated Interconnect – (Regional, Multi-Regional) Direct physical link 🔗 between VPC and on-prem for high-volume data connections

  • VLAN attachment is private connection to VPC in one region: no public GCP☁️ APIs
    • Region chosen from those supported by particular Interconnect Location
  • Links are private but not Encrypted 🔐; can layer your own encryption 🔐
    • Redundancy achieves 99.99% availability: otherwise, 99.9% SLA
  • Pay fee 10 Gbps link, plus (relatively small) fee per VLAN attachment
  • Pay reduced egress rates from VPC through Dedicated Interconnect

Cloud Router 👮‍♀️ – (Regional) Dynamic routing (BGP) for hybrid networks linking GCP VPCs to external networks

  • Works with Cloud VPN and Dedicated Interconnect
  • Automatically learns subnets in VPC and announces them to on-prem network
  • Without Cloud Router👮‍♀️ you must manage static routes for VPN
    • Changing the IP addresses on either side of VPN requires recreating it
  • Free to set up
  • Pay for usual VPC egress

CDN Interconnect – (Regional, Multi-Regional) Direct, low-latency connectivity to certain CDN providers, with cheaper egress

  • For external CDNs, not Google’s Cloud CDN service
    • Supports Akamai, Cloudflare, Fastly, and more
  • Works for both pull and push cache fills
    • Because it’s for all traffic🚦 with that CDN
  • Contact CDN provider to set up for GCP☁️ project and which regions
  • Free to enable, then pay less for the egress you configured

Machine Learning/AI 🧠

Cloud Machine Learning (ML) Engine – (Regional) Massively Scalable ⚖️ managed service for training ML models & making predictions

  • Enables apps/devs to use TensorFlow on datasets of any size, endless use cases
  • Integrates: GCS/BQ (storage), Cloud Datalab (dev), Cloud Dataflow (preprocessing)
  • Supports online & batch predictions, anywhere: desktop, mobile, own servers
  • HyperTune🎶 automatically tunes 🎶model hyperparameters to avoid manual tweaking
  • Training: Pay per hour depending on chosen cluster capabilities (ML training units)
  • Prediction: Pay per provisioned node-hour plus by prediction request volume made

Cloud Vison API👓 – (Global) Classifies images🖼 into categories, detects objects/faces, & finds/reads printed text

  • Pre-trained ML model to analyze images🖼 and discover their contents
  • Classifies into thousands of categories (e.g., “sailboat”, “lion”, “Eiffel Tower”)
  • Upload images🖼 or point to ones stored in GCS
  • Pay per image, based on detection features requested
    • Higher price for OCR of Full documents📃 and finding similar images🖼 on the web🕸
    • Some features are prices together: Labels🏷 + SafeSearch, ImgProps + Cropping
    • Other features priced individually: Text, Faces, Landmarks, Logos

Cloud Speech API🗣 – (Global) Automatic Speech Recognition (ASR) to turn spoken word audio files into text

  • Pre-trained ML model for recognizing speech in 110+ languages/variants
  • Accepts pre-recorded or real-time audio, & can stream results back in real-time
  • Enables voice command-and-control and transcribing user microphone dictations
  • Handles noisy source audio
  • Optionally filters inappropriate content in some languages
  • Accepts contextual hints: words and names that will likely be spoken
  • Pay per 15 seconds of audio processed

Cloud Natural Language API 💬 – (Global) Analyzes text for sentiment, intent, & content classification, and extracts info

  • Pre-trained ML model for understanding what text means, so you can act on it
  • Excellent with Speech API (audio), Vision API (OCR), & Translation API (or built-ins)
  • Syntax analysis extracts tokens/sentences, parts of speech & dependency trees
  • Entity analysis finds people, places, things, etc., labels🏷 them & links🔗 to Wikipedia
  • Analysis for sentiment (overall) and entity sentiment detect +/- feelings & strength
  • Content classification puts each document📃 into one of 700+ predefined categories
  • Charged per request of 1000 characters, depending on analysis types requested

Cloud Translation API –(Global) Translate text among 100+ languages; optionally auto-detects source language

  • Pre-trained ML model for recognizing and translating semantics, not just syntax
  • Can let people support multi-regional clients in non-native languages,2-way
  • Combine with Speech, Vision, & Natural Language APIs for powerful workflows
  • Send plain text or HTML and receive translation in kind
  • Pay per character processed for translation
  • Also pay per character for language auto-detection

Dialogflow – (Global) Build conversational interfaces for websites, mobile apps, messaging, IoT devices

  • Pre-trained ML model and service for accepting, parsing, lexing input & responding
  • Enables useful chatbot and other natural user interactions with your custom code
  • Train it to identify custom entity types by providing a small dataset of examples
  • Or choose from 30+ pre-built agents (e.g. car🚙, currency฿, dates as starting template
  • Supports many different languages and platforms/devices
  • Free plan had unlimited text interactions and capped voice interactions
  • Paid plan is unlimited but charges per request: more for voice, less for text

Cloud Video Intelligence API 📹 – (Regional, Global) Annotates videos in GCS (or directly uploaded) with info about what they contain

  • Pre-trained ML model for video scene analysis and subject identification
  • Enables you to search a video catalog the same way you search text documents📃
  • “Specify a region where processing will take place (for regulatory compliance)”
  • Label Detection: Detect entities within the video, such as “dog” 🐕, “flower” 🌷 or “car”🚙
  • Shot Change Detection: Detect scene changes within the video🎞
  • SafeSearch Detection: Detect adult content within the video🎞
  • Pay per minute of video🎞 processed, depending on requested detection modes

Cloud Job Discovery– (Global) Helps career sites, company job boards, etc. to improve engagement & conversion

  • Pre-trained ML model to help job seekers search job posting databases
  • Most job sites rely on keyword search to retrieve content which often omits relevant jobs and overwhelms the job seeker with irrelevant jobs. For example, a keyword search with any spelling error returns) results, and a keyword search for “dental assistant” returns any “assistant” role that offers dental benefits.’
  • Integrates with many job/hiring systems
  • Lots of features, such as commute distance and recognizing abbreviations/jargon
  • “Show me jobs with a 30-minute commute on public transportation from my home”

Big Data and IoT

            Four Different Stages:

  1. Ingest – Pull in all the raw data in
  2. Store – Store data without data loss and easy retrieval
  3. Process – transform that raw data into some actionable information
  4. Explore & Visualize – turn the results of that analysis into something that’s valuable for your business

Cloud Internet of Things (IoT) Core– (Global) Fully managed service to connect, manage, and ingest data from device Globally

  • Device Manager handles device identity, authentication, config & control
  • Protocol Bridge publishes incoming telemetry to Cloud☁️ Pub/Sub for processing
  • Connect securely using IoT industry standard MQTT or HTTPS protocols
  • CA signed certificates can be used to verify device ownership on first connect
  • Two-way device communication enables configuration & firmware updates
  • Device shadows enable querying & making control changes while devices offline
  • Pay per MB of data exchanged with devices; no per-device charge

Cloud Pub/Sub– (Global) Infinitely Scalable⚖️ at-least-once messaging for ingestion, decoupling, etc.

  • “Global🌎 by default: Publish… and consume from anywhere, with consistent latency”.
  • Messages can be up to 10 MB and undelivered ones stored for 7 days-but no DLQ
  • Push mode delivers to HTTPS endpoints & succeeds on HTTP success status code
    • “Slow-start” algorithm ramps up on success and backs off & retries, on failures
  • Pull mode delivers messages to requestion clients and waits for ACK to delete
    • Let’s clients set rate of consumption, and supports batching and long-polling
  • Pay for data volume
    • Min 1KB per publish/Push/Pull request (not by message)

Cloud Dataprep– (Global) Visually explore, clean, and prepare data for analysis without running 🏃‍♂️ servers

  • “Data Wrangling” (i.e. “ad-hoc ETL”) for business analysts, not IT pros
    • Who might otherwise spend 80% of their time cleaning data?
  • Managed version of Trifacta Wrangler – and managed by Trifacta, not Google
  • Source data from GCS, BQ, or file upload – formatted in CSV, JSON, or relational
  • Automatically detects schemas, datatypes, possible joins, and various anomalies
  • Pay for underlying Daaflow job, plus management overhead charge
  • Pay for other accessed services (e.g. GCS, BQ)

Cloud Dataproc– (Zonal) Batch MapReduce processing via configurable, managed Spark & Hadoop clusters

  • Handles being told to scale (adding or removing nodes) even while running 🏃‍♂️ jobs
  • Integrated with Cloud☁️ Storage, BigQuery, Bigtable, and some Stackdriver services
  • “Image versioning” switches between versions of Spark, Hadoop, & other tools
  • Pay directly for underlying GCE servers used in the cluster – optionally preemptible
  • Pay a Cloud Dataproc management fee per vCPU-hour in the cluster
  • Best for moving existing Spark/Hadoop setups to GCP☁️
    • Prefer Cloud Dataflow for new data processing pipelines – “Go with the flow”

Cloud Datalab 🧪– (Regional) Interactive tool 🔧 for data exploration🔎, analysis, visualization📊 and machine learning

  • Uses Jupyter Notebook📒
    • “[A]n open-source web🕸 application that allows you to create and share documents📃 that contain live code, equations, visualizations and narrative text. Use include data cleaning and transformation, numerical simulation, statistical modeling, data visualization, machine learning, and much more.”
  • Supports iterative development of data analysis algorithms in Python🐍/ SQL/~JS
  • Pay for GCE/GAE instance hosting and storing (on PD) your notebook📒
  • Pay for any other resources accessed (e.g. BigQuery)

Cloud Data Studio– (Global) Big Data Visualization📊 tool 🔧 for dashboards and reporting

  • Meaningful data stories/presentations enable better business decision making
  • Data sources include BigQuery, Cloud SQL, other MySQL, Google Sheets, Google Analytics, Analytics 360, AdWords, DoubleClick, & YouTube channels
  • Visualizations include time series, bar charts, pie charts, tables, heat maps, geo maps, scorecards, scatter charts, bullet charts, & area charts
  • Templates for quick start; customization options for impactful finish
  • Familiar G Suite sharing and real-time collaboration

Cloud Genomics 🧬– (Global) Store and process genomes🧬 and related experiments

  • Query complete genomic🧬 information of large research projects in seconds
  • Process many genomes🧬 and experiments in parallel
  • Open Industry standards (e.g. From Global🌎 Alliance for Genomics🧬 and Health)
  • Supports “Requester Pays” sharing

Identity and Access – Core Security🔒

Roles– (Global) collections of Permissions to use or manage GCP☁️ resources

  • Permissions allow you to perform certain actions: Service.Resource.Verb
  • Primitive Roles: Owner, Editor, Viewer
    • Viewer is read-only; Editor can change things; Owner can control access & billing
    • Pre-date IAM service, may still be useful (e.g., dev/test envs), but often too broad
  • Predefined Roles: Give granular access to specific GCP☁️ resources (IAM)
    • E.g.: roles/bigquery.dataEditor, roles/pub/sub.subscriber
  • Custom Roles: Project- or Org-level collections you define of granular permissions

Cloud Identity and Access Management (IAM)– (Global) Control access to GCP☁️ resources: authorization, not really authentication/identity

  • Member is user, group, domain, service account, or the public (e.g. “allUsers”)
    • Individual Google account, Google group, G Suite/ Cloud Identity domain
    • Service account belongs to application/instance, not individual end user
    • Every identity has a unique e-mail address, including service accounts
  • Policies bind Members to Roles at a hierarchy👑 level: Org, Folder📂, Project, Resource⚙️
    • Answer: Who can do what to which thing(s)?
  • IAM is free; pay for authorized GCP☁️ service usage

Service Accounts– (Global) Special types of Google account that represents an application, not an end user

  • Can be “assumed” by applications or individual users (when so authorized)
  • “Important: For almost all cases, whether you are developing locally or in a production application, you should use service accounts, rather than user accounts or API keys🔑.”
  • Consider resources and permissions required by application; use least privilege
  • Can generate and download private keys🔑 (user-managed keys🔑), for non-GCP☁️
  • Cloud-Platform-managed keys🔑 are better, for GCP☁️ (i.e. GCF, GAE, GCE, and GKE)
    • No direct downloading: Google manages private keys🔑 & rotates them once a day

Cloud Identity– (Global) Identity as a Service (IDaaS, not DaaS) to provision and manage users and groups

  • Free Google Accounts for non-G-Suite users, tied to a verified domain
  • Centrally manage all users in Google Admin console; supports compliance
  • 2-Step verification (2SV/MFA) and enforcement, including security🔒 keys🔑
  • Sync from Active Directory and LDAP directories via Google Cloud☁️ Directory Sync
  • Identities work with other Google services (e.g. Chrome)
  • Identities can be used to SSO with other apps via OIDC, SAML, OAuth2
  • Cloud Identity is free; pay for authorized GCP☁️ service usage

Security Key Enforcement– (Global) USB or Bluetooth 2-step verification device that prevents phishing🎣

  • Not like just getting a code via email📧 or text message…
  • Eliminates man-in-the-middle (MITM) attacks against GCP☁️ credentials

Cloud Resource Manager– (Global) Centrally manage & secure organization’s projects with custom Folder📂 hierarchy👑

  • Organization Resource⚙️ is root node in hierarchy👑, folders📂 per your business needs
  • Tied 1:1 to a Cloud Identity / G Suite domain, then owns all newly created projects
  • Without this organization, specific identities (people) must own GCP☁️ projects
  • “Recycle bin” allows undeleting projects
  • Define custom IAM policies at organization, Folder📂, or project levels
  • No charge for this service

Cloud Identity-Aware Proxy (IAP)– (Global) Guards apps running 🏃‍♂️ on GCP☁️ via identity verification, not VPN access

  • Based on CLB & IAM, and only passes authed requests through
  • Grant access to any IAM identities, incl. group & service accounts
  • Relatively straightforward to set up
  • Pay for load balancing🏋️‍♀️ / protocol forwarding rules and traffic🚦

Cloud Audit Logging– (Global) “Who did what, where and when?” within GCP☁️ projects

  • Maintains non-tamperable audit logs for each project and organization:
  • Admin Activity and System Events (400-day retention)
  • Access Transparency (400-day retention)
  • Shows actions by Google support staff
  • Data Access (30-day retention)
  • For GCP-visible services (e.g. Can’t see into MySQL DB on GCE)
  • Data Access logs priced through Stackdriver Logging; rest are free

Security Management – Monitoring🎛 and Response

Cloud Armor🛡 – (Global) Edge-level protection from DDoS & other attacks on Global🌎 HTTP(S) LB🏋️‍♀️

  • Offload work: Blocked attacks never reach your systems
  • Monitor: Detailed request-level logs available in Stackdriver Logging
  • Manage Ips with CIDR-based allow/block lists (aka whitelist/blacklist)
  • More intelligent rules forthcoming (e.g. XSS, SQLi, geo-based🌎, custom)
  • Preview effect of changes before making them live
  • Pay per policy and rule configured, plus for incoming request volume

Cloud Security Scanner– (Global) Free but limited GAE app vulnerability scanner with “very low false positive rates”

  • “After you set up a scan, Cloud☁️ Security🔒 Scanner automatically crawls your application, following all links🔗 within the scope of your starting URLs, and attempts to exercise as many user inputs and event handlers as possible.”
  • Can identify:
    • Cross-site-scripting (XSS)
    • Flash🔦 injection💉
    • Mixed content (HTTP in HTTPS)
    • Outdated/insecure libraries📚

Cloud Data Loss Prevention API (DLP) – (Global) Finds and optionally redacts sensitive info is unstructured data streams

  • Helps you minimize what you collect, expose, or copy to other systems
  • 50+ sensitive data detectors, including credit card numbers, names, social security🔒 numbers, passport numbers, driver’s license numbers (US and some other jurisdictions), phone numbers, and other personally identifiable information (PII)
  • Data can be sent directly, or API can be pointed at GCS, BQ, or Cloud☁️ DataStore
  • Can scan both text and images🖼
  • Pay for amount of data processed (per GB) –and gets cheaper when large volume
    • Pricing for storage 🗄now very simple (June 2019), but for streaming is still a mess

Event Threat Detection (ETD)– (Global) Automatically scans your Stackdriver logs for suspicious activity

  • Uses industry-leading threat intelligence, including Google Safe Browsing
  • Quickly detects many possible threats, including:
    • Malware, crypto-mining, outgoing DDoS attacks, port scanning, brute-force SSH
    • Also: Unauthorized access to GCP☁️ resources via abusive IAM access
  • Can export parsed logs to BigQuery for forensic analysis
  • Integrates with SIEMs like Google’s Cloud☁️ SCC or via Cloud Pub/Sub
  • No charge for ETD, but charged for its usage of other GCP☁️ services (like SD Logging)

Cloud Security Command Center (SCC) – (Global)

  • “Comprehensive security🔒 management and data risk platform for GCP☁️”
  • Security🔒 Information and Event Management (SIEM) software
  • “Helps you prevent, detect & respond to threats from a single pane of glass”
  • Use: Security🔒 Marks” (aka “marks”) to group, track, and manage resources
  • Integrate ETD, Cloud☁️ Scanner, DLP, & many external security🔒 finding sources
  • Can alert 🔔 to humans & systems; can export data to external SIEM
  • Free! But charged for services used (e.g. DLP API, if configured)
  • Could also be charged for excessive uploads of external findings

Encryption Key Management 🔐

Cloud Key Management Services (KMS)– (Regional, Multi-Regional, Global) Low-latency service to manage and use cryptographic keys🔑

  • Supports symmetric (e.g. AES) and asymmetric (e.g. RSA, EC) algorithms
  • Move secrets out of code (and the like) and into the environment, in a secure way
  • Integrated with IAM & Cloud☁️ Audit Logging to authorize & track key🔑 usage
  • Rotate keys🔑 used for new encryption 🔐 either automatically or on demand
    • Still keeps old active key🔑 versions, to allow decrypting
  • Key🔑 deletion has 24-hour delay, “to prevent accidental or malicious data loss”
  • Pay for active key🔑 versions stored over time
  • Pay for key🔑 use operations (i.e. encrypt/decrypt; admin operation are free)

Cloud Hardware Security Module (HSM)– (Regional, Multi-Regional, Global) Cloud KMS keys🔑 managed by FIPS 140-2 Level 3 certified HSMs

  • Device hosts encryption 🔐 keys🔑 and performs cryptographic operations
  • Enables you to meet compliance that mandates hardware environment
  • Fully integrated with Cloud☁️ KMS
    • Same API, features, IAM integration
  • Priced like Cloud KMS: Active key🔑 versions stored & key🔑 operations
    • But some key🔑 types more expensive: RSA, EC, Long AES

Operations and Management

Google Stackdriver– (Global) Family of services for monitoring, logging & diagnosing apps on GCP/AWS/hybrid

  • Service integrations add lots of value – among Stackdriver and with GCP☁️
  • One Stackdriver account can track multiple:
    • GCP☁️ projects
    • AWS☁️ accounts
    • Other resources
  • Simple usage-based pricing
    • No longer previous system of tiers, allotments, and overages

Stackdriver Monitoring– (Global) Gives visibility into perf, uptime, & overall health of Cloud☁️ apps (based on collectd)

  • Includes built-in/custom metrics, dashboards, Global🌎 uptime monitoring, & alerts
  • Follow the trail: Links🔗 from alerts to dashboards/charts to logs to traces
  • Cross-Cloud☁️: GCP☁️, of course, but monitoring🎛 agent also supports AWS
  • Alerting policy config includes multi-condition rules & Resource⚙️ organization
  • Alert 🔔 via email, GCP☁️ Mobile App, SMS, Slack, PagerDuty, AWS SNS, webnook, etc.
  • Automatic GCP☁️/Anthos metrics always free
  • Pay for API calls & per MB for custom or AWS metrics

Stackdriver Logging – (Global) Store, search🔎, analyze, monitor, and alert 🔔 on log data & events (based on Fluentd)

  • Collection built into some GCP☁️, AWS support with agent, or custom send via API
  • Debug issues via integration with Stackdriver Monitoring, Trace & Error Reporting
  • Crate real-time metrics from log data, then alert 🔔 or chart them on dashboards
  • Send real-time log data to BigQuery for advanced analytics and SQL-like querying
  • Powerful interface to browse, search, and slice log data
  • Export log data to GCS to cost-effectively store log archives
  • Pay per GB ingested & stored for one month, but first 50GB/project free

Stackdriver Error Reporting– (Global) Counts, analyzes, aggregates, & tracks crashes in helpful centralized interface

  • Smartly aggregates errors into meaningful groups tailored to language/framework
  • Instantly alerts when a new app error cannot be grouped with existing ones
  • Link🔗 directly from notifications to error details:
    • Time chart, occurrences, affected user count, first/last seen dates, cleaned stack
  • Exception stack trace parser know Java☕️, Python🐍, JavaScript, Ruby💎,C#,PHP, & Go🟢
  • Jump from stack frames to source to start debugging
  • No direct charge; pay for source data in Stackdriver Logging

Stackdriver Trace– (Global) Tracks and displays call tree 🌳 & timings across distributed systems, to debug perf

  • Automatically captures traces from Google App Engine
  • Trace API and SDKs for Java, Node.js, Ruby, and God Capture traces from anywhere
  • Zipkin collector allows Zipkin tracers to submit data to Stackdriver Trace
  • View aggregate app latency info or dig into individual traces to debug problems
  • Generate reports on demand and get daily auto reports per traced app
  • Detects app latency shift (degradation) over time by evaluating perf reports
  • Pay for ingesting and retrieving trace spans

Stackdriver Debugger– (Global) Grabs program state (callstack, variables, expressions) in live deploys, low impact

  • Logpoints repeat for up to 24h; fuller snapshots run once but can be conditional
  • Source view supports Cloud Source Repository, Github, Bitbucket, local, & upload
  • Java☕️ and Python🐍 supported on GCE, GKE, and GAE (Standard and Flex)
  • Node.js and Ruby💎supported on GCE, GKE, and GAE Flex; Go only on GCE & GKE
  • Automatically enabled for Google App Engine apps, agents available for others
  • Share debugging sessions with others (just send URL)

Stackdriver Profiler– (Global) Continuous CPU and memory profiling to improve perf & reduce cost

  • Low overhead (typical: 0.5%; Max: 5%) – so use it in prod, too!
  • Supports Go, Java, Node,js, and Python (3.2+)
  • Agent-based
  • Saves profiles for 30 days
  • Can download profiles for longer-term storage

Cloud Deployment Manager– (Global) Create/manage resources via declarative templates: “Infrastructure as Code”

  • Declarative allows automatic parallelization
  • Templated written in YAML, Python🐍, or Jinja2
  • Supports input and output parameters, with JSON schema
  • Create and update of deployments both support preview
  • Free service: Just pay for resources involved in deployments

Cloud Billing API 🧾– (Global) Programmatically manage billing for GCP☁️ projects and get GCP☁️ pricing

  • Billing 🧾 Config
    • List billing🧾 accounts; get details and associated projects for each
    • Enable (associate), disable (disassociated), or change project’s billing account
  • Pricing
    • List billable SKUs; get public pricing (including tiers) for each
    • Get SKU metadata like regional availability
  • Export of current bill to GCS or BQ is possible – but configured via console, not API

Development and APIs

Cloud Source Repositories – (Global) Hosted private Git repositories, with integrations to GCP☁️ and other hosted repos

  • Support standard Git functionality
  • No enhanced workflow support like pull requests
  • Can set up automatic sync from GitHub or Bitbucket
  • Natural integration with Stackdriver debugger for live-debugging deployed apps
  • Pay per project-user active each month (not prorated)
  • Pay per GB-month of data storage 🗄(prorated), Pay per GB of Data egress

Cloud Build 🏗 – (Global) Continuously takes source code and builds, tests and deploys it – CI/CD service

  • Trigger from Cloud Source Repository (by branch, tag or commit) or zip🤐 in GCS
    • Can trigger from GitHub and Bitbucket via Cloud☁️ Source Repositories RepoSync
  • Runs many builds in parallel (currently 10 at a time)
  • Dockerfile: super-simple build+push – plus scans for package vulnerabilities
  • JSON/YAML file: Flexible🧘‍♀️ & Parallel Steps
  • Push to GCR & export artifacts to GCS – or anywhere your build steps wrtie
  • Maintains build logs and build history
  • Pay per minute of build time – but free tier is 120 minutes per day

Container Registry (GCR) 📦– (Regional, Multi-Regional) Fast🏃‍♂️, private Docker🐳 image storage 🗄 (based on GCS) with Docker🐳 V2 Registry API

  • Creates & manages a multi-regional GCS bucket 🗑, then translates GCR calls to GCS
  • IAM integration simplifies builds and deployments within GCP☁️
  • Quick deploys because of GCP☁️ networking to GCS
  • Directly compatible with standard Docker🐳 CLI; native Docker🐳 Login Support
  • UX integrated with Cloud☁️ Build & Stackdriver Logs
  • UI to manage tags and search for images🖼
  • Pay directly for storage 🗄and egress of underlying GCS (no overhead)

Cloud Endpoints – (Global) Handles authorization, monitoring, logging, & API keys🔑 for APIs backed by GCP☁️

  • Proxy instances are distributed and hook into Cloud Load Balancer 🏋️‍♀️
  • Super-fast🏃‍♂️ Extensible Service Proxy (ESP) container based on nginx: <1 ms /call
  • Uses JWTs and integrates with Firebase 🔥, AuthO, & Google Auth
  • Integrates with Stackdriver Logging and Stackdriver Trace
  • Extensible Service Proxy (ESP) can transcode HTTP/JSON to gRPC
    • But API needs to be Resource⚙️-oriented (i.e RESTful)        
  • Pay per call to your API

Apigee API Platform – (Global) Full-featured & enterprise-scale API management platform for whole API lifecycle

  • Transform calls between different protocols: SOAP, REST, XML, binary, custom
  • Authenticate via OAuth/SAML/LDAP: authorize via Role-Based Access Control
  • Throttle traffic🚦 with quotas, manage API versions, etc.
  • Apigee Sense identifies and alerts administrators to suspicious API behaviors
  • Apigee API Monetization supports various revenue models /rate pans
  • Team and Business tiers are flat monthly rate with API call quotas & feature sets
  • “Enterprise” tier and special feature pricing are “Contact Sales”

Test Lab for Android – (Global) Cloud☁️ infrastructure for running 🏃‍♂️ test matrix across variety of real Android devices

  • Production-grade devices flashed with Android version and locale you specify
  • Robo🤖 test captures log files, saves annotated screenshots & video to show steps
    • Default completely automatic but still deterministic, so can show regressions
    • Can record custom script
  • Can also run Espresso and UI Automator 2.0 instrumentation tests
  • Firebase Spark and Flame plans have daily allotment of physical and virtual tests
  • Blaze (PAYG) plan charges per device-hour-much less for virtual devices

“Well, we all shine☀️ on… Like the moon🌙 and the stars🌟 and the sun🌞

Thanks –


Week of October 23rd

Part III of a Cloud️ Journey

“I’m learning to fly ✈️, around the clouds ☁️

Hi All –

Happy National Mole Day! 👨‍🔬👩🏾‍🔬

“Open your eyes 👀, look up to the skies🌤 and see 👓

As we have all learned, Cloud computing ☁️ empowers us all to focus our time 🕰 on dreaming 😴 up and creating the next great scalable ⚖️ applications. In addition, Cloud computing☁️ enables us less worry😨 time 🕰 about infrastructure, managing and maintaining deployment environments or agonizing😰 over security🔒. Google evangelizes these principals stronger💪 than any other company in the world 🌎.

Google’s strategy for cloud computing☁️ is differentiated by providing open source runtime systems and a high-quality developer experience where organizations could easily move workloads from one cloud☁️ provider to another.

Once again, this past week we continued our exploration with GCP by finishing the last two courses as part of Google Cloud Certified Associate Cloud Engineer Path on Pluralsight. Our ultimate goal was to have a better understanding of the various GCP services and features and be able to apply this knowledge, to better analyze requirements and evaluate numerous options available in GCP. Fortunately, we gained this knowledge and a whole lot more! 😊

Guiding us through another great introduction on Elastic Google Cloud Infrastructure: Scaling and Automation were well-known friends Phillip Maier and Mylene Biddle. Then taking us through the rest of the way through this amazing course was the always passionate Priyanka Vergadia.

Then finally taking us down the home stretch 🏇 with Architecting with Google Kubernetes Engine – Foundations (which was last of this amazing series of Google Goodness 😊) were famous Googlers Evan Jones and Brice Rice.  …And Just to put the finishing touches 👐 on this magical 🎩💫 mystery tour was Eoin Carrol who gave us in depth look at Google’s game changer in Modernizing existing applications and building cloud-native☁️ apps anywhere with Anthos.

After a familiar introduction by Philip and Mylene we began delving into the comprehensive and flexible 🧘‍♀️ infrastructure and platform services provided by GCP.

“Across the clouds☁️ I see my shadow fly✈️… Out of the corner of my watering💦 eye👁

Interconnecting Networks – There are 5 ways of connecting your infrastructure to GCP:

  1. Cloud VPN
  2. Dedicated interconnect
  3. Partner interconnect
  4. Direct peering
  5. Carrier peering

Cloud VPN – securely connects your on-premises network to your GCP VPC network. In order to connect to your on-premise network via Cloud VPN configure cloud VPN, VPN Gateway, and to repeat in tunnels.

  • Useful for low-volume connections
  • 99.9% SLA
  • Supports:
    • Site-to-site VPN
    • Static routes
    • Dynamic routes (Cloud Router)
    • IKEv1 and IKEv2 ciphers

Please note: The maximum transmission unit or MTU for your on-premises VPN gateway cannot be greater than 1460 bytes.

$gcloud compute --project "qwiklabs-GCP -02-9474b560327d" target-vpn-gateways create "vpn-1" --region "us-central1" --network "vpn-network-1"

$gcloud compute --project "qwiklabs-GCP -02-9474b560327d" target-vpn-gateways create "vpn-1" --region "us-central1" --network "vpn-network-1"
$gcloud compute --project "qwiklabs-GCP -02-9474b560327d" forwarding-rules create "vpn-1-rule-esp" --region "us-central1" --address "" --IP -protocol "ESP" --target-vpn-gateway "vpn-1"

$gcloud compute --project "qwiklabs-GCP -02-9474b560327d" forwarding-rules create "vpn-1-rule-udp500" --region "us-central1" --address "" --IP -protocol "UDP" --ports "500" --target-vpn-gateway "vpn-1"

$gcloud compute --project "qwiklabs-GCP -02-9474b560327d" forwarding-rules create "vpn-1-rule-udp4500" --region "us-central1" --address "" --IP -protocol "UDP" --ports "4500" --target-vpn-gateway "vpn-1"

$gcloud compute --project "qwiklabs-GCP -02-9474b560327d" vpn-tunnels create "tunnel1to2" --region "us-central1" --ike-version "2" --target-vpn-gateway "vpn-1"

$gcloud compute --project "qwiklabs-GCP -02-9474b560327d" vpn-tunnels create "vpn-1-tunnel-2" --region "us-central1" --peer-address "" --shared-secret "GCP  rocks" --ike-version "2" --local-traffic-selector "" --target-vpn-gateway "vpn-1"

Cloud Interconnect and Peering

Dedicated connections provide a direct connection to Google’s network while shared connections provide a connection to Google’s network through a partner

Comparison of Interconnect Options

  • IPsec VPN Tunnel – Encrypted tunnel to VPC networks through the public internet
    • Capacity 1.5-3 Gbps/Tunnel
    • Requirements VPN Gateway
    • Access Type – Internal IP Addresses
  • Cloud Interconnect – Dedicated interconnect provides direct physical connections
    • Capacity 10 Gbps/Link -100 Gbps/ link
    • Requirements – connection in colocation facility
    • Access Type– Internal IP Addresses
  • Partner Interconnect– provides connectivity through a supported service provider
    • 50 Mbps -10 Gbps/connection
    • Requirements – Service provider
    • Access Type– Internal IP Addresses


Direct Peering provides a direct connection between your business network and Google.

  • Broad-reaching edge network locations
  • Capacity 10 Gbps/link
  • Exchange BGP routes
  • Reach all of Google’s services
  • Peering requirement (Connection in GCP PoPs)
  • Access Type: Public IP Addresses

Carrier Peering provides connectivity through a supported partner

  • Carrier Peering partner
  • Capacity varies based on parent offering
  • Reach all of Google’s services
  • Partner requirements
  • No SLA
  • Access Type: Public IP Addresses

Choosing the right connection – Decision Tree 🌳

Shared VPC and VPC Peering

Shared VPC allows an organization to connect Resource is from multiple projects to a common VPC network.

VPC Peering is a decentralized or distributed approach to multi project networking because each VPC network may remain under the control of separate administrator groups and maintains its own global firewall and routing tables.

Load Balancing 🏋️‍♂️ and Autoscaling

Cloud Load Balancing 🏋️‍♂️ – distributes user traffic 🚦across multiple instances of your applications. By spreading the load, load balancing 🏋️‍♂️ reduces the risk that your applications experience performance issues. There are 2 basic categories of Load balancers:  Global load balancing 🏋️‍♂️ and Regional load balancing 🏋️‍♂️.

Global load balancers – when workloads are distributed across the world 🌎 Global load balancers route traffic🚦 to a backend service in the region closest to the user, to reduce latency.

They are software defined distributed systems using Google Front End (GFE) reside in Google’s PoPs are distributed globally

Types of Global Load Balancers

  • External HTTP and HTTPS (Layer 7)
  • SSL Proxy (Layer 4)
  • TCP Proxy (Layer 4)

Regional load balancers – when all workloads are in the same region

  • Regional load balancing 🏋️‍♂️ route traffic🚦within a given region.
  • Regional Load balancing 🏋️‍♂️ uses internal and network load balancers.

Internal load balancers are software defined distributed systems (using Andromeda) and network load balancers which use  Maglev distributed system.

Types of Global Load Balancers

  • Internal TCP/UDP (Layer 4)
  • Internal HTTP and HTTPS (Layer 7)
  • TCP/UDP Network (Layer 4)

Managed instance groups – is a collection off identical virtual machine instances that you control as a single entity. (Same as creating a VM but applying specific rules to an Instance group)

  • Deploy identical instances based on instance template
  • Instance group can be resized
  • Manager ensures all instances are Running
  • Typically used with autoscaling ⚖️
  • Can be single zone or regional

Regional managed instance groups are usually recommended over zonal managed instance groups because this allow you to spread the application’s load across multiple zones through replication and protect against zonal failures.

         Steps to create a Managed Instance Group:

  1. Need to decide the location and whether instance group will be Single or multi-zones
  2. Choose ports that are going allow load balancing🏋️‍♂️ across.
  3. Select Instance template
  4. Decide Autoscaling ⚖️ and criteria for use
  5. Creatine a health ⛑ check to determine instance health and how traffic🚦should route

Autoscaling and health checks

Managed instance groups offer autoscaling ⚖️ capabilities

  • Dynamically add/remove instances
    • Increases in load
    • Decrease in load
  • Autoscaling policy
    • CPU Utilization
    • Load balancing 🏋️‍♂️capacity
    • Monitoring 🎛 metrics
    • Queue-based workload

HTTP/HTTPs load balancing

  • Target HTTP/HTTPS proxy
  • One singed SSL certificate installed (minimum)
  • Client SSL session terminates at the load balancer 🏋️‍♂️
  • Support the QUIC transport layer protocol
  • Global load balancing🏋️‍♂️
  • Anycast IP address
  • HTTP or port 80 or 8080
  • HTTPs on port 443
  • IPv4 or IP6
  • Autoscaling⚖️
  • URL maps 🗺

Backend Services

  • Health ⛑ check
  • Session affinity (Optional)
  • Time out setting (30-sec default)
  • One or more backends
    • An instance group (managed or unmanaged)
    • A balancing mode (CPU utilization or RPS)
    • A capacity scaler ⚖️ (ceiling % of CPU/Rate targets)

SSL certificates

  • Required for HTTP/HTTPS load balancing 🏋️‍♂️
  • Up to 10 SSL certificates /target proxy
  • Create an SSL certificate resource 

SSL proxy load balancing – global 🌎 load balancing service for encrypted non-http traffic.

  • Global 🌎 load balancing for encrypted non-HTTP traffic 🚦
  • Terminate SSL session at load balancing🏋️‍♂️ Layer
  • IPv4 or IPv6 clients
  • Benefits:
    • Intelligent routing
    • Certificate management
    • Security🔒 patching
    • SSL policies

TCP proxy load balancing – a global load balancing service for unencrypted non http traffic.

  • Global load balancing for encrypted non-HTTP traffic 🚦
  • Terminates TCP session at load balancing🏋️‍♂️ Layer
  • IPv4 or IPv6 clients
  • Benefits:
    • Intelligent routing
    • Security🔒 patching

Network load balancing – is a regional non-proxy load balancing service.

  • Regional, non-proxied load balancer
  • Forwarding rules (IP protocol data)
  • Traffic:
    • UDP
    • TCP/SSL ports
  • Backends:
    • Instance group
    • Target pool🏊‍♂️ resources – define a group of instances that receive incoming traffic 🚦from forwarding rules:
      • Forwarding rules (TCP and UDP)
      • Up to 50 per project
      • One health check
      • Instances must be in the same region

Internal load balancing – is a regional private load balancing service for TCP and UDP based traffic🚦

  • Regional, private load balancing 🏋️‍♂️
    • VM instances in same region
    • RFC 1918 IP address
  • TCP/UDP traffic 🚦
    • Reduced latency, simpler configuration
    • Software-defined, fully distributed load balancing (is not based on a device or a virtual machine.)

“..clouds☁️ roll by reeling is what they say … or is it just my way?”

Infrastructure Automation – Infrastructure as code (IaC)

Automate repeatable tasks like provisioning, configuration, and deployments for one machine or millions.

Deployment Manager -is an infrastructure deployment service that automates the creation and management of GCP. By defining templates, you only have to specify the resources once and then you can reuse them whenever you want.

  • Deployment Manager is an infrastructure automation tool🛠
  • Declarative Language (allows you to specify what the configuration should be and let the system figure out the steps to take)
  • Focus on the application
  • Parallel deployment
  • Template-driven

Deployment manager creates all the resources in parallel.

Additional infrastructure as code tools🛠 for GCP

  • Terraform 🌱
  • Chef 👨‍🍳
  • Puppet
  • Ansible
  • Packer 📦

It’s recommended that you provisioned and managed resource is on GCP with the tools🛠 you are already familiar with

GCP Marketplace 🛒 – lets you quickly deploy functional software packages that run 🏃‍♂️ on GCP.

  • Deploy production-grade solutions
  • Single bill for GCP and third-party services
  • Manage solutions using Deployment Manager
  • Notification 📩 when a security update is available
  • Direct access to partner support

“… Must be the clouds☁️☁️ in my eyes 👀

Managed Services – automates common activities, such as change requests, monitoring 🎛, patch management, security🔒, and backup services, and provides full lifecycle 🔄 services to provision, run🏃‍♂️, and support your infrastructure.

BigQuery is GCP serverless, highly scalable⚖️, and cost-effective cloud Data warehouse

  • Fully managed
  • Petabyte scale
  • SQL interface
  • Very fast🏃‍♂️
  • Free usage tier

Cloud Dataflow🚰 executes a wide variety of data processing patterns

  • Server-less, fully managed data processing
  • Batch and stream processing with autoscale ⚖️
  • Open source programming using Apache beam 🎇
  • Intelligently scale to millions of QPS

Cloud Dataprep – visually explore, clean, and prepare data for analysis and machine learning

  • Serverless, works at any scale⚖️
  • Suggest ideal analysis
  • Integrated partner service operated by Trifacta

Cloud Dataproc – is a service for running Apache Spark 💥 and Apache Hadoop 🐘 clusters

  • Low cost (per-second, preemptible)
  • Super-fast🏃‍♂️ to start, scale ⚖️, and shut down
  • Integrated with GCP
  • Managed Service
  • Simple and familiar

“Captain Jack will get you by tonight 🌃… Just a little push, and you’ll be smilin’😊 “

Architecting with Google Kubernetes Engine – Foundations

After Steller🌟 job by was Priyanka taking us through the load balancing 🏋️‍♂️ options, infrastructure as code and some of the managed service options in GCP it was time to take the helm⛵️ and get our K8s ☸️ hat 🧢 on.

Cloud Computing and Google Cloud

Just to wet our appetite 😋 for Cloud Computing ☁️and Evan takes through 5 fundamental attributes:

  1. On-demand self-services (No🚫 human intervention needed to get resources)
  2. Broad network access (Access from anywhere)
  3. Resource Pooling🏊‍♂️ (Provider shares resources to customers)
  4. Rapid Elasticity 🧘‍♀️ (Get more resources quickly as needed)
  5. Measured 📏Service (Pay only for what you consume)

Next, Evan introduced some of GCP Services under Compute like Compute Engine, Google Kubernetes Engine (GKE), App Engine, Cloud Functions. He then discussed some Google’s managed services with Storage, Big Data, Machine Learning Services.

Resource Management


  • GCP provides resource is in multi-regions, regions and zones.
  • GCP divides the world 🌎 up into 3 multi-regional areas the Americas, Europe and Asia Pacific.
  • 3 multi regional areas are divided into regions which are independent geographic areas on the same continent.
  • Regions are divided into zones (like a data center) which are deployment areas for GCP resources.

The network interconnect with the public internet at more than 90 internet exchanges and more than 100 points of presence worldwide🌎 (and growing)

  • Zonal resources operate exclusively in a single zone
  • Regional resources span multiple zones
  • Global resources could be managed across multiple regions.
  • Resources have hierarchy
    • Organization is the root node of a GCP resource hierarchy
    • Folders reflect their hierarchy of your enterprise
    • Projects are identified by unique Project ID and Project Number
  • Cloud Identity Access Management (IAM) allows you to fine tune access controls on all resources in GCP.


  • Billing account pay for project resources
  • A billing account in linked to one or more projects
  • Charged automatically or invoiced every month or at threshold limit
  • Subaccounts can be used for separate billing for projects

How to keep billing under control

  1. Budgets and alerts 🔔
  2. Billing export 🧾
  3. Reports 📊
    • Quotas are helpful limits
      • Quotas apply at the level of GCP Project
      • There two types of quotas
        • Rate quotas reset after a specific time 🕰
        • Allocation quotas govern the number of resources in projects

GCP implements quotas which limit unforeseen extra billing charges. Quotas error designed to prevent the over consumption of resources because of an error or a malicious attack 👿.

Interacting with GCP -There are 4 ways to interact with GCP:

  1. Cloud Console
  • Web-based GUI to manage all Google Cloud resources
  • Executes common task using simple mouse clicks
  • Provides visibility into GCP projects and resources

2. SDK

3. Cloud Shell

  • gcloud
  • kubectl
  • gsutil
  • bq
  • bt
  • Temporary Compute Engine VM
  • Command-line access to the instance through a browser
  • 5 GB of persistent disk storage ($HOME dir)
  • Preinstalled Cloud SDK and other tools🛠
    • gcloud: for working with Compute Engine, Google Kubernetes Engine (GKE) and many Google Cloud services
    • gsutil: for working with Cloud Storage
    • kubectl: for working with GKE and Kubernetes
    • bq: for working with BigQuery
  • Language support for Java ☕️Go, Python 🐍 , Node.js, PHP, and Ruby♦️
  • Web 🕸preview functionality
  • Built-in authorization for access to resources and instances

4. Console mobile App

“Cloud☁️ hands🤲 reaching from a rainbow🌈 tapping at the window, touch your hair”

Introduction to Containers

Next Evan took us through a history of computing. First starting with deploying applications on its physical servers. This solution wasted resources and took a lot of time to deploy maintaining scale. It also wasn’t very portable. It all applications were built for a specific operating system, and sometimes even for specific hardware as well.

Next transitioning to Virtualization. Virtualization made it possible to run multiple virtual servers and operating systems on the same physical computer. A hypervisor is the software layer that removes the dependencies of an operating system with its underlying hardware. It allows several virtual machines to share that same hardware.

  • Hypervisors creates and manages virtual machines
  • Running multiple apps on a single VM
  • VM-centric way to solve this problem (run a dedicated virtual machine for each application.)

Finally, Evan introduced us to containers as they solve a lot of the short comings of Virtualization like:

  • Applications that share dependencies are not isolated from each other
  • Resource requirements from one application can starve out another application
  • A dependency upgrade for one application might cause another to simply stop working

Containers are isolated user spaces for running application code. Containers are lightweight as they don’t carry a full operating system. They could be scheduled or packed 📦tightly onto the underlying system, which makes them very efficient.

Containerization is the next step in the evolution of managing code.

Benefits of Containers:

  • Containers appeal to developers 👩🏽‍💻
  • Deliver high performing and scalable ⚖️ applications.
  • Containers run the same anywhere
  • Containers make it easier to build applications that use Microservices design pattern
    • Microservices
      • Highly maintainable and testable.
      • Loosely coupled. Independently deployable. Organized around business capabilities.

Containers and Container Images

An Image is an application and its dependencies

A container is simply a running 🏃‍♂️ instance image.

Docker 🐳 is an open source technology that allows you to create and run 🏃‍♂️ applications and containers, but it doesn’t offer away to orchestrate those applications at scale ⚖️

  • Containers use a varied set of Linux technologies
    • Containers use Linux name spaces to control what an application can see
    • Containers used Linux C groups to control what an application can use.
    • Containers use union file systems to efficiently encapsulate applications and their dependencies into a set of clean, minimal layers.
  • Containers are structured in layers
    • Container manifest is tool🛠 you used to build the image reads instructions from a file
    • Docker 🐳 file is a formatted container image
      • Each instruction in the Docker file specifies a layer (Read Only) inside the container image.
      • Readable ephemeral top layer
  • Containers promote smaller shared images

How to get containers?

  • Download containerized software from a container registry gcr.io
  • Docker 🐳 – Build your own container using the open-source docker 🐳 command
  • Build your own container using Cloud Build
    • Cloud Build is a service that executes your builds on GCP.
    • Cloud Build can import source code from
      • Google Cloud Storage
      • Cloud Source Repositories
      • GitHub,
      • Bitbucket

Introduction to Kubernetes ☸️

Kubernetes ☸️ is an open source platform that helps you orchestrate and manage your container infrastructure on premises or in the cloud☁️.

It’s a container centric management environment. Google originated it and then donated it to the open source community.

K8s ☸️ automates the deployment, scaling ⚖️, load balancing🏋️‍♂️, logging, monitoring 🎛 and other management features of containerized applications.

  • Facilitates the features of an infrastructure as a service
  • Supports Declarative configurations.
  • Allows Imperative configuration
  • Open Source

K8s ☸️ features:

  • Supports both stateful and stateless applications
  • Autoscaling⚖️
  • Resource limits
  • Extensibility

Kubernetes also supports workload portability across on premises or multiple cloud service providers. This allows Kubernetes to be deployed anywhere. You could move Kubernetes ☸️ workloads freely without vendor lock🔒 in

Google Kubernetes Engine (GKE)

GKE easily deploys, manages and scales⚖️ Kubernetes environments for your containerized applications on GCP.

GKE Features:

  • Fully managed
    • Cluster represent your containerized applications all run on top of a cluster in GKE.
    • Nodes are the virtual machines that host your containers inside of a GKE app cluster
  • Container-optimized OS
  • Auto upgrade
  • Auto repair🛠
  • Cluster Scaling⚖️
  • Seamless Integration
  • Identity and access management (IAM)
  • Integrated logging and monitoring (Stack Driver)
  • Integrated networking
  • Cloud Console

Compute Options Detail

 Computer Engine

  • Fully customizable virtual machines
  • Persistent disks and optional local SSDs
  • Global load balancing and autoscaling⚖️
  • Per-second billing

                  Use Cases

  • Complete control over the OS and virtual hardware
    • Well Suited for lift-and shift migrations to the cloud
    • Most flexible 🧘‍♀️ compute solution, often used when a managed solution is too restrictive

  App Engine

  • Provides a fully managed code-first platform
  • Streamlines application deployment and scalability⚖️
  • Provides support for popular programming language and application runtimes
  • Supports integrated monitoring 🎛, logging and diagnostics.

Use Cases

  • Websites
    • Mobile app📱 and gaming backends
    • Restful APIs

Google Kubernetes Engine

  • Fully managed Kubernetes Platform
  • Supports cluster scaling ⚖️, persistent disk, automated upgrades, and auto node repairs
  • Built-in integration with GCP
  • Portability across multiple environments
    • Hybrid computing
    • Multi-cloud computing

Use Cases

  • Containerized applications
    • Cloud-native distributed systems
    • Hybrid applications

Cloud Run

  • Enable stateless containers
  • Abstracts away infrastructure management
  • Automatically scales ⚖️ up⬆️ and down⬇️
  • Open API and runtime environment

Use Cases

  • Deploy stateless containers that listen for requests or events
    • Build applications in any language using any frameworks and tool🛠

Cloud Functions

  • Event-driven, serverless compute services
  • Automatic scaling with highly available and fault-tolerant design
  • Charges apply only when your code runs 🏃‍♂️
  • Triggered based on events in GCP, HTTP endpoints, and Firebase

Use Cases

  • Supporting microservice architecture
    • Serverless application backends
      • Mobile and IoT backends
      • Integrate with third-party services and APIs
    • Intelligent applications
      • Virtual assistant and chat bots
      • Video and image analysis

Kubernetes ☸️ Architecture

There are two related concepts in understanding K8s ☸️ works object model and principle of declarative management

Pods – the basic building block of K8s ☸️

  • Smallest deployable object.
  • Containers in a Pod share resources
  • Pods are not self-healing

Principle of declarative management – Declare some objects to represent those in generic containers.

  • K8s ☸️ creates and maintains one or more objects.
  • K8s ☸️ compares the desired state to the current state.

The Kubernetes ☸️Control Plane ✈️continuously monitor the state of the cluster, endlessly comparing reality to what has been declared and remedying the state has needed.

K8s ☸️ Cluster consists of a Master and Nodes

Master is to coordinate the entire cluster.

  • View or change the state of the cluster including launching pods.
  • kube-API server – the single component that interacts with the Cluster
    • kubectl server interacts with the database on behalf of the rest of the system
  • etcd – key-value store for the most critical data of a distributed system
  • kube-scheduler – assigns Pods to Nodes
  • kube-cloud-manager – embeds cloud-specific control logic.
  • Kube-controller-manager- daemon that embeds the core control loops

Nodes runs run pods.

  • kubelet is the primary “node agent” that runs on each node.
  • kube-proxy is a network proxy that runs on each node in your cluster

Google Kubernetes ☸️ Engine Concepts

GKE makes administration of K8s ☸️ much simpler

  • Master
    • GKE manages all the control plane components
    • GKE provisions and manages all the master infrastructure
  • Nodes
    • GKE manages this by deploying and registering Compute Engine instances as Nodes
    • Use node pools to manage different kinds of nodes
      • node pool (using nodemon) is a subset of nodes within a cluster that share a configuration, such as their amount of memory or their CPU generation.
      • nodemon is GKE specific feature
        • enable automatic node upgrades
        • automatic node repairs 🛠
        • cluster auto scaling ⚖️

Zonal Cluster – has a single control plane in a single zone.

  • single-zone cluster has a single control plane running in one zone
  • multi-zonal cluster has a single replica of the control plane running in a single zone, and has nodes running in multiple zones.

Regional Cluster – has multiple replicas of the control plane, running in multiple zones within a given region.

Private Cluster – provides the ability to isolate nodes from having inbound and outbound connectivity to the public internet.

Kubernetes ☸️ Object Management – identified by a unique name and a unique identifier.

  • Objects are defined in a YAML file
  • Objects are identified by a name
  • Objects are assigned a unique identifier (UID) by K8s ☸️
  • Labels 🏷 are key value pairs that tag your objects during or after their creation.
  • Labels 🏷 help you identify and organize objects and subsets of objects.
  • Labels 🏷 can be matched by label selectors

Pods and Controller Objects

Pods have a life cycle🔄

  • Controller Object types
    • Deployment – ensure that sets of Pods are running
      • To perform the upgrade, the Deployment object will create a second ReplicaSet object, and then increase the number of (upgraded) Pods in the second ReplicaSet while it decreases the number in the first ReplicaSet
    • StatefulSet
    • DaemonSet
    • Job
  • Allocating resource quotas
  • Namespaces – provide scope for naming resources (pods, deployments and controllers.)

There are 3 initializer spaces in the cluster.

  1. Default name space for objects with no other name space defined.
  2. Kube-system named Space for objects created by the Kubernetes system itself.
  3. Kube-Public name space for objects that are publicly readable to all users.

Best practice tip: namespace neutral YAML

  • Apply name spaces at the command line level which makes YAML files more flexible🧘‍♀️.

Advanced K8s ☸️ Objects


  • set of Pods and assigns a policy by which you can access those pods
    • Services provide load-balanced 🏋️‍♂️ access to specified Pods. There are three primary types of Services:
      • ClusterIP: Exposes the service on an IP address that is only accessible from within this cluster. This is the default type.
      • NodePort: Exposes the service on the IP address of each node in the cluster, at a specific port number.
      • LoadBalancer 🏋️‍♂️: Exposes the service externally, using a load balancing 🏋️‍♂️ service provided by a cloud☁️ provider.


  • A directory that is accessible to all containers in a Pod
    • Requirements of the volume can be specified using Pod specification
    • You must mount these volumes specifically on each container within a Pod
    • Set up Volumes using external storage outside of your Pods to provide durable storage

Controller Objects

  • ReplicaSets – ensures that a population of Pods
    • Deployments
      • Provides declarative updates to ReplicaSets and Pods
      • Create, update, roll back, and scale⚖️ Pods, using ReplicaSets
      • Replication Controllers – perform a similar role to the combination of ReplicaSets and Deployments, but their use is no longer recommended.
    • StatefulSets – similar to a Deployment, Pods use the same container specs
    • DaemonSets – ensures that a Pod is running on all or some subset of the nodes.
    • Jobs – creates one or more Pods required to run a task

“Can I get an encore; do you want more?”

Migrate for Anthos – tool🛠 for getting workloads into containerized deployments on GCP

  • Automated process that moves your existing applications into a K8s ☸️ environment.

Migrate for Anthos moves VMs to containers

  • Move and convert workloads into containers
  • Workloads can start as physical servers or VMs
  • Moves workload compute to container immediately (<10 min)
  • Data can be migrated all at once or “streamed” to the cloud☁️ until the app is live in the cloud☁️

Migrate for Anthos Architecture

  • A migration requires an architecture to be built
  • A migration is a multi-step process
  • Configure processing cluster
  • Add migration source
  • Generate and review plan
  • Generate artifacts
  • Test
  • Deploy

Migrate for Anthos Installation -requires a processing cluster

         Installing Migrate for Anthos uses migctl

$migctl setup install

         Adding a source enables migration from a specific environment

$migctl source create cd my-ce-src –project my-project –zone zone

         Creating a migration generates a migration plan

$migctl migration create test-migration –source my-ce-src –vm- id my-id –intent image

         Executing a migration generates resource and artifacts

$migctl migration generate-artifacts my-migration

         Deployment files typically need modification

$migctl migration get-artifacts test-migration

Apply the configuration to deploy the workload

$Kubectl apply -f deployment_sepc.yaml

“And we’ll bask 🌞 in the shadow of yesterday’s triumph🏆 And sail⛵️ on the steel breeze🌬

Below are some of the destinations I am considering for my travels for next week:

Thanks –


Week of October 16th

Part II of a Cloud☁️ Journey

“Cause I don’t want to come back… Down from this Cloud☁️”

Hi All –

Happy Global 🌎 Cat 😺 Day!

Last week, we started our continuous Cloud☁️ journey exploring Google Cloud☁️ to help us better understand the core services and the full value proposition that GCP can offer.

It has been said “For modern enterprise, that Cloud☁️ is the closest thing to magic🎩🐰 that we have.” Cloud☁️ enables companies large🏢 and small🏠 to be more agile and nimble. It also empowers employees of these companies🏭 to focus on being more creative and innovative and not being bogged down in the minutiae and rigors of managing IT infrastructure. In addition, customers of these companies benefit from an overall better Customer experience as applications are more available and scalable ⚖️.

As you might know, “Google’s mission is and has as always been to organize the world’s🌎 information and make it universally accessible and useful and as result playing a meaningful role in the daily lives of billions of people” Google has been able to hold true to this mission statement through its unprecedented success from products and platforms like Search 🔎, Maps 🗺, Gmail 📧, Android📱, Google Play, Chrome and YouTube 📺. Google continues to strive for the same kind of success with their Cloud Computing☁️ offering with GCP.

This week we continued our journey with GCP and helping us through the Google Cloud☁️ Infrastructure Essentials (Essential Google Cloud☁️ Infrastructure: Foundation & Essential Google Cloud☁️ Infrastructure: Core Services) through a combination of lectures and qwiklabs were esteemed Googlers Phillip Maier who seems to have had more cameos in the Google Cloud☁️ training videos than Stan Lee has made in all of the Marvel Movies combined and the very inspirational Mylene Biddle who exemplifies transformation both in the digital and real world.

Phillip and Mylene begin the course discussing Google Cloud☁️ which is a much larger ecosystem than just GCP. This ecosystem consists of open source software providers, partners, developers, third party software and other Cloud☁️ providers.

GCP uses a state-of-the-art software defined networking and distributed systems technologies to host and deliver services around the world🌎. GCP offers over 90 products and Services that continues to expand. GCP spans from infrastructure as a service or (IaaS) to software as a service (SaaS).

Next, Philip presented an excellent analogy comparing IT infrastructure to one of a city’s 🏙 infrastructure. “Infrastructure is the basic underlying framework of fundamental facilities and systems such as transport, 🚆communications 📞, power🔌, water🚰, fuel ⛽️ and other essential services. The people 👨‍👩‍👧‍👦 in the city 🏙 are like users 👥, and the cars 🚙🚗 and bikes 🚴‍♀️🚴‍♂️ buildings🏬 in the city 🏙 are like applications. Everything that goes into creating and supporting those applications for the users is the infrastructure.

GCP offers wide range of compute services including:

  • Compute Engine – (IaaS) runs virtual machines on demand
  • Google Kubernetes ☸️ Engine (IaaS/PaaS) – run containerized applications on a Cloud☁️ environment that Google manages under your administrative control.
  • App Engine (PaaS) is fully managed platform as a service framework. Run code in the Cloud☁️ without having to worry about infrastructure.
  • CloudFunctions (Serverless) It executes your code in response to events, whether those events occur once a day or many times ⏳

There are Four ways to interact with GCP

  1. Google Cloud Platform Console or GCP Console
  2. CloudShell and the Cloud SDK
  3. API
  4. Cloud Mobile 📱 App

CloudShell provides the following:

  • Temporary Compute Engine VM
  • Command-line access to the instance via a browser
  • 5 GB of persistent disk storage 🗄 ($HOME dir)
  • Pre-installed Cloud☁️ SDK and other tools 🛠
  • gCloud: for working with Compute Engine and many Google Cloud☁️ services
  • gsutil: for working with Cloud Storage 🗄
  • kubectl: for working with Google Container Engine and Kubernetes☸️
  • bq: for working with BigQuery
  • bt: for working with BigTable
  • Language support for Java☕️, Go, Python🐍, Node.js, PHP, and Ruby♦️
  • Web 🕸 preview functionality
  • Built-in authorization for access to resources and instances

“Virtual insanity is what we’re living in”

Virtual Networks

GCP uses a software defined network that is built on a Global 🌎 fiber infrastructure. This infrastructure makes GCP, one of the world’s 🌎 largest and fastest 🏃‍♂️networks.

Virtual Private Cloud☁️

Virtual Private Cloud☁️ (VPC) provides networking functionality to Compute Engine virtual machine (VM) instances, Google Kubernetes Engine (GKE) ☸️ containers, and the App Engine standard and flexible🧘‍♀️ environment. VPC provides networking for your Cloud-based☁️ services that is Global 🌎, scalable⚖️, and flexible🧘‍♀️.

VPC is a comprehensive set of Google managed networking objects:

  • Projects are used to encompass the Network Service as well as all other service in GCP
  • Networks come in three different flavors 🍨.
    • Default
    • Automotive
    • Custom mode.
  • Subnets allow for division or segregation of the environment.
  • Regions and zones (GCP DC) provide continues data protection and high availability.
  • IP addresses provided are internal or external
  • Virtual machines – instances from a networking perspective.
  • Routes and Firewall🔥rules allow or deny connections to or from VMs based specified configuration

Projects, Networks, and Subnets

A Project is the key organizer of infrastructure resources.

  • Associates objects and services with billing🧾
  • Contains networks (up to 5) that can be shared/peered

Networks are Global 🌎 are spans all available regions.

  • Has no IP address range
  • Contains subnetworks
  • Has three different options:
    • Default
      • Every Project
      • One subnet per region
      • Default firewall rules🔥
    • Automotive
      • Default network
      • One subnet per region
      • Regional IP allocation
      • Fixed /20 subnetwork per region
      • Expandable up to /16
    • Custom mode
      • No default subnets created
      • Full control of IP ranges
      • Regional IP allocation

VMs despite different locations geographically 🌎 take advantage of Google’s Global 🌎 fiber network. VMs appear as though they’re sitting in the same rack when it comes to a network configuration protocol.

  • VMs can be on the same subnet but in different zones
  • A single firewall rule 🔥 can apply to both VMs

Subnet is a ranges of IP addresses.

  • Every subnet has 4 reserved IP addresses in its primary IP Range.
  • Subnets can be expanded without re-creating instances or any down Time ⏳
    • Cannot overlap with other subnets
    • Must be inside the RFC 1918 address spaces
    • Can be expanded but cannot be shrunk
    • Auto mode can be expanded from /20 to /16
    • Avoid large subnets (don’t scale ⚖️ beyond what is actually needed)

IP addresses

VMs can have internal and external IP addresses

You also can assign a range of IP addresses as aliases to a VM’s network interface using IP range

Internal IP

  • Allocated from a subnet range to VMs by DHCP
  • DHCP lease is removed every 24 hours
  • VM name + IP is registered with network-scoped DNS

External IP

  • Assigned from pool (ephemeral)
  • Reserved (static)
  • VMs don’t know external IP
  • VMs are mapped to the internal IP

Mapping 🗺 IP addresses

DNS resolution for internal addresses

  • Each instance has a hostname that can be resolved to an internal IP address:
    • The hostname is the same as the instance name
    • FQDN is [hostname]. [zone].c.[project-id].internal
  • Name resolution is handled by internal DNS resolver
  • Configured for use on instance via DHCP
  • Provides answer for internal and external addresses

DNS resolution for external address

  • Instances with external IP addresses can allow connections from hosts outside the project
    • Users connect directly using external IP address
    • Admins can also publish public DNS records pointing to the instance
    • Public DNS records are not published automatically
  • DNS records for external addresses can be published using existing DNS servers (outside of GCP)
  • DNS zones can be hosted using Cloud DNS.

Host DNS zones using Cloud DNS

  • Google DNS service
  • Translate domain name into IP address
  • Low latency
  • High availability (100% uptime SLA)
  • Create and update millions of DNS records

Routes and firewall rules 🔥

Every network has:

  • Routes that let instances in a network send traffic🚦 directly to each other.
  • A default route that directs to destinations that are outside the network.

Routes map🗺 traffic🚦 to destination networks

  • Apply to traffic🚦 egress to a VM
  • Forward traffic🚦 to most specific route
  • Created when a subnet is created
  • Enable VMs on same network to communicate
  • Destination is in CIDR notation
  • Traffic🚦 is delivered only if it also matches a firewall rule 🔥

Firewall rules🔥 protect your VM instances from unapproved connections

  • VPC network functions as a distributed firewall. 🔥
  • Firewall rules🔥 are applied to the network as whole
  • Connections are allowed or denied at the instance level.
  • Firewall rules🔥 are stateful
  • Implied deny all ingress and allow all egress

Create Network

$gcloud compute networks create privatenet --subnet-mode=custom
$gcloud compute networks subnets create privatesubnet-us --network=privatenet --region=us-central1 --range=
$gcloud compute networks subnets create privatesubnet-eu --network=privatenet --region=europe-west1 --range=
$gcloud compute networks list
$gcloud compute networks subnets list --sort-by=NETWORK

Create firewall Rules 🔥

$gcloud compute firewall-rules create privatenet-allow-icmp-ssh-rdp --direction=INGRESS --priority=1000 --network=privatenet --action=ALLOW --rules=icmp,tcp:22,tcp:3389 --source-ranges=

Common network designs

  • Increased availability with multiple zones
    • A regional managed instance group contains instances from multiple zones across the same region, which provides increased availability.
  • Globalization 🌎 with multiple regions
    • Putting resource is in different regions, provides an even higher degree of failure independence by spreading resources across different failure domains
  • Cloud NAT provides internet access to private instances
    • Cloud NAT is Google’s Mesh Network address translation service. Provision application instances without public IP addresses, while also allowing them to access the Internet in a controlled and efficient manner.
  • Private Google Access to Google APIs and services
    • Private Google access to allow VM instances that only have internal IP addresses to reach the external IP addresses of Google APIs and services.

“Know you’re nobody’s fool… So welcome to the machine”

Compute Engine (IaaS)

Predefined or custom Machines types:

  • vCPUs (cores and Memory (RAM)
  • Persistent disks: HDD, SDD, and Local SSD
  • Networking
  • Linux or Windows


Several machine types

  • Network throughput scales⚖️ 2 Gbps per vCPU (small exceptions)
  • Theoretical max of 32 Gbps with 16 vCPU or 100 Gbps with T4 or V100 GPUs

A vCPU is equal to 1 hardware hyper-thread.

Storage 🗄


  • Standard, SSD, or Local SSD
  • Standard and SSD PB scale⚖️ in performance for each GB of space allocated

Resize disks or migrate instances with no downTime ⏳

Local SSD have even higher throughput and lower latency than SSD persistent disks because there are attached to the physical hardware. However, the data that you store local s SSDs persists only until you stop 🛑 or delete the instance.


         Robust network features:

  • Default, custom networks
  • Inbound/outbound firewall rules🔥
    • IP based
    • Instance/group tags
  • Regional HTTPS load balancing
  • Network load balancing
    • Does not require pre-warming
  • Global 🌎 and multi-regional subnetworks

VM access

Linux🐧 SSH (requires firewall to allow tcp:22)

  • SSH from Console CloudShell via Cloud SDK, computer

Windows RDP (requires firewall to allow tcp:3389)

  • RDP clients, PowerShell terminal

VM Lifecycle

Compute Engine offers live migration to keep your virtual machine instances running even when a host system event, such as a software or hardware update, occurs. Live migration keeps your instances running during the following events:

  • Regular infrastructure maintenance and upgrades.
  • Network and power 🔌 grid maintenance in the data centers.
  • Failed hardware such as memory, CPU, network interface cards, disks, power, and so on. This is done on a best-effort basis; if a hardware fails completely or otherwise prevents live migration, the VM crashes and restarts automatically and a hostError is logged.
  • Host OS and BIOS upgrades.
  • Security-related updates, with the need to respond quickly.
  • System configuration changes, including changing the size of the host root partition, for storage 🗄 of the host image and packages.

Compute Options

Machine Types

Predefined machine types Ratio of GB of memory per VCPU

  • Standard machine types
  • High-memory machine types
  • High-CPU machine types
  • Memory-optimized machine types
  • Compute-optimized machine types
  • Shared core machine types

Custom machine types:

  • You specify the amount of memory and number of VCPUs

Special compute configurations

Preemptible (ideal for running batch processing jobs)

  • Lower price for interruptible service (up to 80%)
  • VM might be terminated at any time ⏳
    • No charge if terminated in the first 10 minutes
    • 24 hours max
    • 30-second terminate warning, but not guaranteed
      • Time⏳ for a shutdown script
  • No live migrate; no auto restart
  • You can request that CPU quota for a region be split between regular and preemption
    • Default: preemptible VMs count against region CPU quota

Sole-tenant nodes -physically isolate workloads (Ideal for workloads that require physical isolation)

  • Sole-tenant node is a physical compute engine server that is dedicated to hosting VM Instances
  • if you have existing operating system licenses, you can bring them to compute engine using sole tenant notes while minimizing physical core usage with the in-place restart feature

Shielded VMs🛡offer verifiable integrity

  • Secure🔒 Boot
  • Virtual trusted platform module (vTPM)
  • Integrity Monitoring 🎛


  • Boot loader
  • Operating system
  • File System Structure 🗂
  • Software
  • Customizations

Disk options

Boot disk

  • VM comes with a single root persistent disk
  • Image is loaded onto root disk during first boot:
    • Bootable: you can attach to a VM and boot from it
    • Durable: can survive VM terminate
  • Some OS images are customized for Compute Engine
  • Can survive VM deletion if “Delete boot disk when instance is deleted” is disabled.

Persistent disks

  • Network storage 🗄 appearing as a block device 🧱
    • Attached to a VM through the network interface
    • Durable storage 🗄: can survive VM terminate
    • Bootable: you can attach to a VM and boot from it
    • Snapshots: incremental backups
    • Performance: Scales ⚖️ with Size
  • Features
    • HDD or SSD
    • Disk resizing
    • Attached in read-only mode to multiple VMs
    • Encryption keys 🔑

Local SSD disks are physically attached to a VM

  • More IOPS, lower latency and higher throughput
  • 375-GB up to 8 disks (3TB)
  • Data Survives a reset but not VM Stop 🛑 or terminate
  • VM Specific cannot be reattached to a different VM

RAM disk

  • tmps
  • Faster than local disk, slower memory
    • Use when your application expects a file system structure and cannot directly store its data in memory
    • Fast scratch disk, or fast cache
  • Very volatile: erase on stop 🛑 or reset
  • May require larger machine is RAM sized for application
  • Consider using persistent disk to back up RAM disk data

Common Compute Engine actions

  • Metadata and scripts (Every VM instance store its metadata on a metadata server)
  • Move an instance to a new zone
    • Automated process (moving within region)
      • gcloud compute instance move
      • updates reference to VM; not automatic
    • Manual process (moving between regions):
      • Snapshots all persistent disks
      • Create new persistent disk in destination zone disks restored from snapshots
      • Create new VMs in the destination zone and attach new persistent
      • Assign static IP to new VM
      • Update references to VM
      • Delete the snapshots, original disks and original VM
  • Snapshot: Back up critical data
  • Snapshot: Migrate data between zones
  • Snapshot: Transfer to SSD to improve performance
  • Persistent disk snapshots
    • Snapshot is not available for local SSD
    • Creates an incremental backup to Cloud Storage 🗄
      • Not visible in your buckets; managed by the snapshot service
      • Consider cron jobs for periodic incremental backup
    • Snapshots can be restored to a new persistent disk
      • New disk can be in another region or zone in the same project
      • Basis of VM migration: “moving” a VM to a new zone
        • Snapshot doesn’t back up VM metadata, tags, etc.
      • Resize persistent disk

You can grow disks, but never shrink them!

“If you had my love❤️ and I gave you all my trust 🤝. Would you comfort🤗 me?”

Identity Access Management (IAM)

IAM is a sophisticated system built on top of email, like address names, job type roles in granular permissions.

Who, Which, What

It is a way of identifying who can do what on which resource the who could be a person, group or application. The what refers to specific privileges or actions and the resource could be any DCP service.

  • Google Cloud☁️ Platform Resource is organized hierarchically
  • If you change the resource hierarchy, the policy hierarchy also changes.
  • A best practice is to follow the “principle of least privilege”.
  • Organization node is the root node in this hierarchy. Represents your company.
  • Folders 📂 are the Children of the organization. A Folder📂 could represent a department Cloud☁️
  • Projects are the Children of the Folders📂. Projects provide a trust boundary for a company
  • Resources are the Children of projects. Each Resource has exactly one parent Cloud☁️.


  • An organization node is a root node for Google Cloud☁️ resources
  • Organization roles:
  • Organization Admin: Control over all Cloud☁️ resources: useful for auditing
  • Project Creator: Controls project creation: Control project creation: control over who can create projects
  • Creating and managing Organization when a Workspace or IAM account creates a GCP Project. There are two roles assigned to t users or groups:
  • Super administrator:
  • Assign the Organization admin role to some users
    • Be the point of contact in case of recovery issues
    • Control the lifecycle 🔄 of the Workspace of Cloud☁️ Identity account and Organization
  • Organization admin:
  • Define IAM policies
    • Determine the structure of the resource hierarchy
    • Delegate responsibility over critical components such as Network, billing, and Resource Hierarchy through IAM roles

Folders 📂

  • Additional grouping mechanism and isolation boundaries between projects:
    • Different legal entities
    • Departments
    • Teams
  • Folders 📂 allow delegation of administration rights.


  • There are three types of roles in GCP:
  1. Primitive roles apply across all GCP services in a project
    • Primitive roles offer fixed, coarse-grained levels of access
      • Owner – Full privileges
      • Editor – Deploy, modify & configure
      • Viewer 👓 – Read-only access
      • *Billing Administrator – Manage Billing, Add Administrators
  2. Predefined roles apply to a particular service in a project
  1. Predefined roles offer more fine-grained permissions on a particular service
    • Example: Compute Engine IAM roles:
      • Compute Admin – Full control of Compute Engine
      • Network Admin – Create, modify, delete Network Resources (except FW rules and SSL Certs)
      • Storage Admin– Create, modify, delete disks, Images, and Snapshots
  2. Custom roles define a precise set of permissions


Defined the “who” part of who can do what on which resource.

There are five different types of members:

  1. Google account represents a developer, an administrator or any other person who interacts with GCP. Any email address can be associated with a Google account
  2. Service account is an account that belongs to your application instead of to an individual and user.
  3. Google group is unnamed collection of Google accounts and service accounts.
  4. Workspace domains represent your organization’s Internet domain name
  5. Cloud Identity domains manage users and groups using the Google Admin console, but you do not pay for or received Workspace collaboration products

Google Cloud Directory Sync ↔️ synchronizes ↔️ users and groups from your existing active directory or LDAP system with the users and groups in your Cloud identity domain. Synchronization ↔️ is one way only

Single Sign-om (SSO)

  • Use Cloud Identity to configure SAML, SSO
  • IF SAML2 isn’t supported, use a third-party solution

Service Accounts

  • Provide an identity for carrying out server-to-server interactions
    • Programs running within Compute Engine instances can automatically acquire access tokens with credentials
    • Tokens are used to access any service API or services in your project granted access to a service account
    • Service accounts are convenient when you’re not accessing user data
  • Service accounts are identified by an email address
    • Three types of Service accounts:
      • User-created (custom)
        • Built-in
          • Compute Engine and App Engine default service accounts
        • Google APIs Service account
          • Runs internal Google processes on your behalf
      • Default Compute Engine Service account
        • Automatically created per project with auto-generated name and email address:
        • Name has -compute suffix
        • [email protected]
        • Automatically added as a project Editor
        • By default, enabled on all instances created using glcoud or GCP console
  • Service account permissions
    • Default service accounts: primitive and predefined roles
    • User-created service accounts: predefined roles
    • Roles for Service accounts can be assigned to groups or users

Authorization is the process of determining what permissions and authenticated identity has on a set of specified resource(s)

Scopes are used to determine whether unauthenticated identity is authorized.

Customizing Scopes for a VM

  • Scopes can be changed after an instance is created
  • For user-created service accounts, use IAM roles instead.

IAM Best Practices

  1. Leverage and understand the resource hierarchy
  • Use Projects to group resources that share the same trust boundary
  • Check the policy granted on each resource and make sure you understand inheritance
  • Use “Principles of Least Privilege” when granting roles
  • Audit policies in Cloud☁️ audit logs: setiampolicy
  • Audit membership of groups used in policies
  • Grant roles to Google groups instead of individuals
  • Update group membership instead of changing IAM Policy
    • Audit membership of groups used in policies
    • Control the ownership of the Google group used in IAM policies
  • Service accounts
  • Be very careful granting serviceAccountUser role
    • When you create a service account, give it a display name that clearly identifies its purpose
    • Establish a naming convention for service accounts
    • Establish key rotation policies and methods
    • Audit with serviceAccount.keys.list() method

Cloud Identity-Aware Proxy (Cloud IAP)

Enforce access control policies for application and resources:

  • Identity-based access control
  • Central authorization layer for applications accessed by HTTPS

IAM policy is applied after authentication

“Never gonna give you up… Never gonna say goodbye.”

Storage 🗄 and Database Services 🛢

Cloud Storage 🗄 (Object Storage 🗄) – It allows worldwide🌎 storage 🗄 and retrieval of any amount of data at any Time ⏳.

  • Scalable ⚖️ to exabytes
  • Time⏳ to first byte in milliseconds
  • Very high availability across all storage 🗄 classes
  • Single API across storage 🗄 classes

Use Cases:

  • Website content
  • Storing data for archiving and disaster recovery
  • Distributing large data objects to users via direct download

Cloud Storage 🗄 has four storage 🗄 classes:

  1. Regional storage 🗄 enables you to store data at lower cost with the tradeoff of data being stored in a specific regional location.
  2. Multi regional storage 🗄 is geo redundant, Cloud Storage 🗄 stores, your data redundantly and at least two geographic locations separated by at least 100 miles within the multi-regional location of the bucket 🗑.
  3. Near line storage 🗄 is a low cost, highly durable storage 🗄 service for storing infrequently accessed data.
  4. Cold Line storage 🗄 is a very low cost, highly durable storage 🗄 service for data, archival, online backup and disaster recovery. Data is available within milliseconds, not hours or days.


  • Naming requirements
  • Cannot be nested
  • Regional bucket 🗑 & Multi-Regional cannot be changed
  • Objects can be moved from bucket 🗑 to bucket 🗑


  • Inherit storage 🗄 class of bucket 🗑 when created
  • No minimum size: unlimited storage 🗄


  • gsutil command

Access control lists (ACLs)

For some applications, it is easier and more efficient to grant limited Time ⏳ access tokens that can be used by any user instead of using account-based authentication for controlling resource access.

Signed URLs

“Valet Key” access to buckets 🗑 and objective via ticket:

  • Ticket is a cryptographically signed URL
  • Time-limited
  • Operations specified in ticket: HTTP GET, PUT DELETE (not POST)
  • Any user with URL can invoke permitted operations

Cloud Storage 🗄 Features

  • Customer-supplied encryption key (CSEK)
    • Use your own key instead of Google-managed keys 🔑
  • Object Lifecycle 🔄Management
    • Automatically delete or archive objects
  • Object Versioning
    • Maintain multiple versions of objects
      • Objects are immutable
      • Object Versioning:
        • Maintain a history of modification of objects
        • List archived versions of an object, restore an object to an older state, or delete a version
  • Directory synchronization ↔️
    • Synchronizes a VM directory with a bucket 🗑
  • Object change notification
  • Data import
  • Strong 💪 consistency

Object Lifecycle 🔄 Management policies specify actions to be performed on objects that meet certain rules

  • Examples:
    • Downgrade storage 🗄 class on objects older than a year.
    • Delete objects created before a specific date.
    • Keep only the 3 most recent versions of an object
  • Object inspection occurs in asynchronous batches
  • Changes can take 24 hours to apply

Object change notification can be used to notify an application when an object is updated or added to a bucket 🗑

Recommended: Cloud Pub/Sub Notifications for Cloud Storage 🗄

Data import services

  • Transfer Appliance: Rack, capture and then ship your data to GCP
  • Storage Transfer Service: Import online data (another bucket 🗑, S3 bucket 🗑, Web Service)
  • Offline Media Import: Third-party provider uploads the data from physical media

Cloud Storage 🗄 provides Strong 💪 global consistency

Cloud SQL is a fully managed database 🛢 service (MySQL or PostgreSQL)

  • Patches and updates automatically applied
  • You administer MySQL users
  • Cloud SQL supports many clients
    • gCloud sql
    • App Engine, Workspace scripts
    • Applications and tools 🛠
      • SQL Workbench Toad
      • External applications using standard MySQL drivers
  • Cloud SQL delivers high performance and scalability ⚖️ with up to 30 TBs of storage 🗄 capacity, 40,000 IOPS and 416 GB of RAM
  • Replica service that can replicate data between multiple zones
  • Cloud SQL also provides automated and on demand backups.
  • Cloud SQL scales ⚖️ up (require a restart)
  • Cloud SQL scales ⚖️ out using read replicas.

Cloud Spanner is a service built for the Cloud☁️ specifically to combine the benefits of relational database 🛢 structure with non-relational horizontal scale⚖️

Data replication is synchronized across zones using Google’s global fiber network

  • Scale⚖️ to petabytes
  • Strong 💪 consistency
  • High Availability
  • Used for financial and inventory applications
  • Monthly uptime ⏳
    • Multi-regional:99.999%
    • Regional: 99.99%

Cloud Firestore is a fast, fully managed serverless Cloud☁️ native NoSQL document database 🛢 that simplifies storing, syncing ↔️ and querying data for your Mobile 📱 Web and ioT applications a global scale⚖️.

  • Simplifies storing, syncing ↔️, and querying data
  • Mobile📱, web🕸, and IoT apps at global scale⚖️
  • Live synchronization ↔️ and offline support
  • Security🔒 features
  • ACID transactions
  • Multi-region replication
  • Powerful query engine

Datastore mode (new server projects):

  • Compatible with Datastore applications
  • Strong 💪 consistency
  • No entity group limits

Native mode (new Mobile 📱 and web🕸 apps):

  • Strongly 💪 consistent storage 🗄 layer
  • Collection and document 📄 data model
  • Real-time updates
  • Mobile📱 and Web🕸 Client libraries📚

Cloud Bigtable (Wide Column DB) is a fully managed, no SQL database 🛢 with petabytes scale⚖️ and very low latency.

  • Petabyte-scale⚖️
  • Consistent sub-10ms latency
  • Seamless scalability⚖️ for throughput
  • Learns and adjusts to access patterns
  • Ideal for Ad Tech, FinTech, and IoT
  • Storage 🗄 engine for ML applications
  • Easy integration with open source big data tools 🛠

Cloud MemoryStore is a fully managed Redis Service built on scalable ⚖️, Secure🔒 and highly available infrastructure managed by Google Applications.

  • In-memory data store service
  • Focus on building great apps
  • High availability, failover, patching and Monitoring 🎛
  • Sub-millisecond latency
  • Instances up to 300 GB
  • Network throughput of 12 Gbps
  • “Easy Lift-and-Shift”

Resource Management lets you hierarchically manage resources

  • Resources can be categorized by Project, Folder📂, and Organization
  • Resources are global🌎, regional, or zonal
  • Resource belongs to only one project
  • Resources inherent policies from their parents
  • Resource consumption is measured in quantities like rate of use or Time⏳
  • Policies contain a set of roles, and members and policies are set on
  • Policy is less restrictive; it overrides the more restrictive resource policy.
  • Organization node is root node for GCP resources
  • Organization contains all billing accounts.
  • Project accumulates the consumption of all its resources
    • Track resource and quota usage
    • Enable billing
    • Manage permissions and credentials
    • Enable services and APIs
  • Project use 3 identifying attributes
    • Project Name
    • Project Number (unique)
    • Project ID (unique)


All resources are subject to project quotas or limits


  • Total resources you can create per project: 5 VPC networks/project
    • Rate you make API requests in a project: 5 admin actions/second (Cloud☁️ Spanner)
    • Total resources you can create per region: 24 CPUs region/project

Increase: Quotas page in GCP Console or a support ticket

Your use of GCP expands over time, your quotas may increase accordingly.

Project Quotas:

  • Prevent runaway consumption in case of an error or malicious attack
    • Prevent billing spikes or surprises
    • Forces sizing consideration and periodic review

Labels and names

Labels 🏷 are a utility for organizing GCP resources

  • Attached to resources: VM, disk, snapshot, image
    • GCP Console, gCloud, or API
  • Example uses of labels 🏷 :
    • Inventory
    • Filter resources
    • In scripts
      • Help analyze costs
      • Run bulk operations

Comparing labels and tags

  • Labels 🏷 are a way to organize resources across GCP
  • Disks, image, snapshots
  • User-defined strings.in key-value format
  • Propagated through billing
  • Tags are applied to instances only
  • User-defined strings
  • Tags are primarily used for networking (applying firewall rules🔥)


  • Set a budget lets you track how you spend
  • Set a budget alerts 🔔send alerts 🔔and emails📧to Billing Admin
  • Use Cloud☁️ pubsub notifications to programmatically receive spend updates about this budget.
  • Optimize your GCP spend by using labels
  • Visualize GCP spend with Data Studio

Its recommended labeling all your resources and exporting billing data to Big Query to Analyze spend.

Every single day. Every word you say… Every game you play. Every night you stay. I’ll be watching you”

Resource Monitoring 🎛


  • Integrated Monitoring 🎛, Logging, Error Reporting, Tracing and Debugging
  • Manages across platforms
    • GCP and AWS
    • Dynamic discovery of GCP with smart defaults
    • Open-source agents and integrations
  • Access to powerful data and analytics tools 🛠
  • Collaboration with third-party software

Monitoring 🎛 is important to Google because it is at the base of Site Reliability Engineering (SRE).

  • Dynamic config and intelligent defaults
  • Platform, system, and application metrics
    • Ingests data: Metrics, events, metadata
    • Generates insights through dashboards, charts, alerts
  • Uptime /health checks⛑
  • Dashboards
  • Alerts 🔔

Workspace is the root entity that hold Monitoring 🎛 and configuration information

  • “Single pane of glass 🍸”
    • Determine your Monitoring 🎛 needs up front
    • Consider using separate Workspace for data and control isolation

To access an AWS account, you must configure a project in GCP to hold the AWS connector because workspaces can monitor all of your GCP projects in a single place.

Stack Driver Monitoring 🎛 allows you to create custom dashboards that contain charts 📊of the metrics that you want a monitor.

Uptime checks test the availability of your public services

Stock driver Monitoring 🎛 can access some metrics without the Monitoring 🎛 agent, including CPU utilization, some disk traffic 🚥metrics, network traffic and up Time⏳ information.

Stack Driver Logging provides logging, error, reporting, tracing and debugging.

  • Platform, systems, and application logs
    • API to write to logs
    • 30-day retention
  • Log search/view/filter
  • Log-based metrics
  • Monitoring 🎛 alerts 🔔can be set on log events
  • Data can be exported to Cloud Storage 🗄, BigQuery, and Cloud Pub/Sub
  • Analyze logs in BigQuery and visualize in Data Studio

Stack driver Error Reporting counts, analyzes and aggravates the errors in your running Cloud☁️ services

  • Error notifications
  • Error dashboard
  • Go, Java☕️, .NET, Node.js, PHP, Python 🐍, and Ruby♦️

Stack Driver Tracing is a distributed tracing system that collects Layton. See data from your applications and displays it in the GCP console.

  • Displays data in near real-time ⏳
  • Latency reporting
  • Per-URL latency sampling
  • Collects latency data
    • App Engine
    • Google HTTP(S) load balancers🏋️‍♀️
    • Applications instrumented with the Stackdriver Trace SDKs


  • Inspect an application without stopping 🛑 it or slowing it down significantly
  • Debug snapshots:
    • Capture call stack and local variables of a running application
  • Debug logpoints:
    • Inject logging into a service without stopping 🛑 it
  • Java☕️, Python🐍, Go, Node.js and Ruby♦️

“At last the sun☀️ is shining, the clouds☁️ of blue roll by”

We will continue next week with Part III of this series….

Thanks –


Week of September 12th

Now they always say, ‘Congratulations'”  🎉🥳

Happy International🌎 Programmers’👩🏽‍💻day!

So last week we decided to get our Microsoft Azure fundies on. So naturally, it would only make sense to test out our knowledge🤓 and take the AZ-900: Microsoft Azure Fundamentals certification exam 📝.

So, this week we decided to go down that road 🚙 and I am happy 😊 to report we passed.

“That’s what the flair is about… it’s about fun 😃!”  — Stan from Chotchkie’s

Ok, so besides picking up some additional flair to match our AWS Practitioner badge it does provide us with some additional street “Cloud ☁️ Cred” and we can also show lack of favoritism of one cloud ☁️ provider over another. The bottom line is all clouds☁️ are good 😊 and very powerful⚡️!

To help prepare for the exam📝, we turned to several resources. Fortunately, there is a lot of great content out there. For a quick primer we turned to John Savill 💪 who literally got pumped up 🏋🏻‍♂️ and delivered an excellent primer on Azure architecture and the Core Concepts through his AZ-900 Azure Fundamentals Hints and Tips Youtube video and to “Tech Trainer” Tim Warner who was happy 😊 to be our instructor 👨‍🏫 through his amazing Microsoft Azure Fundamentals Study Guide YouTube Series

Finally, just to get a feel for types of questions that might appear on the exam 📝 I purchased TestPrepTraining – Microsoft Azure Fundamentals (AZ-900) Practice Exam which provided over 1,000 Azure Fundamental-type questions. With that being said there were some similar questions I saw in the course’s practice tests but there were still many questions on my actual exam 📝 that I had never seen 👀 before. In fact, the questions were even in a completely different format then the courseware. Overall, I was quite impressed with how Microsoft structured this test. They made it quite challenging and with the expectations that you had to have practical experience using the Azure Portal and that you just didn’t memorize a bunch concepts and plethora of questions.

“My, my, my, I’m once bitten 🦈 twice shy baby”

For those who might remember when I first attempted to take the AWS Cloud Practitioner Exam, it didn’t exactly go as planned. So perhaps now, I might be a little gun shy ☺️ or just proponent of having a highly available and fault tolerant environment (like the cloud ☁️ itself) to be used for taking these exams 📝. So, in preparation of the exam 📝, I prepared 2 Mac books 💻 and one Windows laptop to be available with the secure OnVUE browser hours ⏳ before the test to mitigate against any unforeseen circumstances.

“Hold your head up… Keep your head up, movin’ on”

Below are some areas I am considering for my travels next week:

  • SQL Server 2019
  • Google Cloud Certified Associate Cloud Engineer Path

Thanks –


Week of September 4th

“You’re my blue sky, you’re my sunny️ day.”

Happy Labor Day👷👷🏽‍♀️ Weekend!

Back at the end of July, we decided to re-route course and go back to the basics with AWS Cloud☁️ focusing on the core concepts and principals of the AWS. Despite hitting a temporary obstacle, we subsequently took and passed the AWS Certified Cloud Practitioner certification exam📝 last week. Feeling the need to spread the love❤️ around the Troposphere we decided we should circle 🔵 back to Microsoft’s very popular cloud☁️ offering Azure and focus on the “fundies” or Fundamentals of Azure. Of course, this wasn’t our first time 🕰 at this rodeo🤠🐴. We spent several occasions in the Microsoft Stratosphere☁️☁️ before. The most recent was looking at Microsoft’s NoSQL Azure offerings. This time 🕰 we would concentrate specifically on General Cloud☁️ Concepts, Azure Architectural Components, Microsoft Azure Core Services, Security🔒, Privacy🤫, Compliance, and Pricing💰, Service Level Agreements, and Lifecycles. To obtain such knowledge we would need to explore several resources starting with our first course 🍽 of Azure fundamentals which was an amazing compilation of rich documentation, vignettes🎞 from current and former blue badgers/ Cloud☁️ Advocates Anthony Bortolo, Sonia Cuff, Phoummala Schmitt, Susan Hinton, Rick Claus, Christina Warren, Pierre Roman and Scott Cate and several short labs🧪 that give you free access to the Azure Cloud☁️ and let you implement solutions. For our second course 🍽 we went out to YouTube and found 5 ½ hours ⏳ of goodness 😊 with Paul Browning’s awesome videos on “Microsoft Azure Fundamentals (AZ 900) – Complete Course and then for encore we went to Pluralsight and visited with both Michael Brown and his Microsoft Azure Security🔒 and Privacy🤫 Concepts and with Steve Buchanan and his Microsoft Azure Pricing and Support Options because who can ever get enough of Security🔒 and Pricing💰?

“So, I look in the sky, but I look in vain…Heavy cloud☁️, but no rain🌧”

General Cloud️ Concepts

First, let’s review… What is cloud☁️ computing anyway? There are numerous meanings out there. According to Wikipedia “Cloud☁️ computing is the on-demand availability of computer system resources, especially data storage🗄 (cloud☁️ storage🗄) and computing power🔌, without direct active management by the user. “

It’s really just a catchy name. So, despite contrary to belief Cloud☁️ Computing has nothing to do with the clouds☁️ ☁️ or the weather☔️ in the sky. In simplest terms it means sharing of pooled computing resources over the Internet. And are you ready for the catcher? “that you can rent”. In other words, you pay for what you use. Opposed to traditional computing way where a company or organization would invest in potentially expensive real estate to house owned Compute🖥, Storage🗄, Networking or fancy Analytics.

So now we are faced with the argument Capital expenditure (traditional computing cost model) versus operational expenditure (Cloud☁️ Computing cost model)

Capital expenditure (CapEx) 💰consists of the funds that a company uses to purchase major physical goods or services that the company will use for more than one year and the value will depreciate over time 🕰

Operational expenditure (OpEx) 💰are deducted in the same year they’re made, allowing you to deduct those from your revenues faster.

So, looking from a cost perspective, the cloud☁️ can offer a better solution for a better cost since the cloud☁️ provider’s already has those, so you would benefit from the economies of scale⚖️.

That’s great but let’s leave the expenses to the bean counters. After all we are technologists and we want the best performance and efficient technology solutions. So, what other benefits does Cloud☁️ provide me? How about Scalability ⚖️, Elasticity 🧘‍♀️, Agility 💃, Fault Tolerance, High Availability, Disaster🌪 Recovery and Security🔒.

  • Scalability ⚖️: Cloud☁️ will increase or decrease resources and services used based on the demand or workload at any given time 🕰. Cloud☁️ supports both vertical ⬆️ and horizontal ↔️ scaling ⚖️ depending on your needs.
  • Elasticity 🧘‍♀️: Cloud☁️ compensate spike 📈 or drop 📉 in demand by automatically adding or removing resources.
  • Agility 💃: Cloud☁️ eliminates the burdens of maintaining software patches, hardware setup, upgrades, and other IT management tasks. All of this is automatically done for you. Allowing you to focus on what matters: building and deploying applications.
  • Fault Tolerance: Cloud☁️ has fully redundant datacenters located in various regions all over the globe.
  • High Availability & Disaster Recovery: Cloud☁️ can replicate your services into multiple regions for redundancy and locality or select a specific region to ensure you meet data-residency and compliance laws for your customers.
  • Security🔒: Cloud☁️ offers a broad set of policies, technologies, controls, and expert technical skills that can provide better security🔒 than most organizations can otherwise achieve.

Ok, now I am sold. But what types of clouds☁️ are there?

There are multiple types of cloud☁️ computing services, but the three main ones are:

Infrastructure as a service (IaaS) – enables applications to run🏃🏻on the cloud☁️ instead of using their own infrastructures. Allows the most control over provided hardware that runs 🏃 your applications

Platform as a Service (PaaS)- enables developers to create as software without investing in expensive 🤑 hardware. Allows you to create an application quickly without managing the underlying infrastructure.

Software as a Service (SaaS) – provides answers to desktop needs for end users. Based on an architecture where one version of the application is used for all customers and licensed through a monthly or annual subscription. 

What about Cloud☁️ Deployment models? Well, there are multiple types of cloud☁️ deployment models out there as well.

Public cloud☁️: cloud☁️ vendor that provides cloud☁️ services to multiple clients. All of the clients securely 🔒 share the same hardware in the back end.

Private cloud️: organization uses their own hardware and software resources to achieve cloud☁️ services.

Hybrid cloud️: this cloud☁️ model is a combination of both private and public cloud☁️ models.

Community cloud☁️: this model consists of a pool of computer resources. These resources are available to the different organizations with common needs. Clients or tenants can access the resources quickly and securely🔒. Clients are referred to as tenants.

“Blue skies, smilin’ 😊 at me Nothin’ but blues skies do I see”

So now that we expounded the virtues of Cloud☁️ computing Concepts let’s take a deeper a look on what we came for…

Microsoft Azure is a cloud☁️ computing service created by Microsoft for building, testing, deploying, and managing applications and services through a global network of Microsoft managed data centers.

“Architecture starts when you carefully put two bricks🧱 together. There it begins.”

Azure Architectural Components

Microsoft Azure is made up of data centers located around the globe 🌎. These data centers are organized and made available to end users by region. A region is a geographical 🌎 area on the planet 🌎 containing at least one, but potentially multiple data centers that are in close proximity and networked together with a low-latency network.

Azure divides the world 🌎 into geographies 🌎 that are defined by geopolitical boundaries or country borders. An Azure geography is a discrete market typically containing two or more regions that preserves data residency and compliance boundaries.

Availability sets are a way for you to ensure your application remains online if a high-impact maintenance event is required, or if a hardware failure occurs. Availability sets are made up of Update domains (UD) and Fault domains (FD).

Fault domains is a logical group of underlying hardware that share a common power🔌 source and network switch, similar to a rack within an on-premise data center.

Update domains is a logical group of underlying hardware that can undergo maintenance or be rebooted at the same time 🕰. An update domain is a group of VMs that are set for planned maintenance events at the same time 🕰.

Paired regions support redundancy across two predefined geographic 🌎 regions, ensuring that even if an outage affects an entire Azure region, your solution is still available.

Additional advantages of regional pairs:

  • In the event of a wider Azure outage, one region is prioritized out of every pair to help reduce the time 🕰 to restore for applications.
  • Planned Azure updates are rolled out to paired regions one at a time 🕰 to minimize downtime 🕰 and risk of application outage.
  • Data continues to reside within the same geography as its pair (except for Brazil South) for tax and law enforcement jurisdiction purposes.

Availability Zones are physically separate locations within an Azure region that use availability sets to provide additional fault tolerance.

Resource group is a unit of management for your resources in Azure. A resource group is like container that allows you to aggregate and manage all the resources required for your application in a single manageable unit.

Azure Resource Manager is a management layer in which resource groups and all the resources within it are created, configured, managed, and deleted.

“It is our choices, Harry, that show what we truly are, far more than our abilities.”― J.K. Rowling

Azure Services

Azure provides over 100 services that enable you to do everything from running 🏃 your existing applications on virtual machines to exploring 🔦new software paradigms such as intelligent bots and mixed reality. Below are some of the services available in Azure:

Azure compute 🖥 – is an on-demand computing service for running cloud-based ☁️ applications.

There are four common techniques for performing compute 🖥 in Azure:

  1. Virtual machines – software emulations of physical computers 🖥 .
  2. Containers – virtualization environment for running 🏃 applications.
  3. Azure App Service – (PaaS) offering in Azure that is designed to host enterprise-grade web-oriented applications
  4. Serverless computing – cloud☁️ hosted execution environment that runs 🏃 your code but completely abstracts the underlying hosting environment.

Azure Virtual Machines (VMs) (IaaS) lets you create and use virtual machines in the cloud☁️.

Scaling VMs in Azure

Availability sets is a logical grouping of two or more VMs that help keep your application available during planned or unplanned maintenance 🧹.

  • Up to three fault domains that each have a server rack with dedicated power🔌 and network resources
  • Five logical update domains which then can be increased to a maximum of 20

Azure Virtual Machine Scale⚖️ Sets let you create and manage a group of identical, load balanced VMs.

Azure Batch enables large-scale⚖️ job scheduling and compute 🖥 management with the ability to scale⚖️ to tens, hundreds, or thousands of VMs.

  • Starts a pool of compute 🖥 VMs for you
  • Installs applications and staging data
  • Runs 🏃jobs with as many tasks as you have
  • Identifies failures
  • Re-queues work
  • Scales ⚖️ down the pool as work completes

Containers in Azure

Azure supports Docker🐳 containers (a standardized container model), and there are several ways to manage containers in Azure.

  • Azure Container Instances (ACI)
  • Azure Kubernetes Service (AKS)☸️
  1. Azure Container Instances (ACI) offers the fastest and simplest way to run🏃🏻 a container in Azure.
  2. Azure Kubernetes Service (AKS)☸️ is a complete orchestration service for containers with distributed architectures with multiple containers.

Containers are often used to create solutions using a microservice architecture. This architecture is where you break solutions into smaller, independent pieces.

Azure App Service

Azure App Service (PaaS) enables you to build and host web apps, background jobs, mobile📱backends, and RESTful 😴 APIs in the programming language of your choice without managing infrastructure. It offers automatic scaling ⚖️ and high availability.

App Service, you can host most common app service styles, including:

  1. Web apps- includes full support for hosting web apps using ASP.NET, ASP.NET Core, Java☕️ , Ruby 💎, Node.js, PHP, or Python 🐍 .
  2. API apps – build REST-based 😴 Web 🕸APIs using your choice of language and framework
  3. Web Jobs – allows you to run🏃🏻 a program (.exe, Java☕️ , PHP, Python 🐍 , or Node.js) or script (.cmd, .bat, PowerShell⚡️🐚, or Bash🥊) in the same context as a web 🕸 app, API app, or mobile 📱 app. They can be scheduled or run🏃🏻 by a trigger 🔫.
  4. Mobile📱app back-ends – quickly build a backend for iOS and Android apps.

Azure Networking

Azure Virtual Network enables many types of Azure resources such as Azure VMs to securely🔒 communicate with each other, the internet, and on-premises networks. Virtual networks can be segmented into one or more subnets. Subnets help you organize and secure🔒 your resources in discrete sections.

Azure Load Balancer is a load balancer service that Microsoft provides that helps take care of the maintenance for you. Load Balancer supports inbound and outbound scenarios, provides low latency and high throughput ↔️, and scales⚖️ up to millions of flows for all Transmission Control Protocol (TCP) and User Datagram Protocol (UDP) applications. Azure Application Gateway

VPN gateway is a specific type of virtual network gateway that is used to send encrypted traffic🚦 between an Azure Virtual Network and an on-premises location over the public internet. It provides a more secure 🔒 connection from on-premises to Azure over the internet.

Application Gateway is a load balancer designed for web applications. It uses Azure Load Balancer at the transport level (TCP) and applies sophisticated URL-based routing rules to support several advanced scenarios.

Here are some of the benefits of using Azure Application Gateway over a simple load balancer:

  • Cookie 🍪 affinity.
  • SSL termination
  • Web 🕸 application firewall🔥 (WAF)
  • URL rule-based routes.
  • Rewrite HTTP headers

Content delivery network (CDN) is a distributed network of servers that can efficiently deliver web 🕸 content to users. It is a way to get content to users in their local region to minimize latency.

Azure Storage🗄 is a service that you can use to store files📁, messages✉️, tables , and other types of information.

Disk storage🗄 provides disks for virtual machines, applications, and other services to access and use as they need, similar to how they would in on-premises scenarios. Disk storage🗄 allows data to be persistently stored and accessed from an attached virtual hard disk. 

Azure Blob storage🗄 is object storage🗄 solution for the cloud☁️. Blob storage🗄 is optimized for storing massive amounts of unstructured data, such as text or binary data.

Blob storage🗄 is ideal for:

  • Serving images or documents directly to a browser.
  • Storing files for distributed access.
  • Streaming video 📽 and audio 📻.
  • Storing data for backup and restore, disaster recovery, and archiving.
  • Storing data for analysis by an on-premises or Azure-hosted service.

Azure Files Storage🗄 enables you to set up highly available network file shares that can be accessed by using the standard Server Message Block (SMB) protocol. That means that multiple VMs can share the same files with both read and write access. You can also read the files using the REST 😴 interface or the storage🗄 client libraries 📚

File shares can be used for many common scenarios:

  • Many on-premises applications use file shares.
  • Configuration files 📂 can be stored on a file share and accessed from multiple VMs.
  • Diagnostic logs, metrics, and crash dumps are just three examples of data that can be written to a file 📂 share and processed or analyzed later.

Azure Archive Blob Storage🗄

Azure Archive Blob storage🗄 is designed to provide organizations with a low cost means of delivering durable, highly available, secure cloud☁️ storage🗄 for rarely accessed data with flexible latency requirements. Azure storage🗄 offers different access tiers include:

  • Hot 🥵 – Optimized for storing data that is accessed frequently.
  • Cool 🥶 – Optimized for storing data that is infrequently accessed and stored for at least 30 days.
  • Archive 🗃 – Optimized for storing data that is rarely accessed and stored for at least 180 days with flexible latency requirements (on the order of hours).

Storage🗄 Replication

Azure regions and geographies🌎 become important when you consider the available storage🗄 replication options. Depending on the storage🗄 type, you have different replication options.

  1. Locally redundant storage🗄 (LRS)- Replicates your data 3x within the region in which you create your storage🗄 account.
  2. Zone redundant storage🗄 (ZRS) – Replicates your data 3x across two to three facilities, either within a single region or across two regions.
  3. Geo-redundant storage🗄 (GRS) – Replicates your data to secondary region that is hundreds of miles away from the primary region.
  4. Read-access Geo-Redundant storage🗄 (RA-GRS)- Replicates your data to a secondary region, as with GRS, but also then provides read only access to the data in the secondary location.

Azure Database🛢 services are fully managed PaaS database🛢 services. Enterprise-grade performance with built-in high availability, scales⚖️ quickly and reach global 🌎 distribution.

Azure Cosmos DB 🪐is a globally 🌎distributed database🛢 service that enables you to elastically and independently scale⚖️ throughput and storage🗄 across any number of Azure’s geographic 🌎 regions. It supports schema-less data that lets you build highly responsive and Always On applications to support constantly changing data.

Azure SQL Database🛢 is a relational database🛢 as a service (DBaaS) based on the latest stable version of Microsoft SQL Server database🛢 engine. SQL Database🛢 is a high-performance, reliable, fully managed and secure database🛢 without needing to manage infrastructure. SQL database🛢 offers 4 service tiers to support lightweight to heavyweight 🏋️‍♀️ database🛢 loads:

  • Basic
  • Standard
  • Premium
  • Premium RS

Azure Database🛢 for MySQL is a relational database🛢 service powered by the MySQL community edition. It’s a fully managed database🛢 as a service offering that can handle mission-critical workloads with predictable performance and dynamic Scalability ⚖️. 

Azure Database🛢 for PostgreSQL is a relational database🛢 service based on the open-source Postgres database🛢 engine. It’s a fully managed database-as-a-service offering that can handle mission-critical workloads with predictable performance, security🔒, high availability, and dynamic Scalability ⚖️.

Azure Database🛢 Migration Service is a fully managed service designed to enable seamless migrations from multiple database🛢 sources to Azure data platforms with minimal downtime 🕰 (online migrations).

Dynamic Scalability enables your database🛢 to transparently respond to rapidly changing resource requirements and enables you to only pay for the resources that you need when you need them.

Elastic pools to maximize resource utilization

Elastic pools are designed dial performance up or down on demand especially if usage patterns are relatively predictable.

Azure Marketplace

Azure Marketplace is a service on Azure that helps connect end users with Microsoft partners, independent software vendors (ISVs), and start-ups that are offering their solutions and services, which are optimized to run🏃🏻on Azure. The solution catalog spans several industry categories:

  • Open-source container platforms
  • Virtual machine images
  • Databases🛢
  • Application build and deployment software
  • Developer tools🛠
  • Threat detection 🛡
  • Blockchain🔗

Internet of Things (IoT) 📲is the ability for devices to garner and then relay information for data analysis. There are many services that can assist and drive end-to-end solutions for IoT on Azure. Two of the core Azure IoT service types are:

  • IoT 📲 Central is a fully managed global IoT software as a service (SaaS) solution that makes it easy to connect, monitor, and manage your IoT assets at scale⚖️.
  • IoT 📲 Hub is a managed service hosted in the cloud☁️ that acts as a central message hub for bi-directional communication between your IoT application and the devices it manages. You can use Azure IoT Hub to build IoT solutions with reliable and secure communications between millions of IoT devices and a cloud☁️-hosted solution backend.
  • IoT📲 Edge is the technology from Microsoft for building Internet of Things (IoT) solutions that utilize Edge Compute. IoT Edge extends IoT Hub. Analyze device data locally instead of in the cloud to send less data to the cloud☁️, react to events quickly, and operate offline.

Big data and analytics – Data comes in all types of forms and formats. When we talk about big data, we’re referring to large volumes of data.

Azure Synapse Analytics is a limitless analytics service that brings together enterprise data warehousing and big data analytics. 

Azure HDInsight is a fully managed, open-source analytics service for enterprises. It is a cloud☁️ service that makes it easier, faster, and more cost-effective to process massive amounts of data.  HDInsight supports open-source frameworks and create cluster types:

  • Apache Spark ⭐️
  • Apache Hadoop 🐘
  • Apache Kafka
  • Apache HBase
  • Apache Storm🌧
  • Machine Learning Services

Microsoft Azure Databricks🧱 provides data science and data engineering teams with a fast, easy and collaborative Spark-based platform on Azure. It gives Azure users a single platform for Big Data processing and Machine Learning.

Artificial Intelligence (AI) 🧠is the creation of software that imitates human behaviors and capabilities. Key🔑 elements include:

  • Machine learning – This is often the foundation for an AI system, and is the way we “teach” a computer in model to make prediction and draw conclusions from data.
  • Anomaly detection – The capability to automatically detect errors or unusual activity in a system.
  • Computer vision👓 – The capability of software to interpret the world visually through cameras, video, and images.
  • Natural language processing – The capability for a computer to interpret written or spoken language and respond in kind.
  • Conversational AI – The capability of a software “agent” to participate in a conversation.

Azure Machine Learning service is a cloud-based☁️ platform for creating, managing, and publishing machine learning models. Azure Machine Learning provides the following features and capabilities:

  • Automated machine learning
  • Azure Machine Learning designer
  • Data and compute🖥 management
  • Pipelines

Azure Machine Learning studio is a web 🕸 portal for data scientist developers in Azure Machine Learning. The studio combines no-code and code-first experiences for an inclusive data science platform.

Serverless computing lets you run🏃🏻 application code without creating, configuring, or maintaining a server.  Azure has two implementations of serverless compute🖥:

  • Azure Functions, which can execute code in almost any modern language.
  • Azure Logic Apps, which are designed in a web-based designer and can execute logic triggered by Azure services without writing any code.

Azure Event Grid allows you to easily build applications with event-based architectures. Event Grid has built-in support for events coming from Azure services, like storage🗄 blobs and resource groups.

“DevOps brings together people, processes, and technology, automating software delivery to provide continuous value to your users.”

Azure DevOps Services allows you to create build and release pipelines that provide CI/CD (continuous integration, delivery, and deployment) for your applications.

  • Azure DevOps – (SaaS) platform from Microsoft that provides an end-to -end DevOps toolchain⛓ for developing and deploying software. It also integrates with most leading tools🛠 on the market and is a great option for orchestrating a DevOps toolchain⛓.
  • Azure DevTest Labs🧪 – (PaaS) enables developers on teams to efficiently self-manage virtual machines (VMs. DevTest Labs🧪 creates labs 🧪 consisting of pre-configured bases or Azure Resource Manager templates.

Azure management options

You can configure and manage Azure using a broad range of tools🛠 and platforms. Tools🛠 that are commonly used for day-to-day management and interaction include:

  • Azure portal for interacting with Azure via a Graphical User Interface (GUI)
  • Azure PowerShell⚡️🐚 cross-platform version of PowerShell⚡️🐚 that enables you to connect to your Azure subscription and manage resources.
  • Azure Command-Line Interface (CLI) cross-platform command-line program that connects to Azure and executes administrative commands
  • Azure Cloud☁️ Shell 🐚 interactive, authenticated, browser-accessible shell🐚 for managing Azure resources.
  • Azure mobile 📱 app access, manage, and monitor 🎛 all your Azure accounts and resources from your iOS 📱 or Android phone or tablet.
  • Azure SDKs for a range of languages and frameworks, and REST 😴 APIs manage and control Azure resources programmatically.

Azure Advisor is a free service built into Azure that provides recommendations on high availability, security🔒, performance, operational excellence, and cost. Advisor analyzes your deployed services and looks for ways to improve your environment across each of these areas. 

“Don’t worry about a thing Cause every little thing gonna be alright”

Security🔒, Privacy🤫, Compliance

Azure Advisor Security🔒 Assistance.

  • Azure Advisor Security🔒 Assistance integrates with Security🔒 Center.
  • Provide best practice security🔒 recommendations.
  • Azure Advisor Security🔒 Assistance helps prevent, detect, and respond to security🔒 threats.
  • You or your team should be using this tool every day to get the latest security🔒 recommendations.
  • Configuration of this tool 🔧, the amount of information it is gathering, the type of information it is gathering, is controlled through Security🔒 Center.

Securing Azure Virtual Networks

Network Security🔒 Groups (NSGs) – filter traffic🚦.

  • NSG has an inbound list and an outbound list.
  • Attached to subnets or network cards
  • Each NSG could be linked to multiple resources
  • NSG are stateful.

Application Security🔒 Groups – allow us to reference a group of resources

  • Used as even a source or destination of traffic🚦.
  • They do not replace network security🔒 groups.
  • Enhance them network security🔒 groups are still required.

When working with application security🔒 groups,

  • create the application security🔒 group
  •  link the application security🔒 group to a resource
  • use the application security🔒 group when working with network security🔒 groups.

Azure Firewall🔥 is a stateful firewall🔥 service and highly available solution provided by Azure. It’s a virtual appliance configured at the virtual network level. It protects access to your virtual networks. Features of Azure Firewall🔥 include:

  • Threat intelligence. 🧠
  • It supports both outbound and inbound NATing
  • Integrates with Azure Monitor 🎛
  • Network traffic🚦filtering rules
  • Unlimited in scale⚖️

Azure DDos protection provides DDoS mitigation for networks and applications.

Always on as a service.

  • Provides protection all the way up to the application layer.
  • Integrates Azure Monitor 🎛 for reporting services.
  • Features offered by Azure DDoS protection include:
    • Multi‑layered support, so protection from layer 4 attacks up to layer 7 attacks.
    • Attack analytics. So, we can get reports on attacks in progress, as well as post attack reports
    • Scale⚖️ and elasticity
    • Provides protection against unplanned costs💰.

Azure DDoS comes in two different service tiers, basic and standard.

Azure Web Application Firewall🔥 is designed to publish your applications to the outside world 🌎 , whether they’re in Azure or on‑premises, and lures bound traffic🚦 towards them.

Forced tunneling allows the control of flow of internet‑bound traffic🚦.

Security🔒 Scenarios

  • Control Internet traffic🚦 – User defined routes, Azure FW 🔥 or marketplace device
  • Azure hosted SQL Server – NSGs
  • VPN – Forced tunneling

Azure identity services

Identity services will help us in the authentication 🔐and authorization👮🏽‍♂️ of our users. Authentication works hand in hand with authorization.

Authentication🔐 – The act of proving who or what something is

Authorization 👮🏽‍♂️ – Granting the correct level of access to a resource or service

In Azure, authentication🔐 is provided by Azure AD and authorization is provided by role‑based access control.

Azure Active Directory is a cloud-based identity service used in Azure. It used to authenticate and authorize users. When we think Azure Active Directory, think single sign on. Azure Active Directory is not to be equivalent of Active Directory Domain Services used on-premise.

Active Directory Domains Services – full Active Directory Domain Service that we’ve used for years on‑premise.

Azure AD Domain Services (PaaS)- introduced to make it easier to migrate legacy applications as it supports both NTLM and Kerberos for authentication🔐. also supports Group Policies, trust🤝 relationships, as well as several over domain service features.

Azure AD Connect as being like a synchronization tool.

Multifactor authentication involves providing several pieces of information to prove who you are. Microsoft strongly recommends that we use multifactor authentication.

Azure Security🔒 Center reports our complying status against certain standards.

It provides continuous assessment of existing and new services that we deploy, it also provides threat protection for both infrastructure and Platform as a Service services.

Azure Key 🔑 Vault is a service we can use to protect our secrets. Azure Key 🔑 Vault uses hardware security🔒 modules. These hardware security🔒 modules have been validated to support the latest Federal Information Processing Standards.

Azure Information Protection (AIP) to classify documents 📃 and emails 📧. AIP applies Labels to documents 📃. Labeled documents 📃 can be protected. There are two sides to Azure Information Protection

These classifications come in the form of metadata that can be attached with a header or added as watermarks to the document you’re trying to protect.  Once classified, then the documents can be protected.

Classification of Documents. 📃

Azure uses Azure Rights Management to encrypt the documents using Rights Managements templates.

Azure Advanced Threat Protection 🛡 (Azure ATP) is a cloud☁️-based security🔒 solution that identifies, detects, and helps you investigate advanced threats, compromised identities, and malicious insider actions directed at your organization. Azure ATP 🛡 portal allows you to create your Azure ATP 🛡 instance, and view the data received from Azure ATP🛡 sensors.

Azure ATP 🛡sensors are installed directly on your domain controllers. The sensor monitors domain controller traffic🚦 without requiring a dedicated server or configuring port mirroring.

Azure ATP cloud☁️ service runs on Azure infrastructure and is currently deployed in the United States, Europe, and Asia. Azure ATP cloud☁️ service is connected to Microsoft’s intelligent security🔒 graph.

Azure Policy is a collection of rules. Each policy we create is assigned to a scope, such as an Azure subscription. When using Azure Policy, we create a policy definition, a policy assignment, and policy parameters,

When we create Azure policies, they can be used by themselves or they can be used with initiatives.  Initiatives are a collection of policies. To use initiatives, we create an initiative definition, an initiative assignment, and initiative parameters

Role Based Access Control (RBAC) is used daily by your organization. It’s central to access control in Azure. Azure provides shared access. RBAC is made up of several different components

  • Roles are groups of permissions that are needed to perform different administrative actions in Azure.  We then assign role members before configuring a scope for the role.
  • Scope details where a role can be used. There are many built‑in roles, each giving different sets of permissions, but three built‑in roles are used more than any other.

Three used most often roles are:

  • Owner role full control of that resource, including the ability to assign other users and group access.
  • Contributor role allows you to do everything except manage permissions.
  • Reader📖 role. This role is read‑only

Always follow the principle of least privilege.

Locks🔒prevent deletions or editing of resource groups and their content Two types Locks 🔒 are Read‑only and Delete.

If you make a resource group read‑only, then all the resources in there can be accessed, but no new resources can be added to the resource group or removed from the resource group.

Delete, then no resources can be deleted from the resource group, but new resources can be added.

Azure Blueprints – advanced way of orchestrating the deployment of resource templates and artifacts.

  • Blueprints maintain a relationship between themselves and the resources that they deployed.
  • Blueprints include Azure policy and initiatives as well as artifacts such as roles.

To use Blueprints, we require a Blueprint definition, we Publish the Blueprint, and then Assign it to a scope.

Blueprint definition

  • Resource groups can be defined and created
  • Azure policy can be included to enforce compliance
  • Azure resource manager templates can be included to deploy resources

Roles can be assigned to resources that blueprints have created

Azure Monitor to collect and analyze metric information both on premise and in Azure

Azure Service Health that we can use to see the health status of the Azure services

  • Personalized dashboards
  • Configurable alerts🔔
  • Guidance and Support

Service Trust🤝 Portal (STP) hosts the Compliance Manager service and is the Microsoft public site for publishing audit reports and other compliance-related information relevant to Microsoft’s cloud☁️ services. 

  • ISO
  • SOC
  • NIST
  • FedRAMP
  • GDPR

Microsoft Privacy🤫 Statement explains what personal data Microsoft processes, how Microsoft processes it, and for what purposes.

Microsoft Trust🤝 Center is a website resource containing information and details about how Microsoft implements and supports security🔒, Privacy🤫, compliance, and transparency in all Microsoft cloud☁️ products and services.

  • In-depth information about security🔒, Privacy🤫, compliance offerings, policies, features, and practices across Microsoft cloud☁️ products.
  • Recommended resources in the form of a curated list of the most applicable and widely used resources for each topic.
  • Information specific to key🔑 organizational roles, including business managers, tenant admins or data security🔒 teams, risk assessment and Privacy🤫 officers, and legal compliance teams.
  • Cross-company document search, which is coming soon and will enable existing cloud☁️ service customers to search the Service Trust🤝 Portal.
  • Direct guidance and support for when you can’t find what you’re looking for.

Compliance Manager is a workflow-based risk assessment dashboard within the Trust🤝 Portal that enables you to track, assign, and verify your organization’s regulatory compliance activities related to Microsoft professional services and Microsoft cloud☁️ services such as Microsoft 365, Dynamics 365, and Azure.

Compliance Manager provides the following features:

  • Detailed information provided by Microsoft to auditors and regulators (ISO 27001, ISO 27018, and NIST).
  • Compliance with regulations (HIPAA).
  • An organization’s self-assessment on compliance with these standards and regulations.
  • Enables you to assign, track, and record compliance and assessment-related activities
  • Provides a Compliance Score to help you track your progress and prioritize auditing
  • Provides a secure repository in which to upload and manage evidence and other artifacts
  • Produces richly detailed reports which can be provided to auditors and regulators

Special Azure regions exist for compliance and legal reasons. These regions are not generally available, and you have to apply to Microsoft if you want to use one of these special regions.

  • US 🇺🇸 Gov regions support US government agencies (US Gov Virginia and US Gov Iowa)
  • China 🇨🇳special regions. China East, China North. (Partnership with 21Vianet)
  • Germany 🇩🇪 regions. Germany Central and German Northeast. (compliant with German data laws)

There are two types of subscription boundaries that you can use, including:

Azure subscriptions provide you with authenticated and authorized access to Azure products and services and allows you to provision resources. An Azure subscription is a logical unit of Azure services that links to an Azure account, which is an identity in Azure Active Directory (Azure AD) or in a directory that an Azure AD trusts🤝.

An account can have one subscription or multiple subscriptions that have different billing models and to which you apply different access-management policies.

  • Billing boundary. This subscription type determines how an Azure account is billed for using Azure. 
  • Access control boundary. Azure will apply access-management policies at the subscription level, and you can create separate subscriptions to reflect different organizational structures. An example is that within a business, you have different departments to which you apply distinct Azure subscription policies. 

The organizing structure for resources in Azure has four levels: management groupssubscriptionsresource groups, and resources. The following image shows the relationship of these levels i.e. the hierarchy of organization for the various objects

Management groups:

Allow you to apply governance conditions (access & policies) a level of scope above subscriptions

These are containers that help you manage access, policy, and compliance for multiple subscriptions. The resources and subscriptions assigned to a management group automatically inherit the conditions applied to the management group.

Azure offers three main types of subscriptions:

  • A free account
  • Pay-As-You-Go
  • Member offers

There are three main customer types on which the available purchasing options for Azure products and services is contingent, including:

  • Enterprise
  • Web 🕸 direct
  • Cloud☁️ Solution Provider

Options for purchasing Azure products and services

  • Pay-As-You-Go Subscriptions
  • Open Licensing
  • Enterprise Agreements
  • Purchase Directly through a Cloud☁️ Solution Provider (CSP)
Azure free account

The Azure free account includes free access to popular Azure products for 12 months, a credit to spend for the first 30 days, and access to more than 25 products that are always free. 

Factors affecting costs💰

Resource type:

When you provision an Azure resource i.e. Compute🖥, Storage🗄, and Networking, Azure creates one or more-meter instances for that resource. The meters track the resources’ usage, and each meter generates a usage record that is used to calculate your bill.

Services: Azure usage rates and billing periods can differ between Enterprise, Web🕸 Direct, and Cloud☁️ Solution Provider (CSP) customers.


The Azure infrastructure is globally distributed, and usage costs💰 might vary between locations that offer Azure products, services, and resources.

All inbound or ingress data transfers to Azure data centers from on-premises environments are free. However, outbound data transfers (except in few cases like backup recovery) incur charges

Zones for billing purposes

A Zone is a geographical grouping of Azure Regions for billing purposes. the following Zones exist and include the sample regions as listed below:

  • Zone 1 – West US, East US 🇺🇸,Canada West 🇨🇦,West Europe🇪🇺,France Central 🇫🇷 and others
  • Zone 2 – Australia Central 🇦🇺, Japan West 🇯🇵 , Central India 🇮🇳 , Korea South 🇰🇷 and others
  • Zone 3 – Brazil South🇧🇷
  • DE Zone 1 – Germany Central, Germany Northeast 🇩🇪

Pricing Calculator 🧮

The Pricing Calculator 🧮 is a tool that helps you estimate the cost of Azure products. It displays Azure products in categories, and you choose the Azure products you need and configure them according to your specific requirements. Azure then provides a detailed estimate of the costs💰 associated with your selections and configurations.

Total Cost of Ownership Calculator

The Total Cost of Ownership Calculator is a tool that you use to estimate cost savings you can realize by migrating to Azure. To use the TCO calculator, complete the three steps that the following sections explain.

  1. Define your workloads
  2. Adjust assumptions
  3. View the report

Best Practices for Minimizing Azure Costs💰

  • Shut down unused resources
  • Right-size underused resources
  • Configure autoscaling
  • Reserved instances pre‑pay for resources at a discounted rate.
  • Azure cost management provides a set of tools🛠 for monitoring🎛, allocating, and optimizing your Azure costs💰. The main features of the Azure Cost Management toolset include:
  • Reporting
  • Data enrichment
  • Budgets
  • Alerting 🔔
  • Recommendations
  • Price
  • Quotas. place around the resources and the amount of resources that you’re using.
  • Spending limits as your approaching that spending limit, you won’t be able to deploy more resources and not going to go over a budget.
  • Azure Hybrid Benefit Migrate your workloads to Azure, the best cloud☁️ for Windows Server and SQL Server
  • Tags🏷 when deploying resources in Azure, you will want to tag your resources. You can use this to identify resources for chargeback in your organization.

SLAs for Azure products or services

An SLA defines performance targets for an Azure product or service. The performance targets that an SLA defines are specific to each Azure product and service.

  • SLAs describe Microsoft’s commitment to providing Azure customers with certain performance standards.
  • There are SLAs for individual Azure products and services.
  • SLAs also specify what happens if a service or product fails to perform to a governing SLA’s specification.

Service Credits

SLAs also describe how Microsoft will respond if an Azure product or service fails to perform to its governing SLA’s specification.

Application SLA

Azure customers can use SLAs to evaluate how their Azure solutions meet their business requirements and the needs of their clients and users. By creating your own SLAs, you can set performance targets to suit your specific Azure application. When creating an Application SLA consider the following:

  • Identify workloads.
  • Plan for usage patterns.
  • Establish availability metrics 
  • Establish recovery metrics 
  • Implement resiliency strategies.
  • Build availability requirements into your design.

Composite SLA

When combining SLAs across different service offerings, the resultant SLA is a called a Composite SLA. The resulting composite SLA can provide higher or lower uptime 🕰 values, depending on your application architecture.

Service lifecycle in Azure

Microsoft offers previews of Azure services, features, and functionality for evaluation purposes. With Azure Previews, you can test pre-release features, products, services, software, and even regions.

There are two categories of preview that are available:

  • Private preview – An Azure feature is available to certain Azure customers for evaluation purposes.
  • Public preview – An Azure feature is available to all Azure customers for evaluation purposes.

General availability

Once a feature is evaluated and tested successfully, the feature may be made available for all Azure customers. A feature released to all Azure customers typically goes to General Availability or GA.

The Azure updates  page provides the latest updates to Azure products, services, and features, as well as product roadmaps and announcements.

“On the road again…I just can’t wait to get on the road again”

Thanks –


Week of July 31st

A new day will dawn 🌄… For those who stand long”

Happy National Avocado🥑 Day!

Our journey 🚞 this week takes us back to our humble beginnings. Well, sort of… If you recall we began our magical✨ mystery tour of learnings back in March with AWS EC2. Then the last 2 weeks we re-routed course back to AWS, concentrating on AWS’s data services. So, we thought it might make sense to take one step 👣 back in order to take two steps 👣 👣 forward by focusing this week’s enlightenments on the fundamentals of the AWS Cloud☁️ and its Key🔑 concepts, core services, security🔐, architecture, pricing 💶, and support.

Fortunately, we knew the right place to load up on such knowledge. Where of course you ask? But to no other than the fine folks at AWS Training through their free online course AWS Cloud☁️ Practitioner Essentials (Second Edition). AWS spared no expense💰 by putting together an all-star🌟 lineup of AWS-er’s led by Kirsten Dupart, an old familiar friend, Blaine Sundrud ,Mike Blackmer,Raf Lopes, Heiwad Osman, Kent Rademacher , Russell Sayers ,Seph Robinson , Andy Cummings , Ian Falconer ,Wilson Santana ,Wes Gruver, Tipu Qureshi, and Alex Buell

The objective of the course was to highlight the following main areas:

  • AWS Cloud☁️ global infrastructure 
  • AWS Cloud☁️ architectural principles 
  • AWS Cloud☁️ value proposition
  • AWS Cloud☁️ overview of security🔐 and compliance
  • AWS Cloud☁️ overview of billing, account management, and pricing 💶 models

The course beings with introduction to the concept of “Cloud☁️ Computing” which of course is the on-demand availability of computing system resources, data Storage 🗄 and computing power⚡️, without direct active management by the user. Instead of having to design and build traditional data centers, Cloud☁️ computing enables us to access a data center and all of its resources, via the Internet or Cloud☁️.

Amazon Web Services (AWS) is a secure🔐 Cloud☁️ services platform, offering compute power⚡️, database Storage 🗄, content delivery and other functionality to help businesses to scale⚖️ up or scale⚖️ down based on actual needs. There are 5 main areas that AWS Cloud☁️ emphases Scalability ⚖️, Agility, Elasticity🧘‍♂️, Reliability and Security🔐.

  1. Scalability ⚖️ is the ability to resize your resources as necessary. AWS Cloud☁️ provides a scalable computing platform designed for high availability and dependability through tools and solutions.
  • Agility is the ability to increase speed🏃🏻, offer an ease of experimentation and promoting innovation. AWS empowers the user to seamlessly spin up servers in minutes, shut down servers when not needed or allow unused resources to be allocated for other purposes
  • Elasticity🧘‍♂️ is the ability to scale ⚖️ computing resources up or down easily. AWS makes it easy to quickly deploy new applications, scale⚖️ up as the workloads increase and shut down resources that are no longer required
  • Reliability is the ability of a system to recover from infrastructure or service failure. AWS provides reliability by hosting your instances and resources across multiple locations utilizing regions, availability zones and edge locations.
  • Security 🔐 is the ability to retain complete control and ownership over your data and meet regional compliance and data residency requirements. AWS provides highly secure 🔐 data centers, continuous monitoring 🎛 and industry-leading capabilities across facilities, networks, software, and business processes

There are three methods in which you can access AWS resources:

  1. AWS management console which provides a graphical user interface (GUI) to access AWS services
  2. AWS command line interface (CLI) which allows you to control AWS services from the command line
  3. AWS Software Development kits (SDK) enables you to access AWS using a variety of programming languages

Next the course provides us with some brief vignettes covering the AWS core services, AWS Global Infrastructure, and AWS Integrated Services.

AWS Core Services

Elastic Compute Cloud☁️ (EC2) is a web service that provides secure, resizable compute capacity in the Cloud☁️. EC2 instances are “pay as go”. You only pay for the capacity you use, and you have the ability to have different Storage 🗄 requirements.

Key🔑 components of EC2 are:

  • Amazon machine image (AMI) which is an OS image used to build an instance
  • Instance Type refers to hardware capabilities (CPU, Memory)
  • Network – Public and Private IP addresses
  • Storage 🗄 – SSDs, Provisioned IOPs SSD, Magnetic disks
  • Keypairs🔑 (Secure) allow you to connect to instances after they are launched
  • Tags 🏷 provide a friendly name to identify resources in an AWS.

Elastic block store (EBS) provides persistent block level Storage🗄 volumes for your EC2 instances

  • EBS volumes are designed for being durable and available volumes that are automatically replicated across multiple servers running in the availability zones.
  • EBS Volumes must be in the same AZ as the instances they are attached to
  • EBS gives you the ability to create point in time⏰ snapshots of your volumes and allows AWS to create a new volumes from a snapshot at any time⏰.
  • EBS volumes have the ability to increase capacity and change to different types
  • Multiple EBS volumes can be attached to an instance

Simple Storage🗄 Service (S3) is a fully managed Storage🗄 service that provides a simple API for storing and retrieving data. S3 uses buckets🗑 to store data. S3 buckets🗑 are associated with a particular AWS region When you store data in a bucket🗑 it’s redundantly stored across multiple AWS availability zones within a given region

  • Data stored in S3 is serverless. So, you do not need to manage any infrastructure.
  • S3 supports objects as large as several terabytes.
  • S3 also provides low latency access to data over HTTP or HTTPS

AWS Global Infrastructure

The AWS Global Infrastructure consists of RegionsAvailability Zones, and Edge locations providing highly available, fault tolerant, and scalable infrastructures. 

AWS regions are multiple geographical 🌎 areas that host two or more availability zones and are the organizing level for AWS services.

Availability zones are a collection of data centers within a specific region. Each availability zone is physically isolated from one another but connected together through low latency, high throughput, and highly redundant networking. AWS recommends provisioning your data across multiple availability zones.

As of April 2020, AWS spans 70 Availability Zones within 22 Regions around the globe 🌎.

Edge locations are where end users access services located at AWS. They are located in most of the major cities 🏙 around the world 🌎 and are specifically used by Amazon CloudFront🌩 which is a content delivery network (CDN) that distributes content to end user to reduce latency.

Amazon Virtual Private Cloud⛅️ (VPC) is a private network within the AWS Cloud☁️ that adheres to networking best practices as well as regulatory and organizational requirements. VPC is an AWS foundational service that integrates with many of the AWS services. VPC leverages the AWS global infrastructure of regions and availability zones. So, it easily takes advantage of high availability provided by AWS. VPC exists within regions and can span across multiple availability zones. You can create multiple subnets in a VPC. Although fewer is recommended to limit the complexity of the network.

Security🔐 Groups acts as a virtual 🔥 firewall for your virtual servers to control incoming and outgoing traffic🚦. It’s another method to filter traffic🚦 to your instances. It provides you control on what traffic🚦 to allow or to deny. To determine who has access to your instances you would configure a Security🔐 group rule.

AWS CloudFormation🌨 “Infrastructure as Code” allows you to use programming languages, JSON files, or simple text files; to model and provision, in an automated and secure manner, all the resources needed for your applications across all regions and accounts.

“Every AWS service that you learned about is another tool 🛠to build solutions. The more tools you can bring to the table, the more powerful 💪 you become.” -Andy Cummings

AWS Integrated Services

AWS offers a variety of services from A-Z. So, it would impossible to review every service in a six-hour course. Below are some of the services highlighted in the course:

Elastic Load Balancing 🏋🏻‍♀️ distributes incoming application traffic🚦across multiple AWS services like EC2 instances, containers, IP addresses, and Lambda functions automatically. There are 3 kinds of load balancers Network Load Balancer🏋🏻‍♂️, Classic Load Balancer 🏋🏻‍♂️ (ELB) and Application Load Balancer🏋🏻‍♂️(ALB).

  1. Network Load Balancer 🏋🏻‍♀️ is best suited for load balancing of TCP, UDP and TLS traffic🚦 where extreme performance is required.
  2. Classic Load Balancer 🏋🏻‍♀️ provides basic load balancing across multiple Amazon EC2 instances and operates at both the request level and connection level.
  3. Application Load Balancer 🏋🏻‍♀️ offers most of the features provided by the classic load Balancer 🏋🏻‍♀️ and adds some important features and enhancements. Its best suited for load balancing of HTTP and HTTPS traffic🚦

AWS Autoscaling⚖️ monitors 🎛 your applications and automatically adjusts capacity to maintain steady, predictable performance at the lowest possible cost. Autoscaling⚖️ removes the guesswork of how many EC2 instances you need at a point in time⏰ to meet your workload requirements. Their are three core components that are need at launch configuration, “the what to deploy”; An Autoscaling⚖️ group (“the where to deploy”) and an Autoscaling⚖️ policy (“when to deploy”).

  • Dynamic auto scaling⚖️ is a common configuration used with AWS CloudWatch⌚️alarms based on performance information from your EC2 instance or load balancer🏋🏻‍♂️.

Autoscaling⚖️ and Elastic load balancing 🏋🏻‍♀️ automatically scale⚖️ up or down based on demands backed by Amazon’s massive infrastructure you have access to compute and Storage 🗄 resources whenever you need them

Amazon route 53👮is a global, highly available DNS service that allows you to easily register and resolve DNS names, providing a managed service of reliable and highly scalable ⚖️ way to route 👮‍♀️ end users to Internet applications. Route 53 offers multiple ways to route👮‍♀️ your traffic🚦 enabling you to optimize latency for your applications and users.

Amazon Relational Database Service (RDS🛢) is a database as a service (DBaaS) that makes provisioning, operating and scaling⚖️ either up or out seamless. In addition, RDS🛢makes other time-consuming administrative tasks such as patching, and backups a thing of the past. Amazon RDS🛢provides high availability and durability through the use of Multi-AZ deployments. It also lets you run your database instances an Amazon VPC, which provides you the control and security🔐.

AWS Lambda is a compute service that lets you run code without provisioning or managing servers. AWS Lambda executes your code only when needed and scales⚖️ automatically to thousands of requests per second.

AWS Elastic Beanstalk🌱 is an easy-to-use service for deploying and scaling web applications and services developed with Java☕️ , NET, PHP, Node.js, Python🐍, Ruby💎, Go, and Docker🐳 on familiar servers such as Apache, Nginx, Passenger, and IIS. Elastic Beanstalk🌱 employs Auto Scaling⚖️ and Elastic Load Balancing🏋🏻‍♂️ to scale⚖️ and balance workloads. It provides tools🛠 in the form of Amazon CloudWatch⌚️ to monitor 🎛 the health❤️ of deployed applications. It also provides capacity provisioning due to its reliance on AWS S3 and EC2.

Amazon Simple Notification Service (SNS) 📨 is a highly available, durable, secure🔐, fully managed pub/sub messaging service like Google’s pub/sub that enables you to decouple microservices, distributed systems, and serverless applications. Additionally, SNS📨 can be used to fan out notifications to end users using mobile push, SMS, and email✉️.

Amazon CloudWatch⌚️ is a monitoring 🎛 service that allows you to monitor 🎛 AWS resources and the applications you run 🏃🏻 on them in real time. Amazon CloudWatch⌚️ features include collecting and tracking metrics like CPU utilization, data transfer, as well as disk I/O and utilization. Some of the components that make up Amazon CloudWatch⌚️ include metrics, alarms, events, logs and dashboards

Amazon CloudFront🌩 uses a global 🌎network of more than 80 locations and more than 10 regional edge caches for content delivery (CDN). It’s integrated with the AWS services such as AWS web application🔥 firewall, certificate manager, route 53, and S3 as well as other AWS services.

AWS CloudFormation🌨 is a fully managed service which acts as an engine 🚂 to automate the provisioning of AWS resources. CloudFormation🌨 reads template files which specify the resources to deploy. Provision resources are known as the stack. Stacks 📚 can be created updated or deleted through CloudFormation🌨.

AWS Architecture

When one refers to AWS Architecture one need to refer to no further than to the AWS Well-Architected Framework. The AWS Well-Architected Framework originally began as a single whitepaper but expanded into more of a doctrine focused on Key🔑 concepts, design principles, and architectural best practices for designing secure, high-performing, resilient, and efficient infrastructure and running 🏃🏻 workloads in the AWS Cloud☁️ .

The AWS Well-Architected Framework is based on five pillars; operational excellencesecurity🔐 , reliabilityperformance efficiency, and cost optimization.

  1. Operational excellence focuses on running 🏃🏻 and monitoring 🎛 systems and continually improving processes and procedures.
  2. Security🔐 centers on protecting information and systems.
  3. Reliability highlights that workload performs consistently as intended and could quickly recover from failure
  4. Performance efficiency concentrates on efficiently using computing resources.
  5. Cost optimization emphasis on cost avoidance.

Reference Architecture – Fault Tolerance and High Availability

Both Fault Tolerance and High Availability are cornerstones of infrastructure design strategies to keep critical applications and data up and running 🏃🏻

Fault Tolerance refers to the ability of a system (computer, network, Cloud☁️ cluster, etc.) to continue operating without interruption when one or more of its components fail.

High availability refers to systems that are durable and likely to operate continuously functioning and accessible and that downtime is minimized as much as possible, without the need for human intervention.

AWS provides services and infrastructure to build reliable, fault-tolerant, and highly available systems in the Cloud☁️.

Some AWS services that can assist in providing high availability:

  • Elastic load balancers🏋🏻‍♀️
  • Elastic IP addresses
  • Route 53 👮‍♀️
  • Auto scaling⚖️
  • CloudWatch⌚️

Some AWS services that provide fault tolerant tools are:

  • SQS
  • S3 🗄
  • RDS🛢

Amazon Web Services offers Cloud☁️ web 🕸 hosting solutions that provide businesses, non-profits, and governmental organizations with low-cost ways to deliver their websites and web 🕸 applications.


When it comes to security🔐 AWS doesn’t take this lightly. So much so that when you are a newbie to AWS it could be quite challenging just to connect to your Cloud☁️ environment. AWS global infrastructure is built with the highest standards to ensure privacy🤫 and data security🔐. AWS infrastructure has strong 💪 safeguards in place to protect customers privacy 🤫. AWS continuously improves and innovates security🔐 incorporating customer feedback and changing requirements. AWS provides security🔐 specific tools🛠 and features across network security🔐, configuration management, access control, and data security🔐. AWS provides monitoring 🎛 and logging tools🛠 to provide full visibility👀 into what is happening in your environment. AWS provides several security🔐 capabilities and services like built-in firewalls🔥 to increase privacy🤫 and control network access. In addition, AWS offers Encryption of data both in transit and data at rest in the Cloud☁️. AWS offers you capabilities to define, enforce, and manage user👤 access policies across AWS services.

The shared👫responsibility model

AWS believes Security🔐 and Compliance is a shared👫responsibility between AWS and the customer. The shared👫responsibility model alleviates the operational burden as AWS operates, manages and controls the components from the host operating system and virtualization layer down to the physical security🔐 of the facilities in which the service operates. The customer assumes responsibility and management of the guest operating system (including updates and security🔐 patches), other associated application software as well as the configuration of the AWS provided security🔐 group🔥 firewalls.

Security🔐 “of” the Cloud☁️ vs Security🔐 “in” the Cloud☁️

  • “Security🔐 of the Cloud☁️ – AWS is responsible for protecting the infrastructure that runs all of the services offered in the AWS Cloud☁️.
  • “Security🔐 in the Cloud☁️ – Customer responsibility will be determined by the AWS Cloud☁️ services that a customer selects.

Inherited Controls are controls that the customer fully inherits from AWS.

Shared👫Controls are controls which apply to both the infrastructure layer and the customer layers, but in completely separate contexts or perspectives. Examples include:

  • Patch Management – AWS is responsible for patching and fixing flaws within the infrastructure, but customers are responsible for patching their guest OS and applications.
  • Configuration Management – AWS maintains the configuration of its infrastructure devices, but a customer is responsible for configuring their own guest operating systems, databases, and applications.
  • Awareness & Training – AWS trains AWS employees, but a customer must train their own employees.

Customer Specific – Controls which are solely the responsibility of the customer based on the application they are deploying within AWS services. Examples include:

AWS Cloud☁️ Security🔐 Infrastructure and services

AWS Identity and Access Management (IAM) is one of the core secure services (at no additional charge) to enforce security🔐 across all AWS Service offerings. IAM provides Authentication, Authorization, User Management and Central User Repository. In IAM, you can create and manage users, groups, and roles to either allow or deny access to specific AWS resources.

  • Users👤 are permanent named Operator (can be either human or machine)
  • Groups 👥 are collections of users (can be either human or machine)
  • Roles – are an authentication method. The Key🔑 part is the credentials with the role are temporary.

As for permissions, it is enforced by a separate object known as a policy document 📜.

A Policy document📜 is JSON document📜 that attaches either directly to a user👤 or a group👥 or it can be attached directly to a role.

AWS CloudTrail 🌨 is a service that enables governance, compliance, operational auditing, and risk auditing CloudTrail🌨 records all successful or declined authentication and authorization.

Amazon Inspector 🕵️‍♂️ is an automated security🔐 and vulnerability assessment service that assesses applications for exposure, vulnerabilities, and deviations from best practices. Amazon Inspector produces a detailed list of security🔐 findings prioritized by level of severity in the following areas:

  • Identify Application Security🔐 Issues
  • Integrate Security🔐 into DevOps
  • Increase Development Agility
  • Leverage AWS Security🔐 Expertise
  • Streamline Security🔐 Compliance
  • Enforce Security🔐 Standards

AWS Shield🛡 is a managed Distributed Denial of Service (DDoS) protection service. There are two tiers of AWS Shield🛡 Standard and Advanced.

  • AWS Shield🛡Standard defends against most common, frequently occurring network and transport layer DDoS attacks that target your web site or applications.
  • AWS Shield🛡Advanced provides additional detection against large and sophisticated DDoS attacks, near real-time visibility into attacks, and integration with AWS web 🕸application 🔥 firewall (WAF).

Pricing and Support

AWS offers a wide range of Cloud☁️ computing services. For each service, you pay for exactly the amount of resources you actually use.

  • Pay as you go.
  • Payless then when you reserve.
  • Pay even less per unit by using more
  • Pay even less as your AWS Cloud☁️ grows

There are three fundamental characteristics you pay for with AWS:

  1. Compute 💻
  2. Storage 🗄
  3. Data transfer out ➡️

Although you are charged for data transfer out, there is no charge for inbound data transfer or for data transfer between other services with the same region.

AWS Trusted Advisor is an online tool🔧 that optimizes your AWS infrastructure, increase reliabilitysecurity🔐 and performance, reduce your overall costs, and monitoring 🎛. AWS Trusted Advisor enforces AWS best practices in five categories:

  1. Cost optimization
  2. Performance
  3. Security🔐
  4. Fault tolerance
  5. Service limits

AWS offers 4 levels of Support

  1. Basic support plan (Included with all AWS Services)
  2. Developer support plan
  3. Business support plan
  4. Enterprise support plan

Obviously, there was a lot to digest😋 but now we have a great overall understanding of the AWS Cloud☁️ concepts, some of the AWS services, security🔐, architecture, pricing 💶, and support and feel confident to continue our journey in the AWS Cloud☁️. 😊

“This is the end, Beautiful friend.. This is the end, My only friend, the end”?

Below are some areas I am considering for my travels next week:

  • Neo4J and Cypher 
  • More with Google Cloud Path
  • ONTAP Cluster Fundamentals
  • Data Visualization Tools (i.e. Looker)
  • Additional ETL Solutions (Stitch, FiveTran) 
  • Process and Transforming data/Explore data through ML (i.e. Databricks)

Thanks –


Week of July 24th

“And you may ask yourself 🤔, well, how did I get here?” 😲

Happy Opening⚾️ Day!

Last week, we hit a milestone of sorts with our 20th submission🎖since we started our journey way back in March.😊 To commemorate the occasion, we made a return back to AWS by gleefully 😊 exploring their data ecosystem. Of course, trying to cover all the data services that are made available in AWS in such a short duration 🕰 would be a daunting task.

So last week, we concentrated our travels to three of their main offerings in the Relational Database, NoSQL, and Data warehouse realms. This being of course RDS🛢, DynamoDB🧨, and Redshift🎆. We felt rather content 🤗 and enlighten💡with AWS’s Relational Database and Data warehouse offerings, but we were still feeling a little less satisfied 🤔 with NoSQL as we really just scratched the surface on what AWS had to offer.

To recap, we had explored 🔦 DynamoDB🧨 AWS’s multi-model NoSQL service which offers support for a key-value🔑and their propriety document📜 database. But we were still curious to learn more about a Document📜 database that offers MongoDB🍃support as well in AWS. In addition, an answer to the hottest🔥 trend in “Not Just SQL Solutions”, Graph📈 Database.

Well of course being the Cloud☁️ Provider that offers every Cloud☁️native service from A-Z, AWS delivered with many great options. So we began our voyage heading straight over to DocumentDB📜. AWS’s fully managed database service with MongoDB🍃compatibility. As with all AWS services, Document DB📜 was designed from the ground up to give the most optimal performance, scalability⚖️, and availability. DocumentDB📜 like the Cosmo DB🪐 MongoDB🍃API makes it easy to set up, operate, and scale MongoDB-compatible databases. In other words, no code changes are required, and all the same drivers can be utilized by existing legacy MongoDB🍃applications.

In addition, Document DB📜 solves the friction and complications of when an application tries to map JSON to a relational model. DocumentDB📜 solves this problem by making JSON documents a first-class object of the database. Data is stored in the form of documents📜. These documents📜 are stored into collections. Each document📜can have a unique combination and nesting of fields or key-value🔑 pairs. Thus, making querying the database faster⚡️, indexing more flexible, and repetitions easier.

Similar to other AWS Data offerings, the core unit that makes up DocumentDB📜 is the cluster. A cluster consists of one or more instances and cluster storage volume that manages the data for those instances. All writes📝 are done through the primary instance. All instances (primary and replicas) support read 📖 operations.  The cluster’s data stores six copies of your data across three different Availability Zones. AWS easily allows you to create or modify clusters. When you modify a cluster, AWS is really just spinning up a new cluster behind the curtains and then migrates the data taking what is an otherwise complex task and making it seamless.

As prerequisite, you first must create and configure a virtual private cloud☁️ (VPC) to place DocumentDB📜 in. You can leverage an existing one or you can create a dedicated one just for DocumentDB📜. Next, you need to configure security🔒 groups for your VPC. Security🔒 groups are what controls who has access to your Document📜 Databases . As for credentials🔐 and entitlements in DocumentDB📜, it is managed through AWS Identity and Access Management (IAM).By default, the cluster Document DB📜accepts secure connections using Transport Layer Security (TLS). So, all traffic in transit is encrypted and Amazon DocumentDB📜 uses the 256-bit Advanced Encryption Standard (AES-256) to encrypt your data or allows you to encrypt your clusters using keys🔑 you manage through AWS Key🔑Management Service (AWS KMS) so data at rest is always encrypted. 

“Such wonderful things surround you…What more is you lookin’ for?”

Lately, we have been really digging Graph📈 Databases. We had our first visit with Graph📈 Databases when we were exposed to the Graph📈 API through Cosmos DB🪐 earlier this month and then furthered our curiosity through Neo4J. Well, now armed with a plethora of knowledge in the Graph📈 Database space we wanted to see what AWS had to offer and once again they did not disappoint.😊

First let me start by writing, It’s a little difficult to compare AWS Neptune🔱 to Neo4J although Nous Populi from Leapgraph does an admirable job. Obviously, both are graph📈 databases but architecturally there some major differences in their graph storage model and query languages. Neo4J uses Cypher and Neptune🔱 uses Apache TinkerPop or Gremlin👹 same as Cosmos DB🪐 as well as SPARQL. Where Neptune🔱 really shines☀️ is that it’s not just another graph database but a great service offering within the AWS portfolio. So, it leverages all the great bells🔔 and whistles like fast⚡️ performance, scalability⚖️, High availability and durability.  As well as being a fully managed service that we have come accustomed too like handling hardware provisioning, software patching, backup, recovery, failure detection, and repair. Neptune🔱  is an optimized for storing billions of relationships and querying the graph with milliseconds latency.

Neptune🔱 uses database instances. The primary database instance supports both read📖 and write📝 operations and performs all the data modifications to the cluster. Neptune🔱  also uses replicas which connects to the same cloud-native☁️ storage service as the primary database instance but only supports read-only operations. There can be up to 15 of these replicas across multiple AZs. In addition, Neptune🔱  supports encryption at rest.

As prerequisite, you first must create and configure a virtual private cloud☁️ (VPC) to place Neptune🔱  in. You can leverage an existing one or you can create a dedicated one just for Neptune🔱  Next, you need to configure security🔒 groups for your VPC. Security🔒 groups are what controls who has access to your Neptune🔱. As for credentials🔐 and entitlements in Neptune🔱  is managed through AWS Identity and Access Management (IAM). Your data at rest in the Neptune🔱  is encrypted using the industry standard AES-256-bit encryption algorithm on the server that hosts your Neptune🔱  instance.  Keys🔑 can also be used, which are managed through AWS Key🔑 Management Service (AWS KMS).

“Life moves pretty fast⚡️. If you don’t stop 🛑 and look 👀 around once in a while, you could miss it.”

So now feeling pretty good 😊 about NoSQL on AWS, where do we go now? 

Well, as mentioned we have been learning so much over the last 5 months it could be very easy to forget somethings especially with limited storage capacity. So we decided to take a pause for the rest of the week and go back and review all that we have learned by re-reading all our previous blog posts as well as engaging in some Google Data Engineered solution Quests🛡to help reinforce our previous learnings

Currently, the fine folks at qwiklabs.com are offering anyone who wants to learn Google Cloud ☁️ skills an opportunity for unlimited access for 30 days.  So with an offer too good to be true as well as an opportunity to add some flare to our linked in profile and who doesn’t like flare?  We dove right in head first!

“Where do we go? Oh, where do we go now? Now, now, now, now, now, now, now”

Below are some topics I am considering for my travels next week:

  • Neo4J and Cypher 
  • More with Google Cloud Path
  • ONTAP Cluster Fundamentals
  • Data Visualization Tools (i.e. Looker)
  • Additional ETL Solutions (Stitch, FiveTran) 
  • Process and Transforming data/Explore data through ML (i.e. Databricks)



Week of July 17th

“Any timeof year… You can find it here”

Happy World🌎 Emoji 😊😜 Day! 

The Last few weeks we have been enjoying our time in Microsoft’s Cloud☁️ Data Ecosystem and It was just last month that we found ourselves hanging out with the GCP☁️ gang and their awesome Data offerings. All seemed well and good😊 except that we had been missing out on excursion to the one cloud☁️ provider where it all began literally and figuratively.

Back when we first began our journey on a cold 🥶 and rainy☔️ day in March just as Covid-19🦠 occupied Wall Street 🏦 and the rest of the planet 🌎 we started at the one place that disrupted how infrastructure and operations would be implemented and supported going forward.

That’s right Amazon Web Services or more endearingly known to humanity as AWS. AWS was released just two decades ago by the its parent company that offers everything from A to Z.

AWS like its parent company has a similar mantra in the cloud ☁️ computing world as they offer 200+ Cloud☁️ Services. So how the heck with so some many months passed that we haven’t been back since? The question is quite perplexing? But like they say “all Clouds☁️☁️ lead to AWS. So, here we are back in the AWS groove 🎶 and eager 😆 to explore 🔦the wondrous world🌎 of AWS Data solutions. Guiding us through this vast journey would be Richard Seroter (who ironically recently joined the team at Google). In 2015, Richard authored an amazing Pluralsight course covering Amazon RDS🛢, Amazon DynamoDB 🧨 and Amazon’s Redshift 🎆. It was like getting 3 courses for the price of 1! 😊

Although the course was several years old, for the most part it still out lasted the test of time ⏰  by providing a strong foundational knowledge for Amazon’s relational, NoSQL, and Data warehousing solutions. But unfortunately in technology years, it’s kind of like dog🐕  years. So obviously, there have been many innovations to all three of these incredible solutions including UI enhancements, architectural improvements and additional features to these great AWS offerings making them even more awesome!

So for a grand finale to our marvelous week of learning and to help us fill in the gaps on some of these major enhancements as well as offering some additional insights were the team from AWS Training and certification which includes the talented fashionista Michelle Metzger, the always effervescent and insightful Blaine Sundrud and on demos the man with a quirky naming convention for database objects the always witty Stephen Cole 

Back in our Amazon Web Services Databases in Depth course and in effort to make our journey that more captivating, Richard provided us with a nifty mobile sports 🏀 ⚾️ 🏈  app solution written in Node.js which leverages the Amazon data offerings covered in the course as components for an end to end solution. As the solution, was written several years back it did require some updating on some deprecated libraries📚 and some code changes in order to make the solution work which made our learning that more fulfilling. So, after a great introduction from Richard where he compares and contrasts RDS🛢, DynamoDB🧨, and Redshift🎆, we began our journey with Amazon’s Relational Database Service (RDS🛢). RDS🛢 is a database as a service (DBaaS) that makes provisioning, operating and scaling⚖️  either up or out seamless. In addition, RDS🛢 makes other time-consuming administrative tasks such as patching, and backups a thing of the past. Amazon RDS🛢 provides high availability and durability through the use of Multi-AZ deployments. In other words, AWS creates multiple instances of the databases in different Availability Zones making recovery from infrastructure failure automatic and almost transparent to the application. Of course like with all AWS offerings there always a heavy emphasis on security🔐 which it’s certainly reassuring when you putting your mission critical data in their hands 🤲 but it could also be a bit challenging at first to get things up and running when you are simply just trying connect to from your home computer 💻  back to the AWS infrastructure

As prerequisite, you first must create and configure a virtual private cloud☁️ (VPC) to put to your RDS🛢instance(s) in. You can leverage an existing one or you can create a dedicated one for RDS🛢instance(s).

It is required that your VPC have at least two subnets in order to support the Availability Zones for high availability. If direct internet access is needed that you will need to add an internet gateway to your VPC.  

Next, you need to configure security🔒 groups for your VPC. Security🔒 groups are what controls who has access to the RDS🛢. RDS🛢 leverages three types of security groups (database, VPC, and EC2). As for credentials🔐 and entitlements in RDS🛢, it is managed through AWS Identity and Access Management (IAM). At the time of the release of Richard’s courseAmazon Aurora was new in the game and was not covered in depth in the course. In addition, at the same time only MySQL, Postgres, Oracle, MS SQL Server and the aforementioned Aurora were only supported at this time. AWS has since added support for MariaDB to their relational database service portfolio.  

Fortunately, our friends from the AWS Training and Certification group gave us the insights that we would require on Amazon’s innovation behind their relational database built for the cloud☁️ better known as Aurora.

So, with six familiar database engines (licensing costs apply) to choose from you have quite a few options. Another key🔑 decision is to determines the resources you want your database to have. RDS🛢offers multiple options for optimized for memory, performance, or I/O.

I would be remised if we didn’t briefly touch on Amazon’s Aurora. As mentioned, it’s one of Amazon’s 6 database options with RDS🛢. Aurora is fully managed by RDS🛢. So it leverages the same great infrastructure and has all the same bells 🔔 and whistles. Aurora comes in two different flavors🍦 MySQL and PostgreSQL. Although, I didn’t benchmark Aurora performance in my evaluation AWS claims that Aurora is 5x faster than the standard MySQL databases. However, it’s probably more like 3x faster. But the bottom line it is that it is faster and more cost-effectiveness for MySQL or PostgreSQL databases that require optimal performances, availability, and reliability. The secret sauce behind Aurora is that automatically maintains 6 copies of your data (which can be increased up to 15 replicas) that is spanned across 3 AZs making data highly available and ultimately providing laser⚡️ fast performance for your database instances.

Please note: There is an option that allows a single Aurora database to span multiple AWS Regions 🌎 for an additional cost

In addition, Aurora uses an innovative and significantly faster log structured distributed storage layer than other RDS🛢offerings.

“Welcome my son, welcome to the machine🤖

Next on our plate 🍽 was to take a deep dive into Amazon’s fast and flexible NoSQL database service a.k.a DynamoDB🧨.. DynamoDB🧨 like Cosmo DB🪐 is a multi-model NoSQL solution.

DynamoDB🧨 combines the best of those two ACID compliant non-relational databases in a key-value🔑 and document database. It is a proprietary engine, so can’t just take your MongoDB🍃 database and convert it to DynamoDB🧨. But don’t worry if you looking to move your MongoDB🍃 works loads to Amazon, AWS offers Amazon DocumentDB (with MongoDB compatibility) but that’s for a later discussion 😉

As for DynamoDB🧨, it delivers a blazing⚡️ single-digit millisecond guaranteed performance at any scale⚖️. It’s a fully managed, multi-Region, multi-master database with built-in security🔐, backup and restore options, and in-memory caching for internet-scale applications. DynamoDB🧨 automatically scales⚖️  up and down to adjust for the capacity and maintain performance of your systems. Availability and fault tolerance are built in, eliminating the need to architect your applications for these capabilities. An important concept to grasp while working with DynamoDB🧨  is that the databases are comprised of tables, items, and attributes. Again, there has been some major architectural design changes to DynamoDB🧨 since Richard’s course was released. Not to go into too many details as its kind or irrelevant but at time⏰ the course was released DynamoDB🧨 used to offer the option to either use a Hash Primary Key🔑 or Hash and Range Primary Key🔑 to organize or partition data and of course as you would imagine choosing the right combination was rather confusing. Intuitively, AWS scrapped this architectural design and the good folks at the AWS Training and Certification group were so kind to offer clarity here as well 😊

Today, DynamoDB🧨 uses partition keys🔑  to find each item in the database similar to Cosmo DB🪐. Data is distributed on physical storage nodes. DynamoDB🧨 uses the partition key🔑  to determine which of the nodes the item is located on. It’s very important to choice the right partition key 🔑 to avoid the dreaded hot 🔥partitions. Again “As rule of thumb 👍, an ideal Partition key🔑 should have a wide range of values, so your data is evenly spread across logical partitions. Also in DynamoDB🧨 items can have an optional sort key🔑 to store related attributes in a sorted order.

One major difference to Cosmos DB🪐 is that DynamoDB🧨 utilizes a primary key🔑 on each table. If there is no sort key🔑, the primary and partition keys🔑 are the same. If there is a sort key🔑, the primary key🔑 is a combination of the partition and sort key 🔑 which is called a composite primary key🔑 .

DynamoDB🧨 allows for secondary indexes for faster searches. It supports two types indexes local (up to 5 per table) and global (up to 20 per table). These indexes can help improve the application’s ability to access data quickly and efficiently.

Differences Between Global and Local Secondary Indexes

Hash or hash and range keyHash and range key
No size limitFor each key, 10GB max
Add during table create, or laterAdd during table create
Query all partitions in a tableQuery single partition
Eventually consistent queriesEventually/strong consistent queries
Dedicated throughput unitsUser table throughput units
Only access projected itemsAccess all attributes from table

DynamoDB🧨 like Cosmo DB🪐 offers multiple Data Consistency Levels. DynamoDB🧨 Offers both Eventually and Strongly consistent Reads but like I said previously “it’s like life itself there is always tradeoffs. So, depending on your application needs. You will need to determine what’s the most important need for your application latency or availability.”

As a prerequisite, you first must create and configure a virtual private cloud☁️ (VPC) to put DynamoDB🧨  in. You can leverage an existing one or you can create a dedicated one for DynamoDB🧨  Next, you need to configure security🔒 groups for your VPC. Security🔒  groups are what controls who has access to DynamoDB🧨. As for authentication🔐  and permission to access a table, it is managed through Identity and Access Management (IAM). DynamoDB🧨 provides end-to-end enterprise-grade encryption for data that is both in transit and at rest. All DynamoDB tables have encryption at rest enabled by default. This provides enhanced security by encrypting all your data using encryption keys🔑stored in the AWS Key🔑Management System, or AWS KMS.

“Quicker than a ray of light I’m flying”

Making our final destination for this week’s explorations would be to Amazon’s fully managed, fast, scalable data warehouse known as Redshift🎆 . A “Red Shift🎆” is when a wavelength of the light is stretched, so the light is seen as ‘shifted’ towards the red part of the spectrum but according to anonymous sources “RedShift🎆 was apparently named very deliberately as a nod to Oracle’ trademark red branding, and Salesforce is calling its effort to move onto a new database “Sayonara,”” Be that what it may, this would be the third Data Warehouse cloud☁️ solution we would have the pleasure of be aquatinted with. 😊

AWS claims Redshift🎆 delivers 10x faster performance than other data warehouses. We didn’t have a chance to benchmark RedShift’s 🎆  performance but based some TPC tests vs some of their top competitors there might be some discrepancies with these claims but either case it’s still pretty darn on fast.

Redshift🎆 uses Massively parallel processing (MPP) and columnar storage architecture. The core unit that makes up Redshift🎆  is the cluster. The Cluster is made up of one or more compute nodes. There is a single leader node and several compute nodes. Clients access to Redshift🎆 is via a SQL endpoint on the leader node. The client sends a query to the endpoint. The leader node creates jobs based on the query logic and sends them in parallel to the compute nodes. The compute nodes contain the actual data the queries need. The compute nodes find the required data, perform operations, and return results to the leader node. The leader node then aggregates the results from all of the compute nodes and sends a report back to the client.

The compute nodes themselves are individual servers, they have their own dedicated memory, CPU, and attached disks. An individual compute node is actually split up into slices🍕, one slice🍕 for every core of that node’s processor. Each slice🍕 is given a piece of memory, disk, and so forth, where it processes whatever part of the workflow that’s been assigned to it by the leader node.

The way the columnar database storage works data is stored by columns rather than by rows. This allows for fast retrieval of columns of data. An additional advantage is that, since each block holds the same type of data, block data can use a compression scheme selected specifically for the column data type, further reducing disk space and I/O. Again, there have been several architectural changes to RedShift🎆 as well since Richard’s course was released.

In the past you needed to pick a distribution style. Today, you still have the option to choose a distribution style but if don’t specify a distribution style, Redshift🎆 will uses AUTO distribution making it little easier not to make the wrong choice 😊. Another recent innovation to Redshift🎆 that didn’t exist when the Richard’s course was released is the ability to build a unified data platform. Amazon Redshift🎆 Spectrum allows you to run queries across your data warehouse and Amazon S3 buckets simultaneously. Allowing you to save time ⏰  and money💰 as you don’t need to load all your data into the data warehouse.

As prerequisite, you first must create and configure a virtual private cloud☁️ (VPC) to place Redshift🎆 in. You can leverage an existing one or you can create a dedicated one just for Redshift🎆. In addition, you will need to create an Amazon Simple Storage Service (S3) bucket and S3 Endpoint to be used with Redshift🎆. Next, you need to configure security🔒 groups for your VPC. Security🔒 groups are what controls who has access to your data warehouse. As for credentials 🔐  and entitlements in Redshift🎆, it is managed through AWS Identity and Access Management (IAM).

One last point worth mentioning is that AWS Cloud ☁️  Watch ⌚️ is included with all the tremendous Cloud☁️ Services offered by AWS. So you get great monitoring 📉right of the box! 😊 We enjoyed 😊 our time⏰ this week in AWS exploring 🔦 some of their data offerings, but we merely just scratched the service.

So much to do, so much to see… So, what’s wrong with taking the backstreets? You’ll never know if you don’t go. You’ll never shine if you don’t glow

Below are some topics I am considering for my travels next week:

  • More with AWS Data Solutions
  • Neo4J and Cypher 
  • More with Google Cloud Path
  • ONTAP Cluster Fundamentals
  • Data Visualization Tools (i.e. Looker)
  • Additional ETL Solutions (Stitch, FiveTran) 
  • Process and Transforming data/Explore data through ML (i.e. Databricks)



Week of May 22nd

And you know that notion just cross my mind…

Happy Bitcoin PizzaEmoji Day!

All aboard! This week our travels would take us on the railways far and high but before, we can hop on the knowledge express we had some unfinished business to attended too.

“Oh, I get by with a little help from my friends”

If you have been following my weekly submissions for the last few weeks I listed as future action item “create/configure a solution that leverages Python to stream market data and insert it into a relational database.

Well last week, I found just the perfect solution. A true master piece by Data Scientist/Physicist extraordinaire AJ Pryor, Ph.D. AJ had created a brilliant multithreaded work of art that continuously queries market data from IEX  and then writes it to a PostgreSQL database. In addition, he built a data visualization front-end that leverages Pandas and Bokeh so the application can run interactively through a standard web browser. It was like a dream come true! Except that the code was written like 3 years ago and referenced a deprecated API from IEX.

Ok, no problem. We will just simply modify AJ’s “Mona Lisa” to reference the new IEX API and off we will go.  Well, what seemed like was a dream turned into a virtual nightmare. I spent most of last week spinning my wheels trying to get the code to work but to no avail. I even reached out to the community on Stack overflow but all I received was crickets..

As I was ready to cut my loses, but I reached out to a longtime good friend who happens to be all-star programmer and a fellow NY Yankees baseball enthusiast. Python wasn’t his specialty (he is really an amazing Java programmer) but he offered to take a look at the code when he had some time… So we set up a zoom call this past Sunday and I let his wizardry take over… After about hour or so he was in a state of flow and had a good pulse of what our maestro AJ’s work was all about. After a few modifications my good chum had the code working and humming along. I ran into a few hiccups along the way with the brokeh code, but my confidant just referred me to run some simpler syntax and then abracadabra… this masterpiece was now working on the Mac!Emoji As the new week started, I was still basking in the radiance of this great coding victory. So, I decided to be a bit ambitious and move this gem Emoji to the cloud Emoji which would be like the crème de la crème of our learnings thus far. Cloud, Python/Pandas, Streaming market data, and Postgres all wrapped up in one! Complete and utter awesomeness! 

Now the question was for which cloud platform to go with? We were well versed in the compute area in all 3 of the major providers as a result of our learnings.

So with a flip of the coin ,we decided to go with Microsoft Azure. That and we had some free credits still available. Emoji

With sugar plum fairies dancing Emoji in our head, we spun up our Ubuntu Image and we followed along the well documented steps on AJ’s Github project 

Now, we were now cooking Emoji with gasoline Emoji! We cloned AJ’s Github repo, modified the code with our new changes, and executed the syntax and just as we were ready to declare victory… Stack overflow Error! Emoji Oh, the pain.

Fortunately I didn’t waste any time, I went right back to my ace Emoji in the hole but with some trepidation that I wasn’t being too much of irritant.

I explained my perplexing predicament and without hesitation my Fidus Achates offered some great trouble shooting tips and quite expeditiously we had the root cause pinpointed. For some peculiar reason, the formatting of URL that worked like a charm on the MacEmoji was a dyspepsia on Ubuntu on Azure. It was certainly a mystery but one that can only be solved by simply rewriting the code.

So once again, my comrade in arms helped me through another quagmire. So, without further ado, may I introduce to you the one and only…


We’ll hit the stops along the way We only stop for the best

After feeling victorious after my own personal Battle of Carthage and with our little streaming market data saga out of our periphery it was to time to hit the rails… Emoji

Our first stop was messaging services which is all the rage now a days.  There are so many choices with data messaging services out there.. So where to start with? We went with Google’s Pub/Sub which turned out to be a marvelous choice! To get enlightened with this solution, we went to Pluralsight where we found excellent course on Architecting Stream Processing Solutions Using Google Cloud Pub/Sub by Vitthal Srinivasan 

Vitthal was a great conductor who navigated us through an excellent overview of Google’s impressive solution, uses cases, and even touched on a rather complex pricing structure in our first lesson. He then takes us deep into the weeds showing us how to create Topics, Publishers, and Subscribers. He goes on further by showing us how to leverage some other tremendous offerings in GCP like Cloud Functions, API & Services, and Storage. 

Before this amazing course my only exposure was just limited to GCP’s Compute Engine so this was eye opening experience to see the great power that GCP had to offer! To round out the course, he showed us how to use GCP Pub/Sub with some client Libraries which was excellent tutorial on how to use Python with this awesome product. There was even two modules on how to integrate Google Hangout Chatbot with Pub/Sub but that required you to be a G Suite User. (There was free trial but skipped the set up and just watched the videos) Details on the work I did on Pub/Sub can be found at

“I think of all the education that I missed… But then my homework was never quite like this”

For Bonus this week, I spent enormous amount of time brushing up my 8th grade Math and Science Curriculum 

  1. Liner Regression
  2. Epigenetics
  3. Protein Synthesis

Below are some topics I am considering for my Journey next week:

  • Vagrant with Docker
  • Continuing with Data Pipelines
  • Google Cloud Data Fusion (ETL/ELT)
  • More on Machine Learning
  • ONTAP Cluster Fundamentals
  • Google Big Query
  • Data Visualization Tools (i.e. Looker)
  • ETL Solutions (Stitch, FiveTran) 
  • Process and Transforming data/Explore data through ML (i.e. Databricks) .
  • Getting Started with Kubernetes with an old buddy (Nigel)

Stay safe and Be well –


Week of April 10th

“…When the Promise of a brave new world unfurled beneath a clear blue Sky”

“Forests, lakes, and rivers, clouds and winds, stars and flowers, stupendous glaciers and crystal snowflakes – every form of animate or inanimate existence, leaves its impress upon the soul of man.” — Orison Swett Marden

My journey for this week turned out to be a sort of potpourri of various technologies and solutions thanks to the wonderful folks at MSFT.  After some heavy soul searching over the previous weekend, I decided that my time would be best spent this week on recreating the SQL Server 2016 with Always On environment (previously created several weeks back on AWS EC2) but in the MS Azure Cloud platform.  The goal would be to better understand Azure and how it works. In addition, I would be able to compare and contrast both AWS EC2 vs. Azure VMs and be able to list both the pros and cons of these cloud providers. 

But before I could get my head into the clouds I was still lingering around in the bamboo forests. This past weekend, I was presented with an interesting scenario to stream market data to pandas from the investors exchange (Thanks to my friend) . So after consulting with Mr. Google, I was pleasantly surprised to find that IEX offered an API that allows you to connect to there service and stream messages directly to Python and use Pandas for data visualization and analysis. Of course being the cheapskate that I am I signed up for a free account and off I went. 

So I started tickling the keys, I produced a newly minted IEX Py script. After some brief testing, I started receiving an obscure error? Of course there was no documented solution on how to the address such an error.. 

So after some fruitless nonstop piping of several modules, I was still getting the same error. 🙁 After a moment of clarity of I deduced there was probably limitation on messages you can stream from the free IEX account..

So I took shot in the dark and decided to register for another account (under a different email address) this way I would receive a new token and give that a try 

… And Oh là là!  My script started working again! 🙂 Of course as I continued to add more functionality and test my script I ran back into the same error but this time I knew exactly how to resolve it. 

So I registered for a third account (to yet again generate a new token ). FortunateIy, I completed my weekend project. See attachments Plot2.png and Plot3.png for pretty graphs

Now that I could see the forest through the trees and it was off to the cloud! I anticipated that it would take me a full week to explore Azure VMs but it actually only took a fews to wrap my head around it..

So this left me chance to pivot again and this time to a Data Warehouse/ Data Lake solution built for the Cloud. Turning the forecast for the rest of the week to Snow.

Here is a summary of what I did this week:


  • Developed Pandas/Python Script in conjunction with iexfinance & matplotlib modules to build graphs to show historical price for MSFT for 2020 and comparison of MSFT vs INTC for Jan 2nd – April 3rd 2020

Monday: (Brief summary)

  • Followed previous steps to build the plumbing on Azure for my small SQL Server farm (See Previous status on AWS EC2  for more details) 
  1. Created Resource Group
  2. Create Application Security Group   
  3. Created 6 small Windows VMs in the same Region and an Availability Zone
  4. Joined them to Windows domain

Tuesday: (Brief summary)

  1. Created Windows Failover Cluster
  2. Installed SQL Server 2016
  3. Setup and configured AlwaysOn AGs and Listeners    

 Observations with Azure VMs:


  • Azure VMS are very slow first time brought up after build
  • Azure VMS has a longer provisioning time than EC2 Instances
  • No UI option to perform bulk tasks (like AWS UI) . Only option is Templating thru scripting 
  • Can not move Resource Group from one Geographical location to another like VMs and other objects within Azure
  • When deleting a VM all child dependencies are not dropped ( Security Groups, NICs, Disks) – Perhaps this is by design?

– Objects need to be dissociated with groups and then deleted for clean up of orphan objects


  • Easy to migrate VMs to higher T-Shirt Sizes
  • Easy to provision Storage Volumes per VM
  • Application Security Groups can be used to manage TCP/UDP traffic for entire resource group


  • You can migrate existing storage volumes to premium or cheaper storage seamlessly
  • Less network administration 
    • less TCP/UDP ports need to be opened especially ports native to Windows domains
  • Very Easy to build Windows Failover clustering services 
    • Natively works in the same subnet
    • Less configuration to get Connectivity to working then AWS EC2
  • Very Easy to configure SQL Server 2016 Always On
    • No need to create 5 Listeners (different per subnet) for a given specific AG 
    • 1 Listener per AG
  • Free Cost, Performance, Operation Excellence Recommendations Pop up after Login


  • Registered for an Eval account for Snowflake instance
  • Attended Zero to Snowflake in 90 Minutes virtual Lab
    • Created Databases, Data Warehouses, User accounts, and Roles
    • Created Stages to be used for Data Import
    • Imported Data Sources (Data in S3 Buckets, CSV, JSON formats) via Web UI and SnowSQL cmd line tool
    • Ran various ANSI-92 T-SQL Queries to generate reports from SnowFlake



**Bonus Points **

  • More Algebra – Regents questions. 
  • More with conjugating verbs in Español (AR Verbs)

Next Steps.. 
Below are some topics I am considering for my voyage next week:

  •  SQL Server Advanced Features:

           – Columnstore Indexes
           – Best practices around SQL Server AlwaysOn (Snapshot Isolation/sizing of Tempdb, etc)

  • Data Visualization Tools (i.e. Looker)
  • ETL Solutions (Stitch, FiveTran) 
  • Process and Transforming data/Explore data through ML (i.e. Databricks) .
  • Getting Started with Kubernetes with an old buddy (Nigel)

Stay safe and Be well


Week of March 20th

Hey, you, get off of my cloud”! (that includes you Corona)

So after taking a brief victory lap for getting my virtual Windows Domain Sandbox up and running last week in AWS EC2. This week I attempted to get SQL Server AlwaysOn running with multiple Availability Groups and multiple Replicas (Async w/Read, Async w/ No read, Sync with Automatic Failover and Sync with Read Automatic Failover options)

After spending multiple days and countless attempts, I was left with numerous disappointments for most of the week as I fruitlessly chased numerous red herrings with non-useful Windows Cluster Log errors.

However, I refused to accept defeat! So I am happy to report that I finally persevered for this week’s mission

Here is what I did:

  • Removed my flaky implementation of AlwaysOn AGs and Listener with some lame hacks that had my AlwaysOn Implementation sudo running earlier in the week
  • Destroyed Windows Failover Cluster

Began the recovery mission:

  • Allocated 5 new class C subsets to existing VPC
  • Added additional Policies to Security Group in VPC
    • Added UDP port for SQL Browser Service
    • Added TCP ports used with AlwaysOn
  • Added a second virtual NIC to designated 5 SQL Severs EC2 instances
  • Configured 2nd IP for 2nd NIC using unique IP from designated Class C subnet
  • Designated DC/DNS Server as my Jump Server as all SQL EC2 instances are no longer routable from internet thru RDP
  •  Recreated Windows Failover Cluster with 5 SQL Servers 
  •  Added 5 virtual Network IPs (1 for each Class C subnet) to be used with Windows Cluster.
  • Recreated Always On with 2 AGs with ASync w/Read, ASync w/ No read, Sync with AF and Sync with Read AF
  • Configured SQL Listener to use 5 IPs in each subnet that hosts SQL Servers.
  • Added 5 Network IP under AG resources in Windows Cluster to be used with SQL Listener
  •  Enabled SQL Browsing Service on all 5 SQL EC2 instances (As I tried to be cute and make all SQL instances a Named Instances instead using default)
  •  Removed all SQL Aliases previously created as they are no longer needed
  •  Resolved “nslookup” issues that were plaguing name resolution on my Network
  •  Added 5 new Class C subnets to Reverse Lookup Zones in DNS
  •  Added Windows Cluster and SQL Listener Names for each subnet to DNS Forward Lookup Zones
  •  Successfully Tested failover scenarios w/ SQL AlwaysOn
  •  Installed and configured Redgate SQL Data Generator software on Jump Server
  •   Successfully Tested various scenarios with pumping millions of records to the SQL Listener while initiating AlwaysOn failover

Next Steps.. 

  • Play around a little bit more with SQL AlwaysOn
  • Maybe take on Snakes and Bears (Start by watching some videos)…
  • Stayed Tuned.

Peace Out –

Week of March 13

Forget Corona! Take me to the cloud

Creating and Configuring a SQL Server AlwaysOn Configuration in AWS EC2

  • Created VPC in AWS
    • Designated Private Subnet address
    • Specified proper routes
  • Created Security Groups
    • Specified specific Protocols, Ports, Source/destination 
  • Created 6 micro AWS Instances of Windows 2016 DC Edition
    • 1 Domain Controller
    • 1 Primary Database Server
    • 4 Replica Database Servers
  • Configured Windows Active Directory, DNS
    • Joined Windows Domain
    • Created FID to be used with SQL Server Services
  • Established connectivity between “MCSSYSTEMS” AWS Instances
    • Configured virtual NICs to point internal DNS and use static IP
  • Configured additional storage volumes
    • 15 (300 GB) Volumes  – 3 volumes (Data, Log, TempDB ) to be assigned to 5 designated SQL Servers)
    • Attached storage to AWS Instances to be used with SQL Server
    • Brought Disks online as Dynamic disks
    • Formatted Disks (64K)
  • Migrated 5 AWS micros instances to medium t-shirt size Instance so to be used with SQL Server
    •  re-configured virtual NICs to point internal DNS and use static IP (as all settings were lost during migration)
  • Installed SQL Server 2016 w/ SP2 and SSMS 2018 on 5 Instances
    • Disabled Windows FW
    • enforced best practices (separation of Data, Log, Temp)
    • Created multiple Temp DB files to match v CPUs
    • Put SQL FID in Lock Pages in Memory Policy, instant file initialization
    • Configured static port # to be used with SQL Server instances
    • Configured SQL Server alias on instances can connect to friendly name (DNS is flaky)
  • Began SQL Server AlwaysOn Prep work
    • Install/Configured Windows Fail Over Cluster Services on 5 SQL Server AWS Instances
    • Enable each SQL Server for Always on availability groups
    • verified all SQL Servers are running properly and prepped for Always 

Next Steps.. 

  • Download and attach AdventureWorks database (Sample)
  • Configure AlwaysOn, Availability Group(s), Listener
  • Create Primary, Secondary, and Read Only Replicas, etc
  • Consult with the Big Guns