Contents

Microsoft Certified: Azure AI Engineer Associate Study Guide

Note: In case you are planning to study to this exam and certification, please keep in mind that this exam will be replaced by the AI-102 exam coming on February 23, 2021 - (AI-102)

The Microsoft Certified: Azure AI Engineer Associate certification enables you to get the knowledge to be subject matter expert using Cognitive Services, Machine Learning (ML), and knowledge mining to architect and implement AI solutions on Azure involving natural language processing, speech, computer vision, conversational AI and more.

In this study guide, I will share with you some of the useful resources you can use to guide you during your learning path to get this certification.

Certification Path

Exam Name Link
Exam AI-100: Designing and Implementing an Azure AI Solution Exam Details

Exam AI-100: Designing and Implementing an Azure AI Solution

The AI-100 has a length of three hours. There are 50+ questions and you need a minimum of 700 of 1000 points to pass the exam.

First place to go is the Microsoft Learn platform where a dedicated learning path is available, for free. Also, you should have a look to the Resources section in this study guide where you have useful resources to help you consolidate the knowledge that will help you get the exam and certification. If you prefer to watch videos, instead of read, explaining these core concepts and showing how to get started using Power Platform, then I invite you to have a look to the Microsoft Azure AI Engineer (AI-100) path, available on Pluralsight.

Skills measured

Analyze solution requirements (25-30%)

Recommend Azure Cognitive Services APIs to meet business requirements

  • select the processing architecture for a solution
  • select the appropriate data processing technologies
  • select the appropriate AI models and services
  • identify components and technologies required to connect service endpoints
  • identify automation requirements

Map security requirements to tools, technologies, and processes

  • identify processes and regulations needed to conform with data privacy, protection, and regulatory requirements
  • identify which users and groups have access to information and interfaces
  • identify appropriate tools for a solution
  • identify auditing requirements

Select the software, services, and storage required to support a solution

  • identify appropriate services and tools for a solution
  • identify integration points with other Microsoft services
  • identify storage required to store logging, bot state data, and Azure Cognitive Services output
Design AI solutions (40-45%)

Design solutions that include one or more pipelines

  • define an AI application workflow process
  • design a strategy for ingest and egress data
  • design the integration point between multiple workflows and pipelines
  • design pipelines that use AI apps
  • design pipelines that call Azure Machine Learning models
  • select an AI solution that meet cost constraints

Design solutions that uses Cognitive Services

  • design solutions that use vision, speech, language, knowledge, search, and anomaly detection APIs

Design solutions that implement the Microsoft Bot Framework

  • integrate bots and AI solutions
  • design bot services that use Language Understanding (LUIS)
  • design bots that integrate with channels
  • integrate bots with Azure app services and Azure Application Insights

Design the compute infrastructure to support a solution

  • identify whether to create a GPU, FPGA, or CPU-based solution
  • identify whether to use a cloud-based, on-premises, or hybrid compute infrastructure
  • select a compute solution that meets cost constraints

Design for data governance, compliance, integrity, and security

  • define how users and applications will authenticate to AI services
  • design a content moderation strategy for data usage within an AI solution
  • ensure that data adheres to compliance requirements defined by your organization
  • ensure appropriate governance of data
  • design strategies to ensure that the solution meets data privacy regulations and industry standards
Implement and monitor AI solutions (25-30%)

Implement an AI workflow

  • develop AI pipelines
  • manage the flow of data through the solution components
  • implement data logging processes
  • define and construct interfaces for custom AI services
  • create solution endpoints
  • develop streaming solutions

Integrate AI services and solution components

  • configure prerequisite components and input datasets to allow the consumption of Azure Cognitive Services APIs
  • configure integration with Azure Cognitive Services
  • configure prerequisite components to allow connectivity to the Microsoft Bot Framework
  • implement Azure Cognitive Search in a solution

Monitor and evaluate the AI environment

  • identify the differences between KPIs, reported metrics, and root causes of the differences
  • identify the differences between expected and actual workflow throughput
  • maintain an AI solution for continuous improvement
  • monitor AI components for availability
  • recommend changes to an AI solution based on performance data

Additional Notes

During the process of studying to this certification and the AI-100 exam, I took some personal notes. So I am sharing below the notes of some important concepts I studied to the exam.

  • Cognitive Services
    • Scenarios
      • Little to no customization needed
      • Team with limited ML and Data Science capabilities
        • Existance of solution to solve your ML problem
      • Categories

        • Decision
          • Anomaly Detector
          • Embed anomaly detection capabilities to apps
            • Identify quickly problems in your apps
            • Ingest timeseries data and select the best fitting detection model for your data
            • Provides Container Support to run your service in the Edge using a container
          • Content Moderator
            • Images, Text and Video moderation
            • Moderate content
            • Identify adult, offensive or inappropriate content
          • Metrics Advisor
          • Personalizer
            • Deliver rich and personalized experience to users, by personalizing content according to user preferences and historic
        • Language
          • Immersive reader
          • Language Understanding (LUIS)
            • Applies ML intelligence to conversational natural language text to predict and extract relevant detailed information
            • Used commonly in chat bots to identify requests from users and provide expected outcomes to users.
            • Provides support to be exported as IoT Edge module
          • QnA maker
            • Extract questions and answers from your content
          • Text Analytics
            • Detect sentiment in text (value 0 (negative sentiment) to 1 (positive sentiment)). Uses classifier model.
              • Provides support to be exported as IoT Edge module
            • Key Phrease extraction
              • Provides support to be exported as IoT Edge module
            • Language Detection
              • Provides support to be exported as IoT Edge module
            • Named Entity Recognition
            • Provides Container Support to run your service in the Edge using a container
          • Translator
            • Detect Languages
            • Translate text into different languages
            • Online and offline translate support
        • Speech
          • Speech to Text: Real-time transcription to text
            • Provides Container Support to run your service in the Edge using a container
          • Text to Speech: Speak to users naturally, convert text to audio near real-time
            • Provides Container Support to run your service in the Edge using a container
          • Speech Translation: Real-time speech translation
          • Speaker Recognition: Verify voice to identify the person talking
          • Custom Models
            • Acoustic Model: e.g. Use for particular environment, such as objects, instruments or specific population
            • Language Model: e.g. Tailoring to the vocabulary and speaking style of users
            • Pronunciation Model: e.g. Specific phonetic form and display of a word or term
            • Voice Font: Create custom voices
        • Vision
          • Computer Vision
            • Analyze Image: Tagging, Object Detection, Adult Content
            • Analyze Text
            • Identify Celebrities
          • Face
            • Detect Faces in Images, Face Identification, identify characteristics in a face (age, gender, smile, etc.)
            • Provides Container Support to run your service in the Edge using a container
            • Provides support to be exported as IoT Edge module
          • Video Indexer
            • Facial, Object recognition, OCR, transcription
            • Topic inference, brands and emotion detection
          • Custom Vision
            • Upload images, label images and train your custom models to label new images
            • Model based Edge support - Export your custom trained model and use it with different frameworks (tensorflow, etc.)
          • Form Recognizer
            • Extract text, key-value pairs and tables from documents, forms and receipts with low or no manual intervention
            • Leverages pre-built components to understand layout and relationship between fields and entries in your documents
            • Provides Container Support to run your service in the Edge using a container
            • Provides support to be exported as IoT Edge module
      • Search

Uses prebuilt models

Capability Input type Key benefit
Text Analytics API Text Evaluate sentiment and topics to understand what users want.
Entity Linking API Text Power your app’s data links with named entity recognition and disambiguation.
Language Understanding Intelligent Service (LUIS) Text Teach your apps to understand commands from your users.
QnA Maker Service Text Distill FAQ formatted information into conversational, easy-to-navigate answers.
Linguistic Analysis API Text Simplify complex language concepts and parse text.
Knowledge Exploration Service Text Enable interactive search experiences over structured data via natural language inputs.
Web Language Model API Text Use predictive language models trained on web-scale data.
Academic Knowledge API Text Tap into the wealth of academic content in the Microsoft Academic Graph populated by Bing.
Bing Autosuggest API Text Give your app intelligent autosuggest options for searches.
Bing Spell Check API Text Detect and correct spelling mistakes in your app.
Translator Text API Text Machine translation.
Recommendations API Text Predict and recommend items your customers want.
Bing Entity Search API Text (web search query) Identify and augment entity information from the web.
Bing Image Search API Text (web search query) Search for images.
Bing News Search API Text (web search query) Search for news.
Bing Video Search API Text (web search query) Search for videos.
Bing Web Search API Text (web search query) Get enhanced search details from billions of web documents.
Bing Speech API Text or Speech Convert speech to text and back again.
Speaker Recognition API Speech Use speech to identify and authenticate individual speakers.
Translator Speech API Speech Perform real-time speech translation.
Computer Vision API Images (or frames from video) Distill actionable information from images, automatically create description of photos, derive tags, recognize celebrities, extract text, and create accurate thumbnails.
Content Moderator Text, Images or Video Automated image, text, and video moderation.
Emotion API Images (photos with human subjects) Identify the range emotions of human subjects.
Face API Images (photos with human subjects) Detect, identify, analyze, organize, and tag faces in photos.
Video Indexer Video Video insights such as sentiment, transcript speech, translate speech, recognize faces and emotions, and extract keywords.

Trained with custom data you provide

Capability Input type Key benefit
Custom Vision Service Images (or frames from video) Customize your own computer vision models.
Custom Speech Service Speech Overcome speech recognition barriers like speaking style, background noise, and vocabulary.
Custom Decision Service Web content (for example, RSS feed) Use machine learning to automatically select the appropriate content for your home page
Bing Custom Search API Text (web search query) Commercial-grade search tool.

What are your options when choosing an NLP service?

In Azure, the following services provide natural language processing (NLP) capabilities:


Key selection criteria

To narrow the choices, start by answering these questions:

  • Do you want to use prebuilt models? If yes, consider using the APIs offered by Microsoft Cognitive Services.
  • Do you need to train custom models against a large corpus of text data? If yes, consider using Azure HDInsight with Spark MLlib and Spark NLP.
  • Do you need low-level NLP capabilities like tokenization, stemming, lemmatization, and term frequency/inverse document frequency (TF/IDF)? If yes, consider using Azure HDInsight with Spark MLlib and Spark NLP.
  • Do you need simple, high-level NLP capabilities like entity and intent identification, topic detection, spell check, or sentiment analysis? If yes, consider using the APIs offered by Microsoft Cognitive Services.

ContentModerator

  • SDK
  • Use the Content Moderator client library for .NET to:
    • Moderate text
    • Moderate images
    • Create a review
  • Image Moderation
    • Create and initialize an instance of the Content Moderator API wrapper. ContentModeratorClient client = new ContentModeratorClient(new ApiKeyServiceClientCredentials(CMSubscriptionKey));

    • Evaluate for adult and racy content. client.ImageModeration.EvaluateUrlInput(“application/json”, url, true);

    • Detect and extract text. client.ImageModeration.OCRUrlInput(“eng”, “application/json”, url, true);

    • Detect faces. client.ImageModeration.FindFacesUrlInput(“application/json”, url, true);

  • Image Reviews
    • Team Name - This must be the team name you used to create your Content Moderator account.
    • Create Review client.Reviews.CreateReviewsWithHttpMessagesAsync(“application/json”, TeamName, requestInfo);
    • Get Review client.Reviews.GetReviewWithHttpMessagesAsync(TeamName, item.ReviewId);
  • Text Moderation
    • Screen the input text: check for profanity, classify the text into three categories, do autocorrect text, and check for personally identifying information (PII) client.TextModeration.ScreenText(“text/plain”, new MemoryStream(Encoding.UTF8.GetBytes(text)), language: “eng”, autocorrect: true, pII: true, listId: null, classify: true);
  • Video Moderation

Language Understanding Intelligent Service (LUIS)

  • LUIS makes use of three key aspects for understanding language:
  • Utterances: An utterance is input from the user that your app needs to interpret.
  • Intents: An intent represents a task or action the user wants to do. It’s a purpose or goal expressed in a user’s utterance.
  • Entities: An entity represents a word or phrase inside the utterance that you want to extract.
  • LUIS config (JSON format)
  • LUIS Pizza App Example (JSON)
  • LUIS API
  • LUIS URL Format: https://<region>.api.cognitive.microsoft.com
  • LUIS Response Example
     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    
    {
      "query": "Book me a flight to Cairo",
      "prediction": {
          "topIntent": "BookFlight",
          "intents": {
              "BookFlight": {
                  "score": 0.9887482
              },
              "None": {
                  "score": 0.04272597
              },
              "LocationFinder": {
                  "score": 0.0125702191
              },
              "Reminder": {
                  "score": 0.00375502417
              },
              "FoodOrder": {
                  "score": 3.765154e-7
              }
          },
          "entities": {
              "Location": [
                  "Cairo"
              ]
          }
      }
    }
    
  • SDK Examples
  • Create Application
    1
    2
    3
    4
    5
    6
    7
    8
    
              var app_info = new ApplicationCreateObject()
              {
                  Name = app_name,
                  InitialVersionId = app_version,
                  Description = app_description,
                  Culture = app_culture
              };
              var app_id = await client.Apps.AddAsync(app_info);
    
  • Create Intents
     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    
      List<String> intents = new List<String>();
              intents.Add("Greeting");
              intents.Add("SearchPic");
              intents.Add("OrderPic");
              intents.Add("SharePic");
              foreach (string intent in intents)
              {
                  await client.Model.AddIntentAsync(app_info.ID, app_info.Version, new ModelCreateObject()
                  {
                      Name = intent
                  });
                  Console.WriteLine("Created intent {0}", intent);
              }
    
  • Create Entity
    1
    2
    3
    4
    
      var facetEntityId = await client.Model.AddEntityAsync(app_info.ID, app_info.Version, new ModelCreateObject()
               {
                   Name = "facet"
               });
    
  • Create Utterances
    1
    
      var resultsList = await client.Examples.BatchAsync(app_info.ID, app_info.Version, utterances);
    
  • Train Model
    1
    
      var response = await client.Train.TrainVersionAsync(app.ID, app.Version);
    
  • Publish App
    1
    2
    
      ApplicationPublishObject obj = new ApplicationPublishObject { VersionId = app.Version, IsStaging = true };
      var info = await client.Apps.PublishAsync(app.ID, obj);
    
  • Check Training Status
    1
    2
    
      var response = await client.Train.GetStatusAsync(app.ID, app.Version); 
      Console.WriteLine(response[0].Details.Status.ToString());
    

Speech API

  • Some Scenarios Speech to Text
    • Translation of live presentations
    • In-person or remote translated communications
    • Customer support
    • Business intelligence
    • Media subtitling
    • Multilingual AI interactions
  • SDK
    • Create configuration
      1
      
      var config = SpeechConfig.FromSubscription("KEY", "LOCATION");
      
    • Create SpeechRecognizer
      1
      
      var recognizer = new SpeechRecognizer(config, audioInput);
      
    • Recognize Async
      1
      
      var result = await recognizer.RecognizeOnceAsync();
      
    • Text to Speech
      1
      2
      3
      
      var fileOutput = AudioConfig.FromWavFileOutput(fileName);
      var synthesizer = new SpeechSynthesizer(config, fileOutput);
      var result = await synthesizer.SpeakTextAsync(text);
      
    • Creates a speech synthesizer using the SpeechSynthesisVoiceName for the audio output.
      1
      
      var speech_synthesizer = new SpeechSynthesizer(config);
      
    • Use the synthesized voice to speak the translation
      1
      
      await speech_synthesizer.SpeakTextAsync(result.Translations[toLanguage]);
      

Text Analytics API

  • Key Phrases Input’s JSON Example

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    
    {
        "documents": [
            {
                "language": "en",
                "id": "1",
                "text": "Document 1 text here"
            },
            {
                "language": "fr",
                "id": "2",
                "text": "Document 2 text here"
            }
        ]
    }
    
  • Key Phrases Output’s JSON Example

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    
    {
      "documents": [
        {
          "id": "1",
          "keyPhrases": [
            "world",
            "input text"
          ]
        },
        {
          "id": "2",
          "keyPhrases": [
            "monde"
          ]
        },
        {
          "id": "3",
          "keyPhrases": [
            "carretera",
            "tráfico",
            "día"
          ]
        }
      ],
      "errors": []
    }
    
    1
    
    
    
  • SDK - Key Phrases

  • Instantiate the client

    1
    2
    3
    4
    5
    
    var credentials = new ApiKeyServiceClientCredentials(key);
    var client = new TextAnalyticsClient(credentials)
    {
        Endpoint = endpoint
    };
    
  • Obtain Key Phrases

    1
    
      var response = await client.KeyPhrasesAsync(doc);
    
  • Process Key Phrases

    1
    2
    3
    4
    
    foreach (string keyphrase in response.KeyPhrases)
    {
      Console.WriteLine($"\t{keyphrase}");
    }
    
  • Language Detection

    • JSON Example Input
       1
       2
       3
       4
       5
       6
       7
       8
       9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      
          {
               "documents": [
                   {
                       "id": "1",
                       "text": "This document is in English."
                   },
                   {
                       "id": "2",
                       "text": "Este documento está en inglés."
                   },
                   {
                       "id": "3",
                       "text": "Ce document est en anglais."
                   },
                   {
                       "id": "4",
                       "text": "本文件为英文"
                   },
                   {
                       "id": "5",
                       "text": "Этот документ на английском языке."
                   }
               ]
           }
      
      
    • JSON Example Output
       1
       2
       3
       4
       5
       6
       7
       8
       9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      
      {
          "documents": [
              {
                  "id": "1",
                  "detectedLanguages": [
                      {
                          "name": "English",
                          "iso6391Name": "en",
                          "score": 1
                      }
                  ]
              },
              {
                  "id": "2",
                  "detectedLanguages": [
                      {
                          "name": "Spanish",
                          "iso6391Name": "es",
                          "score": 1
                      }
                  ]
              }
           ]
      }
      
    • NOTE: In case of a multi-language text, the service will identify the predominant language and return that single language
    • The last condition to consider is when there is ambiguity as to the language content. The scenario might happen if you submit textual content that the analyzer is not able to parse. As a result, the response for the language name and ISO code will indicate (unknown) and the score value will be returned as NaN, or Not a Number. The following example shows how the response would look.
    • SDK Example
      • Instantiate client
        1
        
          var client = new TextAnalyticsClient(endpoint, credentials);
        
      • Detect Language
        1
        
            DetectedLanguage detectedLanguage = client.DetectLanguage("Ce document est rédigé en Français.");
        

Vision Services

  • Face API, Emotion API, Vision API
  • Face API
  • The Face API tasks fall into five categories:
    • Verification: Check the likelihood that two faces belong to the same person.
    • Detection: Detect human faces in an image.
    • Identification: Search and identify faces.
    • Similarity: Find similar faces.
    • Grouping: Organize unidentified faces into groups, based on their visual similarity.
  • Face API Attributes
    • Age
    • Gender
    • Smile intensity
    • Facial hair
    • Head pose (3D)
    • Emotion
  • Vision API
  • Urls
    • Analyze images: https://<region>.api.cognitive.microsoft.com/vision/v2.0/analyze?visualFeatures=<…>&details=<…>&language=<…> The parameters that are sent to the service are visualFeatures, details, and languages. Set the details parameter to Landmarks or Celebrities to help you identify landmarks or celebrities. visualFeatures identify what kind of information you want the service to return you. The Categories option will categorize the content of the images like trees, buildings, and more. Faces will identify people’s faces and give you their gender and age.

    • Recognize Text: https://<region>.api.cognitive.microsoft.com/vision/v2.0/recognizeText?mode=<…> If present, the mode parameter must be set to Handwritten or Printed and is case-sensitive. If the parameter is set to Handwritten or is not specified, handwriting recognition is performed. If the parameter is set to Printed, then printed text recognition is performed. The time it takes to get a result from this call depends on the amount of writing in the image.

      • Response’s Example:
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        12
        13
        14
        15
        16
        17
        18
        19
        20
        21
        22
        23
        24
        25
        26
        27
        28
        29
        30
        31
        32
        33
        34
        35
        
        {
          "status":"Succeeded",
          "recognitionResult":{
              "lines":[
                {
                    "boundingBox":[
                      123,
                      131,
                      404,
                      138,
                      403,
                      169,
                      123,
                      162
                    ],
                    "text":"It does not matter",
                    "words":[
                      {
                          "boundingBox":[
                            128,
                            132,
                            163,
                            133,
                            164,
                            163,
                            129,
                            163
                          ],
                          "text":"It"
                      }, (... - other words)
          ]
                }, (... - other lines)
                        ]
          }
        }
        
    • Get an image’s thumbnail: https://<region>.api.cognitive.microsoft.com/vision/v2.0/generateThumbnail?width=<…>&height=<…>&smartCropping=<…> Different parameters can be provided to the API to generate the proper thumbnail for your needs. The width and height parameters are required. They tell the API which size you need for a specific image. The smartCropping parameter generates smarter cropping by analyzing the region of interest in your image to keep it within the thumbnail. As an example, with smart cropping enabled, a cropped profile picture would keep someone’s face within the picture frame even when the picture has a different aspect ratio.

    • Extract Printed Text in Image: https://<region>.api.cognitive.microsoft.com/vision/v2.0/ocr?language=<…>&detectOrientation=<…> As usual, all calls must be made to the region where the account was created. The call accepts two optional parameters: • language: The language code of the text to be detected in the image. The default value is unk,or unknown. This let’s the service auto detect the language of the text in the image. • detectOrientation: When true, the service tries to detect the image orientation and correct it before further processing, for example, whether the image is upside-down.

Custom AI Solutions

  • Scenarios:
    • Highly customized, custom algorithms
    • Use existing custom workflows/solutions
    • Team with ML capabilities

Azure Machine Learning

  • Compute: Local & Remote

    • Options
      • Azure ML Compute Cluster
      • Remote VM
      • Azure Databricks
      • Azure Data Lake Analytics
      • Azure HDInsight
      • Azure Batch
  • Compute Targets for inference

    Compute target Used for GPU support FPGA support Description
    Local web service Testing/debugging Use for limited testing and troubleshooting. Hardware acceleration depends on use of libraries in the local system.
    Azure Kubernetes Service (AKS) Real-time inference Yes (web service deployment) Yes Use for high-scale production deployments. Provides fast response time and autoscaling of the deployed service. Cluster autoscaling isn’t supported through the Azure Machine Learning SDK. To change the nodes in the AKS cluster, use the UI for your AKS cluster in the Azure portal. Supported in the designer.
    Azure Container Instances Testing or development Use for low-scale CPU-based workloads that require less than 48 GB of RAM. Supported in the designer.
    Azure Machine Learning compute clusters Batch inference Yes (machine learning pipeline) Run batch scoring on serverless compute. Supports normal and low-priority VMs. No support for realtime inference.
  • Training Compute Targets

    Training  targets Automated machine learning Machine learning pipelines Azure Machine Learning designer
    Local computer Yes
    Azure Machine Learning compute cluster Yes Yes Yes
    Azure Machine Learning compute instance Yes (through SDK) Yes
    Remote VM Yes Yes
    Azure Databricks Yes (SDK local mode only) Yes
    Azure Data Lake Analytics Yes
    Azure HDInsight Yes
    Azure Batch Yes

Azure Databricks

  • Analytics Spark Platform
  • Integrated with Azure
  • Build streamlined workflows and interactive workspace to enable collaboration between data science teams
  • Provides Spark SQL to perform analysis and DataFrames to store your data in distributed collections composed by named columns
  • Real-time data processing and analysis for analytical and interactive applications
  • Built-in integration with Hadoop FS, Flume and Kafka
  • Provides MLib (ML library with common ML algorithms and utilities including classification and regression, dimensionality reduction, optimization primitives and more.
  • Has GraphX which graphs and graph computation for a broad scope of use cases from Cognitive Analytics to Data Exploration
  • Supports Spark Core API and multiple languages including R, SQL, Python, Scala & Java
  • Advantages
    • Fully Managed clusters
    • Interactive workspace for exploration and visualization
    • Platform for powering your spark-based apps
    • Serverless - Abstracts the users from the infrastructure being used by your data-driven solutions
  • Data Exploration, Notebooks, Familiar Tools to visualize data

Azure Data Factory

  • Integration Runtimes (Public & Private Networks)
    • Azure: Enables Data Flow, Data Movement and Activity Dispatch
    • Self-hosted: Enables Data movement and Activity dispatch
    • Azure-SSIS: Enables SSIS package execution Note: Use self-hosted integration runtime to support data stores that requires bring-your-own driver such as SAP Hana, MySQL, etc. For more information, see supported data stores.

Big Data

  • Spark:
    • Process streaming data in real-time or in batches (uses RAM to process dataset - RDD)
    • Uses Hadoop MapReduce but with some improvements
    • Spark uses for a real-time stream, batch process and ETL also.
  • Kafka:
    • Kafka is a Message broker.
    • Kafka provides real-time streaming, window process (record by record)
    • Kafka has Producer, Consumer, Topic to work with data
    • In Kafka, we cannot perform a transformation
    • Kafka does not support any programming language to transform the data
    • So Kafka is used for real-time streaming as Channel or mediator between source and target
  • Storm
    • Distributed real-time processing of large volumes of high-velocity data.
    • Supports micro-batching processing
    • Data is mostly dynamic and continuously streamed.
    • Relatively slow.
    • Architecture consists of sprouts and bolts.
    • Scalable and fault-tolerant.
    • Implemented in Clojure.
    • Simple and can be used with any programming language.
    • Easy to set up and operate.
    • Used in business intelligence and Big Data analytics.
    • Open source (Apache license).
  • Hadoop
    • Distributed batch processing of large volumes of high-velocity data.
    • Persists data to disk after completing operations, so slower than Spark
    • Data is mostly static and stored in persistent storage.
    • Relatively fast.
    • Architecture consists of HDFS and MapReduce.
    • Scalable and fault-tolerant.
    • Implemented in Java.
    • Complex and can be used with any programming language.
    • Easy to set up but difficult to operate.
    • Used in business intelligence and Big Data analytics.
    • Open source (Apache license).

VM Series

  • A-Series: Entry-level economical VMs for dev/test
  • Bs-Series: Economical burstable VMs
  • D-Series: General purpose compute
  • DC-series: Protect data in use
  • E-Series: Optimised for in-memory hyper-threaded applications
  • F-Series: Compute optimised virtual machines
  • G-Series: Memory and storage optimised virtual machines
  • H-Series: High Performance Computing virtual machines
  • Ls-Series: Storage optimised virtual machines
  • M-Series: Memory optimised virtual machines
  • Mv2-Series: Largest memory optimised virtual machines
  • N-Series: GPU enabled virtual machines
  • Versions of VMs for ML
    • Pre-configured and pre-installed ML tools in these VMs
      • Data Science VM
      • Deep Learning VM
    • Geo AI Data Science VM

Azure CLI

  • Cognitive Services

    Command Description
    az cognitiveservices list List available Azure Cognitive Services accounts.
    az cognitiveservices account show Get the details of an Azure Cognitive Services account.
    az cognitiveservices account create Create an Azure Cognitive Services account.
    az cognitiveservices account delete Delete an Azure Cognitive Services account.
    az cognitiveservices keys list List the keys of an Azure Cognitive Services account.
  • ML

    Command Description
    az ml computetarget create aks Create an AKS compute target.
    az ml computetarget create amlcompute Create an AzureML compute target.
    az ml computetarget create computeinstance Create an AzureML compute instance target.
    az ml computetarget create datafactory Create a data factory compute target.
    az ml run submit-script Submit a script for execution.
    az ml run submit-pipeline Submit a pipeline for execution, from a published pipeline ID or pipeline YAML file.
    az ml model register -n sklearn_mnist –asset-path outputs/sklearn_mnist_model.pkl –experiment-name myexperiment –run-id myrunid –tag area=mnist Register a ML model

Resources

Resource
Microsoft Learning Path - Evaluate text with Azure Cognitive Language Services
Microsoft Learning Path - Process and Translate Speech with Azure Cognitive Speech Services
Microsoft Learning Path - Create Intelligent Bots with the Azure Bot Service
Microsoft Learning Path - Process and classify images with the Azure cognitive vision services
Microsoft Certified: Azure AI Engineer Associate
Microsoft Azure AI Engineer (AI-100) Pluralsight Certification Path