{"id":28402,"date":"2020-12-28T12:09:02","date_gmt":"2020-12-28T12:09:02","guid":{"rendered":"https:\/\/www.testpreptraining.com\/tutorial\/?page_id=28402"},"modified":"2020-12-28T12:09:03","modified_gmt":"2020-12-28T12:09:03","slug":"create-pipelines-and-activities","status":"publish","type":"page","link":"https:\/\/www.testpreptraining.ai\/tutorial\/create-pipelines-and-activities\/","title":{"rendered":"Create pipelines and activities"},"content":{"rendered":"\n<p><a href=\"https:\/\/www.testpreptraining.ai\/tutorial\/exam-dp-200-implementing-an-azure-data-solution\/\" target=\"_blank\" rel=\"noreferrer noopener\">Go back to DP-200 Tutorials<\/a><\/p>\n\n\n\n<p>In this we will learn about creating pipelines and activities. However, we will create a pipeline with a copy activity that uses the input and output datasets. And, the copy activity copies data from the file you specified in the input dataset settings to the file you specified in the output dataset settings.<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Firstly, create a JSON file named Adfv2QuickStartPipeline.json in the C:\\ADFv2QuickStartPSH folder with the following content:<\/li><\/ul>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2020\/12\/commands-11-381x400.png\" alt=\"Create pipelines and activities\" class=\"wp-image-28420\" width=\"470\" height=\"494\" srcset=\"https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2020\/12\/commands-11-381x400.png 381w, https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2020\/12\/commands-11.png 750w\" sizes=\"auto, (max-width: 470px) 100vw, 470px\" \/><figcaption>Image Source: Micrsoft<\/figcaption><\/figure><\/div>\n\n\n\n<ul class=\"wp-block-list\"><li>Secondly, to create the pipeline: Adfv2QuickStartPipeline, Run the Set-AzDataFactoryV2Pipeline cmdlet.<\/li><\/ul>\n\n\n\n<p><strong>PowerShell<\/strong><br><em>$DFPipeLine = Set-AzDataFactoryV2Pipeline <code>-DataFactoryName $DataFactory.DataFactoryName<\/code><br>-ResourceGroupName $ResGrp.ResourceGroupName <code>-Name \"Adfv2QuickStartPipeline\"<\/code><br>-DefinitionFile &#8220;.\\Adfv2QuickStartPipeline.json&#8221;<\/em><\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Pipelines and activities in Azure Data Factory<\/strong><\/h4>\n\n\n\n<p>A data factory can have one or more pipelines. A pipeline is a logical grouping of activities that together perform a task. However, the pipeline allows you to manage the activities as a set instead of each one individually. You deploy and schedule the pipeline instead of the activities independently.<\/p>\n\n\n\n<p>Data Factory has three groupings of activities: data movement activities, data transformation activities, and control activities. An activity can take zero or more input datasets and produce one or more output datasets. The following diagram shows the relationship between pipeline, activity, and dataset in Data Factory:<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter\"><img decoding=\"async\" src=\"https:\/\/docs.microsoft.com\/en-us\/azure\/data-factory\/media\/concepts-pipelines-activities\/relationship-between-dataset-pipeline-activity.png\" alt=\"Relationship between dataset, activity, and pipeline\"\/><\/figure><\/div>\n\n\n\n<p>Here, an input dataset represents the input for an activity in the pipeline, and an output dataset represents the output for the activity. Datasets identify data within different data stores, such as tables, files, folders, and documents. After you create a dataset, you can use it with activities in a pipeline.<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-full\"><a href=\"https:\/\/www.testpreptraining.ai\/implementing-an-azure-data-solution-dp-200-free-practice-test\" target=\"_blank\" rel=\"noopener noreferrer\"><img loading=\"lazy\" decoding=\"async\" width=\"961\" height=\"150\" src=\"https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2020\/08\/dp-200-pracice-tests-1.png\" alt=\"DP-200 practice tests\" class=\"wp-image-18535\" srcset=\"https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2020\/08\/dp-200-pracice-tests-1.png 961w, https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2020\/08\/dp-200-pracice-tests-1-750x117.png 750w\" sizes=\"auto, (max-width: 961px) 100vw, 961px\" \/><\/a><\/figure><\/div>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Data transformation activities<\/strong><\/h4>\n\n\n\n<p>Azure Data Factory supports the following transformation activities that can be added to pipelines either individually or chained with another activity.<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"546\" height=\"400\" src=\"https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2020\/12\/source-18-546x400.png\" alt=\"Create activities\" class=\"wp-image-28423\" srcset=\"https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2020\/12\/source-18-546x400.png 546w, https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2020\/12\/source-18.png 776w\" sizes=\"auto, (max-width: 546px) 100vw, 546px\" \/><figcaption>Image Source: Microsoft<\/figcaption><\/figure><\/div>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Activity policy<\/strong><\/h4>\n\n\n\n<p>Policies affect the run-time behavior of an activity, giving configurability options. However, Activity Policies are only available for execution activities.<\/p>\n\n\n\n<p><strong>Activity policy JSON definition<\/strong><\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"470\" height=\"400\" src=\"https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2020\/12\/commands-12-470x400.png\" alt=\"activity policy\" class=\"wp-image-28424\" srcset=\"https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2020\/12\/commands-12-470x400.png 470w, https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2020\/12\/commands-12.png 786w\" sizes=\"auto, (max-width: 470px) 100vw, 470px\" \/><figcaption>Image Source: Microsoft<\/figcaption><\/figure><\/div>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Control activity<\/strong><\/h4>\n\n\n\n<p>Control activities have the following top-level structure:<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"532\" height=\"400\" src=\"https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2020\/12\/commands-13-532x400.png\" alt=\"control activity\" class=\"wp-image-28425\" srcset=\"https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2020\/12\/commands-13-532x400.png 532w, https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2020\/12\/commands-13.png 789w\" sizes=\"auto, (max-width: 532px) 100vw, 532px\" \/><figcaption>Image Source: Microsoft<\/figcaption><\/figure><\/div>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Multiple activities in a pipeline<\/strong><\/h4>\n\n\n\n<p>You should know that you can have more than one activity in a pipeline. And, if you have multiple activities in a pipeline and subsequent activities are not dependent on previous activities, then, the activities may run in parallel. Also, you can chain two activities by using activity dependency, which defines how subsequent activities depend on previous activities.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Scheduling pipelines<\/strong><\/h4>\n\n\n\n<p>Pipelines are scheduled by triggers. And, there are different types of triggers (Scheduler trigger, which allows pipelines to be triggered on a wall-clock schedule, as well as the manual trigger, which triggers pipelines on-demand). <\/p>\n\n\n\n<p>However, to have your trigger kick off a pipeline run, you must include a pipeline reference of the particular pipeline in the trigger definition. Pipelines &amp; triggers have an n-m relationship. Multiple triggers can kick off a single pipeline, and the same trigger can kick off multiple pipelines. Once the trigger is defined, you must start the trigger to have it start triggering the pipeline.<\/p>\n\n\n\n<p><em>For example, say you have a Scheduler trigger, &#8220;Trigger A,&#8221; that I wish to kick off my pipeline, &#8220;MyCopyPipeline.&#8221; You define the trigger, as shown in the following example:<\/em><\/p>\n\n\n\n<p><strong>Trigger A definition<\/strong><\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"750\" height=\"329\" src=\"https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2020\/12\/commands-14-750x329.png\" alt=\"scheduling pipelines\" class=\"wp-image-28426\" srcset=\"https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2020\/12\/commands-14-750x329.png 750w, https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2020\/12\/commands-14.png 786w\" sizes=\"auto, (max-width: 750px) 100vw, 750px\" \/><figcaption>Image Source: Microsoft<\/figcaption><\/figure><\/div>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-full\"><a href=\"https:\/\/www.testpreptraining.ai\/implementing-an-azure-data-solution-dp-200-practice-exam\" target=\"_blank\" rel=\"noopener noreferrer\"><img loading=\"lazy\" decoding=\"async\" width=\"961\" height=\"150\" src=\"https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2020\/08\/dp-200-online-course-1.png\" alt=\"Create pipelines and activities DP-200 Online course\" class=\"wp-image-18534\" srcset=\"https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2020\/08\/dp-200-online-course-1.png 961w, https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2020\/08\/dp-200-online-course-1-750x117.png 750w\" sizes=\"auto, (max-width: 961px) 100vw, 961px\" \/><\/a><\/figure><\/div>\n\n\n\n<p class=\"has-text-align-right\"><strong>Reference: <\/strong><a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/data-factory\/concepts-pipelines-activities#next-steps\" target=\"_blank\" rel=\"noreferrer noopener\">Microsoft Documentation<\/a>, <a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/data-factory\/quickstart-create-data-factory-powershell#create-a-pipeline\" target=\"_blank\" rel=\"noreferrer noopener\">Documentation 2<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/www.testpreptraining.ai\/tutorial\/exam-dp-200-implementing-an-azure-data-solution\/\" target=\"_blank\" rel=\"noreferrer noopener\">Go back to DP-200 Tutorials<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Go back to DP-200 Tutorials In this we will learn about creating pipelines and activities. However, we will create a pipeline with a copy activity that uses the input and output datasets. And, the copy activity copies data from the file you specified in the input dataset settings to the file you specified in the&#8230;<\/p>\n","protected":false},"author":2,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_acf_changed":false,"footnotes":""},"categories":[],"tags":[],"class_list":["post-28402","page","type-page","status-publish","hentry"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v22.1 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Create pipelines and activities - Testprep Training Tutorials<\/title>\n<meta name=\"description\" content=\"Enhance your skills by learning about pipelines and activities using Microsoft DP-200 online course and Practice Exam Now!\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.testpreptraining.ai\/tutorial\/create-pipelines-and-activities\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Create pipelines and activities - Testprep Training Tutorials\" \/>\n<meta property=\"og:description\" content=\"Enhance your skills by learning about pipelines and activities using Microsoft DP-200 online course and Practice Exam Now!\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.testpreptraining.ai\/tutorial\/create-pipelines-and-activities\/\" \/>\n<meta property=\"og:site_name\" content=\"Testprep Training Tutorials\" \/>\n<meta property=\"article:modified_time\" content=\"2020-12-28T12:09:03+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.testpreptraining.com\/tutorial\/wp-content\/uploads\/2020\/12\/commands-11-381x400.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/create-pipelines-and-activities\/\",\"url\":\"https:\/\/www.testpreptraining.ai\/tutorial\/create-pipelines-and-activities\/\",\"name\":\"Create pipelines and activities - Testprep Training Tutorials\",\"isPartOf\":{\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/#website\"},\"datePublished\":\"2020-12-28T12:09:02+00:00\",\"dateModified\":\"2020-12-28T12:09:03+00:00\",\"description\":\"Enhance your skills by learning about pipelines and activities using Microsoft DP-200 online course and Practice Exam Now!\",\"breadcrumb\":{\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/create-pipelines-and-activities\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.testpreptraining.ai\/tutorial\/create-pipelines-and-activities\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/create-pipelines-and-activities\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.testpreptraining.ai\/tutorial\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Create pipelines and activities\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/#website\",\"url\":\"https:\/\/www.testpreptraining.ai\/tutorial\/\",\"name\":\"Testprep Training Tutorials\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.testpreptraining.ai\/tutorial\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/#organization\",\"name\":\"Testprep Training\",\"url\":\"https:\/\/www.testpreptraining.ai\/tutorial\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.testpreptraining.com\/tutorial\/wp-content\/uploads\/2020\/07\/tpt-logo-6.png\",\"contentUrl\":\"https:\/\/www.testpreptraining.com\/tutorial\/wp-content\/uploads\/2020\/07\/tpt-logo-6.png\",\"width\":583,\"height\":153,\"caption\":\"Testprep Training\"},\"image\":{\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/#\/schema\/logo\/image\/\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Create pipelines and activities - Testprep Training Tutorials","description":"Enhance your skills by learning about pipelines and activities using Microsoft DP-200 online course and Practice Exam Now!","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.testpreptraining.ai\/tutorial\/create-pipelines-and-activities\/","og_locale":"en_US","og_type":"article","og_title":"Create pipelines and activities - Testprep Training Tutorials","og_description":"Enhance your skills by learning about pipelines and activities using Microsoft DP-200 online course and Practice Exam Now!","og_url":"https:\/\/www.testpreptraining.ai\/tutorial\/create-pipelines-and-activities\/","og_site_name":"Testprep Training Tutorials","article_modified_time":"2020-12-28T12:09:03+00:00","og_image":[{"url":"https:\/\/www.testpreptraining.com\/tutorial\/wp-content\/uploads\/2020\/12\/commands-11-381x400.png"}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.testpreptraining.ai\/tutorial\/create-pipelines-and-activities\/","url":"https:\/\/www.testpreptraining.ai\/tutorial\/create-pipelines-and-activities\/","name":"Create pipelines and activities - Testprep Training Tutorials","isPartOf":{"@id":"https:\/\/www.testpreptraining.ai\/tutorial\/#website"},"datePublished":"2020-12-28T12:09:02+00:00","dateModified":"2020-12-28T12:09:03+00:00","description":"Enhance your skills by learning about pipelines and activities using Microsoft DP-200 online course and Practice Exam Now!","breadcrumb":{"@id":"https:\/\/www.testpreptraining.ai\/tutorial\/create-pipelines-and-activities\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.testpreptraining.ai\/tutorial\/create-pipelines-and-activities\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.testpreptraining.ai\/tutorial\/create-pipelines-and-activities\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.testpreptraining.ai\/tutorial\/"},{"@type":"ListItem","position":2,"name":"Create pipelines and activities"}]},{"@type":"WebSite","@id":"https:\/\/www.testpreptraining.ai\/tutorial\/#website","url":"https:\/\/www.testpreptraining.ai\/tutorial\/","name":"Testprep Training Tutorials","description":"","publisher":{"@id":"https:\/\/www.testpreptraining.ai\/tutorial\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.testpreptraining.ai\/tutorial\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.testpreptraining.ai\/tutorial\/#organization","name":"Testprep Training","url":"https:\/\/www.testpreptraining.ai\/tutorial\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.testpreptraining.ai\/tutorial\/#\/schema\/logo\/image\/","url":"https:\/\/www.testpreptraining.com\/tutorial\/wp-content\/uploads\/2020\/07\/tpt-logo-6.png","contentUrl":"https:\/\/www.testpreptraining.com\/tutorial\/wp-content\/uploads\/2020\/07\/tpt-logo-6.png","width":583,"height":153,"caption":"Testprep Training"},"image":{"@id":"https:\/\/www.testpreptraining.ai\/tutorial\/#\/schema\/logo\/image\/"}}]}},"_links":{"self":[{"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/pages\/28402","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/comments?post=28402"}],"version-history":[{"count":8,"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/pages\/28402\/revisions"}],"predecessor-version":[{"id":28567,"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/pages\/28402\/revisions\/28567"}],"wp:attachment":[{"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/media?parent=28402"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/categories?post=28402"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/tags?post=28402"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}