{"id":31844,"date":"2021-01-14T11:52:59","date_gmt":"2021-01-14T11:52:59","guid":{"rendered":"https:\/\/www.testpreptraining.com\/tutorial\/?page_id=31844"},"modified":"2021-01-14T11:53:00","modified_gmt":"2021-01-14T11:53:00","slug":"choosing-data-processing-technologies","status":"publish","type":"page","link":"https:\/\/www.testpreptraining.ai\/tutorial\/choosing-data-processing-technologies\/","title":{"rendered":"Choosing data processing technologies"},"content":{"rendered":"\n<p><strong><a href=\"https:\/\/www.testpreptraining.ai\/tutorial\/google-certified-professional-cloud-architect\/\" target=\"_blank\" rel=\"noreferrer noopener\">Go back to GCP Tutorials<\/a><\/strong><\/p>\n\n\n\n<p>In this we will learn and understand about data processing technologies.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Data processing<\/strong><\/h3>\n\n\n\n<p>Dataproc and Dataflow offer autoscaling options to scale your data pipelines and data processing. These options to allow your pipelines to access more computing resources based on the processing load.<\/p>\n\n\n\n<p><strong>Recommendations<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Firstly, use Google Cloud Load Balancers to provide a global endpoint.<\/li><li>Secondly, use managed instance groups with Compute Engine to automatically scale.<\/li><li>Thirdly, use the cluster autoscaler in GKE to automatically scale the cluster.<\/li><li>Then, use App Engine to autoscale your Platform-as-a-Service (PaaS) application.<\/li><li>Lastly, use Cloud Run or Cloud Functions to autoscale your function or microservice.<\/li><\/ul>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Dataproc<\/strong><\/h4>\n\n\n\n<p>Dataproc is a managed Spark and Hadoop service that lets you take advantage of open source data tools for batch processing, querying, streaming, and machine learning. Moreover, dataproc automation helps you create clusters quickly, manage them easily, and save money by turning clusters off when you don&#8217;t need them.<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-large\"><a href=\"https:\/\/www.testpreptraining.ai\/google-cloud-certified-professional-cloud-architect-free-practice-test\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" width=\"750\" height=\"117\" src=\"https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2021\/01\/Google-Certified-Professional-Cloud-Architect-prac-tests-750x117.png\" alt=\"gcp cloud architect practice tests\" class=\"wp-image-31460\" srcset=\"https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2021\/01\/Google-Certified-Professional-Cloud-Architect-prac-tests-750x117.png 750w, https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2021\/01\/Google-Certified-Professional-Cloud-Architect-prac-tests.png 961w\" sizes=\"auto, (max-width: 750px) 100vw, 750px\" \/><\/a><\/figure><\/div>\n\n\n\n<p>Further, when compared to traditional, on-premises products and competing cloud services, Dataproc has a number of unique advantages for clusters of three to hundreds of nodes:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Firstly, <strong>Low cost<\/strong>. Dataproc is priced at only 1 cent per virtual CPU in your cluster per hour, on top of the other Cloud Platform resources you use. In addition to this low price, Dataproc clusters can include preemptible instances that have lower compute prices. Further, it reduces your costs.<\/li><li>Secondly, <strong>Super fast<\/strong>. Without using Dataproc, it can take from five to 30 minutes to create Spark and Hadoop clusters on-premises or through IaaS providers. Thirdly, <strong>Integrated<\/strong>. Dataproc has built-in integration with other Google Cloud Platform services. This include BigQuery, Cloud Storage, Cloud Bigtable, Cloud Logging, and Cloud Monitoring.<\/li><li>Next, <strong>Managed<\/strong>. Use Spark and Hadoop clusters without the assistance of an administrator or special software. Moreover, you can easily interact with clusters and Spark or Hadoop jobs through the Google Cloud Console, the Cloud SDK, or the Dataproc REST API. <\/li><li>Lastly, <strong>Simple and familiar<\/strong>. You don\u2019t need to learn new tools or APIs to use Dataproc, making it easy to move existing projects into Dataproc without redevelopment. <\/li><\/ul>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Dataflow <\/strong><\/h4>\n\n\n\n<p>Dataflow is a managed service for executing a wide variety of data processing patterns. This helps you how to deploy your batch and streaming data processing pipelines using Dataflow. It also include directions for using service features. Further, the Apache Beam SDK is an open source programming model that enables you to develop both batch and streaming pipelines. You create your pipelines with an Apache Beam program and then run them on the Dataflow service. <\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-large\"><a href=\"https:\/\/www.testpreptraining.ai\/google-cloud-certified-professional-cloud-architect-practice-exam\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" width=\"750\" height=\"117\" src=\"https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2021\/01\/Google-Certified-Professional-Cloud-Architect-online-course-750x117.png\" alt=\"data processing technologies GCP cloud architect  online course\" class=\"wp-image-31461\" srcset=\"https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2021\/01\/Google-Certified-Professional-Cloud-Architect-online-course-750x117.png 750w, https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2021\/01\/Google-Certified-Professional-Cloud-Architect-online-course.png 961w\" sizes=\"auto, (max-width: 750px) 100vw, 750px\" \/><\/a><\/figure><\/div>\n\n\n\n<p><strong>Reference:<\/strong> <a href=\"https:\/\/cloud.google.com\/architecture\/framework\/performance-cost-optimization#data_processing\" target=\"_blank\" rel=\"noreferrer noopener\">Google Documentation<\/a>, <a href=\"https:\/\/cloud.google.com\/dataflow\/docs\" target=\"_blank\" rel=\"noreferrer noopener\">Doc 1<\/a>, <a href=\"https:\/\/cloud.google.com\/dataproc\/docs\/concepts\/overview\" target=\"_blank\" rel=\"noreferrer noopener\">Doc 2<\/a><\/p>\n\n\n\n<p><strong><a href=\"https:\/\/www.testpreptraining.ai\/tutorial\/google-certified-professional-cloud-architect\/\" target=\"_blank\" rel=\"noreferrer noopener\">Go back to GCP Tutorials<\/a><\/strong><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Go back to GCP Tutorials In this we will learn and understand about data processing technologies. Data processing Dataproc and Dataflow offer autoscaling options to scale your data pipelines and data processing. These options to allow your pipelines to access more computing resources based on the processing load. Recommendations Firstly, use Google Cloud Load Balancers&#8230;<\/p>\n","protected":false},"author":2,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_acf_changed":false,"footnotes":""},"categories":[],"tags":[],"class_list":["post-31844","page","type-page","status-publish","hentry"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v22.1 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Choosing data processing technologies - Testprep Training Tutorials<\/title>\n<meta name=\"description\" content=\"Enhance your knowledge and learn about data processing technologies using the Google Certified Professional Cloud Architect Course Now!\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.testpreptraining.ai\/tutorial\/choosing-data-processing-technologies\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Choosing data processing technologies - Testprep Training Tutorials\" \/>\n<meta property=\"og:description\" content=\"Enhance your knowledge and learn about data processing technologies using the Google Certified Professional Cloud Architect Course Now!\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.testpreptraining.ai\/tutorial\/choosing-data-processing-technologies\/\" \/>\n<meta property=\"og:site_name\" content=\"Testprep Training Tutorials\" \/>\n<meta property=\"article:modified_time\" content=\"2021-01-14T11:53:00+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2021\/01\/Google-Certified-Professional-Cloud-Architect-prac-tests-750x117.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/choosing-data-processing-technologies\/\",\"url\":\"https:\/\/www.testpreptraining.ai\/tutorial\/choosing-data-processing-technologies\/\",\"name\":\"Choosing data processing technologies - Testprep Training Tutorials\",\"isPartOf\":{\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/#website\"},\"datePublished\":\"2021-01-14T11:52:59+00:00\",\"dateModified\":\"2021-01-14T11:53:00+00:00\",\"description\":\"Enhance your knowledge and learn about data processing technologies using the Google Certified Professional Cloud Architect Course Now!\",\"breadcrumb\":{\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/choosing-data-processing-technologies\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.testpreptraining.ai\/tutorial\/choosing-data-processing-technologies\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/choosing-data-processing-technologies\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.testpreptraining.ai\/tutorial\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Choosing data processing technologies\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/#website\",\"url\":\"https:\/\/www.testpreptraining.ai\/tutorial\/\",\"name\":\"Testprep Training Tutorials\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.testpreptraining.ai\/tutorial\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/#organization\",\"name\":\"Testprep Training\",\"url\":\"https:\/\/www.testpreptraining.ai\/tutorial\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.testpreptraining.com\/tutorial\/wp-content\/uploads\/2020\/07\/tpt-logo-6.png\",\"contentUrl\":\"https:\/\/www.testpreptraining.com\/tutorial\/wp-content\/uploads\/2020\/07\/tpt-logo-6.png\",\"width\":583,\"height\":153,\"caption\":\"Testprep Training\"},\"image\":{\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/#\/schema\/logo\/image\/\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Choosing data processing technologies - Testprep Training Tutorials","description":"Enhance your knowledge and learn about data processing technologies using the Google Certified Professional Cloud Architect Course Now!","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.testpreptraining.ai\/tutorial\/choosing-data-processing-technologies\/","og_locale":"en_US","og_type":"article","og_title":"Choosing data processing technologies - Testprep Training Tutorials","og_description":"Enhance your knowledge and learn about data processing technologies using the Google Certified Professional Cloud Architect Course Now!","og_url":"https:\/\/www.testpreptraining.ai\/tutorial\/choosing-data-processing-technologies\/","og_site_name":"Testprep Training Tutorials","article_modified_time":"2021-01-14T11:53:00+00:00","og_image":[{"url":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2021\/01\/Google-Certified-Professional-Cloud-Architect-prac-tests-750x117.png"}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.testpreptraining.ai\/tutorial\/choosing-data-processing-technologies\/","url":"https:\/\/www.testpreptraining.ai\/tutorial\/choosing-data-processing-technologies\/","name":"Choosing data processing technologies - Testprep Training Tutorials","isPartOf":{"@id":"https:\/\/www.testpreptraining.ai\/tutorial\/#website"},"datePublished":"2021-01-14T11:52:59+00:00","dateModified":"2021-01-14T11:53:00+00:00","description":"Enhance your knowledge and learn about data processing technologies using the Google Certified Professional Cloud Architect Course Now!","breadcrumb":{"@id":"https:\/\/www.testpreptraining.ai\/tutorial\/choosing-data-processing-technologies\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.testpreptraining.ai\/tutorial\/choosing-data-processing-technologies\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.testpreptraining.ai\/tutorial\/choosing-data-processing-technologies\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.testpreptraining.ai\/tutorial\/"},{"@type":"ListItem","position":2,"name":"Choosing data processing technologies"}]},{"@type":"WebSite","@id":"https:\/\/www.testpreptraining.ai\/tutorial\/#website","url":"https:\/\/www.testpreptraining.ai\/tutorial\/","name":"Testprep Training Tutorials","description":"","publisher":{"@id":"https:\/\/www.testpreptraining.ai\/tutorial\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.testpreptraining.ai\/tutorial\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.testpreptraining.ai\/tutorial\/#organization","name":"Testprep Training","url":"https:\/\/www.testpreptraining.ai\/tutorial\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.testpreptraining.ai\/tutorial\/#\/schema\/logo\/image\/","url":"https:\/\/www.testpreptraining.com\/tutorial\/wp-content\/uploads\/2020\/07\/tpt-logo-6.png","contentUrl":"https:\/\/www.testpreptraining.com\/tutorial\/wp-content\/uploads\/2020\/07\/tpt-logo-6.png","width":583,"height":153,"caption":"Testprep Training"},"image":{"@id":"https:\/\/www.testpreptraining.ai\/tutorial\/#\/schema\/logo\/image\/"}}]}},"_links":{"self":[{"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/pages\/31844","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/comments?post=31844"}],"version-history":[{"count":6,"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/pages\/31844\/revisions"}],"predecessor-version":[{"id":32005,"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/pages\/31844\/revisions\/32005"}],"wp:attachment":[{"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/media?parent=31844"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/categories?post=31844"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/tags?post=31844"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}