{"id":4797,"date":"2020-04-19T18:31:03","date_gmt":"2020-04-19T18:31:03","guid":{"rendered":"https:\/\/www.testpreptraining.com\/tutorial\/?page_id=4797"},"modified":"2020-04-19T18:31:03","modified_gmt":"2020-04-19T18:31:03","slug":"cloud-pub-sub-overview-google-professional-data-engineer-gcp","status":"publish","type":"page","link":"https:\/\/www.testpreptraining.ai\/tutorial\/cloud-pub-sub-overview-google-professional-data-engineer-gcp\/","title":{"rendered":"Cloud Pub\/Sub Overview Google Professional Data Engineer GCP"},"content":{"rendered":"<ul>\n<li>use Pub\/Sub as messaging-oriented middleware<\/li>\n<li>use as event ingestion and delivery for streaming analytics pipelines.<\/li>\n<li>offers durable message storage and real-time message delivery<\/li>\n<li>gives high availability and consistent performance at scale.<\/li>\n<li>It is a publish\/subscribe (Pub\/Sub) service<\/li>\n<li>senders of messages are decoupled from the receivers of messages<\/li>\n<li>Main terms<\/li>\n<li>Message: the data that moves through the service.<\/li>\n<li>Topic: a named entity that represents a feed of messages.<\/li>\n<li>Subscription: a named entity that represents an interest in receiving messages on a particular topic.<\/li>\n<li>Publisher (also called a producer): creates messages and sends (publishes) them to the messaging service on a specified topic.<\/li>\n<li>Subscriber (also called a consumer): receives messages on a specified subscription.<\/li>\n<li>publisher creates and sends messages to a topic.<\/li>\n<li>Subscriber applications create a subscription to a topic to receive messages from it.<\/li>\n<li>Communication can be\n<ul>\n<li>one-to-many (fan-out)<\/li>\n<li>many-to-one (fan-in),<\/li>\n<li>many-to-many.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>The flow of messages through Pub\/Sub is as<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-4673\" src=\"https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2020\/04\/Professional-Data-Engineer-Google-Cloud-image014.png\" alt=\"\" width=\"601\" height=\"197\" \/><\/p>\n<p>In above figure<\/p>\n<ul>\n<li>There are two publishers publishing messages on a single topic.<\/li>\n<li>2 subscriptions to the topic<\/li>\n<li>The first subscription has two subscribers, so messages will be load-balanced across them<\/li>\n<li>each subscriber receiving a subset of the messages<\/li>\n<li>The second subscription has one subscriber that will receive all of the messages.<\/li>\n<li>The bold letters are messages.<\/li>\n<li>Message A comes from Publisher 1 and sent to Subscriber 2 via Subscription 1, and to Subscriber 3 via Subscription 2.<\/li>\n<li>Message B comes from Publisher 2 and is sent to Subscriber 1 via Subscription 1 and to Subscriber 3 via Subscription 2.<\/li>\n<\/ul>\n<p><strong>Publisher and subscriber endpoints<\/strong><\/p>\n<ul>\n<li>Publishers should make HTTPS requests to pubsub.googleapis.com<\/li>\n<li>It can be\n<ul>\n<li>an App Engine app<\/li>\n<li>a web service hosted on Google Compute Engine<\/li>\n<li>other third-party network, an app installed on a desktop or mobile device, or even a browser.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<ul>\n<li><img loading=\"lazy\" decoding=\"async\" class=\"size-medium wp-image-4674 aligncenter\" src=\"https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2020\/04\/Professional-Data-Engineer-Google-Cloud-image015-551x400.png\" alt=\"\" width=\"551\" height=\"400\" srcset=\"https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2020\/04\/Professional-Data-Engineer-Google-Cloud-image015-551x400.png 551w, https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2020\/04\/Professional-Data-Engineer-Google-Cloud-image015.png 606w\" sizes=\"auto, (max-width: 551px) 100vw, 551px\" \/><\/li>\n<li>Pull subscribers make HTTPS requests to pubsub.googleapis.com.<\/li>\n<li>Push subscribers must be Webhook endpoints that can accept POST requests over HTTPS.<\/li>\n<\/ul>\n<p><strong>Common use cases<\/strong><\/p>\n<ul>\n<li>Balancing workloads in network clusters.<\/li>\n<li>Implementing asynchronous workflows.<\/li>\n<li>Distributing event notifications.<\/li>\n<li>Refreshing distributed caches.<\/li>\n<li>Logging to multiple systems.<\/li>\n<li>Data streaming from various processes or devices.<\/li>\n<\/ul>\n<p><strong>Architecture<\/strong><\/p>\n<ul>\n<li>Publisher and subscriber clients are not aware of the location of the servers to which they connect or how those services route the data.<\/li>\n<li>load balancing direct publisher traffic to the nearest GCP data center<\/li>\n<li>individual message is stored in a single region.<\/li>\n<li>topic may have messages stored in many regions.<\/li>\n<li>When a subscriber client requests messages published to this topic, it connects to the nearest server<\/li>\n<li>Pub\/Sub is divided into two primary parts:\n<ul>\n<li>the data plane managing moving messages between publishers and subscribers,<\/li>\n<li>the control plane, managing assignment of publishers and subscribers to servers on the data plane.<\/li>\n<\/ul>\n<\/li>\n<li>data plane servers are called forwarders<\/li>\n<li>control plane servers are called routers<\/li>\n<li>publishers and subscribers are connected to their assigned forwarders<\/li>\n<li>so easily upgrade the control plane of Pub\/Sub without affecting any clients<\/li>\n<li>All message is a base64-encoded message body and an arbitrary set of key-value pairs called attributes.<\/li>\n<li>There is no structure or context to the message so, JSON or XML entities must be enforced. Each message has a globally unique message ID, to identify if it has already been processed.<\/li>\n<li>Messages may be up to 10 MB in total size<\/li>\n<li>Two methods for message delivery: push subscriptions and pull subscriptions\n<ul>\n<li>In a push subscription, server sends a request to the subscriber app at a preconfigured URL endpoint.<\/li>\n<li>In the pull model, the subscriber requests messages from the server and acknowledges receipt.<\/li>\n<\/ul>\n<\/li>\n<li>Push subscriptions have a limits of 10,000 messages per second and 10,000 concurrent message deliveries.<\/li>\n<li>By default, subscriptions are created with an ack deadline of 10 seconds and the message deadline may be increased to up to 600 seconds.<\/li>\n<li>By default, subscriptions expire after 31 days of inactivity<\/li>\n<li>Using subscription expiration policies, can configure the inactivity duration<\/li>\n<\/ul>\n<p><strong>Topic and Message Management<\/strong><\/p>\n<ul>\n<li>can create, delete, and view topics using\n<ul>\n<li>the API<\/li>\n<li>the Google Cloud Console<\/li>\n<li>the gcloud command-line tool<\/li>\n<\/ul>\n<\/li>\n<li>must create a subscription to a topic before subscribers can receive messages published to the topic.<\/li>\n<li>create subscriptions with\n<ul>\n<li>the API<\/li>\n<li>the Google Cloud Console<\/li>\n<li>the gcloud command-line tool<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p><strong>Resource Name<\/strong><\/p>\n<ul>\n<li>Resource name uniquely identifies a Pub\/Sub resource<\/li>\n<li>Resource can be a subscription or topic<\/li>\n<li>must fit the following format: projects\/project-identifier\/collection\/relative-name<\/li>\n<li>The project-identifier must be the project ID, available from the Google Cloud Console. For example, projects\/myproject\/topics\/mytopic.<\/li>\n<li>The collection must be one of subscriptions or topics.<\/li>\n<li>The relative-name must:\n<ul>\n<li>Not begin with the string goog.<\/li>\n<li>Start with a letter<\/li>\n<li>Contain between 3 and 255 characters<\/li>\n<li>Contain only the following characters:\n<ul>\n<li>Letters: [A-Za-z]<\/li>\n<li>Numbers: [0-9]<\/li>\n<li>Dashes: &#8211;<\/li>\n<li>Underscores: _<\/li>\n<li>Periods: .<\/li>\n<li>Tildes: ~<\/li>\n<li>Plus signs: +<\/li>\n<li>Percent signs: %<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p><strong>Message Storage Security<\/strong><\/p>\n<ul>\n<li>If publish messages to a global endpoint, automatic storage in the nearest Google Cloud region.<\/li>\n<li>topic message storage policy ensure all data published to a topic is persisted in a specific region or set of regions, regardless of the publish request&#8217;s origin.<\/li>\n<li>When multiple regions are allowed by the policy, Pub\/Sub chooses the nearest allowed region.\n<ul>\n<li>To configure all of the topics in an organization-wide scope, use the Resource Location Restriction organization policy.<\/li>\n<li>For fine-grained control, configure a topic&#8217;s message storage policy at topic creation, or via the UpdateTopic operation.<\/li>\n<\/ul>\n<\/li>\n<li>You can configure the policy using the:\n<ul>\n<li>Topic details view<\/li>\n<li>gcloud command-line tool<\/li>\n<li>Service API (using client libraries)<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p><strong>Authentication and Access<\/strong><\/p>\n<ul>\n<li>Following authentication methods allowed\n<ul>\n<li>Service accounts<\/li>\n<li>User accounts &#8211; You can authenticate users directly to application, when the application needs to access resources on behalf of an end user.<\/li>\n<\/ul>\n<\/li>\n<li>Uses Cloud IAM for access control<\/li>\n<li>access control can be configured at the project level and at the individual resource level.<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>use Pub\/Sub as messaging-oriented middleware use as event ingestion and delivery for streaming analytics pipelines. offers durable message storage and real-time message delivery gives high availability and consistent performance at scale. It is a publish\/subscribe (Pub\/Sub) service senders of messages are decoupled from the receivers of messages Main terms Message: the data that moves through&#8230;<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_acf_changed":false,"footnotes":""},"categories":[617],"tags":[689,690,619,623,622,618,621],"class_list":["post-4797","page","type-page","status-publish","hentry","category-google-gcp","tag-cloud-pub-sub","tag-cloud-pub-sub-overview","tag-data-engineer","tag-gcp","tag-google-certification","tag-google-cloud","tag-professional-data-engineer"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v22.1 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Cloud Pub\/Sub Overview Google Professional Data Engineer GCP - Testprep Training Tutorials<\/title>\n<meta name=\"description\" content=\"Google Cloud Certified Professional Data Engineer Tutorial, dumps, brief notes on Cloud Pub\/Sub Overview\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.testpreptraining.ai\/tutorial\/cloud-pub-sub-overview-google-professional-data-engineer-gcp\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Cloud Pub\/Sub Overview Google Professional Data Engineer GCP - Testprep Training Tutorials\" \/>\n<meta property=\"og:description\" content=\"Google Cloud Certified Professional Data Engineer Tutorial, dumps, brief notes on Cloud Pub\/Sub Overview\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.testpreptraining.ai\/tutorial\/cloud-pub-sub-overview-google-professional-data-engineer-gcp\/\" \/>\n<meta property=\"og:site_name\" content=\"Testprep Training Tutorials\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2020\/04\/Professional-Data-Engineer-Google-Cloud-image014.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/cloud-pub-sub-overview-google-professional-data-engineer-gcp\/\",\"url\":\"https:\/\/www.testpreptraining.ai\/tutorial\/cloud-pub-sub-overview-google-professional-data-engineer-gcp\/\",\"name\":\"Cloud Pub\/Sub Overview Google Professional Data Engineer GCP - Testprep Training Tutorials\",\"isPartOf\":{\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/#website\"},\"datePublished\":\"2020-04-19T18:31:03+00:00\",\"dateModified\":\"2020-04-19T18:31:03+00:00\",\"description\":\"Google Cloud Certified Professional Data Engineer Tutorial, dumps, brief notes on Cloud Pub\/Sub Overview\",\"breadcrumb\":{\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/cloud-pub-sub-overview-google-professional-data-engineer-gcp\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.testpreptraining.ai\/tutorial\/cloud-pub-sub-overview-google-professional-data-engineer-gcp\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/cloud-pub-sub-overview-google-professional-data-engineer-gcp\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.testpreptraining.ai\/tutorial\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Cloud Pub\/Sub Overview Google Professional Data Engineer GCP\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/#website\",\"url\":\"https:\/\/www.testpreptraining.ai\/tutorial\/\",\"name\":\"Testprep Training Tutorials\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.testpreptraining.ai\/tutorial\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/#organization\",\"name\":\"Testprep Training\",\"url\":\"https:\/\/www.testpreptraining.ai\/tutorial\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.testpreptraining.com\/tutorial\/wp-content\/uploads\/2020\/07\/tpt-logo-6.png\",\"contentUrl\":\"https:\/\/www.testpreptraining.com\/tutorial\/wp-content\/uploads\/2020\/07\/tpt-logo-6.png\",\"width\":583,\"height\":153,\"caption\":\"Testprep Training\"},\"image\":{\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/#\/schema\/logo\/image\/\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Cloud Pub\/Sub Overview Google Professional Data Engineer GCP - Testprep Training Tutorials","description":"Google Cloud Certified Professional Data Engineer Tutorial, dumps, brief notes on Cloud Pub\/Sub Overview","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.testpreptraining.ai\/tutorial\/cloud-pub-sub-overview-google-professional-data-engineer-gcp\/","og_locale":"en_US","og_type":"article","og_title":"Cloud Pub\/Sub Overview Google Professional Data Engineer GCP - Testprep Training Tutorials","og_description":"Google Cloud Certified Professional Data Engineer Tutorial, dumps, brief notes on Cloud Pub\/Sub Overview","og_url":"https:\/\/www.testpreptraining.ai\/tutorial\/cloud-pub-sub-overview-google-professional-data-engineer-gcp\/","og_site_name":"Testprep Training Tutorials","og_image":[{"url":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2020\/04\/Professional-Data-Engineer-Google-Cloud-image014.png"}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.testpreptraining.ai\/tutorial\/cloud-pub-sub-overview-google-professional-data-engineer-gcp\/","url":"https:\/\/www.testpreptraining.ai\/tutorial\/cloud-pub-sub-overview-google-professional-data-engineer-gcp\/","name":"Cloud Pub\/Sub Overview Google Professional Data Engineer GCP - Testprep Training Tutorials","isPartOf":{"@id":"https:\/\/www.testpreptraining.ai\/tutorial\/#website"},"datePublished":"2020-04-19T18:31:03+00:00","dateModified":"2020-04-19T18:31:03+00:00","description":"Google Cloud Certified Professional Data Engineer Tutorial, dumps, brief notes on Cloud Pub\/Sub Overview","breadcrumb":{"@id":"https:\/\/www.testpreptraining.ai\/tutorial\/cloud-pub-sub-overview-google-professional-data-engineer-gcp\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.testpreptraining.ai\/tutorial\/cloud-pub-sub-overview-google-professional-data-engineer-gcp\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.testpreptraining.ai\/tutorial\/cloud-pub-sub-overview-google-professional-data-engineer-gcp\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.testpreptraining.ai\/tutorial\/"},{"@type":"ListItem","position":2,"name":"Cloud Pub\/Sub Overview Google Professional Data Engineer GCP"}]},{"@type":"WebSite","@id":"https:\/\/www.testpreptraining.ai\/tutorial\/#website","url":"https:\/\/www.testpreptraining.ai\/tutorial\/","name":"Testprep Training Tutorials","description":"","publisher":{"@id":"https:\/\/www.testpreptraining.ai\/tutorial\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.testpreptraining.ai\/tutorial\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.testpreptraining.ai\/tutorial\/#organization","name":"Testprep Training","url":"https:\/\/www.testpreptraining.ai\/tutorial\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.testpreptraining.ai\/tutorial\/#\/schema\/logo\/image\/","url":"https:\/\/www.testpreptraining.com\/tutorial\/wp-content\/uploads\/2020\/07\/tpt-logo-6.png","contentUrl":"https:\/\/www.testpreptraining.com\/tutorial\/wp-content\/uploads\/2020\/07\/tpt-logo-6.png","width":583,"height":153,"caption":"Testprep Training"},"image":{"@id":"https:\/\/www.testpreptraining.ai\/tutorial\/#\/schema\/logo\/image\/"}}]}},"_links":{"self":[{"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/pages\/4797","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/comments?post=4797"}],"version-history":[{"count":1,"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/pages\/4797\/revisions"}],"predecessor-version":[{"id":4808,"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/pages\/4797\/revisions\/4808"}],"wp:attachment":[{"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/media?parent=4797"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/categories?post=4797"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/tags?post=4797"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}