{"id":1080,"date":"2019-07-09T10:57:14","date_gmt":"2019-07-09T10:57:14","guid":{"rendered":"https:\/\/www.testpreptraining.com\/tutorial\/?page_id=1080"},"modified":"2020-05-01T10:06:05","modified_gmt":"2020-05-01T10:06:05","slug":"determine-and-optimize-the-operational-characteristics-of-the-storage-solution","status":"publish","type":"page","link":"https:\/\/www.testpreptraining.ai\/tutorial\/aws-certified-big-data-specialty\/determine-and-optimize-the-operational-characteristics-of-the-storage-solution\/","title":{"rendered":"Determine and Optimize the Operational Characteristics of the Storage Solution"},"content":{"rendered":"\n
The following questions can help you segment data\nwithin each of your workloads and determine your storage requirements:<\/p>\n\n\n\n
How often and how quickly do you need to access\nyour data? AWS offers storage options and pricing tiers for frequently\naccessed, less frequently accessed, and infrequently accessed data.<\/li>
Does your data store require high IOPS or\nthroughput? AWS provides categories of storage that are optimized for\nperformance and throughput. Understanding IOPS and throughput requirements will\nhelp you provision the right amount of storage and avoid overpaying.<\/li>
How critical (durable) is your data? Critical or\nregulated data needs to be retained at almost any expense and tends to be\nstored for a long time.<\/li>
How sensitive is your data? Highly sensitive\ndata needs to be protected from accidental and malicious changes, not just data\nloss or corruption. Durability, cost, and security are equally important to\nconsider.<\/li>
How large is your data set? Knowing the total\nsize of the data set helps in estimating storage capacity and cost.<\/li>
How transient is your data? Transient data is\nshort-lived and typically does not require high durability. (Note: Durability\nrefers to average annual expected data loss.) Clickstream and Twitter data are\ngood examples of transient data.<\/li>
How much are you prepared to pay to store the\ndata? Setting a budget for data storage will inform your decisions about\nstorage options.<\/li><\/ul>\n\n\n\n
Optimize Amazon S3 Storage<\/h2>\n\n\n\n
Amazon S3 lets you analyze data access patterns, create\ninventory lists, and configure lifecycle policies. You can set up rules to\nautomatically move data objects to cheaper S3 storage tiers as objects are\naccessed less frequently or to automatically delete objects after an expiration\ndate. To manage storage data most effectively, you can use tagging to\ncategorize your S3 objects and filter on these tags in your data lifecycle\npolicies.<\/p>\n\n\n\n
To determine when to transition data to another storage\nclass, you can use Amazon S3 analytics storage class analysis to analyze\nstorage access patterns. Analyze all the objects in a bucket or use an object\ntag or common prefix to filter objects for analysis. If you observe infrequent\naccess patterns of a filtered data set over time, you can use the information\nto choose a more appropriate storage class, improve lifecycle policies, and\nmake predictions around future usage and growth.<\/p>\n\n\n\n
Another management tool is Amazon S3 Inventory, which\naudits and reports on the replication and encryption status of your S3 objects\non a weekly or monthly basis. This feature provides CSV output files that list\nobjects and their corresponding metadata and lets you configure multiple\ninventory lists for a single bucket, organized by different S3 metadata tags.\nYou can also query Amazon S3 inventory using standard SQL by using Amazon\nAthena, Amazon Redshift Spectrum, and other tools, such as Presto, Apache Hive,\nand Apace Spark.<\/p>\n\n\n\n
Amazon S3 can also publish storage, request, and data\ntransfer metrics to Amazon CloudWatch. Storage metrics are reported daily, are\navailable at one-minute intervals for granular visibility, and can be collected\nand reported for an entire bucket or a subset of objects (selected via prefix\nor tags).<\/p>\n\n\n\n
With all the information these storage management tools\nprovide, you can create policies to move less-frequently-accessed data S3 data\nto cheaper storage tiers for considerable savings. For example, by moving data\nfrom Amazon S3 Standard to Amazon S3 Standard-IA, you can save up to 60% (on a\nper-gigabyte basis) of Amazon S3 pricing. By moving data that is at the end of\nits lifecycle and accessed on rare occasions to Amazon Glacier, you can save up\nto 80% of Amazon S3 pricing. <\/p>\n\n\n\n
Optimize Amazon EBS Storage<\/h2>\n\n\n\n
With Amazon EBS, it\u2019s important to keep in mind that\nyou are paying for provisioned capacity and performance\u2014even if the volume is\nunattached or has very low write activity. To optimize storage performance and\ncosts for Amazon EBS, monitor volumes periodically to identify ones that are\nunattached or appear to be underutilized or overutilized, and adjust\nprovisioning to match actual usage.<\/p>\n\n\n\n
AWS offers tools that can help you optimize block\nstorage. Amazon CloudWatch automatically collects a range of data points for\nEBS volumes and lets you set alarms on volume behavior. AWS Trusted Advisor is\nanother way for you to analyze your infrastructure to identify unattached,\nunderutilized, and overutilized EBS volumes. Third-party tools, such as\nCloudability, can also provide insight into performance of EBS volumes.<\/p>\n\n\n\n
An easy way to reduce wasted spend is to find and\ndelete unattached volumes. However, when EC2 instances are stopped or\nterminated, attached EBS volumes are not automatically deleted and will continue\nto accrue charges since they are still operating. To find unattached EBS\nvolumes, look for volumes that are available, which indicates that they are not\nattached to an EC2 instance. You can also look at network throughput and IOPS\nto see whether there has been any volume activity over the previous two weeks.\nIf the volume is in a nonproduction environment, hasn\u2019t been used in weeks, or\nhasn\u2019t been attached in a month, there is a good chance you can delete it.<\/p>\n\n\n\n
Before deleting a volume, store an Amazon EBS snapshot\n(a backup copy of an EBS volume) so that the volume can be quickly restored\nlater if needed. You can automate the process of deleting unattached volumes by\nusing AWS Lambda functions with Amazon CloudWatch.<\/p>\n\n\n\n
Resize or Change\nthe EBS Volume Type<\/strong><\/p>\n\n\n\n
Another way to optimize storage costs is to identify\nvolumes that are underutilized and downsize them or change the volume type.\nMonitor the read-write access of EBS volumes to determine if throughput is low.\nIf you have a current-generation EBS volume attached to a current-generation\nEC2 instance type, you can use the elastic volumes feature to change the size\nor volume type, or (for an SSD io1 volume) adjust IOPS performance without\ndetaching the volume.<\/p>\n\n\n\n
The following tips can help you optimize your EBS\nvolumes:<\/p>\n\n\n\n
For General Purpose SSD gp2 volumes, you\u2019ll want\nto optimize for capacity so that you\u2019re paying only for what you use.<\/li>
With Provisioned IOPS SSD io1 volumes, pay close\nattention to IOPS utilization rather than throughput, since you pay for IOPS\ndirectly. Provision 10\u201320% above maximum IOPS utilization.<\/li>
You can save by reducing provisioned IOPS or by\nswitching from a Provisioned IOPS SSD io1 volume type to a General Purpose SSD\ngp2 volume type.<\/li>
If the volume is 500 gigabytes or larger,\nconsider converting to a Cold HDD sc1 volume to save on your storage rate.<\/li>
You can always return a volume to its original\nsettings if needed.<\/li><\/ul>\n\n\n\n
If you have a backup policy that takes EBS volume\nsnapshots daily or weekly, you will quickly accumulate snapshots. Check for\nstale snapshots that are over 30 days old and delete them to reduce storage\ncosts. Deleting a snapshot has no effect on the volume. You can use the AWS\nManagement Console or AWS Command Line Interface (CLI) for this purpose or\nthird-party tools such as Skeddly. <\/p>\n\n\n\n
Storage Optimization is an Ongoing Process<\/h2>\n\n\n\n
Maintaining a storage architecture that is both\nright-sized and right-priced is an ongoing process. To get the most efficient\nuse of your storage spend, you should optimize storage on a monthly basis. You\ncan streamline this effort by:<\/p>\n\n\n\n
Establishing an ongoing mechanism for optimizing storage and setting up storage policies.<\/li>
Monitoring costs closely using AWS cost and reporting tools, such as Cost Explorer, budgets, and detailed billing reports in the Billing and Cost Management console.<\/li>
Enforcing Amazon S3 object tagging and establishing S3 lifecycle policies to continually optimize data storage throughout the data lifecycle.<\/li><\/ul>\n\n\n\n
The following questions can help you segment data within each of your workloads and determine your storage requirements: How often and how quickly do you need to access your data? AWS offers storage options and pricing tiers for frequently accessed, less frequently accessed, and infrequently accessed data. Does your data store require high IOPS or…<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":1031,"menu_order":7,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_acf_changed":false,"footnotes":""},"categories":[2],"tags":[],"class_list":["post-1080","page","type-page","status-publish","hentry","category-amazon-aws"],"acf":[],"yoast_head":"\n
Determine and Optimize the Operational Characteristics of the Storage Solution - Testprep Training Tutorials<\/title>\n<meta name=\"description\" content=\"Determine and optimize the operational characteristics of the storage solution tutorial, notes\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.testpreptraining.ai\/tutorial\/aws-certified-big-data-specialty\/determine-and-optimize-the-operational-characteristics-of-the-storage-solution\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Determine and Optimize the Operational Characteristics of the Storage Solution - Testprep Training Tutorials\" \/>\n<meta property=\"og:description\" content=\"Determine and optimize the operational characteristics of the storage solution tutorial, notes\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.testpreptraining.ai\/tutorial\/aws-certified-big-data-specialty\/determine-and-optimize-the-operational-characteristics-of-the-storage-solution\/\" \/>\n<meta property=\"og:site_name\" content=\"Testprep Training Tutorials\" \/>\n<meta property=\"article:modified_time\" content=\"2020-05-01T10:06:05+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2019\/07\/determine-and-optimize-the-operational-characteristics-of-the-storage-solution.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/aws-certified-big-data-specialty\/determine-and-optimize-the-operational-characteristics-of-the-storage-solution\/\",\"url\":\"https:\/\/www.testpreptraining.ai\/tutorial\/aws-certified-big-data-specialty\/determine-and-optimize-the-operational-characteristics-of-the-storage-solution\/\",\"name\":\"Determine and Optimize the Operational Characteristics of the Storage Solution - Testprep Training Tutorials\",\"isPartOf\":{\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/#website\"},\"datePublished\":\"2019-07-09T10:57:14+00:00\",\"dateModified\":\"2020-05-01T10:06:05+00:00\",\"description\":\"Determine and optimize the operational characteristics of the storage solution tutorial, notes\",\"breadcrumb\":{\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/aws-certified-big-data-specialty\/determine-and-optimize-the-operational-characteristics-of-the-storage-solution\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.testpreptraining.ai\/tutorial\/aws-certified-big-data-specialty\/determine-and-optimize-the-operational-characteristics-of-the-storage-solution\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/aws-certified-big-data-specialty\/determine-and-optimize-the-operational-characteristics-of-the-storage-solution\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.testpreptraining.ai\/tutorial\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"AWS Certified Big Data Specialty\",\"item\":\"https:\/\/www.testpreptraining.ai\/tutorial\/aws-certified-big-data-specialty\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Determine and Optimize the Operational Characteristics of the Storage Solution\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/#website\",\"url\":\"https:\/\/www.testpreptraining.ai\/tutorial\/\",\"name\":\"Testprep Training Tutorials\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.testpreptraining.ai\/tutorial\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/#organization\",\"name\":\"Testprep Training\",\"url\":\"https:\/\/www.testpreptraining.ai\/tutorial\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.testpreptraining.com\/tutorial\/wp-content\/uploads\/2020\/07\/tpt-logo-6.png\",\"contentUrl\":\"https:\/\/www.testpreptraining.com\/tutorial\/wp-content\/uploads\/2020\/07\/tpt-logo-6.png\",\"width\":583,\"height\":153,\"caption\":\"Testprep Training\"},\"image\":{\"@id\":\"https:\/\/www.testpreptraining.ai\/tutorial\/#\/schema\/logo\/image\/\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Determine and Optimize the Operational Characteristics of the Storage Solution - Testprep Training Tutorials","description":"Determine and optimize the operational characteristics of the storage solution tutorial, notes","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.testpreptraining.ai\/tutorial\/aws-certified-big-data-specialty\/determine-and-optimize-the-operational-characteristics-of-the-storage-solution\/","og_locale":"en_US","og_type":"article","og_title":"Determine and Optimize the Operational Characteristics of the Storage Solution - Testprep Training Tutorials","og_description":"Determine and optimize the operational characteristics of the storage solution tutorial, notes","og_url":"https:\/\/www.testpreptraining.ai\/tutorial\/aws-certified-big-data-specialty\/determine-and-optimize-the-operational-characteristics-of-the-storage-solution\/","og_site_name":"Testprep Training Tutorials","article_modified_time":"2020-05-01T10:06:05+00:00","og_image":[{"url":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2019\/07\/determine-and-optimize-the-operational-characteristics-of-the-storage-solution.png"}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.testpreptraining.ai\/tutorial\/aws-certified-big-data-specialty\/determine-and-optimize-the-operational-characteristics-of-the-storage-solution\/","url":"https:\/\/www.testpreptraining.ai\/tutorial\/aws-certified-big-data-specialty\/determine-and-optimize-the-operational-characteristics-of-the-storage-solution\/","name":"Determine and Optimize the Operational Characteristics of the Storage Solution - Testprep Training Tutorials","isPartOf":{"@id":"https:\/\/www.testpreptraining.ai\/tutorial\/#website"},"datePublished":"2019-07-09T10:57:14+00:00","dateModified":"2020-05-01T10:06:05+00:00","description":"Determine and optimize the operational characteristics of the storage solution tutorial, notes","breadcrumb":{"@id":"https:\/\/www.testpreptraining.ai\/tutorial\/aws-certified-big-data-specialty\/determine-and-optimize-the-operational-characteristics-of-the-storage-solution\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.testpreptraining.ai\/tutorial\/aws-certified-big-data-specialty\/determine-and-optimize-the-operational-characteristics-of-the-storage-solution\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.testpreptraining.ai\/tutorial\/aws-certified-big-data-specialty\/determine-and-optimize-the-operational-characteristics-of-the-storage-solution\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.testpreptraining.ai\/tutorial\/"},{"@type":"ListItem","position":2,"name":"AWS Certified Big Data Specialty","item":"https:\/\/www.testpreptraining.ai\/tutorial\/aws-certified-big-data-specialty\/"},{"@type":"ListItem","position":3,"name":"Determine and Optimize the Operational Characteristics of the Storage Solution"}]},{"@type":"WebSite","@id":"https:\/\/www.testpreptraining.ai\/tutorial\/#website","url":"https:\/\/www.testpreptraining.ai\/tutorial\/","name":"Testprep Training Tutorials","description":"","publisher":{"@id":"https:\/\/www.testpreptraining.ai\/tutorial\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.testpreptraining.ai\/tutorial\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.testpreptraining.ai\/tutorial\/#organization","name":"Testprep Training","url":"https:\/\/www.testpreptraining.ai\/tutorial\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.testpreptraining.ai\/tutorial\/#\/schema\/logo\/image\/","url":"https:\/\/www.testpreptraining.com\/tutorial\/wp-content\/uploads\/2020\/07\/tpt-logo-6.png","contentUrl":"https:\/\/www.testpreptraining.com\/tutorial\/wp-content\/uploads\/2020\/07\/tpt-logo-6.png","width":583,"height":153,"caption":"Testprep Training"},"image":{"@id":"https:\/\/www.testpreptraining.ai\/tutorial\/#\/schema\/logo\/image\/"}}]}},"_links":{"self":[{"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/pages\/1080","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/comments?post=1080"}],"version-history":[{"count":4,"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/pages\/1080\/revisions"}],"predecessor-version":[{"id":5070,"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/pages\/1080\/revisions\/5070"}],"up":[{"embeddable":true,"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/pages\/1031"}],"wp:attachment":[{"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/media?parent=1080"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/categories?post=1080"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.testpreptraining.ai\/tutorial\/wp-json\/wp\/v2\/tags?post=1080"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
![\"determine](\"https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2019\/07\/determine-and-optimize-the-operational-characteristics-of-the-storage-solution.png\")
<\/figure><\/div>\n","protected":false},"excerpt":{"rendered":"