{"id":1064,"date":"2019-07-09T10:51:20","date_gmt":"2019-07-09T10:51:20","guid":{"rendered":"https:\/\/www.testpreptraining.com\/tutorial\/?page_id=1064"},"modified":"2020-05-01T09:56:36","modified_gmt":"2020-05-01T09:56:36","slug":"aws-collection-system","status":"publish","type":"page","link":"https:\/\/www.testpreptraining.ai\/tutorial\/aws-certified-big-data-specialty\/aws-collection-system\/","title":{"rendered":"Select a Collection System that handles the frequency of data change and type of data being ingested"},"content":{"rendered":"\n
Data ingestion and synchronization into a big data\nenvironment is harder than most people think. \nLoading large volumes of data at high speed and managing the incremental\ningestion and synchronization of data at scale into an on premise or cloud data\nlake can present significant technical challenges.<\/p>\n\n\n\n
Amazon Kinesis Firehose<\/h2>\n\n\n\n
Amazon Kinesis Firehose is a fully managed service for\ndelivering real-time streaming data directly to Amazon S3. Kinesis Firehose\nautomatically scales to match the volume and throughput of streaming data, and\nrequires no ongoing administration. Kinesis Firehose can also be configured to\ntransform streaming data before it\u2019s stored in Amazon S3. Its transformation\ncapabilities include compression, encryption, data batching, and Lambda\nfunctions.<\/p>\n\n\n\n
Kinesis Firehose can compress data before it\u2019s stored\nin Amazon S3. It currently supports GZIP, ZIP, and SNAPPY compression formats.\nGZIP is the preferred format because it can be used by Amazon Athena, Amazon\nEMR, and Amazon Redshift. Kinesis Firehose encryption supports Amazon S3\nserver-side encryption with AWS Key Management Service (AWS KMS) for encrypting\ndelivered data in Amazon S3. You can choose not to encrypt the data or to\nencrypt with a key from the list of AWS KMS keys that you own (see the section\nEncryption with AWS KMS). Kinesis Firehose can concatenate multiple incoming\nrecords, and then deliver them to Amazon S3 as a single S3 object. This is an\nimportant capability because it reduces Amazon S3 transaction costs and\ntransactions per second load.<\/p>\n\n\n\n
Finally, Kinesis Firehose can invoke Lambda functions to transform incoming source data and deliver it to Amazon S3. Common transformation functions include transforming Apache Log and Syslog formats to standardized JSON and\/or CSV formats. The JSON and CSV formats can then be directly queried using Amazon Athena. If using a Lambda data transformation, you can optionally back up raw source data to another S3 bucket.<\/p>\n\n\n\n
<\/figure><\/div>\n\n\n\n
AWS Snowball<\/h2>\n\n\n\n
You can use AWS Snowball to securely and efficiently\nmigrate bulk data from on-premises storage platforms and Hadoop clusters to S3\nbuckets. After you create a job in the AWS Management Console, a Snowball\nappliance will be automatically shipped to you. After a Snowball arrives,\nconnect it to your local network, install the Snowball client on your\non-premises data source, and then use the Snowball client to select and\ntransfer the file directories to the Snowball device. The Snowball client uses\nAES-256-bit encryption. Encryption keys are never shipped with the Snowball\ndevice, so the data transfer process is highly secure. After the data transfer\nis complete, the Snowball\u2019s E Ink shipping label will automatically update.\nShip the device back to AWS. Upon receipt at AWS, your data is then transferred\nfrom the Snowball device to your S3 bucket and stored as S3 objects in their\noriginal\/native format. Snowball also has an HDFS client, so data may be\nmigrated directly from Hadoop clusters into an S3 bucket in its native format.<\/p>\n\n\n\n
AWS Storage Gateway<\/h2>\n\n\n\n
AWS Storage Gateway can be used to integrate legacy on-premises data processing platforms with an Amazon S3-based data lake. The File Gateway configuration of Storage Gateway offers on-premises devices and applications a network file share via an NFS connection. Files written to this mount point are converted to objects stored in Amazon S3 in their original format without any proprietary modification. This means that you can easily integrate applications and platforms that don\u2019t have native Amazon S3 capabilities\u2014such as on-premises lab equipment, mainframe computers, databases, and data warehouses\u2014with S3 buckets, and then use tools such as Amazon EMR or Amazon Athena to process this data. <\/p>\n\n\n\n
![\"select](\"https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2019\/07\/select-a-collection-system-that-handles-the-frequency-of-data-change-and-type-of-data-being-ingested.png\")
<\/figure><\/div>\n\n\n\n