Uploading Large Files into Amazon S3

I recently had to upload some VHDs to Amazon S3 and found myself going beyond the upload limits for the web upload.  In order to accomplish uploading the 50GB files I used the AWS Command Line Interface (CLI).  If you need to know how to install the CLI check here (Install CLI)

1. Open a command prompt As Administrator.

2. Make sure your CLI configuration is up to date.

C:\Windows\System32> aws configure
AWS Access Key ID: yourkey
AWS Secret Access Key: youraccesskey
Default region name [us-east-1]: yourregion
Default output format [None]: json

3. Create your bucket on S3 if it's not already there.

4. Run the following command.  

C:\Windows\System32> aws s3 cp c:\temp\yourfile.vhd s3://yourbucket/VHDs

5. After you run the command the output window will provide you feedback as to how much of your download is complete and your current upload speed. 

C:\Windows\System32> aws s3 cp c:\temp\yourfile.vhd s3://yourbucket/VHDs
Completed 1.1 GiB/50 Gib (14.1 MiB/s) with 1 file(s) remaining

Comments

Post a Comment

Popular posts from this blog

Add User As Local Administrator On Domain Controller

I recently was settting up a new Microsoft SharePoint 2010 machine and had promoted the machine to a domain controller before creating my SharePoint admin accounts.  I needed to add several of my accounts to the local Administrators group.  Unfortunately after you promote a server to a domain controller you can no longer access the GUI for Local Users and Groups.  Instead I had to use the command line to add the users. Open a command promt using the "Run as administrator" function and then run the following command. net localgroup Administrators /add {domain}\{user} Note: do not include the {} brackets.
Read more

Calling Dataverse Web API in PowerShell using Client Credentials

Image
Connecting to Dataverse using PowerShell can be very helpful for data migrations and use within Azure DevOps. Connecting to an instance in a non-interactive way can be tricky though. This article will provide you the links you need for creation and App registration and adding the app user to your environment. You can then utilize the script provided to call Web API requests including ones you define using the new Custom API functionality now available. The script was written so that it is not dependent on any outside libraries such as the Microsoft.Xrm.Tooling connector. This is helpful in situation where involving an outside library will slow down your deployment time by having to be approved in a change control board. If utilizing an outside code library is not a concern you can create a connection to Dataverse utilizing the Microsoft.Xrm.Tooling connector Get-CrmConnection cmdlet. Create App Registration The first thing to do is create an App Registration within Azure AD for ...
Read more

Windows Server 2008R2 VMs Shut Down After 1 to 2 Hours

Image
When created a lab environment to test ADFS 2.0 I utilized the Windows 2008R2 VM baselines distributed by Microsoft.  After a few days I was told that I had to activate.  The VMs included a 180 day license for use but I didn't feel like adding another network adapter into Hyper-V to connect them to the internet.  I started having issues though where the servers would shut down every hour or so.  I though that maybe there was a memory issue and Hyper-V was shutting them down in order to free up RAM.  Turns out that if Server 2008R2 is not activated it automatically shuts down after a period of time.  After connecting the server to the internet and activating them the problem went away. Finally the madness of the unknown shut downs has ended :)
Read more