databrick add data that resides in AWS s3


To connect to your data source, you have placed in an s3 bucket, goto DataBricks -> Catalog ->  


Then



Then click on "+" to start the integration process




Then select AWS Quickstart. This will help you to create the necessary role and permission in your AWS stack.




Then click on "Next".  Provide your s3 bucket name and then you can click on Generate PAT token. Make sure you copy it and then click on "Launch Quickstart" - and you need to paste this information into the AWS cloud formation. 





The rest of the information would have been populated in AWS console and all you need to do is paste in your PAT token. Click on next. Once it is completed - about 5-10 minutes, refresh your databrick workspace and you should be able to see all your files there. 












Comments

Post a Comment

Popular posts from this blog

The specified initialization vector (IV) does not match the block size for this algorithm

Image
If you're getting  exception when trying to assigned key to a symmetric key and getting error message below :- The specified initialization vector (IV) does not match the block size for this algorithm And your code probably look like something below :- Then you need to go into debug mode and start looking into the supported size for IV as shown in diagram below :- As you can see, IV is 16 bytes, so you need to provide 16 bytes. The same goes for Key field too, if you're adding anything to it. As long as you provide a valid key size, then we're good and you're code will be ready.
Read more