Hot questions for Using Azure in azure storage account

Top Java Programmings / Azure / azure storage account

Question:

Is there a way to connect to a storage account or container only using a SAS token and the endpoint for that storage account? I’ve already done the demo/sample and it is using the connection string (which works) but I don’t see how to connect to my storage using only the SAS? Is there any example how to do this in java?


Answer:

I generate SAS Token for my specific blob on the portal then download it successfully via the token.

sample code:

import com.microsoft.azure.storage.blob.CloudBlockBlob;

import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.URI;

public class DownloadBlobSAS {

    public static final String blobSasToken = "***";

    public static void main(String[] args) {
        try {
            CloudBlockBlob sasBlob = new CloudBlockBlob(new URI(blobSasToken));

            InputStream input =  sasBlob.openInputStream();
            InputStreamReader inr = new InputStreamReader(input, "UTF-8");
            String utf8str = org.apache.commons.io.IOUtils.toString(inr);
            System.out.println(utf8str);

            System.out.println("print done");

        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

Output Result:

If you want to generate SAS token, you need to provide connection string first to access container or blob.

Then you could use generateSharedAccessSignature() method to create SAS token.You could refer to the sample code:

SharedAccessBlobPolicy sp = createSharedAccessPolicy(
                EnumSet.of(SharedAccessBlobPermissions.READ, SharedAccessBlobPermissions.LIST), 300);
        BlobContainerPermissions perms = new BlobContainerPermissions();

        perms.getSharedAccessPolicies().put("readperm", sp);
        this.container.uploadPermissions(perms);
        Thread.sleep(30000);

        CloudBlockBlob sasBlob = new CloudBlockBlob(new URI(this.blob.getUri().toString() + "?"
                + this.blob.generateSharedAccessSignature(null, "readperm")));

Hope it helps you.

Question:

Our blob storage account structure:

container name: simple

inside this container we have blobs:

aa/one.zip
aa/two.zip
bb/ss.zip
bb/dd.zip

Is it possible to generate SAS with write permission for aa "directory", but no access for bb "directory"?

With Amazon AWS we can easily create restrictions based on object/blob prefix name, but I can't find similar functionality in azure storage sdk for java.


Answer:

As of today, it is not possible to do so at the folder level because as such there's no folder in Azure Blob Storage.

You can either create a SAS at a container level or at the blob level. If you want your users to only upload files with certain names, you can very well create a SAS with write permission on the blob name. Blob need not be present when you're creating a SAS.

Thus when your users upload a file using this SAS, it will be saved as a blob that you have chosen to name. So what you will do is create a SAS for a blob named bb/ss.zip and give it to a user. When the user uploads the file, it will be saved as bb/ss.zip in your blob container.

Question:

We can set if secure tansfer is enabled or not when creating storage account in azure portal but is there a way to check if a storage account is enabled or not through api/sdk?


Answer:

You can get this information in multiple ways depending on your preference:

  • Azure CLI
  • Azure PowerShell
  • .Net Fluent SDK
  • Java SDK
  • and other SDKs that exist

Here are the snippets reduced to only display the value of the field. Replace the placeholders <..> with their real values. The storage account will be identified by resource group name and storage account name. The solutions presume that you now how to authenticate.

Azure CLI
az storage account show --resource-group <ResourceGroupName> --name <StorageAccountName> --query enableHttpsTrafficOnly
Azure PowerShell
Get-AzStorageAccount -ResourceGroupName <ResourceGroupName> -Name <StorageAccountName> | Select-Object EnableHttpsTrafficOnly
Fluent SDK (C# console app)
//requires references for Microsoft.Azure.Management.Fluent and Microsoft.Azure.Management.Storage.Fluent

IAzure myAzure = Azure.Authenticate("azure.auth").WithDefaultSubscription();    
Console.WriteLine(myAzure.StorageAccounts.GetByResourceGroup("<ResourceGroupName>", "<StorageAccountName>").Inner.EnableHttpsTrafficOnly);
Java SDK

There is also an SDK for Java and it seems to work in an identical fashion. Looking at the code, you should be able to achieve the same, as with the .NET SDK.

Here is a link for storage account management samples with Java and the SDK: Java SDK Storage Account Management Go to the section List storage accounts and adapt the sample similar to my C# code (apply getByResourceGroup(...) and .Inner.enableHttpsTrafficOnly

I hope this is of some help.

Question:

Is there a way to get the total size of the blobs in a storage account blobs container without iterating over them and summing up the length?

Preferably from the Java API or CLI.

We are required to get the up to date size on a regular basis and will have a large amount of blobs in the container, and we will have a lot of containers, therefore its not feasible to iterate over the blobs every time to get the sizes.


Answer:

I am afraid it's not possible to bypass blob listing and summing up operation to get the size of container(i.e. blobs inside it). All SDKs are based on REST API, which includes all operation interfaces exposed to us. According to those APIs, we can only get the list of blobs and sum up their content length.

See an official example. There's a Calculate size on Azure portal, right click on our container and click Container properties, we can see panel as below. Try to catch the http traffic, we can see it calls list blob operation. The Learn more link shows a powershell script to realize the unavoidable process. (Note that once blob count is over 5000, Calculate size displays Blob Count>5000 and Size>XXMiB).

Question:

I have a requirement to fetch all storage accounts from my azure account and list all of their properties. I have configured azure object and using below call to get all storageAccounts.

Azure azure = Azure
                        .configure()
                        .withLogLevel(LogLevel.NONE)
                        .authenticate(credFile)
                        .withDefaultSubscription();                     

StorageAccounts storageAccounts  =  azure.storageAccounts();

I see that in com.microsoft.azure.management.storage.StorageAccounts interface there are many methods, but I am looking for a model object to fetch and list its details just like in case of aws we have com.amazonaws.services.s3.model.Bucket

My question is: which will be the model object in case of azure for the StorageAccounts ?


Answer:

You could use method .list() to list storage accounts in your subscription. Please check SDK in this link. You could use the following code to get storage accounts properties.

 List<StorageAccount> accounts = azure.storageAccounts().list();
            for (StorageAccount sa : accounts) {
                System.out.println("Storage Account " + sa.name() + " created @ " + sa.creationTime());
            } 

You also could check this example to manage your storage accounts.

Question:

I am trying to get the details of azure storage account containers from rest API using the credentials of a an registered application in azure. I have built the authentication header parameter and when I am calling the rest API I get this error which says Audience validation failed. Audience did not match.

19:47:16.826 [main] INFO - <-- 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. https://shagundonotdeleteaccount.blob.core.windows.net/?comp=list (288 ms, 426-byte body) 19:47:16.828 [main] INFO - 426-byte body: ?<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. RequestId:deb7a983-401e-008d-30e7-0c884e000000 Time:2020-04-07T14:17:14.0081901Z</Message><AuthenticationErrorDetail>Audience validation failed. Audience did not match.</AuthenticationErrorDetail></Error> 19:47:16.828 [main] INFO - <-- END HTTP

This is my code:

        ApplicationTokenCredentials credentials = new ApplicationTokenCredentials("clientID",
                "domain", "secret", AzureEnvironment.AZURE);
RestClient restClient = new RestClient.Builder()
                .withBaseUrl(AzureEnvironment.AZURE, AzureEnvironment.Endpoint.RESOURCE_MANAGER)
                .withSerializerAdapter(new AzureJacksonAdapter())
                .withReadTimeout(150, TimeUnit.SECONDS)
                .withLogLevel(LogLevel.BODY)
                .withResponseBuilderFactory(new AzureResponseBuilder.Factory())
                .withCredentials(credentials).build();
   Azure azure = Azure.authenticate(restClient, credentials.domain()).withDefaultSubscription();
             Base64 base64 = new Base64();

  for (StorageAccount account : azure.storageAccounts().list()) {

            URL url = new URL("https://" + account.name() + ".blob.core.windows.net/?comp=list");
            StorageAccountKey key = account.getKeys().get(0);
            SimpleDateFormat fmt = new SimpleDateFormat("EEE, dd MMM yyyy HH:mm:ss");
            fmt.setTimeZone(TimeZone.getTimeZone("GMT"));
            String date = fmt.format(Calendar.getInstance().getTime()) + " GMT";

            StringBuilder sb = new StringBuilder();
            sb.append("GET\n"); // method
            sb.append('\n'); // md5 (optional)
            sb.append('\n'); // content type
            sb.append('\n'); // legacy date
            sb.append("x-ms-date:" + date + '\n'); // headers
            sb.append("x-ms-version:2017-11-09\n");
            sb.append("/" + account.name() + url.getPath() + "?comp=list");

            Mac mac = Mac.getInstance("HmacSHA256");
            mac.init(new SecretKeySpec(base64.decode(key.value()), "HmacSHA256"));

            String authKey = new String(base64.encode(mac.doFinal(sb.toString().getBytes("UTF-8"))));
            String auth = "SharedKeyLite " + account.name() + ":" + authKey;
         OkHttpClient httpClient = restClient.httpClient();
         Request request = new Request.Builder()
                    .url(url)
                    //                    .addHeader("content-type", "application/json")
                    //                    .addHeader("cache-control", "no-cache")
                    .addHeader("x-ms-version", "2017-11-09")
                    .addHeader("x-ms-date", date)
                    .addHeader("Authorization", auth)
                    .get()
                    .build();

            okhttp3.Response response = httpClient.newCall(request).execute();
            httpClient.newCall(request);

            if (!response.isSuccessful()) {
                throw new RuntimeException("Request Failed" + response.code() + "\n" + response.message());
            }
            JSONParser parser = new JSONParser();
            String apiResponse = response.body().string();

            if (apiResponse != null) {
                org.json.simple.JSONObject responseJSON = (org.json.simple.JSONObject) parser.parse(apiResponse);
                System.out.println(responseJSON);

            }

        }

This is the authsignstring

GET



x-ms-date:Tue, 07 Apr 2020 14:17:13 GMT
x-ms-version:2017-11-09
/mystorageaccount/?comp=list

I have tried this as well:

StringBuilder sb = new StringBuilder();
        sb.append("GET\n"); // method
        sb.append('\n'); // content encoding
        sb.append('\n'); // content language
        sb.append('\n'); // content length
        sb.append('\n'); // md5 (optional)
        sb.append('\n'); // content type
        sb.append('\n'); // legacy date
        sb.append('\n'); // if-modified-since
        sb.append('\n'); // if-match
        sb.append('\n'); // if-none-match
        sb.append('\n'); // if-unmodified-since
        sb.append('\n'); // range
        sb.append("x-ms-date:" + date + '\n'); // headers
        sb.append("x-ms-version:2017-11-09\n");

        sb.append("/" + account.name() + "/" + "\ncomp:list");

and

String auth = "SharedKey " + account.name() + ":" + authKey;

Can you help me with this? I am stuck here from a long time. I want the JSON response for the list of containers.


Answer:

If you want to use SharedKeyLite auth to call Azur blob rest api, please refer to the following code

 String StorageAccountName = "blobstorage0516";
         String StorageAccountKey = "";
        URL url = new URL("https://" + StorageAccountName + ".blob.core.windows.net/?comp=list");
        SimpleDateFormat fmt = new SimpleDateFormat("EEE, dd MMM yyyy HH:mm:ss");
        fmt.setTimeZone(TimeZone.getTimeZone("GMT"));
        String date = fmt.format(Calendar.getInstance().getTime()) + " GMT";
        StringBuilder sb = new StringBuilder();
        sb.append("GET\n"); // method
        sb.append("\n") ;//Content-MD5
        sb.append("\n") ;//Content-Type
        sb.append("\n") ;//data
        sb.append("x-ms-date:" + date + '\n'); // headers
        sb.append("x-ms-version:2017-11-09\n");
        sb.append("/" + StorageAccountName + url.getPath() + "?comp=list");
 System.out.println(sb.toString()); // print stringtosign
SecretKeySpec secretKey = new SecretKeySpec(Base64.getDecoder().decode(StorageAccountKey), "HmacSHA256");
        Mac sha256HMAC = Mac.getInstance("HmacSHA256");
        sha256HMAC.init(secretKey);
        String authKey=Base64.getEncoder().encodeToString(sha256HMAC.doFinal(sb.toString().getBytes("UTF-8")));
        String auth = "SharedKeyLite " + StorageAccountName + ":" + authKey;
        System.out.println(auth);
        OkHttpClient client = new OkHttpClient().newBuilder().build();
        Request request = new Request.Builder()
                             .url(url)
                             .method("GET", null)
                             .addHeader("x-ms-version", "2017-11-09")
                             .addHeader("x-ms-date", date)
                             .addHeader("Authorization", auth)
                             .build();
        Response response = client.newCall(request).execute();
        if(response.isSuccessful()){

            System.out.println(response.body().string());
        }


Update

When we use RestClient to call the Azure rest api, it will use Azure AD access token to do auth. But the restclient use wrong resource to get Azure AD access token. It uses https://management.core.windows.net/ but we needs https:\\storage.azure.com\

My test code

 OkHttpClient httpClient = restClient.httpClient().newBuilder().build();
        //OkHttpClient client = new OkHttpClient().newBuilder().build();
        Request request = new Request.Builder()
                .url(url)
                .method("GET", null)
                .addHeader("x-ms-version", "2017-11-09")
                //.addHeader("x-ms-date", date)
                .build();
        //Response response = client.newCall(request).execute();

        okhttp3.Response response1 = httpClient.newCall(request).execute();



            System.out.println(response1.body().string());

Analyze access token via the link

So if you want to use sharekey to call the api, please create a new httoclient.

Question:

For instance I can authenticate through graph api by getaccesstokencredentials(username, password) Can I use this token to access Azure? Current we can use usertokencredentials and applicationtokencredentials from management library then once done you can create instance of azure class. Azure azure = Azure.authenticate(credentials).withdefaultsubscription. I'm wondering if we can use the token from getaccesstokencredentials instead of usertokentcredentials and applicationtokencredentials


Answer:

We cannot use the the same access token to call graph api and call api to manage Azure resource. Because the resource url for graph api ishttps://graph.microsoft.com/ but the resource url for Azure management rest api is https://management.azure.com/. For more details, please refer to https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-manager-api-authentication.

Besides, regarding how to use Azure AD to access Azure storage, please refer to the following steps:

  1. Add role assignment to your principal.

  1. Get token.

    public static String getToken() throws Exception {
        String TENANT_ID = "your tenant id or name, e4c9*-*-*-*-*57fb";
        String AUTHORITY = "https://login.microsoftonline.com/" + TENANT_ID;
        String CLIENT_ID = "your application id, dc17*-*-*-*a5e7";
        String CLIENT_SECRET = "the secret, /pG*32";
        String RESOURCE = "https://storage.azure.com/";
        String ACCESS_TOKEN = null;
        ExecutorService service = Executors.newFixedThreadPool(1);
        AuthenticationContext context = null;
        try {
            context = new AuthenticationContext(AUTHORITY, false, service);
            ClientCredential credential = new ClientCredential(CLIENT_ID, CLIENT_SECRET);
            Future<AuthenticationResult> future = context.acquireToken(RESOURCE, credential, null);
            ACCESS_TOKEN = future.get().getAccessToken();
        } catch (InterruptedException e) {
            e.printStackTrace();
        } catch (ExecutionException e) {
            e.printStackTrace();
        } catch (MalformedURLException e) {
            e.printStackTrace();
        } finally {
            service.shutdown();
        }
        return ACCESS_TOKEN;
    }
    
  2. Access blob.

    public static void main(String[] args) throws Exception {
        String token = getToken();
        StorageCredentialsToken credentialsToken = new StorageCredentialsToken("storagetest789", token);
        CloudBlobClient blobClient = new CloudBlobClient(new URI("https://storagetest789.blob.core.windows.net/"), credentialsToken);
        CloudBlobContainer blobContainer = blobClient.getContainerReference("pub");
        CloudBlockBlob blockBlob = blobContainer.getBlockBlobReference("test1.txt");
        blockBlob.uploadText("mytest");
    }
    

For more details, please refer to https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad.

Question:

I am creating an application where I do interaction with files in Azure storage account with java services. I want to open uploaded files in office 365 so that I can edit them. How I can do that?

Is there any reference code available for the same to do it with service from java?


Answer:

To view the Office files (Word, Excel, PowerPoint) in Office 365 online document viewer, you can create a link to your blob by using Online Doc Viewer utility. Essentially the link that will be created will be:

https://view.officeapps.live.com/op/view.aspx?src=<Encoded URL of Blob>

Please note that:

  • The documents you open through this utility will be opened in read-only mode. You won't be able to edit these documents unless you save those documents in One Drive.
  • If the blob container containing the documents has private ACL, then you will need to create a Shared Access Signature with at least Read permission to view the documents.