Hot questions for Using Azure in azure table storage

Top Java Programmings / Azure / azure table storage

Question:

I need to give read-only access to a third party to one of our Azure tables. Is there a clean, simple way of doing that or should I clone the table? If so, how do I clone the table?

Edit:

Here's the documentation on how to do it in Java.


Answer:

There's no built-in mechanism for cloning a table; you'd need to do the copy yourself.

However: Your assertion about Azure Tables not supporting read-only credentials is incorrect. If all you want to do is grant read-only access, you can just create a Shared Access Signature (SAS) for the table in question, granting just query permissions, on the specific table. You can even limit it to a particular range, as defined by partition key + row key.

More info about that is here.

Question:

Using Azure Storage SDK for Java, I am trying to perform basic create, read, update, delete operations on Azure Table Storage as given in the link below: https://azure.microsoft.com/en-us/documentation/articles/storage-java-how-to-use-table-storage/

Sample program for creating a table:

package com.azure.test;
import java.io.UnsupportedEncodingException;
import com.microsoft.azure.storage.*;
import com.microsoft.azure.storage.table.CloudTable;
import com.microsoft.azure.storage.table.CloudTableClient;
import com.microsoft.windowsazure.core.utils.Base64;

public class App 
{

    public static void main( String[] args ) throws StorageException,    UnsupportedEncodingException
{

    String storageConnectionString =
            "DefaultEndpointsProtocol=http;" +
            "AccountName=accountname;" + 
            "AccountKey=storagekey;"+
           "EndpointSuffix=table.core.windows.net"; 

    try
    {
        // Retrieve storage account from connection-string.
        CloudStorageAccount storageAccount =
           CloudStorageAccount.parse(storageConnectionString);

        CloudTableClient tableClient =               storageAccount.createCloudTableClient();

      //Create the table if it doesn't exist.
       String tableName = "MyTable";
       CloudTable cloudTable = tableClient.getTableReference(tableName);
       cloudTable.createIfNotExists();               

    }
    catch (Exception e)
    {
        // Output the stack trace.
        e.printStackTrace();
        System.out.println(e.getMessage());
        }
    }
}

The code seems to be fairly simple to understand. It would connect to the Azure table storage and if a table with a given name does not exist it will create it. But I am getting a InvalidKeyException(full exception pasted below).

java.security.InvalidKeyException: Storage Key is not a valid base64 encoded string. at com.microsoft.azure.storage.StorageCredentials.tryParseCredentials(StorageCredentials.java:68) at com.microsoft.azure.storage.CloudStorageAccount.tryConfigureServiceAccount(CloudStorageAccount.java:408) at com.microsoft.azure.storage.CloudStorageAccount.parse(CloudStorageAccount.java:259) at com.azure.test.App.main(App.java:71)

I am surprised that not many people using Azure Storage are facing this issue. I tried to encode the storage key using and used the encoded key in the connection string but still no use.

String encodedKey=Base64.encode(storageKey.getBytes())

String storageConnectionString =
            "DefaultEndpointsProtocol=http;" +
            "AccountName=accountname" + 
            "AccountKey="+encodedKey+
           "EndpointSuffix=table.core.windows.net;"; 

Can anyone please help me with this? I searched in google a lot and I am able to find one user raised a similar issue on discus but there is no answer provided for that or rather that answer was not helpful.


Answer:

Update:/Resolution of the issue

First of all I ensured that all the properties in connection string are separated by ';' as suggested by Gaurav(below)

It turns out that I have to manually set the proxy settings in my program since my company work machine is using a proxy to connect to the internet.

System.getProperties().put("http.proxyHost", "myproxyHost");
System.getProperties().put("http.proxyPort", "myProxyPort");
System.getProperties().put("http.proxyUser", "myProxyUser");
System.getProperties().put("http.proxyPassword","myProxyPassword");

Updating the proxy settings solved the issue for me.

Question:

I'm looking for simple way of verifying an arbitrary Azure Table connection string that uses a SAS such as the one below using the Azure Storage's Java SDK:

https://example.table.core.windows.net/example?sig=aaabbbcccdddeeefffggghhh%3D&se=2020-01-01T00%3A00%3A00Z&sv=2015-04-05&tn=example&sp=raud

I tried a bunch of different methods exposed by the CloudTable api, but none of them works.

  • CloudTable.exists() throws a StorageException, regardless of whether the credentials are valid

  • getName(), getStorageUri(), getUri(), and other getters - all work locally, regardless of the credentials

  • getServiceClient().downloadServiceProperties() and getServiceClient().getServiceStats() also throw various exceptions, while getServiceClient().getEndpoint() and getServiceClient().getCredentials() and others always work locally.

  • Why don't I just query the Table for a row or two? Well, in many cases I need to verify a SAS that gives only write or update premissions (without delete or read permissions), and I do not want to execute a statement that changes something in the table just to check the credentials.


Answer:

To answer your questions:

CloudTable.exists() throws a StorageException, regardless of whether the credentials are valid

I believe there's a bug with the SDK when using this method with SAS Token. I remember running into the same issue some time back.

getName(), getStorageUri(), getUri(), and other getters - all work locally, regardless of the credentials

These will work as they don't make network call. They simply use the data available to them in the different instance variables and return the data.

getServiceClient().downloadServiceProperties() and getServiceClient().getServiceStats() also throw various exceptions, while getServiceClient().getEndpoint() and getServiceClient().getCredentials() and others always work locally.

In order for getServiceClient().someMethod() to work using SAS, you would need Account SAS instead of Service SAS (which you're using right now).

Why don't I just query the Table for a row or two? Well, in many cases I need to verify a SAS that gives only write or update premissions (without delete or read permissions), and I do not want to execute a statement that changes something in the table just to check the credentials.

One possible way to check the validity of a SAS Token for write operation is to perform a write operation which you know will fail with an error. For example, you can try to insert an entity which is already there. In this case, you should get a Conflict (409) error. Other thing you could try to do is perform an optimistic write by specifying some random Etag value and check for Precondition Failed (412) error. If you get a 403 error or 404 error, that would indicate there's something wrong with your SAS token.

Question:

I'm trying to add an entity to a table at Azure but I'm getting many errors. Here is my code:

package table;
import com.microsoft.azure.storage.*;
import com.microsoft.azure.storage.table.*;
import com.microsoft.azure.storage.table.TableQuery.*;

public class tableTutorial {

 public static final String storageConnectionString=
"DefaultEndpointsProtocol=https;"+
"AccountName=my_storage_name;"+
"AccountKey=my_storage_account_key;"+
"EndpointSuffix=core.windows.net";

public static void main (String args[]) {
try
    {
        // Retrieve storage account from connection-string.
        CloudStorageAccount storageAccount =
            CloudStorageAccount.parse(storageConnectionString);

        // Create the table client.
        CloudTableClient tableClient = 
 storageAccount.createCloudTableClient();

        // Create a cloud table object for the table.
        CloudTable cloudTable = tableClient.getTableReference("people");

        // Create a new customer entity.
        CustomerEntity customer1 = new CustomerEntity("Harp", "Walter");
        customer1.setEmail("Walter@contoso.com");
        customer1.setPhoneNumber("425-555-0101");

        // Create an operation to add the new customer to the people table.
        TableOperation insertCustomer1 = 
      TableOperation.insertOrReplace(customer1);

        // Submit the operation to the table service.
        cloudTable.execute(insertCustomer1);
        }
        catch (Exception e)
        {
        // Output the stack trace.
        e.printStackTrace();
       }

    }

}



class CustomerEntity extends TableServiceEntity {
public CustomerEntity(String lastName, String firstName) {
    this.partitionKey = lastName;
    this.rowKey = firstName;
   }

 public CustomerEntity() { 

   String email;
   String phoneNumber;
 }
 public String getEmail() {
    return this.email;
  }

 public void setEmail(String email) {
    this.email = email;
  }

  public String getPhoneNumber() {
    return this.phoneNumber;
  }

  public void setPhoneNumber(String phoneNumber) {
    this.phoneNumber = phoneNumber;
  }
}

The errors I'm getting are:

com.microsoft.azure.storage.StorageException: An attempt was made to access an inaccessible member of the entity during serialization. at com.microsoft.azure.storage.table.TableServiceEntity.writeEntity(TableServiceEntity.java:468) at com.microsoft.azure.storage.table.TableEntitySerializer.getPropertiesFromDictionary(TableEntitySerializer.java:213) at com.microsoft.azure.storage.table.TableEntitySerializer.writeJsonEntity(TableEntitySerializer.java:129) at com.microsoft.azure.storage.table.TableEntitySerializer.writeSingleEntityToStream(TableEntitySerializer.java:63) at com.microsoft.azure.storage.table.TableOperation.insertImpl(TableOperation.java:381) at com.microsoft.azure.storage.table.TableOperation.performInsert(TableOperation.java:362) at com.microsoft.azure.storage.table.TableOperation.execute(TableOperation.java:682) at com.microsoft.azure.storage.table.CloudTable.execute(CloudTable.java:529) at com.microsoft.azure.storage.table.CloudTable.execute(CloudTable.java:496) at table.tableTutorial.main(tableTutorial.java:86) Caused by: java.lang.IllegalAccessException: class com.microsoft.azure.storage.table.PropertyPair cannot access a member of class table.CustomerEntity with modifiers "public" at java.base/jdk.internal.reflect.Reflection.newIllegalAccessException(Unknown Source) at java.base/java.lang.reflect.AccessibleObject.checkAccess(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at com.microsoft.azure.storage.table.PropertyPair.generateEntityProperty(PropertyPair.java:291) at com.microsoft.azure.storage.table.TableServiceEntity.writeEntityWithReflection(TableServiceEntity.java:211) at com.microsoft.azure.storage.table.TableServiceEntity.writeEntity(TableServiceEntity.java:465)


Answer:

Caused by: java.lang.IllegalAccessException: class com.microsoft.azure.storage.table.PropertyPair cannot access a member of class table.CustomerEntity with modifiers "public"

To serialize your entity, your CustomerEntity should be public, so it should be defined in a separate file. Also note not to declare properties in constructor method.

Here's the code for you to refer.

package table;
import com.microsoft.azure.storage.table.TableServiceEntity;

public class CustomerEntity extends TableServiceEntity {

public CustomerEntity(String lastName, String firstName) {
    this.partitionKey = lastName;
    this.rowKey = firstName;
}

public CustomerEntity() { }

String email;
String phoneNumber;

public String getEmail() {
    return this.email;
}

public void setEmail(String email) {
    this.email = email;
}

public String getPhoneNumber() {
    return this.phoneNumber;
}

public void setPhoneNumber(String phoneNumber) {
    this.phoneNumber = phoneNumber;
}
}

Question:

I'm using Java to query Azure Table and I'd like to retrieve all entities with a given partitionid using the following code. However as there are the following restriction in Azure Table, I'd like to know how they should be resolved. The restrictions I'm concerned are:

1) As Azure Table has paging mechanism in place, if a partition contains more a certain number of entities (I believe it's 1000), it return the first 1000 entities. How can I get all the records in m code.

2) As the entries per second for each partition is 2000, what happens if the partition contains more than 2000 entries and multiple instances of my application queries the same partition at the same time?

String partitionFilter = TableQuery.generateFilterCondition(
       "PartitionKey", 
       QueryComparisons.EQUAL,
       term);


   TableQuery<AzureTableDFEntity> partitionQuery =
       TableQuery.from(AzureTableDFEntity.class)
       .where(partitionFilter);

    // Loop through the results
  int count = 0;
  for (AzureTableDFEntity entity : cloudTableIDF.execute(partitionQuery)) {
        count++;                            
   }

Answer:

1) Using execute will automatically and lazily follow the continuation tokens from page to page of results. executeSegmented will not. So, the code you have should work just fine for paging.

2) You could get throttled if you pass the limit or have spikes. For example, 0 to 2000 very quickly will get throttled as the system load balances. The library has a retry policy mechanism in place by default (exponential) which will retry throttled requests a few times. You can change the retry policy settings or use a custom retry policy as well. See the TableRequestOptions class and the Retry*Retry classes for more info.

Question:

I am using Java to Query Azure table storage service.

I am querying a Azure table based on TimeStamp. Before executing query I converted the local java date to UTC.

Query is throwing an TableServiceException "One of the request inputs is not valid. RequestId:59445f16-0002-007e-152d-b3e24d000000 Time:2016-05-21T06:50:34.5077574Z"

When I used same date and query data using Azure Storage explorer; I am able to get the desired value as shown in below screen shot.

http://puu.sh/oZD6y/eadcc45859.png This makes me wonder what am I doing wrong.

below is the code that I use to query Azure Table. endDate is of Type Date

String partitionFilter = TableQuery.generateFilterCondition(PARTITION_KEY,   QueryComparisons.EQUAL,
            partitionKey);
    String date2 = TableQuery.generateFilterCondition(TIMESTAMP, QueryComparisons.LESS_THAN_OR_EQUAL, endDate.getTime());
    String finalFilter = TableQuery.combineFilters(partitionFilter, Operators.AND, date2);

    CloudTable table = getTable(tableName);
    TableQuery<TableServiceEntity> query = TableQuery.from(TableServiceEntity.class).where(finalFilter);
    Iterable<TableServiceEntity> tableEntries = table.execute(query);
    return tableEntries;

Below code is used to convert local date

    long ts = System.currentTimeMillis();
    Date localTime = new Date(ts);
    String format = "yyyy/MM/dd HH:mm:ss";
    SimpleDateFormat sdf = new SimpleDateFormat (format);


    sdf.setTimeZone(TimeZone.getTimeZone("UTC"));
    Date gmtTime = new Date(sdf.format(localTime));
    System.out.println("Local:" + localTime.toString() + "," + localTime.getTime() + " --> UTC time:" + gmtTime.toString() + "-" + gmtTime.getTime());

This already wasted my time a lot. Any help would be highly appreciated.


Answer:

(PartitionKey eq '1') and (Timestamp le 1463796341217L)

Basically you have an issue with your query. Essentially Timestamp is a date/time kind of attribute so if you were writing ODATA query yourself, your query should be something like:

(PartitionKey eq '1') and (Timestamp le datetime'some-date-time-value')

If you notice, this is something you're doing in Azure Storage Explorer as well.

Looking at your code, you are calling getTime() method on endDate which will return you milliseconds elapsed since January 1, 1970, 00:00:00 GMT. What you need to do is use endDate as is. Then the SDK will convert the query in the format understood by Azure Table Service.

Question:

I have an Azure Table where the partition key is DateTime.MaxValue.Ticks - DateTime.UtcNow.Ticks that means the records are stored based on the partition key. I'm using java and would like to retrieve only the last record. How can I do that?


Answer:

Please use setTakeCount() in class TableQuery.

Question:

I've created an Entity class that extends TableServiceEntity. My class has a single field called "value" of type Byte:

public class TestEntity extends TableServiceEntity{

    public Byte value;

    public TestEntity(){
        super("some_partition","somekey");
    }
    public Byte getValue() {
        return value;
    }
    public void setValue(Byte value) {
        this.value = value;
    }
}

According to the documentation for TableServiceEntity, Byte is one of the supported field types.

However when I try to store my entity, I get the following exception:

java.lang.IllegalArgumentException: Type class java.lang.Byte is not supported.
    at com.microsoft.azure.storage.table.EntityProperty.<init>(EntityProperty.java:175)
    at com.microsoft.azure.storage.table.PropertyPair.generateEntityProperty(PropertyPair.java:271)
    at com.microsoft.azure.storage.table.TableServiceEntity.writeEntityWithReflection(TableServiceEntity.java:217)
    at com.microsoft.azure.storage.table.TableServiceEntity.writeEntity(TableServiceEntity.java:470)
    at com.microsoft.azure.storage.table.TableEntitySerializer.writeJsonEntity(TableEntitySerializer.java:317)
    at com.microsoft.azure.storage.table.TableEntitySerializer.writeSingleJsonEntity(TableEntitySerializer.java:411)
    at com.microsoft.azure.storage.table.TableEntitySerializer.writeSingleEntityToStream(TableEntitySerializer.java:74)
    at com.microsoft.azure.storage.table.TableOperation.insertImpl(TableOperation.java:389)
    at com.microsoft.azure.storage.table.TableOperation.performInsert(TableOperation.java:370)
...

The fault seems to be with the EntityProperty constructor (see source), which doesn't seem to support Byte or byte types, only Byte[] and byte[]:

protected EntityProperty(final String value, final Class<?> type) {
   this.type = type;
   this.value = value;
   if (type.equals(byte[].class)) {
      this.getValueAsByteArray();
      this.edmType = EdmType.BINARY;
   }
   else if (type.equals(Byte[].class)) {
      this.getValueAsByteObjectArray();
      this.edmType = EdmType.BINARY;
   }
   ...

Am I doing something wrong, or is this a documentation error?


Answer:

According to MSDN, the list of property types allowed for the Azure Table service does not include Edm.Byte, but does include Edm.Binary (which is byte[]). For a more in-depth understanding of the data types allowed and other limitations see Understanding the Table Service Data Model.