Hot questions for Using Azure in azure cosmosdb

Top Java Programmings / Azure / azure cosmosdb

Question:

I have created JSON that byte.length is too large, and when I try to use the api java:

documentClient.createDocument(
                    getTodoCollection().getSelfLink(), document, null,
                    false)

returns an

DocumentClientException: Message: {"Errors":["Request size is too large"]}

Is there any way to save this kind of files in the documentDB?


Answer:

From DocumentDB limits and quotas page, maximum size of a document can be 512 KB. If your JSON size is more than that, you will get this error.

If the size of your document is more than 512 KB, then you can't save the document as is in DocumentDB. One thing you could do is save the JSON in Blob Storage as a block blob and save the Blob URL in DocumentDB.

P.S. For some reason I thought the size of an attachment in DocumentDB is 2 MB and I thought of suggesting that but it seems they have changed it to 512 KB as well.

Question:

I have an employee dto with department as partitionKey.

  1. How to get number of partition at any given of point of time?
  2. How to query cosmosdb(documentDB) using partitionKey, either query explorer or java api will do?

What I tried:

List<Document> documentList = documentClient.queryDocuments(getCollection().getSelfLink(),"SELECT * FROM root r WHERE r.partitionKey ='" + partitionKey+"'", null).getQueryIterable().toList();

I end up getting IllegalStateException state exception from java api and query exporter also did't give any output. Any help is highly appreciable.


Answer:

Partial answer:

Given the name of Partition Key attribute is department, please change your query to:

List<Document> documentList = 
    documentClient.queryDocuments(
         getCollection().getSelfLink(),
              "SELECT * FROM root r WHERE r.department ='" + 
              partitionKey + "'", null).getQueryIterable().toList();

Question:

I have only a single collection and when I try to delete the document using the code below

    PartitionKey partitionKey = new PartitionKey("undefined");
    RequestOptions requestOptions=new RequestOptions();
    requestOptions.setPartitionKey(partitionKey);
    for(Document currentDocument: existingIMEIDevice){
        try {
            ConfigVariables.documentClient.deleteDocument(currentDocument.getSelfLink(), requestOptions);
        } catch (DocumentClientException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }

it throws exception.

com.microsoft.azure.documentdb.DocumentClientException: Message: {"Errors":["Resource Not Found"]} ActivityId: 4353e7c0-0b24-4b2a-8ec6-fc2db4059aa0, Request URI: /apps/708ed403-166f-44e4-847f-ccaa0cd22d9c/services/d1e2ed4d-7e69-4a3d-9575-3e24b96621b4/partitions/e3fc6138-06a5-4876-a629-a4be69917ded/replicas/131533416718986721p, StatusCode: NotFound at com.microsoft.azure.documentdb.internal.ErrorUtils.maybeThrowException(ErrorUtils.java:69) at com.microsoft.azure.documentdb.internal.GatewayProxy.performDeleteRequest(GatewayProxy.java:187) at com.microsoft.azure.documentdb.internal.GatewayProxy.doDelete(GatewayProxy.java:99) at com.microsoft.azure.documentdb.internal.GatewayProxy.processMessage(GatewayProxy.java:332) at com.microsoft.azure.documentdb.DocumentClient$7.apply(DocumentClient.java:2877) at com.microsoft.azure.documentdb.internal.RetryUtility.executeDocumentClientRequest(RetryUtility.java:58) at com.microsoft.azure.documentdb.DocumentClient.doDelete(DocumentClient.java:2883) at com.microsoft.azure.documentdb.DocumentClient.deleteDocument(DocumentClient.java:956) at com.moveinsync.centraldevices.persistance.AzureCommDAOImpl.replaceDocument(AzureCommDAOImpl.java:45) at com.moveinsync.centraldevices.persistance.AzureCommDAOImpl.documentDbBulkInsert(AzureCommDAOImpl.java:85) at com.moveinsync.centraldevices.jobs.ToAzureJob.executeInternal(ToAzureJob.java:27) at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:75) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) If I do not provide RequestOptions it asks me to provide a partition key. I don't have a partition key as the below does not returns anything

SELECT c.partitionKey FROM c ORDER BY c.partitionKey

How do I solve this?


Answer:

Per my experience, if your collection does not have a partitioning key, you do not need to set the query criteria for the partitioning key when you operate the database.

In you post, the collection does not have a partitioning key, and you set the partitioning key into RequestOption. So , the database certainly doesn't know where to find the document to operate.

You could refer to my snippet of code:

import com.microsoft.azure.documentdb.*;

public class DeleteDocuments {
    private static String accountName="https://jay.documents.azure.com:443/";

    private static String accountKey="Czi66skfjZYLTaXuDhoxNb2JHL4DR98VxAxGXtLkWFnjCa5e7gUXQuPgemlXwyPWjjWJpwrseH1wPMfhkqA8cQ==";

    private static String databaseName = "db";

    private static String collectionName = "coll";

    public static void main(String[] args) throws DocumentClientException {

        DocumentClient client = new DocumentClient(
                accountName,
                accountKey
                , new ConnectionPolicy(),
                ConsistencyLevel.Session);

        FeedOptions options = null;
        String sql = "select * from c";
        FeedResponse<Document> queryResults  = client.queryDocuments("dbs/"+databaseName+"/colls/"+collectionName, sql, options);

        System.out.println("before delete :");
        for (Document d : queryResults.getQueryIterable()) {
            System.out.println(String.format("\tRead %s", d));
        }

        RequestOptions requestOptions = new RequestOptions();
        requestOptions.setOfferThroughput(400);

        client.deleteDocument("/dbs/"+databaseName+"/colls/"+collectionName+"/docs/1",requestOptions);

        queryResults  = client.queryDocuments("dbs/"+databaseName+"/colls/"+collectionName, sql, options);


        System.out.println("after delete :");

        for (Document d : queryResults.getQueryIterable()) {
            System.out.println(String.format("\tRead %s", d));
        }
    }
}

Update Answer:

I think you misunderstand the meaning of partitionkey property in the options[].

For example , my container is created like this:

The partition key is "name" for my collection here. You could check your collection's partition key.

And my documents as below :

{
    "id": "1",
    "name": "jay"
}

{
    "id": "2",
    "name": "jay2"
}

My partitionkey is 'name', so here I have two paritions : 'jay' and 'jay1'.

So, here you should set the partitionkey property to 'jay' or 'jay2',not 'name'.

At this point, if I run the code below without setting partition key into RequestOptions, I'll meet the same issue as you.

  RequestOptions requestOptions = new RequestOptions();
  requestOptions.setOfferThroughput(400);

        client.deleteDocument("/dbs/"+databaseName+"/colls/"+collectionName+"/docs/1",requestOptions);

Exception in thread "main" java.lang.UnsupportedOperationException: PartitionKey value must be supplied for this operation. at com.microsoft.azure.documentdb.DocumentClient.addPartitionKeyInformation(DocumentClient.java:3199) at com.microsoft.azure.documentdb.DocumentClient.addPartitionKeyInformation(DocumentClient.java:3180) at com.microsoft.azure.documentdb.DocumentClient.deleteDocument(DocumentClient.java:959) at DeleteDocuments.main(DeleteDocuments.java:32)

I need to set partition key parameter to be the partition where the operated document stored.

 RequestOptions requestOptions = new RequestOptions();
 requestOptions.setOfferThroughput(400);
 PartitionKey partitionKey = new PartitionKey("jay");
 requestOptions.setPartitionKey(partitionKey);

        client.deleteDocument("/dbs/"+databaseName+"/colls/"+collectionName+"/docs/1",requestOptions);

Update Answer 2:

I guess you want to operate the document which does not set partition key.

Please refer to this perfect blog, you'll find the answer!

In java code , just set partition key to Undefined.Value() then everything will be done.

 RequestOptions requestOptions = new RequestOptions();
 requestOptions.setOfferThroughput(400);
 PartitionKey partitionKey = new PartitionKey(Undefined.Value());
 requestOptions.setPartitionKey(partitionKey);

        client.deleteDocument("/dbs/"+databaseName+"/colls/"+collectionName+"/docs/3",requestOptions);

Hope it helps you.

Question:

I am using CosmosDb for a project, and was first trying the Document-db java SQL API. With that API I can set item-level TTLs right on the Document itself. Ex.

Document document = new Document(new Gson().toJson(testObject));
    document.setTimeToLive(10); // Time to live 10 seconds
    documentClient.createDocument(documentCollection.getSelfLink(), document, null,
        false).getResource();

However, I am also interested in using the Table API. I could not find anything in the documentation about setting item-level TTL. https://docs.microsoft.com/en-us/azure/cosmos-db/table-storage-how-to-use-java I am using https://github.com/Azure/azure-storage-java version 8.0.0 https://mvnrepository.com/artifact/com.microsoft.azure/azure-storage

Is it possible to do this, or at least set a server-side TTL for all items in a table? Thanks!


Answer:

Today, Table API does not expose any TTL feature, so unfortunately you cannot use TTL with the Table API, even though Cosmos supports it under the covers.

Question:

I gone through the documentation , there are two ways to fetch the documents

  1. using Azure's Endpoint https://{databaseaccount}.documents.azure.com/dbs/{db-id}/colls/{coll-id}/docs even i set the required headers Token and all I am getting some exception

{ "code": "Forbidden", "message": "Sql api is not supported for this database account\r\nActivityId: fc5ec296-b7a1- 44df-9c69-42e804177242, Microsoft.Azure.Documents.Common/2.7.0" }

  1. using SDK through the sdk also i m getting the same above exception , below is the java code which i m using for fetching the documents

    public class App {
    private static final String END_POINT = "https://***.documents.azure.com/";
    private static final String MASTER_KEY = "***";
    
    // Define an id for your database and collection
    private static final String DATABASE_ID = "iotdata";
    private static final String COLLECTION_ID = "details";
    
    public static void main(String[] args) {
    
        DocumentClient documentClient = new DocumentClient(END_POINT, MASTER_KEY, new ConnectionPolicy(),
                ConsistencyLevel.Session);
        System.out.println("Check if database " + DATABASE_ID + " exists.");
        String databaseLink = String.format("/dbs/%s", DATABASE_ID);
        try {
            ResourceResponse<Database> readDatabase = documentClient.readDatabase(databaseLink, null);
            if (readDatabase.getStatusCode() == 200) {
                System.out.println("Connection Established");
    
            }
    
            FeedResponse<Document> queryResults = documentClient.queryDocuments("/dbs/iotdata/colls/details",
                    "SELECT * FROM details WHERE mac_address = '28:b2:bd:01:d0:94'", null);
    
            System.out.println("Running SQL query...");
            for (Document family : queryResults.getQueryIterable()) {
                System.out.println(String.format("\tRead %s", family));
            }
    
        } catch (Exception e) {
            e.printStackTrace();
        }
    
    }
    

    }


Answer:

Firstly,i have to say that your sample code in second point is totally right.I tested it and works with Cosmos DB Sql Api.

According to the error:

Sql api is not supported for this database account

It seems that you want to access cosmos db account of other type(maybe mongo api i guess) with SQL API REST API or SQL API SDK. That's forbidden and i'm afraid that's the reason for above error.If your Cosmos DB account is using the Mongo API, you should be using tools and drivers that use the Mongo API. Please refer to this case:Sql api is not supported for this database Error

Question:


Answer:

Below Code Should Work. It's a Simple Spring-Boot Application. If required you can convert it into normal maven application.

All the methods are getting called from connectToDB().

package com.example.demo;

import java.io.FileReader;
import java.io.IOException;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.Date;

import org.json.simple.JSONObject;
import org.json.simple.parser.JSONParser;
import org.json.simple.parser.ParseException;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import com.microsoft.azure.documentdb.ConnectionPolicy;
import com.microsoft.azure.documentdb.ConsistencyLevel;
import com.microsoft.azure.documentdb.DataType;
import com.microsoft.azure.documentdb.Database;
import com.microsoft.azure.documentdb.DocumentClient;
import com.microsoft.azure.documentdb.DocumentClientException;
import com.microsoft.azure.documentdb.DocumentCollection;
import com.microsoft.azure.documentdb.Index;
import com.microsoft.azure.documentdb.IndexingPolicy;
import com.microsoft.azure.documentdb.RangeIndex;
import com.microsoft.azure.documentdb.RequestOptions;

@SpringBootApplication
@RestController
public class HelloCosmosApplication {

    public static void main(String[] args) {
        SpringApplication.run(HelloCosmosApplication.class, args);
    }

    private DocumentClient client;

    @RequestMapping("Connect")
    public String connectToDB() throws DocumentClientException, IOException, ParseException {

        // Making the connection with COSMos DB account
        client = new DocumentClient("https://something-something.documents.azure.com:443/",
                "someKeyShouldBeYourPrimaryKeyIfYouWantToPerformReadWriteOperation==",
                new ConnectionPolicy(), ConsistencyLevel.Session);

        JSONParser parser = new JSONParser();
        // Use JSONObject for simple JSON and JSONArray for array of JSON.
        JSONObject data = (JSONObject) parser
                .parse(new FileReader("C:/STSTestWorkspace/HelloCosmos/src/main/resources/test.json"));


        //This one is added to take date and time.
        DateFormat dateFormat = new SimpleDateFormat("yyyy/MM/dd HH:mm:ss");
        Date date = new Date();
        data.put("id", date.toString());   // we are taking ID as a random value.

        this.createDatabaseIfNotExists("javaDB");
        this.createDocumentCollectionIfNotExists("javaDB", "testJava");
        this.createDocumentIfNotExists("javaDB", "testJava", data);

        return "Success";
    }

    // Create Database
    private void createDatabaseIfNotExists(String databaseName) throws DocumentClientException, IOException {
        String databaseLink = String.format("/dbs/%s", databaseName);

        // Check to verify a database with the id=FamilyDB does not exist
        try {
            client.readDatabase(databaseLink, null);
        } catch (DocumentClientException de) {
            // If the database does not exist, create a new database
            if (de.getStatusCode() == 404) {
                Database database = new Database();
                database.setId(databaseName);

                client.createDatabase(database, null);
            } else {
                throw de;
            }
        }
    }

    // Create Collection
    private void createDocumentCollectionIfNotExists(String databaseName, String collectionName)
            throws IOException, DocumentClientException {
        String databaseLink = String.format("/dbs/%s", databaseName);
        String collectionLink = String.format("/dbs/%s/colls/%s", databaseName, collectionName);

        try {
            client.readCollection(collectionLink, null);
        } catch (DocumentClientException de) {
            // If the document collection does not exist, create a new
            // collection
            if (de.getStatusCode() == 404) {
                DocumentCollection collectionInfo = new DocumentCollection();
                collectionInfo.setId(collectionName);

                // Optionally, you can configure the indexing policy of a
                // collection. Here we configure collections for maximum query
                // flexibility including string range queries.
                RangeIndex index = new RangeIndex(DataType.String);
                index.setPrecision(-1);

                collectionInfo.setIndexingPolicy(new IndexingPolicy(new Index[] { index }));

                // DocumentDB collections can be reserved with throughput
                // specified in request units/second. 1 RU is a normalized
                // request equivalent to the read of a 1KB document. Here we
                // create a collection with 400 RU/s.
                RequestOptions requestOptions = new RequestOptions();
                requestOptions.setOfferThroughput(400);

                client.createCollection(databaseLink, collectionInfo, requestOptions);

            } else {
                throw de;
            }
        }

    }

    // create Document
    private void createDocumentIfNotExists(String databaseName, String collectionName, JSONObject json)
            throws DocumentClientException, IOException {
        try {
            String documentLink = String.format("/dbs/%s/colls/%s/docs/%s", databaseName, collectionName, json);
            client.readDocument(documentLink, new RequestOptions());
        } catch (DocumentClientException de) {
            if (de.getStatusCode() == 404) {
                String collectionLink = String.format("/dbs/%s/colls/%s", databaseName, collectionName);
                this.client.createDocument(collectionLink, json, new RequestOptions(), true);
            } else {
                throw de;
            }
        }
    }

}