Hot questions for Using Azure in scala

Question:

How can I connect to Databricks Delta table using JDBC?

I have tried connecting simba driver but im getting hard time for driver class name and url configuration.

Any solution is appreciated. I cannot paste code here as its company code.

Thanks in advance.


Answer:

Check this link below. This has steps to configure delta using JDBC

http://sedeks.blogspot.com/2019/05/how-to-connect-to-databricks-delta.html

code provided in this link:

    import java.sql.DriverManager 
import java.sql.Driver
import java.sql.Connection 
import javax.sql.DataSource

object ScalaJdbcConnectSelect {
    def main(args: Array[String]) {
        val driver = "com.simba.spark.jdbc41.Driver"  //attach the Spark jar to the Classpath.
        val url = "jdbc:spark://field-eng.cloud.databricks.com:443/default;transportMode=http;ssl=true;httpPath=sql/protocolvl/o/0/0911-153027-hopes19";    
        val username = "token" 
        val password = "<token-value>" //Token generated from databricks profile page.
        var connection:Connection = null
        try {
        // Create the connection
            Class.forName(driver)
            connection = DriverManager.getConnection(url, username, password)
            if(connection != null){
                println("Connection Established");

            }
            else {
                println("Connection Failed");
            }
            // create the statement
            val statement = connection.createStatement()
            val resultSet = statement.executeQuery("<<Query>") // Profile your query here.
            while ( resultSet.next() ) {
                // Iterate through Result set
            } 
        catch {
            case e => e.printStackTrace
        }
        connection.close()
    }
}

Question:

My cluster should read some input files that are located in my azure storage. I am submitting my .jar to the cluster through livy but it always dies because I cannot locate my files -> User class threw exception: java.io.FileNotFoundException. What am I missing? I dont want to use sc.textFile to open the files because it would make them into RDD structures and I need their structure correct.

val Inputs : String = scala.io.Source.fromFile("wasbs:///inputs.txt").mkString

I believe that I am trying to read from the wrong locationo or with the wrong method, any ideas?

Thanks!


Answer:

According to your description, based on my understanding, I think you want to load the plain text file on Azure Storage using Scala running on HDInsight.

Per my experience, there are two ways which you can try to implement your needs.

  1. Just using Scala within Azure Java Storage SDK to get the content of the text blob, please refer to the tutorial How to use Blob storage from Java, and I think using Scala to rewrite the sample code in the tutorial is very simple.

  2. Using Hadoop Filesystem API within Hadoop Azure Support library to load file data, please refer to the hadoop example wiki https://wiki.apache.org/hadoop/HadoopDfsReadWriteExample to write your code in Scala.