SparkHadoopUtil

Value Members

1. final def !=(arg0: AnyRef): Boolean

   Definition Classes

   AnyRef

2. final def !=(arg0: Any): Boolean

   Definition Classes

   Any

3. final def ##(): Int

   Definition Classes

   AnyRef → Any

4. final def ==(arg0: AnyRef): Boolean

   Definition Classes

   AnyRef

5. final def ==(arg0: Any): Boolean

   Definition Classes

   Any

6. def addCredentials(conf: JobConf): Unit

   Add any user credentials to the job conf which are necessary for running on a secure Hadoop cluster.

7. def addCurrentUserCredentials(creds: Credentials): Unit

8. def addSecretKeyToUserCredentials(key: String, secret: String): Unit

9. final def asInstanceOf[T0]: T0

   Definition Classes

   Any

10. def clone(): AnyRef

    Attributes

    protected[java.lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

11. val conf: Configuration

12. final def eq(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

13. def equals(arg0: Any): Boolean

    Definition Classes

    AnyRef → Any

14. def finalize(): Unit

    Attributes

    protected[java.lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( classOf[java.lang.Throwable] )

15. final def getClass(): Class[_]

    Definition Classes

    AnyRef → Any

16. def getConfigurationFromJobContext(context: JobContext): Configuration

    Using reflection to get the Configuration from JobContext/TaskAttemptContext.

    Using reflection to get the Configuration from JobContext/TaskAttemptContext. If we directly call JobContext/TaskAttemptContext.getConfiguration, it will generate different byte codes for Hadoop 1.+ and Hadoop 2.+ because JobContext/TaskAttemptContext is class in Hadoop 1.+ while it's interface in Hadoop 2.+.

17. def getCurrentUserCredentials(): Credentials

18. def getSecretKeyFromUserCredentials(key: String): Array[Byte]

19. def getTimeFromNowToRenewal(sparkConf: SparkConf, fraction: Double, credentials: Credentials): Long

    How much time is remaining (in millis) from now to (fraction * renewal time for the token that is valid the latest)? This will return -ve (or 0) value if the fraction of validity has already expired.

20. def globPath(pattern: Path): Seq[Path]

21. def hashCode(): Int

    Definition Classes

    AnyRef → Any

22. final def isInstanceOf[T0]: Boolean

    Definition Classes

    Any

23. def isTraceEnabled(): Boolean

    Attributes

    protected

    Definition Classes

    Logging

24. def isYarnMode(): Boolean

25. def listFilesSorted(remoteFs: FileSystem, dir: Path, prefix: String, exclusionSuffix: String): Array[FileStatus]

    Lists all the files in a directory with the specified prefix, and does not end with the given suffix.

    Lists all the files in a directory with the specified prefix, and does not end with the given suffix. The returned {{FileStatus}} instances are sorted by the modification times of the respective files.

26. def listLeafDirStatuses(fs: FileSystem, baseStatus: FileStatus): Seq[FileStatus]

27. def listLeafDirStatuses(fs: FileSystem, basePath: Path): Seq[FileStatus]

28. def listLeafStatuses(fs: FileSystem, baseStatus: FileStatus): Seq[FileStatus]

    Get FileStatus objects for all leaf children (files) under the given base path.

    Get FileStatus objects for all leaf children (files) under the given base path. If the given path points to a file, return a single-element collection containing FileStatus of that file.

29. def listLeafStatuses(fs: FileSystem, basePath: Path): Seq[FileStatus]

    Get FileStatus objects for all leaf children (files) under the given base path.

    Get FileStatus objects for all leaf children (files) under the given base path. If the given path points to a file, return a single-element collection containing FileStatus of that file.

30. def log: Logger

    Attributes

    protected

    Definition Classes

    Logging

31. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

32. def logDebug(msg: ⇒ String): Unit

    Attributes

    protected

    Definition Classes

    Logging

33. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

34. def logError(msg: ⇒ String): Unit

    Attributes

    protected

    Definition Classes

    Logging

35. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

36. def logInfo(msg: ⇒ String): Unit

    Attributes

    protected

    Definition Classes

    Logging

37. def logName: String

    Attributes

    protected

    Definition Classes

    Logging

38. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

39. def logTrace(msg: ⇒ String): Unit

    Attributes

    protected

    Definition Classes

    Logging

40. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

41. def logWarning(msg: ⇒ String): Unit

    Attributes

    protected

    Definition Classes

    Logging

42. def loginUserFromKeytab(principalName: String, keytabFilename: String): Unit

43. final def ne(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

44. def newConfiguration(conf: SparkConf): Configuration

    Return an appropriate (subclass) of Configuration.

    Return an appropriate (subclass) of Configuration. Creating config can initializes some Hadoop subsystems.

45. def newConfiguration(): Configuration

    Annotations

    @Deprecated

46. final def notify(): Unit

    Definition Classes

    AnyRef

47. final def notifyAll(): Unit

    Definition Classes

    AnyRef

48. def runAsSparkUser(func: () ⇒ Unit): Unit

    Runs the given function with a Hadoop UserGroupInformation as a thread local variable (distributed to child threads), used for authenticating HDFS and YARN calls.

    Runs the given function with a Hadoop UserGroupInformation as a thread local variable (distributed to child threads), used for authenticating HDFS and YARN calls.

    IMPORTANT NOTE: If this function is going to be called repeated in the same process you need to look https://issues.apache.org/jira/browse/HDFS-3545 and possibly do a FileSystem.closeAllForUGI in order to avoid leaking Filesystems

49. def substituteHadoopVariables(text: String, hadoopConf: Configuration): String

    Substitute variables by looking them up in Hadoop configs.

    Substitute variables by looking them up in Hadoop configs. Only variables that match the ${hadoopconf- .. } pattern are substituted.

50. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes

    AnyRef

51. def toString(): String

    Definition Classes

    AnyRef → Any

52. def transferCredentials(source: UserGroupInformation, dest: UserGroupInformation): Unit

53. final def wait(): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

54. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

55. final def wait(arg0: Long): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )