org.apache.spark.streaming.util

Class WriteAheadLogUtils

Object

org.apache.spark.streaming.util.WriteAheadLogUtils

public class WriteAheadLogUtils
extends Object

A helper class with utility functions related to the WriteAheadLog interface

Constructor Summary

Constructors

Constructor and Description
WriteAheadLogUtils()

Method Summary

Methods

Modifier and Type	Method and Description
static WriteAheadLog	createLogForDriver(SparkConf sparkConf, String fileWalLogDirectory, org.apache.hadoop.conf.Configuration fileWalHadoopConf) Create a WriteAheadLog for the driver.
static WriteAheadLog	createLogForReceiver(SparkConf sparkConf, String fileWalLogDirectory, org.apache.hadoop.conf.Configuration fileWalHadoopConf) Create a WriteAheadLog for the receiver.
static int	DEFAULT_MAX_FAILURES()
static int	DEFAULT_ROLLING_INTERVAL_SECS()
static String	DRIVER_WAL_BATCHING_CONF_KEY()
static String	DRIVER_WAL_BATCHING_TIMEOUT_CONF_KEY()
static String	DRIVER_WAL_CLASS_CONF_KEY()
static String	DRIVER_WAL_CLOSE_AFTER_WRITE_CONF_KEY()
static String	DRIVER_WAL_MAX_FAILURES_CONF_KEY()
static String	DRIVER_WAL_ROLLING_INTERVAL_CONF_KEY()
static boolean	enableReceiverLog(SparkConf conf)
static long	getBatchingTimeout(SparkConf conf) How long we will wait for the wrappedLog in the BatchedWriteAheadLog to write the records before we fail the write attempt to unblock receivers.
static int	getMaxFailures(SparkConf conf, boolean isDriver)
static int	getRollingIntervalSecs(SparkConf conf, boolean isDriver)
static boolean	isBatchingEnabled(SparkConf conf, boolean isDriver)
static String	RECEIVER_WAL_CLASS_CONF_KEY()
static String	RECEIVER_WAL_CLOSE_AFTER_WRITE_CONF_KEY()
static String	RECEIVER_WAL_ENABLE_CONF_KEY()
static String	RECEIVER_WAL_MAX_FAILURES_CONF_KEY()
static String	RECEIVER_WAL_ROLLING_INTERVAL_CONF_KEY()
static boolean	shouldCloseFileAfterWrite(SparkConf conf, boolean isDriver)

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

WriteAheadLogUtils

public WriteAheadLogUtils()

Method Detail

RECEIVER_WAL_ENABLE_CONF_KEY

public static String RECEIVER_WAL_ENABLE_CONF_KEY()

RECEIVER_WAL_CLASS_CONF_KEY

public static String RECEIVER_WAL_CLASS_CONF_KEY()

RECEIVER_WAL_ROLLING_INTERVAL_CONF_KEY

public static String RECEIVER_WAL_ROLLING_INTERVAL_CONF_KEY()

RECEIVER_WAL_MAX_FAILURES_CONF_KEY

public static String RECEIVER_WAL_MAX_FAILURES_CONF_KEY()

RECEIVER_WAL_CLOSE_AFTER_WRITE_CONF_KEY

public static String RECEIVER_WAL_CLOSE_AFTER_WRITE_CONF_KEY()

DRIVER_WAL_CLASS_CONF_KEY

public static String DRIVER_WAL_CLASS_CONF_KEY()

DRIVER_WAL_ROLLING_INTERVAL_CONF_KEY

public static String DRIVER_WAL_ROLLING_INTERVAL_CONF_KEY()

DRIVER_WAL_MAX_FAILURES_CONF_KEY

public static String DRIVER_WAL_MAX_FAILURES_CONF_KEY()

DRIVER_WAL_BATCHING_CONF_KEY

public static String DRIVER_WAL_BATCHING_CONF_KEY()

DRIVER_WAL_BATCHING_TIMEOUT_CONF_KEY

public static String DRIVER_WAL_BATCHING_TIMEOUT_CONF_KEY()

DRIVER_WAL_CLOSE_AFTER_WRITE_CONF_KEY

public static String DRIVER_WAL_CLOSE_AFTER_WRITE_CONF_KEY()

DEFAULT_ROLLING_INTERVAL_SECS

public static int DEFAULT_ROLLING_INTERVAL_SECS()

DEFAULT_MAX_FAILURES

public static int DEFAULT_MAX_FAILURES()

enableReceiverLog

public static boolean enableReceiverLog(SparkConf conf)

getRollingIntervalSecs

public static int getRollingIntervalSecs(SparkConf conf,
                         boolean isDriver)

getMaxFailures

public static int getMaxFailures(SparkConf conf,
                 boolean isDriver)

isBatchingEnabled

public static boolean isBatchingEnabled(SparkConf conf,
                        boolean isDriver)

getBatchingTimeout

public static long getBatchingTimeout(SparkConf conf)

How long we will wait for the wrappedLog in the BatchedWriteAheadLog to write the records before we fail the write attempt to unblock receivers.

Parameters:

conf - (undocumented)

Returns:

(undocumented)

shouldCloseFileAfterWrite

public static boolean shouldCloseFileAfterWrite(SparkConf conf,
                                boolean isDriver)

createLogForDriver

public static WriteAheadLog createLogForDriver(SparkConf sparkConf,
                               String fileWalLogDirectory,
                               org.apache.hadoop.conf.Configuration fileWalHadoopConf)

Create a WriteAheadLog for the driver. If configured with custom WAL class, it will try to create instance of that class, otherwise it will create the default FileBasedWriteAheadLog.

Parameters:

sparkConf - (undocumented)

fileWalLogDirectory - (undocumented)

fileWalHadoopConf - (undocumented)

Returns:

(undocumented)

createLogForReceiver

public static WriteAheadLog createLogForReceiver(SparkConf sparkConf,
                                 String fileWalLogDirectory,
                                 org.apache.hadoop.conf.Configuration fileWalHadoopConf)

Create a WriteAheadLog for the receiver. If configured with custom WAL class, it will try to create instance of that class, otherwise it will create the default FileBasedWriteAheadLog.

Parameters:

sparkConf - (undocumented)

fileWalLogDirectory - (undocumented)

fileWalHadoopConf - (undocumented)

Returns:

(undocumented)