SparkReceiver IO

SparkReceiverIO 是一種轉換，用於從 Apache Spark Receiver 讀取資料作為無邊界來源。

Spark Receivers 支援

SparkReceiverIO 目前支援 Apache Spark Receiver。

Spark Receiver 的需求

如需更多詳細資訊，請參閱 SparkReceiverIO 自述檔。

使用 SparkReceiverIO 進行串流讀取

為了從 Spark Receiver 讀取，您需要傳遞

您可以透過傳遞以下參數輕鬆建立 receiverBuilder 物件

例如

//In this example, MyReceiver accepts a MyConfig object as its only constructor parameter.
MyConfig myPluginConfig = new MyConfig(authToken, apiServerUrl);
Object[] myConstructorArgs = new Object[] {myConfig};
ReceiverBuilder<String, MyReceiver<String>> myReceiverBuilder =
  new ReceiverBuilder<>(MyReceiver.class)
    .withConstructorArgs(myConstructorArgs);

然後您就可以將此 receiverBuilder 物件傳遞給 SparkReceiverIO。

例如

SparkReceiverIO.Read<String> readTransform =
  SparkReceiverIO.<String>read()
    .withGetOffsetFn(Long::valueOf)
    .withSparkReceiverBuilder(myReceiverBuilder)
p.apply("readFromMyReceiver", readTransform);

讀取具有可選參數的資料

您可以選擇性地傳遞以下可選參數

例如

SparkReceiverIO.Read<String> readTransform =
  SparkReceiverIO.<String>read()
    .withGetOffsetFn(Long::valueOf)
    .withSparkReceiverBuilder(myReceiverBuilder)
    .withPullFrequencySec(1L)
    .withStartOffset(1L)
    .withTimestampFn(Instant::parse);
p.apply("readFromReceiver", readTransform);

特定 Spark Receiver 的範例

CDAP Hubspot Receiver

ReceiverBuilder<String, HubspotReceiver<String>> hubspotReceiverBuilder =
  new ReceiverBuilder<>(HubspotReceiver.class)
    .withConstructorArgs(hubspotConfig);
SparkReceiverIO.Read<String> readTransform =
  SparkReceiverIO.<String>read()
    .withGetOffsetFn(GetOffsetUtils.getOffsetFnForHubspot())
    .withSparkReceiverBuilder(hubspotReceiverBuilder)
p.apply("readFromHubspotReceiver", readTransform);