Examples of PartitionStrategy

com.cloudera.cdk.data.PartitionStrategy

The strategy used to determine how a dataset is partitioned.

A {@code PartitionStrategy} is configured with one or more{@link FieldPartitioner}s upon creation. When a {@link Dataset} is configuredwith a partition strategy, we say that data is partitioned. Any entities written to a partitioned dataset are evaluated with its {@code PartitionStrategy} which, in turn, produces a {@link PartitionKey}that is used by the dataset implementation to select the proper partition.

Users should use the inner {@link Builder} to create new instances.
@see FieldPartitioner @see PartitionKey @see DatasetDescriptor @see Dataset
org.kitesdk.data.PartitionStrategy

The strategy used to determine how a dataset is partitioned.

When a {@link Dataset} is configuredwith a partition strategy, that data is considered partitioned. Any entities written to a partitioned dataset are evaluated with its {@code PartitionStrategy} to determine which partition to write to.

You should use the inner {@link Builder} to create new instances.
@see DatasetDescriptor @see Dataset

Examples of org.kitesdk.data.PartitionStrategy

  }


  @Test
  @SuppressWarnings("unchecked")
  public void toDirNameIdentityWithNonString() {
    PartitionStrategy strategy = new PartitionStrategy.Builder()
        .identity("id")
        .build();


    StorageKey expected = new StorageKey(strategy);
    expected.replace(0, 0L);

View Full Code Here

Examples of org.kitesdk.data.PartitionStrategy

  }


  @Test
  @SuppressWarnings("deprecation")
  public void testPartitionedWriterSingle() throws IOException {
    PartitionStrategy partitionStrategy = new PartitionStrategy.Builder().hash(
      "username", 2).build();


    FileSystemDataset<Record> ds = new FileSystemDataset.Builder<Record>()
        .namespace("ns")
        .name("partitioned-users")

View Full Code Here

Examples of org.kitesdk.data.PartitionStrategy

  }


  @Test
  @SuppressWarnings("deprecation")
  public void testPartitionedWriterDouble() throws IOException {
    PartitionStrategy partitionStrategy = new PartitionStrategy.Builder()
      .hash("username", 2).hash("email", 3).build();


    FileSystemDataset<Record> ds = new FileSystemDataset.Builder<Record>()
        .namespace("ns")
        .name("partitioned-users")

View Full Code Here

Examples of org.kitesdk.data.PartitionStrategy

  }


  @Test
  @SuppressWarnings("deprecation")
  public void testGetPartitionReturnsNullIfNoAutoCreate() throws IOException {
    PartitionStrategy partitionStrategy = new PartitionStrategy.Builder().hash(
      "username", 2).build();


    FileSystemDataset<Record> ds = new FileSystemDataset.Builder<Record>()
        .namespace("ns")
        .name("partitioned-users")

View Full Code Here

Examples of org.kitesdk.data.PartitionStrategy

  }


  @Test
  @SuppressWarnings("deprecation")
  public void testWriteToSubpartition() throws IOException {
    PartitionStrategy partitionStrategy = new PartitionStrategy.Builder()
      .hash("username", "username_part", 2).hash("email", 3).build();


    FileSystemDataset<Record> ds = new FileSystemDataset.Builder<Record>()
        .namespace("ns")
        .name("partitioned-users")

View Full Code Here

Examples of org.kitesdk.data.PartitionStrategy

  }


  @Test
  @SuppressWarnings("deprecation")
  public void testDropPartition() throws IOException {
    PartitionStrategy partitionStrategy = new PartitionStrategy.Builder()
      .hash("username", 2).build();


    FileSystemDataset<Record> ds = new FileSystemDataset.Builder<Record>()
        .namespace("ns")
        .name("partitioned-users")

View Full Code Here

Examples of org.kitesdk.data.PartitionStrategy

    Assert.assertNotNull(caught);
  }
  
  @Test
  public void testMerge() throws IOException {
    PartitionStrategy partitionStrategy = new PartitionStrategy.Builder().hash(
        "username", 2).build();


    FileSystemDataset<Record> ds = new FileSystemDataset.Builder<Record>()
        .namespace("ns")
        .name("partitioned-users")

View Full Code Here

Examples of org.kitesdk.data.PartitionStrategy

  }


  @Test
  @SuppressWarnings("deprecation")
  public void testPathIterator_Partition_Directory() {
    PartitionStrategy partitionStrategy = new PartitionStrategy.Builder()
        .hash("username", 2).hash("email", 3).build();


    final FileSystemDataset<Record> ds = new FileSystemDataset.Builder<Record>()
        .namespace("ns")
        .name("partitioned-users")

View Full Code Here

Examples of org.kitesdk.data.PartitionStrategy

        .size());
  }


  @Test
  public void testOverridePartitionStrategy() {
    PartitionStrategy strat = new PartitionStrategy.Builder()
        .hash("keyPart1", "keyPart1", 10).build();
    AvroKeySchema avroKeySchema = parser.parseKeySchema(entitySchema, strat);
    assertEquals(Type.INT, avroKeySchema.getAvroSchema().getField("keyPart1")
        .schema().getType());
    assertEquals(1, avroKeySchema.getPartitionStrategy().getFieldPartitioners()

View Full Code Here

Examples of org.kitesdk.data.PartitionStrategy

    repo.delete("ns", "records"); // clean up
  }


  @Test
  public void testProvidedPartitions() {
    PartitionStrategy expected = new PartitionStrategy.Builder()
        .provided("year", "int")
        .provided("month", "int")
        .provided("day", "long")
        .provided("us_state")
        .build();

View Full Code Here

0 1 2 3 4 5

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.