GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Change register to apply by zhilingc · Pull Request #4 · feast-dev/feast

Change register to apply #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
woop merged 6 commits into master from feast-apply
Dec 13, 2018

Uh oh!

There was an error while loading. Please reload this page.
Merged

Change register to apply #4

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
4337787
Add eq functions for checking diff between specs
Dec 11, 2018
3fb8581
Add apply functions to core service
Dec 12, 2018
5eb5c1d
Update go package to reflect new repository
Dec 12, 2018
3443bae
Change register command to apply
Dec 12, 2018
0c66305
Create audit log objects
Dec 12, 2018
f878d3f
Clean up update code
Dec 13, 2018
File filter
Filter by extension
Clear filters
Conversations
Failed to load comments.
Loading

Uh oh!

There was an error while loading. Please reload this page.
Jump to
Jump to file
Failed to load files.
Loading

Uh oh!

There was an error while loading. Please reload this page.
Diff view
Diff view

There are no files selected for viewing

18 changes: 9 additions & 9 deletions 18 core/src/main/java/feast/core/grpc/CoreServiceImpl.java
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -237,16 +237,16 @@ public void registerFeature(
FeatureSpec request, StreamObserver<RegisterFeatureResponse> responseObserver) {
try {
validator.validateFeatureSpec(request);
FeatureInfo feature = specService.registerFeature(request);
FeatureInfo feature = specService.applyFeature(request);
RegisterFeatureResponse response =
RegisterFeatureResponse.newBuilder().setFeatureId(feature.getId()).build();
responseObserver.onNext(response);
responseObserver.onCompleted();
} catch (RegistrationException e) {
log.error("Error in registerFeature: {}", e);
log.error("Error in applyFeature: {}", e);
responseObserver.onError(getRuntimeException(e));
} catch (IllegalArgumentException e) {
log.error("Error in registerFeature: {}", e);
log.error("Error in applyFeature: {}", e);
responseObserver.onError(getBadRequestException(e));
}
}
Expand All @@ -262,16 +262,16 @@ public void registerFeatureGroup(
StreamObserver<RegisterFeatureGroupResponse> responseObserver) {
try {
validator.validateFeatureGroupSpec(request);
FeatureGroupInfo featureGroup = specService.registerFeatureGroup(request);
FeatureGroupInfo featureGroup = specService.applyFeatureGroup(request);
RegisterFeatureGroupResponse response =
RegisterFeatureGroupResponse.newBuilder().setFeatureGroupId(featureGroup.getId()).build();
responseObserver.onNext(response);
responseObserver.onCompleted();
} catch (RegistrationException e) {
log.error("Error in registerFeatureGroup: {}", e);
log.error("Error in applyFeatureGroup: {}", e);
responseObserver.onError(getRuntimeException(e));
} catch (IllegalArgumentException e) {
log.error("Error in registerFeatureGroup: {}", e);
log.error("Error in applyFeatureGroup: {}", e);
responseObserver.onError(getBadRequestException(e));
}
}
Expand All @@ -286,16 +286,16 @@ public void registerEntity(
EntitySpec request, StreamObserver<RegisterEntityResponse> responseObserver) {
try {
validator.validateEntitySpec(request);
EntityInfo entity = specService.registerEntity(request);
EntityInfo entity = specService.applyEntity(request);
RegisterEntityResponse response =
RegisterEntityResponse.newBuilder().setEntityName(entity.getName()).build();
responseObserver.onNext(response);
responseObserver.onCompleted();
} catch (RegistrationException e) {
log.error("Error in registerEntity: {}", e);
log.error("Error in applyEntity: {}", e);
responseObserver.onError(getRuntimeException(e));
} catch (IllegalArgumentException e) {
log.error("Error in registerEntity: {}", e);
log.error("Error in applyEntity: {}", e);
responseObserver.onError(getBadRequestException(e));
}
}
Expand Down
11 changes: 11 additions & 0 deletions 11 core/src/main/java/feast/core/model/EntityInfo.java
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,4 +86,15 @@ public EntityDetail getEntityDetail() {
.setLastUpdated(convertTimestamp(this.getLastUpdated()))
.build();
}

/**
* Updates the entity info with specifications from the incoming entity spec.
*
* @param update new entity spec
*/
public void update(EntitySpec update) {
this.description = update.getDescription();
this.tags = String.join(",", update.getTagsList());
}

}
89 changes: 57 additions & 32 deletions 89 core/src/main/java/feast/core/model/FeatureGroupInfo.java
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,14 @@

package feast.core.model;

import lombok.AllArgsConstructor;
import lombok.Getter;
import lombok.Setter;
import feast.core.UIServiceProto.UIServiceTypes.FeatureGroupDetail;
import feast.core.util.TypeConversion;
import feast.specs.FeatureGroupSpecProto.FeatureGroupSpec;
import feast.specs.FeatureSpecProto.DataStore;
import feast.specs.FeatureSpecProto.DataStores;
import lombok.AllArgsConstructor;
import lombok.Getter;
import lombok.Setter;

import javax.persistence.*;

Expand All @@ -39,8 +39,7 @@
@Table(name = "feature_groups")
public class FeatureGroupInfo extends AbstractTimestampEntity {

@Id
private String id;
@Id private String id;

@Column(name = "tags")
private String tags;
Expand All @@ -63,52 +62,78 @@ public FeatureGroupInfo() {
super();
}

public FeatureGroupInfo(FeatureGroupSpec spec,
StorageInfo servingStore,
StorageInfo warehouseStore) {
public FeatureGroupInfo(
FeatureGroupSpec spec, StorageInfo servingStore, StorageInfo warehouseStore) {
this.id = spec.getId();
this.tags = String.join(",", spec.getTagsList());
this.servingStore = servingStore;
this.warehouseStore = warehouseStore;
this.servingStoreOpts =
TypeConversion.convertMapToJsonString(spec.getDataStores().getServing().getOptionsMap());
TypeConversion.convertMapToJsonString(spec.getDataStores().getServing().getOptionsMap());
this.warehouseStoreOpts =
TypeConversion.convertMapToJsonString(spec.getDataStores().getWarehouse().getOptionsMap());
TypeConversion.convertMapToJsonString(spec.getDataStores().getWarehouse().getOptionsMap());
}

/**
* Get the feature group spec associated with this record.
*/
/** Get the feature group spec associated with this record. */
public FeatureGroupSpec getFeatureGroupSpec() {
DataStore servingDataStore =
DataStore.newBuilder()
.setId(servingStore.getId())
.putAllOptions(TypeConversion.convertJsonStringToMap(servingStoreOpts))
.build();
DataStore.newBuilder()
.setId(servingStore.getId())
.putAllOptions(TypeConversion.convertJsonStringToMap(servingStoreOpts))
.build();
DataStore warehouseDataStore =
DataStore.newBuilder()
.setId(warehouseStore.getId())
.putAllOptions(TypeConversion.convertJsonStringToMap(warehouseStoreOpts))
.build();
DataStore.newBuilder()
.setId(warehouseStore.getId())
.putAllOptions(TypeConversion.convertJsonStringToMap(warehouseStoreOpts))
.build();
DataStores dataStores =
DataStores.newBuilder()
.setWarehouse(warehouseDataStore)
.setServing(servingDataStore)
.build();
return FeatureGroupSpec.newBuilder()
.setId(id)
.addAllTags(TypeConversion.convertTagStringToList(tags))
.setDataStores(dataStores)
DataStores.newBuilder()
.setWarehouse(warehouseDataStore)
.setServing(servingDataStore)
.build();
return FeatureGroupSpec.newBuilder()
.setId(id)
.addAllTags(TypeConversion.convertTagStringToList(tags))
.setDataStores(dataStores)
.build();
}

/**
* Get the feature group detail containing both spec and metadata, associated with this record.
*/
public FeatureGroupDetail getFeatureGroupDetail() {
return FeatureGroupDetail.newBuilder()
.setSpec(this.getFeatureGroupSpec())
.setLastUpdated(TypeConversion.convertTimestamp(this.getLastUpdated()))
.build();
.setSpec(this.getFeatureGroupSpec())
.setLastUpdated(TypeConversion.convertTimestamp(this.getLastUpdated()))
.build();
}

public void update(FeatureGroupSpec update) throws IllegalArgumentException {
if (!isLegalUpdate(update)) {
throw new IllegalArgumentException(
"Feature group already exists. Update only allowed for fields: [tags]");
}
this.tags = String.join(",", update.getTagsList());
}

private boolean isLegalUpdate(FeatureGroupSpec update) {
DataStore updatedWarehouseStore =
update.getDataStores().hasWarehouse() ? update.getDataStores().getWarehouse() : null;
DataStore updatedServingStore =
update.getDataStores().hasServing() ? update.getDataStores().getServing() : null;
return isStoreEqual(this.warehouseStore, this.warehouseStoreOpts, updatedWarehouseStore)
&& isStoreEqual(this.servingStore, this.servingStoreOpts, updatedServingStore);
}

private boolean isStoreEqual(StorageInfo oldStore, String oldStoreOpts, DataStore newStore) {
return getStorageId(oldStore).equals(newStore == null ? "" : newStore.getId())
&& oldStoreOpts.equals(
newStore == null
? ""
: TypeConversion.convertMapToJsonString(newStore.getOptionsMap()));
Comment thread Outdated

@woop woop Dec 12, 2018

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need to have oldStoreOpts here? Can this also be a method on StorageInfo?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
}

private String getStorageId(StorageInfo storage) {
return storage == null ? "" : storage.getId();
}
}
58 changes: 52 additions & 6 deletions 58 core/src/main/java/feast/core/model/FeatureInfo.java
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,6 @@

package feast.core.model;

import com.google.common.collect.Lists;
import com.google.common.collect.Sets;
import feast.core.UIServiceProto.UIServiceTypes.FeatureDetail;
import feast.core.storage.BigQueryStorageManager;
import feast.core.util.TypeConversion;
Expand All @@ -30,11 +28,9 @@
import lombok.AllArgsConstructor;
import lombok.Getter;
import lombok.Setter;
import org.apache.logging.log4j.util.Strings;

import javax.persistence.*;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

Expand Down Expand Up @@ -133,10 +129,11 @@ public FeatureInfo(
if (spec.getDataStores() != null) {
this.servingStore = servingStore;
this.servingStoreOpts =
TypeConversion.convertMapToJsonString(spec.getDataStores().getServing().getOptionsMap());
TypeConversion.convertMapToJsonString(spec.getDataStores().getServing().getOptionsMap());
this.warehouseStore = warehouseStore;
this.warehouseStoreOpts =
TypeConversion.convertMapToJsonString(spec.getDataStores().getWarehouse().getOptionsMap());
TypeConversion.convertMapToJsonString(
spec.getDataStores().getWarehouse().getOptionsMap());
}
this.bigQueryView = createBigqueryViewLink(warehouseStore);
}
Expand Down Expand Up @@ -256,4 +253,53 @@ private String createBigqueryViewLink(StorageInfo warehouseStore) {
"https://bigquery.cloud.google.com/table/%s:%s.%s_%s_view",
projectId, dataset, entity.getName(), granularity.toString().toLowerCase());
}

/**
* Updates the feature info with specifications from the incoming feature spec.
*
* <p>TODO: maybe allow changes to id, store etc if no jobs are feeding into this feature
*
* @param update new feature spec
*/
public void update(FeatureSpec update) throws IllegalArgumentException {
if (!isLegalUpdate(update)) {
throw new IllegalArgumentException(
"Feature already exists. Update only allowed for fields: [owner, description, uri, tags]");
}
this.owner = update.getOwner();
this.description = update.getDescription();
this.uri = update.getUri();
this.tags = String.join(",", update.getTagsList());
}

private boolean isLegalUpdate(FeatureSpec update) {
DataStore updatedWarehouseStore =
update.getDataStores().hasWarehouse() ? update.getDataStores().getWarehouse() : null;
DataStore updatedServingStore =
update.getDataStores().hasServing() ? update.getDataStores().getServing() : null;
return update.getName().equals(this.name)
&& update.getEntity().equals(this.entity.getName())
&& update.getGranularityValue() == this.granularity.getNumber()
&& update.getValueTypeValue() == this.valueType.getNumber()
&& update.getGroup().equals(getFeatureGroupId(this.featureGroup))
&& TypeConversion.convertMapToJsonString(update.getOptionsMap()).equals(this.options)
&& isStoreEqual(this.warehouseStore, this.warehouseStoreOpts, updatedWarehouseStore)
&& isStoreEqual(this.servingStore, this.servingStoreOpts, updatedServingStore);
}

private boolean isStoreEqual(StorageInfo oldStore, String oldStoreOpts, DataStore newStore) {
return getStorageId(oldStore).equals(newStore == null ? "" : newStore.getId())
&& oldStoreOpts.equals(
newStore == null
? ""
: TypeConversion.convertMapToJsonString(newStore.getOptionsMap()));
}

private String getFeatureGroupId(FeatureGroupInfo featureGroupInfo) {
return featureGroupInfo == null ? "" : featureGroupInfo.getId();
}

private String getStorageId(StorageInfo storage) {
return storage == null ? "" : storage.getId();
}
}
4 changes: 2 additions & 2 deletions 4 core/src/main/java/feast/core/model/StorageInfo.java
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,11 @@

package feast.core.model;

import feast.core.UIServiceProto.UIServiceTypes.StorageDetail;
import feast.specs.StorageSpecProto.StorageSpec;
import lombok.AllArgsConstructor;
import lombok.Getter;
import lombok.Setter;
import feast.core.UIServiceProto.UIServiceTypes.StorageDetail;
import feast.specs.StorageSpecProto.StorageSpec;

import javax.persistence.Column;
import javax.persistence.Entity;
Expand Down
Loading
Oops, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.