job.proto

path mgmt/v1alpha1/job.proto

package mgmt.v1alpha1

---

Messages

ActivityFailure

Name	Type	Description
message	string	The message of the failure

ActivityOptions

Config that contains various timeouts that are configured in the underlying temporal workflow(s) and activities

Name	Type	Description
schedule_to_close_timeout	optional int64	Total time that a workflow is willing to wait for an activity to complete, including retries. Measured in seconds
start_to_close_timeout	optional int64	Max time of a single Temporal Activity execution attempt. This timeout should be as short as the longest psosible execution of any activity (e.g. table sync). Important to know that this is per retry attempt. Defaults to the schedule to close timeout if not provided. Measured in seconds
retry_policy	RetryPolicy	Optionally define a retry policy for the activity If max attempts is not set, the activity will retry indefinitely until the start to close timeout lapses

AiGenerateSourceOptions

Name	Type	Description
ai_connection_id	string	The connection id that corresponds with an AI-based Neosync connection
schemas	repeated AiGenerateSourceSchemaOption	The list of schemas (and their tables) along with any configuration options that will be used to generate data for.
fk_source_connection_id	optional string	An optional connection id that will be used as the basis for the shape of data to be generated.
model_name	string	The name of the model to use
user_prompt	optional string	Optionally provide a user prompt to give more context to the schema
generate_batch_size	optional int64	The batch size that will be used to generate X number of records. This is global and will be applied to all tables configured.

AiGenerateSourceSchemaOption

Name	Type	Description
schema	string	The dataabase schema
tables	repeated AiGenerateSourceTableOption	The list of tables (and their configuration) that reside within the schema to receive generated data

AiGenerateSourceTableOption

Name	Type	Description
table	string	The table that will be used to generate data for
row_count	int64	The total number of records to be generated.

AwsS3DestinationConnectionOptions

Name	Type	Description
storage_class	AwsS3DestinationConnectionOptions.StorageClass	The storage class that will be used when objects are written to S3
max_in_flight	optional uint32	The maximum number of batched messages to have in flight at a given time. Increase this to improve throughput.
timeout	optional string	The maximum period (duration string) to wait on an upload before abandoning it and reattempting.
batch	BatchConfig	Configure batching options to more efficiently store records in S3

AwsS3SourceConnectionOptions

Name	Type	Description
connection_id	string	The unique connection id to a aws s3 connection configuration

BatchConfig

Name	Type	Description
count	optional uint32	The max allowed in a batch before it is flushed. 0 to disable.
period	optional string	A duration string in which an incomplete batch should be flushed regardless of the count. Examples are 1s, 1m, 500ms

CancelJobRunRequest

Name	Type	Description
job_run_id	string	The unique identifier of the job run
account_id	string	The unique identifier of the account

CancelJobRunResponse

Name	Type	Description

ColumnError

Name	Type	Description
schema	string	The schema of the table
table	string	The table of the column
column	string	The column of the error
errors	repeated string	The list of errors

ColumnWarning

Name	Type	Description
schema	string	The schema of the table
table	string	The table of the column
column	string	The column of the warning
warnings	repeated string	The list of warnings

CreateJobDestination

Name	Type	Description
connection_id	string	The connection id to use for the job destination
options	JobDestinationOptions	The destination options to use for the job destination

CreateJobDestinationConnectionsRequest

Name	Type	Description
job_id	string	The unique identifier of the job
destinations	repeated CreateJobDestination	The list of destinations to create and associate with the job

CreateJobDestinationConnectionsResponse

Name	Type	Description
job	Job	The job that was updated

CreateJobHookRequest

Name	Type	Description
job_id	string	The unique identifier of the job
hook	NewJobHook	The new hook configuration

CreateJobHookResponse

Name	Type	Description
hook	JobHook	The newly created hook

CreateJobRequest

Name	Type	Description
account_id	string	The unique account identifier that this job will be associated with
job_name	string	The unique, friendly name of the job. This is unique per account
cron_schedule	optional string	Optionally provide a cron schedule. Goes into effect if the job status is set to enabled
mappings	repeated JobMapping	The list of mappings that will be used to transform the data
source	JobSource	The source connection configuration
destinations	repeated CreateJobDestination	The list of destinations that will be used to store the data
initiate_job_run	bool	Initially trigger a run of this job regardless of its status or cron schedule
workflow_options	WorkflowOptions	Specify timeouts and other workflow options for the underlying temporal workflow
sync_options	ActivityOptions	Specify timeout and retry options for data synchronization activities Data sync activities are any piece of work that involves actually synchronizing data from a source to a destination For the data sync and generate jobs, this will be applied per table
virtual_foreign_keys	repeated VirtualForeignConstraint	The list of virtual foreign keys that will be used to further constrain the data ontop of the database defined constraints

CreateJobResponse

Name	Type	Description
job	Job	The job that was created

CreateJobRunRequest

Name	Type	Description
job_id	string	The unique identifier of the job

CreateJobRunResponse

Name	Type	Description

DatabaseError

Name	Type	Description
errors	repeated string	The list of errors

DeleteJobDestinationConnectionRequest

Name	Type	Description
destination_id	string	The unique identifier of the destination to delete

DeleteJobDestinationConnectionResponse

Name	Type	Description

DeleteJobHookRequest

Name	Type	Description
id	string	The unique identifier of the hook

DeleteJobHookResponse

Name	Type	Description

DeleteJobRequest

Name	Type	Description
id	string	The unique identifier of the job to delete

DeleteJobResponse

Name	Type	Description

DeleteJobRunRequest

Name	Type	Description
job_run_id	string	The unique identifier of the job run
account_id	string	The unique identifier of the account

DeleteJobRunResponse

Name	Type	Description

DynamoDBDestinationConnectionOptions

Configuration for DynamoDB Destination Connection Job Options

Name	Type	Description
table_mappings	repeated DynamoDBDestinationTableMapping	List of table mappings when piping data from a dynamoDB table to another dynamoDB table

DynamoDBDestinationTableMapping

Configuration for mapping a source table to a destination table for DynamoDB

Name	Type	Description
source_table	string	The name of the incoming source table
destination_table	string	The name of the outgoing destination table

DynamoDBSourceConnectionOptions

DynamoDB connection options for a job source

Name	Type	Description
connection_id	string	The unique connection id to a dynamodb connection configuration
tables	repeated DynamoDBSourceTableOption	List of table option configurations for any mapped source table. Any table listed in this must also be present as a job mapping table to be applied.
unmapped_transforms	DynamoDBSourceUnmappedTransformConfig	Default transformations for any unmapped keys
enable_consistent_read	bool	Enforces strong read consistency False: Eventually Consistent Reads, True: Strongly Consistent Reads https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadConsistency.html

DynamoDBSourceSchemaSubset

Name	Type	Description
tables	repeated DynamoDBSourceTableOption	The list of tables (and their configuration) that reside within the schema

DynamoDBSourceTableOption

Name	Type	Description
table	string	The table that this configuration will be applied to
where_clause	optional string	An optional PartiQL query that may be used for subsetting the DynamoDB table. This is not a parameterized query and must be valid. Intended to be everything after the WHERE keyword.

DynamoDBSourceUnmappedTransformConfig

Name	Type	Description
b	JobMappingTransformer	Byte
boolean	JobMappingTransformer	Boolean
n	JobMappingTransformer	Number
s	JobMappingTransformer	String

GcpCloudStorageDestinationConnectionOptions

Configuration for Google Cloud Storage Destination Connection Job Options

Name	Type	Description

GenerateSourceOptions

Name	Type	Description
schemas	repeated GenerateSourceSchemaOption	The list of schemas (and their tables) along with any configuration options that will be used to generate data for.
fk_source_connection_id	optional string	An optional connection id that will be used as the basis for the shape of data to be generated.

GenerateSourceSchemaOption

Name	Type	Description
schema	string	The database schema
tables	repeated GenerateSourceTableOption	The list of tables (and their configuration) that reside within the schema to receive generated data

GenerateSourceTableOption

Name	Type	Description
table	string	The table that will be used to generate data for.
row_count	int64	The total number of records to be generated.

GetActiveJobHooksByTimingRequest

Name	Type	Description
job_id	string	The unique identifier of the job
timing	GetActiveJobHooksByTimingRequest.Timing	The timing desired.

GetActiveJobHooksByTimingResponse

Name	Type	Description
hooks	repeated JobHook	The active job hooks

GetJobHookRequest

Name	Type	Description
id	string	The unique identifier of the hook

GetJobHookResponse

Name	Type	Description
hook	JobHook	The found hook

GetJobHooksRequest

Name	Type	Description
job_id	string	The unique identifier of the job

GetJobHooksResponse

Name	Type	Description
hooks	repeated JobHook	The list of hooks found attached to the job

GetJobNextRunsRequest

Name	Type	Description
job_id	string	The unique identifier of the job

GetJobNextRunsResponse

Name	Type	Description
next_runs	JobNextRuns	The list of next run times

GetJobRecentRunsRequest

Name	Type	Description
job_id	string	The unique identifier of the job

GetJobRecentRunsResponse

Name	Type	Description
recent_runs	repeated JobRecentRun	The list of recent job runs

GetJobRequest

Name	Type	Description
id	string	The unique identifier of the job

GetJobResponse

Name	Type	Description
job	Job	The job that was retrieved

GetJobRunEventsRequest

Name	Type	Description
job_run_id	string	The unique identifier of the job run
account_id	string	The unique identifier of the account

GetJobRunEventsResponse

Name	Type	Description
events	repeated JobRunEvent	The list of events
is_run_complete	bool	Whether the run is complete

GetJobRunLogsStreamRequest

Name	Type	Description
job_run_id	string	The unique identifier of the job run
account_id	string	The unique identifier of the account
window	LogWindow	The time window in which to retrieve the logs
should_tail	bool	Whether or not to tail the stream. Note: only works with k8s-pods and is not currently supported with Loki logs
max_log_lines	optional int64	Optionally provide a max log limit
log_levels	repeated LogLevel	Provide a list of log levels to filter by. If any of these are UNSPECIFIED, all log levels are returned.

GetJobRunLogsStreamResponse

Name	Type	Description
log_line	string	The log line
timestamp	optional google.protobuf.Timestamp	The timestamp of the log line

GetJobRunRequest

Name	Type	Description
job_run_id	string	The unique identifier of the job run
account_id	string	The unique identifier of the account

GetJobRunResponse

Name	Type	Description
job_run	JobRun	The job run that was retrieved

GetJobRunsRequest

Name	Type	Description
job_id	string	Retireve runs for a specific job
account_id	string	Retrieve runs for all jobs in an account

GetJobRunsResponse

Name	Type	Description
job_runs	repeated JobRun	The list of job runs

GetJobStatusRequest

Name	Type	Description
job_id	string	The unique identifier of the job

GetJobStatusResponse

Name	Type	Description
status	JobStatus	The status of the job

GetJobStatusesRequest

Name	Type	Description
account_id	string	The unique identifier of the account

GetJobStatusesResponse

Name	Type	Description
statuses	repeated JobStatusRecord	The list of job statuses

GetJobsRequest

Name	Type	Description
account_id	string	The unique identifier of the account to get jobs for

GetJobsResponse

Name	Type	Description
jobs	repeated Job	The list of jobs

GetRunContextRequest

Name	Type	Description
id	RunContextKey	The run context key

GetRunContextResponse

Name	Type	Description
value	bytes	The returned value in bytes. The format is determined by the key when it is set.

IsJobHookNameAvailableRequest

Name	Type	Description
job_id	string	The unique identifier of the job
name	string	The hook name to check.

IsJobHookNameAvailableResponse

Name	Type	Description
is_available	bool	Whether or not the hook name is available

IsJobNameAvailableRequest

Name	Type	Description
name	string	The name to check for availability
account_id	string	The unique identifier of the account to check for availability

IsJobNameAvailableResponse

Name	Type	Description
is_available	bool	Whether the name is available

Job

Name	Type	Description
id	string	The unique identifier of the job
created_by_user_id	string
created_at	google.protobuf.Timestamp
updated_by_user_id	string
updated_at	google.protobuf.Timestamp
name	string	The unique, friendly name of the job
source	JobSource	The source connection configuration
destinations	repeated JobDestination	The list of destinations that will be used to store the data
mappings	repeated JobMapping	The list of mappings that will be used to transform the data
cron_schedule	optional string	The cron schedule that will be used to trigger the job
account_id	string	The account identifier that a job is associated with
sync_options	ActivityOptions	Specify timeout and retry options for data synchronization activities Data sync activities are any piece of work that involves actually synchronizing data from a source to a destination For the data sync and generate jobs, this will be applied per table
workflow_options	WorkflowOptions	Specify timeouts and other workflow options for the underlying temporal workflow
virtual_foreign_keys	repeated VirtualForeignConstraint	Any virtual foreign keys that are configured as a part of this job

JobDestination

Name	Type	Description
connection_id	string	The connection id to use for the job destination
options	JobDestinationOptions	The destination options to use for the job destination
id	string	The unique identifier of the job destination

JobDestinationOptions

Name	Type	Description
postgres_options	PostgresDestinationConnectionOptions
aws_s3_options	AwsS3DestinationConnectionOptions
mysql_options	MysqlDestinationConnectionOptions
mongodb_options	MongoDBDestinationConnectionOptions
gcp_cloudstorage_options	GcpCloudStorageDestinationConnectionOptions	Destination Connecton options for Google Cloud Storage
dynamodb_options	DynamoDBDestinationConnectionOptions	Destination Connection options for DynamoDB
mssql_options	MssqlDestinationConnectionOptions	Destination Connection options for Microsoft SQL Server

JobHook

Name	Type	Description
id	string	The unique identifier of this hook.
name	string	Name of the hook for display/reference.
description	string	Description of what this hook does.
job_id	string	The unique identifier of the job this hook belongs to.
config	JobHookConfig	Hook-type specific configuration.
created_by_user_id	string	The user that created this hook.
created_at	google.protobuf.Timestamp	The time this hook was created.
updated_by_user_id	string	The user that last updated this hook.
updated_at	google.protobuf.Timestamp	The last time this hook was updated.
enabled	bool	Whether or not the hook is enabled.
priority	uint32	The priority of the hook (0-100). This determines the execution order. Lower values are higher priority (priority=0 is the highest). Tie Breaking is determined by the following: (priority, created_at, id) in ascending order.

JobHookConfig

Contains the specific hook type configurations.

Name	Type	Description
sql	JobHookConfig.JobSqlHook	Configuration for SQL-specific hooks.

JobHookConfig.JobSqlHook

Configuration for SQL-based hooks

Name	Type	Description
query	string	The SQL query to execute
connection_id	string	Unique identifier of the Neosync connection to run this hook for. Must be a connection id that is present in the job.
timing	JobHookConfig.JobSqlHook.Timing	The timing of when the hook will run

JobHookConfig.JobSqlHook.Timing

Name	Type	Description
pre_sync	JobHookTimingPreSync	A Pre-Sync timing. Will run before the first table sync. Will run before Truncation, if enabled. Will run before Schema Init, if enabled.
post_sync	JobHookTimingPostSync	A Post-Sync timing. Will run after the last table sync.

JobHookTimingPostSync

Configures the job hook to run after the last table sync.

Name	Type	Description

JobHookTimingPreSync

Configures the job hook to run before the first table sync. Will run before Truncation, if enabled. Will run before Schema Init, if enabled.

Name	Type	Description

JobMapping

Name	Type	Description
schema	string	The database schema
table	string	The database table.
column	string	The column in the configured schema.table
transformer	JobMappingTransformer	The transformer configuration that will be applied to each cell in the column

JobMappingTransformer

Name	Type	Description
config	TransformerConfig	The transformer configuration

JobNextRuns

Name	Type	Description
next_run_times	repeated google.protobuf.Timestamp	The list of next run times

JobRecentRun

Name	Type	Description
start_time	google.protobuf.Timestamp	The start time of the job run
job_run_id	string	The unique identifier of the job run

JobRun

Name	Type	Description
id	string	The id of the job run. This will currently be equivalent to the temporal workflow id
job_id	string	The unique identifier of the job id this run is associated with
name	string	The name of the job run.
status	JobRunStatus	the status of the job run
started_at	google.protobuf.Timestamp	A timestamp of when the run started
completed_at	optional google.protobuf.Timestamp	Available if the run completed or has not yet been archived by the system
pending_activities	repeated PendingActivity	Pending activities are only returned when retrieving a specific job run and will not be returned when requesting job runs in list format

JobRunEvent

Name	Type	Description
id	int64	The unique identifier of the event
type	string	The type of the event
start_time	google.protobuf.Timestamp	The start time of the event
close_time	google.protobuf.Timestamp	The close time of the event
metadata	JobRunEventMetadata	The metadata of the event
tasks	repeated JobRunEventTask	The list of tasks associated with the event

JobRunEventMetadata

Name	Type	Description
sync_metadata	JobRunSyncMetadata	The metadata of the sync

JobRunEventTask

Name	Type	Description
id	int64	The unique identifier of the task
type	string	The type of the task
event_time	google.protobuf.Timestamp	The time of the event
error	JobRunEventTaskError	The error of the task

JobRunEventTaskError

Name	Type	Description
message	string	The message of the error
retry_state	string	The retry state of the error

JobRunSyncMetadata

Name	Type	Description
schema	string	The schema of the table
table	string	The table of the sync

JobSource

Name	Type	Description
options	JobSourceOptions	The source options to use for the job

JobSourceOptions

Name	Type	Description
postgres	PostgresSourceConnectionOptions
aws_s3	AwsS3SourceConnectionOptions
mysql	MysqlSourceConnectionOptions
generate	GenerateSourceOptions
ai_generate	AiGenerateSourceOptions
mongodb	MongoDBSourceConnectionOptions
dynamodb	DynamoDBSourceConnectionOptions
mssql	MssqlSourceConnectionOptions

JobSourceSqlSubetSchemas

Name	Type	Description
postgres_subset	PostgresSourceSchemaSubset	The list of schemas (and their tables) along with any configuration options that will be used.
mysql_subset	MysqlSourceSchemaSubset	The list of schemas (and their tables) along with any configuration options that will be used.
dynamodb_subset	DynamoDBSourceSchemaSubset	The list of tables (and their configuration) that reside within the schema
mssql_subset	MssqlSourceSchemaSubset	The list of schemas (and their tables) along with any configuration options that will be used.

JobStatusRecord

Name	Type	Description
job_id	string	The unique identifier of the job
status	JobStatus	The status of the job

MongoDBDestinationConnectionOptions

Name	Type	Description

MongoDBSourceConnectionOptions

MongoDB connection options for a job source

Name	Type	Description
connection_id	string	The unique connection id to a mongo connection configuration

MssqlDestinationConnectionOptions

Name	Type	Description
truncate_table	MssqlTruncateTableConfig	Whether to truncate the table before inserting data
init_table_schema	bool	Whether to initialize the table schema before inserting data
on_conflict	MssqlOnConflictConfig	The configuration for handling conflicts when inserting data
skip_foreign_key_violations	bool	Insert all valid records, skipping any that violate foreign key constraints.
batch	BatchConfig	Configure batching options to handle how much data is sent to your database at once.
max_in_flight	optional uint32	Determines the maximum number of parallel batched inserts.

MssqlOnConflictConfig

Name	Type	Description
do_nothing	bool	Whether to do nothing when a conflict occurs

MssqlSourceConnectionOptions

Name	Type	Description
halt_on_new_column_addition	bool	Whether to halt the job if a new column is added
schemas	repeated MssqlSourceSchemaOption	The list of schemas (and their tables) along with any configuration options that will be used.
connection_id	string	The unique connection id to a mssql connection configuration
subset_by_foreign_key_constraints	bool	Whether to subset the table by foreign key constraints
column_removal_strategy	MssqlSourceConnectionOptions.ColumnRemovalStrategy	Provide a strategy of what to do in the event Neosync encounters a column that is removed from the source table.

MssqlSourceConnectionOptions.ColumnRemovalStrategy

Name	Type	Description
halt_job	MssqlSourceConnectionOptions.ColumnRemovalStrategy.HaltJob	halt job if a column is removed
continue_job	MssqlSourceConnectionOptions.ColumnRemovalStrategy.ContinueJob	continue job if a column is removed

MssqlSourceConnectionOptions.ColumnRemovalStrategy.ContinueJob

Configuration for the ContinueJob strategy

Name	Type	Description

MssqlSourceConnectionOptions.ColumnRemovalStrategy.HaltJob

Configuration for the HaltJob strategy

Name	Type	Description

MssqlSourceSchemaOption

Name	Type	Description
schema	string	The database schema
tables	repeated MssqlSourceTableOption	The list of tables (and their configuration) that reside within the schema

MssqlSourceSchemaSubset

Name	Type	Description
mssql_schemas	repeated MssqlSourceSchemaOption	The list of schemas (and their tables) along with any configuration options that will be used.

MssqlSourceTableOption

Name	Type	Description
table	string	The table that will be used to subset the data for.
where_clause	optional string	This is not a parameterized query and must be valid. Intended to be everything after the WHERE keyword. Is used to subset the table.

MssqlTruncateTableConfig

Name	Type	Description
truncate_before_insert	bool	Whether to truncate the table before inserting data

MysqlDestinationConnectionOptions

Name	Type	Description
truncate_table	MysqlTruncateTableConfig	Whether to truncate the table before inserting data
init_table_schema	bool	Whether to initialize the table schema before inserting data
on_conflict	MysqlOnConflictConfig	The configuration for handling conflicts when inserting data
skip_foreign_key_violations	bool	Insert all valid records, skipping any that violate foreign key constraints.
batch	BatchConfig	Configure batching options to handle how much data is sent to your database at once.
max_in_flight	optional uint32	Determines the maximum number of parallel batched inserts.

MysqlOnConflictConfig

Name	Type	Description
do_nothing	bool	@deprecated - Use strategy nothing instead
nothing	MysqlOnConflictConfig.MysqlOnConflictDoNothing	Do nothing when a conflict occurs
update	MysqlOnConflictConfig.MysqlOnConflictUpdate	Update columns when a conflict occurs

MysqlOnConflictConfig.MysqlOnConflictDoNothing

Do nothing strategy

Name	Type	Description

MysqlOnConflictConfig.MysqlOnConflictUpdate

Update strategy // List of table-specific update configurations repeated MysqlOnConflictSchemaUpdate schemas = 1;

Name	Type	Description

MysqlSourceConnectionOptions

Name	Type	Description
halt_on_new_column_addition	bool	Whether to halt the job if a new column is added
schemas	repeated MysqlSourceSchemaOption	The list of schemas (and their tables) along with any configuration options that will be used.
connection_id	string	The unique connection id to a mysql connection configuration
subset_by_foreign_key_constraints	bool	Whether to subset the table by foreign key constraints
column_removal_strategy	MysqlSourceConnectionOptions.ColumnRemovalStrategy	Provide a strategy of what to do in the event Neosync encounters a column that is removed from the source table.

MysqlSourceConnectionOptions.ColumnRemovalStrategy

Name	Type	Description
halt_job	MysqlSourceConnectionOptions.ColumnRemovalStrategy.HaltJob	halt job if a column is removed
continue_job	MysqlSourceConnectionOptions.ColumnRemovalStrategy.ContinueJob	continue job if a column is removed

MysqlSourceConnectionOptions.ColumnRemovalStrategy.ContinueJob

Configuration for the ContinueJob strategy

Name	Type	Description

MysqlSourceConnectionOptions.ColumnRemovalStrategy.HaltJob

Configuration for the HaltJob strategy

Name	Type	Description

MysqlSourceSchemaOption

Name	Type	Description
schema	string	The database schema
tables	repeated MysqlSourceTableOption	The list of tables (and their configuration) that reside within the schema

MysqlSourceSchemaSubset

Name	Type	Description
mysql_schemas	repeated MysqlSourceSchemaOption	The list of schemas (and their tables) along with any configuration options that will be used.

MysqlSourceTableOption

Name	Type	Description
table	string	The table that will be used to subset the data for.
where_clause	optional string	This is not a parameterized query and must be valid. Intended to be everything after the WHERE keyword. Is used to subset the table.

MysqlTruncateTableConfig

Name	Type	Description
truncate_before_insert	bool	Whether to truncate the table before inserting data

NewJobHook

Name	Type	Description
name	string	Name of the hook for display/reference.
description	string	Description of what this hook does.
config	JobHookConfig	Hook-type specific configuration.
enabled	bool	Whether or not the hook is enabled.
priority	uint32	The priority of the hook (0-100). This determines the execution order. Lower values are higher priority (priority=0 is the highest). Tie Breaking is determined by the following: (priority, created_at, id) in ascending order.

PauseJobRequest

Name	Type	Description
id	string	The unique identifier of the job
pause	bool	Whether to pause or unpause the job
note	optional string	An optional note to be associated with the pause

PauseJobResponse

Name	Type	Description
job	Job	The job that was updated

PendingActivity

Name	Type	Description
status	ActivityStatus	The status of the activity
activity_name	string	The name of the activity
last_failure	optional ActivityFailure	The last failure of the activity

PostgresDestinationConnectionOptions

Name	Type	Description
truncate_table	PostgresTruncateTableConfig	Whether to truncate the table before inserting data
init_table_schema	bool	Whether to initialize the table schema before inserting data
on_conflict	PostgresOnConflictConfig	The configuration for handling conflicts when inserting data
skip_foreign_key_violations	bool	Whether to skip records that violate foreign key constraints
batch	BatchConfig	Configure batching options to handle how much data is sent to your database at once.
max_in_flight	optional uint32	Determines the maximum number of parallel batched inserts.

PostgresOnConflictConfig

Name	Type	Description
do_nothing	bool	@deprecated - Use strategy nothing instead
nothing	PostgresOnConflictConfig.PostgresOnConflictDoNothing	Do nothing when a conflict occurs
update	PostgresOnConflictConfig.PostgresOnConflictUpdate	Update columns when a conflict occurs

PostgresOnConflictConfig.PostgresOnConflictDoNothing

Do nothing strategy

Name	Type	Description

PostgresOnConflictConfig.PostgresOnConflictUpdate

Update strategy // List of table-specific update configurations repeated PostgresOnConflictSchemaUpdate schemas = 1;

Name	Type	Description

PostgresSourceConnectionOptions

Name	Type	Description
schemas	repeated PostgresSourceSchemaOption	The list of schemas (and their tables) along with any configuration options that will be used.
connection_id	string	The unique connection id to a postgres connection configuration
subset_by_foreign_key_constraints	bool	Whether to subset the table by foreign key constraints
new_column_addition_strategy	PostgresSourceConnectionOptions.NewColumnAdditionStrategy	Provide a strategy of what to do in the event Neosync encounters an unmapped column for the job's mapped tables.
column_removal_strategy	PostgresSourceConnectionOptions.ColumnRemovalStrategy	Provide a strategy of what to do in the event Neosync encounters a column that is removed from the source table.

PostgresSourceConnectionOptions.ColumnRemovalStrategy

Name	Type	Description
halt_job	PostgresSourceConnectionOptions.ColumnRemovalStrategy.HaltJob	halt job if a column is removed
continue_job	PostgresSourceConnectionOptions.ColumnRemovalStrategy.ContinueJob	continue job if a column is removed

PostgresSourceConnectionOptions.ColumnRemovalStrategy.ContinueJob

Configuration for the ContinueJob strategy

Name	Type	Description

PostgresSourceConnectionOptions.ColumnRemovalStrategy.HaltJob

Configuration for the HaltJob strategy

Name	Type	Description

PostgresSourceConnectionOptions.NewColumnAdditionStrategy

Name	Type	Description
halt_job	PostgresSourceConnectionOptions.NewColumnAdditionStrategy.HaltJob	halt job if a new column is detected.
auto_map	PostgresSourceConnectionOptions.NewColumnAdditionStrategy.AutoMap	automatically handle unmapped columns. It handles this by using the DBs default/nullable values. If this doesn't exist, will fall back to configuring generators for supported datatypes. If none of the criteria above can be met, the job run will fail to prevent leaking of PII.

PostgresSourceConnectionOptions.NewColumnAdditionStrategy.AutoMap

Configuration for the AutoMap strategy

Name	Type	Description

PostgresSourceConnectionOptions.NewColumnAdditionStrategy.HaltJob

Configuration for the HaltJob strategy

Name	Type	Description

PostgresSourceSchemaOption

Name	Type	Description
schema	string	The database schema
tables	repeated PostgresSourceTableOption	The list of tables (and their configuration) that reside within the schema

PostgresSourceSchemaSubset

Name	Type	Description
postgres_schemas	repeated PostgresSourceSchemaOption	The list of schemas (and their tables) along with any configuration options that will be used.

PostgresSourceTableOption

Name	Type	Description
table	string	The table that will be used subset the data for.
where_clause	optional string	This is not a parameterized query and must be valid. Intended to be everything after the WHERE keyword. Is used to subset the table.

PostgresTruncateTableConfig

Name	Type	Description
truncate_before_insert	bool	Whether to truncate the table before inserting data
cascade	bool	Whether to cascade the truncate to child tables

RetryPolicy

Defines the retry policy for an activity

Name	Type	Description
maximum_attempts	optional int32	Maximum number of attempts. When exceeded the retries stop even if not expired yet. If not set or set to 0, it means unlimited, and rely on activity ScheduleToCloseTimeout to stop.

RunContextKey

Name	Type	Description
job_run_id	string	The Neosync Run ID
external_id	string	An opaque identifier that will be used to store specific items
account_id	string	The Neosync Account ID

SetJobHookEnabledRequest

Name	Type	Description
id	string	The unique identifier of the hook
enabled	bool	Whether or not the hook is enabled.

SetJobHookEnabledResponse

Name	Type	Description
hook	JobHook	The updated job hook

SetJobSourceSqlConnectionSubsetsRequest

Name	Type	Description
id	string	The unique identifier of the job to update subsets for
schemas	JobSourceSqlSubetSchemas	The subset configuration
subset_by_foreign_key_constraints	bool	Whether or not to have subsets follow foreign key constraints (for connections that support it)

SetJobSourceSqlConnectionSubsetsResponse

Name	Type	Description
job	Job	The job that was updated

SetJobSyncOptionsRequest

Name	Type	Description
id	string	The unique identifier of the job
sync_options	ActivityOptions	The sync options object. The entire object must be provided and will fully overwrite the previous result

SetJobSyncOptionsResponse

Name	Type	Description
job	Job	The updated job

SetJobWorkflowOptionsRequest

Name	Type	Description
id	string	The unique identifier of the job
worfklow_options	WorkflowOptions	The workflow options object. The entire object must be provided and will fully overwrite the previous result

SetJobWorkflowOptionsResponse

Name	Type	Description
job	Job	The updated job

SetRunContextRequest

Name	Type	Description
id	RunContextKey	The run context key
value	bytes	An opaque value that is to be determined by the key

SetRunContextResponse

Name	Type	Description

SetRunContextsRequest

Name	Type	Description
id	RunContextKey	The run context key
value	bytes	An opaque value that is to be determined by the key

SetRunContextsResponse

Name	Type	Description

TerminateJobRunRequest

Name	Type	Description
job_run_id	string	The unique identifier of the job run
account_id	string	The unique identifier of the account

TerminateJobRunResponse

Name	Type	Description

UpdateJobDestinationConnectionRequest

Name	Type	Description
job_id	string	The unique identifier of the job
connection_id	string	The unique identifier of the connection
options	JobDestinationOptions	The destination connection options
destination_id	string	The unique identifier of the destination

UpdateJobDestinationConnectionResponse

Name	Type	Description
job	Job	The job that was updated

UpdateJobHookRequest

Name	Type	Description
id	string	The unique identifier of the hook
name	string	Name of the hook for display/reference.
description	string	Description of what this hook does.
config	JobHookConfig	Hook-type specific configuration.
enabled	bool	Whether or not the hook is enabled.
priority	uint32	The priority of the hook (0-100). This determines the execution order. Lower values are higher priority (priority=0 is the highest). Tie Breaking is determined by the following: (priority, created_at, id) in ascending order.

UpdateJobHookResponse

Name	Type	Description
hook	JobHook	The updated job hook

UpdateJobScheduleRequest

Name	Type	Description
id	string	The unique identifier of the job
cron_schedule	optional string	The new cron schedule

UpdateJobScheduleResponse

Name	Type	Description
job	Job	The job that was updated

UpdateJobSourceConnectionRequest

Name	Type	Description
id	string	The unique identifier of the job
source	JobSource	The new source connection configuration
mappings	repeated JobMapping	The new list of mappings that will be used to transform the data
virtual_foreign_keys	repeated VirtualForeignConstraint	The new list of virtual foreign keys that will be used to further constrain the data ontop of the database defined constraints

UpdateJobSourceConnectionResponse

Name	Type	Description
job	Job

ValidateJobMappingsRequest

Name	Type	Description
account_id	string	The unique identifier of the account
mappings	repeated JobMapping	The list of mappings to validate
connection_id	string	The unique identifier of the connection
virtual_foreign_keys	repeated VirtualForeignConstraint	The list of virtual foreign keys
job_source	optional JobSource	The source options of the job

ValidateJobMappingsResponse

Name	Type	Description
column_errors	repeated ColumnError	The list of column errors
database_errors	DatabaseError	The database error
column_warnings	repeated ColumnWarning	The list of column warnings

VirtualForeignConstraint

Name	Type	Description
schema	string	The schema of the table
table	string	The table of the virtual foreign key
columns	repeated string	The list of columns
foreign_key	VirtualForeignKey	The foreign key

VirtualForeignKey

Name	Type	Description
schema	string	The schema of the table
table	string	The table of the virtual foreign key
columns	repeated string	The list of columns

WorkflowOptions

Config that contains various timeouts that are configured in the underlying temporal workflow More options will come in the future as needed

Name	Type	Description
run_timeout	optional int64	The timeout for a single workflow run. Measured in seconds

---

Enums

ActivityStatus

Name	Number	Description
ACTIVITY_STATUS_UNSPECIFIED	0
ACTIVITY_STATUS_SCHEDULED	1
ACTIVITY_STATUS_STARTED	2
ACTIVITY_STATUS_CANCELED	3
ACTIVITY_STATUS_FAILED	4

AwsS3DestinationConnectionOptions.StorageClass

Name	Number	Description
STORAGE_CLASS_UNSPECIFIED	0
STORAGE_CLASS_STANDARD	1
STORAGE_CLASS_REDUCED_REDUNDANCY	2
STORAGE_CLASS_GLACIER	3
STORAGE_CLASS_STANDARD_IA	4
STORAGE_CLASS_ONEZONE_IA	5
STORAGE_CLASS_INTELLIGENT_TIERING	6
STORAGE_CLASS_DEEP_ARCHIVE	7

GetActiveJobHooksByTimingRequest.Timing

Name	Number	Description
TIMING_UNSPECIFIED	0	If unspecified, returns all active job hooks
TIMING_PRESYNC	1	Only returns presync hooks
TIMING_POSTSYNC	2	Only returns postsync hooks

JobRunStatus

An enumeration of job run statuses.

Name	Number	Description
JOB_RUN_STATUS_UNSPECIFIED	0	if the job run status is unknown
JOB_RUN_STATUS_PENDING	1	the run is pending and has not started yet
JOB_RUN_STATUS_RUNNING	2	the run is currently in progress
JOB_RUN_STATUS_COMPLETE	3	the run has successfully completed
JOB_RUN_STATUS_ERROR	4	the run ended with an error
JOB_RUN_STATUS_CANCELED	5	the run was cancelled
JOB_RUN_STATUS_TERMINATED	6	the run was terminated
JOB_RUN_STATUS_FAILED	7	the run ended in failure
JOB_RUN_STATUS_TIMED_OUT	8	the run was ended pre-maturely due to timeout

JobStatus

Name	Number	Description
JOB_STATUS_UNSPECIFIED	0
JOB_STATUS_ENABLED	1
JOB_STATUS_PAUSED	3
JOB_STATUS_DISABLED	4

LogLevel

Name	Number	Description
LOG_LEVEL_UNSPECIFIED	0
LOG_LEVEL_DEBUG	1
LOG_LEVEL_INFO	2
LOG_LEVEL_WARN	3
LOG_LEVEL_ERROR	4

LogWindow

Name	Number	Description
LOG_WINDOW_NO_TIME_UNSPECIFIED	0
LOG_WINDOW_FIFTEEN_MIN	1
LOG_WINDOW_ONE_HOUR	2
LOG_WINDOW_ONE_DAY	3

---

Services

JobService

Service that handles jobs, runs, and hooks

GetJobs

Method	GetJobs
Request	GetJobsRequest
Response	GetJobsResponse
Description	Returns a list of jobs by either account or job

GetJob

Method	GetJob
Request	GetJobRequest
Response	GetJobResponse
Description	Returns a specific job

CreateJob

Method	CreateJob
Request	CreateJobRequest
Response	CreateJobResponse
Description	Creates a new job

DeleteJob

Method	DeleteJob
Request	DeleteJobRequest
Response	DeleteJobResponse
Description	Deletes a job

IsJobNameAvailable

Method	IsJobNameAvailable
Request	IsJobNameAvailableRequest
Response	IsJobNameAvailableResponse
Description	Checks if a job name is available

UpdateJobSchedule

Method	UpdateJobSchedule
Request	UpdateJobScheduleRequest
Response	UpdateJobScheduleResponse
Description	Updates the schedule of a job

UpdateJobSourceConnection

Method	UpdateJobSourceConnection
Request	UpdateJobSourceConnectionRequest
Response	UpdateJobSourceConnectionResponse
Description	Updates the source connection of a job

SetJobSourceSqlConnectionSubsets

Method	SetJobSourceSqlConnectionSubsets
Request	SetJobSourceSqlConnectionSubsetsRequest
Response	SetJobSourceSqlConnectionSubsetsResponse
Description	Sets the source sql connection subsets of a job

UpdateJobDestinationConnection

Method	UpdateJobDestinationConnection
Request	UpdateJobDestinationConnectionRequest
Response	UpdateJobDestinationConnectionResponse
Description	Updates the destination connection of a job

DeleteJobDestinationConnection

Method	DeleteJobDestinationConnection
Request	DeleteJobDestinationConnectionRequest
Response	DeleteJobDestinationConnectionResponse
Description	Deletes the destination connection of a job

CreateJobDestinationConnections

Method	CreateJobDestinationConnections
Request	CreateJobDestinationConnectionsRequest
Response	CreateJobDestinationConnectionsResponse
Description	Creates the destination connections of a job

PauseJob

Method	PauseJob
Request	PauseJobRequest
Response	PauseJobResponse
Description	Pauses or unpauses a job

GetJobRecentRuns

Method	GetJobRecentRuns
Request	GetJobRecentRunsRequest
Response	GetJobRecentRunsResponse
Description	Returns a list of recently invoked job runs based on the Temporal cron scheduler. This will return a list of job runs that include archived runs

GetJobNextRuns

Method	GetJobNextRuns
Request	GetJobNextRunsRequest
Response	GetJobNextRunsResponse
Description	Returns a list of runs that are scheduled for execution based on the Temporal cron scheduler.

GetJobStatus

Method	GetJobStatus
Request	GetJobStatusRequest
Response	GetJobStatusResponse
Description	Returns the status of a job

GetJobStatuses

Method	GetJobStatuses
Request	GetJobStatusesRequest
Response	GetJobStatusesResponse
Description	Returns the statuses of jobs within an account

GetJobRuns

Method	GetJobRuns
Request	GetJobRunsRequest
Response	GetJobRunsResponse
Description	Returns a list of job runs by either account or job

GetJobRunEvents

Method	GetJobRunEvents
Request	GetJobRunEventsRequest
Response	GetJobRunEventsResponse
Description	Returns a list of events for a job run to understand more details of the run itself

GetJobRun

Method	GetJobRun
Request	GetJobRunRequest
Response	GetJobRunResponse
Description	Returns a specific job run, along with any of its pending activities

DeleteJobRun

Method	DeleteJobRun
Request	DeleteJobRunRequest
Response	DeleteJobRunResponse
Description	Deletes a job run

CreateJobRun

Method	CreateJobRun
Request	CreateJobRunRequest
Response	CreateJobRunResponse
Description	Creates a new job run

CancelJobRun

Method	CancelJobRun
Request	CancelJobRunRequest
Response	CancelJobRunResponse
Description	Cancels a job run. This is a graceful termination and allows the workflow to clean up and exit gracefully.

TerminateJobRun

Method	TerminateJobRun
Request	TerminateJobRunRequest
Response	TerminateJobRunResponse
Description	Terminates a job run. This is an immediate termination and will not allow the workflow to clean up and exit gracefully.

GetJobRunLogsStream

Method	GetJobRunLogsStream
Request	GetJobRunLogsStreamRequest
Response	GetJobRunLogsStreamResponse stream
Description	Returns a stream of logs from the worker nodes that pertain to a specific job run

SetJobWorkflowOptions

Method	SetJobWorkflowOptions
Request	SetJobWorkflowOptionsRequest
Response	SetJobWorkflowOptionsResponse
Description	Set any job workflow options. Must provide entire object as is it will fully override the previous configuration

SetJobSyncOptions

Method	SetJobSyncOptions
Request	SetJobSyncOptionsRequest
Response	SetJobSyncOptionsResponse
Description	Set the job sync options. Must provide entire object as it will fully override the previous configuration

ValidateJobMappings

Method	ValidateJobMappings
Request	ValidateJobMappingsRequest
Response	ValidateJobMappingsResponse
Description	Validates that the jobmapping configured can run with table constraints

GetRunContext

Method	GetRunContext
Request	GetRunContextRequest
Response	GetRunContextResponse
Description	Gets a run context to be used by a workflow run

SetRunContext

Method	SetRunContext
Request	SetRunContextRequest
Response	SetRunContextResponse
Description	Sets a run context to be used by a workflow run

SetRunContexts

Method	SetRunContexts
Request	SetRunContextsRequest stream
Response	SetRunContextsResponse
Description	Sets a stream of run contexts to be used by a workflow run

GetJobHooks

Method	GetJobHooks
Request	GetJobHooksRequest
Response	GetJobHooksResponse
Description	Retrieves all job hooks

GetJobHook

Method	GetJobHook
Request	GetJobHookRequest
Response	GetJobHookResponse
Description	Retrieves a specific job hook

CreateJobHook

Method	CreateJobHook
Request	CreateJobHookRequest
Response	CreateJobHookResponse
Description	Creates a new job hook

DeleteJobHook

Method	DeleteJobHook
Request	DeleteJobHookRequest
Response	DeleteJobHookResponse
Description	Removes a job hook

IsJobHookNameAvailable

Method	IsJobHookNameAvailable
Request	IsJobHookNameAvailableRequest
Response	IsJobHookNameAvailableResponse
Description	Check if a specific job hook name is available

UpdateJobHook

Method	UpdateJobHook
Request	UpdateJobHookRequest
Response	UpdateJobHookResponse
Description	Updates a job hook

SetJobHookEnabled

Method	SetJobHookEnabled
Request	SetJobHookEnabledRequest
Response	SetJobHookEnabledResponse
Description	Enables or disables a job hook

GetActiveJobHooksByTiming

Method	GetActiveJobHooksByTiming
Request	GetActiveJobHooksByTimingRequest
Response	GetActiveJobHooksByTimingResponse
Description	Returns job hooks that are enabled by a specific timing. They will be sorted by priority, created_at, and id ascending.

---