@natlibfi/melinda-rest-api-commons NPM

Shared modules for microservices of Melinda rest api import system

Node Version

Shared modules for microservices of Melinda rest api import system

Constants

QUEUE_ITEM_STATE

QueueItemState flow for bulk jobs (multiple records, no validations):

microservice	states
`http`	-> VALIDATOR.UPLOADING -> VALIDATOR.PENDING_QUEUING ->
`validator`	VALIDATOR.QUEUING_IN_PROGRESS -> IMPORTER.IN_QUEUE ->
`importer`	IMPORTER.IMPORTING -> IMPORT_JOB_STATEs* -> DONE

any	any -> ERROR if errored

*IMPORT_JOB_STATEs: EMPTY or IN_QUEUE, PROCESSING -> final: DONE, ERROR, ABORT

QueueItemState flow for prio jobs (single record):

microservice	states
`http`	-> VALIDATOR.PENDING_VALIDATION ->
`validator`	VALIDATOR.VALIDATING -> IMPORTER.IN_QUEUE ->
`importer`	IMPORTER.IMPORTING -> IMPORT_JOB_STATEs* -> DONE

`http`	any -> ABORT if job has stayed in an active state too long
`validator`	VALIDATOR.VALIDATING -> DONE for noop (no-operation) jobs

any	any -> ERROR if errored

*IMPORT_JOB_STATEs: EMPTY or IN_QUEUE, PROCESSING -> final: DONE, ERROR, ABORT

CHUNK_SIZE

Chunck size to execution time ratio (From file to queue: ~450 records in 15sec)

Total records	Chunk size	Chunks	Execution time	Time / chunk	Operation
9248	100	93	~597 sec	~6.4 sec	Update

Executed on test server using bridge to docker server and back to server

Common confs for aleph-record-load-api

P_manage_18 name	Node name	Prio update	Prio create	Bulk update	Bulk create	Description
p_active_library	pActiveLibrary	`params`	`params`	`params`	`params`	Library to use
p_input_file	pInputFile	`generated`	`generated`	`generated`	`generated`	Source file location
p_reject_file	pRejectFile	`generated`	`generated`	`generated`	`generated`	Log file for rejected records
p_log_file	pLogFile	`generated`	`generated`	`generated`	`generated`	Log file for updated/created record ids
p_old_new	pOldNew	OLD	NEW	OLD	NEW	Method of operation. Either NEW or OLD
p_fix_type	pFixType	API	API	INSB	INSB	Aleph fix routine code
p_check_references	pCheckReferences
p_update_f	pUpdateF	FULL	FULL	FULL	FULL	Indexing action
p_update_type	pUpdateType	REP	REP	REP	REP	REP or APP (REPlace or APPend)
p_update_mode	pUpdateMode	M	M	M	M	User mode. Either M (Multi-user) or S (Single-user)
p_char_conv	pCharConv					Character conversion to apply
p_merge_type	pMergeType					Merge/Preferred routine
p_cataloger_in	pCatalogerIn	`params`	`params`	`params`	`params`	Value which is written to CAT fields
p_cataloger_level_x	pCatalogerLevelX					Cataloger lever
p_z07_priority_year	pZ07PriorityYear	1998	1990	2099	2099	Override indexing priority
p_redirection_field	pRedirectionField

License and copyright

This project's source code is licensed under the terms of MIT or any later version.

@natlibfi/marc-record @natlibfi/marc-record-serializers @natlibfi/marc-record-validate @natlibfi/marc-record-validators-melinda @natlibfi/melinda-backend-commons @natlibfi/melinda-commons amqplib debug http-status moment mongo-sanitize mongodb

@infinitebrahmanuniverse/nolb-_natl @everything-registry/sub-chunk-635

10 months ago

10 months ago

11 months ago

11 months ago

11 months ago

11 months ago

11 months ago

11 months ago

11 months ago

11 months ago

11 months ago

12 months ago

11 months ago

11 months ago

12 months ago

1 year ago

1 year ago

1 year ago

1 year ago

1 year ago

1 year ago

1 year ago

1 year ago

1 year ago

1 year ago

1 year ago

1 year ago

1 year ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago

5 years ago