@natlibfi/melinda-rest-api-commons NPM

Shared modules for microservices of Melinda rest api import system

Node Version

Shared modules for microservices of Melinda rest api import system

Constants

QUEUE_ITEM_STATE

QueueItemState flow for bulk jobs (multiple records, no validations):

microservice	states
`http`	-> VALIDATOR.UPLOADING -> VALIDATOR.PENDING_QUEUING ->
`validator`	VALIDATOR.QUEUING_IN_PROGRESS -> IMPORTER.IN_QUEUE ->
`importer`	IMPORTER.IMPORTING -> IMPORT_JOB_STATEs* -> DONE

any	any -> ERROR if errored

*IMPORT_JOB_STATEs: EMPTY or IN_QUEUE, PROCESSING -> final: DONE, ERROR, ABORT

QueueItemState flow for prio jobs (single record):

microservice	states
`http`	-> VALIDATOR.PENDING_VALIDATION ->
`validator`	VALIDATOR.VALIDATING -> IMPORTER.IN_QUEUE ->
`importer`	IMPORTER.IMPORTING -> IMPORT_JOB_STATEs* -> DONE

`http`	any -> ABORT if job has stayed in an active state too long
`validator`	VALIDATOR.VALIDATING -> DONE for noop (no-operation) jobs

any	any -> ERROR if errored

*IMPORT_JOB_STATEs: EMPTY or IN_QUEUE, PROCESSING -> final: DONE, ERROR, ABORT

CHUNK_SIZE

Chunck size to execution time ratio (From file to queue: ~450 records in 15sec)

Total records	Chunk size	Chunks	Execution time	Time / chunk	Operation
9248	100	93	~597 sec	~6.4 sec	Update

Executed on test server using bridge to docker server and back to server

Common confs for aleph-record-load-api

P_manage_18 name	Node name	Prio update	Prio create	Bulk update	Bulk create	Description
p_active_library	pActiveLibrary	`params`	`params`	`params`	`params`	Library to use
p_input_file	pInputFile	`generated`	`generated`	`generated`	`generated`	Source file location
p_reject_file	pRejectFile	`generated`	`generated`	`generated`	`generated`	Log file for rejected records
p_log_file	pLogFile	`generated`	`generated`	`generated`	`generated`	Log file for updated/created record ids
p_old_new	pOldNew	OLD	NEW	OLD	NEW	Method of operation. Either NEW or OLD
p_fix_type	pFixType	API	API	INSB	INSB	Aleph fix routine code
p_check_references	pCheckReferences
p_update_f	pUpdateF	FULL	FULL	FULL	FULL	Indexing action
p_update_type	pUpdateType	REP	REP	REP	REP	REP or APP (REPlace or APPend)
p_update_mode	pUpdateMode	M	M	M	M	User mode. Either M (Multi-user) or S (Single-user)
p_char_conv	pCharConv					Character conversion to apply
p_merge_type	pMergeType					Merge/Preferred routine
p_cataloger_in	pCatalogerIn	`params`	`params`	`params`	`params`	Value which is written to CAT fields
p_cataloger_level_x	pCatalogerLevelX					Cataloger lever
p_z07_priority_year	pZ07PriorityYear	1998	1990	2099	2099	Override indexing priority
p_redirection_field	pRedirectionField

License and copyright

This project's source code is licensed under the terms of MIT or any later version.

@natlibfi/marc-record @natlibfi/marc-record-serializers @natlibfi/marc-record-validate @natlibfi/marc-record-validators-melinda @natlibfi/melinda-backend-commons @natlibfi/melinda-commons amqplib debug http-status moment mongo-sanitize mongodb

@infinitebrahmanuniverse/nolb-_natl @everything-registry/sub-chunk-635

2 months ago

2 months ago

2 months ago

2 months ago

3 months ago

3 months ago

3 months ago

3 months ago

3 months ago

4 months ago

4 months ago

4 months ago

4 months ago

4 months ago

4 months ago

4 months ago

4 months ago

4 months ago

4 months ago

5 months ago

5 months ago

5 months ago

10 months ago

8 months ago

9 months ago

9 months ago

9 months ago

9 months ago

6 months ago

6 months ago

6 months ago

7 months ago

7 months ago

7 months ago

7 months ago

6 months ago

7 months ago

6 months ago

8 months ago

8 months ago

6 months ago

7 months ago

9 months ago

8 months ago

8 months ago

8 months ago

8 months ago

8 months ago

8 months ago

8 months ago

8 months ago

8 months ago

8 months ago

8 months ago

10 months ago

10 months ago

7 months ago

7 months ago

7 months ago

8 months ago

8 months ago

6 months ago

6 months ago

6 months ago

9 months ago

9 months ago

9 months ago

7 months ago

7 months ago

7 months ago

7 months ago

7 months ago

7 months ago

7 months ago

12 months ago

11 months ago

11 months ago

12 months ago

1 year ago

1 year ago

1 year ago

1 year ago

1 year ago

1 year ago

1 year ago

1 year ago

1 year ago

1 year ago

1 year ago

1 year ago

1 year ago

1 year ago

1 year ago

1 year ago

1 year ago

1 year ago

1 year ago

1 year ago

1 year ago

1 year ago

1 year ago

1 year ago

1 year ago

1 year ago

1 year ago

1 year ago

1 year ago

1 year ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

2 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

3 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago

4 years ago