py_pool v1.0.0

Advanced Python Process Pool

This project implements an advanced Python process pool that allows for secure and efficient execution of Python code from a Node.js environment. The pool leverages Python subprocesses, hashing for security, memory usage tracking, and data compression to optimize performance and ensure secure code execution.

Table of Contents

• Introduction
• Features
• Directory Structure
• Installation
• Usage
  • Executing Tasks
  • Handling Errors
  • Checking Worker Status
• Testing
• Error Handling
• Contributing
• License

Introduction

The Advanced Python Process Pool project provides a robust and secure framework for executing Python code in a Node.js environment. By maintaining a pool of persistent Python subprocesses, the system can handle concurrent tasks efficiently. Additionally, it incorporates advanced security measures, error handling, and data compression to ensure reliability and performance.

Features

• Persistent Python Subprocesses: Maintains a pool of Python processes to handle incoming tasks, reducing overhead.
• Advanced Security: Uses hashing, blacklisting of dangerous keywords, and RestrictedPython for secure code execution.
• Memory Usage Tracking: Monitors memory usage of each subprocess for better resource management.
• Adaptive Load Balancing: Selects workers based on memory usage and queue size to balance the load.
• Data Compression: Compresses data sent between Node.js and Python processes to improve performance.
• Error Handling: Detects and handles errors, ensuring the subprocesses are restarted if they fail.

Directory Structure

plaintext advanced-python-process-pool/ ├── src/ │ ├── py_pool.js # Node.js client that interacts with Python processes ├── test/ │ ├── test.js # Test suite for the Node.js client ├── .gitignore # Git ignore file ├── LICENSE # License information ├── package.json # NPM package metadata and scripts ├── README.md # Project documentation

• src/: Contains the source files related to the project.
  • py_pool.js: The Node.js client script that interacts with the Python processes.
• test/: Contains the test files.
  • test.js: The test suite for the Node.js client.
• .gitignore: Specifies files and directories Git should ignore.
• LICENSE: Contains the license information for the package.
• package.json: Contains metadata about the package and its dependencies.
• README.md: Provides documentation for the package.

Installation

1. Clone the Repository: bash git clone https://github.com/kunaalhere/PyPooling.git cd PyPooling

2. Install Dependencies: bash npm install

Usage

Executing Tasks

Use the execute method to send tasks to the pool and receive results.

javascript const PythonPool = require('./src/py_pool');

async function runTask() { try { const result = await PythonPool.execute('2 + 2'); console.log(Result: ${result}); // Output: Result: 4 } catch (error) { console.error(Error: ${error.message}); } }

runTask();

Handling Errors

The execute method handles errors gracefully, restarting failed processes if necessary.

javascript async function runTaskWithError() { try { const result = await PythonPool.execute('raise ValueError("Test Error")'); } catch (error) { console.error(Error: ${error.message}); // Output: Error: Test Error } }

runTaskWithError();

Checking Worker Status

Use the getWorkerStatus method to check the status of each worker in the pool.

javascript function checkStatus() { const status = PythonPool.getWorkerStatus(); console.log('Worker Status:', status); }

checkStatus();

Testing

Run the tests to ensure the system behaves as expected.

Running Tests: 1. Ensure you have the necessary dependencies installed: bash npm install

2. Execute the tests: bash npm test

Error Handling

The module includes comprehensive error handling to ensure the Python subprocesses are restarted if they encounter an error or exit unexpectedly. Errors from the Python processes are logged to the console, and the processes are restarted as needed.

Contributing

Contributions are welcome! Please follow these steps: 1. Fork the repository. 2. Create a new branch (git checkout -b feature/your-feature). 3. Commit your changes (git commit -am 'Add your feature'). 4. Push to the branch (git push origin feature/your-feature). 5. Create a new Pull Request.

License

This project is licensed under the MIT License. See the LICENSE file for details.

Conclusion

The Advanced Python Process Pool project provides a secure, efficient, and scalable solution for executing Python code in a Node.js environment. With advanced features like adaptive load balancing, data compression, and comprehensive error handling, this project ensures reliable and performant code execution.