Open Source Projects

Task / worker pool manager in Go

* Start cli tasks automatically * Maintain the desired number of worker processes for each task * Handle automatic restarts when a worker dies or stalls The task manager will be able to start any cli (shell) script from the chosen directory. For tasks that are long-running and meant to be monitored continuously, each worker process should regular keep-alive messages via a ZeroMQ PUB-SUB channel to communicate its health, and should handle SIGTERM messages when asked to terminate. If the worker doesn't respond to a SIGTERM signal, it will be killed with SIGKILL after a (configurable) grace period. The number of workers stalled/stopped since the task manager was started is reported in the task status.

The main package is the Task Manager (wpmanager), which can load the configuration, start some tasks automatically, and handle signals (CTRL+C) and HTTP requests to control the state of the tasks.
There's also a sample console application (wpconsole) to control the status and the configuration of each task from the command line. It supports controlling task managers running on different hosts/ports, and has a tab-completion interface (for both commands and task names).

Lorenzo Alberton

Lorenzo Alberton Lorenzo has been working with large enterprise UK companies for the past 10+ years and is currently CTO at DataSift. He's an international conference speaker and a long-time contributor to many open source projects. Lorenzo Alberton's profile on GitHub Lorenzo Alberton's profile on LinkedIN View Lorenzo Alberton's profile on PHP PEAR
View Lorenzo Alberton's Twitter stream Lorenzo Alberton - Sun Certified MySQL 5 Developer PHP5 ZCE - Zend Certified Engineer

Tags

AJAX, Apache, Book Review, Charset, Cheat Sheet, Data structures, Database, Firebird SQL, Hadoop, Imagick, INFORMATION_SCHEMA, JavaScript, Kafka, Linux, Message Queues, mod_rewrite, Monitoring, MySQL, NoSQL, Oracle, PDO, PEAR, Performance, PHP, PostgreSQL, Profiling, Scalability, Security, SPL, SQL Server, SQLite, Testing, Tutorial, TYPO3, Windows, Zend Framework