svc_topics v0.1.1

Topics Service

Docker Image

docker build -t svc_topics . // OR
docker build .

docker run -d -p 3000:3000 -v $(pwd)/data/log:/data/log svc_topics
docker stop -t 0 imageId
docker rm `docker ps -a -q`

#Kubectl

gcloud container clusters get-credentials jin-microservices-cluster-staging --zone us-central1-b # Setup gcloud and kubectl credentials
kubectl create -f all-in-one-service-topics.yaml # Create a Service and a Replica Set

Mission

• Manage Topics in a "CRUD" way
• Split Topics by source
• Instruct the Crawler to fetch data

Persitance

• Topics model => MySQL
• Topics health check (exhaustivity, volumetry, real-time Index) => Redis
• Feeds health check (exhaustivity, volumetry, real-time Index) => Redis
• Service Status => Redis

Definitions

Topic

Entity with a name and id, owned by a single client, that is created, updated and deleted by an admin user through the Plugr Interface. Each topic is split by source in feeds.

Source

One of the following:

• Facebook
• Twitter
• Instagram
• Google Plus
• Youtube
• Web (Google News & soon Wordpress)
• Forum

Feed

A feed represents a fragment of a topic. Because crawling feeds will be done in various ways and following different constraints, each feed shall be managed and optimized accordingly.

Interface

• /api/v1/topics/:topicId
• /api/v1/feeds/:feedId (not available for the moment, but shall be implemented)
• POST /api/v1/topics/:topicId/fetch-now (when a new topic is created or updated, or a topic needs to be speeded up. This shall be throttled)
• GET /api/v1/topics/:topicId/health-check (returns data for pie or series of the three indices mentioned above)
• GET /api/v1/feeds/:feedId/health-check (returns data for pie or series of the three indices mentioned above)
• PUT /api/v1/feeds/:feedId/metas (the crawler will send meta data here)
• GET /api/v1/health-check (returns uptime information, and health check of past hour operations)

Outerface

• POST {crawler-service-host}/api/v1/feeds/:feedId

  {
      success: true,
      crawled: {
          timestamp_from: 1407986200,
          timestamp_to: 14079900,
      }
  }

TODO

• read all topics for all clients from mySQL and store it in Redis as resilience manager
• chunk topics down to feeds
• manager controller that pings ControllerService
• routes & interfaces

  6b3d0185f1a19a59640aeb43c2d28f229cbe13bb